Crash of the replica-recoverer when adding rules

[cserf]

Description

The replica-recoverer sometimes crashes if it tries to add a rule that already exists :

{"process": {"pid": 1, "name": "suspicious-replica-recoverer", "thread": {"name": "Thread-2", "id": 139697899333184}}, "log": {"level": "CRITICAL", "logger": "root"}, "message": "[1/2]: Exception\nA duplicate rule for this account, did, rse_expression, copies already exists.\nDetails: (cx_Oracle.IntegrityError) ORA-00001: unique constraint (ATLAS_RUCIO.RULES_SCOPE_ACC_NAME_CO_RSE_UQ) violated\n  File \"/usr/local/lib/python3.9/site-packages/rucio/daemons/common.py\", line 216, in _generator\n    result = run_once_fnc(heartbeat_handler=heartbeat_handler, activity=activity)\n  File \"/usr/local/lib/python3.9/site-packages/rucio/daemons/replicarecoverer/suspicious_replica_recoverer.py\", line 364, in run_once\n    add_rule(dids=dids_nattempts_1, account=InternalAccount('root', vo=vo), copies=nattempts, rse_expression='type=SCRATCHDISK', grouping=None, weight=None, lifetime=5 * 24 * 3600, locked=False, subscription_id=None)\n  File \"/usr/local/lib/python3.9/site-packages/rucio/db/sqla/session.py\", line 454, in new_funct\n    result = function(*args, session=session, **kwargs)\n  File \"/usr/local/lib/python3.9/site-packages/rucio/core/rule.py\", line 366, in add_rule\n    raise DuplicateRule(error.args[0]) from error\n", "error": {"type": "DuplicateRule", "message": "A duplicate rule for this account, did, rse_expression, copies already exists.\nDetails: (cx_Oracle.IntegrityError) ORA-00001: unique constraint (ATLAS_RUCIO.RULES_SCOPE_ACC_NAME_CO_RSE_UQ) violated", "stack_trace": "  File \"/usr/local/lib/python3.9/site-packages/rucio/daemons/common.py\", line 216, in _generator\n    result = run_once_fnc(heartbeat_handler=heartbeat_handler, activity=activity)\n  File \"/usr/local/lib/python3.9/site-packages/rucio/daemons/replicarecoverer/suspicious_replica_recoverer.py\", line 364, in run_once\n    add_rule(dids=dids_nattempts_1, account=InternalAccount('root', vo=vo), copies=nattempts, rse_expression='type=SCRATCHDISK', grouping=None, weight=None, lifetime=5 * 24 * 3600, locked=False, subscription_id=None)\n  File \"/usr/local/lib/python3.9/site-packages/rucio/db/sqla/session.py\", line 454, in new_funct\n    result = function(*args, session=session, **kwargs)\n  File \"/usr/local/lib/python3.9/site-packages/rucio/core/rule.py\", line 366, in add_rule\n    raise DuplicateRule(error.args[0]) from error\n"}, "@timestamp": "2024-03-25T08:11:08.355Z"}

A protection needs to be added here : https://github.com/rucio/rucio/blob/master/lib/rucio/daemons/replicarecoverer/suspicious_replica_recoverer.py#L363

Steps to reproduce

Rucio Version

No response

Additional Information

No response