Rucio 32.5.0.post1 hotfix release

Hotfix release due to rebuild of the containers.

Rucio 32.5.0 minor release

General

Features

• Authentication & Authorisation: Incorporate IAM-based testing for token workflows #6188

Bugs

• Functional tests: Automatix.json default path is incorrect. #6352

Rucio 32.4.0 minor release

General

Features

• Transfers: scitags/packet marking support #5856

Enhancements

• Core & Internals: Improve error reporting during upload when reusing a deleted DID #6091

Bugs

• Testing: which executable is not available in the testing environment #6303

Rucio 32.3.1 patch release

General

Bugs

• Core & Internals: Deprecated minio variables in dev docker-compose #6324
• Documentation: Confusing REST API Docs for/rses (list rses) #6327
• Protocols: Webdav Protocol stat does not return data as specified #5977
• Recovery: Files without datatype being declared bad. #6318

Rucio 32.3.0 minor release

General

Enhancements

• Authentication & Authorisation: sqlalchemy deprecation warning in oauthmanager #5838
• Core & Internals: rse_settings dictionary datatype #5972
• Transfers: Allow for tape-to-tape transfer for QoS RSE #6259

Rucio 1.29.14 LTS patch release

General

Bugs

• Authentication & Authorisation: Restrict permissions for adding scopes or manipulating identities #6267

Rucio 32.2.0 minor release

General

Enhancements

• Core & Internals: Replace most uses of locals() #6310

Bugs

• Deletion: Remove mentions of the MaxBeingDeletedFiles RSE limit #6313

Rucio 32.1.0 minor release

General

Features

• Rules: Add size information to list_rules() #5978

Enhancements

• Core & Internals: Make use of ModelBase.to_dict() wherever applicable #6307

Rucio 32.0.0 "The Good, The Bad and the Donkey" major release

Long Term Support (LTS)

This release marks the start of the 32 Long Term Support (LTS) release line. This release line will be supported with security and critical patches until at least August 2025.

Upgrade Instructions

This feature release requires a database schema upgrade. Please consult the documentation about upgrading your database schema.

The following changes are necessary and are covered by the alembic upgrade head command:

1. Renaming index RULES_STUCKSTATE_IDX to RULES_STATE_IDX (Alembic revision a6eb23955c28)

2. Adding last_processed_by and last_processed_at column to requests table (Alembic revision 295289b5a800)

3. Drop of indices REPLICAS_TOMBSTONE_IDX and REPLICAS_RSE_ID_IDX (Alembic revision 27e3a68927fb)

4. Only for ATLAS and CMS, because they run specific oracle database schemas which diverge from the standard SQLAlchemy schema in alembic: Some additional steps have to be performed during database schema update by these two communities. See the relevant commit message: 81bacae

Important changes

• Drop support for python < 3.9.

• The temporary tables workflow is now mandatory. This feature was gradually introduced over many previous releases. First it was opt-in, since 1.31.0 it is an opt-out. Starting the current release, the old behavior is completely removed and the usage of temporary tables is mandatory. The update should be transparent, but requires the database user used by rucio to be allowed to create temporary tables in the database. The configuration switch [core]use_temp_tables=[True|False] can now be removed, as it does not do anything.

• The configuration switch [transfers]use_multihop=[True|False] isn't used by rucio anymore and can be removed. Use [transfers]multihop_rse_expression=<some_rse_expression> to define which RSEs can be used for multihop. It defaults to the RSE-expression available_for_multihop=true, meaning that the update should be transparent for everybody. Communities who did not use multi-hop should not have any RSEs with this attribute set. Communities who used multi-hop already had this attribute set on all required RSEs, as it was mandatory for multi-hopping to work correctly. To explicitly disable multi-hopping, set [transfers]multihop_rse_expression= to an empty value.

• The transfers(conveyor) daemons where subjected to a huge internal re-design. The --threads N CLI switch now has a slightly different meaning. Instead of having N threads getting data from database and working on the retrieved data, there will now be 1 thread getting data from the database + N threads working on that data. This change should have no particular impact.

• As part of the token workflow evolution, Rucio will need to cache already-acquired tokens so that it may reuse them when possible (thereby reducing the amount of token requests sent to the Identity Provider). Consequently, Rucio instance administrators should be aware that the Memcached cache may now hold security-critical information and should take all necessary steps to ensure that access to it is restricted.

• This release allows configuring RSE protocols with NULL priorities to mean disabled for the respective activities. However, instance administrators should be advised that making use of this breaks backward compatibility with clients older than 32.0.0.

Helm chart changes

Note: Technically, rucio 32.0.0 can still be run with the last helm charts of the 1.31 line. You are not required to update the helm values at the same time you update rucio if you want to dissociate updating rucio from updating the helm charts. However, it's strongly advised to update helm charts rapidly after rucio update. We don't guarantee that this compatibility will be maintained in rucio 32.1.0+

breaking change: rework of the additionalSecrets section

This change was anounced in previous rucio release 1.31.0. The current release makes the new behavior mandatory. Follow the instructions from the previous release to substitute the additionalSecrets sections in helm values with secretMounts sections.

breaking change introduced in the previous release: 1.31.x of rucio-servers helm chart is incompatible with old values

If you are updating from a helm chart prior to 1.31.0, please follow the instruction from the previous rucio release 1.31.0 to upgrade the value of the rucio-server helm charts.

General

Features

• Authentication & Authorisation: Replace all instances of X509 identities presented in the legacy format #6182
• Core & Internals: make temporary tables mandatory #6238
• Database: Exchange of function-based indices with normal index #6143
• Release management: drop support for python 3.6, 3.7 and 3.8 #6251
• Transfers: New dispatcher logic for conveyors #6190

Enhancements

• Authentication & Authorisation: Extend /auth/x509 endpoint for browser-based clients #6048
• Core & Internals: Protocol Priority Renumbering #6083
• Core & Internals: finalize migration to unified config table + config file #6107
• Core & Internals: Customisable replica sorter algorithm #6114
• Core & Internals: Support protocols with NULL priority #6201
• Core & Internals: Remove prefixes for unicode string literals #6268
• Messaging: Get rid of hermes-legacy #6213
• Remove deprecated constructs from the typing module #6276
• Replicas: remove REPLICAS_TOMBSTONE_IDX and REPLICAS_RSE_ID_IDX indexes #6287
• Transfers: get rid of the use_multihop configuration option #6164
• Transfers: internal refactoring in the conveyor #6207

Bugs

• Authentication & Authorisation: Restrict permissions for adding scopes or manipulating identities #6267
• Metadata: set_metadata_bulk doesn't work for external postgres / mongo metadata plugins #6246
• x509 authentication fails for baseclient.py #6294
• Protocols: CMS LFN to PFN translation gives double hosts #6096
• Transfers: The Conveyor is not transitioning some requests in failed states #6300

Clients

Bugs

• Clients: broken stats retrieval in the xrootd rse protocol #6265

Rucio 32.0.0rc2 major release candidate

Release Candidate

This is a release candidate! Only to be used in testing environments!

Upgrade Instructions

This feature release requires a database schema upgrade. Please consult the documentation about upgrading your database schema.

The following changes are necessary and are covered by the alembic upgrade head command:

1. Renaming index RULES_STUCKSTATE_IDX to RULES_STATE_IDX (Alembic revision a6eb23955c28)

2. Adding last_processed_by and last_processed_at column to requests table (Alembic revision 295289b5a800)

3. Drop of indices REPLICAS_TOMBSTONE_IDX and REPLICAS_RSE_ID_IDX (Alembic revision 27e3a68927fb)

4. Only for ATLAS and CMS, because they run specific oracle database schemas which diverge from the standard SQLAlchemy schema in alembic: Some additional steps have to be performed during database schema update by these two communities. See the relevant commit message: 81bacae

Important changes

• Drop support for python < 3.9.

• The temporary tables workflow is now mandatory. This feature was gradually introduced over many previous releases. First it was opt-in, since 1.31.0 it is an opt-out. Starting the current release, the old behavior is completely removed and the usage of temporary tables is mandatory. The update should be transparent, but requires the database user used by rucio to be allowed to create temporary tables in the database. The configuration switch [core]use_temp_tables=[True|False] can now be removed, as it does not do anything.

• The configuration switch [transfers]use_multihop=[True|False] isn't used by rucio anymore and can be removed. Use [transfers]multihop_rse_expression=<some_rse_expression> to define which RSEs can be used for multihop. It defaults to the RSE-expression available_for_multihop=true, meaning that the update should be transparent for everybody. Communities who did not use multi-hop should not have any RSEs with this attribute set. Communities who used multi-hop already had this attribute set on all required RSEs, as it was mandatory for multi-hopping to work correctly. To explicitly disable multi-hopping, set [transfers]multihop_rse_expression= to an empty value.

• The transfers(conveyor) daemons where subjected to a huge internal re-design. The --threads N CLI switch now has a slightly different meaning. Instead of having N threads getting data from database and working on the retrieved data, there will now be 1 thread getting data from the database + N threads working on that data. This change should have no particular impact.

Helm chart changes

Note: Technically, rucio 32.0.0 can still be run with the last helm charts of the 1.31 line. You are not required to update the helm values at the same time you update rucio if you want to dissociate updating rucio from updating the helm charts. However, it's strongly advised to update helm charts rapidly after rucio update. We don't guarantee that this compatibility will be maintained in rucio 32.1.0+

breaking change: rework of the additionalSecrets section

This change was anounced in previous rucio release 1.31.0. The current release makes the new behavior mandatory. Follow the instructions from the previous release to substitute the additionalSecrets sections in helm values with secretMounts sections.

breaking change introduced in the previous release: 1.31.x of rucio-servers helm chart is incompatible with old values

If you are updating from a helm chart prior to 1.31.0, please follow the instruction from the previous rucio release 1.31.0 to upgrade the value of the rucio-server helm charts.

General

Features

• Authentication & Authorisation: Replace all instances of X509 identities presented in the legacy format #6182
• Core & Internals: make temporary tables mandatory #6238
• Database: Exchange of function-based indices with normal index #6143
• Release management: drop support for python 3.6, 3.7 and 3.8 #6251
• Transfers: New dispatcher logic for conveyors #6190

Enhancements

• Authentication & Authorisation: Extend /auth/x509 endpoint for browser-based clients #6048
• Core & Internals: Protocol Priority Renumbering #6083
• Core & Internals: finalize migration to unified config table + config file #6107
• Core & Internals: Customisable replica sorter algorithm #6114
• Core & Internals: Support protocols with NULL priority #6201
• Core & Internals: Remove prefixes for unicode string literals #6268
• Messaging: Get rid of hermes-legacy #6213
• Remove deprecated constructs from the typing module #6276
• Replicas: remove REPLICAS_TOMBSTONE_IDX and REPLICAS_RSE_ID_IDX indexes #6287
• Transfers: get rid of the use_multihop configuration option #6164
• Transfers: internal refactoring in the conveyor #6207

Bugs

• Authentication & Authorisation: Restrict permissions for adding scopes or manipulating identities #6267
• Metadata: set_metadata_bulk doesn't work for external postgres / mongo metadata plugins #6246
• x509 authentication fails for baseclient.py #6294
• Protocols: CMS LFN to PFN translation gives double hosts #6096

Clients

Bugs

• Clients: broken stats retrieval in the xrootd rse protocol #6265