forked from rucio/rucio
/
rucio-judge-evaluator
executable file
·137 lines (106 loc) · 7.29 KB
/
rucio-judge-evaluator
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
#!/usr/bin/env python3
# Copyright European Organization for Nuclear Research (CERN) since 2012
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Judge-Evaluator is a daemon to re-evaluate replication rules.
"""
import argparse
import signal
from rucio.daemons.judge.evaluator import run, stop
def get_parser():
"""
Returns the argparse parser.
"""
parser = argparse.ArgumentParser(description="The Judge-Evaluator daemon is responsible for execution and reevaluation of replication rules.\
First it checks if there are DIDs that have changed content e.g. attached or detached DIDs.In case of a new attachment, the replication rule for\
the dataset has to be applied to the attached DID, too. If the attached DID has already a replica on a RSE that satisfies the RSE expression of the\
rule, then the lock counter of that replica gets increased. If it does not have any replica satisfying the rule, then a new replica has to be created.\
In case of a new detachment, the replica has to be removed or the lock counter of the replica has to be decreased, depending on which RSE the replica exist.\
If the DID is a dataset, its properties like size and length also get updated and also an entry is saved to mark a change for possible collection replicas which have to be updated by another daemon.", epilog='''
Same RSEs:
Create a dataset with a replication rule and upload a file to the same RSE. Then attach it to the dataset::
$ rucio add-dataset mock:dataset
$ rucio add-rule mock:dataset 1 MOCK
$ rucio upload --scope mock --rse MOCK --name file filename.txt
$ rucio attach mock:dataset mock:file
Check the rules and locks for the dataset and the replica::
$ rucio list-rules mock:dataset
ID ACCOUNT SCOPE:NAME STATE[OK/REPL/STUCK] RSE_EXPRESSION COPIES EXPIRES (UTC) CREATED (UTC)
-------------------------------- --------- ------------ ---------------------- ---------------- -------- --------------- -------------------
e95941c165d54e38b6e46990de06d3d4 root mock:dataset OK[0/0/0] MOCK 1 2018-12-03 12:35:43
$ rucio list-rule mock:file
ID ACCOUNT SCOPE:NAME STATE[OK/REPL/STUCK] RSE_EXPRESSION COPIES EXPIRES (UTC) CREATED (UTC)
-------------------------------- --------- ------------ ---------------------- ---------------- -------- --------------- -------------------
cfec9a944cdd4543b7267a34a2584631 root mock:file OK[1/0/0] MOCK 1 2018-12-11 08:29:49
$ python
from rucio.db.sqla import session, models
from rucio.core.rse import get_rse_id
rse_id = get_rse_id('MOCK')
session.get_session().query(models.RSEFileAssociation).filter_by(name='file', scope='mock', rse_id=rse_id).first().lock_cnt // 1
There is one rule for the dataset which we created before and one lock and one rule for the replica which got created with the upload of the file.
Run the daemon::
$ rucio-judge-evaluator --run-once
Check the locks for the replica again::
$ python
from rucio.db.sqla import session, models
from rucio.core.rse import get_rse_id
rse_id = get_rse_id('MOCK')
session.get_session().query(models.RSEFileAssociation).filter_by(name='file', scope='mock', rse_id=rse_id).first().lock_cnt // 2
The replica of the DID mock:file has now 2 locks on RSE MOCK, because it is protected by the replication rule of the dataset and the first replication rule
Different RSEs:
Create a dataset with a replication rule and upload a file to another RSE. Then attach it to the dataset::
$ rucio add-dataset mock:dataset
$ rucio add-rule mock:dataset 1 MOCK
$ rucio upload --scope mock --rse MOCK2 --name file filename.txt
$ rucio attach mock:dataset mock:file
Check the rules and locks for the dataset and the replica::
$ rucio list-rules mock:dataset
ID ACCOUNT SCOPE:NAME STATE[OK/REPL/STUCK] RSE_EXPRESSION COPIES EXPIRES (UTC) CREATED (UTC)
-------------------------------- --------- ------------ ---------------------- ---------------- -------- --------------- -------------------
e95941c165d54e38b6e46990de06d3d4 root mock:dataset OK[0/0/0] MOCK 1 2018-12-03 12:35:43
$ rucio list-rule mock:file
ID ACCOUNT SCOPE:NAME STATE[OK/REPL/STUCK] RSE_EXPRESSION COPIES EXPIRES (UTC) CREATED (UTC)
-------------------------------- --------- ------------ ---------------------- ---------------- -------- --------------- -------------------
cfec9a944cdd4543b7267a34a2584631 root mock:file OK[1/0/0] MOCK2 1 2018-12-11 08:29:49
$ python
from rucio.db.sqla import session, models
from rucio.core.rse import get_rse_id
rse_id = get_rse_id('MOCK2')
session.get_session().query(models.RSEFileAssociation).filter_by(name='file', scope='mock', rse_id=rse_id).first().lock_cnt // 1
There is one rule for the dataset which we created before and one lock and one rule for the replica which got created with the upload of the file.
Run the daemon::
$ rucio-judge-evaluator --run-once
Check the replicas for the DID mock:file::
$ python
from rucio.db.sqla import session, models
session.get_session().query(models.RSEFileAssociation).filter_by(name='file', scope='mock').first()
// [{'name': 'file','lock_cnt': 1, 'state': COPYING, 'scope': 'mock', 'rse_id': 'f81f366593754c01b0c340fa5ea0ab90'},
// {'scope': 'mock', 'rse_id': '1330d5daee37474c88ba888101d7b859', 'name': 'file', 'state': AVAIABLE, 'lock_cnt': 1}]
The DID mock:file has now two replicas with one lock each.
As the file replica is attached to the dataset and the rule for the dataset specifies another RSE MOCK instead of the upload RSE, it has to be replicated to this RSE.
Therefor a second replica in state COPYING got created on RSE MOCK.
''', formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--run-once", action="store_true", default=False, help='One iteration only')
parser.add_argument("--threads", action="store", default=1, type=int, help='Concurrency control: total number of threads for this process')
parser.add_argument('--sleep-time', action="store", default=30, type=int, help='Concurrency control: thread sleep time after each chunk of work')
parser.add_argument("--did-limit", action="store", default=100, type=int, help='Maximum number of dids to evaluate')
return parser
if __name__ == "__main__":
signal.signal(signal.SIGTERM, stop)
parser = get_parser()
args = parser.parse_args()
try:
run(once=args.run_once, threads=args.threads, sleep_time=args.sleep_time, did_limit=args.did_limit)
except KeyboardInterrupt:
stop()