-
Notifications
You must be signed in to change notification settings - Fork 302
/
rucio-necromancer
executable file
·120 lines (90 loc) · 5.49 KB
/
rucio-necromancer
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
#!/usr/bin/env python3
# Copyright European Organization for Nuclear Research (CERN) since 2012
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
'''
Necromancer Daemon : Bring the dead files back to life
'''
import argparse
import signal
from rucio.daemons.badreplicas.necromancer import run, stop
def get_parser():
"""
Returns the argparse parser.
"""
parser = argparse.ArgumentParser(description="The Necromancer daemon is responsible for managing bad replicas. If a replica that got declared bad has other replicas, it will try to recover it by requesting a new transfer. If there are no replicas anymore, then the file gets marked as lost.", epilog='''
Lost replica:
In this example the file gets uploaded and will only have this replica as there are no replication rules. If it gets declared bad, there will be no replica to recover from.
Therefor the replica gets marked as lost.
Upload a file::
$ rucio upload --scope mock --rse MOCK --name file filename.txt
Check replicas::
$ rucio list-file-replicas mock:file
+---------+--------+------------+-----------+---------------------------------------------------------+
| SCOPE | NAME | FILESIZE | ADLER32 | RSE: REPLICA |
|---------+--------+------------+-----------+---------------------------------------------------------|
| mock | file | 149.000 B | 948240 | MOCK: file://localhost:0/tmp/rucio_rse/mock/fb/d1/file |
+---------+--------+------------+-----------+---------------------------------------------------------+
Declare it as bad::
$ rucio-admin replicas declare-bad file://localhost:0/tmp/rucio_rse/mock/fb/d1/file --reason 'bad'
Run the daemon::
$ rucio-necromancer --run-once
Check replicas again::
$ rucio list-file-replicas mock:file
+---------+--------+------------+-----------+----------------+
| SCOPE | NAME | FILESIZE | ADLER32 | RSE: REPLICA |
|---------+--------+------------+-----------+----------------|
+---------+--------+------------+-----------+----------------+
Bad replica:
In this example the file gets uploaded and will have two replicas. If it gets declared bad, then the daemon will try to recover it from the second replica.
Upload a file and replicate it::
$ rucio upload --scope mock --rse MOCK filename.txt
$ rucio add-rule mock:file 1 MOCK2
$ rucio-conveyor-submitter --run-once
Check replicas::
$ rucio list-file-replicas mock:file
+---------+--------+------------+-----------+---------------------------------------------------------+
| SCOPE | NAME | FILESIZE | ADLER32 | RSE: REPLICA |
|---------+--------+------------+-----------+---------------------------------------------------------|
| mock | file | 149.000 B | 948240 | MOCK: file://localhost:0/tmp/rucio_rse/mock/fb/d1/file |
|---------+--------+------------+-----------+---------------------------------------------------------|
| mock | file | 149.000 B | 948240 | MOCK2: file://localhost:1/tmp/rucio_rse/mock/fb/d1/file |
+---------+--------+------------+-----------+---------------------------------------------------------+
Declare one replica as bad::
$ rucio-admin replicas declare-bad file://localhost:1/tmp/rucio_rse/mock/fb/d1/file --reason 'bad'
Run the daemon::
$ rucio-necromancer --run-once
Check replicas again::
$ rucio list-file-replicas mock:file
+---------+--------+------------+-----------+---------------------------------------------------------+
| SCOPE | NAME | FILESIZE | ADLER32 | RSE: REPLICA |
|---------+--------+------------+-----------+---------------------------------------------------------|
| mock | file | 149.000 B | 948240 | MOCK: file://localhost:0/tmp/rucio_rse/mock/fb/d1/file |
|---------+--------+------------+-----------+---------------------------------------------------------|
| mock | file | 149.000 B | 948240 | MOCK2: file://localhost:1/tmp/rucio_rse/mock/fb/d1/file |
+---------+--------+------------+-----------+---------------------------------------------------------+
''', formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--run-once", action="store_true", default=False, help='Runs one loop iteration')
parser.add_argument("--threads", action="store", default=1, type=int, help='Concurrency control: number of threads')
parser.add_argument("--bulk", action="store", default=1000, type=int, help='Bulk control: number of requests per cycle')
parser.add_argument('--sleep-time', action="store", default=60, type=int, help='Concurrency control: thread sleep time after each chunk of work')
return parser
if __name__ == "__main__":
# Bind our callback to the SIGTERM signal and run the daemon:
signal.signal(signal.SIGTERM, stop)
parser = get_parser()
args = parser.parse_args()
try:
run(threads=args.threads, bulk=args.bulk, once=args.run_once, sleep_time=args.sleep_time)
except KeyboardInterrupt:
stop()