Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Should process_s3_log_file_task be removed/turned off? #1900

Open
yarikoptic opened this issue Mar 20, 2024 · 0 comments
Open

Should process_s3_log_file_task be removed/turned off? #1900

yarikoptic opened this issue Mar 20, 2024 · 0 comments
Assignees
Labels
performance Improve performance of an existing feature

Comments

@yarikoptic
Copy link
Member

yarikoptic commented Mar 20, 2024

It was added, improved in #1605 .
But AFAIK we do never cared about that collected global download count statistic and it is not displayed/used anywhere. Or is it? (I even failed to figure out in admin django interface where it is )

But I see it quite active.
But if not used, why bother? (especially since we are trying to figure out what makes dandi archive slow ATM).

relevant issues:

and ATM we are just harvesting full S3 logs on drogon server with @CodyCBakerPhD doing initial analytics/visualization so we pin point target needed collection to actually implement/collect long term.

here are some stats for the recent days when we experience slow downness -- cannot be attributed to this really -- we have been consistent in wasting cycles
dandi@drogon:/mnt/backup/dandi/heroku-logs/dandi-api$ for d in 0{1..9} {10..21}; do echo -n "$d "; grep process_s3_log_file_task.*received 202403$d-*.log | wc -l; done
01 25726
02 12579
03 2225
04 28647
05 28418
06 26219
07 33895
08 12287
09 33439
10 30243
11 39352
12 39071
13 45778
14 39987
15 37676
16 39987
17 39490
18 44155
19 38521
20 24035
21 7703
@yarikoptic yarikoptic added the performance Improve performance of an existing feature label Mar 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Improve performance of an existing feature
Projects
None yet
Development

No branches or pull requests

2 participants