Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add check/handle gracefully attempt to upload/rename asset to have path longer than 512 characters #1889

Open
yarikoptic opened this issue Mar 7, 2024 · 1 comment
Open

Add check/handle gracefully attempt to upload/rename asset to have path longer than 512 characters #1889

yarikoptic opened this issue Mar 7, 2024 · 1 comment
Labels
bug Something isn't working UX Affects usability of the system

Comments

@yarikoptic
Copy link
Member

512 is current length limit for the path, and it is more than enough. (the longest path in archive I found is 192:

dandi@drogon:/mnt/backup/dandi/dandisets$ declare m=0;for ds in 00*; do git -C $ds ls-tree -r --name-only HEAD; done | awk '{ if (length($0) > max) max = length($0) } END { print max }'
192
dandi@drogon:/mnt/backup/dandi/dandisets$ pwd
/mnt/backup/dandi/dandisets

but if I do attempt to upload a file with path length greater -- it causes 500

❯ DANDI_DEVEL=1 dandi upload --allow-any-path $fn
2024-03-06 20:34:17,039 [    INFO] Found 2 files to consider
PATH                              SIZE    ERRORS PROGRESS STATUS          MESSAGE        
dandiset.yaml                     3.0 kB                  skipped         should be ed...
...901234567890123456789_data.dat 4 Bytes   0             producing asset                
Summary:                          3.1 kB                  1 skipped       1 should be ...
                                                          1 producing ...                
...901234567890123456789_data.dat 4 Bytes   0             ERROR           500 Server E...
Summary:                          3.1 kB                  1 skipped       1 should be ...
                                                          1 ERROR         1 500 Server...
2024-03-06 20:35:13,412 [    INFO] Logs saved in /home/yoh/.local/state/dandi-cli/log/20240307013414Z-313889.log
Traceback (most recent call last):
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/bin/dandi", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/click/decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/dandi/cli/base.py", line 126, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/dandi/cli/cmd_upload.py", line 103, in upload
    upload(
  File "/home/yoh/proj/dandi/dandi-cli-master/dandi/upload.py", line 459, in upload
    raise upload_err
  File "/home/yoh/proj/dandi/dandi-cli-master/dandi/upload.py", line 357, in process_path
    for r in dfile.iter_upload(
  File "/home/yoh/proj/dandi/dandi-cli-master/dandi/files/bases.py", line 455, in iter_upload
    r = client.post(
        ^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/dandi/dandiapi.py", line 306, in post
    return self.request("POST", path, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/dandi/dandiapi.py", line 200, in request
    for i, attempt in enumerate(
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 347, in __iter__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 325, in iter
    raise retry_exc.reraise()
          ^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 158, in reraise
    raise self.last_attempt.result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/yoh/proj/dandi/dandi-cli-master/dandi/dandiapi.py", line 236, in request
    result.raise_for_status()
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.dandiarchive.org/api/dandisets/000029/versions/draft/assets/
DANDI_DEVEL=1 dandi upload --allow-any-path $fn  2.27s user 2.93s system 8% cpu 58.988 total
❯ 
❯ echo $fn
sub-123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789/sub-123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789_ses-1/sub-123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789_data.dat
❯ echo ${#fn}
596

and as we on 500 re-try -- poor dandi-archive keeps 500ing:

2024-03-07T01:40:14.785006+00:00 app[web.1]: 10.1.61.154 - - [07/Mar/2024:01:40:14 +0000] "POST /api/dandisets/000029/versions/draft/assets/ HTTP/1.1" 500 145 "-" "dandi/0.60.0+11.gee18a15e requests/2.31.0 CPython/3.11.8"
2024-03-07T01:40:14.785346+00:00 heroku[router]: at=info method=POST path="/api/dandisets/000029/versions/draft/assets/" host=api.dandiarchive.org request_id=4a509b4c-8716-4350-973c-33deaa1761f1 fwd="73.218.136.210" dyno=web.1 connect=0ms service=48ms status=500 bytes=512 protocol=https
2024-03-07T01:40:16.133195+00:00 heroku[router]: at=info method=POST path="/api/dandisets/000029/versions/draft/assets/" host=api.dandiarchive.org request_id=812d665f-2909-42de-88f3-8c400639f8fc fwd="73.218.136.210" dyno=web.1 connect=0ms service=47ms status=500 bytes=512 protocol=https
2024-03-07T01:40:16.132901+00:00 app[web.1]: 10.1.61.154 - - [07/Mar/2024:01:40:16 +0000] "POST /api/dandisets/000029/versions/draft/assets/ HTTP/1.1" 500 145 "-" "dandi/0.60.0+11.gee18a15e requests/2.31.0 CPython/3.11.8"
2024-03-07T01:40:17.783794+00:00 app[web.1]: 10.1.61.154 - - [07/Mar/2024:01:40:17 +0000] "POST /api/dandisets/000029/versions/draft/assets/ HTTP/1.1" 500 145 "-" "dandi/0.60.0+11.gee18a15e requests/2.31.0 CPython/3.11.8"
2024-03-07T01:40:17.783956+00:00 heroku[router]: at=info method=POST path="/api/dandisets/000029/versions/draft/assets/" host=api.dandiarchive.org request_id=f4646ed1-bd1e-4967-b18e-3a267cf8b4e3 fwd="73.218.136.210" dyno=web.1 connect=0ms service=38ms status=500 bytes=512 protocol=https
2024-03-07T01:40:19.934664+00:00 heroku[router]: at=info method=POST path="/api/dandisets/000029/versions/draft/assets/" host=api.dandiarchive.org request_id=bf87983c-40ba-41d9-ad2e-7ea1e5c53730 fwd="73.218.136.210" dyno=web.1 connect=0ms service=153ms status=500 bytes=512 protocol=https
2024-03-07T01:40:19.934340+00:00 app[web.1]: 10.1.61.154 - - [07/Mar/2024:01:40:19 +0000] "POST /api/dandisets/000029/versions/draft/assets/ HTTP/1.1" 500 145 "-" "dandi/0.60.0+11.gee18a15e requests/2.31.0 CPython/3.11.8"
2024-03-07T01:40:22.552730+00:00 app[web.1]: 10.1.61.154 - - [07/Mar/2024:01:40:22 +0000] "POST /api/dandisets/000029/versions/draft/assets/ HTTP/1.1" 500 145 "-" "dandi/0.60.0+11.gee18a15e requests/2.31.0 CPython/3.11.8"
2024-03-07T01:40:22.552988+00:00 heroku[router]: at=info method=POST path="/api/dandisets/000029/versions/draft/assets/" host=api.dandiarchive.org request_id=bf4b5372-c419-4939-aed2-682773abd707 fwd="73.218.136.210" dyno=web.1 connect=0ms service=48ms status=500 bytes=512 protocol=https
2024-03-07T01:40:25.712900+00:00 app[web.1]: 10.1.61.154 - - [07/Mar/2024:01:40:25 +0000] "POST /api/dandisets/000029/versions/draft/assets/ HTTP/1.1" 500 145 "-" "dandi/0.60.0+11.gee18a15e requests/2.31.0 CPython/3.11.8"
2024-03-07T01:40:25.713061+00:00 heroku[router]: at=info method=POST path="/api/dandisets/000029/versions/draft/assets/" host=api.dandiarchive.org request_id=690a9bbf-d31d-41f5-9d09-38335a3620de fwd="73.218.136.210" dyno=web.1 connect=0ms service=62ms status=500 bytes=512 protocol=https

I guess should be some kind of 400 code
@yarikoptic yarikoptic added the UX Affects usability of the system label Mar 7, 2024
yarikoptic added a commit to dandi/handbook that referenced this issue Mar 7, 2024
@yarikoptic
State explicitly that our current maximal length of the path in dandi…
b607838 
…set must not exceed 512

ATM, if my check is right -- maximal length we can encounter is only 192, so we
are quite ok

	dandi@drogon:/mnt/backup/dandi/dandisets$ pwd
	/mnt/backup/dandi/dandisets

	dandi@drogon:/mnt/backup/dandi/dandisets$ declare m=0;for ds in 00*; do git -C $ds ls-tree -r --name-only HEAD; done | awk '{ if (length($0) > max) max = length($0) } END { print max }'
	192

When user manages to get over -- server 500s. Filed
dandi/dandi-archive#1889
@yarikoptic yarikoptic mentioned this issue Mar 7, 2024
Merged
@waxlamp waxlamp added the bug Something isn't working label Mar 11, 2024
@waxlamp
Copy link
Member

waxlamp commented Mar 11, 2024

Thanks Yarik -- I agree this is a bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working UX Affects usability of the system
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yarikoptic @waxlamp