zarr re-upload might still be suffering from "A header you provided implies functionality that is not implemented" #1408
Comments
|
@yarikoptic Do the logs contain any messages about |
|
I don't think so: ❯ zgrep -e super_len -e stat 20240216201625Z-3182147.log.gz
do = self.iter(retry_state=retry_state)
result.raise_for_status()
File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
2024-02-16T15:37:30-0500 [INFO ] dandi 3182147:140109695737920 Logs saved in /home/yoh/.local/state/dandi-cli/log/20240216201625Z-3182147.logfull log (compressed) now available from |
|
@yarikoptic The errors all seem to be associated with the file |
|
@yarikoptic Can you try redoing the upload with |
looks "regular" to me...❯ ls sub-randomzarrlike_junk.zarr/0/0/d0/d2/d26/f1.dat -l
-rw------- 1 yoh yoh 1024 Feb 16 15:14 sub-randomzarrlike_junk.zarr/0/0/d0/d2/d26/f1.dat
❯ md5sum -l
md5sum: invalid option -- 'l'
Try 'md5sum --help' for more information.
❯ md5sum sub-randomzarrlike_junk.zarr/0/0/d0/d2/d26/f1.dat
4bd9951479a6a7b2b36b6c8b80f1e5f5 sub-randomzarrlike_junk.zarr/0/0/d0/d2/d26/f1.dat
❯ cat sub-randomzarrlike_junk.zarr/0/0/d0/d2/d26/f1.dat
��
�Կ�[�\������#;r�1�>��G/1{���J�/�ճ]��,L-z����%AT�J��驇c�S��%�
�7f�����,����$�,�<@��⍩&�P��ߑ�����...seems to be a new file since there is no d26 under https://github.com/dandizarrs/fd6ab3ea-cff6-4006-a9bf-acfa5d983985/tree/0.231017.2004%2Bzarr1/0/0/d0 which should reflect prior version . |
|
@yarikoptic What if you set the envvar |
|
that re-upload succeeded without error log is at |
|
uploaded also 20240223192140Z-306046.log.gz from another successful upload. I now wonder if that instrumentation is what makes it pass. |
|
@jwodder - please try (script) a few uploads like that. May be against staging instance. Original description provides all the commands needed. Would be nice to catch this and figure out why AWS wasn't happy to not make users surprised later on. |
|
@yarikoptic As I have stated previous times this has come up, the "A header you provided implies functionality that is not implemented" error from AWS occurs when an upload is made using "chunked" transfer-encoding, and |
|
Figure out either it is a. or b. or some other c. and in the end/accordingly make sure that we do not error out when we |
|
@yarikoptic I'm currently uploading a randomly-generated Zarr to staging from my MacBook Pro with
#!/bin/bash
set -x
cd "$(dirname "$0")"
mkdir -p logs
for i in {1..10}
do
DANDI_DEVEL_INSTRUMENT_REQUESTS_SUPERLEN=1 \
./run.sh |& tee logs/"$(date -u +%Y.%m.%d.%H.%M.%SZ)"-spy.log
./run.sh |& tee logs/"$(date -u +%Y.%m.%d.%H.%M.%SZ)"-nospy.log
done
#!/bin/bash
export DANDI_API_KEY=---REDACTED---
export DANDI_DEVEL=1
set -ex
cd "$(dirname "$0")"
. venv/bin/activate
cd 214256
for layout in zarr128-smallfiles zarr64-smallfiles
do
rm -rf random.zarr
echo Generating random.zarr from $layout ...
chronic python3 ../zarr-digest-timings/mktree.py \
random.zarr \
../zarr-digest-timings/layouts/$layout.json
echo "Now: $(date)"
if ! dandi upload --devel-debug -i dandi-staging --validation ignore -J 5:200 random.zarr
then echo "Failed at: $(date)"
exit 1
fi
echo "Now: $(date)"
done |
|
Update: I just got the following errors on smaug:
According to EDIT: And now I've gotten "Error: requests.utils.super_len() reported size of 0 for '/Users/jwodder/dartmouth/tmp/dandi-1408/214255/random.zarr/1/0/d0/d1/d9/f34.dat', but os.stat() reported size 1024 bytes 1 tries later" on macOS (filesystem type: apfs). |
|
"fun!" .
|
|
@yarikoptic I'm going to try editing the source code of |
jwodder
added
the
HIFNI
|
@yarikoptic See #1444. |
|
🚀 Issue was released in |
I was trying to produce 2nd version of a sample zarr under https://dandiarchive.org/dandiset/000029/draft/files?location=sub-randomzarrlike&page=1
https://dandi.centerforopenneuroscience.org/zarrs/fd6/ab3/fd6ab3ea-cff6-4006-a9bf-acfa5d983985/
which is also freshly created https://github.com/dandizarrs/fd6ab3ea-cff6-4006-a9bf-acfa5d983985/ .
Was created using
python3 mktree.py /home/yoh/proj/dandi/dandisets/000029/sub-randomzarrlike/sub-randomzarrlike_junk.zarr layouts/zarr128-smallfiles.jsonand now "modified" usingpython3 mktree.py /home/yoh/proj/dandi/dandisets/000029/sub-randomzarrlike/sub-randomzarrlike_junk.zarr layouts/zarr64-smallfiles.json(afterrm -rfprior existing subfolders) so that checksum went fromaf1b4b5849cd2b79a29762d716863379-37480--38379520to422bd4b0fa7a06d2655d3c1f67fd4a6f-263522--269846528.During re-upload screen was showing
and that "producing asset" made me worry since AFAIK we should have just modified existing zarr, not generate a new one... but it talks about "asset", not "zarr", so hope remained...
Then dandi CLI was showing no progress or anything with 100% CPU in comparing against remote Zarr exists - reuploading state for a few minutes but then upload commenced. I have used
-J 5:200which resulted in dandi CLI somehow taking over 400% CPU according to top... I decided to look into the log file (on my laptop)20240216201625Z-3182147.logand saw those errors search for which brought me tobut it seems that we might be getting them also not quite handled since errors seems to bubble up all the way to the top?
upload did after all fail with
The text was updated successfully, but these errors were encountered: