Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed hash check logic passes as a completed download, and a typo #15

Open
InvalidAdmin opened this issue Nov 18, 2023 · 0 comments
Open

Comments

@InvalidAdmin
Copy link

I have been downloading several models but occasionally notice that there are model folders much smaller than they should be. My internet connection is a crappy long distance microwave connection that is error prone. It turns out that, when there is a hash fail, somehow, the program then exits with a download success, like this:

Hash failed for LFS file: Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors
Download of alpindale/goliath-120b completed successfully

I am using the command:

bash <(curl -sSL https://g.bodaay.io/hfd) -m alpindale/goliath-120b

Here is the full output from this particular run, which was actually a re-run because the download must have quietly failed before too:

File already exists. Skipping download.
Model: alpindale/goliath-120b
Branch: main
Storage: Storage
NumberOfConcurrentConnections: 5
Append Filter Names to Folder: false
Skip SHA256 Check: false
Token: 

Getting File Download Files List Tree from: https://huggingface.co/api/models/alpindale/goliath-120b/tree/main/
Checking Existsing file: Storage/alpindale_goliath-120b/.gitattributes
file size matched for non LFS file: Storage/alpindale_goliath-120b/.gitattributes
Checking Existsing file: Storage/alpindale_goliath-120b/README.md
file size matched for non LFS file: Storage/alpindale_goliath-120b/README.md
Checking Existsing file: Storage/alpindale_goliath-120b/config.json
file size matched for non LFS file: Storage/alpindale_goliath-120b/config.json
Skipping: Storage/alpindale_goliath-120b/.gitattributes
Skipping: Storage/alpindale_goliath-120b/README.md
Skipping: Storage/alpindale_goliath-120b/config.json

Downloading Storage/alpindale_goliath-120b/model-00001-of-00024.safetensors Speed: 4.70 MB/sec, 100.00% 
Merging Storage/alpindale_goliath-120b/model-00001-of-00024.safetensors Chunks
Checking SHA256 Hash for LFS file: Storage/alpindale_goliath-120b/model-00001-of-00024.safetensors
Hash Matched for LFS file: Storage/alpindale_goliath-120b/model-00001-of-00024.safetensors

Downloading Storage/alpindale_goliath-120b/model-00002-of-00024.safetensors Speed: 4.84 MB/sec, 100.00% 
Downloading Storage/alpindale_goliath-120b/model-00002-of-00024.safetensors Speed: 4.84 MB/sec, 100.00% 
Checking SHA256 Hash for LFS file: Storage/alpindale_goliath-120b/model-00002-of-00024.safetensors
Hash Matched for LFS file: Storage/alpindale_goliath-120b/model-00002-of-00024.safetensors

Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 4.95 MB/sec, 59.40% Error downloading chunk 2: stream error: stream ID 1; INTERNAL_ERROR; received from peer
warning: attempt 1 / 3 failed, error: Error downloading chunk 2: stream error: stream ID 1; INTERNAL_ERROR; received from peer
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 4.95 MB/sec, 59.62% 
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 4.95 MB/sec, 59.78% ee/main/
Checking Existsing file: Storage/alpindale_goliath-120b/.gitattributes
file size matched for non LFS file: Storage/alpindale_goliath-120b/.gitattributes
Checking Existsing file: Storage/alpindale_goliath-120b/README.md
file size matched for non LFS file: Storage/alpindale_goliath-120b/README.md
Checking Existsing file: Storage/alpindale_goliath-120b/config.json
file size matched for non LFS file: Storage/alpindale_goliath-120b/config.json
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 4.93 MB/sec, 61.79% 
Hash Matched for LFS file: Storage/alpindale_goliath-120b/model-00001-of-00024.safetensors
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 4.92 MB/sec, 63.59% 
Hash Matched for LFS file: Storage/alpindale_goliath-120b/model-00002-of-00024.safetensors
Skipping: Storage/alpindale_goliath-120b/.gitattributes
Skipping: Storage/alpindale_goliath-120b/README.md
Skipping: Storage/alpindale_goliath-120b/config.json
Skipping: Storage/alpindale_goliath-120b/model-00001-of-00024.safetensors
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 4.91 MB/sec, 63.60% 
Found existing incomplete download for the file: model-00003-of-00024.safetensors
Forcing Number of connections to: 5



Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 8.24 MB/sec, 100.00%    0% 
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 3.54 MB/sec, 89.38% 
Downloading Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors Speed: 3.54 MB/sec, 89.41% 
Hash failed for LFS file: Storage/alpindale_goliath-120b/model-00003-of-00024.safetensors
Download of alpindale/goliath-120b completed successfully

And secondarily, because I pasted the code above and it had spellcheck for the output, and I noticed a typo. I'll put it here, but would you rather me make a new issue just for this?

Checking Existsing file: Storage/alpindale_goliath-120b/.gitattributes

"Existsing" is spelled wrong, it should be, "Existing"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant