Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add HfFileSystem.url method #2027

Merged
merged 5 commits into from
Feb 15, 2024
Merged

Add HfFileSystem.url method #2027

merged 5 commits into from
Feb 15, 2024

Conversation

mariosasko
Copy link
Contributor

Adds a url method to the HfFileSystem to simplify converting HF paths to HTTP URLs, which should be useful when working with libs that support HTTP URLs but not fsspec paths as input/output (e.g., webdataset, polars, etc.).

PS: The url method is not part of the official fsspec specification, but popular filesystem implementations such as gcsfs and s3fs also have it

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Copy link

codecov bot commented Feb 14, 2024

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (0c272d5) 82.37% compared to head (ab935a4) 82.37%.
Report is 2 commits behind head on main.

❗ Current head ab935a4 differs from pull request most recent head be04be6. Consider uploading reports for the commit be04be6 to get more accurate results

Files Patch % Lines
src/huggingface_hub/hf_file_system.py 77.77% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2027      +/-   ##
==========================================
- Coverage   82.37%   82.37%   -0.01%     
==========================================
  Files          66       66              
  Lines        8240     8249       +9     
==========================================
+ Hits         6788     6795       +7     
- Misses       1452     1454       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@Wauplin Wauplin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the addition! Left a few nits but looks good to me :) Feel free to merge.

src/huggingface_hub/hf_file_system.py Outdated Show resolved Hide resolved
src/huggingface_hub/hf_file_system.py Outdated Show resolved Hide resolved
src/huggingface_hub/hf_file_system.py Outdated Show resolved Hide resolved
src/huggingface_hub/hf_file_system.py Outdated Show resolved Hide resolved
Copy link
Member

@lhoestq lhoestq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice thanks !

mariosasko and others added 3 commits February 15, 2024 17:17
Co-authored-by: Lucain <lucainp@gmail.com>
Co-authored-by: Lucain <lucainp@gmail.com>
Copy link
Contributor

@Wauplin Wauplin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for applying the changes!

@Wauplin Wauplin merged commit f386b2a into main Feb 15, 2024
16 checks passed
@Wauplin Wauplin deleted the hffs-url branch February 15, 2024 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants