Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HfFileSystem] Copy non lfs files #1996

Merged
merged 9 commits into from
Feb 15, 2024
Merged

[HfFileSystem] Copy non lfs files #1996

merged 9 commits into from
Feb 15, 2024

Conversation

lhoestq
Copy link
Member

@lhoestq lhoestq commented Jan 22, 2024

Added CommitOperationCopy support for non-lfs files.

This test already exists to check that copying works with hffs:

def test_copy_file(self):
# Non-LFS file
self.assertIsNone(self.hffs.info(self.hf_path + "/data/text_data.txt")["lfs"])
self.hffs.cp_file(self.hf_path + "/data/text_data.txt", self.hf_path + "/data/text_data_copy.txt")
with self.hffs.open(self.hf_path + "/data/text_data_copy.txt", "r") as f:
self.assertEqual(f.read(), "dummy text data")
self.assertIsNone(self.hffs.info(self.hf_path + "/data/text_data_copy.txt")["lfs"])

I simply removed the condition in hffs to use CommitOperationCopy even for non-lfs files.

Close huggingface/dataset-viewer#2317

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq lhoestq marked this pull request as ready for review January 26, 2024 18:09
@lhoestq lhoestq requested a review from Wauplin January 30, 2024 13:48
Copy link
Contributor

@Wauplin Wauplin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @lhoestq :) I would prefer not to rely on the fsspec overhead when downloading the regular files since a simple http request is also fine. All comments are about making this change.

As an additional change, could you update the docstring of CommitOperationCopy now that non-LFS copies are enabled on the repo. Otherwise, looks good to me :) Glad that the testing part was that easy :D Thanks in advance!

src/huggingface_hub/_commit_api.py Outdated Show resolved Hide resolved
src/huggingface_hub/_commit_api.py Outdated Show resolved Hide resolved
src/huggingface_hub/_commit_api.py Outdated Show resolved Hide resolved
src/huggingface_hub/_commit_api.py Outdated Show resolved Hide resolved
src/huggingface_hub/_commit_api.py Outdated Show resolved Hide resolved
src/huggingface_hub/_commit_api.py Outdated Show resolved Hide resolved
src/huggingface_hub/_commit_api.py Outdated Show resolved Hide resolved
src/huggingface_hub/_commit_api.py Outdated Show resolved Hide resolved
Copy link

codecov bot commented Feb 15, 2024

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (c524a86) 82.39% compared to head (1656953) 82.32%.

Files Patch % Lines
src/huggingface_hub/_commit_api.py 93.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1996      +/-   ##
==========================================
- Coverage   82.39%   82.32%   -0.07%     
==========================================
  Files          66       66              
  Lines        8240     8247       +7     
==========================================
  Hits         6789     6789              
- Misses       1451     1458       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@Wauplin Wauplin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the changes @lhoestq! Ready to be merged :)

@Wauplin Wauplin merged commit 434c60c into main Feb 15, 2024
16 checks passed
@Wauplin Wauplin deleted the copy-non-lfs-files branch February 15, 2024 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

NotImplementedError: Copying a non-LFS file is not implemented
3 participants