Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make hardcoded fetch 100 each worker user configurable #4881

Closed
lxj616 opened this issue May 21, 2024 · 1 comment · Fixed by #4886
Closed

Make hardcoded fetch 100 each worker user configurable #4881

lxj616 opened this issue May 21, 2024 · 1 comment · Fixed by #4886
Labels
kind/feature New feature or request

Comments

@lxj616
Copy link

lxj616 commented May 21, 2024

What would you like to be added:

https://github.com/juicedata/juicefs/blob/main/pkg/sync/cluster.go#L137-L166

Make

				objs = append(objs, obj)
				if len(objs) > 100 {
					break LOOP
				}

User configurable

Why is this needed:

We are using juicefs sync --workers to get 16 very large files from tencent COS to KS3 local jfs

And as we have 8 workers, however 16 is lesser than 100, so all 16 files are using one worker instead of all

Currently our temporary solution is to PAD each big file with 99 empty files, which is, well, working at least

However it would be nicer if I can use juicefs sync --workers on 16 files directly, so

@davies
Copy link
Contributor

davies commented May 22, 2024

We should have a limit on the total size of fetched keys.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants