Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement tunable registry defaults for registry and update mirrors #4521

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

protosam
Copy link

What is the problem this change solves?

In large environments with many cloud instances are running ollama serve, accidentally pushing code to run ollama pull llama3 can result in 100's of cloud instances are trying to download from ollama.ai.

The correct change for production should have been ollama pull https://registry.prod.someside.tld/library/llama3. The registry mirror at registry.prod.someside.tld is necessary to reduce bandwidth costs for high volume data, like an AI model or container image.

Mistakes like this can go unnoticed by novices building scalable infrastructure for their developers, until they get the resulting bill.

Also registry owners often have to implement rate limiting to keep bandwidth costs down. Hitting a rate limit in a production environment often results in an outage. Further making convenient mirroring options desirable.

What are the changes being made?

  • Created a new package called defaults to hold tunable values.
  • Moved variables related to endpoints to a single package called github.com/ollama/ollama/types/defaults
  • Exposes control to admins via environment variables.

Are there any tasks remaining?

I need some guidance on how testing should work for these changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant