About fallback models and environment files

[patillacode]

Hi everyone,

There have been numerous questions, issues and PRs (#14, #22, #29 , #41, #42, #83, #92, #106, #144) regarding these matters, meaning there is definitely some interest and/or use cases.

I think we should make a final decision.
In order to do that I think a discussion would be valuable.

My two cents: if we adopted using a .env file, each of us could set the following:

OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxx
MODEL=gpt-4
...

Which for example, solves the fallback issue, where some people want to use gpt-4 and others can't and have to use gpt-3.5-turbo (or -16k or -16k-0613)

I personally like the idea of each of us tweaking things like this instead of taking care of them in the code trying to fulfil everyone's need, each user can be responsible of their own settings, if you will.
So if I want to use a model that will soon be deprecated, like these new -0613 which will eventually be deprecated, it is on me, to properly modify my .env or settings.py.

Also, many users have asked about the issue where they forgot to export their KEY and honestly, from my point of view, it is just annoying having to do the export every time I open a terminal to run GPT Engineer.
So, this solution would also solve that.

I am aware that @AntonOsika (repo owner) is not convinced about this.

Just to be clear, maybe .env is not the way, maybe a proper settings.py file would do the trick. But I think we need something to cover this cases.

I just thought we should discuss openly here and get some feedback from the community.

Please keep your comments on the matter, don't get too sidetracked if possible.

Let's discuss!

[djpapzin]

I support your point, I having to input my API key everytime I run is annoying

[Vandivier]

imo we should use .env for transparency, convenience, and security:

1. I like to stream programming. I can't use the cli like OPENAI_KEY=foo python3 ... on screen while streaming. it's more convenient and secure to have a .env file
2. A .env.template file makes it transparent what variables are expected.
3. A .env.template file is convenient because we can place default values where it makes sense

[jebarpg]

This is the way

[jnichols0]

I think having the desired model as an environment variable is a great idea and would not add to complexity or lines of code. Additionally support for loading a local .env file would add value and could be implemented while respecting the goal of minimal lines of code. Reliable fallback logic has been added to support users without access to gpt-4, which is great because it should work "out of the box" for anyone with an openai account but I think it's reasonable that users shouldn't have to read and edit code to change something that could easily treated as a setting.

Using python-dotenv, adding just two lines of code to main.py :

from dotenv import load_env
load_dotenv()

change the default arg in main() to

model: str = os.getenv("GPT_ENGINEER_MODEL", "gpt-4")

would keep current behavior while adding support for configurable model via .env file or host env var GPT_ENGINEER_MODEL ( I think MODEL is too vague an env var name, might be safer to namespace it a bit). If no env var is set, it'd still try gpt-4, falling back on gpt-3.5-turbo.

The README.md would need be updated to reflect the new env var and the support for a .env file.

For one dependency and two more total lines of code, this seems like a low cost high value feature to add.

[yaJannik]

+1 for me.

[yhyu13]

Just leave a trail on the previous pr on .env staff : #29

Here is my fork on this project with .env support, just to keep it around as a reference in case anyone is interested: https://github.com/yhyu13/gpt-engineer/tree/local_dev

[RareMojo]

Big agree. This is how my fork works as it is more cost efficient to switch models depending on the task. Also, it's just more accessible in general, in my opinion.

+1

[AntonOsika]

Surprised by so many preferring env variables!

Why not just pass in --model parameter if you need it?

Is it because you run it from IDE / vscode?

[patillacode]

It's common practice and very handy especially for secrets!

[patillacode]

Sorry @noxiouscardiumdimidium

This is not the place for this, this is a discussion about something else.

If you wish you can open an issue, please do it following the new issue template.

[jnichols0]

Given that gpt-engineer also accepts a model as an argument, there could be discrepancy between the two. E.g. supplying gpt-3 via command line and gpt-3.5-turbo via the environment variables. I am wondering now if it could get confusing for the user to accept either or.

Additionally, regarding model fall-back logic: my first contribution to this project was implementing a hard-coded fall-back to gpt-3.5-turbo since I don't have access to gpt-4 and I believe at the time there was no way to provide an alternative model to gpt-engineer w/o changing the code yourself. My intention was to make it work "out of the box" for whoever regardless of their API key. Now that we can provide the model as a cli param, I think we should rethink that pattern.

If a user pip installs gpt-engineer, sets up their openai key env var and runs gpt-engineer projects/example it should just work, try gpt-4 and fallback to gpt-3.5-turbo. But if the user asks for a specific model gpt-engineer can't do, e.g. gpt-engineer projects/example --model no-such-model, it should not just print a warning and revert to gpt-3.5-turbo and continue like that's just as good as what they asked for, because it might not be. I think we should only fall back from gpt-4 to gpt-3.5-turbo if the user has not specified any model explicitly.