Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dotenv integration captures quotes in value/improperly stores json #772

Closed
ParetoOptimalDev opened this issue Aug 4, 2023 · 6 comments · Fixed by #1040 · May be fixed by #778
Closed

dotenv integration captures quotes in value/improperly stores json #772

ParetoOptimalDev opened this issue Aug 4, 2023 · 6 comments · Fixed by #1040 · May be fixed by #778
Labels
bug Something isn't working

Comments

@ParetoOptimalDev
Copy link

ParetoOptimalDev commented Aug 4, 2023

Describe the bug

When using quoted json in .env the resulting environmental variable includes both the quotes and json. If something like "read" in Haskell is called on that environmental variable it will predictably fail.

The reason is because of this regex in the dotenv integration:

(.+) *= *(.+)

Which results in the single quotes being in the resulting value:

nix-repl> builtins.match "(.+) *= *(.+)" "a='{\"a\":\"b\"}'"
builtins.match "(.+) *= *(.+)" "a='{\"a\":\"b\"}'"
[ "a" "'{\"a\":\"b\"}'" ]

I can get the expected behavior by optionally matching ', but this might not be how you want to write it:

nix-repl> builtins.match "(.+) *= *'?(.+)?'" "a='{\"a\":\"b\"}'"
builtins.match "(.+) *= *'?(.+)?'" "a='{\"a\":\"b\"}'"
[ "a" "{\"a\":\"b\"}" ]

Some related issues:

theskumar/python-dotenv#35
theskumar/python-dotenv#285
theskumar/python-dotenv#381

To reproduce
Please provide an Short, Self Contained, Correct (Compilable), Example by creating a gist using devenv.nix, devenv.yaml, and optionally devenv.lock.

Given .env:

data='{"a":"b"}'

Running echo $data with dotenv integration enabled results in:

$ echo $data
'{"a":"b"}'

I would expect only the json value and not the surrounding single quotes to be captured.

Version

Paste the output of $ devenv version here.

@ParetoOptimalDev ParetoOptimalDev added the bug Something isn't working label Aug 4, 2023
@ParetoOptimalDev ParetoOptimalDev changed the title devenv dotenv integration captures quotes in value/improperly stores json dotenv integration captures quotes in value/improperly stores json Aug 4, 2023
@domenkozar
Copy link
Member

Seems like we'll want to cover these tests https://github.com/theskumar/python-dotenv/blob/main/tests/test_parser.py

We'd want to use https://github.com/tweag/nix-unit-testing/tree/main/flake to write the tests.

@ento
Copy link

ento commented Nov 27, 2023

#718 can also fix this issue by offloading the whole parsing logic to the shell. It also means that config.env won't hold variables extracted from the .env file, but it's not like config.env was guaranteed to hold all the environment variables available in devenv shell, since any enterShell script can add its own environment variable.

I was looking into tweaking the regex used in parseLine to make it ignore comments, but I'm beginning to think #718 may be the smarter approach. (Or at least in that general direction - #718 doesn't handle comments well as-is)

@domenkozar What do you think?

@domenkozar
Copy link
Member

#718 can also fix this issue by offloading the whole parsing logic to the shell. It also means that config.env won't hold variables extracted from the .env file, but it's not like config.env was guaranteed to hold all the environment variables available in devenv shell, since any enterShell script can add its own environment variable.

I was looking into tweaking the regex used in parseLine to make it ignore comments, but I'm beginning to think #718 may be the smarter approach. (Or at least in that general direction - #718 doesn't handle comments well as-is)

@domenkozar What do you think?

My worry is that if someone relies on those variables to be used in Nix, this will stop working.

@RocketPuppy
Copy link

I don't know if this needs a new issue or not, but the regex in question also seems to cause problems with values that have = in them. The "(.+) *= *(.+)" regex is greedy so everything up to the last = is captured as the variable name. For example, a typical database url environment variable such as DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres?schema=public seems to be read in with a name of DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres?schema instead of the expected name of DATABASE_URL.

I will second the ability to have the .env variables accessed in Nix is pretty useful. It lets me easily parameterize parts of an environment variable across environments, such as the port of the database url in the example above.

@ento
Copy link

ento commented Nov 29, 2023

Yeah, on second thoughts, I can use direnv to source .env if I don't need the env vars to be available in config.env.

I have a regex that can handle comments and values with = somewhat, but the actual values don't get stripped of comments until after they've been exported as environment variables: https://gist.github.com/ento/765b47da99cc05d4fec508632e8d1f64 (and it doesn't handle quotes at all, as demonstrated by the test cases in that gist)

@Atry
Copy link
Contributor

Atry commented Mar 22, 2024

I will second the ability to have the .env variables accessed in Nix is pretty useful.

I am not sure if it is a good idea to access .env variables in nix expressions. For example, if you create a script via pkgs.writeShellScript, in which the variable is referenced via env.DATABASE_URL, then your pgsql password is leaked to the nix store, which is visible to all unix users. On the other hand, if your shell script references it via $DATABASE_URL, your pgsql password would not be leaked to the nix store.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants