feat: support reading/loading multiple .env files #424

duarte-pompeu · 2022-09-10T14:13:09Z

This PR allows reading from multiple .env files in the dotenv CLI, specifically the targets get, list and run.

This will allow users to have a big .env file with default configs, and smaller ones which only incremental changes - therefore removing the burden of repeating unchanged key-values.

The goal for the new behavior is as follows:

files are processed in order
new keys lead to creating new key-values
repeated keys lead to overriding previous values

Extra remarks:

this PR preserves the behavior of working with single files, and falling back to .env when no file is specified
this PR forbids write operations on multiple files, specifically set and unset

Example:

$ echo -e 'URL="https://www.example.com"\nUSER="user"\nPASSWORD="secret"' > .env1
$ echo -e 'USER="admin"\nPASSWORD="admin"' > .env2
$ python -m dotenv -f .env1 -f .env2 list
PASSWORD=admin
URL=https://www.example.com
USER=admin

Ideas for the future:

also allow loading multiple .env files from the code, eg load_dotenv(f1, f2, ...)
also allow setting and unsetting values for multiple .env files

Relates to #418 .

duarte-pompeu

could be improved

src/dotenv/cli.py

tests/test_main.py

src/dotenv/cli.py

duarte-pompeu

double checked

in cli.py we may use multiple files and getting multiple errors but want to avoid multiple errors if at least one file has the key

src/dotenv/main.py

pheanex · 2023-12-04T15:16:55Z

@duarte-pompeu will you take care of resolving the conflicts before we ping the maintainer for review?

duarte-pompeu · 2023-12-04T19:54:42Z

I can give it a try tonight, it would be interesting to merge this feature.

duarte-pompeu · 2023-12-04T20:32:42Z

Solved conflicts but there's still some work to do regarding failing tests.

duarte-pompeu · 2023-12-04T22:54:29Z

Think it looks better now @pheanex 🙂

It's missing more tests and documentation, but I'd prefer to have feedback from a maintainer before tackling those.

pheanex · 2023-12-05T07:57:42Z

Awesome! Let's see if @bbc2 or @theskumar is available for a quick review?

theskumar · 2023-12-07T13:16:18Z

Fundamentally as feature I'm good with supporting this. I like your approach and seeing how you are going about it. I haven't had time to look into the implementation details. But I feel free to improve on your implementation. Would really appreciate keeping it backward compatible. Thank you @duarte-pompeu for your contribution.

duarte-pompeu · 2023-12-07T14:07:15Z

Thanks for the feedback @theskumar. I can keep working on this over the next days.

I'm thinking about reducing the scope in this PR though - would it be OK to only support read operations, such as get and list? Seems like a useful increment and later we can think about how to handle write operations.

theskumar · 2023-12-07T19:26:26Z

Read is completely fine, I could not see how the write be extended to multiple files. Would be curious if you have some ideas around it.

duarte-pompeu · 2023-12-08T10:27:41Z

Read is completely fine, I could not see how the write be extended to multiple files. Would be curious if you have some ideas around it.

Actually I already implemented it: unset looks for the key in all files and deletes it when present. But I have some doubts about this usefulness, or if this behavior is intuitive to users - as such, I'd rather remove it from this PR and discuss it later.

Example as of 49c34a5:

$ for f in $(ls .env*); do echo $f; cat $f; echo; done
.env1
a=1
b=2
.env2
b=2
c=3
# unset 'a', which is only present in .env1
$ python -m dotenv -f .env1 -f .env2 unset a
Successfully removed a from .env1.
$ for f in $(ls .env*); do echo $f; cat $f; echo; done
.env1
b=2
.env2
b=2
c=3
# unset 'b', which is present in both .env1 and .env2
$ python -m dotenv -f .env1 -f .env2 unset b
Successfully removed b from ['.env1', '.env2'].
$ for f in (ls .env*); do echo $f; cat $f; echo; done
.env1
.env2
c=3
# try to unset key which does not exist
$ python -m dotenv -f .env1 -f .env2 unset d
Key d not removed from ('.env1', '.env2') - key doesn't exist.

duarte-pompeu · 2023-12-08T12:40:33Z

tests/test_cli.py

+    dotenv_path.write_text(contents[0] + '\n')
+    extra_dotenv_path.write_text(contents[1] + '\n')
+
+    args = ['--file', dotenv_path, '--file', extra_dotenv_path, 'list']


I omitted tests with multiple formatting because that seems to me like a separate concern - if the variables are loaded correctly, the formatting behavior shouldn't change.

Do you agree?

You are right. This is fine.

duarte-pompeu · 2023-12-08T12:45:16Z

I think this is ready for review @theskumar:

implemented changes
kept backward compatibility (I think so, someone please double check for peace of mind 😃)
added tests
updated docs
improved PR description

We didn't talk about this, but I only implemented this for the CLI utility. If there's interest to also enable this behavior from the library code, I'm up to contribute to it (but would prefer to have it in a separate PR, as I find this one already useful as is).

duarte-pompeu · 2023-12-16T14:07:26Z

Hi @theskumar and @pheanex , any feedback?

pheanex · 2023-12-17T06:47:58Z

Lgtm. Looking forward to this ☺️👍

theskumar · 2023-12-21T18:22:25Z

@duarte-pompeu thanks for all the hard work. Unfortunately, I've had not been able to review this yet. Looks like we Christmas 🎄 break would be a good time to taking care of it for me, along with few more pull requests.

duarte-pompeu · 2023-12-27T11:58:16Z

@duarte-pompeu thanks for all the hard work. Unfortunately, I've had not been able to review this yet. Looks like we Christmas 🎄 break would be a good time to taking care of it for me, along with few more pull requests.

Cool, I also have some free time now so it would also work for me 😃

theskumar · 2023-12-28T18:54:57Z

README.md

+a=1
+b=20
+c=30
+```


@duarte-pompeu Consider document the precedence/loading order values from the of the files. I didn't see a test but based on the this example it seems the .env2 overrides the previous value from .env1.

I agree: it's important to document this. I chose an overriding behavior, as my use case would be:

have a big file with all values

have smaller .env files with specific values to override in certain situations

I tried to improve it on 363aab5

theskumar · 2023-12-28T18:55:58Z

README.md

@@ -3,7 +3,7 @@
 [![Build Status][build_status_badge]][build_status_link]
 [![PyPI version][pypi_badge]][pypi_link]

-Python-dotenv reads key-value pairs from a `.env` file and can set them as environment
+Python-dotenv reads key-value pairs from `.env` files and can set them as environment


I'm hesitant to change this yet as the core/python API, doesn't support reading from multiple files.

Good point. I agree it makes sense to only document this in the CLI section.

Resolved in 1d33b6a

changes only affect the CLI, not the whole package

duarte-pompeu · 2023-12-28T22:41:00Z

Hi @theskumar , I agree with your feedback and addressed in the last commits. What do you think?

pheanex · 2024-01-12T20:53:13Z

@theskumar any chance to take a quick look again?

pheanex · 2024-03-05T13:22:17Z

@bbc2 any chance you can take a look at this? It seems @theskumar is currently unavailable.

bbc2

I'm OK with this (we might be opening Pandora's box but so be it) and it looks like you addressed all the concerns raised previously, so if no one objects for a few more days I'll go ahead and merge it. Thank you for your contribution!

pheanex · 2024-03-27T07:50:25Z

@bbc2 as two more weeks have passed: Are we good to move forward with this?

accept multiple files in dotenv run

71f1730

duarte-pompeu mentioned this pull request Sep 10, 2022

feature request: load multiple .env file before running a command #418

Open

duarte-pompeu changed the title ~~Load multiple .env file (WIP)~~ feat: multiple .env file (WIP) Sep 10, 2022

duarte-pompeu changed the title ~~feat: multiple .env file (WIP)~~ feat: support multiple .env files (WIP) Sep 10, 2022

duarte-pompeu added 2 commits September 10, 2022 18:37

allow multiple files for other cli targets

5f7d1f8

fix behavior or adapt tests

6718b16

duarte-pompeu changed the title ~~feat: support multiple .env files (WIP)~~ feat: support multiple .env files Sep 10, 2022

duarte-pompeu commented Sep 10, 2022

View reviewed changes

src/dotenv/cli.py Outdated Show resolved Hide resolved

src/dotenv/cli.py Outdated Show resolved Hide resolved

src/dotenv/cli.py Outdated Show resolved Hide resolved

tests/test_main.py Outdated Show resolved Hide resolved

duarte-pompeu added 3 commits September 10, 2022 19:15

improve readability

b4f335b

remove f-string

0660ded

improve formatting

d46b1e6

duarte-pompeu commented Sep 10, 2022

View reviewed changes

src/dotenv/cli.py Outdated Show resolved Hide resolved

restore code to test for warning

bc474ca

duarte-pompeu commented Sep 10, 2022

View reviewed changes

duarte-pompeu added 2 commits October 1, 2022 12:39

format code

2b3a28b

add flag to allow suppressing warning in get_key

f730c6d

in cli.py we may use multiple files and getting multiple errors but want to avoid multiple errors if at least one file has the key

duarte-pompeu commented Oct 1, 2022

View reviewed changes

src/dotenv/main.py Outdated Show resolved Hide resolved

fix warning and test in cli

d6c1385

Merge branch 'main' into multiple-files

630c0bd

duarte-pompeu added 5 commits December 4, 2023 21:08

fix lint warnings

c585a7b

fix some problems

9b1b4af

fix tests

a2d9365

fix type hint

f5ed3e9

add some tests

49c34a5

theskumar changed the title ~~feat: support multiple .env files~~ feat: support reading/loading multiple .env files Dec 7, 2023

duarte-pompeu added 11 commits December 8, 2023 10:45

remove support for unsetting multiple files

437a921

improve backward compatibility

a10e462

remove unused logger

186dc81

remove support for set on multiple files

dde1654

print errors to stderr

ec23819

improve params for test_lit_multi_file

febc153

test get for multiple files

c52dd9e

test run with multiple envs

9e3df30

update docs

6090d64

remove unecessary param

d61c1ed

remove format param for test_list_multi_file

46f1b75

duarte-pompeu commented Dec 8, 2023

View reviewed changes

theskumar requested changes Dec 28, 2023

View reviewed changes

duarte-pompeu added 2 commits December 28, 2023 22:35

document loading order of multiple .envs

363aab5

revert change in description

1d33b6a

changes only affect the CLI, not the whole package

duarte-pompeu requested a review from theskumar December 28, 2023 22:40

bbc2 approved these changes Mar 16, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support reading/loading multiple .env files #424

feat: support reading/loading multiple .env files #424

duarte-pompeu commented Sep 10, 2022 •

edited

duarte-pompeu left a comment

duarte-pompeu left a comment

pheanex commented Dec 4, 2023

duarte-pompeu commented Dec 4, 2023

duarte-pompeu commented Dec 4, 2023

duarte-pompeu commented Dec 4, 2023

pheanex commented Dec 5, 2023

theskumar commented Dec 7, 2023

duarte-pompeu commented Dec 7, 2023

theskumar commented Dec 7, 2023

duarte-pompeu commented Dec 8, 2023 •

edited

duarte-pompeu Dec 8, 2023

theskumar Dec 28, 2023

duarte-pompeu commented Dec 8, 2023

duarte-pompeu commented Dec 16, 2023

pheanex commented Dec 17, 2023

theskumar commented Dec 21, 2023 •

edited

duarte-pompeu commented Dec 27, 2023

theskumar Dec 28, 2023

duarte-pompeu Dec 28, 2023

theskumar Dec 28, 2023

duarte-pompeu Dec 28, 2023

duarte-pompeu commented Dec 28, 2023

pheanex commented Jan 12, 2024

pheanex commented Mar 5, 2024

bbc2 left a comment

pheanex commented Mar 27, 2024

feat: support reading/loading multiple .env files #424

Are you sure you want to change the base?

feat: support reading/loading multiple .env files #424

Conversation

duarte-pompeu commented Sep 10, 2022 • edited

duarte-pompeu left a comment

Choose a reason for hiding this comment

duarte-pompeu left a comment

Choose a reason for hiding this comment

pheanex commented Dec 4, 2023

duarte-pompeu commented Dec 4, 2023

duarte-pompeu commented Dec 4, 2023

duarte-pompeu commented Dec 4, 2023

pheanex commented Dec 5, 2023

theskumar commented Dec 7, 2023

duarte-pompeu commented Dec 7, 2023

theskumar commented Dec 7, 2023

duarte-pompeu commented Dec 8, 2023 • edited

duarte-pompeu Dec 8, 2023

Choose a reason for hiding this comment

theskumar Dec 28, 2023

Choose a reason for hiding this comment

duarte-pompeu commented Dec 8, 2023

duarte-pompeu commented Dec 16, 2023

pheanex commented Dec 17, 2023

theskumar commented Dec 21, 2023 • edited

duarte-pompeu commented Dec 27, 2023

theskumar Dec 28, 2023

Choose a reason for hiding this comment

duarte-pompeu Dec 28, 2023

Choose a reason for hiding this comment

theskumar Dec 28, 2023

Choose a reason for hiding this comment

duarte-pompeu Dec 28, 2023

Choose a reason for hiding this comment

duarte-pompeu commented Dec 28, 2023

pheanex commented Jan 12, 2024

pheanex commented Mar 5, 2024

bbc2 left a comment

Choose a reason for hiding this comment

pheanex commented Mar 27, 2024

duarte-pompeu commented Sep 10, 2022 •

edited

duarte-pompeu commented Dec 8, 2023 •

edited

theskumar commented Dec 21, 2023 •

edited