.env values (potentially secrets) printed to stderr

[alexcb]

> Earthfile
echo "secretkey=keep-it-secret" > .env
earthly +this-doesnt-exist

will output

Error: build target: build main: failed to solve: Earthfile line 0:0 target this-doesnt-exist not found
in		+this-doesnt-exist --secretkey=keep-it-secret

[SimonAdameit]

Earthly will also output the secrets on the command line when being interrupted (Ctrl-C, SIGINT) during the build.

[alexcb]

The .env values are being converted to the stack string which gets formatted into an error under

earthly/earthfile2llb/interpretererror.go

Line 54 in e5d6997

ret = fmt.Sprintf("%s\nin\t\t%s", ret, ie.stack)

[alexcb]

An initial work-in-progress has been opened in #1801; however, it sounds like we may need to rework how .env is used to be able to list build-args seperate from secrets.

If we're forced to change the syntax, we could implement a credential-helper plugin, perhaps supporting a shell-out like: MY_PASSWORD=$(cat /etc/password.insecure.example | grep user | awk '{print $1}')?

or even a generic helper that passes the required token to a program, .e.g. /bin/secret MY_PASSWORD.

[alexcb]

according to #179 (comment)

The build args only show up in the history if they are declared as such via ARG. But if an arg is passed on the command-line, but is never used in the build with an ARG declaration, then it will not show up.

makes it sound like they should only be displayed when explicitly used by the target.

In the case of referencing a target that doesn't exist, I think it should be fine to simply say:

Earthfile line 0:0 target this-doesnt-exist not found

[SimonAdameit]

I think the credential helper and separation in .env of secrets and variables are a bit too complex as solutions to this. How about this approach:

All input (e.g. from .env) is effectively treated as secret, until it is explicitly referenced as a build arg at which moment it is promoted to non-secret at exactly the place where it is referenced.

WDYT?

[alexcb]

until it is explicitly referenced as a build arg

This sounds like a good idea.

[alexcb]

Some other potential fixes from #2336

• Update the documentation to no longer recommend putting secrets in .env
• Add support for a different file just for secrets (e.g. .secrets)
• No longer allow loading secrets from .env (could break existing builds)

[alexcb]

This issue was originally fixed in #2365 and should have been closed; however, due to use cases surrounding the streaming of logs (and underlying https://github.com/earthly/earthly/blob/ceda09e5c4c5b44885daf5b89d723828a9301fcb/logbus/bus.go implementation), it has been "reopened".

@vladaionescu and I have decided to introduce a backwards breaking change for the upcoming v0.7.0 release:

• .arg will be used for any --build-arg values that are input into earthly (e.g. non-sensitive key-value pairs from the .env file)
• .secret will contain sensitive key-value pairs that would be used as if they were specified via --secret
• .env will only contain environment variables used for configuring earthly itself (e.g. EARTHLY_VERBOSE, EARTHLY_INTERACTIVE, EARTHLY_PUSH, etc); any unexpected keys will result in a warning being printed to stderr (to help with transitioning from 0.6 to 0.7 versions of earthly)

While a backwards breaking change is unpleasant, it will allow us to better isolate sensitive information, which will prevent the regression of this issue from re-appearing.

[alexcb]

This has been released under v0.7.0