Issue with configuration file to render the values from environment variables

[akoserwal]

What happened:

There is an issue with the configuration file to render the values from environment variables.

I am setting endpoint as environment variable
export TEST_ENDPOINT=localhost:8081

Setting $TEST_ENDPOINT in the config file.

server:
  http:
    addr: 0.0.0.0:8000
    timeout: 1s
  grpc:
    addr: 0.0.0.0:9000
    timeout: 1s
data:
  testDb:
    endpoint: "$TEST_ENDPOINT"

main.go

c := config.New(
		config.WithSource(
			env.NewSource("TEST_"),
			file.NewSource(flagconf),
		),
	)
	defer c.Close()

	if err := c.Load(); err != nil {
		panic(err)
	}

The environment variable gets resolved with KEY: "ENDPOINT," but it doesn't override the key value as "data.testDB.endpoint."

What you expected to happen:

Environment variable to get resolved as key value as "data.testDB.endpoint."

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kratos version (use kratos -v): v2.7.2
• Go version (use go version): 1.19-1.21
• OS (e.g: cat /etc/os-release):Mac os
• Others:

[dosubot]

🤖

Hey @akoserwal, great to see you back! 😊 Hope you've been doing awesome since our last chat.

The issue you're experiencing is due to the way the default resolver in Kratos handles environment variables. The default resolver only supports flat keys, meaning it can only resolve environment variables that correspond to top-level keys in the configuration. It doesn't support nested keys like "data.testDB.endpoint".

Here's a snippet of the default resolver code for reference:

o := options{
		decoder:  defaultDecoder,
		resolver: defaultResolver,
		merge: func(dst, src interface{}) error {
			return mergo.Map(dst, src, mergo.WithOverride)
		},
	}

If you want to override nested keys with environment variables, you would need to implement a custom resolver that supports this feature. This custom resolver should be able to parse the environment variable names into nested keys.

Unfortunately, without the implementation of the defaultResolver, I can't provide a specific code example on how to modify it to support nested keys. However, the general idea would be to split the environment variable name on a certain character (like underscore or dot) and use the resulting parts as nested keys.

You can find more information about the default resolver and its limitations in the Kratos repository.

I hope this helps clarify the issue. If you have any further questions, feel free to ask.

Sources

• config/config.go
• transport/grpc/resolver/direct/builder.go

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[kvii]

I think here is a bug. Consider to change "$TEST_ENDPOINT" to "${ENDPOINT}" to avoid this temporary.

I made a reproduce project at local. Some of key codes:

server:
  http:
    addr: "${ADDR}"

c := config.New(
	config.WithSource(
		env.NewSource("TEST_"),
		file.NewSource(flagconf),
	),
)
c.Load()

var bc conf.Bootstrap
c.Scan(&bc)

fmt.Println(bc.Server.Http.Addr)
fmt.Println(c.Value("ADDR").String())
fmt.Println(c.Value("server.http.addr").String())

After I execute export TEST_ADDR=0.0.0.0:8000, codes run properly(print "0.0.0.0:8000").

I guess there has some mistakes of the format of placeholder. The comment say that the correct format is "${key:default}".

kratos/config/options.go

Lines 85 to 97 in 9106991

	// defaultResolver resolve placeholder in map value,
	// placeholder format in ${key:default}.
	func defaultResolver(input map[string]interface{}) error {
	mapper := func(name string) string {
	args := strings.SplitN(strings.TrimSpace(name), ":", 2) //nolint:gomnd
	if v, has := readValue(input, args[0]); has {
	s, _ := v.String()
	return s
	} else if len(args) > 1 { // default value
	return args[1]
	}
	return ""
	}

[kvii]

It's a mistake of doc. kratos don't support "$key" format by default. I found this test:

kratos/config/options_test.go

Lines 132 to 136 in 9106991

	{
	name: "test $PORT",
	path: "foo.bar.value2",
	expect: "$PORT",
	},

[akoserwal]

Thanks, @kvii, for clarifying.

It works if the export TEST_ADDR=0.0.0.0:8000 and use fmt.Println(bc.Server.Http.Addr)

For boolean, I am getting

Error
panic: proto: (line 1:133): invalid value for bool type: "true"

data:
  test:
    endpoint: "${ENDPOINT}"
    useTLS: "${TLS:false}"

export TEST_TLS=true

[kvii]

Same issue at #1565. Unfortunately, non-string types are not supported in Scan because that will include more complexity. You can use c.Value("data.test.useTLS").Bool() to get the bool value explicitly. But I think this doesn't match your case.

[dosubot]

Hi, @akoserwal,

I'm helping the Kratos team manage their backlog and am marking this issue as stale. From what I understand, the issue involves a configuration file not rendering values from environment variables as expected in Kratos version v2.7.2 and Go version 1.19-1.21 on Mac OS. The environment variable "TEST_ENDPOINT" is set in the config file, but it does not override the key value as "data.testDB.endpoint." The default resolver in Kratos only supports flat keys, not nested keys, and a custom resolver that supports this feature would need to be implemented. There is also a discussion about the correct format of placeholders and handling non-string types in the comments.

Could you please confirm if this issue is still relevant to the latest version of the Kratos repository? If it is, please let the Kratos team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you!

[akoserwal]

Issue is still relevant, I have PR opened to fix this issue