Unmarshaling nested keys does not read from env

[andig]

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.
• I am not looking for support or already pursued the available support channels without success.
• I have checked the troubleshooting guide for my problem, without success.

Viper Version

1.8.1

Go Version

1.20.1

Config Source

Environment variables

Format

Dotenv

Repl.it link

https://replit.com/@andig1/nested-env?v=1

Code reproducing the issue

func TestEnv(t *testing.T) {
	viper.SetEnvPrefix("spf")
	viper.AutomaticEnv()
	os.Setenv("SPF_ID", "id")
	os.Setenv("SPF_FOO_BAR", "bar")
	viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))

	type Inner struct {
		Bar string
	}

	config := struct {
		Id  string
		Foo Inner
	}{
		Id: "1",
		Foo: Inner{
			Bar: "1",
		},
	}

	viper.Unmarshal(&config)
	fmt.Println(config)

	assert.Equal(t, "id", config.Id, "struct")
	assert.Equal(t, "bar", config.Foo.Bar, "struct")
}

Expected Behavior

Nested key foo.bar should be read from env.

Actual Behavior

Only root key id is read from env.

Steps To Reproduce

Run test

Additional Information

It seems the repl doesn't read either key from env. Not sure why.

[github-actions]

👋 Thanks for reporting!

A maintainer will take a look at your issue shortly. 👀

In the meantime: We are working on Viper v2 and we would love to hear your thoughts about what you like or don't like about Viper, so we can improve or fix those issues.

⏰ If you have a couple minutes, please take some time and share your thoughts: https://forms.gle/R6faU74qPRPAzchZ9

📣 If you've already given us your feedback, you can still help by spreading the news,
either by sharing the above link or telling people about this on Twitter:

https://twitter.com/sagikazarmark/status/1306904078967074816

Thank you! ❤️

[andig]

Update: even specifically binding the env key does not help:

viper.BindEnv("foo_bar")

Actually it does when using dot style:

viper.BindEnv("foo.bar")

Seems there are two issues here:

• nested struct keys are ignored by default and
• BindEnv requires dot style, not env style or does not pass the keys through the env replacer (more a documentation thing, but a bit unfortunate?)

[andig]

Finally, to workaround, I'm now pre-registering all env keys for the config struct like this:

// register all known config keys using github.com/fatih/structs and github.com/jeremywohl/flatten
flat, _ := flatten.Flatten(structs.Map(conf), "", flatten.DotStyle)
for _, v := range maps.Keys(flat) {
	viper.BindEnv(v)
}

[chr0n1x]

I think that I recently just hit the same issue as you but in a different form.

package main

import (
	"bytes"
	"fmt"
	"strings"

	"github.com/spf13/viper"
)

type (
	NestedConfig struct {
		Token string `mapstructure:"token"`
		Thing string `mapstructure:"thing"`
	}

	Config struct {
		Nested NestedConfig `mapstructure:"nested",omitempty`
	}

	ConfigWrapper struct {
		Config Config `mapstructure:"wrapper_config_attr"`
	}
)

func main() {
	viper.SetConfigType("yaml")
	viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
	var yamlExample = []byte(`
wrapper_config_attr:
    nested:
        thing: weeee
`)

	viper.ReadConfig(bytes.NewBuffer(yamlExample))
	viper.AutomaticEnv()

	var wrapper ConfigWrapper
	if err := viper.Unmarshal(&wrapper); err != nil {
		fmt.Println(err)
		return
	}
	fmt.Printf("%+v\n", wrapper)
	fmt.Printf("wrapper_config_attr.nested.token=%+v\n", viper.Get("wrapper_config_attr.nested.token"))
}

Note that in the initial YAML provided to viper to read, wrapper_config_attr.nested.token does not exist. So running the program above as-is:

# WRAPPER_CONFIG_ATTR_NESTED_TOKEN=bar go run testing.go
{Config:{Nested:{Token: Thing:weeee}}}
wrapper_config_attr.nested.token=bar

The way that I got around this was to add a line somewhere before the Unmarshal call specifying a default value:

viper.SetDefault("wrapper_config_attr.nested.token", "<replace-me>")

# WRAPPER_CONFIG_ATTR_NESTED_TOKEN=bar go run testing.go
{Config:{Nested:{Token:bar Thing:weeee}}}
wrapper_config_attr.nested.token=bar

If I had to guess, this has something to do with the dynamic nature in which viper reads environment variables. But that's speculation, I did not dive into the code to find a root cause.

[andig]

I've just come across #761 (comment) but didn't get time to test that yet. I assume this would do some sort of double roundtrip to fetch all the keys first and then register them explicitly.

In fact, I think I should close this issue given #761.

[chr0n1x]

ahhhh good find @andig , thank you!