Allow to read config also from environment #14

mudler · 2020-04-21T09:40:29Z

Fixes #13

With this change, now we don't fail anymore if a config file is not provided:

$ ./binaries/eirini-loggregator-bridge                     
Namespace:  
Loggregator-endpoint:  
Loggregator-ca-path:  
Loggregator-cert-path:  
Loggregator-key-path:  
namespace is missing from configuration

With a config file, we keep reading it correctly:

$ ./binaries/eirini-loggregator-bridge --config config.yaml
Namespace:  default
Loggregator-endpoint:  ca_path
Loggregator-ca-path:  ca_path
Loggregator-cert-path:  ca_path
Loggregator-key-path:  ca_path
{"level":"info","ts":1587461901.3673894,"caller":"kubeconfig/getter.go:79","msg":" does not exist, using default kube config"}

while we allow now to setup configuration via environment variables and also override the config file:

$ NAMESPACE=foo ./binaries/eirini-loggregator-bridge --config config.yaml 
Namespace:  foo
Loggregator-endpoint:  ca_path
Loggregator-ca-path:  ca_path
Loggregator-cert-path:  ca_path
Loggregator-key-path:  ca_path

Note: EIRINI_LOGGREGATOR_BRIDGE_LOGLEVEL=DEBUG must be set to show the config values that are used

Fixes #13

mook-as

Thanks for doing this! However, it doesn't seem to work for me:

$ env NAMESPACE=ns LOGGREGATOR-ENDPOINT=foo LOGGREGATOR-CA-PATH=bar LOGGREGATOR-CERT-PATH=baz LOGGREGATOR-KEY-PATH=quux ./binaries/eirini-loggregator-bridge
Namespace:  ns
Loggregator-endpoint:  foo
Loggregator-ca-path:  bar
Loggregator-cert-path:  baz
Loggregator-key-path:  quux
namespace is missing from configuration

While the debugging bits seem to get the settings, the viper.Unmarshal didn't seem to. Note that I didn't have a --config anywhere.

mudler · 2020-04-22T08:50:18Z

Thanks for the review! you are right, I think we are hitting spf13/viper#188 , I'll have a look!

mudler · 2020-04-22T09:58:03Z

I think I've got it:

$ NAMESPACE=too LOGGREGATOR_KEY_PATH=testkey LOGGREGATOR_CERT_PATH=cert_path LOGGREGATOR_ENDPOINT=loggregator_endpoint_test LOGGREGATOR_CA_PATH=test ./binaries/eirini-loggregator-bridge
Namespace:  too
Loggregator-endpoint:  loggregator_endpoint_test
Loggregator-ca-path:  test
Loggregator-cert-path:  cert_path
Loggregator-key-path:  testkey
{"level":"info","ts":1587549438.6308644,"caller":"kubeconfig/getter.go:79","msg":" does not exist, using default kube config"}
{"level":"info","ts":1587549438.6309066,"caller":"kubeconfig/checker.go:36","msg":"Checking kube config"}
invalid kube config: Get "http://localhost:8080/version?timeout=32s": dial tcp [::1]:8080: connect: connection refused

I've also changed the debug lines to print the data from the config, so it's more easy to spot issues

WIP: doing this aside with the InitContainer. The InitContainer can be dropped once cloudfoundry-incubator/eirini-loggregator-bridge#14 is merged.

mook-as · 2020-04-22T16:34:48Z

cmd/root.go

@@ -63,18 +66,45 @@ func init() {
 	rootCmd.PersistentFlags().StringVar(&kubeconfig, "kubeconfig", "", "kubeconfig file path. This is optional, in cluster config will be used if not set")
 }

+// See: https://github.com/spf13/viper/issues/188#issuecomment-399884438


This code is copied from a GitHub comment; what license is it under? We may need to rewrite it instead. (Also, I find the variable names to be opaque.)

Right, but otoh it won't get much different if we end up using reflection. I'll try a different approach then

mook-as · 2020-04-22T16:36:59Z

cmd/root.go

+		t := ift.Field(i)
+		tv, ok := t.Tag.Lookup("mapstructure")
+		if !ok {
+			continue


If I read the mapstructure docs correctly, this is doing the wrong thing: if the tag is missing, it should by default use the field name. Also, the tag may contain suffixes like ,squash and ,remain, so we should strip anything after a comma.

Alternatively, we can actually import mapstructure, and convert an instance of Config to JSON/YAML and have viper load that before loading the real config, according to a different comment in that issue.

If I read the mapstructure docs correctly, this is doing the wrong thing: if the tag is missing, it should by default use the field name. Also, the tag may contain suffixes like ,squash and ,remain, so we should strip anything after a comma.

Don't see the use case here, and the code would grow too much of complexity for a very tiny improvement that will make it even more fragile. I prefer to look after another way of doing it, avoding reflection at all ( I was unattracted by this solution as well, too ! )

I've been playing around with spf13/viper#188 (comment) here b1c8bd7 without luck - it doesn't seem to work anymore, and moreover forces to ship "defaults" which we tried to avoid here, prefering validation of proper input ( to avoid surprises during configuration )

WIP: it's not working yet

mook-as · 2020-04-23T14:06:20Z

cmd/root.go

+		}
+
+		emptyConfigReader := bytes.NewReader(emptyConfigBytes)
+		viper.MergeConfig(emptyConfigReader)


If I'm reading that viper bug correctly, the MergeConfig call needs to be done even if cfgFile == "".

I've tried already few times as well (also by dropping any check to cfgFile) , I just ended up defining the binds manually: 32059c9

See spf13/viper#761 to check how Unmarshal in Viper works under the hood

mudler · 2020-04-23T14:21:59Z

If looks good, feel free to squash when merging, no need for the whole history ( I kept it here for reference )

mook-as

Thanks! Sad that viper doesn't seem to work as well as we'd like, and we need the workarounds… ah well.

I'm happy with squashing as well, but I'll let you choose if you want to take the suggestions first.

mook-as · 2020-04-23T22:56:48Z

cmd/root.go

+	viper.SetDefault("NAMESPACE", "")
+	viper.SetDefault("LOGGREGATOR_KEY_PATH", "")
+	viper.SetDefault("LOGGREGATOR_ENDPOINT", "")
+	viper.SetDefault("LOGGREGATOR_CA_PATH", "")
+	viper.SetDefault("LOGGREGATOR_CERT_PATH", "")


I don't think we actually need these (since they're the zero values already).

Suggested change

viper.SetDefault("NAMESPACE", "")

viper.SetDefault("LOGGREGATOR_KEY_PATH", "")

viper.SetDefault("LOGGREGATOR_ENDPOINT", "")

viper.SetDefault("LOGGREGATOR_CA_PATH", "")

viper.SetDefault("LOGGREGATOR_CERT_PATH", "")

we need that, otherwise you cannot set them up properly without providing a config 🤷‍♂️

cmd/root.go

Co-Authored-By: Mark Yen <3977982+mook-as@users.noreply.github.com>

Allow to read config also from environment

3f23877

Fixes #13

mudler mentioned this pull request Apr 21, 2020

Allow config to be passed in individually #13

Closed

mudler requested a review from mook-as April 21, 2020 13:13

mook-as suggested changes Apr 21, 2020

View reviewed changes

fix: populate config struct with params from environment

3c1855d

mudler requested a review from mook-as April 22, 2020 11:17

mook-as suggested changes Apr 22, 2020

View reviewed changes

fix: use yaml annotation for config

b1c8bd7

WIP: it's not working yet

mudler force-pushed the read-env branch from 30730a5 to b1c8bd7 Compare April 23, 2020 13:03

mook-as reviewed Apr 23, 2020

View reviewed changes

fix: workaround viper issue when reading config from env

32059c9

mudler requested a review from mook-as April 23, 2020 14:21

mook-as approved these changes Apr 23, 2020

View reviewed changes

Do sprintf only when necessary

39151b7

Co-Authored-By: Mark Yen <3977982+mook-as@users.noreply.github.com>

mudler merged commit 3b15598 into master Apr 24, 2020

mudler deleted the read-env branch April 24, 2020 06:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow to read config also from environment #14

Allow to read config also from environment #14

mudler commented Apr 21, 2020

mook-as left a comment

mudler commented Apr 22, 2020

mudler commented Apr 22, 2020

mook-as Apr 22, 2020

mudler Apr 23, 2020

mook-as Apr 22, 2020

mudler Apr 23, 2020

mudler Apr 23, 2020 •

edited

mook-as Apr 23, 2020

mudler Apr 23, 2020 •

edited

mudler Apr 23, 2020 •

edited

mudler commented Apr 23, 2020

mook-as left a comment

mook-as Apr 23, 2020

mudler Apr 24, 2020

Allow to read config also from environment #14

Allow to read config also from environment #14

Conversation

mudler commented Apr 21, 2020

mook-as left a comment

Choose a reason for hiding this comment

mudler commented Apr 22, 2020

mudler commented Apr 22, 2020

mook-as Apr 22, 2020

Choose a reason for hiding this comment

mudler Apr 23, 2020

Choose a reason for hiding this comment

mook-as Apr 22, 2020

Choose a reason for hiding this comment

mudler Apr 23, 2020

Choose a reason for hiding this comment

mudler Apr 23, 2020 • edited

Choose a reason for hiding this comment

mook-as Apr 23, 2020

Choose a reason for hiding this comment

mudler Apr 23, 2020 • edited

Choose a reason for hiding this comment

mudler Apr 23, 2020 • edited

Choose a reason for hiding this comment

mudler commented Apr 23, 2020

mook-as left a comment

Choose a reason for hiding this comment

mook-as Apr 23, 2020

Choose a reason for hiding this comment

mudler Apr 24, 2020

Choose a reason for hiding this comment

mudler Apr 23, 2020 •

edited

mudler Apr 23, 2020 •

edited

mudler Apr 23, 2020 •

edited