Overriding with YAML defaults on a dataclass config

[tovacinni]

Hi again,

Today I was trying to override a config defined by a dataclass using a YAML file. The docs (https://brentyi.github.io/tyro/examples/03_config_systems/02_overriding_yaml/) seem to show that the use of a simple dictionary does work to override- but for a dataclass based config, it yields a bunch of warnings. A look into the source looks like it's looking for attributes, hence failing on a dictionary.

Is this the intended behaviour? (maybe it makes sense to assume attribute based accessors considering the config itself is a dataclass- but I found this discrepancy with what's indicated in the docs a bit confusing, unless I missed something that specifies this behaviour)

For completeness here's a small example to repro. Replacing the dict with an attrdict does work.

import yaml

import tyro
import dataclasses
import attrdict

@dataclasses.dataclass
class Config:
    exp_name : str
    batch_size : int

# YAML configuration. Note that this could also be loaded from a file! Environment
# variables are an easy way to select between different YAML files.
default_yaml = r"""
exp_name: test
batch_size: 10
""".strip()

if __name__ == "__main__":
    # Convert our YAML config into a nested dictionary.
    default_config = dict(yaml.safe_load(default_yaml))
    
    # Using attrdict here instead will work
    #default_config = attrdict.AttrDict(default_config)

    # Override fields in the dictionary.
    overridden_config = tyro.cli(Config, default=default_config)

[brentyi]

Hi!

So the way default is set up is if you write tyro.cli(SomeType, default=some_default), some_default should be an instance of SomeType.

This is hinted at in the signature of tyro.cli(), where the first argument (loosely) has type Type[OutT] and default has type Optional[OutT].

• The idea is that when f is set to Config and has type Type[Config], default should be of type Optional[Config].
• In the YAML example you pointed out, then when f is set to dict and has type Type[dict], default should be of type Optional[dict].

Does that general pattern make sense?

Another way to think about it is that the default value we expect should be valid on the right side of the equality here:

def main(config: Config = some_default) -> None:
    pass

tyro.cli(main)

[brentyi]

In terms of solutions, here are a couple I can think of. I personally prefer to avoid YAML files when possible, so don't feel super strongly about any particular one:

1. The first is to just go all-in on dictionaries. If you make Config a TypedDict:

from typing import TypedDict

import yaml

import tyro


class Config(TypedDict):
    exp_name: str
    batch_size: int


# YAML configuration. Note that this could also be loaded from a file! Environment
# variables are an easy way to select between different YAML files.
default_yaml = r"""
exp_name: test
batch_size: 10
""".strip()

if __name__ == "__main__":
    # Convert our YAML config into a nested dictionary.
    default_config = dict(yaml.safe_load(default_yaml))

    # Override fields in the dictionary.
    overridden_config = tyro.cli(Config, default=default_config)

2. You can convert the YAML into a dataclass instance after reading it. dacite is a nice library that should work:

import yaml

import tyro


@dataclasses.dataclass
class Config:
    exp_name: str
    batch_size: int


# YAML configuration. Note that this could also be loaded from a file! Environment
# variables are an easy way to select between different YAML files.
default_yaml = r"""
exp_name: test
batch_size: 10
""".strip()

if __name__ == "__main__":
    import dacite

    # Convert our YAML config into a dataclass instance.
    default_config = dacite.from_dict(Config, yaml.safe_load(default_yaml))

    # Override fields in the dictionary.
    overridden_config = tyro.cli(Config, default=default_config)

3. If you don't mind an unsafe load, you can also just have PyYAML directly serialize/deserialize dataclass instances.

4. tyro.extras.from_yaml / tyro.extras.to_yaml are deprecated, but let us construct dataclass instances from YAML-compatible strings via a custom dumper/loader:

import dataclasses

import yaml

import tyro


@dataclasses.dataclass
class Config:
    exp_name: str
    batch_size: int


# YAML configuration. Note that this could also be loaded from a file! Environment
# variables are an easy way to select between different YAML files.
default_yaml = r"""
!dataclass:Config
    exp_name: test
    batch_size: 10
""".strip()

if __name__ == "__main__":
    # Convert our YAML config into a dataclass instance.
    default_config = tyro.extras.from_yaml(Config, default_yaml)

    # Override fields in the dictionary.
    overridden_config = tyro.cli(Config, default=default_config)

[jhliu17]

This problem has puzzled me for a long time. Thanks for your clear explanations. I think parsing configurations from a given YAML file is an useful feature. Could you please explain what is the reason for making tyro.extras.from_yaml / tyro.extras.to_yaml deprecated?

[brentyi]

Could you please explain what is the reason for making tyro.extras.from_yaml / tyro.extras.to_yaml deprecated?

Mostly because serialization is hard and I wanted to focus more development effort on tyro's core competencies. The advantages over the standard yaml.dump(), yaml.load(), as well as libraries like dacite are also small.

Here are some thoughts I shared recently about config files + tyro, unfortunately for many people the experience may just be not good:

• If you don't have too many config files and are fine with reading all of them, one naive thing to do is just read them into a {filename: config} dictionary, which you can feed into tyro.extras.subcommand_type_from_defaults() like this:

  tyro/examples/03_config_systems/01_base_configs.py

  Lines 60 to 81 in 28082f8

  	{
  	"small": ExperimentConfig(
  	dataset="mnist",
  	optimizer=AdamOptimizer(),
  	batch_size=2048,
  	num_layers=4,
  	units=64,
  	train_steps=30_000,
  	seed=0,
  	activation=nn.ReLU,
  	),
  	"big": ExperimentConfig(
  	dataset="imagenet-50",
  	optimizer=AdamOptimizer(),
  	batch_size=32,
  	num_layers=8,
  	units=256,
  	train_steps=100_000,
  	seed=0,
  	activation=nn.GELU,
  	),
  	}

• You can also do some sys.argv hacking? If you assume the first positional argument of your script is that config file name, you can load a config file from sys.argv[1] and pass args=sys.argv[2:] into tyro.cli().
• If you haven't considered it already, I'd recommend just not using yaml-style config files entirely: in my experience just writing configs out in Python can be easier to maintain. There are some recent projects like LGM which have been fairly successful in following this pattern.