Bug in tutorial with references

[createyourpersonalaccount]

In https://reframe-hpc.readthedocs.io/en/stable/tutorial.html#adding-performance-references there's a definition for myhost:baseline. But as I've discovered after testing, it is more like mysystem:mypartition.

This becomes more confusing since later in the tutorial (see https://reframe-hpc.readthedocs.io/en/stable/tutorial.html#id11) a baseline.py configuration is shown where a hostname myhost is defined for the mytutorialsys system and an environment with name baseline. The correct reference for that would be mytutorialsys:default.

There are three related issues overall with the presentation:

1. The issue I've described above; it is mysystem:mypartition for references and needs to be fixed in the tutorial.
2. The order of presentation should be reversed; we should first discuss configuration files and then references.
3. The valid_prog_environs is not clearly explained. It took me a while to figure out how to request two features, (with valid_prog_environs = ['+A +B'].) In retrospect, it is explained in https://reframe-hpc.readthedocs.io/en/stable/regression_test_api.html#reframe.core.pipeline.RegressionTest.valid_systems but it would be nice if the example showed more than just one feature.

If you agree with these issues I can fix them.

[vkarak]

In https://reframe-hpc.readthedocs.io/en/stable/tutorial.html#adding-performance-references there's a definition for myhost:baseline. But as I've discovered after testing, it is more like mysystem:mypartition.

Yes, this need fixing. I would probably go with generic:default which is what we are using and explain its meaning pointing to the explanation of systems/environment in the following section.

The issue I've described above; it is mysystem:mypartition for references and needs to be fixed in the tutorial.

Agreed.

The order of presentation should be reversed; we should first discuss configuration files and then references.

I'm not sure I'm in favor of this. Because if you want to run local run-only tests you can even go without dealing with the configuration. Configuration it's a whole chapter of its own and if it comes first, I think the point about how writing tests will be missed. The idea is to introduce the necessary parts of the configuration as the reader progresses with the tutorial.

The valid_prog_environs is not clearly explained. It took me a while to figure out how to request two features, (with valid_prog_environs = ['+A +B'].) In retrospect, it is explained in https://reframe-hpc.readthedocs.io/en/stable/regression_test_api.html#reframe.core.pipeline.RegressionTest.valid_systems but it would be nice if the example showed more than just one feature.

Agreed, but I would rather make the reader aware of this possibility and point them to the reference for more examples.

[createyourpersonalaccount]

@vkarak I'll take a look on Monday to submit a patch.

[createyourpersonalaccount]

I'll push it back a bit until the end of the week.

[vkarak]

I'll push it back a bit until the end of the week.

No rush, thanks!