Support installing and running tests on a target

[LeBlue]

Is your feature request related to a problem?
I have been working on running the rauc unit-tests on a target system through the yocto ptest framework. To get it to work, i have patched the rauc (test) build process a bit.

If there is general interest in these changes, I can submit the related PRs.

Describe the solution you'd like
To support the on-target tests/ptest, the rauc build could support installing all unit-test programs and assets.

Additional changes i had to make/want to make:

• Improve test selection based on selected compile time features (bundle create, service, etc.)
• Improve/fix drop-ins e.g. grub-editenv

Describe alternatives you've considered
Most (all?) of the changes could be done in the yocto recipe, but:

• rauc version specific test changes do not have to be maintained in the recipe.
• implementing the necessary bits here instead of the yocto recipe makes the feature (running tests on a target machine) available outside of the yocto build.

Additional context
For the yocto bits, i can only provide and test the changes on the 'kirkstone' branch. I would need some help with that.

In addition I have serveral features disabled, namely casync, networking, streaming. Therfore I cannot really verify the tests will run successfully on the target.

[ejoerns]

@LeBlue Many of the tests require root access or specific kernel features to be enabled and are meant to be run in our qemu environment, but that should not prevent allowing those that do not have such strong restrictions.

[ejoerns]

[Pressed Send accidentially] 😏

If there is general interest in these changes, I can submit the related PRs.

I'd be interested. I guess having ptest support is at least good to align with other oe-integrated packages.

Improve test selection based on selected compile time features (bundle create, service, etc.)

If there are missing bits, I'd be happy to have them in the general test suite, yes.

I am also working on migrating the sharness-based tests (rauc.t) to pytest currently, thus I would be really interested if there are things I should pick up.

For the yocto bits, i can only provide and test the changes on the 'kirkstone' branch. I would need some help with that.

I could support with this, probably best in the context of meta-rauc repo then.

[LeBlue]

@ejoerns I have added a PR for the basic install changes. I am still working moving the recipe appends into meta-rauc.

If there are missing bits, I'd be happy to have them in the general test suite, yes.

I am aware the the rauc.t changes are probably outdated, but the idea of filtering for the create build option is still valid.

I am also working on migrating the sharness-based tests (rauc.t) to pytest currently, thus I would be really interested if there are things I should pick up.

For running the tests on a target, this is not a pleasant change, because now the runtime dependency of pytest is added. In addtition, the "ptest" API implementation is relativlely simple for the rauc.t script (Definitely simpler than pytest). But from a perspective of writing and (partially) executing test, pytest is probably superior.

[ejoerns]

@ejoerns I have added a PR for the basic install changes. I am still working moving the recipe appends into meta-rauc.

Thank you! I've added my comments there.

If there are missing bits, I'd be happy to have them in the general test suite, yes.

I am aware the the rauc.t changes are probably outdated, but the idea of filtering for the create build option is still valid.

Yes. It is inconsistent at the moment. Maybe this is something worth adding no matter if we support target tests or not.

I am also working on migrating the sharness-based tests (rauc.t) to pytest currently, thus I would be really interested if there are things I should pick up.

For running the tests on a target, this is not a pleasant change, because now the runtime dependency of pytest is added. In addtition, the "ptest" API implementation is relativlely simple for the rauc.t script (Definitely simpler than pytest). But from a perspective of writing and (partially) executing test, pytest is probably superior.

Maybe add a link to your corresponding commit here for reference. Not sure if the pytest tests are useful on the target anyway 😏

[LeBlue]

Not sure if the pytest tests are useful on the target anyway

The sharness tests (soon pytest) are usefull. e.g.
rauc.t: install (crypt). That fails if the kernel is missing the encryption options.

I just added the necessary options for encryption into the .config and this test failed on the target. Because i missed something.

[ejoerns]

Not sure if the pytest tests are useful on the target anyway

The sharness tests (soon pytest) are usefull. e.g. rauc.t: install (crypt). That fails if the kernel is missing the encryption options.

But that should have been revealed by the dm unit-test, too, doesn't it?

I just added the necessary options for encryption into the .config and this test failed on the target. Because i missed something.

The other thing is, if you really don't intend to use encryption. Then this is a false positive somehow.

[LeBlue]

But that should have been revealed by the dm unit-test, too, doesn't it?

Indeed it also failed.

The other thing is, if you really don't intend to use encryption. Then this is a false positive somehow.

A problem I also thought about already. One would need to be able to explicitly disable this test. This also applies to the verity format requirements.