Feature request summary

This issue tracks the addition of a new intermediate assertion to be used in uktest that is used for known failures which are temporarily and purposefully omitted from the failure count such that the internal test suite can pass.

This intermediate assertion level, to be introduced as FIX ME or UK_FIXME_{ASSERTF,EXPECT,etc...} is used to downgrade known failing test assertions and therefore not to be counted from the overall number of test failures. Explicitly, this makes the unikernel pass a test suite which would otherwise fail.

This is useful in a number of contexts and comes with explicit requirements so as to prevent misuse or to purposefully introduce error-prone code.

Unikraft is still under development and as a result, some tests from libc-test simply do not yet apply and therefore fail. This does not prevent Unikraft to operate under a number of known working conditions. Instead of simply disabling this test in lieu of this intermediate assertion level, the replacement with a FIX ME test is used to signify a test for a set of functionality that has yet to be implemented. Otherwise removing the test would a). potentially lead the test to be forgotten in the sands of time and never to be run again and b). mislead overall coverage of Unikraft's ability to handle the full suite of libc-tests.

To this end, introducing a new middle-ground that simply warns the user that a test has been omitted from the general failure count is intended to not only allow tests to pass that would enable the unikernel to operate but also help identify future sets of functionality that need to be implemented by Unikraft.

This addition comes with a strict requirement that must be included in uktest's documentation: changes from a general assertion to a FIXME assertion must come with an associated GitHub issue that is addressed as a comment preamble such that discussion and tracking can be done for the specific assertion.

For example, for a given assertion, the following diff is considered valid:

- UK_TEST_EXPECT_NOT_NULL(var)
+ FIXME(nderjung): See https://github.com/unikraft/unikraft/issue/xyz
+ UK_FIXME_EXPECT_NOT_NULL(var)

Finally, a strict-mode should be provided as an option to uktest which when enabled, still counts FIXME assertions towards the over failure count. This can be useful for debugging purposes.

Describe alternatives

No response

Related architectures

None

Related platforms

None

Additional context

This issue also acts as an RFC.

Seeking comments and feedback from: @skuenzer @marcrittinghaus @michpappas