Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat request: Markdown format #1200

Open
huitseeker opened this issue Jan 1, 2024 · 1 comment
Open

feat request: Markdown format #1200

huitseeker opened this issue Jan 1, 2024 · 1 comment

Comments

@huitseeker
Copy link
Contributor

huitseeker commented Jan 1, 2024

Summary

Github actions recently added the ability to output a Markdown-formatted test summary :
https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/

It would be great to have a straightforward Markdown output mode that would populate the test outcomes as a Markdown table.

Details

The nextest output is awesome for the CLI (seriously, thanks!), but contains some duplicate formatting for tests marked as slow, and is close to (but not quite a) Markdown table. For a (biased) example:

        PASS [  43.808s] arecibo tests::test_ivc_nondet_with_compression
        SLOW [> 60.000s] arecibo supernova::snark::test::test_compression_with_circuit_size_difference
        SLOW [> 60.000s] arecibo supernova::snark::test::test_nivc_trivial_with_compression
        SLOW [> 60.000s] arecibo tests::test_ivc_nontrivial_with_compression
        SLOW [> 60.000s] arecibo tests::test_ivc_nontrivial_with_spark_compression
        SLOW [> 60.000s] arecibo tests::test_pp_digest
        PASS [  81.895s] arecibo tests::test_ivc_nontrivial_with_compression
        PASS [  81.349s] arecibo tests::test_pp_digest
        SLOW [>120.000s] arecibo supernova::snark::test::test_compression_with_circuit_size_difference
        SLOW [>120.000s] arecibo supernova::snark::test::test_nivc_trivial_with_compression
        SLOW [>120.000s] arecibo tests::test_ivc_nontrivial_with_spark_compression
        PASS [ 138.131s] arecibo supernova::snark::test::test_nivc_trivial_with_compression
        PASS [ 136.331s] arecibo tests::test_ivc_nontrivial_with_spark_compression
        PASS [ 148.281s] arecibo supernova::snark::test::test_compression_with_circuit_size_difference

Other approaches / relevant resources

It's hard to find public repos using nextest in conjunction with $GITHUB_STEP_SUMMARY today.

It may be possible to scrounge up something out of the tool from the following (incomplete, not rigorously tested, doesn't cover ignored tests):

# Extract lines, then use awk to format as markdown table
grep -E "PASS|FAIL" | awk 'BEGIN { 
    print "| status | runtime | test |"; 
    print "| --- | --- | --- |"; 
} 
{
    match($0, /(PASS|FAIL) \[\s*([^\]]+)\]( .*?)/, groups);
    status = groups[1];
    runtime = groups[2];
    test = groups[3]
    gsub(/^ /, "", test); 
    print "| " status " | " runtime " | " test " |";
}'

... or something even more rigorous from the json output.

But this issue is written in the hope contributors might find a maintained markdown-summary output mode a fun endeavor (and that the maintainers would find it a valuable contribution), based on the GH action use case.

By comparison, there is the cargo pretty-test tool, which contains fixed-width output suitable for display with $GITHUB_STEP_SUMAMRY (as text).

See also #20

@sunshowers
Copy link
Member

Hi François, hope you're well! I agree that this would be great and I think nextest should learn how to write a markdown summary to a file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants