Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve output of tests for frameworks which have no <IT::> #262

Open
hobovsky opened this issue Sep 5, 2023 · 3 comments
Open

Improve output of tests for frameworks which have no <IT::> #262

hobovsky opened this issue Sep 5, 2023 · 3 comments

Comments

@hobovsky
Copy link

hobovsky commented Sep 5, 2023

Initially asked on Discord:

Codewars runner supports some languages and testing frameworks which do not fit well into the BDD pattern and do not have constructs which map well into <DESCRIBE::> and <IT::> markers used by test output panel. As a consequence, some reporters emit artificially created <IT::> tags, with titles derived from some additional context data because authors cannot assign any title to them. This often results in an awkward output with confusing or misplaced titles (see #239 , #241 , codewars/FactCheck.jl#3 ), or which spam <IT::> beyond reason.

My proposal would be to get rid of artificial <IT::>'s from all reporters for frameworks which do not have a corresponding construct, and add support for assertions which are not in scope of any <IT::>. Maybe it would be good enough to allow for assertions inside of <DESCRIBE::>, or maybe a better solution would be to provide some new marker, which would be used for frameworks which do not map well into BDD and use simpler concept of a test and a context, for exampel something like:

<TEST::>
  <TEST::>                     (if nesting is allowed)
    <ASSERTIONS_CONTEXT::>     (optional)
      <ASSERTIONS_CONTEXT::>   (if nesting is possible)
        <ASSERTION::>          (probably <PASSED::> or <FAILED::>)
        <ASSERTION::>          (if not stop on first failure)
      <COMPLETEDIN::>
    <COMPLETEDIN::>
  <COMPLETEDIN::>
<COMPLETEDIN::>

With such setup, Julia reporter would emit <TEST::> for @facts (which is optional), <ASSERTIONS_CONTEXT::> for @context (which is optional), and <PASSED::> or <FAILED::> for @fact. For Clojure, we'd get <TEST::> for deftest (whici is mandatory), <ASSERTIONS_CONTEXT::> for testing (which is optional), and <PASSED::> or <FAILED::> for is.

@hobovsky
Copy link
Author

hobovsky commented Sep 5, 2023

For what it's worth I admit that the main reason behind this ticket is the fact how discouraged I can sometimes be to fix some translations, because no matter what I try, there's simply no way to get a nice and clear output of tests for some languages. This can be also a problem for reviewers who can be not familiar with intricacies of setup of some languages on Codewars and sometimes request to make the output of tests clearer before a translation gets approved, even though it's not possible/not easy.

@hobovsky
Copy link
Author

Another example: Perl

image

The test 1 blocks have no corresponding construct in code of tests and are most probably emitted by CW reporter to provide an obligatory <IT::>. It is not known how to provide a meaningful title for such it instead of the test N.

@hobovsky
Copy link
Author

Another affected language: Raku

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant