Add a description with the reason why a test is pending #3324

rdennis · 2018-04-13T14:08:32Z

Requirements

Filling out the template is required. Any pull request that does not include enough information to be reviewed in a timely manner may be closed at the maintainers' discretion.
All new code requires tests to ensure against regressions.

Description of the Change

Added a reason property to Runnable and an optional reason parameter for it, specify, xit, xspecify, and it.skip. Also added the reason to reporters that made sense.

Alternate Designs

I considered a .because or .reason method that would accept the reason a test is marked pending, but it would require a larger change and can still be done after these changes.

Why should this be in core?

It is adding functionality to the core system.

Benefits

It will allow users to give a reason tests are pending. This functionality is helpful in that it lets teams track the status of pending tests in the test report. Right now, if a test is pending, it has to be documented in the test itself, or by some other external means.

Possible Drawbacks

It is possible that this implementation breaks existing tests in some case I have not considered. If those are found, it may be better to use a .reason or .because method instead.

Applicable issues

I have seen this functionality requested in #2026 and #2572.
This is an enhancement (minor release).

jsf-clabot · 2018-04-13T14:08:33Z

All committers have signed the CLA.

boneskull

Thanks for the PR!

While I think this feature is useful, I'm feeling modification of it's signature--and also the Runnable and Test constructors--is not on the table.

Instead, perhaps this.skip() could accept a reason argument. Then, we wouldn't need to modify each interface nor the constructors. Please see my code comments for further discussion.

Because I generally see it.skip() used in an ad-hoc, temporary manner, let's avoid modifying that API for now, and see if there's a pressing need, or if a change to this.skip() will be sufficient.

boneskull · 2018-04-18T18:26:21Z

lib/reporters/xunit.js

@@ -155,7 +155,8 @@ XUnit.prototype.test = function (test) {
    var err = test.err;
    this.write(tag('testcase', attrs, false, tag('failure', {}, false, escape(err.message) + '\n' + escape(err.stack))));
  } else if (test.isPending()) {
-    this.write(tag('testcase', attrs, false, tag('skipped', {}, true)));
+    var reasonTag = test.reason ? tag('reason', {}, false, '<![CDATA[' + test.reason + ']]>') : '';


I'm going to assume "reason" is a real thing per the XUnit spec? (I don't use XUnit, so I don't know)

@boneskull yes, in XUnit.net [Fact(Skip="reason")] or [Ignore("reason")] are used to skip a Fact/Test and provide a reason for skipping.

@Bamieh happy to have someone around that actually uses XUnit 😄

boneskull · 2018-04-18T18:28:05Z

lib/reporters/json.js

@@ -83,7 +83,8 @@ function clean (test) {
    fullTitle: test.fullTitle(),
    duration: test.duration,
    currentRetry: test.currentRetry(),
-    err: cleanCycles(err)
+    err: cleanCycles(err),
+    reason: test.reason


Can we suppress addition of the reason property in the case of a non-pending test?

Can we coerce reason to a String (e.g. reason: String(test.reason))

boneskull · 2018-04-18T18:28:42Z

lib/interfaces/bdd.js

@@ -78,12 +78,16 @@ module.exports = function (suite) {
     * acting as a thunk.
     */

-    context.it = context.specify = function (title, fn) {
+    context.it = context.specify = function (title, fn, reason) {


Please see my review summary regarding this API.

boneskull · 2018-04-18T18:30:35Z

lib/runnable.js

 */
-function Runnable (title, fn) {
+function Runnable (title, fn, reason) {


Please revert changes to this file (see review summary)

boneskull · 2018-04-18T18:30:48Z

lib/test.js

@@ -20,12 +20,13 @@ module.exports = Test;
 * @api private
 * @param {String} title
 * @param {Function} fn
+ * @param {String} reason


Please revert changes to this constructor. Even if reason must be a property of a Test or Runnable (I'm not entirely convinced), it should be renamed pendingReason and settable/gettable via methods.

Please investigate using the reason as the message of a thrown Pending object instead. I see at least one place where the Runner emits pending where we have access to the Pending instance itself... that might be all we need. Then, we can do something like this.emit('pending', test, pending.message) and the reporter can handle it.

boneskull · 2018-04-18T18:42:36Z

test/integration/pending.spec.js

@@ -32,6 +46,25 @@ describe('pending', function () {
        done();
      });
    });
+    it('should accept a reason when used via shorthand methods', function (done) {


tests look great 👍

boneskull · 2018-04-18T18:44:27Z

(Also this needs to be rebased onto master w/ conflicts resolved)

rdennis · 2018-04-23T02:57:23Z

Thanks for the feedback, I'll make these changes.

pdulapalli · 2018-05-17T16:54:22Z

Hello, I was curious about the status of this PR. This is something I would really like implemented in Mocha.

Would it be possible for me to make some of the changes you've recommended @rdennis to make?

rdennis · 2018-05-19T06:25:50Z

I think I've made all of the requested changes. The best source I could find for the xunit XML format (https://xunit.github.io/docs/format-xml-v2) implies a completely different format from that of the current xunit reporter, so I didn't bother adding reason to it this time. If anyone can find me an authoritative source for what the current reporter should output to properly display reason, I'll gladly add it.

outsideris · 2018-05-19T10:51:08Z

lib/runnable.js

+  if (arguments.length > 0) {
+    reason = String(reason);
+    this.reason = reason;
+  }


Why just check if (reason) {?

In a case ofit.skip(), arguments is { '0': undefined } and arguments.length is 1, because of this.runnable().skip(reason) in lib/context.js.
Therefore, reason is 'undefined' as string and it print like this:

- should skip immediately with reason (undefined)

Ah, good catch. Originally I has if (reason) {, but thought this would be better for some reason. I'll change it back.

I also considered using utils.isString to validate the input and throw an exception if invalid, but wasn't sure if that was needed here.

outsideris

undefinded is printed as string.

outsideris · 2018-05-19T11:05:40Z

I didn't use xunit as well.
Jenkins's junit xsd looks same with our format.

I guess we can use message in skipped element for skip reason.

<xs:element name="skipped">
    <xs:complexType mixed="true">
        <xs:attribute name="type" type="xs:string" use="optional"/>
        <xs:attribute name="message" type="xs:string" use="optional"/>
    </xs:complexType>
</xs:element>

outsideris · 2018-05-19T11:25:31Z

Additionally, it will be awesome if you add about reason in document.
Maybe here?

…results in no reason property on the result (instead of reason: undefined)

rdennis · 2018-05-20T04:26:53Z

Ok, I added support for before and beforeEach, but it required not passing a default message to new Pending in Runnable::skip (sync and async). I'm not sure if this is an acceptable change as I'm not sure what side effects it may have. Also, it just feels a little weird having before give the reason to every test in the suite. Maybe it just shouldn't be supported, but it seems counter intuitive to allow this.skip(reason) in spec, but not in before or beforeEach.

plroebuck · 2018-09-01T19:24:03Z

I hate to ask this (seeing how much work you invested), but why couldn't this problem be handled like this?

describe('suite description', function() {
  context('reason why this test is pending', function() {
    it.skip('test description');
  });
});

Though not as customized as your suggestion, would seem to give much of the same effect without any code changes to Mocha.

craigtaub · 2018-12-15T12:01:23Z

@rdennis have u had any time to look at this again?

JoshuaKGoldberg · 2023-12-02T21:05:46Z

👋 coming back to this @rdennis, are you still interested in working on this PR? As of #5027 there are again maintainers who can review it now. No worries if you no longer have time - but if you do that'd be great!

boneskull requested review from Bamieh and outsideris and removed request for Bamieh April 18, 2018 18:17

boneskull suggested changes Apr 18, 2018

View reviewed changes

outsideris added type: feature enhancement proposal area: usability concerning user experience or interface labels Apr 19, 2018

boneskull force-pushed the master branch 2 times, most recently from 4547268 to 7613521 Compare May 4, 2018 16:47

rdennis closed this May 19, 2018

rdennis force-pushed the issue/2026 branch from 983715e to df69279 Compare May 19, 2018 03:05

rdennis added 5 commits May 18, 2018 22:53

Added skip with message test for mochajs#2026

81a84be

Changed skip-with-message to skip-with-reason

24eed77

Added support for Runnable::skip(reason)

e36aefe

Coercing reason to string

198591b

Added reason to spec reporter

3f35fae

rdennis reopened this May 19, 2018

outsideris reviewed May 19, 2018

View reviewed changes

outsideris suggested changes May 19, 2018

View reviewed changes

rdennis added 5 commits May 19, 2018 09:08

Using if (reason) to avoid setting reason to undefined

08bf58c

Using message attribute of skipped element in xunit

8e45987

Added this.skip(reason) to docs

d9f8822

Added test to check that not using the reason argument for this.skip …

5e7a073

…results in no reason property on the result (instead of reason: undefined)

Added support for this.skip(reason) in async spec

9f64222

rdennis added 2 commits May 19, 2018 23:07

Changed no reason tests to make more sense

d575e92

Added support for this.skip(reason) in before and beforeEach

1064192

boneskull added the stale this has been inactive for a while... label Dec 18, 2018

plroebuck mentioned this pull request Feb 25, 2019

Fix this.skip() async calls #3745

Merged

stale bot closed this May 4, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a description with the reason why a test is pending #3324

Add a description with the reason why a test is pending #3324

rdennis commented Apr 13, 2018

jsf-clabot commented Apr 13, 2018 •

edited

boneskull left a comment

boneskull Apr 18, 2018

Bamieh Apr 19, 2018

boneskull Dec 18, 2018

boneskull Apr 18, 2018

boneskull Apr 18, 2018

boneskull Apr 18, 2018

boneskull Apr 18, 2018

boneskull Apr 18, 2018

boneskull commented Apr 18, 2018

rdennis commented Apr 23, 2018

pdulapalli commented May 17, 2018

rdennis commented May 19, 2018

outsideris May 19, 2018 •

edited

rdennis May 19, 2018

rdennis May 19, 2018 •

edited

outsideris left a comment

outsideris commented May 19, 2018

outsideris commented May 19, 2018

rdennis commented May 20, 2018

plroebuck commented Sep 1, 2018

craigtaub commented Dec 15, 2018

JoshuaKGoldberg commented Dec 2, 2023

Add a description with the reason why a test is pending #3324

Add a description with the reason why a test is pending #3324

Conversation

rdennis commented Apr 13, 2018

Requirements

Description of the Change

Alternate Designs

Why should this be in core?

Benefits

Possible Drawbacks

Applicable issues

jsf-clabot commented Apr 13, 2018 • edited

boneskull left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

boneskull commented Apr 18, 2018

rdennis commented Apr 23, 2018

pdulapalli commented May 17, 2018

rdennis commented May 19, 2018

outsideris May 19, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rdennis May 19, 2018 • edited

Choose a reason for hiding this comment

outsideris left a comment

Choose a reason for hiding this comment

outsideris commented May 19, 2018

outsideris commented May 19, 2018

rdennis commented May 20, 2018

plroebuck commented Sep 1, 2018

craigtaub commented Dec 15, 2018

JoshuaKGoldberg commented Dec 2, 2023

jsf-clabot commented Apr 13, 2018 •

edited

outsideris May 19, 2018 •

edited

rdennis May 19, 2018 •

edited