Cypress test results not showing up in DataDog for a monorepo sub-project

[jdborneman-terminus]

Alright, I'm at my wit's end. Posting this in Discussion though since it's likely User Error more than a Bug. 😬

I've got Project A and Project B.

Project A: A stand alone Cypress project with dd-agent integrated. When I run $ DD_ENV=local DD_SERVICE=cypress-poc npx cypress run everything works great, stuff shows up in DataDog just as I'd expect.

Project A Details
Execution Command
From root directory of project: $ DD_ENV=local DD_SERVICE=cypress-poc npx cypress run

plugins/index.js

/**
 * @type {Cypress.PluginConfig}
 */
// eslint-disable-next-line no-unused-vars

module.exports = (on, config) => {
  // `on` is used to hook into various events Cypress emits
  // `config` is the resolved Cypress config
  require('cypress-grep/src/plugin')(config)  
  require('dd-trace/ci/cypress/plugin')(on, config)
  on('task', {
    log (message) {
      console.log(message)
      return null
    }
  })

  return config
}

support/index.js

// Import commands.js using ES2015 syntax:
import './commands'

// Alternatively you can use CommonJS syntax:
// require('./commands')

require('cypress-grep')()
require('dd-trace/ci/cypress/support')

Project B: A monorepo project with the Cypress tests existing as a sub-project within the monorepo. When I run $ DD_ENV=local DD_SERVICE=my-app npx cypress run nothing errors but nothing shows up in DataDog.

Project B Details
Execution Command
From cypress project subdirectory (./apps/my-app-e2e/) directory of project: $ DD_ENV=local DD_SERVICE=my-app npx cypress run

plugins/index.js

module.exports = (on, config) => {
  // register cypress-grep plugin code
  // https://github.com/cypress-io/cypress-grep
  require('cypress-grep/src/plugin')(config)

  // register dd-trace for DataDog reporting
  // https://docs.datadoghq.com/continuous_integration/setup_tests/javascript/?tab=cypress#compatibility
  require('dd-trace/ci/cypress/plugin')(on, config)

  on('task', {
    log (message) {
      console.log(message)
      return null
    }
  })
  return config
}

support/index.ts

// load and register the grep feature
// https://github.com/cypress-io/cypress-grep
require('cypress-grep')();

// Import commands.js using ES2015 syntax:
import './commands';

// Alternatively you can use CommonJS syntax:
// require('./commands')


// ... removed some of our own support methods here ...

// load and register DataDog support
// https://docs.datadoghq.com/continuous_integration/setup_tests/javascript/?tab=cypress#compatibility
require('dd-trace/ci/cypress/support')

// returning false here prevents Cypress from failing the test
Cypress.on('uncaught:exception', (err, runnable) => false);

"dd-trace": "2.0.1" is set up in our package.json (I was originally at 1.6.0, upgraded to 2.0.1 in an attempt to fix this.)

The biggest difference I see as I type this up is that in our failing Project B, the support index is a .ts file, not a .js. But I changed Project A's support index to be a .ts and it still reports to DD just fine, so that's not it.

I'm testing this on my local dev machine (a Mac) with the datadog agent installed. Looking at the agent logs, I do see a line after I kick off Project B that says 2022-02-07 10:46:03 EST | CORE | INFO | (pkg/serializer/serializer.go:371 in sendMetadata) | Sent metadata payload, size (raw/compressed): 957/291 bytes but not sure if that's what I need to look for in the Agent logs, or if there's a way to turn on some sort of verbose mode to see more there.

[jdborneman-terminus]

@juan-fernandez you were such a great help awhile back on an issue I had getting this working in my poc, I figured I'd tag you here as well. Appreciate any feedback you may have as to why this isn't working.

My head's sore from banging it against my desk trying to figure this out 😁

[jdborneman-terminus]

I even just now saw Successfully posted payload to "https://7-32-1-app.agent.datadoghq.com/api/v1/check_run?api_key=***************************xXXXX" in the DD Agent log. So it seems like stuff is getting posted....... so why can't I find it in DD? 😞

[jdborneman-terminus]

My Project A POC is showing up fine when I run it locally, but my other project is not.

[juan-fernandez]

hi @jdborneman-terminus ! Thanks for sharing your issue! Let's see if we can sort it out 😄 .

My first suspicion when seeing the post was an issue with the agent's configuration, but seeing your replies, you're able to report tests with Project A, so it must be something else.

Let's try to run Project B with these env vars: DD_TRACE_DEBUG=true DD_TRACE_LOG_LEVEL=error
Also, what cypress version are you using in both project A and B?

[jdborneman-terminus]

Yeah confirmed, the running specs (that ALL run the before defined in support/index.ts) all say Because this error occurred during a 'before all' hook we are skipping all of the remaining tests.

[jdborneman-terminus]

So yeah, with before..... I'm seeing how this is a tough nut to crack. It only runs ONE test per spec, what should it even report? Should it just report 1 failed test I guess for the single test that ran (and the rest were skipped)?

[juan-fernandez]

I see. Thanks for the clarification.

I think this is what's happening: no test in the test suite runs if before fails, not even the first one (its beforeEach, it and afterEach functions do not run), so we have no test to associate the error with. Correct me if I'm wrong on this, but from what I'm seeing, no actual test in the test suite is run.

This is indeed a gap in our product currently, and it is in our roadmap to come up with a solution: we map tests to traces but we don't map test suites to anything. This means that before hooks can't be associated to any trace. What test would we associate an error happening in the before with?

Does that make sense?

[jdborneman-terminus]

Makes perfect sense @juan-fernandez. Thanks for taking a look and clarifying this issue for me. I'll keep an eye on your release notes to see if you guys are able to come up with a solution!

My 2 cents is that it should map as one failure to the first test that was queued up to run (similar to how Cypress and Cypress.io dashboard do it) Just not sure if that's possible from a dd-trace standpoint.

Thanks again!

[juan-fernandez]

thank you! I'll share this feedback with the team. It might make sense as a temporary solution

[juan-fernandez]

To sum up:

If a before hook fails and no other test is run for that test suite, no test data for that test suite reaches Datadog. This is because currently we have no support for "test suite visibility", that is, we don't map test suites to anything (it is in our roadmap to come up with a solution to this). A temporary workaround could be to associate the before error to the first test that was queued up to run.