Shouldn't this be part of the VC API test suite?

[peacekeeper]

Maybe it would be simpler to add (potentially optional) features to be tested by the existing VC API test suite, rather than creating a lot of separate test suites.

[OR13]

strongly agree.

[msporny]

The VC API test suite should be broken apart along "features". We are currently working w/ industry partners to create a more modular test suite for the Verifiable Credentials ecosystem, composed of a variety of different test suites that test individual features (such as status checking, credential refresh, issuing APIs, verification APIs, presentation APIs, etc.).

Having a giant monolithic test suite requires that all tests are run using the same framework and that just hasn't worked well for us.

Our goal here is to have a number of modular test suites that output the result of the tests in a common format. We are then going to build an VC ecosystem dashboard that shows how every vendor does against every test suite that we know of. We've been discussing this with Mesur.io and we seem to be on the same page as them.

I will also note that this is an experimental approach, and if it fails, the fallback is to integrate all of this back into the vc-api-test-suite. However, we don't see that as a scalable solution for doing VC ecosystem-wide testing for key features (and testing at different levels unit testing vs. integration testing vs. workflow testing).

Hope this helps explain some of the background on why we broke this test suite out into a separate one.

/cc @mprorock

[OR13]

The VC API test suite should be broken apart along "features".

No, it should not be broken apart, it should have small tests that only cover small features, and those tests should be co-located so developers don't have to crawl 90 ccg repos to figure out all the features and who supports what.

Having a giant monolithic test suite requires that all tests are run using the same framework and that just hasn't worked well for us.

Yes, don't rely on 1 test suite...rely on many test suites of varying complexity from "simple feature" to "complicated workflow"... then let folks decide which suites they want to demonstrate support for.

However, we don't see that as a scalable solution for doing VC ecosystem-wide testing for key features (and testing at different levels unit testing vs. integration testing vs. workflow testing).

I don't see splitting these test suites up and handling each of them in a different ccg repo as scalable, and I can't commit to supporting that approach, but that doesn't mean others are in the same boat... and I welcome copying from the trace interop repo if you want to split up its features across a wider range of repos... I just want to make it clear I am not going to help maintain any of that :)

[aljones15]

We could create a repo that clones all the various test suite repos and then runs the tests, but that would be a task for somewhere down the line.