Add support for parallel tests to stub server

[maoberlehner]

Is your feature request related to a problem? Please describe.
In addition to classic contract testing, I also use the Specmatic stub server for my E2E web application tests. To make the tests run as fast as possible, I must run them in parallel. But this means that the stub server must be able to return different responses depending on the test that triggers the request.

Describe the solution you'd like
My test runner Playwright, gives me a TEST_PARALLEL_INDEX environment variable. I pass this index to /_specmatic/expectations as a x-parallel-index header. The web application under test sends this header to the Specmatic stub endpoint and therefore, it returns the correct stubs for my test. But there is one downside to this: I need to specify the x-parallel-index header in the API specification yaml. Which I want to avoid.

My current workaround:

{
    "http-request": {
        "headers": {
            "x-parallel-index": "env(TEST_PARALLEL_INDEX)"
        },
        "method": "get",
        "path": "/items"
    },
    "http-response": {
        "body": "..."
    }
}

openapi: 3.0.0
info:
  title: My API
  version: 1.0.0
paths:
  /items:
    get:
      # I don't want to specify this here!
      parameters:
        - in: header
          name: x-parallel-index
          schema:
            type: string
          required: false
      responses:
        '200':
          description: OK
          content:
            ...

Proposed solution:

{
    "http-stub-group": "env(TEST_PARALLEL_INDEX)",
    "http-request": {
        "method": "get",
        "path": "/items"
    },
    "http-response": {
        "body": "..."
    }
}

Additionally, I propose that sending an X-Specmatic-Group header would ensure that the stub server responds with the stub specified for the provided group.

Describe alternatives you've considered
I've also considered running multiple instances of my web app and Specmatic in parallel in Docker containers, and then parallelizing test runs on a file basis (run N test files in Docker container A, N in Docker container B...). But this would mean a lot of overhead.

[harikrishnan83]

@maoberlehner Thanks for reaching out. And appreciate the well articulated request. We will get back to you on this shortly.

[harikrishnan83]

@maoberlehner We went through your use case and your proposed solution. It is indeed something that we have also encountered the need for in the past. However in terms of the solution we would like to add to your suggestion and also ahve other community members to chime in (since we believe this will be useful to many people), so that we can discuss and finalise the approach. So I am converting this to a Github discussion to facilitate the same. Thanks. cc @jaydeepk @joelrosario

[harikrishnan83]

Hello @maoberlehner,

I am thinking that we make the x-parallel-index header possible to be included in your expectation, as you are already doing in your workaround, without having to define the same in the specification, you should be able to achieve the desired outcome. Will this work for you?

There by we may not have the need for the adding another property in the expectation ("http-stub-group": "env(TEST_PARALLEL_INDEX)").

Also just of curiosity wanted to check if you have had a chance to try our dynamic expectations. On a related note we also have transient expectations. Do have a look and let us know your thoughts.

Thanks,
@jaydeepk and @harikrishnan83

[maoberlehner]

Hello @harikrishnan83

I am thinking that we make the x-parallel-index header possible to be included in your expectation, as you are already doing in your workaround, without having to define the same in the specification, you should be able to achieve the desired outcome. Will this work for you?

Sounds good to me! Would be happy with this solution!

Also just of curiosity wanted to check if you have had a chance to try our dynamic expectations. On a related note we also have transient expectations. Do have a look and let us know your thoughts.

I used dynamic expectations but did not have a use case where I needed transient expectations.

Thx for your excellent work!

[harikrishnan83]

@maoberlehner We have an official docker image now. Please do have a look and let us know your thoughts on this.