Missing documentation on using BUILDKITE_SPLITTER_IDENTIFIER when running multiple split steps

[TJC]

My use case:

• Run a step with test-splitter parallelism for regular unit tests
• Run a step with test-splitter parallelism for integration tests

I'm attempting to achieve this with the first step using BUILDKITE_SPLITTER_TEST_FILE_EXCLUDE_PATTERN to exclude the integration tests, and the second step using BUILDKITE_SPLITTER_TEST_FILE_PATTERN to only include the integration tests.

However in practice, the second step only picks up specs that should have been part of the first step!

ie. running BUILDKITE_SPLITTER_TEST_FILE_PATTERN=spec/foo/**/*_spec.rb test-splitter results in running tests from spec/bar which were definitely not part of that pattern.

It looks like our fix was to set BUILDKITE_SPLITTER_IDENTIFIER to a different unique value per step, however I note that this doesn't seem to be documented yet. Currently the documentation reads as follows:

[amybiyuliu]

Hi @TJC Thanks for reaching out!

Please try setting BUILDKITE_SPLITTER_IDENTIFIER to a different unique value for those two steps (eg BUILDKITE_SPLITTER_IDENTIFIER: "${BUILDKITE_BUILD_ID}_${BUILDKITE_STEP_ID}".

I think what happens with providing same identifier is that the test plan is cached after first time making request to our test plan API. In you second step, even you provide exclude/include pattern the filter won't be applied due to caching.

Let us know how you go with using a different identifier. We will also keep improving the documentation!

[TJC]

Yes, as I noted in the description, our fix did seem to be setting BUILDKITE_SPLITTER_IDENTIFIER.

The issue I'm raising is that this isn't really documented.

[amybiyuliu]

Thanks for providing us the feedback 💚
I've created an internal ticket to improve the documentation.

[TJC]

Is the buildkite splitter identifier truncated to a certain length in your systems?

I ask because when I tried using ${BUILDKITE_BUILD_ID}_${BUILDKITE_STEP_ID} I saw that the id was different on separate steps, but still encountered the problem that the second step re-used the plan from the first.

Switching back to my original method, things work as expected again.

[niceking]

Hey @TJC! Thanks for this additional feedback, and this also helps contextualise the other issue that you raised.

Just for background information, we cache the plan on unique combinations of BUILDKITE_SPLITTER_IDENTIFIER and BUILDKITE_SPLITTER_SUITE_TOKEN, so you can ensure uniqueness by either changing the value of BUILDKITE_SPLITTER_IDENTIFIER or BUILDKITE_SPLITTER_SUITE_TOKEN

Changing the value of BUILDKITE_SPLITTER_IDENTIFIER to ${BUILDKITE_BUILD_ID}_${BUILDKITE_STEP_ID} should work, and while we do have truncation for this field it doesn't kick in until 1024 characters so this is unlikely to be the issue you're seeing! It may be to do with how you're interpolating and passing through this value? If you're interpolating the value of BUILDKITE_STEP_ID on your pipeline definition, then it will have the step id of the upload step, rather than the step that is actually running the rspec specs. You might need to change your interpolation to happen within the testing environment so the correct step id value is used. Would appreciate if you could double check this and give that a go!

Alternatively, you could set up a second Test Suite to run your integration tests. That way, the data for your regular unit tests and the data for your integration tests are collected in different Test Suites, with different suite api tokens. This is what we do on our pipeline, as there are actually differences between the rspec setup and teardown for the two suites, and having the isolated data gave us better test splitting results. Does this sound like a use case for you as well? This would also solve the uniqueness issue :)

Sorry about our lack of documentation about this, we're still trying to figure out what other people's use cases are like, and what is sensible to recommend during the set up process!

[TJC]

Ah, thanks for explaining that aspect of the Step Id -- now I check the logs, it does rather look like the BUILDKITE_SPLITTER_IDENTIFIER ended up being the same everywhere, I think due to the interpolation happening too early.

[amybiyuliu]

Hi @TJC

We just released a new version 0.4.0 of the test splitter.

One of the update in this new release is to set theBUILDKITE_SPLITTER_IDENTIFIER default to BUILDKITE_BUILD_ID/BUILDKITE_STEP_ID.

If BUILDKITE_SPLITTER_IDENTIFIER is not provided, as long as we make BUILDKITE_BUILD_ID and BUILDKITE_STEP_ID accessible by the test splitter, it will use the default value and the uniqueness of the identifier can be guaranteed across different test steps.

As part of the release we also updated the documentation of the identifier. Please let us know if there's anything unclear about the docs 😊

[TJC]

One of the update in this new release is to set theBUILDKITE_SPLITTER_IDENTIFIER default to BUILDKITE_BUILD_ID/BUILDKITE_STEP_ID.

If BUILDKITE_SPLITTER_IDENTIFIER is not provided, as long as we make BUILDKITE_BUILD_ID and BUILDKITE_STEP_ID accessible by the test splitter, it will use the default value and the uniqueness of the identifier can be guaranteed across different test steps.

I tested this out, but it didn't seem to work for us when I removed the setting of BUILDKITE_SPLITTER_IDENTIFIER.

Perhaps I should check what exactly you mean by "as long as we make BUILDKITE_BUILD_ID and BUILDKITE_STEP_ID accessible by the test splitter" -- is there something specific required to enable that?

[wooly]

Perhaps I should check what exactly you mean by "as long as we make BUILDKITE_BUILD_ID and BUILDKITE_STEP_ID accessible by the test splitter" -- is there something specific required to enable that?

@TJC no action required to enable that in the tool. To clarify, the BUILDKITE_BUILD_ID and BUILDKITE_STEP_ID environment variables must be present in the environment in which the test-splitter is being run.

Can I confirm that these two environment variables are present when the BUILDKITE_SPLITTER_IDENTIFIER environment variable is removed?

[TJC]

Looking at the "Environment" tab of the Buildkite steps involved, I see that in both steps, those values are present.
The build id is the same for both, and the step id is unique to each.

[wooly]

Thanks for confirming.

Are you using the docker-compose plugin? If so, are the environment variables being passed through to the docker container?

[TJC]

Are you using the docker-compose plugin? If so, are the environment variables being passed through to the docker container?

We are using the docker-compose plugin. Looks like we needed to set propagate-environment: true. That has resolved the issue.

[wooly]

Good to hear!

I’ll add another clarification to the README.md to explicitly call out the environment in the wording.

I’ll close this issue now, much appreciated for your input @TJC!

[TJC]

Thanks for your help; glad I could help improve the docs.