knapsack is skipping some rspecs

[hariapollo]

Hi, We are seeing that there are certain issues with knapsack lib with rspec execution. We have verified the following cases to confirm what could be the root cause.

Configuration details:

1. We use Github Actions to run our Rspec test cases.
2. We use same configuration for stage and production to run rspec with knapsack. Only difference is that we use 30 CI nodes in production and 20 CI nodes in Staging for rspec to run using Knapsack.

Cases we observed:

1. Commonly same specs are missing/skipping to execute in master branch ( But it is not consistent, sometimes it differes)
2. Commonly same specs are missing/skipping to execute in side branches in PR ( But it is not consistent, sometimes it differes)
3. Specs that are skipping/missing to execute in side branches and master are not same. What i mean is specs missing/skipping in side branch and master branch are different and vice versa. Due to this problem we are facing issues that some specs are not executing in side branches but when it comes to actions shows all tests passed and when PR merged to master those specs executed in master thus failed there.
4. Also, I have verifed the specs which are part of knapsack_rspec_report.json, the specs are part of this json file are not 100% executing in master or side branches.
5. The parallelism is not working properly. We are seeing some of the CI nodes are completing in 2/3 minutes where some completes in 20/30 minutes.

Please suggest us what could be the wrong here or is there any issue with knapsack distribution and execution.?

[ArturT]

Hi @hariapollo

If you could provide a reproducable repo with Github Actions setup and logs that confirm issues that would be helpful.

In the meantime you may consider trying the free plan for Knapsack Pro. It has Queue Mode, which dynamically split tests so that tests are auto-balanced between parallel CI nodes. You can learn more about it here.
https://docs.knapsackpro.com/ruby/queue-mode/

The parallelism is not working properly. We are seeing some of the CI nodes are completing in 2/3 minutes where some completes in 20/30 minutes.

How many parallel CI nodes do you have? If too many, you might saturate the CI machine and tests are getting executed slower.

With Knapsack Pro you can track the test files execution trend and observe if tests are becoming slower so that you can detect CI saturation issues. Add more CPU/RAM or reduce the number of parallel CI nodes to stabilize your CI server performance.

[hariapollo]

Hey @ArturT,

I'm sorry I would not be able to provide the repo to reproduce it.

Is Knapsack pro free plan is forever ?

I've one doubt are there any limitations for knapsack?. I mean is there any limitation for number of tests that only knapsack will run overall or anything like that?

How many parallel CI nodes do you have? If too many, you might saturate the CI machine and tests are getting executed slower.

We have 20 nodes running in Staging and 30 nodes on Prod. We use github actions to run our pipelines.

Is there anything I can look at to reproduce this issue and find the root cause?

[ArturT]

I've one doubt are there any limitations for knapsack?. I mean is there any limitation for number of tests that only knapsack will run overall or anything like that?

There is no limitation like that for the knapsack free gem.

Is Knapsack pro free plan is forever ?

You can check free plan limits here.

How many parallel CI nodes do you have? If too many, you might saturate the CI machine and tests are getting executed slower.

We have 20 nodes running in Staging and 30 nodes on Prod. We use github actions to run our pipelines.

Is there anything I can look at to reproduce this issue and find the root cause?

You can run one of your slow test files just with RSpec (rspec spec/my_slow_test_spec.rb) on your local machine and see what's the execution time.

Then compare it to the execution time of the test file on the GitHub Actions. If it's much slower on CI then CI might be saturated.

You can run the following example and preview recorded tests on a given CI node to check how long it took to execute the test files.

KNAPSACK_GENERATE_REPORT=true bundle exec rake "knapsack:rspec"
cat knapsack_rspec_report.json