Migrate end-to-end tests to staging environment and fix related issues #6263
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In this branch the tests are running against staging production instead of production. The following Linear tickets have been addressed:
When running tests against staging instead of production there was some unexpected issues which has been addressed too, such as timing issues caused by staging responding faster than production and differences in number location and relay availability.
Regressions in
main
has also been adressed in this PR, and an input parameter was added for when the workflow is manually triggered. You can now optionally specify a test case/suite to run. The format must beMullvadVPNUITests/test-suite-name/test-case-name
for exampleMullvadVPNUITests/RelayTests/testAppConnection
to runtestAppConnection
orMullvadVPNUITests/RelayTests
to run all test cases under the suiteRelayTests
.To try out the changes in this PR you might want to run all end to end tests. Do so either by triggering the workflow and specifying branch
create-a-client-for-partner-api-ios-611
https://github.com/mullvad/mullvadvpn-app/actions/workflows/ios-end-to-end-tests.yml or by running them all locally from Xcode.If running the tests locally you must first make sure your
UITests.xcconfig
has the two new configuration properties introduced:This change is