Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate end-to-end tests to staging environment and fix related issues #6263

Closed
wants to merge 5 commits into from

Conversation

niklasberglund
Copy link
Collaborator

@niklasberglund niklasberglund commented May 17, 2024

In this branch the tests are running against staging production instead of production. The following Linear tickets have been addressed:

When running tests against staging instead of production there was some unexpected issues which has been addressed too, such as timing issues caused by staging responding faster than production and differences in number location and relay availability.

Regressions in main has also been adressed in this PR, and an input parameter was added for when the workflow is manually triggered. You can now optionally specify a test case/suite to run. The format must be MullvadVPNUITests/test-suite-name/test-case-name for example MullvadVPNUITests/RelayTests/testAppConnection to run testAppConnection or MullvadVPNUITests/RelayTests to run all test cases under the suite RelayTests.
image

To try out the changes in this PR you might want to run all end to end tests. Do so either by triggering the workflow and specifying branch create-a-client-for-partner-api-ios-611 https://github.com/mullvad/mullvadvpn-app/actions/workflows/ios-end-to-end-tests.yml or by running them all locally from Xcode.

If running the tests locally you must first make sure your UITests.xcconfig has the two new configuration properties introduced:

// URL for Mullvad provided JSON data with information about the connection. https://am.i.mullvad.net/json for production, https://am.i.stagemole.eu/json for staging.
AM_I_JSON_URL = https:/${}/am.i.stagemole.eu/json

// Specify whether app logs should be extracted and attached to test report for failing tests
ATTACH_APP_LOGS_ON_FAILURE = 0

⚠️ Note that there's a backend bug causing connecting to relay to sometimes fail when doing so immediately after creating an account and adding time to it using partner API ⚠️


This change is Reviewable

@niklasberglund niklasberglund added the iOS Issues related to iOS label May 17, 2024
Copy link

linear bot commented May 17, 2024

@niklasberglund niklasberglund changed the title Create a client for partner api ios 611 Migrate end-to-end tests to staging environment and fix related issues May 20, 2024
@niklasberglund niklasberglund self-assigned this May 20, 2024
@niklasberglund niklasberglund force-pushed the create-a-client-for-partner-api-ios-611 branch from e9b0bc4 to 12038c1 Compare May 20, 2024 10:50
@niklasberglund niklasberglund force-pushed the create-a-client-for-partner-api-ios-611 branch 2 times, most recently from e1e6c50 to 49a7434 Compare May 20, 2024 11:55
@niklasberglund niklasberglund force-pushed the create-a-client-for-partner-api-ios-611 branch from 49a7434 to c36c6a9 Compare May 20, 2024 12:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
iOS Issues related to iOS
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants