[scan] replace the simctl boot command with simctl bootstatus, potentially fixing signal kill before running tests

[testableapple]

Checklist

• I've run bundle exec rspec from the root directory to see all new and existing tests pass
• I've followed the fastlane code style and run bundle exec rubocop -a to ensure the code style is valid
• I see several green ci/circleci builds in the "All checks have passed" section of my PR (connect CircleCI to GitHub if not)
• I've read the Contribution Guidelines
• I've updated the documentation if necessary.

Motivation and Context

Resolves #21025
Resolves #20341

Details

Sometimes on CI, the scan action fails with Early unexpected exit, operation never finished bootstrapping - no restart will be attempted. (Underlying Error: Test crashed with signal kill before starting test execution. error message. This problem disappears if hard-booting the simulator and waiting for it to launch before running the scan action.

So if we use the simctl bootstatus command instead of boot to load the simulator as part of the scan action, it will potentially solve this issue (at least it solves it for me). Keep in mind, that this will still require users to pass :prelaunch_simulator option to make it work:

scan(prelaunch_simulator: true)

To see the difference between bootstatus and boot use xcrun simctl help command, e.g.: xcrun simctl help bootstatus.

Testing

To test this branch, modify your Gemfile as:

gem 'fastlane', git: 'https://github.com/testableapple/fastlane.git', branch: 'wait-for-simulator-to-boot-before-scanning'

And run bundle install to apply the changes.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[foxware00]

@testableapple Do you have plans to get this merged?

[testableapple]

Hey @foxware00, I'd love to get this merged, but the PR first needs to be approved by the fastlane core team.

[foxware00]

Hey @foxware00, I'd love to get this merged, but the PR first needs to be approved by the fastlane core team.

@testableapple Might be worth adding @joshdholtz to the reviewers list. I'm also intermitently seeing this issue so would love to see if this resolves the stability of my CI boxes.

[rogerluan]

Hi @testableapple 👋

Sorry for the delay in reviewing this PR!

I did some digging and sounds like the 2 commands are actually equivalent from what I can tell, at least for the purpose we need here. Based on my research the diff would be that boot's documentation doesn't say it boots only "if it isn't already booted" like boostatus -b does, but calling this while the device is already booted shouldn't be causing any problems AFAICT, because this is not uncommon to happen — we'd be seeing more issues I think. The other diff would be that bootstatus -b reports boot status alongside the booting process, which you probably figured out too hence why you're piping not only the stderr to null but also stdout 👍 so this also shouldn't matter for the purpose we're trying to achieve here.

With that being said, if the two are equivalent and you somehow observed improvements using this different approach, we might as well give it a try and see if it solves issues for more people :) I'm ok merging this in, but perhaps we could check if other people observed improvements too!

It seems like many people emoji'd your PR, let's ask them for some help?

Summoning @dfed, @dcordero, @yichengchen, @TomaszLizer, @MihaiEros, @djtarazona, @pschneider, @spencerwhyte, @AlexPan1992, @jostnes, @Paul-van-Klaveren, @foxware00, since you guys emoji'd this PR (I suppose you're interested in the outcome of this PR, hence why I'm spam-tagging you all 🙇 ) — Would you share with us whether you tested this PR and observed improvements in this flaky behavior of the device booting? 🙇

Looking forward to seeing if others have noticed improvements too! Thanks everyone!

And special thanks to you @testableapple for opening this PR, your first here in fastlane land! 😁

[vince1393]

Our team is experiencing the same issue on our CI for Xcode 15 specifically (and 15.1). We tried using this branch but it didn't seem to fix it. We're using an iOS 16.4 simulator

[rogerluan]

@vince1393 the issue you're facing might be related to Xcode 15.x performance issues that Apple allegedly fixed in 15.0.1 and 15.1.0, but their fix isn't really working for 100% of people under all environments — especially under virtualized environments. For more info: https://discuss.circleci.com/t/severe-performance-problems-with-xcode-15/49205/126 (despite being a CircleCI thread, pretty much everything discussed there applies to all CI providers except Xcode Cloud).

I know this is not very helpful, but I thought it'd share what I know about the issue 🙇

[rogerluan]

It's been nearly 4 weeks since I asked for feedback from other users, I think we won't be getting any feedback, especially as the holidays season approaches 😄 I'll be merging this in as I find it a safe change and we can monitor whether it improves things or not!

If you're reading this after this PR has been released, feedback about this PR (or anything else hah!) is always welcome! 😃