Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fastlane-core] recommends to retry uploading when AltoolTransporterExecutor crashes #21536

Merged

Conversation

TheMetalCode
Copy link
Contributor

@TheMetalCode TheMetalCode commented Sep 27, 2023

Checklist

  • I've run bundle exec rspec from the root directory to see all new and existing tests pass
  • I've followed the fastlane code style and run bundle exec rubocop -a to ensure the code style is valid
  • I see several green ci/circleci builds in the "All checks have passed" section of my PR (connect CircleCI to GitHub if not)
  • I've read the Contribution Guidelines
  • I've updated the documentation if necessary.
  • I've added or updated relevant unit tests.

Motivation and Context

Resolves #21535

After recently noticing some of my pilot runs fail due to no exit code being returned when running altool via FastlanePty.spawn, I investigated and eventually realized the error handling in AltoolTransporterExecutor#execute will work as designed if we simply force the return of some kind of exit code when this mysterious issue with altool occurs.

I do not know why altool is failing in this manner and hence this change isn't exactly a solution. What it does do though is provide a better error output when this condition does occur, such that it's much easier for a fastlane user to detect this precise failure and automatically retry the upload (which in my experience over the last few days always works successfully). With this change, paired with setting DELIVER_ALTOOL_ADDITIONAL_UPLOAD_PARAMETERS="--verbose" in the environment so altool is guaranteed to have some kind of stdout, the rather cryptic error output in #21535 turns into something much more informative:

 [23:00:55]: -------------------
[23:00:55]: --- Step: pilot ---
[23:00:55]: -------------------
[23:00:55]: Creating authorization token for App Store Connect API
[23:00:55]: Ready to upload new build to TestFlight (App: REDACTED)...
[23:00:57]: Going to upload updated app to App Store Connect
[23:00:57]: This might take a few minutes. Please don't interrupt the script.
[23:02:26]: [altool] 2023-09-26 23:02:26.633 DEBUG: [ContentDelivery.Uploader] There are 15 parts remaining to upload.

[23:02:26]: [altool] 2023-09-26 23:02:26.634 DEBUG: [ContentDelivery.Uploader] PROGRESS - PART 24 (5242880) - '38b61411-f580-4360-84be-927836472cb9.ipa' 100.00% (5242880/5242880)

[23:02:26]: [altool] 2023-09-26 23:02:26.636 DEBUG: [ContentDelivery.Uploader] PROGRESS - PART 25 (5242880) - '38b61411-f580-4360-84be-927836472cb9.ipa' 100.00% (5242880/5242880)

[23:02:26]: [altool] 2023-09-26 23:02:26.650 DEBUG: [ContentDelivery.Uploader] Wrote part 27 to a temp file '/private/var/folders/6j/z_jm_fbd1rb44rx_9513tbk40000gn/T/com.apple.cds.vbtx/com.apple.cds_5D5786F6-8A6D-4CD2-84AE-BA93E25696F8/Part-27.tmp'.

[23:02:26]: [altool] 2023-09-26 23:02:26.661 DEBUG: [ContentDelivery.Uploader] Adding upload task 32 for part 27.

[23:02:26]: [altool] 2023-09-26 23:02:26.662 DEBUG: [ContentDelivery.Uploader] Saving uploader state (CDUploaderStateCreateAssetChunksForUpload) for identifier 'com.apple.cds_5D5786F6-8A6D-4CD2-84AE-BA93E25696F8'.

[23:02:26]: [altool] 2023-09-26 23:02:26.679 DEBUG: [ContentDelivery.Uploader] Uploading part 20 COMPLETE.

[23:02:26]: [altool] 2023-09-26 23:02:26.679 DEBUG: [ContentDelivery.Uploader] COMPLETED - PART 20 - 38b61411-f580-4360-84be-927836472cb9.ipa - eTag: "AA281F37DC813E9F220C7616FC6080F7"

[23:02:26]: [altool] 2023-09-26 23:02:26.680 DEBUG: [ContentDelivery.Uploader] Wrote part 28 to a temp file '/private/var/folders/6j/z_jm_fbd1rb44rx_9513tbk40000gn/T/com.apple.cds.vbtx/com.apple.cds_5D5786F6-8A6D-4CD2-84AE-BA93E25696F8/Part-28.tmp'.

[23:02:26]: [altool] 2023-09-26 23:02:26.681 DEBUG: [ContentDelivery.Uploader] Removed temporary part file '/private/var/folders/6j/z_jm_fbd1rb44rx_9513tbk40000gn/T/com.apple.cds.vbtx/com.apple.cds_5D5786F6-8A6D-4CD2-84AE-BA93E25696F8/Part-20.tmp'.

[23:02:26]: [altool] 2023-09-26 23:02:26.682 DEBUG: [ContentDelivery.Uploader] Saving uploader state (CDUploaderStateCreateAssetChunksForUpload) for identifier 'com.apple.cds_5D5786F6-8A6D-4CD2-84AE-BA93E25696F8'.

[23:02:26]: [altool] 2023-09-26 23:02:26.686 DEBUG: [ContentDelivery.Uploader] There are 14 parts remaining to upload.

[23:02:26]: [altool] 2023-09-26 23:02:26.687 DEBUG: [ContentDelivery.Uploader] Uploading part 21 COMPLETE.

[23:02:26]: [altool] 2023-09-26 23:02:26.687 DEBUG: [ContentDelivery.Uploader] COMPLETED - PART 21 - 38b61411-f580-4360-84be-927836472cb9.ipa - eTag: "1F12B3E1BFAC15C3ACDCCF72A5B89EFE"

[23:02:26]: [altool] 2023-09-26 23:02:26.689 DEBUG: [ContentDelivery.Uploader] Removed temporary part file '/private/var/folders/6j/z_jm_fbd1rb44rx_9513tbk40000gn/T/com.apple.cds.vbtx/com.apple.cds_5D5786F6-8A6D-4CD2-84AE-BA93E25696F8/Part-21.tmp'.

[23:02:26]: [altool] 2023-09-26 23:02:26.689 DEBUG: [ContentDelivery.Uploader] Saving uploader state (CDUploaderStateCreateAssetChunksForUpload) for identifier 'com.apple.cds_5D5786F6-8A6D-4CD2-84AE-BA93E25696F8'.

[23:02:26]: [altool] 2023-09-26 23:02:26.693 DEBUG: [ContentDelivery.Uploader] Adding upload task 33 for part 28.

[23:02:26]: [altool] 2023-09-26 23:02:26.694 DEBUG: [ContentDelivery.Uploader] Saving uploader state (CDUploaderStateCreateAssetChunksForUpload) for identifier 'com.apple.cds_5D5786F6-8A6D-4CD2-84AE-BA93E25696F8'.

[23:02:26]: [altool] 2023-09-26 23:02:26.694 DEBUG: [ContentDelivery.Uploader] There are 13 parts remaining to upload.

[23:02:26]: [altool] 2023-09-26 23:02:26.695 DEBUG: [ContentDelivery.Uploader] PROGRESS - PART 27 (5242880) - '38b61411-f580-4360-84be-927836472cb9.ipa' 100.00% (5242880/5242880)

[23:02:26]: Application Loader output above ^
[23:02:26]: The call to the altool completed with a non-zero exit status: -1. This indicates a failure.
[23:02:26]: Could not download/upload from App Store Connect!
tput: unknown terminal "unknown"
tput: unknown terminal "unknown"
[23:02:26]: -------------------
[23:02:26]: --- Step: is_ci ---
[23:02:26]: -------------------
+-----------------------------------------------------+
| Lane Context |
+------------------+----------------------------------+
| DEFAULT_PLATFORM | ios |
| PLATFORM_NAME | ios |
| LANE_NAME | ios publish_testflight |
| FL_CHANGELOG | thank you sir may I have another |
+------------------+----------------------------------+
[23:02:26]: Error uploading ipa file:
[Application Loader Error Output]: The call to the altool completed with a non-zero exit status: -1. This indicates a failure.

+--------------------------------------------------------------------------+
| fastlane summary |
+------+-----------------------------------------------------+-------------+
| Step | Action | Time (in s) |
+------+-----------------------------------------------------+-------------+
| 1 | Verifying fastlane version | 0 |
| 2 | default_platform | 0 |
| 3 | whoami | 0 |
| 4 | Switch to ios set_appstore_connect_api_key lane | 0 |
| 5 | mkdir -p /tmp/appstore_connect || true | 0 |
| 6 | Switch to ios generate_release_build_changelog lane | 0 |
| 7 | changelog_from_git_commits | 0 |
| 8 | git_branch | 0 |
| 💥 | pilot | 91 |
| 10 | is_ci | 0 |
+------+-----------------------------------------------------+-------------+

[23:02:26]: fastlane finished with errors

[!] Error uploading ipa file:
[Application Loader Error Output]: The call to the altool completed with a non-zero exit status: -1. This indicates a failure.

Description

Before, AltoolTransporterExecutor#execute simply attempted to assert exit_code.zero? on the assumption that it was being given an integer exit code from the command when executed via FastlanePty.spawn. Now, it forces a -1 if it receives a nil from FastlanePty.spawn.

Testing Steps

@TheMetalCode TheMetalCode changed the title [fastlane-core] Force FastlanePty.spawn to return exit code [fastlane-core] AltoolTransporterExecutor#execute Handles Nil Exit Code from FastlanePty.spawn Sep 27, 2023
@lacostej
Copy link
Collaborator

lacostej commented Nov 5, 2023

Isn't the error that spawn returns a nil error code instead? I'd rather we fix that.

Looking at

def self.spawn(command)
require 'pty'
PTY.spawn(command) do |command_stdout, command_stdin, pid|
begin
yield(command_stdout, command_stdin, pid)
rescue Errno::EIO
# Exception ignored intentionally.
# https://stackoverflow.com/questions/10238298/ruby-on-linux-pty-goes-away-without-eof-raises-errnoeio
# This is expected on some linux systems, that indicates that the subcommand finished
# and we kept trying to read, ignore it
ensure
begin
Process.wait(pid)
rescue Errno::ECHILD, PTY::ChildExited
# The process might have exited.
end
end
end
$?.exitstatus
rescue LoadError
require 'open3'
Open3.popen2e(command) do |command_stdin, command_stdout, p| # note the inversion
yield(command_stdout, command_stdin, p.value.pid)
command_stdin.close
command_stdout.close
p.value.exitstatus
end
rescue StandardError => e
# Wrapping any error in FastlanePtyError to allow
# callers to see and use $?.exitstatus that
# would usually get returned
raise FastlanePtyError.new(e, $?.exitstatus)
end
, I am not 100% sure where it happens.

The last rescue block could maybe fail if an exception is thrown in the previous block, which could happen if we use popen2e, and an exception happens in the yield block, here a complex parse_line function. In that case there's no status to return either.

We could instead make the Pty classes more resilient and return -1 when the status is nil.

@lacostej
Copy link
Collaborator

lacostej commented Nov 6, 2023

I've made #21618 to handle the cases of a program crashing. In that case exitstatus could be nil which could be the explanation as to why it was nil here.

These extra lines shouldn't be necessary.

@TheMetalCode
Copy link
Contributor Author

I've made #21618 to handle the cases of a program crashing. In that case exitstatus could be nil which could be the explanation as to why it was nil here.

These extra lines shouldn't be necessary.

@lacostej To be clear: I was never confident that this code ever solved the issue with altool uploads intermittently failing and am really not attached to this code being merged or released. All this change did was give me just enough information to be able to know why my fastlane pilot actions were failing and to have enough info in the command output to trigger a retry on my end.

Today I have tried using #21618 instead of this code and it seems to work fine. The only hesitation being that since moving up to Xcode 15.0.1 I'm no longer seeing my pilot actions fail, so I cannot say for sure that your code handles the failure any better than mine. 🤷‍♂️

I'm a big fan of the added testing in #21618 so for that reason alone I'm totally fine if this PR closes and yours gets merged instead.

@lacostej
Copy link
Collaborator

lacostej commented Nov 17, 2023

I've made #21618 to handle the cases of a program crashing. In that case exitstatus could be nil which could be the explanation as to why it was nil here.
These extra lines shouldn't be necessary.

@lacostej To be clear: I was never confident that this code ever solved the issue with altool uploads intermittently failing and am really not attached to this code being merged or released. All this change did was give me just enough information to be able to know why my fastlane pilot actions were failing and to have enough info in the command output to trigger a retry on my end.

Today I have tried using #21618 instead of this code and it seems to work fine. The only hesitation being that since moving up to Xcode 15.0.1 I'm no longer seeing my pilot actions fail, so I cannot say for sure that your code handles the failure any better than mine. 🤷‍♂️

I'm a big fan of the added testing in #21618 so for that reason alone I'm totally fine if this PR closes and yours gets merged instead.

I think we should merge both PRs

  • mine improves the core
  • yours adds a useful tips to the user, as atool might still crash.

Once #21618 is merged, I also think you can remove the catch for nil in yours. I'd rather we bubble up those type of internal issues as they help improving the overall core, which benefits the whole fastlane overall. If it had been hidden, we would never have improved FastlanePty.

@@ -215,6 +215,8 @@ class AltoolTransporterExecutor < TransporterExecutor

private_constant :ERROR_REGEX

attr_reader :errors
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems safe, it makes testing the specific error line we're adding for the -1 return more straightforward

@TheMetalCode
Copy link
Contributor Author

Saw that #21618 merged. In response, I've removed the nil catch in fastlane_core/lib/fastlane_core/itunes_transporter.rb and have adjusted the test.

Copy link
Collaborator

@lacostej lacostej left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for updating the PR.

@lacostej lacostej changed the title [fastlane-core] AltoolTransporterExecutor#execute Handles Nil Exit Code from FastlanePty.spawn [fastlane-core] recommends to retry uploading when AltoolTransporterExecutor crashes Dec 14, 2023
@lacostej lacostej merged commit 07c62c5 into fastlane:master Dec 14, 2023
2 checks passed
SubhrajyotiSen pushed a commit to KeepTruckin/fastlane that referenced this pull request Jan 17, 2024
…xecutor crashes (fastlane#21536)

* Force FastlanePty.spawn to return exit code if it ends up being nil for whatever reason

* Actually, let's implement the fix right where the problem occurs, and add a test

* Update fastlane_core/lib/fastlane_core/itunes_transporter.rb

Co-authored-by: Aaron Brager <getaaron@gmail.com>

* Remove unneeded nil catch given fastlane#21618

* Write more accurate test

---------

Co-authored-by: Aaron Brager <getaaron@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[pilot] App Upload via altool Can Sometimes Exit Early With No Exit Code Or Stdout
3 participants