Bitcode enabled framework not accepted by Appstore

[legion151]

Please ensure you have given all the following requested information in your report.

Issue details

Bitcode support was added in
#443
which works so far.
At least for framework target. If framework is build with enabled Bitcode one can also build an app with with bitcode.
However the very app is then rejected by apples store during upload:

Invalid Bundle* - The app cannot be processed because options not
allowed to be embedded in bitcode are detected in the submission. It is likely
that you are not building the app with the toolchain provided in Xcode. Rebuild
your entire app with the latest GM Xcode and submit the app again.

Also recompiling the app with "Rebuild from Bitcode" option set, fails, resulting in:

ipatool failed with an exception [...]

Reproduction steps/code

One might use a clone of
https://github.com/ecsec/open-ecard-ios
which should allow to reproduce the steps mentioned above.

Versions:

• Robovm: 2.3.10
• XCode: 11.x and 12.x
• JDK: 14

Build Targets:

• Appstore

[dkimitsa]

@legion151
played with simple case:

• created a framework project from template.
• created an empty Xcode ObjC project
• added framework there and used it

Archived and was able to rebuild from bitcode. Could you please try with simple case ? As there might other points of failure in your project.

PS: It was required to fix a bit ipatool in Xcode12 as it failed with exception:

Assertion failed: Expected 4 archs in otool output:

added -arch all

  CmdSpec.new(OTOOL_PATH, ["-arch", "all", "-h", "-l", "-v", "-m", path.to_s]).run(0, false, true)

[legion151]

I now tried with

• Swift-project (freshly created)
• our roboface-example framwork

The first weird thing is, that by normal adding our framework, XCode complains that it lacks bitcode (which it doesn't).
After adding the framework via cocoapod i can compile and use the testapp with bitcode switched on.

However, recompiling it fails for me, also when i use your mentioned patch.
I get another error anyway:

ipatool failed with an exception: #<CmdSpec::NonZeroExitException: $ /Users/ecsec/Downloads/Xcode-beta.app/Contents/Developer/usr/bin/python3 /Users/ecsec/Downloads/Xcode-beta.app/Contents/Developer/usr/bin/bitcode-build-tool -v -t /Users/ecsec/Downloads/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin --sdk /Users/ecsec/Downloads/Xcode-beta.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.2.sdk -o /var/folders/5v/r5693gyx7lv1mh9q4fzng2w80000gq/T/ipatool20201022-3881-5ovwwl/thinned-out/arm64/Payload/roboface-test.app/Frameworks/MyFrameworkProtocol.framework/MyFrameworkProtocol --generate-dsym /var/folders/5v/r5693gyx7lv1mh9q4fzng2w80000gq/T/ipatool20201022-3881-5ovwwl/thinned-out/arm64/Payload/roboface-test.app/Frameworks/MyFrameworkProtocol.framework/MyFrameworkProtocol.dSYM --strip-swift-symbols /var/folders/5v/r5693gyx7lv1mh9q4fzng2w80000gq/T/ipatool20201022-3881-5ovwwl/thinned-in/arm64/Payload/roboface-test.app/Frameworks/MyFrameworkProtocol.framework/MyFrameworkProtocol
Status: pid 3923 exit 1
Stdout:

I will now try with ObjC.

[legion151]

ObjC is exactly the same.

I now try to build a completey new framework.

[legion151]

So i played around a little. I have to admit that I am not really sure how to build a template robovm framework via mvn compiler plugin. Since the newest sample for this is from 2015.
I used our roboface-example framework and have the same issues as mentioned before.

Is it possible, that the maven stack can introduce the problems?

[dkimitsa]

@legion151 create a new framework project using Idea -- and then build it with maven ?

[legion151]

Hi and sry for the delay. Today i had some time to look this again.

I now created new Framework via your description.
With the exception of enabling bitcode.

Using gradle i must have set that wrong because XCode tells me the framework doesn't contain any bitcode.
I used build.gradle option and robovm.xml option.

Using maven i got a framework which i could pack into an app and run it on device with bitcode enabled.
However I still get the ipatool error mentioned above.

i tried this with several versions of XCode with same results and wasn't able to find a way to fix ipatool.
i found a better log stating:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang\n Clang option verification failed for bitcode 0001

which comes from ipatool line 373.

May i ask you, which versions you used to get a recompiling done?

I have:

• macOS Big Sur 11.0.1
• Xcode 12.2
• IntelliJ IDEA 2020.2
• roboplugin (updated today)
• robvm-version 2.3.11
• openjdk8 installed via brew and set manually in intelliJ

[dkimitsa]

@legion151 Hi,
have retried again using Xcode 12.2/macOS 11.0.1
have same results as before on simple/empty project. after adapting otool invocation as mentioned above -- was able to rebuild from bitcode.
My steps:

• used Idea to create new framework project. enabled bitcode in robovm.xml and built framework using RoboVM menu in idea;
• created empty Xcode objc project, dropped framework there, called test method to check if it works.
• adapted ipatool
• archive/rebuild from bitcode

[legion151]

Followed these steps again. No success for me.
As said ipatool crashes for me but with an an error which is not solved with your patch and i guess is a different problem.

Anyway since not reproducable i will close this issue.

You could do me a favour if u've got the time.
I just uploaded frmwk.zip which is the unchanged framework built via the plugin of intelliJ. If you have the time you could try to recompile with that. This would tell me if i at least can generate correct frameworks and my Xcode stuff is broken or if the problem exists in earlier steps.
You don't have to, i feel i've stolen enough of your time...

And in the end - bitcode seems not to be that critical as long as it is not enforced by apple.

Thx so far.

[dkimitsa]

@legion151
hey, played with your framework -- and it fails indeed.
digged a bit -- it fails while trying to validate Clang options saved to rebuild bitcode. These can be extracted using following command line:

otool -v -s __LLVM __bundle bitcodetest202011_mvn

Xcode script fail to validate due to lot of empty parameters presents (in general the list of parameters looks as repeatable mess):

   <clang>
    <cmd>-triple</cmd>
    <cmd>arm64-apple-ios8.0.0</cmd>
    <cmd>-emit-obj</cmd>
    <cmd>-disable-llvm-passes</cmd>
    <cmd>-faligned-alloc-unavailable</cmd>
    <cmd>-target-sdk-version=13.6</cmd>
    <cmd>-target-abi</cmd>
    <cmd>darwinpcs</cmd>
    <cmd>-Os</cmd>
    <cmd/>
    <cmd/>
    <cmd/>
    <cmd/>
    <cmd/>
    <cmd>-triple</cmd>
    <cmd>arm64-apple-ios8.0.0</cmd>
    <cmd>-emit-obj</cmd>
    <cmd>-disable-llvm-passes</cmd>
    <cmd>-faligned-alloc-unavailable</cmd>
    <cmd>-target-sdk-version=13.6</cmd>
    <cmd>-target-abi</cmd>
    <cmd>darwinpcs</cmd>
    <cmd>-Os</cmd>
    <cmd/>
    <cmd/>
    <cmd/>
    <cmd/>
    <cmd/>
    <cmd>-triple</cmd>
    <cmd>arm64-apple-ios8.0.0</cmd>
    <cmd>-emit-obj</cmd>
    <cmd>-disable-llvm-passes</cmd>
// another 14000 similar lines go there

Also have compared your binary with my. and yours contains too little of bitcode file section, like 48 vs 1103.

In general it would be great if you share a source code of dummy framework and command line parameters used to build it, just to check if I get result similar to yours.
Thanks

[legion151]

First things first: Thank you very much for looking at this.

Your findings show, that my machine allready fails at gettin the framework correctly build.

I assume the otool command you gave lacks a space between LLVM and bundle.
Calling that returns an empty result for me.

I didn't really change a thing in the framework template which comes with robovm intelliJ plugin, except for enabling bitcode in config-files.
I didn't use commandline commands, only the robovm menu in intellij ,so no parameters involved.

The source can be found here

I there will also push a logfile of intelliJ building the framework

[dkimitsa]

@legion151 hi,
here few moments while I was asking for step about the broken frameworks. it doesn't seem that it corresponds the source/was built with Idea plugin as:

• it enables only ARM64 arch. But logs in your repo tells that fat binary of 4 arches was built;
• it name of shared framework bitcodetest202011_mvn doesn't match one in log.

It seems that shared (and one with defects) probably was built using maven tool. Would be great to have step you were following to build it.
Please correct me if I'm wrong. Thx!

[legion151]

Hi there. I setup a new framework project for the git repo which i linked.
There I used gradle for building.
However I get the same results, no matter if I use gradle or maven.
With maven I didn't do changes, except for enabling the bitcode either.

[dkimitsa]

@legion151 can you please share new binary ? thx

[legion151]

I'll try.
gradle:

• github
• binary

mvn (created today):

• github
• binary

each framework shares the same problem for me.

Another example can be found here, which is the actual framework we're working at.

[dkimitsa]

thanks, same thing there, empty <cmd/>
one more moment, please share gradle/maven command line you used to build.

[dkimitsa]

ok, reproduced on different Mac. will check what is a difference between these two

[legion151]

What do you mean by "command line".
Creating and consuming the template projects, i don't use comands at any point?
Do I miss a step?

[dkimitsa]

@legion151 when you mentioned that you were using gradle for building -- I thought that you were building from command line using gradlew. But if you just created gradle kind of project and build with RoboVm-Create Framework menu -- then it doesn't matter.

[legion151]

I couldn't find new information, yet.
Did you?

[dkimitsa]

@legion151
yep, I've found the root case while bc got packed.
it happens when librobovm-rt is partial linking during the build. it also has different side effect so I had to revert another improvement (check #544 for details)

in case of BC it can be solved. Will create a PR tomorrow. But probably we can target it for testing to v2.3.13

[legion151]

Awesome! \o/

Once again thanks for your great work.
Looking forward to test this.

Thx

[legion151]

Since this is closed but open (tested today with current main-branch, which still shows this), i would like to bump. Don't know when the next release is planned though.

[novacom34]

Hi @legion151 @dkimitsa any updates?
I faced with following issue while uploading app to AppStoreConnect.
Here is the error message that I have:

ipatool failed with an exception: #<CmdSpec::NonZeroExitException: $ /Applications/Xcode.app/Contents/Developer/usr/bin/python3 /Applications/Xcode.app/Contents/Developer/usr/bin/bitcode-build-tool -v -t /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin --sdk /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.4.sdk -o /var/folders/3k/8mdf_w554n5dnlzbxs4p2ztw0000gp/T/ipatool20210611-22236-3z7s8t/thinned-out/arm64/Payload/Application.app/Frameworks/NGMGamesFramework.framework/NGMGamesFramework --generate-dsym /var/folders/3k/8mdf_w554n5dnlzbxs4p2ztw0000gp/T/ipatool20210611-22236-3z7s8t/thinned-out/arm64/Payload/Application.app/Frameworks/NGMGamesFramework.framework/NGMGamesFramework.dSYM --strip-swift-symbols /var/folders/3k/8mdf_w554n5dnlzbxs4p2ztw0000gp/T/ipatool20210611-22236-3z7s8t/thinned-in/arm64/Payload/Application.app/Frameworks/NGMGamesFramework.framework/NGMGamesFramework
Status: pid 41773 exit 1
Stdout:
    SDK path: /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/

Do you know how it can be fixed? Using RoboVM version 2.3.13 with enabled bitcode.

[legion151]

Hi @novacom34,

unfortunately not. Only switching off bitcode works for me. @dkimitsa assumed this to be an issue with missing libraries on other machines than his.

I don't really know the state of this problem. The hope was that v2.3.13 would fix it.
I will test it again, since i didn't use the new release yet.

[dkimitsa]

@legion151 there is possible workaround for case we discussed, will check it later this week

[legion151]

sounds great.

[novacom34]

@dkimitsa Any updates?

[legion151]

Just tested with current master 887151b and still got the ipatool failed error as described above.

[timower]

I discovered that this issue is caused by the single object prelink you're doing.
Commenting out the contents of

robovm/compiler/vm/MergeStaticLibObjectFiles.cmake

Line 23 in c1a5ba9

function(merge_static_lib_object_files lib)

fixes the issue for me (at least the IPA can be recompiled with ipatool).

I'm not sure how to resolve this though, is the single object prelink something you want to keep?
If so we'll have to fix the implementation in order to preserve the bitcode sections.

[dkimitsa]

@timower prelink causes issue as mentioned few comments above.
at some point with another XCode update issue was fixed -- and I stopped looking into it and bitcode generation was working ok for me.
What XCode are you using when you see the issue ?

Anyway bitcode is not required anymore since XCode 14.
More over, bitcode support was declarative, e.g. there was no bitcode for java generated code

[timower]

I'm using Xcode 13.4

[dkimitsa]

pardon, issue is here, also there is a discussion somewhere at llvm without any solution to change behaviour.

meanwhile do you still need bitcode stuff considering recent apple changes ?