Display a "Binary size impact" for a package

[zntfdr]

Today a new package has been added to SPI that promises to report the binary size impact of any package into the final product binary.

I believe it would be very cool/useful if SPI itself could display this size 😃:
for example this would prevent surprises when shipping an app update to the store!

Just a thought:
I'm sure there are challenges in adding such feature, thank you for your consideration 🙏🏻

Here are a few packages that offer this functionality

• swift-package-info
• Tron

[daveverwer]

Yea this is interesting and that's a great, real-world stat that people do/should care about. It's worth checking out how accurate it turns out to be, and how best to display it.

[Sherlouk]

Worth noting that @marinofelipe's swift-package-info was updated recently to support JSON output as well as calculating sizes for a local package (doesn't duplicate the download time which it was doing before)

So we'd essentially just need to run:

$ swift-package-info --for ./relative/path/to/package --report jsonDump

// output

{
  "binarySize" : {
    "amount" : 962560,
    "formatted" : "963 KB"
  }
}

I'm not sure exactly how, but I feel like it could do with it's own "job" triggered at the same time as the iOS, macOS (etc) builds that we currently do?

It would essentially be an "analysis" phase which uploads its data slightly differently. It would power things such as the binary size impact and, in the future, things like documentation generation?

@daveverwer @finestructure I wonder if you have any thoughts on this?

[finestructure]

Yes, the integration is the slightly tricky bit. I haven't thought this through in detail but my hunch is that we could build and add the binary to the docker image and then shell out just like we do for swift package dump-package, probably in that same processing phase.

Alternatively, a cleaner approach would be to see if swift-package-info could ship a library (if it doesn't already) that we embed and use.

[Sherlouk]

It doesn't current surface itself as a library, but that's a relatively minor change that I'm sure we (I or Felipe) could add to the package upstream.

I suppose my concern would be over the time it takes. It essentially has to archive an application using the package which will take minutes - would we want to block the ingest for that long while we run these calculations?

I'm not saying yes/no - genuinely unsure if that would cause problems given some packages take 15 odd minutes to build!

[marinofelipe]

It doesn't current surface itself as a library, but that's a relatively minor change that I'm sure we (I or Felipe) could add to the package upstream.

Jumping in here... Yep, swift-package-info is done with modularity in mind, so we could without much work and some tweaks provide a library a part from the executable module Run.

I suppose my concern would be over the time it takes. It essentially has to archive an application using the package which will take minutes...

Yes, for packages that have a deep dependency graph, it takes a while. Unfortunately we are constrained to xcodebuild archive which doesn't perform quite well.
Another downside is that, even more for virtual machines, the memory consumption of it can be high, which can be tricky.
For example, running it for FirebaseCrashlytics in my machine (1,4 GHz Quad-Core Intel Core i5 / 16 GB RAM / with Xcode closed 😛 ):

/usr/bin/time -lp swift-package-info --for https://github.com/firebase/firebase-ios-sdk -v 7.6.0 --product FirebaseCrashlytics --report jsonDump

real        55.33
user        25.69
sys         11.82
           284663808  maximum resident set size

[finestructure]

Phew, that’s going to be too harsh for our runners (although it might an extreme case) 😅 It just occurred to me that we’re already building every package and perhaps we can get a decent estimate of binary size from the build products we already have. A big problem here is that they’re debug builds and I don’t think we want to switch to release builds, for performance reasons. But maybe even so it might be good enough for t shirt size estimates? Surely there must be enough correlation between debug and release build size for all but the most convoluted cases. Might be worth running some tests.

…

On 24 Mar 2021, at 22:00, Felipe Lefèvre Marino ***@***.***> wrote:  It doesn't current surface itself as a library, but that's a relatively minor change that I'm sure we (I or Felipe) could add to the package upstream. Jumping in here... Yep, swift-package-info is done with modularity in mind, so we could without much work and some tweaks provide a library a part from the executable module Run. I suppose my concern would be over the time it takes. It essentially has to archive an application using the package which will take minutes... Yes, for packages that have a deep dependency graph, it takes a while. Unfortunately we are constrained to xcodebuild archive which doesn't perform quite well. Another downside is that, even more for virtual machines, the memory consumption of it can be high, which can be tricky. For example, running it for FirebaseCrashlytics in my machine (1,4 GHz Quad-Core Intel Core i5 / 16 GB RAM / with Xcode closed 😛 ): /usr/bin/time -lp swift-package-info --for https://github.com/firebase/firebase-ios-sdk -v 7.6.0 --product FirebaseCrashlytics --report jsonDump real 55.33 user 25.69 sys 11.82 284663808 maximum resident set size — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[Sherlouk]

Firebase is definitely a bit of an edge case, it's something I've experienced in my own use cases where it spiked my CI build times (on CircleCI) by over 10 minutes once I added the dependency.

As you say precision doesn't have to be perfect, T-Shirt sizing could work if we have well defined boundaries for that.

Definitely worth exploring if there is any correlation between the build products and the archive size. I'm sure we could write a script which runs swift-package-info against a random subset of the top packages and then compares to the basic build products.

[daveverwer]

Echoing what @finestructure says here, we want to be very careful with significantly increasing the load on our build machines. We have just had an injection of fresh hardware which has allowed us to catch up with builds again and we're in a good place right now, but we should approach additional significant processing tasks cautiously!

I still like the idea, but we'll be cautious when adding features that take large amounts of processing time. Managing the service when it's struggling can be challenging.

[marinofelipe]

I still like the idea, but we'll be cautious when adding features that take large amounts of processing time. Managing the service when it's struggling can be challenging.

Definitely understand and agree with yours and @finestructure points.

I don't know what would be the minimum accuracy of binary size report you'd see as satisfactory, but surely a good analyzes of tradeoff between precision and performance is important here.

Definitely worth exploring if there is any correlation between the build products and the archive size. I'm sure we could write a script which runs swift-package-info against a random subset of the top packages and then compares to the basic build products.

Yep, I'd up for it, @Sherlouk.

Yesterday I was doing some quick tests here, and I'm not sure yet how precise it would be.
Throwing out some findings:

Package	Debug build product	Via archive
Crashlytics 7.9.1
Datadog 1.5.1
RxSwift 6.0.0
ReactiveMoya 15.0.0-alpha.1

It seems that reading the content for arm64 would have a close size for most cases, although not all frameworks generate .swiftmodule/.swiftdoc in debug build (see Crashlytics) - not sure why - would have to dig deep into it.

Another thing I don't know is if both .swiftmodule and .swiftdoc should be accounted into the size calculation. For some cases, like ReactiveMoya summing up both for it and its dependencies leads to a very close value, while for Datadog and RxSwift, that doesn't have external dependencies, it has the opposite effect as just getting the size of .swiftmodule.
Things we'd have to explore further to better understand.

The archiving strategy is a bit simpler, since it's based on the package contents size of the generated application product, which contains the framework/package content "embedded" into it.

[finestructure]

Moving this to a discussion.

[marinofelipe]

Hey @finestructure and @daveverwer 👋

Is binary size report still interesting to you?

As we talked before, just as a quick recap, I guess the two main options would be:

• Further evaluate how precise built products are and if that would give a good enough size
  • Seems the best option in terms of performance cost and feasibility given you already build packages in debug (based on @finestructure's comment)
• Consume swift-package-info as a library
  • Even if it wastes a considerable amount of CPU and memory (xcodebuild), is there any option like scheduling runs on the CI/CD pipelines to collect binary size for latest version of packages? I imagine it would be very tricky depending on the complexities of your CI/CD workflows, and considering it would have to be run for thousands(?) of packages 😅

If you are still interested I could take a look into the preferred option.
I have collected more knowledge around frameworks and libraries in past few months, so I might be able to do a detailed analyzes of debug vs release built products, including for different types of products, and how good that estimated size would be.

Let me know if you have any thoughts.

[mackoj]

Hello, I believe incorporating this metric would be beneficial. What are your thoughts on its feasibility now, @daveverwer and @finestructure? Do you think the potential gains outweigh the associated costs? Thanks.

[finestructure]

It definitely is but unless it can run on top of our debug builds in a lightweight fashion (i.e. by just summing up sizes for already existing build products and "costing" no more than say 10-20s* of wall clock time) I don't think we can make this work given our current hardware situation.

Also apologies @marinofelipe ! I completely missed your reply from March last year :(

My understanding from your analysis regarding binary sizes is that you got mixed results, right? I wonder how valuable sizes are if we can't be sure they're reasonably accurate. It seems to me only release builds would really provide a good answer here - and they are currently prohibitive.

I think we'd need a comparison for release/debug sizes across a decent sample of representative packages (big frameworks, no dep packages, binary artefact framework, various platforms) in order to decide if this is worthwhile. This could be a nice Swift mentorship program project :)

*) rather arbitrary figure but let's just say it can't be minutes - we only have max 10min per build before we time out, so 30s is already 5% of our build budget