Make lockfiles generated on macOS include a lock for Linux by default

[deivid-rodriguez]

What was the end-user or developer problem that led to this PR?

Locking only the specific platform in lockfiles causes issues in CI platforms or Heroku, that run on Linux. This issue only appeared after we started locking the specific platform in the lockfile. Previously, although platform specific resolution would not always be correct, the general case would just work because platform specific variants where chosen at install time (they were not recorded in the lockfile at all).

What is your fix for the problem, implemented in this PR?

My fix is to, after resolving for the current platform like we do now, check whether the resolution found is also compatible with Linux (the 99% case), and if so, add the linux platform and linux specific variants to the lockfile.

This way, new lockfiles generated on Darwin should just work on Linux.

For example, for the following Gemfile,

# frozen_string_literal: true

source "https://rubygems.org"

# gem "rails"

gem "nokogiri", "~> 1.13"

running bundle install on my MacOS laptop results in

$ bundle install
Fetching gem metadata from https://rubygems.org/.......
Resolving dependencies...
Using racc 1.6.0
Using bundler 2.4.0.dev
Using nokogiri 1.13.6 (arm64-darwin)
Bundle complete! 1 Gemfile dependency, 3 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

generating the following Gemfile.lock file, that should just work on Linux

GEM
  remote: https://rubygems.org/
  specs:
    nokogiri (1.13.6-arm64-darwin)
      racc (~> 1.4)
    nokogiri (1.13.6-x86_64-linux)
      racc (~> 1.4)
    racc (1.6.0)

PLATFORMS
  arm64-darwin-21
  x86_64-linux

DEPENDENCIES
  nokogiri (~> 1.13)

BUNDLED WITH
   2.4.0.dev

Closes #4269.
Closes #5338.

Make sure the following tasks are checked

• Describe the problem / feature
• Write tests for features and bug fixes
• Write code to solve the problem
• Make sure you follow the current code style and write meaningful commit messages without tags

[indirect]

This is very much a new idea and I might be wrong, but should we print a message when we do this? “Automatically adding linux-x86 to the lockfile for future deployments.” or something? Also, what if I deploy to linux-arm machines and I don’t want to automatically lock to linux-x86? Can we provide an option to disable this, or configure the platform that is always automatically added?

[deivid-rodriguez]

Yeah, I figured the desire to disable this could come up but then I thought that this doesn't really affect end users in any way, every thing should work just as it worked before, so I'm not sure anybody will want to disable this or configure it in any way. The only reason would be "lockfile aesthetics"?

Anyways, I'm not opposed to adding a configuration to toggle this off. I wouldn't like to allow configuring the specific extra platform because Linux is the 99% case and because there's a bit of confusion regarding platforms in the codebase and I'm really not sure how this will behave for other platforms (in particular Windows and Java).

I would like to figure things out before committing to adding things, but this seems like a handy, mostly transparent, improvement. Basically, it's just running bundle lock --add-platform x86_64-linux for our users so things just work for them.

[deivid-rodriguez]

Regarding the message, I think it goes together with configuring this. If we add a configuration option, then logging something definitely makes sense. But if not, I wouldn't log anything because then maybe someone will want to disable it just because they saw the message 😆.

Also, something that makes me hesitate allowing this to be configured is that my initial plan was to "just add all platforms where the resolution is valid". So allowing this to be configured feels like an opposite direction.

Finally, if we configure this, I think there's two options:

• A standard configuration option.
• A Gemfile DSL for the platforms to be locked.

[deivid-rodriguez]

Well, after a bit more thought, users are probably going to request this anyways, and making this a configuration that defaults to linux seems simple enough. Plus logging a message like the one you mentioned makes things also clear for users. Plus @schneems already suggested exactly this solution at #4269 (comment).

So I think let's implement both of your suggestions @indirect.

[deivid-rodriguez]

Sorry for too many message but also I just noticed that I did a bit of diagonal reading of your first message and missed this part 😳

what if I deploy to linux-arm machines

So definitely this is not just lockfile aesthetics and makes sense to allow configuring it 👍. I will work on it!

[fauno]

> what if I deploy to linux-arm machines So definitely this is not just lockfile aesthetics and makes sense to allow configuring it 👍. I will work on it!

also `x86_64-linux-musl`!

[indirect]

@deivid-rodriguez If you think it makes sense, that sounds good to me. :)

[pond]

As others have pointed out, there's a question of which Linux.

I don't understand why ruby isn't just added as a fallback, since this has historically worked "everywhere" for Heroku, CI systems etc., but would still allow for the local developer's lock to be added to.

I still also don't see why the local developer machine's architecture should have any bearing on lock file whatsoever since it says nothing about anywhere else that the gems might be installed and the whole point of a Gemfile is to be a platform-independent way to specify required Ruby gem versions, but that's another debate.

[indirect]

I have no idea where you got the idea that the entire point of a Gemfile is to be platform independent, but for the record: it is not.

Gemfiles are usually able to be applied to multiple platforms, but sometimes they can’t be. The generic ruby platform is often workable, but sometimes isn’t. The lockfile in particular cannot provide Bundler’s basic guarantee of “the same code” as soon as two platforms are involved—Heroku has never actually “just worked” this entire time, it has just been true that the most common case didn’t run into any of the cross platform issues.

[indirect]

(Bundler creating an additional lock resolution for each platform is the only solution we have for this set of problems, and it is why we now have multiple platforms in the lockfile.)

[deivid-rodriguez]

The time RubyGems allowed versions of gems precompiled for specific architectures, the lockfile stopped being platform independent (and also stopped working at all in some cases). I know you @pond did not run into these issues, but that doesn't mean everything worked perfectly.

Anyways, adding back the "ruby" platform as a fallback for this case is an option. We would have to warn (and recommend adding the specific platform instead) when this "generic" platform gets actually used for materializing the lockfile into actual gems. This is because by doing this we would lose the guarantee mentioned by @indirect that in frozen mode the same lockfile always deploys the exact same set of gems. But at least things would "work" again by default.

These two solutions are not mutually exclusive, but the current one is on the one hand, simpler to implement, and on the other hand, it already covers the most common cases. So I'd go with this one for now and see how it goes, and whether we should also add the "fallback platform" to cover more cases by default.

[pond]

I have no idea where you got the idea that the entire point of a Gemfile is to be platform independent, but for the record: it is not.

Well, OK; that's surprising. I had this idea that Ruby and its gems were meant to run on more than one operating system and architecture. The only mechanism in this ecosystem that lets us specify exact versions of gems and their dependencies is Bundler. If Gemfile.lock is not platform-independent, the entire bundled Ruby language and its gems will not be platform independent either. Of course there will be limits - up to whatever is reasonably possible within Bundler developer time and technology constraints. An example would be native components building against locally included or system libraries, for example, which gem authors have been free to decide on their own and which therefore Bundler has no control over; or even a Gem that's specifically bound to (say) a Windows-only x86 library and is itself, by virtue of native extensions, explicitly not cross-platform. But those are the edge cases, not the major use cases.

Gemfiles are usually able to be applied to multiple platforms, but sometimes they can’t be. The generic ruby platform is often workable, but sometimes isn’t.

So as a suggestion or discussion point - can we have Bundler write the ruby platform for the majority use case where it's workable and the Gemfile.lock is cross-platform? It would only lock to a developer's local machine - though I still struggle to see how that is ever meaningful - if one of the edge cases you mention arise? Could it not always write the ruby entry under platforms in addition to the local machine maybe?

(EDIT: @deivid-rodriguez's follow up hadn't shown when I reloaded the page for this issue in my browser, but after submitting this answer it showed up - GitHub's page cacheing can be weird. I see that it's being considered as a "maybe" - cool!).

After all, what's the lockfile for?

• If I specify the precise semver-orientated version constraints that I require in my Gemfile, then there would never be a reason to commit a Gemfile.lock to any source control system, nor would I need to send it to any deployed system; the Gemfile would be enough and lockfiles can go away completely.
• If the Gemfile version constraints are insufficient for a developer to be sure that their tested software would be deployed and running under the same versions of associated libraries (gems), then a Gemfile.lock must be sent so that that the developer has the same gem versions as the deployed software in which case the developer's local machine architecture and operating system is irrelevant to that developer's intent unless it just happens to be precisely identical to the deployed system.

...though of course:

• We can't ever be sure of precisely the same behaviour even if only because of operating system differences, so usually the best you can do is have a CI system that matches and run tests there; and...
• Likewise native compilation of gems can introduce other variances that might include bugs due to endianness, system vs local bundled libraries and more, so...
• The lockfile in such cases is invaluable because it makes sure that CI and deployment use precisely the same gem versions; but then...
• The platform architecture that would need to be specified therein is once again clearly not the local developer's machine architecture and Bundler writing such by default is unhelpful.

[pond]

(Also as a note, the engagement on this discussion by the community & maintainers is excellent & helps everyone understand what's being done, and why, a lot more; it'll help in future too as this thread can be pointed to; so, thanks for that!)

[deivid-rodriguez]

If I specify the precise semver-orientated version constraints that I require in my Gemfile, then there would never be a reason to commit a Gemfile.lock to any source control system, nor would I need to send it to any deployed system; the Gemfile would be enough and lockfiles can go away completely.

Yes, if you specified exact requirements for ALL of your dependencies (including all transitive dependencies), and if the Gemfile DSL was fully complete and allowed you to specific the platforms your application should work on. I guess you can imagine why nobody would do this (even if we made it possible by expanding the Gemfile DSL).

If the Gemfile version constraints are insufficient for a developer to be sure that their tested software would be deployed and running under the same versions of associated libraries (gems), then a Gemfile.lock must be sent so that that the developer has the same gem versions as the deployed software in which case the developer's local machine architecture and operating system is irrelevant to that developer's intent unless it just happens to be precisely identical to the deployed system.

Well, not only that. The Gemfile.lock, specially in "frozen" mode which makes this intent very clear, is also there to ensure that two different deployments of the same lockfile result in exactly the same code being deployed. And by locking only the generic "ruby" platform, this is not possible. The way Bundler worked before, one deployment could result in the nokogiri-1.13.1 gem being installed, and a subsequent deployment installing the nokogiri-1.13.1-x86_64-linux gem instead, or viceversa.

The above is in addition to the resolution not guaranteed to be correct if specific platforms are not recorded in the Gemfile.lock file, because the same version of a gem, but for a different platform, may bring requirements incompatible with the rest of gems recorded in the Gemfile.lock file. I guess this last issue could be improved by adding additional version resolution work at install time, but that defeats another point of having a lockfile: avoid having to resolve gem dependencies every time.

Anyways, with this feature you should be able to specify not only x86_64-linux, but also ruby as an additional platform by default if you really want to, so that should fix things for you without needing anything else. But still the extra improvement I suggested above of also adding the "ruby" platform by default (but deprecating it by warning when it gets actually used) might not be a bad idea. Food for future discussion, I'll now focus on getting this one ready :)

[indirect]

Updated name to make it slightly clearer that we will resolve and lock for linux, but we have no idea if your bundle will work on linux. 😅

[deivid-rodriguez]

Thanks @indirect :)

[deivid-rodriguez]

This is ready for a review now!

[deivid-rodriguez]

Just to clarify what I ended up implementing here. I did not make this specific to Linux/macOS, but to any "ruby platforms" (not Windows, not Java). So, for fresh lockfiles, we now resolve for the running platform, and if it's a "ruby platform", we'll also lock any other "ruby platforms" that are compatible with the resolution that we found.

[ansonhoyt]

Thank you! This solves one of our regular first-deploy issues! ❤️