Fix packages for external platforms being introduced in lockfile when Bundler retries resolution #6285
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously they were properly ignored during an initial resolve, but if we had to retry resolution under different conditions (for example, retrying considering prereleases), they were reintroduced.
deivid-rodriguez
force-pushed
the
fix-prerelease-regression
branch
from
8b1a5e6
to
abb8c49
Compare
|
I improved the initial approach. Instead of removing the requirement from the resolver when it's for a different platform, we avoid passing it to the resolver at all in the first place. |
|
@deivid-rodriguez Out of curiosity, Is there a way we can bump rubygems in the Rails CI to verify this fixes our builds? |
|
@zzak add a step at the beginning that runs |
yahonda
added a commit
to yahonda/buildkite-config
that referenced
this pull request
Jan 20, 2023
This commit locks RubyGems and Bundler version to 3.4.3 and 2.4.3 until the newer verions of them including rubygems/rubygems#6285 I do not know how to run `pipeline-generate` then validated using docker-compose command with RUBYGEMS and BUNDLER args given as follows. it builds and `RubyGems 3.4.3 installed` and `Bundler version 2.4.3` - Command to validate ``` docker-compose build --build-arg RUBY_IMAGE=ruby:3.2 --build-arg RUBYGEMS="3.4.3" --build-arg BUNDLER="< 2.4.4" ``` - Result - `RubyGems 3.4.3 installed` and `Bundler version 2.4.3` ```ruby $ docker-compose build --build-arg RUBY_IMAGE=ruby:3.2 --build-arg RUBYGEMS="3.4.3" --build-arg BUNDLER="< 2.4.4" memcached uses an image, skipping redis uses an image, skipping default uses an image, skipping mysql uses an image, skipping mysqldb uses an image, skipping postgres uses an image, skipping railties uses an image, skipping postgresdb uses an image, skipping rabbitmq uses an image, skipping activejob uses an image, skipping chrome uses an image, skipping actionview uses an image, skipping Building base Sending build context to Docker daemon 725.8MB Step 1/22 : ARG RUBY_IMAGE Step 2/22 : FROM ${RUBY_IMAGE:-ruby:latest} ---> 1e6b2a26f38d Step 3/22 : ARG BUNDLER ---> Using cache ---> 07bc4f9cde4d Step 4/22 : ARG RUBYGEMS ---> Using cache ---> c0cf1cff98f2 Step 5/22 : RUN echo "--- :ruby: Updating RubyGems and Bundler" && (gem update --system ${RUBYGEMS:-} || gem update --system 3.3.3) && (gem install bundler -v "${BUNDLER:->= 0}" || gem install bundler -v "< 2") && ruby --version && gem --version && bundle --version && echo "--- 📦 Installing system deps" && codename="$(. /etc/os-release; x="${VERSION_CODENAME-${VERSION#*(}}"; echo "${x%%[ )]*}")" && if [ "$codename" = jessie ]; then sed -i -e '/jessie-updates/d' /etc/apt/sources.list && echo 'deb http://archive.debian.org/debian jessie-backports main' > /etc/apt/sources.list.d/backports.list && echo 'Acquire::Check-Valid-Until "false";' > /etc/apt/apt.conf.d/backports-is-unsupported; fi && if ! which gpg || ! which curl; then apt-get update && apt-get install -y --no-install-recommends gnupg curl; fi && curl -sS https://www.postgresql.org/media/keys/ACCC4CF8.asc | APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 apt-key add - && echo "deb http://apt.postgresql.org/pub/repos/apt/ ${codename}-pgdg main" > /etc/apt/sources.list.d/pgdg.list && curl -sS https://deb.nodesource.com/gpgkey/nodesource.gpg.key | APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 apt-key add - && echo "deb http://deb.nodesource.com/node_16.x ${codename} main" > /etc/apt/sources.list.d/nodesource.list && curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 apt-key add - && echo "deb http://dl.yarnpkg.com/debian/ stable main" > /etc/apt/sources.list.d/yarn.list && apt-get update && apt-get install -y --no-install-recommends autoconf automake bzip2 dpkg-dev file g++ gcc imagemagick libbz2-dev libc6-dev libcurl4-openssl-dev libdb-dev libevent-dev libffi-dev libgdbm-dev libgeoip-dev libglib2.0-dev libjpeg-dev libkrb5-dev liblzma-dev libmagickcore-dev libmagickwand-dev libncurses5-dev libncursesw5-dev libpng-dev libpq-dev libreadline-dev libsqlite3-dev libssl-dev libtool libvips-dev libwebp-dev libxml2-dev libxslt-dev libyaml-dev make patch unzip xz-utils zlib1g-dev $( if apt-cache show 'default-libmysqlclient-dev' 2>/dev/null | grep -q '^Version:'; then echo 'default-libmysqlclient-dev'; else echo 'libmysqlclient-dev'; fi ) && apt-get install -y --no-install-recommends postgresql-client default-mysql-client sqlite3 git nodejs yarn lsof ffmpeg mupdf mupdf-tools poppler-utils && curl -fLsS -o /tmp/await-linux-amd64 https://github.com/betalo-sweden/await/releases/download/v0.4.0/await-linux-amd64 && install /tmp/await-linux-amd64 /usr/local/bin/await && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* && mkdir /rails ---> Running in 8a87ed0e7f8d --- :ruby: Updating RubyGems and Bundler Updating rubygems-update Successfully installed rubygems-update-3.4.3 Installing RubyGems 3.4.3 Successfully built RubyGem Name: bundler Version: 2.4.3 File: bundler-2.4.3.gem Bundler 2.4.3 installed RubyGems 3.4.3 installed Regenerating binstubs Regenerating plugins * Installs bundler 2.4.3 as a default gem. * Fix several typos. Pull request #6224 by jdufresne * Add global flag (`-C`) to change execution directory. Pull request #6180 by gustavothecoder * Installs bundler 2.4.2 as a default gem. ------------------------------------------------------------------------------ RubyGems installed the following executables: /usr/local/bin/gem /usr/local/bin/bundle /usr/local/bin/bundler RubyGems system software updated Successfully installed bundler-2.4.3 1 gem installed ruby 3.2.0 (2022-12-25 revision a528908271) [x86_64-linux] 3.4.3 Bundler version 2.4.3 ```
yahonda
added a commit
to yahonda/buildkite-config
that referenced
this pull request
Jan 20, 2023
This commit locks RubyGems and Bundler version to 3.4.3 and 2.4.3 until the newer verions of them including rubygems/rubygems#6285 I do not know how to run `pipeline-generate` then validated using docker-compose command with RUBYGEMS and BUNDLER args given as follows. it builds and `RubyGems 3.4.3 installed` and `Bundler version 2.4.3` - Command to validate ``` docker-compose build --build-arg RUBY_IMAGE=ruby:3.2 --build-arg RUBYGEMS="3.4.3" --build-arg BUNDLER="< 2.4.4" ``` - Result - `RubyGems 3.4.3 installed` and `Bundler version 2.4.3` ```ruby $ docker-compose build --build-arg RUBY_IMAGE=ruby:3.2 --build-arg RUBYGEMS="3.4.3" --build-arg BUNDLER="< 2.4.4" ... snip ... Building base Sending build context to Docker daemon 725.8MB Step 1/22 : ARG RUBY_IMAGE Step 2/22 : FROM ${RUBY_IMAGE:-ruby:latest} ---> 1e6b2a26f38d Step 3/22 : ARG BUNDLER ---> Using cache ---> 07bc4f9cde4d Step 4/22 : ARG RUBYGEMS ---> Using cache ---> c0cf1cff98f2 Step 5/22 : RUN echo "--- :ruby: Updating RubyGems and Bundler" && (gem update --system ${RUBYGEMS:-} || gem update --system 3.3.3) && (gem install bundler -v "${BUNDLER:->= 0}" || gem install bundler -v "< 2") && ruby --version && gem --version && bundle --version && echo "--- 📦 Installing system deps" && codename="$(. /etc/os-release; x="${VERSION_CODENAME-${VERSION#*(}}"; echo "${x%%[ )]*}")" && if [ "$codename" = jessie ]; then sed -i -e '/jessie-updates/d' /etc/apt/sources.list && echo 'deb http://archive.debian.org/debian jessie-backports main' > /etc/apt/sources.list.d/backports.list && echo 'Acquire::Check-Valid-Until "false";' > /etc/apt/apt.conf.d/backports-is-unsupported; fi && if ! which gpg || ! which curl; then apt-get update && apt-get install -y --no-install-recommends gnupg curl; fi && curl -sS https://www.postgresql.org/media/keys/ACCC4CF8.asc | APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 apt-key add - && echo "deb http://apt.postgresql.org/pub/repos/apt/ ${codename}-pgdg main" > /etc/apt/sources.list.d/pgdg.list && curl -sS https://deb.nodesource.com/gpgkey/nodesource.gpg.key | APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 apt-key add - && echo "deb http://deb.nodesource.com/node_16.x ${codename} main" > /etc/apt/sources.list.d/nodesource.list && curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 apt-key add - && echo "deb http://dl.yarnpkg.com/debian/ stable main" > /etc/apt/sources.list.d/yarn.list && apt-get update && apt-get install -y --no-install-recommends autoconf automake bzip2 dpkg-dev file g++ gcc imagemagick libbz2-dev libc6-dev libcurl4-openssl-dev libdb-dev libevent-dev libffi-dev libgdbm-dev libgeoip-dev libglib2.0-dev libjpeg-dev libkrb5-dev liblzma-dev libmagickcore-dev libmagickwand-dev libncurses5-dev libncursesw5-dev libpng-dev libpq-dev libreadline-dev libsqlite3-dev libssl-dev libtool libvips-dev libwebp-dev libxml2-dev libxslt-dev libyaml-dev make patch unzip xz-utils zlib1g-dev $( if apt-cache show 'default-libmysqlclient-dev' 2>/dev/null | grep -q '^Version:'; then echo 'default-libmysqlclient-dev'; else echo 'libmysqlclient-dev'; fi ) && apt-get install -y --no-install-recommends postgresql-client default-mysql-client sqlite3 git nodejs yarn lsof ffmpeg mupdf mupdf-tools poppler-utils && curl -fLsS -o /tmp/await-linux-amd64 https://github.com/betalo-sweden/await/releases/download/v0.4.0/await-linux-amd64 && install /tmp/await-linux-amd64 /usr/local/bin/await && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* && mkdir /rails ---> Running in 8a87ed0e7f8d --- :ruby: Updating RubyGems and Bundler Updating rubygems-update Successfully installed rubygems-update-3.4.3 Installing RubyGems 3.4.3 Successfully built RubyGem Name: bundler Version: 2.4.3 File: bundler-2.4.3.gem Bundler 2.4.3 installed RubyGems 3.4.3 installed ... snip ... RubyGems system software updated Successfully installed bundler-2.4.3 1 gem installed ruby 3.2.0 (2022-12-25 revision a528908271) [x86_64-linux] 3.4.3 Bundler version 2.4.3 ```
|
I see that you already pinned Bundler & RubyGems 👍. Trying this PR is not as easy since the PR is not released nor even merged, but trying it should be a matter of cloning this repo, checking out this branch, and running |
deivid-rodriguez
added a commit
that referenced
this pull request
Jan 21, 2023
Fix packages for external platforms being introduced in lockfile when Bundler retries resolution (cherry picked from commit 8972779)
atosbucket
added a commit
to atosbucket/rails-buildkit
that referenced
this pull request
Mar 22, 2024
This commit locks RubyGems and Bundler version to 3.4.3 and 2.4.3 until the newer verions of them including rubygems/rubygems#6285 I do not know how to run `pipeline-generate` then validated using docker-compose command with RUBYGEMS and BUNDLER args given as follows. it builds and `RubyGems 3.4.3 installed` and `Bundler version 2.4.3` - Command to validate ``` docker-compose build --build-arg RUBY_IMAGE=ruby:3.2 --build-arg RUBYGEMS="3.4.3" --build-arg BUNDLER="< 2.4.4" ``` - Result - `RubyGems 3.4.3 installed` and `Bundler version 2.4.3` ```ruby $ docker-compose build --build-arg RUBY_IMAGE=ruby:3.2 --build-arg RUBYGEMS="3.4.3" --build-arg BUNDLER="< 2.4.4" ... snip ... Building base Sending build context to Docker daemon 725.8MB Step 1/22 : ARG RUBY_IMAGE Step 2/22 : FROM ${RUBY_IMAGE:-ruby:latest} ---> 1e6b2a26f38d Step 3/22 : ARG BUNDLER ---> Using cache ---> 07bc4f9cde4d Step 4/22 : ARG RUBYGEMS ---> Using cache ---> c0cf1cff98f2 Step 5/22 : RUN echo "--- :ruby: Updating RubyGems and Bundler" && (gem update --system ${RUBYGEMS:-} || gem update --system 3.3.3) && (gem install bundler -v "${BUNDLER:->= 0}" || gem install bundler -v "< 2") && ruby --version && gem --version && bundle --version && echo "--- 📦 Installing system deps" && codename="$(. /etc/os-release; x="${VERSION_CODENAME-${VERSION#*(}}"; echo "${x%%[ )]*}")" && if [ "$codename" = jessie ]; then sed -i -e '/jessie-updates/d' /etc/apt/sources.list && echo 'deb http://archive.debian.org/debian jessie-backports main' > /etc/apt/sources.list.d/backports.list && echo 'Acquire::Check-Valid-Until "false";' > /etc/apt/apt.conf.d/backports-is-unsupported; fi && if ! which gpg || ! which curl; then apt-get update && apt-get install -y --no-install-recommends gnupg curl; fi && curl -sS https://www.postgresql.org/media/keys/ACCC4CF8.asc | APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 apt-key add - && echo "deb http://apt.postgresql.org/pub/repos/apt/ ${codename}-pgdg main" > /etc/apt/sources.list.d/pgdg.list && curl -sS https://deb.nodesource.com/gpgkey/nodesource.gpg.key | APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 apt-key add - && echo "deb http://deb.nodesource.com/node_16.x ${codename} main" > /etc/apt/sources.list.d/nodesource.list && curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 apt-key add - && echo "deb http://dl.yarnpkg.com/debian/ stable main" > /etc/apt/sources.list.d/yarn.list && apt-get update && apt-get install -y --no-install-recommends autoconf automake bzip2 dpkg-dev file g++ gcc imagemagick libbz2-dev libc6-dev libcurl4-openssl-dev libdb-dev libevent-dev libffi-dev libgdbm-dev libgeoip-dev libglib2.0-dev libjpeg-dev libkrb5-dev liblzma-dev libmagickcore-dev libmagickwand-dev libncurses5-dev libncursesw5-dev libpng-dev libpq-dev libreadline-dev libsqlite3-dev libssl-dev libtool libvips-dev libwebp-dev libxml2-dev libxslt-dev libyaml-dev make patch unzip xz-utils zlib1g-dev $( if apt-cache show 'default-libmysqlclient-dev' 2>/dev/null | grep -q '^Version:'; then echo 'default-libmysqlclient-dev'; else echo 'libmysqlclient-dev'; fi ) && apt-get install -y --no-install-recommends postgresql-client default-mysql-client sqlite3 git nodejs yarn lsof ffmpeg mupdf mupdf-tools poppler-utils && curl -fLsS -o /tmp/await-linux-amd64 https://github.com/betalo-sweden/await/releases/download/v0.4.0/await-linux-amd64 && install /tmp/await-linux-amd64 /usr/local/bin/await && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* && mkdir /rails ---> Running in 8a87ed0e7f8d --- :ruby: Updating RubyGems and Bundler Updating rubygems-update Successfully installed rubygems-update-3.4.3 Installing RubyGems 3.4.3 Successfully built RubyGem Name: bundler Version: 2.4.3 File: bundler-2.4.3.gem Bundler 2.4.3 installed RubyGems 3.4.3 installed ... snip ... RubyGems system software updated Successfully installed bundler-2.4.3 1 gem installed ruby 3.2.0 (2022-12-25 revision a528908271) [x86_64-linux] 3.4.3 Bundler version 2.4.3 ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What was the end-user or developer problem that led to this PR?
After we started ignoring prereleases and only considering them if we can't resolve without them at #6246, we got reports that Bundler was sometimes introducing gems for external platforms in the lockfile. For example, when using
gem "tzinfo-data", platform: :windowson non Windows platforms.What is your fix for the problem, implemented in this PR?
The problem was that while on initial resolution we were properly ignoring these dependencies, they were not being properly ignored when Bundler had to resort to re-resolving including prereleases (and that's the case of Rails main branch).
The fix is to properly remove the dependency so that it's not considered on retries either.
Another improvement we can do here, but I'll leave that for another PR, is to not exclude prereleases from gems configured to use a gemspec/path source. But this bug fix is independent from that enhancement.
Make sure the following tasks are checked