Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run Cygwin CI workflow commands in login shells #1709

Merged
merged 1 commit into from Oct 14, 2023

Conversation

EliahKagan
Copy link
Contributor

@EliahKagan EliahKagan commented Oct 14, 2023

This passes --login to the bash shell used to run commands in the Cygwin environment on CI. This eliminates the need to work around a partly broken environment, and the extra code what was used to do that is accordingly removed. There are two benefits of this change:

  • The PATH is correct: Cygwin's /usr/local/bin and /usr/bin are present at the beginning of PATH. Otherwise, it is easy to get /usr/bin at the front, but rather involved to get /usr/local/bin to precede it. Because Python on Cygwin puts scripts/executables such as the upgraded pip and the pytest command in /usr/local/bin, it is valuable to have that directory in the PATH and best to have it before /usr/bin. (I have set CYGWIN_NOWINPATH to omit other directories, since finding any of the commands to be run in the Cygwin environment outside that environment is unintended.)

  • Every step automatically has correct temporary directories: When Cygwin commands were not being run in login shells, they didn't automatically get correct values for TMP and TEMP for their environment. To work around this, those environment variables were set globally, for every step. But that caused them to refer to nonexistent locations for steps such as actions/checkout. Most likely this would not cause any errors, but it did cause copious warnings about a nonexistent temporary directory, which risked obscuring other potentially important output. Now that Cygwin commands run in login shells, both the few non-Cygwin steps, and the steps run in the Cygwin environment, all get correct temporary directories (with TMP and TEMP set in the prewritten startup script the login shell uses).

A theoretical disadvantage of this is that login shells take slightly longer to start up, but that delay is insignificant in this application. A more significant disadvantage is that setting the -x shell option the way it was done before would produce a lot of noise at the beginning of the output for every command-running step. To work around that, -x is omitted from the value of shell and set -x is added at the end of the startup script for login shells, so it runs before each step's "payload" command, but without applying to the commands run in the startup script itself.

This passes --login to the bash shell used to run commands in the
Cygwin environment on CI. This eliminates the need to work around a
partly broken environment, and the extra code what was used to do
that is accordingly removed. There are two benefits of this change:

- The PATH is correct: Cygwin's /usr/local/bin and /usr/bin are
  present at the beginning of PATH. Otherwise, it is easy to get
  /usr/bin at the front, but rather involved to get /usr/local/bin
  to precede it. Because Python on Cygwin puts scripts/executables
  such as the upgraded "pip" and the "pytest" command in
  /usr/local/bin, it is valuable to have that directory in the PATH
  and best to have it before /usr/bin. (I have set CYGWIN_NOWINPATH
  to omit other directories, since finding any of the commands to
  be run in the Cygwin environment outside that environment is
  unintended.)

- Every step automatically has correct temporary directories: When
  Cygwin commands were not being run in login shells, they didn't
  automatically get correct values for TMP and TEMP for their
  environment. To work around this, those environment variables
  were set globally, for every step. But that caused them to refer
  to nonexistent locations for steps such as actions/checkout. Most
  likely this would not cause any errors, but it did cause copious
  warnings about a nonexistent temporary directory, which risked
  obscuring other potentially important output. Now that Cygwin
  commands run in login shells, both the few non-Cygwin steps, and
  the steps run in the Cygwin enviroment, all get correct temporary
  directories (with TMP and TEMP set in the prewritten startup
  script the login shell uses).

A theoretical disadvantage of this is that login shells take
slightly longer to start up, but that delay is insigificant in
this application. A more significant disadvantage is that setting
the -x shell option the way it was done before would produce a lot
of noise at the beginning of the output for every command-running
step. To work around that, -x is omitted from the value of "shell"
and "set -x" is added at the end of the startup script for login
shells, so it runs before each step's "payload" command, but
without applying to the commands run in the startup script itself.
@EliahKagan EliahKagan marked this pull request as ready for review October 14, 2023 07:10
Copy link
Member

@Byron Byron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's so awesome! Great discovery!

I can sense that native Windows support is just around the corner :).

@Byron Byron merged commit 97c59ae into gitpython-developers:main Oct 14, 2023
8 checks passed
@EliahKagan EliahKagan deleted the cygwin-ci-path branch October 14, 2023 11:01
renovate bot added a commit to allenporter/flux-local that referenced this pull request Oct 20, 2023
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [GitPython](https://togithub.com/gitpython-developers/GitPython) |
`==3.1.37` -> `==3.1.40` |
[![age](https://developer.mend.io/api/mc/badges/age/pypi/GitPython/3.1.40?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/GitPython/3.1.40?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/GitPython/3.1.37/3.1.40?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/GitPython/3.1.37/3.1.40?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|

---

### Release Notes

<details>
<summary>gitpython-developers/GitPython (GitPython)</summary>

###
[`v3.1.40`](https://togithub.com/gitpython-developers/GitPython/compare/3.1.38...3.1.40)

[Compare
Source](https://togithub.com/gitpython-developers/GitPython/compare/3.1.38...3.1.40)

###
[`v3.1.38`](https://togithub.com/gitpython-developers/GitPython/releases/tag/3.1.38)

[Compare
Source](https://togithub.com/gitpython-developers/GitPython/compare/3.1.37...3.1.38)

#### What's Changed

- Add missing assert keywords by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1678
- Make clear every test's status in every CI run by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1679
- Fix new link to license in readme by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1680
- Drop unneeded flake8 suppressions by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1681
- Update instructions and test helpers for git-daemon by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1684
- Fix Git.execute shell use and reporting bugs by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1687
- No longer allow CI to select a prerelease for 3.12 by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1689
- Clarify Git.execute and Popen arguments by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1688
- Ask git where its daemon is and use that by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1697
- Fix bugs affecting exception wrapping in rmtree callback by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1700
- Fix dynamically-set **all** variable by
[@&#8203;DeflateAwning](https://togithub.com/DeflateAwning) in
[gitpython-developers/GitPython#1659
- Fix small
[#&#8203;1662](https://togithub.com/gitpython-developers/GitPython/issues/1662)
regression due to
[#&#8203;1659](https://togithub.com/gitpython-developers/GitPython/issues/1659)
by [@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1701
- Drop obsolete info on yanking from security policy by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1703
- Have Dependabot offer submodule updates by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1702
- Bump git/ext/gitdb from `49c3178` to `8ec2390` by
[@&#8203;dependabot](https://togithub.com/dependabot) in
[gitpython-developers/GitPython#1704
- Bump git/ext/gitdb from `8ec2390` to `6a22706` by
[@&#8203;dependabot](https://togithub.com/dependabot) in
[gitpython-developers/GitPython#1705
- Update readme for milestone-less releasing by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1707
- Run Cygwin CI workflow commands in login shells by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1709

#### New Contributors

- [@&#8203;DeflateAwning](https://togithub.com/DeflateAwning) made their
first contribution in
[gitpython-developers/GitPython#1659

**Full Changelog**:
gitpython-developers/GitPython@3.1.37...3.1.38

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/allenporter/flux-local).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4xOS4yIiwidXBkYXRlZEluVmVyIjoiMzcuMTkuMiIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
EliahKagan added a commit to EliahKagan/GitPython that referenced this pull request Dec 23, 2023
This makes a pip -> pip3 symlink in /usr/bin in a new step prior to
the first step that runs the pip command. Using "pip3", "pip3.9",
or a command like "python -m pip" would work, but this allows the
Cygwin workflow to continue using the same installation commands as
the main testing workflow.

Adding this fixes two problems:

1. When the pip version installed by the python39-pip package is
   current, so that upgrading pip doesn't install any new commmand,
   no "pip" command is created in /usr/local. This has happened not
   to be the case for a long time, which is why the Cygwin workflow
   was able to pass. (That the recent failures started at the merge
   of gitpython-developers#1783 turns out to be a coincidence: rerunning jobs on prior
   commits has the failure, as does experimentally reverting it.)

2. Even when the pip version installed by python39-pip is behind
   the latest available version, pip is still used before being
   upgraded to check if setuptools is installed, to decide whether
   to upgrade it. This is to keep similar steps in the two testing
   workflows similar, since the Cygwin workflow only uses Python
   3.9, which always has setuptools. Because pip was never in $PATH
   in that step, the Cygwin workflow wrongly refrained from trying
   to upgrade setuptools.

When the "Update PyPA packages" step does find a newer version of
pip to upgrade to, it installs it in /usr/local/bin, which we have
in $PATH before /usr/bin, so the upgraded version, when present,
will still be preferred in subsequent commands, as before.

Running "pip" on Cygwin when it may not be in $PATH -- and, for one
step, never is -- was a bug introduced in e8956e5 (gitpython-developers#1709). Before
that, "pip" still was not always available, but it was not used.
This change fixes the bug by making sure "pip" is always available.
EliahKagan added a commit to EliahKagan/GitPython that referenced this pull request Dec 23, 2023
This makes a pip -> pip3 symlink in /usr/bin in a new step prior to
the first step that runs the pip command. Using "pip3", "pip3.9",
or a command like "python -m pip" would work, but this allows the
Cygwin workflow to continue using the same installation commands as
the main testing workflow.

Adding this fixes two problems:

1. When the pip version installed by the python39-pip package is
   current, so that upgrading pip doesn't install any new commmand,
   no "pip" command is created in /usr/local. This has happened not
   to be the case for a long time, which is why the Cygwin workflow
   was able to pass. (That the recent failures started at the merge
   of gitpython-developers#1783 turns out to be a coincidence: rerunning jobs on prior
   commits has the failure, as does experimentally reverting it.)

2. Even when the pip version installed by python39-pip is behind
   the latest available version, pip is still used before being
   upgraded to check if setuptools is installed, to decide whether
   to upgrade it. This is to keep similar steps in the two testing
   workflows similar, since the Cygwin workflow only uses Python
   3.9, which always has setuptools. Because pip was never in $PATH
   in that step, the Cygwin workflow wrongly refrained from trying
   to upgrade setuptools.

When the "Update PyPA packages" step does find a newer version of
pip to upgrade to, it installs it in /usr/local/bin, which we have
in $PATH before /usr/bin, so the upgraded version, when present,
will still be preferred in subsequent commands, as before.

Running "pip" on Cygwin when it may not be in $PATH -- and, for one
step, never is -- was a bug introduced in e8956e5 (gitpython-developers#1709). Before
that, "pip" still was not always available, but it was not used.
This change fixes the bug by making sure "pip" is always available.
EliahKagan added a commit to EliahKagan/GitPython that referenced this pull request Dec 23, 2023
This makes a pip -> pip3 symlink in /usr/bin in a new step prior to
the first step that runs the pip command. Using "pip3", "pip3.9",
or a command like "python -m pip" would work, but this allows the
Cygwin workflow to continue using the same installation commands as
the main testing workflow.

Adding this fixes two problems:

1. When the pip version installed by the python39-pip Cygwin
   package is current, so that upgrading pip doesn't install any
   new commmand, no "pip" command is created in /usr/local. This
   has happened not to be the case for a long time, which is why
   the Cygwin workflow was able to pass. (That the recent failures
   started at the merge of gitpython-developers#1783 turns out to be a coincidence:
   rerunning jobs on prior commits has the failure, as does
   experimentally reverting it.)

2. Even when the pip version installed by python39-pip is behind
   the latest available version, pip is still used before being
   upgraded to check if setuptools is installed, to decide whether
   to upgrade it. This is to keep similar steps in the two testing
   workflows similar, since the Cygwin workflow only uses Python
   3.9, which always has setuptools. Because pip was never in $PATH
   in that step, the Cygwin workflow wrongly refrained from trying
   to upgrade setuptools.

When the "Update PyPA packages" step does find a newer version of
pip to upgrade to, it installs it in /usr/local/bin, which we have
in $PATH before /usr/bin, so the upgraded version, when present,
will still be preferred in subsequent commands, as before.

Running "pip" on Cygwin when it may not be in $PATH -- and, for one
step, never is -- was a bug introduced in e8956e5 (gitpython-developers#1709). Before
that, "pip" still was not always available, but it was not used.
This change fixes the bug by making sure "pip" is always available.
EliahKagan added a commit to EliahKagan/GitPython that referenced this pull request Dec 23, 2023
This makes a pip -> pip3 symlink in /usr/bin in a new step prior to
the first step that runs the pip command. Using "pip3", "pip3.9",
or a command like "python -m pip" would work, but this allows the
Cygwin workflow to continue using the same installation commands as
the main testing workflow.

Adding this fixes two problems:

1. When the pip version installed by the python39-pip Cygwin
   package is current, so that upgrading pip doesn't install any
   new commmand, no "pip" command is created in /usr/local. This
   has happened not to be the case for a long time, which is why
   the Cygwin workflow was able to pass. (That the recent failures
   started at the merge of gitpython-developers#1783 turns out to be a coincidence:
   rerunning jobs on prior commits has the failure, as does
   experimentally reverting it.)

2. Even when the pip version installed by python39-pip is behind
   the latest available version, pip is still used before being
   upgraded to check if setuptools is installed, to decide whether
   to upgrade it. This is to keep similar steps in the two testing
   workflows similar, since the Cygwin workflow only uses Python
   3.9, which always has setuptools. Because pip was never in $PATH
   in that step, the Cygwin workflow wrongly refrained from trying
   to upgrade setuptools.

When the "Update PyPA packages" step does find a newer version of
pip to upgrade to, it installs it in /usr/local/bin, which we have
in $PATH before /usr/bin, so the upgraded version, when present,
will still be preferred in subsequent commands, as before.

Running "pip" on Cygwin when it may not be in $PATH -- and, for one
step, never is -- was a bug introduced in e8956e5 (gitpython-developers#1709). Before
that, "pip" still was not always available, but it was not used.
This change fixes the bug by making sure "pip" is always available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants