Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Git.execute shell use and reporting bugs #1687

Merged
merged 7 commits into from Oct 2, 2023

Conversation

EliahKagan
Copy link
Contributor

@EliahKagan EliahKagan commented Oct 2, 2023

Fixes #1685
Fixes #1686

Changes to git module code

This updates the shell variable itself, only when it is None, from self.USE_SHELL.

(That attribute usually gets its value from Git.USE_SHELL rather than being separately an instance attribute. But self.USE_SHELL is an idiomatic way to access it. Furthermore, although people probably shouldn't set it on instances, people may expect that this is permitted.)

Now:

  • USE_SHELL is used as a fallback only. When shell=False is passed, USE_SHELL is no longer consulted. Thus shell=False always keeps a shell from being used, even in the non-default case where the USE_SHELL attribute has been set to True.
  • The debug message printed to the log shows the actual value that is being passed to Popen, because the updated shell variable is used both to produce that message and in the Popen call.

New tests

This also (actually, first) adds tests, some to check that the correct value is always passed as shell= when Git.execute calls Popen, and others to check that whatever it passes is the same as it claims to be passing in the log. I've made them two separate test methods rather than a test method with multiple assertions, because I think the results of these particular tests are easier to understand that way (even compared to the option of using self.subTest). But the major shared logic is extracted to a helper function, where its important subtleties are commented. I have verified that the tests fail, in the expected way, before the bugs are fixed, and then pass, and that these are both the case even on native Windows where we don't (yet) have CI jobs.

The tests cover the six combinations of valid values of the shell= argument to Git.execute (None, False, and True) and Git.USE_SHELL (False and True). I used ddt for this, so there are only two test methods, even though there are six test cases. The tests are simple in principle, with the main source of complexity being the question of how to deal with how the command argument to Popen typically differs, including by type, based on the shell argument, yet the tests must not misbehave, nor induce extraneous misbehavior, even when the code under test has a bug that would ordinarily prevent such agreement.

On Python 3.7 only, this adds mock as a development dependency (in the test extra, not as a regular dependency). mock is a backport of unittest.mock, and a couple of my assertions use a handy feature that was introduced to the mock library in Python 3.8. Besides being harder to write without it, I believe it would be less clearly expressed. (Details are commented with the import logic.)

While adding the new tests, I also renamed test_it_executes_git_to_shell_and_returns_result to test_it_executes_git_and_returns_result, because that test case does not test that a shell is used to run git (and a shell is not used in that test).

That test is not testing whether or not a shell is used (nor does
it need to test that), but just whether the library actually runs
git, passes an argument to it, and returns text from its stdout.
In the Popen calls in Git.execute, for all combinations of allowed
values for Git.USE_SHELL and the shell= keyword argument.
The log message shows "Popen(...)", not "execute(...)". So it
should faithfully report what is about to be passed to Popen in
cases where a user reading the log would otherwise be misled into
thinking this is what has happened.

Reporting the actual "shell=" argument passed to Popen is also more
useful to log than the argument passed to Git.execute (at least if
only one of them is to be logged).
Both new shell-related tests were causing the code under test to
split "git" into ["g", "i", "t"] and thus causing the code under
test to attempt to execute a "g" command. This passes the command
as a one-element list of strings rather than as a string, which
avoids this on all operating systems regardless of whether the code
under test has a bug being tested for.

This would not occur on Windows, which does not iterate commands of
type str character-by-character even when the command is run
without a shell. But it did happen on other systems.

Most of the time, the benefit of using a command that actually runs
"git" rather than "g" is the avoidance of confusion in the error
message. But this is also important because it is possible for the
user who runs the tests to have a "g" command, in which case it
could be very inconvenient, or even unsafe, to run "g". This should
be avoided even when the code under test has a bug that causes a
shell to be used when it shouldn't or not used when it should, so
having separate commands (list and str) per test case parameters
would not be a sufficient solution: it would only guard against
running "g" when a bug in the code under test were absent.
This also helps mock Popen over a smaller scope, which may be
beneficial (especially if it is mocked in the subprocess module,
rather than the git.cmd module, in the future).
Because mock.call.kwargs, i.e. the ability to examine
m.call_args.kwargs where m is a Mock or MagicMock, was introduced
in Python 3.8.

Currently it is only in test/test_git.py that any use of mocks
requires this, so I've put the conditional import logic to import
mock (the top-level package) rather than unittest.mock only there.

The mock library is added as a development (testing) dependency
only when the Python version is lower than 3.8, so it is not
installed when not needed.

This fixes a problem in the new tests of whether a shell is used,
and reported as used, in the Popen call in Git.execute. Those
just-introduced tests need this, to be able to use
mock_popen.call_args.kwargs on Python 3.7.
This updates the shell variable itself, only when it is None, from
self.USE_SHELL.

(That attribute is usually Git.USE_SHELL rather than an instance
attribute. But although people probably shouldn't set it on
instances, people will expect that this is permitted.)

Now:

- USE_SHELL is used as a fallback only. When shell=False is passed,
  USE_SHELL is no longer consuled. Thus shell=False always keeps a
  shell from being used, even in the non-default case where the
  USE_SHELL attribue has been set to True.

- The debug message printed to the log shows the actual value that
  is being passed to Popen, because the updated shell variable is
  used both to produce that message and in the Popen call.
Copy link
Member

@Byron Byron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks so much for the fix which brings the behaviour in line with the documentation, and for adding regression protection.

This PR is also a wonderful example on what's possible with the unittest and mock libraries and generally what's possible with python testing.

🙏

@Byron Byron merged commit 874f0bf into gitpython-developers:main Oct 2, 2023
8 checks passed
@EliahKagan EliahKagan deleted the popen-shell branch October 2, 2023 19:59
renovate bot added a commit to allenporter/flux-local that referenced this pull request Oct 20, 2023
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [GitPython](https://togithub.com/gitpython-developers/GitPython) |
`==3.1.37` -> `==3.1.40` |
[![age](https://developer.mend.io/api/mc/badges/age/pypi/GitPython/3.1.40?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/GitPython/3.1.40?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/GitPython/3.1.37/3.1.40?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/GitPython/3.1.37/3.1.40?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|

---

### Release Notes

<details>
<summary>gitpython-developers/GitPython (GitPython)</summary>

###
[`v3.1.40`](https://togithub.com/gitpython-developers/GitPython/compare/3.1.38...3.1.40)

[Compare
Source](https://togithub.com/gitpython-developers/GitPython/compare/3.1.38...3.1.40)

###
[`v3.1.38`](https://togithub.com/gitpython-developers/GitPython/releases/tag/3.1.38)

[Compare
Source](https://togithub.com/gitpython-developers/GitPython/compare/3.1.37...3.1.38)

#### What's Changed

- Add missing assert keywords by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1678
- Make clear every test's status in every CI run by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1679
- Fix new link to license in readme by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1680
- Drop unneeded flake8 suppressions by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1681
- Update instructions and test helpers for git-daemon by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1684
- Fix Git.execute shell use and reporting bugs by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1687
- No longer allow CI to select a prerelease for 3.12 by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1689
- Clarify Git.execute and Popen arguments by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1688
- Ask git where its daemon is and use that by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1697
- Fix bugs affecting exception wrapping in rmtree callback by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1700
- Fix dynamically-set **all** variable by
[@&#8203;DeflateAwning](https://togithub.com/DeflateAwning) in
[gitpython-developers/GitPython#1659
- Fix small
[#&#8203;1662](https://togithub.com/gitpython-developers/GitPython/issues/1662)
regression due to
[#&#8203;1659](https://togithub.com/gitpython-developers/GitPython/issues/1659)
by [@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1701
- Drop obsolete info on yanking from security policy by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1703
- Have Dependabot offer submodule updates by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1702
- Bump git/ext/gitdb from `49c3178` to `8ec2390` by
[@&#8203;dependabot](https://togithub.com/dependabot) in
[gitpython-developers/GitPython#1704
- Bump git/ext/gitdb from `8ec2390` to `6a22706` by
[@&#8203;dependabot](https://togithub.com/dependabot) in
[gitpython-developers/GitPython#1705
- Update readme for milestone-less releasing by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1707
- Run Cygwin CI workflow commands in login shells by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[gitpython-developers/GitPython#1709

#### New Contributors

- [@&#8203;DeflateAwning](https://togithub.com/DeflateAwning) made their
first contribution in
[gitpython-developers/GitPython#1659

**Full Changelog**:
gitpython-developers/GitPython@3.1.37...3.1.38

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/allenporter/flux-local).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4xOS4yIiwidXBkYXRlZEluVmVyIjoiMzcuMTkuMiIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
2 participants