feat(manager/pip-compile): add full support for lock files

[not7cd]

Changes

As this is a draft PR, the list below is a planned scope.

• Extract dependencies from lock files, as only requirements.in is handled at the moment.
• Run pip-compile --upgrade-package for indirect dependencies that exist only in requirements.txt.
  • Bundle updates to lock file and run a single command if grouped.
• Better command extraction handling. Allow for CUSTOM_COMPILE_COMMAND if it doesn't obstruct original arguments. See Feature request: pip-compile support #24725 (reply in thread)
• Handle other pip-compile arguments, pass them without change as to not change command in the header.

Context

Support for pip-compile is partial. It doesn't parse lock files like other implemented managers do. In turn it will ignore updates for dependencies in the lock file. It also "eats" arguments extracted from header in the lock file, for example when --no-emit-index-url is dropped, this is very undesirable as PIP_INDEX_URL can contain secrets like HTTP passwords. Lastly hard-coded pip-compile command doesn't allow for CUSTOM_COMPILE_COMMAND, and will not allow for setups that require docker containers that have system depenencies pre-installed.

Discussion 2018-2022: #24725

Documentation (please check one with an [x])

• I have updated the documentation, or
• No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

• Code inspection only, or
• Newly added/modified unit tests, or
• No unit tests but ran on a real repository, or
• Both unit tests + ran on a real repository

[rarkins]

Hi @not7cd thanks very much for taking a shot at this! One thing I'd like to say though is that design decisions or review are better done in issues and not inline with PRs. Could you summarize your design decisions in #24725 so we can discuss there?

This includes some fundamental decisions such as how to map pip compile concepts to Renovate concepts (and whether any new concepts need adding), e.g. should the file.in be considered "package file" and the file.out be "lock file"? Should Renovate update individual requirements in file.out or only from file.in? etc

[not7cd]

Thanks for quick response @rarkins. I need to figure out what Renovate concepcts are. And this draft PR is a way to do it. I have already added one comment about package file discovery based on lock files.
#24725 (comment)

[not7cd]

Dropped in favour of #26858