Provide a way to replicate Ansible/Python environment of Docker image

[dalbani]

Feature Request

Describe the problem you need a feature to resolve.

In order to run the ansible-operator locally for development purposes (with make run), one has to setup an Ansible / Python (virtual) environment.
Ideally, it would have to be identical to the one found in the Docker image quay.io/operator-framework/ansible-operator which is then deployed on production.
However, due to how the Python dependencies are declared in the Dockerfile, it is difficult to do so accurately.
See https://github.com/operator-framework/operator-sdk/blob/master/images/ansible-operator/Dockerfile#L19:

RUN yum clean all && rm -rf /var/cache/yum/* \
  && yum -y update \
  && yum install -y libffi-devel openssl-devel python36-devel gcc python3-pip python3-setuptools \
  && pip3 install --no-cache-dir \
    ipaddress \
    ansible-runner==1.3.4 \
    ansible-runner-http==1.0.0 \
    openshift~=0.10.0 \
    ansible~=2.9 \
    jmespath \
  && yum remove -y gcc libffi-devel openssl-devel python36-devel \
  && yum clean all \
  && rm -rf /var/cache/yum

The first solution is of course to track down the Dockerfile on Github and copy/paste the pip3 install ... command locally.
It works, but it's not very elegant and requires manual work which would be tricky to automate reliably.

Then I thought that I could simply extract the list of installed Python packages with pip install $(docker run --rm -ti --entrypoint bash quay.io/operator-framework/ansible-operator:v1.2.0 -c 'pip3 freeze').
But I ran into the problem of system packages: the freeze list also contains Python packages installed system-wide via YUM.
Which means I ended up getting gpg==1.13.1 in the freeze list, for example, because the RPM package gpg is present on the system. And gpg==1.13.1 is not a package available on PyPI...

So the pip freeze command becomes a bit more complicated (and still depending on the content of the Dockerfile!):

$ pip3 freeze | grep -E '^ipaddress|ansible-runner|openshift|ansible|jmespath'

This solution works quite well, even though it's not perfect: there's always the possibility of locally installing a different version of some dependencies of those packages, should a new release be made since the Docker image has been built for example.

Describe the solution you'd like.

My suggestion would be to have a requirements.txt file available in the Docker image at a documented location.
And that the packages mentioned inside would have to be pinned to specific versions for reproducibility purposes.
So one can easily set up a local environment with:

pip install $(docker run --rm -ti --entrypoint cat quay.io/operator-framework/ansible-operator:v1.2.0 /path/to/requirements.txt)

Any other idea?

/language ansible

[estroz]

Steps to install these deps need to be documented.

/kind documentation
/assign @asmacdo

[reinvantveer]

👍 for the suggestion of transparent local dependency installation. A couple of thoughts. There's a lot of potential for dependency hell here, pip3 install-ing in the global site-packages, I'd much rather see a solution that uses virtualenvs and deterministic builds through the Makefile. We've been using pipenv successfully for quite some time now, and it would be a welcome addition for managing python dependencies in custom ansible modules as well.

[asmacdo]

Adding release blocker since this blocks #4202

[asmacdo]

This no longer blocks #4202, but this turned out to be a little more involved than I realized. I dont want this to block the release, so I'm bumping to 1.4

I think I agree with @reinvantveer. What I have in mind is creating a virtualenv in the image, and installing the top-level deps via a pinned requirements.txt.

TODO:

• Base image:
  • pull the top level Python dependencies out of Dockerfile and into a requirements.txt.
• Local development:
  • Add a requirements.txt to the Ansible scaffolding with the same top level dependencies
  • Add a test-requirements.txt to the Ansible scaffolding that installs molecule
  • Add a make target that creates a virtualenv and installs both prod and dev requirements
  • update make run to use the venv
  • update the installation docs
• CI:
  • Install from requirements.txt and test-requirements.txt
• Update Script should allow all references to be maintained in a single place

[reinvantveer]

Agreed that having a *requirements.txt for a virtual environment is a huge improvement. However, this leaves one thing still in limbo: the Python version itself - a requirements.txt file doesn't allow you to specify this, while pipenv can (I'm unsure whether poetry can). In a nutshell: I don't quite feel that a requirements.txt file is a substitute for proper dependency management.

On this note on dependency management: it's becoming harder to keep Python 3.6 for local development - I switched to Ubuntu Focal recently and now you need something like the deadsnakes repo to get a hold of mothballed Python versions. 3.6 now only gets security fixes: see https://www.python.org/downloads/release/python-3612/. It's time to move on to a more recent version.

[reinvantveer]

I'm having a first go at an attempt to contribute here: upgrading Python in the ansibe operator Docker image.
#4413

[estroz]

@reinvantveer thanks! I can assign you to this issue in that case.

/assign @reinvantveer

[reinvantveer]

Following comments #4413 (comment) and #4413 (comment) now the PR for pinning the base images for Ansible and Helm operators

[dalbani]

Another component which probably needs to have its version pinned: the Ansible Galaxy collections.
https://github.com/operator-framework/operator-sdk/blob/master/internal/plugins/ansible/v1/scaffolds/internal/templates/requirements.go#L35

[reinvantveer]

I agree

[reinvantveer]

Now #4417 and #4413 have been merged, here PR #4477 to use a dependency resolution system for the main Python dependencies

[reinvantveer]

Fixed a few things in #4477 so that the checks now pass

[reinvantveer]

Something went horribly wrong in #4477 so I'm retrying in #4494

[reinvantveer]

The reproducibility issue just got a little hairier: I've just become aware of pyca/cryptography#5753. Seems like a lot of dependents are pinning cryptography to version 3.3.2 due to breaking builds. Cryptography v 3.4 requires a Rust compiler to be present to build. The ansible package requires ansible-build, which in turn depends on cryptography, but fortunately no specific version is required.

[reinvantveer]

Hm, maybe the cryptography storm will blow over again, now cryptography v3.4.3 was released just a couple of hours ago.
EDIT: It has to do with the pip version, when >= 19.0 it will not require a Rust compiler, as documented at https://cryptography.io/en/latest/installation.html#building-cryptography-on-linux

[reinvantveer]

Another component which probably needs to have its version pinned: the Ansible Galaxy collections.
https://github.com/operator-framework/operator-sdk/blob/master/internal/plugins/ansible/v1/scaffolds/internal/templates/requirements.go#L35

#4529