How to install Python dependencies

[yhoiseth]

In https://github.com/yhoiseth/python-prediction-scorer/runs/786096970, pylint is complaining about missing imports. They are already installed in a previous step, but it doesn’t seem like they are “kept”. How can I run e.g. python -m pip install --upgrade pip pip-tools setuptools wheel && pip-sync so that it works with Super-Linter?

[admiralAwkbar]

This is a great question!
I know we have had to add the lines like below before...

import click # pylint: disable=import-error

But I think it may stem from where the linter actually runs, like directory wise...
I need to do some more testing to help understand this...
I know there is a way to tell it where to find the import like: import click # pylint: import-path: /tmp/here or something like that but can't find the documentation... maybe someone more versed in pylint could help out?

[kevinfealey]

I came here looking for an answer to this same question - just wanted to bump it for visibility.

I had tried the following:

    - name: Pip Installer (application dependencies)
      uses: BSFishy/pip-action@v1
      with:
         requirements: requirements.txt

But I still get import errors.

[dang3r]

I attempted to handle this by disabling import-errors using a custom pylint configuration. Prepending import-error to the list of disabled errors at https://github.com/github/super-linter/blob/master/TEMPLATES/.python-lint#L57 prevents this error from being caught.

However, this project runs pylint using the -E/--errors-only flag in addition to using the custom configuration file. See here. The use of this flag seems to override the disabled errors present in the pylint configuration file.

The one workaround I found was to use --disable=R,C,W instead of the -E flag. This seems to disable all other types of checks except for errors, and it properly handles the disabled checks in the config file. I'm not sure if this is the right solution, but it could work.

[jgaffiot]

I finally found a workaround (works with pip + requirements.txt or pipenv). The problem is to find a way to provide your dependencies to the super-linter container, and then to tell pylint where they are.

For the first step, we can install our dependencies in the container running the action, but the super-linter runs in a separate container. So we have to share the dependencies with the super-linter container, something we can do with container volumes. But we can not declare new volumes... So let's look at the volume already mounted when the super-linter runs:

-v "/var/run/docker.sock":"/var/run/docker.sock"
-v "/home/runner/work/_temp/_github_home":"/github/home"
-v "/home/runner/work/_temp/_github_workflow":"/github/workflow"
-v "/home/runner/work/<your_repo>/<your_repo>":"/github/workspace"

We could put our dependencies in /home/runner/work/<your_repo>/<your_repo>, but they will be linted too, so let's use /home/runner/work/_temp/_github_workflow.

Then we can tell pylint where are the dependencies by using the environment variable PYTHONPATH. Right, all together we get:

---
# Beginning is the same than the official documentation
name: Lint Code Base
on:
  push:
    branches-ignore:
      - 'master'
jobs:
  build:
    name: Lint Code Base
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v2
      # Here I use the system Python, which is 3.6.
      # You may install another Python version with the proper action
      - name: Install missing python packages
        run: sudo apt-get install -y --no-install-recommends python3-venv python3-setuptools
      # USE ONLY ONE THE 2 STEPS BELOW
      ## Dependencies with pip + requirements.txt
      - name: Install dependencies with pip + requirements.txt
        run: |
          python3 -m venv .venv
          .venv/bin/pip install -r requirements.txt
      ## Dependencies with pipenv
      - name: Install dependencies with pipenv
        run: |
          python3 -m pip install pipenv
          python3 -m pipenv sync --python 3.6
        env:
          PIPENV_VENV_IN_PROJECT: 1
      # Now we move the dependencies where super-linter can see them
      - name: Move the dependencies
        run: mv .venv /home/runner/work/_temp/_github_workflow
      # Now we set the PYTHONPATH to the path of the dependencies *inside* the container
      - name: Lint Code Base
        uses: docker://github/super-linter:v2.2.0
        env:
          VALIDATE_ALL_CODEBASE: true
          PYTHONPATH: "/github/workspace/:\
            /github/workflow/.venv/lib/python3.6/site-packages"
...

Hope this helps!

[tunetheweb]

The other workaround is to copy the template into .github/linters/.python-lint and then add these to ignored modules at line 246. For example:

# List of module names for which member attributes should not be checked
# (useful for modules/projects where namespaces are manipulated during runtime
# and thus existing member attributes cannot be deduced by static analysis. It
# supports qualified module names, as well as Unix pattern matching.
ignored-modules=flask,
                flask_talisman,
                matplotlib,
                matplotlib.pyplot,
                mistune,
                pandas,
                scour,
                werkzeug.routing,
                werkzeug.http

[elopez]

@jgaffiot I'm struggling to get your workaround to work, I suspect it is because some of the dependencies I need have binary components (.so libraries), and the super-linter docker image uses Alpine Python, which has musl libc, in contrast to the Ubuntu host where the venv is created, which uses glibc. Do you have any ideas on how we could make it work? 🤔 Thanks!

[jgaffiot]

If the problem really comes from incompatibilities between glibc and musl, I suppose you would have to build your Python dependencies in an Alpine-based image. Unfortunately, it seems that Github does not provide Alpine-based image for actions, neither natively and neither with self-hosted runner. Perhaps by building your own action based on an image taken on DockerHub ?
The other way would be to convince Github to provide another super-linter action, based on Ubuntu, which I suspect would not be much larger than an Alpine-based one once Python is installed.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity.
It will be closed in 14 days if no further activity occurs.
Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[alerque]

@github-actions The problem is not fixed, ergo the issue should not be closed.

[tylermilner]

I'm a bit surprised there is not more mention in the documentation of needing this workaround for linting Python code. As a python newbie, it took me a while to track down why linting with the super-linter was producing errors when linting locally was working fine.

Thanks to jgaffiot's solution, I was able to put together something that works for my use case:

• Uses ruff for linting and mypy for type checking, configured using a pyproject.toml file
• Uses pipenv for dependency management
• Uses .python-version file to specify the Python version

Here is what my lint job looks like:

jobs:
  lint:
    name: Lint Codebase
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        id: checkout
        uses: actions/checkout@v4
        with:
          # super-linter needs the full git history to get the
          # list of files that changed across commits
          fetch-depth: 0
      - name: Set up Python, including cache for pipenv virtual environment
        uses: actions/setup-python@v5
        with:
          python-version-file: .python-version
          cache: pipenv
      - name: Install pipenv
        run: |
          pip install --upgrade pip
          pip install pipenv
      - name: Install project dependencies
        run: pipenv install --deploy --dev
      - name: Get virtual environment path
        id: get-venv-path
        run: echo "venv-path=$(pipenv --venv)" >> "$GITHUB_OUTPUT"
      # Copy python dependencies to a location that the super-linter will be
      # able to access when running inside its Docker container
      # '/home/runner/work/_temp/_github_workflow' maps to '/github/workflow'
      # in the Docker container
      - name: Copy python dependencies
        run: cp -r "${{ steps.get-venv-path.outputs.venv-path }}" /home/runner/work/_temp/_github_workflow/.venv
      # Extract MAJOR.MINOR version from .python-version file to be used in
      # Python folder name when setting PYTHONPATH for super-linter
      - name: Get Python version from .python-version file
        id: get-python-version
        run: echo "python-version=$(cut -d '.' -f 1,2 .python-version)" >> "$GITHUB_OUTPUT"
      - name: Lint Codebase
        id: super-linter
        uses: super-linter/super-linter/slim@v6
        env:
          DEFAULT_BRANCH: main
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          LINTER_RULES_PATH: .  # Set linter rules directory to repo root
          MARKDOWN_CONFIG_FILE: .github/linters/.markdown-lint.yml
          PYTHONPATH: "/github/workspace:/github/workflow/.venv/lib/python${{ steps.get-python-version.outputs.python-version }}/site-packages"
          PYTHON_MYPY_CONFIG_FILE: pyproject.toml
          PYTHON_RUFF_CONFIG_FILE: pyproject.toml
          VALIDATE_ALL_CODEBASE: true
          VALIDATE_PYTHON_BLACK: false
          VALIDATE_PYTHON_FLAKE8: false
          VALIDATE_PYTHON_PYLINT: false
          VALIDATE_JSCPD: false
          YAML_CONFIG_FILE: .github/linters/.yaml-lint.yml