Using docker image with postgres

[Nikos410]

Hey there!

I would like to use the docker image with a PostgreSQL database, but psycopg2 is not installed in the docker image.

I tried adding it afterwards by running from a simple local Dockerfile like this:

FROM ihatemoney/ihatemoney:5.1.1
RUN pip install --no-cache-dir psycopg2

But building fails:

Step 1/2 : FROM ihatemoney/ihatemoney:5.1.1
 ---> 08d796f3a6e6
Step 2/2 : RUN pip install --no-cache-dir psycopg2
 ---> Running in cdb4fb5e4801
Collecting psycopg2
  Downloading psycopg2-2.9.2.tar.gz (380 kB)
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-z6s31rox/psycopg2_907bd598618b4dd796894dbecda3d67b/setup.py'"'"'; __file__='"'"'/tmp/pip-install-z6s31rox/psycopg2_907bd598618b4dd796894dbecda3d67b/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-1wer96lf
         cwd: /tmp/pip-install-z6s31rox/psycopg2_907bd598618b4dd796894dbecda3d67b/
    Complete output (23 lines):
    running egg_info
    creating /tmp/pip-pip-egg-info-1wer96lf/psycopg2.egg-info
    writing /tmp/pip-pip-egg-info-1wer96lf/psycopg2.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-1wer96lf/psycopg2.egg-info/dependency_links.txt
    writing top-level names to /tmp/pip-pip-egg-info-1wer96lf/psycopg2.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-1wer96lf/psycopg2.egg-info/SOURCES.txt'
    
    Error: pg_config executable not found.
    
    pg_config is required to build psycopg2 from source.  Please add the directory
    containing pg_config to the $PATH or specify the full executable path with the
    option:
    
        python setup.py build_ext --pg-config /path/to/pg_config build ...
    
    or with the pg_config option in 'setup.cfg'.
    
    If you prefer to avoid building psycopg2 from source, please install the PyPI
    'psycopg2-binary' package instead.
    
    For further information please check the 'doc/src/install.rst' file (also at
    <https://www.psycopg.org/docs/install.html>).
    
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/33/ed/79434011d773e5ea4c51262f6ebfb86680c2908d7677f31ebccd5aa9f1b3/psycopg2-2.9.2.tar.gz#sha256=a84da9fa891848e0270e8e04dcca073bc9046441eeb47069f5c0e36783debbea (from https://pypi.org/simple/psycopg2/) (requires-python:>=3.6). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Downloading psycopg2-2.9.1.tar.gz (379 kB)
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-z6s31rox/psycopg2_a59a9d084ba24aaeaf5077bcf4323ec0/setup.py'"'"'; __file__='"'"'/tmp/pip-install-z6s31rox/psycopg2_a59a9d084ba24aaeaf5077bcf4323ec0/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-acubs1ih
         cwd: /tmp/pip-install-z6s31rox/psycopg2_a59a9d084ba24aaeaf5077bcf4323ec0/
    Complete output (23 lines):
    running egg_info
    creating /tmp/pip-pip-egg-info-acubs1ih/psycopg2.egg-info
    writing /tmp/pip-pip-egg-info-acubs1ih/psycopg2.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-pip-egg-info-acubs1ih/psycopg2.egg-info/dependency_links.txt
    writing top-level names to /tmp/pip-pip-egg-info-acubs1ih/psycopg2.egg-info/top_level.txt
    writing manifest file '/tmp/pip-pip-egg-info-acubs1ih/psycopg2.egg-info/SOURCES.txt'
    
    Error: pg_config executable not found.
    
    pg_config is required to build psycopg2 from source.  Please add the directory
    containing pg_config to the $PATH or specify the full executable path with the
    option:
    
        python setup.py build_ext --pg-config /path/to/pg_config build ...
    
    or with the pg_config option in 'setup.cfg'.
    
    If you prefer to avoid building psycopg2 from source, please install the PyPI
    'psycopg2-binary' package instead.
    
    For further information please check the 'doc/src/install.rst' file (also at
    <https://www.psycopg.org/docs/install.html>).
    
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/aa/8a/7c80e7e44fb1b4277e89bd9ca509aefdd4dd1b2c547c6f293afe9f7ffd04/psycopg2-2.9.1.tar.gz#sha256=de5303a6f1d0a7a34b9d40e4d3bef684ccc44a49bbe3eb85e3c0bffb4a131b7c (from https://pypi.org/simple/psycopg2/) (requires-python:>=3.6). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

There is a lot more output which seems to be pip trying to install previous versions of psycopg2.

I don't have a lot of experience with python, so I'm not really sure how to solve this. Maybe it would be best to install the dependency in the default docker image so it can be used with postgres out of the box?

[almet]

It seems to tell you that you're missing some dependencies in order to install psycopg in your docker environment. Specifically, you're missing the pg_config executable.

It seems that this should solve your issue :

RUN apk update && apk add postgresql-dev gcc python3-dev musl-dev

[Nikos410]

Thanks for the quick response @almet, that solved my issue!

What do you think about including psycopg2 in the docker image by default?

[almet]

Yeah, why not!

[zorun]

Last time I checked, the psycopg2 binary wheel was not supported on Alpine (because of musl). So, there are two options:

• either install all build dependencies and build psycopg2 (what you did, as far as I can tell)
• either switch to a debian-based base image so that we can simply install the binary wheel. But the image will probably be a bit larger.

In any case, +1 for supporting postgresql by default in the docker image, the doc needs to be updated in consequence.

[Nikos410]

Hey, I tested a few variants:

• Current master, without psycopg2: ~20 seconds build time, 110 MB image size
• Alpine, compiling psycopg2 from source: ~50 seconds build time, 429 MB image size
• Debian bullseye: ~20 seconds build time, 973 MB image size
• Debian bullseye slim: ~20 seconds build time, 196 MB image size

It seems like using the slim-bullseye base image is the best choice. However, the description says the following about that image variant:

This image does not contain the common packages contained in the default tag and only contains the minimal packages needed to run python. Unless you are working in an environment where only the python image will be deployed and you have space constraints, we highly recommend using the default image of this repository.

Again, since I don't have a lot of experiance using python, I'm not sure whether this is an issue for this project.

What do you think?

About my test setup:

I used the following command to measure build time and size:

date \
  && docker build --no-cache -t ihatemoney-size-test . > /dev/null \
  && date \
  && docker image ls | grep ihatemoney-size-test;

The build speed will probably vary depending on your internet connection speed, especially for the second variant, where downloading the additional build-time dependencies takes quite a while. I tested the above builds on a machine with a 100 Mbit/s connection.

[Nikos410]

Also tagging @youegraillot who is working on docker-related stuff in #919

[youegraillot]

having Postgre compatibility would be great!
should I add this to #919 or should we create another PR ?

[zorun]

Thanks for running the numbers!

Indeed, the debian-based slim image looks like the best compromise. I don't think the "warning" is particularly relevant for us.

@youegraillot I guess it could go in #919 yes!

[youegraillot]

#919 is updated

I went with adding build dependencies at build and keeping only postgresql-libs for pg_config executable, this is rather standard when needing musl

I end up with a 101MB image

[zorun]

Excellent 👍

There was something in the doc about missing support for postgresql, but it apparently disappeared with the conversion to markdown, so all good here.