Pin dependency versions in requirements.txt for Docker Image

[cameronbronstein]

The local docker build is a bit slow, as conda has to work through several build attempts. This could maybe be sped up if specific versions were pinned in the requirements.txt. It could also improve local re-builds when adding dependencies that support new notebooks.

[cameronbronstein]

When adding the following libraries to the requirements.txt:

• datashader
• descartes
• geoviews
• holoviews
• xarray

Build attempt taken ~20 minutes before crashing.

Step 6/9 : RUN conda config --set channel_priority strict &&     conda install -y -c conda-forge --file /tmp/requirements.txt
 ---> Running in 9f1176ea350c
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... Solving environment: ...working... Killed
The command '/bin/sh -c conda config --set channel_priority strict &&     conda install -y -c conda-forge --file /tmp/requirements.txt' returned a non-zero code: 137

I'll investigate if these new libraries are making the conda solve more difficult or if it happens regardless.

[cameronbronstein]

A few more tests:

From the latest planetlabs/notebooks docker image, enter the container via /bin/bash and install new libraries one by one (alphabetically) with conda:

• conda install descartes
• conda install geoviews (holoviews is a dependency of geoviews, so we can remove that).

This verifies that we can conda install these.

Next, I ran conda list -e > pinned-requirements.txt. I was able to use these versions to modify the requirements.txt and rebuild the docker image. This does not appear to speed up the conda solve compared to the build pre-analytics notebooks dependencies. However, I retried the docker build w/o pinned dependencies again (but with these additional two libraries), and the build is killed.

My resolution is that pinning the dependencies, including the newly added descartes and geoviews, is the way to go.

[jreiberkyle]

@cameronbronstein so there was no issue with the build when you pinned the dependencies? My thought was that the build was being killed because conda solve timed out, so I'm surprised the build is successful even though conda solve takes just as long

[cameronbronstein]

Looking more into the killed problem, I now think it is a memory issue. Killed is the response from a system issue, not a conda issue.

I increased memory allocation for docker on my laptop to 8 BG. Looking at my activity monitor, docker was using nearly 9 GB of ram trying to solve the conda build. After more than 30 min, I exited the build.

This is further indication that pinning the dependencies improved conda solve times, however, pinning could cause issues in the future with new updates to notebooks and/or the dependencies themselves.