Add support for git tokens, to be used for python dependencies from private git repos, in the main python template. #292
Conversation
|
Thank you for your contribution. unfortunately, one or more of your commits are missing the required "Signed-off-by:" statement. Signing off is part of the Developer Certificate of Origin (DCO) which is used by this project. Read the DCO and project contributing guide carefully, and amend your commits using the git CLI. Note that this does not require any cryptography, keys or special steps to be taken. 💡 Shall we fix this?This will only take a few moments. First, clone your fork and checkout this branch using the git CLI. Next, set up your real name and email address:
Finally, run one of these commands to add the "Signed-off-by" line to your commits. If you only have one commit so far then run: Check that the message has been added properly by running "git log". |
|
Thank you for your contribution. unfortunately, one or more of your commits are missing the required "Signed-off-by:" statement. Signing off is part of the Developer Certificate of Origin (DCO) which is used by this project. Read the DCO and project contributing guide carefully, and amend your commits using the git CLI. Note that this does not require any cryptography, keys or special steps to be taken. 💡 Shall we fix this?This will only take a few moments. First, clone your fork and checkout this branch using the git CLI. Next, set up your real name and email address:
Finally, run one of these commands to add the "Signed-off-by" line to your commits. If you only have one commit so far then run: Check that the message has been added properly by running "git log". |
These tokens can be used for python dependencies for private git repos. These changes are made in the main python template. Signed-off-by: CC007 <Coolcat_the_best2@Hotmail.com>
CC007
force-pushed
the
feature/private-git-dependency-support
branch
from
032868c
to
61bcda0
Compare
| WORKDIR /home/app/ | ||
| COPY --from=builder /home/app/.cache /home/app/.cache |
There was a problem hiding this comment.
My concern would be whether any C/C++ libraries that were built or installed into the system are still available at this point such as numpy, pillow or pandas.
There was a problem hiding this comment.
Good point. I'll see if I can test this. I believe requests also requires C/C++ libs.
There was a problem hiding this comment.
Confirmed that numpy only installs to /home/app/.cache and /home/app/python. I'll try pillow and pandas next.
There was a problem hiding this comment.
Pillow doesn't correctly install, due to a missing zlib dependency (and even when that is added, it is installed to /lib instead of /usr/lib, so python still can't find it)
See: link
There was a problem hiding this comment.
Pandas seems to work fine too, only using those 2 folders
template/python3/Dockerfile
Outdated
| # Install git and make the git token available as environment variable | ||
| USER root | ||
| RUN apk --no-cache add git | ||
| ENV GIT_TOKEN=${GIT_TOKEN} |
There was a problem hiding this comment.
It may be better to avoid leaking the ENV into the build, and instead prefixing the value in the pip install command.
There was a problem hiding this comment.
I was going to say the same thing, like this
RUN GIT_TOKEN=${GIT_TOKEN} pip install -r requirements.txt --target=/home/app/python
There was a problem hiding this comment.
I didn't think that the env var would be leaked if used in the builder stage, but I do agree that it's neater the way that @LucasRoesler proposes. That way it won't unnecessarily be available to all the other commands.
…ment variable for the whole builder stage Signed-off-by: CC007 <Coolcat_the_best2@Hotmail.com>
|
Hmm, I found this article: https://pythonspeed.com/articles/docker-build-secrets/
So it would be safe with a multi-stage build like in this pull request, but it is recommended to use Buildkit's --secret feature instead... I'm going to look into this, because this will also make sure that C/C++ dependencies like numpy and pandas work out of the box, because there is no need for a builder stage. Since we are using an alpine-based image, Buildkit should be available, since it works for linux-based images. |
|
Thanks for the suggestions. We had a customer ask for this, and we implemented it in OpenFaaS Pro Introducing our new Python template for production. See also: Private npm modules |
|
/lock: resolved |
Description
This change adds the ability to provide a GIT_TOKEN as a build argument. This git token is then set as an environment variable.
Also the pip install code is moved to a builder stage.
Motivation and Context
The reason to make this GIT_TOKEN environment variable available, is so that it can be used when adding the following kind of dependency to requirements.txt:
Which issue(s) this PR fixes
Fixes openfaas/faas#1723
How Has This Been Tested?
It has been tested by creating a python module in a private github repo (using this video as an instruction). Then I created a python project with a requirements.txt file that uses this private git repo as a dependency.
Types of changes
Impact to existing users
This doesn't have any direct impact on the end users, as the resulting image will contain the same files. The image layers are slightly different though, due to the pip install commands being moved to a builder stage. This has a very small impact on how the layers are cached, but since the the default requirements.txt in the template doesn't have any dependencies, this doesn't matter all that much.
Checklist:
git commit -s