Custom Go plugins on Kong 2.0 base

[joseandrespg]

We are trying to test the new capabilities of Kong with custom Go plugins. We are doing so using the official images and the example plugins as a base.

After copying the necessary files and configuring the paths suggested in here https://docs.konghq.com/2.0.x/go/ we see that Kong tries to execute the server, but it inmediately exit, trying to use one of the plugins will result in an error, as the socket for the Go Server hasn't been yet created.

To reproduce

These are files we are using to create the custom build:

Dockerfile:

## Compile Go files
FROM golang:1.13.5 as compiler

RUN git clone https://github.com/Kong/go-plugins /usr/src/go-plugins
RUN cd /usr/src/go-plugins && make

RUN git clone https://github.com/Kong/go-pluginserver /usr/src/go-pluginserver
RUN cd /usr/src/go-pluginserver && make

## Build custom Kong image
FROM kong:2.0-ubuntu
LABEL maintainer="HP GSB Solutions"

# Copy Kong configuration
COPY conf/kong.conf /etc/kong/kong.conf

# Copy Go files
COPY --from=compiler /usr/src/go-plugins/*.so /usr/local/go-plugins/
RUN chmod -R 755 /usr/local/go-plugins/*.so

# Copy Go Plugin Server executable
COPY --from=compiler /usr/src/go-pluginserver/go-pluginserver /usr/local/go-pluginserver
RUN chmod -R 755 /usr/local/go-pluginserver

conf/kong.conf:

plugins=bundled,go-hello
go_plugins_dir=/usr/local/go-plugins
go_pluginserver_exe=/usr/local/go-pluginserver

And building like:

docker build --rm=true --force-rm=true -t kong2experiment .

We are executing this image within a Docker Compose inspired in https://github.com/Kong/docker-kong/blob/master/compose/docker-compose.yml

Logs

The results will be an infinite loop of messages like:

kong_1            | 2020/01/23 16:10:16 [notice] 31#0: *22 [kong] go.lua:84 Starting go-pluginserver, context: ngx.timer
kong_1            | 2020/01/23 16:10:16 [notice] 31#0: signal 17 (SIGCHLD) received from 12562
kong_1            | 2020/01/23 16:10:16 [notice] 31#0: *22 [kong] go.lua:95 go-pluginserver terminated: exit 0, context: ngx.timer

---

We tried with the server shipped in the image, but it was complaining about a missmatch of the versions (like the docs predicted), therefore we compile both things within the same environemt.

Could you please advise on what are we missing to configure or doing wrong?

Thanks.

[gszr]

@joseandrespg Thank you for the detailed writeup. I suspect this might be due different base OSes being used. Could you try using Ubuntu Xenial in the compiler stage? You could, for example, download the Go compiler from https://golang.org/dl/go1.13.5.amd64.tar.gz (if you can't find a Golang image based on Xenial).

[joseandrespg]

Hi @gszr thanks for your reply.

We have tried with three different bases for the compiler stage: ubuntu:16.04, ubuntu:18.04 and the same image as Kong kong:2.0-ubuntu, but we always get the same result:

kong_1            | 2020/02/10 10:55:30 [notice] 31#0: signal 17 (SIGCHLD) received from 2654
kong_1            | 2020/02/10 10:55:30 [notice] 31#0: *20 [kong] go.lua:95 go-pluginserver terminated: exit 0, context: ngx.timer
kong_1            | 2020/02/10 10:55:30 [notice] 31#0: *20 [kong] go.lua:84 Starting go-pluginserver, context: ngx.timer

Here the new version of the Dockerfile:

#####################
## Go plugins
#####################
FROM kong:2.0-ubuntu as compiler

## We've tried also :
## FROM ubuntu:16.04 as compiler
## FROM ubuntu:18.04 as compiler

# Install build tools
RUN apt-get update -y && DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y -q curl build-essential ca-certificates git

# Download and configure Go compiler
RUN curl -s https://storage.googleapis.com/golang/go1.13.5.linux-amd64.tar.gz | tar -v -C /usr/local -xz
ENV GOPATH /go
ENV GOROOT /usr/local/go
ENV PATH $PATH:/usr/local/go/bin

# Copy and compile go-plugins
RUN git clone https://github.com/Kong/go-plugins /usr/src/go-plugins
RUN cd /usr/src/go-plugins && make

RUN git clone https://github.com/Kong/go-pluginserver /usr/src/go-pluginserver
RUN cd /usr/src/go-pluginserver && make

#####################
## Release image
#####################
FROM kong:2.0-ubuntu

# Copy Kong configuration
COPY conf/kong.conf /etc/kong/kong.conf

# Copy Go files
COPY --from=compiler /usr/src/go-plugins/*.so /usr/local/go-plugins/
RUN chmod -R 755 /usr/local/go-plugins/*.so

COPY --from=compiler /usr/src/go-pluginserver/go-pluginserver /usr/local/go-pluginserver
RUN chmod -R 755 /usr/local/go-pluginserver

[gszr]

@joseandrespg,

We've identified two issues here:
1 - and what you are facing: Go mod and the plugin buildmode don't play nice with each other, as you can see here. As a result, you have to build in an alternative way - e.g., outside the directory where the go mod is.

2 - Once you fix 1., you will notice another issue. Our Ubuntu Dockerfile runs by default as root; as a result, Nginx workers will drop to nobody (or whatever user was specified in the configuration file), causing the pluginserver to fail initializing. Kong will keep trying to start the pluginserver, which will keep failing. A solution is to avoid running as root, either specifying USER in your Dockerfile or passing the --user flag to docker run.

Here is a slight variation of your Dockerfile that works around these issues. The declarative config file I used is this and you can test with the following command:

 docker run -ti --rm --name kong-go-plugins \ 
  -e "KONG_DATABASE=off" \
  -e "KONG_GO_PLUGINS_DIR=/usr/local/kong" \
  -e "KONG_PLUGINS=go-hello" \
  -e "KONG_DECLARATIVE_CONFIG=/tmp/config.yml" \
  -e "KONG_PROXY_LISTEN=0.0.0.0:8000" \
  -p 8000:8000 \
  kong-demo

Once again, thanks for reporting. We will post an update here once we have addressed the Dockerfile issue on our end.

[joseandrespg]

@gszr I can confirm that your workaround and your suggestions have been very helpful, they do work 👏

Thank you so much for your help!

[gszr]

Hi all - our new Docker image (for Kong 2.0.4) already incorporates the fix for the issue 2. pointed out above. I'm closing this for now, but please reopen if needed.

[Tharunvarshanth]

In our application we wrote go custom plugin using Kong Plugin Development Kit (https://pkg.go.dev/github.com/Kong/go-pdk) when we run the application plugin is started working after first a api request completed plugin is getting terminated ,we are getting error message like this external plugin server "plugin name" terminated , can some one help how to fix this issue

[CoffeeSlayer007]

I am facing same issue as @Tharunvarshanth. Can someone pls help.