Girder does not restart in Docker

[Billuc]

Following this discussion on the forum, I wanted to bring you a problem I have encountered.

I am working with Docker containers and one task I've had recently is to create dev Dockers that would automatically update when a change is made to the code (in this case a plugin's code).

In order to do this, I copied my plugin in the container, built the server, mounted the same plugin at the same adsress with a volume and started the server with the --dev option.
The autoreloader is correctly started. However, when I modify the server stops, tries to restart and never does. Do you have an idea of what could cause this ? I have looked at similar issues, but it doesn't look like this issue has been encountered before...

Here is my repro.

Here are the logs I get in my docker :

girder-1  | [26/Feb/2024:10:17:37] ENGINE Restarting because /app/girder/plugin/girder_plugin/test_rest.py changed.
girder-1  | [26/Feb/2024:10:17:37] ENGINE Stopped thread 'Autoreloader'.
girder-1  | [26/Feb/2024:10:17:37] ENGINE Bus STOPPING
girder-1  | [26/Feb/2024:10:17:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('0.0.0.0', 8080)) shut down
girder-1  | [26/Feb/2024:10:17:38] ENGINE Bus STOPPED
girder-1  | [26/Feb/2024:10:17:38] ENGINE Bus EXITING
girder-1  | Stopped asynchronous event manager thread.
girder-1  | [26/Feb/2024:10:17:38] ENGINE Bus EXITED
girder-1  | [26/Feb/2024:10:17:38] ENGINE Waiting for child threads to terminate...
girder-1  | [26/Feb/2024:10:17:38] ENGINE Re-spawning /usr/bin/python3 /usr/local/bin/girder serve --dev

[zachmullen]

This is fixed in v5, FWIW. #3523

[Billuc]

@zachmullen That's great news ! Any idea of when that could come out ?
I can't seem to make it work with v3.2.2, so I am excited for this ! 😃

[zachmullen]

We don't have an official date but I would expect sometime in the next month or two. Prerelease will likely be within the next couple of weeks.

[Billuc]

I just ran a test with a docker I am using. Girder version is 3.2.2.
Girder actually restarts when a change is made !
The problem is that it takes 10 min for Girder to restart, which explains why I thought it didn't restart !
Any idea of why it takes so much time ?

Here are some logs :

girder-dev-1  | Running in mode: development
girder-dev-1  | Connecting to MongoDB: mongodb://mongodb:27017/girder
girder-dev-1  | Loaded plugin "advanced_management"
girder-dev-1  | WARNING: No description docs present for route PUT collection/:/i/d
girder-dev-1  | WARNING: No access level specified for route PUT collection/:/i/d
girder-dev-1  | Loaded plugin "aimodules_ohif"
girder-dev-1  | Loaded plugin "auth_ohif"
girder-dev-1  | Loaded plugin "resources_ohif"
girder-dev-1  | /usr/local/lib/python3.8/site-packages/pymongo/collection.py:1696: UserWarning: use an explicit session with no_cursor_timeout=True otherwise the cursor may still timeout after 30 minutes, for more info see https:
//docs.mongodb.com/v4.4/reference/method/cursor.noCursorTimeout/#session-idle-timeout-overrides-nocursortimeout
girder-dev-1  |   return Cursor(self, *args, **kwargs)
girder-dev-1  | [29/Mar/2024:13:11:35] ENGINE Listening for SIGTERM.
girder-dev-1  | [29/Mar/2024:13:11:35] ENGINE Listening for SIGHUP.
girder-dev-1  | [29/Mar/2024:13:11:35] ENGINE Listening for SIGUSR1.
girder-dev-1  | [29/Mar/2024:13:11:35] ENGINE Bus STARTING
girder-dev-1  | [29/Mar/2024:13:11:35] ENGINE Started monitor thread 'Autoreloader'.
girder-dev-1  | Started asynchronous event manager thread.
girder-dev-1  | [29/Mar/2024:13:11:36] ENGINE Serving on http://0.0.0.0:8080
girder-dev-1  | [29/Mar/2024:13:11:36] ENGINE Bus STARTED
girder-dev-1  | [29/Mar/2024:13:35:24] ENGINE Restarting because /girder/plugins/auth_ohif/girder_auth_ohif/routes/authenticate.py changed.
girder-dev-1  | [29/Mar/2024:13:35:24] ENGINE Stopped thread 'Autoreloader'.
girder-dev-1  | [29/Mar/2024:13:35:24] ENGINE Bus STOPPING
girder-dev-1  | [29/Mar/2024:13:35:24] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('0.0.0.0', 8080)) shut down
girder-dev-1  | [29/Mar/2024:13:35:24] ENGINE Bus STOPPED
girder-dev-1  | [29/Mar/2024:13:35:24] ENGINE Bus EXITING
girder-dev-1  | Stopped asynchronous event manager thread.
girder-dev-1  | [29/Mar/2024:13:35:24] ENGINE Bus EXITED
girder-dev-1  | [29/Mar/2024:13:35:24] ENGINE Waiting for child threads to terminate...
girder-dev-1  | [29/Mar/2024:13:35:24] ENGINE Re-spawning /usr/local/bin/python3.8 /usr/local/bin/girder serve --dev



girder-dev-1  | Running in mode: development
girder-dev-1  | Connecting to MongoDB: mongodb://mongodb:27017/girder
girder-dev-1  | Loaded plugin "advanced_management"
girder-dev-1  | WARNING: No description docs present for route PUT collection/:/i/d
girder-dev-1  | WARNING: No access level specified for route PUT collection/:/i/d
girder-dev-1  | Loaded plugin "aimodules_ohif"
girder-dev-1  | Loaded plugin "auth_ohif"
girder-dev-1  | Loaded plugin "resources_ohif"
girder-dev-1  | /usr/local/lib/python3.8/site-packages/pymongo/collection.py:1696: UserWarning: use an explicit session with no_cursor_timeout=True otherwise the cursor may still timeout after 30 minutes, for more info see https:
//docs.mongodb.com/v4.4/reference/method/cursor.noCursorTimeout/#session-idle-timeout-overrides-nocursortimeout
girder-dev-1  |   return Cursor(self, *args, **kwargs)
girder-dev-1  | [29/Mar/2024:13:46:29] ENGINE Listening for SIGTERM.
girder-dev-1  | [29/Mar/2024:13:46:29] ENGINE Listening for SIGHUP.
girder-dev-1  | [29/Mar/2024:13:46:29] ENGINE Listening for SIGUSR1.
girder-dev-1  | [29/Mar/2024:13:46:29] ENGINE Bus STARTING
girder-dev-1  | Started asynchronous event manager thread.
girder-dev-1  | [29/Mar/2024:13:46:29] ENGINE Started monitor thread 'Autoreloader'.
girder-dev-1  | [29/Mar/2024:13:46:29] ENGINE Serving on http://0.0.0.0:8080
girder-dev-1  | [29/Mar/2024:13:46:29] ENGINE Bus STARTED

[zachmullen]

That's definitely odd! I cannot explain it. I've tested on the version 5 integration branch (which is inaptly named v4-integration) and it restarts in about 1-2 seconds in docker-compose.

[Billuc]

Alright, I just tried to use the code on branch v4-integration and when I modify the code of my plugin, I still have to wait quite some time... I guess the problem is not directly linked to girder then.

BTW, I used an image that was very similar to yours, so the file system should be pretty much the same !

Here is the Dockerfile :

FROM ubuntu:22.04 AS base

LABEL maintainer="Kitware, Inc. <kitware@kitware.com>"

ENV DEBIAN_FRONTEND=noninteractive \
    LANG=en_US.UTF-8 \
    LC_ALL=C.UTF-8

RUN apt-get update && apt-get install -qy \
    gcc \
    libpython3-dev \
    git \
    libldap2-dev \
    libsasl2-dev \
    python3-pip \
    curl \
&& apt-get clean && rm -rf /var/lib/apt/lists/* \
&& python3 -m pip install --upgrade --no-cache-dir \
    pip \
    setuptools \
    setuptools_scm \
    wheel

RUN curl -LJ https://github.com/krallin/tini/releases/download/v0.19.0/tini -o /sbin/tini && \
    chmod +x /sbin/tini

RUN curl -sL https://deb.nodesource.com/setup_18.x | bash - && \
    apt-get install -qy nodejs

RUN mkdir /girder
# COPY . /girder/
RUN git clone -b v4-integration https://github.com/girder/girder /girder

RUN cd /girder/girder/web && npm i && npm run build

RUN pip install -e /girder

EXPOSE 8080

ENTRYPOINT ["/sbin/tini", "--", "girder", "serve"]

FROM base AS custom

ENV GIRDER_CONFIG /girder/config/girder.cfg
COPY girder.dev.cfg ${GIRDER_CONFIG}
ENV GIRDER_TEST_DB mongodb://mongodb:27017/girder
ENV GIRDER_HOST 0.0.0.0
ENV GIRDER_PORT 8080

COPY plugin /girder/plugin
RUN pip install -e /girder/plugin

ENTRYPOINT ["/sbin/tini", "--", "girder", "serve"]

[Billuc]

Hi,

I don't know if this is relevant, but cpu usage by girder in my Docker is 100% during the 10ish minutes between reloads.
I guess something is using a lot of resources during shutdown/startup (maybe ?)

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                      
      7 root      20   0 7081588  67840  16320 R 100.3   0.2   2:29.93 girder                                                                                                                                                       
      1 root      20   0    2780   1120   1120 S   0.0   0.0   0:00.02 tini                                                                                                                                                         
    133 root      20   0    7752   3520   3040 R   0.0   0.0   0:00.01 top