What is the current state of GitLab Continuous Integration testing in Godot 4?

[SdgGames]

I was curious about the state of Docker-CI. I am using the Barichello-CI docker image, which is specifically mentioned in the docs.

I tried running the scripts from the faq, but I got an error regarding DisplayServerX11.

That's fine, I just want to run unit tests against my functions in my scripts, I'm not actually testing anything within a scene. I tried adding the --headless option to bypass the display errors, but I got the error:

    Headless mode is not supported!
    
    Tests that use UI interaction do not work in headless mode because 'InputEvents' are not transported by the Godot engine and thus have no effect!

This seems like a very intentional failure mode, and it sounds like the problem is with Godot itself, but I was wondering if there are any possible workarounds. Again, I just want to write unit tests against my own code, I'm not interested in integration tests or testing in-scene behaviors like pathfinding, etc. I am writing functions like add(a, b): return a + b (simplified example), and want to make sure that the functions keep returning the expected value as my addon grows.

If Godot is just broken for CI, then is there a bug report that I can go throw my support behind? I'd love to be able to use this tool as a part of my CICD pipeline, but I haven't gotten it to work yet.

[MikeSchulze]

Hi, CI is fully functional, look here to see my workflows, your problem is the Docker image you are using.

libX11.so.6: cannot open shared object file: No such file or directory
ERROR: Can't load Xlib dynamically.
   at: DisplayServerX11 (platform/linuxbsd/x11/display_server_x11.cpp:5725)
ERROR: Unable to create DisplayServer, all display drivers failed.
   at: setup2 (main/main.cpp:2394)
Your video card drivers seem not to support the required Vulkan version.

Means you need to install in addition the missing X11 libs.

I personally do not use the docker image anymore, it is just simpler to download and cache the specific Godot version instead of downloading and running the full image.
See https://github.com/MikeSchulze/gdUnit4/blob/master/.github/actions/godot-install/action.yml

At least, you need to run the test script it with a display server.
xvfb-run --auto-servernum ./addons/gdUnit4/runtest.sh --add ${{ inputs.test-includes }} --audio-driver Dummy --display-driver x11 --rendering-driver opengl3 --screen 0

And it is also important to update the project about the Godot cache issues

    - name: "Update Project"
        if: ${{ !cancelled() }}
        timeout-minutes: 1
        continue-on-error: true # we still ignore the timeout, the script is not quit and we run into a timeout
        shell: bash
        run: |
          xvfb-run --auto-servernum ~/godot-linux/godot -e --path . -s res://addons/gdUnit4/bin/ProjectScanner.gd --headless --audio-driver Dummy

From the documentation, it needs to update the GitLab CI example. I have no GitLab account to verify, it would be nice when you adapt the GitHub CI and give me a link to it.

[SdgGames]

Thanks for the help! I ended up just making my own Docker images instead of modifying the Barichello one. Ultimately, custom images made more sense for me since building code is a constant, but updating versions is fairly rare. If it's useful, here's my Dockerfile.
I was able to get unit tests working for my project template. Not sure if it's useful to anyone else, but I thought I'd link it in case it can help with the documentation.

[MikeSchulze]

If you're ok, then I can link the project template to my GitLab CI documentation

[SdgGames]

Feel free!

[bugbountyguy]

Hi, I was wondering if I could followup with this, and hope someone else came up with a working solution. I've been banging my head against the wall for a couple of days trying to get unit tests to work in a pipeline flow for a new project, using GitLab, and have had zero luck with it as well. I started w/ the error mentioned above, and after trying to follow the same flow provided in the above github links, I run into other issues (using ubuntu:22.04 docker image).

To start with, here's my current (WIP) gitlab-ci.yml script. Note that I had to add a bunch of libx libraries to get rid of a bunch of errors related to some libraries apparently missing:

  script:
    - |
      apt-get update
      apt-get install -y xvfb fontconfig libxcursor1 libxkbcommon-dev libxinerama-dev libxrandr-dev libxi-dev xorg-dev libglu1-mesa libgl1-mesa-dev libdbus-1-3 libxinerama1 libxcursor1
      export GODOT_BIN="${PWD}/godot/godot"
      chmod u+x ./addons/gdUnit4/runtest.sh
      xvfb-run --auto-servernum ./addons/gdUnit4/runtest.sh -a test --audio-driver Dummy --display-driver x11 --rendering-driver opengl3 --screen 0 --continue --verbose

Depending on the Godot version I run, I get one of two problem outputs.

Running with v4.2.1 results in the following:

Godot Engine v4.2.1.stable.official.b09f793f5 - https://godotengine.org/
TextServer: Added interface "Dummy"
TextServer: Added interface "ICU / HarfBuzz / Graphite (Built-in)"
Xshape 1.1 detected.
Xinerama 1.1 detected.
Xrandr 1.6 detected.
Xrender 0.11 detected.
Xinput 2.2 detected.
XInput: Refreshing devices.
XInput: No touch devices found.
Detecting GPUs, set DRI_PRIME in the environment to override GPU detection logic.
Only one GPU found, using default.
WARNING: Could not set V-Sync mode. V-Sync is not supported.
     at: set_use_vsync (platform/linuxbsd/x11/gl_manager_x11.cpp:374)
XcursorGetTheme could not get cursor theme
Failed loading custom cursor: left_ptr
Failed loading custom cursor: xterm
Failed loading custom cursor: hand2
Failed loading custom cursor: cross
Failed loading custom cursor: watch
Failed loading custom cursor: left_ptr_watch
Failed loading custom cursor: fleur
Failed loading custom cursor: dnd-move
Failed loading custom cursor: crossed_circle
Failed loading custom cursor: v_double_arrow
Failed loading custom cursor: h_double_arrow
Failed loading custom cursor: size_bdiag
Failed loading custom cursor: size_fdiag
Failed loading custom cursor: move
Failed loading custom cursor: row_resize
Failed loading custom cursor: col_resize
Failed loading custom cursor: question_arrow
ScreenSaver: DBus 1.12.20 detected.
PortalDesktop: DBus 1.12.20 detected.
Using "default" pen tablet driver...
Shader 'CanvasSdfShaderGLES3' SHA256: bcece59dd27a5c049578235110c0ceb0c5981d5d5c9b8a5d0941f3cf737fdc4e
Shader 'SkeletonShaderGLES3' SHA256: e6a94dca2424810047fe171f750e7069b7b10ad6f4bd296b0aea70c19b6ff340
Shader 'ParticlesShaderGLES3' SHA256: 2971449e7281485de5b3d0dee82dc0524293abd890386e482af2cdcfe94da050
Shader 'ParticlesCopyShaderGLES3' SHA256: 90c293abbec45a67826d18bb41d9a495cfdfeb7c8530d6a67f8e60500e5e9918
Shader 'CopyShaderGLES3' SHA256: 6c9f817005db27c5f88b3cff91c4e2a27c9145dd0778112aafc471b9d58ef122
Shader 'CanvasShaderGLES3' SHA256: 76c9eeca46da51ef56f08c625db269fe1d7f02d8c0ef3452e2fa42b84428794e
Shader 'CanvasOcclusionShaderGLES3' SHA256: 4dcdf74c9a9ba68de8dd7927a38fe9dbdf2a8d76fe65f0be1f0cfc97cb58784a
Shader 'SceneShaderGLES3' SHA256: 6afb291b5b95929bb2fd5f83153452d8863147a4411b4a2a17ab838e0bd1d4d2
Shader 'SkyShaderGLES3' SHA256: 1bcf565b2429112955f36c5799baee51a7839375ba963e588e02c8efa17db041
Shader 'CubemapFilterShaderGLES3' SHA256: 40e4c6b79ba58858af592fded5489803ed092e8d8bfba9890f82627bafa6bd8b
OpenGL API 4.5 (Core Profile) Mesa 23.0.4-0ubuntu1~22.04.1 - Compatibility - Using Device: Mesa - llvmpipe (LLVM 15.0.7, 256 bits)
JoypadLinux: udev enabled and loaded successfully.
 
TextServer: Primary interface set to: "ICU / HarfBuzz / Graphite (Built-in)".
CORE API HASH: 915626997
EDITOR API HASH: 1313472085
Debugger Break, Reason: 'Parser Error: Could not find type "GdUnitHtmlReport" in the current scope.'
*Frame 0 - res://./addons/gdUnit4/bin/GdUnitCmdTool.gd:27 in function ''
Enter "help" for assistance.
debug> 
================================================================
handle_crash: Program crashed with signal 11
Engine version: Godot Engine v4.2.1.stable.official (b09f793f564a6c95dc76acc654b390e68441bd01)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
ERROR: FATAL: Index p_index = 1 is out of bounds (size() = 0).
   at: get (./core/templates/cowdata.h:158)
Illegal instruction
WARNING: Could not set V-Sync mode. V-Sync is not supported.
     at: set_use_vsync (platform/linuxbsd/x11/gl_manager_x11.cpp:374)
================================================================
handle_crash: Program crashed with signal 11
Engine version: Godot Engine v4.2.1.stable.official (b09f793f564a6c95dc76acc654b390e68441bd01)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
ERROR: FATAL: Index p_index = 1 is out of bounds (size() = 0).
   at: get (./core/templates/cowdata.h:158)
Illegal instruction

Running the older v4.0.3 results in the following error looping indefinitely (have to force kill the job), which interestingly enough shows in the 4.2.1 release but only once - not forever:

Debugger Break, Reason: 'Parser Error: Could not find type "GdUnitHtmlReport" in the current scope.'
*Frame 0 - res://./addons/gdUnit4/bin/GdUnitCmdTool.gd:27 in function ''
Enter "help" for assistance.
debug> 

I'm running out of ideas on how to get this to work.... Would love any assistsance anyone may have on this, and whatever the solution is, should probably be updated onto the docs.

[bugbountyguy]

Ugh, I guess it was a rubber ducky situation. Spend forever on it, finally get a fix shortly after asking for help. For anyone who wants to get this thing working in their gitlab-ci, here's my current yaml which appears to be working. 1000% open to cleanup suggestions (I know - godot install is during import-assets...), especially since it's kind of slow as it's pulling software down (couldn't find an existing ubuntu docker image that had all these libraries/tools already built). I went with using both ubuntu:22.04 and godot-ci:4.2.1 in my case due to the export templates, but wouldn't mind hearing of a solution to make it a global ubuntu:22.04 image instead so I only have to change one variable.

Future plans would be to integrate this into gitlab pages and junit (or just show the html page that's already given as a result). Would love to know how to get code coverage reports... Is that possible with gdunit4? Haven't looked into that much yet for it :)

Anyway here it is:

stages:
  - import-assets
  - build
  - tests

variables:
  EXPORT_NAME: TestProject
  GODOT_VERSION: 4.2.1

# Cache imported assets between runs
cache:
  key: import-assets
  paths:
      - .godot/imported/
      - godot/

# Things each job should have
.common:
  tags:
    - docker

import-assets:
  extends: .common
  image: ubuntu:22.04
  stage: import-assets
  script:
    - |
      apt-get update
      apt-get install -y wget unzip
      filename=Godot_v${GODOT_VERSION}-stable_linux.x86_64.zip
      wget https://github.com/godotengine/godot-builds/releases/download/${GODOT_VERSION}-stable/${filename}
      unzip "$filename" -d godot
      rm "$filename"
      mv godot/Godot_v* godot/godot
      chmod u+x godot/godot
      ./godot/godot --headless --verbose --editor --quit

linux:
  extends: .common
  image: barichello/godot-ci:4.2.1
  only:
    - tags
  stage: build
  script:
    - |
      apt-get update
      apt-get install -y fontconfig
      mkdir -v -p build/linux
      godot --headless --verbose --export-release "Linux/X11" build/linux/$EXPORT_NAME.x86_64
  artifacts:
    name: $EXPORT_NAME-$CI_JOB_NAME
    paths:
      - build/linux/

windows:
  extends: .common
  image: barichello/godot-ci:4.2.1
  only:
    - tags
  stage: build
  script:
    - mkdir -v -p build/windows
    - godot --headless --verbose --export-release "Windows Desktop" build/windows/$EXPORT_NAME.exe
  artifacts:
    name: $EXPORT_NAME-$CI_JOB_NAME
    paths:
      - build/windows

mac:
  extends: .common
  image: barichello/godot-ci:4.2.1
  only:
    - tags
  stage: build
  script:
    - mkdir -v -p build/mac
    - godot --headless --verbose --export-release "Mac OSX" build/mac/$EXPORT_NAME.zip
  artifacts:
    name: $EXPORT_NAME-$CI_JOB_NAME
    paths:
      - build/mac

gdunit4:
  extends: .common
  image: ubuntu:22.04
  stage: tests
  dependencies:
    - linux
  script:
    - |
      apt-get update
      apt-get install -y xvfb fontconfig libxcursor1 libxkbcommon-dev libxrandr-dev libxi-dev xorg-dev libglu1-mesa libgl1-mesa-dev libdbus-1-3 libxinerama1 libxcursor1
      export GODOT_BIN="${PWD}/godot/godot"
      chmod u+x ./addons/gdUnit4/runtest.sh
      ./godot/godot -e --path . -s res://addons/gdUnit4/bin/ProjectScanner.gd --headless --audio-driver Dummy
      xvfb-run --auto-servernum ./addons/gdUnit4/runtest.sh -a test --audio-driver Dummy --display-driver x11 --rendering-driver opengl3 --screen 0 --continue --verbose
  artifacts:
    when: always
    reports:
      junit: ./reports/report_1/results.xml

[MikeSchulze]

Hi, i just wounder why you need to install in addition the barichello/godot-ci?

My workflows just do

• run on ubuntu-22.04
• no need to install additional libraries
• download and cache and install a specific Godot version -> https://github.com/MikeSchulze/gdUnit4/blob/master/.github/actions/godot-install/action.yml
• using xvfb-run to run the display server and
  • run the ProjectScanner.gd script
  • run the runtest.sh script
• run action dorny/test-reporter@v1.6.0 to collect test reports -> https://github.com/MikeSchulze/gdUnit4/blob/master/.github/actions/publish-test-report/action.yml
• and upload the test artifacts -> https://github.com/MikeSchulze/gdUnit4/blob/master/.github/actions/upload-test-report/action.yml

[bugbountyguy]

The godot-ci was only for the builds if I wanted them (in my case, only for tagged builds with version numbers). I didn't need it for the unit tests or cache. For the test stages, The ubuntu:22.04 docker image doesn't come with xvfb or any other libraries needed with it which is why I had to install those on it. I'm guessing whatever github uses already comes with that stuff pre-installed or something.

The reason I went with just using godot-ci was because the following error shows up in the pipeline when attempting to create a build with just the regular linux binary:

ERROR: Cannot export project with preset "Linux/X11" due to configuration errors:
No export template found at the expected path:
/root/.local/share/godot/export_templates/4.2.1.stable/linux_debug.x86_64
No export template found at the expected path:
/root/.local/share/godot/export_templates/4.2.1.stable/linux_release.x86_64
   at: _fs_changed (editor/editor_node.cpp:978)
ERROR: Project export for preset "Linux/X11" failed.
   at: _fs_changed (editor/editor_node.cpp:994)

The godot-ci already has these installed, so I was able to just grab them that way to have a successful build.