Skip to content

mbland/tomcat-servlet-testing-example

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

284 Commits

.github

.github

 
 

bin

bin

 
 

dockerfiles

dockerfiles

 
 

gradle

gradle

 
 

strcalc

strcalc

 
 

.gitattributes

.gitattributes

 
 

.gitignore

.gitignore

 
 

LICENSE.txt

LICENSE.txt

 
 

README.md

README.md

 
 

gradle.properties

gradle.properties

 
 

gradlew

gradlew

 
 

gradlew.bat

gradlew.bat

 
 

settings.gradle.kts

settings.gradle.kts

 
 

Repository files navigation

Tomcat Servlet Testing Example

My attempt at an outside-in testing example in Java using a Tomcat servlet container.

Source: https://github.com/mbland/tomcat-servlet-testing-example

License CI status Test results Coverage Status

This project incorporates the String Calculator kata to demonstrate Test-Driven Development in the context of a Test Pyramid-based testing strategy.

Though I've been a programmer for years across many other programming languages, I'm learning a lot about the Java ecosystem for the first time. It will take some time before this project looks like a complete working example that makes sense.

The plan is to develop an exercise comprised of the following steps:

  • Set up a continuous integration pipeline based on GitHub Actions.
  • Creating a walking skeleton implementation and adding a large end-to-end test to validate it, likely using Selenium WebDriver as well as headless Chrome.
  • Developing the String Calculator using TDD and small unit tests.
  • Adding a medium integration test to ensure the Servlet passes parameters to the internal String Calculator logic and passes back the results.
  • Adding tests for frontend JavaScript components.
  • Using test doubles in unit tests. This may involve extending the String Calculator example or adding a completely different one, possibly based on Apache Solr.

Status

I've got an initial Hello, World! servlet running under Tomcat, validated by straightforward medium tests and an initial Selenium WebDriver test, all running under JUnit. Everything runs from the command line, IntelliJ IDEA, and VSCode (all on macOS), and in the GitHub Actions continuous integration pipeline.

The next step is to add a proper HTML <form> and WebDriver test to complete the walking skeleton (i.e., a complete, minimally functional application deployment).

OS Compatibility

I run Arm64/aarch64 builds of Ubuntu Linux and Windows 11 Professional under Parallels Desktop for Mac on Apple Silicon. vite.config.js contains a workaround to allow WebDriver to use Chromium or Firefox on Linux, as no aarch64 build of Google Chrome is available. (I hope to contribute this workaround upstream, at which point I'll remove it from vite.config.js.)

Also, it doesn't appear as though nested virtualzation will ever be supported by the aarch 64 Windows 11 on an Apple M1. This means the Docker-based tests won't work in this situation.

FWIW, Windows Subsystem for Linux 2 also requires nested virtualization. WSL 1 will work fine, but it appears prospects to run Docker in WSL 1 are rather dim.

Development environment setup

Install the Java Development Kit

This project uses the Java® Platform, Standard Edition & Java Development Kit Version 21.

On macOS and Linux, I installed the latest JDK 21.0.1 from Eclipse Temurin™ Latest Releases via SDKMAN!. On Windows, I downloaded it from the Download the Microsoft Build of OpenJDK page.

Setup the frontend JavaScript environment

See strcalc/src/main/frontend/README.md for guidance on setting up the frontend development and build environment.

The frontend environment isn't required for ./gradlew test, which runs the small Java unit tests from the exercise. Conversely, the Java environment isn't strictly required to develop, build, and test the frontend in isolation.

However, you will need to have both environments installed to build most of the Gradle targets. This includes the medium integration and large system tests in strcalc/src/test/java, which also depend on the frontend build.

Optional: Install the Tomcat servlet container

This step is optional, as the bin/tomcat-docker.sh script will run Tomcat locally in a Docker container defined by dockerfiles/Dockerfile.tomcat-test.

I followed the Tomcat 10.1 Setup instructions to install Tomcat locally at /opt/tomcat/apache-tomcat-10.1.16. I created bin/tomcat.sh as a thin wrapper around Tomcat's bin/catalina.sh that detects and sets JAVA_HOME and sets CATALINA_HOME. Verified that it was installed correctly via bin/tomcat.sh start and visiting http://localhost:8080/.

Optional: Configure Tomcat to emit HTTP access logs to standard output

This step is optional, as the Docker container launched by bin/tomcat-docker.sh is already configured to do this by running bin/update-tomcat-config-logging.sh.

By default, Tomcat emits its HTTP access logs to $CATALINA_HOME/logs/localhost_access_log.YYYY-MM-DD.txt. This is configured by the following block in $CATALINA_HOME/conf/server.xml:

        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html
             Note: The pattern used is equivalent to using pattern="common" -->
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log" suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />

When running and experimenting locally, it can be helpful and convenient to emit these logs directly to the terminal instead. To emit the HTTP access logs to standard output instead, edit or replace this <Valve> element with the following:

        <Valve className="org.apache.catalina.valves.AccessLogValve"
               directory="/dev" prefix="stdout"
               suffix="" rotatable="false" buffered="false"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />

(The above is based on the answer to "Stack Overflow: Docker, Tomee, Logging, STDOUT, AWS".)

Alternatively, if you're into the YOLO thing, apply this update by running:

bin/update-tomcat-config-logging.sh $CATALINA_HOME/conf/server.xml

For more information on Tomcat access logging, see:

Optional: Install Gradle

This step is optional, as the gradlew and gradlew.bat wrapper scripts in the root directory of the repository will install Gradle.

On macOS and Linux, I installed Gradle 8.4 via SDKMAN!. On Windows, I followed the Gradle manual installation steps.

(When I started the project, this also required installing the 20.0.2 JDK, since Gradle 8.4 could build Java 21, but needed Java 20 to run. At some point this no longer appeared necessary, though the Gradle compatibility matrix for Java has yet to reflect this. The project now uses Gradle 8.5, which is explicitly supported on Java 21, but it's still important to know about this potential mismatch.)

Create a Java project with Gradle

After running gradle init to create a library as part of the Gradle Tutorial:

Install IntelliJ IDEA and/or Visual Studio Code

The project supports both IntelliJ IDEA and VSCode.

I recently used VSCode extensively to develop my EListMan project in Go, but this being Java, I primarly use IntelliJ IDEA. (I still prefer to edit this file and other Markdown files in VSCode, however.)

VSCode Java configuration

I installed all the extensions from the VSCode: Extension Pack for Java except for Visual Studio IntelliCode.

A lot of VSCode references refer to Java: Configure Java Runtime in the Command Palette, but I can't find it across macOS, Linux, or Windows. I can't seem to find any online resources from anyone else for whom it also doesn't exist.

I did eventually find Managing Java Projects in VS Code: Configure Runtime for Projects, which provides guidance on how to configure Java via settings.json. On macOS and Linux, my config looks like the following (replacing $HOME with my actual home directory):

    "java.configuration.runtimes": [
        {
          "name": "JavaSE-21",
          "path": "$HOME/.sdkman/candidates/java/21.0.1-tem",
          "sources" : "$HOME/.sdkman/candidates/java/21.0.1-tem/lib/src.zip",
          "javadoc" : "https://docs.oracle.com/en/java/javase/21/docs/api",
          "default":  true
        }
    ],

and on Windows:

    "java.configuration.runtimes": [
        {
            "name": "JavaSE-21",
            "path": "C:\\Program Files (Arm)\\Microsoft\\jdk-21.0.1.12-hotspot",
            "sources" : "C:\\Program Files (Arm)\\Microsoft\\jdk-21.0.1.12-hotspot\\lib\\src.zip",
            "javadoc" : "https://docs.oracle.com/en/java/javase/21/docs/api",
            "default":  true
          }
    ]

Additional VSCode extensions and references

Running Tomcat and adding a Tomcat test helper

Optional: Create Tomcat > Local run configuration

This step is optional, as the bin/tomcat-docker.sh script will run Tomcat locally in a Docker container defined by dockerfiles/Dockerfile.tomcat-test.

In IntelliJ, you can run your locally installed Tomcat server via Run > Edit Configurations... > Add New Configuration (+) > Tomcat Server > Local.

  • Under Server > Before launch, remove the "Build" item and add a new "Build Artifacts" task.
  • From the popup window for that operation, select: Build 'Gradle : tomcat-servlet-testing-example : strcalc.war' artifact.

This installs the WAR file (as described by the Tomcat deployment document) under the /strcalc context (i.e., Tomcat serves the servlet at http://localhost:8080/strcalc).

Write "Hello, World!" servlet and JUnit test

The initial servlet responds with the classic hardcoded value Hello, World!. The initial test expects Tomcat to be running and for http://localhost:8080/strcalc to return this value.

Note that the value of the @WebServlet annotation is /add. Tomcat appends this value to the context path (/strcalc), creating the servlet path /strcalc/add.

Add a bin/tomcat-docker.sh script to launch the Tomcat Docker image

This script can be run manually.

Add the LocalServer test helper

The full path is com.mike_bland.training.testing.utils.LocalServer.

This class runs git and docker commands to emulate the bin/tomcat-docker.sh script on demand for StringCalculatorTomcatContractTest. A key difference is that LocalServer will allocate a unique port for every test run, so that it won't conflict with an existing local instance.

Partitioning tests into small, medium, and large test sizes

Add the @SmallTest, @MediumCoverageTest, @MediumTest, and @LargeTest annotations

These are JUnit composed annotations based on the guidance from:

Add test-medium-coverage, test-medium, test-large, test-all tasks, update test and check

These tasks and updates use JUnit composed annotations and the includeTags config option based on guidance from:

Note that with no @SmallTest methods defined, running the test task produces the following warning:

$ ./gradlew test --warning-mode all --rerun-tasks

[...snip...]

> Task :strcalc:test
No test executed. This behavior has been deprecated. This will fail with an
error in Gradle 9.0. There are test sources present but no test was executed.
Please check your test configuration. Consult the upgrading guide for further
information:
https://docs.gradle.org/8.4/userguide/upgrading_version_8.html#test_task_fail_on_no_test_executed

[...snip...]

The same is true for test-medium and test-large when no @MediumTest or @LargeTest methods are present. The SmallPlaceholderTest and LargePlaceholderTest classes exist to silence this warning until actual @SmallTest and @LargeTest methods appear.

Use IntelliJ IDEA Gradle integration to define run configurations for test targets

After syncing the Gradle project, you can use IntelliJ IDEA's Gradle integration to convert the Gradle test tasks from strcalc/build.gradle.kts directly into IntelliJ run configurations:

  • Navigate to View > Tool Windows > Gradle or click the Gradle elephant logo in the right side toolbar to open the Gradle tool window.
  • Navigate through the Gradle project hierarchy and select the target you'd like to turn into a run configuration, e.g., test.
  • Double click the target or hit enter/return to create and run the run configuration for that target.
  • Open the run configuration and modify it as desired.

Benefits

Doing this instead of creating a typical JUnit run configuration ensures that IntelliJ runs the same tests the same way as the gradlew command. This, in turn, helps ensure that IntelliJ runs the same tests the same way as they're run in continuous integration. This includes running with the same code coverage configuration and output. See Setting up continuous integration below.

You can still create JUnit run configurations, but these gradlew targets already ensure that all dependencies of the task are built first. This is particularly helpful when it comes to medium and large tests.

  • Specifically for this project, the medium and large tests depend on the strcalc.war artifact, used to build a temporary Tomcat Docker image for testing. The Gradle test tasks are already configured to depend on the war task.

  • Setting this up in a JUnit run configuration would require setting up Modify options (⌥M) > Before Launch: Add before launch task > Run Gradle task.

Code coverage is always generated for these targets (though not automatically displayed) when using the regular Run command. To display this output, see Showing JaCoCo coverage generated by Gradle tasks.

Drawbacks

Viewing the coverage generated by the Gradle test task in IntelliJ may or may not work. See the Viewing code coverage in IntelliJ IDEA section below.

Adding the /strcalc landing page

Add src/main/webapp/index.html and Hamcrest matchers

Per the Gradle WAR plugin, files in src/main/webapp are copied to the root of the servlet WAR file. Adding the src/main/webapp/index.html landing page necessitated two things:

  • Updating the @WebServlet annotation on StringCalculatorServlet from "" to "/add". This enables Tomcat to serve index.html as /strcalc/, and the servlet now serves /strcalc/add/.
  • Updating the helloWorldPlaceholder test method to landingPageHelloWorld. This test now checks that the response is of Content-Type text/html instead of text/plain, and that it contains "Hello, World!", not that it matches exactly.

Changing the test assertion to check that the response body contains "Hello, World!" necessitated using the Hamcrest matcher library. We could've used:

assertTrue(resp.body().contains("Hello, World!"));

But when it fails, we get very little information about the problem:

Expected :true
Actual   :false

Using the Hamcrest matcher:

assertThat(resp.body(), containsString("Hello, World!"));

we're able to see more information:

java.lang.AssertionError:
Expected: a string containing "Hello, World!"
     but: was "<DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>String Calculator - Mike Bland Training</title>
</head>
<body>
<p>Hello, world!</p>
</body>
</html>
"

Setting up continuous integration using GitHub Actions

Added the .github/CODEOWNERS file and .github/workflows/run-tests.yaml file for GitHub Actions. Configured using the setup-java GitHub Actions plugin and the gradle/gradle-build-action GitHub Actions plugin.

Publishing JUnit test results

run-tests.yaml runs gradlew merge-test-reports to aggregate individual TEST-*.xml JUnit results files into TESTS-TestSuites.xml files. This task uses org.apache.ant:ant-junit based on advice from Merging Per-Suite JUnit Reports into Single File with Gradle + Kotlin. (See also: Using Ant from Gradle.)

The aggregated files are uploaded via the actions/upload-artifact GitHub Actions plugin. .github/workflows/publish-test-results.yaml then uses the dorny/test-reporter GitHub Actions plugin. That plugin downloads the test results and makes them available via the status icon next to the commit hash in the GitHub UI.

As explained by the dorny/test-reporter page, the separate publish-test-results.yaml file is necessary to ensure the plugin has permission to launch a check run:

When someone pushes code to the repository, GitHub automatically sends the check_suite event with an action of requested to all GitHub Apps installed on the repository that have the checks: write permission.

Not having checks:write permission for the plugin can lead to the cryptic error:

Error: HttpError: Resource not accessible by integration

For more on GITHUB_TOKEN permissions, see Assigning permissions to jobs.

Setting up code coverage

Code coverage shows which lines of code are executed by a test or test suite and which aren't. It is one common and important component of a suite of Vital Signs to ensure high code and software quality.

JaCoCo (i.e., Java Code Coverage) is a popular framework for collecting code coverage in Java. IntelliJ IDEA can display code coverage statistics and display which lines are covered, partially covered, and uncovered. It can generate and consume JaCoCo coverage in addition to its own builtin coverage implementation.

Code coverage guidelines

  • Code coverage does not indicate that you have enough tests, or that your tests are any good, or they're validating what you expect they are. It can definitively tell whether specific code is exercised by a test (or any test at all), which could reveal gaps in your test suite. Used skillfully, it can be of great use in the process of refactoring, making small changes that improve code quality while preserving behavior. (Using code coverage in this way requires a set of techniques that I hope to cover in their own separate training module(s).)

  • The ideal is to have high code coverage from your entire small (or small-ish) test suite. Each test should cover relatively little, very specific areas of the code. Larger tests should not generate code coverage, as such tests by definition will execute broader areas of the code, diluting the value of the measurements.

    The way to think about this is to think of the information available when a smaller test fails versus a larger test:

    • When a smaller test fails, the potential area of the code responsible for the failure is fairly narrow.

    • When a larger test fails, the potential area of the code responsible for the failure is potentially the entire application.

  • Also consider the time required to run them: Smaller tests run multiple orders of magnitude faster than larger tests. Having high code coverage from a suite of highly targeted smaller tests that you can run frequently while developing enables tighter feedback and recovery loops. This translates into higher confidence and faster velocity.

    On the contrary, larger tests have a broader scope, consume more resources, and run more slowly. While still critically important, these qualities make running larger tests frequently while coding impractical. They should be used sparingly to validate higher level systemic properties, as using them to validate lower level details renders them brittle.

    Therefore, code coverage from larger tests does practically nothing to tighten feedback loops and to help resolve coding errors quickly, rendering it far less useful.

  • That said, if you do choose to collect code coverage from all tests, take care to partition the reports by test size. Coverage from the larger test suites should be less than or equal to that from the smaller test suite—not the other way around!

    Relying on code coverage percentages inflated by coverage from large tests risks instilling a false sense of security. It could potentially mask gaps in smaller test coverage that could in turn mask serious, yet preventable problems.

Enabling the Gradle JaCoCo Plugin and generating HTML and XML reports

Adding the Gradle JaCoCo Plugin to the strcalc/build.gradle.kts file was straightforward. With the plugin enabled, the test task automatically generates JaCoCo coverage data at strcalc/build/jacoco/test.exec. Then the jacocoTestReport task creates an HTML report viewable via:

open -a safari strcalc/build/reports/jacoco/test/html/index.html

The jacocoXmlTestReport task generates an XML report which our GitHub Actions continuous integration system will publish. For details on publishing options, see:

Viewing code coverage in IntelliJ IDEA

The options for viewing code coverage directly in IntelliJ IDEA include:

  • Using a run configuration to generate and display code coverage
  • Showing JaCoCo coverage generated by Gradle tasks

For information on IntelliJ's code coverage options and operations beyond what's described below, see IntelliJ Idea: Code coverage.

Using a run configuration to generate and display code coverage

IntelliJ allows you to configure specific run configurations to generate and display code coverage via Run > Run CONFIGURATION_NAME with Coverage. It allows you to use wither JaCoCo or IntelliJ's own builtin code coverage implementation. You can configure the Gradle run configurations or any JUnit run configuration you define yourself.

For instructions, see IntelliJ: Run with coverage.

Showing JaCoCo coverage generated by Gradle tasks

The JaCoCo coverage results automatically generated by the Gradle test tasks won't automatically display in IntelliJ, but you can import them. The IntelliJ IDEA: Manage coverage suites documentation covers most of the process, but specifically for this project:

  • Open Run > Show Coverage Data... (⌥⌘F6) to open the Choose Coverage Suite to Display window.
  • Click the + symbol to open the file picker.
  • Select strcalc/build/jacoco/test.exec and click the Open button.
  • Make sure the JaCoCo Coverage > test.exec item is checked and click the Show Selected button.

Code coverage markers should now appear in the left gutter of the code editor of each src/main/java file (green for covered, red for uncovered).

IntelliJ IDEA code coverage display details

Here are a few fine details to be aware of:

  • The coverage percentage reported from IntelliJ's builtin coverage reporter will differ from JaCoCo's slightly, as documented in:

    As mentioned in that answer, it's not that one version is wrong and the other is right. Differences in implementation account for differences in reported percentages, particularly when it comes to differences in counting implicitly generated methods. Both will still show which code is or isn't covered in every source file.

  • However, the Principle of Least Surprise could apply here. Relying on JaCoCo's results is likely to avoid misunderstandings between coverage percentages reported in IntelliJ and from command line and continuous integration runs.

  • The Run > Show Coverage Data... (⌥⌘F6) option actually allows you to show both IntelliJ and JaCoCo coverage results together, or to switch between them. It also allows you to select .exec files from different Gradle subprojects and test targets, as well as coverage from different run configurations in general.

JaCoCo and Java version compatibility between the project and IntelliJ

The JaCoCo version bundled with IntelliJ may not be the latest available. This can prevent JaCoCo coverage from appearing in IntelliJ if your project uses a newer version of Java than the builtin JaCoCo version can support. There appears to be no way for users to update the JaCoCo version bundled with IntelliJ.

If you encounter this problem, I recommend using IntelliJ's builtin in coverage in your run configurations until a new IntelliJ release resolves the issue. Just be aware, as noted above, that while the reported percentages may differ from JaCoCo, the code editor coverage markings should be largely the same.

Diagnosis

This problem will manifest by producing output similar to the following (edited for readability) when using:

  • Run > Run CONFIGURATION_NAME with Coverage: this output will appear in the output pane

  • Run > Show Coverage Data... (⌥⌘F6): this output will appear in the IntelliJ debug log. You can find this log via: Help > Show Log in Finder.

    Note that this problem will not produce a modal error window! If all the percentages in the Coverage pane show 100% (0/0), this is likely the culprit.

2023-11-05 14:47:39,944 [6679894]   INFO - #c.i.c.JaCoCoCoverageRunner -
Error while analyzing .../strcalc/build/classes/java/.../*.class
with JaCoCo 0.8.8.202204050719/5dcf34a.

java.io.IOException:
Error while analyzing .../strcalc/build/classes/java/.../*.class
with JaCoCo 0.8.8.202204050719/5dcf34a.
  at org.jacoco.core.analysis.Analyzer.analyzerError(Analyzer.java:163)
  at org.jacoco.core.analysis.Analyzer.analyzeClass(Analyzer.java:135)
  [ ...snip... ]

Caused by: java.lang.IllegalArgumentException:
Unsupported class file major version 65
  at org.objectweb.asm.ClassReader.<init>(ClassReader.java:199)
  at org.objectweb.asm.ClassReader.<init>(ClassReader.java:180)
  [ ...snip... ]

Publishing continuous integration coverage results to Coveralls

Coveralls is a commercial product for visualizing current code coverage and historical trends. It's free for all open source projects, and offers pricing tiers for commercial customers. See the tomcat-servlet-testing-example Coveralls report (also linked from the coverage badge at the top of this file) to see Coveralls in action.

The continuous integration system publishes JaCoCo code coverage results to Coveralls using the coverallsapp/github-action GitHub Actions plugin. Coveralls will make the coverage results for the tested commit available via the status icon next to the commit hash in the GitHub UI. It will also add comments to pull requests summarizing how the changes affect code coverage.

Alternative coverage reporting

If you prefer not to use Coveralls, you can search the GitHub Actions marketplace for JaCoCo related GitHub Actions plugins. A couple of promising ones are:

Note that, though the instructions of neither of the above plugins show it, you may need to split the configuration similarly to the dorny/test-reporter plugin:

Allow frontend dev server to access Tomcat backend

It's possible to develop the frontend code served by Vite and the backend code served by Tomcat simultaneously.

Start Tomcat

For now, this requires running a fresh build of the app in Docker or setting up your own IntelliJ IDEA Run Configuration. I hope to have another solution set up shortly that would require neither.

# Compile the current code into strcalc.war.
./gradlew build

# Serve the strcalc.war file via Docker
./bin/tomcat-docker.sh

This is enabled by the CORSFilter settings in the application's web.xml file. See the comment in that file for further references.

Start frontend dev server

cd strcalc/src/main/frontend
STRCALC_BACKEND='http://localhost:8080/strcalc/' pnpm dev

Start frontend preview server

In preview mode (pnpm build && pnpm preview), the STRCALC_BACKEND value will not propagate to the compiled bundle. However, entering the following in the browser console will enable the compiled version to communciate with the backend:

globalThis.STRCALC_BACKEND='http://localhost:8080/strcalc/'

Adding large tests

Coming soon...

TODO(mbland): Document how the following are configured:

Implementing core logic using Test Driven Development and unit tests

Coming soon...

Additional References

Copyright

© 2023 Mike Bland <mbland@acm.org> (https://mike-bland.com/)

Open Source License

This software is made available as Open Source software under the Mozilla Public License 2.0. For the text of the license, see the LICENSE.txt file. See the MPL 2.0 FAQ for a higher level explanation.

About

An outside-in testing example in Java using a Tomcat servlet container

Resources

Readme

License

MPL-2.0 license
Activity

Stars

1 star

Watchers

2 watching

Forks

0 forks
Report repository

Releases

3 tags

Packages

No packages published

Languages