Skip to content

JUnit XML file format & JUnit XML examples. Including format specification, description & conventions.

Notifications You must be signed in to change notification settings

testmoapp/junitxml

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 

Repository files navigation

Common JUnit XML Format & Examples

Summary: This project documents the common use of JUnit-style XML files by testing and CI tools, including full examples & common conventions. The goal of this project is to be the reference guide for JUnit-style XML files.

The JUnit XML file format has been made popular by the JUnit project1 and has since become the de facto standard format to exchange test results between tools. JUnit-style XML files are generated by countless testing frameworks and are processed by CI tools, build servers and test management software to read and display test results.

There is no official specification for the JUnit XML file format and various tools generate and support different flavors of this format. The goal of this project here is to document a common set of elements, attributes and conventions supported by many tools2. If you are building software that produces or consumes JUnit-style XML files, this project might be a good starting point to learn about the format.

  1. Basic JUnit XML structure
  2. Complete JUnit XML example
  3. Properties for suites & cases
    1. Properties in test output
  4. Attachments, screenshots and artifacts
    1. Attachment file properties
    2. Attachment URL properties
    3. Attachment inline properties
    4. Attachments in test output
  5. Common properties and conventions
    1. Attachments
    2. Steps
    3. Type hints
  6. About this project

Basic JUnit XML structure

The following example shows the basic structure of a JUnit-style XML file. In this example we have two root test suites (think classes/folders), one sub test suite, and a couple of test cases (test results) in each suite.

A test case is considered successful (passed) unless there is another result element underneath it. Most tools support the result elements skipped, failure and error.

This basic example is not complete as it leaves out various attributes and additional elements supported by most tools. See the next section for a full example.

Open example XML file (junit-basic.xml)

<?xml version="1.0" encoding="UTF-8"?>
<testsuites time="15.682687">
    <testsuite name="Tests.Registration" time="6.605871">
        <testcase name="testCase1" classname="Tests.Registration" time="2.113871" />
        <testcase name="testCase2" classname="Tests.Registration" time="1.051" />
        <testcase name="testCase3" classname="Tests.Registration" time="3.441" />
    </testsuite>
    <testsuite name="Tests.Authentication" time="9.076816">
        <testsuite name="Tests.Authentication.Login" time="4.356">
            <testcase name="testCase4" classname="Tests.Authentication.Login" time="2.244" />
            <testcase name="testCase5" classname="Tests.Authentication.Login" time="0.781" />
            <testcase name="testCase6" classname="Tests.Authentication.Login" time="1.331" />
        </testsuite>
        <testcase name="testCase7" classname="Tests.Authentication" time="2.508" />
        <testcase name="testCase8" classname="Tests.Authentication" time="1.230816" />
        <testcase name="testCase9" classname="Tests.Authentication" time="0.982">
            <failure message="Assertion error message" type="AssertionError">
                <!-- Call stack printed here -->
            </failure>            
        </testcase>
    </testsuite>
</testsuites>

Complete JUnit XML example

The following example lists all commonly supported elements, attributes and typical values used by tools that generate or read JUnit-style XML files. Not all elements are supported by all tools, and not all tools follow and understand the same conventions.

For example, test cases properties are not supported by all tools yet. Likewise, not all tools have options to output attachments. And different tools also expect or use different conventions for names and values.

That said, if you are building tools that work with JUnit-style XML files, it can be a good idea to try to support most or all of these common elements and attributes.

Open example XML file (junit-complete.xml)

<?xml version="1.0" encoding="UTF-8"?>

<!-- <testsuites> Usually the root element of a JUnit XML file. Some tools leave out 
the <testsuites> element if there is only a single top-level <testsuite> element (which
is then used as the root element).

name        Name of the entire test run
tests       Total number of tests in this file
failures    Total number of failed tests in this file
errors      Total number of errored tests in this file
skipped     Total number of skipped tests in this file
assertions  Total number of assertions for all tests in this file
time        Aggregated time of all tests in this file in seconds
timestamp   Date and time of when the test run was executed (in ISO 8601 format)
-->
<testsuites name="Test run" tests="8" failures="1" errors="1" skipped="1" 
    assertions="20" time="16.082687" timestamp="2021-04-02T15:48:23">
    
    <!-- <testsuite> A test suite usually represents a class, folder or group of tests.
    There can be many test suites in an XML file, and there can be test suites under other
    test suites.

    name        Name of the test suite (e.g. class name or folder name)
    tests       Total number of tests in this suite
    failures    Total number of failed tests in this suite
    errors      Total number of errored tests in this suite
    skipped     Total number of skipped tests in this suite
    assertions  Total number of assertions for all tests in this suite
    time        Aggregated time of all tests in this file in seconds
    timestamp   Date and time of when the test suite was executed (in ISO 8601 format)
    file        Source code file of this test suite
    -->
    <testsuite name="Tests.Registration" tests="8" failures="1" errors="1" skipped="1" 
        assertions="20" time="16.082687" timestamp="2021-04-02T15:48:23" 
        file="tests/registration.code">

        <!-- <properties> Test suites (and test cases, see below) can have additional 
        properties such as environment variables or version numbers. -->
        <properties>
            <!-- <property> Each property has a name and value. Some tools also support
            properties with text values instead of value attributes. -->
            <property name="version" value="1.774" />
            <property name="commit" value="ef7bebf" />
            <property name="browser" value="Google Chrome" />
            <property name="ci" value="https://github.com/actions/runs/1234" />
            <property name="config">
                Config line #1
                Config line #2
                Config line #3
            </property>
        </properties>

        <!-- <system-out> Optionally data written to standard out for the suite. 
        Also supported on a test case level, see below. -->
        <system-out>Data written to standard out.</system-out>

        <!-- <system-err> Optionally data written to standard error for the suite. 
        Also supported on a test case level, see below. -->
        <system-err>Data written to standard error.</system-err>

        <!-- <testcase> There are one or more test cases in a test suite. A test passed
        if there isn't an additional result element (skipped, failure, error).

        name        The name of this test case, often the method name
        classname   The name of the parent class/folder, often the same as the suite's name
        assertions  Number of assertions checked during test case execution
        time        Execution time of the test in seconds
        file        Source code file of this test case
        line        Source code line number of the start of this test case
        -->
        <testcase name="testCase1" classname="Tests.Registration" assertions="2"
            time="2.436" file="tests/registration.code" line="24" />
        <testcase name="testCase2" classname="Tests.Registration" assertions="6"
            time="1.534" file="tests/registration.code" line="62" />
        <testcase name="testCase3" classname="Tests.Registration" assertions="3"
            time="0.822" file="tests/registration.code" line="102" />
        
        <!-- Example of a test case that was skipped -->
        <testcase name="testCase4" classname="Tests.Registration" assertions="0"
            time="0" file="tests/registration.code" line="164">
            <!-- <skipped> Indicates that the test was not executed. Can have an optional
            message describing why the test was skipped. -->
            <skipped message="Test was skipped." />
        </testcase>

        <!-- Example of a test case that failed. -->
        <testcase name="testCase5" classname="Tests.Registration" assertions="2"
            time="2.902412" file="tests/registration.code" line="202">
            <!-- <failure> The test failed because one of the assertions/checks failed.
            Can have a message and failure type, often the assertion type or class. The text
            content of the element often includes the failure description or stack trace. -->
            <failure message="Expected value did not match." type="AssertionError">
                <!-- Failure description or stack trace -->
            </failure>
        </testcase>

        <!-- Example of a test case that had errors. -->
        <testcase name="testCase6" classname="Tests.Registration" assertions="0"
            time="3.819" file="tests/registration.code" line="235">
            <!-- <error> The test had an unexpected error during execution. Can have a 
            message and error type, often the exception type or class. The text
            content of the element often includes the error description or stack trace. -->
            <error message="Division by zero." type="ArithmeticError">
                <!-- Error description or stack trace -->
            </error>
        </testcase>

        <!-- Example of a test case with outputs. -->
        <testcase name="testCase7" classname="Tests.Registration" assertions="3"
            time="2.944" file="tests/registration.code" line="287">
            <!-- <system-out> Optional data written to standard out for the test case. -->
            <system-out>Data written to standard out.</system-out>

            <!-- <system-err> Optional data written to standard error for the test case. -->
            <system-err>Data written to standard error.</system-err>
        </testcase>

        <!-- Example of a test case with properties -->
        <testcase name="testCase8" classname="Tests.Registration" assertions="4"
            time="1.625275" file="tests/registration.code" line="302">
            <!-- <properties> Some tools also support properties for test cases. -->
            <properties>
                <property name="priority" value="high" />
                <property name="language" value="english" />
                <property name="author" value="Adrian" />
                <property name="attachment" value="screenshots/dashboard.png" />
                <property name="attachment" value="screenshots/users.png" />
                <property name="description">
                    This text describes the purpose of this test case and provides
                    an overview of what the test does and how it works.
                </property>
            </properties>
        </testcase>
    </testsuite>
</testsuites>

Properties for suites and cases

Properties can be included for suites and cases to record additional information, such as version numbers, environment variables or links to CI pipelines. Properties are name / value pairs. Some tools support multi-line properties by including the property value as the element's text content.

Open example XML file (testcase-properties.xml)
Open example XML file (testcase-output.xml)

<testsuite name="Tests.Registration" time="7.005871">
    <properties>
        <property name="version" value="1.774" />
        <property name="commit" value="ef7bebf" />
        <property name="browser" value="Google Chrome" />
        <property name="ci" value="https://github.com/actions/runs/1234" />
        <property name="config"> <!-- Some tools support properties with text content -->
            Config line #1
            Config line #2
            Config line #3
        </property>
    </properties>

    <!-- Test cases here -->
</testsuite>

Originally, properties were only supported for test suite elements, not cases. However, many tools support properties for test cases now, including test frameworks such as pytest and Playwright, or test management tools such as Testmo and many others. Often it can be useful to include additional details for test cases such as custom fields, test data or attachments.

<testcase name="testCase3" classname="Tests.Registration" time="3.441">
    <properties>
        <property name="priority" value="high" />
        <property name="language" value="english" />
        <property name="author" value="Adrian" />
        <property name="attachment" value="screenshots/dashboard.png" />
        <property name="attachment" value="screenshots/users.png" />

        <!-- Optional support for properties with text values -->
        <property name="description">
            This text describes the purpose of this test case and provides
            an overview of what the test does and how it works.
        </property>
    </properties>

    <!-- Optional output or error/failure/skipped -->
</testcase>

There can be multiple properties with the same name (e.g. multiple properties with the name attachment). Tools that read XML files should support this. However, many test frameworks have APIs that only support one property per name. In this case you should choose unique names for properties (e.g. attachment1 and attachment2, see below).

Properties in test output

Many test frameworks don't support properties for test cases yet. Another option to include properties in XML files is to add them to the test output (similar to referencing attachments in the output). The advantage of this is that developers and testers can output properties even if their testing tool doesn't support test case properties.

<testcase name="testCase3" classname="Tests.Registration" time="3.441">
    <system-out>
Output line #1
Output line #2

[[PROPERTY|author=Adrian]]
[[PROPERTY|language=english]]

[[PROPERTY|browser-log]]
Log line #1
Log line #2
Log line #3
[[/PROPERTY]]
    </system-out>
</testcase>

Each property is listed on its own line without any additional text. Each property consists of a name / value pair. Optionally, multi-line properties just specify the name without a value in the opening tag and end with a closing tag on its own line.

Attachments, screenshots and artifacts

Tests often generate or use additional files as part of a test run, such as log files, screenshots or build artifacts. Sometimes it can be useful to reference or include such files in the XML file. For example, for browser and UI tests it can be useful to include one or more screenshots to better understand why a test failed. Many testing tools support one or more of the following options for attachments.

Attachment file properties

The simplest way to reference files is to list local file paths as properties. This can be included for test suites and test cases. Different tools expect different property names for attachments, but many tools simply expect the properties to be named attachment (or alternatively attachment1 etc.).

<testcase name="testCase3" classname="Tests.Registration" time="3.441">
    <properties>
        <property name="attachment" value="screenshots/dashboard.png" />
        <property name="attachment" value="screenshots/users.png" />
    </properties>
</testcase>

Attachment URL properties

Files might also be uploaded to and stored with CI services or on build servers. For such scenarios it can be useful to reference files as URLs.

<testcase name="testCase3" classname="Tests.Registration" time="3.441">
    <properties>
        <property name="attachment1" value="https://build-server/runs/45112/screenshot1.png" />
        <property name="attachment2" value="https://build-server/runs/45112/screenshot2.png" />
    </properties>
</testcase>

Attachment inline properties

For smaller files and to make it easier to record and transfer files together with an XML file, attachments can also be included inline as data URIs. Filenames can optionally be included with the non-standard filename attribute. Data should be encoded with base64.

<testcase name="testCase3" classname="Tests.Registration" time="3.441">
    <properties>
        <property name="attachment1" value="data:image/png;filename=dot.png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAATXL0Y4OHwAAAABJRU5ErkJggg==" />
        <property name="attachment2" value="data:text/html;base64,SGVsbG8sIFdvcmxkIQ==" />
    </properties>
</testcase>

Attachments in test output

Some tools support another way to reference files from the test output. This convention was first introduced by the Jenkins JUnit Attachments plugin. This format is often used with tools that don't support test case properties. Each attachment is listed on a separate line (without any additional text) as part of the test's output (system-out or system-err) using the following format.

<testcase name="testCase3" classname="Tests.Registration" time="3.441">
    <system-out>
Output line #1
Output line #2
[[ATTACHMENT|screenshots/dashboard.png]]
[[ATTACHMENT|screenshots/users.png]]
    </system-out>
</testcase>

Common properties and conventions

Test frameworks and testing tools can generally write and read properties with any name and value. There aren't any standards on how to name properties or how to format your values. However, various testing tools that read JUnit-style XML files expect certain property names and values. This section lists examples for commonly used property names and how tools format property values.

Open example XML file (conventions.xml)

Attachments

Name: attachment*
Files attached to suites or cases, such as log files, screenshots, build artifacts or configurations.

<!-- Either use multiple properties named "attachment" -->
<property name="attachment" value="screenshots/failure1.png" />
<property name="attachment" value="screenshots/failure2.png" />

<!-- Or append a unique index if your test framework requires this -->
<property name="attachment1" value="browser/console.log" />
<property name="attachment2" value="browser/config.json" />

Steps

Name: step*
Steps can be useful to document the exact steps a test case performs. Tools can render all steps with optional statuses and sub fields.

<!-- Either use multiple properties named "step" -->
<property name="step" value="This is a test step." />
<property name="step" value="Tools will automatically index all steps." />

<!-- Or append a unique index if your test framework requires this -->
<property name="step1" value="You can also use unique step property names if needed." />
<property name="step2" value="And this is another test step." />

<!-- Some tools support properties with text content -->
<property name="step">
    This is a more complex test step with multiple lines
    of text using a property with a text value.
</property>

<!-- You can optionally also include a step result as part of the property name.
Check your CI or test management tool if it supports statuses. -->
<property name="step[passed]" value="This is a test step that passed." />
<property name="step[failure]" value="And this step had a failure." />
<property name="step[skipped]" value="This step wasn't executed." />
<property name="step[error]" value="And this step had an error." />

<!-- Some tools support custom HTML tags to document step sub fields. Check
your CI or test management tool to learn about HTML support. -->
<property name="step[passed]"><![CDATA[
    <test-step-subfield name="Step">
        This line describes this step. You can also use <strong>HTML</strong>.
    </test-step-subfield>
    <test-step-subfield name="Expected">
        You can optionally include more sub fields, e.g. for the expected results.
    </test-step-subfield>
]]></property>

Type hints

Sometimes it can be useful to use more complex types for property values, such as HTML. Or tools support rendering specific property values in a special way. For example, URLs could be rendered as clickable links.

To avoid tools having to guess the type of a property value by its content, some tools support type hints for property values. Tools that generate JUnit-style XML files (e.g. test automation frameworks) do not need to provide special support for this, as type hints are just provided as part of property names, using the format type:name.

There is no standard of types that tools support, so refer to your CI or test management tool's documentation to see if and which types might be supported. Here are some examples for types that tools might support:

<!-- Some tools also support type hints as part of the property names. Check your
CI or test management tool to learn if and which type hints are supported. Type hints
are removed from the property names. -->

<!-- URLs can be rendered as links -->
<property name="url:github" value="https://github.com/org/project/blob/code.js#L200" />

<!-- Console values are rendered as monospaced code block -->
<property name="console:log">
Log line #1
Log line #2
Log line #3
</property>

<!-- Some tools might support a limited subset of HTML for rich content -->
<property name="html:output">
    <![CDATA[<h1>Title</h1><p>Text</p>]]>
</property>

About this project

The goal of this project is to document the common use of JUnit-style XML files and describe how many tools implement support for this format. This project can be a starting point for builders of tools that generate or consume JUnit-style XML files. This file format has evolved over time and tools are adding support for test case properties and attachments as outlined in the above examples.

The information in this project has been collected by the team at Testmo. Please submit an issue if you have questions or feedback or think some additional details should be added.

Footnotes

  1. More precisely, the JUnit XML format was first introduced by the Ant build tool.

  2. The format has evolved over time and new elements are used by tools now, so any original schema & XSD files are usually outdated.

About

JUnit XML file format & JUnit XML examples. Including format specification, description & conventions.

Topics

Resources

Stars

Watchers

Forks