Unit test reports

Version history

It is very common that a CI/CD pipeline contains a test job that verifies your code. If the tests fail, the pipeline fails and users get notified. The person that works on the merge request has to check the job logs and see where the tests failed so that they can fix them.

You can configure your job to use Unit test reports, and GitLab displays a report on the merge request so that it’s easier and faster to identify the failure without having to check the entire log. Unit test reports currently only support test reports in the JUnit report format.

If you don’t use merge requests but still want to see the unit test report output without searching through job logs, the full Unit test reports are available in the pipeline detail view.

Consider the following workflow:

  1. Your default branch is rock solid, your project is using GitLab CI/CD and your pipelines indicate that there isn’t anything broken.
  2. Someone from your team submits a merge request, a test fails and the pipeline gets the known red icon. To investigate more, you have to go through the job logs to figure out the cause of the failed test, which usually contain thousands of lines.
  3. You configure the Unit test reports and immediately GitLab collects and exposes them in the merge request. No more searching in the job logs.
  4. Your development and debugging workflow becomes easier, faster and efficient.

How it works

First, GitLab Runner uploads all JUnit report format XML files as artifacts to GitLab. Then, when you visit a merge request, GitLab starts comparing the head and base branch’s JUnit report format XML files, where:

  • The base branch is the target branch (usually the default branch).
  • The head branch is the source branch (the latest pipeline in each merge request).

The reports panel has a summary showing how many tests failed, how many had errors and how many were fixed. If no comparison can be done because data for the base branch is not available, the panel just shows the list of failed tests for head.

There are four types of results:

  1. Newly failed tests: Test cases which passed on base branch and failed on head branch
  2. Newly encountered errors: Test cases which passed on base branch and failed due to a test error on head branch
  3. Existing failures: Test cases which failed on base branch and failed on head branch
  4. Resolved failures: Test cases which failed on base branch and passed on head branch

Each entry in the panel shows the test name and its type from the list above. Clicking on the test name opens a modal window with details of its execution time and the error output.

Test Reports Widget

Number of recent failures

Version history

If a test failed in the project’s default branch in the last 14 days, a message like Failed {n} time(s) in {default_branch} in the last 14 days is displayed for that test.

How to set it up

To enable the Unit test reports in merge requests, you must add artifacts:reports:junit in .gitlab-ci.yml, and specify the paths of the generated test reports. The reports must be .xml files, otherwise .

In the following examples, the job in the test stage runs and GitLab collects the Unit test report from each job. After each job is executed, the XML reports are stored in GitLab as artifacts and their results are shown in the merge request widget.

To make the Unit test report output files browsable, include them with the artifacts:paths keyword as well, as shown in the Ruby example. To upload the report even if the job fails (for example if the tests do not pass), use the artifacts:when:always keyword.

You cannot have multiple tests with the same name and class in your JUnit report format XML file.

Ruby example

Use the following job in .gitlab-ci.yml. This includes the artifacts:paths keyword to provide a link to the Unit test report output file. 

## Use https://github.com/sj26/rspec_junit_formatter to generate a JUnit report format XML file with rspec
ruby:
  stage: test
  script:
    - bundle install
    - bundle exec rspec --format progress --format RspecJunitFormatter --out rspec.xml
  artifacts:
    when: always
    paths:
      - rspec.xml
    reports:
      junit: rspec.xml

Go example

Use the following job in .gitlab-ci.yml: 

## Use https://github.com/gotestyourself/gotestsum to generate a JUnit report format XML file with go
golang:
  stage: test
  script:
    - go get gotest.tools/gotestsum
    - gotestsum --junitfile report.xml --format testname
  artifacts:
    when: always
    reports:
      junit: report.xml

Java examples

There are a few tools that can produce JUnit report format XML file in Java.

Gradle

In the following example, gradle is used to generate the test reports. If there are multiple test tasks defined, gradle generates multiple directories under build/test-results/. In that case, you can leverage glob matching by defining the following path: build/test-results/test/**/TEST-*.xml: 

java:
  stage: test
  script:
    - gradle test
  artifacts:
    when: always
    reports:
      junit: build/test-results/test/**/TEST-*.xml

In and later, you can use **.

Maven

For parsing Surefire and .gitlab-ci.yml: 

java:
  stage: test
  script:
    - mvn verify
  artifacts:
    when: always
    reports:
      junit:
        - target/surefire-reports/TEST-*.xml
        - target/failsafe-reports/TEST-*.xml

Python example

This example uses pytest with the --junitxml=report.xml flag to format the output into the JUnit report XML format: 

pytest:
  stage: test
  script:
    - pytest --junitxml=report.xml
  artifacts:
    when: always
    reports:
      junit: report.xml

C/C++ example

There are a few tools that can produce JUnit report format XML files in C/C++.

GoogleTest

In the following example, gtest is used to generate the test reports. If there are multiple gtest executables created for different architectures (x86, x64 or arm), you are required to run each test providing a unique filename. The results are then aggregated together. 

cpp:
  stage: test
  script:
    - gtest.exe --gtest_output="xml:report.xml"
  artifacts:
    when: always
    reports:
      junit: report.xml

CUnit 

cunit:
  stage: test
  script:
    - ./my-cunit-test
  artifacts:
    when: always
    reports:
      junit: ./my-cunit-test.xml