reporting.md

This specification covers some improvements to how reports are generated and presented to the user.

Use cases

Developer views a summary of all tests executed by the build

When a build runs multiple test suites, it can be difficult to determine exactly which tests were executed and which ones failed, particularly if the build console output is not retained.

It should be possible to view a persistent summary of which tests were executed by the build and a summary of their results.

Developer views a summary of all code quality checks executed by the build

As for test execution, it can be difficult to determine exactly which code quality checks were executed and which ones failed.

It should be possible to view a persistent summary of which code quality checks were executed by the build and a summary of their results.

Developer views a summary of all reports generated by the build

The reports generated by a build contain useful information about the things that the build has done, or useful information about the build itself.

It should be possible to view a persistent summary of the reports that the build has generated. It should be possible for a build to declare custom reports.

Developer views a summary of all documentation generated by the build

It should be possible to view a persistent summary of the documentation that the build has generated. It should be possible for a build to declare custom documentation.

Developer views a summary of the test coverage for all tests executed by the build

As for test execution. Test results and summaries should include coverage information.

CI build publishes the reports, documentation and summary generated by the build

The CI build is often the reference build for generating documentation and reports, and for executing tests and other code quality checks. The build should generate these artifacts and any summaries in a way that makes it easy to use the CI server's artifact publishing mechanism to publish this output.

Implementation plan

The approach used here is to incrementally grow a dashboard report that summarises the reports, verifications and documentation produced by the build. The report will start off extremely basic, and over a number of steps, more content will be added to the report. Both built-in and custom reports, verifications and documentation will be supported.

In addition, the task graph will be changed to make the dashboard report more usable and reliable in various cases.

Add a build dashboard report (DONE)

Add a basic dashboard HTML report that links to the reports declared in the build.

User visible changes

Add build-dashboard plugin that, by convention, generates a dashboard HTML report that links to all enabled reports generated by the current project and all subprojects.

To use in a single project build:

apply plugin: 'java'
apply plugin: 'checkstyle'
apply plugin: 'build-dashboard'

Running gradle buildDashboard will generate an HTML report that links to the test and checkstyle reports, if they exist. It will not generate those reports.

Running gradle check buildDashboard will generate the test and checkstyle reports, and an HTML report that links to them.

To use in a multi-project build:

apply plugin: 'build-dashboard'

subprojects {
    apply plugin: 'java'
    apply plugin: 'checkstyle'
}

Running gradle check buildDashboard in the root project will generate the test and checkstyle reports for all projects, and an HTML report that links to them.

Note that running gradle check buildDashboard will not generate the dashboard report if any check fails. Either --continue or VerificationTask.ignoreFailures can be used as workarounds. Later stories will make this work better.

Note that running gradle buildDashboard check will not do anything very useful at this stage. Later stories will make this work better.

Implementation approach

1. Add a GenerateBuildDashboard task type.
   • Implements Reporting.
   • Takes a set of Report instances as input.
   • Generates an HTML report that is more or less a list of reports and a link to their output.
2. Add a build-dashboard plugin.
   • Adds a buildDashboard task of type GenerateBuildDashboard.
   • Configures buildDashboard to report on all enabled Report instances for all tasks of type Reporting in the current project and all subprojects.

Schedule dashboard report task to run after all reporting and verification tasks

Run any reporting tasks that are to be executed by the build before running the dashboard report task, so that the dashboard report includes results from the current build.

User visible changes

Running gradle dashboardReport check will run the reporting and verification tasks first, then the dashboardReport task, followed by the remaining check tasks.

Running gradle check dashboardReport --continue will generate the dashboard report on failures, regardless of whether any reporting tasks have been succefully executed.

For this story, running gradle check dashboardReport without --continue will not generate the dashboard report on failures. Later stories will improve this behaviour.

Implementation approach

1. Add a "always runs after" dependency type to the task graph.
   • When "taskA always runs after taskB" and both are in the task graph, then taskA must not be executed until after taskB has been executed, and must execute regardless of whether taskB succeeds or fails. Execution of taskA must honour the other dependencies of taskA.
2. When dashboardReport task is added to the graph, add "always runs after" dependencies from dashboardReport to any task that generates one of its input reports and that is also included in the task graph.

Test coverage

• For a multi-project build where rootProject.check.dependsOn(dashboardReport) and that applies the code quality plugins:
  • verify that running gradle check generates the report after each of the code quality tasks have completed.
  • verify that running gradle check --continue generates the report when one of the code quality tasks fails.
• When a.mustRunAfter b and a.dependsOn c and b.dependsOn d
  • running gradle a runs c then a.
  • running gradle a b runs d then b then c then a.
  • running gradle a b --continue where b is broken, then runs d then b then c then a.
  • running gradle a b --continue where d is broken, then runs d then c then a.
• When a.mustRunAfter b and b.mustRunAfter a
  • running gradle a runs a only.
  • running gradle a b gives a reasonable error message
• When a.mustRunAfter b and b.dependsOn a
  • running gradle a runs a only.
  • running gradle b gives a reasonable error message
• When a.mustRunAfter b and b.mustRunAfter c
  • running gradle a b c runs c then b then a.
• In parallel mode:
  • when a.mustRunAfter b and b depends on a slow task c in another project, verify that c runs before b runs before a.

Automatically add dashboard report task to task graph

A DSL element will be added to allow a finaliser task to be automatically added to the task graph when some other task is scheduled to run. This task and its dependencies are run only if the target task is executed.

The build-dashboard plugin will use this to ensure that the dasboard report task is always updated when any of the reports are generated.

This type of task relationship will also be used to generate test and code coverage reports after the tests have been executed. It will also be used to undeploy applications and stop servers and clean up other resources when the tests that use them have finished executing.

User visible changes

Running gradle checkstyleMain will also execute dashboardReport. The dashboardReport task will be executed regardless of whether the checkstyleMain task succeeds or fails and regardless of whether --continue is used or not.

The dashboardReport task is not executed when none of the reporting tasks in the graph are executed due to failed dependencies.

TBD - Need to some reasonable behaviour when:

1. A build has the checkstyle plugin applied.
2. gradle check is run and implcitly generates the dashboard.
3. The checkstyle plugin is removed.
4. gradle check is run. The dashboard should still be generated or removed. Currently, the old report would not be regenerated.

Implementation approach

When a task a is a finaliser for task b, then:

• when task b is added to the graph, then task a must be added to the graph.
• task a must always execute after task b.
• task a must be executed if task b is executed, and regardless of whether task b succeeds or fails.
• task a should not be executed if task b is not executed and task a is otherwise not required to be executed. Here 'not executed' includes:
  • was not executed due to a failed dependency.
  • was considered up-to-date.
  • was disabled or skipped due to an onlyIf { } constraint.
  • is a finaliser for tasks which were not executed for some reason.
• A dependency task c of task a should not be executed if task b is not executed (as above) and the task c is otherwise not required to be executed

Test coverage

For build with the build-dashboard plugin applied to the root project and the (say) checkstyle plugin applied:

• running gradle check will run the buildDashboard task.
• running gradle check will run the buildDashboard task when the checkstyle task fails.
• running gradle check will not run the buildDashboard task when the checkstyle task is not executed due to a failed dependency.
• running gradle check --continue will not run the buildDashboard task when the checkstyle task is not executed due to a failed dependency.

TBD - some general int tests

Separate test report generation from test execution

Configure a TestReport task to always run after the Test task, and change the TestReport type to implement Reporting. Remove (via deprecation) the reporting from the Test task type.

Include summary of all verification task results in the dashboard report

Change the dashboard report so that for each report generated by a verification task, include a visual indication of whether the verification has succeeded or failed.

User visible changes

The report will show more information.

Implementation approach

1. Add VerificationFailedException.
2. Change the contract for verification tasks, so that they must throw VerificationFailedException on verification failure. Any other exception is treated as a general failure and the outputs generated by that task are assumed to be incomplete.
3. Persist the verification exception.
4. Dashboard report includes message from verification exception.

Include test summary in the dashboard report

Change the dashboard report so that for each report generated by a Test task, include summary information about how many tests where executed, passed, failed and ignored.

User visible changes

The report will show more information.

Implementation approach

1. Add VerificationSummary interface.
2. Add TestSummary interface.
   • How many tests were executed.
   • How many tests failed.
   • How many tests were ignored.
3. Change contract of verification tasks, so that they must provide a serializable VerificationSummary result after execution.
4. Persist the verification result.
5. Dashboard report includes test summary when verification result is an instanceof TestSummary.

Allow a custom verification task to be implemented

Allow a custom reporting task to be implemented

Include code coverage summary in the dashboard report

Link to the API documentation in the dashboard report

Link to custom documentation in the dashboard report

Link to the dependency reports in the dashboard report

Link to the project structure reports in the dashboard report

Link to the build comparison report in the dashboard report

Link to the profile report in the dashboard report

Add an HTML report for Checkstyle

Include Checkstyle summary in the dashboard report

Include CodeNarc summary in the dashboard report

Include PMD summary in the dashboard report

Include FindBugs summary in the dashboard report

Include JDepend summary in the dashboard report

Include a summary of custom reporting task in the dashboard report

Open issues