Skip to content

Commit 67bc432

Browse files
dnozaydnozay
authored
info / fixes about xUnit plugin (#215)
- see #209; add a utility function for XSLT transformation. - minor doc fixes. - fix incorrect version, I called out xunit plugin 2.2.4 earlier and I must not have looked at the version properly.
1 parent f58d7b7 commit 67bc432

File tree

5 files changed

+273
-29
lines changed

5 files changed

+273
-29
lines changed

‎README.md

+65-18
Original file line numberDiff line numberDiff line change
@@ -14,35 +14,82 @@ A unittest test runner that can save test results to XML files in xUnit format.
1414
The files can be consumed by a wide range of tools, such as build systems, IDEs
1515
and continuous integration servers.
1616

17-
## Schema
1817

19-
There are many schemas with minor differences.
20-
We use one that is compatible with Jenkins xUnit plugin, a copy is
21-
available under `tests/vendor/jenkins/xunit-plugin/junit-10.xsd` (see attached license).
18+
## Requirements
19+
20+
* Python 3.5+
21+
* Please note Python 2.7 end-of-life was in Jan 2020, last version supporting 2.7 was 2.5.2
22+
* Please note Python 3.4 end-of-life was in Mar 2019, last version supporting 3.4 was 2.5.2
23+
* Please note Python 2.6 end-of-life was in Oct 2013, last version supporting 2.6 was 1.14.0
24+
25+
26+
## Limited support for `unittest.TestCase.subTest`
27+
28+
https://docs.python.org/3/library/unittest.html#unittest.TestCase.subTest
29+
30+
`unittest` has the concept of sub-tests for a `unittest.TestCase`; this doesn't map well to an existing xUnit concept, so you won't find it in the schema. What that means, is that you lose some granularity
31+
in the reports for sub-tests.
32+
33+
`unittest` also does not report successful sub-tests, so the accounting won't be exact.
2234

23-
- [Jenkins (junit-10.xsd), xunit plugin (2014-2018)](https://github.com/jenkinsci/xunit-plugin/blob/14c6e39c38408b9ed6280361484a13c6f5becca7/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd), please note the latest versions (2.2.4 and above are not backwards compatible)
35+
## Jenkins plugins
36+
37+
- Jenkins JUnit plugin : https://plugins.jenkins.io/junit/
38+
- Jenkins xUnit plugin : https://plugins.jenkins.io/xunit/
39+
40+
### Jenkins JUnit plugin
41+
42+
This plugin does not perform XSD validation (at time of writing) and should parse the XML file without issues.
43+
44+
### Jenkins xUnit plugin version 1.100
45+
46+
- [Jenkins (junit-10.xsd), xunit plugin (2014-2018)](https://github.com/jenkinsci/xunit-plugin/blob/14c6e39c38408b9ed6280361484a13c6f5becca7/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd), version `1.100`.
47+
48+
This plugin does perfom XSD validation and uses the more lax XSD. This should parse the XML file without issues.
49+
50+
### Jenkins xUnit plugin version 1.104+
51+
52+
- [Jenkins (junit-10.xsd), xunit plugin (2018-current)](https://github.com/jenkinsci/xunit-plugin/blob/ae25da5089d4f94ac6c4669bf736e4d416cc4665/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd), version `1.104`+.
53+
54+
This plugin does perfom XSD validation and uses the more strict XSD.
55+
56+
See https://github.com/xmlrunner/unittest-xml-reporting/issues/209
57+
58+
```
59+
import io
60+
import unittest
61+
import xmlrunner
62+
63+
# run the tests storing results in memory
64+
out = io.BytesIO()
65+
unittest.main(
66+
testRunner=xmlrunner.XMLTestRunner(output=out),
67+
failfast=False, buffer=False, catchbreak=False, exit=False)
68+
```
69+
70+
Transform the results removing extra attributes.
71+
```
72+
from xmlrunner.extra.xunit_plugin import transform
73+
74+
with open('TEST-report.xml', 'wb') as report:
75+
report.write(transform(out.getvalue()))
76+
77+
```
78+
79+
## JUnit Schema ?
80+
81+
There are many tools claiming to write JUnit reports, so you will find many schemas with minor differences.
82+
83+
We used the XSD that was available in the Jenkins xUnit plugin version `1.100`; a copy is available under `tests/vendor/jenkins/xunit-plugin/.../junit-10.xsd` (see attached license).
2484

2585
You may also find these resources useful:
2686

2787
- https://stackoverflow.com/questions/4922867/what-is-the-junit-xml-format-specification-that-hudson-supports
2888
- https://stackoverflow.com/questions/11241781/python-unittests-in-jenkins
29-
- [Jenkins (junit-10.xsd), xunit plugin 2.2.4+](https://github.com/jenkinsci/xunit-plugin/blob/master/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd)
3089
- [JUnit-Schema (JUnit.xsd)](https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd)
3190
- [Windyroad (JUnit.xsd)](http://windyroad.com.au/dl/Open%20Source/JUnit.xsd)
3291
- [a gist (Jenkins xUnit test result schema)](https://gist.github.com/erikd/4192748)
3392

34-
## Things that are somewhat broken
35-
36-
Python 3 has the concept of sub-tests for a `unittest.TestCase`; this doesn't map well to an existing
37-
xUnit concept, so you won't find it in the schema. What that means, is that you lose some granularity
38-
in the reports for sub-tests.
39-
40-
## Requirements
41-
42-
* Python 3.5+
43-
* Please note Python 2.7 end-of-life was in Jan 2020, last version supporting 2.7 was 2.5.2
44-
* Please note Python 3.4 end-of-life was in Mar 2019, last version supporting 3.4 was 2.5.2
45-
* Please note Python 2.6 end-of-life was in Oct 2013, last version supporting 2.6 was 1.14.0
4693

4794
## Installation
4895

‎tests/testsuite.py

+33-11
Original file line numberDiff line numberDiff line change
@@ -27,23 +27,21 @@
2727
from unittest import mock
2828

2929

30-
def _load_schema():
31-
path = os.path.join(os.path.dirname(__file__),
32-
'vendor/jenkins/xunit-plugin',
33-
'junit-10.xsd')
30+
def _load_schema(version):
31+
path = os.path.join(
32+
os.path.dirname(__file__),
33+
'vendor/jenkins/xunit-plugin', version, 'junit-10.xsd')
3434
with open(path, 'r') as schema_file:
3535
schema_doc = etree.parse(schema_file)
3636
schema = etree.XMLSchema(schema_doc)
3737
return schema
3838
raise RuntimeError('Could not load JUnit schema') # pragma: no cover
3939

4040

41-
JUnitSchema = _load_schema()
42-
43-
44-
def validate_junit_report(text):
41+
def validate_junit_report(version, text):
4542
document = etree.parse(BytesIO(text))
46-
JUnitSchema.assertValid(document)
43+
schema = _load_schema(version)
44+
schema.assertValid(document)
4745

4846

4947
class TestCaseSubclassWithNoSuper(unittest.TestCase):
@@ -650,7 +648,7 @@ def test_junitxml_xsd_validation_order(self):
650648
self.assertTrue(i_properties < i_testcase <
651649
i_system_out < i_system_err)
652650
# XSD validation - for good measure.
653-
validate_junit_report(output)
651+
validate_junit_report('14c6e39c38408b9ed6280361484a13c6f5becca7', output)
654652

655653
def test_junitxml_xsd_validation_empty_properties(self):
656654
suite = unittest.TestSuite()
@@ -665,7 +663,31 @@ def test_junitxml_xsd_validation_empty_properties(self):
665663
outdir.seek(0)
666664
output = outdir.read()
667665
self.assertNotIn('<properties>'.encode('utf8'), output)
668-
validate_junit_report(output)
666+
validate_junit_report('14c6e39c38408b9ed6280361484a13c6f5becca7', output)
667+
668+
def test_xunit_plugin_transform(self):
669+
suite = unittest.TestSuite()
670+
suite.addTest(self.DummyTest('test_fail'))
671+
suite.addTest(self.DummyTest('test_pass'))
672+
suite.properties = None
673+
outdir = BytesIO()
674+
runner = xmlrunner.XMLTestRunner(
675+
stream=self.stream, output=outdir, verbosity=self.verbosity,
676+
**self.runner_kwargs)
677+
runner.run(suite)
678+
outdir.seek(0)
679+
output = outdir.read()
680+
681+
validate_junit_report('14c6e39c38408b9ed6280361484a13c6f5becca7', output)
682+
with self.assertRaises(etree.DocumentInvalid):
683+
validate_junit_report('ae25da5089d4f94ac6c4669bf736e4d416cc4665', output)
684+
685+
from xmlrunner.extra.xunit_plugin import transform
686+
transformed = transform(output)
687+
validate_junit_report('14c6e39c38408b9ed6280361484a13c6f5becca7', transformed)
688+
validate_junit_report('ae25da5089d4f94ac6c4669bf736e4d416cc4665', transformed)
689+
self.assertIn('test_pass'.encode('utf8'), transformed)
690+
self.assertIn('test_fail'.encode('utf8'), transformed)
669691

670692
def test_xmlrunner_elapsed_times(self):
671693
self.runner_kwargs['elapsed_times'] = False

‎tests/vendor/jenkins/xunit-plugin/junit-10.xsd renamed to ‎tests/vendor/jenkins/xunit-plugin/14c6e39c38408b9ed6280361484a13c6f5becca7/junit-10.xsd

File renamed without changes.

‎tests/vendor/jenkins/xunit-plugin/ae25da5089d4f94ac6c4669bf736e4d416cc4665/junit-10.xsd

+147
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
<?xml version="1.0" encoding="UTF-8" ?>
2+
<!--
3+
The MIT License (MIT)
4+
5+
Copyright (c) 2014, Gregory Boissinot
6+
7+
Permission is hereby granted, free of charge, to any person obtaining a copy
8+
of this software and associated documentation files (the "Software"), to deal
9+
in the Software without restriction, including without limitation the rights
10+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
11+
copies of the Software, and to permit persons to whom the Software is
12+
furnished to do so, subject to the following conditions:
13+
14+
The above copyright notice and this permission notice shall be included in
15+
all copies or substantial portions of the Software.
16+
17+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
18+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
19+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
20+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
21+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
22+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
23+
THE SOFTWARE.
24+
-->
25+
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
26+
<xs:simpleType name="SUREFIRE_TIME">
27+
<xs:restriction base="xs:string">
28+
<xs:pattern value="(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?"/>
29+
</xs:restriction>
30+
</xs:simpleType>
31+
32+
<xs:complexType name="rerunType" mixed="true"> <!-- mixed (XML contains text) to be compatible with version previous than 2.22.1 -->
33+
<xs:sequence>
34+
<xs:element name="stackTrace" type="xs:string" minOccurs="0" /> <!-- optional to be compatible with version previous than 2.22.1 -->
35+
<xs:element name="system-out" type="xs:string" minOccurs="0" />
36+
<xs:element name="system-err" type="xs:string" minOccurs="0" />
37+
</xs:sequence>
38+
<xs:attribute name="message" type="xs:string" />
39+
<xs:attribute name="type" type="xs:string" use="required" />
40+
</xs:complexType>
41+
42+
<xs:element name="failure">
43+
<xs:complexType mixed="true">
44+
<xs:attribute name="type" type="xs:string"/>
45+
<xs:attribute name="message" type="xs:string"/>
46+
</xs:complexType>
47+
</xs:element>
48+
49+
<xs:element name="error">
50+
<xs:complexType mixed="true">
51+
<xs:attribute name="type" type="xs:string"/>
52+
<xs:attribute name="message" type="xs:string"/>
53+
</xs:complexType>
54+
</xs:element>
55+
56+
<xs:element name="skipped">
57+
<xs:complexType mixed="true">
58+
<xs:attribute name="type" type="xs:string"/>
59+
<xs:attribute name="message" type="xs:string"/>
60+
</xs:complexType>
61+
</xs:element>
62+
63+
<xs:element name="properties">
64+
<xs:complexType>
65+
<xs:sequence>
66+
<xs:element ref="property" minOccurs="0" maxOccurs="unbounded"/>
67+
</xs:sequence>
68+
</xs:complexType>
69+
</xs:element>
70+
71+
<xs:element name="property">
72+
<xs:complexType>
73+
<xs:attribute name="name" type="xs:string" use="required"/>
74+
<xs:attribute name="value" type="xs:string" use="required"/>
75+
</xs:complexType>
76+
</xs:element>
77+
78+
<xs:element name="system-err" type="xs:string"/>
79+
<xs:element name="system-out" type="xs:string"/>
80+
<xs:element name="rerunFailure" type="rerunType"/>
81+
<xs:element name="rerunError" type="rerunType"/>
82+
<xs:element name="flakyFailure" type="rerunType"/>
83+
<xs:element name="flakyError" type="rerunType"/>
84+
85+
<xs:element name="testcase">
86+
<xs:complexType>
87+
<xs:sequence>
88+
<xs:choice minOccurs="0" maxOccurs="unbounded">
89+
<xs:element ref="skipped"/>
90+
<xs:element ref="error"/>
91+
<xs:element ref="failure"/>
92+
<xs:element ref="rerunFailure" minOccurs="0" maxOccurs="unbounded"/>
93+
<xs:element ref="rerunError" minOccurs="0" maxOccurs="unbounded"/>
94+
<xs:element ref="flakyFailure" minOccurs="0" maxOccurs="unbounded"/>
95+
<xs:element ref="flakyError" minOccurs="0" maxOccurs="unbounded"/>
96+
<xs:element ref="system-out"/>
97+
<xs:element ref="system-err"/>
98+
</xs:choice>
99+
</xs:sequence>
100+
<xs:attribute name="name" type="xs:string" use="required"/>
101+
<xs:attribute name="time" type="xs:string"/>
102+
<xs:attribute name="classname" type="xs:string"/>
103+
<xs:attribute name="group" type="xs:string"/>
104+
</xs:complexType>
105+
</xs:element>
106+
107+
<xs:element name="testsuite">
108+
<xs:complexType>
109+
<xs:choice minOccurs="0" maxOccurs="unbounded">
110+
<xs:element ref="testsuite"/>
111+
<xs:element ref="properties"/>
112+
<xs:element ref="testcase"/>
113+
<xs:element ref="system-out"/>
114+
<xs:element ref="system-err"/>
115+
</xs:choice>
116+
<xs:attribute name="name" type="xs:string" use="required"/>
117+
<xs:attribute name="tests" type="xs:string" use="required"/>
118+
<xs:attribute name="failures" type="xs:string" use="required"/>
119+
<xs:attribute name="errors" type="xs:string" use="required"/>
120+
<xs:attribute name="group" type="xs:string" />
121+
<xs:attribute name="time" type="SUREFIRE_TIME"/>
122+
<xs:attribute name="skipped" type="xs:string" />
123+
<xs:attribute name="timestamp" type="xs:string" />
124+
<xs:attribute name="hostname" type="xs:string" />
125+
<xs:attribute name="id" type="xs:string" />
126+
<xs:attribute name="package" type="xs:string" />
127+
<xs:attribute name="file" type="xs:string"/>
128+
<xs:attribute name="log" type="xs:string"/>
129+
<xs:attribute name="url" type="xs:string"/>
130+
<xs:attribute name="version" type="xs:string"/>
131+
</xs:complexType>
132+
</xs:element>
133+
134+
<xs:element name="testsuites">
135+
<xs:complexType>
136+
<xs:sequence>
137+
<xs:element ref="testsuite" minOccurs="0" maxOccurs="unbounded" />
138+
</xs:sequence>
139+
<xs:attribute name="name" type="xs:string" />
140+
<xs:attribute name="time" type="SUREFIRE_TIME"/>
141+
<xs:attribute name="tests" type="xs:string" />
142+
<xs:attribute name="failures" type="xs:string" />
143+
<xs:attribute name="errors" type="xs:string" />
144+
</xs:complexType>
145+
</xs:element>
146+
147+
</xs:schema>

‎xmlrunner/extra/xunit_plugin.py

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
import io
2+
import lxml.etree as etree
3+
4+
5+
TRANSFORM = etree.XSLT(etree.XML('''\
6+
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">
7+
<xsl:output method="xml" indent="yes" />
8+
9+
<!-- /dev/null for these attributes -->
10+
<xsl:template match="//testcase/@file" />
11+
<xsl:template match="//testcase/@line" />
12+
<xsl:template match="//testcase/@timestamp" />
13+
14+
<!-- copy the rest -->
15+
<xsl:template match="node()|@*">
16+
<xsl:copy>
17+
<xsl:apply-templates select="node()|@*" />
18+
</xsl:copy>
19+
</xsl:template>
20+
</xsl:stylesheet>'''))
21+
22+
23+
def transform(xml_data):
24+
out = io.BytesIO()
25+
xml_doc = etree.XML(xml_data)
26+
result = TRANSFORM(xml_doc)
27+
result.write(out)
28+
return out.getvalue()

0 commit comments

Comments
 (0)