Jenkins Integration #186

ghost · 2018-11-01T22:33:33Z

Description

Adding a Jenkinsfile and clang_patch script to support continuous integration testing on pull requests.
This Jenkinsfile will execute the following as a basic first step towards CI:

Run the linter (assuming clang-format for now, if that changes I will update this) and verify that no files modified by current pull request have linting issues. This doesn't check every file, only those modified from master (line 4 of .ci/clang_patch) so that lint errors won't be reported on files not touched. The clang_format.patch file is always archived by Jenkins for future reference, but when lint passes it should be a 0B file.
Run make ci for firmware/ec. This will build the firmware image with GCC and create an initial coverage report that shows 0 coverage for all source files. The unit tests are then run and coverage gathered for each one. The results are output as junit xml file that Jenkins can interpret, and .ci/coverage.py is run to convert lcov output to Cobertura format that Jenkins can interpret. After all these runs, the warnings output by the build are gathered by the console so that this can also be tracked by Jenkins, but will not cause a test failure.

Test Plan

Verify that the tests pass on this branch. Linter is not currently run unless PR #157 is closed and merged, then it should run and pass.
Unit tests should be reported as failed until #185 is closed and merged.
Verified clang patch by running it in top directory of the repo. Modified src/main.c by adding an extra empty line, just to verify that a modified file would be detected and patch file would be created.

Issues

#170

… script

ghost · 2018-11-12T18:06:16Z

Jenkins firmware test failed.
No test results found.

… utilities/jenkins dir to match schema PR #180.

ghost · 2018-11-12T19:06:15Z

Jenkins firmware test failed.
No test results found.

ghost · 2018-11-12T19:12:28Z

There will be a bunch of comments and updates on this PR as I mess with Jenkins stuff today, just giving a heads up.
Rerun tests.

ghost · 2018-11-12T19:22:16Z

rerun test

ghost · 2018-11-12T19:58:37Z

run tests

ghost · 2018-11-12T20:05:34Z

rerun tests

ghost · 2018-11-12T20:18:41Z

run test

…ter.

JoshuaJeyaraj

I have suggestion in addition to the lint we can do more ,
The current code base has a certain amount of warnings in linux build and CCS build . Can we use jenkins to ensure that the code which is getting added as part of PR doesn't add more warnings .

JoshuaJeyaraj · 2018-11-15T11:57:29Z

firmware/ec/src/main.c

@@ -9,6 +9,7 @@
 //*****************************************************************************
 //                                HEADER FILES
 //*****************************************************************************
+


I suppose that this extra line is to validate the script , please remove it before merging to master .

Yes, that was just done because linter only runs on changed files in Jenkins, so I needed a .c file to be be detected as changed so that the linter would be run. I will change it back with next commit.

JoshuaJeyaraj · 2018-11-15T12:00:18Z

firmware/ec/Jenkinsfile

@@ -0,0 +1,37 @@
+node {


Could we add some headers for all the new files

JoshuaJeyaraj · 2018-11-16T10:52:48Z

Test Plan

Verified clang patch by running it in top directory of the repo. Modified src/main.c by adding an extra empty line, just to verify that a modified file would be detected and patch file would be created.

Can we do some more testing for valid/invalid entries?
As per the issue #170 description , does jenkins call these scripts only during creation of PR ? or does it do it for all proceeding commits ?

ghost · 2018-11-16T15:49:38Z

Can we do some more testing for valid/invalid entries?

I'm not sure what you mean by this, do you want the Jenkins tests to be added to or the Jenkinsfile to be tested more? Or more commits with errors to be made to verify the errors are caught?

As per the issue #170 description , does jenkins call these scripts only during creation of PR ? or does it do it for all proceeding commits ?

Jenkins is rerun every commit, and can be forced to re-run by commenting on the PR (Note my comments above.) Right now I have it set to only run my commits automatically so that all other PR's don't get reported as failing, but once this is merged I will add all of us to the list for automatic running.

I have suggestion in addition to the lint we can do more ,
The current code base has a certain amount of warnings in linux build and CCS build . Can we use jenkins to ensure that the code which is getting added as part of PR doesn't add more warnings .

Yes, right now in addition to lint it builds the firmware (currently only with GCC, looking to add a Windows node and CCS build in the future) and runs the unit tests. For the build, it collects and reports all the warnings (330 right now...), but as far as I can tell there is no way to automate that it fails if new warnings appear, we would have to manually check status ourselves...will look into that some more, but that would be changes on Jenkins server side I think, not in the Jenkinsfile itself.
It also tracks testing coverage, something else I would like to see if we can only improve or maintain with each commit, but that also is just tracked right now and not automatically shown.

JoshuaJeyaraj · 2018-11-16T17:05:01Z

I'm not sure what you mean by this, do you want the Jenkins tests to be added to or the Jenkinsfile to be tested more? Or more commits with errors to be made to verify the errors are caught?

more commits with errors to be made to verify the errors are caught assuming it is not yet done .

but once this is merged I will add all of us to the list for automatic running.

Great !!

For the build, it collects and reports all the warnings (330 right now...), but as far as I can tell there is no way to automate that it fails if new warnings appear, we would have to manually check status ourselves...will look into that some more, but that would be changes on Jenkins server side I think, not in the Jenkinsfile itself.

Can we count the number of warning from the build logs , if it goes above 330 we can treat it as new warnings in the PR ?

ghost · 2018-11-16T18:59:12Z

rerun tests

ghost · 2018-11-16T19:09:34Z

Rerun tests

jspectacular · 2018-11-26T20:04:46Z

Tests OK

ghost · 2018-11-28T16:17:59Z

more commits with errors to be made to verify the errors are caught assuming it is not yet done

I'll add a few failed commits today that should show the red x next to them indicating Jenkins test failures.

Can we count the number of warning from the build logs , if it goes above 330 we can treat it as new warnings in the PR ?

I suppose could hardcode the jenkinsfile to compare number of warnings to 330, but I don't really like that solution. Let's get the repo running CI first and I'll create an issue to make Jenkins fail on new build warnings for a future PR.

rkfillman

Some of the discussion/comments here talk about upcoming work. Just want to make sure that its tracked somewhere so we don't forget about it.

ghost · 2018-12-03T21:01:52Z

Some of the discussion/comments here talk about upcoming work. Just want to make sure that its tracked somewhere so we don't forget about it.

Yup, added issue #210 last week to keep track of all the enhancements mentioned and other ones planned for future work.

mdlewisfb added 7 commits November 1, 2018 14:46

Adding Jenkinsfile for firmware

983791a

Updating jenkinsfile and adding lint script.

e187562

Updating Jenkinsfile and test/Makefile to support junit test reports.

bfb75c4

Updating clang_patch script to only check files modified by current PR

5161301

Updating Jenkinsfile for coverage, adding comment to clang_patch bash…

84586d8

… script

Adding src/main.c to commit to verify linter

5eb836e

Updating Jenkinsfile to track coverage results with Cobertura plugin.

d99c8bf

jspectacular requested review from jspectacular and markhor November 8, 2018 04:03

mdlewisfb added 2 commits November 12, 2018 09:10

Merge branch 'master' into jenkins_file

d7aa9e3

Fixing default LCOV_FILE in coverage.py script.

834bf4e

Updating coverage script to be more robust. Moving jenkins scripts to…

3309172

… utilities/jenkins dir to match schema PR #180.

Updating Jenkinsfile to point to new script dir

b2db5be

Adding warning cflags to ci build to track more warnings with Jenkins.

e95a47e

Updating clang_patch to compare to origin/master instead of local mas…

9119a75

…ter.

jspectacular requested a review from mayuri2016 November 14, 2018 19:29

ghost requested a review from dleang November 14, 2018 23:43

JoshuaJeyaraj self-requested a review November 15, 2018 12:07

JoshuaJeyaraj reviewed Nov 16, 2018

View reviewed changes

Adding headers to scripts and removing empty line from main.c

dd403f6

jspectacular approved these changes Nov 26, 2018

View reviewed changes

mdlewisfb added 4 commits November 28, 2018 08:22

Lint error for Jenkins to catch

b1be82e

Fix lint issue, add unit test failure for Jenkins to catch

f00aef2

Fixing unit test modification.

4ac7d07

Fixing last lint error

0003280

ghost requested a review from rkfillman December 3, 2018 17:41

rkfillman approved these changes Dec 3, 2018

View reviewed changes

ghost merged commit b05db18 into master Dec 3, 2018

ghost deleted the jenkins_file branch December 3, 2018 21:06

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jenkins Integration #186

Jenkins Integration #186

ghost commented Nov 1, 2018

ghost commented Nov 12, 2018

ghost commented Nov 12, 2018

ghost commented Nov 12, 2018

ghost commented Nov 12, 2018

ghost commented Nov 12, 2018

ghost commented Nov 12, 2018

ghost commented Nov 12, 2018

JoshuaJeyaraj left a comment •

edited

Loading

JoshuaJeyaraj Nov 15, 2018

ghost Nov 16, 2018

JoshuaJeyaraj Nov 15, 2018

JoshuaJeyaraj commented Nov 16, 2018 •

edited

Loading

Test Plan

ghost commented Nov 16, 2018

JoshuaJeyaraj commented Nov 16, 2018

ghost commented Nov 16, 2018

ghost commented Nov 16, 2018

jspectacular commented Nov 26, 2018

ghost commented Nov 28, 2018

rkfillman left a comment

ghost commented Dec 3, 2018

Jenkins Integration #186

Jenkins Integration #186

Conversation

ghost commented Nov 1, 2018

Description

Test Plan

Issues

ghost commented Nov 12, 2018

ghost commented Nov 12, 2018

ghost commented Nov 12, 2018

ghost commented Nov 12, 2018

ghost commented Nov 12, 2018

ghost commented Nov 12, 2018

ghost commented Nov 12, 2018

JoshuaJeyaraj left a comment • edited Loading

Choose a reason for hiding this comment

JoshuaJeyaraj Nov 15, 2018

Choose a reason for hiding this comment

ghost Nov 16, 2018

Choose a reason for hiding this comment

JoshuaJeyaraj Nov 15, 2018

Choose a reason for hiding this comment

JoshuaJeyaraj commented Nov 16, 2018 • edited Loading

Test Plan

ghost commented Nov 16, 2018

JoshuaJeyaraj commented Nov 16, 2018

ghost commented Nov 16, 2018

ghost commented Nov 16, 2018

jspectacular commented Nov 26, 2018

ghost commented Nov 28, 2018

rkfillman left a comment

Choose a reason for hiding this comment

ghost commented Dec 3, 2018

JoshuaJeyaraj left a comment •

edited

Loading

JoshuaJeyaraj commented Nov 16, 2018 •

edited

Loading