CI/CD | Determine why --fail-low-coverage Always Fails in Pipeline #53
Labels
enhancement
New feature or request
good first issue
Good for newcomers
help wanted
Extra attention is needed
Comments
ReedClanton
added
enhancement
ReedClanton
added a commit
that referenced
this issue
Apr 26, 2023
ReedClanton
added a commit
that referenced
this issue
Apr 26, 2023
* Going to get output() constants unit tests runnin in GitHub actions first. This is going to involve a **lot** of commits... * Pipeline didn't run, testing if yml file location was the issue. * Pipeline still not running... * Moved yml file to root of workflows dir. * GitHub says job name is invalid, changed it * same issue * same * same * same * same * Think I figured it out, can't use tabs. * Figuring out what's installed and what's not. * same * Working on getting shellspec installed * same * same * Working on isntalling shellspec on the runner * This won't work, but let's see how it breaks * testing out usage of sudo * this could work... * Holly shit... it worked. Moving on to getting the tests running * same * OMG... it's just working. Now working on getting Kcov installed on the runner. * Added sudo to Kcov install command * That worked. I'm on a roll. Now let's set shellspec to use Kcov * Still working on getting Kcov working with shellspec * same * Going to figure out why Kcov doesn't work latter, for now I'll test the unit tests on other shells * Tests running on bash, sh, and dash. Now working on isntalling BusyBox shell so tests may be run on that. * Busybox was installed, but not listed as a login shell, checking help doc of busybox * Checking where busybox was installed * Found install location of busybox, now running tests via that shell * Tests failed on busybox, checking version * BusyBox version should be okay... Now I gota figure out why shellspec thinks busyBox isn't supported. shellspec's docs say it should be. * Action now installed busybox prior to shellspec. I also moved the install of Kcov to before the instalation of shellspec as well. * Checking out the repo in a diffrent job won't work, but I'm trying it anyways. * Learning about GitHub Action environent variables * Fixed typos in action file * Fixed more of my f ups in the action file * Playing around with how echo works in github actions * more of the same * same * same * Implementing running of tests on ksh * Now run tests on ksh shell * Added zsh shell and running test via it * Made a copy of `output()`'s constants file that doesn't have an extention to see if that fixes the unit tests running via zsh shell * Still troubleshooting zsh tests * same * same * same * same * same * same * same * same * same * same * same * Moved yml file to see if GitHub will still run it. Will go back to troubleshooting zsh unit tests after I answer this * Back to troubleshooting tests run in zsh * same * same * same * same * same * same * Rifgured out that zsh doesn't like it when a varaible is set to the value of another... oooook. Still need to figure out why the readonly tests are failing * same * Same troubleshooting as I've been doing, however, I also refactored the `.gitignore` so it's organized and added the reports directory that gnerated by shellspec when it's run with -p. * Still troubleshooting same thing * same * same * same * same * same * same * same * same * same * same * same * same * same * same * same * same * same * same * same * same * same * same * same * same * same * same * Nothing I tried worked, so I'm going to trouble shoot it in a vm for a while. * Added basicly the same action yml file I've been working on, but modified to run unit tests of output() and createHeaderFooter(). * Commented out failing steps * Troubleshooting why unit tests of createHeaderFooter() are failing when run by sh but not bash. While typing this I relized what's going on. Ubuntu, the runner, has sh point to dash rather than bash. So bash can run the tests, but dash can't * I don't want to pass printf commands through in the end, but I'm doing it for now to see if that fixes this issue. * I didn't fix it. * Troubleshooting why one output() test is failing on the runner. * Fixed path in unit test yml file for output().\ * Removed steps that ran tests under sh because other steps already run them under dash and sh points to dash. * Added directory that contains OS iso files to list of things backUp() won't backup. * Added GitHub action that runs style guide checker against createHeaderFooter() and updated createHeaderFooter() so it passes the style guide check. I also changed the Kcov passing percentage so the min % coverage is effectivly 100% with the new code changes. * Changed step that installs shfmt on the runner to use snap rather than apt-get * Updated output()'s constants file so it passes the style guide check, added a GitHub action to run the style guide check of that constants file, and corrected some typos in the action file that runs the style guide check of createHeaderFooter() * Checking if zsh tests are still broken and added blank line at end of cosntants file in order to make sure that the style guide check fails. * Undid last commit's changes. * Added debugging stuff to action file * Troubleshooting unit tests in busybox shell. * same * same * same * same * Revampted output() constants unit tests so they will run on both bash and zsh shells. * Commented out busybox stuff and uncommented zsh unit tests * Added default `shellspec` and `kcov` options to `.shellspec`, removed those options from action files, and removed option that doesn't do anything from commented out lines in action files. * Added option that I thought didn't do anything back, made more changes to default `shellspec` options, and `kcov` install command should work now, although it's still commented out. * Removed `shellspec` option I forgot to remove in the last commit, and un-commented stuff so i can start testing `shellspec` with `kcov`. * Added `shellspec` option. * Pipeline step is successful, but return code is not 0... troubleshooting this. * same * same * same. * same * same. * same. * same * same * same * same * same * same * same * same * same * same * same * same * The --fail-low-coverage option is what's causing the failure. Even though coverage is 100%... * Testing if the issue is that coverage is 100% * Still troubleshooting. * same * Updating action files with what I've learned. * Told `shellspec` to pass through cat command, however, I'll need to mock out cat in every test because they all will be using cat and I don't really want to pass cat commands through. I have to becuase even though I'm mocking out cat, `shellspec` says stuff fails if I don't tell it to pass it through. This is despite the fact that tests pass. Basiclly, I have to mock out cat in every test and tell `shellspec` to pass cat calls through because `shellspec` has a bug (I think). * While I was writting the last commit message I relized that if I'm going to need to always mock out the cat command, then I should do it in spec_helper.sh, so I did. I then removed mocking from each test that I had added it to. * Revampted text that's checked upon failure of output() tests so i don't have to correctly mock out cat. Converted output() doc over to new standard so tests will have a chance of passing on dash shells. * Corrected spacing. * The `shellspec` tool doesn't work correctly when it runs tests via a dash shell, I'm now going to move on to testing createHeaderFooter() test via a ksh shell. * createHeaderFooter() tests run via a ksh shell, moving on to zsh * Broke up running of unit tests into diffrent jobs, one for each shell. This should allow me to run `shellspec` directly from mocos. My hope is that this will allow the zsh tests to pass. * Troubleshooting why `shellspec` install works, but can't be found. * same * same * same * same * same * same * same * same * same * same * same * same * same * same * same * same * Updated example script with what I've learned working this story, refactored createHeaderFooter() so it will work via dash shells, and added skips in shellspec tests so those tests will pass when run via a dash shell. This was done because shellspec removes the trailing '\n' from the end of the line returned by createHeaderFooter() (no idea why). I manually tested the method in a dash shell and it worked as expected, so it's a shellspec issue. * Refactored output() so it's closer to being POSIX complient, removed debug code from createHeaderFooter() and fixed issues related style checker, and moved mocking of inScriptSource() to spec_helper.sh and removed it from each of the output() unit tests. * Un-commented running of unit tests via dash on macos for createHeaderFooter() to see if fixing them on ubuntu also fixed them on macos. * Refactored GitHub action file that runs unit tests of output()'s constants file so it's in line with what I've done in the one for createHeaderFooter(). I also should have fixed the last issues with style guide violations in createHeaderFooter(). * Fixed error in constants unit test action file * Corrected two more issues caused by copy pasting * Removed debug code from action files. I also removed the sleep because I don't think it helps. At some point I'll try calling shellspec with a full path. * Refactored output() unit test action file so it's in-line with the rest. * Troubleshooting why pipeline sometimes says shellspec isn't installed. If it's because PATH hasn't been updated then I can just call it with the full path. * Refactored output() so it's a little more POSIX complient, and used full path to shellspec to see if that fixes pipeline reliability issue. * Using full path to shellspec has worked so far, so I updated all action files. * Turns out the issue is a server side one on the server that hosts shellspec. I'll need to either have the step that installs shellspec check if it was really installed, or just run everything on a docker image that already has it installed. I'm not going to bother with this for now because I'm thinking I'll be doing the latter. * Removed debug code that was breaking the pipeline * Troubleshooting why date mock test of createHeaderFooter() is failing when run via zsh shell on macos. * Tests work locally when I run them via zsh. Now testing if they work on an Ubuntu runner via zsh * The issue is that zsh doesn't support readonly. I'm now testing out local -r * Cleaned up last changes * Simplified changes made in last few commits. * Minor changes to bring output() a little closer to passing the style guide check. Also implemented readonly cahnges made to createHeaderFooter() in otuput(). * Minor updates to example script to bring it more in line with what the style checker is looking for. * Not very motivated today, so I'm just playing around with docs. * Changed link to diagram so it should work in testing. Will need to change prior to merging. Also re-rendered the diagram. * Playing around with how I link to diagrams. * same * same * Figured out linking and fixed a broken link in another README.md. * Updated diagram so it reflect directory structure and added title to README.md. * Corrected diagram export * Created diagram for log() in the same format as I just did for output() * Created diagram and README.md for backUp() and added links to the README.md files I've been creating to the README.md one level up. * Missed this file in last commit. * Moved checkRequriedOpts() into a util directory, updated all uses of it so they don't break, and made minor formatting changes across many files so as to bring them closer in line with what the style checker will look for when I implement it for those files. * Updated log diagram to reflect the moving of checkRequiredOpts() done by last commit. * Updated diagrams to reflect changes made in next few commits. * Updated log() and backUp() to have a main.sh that handles sourcing. Also made a few simplifications to the code and fixed return value variable name bug in backUp. * Removed top section because it's not used. Also refactored how the method doc is saved so it's POSIX complient. I also did that for the files changed in the last commit. * Updated top secton to reflect the new way I'm dealing with imports, made the method doc saving POSIX complient, and simplified some code. * Updated main.sh so it will be easier to use it as a template for other functions that use main.sh. Updated how method doc is saved so it's POSIX complient and refactored one use of [[ ]] because that's not POSIX complient. * Changed how method docs are saved to be POSIX complient, removed top section when it wasn't used, and made minor changes so code is closer to being POSIX complient. * Refactored output() so it no longer uses arrays. Also made a bunch of changes to it in order to make it more posix complient and more in line with what the style checker is looking for. I also removed some duplicate code from the unit test of log level character (I moved that code to spec_helper a while ago). I also changed output() so you don't have to provide message text, but you do still need to provided at least one message option. I had to update some tests to reflect this change. shellspec can't deal with stdout that's just new lines, so I added skips in the new tests. * Refactored output() to be a function and POSIX complient and Refactored tests. Added pass through of cut and rm commands because they generally work the same on all shells and it would be a LOT of work to mock them out (for little to no benefit). Removed isNotBash() from spec_helper because I changed the CUT so it works on all shells (so it's not needed). I also added return code 200 global. Refactored output() to be a function. This required a LOT of changes. This was done because it encapsulates the code. I hadn't been doing it that way because I thought POSIX didn't support functions, but POSIX just doesn't support the 'function' keyword. I refactored tests such that invalid input tests are in a different file. I did this because: - Most of those tests are almost identical so to create more in the future, I can just make a copy of one of the existing ones as the base for those sorts of test in the future - It splits up some of the largest test files - It should make it easier to make sweeping changes to those tests in the future if need be I refactored how test files are named so the file names match the 'Describe" titles. Refactored backUp() and log() to use a main.sh to handle importing needed stuff so that code is more like output(). Refactored output() to be fully POSIX complaint. Also updated it so it returns more helpful log messages and more accurate return codes. Refactored createHeaderFooter() so it no longer include a new line character at the end of it's stdout. This was done because dash removes the new line character. Updated code coverage min percentage to reflect what it currently is for output() and createHeaderFooter(). * Updated output() so it should pass the style guide checker now. * Testing out the uses keyword and testing the ability to run actions from GitHub interface. * Updated all actions to use the action that installs shellspec. Created action that installs Kcov and shellspec and updated actions to use that as well. Finally, added 'workflow_dispatch' to 'on:' in each action. Still don't really know what it does (I think it allows you to run them from the GitHub UI, but I don't see where). Also added descriptions to existing actions. * Removed descriptions for existing actions. It broke the pipeline. * Broke out the instilation of zsh and ksh shells into actions and updated unit test actions to use them. * Created action for isntalling shfmt (style guide checking tool) and updated style checking actions to use it. * Removed commented out ash shell tests because shellspec says it supports ash but it doesn't, and corrected copy past errors in shfmt install action. * Reduced default max line length in a number of tests so if I want to test that in the future, test input won't take up as much space. * Added retry to shellspec install because shellspec's server is so un-reliable. * Troubleshooting why manuall retry of shellspec install didn't work * same * same * same * Got it working, now tweaking retry values to get it as reliable as possible. * same * same * Retrying the manual way to see if that works better * same * Implemented manual retrys in kcov install as well. * For some reason doing manual retries is reliable, but using nick-fields/retry is not, so I'm implementing retry for everything the manual way. * Implemented retrys for ksh shell install * Retries are now implemented across all common actions. I also removed one retry from shellspec install. If it ends up be un-reliable, I'll add it back. * Removed commended out code from action file. * Removed commended out code from action file. * Removed commended out code from action file. * Removed commended out code from action file. * Removed commended out code from action file. * Apply suggestions from code review * Update src/shell/functions/backUp/backUp.sh * Removed usage of type -t and replaced it witht eh POSIX complient command -v. Also switched example_function.sh to use return rather than exit. * Added TODO comments for #53. * Updated comment to specify what shells don't handle readonly like the rest. * Update src/.shellrc
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Currently when unit tests are run by GitHub actions, code coverage is correctly reported and minimum coverage is met, however, when
--fail-low-coveragestill causes a failure. Figure out why and fix it so the pipeline fails when coverage is under limit but not when over.High Level Solution Description
One way I've thought of is to have logic in the GitHub action file that processes unit test output and checks manually if coverage was met.
Also, to enable
--fail-low-coverage, uncomment it in.shellspec.Alternative Solution(s)
None that I know of.
Blocked By
Blocking
Related Issue(s)
Additional Context
None.
The text was updated successfully, but these errors were encountered: