-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Collect code coverage after running every test case #219
Comments
We actually have already done something like this before, but to collect code coverage at the end of entire testing. I think it is a good idea to use code coverage for input generation. One idea I had was to first generates a lot of inputs guided by coverage running integration test. Once we have the list of inputs which can have large code coverage, then we run e2e tests to test their correctness. |
Great post @Spedoske ! I love the idea of collecting code coverage, but code coverage per test case is too fine-grained. If there is a way to aggregate to code coverage per test campaign, it will be more useful. |
So, that's a reason we didn't put it in the draft, but I think adding the code coverage support is very useful for us. |
We will need a deeper discussion on how to leverage some idea(s) from fuzzing to do smarter things. |
@Spedoske have you worked on this task? Just check. |
No. I am currently working on other tasks. If we want to push the project further, we can have a discussion first. |
What tasks are you focusing now? @tylergu and I want to prioritize this task. By I need to know what you are working on to be able to "prioritize." |
One example of making the operator to output coverage is written here: tylergu/zookeeper-operator@3d3a1e2 While the changes needed for each operator are different, I am going to document the highlevel steps at https://github.com/xlab-uiuc/acto/blob/main/docs/resource.md#measure-code-coverage-for-acto |
Thanks @tylergu ! This'll be super useful. |
Go 1.20 ("runtime/coverage" package) enable us to collect real-time coverage, which means we can collect coverage after each step in one single trial. Here is an example. |
WOW |
This looks very cool because it seems there is more standard procedures to collect coverage? |
Nice work! So you are going to turn the operator an http server to respond the coverage requests I guess? |
@tianyin , the difference this runtime/coverage makes is that, we can collect the coverage information while the operator is still running. Using my old method the coverage is only written to the disk when the operator is terminated. We still need to go through all the hacks of changing operators' source code and deployment script though. |
@tylergu got it -- thank you for the clarification. btw I don't really get why we call them "hacks" (which indicates that it was an ad hoc process). To me, collecting code coverage should be a standard practice, despite that different projects are different. ~t |
Here is demo of collecting code coverage at runtime. To collect coverage, the following code should add to the operator. We can also make the code as a module and use anonymous import to run the code. The coverage can be collected after every test case.
Coverage-guided testing can be done though this, but it should be hard. As we must wait for the states of the cluster converges. Therefore, the procedure of generating new tests must be very precise, so that we can discover new paths and improve code coverage using these new tests effectively.
The text was updated successfully, but these errors were encountered: