Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Collect code coverage after running every test case #219

Open
Tracked by #232 ...
Spedoske opened this issue Jun 3, 2023 · 15 comments
Open
Tracked by #232 ...

Collect code coverage after running every test case #219

Spedoske opened this issue Jun 3, 2023 · 15 comments
Labels
high-pri high priority task that blocks the progress In-progress The task is working in process at the moment task

Comments

@Spedoske
Copy link
Collaborator

Spedoske commented Jun 3, 2023

Here is demo of collecting code coverage at runtime. To collect coverage, the following code should add to the operator. We can also make the code as a module and use anonymous import to run the code. The coverage can be collected after every test case.

package main

// See https://go.dev/testing/coverage/
// Build with -cover
// go 1.20+ required

// $ mkdir coverage
// $ go run -cover main.go
// $ curl localhost:8888
// $ curl localhost:8888/dump/coverage
// $ go tool covdata textfmt -o foo -i=coverage
// $ go tool covdata percent -i=coverage
// $ curl localhost:8888 --cookie "bar=baz"
// $ curl localhost:8888/dump/coverage
// $ go tool covdata textfmt -o bar -i=coverage
// $ go tool covdata percent -i=coverage
import (
	"net/http"
	"runtime/coverage"
)

const CoverageDir = "./coverage"

func dumpCoverage(http.ResponseWriter, *http.Request) {
	err := coverage.WriteCountersDir(CoverageDir)
	if err != nil {
		panic(err)
	}
}

func handle(rw http.ResponseWriter, r *http.Request) {
	if len(r.Cookies()) == 0 {
		_, err := rw.Write([]byte("Foo"))
		if err != nil {
			panic(err)
		}
		return
	}
	_, err := rw.Write([]byte("Bar"))
	if err != nil {
		panic(err)
	}
}

func main() {
	err := coverage.WriteMetaDir(CoverageDir)
	if err != nil {
		panic(err)
	}
	http.HandleFunc("/", handle)
	http.HandleFunc("/dump/coverage", dumpCoverage)
	err = http.ListenAndServe(":8888", nil)
	if err != nil {
		panic(err)
	}
}

Coverage-guided testing can be done though this, but it should be hard. As we must wait for the states of the cluster converges. Therefore, the procedure of generating new tests must be very precise, so that we can discover new paths and improve code coverage using these new tests effectively.

@tylergu
Copy link
Member

tylergu commented Jun 3, 2023

We actually have already done something like this before, but to collect code coverage at the end of entire testing.
We ended up having to rewrite the main functions for each operator to collect code coverage for e2e tests.
For example, collecting CassOp's codecov: https://github.com/xlab-uiuc/acto/blob/main/data/cass-operator/config_cov.json
It turned out that our coverage is not as good as existing operators' tests, I think it is mainly because the codecov is harder to achieve for e2e test versus unittest.

I think it is a good idea to use code coverage for input generation.
One of the challenges is that e2e tests are too expensive to try a lot of inputs.

One idea I had was to first generates a lot of inputs guided by coverage running integration test. Once we have the list of inputs which can have large code coverage, then we run e2e tests to test their correctness.

@tianyin
Copy link
Member

tianyin commented Jun 3, 2023

Great post @Spedoske !

I love the idea of collecting code coverage, but code coverage per test case is too fine-grained. If there is a way to aggregate to code coverage per test campaign, it will be more useful.

@tianyin
Copy link
Member

tianyin commented Jun 3, 2023

It turned out that our coverage is not as good as existing operators' tests, I think it is mainly because the codecov is harder to achieve for e2e test versus unittest.

So, that's a reason we didn't put it in the draft, but I think adding the code coverage support is very useful for us.

@tianyin
Copy link
Member

tianyin commented Jun 3, 2023

Coverage-guided testing can be done though this, but it should be hard. As we must wait for the states of the cluster converges. Therefore, the procedure of generating new tests must be very precise, so that we can discover new paths and improve code coverage using these new tests effectively.

One idea I had was to first generates a lot of inputs guided by coverage running integration test. Once we have the list of inputs which can have large code coverage, then we run e2e tests to test their correctness.

We will need a deeper discussion on how to leverage some idea(s) from fuzzing to do smarter things.

@tianyin
Copy link
Member

tianyin commented Jul 4, 2023

@Spedoske have you worked on this task? Just check.

@Spedoske
Copy link
Collaborator Author

Spedoske commented Jul 4, 2023

@Spedoske have you worked on this task? Just check.

No. I am currently working on other tasks. If we want to push the project further, we can have a discussion first.

@tianyin
Copy link
Member

tianyin commented Jul 4, 2023

What tasks are you focusing now?

@tylergu and I want to prioritize this task. By I need to know what you are working on to be able to "prioritize."

@tianyin tianyin added the high-pri high priority task that blocks the progress label Jul 4, 2023
@tianyin tianyin changed the title [Proposal] Collect code coverage after running every test case Collect code coverage after running every test case Jul 4, 2023
@tianyin tianyin added the task label Jul 4, 2023
@tianyin tianyin added the In-progress The task is working in process at the moment label Jul 4, 2023
@tylergu
Copy link
Member

tylergu commented Jul 9, 2023

One example of making the operator to output coverage is written here: tylergu/zookeeper-operator@3d3a1e2

While the changes needed for each operator are different, I am going to document the highlevel steps at https://github.com/xlab-uiuc/acto/blob/main/docs/resource.md#measure-code-coverage-for-acto

@tianyin
Copy link
Member

tianyin commented Jul 9, 2023

Thanks @tylergu ! This'll be super useful.

@Spedoske
Copy link
Collaborator Author

Spedoske commented Jul 9, 2023

Go 1.20 ("runtime/coverage" package) enable us to collect real-time coverage, which means we can collect coverage after each step in one single trial.

Here is an example.
https://gitlab.isp.moe/Spedoske/cluster-operator/-/commit/0f8e33cb615c894b030504f8fe0b16902d3afd05

@tianyin
Copy link
Member

tianyin commented Jul 9, 2023

WOW

@tianyin
Copy link
Member

tianyin commented Jul 9, 2023

This looks very cool because it seems there is more standard procedures to collect coverage?

@tylergu
Copy link
Member

tylergu commented Jul 9, 2023

Nice work! So you are going to turn the operator an http server to respond the coverage requests I guess?

@tylergu
Copy link
Member

tylergu commented Jul 9, 2023

This looks very cool because it seems there is more standard procedures to collect coverage?

@tianyin , the difference this runtime/coverage makes is that, we can collect the coverage information while the operator is still running. Using my old method the coverage is only written to the disk when the operator is terminated.

We still need to go through all the hacks of changing operators' source code and deployment script though.

@tianyin
Copy link
Member

tianyin commented Jul 9, 2023

@tylergu got it -- thank you for the clarification.

btw I don't really get why we call them "hacks" (which indicates that it was an ad hoc process). To me, collecting code coverage should be a standard practice, despite that different projects are different.

~t

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
high-pri high priority task that blocks the progress In-progress The task is working in process at the moment task
Projects
None yet
Development

No branches or pull requests

3 participants