diff --git a/content/guides/testing.mdx b/content/guides/testing.mdx index a66eed54..6722dc6b 100644 --- a/content/guides/testing.mdx +++ b/content/guides/testing.mdx @@ -5,7 +5,7 @@ description: | The main focus will be testing but we also tacke other validation methods such as fuzzing and symbolic execution. --- -## 00. The Concept of Testing +## The Concept of Testing Before diving into how we can do testing on Unikraft, let’s first focus on several key concepts that are used when talking about testing. @@ -57,7 +57,7 @@ An example of a program being symbolically executed can be seen in the figure be The most popular symbolic execution engines are [`KLEE`](https://klee.github.io/), [`S2E`](https://s2e.systems/docs/) and [`angr`](https://angr.io/). -## 01. Existing Testing Frameworks +## Existing Testing Frameworks Nowadays, testing is usually done using a framework. There is no single testing framework that can be used for everything but one has plenty of options to chose from. @@ -186,7 +186,7 @@ int main(int argc, char ∗∗argv) { Easy? This is not always the case, for example this [sample](https://github.com/google/googletest/blob/master/googletest/samples/sample9_unittest.cc) shows a more advanced and nested test. -## 02. Unikraft's Testing Framework +## Unikraft's Testing Framework Unikraft's testing framework, [`uktest`](https://github.com/unikraft/unikraft/tree/staging/lib/uktest), has been inspired by KUnit and provides a flexible testing API. @@ -213,7 +213,7 @@ UK_TESTCASE(testsuite_name, testcase2_name) } ``` -## 03. The Design behind Unikraft's Testing Framework +## The Design behind Unikraft's Testing Framework The key ideas that were followed when writing `uktest` are: @@ -316,320 +316,6 @@ int uk_testsuite_run(struct uk_testsuite *suite) } ``` -## Work Items - -In this work session we will go over writing and running tests for Unikraft. -We will use `uktest` and [`Google Test`](https://github.com/google/googletest). -Make sure you are on the `usoc21` branch on the core Unikraft repo and `staging` on all others. -`uktest` should be enabled from the Kconfig. - -### Support Files - -Session support files are available on [this GitHub repo](https://github.com/unikraft-upb/guides-exercises/tree/master/testing). -You can use your own setup or start from the files in `work/`. -If you get stuck take a peek at the solutions in `sol/`. - -### 01. Tutorial: Testing a Simple Application - -We will begin this session with a very simple example. -We can use the `app-helloworld` as a starting point. -In `main.c` remove all the existing code. -The next step is to include `uk/test.h` and define the factorial function: - -```C++ -#include - -int factorial(int n) { - int result = 1; - for (int i = 1; i <= n; i++) { - result *= i; - } - - return result; -} -``` - -We are now ready to add a test suite with a test case: - -```C++ -UK_TESTCASE(factorial_testsuite, factorial_test_positive) -{ - UK_TEST_EXPECT_SNUM_EQ(factorial(2), 2); -} - -uk_testsuite_register(factorial_testsuite, NULL); -``` - -When we run this application, we should see the following output: - -```console -test: factorial_testsuite->factorial_test_positive - : expected `factorial(2)` to be 2 but was 2 ....................................... [ PASSED ] -``` - -Throughout this session we will extend this simple app that we have just written. - -### 02. Adding a New Test Suite - -For this task, you will have to modify the existing factorial application by adding a new function that computes if a number is prime. -Add a new testsuite for this function. - -### 03. Tutorial: Testing vfscore - -We begin by adding a new file for the tests called `test_stat.c` in a newly created folder `tests` in the `vfscore` internal library: - -```Makefile -LIBVFSCORE_SRCS-$(CONFIG_LIBVFSCORE_TEST_STAT) += \ - $(LIBVFSCORE_BASE)/tests/test_stat.c -``` - -We then add the menuconfig option in the `if LIBVFSCORE` block: - -```KConfig -menuconfig LIBVFSCORE_TEST - bool "Test vfscore" - select LIBVFSCORE_TEST_STAT if LIBUKTEST_ALL - default n - -if LIBVFSCORE_TEST - -config LIBVFSCORE_TEST_STAT - bool "test: stat()" - select LIBRAMFS - default n - -endif -``` - -And finally add a new testsuite with a test case. - -```C++ -#include - -#include -#include -#include -#include -#include - -typedef struct vfscore_stat { - int rc; - int errcode; - char *filename; -} vfscore_stat_t; - -static vfscore_stat_t test_stats [] = { - { .rc = 0, .errcode = 0, .filename = "/foo/file.txt" }, - { .rc = -1, .errcode = EINVAL, .filename = NULL }, -}; - -static int fd; - -UK_TESTCASE(vfscore_stat_testsuite, vfscore_test_newfile) -{ - /* First check if mount works all right */ - int ret = mount("", "/", "ramfs", 0, NULL); - UK_TEST_EXPECT_SNUM_EQ(ret, 0); - - ret = mkdir("/foo", S_IRWXU); - UK_TEST_EXPECT_SNUM_EQ(ret, 0); - - fd = open("/foo/file.txt", O_WRONLY | O_CREAT, S_IRWXU); - UK_TEST_EXPECT_SNUM_GT(fd, 2); - - UK_TEST_EXPECT_SNUM_EQ( - write(fd, "hello\n", sizeof("hello\n")), - sizeof("hello\n") - ); - fsync(fd); -} - -/* Register the test suite */ -uk_testsuite_register(vfscore_stat_testsuite, NULL); -``` - -We will be using a simple app without any main function to run the testsuite, the output should be similar with: - -```console -test: vfscore_stat_testsuite->vfscore_test_newfile - : expected `ret` to be 0 but was 0 ................................................ [ PASSED ] - : expected `ret` to be 0 but was 0 ................................................ [ PASSED ] - : expected `fd` to be greater than 2 but was 3 .................................... [ PASSED ] - : expected `write(fd, "hello\n", sizeof("hello\n"))` to be 7 but was 7 ............ [ PASSED ] -``` - -### 04. Add a Test Suite for `nolibc` - -Add a new test suite for nolibc with four test cases in it. -You can use any POSIX function from nolibc for this task. -Feel free to look over the [documentation](https://github.com/unikraft/unikraft/blob/staging/lib/uktest/README.md) to write more complex tests. - -### 05. Tutorial: Running Google Test on Unikraft - -For this tutorial, we will use Google Test under Unikraft. -Aside from `lib-googletest`, we'll also need to have `libcxx`, `libcxxabi`, `libunwind`, `compiler-rt` and `newlib` because we're testing C++ code. - -The second step is to enable the Google Test library and its config option `Build google test with main`. - -We can now add a new cpp file, `main.cpp`. -Make sure that the files end in `.cpp` and not `.c`, otherwise you'll get lots of errors. -In the source file we'll include `gtest/gtest.h` -We will now be able to add our factorial function and test it. - -```C++ -int Factorial(int n) { - int result = 1; - for (int i = 1; i <= n; i++) { - result *= i; - } - - return result; -} - -TEST(FactorialTest, Negative) { - EXPECT_EQ(1, Factorial(-5)); - EXPECT_EQ(1, Factorial(-1)); - EXPECT_GT(Factorial(-10), 0); -} -``` - -If we run our unikernel, we should see the following output: - -```console -[==========] Running 1 test from 1 test case. -[----------] Global test environment set-up. -[----------] 1 test from FactorialTest -[ RUN ] FactorialTest.Negative -[ OK ] FactorialTest.Negative (0 ms) -[----------] 1 test from FactorialTest (0 ms total) - -[----------] Global test environment tear-down -[==========] 1 test from 1 test case ran. (0 ms total) -[ PASSED ] 1 test. -``` - -We can see that in this case, the tests are being run after the main call, not before! - -### 06. Tutorial (Bonus): Using KLEE for Symbolic Execution - -One of the most popular symbolic execution engine is [KLEE](https://klee.github.io/). -For convenience, we'll be using Docker. - -```Bash -docker pull klee/klee:2.1 -docker run --rm -ti --ulimit='stack=-1:-1' klee/klee:2.1 -``` - -Let's look over this regular expression program, can you spot any bugs? -We'll create a file `ex.c` with this code: - -```C++ -#include - -static int matchhere(char*,char*); - -static int matchstar(int c, char *re, char *text) { - do { - if (matchhere(re, text)) - return 1; - } while (*text != '\0' && (*text++ == c || c== '.')); - return 0; -} - -static int matchhere(char *re, char *text) { - if (re[0] == '\0') - return 0; - if (re[1] == '*') - return matchstar(re[0], re+2, text); - if (re[0] == '$' && re[1]=='\0') - return *text == '\0'; - if (*text!='\0' && (re[0]=='.' || re[0]==*text)) - return matchhere(re+1, text+1); - return 0; -} - -int match(char *re, char *text) { - if (re[0] == '^') - return matchhere(re+1, text); - do { - if (matchhere(re, text)) - return 1; - } while (*text++ != '\0'); - return 0; -} - -#define SIZE 7 - -int main(int argc, char **argv) { - char re[SIZE]; - - int count = read(0, re, SIZE - 1); - //klee_make_symbolic(re, sizeof re, "re"); - - int m = match(re, "hello"); - if (m) printf("Match\n", re); - - return 0; -} -``` - -Now, let's run this program symbolically. -To do this, we'll uncomment the `klee_make_symbol` line, and comment the line with `read` and `printf`. -We'll compile the program with `clang` this time: - -```Bash -clang -c -g -emit-llvm ex.c -``` - -And run it with KLEE: - -```Bash -klee ex.bc -``` - -We'll see the following output: - -```console -KLEE: output directory is "/home/klee/klee-out-4" -KLEE: Using STP solver backend -KLEE: ERROR: ex1.c:13: memory error: out of bound pointer -KLEE: NOTE: now ignoring this error at this location -KLEE: ERROR: ex1.c:15: memory error: out of bound pointer -KLEE: NOTE: now ignoring this error at this location - -KLEE: done: total instructions = 5314314 -KLEE: done: completed paths = 7692 -KLEE: done: generated tests = 6804 -``` - -This tells us that KLEE has found two memory errors. -It also gives us some info about the number of paths and instructions executed. -After the run, a folder `klee-last` has been generated that contains all the test cases. -We want to find the ones that generated memory errors: - -```console -klee@affd7769bb39:~/klee-last$ ls | grep err -test000018.ptr.err -test000020.ptr.err -``` - -We look at testcase 18: - -```console -klee@affd7769bb39:~/klee-last$ ktest-tool test000018.ktest -ktest file : 'test000018.ktest' -args : ['ex1.bc'] -num objects: 1 -object 0: name: 're' -object 0: size: 7 -object 0: data: b'^\x01*\x01*\x01*' -object 0: hex : 0x5e012a012a012a -object 0: text: ^.*.*.* -``` - -This is just a quick example of the power of symbolic execution, but it comes with one great problem: path explosion. -When we have more complicated programs that have unbounded loops, the number of paths grows exponentially and thus symbolic execution is not viable anymore. - ## Further Reading * [6.005 Reading 3: Test](https://ocw.mit.edu/ans7870/6/6.005/s16/classes/03-testing/index.html#automated_testing_and_regression_testing)