Skip to content

Commit

Permalink
guides: Remove indexes and work items in testing guide
Browse files Browse the repository at this point in the history 
Actual testing guide from hackathons is not aligned with the others. It has indexes and work items.
Remove all indexes and all work items in order to maintain consistency.

Signed-off-by: Alin Andrei Abahnencei <[email protected]>
  • Loading branch information
@alinandrei2004
alinandrei2004 committed Mar 2, 2024
1 parent 9602143 commit 196a00f
Showing 1 changed file with 4 additions and 318 deletions.
322 changes: 4 additions & 318 deletions content/guides/testing.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: |
The main focus will be testing but we also tacke other validation methods such as fuzzing and symbolic execution.
---

## 00. The Concept of Testing
## The Concept of Testing

Before diving into how we can do testing on Unikraft, let’s first focus on several key concepts that are used when talking about testing.

Expand Down Expand Up @@ -57,7 +57,7 @@ An example of a program being symbolically executed can be seen in the figure be

The most popular symbolic execution engines are [`KLEE`](https://klee.github.io/), [`S2E`](https://s2e.systems/docs/) and [`angr`](https://angr.io/).

## 01. Existing Testing Frameworks
## Existing Testing Frameworks

Nowadays, testing is usually done using a framework.
There is no single testing framework that can be used for everything but one has plenty of options to chose from.
Expand Down Expand Up @@ -186,7 +186,7 @@ int main(int argc, char ∗∗argv) {
Easy?
This is not always the case, for example this [sample](https://github.com/google/googletest/blob/master/googletest/samples/sample9_unittest.cc) shows a more advanced and nested test.

## 02. Unikraft's Testing Framework
## Unikraft's Testing Framework

Unikraft's testing framework, [`uktest`](https://github.com/unikraft/unikraft/tree/staging/lib/uktest), has been inspired by KUnit and provides a flexible testing API.

Expand All @@ -213,7 +213,7 @@ UK_TESTCASE(testsuite_name, testcase2_name)
}
```

## 03. The Design behind Unikraft's Testing Framework
## The Design behind Unikraft's Testing Framework

The key ideas that were followed when writing `uktest` are:

Expand Down Expand Up @@ -316,320 +316,6 @@ int uk_testsuite_run(struct uk_testsuite *suite)
}
```
## Work Items
In this work session we will go over writing and running tests for Unikraft.
We will use `uktest` and [`Google Test`](https://github.com/google/googletest).
Make sure you are on the `usoc21` branch on the core Unikraft repo and `staging` on all others.
`uktest` should be enabled from the Kconfig.
### Support Files
Session support files are available on [this GitHub repo](https://github.com/unikraft-upb/guides-exercises/tree/master/testing).
You can use your own setup or start from the files in `work/`.
If you get stuck take a peek at the solutions in `sol/`.
### 01. Tutorial: Testing a Simple Application
We will begin this session with a very simple example.
We can use the `app-helloworld` as a starting point.
In `main.c` remove all the existing code.
The next step is to include `uk/test.h` and define the factorial function:
```C++
#include <uk/test.h>
int factorial(int n) {
int result = 1;
for (int i = 1; i <= n; i++) {
result *= i;
}
return result;
}
```

We are now ready to add a test suite with a test case:

```C++
UK_TESTCASE(factorial_testsuite, factorial_test_positive)
{
UK_TEST_EXPECT_SNUM_EQ(factorial(2), 2);
}

uk_testsuite_register(factorial_testsuite, NULL);
```
When we run this application, we should see the following output:
```console
test: factorial_testsuite->factorial_test_positive
: expected `factorial(2)` to be 2 but was 2 ....................................... [ PASSED ]
```

Throughout this session we will extend this simple app that we have just written.

### 02. Adding a New Test Suite

For this task, you will have to modify the existing factorial application by adding a new function that computes if a number is prime.
Add a new testsuite for this function.

### 03. Tutorial: Testing vfscore

We begin by adding a new file for the tests called `test_stat.c` in a newly created folder `tests` in the `vfscore` internal library:

```Makefile
LIBVFSCORE_SRCS-$(CONFIG_LIBVFSCORE_TEST_STAT) += \
$(LIBVFSCORE_BASE)/tests/test_stat.c
```

We then add the menuconfig option in the `if LIBVFSCORE` block:

```KConfig
menuconfig LIBVFSCORE_TEST
bool "Test vfscore"
select LIBVFSCORE_TEST_STAT if LIBUKTEST_ALL
default n
if LIBVFSCORE_TEST
config LIBVFSCORE_TEST_STAT
bool "test: stat()"
select LIBRAMFS
default n
endif
```

And finally add a new testsuite with a test case.

```C++
#include <uk/test.h>

#include <fcntl.h>
#include <errno.h>
#include <unistd.h>
#include <sys/stat.h>
#include <sys/mount.h>

typedef struct vfscore_stat {
int rc;
int errcode;
char *filename;
} vfscore_stat_t;

static vfscore_stat_t test_stats [] = {
{ .rc = 0, .errcode = 0, .filename = "/foo/file.txt" },
{ .rc = -1, .errcode = EINVAL, .filename = NULL },
};

static int fd;

UK_TESTCASE(vfscore_stat_testsuite, vfscore_test_newfile)
{
/* First check if mount works all right */
int ret = mount("", "/", "ramfs", 0, NULL);
UK_TEST_EXPECT_SNUM_EQ(ret, 0);

ret = mkdir("/foo", S_IRWXU);
UK_TEST_EXPECT_SNUM_EQ(ret, 0);

fd = open("/foo/file.txt", O_WRONLY | O_CREAT, S_IRWXU);
UK_TEST_EXPECT_SNUM_GT(fd, 2);

UK_TEST_EXPECT_SNUM_EQ(
write(fd, "hello\n", sizeof("hello\n")),
sizeof("hello\n")
);
fsync(fd);
}

/* Register the test suite */
uk_testsuite_register(vfscore_stat_testsuite, NULL);
```
We will be using a simple app without any main function to run the testsuite, the output should be similar with:
```console
test: vfscore_stat_testsuite->vfscore_test_newfile
: expected `ret` to be 0 but was 0 ................................................ [ PASSED ]
: expected `ret` to be 0 but was 0 ................................................ [ PASSED ]
: expected `fd` to be greater than 2 but was 3 .................................... [ PASSED ]
: expected `write(fd, "hello\n", sizeof("hello\n"))` to be 7 but was 7 ............ [ PASSED ]
```
### 04. Add a Test Suite for `nolibc`
Add a new test suite for nolibc with four test cases in it.
You can use any POSIX function from nolibc for this task.
Feel free to look over the [documentation](https://github.com/unikraft/unikraft/blob/staging/lib/uktest/README.md) to write more complex tests.
### 05. Tutorial: Running Google Test on Unikraft
For this tutorial, we will use Google Test under Unikraft.
Aside from `lib-googletest`, we'll also need to have `libcxx`, `libcxxabi`, `libunwind`, `compiler-rt` and `newlib` because we're testing C++ code.
The second step is to enable the Google Test library and its config option `Build google test with main`.
We can now add a new cpp file, `main.cpp`.
Make sure that the files end in `.cpp` and not `.c`, otherwise you'll get lots of errors.
In the source file we'll include `gtest/gtest.h`
We will now be able to add our factorial function and test it.
```C++
int Factorial(int n) {
int result = 1;
for (int i = 1; i <= n; i++) {
result *= i;
}

return result;
}

TEST(FactorialTest, Negative) {
EXPECT_EQ(1, Factorial(-5));
EXPECT_EQ(1, Factorial(-1));
EXPECT_GT(Factorial(-10), 0);
}
```
If we run our unikernel, we should see the following output:
```console
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from FactorialTest
[ RUN ] FactorialTest.Negative
[ OK ] FactorialTest.Negative (0 ms)
[----------] 1 test from FactorialTest (0 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (0 ms total)
[ PASSED ] 1 test.
```
We can see that in this case, the tests are being run after the main call, not before!
### 06. Tutorial (Bonus): Using KLEE for Symbolic Execution
One of the most popular symbolic execution engine is [KLEE](https://klee.github.io/).
For convenience, we'll be using Docker.
```Bash
docker pull klee/klee:2.1
docker run --rm -ti --ulimit='stack=-1:-1' klee/klee:2.1
```
Let's look over this regular expression program, can you spot any bugs?
We'll create a file `ex.c` with this code:
```C++
#include <stdio.h>

static int matchhere(char*,char*);

static int matchstar(int c, char *re, char *text) {
do {
if (matchhere(re, text))
return 1;
} while (*text != '\0' && (*text++ == c || c== '.'));
return 0;
}

static int matchhere(char *re, char *text) {
if (re[0] == '\0')
return 0;
if (re[1] == '*')
return matchstar(re[0], re+2, text);
if (re[0] == '$' && re[1]=='\0')
return *text == '\0';
if (*text!='\0' && (re[0]=='.' || re[0]==*text))
return matchhere(re+1, text+1);
return 0;
}

int match(char *re, char *text) {
if (re[0] == '^')
return matchhere(re+1, text);
do {
if (matchhere(re, text))
return 1;
} while (*text++ != '\0');
return 0;
}

#define SIZE 7

int main(int argc, char **argv) {
char re[SIZE];

int count = read(0, re, SIZE - 1);
//klee_make_symbolic(re, sizeof re, "re");

int m = match(re, "hello");
if (m) printf("Match\n", re);

return 0;
}
```

Now, let's run this program symbolically.
To do this, we'll uncomment the `klee_make_symbol` line, and comment the line with `read` and `printf`.
We'll compile the program with `clang` this time:

```Bash
clang -c -g -emit-llvm ex.c
```

And run it with KLEE:

```Bash
klee ex.bc
```

We'll see the following output:

```console
KLEE: output directory is "/home/klee/klee-out-4"
KLEE: Using STP solver backend
KLEE: ERROR: ex1.c:13: memory error: out of bound pointer
KLEE: NOTE: now ignoring this error at this location
KLEE: ERROR: ex1.c:15: memory error: out of bound pointer
KLEE: NOTE: now ignoring this error at this location
KLEE: done: total instructions = 5314314
KLEE: done: completed paths = 7692
KLEE: done: generated tests = 6804
```

This tells us that KLEE has found two memory errors.
It also gives us some info about the number of paths and instructions executed.
After the run, a folder `klee-last` has been generated that contains all the test cases.
We want to find the ones that generated memory errors:

```console
klee@affd7769bb39:~/klee-last$ ls | grep err
test000018.ptr.err
test000020.ptr.err
```

We look at testcase 18:

```console
klee@affd7769bb39:~/klee-last$ ktest-tool test000018.ktest
ktest file : 'test000018.ktest'
args : ['ex1.bc']
num objects: 1
object 0: name: 're'
object 0: size: 7
object 0: data: b'^\x01*\x01*\x01*'
object 0: hex : 0x5e012a012a012a
object 0: text: ^.*.*.*
```

This is just a quick example of the power of symbolic execution, but it comes with one great problem: path explosion.
When we have more complicated programs that have unbounded loops, the number of paths grows exponentially and thus symbolic execution is not viable anymore.

## Further Reading
* [6.005 Reading 3: Test](https://ocw.mit.edu/ans7870/6/6.005/s16/classes/03-testing/index.html#automated_testing_and_regression_testing)
Expand Down

0 comments on commit 196a00f

Please sign in to comment.