Skip to content

Automated Testing Framework

Jump to bottom Edit New page
WALDEMAR KOZACZUK edited this page May 27, 2023 · 11 revisions

The main idea of the framework is quite simple: have a way to compose test images out of the pre-built packages using capstan and then run those images and validate their behavior by using app-specific test.sh shell scripts found in each app. Also instead of building each app from scratch every time we want to test it, we re-use pre-built capstan packages created using ./scripts/build-capstan-mpm-packages. This means that one can keep handy a capstan repo (or many) with a set of test packages that can be used to test and catch any regressions in OSv beyond what the regular unit tests do.

Usage

Typically one would first build test appscapstan packages (MPM files under~/.capstan/packages) using the ./scripts/build-capstan-mpm-packages` script. This is a pretty time expensive step as many apps are built from sources while others are taken from the host. But one would do it only once and re-use those MPM files from local capstan repo with every test run. Some MPMs (mostly of OSv modules) would be rebuilt every time given module changes: for example unit tests, run java, etc.

Once test apps MPMs are available in local capstan repo, users would use the script ./scripts/tests/compose_and_test_selected_apps.sh which automates the process of composing the images and running the test.sh in the corresponding apps or modules directory. The test.sh normally delegates to one of following Python scripts:

  • ./scripts/tests/test_app.py - run an app and validate its standard output for expected text messages

  • ./scripts/tests/test_http_app.py- run an http app, validate its http responses and stress test using apache bench (ab) or wrk

  • ./scripts/tests/test_app_with_test_script.py - run an app and delegate to arbitrary Python tester script located in the app or module directory that would test an app in a more sophisticated way like for example using sysbench for mysql or redis-benchmark for redis.

As you can see below, the compose_and_test_selected_apps.sh can build and execute all test images (default mode), a category of test images (simple, http, with_tester, unit_tests) or single app. You can also run the script to compose the tests images (-c) or run them only (-r). In addition, you can build images with RoFS or ZFS filesystem. Finally, you can run the tests on QEMU (default) or firecracker (-f). Running all tests on firecracker requires setting up networking that uses NAT setup and requires physical NIC that should be passed by setting the environment variable OSV_FC_NIC.

./scripts/tests/compose_and_test_selected_apps.sh 
Compose and test apps out ouf pre-built capstan packages

Usage: compose_and_test_selected_apps.sh [options] <group_of_tests> | <test_app_name>
Options:
  -c              Compose test app image only
  -r              Run test app only (must have been composed earlier)
  -R              Compose test app image with RoFS (ZFS is the default)
  -l              Use latest OSv kernel from build/last to build test image
  -f              Run OSv on firecracker

Test groups:
  simple - simple apps like golang-example
  http - httpserver apps
  http-java - java httpserver apps
  java <name> - java app
  http-node - node http apps
  node <name> - node app
  with_tester - apps tested with extra tester script like redis
  unit_tests - unit tests
  httpserver_api_tests - httpserver API unit tests
  all - all apps

Usage examples

  • ./scripts/build-capstan-mpm-packages kernel - build kernel and bootstrap mpm packages
  • ./scripts/build-capstan-mpm-packages kernel_and_modules - build kernel and standard modules
  • ./scripts/tests/compose_and_test_selected_apps.sh simple - composes ZFS test images for simple apps and runs them on QEMU
  • ./scripts/tests/compose_and_test_selected_apps.sh -rf simple - runs ZFS test images for simple apps on Firecracker
  • OSV_KERNEL=build/release/kernel.elf TESTER=wrk ./scripts/tests/compose_and_test_selected_apps.sh -r http - runs ZFS test images for HTTP apps using OSv kernel (direct kernel mode) located at build/release/kernel.elf and uses wrk to stress test each app
  • YCSB_HOME=~/projects/YCSB OSV_KERNEL=build/release/kernel.elf ./scripts/tests/compose_and_test_selected_apps.sh -rR keydb - run ROFS image of the keydb app

Test Apps

At this point testing of following apps is automated:

  • unit tests - all unit tests typically run with build check except for tracing_smoke_test
  • httpserver API - httpserver API tests typically executed by cd modules/httpserver-api && make check-http && make check-ssl
  • golang-example - 'hello world' Golang app built as shared library executed to check output for expected message
  • golang-pie-example - 'hello world' Golang app built as a PIE executed to check output for expected message
  • graalvm-example - 'hello world' GraalVM app to check output for expected message
  • lua-hello-from-host - 'hello world' Lua app to check output for expected message
  • rust-example - 'hello world' Rust app to check output for expected message
  • stream
  • python2-from-host|python3-from-host - 'hello world' Python 2/3 app to check output for expected message
  • golang-httpserver
  • golang-pie-httpserver
  • graalvm-httpserver
  • lighttpd
  • nginx
  • rust-httpserver
  • jetty
  • tomcat
  • vertx
  • spring-boot-example
  • node-express-example
  • node-socketio-example
  • specjvm - SpecJVM 2008 test suite intended to test JVM on OSv
  • iperf3
  • graalvm-netty-plot
  • ffmpeg - various tests including extracting images and transcoding to another format
  • redis-memonly - tested with redis-benchmark and YCSB
  • keydb - tested with redis-benchmark and YCSB
  • cli
  • mysql - tested with sysbench
  • apache-derby
  • apache-kafka
  • elasticsearch

Prerequisites

In order for all apps to be tested following tools need to be installed:

Environment Variables

Following environment variables can be used to customize running the tests:

  • OSV_KERNEL - the path of the kernel.elf used to run the tests apps on
  • TESTER - the name of the tester app used to stress test http apps: ab (default) or wrk
  • TESTER_CONCURRENCY - number of concurrent requests used when stress-testing http apps
  • TESTER_COUNT - number of requests made by ab
  • TESTER_DURATION - duration of a stress test run in seconds if TESTER is wrk
  • TESTER_THREADS - number of worker threads if TESTER is wrk
  • OSV_FC_NIC - the name of the physical NIC to be used when setting up firecracker networking (see this wiki page)
  • YCSB_HOME - location of the YCSB testing project directory used to stress tests redis and keydb

To Improve

  • Ffmpeg tests do not work on firecracker on Fedora (DNS resolution fails as some firewall rules are not configured correctly).
  • It would be nice to modify the main scripts to check for all prerequisites (tools, networking) ahead of time.
Add a custom footer
Add a custom sidebar
Clone this wiki locally