Skip to content

Latest commit

 

History

History
218 lines (163 loc) · 7.82 KB

WritingTests.asciidoc

File metadata and controls

218 lines (163 loc) · 7.82 KB

openQA tests developer guide

Table of Contents

Introduction

openQA is an automated test tool that makes it possible to test the whole installation process of an operating system. It’s free software released under the GPLv2 license. The source code and documentation are hosted in the os-autoinst organization on GitHub.

This document provides the information needed to start developing new tests for openQA or to improve the existing ones. It’s assumed that the reader is already familiar with openQA and has already read the Starter Guide, available at the official repository.

Basic

This section explains the basic layout of openQA tests and the API available in tests. openQA tests are written in the Perl programming language. Some basic but no in-depth knowledge of Perl is needed. This document assumes that the reader is already familiar with Perl.

API

os-autoinst provides the API for the tests. The most commonly used functions are:

  • check_var NAME, CONTENT Check content of a variable, treating undef as empty.

  • send_key KEY[, WAITIDLE] Send key to openQA instance. Call wait_idle at the end if WAITIDLE is set. (e.g. send_key "alt-o", 1)

  • type_string STRING Typing the given string. (e.g. type_string "yes\n")

  • save_screenshot Capture and save a screenshot in the test results.

  • check_screen NEEDLE[, TIMEOUT] Wait until NEEDLE is seen on the screen. Return undef if not found within TIMEOUT (default 30s).

  • assert_screen NEEDLE[, TIMEOUT] Wait until NEEDLE is seen on the screen. Abort test and mark as failed if not found within TIMEOUT (default 30s).

  • wait_idle TIMEOUT Wait for max TIMEOUT seconds until system is idle.

  • wait_serial REGEX[, TIMEOUT] Wait until REGEX is seen on serial port. Return 0 if not found after TIMEOUT (default 90s)

  • upload_logs FILE Type in necessary shell commands to save FILE in the test results

The following API is distribution specific and subject to change

  • ensure_installed PACKAGES…​ Check that the specific application was installed in the system, if not then it will try to install it from download repositories.

  • x11_start_program NAME Execute the graphical application NAME. How exactly depends on the desktop environment.

  • become_root Become root (in a shell)

  • script_run COMMAND[, TIMEOUT] Wrapper for type_string meant to type in shell commands. Waits TIMEOUT until idle (default 9s)

  • script_sudo COMMAND Use sudo to execute COMMAND, handling the optional password prompt.

  • sendpassword Type default root password.

How to write tests

openQA tests need to implement at least the run subroutine to contain the actual test code.

The is_applicable subroutines is called by the engine to determine whether a test applies and should run. The method can check external conditions, like variable settings.

The test_flags subroutine specifies what happens when the test fails.

The following example is a basic test that assumes some live image that boots into the desktop when pressing enter at the boot loader:

use base "basetest";
use strict;
use bmwqemu;

sub is_applicable {
    return 1; # check for $vars{SOMETHING}
}

sub run {
    # wait for bootloader to appear
    assert_screen "bootloader", 30; # timeout 30 seconds

    # press enter to boot right away
    send_key "ret";

    # wait for the desktop to appear
    assert_screen "desktop", 300;
}

sub test_flags {
    # without anything - rollback to 'lastgood' snapshot if failed
    # 'fatal' - whole test suite is in danger if this fails
    # 'milestone' - after this test succeeds, update 'lastgood'
    # 'important' - if this fails, set the overall state to 'fail'
    return { important => 1 };
}

1;

Test Case Examples

  • Console test that installs software from remote repository via zypper command

sub run() {

    # change to root
    become_root();

    # output zypper repos to the serial
    script_run("zypper lr -d > /dev/$serialdev");

    # install xdelta and insert a string 'xdelta_installed' to the serial
    script_run("zypper --gpg-auto-import-keys -n in xdelta && echo 'xdelta_installed' > /dev/$serialdev");

    # detecting whether 'xdelta_installed' appears in the serial within 200 seconds
    wait_serial "xdelta_installed", 200  || die "zypper install failed";

    # capture a screenshot and compare with needle 'test-zypper_in-1'
    assert_screen 'test-zypper_in-1', 3;
}

  • Typical X11 test testing kate

sub is_applicable {
    # do this test if desktop environment is KDE
    return ( $vars{DESKTOP} eq "kde" );
}

sub run() {

    # make sure kate was installed
    # if not ensure_installed will try to install it
    ensure_installed("kate");

    # start kate
    x11_start_program("kate");

    # check that kate execution succeeded
    assert_screen 'test-kate-1', 10;

    # close kate's welcome window and wait for system becoming idle
    send_key 'alt-c', 1;

    # typing the string on kate
    type_string "If you can see this text kate is working.\n";

    # check the result
    assert_screen 'test-kate-2', 5;

    # quit kate
    send_key "ctrl-q";

    # make sure kate was closed
    assert_screen 'test-kate-3', 5;
}

Variables

Test case behavior can be controlled via variables. Some basic variables like DISTRI, VERSION, ARCH are always set. Others like DESKTOP are defined by the Test suites in the openQA web UI. Check the existing tests at os-autoinst-distri-opensuse on GitHub for examples.

Variables are accessible via the %vars hash, e.g $vars{DESKTOP} or better use the API method check_var.

Modifying setting of an existing test

There is no interface to modify existing tests but the clone script can be used to create a new job that adds, removes or changes settings:

/usr/share/openqa/script/clone_job.pl --from localhost --host localhost 42 FOO=bar BAZ=

Using Snapshots to speed up development of tests

Sometimes it’s annoying to run the full installation to adjust some test. It would be nice to have the VM jump to a certain point. There is an experimental hidden feature that allows to start from a snapshot that might help in that situation:

  1. run the worker with --no-cleanup parameter. This will preserve the hard disks after test runs.

  2. set MAKETESTSNAPSHOTS=1 on a job. This will make openQA save a snapshot for every test run. One way to do that is by cloning an existing job and adding the setting:

$ /usr/share/openqa/script/clone_job.pl --from https://openqa.opensuse.org --host localhost 24 MAKETESTSNAPSHOTS=1

  1. create a job again, this time setting the SKIPTO variable to the snapshot you need. Again, clone_job.pl comes handy here:

$ /usr/share/openqa/script/clone_job.pl --from https://openqa.opensuse.org --host localhost 24 SKIPTO=consoletest-yast2_i

Use qemu-img snapshot -l something.img to find out what snapshots are in the image.