Miscellaneous Test Suite And Userland Core Support Changes (#324)

Contains the following changes:

1] New Crash Dump Directory Hierarchy

Instead of unarchiving one reference dump at a time and testing it it is
now possible to have multiple unpacked crash dumps at the same time and
run the regression tests against both. Each dump is now automatically
unpacked under `test/integration/data/dumps/<name>`.

It is also possible now to generate regression output for multiple dumps
at the same time.

2] Better support for automatically loading userland shared libs

The `-s`/`load_debug_info()` functionality would only look for `.ko`
files (e.g. kernel modules) given a directory. Now it can also
automatically detect debug links and shared objects.

3] Self-Descriptive Regression Output

All regression output files now contain the following appendix:
```
@#$ EXIT CODE $#@
<exit code number>
```

This makes it easier for us to not have to declare in the test suite
code whether a test case is negative or positive. Simplifying our test
code.

4] Test Infrastructure Refactor & Userland Support

The test suite infrastructure code has been refactored to handle
userland core dumps together with kernel crash dumps. Unfortuantely due
to a bug in drgn we can't use a ztest core dump for regressions (even
though sdb/drgn can handle a userland core dump just fine in the host
system). I'll come up with a smaller userland core dump soon so we can
test the userland version of stacks and work on the drgn bug when my
free time allows.

.github/scripts/clear-dump.sh

@@ -7,17 +7,7 @@ if [ ! -d $DATA_DIR ]; then
 	exit 1
 fi
 
-echo "removing current crash dump if any ..."
-rm -f $DATA_DIR/dump.*
-
-echo "removing any extracted vmlinux ..."
-rm -f $DATA_DIR/vmlinux*
-
-echo "removing any extracted modules ..."
-rm -rf $DATA_DIR/mods
-rm -rf $DATA_DIR/usr
-
-echo "removing any savedump scripts ..."
-rm -rf $DATA_DIR/run-*.sh
+echo "removing all crash/core dumps ..."
+rm -rf $DATA_DIR/dumps
 
 echo "Done"

.github/scripts/download-dump-from-s3.sh

@@ -41,31 +41,33 @@ else
 	[ $? -eq 0 ] || exit 1
 fi
 
-if [[ $1 == *.lzma ]]; then
+if [[ $1 == *.tar.lzma ]]; then
 	# Profile A
+	dump_name=${1%.tar.lzma}
 
 	echo "decompressing dump ..."
 	tar -x --lzma -f $1
 
-	echo "moving contents to tests/integration/data ..."
-	mv dump-data/* $DATA_DIR
+	echo "moving contents to tests/integration/data/dumps/${dump_name} ..."
+	mkdir -p $DATA_DIR/dumps/${dump_name}
+	mv dump-data/* $DATA_DIR/dumps/${dump_name}
 	[ $? -eq 0 ] || exit 1
 
 	rmdir dump-data
 	[ $? -eq 0 ] || exit 1
 elif [[ $1 == *.tar.gz ]]; then
 	# Profile B
+	dump_name=${1%.tar.gz}
 
 	echo "decompressing dump ..."
 	tar xzf $1
 
-	decompressed_dir=${1%.tar.gz}
-
-	echo "moving contents to tests/integration/data ..."
-	mv *$decompressed_dir/* $DATA_DIR
+	echo "moving contents to tests/integration/data/dumps/${dump_name} ..."
+	mkdir -p $DATA_DIR/dumps/${dump_name}
+	mv *${dump_name}/* $DATA_DIR/dumps/${dump_name}
 	[ $? -eq 0 ] || exit 1
 
-	rmdir *$decompressed_dir
+	rmdir *${dump_name}
 	[ $? -eq 0 ] || exit 1
 else
 	echo "unknown dump profile"

.gitignore

@@ -114,3 +114,6 @@ tests/integration/data/mods/
 # zipped folders - usually crash dumps
 *.lzma
 *.tar.gz
+
+# crash dump directory
+tests/integration/data/dumps/

README.md

@@ -103,26 +103,8 @@ If you want `pytest` to stop on the first failure it encounters add
 `-x/--exitfirst` in the command above.
 
 If you've added new test commands or found mistakes in the current reference
-output and you want (re)generate reference output for a crash dump - let's say
-`dump.201912060006` from above:
+output and you want (re)generate some reference output download all crash/core
+dumps (or the specific one you want to correct) and run the following:
 ```
-$ PYTHONPATH=$(pwd) python3 tests/integration/gen_regression_output.py dump.201912060006
-```
-
-or more generically:
-```
-$ PYTHONPATH=$(pwd) python3 tests/integration/gen_regression_output.py <dump name>
-```
-
-For the time being, the test suite is not smart enought to handle the testing
-of multiple crash dumps at the same time. Until that happens, developers that
-want to test multiple crash dumps need to delete their current crash dump
-before downloading the next to run `pytest`. Here is a sequence of commands to
-run `pytest` against two crash dumps:
-```
-$ .github/scripts/download-dump-from-s3.sh dump.201912060006.tar.lzma
-$ python3 -m pytest -v --cov sdb --cov-report xml tests
-$ .github/scripts/clear-dump.sh
-$ .github/scripts/download-dump-from-s3.sh dump.202303131823.tar.gz
-$ python3 -m pytest -v --cov sdb --cov-report xml tests
+$ PYTHONPATH=$(pwd) python3 tests/integration/gen_regression_output.py
 ```

sdb/internal/cli.py

@@ -20,6 +20,7 @@
 
 import argparse
 import os
+import re
 import sys
 
 from typing import List
@@ -126,7 +127,8 @@ def parse_arguments() -> argparse.Namespace:
     return args
 
 
-def load_debug_info(prog: drgn.Program, dpaths: List[str]) -> None:
+def load_debug_info(prog: drgn.Program, dpaths: List[str], quiet: bool,
+                    no_filter: bool) -> None:
     """
     Iterates over all the paths provided (`dpaths`) and attempts
     to load any debug information it finds. If the path provided
@@ -140,9 +142,28 @@ def load_debug_info(prog: drgn.Program, dpaths: List[str]) -> None:
             kos = []
             for (ppath, __, files) in os.walk(path):
                 for i in files:
-                    if i.endswith(".ko"):
+                    if i.endswith(".ko") or i.endswith(".debug") or re.match(
+                            r".+\.so(\.\d)?", i) or no_filter:
+                        # matches:
+                        #     kernel modules - .ko suffix
+                        #     userland debug files - .debug suffix
+                        #     userland shared objects - .so suffix
                         kos.append(os.sep.join([ppath, i]))
-            prog.load_debug_info(kos)
+            try:
+                prog.load_debug_info(kos)
+            except drgn.MissingDebugInfoError as debug_info_err:
+                #
+                # If we encounter such an error it means that we can't
+                # find the debug info for one or more kernel modules.
+                # That's fine because the user may not need those, so
+                # print a warning and proceed.
+                #
+                # Again because of the aforementioned short-coming of drgn
+                # we quiet any errors when loading the *default debug info*
+                # if we are looking at a crash/core dump.
+                #
+                if not quiet:
+                    print("sdb: " + str(debug_info_err), file=sys.stderr)
         else:
             print("sdb: " + path + " is not a regular file or directory")
 
@@ -191,7 +212,7 @@ def setup_target(args: argparse.Namespace) -> drgn.Program:
 
     if args.symbol_search:
         try:
-            load_debug_info(prog, args.symbol_search)
+            load_debug_info(prog, args.symbol_search, args.quiet, False)
         except (
                 drgn.MissingDebugInfoError,
                 OSError,

tests/integration/data/regression_output/dump.201912060006/core/addr bogus

@@ -1 +1,3 @@
 sdb: addr: symbol not found: bogus
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/addr fget | deref

@@ -1 +1,3 @@
 sdb: deref: cannot dereference function pointer
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/addr jiffies | deref

@@ -1 +1,3 @@
 (volatile unsigned long)4294968498
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/addr jiffies | deref | deref

@@ -1 +1,3 @@
 sdb: deref: 'volatile unsigned long' is not a valid pointer type
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/addr spa_namespace_avl

@@ -1 +1,3 @@
 (avl_tree_t *)spa_namespace_avl+0x0 = 0xffffffffc07d0fe0
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/addr spa_namespace_avl zfs_dbgmsgs | print -d

@@ -74,3 +74,5 @@
 		.kpe_proc = (struct proc_dir_entry *)0xffffa089659cbf00,
 	},
 }
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/addr spa_namespace_avl | deref

@@ -5,3 +5,5 @@
 	.avl_numnodes = (ulong_t)3,
 	.avl_size = (size_t)9176,
 }
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/addr spa_namespace_avl | deref | addr

@@ -1 +1,3 @@
 (avl_tree_t *)spa_namespace_avl+0x0 = 0xffffffffc07d0fe0
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/addr spa_namespace_avl | deref | print

@@ -5,3 +5,5 @@
 	.avl_numnodes = (ulong_t)3,
 	.avl_size = (size_t)9176,
 }
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/addr spa_namespace_avl | member avl_root->avl_child[0]->avl_child

@@ -1 +1,3 @@
 (struct avl_node *[2]){}
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/addr spa_namespace_avl | member avl_root->avl_child[1

@@ -1 +1,3 @@
 sdb: member: incomplete array expression: please use something of the format 'array_name[index]'
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/addr spa_namespace_avl | member avl_root->avl_child[3]

@@ -1,2 +1,4 @@
 warning: member: index out of bounds for array of type 'struct avl_node *[2]' (requested index: 3)
 (struct avl_node *)0xffffa08892703f60
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/addr spa_namespace_avl | member avl_root->avl_child[a]

@@ -1 +1,3 @@
 sdb: member: incorrect index: 'a' is not a number
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/addr spa_namespace_avl | member avl_root.avl_child[0].avl_child

@@ -1 +1,3 @@
 (struct avl_node *[2]){}
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/addr spa_namespace_avl | member avl_root.avl_pcb avl_size

@@ -1,2 +1,4 @@
 (uintptr_t)1
 (size_t)9176
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/addr spa_namespace_avl | print

@@ -1 +1,3 @@
 (avl_tree_t *)spa_namespace_avl+0x0 = 0xffffffffc07d0fe0
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/addr spa_namespace_avl | print --RAW

@@ -1 +1,3 @@
 0xffffffffc07d0fe0
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/addr spa_namespace_avl | print -d

@@ -5,3 +5,5 @@
 	.avl_numnodes = (ulong_t)3,
 	.avl_size = (size_t)9176,
 }
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/addr spa_namespace_avl | print -n

@@ -1 +1,3 @@
 (avl_tree_t *)0xffffffffc07d0fe0
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/addr spa_namespace_avl | print -nr

@@ -1 +1,3 @@
 0xffffffffc07d0fe0
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/addr spa_namespace_avl | print -r

@@ -1 +1,3 @@
 spa_namespace_avl+0x0 = 0xffffffffc07d0fe0
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/echo 0x0 0x1 0x2 | filter 'obj < 1'

@@ -1 +1,3 @@
 (void *)0x0
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/echo 0x0 0x1 0x2 | filter 'obj <= 1'

@@ -1,2 +1,4 @@
 (void *)0x0
 (void *)0x1
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/echo 0x0 0x1 0x2 | filter 'obj == 1'

@@ -1 +1,3 @@
 (void *)0x1
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/echo 0x0 0x1 0x2 | filter 'obj > 1'

@@ -1 +1,3 @@
 (void *)0x2
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/echo 0x0 0x1 0x2 | filter 'obj >= 1'

@@ -1,2 +1,4 @@
 (void *)0x1
 (void *)0x2
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/echo 0x0 | addr spa_namespace_avl | echo 0x1 | cast avl_tree_t * | member avl_root

@@ -1,3 +1,5 @@
 sdb: member: invalid memory access: addresss 0x0
 (struct avl_node *)0xffffa089413b8108
 sdb: member: invalid memory access: addresss 0x1
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/echo 0x0 | cast int * | array 1

@@ -1 +1,3 @@
 sdb: array: invalid memory access: addresss 0x0
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/echo 0x0 | cast int * | deref

@@ -1 +1,3 @@
 sdb: deref: invalid memory access: addresss 0x0
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/echo 0x0 | cast spa_t * | member spa_name

@@ -1 +1,3 @@
 sdb: member: invalid memory access: addresss 0x0
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/echo 0x0 | cast void * | array 1

@@ -1 +1,3 @@
 sdb: array: can't walk pointer array of incomplete type 'void'
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/echo 0x0 | filter 'obj == 0'

@@ -1 +1,3 @@
 (void *)0x0
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/echo 0x0 | filter 'obj == 1'

@@ -0,0 +1,2 @@
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/echo 0x1 | filter 'obj == obj'

@@ -1 +1,3 @@
 (void *)0x1
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/echo 0x10 | cast int * | deref

@@ -1 +1,3 @@
 sdb: deref: invalid memory access: addresss 0x10
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/echo 0x1234 | cast dmu_recv_cookie_t * | member drc_os

@@ -1 +1,3 @@
 sdb: member: invalid memory access: addresss 0x12c4
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/echo 0xffff90cc11b28000 | deref

@@ -1 +1,3 @@
 sdb: deref: cannot dereference a void pointer
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/echo 1 | echo 2 | sum

@@ -1 +1,3 @@
 (uint64_t)3
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/echo 1234 | cast int | array

@@ -1 +1,3 @@
 sdb: array: 'int' is not an array nor a pointer type
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/filter 'obj == 1'

@@ -0,0 +1,2 @@
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/member no_object

@@ -0,0 +1,2 @@
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/ptype $abc

@@ -1 +1,3 @@
 sdb: ptype: input '$abc' is not a valid type name
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/ptype 'a b c'

@@ -1 +1,3 @@
 sdb: ptype: input 'a b c' is not a valid type name
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/ptype 'bogus union'

@@ -1 +1,3 @@
 sdb: ptype: input 'bogus union' is not a valid type name
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/ptype 'struct bogus'

@@ -1 +1,3 @@
 sdb: ptype: couldn't find type 'struct bogus'
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/ptype 'struct spa'

@@ -193,3 +193,5 @@ struct spa {
 	zfs_refcount_t spa_refcount;
 	taskq_t *spa_upgrade_taskq;
 }
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/ptype 'struct union'

@@ -1 +1,3 @@
 sdb: ptype: input 'struct union' is not a valid type name
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/ptype 'struct'

@@ -1 +1,3 @@
 sdb: ptype: skip keyword 'struct' or quote your type "struct <typename>"
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/ptype 'union struct struct'

@@ -1 +1,3 @@
 sdb: ptype: input 'union struct struct' is not a valid type name
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/ptype 2abc

@@ -1 +1,3 @@
 sdb: ptype: input '2abc' is not a valid type name
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/ptype @

@@ -1 +1,3 @@
 sdb: ptype: input '@' is not a valid type name
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/ptype bogus_t

@@ -1 +1,3 @@
 sdb: ptype: couldn't find typedef, struct, enum, nor union named 'bogus_t'
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/ptype spa vdev

@@ -335,3 +335,5 @@ struct vdev {
 	zfs_ratelimit_t vdev_delay_rl;
 	zfs_ratelimit_t vdev_checksum_rl;
 }
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/ptype spa_t

@@ -1 +1,3 @@
 typedef struct spa spa_t
+@#$ EXIT CODE $#@
+0

tests/integration/data/regression_output/dump.201912060006/core/ptype struct spa

@@ -1 +1,3 @@
 sdb: ptype: skip keyword 'struct' or quote your type "struct <typename>"
+@#$ EXIT CODE $#@
+1

tests/integration/data/regression_output/dump.201912060006/core/ptype zfs_case 'struct v' thread_union

@@ -10,3 +10,5 @@ union thread_union {
 	struct task_struct task;
 	unsigned long stack[2048];
 }
+@#$ EXIT CODE $#@
+0