tplist_example.txt

Demonstrations of tplist.


tplist displays kernel tracepoints and USDT probes, including their
format. It can be used to discover probe points for use with the trace
and argdist tools. Kernel tracepoints are scattered around the kernel
and provide valuable static tracing on block and network I/O, scheduling,
power events, and many other subjects. USDT probes are placed in libraries
(such as libc) and executables (such as node) and provide static tracing
information that can (optionally) be turned on and off at runtime.

For example, suppose you want to discover which USDT probes a particular
executable contains. Just run tplist on that executable (or library):

$ tplist -l basic_usdt
/home/vagrant/basic_usdt basic_usdt:start_main
/home/vagrant/basic_usdt basic_usdt:loop_iter
/home/vagrant/basic_usdt basic_usdt:end_main

The loop_iter probe sounds interesting. How many arguments are available?

$ tplist '*loop_iter' -l basic_usdt -v
basic_usdt:loop_iter [sema 0x601036]
  2 location(s)
  2 argument(s)

This output indicates that the loop_iter probe is used in two locations
in the basic_usdt executable, and that it has two arguments. Fortunately,
the argdist and trace tools understand the probe format and can print out
the arguments automatically -- you can refer to them as arg1, arg2, and
so on.

Try to explore with some common libraries on your system and see if they
contain UDST probes. Here are two examples you might find interesting:

$ tplist -l pthread     # list probes in libpthread
/lib64/libpthread.so.0 libpthread:pthread_start
/lib64/libpthread.so.0 libpthread:pthread_create
/lib64/libpthread.so.0 libpthread:pthread_join
/lib64/libpthread.so.0 libpthread:pthread_join_ret
/lib64/libpthread.so.0 libpthread:mutex_init
... more output truncated

$ tplist -l c           # list probes in libc
/lib64/libc.so.6 libc:setjmp
/lib64/libc.so.6 libc:longjmp
/lib64/libc.so.6 libc:longjmp_target
/lib64/libc.so.6 libc:memory_arena_reuse_free_list
/lib64/libc.so.6 libc:memory_heap_new
... more output truncated

tplist also understands kernel tracepoints, and can list their format
as well. For example, let's look for all block I/O-related tracepoints:

# tplist 'block*'
block:block_touch_buffer
block:block_dirty_buffer
block:block_rq_abort
block:block_rq_requeue
block:block_rq_complete
block:block_rq_insert
block:block_rq_issue
block:block_bio_bounce
block:block_bio_complete
block:block_bio_backmerge
block:block_bio_frontmerge
block:block_bio_queue
block:block_getrq
block:block_sleeprq
block:block_plug
block:block_unplug
block:block_split
block:block_bio_remap
block:block_rq_remap

The block:block_rq_complete tracepoints sounds interesting. Let's print
its format to see what we can trace with argdist and trace:

$ tplist -v block:block_rq_complete
block:block_rq_complete
    dev_t dev;
    sector_t sector;
    unsigned int nr_sector;
    int errors;
    char rwbs[8];

The dev, sector, nr_sector, etc. variables can now all be used in probes
you specify with argdist or trace.


For debugging USDT probes, it is sometimes useful to see the exact locations
and arguments of the probes, including the registers or global variables from
which their values are coming from. In super-verbose mode, tplist will print
this information (note the -vv):

$ tplist -vv -l c *alloc*
libc:memory_malloc_retry [sema 0x0]
  location #0 /lib64/libc.so.6 0x835c0
    argument #0 8 unsigned bytes @ bp
  location #1 /lib64/libc.so.6 0x83778
    argument #0 8 unsigned bytes @ bp
  location #2 /lib64/libc.so.6 0x85a50
    argument #0 8 unsigned bytes @ bp
libc:memory_realloc_retry [sema 0x0]
  location #0 /lib64/libc.so.6 0x84b90
    argument #0 8 unsigned bytes @ r13
    argument #1 8 unsigned bytes @ bp
  location #1 /lib64/libc.so.6 0x85cf0
    argument #0 8 unsigned bytes @ r13
    argument #1 8 unsigned bytes @ bp
libc:memory_calloc_retry [sema 0x0]
  location #0 /lib64/libc.so.6 0x850f0
    argument #0 8 unsigned bytes @ bp


USAGE message:

$ tplist -h
usage: tplist.py [-h] [-p PID] [-l LIB] [-v] [filter]

Display kernel tracepoints or USDT probes and their formats.

positional arguments:
  filter             A filter that specifies which probes/tracepoints to print

optional arguments:
  -h, --help         show this help message and exit
  -p PID, --pid PID  List USDT probes in the specified process
  -l LIB, --lib LIB  List USDT probes in the specified library or executable
  -v                 Increase verbosity level (print variables, arguments, etc.)