trace stream does not match query

[vanhauser-thc]

I have a current kernel, booted with nopti, inet_pt is supported and built from the current github state:

# dmesg|grep nopti
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.19.0-kali1-amd64 root=/dev/mapper/kali--vg-root ro quiet nopti
# ./ptfeature 
Supports PT
toPA output support:            1
multiple toPA entries:          1
single range:                   1
trace transport output:         0
payloads are LIP:               0
cycle accurate mode / psb freq: 1
filtering / stop / mtc:         1
CR3 match:                      1
Number of address ranges:       2
Supports filter ranges:         1
Supports stop ranges:           1
Valid cycles thresholds:        1 2 3 4 5 6 7 8 9 10 11 12 13 14 
Valid PSB frequencies:          1 2 3 4 5 6 
Valid MTC frequencies:          1 4 7 10 
TSC ratio:                      2 242
Bus frequency:                  121.000008
Family:                         6
Model:                          142
Stepping:                       9

to show that the issue is present with the example commands in the README:

 # ./sptcmd -c ls ls
[... ls output ...]
cpu   0 offset    176,     0 KB, writing to ptout.0
cpu   1 offset    272,     0 KB, writing to ptout.1
cpu   2 offset    144,     0 KB, writing to ptout.2
cpu   3 offset    416,     0 KB, writing to ptout.3
Wrote sideband to ptout.sideband
# ./sptdecode --sideband ptout.sideband --pt ptout.0
TIME      DELTA         INSNs   OPERATION
b0:0: error trace stream does not match query
b0: sync forward: trace stream does not match query

and this happens for any command. the kernel module is present.
when I unload the kernel module and use perf I get stuff (and perf report shows branches etc.):

# perf record -e intel_pt//u ls
[... ls output ...]
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.068 MB perf.data ]

any idea what the issue could be?
thanks!

[andikleen]

-c filtering doesn't work anymore since the kernel added page table isolation, because there are multiple page tables per process. You can either boot with nopti or avoid using -c

[vanhauser-thc]

but I bootet with nopti (see the very first line I postet: [0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.19.0-kali1-amd64 root=/dev/mapper/kali--vg-root ro quiet nopti).

thats why I am surprised its not working, but working when I use perf

[nkurz]

Just confirming that I see the same thing as vanhauser-thc. I'm using kernel 5.1.14 on an i3-8121U, and have booted with "nopti". The suggested sample code for sptcmd using -c always results in the error "trace stream does not match query" when I run sptdecode. Using sptcmd without -c gives me apparently mostly results from sptdecode, but still interspersed with occasional inline "trace stream does not match query" errors.

[Bitterino95]

Same problems for me: kernel 5.1.15 and CPU i5-8250U.
Running without any cpu vulnerabilities mitigations.

[huangya90]

Same problem for me . I 'm using Ubuntu 18.04
XXX@XXX:/project/ipt/simple-pt.git$ dmesg |grep command
[ 0.000000] Kernel/User page tables isolation: disabled on command line.
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-65-generic root=UUID=0892bd19-848d-453c-ae3c-83607cc7dbfc ro quiet splash vt.handoff=1 nopti

[okilic1]

Hello,
I also have the same issue even without using -c option. Is there any update on this.
Thanks

[nanananananana]

Hello, I also have the same issue......
With using -c option, it will show that 'error trace stream does not match query'.

Without using -c option, this problem also happens by accident.

When I use 'sudo ./tester',

ubuntu 4.15.18 i5-8500

[nanananananana]

Just confirming that I see the same thing as vanhauser-thc. I'm using kernel 5.1.14 on an i3-8121U, and have booted with "nopti". The suggested sample code for sptcmd using -c always results in the error "trace stream does not match query" when I run sptdecode. Using sptcmd without -c gives me apparently mostly results from sptdecode, but still interspersed with occasional inline "trace stream does not match query" errors.

My question is exactly the same as yours. Did you solve this problem?

[prasun3]

I am seeing this issue as well.

Looking at the code, I found that this error being hit in pt_insn_event

    /* We must currently process an event. */
    if (!decoder->process_event)
        return -pte_bad_query;

process_event is normally set in event_pending, but in the error case it hits this case and returns without setting this flag

     if (!(status & pts_event_pending))

[vanhauser-thc]

@prasun3 maybe send andi a pull request with a fix?

[prasun3]

I'm not sure what the fix or even the root cause is. Hoping this provides some insight to others more familiar with the code.

[andikleen]

I assume when you hexdump the stream it will be all zeroes.

I think the problem is that the PTI kernel puts a PCID into CR3, but simple-pt doesn't know about, so the CR3 filter never matches.

Some variant of #29 would likely fix it.

For now I would just not use the CR3 filter.

[zhangbonian]

try this patch
fix-query-not-match.patch.zip