Skip to content

Commit 994dd12

Browse files
lindongwlindongw
committed
add trace tool to support libva trace process
Signed-off-by: Lindong Wu <[email protected]>
1 parent 11bc130 commit 994dd12

27 files changed

+6358
-0
lines changed

‎tracetool/README.md

+126
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
# Linux trace tool
2+
3+
4+
## Introduction
5+
6+
This is python tool helps to analysis media stack trace logs combining ftrace events from libva, media driver and Linux kernel mode driver (e.g. i915).
7+
8+
9+
## Linux trace capture
10+
11+
1. Install trace-cmd:
12+
13+
sudo apt-get install trace-cmd
14+
15+
2. Grant write access to trace node for application:
16+
17+
sudo chmod 777 /sys/kernel/debug/
18+
sudo chmod 777 /sys/kernel/debug/tracing/
19+
sudo chmod 777 /sys/kernel/debug/tracing/trace_marker_raw
20+
21+
3. Enable libva trace:
22+
23+
export LIBVA_TRACE = FTRACE
24+
25+
to enable libva buffer data capture
26+
27+
export LIBVA_TRACE_BUFDATA = 1
28+
29+
4. Run application under trace-cmd in a proxy mode:
30+
31+
trace-cmd record -e i915 <workflow-cmd-line>
32+
33+
5. Output is "trace.dat"
34+
35+
Alternatively you can collect trace data in separate terminal.
36+
It is useful if you want to profile daemon or a service:
37+
38+
1. Start trace capture:
39+
40+
sudo trace-cmd record -e i915
41+
42+
2. Run test app in another terminal
43+
3. Stop capturing in the first terminal
44+
4. Output is "trace.dat"
45+
46+
47+
## Trace post-processing and analysis
48+
49+
python3 main.py [-raw] file.dat|file.etl [file.dat|file.etl ...]
50+
51+
Options:
52+
53+
* `-raw` - Parse trace events and dump into <trace-file>.csv file.
54+
55+
Output:
56+
57+
* `<trace-file>.json.gz` - visualized trace activity, open in `<chrome://tracing/>` or `<edge://tracing/>`
58+
* `<trace-file>_stat.csv` - statistic of trace activity, open in Excel
59+
* `<trace-file>_surface.csv` - surface attributes and runtime usage in GPU, open in Excel
60+
* `<trace-file>_rtlog.txt` - iHD driver runtime log
61+
62+
63+
## Trace tool manifests
64+
65+
Manifests are for decoding trace data. Each trace system supported by trace tool need have a manifest file in manifests folder.
66+
The manifest file could be in MSFT ETW XML manifest format or json format. Current supported traces:
67+
* libva_trace.man - libva trace manifest in MSFT ETW XML
68+
* Intel-Media-Open.json - iHD open source media driver trace manifest in json.
69+
70+
71+
## Trace Modules
72+
73+
Trace tool dynamic load functional module in modules folders. Each module should have its own standalone function.
74+
Two types of module supported: traceReader and traceHandler.
75+
Each module should have 1 class traceReader or class traceHandler implemented. Trace tool load module by class name, so no change on class name.
76+
### class traceReader
77+
trace file reader module, responsible for parsing trace file into trace events, and call core handler one by one in event timestamp order.
78+
trace reader is driven by core, following interfaces are required by core:
79+
|interface|description|
80+
|---|---|
81+
|open(self, file, options)->int|open trace file, return 0 for sucess, negative for failure. user command line options are provided in args.|
82+
|setParser(self, parsers)->None|set trace head parsers. Since all Linux user space trace share single trace_marker_raw entry, each user trace(libva and iHD) need register its own header parser to identify itself.|
83+
|syncSource(self, src)->int|for sync timestamp across multi trace readers. return -1 for no support.|
84+
|process(self, filter, handler)->None|start trace event process with filter and handler callbacks. Filt events with filter callback could speed up event process.|
85+
86+
add new reader to support more trace file format
87+
### class traceHandler
88+
trace event handler module to customize event handling.
89+
core interface for trace handler:
90+
|interface|description|
91+
|---|---|
92+
|core.regParser(id, parser)->int|register trace header parser, id is 4bytes trace identifier, parer callback is to parser event header.|
93+
|core.regHandler(sys, name, handler)->None|register event handler, set name to None for common trace handler.|
94+
|core.getContext()->dict|get share context from core.|
95+
96+
Since all handler modules are in seperated instance, could not share anything across modules, core provide shared context for sharing.
97+
by default, shared context provide following class instance:
98+
|name in context|access example| description|
99+
|---|---|---|
100+
|UI|sharedCtx['UI']|instance of class writeUI.py, wrap UI event for chrome://tracing.|
101+
|Stack|sharedCtx['Stack']|instance of class callStack.py, provide call stack of trace event. call stack is built from event sequence from the same process id and thread id.|
102+
|Stat|sharedCtx['Stat']|instace of class statistic.py, build statistic for specific events.|
103+
|Output|sharedCtx['Output']|output path string, in case module need to create its own output file.|
104+
105+
developer could add more in shared context.
106+
107+
Note: It is possible multi modules register their own handers for one specific event. core will call these callback one by one when target event occurs.
108+
109+
current modules:
110+
|module|description|
111+
|---|---|
112+
|ftrace.py|Linux ftace file parser, trace file from trace-cmd.|
113+
|i915.py|i915 trace handler to extract GPU workload submit&execution timing.|
114+
|libva.py|libva trace handler.|
115+
|iHD.py|Intel iHD open source media driver trace handler.|
116+
|surface.py|track surface object & attributes across iHD and i915 traces.|
117+
118+
## Making changes in the tool
119+
120+
Make sure to run unit tests before creating PR:
121+
122+
cd tracetool
123+
python3 -m unittest
124+
125+
Make sure trace event and event data are backward compatible.
126+

‎tracetool/callStack.py

+70
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
#
2+
# Copyright (C) Intel Corporation. All rights reserved.
3+
# Licensed under the MIT License.
4+
#
5+
6+
# build call stack from events with the same process and thread id
7+
class callStack:
8+
9+
def __init__(self):
10+
self.context = {} # maintain call stack
11+
12+
# get latest pushed call event in stack
13+
def current(self, pid, tid):
14+
if pid not in self.context or tid not in self.context[pid]:
15+
return None
16+
if self.context[pid][tid]:
17+
return self.context[pid][tid][0]
18+
return None
19+
20+
# get full call stack record
21+
def get(self, pid, tid):
22+
if pid not in self.context:
23+
self.context[pid] = {}
24+
if tid not in self.context[pid]:
25+
self.context[pid][tid] = []
26+
return self.context[pid][tid]
27+
28+
# push event into stack
29+
def push(self, evt):
30+
if evt['pid'] not in self.context:
31+
self.context[evt['pid']] = {}
32+
if evt['tid'] not in self.context[evt['pid']]:
33+
self.context[evt['pid']][evt['tid']] = []
34+
self.context[evt['pid']][evt['tid']].insert(0, evt)
35+
36+
# pop event from stack
37+
def pop(self, evt):
38+
if evt['pid'] not in self.context:
39+
return None
40+
if evt['tid'] not in self.context[evt['pid']] or not self.context[evt['pid']][evt['tid']]:
41+
thrds = self.context[evt['pid']]
42+
for t in thrds.values():
43+
if t and t[0]['name'] == evt['name']:
44+
return t.pop(0)
45+
return None
46+
ctx = self.context[evt['pid']][evt['tid']]
47+
name = evt['name']
48+
idx = 0
49+
ret = None
50+
# find target in the stack
51+
for i in range(len(ctx)):
52+
if ctx[i]['name'] == name:
53+
idx = i+1
54+
ret = ctx[i]
55+
break
56+
# remove target from stack
57+
del ctx[0:idx]
58+
return ret
59+
60+
# find top event with the same sys id + pid + tid
61+
def find(self, evt):
62+
if evt['pid'] not in self.context or evt['tid'] not in self.context[evt['pid']]:
63+
return None
64+
for e in self.context[evt['pid']][evt['tid']]:
65+
if e['sys'] == evt['sys']:
66+
return e
67+
return None
68+
69+
def __del__(self):
70+
del self.context

‎tracetool/core.py

+160
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
#
2+
# Copyright (C) Intel Corporation. All rights reserved.
3+
# Licensed under the MIT License.
4+
#
5+
6+
import os
7+
import sys
8+
import importlib
9+
from writeUI import writeUI
10+
from statistic import statistic
11+
from callStack import callStack
12+
from util import *
13+
14+
class core:
15+
16+
def __init__(self):
17+
self.source = None
18+
self.sharedCtx = {} # shared context for all handlers, dict for flexible usage
19+
self.handlers = {}
20+
self.instances = []
21+
self.readers = []
22+
self.parsers = {}
23+
self.dumpRaw = False
24+
25+
cur = os.path.abspath(os.path.dirname(__file__))
26+
sys.path.append(cur+os.sep+'modules')
27+
for py in os.listdir('modules'):
28+
name = os.path.splitext(py)[0]
29+
m = importlib.import_module(name)
30+
if hasattr(m, 'traceHandler'):
31+
cls = getattr(m, 'traceHandler')
32+
# create handler instace, the class init should call register of this instance
33+
instance = cls(self)
34+
# just for keep instance ref
35+
self.instances.append(instance)
36+
elif hasattr(m, 'traceReader'):
37+
cls = getattr(m, 'traceReader')
38+
self.readers.append(cls)
39+
40+
# open trace file
41+
def open(self, input, options) -> int:
42+
ret = -1
43+
if isinstance(input, list) and len(input) == 1:
44+
input = input[0]
45+
if isinstance(input, list):
46+
# enumerate and open trace files
47+
names = []
48+
readers = []
49+
for i in input:
50+
for cls in self.readers:
51+
reader = cls()
52+
sts = reader.open(i, options)
53+
if sts == 0:
54+
names.append(i)
55+
readers.append(reader)
56+
break
57+
if len(input) == len(readers):
58+
# sync time stamp across multi trace files, need find single source reader
59+
print('Multi trace input files, sync time line ...')
60+
for i in readers:
61+
for j in readers:
62+
if i != j and i.syncSource(j) == 0:
63+
self.source = i
64+
self.sharedCtx['sourceFile'] = names[readers.index(i)]
65+
break
66+
if self.source != None:
67+
break
68+
if self.source != None:
69+
print('done')
70+
ret = 0
71+
else:
72+
print('Error! could not syn time line')
73+
else:
74+
for cls in self.readers:
75+
reader = cls()
76+
sts = reader.open(input, options)
77+
if sts == 0:
78+
self.source = reader
79+
self.sharedCtx['sourceFile'] = input
80+
ret = 0
81+
break
82+
# setup handlers and output if success
83+
if ret == 0:
84+
self.source.setParser(self.parsers)
85+
86+
baseName = self.sharedCtx['sourceFile']
87+
baseName = os.path.splitext(baseName)[0]
88+
self.sharedCtx['Output'] = baseName
89+
self.sharedCtx['UI'] = writeUI(baseName)
90+
self.sharedCtx['Stat'] = statistic(baseName)
91+
self.sharedCtx['Stack'] = callStack()
92+
self.sharedCtx['Opt'] = options
93+
return ret
94+
95+
# start process event from trace file
96+
def process(self) -> None:
97+
self.source.process(self.filter, self.callback)
98+
99+
# close
100+
def __del__(self):
101+
del self.source
102+
del self.readers
103+
del self.instances
104+
del self.handlers
105+
del self.sharedCtx
106+
107+
# test if event handler is set for this event
108+
def filter(self, evt) -> bool:
109+
if 'raw' in self.sharedCtx['Opt']:
110+
return True
111+
if 'sys' not in evt or 'name' not in evt:
112+
return False
113+
if evt['sys'] not in self.handlers:
114+
return False
115+
handler = self.handlers[evt['sys']]
116+
if evt['name'] not in handler and 'allEvent' not in handler:
117+
return False
118+
return True
119+
120+
# call back function to process event with handler
121+
def callback(self, evt) -> None:
122+
if evt['sys'] not in self.handlers:
123+
return
124+
# get handler, could be a list, multi handler for single event
125+
hnd = self.handlers[evt['sys']]
126+
flag = 0
127+
if evt['name'] in hnd:
128+
for h in hnd[evt['name']]:
129+
sts = h(evt)
130+
if sts != None and sts < 0:
131+
flag = 1
132+
# call all event handler at last step, skip if any handler has returned -1
133+
if 'allEvent' in hnd and flag == 0:
134+
for h in hnd['allEvent']:
135+
h(evt)
136+
137+
# register event handler
138+
def regHandler(self, sys, name, handler) -> None:
139+
if name == None:
140+
name = 'allEvent' # name = None means handler for all events of this trace system
141+
if sys not in self.handlers:
142+
self.handlers[sys] = {}
143+
# add handler to list
144+
hnd = self.handlers[sys]
145+
if name in hnd:
146+
hnd[name].append(handler)
147+
else:
148+
hnd[name] = [handler]
149+
150+
# register event head parser from raw message
151+
def regParser(self, id, parser) -> int:
152+
if id in self.parsers:
153+
print('Warning! duplicated event header id')
154+
return -1
155+
self.parsers[id] = parser
156+
return 0
157+
158+
# get shared context
159+
def getContext(self) -> dict:
160+
return self.sharedCtx

0 commit comments

Comments
 (0)