-
Notifications
You must be signed in to change notification settings - Fork 13
/
Makefile.genspec
266 lines (235 loc) · 8.45 KB
/
Makefile.genspec
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
## This Makefile.genspec is used to generate a new test configuration
## for dyninst.
##
## The specification is written in prolog, a logic language.
## The generator uses the logic to omit cases which can not
## be tested, which reduces the test space greatly.
##
## The prolog is written with gprolog (GNU Prolog) constructs
## for some non-common but useful prolog-isms. A different
## prolog, such as swipl, could be used; however gnu-isms
## would need to be switched to the equivalent swipl-isms.
##
## If a prolog standard and support for those constructs
## is supported by the prolog vendors, consider switching
## to using those standard constructs.
##
## In Fall of 2020, I've successfully re-created the testsuite using
## yum provided prolog versions on:
##
## CentOS-7 gprolog-1.4.4
## CentOS-8 gprolog-1.4.4
##
## I have found that the gprolog which ubuntu distributes via apt-get
## does not work reliably, and sometimes crashes.
##
## If you download and compile gprolog yourself, it works
## out of the box on the following platforms & prologs:
##
## ubuntu-18.04 gprolog-1.4.4
## ubuntu-18.04 gprolog-1.4.5
## ubuntu-20.04 gprolog-1.4.5
##
## For ease of use; here is the URL for the current version of gprolog
##
## http://www.gprolog.org/gprolog-1.4.5.tar.gz
##
##
## Previously, the platform directories were somewhat hand-generated.
## With this Fall 2020 version, a complete test hierarchy is generated
## via a test run, including supporting files such as .gitignore.
##
## Care is now taken in the compression of the "tuples" output
## so that, as long as the results don't change, the compressed file
## will not change. Previously, re-producing the platform directory
## from the specification would create a change in tuples.gz due to
## compression artifacts, and resulted in VCS churn.
##
## The test specification is in
##
## src/specification/spec-grouped.pl
##
## Changes there will result in a different set of test files
## being generated.
##
## This Makefile has "SUBDIRS" .. aka DynInst "Platforms", each containing
## the test harness generated for that platform.
##
## 1) gprolog uses its logic to render tests from the specification.
## This is written to 'tuples', and compressed to 'tuples.gz'
##
## 2) A set of python scripts reads the tuples rendered by the prolog
## engine, and creates CMakefiles, cmake lists, directories, and
## source code as appropriate for the test harness for a
## selected platform.
##
## This is done in a platform directory hierarchy, which enumerates
## the various modes of operation of the testsuite, such as
## pic/non-pic, static-linked/dynamic-linked, compiler, and other
## dyninst variants.
##
## The "test groups" are also generated, source files which
## allow for grouping of tests, which speeds test harness
## execution greatly by reducing overhead ... when it works.
## My recommended method to generate a new specification, is to:
## 1) move the existing platform directory someplace else
## 2) gunzip -c tuples.gz >tuples for each OLD platform in that subdir.
## 3) generate the new specification
##
## make -f Makefile.genspec gen-clean gen-all
##
## nB: You can make 1 platform by specifying it as a target;
##
## make -f Makefile.genspec x86_64-unknown-linux2.4
##
## 4) OPTIONAL
## You can use both diff and git to compare the changes to
## the files in the platform directories. This will give you
## some idea of the magnitude of changes you are making to
## the testsuite from the change to the test specification.
##
## Things I look at, for example, are:
## a) Examine the generated tuples, to see if your
## changes are present, and/or as you expected.
## b) Diff the generated source & build files to see how much
## impact they may have on the generated tests, and to see
## if/how the structure of the testsuite changed.
##
## I typically try to build things first, then browse the changes afterward
## if everything looks as you would expect.
##
## 5) Test on changed and unchanged platforms using the new config
## a) examine results.log to verify that tests are run
## b) see if there are unexpected test differences, such as
## missing tests, regressions in testing, failure changes, etc
##
## 6) Once everything is OK, add/delete files and directories as needed
## in git to match the new harness, and commit it.
##
## nB: don't commit the giant 'tuples' files, just the already-in-git
## .gz version
## Tuples remains to allow compare the tuples
## files for correctness against a prolog invocation on
## a different machine, or the prior version.
##
## The first thing to do when wanting to change the test specification
## is to re-generate the existing specification, using the above process.
##
## 1) If you get byte-for-byte results, everything is OK
## 2) If not, there is likely a specification issue, or a prolog issue,
## and you need to find a platform which generates a byte-for-byte
## identical platform directory before proceeding.
##
## Use that compatible platform to make the next test of platform
## directories from the specification.
## These are the DynInst "platforms" (arch+os) that are the tests built for.
## The order of entries is historical... aka what was here!
SUBDIRS = \
i386-unknown-linux2.4 \
x86_64-unknown-linux2.4 \
ppc64_linux \
i386-unknown-freebsd7.2 \
amd64-unknown-freebsd7.2 \
i386-unknown-nt4.0 \
aarch64-unknown-linux \
## BLANK ABOVE
## The test specification directory
TO_SPEC = src/specification
## The GNU prolog files and test specification
## The test specification is in prolog.
## Prolog generates "tuples" from the prolog specification.
## A set of python scripts then turns the "tuples" into the
## files in a platform directory, both source code, and a build
## system to compile that source.
SPEC=spec-grouped.pl
PROLOG_FILES = \
$(TO_SPEC)/$(SPEC) \
$(TO_SPEC)/util.pl \
$(TO_SPEC)/test.pl \
## BLANK ABOVE
## The python source used to generate the platform dirs from the 'tuples'
PYTHON_FILES = \
$(TO_SPEC)/cmake_mutatees.py \
$(TO_SPEC)/cmake_mutators.py \
$(TO_SPEC)/generate.py \
$(TO_SPEC)/group_boilerplate.py \
$(TO_SPEC)/parse.py \
$(TO_SPEC)/test_info_new_gen.py \
$(TO_SPEC)/tuples.py \
$(TO_SPEC)/utils.py \
## BLANK ABOVE
.PHONY: usage gen-all echo $(SUBDIRS) $(SUBDIR_WINDOWS)
usage:
@echo "Use target 'gen-all' to regenerate generated files for all supported"
@echo "platforms"
@echo "Use target 'gen-clean' to remove generated files for all supported"
@echo "platforms"
@echo "Use target PLATFORM to make for a specific platform"
ONE_GENERATED_FILE = tuples tuples.gz cmake-mutatees.txt .gitignore
ALL_GENERATED_FILES = $(foreach dir,$(SUBDIRS),$(ONE_GENERATED_FILE:%=$(dir)/%))
gen-all: $(ALL_GENERATED_FILES)
gen-clean:
-rm -f $(ALL_GENERATED_FILES)
## gzip
## -f force compression
## -c keep tuples around for comparison
## -n don't embed timestamp, so identical contents have identical compressed
## This makes a rebuild not be a git change, unless actual changes occur.
GPROLOG=gprolog
GZIP=gzip
MKDIR=mkdir
$(SUBDIRS:%=%/tuples): %/tuples: $(PROLOG_FILES)
cd $(TO_SPEC); \
$(MKDIR) -p `dirname ../../$@` ; \
$(GPROLOG) \
--entry-goal "['$(SPEC)']" \
--entry-goal "test_init('$*')" \
--entry-goal "write_tuples('../../$@', '$*')" \
--entry-goal "halt" ; \
$(GZIP) -f -n -c ../../$@ >../../[email protected]
$(SUBDIRS:%=%/cmake-mutatees.txt): %/cmake-mutatees.txt: $(PYTHON_FILES) %/tuples
python -c "import sys; import os; os.environ['PLATFORM'] = '$*'; sys.path.append('$(TO_SPEC)'); import generate ; generate.generate('$*')"
$(SUBDIRS:%=%): %:%/cmake-mutatees.txt
## Generate common .gitignore files across platforms.
## Basically Unix, except for the NT platforms
## XXX could use filter/filter-out, or put this in a script, leave it for now.
$(SUBDIRS:%=%/.gitignore): Makefile.genspec
@echo Generating $@
@case $@ in \
i386-unknown-nt4.0*) \
( \
echo Debug ; \
echo resumelog ; \
echo mutatee_resumelog ; \
echo tuples ; \
echo binaries/ ; \
echo '*.obj' ; \
echo '*.pdb' ; \
echo '*.exe' ; \
echo '*.manifest' ; \
echo '*.ilk' ; \
echo '*.dll' ; \
echo '*.lib' ; \
echo '*.exp' ; \
) >$@ ; \
;; \
*) \
( \
echo depends/ ; \
echo runTests ; \
echo test_driver ; \
echo testdriver_wrapper ; \
echo resumelog ; \
echo mutatee_resumelog ; \
echo tuples ; \
echo binaries/ ; \
echo '*.a' ; \
echo '*.o' ; \
echo '*.so' ; \
echo '*_none' ; \
echo '*_low' ; \
echo '*_high' ; \
echo '*_max' ; \
) >$@ ; \
;; \
esac