Skip to content

Commit

Permalink
Update bench and timings dirs for Py3
Browse files Browse the repository at this point in the history
Minor changes for compatibility.

Cannot currently run these tests effectively:

1. One of the bench runs doesn't work on Py3 yet.  is_types.py fails
with:

```
Traceback (most recent call last):
  File "bench.py", line 91, in <module>
    exec(open(args[0], 'r').read())
  File "<string>", line 16, in <module>
AttributeError: module 'types' has no attribute 'InstanceType'
```

The test fundamentally depends on checking if a type is an instance
as part of what's being timed; the problem is types.InstanceType
was only meaningful for old-style classes where all instances have
the same type.  Since that can't happen in Python 3 with new-style
classes only, types.InstanceType has been removed entirely. Is
it worth trying to figure out how to rejigger the test?

2. The calibrarions on the timings configs are horribly wrong for the
local machine (10x too small), but not sure what to calibrate them for.

3. The calibration algorithm sometimes goes off base. The objective
is to converge on something just under 10 seconds, and hit it three
times in a row. In this snippet #25 got close, then we keep iterating
and get further away in #27, #29, #31, #33 (this particular calibration
eventually hit it's three with runs #55-57.

```
run  25:   9.207:  TARGET_COUNT=1019
run  26:  11.026:  TARGET_COUNT=1106
run  27:   8.630:  TARGET_COUNT=1003
run  28:  12.046:  TARGET_COUNT=1162
run  29:   8.240:  TARGET_COUNT=964
run  30:  12.062:  TARGET_COUNT=1169
run  31:   8.232:  TARGET_COUNT=969
run  32:  12.470:  TARGET_COUNT=1177
run  33:   8.185:  TARGET_COUNT=943
```

4. Running any of the timings fails with stdout mismatch, as there is
(unexpected?) timing output. Example CPPPATH - see the second line
of the captured output for the extra data. Is this new data emitted
by debug=time that the timings code is not expecting?

```
/home/mats/.pyenv/versions/python-3.7.3/bin/python /home/mats/github/scons/src/script/scons.py DIR_COUNT=813 --debug=memory,time .
STDOUT =========================================================================
1,6c1,14
< scons:\ Reading\ SConscript\ files\ \.\.\.\
< scons:\ done\ reading\ SConscript\ files\.\
< scons:\ Building\ targets\ \.\.\.\
< scons:\ `\.'\ is\ up\ to\ date\.\
< scons:\ done\ building\ targets\.\
< .*
---
> scons: Reading SConscript files ...
> SConscript:/tmp/testcmd.256448.xbxk_gj1/SConstruct  took 107.051 ms
> scons: done reading SConscript files.
> scons: Building targets ...
> scons: `.' is up to date.
> scons: done building targets.
> Memory before reading SConscript files:    139239424
> Memory after reading SConscript files:     145399808
> Memory before building targets:            145399808
> Memory after building targets:             148684800
> Total build time: 0.311640 seconds
> Total SConscript file execution time: 0.107460 seconds
> Total SCons execution time: 0.204180 seconds
> Total command execution time: 0.000000 seconds
FAILED test of /home/mats/github/scons/src/script/scons.py
...
	from line 53 of timings/CPPPATH/TimeSCons-run.py
```

5. CPPPATH timing config runs into a command-line length problem if
calibrating.  Might be a testcase for using a file for long command
lines even in the gcc case?

```
run   1:   0.490:  DIR_COUNT=813
1/1 (100.00%) /home/mats/.pyenv/versions/python-3.7.3/bin/python timings/CPPPATH/TimeSCons-run.py
/home/mats/github/scons/src/script/scons.py returned 2
STDOUT =========================================================================
scons: Reading SConscript files ...
SConscript:/tmp/testcmd.253125.kkqcvz12/SConstruct  took 1186.979 ms
scons: done reading SConscript files.
scons: Building targets ...
gcc -o foo.o -c -Iinc_0000 -Iinc_0001 -Iinc_0002
... incredibly long line chopped ...
Iinc_16574 -Iinc_16575 -Iinc_16576 -Iinc_16577 -Iinc_16578 -Iinclude foo.c
scons: building terminated because of errors.
Memory before reading SConscript files:    139243520
Memory after reading SConscript files:     233705472
Memory before building targets:            233705472
Memory after building targets:             255672320
Total build time: 5.079720 seconds
Total SConscript file execution time: 1.189501 seconds
Total SCons execution time: 3.890219 seconds
Total command execution time: 0.000000 seconds
FAILED test of /home/mats/github/scons/src/script/scons.py
...
	from line 53 of timings/CPPPATH/TimeSCons-run.py

STDERR =========================================================================
scons: *** [foo.o] sh: Argument list too long

Traceback (most recent call last):
  File "bin/calibrate.py", line 88, in <module>
    sys.exit(main())
  File "bin/calibrate.py", line 69, in main
    elapsed = float(em.group(1))
AttributeError: 'NoneType' object has no attribute 'group'
```

Signed-off-by: Mats Wichmann <[email protected]>
  • Loading branch information
mwichmann committed Aug 31, 2019
1 parent 7476da3 commit d9f49c4
Show file tree
Hide file tree
Showing 5 changed files with 12 additions and 8 deletions.
3 changes: 2 additions & 1 deletion bench/bench.py
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,8 @@
FunctionList
except NameError:
function_names = sorted([x for x in list(locals().keys()) if x[:4] == FunctionPrefix])
l = [locals()[f] for f in function_names]
lvars = locals()
l = [lvars[f] for f in function_names]
FunctionList = [f for f in l if isinstance(f, types.FunctionType)]

IterationList = [None] * Iterations
Expand Down
2 changes: 1 addition & 1 deletion bench/env.__setitem__.py
Original file line number Diff line number Diff line change
Expand Up @@ -289,7 +289,7 @@ def __setitem__(self, key, value):
# that the timer will use to get at these classes.

class_names = []
for n in locals().keys():
for n in list(locals().keys()):
#if n.startswith('env_'):
if n[:4] == 'env_':
class_names.append(n)
Expand Down
10 changes: 6 additions & 4 deletions bin/calibrate.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@
import subprocess
import sys

variable_re = re.compile('^VARIABLE: (.*)$', re.M)
elapsed_re = re.compile('^ELAPSED: (.*)$', re.M)
variable_re = re.compile(r'^VARIABLE: (.*)$', re.M)
elapsed_re = re.compile(r'^ELAPSED: (.*)$', re.M)

def main(argv=None):
if argv is None:
Expand Down Expand Up @@ -60,7 +60,8 @@ def main(argv=None):
while good < 3:
p = subprocess.Popen(command,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
stderr=subprocess.STDOUT,
universal_newlines=True)
output = p.communicate()[0]
vm = variable_re.search(output)
em = elapsed_re.search(output)
Expand All @@ -70,12 +71,13 @@ def main(argv=None):
print(output)
raise
print("run %3d: %7.3f: %s" % (run, elapsed, ' '.join(vm.groups())))
if opts.min < elapsed and elapsed < opts.max:
if opts.min < elapsed < opts.max:
good += 1
else:
good = 0
for v in vm.groups():
var, value = v.split('=', 1)
# TODO: this sometimes converges slowly, better algorithm?
value = int((int(value) * opts.max) // elapsed)
os.environ[var] = str(value)
run += 1
Expand Down
2 changes: 1 addition & 1 deletion timings/README.txt
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ hierarchy, you use a normal runtest.py invocation to run the timings
configuration:

$ python runtest.py timings/Configuration/TimeSCons-run.py

This runs the entire timing configuration, which actually runs SCons
itself three times:

Expand Down
3 changes: 2 additions & 1 deletion timings/hundred/TimeSCons-run.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,8 @@
test = TestSCons.TimeSCons(variables={'TARGET_COUNT':139})

for t in range(test.variables['TARGET_COUNT']):
open('source_%04d' % t, 'wb' ).write('contents\n')
with open('source_%04d' % t, 'w') as f:
f.write('contents\n')

test.main()

Expand Down

0 comments on commit d9f49c4

Please sign in to comment.