forked from ctuning/ck-env
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathCHANGES
501 lines (323 loc) · 23.1 KB
/
CHANGES
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
* 2017.12.21 - added lib-xml2 (Linux); HPCToolKit
* 2017.12.06 - added PAPI and HPCToolKit libmonitor
* 2017.11.21 - added extra_name to ck detect soft to separate various installations
such as Python 3.6 vs Anaconda Python 3.6
* 2017.11.20 - improved Intel OpenCL detection for 2017 SDK
- improved CK env for latest Visual Studio 2017 (15.4)
- fixed detection of latest Android NDK on Windows
- fixed detection of cuDNN on Windows
- added possibility to turn on/off specific cores during crowd-tuning
* 2017.11.17 - improved ArmCL detection
* 2017.11.12 - added possibility to enable dependency based on env during package installation
(for example, ck install package --env.OPENCL=ON)
* 2017.11.11 - added detection of NN devices (Movidus)
* 2017.11.10 - moved run_and_get_stdout from CK kernel to module OS
to keep compatibility with the CK now available in official Ubuntu 17.04+
* 2017.11.09 - added possibility to skip tool version recheck (to see if environment changed during expeirments)
- added "script_set_ondemand_cpu_freq": "ck-set-cpu-ondemand"
and "script_set_ondemand_gpu_freq": "ck-set-gpu-ondemand"
to all related OS entries to "calm" CPU/GPU experiments between repetitions
to avoid system overheating
- added possibility to customize package tags from pre_path function:
currently used in ARMCL to add tags based on env such as (USE_OPENCL, USE_GRAPH, USE_NEON, etc)
* 2017.11.08 - added ninja.exe code generator for Windows
* 2017.11.02 - fixing issue #16 - .dylib for MacOS (32&64 bits)
* 2017.10.26 - detect ipython (useful for interactive DNN)
- pass extra params to git (useful for --recursive for Caffe2)
* 2017.10.24 - fixed bug with CUDA 9.0 detection on Windows
* 2017.10.23 - added support for OS extensions when detecting soft (useful for .dylib on MacOS)
* 2017.10.18 - added latest JDK 9.0.1 and stable 8u131
- added LLVM 5.0.0
* 2017.10.16 - added scons (soft & latest package)
* 2017.10.14 - added newer wget.exe for Windows (to support latest SSL)
- added newer unzip.exe for Windows
- added channel-stable to stable packages
* 2017.10.13 - added possibility to specify "$#path_from_cid=" in install_env in packages
* 2017.10.12 - added "or_tags" to deps to select soft/env/package by groups of tags with OR operator
* 2017.10.10 - added support to rebuild deps during replay from another machine using package UOA
* 2017.10.09 - changed lib.acl to lib.armcl
* 2017.10.05 - added soft:tool.valgrind
* 2017.10.04 - added dvdt-prof2 to latest dvdt prof and asking for Boost >= 1.64.0
- added package sorting by "sort" key besides version and name
(to be able to suggest recommended)
* 2017.10.03 - added support to customize detect soft via --env.XYZ=abc
- added support to set version_from and version_to in deps
and in ck detect soft, for example
$ ck detect soft --tags=boost --version_from=1.62.0 --version_to=1.65.0
- added "get_all_versions_in_deps" function to module:env
to list recursively all versions of SW deps
* 2017.10.01 - minimized questions when installing packages
* 2017.09.29 - added support for package channels and comments
When package channel is specified via "ck set kernel var.package_channel={stable}",
"channel-stable" will be added to all package tags during installation thus stabilizing package
Moving back to development version is by "ck set kernel var.package_channel="
- If package meta has key "comment", it will be printed in a package selector
* 2017.09.28 - added support to run Python scripts from CK entries
* 2017.09.21 - added support to cache and reuse dependencies by tags (for automating crowdbenchmarking)
- reverted update of symlinks in soft
* 2017.09.19 - minor output improvement
* 2017.09.03 - added "ck virtual env" to pre-set environment in a given/new shell for debugging
* 2017.06.06 - added flag --full_path to specify path to the installed soft during "ck detect soft"
* 2017.05.25 - added flag --soft_name to specify non-standard tool name to search
* 2017.05.22 - added --search_dirs in ck detect soft to specify extra dirs where to search soft during auto-detection
- improving GCC detection on non-standard platforms (useful for Caffe* installations)
* 2017.05.15 - improving detection of different versions of python and pip (useful for TensorFlow)
- added --extra_tags to install package
* 2017.05.10 - added protobuf 3.2 & 3.3 for Linux, Windows and Android (needed for Caffe2)
* 2017.05.08 - added ARM compute lib description
- passing updated env to all package/soft customization scripts
* 2017.04.28 - added 'ck distribution package' to create binary distributions
(useful for Caffe crowd-benchmarking and crowd-tuning)
* 2017.04.26 - added support for LMDB lib for Windows
* 2017.04.25 - added experimental mode to rebuild package without download (ck install package:{name} --rebuild)
- added ck xset env with default flags to quickly set env from command line (useful for caffe python)
* 2017.04.23 - added possibility to customize path during package installation (--path=)
* 2017.04.22 - fixed bug when detecting already installed via CK software (restoring tags!)
* 2017.04.21 - fixed recording of package UOA when automatically detecting already installed soft (restoring env - needed for Caffe crowd-benchmarking)
* 2017.04.20 - added LMDB universal package (not completely working on Windows) ...
* 2017.04.19 - added adb.exe for Windows to os:windows-32
* 2017.04.18 - added package to automatically download and install GCC 6.3.0 for Windows
(needed for OpenBLAS + Caffe)
- added perl-strawberry-5.24.1.1-win (also for OpenBLAS on Windows)
- added use_locale_for_version to detect Microsoft compiler version on non-English Windows
(tested on French one)
* 2017.04.17 - added support for AMD OpenCL lib
* 2017.04.15 - moved many generic packages from ck-autotuning here
- added LLVM 4.0.0 universal package (still need to fix for Linux)
- fixed env refresh when extra package name is available (useful for caffe/tensorflow models)
* 2017.04.14 - fixed mingw gfortran detection on Windows (for Caffe)
* 2017.04.11 - improved LLVM description for Android targets
* 2017.03.28 - fixed java detection
* 2017.03.26 - added soft:env.msys2 needed by bazel for Windows (to build TensorFlow, etc)
* 2017.03.24 - added tag modification in deps for host/target OS
- improved support for customized scripts in packages (to update URLs for ARM/Intel + Win/Linux/MacOS/Android)
* 2017.03.16 - added support for VS2017 Community edition (cl + clang)
- added support for PACKAGE_URL_MACOS and PACKAGE_GIT_CHECKOUT_MACOS in universal packages
- improved universal scripts
- added soft:compiler.perl (to compile OpenBLAS on Windows)
- added soft:utils.unix.win (set of Unix Utils on Windows useful to compile OpenBLAS on Windows)
* 2017.03.02 - added VIENNACL_CACHE_PATH to soft:lib.viennacl
(now can cache kernels - critical to cache Caffe OpenCL init time by 5x)
* 2017.02.27 - remove equal paths pointed by symbolic links when detecting installed soft
* 2017.02.09 - added "force_cmd_version_detection" and "soft_version_skip_add_target_file"
to package meta (useful for ARM WA version detection even if taken from GIT)
* 2017.01.17 - added soft:lib.libdnn and moved soft:lib.tiny-dnn here
* 2017.01.08 - many fixes for Caffe deps for Windows/Android
* 2017.01.07 - various fixes for Caffe deps on Windows/Android
* 2017.01.03 - added '--safe' mode to check environment (will first search for CK packages and only then detecting soft)
* 2017.01.01 - added patch.exe (to support compiling Caffe on Windows)
* 2016.12.30 - added installation to env (ck install package:{name} --install_to_env
or persistently: 'ck setup kernel --var.install_to_env="yes"'
* 2016.12.29 - automatically duplicating all deps with 'compiler' in the key
to avoid mix ups of env by packages compiled with different compiler ...
- added OS android19-arm-v7a-hard-neon, android19-arm-v7a-neon, android21-arm-v7a-hard-neon, android21-arm-v7a-neon
* 2016.12.28 - supporting duplication of deps (primarily to reset compiler vars to correct values at the end)
* 2016.12.27 - supporting LLVM from the new Android NDK (for Caffe)
* 2016.12.26 - updated wget for Windows (to download Caffe deps without SSL certificate)
* 2016.12.21 - major update to record package configuration during installation
to be able to correctly re-register soft when env is cleaned ...
It also allows to pre-record information about version
and other hints to CK in the CK-compatible third-party packages (bright future ;) ).
* 2016.12.20 - added detection of Raspberry Pi platform features
- added detection of Odroid platform
- added "ck clean env:*" to remove both CK entries and related packages
* 2016.12.19 - fixing typos in soft scripts
* 2016.12.13 - fixing bug in module 'machine'
* 2016.12.10 - added script:download-and-install-package-with-cmake to start unifying download + cmake + install
- added 'ask_version' for packages
* 2016.12.07 - fixing Bazel soft desc
* 2016.12.05 - added os:*arm-v7a (mainly for Caffe for Android)
* 2016.12.02 - added user-friendly print of OS alias
- added better error reporting when host/target OS is not matching env entries
* 2016.12.01 - added soft:compiler.llvm.android.ndk (and started preparing paths without GCC)
- a few cosmetic changes
* 2016.11.30 - do not show empty dependencies which are not for a used target
- fixed using of preprocess scripts from another entry
* 2016.11.28 - do not look for connected Android device, when just refreshing existing env for it
* 2016.11.24 - fixing version detection of texinfo
* 2016.11.20 - detect OS ABI using uname -m in 'detect platform.os'
- added container for any result
* 2016.11.19 - added possibility to skip setup during package installation
(for example for root installation)
- added soft:lib.ncurses
- added "force_env_init" to dependency description to force env script execution
(useful for LLVM plugins)
* 2016.11.18 - added LLVM plugin soft description
* 2016.11.16 - added LLVM lib and include to be able to compile plugins
* 2016.11.14 - added support to reuse preinstall scripts from other entries during package installation
- added support for CrystaX Android NDK
- added extra soft name customization from soft customize.py scripts
- added soft:process-compiler-for-cmake (to compile complex projects such as Caffe with all deps via CK)
* 2016.11.11 - fixed bug: no possible not to search for installed software
but install from packages (useful for Caffe for Android via CK)
* 2016.11.10 - skipping Android device detection during software detection (if not needed)
- added gflags soft description
- added glog soft description
- added lmdb soft description
- removed soft:lib.opencv.android and moved Android target to lib.opencv
- added possibility to set ABI in the installation path (for Android)
- added protobuf soft description
* 2016.11.08 - improved software detection (now can include sub-path in search)
- can add GPGPU features to installation scripts
- added soft:compiler.computecpp
* 2016.11.07 - added soft description 'tool.bazel' (useful to recompile Google's TensorFlow)
* 2016.11.04 - making os.getcwd() safer
* 2016.10.24 - added lib.tiny-dnn
- fixes possible issues when installing packages for Android devices
* 2016.10.21 - added dependency on ARM WA tool if adding target machine that uses WA ...
- added soft:android.sdk description and detection
* 2016.10.18 - added 'no_tags' to prune environments (set function) ...
* 2016.10.12 - added latest x86/x86_64 android OS descriptions (needed for ARM Workload Automation via CK)
- added 'ck browse platform.*'
* 2016.10.11 - added 'shell' function to 'os' module to automate simple script execution
* 2016.10.10 - added detection of CPU clusters for Workload automation
* 2016.10.05 - obtaining real serial number of ADB devices (if via network) + adding dep on ADB tool for remote devices
* 2016.09.27 - added support for remote execution unified across all platforms
* 2016.09.26 - changed 'device' to 'machine'
* 2016.09.25 - various improvements in package/soft ...
* 2016.09.24 - added tool.show-cuda-devices and tool.show-opencl-devices
for CK widget "platform.gpgpu"
* 2016.09.23 - added optional flag --target to all platform.*
* 2016.09.21 - improved "device" for ARM WA
* 2016.09.19 - added soft:compiler.gfortran
- added soft:tool.adb (for ARM workload automation)
* 2016.09.18 - added many new Android descriptions (past and new such as Android 7.0.0)
- added 'add', 'browse' and 'show' functions to 'device'
- improving new CPU format detection (Linux/Android/MacOS)
* 2016.09.10 - added 'device' module to unify description of available user devices for crowd-benchmarking/crowd-tuning
* 2016.09.09 - updated DVDT OpenCL profiler soft description
* 2016.09.07 - added --env and --env. keys to ck install package to be able to preset env variables
* 2016.09.05 - added HDF5 library description
- changed soft functions 'detect' to 'internal_detect' and 'check' to 'detect'
* 2016.09.01 - fixed recursive search of directories in Python 2.x with non-ascii directories
* 2016.08.30 - added -fPIE to LLVM soft desc for Android based on ARM64
* 2016.08.11 - fixed detection of environment if already installed when customized version have changed
- added record_script flag to quickly rebuild packages without CK
* 2016.08.08 - added flag "extra_path" to package install
- added detection of GIT info (revision + date time of last commit) with possibility to add to version
(to differentiate the same packages with different GIT revisions)
* 2016.08.03 - fixing static library compilation on MacOS using CLANG (such as OpenME)
* 2016.08.02 - fixed various problems with crowdtuning on MacOS
* 2016.07.29 - added possibility to skip compiler name from suggested path during installation (no_compiler_in_suggested_path)
* 2016.07.24 - added detection of version change (for tools) when setting environments in deps
- added dependency on GCC in NVCC (Linux)
* 2016.07.12 - added 'extra_suggested_path' to customization of packages
* 2016.07.07 - added 'no_os_in_suggested_path' to customization of packages to avoid
adding OS info at the end of the suggested path. Useful when installing
various OS-independent datasets;
added 'no_ver_in_suggested_path';
remove the same paths when automatically detecting software
(due to possible soft links);
added sha1sum.exe to windows
* 2016.07.06 - added 'force_ask_path' to customization of packages to ask path
in interactive mode. Useful when installing large training sets
for DNN
* 2016.07.04 - checking if package was already installed...
* 2016.07.03 - fixed freeglut description
* 2016.07.01 - fixed a few bugs reported by Michael Mcgeagh
* 2016.06.29 * added more soft descriptions (Apache thrift, commons-io)
* 2016.06.28 * added support to pass environment variables to installation script via -Dkey=value (similar to cMAKE)
* 2016.06.26 * added support for the new CPU format in Linux/Android
* 2016.06.22 * added support to customize installation path with any extra deps ('add_to_path')
* 2016.06.20 * added sh to Android description
* 2016.06.19 * fixing Boost description for Linux
* fixing openBLAS description for Linux and Android
* fixing PENCIL benchmark description
* fixing ViennaCL and TBB for Linux
* 2016.06.11 * added support to skip GPU detection (for example, not needed
during compiler flag crowd-tuning but may cause problems),
and to force platform.init UOA
* 2016.06.07 * improving package manager:
- detecting host CPU info and passing CK_HOST_CPU_NUMBER_OF_PROCESSORS to process script
- passing extra env to process script
- running pre-processing python script
- running scripts from another entry (light inheritance)
- adding CLBLAST soft description
* 2016.05.30 * support for new CK web server
* 2016.05.29 - adding 'exchange_locally' mode to platform.* detection and info exchange
* 2016.05.17 - improving different soft description
- adding soft description for cuBLAS, cuFFT and cuDNN
* 2016.05.16 - changing CK_CC to CK_NVCC for CUDA soft description
* 2016.05.15 - removing outdated remote-ck1 repository
* 2016.05.13 - moving platform.gpgpu to ck-autotuning repo
* 2016.05.09 - adding 7z to Windows OS env
* 2016.05.06 - adding Julia compiler (after MIT visit)
* 2016.04.28 - major update to automate detection of installed software (for Windows, Linux and Android).
To some extent, CK now works as a customizable cmake with the ability to share
software detection modules in the CK format via GitHub.
* 2016.04.18 - adding compiler flag via environment -std=c++0x (CK_COMPILER_FLAG_CPP0X)
* 2016.04.15 - moved platform.init entries from ck-autotuning here
- added windows tools to windows-32 OS entry
- added DragonBoard platform.init
- improving software/package installation; showing wiki page when problem
- added possibility to show user-friendly soft description when resolving dependencies
* 2016.04.14 - added lib.toon for PAMELA project
improving customize.py format to setup up software
* 2016.04.12 - added software descriptions for TETRACOM project:
freeglut, glew, antlr, glslangvalidator, ogl.get-image, ogl.parser
* 2016.04.08 - started improving automatic detection of software and registration in CK
* 2016.03.25 - improving NN visualization
* 2016.03.24 - adding legacy support and demo for MILEPOST GCC, cTuning plugins, ctuning-cc
* 2016.03.21 - adding universal OpenCL lib soft description for Android devices
* 2016.03.09 - adding soft description for compiler.llvm.android (arm,x86,x86_64)
adding soft description for texinfo to reproduce MILEPOST GCC
(for semantic features extraction)
* 2016.03.01 - adding soft description for CLBLAS, CLBLAST, CLTUNE
* 2016.02.18 - changing platform.accelerator -> platform.gpu
* 2016.02.13 - * adding support to automatically resolve, search and install dependencies
(flag - quiet - useful when crowdsourcing optimization and machine learning)
* 2016.02.10 - * improving CLANG description for Android/ARM via Android NDK
* improving CLANG version detection
* fixing detection of platform features (correct reloading of existing features
- useful for arch specific compiler flag crowdtuning)
* 2016.02.09 - * adding postfix for GCC env registration (such as -4.6)
* 2016.02.05 - * record all info during platform detection only
when ck.cfg['record_all_platform_info']=='yes'.
Needed to avoid mess when recording local or global info
during experiment crowdsourcing.
* 2016.01.27 - * fixing installation of GCC trunk and LLVM trunk
(preparing new engine to crowdsource optimizations)
* adding support to select SW dependencies randomly
(useful to crowdsource experiments)
* 2016.01.16 - * adding UOA and UID of found platforms into detection stream
* 2016.01.12 - * adding possibility to enter and record name of platform, accelerator, cpu and OS
to local cfg if not automatically detected (for crowdtuning)
* 2016.01.11 - * fixing remote repo recaching if selecting different public CK server
during program optimization crowdsourcing
* fixing automatic package installation bug (when crowdsourcing autotuning)
* 2016.01.04 - * adding 'show' function to platform.* to list all platforms
participating in crowd-tuning and their parameters
* 2016.01.03 - * fixing exchange of information during crowd-tuning
(possibility to find platform description in all repos)
* adding possibility to identify user when recording
results during crowdtuning
* 2016.01.02 - * fixing GPU/accelerator detection for ARM/Qualcomm (for crowdtuning)
* fixing platform name detection (to crowdsource optimization)
* 2015.12.19 - * adding env var CK_PLATFORM_INIT_UOA to automatically
add platform scripts to PATH when dealing with CPU/GPU frequency
* 2015.12.14 - * fixing soft:lib.opencl.mali compilation for Android
* added /data/local/tmp/lib to LD_LIBRARY_PATH when
benchmarking/running/autotuning/crowdtuning on Android
(to copy there development libs for dynamic apps
while keeping directory clean)
* 2015.11.21 - * adding "batch_exit" : "exit /b $#return_code#$" to Windows and MingW OS descriptions
* 2015.11.20 - * improving OpenCL lib setup on Ubuntu
* 2015.11.09 - * extending soft:compiler.llvm - adding tool_postfix, for example -3.6 for clang-3.6
* 2015.10.29 - * adding env "CK_COMPILER_FLAG_MFLOAT_ABI_HARD": "-mfloat-abi=hard" to LLVM
* 2015.10.28 - * when automatically installing packages, add not only 'compiler' to dir name,
but also 'support_compiler' ...
* changing 'compiler' to 'source-to-source-compiler' tag for pencilcc
* 2015.10.27 - * if more than one package available, allow selection
* fixing bug with compiler dependencies when installing packages
* fixing pencil deps
* 2015.10.21 - * fixing pencil description from CARP project
* adding pencil benchmark env description
* 2015.09.25 - * renaming soft:compiler.mcl to compiler.msvc (for Microsoft C compilers)
(suggested by Anton)
* 2015.09.17 - * automatically calculating integer soft version for internal comparisons
* 2015.09.10 - * adding 'platform.init' module to share scripts and tools that can initialize a given platform
* 2015.08.24 - * adding support to detect Windows 10 ...
* 2015.08.11 - * fixing bug with detection of standard Linux
* 2015.08.09 - * fixing stupid bug with Y/N when installing packages automatically (detected by Cedric)
* fixing "ck show env" when using tags and target_os -> was no comma separating sub-tags
* 2015.08.07 - fixed a small bug with setting environment for a given target OS