forked from Rdatatable/data.table
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathCRAN_Release.cmd
533 lines (466 loc) · 23.6 KB
/
CRAN_Release.cmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
###############################################
# Basic checks
###############################################
sudo apt-get update
sudo apt-get upgrade
R
update.packages(ask=FALSE)
# See here for getting R.oo to install: https://github.com/HenrikBengtsson/R.utils/issues/29
q("no")
# Ensure latest version of R otherwise problems with CRAN not finding dependents that depend on latest R
# e.g. mirror may have been disabled in sources.list when upgrading ubuntu
rm ./src/*.so
rm ./src/*.o
rm -rf ./data.table.Rcheck
# Ensure no non-ASCII, other than in README.md is ok
# tests.Rraw in particular have failed CRAN Solaris (only) due to this.
grep -RI --exclude-dir=".git" --exclude="*.md" --exclude="*~" --color='auto' -P -n "[\x80-\xFF]" ./
# Unicode is now ok. This unicode in tests.Rraw is passing on CRAN.
grep -RI --exclude-dir=".git" --exclude="*.md" --exclude="*~" --color='auto' -n "[\]u[0-9]" ./
# Ensure no calls to omp_set_num_threads() [to avoid affecting other packages and base R]
# Only comments referring to it should be in openmp-utils.c
grep omp_set_num_threads ./src/*
# Ensure no calls to omp_get_max_threads() also since access should be via getDTthreads()
grep --exclude="./src/openmp-utils.c" omp_get_max_threads ./src/*
# Ensure all #pragama omp parallel directives include a num_threads() clause
grep "pragma omp parallel" ./src/*.c
# Ensure all .Call's first argument are unquoted. TODO - change to use INHERITS()
grep "[.]Call(\"" ./R/*.R
# Ensure no Rprintf in init.c
grep "Rprintf" ./src/init.c
# workaround for IBM AIX - ensure no globals named 'nearest' or 'class'.
# See https://github.com/Rdatatable/data.table/issues/1351
grep "nearest *=" ./src/*.c # none
grep "class *=" ./src/*.c # quite a few but none global
# No undefined type punning of the form: *(long long *)&REAL(column)[i]
# Failed clang 3.9.1 -O3 due to this, I think.
grep "&REAL" ./src/*.c
# No use of long long, instead use int64_t. TODO
# grep "long long" ./src/*.c
# No tabs in C or R code (sorry, Richard Hendricks)
grep -P "\t" ./R/*.R
grep -P "\t" ./src/*.c
# No T or F symbols in tests.Rraw. 24 valid F (quoted, column name or in data) and 1 valid T at the time of writing
grep -n "[^A-Za-z0-9]T[^A-Za-z0-9]" ./inst/tests/tests.Rraw
grep -n "[^A-Za-z0-9]F[^A-Za-z0-9]" ./inst/tests/tests.Rraw
# No system.time in main tests.Rraw. Timings should be in benchmark.Rraw
grep -n "system[.]time" ./inst/tests/tests.Rraw
# seal leak potential where two unprotected API calls are passed to the same
# function call, usually involving install() or mkChar()
# Greppable thanks to single lines and wide screens
# See comments in init.c
cd ./src
grep install.*alloc *.c --exclude init.c
grep install.*Scalar *.c
grep alloc.*install *.c --exclude init.c
grep Scalar.*install *.c
grep mkChar.*alloc *.c
grep mkChar.*Scalar *.c
grep alloc.*mkChar *.c
grep Scalar.*mkChar *.c
grep "getAttrib.*mk" *.c # use sym_* in getAttrib calls
grep "PROTECT *( *getAttrib" *.c # attributes are already protected
grep "\"starts\"" *.c --exclude init.c
grep "setAttrib(" *.c # scan all setAttrib calls manually as a double-check
grep "install(" *.c --exclude init.c # TODO: perhaps in future pre-install all constants
grep mkChar *.c # see comment in bmerge.c about passing this grep. I'm not sure char_ is really necessary, though.
# ScalarInteger and ScalarString allocate and must be PROTECTed unless i) returned (which protects),
# or ii) passed to setAttrib (which protects, providing leak-seals above are ok)
# ScalarLogical in R now returns R's global TRUE from R 3.1.0; Apr 2014. Before that it allocated.
# Aside: ScalarInteger may return globals for small integers in future version of R.
grep ScalarInteger *.c # Check all Scalar* either PROTECTed, return-ed or passed to setAttrib.
grep ScalarLogical *.c # Now we depend on 3.1.0, check ScalarLogical is NOT PROTECTed.
grep ScalarString *.c
# Inspect missing PROTECTs
# To pass this grep is why we like SET_VECTOR_ELT(,,var=allocVector()) style on one line.
# If a PROTECT is not needed then a comment is added explaining why and including "PROTECT" in the comment to pass this grep
grep allocVector *.c | grep -v PROTECT | grep -v SET_VECTOR_ELT | grep -v setAttrib | grep -v return
cd ~/GitHub/data.table
R
cc(clean=TRUE) # to compile with -pedandic. Also use very latest gcc (currently gcc-7) as CRAN does
saf = options()$stringsAsFactors
options(stringsAsFactors=!saf) # check tests (that might be run by user) are insensitive to option, #2718
test.data.table()
q("no")
R CMD build .
R CMD check data.table_1.11.3.tar.gz --as-cran # remove.packages("xml2") first to prevent the 150 URLs in NEWS.md being pinged by --as-cran
R CMD INSTALL data.table_1.11.3.tar.gz
R
require(data.table)
test.data.table()
test.data.table(verbose=TRUE) # since main.R no longer tests verbose mode
gctorture2(step=50)
system.time(test.data.table()) # apx 75min
# Test C locale doesn't break test suite (#2771)
echo LC_ALL=C > ~/.Renviron
R
Sys.getlocale()=="C"
q("no")
R CMD check data.table_1.11.3.tar.gz
rm ~/.Renviron
# Upload to win-builder: release, dev & old-release
###############################################
# R 3.1.0 (stated dependency)
###############################################
### ONE TIME BUILD
sudo apt-get -y build-dep r-base
cd ~/build
wget http://cran.stat.ucla.edu/src/base/R-3/R-3.1.0.tar.gz
tar xvf R-3.1.0.tar.gz
cd R-3.1.0
./configure --without-recommended-packages
make
alias R310=~/build/R-3.1.0/bin/R
### END ONE TIME BUILD
cd ~/GitHub/data.table
R310 CMD INSTALL ./data.table_1.11.3.tar.gz
R310
require(data.table)
test.data.table()
############################################
# Check compiles ok when OpenMP is disabled
############################################
vi ~/.R/Makevars
# Make line SHLIB_OPENMP_CFLAGS= active to remove -fopenmp
R CMD build .
R CMD INSTALL data.table_1.11.3.tar.gz # ensure that -fopenmp is missing and there are no warnings
R
require(data.table) # observe startup message about no OpenMP detected
test.data.table()
#####################################################
# R-devel with UBSAN, ASAN and strict-barrier on too
#####################################################
cd ~/build
wget -N https://stat.ethz.ch/R/daily/R-devel.tar.gz
rm -rf R-devel
tar xvf R-devel.tar.gz
cd R-devel
# Following R-exts#4.3.3
./configure --without-recommended-packages --disable-byte-compiled-packages --disable-openmp --enable-strict-barrier CC="gcc -fsanitize=undefined,address -fno-sanitize=float-divide-by-zero -fno-omit-frame-pointer" CFLAGS="-O0 -g -Wall -pedantic" LIBS="-lpthread"
# For ubsan, disabled openmp otherwise gcc fails in R's distance.c:256 error: ‘*.Lubsan_data0’ not specified in enclosing parallel
# UBSAN gives direct line number under gcc but not clang it seems. clang-5.0 has been helpful too, though.
# If use later gcc-8, add F77=gfortran-8
# LIBS="-lpthread" otherwise ld error about DSO missing
# -fno-sanitize=float-divide-by-zero, otherwise /0 errors on R's summary.c (tests 648 and 1185.2) but ignore those:
# https://bugs.r-project.org/bugzilla3/show_bug.cgi?id=16000
make
alias Rdevel='~/build/R-devel/bin/R --vanilla'
cd ~/GitHub/data.table
Rdevel CMD INSTALL data.table_1.11.3.tar.gz
# Check UBSAN and ASAN flags appear in compiler output above. Rdevel was compiled with them so should be passed through to here
Rdevel
install.packages(c("bit64","xts","nanotime"), repos="http://cloud.r-project.org") # minimum packages needed to not skip any tests in test.data.table()
require(data.table)
test.data.table() # 7 mins (vs 1min normally) under UBSAN, ASAN and --strict-barrier
for (i in 1:100) if (!test.data.table()) break # try several runs; e.g a few tests generate data with a non-fixed random seed
# gctorture(TRUE) # very slow, many days
gctorture2(step=100) # [12-18hrs] under ASAN, UBSAN and --strict-barrier
print(Sys.time()); started.at<-proc.time(); try(test.data.table()); print(Sys.time()); print(timetaken(started.at))
## In case want to ever try again with 32bit on 64bit Ubuntu for tracing any 32bit-only problems
## dpkg --add-architecture i386
## apt-get update
## apt-get install libc6:i386 libstdc++6:i386 gcc-multilib g++-multilib gfortran-multilib libbz2-dev:i386 liblzma-dev:i386 libpcre3-dev:i386 libcurl3-dev:i386 libstdc++-7-dev:i386
## sudo apt-get purge libcurl4-openssl-dev # cannot coexist, it seems
## sudo apt-get install libcurl4-openssl-dev:i386
## cd ~/build/32bit/R-devel
## ./configure --without-recommended-packages --disable-byte-compiled-packages --disable-openmp --without-readline --without-x CC="gcc -m32" CXX="g++ -m32" F77="gfortran -m32" FC=${F77} OBJC=${CC} LDFLAGS="-L/usr/local/lib" LIBnn=lib LIBS="-lpthread" CFLAGS="-O0 -g -Wall -pedantic"
##
###############################################
# valgrind in R-devel (see R-exts$4.3.2)
###############################################
cd ~/build
rm -rf R-devel # easiest way to remove ASAN from compiled packages in R-devel/library
# to avoid "ASan runtime does not come first in initial library list" error; no need for LD_PRELOAD
tar xvf R-devel.tar.gz
cd R-devel
./configure --without-recommended-packages --disable-byte-compiled-packages --disable-openmp --with-valgrind-instrumentation=1 CC="gcc" CFLAGS="-O0 -g -Wall -pedantic" LIBS="-lpthread"
make
cd ~/GitHub/data.table
vi ~/.R/Makevars # make the -O0 -g line active, for info on source lines with any problems
Rdevel CMD INSTALL data.table_1.11.3.tar.gz
Rdevel -d "valgrind --tool=memcheck --leak-check=full --track-origins=yes --show-leak-kinds=definite"
# gctorture(TRUE) # very slow, many days
# gctorture2(step=100)
print(Sys.time()); require(data.table); print(Sys.time()); started.at<-proc.time(); try(test.data.table()); print(Sys.time()); print(timetaken(started.at))
# 3m require; 62m test
# Investigated and ignore :
# Tests 648 and 1262 (see their comments) have single precision issues under valgrind that don't occur on CRAN, even Solaris.
# Ignore all "set address range perms" warnings :
# http://stackoverflow.com/questions/13558067/what-does-this-valgrind-warning-mean-warning-set-address-range-perms
# Ignore heap summaries around test 1705 and 1707/1708 due to the fork() test opening/closing, I guess.
# Tests 1729.4, 1729.8, 1729.11, 1729.13 again have precision issues under valgrind only.
# Leaks for tests 1738.5, 1739.3 but no data.table .c lines are flagged, rather libcairo.so
# and libfontconfig.so via GEMetricInfo and GEStrWidth in libR.so
vi ~/.R/Makevars # make the -O3 line active again
#############################################################################
# TODO: recompile without USE_RINTERNALS and recheck write barrier under ASAN
#############################################################################
There are some things to overcome to achieve compile without USE_RINTERNALS, though.
########################################################################
# Single precision e.g. CRAN's Solaris-Sparc when it was alive on CRAN.
# Not Solaris-x86 which is still on CRAN and easier.
########################################################################
#
# Adding --disable-long-double (see R-exts) in the same configure as ASAN/UBSAN fails. Hence separately.
## cd ~/build
## rm -rf R-devel # 'make clean' isn't enough: results in link error, oddly.
## tar xvf R-devel.tar.gz
## cd R-devel
## ./configure CC="gcc -std=gnu99" CFLAGS="-O0 -g -Wall -pedantic -ffloat-store -fexcess-precision=standard" --without-recommended-packages --disable-byte-compiled-packages --disable-long-double
## make
## Rdevel
## install.packages("bit64")
## q("no")
## Rdevel CMD INSTALL ~/data.table_1.9.9.tar.gz
## Rdevel
## .Machine$sizeof.longdouble # check 0
## require(data.table)
## require(bit64)
## test.data.table()
## q("no")
###############################################
# QEMU to emulate big endian
###############################################
# http://www.aurel32.net/info/debian_sparc_qemu.php
# https://people.debian.org/~aurel32/qemu/powerpc/
### IF NOT ALREADY DONE
sudo apt-get -y install qemu
sudo apt-get -y install openbios-ppc
cd ~/build
wget https://people.debian.org/~aurel32/qemu/powerpc/debian_wheezy_powerpc_standard.qcow2
### ENDIF
qemu-system-ppc -hda debian_wheezy_powerpc_standard.qcow2 -nographic
root, root
lscpu # confirm big endian
wget http://cran.r-project.org/src/base/R-latest.tar.gz
tar xvf R-latest.tar.gz
cd R-3.2.1
apt-get update
apt-get build-dep r-base
uptime # the first time, several hours. But it saves the disk image in the .qcow2 file so be sure not to wipe it
./configure CFLAGS="-O0 -g" --without-recommended-packages --disable-byte-compiled-packages
# -O0 to compile quicker. Keeping -g in case needed to debug
make
alias R=$PWD/bin/R
cd ~
wget -N https://github.com/Rdatatable/data.table/archive/master.zip # easiest way; avoids setting up share between host and qemu
apt-get install unzip
unzip master.zip
mv data.table-master data.table
R CMD build --no-build-vignettes data.table
# R CMD check requires many packages in Suggests. Too long under emulation. Instead run data.table's test suite directly
R
options(repos = "http://cran.stat.ucla.edu")
install.packages("bit64") # important to test data.table with integer64 on big endian
q("no")
R CMD INSTALL data.table_1.9.5.tar.gz
R
.Platform$endian
require(data.table)
test.data.table()
q()
shutdown now # doesn't return you to host prompt properly so just kill the window. http://comments.gmane.org/gmane.linux.embedded.yocto.general/3410
###############################################
# Downstream dependencies
###############################################
# IF NOT ALREADY INSTALLED
sudo apt-get update
sudo apt-get -y install htop
sudo apt-get -y install r-base r-base-dev
sudo apt-get -y build-dep r-base-dev
sudo apt-get -y build-dep qpdf
sudo apt-get -y install aptitude
sudo aptitude build-dep r-cran-rgl # leads to libglu1-mesa-dev
sudo apt-get -y build-dep r-cran-rmpi
sudo apt-get -y build-dep r-cran-cairodevice
sudo apt-get -y build-dep r-cran-tkrplot
sudo apt-get -y install libcurl4-openssl-dev # needed by devtools
sudo apt-get -y install xorg-dev x11-common libgdal-dev libproj-dev mysql-client libcairo2-dev libglpk-dev
sudo apt-get -y install texlive texlive-latex-extra texlive-bibtex-extra texlive-science texinfo texlive-fonts-extra
sudo apt-get -y install libv8-dev
sudo apt-get -y install gsl-bin libgsl0-dev
sudo apt-get -y install libgtk2.0-dev netcdf-bin
sudo apt-get -y install libcanberra-gtk-module
sudo apt-get -y install git
sudo apt-get -y install openjdk-8-jdk
sudo apt-get -y install libnetcdf-dev udunits-bin libudunits2-dev
sudo apt-get -y install tk8.6-dev
sudo apt-get -y install clustalo # for package LowMACA
sudo apt-get -y install texlive-xetex # for package optiRum
sudo apt-get -y install texlive-pstricks # for package GGtools
sudo apt-get -y install libfftw3-dev # for package fftwtools
sudo apt-get -y install libgsl-dev
sudo apt-get -y install octave liboctave-dev
sudo apt-get -y install jags
sudo apt-get -y install libmpfr-dev
sudo apt-get -y install bwidget
sudo apt-get -y install librsvg2-dev # for rsvg
sudo apt-get -y install libboost-all-dev libboost-locale-dev # for textTinyR
sudo apt-get -y install libsndfile1-dev # for seewave
sudo apt-get -y install libpoppler-cpp-dev # for pdftools
sudo apt-get -y install libapparmor-dev # for sys
sudo apt-get -y install libmagick++-dev # for magick
sudo apt-get -y install libjq-dev libprotoc-dev libprotobuf-dev and protobuf-compiler # for protolite
sudo R CMD javareconf
# ENDIF
cd ~/build/revdeplib/
export R_LIBS=~/build/revdeplib/
export R_LIBS_SITE=none
export _R_CHECK_FORCE_SUGGESTS_=false # in my profile so always set
R
.libPaths() # should be just 2 items: revdeplib and the base R package library
update.packages(ask=FALSE)
# if package not found on mirror, try manually a different one:
install.packages("<pkg>", repos="http://cloud.r-project.org/")
update.packages(ask=FALSE) # a repeat sometimes does more, keep repeating until none
# Follow: https://bioconductor.org/install/#troubleshoot-biocinstaller
# Ensure no library() call in .Rprofile, such as library(bit64)
source("http://bioconductor.org/biocLite.R")
biocLite()
biocLite() # keep repeating until returns with nothing left to do
# biocLite("BiocUpgrade")
# This error means it's up to date: "Bioconductor version 3.4 cannot be upgraded with R version 3.3.2"
avail = available.packages(repos=biocinstallRepos()) # includes CRAN at the end from getOption("repos")
deps = tools::package_dependencies("data.table", db=avail, which="most", reverse=TRUE, recursive=FALSE)[[1]]
table(avail[deps,"Repository"])
length(deps)
old = 0
new = 0
if (basename(.libPaths()[1]) != "revdeplib") stop("Must start R with exports as above")
for (p in deps) {
fn = paste0(p, "_", avail[p,"Version"], ".tar.gz")
if (!file.exists(fn) ||
identical(tryCatch(packageVersion(p), error=function(e)FALSE), FALSE) ||
packageVersion(p) != avail[p,"Version"]) {
system(paste0("rm -f ", p, "*.tar.gz")) # Remove previous *.tar.gz. -f to be silent if not there (i.e. first time seeing this package)
system(paste0("rm -rf ", p, ".Rcheck")) # Remove last check (of previous version)
install.packages(p, repos=biocinstallRepos(), dependencies=TRUE) # again, bioc repos includes CRAN here
# To install its dependencies. The package itsef is installed superfluously here because the tar.gz will be passed to R CMD check.
# Not using biocLite() because it does not download dependencies and does not appear to pass dependencies= on.
download.packages(p, destdir="~/build/revdeplib", contriburl=avail[p,"Repository"]) # So R CMD check can run on these
if (!file.exists(fn)) stop("Still does not exist!:", fn)
new = new+1
} else {
old = old+1
}
}
cat("New downloaded:",new," Already had latest:", old, " TOTAL:", length(deps), "\n")
table(avail[deps,"Repository"])
# To identify and remove the tar.gz no longer needed :
for (p in deps) {
f = paste0(p, "_", avail[p,"Version"], ".tar.gz")
system(paste0("mv ",f," ",f,"_TEMP"))
}
system("ls *.tar.gz")
system("rm *.tar.gz")
for (p in deps) {
f = paste0(p, "_", avail[p,"Version"], ".tar.gz")
system(paste0("mv ",f,"_TEMP ",f))
}
system("ls *.tar.gz | wc -l")
status = function(which="both") {
if (which=="both") {
cat("CRAN:\n"); status("cran")
cat("BIOC:\n"); status("bioc")
cat("Oldest 00check.log (to check no old stale ones somehow missed):\n")
system("find . -name '00check.log' | xargs ls -lt | tail -1")
cat("\n")
if (file.exists("/tmp/started.flag")) {
system("ls -lrt /tmp/*.flag")
tt = as.POSIXct(file.info(c("/tmp/started.flag","/tmp/finished.flag"))$ctime)
if (is.na(tt[2])) { tt[2] = Sys.time(); cat("Has been running for "); }
else cat("Ran for ");
cat(round(diff(as.numeric(tt))/60, 1), "mins\n")
}
return(invisible())
}
if (which=="cran") deps = deps[-grep("bioc",avail[deps,"Repository"])]
if (which=="bioc") deps = deps[grep("bioc",avail[deps,"Repository"])]
x = unlist(sapply(deps, function(x) {
fn = paste0("./",x,".Rcheck/00check.log")
if (file.exists(fn)) {
v = suppressWarnings(system(paste0("grep 'Status:' ",fn), intern=TRUE))
if (!length(v)) return("RUNNING")
return(substring(v,9))
}
if (file.exists(paste0("./",x,".Rcheck"))) return("RUNNING")
return("NOT STARTED")
}))
e = grep("ERROR",x)
w = setdiff( grep("WARNING",x), e)
n = setdiff( grep("NOTE",x), c(e,w) )
ok = setdiff( grep("OK",x), c(e,w,n) )
r = grep("RUNNING",x)
ns = grep("NOT STARTED", x)
cat(" ERROR :",sprintf("%3d",length(e)),":",paste(sort(names(x)[e])),"\n",
"WARNING :",sprintf("%3d",length(w)),":",paste(sort(names(x)[w])),"\n",
"NOTE :",sprintf("%3d",length(n)),"\n", #":",paste(sort(names(x)[n])),"\n",
"OK :",sprintf("%3d",length(ok)),"\n",
"TOTAL :",length(e)+length(w)+length(n)+length(ok),"/",length(deps),"\n",
"RUNNING :",sprintf("%3d",length(r)),":",paste(sort(names(x)[r])),"\n",
if (length(ns)==0) "\n" else paste0("NOT STARTED (first 20 of ",length(ns),") : ",paste(sort(names(x)[head(ns,20)]),collapse="|"),"\n")
)
invisible()
}
status()
run = function(all=FALSE) {
numtgz = as.integer(system("ls -1 *.tar.gz | wc -l", intern=TRUE))
stopifnot(numtgz==length(deps))
cat("Installed data.table to be tested against:",as.character(packageVersion("data.table")),"\n")
if (all) {
cmd = "rm -rf *.Rcheck ; ls -1 *.tar.gz | parallel R CMD check"
# apx 7.5 hrs for 582 packages on my 4 cpu laptop with 8 threads
} else {
x = sapply(deps, function(p) {
if (!file.exists(paste0("./",p,".Rcheck"))) return(TRUE)
fn = paste0("./",p,".Rcheck/00check.log")
length(suppressWarnings(system(paste0("grep -E 'Status:.*(ERROR|WARNING)' ",fn), intern=TRUE)))>0L
})
x = deps[x]
cat("Running",length(x),"packages that have no status (no .Rcheck directory) or are in ERROR or WARNING status\n")
cmd = paste0("ls -1 *.tar.gz | grep -E '", paste0(x,collapse="|"),"' | parallel R CMD check")
}
cat("Command:",cmd,"\n")
if (as.integer(system("ps -a | grep perfbar | wc -l", intern=TRUE)) < 1) system("perfbar",wait=FALSE)
cat("Proceed? (ctrl-c or enter)\n")
scan(quiet=TRUE)
system("touch /tmp/started.flag ; rm -f /tmp/finished.flag")
system(paste("((",cmd,">/dev/null 2>&1); touch /tmp/finished.flag)"), wait=FALSE)
}
# ** ensure latest version installed into revdeplib **
system("R CMD INSTALL ~/GitHub/data.table/data.table_1.11.1.tar.gz")
run()
# Investigate and fix the fails ...
find . -name 00check.log -exec grep -H -B 20 "Status:.*ERROR" {} \;
find . -name 00check.log | grep -E 'AFM|easycsv|...|optiSel|xgboost' | xargs grep -H . > /tmp/out.log
# For RxmSim: export JAVA_HOME=/usr/lib/jvm/java-8-oracle
more <failing_package>.Rcheck/00check.log
R CMD check <failing_package>.tar.gz
R CMD INSTALL ~/data.table_1.9.6.tar.gz # CRAN version to establish if fails are really due to data.table
R CMD check <failing_package>.tar.gz
ls -1 *.tar.gz | grep -E 'Chicago|dada2|flowWorkspace|LymphoSeq' | parallel R CMD check &
# Warning: replacing previous import robustbase::sigma by stats::sigma when loading VIM
# Reinstalling robustbase fixed this warning. Even though it was up to date, reinstalling made a difference.
###############################################
# Release to CRAN
###############################################
Bump versions in DESCRIPTION and NEWS (without 'on CRAN date' text as that's not yet known) to even release number
DO NOT push to GitHub. Prevents even a slim possibility of user getting premature version. Even release numbers must have been obtained from CRAN and only CRAN. (Too many support problems in past before this procedure brought in.)
R CMD build .
R CMD check --as-cran data.table_1.11.4.tar.gz # install.packages("xml2") first to check the 150 URLs in NEWS.md under --as-cran, then remove xml2 again
Resubmit to winbuilder (R-release, R-devel and R-oldrelease)
Submit to CRAN. Message template :
-----
475 CRAN + 111 BIOC rev deps checked ok.
9 CRAN packages will break : easycsv, fst, iml, PhenotypeSimulator, popEpi, sdcMicro, SIRItoGTFS, SpaDES.core, splitstackshape
The maintainers have been contacted; some may have updated already.
Thanks and best, Matt
-----
1. Bump version in DESCRIPTION to next odd number
2. Add new heading in NEWS for the next dev version. Add "(on CRAN date)" on the released heading if already accepted.
3. Bump 3 version numbers in Makefile
Push to GitHub so dev can continue. Commit message format "1.11.0 submitted to CRAN. Bump to 1.11.1"
Bump dev badge on homepage
Cross fingers accepted first time. If not, push changes to devel and backport locally
Close milestone
** If on EC2, shutdown instance. Otherwise get charged for potentially many days/weeks idle time with no alerts **