make sqlite v3.20.0

[zirill]

make

Generate llvm bitcode

/home/kirill/emsdk/emscripten/1.37.13/emcc -DSQLITE_OMIT_LOAD_EXTENSION -DSQLITE_DISABLE_LFS -DLONGDOUBLE_TYPE=double -DSQLITE_INT64_TYPE="long long int" -DSQLITE_THREADSAFE=0 -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS c/sqlite3.c -o c/sqlite3.bc
c/sqlite3.c:385:5: error: token is not a valid binary operator in a preprocessor subexpression
#if SQLITE_INT64_TYPE
^~~~~~~~~~~~~~~~~
:8:32: note: expanded from here
#define SQLITE_INT64_TYPE long long int
~~~~ ^
1 error generated.
ERROR:root:compiler frontend failed to generate LLVM bitcode, halting
Makefile:42: ошибка выполнения рецепта для цели «c/sqlite3.bc»
make: *** [c/sqlite3.bc] Ошибка 1

[dinedal]

@kripken @lovasoa I'm able to reproduce the issue. Is this a emscripten bug?

[kripken]

Seems unlikely it's an emscripten bug, but in theory, maybe clang fixed a bug in the preprocessor which sqlite depended on? If so, you can bisect emscripten versions if you find a good one and a bad one, to see where the problem started.

[dinedal]

Only reason I mention it is because this occurs only in 3.2.0 of SQLite, and it appears they started using the preprocessor differently in this version compared to 3.1.9

If you look at sqlite.c, you see that on 3.2.0 a new compiler preprocessor directive that's causing the issue:

#if SQLITE_INT64_TYPE
  "INT64_TYPE",
#endif

This doesn't exist in 3.1.9.

Per the comment in 3.2.0@line 69 :

** BEGIN CODE GENERATED BY tool/mkctime.tcl 

I believe new code was added into the amalgamation that previous builds haven't seen before.

The only way the preprocessor was used before was when the flag was not treated as a bool but as a literal.

[dinedal]

Here's how the same SQLITE_INT64_TYPE flag was used before:

#ifdef SQLITE_INT64_TYPE
  typedef SQLITE_INT64_TYPE sqlite_int64;

Notice now how it fails if it's used as #if instead of #ifdef ?

SQLITE_INT64_TYPE is the only type flag modified in the Makefile that is also in the new tool/mkctime.tcl codegen

[kripken]

Hmm, if there's something wrong in sqlite 2.0 in those macros, I would expect it to also cause problems in a native build with clang. Might be worth checking that (but I would expect a stable release of sqlite to work properly, so maybe unlikely to fail).

[dinedal]

@kripken

$ clang sqlite3.c -fsyntax-only
$ clang sqlite3.c -S -O3 -o -
# ... clean outout no errors

This was on sqlite 3.2 on the amalgamation file, MD5 6fcf441de1e06b11dbdf231a113637bd

[kripken]

Very strange.

I would try to reduce this to a small standalone testcase. Then either file that on emscripten, or maybe i the process the underlying bug will reveal itself.

[zirill]

What has changed or is everything the same?
Is there a solution how to make a new version?

[renesugar]

Compiling sql.js with sqlite 3.20.1:

http://sqlite.org/2017/sqlite-amalgamation-3200100.zip

1. Changing "#if" to "#ifdef" fixes this problem:

c/sqlite3.c:385:5: error: token is not a valid binary operator in a preprocessor subexpression
#if SQLITE_INT64_TYPE
    ^~~~~~~~~~~~~~~~~
<command line>:8:32: note: expanded from here
#define SQLITE_INT64_TYPE long long int
                          ~~~~ ^
1 error generated.
ERROR:root:compiler frontend failed to generate LLVM bitcode, halting
make: *** [c/sqlite3.bc] Error 1

2. All tests pass except one (unrelated to this version of sqlite):

Testing ./test_worker.js...	writeStringToMemory is deprecated and should not be called! Use stringToUTF8() instead!
assert.js:41
  throw new errors.AssertionError({
  ^

AssertionError [ERR_ASSERTION]: Reading BLOB
    at Worker.worker.onmessage (/Users/rene/projects/sql.js/test/test_worker.js:23:14)
    at ChildProcess.<anonymous> (/Users/rene/projects/sql.js/node_modules/workerjs/index.js:25:38)
    at emitTwo (events.js:125:13)
    at ChildProcess.emit (events.js:213:7)
    at emit (internal/child_process.js:774:12)
    at _combinedTickCallback (internal/process/next_tick.js:141:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
Fail!\e[0m
Warning\e[0m : 12 tests passed out of 13

3. Makefile changes to use ES6 generated by CoffeeScript 2.0.1:

-       $(EMCC) $(EMFLAGS) -s EXPORTED_FUNCTIONS=@exported_functions c/extension-functions.bc c/sqlite3.bc --post-js js/api.js -o $@ ;\
+       EMCC_CLOSURE_ARGS="--language_in ECMASCRIPT6 --language_out ES5" $(EMCC) $(EMFLAGS) -s EXPORTED_FUNCTIONS=@exported_functions c/extension-functions.bc c/sqlite3.bc --post-js js/api.js -o $@ ;\

4. Versions used to compile sql.js:

coffee --version
CoffeeScript version 2.0.1

emcc --version
emcc (Emscripten gcc/clang-like replacement) 1.37.22 ()

[zirill]

problems again)

coffee --version
CoffeeScript version 2.2.4

emcc --version
emcc (Emscripten gcc/clang-like replacement) 1.37.37 ()

sqlite
Source Code version 3.23.1

error:
WARNING:root:not all asm.js optimizations are possible with ALLOW_MEMORY_GROWTH, disabling those. [-Walmost-asm]

Uncaught TypeError: Cannot read property 'Database' of undefined

[zirill]

clear install for linux

make

/home/user/prog/ems/emsdk/emscripten/1.37.37/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:272
throw new JS_Parse_Error(message, line, col, pos);
^

[zirill]

CoffeeScript version 1.9.3
emcc (Emscripten gcc/clang-like replacement) 1.38.0 (commit a4474e59db658cea570c78254fa71119cf688db5)

Uncaught abort("Assertion failed: Cannot call unknown function RegisterExtensionFunctions, make sure it is exported"). Build with -s ASSERTIONS=1 for more info.

[Taytay]

I've filed a bug with SQLite for the #if vs #ifdef issue above in hopes that they can fix it upstream.

[Taytay]

@kripken : Can you elaborate on what the original intent of setting SQLITE_INT64_TYPE to long long int was?

[Taytay]

Ah, nevermind. This bug was fixed in SQL.js with this commit, that removed the -DSQLITE_INT64_TYPE:
kripken/sql.js@5467112#diff-b67911656ef5d18c4ae36cb6741b7965

[Taytay]

This issue is fixed by #255