Merge Next major #7826
Draft
Merge Next major #7826
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* code review --------- Co-authored-by: Finn Schiermer Andersen <[email protected]> Co-authored-by: Jørgen Edelbo <[email protected]>
* update next-major * specified part of new layout (new width encoding) * new header format for compressed arrays * code review * code review * start of classifying arrays for compression * classification down to column types * first attempt to cut through the BPlusTree madness * [wip] start on 'type driven' write process * all tests passing (but no compression enabled) * enabled compression for signed integer leafs only * removed some dubious constructions in cluster tree * delete tmp array while classifying arrays * enabled compression of links and backlinks (excl collections) * also compress bplustree of integers/links (experimental) * pref for compressing dicts (not working) * wip * wip * finally: compressing collections (incl dicts) * compressing timestamps now * enabled compression on ObjectID, TypedLink and UUID * also compressing Mixed properties (not list/dicts of Mixed) * Array compression with collections in Mixed (#7412) --------- Co-authored-by: Finn Schiermer Andersen <[email protected]> * merge next-major + collection in mixed * enable dynamic choice of compression method * moved typed_write/typed_print for bptree into class * Merge pull request #7432 from realm/fsa/clean_typed_write moved typed_write/typed_print for bptree into class * cleanup unrelated code changes * fix compilation * cleanup * code review * code review * swap byte 3&4 with byte 6&7 for flex formats for storing A and B sizes * Some modifications * Move Encoding definition * Testing * Perserve type information in typed_write (#7598) * call directly Array::destroy() * Fix issue * remove table from typed_print * lint * point fix avoid compressing history array --------- Co-authored-by: Finn Schiermer Andersen <[email protected]> Co-authored-by: Jørgen Edelbo <[email protected]> Co-authored-by: Finn Schiermer Andersen <[email protected]>
…es for compressed arrays (#7571) * update next-major * specified part of new layout (new width encoding) * new header format for compressed arrays * code review * code review * start of classifying arrays for compression * classification down to column types * first attempt to cut through the BPlusTree madness * [wip] start on 'type driven' write process * all tests passing (but no compression enabled) * enabled compression for signed integer leafs only * removed some dubious constructions in cluster tree * delete tmp array while classifying arrays * enabled compression of links and backlinks (excl collections) * also compress bplustree of integers/links (experimental) * pref for compressing dicts (not working) * wip * wip * finally: compressing collections (incl dicts) * compressing timestamps now * enabled compression on ObjectID, TypedLink and UUID * also compressing Mixed properties (not list/dicts of Mixed) * Array compression with collections in Mixed (#7412) --------- Co-authored-by: Finn Schiermer Andersen <[email protected]> * merge next-major + collection in mixed * enable dynamic choice of compression method * moved typed_write/typed_print for bptree into class * Merge pull request #7432 from realm/fsa/clean_typed_write moved typed_write/typed_print for bptree into class * cleanup unrelated code changes * fix compilation * cleanup * code review * code review * scaffolding for accessing unaligned memory inside compressed arrays * introducing parallel scan for scanning in one go multiple compressed arrays * Silence warning * tests + point fix for upper bound * fix ubsan * remove dead code in subword search * remove tests of 'unaligned iterator' for flexibility * Some fixes to conform with coding polycies * Enhance test * bugfix for serch with width=31 bits * bugfix for search with width=31 bits * bugfix for search with width=31 bits --------- Co-authored-by: Finn Schiermer Andersen <[email protected]> Co-authored-by: Jørgen Edelbo <[email protected]> Co-authored-by: Finn Schiermer Andersen <[email protected]>
…ompression (#7664) * update next-major * specified part of new layout (new width encoding) * new header format for compressed arrays * code review * code review * start of classifying arrays for compression * classification down to column types * first attempt to cut through the BPlusTree madness * [wip] start on 'type driven' write process * all tests passing (but no compression enabled) * enabled compression for signed integer leafs only * removed some dubious constructions in cluster tree * delete tmp array while classifying arrays * enabled compression of links and backlinks (excl collections) * also compress bplustree of integers/links (experimental) * pref for compressing dicts (not working) * wip * wip * finally: compressing collections (incl dicts) * compressing timestamps now * enabled compression on ObjectID, TypedLink and UUID * also compressing Mixed properties (not list/dicts of Mixed) * Array compression with collections in Mixed (#7412) --------- Co-authored-by: Finn Schiermer Andersen <[email protected]> * merge next-major + collection in mixed * enable dynamic choice of compression method * moved typed_write/typed_print for bptree into class * Merge pull request #7432 from realm/fsa/clean_typed_write moved typed_write/typed_print for bptree into class * cleanup unrelated code changes * fix compilation * cleanup * code review * code review * scaffolding for accessing unaligned memory inside compressed arrays * introducing parallel scan for scanning in one go multiple compressed arrays * Silence warning * tests + point fix for upper bound * fix ubsan * added clickbench data ingestor * Added 2 queries to clickbench * separated out a clickquery benchmark * separated out a clickquery benchmark * update to clickbench and query * fix warnings * attempt to fix windows builders * silence warnings * fix importer --------- Co-authored-by: Finn Schiermer Andersen <[email protected]> Co-authored-by: Jørgen Edelbo <[email protected]> Co-authored-by: Finn Schiermer Andersen <[email protected]>
…ectly (#7667) * update next-major * specified part of new layout (new width encoding) * new header format for compressed arrays * code review * code review * start of classifying arrays for compression * classification down to column types * first attempt to cut through the BPlusTree madness * [wip] start on 'type driven' write process * all tests passing (but no compression enabled) * enabled compression for signed integer leafs only * removed some dubious constructions in cluster tree * delete tmp array while classifying arrays * enabled compression of links and backlinks (excl collections) * also compress bplustree of integers/links (experimental) * pref for compressing dicts (not working) * wip * wip * finally: compressing collections (incl dicts) * compressing timestamps now * enabled compression on ObjectID, TypedLink and UUID * also compressing Mixed properties (not list/dicts of Mixed) * Array compression with collections in Mixed (#7412) --------- Co-authored-by: Finn Schiermer Andersen <[email protected]> * merge next-major + collection in mixed * enable dynamic choice of compression method * moved typed_write/typed_print for bptree into class * Merge pull request #7432 from realm/fsa/clean_typed_write moved typed_write/typed_print for bptree into class * cleanup unrelated code changes * fix compilation * cleanup * code review * code review * scaffolding for accessing unaligned memory inside compressed arrays * introducing parallel scan for scanning in one go multiple compressed arrays * Silence warning * tests + point fix for upper bound * fix ubsan * added clickbench data ingestor * Added 2 queries to clickbench * separated out a clickquery benchmark * separated out a clickquery benchmark * update to clickbench and query * fix warnings * add new builder with compression always enabled for verifying that compression works correctly (#7524) * attempt to fix windows builders * silence warning windows * silence warnings --------- Co-authored-by: Finn Schiermer Andersen <[email protected]> Co-authored-by: Jørgen Edelbo <[email protected]> Co-authored-by: Finn Schiermer Andersen <[email protected]>
* Add integer compressor for arrays that have type WTypBits * Add packed handler * Add flex handler * Embed integer compression logic inside commit machinery * Tests * Optimization for integer array decompression and Array::find for compressed formats * code review --------- Co-authored-by: Finn Schiermer Andersen <[email protected]> Co-authored-by: Jørgen Edelbo <[email protected]>
feature release
Pull Request Test Coverage Report for Build jorgen.edelbo_322Details
💛 - Coveralls |
Pull Request Test Coverage Report for Build jorgen.edelbo_333Details
💛 - Coveralls |
Pull Request Test Coverage Report for Build jorgen.edelbo_337Details
💛 - Coveralls |
Pull Request Test Coverage Report for Build jorgen.edelbo_391Details
💛 - Coveralls |
Create test file in file-format 24
This will allow some initial experiments in the SDKs. This is not a fileformat breaking change as we just use already existing features.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What, How & Why?
☑️ ToDos
bindgen/spec.yml, if public C++ API changed