Drastic bottleneck: array joining

[florian-huber]

The current implementation for merging arrays is FAR TOO SLOW.
Slicing and other operations seem fine, but adding data to a StackedSparseArray is painfully slow (even though the implementation was done with numba, but probably not the best way...)

[florian-huber]

I did a lot of profiling on this (see #11 for more details), including Numpy (multiple variants), Pandas, Polars, Numba (multiple variants). A new Numba try seems to improve this quite a bit --> #12

With a simple profiling script I got the following.

Current implementation:

Addition of first layer
 ----------------------------------------
Computation took 29.094916343688965 s.

Addition of new layer
 ----------------------------------------
Computation took 42.284873962402344 s.

Slicing 1
 ----------------------------------------
Computation took 0.010562658309936523 s.

Slicing 2
 ----------------------------------------
Computation took 0.01617717742919922 s.

Slicing 3
 ----------------------------------------
Computation took 0.01498723030090332 s.

New variant (#12):

Addition of first layer
 ----------------------------------------
Computation took 0.007983207702636719 s.

Addition of new layer
 ----------------------------------------
Computation took 0.7480928897857666 s.

Slicing 1
 ----------------------------------------
Computation took 0.008943796157836914 s.

Slicing 2
 ----------------------------------------
Computation took 0.013475418090820312 s.

Slicing 3
 ----------------------------------------
Computation took 0.014035463333129883 s.

[florian-huber]

I consider this fixed with #12