Join with constant memory footprint

[root-11]

@realratchet @cerv15

Join has been optimized for speed using numpy and to maintain a constant memory footprint equal to 4 pages (4M values).

Occasionally (1 in 100) I can trigger the following error during multiprocessing:

/home/bjorn/github/tablite/tests/test_join.py::test_left_join_mp failed: def test_left_join_mp():
        Config.MULTIPROCESSING_MODE = Config.FORCE
>       do_left_join()

tests/test_join.py:52: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_join.py:29: in do_left_join
    left_join.show()
tablite/base.py:1748: in show
    print(self.to_ascii(slice_=slice_, blanks=blanks, dtype=dtype))
tablite/base.py:1712: in to_ascii
    for name, values in self.display_dict(slice_=slice_, blanks=blanks, dtype=dtype).items():
tablite/base.py:1663: in display_dict
    data[name] = list(chain(iter(col), repeat(blanks, times=n - len(col))))[slc]
tablite/base.py:694: in __iter__
    data = page.get()
tablite/base.py:147: in get
    array = load_numpy(self.path)
tablite/utils.py:472: in load_numpy
    return np.load(path, allow_pickle=True, fix_imports=False)
../../venv/py310tablite/lib/python3.10/site-packages/numpy/lib/npyio.py:432: in load
    return format.read_array(fid, allow_pickle=allow_pickle,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

fp = <_io.BufferedReader name='/tmp/tablite-tmp/pid-30773/pages/3.npy'>
allow_pickle = True, pickle_kwargs = {'encoding': 'ASCII', 'fix_imports': False}

    def read_array(fp, allow_pickle=False, pickle_kwargs=None, *,
                   max_header_size=_MAX_HEADER_SIZE):
        """
        Read an array from an NPY file.
    
        ...
    
        """
        if allow_pickle:
            # Effectively ignore max_header_size, since `allow_pickle` indicates
            # that the input is fully trusted.
            max_header_size = 2**64
    
        version = read_magic(fp)
        _check_version(version)
        shape, fortran_order, dtype = _read_array_header(
                fp, version, max_header_size=max_header_size)
        if len(shape) == 0:
            count = 1
        else:
            count = numpy.multiply.reduce(shape, dtype=numpy.int64)
    
        # Now read the actual data.
        if dtype.hasobject:
            # The array contained Python objects. We need to unpickle the data.
            if not allow_pickle:
                raise ValueError("Object arrays cannot be loaded when "
                                 "allow_pickle=False")
            if pickle_kwargs is None:
                pickle_kwargs = {}
            try:
>               array = pickle.load(fp, **pickle_kwargs)
E               _pickle.UnpicklingError: unpickling stack underflow

../../venv/py310tablite/lib/python3.10/site-packages/numpy/lib/format.py:792: UnpicklingError

I believe this is not a join bug, but rather a multiprocessing problem. @realratchet ?

[realratchet]

I'll close the and re-open it from my fork, seems that my git decided to push to the master repo instead of fork for whatever reason.