pyarrow_plasma_kv_store

• Pyarrow's plasma KV store offers the following functionality:

  • Shared memory based storage
  • Python dictionary like interface
  • Multi-process read and write safe

• Methods

  • setitem
  • getitem
  • delitem
  • contains
  • iter
  • len
  • keys
  • values
  • replace
  • delete
  • get_multi
  • set_multi
  • get_available_memory
  • get_used_memory
  • get_store_capacity
  • connect
  • disconnect
  • cleanup

• Open Python terminal from the conda environment

# Activate conda env
conda activate pykvstores

# Open Python terminal
python

Examples

# Import the PyArrowPlasmaKVStore class
>>> from pykvstores.pyarrow_plasma_kvstore import PyArrowPlasmaKVStore
# Create a PyArrowPlasmaKVStore object
>>> store = PyArrowPlasmaKVStore('/tmp/plasma')
# Connect to the plasma object store
>>> store.connect()
# Set an item
>>> store['key'] = 'value'
# Get an item
>>> store['key']
'value'
# Set multiple items
>>> store.set_multi(['key1', 'key2'], ['value1', 'value2'])
# Get multiple items
>>> store.get_multi(['key1', 'key2'])
['value1', 'value2']
# Get the available free memory in human readable format
>>> store.available_memory()
# Get the used memory in human readable format
>>> store.get_used_memory()
# Get the store size in human readable format
>>> store.get_store_size()
# Delete multiple items
>>> store.delete(['key', 'key1', 'key2'])
# Cleanup the plasma object store and the socket file
>>> store.cleanup()
# For more details
>>> help(PyArrowPlasmaKVStore)

• Multiprocessing safe (write and read example)

from multiprocessing import Process
from pykvstores.pyarrow_plasma_kvstore import PyArrowPlasmaKVStore

#### Write example ####
# Create the plasma store
store = PyArrowPlasmaKVStore("/tmp/plasma")
processes = []

def square(num_list, store):
    """ Square the numbers in the list and store the result in the store"""
    # Connect to the plasma object store (Every process needs to connect to the store)
    store.connect()

    # Square the numbers
    for num in num_list:
        store[num] = num * num

# Let's square the numbers in the list and write the result to the store using multiple processes (in this example, we use 4 processes)
for i in range(0, 400, 100):
    p = Process(target=square, args=(range(i, i + 100), store))
    processes.append(p)
    p.start()

for proc in processes:
    proc.join()

#### Read example ####
# We can read the data from the store in a multi-process safe way
processes = []

def read_square(num_list, store):
    """ Read the square of the numbers in the list from the store"""
    # Connect to the plasma object store (Every process needs to connect to the store)
    store.connect()

    # Read the numbers from the store
    for num in num_list:
        assert store[num] == num * num

# Let's read the squares from the store using multiple processes (in this example, we use 4 processes)
for i in range(0, 400, 100):
    p = Process(target=read_square, args=(range(i, i + 100), store))
    processes.append(p)
    p.start()

for proc in processes:
    proc.join()

# Disconnect the store and cleanup
store.disconnect()
store.cleanup()