Create a comparison of all known stores

[nicholascar]

Let's enhance the list of stores at https://rdflib.readthedocs.io/en/stable/plugin_stores.html with both contained stores and any community made stores, for example AWS Neptune (https://lawlesst.github.io/notebook/rdflib-neptune.html) and others.

Let's also add pros & cons of the various stores.

[ashleysommer]

Good timing with this. I've been doing some experimental background work, and we will have a new Store to add to the list soon 😉

[nicholascar]

Great! I think there are about 5 other stores int he RDFlib GitHub org that should be listed and I know of at least another 3m including Neptune.

I've actually made the docco but something in Sphinx is really broken, so I haven't PRed it yet.

[nicholascar]

I've made a start on better Store documentation in the 6.0.1 latest release documentation: https://rdflib.readthedocs.io/en/latest/plugin_stores.html

A few uglinesses there and incompletness for sure. If you'd like to improve that documentation, please add to this Issue or put in PRs to update the plugin store file directly.

[ghost]

The pertinent document “A Universal RDF Store Interface” seems to have dropped off the map. I retrieved a copy from the googlecode archive snapshot tarball and in the interests of completeness and coherence, I'm uploading it here.
univrdfstore.txt

[rchateauneu]

Thanks. How could it replace or complement https://github.com/RDFLib/rdflib-sqlalchemy ?

[ghost]

It wouldn't replace it as such because the Universal RDF Store Interface only addresses key-value persistences and not relational persistences (although there is a small degree of crossover with the PostGres hstore). The “Universal RDF Store Interface” is basically a specification, a generalisation of the was-SleepyCat-now-BerkeleyDB Store berkeleydb.py.

In practice, the main difference is that the BerkelyDB class assigns the five necessary k-v store databases (subjects, predicates, objects, contexts, namespaces) to class attributes and uses the standard BerkelyDB get/put calls, munging the bytes<->string processing for each get/put call, as in the contexts method (in which the call to bb is doing the bytes<->string munging) whereas the generalisation uses store-specific _get/_put definitions (which, IME do vary with the key-value store chosen) which take a db as a positional parameter, offering the potential of doing the bytes<->string munging at the get/put point of DBAPI contact, thus avoiding most of the cruft-per-call prolixity.

Other significant differences are:

1. BerkeleyDB is transaction-aware and many key-value stores (for example, leveldb) are not, so the BerkeleyDB class is obliged to have a txn keyword arg for get/put whereas (for example) the Plyvel leveldb wrapper does not
2. BerkeleyDB key iteration uses a cursor-based approach whereas (for example) the Plyvel wrapper for leveldb simply provides a choice of iterators over the key space.
3. BerkeleyDB has a sync() function, mediated by threading which some key-value stores don't have or, if they do, the API differs - using Plyvel's leveldb wrapper as an example, Plyvel's put has a sync keyword arg. (It's this kind of difference which rather necessitates the splitting out of the get/put calls into separate store-specific implementations.)

Well, that's the principle at least. I'm currently investigating what's involved in upgrading the rdflib-leveldb module which is based on this standard and, if successful, I should be able to apply the same changes to upgrade the rdflib_kyotocabinet similarly.

Back in the day, these key-value (aka “NoSQL”) stores were just starting to make their appearance (I recall investigating Riak as well as Tokyo/Kyoto Cabinet(https://dbdb.io/db/tokyo-cabinet). The Universal RDF Store Interface provides a development path towards recruiting a number of key-value stores (I note a useful review) as persistences for RDFLib graphs.

In summary: there are three different approaches to achieving persistence of RDFLib graphs:

1. relational-backed stores - now collectively mediated by RDFLib-SQLAlchemy
2. key/value-backed stores - mediated (hopefully) by implementations using the Universal RDF Store Interface
3. SPARQL-backed stores - mediated by SPARQLStore

(Yes, technically, it may be possible to implement a higher level generalisation of the Universal RDF Store Interface that provides a wrapper which enables simple k-v backend-store switching but that's quite ambitious and I shan't be holding my breath)

[ghost]

fwiw, the origins of the SLQAlchemy implementation lie not the Universal RDF Store Interface but in the AbstractSQLStore.py - included below because there is no current resolvable URI - but it's in the layercake section of the python-dlp archive on googlecode (it predated RDFLib's migration to Github).

Over a decade ago, there was a whole buncha stores implementing different persistences, many if not most of them unmaintained/unmaintainable and so they have been culled from the repos over the years:

• AbstractSQLStore.py
• BDBOptimized.py
• MySQL.py
• PostgreSQL.py
• Redland.py
• SQLite.py
• _sqlobject.py
• Sleepycat.py
• ZODB.py

For a while, I was able to preserve the PostgreSQL and SQLite stores as separate modules. Both implementations were subclassed from AbstractSQLStore and, given the commonalities in both implementations, it seemed obvious to have a go at implementing a SQLAlchemy subclass of AbstractSQLStore and that was the principled basis for my original (later improved beyond all recognition by subsequent contributors) implementation of the rdflib-sqlalchemy module.

AbstractSQLStore.py.txt

[ghost]

Weellll, TIL ZODB's been updated to run under Python3, so on a whim I revisited rdflib-zodb and after updating the code to Py3, happily the standard store tests are now all passing. There remains some work to be done in terms of bringing the module up to modern Python packaging and testing standards but that's no biggie.

[ghost]

I successfully managed to migrate the rdflib-leveldb Store to Python 3 - py3-migration branch. Devops and docs still WIP but the tests all pass and the store performance stats of reading up to 50K triples suggest a performance on a par with BerkeleyDB.

[ghost]

I successfully managed to migrate the rdflib-leveldb Store to Python 3 - py3-migration branch. Devops and docs still WIP

Happy to report that it's pretty much done. I stole and modified the rdflib validate.yaml GH action added by aucampia and the rdflib-leveldbstore is testing out okay across the marix.

[ghost]

Just for the record, there's something of a forgotten trove of store-related docs in the discontinued rdfextras package.