We're looking for volunteers on developing ArcadeDB's Python Driver

[lvca]

COMMUNITY HELP

Hi guys, we're working on the new Vector Model in ArcadeDB, and one fundamental piece of the puzzle is having a Python driver. This is because most of the AI/ML projects/frameworks out there are in Python. You can use ArcadeDB with HTTP/JSON from Python right now, but a simple driver that wraps the HTTP/JSON calls would be fantastic to have. Is anybody fluent in Python that can help? Please write here or DM me. Thanks!

[adaros92]

I can help with this as we've started to do something similar against our Arcade server where I work.

[lvca]

@adaros92 that would be awesome! Please let us know any news about this.

[danielcamara]

Hi I am evaluating some DB for a new project, and ArcadeDB seems quite promising, in my point of view. However, the missing python lib is for sure a drawback. We had considered doing our own wrappers using the REST interface, and packaging it in a way that we "forget" the calls, and that is it. We have done like that in the past with other things, like Elasticseach before the pip packaged lib was available. One thing we could do is to share that, after it is ready (but the project didn't even started and we are not sure we will infact use arcadedb yet...)

May I ask if there were some advances on this, and if yes, how are you thinking to do it? A wrapper over the REST, using something like JPype to access directly java's bytecode?

[danielcamara]

I agree that a "native" would be the best way to go, directly from python to the java implementation, this is why I asked about the JPype. Regarding your question, it seems that there is already a pyton-based lib to connect jdbc (https://pypi.org/project/JayDeBeApi/), if this kind of API is available for ArcadeDB could be the simples way (I am really sorry, long time without programming in Java so... now I have just a vage memory about jdbc :(, and none of the implications. )

OK the next request will seem stupid, and I apologise for that, but it is life not everyone can be smart :)! Could you point me to a simple and thorough example of creating a DB, inserting data and searching for it in Java? In the documentation, I found this entry https://docs.arcadedb.com/#Embedded-Server, but it seems a little confusing! The direct use of arcadebd from java on doc seems kind of over-complexified, if it is the way, so it is the way. However if I have a pointer to a simple "how to" to use arcadedb directly in Java, if I manage to have the time during the weekend I can try to "translate" a simple yet complete example, to see how hard/simple different ways to do it would be.... and who knows, maybe the solution is simpler than we think ;) Again, sorry for asking like that, but I just arrived and I don't really know where to look for :(!

[lvca]

I forgot, but we implement most of the Postgres driver specs. So you can use ArcaedDB with the Postgres Python driver.

[danielcamara]

Great but then wouldn't I be constrained by SQL as the main language? Would I be able to use Gremlin or Cypher? If not, than... I still think could worth the try :)

[robfrank]

take a look at the JDBC e2e test: https://github.com/ArcadeData/arcadedb/blob/23.10.1/e2e/src/test/java/com/arcadedb/e2e/JdbcQueriesTest.java

Arcade SQL
ResultSet rs = st.executeQuery("SELECT * FROM Beer limit 1")

gremlin:
try (final ResultSet rs = st.executeQuery("{gremlin}g.V().limit(1)")) {

chyper
ResultSet rs = st.executeQuery("{cypher}MATCH(p:Beer) RETURN * LIMIT 1")

[ExtReMLapin]

@lvca

A native bytecode would be also nice only if you intend to use ArcadeDB in embedded mode that is the fastest way to crunch data because you avoid tcp/ip, marshalling/unmarshalling overhead, typical with any client/server approach.

Could you quantify this performance increase, I talked with Daniel at the office and I'm wondering if the overhead you're talking about is just micro-optimization tier overhead or if it's really worth it, have you experienced any case where the REST interface wasn't fast enough ?

Did you quantify the speed increase ?

[robfrank]

Just a silly question.
I guess that Python has a PostgreSQL driver, as Java has. Can the Python community use that driver to connect to Arcade on the Postgres port?

[ExtReMLapin]

While the usefulness has not been exactly proved, I started toying around with it using jpype.

Beside this error, just starting it seems to work :

I tested creating a database and running a "create" and "match" cypher query, it works

monitor.ServerMonitor (in unnamed module @0x60611244) cannot access class sun.management.ManagementFactoryHelper (in module java.management) because module java.management does not export sun.management to unnamed module @0x60611244
        at com.arcadedb.server.monitor.ServerMonitor.checkJVMHotSpot(ServerMonitor.java:107)
        at com.arcadedb.server.monitor.ServerMonitor.monitor(ServerMonitor.java:61)
        at com.arcadedb.server.monitor.ServerMonitor.lambda$start$0(ServerMonitor.java:52)
        at java.base/java.lang.Thread.run(Thread.java:1583)

The python code is the following :

import jpype
import jpype.imports
from jpype.types import *

# Launch the JVM
jpype.startJVM(classpath=['G:/arcadedb-23.10.1/lib/*'])


import com.arcadedb 


config =  com.arcadedb.ContextConfiguration()
server =  com.arcadedb.server.ArcadeDBServer(config)

server.start()

Now the issue is that you would expect that replacing server.start()
by this threaded server start version to be able to run commands on the fly, like creating objects :

import threading
def threadedFunc (server):
    server.start()

thread = threading.Thread(target = threadedFunc, args = (server, ))
import traceback
thread.start()



while True:
    pass
    #keep the whole thing running

But surprisingly it doesn't work that well, the server starts but running any cypher command fails with the following error

<ArcadeDB_0> Error on initializing Cypher query engine
java.lang.NullPointerException: Cannot invoke "java.lang.ClassLoader.loadClass(String)" because "ctxtLoader" is null
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.getParentLoader(GroovyScriptEngineImpl.java:452)
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.access$000(GroovyScriptEngineImpl.java:89)
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl$1.run(GroovyScriptEngineImpl.java:119)
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl$1.run(GroovyScriptEngineImpl.java:116)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.createClassLoader(GroovyScriptEngineImpl.java:116)
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.<init>(GroovyScriptEngineImpl.java:111)
        at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.<init>(GremlinGroovyScriptEngine.java:215)
        at com.arcadedb.gremlin.ArcadeGraph.init(ArcadeGraph.java:468)
        at com.arcadedb.gremlin.ArcadeGraph.<init>(ArcadeGraph.java:120)
        at com.arcadedb.gremlin.ArcadeGraph.open(ArcadeGraph.java:143)
        at com.arcadedb.cypher.query.CypherQueryEngineFactory.getInstance(CypherQueryEngineFactory.java:44)
        at com.arcadedb.query.QueryEngineManager.getInstance(QueryEngineManager.java:66)
        at com.arcadedb.database.EmbeddedDatabase.getQueryEngine(EmbeddedDatabase.java:1298)
        at com.arcadedb.database.EmbeddedDatabase.command(EmbeddedDatabase.java:1334)
        at com.arcadedb.server.ServerDatabase.command(ServerDatabase.java:467)
        at com.arcadedb.server.http.handler.PostCommandHandler.executeCommand(PostCommandHandler.java:129)
        at com.arcadedb.server.http.handler.PostCommandHandler.execute(PostCommandHandler.java:96)
        at com.arcadedb.server.http.handler.DatabaseAbstractHandler.execute(DatabaseAbstractHandler.java:99)
        at com.arcadedb.server.http.handler.AbstractServerHttpHandler.handleRequest(AbstractServerHttpHandler.java:126)
        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393)
        at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:859)
        at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
        at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
        at java.base/java.lang.Thread.run(Thread.java:1583)
<ArcadeDB_0> Error on command execution (PostCommandHandler)
com.arcadedb.exception.CommandParsingException: Error on initializing Cypher query engine
        at com.arcadedb.cypher.query.CypherQueryEngineFactory.getInstance(CypherQueryEngineFactory.java:50)
        at com.arcadedb.query.QueryEngineManager.getInstance(QueryEngineManager.java:66)
        at com.arcadedb.database.EmbeddedDatabase.getQueryEngine(EmbeddedDatabase.java:1298)
        at com.arcadedb.database.EmbeddedDatabase.command(EmbeddedDatabase.java:1334)
        at com.arcadedb.server.ServerDatabase.command(ServerDatabase.java:467)
        at com.arcadedb.server.http.handler.PostCommandHandler.executeCommand(PostCommandHandler.java:129)
        at com.arcadedb.server.http.handler.PostCommandHandler.execute(PostCommandHandler.java:96)
        at com.arcadedb.server.http.handler.DatabaseAbstractHandler.execute(DatabaseAbstractHandler.java:99)
        at com.arcadedb.server.http.handler.AbstractServerHttpHandler.handleRequest(AbstractServerHttpHandler.java:126)
        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393)
        at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:859)
        at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
        at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.NullPointerException: Cannot invoke "java.lang.ClassLoader.loadClass(String)" because "ctxtLoader" is null
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.getParentLoader(GroovyScriptEngineImpl.java:452)
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.access$000(GroovyScriptEngineImpl.java:89)
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl$1.run(GroovyScriptEngineImpl.java:119)
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl$1.run(GroovyScriptEngineImpl.java:116)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.createClassLoader(GroovyScriptEngineImpl.java:116)
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.<init>(GroovyScriptEngineImpl.java:111)
        at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.<init>(GremlinGroovyScriptEngine.java:215)
        at com.arcadedb.gremlin.ArcadeGraph.init(ArcadeGraph.java:468)
        at com.arcadedb.gremlin.ArcadeGraph.<init>(ArcadeGraph.java:120)
        at com.arcadedb.gremlin.ArcadeGraph.open(ArcadeGraph.java:143)
        at com.arcadedb.cypher.query.CypherQueryEngineFactory.getInstance(CypherQueryEngineFactory.java:44)
        ... 15 more

[lvca]

@ExtReMLapin let us know your findings. Since >95% of ML/AI libs and frameworks are in Python, having ArcadeDB embedded means you can use vector embeddings and possibly outperform other DBMS just because you avoid TCP/IP. How much embedded can be faster than a remote approach? It depends on the use case, but it could be easily 3-10X even if the DBMS is located on the same server. If the DBMS server is located on a different server, then it could be 10X or much more.

[ExtReMLapin]

Alright, for now as my coworkers needed a ready to use API, I crafted a rest one from this work, with few changes .

As for the "embedded" approach, this morning I found something called stubgenj that generates Python documentation from jar files.

Which in other words, allows this :

Which will allows much better support for auto-completion, I didn't spend more time working on the threads problems, will keep everyone updated here as soon as i have time to work on it

[lvca]

That's cool, I didn't know about stubgenj. We should create a Python chapter about Python and this tool. @gramian WDYT?

[gramian]

I agree.

[ExtReMLapin]

If there is no need to rush this, give me some time to sort out this thread error thing and I'll give you a proper PoC python file with guide to generate documentation for python

[ExtReMLapin]

Posted my rework of the rest api for python : https://github.com/ExtReMLapin/pyarcade