The migrator does not take advantage of the ScyllaDB driver

[julienrf]

We communicate with ScyllaDB via the the spark-cassandra-connector, which uses the Apache Cassandra driver under the hood (version 4.13 at the time of writing).

This prevents us from taking advantage of the specific ScyllaDB driver such as shard awareness.

We should consider swaping the Cassandra driver with the ScyllaDB driver. We would probably have to change the spark-cassandra-connector itself, though.

[tarzanek]

so we'd first need to merge to our fork ( https://github.com/scylladb/spark-cassandra-connector/ ) and release spark connectors built on shard aware driver - e.g. https://github.com/tarzanek/spark-cassandra-connector/tree/v3.0.0-scylla
Until 3.1 I checked and made sure we can have similar version of respective driver version in https://github.com/scylladb/java-driver/tags (above 3.0 is bad example, but we have the respective version now, back then it was missing) , R&D is also releasing this and afaik only spark 3.5.1 does an upgrade of driver (and even to version that should be part of released ones already)

that said that above simple patch for connector might not be everything, special extensions will need similar changes as scylladb/java-driver#156 to be usable from rdds or leveraged in rdds (BYPASS CACHE being the most important one)
for all above we'd need some QA, which I think is biggest blocker now
technically the first step should be doable, so if we'd release it now, it would be without support

[guy9]

Thanks @tarzanek , @julienrf , let's wait with this until I verify we have the resources from the QA team.

[julienrf]

The first step to move forward is to publish our fork of spark-cassandra-connector, so that the Migrator can use it directly instead of building it from source.

I suggest creating a branch scylla-4.x in our repository with the following changes:

• Rename organization from com.datastax to com.scylladb
• Rename artifact from spark-cassandra-connector to spark-scylladb-connector
• Document the motivation for the fork and the branching policy

I implemented such changes under my fork. Please @tarzanek review this branch and push it to the ScyllaDB fork if you approve it.

The next steps will be as follows (also tracked in scylladb/spark-scylladb-connector#6):

• Rename our repository fork from spark-cassandra-connector to spark-scylladb-connector
• Fast-forward merge the branch feature/track-token-ranges-3.5.0 into scylla-4.x and document the feature in the README
• Set up a release process in the ScyllaDB fork
• Release version 4.0.0
• Use it in the Migrator in place of our submodule

And then we will be able to gradually introduce more features to the connector.

[julienrf]

I am closing this issue since scylladb/spark-scylladb-connector#6 is (almost) done, and the next steps are tracked by the following issues:

• Run the test against ScyllaDB spark-scylladb-connector#13
• The ScyllaDB Java driver should be used instead of the usual Cassandra driver spark-scylladb-connector#14