backport uuid identifier related changes to support uuid based sstable identifiers

[tchaikov]

uuid-based sstable identifier was introduced by Cassandra upstream in apache/cassandra@0040fea. but our fork was based 3.x. so to support this feature, we need to merge the upstream changes or rebase on top of it. or, as put by Avi in scylladb/scylladb#13932 (comment), quoted here as well

We should probably drop sstableloader. We might develop a standalone format converter from Cassandra formats to our format, based on the Cassandra codebase. So migration would use either the Spark migrator, or (Cassandra sstables -> Scylla sstables) -> load'n'stream. This gives us one less reason to keep having tools/java/.

Note that Cassandra is now at "oa" format: apache/cassandra@f16fb67.

without this change following test would be failing after enabling the uuid_sstable_identifier_enabled option introduced by scylladb/scylladb#13932

• Tests / Sanity Tests / alternator_tests.TesterAlternator.test_sstableloder_scenario[without_flag]: depends on sstableloder
• Tests / Sanity Tests / compaction_additional_test.TestCompactionAdditionalStrategy.test_compaction_removes_ttld_data_after_gc_period[LeveledCompactionStrategy]: depends on sstabledump
• Tests / Sanity Tests / snapshot_test.TestSnapshot.test_basic_snapshot_and_restore_with_sstableloader: depends on sstableloder
• Tests / Sanity Tests / snapshot_test.TestSnapshot.test_basic_mulitple_tables_snapshot_and_restore_with_sstableloader: depends on sstableloder
• Tests / Sanity Tests / sstabledump_test.TestSSTableDump.test_sstabledump_basic: depends on sstabledump
• Tests / Sanity Tests / sstabledump_test.TestSSTableDump.test_sstabledump_counter_basic: depends on sstabledump
• Tests / Sanity Tests / sstabledump_test.TestSSTableDumpAllDatatypes.test_sstabledump_all_datatypes: depends on sstabledump
• Tests / Sanity Tests / repair_additional_test.TestRepairAdditional.test_repair_ttl_update: depends on sstabledump

the tests above are only a subset. following tools should be updated

• sstabledump
• sstableloader
• sstableverify
• sstablemetadata

[tchaikov]

see also scylladb/scylladb#13932

[denesb]

I want to patch out sstabledump from all dtests. But I don't know when I will get around to do that, and certainly don't want to block the uuid migration on this.

[tchaikov]

@denesb hi Botond, thank you for the remarks! the recent changes of

• https://github.com/scylladb/scylla-dtest/pull/3220
• https://github.com/scylladb/scylla-dtest/pull/3232

can enable us to proceed without being blocked by this issue or your initiative to ditch sstabledump. and your change will allow revert some of these dtest changes.

[tchaikov]

@denesb hi Botond, i am going to add a wrapper around scylla dump-data to replace sstabledump, what do you think? i think once we have scylladb/scylladb#14726 . we will be able to drop sstabledump.

[juliayakovlev]

sstablemetadata also fails on uuid-based sstable identifier

[tchaikov]

sstablemetadata also fails on uuid-based sstable identifier

thank you. added to the list.

[tchaikov]

an alternative of backport uuid identifier changes is to implement it right in scylla. see scylladb/scylladb#14856

[fruch]

@tchaikov

this is breaking rolling upgrades tests as well
https://github.com/scylladb/scylla-cluster-tests/blob/6e3d34cd1b9fc6533a4e5a3874fd36ad401b9a81/upgrade_test.py#L698

and a test for GC of tomestones:
https://github.com/scylladb/scylla-cluster-tests/blob/6e3d34cd1b9fc6533a4e5a3874fd36ad401b9a81/longevity_tombstone_gc_test.py#L42

[mykaul]

@denesb hi Botond, i am going to add a wrapper around scylla dump-data to replace sstabledump, what do you think? i think once we have scylladb/scylladb#14726 . we will be able to drop sstabledump.

@tchaikov - is this still the plan?

[tchaikov]

@mykaul no, in an offline discussion with Botond, he warned me that the output format of scylla dump-data was different from that of sstabledump. so unless we translate the former into the latter in the wrapper i imagined, it's a no-go. so we are actively replacing sstabledump in our tests. and mark it deprecated, and then plan to remove it in favor of scylla sstable after a grace period.

[mykaul]

@mykaul no, in an offline discussion with Botond, he warned me that the output format of scylla dump-data was different from that of sstabledump. so unless we translate the former into the latter in the wrapper i imagined, it's a no-go. so we are actively replacing sstabledump in our tests. and mark it deprecated, and then plan to remove it in favor of scylla sstable after a grace period.

I don't mind deprecating it, but it has docs implications. https://opensource.docs.scylladb.com/stable/operating-scylla/admin-tools/sstabledump.html for example.

[tchaikov]

https://opensource.docs.scylladb.com/stable/operating-scylla/admin-tools/sstabledump.html

@mykaul hi Yaniv, please take a look at the master version https://opensource.docs.scylladb.com/master/operating-scylla/admin-tools/sstabledump.html . we are deprecating it.