sstabledump failed because AzureSnitch ClassNotFoundException #338

fgelcer · 2023-08-09T14:03:03Z

Issue description

This issue is a regression.
It is unknown if this issue is a regression.

Describe your issue in detail and steps it took to produce it.

Impact

since 5.2 we officially support Azure, and in this job, we are upgrading from 5.2.6 to 2023.1.0-rc8
in this test, we run the following command, to ensure it works:

for i in `sudo find /var/lib/scylla/data/keyspace_complex/ -type f |grep -v manifest.json |grep -v snapshots |head -n 1`; do echo $i; sudo sstabledump $i 1>/tmp/sstabledump.output || exit 1; done

but it failed:

Exit code: 1
Stdout:
/var/lib/scylla/data/keyspace_complex/user_with_ck-93a3e700360611eebccc2e7155ec929d/me-57-big-Index.db
Stderr:
at org.apache.cassandra.tools.SSTableExport.<clinit>(SSTableExport.java:69)
Caused by: java.lang.ClassNotFoundException: org.apache.cassandra.locator.AzureSnitch
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.cassandra.utils.FBUtilities.classForName(FBUtilities.java:556)
... 6 more

it happened once we upgraded all the nodes

How frequently does it reproduce?

did run it once, but it is clear that it will fail on every run

Installation details

Kernel Version: 5.15.0-1042-azure
Scylla version (or git commit hash): 5.2.6-20230730.58acf071bf28 with build-id 17961be569f8503b27ff284a8de1e00a9d83811e
Target Scylla version (or git commit hash): 2023.1.0~rc8-20230731.b6f7c5a6910c with build-id f6e718548e76ccf3564ed2387b6582ba8d37793c

Cluster size: 4 nodes (Standard_L8s_v3)

Scylla Nodes used in this run:

rolling-upgrade--ubuntu-focal-db-node-a0dba1a8-eastus-4 (172.190.112.196 | 10.0.0.8) (shards: 7)
rolling-upgrade--ubuntu-focal-db-node-a0dba1a8-eastus-3 (172.190.117.226 | 10.0.0.7) (shards: 7)
rolling-upgrade--ubuntu-focal-db-node-a0dba1a8-eastus-2 (172.190.117.166 | 10.0.0.6) (shards: 7)
rolling-upgrade--ubuntu-focal-db-node-a0dba1a8-eastus-1 (172.190.117.164 | 10.0.0.5) (shards: 7)

OS / Image: /subscriptions/6c268694-47ab-43ab-b306-3c5514bc4112/resourceGroups/SCYLLA-IMAGES/providers/Microsoft.Compute/images/scylla-5.2.6-x86_64-2023-08-01T05-27-35 (azure: eastus)

Test: rolling-upgrade-azure-image-test
Test id: a0dba1a8-303f-46d7-8aba-0986c71db060
Test name: enterprise-2023.1/rolling-upgrade/rolling-upgrade-azure-image-test
Test config file(s):

rolling-upgrade.yaml

Logs and commands

Restore Monitor Stack command: $ hydra investigate show-monitor a0dba1a8-303f-46d7-8aba-0986c71db060
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs a0dba1a8-303f-46d7-8aba-0986c71db060

Logs:

db-cluster-a0dba1a8.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/a0dba1a8-303f-46d7-8aba-0986c71db060/20230808_192101/db-cluster-a0dba1a8.tar.gz
sct-runner-events-a0dba1a8.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/a0dba1a8-303f-46d7-8aba-0986c71db060/20230808_192101/sct-runner-events-a0dba1a8.tar.gz
sct-a0dba1a8.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/a0dba1a8-303f-46d7-8aba-0986c71db060/20230808_192101/sct-a0dba1a8.log.tar.gz
monitor-set-a0dba1a8.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/a0dba1a8-303f-46d7-8aba-0986c71db060/20230808_192101/monitor-set-a0dba1a8.tar.gz
loader-set-a0dba1a8.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/a0dba1a8-303f-46d7-8aba-0986c71db060/20230808_192101/loader-set-a0dba1a8.tar.gz

Jenkins job URL
Argus

The text was updated successfully, but these errors were encountered:

fgelcer · 2023-08-09T14:03:26Z

@DoronArazii FYI

DoronArazii · 2023-08-09T15:21:44Z

@syuu1228 can you please provide some first assessment

soyacz · 2023-08-16T12:14:54Z

Issue reproduced but it's not reproducing in all the runs.

Issue description

This issue is a regression.
It is unknown if this issue is a regression.

Installation details

Kernel Version: 5.15.0-1042-azure
Scylla version (or git commit hash): 5.2.6-20230730.58acf071bf28 with build-id 17961be569f8503b27ff284a8de1e00a9d83811e
Target Scylla version (or git commit hash): 2023.1.0-20230813.68e9cef1baf7 with build-id c7f9855620b984af24957d7ab0bd8054306d182e

Cluster size: 4 nodes (Standard_L8s_v3)

Scylla Nodes used in this run:

rolling-upgrade--ubuntu-focal-db-node-a838df17-eastus-4 (4.236.182.172 | 10.0.0.8) (shards: 7)
rolling-upgrade--ubuntu-focal-db-node-a838df17-eastus-3 (4.236.182.155 | 10.0.0.7) (shards: 7)
rolling-upgrade--ubuntu-focal-db-node-a838df17-eastus-2 (4.236.182.150 | 10.0.0.6) (shards: 7)
rolling-upgrade--ubuntu-focal-db-node-a838df17-eastus-1 (4.236.182.96 | 10.0.0.5) (shards: 7)

OS / Image: /subscriptions/6c268694-47ab-43ab-b306-3c5514bc4112/resourceGroups/SCYLLA-IMAGES/providers/Microsoft.Compute/images/scylla-5.2.6-x86_64-2023-08-01T05-27-35 (azure: eastus)

Test: rolling-upgrade-azure-image-test
Test id: a838df17-0472-45e1-90ce-0bf4422a23e3
Test name: enterprise-2023.1/rolling-upgrade/rolling-upgrade-azure-image-test
Test config file(s):

rolling-upgrade.yaml

Logs and commands

Restore Monitor Stack command: $ hydra investigate show-monitor a838df17-0472-45e1-90ce-0bf4422a23e3
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs a838df17-0472-45e1-90ce-0bf4422a23e3

Logs:

db-cluster-a838df17.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/a838df17-0472-45e1-90ce-0bf4422a23e3/20230815_103803/db-cluster-a838df17.tar.gz
sct-runner-events-a838df17.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/a838df17-0472-45e1-90ce-0bf4422a23e3/20230815_103803/sct-runner-events-a838df17.tar.gz
sct-a838df17.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/a838df17-0472-45e1-90ce-0bf4422a23e3/20230815_103803/sct-a838df17.log.tar.gz
monitor-set-a838df17.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/a838df17-0472-45e1-90ce-0bf4422a23e3/20230815_103803/monitor-set-a838df17.tar.gz
loader-set-a838df17.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/a838df17-0472-45e1-90ce-0bf4422a23e3/20230815_103803/loader-set-a838df17.tar.gz

Jenkins job URL
Argus

juliayakovlev · 2023-09-07T10:17:24Z

The issue reproduced with 2023.1.1-20230903
Does this issue should be opened in scylla-enterprise-tools-java as well?

Command: 'for i in `sudo find /var/lib/scylla/data/keyspace_complex/ -type f |grep -v manifest.json |grep -v snapshots |head -n 1`; do echo $i; sudo sstabledump $i 1>/tmp/sstabledump.output || exit 1; done'
Exit code: 1
Stdout:
/var/lib/scylla/data/keyspace_complex/user_with_ck-709458004cac11eeb2e7606004b5a623/me-57-big-Statistics.db
Stderr:
at org.apache.cassandra.tools.SSTableExport.<clinit>(SSTableExport.java:69)
Caused by: java.lang.ClassNotFoundException: org.apache.cassandra.locator.AzureSnitch
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.cassandra.utils.FBUtilities.classForName(FBUtilities.java:556)

Issue description

This issue is a regression.
It is unknown if this issue is a regression.

Describe your issue in detail and steps it took to produce it.

Impact

Describe the impact this issue causes to the user.

How frequently does it reproduce?

Describe the frequency with how this issue can be reproduced.

Installation details

Kernel Version: 5.15.0-1045-azure
Scylla version (or git commit hash): 2023.1.1-20230903.68fa1ae4eb94 with build-id 15dd29a34d99a3c9b2196c362264a50c1579085a
Target Scylla version (or git commit hash): 2023.1.1-20230906.f4633ec973b0 with build-id b454e7a22f80cf71a33b2f39e47127225e8fbc13

Cluster size: 4 nodes (Standard_L8s_v3)

Scylla Nodes used in this run:

rolling-upgrade--ubuntu-focal-db-node-7ef06c73-eastus-4 (172.178.109.57 | 10.0.0.8) (shards: 7)
rolling-upgrade--ubuntu-focal-db-node-7ef06c73-eastus-3 (172.178.109.45 | 10.0.0.7) (shards: 7)
rolling-upgrade--ubuntu-focal-db-node-7ef06c73-eastus-2 (172.178.109.38 | 10.0.0.6) (shards: 7)
rolling-upgrade--ubuntu-focal-db-node-7ef06c73-eastus-1 (172.178.109.34 | 10.0.0.5) (shards: 7)

OS / Image: /subscriptions/6c268694-47ab-43ab-b306-3c5514bc4112/resourceGroups/SCYLLA-IMAGES/providers/Microsoft.Compute/images/scylla-2023.1.1-x86_64-2023-09-03T19-09-47 (azure: eastus)

Test: rolling-upgrade-azure-image-test
Test id: 7ef06c73-07f3-466f-9836-e2e5e64e54dd
Test name: enterprise-2023.1/rolling-upgrade/rolling-upgrade-azure-image-test
Test config file(s):

rolling-upgrade.yaml

Logs and commands

Restore Monitor Stack command: $ hydra investigate show-monitor 7ef06c73-07f3-466f-9836-e2e5e64e54dd
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs 7ef06c73-07f3-466f-9836-e2e5e64e54dd

Logs:

db-cluster-7ef06c73.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7ef06c73-07f3-466f-9836-e2e5e64e54dd/20230906_152409/db-cluster-7ef06c73.tar.gz
sct-runner-events-7ef06c73.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7ef06c73-07f3-466f-9836-e2e5e64e54dd/20230906_152409/sct-runner-events-7ef06c73.tar.gz
sct-7ef06c73.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7ef06c73-07f3-466f-9836-e2e5e64e54dd/20230906_152409/sct-7ef06c73.log.tar.gz
monitor-set-7ef06c73.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7ef06c73-07f3-466f-9836-e2e5e64e54dd/20230906_152409/monitor-set-7ef06c73.tar.gz
loader-set-7ef06c73.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7ef06c73-07f3-466f-9836-e2e5e64e54dd/20230906_152409/loader-set-7ef06c73.tar.gz

Jenkins job URL
Argus

mykaul · 2023-09-07T13:26:32Z

It will take less time for @denesb to replace sstabledump with a native tool than fix this, I reckon.

denesb · 2023-09-07T14:23:22Z

scylla sstable is ready to use for some time now. It is also included in 5.2. There is one inconvenience though, the 5.2. version of scylla-sstable does not yet know to read the schema from system tables, so it has to be provided in a separate schema.cql file.

fruch · 2023-09-10T15:34:29Z

scylla sstable is ready to use for some time now. It is also included in 5.2. There is one inconvenience though, the 5.2. version of scylla-sstable does not yet know to read the schema from system tables, so it has to be provided in a separate schema.cql file.

so a snapshot is needed first ? to generate the schema.cql?

can you backport the code that read the system tables ? into 5.2/2023.1 ?

denesb · 2023-09-11T06:57:01Z

scylla sstable is ready to use for some time now. It is also included in 5.2. There is one inconvenience though, the 5.2. version of scylla-sstable does not yet know to read the schema from system tables, so it has to be provided in a separate schema.cql file.

so a snapshot is needed first ? to generate the schema.cql?

Either that or produce it manually.

can you backport the code that read the system tables ? into 5.2/2023.1 ?

I will look into backporting it. It is a well delimited change, not affecting other parts of the system, so it is not a risk to scylla itself.

denesb · 2023-09-13T08:01:44Z

Backport PR: scylladb/scylladb#15386

yaronkaikov · 2023-09-21T08:37:49Z

@denesb I guess this can be closed , right ?

denesb · 2023-09-21T08:43:28Z

@denesb I guess this can be closed , right ?

The backport PR was not merged yet.

mykaul · 2024-03-06T12:22:40Z

This is unlikely to get fixed, as newer (native scylla sstable ... tool) is working well and we have no plans to fix it for the Java based tool.

denesb · 2024-03-06T12:24:30Z

It seems like @yaronkaikov considered this good to close, as soon as scylladb/scylladb#15386 lands in 5.2, and it already did.

mykaul · 2024-03-06T13:48:04Z

It seems like @yaronkaikov considered this good to close, as soon as scylladb/scylladb#15386 lands in 5.2, and it already did.

@denesb - I'm not sure I understand - was this fixed? If it was, then it wasn't fixed properly (see scylladb/scylladb#17659 )

denesb · 2024-03-06T13:49:44Z

It seems like @yaronkaikov considered this good to close, as soon as scylladb/scylladb#15386 lands in 5.2, and it already did.

@denesb - I'm not sure I understand - was this fixed? If it was, then it wasn't fixed properly (see scylladb/scylladb#17659 )

No, we didn't do anything with sstabledump. Instead, the "fix" was backporting enough of the scylla-sstable improvements that went in after 5.2, to make it usable enough to replace sstabledump in daily usage.

fruch · 2024-03-06T14:02:16Z

It seems like @yaronkaikov considered this good to close, as soon as scylladb/scylladb#15386 lands in 5.2, and it already did.

@denesb - I'm not sure I understand - was this fixed? If it was, then it wasn't fixed properly (see scylladb/scylladb#17659 )

No, we didn't do anything with sstabledump. Instead, the "fix" was backporting enough of the scylla-sstable improvements that went in after 5.2, to make it usable enough to replace sstabledump in daily usage.

so it's there in 5.2, and we can backport the usage of the new tool into SCT branch-5.2 ?

fruch · 2024-03-06T14:04:51Z

@vponomaryov

once @denesb confirm it's in place,

we'll need to backport
scylladb/scylla-cluster-tests#6463

denesb · 2024-03-06T14:31:56Z

so it's there in 5.2, and we can backport the usage of the new tool into SCT branch-5.2 ?

Well... I backported the schema auto-detect, but not the bunch of follow-up fixes we found when using SCT w.r.t. EAR (enterprise). So maybe if you enable it in SCT, you will start hitting problems.

Ref: scylladb/scylladb#17659 Ref: scylladb/scylla-tools-java#338

Skip the sstables validation after the DB nodes upgrades due to the known issue [1]. [1] scylladb/scylla-tools-java#338

DoronArazii assigned syuu1228 Aug 9, 2023

denesb mentioned this issue Sep 13, 2023

[5.2 backport] tool/scylla-sstable: more flexibility in obtaining the schema scylladb/scylladb#15386

Closed

mykaul mentioned this issue Mar 6, 2024

[Azure][5.2] sstabledump fails to upgrade sstables with Unable to find snitch class 'org.apache.cassandra.locator.AzureSnitch' error scylladb/scylladb#17659

Closed

2 tasks

mykaul closed this as not planned Won't fix, can't repro, duplicate, stale Mar 6, 2024

vponomaryov added a commit to vponomaryov/scylla-cluster-tests that referenced this issue Mar 6, 2024

fix(azure): make upgrade work on Azure

19dad77

Ref: scylladb/scylladb#17659 Ref: scylladb/scylla-tools-java#338

vponomaryov mentioned this issue Mar 6, 2024

fix(azure): make upgrade work on Azure scylladb/scylla-cluster-tests#7249

Merged

2 tasks

vponomaryov added a commit to vponomaryov/scylla-cluster-tests that referenced this issue Mar 7, 2024

fix(azure): make upgrade work on Azure

ba84c91

Skip the sstables validation after the DB nodes upgrades due to the known issue [1]. [1] scylladb/scylla-tools-java#338

vponomaryov added a commit to vponomaryov/scylla-cluster-tests that referenced this issue Mar 7, 2024

fix(azure): make upgrade work on Azure

193eab0

Skip the sstables validation after the DB nodes upgrades due to the known issue [1]. [1] scylladb/scylla-tools-java#338

vponomaryov added a commit to vponomaryov/scylla-cluster-tests that referenced this issue Mar 7, 2024

fix(azure): make upgrade work on Azure

2861fc4

Skip the sstables validation after the DB nodes upgrades due to the known issue [1]. [1] scylladb/scylla-tools-java#338

fruch pushed a commit to scylladb/scylla-cluster-tests that referenced this issue Mar 7, 2024

fix(azure): make upgrade work on Azure

96572aa

Skip the sstables validation after the DB nodes upgrades due to the known issue [1]. [1] scylladb/scylla-tools-java#338

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sstabledump failed because AzureSnitch ClassNotFoundException #338

sstabledump failed because AzureSnitch ClassNotFoundException #338

fgelcer commented Aug 9, 2023

Logs:

fgelcer commented Aug 9, 2023

DoronArazii commented Aug 9, 2023

soyacz commented Aug 16, 2023

Logs:

juliayakovlev commented Sep 7, 2023 •

edited

Loading

Logs:

mykaul commented Sep 7, 2023

denesb commented Sep 7, 2023

fruch commented Sep 10, 2023

denesb commented Sep 11, 2023

denesb commented Sep 13, 2023

yaronkaikov commented Sep 21, 2023

denesb commented Sep 21, 2023

mykaul commented Mar 6, 2024

denesb commented Mar 6, 2024

mykaul commented Mar 6, 2024

denesb commented Mar 6, 2024

fruch commented Mar 6, 2024

fruch commented Mar 6, 2024

denesb commented Mar 6, 2024

sstabledump failed because AzureSnitch ClassNotFoundException #338

sstabledump failed because AzureSnitch ClassNotFoundException #338

Comments

fgelcer commented Aug 9, 2023

Issue description

Impact

How frequently does it reproduce?

Installation details

Logs:

fgelcer commented Aug 9, 2023

DoronArazii commented Aug 9, 2023

soyacz commented Aug 16, 2023

Issue description

Installation details

Logs:

juliayakovlev commented Sep 7, 2023 • edited Loading

Issue description

Impact

How frequently does it reproduce?

Installation details

Logs:

mykaul commented Sep 7, 2023

denesb commented Sep 7, 2023

fruch commented Sep 10, 2023

denesb commented Sep 11, 2023

denesb commented Sep 13, 2023

yaronkaikov commented Sep 21, 2023

denesb commented Sep 21, 2023

mykaul commented Mar 6, 2024

denesb commented Mar 6, 2024

mykaul commented Mar 6, 2024

denesb commented Mar 6, 2024

fruch commented Mar 6, 2024

fruch commented Mar 6, 2024

denesb commented Mar 6, 2024

juliayakovlev commented Sep 7, 2023 •

edited

Loading