Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

disrupt_remove_node_then_add_node: AssertionError: retry is not supported for long running commands, always use it with retry=0 #9534

Closed
pehala opened this issue Dec 11, 2024 · 1 comment · Fixed by #9546
Closed

disrupt_remove_node_then_add_node: AssertionError: retry is not supported for long running commands, always use it with retry=0 #9534

pehala opened this issue Dec 11, 2024 · 1 comment · Fixed by #9546
Assignees
@fruch @pehala
Labels
Bug Something isn't working right

Comments

@pehala
Copy link
Contributor

pehala commented Dec 11, 2024

Seen here (https://argus.scylladb.com/tests/scylla-cluster-tests/0bc52ea0-4703-4855-8ba2-51bead782590), but also in other runs. Seems like a nemesis bug

2024-12-11 01:58:58.053: (DisruptionEvent Severity.ERROR) period_type=end event_id=eca41cbf-cd85-41fb-ad92-feefae7994f6 duration=2m2s: nemesis_name=RemoveNodeThenAddNode target_node=Node longevity-10gb-3h-master-db-node-0bc52ea0-0-8 [34.139.132.146 | 10.142.0.61] errors=retry is not supported for long running commands, always use it with retry=0
Traceback (most recent call last):
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 5443, in wrapper
result = method(*args[1:], **kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 3712, in disrupt_remove_node_then_add_node
exit_status = remove_node()
File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/decorators.py", line 72, in inner
return func(*args, **kwargs)
File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 3689, in remove_node
res = rnd_node.run_nodetool("removenode {}".format(
File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 2676, in run_nodetool
runner(cmd, timeout=timeout, ignore_status=ignore_status, verbose=verbose, retry=retry)
File "/home/ubuntu/scylla-cluster-tests/sdcm/remote/remote_long_running.py", line 31, in run_long_running_cmd
assert retry == 0, "retry is not supported for long running commands, always use it with retry=0"
AssertionError: retry is not supported for long running commands, always use it with retry=0
@pehala pehala added the Bug Something isn't working right label Dec 11, 2024
@pehala
Copy link
Contributor Author

pehala commented Dec 11, 2024

I found this error in both Azure & GCE runs, no AWS run where this happened in the last week

@fruch fruch self-assigned this Dec 11, 2024
fruch added a commit to fruch/scylla-cluster-tests that referenced this issue Dec 12, 2024
@fruch
fix(nemesis): disrupt_remove_node_then_add_node was missing retry=0
aa0af65 
in ecc2c7e `long_running=True` paramter
was introduced, but it must come togther with `retry=0`, and that
was forgotten

Fixes: scylladb#9534
@fruch fruch mentioned this issue Dec 12, 2024
Merged
3 tasks
@fruch fruch closed this as completed in e07010b Dec 15, 2024
mergify bot pushed a commit that referenced this issue Dec 15, 2024
@fruch @mergify
fix(nemesis): disrupt_remove_node_then_add_node was missing retry=0
6aa7fc5 
in ecc2c7e `long_running=True` paramter
was introduced, but it must come togther with `retry=0`, and that
was forgotten

Fixes: #9534
(cherry picked from commit e07010b)
mergify bot pushed a commit that referenced this issue Dec 15, 2024
@fruch @mergify
fix(nemesis): disrupt_remove_node_then_add_node was missing retry=0
3da7f89 
in ecc2c7e `long_running=True` paramter
was introduced, but it must come togther with `retry=0`, and that
was forgotten

Fixes: #9534
(cherry picked from commit e07010b)
mergify bot pushed a commit that referenced this issue Dec 15, 2024
@fruch @mergify
fix(nemesis): disrupt_remove_node_then_add_node was missing retry=0
7c3e55b 
in ecc2c7e `long_running=True` paramter
was introduced, but it must come togther with `retry=0`, and that
was forgotten

Fixes: #9534
(cherry picked from commit e07010b)
This was referenced Dec 15, 2024
Merged
Merged
Merged
fruch added a commit that referenced this issue Dec 15, 2024
@fruch
fix(nemesis): disrupt_remove_node_then_add_node was missing retry=0
a31a657 
in ecc2c7e `long_running=True` paramter
was introduced, but it must come togther with `retry=0`, and that
was forgotten

Fixes: #9534
(cherry picked from commit e07010b)
fruch added a commit that referenced this issue Dec 15, 2024
@fruch
fix(nemesis): disrupt_remove_node_then_add_node was missing retry=0
fc3707a 
in ecc2c7e `long_running=True` paramter
was introduced, but it must come togther with `retry=0`, and that
was forgotten

Fixes: #9534
(cherry picked from commit e07010b)
fruch added a commit that referenced this issue Dec 15, 2024
@fruch
fix(nemesis): disrupt_remove_node_then_add_node was missing retry=0
ffd3996 
in ecc2c7e `long_running=True` paramter
was introduced, but it must come togther with `retry=0`, and that
was forgotten

Fixes: #9534
(cherry picked from commit e07010b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fruch fruch

@pehala pehala

Labels
Bug Something isn't working right
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(nemesis): disrupt_remove_node_then_add_node was missing retry=0 fruch/scylla-cluster-tests
2 participants
@fruch @pehala