Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Race condition in dashboard signal deque #21

Open
lampep opened this issue Mar 31, 2021 · 1 comment
Open

Race condition in dashboard signal deque #21

lampep opened this issue Mar 31, 2021 · 1 comment
Labels
enhancement New feature or request

Comments

@lampep
Copy link
Member

lampep commented Mar 31, 2021 

Mar 23 13:34:14 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:14] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:14 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:14] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:14 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:14] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:14 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:14] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:14 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:14] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:14 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:14] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:14 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:14] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:15 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:15] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:15 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:15] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:15 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:15] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:15 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:15] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:15 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:15] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:15 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:15] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:16 mof-rts-00011 bash[1708]: Exception on /radiotracking/_dash-update-component [POST]
Mar 23 13:34:16 mof-rts-00011 bash[1708]: Traceback (most recent call last):
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 2447, in wsgi_app
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     response = self.full_dispatch_request()
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1952, in full_dispatch_request
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     rv = self.handle_user_exception(e)
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1821, in handle_user_exception
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     reraise(exc_type, exc_value, tb)
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/_compat.py", line 39, in reraise
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     raise value
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1950, in full_dispatch_request
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     rv = self.dispatch_request()
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1936, in dispatch_request
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     return self.view_functions[rule.endpoint](**req.view_args)
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/dash/dash.py", line 1078, in dispatch
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     response.set_data(func(*args, outputs_list=outputs_list))
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/dash/dash.py", line 1009, in add_context
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     output_value = func(*args, **kwargs)  # %% callback invoked %%
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/home/pi/pyradiotracking/radiotracking/present.py", line 448, in update_signal_variance
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     sigs = self.select_sigs(power, snr, freq, duration)
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/home/pi/pyradiotracking/radiotracking/present.py", line 382, in select_sigs
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     return [sig for sig in self.signal_queue
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/home/pi/pyradiotracking/radiotracking/present.py", line 382, in <listcomp>
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     return [sig for sig in self.signal_queue
Mar 23 13:34:16 mof-rts-00011 bash[1708]: RuntimeError: deque mutated during iteration
Mar 23 13:34:16 mof-rts-00011 bash[1708]: ERROR:radiotracking.present:Exception on /radiotracking/_dash-update-component [POST]
Mar 23 13:34:16 mof-rts-00011 bash[1708]: Traceback (most recent call last):
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 2447, in wsgi_app
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     response = self.full_dispatch_request()
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1952, in full_dispatch_request
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     rv = self.handle_user_exception(e)
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1821, in handle_user_exception
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     reraise(exc_type, exc_value, tb)
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/_compat.py", line 39, in reraise
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     raise value
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1950, in full_dispatch_request
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     rv = self.dispatch_request()
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1936, in dispatch_request
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     return self.view_functions[rule.endpoint](**req.view_args)
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/dash/dash.py", line 1078, in dispatch
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     response.set_data(func(*args, outputs_list=outputs_list))
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/dash/dash.py", line 1009, in add_context
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     output_value = func(*args, **kwargs)  # %% callback invoked %%
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/home/pi/pyradiotracking/radiotracking/present.py", line 448, in update_signal_variance
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     sigs = self.select_sigs(power, snr, freq, duration)
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/home/pi/pyradiotracking/radiotracking/present.py", line 382, in select_sigs
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     return [sig for sig in self.signal_queue
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/home/pi/pyradiotracking/radiotracking/present.py", line 382, in <listcomp>
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     return [sig for sig in self.signal_queue
Mar 23 13:34:16 mof-rts-00011 bash[1708]: RuntimeError: deque mutated during iteration
@jonashoechst
Copy link
Member

I share this observation, which is indeed a race condition inserting new signals while accessing the signal deque. However it is not interrupting operations, other than skipping a single dashboard refresh.

@jonashoechst jonashoechst added the enhancement New feature or request label Apr 13, 2021
@jonashoechst jonashoechst changed the title Observed some error messages while digging into logs from mof-rts-00011 Race condition in dashboard signal deque Apr 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jonashoechst @lampep