Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TTS dies and does not recover when site freezes momentarily. #2

Open
bwookieeeee opened this issue Oct 15, 2018 · 9 comments
Open

TTS dies and does not recover when site freezes momentarily. #2

bwookieeeee opened this issue Oct 15, 2018 · 9 comments
Assignees
Labels
bug Something isn't working low priority Not a big deal, has workaround or will be deprecated/obsolete soon.

Comments

@bwookieeeee
Copy link
Collaborator

SocketIO-Client does not recover for TTS during site-wide 'hiccups'. No other information available

@bwookieeeee bwookieeeee added bug Something isn't working low priority Not a big deal, has workaround or will be deprecated/obsolete soon. labels Oct 15, 2018
@Nocturnal42
Copy link
Collaborator

I'm going to need an output from the controller at the time the tts dies, in order to diagnose and fix this. There is also at least one place that if a site side 'hiccup' occurs, causes an unrecoverable error inside the socket-IO client library that cannot be fixed other than restarting the controller.

That said, generally when this sort of error occurs, the end result is to socket failing multiple times in a short period. Adding a check for this behaviour, and initiating a restart when it occurs is on my TODO list.

@bwookieeeee
Copy link
Collaborator Author

Can I assign you this issue?

@Nocturnal42 Nocturnal42 self-assigned this Oct 15, 2018
@reltham
Copy link
Contributor

reltham commented Oct 15, 2018

Since updating to 0.7.2 of socketIO I have yet to lose TTS on my music bot. Going on like 3-4 days of continuous running. I believe the common case is fixed by that update. However, you said it still happens occasionally even with 0.7.2, so once I update that bot to this repo, I'll have to enable logging to try and capture if/when it fails.

@bwookieeeee
Copy link
Collaborator Author

I'm not sure if it's the same problem, but Dan's Channel will have bots failing TTS at the exact same time sometimes multiple times a day and updating socketio-client has not fixed the issue.

@Nocturnal42
Copy link
Collaborator

Generally when the server "hiccups", everyone is affected.

@reltham
Copy link
Contributor

reltham commented Oct 15, 2018

However, at the same time Dan's bots were losing TTS, my music bot was not affected. Prior to updating, I was having TTS drop constantly. Like usually within an hour or so of rebooting.

@Nocturnal42
Copy link
Collaborator

Affected != same results. Depending on where the code on the bot is currently at when the hiccup occurs, can result in substantially different outcomes.

For instance, if something tries to communicate via an emit, during the reconnection process, that will cause an exception that can lead to a thread dying. While another bot that does not attempt an emit during the same period will be fine.

Note: the described behavior above is with the standard controller, this code has several alterations to the networking code to catch such situations and recover.

@Nocturnal42
Copy link
Collaborator

image

This is one of the end results of a server hiccup. It's hard to catch, because the controller doesn't actually see those warnings. I'm hoping that by registering a on_error or on_reconnect handler for the socket, that I catch this behaviour, and trigger either a refresh of the connection details or a controller restart.

@bwookieeeee
Copy link
Collaborator Author

Should I close this? There hasn't been any site-wide hiccups since this issue was posted.

bwookieeeee added a commit that referenced this issue Aug 19, 2019
Remove Language Code from Google Cloud
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working low priority Not a big deal, has workaround or will be deprecated/obsolete soon.
Projects
None yet
Development

No branches or pull requests

3 participants