Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

trantor::ConcurrentTaskQueue::runTaskInQueue hangs indefinitely #324

Open
Mis1eader-dev opened this issue Feb 26, 2024 · 1 comment
Open

trantor::ConcurrentTaskQueue::runTaskInQueue hangs indefinitely #324

Mis1eader-dev opened this issue Feb 26, 2024 · 1 comment

Comments

@Mis1eader-dev
Copy link
Contributor

Mis1eader-dev commented Feb 26, 2024 •
edited
Loading

On a hosted server with SSL certificates, all functions and API endpoints that relied on trantor::ConcurrentTaskQueue::runTaskInQueue started hanging indefinitely as if the mutex is locked and never unlocked, or something is blocking the queue. runTaskInQueue itself doesn't hang, print statements after it get printed as expected, so it is not an issue of the mutex, it is the task just never executing.

I checked with trantor::ConcurrentTaskQueue::getTaskCount(), and it reported 0 just before the call to runTaskInQueue.

Had to strip all logic that relied on it for the sake of allowing it to work.

Logs were on debug, and may not be of much use.

20240226 14:47:23.450161 UTC 453455 INFO  Start child process - HttpAppFrameworkImpl.cc:580
20240226 15:09:03.516471 UTC 453457 FATAL Transport endpoint is not connected (errno=107) sockets::shutdownWrite - Socket.cc:110
20240226 15:36:22.890503 UTC 453458 FATAL Transport endpoint is not connected (errno=107) sockets::shutdownWrite - Socket.cc:110
20240226 17:40:28.126379 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:40:31.175445 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:40:41.949825 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:40:58.126485 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:41:01.175529 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:41:11.949927 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:41:13.960462 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:41:28.126542 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:41:31.175616 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:41:41.949993 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:41:43.960552 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:41:58.126624 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:42:01.175703 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:42:11.950088 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:42:13.960643 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:42:28.126716 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:42:31.175792 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:42:41.950183 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:42:43.960743 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:42:58.126812 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:43:01.175919 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:43:11.950284 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:43:13.960845 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:43:26.760234 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:43:28.126923 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:43:29.283758 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:43:31.176001 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:43:41.950373 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:43:43.960968 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:43:56.760326 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:43:58.127024 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:43:59.283924 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:44:01.176101 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:44:11.950487 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:44:13.961030 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:44:26.760437 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:44:28.127109 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:44:29.284003 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:44:31.176211 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:44:41.950573 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:44:43.961129 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:44:56.760518 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:44:58.127206 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:44:59.284053 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:45:01.176295 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:45:11.950680 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:45:13.961233 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:45:26.760619 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:45:28.127291 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:45:29.284147 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:45:31.176335 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:45:41.950777 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:45:43.961355 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:45:56.760718 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:45:58.127397 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:45:59.284239 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:46:01.176425 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:46:11.950871 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:46:13.961458 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:46:26.760789 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:46:28.127483 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:46:29.284349 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:46:31.176532 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:46:41.950975 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:46:43.961570 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:46:56.760900 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:46:58.127574 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:46:59.284457 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:47:01.176632 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:47:11.951066 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:47:13.961679 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:47:26.760994 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:47:28.127679 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:47:29.284545 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:47:31.176727 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:47:41.951143 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:47:43.961769 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:47:56.761100 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:47:58.127788 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:47:59.284652 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:48:01.176848 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:48:11.951228 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:48:13.961884 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:48:26.761188 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:48:28.127899 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:48:29.284742 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:48:31.176933 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:48:41.951318 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:48:43.961970 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:48:56.761327 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:48:58.127984 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:48:59.284841 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:49:01.177047 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:49:11.951407 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:49:13.962065 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:49:26.761436 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:49:28.128060 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:49:29.284929 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:49:31.177106 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:49:41.951502 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:49:43.962153 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:49:56.761538 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:49:58.128170 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:49:59.284980 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:50:01.177197 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:50:11.951614 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:50:13.962248 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:50:26.761621 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:50:28.128257 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:50:29.285074 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:50:31.177287 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:50:41.951701 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:50:43.962348 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:50:56.761741 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:50:58.128344 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:50:59.285161 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:51:01.177489 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:51:11.951828 UTC 453457 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:51:13.962451 UTC 453458 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:51:26.761842 UTC 453459 WARN  Connection is not connected,give up sending - TcpConnectionImpl.cc:390
20240226 17:51:36.907095 UTC 457571 INFO  Start child process - HttpAppFrameworkImpl.cc:580
20240226 17:53:16.746372 UTC 457809 INFO  Start child process - HttpAppFrameworkImpl.cc:580
20240226 17:58:47.184918 UTC 458159 INFO  Start child process - HttpAppFrameworkImpl.cc:580
20240226 18:00:22.936246 UTC 458448 INFO  Start child process - HttpAppFrameworkImpl.cc:580
20240226 18:02:33.937763 UTC 458734 INFO  Start child process - HttpAppFrameworkImpl.cc:580
20240226 18:03:48.502096 UTC 459007 INFO  Start child process - HttpAppFrameworkImpl.cc:580
20240226 18:07:38.125654 UTC 1276 INFO  Start child process - HttpAppFrameworkImpl.cc:580
20240226 18:14:08.138407 UTC 2319 INFO  Start child process - HttpAppFrameworkImpl.cc:580
20240226 18:17:31.759736 UTC 2632 INFO  Start child process - HttpAppFrameworkImpl.cc:580
20240226 18:18:04.478187 UTC 2632 WARN  SIGINT signal received. - HttpAppFrameworkImpl.cc:177
20240226 18:19:32.625443 UTC 2632 WARN  SIGTERM signal received. - HttpAppFrameworkImpl.cc:172

Additional context
The server was working fine for the past few days, but an hour ago it started not responding to any API endpoints.
Restarting the host did not help either.
And it serves files just fine.

Things that were changed which could have potentially led to this:

  1. Set relaunchOnError to true
  2. Enabled file based logging
@Mis1eader-dev
Copy link
Contributor Author

Conclusive update, it turned out drogon was causing this issue, I'll open an issue there and link back.

@Mis1eader-dev Mis1eader-dev mentioned this issue Feb 27, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Mis1eader-dev