-
Notifications
You must be signed in to change notification settings - Fork 92
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix Github Actions #834
Fix Github Actions #834
Conversation
… comment" This reverts commit 7e137cb.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I give my approve, ONLY for running tests.
This does not mean that this PR is ready to merge in master.
225097d
to
4aa15cd
Compare
# def test_job_stack_overflow_error(self): | ||
# error_code = self.JOB_STACK_OVERFLOW_ERROR | ||
# data = [[1, 2, 3, 4], [7, 9, 12]] | ||
# buffers = 4 | ||
# stats_before = self.kphp_server.get_stats() | ||
# resp = self.kphp_server.http_post( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we'll remove this completely?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, done.
The problem
Github Actions fails randomly. On my fork i rerun all jobs for same code many times. And got this:
I think it's very important to fix actions, because without all checks passed branch can't be merged and of course there is no trust to PR. For example my previous PR Fix build C libraries inside kphp. @Danil42Russia rerun checks on my PR 3 times and no one succed. So branch with useful fix can't be merged, because of random. And this is not about code in PR, this is about randomly failed tests. You can check my fork for more (i tested another my PR Add mbstring functions to kphp): my fork.
The path and fix
Linux build
So, there is 5 randomly failing python tests (timeout error):
The cause is that the latest version of
requests
does not supporturllib3
2.0.This is triggered by versions of
requests-toolbelt
andurllib3
that were both released in the past few weeks (May 1 and May 4, 2023). This is already an issue inrequests-toolbelt
. This is refer tourllib3
2.0 bug: urllib3 keeps a reference to that exception to include it in theMaxRetryError
exception it will eventually raise. This means the garbage collector will not close the socket on its own, which means the connection will be kept open. So when urllib3 retries, it won't be able to open a connection, because the server is waiting for the previous try to finish. That's a deadlock!Note! I've tried to rewrite
send_http_request
function fromtests/python/lib/http_server.py
with urllib requests. This is wrong way, urllib have different work style. Downgrade urllib much more safety.And 1 test, randomly failing with
RuntimeError: Got bad stat line
errorTo fix this, i changed
tests/python/lib/stats_receiver.py
that way, so now when we receive not complete stats key, we just skip it. That's right because after this test fails, even logs shows that stats file is full and right. So sometimes engine don't be on time to write full string, when test runs. But old code already have timeout function for that (wait_next_stats
). So i think when we wait for updating stats and got not full key, we must't throw runtime error. Also when getValueError
when unpacking key, this is also reason to wait for engine to write.And 1 test, randomly failing because of timeout:
Test deleted, because of incorrect work with ASAN. ASAN randomly shutdown worker when getting stackoverflow error, so we have timeout.
MacOS build
So, there is 3 failed cpp tests:
All tests deal with threads. I think that slow syscalls might cause other threads to spawn to take over. Github Actions default mac os is 3 cores machine, so maybe the efficiency cores are too slow to take over and kphp thinks the syscall takes too long. So i disable these tests similar to
maximum-test.cpp
(which also deals with threads).Result
Result