Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix log dev load issue. #299

Merged
merged 1 commit into from
Feb 2, 2024
Merged

Fix log dev load issue. #299

merged 1 commit into from
Feb 2, 2024

Conversation

sanebay
Copy link
Contributor

@sanebay sanebay commented Jan 31, 2024

Issue.
Getting asserts in logdev during recovery with mismatch in logdev idx and complaining header corrupted.

Cause
One of the logdev family 1 was not used for writes in UT and has no chunks. During recovery, when we read m_vdev_jd->sync_next_read(), we ignored the return value whether we read anything or not. On build machines, the return buffer contained garbage values, which later was considered as invalid header and end of log. log_stream_reader::next_group() line 58. For some reason on github actions, we get a proper loggroup header(from logdev family 0, because crc was matching with sequence number) as garbage value and causing it validate against a invalid header and doing a assert. This issue didnt happen before dynamic logdev changes because we had fixed number chunks for both logdev families and it read some junk and returned error with no more logdev data. Now we have situation of 0 chunks and no garbage data.

Fix
handle the return code m_vdev_jd->sync_next_read() and handle case where then is no data needs to be read.

@sanebay sanebay force-pushed the logstore_issue branch 13 times, most recently from 64df23d to 918ce21 Compare February 1, 2024 19:00
@codecov-commenter
Copy link

codecov-commenter commented Feb 1, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (ec25d4c) 62.18% compared to head (a0546d5) 60.89%.
Report is 2 commits behind head on master.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #299      +/-   ##
==========================================
- Coverage   62.18%   60.89%   -1.29%     
==========================================
  Files         137      109      -28     
  Lines       14022     9180    -4842     
  Branches     1693     1184     -509     
==========================================
- Hits         8719     5590    -3129     
+ Misses       4604     3061    -1543     
+ Partials      699      529     -170     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@sanebay sanebay changed the title Fix log stream reader. Fix log dev load issue. Feb 1, 2024
@sanebay sanebay requested review from yamingk and hkadayam February 1, 2024 19:57
@sanebay sanebay force-pushed the logstore_issue branch 3 times, most recently from 8679a50 to 529f5a9 Compare February 2, 2024 17:53
hkadayam
hkadayam previously approved these changes Feb 2, 2024
In UT's, logdev may not be used and dont have chunk.
Return empty buffer and dont check for corruption in those cases.
@sanebay sanebay merged commit 4bbea89 into eBay:master Feb 2, 2024
20 checks passed
@sanebay sanebay deleted the logstore_issue branch February 2, 2024 20:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants