Skip to content
This repository has been archived by the owner on Jun 20, 2024. It is now read-only.

try to fix coordinates #347

Merged
merged 6 commits into from
Jan 16, 2024
Merged

try to fix coordinates #347

merged 6 commits into from
Jan 16, 2024

Conversation

blankdots
Copy link
Contributor

@blankdots blankdots commented Dec 14, 2023

does this fix the coordinates ... well yes :) does it remove a lot of the unit tests ... yes
Tests will be revamped when we move to new repo with issue neicnordic/sensitive-data-archive#410

inspired by: https://github.com/neicnordic/crypt4gh/blob/master/examples/reader/main.go

  • disabled logging health endpoint as it generates unnecessary logs

@blankdots blankdots self-assigned this Dec 14, 2023
@blankdots blankdots force-pushed the bugfix/coordinates branch 4 times, most recently from a8f53aa to 7dc0905 Compare December 14, 2023 22:18
@blankdots blankdots linked an issue Dec 14, 2023 that may be closed by this pull request
@blankdots blankdots requested a review from a team December 14, 2023 22:32
@blankdots blankdots marked this pull request as ready for review December 14, 2023 22:32
@codecov-commenter
Copy link

codecov-commenter commented Dec 14, 2023

Codecov Report

Attention: 74 lines in your changes are missing coverage. Please review.

Comparison is base (5dc4fe0) 67.03% compared to head (03449f4) 62.15%.

Files Patch % Lines
api/sda/sda.go 0.00% 74 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #347      +/-   ##
==========================================
- Coverage   67.03%   62.15%   -4.89%     
==========================================
  Files          10       10              
  Lines        1611     1633      +22     
==========================================
- Hits         1080     1015      -65     
- Misses        493      583      +90     
+ Partials       38       35       -3     
Flag Coverage Δ
unittests 62.15% <6.32%> (-4.89%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@blankdots blankdots requested a review from a team December 15, 2023 11:38
api/sda/sda.go Outdated Show resolved Hide resolved
@pontus
Copy link
Contributor

pontus commented Dec 18, 2023

Code looks good, but I'm a bit worried about the amount of unit tests cut, do all need to go? I see the added other tests ("integration"), but think we could have both.

(and now I reread the top again and is less worried.)

@pontus
Copy link
Contributor

pontus commented Dec 18, 2023

Also, I'm curious, I hope (and believe) I didn't break anything when I added random access to the crypt4gh golang.

@blankdots
Copy link
Contributor Author

Also, I'm curious, I hope (and believe) I didn't break anything when I added random access to the crypt4gh golang.

We were using the datalistpackets, and I am sure they worked previously. I remember testing, but we don't have a test to see when and if they were actually broken

@blankdots blankdots marked this pull request as draft December 18, 2023 15:29
@blankdots
Copy link
Contributor Author

marking as draft as we test more things

@blankdots
Copy link
Contributor Author

blankdots commented Dec 19, 2023

@pontus we tried to make it work with large file chunks but we see that chunks larger than ~5Mb don't work
we pinpointed to: neicnordic/crypt4gh#83 changes as after those changes the way we did it previously worked

@blankdots blankdots force-pushed the bugfix/coordinates branch 2 times, most recently from d6a6e93 to 53c1c36 Compare December 19, 2023 16:09
@blankdots
Copy link
Contributor Author

@pontus i made it work with the example reader you created; it was my mistake trying to be fancy about how to work with it, now sda-download works with larger files as well, however the problem with the data edit list still exists in crypt4gh how we were using it previously

@blankdots blankdots marked this pull request as ready for review December 19, 2023 16:13
@blankdots blankdots requested a review from a team December 19, 2023 17:51
@blankdots blankdots requested a review from a team January 2, 2024 09:37
@blankdots blankdots force-pushed the bugfix/coordinates branch 2 times, most recently from f19c273 to 2cc065c Compare January 2, 2024 12:39
api/sda/sda.go Show resolved Hide resolved
@jbygdell
Copy link
Contributor

jbygdell commented Jan 9, 2024

When this PR is merged can we archive this repo so that all future work gets done in neicnordic/sensitive-data-archive?
Also these changes needs to be moved into the sensitive-data-archive repo as well.

@blankdots
Copy link
Contributor Author

When this PR is merged can we archive this repo so that all future work gets done in neicnordic/sensitive-data-archive?
Also these changes needs to be moved into the sensitive-data-archive repo as well.

yes, but there were some issues @aaperis reported i will see if i can replicate

Copy link
Contributor

@aaperis aaperis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hopefully, problem solved :-)

I believe the issue was that the logic for seeking the first byte where the byte range will be read was missing. As a consequence, download was always starting from byte 1. With the suggested change, sda-download, the read example from the crypt4gh repo and dd tests all agree. I also made some suggestions about the integration tests. Ideally, for the later we should use a small bam file when we migrate to the merged repo.

api/sda/sda.go Show resolved Hide resolved
.github/integration/tests/common/50_check_endpoint.sh Outdated Show resolved Hide resolved
.github/integration/tests/common/50_check_endpoint.sh Outdated Show resolved Hide resolved
.github/integration/tests/s3notls/52_check_endpoint.sh Outdated Show resolved Hide resolved
.github/integration/tests/s3notls/52_check_endpoint.sh Outdated Show resolved Hide resolved
@blankdots blankdots requested a review from aaperis January 15, 2024 19:34
@blankdots
Copy link
Contributor Author

I believe the issue was that the logic for seeking the first byte where the byte range will be read was missing.

you are right, i think i had it there, but in the confusion of me rewriting that piece many times and bad tests, i did not spot it

@blankdots blankdots merged commit d8fb629 into main Jan 16, 2024
8 checks passed
@blankdots blankdots deleted the bugfix/coordinates branch January 16, 2024 11:36
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

start and endcoordinate requests sometimes return too much data.
6 participants