Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BufferedReader/writer for general IO? #233

Closed
grero opened this issue Sep 17, 2020 · 6 comments
Closed

BufferedReader/writer for general IO? #233

grero opened this issue Sep 17, 2020 · 6 comments

Comments

@grero
Copy link
Contributor

grero commented Sep 17, 2020

I was wondering whether it would be feasible to have BufferedReader/Writer work for general io, rather than just for IOStream? The reason I'm asking is that I'm experimenting with reading parts of large jld2 files (several 10s of G) over sftp. I implemented an IO compliant interface for reading/writing and seeking, and now I would like to have BufferedReader be able to read from this IO. Looking at the source, it seems that if the io field of BufferedReader were of type IO rather than IOStream, I could use it without having to re-implement the BufferedReader interface. I'd be interesting in any thoughts on this approach. Thanks!

@JonasIsensee
Copy link
Collaborator

Hi @grero ,

this sounds like an interesting idea that could be useful in some applications.
I must admit I have not looked at that part of the code before and at this point cannot help with that
but your suggestion sounds plausible.

I would recommend that you fork JLD2 and just try it out
and you are very welcome to report your findings.
If it works and doesn't break anything else in JLD2, we could then likely adapt the signatures to your needs.

@grero
Copy link
Contributor Author

grero commented Sep 17, 2020

I'm close to having something that sort of works here: grero/SecureRemoteFiles.jl#4
Once I have something a bit more polished perhaps we can discuss whether to include the functionality in JLD2 itself. Thanks for your reply!

@JonasIsensee
Copy link
Collaborator

Hi @grero,

I just wanted to ask whether you've made any progress on this

@grero
Copy link
Contributor Author

grero commented Oct 22, 2020

Hey,
I'm so sorry, I sort of got distracted with other projects. Regarding the tests, I was trying to figure out how best to structure them. I gather there aren't currently any tests that explicitly tests the BufferedReader? Would the best approach be to add those, or to basically run the current tests again, using a mocked up bytesavailble?

@JonasIsensee
Copy link
Collaborator

Hi,
no worries. This isn't urgent but I just didn't want the idea to be forgotten.

I think you're right. I haven't seen any explicit tests for that either.
If you've had a look at the tests you'll realize that they are in a bad state in terms of organization.
Anything is appreciated but I think it'll be best to just add a few small tests for your mockup bytesavailable.
(Really just some tiny tests that reproduce the bug you found)

@JonasIsensee
Copy link
Collaborator

Closing as duplicate of #304 and #346

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants