Skip to content

Fsspec: Convert async methods that open sync file handles to use LocalStore #258

Open
@kylebarron

Description

@kylebarron

Async functions should not open files with blocking methods like open.

# TODO: convert to use async file system methods using LocalStore
# Async functions should not open files with blocking methods like `open`
with open(lpath, "rb") as f: # noqa: ASYNC230
await obs.put_async(self.store, rpath, f)

# TODO: convert to use async file system methods using LocalStore
# Async functions should not open files with blocking methods like `open`
with open(lpath, "wb") as f: # noqa: ASYNC230
resp = await obs.get_async(self.store, rpath)
async for buffer in resp.stream():
f.write(buffer)

Instead we should use something like:

    async def _put_file(
        self,
        lpath: str,
        rpath: str,
        mode: str = "overwrite",  # noqa: ARG002
        **_kwargs: Any,
    ) -> None:
        local_store = LocalStore("/")
        local_file = await obs.get_async(local_store, lpath)
        await obs.put_async(self.store, rpath, local_file)
  • We may want to cache the construction of LocalStore("/")? It looks like constructing that only takes 1.22 μs on my machine, so maybe not important.

  • It looks like the leading "/" on a path is not an issue. I.e. this works even though there's a leading "/" in the path:

    lpath = "/Users/kyle/github/developmentseed/obstore/README.md"
    local_store = LocalStore("/")
    local_file = obs.get(local_store, lpath)
    print(local_file.bytes().to_bytes().decode())
  • This may deadlock in the case that the async implementation is used synchronously, but that's something we can come back to in the future.

Metadata

Metadata

Assignees

No one assigned

    Labels

    fsspecPertaining to fsspec integrationgood first issueGood for newcomers

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions