ChunkedEncodingError while fetching run metrics

[feldlime]

Hi,

when I fetching run metics with

repo = aim.Repo(REPO)
run = repo.get_run(RUN_HASH)
list(run.metrics())

I'm getting an error:

ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

Before it worked smoothly (same version, didn't change anything explicitly).

Other way of doing this

run = aim.Run(run_hash=RUN_HASH, repo=REPO)
list(run.metrics())

is working.

What could it be?

Environment

• Aim Version: 3.24.0
• Python version: 3.9
• OS: Ubuntu

[mihran113]

Hey @feldlime! There's actually a difference between the 2 versions that you've provided. In the first case Run is opened in read mode and in the second case it's in write mode, so that's why one might be working and the other - not. But still, there should be no such issue.
If you can provide a minimal repo/example to reproduce the issue it would be very helpful.

[feldlime]

Hi @mihran113, thanks for your answer!
Yes, I know the difference, but I expected it should work in both cases.
Unfortunately, I cannot provide an example to reproduce. We have hundreds of runs every day, and this error appears randomly 1-2 times a month. I tried to figure out the difference between the broken runs and normal ones but couldn't.

[mihran113]

@feldlime yeah, that's definitely not a designed behavior.
I'll look into it, will update here once I find anything useful.

[mihran113]

@feldlime just to clarify: you are using remote server for this, right?

[feldlime]

@feldlime just to clarify: you are using remote server for this, right?

yes

[mihran113]

Hey @feldlime! I've tried a lot of scenarios locally but couldn't reproduce the issue in any way. I'll add some retry logic in case ChunkedEncodingError happens, that won't harm anyone but still might help in case it happens randomly.
In the meantime, can I ask you to check if that error happens repeatably on the same run, or if it's totally random? (In case you encounter that again)