Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Catch member errors #24

Merged
merged 1 commit into from
Apr 18, 2024
Merged

Catch member errors #24

merged 1 commit into from
Apr 18, 2024

Conversation

ajuvercr
Copy link
Member

  • Actually use concurrent requests
  • cleanup logging a bit
  • catch member extract errors

@pietercolpaert
Copy link
Member

The concurrent requests now also refetch concurrently, which of course then keeps resulting in problems:

pieter@pieter-Latitude-7430:~/Projects/LDES-client$ export DEBUG=*;ts-node bin/cli.ts https://marineregions.org/feed.ttl --retry-count 0
...
  fetch:retry Retry https://www.marineregions.org/feed.ttl?page=2023-10-02T11%3A00%3A00Z%2F2023-10-02T12%3A00%3A00Z 114/0 +5ms
  fetch:retry Retry http://marineregions.org/mrgid/7494?t=1704805889 156/0 +1ms
  fetch:retry Retry http://marineregions.org/mrgid/7496?t=1704805889 153/0 +5ms
  ...

I’d also look rename the --retry-count parameter to --max-retries

@pietercolpaert
Copy link
Member

I sit also possible there’s no exponential back-off anymore, and it just retires as fast as possible?

@pietercolpaert
Copy link
Member

I think this is fixed now - can you check?

small cleanup membermanager

catch member errors

don't error the stream controller on error

Failing to fetch a page, is actually fetching an empty page

stop emitting when errored

better start up and adapt rdf-connect processor

Bump version extract-cbd-shape
@ajuvercr ajuvercr force-pushed the catch-member-errors branch from 6302798 to 2d25b43 Compare April 18, 2024 13:15
@pietercolpaert pietercolpaert merged commit 5432217 into main Apr 18, 2024
1 check passed
@pietercolpaert pietercolpaert deleted the catch-member-errors branch April 18, 2024 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants