Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve progress reporting #32

Merged
merged 1 commit into from
Oct 29, 2024
Merged

Conversation

HappyTetrahedron
Copy link
Contributor

Summary

  • I changed the log line that reports on progress after each query run to include more useful details. These are particularly needed for the use-case of backfilling data over a larger timeframe and across multiple products - it needs to be visible at all time where the processing currently is, so that it can be restarted in the case of transient errors.
  • I furthermore removed the printing of the query results. It is extremely spammy, printing up to a hundred results for each timestep, making the progress report log lines impossible to find (or keep in scrollback buffer) while providing little actual utility.

Checklist

  • Categorize the PR by setting a good title and adding one of the labels:
    bug, enhancement, documentation, change, breaking, dependency
    as they show up in the changelog

@HappyTetrahedron HappyTetrahedron added the enhancement New feature or request label Oct 29, 2024
@HappyTetrahedron HappyTetrahedron requested a review from a team as a code owner October 29, 2024 12:20
@HappyTetrahedron HappyTetrahedron merged commit 2079fd1 into master Oct 29, 2024
3 checks passed
@HappyTetrahedron HappyTetrahedron deleted the local-backfill-changes branch October 29, 2024 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants