Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gleaner and Nabu progress logs are not helpful #60

Open
webb-ben opened this issue Dec 10, 2024 · 5 comments
Open

Gleaner and Nabu progress logs are not helpful #60

webb-ben opened this issue Dec 10, 2024 · 5 comments
Assignees

Comments

@webb-ben
Copy link
Member

webb-ben commented Dec 10, 2024

Inside Dagster, Gleaner and Nabu are printing the progress they are making. However it appears to get lost in the Dagster UI as a hex string instead of what should be a progress bar:

image

We should make sure this gets serialized as UTF-8 if we are going to be printing the logs to the UI. Otherwise I would request we remove this feature because it makes it hard to track the progress of the run.

@webb-ben webb-ben changed the title Gleaner and Nabu logs are not helpful Gleaner and Nabu progress logs are not helpful Dec 10, 2024
@C-Loftus
Copy link
Member

I agree with this. And maybe further we should remove progress bar stuff in general. Checking if we are in a headless env is flakey and adds another dependency.

@webb-ben
Copy link
Member Author

Yeah seems like it was built for CLI monitoring... and doesn't translate into the web UI

@webb-ben
Copy link
Member Author

fwiw decoding the logs to UTF-8 instead of latin does not help

@C-Loftus
Copy link
Member

C-Loftus commented Jan 2, 2025

We should align on which program is more important to prioritize since both programs have a fair bit of logging.

@webb-ben
Copy link
Member Author

webb-ben commented Jan 2, 2025

Nabu is likely the worse culprit at the moment. Gleaner prints semi-regular (based on timestep) logs. Nabu prints 4-5 logs per item/transaction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants