Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dashboard shows unknown for job that should be known #2408

Open
akremin opened this issue Nov 7, 2024 · 2 comments
Open

dashboard shows unknown for job that should be known #2408

akremin opened this issue Nov 7, 2024 · 2 comments
Assignees

Comments

@akremin
Copy link
Member

akremin commented Nov 7, 2024

On Nov 5th 2024 in daily a tile was observed on two exposures. The first was good, but the second had low exposure time (aborted) and was only run through sky subtractions. The column coloring is all correct but the status shows as "unknown" and is colored orange for the skysub exposure. Below is a screenshot. We should fix the dashboard to do the correct thing in this case.

image
@akremin akremin self-assigned this Nov 14, 2024
@akremin
Copy link
Member Author

akremin commented Nov 16, 2024

The issue was that for this tile and 2 other examples I found, the skysub exposure was not in the processing_table for the tile but was rather in the unprocessed_table.

My hypothesis is that desi_proc_night was run without --daily mode, such that the script would not submit skysub jobs. However, that doesn't explain why the outputs exist on disk, as I would have expected them to have been removed if the tile was purged.

Because I don't have an explanation for the root cause, I will leave this ticket open, even if the issue is not in the dashboard.

@sbailey
Copy link
Contributor

sbailey commented Dec 23, 2024

Other recent examples of this issue are nights 20241205 (expid 266913), 20241206 (expid 267061), and 20241209 (expid 267511), all of which were LASTSTEP=skysub. We're starting to build up several false positive orange alerts per month, so let's revisit this to try to identify and fix the underlying cause.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants