Tile 7741 from 2023-04-16 was never reduced #116
Comments
|
for what is worth, as of today, I see 23 possible main tiles concerned by such an issue (with 28 exposures): one dark (7741), four bright (23046, 24410, 24459, 25880), and some backup. |
|
small follow-up for the dark+bright tiles, if useful (though I suspect that folks from the data team would be better at tracking the history here):
from a quick search in the desisurveyops issues, I don t see mention of those. @abhi0395 : could take care of bringing those bright/dark tiles to full processing + having exposures-daily and tiles-daily ingesting those? I leave the backup tiles aside for now; if you ve time to also investigate + solve those, it d be great; but that s much lower priority. thanks! |
|
Hi Anand,
To reduce false positives, can you please reference the underlying
exposure_tables? They are simple csv's. If you want I can provide a python
snippet using desispec functions or using astropy-only.
Naming convention:
*${DESI_SPECTRO_REDUX}/${SPECPROD}/exposure_tables/*${NIGHT::6}
*/exposure_table_${NIGHT}.csv*
Best,
Anthony
…On Mon, May 15, 2023 at 2:34 PM araichoor ***@***.***> wrote:
for what is worth, as of today, I see 23 possible main tiles concerned by
such an issue (with 28 exposures): one dark (7741), four bright (23046,
24410, 24459, 25880), and some backup.
below is the list.
some might be due to exposures marked as bad, and not processed by the
pipeline, or things like that.
I can try to follow-up a bit on those cases.
>>> a = Table.read("/global/cfs/cdirs/desi/survey/ops/surveyops/trunk/ops/exposures.ecsv")
>>> sel = (a["TILEID"] >= 1000) & (a["TILEID"] < 60000) & (np.in1d(a["PROGRAM"], ["BACKUP", "BRIGHT", "DARK"]))
>>> a = a[sel]
>>> b = Table.read("/global/cfs/cdirs/desi/spectro/redux/daily/exposures-daily.csv")
>>> sel = ~np.in1d(a["EXPID"], b["EXPID"])
>>> sel.sum()
28
>>> a = a[sel]
>>> a[a["TILEID"].argsort()].pprint_all()
NIGHT TILEID EXPID OBSTYPE PROGRAM EXPTIME EFFTIME_ETC EFFTIME_SPEC EFFTIME GOALTIME QUALITY COMMENTS
-------- ------ ------ ------- ------- --------- ----------- ------------ -------- -------- ------- --------
20230416 7741 176606 SCIENCE DARK 1685.884 1005.371 -1.000 1005.371 -1.0 good --
20211220 23046 114876 SCIENCE BRIGHT 1455.613 105.105 -1.000 105.105 -1.0 good --
20220110 24410 117827 SCIENCE BRIGHT 916.408 17.409 -1.000 17.409 -1.0 good --
20220110 24410 117828 SCIENCE BRIGHT 1228.330 7.778 -1.000 7.778 -1.0 good --
20220110 24410 117829 SCIENCE BRIGHT 826.652 0.469 -1.000 0.469 -1.0 good --
20220216 24459 122527 SCIENCE BRIGHT 1560.731 55.470 -1.000 55.470 -1.0 good --
20220216 24459 122528 SCIENCE BRIGHT 1330.230 66.059 -1.000 66.059 -1.0 good --
20220216 24459 122529 SCIENCE BRIGHT 909.493 28.064 -1.000 28.064 -1.0 good --
20220216 24459 122530 SCIENCE BRIGHT 42.099 1.116 -1.000 1.116 -1.0 good --
20220221 25880 123283 SCIENCE BRIGHT 792.406 181.242 -1.000 181.242 -1.0 good --
20220911 40065 141865 SCIENCE BACKUP 601.838 17.400 -1.000 17.400 -1.0 good --
20220911 40835 141795 SCIENCE BACKUP 137.773 0.000 -1.000 0.000 -1.0 good --
20220911 40895 141870 SCIENCE BACKUP 607.867 30.101 -1.000 30.101 -1.0 good --
20220911 40896 141872 SCIENCE BACKUP 445.460 61.653 -1.000 61.653 -1.0 good --
20220911 40935 141871 SCIENCE BACKUP 606.913 52.994 -1.000 52.994 -1.0 good --
20220911 41334 141794 SCIENCE BACKUP 600.729 4.951 -1.000 4.951 -1.0 good --
20220911 41419 141866 SCIENCE BACKUP 602.791 27.518 -1.000 27.518 -1.0 good --
20220911 41421 141867 SCIENCE BACKUP 602.802 34.934 -1.000 34.934 -1.0 good --
20220911 41461 141869 SCIENCE BACKUP 607.988 29.690 -1.000 29.690 -1.0 good --
20220216 41654 122535 SCIENCE BACKUP 116.539 0.014 -1.000 0.014 -1.0 good --
20230503 41894 178901 SCIENCE BACKUP 601.914 1.044 -1.000 1.044 -1.0 good --
20220415 42416 130372 SCIENCE BACKUP 147.499 0.065 -1.000 0.065 -1.0 good --
20220216 42660 122532 SCIENCE BACKUP 601.416 22.097 -1.000 22.097 -1.0 good --
20220216 42665 122534 SCIENCE BACKUP 601.498 15.680 -1.000 15.680 -1.0 good --
20220216 42666 122533 SCIENCE BACKUP 602.993 16.234 -1.000 16.234 -1.0 good --
20220216 42669 122531 SCIENCE BACKUP 602.955 25.048 -1.000 25.048 -1.0 good --
20220911 42809 141868 SCIENCE BACKUP 608.612 60.675 -1.000 60.675 -1.0 good --
20220911 42873 141873 SCIENCE BACKUP 156.037 14.020 -1.000 14.020 -1.0 good --
—
Reply to this email directly, view it on GitHub
<#116 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACBM5O4FIEHOXEDYD5GQPE3XGKOPNANCNFSM6AAAAAAX62OUGY>
.
You are receiving this because you were assigned.Message ID:
***@***.***>
|
|
from my earlier post, you ve the (tileid, night, expid), so I guess you can easily infer the underlying exposure_table, right? |
|
I am happy to investigate the above. This was a general comment for the future and to advertise these important files. My point is that a common occurrence is that multiple tables are being cross-matched to generate an ascii table like the one above, but the exposure_table's data aren't included. I then have to manually cross-match row-by-row to the exposure_table's, where the majority will be answered/resolved. The exposure_table's should be just as easily checked as the |
|
I promised at the meeting on Monday that I would add warning messages to afternoon planning mentioning these tiles. Here is what I've planned: @sbailey , @akremin , does that look like roughly the relevant information? Running this tonight, I see the ~expected thing that there are 27 exposures from tonight since we have the files but haven't processed them; I'm hoping that by breaking them out separately by night we have the right balance of notifying and maing it easy to see why. |
|
A reminder that we want to make sure that the tiles in Anand's list above get the processing information propagated into the exposures-daily.ecsv file; they have presently been lost. |
|
Pinging this outstanding issue. |
|
Note that we found some portion of this again in #132. |
|
Pinging this outstanding issue; @akremin ? |
|
I can add some details about the raw data transfer. For raw data in 20230416/00176606, the file timestamps are consistent with the surrounding exposure ids. However the directory timestamp is hours later; note The most likely explanation is that the exposure was not linked into |
|
Further details: the directory timestamp indicates that the morning "catch-up" transfer successfully copied that exposure id. There was a one-off failure to copy that exposure id earlier in the night.. This appears to have been sheer bad luck; a momentary network glitch or something. Does the pipeline take into account the "catch-up" transfers that we do in the morning? |
|
@weaverba137 Apologies for the late response. The daily pipeline will find anything that has a request and data file in the I agree with other recommendations that we should have a script running roughly weekly to re-check to see if any late exposures did arrive. I'll make a desispec ticket about that. |
|
All current cases have been resolved. I developed a script in desispec PR #2124 to help debug and understand each of these cases. There was a wide range of issues from data arriving late to what appear to be manually deleted files presumably left over from partial cleanup activities, to data that was perfectly fine but the tsnr afterburner needed to be rerun to update the entries. Since the original posts, the numbers of exposures increased. I used the same code snippet to identify an updated list: These are now fixed. Using the same code snippet Anand provided above: Now gives: |
|
I confirm that today s (20231007) AP does not report anything anymore, great. |
|
Objections to closing this, @araichoor , @akremin ? |
|
no, looks good to me! |
The main survey dark tile 7741 from a single exposure 176606 on 2023-04-16 was never reduced. Anthony reports the data transfer was delayed by 8 hours, after the daily pipeline shut down the following morning, and therefore wasn’t registered in the exposure_table_20230416.csv . Since this exposure was never reduced, it's still just pending and blocking overlapping tiles at RA,Dec=166,+8. Probably not urgent, since this RA has just set for the season.
The text was updated successfully, but these errors were encountered: