Missing tiles/cumulative folders in daily for main backup tiles

[araichoor]

if I m correct, for the following tiles 24 main backup tiles, the expected tiles/cumulative/TILEID/LASTNIGHT folder is missing.

I obtain this statement with reading the exposures-daily.csv file, restrict to (survey=="main") & (efftime_spec > 0) exposures, and retain the latest exposure for each tile; I then check if the on-disk latesttiles/cumulative/TILEID/LASTNIGHT folder corresponds to what I d expect.

as those are backup tiles, I guess there s no urgency.
I noticed that because the 20240818-19 ones make my main-status webpage code crash, for whatever reason.

TILEID EFFTIME_SPEC OBSSTATUS IS_EXPECTED_DIR  NIGHT   ONDISK_LASTNIGHT
------ ------------ --------- --------------- -------- ----------------
 40062         63.3    obsend           False 20240818         20230806
 40237         33.0    obsend           False 20220315              -99
 40533         37.9    obsend           False 20220113         20211217
 40535         62.9    obsend           False 20220113              -99
 40554         17.7    obsend           False 20220210         20220127
 40689         40.6    obsend           False 20240819         20240723
 41253         14.6    obsend           False 20230112         20211216
 41273          7.8    obsend           False 20220227         20220112
 41371         29.2    obsend           False 20230209              -99
 41484         22.7    obsend           False 20230206         20230110
 41680         43.6    obsend           False 20220922         20211218
 41685        121.0   unknown           False 20220101         20211218
 41721         29.9    obsend           False 20211218         20211206
 42620         35.7    obsend           False 20230107              -99
 42660         54.5    obsend           False 20220217         20220216
 42665         59.1    obsend           False 20220217         20220216
 42666         21.0    obsend           False 20220316         20220216
 42669         51.7    obsend           False 20220217         20220216
 42767         44.2    obsend           False 20220910         20220123
 42946         11.2    obsend           False 20240818         20240811
 42980         33.6    obsend           False 20220112              -99
 43009         51.2    obsend           False 20220114              -99
 43017         33.7    obsend           False 20230130              -99
 43072         56.5    obsend           False 20221202              -99

tileid=41685 have a efftime=nan exposure (114859 on 20211220), that probably explains it.
I ve not dug into the other cases, sorry!

and for those two backup tiles have later efftime_spec=0 exposures, so I guess it s ok:

TILEID EFFTIME_SPEC OBSSTATUS IS_EXPECTED_DIR  NIGHT   ONDISK_LASTNIGHT
------ ------------ --------- --------------- -------- ----------------
 42282         34.6  obsstart            True 20240723         20240818
 42994          7.4  obsstart            True 20220112         20220113

for completeness, there also are the three retired bright/dark tiles showing up; but I wouldn t worry about those, as I guess they ve been retired because the data were deemed non-valid / non-recoverable:

TILEID EFFTIME_SPEC OBSSTATUS IS_EXPECTED_DIR  NIGHT   ONDISK_LASTNIGHT
------ ------------ --------- --------------- -------- ----------------
  2704        524.1   retired           False 20210627              -99
  2786        283.9   retired           False 20210627              -99
 20307         59.7   retired           False 20210707              -99

[araichoor]

for reproducibility / clarity, here are the lines of code I used.

from glob import glob
import numpy
from desitarget.geomask import match_to

prod = "daily"
prod_dir = "/global/cfs/cdirs/desi/spectro/redux/{}".format(prod)

# read exposures file
d = Table.read(os.path.join(prod_dir, "exposures-{}.csv".format(prod)))
d = d[(d["SURVEY"] == "main") & (d["EFFTIME_SPEC"] > 0)]

# pick latest night for each tileid
d = d[np.lexsort((-d["NIGHT"], d["TILEID"]))]
_, ii = np.unique(d["TILEID"], return_index=True)
d = d[ii]

# propagate the obstatus from the tiles file
t = Table.read(os.path.join(prod_dir, "tiles-{}.csv".format(prod)))
ii = match_to(t["TILEID"], d["TILEID"])
assert ii.size == len(d)
d["OBSSTATUS"] = t["OBSSTATUS"][ii]

# expected "latest" tiles/cumulative folder
d["EXPECTED_DIR"] = [os.path.join(prod_dir, "tiles", "cumulative", str(tileid), str(night)) for tileid, night in zip(d["TILEID"], d["NIGHT"])]
d["IS_EXPECTED_DIR"] = [os.path.isdir(_) for _ in d["EXPECTED_DIR"]]

# actual "latest" tiles/cumulative folder
d["ONDISK_LASTNIGHT"] = -99
for i, tileid in enumerate(d["TILEID"]):
    mydirs = sorted(glob(os.path.join(prod_dir, "tiles", "cumulative", str(tileid), "*")))
    mydirs = [_ for _ in mydirs if (os.path.basename(_)[:3] == "202") & (len(os.path.basename(_)) == 8)]
    if len(mydirs) > 0:
        d["ONDISK_LASTNIGHT"][i] = int(os.path.basename(mydirs[-1]))

# discrepant tiles
sel = d["ONDISK_LASTNIGHT"] != d["NIGHT"]
d["TILEID","EFFTIME_SPEC","OBSSTATUS","IS_EXPECTED_DIR","NIGHT","ONDISK_LASTNIGHT"][sel].pprint_all()