Landsat Appears to have Missing Scenes between the PC and USGS

[alexgleith]

Hey folks

I've been experimenting with the Planetary Computer and have found a few examples where there are missing scenes, and within scenes, missing files.

Here's a notebook that demonstrates a discrepancy of 30 scenes over fiji: https://gist.github.com/alexgleith/8ca78959c03d4ff943dd4888432f84e4

And this scene has missing .tif files, or perhaps, the file paths are incorrect: https://planetarycomputer.microsoft.com/api/stac/v1/collections/landsat-c2-l2/items/LC08_L2SR_074073_20221105_02_T1

I'd be interested to know if anyone has any ideas on why this is happening or if there are plans to correct any issues that may be causing the discrepancy?

[TomAugspurger]

Thanks for the report. We've had some trouble with getting a 100% robust pipeline working with USGS's servers.

The current version of the pipeline should be robust. It looks like the most recent missing scene from your list is from October 2022. For the missing scenes, I plan to start a job to backfill the archive sometime later this year, but haven't been able to get to it yet.

And this scene has missing .tif files, or perhaps, the file paths are incorrect: https://planetarycomputer.microsoft.com/api/stac/v1/collections/landsat-c2-l2/items/LC08_L2SR_074073_20221105_02_T1

Yet another issue with our old pipeline, we'd get a successful HTTP response and upload the data, when in fact it was an HTML error page. Follow #101 for updates on that. We're working on it now.

[alexgleith]

Sounds good!

For DE Africa, we noticed these reprocessed scenes too. We subscribed to their notifications and I believe we were able to keep up with the changes using that method. I guess that's what you're doing?

[TomAugspurger]

We subscribed to their notifications and I believe we were able to keep up with the changes using that method.

I have a prototype that does that, but it isn't deployed yet. We're currently polling their API at regular intervals to find new scenes.

[alexgleith]

polling their API at regular intervals

Polling by date range? Reprocessing obviously won't be identified this way, unfortunately.

[TomAugspurger]

By publishing date, so we do pick up reprocessed scenes.

[alexgleith]

By publishing date

Oh cool :-)

[alexgleith]

Hey @TomAugspurger I re-ran my comparison over multiple years over Fiji, and it looks like the issue is only there for recent scenes, i.e., 2022 and some 2023. So this issue is less of a big deal than I thought it might be. Still, those missing scenes would be nice to have!

Notebook: https://gist.github.com/alexgleith/8ca78959c03d4ff943dd4888432f84e4