You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently we have the CSS caching that helps handle the failure case of "can't download CSS file" but lots of other failure cases still end up with the subreddit being 'lost' in that scrape.
The proposed change would be something like this (I don't know Python at the moment, so bear with me)
try:
subreddit_emotes = scrape(subreddit_name)
except
old_emotes = read_json_file(existing_json_file_path)
old_sub_emotes = [emote for emote in old_emotes if emote.sr == subreddit_name]
subreddit_emotes = old_sub_emotes
This means that all failure scenarios for scraping a sub (other download problems, bad CSS, bugs in CSS parsing/emote-extraction code, etc) mean we don't completely lose the sub, since we'll just use the emotes we have from the last scrape.
This would also mean potentially being able to get rid of the any lower-level code that was made to try and handle failure scenarios by falling back to the last successful run (like the CSS file cache AFAICT?) but that's not necessary.
The text was updated successfully, but these errors were encountered:
After checking the logs[1] and the code I see the following:
There is one emote that has an invalid image. The download fails and an error is logged, but the other images download fine.
Since the download failed, the images file can't be loaded later on, and another error is logged for this emote group.
All the other emotes are processed OK.
Apparently, some emotes are missing from marms scrapes, but without seeing his logs, I have no idea why.
Currently we have the CSS caching that helps handle the failure case of "can't download CSS file" but lots of other failure cases still end up with the subreddit being 'lost' in that scrape.
The proposed change would be something like this (I don't know Python at the moment, so bear with me)
This means that all failure scenarios for scraping a sub (other download problems, bad CSS, bugs in CSS parsing/emote-extraction code, etc) mean we don't completely lose the sub, since we'll just use the emotes we have from the last scrape.
This would also mean potentially being able to get rid of the any lower-level code that was made to try and handle failure scenarios by falling back to the last successful run (like the CSS file cache AFAICT?) but that's not necessary.
The text was updated successfully, but these errors were encountered: