-
Notifications
You must be signed in to change notification settings - Fork 212
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use upstream thumbnail if available #898
Conversation
Full-stack documentation: Ready https://WordPress.github.io/openverse/_preview/898 Please note that GitHub pages takes a little time to deploy newly pushed code, if the links above don't work or you see old versions, wait 5 minutes and try again. You can check the GitHub pages deployment action list to see the current status of the deployments. |
Based on the high urgency of this PR, the following reviewers are being gently reminded to review this PR: @obulat Excluding weekend1 days, this PR was updated 2 day(s) ago. PRs labelled with high urgency are expected to be reviewed within 2 weekday(s)2. @krysal, if this PR is not ready for a review, please draft it to prevent reviewers from getting further unnecessary pings. Footnotes |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, just one question but not a blocker as I don't think it'll break anything, realistically.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
* cleaning and temp table in pg * sketch of full dag NOT TESTED * inaturalist dag without tests or reporting (yet) * complete dag, 25 mill recs in 5.5 hours local test * Add passwords for s3 testing with new docker * make temp loading table UNLOGGED to load it faster * inat with translation 75 million recs in 8 hrs * using OUTPUT_DIR for API files * clarify delayed requester vs requester * DRYer approach to tags TO DO * comments on taxa transformation * scientific names not ids for manual translation * TO DO comment clean-up * fix name insert syntax * Merge 'main' into feature/inaturalist-performance * add clarity on batch limit override * missing piece of merge from main * limit to 20 tags per photo * add option to use alternate dag creation for sql * adjust tests see issue #898 * slightly faster way to pull medium test sample * Note another data source for vernacular names * remove unnecessary test code * clean and upsert one batch at a time * log parsing resource doc * use common.constants.IMAGE instead of MEDIA_TYPE * add explanation of ancestry joins and taxa tags * use existing clean_intermediate_table_data * remove unnecessary env vars from load_to_s3 * declarative doc string for file update check * update iNaturalist description * remove message to Staci :) * use dynamically generated load subtasks * clarify taxa comments and include languages * consolidate consolidation code * add testing for consolidated metrics * separate ti_mock instances per test * test get batches * shorter titles to save space * add better testing instructions * dag parameter to manage post-ingestion deletions * Add kwargs to get_response_json call * get_media_type can be static method Co-authored-by: Krystle Salazar <[email protected]> * link to original inaturalist photo, rather than medium Co-authored-by: Krystle Salazar <[email protected]> * prefer creator name over login * remove unused constants * add to do for extension cleanup Co-authored-by: Madison Swain-Bowden <[email protected]> Co-authored-by: Krystle Salazar <[email protected]>
Fixes
Fixes #675 by @stacimc
Description
Sends the URL of the upstream thumbnail to Photon if it's available and it's not from Phylopic (since these are not working currently), so we can get the thumbnail faster and prevent timing out when the image is too big. See this comment from the linked issue for more details on the thumbnail investigation.
Testing Instructions
You can manually set a thumbnail URL from the database for any image and test if it's retrieved from the
/thumb
endpoint.Checklist
Update index.md
).main
) or a parent feature branch.errors.
Developer Certificate of Origin
Developer Certificate of Origin