Improved Exception Handling during website-screenshot fallback and several fixes for pydantic ValidationErrors
#115
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…_connector (expected type: str) - feat: additional check in the pipelines for "lom.technical.duration"-values and conversion to string
background: - some old crawlers (e.g. merlin_spider) returned an Integer value in their "getHash()"-method, which now causes pydantic ValidationErrors with the new API client (since the API expects a string value for this property)
- this fixes pydantic ValidationErrors in the es_connector during POST requests
…peline - fix AttributeErrors when a website-screenshot fails: - if the first website-screenshot fails (e.g. when the first URL in LOM technical location points towards a .mp3 or .mp4), screenshot_bytes won't be available to work with - this caused an AttributeError when the pipeline tried to convert a "None"-type to a thumbnail - refactor: extracted the functionality of taking a website screenshot with playwright in the pipeline into its own method - feat: second website screenshot fallback - if a second URL is available in LOM technical location, the pipeline will try to take a screenshot of that URL before finally giving up - this could happen in edge-cases where the first URL in LOM technical location is unavailable for website-screenshots, but the array contains a second URL that might be more fruitful for a screenshot (e.g. a landing page) - fix: 2 warnings w.r.t. too broad exception clauses
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR includes the following changes:
sodix_spider) under two circumstances:thumbnailURL, therefore the pipeline tried to fall back to a website screenshotlocation(cclom:location) would point towards a binary file (e.g..mp3or.mp4), the headless browser wouldn't be able to render / take a website screenshot either, causing anAttributeErrorException when trying to access thescreenshot_bytes-object (expected type:bytes, but receivedNone)locationpydanticValidationErrors during POST requests to edu-sharingduration:intvalues are wrapped in a string before submitting the itemBaseItem.hash: making sure that (old) crawlers which returned anint-value ingetHash()are typecast tostrbefore submitting the itemaggregationLevel:intvalues are wrapped in a string before submitting the itemlicense.internalmapping for"NONPUBLIC"license valuespipelines.pyPS: Thank you, Constantin, for the useful debug logs!