Add optional orjson support for faster json reading and writing #2854
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t stages of json-ld parser. This relies on merging of the BytesIOWrapper PR.
Add orjson to sparql-results-json parser, and sparql-results-json serializer Tangential fixes to all the other non-json SPARQL-Results serializers Adding better typing to all SPARQL-Results Serializers. Got 7 ignored tests passing for SPARQL-Results Serializers.
ashleysommer
requested review from
edmondchuc and
nicholascar
and removed request for
edmondchuc
|
Looks like I need to update this a bit to resolve some conflicts with the changes that were introduced with the "JSON-LD from HTML" feature that was recently merged. I'll get to that later today or tomorrow. |
# Conflicts: # rdflib/plugins/shared/jsonld/util.py
…e after it already has a type associated.
…nch after resolving recent merge conflicts.
…y the `orjson` pyproject addition.
…n is not installed throwing a mypy error, but they are needed when orjson is installed to prevent different mypy errors. So add additional unused-ignore suppressions.
… and stdlib json outputs are the same for docs and comparison purposes.
|
Finally, all tests and lints passing. @nicholascar I'm merging this now. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds optional support for orjson, that can be enabled by installing rdflib with pip extras syntax like
rdflib[orjson], or poetry extras syntax like--extras orjson, or finally it will be detected and used if you simply installorjson>=3.9.14in your python environment.This PR touches a lot of files, because JSON is surprisingly used in a whole lot of different places in rdflib.
There are also some tangential non-JSON related changes to stream handling in a bunch of other SPARQLResult serializers. While implementing the orjson support for sparql-results-json serializer I found some errors in the way all of the different Sparql-Results-Serializers treat TextIO and BinaryIO streams. This was causing 7 errors to be thrown by the rdflib serializer tests, but they were marked as ignored in the test suite.
These additional changes include much better Typing to the Sparql-Results-Serializer subclasses, which exposed where the problems were (hooray for typed Python exposing actual errors). Fixes were made to all of the failing Sparql-Results-Serializer subclasses, and there are no skipped tests now. This also allowed the removal of a bunch of mypy
type: ignorepatches that were in place to silence the complaining type checker.I know it would be great to move all those additional changes to a different PR, but there are two reasons I didnt:
Fixes #2784