Handling non-ASCII characters #47
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… DataSource rather than trying to decode fields after they have passed through feature.get(), which causes errors with non-ASCII characters. Changing str to unicode inside simple_namer for same reason.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I had a lot of trouble getting non-ASCII characters to load into boundaryservice with loadshapefiles. I noticed that the geodjango DataSource that is getting created as part of loadshapefiles doesn't receive the config['encoding'] value, and that non-ASCII characters were erroring out at the feature.get() stage before they even got to the decode() calls in the previous code. Setting the encoding at the DataSource level obviates the other decode calls, as far as I can tell.
I also added some methods to BoundaryResource to properly set the charset when needed, so the API will spit out the characters correctly.