Prepare mktables for Unicode 15.1 and 16.0 #23133
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Leont
reviewed
Mar 19, 2025
| if (defined (my $bmg = property_ref('Bidi_Mirroring_Glyph'))) { | ||
| $bmg->set_to_output_map($EXTERNAL_MAP); | ||
| $bmg->set_range_size_1(1); | ||
| } | ||
|
|
||
| property_ref('Numeric_Value')->set_to_output_map($OUTPUT_ADJUSTED); | ||
|
|
||
| # These two properties have no short names and the file names for them | ||
| # clash in DOS 8.3. Work around this by creating shorter file names that |
There was a problem hiding this comment.
Where are we still limited by 8.3?
There was a problem hiding this comment.
On IRC the other day, I asked if we were still limited, and the answer was yes.
There was a problem hiding this comment.
For unicode filenames yes, but for ASCII filenames we don't AFAIK.
Leont
reviewed
Mar 19, 2025
Add comments, and rewrap comment lines to fit 80 columns
Unicode 15.1 introduces this new property, which needs the same special handling as plain NFKC_Casefold does.
Unicode 15.1 introduces new line breaking rules for Indic languages, via a new property Indic_Conjunct_Break. mktables works in conjunction with regen/mk_invlists.pl to construct tables and DFAs for handling these. This commit prepares mktables to do its part for Unicode versions that have these new rules.
These files are changed in 15.1 to have @missings lines, whereas they didn't before. This leads to some warnings messages, so turn off looking at them, as we do for a number of other files.
Unicode 15.1 changes the rules for line breaking with regards to Quotation marks. This prepares for that.
Unicode 15.1 adds new line breaking rules that depend on the dotted circle. This creates a table for that so that mk_invlists.pl doesn't have to have exception code for handling it.
khwilliamson
force-pushed
the
mktables_15.1
branch
from
4894f2a to
1f07a91
Compare
This is handled by ignoring it for now, and letting mktables know that the properties it contains are empty. This file, new in 16.0, gives extra information about Egyption Hieroglyphics newly encoded in 16.0. It is intended only for scholars of these ancient symbols. mktables normally handles new properties automatically, but this file is in a completely different format than previous ones, so mktables would have to be adapted to understand that. That might not be too hard, given that mktables has infrastructure to handle other outliers that have come along over the years from Unicode. But, by ignoring this file, we create empty tables which generate errors in other places in perl. These are real bugs that ought to be fixed, and will be before 16.0 is incorporated into blead. And how many Egyptologists are there in the world, much less how many use the latest Perl? So the perldelta will say that 16.0's support doesn't include these, which are mostly provisional anyway.
These new properties are automatically handled, but there is a problem. They have no short form names. Files are written for them based on their names, and those files are not distinguishable on a DOS 8.3 file system. The solution here is to manually override the automatically generated file names with distinguishable ones.
khwilliamson
force-pushed
the
mktables_15.1
branch
from
1f07a91 to
de01c61
Compare
|
This p.r. for Unicode mktables did not make it into the March 20 dev release. Does that mean we have to defer it to the 5.43 dev cycle? |
The change isn't really user visible, it would only affect people who would want to patch in a more recent Unicode version. |
jkeenan
reviewed
Apr 2, 2025
There was a problem hiding this comment.
@khwilliamson there's one unresolved conversation in this p.r. If you mark that resolved, then I think this is okay to merge.
|
There are more commits coming |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
perldelta not needed until the actual releases are incorporated.