display-v2: lexer #22216
Open
display-v2: lexer #22216
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## Description Remove support for showing Display fields from `IndexerReader` and `sui-indexer`'s JSON-RPC implementation. This implementation is not in use, and the removal is intended to simplify future work on Display. ## Test plan ``` sui$ cargo nextest run \ -p sui-indexer \ -p sui-graphql-rpc \ -p sui-graphql-e2e-tests ```
## Description Switch the implementation of `Display` in legacy GraphQL from a copy of the fullnode implementation to the `sui-display` implementation. This is a trial run for making the same change to fullnodes. ## Test plan E2E tests: ``` sui$ cargo nextest run -p sui-graphql-e2e-tests ``` Changes affect output order, (entries are now ordered by key name), and wording for errors.
## Description Use the `sui-display` crate to implement Display v1 handling on fullnodes. This change brings all code responsible for rendering Display strings inline to using the same implementation, which is better tested and has better support for enforcing limits etc. ## Test plan Manually tested that fullnode Display support continue to work after this change. `sui-display` has already been tested against all existing Display strings on mainnet to check that they parse successfully or if they produce an error, it's for an expected reason.
## Description Starting work on Display v2, with the lexer for the base grammar (the grammar supporting the same feature set as Display v1). The only difference is that curlies are escaped by doubling them up, instead of using the `\` character. ## Test plan New unit tests: ``` sui$ cargo nextest run -p sui-display ```
## Description Tokenize syntax for indexing into things (both the single and double bracket variants), and alternates. This is enough syntax to implement data structure and dynamic field accesses which may fail. ## Test plan New unit tests: ``` sui$ cargo nextest run -p sui-display ```
## Description Lexing decimal and hexadecimal numbers with optional underscore separators. ## Test plan New unit tests: ``` sui$ cargo nextest run -p sui-display ```
## Description Lexing tokens for types and vector literals (fully-qualified struct types and type parameters). ## Test plan ``` sui$ cargo nextest run -p sui-display ```
## Description Parentheses are used in the representation of positional structs and variants. ## Test plan New unit tests ``` sui$ cargo nextest run -p sui-display ```
## Description '@' will be used to indicate that a number should be interpreted as an address literal. ## Test plan New unit tests: ``` sui$ cargo nextest run -p sui-display ```
## Description Tokenize colons, making it possible to lex all parts of a struct literal with field names. ## Test plan ``` sui$ cargo nextest run -p sui-display ```
## Description Tokenize `#` which separates an enum variant's optional name and its index. ## Test plan ``` sui$ cargo nextest run -p sui-display ```
## Description Tokenize strings, byte-strings and hex-strings. ## Test plan New unit tests: ``` sui$ cargo nextest run -p sui-display ```
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ |
There was a problem hiding this comment.
Pull Request Overview
This PR implements a new lexer for the Display v2 grammar and updates module exports to support the new implementation.
- Added a new lexer module in the v2 directory.
- Clarified comment language in the v1 lexer regarding escape sequence tokenization.
- Updated the library module to expose the new v2 module.
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| crates/sui-display/src/v2/mod.rs | Introduces the new lexer module for Display v2. |
| crates/sui-display/src/v1/lexer.rs | Updates comment to clarify tokenization of escapes. |
| crates/sui-display/src/lib.rs | Exposes the new v2 module. |
amnn
temporarily deployed
to
sui-typescript-aws-kms-test-env
GitHub Actions
Inactive
amnn
commented
May 23, 2025
| Unexpected, | ||
|
|
||
| /// Whitespace around expressions. | ||
| Whitespace, |
There was a problem hiding this comment.
Treating whitespace as an explicit token because there are some forms that are whitespace sensitive:
- Double square brackets.
- String literals with prefixes.
- Numeric literals with suffixes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Implementing a lexer that tokenizes everything needed for the Display v2 grammar.
Test plan
New unit tests:
Stack
Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.