Add syntax type and definiens to visualization

[hendrikvanantwerpen]

• Do not serialize empty spans
• Add syntax type to visualization
• Serialize definiens span and add to visualization

[github-actions]

Performance Summary

Comparing base 1b01a8c with head c8ee438 on typescript_benchmark benchmark. For details see workflow artifacts. Note that performance is tested on the last commits with changes in stack-graphs, not on every commit.

Before

--------------------------------------------------------------------------------
Command:            base/target/release/tree-sitter-stack-graphs-typescript index -D base.sqlite --max-file-time=30 --hide-error-details -- base/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 base-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
1.702^                                             ::::#                      
     |                                             :   #                      
     |                                             :   #                      
     |                                             :   #                     :
     |                                           @@:   #                    ::
     |                                        :::@ :   #                    ::
     |                                        : :@ :   #                ::::::
     |                                       :: :@ :   #                :   ::
     |                                @ @   ::: :@ :   #         ::     :   ::
     |                                @:@   ::: :@ :   #       @::: :::::   ::
     |              :  ::@     :     :@:@:::::: :@ :   #       @:::@::: :   ::
     |             @:::::@::::::: : ::@:@:::::: :@ :   #   @  :@:::@::: :   ::
   0 +----------------------------------------------------------------------->Gi
     0                                                                   74.56

After

--------------------------------------------------------------------------------
Command:            head/target/release/tree-sitter-stack-graphs-typescript index -D head.sqlite --max-file-time=30 --hide-error-details -- head/data/typescript_benchmark
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 head-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
1.406^                                                #                       
     |                                                #                      :
     |                                             :::#                      :
     |                                            ::  #                     ::
     |                                          ::::  #                     ::
     |                                          : ::  #                 ::::::
     |                                          : ::  #                 :   ::
     |                                   @    ::: ::  #          :     @:   ::
     |                    @            ::@:   : : ::  #         ::   ::@:   ::
     |              @@  ::@            : @: : : : ::  #       ::::  :: @:   ::
     |              @ ::::@  :        @: @::::: : ::  #       : :::::: @:   ::
     |             @@ ::::@  ::::::   @: @::::: : ::  #      @: :::::: @:   ::
   0 +----------------------------------------------------------------------->Gi
     0                                                                   71.17

[robrix]

Should empty spans include ones where the start and end are equal?

[hendrikvanantwerpen]

Should empty spans include ones where the start and end are equal?

Good question. My thinking was that an empty span starting beyond 0:0 means that that information was explicitly set. Now I realize that an empty span at 0:0 sometimes has been explicitly set as well. An example is a module name with empty_source_span set.

So the test was intended to correspond to "the span was explicitly set", but now it seems we cannot reliably do it.

We could make the spans optional, like syntax type, so that we have an explicit signal. But at that point, it might be more idiomatic to have separate supplemental arenas for the different fields of source info...

[hendrikvanantwerpen]

@dcreager I tried using is empty and starts at zero to detect whether a span or definiens span was actually set by the user. However, we realized that this is not valid.

If I wanted to make that explicit, I see two options:

1. Similarly to syntax type, we wrap the spans in Option. The downside is that we'll always allocate the memory, even if we don't set those fields for most nodes.

2. Split source info into multiple arenas. I hoped that this might also result in improved memory behavior, but I think supplemental arena's are regular vectors, not sparse vectors, and we'd allocate still for all nodes that don't get a value set in the end.

What are your thoughts here? Perhaps the planned changes to lsp positions are also relevant to this?