Change tokens and tree struct to support loops and conditionals

[williamthome]

Description

This PR introduces :if, :case and :for directives.
:if and :case are conditionals and cannot be used together, the consumer should use one or another.
:for is a loop directive and can be used with a conditional.
Example:

~"""
<div :if={_@bool}>
  I'll be visible when _@bool equals true
  {% NOTE: _@bool must be a boolean() }
</div>

<ul>
  <li 
    :for={Item = #{name := ItemName}} :in={_@list}
    :case={maps:get(available, Item)} :of={true}
    {% NOTE: :of accepts guards, like :of={X when is_integer(X)} }
  >
    <b>{ItemName}</b>
  </li>
</ul>

{% NOTE: blocks/components can also make use of :if, :case and : for, }
{%       but I've not tested this yet, e.g.:                          }
<.comp_mod:comp_fun :if={_@bool} />
"""

The compiled template:

#{block =>
     #{0 =>
        {'if',
         {expr,{#Fun<erl_eval.42.39164016>,[bool]}},
         #{tag =>
            #{0 => #{id => [0,0],text => <<"<div>">>},
              1 =>
               #{id => [0,1],
                 text =>
                  <<"I'll be visible when _@bool equals true">>},
              2 => #{id => [0,2],text => <<"</div>">>}},
           indexes => [0,1,2]}},
       1 =>
        #{tag =>
           #{0 => #{id => [1,0],text => <<"<ul>">>},
             1 =>
              % :for has Static and Dynamic code, we zip them to get the rendered result.
              % IMPORTANT: We send Static only once to the client, after we send just what changes in Dynamic expressions.  
              {for,
               {expr,{#Fun<erl_eval.42.39164016>,[list]}},
               % IMPORTANT: :case is not implemented yet for loops, just :if. I'll implement it later in this PR.
               {'if',true},
               % The list below is static code, it never changes...
               [<<"<li>">>,<<"</li>">>],
               % and below is dynamic code (expressions), they could change...
               {[0],
                #{0 =>
                   #{tag =>
                      #{0 =>
                         #{id => [1,1,0,0],text => <<"<b>">>},
                        1 =>
                         #{id => [1,1,0,1],
                           expr => #Fun<erl_eval.42.39164016>,
                           vars => []},
                        2 =>
                         #{id => [1,1,0,2],
                           text => <<"</b>">>}},
                     indexes => [0,1,2]}}}},
             2 => #{id => [1,2],text => <<"</ul>">>}},
          indexes => [0,1,2]}},
    id => [0],
    view => arizona_tpl_compile,
    directives =>
     #{'if' => {expr,{<<"_@bool">>,{#{},false}}},
       stateful => true},
    attrs => #{},
    indexes => [0,1],
    % IMPORTANT: I'm not collecting vars right now. I'll fix it later in this PR.
    vars => #{}}

We have a simple, compact, and powerful template.

Why this and not implement it in pure Erlang?
The implementation of this PR provides a clear understanding of the consumer's intent. Given the flexible nature of the Erlang code, we must traverse the AST to accurately determine the consumer's objectives, and that's not so easy.
We must definitely provide the same functionality by writing Erlang code. However, for now, this "shortened" version addresses some important gaps in Arizona's templates.

Also, this PR changes our scanner, parser, and compiler, but just a bit:

• Comments are now tokenized in the scanner and not in the parser;
• Expansion code was moved from the parser to the compiler, so now we only transform data in the compiler. This was changed to make it possible for nested tokens to make use of loop and conditionals data, e.g., a loop has a list, and this list has items, we should be able to use the item in the nested tokens.

I haven't implemented the diff for the new struct yet. Due to the new loop rendering, I'll also need to change the JS code a bit.
The behavior of the diff patch remains the same even for loops and conditionals.

Important

I've removed some tokens concat to simplify the understanding and the development, e.g.:

% Before this PR (optimized)
5 =>
{'case',
{expr,{#Fun<erl_eval.42.39164016>,[foo]}},
{text, <<"<p>bar</p>">>}},

% After this PR
5 =>
{'case',
{expr,{#fun<erl_eval.42.39164016>,[foo]}},
#{tag =>
   #{0 => #{id => [5,0],text => <<"<p>">>},
     1 => #{id => [5,1],text => <<"bar">>},
     2 => #{id => [5,2],text => <<"</p>">>}},
  indexes => [0,1,2]}},

We can improve this in another PR.

Note

• This PR introduced some TODOs, but they are just reminders of things to be done in this same PR;
• The ugly code is still there, and bad tests also, but this is a task to be done in Enhance code quality and maintainability #8 🙂
• I started to look into this today, but I will have more time next week. It's 2:50 AM, I need to rest 💤

⚠️ Work in progress ⚠️

Closes #17, closes #18, closes #19.

• I have performed a self-review of my changes
• I have read and understood the contributing guidelines

[williamthome]

I'll keep this PR in draft but split it into more PRs.
It is getting very big and difficult to adapt to the CI.
I'll propose three new modules:

• arizona_template_scanner
• arizona_template_parser
• arizona_template_compiler

After these modules are implemented, we can exclude the following:

• arizona_tpl_scan
• arizona_tpl_parse
• arizona_tpl_compile

[williamthome]

The new scanner, parser, and compiler have been implemented and merged, but they do not implement conditionals nor loops. Anyway, I'll close this PR since it does not make sense with the new implementations.