feat!: data items local

[chrismclarke]

PR Checklist

• PR title descriptive (can be used in release notes)

TODO

• Merge after Chore: ng test ci and mock updates #2775
• Merge after hotfix: data-items local variables #2780
• Add templateCalcService specs
• Calc service changes rebased to refactor!: template-calc service #2793, merge after
• Screenshot tests (trigger on master and compare PR)
• Check whether changes fixes bug noted in example_auth Feat!: auth restore data #2795

Description

Add support for creating local variables within a data_items loop

Breaking changes

@jfmcquade:

These changes may break some functionality that relies upon evaluating local variables in order within a data-items loop, and persisting these changes across multiple iterations within the loop. These authoring patterns were never explicitly supported, and refactoring to use the new functionality would be desirable, but it's possible there are cases of production deployments that will need to be updated if targeting a code version including these changes.

Author Notes

Now any set_variable (or unnamed) row types within a data_items loop will create a temporary variable, available only within that loop. This can be used to generate more complex expressions to include within templated child rows

As a side-effect, more complex data can be used within data_items loops via intermediate variables, e.g. example_auth can now show app theme via nested @item.contact_fields["rp-contact-field._app_theme"]

Review Notes

A new comp_data_items_dev sheet has been created for the purpose of testing. Once in a good state could ideally be merged into main comp_data_items sheet (or renamed and linked from the comp_data_items sheet)

Should also check whether addresses issue in original issue

There is some scope for knock-on to data_items sheets more generally, so would be good to cross-check any advanced authoring examples. There are fairly comprehensive examples in comp_data_items and feat_data_actions - I've gone through each of these and they appear to be working as expected, but would be good to get a second pass.

I've also triggered a new set of screenshots generated on master to compare with this PR and the only thing that stands out to me as related to data_items is debug_data_items_condition sheet which appears broken on master and maybe fixed a bit on this branch (didn't look at the sheet itself as assuming it's an old sheet)

Dev Notes

This PR further decouples the data-items system from the rest of templating. It no longer processes item loops using the same ItemProcessor (but does use the processor to pipe operators like filter, sort etc.). This is because local context requires data-items to process in chunks by item (so that local context can be reset between evaluation), but itemProcessor outputs a single array of all processed items. So new methods have been added to the data_items service and utils to handle row generation and processing in a more linear way.

Handling support for @calc(...) statements within the data-items local variables was complicated, as up till now the new AppDataEvaluator hasn't had to deal with them. As a short term solution these expressions are being explicitly checked for and handled separately, but in the future it might be useful to consider how this should interoperate with the rest of the templated data system. Ideally methods to extract lists of local variables used should also be able to extract any calc references along with list of global functions used within the expression. All of this should ideally happen at the parser level, however that is beyond the scope of this PR

Git Issues

Closes #2766

Screenshots/Videos

comp_data_items_dev

Example - data_item loop now able to show deep nested contact field

[esmeetewinkel]

@chrismclarke thanks for the example, I wasn't aware of the syntax @items[0]?.id.

This almost works for my use case, only that there seems to be some sort of order-of-rendering issue:

• in the module_path_debug template, I can see a little flicker before the "locked" status (which depends on a variable set in data actions) displays correctly.

Screenity.video.-.Feb.20.2025.1.webm

• (I think this is equivalent) in the module_path template the "locked" status only shows correctly when navigating back to the page, or by toggling debug mode. on first render or on refresh the "locked" status of the current module is incorrect.

Screenity.video.-.Feb.20.2025.webm

[chrismclarke]

@chrismclarke thanks for the example, I wasn't aware of the syntax @items[0]?.id.

This almost works for my use case, only that there seems to be some sort of order-of-rendering issue:

• in the module_path_debug template, I can see a little flicker before the "locked" status (which depends on a variable set in data actions) displays correctly.

• (I think this is equivalent) in the module_path template the "locked" status only shows correctly when navigating back to the page, or by toggling debug mode. on first render or on refresh the "locked" status of the current module is incorrect.

That makes sense, as the data-items will only correctly update after being rendered for the first time, prompting a quick re-render after first evaluation. If moving to a data_query type it might be possible to more easily prevent rendering until data evaluated for the first time, although for now the simplest solution is to just use a condition to hide the content until the active module has been evaluated,e.g.

type	name	value	action_list	parameter_list	condition
begin_data_items		@data.module_tasks			@local.current_module_id
...display content
end_data_items

HOWEVER - the one issue with doing this is that if all the modules are completed then there would be no current_module_id as evaluated as first non-completed, and no content would show.

So we need an additional to identify the case where all modules are completed, or at least to know the difference between current module being undefined due to all completed vs not yet evaluated.

In this case I think it makes most sense to add a completed_count, which can evaluate from a second data_items loop and keep track of total number of completed modules. Then we can update the condition statement to show/hide content iff it has been computed, i.e.

type	name	value	action_list	parameter_list	condition
	completed_count
data_items		@data.module_tasks	data_changed | set_local : completed_count : @items.length	filter: @item.completed;
begin_data_items		@data.module_tasks			@local.completed_count !== undefined

Sidenote
as we're not including any inner items for the data_items loop, we don't actually need the begin/end statement. This might be a nicer way to implement item loops used for querying, as the syntax could later just be swapped out for data_query or whatever we settle on

---

I've updated the demo to remove the unused next_module_id snippets, and the counter variable, and include a display condition. I've also added an intermediate unlocked variable (as that's the main purpose of this PR :p ), which simplifies some of the other conditions a bit. Overall the authoring pattern feels pretty nice to me, so might be good to copy across some to the comp_data_items example

So hopefully that should be closer to what you need, but let me know if anything doesn't fully make sense or isn't working as you want it to.

[jfmcquade]

The updates in e0e25f7 make sense, and good to see the final authoring patterns in module_path_debug looking pretty neat.

[esmeetewinkel]

Thanks @chrismclarke, that works!

as we're not including any inner items for the data_items loop, we don't actually need the begin/end statement. This might be a nicer way to implement item loops used for querying, as the syntax could later just be swapped out for data_query or whatever we settle on

I hadn't realised this was possible (so thanks for pointing out), makes sense.

I've also added an intermediate unlocked variable (as that's the main purpose of this PR :p ), which simplifies some of the other conditions a bit.

I also had that at first, but the non-debug template for some reason doesn't work when the condition is put in an intermediate variable. No clue why but no strict need to get to the bottom of right now...

Overall the authoring pattern feels pretty nice to me, so might be good to copy across some to the comp_data_items example

Done, see example_data_actions