Collapsible Nested Arrays

[mozfet]

The current implementation of quickfield for nested arrays (and arrays in general) results in messy user interfaces, ala:

This needs to improve, and in order to improve, less data needs to be shown at any given time, and a hide and show strategy needs to be used. This will allow arrow rows to become smaller to enable effective drag and drop sorting.

We need to make array input to a more compact format, something like this, combined with some form of a collapsible part to display the content of the current level:

[pouya-eghbali]

OK so, here's what I have so far:

Header is one of these:

• arrayHeaderTemplate if provided
• current value of field name defined in arrayHeaderField
• current value of first field name in array (sorted alphabetically)

I start working on nested arrays now. Let me know if you have any ideas or you want any changes.

(sorry for the horrible gif)

[mozfet]

Looking good already. I do not think we need a title inside the collapsible body... its title is in the collapsible header. Also, an edit icon on the right will inform the user than clicking is possible to edit, just like the drag icon indicates that it can be dragged.

[mozfet]

Perhaps also move delete per item also on the right (on left of edit). With delete, make sure to have a confirm dialogue to avoid accidental delete.

[mozfet]

For the header of the collapsible, it may need to support a callback function to allow custom styling and summarization.

[pouya-eghbali]

Yes, I will add that too. Currently, for the header we can define a custom template OR a field name, we must add function support too because it's so much easier than custom templates.

[mozfet]

For the header of the collapsible, it may need to support a callback function to allow custom styling and summarization.

Scrap that - a custom template will be better, dynamically rendered inside the collection item header.
If the custom template is not supplied, then arrayItemHeaderField. Else... whatever else is a best attempt...

[pouya-eghbali]

I managed to fix the nested draggable hell by adding a unique class to each item's handle and also their container. Now I just need to fix field names, I have some plans to simplify this part, I hope that works. I'll start this after lunch time!

[pouya-eghbali]

Ok for some reason instance.$ doesn't work for me, but template.$ does:

Template.afArrayField_materialize.onRendered(() => {

  const instance = Template.instance()
  const template = this;
  ...

Any idea why it doesn't work? There's also an issue with mirror:destroy event, it gets called before the DOM is cleaned, mirror elements are still in DOM when our callback is called.

[pouya-eghbali]

There's also an issue with blaze, it updates the wrong header text when it is changed. Trying to figure out why? Possibly because moving items doesn't change the DOM tree.

[pouya-eghbali]

OR, maybe ^ is because autoform. Investigating.

[pouya-eghbali]

There's also an issue with blaze, it updates the wrong header text when it is changed. Trying to figure out why? Possibly because moving items doesn't change the DOM tree.

Even doing a flex-box order doesn't fix the issue

[pouya-eghbali]

OK I see what's wrong

[pouya-eghbali]

I found some magical ways to fix the problem

[pouya-eghbali]

I'm reading autoform source. afEachArrayItem calls and returns arrayTracker.getField(formId, name); which is:

ArrayTracker.prototype.getField = function atGetField(formId, field) {
  var self = this;
  self.ensureField(formId, field);
  self.info[formId][field].deps.depend();
  return self.info[formId][field].array;
};

If we can change AutoForm.arrayTracker.info[formId][field].array and trigger deps.depend(), we'll be able to force the array to re-render with correct name, data-schema-key and id.

[pouya-eghbali]

AutoForm.arrayTracker.info[formId][field].deps.changed()

Causes a re-render of the array field. Need to figure out how to set field data now.

[pouya-eghbali]

OK. Here's the current situation:

1. To change the order of the items, the only thing we need to change is data-schema-key (confirmed by checking autoform source, but we'll change the name too, just to make sure things won't break)

2. We cannot change the id of the fields. Problem is, if a nested item with 5 array items is moved to replace another item with 2 inner array items, well, because the count isn't equal we'll not have enough ids to exchange.

3. Autoform doesn't keep a record of current state of the fields and their value, so we cannot depend on autoform internals in order to change the order (getFieldValue checks the dom)

4. I'm doing

{{#if ../atts.arrayHeaderField}}
    {{fieldValue current formId ../atts.arrayHeaderField}}
{{else}}
    {{fieldValue current formId null}}
{{/if}}

to get the header value. Here's the fieldValue helper:

fieldValue(current, formId, fieldName) {
    if (!fieldName) {
      // sort alphabetically and get first fieldName
      fieldName = Object.keys(current).sort((a, b) => a > b ? 1 : -1)[0]
    }    
    return AutoForm.getFieldValue(current[fieldName], formId)
  }

in the above function, AutoForm.getFieldValue makes our function depend on the value of this field, so every time the value is changed the value shown in header is changed, that's fine, that's what we want, but problem is, after reordering this still depends on the previous value.

I managed to solve all problems, except this one, but i'm working on it

[pouya-eghbali]

I don't know if this is happening because blaze or because afEachArrayItem, this isn't an issue with draggable and dom/schema-key manipulation, it happens even if I don't drag. Simply adding and removing items causes the headers to be wrong.

[pouya-eghbali]

Looks like I found a way. If I do this:

1. Reorder AutoForm.arrayTracker.info[formId][field].array
2. Call AutoForm.arrayTracker.info[formId][field].deps.changed()
3. Manipulate dom (update names and data-schema-key)

it seems to work. let me write a function that does this automatically, hopefully it solves the issue.

[pouya-eghbali]

Doesn't work. What I've tried:

1. Forcing the entire template to re-render
2. Getting field value from autoform insertDoc instead of autoform getFieldValue
3. Using afArrayItem index instead of current to get field value
4. Updating atts.docs with new field value
5. Reordering AutoForm.arrayTracker.info[formId][field].array

And a few other things I tried. Well, I don't know how to solve the header problem. Any ideas?

[pouya-eghbali]

Last idea I have for the header:

1. On onRendered we install a Tracker.autorun to listen to field value changes
2. Each time it runs we dynamically update header values

That's my last idea. None of the others worked.

[pouya-eghbali]

Yeah, Tracker.autorun does the trick.

[mozfet]

Seems I missed all the action, bravo for finding a solution. I remember when I wrote the array drag stuff before it was quite challenging to get it to work with autoform.

[mozfet]

I've also remembered something about when array items are removed, it leaves undefined items the array in the DB instead of replacing the entire array.

[mozfet]

Yeah, Tracker.autorun does the trick.

I generally find a lot of reactive solutions are possible using autorun and sub-templates.

[pouya-eghbali]

I've also remembered something about when array items are removed, it leaves undefined items the array in the DB instead of replacing the entire array.

Yes, I saw your messages in autoform issues! I checked a lot of them today

[pouya-eghbali]

Ok it works perfectly, I'll do a little polishing and cleaning

[pouya-eghbali]

Another issue! After reordering nested arrays and submitting the form, I get a million errors and warnings in the console. Form submit works, values submit correctly, reordering of arrays works, but there's this error.

TypeError: Cannot read property 'destroy' of undefined
    at Blaze.TemplateInstance.<anonymous> (tooltippedIcon.js:119)
    at blaze.js?hash=51f4a3bdae106610ee48d8eff291f3628713d847:3398
    at Function.Template._withTemplateInstanceFunc (blaze.js?hash=51f4a3bdae106610ee48d8eff291f3628713d847:3769)
    at fireCallbacks (blaze.js?hash=51f4a3bdae106610ee48d8eff291f3628713d847:3394)
    at Blaze.View.<anonymous> (blaze.js?hash=51f4a3bdae106610ee48d8eff291f3628713d847:3500)
    at fireCallbacks (blaze.js?hash=51f4a3bdae106610ee48d8eff291f3628713d847:2014)
    at Object.Tracker.nonreactive (tracker.js:603)
    at blaze.js?hash=51f4a3bdae106610ee48d8eff291f3628713d847:2011
    at Object.Blaze._withCurrentView (blaze.js?hash=51f4a3bdae106610ee48d8eff291f3628713d847:2271)
    at Object.Blaze._fireCallbacks (blaze.js?hash=51f4a3bdae106610ee48d8eff291f3628713d847:2010)

and

TypeError: Cannot set property 'tabIndex' of null
    at Dropdown._makeDropdownFocusable (modules.js?hash=fe957d8420951119a0479225698b779e441bad0a:82369)
    at new Dropdown (modules.js?hash=fe957d8420951119a0479225698b779e441bad0a:82097)
    at Function.init (modules.js?hash=fe957d8420951119a0479225698b779e441bad0a:80843)
    at Function.init (modules.js?hash=fe957d8420951119a0479225698b779e441bad0a:82623)
    at FormSelect._setupDropdown (modules.js?hash=fe957d8420951119a0479225698b779e441bad0a:91658)
    at new FormSelect (modules.js?hash=fe957d8420951119a0479225698b779e441bad0a:91443)
    at Function.init (modules.js?hash=fe957d8420951119a0479225698b779e441bad0a:80843)
    at Function.init (modules.js?hash=fe957d8420951119a0479225698b779e441bad0a:91844)
    at Blaze.View.<anonymous> (select.js:60)
    at blaze.js?hash=51f4a3bdae106610ee48d8eff291f3628713d847:1934

and finally

TypeError: Cannot read property 'insertBefore' of null
    at HTMLSelectElement.<anonymous> (modules.js?hash=fe957d8420951119a0479225698b779e441bad0a:80618)
    at each (modules.js?hash=fe957d8420951119a0479225698b779e441bad0a:79965)
    at Init.each (modules.js?hash=fe957d8420951119a0479225698b779e441bad0a:80214)
    at HTMLDivElement.<anonymous> (modules.js?hash=fe957d8420951119a0479225698b779e441bad0a:80617)
    at each (modules.js?hash=fe957d8420951119a0479225698b779e441bad0a:79965)
    at Init.each (modules.js?hash=fe957d8420951119a0479225698b779e441bad0a:80214)
    at Init.insertBefore (modules.js?hash=fe957d8420951119a0479225698b779e441bad0a:80615)
    at Init.before (modules.js?hash=fe957d8420951119a0479225698b779e441bad0a:80574)
    at FormSelect._removeDropdown (modules.js?hash=fe957d8420951119a0479225698b779e441bad0a:91694)
    at FormSelect.destroy (modules.js?hash=fe957d8420951119a0479225698b779e441bad0a:91458)

These get repeated a million times. The first two are coming from an instance.autorun, the destroy function gets called twice? investigating.

[pouya-eghbali]

Ok, after submitting the form, in dom I see the .material-tooltip elements, they don't get removed. Could it be because their trigger element has been moved in the dom?

[pouya-eghbali]

Here's the situation:

1. Add/Remove an item from an array
2. It triggers autoform array tracker for all arrays in the array
3. It triggers a value change for all fields in the array
4. Autoform redraws the form on each trigger (76 times in my case)

[mozfet]

This is an autoform issue. I tried many different ways to solve this, none of them worked, they just made it worse

How many levels deep are you going?

[mozfet]

Here's the situation:

1. Add/Remove an item from an array
2. It triggers autoform array tracker for all arrays in the array
3. It triggers a value change for all fields in the array
4. Autoform redraws the form on each trigger (76 times in my case)

I will check it out in the playground later this week.

[pouya-eghbali]

How many levels deep are you going?

3 levels

I will check it out in the playground later this week.

Pretty sure this is related: Meteor-Community-Packages/meteor-autoform#638
I honestly don't want to fork and maintain autoform [at least not rn], but if that's the only solution...

[pouya-eghbali]

I started investigating this in autoform source, seems like it is caused by arrayTracker.getVisibleCount and arrayTracker.getField

[pouya-eghbali]

This isn't directly caused by arrayTracker, but caused by helpers that use the above methods:

• afArrayFieldHasMoreThanMinimum and afArrayFieldHasLessThanMaximum are the only helpers using arrayTracker.getVisibleCount
• innerContext helper in afEachArrayItem is the only helper using arrayTracker.getField

afEachArrayItem gets called 183 times in my case.

[pouya-eghbali]

in innerContext helper we have:

arrayTracker.initField(formId, name, ss, docCount, c.atts.minCount, c.atts.maxCount);    
return arrayTracker.getField(formId, name);

first one does a deps.changed() and the second one depends on this deps. Maybe this is causing the issue? Making a cache for initField and preventing further deps.changed() calls to see if it works...

[pouya-eghbali]

No effect, something is resetting ALL arrayTracker data.

[pouya-eghbali]

Tracing back to mozfet:meteor-autoform-materialize, the afArrayField_materialize template is rendered X times, investigating what causes that

[pouya-eghbali]

Tracing back to autoform again, Template.afFormGroup.innerContext is called >1500 times on array item add/remove. I need to fix this otherwise nested arrays aren't usable, it takes ~14 seconds in my case for the UI to become responsive after adding or removing an item from the list. Not even sure if this is some bug in mozfet:meteor-autoform-materialize or in autoform, in any case opening a new issue on autoform github page won't do us any help...

[pouya-eghbali]

Tracing back to quickForm_materialize, this template is only rendered once, but in this template afQuickField is rendered +1000 times on item add/remove. Investigating

[pouya-eghbali]

Tracing back to autoform, the afOptionsFromSchema helper is causing reactivity/rerenders in afQuickField, it's getting called for ALL subfields in array.

[pouya-eghbali]

I'm going to stop working on this for a while, I'm a little tired of trying to solve it all day everyday. I need to clear my mind. Apparently afOptionsFromSchema isn't even reactive. I'm seeing these calls only because we use it in afArrayField_materialize and it gets re-rendered a lot.

[pouya-eghbali]

Further investigation:

• adding an item causes afFormGroup rendering for the new item
• after rendering the new item is finished, afFormGroup re-renders all array items

[mozfet]

This is heavy stuff, I know. Thank you for your effort so far. Let me play with it until next week Wed.

[pouya-eghbali]

This may be the issue:

• afEachArrayItem calls arrayTracker.getField
• arrayTracker.getField depends on arrayTracker.info[formId][field].deps
• arrayTracker.addOneToField (called by .autoform-add-item) triggers arrayTracker.info[formId][field].deps.changed()

This means, each time we add an item it triggers a re-render for afEachArrayItem and when this is the outermost array in our nested arrays it means a re-render of all fields. But still, this doesn't explain why the form gets re-rendered several times. If I observe the dom I can see the field id getting changed 7-8 times.

[pouya-eghbali]

Ok this isn't related to mozfet:meteor-autoform-materialize

For references: Meteor-Community-Packages/meteor-autoform#795

[pouya-eghbali]

Some more clues:

1. arrayTracker.addOneToField calls self.info[formId][field].deps.changed();
2. None of the self.info[formId][field].deps.depend();s in arrayTracker cause reactivity except arrayTracker.getField
3. afEachArrayItem.helpers.innerContext is the only helper using arrayTracker.getField

[pouya-eghbali]

Maybe I can solve this by providing a afEachUnrenderedArrayItem

[pouya-eghbali]

Some hints for a solution: http://blazejs.org/api/spacebars.html#Reactivity-Model-for-Each

[pouya-eghbali]

More clues: meteor/blaze#241

[pouya-eghbali]

I solved the issue :)

[pouya-eghbali]

Sorry for the amount of comments I made, I was just logging what I'm doing so if in the future someone wants to work on this they have a clue where to start and what we already tried. Anyways, I forked autoform and I have my changes here: https://github.com/pouya-eghbali/meteor-autoform

I'm not going to make a pull request, there are 22 open pull requests and 530 open issues on the original repo, this issue was posted there a few years ago and nobody cared to solve it, or nobody figured how to, well I guess I have to maintain my own copy of autoform.

So, what was the issue? It's actually, simple. Autoform maintains a list (a JavaScript Array) of items for each array field, in its ArrayTracker. In afEachArrayItem, it calls arrayTracker.getField which returns this list of items. Each time an item is added to the list or removed from it, it forces arrayTracker.getField to re-run and that causes a re-render for afEachArrayItem.

Checking the source code of afEachArrayItem, it is using a Blaze #each block. Checking Blaze docs and issues posted on the repository, I figured there are issues with their diffing function. The #each block, tries to figure out if items are new/modified or they're the same as before, if it recognizes an item is changed it does a re-render for that item.

Problem is, this diffing function doesn't work well with arrays of objects, so when the arrayTracker.getField returns the new list of items and afEachArrayItem passes this array to #each block, the diffing function fails to recognize the items aren't modified and causes a re-render of all fields.

Reading the Blaze documentation and issues, I figured the diffing function works really well with mongo queries and cursors. So, I modified autoform ArrayTracker and added a new Mongo.Collection(null) for each tracked array field. Next I modified add/remove/getField and some other functions in the tracker to use this collection instead of just a JavaScript Array. By passing a cursor to the #each block instead of the array, Blaze correctly recognizes that the items aren't changed and a re-render isn't necessary.

This solves a lot of performance issues and makes the forms more responsive. It also makes form submits faster. So, now, there are two ways to put this in autoform. First one is forking/cloning and maintaining an independent copy of autoform. Second one is monkey patching arrayTracker.

I'm against the second one because it's a little hacky. I'm planning to go the first way, because autoform isn't getting any updates from aldeed anyways, at least this way we can fix the issues we may face in the future.

This solves the following issues:

• Collapsible Nested Arrays #71 (comment)
• "unresponsive script" warning / performance of larger forms Meteor-Community-Packages/meteor-autoform#795
• Update-form elements has rendered 5 times? Meteor-Community-Packages/meteor-autoform#638

And possibly others.

[mozfet]

I am only now getting the time to look at this properly, and need it for a project as well. I was noticing AutoForm going stale - too bad. I use AutoForm a lot and will offer to try and help maintain on your fork as well.

I will test your array work today and give feedback.

[mozfet]

I notice an issue with Text fields inside arrays. They are not showing their labels... and typing in them does not update the summary in the collapsible header (only the first letter). This happens with original AutoForm, as well as the fork.

new SimpleSchema({
  labelString: {
    type: String,
    label: 'I am a label.',
    optional: true
  },

  placeholderString: {
    type: String,
    optional: true,
    label: 'I am also a label.',
    autoform: {
     placeholder: 'I am a placeholder.'
    }
  },

  arrayOfStrings: {
    type: Array,
    minCount: 1,
    maxCount: 3,
    optional: true  // but, but, with a min or max count it aint optional?
  },
  'arrayOfStrings.$': {
    type: String,
    label: 'A string field with placeholder',
    autoform: {
     placeholder: 'Placeholder'
    }
  }
}

[mozfet]

I notice an issue with Text fields inside arrays. They are not showing their labels... and typing in them does not update the summary in the collapsible header (only the first letter). This happens with original AutoForm, as well as the fork.

new SimpleSchema({
  labelString: {
    type: String,
    label: 'I am a label.',
    optional: true
  },

  placeholderString: {
    type: String,
    optional: true,
    label: 'I am also a label.',
    autoform: {
     placeholder: 'I am a placeholder.'
    }
  },

  arrayOfStrings: {
    type: Array,
    minCount: 1,
    maxCount: 3,
    optional: true  // but, but, with a min or max count it aint optional?
  },
  'arrayOfStrings.$': {
    type: String,
    label: 'A string field with placeholder',
    autoform: {
     placeholder: 'Placeholder'
    }
  }
}

Oops... Actually, that is not a bug ... its the same in the current version. Individual labels of an item of an array are shared by the collapsible header.

[pouya-eghbali]

Hi @mozfet
I cannot work on this right now, I can do a little tonight. I'm planning to make lts versions of some of the meteor packages I'm using in my projects and fix their bugs and maintain them.

[pouya-eghbali]

I detached the fork, it's independent now. I will look into the label issue tomorrow. I made 2 changes in autoform:

a) the array thing (to use minimongo collections instead of arrays for tracking)
b) a bug fix to wait for form context to be ready before we render the fields.

I'll fix bugs as I find them.

[mozfet]

I detached the fork, it's independent now. I will look into the label issue tomorrow. I made 2 changes in autoform:

a) the array thing (to use minimongo collections instead of arrays for tracking)
b) a bug fix to wait for form context to be ready before we render the fields.

I'll fix bugs as I find them.

Please see previous post... the label not showing for the text field is not a bug but per design, however the default label header of the collapsible does not update automatically for String (only first letter).

[pouya-eghbali]

I fixed the header issue in pouya-eghbali/meteor-autoform-materialize@d07e0dd