Can jsonmigrator handle references at all?

[jean]

  Module Products.Archetypes.Field, line 1984, in set
   - __traceback_info__: (...)
  Module Products.Archetypes.ReferenceEngine, line 309, in addReference
ReferenceException: Invalid target UID

[plone-foundation]

Quoting Jean Jordaan (2014-03-29 11:48:29)

Module Products.Archetypes.Field, line 1984, in set

• traceback_info: (...)
  Module Products.Archetypes.ReferenceEngine, line 309, in addReference
  ReferenceException: Invalid target UID

as far as i can remember references were also migrated. not sur what the above
error would be.

Rok Garbas - http://www.garbas.si

[jean]

The above error is what happens when a document with references is encountered.
Sorry, I should have included the whole traceback, but thought that the above would look familiar.
In short, when creating the migrated document, set is called on the reference field with a list of UIDs. Since migrated documents will have different UIDs, and referenced documents may not have been migrated yet, I thought that references would break, unless there is a mechanism to keep track of them.

2014-03-29 11:36:52 INFO JSON Migrator Read item from http://127.0.0.1:26026/manuals/content/documents/climate-change/secretariat-for-the-pacific-community-best-practices/get_item
[...]
2014-03-29 11:36:54 INFO collective.simserver.core indexed 1 documents
2014-03-29 11:36:55 ERROR Zope.SiteErrorLog 1396093015.140.98324413935 http://localhost:38521/iwlearn/@@jsonmigrator-run
Traceback (innermost last):
  Module ZPublisher.Publish, line 138, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module plone.z3cform.layout, line 70, in __call__
  Module plone.z3cform.layout, line 54, in update
  Module z3c.form.form, line 210, in update
  Module z3c.form.action, line 99, in execute
  Module z3c.form.button, line 315, in __call__
  Module z3c.form.button, line 170, in __call__
  Module collective.jsonmigrator.helper, line 133, in handleRun
  Module collective.transmogrifier.transmogrifier, line 63, in __call__
  Module collective.jsonmigrator.mimetype, line 40, in __iter__
  Module collective.jsonmigrator.local_roles, line 39, in __iter__
  Module collective.jsonmigrator.owner, line 41, in __iter__
  Module collective.jsonmigrator.permissions, line 41, in __iter__
  Module collective.jsonmigrator.properties, line 43, in __iter__
  Module collective.jsonmigrator.workflowhistory, line 41, in __iter__
  Module collective.jsonmigrator.datafields, line 23, in __iter__
  Module plone.app.transmogrifier.uidupdater, line 34, in __iter__
  Module plone.app.transmogrifier.atschemaupdater, line 67, in __iter__
  Module zope.event, line 31, in notify
  Module zope.component.event, line 24, in dispatch
  Module zope.component._api, line 136, in subscribers
  Module zope.component.registry, line 321, in subscribers
  Module zope.interface.adapter, line 585, in subscribers
  Module zope.component.event, line 32, in objectEventNotify
  Module zope.component._api, line 136, in subscribers
  Module zope.component.registry, line 321, in subscribers
  Module zope.interface.adapter, line 585, in subscribers
  Module collective.simserver.core.utils, line 156, in index_and_relate
  Module Products.Archetypes.ClassGen, line 76, in generatedMutator
  Module Products.Archetypes.Field, line 1984, in set
   - __traceback_info__: (<ATFile at /iwlearn/content/documents/climate-change/secretariat-for-the-pacific-community-best-practices>, '99c75332408d4d27a206305c3ea58b91', ['410dee33639f93bc0f275f92462daea1', '99c75332408d4d27a206305c3ea58b91', '6dd36755c8eb43d2dd85232b85f5225a', '14749bc63d5472321ea7c77da4b41c58', '5107d15e3251cfd2165327fa8554f088', '83072d3e2aae35228d07d6f1b035bcb6', 'd946621be8c483df4931cabc5fe16238'], [])
  Module Products.Archetypes.ReferenceEngine, line 309, in addReference
ReferenceException: Invalid target UID

[jean]

Looks like it was collective.simserver getting in the way.
I disabled automatic indexing by simservrer, and migration continues.
Now it stops with the following traceback, and binary gobbledygook in the browser.
I suspect it may need to commit more often ..

2014-03-30 08:42:03 INFO JSON Migrator Stop importing profile: collective.jsonmigrator.catalog
2014-03-30 08:42:11 ERROR Zope.SiteErrorLog 1396168931.70.559170359242 http://localhost:38521/iwlearn/@@jsonmigrator-run
Traceback (innermost last):
  Module ZPublisher.Publish, line 146, in publish
  Module Zope2.App.startup, line 301, in commit
  Module transaction._manager, line 89, in commit
  Module transaction._transaction, line 329, in commit
  Module transaction._transaction, line 446, in _commitResources
  Module ZODB.Connection, line 781, in tpc_vote
  Module ZEO.ClientStorage, line 1097, in tpc_vote
  Module ZEO.ServerStub, line 249, in vote
  Module ZEO.zrpc.connection, line 768, in call
FileStorageError: description too long

[jean]

I added partialcommit to the pipeline (3e46d28, 782da88 in #4), and now the migration run completes without errors.

The resulting content is still incomplete, though. The content is File, PloneGlossary, and PloneHelpCenter objects. Files are listed, but return a 404 when browsing to one. The glossary and manual objects are there when browsing folder_contents, but views come up blank. I guess they need blueprints of their own.