Meta-populate

[CBroz1]

Problem

It's often the case that a pipeline will have a single parent table, like Session, that motivates the populate of a number of children for each data modality (e.g., TaskEpoch, VideoFile). We found that when someone encounters an error with one such table, they often want to rollback, edit the files, and start over.

Ideas

The way DataJoint pipelines often handle this is a pipeline.py script that
goes through each table and calls populate on them one at a time.

def populate_all('some_key'):
    for table in [Session, TaskEpoch, VideoFile]:
        table.populate('some_key')

To allow rollback, we created a function to wrap a set of populates in a single transaction. The user calls populate_all_common here, which runs as many populates as is can in a single transaction via this method, allowing the user to decide how they want to treat errors/rollback.

A future DataJoint function my allow...

dj.populate_all([Session, TaskEpoch, VideoFile], key='some_key', rollback_on_fail=True)

The less ambitious version would only allow siblings in the passed list of tables

[dimitri-yatsenko]

This is exactly the reason for the master-part relationship. In this case, TaskEpoch and VideoFile would be organized as part tables of Session. Is that feasable?

[CBroz1]

In our case, at least one such downstream table is itself a master. I don't think datajoint permits parts of parts. It has also made more sense for us to conceptually separate different modalities into different schemas.

While it might be feasible to declare a new pipeline with all such imports as parts of the same master, it's not a decision we can revisit at this point