IVS-235 - Bulk insert ModelInstances

[rw-bsi]

Requires DB constraint to be in place to prevent duplicates / FK constraint errors
(see PR 27)

[aothms]

I think the combination of ModelInstance and ValidationOutcome bulk_creation better be wrapped in a transaction.atomic? Just so that the chance of orphaned instances is reduced even more.

Also, I still don't really know how this now actually works with transactions across multiple tasks in parallel and the isolation level of the transactions. It would be good to at least conceptually understand when multiple tasks in parallel try to create the same instances simultaneously. Should we set the isolation level to Serializable?

[aothms]

Suggested change

	if outcomes_to_save:
	if outcomes_to_save and context.validation_task_id is not None:

Any reason why you removed the ... and context.validation_task_id is not None? The code is a bit spaghetti with all the operating modes, I guess it doesn't hurt to keep it in?

[rw-bsi]

it's at best redundant, worst it silently fails because of a failed assertion (has outcomes, but somehow task_id is lost?) and I prefer to bomb out loudly :-)

[rw-bsi]

I think the combination of ModelInstance and ValidationOutcome bulk_creation better be wrapped in a transaction.atomic? Just so that the chance of orphaned instances is reduced even more.

Also, I still don't really know how this now actually works with transactions across multiple tasks in parallel and the isolation level of the transactions. It would be good to at least conceptually understand when multiple tasks in parallel try to create the same instances simultaneously. Should we set the isolation level to Serializable?

There is a transaction.atomic block at line 109, no harm to push it down, but has same effect?

serializable is not required imo - each instance is linked to one or more outcomes, writing instances with ignore_conflicts everywhere prevents duplicates (by definition only from something running in parallel), and reading them all again makes sure we get current and previously created instances. Key is we do it everwhere the same: schema and gherkin_rules

ModelInstance.objects.bulk_create(outcomes_instances_to_save, ignore_conflicts=True) # ignore conflicts with existing
model_instances = dict(ModelInstance.objects.filter(model_id=model_id).values_list('stepfile_id', 'id')) # retrieve all