Simplify implementation of fmu.context

[perolavsvendsen]

Based on discussions 10/4-2024 @tnatt @jcrivenaes @HansKallekleiv

Current implementation/handling of context is cumbersome, and somewhat complicated (for many reasons).

Related issues/tasks:

• Include context in integration test #588
• Mismatch between pydantic model used for fmu.context and FmuContext enum #544
• standardize FMU contexts #293
• Basic support for using Everest #408 (comment)
• Improve how we signal "preprocessed" #593
• Deprecate fmu_context="case_symlink_realization" #615

The FMU block of the metadata

These are the principles

fmu:
  model: ... # always present
  case: ...  # always present
  iteration: ... # present if data belongs to an iteration/ensemble
  realization: ... # present if data belongs to a realization

  (aggregation: ... # present if data represents an aggregation)

There is implicit logic embedded in this, e.g. "if iteration and not realization then data is on iteration level". However, embedding such logic in all consumers would be cumbersome. Therefore, we mirror this logic in fmu.context:

  context: ...

Suggested tasks

• Enum allowed input to "context"
• Remove use of "context" for handling "preprocessed" or "symlink" logic
• Set context primarily based on detected context, rather than given context (goal: Generally not needed as an argument)

[tnatt]

I did a bit of digging to see how we treat the fmu_context="case_symlink_realization":

The option can only be used in ERT forward model setting.
If this option is given:

• export of data object and metadata to case/share
• create symlink to runpath/share for a realisation

In order to get this behaviour we run extraction of filepaths twice in the FiledataProvider :

1. one with mode fmu_context="case" to get paths that we store into the metadata as absolute_path / relative_path
2. one with mode fmu_context="realization" to get paths that we store into the metadata as absolute_path_symlink / relative_path_symlink

Then after the export to case/share we check if absolute_path_symlink is present in the metadata and create symlinks for the object and its metadata file to runpath/share

Main problems I see with this option is:

• multiple realisations writing to the same data object in case/share -- which can't be a stable solution
• it will set the fmu.context.stage="case_symlink_realization" which will not be pass schema validation and be rejected by sumo
• it is one of two options that prevents us from keeping fmu_context in sync with the schema definition Mismatch between pydantic model used for fmu.context and FmuContext enum #544

I highly recommend to remove this as a valid option to the fmu_context.
After speaking with @ezaezaeza, she confirmed that this option is not used in the sim2seis workflows. They export the seismic data as "preprocessed" and then copy it with an ERT pre-sim workflow to share/case, then they have an ERT forward_model to create the symlinks. Which in my opinion should be the recommended way 🙂

Any objections to removing this option @perolavsvendsen @jcrivenaes ?

[perolavsvendsen]

Surely the intention could never have been that multiple processes inside realizations should write to something under <case>/share?

Anyways, yes, sounds like it should definitely be removed as a valid option for fmu.context.

[tnatt]

No probably not 🙂 and if we were to continue with the option we could change to only export if it is not already present when running with this fmu_context. and we could change the fmu.context.stage in the metadata so that it is valid.
But I'd rather just remove it