Model updates, and some conversion logic #123
Conversation
sherwoodf
commented
Jul 12, 2024
•
sherwoodf
commented
- Model updates from discussions with team. Mostly minor name changes, removing/adding fields that were dulpicates/i thought were duplicates but actually are not. Also created separate signal_channel_information object to hold that information within the image acquisition.
- Created conversion logic for Specimen Preparation Protocol, Specimen Growth Protocol, Image Acquisition. Removed some unused code from the test/utils.py file in the ingest shared models package (which is why the diff is so big for that file)
- Consolidated all the persist artefact logic used in multiple places in the ingest package into a util function. Did not use this for the study as that has more complex logic.
…ields will actually be generated endpoints
…sition conversion logic
| @@ -6,7 +6,8 @@ | |||
| get_generic_section_as_dict, | |||
| mattributes_to_dict, | |||
| dict_to_uuid, | |||
| find_sections_recursive | |||
| find_sections_recursive, | |||
| persist | |||
There was a problem hiding this comment.
Didn't end up using this - so should remove the unused import (will do it in this PR if other changes are needed)
|
|
||
|
|
||
| def get_template_channel() -> semantic_models.Channel: |
There was a problem hiding this comment.
template methods were copied over from the shared_models code package, but aren't actually needed here, so i decided to remove them.
| @@ -0,0 +1,2 @@ | |||
| [ | |||
There was a problem hiding this comment.
I had to create this file to get the experimental_imaging_dataset test working. It was checking the Annotation Dataset, and couldn't find this file & was therefore complaining. I don't understand why it was checking this, but i wanted to move on with this PR without figuring out the EID code right now.
There was a problem hiding this comment.
This was needed because the function biostudies.find_file_lists_in_submission goes through all file lists in the submissions when trying to create file references. Since a file list was added to the "Segmentation masks" "Annotations" section in data/S-BIADTEST.json, find_file_lists_in_submission was looking for the file list ...
…ared models, and fixed file_name -> file_path as per model change for file references
bia-ingest-shared-models/bia_ingest_sm/conversion/annotation_method.py
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
I think we need to revisit the logic of this file in light of the fact that we are not storing the list of file_references anymore. We may have to trigger the generation of file_references after obtaining the uuid for the experimental dataset, so we can pass this to the function that creates file_references, allowing them to point to their parent expermental imaging dataset.
There was a problem hiding this comment.
With new approach (file_reference points to parent EID) we may have to re-write this function. (see comment on assignment of submission_dataset)
| if persist_artefacts: | ||
| file_dict["uuid"] = fileref_uuid | ||
| file_dict["uri"] = file_uri(submission.accno, f) | ||
| file_dict["submission_dataset"] = fileref_uuid |
There was a problem hiding this comment.
This was just a place holder - we need to pass the actual submission_dataset uuid (especially as this will now be the only link to its parent)
There was a problem hiding this comment.
I've not touched the file_reference code. That wasn't the intent of this PR.
There was a problem hiding this comment.
Ok - I have created a clickup ticket to fix this which is assigned to me.
bia-ingest-shared-models/bia_ingest_sm/conversion/image_acquisition.py
Outdated
Show resolved
Hide resolved
bia-ingest-shared-models/bia_ingest_sm/conversion/specimen_growth_protocol.py
Outdated
Show resolved
Hide resolved
bia-ingest-shared-models/bia_ingest_sm/conversion/specimen_imaging_protocol.py
Outdated
Show resolved
Hide resolved
bia-ingest-shared-models/bia_ingest_sm/conversion/specimen_imaging_protocol.py
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
I don't think re module is used
There was a problem hiding this comment.
It's used in:
def get_licence(study_attributes: Dict[str, Any]) -> semantic_models.LicenceType:
"""
Return enum version of licence of study
"""
licence = re.sub(r"\s", "_", study_attributes.get("License", "CC0"))
return semantic_models.LicenceType(licence)
There was a problem hiding this comment.
But i guess we've changed the enums now, so we don't need that?
There was a problem hiding this comment.
I missed this - no I think we still need it!
| # file extension | ||
|
|
||
| file_path: str = Field(description="""The path (including the name) of the file.""") | ||
| # TODO: Clarify if this should be biostudies 'type' or derived from file extension |
There was a problem hiding this comment.
Does this field relate to the biostudies 'type' - which is not really related to the extension of the file e.g. fire_object, directory etc as opposed to the file type derived from the file extension?
There was a problem hiding this comment.
It's not even that: it's sort of 3 different types that we care about mushed together. But i've cut a ticket to deal with that later.
|
|
||
|
|
||
| def get_template_channel() -> semantic_models.Channel: | ||
| return semantic_models.Channel.model_validate( | ||
| def get_test_specimen_growth_protocol() -> List[bia_data_model.ImageAcquisition]: |
There was a problem hiding this comment.
should this function be get_test_image_acquisition ? Or should it be deleted as there is a get_test_image_acquisition on line 221
* model updates to standardise names further and to account for which fields will actually be generated endpoints * model updates and added specimen growth, preparation, and image acquisition conversion logic * added logic to generate annotation method objects * tidied up imports * created empty annotation file list to make tests pass * moved file reference conversion to it's own file, fixed imports of shared models, and fixed file_name -> file_path as per model change for file references * updated models and ingest code
* model updates to standardise names further and to account for which fields will actually be generated endpoints * model updates and added specimen growth, preparation, and image acquisition conversion logic * added logic to generate annotation method objects * tidied up imports * created empty annotation file list to make tests pass * moved file reference conversion to it's own file, fixed imports of shared models, and fixed file_name -> file_path as per model change for file references * updated models and ingest code
* model updates to standardise names further and to account for which fields will actually be generated endpoints * model updates and added specimen growth, preparation, and image acquisition conversion logic * added logic to generate annotation method objects * tidied up imports * created empty annotation file list to make tests pass * moved file reference conversion to it's own file, fixed imports of shared models, and fixed file_name -> file_path as per model change for file references * updated models and ingest code
