Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: update core schema v2 for alignments and mesh #273

Merged
merged 68 commits into from
Sep 27, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
f284234
add collection_metadata to template and update configs
Bento007 Sep 5, 2024
bb63951
add collection metadata to ingestion config
Bento007 Sep 5, 2024
99adb45
add collection metadata to ingest_config_schema
Bento007 Sep 5, 2024
a7b83cc
address comments
Bento007 Sep 5, 2024
b4319d5
address comments
Bento007 Sep 6, 2024
9aaf3f4
add extended validation check for collection metadata (#259)
daniel-ji Sep 6, 2024
fa05f4f
Merge branch 'refs/heads/main' into tsmith/997-ctf
Bento007 Sep 6, 2024
b4c611b
update schema
Bento007 Sep 6, 2024
87f7732
remove rawtilts from 10010.yaml
Bento007 Sep 6, 2024
7421f5d
lint
Bento007 Sep 7, 2024
bf3a2cb
Merge branch 'refs/heads/main' into tsmith/997-ctf-alignment
Bento007 Sep 9, 2024
35b05ee
update ingest schema
Bento007 Sep 9, 2024
991b9d7
update ingest config template
Bento007 Sep 9, 2024
06e1ea0
add AlignmentImporter
Bento007 Sep 9, 2024
e9b190d
lint
Bento007 Sep 9, 2024
8473693
correct offset format
Bento007 Sep 9, 2024
df622bb
update existing ingestion_configs
Bento007 Sep 9, 2024
49e9290
Add alignment to ingest_config validation
Bento007 Sep 11, 2024
ac69f5e
- add format, volume_dimensions to ingest config schema
Bento007 Sep 12, 2024
94f2c02
Merge remote-tracking branch 'origin/tsmith/997-ctf-alignment' into t…
Bento007 Sep 12, 2024
b92c420
add alignment metadata
Bento007 Sep 12, 2024
1e8fe98
fix format documentation
Bento007 Sep 12, 2024
afe9d0c
Update config for alignment
Bento007 Sep 13, 2024
f89290d
Update config for tomogrames and frames
Bento007 Sep 13, 2024
72208ba
update template and schema
Bento007 Sep 13, 2024
a48b4a4
update all configs
Bento007 Sep 13, 2024
796d647
Merge branch 'refs/heads/main' into tsmith/997-ctf-alignment
Bento007 Sep 13, 2024
97ab3c9
add docstring to migrate script
Bento007 Sep 13, 2024
3ec3c58
run linter
Bento007 Sep 13, 2024
517df5c
removed extra changes and run schema gen
Bento007 Sep 13, 2024
2a2ca56
lint
Bento007 Sep 13, 2024
af75836
move mdoc_name to collection_metadata
Bento007 Sep 13, 2024
125fce7
extract txt and csv to from rawtilts to alignment
Bento007 Sep 14, 2024
f91a1e7
add correct dates
Bento007 Sep 16, 2024
a5bfe46
added is_portal_standard
Bento007 Sep 16, 2024
32a1523
undo v2.0 changes
Bento007 Sep 16, 2024
f978bca
Apply suggestions from code review
Bento007 Sep 16, 2024
6244d11
add affinity matrix back to tomograms
Bento007 Sep 16, 2024
b8a8514
add affinity matrix back to tomograms
Bento007 Sep 16, 2024
8fe2d77
update core schema v2.0.0 for alignments and mesh
Bento007 Sep 17, 2024
2e58482
add identity matrix back in
Bento007 Sep 21, 2024
465893c
remove voxel spacing from parent filter of alignment
Bento007 Sep 21, 2024
54c811f
lock the verison of linkml and regenerate files
Bento007 Sep 21, 2024
90734f1
add xtlt and copy to transform_ingestion_configs.py
Bento007 Sep 23, 2024
a3394fb
lint
Bento007 Sep 23, 2024
e2072d8
iron out tilt formats
Bento007 Sep 24, 2024
2096a74
iron out tilt formats
Bento007 Sep 24, 2024
4b627a2
update schema
Bento007 Sep 24, 2024
05c4769
Merge branch 'refs/heads/main' into tsmith/997-ctf-alignment
Bento007 Sep 24, 2024
61c75e9
merge main
Bento007 Sep 24, 2024
ffafd79
update transform_ingestion_configs.py
Bento007 Sep 24, 2024
9c622b7
Merge branch 'refs/heads/tsmith/997-ctf-alignment' into tsmith/997-ct…
Bento007 Sep 25, 2024
e7d80f2
regenerate schema files
Bento007 Sep 25, 2024
326f227
flatten code in ingest_997.py add dosctring to transform_ingestion_co…
Bento007 Sep 25, 2024
4048d79
remove tomogram from alignment in ingest-config
Bento007 Sep 26, 2024
10fac28
Merge branch 'refs/heads/main' into tsmith/997-ctf-alignment
Bento007 Sep 26, 2024
dded1d1
move migrate script to ingest-tools
Bento007 Sep 26, 2024
918ed8e
lint
Bento007 Sep 26, 2024
6912e06
update configs
Bento007 Sep 26, 2024
15f0309
liknt
Bento007 Sep 26, 2024
aacf325
remove is_portal_standard
Bento007 Sep 26, 2024
80baa95
Merge branch 'refs/heads/tsmith/997-ctf-alignment' into tsmith/997-ct…
Bento007 Sep 26, 2024
48b33cb
remove is_portal_standard
Bento007 Sep 26, 2024
2a45a31
linkml
Bento007 Sep 26, 2024
5eb546b
lint
Bento007 Sep 26, 2024
54a61c4
Merge branch 'refs/heads/main' into tsmith/997-ctf-corev2
Bento007 Sep 27, 2024
7404d38
run linkml
Bento007 Sep 27, 2024
f1b8e9e
lint
Bento007 Sep 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 15 additions & 15 deletions ingestion_tools/dataset_configs/10000.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ annotations:
columns: xyz
file_format: csv
glob_string: particle_lists/{run_name}_fas.csv
is_portal_standard: false

is_visualization_default: false
- metadata:
annotation_method: Cumulative template-matching trained 2D CNN predictions + visual
Expand All @@ -85,7 +85,7 @@ annotations:
columns: xyz
file_format: csv
glob_string: particle_lists/{run_name}_cyto_ribosomes.csv
is_portal_standard: false

is_visualization_default: true
- metadata:
annotation_method: spectrum equalization filter + 2D CNN prediction + manual correction
Expand All @@ -106,7 +106,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false

is_visualization_default: false
mask_label: 1
- metadata:
Expand All @@ -128,7 +128,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false

is_visualization_default: true
mask_label: 2
- metadata:
Expand All @@ -150,7 +150,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false

is_visualization_default: false
mask_label: 3
- metadata:
Expand All @@ -172,7 +172,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false

is_visualization_default: true
mask_label: 4
- metadata:
Expand All @@ -194,7 +194,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false

is_visualization_default: true
mask_label: 5
- metadata:
Expand All @@ -216,7 +216,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false

is_visualization_default: true
mask_label: 6
- metadata:
Expand All @@ -238,7 +238,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false

is_visualization_default: false
mask_label: 7
- metadata:
Expand All @@ -260,7 +260,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false

is_visualization_default: false
mask_label: 8
- metadata:
Expand All @@ -282,7 +282,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false

is_visualization_default: false
mask_label: 9
- metadata:
Expand All @@ -304,7 +304,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false

is_visualization_default: true
mask_label: 10
- metadata:
Expand All @@ -326,7 +326,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false

is_visualization_default: false
mask_label: 11
- metadata:
Expand All @@ -348,7 +348,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false

is_visualization_default: false
mask_label: 13
- metadata:
Expand All @@ -370,7 +370,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_membranes.mrc
is_portal_standard: false

is_visualization_default: false
mask_label: 1
collection_metadata:
Expand Down
30 changes: 15 additions & 15 deletions ingestion_tools/dataset_configs/10001.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ annotations:
columns: xyz
file_format: csv
glob_string: particle_lists/{run_name}_fas.csv
is_portal_standard: false

is_visualization_default: false
- metadata:
annotation_method: Cumulative template-matching trained 2D CNN predictions + visual
Expand All @@ -85,7 +85,7 @@ annotations:
columns: xyz
file_format: csv
glob_string: particle_lists/{run_name}_cyto_ribosomes.csv
is_portal_standard: false

is_visualization_default: true
- metadata:
annotation_method: manual segmentation
Expand All @@ -105,7 +105,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false

is_visualization_default: false
mask_label: 1
- metadata:
Expand All @@ -126,7 +126,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false

is_visualization_default: true
mask_label: 2
- metadata:
Expand All @@ -147,7 +147,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false

is_visualization_default: false
mask_label: 3
- metadata:
Expand All @@ -168,7 +168,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false

is_visualization_default: true
mask_label: 4
- metadata:
Expand All @@ -189,7 +189,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false

is_visualization_default: true
mask_label: 5
- metadata:
Expand All @@ -210,7 +210,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false

is_visualization_default: true
mask_label: 6
- metadata:
Expand All @@ -231,7 +231,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false

is_visualization_default: false
mask_label: 7
- metadata:
Expand All @@ -252,7 +252,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false

is_visualization_default: false
mask_label: 8
- metadata:
Expand All @@ -273,7 +273,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false

is_visualization_default: false
mask_label: 9
- metadata:
Expand All @@ -294,7 +294,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false

is_visualization_default: true
mask_label: 10
- metadata:
Expand All @@ -315,7 +315,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false

is_visualization_default: false
mask_label: 11
- metadata:
Expand All @@ -336,7 +336,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false

is_visualization_default: false
mask_label: 13
- metadata:
Expand All @@ -358,7 +358,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_membranes.mrc
is_portal_standard: false

is_visualization_default: false
mask_label: 1
collection_metadata:
Expand Down
4 changes: 2 additions & 2 deletions ingestion_tools/dataset_configs/10002.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_actin_ground_truth.mrc
is_portal_standard: false

is_visualization_default: true
- metadata:
annotation_method: 2D CNN + 3D CNN hyperparameter tuning
Expand All @@ -99,7 +99,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: prediction/{run_name}_*.mrc
is_portal_standard: false

is_visualization_default: false
collection_metadata:
- sources:
Expand Down
2 changes: 1 addition & 1 deletion ingestion_tools/dataset_configs/10003.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ annotations:
file_format: relion3_star
filter_value: '{mapped_tomo_name}.tomostar'
glob_string: run_ct17_it034_data.star
is_portal_standard: false

is_visualization_default: true
order: xyz
dataset_keyphotos:
Expand Down
2 changes: 1 addition & 1 deletion ingestion_tools/dataset_configs/10004.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ annotations:
columns: xyz
file_format: csv_with_header
glob_string: '{run_name}/annoRibosome_bin4.txt'
is_portal_standard: false

is_visualization_default: true
collection_metadata:
- sources:
Expand Down
4 changes: 2 additions & 2 deletions ingestion_tools/dataset_configs/10005.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ annotations:
delimiter: ' '
file_format: csv
glob_string: coorinates_dmt48/s{run_name}_dmt48_coordinates_bin2.txt
is_portal_standard: false

is_visualization_default: true
- metadata:
annotation_method: Template matching + manual filtering + 3D classification filtering
Expand All @@ -51,7 +51,7 @@ annotations:
delimiter: ' '
file_format: csv
glob_string: coordinates_dmt96/s{run_name}_dmt96_coordinates_bin2.txt
is_portal_standard: false

is_visualization_default: true
dataset_keyphotos:
- sources:
Expand Down
2 changes: 1 addition & 1 deletion ingestion_tools/dataset_configs/10006.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ annotations:
file_format: relion4_star
filter_value: '{run_name}'
glob_string: motl_star/czii-remapped-points.star
is_portal_standard: false

is_visualization_default: true
order: xyz
dataset_keyphotos:
Expand Down
2 changes: 1 addition & 1 deletion ingestion_tools/dataset_configs/10007.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ annotations:
binning: 4
file_format: tomoman_relion_star
glob_string: '{run_name}/metadata/particles/*.star'
is_portal_standard: false

is_visualization_default: true
order: xyz
collection_metadata:
Expand Down
4 changes: 2 additions & 2 deletions ingestion_tools/dataset_configs/10008.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ annotations:
file_format: stopgap_star
filter_value: '{run_name}'
glob_string: subtomo/hdcr/hdcr_tm_bin2_motl_12.star
is_portal_standard: false

is_visualization_default: true
order: xyz
- metadata:
Expand All @@ -76,7 +76,7 @@ annotations:
file_format: stopgap_star
filter_value: '{run_name}'
glob_string: subtomo/ribo/ribo_tm_bin1_motl_12.star
is_portal_standard: false

is_visualization_default: true
order: xyz
collection_metadata:
Expand Down
Loading
Loading