-
Notifications
You must be signed in to change notification settings - Fork 174
Imaging Database
HOME > SETUP > IMAGING DATABASE
- Install the imaging pipeline codebase
- Set up imaging insertion scripts
- Executing and Troubleshooting the imaging insertion scripts
- Imaging Uploader
- Queue Manager
- Email Notifications
- Visualization: BrainBrowser
- Quality Control within the Imaging Browser
- Anonymization
- Post-processing (CBRAIN)
LORIS' Imaging insertion pipeline is composed of several Perl-based scripts and configuration files, which are housed in the Loris-MRI git repo.
View schematic for the LORIS Imaging insertion pipeline scripts
View step-by-step LORIS MRI Pipeline Flowchart
By default the pipeline is designed for raw DICOM MRI data, collected by a longitudinally-organized multisite study with a defined data acquisition protocol. With modifications and further customization, it can handle any modality of imaging data.
Formats: By default, all DICOMs are converted both to MINC and NIfTI formats by LORIS' imaging insertion pipeline.
After installation, use the Config module to setting to enable/disable automatic conversion to NIfTI.
1) Install the imaging pipeline codebase
Follow the Loris-MRI Readme.
Key configuration points to verify:
1. /data/* subdirectories were created by the imaging install script.
If not, it may be due to root:root ownership of the /data/ mount on your system.
Ensure these subdirectories are created manually, particularly:
/data/$PROJ/data/* /data/$PROJ/bin/mri/ and /data/incoming/
2. /data/$PROJ directory and subdirectories must be readable and executable by the Apache linux user. It may also help to ensure the /data/ mount is executable. After any modifications, ensure you restart apache.
-
Depending on your operating system, some dependencies might be missing. During initial troubleshooting of the imaging pipeline, note any related error messages (e.g.
install_driver(mysql) failed: Can't locate DBD/mysql.pm) and install missing packages as needed (e.g.sudo apt-get install libdbi-perl,sudo apt-get install libdbd-mysql-perl). -
If your MINC toolkit is older than 1.9.14 and your scans have no Date of Birth value, you may see an age unit error during DICOM to MINC conversion. Instructions for compiling a more recent version available on MNI-BIC GitHub.
-
-
The MRI_alias field must be populated for each site contributing scans for its candidates
-
Since the MINC insertion script checks the visit label against the Visit_Windows table, please ensure this table is fully populated. Please see Loris Setup: Visit Windows
3. ##### mri_protocol table
Populate with one entry for every (type of) acquisition in the study protocol. Ranges of values (e.g. TE) are permitted. Scan_type column must match id from mri_scan_type table (e.g. t1=44), which was populated by default (fmri/bold entries should be added). Note: Do not include hyphens, spaces or periods in your mri_scan_type.Scan_type column values.
##### Notes on Scan type identification
* By default, any scan will be inserted if it matches an _mri_protocol_ table entry.
* To force-load a specific MRI acquisition see (below) [bypassing protocol violation checks](Bypassing-protocol-violation-checks)
* To whitelist/blacklist specific scan types -- e.g. in case of protocol exclusion, case sensitivity or labelling variance -- modify _/data/$PROJ/bin/mri/dicom-archive/.loris_mri/prod_ subroutine: _isFileToBeRegisteredGivenProtocol()_, e.g.:
```
if($acquisitionProtocol eq 't1' or $acquisitionProtocol eq 't2' or $acquisitionProtocol eq 'dti' or $acquisitionProtocol eq 'bold' or $acquisitionProtocol =~ /fmri/) { return 1; }
```
4. ##### Configuration module
-
"DICOM Archive" section: Enter regex pattern to ensure identifying information in the PatientID or PatientHeader field is not displayed in the DICOM Archive module. Living and Lego Phantom regex fields in this section enable identification of these special scans.
-
"WWW" section: Verify your Host and main project URL config settings
-
"Paths" section: Verify LORIS-MRI code, MRI-Upload Directory, Images settings
-
"Imaging Pipeline" section: Verify all paths
-
"Study" section: Set ImagingUploader Auto-Launch to "Yes" only if you want files to be automatically inserted after they are uploaded to the server. For initial upload troubleshooting, it is recommended to leave the default "No".
The Imaging Uploader module provides a user-friendly interface for transferring an imaging dataset to the Loris server, before it is handled by the imaging pre-processing and insertion pipeline scripts.
To use the Imaging Uploader (formerly MRI Upload) module for upload and insertion of scans via a browser, first please follow the Imaging Uploader Readme. Please note this file is stored under the modules/imaging_uploader directory in the Loris codebase.
Missing visit label options? The Imaging Uploader's Visit label options are drawn from the list of all timepoints registered in the database where CenterID != 1 (this CenterID is reserved for DCC candidates). If you do not see a particular visit label option in the Uploader's dropdown select, simply create a new timepoint for any (non-DCC) candidate with that visit label (via Candidate menu, Access Profiles). The visit label should then automatically appear in the Uploader's dropdown options.
After an imaging dataset is uploaded to the Loris server, run the script imaging_upload_file.pl to run the pre-processing and insertion pipeline scripts that load the imaging data into the Loris database tables. Provide the upload_id value and uploaded dataset name (e.g. 608, AAA0001_513067_V01.zip):
cd /data/$PROJ/bin/mri
uploadNeuroDB/imaging_upload_file.pl -profile prod -verbose -upload_id 608 /data/incoming/AAA0001_513067_V01.zip See also: Logs
If upload was successful but issues were encountered with the imaging insertion pipeline scripts:
- CentOS: check for additional dependencies/configurations (e.g. Dicom Dictionary path) in the detailed [CentOS Imaging Installation transcript],
- Manually re-run the entire pipeline sequence: imaging_upload_file.pl
- If one of the final steps such as the MINC conversion is failing, you may wish to just re-run the tarchiveLoader script.
- See also re-running the Imaging pipeline section for troubleshooting information.
- To automatically preprocess, validate and insert all uploaded scans into the database, set ImagingUploader Auto-Launch to "Yes" in the
Configmodule, "Study" section. - For initial setup and configuration, it is recommended to manually run the imaging pipeline scripts for each uploaded dataset.
- Note that your lorisadmin user must also be part of the apache group (e.g. www-data).
The Server Processes Manager module (Admin menu) shows all server jobs launched by Imaging Uploader. The exact Output and Error file names for each upload/insertion are easily found in this module. The Exit Code file describes the exit status of the job.
Caveat: By default these log files are output to /tmp/ and deleted. To avoid deletion, edit deleteProcessFiles() to return false. (See also: Logs).
An alternative to the front-end Imaging Uploader is to load scans in LORIS via the back-end pipeline scripts. This is commonly used for automating the image insertion process. The pipeline can be launched in a single command, or by calling the constituent scripts in sequence:
The batch_uploads_imageuploader script runs in a single step the imaging pre-processing and insertion pipeline sequence (details: MRI-PR#133).
cd /data/$PROJ/bin/mri
./batch_uploads_imageuploader -profile prod < imageuploader_list >log_batch_imageuploader.txt 2>&1 The input file imageuploader_list must be a space-delimited text file, listing one line per dataset (see example below). Each line gives:
-
- the full path to the DICOM zipped scanset,
-
- Y or N depending on whether the scan is for a phantom or not
-
- PSCID_CandID_VisitLabel Loris convention for "patient name". Leave blank for phantoms.
An example imageuploader_list file for 2 entries/scans to be uploaded (one human subject and one phantom):
/data/incoming/PSC0001_123457_V1.tar.gz N PSC0000_123456_V1
/data/incoming/Lego_Phantom_MNI_20140101.zip Y
Projects wishing to customize the execution of pipeline steps can run the following scripts in sequence:
See also 2015 Imaging Pipeline Flow Schematic and older UML diagrams of MRI and DICOM tar pipeline
-
- dicomTar.pl: Tars DICOMs into /data/$PROJ/data/tarchive and loads the tarchive table with DICOM header data
E.g. Sample command to load a pre-registered subject visit from site DCC, given a DICOM dataset organized under $IncomingPath/PSCID_DCCID_VisitLabel/ :
cd /data/$PROJ/bin/mri
dicom-archive/dicomTar.pl /data/$PROJ/data/incoming/DCC0001_513066_V01 /data/$PROJ/data/tarchive -verbose -clobber -database -profile prod -centerName DCC -mri_upload_update > yourLogFile1.txt 2>&1 -
- tarchive_loader script (or batch_uploads_tarchive): loads imaging datasets, runs minc conversion and stores header information into the files and parameter_file tables; Creates candidate sessions if none exists.
cd /data/$PROJ/bin/mri uploadNeuroDB/tarchiveLoader -globLocation -profile prod /data/$PROJ/data/tarchive/DCM_2009-03-19_DCC0001_513066_V01.tar > yourLogFile2.txt 2>&1
### Verify loaded images and Troubleshooting
Once an MRI scan session has been inserted, you should see a link in the Imaging Browser main page to the dataset, as well as a link from the Dicom Archive module "View Images". If you cannot download Mincs or Dicoms, ensure that the /data/_$project_/data/assembly directory and subdirectories are executable.
Verify in the Imaging Browser's View Session page that a jpg showing 3 slice orientations displays properly; if not, verify your permissions and restart apache:
```bash
sudo chmod o+r /data/$PROJ/bin
sudo chmod o+r /data/$PROJ/data
sudo service apache2 restart
Error and output messages from the imaging insertion scripts are logged in files created under the /data/$PROJ/data/logs/ directory. To view messages from the last script run, consult the most recent log file modified in this directory. These log files reference an uploadID used to identify each imaging dataset -- consult the mri_upload database table to look up which uploadID has been assigned to your scans.
Caveat When the imaging insertion pipeline is auto-launched by the Imaging Uploader module, the pipeline scripts' log files are output to /tmp/ and deleted. To avoid deletion, edit the Server Processes Manager function deleteProcessFiles() to return false instead of true.
Scans whose parameters can't be matched against the mri_protocol table during the imaging insertion process, will be flagged as protocol violations and will not have their minc/nifti volumes loaded in the database. Review these in the front-end Mri Violations module (Imaging menu). The type of error (scan identification, protocol violation) will be listed.
See also notes on protocol checks and flagging of protocol violations in LORIS MRI Pipeline Flowchart
For cases when a scan has triggered a protocol violation, the minc volume can be force-loaded into LORIS by running:
uploadNeuroDB/minc_insertion.pl -acquisition_protocol t2w -bypass_extra_file_checks -create_minc_pics -profile prod -globLocation -force -tarchivePath _/data/project/dataTransfer/library/2009/DCM_2009-09-25_project_20110214_185904581.tar_ -mincPath _/data/project/data/trashbin/TarLoad-3-34-pVzGC5/xxx0067_703739_v12_20090925_222403_18e1_mri.mnc_Note carefully the following arguments:
-
acquisition_protocol : must be a known scan type according to the
mri_scan_typetable - tarchive_Path : the DICOM tarball
- mincPath : note this file may haven placed in the trashbin directory
See also: MRI-PR#141 for more examples.
When the need arises to re-load imaging data in LORIS, it is generally not sufficient to just re-run the minc/nifti loading step (tarchiveLoader or batch_uploads_tarchive). The pipeline steps must be re-run starting with dicomTar.pl.
In general, to re-load an imaging dataset through the pipeline from start (from dicomTar.pl) -- Ensure entries from the previous attempt to load the dataset have been removed from the following database tables:
- parameter_file
- tarchive
- mri_acquisition_dates
- files (best to delete from this table last)
- session - not recommended - only if necessary, and only if no other data is associated to this session e.g. on the Behavioural side of Loris.
It is also recommended to remove from the tarchive directory the last generated *.tar package for this dataset.
If any Quality Control flags or comments exist for these scans, you may also wish to delete specific records from files_qcstatus and the mri_feedback_* tables.
For backing up, re-labelling and re-loading MRI datasets with QC information, see Beta Tutorial
In cases where a subject was scanned in two scanner sessions within a single study Timepoint, both datasets should be loaded and associated to the same visit label / session table record. Create separate tarchives for each DICOM dataset upload each to the same visit.
Installing Sun GridEngine (SGE) is useful for managing the server processing load for all scripts. Use the Configuration module setting isqsub to tell the pipeline whether a queue manager is installed.
Installing a mail server is recommended, as the LORIS Imaging pipeline by default attempts to send notifications about completed or failed uploads.
7) Visualization: BrainBrowser
BrainBrowser comes embedded within LORIS.
Note: /data/$PROJ directory and subdirectories must be readable and executable by the Apache linux user.
-
Verify in the Configuration module under "Path settings" that the "Path to MINC files" has been configured correctly. By default this path is set to /data/$PROJ/data/ - set during the imaging_install.sh run. MINC Apache’s path configuration for mincs should be verified as well in the Configuration module. There may be compatibility issue with older graphics cards.
-
A web-enabled viewer for real-time exploration of 3D images.
Note: If database queries appear, image volumes will not display properly in the Imaging Browser.
The Imaging Browser enables web-based Quality Control (QC) of imaging data, viewable in 3D space for ease of examination. QC flags, comments and statistics are fully integrated and can be enabled by: Grant QC permissions in User Accounts (or add 'mri_feedback' permission in user_perm_rel table). Scan types will be populated automatically once images are inserted in the database. QC comments should already be enabled via the feedback_mri_comment_types table.
All DICOM data transferred to the DCC must be free of any identifying information (e.g. patient name). A tool can be provided to the sites to facilitate anonymization. Please contact the LORIS team for details.
LORIS is agnostic to external processing pipelines. Contact us for more information about CBRAIN links.