-
Notifications
You must be signed in to change notification settings - Fork 174
Imaging Database
HOME > SETUP > IMAGING DATABASE
- Install the imaging pipeline codebase
- Set up imaging insertion scripts
- Executing and Troubleshooting the imaging insertion scripts
- Imaging Uploader
- Queue Manager
- Email Notifications
- Visualization: BrainBrowser
- Quality Control within the Imaging Browser
- Anonymization
- Post-processing (CBRAIN)
LORIS' Imaging insertion pipeline is composed of several Perl-based scripts and configuration files, which are housed in the Loris-MRI git repo. With modifications, it can handle any modality of imaging data, and by default is designed for DICOM MRI data collected by a longitudinally-organized multisite study with a defined data acquisition protocol.
View recent schematic for the LORIS Imaging insertion pipeline flow
Each new instance of LORIS will require some customization in these scripts in order to run the Imaging insertion pipeline, including the Imaging Uploader. For projects with multiple modalities of imaging data, and/or heterogeneous data collection, further customization and development of these scripts will be required.
Formats: By default, all DICOMS are converted both to MINC and NIfTI formats by LORIS' imaging insertion pipeline. After installation, conversion to NIfTI can be disabled (or re-enabled) in the prod file via the variable $no_nii. (To disable, set $no_nii = 1;)
1) Install the imaging pipeline codebase
Please follow Readme in the Loris-MRI git repository.
Note: To enable Download and Visualization of imaging files, /data/$PROJ directory and subdirectories must be readable and executable by the Apache linux user.
CentOS installations: please also see CentOS Imaging installation transcript
Troubleshooting: Note the configuration details described in the Loris-MRI Readme. Verify if your /data/* subdirectories were created by the imaging install script - if not, it may be due to root:root ownership of the /data/ mount on your system. Ensure these subdirectory structures are created manually (particularly subdirectories /data/$PROJ/data/* /data/$PROJ/bin/mri/ and /data/incoming/), and that your /data/ mount is executable.
Before attempting to run these Perl-based scripts on raw DICOM files, first configure LORIS for the imaging pipeline:
-
-
The MRI_alias field must be populated for each site contributing scans for its candidates
-
Populate with one entry for every (type of) acquisition in the study protocol. Ranges of values (e.g. TE) are permitted. Scan_type column must match id from mri_scan_type table (e.g. t1=44), which was populated by default (fmri/bold entries should be added). Note: Do not include hyphens, spaces or periods in your mri_scan_type.Scan_type column values.
- Note as of Loris 16.1, an option exists to force-load an MRI acquisition for a specified scan type, regardless of defined protocol checks. See Loris-MRI PR#141 for more information.
-
Since the MINC insertion script checks the visit label against the Visit_Windows table, please ensure this table is fully populated. Please see Loris Setup: Study Variables
-
-
Modify the imaging pipeline configuration file called /data/$PROJ/bin/mri/dicom-archive/.loris_mri/prod :
In the subroutine isFileToBeRegisteredGivenProtocol, modify the if statement to include all modalities. Labels must match Scan_type values in the mri_scan_type table:
if($acquisitionProtocol eq 't1' or $acquisitionProtocol eq 't2' or $acquisitionProtocol eq 'dti' or $acquisitionProtocol eq 'bold' or $acquisitionProtocol =~ /fmri/) { return 1; } -
In the Configuration module, section "DICOM Archive" settings, enter regex patterns which will detect and mask identifying information in the PatientID and PatientName DICOM header fields, when displayed in the DICOM Archive module. Additional regex fields in this Configuration section enable identification of living and Lego phantom scans.
Note: Host and main project URL fields in Configuration module (within the "WWW" settings section) may default to localhost, requiring modification.
For projects using the Imaging Uploader front-end module, see Imaging Uploader section on configuring and running the Imaging pipeline together with the Imaging Uploader. Note the Imaging pre-processing and insertion pipeline has been designed by default for MRI, and assumes raw DICOM data insertion.
As of Loris 16.1, a single script is provided to run the imaging pre-processing and insertion pipeline sequence in one step (details: MRI-PR#133)
cd /data/$PROJ/bin/mri
./batch_uploads_imageuploader -profile prod < imageuploader_list >log_batch_imageuploader.txt 2>&1 The input file imageuploader_list is a text file and must be structured as one line per dataset (example for 2 entries shown below). Each line (for each scan) consists of space delimited 1) the full path to the DICOM zipped scan, 2) Y or N depending on whether the scan is for a phantom or not, and 3) patient name following the PSCID_CandID_VisitLabel Loris convention for real candidates and left BLANK for phantoms.
An example of uploading 2 entries/scans to be uploaded (one human subject and one phantom):
/data/incoming/PSC0001_123457_V1.tar.gz N PSC0000_123456_V1
/data/incoming/Lego_Phantom_MNI_20140101.zip Y
Projects wishing to customize the execution of pipeline steps can run the following scripts in sequence:
See also 2015 Imaging Pipeline Flow Schematic and older UML diagrams of MRI and DICOM tar pipeline
- dicomTar.pl: Tars DICOMs into /data/$PROJ/data/tarchive and loads the tarchive table with DICOM header data.
E.g. Sample command to load a pre-registered subject visit from site DCC, given a DICOM dataset organized under $IncomingPath/PSCID_DCCID_VisitLabel :
cd /data/$PROJ/bin/mri
dicom-archive/dicomTar.pl /data/$PROJ/data/incoming/DCC0001_513066_V01 /data/$PROJ/data/tarchive -verbose -clobber -database -profile prod -centerName DCC -mri_upload_update > yourLogFile1.txt 2>&1 - tarchive_loader script (or batch_uploads_tarchive): loads imaging datasets, runs minc conversion and stores header information into the files and parameter_file tables; Creates candidate sessions if none exists.
cd /data/$PROJ/bin/mri
uploadNeuroDB/tarchiveLoader -globLocation -profile prod /data/$PROJ/data/tarchive/DCM_2009-03-19_DCC0001_513066_V01.tar > _yourLogFile2.txt_ 2>&1Alternatively, the batch_uploads_tarchive script may be used to process multiple dicom tarchives at a time, managed serially or by a queue manager such as qsub. Note the batch_uploads_tarchive script requires an input file stream containing a list of just tarchive names (DCM*.tar) :
cd /data/$PROJ/bin/mri
./batch_uploads_tarchive < $file_of_tar_names.txtOnce a scan session has been uploaded, verify in the Imaging Browser's View Session page that a jpg showing 3 slice orientations displays properly; if not, verify your permissions and restart apache:
sudo chmod o+r /data/$PROJ/bin
sudo chmod o+r /data/$PROJ/data
sudo service apache2 restartError and output messages from the imaging insertion scripts are logged in files created under the /data/$PROJ/data/logs/ directory. To view messages from the last script run, consult the most recent log file modified in this directory. These log files reference an uploadID used to identify each imaging dataset -- consult the mri_upload database table to look up which uploadID has been assigned to your scans.
Caveat When the imaging insertion pipeline is auto-launched by the Imaging Uploader module, the pipeline scripts' log files are output to /tmp/ and deleted. To avoid deletion, edit the Server Processes Manager function deleteProcessFiles() to return false instead of true.
When the need arises to re-load imaging data in LORIS, it is not sufficient to just re-run batch_uploads_tarchive. The pipeline must be re-run from dicomTar.pl. Ensure entries from the previous attempt to load the dataset have been removed from the following database tables:
- parameter_file
- tarchive
- mri_acquisition_dates
- files (best to delete from this table last)
- session - not recommended - only if necessary, and only if no other data is associated to this session e.g. on the Behavioural side of Loris.
It is also recommended to remove from the tarchive directory the last generated *.tar package for this dataset.
For backing up, re-labelling and re-loading MRI datasets with QC information, see Beta Tutorial
In cases where a subject was scanned in two scanner sessions within a single study Timepoint, both datasets should be loaded and associated to the same visit label / session table record. Create separate tarchives for each DICOM dataset upload each to the same visit.
The Imaging Uploader provides a user-friendly interface for transferring an imaging dataset to the Loris server, before it is handled by the imaging pre-processing and insertion pipeline scripts.
Note that once the data is uploaded onto your Loris server, the Imaging Uploader will not by default automatically launch the imaging pipeline scripts to preprocess, validate and insert the data into the database. The Imaging AutoLaunch feature can be used to automatically run these imaging pipeline scripts, or a cronjob can be used to run the scripts.
For initial setup and configuration, it is recommended to launch the imaging pipeline via the back-end for each uploaded dataset. Note that your lorisadmin user must also be part of the apache group (e.g. www-data).
To use the Imaging Uploader (formerly MRI Upload) module for upload and insertion of scans via a browser, first please follow the Imaging Uploader Readme. Please note this file is stored under the modules/imaging_uploader directory in the Loris codebase.
Second, in the Configuration module in the Loris front end:
Study section: Set ImagingUploader Auto-Launch as desired. For initial data upload troubleshooting, it is set to "No" by default.
Paths section:
- LORIS-MRI code : /data/$PROJ/bin/mri/
- MRI-Upload Directory : /data/incoming/
- Images /data/$PROJ/data/
Third, note carefully the configuration settings for apache, unix user accounts, and directories described in Loris-MRI Installation Readme. After any modifications, ensure you restart apache.
Missing visit label options? The Imaging Uploader's Visit label options are drawn from the list of all timepoints registered in the database where CenterID != 1 (this CenterID is reserved for DCC candidates). If you do not see a particular visit label option in the Uploader's dropdown select, simply create a new timepoint for any (non-DCC) candidate with that visit label (via Candidate menu, Access Profiles). The visit label should then automatically appear in the Uploader's dropdown options.
After an imaging dataset is uploaded to the Loris server, run the script imaging_upload_file.pl to run the pre-processing and insertion pipeline scripts that load the imaging data into the Loris database tables. Provide the upload_id value and uploaded dataset name (e.g. 608, AAA0001_513067_V01.zip):
cd /data/$PROJ/bin/mri
uploadNeuroDB/imaging_upload_file.pl -profile prod -verbose -upload_id 608 /data/incoming/AAA0001_513067_V01.zip See also: Logs
If upload was successful but issues were encountered with the imaging insertion pipeline scripts:
- CentOS: check for additional dependencies/configurations (e.g. Dicom Dictionary path) in the detailed [CentOS Imaging Installation transcript],
- Manually re-run the entire pipeline sequence using imaging_upload_file.pl
- For further information on the scripts also see Imaging Insertion Pipeline Troubleshooting steps.
- If one of the final steps such as the MINC conversion is failing, you may wish to just re-run the tarchiveLoader script.
- See also re-running the Imaging pipeline section for troubleshooting information.
Note: This feature is in Beta
The Configuration module (Study section) features a "Imaging Uploader Autolaunch" setting which is set to No for new installation of Loris, so that initial pipeline setup and troubleshooting can be supervised manually. Once the pipeline is configured and running smoothly, this setting may be used to run the Imaging Upload and insertion pipeline in one step, with a few additional modifications.
The Server Processes Manager module (Admin menu) shows all server jobs launched by Imaging Uploader to run the imaging insertion pipeline on an uploaded MRI dataset. Paths to the Stdout and Stderr log files and Exit Code file generated by each job are indicated. The Exit Code file contains a descriptive message regarding the exit status of the job.
By default these log files are output to /tmp/ and deleted. To avoid deletion, edit deleteProcessFiles() to return false. (See also: Logs).
Installing Sun GridEngine (SGE) is useful for managing the server processing load for all scripts.
Installing a mail server is recommended, as the LORIS Imaging pipeline by default attempts to send notifications about completed or failed uploads.
7) Visualization: BrainBrowser
BrainBrowser comes embedded within LORIS.
Note: /data/$PROJ directory and subdirectories must be readable and executable by the Apache linux user.
-
Verify in the Configuration module under "Path settings" that the "Path to MINC files" has been configured correctly. By default this path is set to /data/$PROJ/data/ - set during the imaging_install.sh run. MINC Apache’s path configuration for mincs should be verified as well in the Configuration module. There may be compatibility issue with older graphics cards.
-
A web-enabled viewer for real-time exploration of 3D images.
Note: If database queries appear, image volumes will not display properly in the Imaging Browser.
The Imaging Browser enables web-based Quality Control (QC) of imaging data, viewable in 3D space for ease of examination. QC flags, comments and statistics are fully integrated and can be enabled by: Grant QC permissions in User Accounts (or add 'mri_feedback' permission in user_perm_rel table). Scan types will be populated automatically once images are inserted in the database. QC comments should already be enabled via the feedback_mri_comment_types table.
All DICOM data transferred to the DCC must be free of any identifying information (e.g. patient name). A tool can be provided to the sites to facilitate anonymization. Please contact the LORIS team for details.
LORIS is agnostic to external processing pipelines. Contact us for more information about CBRAIN links.