Skip to content

Latest commit

 

History

History
131 lines (95 loc) · 5.92 KB

README.md

File metadata and controls

131 lines (95 loc) · 5.92 KB
Raw

NDS bioCADDIE Pilot

NDS bioCADDIE

This repository contains code for the NDS bioCADDIE pilot and submission to the 2016 bioCADDIE Dataset Retrieval Challenge. The pilot project explores expansion models for biomedical data search focusing on query expansion using external collections (e.g., PubMed, Wikipedia), document expansion, and document priors based on dataset source repository.

This repository includes:

Prerequisites

The pilot code in this repository requires the following software:

  • Indri 5.8 with JNI support (liblemur.so, liblemur_jni.so, libindri.so, libindri_jni.so)
  • Java 1.8 (JDK) with Maven
  • R

This submission relies on the ir-tools framework maintained by Miles Efron's lab at the University of Illinois at Urbana-Champaign.

You can either install prerequisites or use our provided Docker container.

Install prerequisites

The following instructions assume an Ubuntu system running as root user:

apt-get update
apt-get install openjdk-8-jdk-headless maven
apt-get install r-base
apt-get install build-essential git parallel vim wget zlibc zlib1g zlib1g-dev

Build and install Indri:

cd /usr/local/src
wget https://sourceforge.net/projects/lemur/files/lemur/indri-5.8/indri-5.8.tar.gz/download -O indri-5.8.tar.gz
tar xvfz indri-5.8.tar.gz
cd indri-5.8
./configure --enable-java --with-javahome=/usr/lib/jvm/java-8-openjdk-amd64
make 
make install

Build and install trec_eval:

cd /usr/local/src
wget http://trec.nist.gov/trec_eval/trec_eval_latest.tar.gz
tar xvfz trec_eval_latest.tar.gz
cd trec_eval.9.0/
make
make install

Run Docker image

Instead of installing the prerequisites on your system, the provided Docker image contains all of the required dependencies. The following example assumes that you've downloaded the BioCADDIE benchmark data to /data/biocaddie.

docker run -it -v /data/biocaddie:/data/biocaddie -v /data/pubmed:/data/pubmed ndslabs/indri bash

Clone this repository and build artifacts

Download and install the ir-tools and indri libraries (Note: we're working to add these to the Maven Central repository):

wget https://github.com/nds-org/biocaddie/releases/download/v0.1/ir-utils-0.0.1-SNAPSHOT.jar
mvn install:install-file -Dfile=ir-utils-0.0.1-SNAPSHOT.jar -DgroupId=edu.gslis -DartifactId=ir-utils -Dversion=0.0.1-SNAPSHOT -Dpackaging=jar
mvn install:install-file -Dfile=/usr/local/share/indri/indri.jar -DgroupId=indri -DartifactId=indri -Dversion=5.8 -Dpackaging=jar
cd ~
git clone https://github.com/nds-org/biocaddie
cd biocaddie
mvn install

Replication steps

This section describes the steps to repeat our 2016 BioCADDIE challenge submissions. The basic steps are:

Qrels and queries

Please see the queries directory for information about the different query formats used for evaluation.

Cross-validation

The mkeval.sh script generates trec_eval output files for each parameter combination and then runs leave-one-query-out cross validation (loocv) on the results, optimizing for specific metrics.

scripts/mkeval.sh <model> <topics> <collection>

The loocv process optimizes for the following metrics: map, ndcg, P_20, ndcg_cut_20. This process generates one output file per metric for the model/collection/topics. For example:

loocv/model.collection.topics.metric.out

The output file is formatted as: <query> <parameter combination> <metric value>

Comparing model output

To compare models, use the compare.R script. This runs a paired one-tailed t-test comparing two models and outputs the metrics averages and p-value:

Rscript scripts/compare.R <collection> <from model> <to model> <topics>

For example:

Rscript scripts/compare.R combined tfidf dir short

This will report the p-values of a paired, one-tailed t-test with the alternative hypothesis that is greater than .

The model comparisons can be used to select the best model from the training data. Model parameter estimates must be determined from the LOOCV output.