Skip to content
/ cite Public
forked from BryanPlummer/cite

Implementation for our paper "Conditional Image-Text Embedding Networks"

License

Notifications You must be signed in to change notification settings

leoozy/cite

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

9 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Conditional Image-Text Embedding Networks

cite contains a Tensorflow implementation for our paper. If you find this code useful in your research, please consider citing:

@inproceedings{plummerCITE2018,
Author = {Bryan A. Plummer and Paige Kordas and M. Hadi Kiapour and Shuai Zheng and Robinson Piramuthu and Svetlana Lazebnik},
Title = {Conditional Image-Text Embedding Networks},
Booktitle  = {ECCV},
Year = {2018}
}

This code was tested on an Ubuntu 16.04 system using Tensorflow 1.2.1.

Phrase Localization Evaluation Demo

After you download our precomputed features/model you can test it using:

python main.py --test --spatial --resume runs/cite_spatial_k4/model_best

You can test the ReferIt dataset by setting the dataset flag and adjusting the number of embeddings to match the trained model:

python main.py --test --spatial --dataset referit --num_embeddings 12 --resume runs/referit_spatial_k12/model_best

Training New Models

Our code contains everything required to train or test models using precomputed features. You can train a new model on Flickr30K Entites using:

python main.py --name <name of experiment>

When it completes training it will output the localization accuracy using the best model on the testing and validation sets. Note that the above does not use the spatial features we used in our paper (needs the --spatial flag). You can see a listing and description of many tuneable parameters with:

python main.py --help

Precomputed Features

Along with our example data processing script in data_processing_example you can download our precomputed (PASCAL) features for the Flickr30K Entities dataset here (126G) and ReferIt dataset here (88G). Unpack the features in a folder named data or update the path in the data loader class.

Our best CITE model using these precomputed features can be on Flickr30K Entities can be found here and ReferIt dataset here.

You can download the raw Flickr30K Entities data here and ReferIt here, but isn't necessary to use our precomputed features.

Many thanks to Kevin Shih and Liwei Wang for providing to their implementation of the Similarity Network that was used as the basis for this repo.

About

Implementation for our paper "Conditional Image-Text Embedding Networks"

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 79.2%
  • MATLAB 20.8%