Skip to content

Latest commit

 

History

History
118 lines (76 loc) · 7.38 KB

README.md

File metadata and controls

118 lines (76 loc) · 7.38 KB

PathGan: Visual Scan-path Prediction with Generative Adversarial Networks

Marc Assens Kevin McGuinness Xavier Giro-i-Nieto Noel O'Connor
Marc Assens Kevin McGuinness Xavier Giro-i-Nieto Noel O'Connor

A joint collaboration between:

logo-insight logo-dcu logo-gpi
Insight Centre for Data Analytics Dublin City University (DCU) UPC Image Processing Group

Abstract

We introduce PathGAN, a deep neural network for visual scanpath prediction trained on adversarial examples. A visual scanpath is defined as the sequence of fixation points over an image defined by a human observer with its gaze. PathGAN is composed of two parts, the generator and the discriminator. Both parts extract features from images using off-the-shelf networks, and train recurrent layers to generate or discriminate scanpaths accordingly. In scanpath prediction, the stochastic nature of the data makes it very difficult to generate realistic predictions using supervised learning strategies, but we adopt adversarial training as a suitable alternative. Our experiments prove how PathGAN improves the state of the art of visual scanpath prediction on the iSUN and Salient360! datasets.

Publication

Find the pre-print version of our work on arXiv.

Image of the paper

Please cite with the following Bibtex code:

@inproceedings{Assens2018pathgan,
title={PathGAN: Visual Scanpath Prediction with Generative Adversarial Networks},
author={Marc Assens, Xavier Giro-i-Nieto, Kevin McGuinness, Noel E. O'Connor},
journal={ECCV Workshop on Egocentric Perception, Interaction and Computing (EPIC)},
year={2018}
}

You may also want to refer to our publication with the more human-friendly Chicago style:

Marc Assens, Xavier Giro-i-Nieto, Kevin McGuinness, Noel E. O’Connor. “PathGAN: Visual Scanpath Prediction with Generative Adversarial Networks”, ECCV Workshop on Egocentric Perception, Interaction and Computing (EPIC), 2018.

Models

The model is composed by two deep neural networks, the generator and the discriminator, whose combined efforts aim at predicting a realistic scanpath from a given image.

Model Architecture: architecture-fig

The model is implemented in Keras, which at its time is developed over Tensorflow.

Examples

We provide examples of predicted object sequences for two datasets.

1. iSun

ex-isun

Our model ranked #2 in the ICME'18 Grand Challenge on Salient360!).

The big dot indicates the first fixation of the scanpath:

ex-360

Acknowledgements

We especially want to thank our technical support team:

AlbertGil-photo
Albert Gil
We gratefully acknowledge the support of NVIDIA Corporation with the donation of the GeForce GTX Titan Z and Titan X used in this work. logo-nvidia
The Image Processing Group at the UPC is a SGR14 Consolidated Research Group recognized and sponsored by the Catalan Government (Generalitat de Catalunya) through its AGAUR office. logo-catalonia
This work has been developed in the framework of projects TEC2013-43935-R and TEC2016-75976-R, financed by the Spanish Ministerio de Economía y Competitividad and the European Regional Development Fund (ERDF). logo-spain