Deep predictive coding networks are neuroscience-inspired unsupervised learning models that learn to predict future sensory states. We build upon the PredNet implementation by Lotter, Kreiman, and Cox (2016) to investigate if predictive coding representations are useful to predict brain activity in the visual cortex. We use representational similarity analysis (RSA) to compare PredNet representations to functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) data from the Algonauts Project (Cichy et al., 2019).
In contrast to previous findings in the literature (Khaligh-Razavi & Kriegeskorte, 2014), we report empirical data suggesting that unsupervised models trained to predict frames of videos may outperform supervised image classification baselines.
This repository contains supporting code for PredNet training, fine-tuning, feature extraction, and evaluation. We also use the Algonauts development kit, which is not distributed here. Experiment workflow is as follows:
- PredNet training on videos
- Feature extraction
- Feature extraction for CORnet (Kubilius et al. 2018)
- Evaluation using representational dissimilarity analysis (RSA)