RF Artifacts after long training times

[alex404]

In certain regimes the receptive fields across all layers decohere and exhibit weird artifacts:

• Green/red 45 degree gabor overlayed on all RFs, with frequencies that are fairly independent of layer.
• General movement upward of RFs, and decoherence into colour splashes in RFs that began as gabors.

This seems to emerge over training, and became more acute when I changed the defaults of the image transformations to be less extreme (mostly not quite so dark). When I switched back to the older config this "fixed" the artifacts. It seems like it might be some kind of overfitting thing, but weight_decay doesn't help. I'll keep running tests. This seems like less of a bug or more like a training issue at this point, so I think we're ready to merge.

[alex404]

This still happens on master before circuit-extensions was merged. I could run simulations going back even more merges, but I have a filling this has always been there and we just haven't tripped over it/noticed.

[alex404]

Current hypothesis that I'm testing:

• Some kind of sparse+noisy autoencoder might might help
  • This in turn might call for post FC decoding
• Might also be helped by managing depth. It'll be okay if very deep layers decohere, we just want a certain number of initial layers not to decohere.