Reduce protein embedding dimensions

[lincoln-harris]

I'm using the ESM-2 model to generate embeddings for proteins. I'm following the instructions on the README, namely, batching the data, generating per-residue representations and marginalizing those to get per-sequence representations. I'm wondering if there is a way to generate an embedding vector for a protein with fewer than 1280 dimensions? I have a small-scale deep neural network model that may struggle to learn linear layer parameters for such high-dimensional vectors. Using the ESM-2 model to generate, say, 32 dimensional protein embeddings would be super useful to me.
Thanks!

[ptynecki]

@lincoln-harris did you try to adopt vector decomposition method (PCA, UMAP or t-SNE) after the origin embedding and before the network? It should meet your requirements.

[lincoln-harris]

Dimensionality reduction with PCA is definitely an option; I was wondering if there was a way to do this within the ESM-2 model code itself. I'm guessing the answer is no?

[cclark1e]

@lincoln-harris Hi Lincoln, did you ever find a way to get this going? I'm facing a similar issue and dimensionality reduction is dropping too much information to be viable.