Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
__pycache__		__pycache__
LICENSE		LICENSE
README.md		README.md
example.py		example.py
mi_lower_bounds.py		mi_lower_bounds.py
utils.py		utils.py

Repository files navigation

MI-Lower-Bounds

This repo. contains a Pytorch implementation of four Mutual Information (MI) lower bounds based on the paper https://arxiv.org/pdf/1905.06922.pdf and the notebook https://github.com/google-research/google-research/blob/master/vbmi/vbmi_demo.ipynb \

The definition of mutual information is
$I(X;Y) = E_{p(x,y)}\Big[log \frac{p(x,y)}{p(x)p(y)}\Big]$
The four MI lower bounds implemented are
$1. I_{TUBA} = E_{p(x,y)}\Big[log \frac{e^{f(x,y)}}{a(y)}\Big] - E_{p(x)p(y)}\Big[ \frac{e^{f(x,y)}}{a(y)} \Big]$

$2.I_{NWJ} = E_{p(x,y)}[f(x,y)] -\frac{1}{e} E_{p(x)p(y)}\Big[ e^{f(x,y)} \Big]$

$3.I_{NCE} = E_{p^K(x,y)}\Big[\frac{1}{K} \sum_{i=1}^{K}log \frac{e^{f(y_i, x_i)}}{\frac{1}{K}\sum_{j=1}^{K}e^{f(y_i,x_j)}}\Big]$

$4.I_{\alpha} =1 + \frac{1}{K} \sum_{j=1}^{K} \Big( \frac{1}{K-1} \sum_{i \neq j}\Big( log\frac{e^{f(x_i,y_i)}}{a(y_i;x_{\neq j})} - \frac{e^{f(x_i,y_j)}}{a(y_i;x_{\neq j})} \Big) \Big)$
where
$a(y; x_{1:M}) = \alpha \frac{1}{M} \sum_{l=1}^{M} e^{f(x_l, y)} + (1-\alpha)q(y)$

About

This Repo. contain a Pytorch implementation of three MI lower bounds based on the paper https://arxiv.org/pdf/1905.06922.pdf and the notebook https://github.com/google-research/google-research/blob/master/vbmi/vbmi_demo.ipynb

Report repository

Releases

No releases published

Packages

No packages published

Languages

Python 100.0%