awesome-information-geometry

This repo is a collection of AWESOME things about information geometry, including papers, code, etc. Feel free to star and fork.

Contents

• awesome-information-geometry
• Contents
• Books
• Papers
  • Survey
  • Information Geometry for Neural Networks
  • Information Geometry for Clustering
  • Information Geometry for NMF
  • Information Geometry for MCMC
  • Information Geometry for HMM
  • Information Geometry for Dimension Reduction
  • Fisher Information Matrix
  • Fisher Kernels
  • Information Diffusion Kernels
  • Natural Gradients
  • alpha-Divergence
  • Bregman Divergence
  • Jensen-Shannon Divergence
  • Exponential and Mixture Families
  • Inequalities
  • Transport Information Geometry
  • Computational Information Geometry
• Library
• Lectures and Tutorials
  • PDF
  • YouTube
• Journals
• Conferences
• Workshops

Books

• Differential Geometrical Foundations of Information Geometry (WorldScientific, 2022)
• Progress in Information Geometry (Springer, 2021)
• Computational Information Geometry (Springer, 2017)
• Information Geometry (Springer, 2017)
• Information Geometry and Population Genetics (Springer, 2017)
• Information Geometry and Its Applications (Springer, 2016)
• Matrix Information Geometry (Springer, 2013)
• Methods of Information Geometry (American Mathematical Soc., 2000)

Papers

Survey

• Fields of Application of Information Geometry (Information Geometry, 2017)
• Information geometry in optimization, machine learning and statistical inference (Front. Electr. Electron. Eng. China, 2010)

Information Geometry for Neural Networks

• The Fisher–Rao loss for learning under label noise (Information Geometry, 2022)
• A Reparametrization-Invariant Sharpness Measure Based on Information Geometry (NeurIPS, 2022)
• IGAGCN: Information geometry and attention-based spatiotemporal graph convolutional networks for traffic flow prediction (Neural Networks, 2021)
• Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization (ICML, 2021)
• An Information-Geometric Distance on the Space of Tasks (ICML, 2021)
• Information Geometry of Orthogonal Initializations and Training (ICLR, 2020)
• Fisher-rao metric, geometry, and complexity of neural networks (AISTATS, 2019)
• Principles of Riemannian Geometry in Neural Networks (NeurIPS, 2017)
• f-GANs in an Information Geometric Nutshell (NeurIPS, 2017)
• Principal whitened gradient for information geometry (Neural Networks, 2008)
• Geometrical Singularities in the Neuromanifold of Multilayer Perceptrons (NeurIPS, 2001)
• Algebraic Information Geometry for Learning Machines with Singularities (NeurIPS, 2000)
• Gradient systems in view of information geometry (Physica D: Nonlinear Phenomena, 1995)
• Information geometry of the EM and em algorithms for neural networks (Neural Networks, 1995)
• Information geometry of Boltzmann machines (IEEE Trans. Neural Networks, 1992)

Information Geometry for Clustering

• A novel clustering algorithm based on information geometry for cooperative spectrum sensing (IEEE Systems Journal, 2020)
• On Clustering Histograms with k-Means by Using Mixed α-Divergences (Entropy, 2014)
• Barycentric distribution estimation for texture clustering based on information-geometry tools (ISETC, 2012)

Information Geometry for NMF

• Non-negative low-rank approximations for multi-dimensional arrays on statistical manifold (Information Geometry, 2023)
• Geometrical formulation of the nonnegative matrix factorization (ICONIP, 2018)
• Generalized alpha-beta divergences and their application to robust nonnegative matrix factorization (Entropy, 2011)

Information Geometry for MCMC

• Geometric Aspects of Data-Processing of Markov Chains (arxiv, 2022)
• Information Geometry of Reversible Markov Chains (Information Geometry, 2021)
• Geometry of Markov Chains, Finite State Machines, and Tree Models (IEICE Tech. Rep., 2014)
• Information geometry approach to parameter estimation in Markov chains (ISIT, 2014)
• Information geometry and sequential Monte Carlo (arXiv, 2012)
• Information Geometry of Contrastive Divergence (ITSL, 2008)
• Information geometry of Gibbs sampler (NNA, 2004)

Information Geometry for HMM

• Information geometry approach to parameter estimation in hidden Markov model (Bernoulli, 2022)
• Local equivalence problem in hidden Markov model (Information Geometry, 2019)

Information Geometry for Dimension Reduction

• Generalized t-SNE Through the Lens of Information Geometry (IEEE Access, 2021)
• Dimension Reduction for Mixtures of Exponential Families (ICANN, 2008)
• The e-PCA and m-PCA: Dimension reduction of parameters by information geometry (IJCNN, 2004)

Fisher Information Matrix

• A Statistical Manifold Framework for Point Cloud Data (ICML, 2022)
• On the Variance of the Fisher Information for Deep Learning (NeurIPS, 2021)
• On the Fisher-Rao Information Metric in the Space of Normal Distributions (GSI, 2019)
• The Spectrum of the Fisher Information Matrix of a Single-Hidden-Layer Neural Network (NeurIPS, 2018)
• General Fisher information matrices of a random vector (Adv. Appl. Math., 2017)
• Evaluating neuronal codes for inference using Fisher information (NeurIPS, 2010)
• Invariant Fisher information (Differ Geom Appl., 1994)

Fisher Kernels

• Learning Representation from Neural Fisher Kernel with Low-rank Approximation (ICLR, 2022)
• Deep active learning for biased datasets via fisher kernel self-supervision (CVPR, 2020)
• Persistence fisher kernel: A riemannian manifold kernel for persistence diagrams (NeurIPS, 2018)
• The fisher kernel: a brief review (RN, 2011)
• Improving the fisher kernel for large-scale image classification (ECCV, 2010)
• Indefinite kernel fisher discriminant (ICPR, 2008)
• A Kullback-Leibler divergence based kernel for SVM classification in multimedia applications (NeurIPS, 2003)
• A novel graph-based fisher kernel method for semi-supervised learning (ICPR, 2014)
• Learning the Similarity of Documents: An Information-Geometric Approach to Document Retrieval and Categorization (NeurIPS, 1999)

Information Diffusion Kernels

• Learning parameterized histogram kernels on the simplex manifold for image and action classification (ICCV, 2011)
• Sentiment classification with interpolated information diffusion kernels (ADKDD, 2007)
• Diffusion kernels on statistical manifolds (JMLR, 2005)

Natural Gradients

• Invariance properties of the natural gradient in overparametrised systems (Information Geometry, 2022)
• Tractable structured natural-gradient descent using local parameterizations (ICML, 2021)
• Marginalized Stochastic Natural Gradients for Black-Box Variational Inference (ICML, 2021)
• Sinkhorn Natural Gradient for Generative Models (NeurIPS, 2020)
• An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methods (NeurIPS, 2020)
• Natural Policy Gradient Primal-Dual Method for Constrained Markov Decision Processes (NeurIPS, 2020)
• Understanding Approximate Fisher Information for Fast Convergence of Natural Gradient Descent in Wide Neural Networks (NeurIPS, 2020)
• Ngboost: Natural gradient boosting for probabilistic prediction (ICML, 2020)
• A Formalization of the Natural Gradient Method for General Similarity Measures (GSI, 2019)
• Fast Convergence of Natural Gradient Descent for Over-Parameterized Neural Networks (NeurIPS, 2019)
• Limitations of the empirical Fisher approximation for natural gradient descent (NeurIPS, 2019)
• Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search (ICML, 2019)
• Online natural gradient as a Kalman filter (Electron. J. Stat., 2018)
• Fast yet simple natural-gradient descent for variational inference in complex models (ISITA, 2018)
• Natural gradient via optimal transport (Information Geometry, 2018)
• Fast Approximate Natural Gradient Descent in a Kronecker Factored Eigenbasis (NeurIPS, 2018)
• Exact natural gradient in deep linear networks and its application to the nonlinear case (NeurIPS, 2018)
• SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient (NeurIPS, 2018)
• Comparison-based natural gradient optimization in high dimension (GECCO, 2014)
• Projected Natural Actor-Critic (NeurIPS, 2013)
• Stochastic Gradient Riemannian Langevin Dynamics on the Probability Simplex (NeurIPS, 2013)
• New Sparse Adaptive Algorithms Based on the Natural Gradient and the $L_0$-Norm (IEEE J. Ocean. Eng., 2012)
• Natural Policy Gradient Methods with Parameter-based Exploration for Control Tasks (NeurIPS, 2010)
• A Generalized Natural Actor-Critic Algorithm (NeurIPS, 2009)
• Stochastic search using the natural gradient (ICML, 2009)
• Incremental Natural Actor-Critic Algorithms (NeurIPS, 2007)
• Topmoumoute Online Natural Gradient Algorithm (NeurIPS, 2007)
• Natural Actor-Critic for Road Traffic Optimisation (NeurIPS, 2006)
• Rprop using the natural gradient (Trends and Applications in Constructive Approximation, 2005)
• A Natural Policy Gradient (NeurIPS, 2001)
• Natural gradient descent for on-line learning (PRL, 1998)
• Natural gradient works efficiently in learning (Neural Computation, 1998)

alpha-Divergence

• Infinite-dimensional gradient-based descent for alpha-divergence minimisation (Ann. Stat., 2021)
• Utilizing amari-alpha divergence to stabilize the training of generative adversarial networks (Entropy, 2020)
• Divergence Functions in Information Geometry (GSI, 2019)
• Log-determinant divergences revisited: Alpha-beta and gamma log-det divergences (Entropy, 2015)
• On the chi square and higher-order chi distances for approximating f-divergences (IEEE Signal Processing Letters, 2013)
• Families of alpha-beta-and gamma-divergences: Flexible and robust measures of similarities (Entropy, 2010)
• Information geometry of divergence functions (Bull. Pol. Acad. Sci. Tech. Sci., 2010)
• $\alpha $-Divergence Is Unique, Belonging to Both $f$-Divergence and Bregman Divergence Classes (IEEE Trans. Information Theory, 2009)
• Non-negative matrix factorization with $\alpha$-divergence (Pattern Recognit. Lett., 2008)
• Integration of Stochastic Models by Minimizing $\alpha$-Divergence (Neural Computation, 2007)

Bregman Divergence

• Transport information Bregman divergences (Information Geometry, 2021)
• Target detection within nonhomogeneous clutter via total bregman divergence-based matrix information geometry detectors (IEEE Trans. Signal Processing, 2021)
• The Bregman Chord Divergence (GSI, 2019)
• Logarithmic Divergences: Geometry and Interpretation of Curvature (GSI, 2019)
• Information geometry for radar target detection with total Jensen–Bregman divergence (Entropy, 2018)
• Information geometry for covariance estimation in heterogeneous clutter with total Bregman divergence (Entropy, 2018)
• The information geometry of mirror descent (IEEE Trans. Information Theory, 2015)
• The information geometry of Bregman divergences and some applications in multi-expert reasoning (Entropy, 2014)
• Information geometry of U-Boost and Bregman divergence (Neural Computation, 2004)

Jensen-Shannon Divergence

• On a variational definition for the Jensen-Shannon symmetrization of distances based on the information radius (Entropy, 2021)
• $\alpha$-Geodesical Skew Divergence (Entropy, 2021)
• On a generalization of the Jensen–Shannon divergence and the Jensen–Shannon centroid (Entropy, 2020)
• On the Jensen–Shannon Symmetrization of Distances Relying on Abstract Means (Entropy, 2019)

Exponential and Mixture Families

• On partial likelihood and the construction of factorisable transformations (Information Geometry, 2022)
• On a convergence property of a geometrical algorithm for statistical manifolds (ICONIP, 2019)
• Testing the Number and the Nature of the Components in a Mixture Distribution (GSI, 2019)
• Sobolev Statistical Manifolds and Exponential Models (GSI, 2019)
• Minimization of the Kullback-Leibler Divergence over a Log-Normal Exponential Arc (GSI, 2019)
• Riemannian Distance and Diameter of the Space of Probability Measures and the Parametrix (GSI, 2019)
• Information geometry of positive measures and positive-definite matrices: decomposable dually flat structure (Entropy, 2014)
• Geometry of deformed exponential families: Invariant, dually-flat and conformal geometries (Physica A, 2012)
• Geometry of q-Exponential Family of Probability Distributions (Entropy, 2011)

Inequalities

• Curvature Inequalities and Simons’ Type Formulas in Statistical Geometry (GSI, 2021)
• Inequalities for Statistical Submanifolds in Hessian Manifolds of Constant Hessian Curvature (GSI, 2019)
• B. Y. Chen Inequalities for Statistical Submanifolds in Sasakian Statistical Manifolds (GSI, 2019)
• Generalized Wintgen Inquality for Legendrian Submanifolds in Sasakian Statistical Manifolds (GSI, 2019)
• Cramér-Rao lower bound and information geometry (Connected at Infinity II., 2013)
• Inequalities for Tsallis relative entropy and generalized skew information (Linear Multilinear Algebra, 2011)

Transport Information Geometry

• Wasserstein information matrix (Information Geometry, 2023)
• Wasserstein Statistics in One-Dimensional Location-Scale Models (GSI, 2021)
• Traditional and Accelerated Gradient Descent for Neural Architecture Search (GSI, 2021)
• Recent Developments on the MTW Tensor (GSI, 2021)
• Wasserstein Proximal of GANs (GSI, 2021)
• Hessian Curvature and Optimal Transport (GSI, 2019)
• Information geometry connecting Wasserstein distance and Kullback–Leibler divergence via the entropy-relaxed transportation problem (Information Geometry, 2018)

Computational Information Geometry

• Computing Statistical Divergences with Sigma Points (GSI, 2021)
• Remarks on Laplacian of Graphical Models in Various Graphs (GSI, 2021)
• Classification in the Siegel Space for Vectorial Autoregressive Data (GSI, 2021)
• Information Metrics for Phylogenetic Trees via Distributions of Discrete and Continuous Characters (GSI, 2021)
• Wald Space for Phylogenetic Trees (GSI, 2021)
• A Necessary Condition for Semiparametric Efficiency of Experimental Designs (GSI, 2021)
• Parametrisation Independence of the Natural Gradient in Overparametrised Systems (GSI, 2021)
• Properties of Nonlinear Diffusion Equations on Networks and Their Geometric Aspects (GSI, 2021)
• Rényi Relative Entropy from Homogeneous Kullback-Leibler Divergence Lagrangian (GSI, 2021)
• Statistical Bundle of the Transport Model (GSI, 2021)
• Topological Methods for Unsupervised Learning (GSI, 2019)
• Geometry and Fixed-Rate Quantization in Riemannian Metric Spaces Induced by Separable Bregman Divergences (GSI, 2019)
• The Statistical Minkowski Distances: Closed-Form Formula for Gaussian Mixture Models (GSI, 2019)
• Parameter Estimation with Generalized Empirical Localization (GSI, 2019)
• Properties of the Cross Entropy Between ARMA Processes (GSI, 2019)
• Computational information geometry for binary classification of high-dimensional random tensors (Entropy, 2018)
• Computational information geometry in statistics: theory and practice (Entropy, 2014)
• Computational Information Geometry and its Applications (ICNNB, 2005)

Library

• NNGeometry
• natural-gradients
• InformationGeometry.jl
• geodesical_skew_divergence

Lectures and Tutorials

PDF

• An elementary introduction to information geometry (Entropy, 2022)
• Divergence function, information monotonicity and information geometry (WITMSE, 2009)

YouTube

• Information Geometry and Its Applications: Survey by Shun-Ichi Amari (YouTube)
• Information Geometry by Microsoft Research (YouTube)
• Nihat Ay : Information Geometric structures in Cognitive Systems Research (YouTube)
• Computational Information Geometry with Frank Nielsen (YouTube)
• Information Geometry and its Application by Melvin Leok (YouTube)

Journals

• Information Geometry (Journal home)

Conferences

• International Conference on Information Geometry for Data Science (IG4DS)
• International Conference on Geometric Science of Information (GSI)

Workshops

• Symmetry and Geometry in Neural Representations (at NeurIPS 2022)
• TAG in Machine Learning (at ICML 2022)
• Deep Learning through Information Geometry (at NeurIPS 2020)