03: Bayesian Multivariate Linear Model

Combining the content from Bayesian Linear Model and Multivariate Linear Model we can show the following. Consider the following multivariate (response) linear model:

$$ \begin{aligned} \mathbf y_i \mid \mathbf B &\sim \text{MvNormal}(\mathbf x_i^T \mathbf B, \boldsymbol\Sigma) \\ \text{vec}(\mathbf B) &\sim \text{MvNormal}(\mathbf m, \mathbf S) \end{aligned} $$

with dimensions:

$\mathbf y_i$	$\mathbf x_i$	$\boldsymbol\Sigma$	$\mathbf B$
$(R \times 1)$	$(P\times 1)$	$(R\times R)$	$(P\times R)$

Construct $\mathbf Y$ ($N \times R$) by stacking $N$ rows of $\mathbf y_i^T$. Similarly, construct $\mathbf X$ ($N \times P$) by stacking $N$ rows of $\mathbf x_i^T$.

Then,

$$ \begin{aligned} \text{vec}(\mathbf B) \mid \text{vec}(\mathbf Y) &\sim \text{MvNormal}(\mathbf m^\star, \mathbf S^\star), \text{ where} \\ \mathbf S^\star &= \left( \boldsymbol\Sigma^{-1} \otimes \mathbf X^T \mathbf X + \mathbf S^{-1} \right) ^ {-1} \\ \mathbf m^\star &= \mathbf S^\star \left( \boldsymbol\Sigma^{-1} \otimes \mathbf X^T)~\text{vec}(\mathbf Y) + \mathbf S^{-1} \mathbf m \right) \\ \Rightarrow \mathbf m^\star &= \mathbf S^\star \left( \text{vec}(\mathbf X^T \mathbf Y \boldsymbol\Sigma^{-1}) + \mathbf S^{-1} \mathbf m \right). \end{aligned} $$

Note that with minor adjustments, instead of $\text{vec}(\cdot)$, we could have used $\text{flat}(\cdot)$ throughout, where $\text{flat}(\cdot)$ stacks the rows into a single column. This is useful in programming languages that store matrices in row-major order (e.g., in python). Python has a flatten method, which is essentially $\text{flat}(\cdot)$. Replacing vec with flat usually changes the order of entries in the Kronecker product. E.g., The previous equations would be rewritten as:

Model

$$ \begin{aligned} \mathbf y_i \mid \mathbf B &\sim \text{MvNormal}(\mathbf x_i^T \mathbf B, \boldsymbol\Sigma) \\ \text{flat}(\mathbf B) &\sim \text{MvNormal}(\mathbf m, \mathbf S) \end{aligned} $$

Posterior

$$ \begin{aligned} \text{flat}(\mathbf B) \mid \text{flat}(\mathbf Y) &\sim \text{MvNormal}(\mathbf m^\star, \mathbf S^\star), \text{ where} \\ \mathbf S^\star &= \left(\mathbf X^T \mathbf X \otimes \boldsymbol\Sigma^{-1} + \mathbf S^{-1}\right)^{-1} \\ \mathbf m^\star &= \mathbf S^\star \left((\mathbf X^T \otimes \boldsymbol\Sigma^{-1}) ~\text{flat}(\mathbf Y) + \mathbf S^{-1} \mathbf m\right) \\ \Rightarrow \mathbf m^\star &= \mathbf S^\star \left(\text{flat}(\mathbf X^T \mathbf Y \boldsymbol\Sigma^{-1}) + \mathbf S^{-1} \mathbf m\right). \end{aligned} $$