Probabilistic Canonical Correlation Analysis in Detail

Probabilistic canonical correlation analysis is a reinterpretation of CCA as a latent variable model, which has benefits such as generative modeling, handling uncertainty, and composability. I define and derive its solution in detail.

Published

10 September 2018

Standard CCA

Canonical correlation analysis (CCA) is a multivariate statistical method for finding two linear projections, one for each set of observations in a paired dataset, such that the projected data points are maximally correlated. For a thorough explanation, please see my previous post.

I will present an abbreviated explanation here for completeness and notation. Let $X_{a} \in R^{n \times p}$ and $X_{b} \in R^{n \times q}$ be two datasets with $n$ samples each and dimensionlity $p$ and $q$ respectively. Let $w_{a} \in R^{p}$ and $w_{b} \in R^{q}$ be two linear projections and $z_{a} = X_{a} w_{a}$ and $z_{b} = X_{b} w_{b}$ be a pair of $n$ -dimensional “canonical variables”. Then the CCA objective is:

$w_{a}^{⋆}, w_{b}^{⋆} = ar g w_{a}, w_{b} max {w_{a}^{⊤} X_{a}^{⊤} X_{b} w_{b}}$

With the constraint that:

$∥ z_{a} ∥_{2} = w_{a}^{⊤} X_{a}^{⊤} X_{a} w_{a} = 1 ∥ z_{b} ∥_{2} = w_{b}^{⊤} X_{b}^{⊤} X_{b} w_{b} = 1$

Since $w_{a}^{⊤} X_{a}^{⊤} X_{b} w_{b} = z_{a}^{⊤} \cdot z_{b}$ , the objective can be visualized as finding linear projections $w_{a}$ and $w_{b}$ such that $z_{a}$ and $z_{b}$ are close to each other on a unit ball in $R^{n}$ . If we find $r$ such projections where $r \leq min (p, q)$ , then we can embed our two datasets into $r$ -dimensional space.

Probabilistic interpretation of CCA

A probabilistic interpetation of CCA (PCCA), is one in which our two datasets, $X_{a}$ and $X_{b}$ , are viewed as two sets of $n$ observations of two random variables, $x_{a}$ and $x_{b}$ , that are generated by a shared latent variable $z$ . Rather than use linear algebra to set up an objective and then solve for two linear projections $w_{a}$ and $w_{b}$ , we instead write down a model that captures these probabilistic relationships and use maximum likelihood estimates to update its parameters. See my previous post on probabilistic machine learning if that statement is not clear. The model is:

$z x_{a} ∣ z x_{b} ∣ z \sim N (0, I_{r}), r \leq min (p, q) \sim N (W_{a} z + μ_{a}, Ψ_{a}) \sim N (W_{b} z + μ_{b}, Ψ_{b}) (1)$

Where $W_{a} \in R^{p \times r}$ and $W_{b} \in R^{q \times r}$ are two arbitrary matrices, and $Ψ_{a}$ and $Ψ_{b}$ are both positive semi-definite. (Bach & Jordan, 2005) proved that the resulting maximum likelihood estimates are equivalent, up to rotation and scaling, to CCA.

It is worth being explicit about the differences between this probabilistic framing and the standard framing. In CCA, we take our data and perform matrix multiplications to get lower-dimensional representations $z_{a}$ and $z_{b}$ :

$z_{a} = X_{a} w_{a} z_{b} = X_{b} w_{b}$

The objective is to find projections $w_{a}$ and $w_{b}$ such that $z_{a}$ and $z_{b}$ are maximally correlated.

But the probabilistic model, Equations ( $1$ ), is a function of random variables. If we marginalize out $z$ for either $x_{a}$ or $x_{b}$ , we get the following generative model:

$x = W z + u (2)$

Where $u \sim N (μ, Ψ)$ . If we assume our data is mean-centered, meaning $μ = 0$ —we make the same assumption in CCA—and rename $W$ to $Λ$ , we can see that PCCA is just group factor analysis with two groups (Klami et al., 2015):

$x_{a} x_{b} = Λ_{a} z + u_{a} = Λ_{b} z + u_{b} (3)$

And this in turn is nice because we can just use the maximum likelihood estimates we computed for factor analysis for PCCA. To see this, let’s use block matrices to represent our data and parameters:

$x = [x_{a} x_{b}], Λ = [Λ_{a} Λ_{b}], u = [u_{a} u_{b}]$

Where $x \in R^{p + q}$ , $Λ \in R^{(p + q) \times k}$ , and $u \in R^{p + q}$ and,

$u \sim N ([00], [Ψ_{a} 0 0 Ψ_{b}])$

Then our PCCA updates are identical to our updates for factor analysis:

$Λ^{⋆} Ψ^{⋆} = (i = 1 \sum n x E_{z ∣ x} [z ∣ x_{i}]^{⊤}) (i = 1 \sum n E_{z ∣ x} [z z^{⊤} ∣ x_{i}])^{- 1} = \frac{1}{n} diag (i = 1 \sum n x_{i} x_{i}^{⊤} - Λ^{⋆} E_{z ∣ x} [z ∣ x_{i}] x_{i}^{⊤})$

Futhermore, this definition gives us a joint density for $x$ that should look similar to the density $p (x)$ in factor analysis:

$p (x) = N ([00], [Λ_{a} Λ_{a}^{⊤} + Ψ_{a} Λ_{b} Λ_{a}^{⊤} Λ_{a} Λ_{b}^{⊤} Λ_{b} Λ_{b}^{⊤} + Ψ_{b}])$

See the Appendix for a derivation.

It’s worth thinking about how the properties of CCA are converted to probabilistic assumptions in PCCA. First, in CCA, $z_{a}$ and $z_{b}$ are a pair of embeddings that we correlate. The assumption is that both datasets have similar low-rank approximations. In PCCA, this property is modeled by having a shared latent variable $z$ .

Furthermore, in CCA, we proved that the canonical variables are orthogonal. In PCCA, there is no such orthogonality constraint. Instead, we assume the latent variables are independent with an isotropic covariance matrix:

$z \sim N (0, I_{r})$

This independence assumption is the probabilistic equivalent of orthogonality. The covariance matrix of the latent variables is diagonal, meaning there is no covariance between the $i$ -th and $j$ -th variables.

The final constraint of the CCA objective is that the vectors have unit length. In probabilistic terms, this is analogous to unit variance, which we have since the identity matrix $I_{r}$ is an isotropic matrix with each diagonal term equal to $1$ .

Code

For an implementation of PCCA, please see my GitHub repository of machine learning algorithms, specifically this file.

Appendix

1. Derivations for $μ_{x}$ and $Σ_{x}$

Let’s solve for $μ_{x}$ and $Σ_{x}$ for the density:

$p ([x_{a} x_{b}]) = N (μ_{x}, Σ_{x})$

First, note that $E [z] = 0$ and $E [u] = μ$ . Then:

$μ_{x} = E [x] = E [W z + u] = W E [z] + E [u] = μ = [μ_{a} μ_{b}]$

If the data are mean-centered, as we assume, then:

$[μ_{a} μ_{b}] = [00]$

To understand the covariance matrix $Σ_{x}$ ,

$Σ_{x} = [Σ_{a a} Σ_{b a} Σ_{a b} Σ_{b b}]$

let’s consider $Σ_{a a}$ and $Σ_{a b}$ . The remaining block matrices, $Σ_{b b}$ and $Σ_{b a}$ have identical proofs, respectively, but with the variables renamed. Both derivations will use the fact that $E [X Y] = E [X] \cdot E [Y]$ if $X$ and $Y$ are independent. First, let’s consider $Σ_{a a}$ :

$Σ_{a a} = E [(x_{a} - μ_{a}) (x_{a} - μ_{a})^{⊤}] = E [(x_{a}) (x_{a})^{⊤}] - μ_{a} μ_{a}^{⊤} = E [(W z + u_{a}) (W z + u_{a})^{⊤}] - μ_{a} μ_{a}^{⊤} = E [Λ_{a} z z^{⊤} Λ_{a}^{⊤} + u_{a} z^{⊤} Λ_{a}^{⊤} + Λ_{a} z u_{a}^{⊤} + u_{a} u_{a}^{⊤}] - μ_{a} μ_{a}^{⊤} = Λ_{a} Λ_{a}^{⊤} I_{r} Λ_{a} Λ_{a}^{⊤} E [z z^{⊤}] + 0 E [u_{a}] E [z^{⊤}] E [Λ_{a}^{⊤}] + 0 Λ_{a} E [z] E [u_{a}^{⊤}] + var (x_{a}) + μ_{a} μ_{a}^{⊤} E [u_{a} u_{a}^{⊤}] - μ_{a} μ_{a}^{⊤} = Λ_{a} Λ_{a}^{⊤} + Ψ_{a}$

So for the two views, $a$ and $b$ , we have:

$Σ_{a a} = Λ_{a} Λ_{a}^{⊤} + Ψ_{a} Σ_{b b} = Λ_{b} Λ_{b}^{⊤} + Ψ_{b}$

Now, let’s consider $Σ_{a b}$ :

$Σ_{a b} = E [(x_{a} - μ_{a}) (x_{b} - μ_{b})^{⊤}] = E [x_{a} x_{b}^{⊤}] - μ_{a} μ_{b}^{⊤} = E [(Λ_{a} z + u_{a}) (Λ_{b} z + u_{b})^{⊤}] - μ_{a} μ_{b}^{⊤} = E [Λ_{a} z z^{⊤} Λ_{b}^{⊤} + u_{a} z^{⊤} Λ_{b}^{⊤} + Λ_{a} z u_{b}^{⊤} + u_{a} u_{b}^{⊤}] - μ_{a} μ_{b}^{⊤} = Λ_{a} Λ_{b}^{⊤} I_{r} Λ_{a} Λ_{b}^{⊤} E [z z^{⊤}] + 0 Λ_{b}^{⊤} E [u_{a}] E [z^{⊤}] + 0 Λ_{a} E [z] E [u_{b}^{⊤}] + cov (u_{a}, u_{b}) + μ_{a} μ_{b}^{⊤} E [u_{a} u_{b}^{⊤}] - μ_{a} μ_{b}^{⊤} = Λ_{a} Λ_{b}^{⊤}$

Thus, the full covariance matrix for $x$ is:

$Σ = [Λ_{a} Λ_{a}^{⊤} + Ψ_{a} Λ_{b} Λ_{a}^{⊤} Λ_{a} Λ_{b}^{⊤} Λ_{b} Λ_{b}^{⊤} + Ψ_{b}]$

Bach, F. R., & Jordan, M. I. (2005). A probabilistic interpretation of canonical correlation analysis.
Klami, A., Virtanen, S., Leppäaho, E., & Kaski, S. (2015). Group factor analysis. IEEE Transactions on Neural Networks and Learning Systems, 26(9), 2136–2147.