Learning the model

Our goal is to find the parameters $e$ which maximizes the incomplete data likelihood $p(Z\vert e)$. Directly marginalizing the complete data likelihood $p(Z,T\vert e)$ cannot give us a close-form solution to the maximization problem. However, it fits well into the category of the problems which can be solved by the EM algorithm [2]. Our target function is given by

$$Q(e, e^g) = E[ \log p(Z,T\vert e) ]$$ (6)

$$= \sum_T {\log p(Z,T\vert e) f(T\vert Z,e^g)}$$ (7)

which can be decoupled into two parts,

$$Q(e, e^g) = \sum_T { \sum_{k=1}^P {\log p(T_k) f(T\vert Z,e^g) }} +$$ (8)

$$\sum_T { \sum_{k=1}^P {\log {N(Z_{k} ; \mu_{T_k}, \phi_{T_k} )}} f(T\vert Z,e^g) }$$ (9)

The first term is not dependent on the parameters $e$, we only need to maximize the second term in the maximization step. A further derivation shows it is equivalent to maximize,

$$\sum_{k=1}^P { \sum_{i\in S_k} \sum_{T_k, T_k(i)=j} (Z_{i,k} - \mu_{T_k(i)})^2 / \phi_{i,k}^2 f(T_k\vert Z_k,e^g) }$$ (10)

The EM is iteratively performed by two steps. In the ``expectation'' step, we compute the posterior probabilities $f(T_k\vert Z_k, e^g)$ based on the current parameters,

$$f(T_k\vert Z_k,e^g) = \frac{p(T_k){N(Z_{k} ; \mu_{T_k}^g, \phi_{T_k}^g) }} {\sum_{T_k}p(T_k){N(Z_{k} ; \mu_{T_k}^g, \phi_{T_k}^g)}}$$ (11)

In the ``maximization'' step, we take the derivatives with respect to $\mu_j$ and $\phi_j$ and let them be zero, we have the estimated epitome means and variances as follows,

$$\mu_j = \frac{\sum_k \sum_{i\in S_k} \sum_{T_k, T_k(i)=j} f(T_k\vert Z_k,e^g) z_{i,k}} {\sum_k \sum_{i\in S_k} \sum_{T_k, T_k(i)=j} f(T_k\vert Z_k,e^g)}$$ (12)

$$\phi_j = \frac{\sum_k \sum_{i\in S_k} \sum_{T_k, T_k(i)=j} f(T_k\vert Z_k,... ...,k}-\mu_j^g)^2} {\sum_k \sum_{i\in S_k} \sum_{T_k, T_k(i)=j} f(T\vert Z_k,e^g)}$$ (13)