Factor Graph

VS DGM & UGM

DGM and UGM: allow a global function of several variables to be expressed as a product of factors over subsets of those variables, designed for conditional independence or potentials.
Factor graphs: make this decomposition explicit by introducing additional nodes for the factors in addition to the nodes representing the variables, designed for more explicit details of the factorization.

Convolution

Convolution is a linear transformation and the output is the feature of the input.

• 1D: text processing
• 2D: image processing
• 3D: 3D data, CT, microscopy, etc

mD: m dimensions where the kernel is shifted

Variable Elimination Algorithm

General probabilistic problem: calculate $p(X_F|X_E)$ for arbitrary disjoint subsets $E$ and $F$: ( $X_F$ are query nodes, $X_E$ are evidence nodes)

$$p(X_F|X_E)=\frac{p(X_F,X_E)}{p(X_E)}=\frac{\sum_{X_R}p(X_E,X_F,X_R)}{\sum_{X_F}p(X_F,X_E)}$$

$X_R$ are nodes that must be marginalized out of joint probability.

Joint probability table size of $n$ variabels that takes $k$ states:
a naive summation: $O(k^n)$.
a factored form: $O(k^r)$ where $r\ll n$

Stochastic Gradient DescentVarious GD’s problems and advantages Forward computer the average loss among: all training samples (GD) randomly pick a single training sample (SGD) randomly pick b training ...

为什么需要无向图模型

使用无向图模型(UGM: Undirected Graphical Models) $\mathcal G(\mathcal V,\mathcal E)$来表示Markov Random Field(Markov network)。
$\mathcal V$ : 顶点集合，每个顶点与一个随机变量 $X_i$ 一一对应。
$\mathcal E$ : 无向边集合。
边没有方向性，这对于一些图像处理或空间分析问题更加自然。
For MRF, conditional independence is determined by simple graph separation, not the defination of “blocked”.

Multilayer Perceptron (MLP)

The Perception

input: $x_1,x_2…,x_N$
weight: $w_1,w_2,…,w_N$
output: $y={\mathcal g }(\boldsymbol{w^Tx})=\begin{cases} 1,\qquad if\quad\boldsymbol{w^Tx}>0 \\ 0,\qquad else\end{cases}$
Linear functions are limited: Linear models cannot fit in data from XOR function:

条件独立 conditional independence

完全独立不足以对现实的随机变量建模，而完全相关会导致计算量过大。
随机变量 $X_A$ 和 $X_C$ 关于 $X_B$ 条件独立：$p(x_A,x_C|x_B)=p(x_A|x_B) p(x_C|x_B)$
等价于：$p(x_A|x_B,x_C)=p(x_A|x_B) \qquad \forall X_B:x_B>0$
也可以写作：$X_A \perp X_C | X_B$

Regression VS Classification

回归问题：定量(quantity)
分类问题：定性(label)
分类问题的标签一般由回归或者直接测量来得到。eg. 预测明天的天气，降水量超过一定阈值$C$为有雨，否则为不下雨。这里的降水量是个连续型随机变量，可以直接测量也可以通过回归来预测。

$$\widetilde{y}=\begin{cases} 1,\qquad\qquad if\boldsymbol{w^Tx}>c \\ 0,\qquad\qquad else\end{cases}$$

也可以将阈值$C$与待学习的参数$w$合并：

$$\widetilde{y}=\begin{cases} 1,\qquad\qquad if\boldsymbol{w^Tx}>0 \\ 0,\qquad\qquad else\end{cases}$$

概率空间 Probability Space

概率空间 $(\Omega,E,P)$包含了三个部分，用来描述概率现象。
样本空间(Outcome/Sample Space) $\Omega$
事件空间(Event Space) E
概率分布(Probability Distribution) $P:E\to R$

神经网络发展历程

神经网络主要依赖于特征转换(feature transformation)，通过准备数据、提取特征来构造模型。