Linear Regression

Linear regression

Univariate: single feature $x\in R$
Multivariate: multiple features $\boldsymbol{x}\in R^m$
Linear Transformation: $\widetilde{y}=\boldsymbol{w^Tx}$
Loss: measures the difference between the prediction and the ground truth
Training: to optimize(i.e. minimize) the loss w.r.t parameters($\boldsymbol{w}$)

Graph Models

DGM

UGM (MRF: Markov Random Field)

Markov Property:

• Global: no path between
• Local: condition upon Markov blanket
• Pair: given the rest, no direct edges

InceptionNetV2

Inception V1 + Batch Normalization(BN), converge faster
After convolution / linear/ fully connected layer: $z=ReLU(BN(f(x)))$
Inreality: $z=BN(ReLU(f(x)))$

_get_jt_clique_and_edges()

Create a graph G using nx.Graph() and add nodes and edges to it using the input and edges. Using a helper function undirected_graph_eliminate() to get the reconstituted graph, and use nx.find_cliques() to get the jt_cliques in this graph. Create a clique graph G_c using the jt_cliques, set the sepsets’ size as edges’ weights. Get G_c’s maximum spanning tree and assign this tree’s edges as jt_edges.

_learn_node_parameter_w()

When input is None

It means there are no observation of its parents. In this case weight size is 1, and weights[0] equals to the average of output using numpy.average().

Motivation: How to get the unknown parameter $\theta$ of a DGM/UGM $p(x_1,x_2…,x_M|\theta)$ from fully observed data?
Given: a set of 𝑁 independent and identically distributed (i.i.d) complete observation of each random variable 𝑋:$x_{1,1}, … x_{1,N},…, x_{M,1},…x_{M,N}$.

• Maximum Likelihood Estimate (MLE) $$\begin{aligned}\hat{\theta}&=\arg\max_{\theta} p(x_1,...,x_M|\theta)\\&=\arg\max_{\theta} \prod_{i=0}^N p(x_{1,i},...,x_{M,i}|\theta)\qquad (i.i.d)\end{aligned}$$
• Maximum a Posteriori (MAP) $$\begin{aligned}\hat{\theta}&=\arg\max_{\theta} p(\theta|x_1,...,x_M)\\&=\arg\max_{\theta}\frac{p(x_1,...,x_M|\theta)p(\theta)}{p(x_1,...,x_M)}\qquad (Bayes'rule)\\&=\arg\max_{\theta}\frac{\prod_{i=0}^N p(x_{1,i},...,x_{M,i}|\theta)p(\theta)}{p(x_1,...,x_M)}\qquad (i.i.d)\\&=\arg\max_{\theta}\prod_{i=0}^N p(x_{1,i},...,x_{M,i}|\theta)p(\theta) \end{aligned}$$

Historical Notes

Fukushima’s Neocognitron

• Hierarchical feature extraction
• Local connectivity field
• Hand crafted weight (before BP was developed)

Paper: https://www.arxiv-vanity.com/papers/1712.01208/

Introduction

There exists various data access patterns, and correspondingly various choices of index structure.
B-tree: range request (from sorted array)
HashMap: single key look-ups (from unsorted array)
Bloom filters: check for record existence

Paper: https://www.arxiv-vanity.com/papers/1712.01208/