The Iris Task

Key Notes

• The problem is almost a Linear Separable Problem
  • We can design an (almost) error free Linear Classifier

Training

Minimize mean square error until convergence using gradient descent

How it is written in the book:

$$g=\left[\begin{array}{cc}W& w_0\end{array}\right]\left[\begin{array}{c}{x}^T\\ 1\end{array}\right]$$

Or more familiar

$$z=Wx+b$$

And applying the bias trick

$$z=\left[\begin{array}{cc}W& b\end{array}\right]\left[\begin{array}{c}{x}^T\\ 1\end{array}\right]$$

---

We write

$$MSE=\frac{1}{2}\sum _{k=1}^N(g_k-t_k{)}^T(g_k-t_k)$$

Where $t_k$ is the ground truth (label).

We also need to squish the results between $0$ and $1$ → Sigmoid (which is an activation function)

---

No explicit solution of the minima of MSE exists. We therefore use a gradient based technique called Gradient Descent.

From page=79 we know that

$$\begin{array}{r}{\nabla }_{W}MSE=\sum _{k=1}^N{\nabla }_{g_k}MSE{\nabla }_{z_k}{g}_k{\nabla }_W{z}_k\end{array}$$

Where

$$\begin{array}{rl}{\nabla }_{g_k}MSE& =g_k-t_k\\ {\nabla }_{z_k}g& =g_k\circ (1-g_k)\\ {\nabla }_W{z}_k& =x_k^T\end{array}$$

Note that $\circ$ means elementwise multiplication.

Combining these results with Gradient Descent, we get

$$W(m)=W(m-1)-\alpha {\nabla }_{W}MSE$$

Where

• ${\nabla }_{W}MSE={\sum }_{k=1}^N[(g_k-t_k)\circ g_k\circ (1-g_k)]x_k^T$
  • $t_k$ is the ground truth
  • $z_k=\left[\begin{array}{cc}W& w_0\end{array}\right]\left[\begin{array}{c}{x}^T\\ 1\end{array}\right]$
  • $g_k=sigmoid(z_k)$
• $m$ is the iteration number
• $\alpha$ is the step factor