Scaling

In unconstrained optimization, a problem is said to be poorly scaled if changes to $x$ in a certain direction produce much larger variations in the value of $f$ than do changes to $x$ in another direction. For example

$$f(x)=10^9{x}_1^2+x_2^2$$

Is an example of a poorly scaled problem. This is very sensitive to changes in $x_1$

---

Note

Newton Direction is scale invariant

Example

Given

$$\underset{x\in {\mathbb{R}}^2}{\min }\frac{1}{2}(x_1^2+\gamma x_2^2)$$

Steepest Descent gives

$$p_k=-\left[\begin{array}{c}{x}_1\\ \gamma x_2\end{array}\right]$$

While Newton Direction

$$p_k=-{\left[\begin{array}{cc}1& 0\\ 0& \gamma \end{array}\right]}^{-1}\left[\begin{array}{c}{x}_1\\ \gamma x_2\end{array}\right]=-\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]$$