Question regarding a part of the paper

[yxchng]

Can I know why "From the geometric perspective, ...normal vector x"?

Also, I can't quite understand your figure 4 too. Why $\frac{\partial L}{\partial \bar{x}}$ points in the south east direction?

Do you mind explaining?

[happynear]

$\frac{\partial L}{\partial \bar{x}}$ is the gradient passed from back-propagation. It can be in any direction.

[happynear]

The proof of why x and its gradient are orthogonal is in Appendix 8.5.

[yxchng]

I am not asking about why they are orthogonal. I am asking about the line after that, the line after you say they are orthogonal. How does this lead to the geometric perspective you mentioned?

[happynear]

Tangent space is a space spanned by all the tangent line. The gradient is orthogonal with x, so it is in the tangent space.

The gradient is actually a vector. I draw it from the end of x because when updating x, we minus lr * gradient from the x.

[yxchng]

Shouldn't the gradient vector $\partial L/\partial \bar{x}$ be perpendicular to x and points to the east and not south east?

[happynear]

The gradient w.r.t. \bar{x} can be in any direction, but the gradient w.r.t. x is orthogonal with x.

[hongxin001]

I'm also confused about that. Why the gradient w.r.t. x is the projection of the gradient w.r.t. \bar{x}