When the dot produt of two vectors is $0$, they are perpendicular to each other. The fomula in the picture is a special case of
\begin{equation}
\vec{A} \cdot \vec{B} = |A||B| \cos \theta,
\end{equation}
where $\theta$ is the angle formed by the vectors. If this angle is $90^\circ$, like in the picture, the dot product is $0$, and the forces act on perpendicular directions. Therefore the box will move, and the total motion will be that caused by force $\vec{A}$ added to that caused by force $\vec{B}$. Since the forces have equal magnitudes, you get that the box will move along the dotted line.
A way to see this clearly is to add the forces before finding the motion of the box. Since $\vec{A} = (10 N) \vec{x}$ and $\vec{B} = (10 N) \vec{y}$, where $\vec{x}$ and $\vec{y}$ are unit vectors pointing on the $x$ and $y$ directions, respectively, the total force will be
\begin{equation}
\vec{F} = \vec{A} + \vec{B} = (10 N) \vec{x} + (10 N) \vec{y} = (10 N) (\vec{x} + \vec{y}).
\end{equation}
The vector $\vec{x} + \vec{y}$ points along the dotted line in the picture, the diagonal, so that is the direction along which $\vec{F}$ pushes the box.
If this is not clear, you can convince yourself by taking the dot product of $\vec{x} + \vec{y}$ it with $\vec{x}$ and $\vec{y}$ and use the formula in the top, to find exactly what angle it makes with these two vectors. You should find that in both cases the angle is $1/\sqrt{2}$ radians, or $45^\circ$.