I understand the stress tensor at a specific point $P$ as follows. You draw a cube with its center at $P$ and look at its faces. The face with its normal vector in the positive $x$ direction has a certain force acting on it. $\sigma_{xx}$ is the limit of the $x$-component of that force over the area of the face as the cube shrinks to size $0$. Similarly, $\sigma_{xy}$ is the limit of the $y$-component and so on.
Specifically, the force on a face is essentially the sum of the forces on all the particles on the face. If the force inside the material is roughly continuous (is that true for the insides of materials?) then as you take the limit as the cube shrinks to size $0$, shouldn't the force on all the particles on each face of the cube be essentially the same, thus leading to the forces on each face being the same and thus making, for example, $\sigma_{xx}=\sigma_{yx}=\sigma_{zx}$?
It seems to me somewhat obvious that all stress tensor components whose second element is $x$ should be the same (along with $y$ and $z$) but given how that is not stated in physics books that is definitely not the case.
Where does my logic go wrong? Also, could people give me some examples of where the forces on the different faces of the cube are clearly different even as the cube size gets extremely small? I would like to gain a better understanding of stresses.