Are non-associative operators (or other kind of elements) used in Physics? For example, in QM I'm looking for something like $A(BC)|\psi\rangle \ne (AB)C|\psi\rangle$.
NOTE: I think that this question does not make much sense, in which case I will close it.
Non-associative algebras represent a challenge in quantum theory because they cannot be realized as linear operators on a Hilbert space which are automatically associative by construction.
These algebras appear in the classical description of a particle moving under the influence of a magnetic monopole in $\mathbb{R}^3$ as well as in string theory in backgrounds with nonvanishing $H$-field (i.e., with nonconstant $B$-field). Please, see for example the following review by Dieter Lüst.
In both cases the classical phase spaces become 2-plectic manifold instead of symplectic.
In the case of the magnetic monopole, The Poisson bracket of two generalized momenta of a particle moving in a magnetic field background is given by:
$\{\pi_i, \pi_j\} = ie\frac{\hbar}{c}\epsilon_{ijk}B^k(x)$
In this case, the finite translation operators $T(\mathbf{a}) = exp(\frac{1}{i \hbar}\mathbf{a}.\mathbf{\pi} ) $ satisfy:
$(T(\mathbf{a_1})T(\mathbf{a_2}))T(\mathbf{a_3}) = exp( -\frac{ie}{\hbar c} \Phi(\mathbf{a_1},\mathbf{a_2}, \mathbf{a_3}))T(\mathbf{a_1})(T(\mathbf{a_2})T(\mathbf{a_3}))$
Where $\Phi(\mathbf{a_1},\mathbf{a_2}, \mathbf{a_3})$ is the flux through the tetrahedron generated by the three translations $\mathbf{a_1}, \mathbf{a_2}, \mathbf{a_3}$.
And the Jacobi relation of three generalized momenta becomes
$ \epsilon_{ijk}\{\pi_i, \{\pi_j, \pi_k\}\}= \frac{2e\hbar^2}{c}\mathbf{\nabla}.\mathbf{B}$
In the presence of a magnetic monopole, the flux through the tetrahedron is nonvanishing, leading to the loss of associativity of the finite translation operators. Also $ \mathbf{\nabla}.\mathbf{B} \ne 0$ leading to the violation of the Jacobi identity.
Due this violation, the Poisson brackets are called twisted Poisson brackets.
In the magnetic monopole case, the quantization of the magnetic charge leads to the quantization of the flux through the tetrahedron and the algebra of the finite translation operators becomes associative. However, the Jacobi identity is still violated in this case. This problem is avoided if the sources of magnetic charge lie outside the configuration space, then the Jacobi relation is satisfied on it. In this case the configuration space will not be $\mathbb{R}^3$. For example, if the particle is restricted to move on a two dimensional sphere, these problems would have been avoided.
Another way out is to declare the generalized momenta as nonohysical and work only with subalgebras of operators which satisfy the Jacobi identity, for example, the angular momenta
$J_k = \epsilon_{ijk} x_i\pi_j$
As mentioned above, nonassociative algebras cannot be represented by linear operators on a Hilbert space. However, it is well known that the Hilbert space description is not the most general description of quantum theory. In the case of nonassociativity, mainly due to the string theory applications, there are attempts to find quantization methods, in which nonassociative algebras can be represented. This subject is very new and not fully understood yet, please see for example the following article by Mylonas Schupp and Szabo.
The main tool used is deformation quantization which is one of the most general methods of quantization. The reason why this method works is that star products can be nonassociative, for example the Moyal star product becomes nonassociative when the Poisson bivector is not the reciprocal of a closed two form.
In M-theory, the worldvolume QFT describing how the world looks to a stack of M2 branes ending on an M5 brane includes 'transverse coordinate' fields which take values in a non-associative algebra.
