I am someone who works in mathematics, works with operators, and tries to interpret things in terms of a "toy" QM that doesn't necessarily have anything to do with actual QM. Two things in the below, which outlines the "toy" QM that I employ, that might not happen in any real QM theory is that A) I have a system with trivial dynamics and B) this means I can talk about sequential measurement.
This might be a maths question rather than a physics question (but if it it a maths question rather than a physics question, then it needs to be asked in a different way somewhere else).
Suppose a QM system given by Hilbert space $\mathsf{H}$, and suppose the system state is given by a state $\varphi:B(\mathsf{H})\to \mathbb{C}$. Consider a finite spectrum observable with eigendecomposition $$f=\sum_{\lambda\in \sigma(f)}\lambda \,p_\lambda,$$ a self-adjoint element of $B(\mathsf{H})$. Given the state $\varphi$, we say that the expectation of $f$ is: $$\mathbb{E}_\varphi(f)=\varphi(f),$$ and the probability that we measure $\varphi$ with $f$ and get $f=\lambda$ is $$\mathbb{P}_\varphi[f=\lambda]=\varphi(p_\lambda),$$ and the state collapses to: $$\widetilde{p_\lambda}\varphi= \frac{\varphi(p_\lambda \cdot p_\lambda)}{\varphi(p_\lambda)}.$$ If we represent $\varphi$ as a vector state $\xi_\varphi$, the wave function collapse is $\xi_\varphi\mapsto p_\lambda \xi_\varphi$ in projective Hilbert space.
If we assume trivial dynamics (the unitary operator is the identity), then we can, after measuring $\varphi$ with $f$, getting $f=\lambda$, and finding the system state is $\widetilde{p_{\lambda}}\varphi$, take another finite spectrum observable $$g=\sum_{\mu \in\sigma(g)}\mu\,q_\mu,$$ and measure the state $\widetilde{p_{\lambda}}\varphi$ with it. We might write that: $$\mathbb{P}_{\widetilde{p_{\lambda}}\varphi}[g=\mu]=\frac{\varphi(p_\lambda q_\mu p_\lambda)}{\varphi(p_\lambda)}.$$
Now we know that in general $gf$ is not an observable but I consider this sequential measurement "$f$ then $g$" (I have used $g\succ f$) an object with values in $\sigma(f)\times \sigma(g)\subset \mathbb{R}^2$, and crucially it gives meaning to quantities like $q_\mu p_\lambda$ via: $$\mathbb{P}_{\varphi}[(g\succ f)=(\mu,\lambda)]=\varphi(|q_\mu p_\lambda|^2),$$
where of course $|q_\mu p_\lambda|^2=p_\lambda q_\mu p_{\lambda}$.
This to me gives an intuitive algebra (product/multiplication) structure on observables: that if $f,g\in B(\mathsf{H})$ are observables, the quantity $gf\in B(\mathsf{H})$, not an observable, is related to an $\mathbb{R}^2$-valued object $g\succ f$. So when I look at the place $\mathcal{A}\subset B(\mathsf{H})$ where observables live, I want it to have an algebra structure, $f,g\in \mathcal{A}\implies gf\in\mathcal{A}$.
What I still don't understand, and I understand it is still an issue with mainstream QM, is that $g+f$ is an observable whenever $f$ and $g$ are. So in addition to the multiplication I have $f,g\in \mathcal{A}\implies f+g\in \mathcal{A}$. I don't expect today for someone to help interpret this (beyond perhaps using some observable-state duality).
Let us summarise right here (glossing over a desire that $\mathcal{A}$ be complete) that we want $\mathcal{A}$ to be a $\mathrm{C}^*$-algebra. It allows us to make sense of $gf$ and we just kind of put up with $g+f$.
Recently however I have seen people in the area of maths I work in use Operator Systems. These are closed, self-adjoint unital subspaces of $\mathcal{O}\subset B(\mathsf{H})$ (or $\mathcal{O}\subset \mathcal{A}$), where we don't require the algebra structure $f,g\in\mathcal{O}\implies gf\in \mathcal{O}$ but we do require the vector structure: $$f,g\in\mathcal{O}\implies g+f\in\mathcal{O}.$$
Finally, hopefully a focused question:
Why would a QM theory work with operator systems rather than operator algebras?
