From an abstract viewpoint, it is indeed not the Hilbert space that is determining the system, but the $C^\ast$-algebra $\mathcal{A}$ of observables, which is a priori just an abstract algebra not represented on any kind of space. This algebra of observables has operators that are not observables because they are not Hermitian/self-adjoint, but it is generated as a $C^\ast$-algebra by the physical observables, which justifies the terminology. The generic isomorphism between separable Hilbert spaces is, in practice, useless because it does not, in general, preserve the representation of the algebra of observables.
States are normalized positive linear functionals on this algebra, and by the GNS construction, every state induces an irreducible representation of the algebra upon a Hilbert space. Every unit vector $v$ in such a Hilbert space in turn induces a normalized positive linear functional on the algebra by
$$ A\mapsto \langle v,Av\rangle$$
so it is justified to call every such Hilbert space a state space. States that arise from such vectors are called pure states in physics. Note that not all states on an algebra of observables are represented in such a way by vectors in a fixed Hilbert space, for instance mixed states (density matrices that are not just the projector onto a single vector on this Hilbert space) can be represented as such, but purifying them requires enlarging the Hilbert space.
If the $C^\ast$-algebra is irreducibly given as an algebra of bounded operators on a Hilbert space, then its pure states in the physical sense in this (faithful, irreducible) representation (the rays) are the extremal points (with respect to convexity) of the set of all states, they carry maximal information. All other states are mixed in the sense that, as abstract states, they arise as linear combinations of these pure states. This justifies the physical usage of a single, fixed Hilbert space as the state of space - using the density matrix formalism, all states may be represented as mixed states on this single, irreducible representation, there is physically no need to consider other representations.
Subtleties arise when the $C^\ast$ algebra is not given as bounded operators (in fact, $C^\ast$ algebras are usually defined such that they cannot contain unbounded operators), as is unfortunately the case for the standard commutation relations $[x,p] = \mathrm{i}\mathbf{1}$. The "states" corresponding to "eigenstates" $\lvert x\rangle$ of $x$ are neither pure nor mixed on the standard representation $L^2(\mathbb{R}^d)$, and auxiliary constructions such as rigged Hilbert spaces are needed to keep the standard physical picture of a single, fixed Hilbert space as the space of states. Another solution is to forget about $x$ and $p$ and consider only their exponentiated forms, which are bounded and everywhere defined, and seek representations and states of this exponentiated version of the operators. The Stone-von Neumann theorem then tells you there is a unique representation of these exponentiated ("Weyl") relations, which is induced by the standard representation of $x$ and $p$ as multiplication and differentiation on $L^2(\mathbb{R}^d)$.