One of the postulates of QM mechanics is that any observable is described mathematically by a hermitian linear operator.
I suppose that an observable means a quantity that can be measured. The mass of a particle is an observable because it can be measured. Why then the mass is not described by a linear hermitian operator in QM?
In non-relativistic quantum mechanics the mass can, in principle, be considered an observable and thus described by a self-adjoint operator.
In this sense a quantum physical system may have several different values of the mass and a value is fixed as soon as one performs a measurement of the mass observable, exactly as it happens for the momentum for instance.
However, it is possible to prove that, as the physical system is invariant under Galileian group (or Galilean group as you prefer), a superselection rule arises, the well-known Bargmann mass superselection rule. It means that coherent superpositions of pure states with different values of the mass are forbidden.
Therefore the whole description of the system is always confined in a fixed eigenspace of the mass operator (in particular because all remaining observables, including the Hamiltonian one, commute with the mass operator).
In practice, the mass of the system behaves just like a non-quantum, fixed parameter. This is the reason, barring subtle technicalities (non-separability of the Hilbert space if the spectrum of the mass operator is continuous), why the mass can be considered a fixed parameter rather than a self-adjoint operator in non-relativistic quantum mechanics.
In relativistic quantum mechanics the picture is quite different. First of all, one has to distinguish between elementary systems (elementary free particles in with Wigner's defintion) and compound (interacting) systems. The formers are defined as irreducible (strongly continuous) unitary representations of Poincaré group. Each such representation is identified by a set of numbers defining the eigenvalues of some observables which attains constant values in the representation because of the irreducibility requirement. The nature of these numbers depend on the structure of the group one is considering.
Each such observable, in the irreducible Hilbert space of the system has the form $\lambda I$ where $\lambda$ is a fixed real number. Referring to the Poincaré group, the mass operator turns out to be one of these elementary observables. Therefore, in relativistic quantum mechanics, the elementary systems must have the trivial mass operator, which as before, can be considered as a fixed, non-quantum parameter.
The picture changes dramatically if one focuses on compound systems: there the mass is simply the energy operator evaluated in the rest frame of the system. It generally shows a mixed spectrum made of a continuous part, due to the "relative" kinetic energy and, below that, a point spectrum describing the possible masses of the overall system.
ADDENDUM. As Arnold Neumaier pointed out to me, neutrinos appear to have non-fixed values of the mass (i.e. the mass operator is not trivial) in view of the presence of the weak interaction. In my view, it is disputable if they can be considered elementary particles since they include weak interaction in their description. Surely they are elementary from a purely physical viewpoint. Maybe Wigner's description is physically inappropriate.
Of course, mass is an observable, although in simple models it is constant.
This is already the case classically. One cannot determine the path of as rocket that burns fuel (which forms a large fraction of its mass) without taking into account that the mass is variable.
The same holds in quantum mechanics, whenever the mass is not fixed by the modeling assumptions. Important case where the mass is given by a nontrivial operator M (then called the mass matrix) are
the mass operator of a relativistic particle in an external field. Its eigenvalues are the masses of the bound states.
the Cabibbo-Kobayashi-Maskawa Quark Mass Matrix http://arxiv.org/abs/hep-ph/9708216
the Neutrino Mass Matrix http://arxiv.org/abs/hep-ph/9802328
On the other hand, if the mass is a constant m, it is also represented by an operator, namely M:=m*1, where 1 is the identity matrix. This is a Hermitian operator, hence fits the standard description.
Mass-squared is a Hermitian linear operator, it's a Casimir operator $\hat{C}_{1}=\hat{P}_{0}\hat{P}_{0}-\hat{P}_{i}\hat{P}_{i}$ for the Poincare group. It's Hermitian because the translation generators $\hat{P}_{\mu}$ are Hermitian. It commutes with all the generators of the Poincare group and so it's eigenvalues (mass-squared) are constant on each irreducible subspace.
I'll just supplement Prof. Kalitvianski's answer by adding that the mantra « observables as operators » only applies to the special kind of measurement called a « quantum measurement », which always involve amplification. Other kinds of measurement, such as measuring physical constants, are not modelled by observables either, like the speed of light, the mass of the electron, the charge on an electron, etc.
The Nobel prize winner Eugene Wigner wrote very careful articles about the theory of measurement, although it is considered superseded by many today, they are still an excellent foundation for understanding the more modern theories. They have been reprinted in the collection of very readable scientific essays of his, Symmetries and Reflections.
On one hand, in non relativistic QM mass is a numerical parameter in a theoretical construction, so there is no need in an additional operator while transition form CM to QM.
On the other hand, in relativistic QM the energy is an operator reducing to mass (rest energy) in a particular case $\vec{p}=0$. Normally the energy of a free particle has a continuous spectrum (not quantized, nor calculated), so it is still an external parameter to the theory for an "elementary" free particle.
In case of a compound system, the rest mass can be (in principle) calculated from masses and interaction forces of its constituents.
Because mass is supposed to be a quantity which is to remain constant for all eternity in quantum mechanics. Mass is energy in the rest frame of the of the particle. The reason why you don’t see this in non-relativistic quantum mechanics is that you can add an arbitrary constant to the Hamiltonian. In relativistic quantum mechanics, that offset is fixed with the rest mass energy becoming a meaningful and important quantity. All wave functions pick up the Compton frequency to their phase factors.
Therefore, the energy operator is the mass operator, thus mass is an observable, but mathematically, it is considered as a constant, and its operator is the energy one.
And one more thing; the neutrino mass matrix is not an operator: it is a matrix in the 3 dimensional space of flavors (electron,muon,tau), not on the Hilbert space of the states of neutrinos.
Rest mass $m$ is conjugate to proper time $\tau$ as such to produce an operator: $$ \hat{m} = −\frac{iℏ}{c^2}\frac{\partial}{\partial \tau} $$ In an analogous way that energy $E$ is conjugate to local time $t$.
Read more in this arXiv paper: https://arxiv.org/abs/1710.06504v2
