I know it's an old question, but it's a topic I love. ;) I always recommend this paper which goes into details in a very clear way. Let's try to sum up the most important parts for your question.
Step 1: momentum is hermitian but not self-adjoint
Let's take the momentum operator $P=-i\hslash\frac{d}{dx}$. It's an operator acting on functions (usually wavefunctions), so its domain of definition is all functions of $L^2$ which have a derivative also in $L^2$ (page 11).
$P$ is a hermitian operator, so $P$ and $P^\dagger$ act in the same way on functions in the domain of definition of $P$ (page 13).
But $P^\dagger$ has a larger domain of definition than $P$ (page 12). Because of this, $P\neq P^\dagger$ for every function, so $P$ is hermitian but not self-adjoint (page 13).
Step 2: spectral theorem
A self-adjoint $A$ operator has a spectrum in two parts (page 14):
- a discrete part, for eigenfunctions that are its domain of definition,
- a continuous part, for eigenfunctions that are generalized eigenfunctions (page 16).
But an operator $A$ that is hermitian but not self-adjoint has a third part to its spectrum:
- a residual part, for functions that are eigenfunctions of $A^\dagger$ but not of $A$ (page 44).
We saw above that this is possible, that something as simple and well-known as the momentum operator is in this situation.
The paper studies a very common example: a wavefunction for a bound state in $x\in[0,1]$, so that it vanishes at the boundaries. It yields the surprising result that both the discrete and continuous spectra are empty, and that all eigenvalues are in the residual spectrum (page 44).