Position and momentum basis

Question

I am studying formalism of QM from the book by Griffiths. He illustrates via two examples that momentum and position operators have no eigenfunctions in Hilbert space. In that case, how can we expand a general state vector (in Hilbert space) in position/momentum eigenbasis, as the eigenvectors constituting the basis do not lie inside the space?

Answer 1

One postulate of QM is that quantum states live in Hilbert space, and thus must be normalisable.

Another postulate of QM is that if you have a physical observable, then it is representable by a Hermitian operator, and any quantum state can be represented by the eigenfunctions of that operator.

Now, if the physical observable has a continuous spectrum, then those eigenfunctions will live in Rigged Hilbert space rather than Hilbert space, and cannot be normalised. That is fine, because it just means that real quantum states must be made of normalisable wavepackets of these eigenfunctions.

Answer 2

how can we expand a general state vector (in Hilbert space) in position/momentum eigenbasis

In continuous space, we can't - as you write, momentum and position operators in continuous space do not have normalizable eigenfunctions that would belong to the Hilbert space. They have "improper eigenfunctions" or "generalized eigenfunctions", like $\delta(x-x_0)$ or $e^{ip_0x/\hbar}$, which satisfy the eigenvalue equation in distribution sense, but do not belong to the Hilbert space.

We can express any well-behaved normalizable wave function as integral of these improper eigenfunctions, but such integral is not, strictly speaking, an expression of a linear combination of functions that form a basis in the Hilbert space. This means the simple Dirac formulation of the postulates which assumes every observable has a set of eigenfunctions that form a basis in the space of quantum states is not mathematically correct for all observables.

The question of definition and finding of eigenvalues and eigenfunctions is studied in functional analysis/spectral theory. The more mathematically correct but also more difficult formalism is by von Neumann. In practical simple calculations physicists often ignore these problems and just use the Dirac formalism, relying on the fact other people have shown that for all practical purposes (e.g. textbook examples) it works, and that it found mathematically acceptable formulation in the so-called rigged Hilbert space formalism. In some cases (e.g. careful explanation of quantum theory) the Dirac/rigged Hilbert space notation is too simplifying and misleading and one better go back to the more traditional mathematician's formalism, like Landau & Lifshitz (Quantum mechanics) or Weinberg (Lectures on Quantum Mechanics) do.