Using separation of variables to solve Schrödinger equation for a free particle

Question

I was reading Introduction to Quantum Mechanics by David Griffiths and I am in Chapter 2, page 45. I know that since the solutions from Schrödinger equation cannot be normalized for a free particle. This must imply that the wavefunction of free particle must be non separable i.e., $$\Psi(x,t)\neq \psi(x)\phi(t)$$ because the solution which we get from solving the Schrödinger equation in this case is $$\psi(x) = Ae^{ikx}+Be^{-ikx}$$ cannot be normalized. Thus, this is not a valid solution.

But Griffiths writes that

The general solution to the time-dependent Schrodinger equation is still a linear combination of separable solutions (only this time it's an integral over the continuous variable $k$, instead of a sum over the discrete index)

But how is this possible, because we already know that we cannot use separation of variables to solve the Schrödinger equation?

Answer 1

The assumption that separability and normalizability are somehow linked is wrong. It can be directly shown to be wrong because we can use the separability ansatz, use $\hat{H}\vert\psi\rangle=E\vert\psi\rangle$, and get a complete set of solutions and they agree with our separability ansatz. So, you can say that the success of the process justifies the ingredient ansatz.

Now, coming to your core confusion. An energy eigenstate of a free particle with energy $E$ is given by $e^{-iEt}(Ae^{ikx}+Be^{-ikx})$ where $E=\frac{k^2}{2m}$. You can check that this solution indeed satisfies the Schrodinger equation. This is not normalizable but it is perfectly separable.

Finally, it should be noted that non-normalizable wavefunctions, indeed, cannot describe a physical particle/system. However, they are useful because they arise as eigenfunctions of position and momentum operators and they form a complete basis for all the normalizable wavefunctions. So, the non-normalizable wavefunctions do not themselves live in the Hilbert space but they provide a basis for the Hilbert space. And thus, solving the Schrodinger equation in this basis is equivalent to solving it for all wavefunctions.

For example, a Gaussian wavepacket of the form $\psi(x,0)=Ae^{-x^2/\Delta}$ can be seen as the linear combination $\psi(x,0)=\int \frac{dp}{2\pi}\ e^{ipx}\tilde\psi(p)$ where $\tilde\psi(p)$ is the Fourier transform of $\psi(x,0)$. Now, we can trivially calculate $\psi(x,t)=\int \frac{dp}{2\pi}\ e^{-ip^2t/2m}e^{ipx}\tilde\psi(p)$. Of course, I simply mentioned the Gaussian wavefunction as an example, the same formulae and the procedure can be used for any initial wavefunction.

Answer 2

There is a far more serious conceptual issue in this question and in Griffith's book that must be addressed.

This is the claim that the (continuous spectrum) eigenfunction solutions of a free particle have no physical interpretation.

This claim (by Griffiths and in the other answers) about the continuous spectrum is unfortunately a very common misunderstanding - specifically Griffith's claim is that (ref. [2], Sec. 2.4)

" In the case of the free particle, then, the separable solutions do not represent physically realizable states. A free particle cannot exist in a stationary state; or, to put it another way, there is no such thing as a free particle with a definite energy.

Later on in his scattering chapter this belief actually forces him to say in a footnote about the general solution of the Schrodinger equation:

For the moment, there's not much quantum mechanics in this: what we're really talking about is the scattering of waves, as opposed to classical particles...

This is his way of dodging the fact that he used a single plane wave to model an incoming free particle with a completely well-defined momentum and had it scatter off a scattering center. By calling them plane waves and saying it's like we're just studying the scattering of waves it somehow means it applies to a scattering problem of a quantum particle but does not model a free particle by a single plane wave. It's completely incoherent.

Taken seriously, since his statements setting up the general solution can be justified from the Born approximation to the integral solution of the Schrodinger equation, he's bizarrely trying to say the setting up of the Born approximation to the general solution does not involve much quantum mechanics it's all just waves man... Of course he and nobody else actually believes that, but it should ring alarm bells to anybody thinking quantum mechanically.

There are so many issues with this belief that I'm going to have to break my answer up into three sections pointing out how problematic this is.

A: Contradicting the Literature

At the very least these claims completely contradict the claims that Born himself, the 'founder' of the 'Born rule', was making when he came up with the rule in the study of a continuous spectrum collision problem in [7]. He very clearly says "there is no escape from the conclusion" that the incoming free particle he is scattering off an atom is described by a "definite state" along a straight line "which corresponds to a plane wave", and uses an incoming plane wave along the $z$-axis to describe it's "state" (i.e. it's wave function). One can just write it off as being old, fine.

More importantly, it also completely contradicts the claims in the absolutely canonical textbook of reference [1], taken extremely seriously even by critics [10], which explicitly refers to the eigenfunctions of a free particle for example as the wave function of the particle ([1], Sec. 17 or Sec. 34 for example). Similarly Dirac ([5], Sec. 30) literally uses a single free particle eigenfunction to illustrate the "suitability" of the terms "wave function" and "wave equation". Another big-name author stating explicitly that a single free particle eigenfunction has "physical meaning" is Kramers in ([6, Sec. 22]).

So to believe this one has to start believing that most of the founders of QM were themselves wrong about the most basic QM problem, and claim their physical interpretation for the non-normalizability (given below) is wrong too.

Even if they are all wrong about a problem as simple as the free particle, the fact that (some of the most) canonical textbooks are claiming a physical interpretation actually exists for something this simple and fundamental should give anybody making such claims serious pause. Even acknowledging such an alternative perspective exists should at least be a litmus test for whether one is getting an honest expression of the situation.

Given how deeply ingrained this belief is, it is useful to quote a (good) textbook ([8], Sec. 2.3) which honestly points out the situation in the literature that it is only one of two possible choices to decide to give up on treating continuous spectrum wave functions seriously, but not the only choice:

There are two ways out of this difficulty. The first is to give up the concept of absolute probabilities when dealing with wave functions such as (2.13) or (2.16) which are not square integrable. Instead, $|\Psi(\mathbf{r}, t)|^2 d \mathbf{r}$ is then interpreted as the relative probability of finding the particle at time t in a volume element $d\mathbf{r}$ centred about $\mathbf{r}$, so that the ratio $|\Psi(\mathbf{r}_1, t)|^2/|\Psi(\mathbf{r}_2, t)|^2$ gives the probability of finding the particle within a volume element centred around $\mathbf{r} = \mathbf{r}_1$, compared with that of finding it within the same volume element at $\mathbf{r} = \mathbf{r}_2$...

This is of course what Dirac, Landau etc... do that is commonly believed to be impossible. After realizing this means a particle's position is completely unknown (a basic implication of the uncertainty principle) they then say

This suggests a second way out of the difficulty, which is to give up the requirement that the free particle should have a precisely defined momentum, and to superpose plane waves corresponding to different momenta to form a localised wave packet, which can be normalised to unity.

While it is this second approach people seem to prefer, and there good are reasons to do it as an approximation [in fact, the use of wave packets is actually a classical approximation tool ([1], Sec. 6), so people are of course really just sneaking in the weakest classical intuition possible with this approximation], it is of course denying/rationalizing-away a completely natural consequence of the theory because of a bias for a simpler case (discrete spectrum) to blindly carry over to the more complicated case. There's no difference between this and a bias for the discrete case in classical discrete vs. continuous probability theory. The classical probability discrete distribution interpretation at a single (or discrete set of) points itself does not carry over blindly to the continuous distribution either, but we don't just pretend it's meaningless (i.e. 'not physical') there.

Thus, there is no problem taking the second approach as an approximation.

Indeed Dirac ([5], Sec. 12) talks about this: he says that although one finds infinite norm states in the continuous spectrum, one would need an infinite amount of precision to actually experimentally realize a continuous spectrum state exactly, i.e. saying that although they are there we can't actually attain infinite precision so the infinite norm shouldn't worry us too much. He then suggests it may be that only the finite norm states are the ones that can be experimentally realized (due to potentially not needing infinite precision). But he just means this with regard to actually precisely experimentally realizing such states, not that they aren't there as a fundamental necessity of the theory. He even says we could not do without them, that they are needed to precisely formulate the theory of QM and that other areas of science do this too.

In other words (now mine not his), he is basically saying that if you think it's okay to just throw the infinite norm states out for not being experimentally realizable then you should also throw out all of science too. Classical mechanics for example says that the position and velocity of a particle are simultaneously knowable in principle theoretically (in contrast to quantum mechanics), the theory doesn't say anything at all otherwise, it's just a fundamental claim of all of classical mechanics that this is possible in principle. Obviously experimentally this is not possible with infinite precision, but that doesn't mean we throw all of classical physics out.

More importantly, Dirac actually does go on to give a physical interpretation to infinite norm states later (see below), so you could probably be forgiven for thinking it is impossible to misread the above discussion Dirac gives as him saying we should just throw out infinite norm states.

In a footnote to the original von Neumann ([9], Sec. II.8, sometimes considered the originator of what is commonly said to be the rigorous approach) discussion of a free particle, he says the wave function

...does not belong to $\mathbb{R}_{\infty}$ because of the unboundedness of the integral of the square of its absolute value. From our point of view, ... what does not belong to $\mathbb{R}_{\infty}$ for us does not exist.

and in a footnote appended to this adds:

Of course, only success in the physical application can justify this point of view or its use in quantum mechanics.

Thus even von Neumann was aware that this approach was at risk of simply being incorrect. If one wants to claim that this approach is the correct approach on a fundamental level, something you can see from the above quotes is not even agreed upon in the literature, they have to not only ignore the above physical interpretations given above by Dirac, Landau etc... but also have to cogently answer all of the gigantic conceptual problems with this approach mentioned in the next section (there are of course more).

So, while it's okay to use wave packets (or eigendifferentials etc...) as an approximation, in terms of the absolutely fundamental nature of quantum mechanics as a theory: there are simply gigantic logical problems with taking this second approach too seriously (which is exactly what is being done), it's as absurd as pretending classical mechanics theoretically just tells us the position/velocity of a particle exists simultaneously only up to the accuracy of a given measurement instead of theoretically existing somewhere in principle. But the theory should internally make no sense at times if we take this approach if this is the case. Lets see this next:

B: Theoretical Contradictions

Lets look at the logical problems of saying the continuous spectrum eigenfunctions aren't physical.

This claim that a continuous spectrum eigenfunction is not physical would imply e.g. the wave functions of the hydrogen atom, which has both a continuous and discrete spectrum, and is built from discrete and continuous spectrum wave functions, is built from physical eigenfunctions on some of the spectrum (on the discrete spectrum) and non-physical eigenfunctions on the rest of it (the continuous spectrum). It's even more absurd than adding fermions and bosons, in this case it's adding 'physical' and 'non-physical'. Thus individual 'scattering states' in the general expansion of the hydrogen atom wave function are for some reason 'not physical' even though a discrete spectrum bound state term in the exact same expansion is physical. This is an example of the gigantic logical flaw one has to ignore regarding why the Schrodinger equation even applies to such a problem if it's stationary states are not physical, and of course it traces back directly to contradicting the superposition principle as we'll see below.

Another extremely important example is Fermi's Golden Rule applied to the continuous spectrum, e.g. quantum scattering which is a continuous spectrum problem. It is precisely because the initial wave functions are in the continuous spectrum that the naive 'transition probability' $dw$ in Fermi's Golden Rule does not have the correct dimensions of probability per unit time, the 'transition probability' now has dimensions which depend on ([1], Sec. 43) the normalization of the initial continuous spectrum wave function normalization (which would make no sense if we believed they could not even be normalized...). If the initial continuous spectrum wave functions are normalized as 'one particle per volume $V$', this is a clear explanation for why we have to take the 'transition probability' in QFT and turn it into a 'decay rate' or a 'cross section' to get measurable results.

Even worse, in QFT, the use of single free particle eigenfunctions as the wave functions of the free particles in a scattering process is absolutely essential, and their continuous spectrum normalization properties vital, when doing any qft scattering problem. One has to believe that any qft problem where the incoming free particle has a (theoretically) precisely known momenta, so that it is described by a single free particle stationary state, is (theoretically) non-physical, as if that makes any sense.

These facts do not disappear just because one uses second quantization and quantum fields, second quantization can be 'defined' in the first place starting from the use of eigenfunctions for a system of identical particles, see for example ([1] ch. IX).

Further, in QFT one commonly takes a box normalization over a finite domain [so the 'free particle' eigenfunctions are now discrete spectrum wave functions, and we can normalize to 'one particle per volume $V$' with little thought, even though we can do this in the continuous spectrum too (see [1], Sec. 15 and 48)] so we can make things easier, but we're still taking a limit at the end of the calculation, they don't all of a sudden turn non-physical in that limit. In other words, the Fermi Golden Rule normalization issue mentioned above is always there in a QFT scattering problem, whether we do it at the end or the beginning.

In other words, one actually has to believe that e.g. the $\frac{1}{\sqrt{2 E_{\mathbf{p}}}}$ normalization factors in Klein-Gordon etc... are all just convenient math tricks that all magically work out, and just ignore why individual 'free particle' plane wave eigenfunctions with a continuous spectrum even have such normalization factors, as if this is all just a happy accident. A common way to ignore all the physical reasoning here is to pretend we add the normalization factors to non-normalizable 'wave functions' that are 'non-physical' because we just want a relativistically invariant volume element, but it amounts to a specific choice of normalization factors from the above perspective, and it's not even that commonly done because it makes setting up the (continuous spectrum) decay rates, cross sections etc... less intuitive than the 'one paticle per volume $V$' normalization approach.

Even in a non-relativistic scattering problem, one now has to believe: a) a free particle with precisely known momenta (thus described by an individual plane wave solution) becomes non-physical; b) the resolution of a non-physical plane wave into angular momenta of that plane wave, that's absolutely vital in scattering when we take e.g. a single incoming free particle $e^{ikz}$ in usual scattering problems, is all just non-physical; c) instead arguments relying classical thinking about scattering of classical waves has to replace actual quantum mechanical thinking; d) any conceptual issues can just be rationalized away as 'math'. This is just the beginning of the so-called rigorous approach.

Another very basic aspect of QM that completely fails if individual continuous spectrum eigenfunctions of non-relativistic free particles are not wave functions is the existence of a 'discrete spectrum' for the $E < 0$ stationary states of a particle in a potential $U$ that vanishes at infinity. Sketching the proof in [1], if a single individual stationary state in such a potential goes off to infinity, there is absolutely nothing stopping them from reaching infinity in principle in general, in fact the potential disappearing as one goes to infinity may make it more likely for the particle to get to infinity the further out it goes. But why is it that they don't reach infinity and instead remain in a finite region (so that the spectrum turns out to be discrete despite the possibility of going off to infinity)? It is precisely because of the fact that if such a stationary state went off to infinity, it would then reduce to the stationary state of a single free particle, but when the energy of a single free particle can be precisely known it means it's wave function is an individual continuous spectrum stationary state i.e. the usual plane wave with $E = \mathbf{p}^2/2m$. But this is always positive/non-negative, yet we were assuming that $E < 0$ held. Thus we get an unavoidable contradiction unless the particle is either a bound and so the wave function simply never reduces to that of a free particle ([1], Sec. 18), or nonsensically when the single stationary state, with it's single energy eigenvalue, reduces to the case of a free particle when the potential goes to zero the physics just stops and the stationary state illogically just stops applying to the system, or at best magically all of a sudden turns into an integral of free particle stationary states even though we were working with a single stationary state whose eigenvalue was precisely known by assumption, for some unexplained reason.

In fact, if we do not accept that continuous spectrum wave functions are actually wave functions, then quantum mechanics does not even exist in the continuous spectrum case, thus scattering etc... is all gone, since the wave function of a system does not even exist in principle. As explained in my answer here, which again this is just summarizing the canonical interpretation of the QM measurement process as described in [1], it is in principle impossible to even determine the wave function of a physical system unless the abstract Fourier expansion of the combined 'measuring apparatus + quantum system' "collapses" down to a single individual eigenfunction, due to the classical nature of the measuring apparatus. If the measuring device is in the continuous spectrum then this means that the wave function unavoidably 'collapses' down to a single continuous spectrum eigenfunction. In reality the 'collapsing' never occurs, the wave function was that single Fourier term the whole time, i.e. the total wave function after a measurement involves two terms, a single eigenfunction of the continuous spectrum measuring apparatus, and a second term in the product related to the wave function of the system that measured after the measurement. If we take the belief that continuous spectrum eigenfunctions aren't wave functions, then we have to believe that a classical measuring apparatus (when a measured eigenvalue is known with complete certainty, which has theoretically possible otherwise even the wave function of a system in principle can never even be known and we have no theory) is precisely described by a 'wave function' that is not a 'wave function', it just makes absolutely no sense.

On an even more primitive level, the reason why the individual eigenfunctions have to be the potential wave functions of a potential physical system is that, beginning from the fundamental concept of 'the superposition principle', we can't even define the 'total wave function' of a system (linear combination of the stationary states) unless the system is described by individual eigenfunctions representing a possible physical states. The things we sum together to get the total wave function of the system in principle have to themselves be potential wave functions for a possible state of that system, otherwise they're not even allowed into the sum. It just completely makes no sense if the eigenfunctions are 'not physical' - there's nothing to 'sum' via the superposition principle to even start to construct the total wave function of a system of a continuous spectrum in the first place.

To make this point again: go read ([1], Sec. 2 and 5) and then tell me why we're even allowed to have a continuous spectrum anywhere in physics if the 'eigenfunctions' are not actually representing a possible physical state of the system - it's precisely because each eigenfunction is a potential state of the system that we can then use the superposition principle to add them together to get the total wave function. It simply contradicts the superposition principle (as described in [1]) to say the ($\hbar = 1$) eigenfunction solutions $e^{i (\mathbf{p} \cdot \mathbf{r} - Et)}$ of the free-particle Schrodinger equation are not physical.

In other words, it is precisely because of the Born rule and the superposition principle that the continuous spectrum wave functions need to be normalized against delta functions and need to be interpreted as 'physical wave functions' of a potential system. If we couldn't do this then even the wave function of a system doesn't even exist because when some measuring apparatus has a continuous spectrum, the only way we can even fix the wave function of a system after a measurement is by invoking the fact that a single eigenstate of that continuous spectrum measuring device (quasi-classical) wave function describes the state of the measuring device after the measurement. That last point is precisely how we know the system after a measurement has a new wave function and that it's different to the one before the measurement. Otherwise we'd have to just incoherently say the wave function describing the measuring apparatus "collapses" to a "non-physical" eigenfunction that describes the system but also doesn't describe it and magically we can just infer out of thin air that the system we measured also gets a new wave function different to the one before the measurement, it's simply absurd.

C: Contradicting the Well-Known Physical Interpretation

Now let me note the physical interpretation of non-normalizable free particle eigenfunctions.

This is given in reference ([1] Sec. 10) for example: the non-normalizabilty of a free particle eigenfunction just corresponds to, and vitally depends on, the fact that on an unbounded domain the system can spend time at 'infinity', i.e. an 'infinite motion' can occur, something that obviously can't occur on a bounded domain, and the non-normalizability is an absolute necessity in order to be able to give this physical interpretation.

Sketching the argument in [1]: The integral $\int dq |\Psi(q)|^2$ diverges for a stationary state because $|\Psi(q)|^2$ (in this case) does not become zero at infinity. In other words, the probability density at a point at infinity does not become zero, so the particle represented by that stationary state could potentially be found at this point at infinity if the probability interpretation is to make sense. If we now assume the position $q$ is known, or rather known on average, we should now allow for the eigenvalue associated to that location to not be known, and so we should take a linear combination of stationary states in the continuous spectrum at that point, $\Psi(q) = \int dE a_E e^{-iE t} \psi_E(q)$. We can interpret the continuous spectrum eigenfunctions at this point on average (with respect to time) by squaring $|\Psi(q)|^2 = \int dE dE' a_E a_{E'}^* e^{-it(E - E')} \psi_E(q) \psi_{E'}^*(q)$ and then averaging over time. Because it involves a Dirac delta function in the continuous spectrum case, the time average will therefore go to zero (in the discrete spectrum case this instead remains finite). Thus the time average of the probability density at any point is zero only in the continuous spectrum case, i.e. the average probability density for finding a particle at any point is zero. This only makes sense if the particle exists on an infinite domain.

But what's this? A physical interpretation of the non-normalizable eigenfunctions that is directly due to the non-normalizability? This is supposed to be impossible...

In other words, just because a particle can spend time at infinity on an infinite domain, it doesn't change a thing about the fact that quantum mechanical wave functions should still be able to describe the fact that they can do this i.e. that a free particle on an unbounded domain can be found at infinity. To claim QM can't describe this is to artificially limit what QM can do due to one's own bias for discrete probability spectra and to ignore the lesson of the difference between 'classical' discrete vs continuous probability distribution theory that the latter case needs to be treated carefully instead of just being thrown out.

One actually has to deny the Heisenberg Uncertainty Principle to say otherwise. Recall that the HUP tells us if we know the momentum precisely, the position is completely unknown, in other words, there is no reason why the free particle couldn't be located 'at infinity'.

One has to actually claim that quantum mechanics is not equipped to be able to say that a free particle with precisely known momenta can potentially be found at infinity using a wave function, even though it's exactly what the HUP tells us should happen.

Indeed even Dirac ([5], Sec. 48) also very clearly gives this physical interpretation, after arguing in a sentence the time-average argument above that a particle on an unbounded domain "spends nearly all it's time at infinity" and then shows how essential it is that the norm diverges in this case order for the relative probability interpretation to make sense, agreeing with the arguments in ([1], Sec. 2 and 10) and the quote from [8] mentioned earlier.

D: Other Comments

The fact people misinterpret this is not so different to the fact that discrete probability distribution theory is slightly different to continuous probability distribution theory, with the latter taking well over a century for it's basic principles to be fully axiomatized, intuition is simply lost in going to the latter, the same loss of intuition is clearly occurring in the quantum case. For classical continuous probability distribution the probability of a single value is zero. This doesn't mean a single outcome in a classical probability experiment is not 'real/physical'.

It's simply not surprising that in quantum mechanics the naive use of the Born rule also 'breaks down' for a single/discrete set of eigenvalues when a continuous spectrum exists and we need to be more careful only in this case. That in no way means we should deny the completely obvious fact that something as simple as an individual free particle whose energy can in principle be precisely (and so as a consequence the position cannot) known exists and has to be describable by quantum mechanics.

On a mathematical level, the belief seems to be that 'rigged Hilbert spaces' are the natural realm where 'continuous spectrum wave functions' can legitimately be called wave functions [3], because these are the spaces where delta functions can be treated properly. Whether it's actually the case that this is the correct space where one can use the obvious physical interpretations sketched above is another question, I'm not sure. For example [4] says of a rigged Hilbert space:

"It allows vectors of infinite norm to be accommodated within the formalism, and eliminates the vagueness that often surrounds the question whether the operators that represent observables possess a complete set of eigenvectors (from the preface);

These two examples suffice to show that rigged Hilbert space seems to be a more natural mathematical setting for quantum mechanics than is Hilbert space (p. 29)."

Finally: if we acknowledge that the obvious free particle eigenfunctions are physical, it means we have to recognize that a devotion to 'separable Hilbert spaces' actually denies physical properties of systems like that a free particle with precisely known momenta obviously could be found ('spend time at') at infinity (as described above). A devotion to separable Hilbert spaces was famously used as an objection to loop quantum gravity. Clearly, at least from the above perspective shared by Dirac, Landau etc..., this is just a very-flawed/bad argument against loop quantum gravity that, in being wrong, actually just enables defenders of it to paint the whole thing as wrong. Indeed the 'testability of the discrete area spectrum' argument there is just as bad as e.g. writing off classical mechanics for our not having infinite accuracy as mentioned above, it's the kind of weak argument used to deny physical predictions about free particles given above and it just doesn't address the theory by the standards of judging other theories. This for example seems like a far stronger argument against it that addresses it's own internal logic.

References:

1. Landau and Lifshitz, "Quantum Mechanics", 3rd Ed.;
2. Griffiths, "Introduction to Quantum Mechanics", 2nd Ed.
3. nlab: "rigged Hilbert space".
4. Ballentine, "Quantum Mechanics, A Modern Development", 1st Ed.
5. Dirac, "Principles of Quantum Mechanics", 4th Ed.
6. Kramers, "Quantum Mechanics", 1st Ed.
7. Born, "On the Quantum Mechanics of Collisions" (1926), J.A.W., W.H.Z. translation (1981).
8. Bransden and Joachain, "Quantum Mechanics", 2nd Ed.
9. von Neumann, "Mathematical Foundations of Quantum Mechanics", 1st Ed.
10. Bell, "Against Measurement", 1990 Phys. World 3 (8) 33.

Answer 3

Thus this is not a valid solution

The solution is a mathematically valid solution but cannot be used to model a particle with a specific (x,y,z,t) .

But how is this possible since we already know that we cannot use separable of variable method to solve the Schrodinger equation

If one uses a different $k$ in the solutions of the plane wave, one has many plane waves that can be used to model a free particle with the wave packet solution of wave equations, giving the quantum mechanical uncertainty to the momentum of the particle.

This is a useful way to think of modeling free particles in quantum mechanics, but fortunately it is not needed when a potential exists with which the particle interacts. There the model is direct solution of the appropriate equation with the potential, or the use of Quantum field theory and Feynman diagrams that allow to fit and predict interactions of particles.

Answer 4

It's certainly true that a generic state vector $\psi_t$ cannot be written in the form $\psi_t = f(t)\phi$ for some vector $\phi$ in the Hilbert space. In $2$ dimensions, for example, one might have something of the form $\psi_t = \pmatrix{e^{i\omega t}\\ e^{-i\omega t}}$, which cannot be written as a time-dependent scalar function multiplying a constant vector.

However, if you choose a fixed basis $\{\phi_n\}$, then any vector can be expressed as a linear combination $\sum_n c_n \phi_n$. In particular, at each instant the state vector can be expressed in this way, with the coefficients changing from moment to moment. As a result, the fact that the time-dependent state vector can be expressed as $\psi_t = \sum_n c_n(t) \phi_n$ is an essentially trivial statement - after all, the $\phi_n$'s are a basis for every $t$ - and separability as you define it becomes a non-issue. Choosing the basis $\{\phi_n\}$ to be eigenvalues of the Hamiltonian makes solving the Schrodinger equation similarly trivial, yielding $c_n(t) = c_n(0) e^{-iE_n t/\hbar}$.

When the spectrum of the Hamiltonian is discrete, then an orthonormal basis of energy eigenvectors is guaranteed to exist, and so the aforementioned procedure is perfectly well-defined. However, when the spectrum is continuous, this is no longer true, as in this case $H$ has no eigenvectors. In our introductory course, we learn that if we forget about the requirements of normalizability, there do exist functions $\phi_k$ which behave somewhat like eigenvectors; furthermore, we can expand geniune, normalizable states as integral superpositions $$\psi_t = \int c_k(t) \phi_k\mathrm dk$$

To be sure, this procedure works just fine, but one may be led to ask how we can justify it. There are essentially two rigorous routes one could take - one developed by John von Neumann, and the other by Israel Gelfand. Gelfand's approach is a formalization of Dirac's heuristic procedure, but a technically detailed discussion of it requires the development of a fairly large additional body of machinery beyond the Hilbert space itself; as a result, I focus here on justifying the "generalized eigenvector" procedure through the technically simpler lens of von Neumann.

---

In this approach$^{1}$, we take as our Hilbert space $\mathscr H := L^2(\mathbb R)$. Our Hamiltonian $H$ with domain $\mathrm{dom}(H)$ is given by $$\mathrm{dom}(H):= \{\psi\in L^2(\mathbb R) \ : \ \psi \text{ is twice weakly-differentiable and }\psi''\in L^2(\mathbb R)\}$$ $$\big(H\psi\big)(x) := -\frac{1}{2}\psi''(x)$$

This domain is also called the Sobolev space $W^{2,2}(\mathbb R)$. $H$ is self-adjoint and has a purely continuous spectrum given by $\sigma(H) = [0,\infty)$. The latter fact implies that it has no eigenfunctions - that is, there are no elements $\psi\in \mathrm{dom}(H)$ which satisfy $H\psi = \lambda \psi$ for some $\lambda\in \mathbb C$. However, the fact that it is self-adjoint implies that there is a unitary operator $U$ such that $\widehat H = U^\dagger H U$ is an $\mathbb R$-multiplication operator, where a multiplication operator $M$ is one such that $\big(M\psi\big)(x) = m(x) \psi(x)$ for some function $m$; this is one of the three equivalent statements of the spectral theorem for self-adjoint operators.$^{2}$

In this case, we may let $U$ be the Fourier operator $$\big(U\psi\big)(k)= \frac{1}{\sqrt{2\pi}} \int_{\mathbb R} \psi(x) e^{-ikx} \mathrm dx$$ It is not difficult to see that $$\big(\widehat H f\big)(k) = \frac{k^2}{2} f(k)$$ and so, given any $\psi\in \mathrm{dom}(H)$, we can equivalently write $$\big(H\psi\big)(x) = \frac{1}{\sqrt{2\pi}} \int_{\mathbb R} \frac{k^2}{2} \tilde\psi(k) e^{-ikx} \mathrm dk$$ where $\tilde \psi \equiv U\psi$ is the Fourier transform of $\psi$. Via the time-independent propagator, time evolution is given by

$$\psi_t = e^{-iHt}\psi_0 = \frac{1}{\sqrt{2\pi}} \int_{\mathbb R} e^{-i\frac{k^2}{2} t} \tilde \psi_0(k) e^{-ikx} \mathrm dk$$ which is obtained straightforwardly by exponentiating the multiplication operator - that is, if $\big(M\psi\big)(x) = m(x)\psi(x)$, then $\big(e^M\psi\big)(x) = e^{m(x)}\psi(x)$. Equivalently, the Schrodinger equation yields $$i\frac{d}{dt}\psi_t = H\psi_t \iff i\frac{d}{dt}\tilde\psi_t=\frac{k^2}{2}\tilde \psi_t$$ $$\implies \tilde\psi_t(k) = e^{-i\frac{k^2}{2} t} \tilde \psi_0(k)$$

---

The fact that $U$ is in this case given by the Fourier operator should be clear to anyone with experience in Fourier analysis. However, for a general Schrodinger Hamiltonian $H:= -\frac{1}{2}\frac{d^2}{dx^2} + V(x)$ it is not so trivial a problem to solve. The correct procedure is to postulate that

$$\big(U\psi\big)(k) = \int_{\mathbb R} \psi(x) \phi_k(x) \mathrm dx$$ for some $\phi_k(x)$. The fact that $U$ should be unitary implies via a few lines of algebra that $$\int_{\mathbb R} \overline{\phi_k(x)} \phi_q(x) \mathrm dx = \delta(k-q) \qquad \int_{\mathbb R} \overline{\phi_k(x)} \phi_k(y) \mathrm dk = \delta(x-y) \quad (\star)$$ Once again defining $\widehat H = UHU^\dagger$ (which, via the ansatz, is a multiplication operator) and $\tilde \psi = U\psi$, we then have $$\big(\widehat H \tilde \psi\big)(k) = E(k) \tilde \psi(k) = \int_{\mathbb R} \big[-\frac{1}{2}\psi''(x) + V(x)\psi(x)\big]\phi_k(x)\mathrm dx$$

This must hold for arbitrary $\psi\in\mathrm{dom}(H)$; integration by parts then tells us that $$-\frac{1}{2}\phi_k''(x) + V(x) \phi_k(x) = E(k) \phi_k(x)$$

Solving this differential equation subject to the $\delta$-normalization and orthogonality conditions $(\star)$ provides the correct form for $\phi_k$, and therefore $U$. In the free particle case, we find that a perfectly good choice is given by $\phi_k(x) = e^{ikx}/\sqrt{2\pi}$ with $k\in\mathbb R$. Crucially however, these $\phi_k$'s are not elements of $L^2(\mathbb R)$ in general.

This procedure is explained$^3$ in detail in many places in the physics literature. However, the unitary transformation $\psi(x) = \big(U^\dagger f\big)(x) = \int_{\mathbb R} f(k) \overline{\phi_k(x)} \mathrm dk$ is typically described as an expansion of $\psi(x)$ in the continuous (generalized) eigenbasis $\{\overline{\phi_k}\}$. This provides intuition for what physical significance the $\phi_k$'s should hold, but it must be quickly clarified that they do not constitute valid physical states for the system to occupy$^\ddagger$.

---

To recapitulate, self-adjoint operators with purely continuous spectra do not have eigenfunctions. However, they are unitarily equivalent to multiplication operators, and in this way they can be fairly easily understood. In order to find the correct unitary transformation, we find ourselves solving a differential equation which looks just like the eigenvalue equation associated to the Hamiltonian except for the fact that (i) the eigenvalues form a continuum rather than a discrete set, and (ii) the "eigenfunctions" $\phi_k$ are not constrained to live in $L^2(\mathbb R)$. We often call the $\phi_k$'s generalized or non-normalizable eigenstates.

For completeness, Gelfand's approach is called the rigged Hilbert space formalism$^4$ in which these $\phi_k$ are understood as (the kernels of) distributions over a suitable space of test functions; this description requires substantially more machinery (nuclear spaces, distribution theory, etc) to make technically rigorous. In that way it is somewhat similar to the use of the hyperreal number system to formalize infinitesimals in calculus - simpler and somewhat more intuitive on the surface, but not quite so simple on a technical level.

---

$^\ddagger$There are a number of ways to define a quantum state, but the bare minimum requirement is that they give rise to a probability distribution for any possible measurement; that is, they must provide some way to answer the question, "what is the probability that I measure the (self-adjoint) observable $A$ to lie in the (Borel-measurable) set $E\subseteq \mathbb R$?" One way to define such a probability distribution is through rays in the Hilbert space underlying the theory, while a more general way is provided by the positive trace-class operators with unit trace; for the even more mathematically ambitious, this can be generalized even further. However, it becomes clear with a bit of work that the generalized eigenstates obtained here do not fall into any of these categories. As a specific example, a fully-delocalized plane wave does not provide a way to answer "what is the probability of measuring the position of the particle to be in the interval $E$?" which satisfies the demands we place on probability distributions.

References:

[1] Hall, Quantum Theory for Mathematicians

[2] For a comprehensive tour de force in spectral theory and the mathematically rigorous foundations of quantum mechanics, see Spectral Theory and Quantum Mechanics by PhysicsSE's own Valter Moretti.

[3] See e.g. p.15 of Landau & Lifshitz, A Course in Theoretical Physics, Vol. III: Quantum Mechanics

[4] A. Bohm, M. Gadella, Dirac Kets, Gamow Vectors and Gel'fand Triplets, Section IV