General analytic methods to prove non-integrability are discussed in e.g. this Phys.SE post. In this answer, we will sketch how to apply the following Poincare corollary.
Poincare corollary: If an autonomous Hamiltonian Liouville-integrable system has a periodic solution $z_0(t)=z_0(t+T)$, then the monodromy matrix for the linearized system along $z_0$ can only have 1 as a (generalized) eigenvalue.
A proof of the Poincare corollary is given in my Phys.SE answer here.
$\uparrow$ Fig. 1. Poincare maps in the $(y,\dot{y})$ plane with $(x,\dot{x})=(0,0)$ for various fixed energy-levels $E$ of the Henon-Heiles (HH) system. We consider below the outermost periodic orbit for $E=1/6$ and $E=1/12$.
Sketched analytic proof that the HH system is not integrable: The HH system has Lagrangian$^1$
$$\begin{align} L ~=~&T-V, \cr
T ~=~&\frac{1}{2}(\dot{x}^2+\dot{y}^2)
~=~\frac{1}{2}(\dot{r}^2+r^2\dot{\theta}^2), \cr
V~=~&\frac{1}{2}(x^2+y^2)+x^2y-\frac{1}{3}y^3
~=~\frac{r^2}{2}+\frac{r^3}{3}\sin 3\theta, \cr
x+iy ~\equiv~& re^{i\theta}. \end{align}\tag{1}$$
The EL equations are
$$ -\ddot{x}_0~=~x_0 +2x_0y_0, \tag{2}$$
$$ -\ddot{y}_0~=~y_0 +x_0^2-y_0^2.\tag{3} $$
We seek a periodic solution $y_0(t)=y_0(t+T)$ with $x_0(t)\equiv 0$. Then eq. (2) is automatically satisfied. The energy
$$ E~=~ \frac{1}{2}\dot{y}_0^2+\frac{1}{2}y_0^2 -\frac{1}{3}y_0^3 \tag{4}$$
is conserved. Consider $(x,y)=(x_0,y_0)+ (\xi,\eta)$, where $(\xi,\eta)$ is an infinitesimal fluctuation. The linearized EL equations
$$ \ddot{\xi}~=~-(1+2y_0)\xi ,\tag{5} $$
$$\ddot{\eta}~=~(2y_0-1)\eta,\tag{6}$$
decouple. [The eq. (6) corresponds to dimensional reduction to the 1D subsystem in the $y$-direction alone. This is manifestly integrable, so we already know that the corresponding 2 generalized eigenvalues for the monodromy matrix in the $y$-direction is $1$. Hence eq. (5) will be the actual integrability test.]
First attempt: $E=\frac{1}{6}$. The first integral (4) becomes
$$\dot{y}_0~=~\pm (1-y_0)\sqrt{y_0+1/2}, \qquad -\frac{1}{2}~\leq~y_0~\leq~1. \tag{7}$$
It has solution
$$ y_0(t)~=~ \frac{3}{2}\tanh u-\frac{1}{2}
~=~1-\frac{3/2}{\cosh^2u},\qquad u~\equiv~\frac{t}{2}.\tag{8} $$
Unfortunately the period is infinite $T=\infty$.
For the record, the linearized eqs. (5) & (6) become Legendre's DE
$$ \frac{d^2\xi}{du^2}~=~- 12\xi \tanh^2 u , \tag{9}$$
$$ \frac{d^2\eta}{du^2}~=~\left(4 - \frac{12}{\cosh^2 u}\right) \eta. \tag{10}$$
Second attempt: $E=\frac{1}{12}$. The first integral (4) becomes
$$\begin{align}\dot{y}_0~=~&\pm \sqrt{\frac{2}{3}(y_0-1)(y_0^2-y_0-1/2)}, \cr
\frac{1-\sqrt{3}}{2}~\leq~&y_0~\leq~\frac{1}{2}.\end{align}\tag{11}$$
It has a solution in terms of Jacobi elliptic functions
$$\begin{align} y_0(t)~=~& \sqrt{3}{\rm sn}^2u +\frac{1-\sqrt{3}}{2}, \cr
{\rm sn} u~\equiv~&{\rm sn}(u|m\!=\!2), \cr
u~\equiv~&\frac{t}{2\sqrt[4]{3}}.\end{align}\tag{12} $$
The linearized eqs. (5) & (6) become Lamé's DE on Jacobi form$^2$
$$ \frac{d^2\xi}{du^2}~=~\left(12-8\sqrt{3}- 24 {\rm sn}^2u \right) \xi \tag{13},$$
$$ \frac{d^2\eta}{du^2}~=~\left(24 {\rm sn}^2u - 12\right) \eta \tag{14}.$$
Eqs. (13) & (14) may in principle be solved analytically. At this stage we admittedly just plugged eq. (13) into Mathematica, and checked numerically that the corresponding 2 generalized eigenvalues for the monodromy matrix in the $x$-direction are far from 1. Hence the HH system is not integrable, cf. the Poincare corollary. $\Box$
References:
- H. Goldstein, Classical Mechanics, 3rd edition, Section 11.6.
--
$^1$ Although the $4\times 4$ monodromy matrix refers to the 4D phase space, it is convenient to perform most of analysis in the 2D configuration space, and postpone the Legendre transformation.
$^2$ One solution to eq. (14) is
$$\eta_1(u)~:=~{\rm sn}u~{\rm cn}u~{\rm dn}u. \tag{15}$$
Another independent solution can in principle be found from the formula
$$ \eta_2(u)~:=~\eta_1(u)\int^u\! \frac{du^{\prime}}{\eta_1(u^{\prime})^2}. \tag{16}$$
