From what I am looking at, it appears you are applying Euler-Lagrange's equations to the Lagrangian of the system of a particle constrained on a certain curve of shape f(x). However, by doing this, you are minimising the action of this particle, and thus you are getting the equations of motion of the particle. Granted, if you where able to solve them in the general case, you would then be able to compute the time it takes for the particle to reach the bottom, and thus minimise by varying f(x), finding the form of the brachistochrone.
However, this seems a little bit overkill to me. Let's see if there is a better functional to consider other than the action of your system.
Just as a side note, the equation you derived seem to be wrong, the correct equation should be
$$\ddot{x}(1+f'^2)=-f'f''\dot{x}^2-gf'$$
Anyway, since Lagrangian formalism is all about minimisation, let's consider, what do we want to minimize here ? It's very straightforward, and it is the time it takes for the particle to fall down the ramp. Now, parametrising the form of the ramp as you did, how can we compute the time for the ball to reach the bottom ?
To cross the infinitesimal distance $ds = \sqrt{1+f'^2}dx$ at speed v takes a time dt given by :
$$ dt = \frac{ds}{v}$$
We're basically done ! The total time for the ball to reach the bottom will then simply be $T = \int dt = \int \frac{ds}{v} = \int \frac{\sqrt{1+f'2}}{v}dx$.
As we can see, we still miss an element to be able to start doing some lagrangian mechanics. Indeed, we have the speed $v$ that we would like to express as a function of $x$, so that we have an integral on $x$. To do this, we exploit the conservation of energy which tells us that $\frac{1}{2}mv^2+mgh = E$. If we set our y-axis to be directed downwards for convenience (so here, $0<f(x)<-a$) as well as set the Energy to be 0 at $x=0$, we obtain $v(x) = \sqrt{2gf(x)}$
Thus, we have the functional that we would like to minimize :
$$ T = \int \sqrt{\frac{1+f'(x)^2}{2gf(x)}}dx$$. We can strip away the constant $2g$ which plays no role (we just showed that the brachistochrone has the same shape here or on the moon, by the way !), and the problem can be formulated as a function of a Lagrangian :
$$ L = \sqrt{\frac{1+f'(x)^2}{f(x)}} $$
Note that here x plays the role of the time in the usual lagrangian, in the sense that the Euler equations are now :
$$ \frac{d}{dx}\frac{\partial L}{\partial f'} = \frac{\partial L}{\partial f} $$
Just for the record, $f(x)$ is usually renamed $y(x)$.
See if you have more luck working on this formulation of the problem, if not, tell me and I'll complete my answer !