I wouldn't say that either of the linked papers "claims the discovery of a fifth force." What we have here is an subtle experimental observation with a very interesting interpretation that can't yet be ruled out, plus a path forward to search in data from upcoming experiments.
The reaction chain of interest in the PRL by Krasznahorkay et al. is \begin{align} \rm^7Li + p &\to{} \rm^8Be^* \\ \rm^8Be^* &\to{} \rm^8Be + \gamma \to{} ^8Be + e^+e^- \\ \rm^8Be &\to 2\alpha \end{align} The gamma rays in the second step are unusually energetic, about 18 MeV. There is some small chance that the gamma ray emitted in the relaxation of the excited nucleus $\rm^8Be^*$ will interact with the electric field around that nucleus to produce an electron-positron pair. These lepton pairs are overwhelmingly likely to leave the nucleus traveling nearly parallel to each other, so that the angle between their momenta is nearly zero. For the $\rm e^+e^-$ pair to be emitted back-to-back, this interaction between the relaxation photon and the nuclear electric field would have to take place in the rest frame of the nucleus. The ratio of pairs separated by 40° to pairs separated by 180° is about a factor of twenty.
The subtlety is that, for a narrow range of proton energies, there are perhaps 30% extra $\rm e^+e^-$ pairs separated by about 140°. It's not impossible this is some acceptance effect in the detectors, since the detector geometry has a feature at 150°. However one would naïvely expect the same sorts of acceptance issues at the other special angles in the detector, at 60°, 90°, and 120°; those are absent. Further arguing against a geometrical detector anomaly, the excess is present at some proton energies but absent at lower and higher energies. However a model where second line in the reaction chain is \begin{align} \rm^8Be^* &\to{} \rm^8Be + \mathit X \to{} ^8Be + e^+e^- \end{align} has a most probable $\rm e^+e^-$ angle which depends on the mass of the $X$. The authors find a best fit to their data a small contribution from $X$ decays for $m_X \approx 17\rm\,MeV$. This is also in the right range for resonant production of the $X$, which would explain why the extra angular correlations are only present at some energies. The branching ratio for $X$ production is quite small.
Krasznahorkay et al. also cite previous observations of this phenomenon, anomalous angular distributions in internal pair production from high-energy nuclear gamma decays, from another group in the era 1996--2001, so the phenomenon is not a bolt from the blue. The abstracts for these older papers suggest a quite different mass for the putative particle; I haven't read to see what is the difference.
A follow-up paper by Feng et al. (now also in PRL) compares this putative particle with known, constrained, and unconstrained particles. Since production of the $X$ seems to proceed from one excited state in $\rm^8Be$ but not another, we know something about its quantum numbers. It can't be a pion, since it's too light by a factor of ten. And it's highly unlikely to be any hadron that's composed of quarks, since the pion is the "massless" Goldstone boson of QCD and the pion mass sets the floor of the strong interaction's mass spectrum. Feng et al. compute the ranges of couplings of the $X$ to electrons, neutrinos, up and down quarks, and nucleons which would permit the $X$ to contribute to $\rm^8Be$ decay but have kept it hidden from a list of about a dozen previous experiments. Feng et al. then take those coupling ranges and claim, in their final figure, that the $X$ should be within the sensitivity ranges of at least five upcoming experiments (looks like the Heavy Photon Search would miss it).
Both of these are soundly reasoned papers which survived peer review and have appeared in the flagship journal of the American Physical Society. There is repeatable (possibly already repeated) evidence for an anomalous result in one system which is consistent with a new force-carrying particle, and theoretical guidance on how this particle would look in other, completely different experiments which have not been completed yet. If we were going to find a new fundamental interaction this is exactly how the early stages would look and it's absolutely worth pursuing further. All those positive comments in mind, I give it a 90% chance that it evaporates under further scrutiny; nothing resembles a new effect quite so much as a mistake.
DavePhD asks me to re-evaluate this answer in light of a new (October 2019) preprint by the Krasznahorkay group which reports evidence for the $X$ in a different small-nucleus reaction,
$$\rm ^3He + p \to {}^4He + \gamma$$
This is an interesting choice because photon emission from excited $\rm^4He$ states is mostly forbidden. (That link includes this $\rm^4He$ level diagram.) The authors explain that
[t]his bombarding energy [$E_\text{proton} = 900\rm\,keV$] is below the threshold of the $\rm(p,n)$ reaction ($E_\text{threshold}=1.018\rm\,keV$) and excites the $\rm^4He$ to $E_x=20.49\rm\,MeV$, which is below the centroid of the wide $0^-$ state.
In other words, the forbidden photon emission is the only way for this nucleus to relax. The photon emission is forbidden not only for the strong-isospin reasons answered in the linked question above, but for the more fundamental reason that transitions from spin and parity $0^-\to0^+$ are forbidden by photon selection rules --- the photon should carry away one unit of angular momentum. However, a pseudoscalar particle with $J^\pi = 0^-$ could be emitted in this transition without violating angular momentum or parity selection rules.
The authors report strong evidence of a signal excess in $\rm e^+e^-$ pair production at the energies and angles consistent with a pseudoscalar boson with mass $m_xc^2 \approx 17\rm\,MeV$. And that $\approx$ is mine, not theirs. The authors report two fits, whose differences aren't immediately clear to me, with mass centroids of $16.84\pm0.16\rm\,MeV$ and $17.00\pm0.13\rm\,MeV$.
This new result has not yet been peer-reviewed, and peer-review isn't what I'm doing in this answer: I have some technical questions about the paper that would take a few days of thinking to answer. But the new result is consistent with the previous paper on beryllium. I would be much more excited about it if it came from a different group, or if there were a discussion of a blind analysis technique. Absent those, I'm not prepared to call this a discovery. However, the final page of the preprint is a list of about five independent experiments that will also be sensitive to this particle. Apparently the community is treating this phenomenon with the seriousness that it deserves.
Rob's answer is more or less complete for the stage of data we are aware of. New experiments are needed to see if a 17 MeV particle exists in e+e- , as the existing data in the particle data group stop at about 100 MeV.
I want to address the title question:
Is the “protophobic fifth force” observed in nuclear decays a new fundamental force?
the difference between "fundamental" force and force in general in particle interactions.
Force in the framework where quantum mechanics reigns can still be defined as dp/dt. Any two particle scattering will either transfer momentum or be elastic. A force is exchanged between initial state and final state measured particles if there is a momentum transfer. The crossections are calculated using the iconal representation of Feynman diagrams. These are integrals in a perturbative expansion series, each order dependent on coupling constants., in diminishing importance.
The standard model of particle physics , which describes successfully the enormous plethora of data gathered the last fifty or so years,depends on three fundamental forces. They are called fundamental because the lowest order diagrams, contributing the most to the crossections, are dependent on the corresponding couplings of electromagnetic forces, weak, or strong. These lowest order diagrams have as an exchanged particle a photon, a Z/W , a gluon corresponding to the force characterization of the interaction. These are identified with the gauge bosons in the SU(3)xSU(2)xU(1) of the standard model.
There is a hypothesis that the gravitational force, once quantized , will have the graviton as the exchanged particle, but this is still a research project.
The SM model is based on a plethora of data and it is doubtful that it can be upset by the introduction of a "fifth force" when just an exchange feynman diagram with this possible X being a complex quark antiquark particle might explain the interaction given an appropriate format. After all, all particles can be exchanged and transfer a dp/dt without introducing any fundamental forces. See my answer here.
There is naturally the question of where would this new boson get its mass from. One could guess that if other SM particles are millicharged under this new symmetry, the Higgs would also have a non-zero charge (should be micro-charged to get to the right ballpark) and that would explain the smallness of the mass.
– Janus Boffin May 27 '16 at 07:52