Some of the major challenges that heralded the need for quantum mechanics we're explaining the photo-electric effect, the double-slit experiment, and electrons behavior in semi conductors.
What are some of the predictions we can expect to see from a theory of quantum gravity?
What types of experiments have shown the necessity for a quantum gravity theory?
A quantum theory of gravity does make definite predictions. One such an example, which is the same for any theory of quantum gravity that reproduce GR at low energy, is the famous correction to the newton $1/r$ potential: $$ V(r)=\frac{M_{star}}{M_{Planck}r}\left(1-\frac{M_{star}}{M_{Planck}^2 r}-\frac{127}{30\pi^2}\frac{1}{M_{Planck}^2 r^2}+\ldots\right). $$ The last term comes from loops of graviton, so it is a genuine quantum gravity contribution, see e.g. Chapter 22.4 in Matt Schwartz book in QFT about it.
The problem is that these corrections from quantum effects are so tiny that is difficult to test them. However, as BICEP as reminded us (even if it will turn out to be wrong) very early cosmology is sensitive to the quantum effects of gravity and we can in principle detect them with present day technology.
I would say that there is not too much experimental evidence for a quantum theory of gravity yet, the reasons why such a theory is desirable are mainly of conceptual/theoretical nature. I will give a (likely to be incomplete) list of motivations for studying quantum gravity.
The list can go on, and I will add more points, if suitable.
I'll also throw in a few words about the issue. As TwoBs points out, the problem is the comparative weakness of gravity, which makes quantum effects only relevant under extreme conditions: Essentially black holes and the big bang. The problem with the former is that we cannot peek beyond the horizon, so it might be more fruitful to try to tackle the issue by looking at the cosmic microwave background (CMB).
Going back to your specific questions:
1. What are some of the predictions we can expect to see from a theory of quantum gravity?
Quantizing gravity predicts fluctuations in the gravitational field, which will be magnified by the expansion of space, in particular during the inflationary period right after the big bang. Certain models of inflation predict that we should see the evidence for quantum gravity literally written across the sky as polarization of the CMB due to gravitational waves, and the BICEP2 experiment might have seen this. It's still possible that they have severly underestimated the effect of foreground dust, but I cannot judge how likely that is - we'll have to wait and see until new data comes in.
2. What types of experiments have shown the necessity for a quantum gravity theory?
None, really - the need for a quantum theory of gravity stems almost entirely from theoretical considerations. While we can do some things with a semi-classical treatment of gravity (quantum fields in curved space-time with curvature sourced by the expectation value of energy-momentum) and use it to predict gaussian-distributed perturbations of the CMB, such a treatment is incompatible with the rules of quantum mechanics.
Personally, I'd also expect a quantum theory of gravity to change our understanding of the vacuum, illuminating the nature of dark energy and possibly dark matter and in particular improving upon the worst theoretical prediction in the history of physics.
If it's asymptotically safe, it can correctly predict the mass of the Higgs boson. (Acknowledgments to Daniel de França for emphasizing that this was a quantum gravity prediction.)
