If a photon gets deflected at the first beam splitter for whatever
reason, it gets deflected at the second BS as well for the same
reason.
The accepted term for this idea is that the photon has a "hidden variable", some aspect or state that we don't know about (and so can't test for and can't include in our frameworks of physics) but which determines whether the photon reflects from or passes through the beam splitters.
The problem is that there cannot be a consistent set of hidden variables underpinning quantum mechanical behaviour that are both a) counterfactual-definite, meaning that the classical result you observe "was there all along", and would have still been there if you hadn't made the measurement you did, b) "local", not involving influences travelling faster than light. That might not sound important, but in a Relativistic context, "travels faster than light" is equivalent to being able to travel backwards in time and most physicists want to avoid that notion if at all possible. For more detail on how we've ruled out hidden variables generally, the terms to look for are Bell's theorem and the EPR experiments that show Bell's theorem does not apply to reality.
As for why the interferometer in particular cannot be explained by hidden variables, there's an extension of the experiment you described that's nicknamed the "quantum bomb tester" and looks like this:

The experiment works like this: a "bomb" that is supposed to be triggered to explode by absorbing a photon is placed at point B. However, some of the bombs are broken in such a way that they do not absorb the photon and do not explode, letting the photon continue unimpeded. This means that a photon passing the bomb may or may not reach the second beam splitter depending on whether or not the bomb is functional. This in turn means (per quantum mechanics) that the probabilities of reaching each of the two final detectors depends on whether bomb is functional.
When you work through all the numbers, you find that the probability of detecting the photon at detector D is non-zero if and only if the bomb is functional. Therefore if you do find the photon there, you know that the bomb is functional - but also that your photon didn't interact with it, because it didn't explode either. (There is a seperate possibility it does explode, but you can make that arbitrarily unlikely with a more complex setup.) The bomb's state is influencing the photon despite not "really" interacting with it, i.e. not exploding, which cannot be explained by classical unique paths or hidden variables.