The underlying framework of the physical world we have studied with our experiments is definitely quantum mechanical. The classical particles and waves emerge from this quantum mechanical level in a mathematically smooth fashion.
If one wants to go to the basic framework for light, one has to use quantum mechanics. Quantum mechanics relies on postulates imposed on the self consistent mathematical functions in addition to the mathematical axioms, in order to connect measurements to the mathematics. Crucial to QM is the Born rule
is a law of quantum mechanics which gives the probability that a measurement on a quantum system will yield a given result. It is named after its originator, the physicist Max Born.
It can be shown that classical electromagnetic waves are built up by a confluence of photons, but this needs the formalism of quantum field theory, which is based on the quantum mechanical postulates . A photon's wavefunction is derived quantum mechanically by a quantized Maxwell's equation and it should not be surprising that the solutions merge at the classical limit.
Note, a single measurement. In the case of the double slit single photon experiment it is the probability of a point being seen at the screen's (x,y).

Viewed quantum mechanically that is what the interference pattern is, the distribution of photons builds up a probability density.
It is only "setting up" for a measurement.
as you say.
Why does interference go away when polarized differently if this is the case?
In principle this probability density is the solution of the quantum mechanical boundary value problem " photon scattering on two specific slits". As the solutions of quantum mechanical equations are mostly sinusoidal, and sines and cosines when functions are added or subtracted usually show interference patterns, it should not be surprising that when the probability of measurement depends on sinusoidal distributions, interference patterns may appear or go away , depending on the boundary conditions, in this case the ones imposing polarization.