Usually, one uses some argument related to finiteness energy/angular momentum/charges, etc. to show that $A \to 0$ at infinity. However, all of these arguments remain unchanged if $A \to pure~gauge$ so we could in principle allow such configurations.
Now, these configurations do actually have an effect on the path integral due to boundary effects. For instance, the gauge field couples to a conserved charge current through the term
$$
S \ni \int A_\mu J^\mu
$$
If $A$ is pure gauge
$$
S \ni \int \partial_\mu \epsilon J^\mu = \int \partial_\mu ( \epsilon J^\mu )
$$
Now, usually we drop such terms because we assume things fall-off sufficiently fast near the boundary. However, we have to be careful here. The current falls of as $r^{-2}$, but then the integration measure grows as $r^2$. Thus, if $\epsilon$ is finite at infinity (i.e. if the gauge field does NOT vanish at infinity), then such a boundary term is non-zero and leads to observable differences in the theory. It follows that including these modes in your phase space leads to interesting new physics.
One question you might ask whether including these modes is necessary, i.e. can we not consistently discuss all of physics without ever introducing these large gauge modes? The answer to this is -- as we have learnt through Strominger's work -- NO! These large gauge modes are canonically conjugate to soft photons. Consequently, if you have soft-photons in your phase space (which we always do in traditional QFT!), then we must also keep the large gauge modes. In other words, soft photons source the large gauge dof so we cannot consistently set one to zero without also doing the same to the other.