Asymptotic symmetry of space-time corresponds to diffeomorphism transformation of physical space-time, which manifests as isometry near Conformal Boundary. Asymptotic symmetry can be defined using various frame-work : such as Gauge fixing approach, geometric approach and Hamiltonian approach (see [1]). In the gauge fixing approach, one can find similarities between role played by electro-magnetic potentials in $U(1)$ theory and space-time metric in gravity. Actually, it is more transparent in principal Lie group bundle formalism where the diffeomorphism group for gravity $GL(3+1,R)$ plays the same role of gauge group as $SU(N)$ for Yang-Mills theory.
However, even in classical gravity, it is possible to construct toy models such as propagating shock waves (with reasonable energy conditions) which induces diffeomorphism transformation of physical co-ordinates (see [2],[3]), or consider any tensor field (e.g. velocity field), whose components will change under diffeomorphism. So observables will depend on choice of gauge, unlike in electro-magnetism.
In quantum field theory, it is known that the $S-$matrix for $U(1)$ theory is gauge invariant and is valid for all orders in perturbative expansion. However, for pure gravity (such as Christodoulou and Klainerman space-times), it has been shown that the quantum gravity $S-$ matrix is invariant under a certain subset of the total asymptotic group (see [4]). Based on similarities with $SU(N)$ theories in gauge theory, we should expect that the gravitational $S$-matrix to be invariant under total asymptotic group. While, the shock wave argument shows that it is possible to physically distinguish two choice of gauge in case of gravity, thus $S$-matrix need not be invariant under full symmetry group.
I apologize for the above convoluted arguments, however, I do not understand the frame-work based on which one can identify whether a certain quantity is observable or not. The close similarity b/w role played by diffeomorphism group and SU(N) group in principal lie group bundle formalism or in gauge fixing approach gives the impression that observables in gravity should not depend on a particular frame, while toy models such as shock waves and $S$-matrix argument implies that this conclusion is not true. The fact that only a subset of the total asymptotic group acts as symmetry group for $S$-matrix, it could imply that a certain subset of the total $GL(3+1,R)$ group should act as the unphysical gauge, on which our observables won't depend. How do we choose such a subset? Can there be a classical interpretation for choosing such a subset?
References:
[1]https://arxiv.org/abs/1910.08367
[2]https://ui.adsabs.harvard.edu/link_gateway/1985NuPhB.253..173D/doi:10.1016/0550-3213(85)90525-5