In confining theories you only observe gauge singlets, as in QCD (hadrons), and you might think of the gauge fields (gluons) as "merely" brilliant indirect calculational devices, leading up to phenomena involving gauge singlet observables--but just try to describe gluon jets without nonsinglet gauge fields!
But what about SSB theories such as the EW interactions?
That is a very long story you have to read about... the "principle of complementarity" of Susskind. The confining phase and the Higgs phase are not as distinct as they appear... So you may define singlet composite operators, hence physical and gauge invariant, corresponding 1-to-1 to the nonsinglet spectrum particles of the SM.
See Sec 3 of T Banks and E Rabinovici (1979), "Finite-temperature behavior of the lattice abelian Higgs model", Nucl Phys B160 (2), pp 349-379 for a quick-and-dirty illustration. (I am sure there are nicer reviews out there, but I haven't paid due diligence to find them. The recondite technical underpinning is in E Fradkin and S Shenker (1979) "Phase diagrams of lattice gauge theories with Higgs fields", Phys Rev D19 (12), p3682.)
Basically, there is a "theoretical" cottage industry out there, fussing the conceptual quandaries you might fear you have stumbled on, Elitzur's Theorem, etc, but, in practical terms, the gauge-fixing monkey-see-monkey-do of standard QFT texts is still the best way to handle the SM and keep track of symmetries and degrees of freedom!