“...while gauge fields go in the adjoint rep”.
It is true that gauge fields are always in the adjoint, except when the gauge group is Abelian (e.g. $U(1)$ EM) in which case the adjoint is just the singlet (or 'trivial') representation. This is why the photon is neutral.
The reason is that gauge fields are defined as connections - they are defined by the covariant derivative $D = d + A$. The gauge field encodes how an object transforms when it is parallel transported an infintesimal difference, hence it is an infinitesimal generator - a Lie algebra element. The adjoint representation is simply the action of a group on it's own generators.
The adjoint representation $\rho _{\rm adjoint}$ is the defined to act by conjugation on group elements $\rho _{\rm adjoint} (U) A = U A U^\dagger$ which is infinitesimally (for $U=e^{i T}$) given by $i \left[T,A\right]$. This is the 'natural' way for the group to act on itself because if $U$ is thought of as a linear basis transformation then $U A U^\dagger$ gives the form of $A$ in the transformed basis.
"...fermionic matter goes in the fundamental rep of $SU(N)$..."
This isn't true. It's true in QCD and in most of the standard model, but there is also a scalar in the standard model (the Higgs field) that transforms in the fundamental of $SU(2)_{\rm L}$. There are many other theories with fundamental scalars, adjoint scalars and Fermions (e.g. $\mathcal{N} = 4$ SYM), bi-adjoint scalars (e.g. ABJ(M)) etc. Also, most of these theories can be formulated with gauge groups other than $SU(N)$.
One note perhaps on the relative lack of scalars charged under gauge symmetries is that their presence will often cause the symmetry to be spontaneously broken (e.g. the Higgs).
"EDIT: my confusion is partially that I feel that a representation is just a representation, and as long as it is faithful (for instance not trivial) it is just a choice and I guess it shouldn’t affect the physics."
Indeed the term 'representation' is a bit misleading.
First, note that many reps that we use are not faithful. The adjoint rep maps all elements of the center of the group to the identity. For the Abelian group $U(1)$ the different reps correspond to different charges, so for instance for charge $n$ the rep is $e^{i \theta} \to e^{i n \theta}$ and therefore cannot distinguish between $\theta$ and $\theta + \frac{2 \pi}{n}$.
Besides that, the representation is significant in other ways:
- What interactions are construct-able? A fundamental scalar has the quartic $\left(\bar{\phi}\phi\right)^2$ vertex while an adjoint has 2 kinds of quartic: ${\rm tr}\left(\phi ^4\right)$ and ${\rm tr}\left(\phi ^2\right)^2$.
- What Casimirs show up in formulas? Feynman diagrams will include 'color factors' that involve the representations matrices $T^a _R$. Gauge invariant quantities will therefore involve Casimir invariants that depend on the rep, e.g. $\sum_a T^a _R T^a _R = C_R$.
- How does the symmetry break when the field in question acquires a v.e.v. (vacuum expectation value). A fundamental field will break the $SU(N)$ symmetry down to $SU(N-1)$ while an adjoint will break it down partially (usually to a bunch of $U(1)$s).
*Note: Previously I wrote that a fundamental field breaks the symmetry completely, which is incorrect. Thanks to Hosein for pointing it out.
There are probably other ways in which the rep is significant that don't come to my mind right now.
