Gauge fields always live in the Lie algebra of their gauge group (or more precisely, in the adjoint representation of the Lie algebra, but in physics we only consider semisimple Lie algebras, whose adjoint representations are always faithful, so we tend to be sloppy about making that distinction).
The way I think about it, the fundamental information contained in a gauge field is the collection of Wilson lines $W_C$ for oriented curves $C$, which are matrices in the adjoint representation of the gauge group. These give the parallel transport map with respect to the chosen gauge, which allows you to compare quantities at different points in spacetime. Basically, they correct for the fact that your coordinate system might be "twisted". (If the curve $C$ is closed, then $\text{Tr } W_C$ is gauge-invariant and provides a measure of electromagnetic flux through the resulting "Wilson loop", which is useful for diagnosing confinement, among other things.)
Wilson lines are nonlocal, so it's more convenient to consider infinitesimally short Wilson lines connecting nearby points in spacetime, which are local. The gauge field is defined to be the deviation of an infinitesimal Wilson line from the identity:
$$A_\mu(x) := -i \lim_{\epsilon \to 0} \frac{W_{x \to x + \epsilon\, \hat{e}_\mu} - I}{\epsilon},$$
or equivalently, the derivative of the Wilson line in the $\mu$-direction:
$$A_\mu(x) := -i (\partial_{\mu'} W_{x \to x'})|_{x' = x}.$$
So basically the gauge field tells you how much the coordinate system (in a particular gauge) is twisting as you go in a particular direction, and the covariant derivative $D_\mu := \partial_\mu - i e A_\mu$ corrects that twist. Since the Wilson line is an element of (the adjoint representation of) the gauge group, its infinitesimal version is an element of (the adjoint representation of) the gauge group's Lie algebra.
Inverting the definition above gives the expression for a Wilson line in terms of the gauge field:
$$W_C := \mathcal{P} \exp \left( i \int_C A_\mu dx^\mu \right),$$
which is the usual formula for converting an element of a Lie algebra to the corresponding element of the associated Lie group.