Please, let me refer you first to Freed's popular lecture in which some general intuition on the role of topology in physics (specifically in quantum phases of matter) is given. The lecture is given only as a general reference; it does not refer to characteristic classes explicitly, and in fact, the classification of the quantum phases of matter requires a finer resolution than characteristic classes alone. But, I'll stick with the characteristic classes for the sake of simplicity. May be the most important statement in this lecture that topology is believed to be important mainly in the low energy limit of physical systems.
I'll base my description on quantum systems described by parameterized Hamiltonians. These Hamiltonians approximate the few low energy excitations of a system when there is a large gap from the higher excited states.
The Hamiltonians are parametrized by a manifold of parameters (parameter space). This parameter space can be fundamental, such as in the case of electron Hamiltonians in crystals parametrized by the Brillouin zone. But it can be also an approximation, such as in the case of the Born-Oppenheimer approximation, when the electron Hamiltonian is parametrized by the nuclear coordinates. Another possibility is that the parameter space is a control manifold of system parameters controlled by the experimentalist and can be changed during the experiment; for example they can be the frequencies of some exciting laser source.
In the very low energy limit, the ground states of these Hamiltonians become of the greatest importance. The ground states energies of these Hamiltonians will depend on the parameter space coordinates and form bands. In addition, these ground states are characterized by their eigenvectors whose dependence on the parameters is quite complicated. In the generic cases, these parametrized eigenvectors define line bundles on the parameter space in the case when the ground state is non-degenerate and vector bundles when the ground states are degenerate. These bundles will be associated to principal bundles $U(1)$ in the case of line bundles and non-Abelian Lie groups in the degenerate case of vector bundles.
Since characteristic classes classify principal and vector bundles, they classify, as a consequence, deformation classes of the parametrized Hamiltonian. This classification will tell us if we can deform two parametrized Hamiltonians into each other, which virtually means that they correspond to the same type of systems. For example, in the Quantum Hall effect we cannot continuously change the system's parameters to deform an integer Hall state into another integer Hall state with a different Chern number.
The (coarse) classification of the Hamiltonian deformation classes is based on the following commutative diagram:
$$\begin{array}{ccc}
\lambda& \overset{f^*}\leftarrow & \eta\\
\downarrow & & \downarrow \\
\mathcal{M} & \overset{f}\rightarrow & B
\end{array}$$
(This figure is taken from Bohm , Boya Mostafazadeh, and Rudolph).
Here, $M$ is the parameter manifold, $\lambda$ is the line or vector eigenbundle over $M$, $B$ is the classifying space of $\lambda$ and $\eta$ is the corresponding universal bundle. Thus the eigenbundle is obtained as a pullback $f*$of an embedding map $f$ of the parameter space in the classifying space. Also, the geometric phase along a closed curve in the parameter space will be obtained as a pullback of a holonomy on the universal bundle which can be expressed as an integral of a universal Berry curvature. Invariant polynomials of this Berry curvature consist of representatives of the characteristic classes.
When the Hamiltonian is real, then the classification will be by means of real universal bundles. When the Hamiltonian is quaternionic, the classification will be by means of quaternionic universal bundles. These cases correspond respectively to integer and half integer spin systems with time reversal symmetry.
Non-trivial Stiefel-Whitney classes emerge when the eigenbundle can be non-orientable. Please see the following work by Kaufman, Li and Wehefritz-Kaufman.
Now, there is an abundance of physical examples fitting the above description. The most known is the Dirac monopole bundle. I'll describe a few not very well known ones.
This is an example by Sonner and Tong: Just as the monopole bundle can be obtained as the eigenbundle of a fixed single spin in a magnetic field, where the magnetic field components serve as the parameter manifold. Sonner and Tong showed that a spinning particle moving on the surface of a sphere in the presence of a magnetic field has a doubly degenerate ground state and the Berry connection on the vector bundle of eigenstates is a t'Hooft-Polyakov monopole. These monopoles are classified by higher Chern classes.
In Johnson and Aitchison construct a Hamiltonian whose eigenbundle is the instanton bundle. The idea is to use higher spin generalization of the Pauli matrices in the form of gamma matrices.
Further examples of monopoles and instantons are given by Landi.
