In QFT, we are used to ignoring boundary terms because they don't affect the perturbative dynamics, but they need not be $0$.
In fact, if one performs the boundary integral on $\partial M = S^3$ (after approprietly Wick rotating to euclidean time) it can be shown that the boundary integral
\begin{equation}
k \propto \int_{\partial M}Tr(A\wedge dA+\frac{2}{3}A\wedge A \wedge A),
\end{equation}
is independant of the radius $R$ of the 3-sphere. It's not too hard to accept that this integral will depend only on the relationship between the gauge group $G$ and the boundary.
I won't go into details here (an excelent reference is David Tong's lecture notes on Gauge Theory), but the instanton number is characterised by $\pi_3(G)$.
From a physical point of view, the instanton number is intepreted as the class of mapping from the boundary 3-sphere into the gauge group. Field configuration belonging in different classes cannot be continuously mapped onto one another.
Hope this helps a little.