I have relatively limited knowledge on the Chern number, and I know that there exists high-level math proofs that the Chern number is an integer, but let me try to focus on the case I have in mind. $\DeclareMathOperator{\Tr}{Tr}$
Let $\Omega(\phi)$ denote the normalized states in a Hilbert space where $\phi\mapsto \Omega(\phi)$ is smooth on some closed orientable 2D manifold $M$. Let $P(\phi)$ denote the corresponding projection operator, i.e., $P(\phi) = |\Omega\rangle \langle \Omega|$. Let me define the connection $A = -i\langle \Omega|d\Omega\rangle$. Then by Stoke's theorem, it seems that $$ \oint_\gamma A=\int_\text{interior}dA=-\int_\text{exterior}dA $$ where the interior and exterior of $\gamma$ in $M$ are determined by the orientation of $M$ and $\gamma$. Notice that $idA = \Tr (PdP\wedge dP)$ is gauge-invariant.
Now, the usual argument is that $\oint_\gamma A$ is the Berry phase and only defined module $2\pi$ and thus $\int_M dA \in 2\pi \mathbb{Z}$. However, I don't quite understand why. Indeed, $A$ is a well-defined differential form, and thus is well-defined without any knowledge of the Berry phase. Indeed, we seem to have $$ \int_M dA= \int_\text{interior} dA +\int_\text{exterior} dA=0 $$ And not $2\pi \mathbb{Z}$. Of course, I know this can't be quite true since there are simple examples such as the Bloch sphere. However, I'm having trouble seeing where I went wrong.
Follow-up. @Heidar' pointed out that $A$ is only locally defined. This can also be seen in the example of the Bloch sphere (my incomplete answer).
Now it seems that the only missing puzzle is why is the Chern number quantized. Indeed, in the Bloch sphere example, I can move the "singularity" of $A$ freely via a gauge transform. In this case, we can use properties of the Berry phase to rigorously show that the Chern number is quantized. However, in general, how would one "move such singularities"?