What the lecturer said seems rather obviously false. For instance, in dimension $2$ there are already up to homeomorphism infinitely many orientable, connected, compact manifolds. The Euler characteristic (or equivalently the genus) is a topological invariant that distinguishes them. If two manifolds are not homeomorphic they are definitely not diffeomorphic. If you allow the manifold to be disconnected then even in dimension $1$ disjoint unions of circles yield an infinite family of compact, orientable manifolds.
What the lecturer could have been referring to is that $\Bbb R^n$ has, up to diffeomorphism, exactly one smooth structure (the "standard" one) in any dimension $n\neq 4$. In dimension $4$, it follows from work by Freedman (on topological properties of $4$-manifolds) and Donaldson (on properties of $4$-manifolds that can detect the smooth structure) that there are uncountably many nondiffeomorphic smooth structures on $\Bbb R^4$: there are many "fake" or "exotic" $\Bbb R^4$'s.
The work by Donaldson has close connections to physics (one proves his main results using Yang-Mills theory or Seiberg-Witten theory, which provide invariants that can distinguish nondiffeomorphic smooth structures). However, there is currently no mainstream physical interpretation for the fact that there are fake $\Bbb R^4$'s. It is not clear that there is any physical relevance to this fact.
As a fun aside, exotic spheres are also a research topic in mathematics. The paper I just linked shows that in dimensions $5-61$, most spheres admit exotic structures. Thus, the phenomenon of "exotic structures" on manifolds is far from unique to dimension four. Note that the problem is open in dimension $4$. As pointed out in the comments by ACuriousMind, there is a paper by Witten that gives an attempt at physically interpreting exotic spheres, though I don't know anything about his arguments.
The usually cited reason why dimension $4$ is special has to do with certain differential-geometric theorems of extraordinary power that require "wiggle room", which causes them to work only in dimensions $\geq 5$. As far as I know, the h cobordism theorem by Smale is one of the main examples of this phenomenon. Thus, $n=4$ is the largest dimension in which "high-dimensional methods" do not work. Dimensions $1$ and $2$ are a bit easier to work with because the phenomena occurring there are comparatively "tame". From a geometric and topological point of view, things start to get really interesting in dimension $3$, while $4$ is "wild". I don't know much more about it than that, but there are several excellent books on $3$- and $4$-manifolds (for the former, I know especially Thurston's book is great, while the latter has books by Donaldson, Freedman, Freed & Uhlenbeck, Gompf & Stipsicz, etc.).
