Remember the definition $D(\alpha)=\exp(\alpha a^\dagger - \alpha^* a)$. Using BCH we can rewrite this in two ways:
$$D(\alpha) = e^{-|\alpha|^2/2}e^{\alpha a^\dagger}e^{-\alpha^* a} = e^{|\alpha|^2/2}e^{-\alpha^* a}e^{\alpha a^\dagger}.\tag1\label1$$
The matrix elements of the displacement operator in the number basis read
$$\langle n|D(\alpha)|m\rangle = \frac{1}{\sqrt{n! m!}}\langle a^n D(\alpha) a^{\dagger m}\rangle,$$
where $\langle \cdot\rangle$ denotes the vacuum expectation value.
We can go two ways for this calculation, using the two different ways to write $D(\alpha)$ given in \eqref{1}.
Using $D(\alpha) = e^{-|\alpha|^2/2}e^{\alpha a^\dagger}e^{-\alpha^* a}$
We have
$$\langle a^n e^{\alpha a^\dagger}e^{-\alpha^* a} a^{\dagger m}\rangle
= \sum_{i=0}^m\frac{(-\alpha^*)^i}{i!}\langle a^n e^{\alpha a^\dagger} a^i a^{\dagger m}\rangle
= \sum_{i=0}^m\sum_{j=0}^\infty \frac{(-\alpha^*)^i\alpha^j}{i!j!}\langle a^n a^{\dagger j} a^i a^{\dagger m}\rangle.\label{sum1}\tag2$$
The expectation value $\langle a^n a^{\dagger j} a^i a^{\dagger m}\rangle$ is only nonzero when $n+i=j+m$. This allows to collapse the sum over $j$ replacing $j=n+i-m$, with the additional constraint that we must have $j\ge0$, which thus translates into $i\ge m-n$. But $i$ must also be positive, while $m-n$ might not be. We thus conclude that the appropriate constraint on $i$ is $\max(0,m-n)\le i\le m$. In summary, \eqref{sum1} becomes
$$\langle a^n e^{\alpha a^\dagger}e^{-\alpha^* a} a^{\dagger m}\rangle
= \sum_{i=\max(0,m-n)}^m \frac{(-\alpha^*)^{i}\alpha^{n+i-m}}{i!(n+i-m)!}
\langle a^n a^{\dagger(n+i-m)}a^i a^{\dagger m}\rangle$$
We then see that
$$\langle a^n a^{\dagger(n+i-m)}a^i a^{\dagger m}\rangle
= \sqrt{m!\frac{m!}{(m-i)!}\frac{((m-i)+(n+i-m))!}{(m-i)!}n!}
= \frac{m!n!}{(m-i)!},$$
where we used the identities
$$a^{\dagger j}|\ell\rangle = \sqrt{\frac{(\ell+j)!}{\ell!}}|\ell+j\rangle,
\qquad a^j |\ell\rangle = \sqrt{\frac{\ell!}{(\ell-j)!}}|\ell-j\rangle \,\,\,\text{(for $\ell\ge j$)}.$$
We conclude that
$$\langle a^n e^{\alpha a^\dagger}e^{-\alpha^* a} a^{\dagger m}\rangle
= \sum_{i=\max(0,m-n)}^m (-\mu)^i \alpha^{n-m}\binom{m}{i}\binom{n}{m-i}(m-i)!,$$
where $\mu\equiv|\alpha|^2$,
and
$$\langle n|D(\alpha)|m\rangle = \frac{e^{-\mu/2}}{\sqrt{n!m!}}
\sum_{i=\max(0,m-n)}^m (-\mu)^i \alpha^{n-m}\binom{m}{i}\binom{n}{m-i}(m-i)!$$
The summation can actually be simplified using the range $\sum_{i=0}^m$, as the corresponding terms vanish whenever $i< m-n$ anyway, due to the $\binom{n}{m-i}$ factor.