I know that the matrix element of momentum operator in position basis is: $\langle x|p|x'\rangle = \frac{d}{dx}\delta(x-x')$. But it seems not a number, because $\langle x|p|x'\rangle$ is rely on the position you place it in. For example, $\langle\psi|p|\psi\rangle = \sum_x\sum_{x'}\langle\psi|x\rangle\langle x|p|x'\rangle\langle x|\psi\rangle$, here you have to keep $\langle x|p|x'\rangle$ in its position, cannot move it like $\sum_x\sum_{x'}\langle x|p|x'\rangle\langle\psi|x\rangle\langle x|\psi\rangle$.
However, in momentum basis, matrix element of momentum operator $\langle k|p|k\rangle = \hbar k$ it's just a number and one can move it anywhere: $$\langle\psi|p|\psi\rangle= \sum_k\sum_{k'}\langle\psi|k\rangle\langle k|p|k'\rangle\langle k|\psi\rangle= \sum_k\sum_{k'}\langle k|p|k'\rangle\langle\psi|k\rangle\langle k|\psi\rangle.$$
Why momentum operator behave so weird? Is it true that a general matrix elements of an operator can be numbers in one basis but may not be numbers in other basis? If so, why in second quantization it can just place the matrix element in front of any creation/annihilation operators since matrix element can be rely on the position where it acts?