In showing that $$\langle x|\hat{p}|x'\rangle = -i \hbar \frac{dδ(x-x')}{dx}$$ I've seen many solutions doing something similar to Can I replace eigenvalue of p operator with position space representation of p operator? which uses the fact that $$ p\langle p|x\rangle = -i \hbar \partial_x \langle p|x\rangle $$ in order to get the desired result. This leads me to think that in calculating $\langle x|\hat{p}|x'\rangle$ we need to know $\langle p|x\rangle$ (i.e. the differential equation it satisfies).
However, much the same way, when calculating $\langle p|x\rangle$ I've seen solutions using the fact that $\langle x|\hat{p}|x'\rangle = -i \hbar \partial_x δ(x-x')$ like: $$ p\langle p|x\rangle = \langle x|\hat p|p\rangle = \int dy \langle x|\hat{p}|y\rangle \langle y | p \rangle =i\hbar \int dy \frac{dδ(x-y)}{dy}\langle y | p \rangle = -i\hbar \frac{d \langle x | p \rangle}{dx}.$$
My question then is, which comes first? We can't use one in calculating the second and at the same time using the second to calculate the first. Can someone indicate a rigorous way of calculating both without such an interference?