I can think of at least two different ways to understand eigenfunctions of operators in quantum mechanics. But neither one seems to provide a good explanation for why we take the position-basis representation of $|y\rangle$, the eigenvector of the position operator $\hat{x}$ with eigenvalue $y$, to be the Dirac delta function $\delta (x-y)$:
Perspective 1: The eigenfunctions of position are those functions which satisfy the eigenvector equation $\hat{x}|\psi\rangle = y |\psi\rangle$, expressed in the position basis. So we have, upon multiplication by the bra $\langle x|$ to move to the position basis:
$$\langle x | \hat{x} | \psi \rangle = \langle x | y | \psi \rangle \tag{1}$$ $$x \psi (x) = y \psi (x)\tag{2}$$ $$(x-y) \psi(x) = 0\tag{3}$$
Here I have taken for granted that the action of the position operator in the position basis is to multiply functions by $x$. So this equation tells us that the eigenfunction $\psi(x) = \langle x |\psi \rangle$ must be zero everywhere except for $x=y$. But it doesn't seem to give us any information about the value of $\psi(x)$ at $x=y$.
Perspective 2: The eigenfunctions of position are the position-basis representations of those state vectors for which we can assign a definite value of position. That is, for systems described by such a state vector, an ideal position measurement could only ever result in one particular value $y$ of position, with probability 1. The probability density "function" that best describes this situation is $P(x)=\delta(x-y)$. But the probability density over the possible values of position upon measurement is given by $P(x) = |\langle x | \psi \rangle|^2=|\psi(x)|^2$. So we have, up to a phase factor, that $\psi(x)=\sqrt{\delta(x-y)}$, which isn't a meaningful mathematical expression as far as I can tell. Alternately, the eigenfunction can't be $\psi(x) = \delta(x-y)$ as we usually claim, because then $P(x) = \delta(x-y)^2$, which doesn't seem to have a rigorous mathematical meaning either.
Both of these routes seem to lead to a dead end. So my question is, how can we claim that the functions $\delta(x-y)$ are eigenfunctions of position? I think that an analogous question can be asked for any other observable that takes on a continuous range of values, such as momentum.