The problem here was identified by Sandejo - i.e. that writing $|ax\rangle = a | x \rangle$ is an "abuse of notation". Recall that (in this context) the ket labels signal the eigenvalue obtained with respect to some (self-adjoint) operator, here the position operator $\hat{x}$. This is not always the case. A ket labels an element of the abstract Hilbert space. Though this label is arbitrary, choosing it to be the eigenvalue of some operator is often a useful and compact notation.
This is a source of confusion for everyone, since a common (but lazy) notation is to use unmodified $x$ to represent three distinct ideas:
- $\hat{x}$, the position operator
- $x'$, the eigenvalue corresponding to an eigenket after being acted on by $\hat{x}$
- $|x'\rangle$, the eigenket with an eigenvalue of $x'$ after being acted on by $\hat{x}$.
Now if we tried to define $|ax'\rangle := a | x' \rangle$, and acted on this state using $\hat{x}$, we'd find
$\hat{x} a | x' \rangle = a \hat{x} | x' \rangle = a x' |x'\rangle = x' ( |ax' \rangle )$
I.e. the eigenvalue of our supposed "$|ax' \rangle$" ket is actually $x'$, and the label is incorrect. Phrased in linear-algebra terms, what we've (re)discovered is that eigenvectors of a linear operator are only unique up to scalar multiples of themselves.
So where to now? I don't believe it's possible to do this question without at least somewhat exploiting the integral properties of either the delta or the inner products, so we'll try the latter. (Note that life is very difficult here because we are living in a rigged Hilbert space, where not every bra has a ket and not every ket has a bra).
Your states must have some kind of normalisation, i.e.
$$ \int_D dx \langle x' | x \rangle =: N(x') < \infty$$
By making the change of variables $x = au$, obtain
$$\int_D a du \langle x' | au \rangle = N(x')$$
Translating these kets into their dirac-delta represntations, and specialising to $x'=0$ yields
$$\int_D \delta(x) dx = \int_D a \delta(au) du$$
It's tempting to think that we're done here, but there is a subtlety: Though the integration domain $D$ is arbitrary, it's been scaled by a factor of $a$ on the right hand side. We have not quite proven that the integrands themselves are identical. To do this, move back to the inner-product notation (where $\langle 0 |$ is taken to be the $x'=0$ bra, not the null bra)
$$ \int_{x_1}^{x_2} dx \langle 0 | x \rangle = \int_{x_1/a}^{x_2/a} du a\langle 0 | au \rangle $$
In the case that $ 0 \not \in [x_1, x_2]\subset \mathbb{R} $, both LHS and RHS are zero by orthogonality. This shows that the integrands are the same "almost everywhere", so long as the integration domain does not include zero.
The interesting case is if we let $x_1 = -\epsilon$, $x_2 = \epsilon$ and take a limit as $\epsilon \to 0^+$: We know that for a given epsilon we can excise arbitrarily large amounts of padding from the integral domain, so long as we do not cross zero: i.e. the limit can be taken, and shrinking epsilon does not in any way change its value.
We therefore conclude that in this weaker, integral sense, $\delta(x) = a \delta(ax)$.