Your reasoning goes off-track here:
this would imply that.. $$|x\rangle=\langle x|=\delta(x-x')$$
There are a few different things that do not make sense here. It might be easiest to work with this statement in terms of a geometric analogy. The state $|x\rangle$ is a member of a Hilbert space; you can think of it as a vector. In keeping with this analogy, $\langle x|$ is the corresponding member of the dual space: you can think of it as a function that takes a vector and returns a number, or equivalently, as the transpose of a vector. By saying $|x\rangle=\langle x|$, you are trying to say that, for example,
$$\begin{bmatrix}1&2&5\end{bmatrix}=\begin{bmatrix}1\\2\\5\end{bmatrix}$$
which is obviously incorrect, as those are two fundamentally different objects. By saying $|x\rangle=\delta(x-x')$, you are saying that a vector is equivalent to a number. By saying $\langle x|=\delta(x-x')$, you are saying that a function on vectors is equivalent to a number. None of these make sense, as they are all different types of objects.
So what is the proper way to make sense of the wavefunction? Let's begin with the question: how do you express an arbitrary state in a particular basis?
Suppose we have a finite orthonormal basis of states $|\phi_1\rangle,|\phi_2\rangle,...,|\phi_n\rangle$. If we have an arbitrary state $|\psi\rangle$, then we can express this state as a linear combination of basis states:
$$|\psi\rangle=c_1|\phi_1\rangle+c_2|\phi_2\rangle+...+c_n|\phi_n\rangle$$
The reason we can do this is because of the definition of a basis, from linear algebra: a basis is a minimal set of vectors whose span (set of linear combinations) is the entire vector space in question. Now, from this, we can easily see that, for any $k$:
$$\langle\phi_k|\psi\rangle=c_k$$
The reason we can say this is because our basis is orthonormal: it's both orthogonal (meaning $\langle \phi_j|\phi_k\rangle=0$ for $j\neq k$) and normal (meaning $\langle \phi_k|\phi_k\rangle=1$ for all $k$). Therefore, we can write:
$$|\psi\rangle = \langle\phi_1|\psi\rangle|\phi_1\rangle+\langle\phi_2|\psi\rangle|\phi_2\rangle+...+\langle\phi_n|\psi\rangle|\phi_n\rangle=\sum_{k=1}^n\langle\phi_k|\psi\rangle|\phi_k\rangle$$
This gives us a general procedure for expanding a function in a finite basis. This procedure also extends to expanding a state in a countably infinite basis; the finite sum at the end simply becomes an infinite sum.
When working with uncountably infinite bases like the position basis, where there is one basis state $|x\rangle$ for any real number $x$, we need to be a bit more careful. In particular, the position basis is not orthonormal. The inner product of two basis states $|x\rangle$ and $|x'\rangle$ is given by:
$$\langle x|x'\rangle=\delta(x-x')$$
So we still have an orthogonal basis ($\langle x|x'\rangle =0$ for $x\neq x'$), but it's not a normal basis ($\langle x|x\rangle$ is infinite).
In any case, the procedure to expand a state in the position basis is similar, but instead of a sum, we have the uncountable analogue of a sum, which is an integral:
$$|\psi\rangle = \int_{-\infty}^\infty \langle x|\psi\rangle |x\rangle dx$$
Instead of having a finite or countable set of coefficients $c_k$, we now have a coefficient for every real number $x$. It is convenient to express these coefficients as a function which takes a real number $x$ and returns the appropriate coefficient $\langle x|\psi\rangle$. This is what we mean when we say $\langle x|\psi\rangle = \psi(x)$, which means we can say:
$$|\psi\rangle=\int_{-\infty}^\infty \psi(x)|x\rangle dx$$
This is how the wavefunction $\psi(x)$ is related to the state vector $|\psi\rangle$. To recover your statement, we simply take the overlap of some particular position basis state $|x\rangle$ with our state $|\psi\rangle$, and expand appropriately:
\begin{align}
\langle x|\psi\rangle&=\langle x|\int_{-\infty}^\infty\langle x'|\psi\rangle |x'\rangle dx'\\
&=\int_{-\infty}^\infty \langle x'|\psi\rangle\langle x|x'\rangle dx'\\
&=\int_{-\infty}^\infty \psi(x')\delta(x-x')dx'\\
&=\psi(x)
\end{align}
which is exactly what should be happening, given the definition.
Now, to answer your questions:
- The question about why being in a eigenstate of position or momentum implies having a position or momentum of x or p respectively.
One of the fundamental postulates of quantum mechanics is the following: measurements of a particular observable give you eigenvalues of the operator associated with that observable. The position operator is associated with the position observable; measuring a particular position means that you have measured a particular eigenvalue of an operator. Therefore, the state post-measurement must be the eigenstate associated with that particular eigenvalue. Conversely, an eigenstate of the position operator is associated with a particular eigenvalue of the position operator, which is, in turn, associated with being measured at a particular position.
- The question at (*) about how this "quantum amplitude for being in position x" can be an apt description of ψ(x) (more specifically, intuitively how |x⟩=δ(x−x′) has one definite position yet ⟨x∣ψ⟩=ψ(x) which doesn't have a definite position).
Another fundamental postulate of quantum mechanics is the following: the probability that a state $|\psi\rangle$ will be in a particular eigenstate $|\phi\rangle$ after measurement of the corresponding observable is $|\langle \phi|\psi\rangle|^2$. Hence, the probability that $|\psi\rangle$ will be in a particular position eigenstate $|x\rangle$ after measurement is $|\langle x|\psi\rangle|^2=|\psi(x)|^2$. Using the answer to the first question, this is the same as the probability that the object described by $|\psi\rangle$ will be measured in position $x$.
- If $|x\rangle = \delta(x-x')$ and $|p\rangle = \delta(p-p')$ then why does
$$
\langle x \mid p \rangle = \int_{-\infty}^{\infty} \delta(x-x') \ \delta(p-p') \ dx = \frac{1}{\sqrt{2 \pi \hbar}} e^{i p/\hbar x}
$$
Where the RHS is clearly a plane wave?
The action of the momentum operator $\hat{p}$ on the state $|\psi\rangle$ can be represented in the position basis as a function of the position wavefunction of $\psi$:
$$\hat{p}|\psi\rangle\to -i\hbar\frac{d\psi(x)}{dx}$$
where the symbol "$\to$" is used to emphasize that these two statements are not strictly equivalent, but rather that the right-hand side is one of many possible representations of the left-hand side. For example, in the momentum basis, the representation of the momentum operator is a function of the momentum wavefunction of $|\psi\rangle$:
$$\hat{p}|\psi\rangle\to p\psi(p)$$
Anyway, suppose we want to find the representation of the momentum eigenstate $|p\rangle$ in position space. This is equivalent to finding the position wavefunction of the momentum eigenstate, which for clarity we will call $\psi_p(x)=\langle x|p\rangle$. Based on the definition of an eigenstate, we know that, regardless of representation,
$$\hat{p}|p\rangle=p|p\rangle$$
where $p$ is the momentum eigenvalue associated with the momentum eigenstate $|p\rangle$. If we put this equation into the momentum representation, we have the following differential equation:
$$-i\hbar\frac{d\psi_p(x)}{dx}=p\psi_p(x)$$
Solving this differential equation gives you:
$$\psi_p(x)=\frac{1}{\sqrt{2\pi\hbar}}e^{\frac{ipx}{\hbar}}$$
where the factor in front is to ensure that the wavefunction is normalized. Therefore, based on the definition of $\psi_p(x)$, we have:
$$\langle x|p\rangle = \frac{1}{\sqrt{2\pi\hbar}}e^{\frac{ipx}{\hbar}}$$
Incidentally, this relation helps you transform a position wavefunction into a momentum wavefunction, and vice-versa. We simply insert a basis expansion into the definition of the position wavefunction as follows:
$$\psi(x)=\langle x|\psi\rangle=\int_{-\infty}^\infty \langle x|p\rangle \langle p|\psi\rangle dp=\int_{-\infty}^\infty \frac{1}{\sqrt{2\pi\hbar}}e^{\frac{ipx}{\hbar}}\psi(p)dp$$
This shows that the position and momentum wavefunctions are related by a Fourier transform.
