First, we need to lay out some preliminaries in the language of Shankar's book. If you want, you can skip to the bottom where I apply the following to your question.
When you express a state $|\psi\rangle$ in the position basis, what you're doing is applying the identity operator $\mathbb{1} = \int dx \ |x\rangle\langle x|$ and defining $\langle x | \psi \rangle \equiv \psi(x)$ to be the position-space wave function corresponding to the state $|\psi\rangle$.
$$ |\psi \rangle = \int dx \ |x \rangle\langle x | \psi \rangle \equiv \underbrace{\int dx \ |x \rangle \underbrace{\psi(x)}_{\text{Wavefunction}}}_{\text{Entire state}}$$
When you express an operator $\hat O$ in the position basis, what you're doing is applying the identity operator $\mathbb{1} = \int dx |x \rangle\langle x |$ from the right and $\mathbb{1} = \int dx' |x' \rangle\langle x' |$ from the left and defining $ \langle x'|\hat O|x\rangle \equiv O_{xx'}$ to be the position-space representation of the operator $\hat O$.
$$\hat O = \iint dx \ dx'\ |x'\rangle\langle x'|\hat O |x \rangle \langle x | \equiv \iint dx\ dx' \ |x'\rangle O_{xx'} \langle x|$$
( Note that $\hat O$ eats states and spits out other states, while $O_{xx'}$ eats (wave)functions and spits out other functions.)
From here, we can write, we can write
$$\hat O |\psi\rangle = \left(\iint dx\ dx' |x'\rangle O_{xx'} \langle x| \right)\left(\int dy \ |y\rangle \psi(y)\right)$$
$$ = \iiint dx \ dx' \ dy \ |x'\rangle O_{xx'} \langle x|y\rangle \psi(y)$$
We can use the fact that $\langle y|x\rangle = \delta(x-y)$ to eliminate the integration over $y$ and yield
$$\hat O | \psi\rangle = \iint dx \ dx' |x'\rangle O_{xx'} \psi(x)$$
This is the most general possible expression for action of the operator $\hat O$ on the state $|\psi\rangle$ as expressed in the position basis.
In your question, you refer to $\hat a \equiv \frac{\hat X + i\hat P}{\sqrt{2}}$ (where we've set all the constants equal to 1 for simplicity - you can put them back in as an exercise). In the position basis, we have that
$$X_{xx'} \equiv \langle x' | \hat X | x\rangle = x \delta(x-x')$$
and
$$P_{xx'} \equiv \langle x' | \hat P | x\rangle = -i\delta'(x-x') = -i \delta(x-x') \frac{d}{dx'}$$
and so
$$a_{xx'}\equiv \langle x' | \hat a | x\rangle = \delta(x-x') \left(\frac{x + \frac{d}{dx'}}{\sqrt{2}}\right)$$
The wavefunction of your state is unknown at the moment, but you can call it $\psi_0(x)$. Plugging the operator $a_{xx'}$ into the integral we found earlier yields
$$\hat a|0\rangle = \iint dx \ dx' |x'\rangle \delta(x-x') \left(\frac{x + \frac{d}{dx'}}{\sqrt{2}}\right) \psi_0(x) = \int dx |x\rangle \frac{1}{\sqrt{2}} \left( x + \frac{d}{dx}\right)\psi_0(x) $$
which means that $\hat a|0\rangle$ yields a new state whose wavefunction is given by
$$\phi(x) = \frac{1}{\sqrt{2}}\left(x + \frac{d}{dx}\right) \psi_0(x) = 0$$
where the last equality arises because we know that $\hat a$ annihilates the state $|0\rangle$.