There seems to be some misconceptions. I'll avoid expressing the field operator in terms of the ladder operator, because the OP does not seem to want that. However, it is perhaps important to note that the field operator can be separated into two parts $\phi(x)=\phi^c(x)+\phi^a(x)$ where $\phi^c(x)$ creates a particle and $\phi^a(x)$ annihilates a particle. So now we can indeed use these operators to produce and manipulate the Fock states.
We need to remember that Fock states do not only contain particle-number degrees of freedom but also the other degrees of freedom in the theory, namely the spatiotemporal degrees of freedom. Therefore, when we produce a one-particle state using $\phi^c(x)$, we can multiply it with some mode function and integrate over $x$ to specify the spatiotemporal degrees of freedom of that state. The requirement for normalization is then carried by the properties of that mode function. If we don't do it that way, we get something that may not be a well-defined state because it is not properly normalized.
To avoid this issue of normalization, let's define a single-mode creation operator as
$$ \phi_0^{\dagger} = \int \phi^c(x) f_0(x) \text{d}x . $$
By using only $\phi_0^{\dagger}$, we are restricting ourselves to the part of the Hilbert space that only deals with states having the spatiotemporal degrees of freedom defined by $f_0$.
When $\phi_0^{\dagger}$ operates on the vacuum it produces a one-particle Fock state
$$ \phi_0^{\dagger}|\text{vac}\rangle = |1\rangle . $$
To produce Fock states with more particles, we need to apply $\phi_0$ multiple times. For example an $n$-particle Fock state is given by
$$ \frac{1}{\sqrt{n!}}\left(\phi_0^{\dagger}\right)^n|\text{vac}\rangle = |n\rangle . $$
Note the factor in front which is required for normalization.
To address the comment: what happens when the field operator is applied to the Fock states? Again, we can separate the field operator into the two parts. If the creating part is applied to a Fock state it adds another particle to the state, but again we need to be careful with the normalization. So if we again use $\phi_0^{\dagger}$, we get
$$ \phi_0^{\dagger}|n\rangle = |n+1\rangle \sqrt{n+1} . $$
When the annihilating part of the field operator is applied to a Fock state, it removes a particle. To do the calculation correctly, we need to use the equal-time commutation relations. It reads$^a$
$$ [\phi^a(\mathbf{x},t), \phi^c(\mathbf{y},t)] = \delta(\mathbf{x}- \mathbf{y}) . $$
Then for the single mode operators, we have
$$ [\phi_0, \phi_0^{\dagger}] = 1 , $$
thanks to the normalization. Now we can apply the single-mode annihilation operator to the single particle Fock state
$$ \phi_0|1\rangle=\phi_0\phi_0^{\dagger}|\text{vac}\rangle
=[\phi_0,\phi_0^{\dagger}]|\text{vac}\rangle
= |\text{vac}\rangle . $$
It can be generalized for the case of a multi particle Fock state.
Hopes this explanation clarifies some of those misconceptions. Let me know if there are more issues.
$a$: It may be better to do these commutations in the momentum basis. What follows is a bit hand wavy.