For quantizing the electromagnetic field authors go to its potential and then find themselves facing to the problems of degree of freedom from gauge transformation.
Why we can't simply quantize electromagnetic field itself: decompose it to wave planes and promote normal modes to quantum harmonic oscillator?
The $\bf E$ and $\bf B$ fields viewed as independent quantum oscillators contain too many DOFs, if that's what you mean. But I'm getting ahead of myself. Here is one line of reasoning:
It is reasonable to expect that for a consistent quantum theory of E&M, it should have a classical limit $\hbar\to 0$ where the classical electromagnetic fields are governed by a classical action $S$.
In particular, in this classical action formulation, we demand that the Maxwell equations are (i) either automatically satisfied or (ii) arise as the Euler-Lagrange (EL) equations for $S$.
It turns out to be very challenging to try to formulate the classical E&M action $S$ without the use of potentials, cf. e.g. this Phys.SE post.
Look at the Lagrangian density for electromagnetism, it's $$\mathcal{L}= \frac{\epsilon_0}{2}\mathbf{E}^2-\frac{1}{2\mu_0}\mathbf{B}^2.$$ The Lagrangian is usually constructed as kinetic minus potential energy, right? Well, the kinetic energy is usually given in terms of the time derivative of some coordinate that appears in the potential, and the only way to get that is to use the 4-vector potential fields \begin{align} \mathbf{E}&=-\nabla\phi-\frac{\partial\mathbf{A}}{\partial t} \\ \mathbf{B}&=\nabla\times\mathbf{A}, \end{align} which fits in nicely with the pattern of $\mathbf{E}^2$ as kinetic energy and $\mathbf{B}^2$ as potential if $\mathbf{A}$ is the coordinate.
This allows us, after fixing a gauge and careful handling of constraints, to define the canonical commutation relationships that define quantum theories. For a detailed description of this process, see Chapter 8 of Weinberg's "The Quantum Theory of Fields, Vol. I".
As long as the electromagnetic field is free, your approach works fine. But as soon as you want to include interactions, things become more complicated. For one thing, the term that couples the electromagnetic field to the rest of fields is $$ A^\mu j_\mu $$ where $j_\mu$ is the electric current of such fields.
You cannot write this term directly in terms of $\boldsymbol E,\boldsymbol B$, so you must work with $A^\mu$. That's pretty much it.
This is even more drastic in the case of Yang-Mills, which is a self-interacting version of electromagnetism. Here, even in the absence of an external current $j_\mu$, and due to the self-interactions, it is impossible to formulate the theory directly in terms of $\boldsymbol E,\boldsymbol B$. Here the vector potential $A^\mu$ is unavoidable.
You'll encounter the same gauge fixing issues since the constraint that the waves only have transverse polarizations is a local constraint (the Gauss law). In quantum mechanics, local constraints and gauge symmetries are the same thing.
Also, it will be hard to couple matter to such a theory since the electric and magnetic fields don't capture the Aharonov-Bohm phase.
To add to and elaborate on the points made by others above:
If you only want to quantise the free EM field, you can probably do so directly after you take into account the constraints they satisfy. Alternatively you can take a phenomenological approach (accept photons) and quantise that way, eg. see this paper.
But the moment you include the sources (matter) that generate the EM fields and want to quantise the whole theory to get QED, then avoiding potentials leads to a nonlocal formulation. Such attempts go back several decades, see eg. this paper
For QED there is a simple physical argument why avoiding potentials (if at all possible consistenly) must lead to a nonlocal field theory: the Aharonov-Bohm effect, which shows that physical effects can be felt by an electron even in regions where where the field strength is zero. So to capture the AB effect without gauge potentials would require you to have nonlocal interactions involving the field strengths and matter fields.
Indeed, an analysis of the Aharonov-Bohm effect (see eg this paper) shows that while the gauge potential "over describes" physics (that is, gauge invariance is a redundancy) both classically and quantum mechanically, the field strength "under describes" physics at the quantum level. So you need gauge-potentials, at least their exponentiated loop integrals, in quantum physics. The non-perturbative study of gauge theories on a lattice make use of such exponentiated integrals (Wilson Loops).
Electric and magnetic fields are not independent, except in the static case. Their interdependency originates from the fact that they derive from the four potential, of which the components are mutually independent. The four potential has too much freedom, while the field tensor has too little. Note that its source, the charge-current, also has too much freedom. Too much freedom works better than too little. Note also that it is very inconvenient to try to set up quantum mechanics with the field tensor. I don't know that anyone succeeded at this.
