The electromagnetic radiation consists of electric $\boldsymbol E(t)$, and magnetic $\boldsymbol B(t)$ fields. The magnetic and electric fields disappear and acquire max values at the same time points.
My question is how energy disappears and appears again, where does it go? The Poynting vector is just average value over a half of cycle.
How is energy conserved if we do not take an average? A photon on the other hand has constant energy of: $$E_p=\hbar\omega$$
I think the confusion here is much simpler. Have a friend clap once a second and stand nearby. Every second, you'll hear a clap followed by silence. That means the energy of the sound waves at your ear will rise every second, then fall. Does that mean energy is 'appearing and disappearing'? Of course not; it's just passing by.
Similarly for a plane electromagnetic wave the fields oscillate in space, $$E(x, t) \propto B(x, t) \propto \cos(kx-\omega t)$$ which means at every moment, the distribution of energy looks like $\cos^2(kx - \phi)$ for some phase $\phi$. That is, the energy of the waves comes in regular lumps, just like the energy in the sound waves your clapping friend makes. (These lumps have absolutely nothing to do with photons.) Then the energy density at a point can of course go up and down as the lumps pass by.
The energy of the electromagnetic field in a volume $V$ is given by $$U=\iiint_{V}\left(\frac{\varepsilon_{0}|E|^2}{2}+\frac{|B|^2}{2\mu_{0}}\right)dV$$ and the poynting vector is defined as $$\vec{S}=\frac{\vec{E}\times\vec{B}}{\mu_{0}}$$ This is not an average value, but a time dependent quantity that encodes the flux of the energy of the electromagnetic field. The conservation of energy is then the statement $$\frac{\partial U}{\partial t}=-\iint_{\partial V}\vec{S}\cdot\vec{dA}$$ which means that the change in the energy of the fields in a volume $V$ is accompanied by a flux in\out that volume. This equation holds for all times, and it describes instantaneous values and not averages.
The conservation here is a statement about changes, rather then just about values at a given instance of time. Thus it is okay for both field to vanish simultaneously, as long as they appear somewhere else or have 'inertia' in some sense. The same happens for a string: at some instances of time it is flat, but has energy in the form of non-zero velocity.
The precise notion of conservation in field theory is given by the concept of a conserved current, that is, a tensor $j^\mu$ satisfying $$ \partial_\mu j^\mu=0 $$
Integrating this equation and using Stokes theorem, you get $$ \frac{\mathrm d}{\mathrm dt}\int_V j^0\mathrm dV=\int_{\partial V} j^i\mathrm d\sigma_i $$ which tells you that the change in time of the integrated density $j^0$ is equal to the flux of $\boldsymbol j$ leaving the integration volume. In this sense, and only in this sense, is a field-theoretic current $j^\mu$ conserved.
This notion of conservation is said to be local. A global conservation law just tells you that the total amount of something doesn't change in time; the local conservation law, on the other hand, is much more restrictive: it tells you that it is conserved in every region of space. If, for example, an electric dipole appears out of nothing, this would satisfy the global conservation of charge, but not the local one. But nature conserves charge locally, so such a phenomenon is disallowed by the laws of physics. This is nicely stressed by Feynman in this lecture of his.
In the case of energy conservation, the current is the so-called energy-momentum tensor, $T^{\mu\nu}$, where $T^{00}$ is to be thought of as an energy density, and $T^{0i}$ as the flux of energy across the surface orthogonal to $x^i$.
In the case of vacuum electrodynamics, $$ T^{\mu \nu} = \frac{1}{\mu_0} \left( F^{\mu \alpha} \eta_{\alpha \beta} F^{\nu \beta} - \frac{1}{4} \eta^{\mu \nu} F_{\delta \gamma} F^{\delta \gamma} \right) $$ and its conservation is easily seen to be a direct consequence of the Maxwell equations. Thus, the electromagnetic energy of any system is conserved locally: it is conserved at each region of space, irrespective of its size and content. See also this PSE post.
I have had the same thoughts viewing the electromagnetic field independent of its source:
where the fields themselves diminish to zero at the appropriate times due to the frequency. Mind you, the Poynting vector is derived as a conservation of energy between generating charges and electromagnetic waves .
Edit after reading the answer by knzhou :
If in the animation we take a perpendicular area to the traveling wave, we see that the energy in that area is propagating with velocity c, and the sinusoidal variation of the energy of consequent planes depends on the way the generating charges created the wave as a function of time.
This becomes clear in the quantum frame where it can be shown that the electromagnetic field emerges from the superposition of an enormous ensemble of photons.
There exists a wavefunction that describes a photon.
The superposition ( not interaction mind you) of these wave functions builds up the electric and magnetic field functions seen in the animation above. In the area perpendicular to the direction of the wave, the energy of the photons propagates with velocity c. The energy is carried by the individual photons as h*nu, the E and B field manifestation is the superposition of all the wavefunctions of the photons in that area ( actually by the heisenberg uncertainty it should be a volume but lets not complicate it more). The following areas can have different energy .
I like this illustration of how photons can build a polarized electromagnetic wave , visualizations help me.
