If you adopt the frequentist perspective, the fact that $\langle E\rangle$ does not exist simply means that if you measure the energies of $N$ identically-prepared systems and average the results, then that number does not approach a well-defined limit as $N\rightarrow \infty$.
If the spectrum of the Hamiltonian is unbounded (as it very often is), then such states must exist in the Hilbert space, so they certainly appear mathematically. I personally don't find anything necessarily unphysical about them either; $\langle E\rangle \rightarrow \infty$ doesn't mean that any measurement returns infinity as a result, but rather that increasing the number of measurements tends to increase the average energy.
For example, consider a particle in a box of length $a$ with wavefunction $\psi(x) = 1/\sqrt{a}$. This corresponds to the position of the particle being completely unknown (other than the fact that it's in the box). If you compute $\langle E\rangle$ in this state, you will find that it diverges to infinity. The particle in a box is of course a mathematical idealization, but so is every model we ultimately use, and since this one is very plausibly a useful model I'm not so willing to write it off.
Your example is tricker than it seems since [$\psi$ does not satisfy the boundary conditions associated with the domain of $H$]
@ZeroTheHero raises an excellent point. $H$ (whose domain is twice weakly differentiable functions with Dirchlet boundary conditions) is self-adjoint (as all Hamiltonians must be), but it is correct that the $\psi$ I've written down is not in its domain. In this respect, we cannot write $\langle E\rangle = \langle \psi,H\psi\rangle$ simply because the right-hand side is illegal!
This does not necessarily mean that $\langle E \rangle$ is undefined, but it should make us suspicious. The correct thing to do is to expand $\psi$ in terms of the normalized eigenvectors of $H$:
$$\psi = \sum_n c_n \phi_n \qquad H\phi_n = E_n\phi_n$$
which can always be done due to the self-adjointness of $H$. We then define $\langle E\rangle := \sum_n E_n |c_n|^2$. It's easy to see that this coincides with $\langle \psi,H\psi\rangle$ when $\psi\in\mathrm{dom}(H)$, but is in fact slightly more general, and it is this calculation which diverges to infinity. Indeed, if we naively calculate $\langle \psi,H\psi\rangle$ by differentiating $\psi$ twice, we get zero.