A symmetric operator has only real eigenvalues and different eigenvectors corresponding to different eigenvalues are orthogonal.
These are exactly what we want for a physical observable. I think this is the all required.
So why is quantum mechanics not satisfied with symmetric operators, but wants more?
I'm pretty sure there exists an answer for this here already, but I can't find it (it's always about unboundedness).
For the Hamiltonian, the answer is basically given by Stone's theorem on one-parameter unitary groups. There is a one-to-one correspondence between self-adjoint operators and strongly continuous one-parameter families of unitaries.
Why is that important? The Hamiltonian is important, because it is supposed to give us the dynamics of our system, i.e. it will define the time development of the quantum system. Such a time development, however, must be a one-parameter unitary group $U(t)$. This is due to the fact that we want Born's rule to hold:
If $|\psi\rangle\in\mathcal{H}$ is a state in your Hilbert space, we need that $1=\langle \psi(t)|\psi(t)\rangle=\langle U(t)\psi|U(t)\psi\rangle$, i.e. the state stays normalized such that it still is a probability distribution. But then $U(t)$ must be unitary by the very definition of a unitary operator. However, Stone's theorem tells us that the generators of unitary groups are self-adjoint operators, not just symmetric ones. Granted, the theorem talks only about strongly continuous groups, but there exist examples of symmetric and non-selfadjoint operators that generate a group that's just not unitary (some form of an operator for the Stark effect does the trick, if I remember correctly)
Another issue alluded to in a comment is the problem with the spectral theorem: Given any observable, its spectrum tells us something about the possible measurement outcomes, but we simply might not have a spectral decomposition, if the operator is merely symmetric. Unbounded operators can even have empty spectrum, but I don't know whether there exist symmetric unbounded operators with empty spectrum (there certainly exist anti-symmetric operators with empty spectrum). If you consider a von-Neumann measurement, one of the axioms tells you how the state after the measurement looks like - there is a projection involved. But this projection might not be well-defined, if you don't have a spectral decomposition.
You might also want to have a look at what is called "Pseudospectra". Given a non-selfadjoint poerator, it might have "approximate eigenfunctions" to "pseudo-eigenvalues" that are very far from the spectrum, (in the sense that given an operator $A$, there exists a $\lambda\notin \operatorname{Spec}(A)$ such that there might be an $x\in\mathcal{H}$ with
$$ \|Ax-\lambda x\|<\varepsilon \|x\|$$
and such things might exist very far from the spectrum for any $\varepsilon>0$ you want; see Davies book on "linear operators and their spectra"). I don't know much about this and how this could be an issue, but this might give you more incentives as to why "just" symmetric operators just might not do the job.
The spectral theorem only holds for normal operators. Self-adjoint operators are normal, symmetric ones not necessarily so.
In physicist-speak, we want the generalized eigenvectors to from a 'complete basis' of the Hilbert space. For example, the generalized eigenvectors of the momentum operator in position representation are plane waves, and even though they aren't part of our Hilbert space themselves, any wave function can be decomposed into a superposition of plane waves. This is only possible because the momentum operator is self-adjoint.
