Every separable Hilbert space $H$ has a countable orthonormal basis, a set of mutually orthogonal unit-norm elements $h_0,h_1,h_2,h_3,...$ such that every element $h\in H$ may be written as a $h=z_0 h_0 + z_1 h_1 + z_2 h_2 +\cdots $ with complex coefficients $z_n$ such that $\sum_{n\geq 0}|z_n|^2$ is finite.
The Hilbert space of all complex-valued square-integrable functions of $x$ (modulo functions whose support has Lebesgue measure zero) is a separable Hilbert space. It has a countable orthonormal basis consisting of the functions
$$
h_n(x)\propto \left(x-\frac{d}{dx}\right)^n \exp(-x^2/2)
\tag{1}
$$
with $n\in\{0,1,2,3,...\}$. Any square-integrable function $h(x)$ can be written in the form
$$
h(x)=\sum_{n\geq 0} z_n h_n(x),
\tag{2}
$$
modulo functions whose support has Lebesgue measure zero. The functions (1) are the eigenfunctions of the quantum harmonic oscillator Hamiltonian $\propto a^\dagger a$ with
$$
a := \frac{1}{\sqrt{2}}\left(x+\frac{d}{dx}\right)
\hskip2cm
a^\dagger := \frac{1}{\sqrt{2}}\left(x-\frac{d}{dx}\right).
$$
The function $h_0(x) \propto \exp(-x^2/2)$ represents the ground state. With this connection in mind, the fact that the functions $h_n(x)$ represent mutually orthogonal elements of the Hilbert space should be familiar, along with the fact that any element of the Hilbert space can be written as in (2).
Distributions like $\delta(x)$ are not functions and do not represent elements of the Hilbert space. They are legitimate mathematical objects when handled in appropriate ways, but they don't represent elements of the Hilbert space.
Appendix
An earlier version of this answer failed to distinguish between two inequivalent notions of "basis" (and the two corresponding notions of "dimension"), and indeed I failed to distinguish between them in my own mind. For the benefit of other visitors that might have the same misconception, I'll show some excerpts from a book that makes the distinctions clear, namely Halmos (1967), A Hilbert Space Problem Book (Springer). Pages 5 and 6 say:
The concept of dimension can mean two different things for a Hilbert space $H$. Since $H$ is a vector space, it has a linear dimension; since $H$ has, in addition, an inner product structure, it has an orthogonal dimension. ... [A linear basis] is a maximal linearly independent subset of $H$. (Recall that an infinite set is called linearly independent if each finite subset of it is linearly independent. ...) An orthonormal basis for $H$ is a maximal orthonormal subset of $H$.
To account for the fact that physicists often use the words "basis" and "dimension" without qualification, page 170 in the same book says this:
...the concept of linear dimension is of no interest in Hilbert space theory. In a Hilbert space context "dimension" always means "orthogonal dimension".
Even if only one of the two concepts is of interest in Hilbert space theory (in Halmos's words), we should still be aware of the distinction (which I was not), because these two types of basis are not equivalent to each other. Page 171 in the same book offers this example of an uncountable linear basis in a Hilbert space that has a countable orthogonal basis: Consider the Hilbert space of all sequences $(z_0,z_1,z_2,...)$ of complex numbers with $\sum_n |z_n|^2$ finite and with the usual inner product. This Hilbert space has a countable orthonormal basis in which each basis vector has a single component equal to $1$ and all other components equal to $0$. But it also has an uncountable linear basis, such as the basis consisting of vectors of the form $(1,x,x^2,x^3,...)$ with $0<x<1$. Any finite set of such vectors (with distinct values of $x$) is linearly independent.
Page 170 also says:
If the orthogonal dimension of a Hilbert space is infinite [even if it's countable], then its linear dimension is greater than or equal to $2^{\aleph_0}$ [uncountable].
In contrast (same page),
...if either the linear dimension or the orthogonal dimension of a Hilbert space is finite, then so is the other, and the two are equal.
Halmos's book is excellent, and I highly recommend it, but beware that it does contain one confusingly-worded statement about this particular subject, namely a statement from pages 5-6 that is discussed in this Math SE post:
Dimension of a Hilbert space
Here are a few other Math SE posts with some information related to bases and dimension in the context of Hilbert spaces:
An orthonormal set cannot be a basis in an infinite dimension vector space?
What is the difference between a Hamel basis and a Schauder basis?
Can we have infinite-dimensional separable Hilbert spaces?
Is every Hilbert space separable?
Can a Hamel basis for an infinite-dimensional Hilbert space be orthonormal?
