The defining equation for a ladder (or lowering) operator $\hat{L}$ with respect to some observable $\hat{X}$ is* $$[\hat{X},\hat{L}] = -\Delta \hat{L},$$ where $\Delta$ is some difference in eigenvalues of $\hat{X}$. It follows that if $|\xi\rangle$ is some eigenvector of $\hat{X}$ with eigenvalue $\xi$, then $\hat{L}|\xi\rangle$ is another eigenvector with eigenvalue $\xi-\Delta$, since
$$ \hat{X}\hat{L}|\xi\rangle = [\hat{X},\hat{L}]|\xi\rangle +\hat{L}\hat{X}|\xi\rangle = -\Delta \hat{L}|\xi\rangle + \hat{L}\xi |\xi\rangle = (\xi-\Delta)\hat{L} |\xi\rangle.$$
So far this is completely general, but there are numerous famous and familiar examples in quantum physics. For the simple harmonic oscillator, the relevant ladder operator is $\hat{L} = \hat{a} = \hat{x}+{\rm i}\hat{p}$ (in appropriate adimensional units), the observable is the Hamiltonian $\hat{X} = \hat{H} = \omega\hat{a}^\dagger\hat{a},$ and we have $[\hat{H},\hat{a}] = -\omega\hat{a}$ (taking $\hbar=1$). The OP raises another example: the unitary translation operator $\hat{T}(y) = {\rm e}^{{\rm i}y \hat{p}}$, whose action is usually written as
$$\hat{T}^\dagger(y) \hat{x}\hat{T}(y) = \hat{x}+y,$$ but this is equivalent to a commutation relation
$$ [\hat{x},\hat{T}(y)] = y \hat{T}(y),$$
which defines $\hat{T}(y)$ as a "ladder operator" that "lowers" the observable $\hat{x}$ by an amount $\Delta = -y$. (The OP describes the example of an infinitesimal transformation $\hat{T}(\varepsilon)$ up to order $O(\varepsilon)$ but the relation actually holds in general).
It should be obvious from the above argument that any other continuous unitary symmetry operator is also a ladder/lowering operator with respect to the transformed observable, which accords intuitively with our notion of translations, rotations etc. as a shift of spatial, angular coordinate etc.
*It is not always so easy to gain physical intuition from commutator relations but you can check out this nice question for some insights. Mathematically, the defining commutation relation of a ladder operator encapsulates the fact that $\hat{L}$ changes the (eigen)value of $\hat{X}$.