First of all, the $E$ one uses in these formulas are macroscopic , and describe accurately only the forces on macroscopically small, yet microscopically large bodies, i.e, a large collection of charge carriers. $J=\sigma E$ is a macroscopic formula (contrary to what high school textbooks like Haliday will tell you).
Next, you are right to expect some slowing down of these electrons, and there certainly are electron-electron collisions (in a rudimentary, classical picture) occuring, and this is accounted for in the derivation of ohm's law, which is not fundamental anyways. Ohm's Law is more of Ohm's specific relation for specific material with some specific considerations. Also keep in mind, this electric field is pretty small (we are inside a conductor, remember?)
In reality, the motion of these electrons are fantastically complicated, with interaction happening with the field, with other electrons, and with other charge carriers.
The electrons have a velocity due to random thermal motion, which is significantly large, but all in all, random. On average, they cancel our to give $\approx 0$. What remains in this average is the small velocity gain in the direction of the field which we call drift velocity.
You wrote down the electrostatic potential, but the velocity we consider here is not the true velocity of each of these particles, this is simply the average velocity we pretend these large collection of charges move with. And also, you are forgetting the important ingredient which will not get accounted for, dissipation. What you claim is only true for a point charge in vaccum in presence of an electric field $E$