The answer to this question depends on what your assumptions about how the vector velocities of the interstellar particles might be, of what kind of particles you might think you would encounter. Voyager does track the protons encountered by the space craft, and is currently colliding with 1-2 protons per second. My initial thought is that the particles would have velocity that is Maxwell-Boltzmann distributed which would lead to a type of Brownian Motion which would make its presence known by a type of wobble in the trajectory of the spacecraft.
Putting that thought aside, let's consider the case where the hydrogen is in-elastically colliding with Voyager. Mass of the Voyager space craft is ~733 kg and velocity is 17,000 m/s. The rest mass of the proton is $1.67 \times 10^{-27} kg$. Average velocity of protons enter the solar system from galactic sources is 26,000 m/s. We'll assume all the particles are impacting Voyager head on, which should approximate the worst case scenario.
The inelastic equation is:
$$\dfrac{m_{voy}v_{voy} + m_{p}v_{p}}{m_{voy}+ m_{p}} = v_{voy}'$$
Inputting our values:
$$\dfrac{(733kg)(17,000\dfrac{m}{s}) + (1.67\times10^{-27}kg)(-26,000\dfrac{m}{s})}{(733kg)+ (1.67\times10^{-27}kg)} = (17,000 - 9.79\times10^{-26})\dfrac{m}{s}$$
Or, in other words, under the inelastic assumption the decrease in velocity per collision is $$~9.79\times10^{-26}\dfrac{m}{s}$$
At 1-2 protons per second, the craft would require ~$10^{26}$ seconds to slow down by 1 meter per second. This is several orders of magnitude greater than the life of the universe.
This simple calculation of course does not take into account collisions with larger objects, or dust particles, but a simple adjustment in mass parameter plus knowledge of the number of collisions would allow for a reasonable estimate. A slightly more sophisticated calculation would consider elastic collisions, which I might add later.
