Newton defined "quantity of motion" at the beginning of Book I in Definition II.
The quantity of motion is the measure of the same, arising from the velocity and quantity of matter conjunctly.
The motion of the whole is the sum of the motions of all the parts; and therefore in a body double in quantity, with equal velocity, the motion is double; with twice the velocity, it is quadruple.
Here, "arising from the velocity and quantity of matter conjunctly" means it is the result of multiplying the velocity and mass.
A comment to another question may answer why Newton didn't express his laws more mathematically:
I'm pretty sure that Newton never wrote his law of gravitation in algebraic form, nor thought in terms of a gravitational constant. In fact the Principia looks more like geometry than algebra. Algebra was not the trusted universal tool that it is today. Even as late as the 1790s, Cavendish's lead balls experiment was described as 'weighing [finding the mass of] the Earth', rather than as determining the gravitational constant.
Addendum:
I've been reading some sections of Newton's Principia more closely and I think Newton is using an ambiguous definition for force. Take the following passage from the Scholium following the Three Laws of Motion:
When a body is falling, the uniform force of its
gravity acting equally, impresses, in equal particles of time, equal forces
upon that body, and therefore generates equal velocities; and in the whole
time impresses a whole force, and generates a whole velocity proportional
to the time.
The first use of the word "force" in "the uniform force of its gravity" would seem to be the usual definition of force that causes acceleration. The second use, "impresses, in equal particles of time, equal forces upon that body, and therefore generates equal velocities," seems to refer to a force acting over time that causes a finite change in velocity. Modern physicists would refer to the latter usage as "impulse," which would be expressed mathematically as $J = F\Delta t$ or $J = \int Fdt$ for varying forces.
Given the novelty of calculus at the time, perhaps Newton could not speak confidently of instantaneous actions over infinitesimal quantities of time, so he only spoke of finite intervals, which would also yield to more geometric arguments of motion through space. If Newton had written more mathematically rather than geometrically, he might have expressed the Second Law as either
$$F = \frac{\Delta(mv)}{\Delta t}$$
or
$$F\Delta t = \Delta(mv)$$
with both of the left-hand sides of the equations being referred to as force.