It comes from experience with magnetic field acting on current-carrying wires (experiments and inference of force formulae from them was by Ampere, Biot, Savart, Laplace) and magnetic field acting on cathode rays (electrons) in electric and magnetic fields (experiments by Crookes, J. J. Thomson).
The generalization of the above formulae to the case of force acting on a charged particle was, although partially implied already by Thomson's work, established some years by H. A. Lorentz in his theory of electrons: he postulated that force on charged particle can be expressed as integral
$$
\int_{volume~containing~the~particle} \rho\mathbf d + \mathbf j\times \mathbf h\,dV
$$
where $\mathbf d,\mathbf h$ are microscopic electric and magnetic field and $\rho,\mathbf j$ are microscopic electric charge density and current density of the particle.
Today, we often simplify this by neglecting any details of internal structure and write the force as function of total charge, velocity and external fields:
$$
q\mathbf E_{ext} + q\mathbf v \times \mathbf B_{ext}
$$
which is more appropriate to description of experiments such as Thomson's cathode rays (we set up the macroscopic fields $\mathbf E_{ext}, \mathbf B_{ext}$, not the microscopic ones, those are hard to control.
In this modern notation $\mathbf E_{ext}$ replaces Lorentz's $\mathbf d$ and $\mathbf B_{ext}$ replaces his $\mathbf h$.