There is more than one approach one can use to calculate the work done by a contact force on a rigid object. All approaches are valid so long as you're careful about how you actually use your calculated work.
One way is to use the infinitesimal displacement of the point of application $d\vec l_\text{app}$:
$$W = \int \vec F \cdot d\vec l_\text{app} \tag{1}$$
In the absence of other forces (and forms of potential energy), this will correctly give you the quantity $1/2 \int v^2 dm$, i.e., the change in total kinetic energy.
In your object-down-the-ramp example, the static frictional force would do zero work.
Another way to approach the same problem is to use the center of mass displacement $d\vec l_\text{CM}$:
$$W = \int \vec F \cdot d\vec l_\text{CM} \tag{2}$$
In the absence of other forces (and forms of potential energy), this will give you the change in center-of-mass kinetic energy $\Delta K_\text{CM} \equiv \Delta\left(\frac{1}{2}mv_\text{CM}^2\right)$.
Yet another way is an extension of method #2, in which one considers the rotational aspects of the motion:
$$W = W_\text{CM} + W_\text{rot} = \int \vec F \cdot d\vec l_\text{CM} + \int \vec\tau \cdot d\vec\theta \tag{3}$$
This latter approach lends itself well to separating out work that causes changes in the center-of-mass kinetic energy and changes in the rotational energy of a rigid body:
$$\Delta K_\text{tot} = \Delta K_\text{CM} + \Delta K_\text{rot}$$
More info on the different definitions of work: Article by Sherwood