It's possible to measure chemical potential directly. Chemical potential is exactly analogous to pressure, via the fundamental relation S(U, V, N).
While P is what is equalized under a moving wall, $\mu$ is what is equalized under a permeable wall.
While V is what changes to equalize P, N is what changes to equalize $\mu$.
We can measure P by placing it next to a reference system via a moveable wall and see if V increases, decreases, or stays the same. Then we say our system has smaller, greater, or equal pressure than the reference.
We can measure $\mu$ by placing it next to a reference system via a permeable wall and see if N increases, decreases, or stays the same. Then we say our system has smaller, greater, or equal chemical potential than the reference.
We will see the pressure of an ideal gas decreases linearly with N. Since the minimum N is zero, the corresponding minimal pressure of an ideal gas is that with zero N. So we then simply define zero pressure as the pressure of an empty container.
The $\mu$ of an ideal gas increases with ln(P), so it is undefined at zero P (it approaches negative infinity). So there's no clear system we should take as having zero $\mu$.
Nevertheless, we can measure the (relative) chemical potential of any system by this method. It is important that the other thermodynamic variables remain constant as we measure.