How to know if the error is in a law or in uncertainty of the measurement?

Question

I read these words in a (great) answer to this question:

There are errors that come from measuring the quantities and errors that come from the inaccuracy of the laws themselves

But how do we know that the errors are in the measuring or in the law about which we make measurements?

Answer 1

Measurement errors or experimental errors can be reduced by, for example, using more accurate and more sensitive equipment; making multiple measurements and taking an average; thinking about possible sources of noise and trying to compensate for or reduce these. If you do all these things and there is still a difference between actual and expected results - and if this difference is larger than can be accounted for by the remaining sources of noise or error - then the problem is with the law or theoretical model that produced the expected results.

Usually, the "null hypothesis" is that the law or theoretical model is correct, so discrepancies in results are due to experimental error. A scientist will typically only consider modifying a law or model after all the alternative explanations of discrepancies have been ruled out.

Answer 2

In my opinion, @gandalf61 has provided the correct answer. Let me just expand on it a little.

Measurement errors
The discrepancy between the law and the results of measurement may come from different sources, notably from the measurement errors (some of which cannot be controlled), but also possibly from sample rpeparation, different conditions at the time of the experiments, etc. Often these errors cannot be controlled or reduced to zero, even by improving the measurement techniques and the rest. The discrepancy between the observed results and the theory may suggest that the theory is not correct.

Hypothesis testing
If we suspect that the theory is not correct, we need to perform hypothesis testing. This is a rather well-defined statistical procedure, but unfortunately not given sufficient attention in modern physics education, since the high precision of measurements makes it rather unnecessary (with the important exception of particle physics, see this review).

One takes as a null hypithesis the assumption that the existing theory is correct and calculates the p-value, which is the probability that the observed anomalous data are due to statistical error only. If the p-value is smaller than the chosen confidence threshold, the null hypothesis is rejected, i.e., we conclude that the theory is wrong.

Note that the whole procedure is statistical in nature - we can never be 100% sure that our conclusions are correct!

The reason why we try to disprove the existing theory, rather than trying to prove it, is is that doing the latter requires also calculating the statistical power, which is usually a more difficult problem, requiring more assumptions.

There exist multitude of statistical tests for various types of the situations, which allows us to adapt to various sources of statistical errors.

Update
Note that familiar to everyone confidence intervals are actually a rather involved concept, grounded in hypothesis testing: The interval has an associated confidence level that gives the probability with which the estimated interval will contain the true value of the parameter. Their everyday interpretation as the spread of measurement values around the "true" value is actually that of the credible interval in Bayesian statistics.

Answer 3

But how do we know that the errors are in the measuring or in the law about which we make measurements?

Laws in physics theories are extra axioms to pick up from mathematical solutions those solutions that are descriptive and predictive of data. Whenever data do not fit predictions , one finds the dimensions of validity for the theory, a new theory needed outside those dimensions.

In the dimensions needed for the GPS system to predict accurately the positions on earth, Newton's laws fail and special and general relativity have to be pulled in.

So it is the failure of predictions for data of a theory that decides the errors on the laws. It is evident that the measurement errors should be small enough to show discrepancy with theoretical predictions using the law.

Answer 4

Every measured value has errors. This is the principle stand of the Guide to the Expression of Uncertainty in Measurements. The magnitudes of the errors can be determined (quantified). They consist of offset (calibration) errors, measurement (device) errors, and random errors. A term with less negative context is uncertainty rather than error.

Suppose that we make a comparison of a measured value to a value predicted under a certain law. This comparison requires that we do include two things. First, we must include the total uncertainty in the measured value. This sets the confidence that we have that our measured value is an accurate representation of the reality (e.g. that the device is well-calibrated). It also sets the confidence that we have that we are using the best precisions in our devices and doing our experiments to the most reproducible manner possible.

The second thing we must do is to assure that the conditions of the experiment are not a prior outside the bounds required to apply the law. This validates in advance why we permit ourselves to use the law for comparison.

We compare whether a measured value is different from the prediction within the confidence level of the measurement and within the validity of the assumptions in the law.

What happens when we discover that a measured value is different from expectation? We can take one of two steps. 1) We can realize that our measurement has an as yet unrealized error. Maybe the devices were not calibrated or not calibrated properly. Maybe we have not done sufficient replicate experiments to encompass a strong confidence range on the population statistics. 2) We can realize that our experiment was not done within the full bounds of the assumptions required to apply law that we chose to use. Maybe we neglected to follow a critical assumption that must be in place to apply the law. Maybe we overstepped a limit on a limiting contingency to apply the law.

In summary, laws do not have errors (uncertainties); measured values have errors and laws have assumptions against which we must verify our measurement process. Laws are not inaccurate in and of themselves; measured values can be deemed accurate or inaccurate in reference to the law against which they are compared.