I would not say there is a difference between the definition of probability in Mathematics and Physics. Probably, what you are referring to as the definition in Physics is the interpretation of probability theory.
I assume that what you mention as the Mathematical definition is the theory as summarized by Kolmogoroff's axioms (or equivalent). That is a special chapter of Measure Theory in Mathematics. However, we have to notice two important things on the mathematical side.
- Kolmogoroff's theory is a general scheme consistent with many interpretations of the theory. As an axiomatic theory, it does not define the basic objects it works on; like in axiomatic geometry, we do not define what a point, a line, or a plane are. Then, for application to the real world, we have to provide an interpretation fixing how we can assign probabilities to the events. It turns out that many interpretations can be used in connection and consistent with Kolmogoroff's axioms (frequentistic, classical "a priori," subjective Bayesian, ...).
- There are other mathematical frameworks accommodating probability theories quite different from Kolmogoroff's version. It is possible to find an extended, even though not up-to-date, report in the classical book by T.L. Fine Theories of Probability, Academic Press (1973). Comparative Probability, Complexity-based Probability, and Carnap's Theory of Logical Probability are a few of them.
In Physics, the axioms by Kolmogoroff are usually used. In many cases, the frequentistic approach prevails regarding the interpretation, although, in different contexts, the Bayesian point of view has been advocated as the most appropriate.
After this summary of the situation, the point about rational or irrational values for probability seems to raise a marginal issue.
First, in a frequentistic approach, one can assume that frequencies are ideally measured over infinite sequences of experiments. But, even without stressing this point of view, one can safely use a theory based on real numbers to approximate rational-number-based measures satisfactorily. That is common in Physics and any application of Mathematics to the real world. Strictly speaking, even measuring a length in the real world can only provide rational number outcomes. This fact does not prevent from using geometrical relations based on real numbers.