The first attempts at formulating a model of nature had no numbers in it. Numbers is the province of mathematics. Arithmetic was created really when bartering became necessary as communities became larger and larger and a number system was needed to keep a fair track of who owes what to whom, as you say. It is arithmetic that was the start of mathematics. Not physics.
When land possession started people used arithmetic to keep track of what was what, and finally geometry and then algebra were invented. Negative numbers started coming in solutions of algebraic equations
For a long time, negative solutions to problems were considered "false". In Hellenistic Egypt, the Greek mathematician Diophantus in the third century A.D. referred to an equation that was equivalent to 4x + 20 = 0 (which has a negative solution) in Arithmetica, saying that the equation was absurd.
Negative numbers appear for the first time in history in the Nine Chapters on the Mathematical Art (Jiu zhang suan-shu), which in its present form dates from the period of the Han Dynasty (202 BC – AD 220), but may well contain much older material. The Nine Chapters used red counting rods to denote positive coefficients and black rods for negative.
It is after all a useful way of keeping count of what is owed and what is gained in any human transaction, in two sets of numbers. Instead of using left hand fingers for owed right hand fingers for gain, a symbol instead meaning "left hand".
Lets go really small. I have 2 grains of salt. I can't take 3 grains away.
No, you can count on your right hand "two in hand", and on your left "one in the bush" meaning expecting to get it from somewhere, or someone else's kitchen.
In any case negative numbers existed in mathematics before ever being used in physics.
Now for physics, mathematics is a tool. I have heard it said that "physics theory is a subset of mathematics" I suppose by mathematicians ? But it is not true. Mathematics allows for beautiful self consistent theories where positive, negative, complex and even worse definitions hang together and produce solutions .
Physics takes mathematical self consistent constructs and fits them, imposes them on physical observables by introducing physics postulates and laws on top of the mathematical axioms . These correlate physical situations, space and time changes, and negative numbers come with the package (also complex ones). Solutions of the mathematical equations predict data to be seen in the future. Physics cannot be blamed for negative numbers.
I believe negative numbers were invented by man to create debt.
You are correct, on the first part, but not to create debt, to keep count of debt in a better way than a left hand and a right hand give and take tally.