Is there an upper theoretical density limit - density being mass in a finite volume, and the part consisting mass has also some finite volume, just with large space in between (particles, atoms, ...).
Is there a theoretical limit how dense can something get?
In the usual model of black hole, with general relativity only and no quantum mechanics, yes. Once the threshold (the center of) a star collapses down to an infinitely dense singularity, there are no longer any repulsing forces to counterbalance auto-gravity.
But if you add quantum mechanics, then such an exact concentrated locality is not physically possible. (Plus also some modern on-going physics envisioning limits to the small scales of space, or different topologies there.) So in practice infinite density is unlikely to exist. But since nobody knows, (at least for the moment) how to marry Quantum Mechanics and General Relativity in black holes, nobody can give you numbers or equations ruling such a maximum value. ;-)
In case you don't want to go up to black holes:
All rest situations results from the balance of contracting and repulsing forces or pseudo-forces. Since repulsing force grows with decreasing distance[1], there is always a shorter radius (and thus higher density) making the balance[2] in each given situation were you increase a bit the contracting force. Repulsing force can be internal mechanical pressure, electrostatic or more complicated repulsion by the electronic layers, etc.
But if... the physics changes under a distance threshold. For mechanical pressure there is no such threshold (molecules can come in close contact, then electrostatic or electronic force come to settle the limit). Then under larger pressure the molecular or cristal configuration can change to a more compact state with higher energy to resist (indeed inside Earth the cristal configuration change many time with depth, causing density steps). Then once minimally packed, passed a threshold of distances electron orbitals can no longer exists and collapse to the nucleus (-> neutron star), causing a sudden stop of the repulsing force, then replaced by another one caused by something else at a smaller scale. So, there is a cascade of limits, which each of them can theoretically break if you really insist. But this correspond to higher and higher huge energy. So depending on your problem, e.g., only things that can occurs on Earth, this settles a practical range to which part of the cascade above is out of reach or not. E.g. your question could be "what is the maximum density we can produce on Earth in stationary conditions (i.e. not during a picosecond of collision in some particle accelerator)".
[1]: ok, apart for quarks ;-)
[2]: assuming instability have not explode the system apart, like in a supernova.