There are a couple of good answers, but I'm going to answer it from a third perspective not related to a specific type of heating used or shared walls/floors/ceiling with neighboring apartments.
Heating a building requires two things: heating the air, and heating the structure itself. The thermostat is set so that the heating unit will run until the air flow around the thermostat is at an upper bound at or slightly above the desired temperature. For some types of heating such as forced air the air temperature will actually heat up quite quickly, because while the heating unit is on the air is cycled through a heat exchanger. For other types of heat such as radiative this is not quite as true because heat transfer is more of a function of distance from the heat source, but generally the air will heat up quickly anyway (although a lot less evenly!) because the thermal mass of the air isn't that high.
So if all there was to it was heating the air, then it would make a lot of sense to simply turn off the heat when you were gone (or turn it just high enough to prevent freezing to prevent damage to pipes, etc.). However, there is also the thermal mass of the building. This is greatly dependent on the construction materials used. A "stick frame" house will not have as much thermal mass as block construction, concrete, brick, etc. largely because there are a lot of air gaps (often filled with insulation).
Whether or not this works out better to using less energy than keeping the temperature constant may seem to depend on how well insulated the building is. But you have the same insulation whether you are keeping it the same temperature or letting it cool. So by that logic you are right that the T in calculating heat loss is less in a cooler building.
If we look at the heat capacity of various materials based on density (so we can compare to air), we get 0.231 MJ/m^3K for wood, approximately 2 for both brick and concrete, and only 0.0012 for air. So from this we can see that the energy required to heat the air itself is small per unit volume, and even wood is only 1/10th that of brick or concrete.
The energy required to heat up the building is defined by Q = CT where Q is the energy required, C is the thermal mass, and T is the temperature difference. So from this it can be seen that a building with a large thermal mass will require a lot of energy to bring it back up to temperature after it has cooled. But conversely, it will also take a long time for the building to cool as more energy much be lost per degree temperature drop. If energy costs vary over the day, thermal mass (if large enough) can be used to shift energy use to times when energy costs less.
So what the physics is telling us (or me, at least) is that keeping the building as cool as possible at all times uses the least amount of energy. But, given all of the above I believe that the recommendation being given to keep the temperature from going to low is based on the subjective perception that since it takes a long time to heat the building structure this is going to cost more than simply keeping the temperature a bit warmer all the time, ignoring the longer time it took the temperature to drop.
From a practical standpoint, heating up the entire structure may overwork a heating system that is undersized (or not oversized) and which must run continuously for an extended period of time to initially heat the entire structure. Depending on how the building was constructed and how heat is transfer this could also result in the system cycling more than designed as the air quickly heats up, the heat shuts off, and then the heat starts to conduct and radiate into the walls causing the temperature to fall and the heat to come on again more quickly than normal. So there could be added costs not directly related to heating energy but related to maintaining the heating system itself.