Temperature is not energy. Temperature is energy per unit entropy. Entropy, most naturally measured in bits, or bytes, or digits, or nepers, or some other quantity that's basically a logarithm, is a measure of what we don't know about a system. In the context of thermodynamics, and therefore what we think of as temperature, this means all the ways the atoms and molecules and assorted what-not in a system can vibrate and still give us the system we observe. Temperature, therefore, is the amount of energy each of those "degrees of freedom" hold, averaged over the lot. The point of the matter is that, where temperature is concerned, what we really care about is the entropy, which is independent of the LTM system.
The problem is, logarithms, like counts, are fundamentally unitless. To properly keep track of entropy, we have to give it a unit. SI gives it the unit joule per kelvin. Or rather, it defines a unit for temperature (kelvin), then divides it into the unit for energy (joule - itself derived) to give units of entropy. SI goes this way because it is actually quite easy to measure temperature (thermometers exist) but almost impossible to measure entropy directly, and SI is fundamentally an engineer's system, not a scientist's. There do exist systems where it is easier to measure entropy - mostly in information science, google "Shannon entropy" when you have the time - and here we see entropy measured in bits (or whatever is most apropos) and temperature measured in units of energy per bit.
So, why do we care about temperature slash entropy in thermodynamics? Because objects in equilibrium have the same temperature, but not necessarily the same energy. A larger thing (which has more atoms, and therefore more ways those atoms can vibrate) will have more energy than a smaller thing at the same temperature. And since thermodynamics is all about how things attain and maintain equilibrium, it helps to know where that equilibrium point actually is, which energy can't help us with - at least not directly. Enter the thermometer and with it, temperature and entropy.