Relation between entropy and information

Question

Lack of information is entropy. But entropy has dimension and information has no dimension. Then how do we relate these two terms..

Answer 1

Information has a pseudo-unit. Like angles can be measured in "radians," "cycles," or "degrees," entropy can be measured in "bits," "digits," or "nats." The units for entropy are $J\,\mathrm{K}^{-1}$. If you look at the Boltzmann or Gibbs formulas for entropy, though, those units are just bolted on in Boltzmann's constant out front. Now, it may be intellectually dissatisfying to theorists who like to work with "natural units", but there's a good reason for it.

The universal gas constant in $PV=nRT$ is approximately $R = 8\,\mathrm{J\,mol^{-1}\,K^{-1}}$. It has a nice value similar to 1. If we look at the Boltzmann constant version, though, we get $PV=NkT$ giving $k$ a value like $10/(\text{Avogadro's number})$ (in fact, $R=kN_A$). In other words, because of the way combinatorics works, real thermodynamic entropies would be stupidly large numbers without a pseudo-unit to bring it down to size, even inside of a logarithm. In other words, you can think of $\mathrm{J}\,\mathrm{K}^{-1}$ as measuring entropy in approximately deci-Avogadro's number of nats.

Now, there are many who object that thermodynamic entropy and information entropy are not the same thing. For starters, information theory entropy is observer dependent, while thermodynamic entropy is, to the best of our ability to determine it, not. I think, honestly, that thermodynamic entropy may end up being observer dependent but only if quantum entanglement can make the number of states available to the system observer dependent.

In a little more detail, the Boltzmann formula for entropy is $$S = k\ln W$$ where $W$ is the number of microstates consistent with the macro-state of the system. This is the form of entropy that you use for an isolated system. If you compare it to the Gibbs entropy, it is the same as if you assume all states with non-zero probability are equally probable.

Information theory borrowed the formula for Gibbs entropy, that's why they called "information" "entropy". There the probabilities aren't tied to "states of the system" but to some enumerated set of symbols that could be produced by a communication channel.

If you say that the output of a communication channel is analogous to the outcome of an experiment measuring the state of a system, it's tempting to say that these quantities are the same thing. They're not, though. Classically, you could invoke a Maxwell's demon type character that knows the micro-state of the system/the future outputs of the channel. For this demon, both systems would have zero information entropy. In information theory that isn't a problem, since the entropy there is just a measure of the surprise of the observer of the channel's output.

For thermodynamics, though, the entropy is tied to things like temperature, chemical reaction rates, internal energy, etc. All of those things have an observer independent status to them. Furthermore, an observer who is simply ignorant of some aspect of the system's macro-state (e.g. she doesn't know the number of moles, or he doesn't know the total internal energy) cannot affect these things, even though there are a whole lot more micro-states consistent with the ignorant observer's picture. Thus the information entropy and thermodynamic entropy cannot be the same things.

Where quantum mechanics comes in is inherently in Schrödinger's cat-like scenarios. Suppose some thermodynamic system $\sigma$ has two macro-states available to it labeled $1$ and $2$. Suppose, also, that these states are, somehow, quantum mechanically coherent and the system has a 50-50 chance of being in either. Finally, let's suppose that the total entropy of $\sigma$ is $S_0$. If observer $A$ measures the system then, in multi-universe type interpretations of quantum mechanics, then they enter into an entangled state where there is a 50-50 chance for $(A,\sigma)$ to jointly be in states 1 or 2. Now, for observer $A$ $\sigma$ is now in a state with an entropy of $S_0 - k_B\ln 2$. For observer $B$, who is not entangled, the combined $(A,\sigma)$ grouping will still have entropy $S_0$.

That's one way that quantum mechanics may introduce an observer dependence to entropy that is still physically sensible. There's an awful lot of "if" in that description, though, so I wouldn't bet on it ultimately being one way or the other.

Answer 2

Entropy has pseudo units. In statistical mechanics we encounter the quantity $\beta = 1/k_B T$ with units of inverse energy, where $T$ is absolute temperature and $k_B$ is Boltzmann's constant. Temperature alone is never encountered, it is always in the form of the product $k_B T$. By a fluke of history and human inertia temperature got the arbitrary unit of kelvin, which necessitates the introduction of Boltzmann's constant $k_B = 1.38\times 10^{-23}$ J/K as a unit conversion between kelvin and joule. But we could as well define the unit of temperature to be joule, which would make Boltzmann's constant dimensionless and equal to 1. So, when in thermodynamics we write Gibbs's entropy, $$S = -k_B \sum_i p_i \ln p_i,$$ we are writing the same equation as Shannon, only reported in arbitrary units.

Answer 3

The other two answers are correct, but here's my take on it: the thermal energy on a particle is k_B/2 per degree of freedom. The units on the entropy come from the k_B, which is simply a conversion factor between joules and kelvins as Themis clearly states. (The 1/2 comes, I think, because in a collision you have two particles hitting each other, each with thermal energy)

If you refine entropy down to degrees of freedom, it ought to have the same unit basis as information. The ln comes in because the degrees of freedom in a number rise more slowly than its magnitude (which is why you can write 1000001 with less than a million and one digits). The binary bits in the number of microstates written in base 2 (which, times k_B ln 2, is the entropy) are the same as the bits of information you would need to obtain to nail that undetermined macrostate down to one particular microstate. (The same would be true for entropy written in base 3, or I suppose base e if you can manage that. :)