Verifying a formula for the entropy of a system

Question

I would like to show that the equation

$$S = -k\sum_{i = 1}^{N} p(n_{i}) \ln p(n_{i}),$$

holds for a binary system with a given definitinon of entropy.

Answer 1

To avoid confusion with $N$, let $M$ be the number of particles. Suppose that each particle can be in one of 2 energy states, $\epsilon_1$, and $\epsilon_2$. Suppose that we know the total energy of the system. Then we know exactly how many particles must be in each energy state. In that case, we will have $\Omega = \frac{M!}{M_1!M_2!}$, where $M_1$ is the number of particles in energy state $\epsilon_1$, and $M_2$ is the number of particles in energy state $\epsilon_2$.

This is the setup of our problem. We would now like to calculate the probability that a given particle is in state $\epsilon_1$. This is just $M_1/M$. Similarly, the probability that the particle is in state $\epsilon_2$ is $M_2/M$.

Therefore, for this particle,

$$ S=-k\sum_{i=1}^{2} p_i\ln p_i = -k \left ( \frac{M_1}{M}\ln(M_1/M) + \frac{M_2}{M}\ln(M_2/M) \right ) $$

$$ = \frac{k}{M} \left ( M\ln M - M_1\ln M_1 - M_2\ln M_2 \right ) $$

If we take the Stirling approximation for the entropy of the overall system, then we get:

$$ S = k \ln \Omega = k \ln\frac{M!}{M_1!M_2!} \approx k \left[ M\ln{M} - M -(M_1\ln M_1 - M_1) -(M_2\ln M_2 - M_2) \right] $$

$$ = k \left ( M\ln M - M_1\ln M_1 - M_2\ln M_2 \right ) $$

But this is just $M$ times the expression for the entropy of a single particle! So we are done. The two expressions for entropy are almost exactly identical.

Note: I say almost exactly, and this is important. We took a Stirling approximation to get this equality, and so it doesn't hold exactly. The error came in when we assumed that all the particles were independent from each other, so we could just add their entropies together to get the total. However, the particles weren't exactly independent of each other, and so this step wasn't really formally justified. You'd have to argue that the particles were independent enough from each other that the approximation could be made in order for this proof to work formally.