55

Our ability to store data on or in physical media continues to grow, with the maximum amount a data you can store in a given volume increasing exponentially from year to year. Storage devices continue to get smaller and their capacity gets bigger.

This can't continue forever, though, I would imagine. "Things" can only get so small; but what about information? How small can a single bit of information be?

Put another way: given a limited physical space -- say 1 cubic centimeter -- and without assuming more dimensions than we currently have access to, what is the maximum amount of information that can be stored in that space? At what point does the exponential growth of storage density come to such a conclusive and final halt that we have no reason to even attempt to increase it further?

Qmechanic
  • 201,751
tylerl
  • 684
  • this is a great question having to do with Bousso's covariant entropy bound - see my answer –  Dec 27 '10 at 03:04
  • a hydrogen atom has infinitely many energy eigenstates... – Mark Eichenlaub Dec 27 '10 at 03:54
  • @MarkEichenlaub But surely the higher and higher energy eigenstates fill up more and more space: IIRC there is no bound on the eigenstate "size" as you go higher in energy. – Selene Routley Sep 30 '13 at 07:54

4 Answers4

38

The answer is given by the covariant entropy bound (CEB) also referred to as the Bousso bound after Raphael Bousso who first suggested it. The CEB sounds very similar to the Holographic principle (HP) in that both relate the dynamics of a system to what happens on its boundary, but the similarity ends there.

The HP suggests that the physics (specifically Supergravity or SUGRA) in a d-dimensional spacetime can be mapped to the physics of a conformal field theory living on it d-1 dimensional boundary.

The CEB is more along the lines of the Bekenstein bound which says that the entropy of a black hole is proportional to the area of its horizon:

$$ S = \frac{k A}{4} $$

To cut a long story short the maximum information that you can store in $1\ \mathrm{cm^3} = 10^{-6}\ \mathrm{m^3}$ of space is proportional to the area of its boundary. For a uniform spherical volume, that area is:

$$ A = V^{2/3} = 10^{-4}\ \mathrm{m^2}$$

Therefore the maximum information (number of bits) you can store is approximately given by:

$$ S \sim \frac{A}{A_\mathrm{pl}} $$

where $A_\mathrm{pl}$ is the planck area $ \sim 10^{-70}\ \mathrm{m^2}$. For our $1\ \mathrm{cm^3}$ volume this gives $ S_\mathrm{max} \sim 10^{66} $ bits.

Of course, this is a rough order-of-magnitude estimate, but it lies in the general ballpark and gives you an idea of the limit that you are talking about. As you can see, we still have decades if not centuries before our technology can saturate this bound!


Edit: Thanks to @mark for pointing out that $1\ \mathrm{cm^3} = 10^{-6}\ \mathrm{m^3}$ and not $10^{-9}\ \mathrm{m^3}$. Changes final result by three orders of magnitude.

On Entropy and Planck Area

In response to @david's observations in the comments let me elaborate on two issues.

  1. Planck Area: From lqg (and also string theory) we know that geometric observables such as the area and volume are quantized in any theory of gravity. This result is at the kinematical level and is independent of what the actual dynamics are. The quantum of area, as one would expect, is of the order of $\sim l_\mathrm{pl}^2$ where $l_\mathrm{pl}$ is the Planck length. In quantum gravity the dynamical entities are precisely these area elements to which one associates a spin-variable $j$, where generally $j = \pm 1/2$ (the lowest rep of SU(2)). Each spin can carry a single qubit of information. Thus it is natural to associate the Planck areas with a single unit of information.

  2. Entropy as a measure of Information: There is a great misunderstanding in the physics community regarding the relationship between entropy $S$ – usually described as a measure of disorder – and useful information $I$ such as that stored on a chip, an abacus or any other device. However they are one and the same. I remember being laughed out of a physics chat room once for saying this so I don't expect anyone to take this at face value.

But think about this for a second (or two). What is entropy?

$$ S = k_\mathrm B \ln(N) $$

where $k_\mathrm B$ is Boltzmann's constant and $N$ the number of microscopic degrees of freedom of a system. For a gas in a box, for eg, $N$ corresponds to the number of different ways to distribute the molecules in a given volume. If we were able to actually use a gas chamber as an information storage device, then each one of these configurations would correspond to a unit of memory. Or consider a spin-chain with $m$ spins. Each spin can take two (classical) values $\pm 1/2$. Using a spin to represent a bit, we see that a spin-chain of length $m$ can encode $2^m$ different numbers. What is the corresponding entropy:

$ S \sim \ln(2^m) = m \ln(2) \sim \textrm{number of bits} $

since we have identified each spin with a bit (more precisely qubit). Therefore we can safely say that the entropy of a system is proportional to the number of bits required to describe the system and hence to its storage capacity.

  • 7
    I've heard this a few times, might as well ask now. What if you take a volume $V_2$ that lies inside your volume $V_1$ such that $A_2 > A_1$. Which one would be able to hold more information? – Malabarba Dec 27 '10 at 04:24
  • 1
    @space_cadet: This has the makings of a great answer; my one (hopefully constructive) criticism is that you don't really explain why the proportionality constant $S/A$ is related to $A_{pl}$. Of course a full proof would be overkill, but I think it'd help to include a few words on the significance of the Planck area in this argument, for people who aren't familiar with it. Also I'd rather see a different symbol used instead of $S$ in your last equation, since entropy doesn't quite measure the number of bits of information. (I know it's just a constant factor difference, it just looks weird) – David Z Dec 27 '10 at 04:42
  • Thank you @mark and @david for your comments. I hope the edit resolves the questions you had. As for what @bruce pointed out (nice one btw) I'm still processing and will come back if I have a response. –  Dec 27 '10 at 05:10
  • 2
    @Bruce: $V_2$ obviously; the whole point of holography is that volume doesn't matter at all, only area does :-) Of course, I am not sure to what degree this has been proved (as in calculated microscopically) for generic surfaces (not smooth even?) rather than for horizons of quite generic BH. – Marek Dec 27 '10 at 09:11
  • This is a nice answer, but I have to wonder, how much of this information can actually be exploited. It's clear that you have given absolute lower bound on that information. But in reality, we wouldn't be able to modify and read bits from BH's horizon. So I guess a bigger lower bound should exist. Or are you suggesting that all of that holographic information can somehow be managed, even in principle? – Marek Dec 27 '10 at 09:17
  • @marek - good point (two comments up). I was thinking along similar lines. Interestingly this line of reasoning sheds light on the geometric nature of the entropy bound, so is worth pursuing in greater detail. –  Dec 27 '10 at 09:19
  • @marek - this is not a lower bound. It is an upper bound. It determines the maximum amount of information that you can store in a given region. Or am I misunderstanding you? Secondly, I'm not suggesting anything about how such information can be managed. That is a separate question that will lead us to consider limits on information processing and transfer as opposed to storage. There is an interesting article on this by JB Pendry available here –  Dec 27 '10 at 09:28
  • @space_cadet: sorry, I reversed the order, I did mean to write upper. And by the way, no, it's not a separate question. If you want to store your information, it's because you want it safe and recover it later. Throwing things into BH doesn't count as storing information in my opinion (at the very least, that information will fly away as Hawking radiation sooner or later) :-) But thanks for the link. – Marek Dec 27 '10 at 09:37
  • 1
    @space_cadet: the answer is incomplete. All that it is saying is that you can only store information on the surface area. It makes sense (you need to have a surface to be able to retrieve the information). On the other hand the question asks about the volume. So the question now is: how much surface area can you fit in a given volume? http://en.wikipedia.org/wiki/Menger_sponge – Sklivvz Dec 28 '10 at 12:33
  • 2
    In the Pleasure of Finding Things out, Richard Feynman also speculates on the limits of information density. He doesn't go as far as the Planck length, as far as I can recall, presumably because that is very far of technologically, even nowadays. Interesting lecture. – Raskolnikov Dec 28 '10 at 14:52
  • @raskolnikov - Feynman was one of the original proponents of quantum computation. There is an article by him from around ~ 1982 on this topic. @Sklivvz - the information that can be stored on a black hole horizon or other surface, is the maximum information that can be stored within any region (i.e. volume) bounded by that surface. At high enough energies, space-time becomes 2+1 dimensional and the notion of volume disappears. –  Dec 28 '10 at 23:07
  • 1
    There are problems with surface-area definition of information: 1) it makes information capacity observer-dependent (for an observer under the named surface the information capacity will be much differnt) 2) it is not additive and depends on the division we choose for the space: sum of many 1 cm^3 volumes evidently can store more information than one area of the same volume and the both smaller than a volume that encompasses both. 3) It does not account differences between quantum and classical information 4) It is not reversible: for an observer the entropy of the universe turns to be limited – Anixx Apr 10 '11 at 10:09
  • it makes information capacity observer-dependent ... I'm not sure what you mean by that. for an observer under the named surface the information capacity will be much differnt I'm sorry, I don't understand this. Do you mean that an observer within the bounded region will see something different from an observer on the outside? it is not additive and depends on the division we choose for the space ... there is a restriction on the sorts of surfaces which saturate the entropy bound. These are generally minimal or "extremal" surfaces which are unique for a given spacetime. –  Apr 10 '11 at 10:26
  • It does not account differences between quantum and classical information ... again I don't understand the point you're trying to make. Please elaborate on what these differences are and how holography fails to account for them. 4) It is not reversible: for an observer the entropy of the universe turns to be limited ... Is that one criticism or two? Holography actually ensures unitarity and hence (microscopic) reversibility of a quantum gravity theory. See Lubos' answer to question 5407. Also, I don't see how the entropy of the universe being a finite quantity is undesirable. –  Apr 10 '11 at 10:30
  • 1
    In the mathematical discipline of information theory, the entropy of a message is the expected value of the information contained in that message. The formulas are the same, so it shouldn't be surprising that entropy is a measure of information content in physical systems as well. – Nick Alger Sep 08 '12 at 04:41
  • Is the answer also true shortly after the big bang ? – jjcale Sep 30 '13 at 08:35
1

Ok then, let's say that for a given volume, and nano molecular data retrieval technology. Assuming that you want the data safe, retrievable, made of a long term stable atom what is the maximum data that can usefully be stored.

So firstly we need 1/2 of the total volume to be used for a single molecular layer of your chosen molecule, this will be the "platter" for our "hard drive".

Onto this you place the atoms that represent bits so you have your volume divided by the volume of your chosen molecule/element divided by 2 as the total number of bits.

But with molecular storage, you could use different molecules and have for example,

No molecule = 0 Gold = 1 Platinum =2 Silver = 3

Then you have 4 bit data storage without much loss in size, throw in some carbon 12 and carbon 13 and your up to 6 bit, find some more stable elements and your up to 8 bit and so on.

Of course data retrieval would be terribly slow, but for long term, small size storage. Your talking quadrillions of bits per cm3

0

The conversion factor from physical entropy to information entropy (in random bits) uses Landauer's limit: (physical entropy)=(information bits)*kb*ln(2). The number of yes/no questions that have to be asked to determine which state a physical system is in is equal to Shannon's entropy in bits, but not Shannon's intensive, specific entropy H, but his extensive, total entropy of a data-generating source: S=N*H where H=1 if the n bits are mutually independent.

Landauer's limit states that 1 bit of information irreversibly changing state releases entropy kb*ln(2), which is a heat energy reelase for a given T: Q=T*kb, implying there was a stored potential energy that was the bit. This shows that entropy is information entropy: the ln(2) converts from ln() to log2(). kb is a simple conversion factor from average kinetic energy per particle (definition of temperature) to heat joules which has units of joules/joules, i.e. unitless. If our T was defined in terms of joules of kinetic energy (average 1/2 mv^2 of the particles) instead of Kelvins, then kb=1. So kb is unitless joules/joules. It's not a fundamental constant like h. c also does not have fundamental units if you accept time=i*distance as Einstein mentioned in appendix 2 of his book, allowing use of the simpler Euclidean space instead of Minkoswki space without error or qualification and in keeping with Occam's razor.

Shannon's "entropy" (specific, intensive) is H=sum(-p*log(p)) and he stated 13 times in his paper that H has units of bits, entropy, or information PER SYMBOL, not bits (total entropy) as most people assume. An information source generates entropy S=N*H where N is the number of symbols emitted. H is a "specific entropy" based on the probability of "n" unique symbols out of N total symbols. H is not a "total entropy" as is usually believed, finding its physical parallel with So=entropy/mole. Physical S=N*So and information S=N*H. It is rare to find texts that explain this.

An ideal monoatomic gas (Sackur-Tetrode equation) has an entropy from N mutually independent gas particles of S=kb*sum(ln(total states/i^(5/2)) where the sum is over i=1 to N. This is approximated by Stirling's formula to be S=kb*N*[ln(states/particle)+5/2]. I can't derive that from Shannon's total entropy S=N*H even though I showed in the first paragraph the final entropies are exactly the same. I am unable to identify an informatic "symbol" in a physical system. The primary problem seems to be that physical entropy is constrained by total energy which gives it more possible ways to use the N particles. 1 particle carrying the total energy is a possible macrostate (not counting the minimal QM state for the others), but information entropy does not have "check sum" like this to use fewer symbols. Physical entropy seems to always(?) be S=kb*N*[ln(states/particle)+c] and the difference from information entropy is the c. But in bulk matter where the energy is spread equally between bulks, physical entropy is S=N*So. Information entropy is perfectly like this (S=N*H), but I can't derive So from H. Again, Sbits=S/(kb*ln(2)).

So Shannon's entropy is a lot simpler and comes out to LESS entropy if you try to make N particles in a physical system equivalent to N unique symbols. The simplest physical entropy is of independent harmonic oscillators in 1D sharing a total energy but not necessarily evenly is S=kb*ln[(states/oscillator)^N / N!] which is S=N*[log(states/particle)+1] for large N. So even in the simplest case, the c remains. Shannon entropy is of a fundamentally different form: S~log((states/symbol)^N) = N*log(states/symbol) when each symbol is mutually independent (no patterns in the data and equal symbol probabilities). For example, for random binary data S=log2(2^N) = N bits. So it is hard to see the precise connection in the simplest case (the +1 is not a minor difference), even as they are immediately shown by true/false questions to be identical quantities with a simple conversion factor. Stirling's approximation is exact in the limit of N and Shannon's H depends in a way on an infinite N to get exact p's, so the approximation is not a problem to me.

I have not contradicted anything user346 has said but I wanted to show why the connection is not trivial except in the case of looking at specific entropy of bulk matter. QM uses S=sum(-p*log(p)) but Shannon entropy is S=N*sum(-p*log(p)). They come out the same because calculating the p's is different. Physical's p=(certain macrostate)/(total microstates) but the numerator and denominator are not simply determined by counting. Information's p=(distinct symbol count)/(total symbols) for a given source. And yet, they both require the same number of bits (yes/no questions) to identify the exact microstate (after applying kb*ln(2) conversion).

But there's a problem which was mentioned in the comments to his answer. In an information system we require the bits to be reliable. We can never get 100% reliability because of thermal fluctuations. At this limit of 1 bit = kb*ln(2) we have a 49.9999% probability of any particular bit not being in the state we expected. The Landuaer limit is definitely a limit. The energy required to break a bond that is holding one of these bits in a potential memory system is "just below" (actually equal) to the average kinetic energy of the thermal agitations. Landauer's limit assumes the energy required to break our memory-bond is E=T*kb*ln(2) which is slightly weaker than a van der waals bond which is about the weakest thing you can call a "bond" in the presence of thermal agitations.

So we have to decide what level of reliability we want our bits. Using the black hole limit also seems to add a problem of "accessibility". It is the information content of the system, but it is not an information storage system.

zawy
  • 249
0

I'm not a physicist, but I do know computer science, and keep up with the basics of physics, so let me give another answer for this:

We don't know yet. As long as there are smaller things that can be found, changed, and observed, we can use them to store information.

For example, if a new quantum property is found which can be in state A or state B, that's a new bit. If that's in every billion atoms of something, that's a billion more bits of data. If we then learn to manipulate that property into two additional states (say, right-way-out, and inside-out), then we've just added a new bit, raising that capacity to the power of 2.

So, the problem is that we're still learning what matter and spacetime are made of. Until we come up with a provably correct, unified theory, we don't know how many varying things there are within any material. Given that every single additional state is at least a ^2 change in information density, it's fairly useless to give "ballpark" figures until we know more. So it's probably just better to give something like Moore's Law - a prediction that we'll double the storage every so often, until we run out of new discoveries/technologies.