The role of context in information theory

Question

Consider Hofstaedter’s jukebox analogy: A jukebox that contains only one record, but many different record players, each of which interprets that one record in a different way to produce an entirely different song.

A DVD contains GBytes of data, but if you cannot decode it, it represents far less information – perhaps the single bit which simply registers that you now have that DVD.

The key concept here is context, and my question is where, in all of information theory (quantum or otherwise), is the concept of context to be found? The expressions used for information seem incomplete; see e.g.: What is information?

Answer 1

There are two basic approaches to information theory, of which I'm aware:

1. Shannon's Information Theory, is based on classical probability. Since all probability is conditioned on an observer, it inherently accounts for context.

2. Kolmogorov Complexity, is based on automata theory, and it defines the information content of a string as the length of the shortest program capable of writing that string. On the one hand, it accounts for context in the form of the computer interpreting that program - for some computers the program will be shorter. On the other hand, since every universal computer can simulate every other, in a sense the definition of information is context-free or absolute.

I'm a little rusty on all this stuff, so there's certainly more to say.

Answer 2

Physical information (i.e. “information” in physics) has a well-defined meaning. One definition the minimum number of states (yes/no answers) needed to fully describe the system.

“Information” as a term is somewhat overloaded e.g. in your example you mention the DVD representing “more” information if you can decode it. Here you are using information to mean conveying of representational and conceptual information about some object (the film on the DVD). There is no film if you cannot decode the DVD!

In some ways, the context is implied, in the sense that you must first define the object(s) that you're trying to describe. In the case of the film, we might determine that a particular number of states are necessary to describe the most essential components for a human to watch the film i.e. we can use lossy compression techniques to reduce the number of states required for a sufficiently good description, taking into account the properties of human vision etc. Thus, for a given film, we can say whether it requires more or less states to describe it i.e. whether it has more or less information (by our definition).

Answer 3

If you go back to Shannon's original paper in 1948, you'll see that in positing the theory he implicitly, if not explicitly, built in the context with one of the few diagrams (top of page 2) which appears in the paper as part of the overall engineering problem. The diagram of which I speak is the one that physically shows what are all now commonly called the source, the encoder, the channel, the decoder, and the receiver of the message. In your question, you're simply changing your decoder and receiver, but the information from the source/encoder which travels through the channel are exactly the same. Thus, the DVD still has the same amount of information encoded on it, it's just that your "new decoder" is only capable of giving you a 0 or a 1 indicating that you either have the disk or not. With the more sophisticated decoder, you obviously get the full message back out of the system.

Taking this all a step further, we might now say from a philosophical standpoint that Shannon has also answered the question "if a tree falls in the woods, and no one is there, does the tree make a sound?" If nothing else, it has at least certainly sent a message!

(The careful reader will also notice that in paragraph two of the paper he specifically states: "These semantic aspects of communication are irrelevant to the engineering problem." This specifically partitions out errors in which a message is perfectly sent and perfectly received, but the proverbial person at the other end misinterpreting the actual semantic meaning.)