Recent Articles

Self-information

Within the context of information theory, self-information is defined as the amount of information that knowledge about (the outcome of) a certain event, adds to someone's overall knowledge. The amount of self-information is expressed in the unit of information: a bit.

By definition, the amount of self-information contained in a probabilistic event dependends only on the probability <math>p<math> that the event happens. More specifically: the smaller this probability is, the larger is the self-information associated with receiving information that the event indeed occurred.

Further, by definition, the measure of self-information has the following property. If an event C is composed of two mutually independent events A and B, then the amount of information at the proclamation that C has happened, equals the sum of the amounts of information at proclamations of event A and event B respectively.

Taking into account these properties, the self-information H(A) associated with event A that has a probability <math>p<math> is defined as:

<math> H(A) = log_2 (1/p) <math>

bits. This definition, using the binary logarithm function, complies with the above conditions.

This definition can be rewritten as:

<math> H(A) = - log_2 (p) <math> (bits).

Examples

• On tossing a coin, the chance of 'tail' is 0.5. When it is proclaimed that indeed 'tail' occurred, this amounts to

H('tail') = log₂ (1/0.5) = log₂ 2 = 1 bits of information.

• When throwing a die, the probability of 'four' is 1/6. When there is proclaimed that 'four' has been thrown, the amount of self-information is

H('four') = log₂ (1/(1/6)) = log₂ (6) = 2.585 bits.

• When, independently, two dice are thrown, the amount of information associated with {throw 1 = 'two' & throw 2 = 'four'} equals

H('throw 1 is two & throw 2 is four') = log₂ (1/Pr(throw 1 = 'two' & throw 2 = 'four')) = log₂ (1/(1/36)) = log₂ (36) = 5.170 bits.
This outcome equals the sum of the individual amounts of self-information associated with {throw 1 = 'two'} and {throw 2 = 'four'}; namely 2.585 + 2.585 = 5.170 bits.