5.1 Information

In this module we introduce the concept of self information for an outcome of a stochastic variable.

The self information formula

An intuitive and meaningful measure of self information in an event should have the following properties:

• The more uncertain you, in advance, are about the outcome, the more new information you get by observing the actual outcome, or equivalently an eventwith low probability, $p_n$ , has high self information $I(p_n)$ . $I(p_n)$ should be a monotonically decreasing function of $p_n$ .
• Oberserving an event with certain outcome, i.e $p_n=1$ , should give zero information. The event $p_n$ is then said to have zero self information. Since $I(p_n)$ is monotonically decreasing for $p_n\in \left[0 , 1\right]$ this implies that the self information can never be less than zero, the observer can never lose informationby observing an outcome.
• If we receive independent messages, the information should accumulate. This means that the measure must be additive.

It can be shown that there only exists one function satisfying the above conditions.

In the above equation the logarithm base can be chosen arbitrary. Usually $b=2$ is chosen so that the denomination is information bit . The choice $b=2$ is made to adapt to a digital "world", that is to facilitate electronic storage and transmission.