7.1 Entropy

Information sources take very different forms. Since the information is not known to the destination, it is then bestmodeled as a random process, discrete-time or continuous time.

Here are a few examples:

• Digital data source ( e.g. , a text) can be modeled as a discrete-time and discrete valued randomprocess $X_1$ , $X_2$ ,…, where $X_i\in \{A, B, C, D, E, \dots \}$ with a particular $p_{X_1}(x)$ , $p_{X_2}(x)$ ,…, and a specific $p_{X_1{X}_2}$ , $p_{X_2{X}_3}$ ,…, and $p_{X_1{X}_2{X}_3}$ , $p_{X_2{X}_3{X}_4}$ ,…, etc.
• Video signals can be modeled as a continuous time random process. The power spectral density is bandlimited toaround 5 MHz (the value depends on the standards used to raster the frames of image).
• Audio signals can be modeled as a continuous-time random process. It has been demonstrated that the power spectraldensity of speech signals is bandlimited between 300 Hz and 3400 Hz. For example, the speech signal can be modeled as aGaussian process with the shown power spectral density over a small observation period.

These analog information signals are bandlimited. Therefore, if sampled faster than the Nyquist rate, they can be reconstructedfrom their sample values.

An intuitive and meaningful measure of information should have the following properties:

• Self information should decrease with increasing probability.
• Self information of two independent events should be their sum.
• Self information should be a continuous function of the probability.

Entropy

A more basic explanation of entropy is provided in another module .

Joint Entropy

The joint entropy of two discrete random variables ( $X$ , $Y$ ) is defined by

$$H(X, Y)=-\sum \sum p(X, , Y, x_i, y_j)\lg p(X, , Y, x_i, y_j)$$

The joint entropy for a random vector $X=\left(\begin{array}{c}{X}_1\\ X_2\\ \dots \\ X_n\end{array}\right)$ is defined as

Conditional Entropy

The conditional entropy of the random variable $X$ given the random variable $Y$ is defined by

$$H(X|Y)=-\sum \sum p(X, , Y, x_i, y_j)\lg p_{X|Y}(x_i|y_j)$$

It is easy to show that

Entropy Rate

The entropy rate of a stationary discrete-time random process is defined by

$$H=\lim_{n\to }n\to$$∞ H X n | X 1 X 2 … X n

The limit exists and is equal to

$$H=\lim_{n\to }n\to$$∞ 1 n H X 1 X 2 … X n

The entropy rate is a measure of the uncertainty of information content per output symbol of the source.

Entropy is closely tied to source coding . The extent to which a source can be compressed is related to its entropy. In 1948,Claude E. Shannon introduced a theorem which related the entropy to the number of bits per second required to representa source without much loss.