<< Chapter < Page Chapter >> Page >

For i.i.d. sources, D ( P 1 ( x n ) | | P 2 ( x n ) ) = n D ( P 1 ( x i ) | | P 2 ( x i ) ) , which means that the divergence increases linearly with n . Not only does the divergence increase, but it does so by a constant per symbol.Therefore, based on typical sequence concepts that we have seen, for an x n generated by P 1 , its probability under P 2 vanishes. However, we can construct a distribution Q whose divergence with both P 1 abd P 2 is small,

Q ( x n ) = 1 2 P 1 ( x n ) + 1 2 P 2 ( x n ) .

We now have for P 1 ,

1 n D ( P 1 n | | Q ) = 1 n E log P 1 ( x n ) 1 2 P 1 ( x n ) + 1 2 P 2 ( x n ) 1 n log ( 2 ) = 1 n .

On the other hand, 1 n D ( P 1 ( x 1 n ) | | Q ( x 1 n ) ) 0 [link] , and so

1 n 1 n D ( P 1 ( x 1 n ) | | Q ( x 1 n ) ) 0 .

By symmetry, we see that Q is also close to P 2 in the divergence sense.

Intuitively, it might seem peculiar that Q is close to both P 1 and P 2 but they are far away from each other (in divergence terms). This intuition stems from the triangle inequality, which holds for all metrics. The contradiction is resolved by realizingthat the divergence is not a metric, and it does not satisfy the triangle inequality.

Note also that for two i.i.d. distributions P 1 and P 2 , the divergence

D ( P 1 ( x n ) | | P 2 ( x n ) ) = n D ( P 1 | | P 2 )

is linear in n . If Q were i.i.d., then D ( P 1 ( x n ) Q ( x 1 n ) ) must also be linear in n . But the divergence is not increasing linearly in n , it is upper bounded by 1. Therefore, we conclude that Q ( · ) is not an i.i.d. distribution. Instead, Q is a distribution that contains memory, and there is dependence in Q between collections of different symbols of x in the sense that they are either all drawn from P 1 or all drawn from P 2 . To take this one step further, consider K sources with

Q ( x n ) = i = 1 K 1 K P i ( x n ) ,

then in an analogous manner to before it can be shown that

D ( P i ( x 1 n ) | | Q ( x 1 n ) ) 1 n log ( K ) .

Sources with memory : Instead of the memoryless (i.i.d.) source,

P ( x n ) = i = 1 n P ( x i ) ,

let us now put forward a statistical model with memory,

P ( x n ) = i = 1 n P ( x i | x 1 i - 1 ) .

Stationary source : To understand the notion of a stationary source, consider an infinite stream of symbols, . . . , x - 1 , x 0 , x 1 , ... . A complete probabilistic description of a stationary distribution is given by the collection of allmarginal distribution of the following form for all t and n ,

P X t , X t + 1 , . . . , X t + n - 1 ( x t , x t + 1 , . . . , x t + n - 1 ) .

For a stationary source, this distribution is independent of t .

Entropy rate : We have defined the first order entropy of an i.i.d. random variable [link] , and let us discuss more advanced concepts for sources with memory.Such definitions appear in many standard textbooks, for example that by Gallager  [link] .

  1. The order- n entropy is defined,
    H n = 1 n H ( x 1 , . . . , x n ) = - 1 n E [ log ( P ( x 1 , . . . , x n ) ) ] .
  2. The entropy rate is the limit of order- n entropy, H ¯ = lim n H n . The existence of this limit will be shown soon.
  3. Conditional entropy is defined similarly to entropy as the expectation of the log of the conditional probability,
    H ( x n | x 1 , . . . , x n - 1 ) = - 1 n E [ log ( P ( x n | x 1 , . . . , x n - 1 ) ) ] ,
    where expectation is taken over the joint probability space, P ( x 1 , . . . , x n ) .

The entropy rate also satisfies H ¯ = lim n H ( x n | x 1 , . . . , x n ) .

Theorem 3 For a stationary source with bounded first order entropy, H 1 ( x ) < , the following hold.

  1. The conditional entropy H ( x n | x 1 , . . . , x n - 1 ) is monotone non-increasing in n.
  2. The order- n entropy is not smaller than the conditional entropy,
    H n ( x ) H ( x n | x 1 , . . . , x n - 1 ) .
  3. The order- n entropy H n ( x ) is monotone non-increasing.
  4. H ¯ ( x ) = lim n H n ( x ) = lim n H ( x n | x 1 , . . . , x n - 1 ) .

Questions & Answers

what does the ideal gas law states
Joy Reply
Three charges q_{1}=+3\mu C, q_{2}=+6\mu C and q_{3}=+8\mu C are located at (2,0)m (0,0)m and (0,3) coordinates respectively. Find the magnitude and direction acted upon q_{2} by the two other charges.Draw the correct graphical illustration of the problem above showing the direction of all forces.
Kate Reply
To solve this problem, we need to first find the net force acting on charge q_{2}. The magnitude of the force exerted by q_{1} on q_{2} is given by F=\frac{kq_{1}q_{2}}{r^{2}} where k is the Coulomb constant, q_{1} and q_{2} are the charges of the particles, and r is the distance between them.
Muhammed
What is the direction and net electric force on q_{1}= 5µC located at (0,4)r due to charges q_{2}=7mu located at (0,0)m and q_{3}=3\mu C located at (4,0)m?
Kate Reply
what is the change in momentum of a body?
Eunice Reply
what is a capacitor?
Raymond Reply
Capacitor is a separation of opposite charges using an insulator of very small dimension between them. Capacitor is used for allowing an AC (alternating current) to pass while a DC (direct current) is blocked.
Gautam
A motor travelling at 72km/m on sighting a stop sign applying the breaks such that under constant deaccelerate in the meters of 50 metres what is the magnitude of the accelerate
Maria Reply
please solve
Sharon
8m/s²
Aishat
What is Thermodynamics
Muordit
velocity can be 72 km/h in question. 72 km/h=20 m/s, v^2=2.a.x , 20^2=2.a.50, a=4 m/s^2.
Mehmet
A boat travels due east at a speed of 40meter per seconds across a river flowing due south at 30meter per seconds. what is the resultant speed of the boat
Saheed Reply
50 m/s due south east
Someone
which has a higher temperature, 1cup of boiling water or 1teapot of boiling water which can transfer more heat 1cup of boiling water or 1 teapot of boiling water explain your . answer
Ramon Reply
I believe temperature being an intensive property does not change for any amount of boiling water whereas heat being an extensive property changes with amount/size of the system.
Someone
Scratch that
Someone
temperature for any amount of water to boil at ntp is 100⁰C (it is a state function and and intensive property) and it depends both will give same amount of heat because the surface available for heat transfer is greater in case of the kettle as well as the heat stored in it but if you talk.....
Someone
about the amount of heat stored in the system then in that case since the mass of water in the kettle is greater so more energy is required to raise the temperature b/c more molecules of water are present in the kettle
Someone
definitely of physics
Haryormhidey Reply
how many start and codon
Esrael Reply
what is field
Felix Reply
physics, biology and chemistry this is my Field
ALIYU
field is a region of space under the influence of some physical properties
Collete
what is ogarnic chemistry
WISDOM Reply
determine the slope giving that 3y+ 2x-14=0
WISDOM
Another formula for Acceleration
Belty Reply
a=v/t. a=f/m a
IHUMA
innocent
Adah
pratica A on solution of hydro chloric acid,B is a solution containing 0.5000 mole ofsodium chlorid per dm³,put A in the burret and titrate 20.00 or 25.00cm³ portion of B using melting orange as the indicator. record the deside of your burret tabulate the burret reading and calculate the average volume of acid used?
Nassze Reply
how do lnternal energy measures
Esrael
Two bodies attract each other electrically. Do they both have to be charged? Answer the same question if the bodies repel one another.
JALLAH Reply
No. According to Isac Newtons law. this two bodies maybe you and the wall beside you. Attracting depends on the mass och each body and distance between them.
Dlovan
Are you really asking if two bodies have to be charged to be influenced by Coulombs Law?
Robert
like charges repel while unlike charges atttact
Raymond
Got questions? Join the online conversation and get instant answers!
Jobilize.com Reply

Get Jobilize Job Search Mobile App in your pocket Now!

Get it on Google Play Download on the App Store Now




Source:  OpenStax, Universal algorithms in signal processing and communications. OpenStax CNX. May 16, 2013 Download for free at http://cnx.org/content/col11524/1.1
Google Play and the Google Play logo are trademarks of Google Inc.

Notification Switch

Would you like to follow the 'Universal algorithms in signal processing and communications' conversation and receive update notifications?

Ask