Finite-precision error analysis

Fundamental assumptions in finite-precision error analysis

Quantization is a highly nonlinear process and is very difficult to analyze precisely. Approximations and assumptions are madeto make analysis tractable.

Assumption #1

The roundoff or truncation errors at any point in a system at each time are random , stationary , and statistically independent (white and independent of all other quantizers in a system).

That is, the error autocorrelation function is $r_e(k)=(e_n{e}_{n+k})={}_q^2(k)$ . Intuitively, and confirmed experimentally in some (but notall!) cases, one expects the quantization error to have a uniform distribution over the interval $\left[-\left(\frac{}{2}\right) , \frac{}{2}\right)$ for rounding, or $\left(- , 0\right]$ for truncation.

In this case, rounding has zero mean and variance $$(Q(x_n)-x_n)=0$$ $${}_Q^2=(e_n^2)=\frac{{}_B^2}{12}$$ and truncation has the statistics $$(Q(x_n)-x_n)=-\left(\frac{}{2}\right)$$ $${}_Q^2=\frac{{}_B^2}{12}$$

Please note that the independence assumption may be very bad (for example, when quantizing a sinusoid with an integerperiod $N$ ). There is another quantizing scheme called dithering , in which the values are randomly assigned to nearby quantizationlevels. This can be (and often is) implemented by adding a small (one- or two-bit) random input to the signal before atruncation or rounding quantizer.

Assumption #2

Pretend that the quantization error is really additive Gaussian noise with the same mean and variance as the uniform quantizer. That is, model

Summary of useful statistical facts

• Correlation function

  $(r_x(k), (x_n{x}_{n+k}))$

• Power spectral density

  $(S_x(w), \mathrm{DTFT}(r_x(n)))$
• Note $r_x(0)={}_x^2=\frac{1}{2\pi }\int_{-\pi }^{\pi } S_x(w)\,d w$
• $(r_{\mathrm{xy}}(k), (x^{*}(n)y(n+k)))$

• Cross-spectral density

  $S_{\mathrm{xy}}(w)=\mathrm{DTFT}(r_{\mathrm{xy}}(n))$
• For $y=(h, x)$ : $$S_{\mathrm{yx}}(w)=H(w)S_x(w)$$ $$S_{\mathrm{yy}}(w)=\left|H(w)\right|^2{S}_x(w)$$
• Note that the output noise level after filtering a noise sequence is $${}_y^2=r_{\mathrm{yy}}(0)=\frac{1}{\pi }\int_{-\pi }^{\pi } \left|H(w)\right|^2{S}_x(w)\,d w$$ so postfiltering quantization noise alters the noise power spectrum and may change its variance!
• For $x_1$ , $x_2$ statistically independent $$r_{x_1+x_2}(k)=r_{x_1}(k)+r_{x_2}(k)$$ $$S_{x_1+x_2}(w)=S_{x_1}(w)+S_{x_2}(w)$$
• For independent random variables $${}_{x_1+x_2}^2={}_{x_1}^2+{}_{x_2}^2$$