3.2 Quantization error in fir filters

In digital filters, both the data at various places in the filter, which are continually varying, and the coefficients, whichare fixed, must be quantized. The effects of quantization on data and coefficients are quite different, so they are analyzedseparately.

Data quantization

Typically, the input and output in a digital filter are quantized by the analog-to-digital and digital-to-analog converters,respectively. Quantization also occurs at various points in a filter structure, usually after amultiply, since multiplies increase the number of bits.

Direct-form structures

There are two common possibilities for quantization in a direct-form FIR filter structure:after each multiply, or only once at the end.

Transpose-form

Similarly, the transpose-form FIR filter structure presents two common options for quantization: after each multiply, or once at the end.

The transpose form is not as convenient in terms of supporting double-precision accumulation, which is asignificant disadvantage of this structure.

Coefficient quantization

Since a quantized coefficient is fixed for all time, we treat it differently than data quantization. The fundamental questionis: how much does the quantization affect the frequency response of the filter?

The quantized filter frequency response is $$\mathrm{DTFT}(h_Q)=\mathrm{DTFT}(h_{\mathrm{inf.\; prec.}}+e)=H_{\mathrm{inf.\; prec.}}(w)+H_e(w)$$ Assuming the quantization model is correct, $H_e(w)$ should be fairly random and white, with the error spread fairly equally over all frequencies $w\in \left[-\pi , \pi \right)$ ; however, the randomness of this error destroys any equiripple property or any infinite-precision optimality of a filter.

Ideally, if one knows the coefficients are to be quantized to $B$ bits, one should incorporate this directly into the filter design problem, and find the $M$ $B$ -bit binary fractional coefficients minimizing the maximum deviation( $L_{}$ error). This can be done, but it is an integer program, which is known to be np-hard (i.e., requires almost abrute-force search). This is so expensive computationally that it'srarely done. There are some sub-optimal methods that are much more efficient and usually produce pretty good results.