8.8 Problems in polyphonic detection

The techniques we used for our recognition system have for the most part been applied to monophonic recordings. In moving to polyphonic recordings, we have to subtract out the portion of the sound signal due to the first note and repeat the pitch detection and instrument recognition algorithm for each successive note. We implemented a simple masking function to remove the portions of the spectrum contributed by the detected note. Our first trial used a simple binary mask and removed all the harmonics given a fundamental frequency, but this has the problem of removing potentially significant information if the next note is a harmonic of the first, as their spectrum would therefore overlap. We thus decided to use a more intelligent mask and remove parts of the spectrum using knowledge about the instrument that produced the note. The mask was constructed from the average harmonic envelope and fundamental frequency in a process similar to the inverse of sinusoidal harmonic modeling. However, we only work with the spectral amplitude and not with phase. Because the spectral amplitude of multiple notes is not linear, however, the harmonic peaks in a polyphonic tune cannot simply be subtracted as we have done. We note that accounting for the phase differences is a non-trivial problem, and the simplifying assumption of linearity in spectral amplitude is often used in polyphonic systems.