Idea: a "hints" parameter for Viterbi in DBNDownBeatTrackingProcessor

[radkoff]

Expected behaviour

If downbeats come back incorrect, it would be nice if we could help the beat/downbeat trackers get things right, by providing a rough estimate of a particular time at which a downbeat occurs. This observation (or observations?) would be passed onto the Viterbi algorithm, which would use it as an anchor while coming up with probabilities for the remainder of the song.

This might be a nice feature any time Viterbi is used in madmom

Actual behaviour

I've noticed RNNDownBeatProcessor can get things wrong, and I don't have an easy way to correct and re-run it (at least, playing with the threshold param doesn't help)

[superbock]

That's indeed something that bothered me as well. I think your idea is definitely worth investigating. What you could try is to manually alter the downbeat activation produced by RNNDownBeatProcessor. You could try to set the downbeat probability (i.e. the second column) of a single frame to 1.0. RNNDownBeatProcessor uses a fixed frame of 100 fps. I am not sure if this does the trick, but it should force Viterbi decoding to always treat this frame as downbeat.

[radkoff]

Thanks for the tip, Sebastian! For some of the incorrect downbeats in my dataset, I just switched to the beat-synchronous RNNBarProcessor/DBNBarTrackingProcessor.

There was one holdout (this song) for which I was able to do what you suggested. I had to give at least 5 hints for it to work, and I had to use .999 to avoid a division by zero, but it worked!

act = madmom.features.downbeats.RNNBarProcessor()((path, beats))
hints = [8.3, 10.35, 12.3, 14.2, 16.2, 18.2, 20.2]

for hint in hints:
    # Find the beat event that is closest to the hint
    idx = (np.abs(act[:,0] - hint)).argmin()
    act[idx][1] = 0.99999999
proc = madmom.features.downbeats.DBNBarTrackingProcessor(fps=100, meter_change_prob=0.000000001)
downbeats = proc(act)

[superbock]

This reminds me that I always wanted clip the input.

Do you mean that this attempt worked only for single song (out of how many)? Could you please send me links to songs it is still failing? That would be of great help.

[radkoff]

No, I only attempted this approach with 1 song

[radkoff]

Actually, I did find a song that this approach isn't working for: Syeeda's Song Flute

Despite adding nearly 30 hints, the downbeats come back one beat off what it should (the true downbeats come on the '4' count). Jazz is probably a difficult genre, given the off-beat percussion.

Ideally the model could handle this, but in the mean time I did some post-processing where I shifted the results by one:

# Shifts downbeats backwards by 'shift'. For example, shifting by one makes it go from 4-1-2-3 to 1-2-3-4
def applyShift(downbeats, shift):
    if shift is not 0:
        downbeats[:, 1] = np.roll(downbeats[:, 1], -1 * shift)

        # The top beats got shifted to the bottom and are unreliable, since the song may have ended mid-bar
        # Guess the meter by sampling the last 10, and fix up the end 'shift' downbeats to be ascending
        meter = downbeats[-10:][:, 1].max()
        last_value = downbeats[int(-1 * shift - 1)][1]
        mask = np.arange(shift) + 1 + last_value
        downbeats[int(-1 * shift):, 1] = 1.0 + np.mod(mask - 1.0, meter)

    return downbeats

[fdlm]

Regarding "giving hints" to the Viterbi algorithm, you are looking for the "evidential Viterbi algorithm, which Florian Krebs et al used for beat tracking in "Unsupervised learning and refinement of rhythmic patterns for beat and downbeat tracking", EUSIPCO 2014. (https://ieeexplore.ieee.org/document/6952181)

Basically, for some time steps t, you want to mask "wrong" states. In this case, if t is a downbeat position, mask all non-downbeat states (with some tolerance around t).

[radkoff]

@fdlm Thanks, that's exactly what I want!