Hard example mining on instance level?

[Jacobew]

OHEMSampler in mmdetection mines the hard RoIs, and in mmsegmentation OHEMSampler mines the hard pixel.
I am trying to mine the hard instance (i.e., image) according to the loss value and let my model train more times on these hard images.
I wonder how I can dynamically update the hard image bank since the sampler seems to determine the whole training list in an epoch in advance.