Temporal editing of video annotations in timeline

[GMDuncan]

(The use case is more for whole-frame annotations than for geometric annotations - but could be applicable to both?)

It would be useful to be able to move keyframes along the video timeline, to allow the user to correct any temporal inaccuracy in their initial annotation, similar to how the current move tool allows for spatial correction. An example of this could be where an annotator has created a whole frame annotation for a habitat, but a reviewer/expert subsequently decides that the habitat is applicable from earlier on in the video. Current workflow would be to delete the existing annotation and remake, which is workable but not necessarily streamlined.

In addition, to aid both the above, and whole frame annotations in general, it would be a good quality control step to allow users to snap (or temporally align?) annotation keyframes from different annotations. This would allow seamless transition from one annotation to the next in the case of changing seafloor descriptors (such as habitats). In the current system, there is the possibility, if the user is not careful in the first place, to leave small gaps between whole frame annotations, or overlapping whole frame annotations where a section of the video could be classed as two things, and it is difficult to correct this once made.

Nice-to-have feature rather than a critical issue, as the above can be mitigated by good initial user behaviour, but it is always good to have the ability to correct bad users ;)

[mzur]

Thanks for the ideas!

Regarding the "temporal correction": Currently, this can be done like this: Select the keyframe that you want to modify, then create a new annotation at this time and let it span the whole "additional" time that the original annotation should have. Now select both annotations and click on the "link selected annotations" instrument. The two keyframes at the same time will be merged into one and the annotation is extended. This may be quite complicated but I think I prefer it that way for the following reasons:

• Moving a keyframe could be very imprecise, especially for long videos. One pixel movement could mean several seconds of offset and I imagine that it could be quite hard to get the position right.
• While the time of the keyframe is updated this way, the coordinates of the annotation are not. With the current method you have to explicitly set new coordinates.
• With the current method you always see the exact video frame that is affected (i.e. where the object should still be visible). You can't do that while a keyframe is moved.

Sorry to sound so dismissive - I think it is a great suggestion! - but right now I'm not sure if we should allow users to do it the way you suggested. What do you think?

Regarding the "snap to keyframe": I like this idea but how can it be implemented? One way could be to allow temporal movement of keyframes as you suggested. Another could be a new sub-instrument to the "link selected annotations" instrument (like the polygon brush is a sub-instrument to the polygon instrument)? You could select the two keyframes, click the instrument and they are automatically snapped together.

[GMDuncan]

Reading your explanation - I can fully understand the potential issues with adding the ability to move keyframes around the timeline and I agree that it would lead to issues. So I think the current method is likely the best one actually.

Just for clarification, and so we can update our workflows, your example would be where the user has not covered enough time with the original annotation - I assume the other situation (the user has made an annotation that covers too much time) would be covered by selecting the annotation, placing the cursor at the right time, using the split tool and deleting the incorrect section?

The "Snap to keyframe" is also potentially solved by the above, by the correct order of placing the cursor at the right time (by clicking the keyframe), and either linking or splitting annotations. Snapping to keyframes even with the second method could still be at its core a "move" of the keyframe and still run in to the same issues you first described? If so, I think it's likely not worthwhile proceeding with and the application "as is" (with new knowledge of best practice) is likely still the best approach? :D

(apologies for the slightly tangential idea!)

[mzur]

Just for clarification, and so we can update our workflows, your example would be where the user has not covered enough time with the original annotation - I assume the other situation (the user has made an annotation that covers too much time) would be covered by selecting the annotation, placing the cursor at the right time, using the split tool and deleting the incorrect section?

Exactly.

If so, I think it's likely not worthwhile proceeding with and the application "as is" (with new knowledge of best practice) is likely still the best approach?

I'm not sure about this one. If you want to close a gap in this scenario, the gap would probably be very small and the marked location in the video would probably be still ok if the keyframe was shifted a little. On the other hand, the current way (first selecting the keyframe of the other annotation that you want to snap onto) is the most exact, requires less interactions and thus should be faster. So maybe we shouldn't allow people to do it differently and force them to do it the "right" way.

For this to work at all you need visible handles of the existing video annotation, otherwise you can't select the keyframes. An improvement related to this is discussed in #418.