Assertion failure when redoing a selection transformation

[scribblemaniac]

Issue Summary

If a selection is transformed and the undo/redo mechanism is used in a specific order, then this assert will be triggered, and crash on debug builds.

Steps to reproduce

1. Create a selection with the select tool.
2. Switch to the move tool.
3. Transform the selection.
4. Undo the selection transformation.
5. Switch to the pen tool.
6. Redo the selection transformation.
7. Attempt to draw through the selection.

Actual Results

The program assert fails on debug builds when following the steps outlined in the section above. On release builds, the selection does not get applied before drawing and the drawing tool will behave as if the selection has not been transformed. This will result in some visual anomalies as the brush data conflicts with the selection data. After each stroke is finished, the program behaves as if the stroke had been drawn before the selection was transformed, and the selection remains active with a transformation.

Expected Results

It is expected that the assert does not fail obviously. When redoing a selection, it should probably apply the transform if the move tool is not active, or perhaps just if there is no active selection transform.

System Information

• Pencil2D Version: ac11a81

• Operating System: 22.04

• RAM Size: 16GB

• Graphics Tablet: N/A

---

Developer Notes

This has many similarities to #1838, likely the selection is in a similar state but through different means. There may be one fix for both issues, or separate fixes depending on where the fix is applied to.