AccessKit + MacOS VoiceOver seems really slow and buggy

[viridia]

I've been working on building a widget library in the Bevy game engine using Bevy's AccessKit integration. However, as you can see from the attached movie, the results when testing via VoiceOver are, well, terrible. I'm not sure at this point whether the fault is in VoiceOver, AccessKit or bevy_a11y.

accesskit-movie.mov

[DataTriny]

Thanks @viridia for giving AccessKit a try. This doesn't look great indeed.

I can run simple apps using AccessKit on my very old MacBook and it feels much more responsive than what you are experiencing.

I have troubles understanding what is going on in the video because the text said by VoiceOver doesn't seem to match the content on screen and the focus highlighter seem to point at random places, but I might have missed something as I am visually impaired. It would be useful to know for instance if the checkbox visual state immediately updates as the toggle action is performed.

What I can guess for now is that the delay doesn't seem to happen in AccessKit's tree update processing: when the focus moves it seem that the new widget is announced immediately. Multiple IPC calls have to take place before this can happen. But without knowing VoiceOver's internals this is only a guess.

I suppose you are in control of handling action requests from AccessKit, logging when it happens should already help you figure out the time between the key press and the action being received by Bevy. I think VoiceOver commands don't actually get passed to the app so you'd have to monitor the logs while you perform the commands.

Finally, the delay as shown in the video always seem to be of around 2 seconds. This might be pure luck, but it is surprizingly consistent.

[viridia]

That's useful information!

Might this be a regression in the Bevy release candidate? When you ran the Bevy example, which version of Bevy did you use?

[viridia]

No, scratch that. I just tried the official example on mainline and it works ok.

[viridia]

I have a partial solution. Adding this line:

.insert_resource(WinitSettings::desktop_app())

to the init code eliminates the delay. However, it doesn't solve the issue of the highlight rect in the wrong place.

[DataTriny]

FWIW I was using the latest commit of the main branch for all my tests. I also just ran your example on my Mac and I don't notice a significant delay. There is definitely a small one, but probably smaller than on Windows. Also, navigating with VoiceOver commands is almost immediate, showing that AccessKit apparently has no problem handling requests from assistive technologies.

The bounds of all AccessKit nodes generated by bevy_a11y are definitely wrong and are the cause for the focus highlight being drawn at the wrong location.

[alice-i-cecile]

Tracked the bounds issue in bevyengine/bevy#18685 for you :)

[ickshonpe]

Just having a look and the bounds calculations are definitely nonsense. I'm not familiar with accesskit or screenreaders though so I'm not sure how things are meant to work or how to test any changes. I'll try my best guess anyway and put up a PR.

PR: bevyengine/bevy#18706

[DataTriny]

@viridia I just tested your example again with the latest changes and it's super responsive on Windows now, so I think you are on the right track.

[DataTriny]

We have optimizations in the pipeline that should improve the situation with Bevy continuous update mode. Now that the bounds issue is fixed, I'll close this discussion.