UIA handler: add ability for plugins and other subsystems to request event tracking for specific objects

[josephsl]

Hi,

With the advent of #3831, it became possible for app modules, global plugins and others to tell NVDA to keep an eye on events for specific objects. This is useful if one can isolate specific process ID's, window class name and name of the event requested, with the intention being to keep this API agnostic as much as possible (see comments on #6240). This works well for IAccessible objects and others where each object can be readily identified via window class name, but it is inadequate for UIA and other layers where objects share a common window class name.

A particular concern is UIA, specifically newer style of applications. Unlike IAccessible/MSAA, UIA uses its own set of properties to identify objects (termed "elements"), including framework ID, automation ID, unique UIA class name (if done properly) and so on. For new universal apps and others, a common window class name is used throughout an app, with UIA elements using unique UIA ID's that does not fit the API agnostic model of #3831. Also, for performance reasons, certain UIA events are blocked by NVDA, and in order to accommodate them, a UIA version of eventHandler.requestEvents is needed.

To solve this dilemma, a number of routines are proposed, united under the idea that UIA handler should be enhanced to allow plugins and other subsystems to request UIA events for specific objects, as well as remove custom event routines in specific situations such as when an app module exits. Currently, in order to request event tracking, one must know to which map an event belongs to, add the event ID and corresponding NVDA event name to the intended map, then tell UIAHandler.handler to call UIAHandler.handler.clientObject.add* function(s). For an actual working example, see Windows 10 App Essentials add-on. The goal of the proposal is to simplify this, with no need to dig into UIA client object just to tell NVDA to track wanted events.

The proposed functions are:

• UIAHandler.addEventHandler(eventType, eventID, correspondingEvent, processID=None, automationID=None, className=None): This function will add event handlers for specific events by calling UIAHandler.handler.clientObject.add* functions. The arguments are:
  • Event type: generic UIA event (e.g. live region changed), property (e.g. controller for or text change), focus change and structure change. The eventType is needed because UIA handler uses two maps to group event types, representing UIA events and property events, respectively.
  • EventID: UIA event constants, ideally those not already defined in UIA handler.
  • Corresponding event e.g. nameChange, stateChange, etc.
  • Process ID (required for app modules and objects, optional for global plugins): to keep track of processes where these events are coming from, similar to required PID argument in eventHandler.requestEvents.
  • UIA ID's (automation ID and class name): if None, all objects are considered. This is meant to let UIA handler be specific as to which events to accept from which objects.
• UIAHandler.removeEventHandler(eventID, processID=None, automationID=None, className=None): tells NVDA to forget the event specified. EventID is a required parameter, with processID being used by app modules.

This necessarily requires changes to uIAHandler.UIAHandler class, specifically when it comes to dealing with events themselves. Currently when an event comes in, it doesn't care where it comes from as long as there is a corresponding event defined in event map(s). This is more prominent with structure change event (see #6240) where a storm of them is fired constantly, thus degrading NVDA's UIA handler performance. One way to address this shortcoming is to somehow come up with a record that allows event handler to let certain objects receive priority (same approach as #3831), hence UIA ID's are needed in the end.

Possible issues and questions:

• What if UIA is unavailable (XP and Vista)? A trivial fix is to let the functions above raise runtime exceptions.
• How could a user know which category an event belongs to? Although defining event type is useful, in order for this to be user friendly, a heuristic might need to be employed in the functions themselves to figure out which map an event belongs to.
• What about specifying multiple events for different objects at once? One way is to make the functions accept an array of event records (event/NVDAEvent/PID/automationID/className), but this runs into readability and complexity issues but with the advantage that event registration becomes a for loop (can be done all at once). In contrast, one line per event requests would simplify readability but it involves calling add* functions each time addEventHandler function is called.

Comments are appreciated. Thanks.

[jcsteh]

It'd be good if you could provide some specific practical use cases where this would be useful. Being future proof is a laudable goal, but practical use cases ensure a solid implementation.

The major problem with this is performance. You noted:

Currently when an event comes in, it doesn't care where it comes from as long as there is a corresponding event defined in event map(s). This is more prominent with structure change event (see #6240) where a storm of them is fired constantly, thus degrading NVDA's UIA handler performance.

Even if we drop events based on raw UIA criteria, much of the performance hit has already been incurred: a COM object has already been created and initial information marshalled between processes. UIA events are much heavier than MSAA events in this respect. With a UIA event, the object has already been instantiated by the time we get the event. With MSAA, we just get the identifiers that allow us to instantiate an object, so if we drop the event at that stage, we don't incur the performance hit of instantiating the object and marshalling it between processes. Thus, with UIA, the only way to avoid that hit is to not register for those events at all.

You might be able to manage this by registering just for UIA events from a specified subtree. For example, you might register for structure change events only beneath the root of the app in question, or maybe even deeper in the tree. The deeper you can go, the less risk in terms of performance.

One other problem is that historically, removing event handlers tends to crash/freeze UIA, which is why we don't register for events more selectively already. (Note the "Timeout or error while waiting for UIAHandler MTA thread" that gets logged most times when exiting NVDA.) We still don't really understand why this happens. It seems to be a bug in UIA Core, but we haven't been able to reproduce it in an isolated test case yet. That essentially means that once we register for events, we can't safely unregister them.

Finally, you asked:

How could a user know which category an event belongs to? Although defining event type is useful, in order for this to be user friendly, a heuristic might need to be employed in the functions themselves to figure out which map an event belongs to.

IMO, if a developer wants to request raw UIA events, they really need to know what they're doing. One of the main reasons eventHandler.requestEvents is so agnostic is that it doesn't require specific knowledge of raw APIs. Once you step outside of that, you need to understand the raw API to some extent. If you need to understand the event ids, it's reasonable to also expect you to understand what event type you're dealing with.

A related question: I understand you want events for specific UIA criteria, but as noted above, this doesn't actually improve performance much anyway. What about just having a way to make eventHandler.requestEvents enable certain UIA events for certain process ids? For example, to get structure change events for your app module, you might do:

eventHandler.requestEvents(eventName="structureChange", processId=self.processID)

Obviously, we'd have to make windowClassName optional.

@michaelDCurran, I'd appreciate any other thoughts, since you know UIA stuff better than I.

[josephsl]

Hi,

At the moment, the most promising practical use case is handling UIA events from controls or apps with no corresponding events defined in Core (for example, listening to structure change event only for Skype Preview, or live region change event for Edge, etc.). Also, as UIA is becoming prominent (along with bugs seeing the light of day), and given the distributed nature of NVDA development, allowing those familiar with UIA could develop app modules or other add-ons that can provide experimental and practical data by investigating UIA issues by using the event tracking functions discussed above, thereby letting NV Access and others to work on other important issues.

Another point to consider is adaptability. One side might say that NVDA Core should have handlers for all possible events, thereby making its UIA handling implementation inclusive. However, real-life data suggests otherwise. A good example is listening to new events introduced in more recent UWP controls, and one way to allow NVDA to showcase its flexibility is let add-ons specify what event(s) NVDA should follow (UWP app modules, for example). Also, in case an Insider build produces UIA event oddities, NVDA can be told to add (or ignore) handlers for new (or deprecated) events for a while, and the best place to do this without impacting Core too much would be add-ons. In other words, for this case, I'd say that the biggest motivation and practical use case would be to allow UIA-aware add-ons to serve as pioneers and provide guidance to Core once NVDA is ready to deal with what's required to support next set of UIA events and/or controls (including new Windows 10 builds).

Thanks.

[josephsl]

Hi,

The last point on making class name optional in eventHandler.requestEvents: This means subscribers will be asked to listen to specific event(s) from all objects, but I think, based on structure of event handler code I've seen (and worked with), it could be a workable compromise for now.

As for handling UIA ones: we haven't found ways to assign many UIA events to ones defined by NVDA, so either we have to rely on a suitable handler to show up in Core or add-ons should provide their own sensible handlers. But this raises a question of my own: is there a way for eventHandler.requestEvents to take API type into account somehow without (or with) user intervention? If there's a way to do this in an API agnostic way, then the primary purpose of this function can be fulfilled - letting users specify events they want NVDA to listen to regardless of API set one is using; then this issue could be modified to making eventHandler function be a bit smarter.

Thanks.

[Adriani90]

cc: @michaelDCurran

[josephsl]

Hi,

I believe IUIAutomation6's event group facility might be of use for us in the long run.

Thanks.

[Adriani90]

@josephsl could you please give an update status on this? What needs to still be implemented here?

[josephsl]

Hi, selective UIA event registration makes things quite complicated as it may involve restarting UIA thread unless there is a way to add support for event tracking from app modules and objects without requiring thread restarts. Thanks.

[Adriani90]

cc: @LeonarddeR