Offscreen mousemovement/click with something like Sikuli?

[rodrigojager]

Is it possible to cast a mouse movement and click inside a browser that is rendering offscreen? So let's say I want the mouse pointer to move to a position (X,Y) related to that of the captured image and perform a mouseclick on it. Is it possible? If so, maybe we can use tools like Sikuli that is capable of make automation using visual cues on the screen, but offscreen. If this is possible, is it possible to run multiple instances and doing this exactly same thing (or even something else), but not synced between each instance? This is useful in case I want to automate a test in a website that uses Flash or Silverlight, where I don't have available the ids of buttons for instance.

[amaitland]

Is it possible to cast a mouse movement and click inside a browser that is rendering offscreen? So let's say I want the mouse pointer to move to a position (X,Y) related to that of the captured image and perform a mouseclick on it. Is it possible?

Yes. There are method for sending mouse and keyboard events. WPF implementation uses the same offscreen rendering engine, you can use it as a reference.

https://github.com/cefsharp/CefSharp/blob/master/CefSharp.Wpf/ChromiumWebBrowser.cs#L2412

Using CefSharp.Dom is likely a better solution than just raw mouse events though.

https://github.com/cefsharp/CefSharp.Dom/blob/main/lib/PuppeteerSharp.Tests/ClickTests/ClickTests.cs

If so, maybe we can use tools like Sikuli that is capable of make automation using visual cues on the screen, but offscreen.

No idea. Never even heard of Sikuli

is it possible to run multiple instances and doing this exactly same thing (or even something else), but not synced between each instance?

If you use CefSharp.Dom then that should be fine. Raw mouse events cannot be sent concurrently two different instances.

This is useful in case I want to automate a test in a website that uses Flash or Silverlight,

Neither of these are supported. Haven't been for years.

---

For general automation of browsers then https://github.com/hardkoded/puppeteer-sharp is likely a better choice.

[rodrigojager]

Thanks a lot for your reply. I will be using CefSharp.NET for some projects for sure. Sikuli is a tool that can be used to cast a click in elements where you don't have another way of identifying the element besides its appearence. Let's say I have a button in a page but for some reason it doesn't have an exposed id (like in a flash or silverlight frame) in the DOM. What you can do is to capture a screenshot of the element beforehand and save it, so later you can "tell" Sikuli to search for that set of pixels on the screen and if it finds it, and to perform a mouseclick on it. It is a computer vision based automation tool.