Selenium – Sikuli

Automate anything you see using Sikuli (an interesting GUI automation tool). As said Sikuli automates anything you see on-screen using the image recognition method to identify GUI elements on the screen.

Most of the automation testing tools will not support flash object automation (E.g. Selenium). Sikuli provides extensive support to automate flash objects. And also Sikuli provides support to automate desktop applications.

Selenium-Sikuli combination provides more support for the automation engineers. Most of the time in automation for web application, we may face to identify the elements from the web UI or the locators values are getting wrong. In such situation Sikuli comes in picture, Sikuli allows to automate GUI interaction by using images of elements instead of using the xpath, id, linktext, classes etc.,

Important steps to proceed

1. Create a Maven project in Eclipse.
2. Add following dependencies in pom.xml file:
   1. selenium-server
   2. selenium-java
   3. sikulixapi
   4. testng
3. Create TestNG classes.
4. Create WebDriver object to start driver session and load website.
5. Create a folder inside src/main/resources and store the images of web elements inside that folder.
6. Add the sikuli scripts to perform operations on the elements (like click, type, wait, doubleClick).
7. Create a testng.xml file for the created TestNG class and execute testng.xml.
8. Observe the results on executions.

Below is the sample Selenium-Sikuli Maven project structure:

Common Sikuli Actions

Create an object for Screen class, say Screen s = new Screen(); and perform various actions:

1. click :  s.click(<path to image>); – helps to click on web element using its image.
2. rightClick : s.rightClick(<path to image>); – helps to right-click on web element using its image.
3. doubleClick : s.doubleClick(<path to image>); – helps to double-click on web element that displaying in the view.
4. find : s.find(<path to image>); –  helps to locate a web element using the image of particular web objects.
5. exists : s.exists(<path to image>); – helps to check the web element is present on the view.
6. type : s.type(<path to image>, <input value to edit-box>); – helps to enter value to an input field.
7. hover : s.hover(<path to image>); – helps to mouse hover a web element in the web page.
8. wait : s.wait(<path to image>); – helps to wait for a web element on the page until it get displayed. There another method for wait is with timeout value, it looks like s.wait(<path to image>, <timeout in seconds>);
9. dragDrop : s.dragDrop(<source image path>, <target image path>); – helps to drag an object(source image object) from one location and drop that object over other object (target image object) at target location.

Click here to get sample Selenium – Sikuli POC project and explore more. If you have any difficulty to get the POC project, please feel free to comment here.