Find Element By Image Locator Strategy in Appium

location-share-top

      Picking the right locator strategy during mobile automation helps to stabilize the automation flow. One of my previous articles, I have already mentioned the selection of the right locator strategy. There I have mentioned about -image selector.

    The -image locator strategy is pretty nifty. Supported on all platforms and drivers, you can pass an image file to Appium and it will try to locate the matching elements on the screen. This supports fuzzy matching and certainly has its applications, but you should only use it when the other locator strategies aren’t working. It may not always behave deterministically, which means it can be a source of test flakiness. Also, unless you have a special code editor, they may be a little more difficult to work with and maintain. Of course, visual UI changes will always break -image selectors, whereas the other selectors only break when the element hierarchy changes. In this article, I would like to explain about find elements By Image. 

     The Appium team finally decided to bite the bullet and support a small set of visual detection features, which are available as of Appium 1.9.0 version and above. An image element looks to your client code exactly like any other element, except that you’ve found it via a new -image locator strategy. Instead of a typical selector (like “foo”), the strings used with this new locator strategy are Base64-encoded image files. The image file used for a particular find action represents a template image that will be used by Appium to match against regions of the screen in order to find the most likely occurrence of the element you’re looking for.

     Suppose you have a scenario like, click on a particular image from your gallery and validate it.  None of the images have any identifying information in the UI tree, and their order changes every time we load the view, so we can’t hardcode an element index if we want to tap a particular image. Find-by-image to the rescue. Actually using this strategy is the same as finding an element using any other strategy:

WebElement element = driver.findElementByImage(base64EncodedImageFile); element.click()

     Of course, for this to work we have to have a Base64-encoded version of your image file. In Java 8 this is pretty straightforward:

// consider that you have a File called myImageFile.You can fetch image path //refImgUrl using ClassLoader class
File myImageFile = Paths.get(refImgUrl.toURI()).toFile();
Base64.getEncoder().encodeToString(Files.readAllBytes(myImageFile.toPath()));

     One another great thing is that finding elements by image supports both implicit and explicit wait strategies, so your tests can robustly wait until your reference image matches something on the screen:

By image = MobileBy.image(base64EncodedImageFile);
new WebDriverWait(driver, 10)
.until(ExpectedConditions.presenceOfElementLocated(image)).click();

     By using the above simple code snippet you can easily navigate to the gallery, click on your particular image, and validate it. Try to use the above find element by image locator strategy in your Appium automation script and enjoy automation.

Reference: Appium Pro

make it perfect!

Find Off-Screen Elements In Android Using Appium

getting_started_appium_-android

     In this article, I would like to share how to find off-screen elements in Android using Appium and also how ‘Android Data Matcher‘ Locator Strategy works better. One of the known limitations of finding elements on Android is that the Android UI renderer doesn’t actually build and render elements until they are about to appear on the screen. If we are looking for an element that is far down in a list, and not yet visible on the screen, the element just does not “exist”. You can prove this to yourself by generating a source snapshot of your app while there are lots of list elements present below the edge of the screen.

     How do we handle a situation like this with Appium? Usually, say is to “think like a user”–how would a user find this item? By swiping to scroll the list and using their eyes to periodically check for the appropriate content. We could, therefore, implement a scroll-and-find helper method with built-in retry loops. If we’re using the Espresso driver, there’s a better way: the ‘Android Data Matcher‘ locator strategy.

      This strategy takes advantage of the fact that Espresso runs in the app context, not on top of the Accessibility layer, so it has access to the data which backs a list, not just the elements which are displayed on the screen. We can use a part of the Espresso API to essentially target this data, and have Espresso automatically scroll to the element which represents that data for us. If we were using Espresso directly (not through Appium), we might accomplish this with the following code:

onData(hasEntry(“title”, “textClock”)
.inAdapterView(withId(“android:id/list))
.perform(click());

      This Espresso code waits for some “data” according to the parameters we’ve given the onData command, namely that we want an entry that has a specific title, within the list that has a certain Android ID. The important bit here is the hasEntry function, which builds a hamcrest matcher. The matcher is used by Espresso to help find the desired data and turn it into a view we can interact with. All kinds of matcher explore and use with Expresso are available now in Appium. Basically, in Appium we need to use -android datamatcher locator strategy, in Java is accessed as MobileBy.androidDataMatcher. And for the selector, rather than a simple string, we construct a JSON object containing the Matcher information, stringify it, and pass it as the selector. Below we can how normally use in automation scripting,

// first, find the AdapterView (not necessary if there’s only one)
WebElement list = wait.until(ExpectedConditions.presenceOfElementLocated(
MobileBy.className(“android.widget.ListView”)
));

// Construct the Hamcrest Matcher spec
String selector = new Json().toJson(ImmutableMap.of(
“name”, “hasEntry”,
“args”, ImmutableList.of(“title”, “Layout”)
));

// find and click on the element using the androidDataMatcher selector
list.findElement(MobileBy.androidDataMatcher(selector)).click();

     In the above example, it will look for the title “Layout” in the list and perform the click action. When you see this work, it’s pretty awesome, the screen will jump very quickly to the expected item in the list view, without having to do a bunch of manual scrolling.

Reference: Appium Pro

make it perfect!

Pick the Right Locator Strategy during Mobile Automation

    APITesting

   Here focus just on the selector strategies provided by Appium for native iOS and Android testing using the UiAutomator2 and XCUITest drivers.  Here’s prioritized list of locator strategies:

  1. accessibility id
  2. id
  3. XPath
  4. Class name
  5. Locators interpreted by the underlying automation frameworks, such as: -android uiautomator, -ios predicate string, -ios class chain
  6. -image

1. accessibility id
This is the top choice should surprise nobody. If you have the option of using accessibility IDs, use them. Normally an app developer needs to add these specifically to UI elements in the code. The major benefit of accessibility IDs over just the id locator strategy is that while app developers add these IDs for testing, users with handicaps or accessibility issues benefit. People who use screen readers or other devices, and algorithms which inspect UIs, can better navigate your application. On Android, this locator strategy uses the accessibility contentDescription property. On iOS, this locator strategy uses the accessibility identifier. Here’s something surprising: in the XCUITest driver, the accessibility idid, and name locator strategies are all identical.

  They are implemented the same way. Go ahead and try switching your locator strategies, you will get the same results. This may change in the future, but for now you can find an element using the name or text in it because iOS has many ways in which it sets a default accessibility identifier if one is not provided by the developer.

2. id
Element IDs need to be added by a developer, but they allow us to pinpoint the exact element in the app we care about, even if the UI changes appearance. The drawback is you need to be able to talk to your developers. Many testing teams do not have this luxury.

   This locator strategy is pretty similar to accessibility id except that you don’t get the added benefit of accessibility. As noted above, on iOS, this is actually identical to accessibility id. On Android, the id locator strategy is implemented using Resource IDs. These are usually added to UI elements manually by app developers, though are not required, so many app developers will omit them if they don’t think they’re important.

3. XPath
Now this is contentious. XPath is the most expressive and commonly accepted locator strategy. Despite Appium developers warning against XPath’s low performance for years, it still seems to be the most popularly used locator strategy. This is probably because there are many selections that can’t easily be made any other way. For example, there’s no way to select the parent of an element using the simple id selectors. The benefit of being able to express more complicated queries must outweigh the cost to performance for all but the testers whose apps have such large XML element hierarchies that XPath is completely unusable.

     XPath selectors can be very brittle, but they can be responsibly wielded to great effect. Being intentional and carefully picking selectors rather than taking whatever an inspector provides can mitigate the brittleness.

    I think part of the popularity of XPath stems from its use with Selenium and web development, as well as it being the default of many tutorials and inspection tools. When working on Appium I always expected our XPath handling to break more often, but I remember few bugs, probably the benefit open source XPath libraries built for more generalized use.

    The Android OS provides a useful dumpWindowHierarchy which gives us an XML document of all the elements on the screen. From there we apply the XPath query and find elements.

    iOS does not supply a method of getting the entire hierarchy. Appium’s implementation starts at the root application element and recurses through each element’s children and populates and XML document which we can then apply the XPath query to. I still think XPath is unintuitive, especially for those new to programming, but at least it’s a well-accepted industry standard.

4. -android uiautomator, -ios predicate string or -ios class chain
These are the “native” locator strategies because they are provided by Appium as a means of creating selectors in the native automation frameworks supported by the device. These locator strategies have many fans, who love the fine-grained expression and great performance (equally or just slightly less performance than accessibility id or id).

    These locator strategies are crucial for those who have UIs which escape the grasp of the other locator strategies or have an element tree which is too large to allow the use of XPath. In my view, they have several drawbacks. These native locator strategies require a more detailed understanding of the underlying automation frameworks. Uiautomator and XCUITest can be hard to use, especially for those less familiar with Android and iOS specifics. These locator strategies are not cross platform, and knowing the ins-and-outs of both iOS and Android is challenging.

   In addition, the selectors passed to these native locator strategies are not directly evaluated by the mobile OS. Java, Kotlin, Objective C and Swift all lack an eval function which would allow interpreting a string of text as code. When you send an android uiautomator selector to Appium, the text passes through a very simplistic parser and uses Reflection to reconstruct the objects referenced in the text. Because of this, small mistakes in syntax can throw off the entire selector and only the most common methods are supported. This system is unreliable and often encounters difficult bugs.

    If Android or iOS change the testing classes in new updates, your selectors might need to be updated. Using XPath, Appium will keep track of the OS updates and your selectors should keep working.

    A personal quibble I have with these locator strategies is that you are essentially writing a different programming language (such as Java) inside of a string in your test code. Your text editor will not offer syntax highlighting or semantic analysis inside of these queries which makes them harder to maintain. On the other hand, sometimes there’s just no other way, and for those who are proficient in these methods XPath can seem clumsy in comparison.

5. -image
This locator strategy is pretty nifty. Supported on all platforms and drivers, you can pass an image file to Appium and it will try to locate the matching elements on the screen. This supports fuzzy matching, and certainly has its applications, but you should only use it when the other locator strategies aren’t working. It may not always behave deterministically, which mean it can be a source of test flakiness. Also, unless you have a special code editor, they may be a little more difficult to work with and maintain. Of course, visual UI changes will always break -image selectors, whereas the other selectors only break when the element hierarchy changes.

Reference: Appium Pro

make it perfect !