Category: Automation

Test Automation Code Review Factors

     A code review is a quality assurance activity to ensure that check-ins are reviewed by someone other than the author. Often practice this activity as it’s an excellent way to catch errors early in the process. The test code should be treated with the same care as feature code, and therefore it should undergo code reviews as well. Test code can be reviewed by other automation engineers or developers who are familiar with the project and codebase.

     In this article, we will discuss what exactly should we look for when reviewing automation test code? I would like to share eight specific factors here,

Does your test case verify what’s needed?

     When verifying something manually, there are a lot of hidden validations that are being made. So if anything was incorrect, we’d likely notice that our tests are not as good at this. In fact, they will only fail if the conditions that we explicitly specify are not met. During automation scripting, we usually do a minimum number of checkpoints from medium to high-level. We should add maximum low-level checkpoint at each step to get maximum test coverage, this will help you to increase the quality of your software. The test automation code review will help to identify the missing checkpoints.

Does the test case focus on one specific thing?

     Each test case should focus on a specific single thing. This may be a bit confusing than a bunch of things that should be asserted. However, all of those assertions work together to verify a single thing. If, however, the test case also verified the company’s logo or some other feature besides the actual automated one, that would be outside of the scope of this test. So that the automation code review helps to identify and differentiate the out-scope items implemented in the test scripts.

Can the test cases run independently?

     Each test should be independent, which means it should not rely on other tests at all. This makes it much easier to track down failures and their root causes, and will also enable the team to run the tests in parallel to speed up execution if needed. Sometimes the automation engineer falls into the trap while isolating the test cases, because they may be using related test runs as a setup for other tests. For example, there is a test to delete an item from the list, in this case, they need to run the cases to add an item to the list and after that, they will execute the delete that item from the list. In this case, we can recommend that the test create and delete whatever it needs, and if possible to do this outside of the GUI (like via API calls or database calls).

How is the test data managed for test case executions?

     The test cases that deal with test data can be the difference for different test suites. As each test case should be developed to run independently and all the test cases should be able to run in parallel at the exact same time, each test should be responsible for its own test data. When the test cases are running in parallel and have different expectations of the test data’s state, the test cases end up failing when there’s no real issue with the application. Recommended that the automation engineer can create whatever data is needed for the test within the test itself or can keep in external files and call those data into their specific test cases.

Whether any separation of concerns?

     You should treat your test code should be in the same care as feature code. That means that clean coding practices, such as separation of concerns, should be followed. The test method should only focus on putting the application in the desired state and verifying that state. The implementation of manipulating the application’s state should not be within the test itself. For example, you have a test case to submit an application let say your method submitApplication() should be added as a test step in the test class and the actual implementation of the submitApplication() should be in the other helper class. Also, the non-test methods should not make any assertions. Their responsibility is to manipulate the state of the application, and the test’s responsibility is to verify that state. Adding assertions within non-tests decreases the reusability of those methods.

Is there anything in the test that should be a utility for reusability?

     Separating concerns already address this in most cases, but double check that the test isn’t implementing something that can be reused by other tests. Sometimes this may be a common series of verification steps that certainly is the test’s responsibility, however, it will be duplicated across multiple tests. In these cases, it’s a good idea to recommend that the automation engineer can move this block of code into a utility method that can be reused by multiple tests.

Are the object locators reliable?

     In the case of Graphical User Interface and mobile tests, the automation engineer will need to use selectors to locate and interact with the web elements. The first thing to ensure is that these locators are not within the tests themselves, need to think of separation of concerns. Also, make sure the selectors can stand the test of time. For example, using selectors that depend on the DOM structure of the page (e.g. index) will only work until the structure of the page is changed. Encourage the automation engineer to use unique IDs to locate any elements they need to interact with, even if that means adding the ID to the element as part of this check-in or create a strong custom XPath to locate and interact with the elements.

Are they using stable wait strategy?

     Mark any hard-coded waits. Automated tests run faster than customers would actually interact with the product and this could lead to issues with the test, such as trying to interact with or verify something that is not yet in the desired state. To solve this, the code needs to wait until the application is in the expected state before proceeding. However, hard-coding a wait is not suitable. It leads to all kinds of problems such as lengthening the duration of execution, or still failing because it didn’t wait long enough. Instead of that, you can recommend that the automation engineer use conditional waits that will only pause execution for the least amount of time that is needed.

     Now you are ready to start the automation test code review. Test code review is a highly recommended process and it should be added to your automation projects life cycle. Test code review will improve the quality of your automation test scripts, that way it helps to improve your application’s quality and your customer will get a quality product that adds more value to their businesses. I hope you got a clear idea of different factors that need to be considered while test code review and try to apply them in your automation project life cycle.

make it perfect!

Toggle between iOS application during AUT

93e9e695edc2eed5986871ebc75cc63e

     According to one recent report, smartphone users use on average 9 apps per day, and 30 apps per month. Many apps today provide value by integrating with other aspects of the mobile phone experience, even if it’s just as simple as taking photos using the device’s camera. Testing of multi-app integrations with iOS has been challenging, because Apple’s automation model allowed automation only so long as the AUT was running. The second you switched to another app to automate another part of your user flow, the automation would die. Thankfully, with Appium’s XCUITest driver (available with iOS 9.3+), we’re no longer limited to this kind of sandbox.

     There are so many kinds of multi-app flows we could experiment with, but let’s imagine a scenario involving the camera. Let’s say your app allows the user to take photos, and modifies them in some way before finally saving them to the system-wide photo library (the Camera Roll). One key verification for testing your app will be ensuring that, after performing the necessary steps in your app, the photo is newly available in the Camera Roll album of the Photos app.

     This is all possible to encode into your Appium scripts using two handy commands: mobile:launchApp and mobile:activateApp. We know that the launchApp before, in the Appium. activateApp is just the same, only the app must already have been launched; activateApp merely brings it back to the foreground. Here are examples of how we’d use these commands, by incorporating the Bundle IDs of the apps we’re dealing with (having these Bundle IDs available is a pre-requisite),

// launch the photos app (with the special bundle id seen below)
HashMap<String, Object> args = new HashMap<>();
args.put(“bundleId”, “com.apple.mobileslideshow”);
driver.executeScript(“mobile: launchApp“, args);

// re-activate that AUT (in this case The App)
args.put(“bundleId”, “io.cloudgrey.the-app”);
driver.executeScript(“mobile: activateApp“, args);

      I believe this is so simple with Appium we can really pretend we are a user and do all the things a user would do in order to check the appropriate conditions. And chances are, you have a different requirement than verifying photo saving. So the thing to take away is the principle of using mobile: launchApp and mobile: activateApp. What’s nice is that this particular strategy will work not only for simulators but also for real devices.

Try to use this logic of switch applications during your iOS Appium automation.

Reference: Appium Pro

make it perfect!

File download with headless automation

1_aoUGQKFGoplkAYSbPv8i5g

      In selenium headless automation, we had a scenario to download a file and verify the file content. We tried with chrome browser as headless for execution but failed to download the file into a respective path mentioned in the ChromeOptions via setExperimentalOption. 

     Finally, we derived a solution to download the file into respective path when the application under headless execution. We used to send the Page.setDownloadBehavior value as a request. This request needs to execute after the headless driver instantiation. Following is the sample code snippet which helped us to solve the file download issue under headless automation execution,

String dowloadPath = “directory path to your file to download”;

System.setProperty(“webdriver.chrome.driver”, “path to chromedriver.exe”);

ChromeOptions options = new ChromeOptions();
options.addArguments(“–test-type”);
options.addArguments(“–headless”);
options.addArguments(“–disable-extensions”);
ChromeDriverService driverService = ChromeDriverService.createDefaultService();
ChromeDriver driver = new ChromeDriver(driverService, options);

Map<String, Object> commandParams = new HashMap<>();
commandParams.put(“cmd”, “Page.setDownloadBehavior”);
Map<String, String> params = new HashMap<>();
params.put(“behavior”, “allow”);
params.put(“downloadPath”, dowloadPath);
commandParams.put(“params”, params);
ObjectMapper objectMapper = new ObjectMapper();
HttpClient httpClient = HttpClientBuilder.create().build();
String command = objectMapper.writeValueAsString(commandParams);
String u = driverService.getUrl().toString() + “/session/” + driver.getSessionId() + “/chromium/send_command”;
HttpPost request = new HttpPost(u);
request.addHeader(“content-type”, “application/json”);
request.setEntity(new StringEntity(command));
httpClient.execute(request);

     Try to use the above code snippet if you are facing the file download issue during headless automation execution.

make it perfect!

Pick the Right Locator Strategy during Mobile Automation

    APITesting

   Here focus just on the selector strategies provided by Appium for native iOS and Android testing using the UiAutomator2 and XCUITest drivers.  Here’s prioritized list of locator strategies:

  1. accessibility id
  2. id
  3. XPath
  4. Class name
  5. Locators interpreted by the underlying automation frameworks, such as: -android uiautomator, -ios predicate string, -ios class chain
  6. -image

1. accessibility id
This is the top choice should surprise nobody. If you have the option of using accessibility IDs, use them. Normally an app developer needs to add these specifically to UI elements in the code. The major benefit of accessibility IDs over just the id locator strategy is that while app developers add these IDs for testing, users with handicaps or accessibility issues benefit. People who use screen readers or other devices, and algorithms which inspect UIs, can better navigate your application. On Android, this locator strategy uses the accessibility contentDescription property. On iOS, this locator strategy uses the accessibility identifier. Here’s something surprising: in the XCUITest driver, the accessibility id, id, and name locator strategies are all identical.

  They are implemented the same way. Go ahead and try switching your locator strategies, you will get the same results. This may change in the future, but for now you can find an element using the name or text in it because iOS has many ways in which it sets a default accessibility identifier if one is not provided by the developer.

2. id
Element IDs need to be added by a developer, but they allow us to pinpoint the exact element in the app we care about, even if the UI changes appearance. The drawback is you need to be able to talk to your developers. Many testing teams do not have this luxury.

   This locator strategy is pretty similar to accessibility id except that you don’t get the added benefit of accessibility. As noted above, on iOS, this is actually identical to accessibility id. On Android, the id locator strategy is implemented using Resource IDs. These are usually added to UI elements manually by app developers, though are not required, so many app developers will omit them if they don’t think they’re important.

3. XPath
Now this is contentious. XPath is the most expressive and commonly accepted locator strategy. Despite Appium developers warning against XPath’s low performance for years, it still seems to be the most popularly used locator strategy. This is probably because there are many selections that can’t easily be made any other way. For example, there’s no way to select the parent of an element using the simple id selectors. The benefit of being able to express more complicated queries must outweigh the cost to performance for all but the testers whose apps have such large XML element hierarchies that XPath is completely unusable.

     XPath selectors can be very brittle, but they can be responsibly wielded to great effect. Being intentional and carefully picking selectors rather than taking whatever an inspector provides can mitigate the brittleness.

    I think part of the popularity of XPath stems from its use with Selenium and web development, as well as it being the default of many tutorials and inspection tools. When working on Appium I always expected our XPath handling to break more often, but I remember few bugs, probably the benefit open source XPath libraries built for more generalized use.

    The Android OS provides a useful dumpWindowHierarchy which gives us an XML document of all the elements on the screen. From there we apply the XPath query and find elements.

    iOS does not supply a method of getting the entire hierarchy. Appium’s implementation starts at the root application element and recurses through each element’s children and populates and XML document which we can then apply the XPath query to. I still think XPath is unintuitive, especially for those new to programming, but at least it’s a well-accepted industry standard.

4. -android uiautomator, -ios predicate string or -ios class chain
These are the “native” locator strategies because they are provided by Appium as a means of creating selectors in the native automation frameworks supported by the device. These locator strategies have many fans, who love the fine-grained expression and great performance (equally or just slightly less performance than accessibility id or id).

    These locator strategies are crucial for those who have UIs which escape the grasp of the other locator strategies or have an element tree which is too large to allow the use of XPath. In my view, they have several drawbacks. These native locator strategies require a more detailed understanding of the underlying automation frameworks. Uiautomator and XCUITest can be hard to use, especially for those less familiar with Android and iOS specifics. These locator strategies are not cross platform, and knowing the ins-and-outs of both iOS and Android is challenging.

   In addition, the selectors passed to these native locator strategies are not directly evaluated by the mobile OS. Java, Kotlin, Objective C and Swift all lack an eval function which would allow interpreting a string of text as code. When you send an android uiautomator selector to Appium, the text passes through a very simplistic parser and uses Reflection to reconstruct the objects referenced in the text. Because of this, small mistakes in syntax can throw off the entire selector and only the most common methods are supported. This system is unreliable and often encounters difficult bugs.

    If Android or iOS change the testing classes in new updates, your selectors might need to be updated. Using XPath, Appium will keep track of the OS updates and your selectors should keep working.

    A personal quibble I have with these locator strategies is that you are essentially writing a different programming language (such as Java) inside of a string in your test code. Your text editor will not offer syntax highlighting or semantic analysis inside of these queries which makes them harder to maintain. On the other hand, sometimes there’s just no other way, and for those who are proficient in these methods XPath can seem clumsy in comparison.

5. -image
This locator strategy is pretty nifty. Supported on all platforms and drivers, you can pass an image file to Appium and it will try to locate the matching elements on the screen. This supports fuzzy matching, and certainly has its applications, but you should only use it when the other locator strategies aren’t working. It may not always behave deterministically, which mean it can be a source of test flakiness. Also, unless you have a special code editor, they may be a little more difficult to work with and maintain. Of course, visual UI changes will always break -image selectors, whereas the other selectors only break when the element hierarchy changes.

Reference: Appium Pro

make it perfect !

Automating Voice Commands With Siri

iphone-6s-siri

     It’s very common with modern mobile devices to rely on virtual “assistants” to get tasks done, whether in a hands-free situation utilizing voice commands, or just to save the trouble of tapping out search queries. On iOS these interactions take place through the Siri interface.

Hey Siri

     How on earth would you test this aspect of your app’s behavior? Ideally you’d be able to have a recording of the particular voice command or phrase used to trigger your app’s Siri integration, which you could then somehow apply to the simulator or device under test. This is not currently possible, outside of rigging up some speakers!

     Fortunately, Appium has recently added a command (as of Appium 1.10.0), that lets you specify the text you want Siri to parse, as if it had been spoken by a person.

The command itself is accessible via the executeScript “mobile” interface:

HashMap<String, String> args = new HashMap<>();
args.put(“text”, “Hey Siri, what’s happening?”);
driver.executeScript(“mobile: siriCommand”, args);

     Essentially, we construct an options hash with our desired text string, and pass it to the siriCommand “mobile” method. We can run this command at any point in our automation, and it will take care of getting to the Siri prompt for us as well (we don’t need to long-hold the home button).

     At this point we can use the typical native automation methods to verify Siri’s response on the screen, tap on action items, etc…

     That’s basically it! There’s not much to it. So let’s have a look at a full example that asks Siri a math question (What’s two plus two?) and verifies the result (notice how the result text shows up as accessibility IDs, which found by looking at the page source).

@Test
public void testSiriTalk() {
HashMap<String, String> args = new HashMap<>();
args.put(“text”, “What’s two plus two?”);
driver.executeScript(“mobile: siriCommand“, args);
wait.until(ExpectedConditions.presenceOfElementLocated(MobileBy.AccessibilityId(“2 + 2 =”)));
wait.until(ExpectedConditions.presenceOfElementLocated(MobileBy.AccessibilityId(“4”)));}

     You have to use java-client version as 6.1.0 and Appium server version as 1.10.0. Try to use above sample test case to see what exactly automating voice command with Siri.

Reference: Appium Pro

make it perfect !