Automate iPadOS Split View Multitasking With Appium

      iPad Pros run a slightly different version of iOS called iPadOS, and this version of iOS comes with several really useful features. One of the favorite is the ability to run two apps side-by-side. This is called Split View Multitasking by Apple, and getting it going involves a fair bit of gestural control for the user. Here’s how a user would turn on Split View:

  1. Open the first app they want to work with
  2. Show the multitasking dock (using a slow short swipe up from the bottom edge)
  3. Touch, hold, and drag the icon of the second app they want to work with to the right edge of the screen

      From this point on, the two apps will be conjoined in Split View until the user drags the app separator all the way to the edge of the screen, turning Split View off. Of course, both apps must be designed to support split view for this to work. 

      Let’s now discuss how we can walk through this same series of steps with Appium to get ourselves into Split View mode, and further be able to automate whichever app of the two we desire. Unfortunately, there’s no single command to make this happen, and we have to use a lot of tricky techniques to mirror the appropriate user behavior. Basically, we need to worry about these things:

  1. Ensuring both apps have been opened recently enough to show up in the dock
  2. Executing the correct gestures to show the dock and drag the app icon to trigger Split View
  3. Telling Appium which of the apps in the Split View we want to work with at any given moment

We’re going to describe how to achieve above steps.

Ensuring applications are in the dock

      For our strategy to work, we need the icon of the app we want to open in Split View in the dock. The best way to make this happen is to ensure that it has been launched recently in fact, most recently apart from the currently-running app. Let’s take a look at the setup for an example where we’ll load up both Reminders and Photos in Split View. In our case, we’ll want Reminders on the left and Photos on the right. Because we’re going to open up Photos on the right, we’ll actually launch it first in our test, so that we can close it down, open up Reminders, and then open up Photos as the second app.

DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(“platformName”, “iOS”);
capabilities.setCapability(“platformVersion”, “13.3”);
capabilities.setCapability(“deviceName”, “iPad Pro (12.9-inch) (3rd generation)”);
capabilities.setCapability(“app”, PHOTOS);
capabilities.setCapability(“simulatorTracePointer”, true);
driver = new IOSDriver(new URL(” http://localhost:4723/wd/hub “), capabilities);
wait = new WebDriverWait(driver, 10);
size = driver.manage().window().getSize();

      In this setUp method, we also construct a WebDriverWait, and store the screen dimensions on a member field, because we’ll end up using them frequently. When we begin our test, the Photos app will be open. What we want to do next is actually terminate Photos, and launch Reminders. At this point, we’ve launched both the apps we want to work with, so they are both the most recently-launched apps, and will both show up in the recent apps section of the dock. Then, we go back to the Home Screen, so that the dock is visible:

// terminate photos and launch reminders to make sure they’re both the most recently launched apps
driver.executeScript(“mobile: terminateApp”, ImmutableMap.of(“bundleId”, PHOTOS));
driver.executeScript(“mobile: launchApp”, ImmutableMap.of(“bundleId”, REMINDERS));

// go to the home screen so we have access to the dock icons
ImmutableMap pressHome = ImmutableMap.of(“name”, “home”);
driver.executeScript(“mobile: pressButton”, pressHome);

      In the next step of this flow, we figure out where the Photos icon is, and save that information for later. Then we re-launch Reminders, so that it is active and ready to share the screen with Photos.

// save the location of the icons in the dock so we know where they are //when we need to drag them later, but no longer have access to them as //elements
Rectangle photosIconRect = getDockIconRect(“Photos”);

// relaunch reminders
driver.executeScript(“mobile: launchApp”, ImmutableMap.of(“bundleId”, REMINDERS));

      There is an interesting helper method here. getDockIconRect just takes an app name, and returns the position of its dock icon in the screen:

protected Rectangle getDockIconRect(String appName) {
By iconLocator = By.xpath(“//[@name=’Multitasking Dock’]//[@name='” + appName + “‘]”);
WebElement icon = wait.until(
ExpectedConditions.presenceOfElementLocated(iconLocator));
return icon.getRect();
}

      Here we use an xpath query to ensure that the element we retrieve is actually the dock icon and not the home screen icon. Then, we return the screen rectangle representing that element, so that we can use it later.

Showing the dock and entering into Split View

      At this point we are ready to call a special helper method designed to slowly drag the dock up in preparation for running the Split View gesture:

// pull the dock up so we can see the recent icons, and give it time to settle
showDock();
Thread.sleep(1000);

protected void showDock() {
swipe(0.5, 1.0, 0.5, 0.92, Duration.ofMillis(1000));
}

      We are using showDock method to perform a slow swipe from the middle bottom of the screen, up just far enough to show the dock. Now that the dock is shown, we can actually enter Split View. To do that, we make use of a special iOS-specific method mobile: dragFromToForDuration, which enables us to perform a touch-and-hold on the location of the Photos dock icon, then drag it to the right side of the screen. We wrap this up in a helper method called dragElement. Below is the implementation:

// now we can drag the photos app icon over to the right edge to enter split view, also give it a bit of time to settle
dragElement(photosIconRect, 1.0, 0.5, Duration.ofMillis(1500));
Thread.sleep(1000);

protected void dragElement(Rectangle elRect, double endXPct, double endYPct, Duration duration) {
Point start = new Point((int)(elRect.x + elRect.width / 2), (int)(elRect.y + elRect.height / 2));
Point end = new Point((int)(size.width * endXPct), (int)(size.height * endYPct));
driver.executeScript(“mobile: dragFromToForDuration”, ImmutableMap.of(“fromX”, start.x, “fromY”, start.y, “toX”, end.x, “toY”, end.y, “duration”, duration.toMillis() / 1000.0));}

      Essentially, we take the rect of a dock icon, pass in the ending x and y coordinate percentages, and the duration of the “hold” portion of the gesture. The dragElement helper converts these to the appropriate coordinates, and calls the mobile: method.

Working with simultaneously open applications

      At this stage in our flow, we’ve got both apps open in Split View! But if we take a look at the page source, we’ll find that we only see the elements for one of the apps. And in fact, we can only work with one app’s elements at a time. We can, however, tell Appium which app we want to work with, by updating the defaultActiveApplication setting to the bundle ID of whichever app you want to work with:

driver.setSetting(“defaultActiveApplication”, PHOTOS);
wait.until(ExpectedConditions.presenceOfElementLocated(MobileBy.AccessibilityId(“All Photos”)));
driver.setSetting(“defaultActiveApplication”, REMINDERS);
wait.until(ExpectedConditions.presenceOfElementLocated(MobileBy.AccessibilityId(“New Reminder”)));

      In the code above, you can see how we call driver.setSetting, with the appropriate setting name and bundle ID. After doing this for a given app, we can find elements within that app, and of course we can switch to any other app if we want as well.

      So that’s the way we can enter into a Split View and automate each application on the screen. Try to utilize above capabilities in your iPadOS automation.

Reference: Appium Pro

make it perfect!