Adam MilliganAdam Milligan
 iPhone UI Automation tests: a decent start
edit Posted by Adam Milligan on Tuesday July 20, 2010 at 12:24AM

Apple's inclusion of the UI Automation component in Instruments with iOS 4 is a definite step in the right direction. It's the first reasonable way to write tests that externally exercise your actual app, rather than weirdly injecting test code into it. It's also the only way to programmatically test lifecycle issues, such as how your app behaves when put in the background, when rotated, when the device locks, etc. Good stuff. Unfortunately, the current implementation of UI Automation also has some significant problems:

  1. There's no way to run tests from the command line. The subtitle of the WWDC talk for UI Automation was "find bugs while you sleep;" unfortunately, you can't find bugs while you sleep if you have to wake up to click the "Run" button.
  2. There's no way to set up or reset state. The lack of fixtures which set up a known state at the beginning of iPhone tests has been a problem for unit testing (with OCUnit, Cedar, or what have you), particularly for apps that use CoreData. Now it's worse than ever, because UI Automation manipulates the actual state of the app on the device, much like Selenium does in a browser. Sadly, UI Automation provides no method for reseting the device's state, making it nigh impossible to prevent tests from affecting one another.
  3. Part of the previous problem is that UI Automation has no concept of discrete tests; it provides no form of organization for your test scripts. No test methods, no set up or tear down methods, just one big stream of consciousness line of execution. Obviously you can break this up into functions as you see fit, but why reinvent the wheel? Since the test script is JavaScript, I like the idea of using Jasmine for this.
  4. There's no way to programmatically retrieve the results of the test run. You could debate the value of solving this issue at the moment, considering there's no way to programmatically start the tests either. However, even if you were to write some clever AppleScript to kick off the tests automatically the only indication of the pass/fail status is in the Instruments UI, so you still have to wake up to check the results. I searched around a bit for information on deconstructing the protocol that UI Automation uses to talk to the device, but I came up empty.

I'll definitely use UI Automation, particularly for app lifecycle testing. But, not being able to add those tests to a CI build definitely stings. I very much hope Apple keeps their momentum for automated testing and makes it more developer-friendly.

Comments

  1. Johan Johan on July 20, 2010 at 04:42AM

    Can't say much else other than I agree. UIAutomation seems like a moving target at the moment (good), but slightly slanted towards implementing repeatable performance tests (bad, if what you really want to do is integration-style testing)

    Have you submitted this to the radar (bugreport.apple.com)? If it's not in there it's generally not on Apple's, uhm, radar...

  2. Nige Nige on July 20, 2010 at 09:15AM

    I've been looking for a way to run my UIAutomation from the instruments command line so I can integrate it in with my Hudson build. So far no luck. I agree UIAutomation feels very 1.0 beta.

  3. Ryan Ausanka-Crues Ryan Ausanka-Crues on July 20, 2010 at 11:22PM

    Additional problems:

    1. Calls are very slow to execute. In my testing on a MacBookPro in iPhone Simulator it took ~5 seconds to execute most of the methods on UIAElement.
    2. Spartan documentation and examples. How do you assert a table row's accessoryType?
    3. Caching issues. If you are on a tableView and get the cells then switch to another tableView and request groups you get the groups of the new tableView but if you request cells, you get the old tableView cells. Since there no access to JavaScript source it's hard to see what's actually happening and why.
    4. UIAElement.logElement() is not very helpful and, since there is no support for breakpoints, debugging your script can be a pain in the ass.
    5. Doesn't automatically terminate unless the app crashes. When all of the statements in your script have been executed you still have to manually stop Instrument execution.
    6. No way to get the cells within a table group.

    I really like the idea and am happy that it's in JS instead of Objective-C but the implementation is more infuriating than it is helpful. I hope it grows into its potential.

  4. Adam Milligan Adam Milligan on July 21, 2010 at 12:06AM

    Interesting. Ryan, you've clearly played around with UI Automation far more than I have. Thanks for the detail.

  5. Vijay Ganesh Vijay Ganesh on September 14, 2010 at 08:47PM

    Hi Adam,

        I am currently working on iPhone automation testing. I wanted to know whether UI Automation can be used to automated the iPhone browser and test a particular website ?
  6. Adam Milligan Adam Milligan on September 14, 2010 at 09:46PM

    Vijay, I don't claim to be a UIAutomation expert just yet, but I don't know of any way to test the content of a web view directly. I believe you can send touch events to particular locations on the device, but you'd have to know the screen coordinates of the elements on the web page you'd like to touch.

    Perhaps someone else will correct me and chime in with some better ideas.

  7. Vijay Ganesh Vijay Ganesh on September 22, 2010 at 09:16PM

    Adam, you said that you have tried writing Applescript to automate UI Automation tool, i tried that too, but could go further except for opening “Instruments” . Is it possible to choose app to test, and test script and press “Start” button ? if yes, could plaza share the script ?

  8. Vijay Ganesh Vijay Ganesh on September 22, 2010 at 11:21PM

    Adam, you said that you have tried writing Applescript to automate UI Automation tool, i tried that too, but couldn't go further except for opening “Instruments” application. Is it possible to automate choosing app to test and test script, then press “Start” button ? if yes, could plz share the script ?

    *excuse my typo in the previous comment

  9. Adam Milligan Adam Milligan on September 23, 2010 at 07:21PM

    Vijay, my comment about using AppleScript to start a UIAutomation test run was purely speculation; I haven't actually tried it. If you get it working please post again and let us know how to go about it.

  10. Sean Woodward Sean Woodward on October 01, 2010 at 03:28PM

    I am rather new to the iOS development environment, and am becoming familiar with the Automation instrument. I appreciate Adam's point of view and suggest that while not as robust as we might like, the Automation instrument is much better than manual testing.

    With regards to Adam's orignal comments:

    1. No way to set up state. Perhaps a combination of duplicated targets or test targets and the Automation instrument could provide the setup you require. Theoretically, one could use specific Core Data sources (SQLite files or XML files for easy viewing) associated with a dedicated target to handle the state for a particular test. Not very sexy, but I think it might work. Worst case you could use a custom class to manage the state.

    2. No concept of discrete tests, and no way to organize scripts You can use the #import statement in the Automation JavaScript files to allow for some code reuse and organization.

    With regards to Ryan's comments:

    1. Caching issues In my tests, I have found I need to implement a delay to allow the application to "catch up" to the script. I have noted variance in element counts with and without delay. A work-around is to use labeled UI elements and access them by name which forces a delay. However, if the element you update is the named element, I think you can only rely on using a delay.

    2. No Automatic termination of the script This seems to be the silliest absence of a feature, but an easy way to terminate the script is to throw an exception. Just put 'throw "end of script";' at the end of your script. Viola, the Automation Instruments session will end quite nicely.

    Thank you to everyone for their contributions. I certainly hope to see more attention paid to UI Automation from the Apple developers.

  11. Nghiem_huong Nghiem_huong on October 13, 2010 at 08:20PM

    Hello! I am using UI Automation to testing iPhone app. I want to test that next screen is true as follow of app. But I don't know how to use? Ex: from Top Screen, when you click to Button "Help", app will go to Help Screen. But app don't work that same.