Testing in the mobile world

Testing in the mobile world is really no different from testing on any other application or platform and all the usual types of testing need to be performed. However the mobile world does offer some additional challenges that you must consider when defining your mobile testing strategy and the testing that needs to be performed.

• Device and Operating System – There are a vast number of combinations of devices, form factors, operating systems and it’s simply impossible to test all of them. As such there are a number of elements you need to consider to ensure your test strategy is providing you with the necessary level of test coverage. These include:
  • The range of devices / operating systems to be tested to ensure optimum coverage in the geographical regions in which the applications will be available. Detailed information on device and operating system use across geographies is available and should be consulted as part identifying proposed test coverage. Examples of available coverage information include:
    • Device share by Android version (Android Developer Website)
    • Device share by IOS version and device type (David Smith Website)
    • Mobile operating system share (Net Market Share)
    • Perfecto Mobile Quarterly Device Coverage Report

• How, when and what mix of devices are tested to ensure required device coverage. This includes the initial use of device emulators for basic functionality, actual devices to test real user conditions, local or internet cloud platforms to provide necessary device and operating system coverage followed by beta or user testing provide necessary usability feedback. A number of local (e.g. Experitest and Mobile Labs platforms) and internet (e.g. Perfecto Mobile, Keynote, Xamarin, Sauce labs and Soasta cloud platforms) solutions exist which allow both manual or automated tests to be run using a wide range of devices, user scenarios and network conditions.
• Where possible the implementation and use of test automation should be included as part of your test strategy to allow required device and operating system coverage to be achieved. Most of the device cloud platforms identified above also offer test automation solutions. Examples of the range of available automation tools and frameworks (not an exhaustive list) include mobile tools provided by test automation vendors such as Borland, HP, Tricentis and Ranorex along with mobile specific solutions and frameworks such as the Appium Mobile Framework, Selenium for Android, Xamarin Calabash and Robotium.
• Pace of change – Rapid changes in and updates to devices, operating systems and applications occur in the mobile world. Ensuring that the required testing is performed for each of the updates requires that in addition to a strategy to cover all required devices and operating systems being in place that a continuous integration and testing programme is considered as part of test strategy to meet the demand placed by this pace of change.
• Network coverage and conditions – Tests must be in place to ensure applications function correctly across all required networks, are able to deal with switches between networks (e.g. Wi-Fi, 3G, and 4G) and also deal with varying network conditions including signal levels and signal loss. The local testing across possible networks (400+ worldwide) with varying network conditions is simply not a realistic proposition and as such consideration must be given to the use Network Simulation and Virtualisation tools. These can be standalone, e.g. the HP Network Visualisation Tool network simulation functionality provided by cloud based tools, e.g. the Perfecto Mobile cloud platform.
• Mobile devices have a number of physical characteristics which you must take into account when defining your tests to ensure your application can deal with these conditions. These include:
  • Screen size / resolution: Reduced screen size and resolutions, variety of screen sizes and resolutions, switching between display orientations.
  • Memory: Reduced memory levels, memory full.
  • Battery levels: Reduced power / battery levels, power loss, switching between batteries and charging.
  • Device interrupts: Messages, calls, alarms and notifications appearing during application use.
  • Cross Platform: Users switching between device platforms (PC, laptop, tablet and mobile).
• Application types: Developed applications may include native (i.e. device only), web (i.e. browser based) or hybrid versions. All of these versions must be tested.
• Application store rules and guidelines: Tests must be in place to ensure applications meet the rules and guidelines of the application stores through which the applications will be delivered. Current examples of guidelines include the Apple Store Guidelinesand Android Developer Checklists.