In this article, you can find the information about special tools that could make your mobile applications testing easier and more effective. [sociallocker]
Nowadays there is a trend of mobile applications rapid growth. There are millions of apps designed for different kinds of platforms that need to be tested. OS platforms variety is not the only challenge tester has to face to. The others are:
- different mobile app types
- multitude mobile devices
- device fragmentation
- test interfaces variety.
The most popular tools, as well as special needs tools, will be considered below
So, let’s start.
The most often used and multipurpose Tools
Appium supposed to be the most well-known tool for mobile applications testing. The special Appium philosophy principles describe this app in the best way:
- The user shouldn’t have to recompile your app or modify it in any way in order to automate it.
- The user shouldn’t be locked into a specific language or framework to write and run your tests.
- A mobile automation framework shouldn’t reinvent the wheel when it comes to automation APIs.
- A mobile automation framework should be open source, in spirit and practice as well as in name!
Using these 4 principles gives the main merits for Appium.
- Appium is a freely distributed open source mobile application testing framework.
- Appium has a framework or wrapper that translate Selenium Webdriver commands into UIAutomation (iOS) or UIAutomator (Android) commands depending on the device type, not any OS type. So, there is no dependency on Mobile device OS.
- Appium supports automation of hybrid, native and mobile web applications on physical devices as well as on an emulator or simulator both.
- Appium supports multiple platforms: iOS, Android, Windows, FirefoxOS
Of course, Appium is the non-ideal tool. So it has some disadvantages too:
- Appium directly supports Android version 17 and later. The elder versions are not supported.
- There is no direct Android alert handling support.
- There is no ability for direct image location. The user had to work with screen coordinates to do it.
- Over 50 open bugs with iOS
Selendroid is one of the most powerful test automation frameworks for the multi-type of mobile applications: native and hybrid Android apps and mobile web. Here are its main features:
- As Selendroid tests are written on the Selenium Web driver client API, so it supports full integration with current Selenium frameworks.
- It can be used on emulators as well on real devices simultaneously. So you have a great tool for the compatibility testing.
- Existing emulators have started automatically.
- Special test case development tool – Inspector for checking the current state of your app’s UI.
- No modification of app under test required in order to automate it.
- Selendroid supports old versions of Android starting from the 10-th.
- The user has an ability to change the hardware devices (plug and unplug) during the testing without restarting or stopping the test.
- Full integration as a node into Selenium Grid for scaling and parallel testing
- Full compatibility with the JSON Wire Protocol/Selenium 3 Ready.
- Click actions recording.
Some disadvantages of Selendroid:
- Selendroid directly supports only Android. Though, it’s fair to mention, that you can use ios-driver for testing any iOS native, hybrid, or mobile web application.
- The user has no ability to automate activities outside of app like camera, maps etc.
- Selendroid is quite slow. It is hard to be used on some machines with less than 4GB RAM.
Calabash is an automated acceptance testing framework maintained by Xamarin. It enables automated UI Acceptance Tests written in Cucumber to be run on iOS and Android applications. Here are the main advantages of Calabash:
- The user can execute Calabash tests on over 1,000 real mobile devices in the Xamarin Test Cloud is a cloud-based mobile testing lab.
- Clear and simple documentation about how the application should work.
- Support of Cucumber for Behavior-Driven Development.
- Parallel Test Case execution.
- Mobile functions like Gesture, swipe are extensively supported.
- Swipe can be automated
Calabash also has some demerits:
- Test Cases are written only in Ruby.
- Special target is Need to prepare to build an iOS application.
- No record and play options.
- No code generation by the tool option.
Robotium is one of the leading test automation framework for Android development. Let’s consider its main features:
- Test case developers can write function, system and acceptance test scenarios, spanning multiple Android activities with the Robotium support.
- It can be used both for testing applications where the source code is available and applications where only the .apk file is available and the implementation details are not known.
- Robotium has support for Android features such as activities, toasts, menus and context menus.
- Robotium is easy to write.
- Fast Test Case execution.
- Integrates smoothly with Maven or
- Automatic timing and delays.
Weak points of Robotium:
- It handles only one application at a time.
- Robotium cannot handle Flash or Web components
Ranorex is an easy-to-use test automation software for developing and managing projects in teams made up of both testers and developers.
The main advantages of Ranorex using:
- Easy installation for non-programmers.
- “Click & Go” XPath editor.
- UI object map editor.
- Support for Dynamic IDs.
- Easy maintenance.
- Easy execution of tests with executable (.exe) files.
- Professional Support Services.
Ranorex has some limitations:
- It does not support MAC and testing of Android web-based and hybrid applications.
- There is no option to export the test-script to different languages as Java, VBScript etc.
- Some Touch gestures/Drag and drop operations not supported.
- Automatic launching of the closed application is not supported in case of crashing/hanging.
- Need to provide long waiting times for the object identification
The eggPlant range of test automation tools comprises a variety of tools to satisfy every testing need – from functional to performance, mobile to desktop, digital to legacy. eggPlant tools can work on their own, with test tools from other vendors, or together in a unified environment.
eggPlant key features:
- Easy integration with quality management software.
- Guided Record Mode.
- eggPlant Test Cases can be run unattended via the command line
- It supports all the main platforms: Android, iOS, BlackBerry, Windows Phone, and Symbian.
- Interacts with the system exactly the same way a user does.
- No plugins required.
- No waiting for an update to the tool to support a newly released OS or development language.
Here are some drawbacks of eggPlant:
- No native object identification.
- Images captured on one operating system cannot work on other OS.
- EggPlant is not very popular in the market. So it is very difficult to find the resource having programming skills to work on Eggplant automation when compared to other competitive tools like Selenium Webdriver.
KeepItFunctional(KIF) is an iOS integration test framework that allows you to implement Test Cases with objective C that can be executed currently only against the iPhone/ iPad simulator.
KIF has following advantages:
- User-like Test Automation.
- Various iOS Versions support.
- Automatic integration with Xcode Testing Tools.
- Easy to learn.
- Tests are easy to write, in the same language as the code (ObjC/Swift), using familiar concepts.
The main demerits of KIF are:
- It uses undocumented Apple APIs. If your test scripts are not part of the production code your app will be rejected by Apple due to the use of undocumented APIs.
- KIF is not consistent at points.
- Tests are written in Objective-C, which is already familiar to iOS developers, but not test teams.
UI Automator testing framework, provided by the Android Testing Support Library. It is a perfect tool for the automation testing outside of the application because this framework allows test system stuff, as for example notifications, access to any application, etc.
The key features of the UI Automator:
- UI Automator View.
- Interacting with any applications.
- Access to the device state.
- Developers can create tests using the API provided by the platform and run tests on it.
Its library comes with the Android SDK and hundreds of tutorials are available
UI Automator has some limitations:
- It requires Android 4.3 (API 18) or higher.
- Java is the only directly supported language.
- Lack of support for hybrid apps.
SOASTA TouchTest provides complete functional test automation for continuous multi-touch, gesture-based mobile applications.
The main merits of SOASTA TouchTest:
- Test Cases are quickly built, edited, executed and analyzed with TouchTest’s patented interface.
- Capture Multi-Touch Gestures on Mobile Devices with native, web and hybrid app tests on iOS and Android.
- Gradle It automatically downloads the necessary jar files, no separate download or build steps is required once the changes are made.
- End-to-end mobile performance testing.
- TouchTest is best for automating critical tests that will be reused a lot because each device needs to be automated separately.
Here are some of TouchTest weak points:
- Scalability precision.
- Timing accuracy.
- No hardware control capabilities.
- No an incoming call simulation option.
SeeTestAutomation is the special tool that allows running automated mobile app testing not only on iOS or Android but also on BlackBerry and Windows Phones on real devices as well on emulators.
There are following features of SeeTestAutomation:
- The same test runs on different devices (e.g. Android test runs on iOS)
- A variety of plug-ins for UFT (QTP), RFT, TestComplete, C#, MSTest/VisualStudio/TFS, Java, Perl, Python, Ruby.
- Increase coverage with full peripheral testing capabilities – GPS, Touch ID, camera, and audio.
- Continuous integration environment integration – HP Quality Center (ALM), Jenkins etc.
- Integrate with WebDriver (Selenium), HP UFT/QTP, JUnit, Microsoft Visual Studio.
Weak points of SeeTestAutomation:
- SeeTestAutomation is a fully commercial testing tool.
- It connects up only to 5 devices.
- SeeTest runs devices serially and requires a separate plug-in for parallel execution.
Performance Testing Tools
Performance Testing is provided to check the application behavior under certain conditions such as bad network coverage, low available memory, and simultaneous access to application’s server by several users, low battery, and other conditions.
Let’s consider some special Performance Testing Tools.
NeoLoad is one of the best tools available in the market for load-testing in terms of price, support, and compatibility with web 2.0.
The main key features and benefits of Neoload are:
- Its cross-platform capabilities allow the loads to be generated on a wide variety of platforms (Unix, Windows, Linux…).
- Supports, cloud integration, live monitoring and real device integration not only on Android or iOS but also on Windows Phone and Blackberry.
- NeoLoad obtains a summary of the test case execution and then examining its details using the graphs and statistical tables. Statistics is got from your servers (CPU, Memory, etc).
- Content validation. Page content is checked under the load.
- NeoLoad gives testers an ability to define Virtual User behavior and set scenario parameters such as load policy (constant, ramp-up, peak or custom) or the number of Virtual Users to be simulated in the test.
- Performance monitors (CPU, memory, disk usage, and so on) are set for your servers.
- It records HTTP traffic between browser and server.
- An ergonomic graphical interface that reduces the man hours required for testing.
Below you can see some drawbacks of NeoLoad:
- There is no facility to create start and end transactions to capture the response time
- NeoLoad does not work in Mac OSX
- No access page DOM
Apteligent: (formerly Crittercism) is a mobile application performance management system that offers both service monitoring and error monitoring solutions.
There are the following advantages of Apteligent:
- Both fatal and non-fatal errors
- Cloud Service API monitoring.
- Hybrid application support.
- Real-time crash monitoring.
- Provides detailed information about the crashes.
- Bug reporting is seamless.
Apteligent also has some disadvantages:
- App management is confusing.
- The user interface is sometimes slow.
New Relic focuses on performance, offering an end-to-end performance monitoring (analytics, A/B testing and crash reporting) for the web and mobile apps.
New Relic’s merits are:
- Easy to implement and easy to understand.
- Data includes sessions that crashed, the number of users impacted, resolved and unresolved crashes.
- Crashes and slowdowns are allocated by geographic parameters, type of crash, devices, operating systems, operator networks etc.
- Performance breakdowns across each wireless network carrier.
There are also some demerits of the New Relic:
- The web UI is a bit difficult to navigate.
- No Android Studio plug-in.
- No ability to get custom attributes on crash reports
Mobile App Distribution and Crash Reporting Tools for Developers
Application stability monitoring is the must have a thing for any project. No doubt, that keeping your crash count down to a minimum is vitally important. There is a variety of crash reporting tools at your disposal with which you can arm yourself.
Let’s consider some of them.
Crashlytics is one of the most popular tools available today. It was acquired by Twitter in 2013 and since that is offered under Twitter’s Fabric umbrella-SDK.
Key features and benefits:
- Easy setup.
- Real-time crash reporting.
- In-depth crash analysis.
- Exception tracking.
- Test life cycle monitoring.
- Beta app distribution.
- Some popular apps like Uber, Spotify and Fruit Ninja are using Crashlytics.
- Lack of service support. Staff communication and responses to emails are inconsistent.
- Crashlytics is no more than a crash reporter.
- No distribution management.
Firebase Crash Reporting is one of the latest features of Google Firebase offers a comprehensive set of tools and services to help kick start the development of new apps and to accelerate the growth of existing ones.
It has such advantages:
- Data includes device characteristics and circumstances, a stack trace, and more.
- Firebase Crash Reporting allows the tester to group an audience out of users that experienced crashes and use Firebase platform to send push notifications and configuration changes.
- Crashes are prioritized by frequency and impact.
- Firebase Crash Reporting collects crashes that occur while the device is offline.
- Native crashes are collected automatically.
And here are some drawbacks of Firebase Crash Reporting:
- No way of changing resolved crash status to “resolved”.
- There is no search option within the crashes.
- New crashes appear slowly in the system (app no less than 15 min.).
- Google Play services must be implemented, otherwise, the Firebase will not work.
Instabug is well-known automatic debugging crash reporting tool.
Its main features are:
- Ability to group all related crashes together and permitting easy analysis regarding their occurrences.
- One dashboard for crashes and bugs.
- Automated symbols upload script.
- Instabug is one of the most powerful tools for beta testing and reporting.
Instabug has following weak points:
- Limited flexibility on reports customization.
- Less of support through social media channels, like FB, Twitter etc.
Hockeyapp is another popular product for a beta testing of mobile apps. This Microsoft product allows to distribute beta versions on Windows phones, Android, and iOS, collect live crash reports, get feedback from testers, analyze the test coverage and recruit new testers.
Hockeyapp has such merits:
- Full manual control
- Easy integration with 3rd party bug tracking systems and development workflows systems
- Wide variety of search tools to help you find the root of an app crash
- It symbolizes the crash reports for easy crash analysis
- Quick to upload new versions, while maintaining access to the old versions
It also has demerits:
- Quite expensive (an initial free plan for 2 apps, and then between $10 to $500 per month)
- The setup is still not optimized for the client
- User management is intuitive at times
Cloud Based Mobile Testing Tools and Services
Cloud Based Mobile Testing Tools and Services give the ability to run devices on multiple systems or networks via the Internet where applications can be tested, updated and managed. Easy availability, high scalability, and low cost are the main benefits of testing in the cloud.
The rising popularity of cloud testing has given rise to a slew of cloud-based testing tools in the market. Some of them will be considered below.
Xamarin Test Cloud provides tools which support automated UI Acceptance Testing of mobile applications across thousands of different devices. It consists of the following parts:
- Calabash (It has been considered above).
- UITest is a framework that gives the ability to write test cases in C# using the popular NUnit testing library.
- Test Cloud is a cloud-based service consisting of thousands of physical mobile devices.
- Xamarin Test Recorder is the tool, still under development, which helps to simplify creating tests and is great for someone who is new to Xamarin.UITest and unfamiliar with the APIs.
Xamarin Test Cloud’s main advantages are:
- Big variety of testing tools for cross-platform mobile apps, native iOS, Android, Windows Mobile and Blackberry
- One Technology Stack to Code for All Platforms
- Close to Native Performance
- Full Hardware Support
- Native User Experiences
- A huge number of devices for testing. The user can run test cases on more than 2 500 devices.
However, there are still some drawbacks of Xamarin:
- Time limitations for different kinds of test cases
- Expensive Xamarin License. The business subscription comes at the annual fee of $999 per developer, per device platform, which might seem a little too high if you plan to create only one small app. For example, it will cost you almost $10,000 annually to run a team of five engineers, each building apps for iOS and Android.
- Limited access to open source libraries.
- Slightly delayed support for the latest platform update
- Hardware features such as Bluetooth, throttling WiFi, Camera, physically rotating the device, simulating different battery conditions are not supported by Xamarin Test Cloud.
Perfecto Mobile Continuous Quality Lab is the special set of tools for mobile applications testing and optimization.
The main merits of Perfecto Mobile Continuous Quality Lab are below:
- Real-time testing of smartphones, phones, and tablets. More than 500 devices, operating on Android, Blackberry, iOS, Symbian and WinPhone platforms are available for testing. Almost all devices have a real connection to major U.S., UK, Indian, Canadian, Israeli and other mobile operators.
- Full control over the device (real and virtual keyboard, touch and slide functions, accelerometer, turn on/off functions).
- Automated procedures for app installation, incoming calls, incoming text messages, file upload, information input from OS clipboard and laptop keyboard.
- Possibility to make calls, send text messages, access to the Internet (devices use tariff plans (service plans) of mobile operators).
- Device sharing
- Simultaneous automation testing on several devices.
Some of Perfecto Mobile Continuous Quality drawbacks are:
- Hot functions do not operate on most of the devices (incoming call and text messages simulation).
- Indistinct (often unreadable) image
- Long response time
- The free trial version using is limited to 60 minutes.
Keynote Mobile Testing cloud-based solution, connecting to the devices by hard-wiring of the devices. All operation is done through Keynote Systems test framework, software installation and continuous maintenance of the software is required.
Keynote Mobile Testing has the following features:
- Comprehensive cloud library of real mobile devices
- Parallel execution of multiple tests on multiple devices
- Run ad hoc anytime or schedule execution
- Import scripts built in Appium and Selenium
- View test results that include success rates, errors, log files, and visuals with high-resolution screenshots and videos
- Customize reports to match your team needs
- Collaboration features to share test results with colleague
Here are some weak points of Keynote Mobile Testing:
- There is no device reservation support
- Keynote Mobile Testing does not examine device vitals
- There is no script portability option
Mobile Applications Testing Services and Service Providers
Ubertesters is crash reporting tool, as well as full bug reporting tool. So, finally, you can get not only crash reports, but also full bug reports, with edited screenshots and full video recording of the test sessions.
The main advantages of Ubertesters:
- Multi-platform support
- Over-the-air (OTA) app distribution
- A detailed report for each test
- Complete support of all development methodologies
- Real-time team management & monitoring
- Integration with external Bug Tracking System such as Jira, Redmine, YouTrack, Mantis, Unfuddle and HP Quality Center. Besides Ubertesters provides its own easy-to-use bug tracker.
- Device management
- Build management
- In-app bug editing, marking, reporting and user feedback
- Video recording tool – the entire test sessions is fully recorded, thus, giving you a real-life recording of what steps the tester actually did prior to the bug/crash.
It also has some demerits:
- Capturing testing time data is sometimes not too accurate (if phone’s auto-lock is enabled)
- Ubertesters does not have screen cast functionality
Applause is one of the leading providers of testing services, usability feedback and research. Worldwide famous companies like Google, eBay, Fox Broadcasting are among the Applause customers.
Key features of Applause:
- More than 200,000 users providing “in-the-wild” app testing
- The variety of testing approaches
- Ability to find multiple issues quickly
- Applause provides mobile beta management by its mobile analytic tool
- It ensures compatibility with a wide range of devices
- Unique abilities to localization testing providing
Applause’s disadvantages are mentioned below:
- The PC dashboard could be a little more streamlined
- Integration with JIRA does not work if you are using single sign-on
- Applause is not flexible with some new environments such as Windows 10
Mobile Emulators to Test Websites Online on Mobile Devices
Mobile phone emulators help to check the functionality of the application across different mobile platforms, such as iPhone, iPad, Android, and BlackBerry, without actually having the devices in hand.
Let’s consider some of the popular tools, which are easy to find on the Internet.
Genymotion is the AOSP-based emulator that for testing Android applications.
Genymotion’s main advantages are:
- It simulates more than 3000 virtual device configurations and every imaginable situation to ensure a better user experience
- Genymotion develops test cases with Java API and automates them with command line tool or with Gradle plugin while you are in development phase.
- Application style and delivery promotion
There is another group of emulators for mobile testing should be mentioned. Each of them duplicates aspects of the original device’s behavior. Here are some examples of them:
Such tools have common for all advantages and disadvantages.
So, the main advantages of mobile testing via emulators are:
- Mobile emulators are completely free
- Testing on emulator is very simple
- It is easy to capture issue of screenshots over emulator
- It can be used to study the behavior of an app
However, there are also disadvantages of such mobile testing:
- Testing on emulator gives false impression
- As they are run on the PC, mobile emulators have completely different network environment.
- The emulators are not able to simulate the incoming interrupts for SMS as well as the incoming calls
- Emulators do not support interoperability testing because they are not connected to the mobile network
- Emulators cannot mimic the following features: Mobile device battery, Mobile device’s camera, etc.
- Emulators are not so much realistic simulation for mobile device memory usage
Mobile Defect Logging and Test Management Tool
There are special tools which report issues directly from your applications. Some of them are considered below.
Snaffu is the simple and effective in-app bug reporting tool that streamlines issue creation.
Key features of Snaffu:
- Any previous experience is not required for using Snaffu
- Standalone issue tracking
- Snaffu allows anyone to report issues from anywhere, at any time
- It captures logs, hardware, software, application and network details automatically for reporting
- Snaffu detects when a crash has occurred
- Email notifications about issues
- Snaffu enables automatic issue reporting from test automation frameworks like Selendroid and Appium to issue tracking tools like Jira, Pivotal Tracker, and Asana
- It continues submitting issues from your app even without a connection
The weak points of Snaffu:
- Pre-filled crash reports are available only for Android
- Video recording is allowed only for Android
BugClipper is the special tool, that makes reporting bugs very simple, smarter and more efficient.
It has following advantages:
- Beta testers or End Users can report issues in just 3 steps: Tap. Describe. Done
- It creates screen recordings
- BugClipper creates a detailed bug report
- It supports all the main platforms: Android, iOS, Windows Phone, Mc OS
- 1-minute integration
- Private cloud for premium users
- It is expensive for small teams. You can report only 50 bugs for free per month if your team is not over 5 persons.
- BugClipper is only iOs apps tool.
Mobile Optimization A/B Testing Tool
A/B testing enables you to test two or more app pages and determine which performs better. Some of the tools for mobile A/B testing are considered below.
Taplytics is a fully integrated mobile A/B testing, push notification, and analytics platform providing the tools you need to optimize your mobile app.
It has the following merits:
- Advanced analytics, custom segmentation
- It is easy to be set up
- Detailed Reporting
- Taplytics BigQuery gets your data as soon as it’s created by users.
Here are some demerits of Taplytics:
- It’s not so easy to use
- Taplytics is currently available for iOS only.
Optimimo provides flexible A/B testing and multivariate testing software that leverages segmentation by screen size, OS version, network connectivity, etc.
There are following advantages of Optimimo:
- Easy implementation
- Deep filtering
- Live update
- Real-Time Metrics
- Idea Factory runs a variety of A/B tests ranging from Icons to Landing page layouts.
There are also some disadvantages of Optimimo:
- Optimimo is currently available for iOS only
- Mobile heat maps are not supported by Optimimo
Different kinds of mobile applications testing tools have been considered above. Naturally, there is no the best one. All of them have advantages as well as weak points. It depends on you how to use them. Complex using seems to be the prospective way to make your testing of mobile applications more flexible and effective.