What is Automation Testing?
While the manual testing executed by successive steps, the automation testing using automation tools, which can be conveniently administered test data, test system to compare the expected and actual results and generate detailed test reports.
Because the development cycles require performing the same tests repeatedly testing requires a significant investment of money and resources.
By using test automation tool you can record test suit and replay it as needed. After the automated test set, no need for human intervention and for through is possible to improve return on investment test automation. The main goal of automation testing is to reduce the number of tests to run manually.
Benefits of Automation Testing
Following are benefits of automated testing:
- Faster than the manual testing
- Wider test coverage of application features
- Reliable in results
- Ensure Consistency
- Saves Time and Resources
- Improves accuracy
- Human Intervention is not required while execution
- Increases effectiveness
- Reusable test scripts
- Test thoroughly and Frequently
Drawbacks of Automation Testing
- False sense of quality
- Not reliable
- Automation is not testing
- Maintenance Time and Effort
- Slow feedback
- Not many bugs found
Automation testing frameworks
Selenium is a Java application that can analyze files of a specific structure in order to find commands for manipulating the browser and commands for performing certain actions and checks.
Being one of the most popular automated testing tools for WEB on different platforms, Selenium has support from some browsers that make Selenium a native part of their browser.
Selenium was originally developed by Jason Huggins in 2004 as an internal tool at ThoughtWorks.
Stable release 3.0 ( October 13, 2016)
- Selenium is pure open source, freeware and portable tool.
- Selenium supports a variety of languages that include Java, Perl, Python, C#, Ruby, Groovy, Java Script, and VB Script. etc.
- Selenium supports many operating systems like Windows, Macintosh, Linux, Unix etc.
- Selenium supports many browsers like Internet Explorer, Chrome, Firefox, Opera, Safari etc.
- Selenium can be integrated with ANT or Maven kind of framework for source code compilation.
Read more about Selenium
Watir is an automation testing tool that using to testing of web apps. It is an open source library of Ruby and anyone can get it on GitHub. It’s very helpful for automating web browsers and allows you to create simple tests
that easy maintain.
Unlike testing tools based on the HTTP protocol that simulates browser requests, Watir simulates user interaction with the browser using the OLE protocol, which is based on the COM architecture.Ruby supports OLE, which allows the automation of Microsoft Internet Explorer.
Stable release 5.0 (October 5, 2013).
- Tests any language-based web application
- Cross-browser testing
- Compatible with business-driven development tools like RSpec, Cucumber, and Test/Unit
- Tests web page’s buttons, forms, links, and their responses
Read more about Watir
SoapUI is a cross–platform engineering tool for functional testing, which was created for automatically be used to test the interface, so SOAP and REST for secure access to the oldest add–ons.It is built entirely on the Java platform and uses Swing for the user interface. This means that SoapUI is cross–platform.
Today, SoapUI also supports IDEA, Eclipse, and NetBeans. SoapUI can test SOAP and REST web services, JMS, AMF, as well as make any HTTP(S) and JDBC calls.
Stable release 5.3 (December 5, 2016)
Pricing: Free (open source). €595 – €1785/per year (professional version).
- User-Friendly GUI
- Easy for Functional Testing
- Vulnerability Testing
- Load Testing using LoadUI
- Automation with Groovy
Read more about SoapUI
JUnit has been very important in test-driven development and testing framework at all. This testing framework using for testing applications written in Java and help to easy design repeatable tests. A research survey performed in 2013 across 10,000 Java projects hosted on
A research survey performed in 2013 across 10,000 Java projects hosted on GitHub found that JUnit was the most commonly included external library. Each library was used by 30.7% of projects.
Stable release 4.12 (December 4, 2014)
- Simple framework for writing automated, self-verifying tests in Java
- Support for test assertions
- Test suite development
- Immediate test reporting
Read more about JUnit
TestNG is also for the Java programming language application. This testing framework was inspired by JUnit and NUnit. It provides few new features for writing tests that make it more powerful convenient and easy to use.
TestNG was created to cover multiple types of tests, such as unit, functional, end-to-end and integration tests.
Stable release 6.9.12(June 21, 2015)
TestNG’s main features include:
- Annotation support.
- Support for parameterized and data-driven testing
- Support for multiple instances of the same test class
- Flexible execution model. TestNG can be run either by Ant via build.xml, or by an IDE plugin with visual results. There isn’t a TestSuite class, while test suites, groups and tests selected to run are defined and configured by XML files.
- Concurrent testing: run tests in arbitrarily big thread pools with various policies available, and test whether the code is multithread safe.
- Embeds BeanShell for further flexibility.
- Default JDK functions for runtime and logging.
- Dependent methods for application server testing.
- Distributed testing: allows distribution of tests on slave machines.
Read more about TestNG
NUnit is an open environment for a unit-testing framework for .NET. It was ported from the Java language (the JUnit library). The first versions of NUnit were written in J #, but then all the code was rewritten in C # using such .NET innovations as attributes.
There are also known extensions of the original NUnit package, most of them also with open source. NUnit.Forms complement NUnit by testing the Windows Forms user interface elements. NUnit.ASP performs the same task for interface elements in ASP.NET.
Stable release 3.2 ( 5 March 2016)
- Tests can be run from a console runner, within Visual Studio through a Test Adapter, or through 3rd party runners.
- Tests can be run in parallel.
- Strong support for data-driven tests.
- Supports multiple platforms including .NET Core, Xamarin Mobile, Compact Framewor, and Silverlight.
- Every test case can be added to one or more categories, to allow for selective running.
Read more about NUnit
Google Test is a library for the C++ programming language apps that based on the xUnit architecture. It’s executing testing individual parts of the program such as classes, functions, modules. They are checked separately from each other, in isolation and it called unit testing.
It can be compiled for a variety of POSIX and Windows platforms, allowing unit–testing of C sources as well as C++ with minimal source modification. One of the biggest advantages of this tool is a possibility to run tests as one and all at once at the same time, which makes testing more convenient and versatile for testers and coders.
Stable release 1.8( August 22, 2016)
- An XUnit test framework.
- Test discovery.
- A rich set of assertions.
- User-defined assertions.
- Death tests.
- Fatal and non-fatal failures.
- Value-parameterized tests.
- Type-parameterized tests.
- Various options for running the tests.
- XML test report generation.
Read more about GoogleTest
Testoob is an acronym which stands for Python Testing Out Of (The) Box. This tool is an advanced unit testing framework for Python language apps. It’s very convenient because easy integrates with existing PyUnit (module “unit test”) test suites. You can get it on GitHub.
Stable release 1.12 (9 September 2006)
- Work with PyUnit tests as-is, no changes necessary
- Filter which tests to run with regular expressions
- Output test results as XML/HTML/PDF
- Test skipping
- Color output on a terminal
- Run pdb post-mortem on failed tests
- Profiling support
- Run tests in parallel in threads or processes and more!
Read more about TestToob
csUnit is a framework for unit testing. It works with compliant language applications. csUnit was designed to work with any .NET compliant languages.
It has specifically been tested with C#, Visual Basic .NET, Managed C++, and J#. The framework is open source and comes with a flexible license that allows cost–free inclusion in commercial closed–source products as well.
The tool offers a native GUI application, a command line, and add-ins for Visual Studio 2005 and Visual Studio 2008.
Stable release 2.6 (March 28, 2009)
- Standard unit testing features such as test, fixture, setup, teardown, ignore an expected exception, etc.
- Easy to use graphical user interface
- Recipes for combining several test assemblies into one test suite
- Add-in for Visual Studio 2005 and Visual Studio 2008
- Search capabilities across tests, output, and statistics
- Statistics per test to create performance baseline
- Categories to group tests for execution
- Support for applications targeting .NET 2.0, .NET 3.0, and .NET 3.5
- Works with any .NET language (C#, VB.NET, Managed C++, etc.)
- Runs on Windows Vista SP 1 or Windows XP SP 3
Read more about csUnit
Htmlunit is a web browser which was created in Java and ensures access to structure and detail within the received web pages.
Using this tool you can imitate emulates the behavior of the browser like fill the fields, submit the form and clicking hyperlinks to websites.
It can deal with HTTPS security, basic HTTP authentication, automatic page redirection and other HTTP headers.
The goal is to simulate real browsers such as Chrome, Firefox ESR 38 and Internet Explorer.
Stable release 2.23 ( July 27, 2016)
- Provides high-level API, taking away lower-level details away from the user.
- Compared to other WebDriver implementations, HtmlUnitDriver is the fastest to implement.
- It can be configured to simulate specific Browser.
Read more about HtmlUnit
One of the most popular unit testing framework for PHP projects is PHPUnit. At the first time, it was an instance of xUnit architecture and after became popular with JUnit. This framework was designed by Sebastian Bergmann and hosted on GitHub. The main goal of the tool allowed developers to find mistakes in their code quickly. And they are able to assert that no code regression has occurred in other parts of the code base.
In order to show that the individual parts are correct, this tool makes their goal of unit testing is isolate each part of the program. As follows, a unit test can find problems in the first times of development cycle. PHPUnit can output test results in a number of different formats, including JUnit XML, Test Anything Protocol, JSON, and TestDox.
Stable Release 6.0 (February 3, 2017)
- PHPUnit is the standard; most frameworks use it (like Zend Framework (1&2), Cake, Agavi, even Symfony is dropping their own Framework in Symfony 2 for PHPUnit).
- PHPUnit is integrated into every PHP IDE (Eclipse, Netbeans, Zend Stuide, PHPStorm) and works nicely.
- PHPUnit works fine with every continuous integration server since it outputs all standard log files for code coverage and test reports.
- PHPUnit is actively maintained, stable and works great for every codebase, every scenario and every way you want to write your tests.
Read more about PHPUnit
SimpleTest is an open source unit test framework for the PHP programming language and was created by Marcus Baker. The test structure is similar to JUnit/PHPUnit. SimpleTest supports mock objects and can be used to automate the regression testing of web applications with a scriptable HTTP Client that can parse HTML pages and simulate things like clicking on links and submitting forms.
Stable release 1.1.7 ( September 21, 2015)
Read more about SimpleTest
It is a testing business development driven that helps testers to test the software. Cucumber does automated tests for acceptance that writing in BDD style. As such, Cucumber allows the execution of feature documentation written in business-facing text. Capybara, which is a part of the Cucumber testing framework, is an example of a web-based test automation software.
Cucumber is written in the Ruby programming language and was originally used exclusively for Ruby testing as a complement to the RSpec BDD framework. Cucumber now supports a variety of different programming languages through various implementations. For example, Cuke4php and Cuke4Lua are software bridges that enable testing of PHP and Lua projects, respectively. Other implementations may simply leverage the Gherkin parser while implementing the rest of the testing framework in the target language.
Stable release 1.3.15 (May 9, 2014)
Pricing: Free (open source), mothly $45/user (Cucumber Pro) .
Read more about Cucumber
Behavior-driven development (or BDD) is an agile software development technique that encourages collaboration between developers, QA and non-technical or business participants in a software project.
Stable release 1.2.6
- Establishing the goals of different stakeholders required for a vision to be implemented
- Drawing out features which will achieve those goals using feature injection
- Involving stakeholders in the implementation process through outside–in software development
- Using examples to describe the behavior of the application, or of units of code
- Automating those examples to provide quick feedback and regression testing
- Using ‘should’ when describing the behavior of software to help clarify responsibility and allow the software’s functionality to be questioned
- Using ‘ensure’ when describing responsibilities of software to different outcomes in the scope of the code in question from side-effects of other elements of the code.
- Using mocks to stand-in for collaborating modules of code which have not yet been written
Read more about Behave
Automation testing tools
SeeTestAutomation is the special tool that allows running automated mobile app testing not only on iOS or Android but also on BlackBerry and Windows Phones on real devices as well on emulators.
Pricing: information is available only upon the direct request. Free trial period is 30 days.
- Perform mobile test automation using real devices and emulators
- Create a test on one device and run it on any other device model or operating system
- Increase coverage with full peripheral testing capabilities – GPS, Touch ID, camera and audio
- Start your mobile test automation quickly with a simple plug, record and play tool
- Enhance your mobile test automation by Integrating into all your major ALM environments
- Integrate with WebDriver (Selenium), HP UFT/QTP, JUnit, Microsoft Visual Studio and more
- Integrate with any continuous integration environment – HP Quality Center (ALM), Jenkins and more
Read more SeeTestAutomation
Rapise is the next generation software test automation platform that leverages the power of open, extensible architecture to provide the most rapid and flexible functional testing tool on the market today. Rapise’s unique Learn and Go testing method is much more time efficient than traditional record-and-playback processes. Objects can be edited during the learning process instead of waiting for all objects to be recorded then edited.
Rapise provides powerful built-in support for the following environments: Java, Google Web Toolkit, Yahoo User Interface, Web applications including AJAX, Windows Forms including Infragistics, DevExpress, Telerik, Adobe Flash/Flex including AIR, Microsoft .NET, Win32 applications and WPF. With Rapise you can use the same environment to test all these technologies.
Rapise’s cross-browser testing capabilities are best in class, with support for multiple versions of Microsoft Internet Explorer, Mozilla Firefox and Google Chrome. Choose Rapise to test web applications in all of the most widely used browsers. Record or create one test script and execute the same script without modification across the major browsers. Built-in support for GWT and YUI. Fully Integrated with SpiraTest for Test Management.
Stable release 5.0(January 12, 2017)
Pricing: $799.99 /user (single) –$359.99 /user (100+ users). Free trial version for the single user.
Read more about Rapise
Ranorex is a GUI test automation framework for testing of desktop, web-based and mobile applications. Ranorex is provided by Ranorex GmbH, a software development company for innovative software test automation solutions. Ranorex does not have a scripting language of its own, instead of using standard programming languages such as C# and VB.NET as a base.
Stable release 6.2.1 (January 18, 2017)
Pricing: € 690 (Runtime) – €3 990 (Premium Floating). Free trial version is available upon the direct request.
The main advantages of Ranorex using:
- Easy installation for non-programmers.
- “Click & Go” XPath editor.
- UI object map editor.
- Support for Dynamic IDs.
- Easy maintenance.
- Easy execution of tests with executable (.exe) files.
- Professional Support Services.
Read more about Ranorex
qTest encompasses the full lifecycle from requirements and test case design, through test execution and defect tracking, to real-time reports on status and quality-metrics. This SaaS solution allows you to get up and running in a matter of minutes. It supports both scripted and unscripted (exploratory) testing approaches.
Stable release 8.1
Pricing: information is available upon the direct request. Free trial period is 14 days.
- Support for Agile testing
- Intelligent multi-screen capture
- Auto recording of all clicks and screen actions
- Cross application capture
- Ability to add notes and annotation
- Test case and defect submission
Read more about qTest eXplorer
Telerik Test Studio
Telerik Test Studio is a software testing tool that based on Windows. It was designed for web and desktop functional testing, software performance testing, load testing and mobile application testing.
Stable release 2012.2 (R2 2012)(September 20, 2012)
Pricing: $349 (Runtime perpetual) – $3 499 (Ultimate perpetual). Free trial version is available upon the direct request.
Test Studio’s features include:
- Scriptless test recording and playback
- Cross-browser test execution – Internet Explorer, Firefox, Chrome and Safari (web browser)
- Support for HTML, AJAX, Silverlight, WPF and ASP.NET MVC application testing
- Element abstraction and reuse
- Integration with Visual Studio 2010 / 2012, MS Team Foundation Server
- NUnit, MbUnit, XUnit integration
- Sentence-based UI validation
- Continuous integration with Microsoft Build Server, CruiseControl, and TeamCity
- Visual Debugger
- Test customization in C# and VB.NET
- Exploratory Testing
- Manual Testing
- Integration with HP Quality Center
- Built-in testing framework
- DOM Explorer
- Fiddler web debugger integration
- Automated data-driven testing
- Bug-tracking tools integration
Get from http://www.telerik.com/teststudio
Read more about TestStudio
For API automated testing
A load testing tool developed by the Apache Software Foundation. Although JMeter was originally developed as a web application testing tool, it is now able to perform load tests for JDBC connections, FTP, LDAP, SOAP, JMS, POP3, IMAP, HTTP, and TCP.
To create a large number of queries using several computers when managing this process from one of them. The architecture that supports plug-ins of third-party developers allows you to supplement the tool with new features.
The program implements mechanisms for authorizing virtual users, user sessions are supported. Logging of test results and various visualization of results in the form of diagrams, tables, etc. are organized. Apache JMeter is a solution for load testing for analyzing and measuring the implementation of various services with a focus on web apps.
Stable release 3.1 (November 19, 2016)
- Being an open source software, it is freely available.
- It has a simple and intuitive GUI.
- JMeter can conduct load and performance test for many different server types − Web – HTTP, HTTPS, SOAP, Database via JDBC, LDAP, JMS, Mail – POP3, etc.
- It is a platform-independent tool. On Linux/Unix, JMeter can be invoked by clicking on the JMeter shell script. On Windows, it can be invoked by starting the jmeter.bat file.
- It has full Swing and lightweight component support (precompiled JAR uses packages javax.swing.* ).
- JMeter store its test plans in XML format. This means you can generate a test plan using a text editor.
- It is full multi threading framework allows concurrent sampling by many threads and simultaneous sampling of different functions by separate thread groups.
- It is highly extensible.
- It can also be used to perform automated and functional testing of the applications.
Read more about JMeter
Pricing: Free version. Paid versions: monthly $8 – $21/user.
Read more about PostMan
The right selection of automation tool, testing process, and team, are important players for automation to be successful. Manual and automation methods go hand-in-hand for successful testing.