When working on software development projects, it is important to keep track of all code as lots of it is written by different teams. Similarly, automated testing of the code may be done in parallel by different teams at various locations and there is a need to track modular and integration test coverage by unified framework.
Alongside software development, Degree 53 provides comprehensive automation testing and consulting services to ensure the products we build can be tested repeatedly and reliably through automation during testing and operation. Automation testing provides quick feedback on the failure and it can be integrated with the release pipeline to save delivery time and to build more confidence with higher test/code coverage.
We use a number of tools and frameworks to run test scripts for both manual and automated tasks. Among various frameworks available, we found that Behaviour Driven Development (BDD) is extremely useful to building traceability and improving collaboration for such projects. BDD is a set of practices designed to reduce wasteful behaviour in software development.
For those who don't have much familiarity with what BDD and what its benefits are, here is a quick snippet:
BDD automation framework vs manual testing
We have recently successfully implemented a complete BDD automation framework for one of our clients.
The requirement was to build an automation framework for a mobile software development kit (SDK) which could take user-defined data (data-driven) and run seamlessly for web and mobile automation testing. Moreover, the framework and scripts both needed to be built with the consideration for scalability and reusability, so that they could be plugged easily with end-to-end automation testing once the client would add their API test scripts.
Since the code was developed on Java, we used Cucumber. Cucumber is used to run the tests. Cucumber tests are added in a feature file. These tests can be understood by anyone as they are written in simple English (Gerkin) format. Also, within the feature file, the browser or device is listed. Here, the device or browser can be edited based on BrowserStack compatibility.
- Functional Library or utilities
- Test Data Files (CSV or JSON)
- Test suite driver, including startup scripts, test scripts, cleanup scripts)
- Selenium Webdriver & Appium
- Serenity reporting
Selenium WebDriver is designed to provide a simpler, more concise programming interface in addition to addressing some limitations in the Selenium-RC API. Selenium WebDriver was developed to better support dynamic web pages where elements of a page may change without the page itself being reloaded. WebDriver's goal is to supply a well designed object-oriented API that provides improved support for modern advanced web app testing problems.
Appium is an open-source tool for automating native, mobile web and hybrid applications on iOS mobile, Android mobile and Windows desktop platforms. Each Appium specific platform is implemented differently due to the nature of each operating system. As Selenium was developed for websites, Appium was developed for automating native applications.
Maven is a build automation tool used primarily for Java projects. Maven is built using a plugin-based architecture that allows it to make use of any application controllable through standard input.
BrowserStack allows you to test your native and hybrid apps on a wide range of physical mobile and tablet devices, including iPhone, iPad, Samsung Galaxy, Nexus, Pixel, etc. BrowserStack can also be used to test manually. It also enables testers to release builds faster by running concurrently across thousands of devices, offering quick results. One of the best things about it is it reduces the test execution time by more than 10x by using parallel tests on the BrowserStack device cloud.
You can debug and fix bugs instantly as BrowserStack record:
- Device Logs
- Text Logs
- Video Recording of the entire testing session
- Appium screenshots with timestamps
Serenity BDD is a testing tool that helps produce illustrated, narrative test reports. It helps to map automated tests to your requirements, see how much your application is being tested and it also helps to track project progress.
IntelliJ is a Java integrated development environment for developing computer software. Selenium, Appium, Maven, BrowserStack and Serenity are added into IntelliJ.
The automation tests are written in Java. Elements on the website/application are looked for and functions are created to use these elements.
Advantages of the automation framework
- Can be used for unit testing, functional testing and end-to-end testing based on requirements.
- Non-functional automation tests like JMeter performance, stress tests and security tests can be easily integrated.
- The framework is very useful for troubleshooting as it gives device log, text log, video recording of the testing session and screenshot, timestamp of Appium. So, the framework does not only provide a faster feedback loop of your code changes, but it also provides a debug and troubleshooting facility to fix and test issues quickly.
- Reporting is flexible and easily customizable, so there is no need to invest any time in post processing test results.
- By creating scripts only once, you can execute it on multiple browsers, devices and operating systems without any extra efforts and catch any browser or device specific issues.
We have benchmarked and compared the testing time, coverage, cost and quality of the project by using the above automation framework, and here are our findings:
- Testing time was reduced by about 68% compared to manual testing for a project of a similar size
- 75% line code coverage was achieved for this project through unit and automation test coverage
- Release delivery time was reduced by about one month for the mobile SDK delivery compared to a project of a similar size previously
- Automation framework helped in debugging and fixing issues quickly. It improved pass % over a period of time and stabilisation achieved sooner, compared to previous projects.
- The automation package was provided to the customer and worked seamlessly in the release pipeline which reduced their end-to-end testing time as well.
The above method proved that creating tests using this automation framework is quicker and more efficient than running tests manually. Each test only took a couple of minutes to run, compared to around 10 minutes manually. The automation tests were ran overnight, saving resources. Overall, throughout the project, Degree 53 reduced test time by 10x.
We have expert automation architecture and testers who can help you with designing/customising an automation framework which gives you a great return on investment. Get in touch with us for any additional questions.