Posts

We compared hours and hourly rates and the outcome might surprise you

Automation testing or manual testing? That is the question! Which one is better? This is something that gives many in the tester community sleepless nights. Especially during a project when we have to determine the testing strategy, even at its later stages. This is also, unfortunately, a question with no clear answer. Why? Because the reality is that these two ways of testing are inseparably connected with each other.

If you’re still wondering which is more profitable, effective, or better suited to your or your client’s project, let’s dive right in. Here we’ll be revealing the true value of automation vs manual testing, based on one of our clients’ applications – CloudAdmin.


Why CloudAdmin?


Together with Pawel Gieniec (CEO & Founder CloudAdmin), we are creating and developing a web application – CloudAdmin. Cloud Admin is a cost-optimization platform, which helps save thousands of dollars by eliminating hidden cloud instance buying costs.

The project turned out to be perfect for introducing an automation architecture for modules that are always tested during regression testing (the process of running older tests to ensure that new updates to a piece of software haven’t introduced or re-introduced previously eradicated bugs). Our goal is to save time, which we will then be able to spend on manually testing new functionalities. The application is developing dynamically, so we’ll be needing more and more time to check all of the fresh new features.

One of the core functionalities is logging. Nothing surprising, this is probably the module most often automated.

We decided to use automated testing with Python and pytest. 

And this is how it went:

We based our login tests on these 7 test cases below.

The first video is from PyCharm (the Python IDE). It shows how long it takes to run and finish all of the tests for the login module. As you can see, all of them passed. This means all of the cases have been tested correctly and have achieved the expected results, i.e.  the functionality works properly. 

Automated tests – this process took 1:39.

Now take a look at the video below. It shows how long it takes to carry out manual testing for the same test cases as for the automated tests. 

Manual tests – this process took 1:57.

How test automation reduces the time

At first, you might think “Hmm, this is no big deal. There’s only an 18-second difference in favor of automation.”. Okay, you’re right, but let’s think more in the long-term and globally.

First of all, it’s not just an 18-second gain. It’s an almost 2-minute yield for one run! This will ultimately give us more time to spend on the project. How? 

If you decide to implement automation architecture for sections that always needs to be covered by manual regression testing, you will gain all of this time for doing other things (manual testing for new features, application development, automation architecture development).

The login module is just a drop in the ocean. But let’s take a look further. Let’s compare working times by adding the next 3 modules: AWS Ri Planner, AWS RI Management, AWS Spend.

As you can see, the automation tests give us a 35% time yield relative to the manual tests (9:16 min vs 14:14 min). And again, this is a 14:14-minute gain, we have saved on regression tests – for only 4 modules.

How test automation increases productivity

As mentioned before, by dint of test automation, the testing team gains more time they can spend on doing other tasks. But this is not the only thing they get. Regression testing can be tiring. Imagine how tedious it would be to repeat the same action over and over again. This might sound trivial, but the time involved in repetitive, manual testing is wasted, especially over a longer course of time. With this in mind, think about how much more productive testers could be if they were able to devote more time to performance testing, security testing, or testing of new features. 

Moreover, testers have to quickly and continuously execute a lot of time-consuming tests simultaneously, just to be sure an application is performing as expected at all times.

How long will setting up an automation architecture take?

It’s rare that everyone on your team has  automation experience, so when it comes to resources, you will really have to gauge how much time it will take for your tool to make an impact. You will either need to invest in expanding your team’s automation knowledge or hire automation specialists.

Let’s go back to our CloudAdmin tests. 

Considering the  4 sections previously mentioned (Login, RI Planner, RI Management, Spend), our QA Engineer (the automation tests were written by one person) devoted 80 hours to:

  • choosing and configurating tools  – 16 hours
  • writing automation tests for every test case – 16 hours per module / 64 hours per 4 modules

It might seem like this is a long period of time, but don’t be fooled. Automation architecture is reusable. We can use it limitlessly.  Maintenance and development of automated test cases is an ongoing process. Unfortunately, there’s no guarantee that  spending eighty hours automating a set of tests means you won’t have to deal with them again. But it will still take definitely less time in general. Take a look at the graph below:

Source: https://www.multisoftsystems.com/

The turnaround time for automatic testing is definitively longer than for manual. However, the benefits of automation outweigh the time benchmark in terms of:

  • Reduction in execution time 
  • Reduction in effort 

These two metrics will provide an incredible boost towards higher efficiency and a better rate of reaction to potential bugs.

What about costs?

Last but not least, is something most crucial when establishing a contract with a client – costs. Test automation might be a catchy slogan during negotiations but it’s worth checking the hidden costs. You don’t want the client to think this is just another fashionable balloon, ready to burst at any time.

If you intend to adopt an automated testing process to meet the rising demand for bug-free releases and faster delivery cycles, it’s vital to assess whether the return on investment (ROI) is worth the changes. Before executing, or even considering building an automation strategy, you will want to calculate the net gain you will achieve from transitioning. Divide this by the net investment needed to transition (i.e., the tools and resources you use), and you will see your ROI for automated testing.

Source: https://www.level12.io/

The maintenance cost of test automation is not usually trivial but if you keep enhancing the test automation framework, improve the application testability and overall testing processes, a high ROI is very much achievable.

To get the total cost, you will also want to take into account the hourly cost of the number of team members executing the tests.

Based on the average earnings for an Automation QA Engineer and a Manual QA Engineer (I rely on data from Glassdoor), we can make a labor cost comparison simulation based on the CloudAdmin case study (hours needed to perform each type of tests for this application).

As I mentioned before, it took 80 hours for our QA Engineer to develop and execute the automated tests for the 4 CloudAdmin sections. How many times might we manually test the same part of the app at the same time? 337 times (for the record: manual testing time -> total time from the module times table). So what’s better: 

  • spending $2,100 on performing  the same set of regression, manual tests 337 times

or

  • spending $2,700 on building an automation architecture – which once created is reusable?

We’ll let you answer that question ;) 

Final battle: Manual Testing vs Automation Testing

To wrap up, is an automation testing a cure-all for time-consuming manual testing? Not really. Manual testing will always be important. There are scenarios that will always require manually executed test cases. This means you will still have to create, run, and maintain the tests. But there are still a lot of situations when you can successfully implement automation. This can be a cost-effective method for regression testing of software products that have a long maintenance life.

With a balanced approach to the testing process and a smart and effective amalgamation of both automation and manual testing, you can spend more time focusing on building new features to enhance your product quality – ultimately giving yourself the opportunity to stay ahead and be more competitive in the market.

Check out other testing-related articles

Check out other business-related articles

As a tester I often get to test functionalities in which the approach to the subject plays a key role. In a previous article, I have already written about how to approach manual testing on mobile devices generally. But here I would like to talk about one of the most interesting functionalities I have had an opportunity to test. Namely the testing of advertisements in different time zones. 

The assumption of the functionality was that a banner placed on an advertisement on a given day and time could be displayed anywhere in the world, making it possible to promote movie premieres or sales promotions. The creator of the advertisement would be able to set the day and time strictly in the “User time zone” but would also be able to enter the time zone manually. 

Why 590 timezones?

Initially, the programmers wanted to use UTC to divide it into and cover 24 time zones. Then all the creator would have to do, is simply enter the country and the program would automatically add the time zone.

However, after initial tests, it turned out that this solution has one main disadvantage, namely, there is no provision for summer and wintertime, which change in virtually every country.

After consultations, they decided that the best solution would be to use the moment.js library, which is based on times in Wikipedia. With this solution, the user could enter any destination region into the field regardless of the current time. The most up-to-date solution was divided into 590 regions (TZ database).

What devices did we have to check? 

This functionality was supposed to work on both desktop and Android devices from 5.1.1 version to the newest (it’s now 10.0) and for iOS from 9.3.5 to the newest (13). There were 13 versions of both systems to cover and 590 possibilities to check. I and a second tester had a maximum of just two days to plan and conduct the tests. So we had to come up with a manual testing approach that would give us sufficient test coverage but in this very limited time. 

Manual testing 590 timezones devices

How to approach manual testing when you have 590 Time zones and 2 days

Finding common ground

We started by listing all 590 zones and finding common features, i.e. the same UTC. For example, times in Europe / Astrakhan, Europe / Samara, Europe / Saratov, Asia / Dubai, Asia / Muscat, Asia / Tbilisi, Asia / Yerevan all have UTC +04: 00 in summer and wintertime so there would only have to be one test for all of these regions on all of the devices. After these merges, according to the UTC, there were 54 zones to check. However, 54 zones is still a lot, so we had to find another elimination factor. 

Priority

The next step was to take all of the 54 available zones and check which ones have high priority, which medium and which did not need to be tested at all.

High importance in our case meant that the timezone was important from a business point of view. For example, the zone in which a large part of Europe is located, i.e. +01: 00 in winter and +02: 00 in summer. Medium priority was given to those that were less significant for business or less used, such as the zones in which the Azores or Egypt are located in.  But also parts of Australia, for example,  because Australia has 5 different time zones and some were more important than others. Zones on smaller islands, practically uninhabited, such as the zone in which Fakaofo is located, i.e. an archipelago from a group of coral islands with just 265 inhabitants, were also considered not important from a business point of view.

Testing timezones
Testing cases for timezones - manually tested

After counting and analyzing the highest priority zones, we managed to reduce it to 25 must-haves and 11 nice-to-haves that would also be checked eventually if time allowed. You should also remember that some of these zones would need to be divided into summer and wintertime as well. So let’s keep that in mind. 

How to change time zones on devices. Preparing for the manual testing

Each of the devices being tested had to have a different zone with each test.  The largest available city was selected from each time zone, and the times were changed based on that. 

Changing the time zone on a specific device is not difficult, you just need to know where to look. I will show how to do it on macOS, iPhone, and Android devices (in this example Sony Xperia).

First device: MacBook

To change the time zone and Date & Time we have to press the date in the upper right-hand corner. Then press “Open Date & Time Preferences” and then go through 4 steps:

1. Unlock the possibility of changes

2. Turn off automatic date updating

3. After going to the Time Zone tab, turn off the ability to automatically load locations

4. Choose the location around the city or by clicking on the map

Second device: iPhone

On the iPhone we have to go to Settings -> General -> Date & Time and disable the option “Set Automatically”. Now we can change the time zone by entering the city name and change the date and time. 

It is important that when you want to test the application on iOS, you need to first download the applications on the phone (from Xcode). Then change the date/time zone because in the reverse order you will not be able to upload the application in the future date (on Android there is no such dependence).

Third device: Sony Xperia

On a Sony Xperia phone (there are no major differences on other Android devices) we go to Settings. Then we go to Date & Time and disable the automatic date and time zone settings.

Test cases

After preparing the test environment and time zones, we could then start on the test cases. After going through the simplest cases of playing advertisements at a given date and time, we also had to check whether everything was working properly in any given summer and winter time. 

A good way to do this was to check the wintertime by setting the date in December or November. Because in October the time changes around the world, but it is worth giving yourself a safety buffer. And then test summertime at the end of May. 

It’s also worth checking testing zones at the end of the day.  This is to check the disappearance of the banner at 23:59 and its appearance at 00:01 the next day. The disappearance of the banner and appearance in two weeks, only on certain days or vice versa was also checked.

Some of the more interesting issues that came up during the whole manual testing process were:

  •  when we changed the orientation, the banner did not appear.
  • Another thing was that when a given advertisement was fired by the user, the hour was not read in real-time. For example the banner was to appear at 17:00 and the user started displaying the advertisement at 16:59, the banner did not appear.

Summary

Thanks for reading the whole article! As (I hope) I have shown you, manual testing of time zones can be quite simple and interesting. You just have to make sure you clarify the most appropriate approach from the beginning. 

In this case, after the feature went into production, we didn’t have to test it any further. If, of course, the tests were supposed to be repeated regularly, it would be necessary to devise a strategy for writing automated tests. A little spoiler alert… We were able to devise the test approach and do the testing itself in just two days! It was a lot of hard work, but the outcomes were totally worth it. 

See also