Posts

We compared hours and hourly rates and the outcome might surprise you

Automation testing or manual testing? That is the question! Which one is better? This is something that gives many in the tester community sleepless nights. Especially during a project when we have to determine the testing strategy, even at its later stages. This is also, unfortunately, a question with no clear answer. Why? Because the reality is that these two ways of testing are inseparably connected with each other.

If you’re still wondering which is more profitable, effective, or better suited to your or your client’s project, let’s dive right in. Here we’ll be revealing the true value of automation vs manual testing, based on one of our clients’ applications – CloudAdmin.


Why CloudAdmin?


Together with Pawel Gieniec (CEO & Founder CloudAdmin), we are creating and developing a web application – CloudAdmin. Cloud Admin is a cost-optimization platform, which helps save thousands of dollars by eliminating hidden cloud instance buying costs.

The project turned out to be perfect for introducing an automation architecture for modules that are always tested during regression testing (the process of running older tests to ensure that new updates to a piece of software haven’t introduced or re-introduced previously eradicated bugs). Our goal is to save time, which we will then be able to spend on manually testing new functionalities. The application is developing dynamically, so we’ll be needing more and more time to check all of the fresh new features.

One of the core functionalities is logging. Nothing surprising, this is probably the module most often automated.

We decided to use automated testing with Python and pytest. 

And this is how it went:

We based our login tests on these 7 test cases below.

The first video is from PyCharm (the Python IDE). It shows how long it takes to run and finish all of the tests for the login module. As you can see, all of them passed. This means all of the cases have been tested correctly and have achieved the expected results, i.e.  the functionality works properly. 

Automated tests – this process took 1:39.

Now take a look at the video below. It shows how long it takes to carry out manual testing for the same test cases as for the automated tests. 

Manual tests – this process took 1:57.

How test automation reduces the time

At first, you might think “Hmm, this is no big deal. There’s only an 18-second difference in favor of automation.”. Okay, you’re right, but let’s think more in the long-term and globally.

First of all, it’s not just an 18-second gain. It’s an almost 2-minute yield for one run! This will ultimately give us more time to spend on the project. How? 

If you decide to implement automation architecture for sections that always needs to be covered by manual regression testing, you will gain all of this time for doing other things (manual testing for new features, application development, automation architecture development).

The login module is just a drop in the ocean. But let’s take a look further. Let’s compare working times by adding the next 3 modules: AWS Ri Planner, AWS RI Management, AWS Spend.

As you can see, the automation tests give us a 35% time yield relative to the manual tests (9:16 min vs 14:14 min). And again, this is a 14:14-minute gain, we have saved on regression tests – for only 4 modules.

How test automation increases productivity

As mentioned before, by dint of test automation, the testing team gains more time they can spend on doing other tasks. But this is not the only thing they get. Regression testing can be tiring. Imagine how tedious it would be to repeat the same action over and over again. This might sound trivial, but the time involved in repetitive, manual testing is wasted, especially over a longer course of time. With this in mind, think about how much more productive testers could be if they were able to devote more time to performance testing, security testing, or testing of new features. 

Moreover, testers have to quickly and continuously execute a lot of time-consuming tests simultaneously, just to be sure an application is performing as expected at all times.

How long will setting up an automation architecture take?

It’s rare that everyone on your team has  automation experience, so when it comes to resources, you will really have to gauge how much time it will take for your tool to make an impact. You will either need to invest in expanding your team’s automation knowledge or hire automation specialists.

Let’s go back to our CloudAdmin tests. 

Considering the  4 sections previously mentioned (Login, RI Planner, RI Management, Spend), our QA Engineer (the automation tests were written by one person) devoted 80 hours to:

  • choosing and configurating tools  – 16 hours
  • writing automation tests for every test case – 16 hours per module / 64 hours per 4 modules

It might seem like this is a long period of time, but don’t be fooled. Automation architecture is reusable. We can use it limitlessly.  Maintenance and development of automated test cases is an ongoing process. Unfortunately, there’s no guarantee that  spending eighty hours automating a set of tests means you won’t have to deal with them again. But it will still take definitely less time in general. Take a look at the graph below:

Source: https://www.multisoftsystems.com/

The turnaround time for automatic testing is definitively longer than for manual. However, the benefits of automation outweigh the time benchmark in terms of:

  • Reduction in execution time 
  • Reduction in effort 

These two metrics will provide an incredible boost towards higher efficiency and a better rate of reaction to potential bugs.

What about costs?

Last but not least, is something most crucial when establishing a contract with a client – costs. Test automation might be a catchy slogan during negotiations but it’s worth checking the hidden costs. You don’t want the client to think this is just another fashionable balloon, ready to burst at any time.

If you intend to adopt an automated testing process to meet the rising demand for bug-free releases and faster delivery cycles, it’s vital to assess whether the return on investment (ROI) is worth the changes. Before executing, or even considering building an automation strategy, you will want to calculate the net gain you will achieve from transitioning. Divide this by the net investment needed to transition (i.e., the tools and resources you use), and you will see your ROI for automated testing.

Source: https://www.level12.io/

The maintenance cost of test automation is not usually trivial but if you keep enhancing the test automation framework, improve the application testability and overall testing processes, a high ROI is very much achievable.

To get the total cost, you will also want to take into account the hourly cost of the number of team members executing the tests.

Based on the average earnings for an Automation QA Engineer and a Manual QA Engineer (I rely on data from Glassdoor), we can make a labor cost comparison simulation based on the CloudAdmin case study (hours needed to perform each type of tests for this application).

As I mentioned before, it took 80 hours for our QA Engineer to develop and execute the automated tests for the 4 CloudAdmin sections. How many times might we manually test the same part of the app at the same time? 337 times (for the record: manual testing time -> total time from the module times table). So what’s better: 

  • spending $2,100 on performing  the same set of regression, manual tests 337 times

or

  • spending $2,700 on building an automation architecture – which once created is reusable?

We’ll let you answer that question ;) 

Final battle: Manual Testing vs Automation Testing

To wrap up, is an automation testing a cure-all for time-consuming manual testing? Not really. Manual testing will always be important. There are scenarios that will always require manually executed test cases. This means you will still have to create, run, and maintain the tests. But there are still a lot of situations when you can successfully implement automation. This can be a cost-effective method for regression testing of software products that have a long maintenance life.

With a balanced approach to the testing process and a smart and effective amalgamation of both automation and manual testing, you can spend more time focusing on building new features to enhance your product quality – ultimately giving yourself the opportunity to stay ahead and be more competitive in the market.

Check out other testing-related articles

Check out other business-related articles

Why do you need this Rest Assured API testing tutorial? In the world of testing – automation can be a real game-changer. That’s why it’s on everyone’s tongues, especially  “automated tests” in the context of GUI tests written in python/java with SeleniumWebDriver. Or possibly with the increasingly popular Cypress.io. However, GUI tests sit right at the top of the basic pyramid when it comes to testing. 

Read more

As Zalando kept growing to become an ecommerce giant, at some point they have decided to switch from Java to Scala. Although it’s a major move that required a lot of effort, they managed to do it in less than three months.

Want to see how they did it? Wondering if this might be the right choice for your company? Keep reading to find out more!

Java vs. Scala – the main differences

We’ll start with a quick introduction so you can see clearly what are the main differences between Java and Scala: 

  • Amount of code – This is the main difference you’ll surely notice at first sight. Scala is a drastically more concise language than Java. The number of lines is visibly reduced, thanks to the use of type inference, function passing and several other factors. 
  • Parallelism – Scala is a much better fit for parallel programming, as it’s designed to express patterns in a very concise manner. 
  • Objects and functions – While Scala is both functional and object-oriented (we’ll cover that in more detail later), you need to keep in mind that functions are acting as objects in Java. Scala treats every function as a variable, which means than one Scala function can accept another. This is one of the most impressive features of this language. 
  • Syntax – This is one of the reasons while Scala’s learning curve can be so steep. Although concise, the syntax seems a little off-putting at first, which is why some people consider Java more readable. This might be a problem for these who want to get started with Scala. 

Why Zalando chose Scala

Alright, so why exactly did Zalando decide to look for a Java alternative, then? Here’s why they thought Scala might be a better choice for some of their needs:

  • The company had years of previous experience with the Java Virtual Machine – Javier Arrieta, one of the Senior Software Engineers at Zalando, had a whopping 18 years of previous experience working with JVM. This solution is fast, efficient and quite convenient for debugging as well. Java worked fine, but there was one problem: the amount of code needed to ensure stability. The team thought that it would also be nice to use lambdas for transforming collections.
  • Scala works well with Java libraries and frameworks – This made the transition from Java to Scala much more convenient. The developers didn’t have to let go of all the libraries they’ve already been used to. They could also stick to Play, which is used for both Java and Scala development. This framework is intuitive, extensive and trustworthy, as it’s backed by Lightbend (formerly Typesafe).
  • It combines functional programming and object-oriented programming – Why choose one when you can have both? Scala presents a hybrid approach, as it combines both object-oriented (OOP) and functional programming. The possibility to choose one or the other paradigm, depending on the situation, makes Scala a really flexible choice. 
  • Scala is great for parallelization – This is one of the main advantages of functional programming. What’s more, Scala comes with a Futures API, which makes parallel programming much smoother. You can use this solution instead of sticking to traditional methods like locks, callbacks and threads. 

What kinds of challenges come with Scala

Although the Zalando team is still positive that moving to Scala was the best choice they could make, there are some challenges they had to face:

  • Compilation times – Advanced languages features in Scala make the compilation time longer. Luckily, there’s a solution to this issue. Zalando team turned to incremental compilation. This trick helped them solve the problem. 
  • Language problems – The same thing in Scala can be written in many different ways, which is both a blessing and a curse. There is no canonical style guide available and the code can get unreadable quite. When implementing Scala, it definitely pays off to implement some sort of in-house style guide. This will help you achieve consistency. 
  • Possible operator overloading – In Scala, the operator can have a different implementation depending on the argument. This case of polymorphism can lead to confusing constructions. 
  • Learning curve – As we’ve already mentioned in the Java vs. Scala comparison, Scala’s syntax can seem highly complicated to those who want to learn it. It’s a hybrid language, which is a double-edged sword: it makes work easier, but you need to consider more aspects to understand how it works. 

How Zalando introduced Scala

The team has to admit that Scala’s learning curve is pretty steep. All developers had to be introduced not only to a new language, but also new frameworks, new build tool and, above all, functional programming concepts. 

So, how do you tackle such a complex project? Of course, you need to start by providing high-quality educational materials. The team at Zalando picked Scala courses from Coursera, which turned out to be a great fit for their needs. 

The developers proceeded to learn through prototypes and greenfield projects. They were creating systems for a new environment, without integrating them with other systems yet. Another method used to introduce Scala was the layered approach. All these measures helped the team learn new skills in a controlled environment. 

Internal workshops worked quite well, as many senior Java developers were willing to learn Scala. Actually, all-Scala developers are quite a rare phenomenon – most of them have transitioned from another programming language. When it comes to exact numbers, 40% of engineers and Zalando expressed interest in learning Scala. 

In Zalando’s Dublin office, most services and data pipelines are written in Scala. The Irish team is using that language to develop the Customer Data Platform.

How the developers at Zalando adapted to Scala

It is also worth mentioning that there were two main types of Scala adoption in the company: 

  • Using Scala as Java – For developers who already had years of experience working with Java, it was almost certain to happen. You could say that they’re using Scala as an improved version of Java, with all the lambdas, pattern matching and case classes involved. 
  • Functional programming nerds – The second group, on the other hand, really got into Scala and all the exciting new opportunities that come with functional programming. They treat Scala like an entirely new programming language and are more open to trying new things. 

It’s good to keep this pattern in mind, as it’s quite likely that introducing Scala will take a similar turn in your company. Some developers may prefer to stick to their own habits and only adopt certain features of the new languages, while some may be particularly excited about implementing a new solution.

What Zalando loves about Scala

Now that Zalando has been working with Scala for quite some time, they can definitely list some of their favorite things about it:

  • Types – The team at Zalando absolutely loves types. They help them understand what kind of value they’re working with, whether it’s the customer’s password, their name, or maybe their email address. They prefer using either tagged types or value classes to manage this data.
  • Referential transparency – It’s about the substitution model. Referential transparent computation allows the team to substitute a function with parameters, which makes work so much easier. It’s also incredibly helpful for testing a program.
  • Using monads for function composition – One of the main advantages of Scala is how it allows composing functions to create more complex ones. Monads one of the most popular ways to make that happen. Thanks to them, many operations can run sequentially.

Key takeaways

As you can see, the Zalando team is very happy with using Scala instead of Java. They love to explore the wide possibilities that come with the hybrid of object-oriented and functional programming. Why not have a closer look at their process and implement a similar solution in your company? Who knows, maybe Scala is the alternative you were looking for. The case of Zalando proves that it can be successfully implemented in just a couple of weeks. 


The article is based on the following sources:

https://www.infoq.com/presentations/scalability-zalando/ 
https://jobs.zalando.com/en/tech/blog/why-we-do-scala/?gh_src=4n3gxh1 
https://www.slideshare.net/ZalandoTech/zalando-tech-from-java-to-scala-in-less-than-three-months 

Check out also:

JVM creators designed it with automatic memory management in mind, which means programmers don’t need to worry about memory allocation and memory. Unused objects can be released automatically in a transparent way, which is really convenient, especially when you’re new to JVM. But even in general, there’s less code to write and it’s less error-prone than the traditional approach which requires you to do everything manually. 

Read more

Scala is well known as a concise, readable programming language. One of the reasons is that many popular libraries offer their users DSLs to work with. These convenient APIs make creating programs simpler by saving our keystrokes and (most importantly) by improving readability. But it’s not always the case …

Sometimes we have to work with what we were given and sometimes it might be a verbose Java API. I will show you how you can make development easier by abstracting away rough edges of the underlying code. Read more