We compared hours and hourly rates and the outcome might surprise you

Automation testing or manual testing? That is the question! Which one is better? This is something that gives many in the tester community sleepless nights. Especially during a project when we have to determine the testing strategy, even at its later stages. This is also, unfortunately, a question with no clear answer. Why? Because the reality is that these two ways of testing are inseparably connected with each other.

If you’re still wondering which is more profitable, effective, or better suited to your or your client’s project, let’s dive right in. Here we’ll be revealing the true value of automation vs manual testing, based on one of our clients’ applications – CloudAdmin.

Why CloudAdmin?

Together with Pawel Gieniec (CEO & Founder CloudAdmin), we are creating and developing a web application – CloudAdmin. Cloud Admin is a cost-optimization platform, which helps save thousands of dollars by eliminating hidden cloud instance buying costs.

The project turned out to be perfect for introducing an automation architecture for modules that are always tested during regression testing (the process of running older tests to ensure that new updates to a piece of software haven’t introduced or re-introduced previously eradicated bugs). Our goal is to save time, which we will then be able to spend on manually testing new functionalities. The application is developing dynamically, so we’ll be needing more and more time to check all of the fresh new features.

One of the core functionalities is logging. Nothing surprising, this is probably the module most often automated.

We decided to use automated testing with Python and pytest. 

And this is how it went:

We based our login tests on these 7 test cases below.

The first video is from PyCharm (the Python IDE). It shows how long it takes to run and finish all of the tests for the login module. As you can see, all of them passed. This means all of the cases have been tested correctly and have achieved the expected results, i.e.  the functionality works properly. 

Automated tests – this process took 1:39.

Now take a look at the video below. It shows how long it takes to carry out manual testing for the same test cases as for the automated tests. 

Manual tests – this process took 1:57.

How test automation reduces the time

At first, you might think “Hmm, this is no big deal. There’s only an 18-second difference in favor of automation.”. Okay, you’re right, but let’s think more in the long-term and globally.

First of all, it’s not just an 18-second gain. It’s an almost 2-minute yield for one run! This will ultimately give us more time to spend on the project. How? 

If you decide to implement automation architecture for sections that always needs to be covered by manual regression testing, you will gain all of this time for doing other things (manual testing for new features, application development, automation architecture development).

The login module is just a drop in the ocean. But let’s take a look further. Let’s compare working times by adding the next 3 modules: AWS Ri Planner, AWS RI Management, AWS Spend.

As you can see, the automation tests give us a 35% time yield relative to the manual tests (9:16 min vs 14:14 min). And again, this is a 14:14-minute gain, we have saved on regression tests – for only 4 modules.

How test automation increases productivity

As mentioned before, by dint of test automation, the testing team gains more time they can spend on doing other tasks. But this is not the only thing they get. Regression testing can be tiring. Imagine how tedious it would be to repeat the same action over and over again. This might sound trivial, but the time involved in repetitive, manual testing is wasted, especially over a longer course of time. With this in mind, think about how much more productive testers could be if they were able to devote more time to performance testing, security testing, or testing of new features. 

Moreover, testers have to quickly and continuously execute a lot of time-consuming tests simultaneously, just to be sure an application is performing as expected at all times.

How long will setting up an automation architecture take?

It’s rare that everyone on your team has  automation experience, so when it comes to resources, you will really have to gauge how much time it will take for your tool to make an impact. You will either need to invest in expanding your team’s automation knowledge or hire automation specialists.

Let’s go back to our CloudAdmin tests. 

Considering the  4 sections previously mentioned (Login, RI Planner, RI Management, Spend), our QA Engineer (the automation tests were written by one person) devoted 80 hours to:

  • choosing and configurating tools  – 16 hours
  • writing automation tests for every test case – 16 hours per module / 64 hours per 4 modules

It might seem like this is a long period of time, but don’t be fooled. Automation architecture is reusable. We can use it limitlessly.  Maintenance and development of automated test cases is an ongoing process. Unfortunately, there’s no guarantee that  spending eighty hours automating a set of tests means you won’t have to deal with them again. But it will still take definitely less time in general. Take a look at the graph below:


The turnaround time for automatic testing is definitively longer than for manual. However, the benefits of automation outweigh the time benchmark in terms of:

  • Reduction in execution time 
  • Reduction in effort 

These two metrics will provide an incredible boost towards higher efficiency and a better rate of reaction to potential bugs.

What about costs?

Last but not least, is something most crucial when establishing a contract with a client – costs. Test automation might be a catchy slogan during negotiations but it’s worth checking the hidden costs. You don’t want the client to think this is just another fashionable balloon, ready to burst at any time.

If you intend to adopt an automated testing process to meet the rising demand for bug-free releases and faster delivery cycles, it’s vital to assess whether the return on investment (ROI) is worth the changes. Before executing, or even considering building an automation strategy, you will want to calculate the net gain you will achieve from transitioning. Divide this by the net investment needed to transition (i.e., the tools and resources you use), and you will see your ROI for automated testing.


The maintenance cost of test automation is not usually trivial but if you keep enhancing the test automation framework, improve the application testability and overall testing processes, a high ROI is very much achievable.

To get the total cost, you will also want to take into account the hourly cost of the number of team members executing the tests.

Based on the average earnings for an Automation QA Engineer and a Manual QA Engineer (I rely on data from Glassdoor), we can make a labor cost comparison simulation based on the CloudAdmin case study (hours needed to perform each type of tests for this application).

As I mentioned before, it took 80 hours for our QA Engineer to develop and execute the automated tests for the 4 CloudAdmin sections. How many times might we manually test the same part of the app at the same time? 337 times (for the record: manual testing time -> total time from the module times table). So what’s better: 

  • spending $2,100 on performing  the same set of regression, manual tests 337 times


  • spending $2,700 on building an automation architecture – which once created is reusable?

We’ll let you answer that question ;) 

Final battle: Manual Testing vs Automation Testing

To wrap up, is an automation testing a cure-all for time-consuming manual testing? Not really. Manual testing will always be important. There are scenarios that will always require manually executed test cases. This means you will still have to create, run, and maintain the tests. But there are still a lot of situations when you can successfully implement automation. This can be a cost-effective method for regression testing of software products that have a long maintenance life.

With a balanced approach to the testing process and a smart and effective amalgamation of both automation and manual testing, you can spend more time focusing on building new features to enhance your product quality – ultimately giving yourself the opportunity to stay ahead and be more competitive in the market.

Check out other testing-related articles

Check out other business-related articles

Introduction to OSI model and TCP/IP for Testers

Most applications out there run on the HTTP protocol, so having a solid understanding of this protocol will make your testing work much more manageable. We explored this in a previous post: What is HTTP protocol – introduction to HTTP for Testers. But there’s more to networks than just HTTP. In this post, we are going to dive deeper into networks by exploring the OSI model.

My main goal in this article is to show you the OSI model and explain how data flows in a network. Then I will go through the differences between the OSI model and TCP/IP. At the end of the article, I will also mention a few protocols used in networks.

But before we get into the details, I should explain some basic terminology.


LAN (Local Area Network) and WLAN (Wireless Local Area Network) 

Networking basics LAN WLAN

LAN is a local network that consists of a group of computers and devices connected via a single physical network (cables). It is limited to a specific geographic area/location.

An excellent example of this kind of network would be a library, office, or home. I don’t think most of us use a LAN in our homes these days, because a LAN connects devices via cables.  Nowadays, our devices are connected wirelessly via WIFI, so we’re talking about WLAN.

WAN (Wide Area Network)

WAN combines numerous sites and covers large geographic regions (connecting physically distant locations). The best example of this is the internet itself – that is, thousands of local networks (LAN / WLAN) connected. 

Another example would be connecting three company offices in different cities. Each office has its LAN. By combining them, we could create the company’s own internal network – WAN.

Networking basics WAN

Differences between IP and MAC address

You have probably already heard of and know something about what an IP is. However, you may not have met the concept of a MAC address. So, let me explain in a few words what an IP is, and then a MAC address, to illustrate the key differences between them.

IP (internet protocol) 

We use IP for communication between different networks (to address and transport data from one network to another). It performs the role of routing, i.e., searches for the fastest route to pass a data packet. An IP address is a logical address – this means that it is allocated depending on which network the device has been connected to. If a device is in two networks, it will have two IP addresses.

MAC address (Media Access Control)

MAC is a physical address with a unique identifier burned out on the network card. It identifies specific devices and is assigned by the manufacturer. MAC addresses are used for communication within one network, e.g., in a home network, if you want to connect a computer to a printer or other devices, it will use MAC addresses to do that.

Key differences to remember



Logical address

Physical address

Identifies connection with a device in the network

Identifies device in the network

Assigned by the network administrator or ISP (internet service provider)

Assigned by the manufacturer

Used in WAN communication

Used in LAN/WLAN communication

OSI model

The OSI model has never been directly implemented as it’s mostly a reference architecture on how data should flow from one application to another through a network. TCP/IP is used, and these days it’s the most popular. After the OSI model, I will say more about TCP/IP. But it’s good to start with the OSI because it’s easier to understand some of the concepts. 

Networking basics OSI model

The OSI model consists of 7 layers divided into two groups:

  • Host layers (happening on the computer side. Responsible for accurate data delivery between devices)
  • Media layers (happening on the network side. Responsible for making sure that the data has arrived at its destination)

7. Application layer

In this layer, the user directly interacts with applications. Here is decided which interfaces are used to interact with the network through the corresponding protocols in this layer. 

Examples of such applications are chrome or Gmail:

  • Chrome uses the HTTP / HTTPS protocol
  • Gmail uses email protocols like SMTP, IMAP.

The applications themselves are not in the application layer – in this layer, there are only the protocols or services that the applications use.

6. Presentation layer

The task of this layer is proper data representation, compression/decompression, encryption/decryption. This ensures that the data sent from the X system application layer can be read by the Y system application layer.

5. Session layer

This layer is responsible for creating, managing, and then closing sessions between two applications that want to communicate with each other. 

4. Transport layer

The task of this layer is to make sure that the data has arrived safely from the sender to the recipient. When it sends data, it breaks it into segments. When it accepts data, it puts it back into a stream of data.

Networking basics Transport Layer

In this layer  two protocols are used: TCP and UDP (later on in the article I’ll be saying more about these)

3. Network layer

Provides addressing and routing services. It defines which routes connect individual computers and decides how much information to send using one connection or another. Data transferred through this layer are called packets.

Places two addresses in the packet sent:

  • Source address
  • Destination address

This layer is based on IP (internet protocol).

2. Data-link layer

This layer deals with packing data into frames and sending them to the physical layer. It also oversees the quality of the information provided by the physical layer. It recognizes errors related to losing packages and damaging frames and deals with their repair. 

1. Physical layer

This is the physical aspect of the network. This applies to cables, network cards, WIFI, etc. It is only used to send logical zeros and ones (bits). It determines how fast the data flows. When this layer receives frames from the data link layer, it changes them to a bitstream.

Encapsulation and decapsulation of data

Encapsulation and decapsulation of data

Encapsulation adds pieces of information to data sent over the network. This occurs when we send data. At each layer, some information is added to our data. We combine the address of the sender and recipient, the encryption method, data format, how the data will be divided, sent, etc.

Decapsulation occurs when we receive information. It consists of removing pieces of information collected from the network. At each layer, some info disappears. In the end, the user only gets what he should receive without the IP, MAC address, etc.

Differences between the OSI model and TCP/IP

The TCP/IP model has a similar organization of layers to the OSI model. However, TCP/IP is not as rigorously divided and better reflects the actual structure of the Internet.

Networking basics Differences between OSI model and TCP:IP

In TCP/IP, there are only four layers:

  • Application layer
  • Transport layer
  • Internet layer
  • Network interface layer

The OSI model makes a clear distinction between layers and some concepts. In TCP/IP, it is harder to make this clear distinction and explain these concepts. Now you can see why I introduced to you the OSI model before the TCP/IP.

The TCP/IP application layer contains three layers from the OSI model:

  • Application layer
  • Presentation layer
  • Session layer 

The working of the application layer in the TCP/IP is a combination of these three layers from the OSI model. In this layer, we have various protocols such as HTTP, DNS, SMTP, FTP. 

The transport and internet layers in TCP/IP work, as I described in the OSI model. But in the next section, I will be revealing more details on how the transport layer protocols (TCP and UDP) work.

The network interface layer in TCP/IP is a combination of two layers form the OSI model (data link and physical layer). I’m not going to go into the details of this layer. But in the OSI model, I described the critical functions of these two last layers. Here in TCP/IP, these functions are realized in one layer.

Protocols in the TCP/IP model

Internet layer protocols

ARP (Address Resolution Protocol)

Used to identify the MAC address. If the device knows the IP address of the target device, then ARP sends a request to all of the devices in the LAN to search for the MAC address of the device with the given IP. Then the device with that IP sends an ARP response with its MAC address. 

This information will be saved in the ARP table. In windows or macOS, open terminal and type arp -a. Then you should see the ARP table.

In the image below, you can see how this process works when an ARP request matches the IP of the device.

Networking basics

The RARP protocol performs the reverse operation.

IP (Internet protocol)

I explained at the beginning of this article what IP is. But I want to make clear that the IP in the TCP/IP model is in the internet layer. It is also good to add that IP has two versions.

  • IPv4
  • IPv6

The second one has been introduced because IPv4 addresses are ending. IPv6 is more efficient, has better routing, and is safer. 

ICMP (Internet Control Message Protocol)

This acts as a tool for solving problems. The ICMP reports any communication errors between hosts. ICMP messages can help to diagnose a problem. For example, if the router or host is overloaded, ICMP can send a message to slow down the transfer rate.

ICMP is used in the ping program, which allows the diagnosis of network connections. Ping lets you check if there is a connection between the hosts. It also allows you to measure the number of packets lost and delays in their transmission.

In the terminal, type ping After ping, you need to provide the host. You can choose any website. I’m going to check my connection with the scalac site. To exit ping, use CTRL + C.

Ping sends an ICMP packet to the host provided. In my case, I sent 17 packets and received back 17 packets. In this short connection, I didn’t lose any packets. The program also counts the time gap between sending and receiving packets. In the end, the program summarizes the connection and shows us the minimal/ average / maximum time gap between sending and receiving packets.

Transport layer protocols

TCP (Transmission Control Protocol)

TCP is a highly reliable and connection-oriented protocol. It applies the 3-way handshake principle. Before it sends any data, it will first establish a connection.

Networking basics - Transmission Control Protocol

This rule consists of three steps, made to establish a connection.

  1. SYN – The device sends a message to the server, “I want to connect with you.”
  2. SYN / ACK – When the server receives the message, it will reply that it is ready for communication.
  3. ACK – The device sends confirmation of receiving the response from the server and that it is ready for communication.

The high reliability of TCP is due to the device, making sure that the data sent has been received by the server. Then the server makes sure that the data sent to you have been collected by you. If the server sends 10 data packets, and for some reason, you do not receive one of them, and you do not confirm the receipt, this server will try to send the lost package again. 

TCP also provides data delivery in order. Each sent packet is numbered. Although packets may still arrive out of order, TCP will arrange them in order before sending them to the application.

To summarize the advantages of TCP:

  • Set up a connection before sending any data
  • Data delivery acknowledgment
  • Retransmission of lost data
  • Deliver data in order

UDP (User Datagram Protocol)

UDP sends data and doesn’t care if the device has received it or not. It also doesn’t care if some packets are lost. But the significant advantage of the User Datagram Protocol is that the packet sizes are smaller than TCP (about 60% lighter). 

Networking basics connection

UDP is an economical version of TCP. 

  • Connectionless and unreliable.
  • No data retransmission
  • No data delivery acknowledgment
  • Data can arrive out of order

You may ask the question, then why use UDP? It’s such an unreliable protocol!

In some cases, UDP is better because TCP has significant overheads (data retransmission, delivery acknowledgment, etc.) UDP is often used to transmit data in real-time: video streaming or audio such as Skype calls.

Application layer protocols

Network management protocols

DNS (Domain Name Services) – Changes the domain name to an IP address. The domain name is used because it’s human-friendly. It’s easier to remember a domain name ( than an IP address ( When you type any website address into a browser, then the browser sends a request to the DNS for the IP address of that domain.

Networking basics management protocols

If you type into a browser IP, then you should see the Google page because this is Google’s IP address. I can get it directly by requesting the DNS in the terminal. Type in terminal: nslookup 

NTP (Network Time Protocol) – This is an uncomplicated and straightforward protocol. It is used for automatic time synchronization in devices connected to a network. Imagine now manually synchronizing time for 10 or 50 devices. This would be ineffective.

Some devices, procedures, or safety mechanisms require accurate time synchronization for proper operation. Also, thanks to the NTP, finding the causes of any network or device errors is easier. Because using the logs, we will be able to find out what the order of events was that caused the failures.

SNMP (Simple Network Management Protocol) – This is used for monitoring, management of updates, and diagnostics of networks and network devices.

Remote authentication protocols

SSH (Secure Shell) – This allows you to remotely log in to the terminal in network devices and administer them (e.,g. router, firewall, a remote server). SSH is secure because communication is encrypted. SSH uses the TCP protocol.

File transfer protocols

FTP (File Transfer Protocol) –  The purpose of this protocol is to display a list of files/folders, adding, deleting, or downloading them from the server. A good example is sending website files to a server. To do this, you need to use an FTP client with which you can authenticate yourself and get access to the FTP server. A popular FTP client is FileZilla. FTP uses TCP. 

A significant flaw of FTP is the lack of data encryption. Therefore, to ensure secure authentication and transfer of files, it is worth using FTPs (FTP Secure and FTP-SSL) or SFTP (SSH File Transfer Protocol). They work in the same way as FTP but extend its functionality by encrypting the transmitted data.

Email protocols

SMTP (Simple Mail Transfer Protocol) and IMAP (Internet Access Message Protocol) are two protocols used in sending and receiving emails. SMTP’s task is to send email messages from a client to an email server or between email servers. IMAP is used to manage and retrieve email messages from an email server.


This image shows an example when a sender ( and a recipient ( have different email service providers.

  1. In the beginning, the email message is sent to the sender’s email server (Gmail)
  2. Then the Gmail email server sends an email message to the recipient’s email server  (WP)
  3. Finally,  IMAP retrieves the email message from the wp email server to our client.

When the sender and recipient have the same email service provider (Gmail), step 2 will be skipped.

Browser protocols

HTTP/HTTPS – I have written a separate article on HTTP. You can read it here: What is HTTP protocol – introduction to HTTP for Testers. I explain there exactly how HTTP works. HTTPS extends HTTP functionality with data encryption protocols.

VoIP protocols (Voice over IP)

SIP (Session Initiation Protocol) – This performs an administrative function (using TCP). It is used only to set and close an audio or video connection.

RTP (Real-Time Transport Protocol) – This is used to transfer data during audio or video calls (using UDP).

For example, let’s say you want to call someone on Skype. SIP will be used to establish the connection. When the connection is established, the RTP springs into action and transmits the data. When you end the conversation, SIP will close the connection.


You have come to know many new concepts today. You now know how data flows in networks. They go through a rather complicated process. All of the topics I have touched are so extensive; they could easily have a separate article for themselves. However, I have tried to present them to you at a fairly general level, easy to understand. Without going too deeply into the more technical aspects.

If you think I have managed to explain things understandably and interestingly, please share this article on social media. And if you have any questions, also feel free to ask them in the comments below.