Testing for mobile apps

How to approach problematic areas using simulators and more

It’s no secret that mobile devices are taking over the world. Today, it’s possible to complete basically any operation, anywhere, within a few seconds – all thanks to smartphones. This market is constantly growing, and slowly outrunning laptops, not to mention other stationary devices.

With the growing number of mobile devices on the market, the problem of how to keep up with the needs of users and provide them with high-quality software is also increasing. To meet these demands, we need a specific approach. That’s why testing for mobile apps is a completely different topic than web application testing.

To automate or not to automate? That is the question!

I think testing mobile applications is a good candidate for automation. It often allows you to provide high test coverage. However, it can be time-consuming and not very profitable due to the specifics of a project.

For over a year, together with my QA team, we tested specific, short advertisements – each of them unique. There were at least 5 of them a day. The space for automation was also limited and covered only basic elements such as detecting html5_interaction or playback of subsequent video elements and detecting whether the game ‘install’ button was clicked.

Test coverage

Our tests covered a wide range of devices: iPhones, the entire range of Android phones, and Amazon devices.

The test coverage included:

Division by system:

• Android from 5.1.1 to the newest ( it’s now 10.0)
• iOS from 9.3.5 to the newest ( it’s now 13)

Division by type of devices:

• Amazon (in this case one device was enough, e.g. Fire HD 8 Tablet)
• Low-spec devices – e.g. iPhone 5s or Samsung J3
• High-spec devices – e.g. iPhone 8+ or Samsung Galaxy J7
• Wide aspect ratios – e.g. Samsung Galaxy S8+ or Google Pixel 2
• Old iPad and New iPad – e.g. iPad 2 Air old and the new generation
• Android Tablet – e.g. Samsung Tab S3
• iPhone X’s – this type of device generates a lot of visual issues so had a separate test case as a type of device

What tests were carried out and what were the most common problem areas?

Testing for mobile devices is not the same as testing desktop applications, not only regarding the number of devices but also the methods of testing them and the focus areas.

Testing for mobile apps problem 1: Scaling

We focused on scaling and loading the ad. When the company logo or inscriptions were covered, the issue had high priority. The phone’s notch, for example on the iPhone X or wide aspect ratio devices such as Samsung Galaxy s8 +, was a big problem (e.g. the notch covering half of the name of the advertised place).

Testing for mobile apps problem 2: iPads

Tests on iPads generated a lot of errors, due to the fact that they have the ability to rotate 360 ​​degrees. For this reason, there were often problems with some missing parts of the images that were not fully displayed, lack of screen adjustment. This sometimes even resulted in the video stopping or the entire advertisement jamming. The problems were so frequent that iPad fixes on the dev side “ruined” the functionalities of other devices or were simply not feasible. After taking all of the conditions under consideration, especially the challenging time frames for our tests, we decided to lower the priority of iPad fixes.

Testing for mobile apps problem 3: Functional side

Functional tests were performed in various combinations. The most problematic area, it turned out, was “the background issue”. Going back to the app after putting it in the background made some of the mechanisms in the ad fail. Another thing was that functions were failing to shut down after switching them to the background – for example, the music from the video remained on. This was the most common thing that happened with the videos.

Testing for mobile apps problem 4: Open store

Going to the store or opening dedicated links were also very important. It was quite a challenge to check the ad specific availability in a given country because when an item is not available in your country, the Android (Google Play) store will simply display this information. However, it’s not as easy with the App Store. In Apple’s case, you will receive a blank page and no information about what’s happened. Which obviously is not what we want for our users to experience.

Testing for mobile apps problem 5: Performance

Performance tests were carried out, using one of the tools for testing mobile apps that I recommend – Charles Proxy – something which I will elaborate on later in the article. It helped to simulate the slowdown of the internet up to 512 kbps, but we most often used 3G which was enough to induce the performance problems we were analyzing.

Tools for testing mobile apps: Charles Proxy

So what is Charles Proxy? According to their website

“Charles is an HTTP proxy / HTTP monitor / Reverse Proxy that enables a developer to view all of the HTTP and SSL / HTTPS traffic between their machine and the Internet. This includes requests, responses and the HTTP headers (which contain the cookies and caching information).”

For me, Charles Proxy helps to monitor requests or to exchange the body of the request. For example, Xcode only has the simulator for iOS 9.3 but our tests had to be performed on iOS 9.3.5. So I had to rewrite the rule but to do that all I needed was to configure the file of the app and simply change the value in the body of the request

Charles Proxy: How to set it up?

To set up Charles Proxy in a way so it can read traffic between machines, all you need are a few, short steps:

1. To Download Charles Proxy you need to go to: https://www.charlesproxy.com/ If you want to try it out first, there’s a free trial for 30 days.
2. After installing and opening the app click on the tab Help -> SSL Proxying -> Instal Charles Certificate on a Mobile Device or Remote Browser

Then we have information about the name and port of the proxy server, there will also be some info about the installation of the certificate from the site chls.pro/ssl on your phone.

Where should you fill this data? It depends on the OS:

Android: Settings -> WiFi -> Manage network settings -> Show advanced options -> Proxy -> Manual -> Enter the Server Name and Port and click Save (example on Samsung Galaxy J3)

iOS: Settings -> Wifi (hold the given network) -> Configure Proxy -> Manually -> Enter Server and Port -> Click “Save” (iPhone 5s example) After entering and saving, go to chls.pro/ssl to download the certificate.

3. The last thing you need to do is enable our SSL proxying to the wild card

Now you can see your traffic. Also, the one thing I mentioned earlier was, namely: slowdown the internet to check the performance of our ad or app – you can find this feature on Throttle Settings in the Proxy tab.

Note: For Android 7+ you need to add the XML file (or ask the developer for it) to your application along with the configuration file that allows you to monitor the connection. You can also find more information on how to do this in the documentation: https://www.charlesproxy.com/documentation/using-charles/ssl-certificates/

The basic test coverage for mobile devices

Buying every possible device to test the hell out of every application is quite expensive and requires you to always be up to date with new devices. If you have a tight budget and you’re not so great with time either, you have to consider which devices and systems are the most important for the tests. To decide the priorities, think about what systems are the most popular and then simply – test them. In the case of iOS, most users update to the latest version. Over time, Apple ceases the support of old versions of applications. Interesting fact: at this point on iOS 10.0.2 (e.g. iPhone 6s) there is no application that would allow us to record the screen.

In the table below you can see the usage of all iOS OS version:

As far as Android is concerned, it is not as obvious as with the OS versions. There are still devices with Android 3 or 4 that people still use on a daily basis. On the plus side, versions of Android usually aren’t too different from each other. When there’s a bug it’s rarely found only in one version, usually, it occurs on most of the other systems too.

When we can’t use physical devices for our tests, we can use tools for testing mobile apps such as iOS Simulator or Android Emulator. From my experience, Apple’s Xcode Simulator is a very useful tool. In contrast to Apple, testing Android apps this way is way harder, and I would opt for using physical devices whenever possible. Why? I’ll explain in a second.

Simulator

Now, let me tell you about the simulators and emulators that are available with Xcode for iOS and Android Studio for Android.

Simulator for iOS

IOS simulators have many options and can really reflect the real devices. Often, the bugs you can find on the simulator match the ones on the physical device. The simulator for IOS, just like a physical device, includes a silent button, sound buttons, locks, and home button. We have many types of devices to choose from and almost every OS version that has ever been released is available.

Tip: To use Charles Proxy for reading traffic with our iOS simulator we need to enable ‘macOS Proxy’ and Install Certificate on iOS simulator.

So if you don’t want to invest a lot of money in buying all types of devices, Xcode simulators will cover most of the basic tests.

Below there is an example of options in a simulator and other types of devices supported in Xcode:

Android emulator

Unfortunately, Emulators in the Android Studio generate a lot more issues than the iOS Simulator. In my experience, a lot of issues found on emulators do not occur on a physical device, so in the case of Android, it is better to buy a device or use device farms.

If you want to try it for yourself – to set up a new emulator you need to choose a new hardware type and OS version.

In the Android emulator, there are a lot of options to use, e.g.: the health of the battery, location, and type of network. It never harms to try all these out and see how they work for you. And if you don’t fall in love with it, just like I didn’t, check out Best device farms for ios and android

Summary

When it comes to testing whole systems, or even a few applications, which for financial reasons and time frames are not profitable to automate, you have to consider what physical phone resources you have, what your base test coverage may be and what tests the crucial ones for the application are. When you figure these things out, testing will be automatically much more effective and cost-effective.

Thanks for reading the whole article! I hope it provided a helpful dose of knowledge and helps you to find your feet in an era of the growing popularity of mobile devices and their testing.

Check out some other articles on testing on our Blog:

• Web Applications: Why They Are Relevant And How To Test Them Manually
• Improving your project with SBT
• Reflections on starting Android project with Scala
• HOW WE MADE MANUAL TESTING OF 590 TIMEZONES IN 2 DAYS POSSIBLE
• Networking basics