To Build or Not to Build? – InApps 2022

When choosing a test automation strategy, there are always some common challenges to consider. For example, what framework are you using for test implementation? What should be automated and how will you ensure the testing process is stable and fast. This article will provide guidance on how to build a test automation infrastructure that will allow you to create a fast, secure automation process, and answer the Shakespeare-like question, “to build or not to build?” — a question every developer must answer before embarking on a new project.

Depending on your project size and the budget available to invest into infrastructure, you have two different options in shaping your infrastructure for QA automation: 1. paid cloud services such as Browserstack, Sauce Labs or Zebrunner, or 2. building in-house mobile device farms.

If you are a small startup and are interested in covering the full matrix of mobile devices for testing, you may be interested in cloud solutions with an average price of $150 per device monthly. However, when you reach the point of increased business growth and increasing automation coverage, you will need more dedicated devices to support your CI/CD pipeline needs. It is at this point that you should consider investing in your own mobile farm infrastructure, fully dedicated to your team and isolated into your VPN space.

Based on our experience with continuous engagement with QA infrastructures, we combined our knowledge with outside expert opinions, which resulted in the following conclusions on the advantages of building in-house mobile farms:

• Enables high capacity tests. A larger number of tests will run faster compared to those executed with cloud providers.
• No waiting in a queue and no limits for sessions.
• Fast farm infrastructure deployment (within a few hours).
• Enables integration with IoT devices.
• Secure solution for companies whose SecOPS restricts cloud usage.
• In the long term, cheaper than cloud providers.
• Open source documentation.
• Active community and support channels.

Mobile farms have some disadvantages as well:

• Expensive for startups and small businesses (in the short term).
• Maintenance expenses (device/component renewal, scheduled inspection, support specialist on-site).
• Keeping devices relevant (typically they are up to date for two years)

You’ve made the choice to build a mobile device farm; where should you start? First of all, you should analyze your project demand: the number of devices and actual models. In our experience, the surest way to define models is your analytics tools. Data analysis provides better understanding of your customers’ most popular mobile devices.

Chart shows most popular devices based on collected analytics.

Once you have this information, select the top 10, 20 or 50 to test according to your budget, and purchase them from your local provider. It is important to ensure you have the original warranty service support, so you can fix problems in a reasonable amount of time. We have found the most stable automation vendors to be the following: Samsung, LG and all generations of Google Nexus devices. Other makes tend to require a little more effort for maintenance. Also, do not forget to analyze all the same statistics for your Apple audience, selecting the most popular iPhones and iPads for your internal farm.

Now it is time to purchase servers and peripheral devices to develop your own cloud. It is important to construct two separate servers for Android and iOS since they use different technologies under the hood.

Let’s start with the Android server. We recommend building servers on an LGA2011-3 socket that supports powerful XEON processors; the configuration is the following:

• ASUS X99-A II motherboard
• SSD Intel 660p 2TB
• Intel Xeon E5-2630
• Corsair Vengeance LPX 2x8GB DDR4

A good power supply is also necessary, as is an external PCI card for the USB bus. The entire configuration is hosted on the Thermaltake full tower case (but you can choose your favorite).

• Thermaltake Core X71
• Startech 4-port PCI USB 3.0 card
• Corsair RM550x

When all the components are in place, then it’s time to consider peripheral devices: USB hubs and cables. We have tested a large variety of USB hubs from different vendors, and we have found the Pluggable USB 3.0 7 port hub in combination with Monoprice USB-A cables to be the most stable. This hub has an external 60W power adapter that allows it to charge mobile phones and tablets without any problems. Also, it supports stable USB data transfer, which helps to speed up your test automation tasks.

Now, let’s move on to an iOS server configuration. Options here are more limited; you can choose between a Mac Mini with custom configuration or a Mac Pro. From our experience, a custom Mac Mini is a better value for the price; it can easily support up to five iOS devices connected at the same time. It is important to use the latest generation of i5 CPUs with four cores onboard and 16 GB of RAM.

In summary, when it comes to server configuration, it is important to note that general Android servers with four USB hubs will be able to manage up to 30 devices connected, while iOS configuration is comfortable with four to five devices connected directly to the Mac Mini.

What about Network Configuration?

Another important topic to highlight is the network configuration. We highly recommend connecting all the servers to the Internet via an Ethernet cable and all mobile devices to WiFi. As access points, we recommend Ubiquiti UniFi [UAP-AC-PRO] — connecting at least two devices into one cluster. This cluster is able to balance network load, and thus provide a stable Internet connection.

When you host tens of mobile devices in the same place, you should bear in mind a very important question about security. As long as all devices are connected 24/7 via USB cables, their batteries are experiencing huge stress, and from time to time, you will have to perform technical checks and replace some accumulators. When this occurs, we highly recommend placing the whole setup onto a metal shelf with good air conditioning to prevent any chance of fire in the office or datacenter. In addition, we recommend using Xiaomi smart home sensors that will be triggered in case of smock or abnormal temperatures. In case of electricity bumps, APC Back-UPS 800VA, which allows you to secure your hardware from electrical damage, can prove quite valuable. If you are using a metal shelf, make sure that it is grounded!

My hope is that you now have the information you need to make informed decisions while choosing the right solution for your mobile automation infrastructure. Ultimately, it will be up to your scale and scope, as well as investing opportunities, to determine which variant best fits your particular needs.

Images provided by the author.