2013 State of Mobile Testing Report

XBOSoft Mobile Testing Report 2013

Mobile platforms are changing the way we do business.

Table of Contents
1 Overview 3 2 Mobile Development Human Resource 4 3 Mobile Development/Testing Processes 6 4 Mobile Testing 8 5 Mobile Organization 10 6 Conclusion 12

XBOSoft Mobile Testing Report 2013

1 Overview
XBOSoft is a software quality assurances services company, providing both consulting services as well as QA services. XBOSoft wanted to understand the state of mobile testing so that it could better provide services to that market. Mobile device internet usage will surpass usage by desk/laptops in the next year or two. Many who have successful websites realize that they cant wait any longer to have a mobile device version of their website or to offer a mobile app for their application. They are asking questions such as: How are other successful companies building development and QA teams for their mobile effort? What testing strategies do other companies use? What are other companies targeting for mobile platforms?

XBOSoft explores these issues (and more) in our survey to find out what people are doing to solve these problems. Purpose: to collect data on how different QA organizations conduct mobile testing and QA, to analyze against industry standards and best practices, to present this data so organizations can compare their own mobile QA efforts to survey results.

Over 830 million

Active users of mobile enterprise application s by 2016

Timeframe: The Survey was conducted between December 2012 and February 2013 by XBOSoft. Participants 100 survey respondents from 6 continents.

XBOSoft Mobile Testing Report 2013

2 Mobile Development Human Resources

Ninety four percent, 94%, of surveyed companies have or are planning to have a mobile presence. And over 68% are planning to grow their mobile development. Larger companies have the largest mobile presence, while the smaller companies are playing catch-up. With the importance of mobile growing, so is the need for QA teams to ensure high quality. Over 60% of respondents have dedicated mobile QA teams. Even though almost everyone, 82%, agreed that specific QA mobile training was needed, those that have dedicated mobile QA teams are three times more likely to give their QA teams specific mobile training. 51% versus 17% for the organizations whose teams that are not dedicated. The reasons for not having a dedicated mobile team are varied, but here are a few representative responses: We are planning to soon. We are too small now, maybe when we get bigger. We use contractors, third party testers. We dont have enough resources; we are understaffed and underfunded. Good testers can test mobile and desktop equally well.

security problems, and varying platforms.

In our opinion these challenges should cause the mobile tester-to-developer ratio to be higher than the traditional IT application. With the size of mobile development teams growing faster than the size of mobile QA teams the problem of shortchanging mobile QA will get worse. Mobile development teams are expecting to grow at 15% in 2013 while mobile QA teams are only expected to grow at 10% in the same year. Additionally, 34% of respondents are planning to decrease their QA staff relative to their development staff, while only 8% are planning to increase their QA staff relative to their development staff. 1. What is the size of your mobile development team (including third party resources)?

The ratio of the size of the surveyed mobile QA teams to the size of their mobile development teams is approximately 1-to-2. As a benchmark, as the number of users and the complexity of the software increase greatly, the ratio starts to approach 1-to-1, which is the ratio that Microsoft uses1. So, is the 1-to-2 ratio sufficient? It depends on the application, but mobile has many challenges that non-mobile either doesnt have or are greatly exacerbated by comparison, such as: network related performance problems, load problems,

2. Do you expect your mobile development team to be larger next year?

XBOSoft Mobile Testing Report 2013

3. Do you have a dedicated QA team for mobile?

Using a third party for part of the QA testing could help, but 75% dont use that option. However 100 % of those respondents that do use third party QA staff think that specific mobile test training is needed, while only 48% of those that dont use 3rd party testers saw the need for specific mobile test training. It seems that those that outsource do so with a more thorough understanding of what is required and how to make outsourcing work. 38% of the respondents gave their mobile QA staff specific training and thought that it was very helpful. However many felt that they dont know what their options are and would like help to upgrade their mobile QA staff.

4. How big is your mobile QA team (including third party resources)?

7. Have your QA testers received specific mobile test training and/or have sufficient experience in structured mobile testing?

5. Do you expect your mobile QA team to be larger next year?

8. Do you think specific mobile testing training is needed?

6. Do you use a third party for mobile QA testing?

XBOSoft Mobile Testing Report 2013

3 Mobile Development Human Resources

The mobile world, like the rest of the software development world, is going Agile. 85% have a production release every quarter or less and 43% every month or less. 9% of the respondents only develops and supports mobile software.

9. How often do you have a production release for your mobile software?

72% use an Agile development process for their mobile applications, which means production releases are numerous and often. Agile is a good fit for mobile development, which requires fast and quick delivery. Evolving devices, security issues, inadequate hardware demand a development process which can adapt quickly to these environmental changes. The advantages of Agile are the same for non-mobile as well as mobile, however with the greatly shortened development cycles, the advantages are more highlighted. 10. Do you use an Agile development process?

Devices will not always be readily available when you need them

XBOSoft Mobile Testing Report 2013

29% said that their development processes were different for mobile than regular software projects and about the same percentage said that their mobile QA processes were also different than their regular QA processes. Those that had dedicated mobile QA teams were more than three times more likely to have a different QA process, thus revealing a more thorough understanding of the needs of mobile testing. 11. If you develop other types of software, does your development process for mobile software differ from the development process of your other types of software?

While the overall QA methodology is the same for mobile and desktop development, there are differences which must be taken into account. A fat broadband vs. a thin mobile data pipe, many cell tower connections, less powerful devices, touch navigation, and being charged for data usage are just a few of the many differences between mobile and desktop which must be taken into account. 54% of those surveyed have different QA processes for mobile and they cite many reasons for the differences, such as: Different network environments (bad weather can cause problems!), Different skills are needed, Different mobile devices, Small screen UI problems, Large number of platforms to support, Very compressed development schedule, and Automation needs are very different in setup and effort.

Responses of those who did not have different processes: We will have to change our processes because of the number of post production defects that were found, We use TMap NEXT as a structured QA process and this also works for Mobile, We havent given it any thought, and The details are different, but testing strategy is the same.

XBOSoft Mobile Testing Report 2013

12. Do you have a different QA process for mobile?

Only 43% of the respondents do any performance testing. We believe this should be near 100%. Letting your customers find your performance problems can be very expensive. Example: Google found that a .5-1 second increase in page load time resulted in a 20% decrease in traffic and revenue. For an Agile environment, many contend that regression testing cant be done for each cycle unless you use automated testing. For an Agile team that has a production release each month, there is only so much time that can be spent on testing with such a short time frame, so often only new features are tested. Regression testing of all those features and functions which shouldnt have been affected is often neglected. Automation is the only way to solve this dilemma. 15. What type of platforms do you support?

13. Do you think a different QA process for mobile is necessary?

4 Mobile Testing
When adding mobile support to your website or application, there are three approaches. Make your desktop website mobile friendly, Develop a different design and implementation for mobile, or Develop an app to reside on the mobile device.

Android and iOS continue to dominate with both having over 80% of the developers support, which tracks with most other surveys. Windows has 41% support, which is surprising since Windows market share is still in the low single digits and 10% are even developing for Windows only to the exclusion of either iOS or Android. They must believe market research company IDCs forecast report 5 that Windows market share will rise to 20% in the next few years and it is hard to believe that that the Nokia-Microsoft alliance wont be successful. Interestingly amongst those surveyed Windows developers are half as likely to have a dedicated development or QA team as the other developers.

In all three approaches, the choices are numerous on what and how to test. 14.what type of mobile testing do you do ?

XBOSoft Mobile Testing Report 2013

16. How many different devices is your SW used on (approximately)? Please count one device that is used with 3 different OS versions as 3 devices.

18. Do you test how your application functions on different networks?

17. How many actual devices do you test on? Please count one device that is used with 3 different OS versions as 3 devices.

Almost 60% do not test their application on different mobile networks. Networks are one of the leading causes of performance degradation on mobile devices. Those that do not test the performance of their application on different networks are going to lose customers and wont know why. Mobile performance is drastically different on the different networks. It may work well on one type of network and not work at all on another type of network. 19. Do you test how your application(s) function in different countries?

Most mobile developers test with a small set of devices. Unfortunately, the number of devices on which your application will run is quite large. Even with testing on more than 24 devices (one company tested on 200 mobile devices), you are testing on only a small fraction of the devices that your mobile application will be used on. Since it is impossible to test on the full universe of devices that your application will be used on, you have to test smart and choose a minimum set of devices that gives you maximum coverage. This is not easy since that set of devices will change from quarter to quarter. This is where third party QA experts can really help. They have sets of devices for different types of application and are continually updating them. They can match you up with one of their predetermined sets of devices, and then modify and add to it based on your specific requirements.

20. Is your application used in different countries?

XBOSoft Mobile Testing Report 2013

10

63% of the respondents plan to have their application used in more than one country. Performance of networks in different counties varies greatly. Even if the network in the other country is the same as the network in your own country that you tested on, the performance can vary greatly. For example, the distance of the server from the user can be dramatically different which can change the performance for the end user. Tools can emulate the different network conditions around the world in a cost effective way. With two thirds of the respondents supporting multiple countries, not only performance and security are issues, but also localization is important. 65% of those mobile applications in multiple countries are not tested in those countries. For example, a word in English can have a very subtle negative connotation in another language.

21. How many employees does your organization have?

22. What industry do you work in?

5 Mobile Organization
QA managers and testers made up about two thirds of the respondents, 11% were in development and the rest were senior manager and consultants. Most industries were represented with software companies making up over 55% of the share. Government representatives were not included in this survey. None of the developers or development Managers saw any difference between processes, (both development and QA processes), for mobile and non-mobile development and QA, however 40% of the QA managers and testers saw a difference.

XBOSoft Mobile Testing Report 2013

11

23. What is your role in your organization?

25. Does your organization only support/develop mobile software?

24. Where are you located?

North America 49%, and Western Europe, 27%, have the largest share of respondents to the survey, with Latin America, Africa, Eastern Europe, East Asia, and South East Asia having very little representation. For our next survey, it would interesting to get more responses from the developing world since mobile has a far greater impact on the people of those countries. There are now over 3 billion mobile phones in these emerging markets, and for many, the mobile phone is their only access to the internet.

XBOSoft Mobile Testing Report 2013

12

6 Conclusion:
Based on our survey a representative mobile development organization: Has ten developers and five testers, Is more likely to expand their development staff before their expanding their QA staff, Does not use a third party for testing, Has not given their staff specific mobile QA training but thinks that they should, Is Agile with a development cycle of less than a quarter, Uses the same processes in QA and Development for mobile and non-mobile, Likely to only use Manual Functional and Usability Testing, Supports mainly Android and iOS, Tests on less than 8 devices, Does not test on different networks, Has an application or website that is used in multiple countries, and Is located in North America or Europe.

XBOSoft greatly appreciates the approximately 100 participants who filled out the survey. The 2013 State of Mobile Testing Survey Report can be downloaded at XBOSoft.com under Knowledge Center/ Software QA Testing Research. The interest in mobile is now ubiquitous. No organization can afford to ignore it and how it affects its profitability.

References and Further Reading

1. Microsoft Secrets, by Michael Cusumano, Richard Selby. 2. Mobile Performance Testing: Part 2 Testing the Server, xbosoft.com. December 2012. 3. Testing Performance of Mobile Apps - Part 1: How Fast Can Angry Birds Run? xbosoft.com, Oct. 2012. 4. http://glinden.blogspot.com/2006/11/marissamayer-at-web-20.html. 5. http://www.idc.com/getdoc.jsp?containerId=prU S23523812#.USxgbTBpB8E. 6. http://gigaom.com/2011/09/26/txteaglemobilize-2011/.

This is typical, but these are not necessarily best practices. The right ratio of testers-to-developers depends on so many factors, but with performance, security, localization, and automation not getting enough attention, more testing needs to be done in those areas. Most want more specific mobile testing training, which is good, but needs to be acted on. The differences between desktop and mobile testing needs to be emphasized and applied to the QA processes. More localization testing is needed.

XBOSoft Inc.
640 Rocca Ave. South San Francisco, CA 94080 +1 408-350-0508 www.xbosoft.com
Cover page photo by: Kooaba http://shortcutmedia.com/