Mobile Testing Report 2013

2013 State of Mobile Testing Report

XBOSoft – Mobile Testing Report 2013 2

Table of Contents

1 Overview ······························································································· 3 2 Mobile Development Human Resource ·················································· 4 3 Mobile Development/Testing Processes ················································· 6 4 Mobile Testing ······················································································ 8 5 Mobile Organization ············································································ 10

6 Conclusion ··························································································· 12

Mobile platforms are changing the way we do business.


1 Overview

XBOSoft is a software quality assurances services company, providing both consulting services as well as QA services. XBOSoft wanted to understand the state of mobile testing so that it could better provide services to that market. Mobile device internet usage will surpass usage by desk/laptops in the next year or two. Many who have successful websites realize that they can’t wait any longer to have a mobile device version of their website or to offer a mobile app for their application. They are asking questions such as:

How are other successful companies building

development and QA teams for their mobile

effort?

What testing strategies do other companies

use?

What are other companies targeting for mobile

platforms?

XBOSoft explores these issues (and more) in our survey to find out what people are doing to solve these problems. Purpose:

to collect data on how different QA

organizations conduct mobile testing and QA,

to analyze against industry standards and best

practices,

to present this data so organizations can

compare their own mobile QA efforts to survey

results.

Timeframe:

The Survey was conducted between December

2012 and February 2013 by XBOSoft.

Participants

100 survey respondents from 6 continents.

Over 830 million Active users of mobile enterprise application s

by 2016


2 Mobile Development Human Resources Ninety four percent, 94%, of surveyed companies have or are planning to have a mobile presence. And over 68% are planning to grow their mobile development. Larger companies have the largest mobile presence, while the smaller companies are playing catch-up. With the importance of mobile growing, so is the need for QA teams to ensure high quality. Over 60% of respondents have dedicated mobile QA teams. Even though almost everyone, 82%, agreed that specific QA mobile training was needed, those that have dedicated mobile QA teams are three times more likely to give their QA teams specific mobile training. 51% versus 17% for the organizations whose teams that are not dedicated. The reasons for not having a dedicated mobile team are varied, but here are a few representative responses:

We are planning to soon.

We are too small now, maybe when we get

bigger.

We use contractors, third party testers.

We don’t have enough resources; we are

understaffed and underfunded.

Good testers can test mobile and desktop

equally well.

The ratio of the size of the surveyed mobile QA teams to the size of their mobile development teams is approximately 1-to-2. As a benchmark, as the number of users and the complexity of the software increase greatly, the ratio starts to approach 1-to-1, which is the ratio that Microsoft uses1. So, is the 1-to-2 ratio sufficient? It depends on the application, but mobile has many challenges that non-mobile either doesn’t have or are greatly exacerbated by comparison, such as:

network related performance problems,

load problems,

security problems, and

varying platforms.

In our opinion these challenges should cause the mobile tester-to-developer ratio to be higher than the traditional IT application. With the size of mobile development teams growing faster than the size of mobile QA teams the problem of shortchanging mobile QA will get worse. Mobile development teams are expecting to grow at 15% in 2013 while mobile QA teams are only expected to grow at 10% in the same year. Additionally, 34% of respondents are planning to decrease their QA staff relative to their development staff, while only 8% are planning to increase their QA staff relative to their development staff. 1. What is the size of your mobile development team (including third party resources)?

2. Do you expect your mobile development team to be larger next year?


3. Do you have a dedicated QA team for mobile?

4. How big is your mobile QA team (including third party resources)?

5. Do you expect your mobile QA team to be larger next year?

6. Do you use a third party for mobile QA testing?

Using a third party for part of the QA testing could help, but 75% don’t use that option. However 100 % of those respondents that do use third party QA staff think that specific mobile test training is needed, while only 48% of those that don’t use 3rd party testers saw the need for specific mobile test training. It seems that those that outsource do so with a more thorough understanding of what is required and how to make outsourcing work. 38% of the respondents gave their mobile QA staff specific training and thought that it was very helpful. However many felt that they don’t know what their options are and would like help to upgrade their mobile QA staff. 7. Have your QA testers received specific mobile test training and/or have sufficient experience in structured mobile testing?

8. Do you think specific mobile testing training is needed?


Devices will not always be readily available when you need them

3 Mobile Development Human Resources

The mobile world, like the rest of the software development world, is going Agile. 85% have a production release every quarter or less and 43% every month or less. 9% of the respondents only develops and supports mobile software. 9. How often do you have a production release for your mobile software?

72% use an Agile development process for their mobile applications, which means production releases are numerous and often. Agile is a good fit for mobile development, which requires fast and quick delivery. Evolving devices, security issues, inadequate hardware demand a development process which can adapt quickly to these environmental changes. The advantages of Agile are the same for non-mobile as well as mobile, however with the greatly shortened development cycles, the advantages are more highlighted. 10. Do you use an Agile development process?

http://www.surveymonkey.com/MySurvey_Responses.aspx?sm=bWhtakqFKmO16lawJpNHjgQtkdb38AXtjuFnf8x453vQe3HiZL7aOAdkZZ91NJwi


29% said that their development processes were different for mobile than regular software projects and about the same percentage said that their mobile QA processes were also different than their regular QA processes. Those that had dedicated mobile QA teams were more than three times more likely to have a different QA process, thus revealing a more thorough understanding of the needs of mobile testing.

11. If you develop other types of software, does your development process for mobile software differ from the development process of your other types of software?

While the overall QA methodology is the same for mobile and desktop development, there are differences which must be taken into account. A fat broadband vs. a thin mobile data pipe, many cell tower connections, less powerful devices, touch navigation, and being charged for data usage are just a few of the many differences between mobile and desktop which must be taken into account. 54% of those surveyed have different QA processes for mobile and they cite many reasons for the differences, such as:

Different network environments (bad weather can cause problems!),

Different skills are needed,

Different mobile devices,

Small screen UI problems,

Large number of platforms to support,

Very compressed development schedule, and

Automation needs are very different in setup and effort.

Responses of those who did not have different processes:

We will have to change our processes because of the number of post

production defects that were found,

We use TMap NEXT as a structured QA process and this also works for Mobile,

We haven’t given it any thought, and

The details are different, but testing strategy is the same.


12. Do you have a different QA process for mobile?

13. Do you think a different QA process for mobile is necessary?

4 Mobile Testing

When adding mobile support to your website or application, there are three approaches.

Make your desktop website mobile friendly,

Develop a different design and implementation

for mobile, or

Develop an app to reside on the mobile device.

In all three approaches, the choices are numerous on what and how to test. 14.what type of mobile testing do you do ?

1

Only 43% of the respondents do any performance testing. We believe this should be near 100%. Letting your customers find your performance problems can be very expensive. Example: Google found that a .5-1 second increase in page load time resulted in a 20% decrease in traffic and revenue. For an Agile environment, many contend that regression testing can’t be done for each cycle unless you use automated testing. For an Agile team that has a production release each month, there is only so much time that can be spent on testing with such a short time frame, so often only new features are tested. Regression testing of all those features and functions which shouldn’t have been affected is often neglected. Automation is the only way to solve this dilemma. 15. What type of platforms do you support?

Android and iOS continue to dominate with both having over 80% of the developer’s support, which tracks with most other surveys.

Windows has 41% support, which is surprising since Windows market share is still in the low single digits and 10% are even developing for Windows only to the exclusion of either iOS or Android. They must believe market research company IDC’s forecast report 5 that Window’s market share will rise to 20% in the next few years and it is hard to believe that that the Nokia-Microsoft alliance won’t be successful. Interestingly amongst those surveyed Window’s developers are half as likely to have a dedicated development or QA team as the other developers.


16. How many different devices is your SW used on (approximately)? Please count one device that is used with 3 different OS versions as 3 devices.

17. How many actual devices do you test on? Please count one device that is used with 3 different OS versions as 3 devices.

Most mobile developers test with a small set of devices. Unfortunately, the number of devices on which your application will run is quite large. Even with testing on more than 24 devices (one company tested on 200 mobile devices), you are testing on only a small fraction of the devices that your mobile application will be used on. Since it is impossible to test on the full universe of devices that your application will be used on, you have to test smart and choose a minimum set of devices that gives you maximum coverage. This is not easy since that set of devices will change from quarter to quarter. This is where third party QA experts can really help. They have sets of devices for different types of application and are continually updating them. They can match you up with one of their pre-determined sets of devices, and then modify and add to it based on your specific requirements.

18. Do you test how your application functions on different networks?

Almost 60% do not test their application on different mobile networks. Networks are one of the leading causes of performance degradation on mobile devices. Those that do not test the performance of their application on different networks are going to lose customers and won’t know why. Mobile performance is drastically different on the different networks. It may work well on one type of network and not work at all on another type of network. 19. Do you test how your application(s) function in different countries?

20. Is your application used in different countries?


63% of the respondents plan to have their application used in more than one country. Performance of networks in different counties varies greatly. Even if the network in the other country is the same as the network in your own country that you tested on, the performance can vary greatly. For example, the distance of the server from the user can be dramatically different which can change the performance for the end user. Tools can emulate the different network conditions around the world in a cost effective way. With two thirds of the respondents supporting multiple countries, not only performance and security are issues, but also localization is important. 65% of those mobile applications in multiple countries are not tested in those countries. For example, a word in English can have a very subtle negative connotation in another language.

5 Mobile Organization

QA managers and testers made up about two thirds of the respondents, 11% were in development and the rest were senior manager and consultants. Most industries were represented with software companies making up over 55% of the share. Government representatives were not included in this survey. None of the developers or development Managers saw any difference between processes, (both development and QA processes), for mobile and non-mobile development and QA, however 40% of the QA managers and testers saw a difference.

21. How many employees does your organization have?

22. What industry do you work in?


23. What is your role in your organization?

24. Where are you located?

North America 49%, and Western Europe, 27%, have the largest share of respondents to the survey, with Latin America, Africa, Eastern Europe, East Asia, and South East Asia having very little representation. For our next survey, it would interesting to get more responses from the developing world since mobile has a far greater impact on the people of those countries. There are now over 3 billion mobile phones in these emerging markets, and for many, the mobile phone is their only access to the internet.

25. Does your organization only support/develop mobile software?


6 Conclusion:

Based on our survey a representative mobile development organization:

Has ten developers and five testers,

Is more likely to expand their development staff

before their expanding their QA staff,

Does not use a third party for testing,

Has not given their staff specific mobile QA

training but thinks that they should,

Is Agile with a development cycle of less than a

quarter,

Uses the same processes in QA and

Development for mobile and non-mobile,

Likely to only use Manual Functional and

Usability Testing,

Supports mainly Android and iOS,

Tests on less than 8 devices,

Does not test on different networks,

Has an application or website that is used in

multiple countries, and

Is located in North America or Europe.

This is typical, but these are not necessarily best practices. The right ratio of testers-to-developers depends on so many factors, but with performance, security, localization, and automation not getting enough attention, more testing needs to be done in those areas. Most want more specific mobile testing training, which is good, but needs to be acted on. The differences between desktop and mobile testing needs to be emphasized and applied to the QA processes. More localization testing is needed.

XBOSoft greatly appreciates the approximately 100 participants who filled out the survey. The 2013 State of Mobile Testing Survey Report can be downloaded at XBOSoft.com under Knowledge Center/ Software QA – Testing Research. The interest in mobile is now ubiquitous. No organization can afford to ignore it and how it affects its profitability.

References and Further Reading

1. Microsoft Secrets, by Michael Cusumano, Richard Selby.

2. Mobile Performance Testing: Part 2 – Testing the Server, xbosoft.com. December 2012.

3. Testing Performance of Mobile Apps - Part 1: How Fast Can Angry Birds Run? xbosoft.com, Oct. 2012.

4. http://glinden.blogspot.com/2006/11/marissa-

mayer-at-web-20.html.

5. http://www.idc.com/getdoc.jsp?containerId=prU

S23523812#.USxgbTBpB8E.

6. http://gigaom.com/2011/09/26/txteagle-

mobilize-2011/.

XBOSoft Inc. 640 Rocca Ave.

South San Francisco, CA 94080

+1 408-350-0508 www.xbosoft.com

Cover page photo by: Kooaba http://shortcutmedia.com/

http://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html

http://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html

http://www.idc.com/getdoc.jsp?containerId=prUS23523812#.USxgbTBpB8E

http://www.idc.com/getdoc.jsp?containerId=prUS23523812#.USxgbTBpB8E

http://gigaom.com/2011/09/26/txteagle-mobilize-2011/

http://gigaom.com/2011/09/26/txteagle-mobilize-2011/

Documents

Mobile Testing Report 2013