COMMUNITY-DRIVEN QA

Six Ways a Virtual Testing Team Can Compliment Your In-House QA Efforts and Raise App Quality
WHITEPAPER: March, 2009

Table of Contents
The Current State of Managing Software QA...... Todays QA Landscape: Trying to Do More with Less..... The Evolution of Quality Assurance: A Quick Look Back. Crowdsourcing: A Complimentary Option for In-House QA Ways in Which Crowdsourcing Compliments In-House QA Teams .. 1. Maintain Control . 2. Testing Coverage.. 3. Increase App Quality............ 4. Tester Diversity .. 5. Efficiency and Ease of Use..... 6. Cost Containment... 2 2 2 3 4 4 4 5 5 5 6

Dig Deeper & Take Action 7 About uTest. 7

As soon as we started programming, we found to our surprise that it wasnt as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent finding mistakes in my own programs. Maurice Wilkes, Author, Memoirs of a Computer Pioneer

WHITEPAPER: Community-Driven QA

The Current State of Managing Software QA

Todays QA Landscape: Trying to Do More with Less More than ever before, testing teams are being stretched thin in just about every organization. QA leaders are being asked to keep their applications bug-free despite shrinking budgets and teams; despite tighter launch deadlines; and despite increased app and testing complexity. In many companies, top-down pressure from management to slash anything that isnt fully perceived as mission-critical (or fully understood) has brought a sharp eye on the QA department. And while testing managers and directors are trying to get their jobs done with smaller budgets and teams, theyre also trying to achieve testing coverage to match the complexity of their users and their apps. This applies to: Operating systems Browsers Plug-ins and anti-virus programs Geographic locations Languages And in the case of mobile applications, handset makers and wireless carriers

All of this is enough to make even the best QA leaders feel like the deck is stacked against them that all their efforts are merely delaying the inevitable crisis.

The Evolution of Quality Assurance: A Quick Look Back Of course, things werent always this tough for QA managers (not that its ever been easy). As desktop applications began migrating to the web as SaaS apps in the early 2000s, there was optimism in many QA departments. This was Testing is complex, but effective based on the premature assumption that web apps would use of exploratory techniques can dramatically reduce the complexity of the testing compliance help tame that complexity and matrix. If it works in one browser, it will work in any browser contribute to the production of was the hope of many testing managers. high quality software. And for many, there was a temporary lessening of the testing - James Whitaker efforts required due to cross-browser, cross-platform Testing Guru, Microsoft consistency of the web apps of those days: Simple HTML content, generated entirely server side, with very little client complexity and light usage of rich content or dynamic content. Yep, those were the days. That euphoria was short-lived, however. As application complexity has increased exponentially, the dimensions of the testing matrix would ultimately burden traditional QA departments. Customers continue to expect more; and so executives and sales & marketing teams continue to demand more, better and richer features. And after the product and engineering teams have managed to build these magical apps, its often the QA team thats left to hold things together.

WHITEPAPER: Community-Driven QA

The traditional solution to this has been to grow the QA department in a never-ending attempt to provide proper coverage for an inherently unmanageable testing matrix. However, given the tightening of budgets everywhere, testing coverage (and long-term app quality) often suffers. Crowdsourcing: A Complimentary Option for In-House QA In response to these mounting challenges, there is a rapidly growing trend of using community-based testing (aka: crowdsourcing). By leveraging a global community of professional testers, QA teams are doing the impossible: maintaining app quality, achieving broad testing coverage, meeting launch dates and working within tighter budgets. Heres how it works: 1. QA teams specify their testing requirements (by location, OS, browser, app type, etc.) 2. They upload their testing scripts and other documentation through a secure platform 3. Bugs are reported and classified by the community, and monitored by the in-house QA team (who can approve, reject or request more information from testers) 4. The QA team pays for each completed test cycle The net effect is that QA teams get instant access to a global community of professional testers via a platform that enables management of full QA cycles (or connects to The next logical step in the their in-house bug tracking system) all without having to commit to long-term contracts. This testing community can function as an on- evolution of testing is for vendors to provide testers and demand extension to your in-house QA team. this is exactly the era weve If this sounds familiar, thats because the crowdsourcing concept entered with crowdsourcing. was inspired by the open source movement. But unlike open source development, the QA team that uses crowdsourced testers - James Whitaker has direct control of test plans, scripts, prioritization and timelines. Testing Guru, Microsoft Also, while the open source model of distributed activity often depends upon altruism and volunteer work, this is not the case with crowdsourced testing. These professional testers are paid for their active and effective participation. With this in mind, let us now look at six ways in which crowdsourcing can provide an ideal compliment to your internal QA department.

WHITEPAPER: Community-Driven QA

Ways in Which Crowdsourcing Compliments In-House QA Teams

1. Maintain Control: One of the greatest concerns with traditional outsourcing is a loss of control, and thus, quality. But with crowdsourcing, the in-house QA leader maintains complete control and oversight of the entire test cycle. With the uTest community, we In fact, the companies that use crowdsourcing understand that effective, detailed communication and project management are critical to successfully managing the crowd. Said differently, you are the brains calling the shots; the community serves as your hands, feet and eyes, reporting their legwork back to you for review and approval. get a complete picture of how our software works, on different environments, systems and locations. - Cosma Dinca QA Manager, IMinent Software

And so while you dont have to micro-manage the crowd in terms of tactical execution and idea generation, strong management enables the processes, plans and deadlines to remain firm. The crowdsourcing model also provides the in-house QA team with the tools they need to oversee the entire testing cycle: Share your application via a secure platform Specify your unique testing requirements to custom-build your virtual QA team Load test scripts or QA scope requirements Specify timelines and other requirements Review bug reports Integrate with in-house bug-tracking systems like Jira, Rally and Bugzilla

Its worth noting that there are many sites that connect employers with freelance workers, but such sites do not provide structured community-building they are simply an online employment agency that helps companies to find the right individual freelancer. Crowdsourcing, therefore, is unique in that it encourages strong management and participation from the companys in-house QA team, rather than replacing it. 2. Testing Coverage: Unless your app is designed for a simple and homogenous audience thats identical to your in-house QA team, its likely that you have gaps in your testing coverage. Yet to fill these voids - whether they be browser, OS, location, language or device would be impractical, expensive and a huge waste of time. This is no longer the case. Crowdsourcing, having already assembled a global testing community, enables QA teams of all sizes to target very specific users (like Mac-users who live in Singapore) for specific assignments. Previously, even the largest software companies could not afford to have their products universally tested. Now, even early-phase start-ups cant afford not to.

WHITEPAPER: Community-Driven QA

3. Increase App Quality: For some strange reason, customers expect applications to be error-free and fully functional from day one. Good enough is no longer an option, nor is beta testing as a substitute for thorough QA . Whereas beta used to mean An early release of a software version, before the bugs have been worked out, it is now understood quite differently. The meaning now is almost entirely business-driven: A stable, first release of an application, before the precise business model has been set in stone. Simply put: web and mobile app companies often do not have the luxury of making two releases to customers. And so from a quality perspective, those considering using the also consider that there are two types of currency in an online community: money and reputation. Of course, money talks, but the transparency of crowdsourcing motivates people to build strong reputations. For these testers, it is the only way to earn the respect of their peers. For a QA manager, this helps to ensure that the people testing their applications have something at stake in the relationship. As with your internal QA team, motivation is the key to success. community as a QA supplement must When it comes to real-world application testing, the crowds in Camden Market are preferable to the elite residents of Buckingham Palace. Diversity yields world-class QA coverage.

- Doron Reuveni 4. Tester Diversity: CEO, uTest Not only does the internal QA leader maintain authority over release cycles, they also exercise crowd control (pardon the pun) in regards to which community members will test their applications. As the scope of your applications evolves, your virtual testing team can change with it. An ideal testing community will be comprised of a wide range of users across: Geographic locations Languages spoken Technical platforms (hardware, operating system, browser, anti-virus, etc.) App type expertise (mobile vs. web vs. desktop, specific industry knowledge, etc.) QA experience levels

For most companies, it is prohibitively expensive to simulate solely within an internal QA department. Also, no matter how intelligent, driven and talented your in-house QA team may be (and weve met some great ones), they are often less effective at generating product ideas and objective feedback than you may think. By crowdsourcing, you can avoid (or at least off-set) the group-think that often plagues so many internal QA teams. 5. Efficiency and Ease of Use: Since it is an on-demand solution, crowdsourcing helps alleviate the pains associated with peak release times (this is especially true of those adhering to Agile). Leveraging a community of testers exactly when you need them can ensure that resources are neither over-worked - nor underutilized - at any point in the product development lifecycle.

WHITEPAPER: Community-Driven QA

The efficiency of any QA department is often a good indication of the efficiency of the business as a whole. And so by smoothing over peak release times to achieve faster time-to-market, everyone reaps the benefits. Vendors are free to pursue new revenue opportunities; the sales team enjoys a higherquality product to pitch and technical support is no longer overburdened with a faulty product. In short, an efficient QA process lends itself to success in other areas. Just dont expect a trophy anytime soon. 6. Cost Containment: The net impact of a user discovering a bug in your software is high for obvious reasons, including customer dissatisfaction, impact on follow-up sales and support costs, to name a few. This impact has been studied greatly, most notably by Basili Boehm as published by IEEE; and by Littlewood as published in his Software Reliability Achievement and Assessment. It turns out the overall cost of fixing bugs that have been discovered by users after launch can be 1,000 times greater than the cost of fixing bugs that are discovered earlier in the process! Dont believe us, do you? Click here for a TechTarget article that highlights these studies. And in todays world of pay-as-you-go products such as mobile apps or any SaaS web app any software bugs that make it in front of your users will immediately decrease usage, dragging revenue down with it. Even for standard license-based applications, long evaluation cycles mean bugs often are found before the purchase decision, and can hamper your reputation and sales. What Would It Cost To Add An Additional QA Analyst? Base Salary: $65,455 Social Security: $5,192 401k: $2,443 Disability: $679 Healthcare: $5,722 Time Off: $8,876 Total: $88,368 Source: Salary.com March, 2009

But when in-house QA teams leverage a global testing community, both parties benefit. The QA team gets on-demand access to testers with the right skills, location and technical profile. Testers get to earn extra money for doing what theyre good at, and build a reputation that helps them win more future testing projects. In short, crowdsourcing is a resource that progressive QA leaders are utilizing to achieve maximum testing coverage, which decreases the likelihood of bugs making it to launch, and helps them get their apps to market faster.

WHITEPAPER: Community-Driven QA

Dig Deeper & Take Action

Want to capture the power of crowds to launch higher quality apps faster? Design: Need help with UI or graphic design? o Guru www.guru.com o CrowdSpring www.crowdspring.com Development: Want to get your app built with the help of the crowd? o TopCoder www.topcoder.com o Rent-A-Coder www.rentacoder.com Testing & Usability: Want to make sure your new app or version is ready for primetime? o uTest www.utest.com

Other inspirations for this paper and valuable resources about crowdsourcing: Wired Magazine Crowdsourcing article by Jeff Howe Jeff Howe Crowdsourcing blog ReadWriteWeb Crowdsourcing review BNet Crowdsourcing review Crowdsourcing Links Crowd-driven companies & news

About uTest
Headquartered near Boston, uTest is the world's largest marketplace for software testing services. The company provides real-world QA services through its community of 25,000+ professional testers from over 150 countries around the world. Hundreds of companies - from web start-ups to enterprise software firms - have signed up to get their apps tested by the uTest community. uTest enables companies to launch higher quality products; get their desktop, web and mobile applications to market faster; and control the cost of testing. Customers specify their QA requirements for tester experience, location, language, OS and browser, and uTest selects the right testers for each project. And because uTest is on-demand, companies pay for completed test cycles. A brief online demo is available at www.utest.com/demo. uTest can be contacted at: uTest, Inc. 153 Cordaville Road Southborough, MA 01772 p: 1.800.445.3914 e: info@utest.com w: www.utest.com

WHITEPAPER: Community-Driven QA