PT Issue21

TE TER
Essential for software testers

4 /5
SUBSCRIBE
Its FREE
for testers
Continuous testing and delivery
June 2013
v2.0
number 21
THIS ISSUE OF PROFESSIONAL TESTER IS SPONSORED BY
Including articles by: Bogdan Bereza VictO Roy de Kleijn Polteq Mark Lehky Jessica Schiffmann Prism Informatics Eric M.S.P. Veith Wilhelm Bchner Hochschule/TU Bergakademie Freiberg
Stopping stopping
In software development, continuous is often used wrongly to mean very frequent. Many developers like frequent release lifecycles because they prefer making mistakes and fixing them over trying harder not to make them in the first place. Everyone who writes code knows how much fun that is. And work should be fun, shouldnt it? Yes, if you bear the consequences yourself. But developers typically dont: it is others who suffer, often badly.
From the editor
try to dress deliberate mistake-making up as something positive by calling it iterative and so on. But rapid change for business reasons is a reality testing must embrace, and that brings us to the correct meaning of continuous: without stopping. In software development, that means eliminating the need for human intervention by automating as much as possible. This issue is about how testing can achieve that. We are very grateful to its sponsor HP for making it possible.
Contact
Edward Bishop editor@professionaltester.com Niels Valkering ops@professionaltester.com Christiaan van Heest art@professionaltester.com Rikkert van Erp advertise@professionaltester.com Jerome H. Mol publisher@professionaltester.com subscribe@professionaltester.com
Editor
Managing Director Art Director Sales
It has been said that change is the enemy of quality. In this issue we see that, increasingly, frequent software change is the friend and ally of business. So the first statement needs qualifying: change due to mistakes is the enemy of quality and must be stamped out, even more so because developers
Edward Bishop
Editor
Publisher
Subscriptions
IN THIS ISSUE
4 We can no longer beat agile so must join it 8 Autometrics
Edward Bishop examines the impact of mobile apps on testing
Contributors to this issue

Jessica Schiffmann Eric M. S. P. Veith Roy de Kleijn Bogdan Bereza Mark Lehky Professional Tester is published by Professional Tester Inc We aim to promote editorial independence and free debate: views expressed by contributors are not necessarily those of the editor nor of the proprietors. Professional Tester Inc 2013 All rights reserved. No part of this publication may be reproduced in any form without prior written permission. Professional Tester is a trademark of Professional Tester Inc.
Jessica Schiffmann and Eric M. S. P. Veith on extending test automation and its measurement in SAP
12 Behave yourself 18 Cross purposes 21 Flash light
BDD for testers with Roy de Kleijn
Bogdan Bereza discusses centralizing mobile app test automation
Mark Lehkys comprehensive guide to automating a technology often perceived as hard to automate
Visit professionaltester.com for the latest news and commentary
PT - June 2013 - professionaltester.com
We can no longer beat agile so must join it

by Edward Bishop
How to stop testing limiting release frequency
When business people say agile they mean agile business: in other words, a business that can change its operations rapidly. In a doomed attempt to avoid confusion, the term nimble has been coined to mean the same thing. Business decision-makers are employed for their ability to make good strategic decisions. But strategy becomes more difficult as things change faster. So successful strategists need their decisions to be applied as immediately as possible. They tend to think of themselves as racing drivers. Whether in F1 (blue chip), go-kart (startup) or anywhere between, they want their car to be as manuvrable as possible, because that will give their driving skills the best chance of success and make them less likely to crash. The majority of HPs clients are actively seeking greater business agility. Is agile development the best way to deliver an agile business? A lot of people are convinced it is, or isnt, but the truth is no-one knows. Like so much in testing, it depends. Much more importantly, soon no-one will care. Mobile apps have not just changed the game, theyve ripped it up and started it again. They are more immediately-business-critical than anything that has existed before. To business strategists focused on sales at any level, the mobile app is their racing car. It must respond to their controls. That requires frequent change to production and that requires continuous delivery. Web changed mainframe development because it depended upon it, yet became as important as it. In the same way, mobile is changing all other development. Consider a mobile app developer. He or she must be fluent in Objective-C/iOS
PT editor Edward Bishop assesses the impact of mobile on testing
Testers have been arguing about agile development and accelerating iteration frequency, with legitimate reason, for a long time. But the growth of mobile apps has now rendered resistance futile. Testing must eliminate testing cycles and become continuous or development cycles will eliminate testing. The agile business When developers say agile they usually mean agile development and are talking about the working practices of programmers: finding the most efficient and productive ways to spend their time. Testers use the same definition, and testers working with agile developers are concerned with helping to make the developers work efficient, productive and safe. Some developers think testing can do that, some dont. Most are somewhere in the middle and want to use testing, but in times and ways of their own choice.
and/or Java/Android and expert in design of responsive/adaptive mobile GUIs. Can you find anyone with those skills who knows or cares about any sequential development lifecycle or independent quality control process? There is no such animal. But consider also a mobile app with many users in which a severe defect is detected in production. The defect is fixed and a new version released. But what about users who dont update immediately? They will probably experience and be influenced by failure. Worse, what if that failure causes wider failure, for example of data integrity, that impacts more users, damaging the business? So shall we prevent use of the defective version(s)? Display a message telling users they cannot use their current version and must update? Deliberately cripple some functionality? Hold some transactions in suspense and start an emergency project aimed at consolidating them? How agile is a business that does any of these? Obviously it is better to avoid them by preventing the defect entering production. Testing is the only way to achieve that. Is agile testing the best way to support agile development? This depends what agile testing means, and I can find no useful definition. Whatever it means, agile development must not require testing itself to change. Although continuous delivery suits agile development and requires testing to be integrated better and to be continuous, good governance requires testing to remain formal and independent. Many testers use HP Quality Center to deliver superb testing practices which were originally designed for application to sequential development practices. The fact that the basis of that design is now obsolete does not render the practices obsolete. They remain as valid and important as ever. They must be done, but they must be done more rapidly and with more granularity, so their risk-reducing power can be applied to the frequent releases to production demanded by mobile apps.
Agile and business continuity One of HPs clients is a multinational retail franchise business. It retails products and services at thousands of outlets. The average retail price per unit is in the hundreds of pounds. The franchisor supplies applications to the franchisee. These are diverse, ranging from POS and retail management to control of onsite manufacturing machinery. One of the largest is used for supply chain management, a critical business function with a close relationship to revenue. Because it knows all projects will soon be agile, the business has started its agile journey by converting this key project to agile development. The first thing an agile project must have is a viable plan B so that plan A can take more risk. The SCM project took the considerable risk of outsourcing most of the development work to two suppliers, both in India. It paid off as both suppliers quickly proved superb. But what if that should change?
If a business depends on a datacentre, its business continuity plan needs to address that. If a project depends on suppliers, in exactly the same way it must consider supplier failure: what if one of them suffers disruption? If you are not in a position to replace them quickly, your project inherits all the business continuity risk of its suppliers and that of their suppliers. HP Application Lifecycle Management records all the information the replacement supplier needs in nearly-real time, and makes it quick and easy to transfer. For that reason alone, ALM is essential to business agility. Testing in the agile project Periodic releases of information about what changed in the last iteration and what change is planned for the next are not sufficient. Testing needs to know about change to the product immediately it occurs, so testing can adapt immediately and is always of the current product, achieving the greatest effectiveness
Figure 1: Application Lifecycle Intelligence
because that would be dangerous and/ or expensive. HPs Service Virtualization keeps testing continuous by enabling testers to create and maintain the interfaces their tests need easily and rapidly. This divergence of test from development environment creates a danger. What if the behaviour of an external service changes? In PTs April 2013 issue Huw Price and Llyr Jones explained why it is vital to carry out frequent empirical testing to detect that: it is untraceable, invalidates testing and is likely to cause regression failure. To ensure reliable detection HP has integrated QuickTest Professional with Service Test, adding very easy code-free API testing to the same QTP IDE used for GUI testing. This fully-automated Unified Functional Testing is assimilated seamlessly to the CI workflow. When change is detected, the test environment is corrected easily using Service Virtualizations visually intuitive interface (figure 3) so it is ready for use to test resulting change to code immediately it takes place. Contributors to PTs February 2011 issue noted that even in the most highly automated software organization some manual test execution is needed. HP Sprinter accelerates this by semiautomating it, eliminating human error in data entry, replicating execution across multiple environments and capturing every detail of what is done and its results directly into ALM. Closing the dev-test-deploy loop using automation means it can be made smaller: frequency of releases to production neither impacts nor is limited by testing. Effective, continuous testing that is integral to continuous delivery maximizes business agility
Figure 2: Quality Center interaction within Eclipse
Figure 3: editing a Service Virtualization definition
possible in the time and with the resources available. HPs Application Lifecycle Intelligence links ALM with build servers and software configuration management. That allows testing to see how many lines of code went into a specific change, who wrote them, which build that change is in, what proportion of relevant unit tests the changed components have passed and exactly what was done, enabling strategic allocation of retesting and regression testing effort (see figure 1). Developers interact with Quality Center within their IDE (figure 2), seeing easily all test results relevant to their current work, and knowing that changes they make and unit tests they execute will influence those results in nearly-real time. This helps to bring developers onboard with testing and its formal defect management lifecycle. In October 2012 Professional Tester considered the interface of testing to
DevOps, a trend closely related to CI and similarly driven by the need for business agility. More frequent app releases are increasingly likely to contain operational as well as code change, therefore continuous testing must be also of the current environment. HPs Lab Management Automation allows deployment and configuration of test environments to be controlled by the same schedule that executes tests. Those environments need to be identical to their production equivalents except, in the case of composite applications such as most mobile apps, in one respect: they cannot be interfaced to external services
To learn more about the HP products mentioned in this article please visit http://hp.com/go/alm
Test Studio
Easily record automated tests for your modern HTML5 apps
Test the reliability of your rich, interactive JavaScript apps with just a few clicks. Benefit from built-in translators for the new HTML5 controls, cross-browser support, JavaScript event handling, and codeless test automation of multimedia elements.
www.telerik.com/html5-testing
Autometrics
by Jessica Schiffmann and Eric M.S.P. Veith
Better automation means automating more, including measurement
Top eCATT Our automation tool of choice is eCATT (extended computer-aided test tool). This achieves a high rate of success in identifying SAP GUI objects and can also access SAP at the database level which thirdparty tools typically cannot. Its weakness is that it is limited to the SAP environment. Our example here has no web or other external interfaces: otherwise it might be necessary to use additional tools, or even make a different choice of main tool. Like most capture-replay test automation tools eCATT records manual user interaction with the GUI of the application under test (see figure 1) as a script (figure 2). Here, the section UserChangedState contains the detail of what the user did to an object or objects: for example making a choice from a drop-down control or selecting a radio button. The command GETGUI is used to read the state of objects for validation. Other commands enable internal validation, for example of database table contents. The script is extended and developed by adding further commands from Inline ABAP, a subset of SAPs Advanced Business Application Programming language, and parameterized, configured and executed using the transaction seCATT (figure 3). Other transactions provide detailed logging of execution and outcomes at GUI, database and system levels. Deciding which tests to automate In the early days of test automation standard advice was automate first or next those tests which will be executed the greatest number of times. This is logically correct, but in this form is applicable only to sequential development lifecycles with formal test planning. In iterative/incremental lifecycles with rapid change the question becomes meaningless and impossible to answer. All valid tests are essential to
Ideas to improve test automation of SAP and other applications from Jessica Schiffmann and Eric M.S.P. Veith
Applications are changing faster than ever. Advanced IDEs, ultra-flexible enterprise platforms such as SAP, continuous integration, automated deployment, cloud and sequential lifecycles all lead to more frequent releases. Test automation needs to keep up, but is still dogged by the same problems it always has been. For example: it takes a lot of effort to vary alreadyautomated tests in order to sustain defect-finding potential there is a tendency to create lab tests that dont simulate realistic actual use its difficult to demonstrate its value in a way that will help sustain management support. In this article we will offer ideas for improving these areas, especially the last, at the same time relating how functional test automation of SAP applications can be achieved.
preventing regression and must be executed against every release. On the other hand any test can immediately become obsolete due to product change. Instead, consider the parameterization (called test data in eCATT). How many lines will be required in the data file? The tests where this number is largest are the ones to automate first/next, because doing so will reduce the need for manual execution fastest. In terms of manual effort, the same test script executed with n parameterizations is in fact n tests. Bearing this fact in mind, choosing tests to automate in this way makes true persuasive statements such as automating these n tests required effort e1, but they can now be executed automatically in time t rather than requiring effort e2 for manual execution. It is also often worth reminding management and ourselves that automatic execution is not prone to human error in action nor in observation. Test data design starting from historical reality The structure unit for test data in eCATT is called test data container. Its important to note that any execution run can be done against any number of containers, making it easy to group, categorize and reuse data. Because applications built in SAP are so configurable for example, they are often rolled out with the GUI in multiple natural languages some useful test data variants are easily found or obvious. So how to set about designing test data? The familiar techniques such as equivalence partitioning and boundary value analysis can be used easily of course. But in the case of most SAP applications, we question their defect-finding potential and also the importance of defects they might find. The inputs they generate are unlikely to occur in real use (the reverse might be true of, say, a public-facing web application which users might deliberately wish to try to break). Furthermore, tests created by application of a too-simple technique will become stale (that is,
Figure 3: seCATT transaction Figure 1: eCATT recorder
Figure 2: eCATT script
their potential to detect important defects even when the AUT changes will diminish) quickly. So actual inputs extracted from previous transactions may be a superior starting point. The data could then be increased in volume by permutation and by addition of transaction information from other systems obtained using data mining techniques. Thus real-life data is evolved into synthetic data and retains the advantages of both.
Test data design starting from randomness Although this approach, we believe, works well, in practice it is not always easy to do. Obtaining the real transaction data may be problematical both technically and because of its implications regarding privacy. In that case an opposite approach, using pure random data as the starting point, may be valuable. What testing really needs to develop that into realistic data
with good potential to detect defects is not the raw real-life data itself, but the coherences and probabilities that can be found within it. That might be obtained from statistical analysis of the data perhaps part of the standard functionality of the system that processes it with no need for sight of the data itself. For example consider an ordering system. Generate a test case including a random number of items to be ordered. Then determine, from analysis of historical data, the average number of items per order, the probability of an order being for only one item and the probability of an order being for the maximum permitted number of items. Use this information to weight the calculation used to generate the random number. Then apply the same procedure to other inputs and permutate the resulting test cases. Metrics to demonstrate ROI Much has been written about selecting test metrics. In general a metric should provide quantitative information and be objective, comprehensible, adaptive (that is, for use at different stages of product development) and possible to generate automatically. Peter Liggesmeyer (Software-Qualitt, Spektrum der Wissenschaft, 2010) suggests and defines further attributes: elementariness, validity, stability, seasonableness, analyzability, and possibility to reproduce. All these things, considered together, can be used to good effect as metric selection criteria. Basic effectiveness One way to quantify the effectiveness of any activity that aims to improve quality of a product is number of defects it detects divided by the effort taken to perform it. The effort to produce a manual test includes the time taken to understand the requirement on which it is based plus that to apply whatever model is used. For automated testing the time to automate must be added. It is possible to consider the time taken to learn how to automate using a given toolset as an additional effort, especially when tools are changed, but as it is approximately a one-time effort we will omit it.
Unfortunately this metric is challenged by one of our key criteria: it is not obvious in SAP development how to measure the time taken to achieve understanding. However SAP comes into its own when measuring the execution and defects: eCATTs logging facilities provide all the information needed. The answer, as for non-SAP development, is integrated application lifecycle management. We achieve this using SAP Solution Manager which allows the use of effort measurement, incident weighting and categorization optimized for the specific product in hand. Test case defect density In their article What should we measure during testing? (Testing Experience magazine September 2010), Yogesh Singh and Ruchika Malhotra define this as number of defects found divided by number of test cases executed. This metric indicates the effort needed to detect defects and is easily and entirely generated from eCATTs logging. Defect detection percentage (DDP) Effectiveness of a quality activity depends very much on when defects are detected and this common metric addresses that. It can be calculated at any time as number
of defects detected by the activity divided by total number of known defects. Again, eCATT makes it easy to count the defects detected by automated test execution and so to monitor DDP continuously using Solution Manager. Meaningfulness is increased further by weighting the defects as discussed above. Transaction coverage An SAP application consists of transactions: some standard built-in ones plus custom, specially-developed ones. The logging functionality of eCATT is based on these transactions and records, by their names, which were performed and when. This enables a simple routine to be written that calculates an SAP-specific metric: number of transactions performed by automated test execution divided by total number of transactions performed. Execution of data-driven test cases with different parameters can also be logged in detail by the eCATT script. This might raise the possibility of measuring depth of coverage of transactions. Unfortunately this information is in the script log not the test execution log and we have so far found no easy way for Solution Manager to obtain it, making it available for automatic metric calculations
Jessica Schiffmann is a senior consultant in SAP technology for Prism Informatics (prism-informatics.com). Eric M.S.P. Veith (eric.veith@wb-fernstudium.de) is a researcher at Wilhelm Bchner Hochschule/TU Bergakademie Freiberg
10
www.neotys.com
Any Device. Any Network. Any Region.

The rst complete mobile performance testing solution.
Records 100% of Mobile Applications Integrated Network Simulation Cloud Integrated
Free Evaluation of NeoLoad

30-Day License with Free Technical Support
www.neotys.com/trial
Behave yourself
by Roy de Kleijn
Acceptance-test-driven development
In this article I will explain an approach to preventing the second and third types of error. Acceptance-test-first Behaviour-driven development grew in the mind of its inventor Dan North (see http:// dannorth.net/introducing-bdd) from testdriven development. However whereas TDD is a development activity dealing with unit testing, BDD is very definitely a testing approach: it aims to deliver effective acceptance testing based on defined requirements, but in an agile way compatible with lifecycles that develop and deliver business value in short iterations. Put another way, TDD works from the inside of software creation, out. BDD works outside-in, assuring that everything done at lower levels is correct according to the high level. So, BDD is an application of the test-first principle used in TDD, but it changes the meaning of that term from unit-test-first to the greatly superior acceptance-test-first. This makes it very suitable for maintenance of legacy software as well as new-build, because knowledge of existing internal design and structure is not needed to do BDD.
Roy de Kleijns tutorial and get-started guide to behaviour-driven development for testers
Defects in software are caused by one of three kinds of human error. In the first, the developer understands what is required and attempts to achieve it but fails. In the second, the developer misunderstands the requirement and so succeeds in achieving the wrong thing. That misunderstanding also causes the third kind of error: working to achieve something that is not required and adds no business value. This also introduces a defect, but typically not one with the potential to cause product failure. More importantly, it wastes effort making project failure more likely.
BDD in practice An agile iteration (sprint or equivalent) begins with a short meeting defining its objectives, that is the requirements it will deliver. Its important that all involved contribute to defining these and their underlying requirements: but even then, the understanding of different people and roles tends to vary because of their different viewpoints and the different languages they use to communicate. BDD aims to solve this problem by having everyone work together also on acceptance criteria, called scenarios. Steve Watson discussed how to do this in the February 2013 issue of Professional Tester.
12
Gothenburg, Sweden 4 - 7 November 2013
d& Early Bir nts iscou Group D le Availab NOW
Europes Biggest Gathering of Software Testers
6 Worldclass Keynotes
50 Exhibitors, 1000 attendees
40+ presentations
7 TUTORIALS
LEARN, NETWORK, ENGAGE & DISCUSS.

NEW
First Time Delegate Connection Session | WELCOME DRINKS

NEW
Speed Networking | The Test Lab | Taste of Gothenburg Community Dinner

NEW
NEW
Active Workshops | Special Interest Discussion Tables | Test Clinic & Tip Exchange
NEW
w w w. e u ro s t a rco n fe re n ce s. co m
The Official EuroSTAR Party | And MUCH More!
In order to <receive benefit> As a <role> I want to <goal/desire>

Figure 1: syntax of a BDD story
Examples: |input1|input2|outcome| |yes|20|accepted| |yes|13|not accepted| [...]

Figure 3: example table
Scenario: <scenario title> Given <pre-condition> And <optional additional pre-condition> [...] When <action> And <optional additional action> [...] Then <post-condition> And <optional additional post-condition> [...]
Figure 2: syntax of a BDD scenario
The syntax of a BDD story is shown in figure 1. Obviously its creation is driven mainly by business representatives. The (usually many) acceptance criteria for the story are defined as BDD scenarios in the syntax shown in figure 2, surrounded if necessary by examples set out in a table as shown in figure 3. Notice here that each entry in the table is either a test data item or an expected outcome, and each line in the table is a test. This defined, uniform syntax makes the acceptance criteria easy to read and understand so that everyone involved can define them together. It also means they can be parsed and managed using tools. The testers present must ensure that they are clear and unambiguous. If everyone agrees that has been achieved, there should be no nasty surprises in store for anyone. After the meeting the testers can get started on writing functional, executable (and therefore automatable) acceptance tests to implement the scenarios according to the acceptance criteria. Each scenario and each of the test steps of which it consists is taken through a series of states as shown in figure 4. Initially, the scenario fails (state 2) because there is no step. Now the step is written but it too fails because there is no test object (3). Now just enough code is created to make the step pass (4) and if necessary previously-created code is refactored (5). When all steps pass, the scenario passes (6) and if necessary code created for other scenarios is refactored (7).
This process is repeated for all scenarios against all acceptance criteria, so in theory minimum code required to pass them should be written. Whether that is true in practice depends of course on how well the code is written and that is where TDD comes in.
org) the stories can be executed immediately. Only the actual test code need be implemented. This is based upon and analogous to the way developers use TDD frameworks. Traceability of course must work in both directions. When a story is executed, the results map back to the scenarios and examples: every line is viewed in red or green type depending on the results of tests associated with it. The exact current state of the product is immediately apparent. In theory, any other documentation describing its functionality becomes superfluous. This information can be interfaced to continuous integration/delivery servers. It provides immediate feedback on the effect on existing functionality of every commit or deployment, in a form that is easily accessible and comprehensible to everyone. Getting started with BDD I hope all this will make those PT readers who have not done so already want to try out BDD. To help, I have set up an example project which you can use to test web applications. It supports parallel test execution and takes a screenshot whenever the actual and expected outcome differs. To set it up, please follow the instructions in figure 5
1 2 o i ar 3 p e
sce n st
4 6
Figure 4: sequence of scenario and step states (after P. Creux, see http://eggsonbread.com)
Executable specifications and living documentation BDDs most important advantage is that it maps acceptance criteria, expressed as scenarios, directly to code. This can be seen as the ultimate in traceability. Using a BDD framework such as JBehave (also originally invented by Dan North, see http://jbehave.
Roy de Kleijn (roy.dekleijn@polteq.com) is a technical test consultant at Polteq where he works with clients on test process and automation, mainly in agile teams, and develops and presents training in Selenium/WebDriver. He is especially interested in web technology, new programming languages and knowledge transfer methods and publishes Selenium-related tutorials at http://selenium.polteq.com and test-related articles at http://rdekleijn.nl. This article is based on one originally published in Dutch by TestNet (see http://testnet.org)
14
First install the following software: 1 Eclipse IDE for Java Developers from http://eclipse.org/downloads 2 Maven. In Eclipse, choose Help > Eclipse Marketplace and search for Maven Integration for Eclipse 3 JBehave Plugin. Follow the instructions at https://github.com/Arnauld/jbehave-eclipse-plugin Now import the example project: 1 Download the sourcecode from http://roydekleijn.github.io/Spring-Jbehave-WebDriver-Example/ 2 Extract the ZIP file to your chosen folder 3 In Eclipse, right-click in the Package Explorer panel and choose Import > Existing Maven Projects 4 Choose the folder where you stored the files as root Directory. 5 Complete the wizard and click Finish The project is in three parts (see figure 5a): the package org.google.pages contains an abstraction of the application under test the package org.google.steps contains the mapping between textual sentences and code the folder org/google/web contains the textual stories To execute the stories: Choose Run > Run Configurations (figure 5b) Right-click on Maven Build and choose New Select the project using Browse Workspace In the Goals field, enter the command integration-test -Dgroup=google Dbrowser=firefox
Figure 5: setting up the example project
Figure 5b
Figure 5a
15
What enterprises need to know
Are you on the right path to mobile?

Creating increased demand for mobile applications.
These trends are impacting enterprises

88% of employees are using their personal devices for business.7
Mobile is exploding
In 2014 more users will connect to the Internet over mobile devices than desktop PCs.1
30% of organizations have or are implementing a private app store.8
Introducing challenges to IT departments trying to develop, test, and support mobile apps.
1.2 billion smartphones will enter the market over the next 5 years, about 40 percent of all handset shipments.2
Performance can make or break a mobile app

Only 30% of organizations have currently implemented any kind of mobile application performance infrastructure or strategy.14
6 billion mobile subscriptions were active at the end of 2011, equivalent to 87% of the world's population.3
:03
64% of mobile phone time is spent on apps.4
48% of organizations listed performance as their main priority for mobile testing.15
But, many organizations are unprepared for mobile testing.

1/3 of organizations lack mobile testing methodologies and processes.15
The proportion of companies increasing their budgets for mobile applications has almost doubled from 28% to 51%.5
By 2015, the number of mobile app downloads is projected to reach 98 billion.6
7
common operating platforms on the market9
29
versions of Android released since 200712
25% of users abandon a mobile app after three seconds of delay.13
And the security risk is growing...

More than 50% of organizations report security or compliance issues in their mobile deployments.16
Only 18% of organizations are testing mobile apps for security.15
18
versions of iOS released since 200711
420
active models of Android mobile phones10
Are you prepared?

Read Change your mobile application challenges into opportunities for more guidance on how you can address these top mobile issues.
The opportunity is huge.

The overall mobility market will top US $1 trillion by 2014.18
What does this all mean?

Develop a user-centric mobile strategy Support multiple devices Integrate with your legacy systems & data Monitor and manage the full mobile user experience Give your app dev a boost Embrace agile dev practices Automate mobile testing Account for mobile network conditions Embed security throughout the mobile lifecycle
2/3 of organizations say they lack the right tools to test mobile apps.15
2013
2014
2015
2016
2017
2018
QR-CODE FPO
In 2016, total global mobile application revenue will reach an estimated $46 billion.19
1. Mary Meeker, Internet Trends, Apr. 2010, Morgan Stanley 2. ABI Research 3. The International Telecommunication Union, Nov. 2011 4. The Digital Revolution: A Look Through the Marketers Lens, Apr. 2012, Nielsen 8. 2012 State of Mobility Survey, Symantec 9. Mobile Operating System, Wikipedia 10. Count from Android forums as of April 11, 2012 11. iOS version history, Wikipedia 12. Android version history, Wikipedia 15. World Quality Report, 201213, Capgemini and HP 16. Worldwide Mobile Security 20102014 Forecast and Analysis, Mar. 2010, IDC 17. World Quality Report, 201213, Capgemini and HP
5. Chris Marsh, Mobility Outlook 2013, Nov. 2012, Yankee Group 6. "The Mobile Application Market," Oct. 2011, Berg Insight 7. Global Survey: Dispelling Six Myths of Consumerization of IT, Jan. 2012, Avenade 13. First Class Mobile Application Performance Management, Aug. 2012, The Aberdeen Group 14. The Challenge of Application Performance in a Mobile Application World, Jul. 2012, The Aberdeen Group 18. Gartner Says Mobility will be a Trillion Dollar Business by 2014, Oct. 21, 2010, Gartner (press release) 19. Mobile Application Business Models, Q1 2012, ABI Research
Cross purposes
by Bogdan Bereza
Complete automation requires central programmatic control
The browser war was won by standards. Users and support technicians learned that using a more standards-compliant browser gave them a better chance of achieving what they wanted on a wider range of sites. That in turn incentivized businesses to make their sites more compliant. Good browsers like FireFox and Safari rose up the priority list. Bad ones like Internet Explorer 5 and 6 fell down it, so their subsequent versions became more compliant. At that point most of the problem disappeared. That is well because since then, while the number of important browsers has remained constant, platforms have proliferated in the form of mobile devices. Browser and platform are coupled tightly: the behaviour of browser A for mobile phone B can be very different to its version for tablet C. Thanks to standards, it is feasible to develop and test adequately a single version of a web application to work well on most clients. Even so, there are often two versions, desktop and mobile, in order to improve the experience of those with small screens and slow connections. Thus testing effort is split in two (perhaps less than two where the two versions share many components). Thats not good, but its better than having to split it into several fragments as was common at the height of the browser war.
Bogdan Bereza on regression testing without human intervention
Multiple versions of applications have always been bad news for testing. Many PT readers will remember from experience the nightmare caused by variation between web clients: platforms, browsers, plugins etc, all in various releases. Test design was shrunk by having to repeat test execution across a large number of combinations: important tests simply could not be done. The many disgracefully simple failures that occurred at the time were no coincidence.
Here we go again If all mobile applications were web applications, cross-client web and mobile development would now be in a strong position. Of course they are not: native apps have gained preference. There are many reasons, good and bad, for this which I will not attempt to analyse here. Some people consider building native apps to be retrograde: for example PhoneGap (http:// phonegap.com) wraps web applications, converting them to native mobile apps: this idea may catch on.
18
01487_pds_ad_hp_cat_proftest_ma_Layout 1 10/08/2012 14:27 Page 1
Agile is going places are you?

01793/PDS/AD/0812
BCS, The Chartered Institute for IT, offers the leading agile testing certication for software testers Certied Agile Tester.
bcs.org/agiletester
The Certied Agile Tester scheme is a trademark of iSQI. BCS, The Chartered Institute for IT, is the business name of The British Computer Society (Registered charity no. 292786) 2012
Event Loading...
London Autumn 2013
This highly anticipated event now returns to London on 24th October 2013. Register by 5th July for your early bird discount! Just visit: testexpo.co.uk
Until and unless it does it is necessary once again to develop for different proprietary client platforms. That means testers are once again faced with multiple versions of their test items, and they wont go away any time soon. The economic forces around users choice of device are more powerful and complex than those affecting free browser software. There is no incentive for standardization and no chance of one platform achieving dominance. Worse still, the most business-critical apps are hybrids, in the sense that they have multiple web and mobile clients sharing the same server/mainframe services and data. Web cross-client failures were usually of presentation; native apps have the dangerous potential to cause organization-wide failure. And this is in continuous delivery environments where they are being changed very frequently with high possibility of regression. Continuous cross-client hybrid app testing So testing needs to become continuous: the growing and evolving regression suite should be executing automatically and always, to assure against defects introduced by constant change to the test items made by automated integration and deployment. Although it is very simple, the test script shown in figure 1 could if executed in time have prevented a serious failure: a lessthan-one-line change to code prevented the app, when running on one specific and superseded version of the mobile device OS, from accessing the account balance. The rarely-executed code that should have trapped that event also failed. An incorrect balance, the last one retrieved, was displayed and transactions which should have been prevented were allowed. So how can a script like this, requiring execution and verification on multiple devices, be fully automated? The different approaches to functional testing of mobile apps taken by different tools suggest some possibilities. As so often in test automation, the key difference
between them is how they recognize interface objects. The device executes the test Most or possibly all tools that do this require the device to be jailbroken or rooted: because of the obvious disadvantages of doing that I wont consider them here. It is possible legitimate tools which are themselves apps might emerge, possibly supported by a change of policy by the device manufacturers of the kind that is easier to imagine happening on, say, Android than on iOS. If it does happen, whether they can be used for our current purpose will depend on (i) at what level they intrude on the app under test and (ii) whether and how they communicate with the cross-client master script so that it can control execution of the test, monitor its progress and verify its outcome. Remote control the device It is possible to execute tests on a device from a workstation, using various desktop sharing systems. The simplicity of this method is both its strength and its weakness. The recorded script is easy to read and can be controlled and verified by the master script as we require, but it is not robust: and because the only information available about what the application is doing is what is in the pixel image displayed on the device screen, modifying, reusing and maintaining it are visual and therefore very manual tasks. Remote control the app The most successful tools such as Ranorex control the app, not the device. This requires its code to be instrumented, a transparent process which makes the app able to automate itself: no other app is needed. User actions on the device are
Access test account A via the web application Transfer the entire balance of test account A to test account B [repeat for all mobile devices and OS versions currently supported] Access test account A via the mobile application and verify that its balance is displayed as zero [end repeat]
Figure 1: cross-client test script
captured and documented by Ranorex Recorder running on a workstation exactly as those performed on any other platform and can then be augmented with additional actions and validations, managed, maintained and reused in all the same ways, with object recognition accomplished in the most appropriate of multiple available ways. Continuous delivery aspires to automate as many deployment operations as possible. That is not done simply by recording them and playing them back: programmatic control is needed. Continuous testing requires exactly the same. Easy approaches designed to appeal to people who cannot understand simple code are not easy except in demonstrations which are always done using trivial example apps. In real testing of real apps they quickly become difficult or impossible. Continuous testing of mobile apps requires not ease, but power: that is what Ranorex offers
Frequent PT contributor Bogdan Bereza is a testing consultant, speaker and trainer, proprietor of VictO (see http://victo.eu) and co-ordinator of the inaugural Good Requirements conference taking place in Warsaw on 3rd and 4th October 2013 (see http://conference.wymagania.org.pl/english.html) A free trial of Ranorex, including all the mobile app testing features described here, is available from http://ranorex.com
20
Flash light
by Mark Lehky
Genie and monster: a groovy soap opera
development (or test) needs arise. If the model is adhered to strictly, test automation becomes simple: any of the tiers can be replaced with virtually any tool. If, due to design and/or development decisions, the boundaries between the tiers start to blur, testing (and future development) become more difficult. This is one of the reasons testing should get involved in the design process as early as possible. The data tier is usually, and should be, unimportant from the test point of view. It should have very little impact on the tiers above, and the choice of database engine is usually dictated by other factors such as performance, security and cost. The logic tier should be responsible for transmitting, recoding, and processing all messages. Transmission of messages is usually done over classic HTTP or Real Time Messaging Protocol (RTMP). The content of the messages is usually encoded using Action Message Format (AMF). If the application is written in Java, then the logic tier server is typically BlazeDS; if the application is written in C# .NET, the server will likely be FluorineFX. Some logic tiers are very complex and use multiple technologies (web server, message queue, etc). Here I will deal with the most common AMF messages only. The presentation tier of a Flash application is most often written in ActionScript. This has the look and feel of Java: object-oriented design, compile-once-run-anywhere to produce an SWF file playable by a large number of available player applications on all desktop and mobile platforms. This technology is completely different to HTML, but the player can be made compatible with a web browser via a plug-in: I am often (ie several times a week) asked: can I automate Flash with Selenium?. The answer is no. Sort of.
Mark Lehky illuminates Flash for testers
Its impending decline has been rumoured for years, but Flash remains one of the most popular ways to create rich Internet applications. By testers, it is often perceived as difficult to automate. In this article I will try to dispel misconceptions and explain how effective test automation of Flash applications can be achieved easily: and when and why it cannot. Technology overview Most web Flash applications can be described very well by the three-tier architecture model (see figure 1). Ideally, none of the tiers knows anything about the platform, technology, or structure of any of the others, and therefore any one can easily be swapped out as new
21
Presentation tier
The top-most level of the application is the user interface. The main function of the interface is to translate tasks and results to something the user can understand.
>GET SALES TOTAL
4 TOTAL SALES
>GET SALES TOTAL
Logic tier
This layer coordinates the application, processes commands, makes logical decisions and evaluations, and performs calculations. It also moves and processes data between the two surrounding layers.
GET LIST OF ALL SALES MADE LAST YEAR ADD ALL SALES TOGETHER
also need to run a Genie Socket Server (reminiscent of the old Selenium Server). After that, things get pretty easy. Genie browser Genie automation scripts refer to objects by their GenieID. This is similar to the HTML element id or the element locators used by Selenium, but more complex: it is formed from several attributes of the Flash application determined by its developers together with some cryptic portions assigned by Genie itself. The result can look quite cryptic, for example SP^botBa rContainer:::FPûserBox:::SEîmgBox:: PX^2::PTR^0::IX^2::ITR^0. GenieIDs are difficult to predict, even in close collaboration with developers. Genie includes a browser (figure 2) that helps you locate elements and discover their GenieIDs and other attributes. Genie recorder User actions within the Flash application are recorded as a generic Java script. Genie has no playback functionality; the script is intended to be pasted into Eclipse and run from there, somewhat similar to recording a script with Selenium IDE and then exporting it to Java. An example Genie script is shown in figure 3. It is somewhat similar to Selenium RC code, and the recorder will help to ease you into Genie programming. There is no contextual menu to help create assertions; they have to be inserted manually, in Eclipse. Once you become proficient at writing code by hand you will almost never use Genie recorder again. As can be seen from the script, Genie has its own test framework called GenieScript. This is all you need to create and execute extensive test suites,
Data tier
Here information is stored and retrieved from a database or file system. The information is then passed back to the logic tier for processing, and then eventually back to the user.
QUERY
SALE 1 SALE 2 SALE 3 SALE 4
Database
Storage
Figure 1: three-tier architecture (public domain image from http://en.wikipedia.org/wiki/ File:Overview_of_a_three-tier_application_vectorVersion.svg)
UI automation with Genie Selenium is not a tool, but a library available in many programming languages that allows the coder to access and manipulate objects (text, controls, images etc) in a web browsers Document Object Model (DOM). Many technologies, eg HTML5, JavaScript, AJAX and others, generate or manipulate the DOM. Flash does not. The fact that a web browser requires a plug-in to be able to display Flash applications is the first giveaway that Selenium cannot automate Flash. A Flash application is separate to the DOM, even though it can be displayed in the browser window alongside DOM objects. Selenium was never intended to automate Flash, and in all likelihood never will. Fortunately, there are other libraries to enable you to manipulate objects in a Flash application. Here I will discuss one of them, Genie (http://sourceforge.net/ adobe/genie/). Another example, also often recommended, is flash-selenium (http://code.google.com/p/flash-selenium/). One reason, and probably the most
important, why you would want to choose Genie over flash-selenium is that flashselenium is dependent on Selenium Remote Control, which was deprecated (in favour of WebDriver) in July 2011. Genie depends on nothing and can be used completely standalone, or combined with anything else you may want to use. It can be plugged into almost any existing framework or toolset (Ant, Maven, JUnit, TestNG, etc etc). Unlike Selenium, Genie is only available in Java and some of its development support tools will only work in Eclipse. This is a barrier for testers using only .NET and Visual Studio. Genie is open source so a port is possible but I dont foresee it. If you need to test Flash and want to use Genie but dont know Eclipse I strongly recommend the tutorials at http://eclipse. org/resources/. Before installing Genie you will need to install Eclipse, the Java Development Kit (JDK) at least version 1.5, and the debug version of Flash Player. The Genie download comes with full instructions. You will
22
although it does not have all the features of more mature frameworks such as JUnit or TestNG which are often used instead. Genie also features built-in custom logging which produces an XML file, with an XSLT provided to convert it to viewable HTML. The file can of course be parsed with any tool, but it does not follow any established format such as JUnits. Furthermore, the log is limited to only Genie commands: if your test fails in any other step (for example an assert), this will not be logged automatically. Genie does have extra commands to log your own events, but this is additional overhead. If you decide to use some other framework such as JUnit or TestNG you will probably use the logging capabilities it provides so the logs can be parsed automatically by your continuous integration server, and once you have finished debugging your test will want to turn off Genies logging. This is done with:
1. LogConfig logger = new LogConfig(); 2. logger. setNoLogging(true);
Figure 2: Genie browser
All your Selenium skills (and frustrations) still apply! Because Genie, like Selenium, is a Java library, having the framework of your choice run your tests is trivial. Figure 4 shows how to do this with JUnit; figure 5 combines it with Selenium. I have also found that all my Genie tests run with no problems extended from GroovyTestCase. This is significant because Groovy, a scripting language for the Java platform, is natively understood by SoapUI. Ill explain this significance further in the section on API automation below.
The Page Object Model Think of your application as a collection of screens, each with a number of controls and actions that it can perform. If you can describe one screens actions as a collection of methods, abstracting away all the controls, and make it independent of any other screen, then you have a PageObject. Alan Richardson described the advantages of this in Professional Tester August 2012 (http://professionaltester. com/magazine/backissue/PT016/ ProfessionalTester-August2012Richardson.pdf).
Genie, with its obscure GenieIDs, is perfect for this approach! Further, because Flash is often embedded in a webpage via a browser plug-in, there is great incentive to combine the automation with Selenium. Using the PageObject pattern, you can effectively hide which actions require accessing Selenium and which are accessing Genie. Page loading delays As with any web technology, there are issues with page loading delays. Since the Flash application runs in a plug-in you will not be able to query the browser to see if the page is loaded it is always loaded.
23
1. package scripts; 2. 3. import com.adobe.genie.genieCom.SWFApp; 4. import com.adobe.genie.executor.GenieScript; 5. import com.adobe.genie.executor.components.*; 6. import com.adobe.genie.executor.uiEvents.*; 7. import static com.adobe.genie.executor.GenieAssertion.*; 8. import com.adobe.genie.executor.enums.GenieLogEnums; 9. 10. 11. /** 12. * This is a sample Genie script. 13. */ 14. //Change name of the class 15. public class Unnamed extends GenieScript { 16. 17. public Unnamed() throws Exception { 18. super(); 19. 20. } 21. 22. @Override 23. public void start() throws Exception { 24. //Turn this on if you want script to exit 25. //when a step fails 26. EXIT_ON_FAILURE = false; 27. 28. //Turn this on if you want a screenshot 29. //to be captured on a step failure 30. CAPTURE_SCREENSHOT_ON_FAILURE = false; 31. 32. SWFApp app1=connectToApp([object paMain]); 33. (new GenieDisplayObject(SP^stageContainer:::FP^loginContainer:::SEûserN ameTxt::PX^6::PTR^0::IX^1::ITR^0,app1)).click(56,17,436,174,1090,821,3,f alse); 34. (new GenieTextInput(SP^stageContainer:::FP^loginContainer:::SEûserNameT xt::PX^6::PTR^0::IX^1::ITR^0,app1)).selectText(0,0); 35. (new GenieTextInput(SP^stageContainer:::FP^loginContainer:::SEûserNameT xt::PX^6::PTR^0::IX^1::ITR^0,app1)).input(147258369); 36. (new GenieDisplayObject(SP^stageContainer:::FP^loginContainer:::SE^passw ordTxt::PX^6::PTR^0::IX^2::ITR^0,app1)).click(63,12,443,246,1090,821,3,f alse); 37. (new GenieTextInput(SP^stageContainer:::FP^loginContainer:::SE^passwordT xt::PX^6::PTR^0::IX^2::ITR^0,app1)).selectText(0,0); 38. (new GenieTextInput(SP^stageContainer:::FP^loginContainer:::SE^passwordT xt::PX^6::PTR^0::IX^2::ITR^0,app1)).input(1234); 39. (new GenieDisplayObject(SP^stageContainer:::FP^loginContainer:::SE^login Btn::PX^6::PTR^0::IX^4::ITR^0,app1)).click(42,44,1011,218,1090,821,3,fal se); 40. } 41. }
Figure 3: Genie script
24
1. import junit.framework.TestCase; 2. 3. import com.adobe.genie.executor.Genie; 4. import com.adobe.genie.executor.LogConfig; 5. import com.adobe.genie.executor.components.GenieDisplayObject; 6. import com.adobe.genie.executor.components.GenieTextInput; 7. import com.adobe.genie.genieCom.SWFApp; 8. 9. public class MyTestSuite extends TestCase { 10. 11. Genie jean; 12. SWFApp app1; 13. 14. protected void setUp() throws Exception { 15. super.setUp(); 16. 17. LogConfig logger = new LogConfig(); 18. logger.setLogFolder(log); 19. jean = Genie.init(logger); 20. app1 = jean.connectToApp([object paMain]); 21. jean.EXIT_ON_FAILURE = true; 22. jean.EXIT_ON_TIMEOUT = true; 23. jean.CAPTURE_SCREENSHOT_ON_FAILURE = false; 24. } 25. 26. protected void tearDown() throws Exception { 27. 28. jean.stop(); 29. 30. super.tearDown(); 31. } 32. 33. void testCase_LogIn() { 34. 35. (new GenieDisplayObject(SP^stageContainer:::FP^loginContainer:::SEûserN ameTxt::PX^6::PTR^0::IX^1::ITR^0, app1)).click(); 36. (new GenieTextInput(SP^stageContainer:::FP^loginContainer:::SEûserNameT xt::PX^6::PTR^0::IX^1::ITR^0, app1)).selectText(0, 100); 37. (new GenieTextInput(SP^stageContainer:::FP^loginContainer:::SEûserNameT xt::PX^6::PTR^0::IX^1::ITR^0, app1)).input(147258369); 38. (new GenieDisplayObject(SP^stageContainer:::FP^loginContainer:::SE^passw ordTxt::PX^6::PTR^0::IX^2::ITR^0, app1)).click(); 39. (new GenieTextInput(SP^stageContainer:::FP^loginContainer:::SE^passwordT xt::PX^6::PTR^0::IX^2::ITR^0, app1)).selectText(0, 100); 40. (new GenieTextInput(SP^stageContainer:::FP^loginContainer:::SE^passwordT xt::PX^6::PTR^0::IX^2::ITR^0, app1)).input(1234); 41. (new GenieDisplayObject(SP^stageContainer:::FP^loginContainer:::SE^login Btn::PX^6::PTR^0::IX^4::ITR^0, app1)).click(); 42. } 43. 44. void testCase2() { 45. ... 46. } 47. }
Figure 4: Genie in JUnit
25
1. import junit.framework.TestCase; 2. 3. import org.openqa.selenium.WebDriver; 4. import org.openqa.selenium.firefox.FirefoxDriver; 5. 6. import com.adobe.genie.executor.Genie; 7. import com.adobe.genie.executor.LogConfig; 8. import com.adobe.genie.executor.components.GenieDisplayObject; 9. import com.adobe.genie.executor.components.GenieTextInput; 10. import com.adobe.genie.genieCom.SWFApp; 11. 12. public class MyTestSuite extends TestCase { 13. 14. Genie jean; 15. SWFApp app1; 16. WebDriver driver; 17. 18. protected void setUp() throws Exception { 19. super.setUp(); 20. 21. driver = new FirefoxDriver(); 22. driver.get(http://some.server.test/login/); 23. 24. LogConfig logger = new LogConfig(); 25. logger.setLogFolder(log); 26. jean = Genie.init(logger); 27. app1 = jean.connectToApp([object paMain]); 28. jean.EXIT_ON_FAILURE = true; 29. jean.EXIT_ON_TIMEOUT = true; 30. jean.CAPTURE_SCREENSHOT_ON_FAILURE = false; 31. } 32. 33. protected void tearDown() throws Exception { 34. 35. jean.stop(); 36. driver.quit(); 37. 38. super.tearDown(); 39. } 40. 41. void testCase2() { 42. // tests now have access to Selenium such as 43. driver.findElement(By...) 44. } 45. }
Figure 5: Genie with Selenium
26
Just as in Selenium, you will need to use tricks to discover if elements are present, detect busy streamers, and so on. Genie has methods for all elements, such as .isEnabled(), .isPresent(), and .isVisible(). So, for example, waiting for something like a busy streamer to go away is simple (figure 6). XPath and PageFactory Neither XPath not PageFactory (a design pattern used in a framework to make discovery of elements simpler and more dynamic) is implemented in Genie. However a useful feature of the Genie browser gives Genie the ability to handle both: it can dump an XML representation of the entire application. The code to do this is shown in figure 7 and you can then search for GenieIDs using XPath as shown in figure 8. The Genie IDs are guaranteed to be consistent from one run of your application to the next. However, it is possible for the developer to create a Flash application in which controls look the same but are instantiated on the fly, making the GenieIDs effectively dynamic. This can be a problem but PageFactory solves it by finding the GenieIDs on the fly. To build your own PageFactory you will need to add the Java Reflection library to your framework. API automation with SoapUI In many Flash projects the underlying API will be available to testing long before the UI materializes (assuming it ever does). It makes sense to test the API first. The popular API testing framework SoapUI has limited support for AMF messages and no support for any flavour of the RTMP protocol. To test the APIs you
1. static void waitForStreamer(GenieMovieClip streamer, int seconds) { 2. Date rightNow = new Date().getTime(); 3. while(streamer.isVisible()) { 4. if(rightNow + (seconds * 1000) < new Date(). getTime()) { 5. throw new TimeoutException(Page busy? Streamer still visible after + seconds + seconds.); 6. } 7. } 8. }
Figure 6: waiting for a busy streamer to go away
1. String appXml = SynchronizedSocket.getInstance(). getAppXMLGeneric(app.name); 2. ByteArrayInputStream inputStream = new ByteArrayInputStream(appXml.getBytes()); 3. DocumentBuilder docBuilder = DocumentBuilderFactory.newInstance(). newDocumentBuilder(); 4. Element docElement = docBuilder.parse(inputStream). documentElement; 5. XPath xpath = XPathFactory.newInstance(). newXPath();
Figure 7: dumping an XML representation of the application
1. 2.
Node page = xpath.evaluate( //*[@name= + fieldname + ], pageNode, XPathConstants.NODE ); return pageNodes.attributes. getNamedItem(genieID).getNodeValue();
Figure 8: searching for a GenieID using XPath
will need to have HTTP enabled on the server: RTMP and HTTP can be enabled simultaneously. Although you may not be doing this in production, bear in mind that (i) you are not testing the protocol nor, probably, the server configuration; (ii) transmission via the RTMP protocol will be validated as part of your end-to-end test, which you will probably do through the UI. Right now we are concerned only with functional testing of the APIs. There is no call discovery for AMF messages in SoapUI. This is the equivalent of reading in a WSDL for SOAP endpoints, or a WADL for REST endpoints. For AMF everything has to be done by hand. However SoapUI does provide other goodies: test scripting, Groovy extensions and XML parsing of responses. We
will now see how to hook up AMF with SoapUI tests. There is a useful, short tutorial on these at http://soapui.org/AMF/ calling-amf-services.html. Method discovery As there is no way to define an AMF endpoint in SoapUI, one just creates an empty project (unless the project also contains SOAP and/or REST endpoints) and a new test case. When you add an AMF step to your test case you will need to provide three pieces of information: the endpoint, the AMF call, and all the inputs (figure 9). If you are lucky you will be able to get these from a detailed technical specification provided by your developers. If you are like me and you want any documentation you will have to write it yourself.
27
Figure 9: adding an AMF step
Figure 10: method calls in BlazeMonster
Figure 11: two arguments of the same type: which is which?
Figure 12: naming inputs
AMF endpoint This is made up of several parts as follows: {protocol}://{server}/{service}/messagebroker/amf. You will have to find out the parts in curly brackets from development. Remember that for SoapUI the protocol has to be http or https: rtmp will not work. For secure messaging the last part is sometimes messagebroker/amf. Note however that this does not make the messaging secure: configured secure transmission (the s part of https) does. Again, remember that for now we are only concerned with functional testing of the APIs and everything else just gets in the way of that. AMF call Now that we have an endpoint, we need to know the method calls. The tool BlazeMonster (http://sujitreddyg. wordpress.com/blazemonster/) can be used for, among other things, discovering them. For that we need only its Existing Services function. To use it, enter the web applications root URL, that is the variable parts of the endpoint, and click Load
Simple inputs These are simply added to the input window of SoapUI. In the example shown, the call is named findCustomerBySSN and it takes only one input: probably the SSN (social security number). I say Note that BlazeMonster is not perfect. It probably, because BlazeMonster tells shows all the method calls on the server, us only that there is one argument and whether they are public (available to be that it is of type java.lang.String. It is able called) or private (not exposed as callable to read only the method signature so methods). Later on, when you are actually knows the number and type of inputs, but making the call using SoapUI, if you get a response that such a method does not exist not what they represent. We can test our guess from within BlazeMonster using the even though BlazeMonster says it does, Invoke selected button. check with your development whether that particular method is public or private. Unfortunately, things are seldom that simple! If the AMF call takes more than You can now determine the AMF call. It is one argument, guessing what they are {destination name}{method} as those are becomes harder. Even if you can do that, shown in BlazeMonster: I usually have guessing which is which becomes very both BlazeMonster and SoapUI open on facing screens, and just copy-paste every- hard, especially if more than one of them are of the same type (figure 11). thing between the two windows. destinations. If everything is configured correctly, BlazeMonster fills in the AMF endpoint URL automatically and all the method calls are loaded up (figure 10). Now you can click on the run step button in SoapUI, and should get an error response saying that you did not supply any inputs (unless that call does not require any). The order in which the parameters are displayed now becomes important. You will need to find out where, in the source code repository, is the interface to the service, then read the code and find the method by
28
name, eg findCustomerByName(String surname, String givenName). Then when creating the AMF step in SoapUI, give the inputs the same or similar names in the same order (figure 12). The name is not actually important, only the order is. Complex inputs Things now (as always in testing) get worse. We are accessing via AMF the underlying methods, right in the technology layer. That means the arguments of some of the calls are probably not simple numerical or string variables. They may be java. util.Collection, or a custom object. Custom objects are easy to identify (their type starts with com.<your_company_name>) but difficult to deal with. Selecting a method with these argument types in BlazeMonster and clicking the Invoke selected button gives the error message Selected operation expects object types which are not supported by this application. Generate code for invoking this operation. And so we return to Groovy, the scripting language that can be used to generate the inputs for the call within SoapUI. Luckily for non-Groovers, Groovy is source-compatible with Java. You can write simple Java code to create the needed object(s) and it should work in your SoapUI AMF step with very little modification. First, create a new AMF step in the way already described: enter the endpoint and AMF call and name the input. There could be more than one input, but in that case you should have a discussion with development about whether they would want to consider redesigning this call. Next, in BlazeMonster, select the appropriate method and click on Generate code. A new window appears (figure 13) containing AS3 (ActionScript) VO (Value Objects) code for Java classes. The lower pane
Figure 13: AS3 VO code and custom objects
1. package com.company.dto 2. { 3. [Bindable] 4. [RemoteClass(alias=com.company.dto.CustomerDTO)] 5. public class CustomerDTO 6. { 7. public var dateOfBirth:Date; 8. public var givenName:String; 9. public var id:Number; 10. public var surname:String; 11. 12. public function CustomerDTO() 13. { 14. super() 15. } 16. } 17. }
Figure 14: AS3 VO code for a specific custom object
1. 2. 3. 4. 5. 6.
import com.company.dto.CustomerDTO def scriptCustomer = new CustomerDTO() scriptCustomer.givenName = Peter scriptCustomer.surname = Parker scriptCustomer.dateOfBirth = new GregorianCalendar (1962, Calendar.AUGUST, 15).getTime() parameters[customer] = scriptCustomer
Figure 15: Groovy script
29
lists the input and output custom classes. Select the class you are trying to create and click on Generate VO. This opens a new tab with more AS3 code, which gives a lot of hints about what is needed. Copypaste it into the script pane of the AMF call in SoapUI. An example of what it looks like is shown in figure 14. Now we will convert this to Groovy code SoapUI can understand. I usually start by enclosing all the code in a comment block. Now change lines 4 and 5 to lines 1 and 2 of the Groovy script as shown in figure 15. Next, we need the jar file that defines the object: place this in SoapUIs bin/ext folder and restart SoapUI to load it into the session. Now click the run button. If you get a dialog saying something like paramets {} amfHeaders {} youre OK! If you get errors, dont move on until you resolve them. Then, continue building the script inputs (lines 3-5 in figure 15, replacing lines 7-9). You will have to find out, from the source code (which may not be trivial) or from developers, which variables are required: in this example, well assume id is optional. Note also that our three new lines do not introduce the variables in the same order as the old ones: this is fine because its Groovy. SoapUI cannot handle the situation where one of the variables is another custom object. Even if you do get all the pieces lined up, the call will probably fail. I dont have any way around this brick wall at present except that, again, the design might be questioned. Once the object has been built up, assign it back to the variable name (line 6 in figure 15) and run the script. If you see a dialog showing your inputs (figure 16), you are ready to run the test step.
Figure 16: call inputs confirmed
1. <flex.messaging.io.amf.ASObject> 2. <dateOfBirth>1962-08-15 00:00:00.0 PDT <dateOfBirth> 3. <givenName>Peter</givenName> 4. <surname>Parker</surname> 5. </flex.messaging.io.amf.ASObject>

Figure 17: the response as XML (simple objects)
1. <flex.messaging.io.amf.ASObject serialization=custom> 2. <string>dateOfBirth</string> 3. <date>1962-08-15 00:00:00.0 PDT</date> 4. <string>givenName</string> 5. <string>Peter</string> 6. <string>surname</string> 7. <string>Parker</string> 8. </flex.messaging.io.amf.ASObject>
Figure 18: the response as XML (custom object)
30
Complex responses The AMF response is a binary object. You can see a representation of sorts on SoapUIs Raw response tab. However SoapUI also goes to great lengths to give you a more familiar representation: Response as XML. For simple response objects it usually does a pretty good job of guessing what belongs where so the output is quite readable (figure 17). For example, givenName could be validated using the XPath //givenName. For complex custom objects, the response can be more verbose (figure 18). In this case XPath axis is needed, eg //string[text()=givenName]/ following-sibling::*[1].
Dealing with multiple architectures Unfortunately this method for handling complex inputs will not work for .NET logic tiers, not least because SoapUI cannot import DLLs and serialization of objects is done differently by .NET and Java. I have seen some workarounds tried: for example publishing the information on a special JMS test queue; duplicating information in the message headers; exposing methods as a SOAP call. All achieved some success but all require additional development work. If you will be testing a Flash application using .NET the only suggestion I can offer at present is that you address this issue as urgently as possible
Mark Lehky has worked in test automation in multiple industries since 1999. He blogs on the subject at http://siking.wordpress.com and keeps in touch via LinkedIn (http://linkedin.com/in/marklehky)
Achieve test coverage of up to 95% Increase test automation up to 90% Reduce your time-to-market
Free Demo
www.tricentis.com/PT

PT Issue21

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

PT Issue21

Transféré par

Droits d'auteur :

Formats disponibles

TE TER

Essential for software testers

Continuous testing and delivery

THIS ISSUE OF PROFESSIONAL TESTER IS SPONSORED BY

From the editor

Managing Director Art Director Sales

4 We can no longer beat agile so must join it 8 Autometrics

Continuous testing and delivery

Edward Bishop examines the impact of mobile apps on testing

Contributors to this issue

12 Behave yourself 18 Cross purposes 21 Flash light

BDD for testers with Roy de Kleijn

Bogdan Bereza discusses centralizing mobile app test automation

Visit professionaltester.com for the latest news and commentary

PT - June 2013 - professionaltester.com

Continuous testing and delivery

We can no longer beat agile so must join it

How to stop testing limiting release frequency

PT editor Edward Bishop assesses the impact of mobile on testing

PT - June 2013 - professionaltester.com

Continuous testing and delivery

Figure 1: Application Lifecycle Intelligence

PT - June 2013 - professionaltester.com

Continuous testing and delivery

Figure 2: Quality Center interaction within Eclipse

Figure 3: editing a Service Virtualization definition

PT - June 2013 - professionaltester.com

Easily record automated tests for your modern HTML5 apps

Continuous testing and delivery

Better automation means automating more, including measurement

PT - June 2013 - professionaltester.com

Continuous testing and delivery

Figure 2: eCATT script

PT - June 2013 - professionaltester.com

Continuous testing and delivery

PT - June 2013 - professionaltester.com

Any Device. Any Network. Any Region.

Records 100% of Mobile Applications Integrated Network Simulation Cloud Integrated

Free Evaluation of NeoLoad

Continuous testing and delivery

PT - June 2013 - professionaltester.com

Gothenburg, Sweden 4 - 7 November 2013

d& Early Bir nts iscou Group D le Availab NOW

Europes Biggest Gathering of Software Testers

50 Exhibitors, 1000 attendees

LEARN, NETWORK, ENGAGE & DISCUSS.

First Time Delegate Connection Session | WELCOME DRINKS

Speed Networking | The Test Lab | Taste of Gothenburg Community Dinner

The Official EuroSTAR Party | And MUCH More!

Continuous testing and delivery

In order to <receive benefit> As a <role> I want to <goal/desire>

Examples: |input1|input2|outcome| |yes|20|accepted| |yes|13|not accepted| [...]

PT - June 2013 - professionaltester.com

Continuous testing and delivery

PT - June 2013 - professionaltester.com

What enterprises need to know

Are you on the right path to mobile?

These trends are impacting enterprises

30% of organizations have or are implementing a private app store.8

Performance can make or break a mobile app

But, many organizations are unprepared for mobile testing.

By 2015, the number of mobile app downloads is projected to reach 98 billion.6

25% of users abandon a mobile app after three seconds of delay.13

And the security risk is growing...