Load Testing Ajax Apps using head-less browser tools

NoVaTAIG April 13, 2011 Gopal Addada and Frank Hurley Cigital Inc.

Agenda

About Cigital Background : AJAX and Load Test requirements Tools research HTMLUnit Virtual User (VU) Generation Results and reporting Pros and Cons Limitations and Future work

Software Assurance services

Software Quality Agile testing Test automation Continuous integration Test process improvement Software Security Secure design Secure coding Security testing Continuous integration
3

Dec10

Background: AJAX

Web 2.0 ! Interactive Several AJAX frameworks are being used

GWT, jQuery, MochiKit, Adobe Flex, Dojo, YUI LoadTester, Neotys, LoadRunner(HP) Very expensive (per license per Virtual User)

COTS tools (claim to) support Ajax load tests:

Open source solutions

Jmeter, Grinder and openSTA

Background: AJAX

Taken from Aztecsoft Whitepaper [1]

Please refer to Descriptive Picture at URL [2]: http://www.adaptivepath.com/uploads/archive/images/publication s/essays/ajax-fig2_small.png

Background: AJAX
Classic Web applications Requests sent as direct response to user actions Requests are normally <form> that use GET/POST methods Browser - server involves requests for complete pages AJAXy web applications Requests can happen due to user actions or asynchronously at intervals determined by AJAX engine Can be of any form supported by remote application server (e.g. XMLHttpRequest) Typical requests involve updating parts of already loaded page

The response from server involves entire new page which is rendered at client browser

The response is intended for small portion of page which is interpreted by AJAX engine on client side and updates DOM of current page

Partially from Aztecsoft Whitepaper [1]

Load Test Tool Requirements for Web

Need to simulate User behavior using Virtual Browsers Real browsers dont work:
Do not scale well as running multiple browser tests from test machines is not feasible Intent is to test the Server performance JMeter and Grinder are prominent open-source tools Use HTTPClient implementations and are multi-threaded GUI to create load tests Do not support Java Script and hence cant simulate AJAX engine behavior

Load Test Tool Requirements for Web

Classic Web applications Need any virtual (headless) browser that simulate user actions using HTTP client AJAXy Web applications Need any virtual (headless) browser that simulate user actions using HTTP client with JavaScript Support

Involves testing mostly server, client behavior may not have to be replicated (e.g. cross-browser does not matter). Simulating concurrent HTTP GETs/POSTs is enough (e.g. Jmeter, Grinder)
Response has to be asserted and response times may need to be captured. Server has to be monitored for CPU performance, memory usage and DB performance

Browser runs AJAX Engine (JS calls) -> need to simulate client behavior to maximum extent possible

Response has to be asserted and response times may need to be captured. Server has to be monitored for CPU performance, memory usage and DB performance

Tool research (open-source)

Jmeter One of the best load test tools, but does not support JS calls HttpUnit Good Headless browser tool and No JS support Grinder Good tool, but no known JS support HtmlUnit 100% Java-based headless browser Supports HTTP Requests, HTML Parsing, JS execution, CSS Easy to script tests (similar to Selenium) WebTest Built on HtmlUnit and tests written in Groovy or XML Can potentially be used for load test design Easier and non-programmer friendly

HtmlUnit
@Test public void testGoogle() throws Exception {
WebClient webClient = new WebClient(BrowserVersion.FIREFOX_3); HtmlPage gPage = webClient.getPage(http://www.google.com); assertEquals(Google", gPage.getTitleText());
HtmlUnit qTxtbox = (HtmlInput)gPage.getElementByName(q).get(0); qTxtbox.setValueAttribute(NoVaTAIG);
HtmlElement btn = gPage.getFirstByXPath(//input[@value=\ I'm Feeling Lucky\]);

HtmlPage gPage2 = btn.click();

asserEquals(Northern Virginia Test Automation Interest Group,gPage2.getTitleText());

} }

HtmlUnit (contd.)

SHOW ARCHITECTURE DIAGRAM from MARCs PRESENTATION

HtmlUnit (contd.)
AJAX Libraries Supported

CurvyCorners Dojo ExtJS GWT JQuery MochiKit Prototype Sarissa Yahoo UI ...

Approach for our load tests

Complete Black-box approach simulating a browser (Vuser) and perform user actions Write functional tests for pages to be load tested Generate desired number of threads Each thread calls functional test(s) Each thread can use different user

To simulate a close-to-reality scenario

Each thread may have to simulate behavior of different browser (IE, Firefox, Chrome)

Approach
ThreadPoolMain
Test Status Data reported back to main

T
Virtual users (VU)

Thread 1

Thread 2 T

Thread 3 T

Thread n

HtmlUnit CodeT

HtmlUnit CodeT

HtmlUnit CodeT

HtmlUnit CodeT

HtmlUnit code demo

//Every thread calls this function which has sequence of tests public void runGTest() throws Exception{ HtmlPage currentPage; webClient.setUseInsecureSSL(true); final HtmlPage loginHomePage = login(url,username,passwd); WebAssert.assertTextPresent(loginHomePage, "Change security question"); currentPage = loginHomePage; goToGmail(loginHomePage); /** INVOLVES AJAX CALLS**/ logout(currentPage); }

HtmlUnit code demo (Contd..)

//Sample test involving Ajax calls
private HtmlPage goToGmail(HtmlPage dashboardPage) throws Exception{
webClient.setAjaxController(new NicelyResynchronizingAjaxController());

final HtmlAnchor gmailLink = dashboardPage.getFirstAnchorByText("Gmail"); final Page gmailPage = gmailLink.click(); // AJAX Engine call (JS) WebAssert.assertElementPresent((HtmlPage)gmailPage,"loading"); WebAssert.assertLinkPresentWithText((HtmlPage)gmailPage,"Load basic); WebAssert.assertTitleContains((HtmlPage)gmailPage, "Gmail"); return dashboardPage; }

Thread class
public class GmailThread implements Runnable { private int userIndex = -1; public GmailThread(int userIndex) { this.userIndex = userIndex; } public void run() { String origName = Thread.currentThread().getName(); Thread.currentThread().setName("user_" + userIndex); GAccountAccess gTest= new GAccountAccess(); gTest.runGTest(); } }

Test Thread Details class (New Object Passed from main to all the way to HtmlTest)
public class TestStatusData { private long threadStartTime; private long threadEndTime; private String tUsername; private String tPasswd; // Dont save password just place holder Private threadPage1ResponseTime; Private threadPage2ResponseTime; Private threadPage3ResponseTime; private long threadLoginTime; private long threadLogoutTime; String threadCurrentStatus; String tComment; //sample setter and getter public void setSFTPLoginTime(long sftpLoginTime){ threadSFTPLoginTime = sftpLoginTime; } public long getSFTPLoginTime(){ return threadSFTPLoginTime; }

Thread Pool (Main class) *** Can be used as

black box for most load tests. Details not really important. Partial code (Will be posted in blog)
private ExecutorService userThreadPool = null; private HashMap<Future<GmailThread>, GmailThread> taskMap = new HashMap<Future<GmailThread>, GmailThread>(); userThreadPool = Executors.newFixedThreadPool(concurrentThreadCount, new NamedThreadFactory("UserPool")); for (int i = 0; i < totalRuns; i++) { testData[i] = new TestStatusData(); // Detailed in previous slide int userIndex = i % usernames.length; //randomize username/paswords to be used testData[i].setTUsername(usernames[userIndex]); TestFlexorAuthThread handler = new TestFlexorAuthThread(testData[i],usernames[userIndex],passwds[userIndex]); Future<TestFlexorAuthThread> serverTask = userThreadPool.submit(handler, handler); taskMap.put(serverTask, handler); } awaitTermination(); userThreadPool.shutdown();

Good

Serves the purpose without COTS tools Complete control and Minimal Java needed Can be cronned for run overnight builds Underlying HtmlUnit continuously updated and supports new frameworks Can simulate multiple Browser in single Load test

HtmlUnit Simulates IE7, IE8, Firefox 2, Firefox3 Can configure each thread for different browser Load generation code can be re-used for other web apps Used for client/server applications (SFTP, FTP, SMTP)

Generic load generation

Bad

Does not have reporting UI

Working on incorporating into JMeter Investigating incorporating into Grinder HtmlUnit does not support all AJAX libraries (RIAs) Adobe Flash, Silverlight, WPF

Scalability: Limited by JVM and CPU availability

Adding additional hardware helps, or making tests distributed Its an issue with every load/functional test tool Very basic programming needed Use Canoo WebTest instead of HtmlUnit We are currently working on this

Hard to maintain as UI/ object properties change

Not very friendly for non-programmer

Lessons Learned

JVM tuning sponse haUse as powerful machine as possible (multi core)

JVM assigned to max possible (32-bit, 64-bit) To be incorporated into load generation framework

All the HTMLUnit tests have to be thread-safe

Fine-grained Logging needed for trouble shooting

E.g. To know what response server is sending back?

Every possible exception better be handled in HtmlUnit code and Thread code for failure analysis

Other ways to perform load tests

Using TestNG ThreadPool framework for HtmlUnit tests WebTest + Groovy multi-threading Jmeter Groovy Sampler+ WebTest (needs extra effort)

Future Work

Use Groovy with WebTest Canoo to make tests nonprogrammer friendly Work on potential integration with Jmeter Isolate Load generation and load execution (LoadRunner/Grinder approach of Controller + VUgen) For large-scale tests

Additional Virtual Machine support Cloud based support

Develop distributed version of tool for CLOUD IN Progress!

Later : Develop WebUI for configuration purposes

Questions, Comments and Suggestions?

Gopals Load testing Blog:

http://webloadtest.blogspot.com
Please post comments and suggestions Updates on load test framework(s) will be posted

27

References
[1] http://www.crn.in/Resources/Docs/Aztecsoft_Whitepaper_Perf ormance_Testing_AJAX-based_Applications.pdf [2] http://www.adaptivepath.com/ideas/e000385

28