PERFORMANC

E TESTING
SCOPE

Contents
Introduction................................................................................................................ 3
Choosing an appropriate performance testing tool....................................................3
Testing Tool Architecture.......................................................................................... 3
Designing an appropriate performance test environment..........................................4
There are three levels of preference when it comes to designing a test environment:
............................................................................................................................. 4
Injection Capacity................................................................................................. 4
Certain other situations will also affect how many injectors you will need to create
application load.................................................................................................... 5
Addressing Different Deployment Models............................................................5
Environment Checklist......................................................................................... 5
Software Installation Constraints..........................................................................6
Setting realistic and appropriate performance targets...............................................6
Making sure application is stable enough for performance testing...........................6
Obtaining a code freeze............................................................................................. 6
Identifying and scripting the business-critical transactions........................................6
Transaction Checklist............................................................................................ 7
Transaction Replay Validation............................................................................... 7
What to Measure?................................................................................................ 7
To Log In or Not to Log In...................................................................................... 7
Peaceful Coexistence........................................................................................... 7
Providing sufficient test data of high quality..............................................................7
Input Data............................................................................................................ 7
Runtime Data....................................................................................................... 8
Data Security....................................................................................................... 8
Ensuring accurate performance test design...............................................................8
Pacing Think Time................................................................................................ 9
Injection (App Load) Profile..................................................................................9
Setting the Number of Virtual Users per Transaction...........................................9
Deciding on Performance Test Types....................................................................9
Load Injection Point of Presence.........................................................................10
Putting It All Together......................................................................................... 10
Identifying the server and network monitoring key Performance Indicators (KPIs). .10
Allocating enough time to performance test effectively...........................................10

Introduction
Performance awareness should be built into the application life cycle as early as possible.
Questions:

How many end users will the application need to support at release? After 6
months, 12 months, 2 years?
Where will these users be located, and how will they connect to the application?
How many of these users will be concurrent at release? After 6 months, 12
months, 2 years?

The answers then lead to other questions, such as:

How many and what specification of servers will I need for each application tier?
What sort of network infrastructure do I need to provide?

Functional Testing
Functional testing involves testing the application against the business
requirements. The goal of functional testing is to verify that the application is
behaving the way it was designed to.
Functional testing ensures that your software is ready to for release to the public.
It also verifies that all the specified requirements have been incorporated.
There are two major categories of functional testing: positive and negative
functional testing. Positive functional testing involves inputting valid inputs to see
how the application responds to these and also testing to determine if outputs are
correct. Negative functional testing involves using different invalid inputs,
unanticipated operating conditions and other invalid operations.

Non-functional Testing
While functional testing is concerned about business requirements, non-functional
testing is designed to figure out if your product will provide a good user
experience. For example, non-functional tests are used to determine how fast the
product responds to a request or how long it takes to do an action. Examples of
non-functional tests include:

Load/Performance testing

Compatibility testing

Localization testing

Security testing

Reliability testing

Stress testing

Usability testing

Compliance testing

Choosing an appropriate performance testing tool

The aim of any automated performance testing tool is to simplify the testing process. This
is normally achieved by providing the ability to record end-user activity and to render this
data as transactions or scripts. The scripts are then used to create load testing sessions
or scenarios that represent a mix of typical end-user activity. These are the actual
performance tests and, once created, they can easily be rerun on demand, which is a big
advantage over any sort of manual testing. Another huge advantage over manual testing
is the ability to quickly correlate performance data from various sources (such as the
servers, network, and application response time) and then present this in a single view.
This information is normally stored for each test run, allowing easy comparison of the
results of multiple test executions.
Talian using Rational Performance Tester

Testing Tool Architecture

Scripting module
Enables recording of end-user activity and may support many different middleware
protocols. Allows modification of the recorded scripts to associate internal/external
data and configure granularity of response-time measurement.

Test management module

Allows the creation and execution of load test sessions or scenarios that represent
different mixes of end-user activity. These sessions make use of nominated scripts
and one or more load injectors.

Load injector(s)
Generates the loadnormally from multiple workstations or servers, depending on
the amount of load required.

Analysis module
Provides the ability to analyze the data collected from each test execution. This
data is typically a mixture of autogenerated reports and graphical or tabular
reporting. There may also be an expert capability that provides automated
analysis of results and highlights areas of concern.

Designing an appropriate performance test environment

The number and specification of servers (probably the most common reason)
If we cannot replicate the numbers of physical servers at each application tier then
do try to match the specification of the live servers
Berapa jumlah server dan spesifikasinya?
Apakah bisa mendapatkan arsitektur servernya?
Apakah memungkinkan testing server sejumlah live server / prodection?

Apakah testing server yang ada spesifikasi testing server sudah sama dengan live
server ?
Bila yang jumlah testing server tidak identic dengan live production, maka paling
tidak testing server punya spesifikasi yang mirip dengan live production; tinggal
karakteristik individual testing server mewakili karakteristik live server per
individual; apakah bila diketahui karakteristik individual per server bisa diprediksi
karakteristik keseluruhan server? (horizontal scaling)

Bandwidth and connectivity of network infrastructure

Apakah memungkinkan mendapatkan arsitektur network / infrastrukturnya?
Bagaimana model pengaturan jaringannya, apakah LB atau SDN ?
Bagaimana split antara primary server dengan backup server?
Apakah Server Apps-Database primary dengan Server Apps-Backup database
mempunya alokasi bandwith sama?
Bagaimana karakteristik network di tempat dilakukannya pengetesan dengan
karakteristik network server maupun database?
Apakah dalam sejarahnya ada terjadi anomaly connectivity di jaringan ? Dalam
keadaan anomaly (tiba-tiba terputus paling cepat berapa lama? Paling lambat
berapa lama?)
Apakah ada riwayat anomaly performansi jaringan antara Server apps dengan
database?
Apakah konektifitas menimbulkan fragility pada system?

Number of application tiers

In many cases, the number of application tiers has greater impact on the validity of
performance testing than simply the number of servers.
Apakah testing server sudah mempunyai application tiers yang tepat sama dengan
production server?
Example:
-

A User Interface (UI) for clients to use the system

An embedded reporting engine for report generation
A cron task engine for scheduled functions
An escalation process engine for escalation activities based on
business rules
An Integration Framework (IF) to allow real time data integration with
external products
EAM_Toolkit_Archit
ectural_Decisions_V2.2.pdf

Sizing of application databases

The size of the test environment database should closely approximate the live one;
otherwise, the difference will have a considerable impact on the validity of
performance test results

Performance Impacts of Data Volume

The amount of data stored in a database has a great impact on its performance. It
is usually accepted that a query becomes slower with additional data in the
database. But how great is the performance impact if the data volume doubles?
And how can we improve this ratio? These are the key questions when discussing
database scalability.
As an example we analyze the response time of the following query when using
two different indexes. The index definitions will remain unknown for the time
beingthey will be revealed during the course of the discussion.
SELECT count(*)
FROM scale_data
WHERE section = ?
AND id2 = ?
The column SECTION has a special purpose in this query: it controls the data
volume. The bigger the SECTION number becomes, the more rows the query
selects. Figure 3.1 shows the response time for a small SECTION.
Figure 3.1. Performance Comparison