Stress Testing Web Intelligence and Crystal Reports

Stress Testing Web Intelligence
and Crystal Reports
Applies to:
SAP BusinessObjects XI 3.1. For more information, visit the Business Objects homepage.
Summary
This paper discusses stress testing an SAP BusinessObjects Enterprise system to compare the efficiency of
Web Intelligence and Crystal Reports. For both Web Intelligence and Crystal Reports it looks at how
document size and number of users can affect the impact on system resources. The results demonstrate that
Crystal Reports is a better fit for deployments where there are a large number of us ers who are viewi ng and
refres hing pre-built reports.
Author:
Al Gulland
Company: gulland
Created on: 8 November 2010
Author Bio
Al Gulland is a senior BI consultant for IBM Global Services and has been active in Business Intelligence for
over 15 years, specializing in SAP BusinessObjects and data warehouse solutions. Al maintains a blog at
www.gulland.com and has posted many articles on SAP BusinessObjects.
SAP COMMUNITY NETWORK

2010 SAP AG
SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com

1
Stress Testing Web Intelligence and Crystal Reports
Table of Contents
Introduction............................................................................................................................................. 3
Test Environment .................................................................................................................................... 3
Test Documents Used .......................................................................................................................... 3
Stress Test Approach .............................................................................................................................. 4
Performance Monitoring ....................................................................................................................... 4
Test Results............................................................................................................................................ 5
Conclusion.............................................................................................................................................. 5
Appendix 1: Server Configuration ............................................................................................................. 6
Web Intelligence Server Configuration ................................................................................................... 6
Crystal Reports Server Configuration .................................................................................................... 7
Properties for Crystal Reports Processing Server ................................................................................................................... 7
Appendix 2: Detailed Results ................................................................................................................... 7

Test: Web Intelligence Small Document ................................................................................................ 7
Test: Web Intelligence Medium Document ............................................................................................. 9
Test: Crystal Report Small Doc ument .................................................................................................. 10
Test: Crystal Report Medium Document .............................................................................................. 12
Relat ed Content .................................................................................................................................... 14
Boest .14
Linear Regression Analysis ................................................................................................................ 14
Other Related Topics ......................................................................................................................... 14
Disclaimer and Liability Notice................................................................................................................ 15

2010 SAP AG

2
Introduction
The overall objecti ve of this exercise was to compare Web Intelligence to Crystal Reports under identical
stress test conditions. Here the stress test was to have a set number of users viewing and refreshing a
document while at the same time measure the memory and CP U consumption on the SAP BusinessObjects
Enterprise (BOE ) server. The test was executed for both a Web Intelligence document and a Crystal Report
of the same size. In both cases the number of users was increased and measures of memory and CP U
usage were taken. The Boe st stress testing application was used to execut e the stress tests and to analyze
the results.
This article reviews the testing environment, how the stress tests were executed, discussion of the test
results and finally conclusions.
Test Environment
The same BOE server was used for both the Web Intelligenc e and Crystal Reports stress tests. As the tests
were a comparison of the two report formats it wasn't necessary to have a high end capacity server and so a
virtual machine was used to host the server. The environment specification is as follows.
OS
Windows 2003 Server
CPU
2 cores from a Intel Core i5 750 of 2.67 Ghz
RAM
2 GB
Virtualisation Technology
VM Player 3.1
SAP BOE Version
XI 3.1 sp3
Database
SQL Server Express 2005
Boest was installed on a Windows 7 desktop that had access to the BOE Server. The same desktop was
host to the virtual machine running the BOE server.
Test Documents Used
The test documents used for both Web Intelligence and Crystal Reports were created to be as much as
possible of the same size and complexity in order to ensure a valid comparison test. In this case the same
SQL statement was used in both the Web Intelligence document and Crystal Report. This ensured that not
only the database ex ecution of the SQL statement would be the same but that the number of rows in the
report would be the same. In addition the display and formatting of the data in the reports was the same for
both document types which was a simple table of dat a.
In both cases a small and a medium sized document was created where the difference in the document size
is purely based on the number of rows returned by the SQL statement.
The SQL used by all reports is,
SELECT
ID, Store, Product, Purchase_Date, Quantity, On_Sale
FROM
SalesDataMartDemo.dbo.TBL_LARGE2
WHERE
ID <= 10000
ORDER BY
Store, Product, Purchase_Date

2010 SAP AG

3
The only difference between small and medium reports is the value used in the condition. For small reports
the condition is ID <= 10000 and for medium it is ID <= 100000.
The following reports are used in these tests.
Name
Type
Size on disk (kb)
Number of Rows
PerfTestWebi_Small
Web Intelligence
224
9,991
PerfTestWebi_Medium
Web Intelligence
1,921
99,991
PerfTestCrystal_Small
Crystal Report
168
9,991
PerfTestCrystal_Medium
Crystal Report
1,214
99,991
For comparison this is same amount of dat a saved as CSV where the file was generat ed from a SQL Server
export.
Name
Type
Size (kb)
Number of rows
Small
CSV
522
9,991
Medium
CSV
4,266
99,991
Stress Test Approach

Boe st is an application designed to stress test an SAP BOE system by simulating multiple parallel us ers all
viewing and refreshing reports. It was used here to stress test both Web Intelligence and Crystal Reports
where the method used was the same in both cases and is described below.
The test involved logging on an initial set of 5 users and once they have all logged on they then
simultaneously view and refresh a test document. Once complet e the users were all logged off. While this
test is being executed Windows Performance Monitor was used to track the memory and CP U being used by
the Web Intelligence or Crystal Report processes on the server. For the test the maximum memory reached
is noted along with maximum percentage CPU reached. As well as maximum CPU the average percentage
CPU was also meas ured during the test in order to obtain a measure of overall load.
The test of five users was repeated five times, and each time the memory and CP U usage was logged. The
purpose of this was to obtain an average of the memory and CPU so as to eliminat e any unexpected spikes
which could be caus ed by other processes on the system.
After the test of five users was complet e the testing was repeated for 10 users and this was also repeated
five times to take an average. The test was repeat ed again for 15 users and then 20 and so on up to 50
users in total.
This process was performed for both the small and medium Web Intelligence documents and then again for
the small and medium Crystal Reports.
Performance Monitoring
Windows Performance Monitor was used to measure the amount of memory and percent age CPU us ed by
the Web Intelligence and Crystal Reports proc esses. The server process that is used by Web Intelligence is
the WIServerProcess and the counters tracked were "% Processor Time" for Percentage CPU and
"Working Set" for memory.
For Crystal Reports the main Crystal Report proc ess crproc will instantiate a further 5 processes across
which the processing is spread. For each of these processes (identified by crproc#1, crproc#2 etc) the "%
Processor Time" and "Working Set" were measured from which the results were aggregated together.

2010 SAP AG

4
Test Results
The results were analyzed by first taking an average across the 5 repeated tests. This average was then
plotted against the number of users to produce a chart of memory usage by number of users and also a
chart of CPU us age by number of users (see appendix). The chart of number of users versus memory was a
good fit for a straight line and so in eac h case linear regression analysis was used to determine the best fit
for a straight line through these points. This straight line can then be used to determine a relation between
number of users and memory which can then be extrapolated to find a maximum load for the system.
The table below summarizes the results. The initial memory is the intersect of the straight line wit h the
vertical axis and is equivalent to the amount of RAM consumed prior to start of test. The maximum memory
is the maximum memory reached during the test and this was always when the full load of 50 users was
reached. The next column displays the gradient of the straight line and this is the inc rease in RAM per user.
The final column is the maximum percent age CPU reached during the test, note this is out of 200% as the
system was configured with two CPUs.
Test
Initial Memory
(m b)
Maximum
Memory (mb)
Increase in RAM Maximum CPU

per user
Web Intelligence Small Document
63.6
168.7
2.17
190.31
Web Intelligence Medium Document
44.2
502.5
8.31
195.02
Crystal Report Small Document
257.5
292.5
0.78
66.87
Crystal Report Medium Document
255.5
291.9
0.81
78.75
From this it can be seen that although Web Intelligenc e starts from a lower bas e the increase in memory per
user is much greater than Crystal Reports and for the medium doc about 0.5GB of RAM is reached for 50
users. For Crystal Reports 50 users viewing a report of equivalent size consumes 291MB of RAM.
It should also be noted that Web Intelligenc e places more demand on CPU usage indicating more
processing required to render the displayed report.
Conclusion
The above analysis clearly shows that there is, as you would expect, a direct correlation between increasing
users and an increase in memory for Web Int elligence and Crystal Report types. However although Crystal
Reports has a higher initial memory the rate of increase in memory is much less than that of Web
Intelligence such that for a large number of users Crystal Reports uses less memory than Web Intelligence.
CPU usage is also different and by inspection of the graphs it can be seen that Crystal Reports is less CPU
intensive than Web Intelligence.
From this it can be concluded that Crystal Reports is a better solution for deployments with a high number of
users who need to view and refresh prebuilt reports and that Web Intelligenc e is more efficient for a low
number of users.
Does this mean then that Web Intelligence should never be used and that Crystal Reports is best used in all
scenarios ? No. These are two different products that meet two similar but different requirements. Web
Intelligence is designed for a business user to create new reports and perform some basic data analysis.
Crystal Reports on the other hand has been designed for large volume report generation and distribution.
This exercise was designed to compare Crystal Reports to Web Intelligence in a tightly controlled
environment. The above values are not expected in a real world scenario and the results should not be
interpreted as a realistic measure of demand in a production system.
Report size has a significant impact on system demand and in a production environment it is expected that
there will be a wide range of reports of different sizes and complexity. Furthermore, user demand will not be
regular and there will be peak times when users are accessing the system.
These factors and more will affect the maximum load that a system can comfortably handle and a more
detailed capacity planning exercise should be undertaken when designing a deployment.

2010 SAP AG

5
Appendix 1: Server Configuration

For both Crystal Reports and Web Intelligence caching was reduced so that memory usage can be
measured per us er per doc ument. Full server settings are as follows.
Web Intelligence Server Configuration

The following are the settings used for the Web Intelligence Proc essing Server.
Property
Setting
Document cache cleanup interval (minutes):
120
Binary Stream Maximum Size (MB):
50
Cache Timeout (minutes):
4370
Memory Maximum Threshold (MB):
1800
Idle Document Timeout (seconds):
100
Server Polling Interval (seconds):
100
Universe Cache Maximum Size (Universes):
20
Disable Cache Sharing:
Off
Images Directory:
Maximum Document Cache Size (KB):
1000000
Output Cache Directory:

Maximum Documents per User:
Allow Document Map Maximum Size Errors:
On
Maximum Documents Before Recycling:
50
Maximum Connections:
100
Idle Connection Timeout (minutes):
Maximum List Of Values Siz e (entries):
50000
Enable List Of Values Cache:
On
Enable Real-time Cache:
Off
Maximum document cache reduction space (%):
70.00%
Maximum Documents in Cache:
Memory Upper Threshold (MB):
1500
Enable Memory Analy sis
Off
Enable Document Cache
On
List Of Values Batch Siz e (entries):
1000
Maximum Custom Sort Size (entries):
100
Memory Lower Threshold (MB):
1000
Timeout Before Recycling (seconds):
1200
Maximum Character Stream Size (MB):

2010 SAP AG

6
Crystal Reports Server Configuration

Properties for Crystal Reports Processing Server
Property
Setting
IdleJobTimeout
Database Records Read When Preview ing or Refreshing
Maximum Lifetime Jobs Per Child:
1000
Viewer Refresh Alw ays Yields Current Data
On
Share Report Data Betw een Clients
On
Idle Connection Timeout (minutes):
20
Maximum Concurrent Jobs (0 for automatic):
Oldest On-Demand Data Given to Clients (seconds):
Maximum Number of Prestarted Children
Allow Report Jobs to Stay Connected to the Database until Off

the Report Job is Closed
Appendix 2: Detailed Results

Below are full results for each test. Each test of a s et number of users is repeated five times in order to
obtain the averages. The average CPU is the average of the average percentage CPU during the five
repeated test runs and the average maximum CPU is the average of the maximum percentage CPU
obtained during each test. The average memory is the average of the maximum memory reached during
each test.
Test: Web Intelligence Small Document
The test results for users viewing and refreshing the small Web Intelligence document are displayed in the
table and graphs below. The inc rease in memory is linearly correlated to the increase in number of
simultaneous users where each user increases memory usage by 2.174MB on average.
Percentage CPU used by the Web Intelligence process quickly reaches 180% after 20 simultaneous users
which is pretty much full load with the remaining perc entage used by other system proc esses. The average
percentage CP U used slowly increases indicating that more CPU cycles are required to handle the inc reased
user demand.
Number Users
Average CPU
Average Max CPU
Average Memory
14.08
88.44
74.34
10
24.28
158.12
86.30
15
30.51
171.25
96.21
20
35.20
183.75
106.89
25
39.36
180.31
114.55
30
43.24
188.12
130.41
35
45.78
185.00
137.69
40
48.47
188.43
155.01
45
51.28
190.31
163.68
50
52.51
189.06
168.73

2010 SAP AG

7
Linear regression analysis of number of users versus average memory gives the following metrics,
Gradient:
2.174
Intersect:
63.598
Correlation Coefficient:
0.997
Graph of memory consumption vs. number of users for the small Web Intelligence document
Graph of average CPU (lower line) and maximum CPU vs. number of users for the small Web
Intelligence document

2010 SAP AG

8
Test: Web Intelligence Medium Document

The test results for users viewing and refreshing the small Web Intelligence document are displayed in the
table and graphs below. In this test the gradient is higher which is due to the fact that a larger document is
being used and so each additional user viewing a larger document has a higher demand on system memory.
A second difference is that the perc entage CP U usage is also much more than with the smaller document in
both maximum and average percentage CP U used.
Number Users
Average CPU
Average Max CPU
Average Memory
48.96
183.74
101.54
10
77.52
190.62
138.43
15
90.11
191.24
163.31
20
98.9
190.96
207.18
25
104.56
193.43
248.38
30
110.52
192.49
273.9
35
115.17
195.02
325.73
40
114.25
193.75
355.32
45
115.7
192.81
410.15
50
118.1
194.68
502.53
Linear regression metrics,

Gradient:
8.306
Intersect:
44.236
0.989
Graph of memory consumption vs. number of users for the medium Web Intelligence document

2010 SAP AG

9
Graph of average CPU (lower line) and maximum CPU vs. number of users for the medium Web
Intelligence document
Test: Crystal Report Small Document

The test results for users viewing and refreshing the small Crystal Report are displayed in the table and
graphs below. Here the increase in memory per increase in user is much less than that of the equivalent
report in the Web Intelligenc e tests. The increase per user is 0.780MB compared to 2.174MB for Web
Intelligence. The initial memory usage is higher than the maximum memory reached in the Web Intelligence
test using the small document.
CPU usage is also significantly lower than the equivalent test in Web Intelligence where the maximum
percentage CP U reaches a peak of about 70%.
Number Users
Average CPU
Average Max CPU
Average Memory
3.16
17.19
257.43
10
4.04
25
264.48
15
5.71
35.94
270.04
20
6.66
46.56
274.37
25
8.33
56.25
279.28
30
8.88
64.37
283.61
35
11.24
66.87
286.48
40
8.41
55.62
289.37
45
10.88
67.81
291.71
50
10.94
63.13
292.53

Gradient:
0.780
Intersect:
257.469
0.980

2010 SAP AG

10
Graph of memory consumption vs. number of users for the small Crystal Report
Graph of average CPU (lower line) and maximum CPU vs. number of users for the small Crystal
Report

2010 SAP AG

11
Test: Crystal Report Medium Document

The test results for users viewing and refreshing the medium Crystal Report are displayed in the table and
graphs below.
Number Users
Average CPU
Average Max CPU
Average Memory
2.67
14.37
255.24
10
2.98
19.69
263.06
15
4.56
28.75
268.57
20
6.13
34.06
273.48
25
8.56
57.19
277.41
30
8.48
52.18
283.42
35
9.55
55.31
286.22
40
10.51
60.94
289.05
45
10.28
71.56
290.8
50
12.81
78.75
291.94

Gradient:
0.814
Intersect:
255.547
0.978
Graph of memory consumption vs. number of users for the small Crystal Report

2010 SAP AG

12
Graph of average CPU (lower line) and maximum CPU vs. number of users for the small Crystal
Report

2010 SAP AG

13
Related Content
Boe st
Boest is a Java application built specifically for stress testing the execution of Crystal Reports and Web
Intelligence documents in an SAP BusinessObjects Enterprise deployment. Further details of the application
and download can be found at Sourceforge.net.
Linear Regression Analysi s
The linear regression analysis was calculated using Web intelligence using the method discussed in the
article Linear Regression Analysis and Web Intelligence.
Other Related Topics
BusinessObjects Enterprise XI Release 2 Sizing Recommendations Guide. Although this was written for XIr2
much of the content remains valid for XI3.
Selecting the Right Reporting Tool A comparison of Web Intelligence and Crystal Reports
Business Objects Community: Crystal Reports
Business Objects Community: Webi
For more information, visit the Business Objects homepage

2010 SAP AG

14
Disclaimer and Liability Notice

This document may discuss sample coding or other information that does not include SAP offic ial interfaces and therefore is not
supported by SAP. Changes made based on this information are not supported and can be overwritten during an upgrade.
SAP w ill not be held liable for any damages caused by using or misusing the information, code or methods suggested in this document,
and anyone using these methods does so at his/her own risk.
SAP offers no guarantees and assumes no responsibility or liability of any type w ith respect to the content of this technical article or
code sample, including any liability resulting from incompatibility betw een the content within this document and the materials and
services offered by SAP. You agree that you will not hold, or seek to hold, SAP responsible or liable w ith respect to the content of this
document.

2010 SAP AG

15

Stress Testing Web Intelligence and Crystal Reports

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Stress Testing Web Intelligence and Crystal Reports

Transféré par

Droits d'auteur :

Formats disponibles

Stress Testing Web Intelligence

and Crystal Reports

SAP COMMUNITY NETWORK

SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com