Siebel 7 Performance and Scalability

Inside the Siebel Server

Richard Sands Siebel Expert Services
Siebel Systems 2003 Do not distribute or re-use without permission

Siebel 7 Performance and Scalability

Inside the Siebel Server Introduction to Siebel Architecture

How components scale

Within servers Across servers

Siebel Performance
Hints Monitoring Diagnosis

Performance and Scalability

Architecture Overview Component Scalability Scalability Across Components Network Scalability Performance Optimization Performance Management

Siebel 7 Infrastructure Overview

Wireless Web
Browser User Interface
WAP Gateway Server

Connected Web User (Employee)

Browser User Interface

Connected Web User (External)

Browser User Interface

PDA

Mobile (Disconnected) Web User

Browser User Interface Object Manager

Web Server(s)
Siebel Web Server Extension

Data Manager
SIEB SYNC
Local DB

Gateway Name Server

Central Dispatch Load Balancer

Siebel Remote External Applications Siebel eAI

Siebel Enterprise Server

Voice Interaction

Siebel Replication
Regional Siebel DB Server

Object Manager Data Manager

Object Manager Data Manager

Email Interaction

Central Siebel DB Server

Major Client Types

All accessed through a browser High Interactivity (Employee facing)

Very demanding on browser Can only run on strictly defined browser configurations Rich user interface

Standard Interactivity (Customer facing)

Less demanding on browser Can run on wide variety of browsers Standard web user interface

Mobile Client
Has local copy of Siebel database User-specific subset of data Local server functionality Quasi-web server Business Object and Data Manager functionality Uses High Interactivity interface

Siebel Enterprise Server SWSE

Web Server
SWSE

Gateway Server Gateway Name Server Resonate Central Dispatch

Siebel Web Server Extensions Web Server Plug-In Manages communications to Siebel Enterprise Includes cache for static files (images, etc)

Enterprise Server Siebel Server Component Siebel Server Component

Siebel Enterprise Server Gateway Server

Web Server
SWSE

Serves as a single entry point Includes Siebel Gateway Name Server

Siebel Enterprise configuration data (static) Siebel Enterprise operations data (dynamic)

Gateway Server Gateway Name Server Resonate Central Dispatch

Optionally includes Resonate Central Dispatch

Load balancing

Enterprise Server Siebel Server Component Siebel Server Component

Siebel Enterprise Server Gateway Name Server

Web Server SWSE

Gateway Server Gateway Name Server Resonate Central Dispatch

Holds Enterprise Configuration Stores component definitions, parameters, and connectivity information
Stored in siebns.dat file Provides Connectivity information when Resonate not used

Enterprise Server Siebel Server Component Siebel Server Component

Dynamically registers Siebel Server and component availability

Siebel Enterprise Server Resonate Central Dispatch

Web Server SWSE

Gateway Server Gateway Name Server Resonate Central Dispatch

Load Balances Object Manager Components Provides resilience for load balanced components Provides session management Simplifies firewall configuration Not always needed

Enterprise Server Siebel Server Component Siebel Server Component

Architecture Overview Siebel Server

Web Server
SWSE

Gateway Server Gateway Name Server Resonate Central Dispatch

Framework for running server components Obtains configuration information from the Gateway Name Server Runs as a Windows service Siebel Enterprise Server is a logical grouping of Siebel Servers

Enterprise Server Siebel Server Component Siebel Server Component

Architecture Overview Server Components

Web Server
SWSE

Gateway Server Gateway Name Server Resonate Central Dispatch

Enterprise Server Siebel Server Component Siebel Server Component

A type of program, executed as a task Examples: Object Manager Handling client requests Workflow Process Manager Processing Workflows File System Manager Handling access to Siebel File System

Architecture Overview Server Component Types

Different types of component run in different ways: Background

Background mode components execute tasks to perform background operations for the Siebel Server. After a background mode component task starts, it runs until you explicitly stop the task, or until the Siebel Server itself is shut down.

Interactive
Interactive mode components start tasks automatically in response to client requests. Interactive mode component tasks execute for as long as the client maintains the session, and end when the client disconnects.

Batch
Batch mode components execute tasks in response to requests. Batch mode component tasks execute until they finish processing. Internal batch tasks are initiated by other Siebel Server tasks External batch tasks are initiated by Server Manager

Architecture Overview Component Execution Platforms

Different types of component execute in different platforms: Single Threaded

Single threaded components have one execution stream per process. So each operating system process supports a single Siebel Task. i.e. EIM

Multi-threaded
Multi-threaded components have multiple execution streams within a single process.. So each operating system process can support multiple Siebel Tasks. i.e. Object Managers

Performance and Scalability

Architecture Overview Component Scalability Scalability Across Components Network Scalability Performance Optimization Performance Management

Component Scalability

How Components Scale in a Siebel 7 Enterprise Scaling within a server

Multi-threaded components Single-threaded components

Scaling across servers

Load balancing Resonate central dispatch Load sharing Server Request Broker Server Request Processor

Scaling Within a Siebel Server

Single Threaded Components

Create multiple processes (tasks) Some components are limited i.e. Transaction Processor max 1 per server Workflow Monitor Agent max 1 per policy group per Enterprise Can be started manually, through Server Request Broker, or automatically (Default Tasks parameter)

Multi-Threaded Components
Create multiple threads &/or processes (tasks) Control distribution through component parameters

Multi-Threaded Components

Can have multiple processes as well as multiple threads Important to control ratio of threads to processes
Can have major impact on performance Determined primarily by rate of switches between threads 100:1 good starting point for Client Object Managers Assumes 30sec think time For 15 sec think time use 50:1

Can set additional processes to spawn on demand

Will always start minimum number specified Will start additional processes as needed to maintain process:thread ratio Limit on maximum number of processes

Memory Scalability

Multi-Process, Multi-Threaded model

All threads in a process share the same memory space Each Process has a separate memory space

No single process needs a large memory space

Operating system manages memory allocation If a single process needs more than 1GB theres normally something wrong No need for large process memory model (/3GB switch) Can use Microsoft PAE for access to large memory capacities Operating system can allocate memory to each process

Multi-Threaded Component Parameters

Typically set per component Maximum number of tasks (MaxTasks)

Maximum number of Tasks per component per server One thread per task Some additional background system threads - not counted by MaxTasks

Maximum number of Multi-Threaded servers (MaxMTServers)

An MTServer is a multi-threaded component process This defines the maximum number of MTServers per component per server

Minimum number of Multi-Threaded servers (MinMTServers)

This defines the minimum number of MTServers per component per server Sets the number of MTServers started on server startup

Configuring the Object Managers

Set MaxTasks = peak (concurrent users + anonymous users)

Anonymous users are used for login screens before user authenticates Typically set to 10%-15% of concurrent user count

Set MaxMTServers = MaxTasks / 100

An MTServer is equivalent to single instance of Object Manager 100 : 1 ratio is assuming average 30 second think time between user operations If average user think time is 15 seconds then ratio is 50 : 1 ( 50% of 100:1) If average user think time is 60 seconds then ratio is 200 : 1 (200% of 100:1)

Set MinMTServers = MaxMTServers

Setting MinMT Servers < MaxMTServers may cause delay of service for new users as MTServer gets initialized.

Multi-Threaded Component Parameter Example

Object Manager configuration for 800 Call Center users

Concurrent Users 800 Call Center Users Anonymous Users 120

15%

Object Manager Level

MaxTasks 920 1000 100:1 MaxMTServers 10

Round up to maintain 100:1 ratio

MinMTServers 10

Scaling Across Siebel Servers

Depends on how component task initiated

Component Group Assignment
Which servers a component is available on

Object Managers
Load balanced (Resonate Central Dispatch) Not valid for eConfigurator Object Manager

Siebel Remote
Manual allocation of mobile client to Siebel Server

Server Requests
Server Request Broker &/or Server Request Processor

Manual Requests (start task)

Manually specified server

Special cases
CTI eConfigurator

Multi-Threaded Component Scalability

Vertical Scalability

Sales Object Manager

Sales Object Manager

Sales Object Manager

Sales Object Manager

Sales Object Manager

Sales Object Manager

Sales Object Manager

Sales Object Manager

Sales Object Manager

Siebel Server
Enterprise Server

Siebel Server

Siebel Server

Horizontal Scalability

Load Balancing

Web Client

Web Client

Web Client

Web Client

Web Client

Web Client

Load Balancing
Web Server + SWSE Web Server + SWSE

Resonate Load Balancing

Thread Process Server

Sales Object Manager

Sales Object Manager

Sales Object Manager

Sales Object Manager

Siebel Server

Siebel Server

Enterprise Server

Performance and Scalability

Architecture Overview Component Scalability Scalability Across Components Network Scalability Performance Optimization Performance Management

Server Request Broker & Server Request Processor

Server Request Broker (SRBroker)

Used to start synchronous Siebel Server tasks Server Request Broker & Server Manager are the only components which directly start tasks. New in Siebel 7 Replaces Server Request Manager (SRMSynch) in Siebel 2000 Background component Multi-threaded component Need to set MaxTasks accordingly

Server Request Processor (SRProc)

Used to start asynchronous Siebel Server tasks Manages queued requests Calls SRBroker to manage task execution Background component

Server Request Broker

Accepts requests from other Components Will try to service request locally
If component is available on same Siebel Server then this is always used

If not available locally then will use other Siebel Servers

Maintains internal table of components available on each Siebel Server Will route requests round Siebel Servers in turn (round-robin)

Multi-threaded component
May need to increase MaxTasks

Should always be running on all servers

Siebel Server Server Request Broker

Assignment Manager
Server Request Broker Workflow Process Manager

Run Object Assignment Manager TaskAssignment Is available on this Run server? Server Server Request Assignment onRequest Broker local server Broker Workflow Process Manager

Assignment Manager

Siebel Server Server Request Broker

Assignment Manager
Server Request Broker Workflow Process Manager

Run Object Workflow Is Workflow Manager Process Manager Server Request Broker

Assignment Manager

available on Server this server? WhichRequest other serversBroker have workflow Choose server online? on roundWorkflow robin basis Process
Manager

Server Request Processor

Processes asynchronous requests Request submitted by creating record in table

S_SRM_REQUEST

Server Request Processor reads from table

Request must: Be active (reached activation time) Not be specified for a different Siebel Server Not being processed by other Server Request Processor

Eligible requests submitted through Server Request Broker Normally runs on all Siebel Servers

Siebel Server Server Request Processor

S_SRM_REQUEST

SRProc

Sleep Interval

Request Queue
SRBroker

Task

Performance and Scalability

Architecture Overview Component Scalability Scalability Across Components Network Scalability Performance Optimization Performance Management

Connection Pooling

Siebel 7 can pool sessions at two levels:

Web server to Siebel Enterprise
SISNAPI Connection Pooling Multiple SISNAPI (Siebel) sessions through one TCP session Reduces operating system overhead and network traffic Enabled by default Set to 20 Controlled by component parameter: Number of Sessions per SISNAPI Connection

Siebel Object Manager to Database

Database connection pooling SQL traffic for multiple Siebel users through one database session Reduces session overheads on database server Disabled by default Suitable for larger deployments (over 500 concurrent users)

Database Connection Pooling

Connections use native database protocols Some components directly access native protocol
Object Managers

Siebel 7 supports its own database connection pooling

Used for connections from Object Managers Two types of connection Shared for general transactions Specialized for long running

Not always appropriate

Should carefully evaluate tradeoffs Benefits of less database sessions to maintain Risk of database session contention

Database Connection Pooling

Database session uses login for first user to establish session

Database connection pooling controls

Defined as component parameters Set to -1 to disable (default)

Specialized (Dedicated) Database sessions

All valid per component process (MT Server) per Siebel Server MaxTrxDbConns - Maximum number of specialized DB sessions MinTrxDbConns - Minimum number of specialized DB sessions to be kept in pool

Shared Database sessions

Valid per component per Siebel Server MaxSharedDbConns - Maximum number of shared DB sessions MinSharedDbConns - Minimum number of shared DB sessions to be kept in pool

Database Network Architecture

Client Connections Siebel Server
Server Request Processor Object Manager

Shared

Shared

Specialized

Native Database Connectivity (ODBC for SQL Server)

Threads (sessions)

Siebel Database

Processes (components)

Performance and Scalability

Architecture Overview Component Scalability Scalability Across Components Network Scalability Performance Optimization Performance Management

General Siebel Server Optimisation

Disable unused components and component groups

These just use up system resources

Consider using common log file for multi-threaded components that generate too many log files Consider using multiple instances/server tasks for certain activities
e.g. Parallel EIM, multiple Workflow Policy Groups, etc.

Monitor resource utilization by processes and threads

Object Manager Optimization

Enabling View Caching for High Interactive mode

Set in user preferences Faster client response times, by as much as 1 second Better network behavior Expensive, 3MB per view cached Most users use < 10 views / pages Uses LRU mechanism to flush out cache

If not using eAdvisor or browser based eConfigurator

On cfg file set EnableCDA = FALSE

Set Timed Statistics Off Set all Log Levels to 1

EAI Optimization
Minimize fields in Integration Components
Keep XML records small
XML is very verbose and needs to be parsed

Lump several records into one transaction

will achieve higher throughput

If possible keep records to 15 fields

More will reduce throughput

Keep in mind that an XML field translates to BC field

Integration Object initialize Business Objects and Business Components The more fields per record the slower it will process

EAI Optimization
Reduce SessPerSisnConn for EAI Adapters
Do this only if you are going to bombard adapter, i.e. little or no think times

For very high throughput systems where you are dedicating EAI machines
Consider letting the SRB forward BIM requests to dedicated machines BIM processing can become bottleneck if BIM processing is complex May be necessary to have a higher EAI Adapter:BIM machine ratio (i.e. 1:x)

Use Session-Mode for high volume inbound HTTP

Greatly increased throughput No overhead for authentication & session startup for each message

Workflow Process Optimisation

Use Run-Time Events for process automation

Minimize use of Workflow Policies since workflow policy work at the database layer and do not take advantage of the level of abstraction provided with the Object Manager Performance gain since the decision event takes place on the business object layer and so no trigger is created on the table No need for scripting

Workflow Process Distribution

Workflow Policies cant be directly load balanced Asynchronous Workflow Processes are distributed by SRProcessor Synchronous Workflow Process requests are distributed by SRBroker

Application Performance - Configuration

Keep scripting small and tight

Avoid the use of script wherever possible Remember it is still interpreted code and will execute as such Ensure all objects are destroyed after use avoid memory leaks

MVG Applets
Keep to a minimum as they increase SQL complexity Use primary joins as much as possible; reduces number of SQL statements

Use SmartScript only where really required

E.g. Novice Call Center users

Avoid putting external news pages on login page

3rd party web sites can be slow

Application Performance - Configuration

Activate only fields that are absolutely necessary
Become active when displayed; Force Active in BC configuration General rule, the more fields on a page the slower the page Recently saw customer with 750 fields and 80 tables for one applet

Include only manageable set of fields on list applets

Many customers use several dozen fields; not usable and slow performance

Pages with more than three applets will perform slower for HI applications
Reason for this is not really the number of applets, rather the number of fields

PickLists set Long List to TRUE if returning large data sets Tree Applets can be slow
They generate Bill of Materials type queries

Network Performance Siebel Configuration

Browser Validation
Reduces the need for server communications to validate data entry Implement through browser script

Immediate Posting of Changes

Where the Immediate Post Changes flag is set against a field data will be transferred whenever a field is changed Incurs additional round trip with approx 2KB data

Keep to no more than two Applets per View

Minimize Popups
Limit columns in List Applets

Network Performance Siebel Settings

View Caching
View definitions cached in browser memory Requires approx 3MB memory per view Typically around 10 cached views is enough Uses LRU algorithm to maintain cache contents Personalization and Applet Toggles wont use view caching Enabled through Object Manager configuration (.cfg) file setting [SWE] EnableViewCache=TRUE Controlled through: User Preferences > Behaviour > View Cache Size Default: 10

Network Performance Siebel Settings

Compression (Dynamic Content)
Performed by SWSE Do not use web server dynamic compression (application files) Enabled through SWSE configuration file (eapps.cfg) [Defaults] DoCompression = TRUE

Compression (Static Content)

Performed by web server (IIS only)

Network Performance Web Server / Browser Settings

Browser Settings
Dont clear cache except when necessary Ensure Empty Temporary Internet Files Folder when browser is closed option is not enabled.

Network Performance Web Server

Web Server Settings

Use HTTP keep-alive Reduces the need to negotiate TCP sessions for each HTTP message

Network Performance Web Server

Web Server Settings

Set Content Expiration 2 days is typical setting Set through Internet Information Services Administration HTTP Headers > Content Expiration

Network Performance Web Caching

Uncached
GET: icon.gif

25KB
icon.gif RESPONSE: icon.gif DATE: 10/10/03 09:25:08 LAST-MODIFIED: 10/08/03 07:14 DATE: 10/08/03 07:14

Network Performance Web Caching

Cached

GET: icon.gif IF-MODIFIED-SINCE: 10/10/03 09:25

2KB
icon.gif
DATE: 10/10/03 09:25 RESPONSE: Not-modified icon.gif DATE: 10/08/03 07:14

Network Performance Web Caching

Cached with Expiration

0KB
icon.gif
DATE: 10/10/03 09:25 EXPIRES: 10/12/03 14:13:08 icon.gif DATE: 10/08/03 07:14

Performance and Scalability

Architecture Overview Component Scalability Scalability Across Components Network Scalability Performance Optimization Performance Management

Performance Problem Diagnosis

What is the problem?

Slow response times reported by User(s)? Resource utilisation problem reported by Administrator?

Where is the problem?

Who is experiencing the problem? One user, a subset of users, all users? Network problem? Database problem? Siebel Server problem? Web Server problem?

How do we resolve the problem?

Performance Problem Diagnosis continued

Problem Identification
User reports slow response times Identify precise actions carried out by user at the precise time of the problem set up a problem reporting process, with required information to be supplied by user Subset of Users report slow response times Are users on the same LAN segment? Are other users experiencing problems? Are users all performing similar tasks? All users report slow response times Is it specific transactions (views) that are slow? Do all transactions run slowly? System Administrators should check system resource utilisation System Administrator(s) report resource utilisation problems Are users experiencing problems as a result? Which area of the system is experiencing problems

Ask questions of the relevant people need more information than The system seems to be slow

Performance Problem Diagnosis continued

Diagnosis Tools
Database SQL Trace (SQL Server Profiler) Siebel Server Object Manager SQL Trace Web Server SWSE Statistics Page Web Server SWSE Log Web Server Log (e.g. IIS log) Server Resource Utilisation tools Windows: PerfMon, Task Manager

TECHNOTE 361
Memory Utilization in Siebel eBusiness Applications

TECHNOTE 382
How can users generate Performance Monitor information into a log file?

Performance SQL Trace

Use Database SQL Trace or Siebel Component SQL Trace to identify poor-performing queries
Correct Application Configuration Add indexes (except unique indexes) Refresh database statistics (not on Oracle)

Siebel Component SQL Trace

Set SQL Trace Flags parameter Set Event Log Levels to 5 for %SQL% Srvrmgr: change evtloglvl %SQL%=5 for component sccobjmgr_enu server <siebel server name> Scan log file for SQL Statement Execute Time

Need to use SQL Server Profiler for full capture

Siebel uses specialized ODBC connection models Need to reproduce fully in order to get correct query execution plan

Performance Siebel Web Server Logs

SWSE Log
<Install Dir>/SWEApp/log/ssyymmdd_nn.log Set log=details in eapps.cfg file for more detailed logging Set environment variables for full detail SIEBEL_SESSMGR_TRACE=1 SIEBEL_SISNAPI_TRACE=1 SIEBEL_LOGEVENTS=ALL Restart Web Server

Web Server Log

IIS: <Windows Install Dir>\system32\Logfiles\W3SVC1\exyymmdd.log In Internet Services Manager, enable extended logging to include Client IP address, User Name, Time Taken Use to identify long-running HTTP requests or individual user machines experiencing problems

Performance Siebel SWSE Statistics Page

URL: <Application URL>/_stats.swe?verbose=high
<Application URL>/_stats.swe?verbose=high&reset=true to reset stats

Eapps.cfg configuration file:

Allowstats=TRUE SessionMonitor=TRUE Allows individual user session statistics, in Current Sessions section:

Event

Total Time

General Stats (count, mean, std dev) 54 0.2718 0.6826

Frequen cy (mean, std dev) 11.33 59 44.6 484

siebel.TCPIP.None.None://jmullisp4:2320/si ebel/SCCObjMgr_enu/jmullisp4/!1.8c0. 14.6779 Total Time: Time for all users HTTP requests 4023.3ec3708d General Stats-Count: # user HTTP requests SADMIN

General Stats-Mean: Average time for each user HTTP request Frequency-Mean: Average time between each user HTTP request

Performance Siebel SWSE Statistics Page continued

Session Identification
!A.B.C.D A = Siebel Server Id B = Siebel Server Component Process Id C = Siebel Server Component Task Id D = Timestamp All values in hexadecimal Use to map session information to Siebel Server Task logfile

Section Current Operations Processing

Operations in bold type have been running > 10 seconds

Volume Performance Testing

Using Volume Performance Testing Tools, such as

Mercury LoadRunner Segue SilkPerformer

Use to measure
Client Response times under high user load Resource usage under high user load Web Servers, Siebel Servers, Database Server, Network Scalability of Siebel environment

Hardware
Use full duplication of production hardware, if possible If not, scale down numbers of servers proportionately

Database
Tune Object Manager SQL prior to volume performance test exercise Utilise Object Manager SQL trace flags

New Feature - SARM Features

Siebel Application Response time profiling SARM (7.5.3) ARM Standard Support (7.7) CPU and Memory Usage Profiling (7.7) Additional Siebel component metrics (7.7)

Benefits
Proactive monitoring of application response time Diagnosis of response time problems in the application and infrastructure Enables tuning of applications to meet service level commitments Enables diagnosis of memory consumption

SARM Run-time Architecture

Siebel Server
SARM Start

Server Component
SARM Stop SARM Correlation Key

SARM Framework

3rd Party ARM API Library

3rd Party ARM Tool

SARM Start

Server Component
SARM Stop

SARM Log

ARM Log

SARM Instrumentation for 7.5.3

User Interface
Physical UI Rendering Logical UI Object Manager Data Manager EAI

Timers
Web Server Time

Network Time SWE Time Workflow Time Script Time SRB Time Comp Time

Database Time

Siebel Database

External Database

Legacy Application

Performance and Scalability

Architecture Overview Component Scalability Scalability Across Components Network Scalability Performance Optimization Performance Management

Questions and Answers

Siebel 7 Performance and Scalability

Inside the Siebel Server
Richard Sands Siebel Expert Services
Siebel Systems 2003 Do not distribute or re-use without permission