WHITE PAPER

Managing Data within

Billing, Mediation and
Rating Systems

Introduction
Building high performance production systems for
billing and mediation relies heavily on making the
right architectural choices needed to achieve the high
availability and system throughput demanded by such
systems. This paper examines some of the common
key functional areas of these processing engines with a
particular focus on data management strategies. We use
the Berkeley DB database engine, an open source devel-
oper database used in many carrier grade applications,
to illustrate how data management strategies
can be implemented.

The range of requirements is broad and there are specific

computing challenges involved in the process of collect-
ing usage statistics, applying a rate to these statistics
and producing a useful subscriber bill at the end. This
paper examines the global requirements which apply at
all steps in the process and the specific requirements
which apply to mediation/aggregation, rating and
billing respectively.

Makers of Berkeley DB
2 MANAGING DATA WITHIN BILLING, MEDIATION AND RATING SYSTEMS

Contents
Introduction 1
Global Requirements 3
Data Throughput 3
Real-Time Data 3
Data Durability 3
Data Availability 3
Module-Secific Data Requirements 3
Reference Data 3
Live Data 3
Critical Design Considerations 5
Predictable Data Access 5
Simple Data Types 5
Service Orientation 5
Repetitive Access 5
Live Data Management Strategies 5
In-Memory Data Storage 5
Client-Server Data Storage 6
Bespoke Flat FIle Data Storage 6
In-Process Data Storage 6
Berkeley DB 6
Database Performance Comparison 7
Aggregating/Mediating CDRs 7
Real-Time Processing of CDRs 8
Rating/Fraud Detection 9
Compiling Reports of CDRs 9
Exporting Data 10
Deployment Considerations 10
Data Replication and Load Balancing 10
Data High Availability 11
Data Security 11
Database Maintenance 11
Summary 11

Global Requirements
Billing and Mediation systems are a key element in any
operator’s OSS. They are often considered the most
mission-critical components of the OSS, as outages
or data loss have a direct impact on operator
revenue. Any application must fulfill the following key
requirements:
Data Throughput
The proliferation of new services, new billing methods
and new subscribers continues to drive data volumes
higher throughout the system. Subscribers number in the
millions and billable events are of the order of tens or
hundreds of millions, depending on the billing cycle.
Real-Time Data
Support for pre-paid services in particular requires that
full end-to-end processing of billable services is possible
in real time. Latency within a system architecture is often
a limiting factor on meeting real-time requirements.
System latency can be more difficult to address than
system throughput as it is often a feature of the software
architecture and cannot be efficiently remedied by the
availability of additional hardware.
Data Durability
Data loss equals revenue loss for an operator. Systems
simply cannot afford to drop data under any conditions.
Data Availability
Billing is a real time operation (particularly when sup-
porting pre-paid services) and must always be avail-
able. Billing requires rating information which, in turn,
requires usage data. Any failure of any part of the sys-
tem (e.g., rating engine) will result in failure of a whole
system or a failure to achieve the operator’s desired
level of revenue assurance.
WHITE PAPER 3
Module-Specific Data
Requirements
In each part of the process, there are distinct problem
types. The problem types can be broadly divided into
two categories; reference data management and live
data management.
Reference Data
Reference data relating to the operational or business
knowledge of an organization – such as subscriber
details, account histories or plans – is generally
accessed infrequently and is of a relatively complex
nature. Examples of reference data applications are
rate plan management and subscriber relationship
management. The frequency of changes/writes is
relatively low (< 10,000-1,000,000 per day), and there
are a high number of queries/data reads relative to data
writes. Additionally, queries are more likely to be of an
ad-hoc nature, and the associated database schema is
generally more complex.
The storage of reference data is usually well understood
and well suited to database management systems with
support for high level schema definition and ad-hoc
queries. Often, the choice of database is dictated by the
demands of specific operators, who may have existing
software or who rely on SQL as the common denominator
between applications for integration and interoperability.
Live Data
Live data is data that has to be processed before it can
become useful business knowledge or reference data.
Using a call data record (CDR) as an example, it is of
little use to the business without significant processing.
Once it has been collected, rated and presented as part
of a subscriber bill, it has distinctly more business value.
Live data is also vital business data that is accessed very
frequently in the course of the system operation, such
as subscriber information caches/lookups, pre-paid bal-
ance management, event/audit logs. By its nature, live
data exists in large amounts since it is the lifeblood of
the business.
4 MANAGING DATA WITHIN BILLING, MEDIATION AND RATING SYSTEMS

Examples of live data applications are usage collection/

mediation, rating and account state management. The
frequency of changes/writes is high (>1 million per day),
and there is a comparatively high ratio of data writes to
queries. It is also generally the case that there is a small
set of defined access methods/queries on the data, and
these are usually known at design time. The correspond-
ing database schema is often quite simple.

��� ������
������� �����
��� ������
������

������� ��� ��������� ����� ����������

���������� �����������

��� ��������� ��� ��

���� ���� ������ ������

������
��� ������
���� ����� ����� �������
�����
���������� ���
������������ ������� ������
��� ���������
�����

������� ��
��� ������
��������� ����
���
���� ����

Sample architecture for mediation/rating billing system


Critical Design Considerations
In order to build robust and scalable systems, it is clearly
important to identify the weakest parts of the architec-
ture. Handling live data places the greatest load in terms
of data volumes and and greatest exposure to critical
latency limits. For this reason, this paper focuses on data
management strategies for handling live data.
There are certain characteristics of live data and the
processing which must be carried out on it which can be
exploited to deliver optimal performance:
Predictable Data Access
This is an excellent characteristic for optimizing sys-
tem performance. Data can be stored and retrieved
using database indices which are based on the most
common access mode for the data. This enables
expensive database join and query operations to be
avoided. As these can involve tens or hundreds of
times more processing than simple index lookup, this
represents a huge potential efficiency improvement.
Simple Data Types
The data types associated with billing and mediation
(for example, CDRs) are often structurally simple and
do not have complex relationships that need to be
specified within a high level data schema. The ability
to dispense with complex schema means that expen-
sive schema can be avoided in both the application
and the underlying database.
Service Orientation
Many development organizations are adopting a
service oriented approach to their systems devel-
opment. This approach is characterized by encap-
sulation of functionality within functional modules
(“services”) whose interfaces are well defined. The
traditional data-oriented approach, which is implied
WHITE PAPER 5
by relational systems, is in opposition to service ori-
ented development, since it brings with it the notion
that there is a separate data storage format and
query language which service modules must supp-
port. This adds additional complexity and, critically,
performance overhead to a service-oriented appli-
cation which is built to use a relational database
system.
Repetitive Access
At high loads, billing and mediation systems make
almost continuous use of the underlying data store
as they write UDRs (Usage Data Records), audit logs
and other data to ensure it is safe in the event of
system failure. Any form of access latency, such as
network/socket latency, will have a significant effect
on performance when multiplied over the total num-
ber of writes/queries.
Live Data Management
Strategies
In-Memory Data Storage
Ideally, all handling of live data would be done in main
memory, since this enables data to be stored and
retrieved at optimal speed. Storage of CDR data in main
memory alone is generally not appropriate, however,
because the risk of data loss is unacceptable. Attention
must be given to any form of internal data buffering or
queuing which will be at risk if the application crashes.
Consequently, all queues/buffers need to be backed up
to some non-volatile storage mechanism. The options
are outlined below.
6 MANAGING DATA WITHIN BILLING, MEDIATION AND RATING SYSTEMS

Client-Server Data Storage

A common storage solution is the use of a client-server, of this kind offers an elegant solution to the problems
relational database management system (RDBMS). associated with handling live data. By removing the
Often this may already be in use for the storage of refer- query latency, read performance and - more importantly
ence data elsewhere in the OSS, and so it is tempting to - write performance can be dramatically improved over
assume that it will be appropriate for storage of data in that offered by a client-server database management
all situations. In fact, RDBMS are very generic storage system. In addition, an in-process database can provide
solutions and bring with them many layers of functional- transactional and recoverability features often lacking in
ity which are redundant when handling live data. Trying bespoke flat file solutions. Berkeley DB is an example of
to improve system performance for the high volume of an in-process database commonly used in a wide range
writes/queries involved in billing and mediation process- of carrier grade applications such as OSS, messaging
ing can involve a significant amount of development services and network infrastructure, and it will be used
work to try and raise the throughput to acceptable levels. to illustrate some of the optimization techniques for han-
Examples of this are the batching of reads/writes to the dling live data throughout this paper.
database to reduce the effect of network latency and
connection pooling used to parallelize access to the
database. In extreme cases, a large in-memory cache is
used in front of the database to improve response time. Berkeley DB
While this improves read times it does little for performance
when faced with the high proportion of database writes
Berkeley DB is an open source developer database
that are associated with live data. The remote nature
which operates in-process after being linked into the
of a client server database places it at a distinct disad-
application. All data is stored as a collection of key/
vantage in ensuring rapid response and low latency to
value pairs within the database, where the key is the
system transactions which must be durable.
equivalent of an index, and the data is equivalent to
a record in a relational database. Keys are typically
Bespoke Flat File Data Storage ordered in a b-tree structure (although they can be
To improve performance, developers often build inter-
structured in other ways), and many sets of keys can
nally-developed file system solutions to store data. In
be used to store multiple views/indices to the same
general, these solutions solve the problem of perfor-
set of data. All data is stored in the native format of the
mance but often lack transaction support for full ACID
controlling application, which eliminates a tremendous
transactions, recoverability and advanced high-avail-
amount of processing overhead associated with
ability features such as clustering. Additionally, this
relational storage systems.
approach diverts valuable development time from build-
ing other features within the product. It can also neces-
Berkeley DB supports concurrent access, transactions,
sitate a cost for porting to new hardware, as well as add
replication and high availability/clustering via C, C++,
on-going maintenance costs to the solution as all of the
and Java APIs, as well as a variety of scripting languages.
code associated with the internally-developed solution
must be maintained and upgraded whenever the appli-
cation is upgraded.

In-Process Data Storage

The ideal storage solution for building massively scalable
billing and mediation solutions is one which combines
the speed of in-memory access with the safety of trans-
actional, secure disk storage. An in-process database
 WHITE PAPER 7

Database Performance
Comparison Write Time (ms)
# Records MySQL Berkeley DB JE Berkeley DB C++
1000 1168 492 10
To understand the magnitude of performance gains 2000 2261 680 20
5000 5673 1103 60
available through using Berkeley DB, it is important
7000 7156 1396 90
to understand its relative strengths. In a recent simple 10000 10167 1905 130
100000 100787 14624 1880
benchmarking exercise, a small data record of a similar 1000000 3167
size to a CDR (~60-80 bytes) was written and read using
Query Time (ms)
three database configurations: # Records MySQL Berkeley DB JE Berkeley DB C++ *
1000 9 1 0
2000 10 2 0
1. A C++ application using Berkeley DB 4.2 5000 15 5 0
2. A Java application using Berkeley DB 1.5 Java Edition (JE) 7000 17 8 10
10000 20 14 10
3. A Java application connecting to a relational data- 100000 142 138 80
base management system (in this case, MySQL 4.0.21 1000000 800
accessed via JDBC).
Comparative time to write data and perform a single query.

This simple test measured the time taken to write a fixed

number of records to the database, one at a time. The read
test measured the time taken to perform a single query of
the database, which returned a given number of events
(records). The query was a simple indexed lookup on a
single column of data. No attempt was made, however, to
optimize or batch queries to try and avoid network/query
latency as would typically be done to improve the perfor-
mance of a client server database.
The test was performed on a notebook PC running Linux
(Specification: Intel Centrino 1.7MHz, 512Mb RAM).
Sample of record stored:
Seq Source Destination EventType Description
1 142.3.1.5 612.3.41.5 16 Link up
As can be clearly seen, the level of performance - particular-
ly for writes - far outstrips conventional relational database
systems. The response of adding an in-memory cache to
a relational database offers little relief here, since perfor-
mance depends heavily on the performance of database
writes, which must necessarily involve disk access.
Aggregating/Mediating CDRs
In some instances, CDR logs need to be aggregated to
produce usable information for billing purposes. As an
example, consider a call log that makes entries for the start
and end of a call/session that then need to be correlated in
order to make complete call records. Typically, the call logs
are imported into the database first to ensure they are per-
sisted and then processed to find matching events. Using
a relational database, this requires several queries or joins
on the database tables to produce the result. With Berkeley
DB, indexing can dramatically improve correlation lookup
times and eliminate entirely the need to perform joins on
database tables.

*The accuracy of the system clock (clock_t) on the Linux x86 test machine
was insufficient for measuring the time taken for small numbers of
queries using Berkeley DB C++. Note that Berkeley DB offers other con-
figurations (DB_TXN_NOSYNC, DB_TXN_WRITE_ NOSYNC) for even higher
performance where relaxing durability is a possibility.
8 MANAGING DATA WITHIN BILLING, MEDIATION AND RATING SYSTEMS

Take an example CDR with the following fields:

LogId Date Time Trunk Number Type
2345 25/12/04 12:01.01 1238523 00441234567 Call connected
2346 25/12/04 12:12.23 2356723 00441456789 Call connected
2347 25/12/04 12:14.06 1238523 00441234567 Call disconnected
2348 25/12/04 12:15.45 1238523 00353874107 Call connected
2349 25/12/04 12:34.12 2356723 00441456789 Call disconnected

To correlate the events, one might employ a join to find store where access speed, isolation and durability are para-
log events where the trunk and dialed number match mount. Berkeley DB supports a configurable cache for data
each other and then pair the results to produce a single storage and can also be configured to run entirely in-memo-
billing object per call. For example: ry if needed. It is routinely deployed as part of server groups
using replication which process submitted events from a
Date Time Trunk Number Duration central event queue/buffer to perform balance updates or
25/12/04 12:01 1238523 00441234567 13:05 record mediation.

Berkeley DB uses a simple key-value model for indexing
data. Typically, the key value will be an index value cor-
responding to a column of a database table. Berkeley
DB supports and manages multiple sets of keys on the
same table of values. In this example, assume the unique
primary key is the LogId. Additionally, we can use [Trunk
+ Number] as a secondary key/index to the data so that
multiple entries can be entered using the same duplicate
secondary key. By using a simple cursor to the data, all
the entries for a particular [Trunk + Number] combination
can be read very efficiently.
Real-Time Processing of CDRs
Berkeley DB supports a number of internal database struc-
tures. The most common and general purpose of these is
the b-tree. Berkeley DB also supports hash, queue and
recno access methods internally. For real-time processing of
CDRs, a recno/queue indexed database operates as a very
efficient queue/buffer which is particularly suited to storing
and retrieveing fixed length records. CDRs can be pushed
into and popped out of the database which prevents data
loss or out-of-memory problems which can occur when real
time billing servers are subjected to “bursty” data traffic.
Berkeley DB is frequently used as a high speed persistent
buffer in data critical and high throughput applications (e.g.,
SMS/MMS messaging and email gateways) to make
applications more resilient to high traffic loads and denial
of service attacks.

Without persistent storage, the buffer would fill up, and

The greatest shakeup of billing, mediation and rating
then data must be dropped or the application will run out
architectures has come from supporting pre-paid
of memory. Neither case is appropriate for applications
services. This places all validation, mediation and
such as billing and mediation, where any data loss in
rating functions directly in the subscriber’s call path.
unacceptable.
The demand placed on the OSS to support this has
made end-to-end system latency the key requirement
for systems handling pre-paid services.

Berkeley DB’s exceptional performance and in-process

architecture make it ideal for implementing subscriber cach-
es, balance management databases and other forms of data
 WHITE PAPER 9

Rating/Fraud Detection Compiling Reports of CDRs

Rating of billing objects can be a difficult process to
Compiling CDR information into useful per-subscriber
get right given the ever-changing list of rate plans and
billing data is another activity which requires intensive
services that can be applied to subscribers. Rate plans
query and join operations on the underlying database.
are becoming increasingly complex as more network
As with the CDR indexing example above, billing objects/
technologies and billable services become available
rated CDRs can be indexed to ensure fast retrieval for
to subscribers. Rate plans are complicated and prone
billing activities. For example, in exporting subscriber
to significant structural revision from time to time. For
bills, the rated billing objects need to be selected for
these reasons, they are often stored in a relational (or
a given subscriber over a given billing period (say 30
potentially XML) database which provides an array of
days). Here, a secondary index is used which combines
tools for managing this complexity. Rating manage-
the subscriber number and the billing period.
ment applications are also well suited to relational
databases as the system performance is a secondary
Imagine all dates between 01/12/04 and 31/12/04 are
consideration to the requirement for ad-hoc query
identified as the same billing period, for example number
support and complex schema and relationship
36. The secondary index would be [1238523 + 36] in this
management capability.
case. As with the CDR example, all entries for a sub-
scriber in a given billing period would be immediately
The additional overhead of database schema process-
accessible without requiring a database join operation to
ing layers make relational databases unsuited for the
be performed. A cursor is used to iterate over the entries
timely bulk processing of CDRs. By contrast, Berkeley DB
associated with the secondary key to retrieve the com-
is ideally suited to serve as the core processing engine
plete subscriber billing information.
for performance-critical components of systems such
as this. Many customers have found an optimal solu-
tion using a rating engine that retrieves rate plans from
a number of SQL databases, combined with core CDR
data processing using an in-process database such as
Berkeley DB.

Date Time Subscriber Number Duration Rate

25/12/04 12:01 1238523 00441234567 13:05 0.45
27/12/04 15:17 1238523 00353234567 08:26 0.30

Our customers have also found that Berkeley DB is a

good solution for implementing correlation engines
which examine trends in usage statistics for fraud detec-
tion and for building other value-added services having
similar requirements for fast access and custom indexing
of usage records.
10 MANAGING DATA WITHIN BILLING, MEDIATION AND RATING SYSTEMS

Exporting Data Deployment Considerations

Exporting processed information to external systems is the
last step in the process. There are myriad ways to export
data to external systems via middleware (Web services,
SOAP, CORBA, MQ Series), direct database connections via
SQL and file based transfer/import/export. Berkeley DB
stores data in the application’s native format and does not
require extraneous mapping prior to export. This optimizes
the export process, preventing it from becoming the weak
link in the processing cycle.
Data Replication and Load Balancing
In order to achieve a five-nines level of availability, it is
necessary to defend against system failure by duplicat-
ing data in multiple locations. Berkeley DB supports
data replication across multiple machines which affords
seamless and automatic horizontal scaling to the appli-
cation. Beneath the covers, Berkeley DB implements a
single master replication model with a replicated envi-
ronment. All replicas are available for reading (all writes
are internally re-directed to the master database) within
the environment.

Developers have complete control over the synchroniza-

tion of replicas and transactions can complete when
the write occurs at the master or when all replicas
receive the update depending on the level of guarantee
required.

��� ��

���� ����
������ ����� �������

������ �������� ��
������ ������� ������
����� �������

������ �������� ��
������ ������� �������
�����

����� �������

������ �������� ��
������ ������� �������
 WHITE PAPER 11

Data High Availability Summary

In a replicated environment, Berkeley DB replicas con-
stantly monitor their connection to the master database.
In the highly complex and demanding realm of media-
In the event of failure of the master or partitioning of
tion and billing, there is no “one size fits all” solution for
the network, the replicas call a Paxos-compliant election
data management. Handling the volume of live data that
process and a new master is chosen from N/2+1 of the
would need to be processed immediately or refined to
interconnected replicas. Selection is based on the most
produce billing and subscriber information is often the
recent log record and on a configurable priority scheme
critical weak point of a solution and a limiting factor for
(e.g., it may be desirable to favor machines with faster
system scalability.
disks or more RAM). This prevents a minority of replicas
from electing a new master if they become isolated due
Reference data is well suited to traditional client-server
to network partition.
relational databases. The ad-hoc query capabilities
of SQL and wealth of tools available to users make
Data Security them appropriate for managing this data. For live data,
Securing data stored in a Berkeley DB database is easier consisting of millions of records which are in constant
than securing data stored in client-server databases motion through the billing system, other architectures
(which must communicate between client and server are preferable. Berkeley DB is a vital component of these
using externally monitorable network traffic) as there is systems for many of the world’s leading operators and
no requirement for SSL or other authentication protocols solutions providers.
to be supported in the application. This is because the
application and the database management code are
linked directly together in a single process. Berkeley DB
supports an encryption plug-in which encrypts the data
directly on the disk, thereafter requiring a password from
the application. In addition, Berkeley DB can restrict
access to its in-memory cache to protect against unau-
thorized access of system memory.

Database Maintenance
Berkeley DB does not require regular maintenance - or
even a DBA - which greatly reduces the operational
expense of systems deployed using it. For this reason,
many of the best known names in the telecommunica-
tions business depend on Berkeley DB. A partial list
includes Cisco, Motorola, Amdocs, Ericsson, LogicaCMG,
Alcatel, Tellabs, Openwave, Jabber, Hitachi, Lucent and
AT&T. With over 200 million deployments, Berkeley DB
is the natural storage solution wherever data is on the
move – in the handset, at the switch, at the message
center, in the OSS and in the billing system. Berkeley
DB offers exceptional performance, zero maintenance
deployments and unmatched reliability.
12 MANAGING DATA WITHIN BILLING, MEDIATION AND RATING SYSTEMS

Sleepycat Software www.sleepycat.com makes Berkeley DB, the most widely used open source devel-
oper database in the world with over 200 million deployments. Customers such as Amazon.com, AOL,
British Telecom, Cisco Systems, EMC, Google, Hitachi, HP, Motorola, RSA Security, Sun Microsystems,
TIBCO and Veritas also rely on Berkeley DB for fast, scalable, reliable and cost-effective data manage-
ment for their mission-critical applications. Profitable since it was founded in 1996, Sleepycat is a
privately held company with offices in California, Massachusetts and the United Kingdom.

For further information, please contact Sleepycat by sending email to info@sleepycat.com or visiting
Sleepycat’s website at www.sleepycat.com

Corporate Headquarters West Coast Office European Office Telephone

Sleepycat Software Inc. Sleepycat Software Inc. Sleepycat Europe Ltd. +1-978-897-6487
118 Tower Road 5858 Horton St. Suite 265 Coronation House, +1-877-SLEEPYCAT
Lincoln, MA 01773 Emeryville, CA 94608 Guildford Rd. (Toll-free, USA only)
Makers of Berkeley DB USA USA Woking, GU22 7QD
United Kingdom wp_billmed_0305