WebSphere Extreme Scale (WXS Components) – Operational readiness Considerations.

By Nitin Gaur

Introduction:

IBM WebSphere eXtreme Scale is specifically designed to help applications lower the load they
place on back-end servers, increase availability and scalability, and provide the facilities for
extreme transaction processing. A software layer between an application and its back-end
resources, WebSphere eXtreme Scale accelerates applications by caching objects obtained from
the back-end, providing a customizable, pluggable data fabric framework that allows applications
to share data across a wide range of application scenarios.
This paper attempts to serve as an operational readiness checklist and is designed for
customers and IBM software services community, who are interested in implementing a highly
available and scalable e-business infrastructure using the IBM WebSphere eXtreme Scale.
Through WebSphere eXtreme Scale, customers can postpone or virtually eliminate costs
associated with upgrading more expensive, heavily loaded back-end database and transactional
systems, while meeting the high availability and scalability requirements for today’s
environments. While not an exhaustive list, this paper includes primarily the infrastructure
planning requirements of WXS environment.

Understanding WXS Components:

From a broad environment perspective WXS has Three management entities:

1. Grid Servers: These are JVM that will contain grid container. Much like a typical
description of a container in a J2EE™ context, grid containers essentially provide the
grid application services such as security, transaction support, JNDI lookup service,
remote connectivity, and so forth. The grid containers house shard distribution and
placement, and enable easy manageability of the grid infrastructure. Much like other
containers (Web and EJB™ container, for example), a grid container can also take
advantage of the configuration service provided by the WebSphere Application Server
infrastructure in a managed environment.

2. Grid Clients: Clients connect to a grid and are attached to the whole grid. Clients need
to examine the key of application data to determine to which partition to route the request.
Any entity that is attached to the grid with any kind of request becomes a client. The
client directly accesses the partition that holds the data. The location information
regarding the partition is provided by the catalog servers.

3. Catalog Servers: The catalog server is the engine that drives the grid operations. The
catalog server maintains the healthy operation of grid servers and containers. The catalog
server becomes the central nervous system of the grid operation by providing the
following essential operation services:

a. Location service to all the clients

b. Health management of the grid itself
c. Shard distribution and re-distribution
d. Policy and rule enforcement
e. High availability and group service
 The client will begin its access to the grid by obtaining a routing table from the
catalog servers, which enables the client to locate the primary partition shard and object.
In the event of a JVM failure, or re-distribution of partitions due to a change in grid
membership, the client is provided with an up-to-date routing table by one of the grid
servers. This mechanism is maintained by an epoch time and version maintained with
the routing table held by the client.

Understanding the WXS component relationship:

The above Data Model describes the relationship between the components in an eXtreme Scale
environment.
1. The JVM can be either an application server or a stand-alone JVM and can host many grid
containers. A JVM contains a runtime and a number of containers, usually one.
2. A grid container that resides in a JVM can host many ObjectGrid instances, or many
ObjectGrid instances spread across many grid containers.
3. A MapSet is a collection of maps that are typically used together. Many MapSets can exist in
one ObjectGrid instance.
4. An ObjectGrid consists of a number of partitions. Each partition has a primary shard and N
replica shards.
5. One partition can host many BackingMaps.
6. An ObjectGrid consists of a number of maps, called a mapSet. A mapSet is partitioned using a
key. Each map in the mapSet is defined by a BackingMap.
7. An ObjectGrid can host a set of grid containers. A grid container can only host
shards from one ObjectGrid. This means that multiple ObjectGrids can be started on a single
JVM. Each grid started has its own container within the JVM. Those grid containers will host
shards from the grid as determined by the catalog server.
WXS Infrastructure Planning Considerations:

1. Grid Server Hosting.

a. Grid Server on WebSphere managed node: WebSphere eXtreme Scale fits
perfectly into an environment managed by WebSphere Application Server
Network Deployment, allowing all the services, such as transaction management,
security and monitoring, to be used by grid components. The decision to host
Grid Servers in a managed environment, while providing features like ease of use
and administration, may have licensing cost considerations.

b. Grid Server on standalone Supported JVM: WebSphere eXtreme Scale has

several flexible deployment options. One of these deployment options is
installing WebSphere eXtreme Scale into a stand-alone Java2 Standard Edition
environment (J2SE). The stand-alone deployment option differs from the
WebSphere Network Deployment-managed option in that it lacks Java Enterprise
Edition support and does not benefit from the management and monitoring
environment that WebSphere Network Deployment provides. It does, however,
provide a simple footprint and runtime for hosting grid containers and catalog
servers. While it may be economical to

2. Catalog servers hosting

a. Catalog Server on WebSphere managed node: As mentioned before, The
catalog server is the engine that drives the grid operations. Hosting Catalog
servers on WebSphere managed node, presents similar benefits, to that of hosting
grid servers on WebSphere managed node.

b. Catalog Server on standalone Supported JVM : In some cases, it may be

imperative to host catalog servers on standalone JVMs. An example may include
when using WXS enabled grid to cache HTTP session, across geographically
disperse data centers. In this specific scenario, while the grid servers may be
hosted in WebSphere managed application servers, and the cache to be replicated
across data centers, the catalog servers, due to HA managers services and quorum
requirements, would be required to be hosted in stand alone JVMs. This will
enable a core group of standalone catalog servers.

c. Multi data Center Catalog Server support : Multi data center would imply a
WXS zoned configuration. In a Zoned configuration, usually one catalog server
per zone ( or data center) is a good start.

3. Co-locating Grid client and server (Application and Grid container in same JVM):
a. This is when the ObjectGrid servers are collocated in the same processes where
the servlets run. The WXS session manager can communicate directly to the
local ObjectGrid instance, since it is collocated with in the same server process.
b. When WebSphere Application Server as runtime and grid container, simply place
the supplied objectGrid.xml and objectGridDeployment.xml files into the
META-INF directories of your WAR files. ObjectGrid will automatically detect
the presence of these files when the application starts and will then automatically
launch the ObjectGrid containers in the same process as the session manager.
You can modify the objectGridDeployment.xml depending on which type of
replication (synchronous or asynchronous) and how many replicas you want
 configured. This step assumes that your have augmented the application server
profile/instance with WXS.

c. When the grid container is collocated with the enterprise application in the same
application server, the grid container becomes more vulnerable. Issues in the
application (for example, memory leaks, deadlock situations) can bring the grid
container down.

4. Isolating the application from the grid (i.e. application and grid in separate JVMs):
a. Unlike the topology discussed above the server process hosting the application is
different from the server process hosting the JVMs. This is beneficial for the
application with either voluminous HTTP session traffic or relatively larger http
session objects. In this case the WXS session manager, which resides on the
application server process, communicates to a remote ObjectGrid server
processes. In effect the WXS session manager becomes the client, to the Object
grid servers, which are the hosts to grid containers.
b. To use a remote, ObjectGrid, the WXS session manager must be configured to
communicate with the catalog servers and grid servers. The session manager will
then use an ObjectGrid client connection to communicate to the catalog server
and the ObjectGrid servers, wherever they reside. If the ObjectGrid servers are to
be started in independent, standalone J2SE processes, then it is advisable to
launch the ObjectGrid containers using the objectGridStandAlone.xml and
objectGridDeploymentStandAlone.xml files supplied in the session manager
samples directory. This differentiation is to indicate that the HTTP session store
is standalone and isolated. The core functionality of the file remains, i.e. like
objectGrid.xml, objectGridStandalone.xml defines the grid for sessions, and
objectGridDeploymentStandalone.xml defines the mechanics of grid deployment.

5. Grid server sizing

a. Calculate the number of JVMs required from the partitioning configuration.

b. When deciding on the number of JVMs required for a grid, a good rule of thumb
is to start with the number of partitions needed multiplied by the number of
replica shards per partition. For example, if you have two partitions and each
partition has one primary and one replica shard, then you would need at least four
JVMs as a starting point for a highly available grid.
c. It is important to plan the number of partitions you will need for both current
capacity and future growth. Once you have deployed your applications you
cannot simply add partitions or scale using more JVMs. A good guideline to
follow is that at the end of your planning horizon, you should end up with 10
shards per partition.

6. Catalog server sizing

a. Always more than one for high availability
b. Recommended prime number greater than 2. This is due to concept of a catalog
service quorum. A catalog service quorum is the minimum number of active
catalog server members required for the grid system to operate correctly (that is,
to accept membership registrations and changes to membership to ensure proper
routing and shard placement). This approach of ensuring the registration and
 consistency of grid servers is achieved only when a quorum is established
between catalog servers. Writes to the catalog service state are committed only
when the majority of the catalog servers participate in the transaction.
c. The decision on the number of catalog servers depends on overall grid size,
desired high availability, and zones configuration.
d. In a Zoned configuration, usually one catalog server per zone is a good start.

7. Secure Transport (SSL)

a. Understanding which end points to secure : It is vital to understand different
transports that ObjectGrid uses in different instances. WXS uses the ORB for
client-server, server-server, client-catalog, server-catalog communications. This
is secure and was added support for SSL-Required in 6.1.0.5.
In addition to the ORB WXS the HAM/DCS/RMM stack for membership
only by the containers. Further the intra-zone (data center) grid servers use group
services (HA manager construct) for server state information. So, the
communications on this protocol is kept local and only contains membership
information, but this is not yet secured. This should not be alarming to enterprise,
as there is no application data shared, but only server state data ( which included
things like server, hostname and port information).

b. Supported Transport layer security: SSL Supported and SSL required are two
supported transport Type. SSL required, transportType support was added in
6.1.0.5. SSLv2, SSLv3, and TLSv1 are all considered SSL protocols.For
Transport, One can choose to use Secure Sockets Layer (SSL), TCP/IP or both as
the inbound transport that a server supports. If you specify TCP/IP, the server
only supports TCP/IP and cannot accept SSL connections. If you specify SSL-
supported, this server can support either TCP/IP or SSL connections. If you
specify SSL-required, then any server communicating with this one must use
SSL.

SSL Supported: If you select SSL-supported, then the server opens both a
TCP/IP and an SSL listener port and most inbound requests are received using
SSL
SSL Required : If you select SSL-required, then the server opens an SSL
listener port only and all inbound requests are received using SSL.

If you specify SSL-supported or SSL-required, decide which set of SSL

configuration settings you want to use for the inbound configuration. This
decision determines which key file and trust file are used for inbound
connections to this server.
In dynamic deployment topology, the ORB protocol is used for client/server
communications. ORB transport security, Transport Layer Security (TLS) or
Secure Sockets Layer (SSL) is supported. However, transportType property must
be set to "SSL-Supported" in both the client and server security property files. If
set to "SSL-Required", the container server will fail to initialize with an error
message: "initial and forwarded IOR inaccessible".
Please note that prior to 6.1.0.5 use SSL-Supported or can use SSL-Required
if 6.1.0.5 is applied. Either case SSL is supported, although with a different type
of support.
 c. TCP and Firewall considerations : Please not that all the relevant ports would
need to be considered while designing a WXS topology. To ensure
communication flow between client-server, server-server, client-catalog, server-
catalog is vital to a smooth grid function. These communication flow ensures
grid data replication, client to grid access, registering membership changes and
subsequent re-routing of client due to membership changes. The following ports
would need to considered, in relation to TCP transport and enterprise firewall
while planning a WXS topology:
i. Client port : port used for client catalog cluster communication
ii. Peer port: port used for peer catalog cluster communication
iii. Listener port : Client’s need to know Catalog service JVMs listener
end points.
iv. JMXServicePort – if using JMX ports.

Note: you can set the transportType property in the following client and server
security configuration files:
• Client ( application - in Managed WAS env) : The transportType property in the
security.ogclient.props file.
• Container or Grid server or cache server: The transportType property in the
containerServer.props file.
• Catalog Server: The transportType property in the security.ogserver.props file or
containerServer.props file, depending on which file you use.

8. Testing the Infrastructure

Testing Methodology : One of the primary purpose of testing is to uncover software
defects and failures and subsequently correct them. Any broad goals of testing
include unit testing, integration testing, system and system integration testing. Each
testing process has a specific goal and objectives. From extreme scale testing, it is
vital to isolate the application test from the WXS ‘Grid’ tests. The implication of
such an approach assumes that ONLY the WXS features are tested independently
while the application itself has been tested for component features. The WXS ‘Grid’
tests would include testing the following:
1. Application accessing the Grid
2. WXS grid’s replication mechanism
3. WXS grid container failover and HA mechanism.
4. Testing of the WXS Grid infrastcruture, including the catalog servers and
shard placement.
5. Various use of configurable grid features such as Memory threshold, zone
replication, locking mechanism , batched update etc.
Integration and system integration testing would however include tests cases that
include verification of interaction between the integrated components, and system
integration tests verifies the integrated system is well aligned with core objectives of
system requirements. It is vital to understand that testing at each level to be
completed, prior to proceeding to integrations or system level testing. This approach
will ensure that each component or module is verified prior to their interaction in a
system as a whole.
9. Monitoring:
Monitoring while important is a complex discussion topic, and many decision points
around monitoring focus on “what to monitor?” Conceivably one can monitor many
aspects of an enterprise application infrastructure, i.e
a. Infrastructure monitoring – Include Monitoring of the overall infrastructure
health, including the resource usage, JVM health and availability and so on.
b. Application Monitoring – Includes application performance, resource usage
and bottlenecks.
WXS introduces another level of monitoring, which is completely optional, and that is the
GRID monitoring.
c. Grid Monitoring – Includes monitoring around cache statistics, cache entries
and cache policy information, grid replication and information on shard
placements.

When WXS is hosted in WebSphere managed environments, the WebSphere

Application Server provides a sophisticated performance monitoring infrastructure (PMI).
It can be used to gather performance statistics of all components inside an application
server. WebSphere eXtreme Scale takes advantage of this infrastructure to collect
performance statistics relevant to the grid. Metrics provided include the following:
d. Actual cache size
e. Cache hit rate
f. Transaction response time
g. Loader update response time.

However, in a standalone environment, The ObjectGrid APIs and MBeans are

designed to allow for easy integration with third-party monitoring applications.
JConsole or MC4J are some examples of lightweight JMX consoles that can be
used to quickly digest information about an ObjectGrid topology. Besides the
JMX based monitoring, WXS, beginning with WXS 6.1.0.3 provides a sample
application called ‘xsadmin’. This nifty application will provide with ‘point-in-
time’ information on topology, shard placement and map sizes in the WXS
enabled grid. These tools are only made available to facilitate WXS specific grid
level monitoring and management. It is assumed that the infrastructure
monitoring tools ( example IBM’s ITCAM and CA’s Wily Introscope) will still
be employed to monitor the underlying application environment.
Acknowledgement
Robert Ringo – WebSphere WW Technical Sales
Douglas Berg – WebSphere Extreme Scale Development
Joshua Dettinger - WebSphere Extreme Scale Development

References:

1. RedBook - Users Guide to WebSphere Extreme Scale

www.redbooks.ibm.com/redpieces/pdfs/sg247683.pdf

2. WebSphere Extreme Scale – Wiki – Users Guide.

http://www.ibm.com/developerworks/wikis/display/objectgrid/Getting+started

3. Billy Newports Blog - http://www.devwebsphere.com/