Cisco IT Case Study July 2012 Enterprise Messaging on Cisco UCS

How Cisco IT Deployed Enterprise Messaging on Cisco UCS

Messaging platform upgrade and new servers reduce costs and improve management, availability, and disaster recovery.
Challenge
EXECUTIVE SUMMARY
CHALLENGE Upgrade to a new messaging platform and new server infrastructure at the same time SOLUTION Microsoft Exchange 2010 messaging platform on Cisco UCS servers RESULTS Consolidated infrastructure with 60% fewer servers, located in three data centers instead of five Larger mailbox storage and improved user experience Improved service resilience and disaster recovery LESSONS LEARNED Analyze network capacity and evaluate multiple hardware configurations Test custom applications Identify special needs for user communications NEXT STEPS Server virtualization to match actual operational needs

In 2005, Cisco IT chose the Microsoft Exchange 2003 platform as the corporate standard for email, calendaring, and contacts software. By 2011, the need to upgrade to Microsoft Exchange 2010 was indicated by:

User requests for more mailbox space The need to provide a better experience for Apple Macintosh and mobile device users The end of Microsofts end support for Exchange 2003

An aging server infrastructure was also a factor. By this time we'd gone so long without an upgrade, the server infrastructure was getting quite old and we were facing a higher risk of hardware failures that could cause unacceptable service outages, says Ken Pauley, senior IT manager for communications and collaborations IT at Cisco. So we made the decision to upgrade to Exchange 2010 and replace the entire underlying server infrastructure at the same time.

Solution
For Exchange 2003, Cisco IT deployed three separate infrastructures, including one for employee user accounts and one for vendor and external user accounts. The third infrastructure, for voicemail accounts, was replaced in 2011 by the Cisco Unity Connection on Cisco UCS deployment. The 220 servers and associated storage systems for these infrastructures were housed in five locations around the world. The previous infrastructures were based on a modular pod design of servers and storage systems. Servers were installed in racks and each rack had a dedicated storage pod, with an additional SAN associated with the rack for backup purposes. Although simple to support and easily scaled by adding a new pod, this design was also very expensive. Adding even just a small number of users required installing a complete pod and all of its associated connectivity and other infrastructure.

2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public.

Page 1 of 6

For Microsoft Exchange 2010, Cisco has consolidated all user accounts onto a single, high-availability infrastructure that is based on 113 Cisco Unified Computing System (Cisco UCS) physical serversa design reflecting the Cisco IT service-oriented architecture for data center virtualization. Replacing these servers allowed the IT team to define a standard hardware and software configuration and fully implement and test the servers before starting the messaging platform upgrade. Although the upgrade could have been installed on virtualized servers from the start, the team decided to gather operational data in order to determine the right plan for server virtualization in the future. The Cisco UCS servers and associated storage systems are configured into multiple clusters at three sites in the United States. The primary clusters (with 30 servers each) are deployed in the Cisco data centers in Richardson and Allen, Texas. These centers are connected by a Metro Virtual Data Center (MVDC) active-active design, which provides for complete, real-time replication and failover between the clusters. The third site, in Research Triangle Park, North Carolina, operates a 27-server cluster as a backup for disaster recovery if both Texas data centers become unavailable. As shown in Figure 1, this design allows for failover at both the hardware and application levels. The server layer handles application failover because each host has eight active and eight passive databases. For example, if an active database fails in Richardson, failover is automatic to the associated passive database in Allen. If the application must fail-over to the disaster recovery site in Research Triangle Park, user accounts are back online within seconds to support business continuity. Load balancing of all client traffic across the two primary clusters is managed by Cisco Application Control Engine (ACE) GSS 4400 Series Global Site Selector Appliances, which also handle failover to the third cluster. To manage these elements, Cisco IT uses internal and vendor tools for the Cisco products, the messaging platform, and the storage systems.

2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public.

Page 2 of 6

Figure 1.

Cluster Design for Cisco UCS Servers that Host the Messaging Platform

Cisco Exchange 2010

Mobile Devices

Cisco ACE GSS

Firewall Clients

Applications

VPN
tion duc Pr o
Prod uctio n

Production Texas
Richardson DC ACE Round Robin

Dis ast er R eco ve r

DR North Carolina
Research Triangle Park DC ACE Round Robin AD Site Exchange DR CAS Array

Allen DC ACE Round Robin AD Site Exchange HA CAS Array

Client Access Server

Client Access Server

Client Access Server

Hub Transport

Hub Transport

Hub Transport

Mailbox Copy 1

Mailbox Copy 2

Database Availability Group (DAG)

Mailbox Copy 3

Storage of email messages, calendars, and contacts data is another significant element of this design. With 5.8 petabytes of data storage allocated, the enterprise messaging infrastructure represents the single largest storage deployment within Cisco and 10 percent of all storage deployed by the company, says Pauley. It is also 10 times more than the amount of storage in the old infrastructure, reflecting our users needs to store more messages for a longer time and Cisco IT's need to provide backup storage. Table 1 shows key statistics for the old and new messaging infrastructures.
Table 1. Key Statistics about the Cisco IT Enterprise Messaging Infrastructures

Exchange 2003 Infrastructure at Cisco

Five sites worldwide 220 servers, 40 rack spaces 10% average CPU utilization 0.44 PB storage 112,000 user mailboxes, 400 MB each No high-availability or disaster recovery configuration

Exchange 2010 Infrastructure at Cisco

Three sites in the United States 113 servers, 30 rack spaces Target of 35% average CPU utilization in normal operation and 70% utilization in a highavailability/disaster recovery scenario 5.8 PB storage 120,000 user mailboxes, 2 GB each Configuration supports high availability and disaster recovery

Upgrade Process
We made the decision to upgrade all users to the new messaging platform within the same timeframe instead of doing a phased upgrade by location, says Pauley. However, this meant the complete infrastructure needed to be
2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 3 of 6

up and running from day one, with all servers and storage systems deployed and

PRODUCT LIST
Data Center and Virtualization Cisco Unified Computing System Cisco ACE Application Control Engine Borderless Networks Cisco Wide-Area Application Services Software Cisco ACE GSS 4400 Series Global Site Selector Appliances

ready to go at the same time. Before beginning the actual upgrade, Cisco IT used a small proof-of-concept project to assess the planned upgrade process, its potential impact on users, and needs for user communication. This project included a small number of Cisco executives in order to verify access and user experience for calendars that are jointly managed with their administrative assistants. For the actual upgrade, the Cisco IT team used a scheduling tool to designate upgrade dates for defined groups of users, automatically send e-mail and voicemail

notices about the planned move, then perform the actual upgrade of the users mailbox from the old to the new platform. The upgrade was initially installed on non-virtualized Cisco UCS servers in order to maintain high levels of performance for the messaging applications. I'd rather over-deploy servers and scale back than under-deploy and have performance issues, says Pauley.

User Features
Today Cisco users enjoy a common experience in the messaging applications, whether accessed from a Windows, Macintosh, or Linux-based client or a web browser interface. The enterprise messaging applications are also integrated with other popular tools such as Cisco TelePresence and the internal Cisco Integrated Workforce Experience (IWE) social sharing site. With 55,000 mobile devices already authorized on the Cisco network, users increasingly expect to access and synchronize their e-mail, calendars, and contacts from their smartphone or tablet computer. Given the growing number of mobile devices used by Cisco employees, the deployment team planned the new messaging infrastructure to support access from three devices per user.

Results
With the mid-2012 completion of the upgrade to the messaging platform and its server infrastructure, Cisco is realizing results in several areas. Simpler infrastructure. Migrating to the Cisco UCS servers allows for a simpler infrastructure design, with 60 percent fewer servers and fewer cluster locations. This design reduces capital and operating costs, and allows for improved resilience and disaster recovery. We were able to achieve this level of reduction because of the performance and integrated protocol cable design of the Cisco UCS servers, says John Kerruish, Cisco IT architect. We were able to configure more users per server, which meant we could use fewer servers even before we looked at virtualizing them. Rapid deployment. Cisco IT completed the upgrade of 88,000 user mailboxes in just six weeks. If we had kept the previous cluster design with multiple locations around the world, it would've taken a year to go to all of the locations and finish the upgrade in a phased schedule, says Pauley. Simpler client interface for Macintosh users. Cisco IT now supports two native Macintosh email clients, replacing the 45 different combinations of tools used previously.

2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public.

Page 4 of 6

More message storage. Users now enjoy a standard mailbox size of 2 GB, compared to the previous limit of 400 MB. To control storage costs, Cisco IT can easily reduce or increase mailbox sizes based on actual usage levels and changes to message retention periods. Improved recovery from cluster failures. The previous design did not allow for complete replication of user accounts and data. In a cluster failure, Cisco IT was able to recover users mailboxes, but not their email messages. Today, Cisco IT can perform full replication of all user messaging data, with automatic failover between the two primary clusters. The disaster recovery site provides an additional cluster for recovering user accounts only.

Lessons Learned
Analyze network capacity. Configuring a messaging platform for a large enterprise in a high-availability failover design requires significant network bandwidth between the data centers. Verify adequate capacity to support both high-volume traffic from remote users and replication traffic. During periods of high network congestion, the bandwidth between the primary clusters can be configured to give priority to real-time user traffic and allow the replication traffic to absorb any latency. Evaluate multiple hardware configurations. Take the time to test various configurations of physical and virtualized servers to identify the best hardware configuration for the specific messaging platform. It is also important to allow sufficient time to accurately configure the storage systems for user messages and related data. Allow time for testing custom applications. Cisco IT operates a lab that allowed internal developers to test their specialized applications and also allowed the deployment team to perform functional and load testing before finalizing the server configurations. With this lab testing, Cisco IT was able to avoid conducting a pilot project, which accelerated the production schedule. Identify special needs for user communication. Cisco IT created special communications for Macintosh users because they would see the most change in their access and experience with the new enterprise messaging system.

Next Steps
The upgrade to the new enterprise messaging infrastructure was complete as of mid-2012. After monitoring operations and utilization levels, Cisco IT will begin planning for partial server virtualization to further reduce costs. Cisco IT will also plan upgrading the Wide-Area Application Services (WAAS) software to accelerate email traffic for Windows devices, which use the native, encrypted Messaging Application Programming Interface (MAPI) to retrieve messages.

2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public.

Page 5 of 6

For More Information

Cisco Unified Computing System: www.cisco.com/go/ucs Cisco Application Control Engine: www.cisco.com/go/ace To read additional Cisco IT case studies on a variety of business solutions, visit Cisco on Cisco: Inside Cisco IT: www.cisco.com/go/ciscoit

Note
This publication describes how Cisco has benefited from the deployment of its own products. Many factors may have contributed to the results and benefits described; Cisco does not guarantee comparable results elsewhere. CISCO PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING THE IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer of express or implied warranties, therefore this disclaimer may not apply to you.

2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public.

Page 6 of 6