CONTINUOUS

INTEGRATION
+ DELIVERY
IN THE CLOUD
How We Do It at RightScale

CONTENTS

CONTINUOUS INTEGRATION + CONTINUOUS DELIVERY EXPLAINED

CI/CD: Why Its a Perfect Fit for the Cloud
Automating Infrastructure

CONTINUOUS TESTING IN THE CLOUD

The Problem: A Test Matrix from Hell
Our Solution: RocketMonkey, Jenkins, and RightScale
Bottom Line: Reduce Manual Effort, Speed Development Cycles

DYNAMIC SCALING OF JENKINS IN THE CLOUD

The Problem: Many Developers + Many Changes = Jenkins Bottleneck
Our Solution: Manage Jenkins with RightScale
Bottom Line: Speed Development, Lower Costs

CONTINUOUS DELIVERY OF SOFTWARE PACKAGES IN THE CLOUD

The Problem: Compressing Long Release Cycles with Continuous Delivery
The Solution: Continuous Delivery of RightLink

KEY TAKEAWAYS + ADDITIONAL RESOURCES

2015 RightScale, Inc. All rights reserved.

CONTINUOUS
INTEGRATION
+ DELIVERY
EXPLAINED

CONTINUOUS INTEGRATION + DELIVERY EXPLAINED

Continuous integration (CI) is the practice of automatically integrating and testing a piece of
software each time code is committed by a developer. Continuous delivery (CD) goes a step
further to package the software, perform regression testing, and ensure that the software is ready
for release. Continuous deployment, sometimes used interchangeably with continuous delivery,
is most relevant for web-hosted and SaaS applications and automates the process all the way
through to deployment into production.
To reduce time-to-market of new features and products, RightScale leverages a variety of modern
development practices, including agile development, CI, and CD. We dont follow a one-sizefits-all approach. Rather, the development practices we choose for each product or component
depend on the business goals we are trying to achieve. In addition, we leverage our RightScale
Cloud Portfolio Management platform heavily to automate provisioning and management of all
of our cloud infrastructure and tightly integrate with our development processes.

2015 RightScale, Inc. All rights reserved.

CONTINUOUS INTEGRATION + DELIVERY EXPLAINED

CI/CD: Why Its a Perfect Fit for the Cloud
In order to accelerate the application lifecycle via CI and CD,
organizations will need to leverage infrastructure resources
(virtual machines, storage, and networking) to automate the build,
integration, and testing the code under development. The amount
of infrastructure required will vary greatly depending on the
number of simultaneous teams or projects, and the phase of each
project in the development process. In addition, resource needs
may increase during business hours and decrease during nonworking time. As a result, companies are increasingly turning to
cloud-based options, both public and private, to supply resources
for CI and CD processes.

Faster application
delivery enables
competitive
differentiation.
Applications have
become the means by
which organizations
attract customers and
deliver value; better
applications, delivered
faster, drive faster
business growth.
Forrester Research

2015 RightScale, Inc. All rights reserved.

CONTINUOUS INTEGRATION + DELIVERY EXPLAINED

Automating Infrastructure
In order to leverage cloud environments as part of a CI/CD development model, companies need
to automate not only the build, integration, and testing processes, but also the provisioning and
de-provisioning of infrastructure needed to support these processes. In addition, organizations
need to ensure consistency among the various infrastructure environments as code moves from
development to test to staging and then production.

2015 RightScale, Inc. All rights reserved.

CONTINUOUS INTEGRATION + DELIVERY EXPLAINED

At RightScale, we leverage our own platform to manage and automate infrastructure as part
of CI/CD processes. We leverage three key capabilities:

Multi-cloud support: The RightScale platform provides a multi-cloud API that enables us
to deploy our infrastructure to any cloud at any point in the lifecycle. For example, we can
leverage capacity in a VMware vSphere environment during development phases then deploy
to a public cloud for testing and production.

Infrastructure-as-code: RightScale supports a variety of configuration management options

that enable us to configure servers using code with such tools as Chef, Puppet, and Ansible.
This ensures consistency between development, test, staging, and production environments.

Automate infrastructure: With the RightScale API, we can automate all of the provisioning,
updating, scaling, and deprovisioning of infrastructure and integrate those processes into
our CI/CD tools and processes.

2015 RightScale, Inc. All rights reserved.

CONTINUOUS INTEGRATION + DELIVERY EXPLAINED

The RightScale development organization uses various tools in addition to the

RightScale platform to efficiently automate our testing across a large matrix of clouds
and operating systems.

2015 RightScale, Inc. All rights reserved.

CONTINUOUS
TESTING
IN THE CLOUD

CONTINUOUS TESTING IN THE CLOUD

At RightScale we strive to automate all aspects of continuous
testing in the cloud from executing tests to the provisioning of
infrastructure. To do this, our development organization leverages
Jenkins, various test tools, and the RightScale platform, itself,
to automate testing of our RightScale ServerTemplatesTM
scripts to launch servers across a large matrix of clouds and
operating systems.

The Problem: A Test Matrix from Hell

As part of our solution, RightScale develops and delivers preconfigured ServerTemplates for popular technologies and stacks,
such as LAMP, MySQL, and more for both Linux and Windows.
These ServerTemplates are designed to operate across a broad
number of clouds (AWS, Azure, Google, Rackspace, CloudStack,
OpenStack, VMware vSphere, and more) as well as a variety of
operating systems such as CentOS, Ubuntu, RHEL, and Windows.
Since RightScale provides ongoing technical support and bug fixes
for these ServerTemplates, its critical that we test them across all
of the platform/OS combinations we support.

2015 RightScale, Inc. All rights reserved.

21
We know that
high performers are
deploying code
30 times more
frequently. We know
they can complete
those deployments in
minutes or hours as
opposed to weeks,
months, or quarters.
In a production
deployment, high
performers are twice as
likely to succeed, and
once something goes
wrong, they can fix the
issues 12 times faster.
Gene Kim
Author of The Phoenix Project:
A Novel About IT, DevOps, and
Helping Your Business Win

10

CONTINUOUS TESTING IN THE CLOUD

21

As the number of ServerTemplates, tests, and clouds grew, the result was a test matrix that had as
many as 2,500 possible combinations:

Although we were using Jenkins as a front end to our homegrown regression testing tool,
VirtualMonkey, the amount of time and manual effort to set up the test matrix (which was
originally a GoogleDoc), execute the tests, and review the results quickly became unmanageable.
In addition, each cell in the above matrix represented a test that might take 20 minutes to 2 hours.
We needed to balance running tests in parallel to reduce the elapsed time with serializing tests
in order to manage the cloud capacity and budget required.
2015 RightScale, Inc. All rights reserved.

11

CONTINUOUS TESTING IN THE CLOUD

21

Our Solution: RocketMonkey, Jenkins, and RightScale

To solve this problem, we developed a new tool, RocketMonkey, that reads a test matrix from a
GoogleDoc and translates each cell into individual jobs loaded into Jenkins. RocketMonkey then
drives these jobs via the Jenkins API. RocketMonkey also allows us to define what tests run in parallel
and how tests are sequenced.

2015 RightScale, Inc. All rights reserved.

12

CONTINUOUS TESTING IN THE CLOUD

21

After the jobs are configured, the test activity is coordinated by RocketMonkey, which selectively
triggers jobs in Jenkins, which in turn uses VIrtualMonkey to execute the test scripts. The test
scripts call the RightScale API to provision, auto-scale, and deprovision cloud deployments as
required for each tests. These cloud deployments are defined within RightScale and pull the latest
versions of the ServerTemplates that are under test.
RocketMonkey provides control of the test execution at a high level, allowing us to address several
issues that are a direct result of large-scale testing. It can be configured to link verticals or rows in
the test matrix, allowing for running tests on a specific cloud or the same test across all the clouds
at will. The tests can be linked to run serially, so they consume a limited level of cloud resources
while they run overnight unattended or in parallel to provide time efficiency.

2015 RightScale, Inc. All rights reserved.

13

CONTINUOUS TESTING IN THE CLOUD

21

As tests complete, a generated HTML report based on the original GoogleDoc is automatically
updated with a status icon so that the QA team can easily see how the tests are progressing.
The automation can also be set to auto-halt testing if repeated failures are detected, say, for
example, if there are network problems. Finally, one of the biggest time savings is in the automatic
management of the result logs, which are available via links on the individual green/red dots:

2015 RightScale, Inc. All rights reserved.

14

CONTINUOUS TESTING IN THE CLOUD

21

Bottom Line: Reduce Manual Effort, Speed Development Cycles

By integrating RightScale into our CI and testing processes, we are able to automate all aspects
of the continuous testing, from the execution of tests to the provisioning of infrastructure needed
to support these tests. As a result, weve been able to greatly reduce the manual effort involved
in testing. It is difficult to estimate the efficiency gain in numbers, but it is safe to say that it
exceeds 10x. Rather than consuming an entire team of 10 people for 4 weeks, we were able to
run regression in 3 weeks with only 2 people dedicated to seeing all the tests through. The rest
of the team was then free to respond to fixes or work on planning the next sprint.

2015 RightScale, Inc. All rights reserved.

15

DYNAMIC
SCALING OF
JENKINS
IN THE CLOUD

DYNAMIC SCALING OF JENKINS IN THE CLOUD

31

The RightScale development organization uses Jenkins to enable us to to automatically and

continuously test our code.

The Problem: Many Developers + Many Changes = Jenkins Bottleneck

At RightScale we have dozens of developers working on the RightScale platform. As each of
these developers commits changes to the code repository, our CI process kicks in to automate
integration and testing of the changes. However, when many changes are being made each day,
this can create a bottleneck in the Jenkins environment.

2015 RightScale, Inc. All rights reserved.

17

DYNAMIC SCALING OF JENKINS IN THE CLOUD

31

Our Solution: Manage Jenkins with RightScale

Jenkins is designed with a master/slave architecture. To process more Jenkins jobs, you need to
add more Jenkins slaves. However, we didnt want to provision a lot of Jenkins slaves that would
sit idle when the demand for CI was low. We wanted to go further and dynamically scale up the
number of Jenkins slaves based on the size of the Jenkins queues.

2015 RightScale, Inc. All rights reserved.

18

DYNAMIC SCALING OF JENKINS IN THE CLOUD

We created a RightScale ServerTemplate to deploy Jenkins in
the cloud along with a RightScale auto-scaling server array. This
Jenkins environment included a RightScale RightLinkTM agent
a software package installed on instances for securely managing
a large numbers of instances in the cloud that monitored the
size of the Jenkins queue and triggered RightScale to auto-scale
the number of Jenkins slaves when the queue got too long. As
the number of Jenkins slaves increased, the Jenkins queue could
be processed more quickly. Similarly, when the queue got shorter,
RightScale would decrease the number of Jenkins slaves.
The number of Jenkins slaves we use ranges from 1 to 6 depending
on rate of new commits entering the build queue and the duration
of the individual CI jobs. We have experimented with changing
the server array scaling sensitivity on weekends and outside of
business hours, but have discovered that its better to optimize the
boot time for new slaves than to try and predict when developers
will have work to test.

2015 RightScale, Inc. All rights reserved.

31
DevOps and
continuous delivery are
most likely to occur with
the help of cloud-based
tools or when operating
in cloud infrastructure.
Seventy-seven percent
[of developers] said
their continuous delivery
effort was executed
either primarily or
entirely through the
cloud, while 18.8 percent
said it was primarily done
through on-premises
infrastructure.
Charles Babcock
Editor-at-large at
InformationWeek,
citing results of the
Evans Data Corporation
North American
Development Survey 2014

19

DYNAMIC SCALING OF JENKINS IN THE CLOUD

31

Bottom Line: Speed Development, Lower Costs

By dynamically scaling Jenkins, we are able to ensure fast turnaround on CI and testing
so that developers can get feedback more quickly and ultimately deliver features to our
customers more quickly. In addition, we are able to right-size our Jenkins slaves and therefore
use only the cloud resources that we need at any point in time. As a result, we can reduce
our cloud costs by eliminating overprovisioning and overspending.

2015 RightScale, Inc. All rights reserved.

20

CONTINUOUS
DELIVERY OF
SOFTWARE
PACKAGES
IN THE CLOUD

CONTINUOUS DELIVERY OF SOFTWARE PACKAGES

IN THE CLOUD

41

Continuous delivery in the cloud enables a more agile software delivery process to enable you
to deliver releases to your customers more quickly. By implementing continuous delivery of
our RightLink package, it has enabled the RightScale development organization to release as
frequently as needed based on customer demands.

The Problem: Compressing Long Release Cycles with Continuous Delivery

Although the RightScale platform is a SaaS solution, we also develop a software agent,
RightLink, that is automatically installed on our customers cloud instances and communicates
with the RightScale platform to monitor these instances and perform management functions.
In the early days of RightScale, RightLink was released every 12-18 months. However, as the
company grew and gained more customers, we wanted to release more frequently. In addition,
the long release cycles with many code changes pushed the discovery of bugs to the end of the
release process long after the code changes had been made. This made it more difficult to find
and fix the issues.

2015 RightScale, Inc. All rights reserved.

22

CONTINUOUS DELIVERY OF SOFTWARE PACKAGES

IN THE CLOUD

41

The Solution: Continuous Delivery of RightLink

To accelerate the release cycle, we implemented a continuous integration and delivery process.
Each time changes are made to the code in the GitHub repo, Jenkins is triggered and unit tests
are run. Every night, we use the RightScale API to launch a number of servers one for each
supported architecture and OS which build the package and upload it into cloud storage.
Then, using the RightScale API, we provision test servers in the cloud to run the newly built
packages and automatically execute Chef test scripts. Any issues found are reported to the
development team so that fixes can be made immediately.

2015 RightScale, Inc. All rights reserved.

23

CONTINUOUS DELIVERY OF SOFTWARE PACKAGES

IN THE CLOUD
The nightly build is made available for both internal use by RightScale
developers as well as for customers who want to test the latest and
greatest features. Because we now have a tested package at all times,
we can make release decisions based on customer needs or market
demand. If a customer needs a particular feature, we simply do final
regression testing and then create a release.

As a result, our releases are now averaging about three months

apart and could be even shorter if needed. This enables us to get
new features in the hands of customers more quickly.

2015 RightScale, Inc. All rights reserved.

41

While a cycle time

measured in months
is common, many
companies do much
worse than this . . .
Automation is the key.
It allows all of the
common tasks involved
in the creation and
deployment of software
to be performed by
developers, testers, and
operations personnel,
at the push of a button.
Jez Humble and David Farley
Authors of Continuous
Delivery: Reliable Software
Releases through Build, Test,
and Deployment Automation

24

KEY TAKEAWAYS
+ ADDITIONAL
RESOURCES

KEY TAKEAWAYS + ADDITIONAL RESOURCES

51

Key Takeaways
Rather than adopting a one-size-fits-all approach for development, the practices we choose at
RightScale for each product or component depend on the business goals we are trying to achieve.
By embracing the agile development practices of continuous integration and continuous delivery
and by leveraging our own RightScale Cloud Portfolio Management platform to automate
provisioning and management of all of our cloud infrastructure, we have reduced the time-tomarket of new features and products.
To be highly effective, your continuous integration and continuous delivery processes might
incorporate some or all of these components:

Multi-cloud access including public, private, and virtualized environments.

Infrastructure-as-Code model.
Consistency across environments (dev, test, staging, and production).
Automation of infrastructure as part of your application lifecycle.
Ability to rapidly provision/scale/update/decommission.
Provision of self-service access to developers.
Reduction of cycle times at every stage.
Additional Resources
DevOps and Cloud: How to Make Business Love IT [Blog]
Become a Cloud Services Broker: How IT Can Deliver Cloud Services [White Paper]
DevOps in the Cloud: How RightScale Does It [Video]
2015 RightScale, Inc. All rights reserved.

26

REQUEST A FREE CONSULTATION

Speak with a RightScale expert who can advise you on how
to best implement or improve the continuous integration and
continuous delivery process in your organization.

REQUEST YOUR CONSULTATION

Or call us at 1.888.989.1856

www.rightscale.com