Proceedings of 11th International Conference on Computer and Information Technology (ICCIT 2008)
25-27 December, 2008, Khulna, Bangladesh
Server Consolidation Using OpenVZ: Performance Evaluation
Mohuiddin Ahmed, Showayb Zahda, Majed Abbas
Computer Science Department
International Islamic University Malaysia
Abstract—Virtualization is abstracting computer resources. It is
the ability to run multiple operating systems simultaneously on a
single physical machine. In this paper we evaluate the
performance of the operating system level virtualization software
(OpenVZ). The evaluation will be based on Web hosting, in other
words, we evaluate the performance of Web servers on two
different machines (dubbed as Baru and Lama) and we observe
the behavior of OpenVZ and its scalability for Web servers on
different architectures (32 bits and 64 bits). The study evaluates
the CPU, Memory and Throughputs under different loading
conditions.
Index Terms — OpenVZ, performance, server consolidation,
virtualization.
I. INTRODUCTION
“Server consolidation is an approach to the efficient usage
of computer server resources in order to reduce the total
number of servers or server locations that an organization
requires”[1]. Simply put, server consolidation is the
combination of several servers (i.e. Web Server, Mail Server,
Java Server, etc) into one physical machine, in order to, save
cost and utilize the computer resources.
One way to achieve server consolidation is to use
virtualization. Virtualization as a word does not exists in most
English dictionaries. However, the closest word to it in
Oxford Concise Dictionary is “virtual” which means
“Computing: not physically existing as such but made by
software to appear to do so” [2]. However, virtual is not a new
term in the computer jargons, it was used before such as
virtual memory, virtual reality, etc.
Virtualization in our context according to [3] is defined as
“the abstraction of computer resources” and “abstraction is the
process of separating hardware functionality from the
underlying hardware.” In other words, virtualization is
sharing, and partitioning the resources of computer hardware
amongst more than one running operating systems on a single
physical machine simultaneously. Each one of the running
operating systems is called a Virtual Machine (VM).
Virtualization offers the ability to run multiple operating
systems (Linux, Windows, MacOS X) on one physical
machine at the same time.
Currently, virtualization is grabbing the attention of the data
centers as well as the small and medium enterprises due to its
enormous advantages. This led processor manufacturer mainly
Intel and AMD to develop processors to support virtualization
i.e. Intel VT and AMD-V respectively.
[1] The corresponding author.
1-4244-2136-7/08/$20.00 ©2008 IEEE
Indeed, virtualization brings a huge economical value to the
organizations. Mainly, it reduces the Total Cost of Ownership
(TCO) by consolidating a bunch of physical servers into one
physical server. For example, 10 Web servers can use one
physical machine instead of using 10 physical machines. This
reduces a lot of cost -beside the cost of the machines- such as
electricity bills, space, air conditioning, etc. In addition,
organizations can couple virtualization with open source
software solutions (e.g. OpenVZ, Xen) to achieve more
reduction in TCO and not to bother anymore about licensing
issues. This is handy when multiple virtual machines or virtual
machine backups are kept online for redundancy, restoration,
or disaster recovery purposes [4].
Virtualization technology can be implemented in several
techniques namely, hardware emulation, full virtualization,
paravirtualization, operating system level virtualization,
application virtualization, and desktop virtualization.
However, the discussion about the differences amongst them
is beyond this paper. But, a brief discussion about operating
system level virtualization will be provided in OpenVZ
section.
Besides cost savings, virtualization in general has great
advantages for the organizations, due to the fact that the
virtualization layer isolates and separates the VMs. None of
the VMs is aware about the existence of other VMs on the
same machine. They seem to each other as if they were
running on different physical machines. The isolation provides
higher availability and security for the organization. If one of
the VMs crashes or goes down, it has no effects on other
VMs. Moreover, system administrators can fire up a VM in
less than a minute. As for security, running different services
on different VMs improves the security. For instance, if an
intruder manages to hack one server, only that particular
server goes down and others will remain running as if nothing
happened.
Live migration of VMs is yet another great advantage of
virtualization. Assume one of the VMs is exhausting the
hardware resources, then the administrator can migrate the
server into another physical machine in less than one minute.
Or if the physical machine should go down for upgrade or
maintenance, the VMs can be migrated to another machine.
Therefore, migration boosts the availability of the running
servers.
Software developers and testers use virtualization to run
several platforms to develop and test the products on (i.e.
Linux, Windows, MacOS X). It reduces the time of installing
the platforms as well as increasing the number of platforms
341
that can be used for testing and development.
Virtualization is indeed a very helpful technology for
education purposes. Now, universities and educational
institutions no longer need to buy extra machines to cater their
student’s need of running several operating systems.
Moreover, each student can have a root access which allows
them to have full authority over the system.
Virtualization has disadvantages too; a big damage to all the
running servers might occur if the power of the host machine
goes down. Or, if the administrator is upgrading the system
hardware, he/she has to shutdown the machine. However, by
using migration, the system administrators can easily migrate
the VMs to another physical machine in less than one second.
The rest of the paper is organized as follows: in section II
OpenVZ is briefly described, in section III we describe the
methodology of the performance evaluation, section IV
describes the test bed and its components, in section V we
analyze the test results followed by concluding remarks.
II. OPENVZ
OpenVZ is a free and open source virtualization solution. It is
the basis of the commercial virtualization solution Virtuozzo
from Parallel (formerly known as SWsoft). OpenVZ
implements the operating system level virtualization technique
which virtualizes servers on the operating system (kernel)
layer [5]. In other words, OS-level Virtualization adds a new
layer on top of the operating system. This layer manages all
the issues related to OpenVZ containers and resource
management.
Fig. 1. Operating System level Virtualization [5]
OpenVZ has its own terminologies regarding virtualization
technology. The physical machine that is running OpenVZ is
called a Host System or Hardware Node. A running instance
of the operating system that partitions and shares the resources
of the hardware node is called a container, Virtual
Environment (VE), and/or Virtual Private Server (VPS) [6]. In
this paper we will refer to the running instance as a container.
Operating system-level virtualization shares the kernel
amongst all running containers. The kernel (OpenVZ kernel
creates the virtualization layer) bears the duty of managing
and partitioning the resources of the computer, and isolating
342
the containers from each other. However, sharing one kernel
introduces a drawback. In case the kernel crashes all the
containers are down.
The performance of OpenVZ is closed to the native
performance of the machine. The overhead only consumes
about 1-3% of the resources. This indeed makes OpenVZ a
good choice for server consolidation especially for web
hosting [7].
OpenVZ is based on Linux operating system and it only
runs in Linux distributions. Moreover, OpenVZ can also be
tested without installation using OpenVZ LiveCD1.
It takes a few minutes to setup one container using
OpenVZ. This is because of the usage of Operating System
Templates. An OS-template is a set of packages from some
Linux distribution used to populate a VE. With OpenVZ,
different distributions can co-exist on the same hardware box,
so multiple OS templates are available. [6]
Furthermore, changing the configuration of any container
can be done on the fly. There is no need to restart any
container in order to give or take from it some resources.
OpenVZ comes with a command line control utilities that
allow the administrators to create, start, stop, destroy and
migrate a container. The utilities also allow him/her to set the
quota, manage the resources and execute commands on the
containers from outside the container itself. For instance,
“vzsplit” is a command that creates a configuration file which
splits the resources of the computer equally amongst a number
of containers the user specifies (vzsplit -n 4).
Fig. 2. WebVZ Management Tool
WebVZ is a web management tool for OpenVZ and was
developed by the authors and available in [8] under GPL
GNU. WebVZ allows system administrators to control the
containers locally and remotely. Fig 2 shows a screenshot of
WebVZ. Resource allocation among the containers is done on
the basis of demand. If there is need to migrate a container to
another server due to resource hogging, this can easily be
attained by migration to a different machine which is lightly
1
Live CD: is an operating system that runs without the need for
installation on the hard disk and it does not have any effect on the existing
data. Once the CD is ejected, everything goes back to its original state
loaded. Fig 3 shows the output of the command “vzlist” (list In order to capture the reading of the CPU and the RAM we
all containers) used “top -b -d 1” to read the values every second. While the
throughput of each test was reported by “httperf’ for plotting
and display the result, “bench2graph” is a tool comes with
autobench, was used to plot the output of the tests.
This paragraph explains how autobench runs httperf. (E.g.
autobench --single_host --host1 192.168.0.1 --low_rate 50 --
high_rate 100 --rate_step 10 --timeout 7 --file ve1.tsv). This
command will run “httperf” 5 times. First time httperf initiates
50 connections per second until the total number of
connections reaches 5000 connections (the default) which is
the total number of connections that are initiated during one
test, and each connection contains 10 requests (the default). In
the second test httperf initiates 60 connections per second
until the total number of connections reaches 5000 and so on,
each time the number of connections per second increases
according to the specified rate step value. The output of
Fig 3 vzlist output httperf is shown in Fig 4 which is extracted and tabulated by
autobench as shown in Fig 5.
III. METHODOLOGY

In order to test the migration between containers we

injected http queries from a server and then used a wrapper to
increase the number of requests until the system chokes or
crashes.
In order to evaluate the performance of the containers on
OpenVZ kernel, first we evaluated the performance of both
machines (we named them Baru and Lama respectively) as a
single unit without running OpenVZ kernel so as to know how
many requests per second each machine can handle and to
find out the relationships between the containers and the Host
System. Second, we evaluated the containers by scaling the
number of containers from 4 to 8, and finally to 12 on the two
machines. All the containers receive the same amount of Fig 4 httperf output sample
requests per second concurrently. In other words, each
container for example is receiving 100 connections per second
at the same time.
In order to measure the performance of each running Web
server we used a free and open source HTTP performance
measurement tool from HP called “httperf” version 0.9.0.
According to the manual page of httperf “httperf speaks the
HTTP protocol both in its HTTP/1.0 and HTTP/1.1 flavors
and offers a variety of workload generators. While running, it Fig. 5. autobench output sample
keeps track of a number of performance metrics that are
summarized in the form of statistics that are printed at the end IV. TESTBED
of a test run. The most basic operation of httperf is to generate The tests were conducted on two different servers.
a fixed number of HTTP GET requests and to measure how First server’s specifications (dubbed as Baru) were Intel
many replies (responses) came back from the server and at Core 2 Duo 1.83 GHz, 2 GB of RAM, and 100 Megabit
what rate the responses arrived.” network interface (64 bit machine). And second server’s
However, httperf is limited to one test at a time which specifications (dubbed as Lama) were Intel Pentium 4 2.6
generates a fixed number of HTTP GET. In order to run GHz, 1 GB of RAM, and 100 Megabit network interface (32
httperf several times and increase the number of HTTP GETs bits machine). Both machines hosted OpenVZ on Centos 5.
each time, another tool called “autobench” can be used which The tests were conducted from two clients both running
automates web server benchmarking using httperf. According Ubuntu 7.10. Fig 6 portrays the architecture of the test bed.
to autobench version 2.1.1 manual page “autobench runs The tests were conducted based on the instructions and
httperf against the specified host or hosts, ramping up the practices of [9]. We conducted the tests in isolated network
number of requested connections, and logging the results in environment.
TSV or CSV format files.”

343
 The following configuration of “autobench” was done to V. DISCUSSION
measure the performance. When we benchmarked the whole
machine, the total number of connections to be sent to the A. OpenVZ
machine was set to 20,000. The number of requests in each There are general observations about OpenVZ that we
connection was set to 10. The client time out was set to 7 would like to talk about in this section.
seconds. However, when we benchmarked the containers, for It is easy to fire up a new container using OpenVZ. It would
each container, 5,000 connections were sent in each test. And take about one minute. Moreover, a container occupies only a
the rest of the configuration parameters were the same as small amount of memory. However, this depends on the size
above. of the distribution as well as the running services. In our case,
OpenVZ kernel version 2.6.18, and Centos 5 templates from we only fired up the web server and the default services that
OpenVZ website were used and Apache web server version run on the Centos 5 template from OpenVZ Website. We
2.2.3 was used as the Web server. The http queries were found that each VE takes around 25 MB of RAM and about
generated by using Httperf version 0.9.0 and autobench 2.1.2. 400 MB of hard disk. Accordingly only 20 containers can be
We conducted various tests by scaling the number of running web servers using 450 MB of RAM and about 8 GB
containers from 4 to 8 and finally to 12. of hard disk.
The configurations of Apache were left to the default We ran 55 containers on Baru and 23 on Lama at the same
except the “KeepAlive Off” which we changed to “On” in time. This almost filled the memory of each machine
order to allow multiple requests in one connection. respectively. Moreover, we ran the containers with no
application overhead within itself.
B. The host machine (Baru)
After running “autobench” (from 1000 to 7000 requests per
second), the machine managed to handle about 6000 requests
per second with no error. This amount of concurrent requests
has saturated the CPUs of the machine. However, the
throughput was not saturated due to the size of the files (~80
Kilo bytes).
Figure 8 shows the results of “autobench” which uses
httperf to benchmark the machines. In Fig 8 the X-axis
represents the number of requests per second that httperf is
supposed to send each test, and Y-axis represents different
values that changes based on the figure’s keys. In the
following paragraph, we will describe what each line means.
Sent request per second is the number of requests that was
Fig. 6. The Test Bed architecture sent to the machine each second. Average reply rate per
second is the average number of replies from the server per
The configuration files of all the containers were the same second. Response time is the reply time of the request in
and a sample is shown in Fig 7. milliseconds. Throughput is the size of the bandwidth that was
used during each test in Kilo bit per second. And finally we
ONBOOT="yes" observed the percentage of errors for each test as packet loss.
# UBC parameters (in form of barrier:limit)
KMEMSIZE="11055923:11377049" As for Fig 8 we see that the first test sent 1000 requests per
LOCKEDPAGES="256:256"
PRIVVMPAGES="65536:69632"
second and the replies were all intact. In the next test which
SHMPAGES="21504:21504" was 1500 requests per second also it was fine. Autobench
NUMPROC="240:240"
PHYSPAGES="0:2147483647" conducted 14 tests starting from 1000 requests per second and
VMGUARPAGES="33792:2147483647"
OOMGUARPAGES="26112:2147483647"
ending at 7000 requests per second. Each test autobench
NUMTCPSOCK="360:360" increases the number of requests per second by 500. So, we
NUMFLOCK="188:206"
NUMPTY="16:16" see that until 6000 requests per second everything was fine
NUMSIGINFO="256:256"
TCPSNDBUF="1720320:2703360"
and no errors occurred. After 6000 requests per second we
TCPRCVBUF="1720320:2703360" notice there is a drop in the number of requests sent to the
OTHERSOCKBUF="1126080:2097152"
DGRAMRCVBUF="262144:262144" server which is a bit steep. And so did every thing. The drop
NUMOTHERSOCK="360:360"
DCACHESIZE="3409920:3624960"
occurred due to the CPUs saturation which was utilized 100%.
NUMFILE="9312:9312" Fig 9 shows the CPU usage in relation with the time. X-axis
AVNUMPROC="180:180"
NUMIPTENT="128:128" represents the time line of the test. We captured the reading of
# Disk quota parameters (in form of softlimit:hardlimit) the CPU each second for the whole duration of the tests. Y-
DISKSPACE="1048576:1153024"
DISKINODES="200000:220000" axis shows the percentage of the idle CPU. Why the idle?
QUOTATIME="0"
# CPU fair sheduler parameter Because the percentage of the idle CPU show the percentage
CPUUNITS="1000" of the free portion of CPU, we can see whether the whole
system including the web server is utilizing the CPU or not.
Fig. 7. Configuration of a typical container
344
 Fig 9 shows the CPU idle percentage. We observe that at
the beginning of the test the usage of the CPU was about 10%.
This value keeps fluctuating each second and keeps going
down until it reaches 0% which means that the CPU is fully
used. In this test, this point was reached at approximately at
210 seconds and error occurrences were increasing from this
point and onward (i.e. 6500 requests per second and beyond).
Fig 10 shows the usage of the RAM of the whole duration
of this test.
The results indicate that Baru machine can handle about
6000 requests per second and the CPU was saturated.
However, in reality, usually benchmarking is done up to 80%
of the CPU usage.
C.The host machine (Lama)
Since we are testing the performance of OpenVZ on two
different machines, section A presented the results of the first
machine (Baru) whereas this section presents the second
machine (Lama). In this section we will not present the graphs
and we will only mention the results.
After running “autobench”, the machine managed to handle
about 3300 requests per second with no error. This amount of
concurrent requests has saturated the CPU of the machine.
However, the throughput was not saturated due to the size of
the files (~80 Kilo bytes).
Fig. 8. The performance of Baru.
D. OpenVZ on Baru and Lama
In this section we will describe the results of running
OpenVZ on the two machines. The results are tabulated but
brief discussion is provided here about the result for
clarification.
As mentioned before we scaled the number of containers
from 4 to 8 and finally to 12 and we conducted the tests on
them so as to find the relationships between the machine and
OpenVZ. Surprisingly, the two machines have had the same
relationship with OpenVZ. The results are described first and
later we show the relationship between the machine and
OpenVZ.
Running 4 containers on Baru:
We set up Baru with 4 identical containers and we ran
autobench against them concurrently.
345
In other words, all the containers receive the same load at
the same time. Fig 11 shows the results of autobench on one
of the containers (the other containers have almost an identical
graphs so we present one of them here). As the graph shows
autobench starts from 200 requests and ends at 2000 requests.
However, the errors started to occur at about 1600 requests
per second. This means that the machine is receiving about
6400 requests per second at the same time. Note that this
number is about the same as the performance of machine Baru
in section A. Surprisingly, the CPU usage (Fig 12) was almost
the same as in section A and is saturated at the same the error
occurrences. As for the RAM (Fig 13) it has also the same
usage as in section A.
Fig. 9. CPU Usage of Baru
Fig. 10. RAM usage of Baru
Running 8, and 12 containers on Baru:
The results of 8 and 12 containers were almost the same as
when we were using four containers. In 8 containers, each
container handled about 750 requests per second, and in 12
containers, each container handles 500 requests per second
with no errors. The graphs for the CPU and RAM were the
same as Baru machine and 4 containers. Table 1 shows the
relationship between the performance of Baru and 4, 8, and 12
containers. We can see from the table that OpenVZ has nearly
0% overhead. This means that we do not lose any hardware
resources when we are running OpenVZ.
 TABLE 1
Performance of Baru and its containers
Machine Number Requests Total requests As for Lama we mentioned before that it handled about
of servers per second per second 3300 requests per second. So, when we ran 4 containers on it
Baru 1 ~6000 ~6000 we found that 700 requests per seconds were handled by each
4 containers 4 ~1600 ~6400 container which means 700 * 4 = 3200 requests per second
8 containers 8 ~750 ~6000 which is about the same number of requests that were handled
12 containers 12 ~500 ~6000 by Lama. Note that we did not manage to run 12 containers on
Lama due to RAM limitations.

TABLE 2
Performance of Lama and its containers
Machine Number of Requests per Total requests
servers second per second
Lama 1 ~3300 ~3300
4 containers 4 ~700 ~3200
8 containers 8 ~400 ~3200

Fig. 11. Performance of one of the containers
VI.CONCLUSION
We found that when using OpenVZ, almost no overhead on
the computer resources and it shares the resources of the
computer fairly amongst the running containers. Moreover,
we found that both machines (Baru and Lama) can handle
about 6000 and 3300 requests per second respectively. And so
do containers of OpenVZ when we run all of them
concurrently on the both Baru and Lama. This means that
OpenVZ performance does not change with the change of the
architecture (32 bits and 64 bits). We can safely say that if we
want to run 15 Web servers using OpenVZ containers, each
VE can handle 400 requests per second at the same time (this
is on Baru). The WebVZ is a very useful tool for
administrating OpenVZ containers both locally and remotely.

Fig. 12. CPU usage of 4 containers
REFERENCES
[1] Information Technology @ Johns Hopkins-Glossary of
Technical Terms and Acronyms. Available
http://it.jhu.edu/glossary/pqrs.html
[2] Concise Oxford English Dictionary (Eleventh Edition).
Computer Software.
[3] Wolf C., Halter E. (2005). Virtualization from the desktop to the
enterprise. Apress, 2005. p.1.
[4] Shields G., The Shortcut Guide to Selecting the Right
Virtualization Solution, electronic book. Available:
http://nexus.realtimepublishers.com/SGSRVS.htm
[5] OS Virtualization. Available:
http://www.parallels.com/en/products/virtuozzo/os/
[6] OpenVZ Wiki. Available:
http://wiki.openvz.org/Category:Definitions
[7] Padala P., Xiaoyun Zhu, Zhikui Wang, Singhal S., Shin K.
Performance Evaluation of Virtualization Technologies for
Server Consolidation. HP Laboratories Palo Alto, April 11,
2007. Available: http://www.hpl.hp.com/techreports/2007/HPL-
2007-59.pdf
[8] WebVZ. Available: http://webvz.sourceforge.net
[9] The Linux HTTP Benchmarking HOWTO. Available:
http://www.xenoclast.org/doc/benchmark/HTTP-benchmarking-
HOWTO/

Fig. 13. RAM usage of the 4 containers

346