WHITE PAPER

Traffic Distribution
over Time and Space
An Operator Focused Approach to
Inter Cloud Scenarios

http://www.smartenit.eu

1. SmartenIT at a Glance
The massive adoption of the cloud computing paradigm and its broad portfolio of service
offerings poses substantial challenges to different stakeholders playing in the cloud landscape, more specifically Internet Service Providers (ISP), Data Center owners, and Cloud
Service Providers (CSP). Most of these challenges derive from the fact that new services
are offered according to the cloud principle of service transparency, where end-users are
fully unaware of heterogeneous underlying cross-layer infrastructures and assets that
compose the service purchased.
Today and at the state of art all stakeholders contributing to the service chain lack a unified
and cooperative view on the end-to-end service to be exposed to the end-user, which
results in the absence of a cross-layer, inter-domain approach to challenges posed by
service composition and proper end-to-end Service Level Agreements (SLA) management.
Major drawback of the depicted landscape is the unfair management of inter-cloud/ interdomain traffic, with extra burden on ISP/CSP owned infrastructure.
Another key topic, which constitutes at the same time a challenge as well as an opportunity, is the growth of traffic driven by social networks, which turns out for its unpredictability and burstiness of network traffic to be running on an increasing number of energy demanding end-user devices and network equipment. These energy awareness considerations play another important role. Cloud computing, by leveraging virtualization technology, is inherently more energy-efficient than an older computing paradigm, but its full
potential of savings has not been exploited yet. In this complex and multi-sided landscape
SmartenITs main idea and approach developed is to innovate on two main aspects, which
are strictly correlated: the first is related to the analysis of new business schemes, which
are intended to forecast cross-layer and inter-domain cooperation with an incentive-based
approach, the second is to develop new traffic management mechanisms, which leverage
newly developed business schemes. More in detail SmartenITs innovation explored the
possibility of embedding new traffic management solutions with main topics as inter-domain
collaboration, energy-aware and social-awareness considerations, to provide a tangible
impact on end-user devices and underlying networking infrastructure as well as application
provisioning infrastructure of service providers.
Therefore, SmartenITs landscape has been divided in two macro perspectives, namely the
Operator Focused Scenario (OFS) and the End-user Focused Scenario (EFS).
The first scenario is dedicated to the analysis of inter-relationships among service providers, whose infrastructure is used to compose cloud services to be offered to end-users,
with a major focus on the management of inter-domain traffic and on the analysis of mutual
interests of those stakeholders involved.
The second scenario sees the end-user as its main focus, dealing with topics of end-user
interaction in a social network landscape and Quality-of-Experience (QoE), when purchasing cloud services. By leveraging these two scenarios, SmartenIT has designed suitable incentive-compatible cross-layer network management schemes to drive an efficient and beneficial interaction among service providers and end-users.
The following sections of this white paper introduce SmartenITs approach and solutions
across the OFS scenario domain.

SmartenIT White Paper OFS

2. Overlay Applications in Cloud Scenarios:

A New Challenge for Stakeholders
Internet services such as IPTV (Internet TV), video and gaming, are used by millions of
geographically dispersed people. Clouds, Content Delivery Networks (CDN), and online
social networks have resulted in complex services where resources, data and service
points are virtualized and instantiated at/migrated to multiple locations so as to efficiently
serve the users. At the same time, the Internet faces well-known inefficiencies.
Asymmetric information between the ISP and its customers (cloud providers, other ISPs) is
a major issue, since ISP are not willing to disclose internal details and policies of owned
infrastructure to other ISPs and cloud/application service providers, rendering inter-domain
monitoring impossible. Furthermore, there are no reward schemes for ISPs willing to
provide assured quality to inter-domain traffic since networks solely exchange data and
BGP (Border Gateway Protocol) information, lacking standardized service-aware inter-provider service coordination in both business and technical layer. This is also due to the fact
that interconnections contracts pertain to large traffic aggregates, thus there is no servicespecific overlay for optimizations, and provide only uptime guarantees and absolutely no
guarantees on quality. This lack of end-to-end inter-provider Service Level Agreements and
respective multi-provider service-aware connectivity products drives high quality out of the
market: Since there are no quality guarantees, it is impossible for buyers to predict the
quality of inter-domain flows, thus they expect and bias their willingness to pay on the
average quality observed in the market, which essentially renders any higher-quality
product unsustainable. ISPs opportunistic routing and prioritizing owned flows over interdomain traffic further mitigates quality. Overall, these inefficiencies are due to the lack of
service-aware coordination and cooperation among ISPs, clouds and ISPs, and clouds.
This further renders inter-provider cloud and content service provisioning problematic and
greatly affecting the end-users Quality-of-Experience (QoE). This is also of particular interest to Europe and similar areas where there are typically multiple geographically-limited
ISPs, datacenters and cloud service providers without global Information and Communication Technology infrastructure, rendering intra-provider traffic management and crosslayer network-cloud layer optimization solutions inapplicable.
To this end, data centers, CDNs and proxy servers attempt to mitigate these inefficiencies
to some extent and improve service performance by bringing data and services closer to
the users. However, this requires continuously transferring potentially vast amounts of data
among clouds and data centers/caches across multiple Internet locations and domains. In
particular, inter-cloud communication constitutes a thriving business and is further growing
due to the emerging federations of clouds and CDNs, such as CDNI (Content Delivery Networks Interconnection), OnApp, XIFI, or Zentera, that enable their members to dynamically
lease presence to multiple geographic regions when and for as long needed. This elastic
way to provision cloud resources and inter-cloud data transfers complement the CDNs
content caching for cloud applications requiring good QoE for end users, redundancy, faulttolerance and reliability.
Those data transfers comprise the opaque to the users back-office, hence not visible to
end-users, Web traffic that generates a large portion of the Internet inter-domain traffic,
being up to 30% and occasionally 40-50% of the total traffic of popular Internet transit links.
The main sources of this traffic, which is expected to increase due to video, have been

SmartenIT White Paper OFS

identified to be mainly related to Web proxies/intermediaries, CDN servers, ad exchanges

and Web crawlers. This large back-office portion of transit traffic is not time-critical for the
end users since it is not directly associated to their flows and thus substantial transit cost
savings can be attained for ISPs and also for clouds by applying smart traffic management.
This motivates SmartenIT research on traffic management mechanisms that consider the
peculiarities of inter-cloud communication and bring benefits for both ISPs and their
business customers, including the cloud service providers.
Cloud applications are hosted either at enterprise-owned private clouds, provider-owned
public and hybrid clouds platforms. With the exception of the enterprise private cloud, the
role and importance of the ISP increases with the openness of the cloud. The provisioning
of efficient and reliable network connectivity is crucial for the success of the end-to-end
cloud service and greatly impacts the customers satisfaction and, thus, his willingness to
pay. ISPs can play a significant role in managed virtual clouds, a new service proposition
as evolution of Virtual Private Networks (VPN); this constitutes an additional service of their
portfolio targeting their existing customer base. Inter-connecting the Cloud IT (Information
Technology) infrastructure requires an active involvement of ISPs in large scale due to wellknown inefficiencies of the Internet: by-passing inefficient BGP routes, provisioned
resources to guarantee Quality-of-Service (QoS), or prioritizing traffic possibly also in interdomain layers. There is potential for ISPs here to combine network and Information Technology resources and act as brokers-integrators, exploiting existing business relationships
(e.g., VPN clients), bundling cloud brokering offers with network connectivity services so as
to provide complete solutions to Small and Medium Businesses with aggressive pricing.
The efficient management of the underlying traffic and the fair and efficient use of cloud
resources, considering also energy efficiency, QoE and stakeholders costs and interests is
the focus of the SmartenIT research agenda. SmartenIT has specified, simulated, evaluated and implemented a set of mechanisms that are tailored to the operators needs and
take into account their existing business contracts and interactions. Since for some intercloud data transfers, such as data replication, there may be multiple candidate destinations, SmartenIT Operator-focused mechanisms can select the optimal destination for such
transfers, considering the senders constraints and preferences, as well as the network and
cloud current load and performance metrics. This allows for joint cross-layer optimization
between the network and cloud layer. Invoking the functionality which is complementing the
network layer inter-domain traffic management done by the SmartenIT traffic management
mechanisms over the ISP transit link(s) is optional.
SmartenIT adopts design-for-tussle principles, respecting the scope of business decisions
and control of the network and cloud layer. In particular, the SmartenIT mechanism design
rationale is to perform win-win' traffic management in the network that respects and meets
the business expectations and constraints of the cloud layer as well. The clear separation
of decisions between the cloud and the network layer is intentional to ensure that
SmartenIT is compliant with current and emerging business models without providing unfair
power to some stakeholder, being in line with the design-for-tussle" and fair competition
best practices. Therefore, the clouds and data centers, rather than the ISP, take all the
cloud-related decisions such as selection of the destination and marking of traffic, while
there is built-in support for both federated and independent data centers and clouds.
Henceforth and in the remainder of this whitepaper we restrict attention to the SmartenIT
network-layer traffic management for which there is a working software prototype and
whose implementation along with a plethora of simulations already conducted prior to the
actual implementation have enabled to document the effectiveness and power of the
SmartenIT traffic management and to quantify the tangible benefits attained for operators.
SmartenIT White Paper OFS

3. SmartenIT for Operators:

Traffic Distribution over Time and Space
SmartenIT Traffic Management solutions have been designed to tackle the needs for InterCloud Communication (ICC) traffic management in an ISP-friendly way. The approach is to
apply novel ICC optimization mechanisms, comprising both network and cloud layer logic.
To this end, SmartenIT has specified, evaluated and implemented the DTM++ (Dynamic
Traffic Management) mechanism, which combines the features of two simpler mechanisms
also developed in the scope of the project, DTM and ICC. Proposed mechanisms use
information from network (transport layer) about topology and traffic samples and overwrite
typical traffic routing by cost-based steering decisions implemented thanks to specially
established tunnels.
The key motivation for the DTM++ traffic management rules is to recognize and then
decide, whether and to what extent it is beneficial to choose the path with lower cost
(distribution over space) or possibly delay a fraction of the traffic (distribution over time).
The former decision pertains to the DTM logic of the DTM++ mechanism, while the latter to
that of ICC traffic management mechanism
ICC is the incentive-compatible traffic management for the cost-efficient use of transit links,
relying on the fact that it is highly beneficial for the ISP to shape delay-tolerant, i.e., timeshiftable traffic such as inter-cloud bulk data transfers for back-ups, during peak epochs
thus reducing the transit link utilization under 95th percentile rule. As also depicted in
Figure 1, removing some time-shiftable portion of the traffic when it exceed a threshold
value and subsequently transferring it when the link usage is lower can enforce that the ISP
th
transit link 95 percentile will not exceed the desired threshold value set by the ISP. This
operation can be performed only as long as there is enough time-shiftable traffic to be
delayed. The delaying of the traffic is incentive-compatible, since the marking of the traffic
that is to be delayed is performed by the ISP business customers (clouds or data centers)
and are in turn compensated for the extra delays incurred by getting a cut of the ISPs transit cost savings.
We now proceed to present the DTM logic of DTM++, which comprises the main core of
the DTM++ traffic management mechanism: The DTM mechanism uses concept of transport costs and flow-based approach. The generic concept of the DTM mechanisms, designed for the optimization of transit costs for multi-homed ISP architecture is intended to
minimize the ISPs costs related to the amount of traffic in the network by optimally utilizing
the multiple transit links of a multi-homed ISP. The cost function defined by the ISP reflects
its requirements and may represent any monetary or non-monetary costs related to the
amount of transmitted traffic. The work on DTM specification, implementation and performance evaluation is focused on monetary costs for inter-domain traffic (e.g., paid by Tier-3
operator to Tier-2 operator).
The principle of DTM mechanism as described in Figure is explained using the following
simple use-case. Lets assume that network of ISP-A is multi-homed and the tariff schemas
used on different inter-domain links differs. In such a case appropriate shifting of a portion
of inter-domain traffic between them may result in lowering total costs. The overall traffic on
each inter-domain link is composed of two types of traffic: manageable and non-manageable (background). The former can be influenced by DTM. In this sample use-case the
manageable traffic is constituted by all transfers between remote Data Centers (DC).

SmartenIT White Paper OFS

Without using DTM all traffic exchanged between DC-A and DC-B would follow a default
BGP path and would traverse, e.g., link AB. When DTM is used, two tunnels (Generic
Route Encapsulation or Multi-Packet Label Switching tunnels) are established between
remote POI (Point-of-Interconnection) to which DCs are connected to. Tunnels pass different inter-domain links. ISP-A may select tunnel 1 or tunnel 2 for the inter-DC traffic and
as a result decide which inter-domain link is used. The decision on the link selection is
taken by the ISP dynamically (e.g., every minute) according to the current situation and
estimated cost of inter-domain traffic. DTM is designed to operate with those tariffs based
on total traffic volume or the 95th percentile.

Shift in space (DTM)

1
k
n
i
L

Shift in time (ICC)

TARGET traffic (cost)

to be achieved in the
current period

TARGET traffic (cost)

to be achieved in the
current period

5 min
samples

5 min
samples

n1

n+1 n+2 n+3 n+4

n1

n+1 n+2 n+3 n+4

TARGET traffic (cost)

to be achieved in the
current period

2
k
n
i
L

Non-manageable traffic
Manageable traffic
Portion of the traffic not affected
Portion of the traffic shifted in space or time

5 min
samples

n1

n+1 n+2 n+3 n+4

Figure 1: SmartenIT DTM++ Traffic Management

DTM consists of two main building blocks:
1. An algorithm to find the optimal solution to be achieved at the end of an accounting period. It makes a prediction for the next period and finds a better traffic distribution in
which the ISP's cost is minimized.
2. The compensation procedure determines how the traffic distribution should be influenced at a given moment to achieve the optimal solution at the end of the accounting
period, i.e., decides on selection of inter-domain link.

SmartenIT White Paper OFS

The DTM++ concept integrates DTM with ICC, thus, allowing for optimizations over both
the space (among multiple links DTM) and time (shifting in time ICC) axes. The
rationale and role of both mechanisms can be summarized as follows:

DTM is responsible for distributing manageable traffic among tunnels and what
follows selects the inter-domain link used. Given the amount of background and
manageable traffic in a previous billing period DTM finds such a traffic distribution
that the cost of the (inbound) traffic is optimized for the ISP of the receiving domain.
DTM does not delay the traffic or drops packets.
ICC additionally influences manageable traffic (delay tolerant component) on each
link independently, performing traffic shaping so that the resulting aggregate rate is
limited trying to keep it below a threshold, which is set for DTM++ to a reference
threshold related to that link.

Figure 1: SmartenIT DTM++ Traffic Management Mechanism Deployment

SmartenITs DTM++ traffic management mechanism defines a consistent strategy to recognize opportunities to manage traffic in a multi-domain, multi-operator cloud framework,
based on topology recognition and traffic measurements, which provide traffic adjustments
over different links to achieve a proper targeted at link cost utilization.

SmartenIT White Paper OFS

4. SmartenIT Solutions Matters to Operators

In order to highlight the efficiency of the SmartenIT Operator-focused Traffic Management
(TM) mechanisms, relevant and realistic (from a business perspective) Use Cases (UC)
have been defined, such as: Bulk data transfers for clouds, Resource allocation in cloud
federations, Video content transfer among clouds, IoT data transfers for clouds. Those UCs
involve Cloud Service Providers/Data Center Operators, possibly in the form of a federation
and ISPs offering the underlying traffic management. These UCs highlight aspects of the
inter-cloud communication paradigm, for which ICC, DTM and DTM++ have been designed. The SmartenIT UCs developed and studied within SmartenIT relies on the provision of proper incentives and the providing tangible benefits to all the stakeholders, so as to
achieve a win-win situation. Those benefits include the minimization of costs in terms of
inter-connection charges due to traffic generated by cloud services and applications for
ISPs, operating costs in terms of connectivity charges and energy costs for Cloud Service
Providers/Data Center Operators. The SmartenIT UCs and traffic management mechanisms are well in line with the existing and emerging business trends and business models
in the context of cloud and Internet services.
More in detail, the SmartenIT Operator-focused mechanisms and DTM++ in particular constitute powerful tools of service differentiation and QoE enhancement, reducing transit and
energy costs, as well as ISPs and clouds operating costs. They are applicable over the
Best Effort Internet and also in the context of Future Internet, where some form of quality of
service could be provisioned at inter-domain level. The SmartenIT Operator-focused mechanisms are applicable under various business models, thus contributing to the operators
agility and competitiveness: Such concrete business models are the TM Forum Repeatable
Digital Services Provisioning or the Federation and the ISP Managed Services business
models. The Repeatable Digital Services Provisioning model is a TM Forum attempt to
adapt the Amazon repeatable retail business model to cloud and Internet services based
on a middleware, the Ecosystem Enablement Platform, providing rich functionality to digital
service providers: DTM++ could constitute a smart traffic management module of this
platform. The Federation model is also a direct match, since SmartenIT mechanisms,
namely ICC and DTM++, have by design a cloud layer facilitating optimal destination
selection of bulk data transfers which explicitly considers the inter-cloud communication
needs and inherent features. Finally, the ISP Managed Services model is also an excellent
match since DTM++ could be an integral part of this paradigm as a whole and also for
particular managed services such as Virtual Private Cloud and Managed Storage and
Backup; in the latter cases SmartenIT intelligence could be integrated and used for the bulk
data transfers among the Virtual Private Cloud end points and in order to perform
scheduled managed back-ups. Also, SmartenIT relates to additional business models to
emerge due to the different forms of customer ownership, brokering, intermediation and
control, and form of the inter-operator business agreements.
Overall, the SmartenIT Operator-focused Traffic Management mechanisms are effective,
incentive-compatible, and in line with best practices and recent evolutions in business
models and networking and cloud services. Their design and features can also be
extended to near-future paradigms, including assured-quality inter-domain connectivity,
Software-Defined Networking/Network Function Virtualization and 5G. Thus, the SmartenIT
mechanisms constitute very good solutions leading to significant benefits also in practical
cases involving commercial networks and application/cloud service providers.

SmartenIT White Paper OFS

Abbreviations
BGP

Border Gateway Protocol

CDN

Content Delivery Network

CDNI

Content Delivery Networks Interconnection

CSP

Cloud Service Provider

DC

Data Center

DTM

Dynamic Traffic Management

DTM++

Dynamic Traffic Management with extensions

EFS

End-user Focused Scenario

ICC

Inter-Cloud Communication

ISP

Internet Service Provider

IT

Information Technology

OFS

Operator Focused Scenario

POI

Point-of-Interconnection

QoE

Quality-of-Experience

QoS

Quality-of-Service

SLA

Service Level Agreements

TM

Traffic Management

UC

Use Case

VPN

Virtual Private Network

SmartenIT White Paper OFS

The SmartenIT Consortium

University of Zrich, UZH, Switzerland
Athens University of Economics and Business - Research Center, AUEB-RC, Greece
Julius-Maximilians Universitt Wrzburg, UniWue, Germany
Technische Universitt Darmstadt, TUD, Germany
Akademia Gorniczo-Hutnicza im. Stanislawa Staszica w Krakowie, AGH, Poland
Intracom S.A. Telecom Solutions, ICOM, Greece
Alcatel Lucent Bell Labs, ALBLF, France
Instytut Chemii Bioorganiicznej Pan, PSNC, Poland
Interroute S.P.A, IRT, Italy
Telkom Deutschland GmbH, TDG, Germany

Copyright 2015, the Members of the SmartenIT Consortium

For more information on this document or the SmartenIT project, please contact:
Prof. Dr. Burkhard Stiller
Universitt Zrich, CSG@IFI
Binzmhlestrasse 14
CH8050 Zrich
Switzerland
Phone: +41 44 635 4331
Fax: +41 44 635 6809
E-mail: info@smartenit.eu

Mr. Matteo Biancani

Enterprise Sales Director, Cloud Infrastructure Sales
Interoute S.P.A.
Via Cornelia, 498, Roma RM
Italy
Phone: +39 06 615 2401
Fax: + +39 06 6152 40 99
E-mail: matteo.biancani@interoute.com

Further information on SmartenIT and its traffic management mechanisms as well as its
prototypes and results achieved can be found in the two sections Overview and Publications of the projects home page http://www.smartenit.eu/.

SmartenIT White Paper OFS

10