BigData Express: Toward Predictable, Schedulable,

and High-performance Data Transfer

Wenji Wu wenj@fnal.gov
Internet2 Global Summit
May 8, 2018
 BigData Express
• Funded by DOE’s office of Advanced Scientific
Computing Research (ASCR)
• Collaborative effort by Fermilab and Oak Ridge
National Laboratory
– KISTI joined as a unfunded partner at 2017
– ESnet provides WAN service
• A three-year research project
– Start: Oct 1, 2015
– End: Sep 30, 2018
• http://bigdataexpress.fnal.gov
BigData Express Research Team
• FNAL • ORNL
– Wenji Wu (PI) – Nageswara Rao
– Qiming Lu – Gary Liu
– Liang Zhang
– Amy Jin • KISTI
– Sajith Sasidharan – Syed Asif Shah
– Phil DeMar – Seo-Young Noh
– Jin Kim

Note: KISTI and ESnet are unfunded project partners

 DOE Leadership
Big dataComputing facilities offer
enables scientific computing and
discoveries
storage resources needed to process and analyze science data
BigData Express Goal
The efficient movement of science data from their sources into
processing and storage facilities and ultimately on to user
A distributed middleware system that provides a
analysis is critical to the success of any such endeavor.
schedulable, predictable, and high-performance
Datadata transfer
transfer is now service for function
an essential the DOE’s large-scale
for science
science
discoveries, facilities
particularly andbig
within their
datacollaborators.
environments.
 Why BigData Express?
• Targeted at optimizing data transfers in high-speed networks
– Large-scale data movement of Big Data Science
– High-speed network environments (40/100GE+)
• Builds on Multicore-Aware Data Transfer Middleware (MDTM)
– mdtmFTP: a high-performance data transfer tool
• Pipelined I/O-centric design to streamline data transfer
• MDTM optimizes use of underlying multicore system
• Extremely efficient in transferring of Lots Of Small Files (LOSF)
– http://mdtm.fnal.gov
• Orchestrates system (DTN), storage, & network (SDN) resources
– To provide full end-to-end performance optimization
 BigData Express versus SENSE
• BigData Express is data transfer middleware
– Uses SENSE for WAN SDN services
• SENSE is a network service
– Provides higher-level applications with SDN-type services
– BigData Express is an application to SENSE
• BigData Express and SENSE are each stand-alone services in their
own right
– BigData Express works fine without SENSE
• WAN component is simply Best Effort
– SENSE is agnostic to higher-level applications using its services
 BigData Express Major Components
• BDE Web Portal
– Allow users to access BigData Express data
transfer services
• BDE Scheduler
– DTN as a service
– Co-scheduling of DTN, storage, and network
• BDE AmoebaNet
– Network as a service
• mdtmFTP
– a high-performance data transfer engine
– http://mdtm.fnal.gov
 BigData Express Major Components (cont.)
• DTN Agents
– Manage and configure DTNs
– Collect and report the DTN configuration and status

• Storage Agents
– Manage and configure storage systems

• Data Transfer Launching Agent

– Launch data transfer jobs
– Support different data transfer protocols
BigData Express -- Distributed

A Peer-to-Peer model
 BigData Express -- Flexible

Data Transfer Federation

…
DTNs

DTNs
• Flexible to set up data
Data Transfer Federation transfer federations
Data Transfer Federation

Networks • Providing inherent support

DTNs for incremental deployment
DTNs
…
…

DTNs
 BigData Express -- Scalable
BDE Web
Portal

SDN Agent
SDN
SDNAgent
Agent BDE Scheduler
(AmoebaNet) BDE Scheduler

Message
Queue

DTN Agent Data Transfer

DTN Agent Data Transfer
Launching Agent
Data Transfer
DTN Agent Launching Agent
Launching Agent

Storage Agent
Storage Agent
Storage Agent

• BigData Express scheduler manages site resources through agents

• Use RabbitMQ as message bus
 BigData Express -- Extensible
BDE Web
Portal

SDN Agent BDE Scheduler

SDN
SDNAgent
Agent BDE Scheduler
(AmoebaNet)
Message
Queue

DTN Agent Data Transfer Launching Agent

DTN Agent Data Transfer Launching Agent
DTN Agent Data Transfer Launching Agent

Storage Agent mdtmFTP GridFTP SRM … XrootD

Storage Agent
Storage Agent Plugin Plugin Plugin Plugin

• Extensible Plugin framework to support various data transfer protocols

• mdtmFTP, GridFTP, SRM, XrootD, …
BigData Express -- End-to-End Data Transfer Model

Site A - Smart E2E Site B - Smart E2E

Data Transfer Orchestrator Resource Data Transfer Orchestrator
negotiation & brokering
Web Portal Web Portal
• Application-aware network service
Scheduler Scheduler

Data Transfer
ne
Data Transfer o On-demand programming
Launching Agent go R ing Launching Agent
tia eso rce ker
tio ur u o
n & ce e so & br
R n
br
ok
e rin
g go
tia
tio
• Fast-provisioning of end-to-end
ne
network paths with guaranteed QoS
mdtmFTP mdtmFTP

DTN Agent AmoebaNet SENSE AmoebaNet DTN Agent

Storage Agent Storage Agent

• Distributed resource negotiation &

LAN WAN LAN
brokering
Storage Edge DTN Edge DTN Storage

A End-to-End Data Transfer Loop with Guaranteed QoS

BigData Express -- Three Types of Data Transfer

• Real-time data transfer

• Deadline-bound data transfer

• Best-effort data transfer

 BigData Express vs. Globus Online
Features BigData Express Globus Online
• Distributed service
Architecture • Centralized service
• Flexible to set up data transfer federations
• Extensible plugin framework to support multiple
protocols:
Supported Protocols • GridFTP
o mdtmFTP
o GridFTP, XrootD, SRM (coming soon)
• Yes, Network as a service
SDN Support • Fast-provisioning end-to-end network paths with • No
guaranteed QoS
• Real-time data transfer
Supported Data Transfers • Deadline-bound data transfer • Best-effort data transfer
• Best-effort data transfer

• Checksum • Checksum
Error Handling
• Retransmit • Retransmit
 BigData Express SC’17 DEMO
• BigData Express: a schedulable, predictable, and high-performance
data transfer service
– QoS-guaranteed data transfer
– DTN as a service
– Network as a service
– Distributed resource brokering/matching

A DOE/SC/ASCR-sponsored research project

Software is available at: http://bigdataexpress.fnal.gov
A Cross-Pacific SDN Testbed
HP Z91000
AmoebaNet

KREONET 40GE BDE BDE

Web Protal Scheduler
STP
192.2.2.7
DTN2 134.75.125.77
STP 10GE

10GE 134.75.125.78
StarLight
192.2.2.9 10GE 192.2.2.8
STP DTN3

DTN4 134.75.125.76

134.75.125.80
ESNET OSCARS KISTI SW
KISTI, South Korea

600W Chicago
To Internet

FNAL Border router

STP
bde-hp1.fnal.gov
40GE Pica8 P5101 40GE Pica8 P3930
ESnet NSI
AmoebaNet
Circuit Service
4/1 4/3 4/5 4/7 47 49 65 51
73

yosemite.fnal.gov 4/2 74 4/4 4/6 4/8 48 50 66 52

BDE BDE 40GE 40GE 40GE 1GE 40GE

Web Protal Scheduler 192.2.2.1 192.2.2.2 192.2.2.3

BDE1 BDE2 BDE3 BDE-hp5 BDE4

Lustre file system

wwwld1 Infiniband Switch wwwld5

(mgt) (oss)

wwwld2 wwwld3 wwwld4 wwwld6

(oss) (oss) (oss) (oss) FNAL, US
 BigData Express Deployment
• Completed deployment: KISTI, UMD, StarLight, FNAL
• Ongoing deployment: KSTAR, ESnet
• Work with StarLight to deploy BDE at XRPs
– Pacific Research Platform (PRP)
– National Research Platform (NRP)
– Global Research Platform (GRP)
– The European Research Platform (ERP)
– Asia Research Platform (ARP)
• Collaborate with SENSE for BDE+SENSE deployment
• Work with US CMS to deploy BDE at US CMS sites
 Support Science
• Fusion community
– Work with KSTAR, KISTI, PPPL, and ORNL to transfer/stream data from
KSTAR to US research institutions

• XRPs (PRP, NRP, GRP, ERP, ARP)

– Work with StarLight to deploy BDE at XRPs to support various science

• HEP community
– Work with US CMS to deploy BDE at US CMS sites
• PI has been invited to give a BDE demo for US CMS
• Tentatively scheduled for the last week of May, 2018
More information about BigData Express

http://bigdataexpress.fnal.gov

PI: Wenji Wu, Fermilab

wenji@fnal.gov