Advanced Data Sharing Technologies

EDUCATION
Advanced Data Sharing

Technologies Part 1
version 9
Philippe Nicolas, Symantec

Jonathan Goldick, ONStor
Abstract
EDUCATION
Advanced Data Sharing Technologies Part 1

The What, Why and How of Data Sharing technologies plus block and filebased approaches
for IT Director and Managers, IT/Storage/System Eng., Admins, Architects
and Trainers
How to deliver more performance and data accessibility with little to no
additional cost? How to take advantage of existing storage infrastructure to
provide more value to end-users and the global enterprise? A clear industry
direction indicates that Data Sharing architectures and technology can be a
good way to achieve these objectives.
The first session offers a definition of Data Sharing and a discussion of its
benefits with examples linked to the SNIA Shared Storage Model. We cover the
main data sharing approaches and describe how they can improve
performance, data accessibility, and manageability. This includes Scale-in and
Scale-out methods based on block, file and application technologies such as:
Cluster Volume Managers, SAN File Systems, Cluster File Systems, Parallel
NFS (pNFS), Object-based Storage Devices (OSD) and Global/Parallel File
System.
SNIA Advanced Data Sharing Technologies Tutorial (v9) - Storage Networking World USA - April 3-6th, 2006
2005-2006 Storage Networking Industry Association. All Rights Reserved.
SNIA Legal Notice

EDUCATION
The material contained in this tutorial is copyrighted by

the SNIA
Member companies and individuals may use this material
in presentations and literature under the following
conditions:
Any slide or slides used must be reproduced without modification
The SNIA must be acknowledged as source of any material used
in the body of any document containing material from these
presentations
This presentation is a project of the SNIA Education

Committee
Agenda
EDUCATION
What is Data Sharing ?

Definition
Why Data Sharing ?

End User Benefits
How is Data shared ?

Block and file-based approaches
EDUCATION

Why Data Sharing ?

Definition
EDUCATION
Shared access to
same data (value & location) by multiple systems
Read/write: changes to data become visible to all servers
Read-only access via mechanisms that support shared read/write
access
Examples
Read/write access to a shared file is data sharing
So is read-only access to a shared file
Clone/Snapshots (read-only or read/write) are not data sharing

Changes do not affect original data
Caching is data sharing when changes propagate

Changes to cached data must become visible to all
Other data changes must become visible via cache
Replication/CDP* is not data sharing because location changes

Potential divergence of data
* CDP: Continuous Data Protection

Why Data Sharing ?

End User Benefits
EDUCATION
Better performance and scalability

Larger server can be expen$$ive
Sharing: apply more servers to same problem

Scales well for some applications, poorly for others
Can avoid replication or cloning
Concurrency and Content access distribution

Use same data for more than one application
Administration
Consolidated shared resource has lower TCO
Data Sharing increases the benefits of Storage/Data
Consolidation
Storage Consolidation
Scale-up by Scale-In
EDUCATION
File
Server
Data Network - LAN
NFS/CIFS
Server
Scale-In
Storage
Network
Shared
Disks
Performance Improvement
Scale-up by Scale-Out
EDUCATION
Application
Server
Application
DB Engine
Cluster Software
Shared Storage Software
Scale-out
Storage
Network
Shared Disks
Why Data Sharing ?

How to apply Data Sharing to do useful things
EDUCATION
High Availability Clusters (local & geographic)

Scaling applications
Web Servers - Read mostly/load balanced
Databases/OLTP/DW - Mostly use direct I/O
Parallel applications and fast failover

Systems and Applications Consolidation/Migration
Off-host processing
Based on shared file system
Can also use by Point-in-Time copy techniques (not related to
our data sharing definition)
address both Performance and Availability with no

administration degradation and overhead
10
Some technologies
and products
IBM AFS
WebNFS
ISO9660
EDUCATION
Apple Xsan
Cisco FileEngine
Coda
VERITAS CFS
RFS
DiskSites FilePort
Distributed,
Samba FineGround
PolyServe Matrix Server
Oracle OPS/RAC
ClusterLUSTRE
or
WebFS
EMC
HighRoad
IBM Storage Tank
IBRIX
FusionFS
SAN
File
System
SGI CXFS
CIFS
DB2
WAFS
ONStor STOR-FS
HP TruCluster/CFS
ADIC StorNext FS
Parallel &Sybase MPP

SUN SAM-FS
Partitioned
Sanbolic Melio FS
IBM SANergy
Applications
SMB
Tacit Networks Ishared

OSD
PVFS
Redhat GFS
PPFS
Sun QFS
Volume
WebNFS
Isilon IQ OneFS
OpenAFS
NFS
Informix XPS
DFS
Nuview StorageX
pNFS
11
EDUCATION
How is Data Shared ?
How is data shared ?

Approaches and methods
EDUCATION
Several levels are possible

Share at the volume level (block based)
Share at the file or file system level (file, block or object* based)
Share at the database or application level (custom)
In all cases, all these methods could occur

among like or dissimilar systems (OS),
concurrently or serially,
directly at the storage or in the network
* For OSD
13

Approaches and methods
EDUCATION
Traditional/Historical
Block level: Volume Management
File/File System (FS) level: Local FS (serial data sharing) and
distributed methods with NAS, Samba, AFP (AppleShare), DFS,
AFS/OpenAFS, RFS, Coda
App./DB level: custom built-in methods (RDBMS, Email
systems)
Check
Check out
out
SNIA
SNIA Tutorial:
Tutorial:
Advanced/Recent - File/FS level

Distributed: WAFS approach (NAS extension)
and Network File Management/Virtualization
(NFM), Global FS, SANFS and Cluster FS
Storage
Storage
Virtualization
Virtualization
Check
Check out
out
SNIA
SNIA Tutorial:
Tutorial:
NAS
NAS &
& iSCSI
iSCSI
14

The SNIA Shared Storage Model
EDUCATION
Application
Application
level
Database
(dbms)
Cluster FS
WAFS
NAS
File system
(FS)
Shared
LVM
SAN FS
Host
Network
Block
aggregation
Device
Storage
Storage devices
devices (disks,
(disks, )
)
Block layer
Discovery, monitoring
monitoring
Discovery,
Resource mgmt,
mgmt, configuration
configuration
Resource
Security, billing
billing
Security,
Redundancy mgmt
mgmt (backup,
(backup, )
)
Redundancy
High availability
availability (fail-over,
(fail-over, )
)
High
Capacity planning
planning
Capacity
File/record layer
Services
Storage domain
GFS
15

Volume level
EDUCATION
LVM
LVM
Data
Path
Data Layout
and
Organization
Examples
EMC PowerPath Volume Manager (PPVM)

HP Shared Logical Volume Manager (SLVM)
IBM Logical Volume Manager (LVM)
MACROIMPACT SANique Cluster Volume Manager (CVM)
REDHAT Logical Volume Manager (LVM)
SANBOLIC LaScala
VERITAS* [Cluster] Volume Manager (CVM/VxVM)
* Merged with Symantec, July 2005
16

Volume level
EDUCATION
Volume Managers allow data to be shared at a low level

(block) without usually a built-in locking mechanism
Higher level applications control concurrent accesses to the data as
needed
Can combine or divide physical resources (e.g., concatenation,
mirroring, striping) and share the result
Volume Managers and the VTOC* problem
Every OS has its own VTOC format
Every VM has its own Volume Header and Table Definition
Same VM everywhere and you can share raw volume or same FS
Byte orders between processor

Big Endian: Sparc, PA-Risc, Power Little Endian: Intel)
Block size on the device and block boundary

could cause troubles
Concurrent or Serial access
* Volume Table Of Contents
Check
Check out
out
SNIA
SNIA Tutorial:
Tutorial:
Storage
Storage
Virtualization
Virtualization
17

Volume level
EDUCATION
In-Band Virtualization
Out-of-Band Virtualization
Application
Application
Servers
Servers
Volume
allocation
Intelligent
switch and/or
Appliance
Storage
Storage
Network
Network
Appliance
Shared
Disks
Volume
creation
Shared
Disks
18

Volume level
EDUCATION
Application
HPC App.
HPC App.
HPC App.
Servers
HPC App.
Shared Volume Manager - Storage Software
Storage
Network
Example:
HPC* Application
- How ?
Shared Disks
Own data format on disk

Own lock mechanism
- Benefits
Increased throughput
More effective use of servers
* High Performance Computing
19

File/File System level
EDUCATION
Share at File or File System (FS) level

Multiple approaches & levels of maturity
Block-based
Local (physical) Disk File System* for serial data sharing
Disk based Cluster File System
SAN File System or SAN File Sharing System
File-based or Distributed File System

NFS/CIFS, WAFS and NFM**
Global, Parallel and Distributed FS
* like UFS, HFS, XFS, JFS, VxFS, NTFS, ext2/3
** Network File Management (also Network File Virtualization)
20
File/record layer
EDUCATION
Cluster
FS
Host
Host. with LVM
Host. with LVM

and software RAID
File/record
layer
Application
Application
level
Dist. FS
LAN
SAN FS
NAS
headCluster FS
Shared
LVM
SN
Block
layer
Host
1. Direct-attach
2. SN-attach
3. NAS head
4. NAS server
NAS
server
Disk array
Host block-aggregation
Network block-aggregation
Device block-aggregation
21

File/File System level Local Disk File System
EDUCATION
Serial File System sharing on same or dissimilar OS via common

Volume Manager & Physical File System
Good for sequential (not concurrent) data processing and data
migration
If OS is different
Same Volume Manager avoids VTOC incompatibilities
Byte order differences may require meta-data conversion
Intel is Little-endian, most others are Big Endian
Examples
Homogeneous OS (common case)
Most file systems (and volume managers) support this
UFS, HFS, XFS, JFS, VxFS, ext2/3 SDS/SVM, LVM, XVM,

VxVM Sanbolic Kayo, DNF Dynamic Share
Heterogeneous OS (need common volume manager)
VERITAS* Storage Foundation with Portable Data Container feature
* Merged with Symantec, July 2005

22

File/File System level Local Disk File System
EDUCATION
Example:
DW* Application
OS #0
OS #1
OS #2
OS #3
- How ?
Storage
Network
Import Disk Group
- Benefits
Start Volume
Mount File System
Deport Disk Group

Stop Volume
Umount File System
OS #0 server stores data

OS #1 server starts
batches
OS #2 server loads data
into the DW
OS #3 server backups
data
Shared
Disks
No data multiplication
Cost effective for Storage
More effective use of
servers
No time wasted in
copying data between
servers
* Data Warehouse
23

File/File System level Cluster File System
EDUCATION
Cluster File System (also called Shared Data Cluster)
A Cluster FS allow a FS and files to be shared

All nodes understand Physical (on-disk) FS structure
The File System is mounted by all the nodes
Single FS Image
Same data view from all nodes
Examples
HP CFS (TruCluster)
HP/Cal. Soft. Monster FS
IBM GPFS
MACROIMPACT SANique CFS
POLYSERVE Matrix Server
REDHAT GFS 1
SANBOLIC MelioFS
VERITAS2 CFS
1 Sistina acquired by RedHat

2 Merged with Symantec, July 2005
24

File/File System level Cluster FS
Web
Server
First
EDUCATION
HeartBeat
Web
Server
Lock Management
Host
Web
Server
Second
Host
Cluster File System

Cluster Volume Manager
Cluster
Storage
Network
Example:
Web Servers Farm
How ?
Shared VM/FS
Load Balancer in front
Shared Disks
Benefits
More effective use of servers
Failure is transparent
SSI/SFSI, High SLAs
Optional Layer
25

File/File System level Cluster File System
EDUCATION
Asymmetric & Symmetric Implementation

Asymmetric uses master node for logging and locking
Lock Mechanism
Distributed or Global Lock Management (DLM/GLM)
Different implementation strategies
Granularity varies: file, record, byte
Cache Coherence Single File System Image

Every modification is seen by all nodes as soon as a
modification in the data sharing domain occurs
Usage Consideration: Concurrent vs Serial data access

Concurrent: multiple systems access the data simultaneously
Serial: one system at a time uses and access the data
26

Advanced methods File/FS Approaches
EDUCATION
SAN File System

1 node (Master) or a set of masters
Understand, manage and use metadata on disk
Use of file system even if portions of it are inaccessible
block addresses distributed to nodes (clients) on request
Other nodes (clients)

connection to SAN storage
Avoid overhead due to Metadata management
access to data directly using blocks addresses sent by Master(s)
Designed to support hundreds or thousands of nodes

Mixed role between direct data access with
host based thin software and NAS access
27

File/File System level SAN File System
EDUCATION
Flexibility of network FS at SAN speed

Long-term goal for the industry development for Capacity and
Performance scaling
Scaling hundreds of PetaBytes of capacity and tens of GigaBytes/sec
More recent File Server Generation

Examples
APPLE Xsan
ADIC StorNext FS
DataPlow SAN FS
& Nasan FS
EMC Celerra HighRoad,
MPFS/MPFSi
HP DirectNFS xNFS (Cal. Soft.)

Transoft Fibrenet
IBM TotalStorage SAN FS,
SANergy
IBRIX FusionFS
SGI CXFS
SUN QFS
28

File/File System level SAN FS
EDUCATION
App.
File
Server
Data Network - LAN

Metadata
Server
App.
App.
Client sw
Client sw
File Request
NFS/CIFS
Server
Block list
Data and
Control Access
Storage
Network
Data Access
Example:
Shared
Disks
Multimedia Application
How ?
Benefits
1 big server with NFS/CIFS layer

Server and Client SAN FS layer
Hundreds of clients
Consolidate storage, very scalable
More effective use of resources
29

File/File System level SAN FS
EDUCATION
How it works ?
Asymmetric or Client/Server model
Server controls client access, resolves conflicts
Thin client software layer handles SAN device and server
interaction
Lock Mechanism
Provided by the server at a central location

Various granularity: file, record, byte
Some implementations use SMB or NFS semantics
The server needs to be protected cause it represents a SPOF
Cache Coherency
Some implementations deliver cache coherency with traditional
validate/invalidate mechanism, others dont offer cache at all
30

File/File System level OSD
EDUCATION
OSD = Object-based Storage Device

An object comprises
Application data (e.g., file, record)
Device-managed metadata (e.g., block allocation)
User-accessible attributes (e.g., access times)
Objects have file-like methods for access

Open, close, read, write, get/set attributes
Commands are authorized
Industry offerings
Lustre (www.lustre.org)
Bull, CFS, Cray, HP, Scali, SUN Lustre based
Panasas
ID x123
Blocks:3,42
Length:512
SNIA OSD Working Group

OSD as a SCSI command set
www.snia.org/tech_activities/workgroups/osd
Check
Check out
out
SNIA
SNIA Tutorial:
Tutorial:
Object-based
Object-based
Storage
Storage
Device
Device
31

EDUCATION
CPU
CPU
Applications
Applications
System Call Interface
System Call Interface
File System
User Component
File System
User Component
File System
Storage Component
Object Interface
Block Interface
File System
Storage Component
Block I/O Manager
Storage Device
Block I/O Manager
Storage Device
32

EDUCATION
Clients
A cc
es s
R eq
uest
SECRET
KEY
A
DAT
EthSAN
switch
NT
E
EM
G
NA
A
M
Managers
Intelligent Device
Space Management
Backup/Recovery
QoS via attributes
Security
SECRET
KEY
SECRET
KEY
Validate
Validate Capability
Capability
Object-based Storage Devices

33

File/File System level Parallel NFS (pNFS)
Now
Client
EDUCATION
pNFS Goal
Client
Host Net
Host Net
NFS
v4
NFS
v4
Data
Storage Net
Storage Net
NFS Server
NFS Server
Data
Storage Servers
Storage Servers
Allow NFSv4, data to bypass NFS server

No application changes, similar management model
pNFS extensions to NFSv4 communicate data location to clients

Clients access data via Fibre Channel, iSCSI, OSD, or even NFS
IETF standardization in progress
34

File/File System level SANFS vs CFS
Characteristics
& Features
EDUCATION
SAN FS
Cluster FS
Tolerance of Distance
(between server and
clients)
Important
Limited
# of nodes
Hundreds
Dozens
Yes
No
Yes, usually
No cluster assigns
functions to nodes
Metadata server only

(clients may understand
if same OS)
All nodes (Cluster FS

currently requires
same OS)
Heterogeneous OS
Dedicated Meta-Data
Server(s) Required
Physical filesystem
layout knowledge
35

File/File System level Parallel File System
EDUCATION
Concept/Idea: Data is striped between servers (I/O

nodes)
Features
Cluster-wide consistent name space
User control for file striping across I/O nodes
Asymmetric (master + slave servers + clients)

GoogleFS, PVFS*, IBRIX, Panasas (osd)
Symmetric (peer servers + clients)

TerraScale, Isilon, Exanet, NetApp (Spinnaker Networks)
* Parallel Virtual File System

36

File/File System level Parallel File System
Asymmetric
EDUCATION
Symmetric
#0
#1
#2
37
EDUCATION
Conclusion
Conclusion
Various ways to Share Data
EDUCATION
Many products and philosophy in the industry
OS, disk (local) file system

Methods to protect data (locking)
Cache coherency mechanisms and semantics
Caused by varied objectives and applications
There is no single, simple, efficient data format

available on all operating systems !! (sorry)
Server and client software needed for Data Sharing
Remember VTOC and Byte ordering potential issue
39
Conclusion
to leverage the infrastructure
EDUCATION
There are a number of things to consider when

choosing a file system or server
Will the application work as desired?

Will it perform and scale?
Does it have the required data management services?
Is it secure enough?
Is it easy to use and manage?
There is no single solution that is superior in all

cases BUT these approaches deliver real
applications and business benefits
Real measured ROI
Performance, Availability and Manageability
40
Q&A / Feedback
EDUCATION
Please send any questions or comments on this

presentation to SNIA: track-storage@snia.org
Many thanks to the following individuals
for their contributions to this tutorial.
SNIA Education Committee
Symantec (Philippe Nicolas)

ONStor
(Jonathan Goldick)
EMC
(David Black)
CA, Cisco, CNT, Crossroads, EvaluatorGroup, HDS,
HGAI, Inrange, Knowledge Transfer, Microsoft,
NationWide, QLogic, Sandia National Laboratories,
Seagate, Solution Technology, Sun Microsystems &
VERITAS Software
41
EDUCATION
Advanced Data Sharing

Technologies
version 9
Philippe Nicolas, Symantec

Jonathan Goldick, ONStor

Advanced Data Sharing Technologies

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Advanced Data Sharing Technologies

Transféré par

Droits d'auteur :

Formats disponibles

EDUCATION

Advanced Data Sharing

Philippe Nicolas, Symantec

Advanced Data Sharing Technologies Part 1

SNIA Legal Notice

The material contained in this tutorial is copyrighted by

This presentation is a project of the SNIA Education

What is Data Sharing ?

Why Data Sharing ?

How is Data shared ?

What is Data Sharing ?

What is Data Sharing ?

Clone/Snapshots (read-only or read/write) are not data sharing

Caching is data sharing when changes propagate

Replication/CDP* is not data sharing because location changes

* CDP: Continuous Data Protection

Why Data Sharing ?

Better performance and scalability

Sharing: apply more servers to same problem

Concurrency and Content access distribution

Why Data Sharing ?

High Availability Clusters (local & geographic)

Parallel applications and fast failover

address both Performance and Availability with no

Parallel &Sybase MPP

Tacit Networks Ishared

How is Data Shared ?

How is data shared ?

Several levels are possible

In all cases, all these methods could occur

How is data shared ?

Advanced/Recent - File/FS level

How is data shared ?

How is data shared ?

EMC PowerPath Volume Manager (PPVM)

How is data shared ?

Volume Managers allow data to be shared at a low level

Byte orders between processor

Block size on the device and block boundary

How is data shared ?

How is data shared ?

Shared Volume Manager - Storage Software

Own data format on disk

How is data shared ?

Share at File or File System (FS) level

File-based or Distributed File System

Host. with LVM

How is data shared ?

Serial File System sharing on same or dissimilar OS via common

UFS, HFS, XFS, JFS, VxFS, ext2/3 SDS/SVM, LVM, XVM,

* Merged with Symantec, July 2005

How is data shared ?

Import Disk Group

Deport Disk Group

OS #0 server stores data

How is data shared ?

Cluster File System (also called Shared Data Cluster)

A Cluster FS allow a FS and files to be shared

1 Sistina acquired by RedHat

How is data shared ?

Cluster File System

Web Servers Farm

How is data shared ?

Asymmetric & Symmetric Implementation