Sebastian Schmid ECM Solution Lab

13 February 2014

ECM Solution Lab

StoredIQ - Large Scale

2014 IBM Corporation

ECM Solution Lab

Team of 8 engineers integrates and deploys the complex defensible disposal software
stacks on a large-scale hardware lab
Care is taken to feed the systems with realistic test data at scale
Systems are sized to process petabytes of structured and unstructured data
Work very closely with the development teams of the individual products
Report defects and provide workarounds or fixes as well as prototypes for product
improvements
Publish scalability guides, best-practices documents as well as sizing recommendations

New StoredIQ development team was founded in Boeblingen to drive scalability and
integration

2014 IBM Corporation

StoredIQ Large Scale System Architecture

NAS Server
2x N6270
Metrocluster
5700 CIFS
Shares
5 Billion Files
1 Petabyte
on SAN
SAN Server
NAS storage
DS8800
3

vSphere Server
X3850 X5 (130 vCPU, 520GB)
Application Server
Gateway Server
32 Data Servers
150 mio objects

SAN Server
VM storage
V7000

SAN Server
VM storage
V7000
2014 IBM Corporation

StoredIQ Large Scale Use Cases

Information Inventory
Metadata only
No hash creation
Provide data structure insights to data expert user using Datamap view

Outdated and Trivial data removal

Identify data older than x years and delete automatically
Identify installer, temporary or log files and delete them

2014 IBM Corporation

Sizing Guidelines

Role

vCPU

Memory (GB)

Storage (GB)

Network

Data Server

16

190 + 330 2.000

1GbE

Gateway

100

1GbE

Application
Server

20

1GbE

The following are general design guidelines for the StoredIQ 7.x release. To precisely size an
environment and understanding of factors such as frequency of harvests and use scenarios
should be considered.
One data server per 30TB of file shares. Varies depending on number of volumes, objects
per volume and object types.
One gateway per 50 data servers.
One application server.

2014 IBM Corporation

Sizing Guidelines Storage Requirements

General sizing limits per dataserver are 150 million objects or 500 defined volumes
Taken 200KB this equals ~30TB of managed storage
Index storage requirement for this is ~330GB (11GB per TB)
Full-text or snippet (auto-classify) index requires ~100GB per TB

30 TB
metadata
8 TB
full-text
8 TB
auto-classify
6

330 GB
+
800 GB
+
800 GB
1.9 TB Total
2014 IBM Corporation

Sizing Guidelines Storage IOPS Requirements

Data server performance is significantly impacted by the IOPS available from the storage
sub-system
Each data server should be sized to consume 650 IOPS in average workload scenarios
Peaks under highest workload can reach up to 7.000 IOPS
Example: Were running the VMs on two V7000 with nearline-SAS disks that deliver 15.000
IOPS each

2014 IBM Corporation