iSCSI and OneFS

A Best Practices Guide for Isilon IQ Storage By Michael Kade Global Systems Engineer

An Isilon Systems Best Practices Guide

ISILON SYSTEMS

February 2010

ISILON SYSTEMS

Table of Contents
1. 2. 3. Introduction .......................................................................................................................................... 3 Isilon Best Practices for various vendor iSCSI initiators ................................................................ 3 Isilon Best Practices for iSCSI with OneFS ...................................................................................... 3 Hardware iSCSI Adapter ...................................................................................................................... 3 Snapshot Licensing .............................................................................................................................. 3 Optimal Block Size ............................................................................................................................... 4 2x Protection Recommended ............................................................................................................... 4 Dedicated Network ............................................................................................................................... 4 Jumbo Frames ..................................................................................................................................... 4 Maximum Number of Targets and Maximum Number of LUNs Per Target ......................................... 4 Write Caching ....................................................................................................................................... 5 Client Logical Volume Manager ........................................................................................................... 6 LUNs are Directly Associated with Targets .......................................................................................... 7 Dynamically Allocated IP Pools result in LUNs not being visible ......................................................... 7 Missing Logical Unit Number ............................................................................................................... 7 Concurrency vs. Streaming Access Pattern ........................................................................................ 7 OneFS Prefetch ................................................................................................................................... 7 LUN Provisioning ................................................................................................................................. 8 LUN Resizing ....................................................................................................................................... 8 LUN Backup ......................................................................................................................................... 8 iSCSI with SyncIQ ............................................................................................................................. 8 4. 5. Conclusion ........................................................................................................................................... 8 References............................................................................................................................................ 9

ISILON SYSTEMS

1. Introduction
The implementation of the iSCSI protocol within Isilon IQ brings unified storage capability to Isilon. Leveraging Isilon industry leading scale-out technology, Isilon iSCSI solutions allows applications like Microsoft Exchange, Microsoft SQL, and block-level file systems like VMware VMFS, to benefit from the Isilon distributed architecture. This guide reviews best practices directly related to the storage of Logical Units (LUNs) on the Isilon OneFS filesystem. This guide is intended to be complementary to the Isilon Installed Product Help, Isilon Online Knowledge Center, and other Isilon iSCSI Best Practices guides. Please see the References section of this guide for a more complete list of documentation covering iSCSI. The Isilon iSCSI implementation features the following capabilities: Implementation of RFCs 3720 and 5048 Support for Thin-Provisioned or Fully-Allocated LUNs Support for One-Way CHAP Authentication Support for dynamically growing LUNs Support for creating LUN clones, snapshots or shadow copies (writable snapshots) Support for SCSI reserve/release LUN arbitration control Client Support for iSNS (iSCSI Name Service) Support for VSS (Volume Shadow Service) when utilizing the Microsoft iSCSI Initiator Isilon iSCSI LUNs are constructed as files that reside within OneFS. Each iSCSI LUN is represented as eight extents within a particular directory. By default, directories lie underneath the target that the LUN is allocated to, although LUNs may be moved or placed anywhere within the directory hierarchy for convenience (e.g. to enforce a single set of SmartQuotas or aid in SyncIQ).

2. Isilon Best Practices for various vendor iSCSI initiators

Different best practices guides are available that include detailed guidance for setting up and configuring iSCSI on the various vendor initiators. This is in addition to Isilon Installed Product Help and Isilon Online Knowledge Center.

3. Isilon Best Practices for iSCSI with OneFS

The following best practices will ensure the best possible performance when using iSCSI with the OneFS filesystem.

Hardware iSCSI Adapter

Hardware iSCSI Adaptors have not been tested and are not currently supported for use with Isilon storage products.

Snapshot Licensing
Cloning allows the user to create a new LUN based upon an exact point-in-time copy of an existing LUN. There are three types of clones available; normal, snapshot and shadow. Each of these types of clones requires that SnapshotIQ be licensed. Without SnapshotIQ licensed, iSCSI will continue to work, but it will be impossible to clone a Logical Unit.
ISILON SYSTEMS 3

Optimal Block Size

LUNs reside as files within the OneFS filesystem. File system objects (directories, files, etc) within OneFS are stored as groups of 8K blocks. The data contained within a LUN is solely under the control of the initiator within the client and, as such, the block size within a LUN as written by the initiating client may be different than the 8K-block size of a OneFS file. If the client block size within the LUN is not a multiple of the 8K-block size of OneFS, then multiple I/Os could occur and this would result in suboptimal performance. When possible, format the data contained within the LUN so that the block size is a multiple of the 8K-block size of OneFS. More detailed information is contained within the Best Practices Guides for the Microsoft, Linux and VMWare initiators.

2x Protection Recommended
In most cases, 2x (mirrored) protection will result in better performance since parity reads arent required during the write process. However, if your workflow is primarily reads, then 2x protection wont provide as much benefit. Additionally if space is a primary concern then you may be willing to sacrifice some write performance to free up space. Setting 2X protection on a LUN may fail if the global setting for FlexProtect is different than 2X (mirrored). You can change the individual protection for a Logical Unit once you have changed the FlexProtect setting to advanced. In order to set the FlexProtect setting to advanced, login to one node of the cluster and type isi flexprotect advanced.

Dedicated Network
iSCSI is a block based protocol and, as such, can be very network intensive. Best performance will be achieved if iSCSI is the only protocol accessing the assigned gigabit or 10-gigabit Ethernet port.

Jumbo Frames
In order to achieve maximum performance, it was previously recommended that the block size of the client system be right-sized to match the block size of OneFS. Since each I/O would then be one or more multiples of 8K, it is recommended that you enable Jumbo Frames of 9K on the Ethernet ports dedicated to iSCSI on both the Isilon node(s) and the client system. This will result in 6 times less I/O and consequently better performance than using the default Ethernet frame size of 1500.

Maximum Number of Targets and Maximum Number of LUNs Per Target

Isilon does not enforce any limit on the number of targets. The number of LUs per target is restricted to 256 (numbered 0-255). Users creating massive numbers of targets or LUNs should be aware of initiator restrictions, and use of target masking is advised to prevent an initiator from being overwhelmed with SendTargets responses.

ISILON SYSTEMS

Write Caching
Write caching on Isilon iSCSI LUNs is turned off by default. Turning Write Caching on can result in write performance improvements, but at a risk of corruption if a node loses power or crashes while uncommitted data is in the write cache. Some software iSCSI initiators protect this data-in-flight, but unless you know whether your client initiator has this commit technology, it is recommended that this setting stay in the default off setting. One exception would be to turn write caching on during the creation of a thickly allocated LUN, and turning write caching off once the LUN creation has completed. If it is discovered that your client initiator does not protect data in-flight, then it may still be possible to use Write Caching by implementing LUN protection using techniques available to Logical Volume Managers available with some client operating systems. One example is given in the next section entitled Client Logical Volume Manager.

ISILON SYSTEMS

Client Logical Volume Manager

Many client operating systems support the concept of a Logical Volume Manager (LVM); whereby multiple smaller physical disks or LUNs can be grouped together to form a single larger logical disk. This is often done for performance, additional protection or both. If you wish to have better performance, it is possible with many LVMs to build RAID-0 (Striped) logical volumes by combining multiple LUNs. The best performance for such a RAID-0 logical volume on an Isilon clustered filesystem would be to have each physical LUN accessed by a separate node in the cluster (E.G. If you have five LUNs striped into a RAID-0 Logical Volume, then access each individual LUN from a separate node of a five node or larger cluster).

Large Server

Target: RAID0 LUN: 0

Target: MIRROR0 LUN: 1

Z:\ Logical Volume Manager Mirrored Set

Target: RAID0 LUN: 1 Target: MIRROR0 LUN: 2

Target: RAID0 LUN: 2

Target: MIRROR0 LUN: 3

Target: RAID0 LUN: 3

Target: MIRROR0 LUN: 4

RAID 0 Stripe Target: RAID0 LUN: 0 LUN: 1 LUN: 2 LUN: 3 LUN: 4

RAID 0 Stripe Target: MIRROR0 LUN: 0 LUN: 1 LUN: 2 LUN: 3 LUN: 4

Target: RAID0 LUN: 4

Target: MIRROR0 LUN: 0

RAID-0 striped logical volumes will give you higher performance, but they offer no additional protection. In some cases, they could offer less protection since the failure of any one LUN will cause data loss. To prevent this, it is possible to build two RAID-0 logical volumes and then mirror them (RAID-1). This is often called RAID-10. This will provide for the temporary removal of any LUN (say through the reboot of a node); thereby keeping the entire logical

ISILON SYSTEMS

volume online and available to your users. When the node reboots or its functions are transferred to another node in the cluster, then the client LVM will detect this and rebuild the RAID-1 mirror automatically. The previous diagram illustrates how such a RAID-10 logical volume could be built. Note that the best protection occurs when the LUNs of the two RAID-0 stripes are accessed through offset nodes of the cluster. This protects against node failures taking down the same LUNs in the RAID-1 (mirrored) set.

LUNs are Directly Associated with Targets

LUNs are directly associated with targets. If you delete a target, all LUNs assigned to that target will be permanently destroyed as well. If you need to delete a target and preserve the LUNs within that target, first move each LUN to a new target, then delete the target only when there are no LUNs associated with it. Moving LUNs can be accomplished via the command line (scripted) or via the Isilon web administration interface.

Dynamically Allocated IP Pools result in LUNs not being visible

If you create a LUN within Isilon, and try to connect to that LUN from an iSCSI Initiator, in which the IP address belongs to an Isilon SmartConnect IP Pool where the IP Allocation method is set to Dynamic, the iSCSI client/initiator will not be able to see any LUNs. You must set the IP Allocation method to Static in order to see the LUNs.

Missing Logical Unit Number

If you create a LUN within Isilon and set the LUN number allocation to manual, it is possible that the client initiator will not be able to detect any LUNs. This generally occurs when there is no LUN with a number of zero. Some initiators require a LUN with a number of zero before they will scan for any other LUNs with higher numbers. If a LUN with the number of zero does not exist, then scanning for additional LUNs will not occur and it will appear as if no LUNs exist.

Concurrency vs. Streaming Access Pattern

Setting the Access Pattern on a LUN can result in different performance only within a very narrow workflow. Very random writes within a large LUN have been shown to benefit from setting the Access Pattern to Streaming. With our current testing, Concurrency versus Streaming has demonstrated no performance increase or decrease except for this one case.

OneFS Prefetch
Occasionally, you will have an application that has an access pattern that is so random in nature, that the prefetching that OneFS does with each file can be too aggressive in regards to the overall performance of the application. You can turn off prefetching either temporarily or permanently. Until you are sure that turning off prefetch results in an increase in performance, you should only disable prefetch temporarily. To turn off prefetching temporarily, login to one node of the cluster and type isi_for_array sysctl efs.bam.enable_prefetch=0; isi_flush To turn off prefetching permanently, login to one node of the cluster and add the following line to the file /etc/mcp/override/sysctl.conf efs.bam.enable_prefetch=0 After you are done adding the line to the file, you must reboot the cluster in order to have prefetching disabled.

ISILON SYSTEMS

LUN Provisioning
Isilon supports provisioning LUNs as thin or thick. Provisioning a LUN as thin will take less time at creation. During testing, it has been demonstrated that Thinly Provisioned LUNs may initially have higher performance than Thickly Provisioned LUNs. This performance improvement occurs because every initial write in a thinly provisioned LUN does not result a read, modify and write operation within the OneFS filesystem. Of course, this improvement goes away once a block has been written once within the thin LUN. Every rewrite within a thin LUN could potentially result in the same read, modify and write operation. As previously written in the sections entitled Optimal Block Size and 2x Protection Recommended, it is possible to reduce or remove the expense of the rewrite operation for both Thin and Thick LUNs.

LUN Resizing
Isilon supports resizing of LUNs to be larger than original size, but does not support shrinking of LUNs. The size of the LUN can be increased while it is being reserved, and while another resize operation is still pending. LUNs that are thinly allocated or fully allocated can be resized.

LUN Backup
There are several ways to backup the contents of a LUN. One method would be to make a snapshot of LUN and then to backup the snapshot directly from the cluster. This method has the advantage of having the highest backup performance, but recovery entails restoring the entire LUN and not just the piece of data that is missing. The preferred method also involves taking a snapshot and assigning that snapshot LUN to a Target. Once it is assigned to a Target, have a client initiator backup the contents of the LUN with the appropriate backup software. This method is just a little bit slower, but has the advantage of allowing the user to recover any specific piece of data within the LUN. Under both methods, it will be necessary to first quiesce the LUN before taking a snapshot. This ensures that all data has been flushed from the client to the LUN prior to the creation of the snapshot. One method to achieve this in a Microsoft environment would be to use Volume Shadow Service (VSS). Please see the Best Practices Guide for the MIcrosoft iSCSI Initiator for more information on this method.

iSCSI with SyncIQ

SyncIQ currently does not replicate snapshot information. As previously discussed, iSCSI can create three different clone types: Normal, Snapshot and Shadow. All three of these types of clones utilize snapshots within OneFS. However, the Normal clone type only uses a snapshot to start the clone and then discards the snapshot once the cloning has completed. The Normal clone is the only type of clone that is currently certified to work with SyncIQ.

4. Conclusion
This Best Practices guide is focused on helping you configure iSCSI within the OneFS filesystem and achieving the best performance possible. I hope this information has been beneficial to you and will allow you to realize potential block-level access use and Unified Storage from Isilon products. The references below should be helpful and contain more in-depth information of many of the topics discussed in this paper.

ISILON SYSTEMS

5. References
Isilon Best Practices Guide for the Microsoft iSCSI Initiator OneFS 5_5_4 iSCSI Module

Isilon Systems (NASDAQ: ISLN) is the proven leader in scale-out NAS. Isilon clustered storage and data management solutions drive unique business value for customers by maximizing the performance of their missioncritical applications, workflows, and processes. Isilon enables enterprises and research organizations worldwide to manage large and rapidly growing amounts of file-based data in a highly scalable, easy-to-manage, and costeffective way. Information about Isilon can be found at http://www.isilon.com.
2009 Isilon Systems, Inc. All rights reserved. Isilon , Isilon Systems, OneFS, SyncIQ are registered trademarks of Isilon Systems, Inc. Isilon IQ, SmartConnect, SnapshotIQ, TrueScale, Autobalance, FlexProtect, SmartCache, HOW BREAKTHROUGHS BEGIN. and the Isilon logo are trademarks or registered trademarks of Isilon. Other product and company names mentioned are the trademarks of their respective owners. U.S. Patent Numbers 7,146,524; 7,346,720; 7,386,675. Other patents pending.

ISILON SYSTEMS