Data Storage Configuration

Here is a summary of how to configure hard disks on Windows and Linux PC machines. It describes SAN and NAS.

Blogs on this topic include:

Topics this page:

Site Map  
About this site  

Data Storage Technologies

"JBOD" (Just a Bunch Of Disks) refers to hard disks that aren't configured according to a RAID subsystem of disk drives that improves performance and fault tolerance.

As of this writing, common capacities on for single hard drive mechanical devices range up to 750GB and 1+ Terrabytes (1000 GB).

Directly Attached Storage (DAS)

These are traditional hard disks, which uses one of two technologies to interface to the motherboard:
1. parallel ATA interface.

2. SCSI (SCSI) from Adaptec and LSI logic

3. FC (Fiber Channel) such as the IBM FastT900 storage array. But Fibre Channel was designed as a campus-wide network, not to exceed 100 miles, and there are many difficult barriers to transporting Fibre Channel over the WAN because of Fibre Channel's sensitivity to delay. Additionally, while there are established protocols such as IFCP (Internet Fibre Channel Protocol) and FCoIP Fibre Channel over IP) to allow Fibre Channel to extend further, Fibre Channel is a closed network and these alternatives are piecemeal approaches that require specialized hardware and introduce additional complexities and vulnerabilities into the network.

To speed things up with ATA:

SCSI provides significant performance gains for drives or controllers because it has a robust tagged command queuing implementation which allows multiple commands to be outstanding and executed in an optimal execution manner.

This intelligence makes SCSI more expensive, since SCSI uses two processors — one for executing the commands and handling the interface and another processor controlling the head positioning through servos.

Ultra320 SCSI

PCI Express replaces PCI and PCI-X on motherboards, and as a serial interconnect, provides high performance (multiple Gbps per connection) and scalability to networked storage systems.

SATA/IDE

SATA (Serial ATA) also known as IDE was defined by the Serial ATA International Organization (SATA-IO) to be software compatible with Parallel ATA, which offer data rates of up to of 100 MB/s. SATA has a max. data transfer rate of 59 MBps through a single port. Serial ATA supports data rates up to 150 MB/s.

It uses ATA disks in a redundant array to provide mass, reliable storage inexpensively.

The new interface also provides for command queuing to further boost system performance.

SATA uses smaller cable connectors with improved silicon design for lower voltage that alleviates current design requirements in Parallel ATA from Seagate

SAS (Serial Attached SCSI)

SAS (Serial Attached SCSI) began shipping in 2005 as Seagate and Maxtor Atlas 10K V and Atlas 15K II drives.
SAS offers a serial point-to-point technology which achieves 3Gb/s because it eliminates the parallel bus overhead of today's parallel SCSI technology. SAS also adds dual porting , full duplex, device addressing up to 128 drives. Low-cost switches, known as expanders, aggregate hundreds of drives. yet SAS offers longer cable distances. The SAS interface is compatible with (provides universal interconnect with) SATA devices and logical SCSI compatibility, but when an SAS system is used as a base. However, SAS connectors are much smaller than SCSI connectors.

Processor.com article

HP

NAS (Networked Data Storage)

Quiz Question: Which three statements describe differences between Storage Area Nework (SAN) and Network Attached Storage (NAS) solutions? (Choose three.) 1. NAS requires a Cluster Server solution. 2. SAN is generally more expensive but provides higher performance. 3. NAS uses TCP/IP for communication between hosts and the NAS server. 4. NAS requires additional hardware on a host: a Host Bus Adapter for connectivity. 5. SAN uses proprietary protocols for communication between hosts and the SAN fabric. Answer: B, C, E

Separating storage from the server reduces the file serving activity and I/O bottlenecks and increases server bandwidth. CPU cycles can then be dedicated to handling application requests, resulting in improved client response time.

There are three major technologies:

• iSCSI (internet SCSI) commands over the internet
• NAS (Network Attached Storage) serving files
• SAN (Storage Area Network) on separate network fabric
• Combinations of the above

iSCSI (Internet SCSI)

The iSCSI (Internet SCSI) RFC 2026 at draft 20 completed in February 2003 specifies how to run SCSI commands over TCP/IP, providing a lower cost alternative for storage area networking, using cards from:
• QLogic SANblade[tm] 4000 Series
• Intel PRO/1000 T IP Storage Adapter
• Emulex GN9000/SI(VI) 1Gb/s iSCSI Host Bus Adapters
• Adaptec ASA-7211 iSCSI HBA

iSCSI IETF RFC 3720 introduces complete error recovery mechanisms called Error Recovery Level Two (ERL 2)

In any SCSI connection there is at least one initiator and one target.
Initiators are the devices which request, or initiate, any SCSI communications. They request data writes, reads and any other SCSI operations. Usually initiator is the HBA in the computer which is using SCSI disks, tapes and other target devices.
Targets are the devices which perform SCSI commands at a request from initiators, but never initiate SCSI activity. Examples of SCSI targets are: disks, tapes, RAID arrays, robotic libraries and many more.

A Performance Comparison of NFS and iSCSI for IP-Networked Storage showed that iSCSI is faster than NFS because iSCSI caches and updates meta-data asynchronously and transfers blocks rather than files. here and Mathias Gug of CERN

Microsoft iSCSI Software Initiator Version 2.0 (build 1653)

Network Attached Storage (NAS)

Benefits of NAS EMC and Network Appliance are two leading vendors. Iomega NAS filer servers. Storage Magazine Enterprise Storage Forum

NAS filers are special-purpose file servers (i.e., "appliances") that attach to a local area network (LAN) to deliver files to client systems - or other servers acting as clients - via TCP/IP within a LAN.

NAS filers are sometimes called NAS "heads" because the NAS "node" is referenced using the IP address of the head device.

Most NAS supports Multi-platform File Sharing by simultaneously supporting Windows Common Internet File System (CIFS) and Unix Network File System (NFS) as well as file systems associated with Macintosh, Novell, and other operating systems. This makes them ideal for sharing files across OS platforms on the same network.

CIFS was formerly known as Server Message Block (SMB) developed by IBM and Microsoft to support file sharing in DOS. This protocol is used today in UNIX systems as part of the Samba open-source utility package.

Many NAS systems also support HTTP so that clients can download files and administer the system using their Web browser.

Since NAS filers do not need a general-purpose operating system, they cost less, have less to go wrong. They also have less avenues of attack, which make them more secure than file servers.

Some NAS systems can expand into multiple terabytes. Non-scaling NAS systems need to be taken offline to redistribute data when adding capacity.

Storage Area Network (SAN) Storage

A SAN provides a high-speed data path between server and storage independent of the LAN (on another NIC/network cable). SAN storage connects to multiple servers through a separate fabric (storage area network), not via the LAN.

With a SAN, app servers access files contained in SAN storage using basic block I/O commands. just as if the storage were part of the server — not by calling for files over a LAN.

Instead of a NIC card, servers access the SAN using an HBA (Host Bus Adapter) board to encode data per the fast & secure 8B/10B scheme that addresses SANs with a hard-coded 64-bit World Wide Name (WWN) and World Wide Port_Name (WWPN).

Although a SAN does not (usually) offer file sharing, it does offer storage sharing to servers. The storage sharing can be physical (with a fixed logical "wall" between servers that run different operating systems), or partitioned logical storage (shared by servers that run the same operating system).

Adapted from this article, from an Aberdeen Group report, and Intel

Due to the high-speed (1 to 2Gb/s data transfer rates in 2006, and 10Gg/s in 2008), SANs usually run though a fiber channel (IEEE 802.2) networking equipment.

At the fabric layer, fibre technology provides sophisticated cascading switches, switch initialization, and zoning.

It is almost a mute point to compare the total costs of a SAN, since in may large/enterprise shops that need highly-available central consolidated data store for clusters of servers to access, it has become a "must-have" for its ability to handle large amounts of data quickly and securely at low per-byte hardware, power, and manpower cost.

Hardware Cost:	SANs have a higher initial cost but lower incremental cost to add storage. Traditional DAS have a lower initial cost and straight-line incremental cost.	-
Scalability:	DAS storage arrays have limited capacity and are difficult to grow. SANs and have larger capacity limits (up to 64 TBs for a FastT900 storage array).	-
Ease of Use:	Additional storage can be added on a SAN dynamically as required, so actions can be taken proactively rather than reactively when problems occur. Large volumes of SAN storage (50TBs) can be managed more easily than managing large amounts of direct attached storage spread over many physical locations.	-
Utilization:	Since individual drives are 30% used, SANs make more free space available by combining small amounts of storage capacity together.	-
Robustness:	A SAN enables data mirroring across multiple sites — this means "inline" data backups — a key feature in business continuity plans for priority one services.	-
Flexibility:	SAN provides quick snapshot technology for data replication and protection, without the down time associated with backups of data that quickly become stale.	-
Performance:	Because a SAN is built with firmware and communicates using more compact protocols, it can serve data quicker than a NAS which uses general software parsign standard IP. This is most pronounced when moving large amounts of data (e.g., for backup or streaming applications).	-

For more information:
Storage Network Industry Association (SNIA)

FREE vendor-neutral SAN Fundamentals 8-hour on-line class professionally developed and offered at HP since Feb. 2004
HP offers free full-line technical on-line classes on Storage software and solutions HP storage technologies such as HP StorageWorks.

Fiber comes with advanced services such as

• Fabric Login (FLOGI) — Enables nodes to be successfully initialized (allocated a unique address) in a switched environment, enabling communication between two nodes
• Simple Name Server (SNS) — Helps a source node to discover the destination node within the fabric without causing unnecessary communication overhead
• Registered State Change Notification (RSCN) that notifies Fibre Channel nodes about the changes in the existing topology

Fibre Channel technology are used on trans-oceanic cables (which have repeaters every 10km, powered by a copper sheath around the fiber.) But the HBA and devices can be up to 500m apart.

A series of standards from the American National Standards Institute (ANSI) defines 3 main topologies:

• point-to-point, where devices are directly connected to each other (without the use of hubs, switches, or routers). Transmissions are sychronous (cannot transmit and receive, simultaneously).
• Fibre Channel Arbitrated Loop (FC-AL), which shares bandwidth with up to 126 nodes on a distributed uni-directional ring topology, connects with hubs — the simplest form of a fabric topology.
• Fibre Channel Switched Fabric (FC-SW) provides nondisruptive scalability and switch connection among up to 16 million nodes — the highest performance and connectivity topology.

Transmits data in frames of 2148 bytes maximum.

SAN Vendors and Products

• HP StorageWorks 1000 Modular Smart Array
• EqualLogic

Converged products

Converged NAS/SAN products are "two dialect device," or 2DD for short.

DLink uses its xStack Storage architecture which uses iSCSI.

BlueArc offers a high-end NAS server with SAN attributes, such as support for 10,000 or more users, multi-gigabit throughput, and up to 250 TB capacity.

Disks under Windows OS

Windows 9x/NT Basic Drives and Partitions

Windows 9x and NT only uses drives with MBR technology, which Windows 2000 calls basic drives. Windows 2000 users can convert to dynamic volumes, but Windows 2000 still needs a basic drive to house a system volume for boot-up.

When the computer powers up, the BIOS looks for boot-up files on the system partition marked as an active partition..

Partitions

Drives using MBR technology can have up to 3 primary partitions, plus one extended partition.

Space on a physical disk not used by a primary partition can be used by the Extended Partition. The extended partition can be formatted for use by other operating systems (such as Linux). DOS divides the extended partition into smaller segments, each assigned a logical drive and formatted with a FAT file system.

Creating Disks

To create MBR partitions, run from a bootable floppy or the Win2000 setup CD:
FDISK

The selection for "large disk" partitions is really about selecting FAT16 or FAT32. FAT16 drives can only accomodate 2048 MB (2 gigabytes) per partition.

For more on this topic, see:

FDISK Guide to Hard Drive Partitioning
FDISK.COM
bootDISK.COM

System Partition

The system partition (also called a system volume) holds the hardware-specific files needed to load the Windows operating system. The boot partition holds the remainder of the operating system files.

The system volume must be the Active partition on the disk partition table on a basic disk, typically drive C. The system volume cannnot use unallocated space on a Win2000-only dynamic disk.

Boot Partition

If you have a (formatted) data diskette inserted during boot-up, the message you get depends of the operating system:
• COMMAND.COM not found, or
• NTLODR not found

Each Primary partition set as active contains boot files to start the operating system. Each OS has a different version of these hidden read-only system boot files:

OS	Boot files	OS Parent directory
Windows 3.1 thru 98 & Me	IO.SYS, MSDOS.SYS, COMMAND.COM	Windows
Windows NT4, XP, 2000, and others in the future	NTLODR	Winnt

Each parent folder contains subfolders such as System32, which holds the Windows Kernel kernel32.dll.

Basic vs Dynamic Disks

Logical drives and partitions are Basic disk storage.

Windows 2000 can create "Dynamic Disks", then -without rebooting- resize, stripe, mirror, and RAID-5 them.

Boot and system partitions can't be in a RAID-5 volume - dynamic volumes (and therefore cannot be spanned).

FDisk Creation

How Fdisk works - step by step with sample screens

A Simple Dynamic volume on Windows 2000 may or MAY NOT have a partition table!

• When FDISK.EXE is used to create a "Basic" disk, it creates a partition table.
• When a disk created with FDISK is then converted by Windows 2000 Disk Management, the legacy partition table is retained on the disk.
• When Windows 2000 Disk Management is used to create a disk, a legacy partition table is NOT created.

Converting Hard Drive Among Operating Systems

Windows 2000 Installation Windows 2000 Administration Backing up data

Troubleshooting Windows Messages

• "Fixed disk error" -- Setup CMOS
• "Drive mismatch error" -- Setup CMOS
• "ROM BASIC Interpreter not found" -- FDISK to create partition info
• "Missing operating system" -- MBR is bad, FDISK \MBR
• "Bad or missing COMMAND.COM" -- SYS C:
• "Disk boot failure" -- reinstall SYS
• "Invalid media" -- FORMAT
• "Low drive space" -- add more RAM (not hard disk space)
• Windows 95 OSR2 ("B" in the System Properties tab part number and VER version number 4.00.950.1111 ) offers APM functions, bus mastering support, MMX multimedia support, and enhanced PCMCIA functions.

• NT uses NTFS v4. Win2000 uses NTFS v5, which stores an Access Control List with each file & folder to enable file-level access-control, encryption, compression, disk quotas, recovery logs. NTFS 5 also enables Distributed Link Tracking to preserve shortcuts when files are moved; and a drive posing as a folder.

• To copy files from an NTFS drive to a DOS drive, use the NTFSDOS driver for Windows 9x from Sysinternals.com

  Basic to Simple Dynamic Disk Convertion

  To convert (upgrade) a basic disk to a simple "Dynamic Disk"
  convert d: /fs:ntfs

  Converted drives have Full Control to Everyone, so set default NTFS permissions:
  secedit.exe

  Unlike drives created as dynamic, converted drives retain partition information.

  Mirroring Disks/Volumes

  Mirrored volumes in Win2000 use the Ftdisk.sys driver to write data to physical disks.

  Spanned (Extended) Disks/Volumes

  Simple dynamic volumes are made complex -spanned- using the Extend command of the Windows 2000 Disk Management MMC snap-in to Computer Management, which does NOT create disk partitions.

  Striped Disks/Volumes

  Striped volumes are good for faster performance because they are written to one drive at a time.

Hard Drive Usage Strategy

The tradeoff is between money and performance. For the maximum performance using 7 hard disk drives, install an NT Server or Exchange Server this way.

For the NT boot/system FAT partition: a single 2 GB FAT16 or NTFS partition.

But keep an alternate install NT handy to swap in case of emergency.

FAT16 partitions are formatted at or below 2 GB using a 32K cluster size, or 4GB if partitioned by NT using a 64K cluster size. This avoids having any file beyond the 1024th cylinder during the installation of service packs. (See KB article Q197295).

Making this a FAT16 partition allows a DOS floppy to be used in booting up the system when recovering from problems with NT. Because FAT partitions are not secure, there should be no public shares on this volume.

The BIOS int13H disk I/O function calls interface limits the system partition to 7.8GB size IF you first partition and format it as NTFS on another PC and move the HD to the system you want to install NT to -- per KB Q119497. 

Plus, a partition formatted NTFS using Disk Administrator performs slightly better than one converted from FAT.

But there's a new ATAPI.SYS in SP4 that handles IDE drives larger than 8GB. Have it on a diskette during the install. KB Q197667 explains it and gives steps to install it during an initial NT setup.

Maxims:

• Don't install anything to the system partition unless it won't work on another drive.
• Spread the workload among physical drives so that roughly equal demands are placed on each drive.
• Use I/O stats to determine which files the most heavily used so that they are placed on very beginning of the disk, to the least often used on the extreme inside. This reduces the average head swing to seek data on the drive, thereby reducing average seek time in the process.

For paging files, a single 2 GB FAT16 drive 

The minimum is 100 MB plus the amount of RAM, which is 128 MB for the smallest servers. More practical amounts of RAM would be 256 MB to 512 MB. Some multiprocessor servers may even require 1 GB of RAM for optimum performance. Although all 2 GB would probably never be needed for paging, you can use the extra space as archival space for rarely used drivers, install files, etc.

Because of their volatility, paging data should not be mirrored.

FAT16 partitions are much faster than NTFS when used in write intensive situations. This is because the architecture of NTFS requires a duplicate copy of the Master File Table (MFT) to be placed in the middle of the drive partition to supplement the one that is at the beginning of the partition. When new data is written to a file it causes properties of the file object to change (such as the file size, the date the file was last changed, the user that last updated the file, and possibly the cluster allocation chain). This information must be updated on both copies of the MFT. This causes a large amount of head swing since the second copy is in the center of the partition.

For user data and applications, a RAID 5 array:

This requires a minimum of three but optimally 5 drives.

All total, this configuration requires seven drives at minimum. I do recommend that if you are going to reinstall your server to redo your disk configuration, that you install separate drives for the system volume and your RAID. There are dozens of valid choices about configuring your drives. You must weigh cost vs. required performance to arrive at the best solution.

For Exchange/SQL server log files: 2 GB FAT partitions which are hardware mirrored.

Since log files are sequentially written to the disk, the best performance occurs if the disk read head doesn't have to move for other tasks. This disk needs to be mirrored for error recovery.

Exchange Server will operate in a Microsoft Cluster Server environment, but there are a bunch of problems. You may wish to check out some of the MSKB articles

If you use a cluster, turn off circular logging if you want a fairly clean backup.

Thanks to Tom Baffy on ZDU

Hard Drive Management

PowerQuest Storage Resource Management

FileScreen from Precise WQuinn monitors file being loaded on the server so that it can block specific file types (such as .mp3).

Wipe Out

These utilities recover information from hard drives gone bad:

$125 R-Studio from R-TT.COM claims to be the most comprehensive data recovery solution for FAT12/16/32, NTFS, NTFS5, and Ext2FS. It recovers data both on local logical and physical disks -- including encrypted files, compressed files on RAID drives -- even if their partition structures are damaged or deleted. Their $180 network edition recovers disks on remote computers over networks. R-Mail from R-TT.COM recovers corrupted *.dbx files where MS Outlook Express 5.0/ 5.01/5.5/6.0 store folders with messages. The new e-mail recovery technology allows the users to rebuild corrupted *.dbx files and restore accidentally deleted messages. The messages are recovered in the .eml format and can be imported into Outlook Express mail bases.

These utilities clean residual data on hard drives:

BCWipe version 1.x and 2.x from Jetico

Eraser 5.3

SecureClean v3 build-2.0

East-Tec Eraser 2000

PGP 6.x freeware and commercial, 7.x, freeware and commercial

On Jan 21, 2002 Kurt Seifried announced that windows file wiping utilities do not wipe alternate data streams with NTFS file systems: In the NTFS file system a facility exists to bind additional data to a file or directory, called an alternate data stream [url1][url2]. These alternate data streams cannot be be removed, unless the parent file or directory is destroyed. Unfortunately most file wiping utilities only deal with the primary data stream and do not wipe the alternate data streams, thus leaving data intact.