dd if=/dev/random of=/dev/null bs=always

Please note that this is only a personal opinion of mine as I have been observing the growth and various decline of storage concepts within the data storage industry. The views of the reader may differ from my own which is why I would invite you to please post your opinions as a comment to this post.

One of the most volatile and yet needed industries is the data storage industry. As computing technologies become more cloud centric and rely upon the web for business, productivity, education to even recreation, there is a constant push to increase capacities but even more so increase I/O throughput. As a result of recent demands, our approach with these technologies need to be re-evaluated. The primary focus of this article is on the future of data storage concepts and the limited life and functionality of RAID.

Back in 1987 when the idea of RAID was first conceived, the goal or vision was to be able to scale multiple drives into a single volume which was represented to a host as such while also offering a form of redundancy with a more sensitive magnetic platter-based disk technology. Flash forward to the present and we are still reliant upon the same technologies. Is that because RAID is so perfect or have we just grown too comfortable and are too afraid of change?

Hardware Vs. Software RAID 

There was a time when processing power was limited and it became advantageous to utilize external methods for creating and managing arrays of data storage, but as time progressed, this approach became increasingly insignificant. At least that is to say for the Small-to-Medium sized Business (SMB). For the last decade, a lot of efforts have been placed toward increasing the reliability, stability and enhanced features with the software-based RAID. This has slowly been eating away at the hardware vendors. Although it has been rarely noticeable.

These software implementations are integrated with methods of Logical Volume Management (with built in redundancy via RAID 1-6), Load Balancing/Multipathing capabilities, data encryption, along with the abilities to utilize incremental snapshot(s) over designated volumes. These software implementations include dynamic resizing, quota/permission management, enhanced copy-on-write file systems that perform very well along with routine checksums to correct noisy and silent data corruption; almost all of which can be managed while volumes are on-line. Some of these volume managers have the capability to export iSCSI & FCoE targets and can also be tuned to support FC targets.

To name a few you have ZFS (an all-in-one solution), Btrfs (still in development and under test), device-mapper / LVM2 / multipath-tools, mdadm, DRBD, etc. The list goes on. What is to stop an SMB from setting up an array of JBODS and (if more redundancy is needed) cluster a couple of Solaris / OpenSolaris or Linux servers to manage their software RAID while also exporting it via a file server or into a SAN? Note that Lustre support for ZFS is still in development. Realistically most entry-level modular external RAID solutions don’t run on the latest and greatest of hardware components (as they are intended for a limited purpose and not to provide other hosting services). You will most likely achieve much greater performance with the software approach while also utilizing a much more efficient virtual memory manager (for enhanced caching) alongside a finely tuned schedular.

On the enterprise end of computing you will find some very impressive storage solutions that are intended to take the workload of the enterprise environment. Such companies as Hitachi Data Systems (HDS) have been doing an excellent job with providing high-quality and well performing storage solutions that are also easily manageable. Other companies have resorted to being a little creative in order to gain some market share with the SMB and larger companies. Such notable companies are NetApp, Data Domain to even Cleversafe.

Earlier I found an interesting link differentiating the positives and negatives of both hardware and software RAID implementations. It should be noted that times have changed and some of the key points highlighted are no longer an issue. For instance, under the category /boot partition, this seems to no longer be an issue with at least ZFS.

Enter the SSD

In more recent years, the Flash-based Solid State Drive (SSD) has been entering into enterprise markets. This is a result from such notable providers as Sun Microsystems, etc. Currently the percentage in SSD usage in the enterprise is somewhat minimal as their is a limit in maximum capacities for the drives. This may soon change as in Q3 of 2009, PureSilicon will release their Nitro 1TB SSD drive. The throughput and performance speeds seem very optimal in arenas where greater speeds are needed, but the technology introduces additional handicaps (in the form of write operations and a limited cell life) which most environments and some manufacturers have a difficult time in accomodating to. To combat the limited cell life, vendors have implemented their own method of wear leveling, transparent to the host. With this concept, the same data cell, when accessed and written to multiple times will not get written to the exact location but instead, through an “intelligent” built in firmware the data will get written to another cell on the drive. To the operating system, it is still the same “sector” location. While there is very little latency in seeking performance (sequential and random), write operations take a huge hit, especially with smaller I/O transfer sizes, when typically the flash medium erase/rewrite a 128K page at a time.

SSD Tuning

With the recent hype of Flash-based SSDs, many vendors and UNIX/Linux distributions have been writing file systems tuned to perform extremely well on SSDs (and limit the impact of these handicaps). For example, Sun Microsystem’s ZFS (available on Solaris, OpenSolaris, MacOS X [read-only], FreeBSD and Linux [over FUSE]) had recently added tunable support for SSDs in their release versions for Solaris & OpenSolaris, while the development of Btrfs for Linux has done the same. In contrast the Microsoft developed NTFS does not offer such features or functionality. In fact the file system has remained somewhat unchanged over the course of the years and is just as inferior now as it was when it was first released as a replacement to the FAT series of file systems. I wrote an entire post explaining why the NTFS file system is not well suited for today’s methods of computing here.

In recent releases it should be noted that Microsoft’s Windows 7 has been tuned for SSDs that are to be provided on netbooks. What this means, I do not know? And by tuned, this is still unclear. You can read some of that information here. The only reason for the lack of changes in NTFS is to preserve backwards compatibility. This approach limits the ability to update a current existing server’s (if not running Windows 7) NTFS module if it needed to serve backend storage utilizing SSD media.

The Impact on RAID Technologies

As SSDs become more popular the advantages to using RAID are reduced, where the only benefits are gained from a simple stripe in a RAID 0 or mirroring to a backup array within a SAN or other form of network using RAID 01 (not to be confused with a RAID 10); just in case access to the first fails for whatever reason. This is where DRBD would come in real handy. As I briefly mentioned earlier, the whole concept of this form of redundancy was dependent upon the problematic nature of a magnetic disk device; where failures were imminent. And for those who are concerned with a method of error detection for both silent and noisy data corruptions, the majority of RAID implementations (both hardware and software) do not validate the data like the ZFS or Btrfs checksum implementation.

Changes in Protocol Layers?

With the popularity of SSD technologies growing and its costs reducing, the one drawback that is setting manufacturers and consumers back are the limitations offered by the protocols that they are working with. Today, Fibre Channel, SAS and SATA are not capable of handling full SSD speeds and serve only as a bottleneck to the technology. There have been recent attempts from vendors as Fusion-io to even PureSilicon to rely on other protocol interfaces such as PCI Express (PCI-E). Capable of handling up to 1 GB per second, it only seems natural for these vendors to move in that direction. I anticipate that shortly, others will follow. Fibre Channel and SAS may continue to serve the SAN (and with the appropriate load balancing mechanisms configured, it will perform well) but when it comes to the drive within the chassis, I expect to see more PCI Express in the near future. But who knows, with the recent drop in prices for 10Gb Ethernet or the supported high throughput offered from Infiniband, things may be moving toward another direction altogether.

In conclusion, I predict that in five years time we will start to see some huge and very interesting changes. I am looking forward to it.