PCI Express, or PCIe, is a computer motherboard interconnect that's used to expand the performance, functionality
or connectivity of a server. HBAs, NICs, graphics accelerator cards and RAM expansion modules are common examples. Solid-state storage devices in the form of NAND flash are a newer application for the PCIe interconnect. This article will detail PCIe SSD: PCIe cards that contain on-board SSDs and provide internal storage to a server as opposed to external SSD arrays that are connected through a PCIe interface card.
Flash-based SSD is seeing increasing use because disk storage can't keep up with CPU performance; hard drive rotational speed hasn't increased in almost a decade. Even though some performance improvement has come from increased data density (more data per rotation), the fact remains that hard disk-based storage can't get data to and from CPUs fast enough to take advantage of continually improving processors. Memory (DRAM) is too expensive to provide the capacity needed, so flash storage has been tapped to fill this gap. With performance that's a lot closer to DRAM than HDDs but a cost per gigabyte that's in between the two, flash offers a viable alternative to spinning disk storage.
PCIe provides an efficient platform for integrating this flash into the compute environment, putting it onto the server motherboard and moving it closer (physically and logically) to the server compute engine. Compared with external devices -- such as traditional disk arrays with SSDs instead of hard drives, dedicated SSD arrays or caching appliances -- flash on the PCIe card storage can bypass the array controllers, interface protocol and cabling that direct-attached external devices have. They also eliminate the switches and networking connections of SAN-attached arrays.
How NAND flash works
As a data recording medium, NAND flash has a number of characteristics that require special processes. These processes represent overhead to be carried out by either a flash controller that's resident on the PCIe card or by the server CPU. Generally, they're associated with flash write operations but also handle reliability processes like error correction and replacement of bad flash blocks.
When data is written to flash, old data must be erased ahead of each write to make room for the new data. Also, this flash erase must occur in blocks, not at the byte level as with HDDs. This means that when a section of data is deleted, the entire block it's located on must be erased and the bytes not marked for deletion must be copied to another location.
NAND flash, unlike magnetic disk substrate, has a finite life span, meaning it can only be written to and erased a certain number of times. In response to this condition, flash controllers spread data writes across all available blocks so that the entire chip ages at a more even rate. Called "wear leveling," this process maximizes the life of the NAND substrate and, consequently, the SSD that contains it.
PCIe-based flash pros and cons
Flash on a PCIe card puts the SSD closest to the CPU memory complex within the server and affords a lower potential latency than external SSD devices. PCIe is bidirectional, meaning reads and writes can occur simultaneously, and it eliminates the protocol translation of SAS, SATA or FC storage. All this adds up to faster I/O than external devices, which use these protocols and often have to share CPU with RAID functions and storage features as well. Obviously, network-attached flash storage systems put the additional network latency into the equation.
Since it's a captive, dedicated storage device, a PCIe SSD card can be the easiest to install and use. External systems, especially those connected to a storage network and shared with other hosts, are typically more complex to implement. These larger-capacity storage devices represent a bigger investment and often involve strategies like storage tiering and caching to improve SSD utilization.
Being an internal, bus-connected card, PCIe SSD devices cause a reboot when installed or replaced, making expansion or servicing more disruptive. Also, since they consume a PCIe slot, they may reduce the number of other PCIe devices, like network adapters (or additional SSD cards) that can be used. As dedicated storage, PCIe SSD can't be shared with other servers, something that could help cost-justify this premium-priced capacity.
PCIe SSD use cases and products
PCIe SSD is ideal for I/O-intensive applications, like Web 2.0, social networking, OLTP, active databases, etc. It can also provide quick performance improvements for localized application problems that require a large, fast buffer area, like video editing, financial modeling, simulations and data acquisition. This direct-attached storage (DAS) implementation may also appeal to first-time SSD users, compared with the potential complexity of an external or network-attached array. This form factor is also ideal for OEMs to include SSD with servers as an upgrade option.
Texas Memory Systems puts the flash controller on the PCIe card, so it doesn't impact the server CPU or memory resources to handle the write overhead mentioned earlier. According to Levi Norman, director of marketing, "NAND flash returns errors; you just have to manage it. Texas Memory Systems designs PCIe cards as a system, with onboard RAM and CPU, giving it the intelligence to conduct more efficient I/O and improve performance, especially for write-heavy applications."
Mitch Crane, senior director of technical marketing at Fusion-io, said, "Everyone else is trying to make flash look like a disk drive; we're making it look like memory." He said Fusion-io's products use physical memory addresses and DMA to treat flash more like system memory than storage. Fusion-io products don't do flash overhead functions on the card, which Crane said increases complexity and chances for failure.
LSI's PCIe SSD solution takes a little different approach. Its card includes six SSD drive modules, as well as a SAS/SATA bridge that acts an interface between PCIe and these modules. The flash controller functions are handled by the SSD modules themselves. JB Baker, the company's principal product manager, said, "The use of SAS controllers in the solution architecture gives LSI the flexibility in future designs to consider including ports to external SAS/SATA devices, which could enable users to expand and scale the system."
Although PCIe-based SSD products are newer and represent a small portion of the total SSD market currently, they're certainly a hot topic. According to Gartner, market share for PCIe-based SSD was 6% of all solid-state shipments in 2009 and is predicted to be four times that by 2013, as all major SSD vendors come out with products. From a performance perspective, PCIe SSD represents a fast and efficient way to apply flash performance to a server IOPS problem. While performance is the key driver for SSDs, actual mileage may vary depending upon the PCIe solution chosen and the environment. But performance is a discussion integrators are used to having with customers. Another one is implementation.
As practitioners are finding out, getting the kinds of utilization they need to see from their SSD investment is easier said than done. Adding SSDs to existing arrays or putting in dedicated SSD appliances can be a complex and expensive solution. As a smaller, localized implementation of SSD, a PCIe card can be easier to get approved, easier to put in and be faster to show the performance improvements expected.
For VARs, this shouldn't be the only SSD product on the line card. Like other technologies, integrators should handle a range of products and represent multiple implementation options. Whether it's localized application acceleration, dedicated high-performance storage or the simplest way to get an IOPS problem resolved, PCIe SSDs provide a strong set of options for VARs.
About the author
Eric Slack, a senior analyst for Storage Switzerland, has more than 20 years of experience in high-technology industries holding technical management and marketing/sales positions in the computer storage, instrumentation, digital imaging and test equipment fields. He's spent the past 15 years in the data storage field, with storage hardware manufacturers and as a national storage integrator, designing and implementing open systems storage solutions for companies in the Western United States. Find Storage Switzerland's disclosure statement here.