Testing Fuels The NVMe Revolution

Game-changing technologies are exciting. In a recent blog, Jim O’Reilly wrote that Non-Volatile Memory express (NVMe) was about to "revolutionize" the solid-state drive (SSD) industry in as radical a change as introduced by Fibre Channel in the 90s and SCSI in the 80s.

Revolutions, however, can be tricky to manage. To benefit from new technologies, an industry needs to harness their power. In the case of a new protocol such as NVMe, the interested parties first must collaborate on writing it. Then comes testing and further documentation. Here are five ways the University of New Hampshire Interoperability Lab (UNH-IOL) is supporting NVMe.

Integrators list: With any disruptive technology, it helps to have a single record for products to interoperate across platforms and conform to the new specification or standard. That is the role of the UNH-IOL’s NVMe Integrators List.

How do the UNH-IOL and the NVM Express Organization vet products for this list? First, we collect requirements, often involving company feedback, issues from previous tests, or interest in new features. We discuss those requirements, and if approved, write up and test the procedures. Then the UNH-IOL debuts those procedures at plugfests. 

Test procedures: A documented and publicly available test program ensures transparency for NVMe. The program involves an interop and conformance component.

The interop procedure tries to mimic the use of a drive in the real world. It is first installed in a system, which is booted from a non-NVMe boot drive. The visibility of the NVMe drive in the OS is verified, and the drive is formatted. Then a cross-platform tool is used to perform a series of tests, carried out across a series of reboot and shutdown cycles across a variety of OS and hardware platforms. A number of optional or informational tests can also be performed.

The conformance tests are aimed at validating that the SSD is conformant to the NVMe specification. To that end, we use a test tool that acts as an NVMe host to exercise and stimulate the NVMe SSD.

Common test tools: To ensure uniformity in the testing performed at a company’s internal lab, UNH-IOL, and at any plugfest, a common set of tools is provided through UNH-IOL’s NVMe Consortium. We use two primary test tools to perform the tests.

The first is the IOL Interact PC Edition tool, based on the open source tNVMe tool that the UNH-IOL administers. It can be run using any PC with an Intel PCIe chipset. The second is the IOL Interact Teledyne-Lecroy Edition. This tool is a set of UNH-IOL authored scripts and automation that uses the Summit series of PCIe tools from Teledyne-LeCroy. It can report exactly what is happening on the PCIe bus, and report on NVMe at the packet level. 

Plugfests: Industry plugfests serve as a public marker of the progress that new technologies make. NVMe has held two plugfests so far, both hosted by the UNH-IOL. 

At the plugfest, SSD implementations are cycled through tests against a variety of host platforms, operating systems, and drivers. Company representatives are present, making debugging easier. The plugfests also serve as a kickoff point for new test procedures and requirements. After the event, the UNH-IOL reviews the results to determine which products have qualified for the Integrators List. 

Testing services: Testing that occurs at the plugfest gets a lot of attention, and rightly so. But NVMe test services continue at the UNH-IOL. Some companies may not have been ready for the plugfest, or the event didn’t line up with their product release schedule. The opportunity to test and place products on the NVMe Integrators List is still open through testing at the UNH-IOL throughout the year. 

Results from these plugfests are promising, and as O’Reilly noted in his column, "justify claims that NVMe has arrived." Recent weeks have yielded additional developments signaling increased support for NVMe, including product announcements, OS support, and support from the test community. Testing can be tedious work; but without it, technological breakthroughs such as NVMe would be less beneficial to the industry at large.