Fibre Channel and InfiniBand vs. Ethernet

by David Gross

As successful as InfiniBand has been with supercomputing centers and financial traders, it has struggled to break into the mass market dominated by Ethernet. Fibre Channel has had great success in SANs, but isn't even trying to go after the mass market anymore. Yet when first standardized over ten years ago, both Fibre Channel and InfiniBand had far greater ambitions.

Through their respective industry associations, InfiniBand and Fibre Channel have responded by developing "over Ethernet" versions of their protocol in recognition of Ethernet's dominance. The InfiniBand Trade Association's RoCE, or RDMA over Converged Ethernet, could have just as easily been called InfiniBand over Ethernet.

The niches that Fibre Channel and InfiniBand currently fill though, aren't going away, but neither protocol is about to challenge Ethernet supremacy. Interestingly, the niche roles each protocol occupy have little to with bandwidth, cost, or price per bit. Fibre Channel has never been cost competitive with Ethernet, even when both topped out at a gigabit, yet has held on strong in storage. InfiniBand is cheaper per bit than Ethernet, particularly at 40G, but is struggling to break out of its supercomputing and financial trading niches.

More than cost per bit or port price, an interesting factor behind the development of the InfiniBand and Fibre Channel niches comes down to message size. InfiniBand is very closely tied to parallel computing, and the shorter messages that result from breaking up a transmission across multiple CPUs and GPUs. Fibre Channel is closely to tied to serial storage networks, particularly the large block transfers that cross SANs, which rely on the protocol's hardware-based error correction and detection, and generally require the link length that comes with a serial protocol. To use a somewhat cheesy analogy, you could say InfiniBand is a little sports car, Fibre Channel a long freight train, and Ethernet a Camry.

Fibre Channel grew on the back of serial storage networks, InfiniBand on parallel supercomputing networks. The biggest threat to either then is not Ethernet, but a revival of parallel SANs and serial supercomputing, neither of which will happen anytime soon.