By David Gross
One of the most interesting topics in data networking over the last few years has been the persistence of Fibre Channel and InfiniBand. Despite cries that both will disappear, neither has faded like ATM, FDDI, and Token Ring did in the 90s.
The case for stand-alone Fibre Channel (not Fibre Channel over Ethernet) remains very strong, especially with progress continuing on the 16G standard. With the ability to transmit large block transfers over 130 Megabytes, not to mention a loyal base of SAN managers, it remains better-suited than Ethernet for many non-attached storage applications. However, the case for InfiniBand still seems unclear to many, including some in the InfiniBand Trade Association, who developed RoCE, or RDMA over Converged Ethernet, which is essentially InfiniBand over Ethernet, out of fear that InfiniBand could struggle to survive on its own.
The fact that so many people compare InfiniBand to Ethernet ignores an important point - InfiniBand's growth has had little to do with taking out Ethernet, but rather taking share from proprietary and specialized supercomputing interconnects like Myrinet and Quadrics. Over the last five years, Gigabit Ethernet's share of interconnects on the Top500 supercomputers has actually increased slightly, from 214 to 227, according to Top500.org. While InfiniBand has soared from just 16 to 210 over the same time, it has mostly come at the expense of Myrinet, which has dropped from 139 to 4. Quadrics, a former Supercomputing favorite, went out of business last year. And even with decline of proprietary interconnects, just one of the top 100 supercomputers is now using Ethernet.
Additionally, while there is a lot of talk about latency and bandwidth, there is another key metric supercomputing networks are built around - MPI Message Passing rate. Mellanox (MLNX) recently announced it had the capability to transmit more than 90 million MPI messages per second. In addition to low cost 40 Gigabit ports, one reason why InfiniBand has such low latency is the protocol's own message size can be cut down to 256 bytes, and most supercomputers achieve high performance by breaking up requests into tiny fragments across multiple nodes. This InfiniBand benefit is directly opposite to the large block transfers Fibre Channel provides SANs, which have preserved that protocol's strength in storage.
Another reason why InfiniBand has offered such great price/performance in supercomputing, but is almost nonexistent in enterprise data centers, is it use of local addressing. InfiniBand switches frames based on a 16-bit local ID, with the expectation that they are not leaving the cluster. It is ultimately an I/O technology, not a networking technology. Ethernet, on the other hand, uses global 48-bit MAC addressing, and many of the frames coming in and out of data center servers are heading back and coming from the public Internet. While InfiniBand has a layer 3 global ID as well, it is built right into the InfiniBand stack. Ethernet's layer 3 forwarding has to be handled by IP, which means buying an expensive router, which is not justified if you've got a high performance network but are not sending much traffic out to the public network. And like supercomputing clusters, financial traders are not using InfiniBand to connect their web servers to public networks, but rather for private networks. Where those same financial firms need to connect public-facing web servers within data centers, they use Ethernet.
Comparing InfiniBand to Ethernet can create interesting debates, but it's mostly a theoretical argument, because it's not a decision many data center managers face now or will face in the future. InfiniBand is an I/O technology mostly serving high-end supercomputing clusters and trading networks, both of which are expanding and promise further growth for the technology. Ethernet is the dominant LAN switching technology in the data center, and no one is seriously talking about replacing it with InfiniBand. Therefore, when looking at the data center networking and supercomputing interconnect market simultaneously, it makes more sense to think about Ethernet and InfiniBand rather than Ethernet vs. InfiniBand.
It is true that in the early days InfiniBand took over the proprietary interconnect on the Top500, but in the last 3 lists, there was not much left on the proprietary side, and InfiniBand start to take over the Ethernet connected clusters – you can see how Ethernet start to decrease while InfiniBand continue to increase. Check – http://www.mellanox.com/pdf/applications/Top500_June_2010.pdf for more details.
ReplyDelete