Top-of-Rack (ToR) Switching – Better or Worse for Your Data Center?

by Lisa Huff

Earlier this month I talked about how ToR switching has become popular in the data center and how large switch manufacturers are telling you that it’s really the only way to implement 10G Ethernet. I left that post with this question – “But is it the best possible network architecture for the data center?”  And, I promised to get back to you with some assessments.

The answer, of course, is that it depends. Mainly, it depends on what you’re trying to achieve. There are many considerations when moving to this topology. Here are just a few:

1)      Cost 1:  ToR switches typically don’t cost that much unless you consider that if you were using a “traditional” structured cabling approach that you would have a passive patch panel that probably costs, at most, 1/10 that of the switch.

2)      Cost 2:  While the installation of the structured cabling is expensive, if you choose the latest and greatest like CAT6A or CAT7 for copper and OM3 or OM4 for fiber, it typically lasts at least 10 years and could last longer. It can stay there for at least two and possibly three network equipment upgrades. How often do you think you’ll need to replace your ToR switches? Probably every three-to-five years.

3)      Cost 3:  Something most data center managers haven’t considered – heat. I’ve visited a couple of data centers that have implemented ToR switching only to see that after about a month, some of the ports were seeing very high BER’s to the point where they would fail. What was happening is that the switch is deeper than the servers that are stacked below it and was trapping the exhaust heat at the top of the rack where some “questionable” patch cords were being used. This heat caused out-of-spec insertion loss on these copper patch cords and therefore bit errors.

4)      Cost 4: More switch ports than you can actually use within a rack. Some people call this oversubscription. My definition (and the industry’s) for oversubscription is just the opposite so I will not use this term. But, the complaint is this – cabinets are typically 42U or 48U. Each server, if you’re using pizza-box servers, are 1U or 2U. You need a UPS, which is typically 2U or 4U and your ToR switch takes up 1U or 2U. So, the maximum amount of servers you can have in a rack would realistically be 40. Most data centers have much less than this – around 30. In order to connect all of these servers to a ToR switch, you would need a 48-port switch. So you’re paying for eighteen ports that you will most likely never use. Or, what I’ve seen happen sometimes, is that data center managers want to use these extra ports so they connect them to servers in the next cabinet – now you have a cabling nightmare.

So I’ve listed some of the downside. In a future post, I’ll give you some advantages of ToR switching.