Have you ever looked at the architecture and over-subscription of your network gear? If you have, were you shocked at the products limited ability to pass traffic? I have and yes, I was surprised that switch was undersized for the function it was serving. As time went on, this over-subscription caused major problems in the network.
It’s very difficult to troubleshoot an over-subscription issue. Customers complain that sessions to servers time out or run very slow. To troubleshoot we look at utilization stats in the monitoring tools and on the physical interfaces. We don’t find a problem of over utilization anywhere. The only good way of identifying this issue is to know your hardware. You need to know what block of ports connect to the same ASIC and the amount of bandwidth that the ASIC has to the backplane. You also need to know the bandwidth of each slot on the backplane.
By knowing the architecture of the product, I was able to add the utilization amounts of all of the ports on that ASIC to find out that the 1Gig connection from the ASIC to the backplane was at capacity.
Vendors will design multiple line cards for the a chassis that look very similar. Even though they are similar, it doesn’t mean they all perform the same. They may make a lower cost 10/100/1000 that is 8 Gig ports to 1 Gig to the backplane (8 to 1). Another line card may be 4 to 1 over-subscribed (Better). The highest priced card may be full line rate (Best). The vendors do this to provide a cost effective solution to fulfill the variety of needs of the customer.
Before selecting a line card that is NOT full line rate, I suggest you ask your sales engineer for a few slides on the architecture of the line card. These slides should show you the architecture and packet flow of the line card. Once you have this information, you will know what ports are oversubscribed.
Anymore, I always ask for full line rate hardware. We pay more for it up front, but I never have to worry about backplane or ASIC capacity issues.
If you have been burnt by an undersized or oversubscribed line card, how did you figure it out?
What change in the network caused the issue?
What did you do to avoid it in the future?