Jul 09

Know what products in your network are oversubscribed!!!!

Have you ever looked at the architecture and over-subscription of your network gear? If you have, were you shocked at the products limited ability to pass traffic? I have and yes, I was surprised that switch was undersized for the function it was serving. As time went on, this over-subscription caused major problems in the network.

It’s very difficult to troubleshoot an over-subscription issue. Customers complain that sessions to servers time out or run very slow. To troubleshoot we look at utilization stats in the monitoring tools and on the physical interfaces. We don’t find a problem of over utilization anywhere. The only good way of identifying this issue is to know your hardware. You need to know what block of ports connect to the same ASIC and the amount of bandwidth that the ASIC has to the backplane. You also need to know the bandwidth of each slot on the backplane.

By knowing the architecture of the product, I was able to add the utilization amounts of all of the ports on that ASIC to find out that the 1Gig connection from the ASIC to the backplane was at capacity.

Vendors will design multiple line cards for the a chassis that look very similar. Even though they are similar, it doesn’t mean they all perform the same. They may make a lower cost 10/100/1000 that is 8 Gig ports to 1 Gig to the backplane (8 to 1). Another line card may be 4 to 1 over-subscribed (Better). The highest priced card may be full line rate (Best). The vendors do this to provide a cost effective solution to fulfill the variety of needs of the customer.

Before selecting a line card that is NOT full line rate, I suggest you ask your sales engineer for a few slides on the architecture of the line card. These slides should show you the architecture and packet flow of the line card. Once you have this information, you will know what ports are oversubscribed.

Anymore, I always ask for full line rate hardware. We pay more for it up front, but I never have to worry about backplane or ASIC capacity issues.

If you have been burnt by an undersized or oversubscribed line card, how did you figure it out?
What change in the network caused the issue?
What did you do to avoid it in the future?

May 07

Do you read the release notes for software updates?

Over the past years I have meet many people and watched how they pick their software for their network equipment. Sometimes they pick a good version and other times, it’s crash and burn. Learning from past co-workers I have learned to always read the release notes for that software version.

Sometimes the release notes are a few hundred pages, so I sift through every single page. I try to pick out the parts that apply to me and skip the rest. There are a few things that I look for when I read through the release notes.

Things I look for when reading the release notes of a new software version

1. Compatibility with my hardware.
– Will all of my line cards work?
– Do I have enough memory?
– Are all of my Wireless Access Points supported?
– Are all of my IP Phones and voice gateways supported?
2. Are there incremental upgrades, or can I go directly to this version?
3. What are the new features that may relate to me?
4. What configuration changes have been made
– Cisco ASA 8.3 code is a great example of this. NAT Completely changed in this version.
5. What are the known caveats? Do they apply to me and my hardware?
6. If I’m upgrading to resolve a Bug, is that bug listed? Is it in the list of resolved caveats?
7. Any performance information to be learned?
8. Upgrade directions

Through reading the release notes I have found some very interesting things, Below I have listed a few of them.

– I have to start this off with the Cisco ASA 8.3 release notes. Cisco changed the way NAT was used. This was a major change and Cisco dedicated many pages in the release notes to the new changes. If I wouldn’t have read the release notes the upgrade would have been a disaster.

– There was a version of software for one of the routers we use that didn’t really encrypt the IPSEC packets. Moved on to the next release and made sure that caveat was resolved, it was.

– I was going to upgrade to be able to add a new line card. I found out that the line card wouldn’t run as many line rate ports if there were specific types of line cards in the chassis. We had those line cards, so the number of line rate ports went way down. This was disappointing, but better to find out now then a year later when customers were complaining of slow connections.

– Where to place the OS file to get it to boot? It will tell you. I failed to look and turned my device into a brick. Now I always read where to put the software on the device.

– The Brocade MLXe is not a simple device to upgrade. There are multiple files that need to be copied to the management modules and then again to the line cards. The release notes spell it all out and make it a lot easier then what it seems.

What wild and crazy things have you found by reading the release notes?
What have you broke because you didn’t read the release notes?
Tell us about your stories about reading or not reading the release notes