Apr 29

Don’t forget to change the Registry before upgrading your Cisco 4500-E with a SUP-8E!!!

I recently acquired a brand new Cisco 4506-E with the Supervisor 8-E. The Supervisor came with the latest K9 version software. This was very odd for me, I almost always have to upgrade my new network equipment when I get it. After configuring the switch I relocated it to the wiring closet where it is going to spend the rest of it’s life.

Two days before the scheduled installation, I turned it on to make a couple changes that needed to be made. Once it was booted up, I issued the “show ip int brief” command and the only interfaces that showed up were the tengigabit interfaces on the Supervisor. I did some more digging and found the following error in the log %C4K_CHASSIS-3-BACKPLANESEEPROMREADFAILED. Cisco’s Error Message Decoder stated that the chassis was bad and to return the chassis.

After replacing the chassis, I still had the same error and none of the line cards works. Cisco then sent me a replacement Supervisor 8-E. I installed the replacement Supervisor 8-E, configured it enough to get on the network, then did a TFTP to get the configuration file on it. Knowing it took 5-7 minutes to boot up, I stepped away and when I came back I was able to validate the config loaded (From the console port).

After the chassis was booted up and the configuration was validated I attempted to configure the SSH version and generate the crypto key. The commands where not there. I thought this was very odd because there was only one software version for this platform and the original Supervisor came with the correct version. After checking, this version did not have the SSH feature so I needed to upgrade the switch to Crypto image cat4500es8-universalk9.SPA.03.03.00.XO.151-1.XO.bin. The only difference in the file name is the K9.

I copied the file to the bootflash and changed the boot statement to “boot system flash bootflash:/cat4500es8-universalk9.SPA.03.03.00.XO.151-1.XO.bin”, saved the configuration and reloaded. The switch ignored the boot statement and used the first file in the bootflash. I thought that I had the boot statement wrong, so I tried it without the /, same thing.

After reading the configuration guide I found that the registry needed to be changed to 0X0102. This registry entry tells the switch to read the boot statement. So I entered the following command config-register 0x0102, saved the configuration and reloaded. The switch now booted up with the new image and I was able to configure SSH on the switch.

Like so many other Cisco products, I thought I could simply change the boot statement, save and reload. My assumption failed me on this upgrade. Because of this, I always recommend reading the configuration guides or release notes. Sometimes I get in a hurry and don’t read them. When I do that, I usually get reminded that I need to read the documentation.

Have you found other networking devices that you have to change the config-register to tell it to read the boot statement? If so, please share your experience

Dec 24

Redundancy testing, Do you do it?

In time of an outage, will your systems continue to run and provide services to your customers? Unless you test it, you won’t know.

Testing the redundancy in your network is very important, unfortunately, it usually doesn’t get done. For years I have suggested that we perform annual redundancy testing. Every year the testing gets denied.

Here is a list of things I would like to test.

– Make sure both power supplies (In dual powered equipment) can handle the load by themselves.
– Dual connected servers stay online when 1 of the 2 switches are powered off.
– Move the entire data center to the same UPS, then to the other to make sure it can handle the load.
– Fail any Active/Standby pair to the Standby to make sure the standby configuration and hardware works.

During a switch upgrade I took down multiple services. All of the servers were dual homed to the other switch, but they still went down. Later we found out that the redundant switch port was not configured. Another server never had the network cards setup for active/standby, but the standby network card was connected to the standby switch. Another server didn’t even have the cable run to the second switch. Everybody thought these services were redundant, but it didn’t work.

Many times I don’t have time to perform failover testing when I deploy new equipment. With out testing in the production environment, we don’t know how the other equipment is going to react to a failure. In my network we have many TAP’s and Bypass switches. If the switch on one side of the TAP fails, does the port go down on the other side of the TAP? I hope it goes down. The device on the other side needs to know that the neighbor just went down.

Do you perform annual redundancy testing? If so, what problems have you found and what outages have you avoided?

Please share, maybe your experiences can help other justify performing these tests!!