Mar 04

Servers keep dropping their network connection, is it the server or the networks fault?

I had just arrived home from work and received a call stating many servers are dropping their network connection. The voice on the other end was very concerned that there was a major problem. I promptly logged into the network and started looking at the network equipment.

The specific servers were connected to Brocade MLXe switches via Multi Chassis Trunking (MCT). If you are not familiar with MCT, it is similar to Cisco’s Virtual Port Channel (VPC). It allows two MLX chassis to act like a single switch from the servers view. LACP is used to create a LAG (Trunk/Etherchannel) to the server.

Upon reviewing my log, I found the following.

Feb 28 16:08:14:W:LACP: 13/32 state changes from LACP_BLOCKED to FORWARD
Feb 28 16:08:14:I:LACP: Port 13/32 mux state transition: not aggregate -> aggregate
Feb 28 16:08:14:I:CLUSTER FSM: Cluster CNS-Cluster (Id: 1), client (RBridge Id: 161) – Remote client CCEP up
Feb 28 16:08:12:I:LACP: Port 13/32 partner port state transition: not aggregate -> aggregate
Feb 28 16:08:12:I:LACP: Port 13/32 rx state transition: defaulted -> current
Feb 28 16:06:43:I:LACP: Port 13/32 rx state transition: current -> expired (reason: timeout)
Feb 28 16:05:29:I:CLUSTER FSM: Cluster CNS-Cluster (Id: 1), client (RBridge Id: 161) – Remote client CCEP down
Feb 28 16:05:29:I:CLUSTER FSM: Cluster CNS-Cluster (Id: 1), client (RBridge Id: 161) – Remote client CCEP up
Feb 28 16:05:26:I:CLUSTER FSM: Cluster CNS-Cluster (Id: 1), client (RBridge Id: 161) – Remote client CCEP down
Feb 28 16:05:26:I:LACP: Port 13/32 mux state transition: aggregate -> not aggregate (reason: peer is out of sync)
Feb 28 16:05:26:W:LACP: 13/32 state changes from FORWARD to DOWN
Feb 28 16:04:36:I:RSTP: VLAN VLAN: 110 Port 12/32 – STP State FORWARDING (EnableFwding)
Feb 28 16:04:36:I:RSTP: VLAN VLAN: 110 Port 12/32 – STP State LEARNING (EnableLearning)
Feb 28 16:03:05:I:RSTP: VLAN VLAN: 110 Port 12/32 – STP State FORWARDING (EnableFwding)
Feb 28 16:03:05:I:RSTP: VLAN VLAN: 110 Port 12/32 – STP State LEARNING (EnableLearning)
Feb 28 14:29:28:I:RSTP: VLAN VLAN: 110 Port 16/6 – STP State FORWARDING (EnableFwding)

One thing I quickly noticed was the lack of interface up/down entries. I finally came to the conclusion that the log entries were a result of the interface switching to UP. The mass outage that I was called about, wasn’t such a mass outage after all. Yes, the log showed many servers going down, but not all at the same time and they were not down now.

After some more conversation with the server team the next day, we came to the conclusion that all of the HP Gen8 servers were having this issue. They would drop their connection, send an SNMP trap, then recover by the time a support engineer could take a look at the server. I was surprised to hear this was going on for many weeks. Knowing that the Brocade MLXe MCT has been stable for a couple of years now, I felt safe suggesting that the server team update the drivers on the server for the NIC. There was an update available and that resolved the issue.

I have had wireless NIC drivers cause connectivity issues in the past, but never on a server.
Can any of you share any stories where a server network card driver caused an issue?

Back to the switch log, where was the port UP/DOWN log?


After some more searching, I figured out that I had the following command applied to the interface “no snmp-server enable traps link-change”. I have this command on every interface that is NOT an uplink interface. This command prevents the switch from sending interface up/down traps to the monitoring system. I do this because I don’t want to receive port up/down traps when the servers do their scheduled reboot, or go down when the server team takes down the server for maintenance.

I removed the “no snmp-server enable traps link-change” command off of a test interface. I then connected my PC to the port, then disconnected it. I received the following log entries.

Feb 26 09:39:40:I:System: Interface ethernet 15/35, state down – link down
Feb 26 09:39:02:I:RSTP: VLAN VLAN: 4 Port 15/35 – STP State FORWARDING (EnableFwding)
Feb 26 09:39:02:I:RSTP: VLAN VLAN: 4 Port 15/35 – STP State LEARNING (EnableLearning)
Feb 26 09:39:02:I:System: Interface ethernet 15/35, state up

I found that the “no snmp-server enable traps link-change” command is preventing the link up/down log entries. After talking to Brocade support, this is a software defect in 5.2d.

Have you run into this software defect on the MLX?
Have you run into this driver issue on the HP Gen8 servers? If so, what switches were you using?

No account needed to post a reply, find the Reply button below and add your comment!!!

Feb 11

Brocade ICX-6430-C12 Compact Switch Review

Over the past few years I have received multiple requests for temporary computer labs in conference rooms. Sometimes I receive a request for a switch to be placed in a conference room for a major systems change and the requester want all of the support staff to be in the same room hardwired specific subnet. An IP phone is always on the request.

I usually struggle to find an extra POE switch that is available, I usually find a 48 port Cisco 3560. Then I place it on top of the conference table so all of the computers can easily reach it. Sometimes I am stuck with a Cisco 2950, no POE, then struggle to find a power brick for the IP Phone and have to make a crossover cable for the uplink.

I was excited to hear about the Compact Switches that do not have fans. I started looking at different vendors with compact switches. I wanted to get a 10/100/1000 switch with 10-12 ports. Out of the compact switches that I looked at, the Brocade ICX-6430-12C was the only one I found with 12 ports of gigabit ethernet. I have used Brocade FastIrons and NetIrons in the data center so I took a close look at the ICX-6430-C12.

Here are the requirements I had for my compact switch to fulfill this need.
– 10 to 12 Gigabit ports
– Fully Managed, SSH, SNMP etc..
– Layer 2 only, Just pass the VLANs through
– Mark QOS via ACL (Mark DSCP Value), Trust DSCP on the uplink
– Provide PoE (for an IP phone and maybe an Access Point)
– No fan, low to no noise
– SFP port (a like to have)

The ICX-6430-C12 has 12 10/100/1000 ports with two Gigabit uplink ports and 2 available SFP ports. Brocade states it is a 12 port switch, but it is really a 16 port switch. The switch can power up to 4 802.3AF devices OR 2 802.3AT devices and 2 802.3AF devices. The switch doesn’t make any noise, no humm or Buzz. The configuration is very simple. TACACS+ works with my Cisco ACS server for authentication.

The switch has a few different mounting kits available. There is a 19 inch rack mount kit, rubber feet and my favorite, the magnet. You can read more about them in the FastIron_08001_ICX6430-C_InstallGuide. I was surprised to see the magnet mounting kit. It would work great if there was a steel wall close. It’s easy to use and no tools are needed to mount it to the wall. Unfortunately, very few of the locations I would use this have steel walls to mount it on.

I was easily able to power my 6900 and 7900 Series Cisco IP Phones with this swtich. I was able to power a Cisco 1252 and 3500 series wireless access points. The Cisco 1252 access point requires PoE+.

QOS marking and queuing worked well. I had very good voice quality while the switch was under heavy load.

Overall, I’m very happy with this product and I do highly recommend the switch to you. The gigabit port density for the cost was a very good value for me.

Brocade ICX-6430-12C Compact Switch

Brocade ICX-6430-12C Compact Switch


Have you used any of the compact switches?
If so, how are you using it?
Please share your experiences with the switch or what you would like to see different?

An account is not needed to comment, just hit reply below to leave your comment!!