Some of you may be interested in QSFP+ connections with Cisco Nexus 5500 and Nexus 6000 series switches. The scope of this post is not to present all the different flavors of QSFP+ pluggables or all QSFP+ capable devices but I’ll present some special features that need to be understood and considered when deploying QSFP+ ports with Nexus switches. Let’s first lay out the interesting physical pieces:
- Nexus 6004 switch has 48 fixed QSFP+ ports (4 “slots” x 12 ports = 48 ports) and 4 module slots (LEM = Line-card Expansion Module) for additional 48 QSFP+ ports. Data sheet: http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps12806/ps12807/data_sheet_c78-723667.html
- Nexus 5500 series switches don’t have fixed QSFP+ ports but there is the N55-M4Q GEM (Generic Expansion Module) that has 4 QSFP+ ports. In Nexus 5548P and 5548UP there is one GEM slot available and in Nexus 5596UP and 5596T there are three GEM slots. Data sheet: http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/data_sheet_c78-618603.html
- Nexus 2248PQ is a Fabric Extender (FEX) with 48 SFP+ host ports and 4 QSFP+ fabric ports. Data sheet: http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps10110/data_sheet_c78-507093.html
In general, a QSFP+ port is meant to be used as a 40G Ethernet port but as it is internally based on 4 x 10G lanes they can in some configurations be splitted in four separate 10G ports. For example in Nexus 6004 it is then possible to use a total of 384 ports of 10G connections (96 QSFP+ ports splitted means 4 x 96 = 384 ports in 10G). There is a special command to be entered to enable splitting of the ports. In the Nexus 6004 case the ASICs mandate that three consecutive ports always share the same mode meaning that you cannot enable splitting in just one port, you always need to enable it for the three adjacent ports. (And you need to reboot the module after making the change to reconfigure the ASIC.)
The real difference in deploying 1 x 40G port and 4 x 10G ports is the usual issue with etherchannels: with 4 x 10G bundled in an etherchannel a single flow cannot exceed 10G, while in a real 40G port it is possible to get the full 40G bandwidth for a single flow (because the 10G lane handling happens internally deep down in the physical layer). Therefore there really are different use cases for these configurations.
The interesting part starts here: The QSFP+ GEM for Nexus 5500, the N55-M4Q module, does not work in 1 x 40G port mode at all. It only supports splitting the QSFP+ ports to 10G ports, meaning that the module is really a 16 x 10G module with QSFP+ physical ports onboard.
The Nexus 2248PQ has the same feature: The four QSFP+ fabric ports are really 16 x 10G ports, not 4 x 40G ports.
The corollary of this is that when you connect a 2248PQ FEX to a Nexus 6004 switch you need to first reconfigure the correct switch ports to 4 x 10G mode before you can access the FEX because the QSFP+ ports on Nexus 6004 are in the 1 x 40G mode by default, whereas the only supported mode for 2248PQ is the 4 x 10G mode. This seems to be a FAQ as Cisco TAC has published a separate document about it: Nexus 2248PQ FEX Connection Problem with a Nexus 6000 40G QSFP Port (cisco.com).
Here is a configuration example from the NX-OS Interface Operations Guide (cisco.com):
N6004-TME3(config)# interface breakout slot 2 port 1-3 map 10g-4x N6004-TME3(config)# interface breakout slot 2 port 7-12 map 10g-4x N6004-TME3(config)# poweroff module 2 N6004-TME3(config)# 2013 Jan 2 23:30:40 N6004-TME3 %$ VDC-1 %$ %PFMA-2-MOD_REMOVE: Module 2 removed (Serial number FOC16422P28) N6004-TME3(config)# no poweroff module 2 N6004-TME3(config)# show interface brief | incl Eth2 Eth2/1/1 1 eth access down SFP not inserted 10G(D) -- Eth2/1/2 1 eth access down SFP not inserted 10G(D) -- Eth2/1/3 1 eth access down SFP not inserted 10G(D) -- Eth2/1/4 1 eth access down SFP not inserted 10G(D) -- Eth2/2/1 1 eth access down SFP not inserted 10G(D) -- Eth2/2/2 1 eth access down SFP not inserted 10G(D) -- Eth2/2/3 1 eth access down SFP not inserted 10G(D) -- Eth2/2/4 1 eth access down SFP not inserted 10G(D) -- Eth2/3/1 1 eth access down SFP not inserted 10G(D) -- Eth2/3/2 1 eth access down SFP not inserted 10G(D) -- Eth2/3/3 1 eth access down SFP not inserted 10G(D) -- Eth2/3/4 1 eth access down SFP not inserted 10G(D) -- Eth2/4 1 eth access down SFP not inserted 40G(D) -- Eth2/5 1 eth access down SFP not inserted 40G(D) -- Eth2/6 1 eth access down SFP not inserted 40G(D) -- Eth2/7/1 1 eth access down SFP not inserted 10G(D) -- Eth2/7/2 1 eth access down SFP not inserted 10G(D) -- Eth2/7/3 1 eth access down SFP not inserted 10G(D) -- Eth2/7/4 1 eth access down SFP not inserted 10G(D) -- <output omitted> Eth2/12/1 1 eth access down SFP not inserted 10G(D) -- Eth2/12/2 1 eth access down SFP not inserted 10G(D) -- Eth2/12/3 1 eth access down SFP not inserted 10G(D) -- Eth2/12/4 1 eth access down SFP not inserted 10G(D) -- N6004-TME3(config)#
In thix example the first three QSFP+ ports were reconfigured in 4 x 10G mode, and also the ports 7-12. That left ports Eth2/4, Eth2/5 and Eth2/6 in the original 40G mode. Remember, the ports must be reconfigured in groups of 3.
As a detail, note also the interface designation change: the 40G ports are designated quite normally as EthX/Y but when the 4 x 10G mode is entered the third level of designation is used to tell the “subport” of the physical port.
There is one other speciality as well in Nexus 2248PQ. The fabric ports in 2248PQ don’t have static pinning at all anymore, the only supported mode is the port-channel mode. This means that the 16 fabric ports cannot be configured as dedicated to any host ports but they all participate in all host ports traffic delivery. This makes sense as the oversubscription ratio between the host and fabric ports is 3:1 (48:16) so you cannot guarantee bandwidth for all the host ports anyway (unless leaving some host ports empty [but the static pinning is still not available]). It is different case with the older 2248TP FEX that has 1G host ports and 4 x 10G fabric ports. There the static pinning can make sense in some designs as the oversubscription ratio is almost 1:1.
After setting this, how do you turn the ports back to 40G?
Hi nab, using the equivalent “no” command. Please see the configuration guide for example in http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus6000/sw/interfaces/6x/b_6k_Interfaces_Config_Guide_Release_6x/b_6k_Interfaces_Config_Guide_Release_602N12_chapter_0101.html.