Dual-homed FEXes and Switch Profiles

Using dual-homed FEXes with Cisco Nexus 5000 switches presents some new issues for the network admins. One important feature is that the FEX port configuration needs to exist on the both N5ks.

Let’s see the topology example where the FEX is connected to the switches with vPC (= dual-homed FEX):

The server is connected to the port 30 of the N2248TP FEX. Since the FEX is configured as the FEX 100 on both switches the interface is called e100/1/30.

Let’s here assume that the vPC basics and the connection to the FEX have been configured: vPC domain, peer keepalive, peer links, fabric interfaces, port-channel and vPC from the switches to the FEX.

The FEX is owned by the both N5k switches so the server-facing interface configuration needs to exist on both switches. If the configurations don’t match between the switches then the interface will not be operational.

Switch profiles can be used to synchronize the configurations between the vPC peer switches. Here is a crash course on configuring the FEX interfaces with switch profiles.

First the switch profile is created on both N5k switches:

conf t
cfs ipv4 distribute
end
conf sync
switch-profile PROFILE
sync-peers destination 10.5.2.x
end

The switch profile name can be selected freely but it needs to match between the peer switches. Note that the switch profiles currently work only through the mgmt0 interfaces!

You can use the show switch-profile status command (shown later below) to check the status before proceeding.

When using dual-homed FEXes and switch profiles it is highly recommended that the FEXes are pre-provisioned. The reason for this is that the configuration synchronization cannot succeed if the FEX does not exist on the peer switch. By pre-provisioning the FEXes you can be sure that the configuration can be applied even though the FEX happens to be offline (due to disconnected link or whatever) at that point. Check my earlier post about FEX pre-provisioning.

Now you can start using the switch profile:

N5k-1# conf sync
Enter configuration commands, one per line.  End with CNTL/Z.
N5k-1(config-sync)#

Note that the prompt (“config-sync”) now indicate that you are in a different configuration mode.
A switch profile called “PROFILE” was already configured on each of the switches, so it should be used:

N5k-1(config-sync)# switch-profile PROFILE
Switch-Profile started, Profile ID is 1
N5k-1(config-sync-sp)#

Again the command prompt changed to indicate that you are in the switch profile configuration mode.
Now you can enter the commands that you want to deploy to both of the vPC peer switches, for example:

N5k-1(config-sync-sp)# int e100/1/30
N5k-1(config-sync-sp-if)# desc Server LAN
N5k-1(config-sync-sp-if)# swi mode access
N5k-1(config-sync-sp-if)# swi access vlan 50
N5k-1(config-sync-sp-if)# no shut
N5k-1(config-sync-sp-if)# exit
N5k-1(config-sync-sp)#

The commands have not been actually applied yet. You can show all the commands you have entered in the switch profile in this session:

N5k-1(config-sync-sp)# sh switch-profile buffer

switch-profile  : PROFILE
----------------------------------------------------------
Seq-no  Command
----------------------------------------------------------
1       interface Ethernet100/1/30
1.1       description Server LAN
1.2       switchport mode access
1.3       switchport access vlan 50
1.4       no shutdown

N5k-1(config-sync-sp)#

After entering all the configuration you need to first verify the configuration for any mismatches:

N5k-1(config-sync-sp)# verify
Verification Successful
N5k-1(config-sync-sp)#

If the verification is not successful then you need to troubleshoot the situation. One reason for verification to fail is that there is already some “normal config mode” configuration entered that conflict with the configuration you entered in the switch profile. In those situations you should remove the conflicting configuration (in the normal “conf t” mode) and then re-enter the switch profile configuration (with a new verification of course). You can also use the import functionality in the switch profiles but that’s not covered here.

If the verification is successful then you can commit the changes:

N5k-1(config-sync-sp)# commit
Verification successful...
Proceeding to apply configuration. This might take a while depending on amount
of configuration in buffer.
Please avoid other configuration changes during this time.
Commit Successful
N5k-1(config-sync)#

Now the commands have been applied to both of the switches.

The switch profile buffer is automatically emptied after committing the configuration:

N5k-1(config-sync)# sh switch-profile buffer

switch-profile  : PROFILE
----------------------------------------------------------
Seq-no  Command
----------------------------------------------------------

N5k-1(config-sync)#

Use the exit command to leave the config sync mode.

Show switch-profile status can be useful if the situation is somehow unclear with the switch profiles:

N5k-1(config-sync)# sh switch-profile status

switch-profile  : PROFILE
----------------------------------------------------------

Start-time: 496010 usecs after Sat Jan  7 13:38:44 2012
End-time: 647451 usecs after Sat Jan  7 13:38:46 2012

Profile-Revision: 2
Session-type: Commit
Session-subtype: -
Peer-triggered: No
Profile-status: Sync Success

Local information:
----------------
Status: Commit Success
Error(s):

Peer information:
----------------
IP-address: 10.5.2.2
Sync-status: In sync
Status: Commit Success
Error(s):

N5k-1(config-sync)#

In the previous output you can see the local and peer status information that can be useful when troubleshooting switch profile problems.

Show run switch-profile can be used to show which commands have actually been entered with the switch profile (instead of the direct “conf t” configuration).

You can read more about switch profiles in the Cisco Nexus 5000 Series NX-OS System Management Configuration Guide: http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/system_management/513_n1_1/b_Cisco_n5k_system_mgmt_cg_rel_513_n1_1_chapter_011.html

All the current Cisco Nexus 5000 Series Configuration Guides can be found in http://www.cisco.com/en/US/products/ps9670/products_installation_and_configuration_guides_list.html.

Update: There is also a Q-and-A list about configuration synchronization in the Cisco Nexus 5548P documentation: http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/qa_c67-618605_ps9670_Products_Q_and_A_Item.html#wp9000151