Dual-homed FEXes and Switch Profiles

Using dual-homed FEXes with Cisco Nexus 5000 switches presents some new issues for the network admins. One important feature is that the FEX port configuration needs to exist on the both N5ks.

Let’s see the topology example where the FEX is connected to the switches with vPC (= dual-homed FEX):

The server is connected to the port 30 of the N2248TP FEX. Since the FEX is configured as the FEX 100 on both switches the interface is called e100/1/30.

Let’s here assume that the vPC basics and the connection to the FEX have been configured: vPC domain, peer keepalive, peer links, fabric interfaces, port-channel and vPC from the switches to the FEX.

The FEX is owned by the both N5k switches so the server-facing interface configuration needs to exist on both switches. If the configurations don’t match between the switches then the interface will not be operational.

Switch profiles can be used to synchronize the configurations between the vPC peer switches. Here is a crash course on configuring the FEX interfaces with switch profiles.

First the switch profile is created on both N5k switches:

conf t
cfs ipv4 distribute
end
conf sync
switch-profile PROFILE
sync-peers destination 10.5.2.x
end

The switch profile name can be selected freely but it needs to match between the peer switches. Note that the switch profiles currently work only through the mgmt0 interfaces!

You can use the show switch-profile status command (shown later below) to check the status before proceeding.

When using dual-homed FEXes and switch profiles it is highly recommended that the FEXes are pre-provisioned. The reason for this is that the configuration synchronization cannot succeed if the FEX does not exist on the peer switch. By pre-provisioning the FEXes you can be sure that the configuration can be applied even though the FEX happens to be offline (due to disconnected link or whatever) at that point. Check my earlier post about FEX pre-provisioning.

Now you can start using the switch profile:

N5k-1# conf sync
Enter configuration commands, one per line.  End with CNTL/Z.
N5k-1(config-sync)#

Note that the prompt (“config-sync”) now indicate that you are in a different configuration mode.
A switch profile called “PROFILE” was already configured on each of the switches, so it should be used:

N5k-1(config-sync)# switch-profile PROFILE
Switch-Profile started, Profile ID is 1
N5k-1(config-sync-sp)#

Again the command prompt changed to indicate that you are in the switch profile configuration mode.
Now you can enter the commands that you want to deploy to both of the vPC peer switches, for example:

N5k-1(config-sync-sp)# int e100/1/30
N5k-1(config-sync-sp-if)# desc Server LAN
N5k-1(config-sync-sp-if)# swi mode access
N5k-1(config-sync-sp-if)# swi access vlan 50
N5k-1(config-sync-sp-if)# no shut
N5k-1(config-sync-sp-if)# exit
N5k-1(config-sync-sp)#

The commands have not been actually applied yet. You can show all the commands you have entered in the switch profile in this session:

N5k-1(config-sync-sp)# sh switch-profile buffer

switch-profile  : PROFILE
----------------------------------------------------------
Seq-no  Command
----------------------------------------------------------
1       interface Ethernet100/1/30
1.1       description Server LAN
1.2       switchport mode access
1.3       switchport access vlan 50
1.4       no shutdown

N5k-1(config-sync-sp)#

After entering all the configuration you need to first verify the configuration for any mismatches:

N5k-1(config-sync-sp)# verify
Verification Successful
N5k-1(config-sync-sp)#

If the verification is not successful then you need to troubleshoot the situation. One reason for verification to fail is that there is already some “normal config mode” configuration entered that conflict with the configuration you entered in the switch profile. In those situations you should remove the conflicting configuration (in the normal “conf t” mode) and then re-enter the switch profile configuration (with a new verification of course). You can also use the import functionality in the switch profiles but that’s not covered here.

If the verification is successful then you can commit the changes:

N5k-1(config-sync-sp)# commit
Verification successful...
Proceeding to apply configuration. This might take a while depending on amount
of configuration in buffer.
Please avoid other configuration changes during this time.
Commit Successful
N5k-1(config-sync)#

Now the commands have been applied to both of the switches.

The switch profile buffer is automatically emptied after committing the configuration:

N5k-1(config-sync)# sh switch-profile buffer

switch-profile  : PROFILE
----------------------------------------------------------
Seq-no  Command
----------------------------------------------------------

N5k-1(config-sync)#

Use the exit command to leave the config sync mode.

Show switch-profile status can be useful if the situation is somehow unclear with the switch profiles:

N5k-1(config-sync)# sh switch-profile status

switch-profile  : PROFILE
----------------------------------------------------------

Start-time: 496010 usecs after Sat Jan  7 13:38:44 2012
End-time: 647451 usecs after Sat Jan  7 13:38:46 2012

Profile-Revision: 2
Session-type: Commit
Session-subtype: -
Peer-triggered: No
Profile-status: Sync Success

Local information:
----------------
Status: Commit Success
Error(s):

Peer information:
----------------
IP-address: 10.5.2.2
Sync-status: In sync
Status: Commit Success
Error(s):

N5k-1(config-sync)#

In the previous output you can see the local and peer status information that can be useful when troubleshooting switch profile problems.

Show run switch-profile can be used to show which commands have actually been entered with the switch profile (instead of the direct “conf t” configuration).

You can read more about switch profiles in the Cisco Nexus 5000 Series NX-OS System Management Configuration Guide: http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/system_management/513_n1_1/b_Cisco_n5k_system_mgmt_cg_rel_513_n1_1_chapter_011.html

All the current Cisco Nexus 5000 Series Configuration Guides can be found in http://www.cisco.com/en/US/products/ps9670/products_installation_and_configuration_guides_list.html.

Update: There is also a Q-and-A list about configuration synchronization in the Cisco Nexus 5548P documentation: http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/qa_c67-618605_ps9670_Products_Q_and_A_Item.html#wp9000151

12 Comments

Add a Comment
  1. Did you see any issues with config sync between the devices.. can you please let me know if you have went through issues…

    Thanks in Advance
    -Pa1

  2. I don’t remember seeing any real issues with config sync. I’d still recommend testing using it outside of any production, as with any new feature that affects your day to day operations.

  3. Thanks for this easy tutorial 🙂

  4. Hi Markku,

    Thanks for the guide. Regarding the pre-provisioning…. I currently have dual-homed FEX modules with manual config on both of them to keep them synced up. I just upgraded to a version with config-sync available. Do I just need to configure the sync and it should be good? Any ideas in a scenario when the FEX is connected & configured on both ends already?

    1. Hi Ryan, it is possible to import the interface configs to the switch profile. Look for import command in the config sync mode after configuring the peer configs.

  5. I have been consistently getting linkflaperror in one port of the FEX 2248. How do I test that the fex port itself is not bad ? I do not think there is any problem on server side or cable.

  6. Great instruction, So in order to have configs sync you have to be in the config-sync mode for any changes you want on both switches? It will not make changes to other switch like an HA device? Is there a way to push current configs to match with out our ports setup for switches and not FEX’s? Just want the FEX’ configs to match and ports on FEX’s. I am just trying to confirm I understand how the config-sync works.

    1. Hi PatV! The idea in the switch profiles is that only the configuration that is entered (and committed) in the switch profile (config-sync mode) is deployed in both switches. Ideally you can enter whatever configurations you want in the switch profile, but in practice (as far as I remember) there are some limitations for which commands you can use in the switch profile configurations. It is up to you to select which configurations you want to deploy using switch profiles. I personally used the style that only the interface configs for dual-homed FEXes were deployed in the switch profiles (because they absolutely need to match between the N5k switches for the interfaces to work correctly), and everything else was still configured in the traditional way in config mode. If you already have some configurations present in the devices that you want to move in the switch profiles, look for the import command in config-sync mode.

      Note that if you are using some kind of “SDN-style” configuration deployment solutions, you probably don’t need switch profiles at all as the config deployment system already should take care of configuring all the switches properly.

  7. Also with upgrading firmware is there any issues we should be aware of? Primary take care of the FEX’s and secondary disabled until update or will not update till both are updated?

Leave a Reply to Markku LeiniöCancel reply