Majornetwork

Gateway of last resort is not set

Private VLANs on Nexus 5000 Series

So it was a usual day in the wonderful world of networking and projects. Then I was suddenly lost in the Private VLANs. You know, the feature that has existed a long time in Cisco switches but still there are some known and unknown limitations in using it. Let’s talk about PVLANs.

First the basics (use configuration guides to support you if you don’t understand what I’m talking about):

  • Primary VLAN: That’s the VLAN that is the “umbrella” around the secondary VLANs.
  • Secondary VLAN: One primary VLAN can have one or more secondary VLANs. Secondary VLAN can be either isolated VLAN (only at most one for each primary VLAN) or community VLAN (zero or more can exist for each primary VLAN).
  • Hosts connected to PVLAN host ports that are configured with isolated VLANs (or PVLAN isolated trunk ports) can only talk to the promiscuous ports. Promiscuous ports are the ports that connect to the common network elements in that VLAN like routers.
  • Hosts connected to PVLAN host ports that are configured with community VLANs can talk to the promiscuous ports and to the other community ports in the same community VLAN. They cannot talk to the other community VLANs.
  • Hosts connected to the promiscuous ports can talk to every port in the area of that particular primary VLAN.

These are kind of hard to explain in very exact way but I hope you got the points:

  • Isolated means that hosts cannot talk to each other, promiscuous port is always an exception
  • Community means that there is normal communication within that community but no access to other communities, and promiscuous port is again the exception that can talk to everyone (and back).

If you want to understand why these primary and secondary VLANs are needed you can think the frame flow like this:

  • The primary VLAN is used to direct traffic from the promiscuous ports to the secondary ports
  • The secondary VLAN is used to direct traffic from the isolated ports to the promiscuous ports, and from the community ports to the other identical community ports and to the promiscuous ports.

At this point we are ready for some configuration.

Let’s say I have VLAN 100 as the primary VLAN and VLAN 101 as the isolated (secondary) VLAN:

n5k-1(config)# feature private-vlan
n5k-1(config)# vlan 101
n5k-1(config-vlan)# private-vlan isolated
n5k-1(config-vlan)# exit
n5k-1(config)# vlan 100
n5k-1(config-vlan)# private-vlan primary
n5k-1(config-vlan)# private-vlan association 101
n5k-1(config-vlan)# exit
n5k-1(config)# sh vlan private-vlan
Primary  Secondary  Type             Ports
-------  ---------  ---------------  -------------------------------------------
100      101        isolated

Now, I have a router that I want to be routing this VLAN to the outside world, and I want to connect it with a PVLAN promiscuous trunk. Note that PVLAN promiscuous port is not supported on FEX so a physical port on Nexus 5000 is needed:

n5k-1(config)# int e1/2
n5k-1(config-if)# switchport mode private-vlan trunk promiscuous
n5k-1(config-if)# switchport private-vlan mapping trunk 100 101

Now the switch knows that the traffic going to the router needs to be mangled specially. It will adjust the outgoing VLAN tags so that the router only sees packets in VLAN 100. (You remember, traffic coming from the secondary VLAN ports will be using VLAN 101, that’s how the switches know how to keep the traffic from going in wrong places.)

I want to connect a server on the FEX with a PVLAN isolated trunk (because I want it as a trunk port on the server for whatever reason, to save interfaces or so):

n5k-1(config-if)# int e100/1/9
n5k-1(config-if)# switchport mode private-vlan trunk promiscuous
ERROR: requested config change not allowed

That was just a distraction to show you that promiscuous port is really not supported on the FEX! Let’s continue with our real configuration, PVLAN isolated trunk:

n5k-1(config-if)# switchport mode private-vlan trunk secondary
n5k-1(config-if)# switchport private-vlan association trunk 100 101

“Secondary” keyword kind of differentiates it from the promiscuous trunk that was used for the router port. The command is accepted also without it, it is then added automatically in the configuration.

The association statement will advise the switch to change the VLAN tags accordingly: the server will only see the secondary VLAN tag entering the NIC. VLAN 101 is then configured on the server side on this trunk.

Let’s see what the switch says about the port status:

n5k-1(config-if)# sh int e100/1/9 switchport
Name: Ethernet100/1/9
  Switchport: Enabled
  Switchport Monitor: Not enabled
  Operational Mode: Private-vlan trunk secondary
  Access Mode VLAN: 1 (default)
  Trunking Native Mode VLAN: 1 (default)
  Trunking VLANs Allowed: 1-3967,4048-4093
  Voice VLAN: none
  Extended Trust State : not trusted [COS = 0]
  Administrative private-vlan primary host-association: none
  Administrative private-vlan secondary host-association: none
  Administrative private-vlan primary mapping: none
  Administrative private-vlan secondary mapping: none
  Administrative private-vlan trunk native VLAN: 1
  Administrative private-vlan trunk encapsulation: dot1q
  Administrative private-vlan trunk normal VLANs: none
  Administrative private-vlan trunk private VLANs: (100,101)
  Operational private-vlan: (100,101)
  Unknown unicast blocked: disabled
  Unknown multicast blocked: disabled

By the way, don’t bother checking the PVLAN trunk ports with “sh int trunk” command, it does not really help you at all regarding the PVLAN situation.

I want also trunk a normal VLAN (30) to the server:

n5k-1(config-if)# switchport private-vlan trunk allowed vlan 30
n5k-1(config-if)# sh run int e100/1/9
...
version 5.1(3)N2(1a)

interface Ethernet100/1/9
  switchport mode private-vlan trunk secondary
  switchport private-vlan trunk allowed vlan 30
  switchport private-vlan association trunk 100 101

n5k-1(config-if)# sh int e100/1/9 switchport
Name: Ethernet100/1/9
  Switchport: Enabled
  Switchport Monitor: Not enabled
  Operational Mode: Private-vlan trunk secondary
  Access Mode VLAN: 1 (default)
  Trunking Native Mode VLAN: 1 (default)
  Trunking VLANs Allowed: 1-3967,4048-4093
  Voice VLAN: none
  Extended Trust State : not trusted [COS = 0]
  Administrative private-vlan primary host-association: none
  Administrative private-vlan secondary host-association: none
  Administrative private-vlan primary mapping: none
  Administrative private-vlan secondary mapping: none
  Administrative private-vlan trunk native VLAN: 1
  Administrative private-vlan trunk encapsulation: dot1q
  Administrative private-vlan trunk normal VLANs: 30
  Administrative private-vlan trunk private VLANs: (100,101)
  Operational private-vlan: (100,101)
  Unknown unicast blocked: disabled
  Unknown multicast blocked: disabled

Note that the primary or secondary VLAN was not needed to be added in the “switchport private-vlan trunk allowed vlan” statement at all. This is kind of logical because we explicitly configured the PVLAN isolated trunk on the interface so why would we then want to prune the isolated VLAN from the trunk? It goes there automatically. If you want to prune it then just don’t configure the PVLAN association at all!

If I would need more PVLANs trunked to the same server (because it is a member of several PVLANs), I can do that as well:

n5k-1(config-if)# vlan 111
n5k-1(config-vlan)# private-vlan isolated
n5k-1(config-vlan)# vlan 110
n5k-1(config-vlan)# private-vlan primary
n5k-1(config-vlan)# private-vlan association 111
n5k-1(config-vlan)# int e100/1/9
n5k-1(config-if)# switchport private-vlan association trunk 110 111
n5k-1(config-if)# sh run int e100/1/9
...
interface Ethernet100/1/9
  switchport mode private-vlan trunk secondary
  switchport private-vlan trunk allowed vlan 30
  switchport private-vlan association trunk 100 101
  switchport private-vlan association trunk 110 111

n5k-1(config-if)# sh int e100/1/9 switchport
Name: Ethernet100/1/9
  Switchport: Enabled
  Switchport Monitor: Not enabled
  Operational Mode: Private-vlan trunk secondary
  Access Mode VLAN: 1 (default)
  Trunking Native Mode VLAN: 1 (default)
  Trunking VLANs Allowed: 1-3967,4048-4093
  Voice VLAN: none
  Extended Trust State : not trusted [COS = 0]
  Administrative private-vlan primary host-association: none
  Administrative private-vlan secondary host-association: none
  Administrative private-vlan primary mapping: none
  Administrative private-vlan secondary mapping: none
  Administrative private-vlan trunk native VLAN: 1
  Administrative private-vlan trunk encapsulation: dot1q
  Administrative private-vlan trunk normal VLANs: 30
  Administrative private-vlan trunk private VLANs: (100,101) (110,111)
  Operational private-vlan: (100,101) (110,111)
  Unknown unicast blocked: disabled
  Unknown multicast blocked: disabled

Let’s return to our PVLAN promiscuous trunk again (port e1/2):

n5k-1(config-if)# sh run int e1/2
...
interface Ethernet1/2
  switchport mode private-vlan trunk promiscuous
  switchport private-vlan mapping trunk 100 101

I just added the VLANs 110 and 111 above, so I’ll configure the router to route those as well, and configure the switch port accordingly:

n5k-1(config-if)# int e1/2
n5k-1(config-if)# switchport private-vlan mapping trunk ?
  <1-3967,4048-4093>  Primary private VLAN

n5k-1(config-if)# switchport private-vlan mapping trunk 110 ?
  <1-3967,4048-4093>  Secondary VLAN IDs
  add                 Add a VLAN to private VLAN list
  remove              Remove a VLAN from private VLAN list

n5k-1(config-if)# switchport private-vlan mapping trunk 110 111
n5k-1(config-if)# sh run int e1/2
...
interface Ethernet1/2
  switchport mode private-vlan trunk promiscuous
  switchport private-vlan mapping trunk 100 101
  switchport private-vlan mapping trunk 110 111

Note that there is “add/remove” syntax as well. In this version of NX-OS (5.1(3)N2(1a)) there is however no difference in using the “add” keyword or not. Let me show that:

n5k-1(config-if)# vlan 102
n5k-1(config-vlan)# private-vlan community
n5k-1(config-vlan)# vlan 100
n5k-1(config-vlan)# private-vlan association 102
n5k-1(config-vlan)# sh vlan id 100 private-vlan
Primary  Secondary  Type             Ports
-------  ---------  ---------------  -------------------------------------------
100      101        isolated         Eth1/2, Eth100/1/9
100      102        community        
n5k-1(config-vlan)# int e1/2
n5k-1(config-if)# switchport private-vlan mapping trunk 100 102
n5k-1(config-if)# sh run int e1/2
...
interface Ethernet1/2
  switchport mode private-vlan trunk promiscuous
  switchport private-vlan mapping trunk 100 101-102
  switchport private-vlan mapping trunk 110 111

So I didn’t use “add” but it was added, it didn’t replace VLAN 101 there. I’ll remove 102 and try with “add”:

n5k-1(config-if)# switchport private-vlan mapping trunk 100 remove 102
n5k-1(config-if)# sh run int e1/2
...
interface Ethernet1/2
  switchport mode private-vlan trunk promiscuous
  switchport private-vlan mapping trunk 100 101
  switchport private-vlan mapping trunk 110 111

n5k-1(config-if)# switchport private-vlan mapping trunk 100 add 102
n5k-1(config-if)# sh run int e1/2
...
interface Ethernet1/2
  switchport mode private-vlan trunk promiscuous
  switchport private-vlan mapping trunk 100 101-102
  switchport private-vlan mapping trunk 110 111

The effect was exactly the same. Note that this behaviour might change, and the configuration guide for 5.1(3)N2(1) don’t mention “add/remove” syntax for promiscuous trunks at all.

Now I have covered the cases of

  • PVLAN promiscuous trunk
  • PVLAN isolated trunk.

Let’s see how goes the other non-trunk PVLAN cases: PVLAN promiscuous port and PVLAN host port.

PVLAN promiscuous port is used for connecting common-access devices that are not using trunk. An example is a monitoring server that needs access to all hosts in the segment. As mentioned above, PVLAN promiscuous ports cannot be FEX ports so a Nexus 5000 port must be used.

n5k-1(config-if)# int e1/3
n5k-1(config-if)# switchport mode private-vlan promiscuous
n5k-1(config-if)# switchport private-vlan mapping 100 101
n5k-1(config-if)# sh run int e1/3
...
interface Ethernet1/3
  switchport mode private-vlan promiscuous
  switchport private-vlan mapping 100 101

n5k-1(config-if)# sh int e1/3 switchport
Name: Ethernet1/3
  Switchport: Enabled
  Switchport Monitor: Not enabled
  Operational Mode: Private-vlan promiscuous
  Access Mode VLAN: 100 (VLAN0100)
  Trunking Native Mode VLAN: 1 (default)
  Trunking VLANs Allowed: 1-3967,4048-4093
  Voice VLAN: none
  Extended Trust State : not trusted [COS = 0]
  Administrative private-vlan primary host-association: none
  Administrative private-vlan secondary host-association: none
  Administrative private-vlan primary mapping: 100
  Administrative private-vlan secondary mapping: 101
  Administrative private-vlan trunk native VLAN: 1
  Administrative private-vlan trunk encapsulation: dot1q
  Administrative private-vlan trunk normal VLANs: none
  Administrative private-vlan trunk private VLANs: none
  Operational private-vlan: (100,101)
  Unknown unicast blocked: disabled
  Unknown multicast blocked: disabled

Since the port was promiscuous the mapping statement is needed to instruct the switch with VLAN mangling. For the host (the monitoring server) this does not look anything special since it is using normal untagged traffic, just like an access port (but this is really a PVLAN promiscuous port). Access VLAN is interestingly shown here as well, that kind of confirms our principles of the forwarding directions (primary VLAN (100) is used to carry traffic from the promiscuous ports).

The “add/remove” syntax is similar to the PVLAN promiscuous trunk example above.

The last case is PVLAN host port. Since it is not a promiscuous port it can reside in the FEX as well:

n5k-1(config-if)# int e100/1/10
n5k-1(config-if)# switchport mode private-vlan host
n5k-1(config-if)# switchport private-vlan host-association 100 101
n5k-1(config-if)# sh run int e100/1/10
...
interface Ethernet100/1/10
  switchport mode private-vlan host
  switchport private-vlan host-association 100 101

n5k-1(config-if)# sh int e100/1/10 switchport
Name: Ethernet100/1/10
  Switchport: Enabled
  Switchport Monitor: Not enabled
  Operational Mode: Private-vlan host
  Access Mode VLAN: 101 (VLAN0101)
  Trunking Native Mode VLAN: 1 (default)
  Trunking VLANs Allowed: 1-3967,4048-4093
  Voice VLAN: none
  Extended Trust State : not trusted [COS = 0]
  Administrative private-vlan primary host-association: 100
  Administrative private-vlan secondary host-association: 101
  Administrative private-vlan primary mapping: none
  Administrative private-vlan secondary mapping: none
  Administrative private-vlan trunk native VLAN: 1
  Administrative private-vlan trunk encapsulation: dot1q
  Administrative private-vlan trunk normal VLANs: none
  Administrative private-vlan trunk private VLANs: none
  Operational private-vlan: (100,101)
  Unknown unicast blocked: disabled
  Unknown multicast blocked: disabled

It looks like a normal access port for the host but the host-association statement again tells the switch what kind of traffic to expect from this port and how to forward it. Now the access VLAN is shown as 101 because that is the VLAN that the switch internally uses for forwarding the frames from this access port.

There you go. I’ll probably write later about other PVLAN trunks, let’s see.

Nexus 5000 Series Configuration Guides: http://www.cisco.com/en/US/products/ps9670/products_installation_and_configuration_guides_list.html

1 Comment

Add a Comment
  1. The association statement will advise the switch to change the VLAN tags accordingly: the server will only see the secondary VLAN tag entering the NIC. VLAN 101 is then configured on the server side on this trunk.

    Thanks, this helped me understand how trunking works with PVLANs.

Leave a Reply

Majornetwork.net © Markku Leiniö 2011-2017 Frontier Theme