- Configure different Port types
- Implement Ethernet end Host Mode
- Implement VLANs and Port Channels.
- Implement Pinning and PIN Groups
- Implement Disjoint Layer 2
I used the Cisco Uplink Ethernet configuration example for some of these notes - http://www.cisco.com/en/US/products/ps10276/products_configuration_example09186a0080ae0691.shtml
Configure different port types
There are three different port types used by UCS Fabric Interconnects
- Server Ports - These can only be configured on the fixed port module
- Uplink Ethernet Ports - Network bound traffic is pinned to one of these ports. Can use either fixed or expansion module.
- Uplink Fiber Channel Ports - Network bound fiber channel traffic is pinned to one of these ports. Only an expansion module can be used for this traffic.
UCS LAN Networking
There are a number of pieces in involved in UCS LAN Networking. These pieces are:
CNA on the blade
FEX in the back of the chassis
Fabric Interconnects (FI)
Upstream Switches (in the example below, Nexus 5K)
Connectivity for LAN consists of Server ports and Uplink ports
Server ports - From the perspective of the FI, these ports face the UCS chassis. These ports can only be configured on the fixed port module
Uplink ports - From the perspective of the FI, these ports face the LAN infrastructure (hopefully a Nexus Switch). These ports can be configured on either fixed or expansion module ports.
The 6100 FI’s can operate in either end host mode or switched mode. With either mode, Server side traffic is layer 2 switched and traffic can flow locally. The uplinks are treated differently between EHM and switched mode.
End Host Mode
This is the default uplink mode of the FI. No spanning-tree is running on the uplinks and traffic is pinned to ensure that loops do not form. All uplinks are active for all VLANs. There is also no MAC learning on the uplink ports which can greatly reduce the size of the MAC table. Traffic cannot be locally switched on the uplinks (due to pinning). Cisco generally recommends this mode as best practice.
Switch mode
In switch mode, the uplinks run spanning-tree just as any other switch would. Spanning-tree will block redundant links to ensure that loops are not formed. One use case for switch mode is if UCS is connected to separate, isolated networks
Uplink Options
Individual Uplinks – End Host Mode
Traffic from/to a given server is automatically pinned to a single uplink. This pinning can also be done statically using a LAN pin group. If an uplink fails, there is a dynamic re-pinning process that moves the traffic to a viable uplink. The FI will send out gratuitous ARPs (GARPs) so that attached devices learn the new location of the MAC addresses that were behind the failed uplink.
When port-channels are used for uplinks, availability in the event of an uplink failure is improved. Traffic is pinned to the port-channel which has multiple links associated with it. If one of links fails, connectivity remains on the other link(s). No GARP or dynamic re-pinning is required. Don't forget that only active mode (LACP) port-channels are used on the FI. The upstream ports must etiehr be active or passive.
vPC uplinks are recommended by Cisco as best practice. All of the advantages of availability that port-channeled uplinks provide are included along with the addition of improved availability of the upstream Nexus switches. A failure of one of the vPC Nexus switches will have minimal impact.
Switch Mode
UCS Switch Mode is rarely used. There are a few use cases for it with the most common being for connecting to separate, isolated networks. Here is an example diagram based on a Cisco document that describes this use case.
When End Host mode is used, UCS assumes that all uplinks are connected to the same, common Layer 2 domain. A designated single uplink is selected at random by the FI to receive all broadcast frames. Since there is no Layer 2 connection between these isolated networks, only one uplink will see broadcasts.
When Switched mode is used, broadcast packets are received normally. In the diagram above, vPC’s are used to prevent spanning-tree from blocking ports which increases the amount of bandwidth available. vPC’s are not required in which case spanning-tree would come into play.
LAN Pin Groups
Pinning is a mechanism where Ethernet traffic from a vNIC can be forced to use a specific uplink or port-channel. Pinning can either be static configured or dynamically set by UCS. Pinning is configured in UCS by using LAN pin groups that are in the LAN cloud tab.
Disjoint L2
Supporting disjoint L2 is covered by assigning VLAN to only those uplinks or port-channels that connect to the appropriate L2 networks.
