IOS XE 15.5
IOS XR 5.3
The topology for this demo:
In this post we will be taking a look at both EVPN and PBB-EVPN. Everything we have previously reviewed has been on the written and lab blueprints, EVPN and PBB-EVPN are only on the written blueprint and per the SPv4 PM, EVPN and PBB EVPN will not be tested on it the lab. I will cover the topics, features, terms and a brief demo to show what it looks like to get PBB-EVPN up and running. The bulk of this post will be covering the terms and an overview of how it all works.
One thing that I noticed as I was going through the documentation, Cisco Live videos, other publicly available white papers, RFCs, etc. It appears that EVPN for IOS XR doesn't exist. It appears that Cisco decided to go forward with PBB which is actually an enhancement to EVPN.
Cisco Live presentations
BRKMPL-2101 - Deploying MPLS-based Layer 2 Virtual Private Networks (2014 San Francisco) - 90 Mins
BRKMPL-2333 - E-VPN & PBB-EVPN: the Next Generation of MPLS-based L2VPN (2015 San Diego) - 90 Mins
RFCs
EVPN for MPLS
https://tools.ietf.org/html/rfc7432
PBB EVPN
https://tools.ietf.org/html/rfc7623
Let's take a look at why EVPN and PBB-EVPN were developed. First of all, DCI or Data Center Interconnect with current L2VPN services like VPLS and H-VPLS don't address the scale, redundancy, load balancing, fast convergence or simplified provisioning that EVPN/PBB offer.
Current VPLS design doesn't allow for All Active per flow redundancy. It uses a PW to exchange label information between PEs/UPEs. If a CE device is connected to 2 different VPLS PEs, only one of the PEs can be used to forward the data, both PEs can learn the MAC info of the CE. The problem that exists, just like in LAN switching, if 2 different MAC addresses are mapped to 1 IP address, how does the switch know which way to forward the data? It will flip-flop back and forth or drop the traffic as a result.
Remember that VPLS is simply providing a "bridge" over the MPLS core and advertising MAC addresses so that the CE knows how to reach other CEs over the VPLS network. If you dual or multi home a CE-PE connection, each PE will learn that MAC address. Multiple ACs can be used on different physical interfaces, they must be physical as subinterfaces borrow the main interface MAC. The issue with this solution is its costly, 2+ ACs back to multiple PEs, $$$$$. That doesn't solve the load balancing issue though, it would require the customer to know how to run ECMP, assuming they know that. The purpose here is to take the burden off the customer and for the SP to solve that issue.
I'm not bashing VPLS by any means, just pointing out some things to keep in mind in case you need all active load balancing, which seems to be the most common request I get when it comes to L2 services.
Ethernet VPN - Next generation solution for E-LAN multipoint services. PEs will run MP-BGP to advertise and learn Customer MAC (C-MAC) over the core, same operationally to L3 VPN. Learning on the PE access circuits or AC via data plane transparently. There is no pseudowire involvement at all. Uses unicast for MP2P or MP2MP depending on the need. Uses Multicast, specifically ingress replication over MP2P tunnel or can use LSM, this is used by the PEs to signal interest in joining the service.
Provider Backbone Bridge EVPN - Takes EVPN to the next level by adding a PBB header as the frame passes through the I and B Components. The EVPN forwarder add the EVPN MPLS Label (VPN Label) and a Control word, which is used to give the VPN Label a "0" value. As the Frame is forwarded out to the MPLS core, a PSN label or the Transport Label with the E-Type of 8847 or MPLS, a source address and destination address, the NH.
So let's hit the components you will see when deploying PBB-EVPN, we'll focus on this aspect, the 802.1ah or Provider backbone bridge MAC-in-MAC forwarding capability. We'll only focus design wise on the E-LAN solution as we only need to know this topic for the written exam.
The components!
Backbone Edge Bridges or BEBs - This is the edge of the MPLS Network.
PBB - 2^24 service instances per B-VLAN, leverages MAC in MAC routing, MAC in MAC means that the B-MAC is used to reach the PE that is advertises that B-MAC. This gives us 16 million I-SIDs per EVI, lots of scalability at the edge. Multiple I-SIDs can be mapped to a single EVI.
I-Component - The I-Component maps service VLAN identifiers to service instance identifiers. It adds a PBB header without a backbone VLAN tag. Learns and forwards C-MACs and maintains a mapping table of the C-MAC to B-MAC. Performs the PBB encapsulation and deencapsulation on the PIP or provider instance port. Similar to the "service instance" in VPLS
B-Component - The B-Component maps I-SIDs to backbond VID B-VIDs and adds a PBB header with a B-Tag. Learns and forwards traffic based on B-MACs learned from other PEs. Pushes and pops the B-VLAN on the CBP or Customer backbone port.
The 802.1ah standard specifies three types of BEBs, B-BEBs, I-BEBs and IB-BEBs. Only IB-BEBs are supported on ASR 9K. IOS XR supports IB-BEB bridge type at the Edge Node. IB-BEBs contain both the I-Component and the B-Component. The bridge selects the B-MAC and inserts the I-SID based on the provider (S) tag and customer (C) tag.
EVPN Instance or EVI - Think of this like the "bridge-domain" in VPLS, it is what binds all of the PEs together to flood data between themselves, EVIs do pretty much the same stuff. The EVI instance number must match among all participating PEs and is configured under the "PBB Core" configuration, ironically enough, pointing into the core of the network. There are several methods to identify traffic as well; Port, VLAN, VLAN Bundling and VLAN Aware Bundling. There can only be one EVI per core bridge.
Ethernet Segment Identifier or ESI - This is a "site" or CE, I also like to think of it like an Attachment Circuit, not 1 for 1 but the fact it is used to connect the PE to the CE. There are several connection styles, Single and Multi Homed Device and Network. This is a uniquely identified 10 byte identifier created under the EVPN configuration.
BGP Routes - 5 new Route Types
[1] Ethernet Auto-Discovery (AD) Route
[2] MAC Advertisement Route
[3] Inclusive Multicast Route
[4] EthernetSegment Route
(5) IP Prefix Advertisement Route
EVPN/PBB-EVPN define a single new BGP NLRI used to carry all EVPN routes
NLRI has a new SAFI (70)
Routes serve control plane purposes, including:
MAC / IP address reachability
MAC mass withdrawal
Split-Horizon label adv.
Aliasing
Multicast endpoint discovery
Redundancy group discovery
Designated forwarder election
BGP Route Attributes
Extended Communities
ESI MPLS Label
ES-Import
MAC Mobility
Default Gateway
Router’s MAC
New BGP extended communities defined
Expand information carried in BGP routes, including:
MAC address moves
C-MAC flush notification
Redundancy mode
MAC / IP bindings of a GW
Split-horizon label encoding
Designated Forwarder Election - Used to determine a designated forwarder in dual-homed or multi-homed devices or networks. The election is performed on a per service basis. The DF filtering function for MHN differs from that for MHD. Directionality is implemented differently depending on the connectivity type. DF filtering for MHN is applied for traffic both ingress and egress on the access-facing Ethernet interfaces. DF filtering for MHD is applied only to traffic that egress the access-facing interfaces. Traffic Types are also very similar to Directionality. DF filtering for MHN impacts both unicast as well as flooded multi-destination traffic. DF filtering for MHD only applies to flooded multi-destination traffic.
The DF idea is basically a way of determining which PE will forward frames received in from the core to the CE if multiple paths to the CE are available. The designated forwarder, depending on the deployment model, mentioned above, is leveraged will determine how the incoming traffic will be forwarded. If traffic is received in on a PE that is not the DF, it is simple discarded.
There is no data plane support for this feature, only a minimal amount of control plane functionality is available. We'll take a look the base PBB-EVPN configuration. Once the configuration is in place, we'll verify the configuration is working and then how the logic is laid out.
From the topology above, XR1, XR2 and XR2 will form a full mesh of iBGP L2VPN EVPN peerings with each other. Both XR and IOS can be used as BGP Route Reflectors, however I chose not to implement that. I will copy in the configuration from all three routers, but only verify on XR3 that it is working correctly.
XR1
l2vpn
bridge group CORE
bridge-domain CORE
pbb core
evpn evi 100
!
!
!
bridge group EVPN
bridge-domain EVPN
interface GigabitEthernet0/0/0/2.100
!
pbb edge i-sid 256 core-bridge CORE
!
router bgp 1
address-family l2vpn evpn
!
neighbor 192.168.1.12
remote-as 1
update-source Loopback0
address-family l2vpn evpn
!
!
neighbor 192.168.1.13
remote-as 1
update-source Loopback0
address-family l2vpn evpn
XR2
l2vpn
bridge group CORE
bridge-domain CORE
pbb core
evpn evi 100
!
!
!
bridge group EVPN
bridge-domain EVPN
interface GigabitEthernet0/0/0/1
!
pbb edge i-sid 256 core-bridge CORE
!
router bgp 1
address-family l2vpn evpn
!
neighbor 192.168.1.11
remote-as 1
update-source Loopback0
address-family l2vpn evpn
!
!
neighbor 192.168.1.13
remote-as 1
update-source Loopback0
address-family l2vpn evpn
XR3
l2vpn
bridge group CORE
bridge-domain CORE
pbb core
evpn evi 100
!
!
!
bridge group EVPN
bridge-domain EVPN
interface GigabitEthernet0/0/0/1
!
pbb edge i-sid 256 core-bridge CORE
!
router bgp 1
address-family l2vpn evpn
!
neighbor 192.168.1.11
remote-as 1
update-source Loopback0
address-family l2vpn evpn
!
!
neighbor 192.168.1.12
remote-as 1
update-source Loopback0
address-family l2vpn evpn
As you can see, the configuration isn't very intensive. However, it is pretty finicky if you don't put things in the right spots.
So let's verify this now:
RP/0/0/CPU0:XR3#sh l2vpn bridge-domain
Sun Jan 1 19:48:03.837 UTC
Legend: pp = Partially Programmed.
Bridge group: CORE, bridge-domain: CORE, id: 0, state: up, ShgId: 0, MSTi: 0
Type: pbb-core
Number of associated pbb-edge BDs: 1
Aging: 300 s, MAC limit: 4000, Action: none, Notification: syslog
Filter MAC addresses: 0
ACs: 0 (0 up), VFIs: 0, PWs: 0 (0 up), PBBs: 1 (1 up)
List of PBBs:
PBB Core, state: up
List of ACs:
List of Access PWs:
List of VFIs:
Bridge group: EVPN, bridge-domain: EVPN, id: 1, state: up, ShgId: 0, MSTi: 0
Type: pbb-edge, I-SID: 256
Aging: 300 s, MAC limit: 4000, Action: none, Notification: syslog
Filter MAC addresses: 0
ACs: 1 (1 up), VFIs: 0, PWs: 0 (0 up), PBBs: 1 (1 up)
List of PBBs:
PBB Edge, state: up, Static MAC addresses: 0
List of ACs:
Gi0/0/0/1, state: up, Static MAC addresses: 0
List of Access PWs:
List of VFIs:
The above output indicates that the 2 bridge domains, the EVPN and the CORE are both up and operational. Both have a MAC limit of 4000 and MAC aging time of 5 minutes.
RP/0/0/CPU0:XR3#sh l2vpn pbb backbone-source-mac
Sun Jan 1 19:49:08.713 UTC
Backbone Source MAC: 0b16.212c.3742
Chassis MAC : 0b16.212c.3742
The source MAC of XR3 is above, that will be the B-MAC that XR1 and XR1 will use as the destination address in the PBB Header.
RP/0/0/CPU0:XR3#show evpn evi detail
Sun Jan 1 19:50:26.267 UTC
EVI Bridge Domain Type
---------- ---------------------------- -------
100 CORE PBB
Unicast Label : 24311
Multicast Label: 24312
Flow Label: N
RD Config: none
RD Auto : (auto) 192.168.1.13:100
RT Auto : 1:100
Route Targets in Use Type
------------------------------ -------
1:100 Import
1:100 Export
You should take note that we didn't configure any VRF like attributes, RD/RT when we configured PBB, the bridge domains or BGP. That is because those values are auto generated based on the EVI value and the lo0 IP address and then EVI value. As you can see, the EVI is using 1:100 as the import and export value.
RP/0/0/CPU0:XR3#show evpn evi inclusive-multicast detail
Sun Jan 1 19:50:04.499 UTC
ISID: 256, Originating IP: 192.168.1.11 100
Nexthop: 192.168.1.11
Label : 24113
Source : Remote
ISID: 256, Originating IP: 192.168.1.12 100
Nexthop: 192.168.1.12
Label : 24211
Source : Remote
ISID: 256, Originating IP: 192.168.1.13 100
Nexthop: ::
Label : 24312
Source : Local
This is really important, in case you missed it eariler, inclusive multicast is critically important in the PBB environment. IM is used by the PEs to signal to the network they would like to join the PBB service. You can also see the label values that were generated by the respective PEs; XR1 - 24113; XR2 - 24211; XR3 - 24312.
RP/0/0/CPU0:XR3#show bgp l2vpn evpn summary
BGP is operating in STANDALONE mode.
Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer
Speaker 27 27 27 27 27 0
Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
192.168.1.11 0 1 401 403 27 0 0 01:39:04 2
192.168.1.12 0 1 399 404 27 0 0 01:39:11 2
We can see that we have 2 BGP L2 VPN EVPN peerings, to XR1 and XR2 respectively. Each advertising 2 prefixes.
RP/0/0/CPU0:XR3#show bgp l2vpn evpn
Sun Jan 1 19:51:26.043 UTC
BGP router identifier 192.168.1.13, local AS number 1
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 0
BGP main routing table version 27
BGP NSR Initial initsync version 7 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs
Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 192.168.1.11:100
*>i[2][0][48][0b16.212c.3742][0]/104
192.168.1.11 100 0 i
*>i[3][256][32][192.168.1.11]/80
192.168.1.11 100 0 i
Route Distinguisher: 192.168.1.12:100
*>i[2][0][48][0b16.212c.3742][0]/104
192.168.1.12 100 0 i
*>i[3][256][32][192.168.1.12]/80
192.168.1.12 100 0 i
Route Distinguisher: 192.168.1.13:100 (default for vrf CORE)
*> [2][0][48][0b16.212c.3742][0]/104
0.0.0.0 0 i
* i 192.168.1.11 100 0 i
* i 192.168.1.12 100 0 i
*>i[3][256][32][192.168.1.11]/80
192.168.1.11 100 0 i
*>i[3][256][32][192.168.1.12]/80
192.168.1.12 100 0 i
*> [3][256][32][192.168.1.13]/80
0.0.0.0 0 i
Processed 8 prefixes, 10 paths
A couple of things to note here. You'll notice that there is some uniqueness to the outputs above, namely the bracketed [2][0][48][MAC] outputs. This is essentially the power behind PBB.
There are 2 very important outputs, the [2] and the [3]. The [2] indicate a MAC advertisement Route and [3] indicates Inclusive Multicast Routes. We see 3 [3] followed by [256] which is the I-SID value. I have not found any documentation explaining the [32] or the reason there are /80 or /104 length prefixes. Those lengths are similar to BGP AD with BGP signaling but not exact.
RP/0/0/CPU0:XR3#show bgp l2vpn evpn bdomain CORE [2][0][48][0b16.212c.3742][0]$
Sun Jan 1 19:53:39.924 UTC
BGP routing table entry for [2][0][48][0b16.212c.3742][0]/104, Route Distinguisher: 192.168.1.13:100
Versions:
Process bRIB/RIB SendTblVer
Speaker 26 26
Local Label: 24311
Last Modified: Jan 1 18:12:01.298 for 01:41:38
Paths: (3 available, best #1)
Advertised to update-groups (with more than one peer):
0.2
Path #1: Received by speaker 0
Advertised to update-groups (with more than one peer):
0.2
Local
0.0.0.0 from 0.0.0.0 (192.168.1.13)
Origin IGP, localpref 100, valid, redistributed, best, group-best, import-candidate, rib-install
Received Path ID 0, Local Path ID 1, version 26
Extended community: RT:1:100
EVPN ESI: 0000.0000.0000.0000.0000
Path #2: Received by speaker 0
Not advertised to any peer
Local
192.168.1.11 (metric 4) from 192.168.1.11 (192.168.1.11)
Received Label 24112
Origin IGP, localpref 100, valid, internal, import-candidate, imported, rib-install
Received Path ID 0, Local Path ID 0, version 0
Extended community: RT:1:100
EVPN ESI: 0000.0000.0000.0000.0000
Source VRF: default, Source Route Distinguisher: 192.168.1.11:100
Path #3: Received by speaker 0
Not advertised to any peer
Local
192.168.1.12 (metric 2) from 192.168.1.12 (192.168.1.12)
Received Label 24210
Origin IGP, localpref 100, valid, internal, import-candidate, imported, rib-install
Received Path ID 0, Local Path ID 0, version 0
Extended community: RT:1:100
EVPN ESI: 0000.0000.0000.0000.0000
Source VRF: default, Source Route Distinguisher: 192.168.1.12:100
This output indicates how XR3 will use the RIB to figure out the path to XR1 and XR2. Both will use LDP to get to their respective locations. Since we are looking at a detailed output of a MAC advertisement route, this is data plane specific, No data plane support currently.
RP/0/0/CPU0:XR3#sh mpls forwarding labels 24311
Sun Jan 1 20:47:09.794 UTC
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
24311 Pop No ID BD=0 PE point2point 0
RP/0/0/CPU0:XR3#sh mpls forwarding labels 24312
Sun Jan 1 20:47:12.864 UTC
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
24312 Pop No ID BD=0 PEIM point2point 0
RP/0/0/CPU0:XR3#show bgp l2vpn evpn bdomain CORE [3][256][32][192.168.1.11]/80
Sun Jan 1 19:54:21.691 UTC
BGP routing table entry for [3][256][32][192.168.1.11]/80, Route Distinguisher: 192.168.1.13:100
Versions:
Process bRIB/RIB SendTblVer
Speaker 27 27
Last Modified: Jan 1 18:12:01.298 for 01:42:20
Paths: (1 available, best #1)
Not advertised to any peer
Path #1: Received by speaker 0
Not advertised to any peer
Local
192.168.1.11 (metric 4) from 192.168.1.11 (192.168.1.11)
Origin IGP, localpref 100, valid, internal, best, group-best, import-candidate, imported
Received Path ID 0, Local Path ID 1, version 27
Extended community: RT:1:100
PMSI: flags 0x00, type 6, label 24113, ID 0xc0a8010b
Source VRF: default, Source Route Distinguisher: 192.168.1.11:100
This is Inclusive Multicast Route specific,
Thanks for stopping by!
Rob Riker, CCIE #50693
