In this post we will take a look at the third option for Inter AS VPN connectivity with Option C. This option is among my favorite SP topics as it gives quite a bit of scale while not being as configuration intensive as Option A or back to back VRFs. We're not looking at all the little variations or options we could used to get end to end reachability. We are looking at the more "common" ways to roll this out. Like the CSC post, we are forming VPNv4 peerings between the RRs, but these are 2 different SPs, so route reflection isn't supported here. We will also advertise local BGP RR and PE loopbacks on the ASBRs and redistribute the learned routes into IGP via a route map that is allowing only /32 prefixes into IGP. This will allow for LDP label allocation of the remote RR and PEs.
The process that is needed to get this configuration up and running is pretty straightforward, there are slight differences/options as we go along but that is for the RIT workbooks and VoDs.
1. eBGP IPv4 unicast peerings with label allocation between the ASBRs in each AS, there maybe several ASBRs.
2. While activating the peering and enabling labeled unicast, advertise the local RR and PE loopback IPs to the remote AS.
3. Redistribute the eBGP learned routes into IGP, filter if desired, or use prefix-list to match /32 routes called from a route-map during redistribution.
4. On IOS XR ASBRs, configure a static /32 route to the connected neighbor pointing out the connected interface. Needed for LDP to allocate a label for the next hop IP. IOS does this by default with the "mpls bgp forwarding" command.
5. From the BGP route reflectors/PE routers, ping and trace route from the local device to the remote RR/PE, if a ping/labeled path is good, the next step is ready.
6. On the RR, PEs can also be configured if needed, configure a VPNv4 eBGP multihop peering between RRs. The VPNv4 peering should form and routes should be propagated between PEs.
7. When traces from CE to CE are done, 2 LSPs should be seen, 1 from ingress PE to the remote ASN RR and then down to the remote PE. This will cause issues for things like QoS.
8. On the RRs, under the VPNv4 AFI, set the "next-hop-unchagned" command to remove the RR from the traffic path.
Now that a process has been laid out, I use a process or order of operations to build with, verify against and troubleshoot from to make sure I didn't miss anything.
To cut down on the amount of configuration here, IGP (IS-IS and OSPF) and LDP are running their respective SP cores. I won't be adding the IGP/LDP/MP-BGP/VRF/PE-CE Routing configuration, there would be a lot. I will add in the associated configuration from the above mentioned bullet points. The traceroute outputs are to prove that IGP/LDP are working. This also assumes that the individual SPs MPLS Cores are ready to go, IGP/LDP/MP-BGP VPNv4 Route Reflected Peerings/VRFs/PE-CE Routing.
IS-IS/BGP ASN 50693
R1#traceroute 192.0.2.15 source lo0 num
Type escape sequence to abort.
Tracing the route to 192.0.2.15
VRF info: (vrf in name/id, vrf out name/id)
1 100.64.18.8 [MPLS: Label 30 Exp 0] 4 msec 3 msec 3 msec
2 100.64.158.15 3 msec * 4 msec
R1#traceroute 192.0.2.21 source lo0 num
Type escape sequence to abort.
Tracing the route to 192.0.2.21
VRF info: (vrf in name/id, vrf out name/id)
1 100.64.18.8 [MPLS: Label 18 Exp 0] 10 msec 6 msec 6 msec
2 100.64.158.15 [MPLS: Label 35 Exp 0] 16 msec 20 msec 20 msec
3 100.64.151.11 20 msec * 6 msec
OSPF/BGP ASN 2
R11#traceroute 192.0.2.24 so lo0 num
Type escape sequence to abort.
Tracing the route to 192.0.2.24
VRF info: (vrf in name/id, vrf out name/id)
1 100.64.110.10 [MPLS: Label 22 Exp 0] 7 msec 6 msec 6 msec
2 100.64.103.13 [MPLS: Label 24008 Exp 0] 15 msec 20 msec 20 msec
3 100.64.134.14 20 msec * 7 msec
The eBGP IPv4 labeled unicast peerings on the ASBRs.
R5 (ASBR)
router bgp 50693
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 100.64.56.6 remote-as 2
!
address-family ipv4
network 192.0.2.1 mask 255.255.255.255
network 192.0.2.8 mask 255.255.255.255
network 192.0.2.15 mask 255.255.255.255
network 192.0.2.17 mask 255.255.255.255
network 192.0.2.21 mask 255.255.255.255
network 192.0.2.25 mask 255.255.255.255
neighbor 100.64.56.6 activate
neighbor 100.64.56.6 send-label
R6 (ASBR)
router bgp 2
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 100.64.56.5 remote-as 50693
neighbor 100.64.166.16 remote-as 50693
!
address-family ipv4
network 192.0.2.11 mask 255.255.255.255
network 192.0.2.22 mask 255.255.255.255
network 192.0.2.23 mask 255.255.255.255
network 192.0.2.24 mask 255.255.255.255
neighbor 100.64.56.5 activate
neighbor 100.64.56.5 send-label
neighbor 100.64.166.16 activate
neighbor 100.64.166.16 send-label
XR6 (ASBR)
router bgp 50693
address-family ipv4 unicast
network 192.0.2.1/32
network 192.0.2.3/32
network 192.0.2.8/32
network 192.0.2.15/32
network 192.0.2.17/32
network 192.0.2.21/32
network 192.0.2.25/32
allocate-label all
!
neighbor 100.64.166.6
remote-as 2
address-family ipv4 labeled-unicast
route-policy RPL_EBGP_PEERINGS in
route-policy RPL_EBGP_PEERINGS out
!
router static
address-family ipv4 unicast
100.64.166.6/32 GigabitEthernet0/0/0/0.166
Now that the configuration is in place, let's see what we've learned.
R5#sh bgp ipv4 unicast summary | b Neighbor
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
100.64.56.6 4 2 1019 1024 166 0 0 15:22:01 6
R5#sh bgp ipv4 unicast | b Network
Network Next Hop Metric LocPrf Weight Path
*> 192.0.2.1/32 100.64.75.7 30 32768 i
*> 192.0.2.3/32 100.64.56.6 0 2 50693 i
*> 192.0.2.8/32 100.64.85.8 20 32768 i
*> 192.0.2.11/32 100.64.56.6 3 0 2 i
*> 192.0.2.15/32 100.64.85.8 30 32768 i
*> 192.0.2.17/32 100.64.175.17 20 32768 i
* 192.0.2.21/32 100.64.56.6 0 2 50693 i
*> 100.64.85.8 40 32768 i
*> 192.0.2.22/32 100.64.56.6 4 0 2 i
*> 192.0.2.23/32 100.64.56.6 3 0 2 i
*> 192.0.2.24/32 100.64.56.6 4 0 2 i
*> 192.0.2.25/32 100.64.75.7 40 32768 i
R6#sh bgp ipv4 unicast summary | b Neighbor
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
100.64.56.5 4 50693 1027 1022 17 0 0 15:25:19 6
100.64.166.16 4 50693 261 288 17 0 0 04:14:49 7
R6#sh bgp ipv4 unicast | b Network
Network Next Hop Metric LocPrf Weight Path
* 192.0.2.1/32 100.64.166.16 40 0 50693 i
*> 100.64.56.5 30 0 50693 i
*> 192.0.2.3/32 100.64.166.16 20 0 50693 i
* 192.0.2.8/32 100.64.166.16 30 0 50693 i
*> 100.64.56.5 20 0 50693 i
*> 192.0.2.11/32 100.64.106.10 3 32768 i
* 192.0.2.15/32 100.64.166.16 30 0 50693 i
*> 100.64.56.5 30 0 50693 i
* 192.0.2.17/32 100.64.166.16 50 0 50693 i
*> 100.64.56.5 20 0 50693 i
*> 192.0.2.21/32 100.64.166.16 20 0 50693 i
* 100.64.56.5 40 0 50693 i
*> 192.0.2.22/32 100.64.106.10 4 32768 i
*> 192.0.2.23/32 100.64.106.10 3 32768 i
*> 192.0.2.24/32 100.64.106.10 4 32768 i
* 192.0.2.25/32 100.64.166.16 50 0 50693 i
*> 100.64.56.5 40 0 50693 i
RP/0/0/CPU0:XR6#sh bgp ipv4 labeled-unicast summary | b Neighbor
Tue Apr 10 12:00:10.744 UTC
Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
100.64.166.6 0 2 289 262 19 0 0 04:15:49 4
RP/0/0/CPU0:XR6#sh bgp ipv4 labeled-unicast | b Network
Tue Apr 10 12:00:29.733 UTC
Network Next Hop Metric LocPrf Weight Path
*> 192.0.2.1/32 100.64.163.3 40 32768 i
*> 192.0.2.3/32 100.64.163.3 20 32768 i
*> 192.0.2.8/32 100.64.163.3 30 32768 i
*> 192.0.2.11/32 100.64.166.6 3 0 2 i
*> 192.0.2.15/32 100.64.116.11 30 32768 i
*> 192.0.2.17/32 100.64.163.3 50 32768 i
*> 192.0.2.21/32 100.64.116.11 20 32768 i
*> 192.0.2.22/32 100.64.166.6 4 0 2 i
*> 192.0.2.23/32 100.64.166.6 3 0 2 i
*> 192.0.2.24/32 100.64.166.6 4 0 2 i
*> 192.0.2.25/32 100.64.163.3 50 32768 i
Processed 11 prefixes, 11 paths
Now that we have learned these routes, we'll configure the BGP to IGP redistribution.
XR6
prefix-set PS_SP_LOOPBACKS
0.0.0.0/0 ge 32
end-set
!
route-policy RPL_SP_LOOPBACKS
if destination in PS_SP_LOOPBACKS then
pass
endif
end-policy
!
router isis 50693
address-family ipv4 unicast
redistribute bgp 50693 level-2 route-policy RPL_SP_LOOPBACKS
R5
router isis 50693
redistribute bgp 50693 route-map RM_BGP_ROUTES
!
ip prefix-list PL_BGP_ROUTES seq 5 permit 0.0.0.0/0 ge 32
!
route-map RM_BGP_ROUTES permit 10
match ip address prefix-list PL_BGP_ROUTES
R6
router ospf 50693
redistribute bgp 2 subnets route-map RM_BGP_ROUTES
!
ip prefix-list PL_BGP_ROUTES seq 5 permit 0.0.0.0/0 ge 32
!
route-map RM_BGP_ROUTES permit 10
match ip address prefix-list PL_BGP_ROUTES
Now that we have this configuration in place, we need to make sure that R8 can talk to XR3.
R8#sh ip route 192.0.2.23
Routing entry for 192.0.2.23/32
Known via "isis", distance 115, metric 10, type level-2
Redistributing via isis 50693
Last update from 100.64.85.5 on GigabitEthernet1.85, 15:27:37 ago
Routing Descriptor Blocks:
* 100.64.85.5, from 192.0.2.5, 15:27:37 ago, via GigabitEthernet1.85
Route metric is 10, traffic share count is 1
We have a route in the RIB, which means we should also have an LDP allocation.
R8#sho mpls forwarding-table 192.0.2.23
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
38 39 192.0.2.23/32 1188 Gi1.85 100.64.85.5
We have a label allocation.
R8#traceroute 192.0.2.23 so lo0 num
Type escape sequence to abort.
Tracing the route to 192.0.2.23
VRF info: (vrf in name/id, vrf out name/id)
1 100.64.85.5 [MPLS: Label 39 Exp 0] 8 msec 7 msec 7 msec
2 100.64.56.6 [MPLS: Label 22 Exp 0] 25 msec 30 msec 30 msec
3 100.64.106.10 [MPLS: Label 18 Exp 0] 20 msec 20 msec 20 msec
4 100.64.103.13 59 msec * 6 msec
We can traceroute to the the remote end via R5.
With the current configuration, we'll have 2 LSPs, 1 from the ingress PE to the remote RR and another from the remote RR to the egress PE.
R1#sh bgp vpnv4 unicast all 192.0.2.4/32
BGP routing table entry for 1:1:192.0.2.4/32, version 173
Paths: (1 available, best #1, table CSC)
Advertised to update-groups:
13
Refresh Epoch 3
2 65004
192.0.2.23 (metric 20) (via default) from 192.0.2.8 (192.0.2.8)
Origin IGP, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
mpls labels in/out nolabel/24015
rx pathid: 0, tx pathid: 0x0
The route update was reflected to us from R8 and R8's next hop is XR3. That means that another LSP from XR3 down to the remote PE is needed.
R2#traceroute 192.0.2.4 source lo0 num
Type escape sequence to abort.
Tracing the route to 192.0.2.4
VRF info: (vrf in name/id, vrf out name/id)
1 100.64.21.1 [AS 50693] 4 msec 4 msec 4 msec
2 100.64.18.8 [MPLS: Labels 38/24015 Exp 0] 11 msec 10 msec 50 msec
3 100.64.85.5 [MPLS: Labels 39/24015 Exp 0] 47 msec 47 msec 47 msec
4 100.64.56.6 [MPLS: Labels 22/24015 Exp 0] 47 msec 47 msec 47 msec
5 100.64.106.10 [MPLS: Labels 18/24015 Exp 0] 47 msec 77 msec 24 msec
6 100.64.103.13 [MPLS: Label 24015 Exp 0] 28 msec 36 msec 35 msec
7 100.64.103.10 [MPLS: Labels 17/57 Exp 0] 36 msec 36 msec 36 msec
8 100.64.114.11 [AS 65004] [MPLS: Label 57 Exp 0] 31 msec 77 msec 145 msec
9 100.64.114.4 [AS 65004] 20 msec * 11 msec
As you can see, traffic flows down to the RR and back to the P router that connects to the PE. This is not the desired result, we can change this with the "next-hop-unchanged" command under the VPNv4 AFI. After applying it, it will take a few moments for the BGP propagation to occur.
R1#sh bgp vpnv4 unicast all 192.0.2.4/32
BGP routing table entry for 1:1:192.0.2.4/32, version 182
Paths: (1 available, best #1, table CSC)
Advertised to update-groups:
13
Refresh Epoch 3
2 65004
192.0.2.11 (metric 20) (via default) from 192.0.2.8 (192.0.2.8)
Origin IGP, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
mpls labels in/out nolabel/57
rx pathid: 0, tx pathid: 0x0
By telling VPNv4 to not change the next hop, we see that R11 advertised the route to XR3 which then advertised it to R8. R1 receives the update from R8, R8 shows us that the route was learned from R11, which is removing the XR3 advertisement thus passing us directly to the remote PE.
R2#traceroute 192.0.2.4 source lo0 num
Type escape sequence to abort.
Tracing the route to 192.0.2.4
VRF info: (vrf in name/id, vrf out name/id)
1 100.64.21.1 [AS 50693] 5 msec 4 msec 4 msec
2 100.64.18.8 [MPLS: Labels 43/57 Exp 0] 9 msec 9 msec 10 msec
3 100.64.85.5 [MPLS: Labels 40/57 Exp 0] 31 msec 31 msec 31 msec
4 100.64.56.6 [MPLS: Labels 21/57 Exp 0] 32 msec 31 msec 31 msec
5 100.64.106.10 [MPLS: Labels 17/57 Exp 0] 31 msec 76 msec 21 msec
6 100.64.114.11 [AS 65004] [MPLS: Label 57 Exp 0] 20 msec 19 msec 19 msec
7 100.64.114.4 [AS 65004] 20 msec
Our traceroute bypasses the RR now and goes directly to the remote PE.
You may have noticed that we were going through R5-R6 and bypassing XR6. I shutdown the eBGP peering on R6s side and that forced the traffic to go through XR6.
R8#traceroute 192.0.2.23 so lo0 num
Type escape sequence to abort.
Tracing the route to 192.0.2.23
VRF info: (vrf in name/id, vrf out name/id)
1 100.64.38.3 [MPLS: Label 42 Exp 0] 10 msec 7 msec 8 msec
2 100.64.163.16 [MPLS: Label 24024 Exp 0] 24 msec 35 msec 30 msec
3 100.64.166.6 [MPLS: Label 22 Exp 0] 31 msec 30 msec 30 msec
4 100.64.106.10 [MPLS: Label 18 Exp 0] 16 msec 19 msec 21 msec
5 100.64.103.13 20 msec * 8 msec
We trace between the RRs and take XR6 on hops 2 and 3.
R2#traceroute 192.0.2.4 source lo0 num
Type escape sequence to abort.
Tracing the route to 192.0.2.4
VRF info: (vrf in name/id, vrf out name/id)
1 100.64.21.1 [AS 50693] 5 msec 3 msec 3 msec
2 100.64.18.8 [MPLS: Labels 43/57 Exp 0] 11 msec 11 msec 11 msec
3 100.64.38.3 [MPLS: Labels 43/57 Exp 0] 22 msec 30 msec 35 msec
4 100.64.163.16 [MPLS: Labels 24025/57 Exp 0] 29 msec 32 msec 32 msec
5 100.64.166.6 [MPLS: Labels 21/57 Exp 0] 198 msec 24 msec 21 msec
6 100.64.106.10 [MPLS: Labels 17/57 Exp 0] 21 msec 21 msec 20 msec
7 100.64.114.11 [AS 65004] [MPLS: Label 57 Exp 0] 19 msec 21 msec 20 msec
8 100.64.114.4 [AS 65004] 19 msec * 10 msec
We confirm that our routing between the CEs is still good as we flow through XR6 with an XR assigned label.
