The goal is to get R2 to learn R4's loopabck address and vice versa.
We can verify R1 is learning routes from both R2 and R4 which means that routing is working correctly.
R1#sh bgp vpnv4 unicast all | b Network
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf CSC)
*> 100.64.21.0/24 0.0.0.0 0 32768 i
*>i 100.64.34.0/24 192.0.2.3 0 100 0 i
*> 192.0.2.2/32 100.64.21.2 0 0 65002 i
*>i 192.0.2.4/32 192.0.2.3 0 100 0 65004 i
The 192.0.2.4/32 routes shows that it was learned from 192.0.2.3 but that is the other side of the VPN. We'll need to prove the routes were learned the right way.
R1#sh bgp vpnv4 unicast all 192.0.2.4/32
BGP routing table entry for 1:1:192.0.2.4/32, version 48
Paths: (1 available, best #1, table CSC)
Advertised to update-groups:
6
Refresh Epoch 2
65004
192.0.2.3 (metric 20) (via default) from 192.0.2.8 (192.0.2.8)
Origin IGP, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
Originator: 192.0.2.3, Cluster list: 192.0.2.8, 192.0.2.26
mpls labels in/out nolabel/17
rx pathid: 0, tx pathid: 0x0
When we expand the route we see that it was learned from 192.0.2.8 which is also present from the cluster list with the same IP address, which happens to be the VPNv4 route reflector. We'll jump to R8 and see what it sees.
R8#sh bgp vpnv4 unicast all 192.0.2.4/32
BGP routing table entry for 1:1:192.0.2.4/32, version 33
Paths: (1 available, best #1, no table)
Advertised to update-groups:
1
Refresh Epoch 1
65004
192.0.2.3 (metric 10) (via default) from 192.0.2.26 (192.0.2.26)
Origin IGP, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:1
Originator: 192.0.2.3, Cluster list: 192.0.2.26
mpls labels in/out nolabel/17
rx pathid: 0, tx pathid: 0x0
After expanding that same route on R8, the next hop is 192.0.2.26 which is also the other side of the VPN. What this means is that there is a VPNv4 peering between R8 and XR6.
R8#sh bgp vpnv4 unicast all summary | b Neighbor
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
192.0.2.1 4 50693 290 289 33 0 0 04:14:45 2
192.0.2.5 4 50693 305 328 33 0 0 04:33:07 0
192.0.2.15 4 50693 647 693 33 0 0 09:42:54 0
192.0.2.21 4 50693 583 685 33 0 0 09:40:06 0
192.0.2.26 4 50693 261 291 33 0 0 04:17:02 2
The reason we need to form an iBGP peering with either the remote route reflector or with the PE router that connects to the customer. This enables customer routes to be propagated between the regional POPs. In order for the routes to get propagated, one of many possible solutions needs to be enabled. Basic redistribution into BGP from IGP and BGP into IGP is one way. In our configuration we went to the ASBR where the PE/RR router loopbacks would be learned via IGP are manually configured as BGP network statements to be advertised into BGP. The opposite needs to happen, BGP learned routes need to be learned by IGP so that the RR/PE routers would learn them.
R5
router isis 50693
redistribute bgp 50693 route-map RM_BGP_ROUTES
mpls ldp autoconfig
!
router bgp 50693
address-family ipv4
network 192.0.2.1 mask 255.255.255.255
network 192.0.2.8 mask 255.255.255.255
network 192.0.2.15 mask 255.255.255.255
!
ip prefix-list PL_BGP_ROUTES seq 5 permit 0.0.0.0/0 ge 32
!
route-map RM_BGP_ROUTES permit 10
match ip address prefix-list PL_BGP_ROUTES
This allows for specific advertisement and learning. This is an important point in the deployment, propagating routes between the Regional SPs is the one thing needed to allow customer routes to be learned.
In order for the routes to be learned between regional SPs, there needs to be a way for those routes to have labels associated between ASNs. To enable this, R5 and R6, R12 and R11 and XR4 and R12 need to exchange labels in their BGP peering capabilities. In IOS, this is with the "send-label" command under the IPv4 unicast peering and XR with the "labeled-unicast" address family.
R5#sh bgp ipv4 unicast labels
Network Next Hop In label/Out label
192.0.2.1/32 100.64.75.7 20/nolabel
192.0.2.3/32 100.64.56.6 nolabel/32
192.0.2.8/32 100.64.85.8 24/nolabel
192.0.2.13/32 100.64.56.6 nolabel/36
192.0.2.15/32 100.64.75.7 22/nolabel
192.0.2.26/32 100.64.56.6 nolabel/27
R5#sh bgp ipv4 unicast neighbors 100.64.56.6 | b Neighbor
ipv4 MPLS Label capability: advertised and received
!
RP/0/0/CPU0:XR4#sh bgp vrf CSC ipv4 labeled-unicast summary | b Neighbor
Sun Apr 8 00:25:32.908 UTC
Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
100.64.142.12 0 50693 560 508 282 0 0 07:56:15 3
100.64.145.15 0 50693 520 556 282 0 0 08:04:49 0
Without this capability, customer routes would not be able to use label switching. On the ASBRs, under the interface facing the core provider, the command "mpls bgp forwarding" is enabled on IOS by default. This helps propagate labels.
IOS XR is a bit unique in this case, manual intervention is required where a static route needs to be applied.
XR4
router static
vrf CSC
address-family ipv4 unicast
100.64.142.12/32 GigabitEthernet0/0/0/0.142
100.64.145.15/32 GigabitEthernet0/0/0/0.145
RP/0/0/CPU0:XR4#show mpls forwarding vrf CSC
Sun Apr 8 00:30:15.909 UTC
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
24011 Pop 100.64.145.15/32[V] \
Gi0/0/0/0.145 100.64.145.15 57589
24026 28 192.0.2.1/32[V] 192.0.2.6 0
24027 20 192.0.2.3/32[V] Gi0/0/0/0.142 100.64.142.12 266919
24029 40 192.0.2.8/32[V] 192.0.2.6 0
24031 26 192.0.2.13/32[V] Gi0/0/0/0.142 100.64.142.12 0
24032 29 192.0.2.15/32[V] 192.0.2.6 0
24035 24 192.0.2.26/32[V] Gi0/0/0/0.142 100.64.142.12 113630
24036 Pop 100.64.142.12/32[V] \
Gi0/0/0/0.142 100.64.142.12 48129900
Without the static routes, labels would not be allocated for the next hop of the customer carrier.
RP/0/0/CPU0:XR4#show route vrf CSC connected
Sun Apr 8 00:32:08.381 UTC
C 100.64.142.0/24 is directly connected, 08:05:22, GigabitEthernet0/0/0/0.142
C 100.64.145.0/24 is directly connected, 08:43:36, GigabitEthernet0/0/0/0.145
LDP won't allocate a label for a non /32 route, so a static /32 host route is needed. IOS does this automatically with the "mpls bgp forwarding" command.
R5
interface GigabitEthernet1.56
mpls bgp forwarding
With all of this configuration and verification, the propagation of routes should now be complete.
R1#sh bgp vpnv4 unicast all | b Network
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf CSC)
*> 100.64.21.0/24 0.0.0.0 0 32768 i
*>i 100.64.34.0/24 192.0.2.3 0 100 0 i
*> 192.0.2.2/32 100.64.21.2 0 0 65002 i
*>i 192.0.2.4/32 192.0.2.3 0 100 0 65004 i
This output proves that R3 has advertised routes to R1 and R1 learned them.
R2#sh ip bgp | b Network
Network Next Hop Metric LocPrf Weight Path
r> 100.64.21.0/24 100.64.21.1 0 0 50693 i
*> 100.64.34.0/24 100.64.21.1 0 50693 i
*> 192.0.2.2/32 0.0.0.0 0 32768 i
*> 192.0.2.4/32 100.64.21.1 0 50693 65004 i
R2 shows in the BGP table.
R2#sh ip route bgp | b Gateway
Gateway of last resort is not set
100.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
B 100.64.34.0/24 [20/0] via 100.64.21.1, 05:19:41
192.0.2.0/32 is subnetted, 2 subnets
B 192.0.2.4 [20/0] via 100.64.21.1, 05:19:41
Those BGP learned routes are then injected into the RIB.
Now we can do a trace and see the results.
R2#traceroute 192.0.2.4 so lo0 num
Type escape sequence to abort.
Tracing the route to 192.0.2.4
VRF info: (vrf in name/id, vrf out name/id)
1 100.64.21.1 [AS 50693] 5 msec 3 msec 4 msec
2 100.64.17.7 [MPLS: Labels 38/17 Exp 0] 14 msec 13 msec 13 msec
3 100.64.75.5 [MPLS: Labels 28/17 Exp 0] 25 msec 36 msec 31 msec
4 100.64.56.6 [MPLS: Labels 32/17 Exp 0] 31 msec 32 msec 31 msec
5 100.64.106.10 [MPLS: Labels 22/24027/17 Exp 0] 34 msec 31 msec 32 msec
6 100.64.103.13 [MPLS: Labels 24008/24027/17 Exp 0] 147 msec 37 msec 85 msec
7 100.64.134.14 [MPLS: Labels 24027/17 Exp 0] 26 msec 19 msec 20 msec
8 100.64.142.12 [MPLS: Labels 20/17 Exp 0] 20 msec 20 msec 31 msec
9 100.64.129.9 [MPLS: Labels 18/17 Exp 0] 31 msec 32 msec 32 msec
10 100.64.34.3 [AS 50693] [MPLS: Label 17 Exp 0] 20 msec 19 msec 22 msec
11 100.64.34.4 [AS 50693] 20 msec * 14 msec
R2 has reachability to R4's loopback, which means that the routes propagated.
Let's walk or hop from R1 to R3 and see how the labels that we see getting allocated are allocated and why.
R1#sh ip route vrf CSC 192.0.2.4
Routing Table: CSC
Routing entry for 192.0.2.4/32
Known via "bgp 50693", distance 200, metric 0
Tag 65004, type internal
Last update from 192.0.2.3 03:33:33 ago
Routing Descriptor Blocks:
* 192.0.2.3 (default), from 192.0.2.8, 03:33:33 ago
Route metric is 0, traffic share count is 1
AS Hops 1
Route tag 65004
MPLS label: 17
MPLS Flags: MPLS Required
We see that MPLS label gets allocated and that 192.0.2.3 is our next hop. This indicates that R3 learned a route from R4 in the VRF and VPNv4 allocated a label for that learned route. In this case label 17 is the VPN label or the label allocated for routes advertised from the customer. You'll notice that label 17 doesn't change in the above traceroute at all, pointing us to the VPN label.
Next we'll see how we get to the next hop of 192.0.2.3, we'll always do an L3 lookup first, which will tell us how to reach the outgoing interface which will point the CEF table to begin forwarding traffic.
R1#sh ip route 192.0.2.3
Routing entry for 192.0.2.3/32
Known via "isis", distance 115, metric 20, type level-2
Redistributing via isis 50693
Last update from 100.64.18.8 on GigabitEthernet1.18, 03:38:20 ago
Routing Descriptor Blocks:
100.64.18.8, from 192.0.2.5, 03:38:20 ago, via GigabitEthernet1.18
Route metric is 20, traffic share count is 1
* 100.64.17.7, from 192.0.2.5, 03:38:20 ago, via GigabitEthernet1.17
Route metric is 20, traffic share count is 1
We have 2 different ways to get to the next hop, but no label is specified, but we know the outgoing interface.
R1#show ip cef 192.0.2.3
192.0.2.3/32
nexthop 100.64.17.7 GigabitEthernet1.17 label 38
nexthop 100.64.18.8 GigabitEthernet1.18 label 25
The labels are 25 and 38. Our second hop shows an outgoing label of 38 which points us to R7. Label 38 is the transport label. Now we'll jump to R7 and repeat the process we did on R1, this happens on a per hop basis until the LSP has been completely traced out.
R7#sh ip route 192.0.2.3
Routing entry for 192.0.2.3/32
Known via "isis", distance 115, metric 10, type level-2
Redistributing via isis 50693
Last update from 100.64.75.5 on GigabitEthernet1.75, 03:47:25 ago
Routing Descriptor Blocks:
* 100.64.75.5, from 192.0.2.5, 03:47:25 ago, via GigabitEthernet1.75
Route metric is 10, traffic share count is 1
R7#sh ip cef 192.0.2.3
192.0.2.3/32
nexthop 100.64.75.5 GigabitEthernet1.75 label 28
Since verification in the customer carrier core is pretty much the same until the ASBR, we'll jump to R5 and continue.
R5#sh bgp vpnv4 unicast all
Since R5 isn't a PE router, there is no VPNv4 learned routes, however, there should be IPv4 learned routes as that is how we are peering with R6.
R5#sh bgp ipv4 unicast
BGP table version is 120, local router ID is 192.0.2.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 192.0.2.1/32 100.64.75.7 30 32768 i
*> 192.0.2.3/32 100.64.56.6 0 2 50693 i
*> 192.0.2.8/32 100.64.85.8 20 32768 i
*> 192.0.2.13/32 100.64.56.6 0 2 50693 i
*> 192.0.2.15/32 100.64.75.7 30 32768 i
*> 192.0.2.26/32 100.64.56.6 0 2 50693 i
We see routes learned both internally and externally, the next thing we need to see is the labels associated to then learned routes.
R5#sh bgp ipv4 unicast labels
Network Next Hop In label/Out label
192.0.2.1/32 100.64.75.7 20/nolabel
192.0.2.3/32 100.64.56.6 nolabel/32
192.0.2.8/32 100.64.85.8 24/nolabel
192.0.2.13/32 100.64.56.6 nolabel/36
192.0.2.15/32 100.64.75.7 22/nolabel
192.0.2.26/32 100.64.56.6 nolabel/27
In our trace from R2 to R4, the third hop uses label 32, which points us to R3, pointing us to R6. We'll jump to there as we're simply tracing the transport labels since we already know the VPN label.
R6#sh mpls forwarding-table labels 32
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
32 24027 192.0.2.3/32[V] 414722 Gi1.106 100.64.106.10
On R6 we receive traffic from R5 with label 32, which is swapped to label 24027 towards R10. But here's the thing, that is to R3, however, we are now in the core carrier. We need to find out the outgoing label for core carrier transport label towards the egress PE router. That router is XR4 or 192.0.2.24.
To find out that we need to use the same recursion process as before.
R6#sh ip route 192.0.2.24
Routing entry for 192.0.2.24/32
Known via "ospf 50693", distance 110, metric 4, type intra area
Last update from 100.64.106.10 on GigabitEthernet1.106, 1d02h ago
Routing Descriptor Blocks:
* 100.64.106.10, from 192.0.2.24, 1d02h ago, via GigabitEthernet1.106
Route metric is 4, traffic share count is 1
We know the destination, and the outgoing interface, we need the label value now.
R6#sh ip cef 192.0.2.24
192.0.2.24/32
nexthop 100.64.106.10 GigabitEthernet1.106 label 22
Now we have the label value, now we need the label stack.
R6#sh mpls forwarding-table labels 26
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
26 22 192.0.2.24/32 0 Gi1.106 100.64.106.10
We have the label stack, it's a repeat of the process again.
The core carrier transport label here is 22, 24027 now becomes the core carrier "VPN" label, but it still remains the transport label position and label 17 is the VPN label.
At this point, its repetitive process, show ip route, show ip cef and show mpls forwarding are going to give the info needed to get to the next hop, in this case the label.
The thing to notice, is that when the trace goes over the core carrier, there is a three label stack. In the customer carrier, it's a 2 label stack.
