MPLS Inter AS L3 VPN Carrier Supporting Carrier or CSC

In this post we will take a look at CSC or Carrier Supporting Carrier. This is a really involved configuration and will take sometime to break down all the individual steps in order to get our Customer sites to be able to talk to each other.

First, lets cover what CSC is. The really basic part is we're connecting regional Tier 2 POPs together via a Tier 1 Backbone provider. That's pretty much it, a bit oversimplified but if you have never seen, heard of or worked with CSC, it's one of the more advanced and configuration intensive IAS capabilities out there.

So what is really going, I know what your thinking, jee.... connecting Tier 2 POPs together via a Tier 1 backbone, jee.... way to make it sound not complicated. That's kind of the point. There are 3 individual MPLS L3 VPN providers in our setup here, 2 of which are in the same BGP AS (Tier 2 POPs) and the 3rd is the Tier 1 Backbone. The Backbone provider treats both Tier 2 POPs as CE sites, the Tier 2 POPs treat their customer as regular CEs. Pretty much a hierarchical L3 VPN at the end of the day. Sort of like the MPLS Superbackbone with OSPF.

Out Topology:

The ASN 50693 is our T1 backbone, R1, R2, R3, XR1, XR2 and XR3.

The ASN 1000 is 2 Regions, R4, R5 and XR4; R13, R14 and R16.

R7 and R15 are our customer sites that need to reach each other.....

First and foremost, IGP/LDP, VPNv4 PE-PE iBGP peerings, and VRFs need to be configured in each site. In my first rollout, I leveraged R1, R5 and R16 as RR for BGP. I kept this design until the very end, we'll discuss this later. 

Once you have Intra AS L3 VPN setup and ready to go, the Inter AS setup comes. We'll start with the Backbone configuration. ****For whatever reason, haven't figured out why, IOS XRv 5.3.0 would not pass VPNv4 traffic down to the CSC_CEs.**** I ended up routing around XRv and using all IOS XE (CSR1000v) to get this up and running. I'll add in the applicable configuration for XRv. 

On the connection from XR2 to R13 I placed g0/0/0/0.1312 into VRF CSC.

XR2, XR3

vrf CSC

 address-family ipv4 unicast

  import route-target

   3:3

  !

  export route-target

   3:3

XR2

int g0/0/0/0.1312

vrf CSC

ipv4 address 13.12.0.12 255.255.255.0

 ipv6 address 2001:13:12::12/64

XR3

interface GigabitEthernet0/0/0/0.134

 vrf CSC

 ipv4 address 13.4.0.13/24

 ipv6 address 2001:13:4::13/64

Now on to the BGP configuration

XR2

vrf CSC

  rd 3:3

  address-family ipv4 unicast

  !

  neighbor 13.12.0.13

   remote-as 1000

   address-family ipv4 labeled-unicast

    route-policy PASS in

    route-policy PASS out

    as-override

The RD is placed under the VRF configuration under BGP

Labeled-unicast is used to advertise BGP labels

AS-override is to remove the AS received from the Customer Carrier and replace it with the Core Carrier AS

XR3

vrf CSC

  rd 3:3

  address-family ipv4 unicast

  !

  neighbor 13.4.0.4

   remote-as 1000

   address-family ipv4 labeled-unicast

    route-policy PASS in

    route-policy PASS out

    as-override

Now we need to make sure that any routes that are learned in the IPv4 Unicast AFI get labels associated to them

XR2 and XR3

router bgp 50693

 address-family ipv4 unicast

  allocate-label all

Allocate label all is used to allocate labels to global BGP routes

Now we have one more issue, the prefix that XR2 and XR3 peer with is /24 to the CSC_CE. No labels will be allocated for a /24, we need to make this a /32.

XR2

show mpls forwarding-table

120012 Aggregate   13.12.0.0/24[V]    CSC                          0

XR2 

router static

 vrf CSC

  address-family ipv4 unicast

   13.12.0.13/32 GigabitEthernet0/0/0/0.1312

XR2

show mpls forwarding-table

120003 Pop         13.12.0.13/32[V]   Gi0/0/0/0.1312 13.12.0.13      6480

You can see how now there is a "Pop" now. That's how we'll get label allocation for this connected prefix that is a /24.

XR3

router static

 vrf CSC

  address-family ipv4 unicast

   13.4.0.4/32 GigabitEthernet0/0/0/0.134

Like I mentioned, XRv wasn't advertising routes to the Customer Carrier, so I modified the configuration and removed XR from the demo. The above configuration is still applicable, use it if you can.

IOS Based setup from here out.

R3 

I created the VRF CSC on R3 and R1, then under BGP I peered with R4 to R3 and R1 to R13. 

R3

int g1.34

vrf for CSC

en do 34

ip add 3.4.0.3 255.255.255.0

R4

int g1.34

en do 34

ip add 3.4.0.4 255.255.255.0

R1

int g1.13

en do 13

vrf for CSC

ip add 1.13.0.1 255.255.255.0

R13

int g1.13

en do 13

ip add 1.13.0.13 255.255.255.0

Now we can configure the BGP IPv4 Unicast peerings and advertise BGP Labels. IGP+LDP is applicable here but not demo'd. It would have been easier but who likes easy?

R1

router bgp 50693

 no bgp default ipv4-unicast

 address-family ipv4 vrf CSC

  neighbor 1.13.0.13 remote-as 1000

  neighbor 1.13.0.13 activate

  neighbor 1.13.0.13 as-override

  neighbor 1.13.0.13 send-label

Send-label is the equivalent to labeled-unicast in XR, it advertises BGP labels. Both R1 and R13 need this configured. AS Override is configured since both Tier 2 POPs are in the same ASN.

R13

router bgp 1000

no bgp default ipv4

neighbor 1.13.0.1 remote-as 50693

address-family ipv4

  redistribute ospf 1

  neighbor 1.13.0.1 activate

  neighbor 1.13.0.1 allowas-in

  neighbor 1.13.0.1 send-label

R3

router bgp 50693

 bgp log-neighbor-changes

 no bgp default ipv4-unicast

address-family ipv4 vrf CSC

  neighbor 3.4.0.4 remote-as 1000

  neighbor 3.4.0.4 activate

  neighbor 3.4.0.4 as-override

  neighbor 3.4.0.4 send-label

R4

router bgp 1000

 bgp log-neighbor-changes

 no bgp default ipv4-unicast

address-family ipv4

  neighbor 3.4.0.3 activate

  neighbor 3.4.0.3 allowas-in

  neighbor 3.4.0.3 send-label

Now routes are propagating via BGP.

R4

show ip bgp sum

3.4.0.3         4        50693     302     288       77    0    0 04:08:16        8

show ip bgp

*>  14.14.14.14/32   3.4.0.3                                0 50693 50693 ?

I omitted everything but R14's loopback, it's all we care about.

R13

show ip bgp sum

1.13.0.1        4        50693     262     265       59    0    0 03:54:21        8

show ip bgp

*>  114.114.114.114/32          1.13.0.1                               0 50693 50693 ?

Now we need to be able to ping/trace R14 to XR4 loopbacks.

R14#ping 114.114.114.114 source lo0

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 2/6/20 ms

R14#traceroute 114.114.114.114 source lo0 num

VRF info: (vrf in name/id, vrf out name/id)

  1 13.14.0.13 [MPLS: Labels 29/44 Exp 0] 5 msec 7 msec 5 msec

  2 1.13.0.1 [MPLS: Label 44 Exp 0] 30 msec 31 msec 152 msec

  3 10.1.2.2 [MPLS: Labels 24/30 Exp 0] 31 msec 32 msec 31 msec

  4 3.4.0.3 [AS 50693] [MPLS: Label 30 Exp 0] 32 msec 30 msec 29 msec

  5 3.4.0.4 [AS 50693] [MPLS: Label 18 Exp 0] 21 msec 20 msec 20 msec

  6 10.4.14.14 [AS 50693] 21 msec *  3 msec

Now we need to peer R14 to XR4 via VPNv4.

R14

router bgp 1000

 bgp log-neighbor-changes

 no bgp default ipv4-unicast

 neighbor 114.114.114.114 remote-as 1000

 neighbor 114.114.114.114 update-source Loopback0

address-family vpnv4

  neighbor 114.114.114.114 activate

  neighbor 114.114.114.114 send-community extended

XR4

router bgp 1000

neighbor 14.14.14.14

  remote-as 1000

  update-source Loopback0

  address-family vpnv4 unicast

R13 and R4 have to redistribute BGP into OSPF

router ospf 1

 redistribute bgp 1000 subnets

RP/0/0/CPU0:XR4#sh bgp vpnv4 unicast summary

14.14.14.14       0  1000     264     239        5    0    0 03:56:05          1

R14 and XR4 are now peered via VPNv4 iBGP. You can also see that "1" route is being propagated.

RP/0/0/CPU0:XR4#sh bgp vpnv4 unicast

*> 14.7.0.0/24        14.7.0.7                 0             0 7 i

*>i14.15.0.0/24       14.14.14.14              0    100      0 15 i

XR4 sees the route of 14.15.0.0/24 that R14 advertised it.

R15

show ip route

B        14.7.0.0/24 [20/0] via 14.15.0.14, 00:14:47

R15 sees the R7 route.

Now lets see if we can ping from R15 to R7

R15#ping 14.7.0.7

.....

Success rate is 0 percent (0/5)

WHAT!!!! Ok, calm down, easily fixed. But first explained. XR4 is learning the 14.7.0.0/24 prefix from R7. XR4 generates a VPNv4 label for it. That info is propagated via VPNv4 to R14. The problem is that XR4 to R4 is IGP+LDP. Therefore, when XR4 goes to forward the traffic, R4 won't know what do with it. The problem is that R13 is receving BGP labels from R1, there has to be a way to bridge the LSP from XR4 to R14, but we can't since R13 terminates the LSP via BGP and R13 to R14 is LDP. So.... we have to extend BGP to R14.

R13

router bgp 1000

 bgp log-neighbor-changes

 no bgp default ipv4-unicast

 neighbor 14.14.14.14 remote-as 1000

 neighbor 14.14.14.14 update-source Loopback0

address-family ipv4

  neighbor 14.14.14.14 activate

  neighbor 14.14.14.14 send-label

R14

router bgp 1000

 bgp log-neighbor-changes

 no bgp default ipv4-unicast

 neighbor 13.13.13.13 remote-as 1000

 neighbor 13.13.13.13 update-source Loopback0

address-family ipv4

  neighbor 13.13.13.13 activate

  neighbor 13.13.13.13 send-label

Now R13 can advertise BGP labels to R14

R15

R15#ping 14.7.0.7

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 3/7/18 ms

R15#traceroute 14.7.0.7 num

VRF info: (vrf in name/id, vrf out name/id)

  1 14.15.0.14 71 msec 3 msec 1 msec

  2 13.14.0.13 [MPLS: Labels 29/44/24006 Exp 0] 29 msec 16 msec 4 msec

  3 1.13.0.1 [MPLS: Labels 44/24006 Exp 0] 12 msec 32 msec 51 msec

  4 10.1.13.13 [MPLS: Labels 130002/30/24006 Exp 0] 122 msec 81 msec 23 msec

  5 10.13.3.3 [MPLS: Labels 30/24006 Exp 0] 32 msec 52 msec 33 msec

  6 3.4.0.4 [MPLS: Labels 18/24006 Exp 0] 20 msec 26 msec 21 msec

  7 10.4.14.14 20 msec 77 msec 15 msec

  8 14.7.0.7 [AS 7] 35 msec *  6 msec

VIOLA! Ok, it's a bit dirty but it's working. You can see there are 3 label stacks on hops 2 and 4

24006 is the VPNv4 label XR4 allocated the 14.7.0.0/24 prefix. This never changes. The 29/44 in our first hop is 44 of the LDP label and 29 of the BGP label. 

That is pretty much it. OK, I know what your thinking. Jee Rob, long enough post? This is the last long one for a while. Not much left to cover for CCNP stuff. I plan on doing a ton of posts for CCIE prep. 

Thanks for stopping by!

Rob Riker, CCIE #50693