Thursday, March 28, 2019

Cisco ASA Multiple Context Mode with Active/Active Stateful Failover


Multiple context mode is the method that ASAs use to “virtualize” or user created firewalls. These new “user” firewalls can be allocated to a customer, business unit or other need. If you are familiar with IOS router/switch VRF or virtual routing and forwarding tables, then learning contexts is going to be simpler. The only similarity is user defined routing table, contexts go beyond with separate configurations and policies per context. The resources of the chassis are shared across all of the contexts, resources can be prioritized and assigned to a context.

Each context is assigned a name, interfaces, resource limits and a firewall policy. The interface allocation can be either physical or sub-interfaces, which is the most common method. The resource limits apply to the number of connections, hosts, NATs and other attributes. The firewall policies are per context and can be modified to meet the needs of the deployment.

The ASA supports a lot of features from the 8.X versions of code, routed mode which is the default mode of operation. Transparent mode in the event of bump in the wire capabilities. Active/Active failover, which is really Active/Standby on a per context basis. Dynamic routing with OSPF and EIGRP is supported in the user contexts, so is static routing. Site to Site VPN is also supported. WebVPN, both SSL and AnyConnect are not supported, and neither is QoS. These features on ASA 9.1.5 code.

There are three different contexts that we need be aware of and how they work in this setup. The first of which is the system context, which is used to create new contexts and define context parameters. The interface to context assignments, the resource allocations and the configuration file location. The system context allows the admin to switch between the different contexts for management. When you first rollout multiple context mode, the system context is only available from the console, which can be changed after the admin context is defined and setup.

The admin context is used for remote access to the system context, it is the only context that allows for remote support where the “changeto system” command can be used to remotely manage the ASA. This context is automatically created when the ASA is configured for multiple context mode. The configuration is saved in the “admin.cfg” file located on disk0:/. Since this is the first actual context, there are no resources allocated by default, like interfaces, remote access or other options. Typically, the management interface of the ASA will be allocated, SSH with AAA for remote access and the ASA is ready for remote management.

The user defined context is the last to be configured, usually, and just like the admin context, needs to be allocated interfaces, resources, firewall policies and any other optional configurations.

There is the high likelihood that an interface will be shared between one or more contexts, like the internet enabled interface for example. Interface sharing is defined in the system context during interface allocation. The ASA must be able to associate inbound traffic with the context the traffic was originated from. There are three ways to classify the traffic, unique physical or sub-interfaces, unique MAC address per context for the shared interfaces or unique NAT translations per context for the shared interfaces.

The unique physical interface separates traffic at the physical or VLAN level which is the most common design and can consume public IPs. When it comes to shared interfaces, the same MAC address is used for all contexts, which will cause recursion issues. The unique MAC address per-context is another method, which is what we’ll be using, when used, new MACs per context are generated. The last option is unique NAT translations per context, with static or dynamic NAT, overlapping addressing supported inside the contexts, every context could use the 10.0.0.0/8 range without issue. However, the outside address must be unique per context.

Context resources can also be limited on a per context basis. This could be used to limit resources if licensing restricts usage to a certain feature. The resources can be limited for connections, NAT translations, dynamic MAC address limits and management sessions. The resource limitations are tied to a resource class which is then applied to the context.





SW3
Vlan 15
 Name TEST15
!
Vlan 16
 Name TEST16
!
interface GigabitEthernet0/0

 switchport trunk encapsulation dot1q
 switchport mode trunk
 media-type rj45
 negotiation auto
!
interface GigabitEthernet0/1

 switchport trunk encapsulation dot1q
 switchport mode trunk
 media-type rj45
 negotiation auto
!
interface GigabitEthernet0/2
 switchport access vlan 15
 switchport mode access
 media-type rj45
 negotiation auto
 spanning-tree portfast edge
!
interface GigabitEthernet0/3
 switchport access vlan 16
 switchport mode access
 media-type rj45
 negotiation auto
 spanning-tree portfast edge



ASA3 system configuration
hostname ASA3
!
mac-address auto prefix 0
!
interface Ethernet0
 no shut
!
interface Ethernet0.15
 vlan 15
!
interface Ethernet0.16
 vlan 16
!
interface Ethernet1
 no shut
!
interface Ethernet2
 no shut
!
interface Ethernet3
 no shut
!
context Test15
  allocate-interface Ethernet0.15 inside15 visible
  allocate-interface Ethernet3 outside visible
  config-url disk0:/TEST15.cfg
!
context Test16
  allocate-interface Ethernet0.16 inside16 visible
  allocate-interface Ethernet3 outside visible
  config-url disk0:/TEST16.cfg
!
failover
failover lan unit primary
failover lan interface FO Ethernet1
failover interface ip FO 10.1.1.3 255.255.255.0 standby 10.1.1.4
failover link SFFO Ethernet2
failover interface ip SFFO 10.1.2.3 255.255.255.0 standby 10.1.2.4
failover group 1
failover group 2
  secondary


By default, when an interface is allocated to a context, its hardware properties are not passed from the system context to the user context. The “invisible” command is used by default, to pass those properties, I added the “visible” command to allow those properties to be visible in the user context.

ASA3 admin context configuration
admin-context admin
context admin
  config-url disk0:/admin.cfg

Currently we don’t have a connection setup to the management interface of the ASA, but in production, you would allocate the management interface to the admin context for remote management.

ASA3 user context Test15 configuration
ASA3# changeto context Test15
ASA3/Test15#
interface inside15
 nameif inside
 security-level 100
 ip address 192.168.15.1 255.255.255.0
!
interface outside
 nameif outside
 security-level 0
 ip address 202.1.10.15 255.255.255.0
!
object network TEST15
 host 192.168.15.15
!
object network TEST15
 nat (inside,outside) dynamic interface
!
route outside 0.0.0.0 0.0.0.0 202.1.10.1 1
!
policy-map global_policy
 class inspection_default
  inspect icmp
!
changeto context Test15

  interface inside15
  ip address 192.168.15.1 255.255.255.0 standby 192.168.15.2
!
monitor-interface inside


ASA3 user context Test16 configuration
ASA3# changeto context Test16
interface inside16
 nameif inside
 security-level 100
 ip address 192.168.16.1 255.255.255.0
!
interface outside
 nameif outside
 security-level 0
 ip address 202.1.10.16 255.255.255.0
!
object network TEST16
 host 192.168.16.16
!
object network TEST16
 nat (inside,outside) dynamic interface
!
route outside 0.0.0.0 0.0.0.0 202.1.10.1 1
!
policy-map global_policy
 class inspection_default
  inspect icmp

ASA4

System context configuration
failover
failover lan unit secondary
failover lan interface FO Ethernet1
failover interface ip FO 10.1.1.3 255.255.255.0 standby 10.1.1.4
failover group 1
failover group 2
  secondary

User context Test16 configuration
changeto context Test16
interface inside16
 ip address 192.168.16.1 255.255.255.0 standby 192.168.16.2
!
monitor-interface inside


Now we can test out the configuration. Test15 and Test16 are routers that will be used to test connectivity through the ASA.

Test15#ping 10.255.1.1 repeat 100
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.255.1.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

ASA3/Test15# show conn long
0 in use, 1 most used

ICMP outside: 10.255.1.1/0 (10.255.1.1/0) inside: 192.168.15.15/4 (202.1.10.15/4), , flags  , idle 0s, uptime 0s, timeout 2s, bytes 72


Test16#ping 10.255.1.1 repeat 100
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.255.1.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 17/40/337 ms

ASA3/Test16# show conn long
1 in use, 1 most used

ICMP outside: 10.255.1.1/0 (10.255.1.1/0) inside: 192.168.16.16/2 (202.1.10.16/2), , flags  , idle 0s, uptime 0s, timeout 2s, bytes 72

You can see that both contexts have the ability to communicate through the ASA by pinging my home router.

R2#show arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  202.1.10.1              -   5000.0008.0000  ARPA   GigabitEthernet0/0
Internet  202.1.10.15           165   a200.0000.0006  ARPA   GigabitEthernet0/0
Internet  202.1.10.16             1   a200.0000.0008  ARPA   GigabitEthernet0/0

You can see that R2 shows 2 different ARP entries from ASA3, one from each context.



ASA3/pri/act(config)# show failover
Failover On
Failover unit Primary
Failover LAN Interface: FO Ethernet1 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 4 of 60 maximum
Version: Ours 9.1(5)16, Mate 9.1(5)16
Group 1 last failover at: 06:13:55 UTC Mar 28 2019
Group 2 last failover at: 03:54:50 UTC Mar 28 2019

  This host:    Primary
  Group 1       State:          Active
                Active time:    27 (sec)
  Group 2       State:          Bulk Sync
                Active time:    6 (sec)

                  Test15 Interface inside (192.168.15.1): Normal (Monitored)
                  Test15 Interface outside (202.1.10.15): Unknown (Waiting)
                  Test16 Interface inside (192.168.16.2): Normal (Monitored)
                  Test16 Interface outside (0.0.0.0): Unknown (Waiting)

  Other host:   Secondary
  Group 1       State:          Bulk Sync
                Active time:    18 (sec)
  Group 2       State:          Active
                Active time:    8629 (sec)

                  Test15 Interface inside (192.168.15.2): Normal (Monitored)
                  Test15 Interface outside (0.0.0.0): Unknown (Waiting)
                  Test16 Interface inside (192.168.16.1): Normal (Monitored)
                  Test16 Interface outside (202.1.10.16): Unknown (Waiting)

Stateful Failover Logical Update Statistics
        Link : SFFO Ethernet2 (up)
        Stateful Obj    xmit       xerr       rcv        rerr
        General         4          0          3          0
        sys cmd         0          0          0          0
        up time         0          0          0          0
        RPC services    0          0          0          0
        TCP conn        0          0          0          0
        UDP conn        0          0          0          0
        ARP tbl         2          0          2          0
        Xlate_Timeout   0          0          0          0
        IPv6 ND tbl     0          0          0          0
        VPN IKEv1 SA    0          0          0          0
        VPN IKEv1 P2    0          0          0          0
        VPN IKEv2 SA    0          0          0          0
        VPN IKEv2 P2    0          0          0          0
        VPN CTCP upd    0          0          0          0
        VPN SDI upd     0          0          0          0
        VPN DHCP upd    0          0          0          0
        SIP Session     0          0          0          0
        Route Session   0          0          0          0
        User-Identity   2          0          1          0
        CTS SGTNAME     0          0          0          0
        CTS PAC         0          0          0          0
        TrustSec-SXP    0          0          0          0
        IPv6 Route      0          0          0          0

        Logical Update Queue Information
                        Cur     Max     Total
        Recv Q:         0       4       6
        Xmit Q:         0       2       13

We can see that based on the failover configuration now being stateful, we see additionally output and the hit counters are incrementing indicating that traffic and its associated connections and resource allocations are being synced between the ASAs. We can now test failover for connections that will be able to be synced. We open a Telnet session to 10.255.1.1, authenticate and send a little data.

Test15#telnet 10.255.1.1
Trying 10.255.1.1 ... Open


User Access Verification

Username: Rob
Password:
Rack1R2#
Rack1R2#
Rack1R2#
Rack1R2#
Rack1R2#

ASA3/pri/act/Test15(config)# show conn long
5 in use, 5 most used

TCP outside: 10.255.1.1/23 (10.255.1.1/23) inside: 192.168.15.15/41152 (202.1.10.15/41152), flags UxIO , idle 2s, uptime 3s,
timeout 1h0m, bytes 178

We have to changeto the user context Test15 to look at the connection table, we see an active connection.

ASA3/sec/stby# failover active group 1
ASA3/sec/act#

ASA3/pri/stby/Test15# show conn long
5 in use, 5 most used

TCP outside: 10.255.1.1/23 (10.255.1.1/23) inside: 192.168.15.15/41152 (202.1.10.15/41152), flags UxO , idle 1s, uptime 23s, timeout 1h0m, bytes 2

After failing group 1 over from the active member to the standby member, a little more traffic is generated and we see the connection is still up.

ASA3/pri/stby# show failover
Failover On
Failover unit Primary
Failover LAN Interface: FO Ethernet1 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 4 of 60 maximum
Version: Ours 9.1(5)16, Mate 9.1(5)16
Group 1 last failover at: 06:15:17 UTC Mar 28 2019
Group 2 last failover at: 03:54:50 UTC Mar 28 2019

  This host:    Primary
  Group 1       State:          Standby Ready
                Active time:    2 (sec)
  Group 2       State:          Standby Ready
                Active time:    6 (sec)

                  Test15 Interface inside (192.168.15.2): Normal (Monitored)
                  Test15 Interface outside (0.0.0.0): Unknown (Waiting)
                  Test16 Interface inside (192.168.16.2): Normal (Monitored)
                  Test16 Interface outside (0.0.0.0): Unknown (Waiting)

  Other host:   Secondary
  Group 1       State:          Active
                Active time:    16 (sec)
  Group 2       State:          Active
                Active time:    8702 (sec)

                  Test15 Interface inside (192.168.15.1): Normal (Waiting)
                  Test15 Interface outside (202.1.10.15): Unknown (Waiting)
                  Test16 Interface inside (192.168.16.1): Normal (Monitored)
                  Test16 Interface outside (202.1.10.16): Unknown (Waiting)

Stateful Failover Logical Update Statistics
        Link : SFFO Ethernet2 (up)
        Stateful Obj    xmit       xerr       rcv        rerr
        General         19         0          19         0
        sys cmd         11         0          11         0
        up time         0          0          0          0
        RPC services    0          0          0          0
        TCP conn        4          0          5          0
        UDP conn        0          0          0          0
        ARP tbl         2          0          2          0
        Xlate_Timeout   0          0          0          0
        IPv6 ND tbl     0          0          0          0
        VPN IKEv1 SA    0          0          0          0
        VPN IKEv1 P2    0          0          0          0
        VPN IKEv2 SA    0          0          0          0
        VPN IKEv2 P2    0          0          0          0
        VPN CTCP upd    0          0          0          0
        VPN SDI upd     0          0          0          0
        VPN DHCP upd    0          0          0          0
        SIP Session     0          0          0          0
        Route Session   0          0          0          0
        User-Identity   2          0          1          0
        CTS SGTNAME     0          0          0          0
        CTS PAC         0          0          0          0
        TrustSec-SXP    0          0          0          0
        IPv6 Route      0          0          0          0

        Logical Update Queue Information
                        Cur     Max     Total
        Recv Q:         0       4       103
        Xmit Q:         0       2       41

Checking the failover outputs, we see more hits on the counters, indicating that traffic is being synced.

ASA3/pri/stby# failover active group 1
ASA3/pri/act#

The failover command is executed again so that the actual active primary is the active node again.

ASA3/pri/act# show failover
Failover On
Failover unit Primary
Failover LAN Interface: FO Ethernet1 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 4 of 60 maximum
Version: Ours 9.1(5)16, Mate 9.1(5)16
Group 1 last failover at: 06:15:39 UTC Mar 28 2019
Group 2 last failover at: 03:54:50 UTC Mar 28 2019

  This host:    Primary
  Group 1       State:          Active
                Active time:    2 (sec)
  Group 2       State:          Standby Ready
                Active time:    6 (sec)

                  Test15 Interface inside (192.168.15.1): Normal (Waiting)
                  Test15 Interface outside (202.1.10.15): Unknown (Waiting)
                  Test16 Interface inside (192.168.16.2): Normal (Monitored)
                  Test16 Interface outside (0.0.0.0): Unknown (Waiting)

  Other host:   Secondary
  Group 1       State:          Standby Ready
                Active time:    23 (sec)
  Group 2       State:          Active
                Active time:    8712 (sec)

                  Test15 Interface inside (192.168.15.2): Normal (Waiting)
                  Test15 Interface outside (0.0.0.0): Unknown (Waiting)
                  Test16 Interface inside (192.168.16.1): Normal (Monitored)
                  Test16 Interface outside (202.1.10.16): Unknown (Waiting)

Stateful Failover Logical Update Statistics
        Link : SFFO Ethernet2 (up)
        Stateful Obj    xmit       xerr       rcv        rerr
        General         19         0          20         0
        sys cmd         11         0          11         0
        up time         0          0          0          0
        RPC services    0          0          0          0
        TCP conn        4          0          6          0
        UDP conn        0          0          0          0
        ARP tbl         2          0          2          0
        Xlate_Timeout   0          0          0          0
        IPv6 ND tbl     0          0          0          0
        VPN IKEv1 SA    0          0          0          0
        VPN IKEv1 P2    0          0          0          0
        VPN IKEv2 SA    0          0          0          0
        VPN IKEv2 P2    0          0          0          0
        VPN CTCP upd    0          0          0          0
        VPN SDI upd     0          0          0          0
        VPN DHCP upd    0          0          0          0
        SIP Session     0          0          0          0
        Route Session   0          0          0          0
        User-Identity   2          0          1          0
        CTS SGTNAME     0          0          0          0
        CTS PAC         0          0          0          0
        TrustSec-SXP    0          0          0          0
        IPv6 Route      0          0          0          0

        Logical Update Queue Information
                        Cur     Max     Total
        Recv Q:         0       4       115
        Xmit Q:         0       2       47

Checking the failover output again, the counters continue to climb.