Thursday, June 16, 2016

Cisco Routers: Easy Hair-Pin NAT for Internal Guest Network

Hey all! 

Recently I've been pouring myself into one particular configuration issue that is remarkably hard to solve on Cisco's IOS platform: Hairpin NAT. 

I've been tasked with designing a complete architecture for almost 20 sites. They want DMZs, Guest networks, lots of static and complex NATs, the works. And guess how many ASAs I can use? None

Which isn't that big of a deal, right? IOS has caught up to most security features over the past 5 or so years. NAT NVI (Nat Virtual Interface) can handle even complex NATs, ZBF (Zone-Based Firewall) is a nuanced and fantastic way to handle access control, etc. Hell, you can even build AnyConnect on IOS these days! 

But there's one problem that is intractable on Cisco's IOS platform: Hairpin-NAT

Because this problem is called so many things, let's define it. It's the issue that is seen when an internal host wants to access an internal server using the server's public (NAT'd) IP. Usually you don't see this with internal hosts, but the issue frequently comes up in designs when there is a "guest" network. Usually the guest network uses the same network infrastructure and internet connection as your other internal servers, but you use public DNS to make network segmentation clean and prevent as much internal access as possible. 
Picture from ServerFault
On an ASA, this problem is easy to solve - in fact, it's a single command.

same-security-traffic permit intra-interface

Surely it's this easy to solve on a router?

I tried to solve the problem in a half-dozen ways, and each has a "gotcha" that prevents it being useful in production. 

Potential Solution #1: Nat Virtual Interface (NVI) NAT'ing

This has the benefit of immediate results. Turn on NVI NAT'ing, modify the NAT statements and "ip nat enable" on a few interfaces, and you're good, all the guest network can use public IPs to access the DMZ servers. This technique is widely recommended as THE solution to this problem on the Internet. However, when there is more than an "inside" and "outside" interface, this type of NAT'ing gets very complex. 

As Alex points out in the comments below, you can make this work by creating exemptions in your NAT traffic selection. However, the complexity of this solution makes it less than ideal. 

Major drawback: Complexity with >2 interfaces and WAN NATing, high router utilization as NVI traffic is punted to the CPU for most ISR platforms, all NATs would have to be rewritten in the new style. 

Verdict: This is a no-go.

Potential Solution #2: Tons of Route-Maps

A widely cited solution from Tassos (who is a gosh-darn genius, and you should read his blog) nearly 8 years ago is to use regular NAT, but use route-maps to push traffic through an "ip nat outside" loopback address with policy routing on several interfaces. 

Major drawback: Super complex, high router utilization doing policy routing on several/all interfaces, I was not smart enough to get it working in a lab. 

Verdict: Not going to fly in a real production network (if I could even build it)

Potential Solution #3: Put the guest network on a separate internet connection

This is the solution often employed by Cisco shops that have become frustrated with the complexity and lack of manageability of Guest networks on the inside of the network. Shops either policy route the guest network out the other internet connection on their firewall or segment the network with a separate SSID/switch for guest that isn't connected to the internet network. 

Major drawback: As a consultant, I can't in good conscience ask the client to turn up 20 new consumer-class Internet circuits because I can't figure this issue out. 

VerdictDammit, I'm a good network engineer, I should be able to do this!

Potential Solution #4: Mix Up Some NATs

At this point I'm pulling my hair out, and really starting to believe that this just isn't possible, an answer that is widely circulated on the internet. I really just wish I could put the "guest" on the outside of my network. So I did. And dammit, it felt good. But of course, with both the "WAN" and "guest" in the same "ip nat outside" zone, there's no overload NAT to give the poor guys some Internets. 

And then I think of a funny joke. I wonder if I could use BOTH zoned NAT (ip nat inside/ip nat outside) AND stateless NAT (NVI) at the same time. I could just turn on stateless NAT on the guest and WAN zones, then write a single NAT statement to give it Internet access. I mean... there's no way Cisco would let you run both types of NAT at the same time, but I'm at wit's end. And low and behold, it flipping works. 

Major drawback: Configuration is slightly unorthodox. Using both NAT types looks funny, and someone might absent-mindedly 'fix' this in the future. Cisco might remove the older-style inside/outside NAT in future releases. 

VerdictSolution! (Short of Cisco actually supporting hairpin-nat, this'll do) 

Final Configuration

! Inside Host (vlan 1)
int eth 0/0
 ip add 192.168.1.50 255.255.255.0
ip route 0.0.0.0 0.0.0.0 192.168.1.1

! Guest Host (vlan 10)
int eth 0/0
 ip add 192.168.10.50 255.255.255.0
ip route 0.0.0.0 0.0.0.0 192.168.50.1

! DMZ Host (vlan 99)
int eth 0/0
 ip add 192.168.99.50 255.255.255.0
ip route 0.0.0.0 0.0.0.0 192.168.99.50

! Edge Router with router on a stick

! Interface configuration
interface Ethernet1/0.1
 encapsulation dot1Q 1 native
 ip address 192.168.1.1 255.255.255.0
 ip nat inside  <-- Most interfaces remain 'ip nat inside,' and NAT statements don't have to be rewritten

interface Ethernet1/0.10
 description GUEST
 encapsulation dot1Q 10
 ip address 192.168.10.1 255.255.255.0
 ip nat outside  <-- Note that 'guest' SVI is in outside zone. This enable all existing zoned-NATs to behave properly, without any config rewrites. So 'guest' users can access DMZ hosts using their public addresses just fine. 
 ip nat enable  <-- Stateless NAT also created on this interface

interface Ethernet1/0.99
 description DMZ
 encapsulation dot1Q 99
 ip address 192.168.99.1 255.255.255.0
 ip nat inside

interface Ethernet1/1
 description WAN
 ip address 20.0.0.2 255.255.255.0
 ip nat outside
 ip nat enable

! Route
ip route 0.0.0.0 0.0.0.0 20.0.0.1

! Object-groups and ACL

object-group network LocalGuest  <-- Highly recommend using object-groups to minimize configuration and simplify
 192.168.10.0 255.255.255.0
object-group network RFC1918Private 
 10.0.0.0 255.0.0.0
 172.16.0.0 255.240.0.0
 192.168.0.0 255.255.0.0

ip access-list extended Guest_2_WAN
 permit ip object-group LocalGuest any

ip access-list extended privateToPublic
 permit ip object-group RFC1918Private any

ip access-list extended siteToSite
 permit ip object-group RFC1918Private object-group RFC1918Private

route-map natOverload deny 10
 match ip address siteToSite
route-map natOverload permit 20
 match ip address privateToPublic

! NAT Configuration

ip nat inside source route-map natOverload interface Ethernet1/1 overload  <-- Regular inside/outside overload NAT for internet traffic

ip nat source list Guest_2_WAN interface Ethernet1/1 overload  <-- The new "NVI" style of NAT. Note the lack of "inside" in this command. Simply provides internet access to "guest" network segment. 

ip nat inside source static 192.168.99.50 20.0.0.50 extendable  <-- Hosts can still be exposed to the internet with regular "ip nat inside source static" commands. 

Lock it Down: ZBFW and Reflexive ACLs

I initially attempted to use ZBFW rules to lock down traffic from WAN-->Self and Self-->WAN, which works perfectly for zoned NAT (inside/outside). However, with NVI-style NAT, return internet traffic is evaluated by the edge router as destined for the edge router itself. Even when the outbound traffic is inspect on leaving this behavior occurs. 

But in engineering, there is always more than one way to skin that darn cat. So let's build some reflexive ACLs to permit return traffic. 

! Configuration: 


ip access-list extended In_2_Out
 permit ip any any reflect StatefulInbound  <-- Allow all traffic outbound, and remember it (default 300 seconds)

ip access-list extended Out_2_In
 evaluate StatefulInbound  <-- Allow all traffic that exited to return, using the stateful ACL table. 
 permit ip any host 20.0.0.50  <-- Permit inbound traffic for this particular DMZ services host


interface Ethernet1/1
 ip access-group Out_2_In in
 ip access-group In_2_Out out

In fact, you can still use ZBFW for most traffic, just make sure to permit it past the interface ACL so it can be picked up by NAT and then ZBF and delivered to internal hosts. 

Pro tip: If you're using this reflexive interface ACL as well as ZBF, remember your order of operations. The interface ACL is evaluated pre-NAT (so inbound traffic destinations will be public IPs), and ZBF is evaluted post-NAT (so inbound traffic dests will be your private IPs)

Download the GNS3 and Do It Yourself

There's nothing better for learning than building the thing yourself. Here's a completed GNS3 file with all features deployed. Please download and play with it yourself!

Files are here: 
https://1drv.ms/f/s!AliOPzHSO-GngrhpfKDb-YNfu1uLXQ

Good luck out there. 
Kyler

15 comments:

  1. How can this solution be adapted to simply access internal servers using the public IP address from the internal network? My setup does not have an guest network, there is no need for that. It feels as if it's possible, and would look like the easiest way to do it, but I'm at a loss.
    I've tried adding the "ip nat enable" alongside the "ip nat inside/outside" and the "ip nat overload" - but this is not working.
    P.S. you have a typo here:
    ! Guest Host (vlan 10)
    int eth 0/0
    ip add 192.168.10.50 255.255.255.0
    ip route 0.0.0.0 0.0.0.0 192.168.50.1.1 - the last ".1" shouldn't be there.

    ReplyDelete
    Replies
    1. Hey Tony,

      Thanks for the catch on the typo - corrected.

      Cisco's ISR/router platform doesn't allow you to easily get around this. That wouldn't be hard at all to achieve on an Cisco ASA, but it's going to get very complex on a Cisco router platform.

      Option A: Move DMZ to "ip nat outside" zone. Create duplicate stateless/NVI NATs from DMZ to outside for overload NAT and static NAT.
      Option B: The "Tassos route-maps" option would be able to do this, but I wasn't able to get it working on a new platform. Maybe you'll have more success. Still, regardless of whether this would actually work, it'd be a huge pain to maintain or teach someone else how it works.

      Sorry to not be much help here -- good luck sorting it, and let me know if you find an elegant solution on router platform.

      Delete
  2. So in this example, are we assuming that LAN traffic will reach the DMZ hosts without using the public IP of the DMZ hosts?

    ReplyDelete
    Replies
    1. Hey Dan,

      The "internal" LAN hosts would reach the DMZ hosts using their private/DMZ IPs. The "guest" and "wan"/internet hosts would reach the DMZ hosts using their public/outside NAT IPs.
      kyler

      Delete
    2. Thanks for the clarification. Trying to figure out how to handle my situation, I have a Sandstorm server on my LAN with a dynamic DNS address (from sandcats.io) that resolves to my public IP, which of course means my LAN hosts can't reach the server. The fact that Sandstorm is very particular as to how you reach the server complicates things; if I try to access it via https://lan.ip.of.server it puts me on an error page, so I HAVE to access it somehow via my public IP from the LAN. Trying to do some mix of nat enable and inside/outside right now but it's not jiving...it didn't BREAK anything yet, but it sure didn't fix it either!

      Delete
    3. Hey Dan,

      That sounds doable, provided you're using the older-style of NAT for your regular internet traffic (ip nat inside/ip nat outside).

      1. Enable NVI nat on your internal and DMZ intefaces
      int XX
      ip nat enable
      int XX
      ip nat enable

      2. Write an NVI NAT statement with your private/public IPs for that DMZ host.
      ip nat source static (DMZ private IP) (DMZ public IP)

      That should do it. You will no longer be able to access that host by its internal IP, but you should be able to access it using the public one. Let me know how it goes!
      kyler

      Delete
    4. Good point, I hadn't thought of doing it that way! I've got one other test to try before I go down that road, I'll post that if it works!

      In general though it's sounding like setting up an actual DMZ (I haven't done that yet because I just started testing this thing...), or just implementing a second router behind a different public IP would be best (I have a few static IPs to play with).

      Delete
    5. If you can get a firewall, then sharing your public IP space isn't hard at all. You can do interesting NATs like this pretty easily.

      If you're only able to use routers, then getting a second entire network and using publics is going to be far simpler.
      kyler

      Delete
    6. I figured it out (this part, at least)! I used nat enable on ALL interfaces, along with a very specific access list for NAT, with explicit deny and permit statements, like so:

      10 deny ip 10.1.0.0 0.0.255.255 10.1.0.0 0.0.255.255
      20 deny ip 10.1.0.0 0.0.255.255 10.70.0.0 0.0.255.255
      30 deny ip 10.1.0.0 0.0.255.255 172.31.0.0 0.0.255.255 (1430 matches)
      31 deny ip 10.1.0.0 0.0.255.255 10.0.0.0 0.0.0.7
      40 deny ip 10.70.0.0 0.0.255.255 10.1.0.0 0.0.255.255
      50 deny ip 10.70.0.0 0.0.255.255 10.70.0.0 0.0.255.255
      60 deny ip 10.70.0.0 0.0.255.255 172.31.0.0 0.0.255.255
      61 deny ip 10.70.0.0 0.0.255.255 10.0.0.0 0.0.0.7
      70 deny ip 172.31.0.0 0.0.255.255 172.31.0.0 0.0.255.255 (2 matches)
      80 deny ip 172.31.0.0 0.0.255.255 10.70.0.0 0.0.255.255
      90 deny ip 172.31.0.0 0.0.255.255 10.1.0.0 0.0.255.255 (3914 matches)
      91 deny ip 10.0.0.0 0.0.0.7 172.31.0.0 0.0.255.255
      92 deny ip 10.0.0.0 0.0.0.7 10.70.0.0 0.0.255.255
      93 deny ip 10.0.0.0 0.0.0.7 10.1.0.0 0.0.255.255
      96 deny ip 172.31.0.0 0.0.255.255 10.0.0.0 0.0.0.7
      100 permit ip 10.1.0.0 0.0.255.255 any (82 matches)
      110 permit ip 10.70.0.0 0.0.255.255 any
      120 permit ip 172.31.0.0 0.0.255.255 any (39 matches)
      130 permit ip 10.0.0.0 0.0.0.7 any

      I was technically lazy here, my subnets are actually like 10.1.110.0, 10.1.120.0, 10.1.130.0...etc., but this works as-is, so I'm OK with it. (My home network/lab has like 15 or so subnets, I need to scale it back a bit!)

      Now the only issue I have is I realized my ipsec VPN, which is hosted by my primary router, wants to use port 443 for authentication...well, so does Sandstorm, and I used static NAT (PAT) to redirect port 443 from the public side to the server! Depending on the real solution there I may yet have to get another router. (Routers are always cheaper than ASAs on the used market for some reason.)

      Anyway thanks for the tips! It's nice to find a helpful link that's not totally stale and abandoned for once.

      Delete
  3. Hi Kyler,

    Very nice write up!

    I have a question for Solution #1. In the major drawback I can see "... prevents internal hosts from using the DMZ server's private addresses to access them".
    Is that really so difficult for avoidance problem if the all translation between those two zones can be denied with few ACL lines?

    Greets,

    Alex

    ReplyDelete
  4. Hey Alex,

    There's no need to block it - by it's nature, it doesn't allow internal hosts to hit the DMZ hosts with their private IPs, which can be annoying and might require split DNZ. It certainly would change how most companies have built their DMZ access unless you start with this setup. And if you ever grow out of a router and into a proper security device like an ASA, you'll need to redo DNS pointers at your DMZ.

    If you'd want to block internal hosts from accessing the DMZ hosts, you'd use ACLs normally - if inbound on the inside interface, block pre-NAT'd IPs, if outbound on the DMZ interface, use post NAT'd IPs.

    Good luck out there!
    kyler

    ReplyDelete
    Replies
    1. Well, it works (NVI NAT) perfectly fine in my home setup

      Internet
      |
      L2 Switch == Router (on a stick, sub-ints for Inet, LAN and DMZ)
      | |
      | DMZ
      LAN

      I can access from the LAN both the public (where the DNS points) and the private IP addresses in the DMZ. The ACL, I was writing about, is used to exclude from the NAT interesting traffic any sessions between the LAN and the DMZ. In that way they are just routed...

      Most likely you should reconsider your NVI NAT understanding about "by it's nature, it doesn't allow internal hosts to hit the DMZ hosts with their private IPs" ;-)

      Delete
    2. Hey Alex,

      That's an excellent point! Thanks for pointing that out. It's still a much more complex solution than I'd like - imagine handing management of that solution over to a support team, and how difficult it'd be for them to understand and manage.

      I'll update the blog with your notes - thanks again!
      kyler

      Delete
    3. It depends how scalable you need to do it. The ACL is the most simple way to introduce some logic in the NAT interesting traffic selection. I used that one to give you just an idea...

      When speaking for more "user-friendly" approach, you can replace the ACL with route-map and to use your beloved :) 'object-group network ...' statements to identify the "LAN" and the "DMZ" set of subnets. The logic will be in the route-map and each area will have conveniently separated (and eventually described) configuration section. Those days, such a sections can be even updated in automated manner depending of the actual platform (google ansible/yaml or netconf/yang for more details in that direction).

      Usually, in a very big deployments, there are IP planning phase and that's the step when those things can be simplified with reserved, zones' dedicated big subnets...

      Good luck,

      Alex

      Delete
  5. Hello,
    If I want to access both the internal IP and public IP of a server from a internal network client PC, How can I do that?
    Thanks in advance.

    ReplyDelete