3

I'm trying to do (what I think is) a simple thing for a custom tcp server (a type of sftp):

Allow traffic to come in on port 2222 via an aws network load balancer and then be routed to instances in a private subnet.

I've done the same with an application load balancer and it works great.

But I can't get the network version working.

I'm wondering if this is even possible? I do get a warning when I create the network load balancer saying that there are no instances in my public subnet (which is true... all the instances are in the private subnet). But I weirdly don't get this error when creating an application load balancer with the same subnets.

Am I missing something here?

(I can ping my server no problem using a bastion host, so I know it's running. I've also opened port 2222 in all related security groups.)

Can network load balancers direct traffic to instances in a private subnet?

Nick Lang
  • 469
  • 6
  • 16
  • 2
    I wouldn't be surprised if that didn't work. NLBs pass the traffic along unchanged, and don't have Security Groups. They are operating at a different network layer than ALBs. There's probably a good reason it is giving you the warning about no instances in the public subnet. You can't assume it will work just because it works for ALBs, they are fundamentally different. – Mark B Nov 09 '17 at 01:31

2 Answers2

8

NLB appears to modify the behavior of the network infrastructure, rather than being "hidden EC2 instance"-based like ALB or classic ELB, so a different network configuration is required.

The instances need to be on a public subnet with their default route pointing to the Internet Gateway in order to work with an outside-facing NLB, because unlike the other load balancer offerings, they don't return their response traffic to "the balancer's internal IP" since the balancer has no instance-facing IP address of its own, and the instances see the traffic as coming directly from the client IP.

Michael - sqlbot
  • 169,571
  • 25
  • 353
  • 427
  • Great feedback. Thank you. Can you point to any relevant documentation which helps support your comments.? – Felipe Alvarez Mar 02 '18 at 13:23
  • 1
    @FelipeAlvarez not really. These insights are mostly the result of observation, testing, and intuition. It turns out, the instances don't actually *need* to be on a public subnet, due to a quirk in the implementation. If they are on a private subnet, a public NLB still works as long as the private subnet has a default route. It doesn't actually matter where the route points, there just needs to be one configured -- the network is rewriting packets, as noted above, so the target of the default route is ignored, but without a default route they are dropped before the rewrite can occur. – Michael - sqlbot Mar 02 '18 at 16:19
  • 1
    A Network Load Balancer is, conceptually, very similar in some senses to a NAT Gateway but it works in the opposite direction, allowing external clients to access internal machines using a public IP on the outside, connecting to private IPs on the inside, and rewriting the packets so that both ends understand the conversation. – Michael - sqlbot Mar 02 '18 at 16:21
  • 1
    Reference for the default route quirk: https://stackoverflow.com/a/48798518/1695906 – Michael - sqlbot Mar 02 '18 at 16:23
  • "The instances need to be on a public subnet with their default route pointing to the Internet Gateway in order to work with an outside-facing NLB" - they do not. You absolutely can run an NLB in a public subnet with the instances in a private subnet without internet. – Alec Fenichel Jul 08 '18 at 03:19
  • 1
    @AlecFenichel read the comments, above. It works, with just the right configuration, but it seems like a bad idea to design based on behavior that essentially works by accident. – Michael - sqlbot Jul 08 '18 at 03:28
  • It does not work by accident, it works by design. It is using a flow hashing algorithm. Having instances in a private subnet offers a huge security benefit over a public subnet. See https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html#requests-not-routed. – Alec Fenichel Jul 08 '18 at 04:54
4

So what everyone responded with is absolutely correct.

The solution was to simply change the security group associated with the network interface that is in turn associated with my private subnets as follows:

2222 0.0.0.0/0

I had previously had it as

2222 10.0.0.0/16 (where 10.0.0.0/16 is my vpc)

and this doesn't work since clients could come from anywhere and they aren't coming from my network because the network load balancer passes off clients directly to the backend infrastructure.

This simple change fixed the problem right away.

Nick Lang
  • 469
  • 6
  • 16