Provider independent IP address todo list

  • Register an LLC -- done! PETERJIN.ORG, LLC
  • Create ARIN online account -- done! Account, POC, and Org ID created
  • Register IPv6 address range -- done! 2602:806:a000::/44
  • Register ASN -- done! AS398565
  • Register IPv4 address range from the ARIN 4.10 ([1]) block -- done! 23.161.208.0/24
  • Announce them via BGP from apps-vm1 apps-vm6 -- done! All peers seeing 2602:806:a000::/47, 2602:806:a002::/47, 23.161.208.0/24
  • Get OV SSL certificate -- done!
  • If allowed by budget constraints, get EV SSL certificate

IPv6 assignments

Range Status Location/BGP Origin Purpose of use
2602:806:a000::/48 active Vultr/Choopa (AS20473); Chicago, IL, United States, but under AS398565 Networking for VPS
2602:806:a001::/48 active Vultr/Choopa (AS20473); Chicago, IL, United States, but under AS398565 Home Network (using Vultr as tunnel endpoint)
2602:806:a004::/48 cancelled Amazon AWS (AS?????); Oregon, United States Networking for EC2
2602:806:a005::/48 cancelled Amazon AWS (AS?????); Ohio, United States Networking for EC2, separate region
2602:806:a002::/48 immediate proposed Multiple locations (AS398565) NTP Anycast services
2602:806:a003::/48 immediate proposed VMHaus (AS1?????); Los Angeles, CA, United States, but under AS398565 Networking for VPS

How we will use our IPv4 addresses

On one server with a static BGP route to it:

iptables -t nat -A PREROUTING -d 192.0.2.6 -p tcp --dport 25 -j DNAT --to [private IP address of real mail server]
iptables -t nat -A PREROUTING -d 192.0.2.5 -p tcp --dport 80 -j DNAT --to [private IP address of real web server]
iptables -t nat -A POSTROUTING -s [private IP range e.g. 172.16.0.0/12] -j SNAT --to 192.0.2.16/28

The important aspects of this are:

  • IP addresses define services, not hosts. This allows us to host servers on certain IP addresses determined entirely by a procedural ruleset, rather than by the actual IP addressing architecture of the downstream hosts. We can freely mix and match IP addresses, both inbound (PREROUTING/DNAT) and outbound (POSTROUTING/SNAT). This also makes renumbering extremely easy since the only thing we need to do is change the ruleset's IP addresses (and possibly update IP addresses in the DNS).
  • No network visible on the public internet has an on-link route of 192.0.2.0/24 (example), so you can also use the .0 and .255 IP addresses since they have no special meaning. Try pinging 172.217.0.0 (a Google IP address); you might be surprised that it actually responds!
  • From the perspective of a third party, it might appear that you have one web server on 192.0.2.5 and one mail server on 192.0.2.6, but these numbers are not special to the third party. Your third party sees the IP addresses, and that's it. They don't know your subnet mask, or your gateway address. All it knows is that to connect to your mail server, the destination IP is set to 192.0.2.6. When it hits your router, even though it is not your mail server, it interprets this destination IP using the ruleset rather than using the conventional routing table, and correctly forwards it to your mail server.

The only disadvantage is that clients on the network cannot access the resources otherwise made available on the public IP addresses, but this could be solved by forcing IPv6 only to those servers on the local network (other than the traditional approach of split-horizon DNS). It is also difficult to multihome or implement anycast, since connection tracking is required, but in some cases, static /32 routes can be used instead.

In general,

  • Networks should be designed with public IPv6 and private IPv4 for hosts.
  • When registering a new /24 of IPv4 from ARIN, APNIC, RIPE, or whatever, think of the IP addresses only as service points and not hosts, and then implement the necessary routing rules for services (not hosts) on your public facing router.

The university I go to has three legacy /16's allocated to them. They run a lot of websites on various subdomains of their primary domain. They also have ten IPs for outbound email and two for inbound. Excluding IPs for their network infrastructure on public BGP/routing/peering arrangements, and outbound IPs for the purpose of auditing outbound network connections from dorm rooms, they could still operate even with just one /24, by putting in a load balancer on a public IP for all of their websites, as long as they deploy IPv6, as long as they have less than 256 web servers (note that virtual hosting allows for more than one website on a web server, so the total number of websites can be greater than 256), 256 email servers (again, RCPT TO allows for multiple mail servers to be aggregated, using a relay to send to downstream mail servers on private IPs, so the total number of email domains can be greater than 256), and 256 SSH servers. The fact that none of the websites actually have IPv6 (except for DNS, the VPN, and their dedicated IP transit provider [2]) was just frustrating. Of course, this could never happen in one night, but this also means that if I were to set up a network for a large enterprise, I would definitely set it up in a way that encourages public IPv4 address conservation.

Hmmmmm... maybe if I ever design networking equipment, I think it might be useful to put an FPGA in it, so that the routing mechanisms could be determined entirely by software, yet still reach the capacity of hardware routers with ASICs in them. Maybe allow custom IP cores and VHDL/Verilog code on it as part of custom firmware images such as OpenWRT?

Technically, we have more IP addresses than the University of Illinois. They have:

  • 3 IPv4 /16's + 1 /18 (200k)
  • One /44 (2^84), only two /48's of which are announced
  • Two /48's (2^81)

And we have

  • One /44 (2^84), of which four /48's are announced
  • One /48 (2^80) and one /64 (2^64) (Hurricane Electric tunnel on apps-vm3)
  • About two IPv4 addresses

One thing to note about our network is that our network infrastructure is more of creative artwork rather than business requirements (although I use it personally to do business [broadly defined] with other people). IP addresses and network design is an attractive target since not only do I know how to do computer networking, but it is an expression which can be easily realized by other people over the Internet simply by connecting to my public servers.

See also