Network

AS398565 Network Diagram (IPv4 omitted)

Peter Jin operates a computer network under AS398565, along with several provider-independent IPv4 and IPv6 ranges.

Inspiration

The peterjin.org Network was inspired by my university's network. They have three /16's worth of IPv4 ranges but only a single /48 for IPv6 (at least on campus). They don't use NAT at all except on the public wifi, yet the /48 is barely used.

Wanting to "man up" a bit, the peterjin.org Network seeks to break the status quo of IPv4-only networks, by making IPv6 deployment a top priority while still maintaining adequate IPv4 connectivity, even if it is heavily degraded.

Todo list (archived)

Click on "expand" on the right to view contents.

  • Register an LLC -- done! PETERJIN.ORG, LLC
  • Create ARIN online account -- done! Account, POC, and Org ID created
  • Register IPv6 address range -- done! 2602:806:a000::/44
  • Register ASN -- done! AS398565
  • Register IPv4 address range from the ARIN 4.10 ([2]) block -- done! 23.161.208.0/24
  • Announce them via BGP from apps-vm1 apps-vm6 -- done! All peers seeing 2602:806:a000::/47, 2602:806:a002::/47, 23.161.208.0/24
  • Get OV SSL certificate -- done!
  • If allowed by budget constraints, get EV SSL certificate

Peering

Our network is open to peering. The main endpoint is located in Los Angeles, advertising 2602:806:a002::/47. The Chicago endpoint is not open to peering.

IPv6 assignments

Further information: Peterjin.org:IP Addressing Plans
Range Status Location/BGP Origin Purpose of use
2602:806:a000::/48 active Vultr/Choopa (AS20473); Chicago, IL, United States, but under AS398565 Networking for VPS
2602:806:a001::/48 active Vultr/Choopa (AS20473); Chicago, IL, United States, but under AS398565 Home Network (using Vultr as tunnel endpoint). Also announced under AS138414 through separate Internet connection.
2602:806:a004::/48 cancelled Amazon AWS (AS?????); Oregon, United States Networking for EC2
2602:806:a005::/48 cancelled Amazon AWS (AS?????); Ohio, United States Networking for EC2, separate region
2602:806:a002::/48 active Multiple locations (AS398565) [1][2] NTP Anycast services
2602:806:a003::/48 active VMHaus (AS136620); Los Angeles, CA, United States, but under AS398565 [2] Networking for VPS

How we will use our IPv4 addresses

Current use of addresses described here. Click "expand" on the right to view historical content.

On one server with a static BGP route to it:

iptables -t nat -A PREROUTING -d 192.0.2.6 -p tcp --dport 25 -j DNAT --to [private IP address of real mail server]
iptables -t nat -A PREROUTING -d 192.0.2.5 -p tcp --dport 80 -j DNAT --to [private IP address of real web server]
iptables -t nat -A POSTROUTING -s [private IP range e.g. 172.16.0.0/12] -j SNAT --to 192.0.2.16/28

The important aspects of this are:

  • IP addresses define services, not hosts. This allows us to host servers on certain IP addresses determined entirely by a procedural ruleset, rather than by the actual IP addressing architecture of the downstream hosts. We can freely mix and match IP addresses, both inbound (PREROUTING/DNAT) and outbound (POSTROUTING/SNAT). This also makes renumbering extremely easy since the only thing we need to do is change the ruleset's IP addresses (and possibly update IP addresses in the DNS).
  • No network visible on the public internet has an on-link route of 192.0.2.0/24 (example), so you can also use the .0 and .255 IP addresses since they have no special meaning. Try pinging 172.217.0.0 (a Google IP address); you might be surprised that it actually responds!
  • From the perspective of a third party, it might appear that you have one web server on 192.0.2.5 and one mail server on 192.0.2.6, but these numbers are not special to the third party. Your third party sees the IP addresses, and that's it. They don't know your subnet mask, or your gateway address. All it knows is that to connect to your mail server, the destination IP is set to 192.0.2.6. When it hits your router, even though it is not your mail server, it interprets this destination IP using the ruleset rather than using the conventional routing table, and correctly forwards it to your mail server.

The only disadvantage is that clients on the network cannot access the resources otherwise made available on the public IP addresses, but this could be solved by forcing IPv6 only to those servers on the local network (other than the traditional approach of split-horizon DNS). It is also difficult to multihome or implement anycast, since connection tracking is required, but in some cases, static /32 routes can be used instead.

In general,

  • Networks should be designed with public IPv6 and private IPv4 for hosts.
  • When registering a new /24 of IPv4 from ARIN, APNIC, RIPE, or whatever, think of the IP addresses only as service points and not hosts, and then implement the necessary routing rules for services (not hosts) on your public facing router.

The university I go to has three legacy /16's allocated to them. They run a lot of websites on various subdomains of their primary domain. They also have ten IPs for outbound email and two for inbound. Excluding IPs for their network infrastructure on public BGP/routing/peering arrangements, and outbound IPs for the purpose of auditing outbound network connections from dorm rooms, they could still operate even with just one /24, by putting in a load balancer on a public IP for all of their websites, as long as they deploy IPv6, as long as they have less than 256 web servers (note that virtual hosting allows for more than one website on a web server, so the total number of websites can be greater than 256), 256 email servers (again, RCPT TO allows for multiple mail servers to be aggregated, using a relay to send to downstream mail servers on private IPs, so the total number of email domains can be greater than 256), and 256 SSH servers. The fact that none of the websites actually have IPv6 (except for DNS, the VPN, and their dedicated IP transit provider [3]) was just frustrating. Of course, this could never happen in one night, but this also means that if I were to set up a network for a large enterprise, I would definitely set it up in a way that encourages public IPv4 address conservation.

Hmmmmm... maybe if I ever design networking equipment, I think it might be useful to put an FPGA in it, so that the routing mechanisms could be determined entirely by software, yet still reach the capacity of hardware routers with ASICs in them. Maybe allow custom IP cores and VHDL/Verilog code on it as part of custom firmware images such as OpenWRT?

Technically, we have more IP addresses than the University of Illinois. They have:

  • 3 IPv4 /16's + 1 /18 (200k)
  • One /44 (2^84), only two /48's of which are announced
  • Two /48's (2^81)

And we have

  • One /44 (2^84), of which four /48's are announced
  • One /48 (2^80) and one /64 (2^64) (Hurricane Electric tunnel on apps-vm3)
  • About two IPv4 addresses

The connection between the native network and the homelab network

On the home network portion, the homelab to Comcast connection (the line going to the left of the bottom-right dashed portion) is a "transit" link over IPv4, but a "peer" link over IPv6. In other words, the default route goes towards the Comcast network on IPv4, but the IPv6 connection is independent and only the two local network routes are exchanged over the homelab-to-Comcast link.

We do have NAT64 gateways that not only facilitate easier IPv6 deployment, but it also allows us to choose which IPv4 outbound address to use just by selecting the appropriate DNS64 server.

Miscellaneous

One thing to note about our network is that our network infrastructure is more of creative artwork rather than business requirements (although I use it personally to do business [broadly defined] with other people). IP addresses and network design is an attractive target since not only do I know how to do computer networking, but it is an expression which can be easily realized by other people over the Internet simply by connecting to my public servers.

In other words, the peterjin.org Network is sort of like my idealistic view of how an enterprise network should be set up, especially in terms of IPv6 deployment.

See also

Notes

  1. Currently only announced in VMHaus Los Angeles location.
  2. 2.0 2.1 VMHaus only announces prefixes under AS174 (Cogent). Due to peering disputes beyond my control, these prefixes would not be visible on the Hurricane Electric or Google networks. We use a third-party service (AS46997) to announce the entire 2602:806:a002::/47 network as a less specific prefix, whereas the two more specific prefixes comprising it are announced only to Cogent. This allows the Cogent network to be favored while still allowing access from Hurricane Electric. See [1].