IPv6 Things/Configuration

This page lists all of the configuration files used for the IPv6 Things project. Many of these configuration files are highly site-specific, so you will need to make a lot of changes if you want to self host. The contents here are provided for reference.

DO NOT COPY AND PASTE THESE SCRIPTS VERBATIM INTO YOUR OWN SERVER!!! IT WILL NOT WORK!!! Instead, use these scripts as a guideline, and modify anything needed to adapt the scripts into your own environment (IP addresses, filesystem layout, etc.).

Summary

The following scripts can be used to create the nginx frontend and the node.js backend. Using the AnyIP trick, we assign the entire /64 to our server. The nginx frontend is capable of determining which IP address within that /64 the incoming connection was directed to, and can change its "personality" (SSL certificate, document root, etc.) based on this IP address. The nginx server performs the "categorical" sorting (i.e. group by subnet), whereas the node.js backend interprets individual IP addresses. The node.js backend runs in an LXC-like unprivileged container with user namespaces to limit damage in case of a security vulnerability. The frontend and backend are linked together with proxy_pass and Unix domain sockets.

Node.js startup script

The following lines of code will initialize the Node.js environment. We use Python for simplicity and because it is both universally available and straight to the point as to what it does: create and bind three listening Unix domain sockets, which the nginx frontend will use to gain access to the node.js backend. Additionally, another socket will be created for the Traceroute Text Generator's telnet functionality; unlike the other Unix domain sockets, this one is accessed directly. The file descriptors were meant to be inherited all the way through to the Node.js program. where environment variables (os.setenv in Python, process.env in Node.js) are used to access the file descriptor numbers.

We create the sockets at this stage because the creation of the sockets requires root access. After that, the process drops privileges and enters a container-like chroot; despite this, it can still listen for and accept incoming connections.

#!/bin/sh
rm -f /containers/nginx-40e/run/*.sock
exec nsenter --net=/run/netns/nginx-40e python3 <<\EOF
import socket
import os
import stat

s_bible_http = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s_bible_http.bind("/containers/nginx-40e/run/ipv6bible.sock")
s_bible_http.set_inheritable(True);

s_bible_telnet = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s_bible_telnet.bind("/containers/nginx-40e/run/ipv6bible-telnet.sock")
s_bible_telnet.set_inheritable(True);

s_i6t_http = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s_i6t_http.bind("/containers/nginx-40e/run/ipv6things.sock")
s_i6t_http.set_inheritable(True);

s_telnet_server = socket.socket(socket.AF_INET6, socket.SOCK_STREAM);
s_telnet_server.setsockopt(41, 75, 1) # set IPV6_TRANSPARENT
s_telnet_server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s_telnet_server.bind(("::ffff:127.0.0.1", 1))
s_telnet_server.set_inheritable(True)

os.chown("/containers/nginx-40e/run/ipv6bible.sock", 0, 124)
os.chmod("/containers/nginx-40e/run/ipv6bible.sock", stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IWGRP)
os.chown("/containers/nginx-40e/run/ipv6things.sock", 0, 124)
os.chmod("/containers/nginx-40e/run/ipv6things.sock", stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IWGRP)
os.chown("/containers/nginx-40e/run/ipv6bible-telnet.sock", 0, 124)
os.chmod("/containers/nginx-40e/run/ipv6bible-telnet.sock", stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IWGRP)

os.putenv("VM5_NODE_LISTEN_FD", str(s_bible_http.fileno()))
os.putenv("VM5_NODE_LISTEN_FD_IPV6THINGS", str(s_i6t_http.fileno()))
os.putenv("VM5_NODE_TELNET_FD", str(s_bible_telnet.fileno()))
os.environ["TTGEN_TELNET_LISTEN_FD"] = str(s_telnet_server.fileno());
os.putenv("NODE_ENV", "production")
os.execv("/bin/sh", ["/bin/sh", "/containers/nginx-40e/conf/part2.sh", "node", "/root/index.js"])
EOF

part2.sh

This script mainly sets up the container and its filesystem environment, as well as changing to the "node-nginx-40e" user (UID/GID 118/125)

#!/bin/sh

exec setpriv --reuid=118 --regid=125 --clear-groups unshare -r -m -p -i --fork --propagation slave --mount-proc sh -s "$@" <<\EOF
set -eu
ip link set lo up || true
mount -t tmpfs -o mode=0755 none /proc/driver
cd /proc/driver
mkdir -p dev/pts dev/mqueue fsroot home oldroot proc root run sys tmp
for x in bin etc lib lib64 opt sbin usr var; do ln -s fsroot/"$x"; done
for x in full null random tty urandom zero; do
	touch dev/"$x"
	mount --bind /dev/"$x" dev/"$x"
done
mount -t devpts -o mode=0600,ptmxmode=0666,newinstance none dev/pts
mount -t mqueue none dev/mqueue
mount --rbind /containers/nginx-40e/home root
mount --rbind /containers/mounts/node fsroot
mount -t sysfs none sys || mount --rbind /sys sys
mount -t proc none proc
echo nameserver 2606:4700:4700::1111 > resolv.conf
mount --rbind ./resolv.conf etc/resolv.conf
# mount --bind /containers/nginx-40e/conf/resolv /run/systemd/resolve
cp /usr/bin/setpriv .
pivot_root . oldroot
exec /bin/sh -c 'umount -l /oldroot && exec setsid /setpriv --bounding-set=-all "$@" </dev/null' - "$@" >/oldroot/containers/nginx-40e/logs/node.log 2>&1
EOF

/containers/nginx-40e/home/index.js

require("./ipv6-things/index.js");
require("./public/index-prod.js");
require("./ttgen-telnet/telnet-prod.js");

Creating and mounting the node.squashfs file

#!/bin/sh

sudo docker run -v /docker-files/node:/buildfiles node:12-slim sh -c 'tar c /bin /etc /lib /lib64 /opt /sbin /usr /var > /buildfiles/node.tar'
unshare -r -m --propagation=slave sh -c 'mount -t tmpfs -o mode=0755 none /tmp && mkdir /tmp/node-build && (cd /tmp/node-build && bsdtar -x --chroot --no-same-owner --no-same-permissions -f /docker-files/node/node.tar && tee etc/resolv.conf etc/hosts etc/hostname </dev/null) && mksquashfs /tmp/node-build node.squashfs -comp lz4 -b 1048576 -Xhc'
mount node.squashfs /containers/mounts/node

Network configuration

This section is highly specific to our environment. You will need to make changes to the IP addresses if you want to self host. The most important part is the "ip route add local" line, which allows us to perform the AnyIP trick.

ip link set lo up
ip route add local 2602:806:a003:40e::/64 dev lo
ip route add local 2602:806:a003:40f::/64 dev lo
# ip addr add fe80::4/64 dev eth0
# ip addr add 2602:806:a003:4ff::ece:329/96 dev eth0
# ip addr add 172.19.28.3/28 dev eth0
# ip link set eth0 up
# ip route add ::/0 via fe80::1 dev eth0
# ip route add 0.0.0.0/0 via 172.19.28.1
ip6tables-nft -t mangle -A PREROUTING -i eth0 -d 2602:806:a003:40f::/64 -p tcp --dport 23 -j TPROXY --on-ip ::ffff:127.0.0.1 --on-port 1
iptables-nft -t mangle -A PREROUTING -i eth0 -d 172.19.28.3 -p tcp --dport 23 -j TPROXY --on-ip 127.0.0.1 --on-port 1

For a better explanation of the last two lines, see Socketbox#Dual-stack TPROXY.

nginx.conf

user nginx-40e;
worker_processes auto;
pid /run/nginx.pid;
# include /etc/nginx/modules-enabled/*.conf;
load_module /usr/share/nginx/modules/ngx_stream_module.so;

events {
	worker_connections 768;
}

http {
	sendfile on;
	tcp_nopush on;
	tcp_nodelay on;
	keepalive_timeout 65;
	types_hash_max_size 2048;
	include /etc/nginx/mime.types;
	default_type application/octet-stream;
	ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
	ssl_ciphers HIGH:!aNULL:!MD5:!AESCCM;
	ssl_prefer_server_ciphers on;
	ssl_session_cache shared:X:10m;
	ssl_stapling on;
	log_format custom '$remote_addr/$http_cf_connecting_ip ($ssl_protocol $ssl_cipher) - $remote_user [$time_iso8601] $host[$server_addr] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"';
	access_log /containers/nginx-40e/logs/access.log custom;
	error_log /containers/nginx-40e/logs/error.log;
	add_header Strict-Transport-Security "max-age=31536000" always;
	root /var/www/empty;
	gzip on;
	geo "$server_addr" $proxy_socket {
		default unix:/containers/nginx-40e/sockets/ipv6things.sock:;
		2602:806:a003:40e:b1be::/80 unix:/containers/nginx-40e/sockets/ipv6bible.sock:;
	}
	geo "$server_addr" $ssl_cert_match {
		default aliases;
		2602:806:a003:40e::1:0/112 ipv6-things-dotcom;
		2602:806:a003:40e:b1be::/80 ipv6bible;
		2602:806:a003:40f::/64 traceroute;
		2602:806:a003:40f::4:1 mbloo-ttgen;
		2602:806:a003:40f::4:3 mbloo-ttgen;
		2602:806:a003:40f:0:1:2:ff01 mbloo-ttgen;
	}
	map "$remote_addr" $remote_addr_string {
		default "\"unknown\"";
		"~^([0-9a-f:]*)$" "\"$1\"";
	}
	geo "$server_addr" $alternative_root {
		default '';
		2602:806:a003:40f::4:0/112 ttgen-videos/bonnie;
		2602:806:a003:40f:0:1:1:0/112 ttgen-videos/mountain;
		2602:806:a003:40f:0:1:2:0/112 ttgen-videos/bonnie;
	}
	map "$time_iso8601" $my_time_simple_iso8601 {
		default "unknown";
		"~([0-9-]+)T([0-9:]+)\+00:00" "$1 $2";
	}
	server {
		listen [::]:80 ipv6only=on;
		server_name .ipv6.bible .peterjin.org .ipv6-things.com;
		location / {
			return 301 https://$host$request_uri;
		}
		location /.well-known/acme-challenge {
			root /run/acme-challenge;
		}
	}
	server {
		listen [::]:80 default_server;
		include /containers/nginx-40e/conf/normal-ipv6.conf;
		location /.well-known/acme-challenge {
			root /run/acme-challenge;
		}
	}
	server {
		listen [::]:443 ssl http2 default_server ipv6only=on;
		ssl_certificate "/containers/nginx-40e/ssl/$ssl_cert_match.crt";
		ssl_certificate_key "/containers/nginx-40e/ssl/$ssl_cert_match.key";
		include /containers/nginx-40e/conf/normal-ipv6.conf;
	}
	server {
		server_name "~^i6t-cf-poc-[0-9]+\.peterjin\.org$";
		listen [::]:443 ssl http2;
		ssl_certificate "/containers/nginx-40e/ssl/i6t-poc.crt";
		ssl_certificate_key "/containers/nginx-40e/ssl/i6t-poc.key";
		include /containers/nginx-40e/conf/normal-ipv6.conf;
	}
}
stream {
	geo "$server_addr" $proxy_socket_telnet {
		default '';
		2602:806:a003:40e:b1be::/80 unix:/containers/nginx-40e/sockets/ipv6bible-telnet.sock;
		2602:806:a003:40e::3000:0/100 unix:/containers/nginx-40e/sockets/aliases3k-telnet.sock;
	}
	server {
		listen [::]:23;
		listen [::]:2323;
		proxy_pass $proxy_socket_telnet;
		proxy_protocol on;
	}
}

normal-ipv6.conf

location / {
	if ($alternative_root = "") {
		add_trailer Content-Security-Policy "default-src 'self'; style-src 'unsafe-inline'";
		proxy_pass http://$proxy_socket;
	}
	if ($alternative_root) {
		root "/containers/nginx-40e/x_homes/$alternative_root";
	}
	proxy_set_header X-Server-IP "$server_addr";
	proxy_set_header Host "$host";
	proxy_set_header X-Ipv6things-LocalIP "$server_addr";
	proxy_set_header X-Ipv6things-RemoteIP "$remote_addr";
	add_header X-Frame-Options DENY always;
	add_header X-Content-Type-Options nosniff always;
	add_header Referrer-Policy strict-origin-when-cross-origin always;
	add_header X-XSS-Protection "1; mode=block" always;
	add_header Strict-Transport-Security "max-age=31536000" always;
}
root /var/www/empty;
location = /ipv6_check.js {
	default_type text/javascript;
	types { }
	add_header Cache-Control "no-cache, no-store, max-age=0, must-revalidate, private";
	return 200 "var __pjo_ipv6_address = $remote_addr_string;\n";
}
location = /ipv6_check.css {
	default_type text/css;
	types { }
	add_header Cache-Control "max-age=10, must-revalidate, private";
	return 200 ".__pjo_ipv6_banner { display: block; }\n.__pjo_no_ipv6_banner { display: none; }";
}