welcome to le-fay.dn42, AS4242421495.
i am accepting peerings in the UK & Ireland, North America, and mainland Europe. please see the list of nodes for information about nodes accepting new peerings. all nodes support multiprotocol BGP with extended next hop for IPv4 prefixes; BGP itself is only supported over link-local IPv6 addresses.
in the UK, please set up one peering with either uk-myb-2 or uk-jmp-2. in all other locations, you may peer with every node, but i'll only accept one peering for each of your nodes (i.e., you cannot peer between two of my nodes and one of your nodes).
peering requests with an excessively high rtt may be rejected to preserve network quality.
to request a peering, please mail ft.dn42@le-fay.org and include:
routes advertised by AS4242421495 may have one or more of the following communities attached:
| community | meaning |
|---|---|
| general | |
| 4242421495:1:1 | route origin: internal (AS4242421495) |
| 4242421495:1:2* | route origin: external, Internet (DFZ) |
| 4242421495:1:42 | route origin: external, DN42 |
| 4242421495:4:1* | internal route policy: export to Internet peers |
| 4242421495:4:2* | internal route policy: export to internal peers only |
| 4242421495:4:42 | internal route policy: export to DN42 |
| 4242421495:5:1 | aggregated route (internal route policy: do not install to FIB) |
| node-specific | |
| 4242421495:2:1 | route originated or received by ny-czy-1 |
| 4242421495:2:2 | route originated or received by mia-czy-1 |
| 4242421495:2:3 | route originated or received by fre-myb-1 |
| 4242421495:2:4 | route originated or received by nl-myb-1 |
| 4242421495:2:5 | route originated or received by uk-myb-2 |
| 4242421495:2:6 | route originated or received by uk-myb-1 |
| 4242421495:2:7 | route originated or received by uk-jmp-2 |
| 4242421495:2:8 | route originated or received by uk-jmp-1 |
| 4242421495:2:9 | route originated or received by willow |
| 4242421495:2:10 | route originated or received by uk-aai-1 |
| region-specific | |
| 4242421495:3:1 | route originated or received in UK region |
| 4242421495:3:2 | route originated or received in NA region |
| 4242421495:3:3 | route originated or received in EU region |
| 4242421495:3:4 | route originated or received in Eden |
* - these communities should never be seen on DN42
the node-specific and region-specific communities are only attached when the route is originated or received from an external peer. for example, a route tagged with 4242421495:2:1 was either received by ny-czy-1 from an external EBGP peer, or was locally originated on that node. therefore, any route should only have a single node-specific and region-specific community.
you will only see the relevant communities for the best route learned by the node you peer with. for example, if you peer with nl-myb-1 and you want to ignore routes that AS4242421495 only learned in the US, you can filter routes tagged with 4242421495:3:2; you will still receive routes that we learned in the UK or EU, even if we also learned the route in the US.
[DN42] [DN42]
│ │
╭──────ny-czy-1 ╭───────────────────nl-myb-1
| │ | │ │
[DN42]────fre-myb-1 │ ╰─────────────────────╮ ╭─────────╯
| │ │ │ │
╰──────mia-czy-1────uk-myb-2─────uk-jmp-2────[DN42]
│ │ │ │
[DN42] [DN42] │ │
│ │
yarrow────uk-myb-1 uk-jmp-1────amaranth
│ │
│ │
│ │
╰──uk-aai-1──╯
│
│
│
willow
nodes are virtual machines running on KVM or Xen at commercial VPS providers, except for uk-aai-1, which runs on physical hardware, a fanless Denverton box with 4x 10GbE and 5x 1GbE ports.
nodes themselves are FreeBSD VNET jails underneath the host VM. the external network interface (e.g. vtnet0 or xn0) is exported to the jail, and Wireguard interfaces are created directly inside the jail. each jail also has one side of an epair(4) connection to provide network connectivity to the host.
each host VM runs a custom build of FreeBSD 15.0-CURRENT pkgbase. since the external interface is exported to the node jail, the host itself uses the other end of the epair(4) interface for its network/Internet connectivity; the hosts have no direct network connection other than via the node jail.
all nodes run OSPF over clearnet Wireguard links to handle internal routing. OSPF carries GUA (clearnet) router loopback addresses for each node, as well as internal GUA routes and our internal DN42 routes, but never carries external DN42 routes. OSPF cost is set manually at link-latency * 10, so a link with 13.5ms latency has a cost of 135.
all nodes run IBGP in a full mesh to distribute routes learned from DN42 EBGP peers. since IBGP learns all available routes to a particular network from the other IBGP routers, it uses IGP (OSPF) cost to select the next-hop. this means packets will take the shortest available path (in terms of latency) to their destination, either internally or to a local EBGP peer (since EBGP routes are preferred over IBGP).
routers with external DN42 peerings run EBGP with those peers over link-local IPv6 addresses. EBGP accepts peer routes of length /44../64, and advertises all routes learned from IBGP, which includes transit routes from other EBGP routers learned over IBGP.
first, we add the host's local IP addresses (GUA and DN42) as loopback addresses in /etc/rc.conf:
ifconfig_lo0_alias0="inet6 2a00:1098:6b:100::2/128" ifconfig_lo0_alias1="inet6 fd5b:a83:b06b:4::1:1/128"
then we need to create an epair(4) interface for traffic between the host and the node jail. this is done in /etc/rc.conf; for example on yarrow, which is the host for uk-myb-1:
cloned_interfaces="epair0" ifconfig_epair0a_name="ep.yarrow" ifconfig_epair0b_name="ep.uk-myb-1" ifconfig_ep_uk_myb_1_ipv6="inet6 fe80::2/64" ipv6_defaultrouter="fe80::1%ep.uk-myb-1"
this creates two new virtual Ethernet interfaces, "ep.uk-myb-1" which will stay on the host, and "ep.yarrow" which will be exported to the jail. to make routing easier, both sides have statically configured IPv6 link-local addresses, and the host uses the node jail's address as its default route.
if the host needs IPv4 connectivity, we also add the IPv4 configuration:
ifconfig_ep_uk_myb_1="inet 176.126.243.79/32" # -inet6 is required here to use an IPv6 address as IPv4 gateway defaultrouter="-inet6 fe80::1%ep.uk-myb-1"
finally, we configure the node jail in /etc/jail.conf
uk-myb-1 {
host.hostname "uk-myb-1.le-fay.org";
vnet new;
vnet.interface = vtnet0, ep.yarrow;
devfs_ruleset 6;
}
devfs ruleset 6 is a ruleset that enables BPF, so that tcpdump works in the jail, configured in /etc/devfs.rules:
[devfsrules_jail_bpf=6] add include $devfsrules_hide_all add include $devfsrules_unhide_basic add include $devfsrules_unhide_login add include $devfsrules_jail add include $devfsrules_jail_vnet add path 'bpf*' unhide
first we configure the node's loopback addresses and the epair link to the host (but we don't add a route here, since this is done in BIRD):
ifconfig_lo0_alias0="inet6 2a00:1098:6b:100::1/128 alias" ifconfig_lo0_alias1="inet6 fd5b:a83:b06b:4::1:2/128 alias" ifconfig_ep_yarrow_ipv6="inet6 fe80::1/64"
then we configure the Internet interface. this is in group "arpa" (for pf firewalling), and since we are a router, we disable LRO and TSO on the interface.
ifconfig_vtnet0_ipv6="inet6 auto_linklocal 2a00:1098:6b::1/128 group arpa -lro -tso"
and our internal and DN42 Wireguard tunnels. again, for pf firewalling, internal tunnels are in group "lf" and "ospf", and external peers are in "dn42peer".
cloned_interfaces="wg0 wg1 wg2 wg3 wg4" ifconfig_wg0_name="wg.uk-myb-2" ifconfig_wg1_name="wg.uk-aai-1" ifconfig_wg2_name="wg.burble" ifconfig_wg3_name="wg.n9" ifconfig_wg4_name="wg.g-load" ifconfig_wg_uk_myb_2_ipv6="inet6 auto_linklocal tunnelfib 1 group ospf group lf" ifconfig_wg_uk_aai_1_ipv6="inet6 auto_linklocal tunnelfib 1 group ospf group lf" ifconfig_wg_burble_ipv6="inet6 fe80::1/64 group dn42peer" ifconfig_wg_n9_ipv6="inet6 fe80::1/64 group dn42peer" ifconfig_wg_g_load_ipv6="inet6 fe80::1/64 group dn42peer"
each Wireguard tunnel has a corresponding start script which loads the Wireguard configuration, e.g. /etc/start_if.wg.uk-myb-2:
#! /bin/sh /usr/bin/wg setconf wg.uk-myb-2 /etc/wg/wg.uk-myb-2.conf
finally, we configure the system's static routes in bird.conf. this includes the static route for the host, a route for the NAT64 network 64:ff9b::/96 (if this node is a NAT64 gateway), an Internet route, and a blackhole route for our DN42 /48 prefix.
we also add a route for Internet access via fib 1 here, which is for Wireguard tunnels to our internal nodes. without this, as soon as we learned a route to those nodes via OSPF, Wireguard would try to send the Wireguard traffic over the Wireguard tunnel itself, which would not work. putting Wireguard in fib 1 ensures this traffic always goes over the Internet.
# IPv6 default table (fib 0).
protocol static static0v6 {
ipv6 {
table master6;
import all;
};
# Internet via Mythic Beasts
route 2000::/3 via fe80::1%vtnet0;
# Blackhole our local networks to prevent routing loops.
route 2a00:1098:6b:100::/56 prohibit;
route fd5b:a83:b06b:4::1:0/112 prohibit;
# yarrow
route 2a00:1098:6b:100::2/128 via fe80::2%'ep.yarrow';
route fd5b:a83:b06b:4::1:1/128 via fe80::2%'ep.yarrow';
# NAT64
route 64:ff9b::/96 blackhole;
# DN42
route fd5b:a83:b06b::/48 prohibit;
}
# IPv6 Wireguard table (fib 1).
protocol static static1v6 {
ipv6 {
table fib1v6;
import all;
};
route 2000::/3 via fe80::1%vtnet0;
}
OSPF configuration in BIRD is fairly straightforward. we configure the interface cost based on link latency in ms * 10; for example the latency from uk-myb-1 to uk-myb-2 is 0.8ms, so the link cost is 8. the LF6 prefix list contains the list of local addresses we want to advertise into OSPF.
define LF6 = [
2a00:1098:6b::/64+,
2a00:1098:6b:100::/56,
fd5b:a83:b06b:4::1:0/112
];
protocol ospf v3 lf6 {
ipv6 {
import all;
export filter {
if net ~ LF6 then accept;
reject;
};
};
area 0 {
interface "wg.uk-aai-1" {
authentication cryptographic;
password "...".
type ptp;
cost 69;
};
interface "wg.uk-myb-2" {
authentication cryptographic;
password "..."
type ptp;
cost 8;
};
};
}
IBGP peers are configured using a BIRD template. IBGP forms a full mesh of all edge routers, so we configure all peers here. the export filter is configured to only export DN42 routes that we received from BGP peers; we don't want internal routes in IBGP.
template bgp lfibgp {
local 2a00:1098:6b:100::1 as MYASN;
neighbor internal;
path metric 1;
strict bind yes;
ipv6 {
next hop self;
extended next hop yes;
import all;
export filter {
if (source = RTS_BGP) && (net ~ [ fd00::/8{44,64} ]) then accept;
reject;
};
};
}
protocol bgp 'uk-aai-1' from lfibgp {
neighbor 2001:8b0:aab5:100::1;
}
protocol bgp 'uk-myb-2' from lfibgp {
neighbor 2a00:1098:6b:200::1;
}
# etc, etc...