My Virtualized Router

This post is mostly for me to remember what I did, but feel free to follow along.


Original Post

Recently I decided to switch jobs, for a number of reasons that aren't germane to this post. I haven't had any proper time off for years so this time I decided to take a big chunk of time between leaving my old job and starting my new one. Two and a half months, to be specific.

I'm spending this time doing a few things. First, I'm being more present with my family. I haven't been the kind of dad or husband that I want to be lately and I'm trying my best to fix that. Second, I fired up my XBox One and started playing Forza Horizon 5. It's ludicrious and mindless in the best possible way.

Third, the topic of this post: I'm building a virtualized router out of a Dell T20 server and a bunch of eBay'd networking gear.

But why?

Yeah, good question. There's a bunch of answers. Comcast is selling us 1.2Gbps service and I want to be able to use it all. I want a reliable failover WAN situation because Comcast goes out for about 5 minutes multiple times a day and that's really annoying in meetings. For example, I was in the middle of a job interview, deep in thought while on camera with two people from a company you've heard of, when Comcast fell over. I tethered to my phone quick but by the time I was reconnected I had lost my train of thought.

Beyond more and better, I want to get some more hands on experience with some tech that I only sort of tangentially know. Specifically, I've been running a Proxmox host for a couple years and it's been solid, but I don't know a lot about the guts. I've also been running a UniFi Security Gateway and then an Edgerouter, but I feel like I don't know how they actually do their job. I also want to play around with a thing called Open vSwitch within Proxmox and this seems like a good opportunity.

Also it's fun and my official job until early 2022 is to follow the dopamine.

Hardware Stack

The hardware is a mix of stuff I had on hand and a few things I've picked up:

  • Dell T20 minitower server with Xeon E3-1225v3 3.2GHz and 32GB of ECC memory
  • Samsung 500GB SSD (side note: SSDs have gotten ridiculously cheap since I last looked at them)
  • Intel X520-DA2 Dual SFP+ 10Gbps network card
  • Wiitek 10Gbase-T SFP+ interface
  • passive SFP+ DAC cable
  • Intel PRO/1000 VT quad port gigabit ethernet card
  • Motorola MB8611 DOCSIS 3.1 multi-gig cable modem
  • Netgear LB1120 LTE modem

The network card decisions deserve some explanation. Comcast gives us a 1.2Gbps cable connection handled by the MB8611 modem. That modem has a 2.5Gbase-T ethernet connection.

One way to handle this would have been a 2.5Gbase-T ethernet card in the router. This would have been a little cheaper but the fast connection would have ended at the router. I want to share the speed with my other server so I need another faster-than-gigabit port while also preserving the ability to, one day, maybe, upgrade to Comcast Gigabit Pro 3 Gbps fiber service. I'm also constrained by the T20's selection of three PCIe slots: 16x, 4x, and 1x. If I tried to do something with 2.5Gbase-T cards I probably would have run out of slots, but with the dual port SFP+ card in the 16x slot and the quad gigabit card in the 4x slot I'm fine.

I actually got two of those dual SFP+ cards, one for the router and another for my other server which will cross-connect with a passive DAC (directly attached copper) cable.

Software Stack

After putting the SFP+ card and SSD into the server I installed Proxmox VE 7.1 on the server and got started evaluating router distros.

The very first thing I installed was OPNSense, a FreeBSD-derived routing and firewall system. It came up wanting to be on the same IP as our current router ( which was problematic for a bit until I figured out what was going on. After installing it I clicked around a bit and read some docs and decided that I wasn't really going to learn what I wanted to learn from it.

Next I installed VyOS, a routing and firewall package derived from Debian and Vyatta, which itself was strongly inspired by Juniper's router OS. This was sort of bewildering and overwhelming and after messing around a bit I moved onto the next thing.

Third, I installed NixOS and futzed around with the config. NixOS is a linux distribution that uses Nix to deploy and configure software. This is interesting but also weird and it doesn't get me multi-WAN failover out of the box. I'd have to build that myself, which is not super appealing.

I think what I'm going to do is reinstall VyOS and actually commit to learning how the CLI works. It gets me everything I want out of the box, it's just slightly more inscruitable.

I'm also planning on running a couple of ancilliary "network service" type VMs on this machine:

  • pi-hole for DNS and network-wide ad blocking
  • Unifi controller for our existing Unifi gear (mostly APs, some switches)
  • Ingress nginx proxy (this probably deserves it's own post)

Open vSwitch

One additional thing that I want to play with is Open vSwitch. This is a software network switch that lives inside Proxmox and ties everything together. It acts like a L3 hardware switch, just implemented entirely in software. It's optional within Proxmox but from what I've read it gives significantly better performace, which is aesthetically attractive if not strictly necessary. Nothing about this project is strictly necessary, though, so I feel justified.

What's next?

  • Set up a basic Open vSwitch configuration within Proxmox
  • Install VyOS and get it working as a basic router
  • When the quad port ethernet card arrives, install it and hook it up to the vSwitch and VyOS
  • Roll out to production!?

Progress Update 2021-11-25

I accomplished a couple of things last night and today:

I got a basic Open vSwitch config working within Proxmox! This was a bit of an ordeal because I installed a package that really shouldn't be installed, because apparently it breaks the entire network stack if you install it. So, protip, just do what the tutorial says and don't get fancy.

Here's the config, for posterity:

auto lo
iface lo inet loopback

# LAN interface, auto-tagged as VLAN-1
auto eno1
iface eno1 inet manual
    ovs_type OVSPort
    ovs_bridge vmbr0
    ovs_options vlan_mode=native-untagged tag=1

# WAN1 10GBase-T SFP+ Module, auto-tagged as VLAN-100
auto enp1s0f1
iface enp1s0f1 inet manual
    ovs_type OVSPort
    ovs_bridge vmbr0
    ovs_options vlan_mode=native_untagged tag=100

# Internal interface for the hypervisor itself attached to VLAN-1
auto vlan1
iface vlan1 inet static
    ovs_type OVSIntPort
    ovs_bridge vmbr0
    ovs_mtu 1500
    ovs_options tag=1

# Just one OVSBridge, the software equivalent of an L3 managed switch
auto vmbr0
iface vmbr0 inet manual
    ovs_type OVSBridge
    ovs_ports eno1 enp1s0f1 vlan1

OVS on Proxmox works like this:

  • Traffic comes into a OVSBridge through physical ports, represented by OVSPorts, as well as OVSIntPorts. These ports can have packets come in already tagged or tag them themselves (or both?)
  • Traffic to and from VMs transits via the virtual network adapters attached to each VM. These network adapters can be assigned to a VLAN or not. If not, they're assumed to be a trunk port (all VLANs).
  • The only things with IP addresses assigned are OVSIntPorts and the VM NICs.

I set up two LXC containers, one for pi-hole and another for the Unifi console. After setting up pi-hole I went into the Edgerouter config and told it to send DNS traffic there, and boy howdy is it interesting how much trash the various things on the network are talking to.

The Unifi console was a bit of work. I installed it using the 5.14.23 script from here and then restored a backup from my remote console. After that I told the Edgerouter to broadcast the new console's IP as the inform URL, which mostly worked. I had to forget one AP from the old controller and re-adopt it on the new one, and one of the switches (which happens to be the basement switch that sits in the critical path between the house where I was sitting and the office where the servers are) needed some hand holding, i.e. SSH'ing into it and running set-inform manually.

Tonight I set up Proxmox to relay email through Postmark with this tutorial and set up a scheduled weekly snapshot of all the VMs on the machine.

Want more? Get emails from me about coding, computers, finance, and business.

I will never send you spam. You can unsubscribe at any time.