Nebula is a cool mesh VPN/SDWAN(ish) project that spawned out of Slack. It’s a VPN in the sense that it connects multiple hosts across the internet (or other networks), but not in the traditional client-server sense. Instead, it’s a self-contained binary that does peer-to-peer networking (securely) across multiple devices.

In this regard, it’s similar to other VPN(ish) projects, like:

The cool part is, it’s totally open source, and they supply binaries for just about every platform you might like to run it on. Or you can build it from source.

Each Nebula network has at least one “lighthouse,” which serves as the coordination “server” for a network. It serves as a kind of registry to help nodes find each other on the network, and to coordinate security through PKI.

Every node has a configuration file that points it to the lighthouse, as well as defines things like ACLs and static peers.

Nodes are issued private certificates by the CA (created first) and use these certificates to prove their identity, establish IP addresses (and external/”unsafe” routes).

Lighthouse

Let’s start by setting up a Lighthouse on a UI EdgeRouter.

First, we need to ssh into the device and enable the installation of packages from the debian repositories. This will make life easier when we need to install tools like wget.

UI has instructions here.

[email protected]:/home/ubnt# configure
[edit]
[email protected]# set system package repository stretch components 'main contrib non-free' 
[edit]
[email protected]# set system package repository stretch distribution stretch
[edit]
[email protected]# set system package repository stretch url http://http.us.debian.org/debian
[edit]
[email protected]# commit; save
Saving configuration to '/config/config.boot'...
Done

Now we need to update apt:

[email protected]:/home/ubnt# apt update
Ign:1 http://http.us.debian.org/debian stretch InRelease
Get:2 http://http.us.debian.org/debian stretch Release [118 kB]
Get:3 http://http.us.debian.org/debian stretch Release.gpg [3177 B]
Get:4 http://http.us.debian.org/debian stretch/main mips Packages [6841 kB]
Get:5 http://http.us.debian.org/debian stretch/main Translation-en [5377 kB]
Get:6 http://http.us.debian.org/debian stretch/contrib mips Packages [39.9 kB]
Get:7 http://http.us.debian.org/debian stretch/contrib Translation-en [45.8 kB]
Get:8 http://http.us.debian.org/debian stretch/non-free mips Packages [50.6 kB]
Get:9 http://http.us.debian.org/debian stretch/non-free Translation-en [80.2 kB]
Fetched 12.6 MB in 43s (290 kB/s)                                                                                                                                                                                                                                                 
Reading package lists... Done
Building dependency tree... Done
22 packages can be upgraded. Run 'apt list --upgradable' to see them.

Now let’s install wget so we can download the prebuilt Nebula binaries:

[email protected]:/home/ubnt# apt install wget
Reading package lists... Done
Building dependency tree... Done
The following NEW packages will be installed:
  wget
0 upgraded, 1 newly installed, 0 to remove and 22 not upgraded.
Need to get 798 kB of archives.
After this operation, 2871 kB of additional disk space will be used.
Get:1 http://http.us.debian.org/debian stretch/main mips wget mips 1.18-5+deb9u3 [798 kB]
Fetched 798 kB in 1s (454 kB/s)      
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package wget.
(Reading database ... 37013 files and directories currently installed.)
Preparing to unpack .../wget_1.18-5+deb9u3_mips.deb ...
Unpacking wget (1.18-5+deb9u3) ...
Setting up wget (1.18-5+deb9u3) ...
[email protected]:/home/ubnt#

All set to download the binaries. We know the EdgeRouter is a MIPS64 platform:

[email protected]:~$ sudo uname -a
Linux br-edge 4.9.79-UBNT #1 SMP Thu Mar 5 16:48:33 UTC 2020 mips64 GNU/Linux

So let’s download and extract the MIPS64 Nebula binary. Releases are here.

[email protected]:/home/ubnt# wget https://github.com/slackhq/nebula/releases/download/v1.5.2/nebula-linux-mips64.tar.gz
--2022-04-16 15:53:55--  https://github.com/slackhq/nebula/releases/download/v1.5.2/nebula-linux-mips64.tar.gz
Resolving github.com (github.com)... 140.82.114.4
Connecting to github.com (github.com)|140.82.114.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/222172014/6acb2e5c-652f-4511-9b34-afe8e5fcb21f?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220416%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220416T155358Z&X-Amz-Expires=300&X-Amz-Signature=a0509180549f92573e36e3357d6d551dbe692b0ef40a29a453048fa88a2c1527&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=222172014&response-content-disposition=attachment%3B%20filename%3Dnebula-linux-mips64.tar.gz&response-content-type=application%2Foctet-stream [following]
--2022-04-16 15:53:58--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/222172014/6acb2e5c-652f-4511-9b34-afe8e5fcb21f?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220416%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220416T155358Z&X-Amz-Expires=300&X-Amz-Signature=a0509180549f92573e36e3357d6d551dbe692b0ef40a29a453048fa88a2c1527&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=222172014&response-content-disposition=attachment%3B%20filename%3Dnebula-linux-mips64.tar.gz&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.109.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10887563 (10M) [application/octet-stream]
Saving to: 'nebula-linux-mips64.tar.gz'

nebula-linux-mips64.tar.gz                                           100%[=====================================================================================================================================================================>]  10.38M  6.72MB/s    in 1.5s    

2022-04-16 15:53:59 (6.72 MB/s) - 'nebula-linux-mips64.tar.gz' saved [10887563/10887563]

[email protected]:/home/ubnt# tar -xvf nebula-linux-mips64.tar.gz 
nebula
nebula-cert
[email protected]:/home/ubnt# ls
nebula                      nebula-cert                 nebula-linux-mips64.tar.gz

Now we have the nebula and nebula-cert binaries ready to go. We can follow the Quick Start Guide and start making some certs.

Certificates

First, let’s create a CA cert (I did this on the router, but you should probably do it on a secure system). As the guide states, the resulting ca.key is the most important file – it is essentially the keys to the kingdom. You will use it to create certs for all other hosts, and it should be protected like the “root key” that it is.

It does NOT need to stay on the Lighthouse, or wherever you generate it. Just keep it safe, because you’ll need it in the future to generate any new node certificates.

[email protected]:/home/ubnt# ./nebula-cert ca -name "mynetwork"

This will create two files: ca.crt (the public key) and ca.key (the private key). The former goes on all nodes to establish trust, and the latter is used to sign all new certs you will make for hosts.

To start, we’ll only have a lighthouse and one node. Let’s create certificates for them.

Lighthouse

I’m going to use a 192.168.254.0/24 subnet for my network, but you can choose whatever you like. This network should be separate and distinct from any other networks you usually connect to so there aren’t any conflicting routes.

./nebula-cert sign -name "lighthouse" -ip "192.168.254.1/24" -subnets "192.168.17.0/24"

Note that I used the subnets option. This allows the host (in this case, the Lighthouse) to route “unsafe” traffic to another local subnet. So, we’ll be able to use the Lighthouse to route traffic from the Nebula network onto the non-Nebula local network.

Out comes two files: lighthouse.key and lighthouse.crt. These will go on the Lightouse (in this case, the router).

First Host

Next, create a cert for the first host. Repeat as many times as necessary for each node that will join the Nebula nework. Obviously, change the IPs. I’ll call mine mbp.

./nebula-cert sign -name "mbp" -ip "192.168.254.2/24" -groups "laptop"

The “laptop” group will be used later to apply ACLs for traffic from this device (and others with the same group).

Configuration File

Now we need to create the configuration file the Lighthouse will use. Start with the example file here.

We’ll make the following changes. Find the related sections in the example and update them to match your config.

pki:
  ca: /home/ubnt/ca.crt
  cert: /home/ubnt/lighthouse.crt
  key: /home/ubnt/lighthouse.key

static_host_map:
  "192.168.254.1": ["<public IP of edgerouter>:4242"]

lighthouse:
  am_lighthouse: true

hosts:
  <should be empty>

tun:
  unsafe_routes:
    - route: 192.168.17.0/24
      via: 192.168.254.1
      mtu: 1300
      metric: 100
firewall:
  outbound:
    # Allow all outbound traffic from this node
    - port: any
      proto: any
      host: any

  inbound:
    # Allow icmp between any nebula hosts
    - port: any
      proto: icmp
      host: any

    # Allow any host with laptop group
    - port: any
      proto: any
      groups:
        - laptop

Note that I kept the cert files in /home/ubnt out of pure laziness. They should probably go somewhere else (/var/lib?), but this will work.

The unsafe routes match what we added in the certificate for the router – this will let us route traffic out the EdgeRouter to the local (non-Nebula) network.

This firewall configuration is pretty insecure – but we can lock it down later. For now we’ll leave it open for testing.

Let’s test the config and see how we did:

[email protected]:/home/ubnt# ./nebula -test -config router.conf 
<config info will be printed here>
INFO[0000] Firewall rule added                           firewallRule="map[caName: caSha: direction:outgoing endPort:0 groups:[] host:any ip: proto:0 startPort:0]"
INFO[0000] Firewall rule added                           firewallRule="map[caName: caSha: direction:incoming endPort:0 groups:[] host:any ip: proto:1 startPort:0]"
INFO[0000] Firewall rule added                           firewallRule="map[caName: caSha: direction:incoming endPort:0 groups:[laptop] host: ip: proto:0 startPort:0]"
INFO[0000] Firewall started                              firewallHash=41...12
INFO[0000] Main HostMap created                          network=192.168.254.1/24 preferredRanges="[]"

No errors. A good sign!

Next, we need to make sure that traffic from outside can hit the EdgeRouter on UDP port 4242. This is as simple as adding a single security rule to the EdgeRouter, applied to the external interface.

In the GUI:

Or in the CLI:

set firewall name outside_in rule 12 action accept
set firewall name outside_in rule 12 description Nebula
set firewall name outside_in rule 12 destination port 4242
set firewall name outside_in rule 12 log disable
set firewall name outside_in rule 12 protocol tcp_udp

This will allow any traffic from the outside destined to port 4242 to hit the EdgeRouter. No need for a NAT rule, because the ER itself is terminating the traffic. I used TCP/UDP but only UDP is required.

You should also see a new route added for the Nebula network:

C    *> 192.168.254.0/24 is directly connected, nebula1

This is automatically created when the nebula interface is built.

In this case, no NAT’ing from the node is required. Because the EdgeRouter is already the default gateway for the devices on the “unsafe” subnet, all traffic will naturally find its way back into the Nebula traffic, because the router knows where to send it.

If the EdgeRouter is not your default gateway for the unsafe subnets, you’ll need to configure NAT/masquerade to make the traffic look like it’s coming from the ER’s IP.

Run as a Service

Finally, let’s have Nebula start as a service on the EdgeRouter. We’ll use the example config from the Nebula docs.

Make a new file at /etc/systemctl/system/nebula.service:

[Unit]
Description=nebula
Wants=basic.target
After=basic.target network.target

[Service]
SyslogIdentifier=nebula
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/home/ubnt/nebula -config /home/ubnt/router.conf
Restart=always

[Install]
WantedBy=multi-user.target

and enable it:

[email protected]:/home/ubnt# systemctl enable nebula.service
Created symlink /etc/systemd/system/multi-user.target.wants/nebula.service -> /etc/systemd/system/nebula.service.

OK, we’re done on the Lighthouse.

Host

Setting up the other node (host) is simple.

Copy the host files made in the last step (for me, mbp.key and mbp.crt) to the host, as well as the ca certificate (ca.crt, not the key file!).

Then, create a new config file, again using the example from Nebula, with the following changes:

pki:
  # The CAs that are accepted by this node. Must contain one or more certificates created by 'nebula-cert ca'
  ca: ca.crt
  cert: mbp.crt
  key: mbp.key

static_host_map:
  "192.168.254.1": ["<public IP of edgerouter>:4242"]

lighthouse:
  hosts:
    - "192.168.254.1"
tun:
  unsafe_routes:
    - route: 192.168.17.0/24
      via: 192.168.254.1
      mtu: 1300
      metric: 100
firewall:
  outbound:
    # Allow all outbound traffic from this node
    - port: any
      proto: any
      host: any

  inbound:
    # Allow icmp between any nebula hosts
    - port: any
      proto: icmp
      host: any

And that’s it. Note that the firewall configuration in this case only allows the host to be pinged from other devices on the network. If you need to expose services or ports on this host, do it under the inbound section.

Let’s give it a try!

I used Homebrew to install Nebula on my mac, so we’ll use it. On Mac, we need to run it as root, so we’ll sudo.


m1 ➜  ~ sudo /opt/homebrew/bin/nebula -config ~/nebula/mbp.conf
INFO[0000] Firewall rule added                           firewallRule="map[caName: caSha: direction:outgoing endPort:0 groups:[] host:any ip: proto:0 startPort:0]"
INFO[0000] Firewall rule added                           firewallRule="map[caName: caSha: direction:incoming endPort:0 groups:[] host:any ip: proto:1 startPort:0]"
INFO[0000] Firewall started                              firewallHash=41c01aec...f1968ffef5ec35
WARN[0000] route MTU is not supported in darwin          route="{1300 100 192.168.17.0/24 192.168.254.1}"
WARN[0000] interface name must be utun[0-9]+ on Darwin, ignoring 
INFO[0000] Main HostMap created                          network=192.168.254.2/24 preferredRanges="[]"
INFO[0000] UDP hole punching enabled                    
INFO[0000] Nebula interface is active                    build=1.5.2 interface=utun6 network=192.168.254.2/24 udpAddr="[::]:4242"
INFO[0000] Handshake message sent                        handshake="map[stage:1 style:ix_psk0]" initiatorIndex=2895366326 udpAddrs="[<edgerouter public>:4242]" vpnIp=192.168.254.1
INFO[0000] Handshake message received                    certName=lighthouse durationNs=61424667 fingerprint=ad48...f555030cf84a99 handshake="map[stage:2 style:ix_psk0]" initiatorIndex=2895366326 issuer=14604...d7e remoteIndex=2895366326 responderIndex=3772549451 sentCachedPackets=1 udpAddr="<edgerouter public>:4242" vpnIp=192.168.254.1

Looks good, but let’s check…

m1 ➜  ~ ping 192.168.254.1   
PING 192.168.254.1 (192.168.254.1): 56 data bytes
64 bytes from 192.168.254.1: icmp_seq=0 ttl=64 time=129.490 ms
64 bytes from 192.168.254.1: icmp_seq=1 ttl=64 time=95.193 ms
64 bytes from 192.168.254.1: icmp_seq=2 ttl=64 time=67.001 ms
64 bytes from 192.168.254.1: icmp_seq=3 ttl=64 time=91.063 ms
^C
--- 192.168.254.1 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 67.001/95.687/129.490/22.289 ms

Sweet, it works!

That’s it, a simple Nebula VPN setup using the EdgeRouter. The performance is really good, and the ease of configuration makes this a pretty neat setup.