Does anyone have any stats comparing throughput of Wireguard vs Edgevpn using between same 2 Nodes? #157

bmullan · 2023-12-12T23:56:51Z

bmullan
Dec 12, 2023

_Just curious about..

Does anyone have any stats comparing throughput of Wireguard vs Edgevpn using between same 2 Nodes?_

tomtom5152 · 2024-04-01T12:55:57Z

tomtom5152
Apr 1, 2024

After looking around this repo a bit I found this could be a potentially very interesteing experiment to determine if
edgevpn would be a suitable replacement for tailscale. As such I decided to use some digitalocean droplets to determine
what the difference is if any.

Test criteria

All droplets located in LON1 region
Mixed sizes (that I had available) using shared cores
Reported network speed of 1Gbps
Speed measured using iperf3 in TCP mode over 10 seconds
Droplets have public IPv4 and IPv6 addresses
Droplets are in different VPCs so no direct LAN connection is possible
Ping tests are run twice with the second result reported, this allows circuits to be fully established.

Baseline tests

iperf3 running on public IP addresses

	Avg ping (10)	Avg TX	Avg Rx
IPv4	1.075ms	2.02Gbps	2.01Gbps
IPv6	1.136ms	2.00Gbps	1.99Gbps
Avg	1.106ms	2.01Gbps	2.00Gbps

Public IPv4

[ 5] 0.00-1.00 sec 264 MBytes 2.21 Gbits/sec 36157 63.6 KBytes
[ 5] 1.00-2.00 sec 239 MBytes 2.00 Gbits/sec 37713 519 KBytes
[ 5] 2.00-3.00 sec 239 MBytes 2.00 Gbits/sec 31315 107 KBytes
[ 5] 3.00-4.00 sec 238 MBytes 1.99 Gbits/sec 37233 191 KBytes
[ 5] 4.00-5.00 sec 238 MBytes 1.99 Gbits/sec 31584 83.4 KBytes
[ 5] 5.00-6.00 sec 239 MBytes 2.00 Gbits/sec 38359 52.3 KBytes
[ 5] 6.00-7.00 sec 238 MBytes 1.99 Gbits/sec 35323 77.8 KBytes
[ 5] 7.00-8.00 sec 238 MBytes 1.99 Gbits/sec 39103 157 KBytes
[ 5] 8.00-9.00 sec 239 MBytes 2.00 Gbits/sec 35278 97.6 KBytes
[ 5] 9.00-10.00 sec 236 MBytes 1.98 Gbits/sec 33434 73.5 KBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.35 GBytes 2.02 Gbits/sec 355499 sender
[ 5] 0.00-10.00 sec 2.35 GBytes 2.01 Gbits/sec receiver

Public IPv6

[ 5] 0.00-1.00 sec 265 MBytes 2.22 Gbits/sec 32990 142 KBytes
[ 5] 1.00-2.00 sec 236 MBytes 1.98 Gbits/sec 25242 83.7 KBytes
[ 5] 2.00-3.00 sec 240 MBytes 2.01 Gbits/sec 35651 89.2 KBytes
[ 5] 3.00-4.00 sec 238 MBytes 1.99 Gbits/sec 23919 116 KBytes
[ 5] 4.00-5.00 sec 238 MBytes 1.99 Gbits/sec 23070 110 KBytes
[ 5] 5.00-6.00 sec 211 MBytes 1.77 Gbits/sec 23289 99.0 KBytes
[ 5] 6.00-7.00 sec 238 MBytes 1.99 Gbits/sec 27666 137 KBytes
[ 5] 7.00-8.00 sec 239 MBytes 2.00 Gbits/sec 27472 78.1 KBytes
[ 5] 8.00-9.00 sec 238 MBytes 1.99 Gbits/sec 22510 404 KBytes
[ 5] 9.00-10.00 sec 238 MBytes 1.99 Gbits/sec 23194 102 KBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.32 GBytes 2.00 Gbits/sec 265003 sender
[ 5] 0.00-10.00 sec 2.32 GBytes 1.99 Gbits/sec receiver

Wireguard

Connections manually created over both IPv4 and IPv6

Test Proto	Tunnel Proto	Avg ping (10)	Avg TX	Avg RX
IPv4	IPv4	1.050ms	778Mbps	776Mbps
IPv6	IPv4	0.954ms	950Mbps	947Mbps
IPv4	IPv6	1.129ms	905Mbps	903Mbps
IPv6	IPv6	0.907ms	803Mbps	801Mbps
Avg		1.010ms	859Mbps	857Mbps

IPv4 over IPv4

[ 5] 0.00-1.00 sec 98.7 MBytes 828 Mbits/sec 38 1.10 MBytes
[ 5] 1.00-2.00 sec 108 MBytes 902 Mbits/sec 0 1.20 MBytes
[ 5] 2.00-3.00 sec 95.0 MBytes 797 Mbits/sec 0 1.28 MBytes
[ 5] 3.00-4.00 sec 70.0 MBytes 587 Mbits/sec 0 1.35 MBytes
[ 5] 4.00-5.00 sec 83.8 MBytes 703 Mbits/sec 9 1.01 MBytes
[ 5] 5.00-6.00 sec 95.0 MBytes 797 Mbits/sec 0 1.06 MBytes
[ 5] 6.00-7.00 sec 92.5 MBytes 776 Mbits/sec 0 1.11 MBytes
[ 5] 7.00-8.00 sec 81.2 MBytes 681 Mbits/sec 0 1.16 MBytes
[ 5] 8.00-9.00 sec 95.0 MBytes 798 Mbits/sec 28 1.21 MBytes
[ 5] 9.00-10.00 sec 109 MBytes 912 Mbits/sec 0 1.26 MBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 927 MBytes 778 Mbits/sec 75 sender
[ 5] 0.00-10.01 sec 926 MBytes 776 Mbits/sec receiver

IPv6 over IPv6

[ 5] 0.00-1.00 sec 102 MBytes 852 Mbits/sec 55 1.10 MBytes
[ 5] 1.00-2.00 sec 116 MBytes 975 Mbits/sec 0 1.20 MBytes
[ 5] 2.00-3.00 sec 114 MBytes 955 Mbits/sec 1 1.28 MBytes
[ 5] 3.00-4.00 sec 120 MBytes 1.01 Gbits/sec 0 1.33 MBytes
[ 5] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 6 1.01 MBytes
[ 5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec 0 1.09 MBytes
[ 5] 6.00-7.00 sec 112 MBytes 944 Mbits/sec 0 1.17 MBytes
[ 5] 7.00-8.00 sec 119 MBytes 996 Mbits/sec 0 1.23 MBytes
[ 5] 8.00-9.00 sec 111 MBytes 933 Mbits/sec 0 1.27 MBytes
[ 5] 9.00-10.00 sec 115 MBytes 965 Mbits/sec 0 1.31 MBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.11 GBytes 950 Mbits/sec 62 sender
[ 5] 0.00-10.00 sec 1.10 GBytes 947 Mbits/sec receiver

IPv4 over IPv6

[ 5] 0.00-1.00 sec 106 MBytes 891 Mbits/sec 94 1.31 MBytes
[ 5] 1.00-2.00 sec 85.0 MBytes 713 Mbits/sec 441 1014 KBytes
[ 5] 2.00-3.00 sec 87.5 MBytes 734 Mbits/sec 0 1.04 MBytes
[ 5] 3.00-4.00 sec 112 MBytes 944 Mbits/sec 0 1.10 MBytes
[ 5] 4.00-5.00 sec 114 MBytes 954 Mbits/sec 0 1.16 MBytes
[ 5] 5.00-6.00 sec 114 MBytes 954 Mbits/sec 0 1.22 MBytes
[ 5] 6.00-7.00 sec 118 MBytes 986 Mbits/sec 0 1.27 MBytes
[ 5] 7.00-8.00 sec 116 MBytes 975 Mbits/sec 0 1.33 MBytes
[ 5] 8.00-9.00 sec 112 MBytes 944 Mbits/sec 0 1.38 MBytes
[ 5] 9.00-10.00 sec 114 MBytes 954 Mbits/sec 10 1.03 MBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.05 GBytes 905 Mbits/sec 545 sender
[ 5] 0.00-10.00 sec 1.05 GBytes 903 Mbits/sec receiver

IPv6 over IPv6

[ 5] 0.00-1.00 sec 100 MBytes 839 Mbits/sec 33 1.06 MBytes
[ 5] 1.00-2.00 sec 111 MBytes 933 Mbits/sec 0 1.16 MBytes
[ 5] 2.00-3.00 sec 104 MBytes 870 Mbits/sec 0 1.24 MBytes
[ 5] 3.00-4.00 sec 101 MBytes 849 Mbits/sec 0 1.31 MBytes
[ 5] 4.00-5.00 sec 86.2 MBytes 723 Mbits/sec 11 986 KBytes
[ 5] 5.00-6.00 sec 83.8 MBytes 703 Mbits/sec 0 746 KBytes
[ 5] 6.00-7.00 sec 87.5 MBytes 734 Mbits/sec 0 810 KBytes
[ 5] 7.00-8.00 sec 98.8 MBytes 828 Mbits/sec 0 870 KBytes
[ 5] 8.00-9.00 sec 92.5 MBytes 776 Mbits/sec 0 932 KBytes
[ 5] 9.00-10.00 sec 92.5 MBytes 776 Mbits/sec 0 991 KBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 958 MBytes 803 Mbits/sec 44 sender
[ 5] 0.00-10.00 sec 955 MBytes 801 Mbits/sec receiver

edgevpn

edgevpn v0.25.0
DHT discovery
Manual addressing
MTU raised to 1350 (generally accepted safe value and increase from 1200 required for IPv6)
All other settings at default

	Avg ping (10)	Avg TX	Avg RX
IPv4	2.243ms	30.0Mbps	29.7Mbps
IPv6	2.344ms	29.9Mbps	29.6Mbps
Avg	2.294ms	30.0Mbps	29.7Mbps

IPv4

[ 5] 0.00-1.00 sec 3.99 MBytes 33.4 Mbits/sec 9 26.6 KBytes
[ 5] 1.00-2.00 sec 4.07 MBytes 34.1 Mbits/sec 18 36.8 KBytes
[ 5] 2.00-3.00 sec 3.92 MBytes 32.9 Mbits/sec 21 22.8 KBytes
[ 5] 3.00-4.00 sec 3.01 MBytes 25.3 Mbits/sec 12 12.7 KBytes
[ 5] 4.00-5.00 sec 3.10 MBytes 26.0 Mbits/sec 15 40.6 KBytes
[ 5] 5.00-6.00 sec 3.51 MBytes 29.4 Mbits/sec 6 20.3 KBytes
[ 5] 6.00-7.00 sec 3.48 MBytes 29.2 Mbits/sec 22 29.2 KBytes
[ 5] 7.00-8.00 sec 3.76 MBytes 31.6 Mbits/sec 13 62.1 KBytes
[ 5] 8.00-9.00 sec 3.37 MBytes 28.3 Mbits/sec 3 90.0 KBytes
[ 5] 9.00-10.00 sec 3.53 MBytes 29.6 Mbits/sec 15 87.5 KBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 35.7 MBytes 30.0 Mbits/sec 134 sender
[ 5] 0.00-10.02 sec 35.5 MBytes 29.7 Mbits/sec receiver

IPv6

[ 5] 0.00-1.00 sec 3.93 MBytes 33.0 Mbits/sec 18 44.9 KBytes
[ 5] 1.00-2.00 sec 4.23 MBytes 35.4 Mbits/sec 15 48.7 KBytes
[ 5] 2.00-3.00 sec 3.09 MBytes 25.9 Mbits/sec 15 9.98 KBytes
[ 5] 3.00-4.00 sec 3.50 MBytes 29.4 Mbits/sec 12 43.7 KBytes
[ 5] 4.00-5.00 sec 3.31 MBytes 27.8 Mbits/sec 5 54.9 KBytes
[ 5] 5.00-6.00 sec 2.62 MBytes 22.0 Mbits/sec 9 71.1 KBytes
[ 5] 6.00-7.00 sec 3.42 MBytes 28.7 Mbits/sec 5 91.1 KBytes
[ 5] 7.00-8.00 sec 3.50 MBytes 29.4 Mbits/sec 3 78.6 KBytes
[ 5] 8.00-9.00 sec 4.16 MBytes 34.9 Mbits/sec 12 74.9 KBytes
[ 5] 9.00-10.00 sec 3.92 MBytes 32.8 Mbits/sec 19 16.2 KBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 35.7 MBytes 29.9 Mbits/sec 113 sender
[ 5] 0.00-10.00 sec 35.3 MBytes 29.6 Mbits/sec receiver

Conclusions

Unladen latency appears to be double over edgevpn vs native and wireguard connections
Wireguard essentially halves native throughput
edgevpn performance is considerably worse in all test cases even with the increased MTU

These results are extremely surprising from my understanding of how edgevpn and libp2p work, so I would be very interested to see if anybody can replicate the tests in a different environment and confirm they get the same approximate results.

0 replies

tomtom5152 · 2024-04-01T17:30:48Z

tomtom5152
Apr 1, 2024

At this suggestion of @bmullan via email, I've also repeated this test using tailscale (via headscale) as that runs using user-space Wireguard rather than kernel space. I had already spun the instances down so these are two new droplets with the same split VPC config. The baseline tests were repeated to ensure the environment was the same and they returned near identical results. Ping results

Tailscale (user-space Wireguard)

Direct connection over IPv6

	Avg ping (10)	Avg TX	Avg RX
IPv4	1.353ms	996Mbps	989Mbps
IPv6	1.368ms	992Mbps	988Mbps
Avg	1.361ms	994Mbps	989Mbps

IPv4

[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 113 MBytes 948 Mbits/sec 0 3.14 MBytes
[ 5] 1.00-2.00 sec 116 MBytes 975 Mbits/sec 199 2.33 MBytes
[ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 0 2.52 MBytes
[ 5] 3.00-4.00 sec 118 MBytes 986 Mbits/sec 0 2.67 MBytes
[ 5] 4.00-5.00 sec 118 MBytes 986 Mbits/sec 0 2.80 MBytes
[ 5] 5.00-6.00 sec 125 MBytes 1.05 Gbits/sec 0 2.90 MBytes
[ 5] 6.00-7.00 sec 110 MBytes 923 Mbits/sec 0 2.98 MBytes
[ 5] 7.00-8.00 sec 122 MBytes 1.03 Gbits/sec 186 2.15 MBytes
[ 5] 8.00-9.00 sec 126 MBytes 1.06 Gbits/sec 0 2.27 MBytes
[ 5] 9.00-10.00 sec 118 MBytes 986 Mbits/sec 0 2.36 MBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.16 GBytes 996 Mbits/sec 385 sender
[ 5] 0.00-10.05 sec 1.16 GBytes 989 Mbits/sec receiver

IPv6

[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 113 MBytes 945 Mbits/sec 0 3.15 MBytes
[ 5] 1.00-2.00 sec 116 MBytes 975 Mbits/sec 0 3.15 MBytes
[ 5] 2.00-3.00 sec 124 MBytes 1.04 Gbits/sec 0 3.15 MBytes
[ 5] 3.00-4.00 sec 115 MBytes 965 Mbits/sec 0 3.15 MBytes
[ 5] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 3.15 MBytes
[ 5] 5.00-6.00 sec 119 MBytes 996 Mbits/sec 0 3.15 MBytes
[ 5] 6.00-7.00 sec 121 MBytes 1.02 Gbits/sec 38 3.15 MBytes
[ 5] 7.00-8.00 sec 122 MBytes 1.03 Gbits/sec 0 3.15 MBytes
[ 5] 8.00-9.00 sec 120 MBytes 1.01 Gbits/sec 0 3.15 MBytes
[ 5] 9.00-10.00 sec 120 MBytes 1.01 Gbits/sec 679 2.37 MBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.15 GBytes 992 Mbits/sec 717 sender
[ 5] 0.00-10.03 sec 1.15 GBytes 988 Mbits/sec receiver

edgevpn take 2

Given the result above is highly unexpected I decided to run iperf3 again over edgevpn using the same settings

Static address
MTU 1350
All other settings default

	Avg ping (10)	Avg TX	Avg RX
IPv4	2.346ms	34.3Mbps	33.7Mbps
IPv6	2.290ms	29.5Mbps	28.9Mbps
Avg	2.318ms	31.9Mbps	31.3Mbps

IPv4

[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 4.25 MBytes 35.7 Mbits/sec 98 231 KBytes
[ 5] 1.00-2.00 sec 3.47 MBytes 29.1 Mbits/sec 18 236 KBytes
[ 5] 2.00-3.00 sec 4.21 MBytes 35.3 Mbits/sec 7 238 KBytes
[ 5] 3.00-4.00 sec 3.96 MBytes 33.2 Mbits/sec 10 246 KBytes
[ 5] 4.00-5.00 sec 4.08 MBytes 34.3 Mbits/sec 20 248 KBytes
[ 5] 5.00-6.00 sec 4.46 MBytes 37.4 Mbits/sec 11 256 KBytes
[ 5] 6.00-7.00 sec 4.58 MBytes 38.4 Mbits/sec 7 261 KBytes
[ 5] 7.00-8.00 sec 4.08 MBytes 34.3 Mbits/sec 6 270 KBytes
[ 5] 8.00-9.00 sec 4.27 MBytes 35.8 Mbits/sec 5 275 KBytes
[ 5] 9.00-10.00 sec 3.53 MBytes 29.6 Mbits/sec 7 280 KBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 40.9 MBytes 34.3 Mbits/sec 189 sender
[ 5] 0.00-10.06 sec 40.5 MBytes 33.7 Mbits/sec receiver

IPv6

[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 3.29 MBytes 27.6 Mbits/sec 3 98.6 KBytes
[ 5] 1.00-2.00 sec 3.60 MBytes 30.2 Mbits/sec 10 115 KBytes
[ 5] 2.00-3.00 sec 3.36 MBytes 28.2 Mbits/sec 8 127 KBytes
[ 5] 3.00-4.00 sec 3.73 MBytes 31.3 Mbits/sec 7 142 KBytes
[ 5] 4.00-5.00 sec 4.10 MBytes 34.4 Mbits/sec 6 157 KBytes
[ 5] 5.00-6.00 sec 3.54 MBytes 29.7 Mbits/sec 10 166 KBytes
[ 5] 6.00-7.00 sec 3.61 MBytes 30.3 Mbits/sec 8 93.6 KBytes
[ 5] 7.00-8.00 sec 3.23 MBytes 27.1 Mbits/sec 6 111 KBytes
[ 5] 8.00-9.00 sec 2.80 MBytes 23.5 Mbits/sec 5 124 KBytes
[ 5] 9.00-10.00 sec 3.85 MBytes 32.3 Mbits/sec 4 140 KBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 35.1 MBytes 29.5 Mbits/sec 67 sender
[ 5] 0.00-10.07 sec 34.8 MBytes 28.9 Mbits/sec receiver

These results very closely mirror the original results only with a greater discrepancy between IPv4 and IPv6 this time for reasons unknown.

An additional observation not commented on in the original test is the startup time on IPv6. I assumed initially it was some quirk of DHT that it took so long to establish the peering with the other node, however again it took a lot longer to establish on IPv6 addresses than IPv4. I don't know what this means, but it could considered interesting.

Conclusions

It would appear either the previous test was running in user-space despite me believing it was kernel-space, tailscale magic is somehow making it quicker with some real voodoo, or the droplets are on less loaded hosts this time which is something Wireguard can take advantage of whilst edgevpn/libp2p still struggles.

I think my ultimate takeaway from this is that I want a hybrid network, edgevpn to establish connectivity for control plane to control plane probably using static IPv6 based on machine-id or something, and then either to run HA headscale and then use tailscale between nodes, or to use some other Wireguard VPN solution (e.g. calico) for service to service communication.

All in all, this has been a very interesting experiment that I'm glad I took the time to run it and thanks again to @bmullan for pointing out that use-space Wireguard may have different results (it did, just not what we expected).

0 replies

tomtom5152 · 2024-04-05T17:35:43Z

tomtom5152
Apr 5, 2024

I really wasn't happy with the previous two results, they left too much unanswered as to why throughput was so poor, so
I've decided to replicate it using local hardware where I know the full stack and everything running on the network. Hopefully now my mind can settle this.

Unless stated otherwise the critera for these tests was the same as in previous tests on digitalocean droplets.

Again these tests were run in unidirectional mode due to being a symmetric connection.

Test overview

Running on repurposed mac minis with the following specs:
- 1.4GHz Dual-Core Intel Core i5 (Turbo Boost up to 2.7GHz)
- 4GB 1600MHz LPDDR3 SDRAM
- 500GB Serial ATA Drive @ 5400 rpm
Running a kairos/opensuse:leap-15.5-standard-amd64-generic-v3.0.4-k3sv1.29.3-k3s1 modified to add wireguard and iperf3
- Default edgevpn and k3s disabled, essentially minimal linux but with an easy install mechanism
Connected with CAT6 to a UniFi 8 port switch with confirmed 1000Mbps links
Both located on the same semi-restricted VLAN, only other traffic on that VLAN is for the control machine.
Overall load on the switch is minimal, well within the backplane bandwidth
Switch configured with jumbo frames enabled (MTU 9000)

Standard 1500MTU

Baseline

	Avg ping (10)	Avg TX	Avg Rx
IPv4	0.277ms	943Mbps	941Mbps
IPv6	0.276ms	930Mbps	928Mbps
Avg	0.277ms	937Mbps	935Mbps

IPv4

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   954 Mbits/sec    0    351 KBytes       
[  5]   1.00-2.00   sec   113 MBytes   945 Mbits/sec    0    351 KBytes       
[  5]   2.00-3.00   sec   112 MBytes   938 Mbits/sec    0    351 KBytes       
[  5]   3.00-4.00   sec   113 MBytes   945 Mbits/sec    0    351 KBytes       
[  5]   4.00-5.00   sec   112 MBytes   938 Mbits/sec    0    351 KBytes       
[  5]   5.00-6.00   sec   113 MBytes   945 Mbits/sec    0    351 KBytes       
[  5]   6.00-7.00   sec   112 MBytes   938 Mbits/sec    0    351 KBytes       
[  5]   7.00-8.00   sec   113 MBytes   945 Mbits/sec    0    351 KBytes       
[  5]   8.00-9.00   sec   112 MBytes   938 Mbits/sec    0    351 KBytes       
[  5]   9.00-10.00  sec   113 MBytes   945 Mbits/sec    0    351 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  receiver

IPv6

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   113 MBytes   945 Mbits/sec    0    383 KBytes       
[  5]   1.00-2.00   sec   111 MBytes   928 Mbits/sec    0    404 KBytes       
[  5]   2.00-3.00   sec   111 MBytes   928 Mbits/sec    0    404 KBytes       
[  5]   3.00-4.00   sec   111 MBytes   928 Mbits/sec    0    404 KBytes       
[  5]   4.00-5.00   sec   111 MBytes   928 Mbits/sec    0    404 KBytes       
[  5]   5.00-6.00   sec   111 MBytes   928 Mbits/sec    0    404 KBytes       
[  5]   6.00-7.00   sec   111 MBytes   928 Mbits/sec    0    404 KBytes       
[  5]   7.00-8.00   sec   111 MBytes   928 Mbits/sec    0    404 KBytes       
[  5]   8.00-9.00   sec   111 MBytes   928 Mbits/sec    0    404 KBytes       
[  5]   9.00-10.00  sec   111 MBytes   928 Mbits/sec    0    404 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.08 GBytes   930 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.08 GBytes   928 Mbits/sec                  receiver

Wireguard

Kernel mode

Test Proto	Tunnel Proto	Avg ping (10)	Avg TX	Avg RX
IPv4	IPv4	0.652ms	902Mbps	901Mbps
IPv6	IPv4	0.709ms	889Mbps	888Mbps
IPv4	IPv6	0.674ms	890Mbps	889Mbps
IPv6	IPv6	0.675ms	890Mbps	889Mbps
Avg		0.678ms	893Mbps	892Mbps

IPv4 over IPv4

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   108 MBytes   909 Mbits/sec    0    310 KBytes       
[  5]   1.00-2.00   sec   107 MBytes   901 Mbits/sec    0    310 KBytes       
[  5]   2.00-3.00   sec   107 MBytes   899 Mbits/sec    0    310 KBytes       
[  5]   3.00-4.00   sec   107 MBytes   902 Mbits/sec    0    310 KBytes       
[  5]   4.00-5.00   sec   107 MBytes   901 Mbits/sec    0    310 KBytes       
[  5]   5.00-6.00   sec   107 MBytes   902 Mbits/sec    0    310 KBytes       
[  5]   6.00-7.00   sec   108 MBytes   903 Mbits/sec    0    345 KBytes       
[  5]   7.00-8.00   sec   107 MBytes   900 Mbits/sec    0    345 KBytes       
[  5]   8.00-9.00   sec   107 MBytes   900 Mbits/sec    0    345 KBytes       
[  5]   9.00-10.00  sec   108 MBytes   904 Mbits/sec    0    345 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.05 GBytes   902 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.05 GBytes   901 Mbits/sec                  receiver

IPv6 over IPv6

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   107 MBytes   895 Mbits/sec    0    337 KBytes       
[  5]   1.00-2.00   sec   106 MBytes   887 Mbits/sec    0    351 KBytes       
[  5]   2.00-3.00   sec   106 MBytes   891 Mbits/sec    0    351 KBytes       
[  5]   3.00-4.00   sec   106 MBytes   888 Mbits/sec    0    351 KBytes       
[  5]   4.00-5.00   sec   106 MBytes   888 Mbits/sec    0    351 KBytes       
[  5]   5.00-6.00   sec   106 MBytes   888 Mbits/sec    0    351 KBytes       
[  5]   6.00-7.00   sec   106 MBytes   888 Mbits/sec    0    351 KBytes       
[  5]   7.00-8.00   sec   106 MBytes   888 Mbits/sec    0    351 KBytes       
[  5]   8.00-9.00   sec   106 MBytes   888 Mbits/sec    0    351 KBytes       
[  5]   9.00-10.00  sec   106 MBytes   888 Mbits/sec    0    351 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.03 GBytes   889 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.03 GBytes   888 Mbits/sec                  receiver

IPv4 over IPv6

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   107 MBytes   896 Mbits/sec    0    314 KBytes       
[  5]   1.00-2.00   sec   106 MBytes   890 Mbits/sec    0    327 KBytes       
[  5]   2.00-3.00   sec   106 MBytes   889 Mbits/sec    0    327 KBytes       
[  5]   3.00-4.00   sec   106 MBytes   888 Mbits/sec    0    327 KBytes       
[  5]   4.00-5.00   sec   106 MBytes   892 Mbits/sec    0    341 KBytes       
[  5]   5.00-6.00   sec   106 MBytes   888 Mbits/sec    0    370 KBytes       
[  5]   6.00-7.00   sec   106 MBytes   890 Mbits/sec    0    370 KBytes       
[  5]   7.00-8.00   sec   106 MBytes   890 Mbits/sec    0    370 KBytes       
[  5]   8.00-9.00   sec   106 MBytes   887 Mbits/sec    0    370 KBytes       
[  5]   9.00-10.00  sec   106 MBytes   889 Mbits/sec    0    370 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.04 GBytes   890 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.04 GBytes   889 Mbits/sec                  receiver

IPv6 over IPv6

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   105 MBytes   880 Mbits/sec    0    337 KBytes       
[  5]   1.00-2.00   sec   104 MBytes   876 Mbits/sec    0    337 KBytes       
[  5]   2.00-3.00   sec   105 MBytes   878 Mbits/sec    0    337 KBytes       
[  5]   3.00-4.00   sec   104 MBytes   876 Mbits/sec    0    351 KBytes       
[  5]   4.00-5.00   sec   105 MBytes   877 Mbits/sec    0    351 KBytes       
[  5]   5.00-6.00   sec   105 MBytes   877 Mbits/sec    0    351 KBytes       
[  5]   6.00-7.00   sec   105 MBytes   877 Mbits/sec    0    351 KBytes       
[  5]   7.00-8.00   sec   104 MBytes   874 Mbits/sec    0    384 KBytes       
[  5]   8.00-9.00   sec   105 MBytes   877 Mbits/sec    0    384 KBytes       
[  5]   9.00-10.00  sec   105 MBytes   877 Mbits/sec    0    384 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.02 GBytes   877 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.02 GBytes   876 Mbits/sec                  receiver

User mode (tailscale)

tailscale status shows direct connection over IPv4

Test Proto	Avg ping (10)	Avg TX	Avg RX
IPv4	1.118ms	893Mbps	891Mbps
IPv6	1.153ms	879Mbps	876Mbps
Avg	1.136ms	886Mbps	884Mbps

IPv4

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   109 MBytes   912 Mbits/sec    0   3.02 MBytes       
[  5]   1.00-2.00   sec   106 MBytes   891 Mbits/sec    0   3.02 MBytes       
[  5]   2.00-3.00   sec   106 MBytes   891 Mbits/sec    0   3.02 MBytes       
[  5]   3.00-4.00   sec   106 MBytes   891 Mbits/sec    0   3.02 MBytes       
[  5]   4.00-5.00   sec   106 MBytes   891 Mbits/sec    0   3.02 MBytes       
[  5]   5.00-6.00   sec   106 MBytes   891 Mbits/sec    0   3.02 MBytes       
[  5]   6.00-7.00   sec   106 MBytes   891 Mbits/sec    0   3.02 MBytes       
[  5]   7.00-8.00   sec   106 MBytes   891 Mbits/sec    0   3.02 MBytes       
[  5]   8.00-9.00   sec   106 MBytes   891 Mbits/sec    0   3.02 MBytes       
[  5]   9.00-10.00  sec   106 MBytes   891 Mbits/sec    0   3.02 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.04 GBytes   893 Mbits/sec    0             sender
[  5]   0.00-10.03  sec  1.04 GBytes   891 Mbits/sec                  receiver

IPv6

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   107 MBytes   898 Mbits/sec    0   3.15 MBytes       
[  5]   1.00-2.00   sec   105 MBytes   881 Mbits/sec    0   3.15 MBytes       
[  5]   2.00-3.00   sec   104 MBytes   870 Mbits/sec    0   3.15 MBytes       
[  5]   3.00-4.00   sec   105 MBytes   881 Mbits/sec    0   3.15 MBytes       
[  5]   4.00-5.00   sec   105 MBytes   881 Mbits/sec    0   3.15 MBytes       
[  5]   5.00-6.00   sec   104 MBytes   870 Mbits/sec    0   3.15 MBytes       
[  5]   6.00-7.00   sec   105 MBytes   881 Mbits/sec    0   3.15 MBytes       
[  5]   7.00-8.00   sec   104 MBytes   870 Mbits/sec    0   3.15 MBytes       
[  5]   8.00-9.00   sec   105 MBytes   881 Mbits/sec    0   3.15 MBytes       
[  5]   9.00-10.00  sec   105 MBytes   881 Mbits/sec    0   3.15 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.02 GBytes   879 Mbits/sec    0             sender
[  5]   0.00-10.03  sec  1.02 GBytes   876 Mbits/sec                  receiver

edgevpn

DHT discovery disabled, mDNS only
Manual addressing
MTU raised to 1350 for a baseline due to
All other settings at default

For these tests the switch was reconfigured to mirror traffic to a second NIC on my desktop, this was done to confirm
how the traffic was travelling across the network. Packet captures are not included due to their size, however they
showed QUIC over IPv6 for the transport.

MTU	Interface MTU	Test Proto	Avg ping (10)	Avg TX	Avg RX
1200	1420	IPv4	1.555ms	102Mbps	102Mbps
1200	1420	IPv6	-	-	-
1280¹	1420	IPv4	1.613ms	122Mbps	122Mbps
1280¹	1420	IPv6	1.835ms	97.3Mbps	97.0Mbps
1350	1420	IPv4	1.862ms	103Mbps	103Mbps
1350	1420	IPv6	1.815ms	96.7Mbps	96.5Mbps
1500²	1500	IPv4	1.826ms	96.9Mbps	96.7Mbps
1500²	1500	IPv6	1.644ms	123Mbps	123Mbps
Avg			1.736ms	106Mbps	106Mbps

IPv4 1200/1420

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  12.2 MBytes   103 Mbits/sec  132   30.3 KBytes       
[  5]   1.00-2.00   sec  12.9 MBytes   108 Mbits/sec   59   23.5 KBytes       
[  5]   2.00-3.00   sec  13.2 MBytes   111 Mbits/sec   76   32.5 KBytes       
[  5]   3.00-4.00   sec  12.0 MBytes   100 Mbits/sec  132   13.5 KBytes       
[  5]   4.00-5.00   sec  10.3 MBytes  86.4 Mbits/sec   48   16.8 KBytes       
[  5]   5.00-6.00   sec  12.7 MBytes   106 Mbits/sec   58   43.7 KBytes       
[  5]   6.00-7.00   sec  11.7 MBytes  98.2 Mbits/sec  111   17.9 KBytes       
[  5]   7.00-8.00   sec  12.4 MBytes   104 Mbits/sec   61   42.6 KBytes       
[  5]   8.00-9.00   sec  12.4 MBytes   104 Mbits/sec   64   38.1 KBytes       
[  5]   9.00-10.00  sec  12.4 MBytes   104 Mbits/sec   53   30.3 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   122 MBytes   102 Mbits/sec  794             sender
[  5]   0.00-10.00  sec   122 MBytes   102 Mbits/sec                  receiver

IPv4 1280/1420

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  14.3 MBytes   120 Mbits/sec   80   42.0 KBytes       
[  5]   1.00-2.00   sec  14.7 MBytes   123 Mbits/sec   54   21.6 KBytes       
[  5]   2.00-3.00   sec  14.7 MBytes   123 Mbits/sec   64   49.2 KBytes       
[  5]   3.00-4.00   sec  14.7 MBytes   123 Mbits/sec   52   25.2 KBytes       
[  5]   4.00-5.00   sec  14.8 MBytes   124 Mbits/sec   57   50.4 KBytes       
[  5]   5.00-6.00   sec  14.2 MBytes   119 Mbits/sec   95   34.8 KBytes       
[  5]   6.00-7.00   sec  14.4 MBytes   121 Mbits/sec   77   33.6 KBytes       
[  5]   7.00-8.00   sec  14.4 MBytes   121 Mbits/sec   61   45.6 KBytes       
[  5]   8.00-9.00   sec  14.3 MBytes   120 Mbits/sec   72   31.2 KBytes       
[  5]   9.00-10.00  sec  14.8 MBytes   124 Mbits/sec   52   34.8 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   145 MBytes   122 Mbits/sec  664             sender
[  5]   0.00-10.00  sec   145 MBytes   122 Mbits/sec                  receiver

IPv6 1280/1420

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  11.3 MBytes  95.0 Mbits/sec  131   60.2 KBytes       
[  5]   1.00-2.00   sec  11.4 MBytes  95.6 Mbits/sec   70   41.3 KBytes       
[  5]   2.00-3.00   sec  11.6 MBytes  97.0 Mbits/sec   45   72.0 KBytes       
[  5]   3.00-4.00   sec  12.0 MBytes   100 Mbits/sec   17   88.5 KBytes       
[  5]   4.00-5.00   sec  11.9 MBytes  99.9 Mbits/sec   11   74.3 KBytes       
[  5]   5.00-6.00   sec  11.5 MBytes  96.8 Mbits/sec   29   57.8 KBytes       
[  5]   6.00-7.00   sec  11.7 MBytes  97.8 Mbits/sec   53   69.6 KBytes       
[  5]   7.00-8.00   sec  11.7 MBytes  98.3 Mbits/sec   19   90.8 KBytes       
[  5]   8.00-9.00   sec  11.5 MBytes  96.7 Mbits/sec   82   37.8 KBytes       
[  5]   9.00-10.00  sec  11.4 MBytes  95.3 Mbits/sec   43   49.5 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   116 MBytes  97.3 Mbits/sec  500             sender
[  5]   0.00-10.00  sec   116 MBytes  97.0 Mbits/sec                  receiver

IPv4 1350/1420

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  12.6 MBytes   106 Mbits/sec  107   15.2 KBytes       
[  5]   1.00-2.00   sec  12.2 MBytes   102 Mbits/sec   72   12.7 KBytes       
[  5]   2.00-3.00   sec  12.4 MBytes   104 Mbits/sec   89   19.0 KBytes       
[  5]   3.00-4.00   sec  12.0 MBytes   101 Mbits/sec   84   15.2 KBytes       
[  5]   4.00-5.00   sec  12.2 MBytes   102 Mbits/sec   84   34.2 KBytes       
[  5]   5.00-6.00   sec  12.2 MBytes   102 Mbits/sec   77   31.7 KBytes       
[  5]   6.00-7.00   sec  12.6 MBytes   106 Mbits/sec  104   25.4 KBytes       
[  5]   7.00-8.00   sec  12.6 MBytes   105 Mbits/sec  101   20.3 KBytes       
[  5]   8.00-9.00   sec  12.4 MBytes   104 Mbits/sec  124   22.8 KBytes       
[  5]   9.00-10.00  sec  11.6 MBytes  97.7 Mbits/sec  100   27.9 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   123 MBytes   103 Mbits/sec  942             sender
[  5]   0.00-10.01  sec   123 MBytes   103 Mbits/sec                  receiver

IPv6 1350/1420

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  11.7 MBytes  97.8 Mbits/sec  215   37.4 KBytes       
[  5]   1.00-2.00   sec  11.6 MBytes  97.5 Mbits/sec   55   44.9 KBytes       
[  5]   2.00-3.00   sec  11.4 MBytes  95.9 Mbits/sec   68   21.2 KBytes       
[  5]   3.00-4.00   sec  11.4 MBytes  95.4 Mbits/sec   77   31.2 KBytes       
[  5]   4.00-5.00   sec  11.4 MBytes  95.4 Mbits/sec   48   22.5 KBytes       
[  5]   5.00-6.00   sec  11.4 MBytes  95.4 Mbits/sec   67   23.7 KBytes       
[  5]   6.00-7.00   sec  11.6 MBytes  97.5 Mbits/sec   56   52.4 KBytes       
[  5]   7.00-8.00   sec  11.4 MBytes  95.4 Mbits/sec   60   43.7 KBytes       
[  5]   8.00-9.00   sec  11.6 MBytes  97.5 Mbits/sec  103   33.7 KBytes       
[  5]   9.00-10.00  sec  11.8 MBytes  99.1 Mbits/sec   48   32.4 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   115 MBytes  96.7 Mbits/sec  797             sender
[  5]   0.00-10.01  sec   115 MBytes  96.5 Mbits/sec                  receiver

IPv4 1500/1500

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  11.8 MBytes  98.6 Mbits/sec   79   21.2 KBytes       
[  5]   1.00-2.00   sec  11.2 MBytes  94.4 Mbits/sec   36   17.0 KBytes       
[  5]   2.00-3.00   sec  11.7 MBytes  98.5 Mbits/sec   45   22.6 KBytes       
[  5]   3.00-4.00   sec  11.9 MBytes   100 Mbits/sec   38   15.6 KBytes       
[  5]   4.00-5.00   sec  11.4 MBytes  95.9 Mbits/sec   51   17.0 KBytes       
[  5]   5.00-6.00   sec  11.0 MBytes  92.3 Mbits/sec   61   15.6 KBytes       
[  5]   6.00-7.00   sec  12.1 MBytes   101 Mbits/sec   44   35.4 KBytes       
[  5]   7.00-8.00   sec  11.3 MBytes  94.9 Mbits/sec   46   28.3 KBytes       
[  5]   8.00-9.00   sec  11.7 MBytes  98.5 Mbits/sec   57   21.2 KBytes       
[  5]   9.00-10.00  sec  11.3 MBytes  94.9 Mbits/sec   45   32.5 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   116 MBytes  96.9 Mbits/sec  502             sender
[  5]   0.00-10.01  sec   115 MBytes  96.7 Mbits/sec                  receiver

IPv6 1500/1500

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  15.8 MBytes   133 Mbits/sec  111   65.5 KBytes       
[  5]   1.00-2.00   sec  15.3 MBytes   128 Mbits/sec   68   29.3 KBytes       
[  5]   2.00-3.00   sec  14.8 MBytes   124 Mbits/sec   49   57.2 KBytes       
[  5]   3.00-4.00   sec  14.9 MBytes   125 Mbits/sec  113   16.7 KBytes       
[  5]   4.00-5.00   sec  14.2 MBytes   119 Mbits/sec   70   32.1 KBytes       
[  5]   5.00-6.00   sec  13.5 MBytes   114 Mbits/sec   89   8.37 KBytes       
[  5]   6.00-7.00   sec  13.1 MBytes   110 Mbits/sec   70   34.9 KBytes       
[  5]   7.00-8.00   sec  15.3 MBytes   129 Mbits/sec   63   20.9 KBytes       
[  5]   8.00-9.00   sec  14.3 MBytes   120 Mbits/sec  102   30.7 KBytes       
[  5]   9.00-10.00  sec  15.5 MBytes   130 Mbits/sec   49   64.1 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   147 MBytes   123 Mbits/sec  784             sender
[  5]   0.00-10.01  sec   146 MBytes   123 Mbits/sec                  receiver

Jumbo frames

Utilising the maximum MTU on my network of 9000 produced the following results.

Ping tests were run with -s 9000 to produce a 9000 byte payload

Baseline

	Avg ping (10)	Avg TX	Avg RX
IPv4	0.797ms	991Mbps	990Mbps
IPv6	0.810ms	989Mbps	988Mbps
Avg	0.804ms	990Mbps	989Mbps

IPv4

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   119 MBytes   998 Mbits/sec    0    367 KBytes       
[  5]   1.00-2.00   sec   118 MBytes   991 Mbits/sec    0    367 KBytes       
[  5]   2.00-3.00   sec   118 MBytes   990 Mbits/sec    0    367 KBytes       
[  5]   3.00-4.00   sec   118 MBytes   990 Mbits/sec    0    367 KBytes       
[  5]   4.00-5.00   sec   118 MBytes   990 Mbits/sec    0    367 KBytes       
[  5]   5.00-6.00   sec   118 MBytes   987 Mbits/sec    0    367 KBytes       
[  5]   6.00-7.00   sec   118 MBytes   991 Mbits/sec    0    367 KBytes       
[  5]   7.00-8.00   sec   118 MBytes   991 Mbits/sec    0    367 KBytes       
[  5]   8.00-9.00   sec   118 MBytes   990 Mbits/sec    0    367 KBytes       
[  5]   9.00-10.00  sec   118 MBytes   987 Mbits/sec    0    367 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.15 GBytes   991 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.15 GBytes   990 Mbits/sec                  receiver

IPv6

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   119 MBytes   995 Mbits/sec    0    349 KBytes       
[  5]   1.00-2.00   sec   118 MBytes   988 Mbits/sec    0    349 KBytes       
[  5]   2.00-3.00   sec   118 MBytes   988 Mbits/sec    0    349 KBytes       
[  5]   3.00-4.00   sec   118 MBytes   988 Mbits/sec    0    349 KBytes       
[  5]   4.00-5.00   sec   118 MBytes   988 Mbits/sec    0    349 KBytes       
[  5]   5.00-6.00   sec   118 MBytes   988 Mbits/sec    0    349 KBytes       
[  5]   6.00-7.00   sec   118 MBytes   988 Mbits/sec    0    349 KBytes       
[  5]   7.00-8.00   sec   118 MBytes   988 Mbits/sec    0    349 KBytes       
[  5]   8.00-9.00   sec   118 MBytes   988 Mbits/sec    0    349 KBytes       
[  5]   9.00-10.00  sec   118 MBytes   988 Mbits/sec    0    349 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.15 GBytes   989 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.15 GBytes   988 Mbits/sec                  receiver

Wireguard

Kernel mode

Test proto	Transport proto	Avg ping (10)	Avg TX	Avg RX
IPv4	IPV4	1.171ms	984Mbps	983Mbps
IPv6	IPv4	1.165ms	982Mbps	981Mbps
IPv4	IPv6	1.141ms	982Mbps	981Mbps
IPv6	IPv6	1.152ms	980Mbps	979Mbps
Avg		1.157ms	982Mbps	981Mbps

IPv4 over IPv4

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   118 MBytes   990 Mbits/sec    0    381 KBytes       
[  5]   1.00-2.00   sec   117 MBytes   984 Mbits/sec    0    398 KBytes       
[  5]   2.00-3.00   sec   117 MBytes   983 Mbits/sec    0    398 KBytes       
[  5]   3.00-4.00   sec   117 MBytes   983 Mbits/sec    0    398 KBytes       
[  5]   4.00-5.00   sec   117 MBytes   983 Mbits/sec    0    398 KBytes       
[  5]   5.00-6.00   sec   117 MBytes   983 Mbits/sec    0    398 KBytes       
[  5]   6.00-7.00   sec   117 MBytes   983 Mbits/sec    0    398 KBytes       
[  5]   7.00-8.00   sec   117 MBytes   983 Mbits/sec    0    398 KBytes       
[  5]   8.00-9.00   sec   117 MBytes   983 Mbits/sec    0    398 KBytes       
[  5]   9.00-10.00  sec   117 MBytes   983 Mbits/sec    0    398 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.15 GBytes   984 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.14 GBytes   983 Mbits/sec                  receiver

IPv6 over IPv4

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   118 MBytes   988 Mbits/sec    0    302 KBytes       
[  5]   1.00-2.00   sec   117 MBytes   982 Mbits/sec    0    302 KBytes       
[  5]   2.00-3.00   sec   117 MBytes   979 Mbits/sec    0    320 KBytes       
[  5]   3.00-4.00   sec   117 MBytes   982 Mbits/sec    0    320 KBytes       
[  5]   4.00-5.00   sec   117 MBytes   982 Mbits/sec    0    320 KBytes       
[  5]   5.00-6.00   sec   117 MBytes   982 Mbits/sec    0    320 KBytes       
[  5]   6.00-7.00   sec   117 MBytes   980 Mbits/sec    0    320 KBytes       
[  5]   7.00-8.00   sec   117 MBytes   980 Mbits/sec    0    320 KBytes       
[  5]   8.00-9.00   sec   117 MBytes   981 Mbits/sec    0    320 KBytes       
[  5]   9.00-10.00  sec   117 MBytes   986 Mbits/sec    0    423 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.14 GBytes   982 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.14 GBytes   981 Mbits/sec                  receiver

IPv4 over IPv6

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   118 MBytes   989 Mbits/sec    0    364 KBytes       
[  5]   1.00-2.00   sec   117 MBytes   981 Mbits/sec    0    364 KBytes       
[  5]   2.00-3.00   sec   117 MBytes   981 Mbits/sec    0    364 KBytes       
[  5]   3.00-4.00   sec   117 MBytes   982 Mbits/sec    0    364 KBytes       
[  5]   4.00-5.00   sec   117 MBytes   981 Mbits/sec    0    364 KBytes       
[  5]   5.00-6.00   sec   117 MBytes   981 Mbits/sec    0    364 KBytes       
[  5]   6.00-7.00   sec   117 MBytes   981 Mbits/sec    0    364 KBytes       
[  5]   7.00-8.00   sec   117 MBytes   981 Mbits/sec    0    364 KBytes       
[  5]   8.00-9.00   sec   117 MBytes   981 Mbits/sec    0    364 KBytes       
[  5]   9.00-10.00  sec   117 MBytes   981 Mbits/sec    0    364 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.14 GBytes   982 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.14 GBytes   981 Mbits/sec                  receiver

IPv6 over IPv6

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   117 MBytes   984 Mbits/sec    1    302 KBytes       
[  5]   1.00-2.00   sec   117 MBytes   982 Mbits/sec    0    302 KBytes       
[  5]   2.00-3.00   sec   117 MBytes   978 Mbits/sec    0    302 KBytes       
[  5]   3.00-4.00   sec   117 MBytes   978 Mbits/sec    0    302 KBytes       
[  5]   4.00-5.00   sec   117 MBytes   978 Mbits/sec    0    302 KBytes       
[  5]   5.00-6.00   sec   117 MBytes   982 Mbits/sec    0    302 KBytes       
[  5]   6.00-7.00   sec   117 MBytes   980 Mbits/sec    0    372 KBytes       
[  5]   7.00-8.00   sec   116 MBytes   977 Mbits/sec    1    372 KBytes       
[  5]   8.00-9.00   sec   117 MBytes   979 Mbits/sec    0    372 KBytes       
[  5]   9.00-10.00  sec   117 MBytes   980 Mbits/sec    1    363 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.14 GBytes   980 Mbits/sec    3             sender
[  5]   0.00-10.00  sec  1.14 GBytes   979 Mbits/sec                  receiver

Usermode (tailscale)

This was specifically not tested, tailscale uses a fixed MTU of 1280 which cannot be overwritten, meaning the results
would be identical to above

edgevpn

Due to the really poor results this test was only performed once.

The MTU choice of 8700/8920 was selected to mirror the difference between edgevpn defaults and the standard MTU of 1500.

MTU	Interface MTU	Test Proto	Avg ping (10)	Avg TX	Avg RX
8700	8920	IPv4	2.361ms	8.94Mbps	8.70Mbps
8700	8920	IPv6	2.259ms	8.34Mbps	8.03Mbps
Avg			2.310ms	8.64Mbps	8.37Mbps

IPv4 8700/8920

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.48 MBytes  12.4 Mbits/sec   37   8.45 KBytes       
[  5]   1.00-2.00   sec   591 KBytes  4.84 Mbits/sec   25   8.45 KBytes       
[  5]   2.00-3.00   sec  1.27 MBytes  10.7 Mbits/sec   37   8.45 KBytes       
[  5]   3.00-4.00   sec  1005 KBytes  8.23 Mbits/sec   25   16.9 KBytes       
[  5]   4.00-5.00   sec  1.27 MBytes  10.7 Mbits/sec   37   16.9 KBytes       
[  5]   5.00-6.00   sec   828 KBytes  6.78 Mbits/sec   26   8.45 KBytes       
[  5]   6.00-7.00   sec   887 KBytes  7.26 Mbits/sec   26   8.45 KBytes       
[  5]   7.00-8.00   sec  1.04 MBytes  8.72 Mbits/sec   34   16.9 KBytes       
[  5]   8.00-9.00   sec  1.27 MBytes  10.7 Mbits/sec   32   8.45 KBytes       
[  5]   9.00-10.00  sec  1.10 MBytes  9.20 Mbits/sec   28   16.9 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.7 MBytes  8.94 Mbits/sec  307             sender
[  5]   0.00-10.01  sec  10.4 MBytes  8.70 Mbits/sec                  receiver

IPv6 8700/8920

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.36 MBytes  11.4 Mbits/sec   37   8.43 KBytes       
[  5]   1.00-2.00   sec  1.32 MBytes  11.1 Mbits/sec   36   16.9 KBytes       
[  5]   2.00-3.00   sec  1003 KBytes  8.21 Mbits/sec   29   8.43 KBytes       
[  5]   3.00-4.00   sec  1.27 MBytes  10.6 Mbits/sec   38   8.43 KBytes       
[  5]   4.00-5.00   sec  1.44 MBytes  12.1 Mbits/sec   40   8.43 KBytes       
[  5]   5.00-6.00   sec   649 KBytes  5.31 Mbits/sec   17   8.43 KBytes       
[  5]   6.00-7.00   sec   767 KBytes  6.28 Mbits/sec   19   8.43 KBytes       
[  5]   7.00-8.00   sec   708 KBytes  5.80 Mbits/sec   24   16.9 KBytes       
[  5]   8.00-9.00   sec   413 KBytes  3.38 Mbits/sec   20   8.43 KBytes       
[  5]   9.00-10.00  sec  1.09 MBytes  9.18 Mbits/sec   29   8.43 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  9.94 MBytes  8.34 Mbits/sec  289             sender
[  5]   0.00-10.18  sec  9.75 MBytes  8.03 Mbits/sec                  receiver

Analysis

Overall these results again support the initial result that edgevpn is notably slower than both native and wireguard
connections. However, from some more experimenting with MTU's I think I may have identified the cause. The average RX
values were also close enough to the average TX that I will not consider them any further than including them for
reading.

My initial hypothesis was that this was MTU related, as it seemed to fit that edgevpn uses multiple nested
encapsulations. This was the motivation for performing the test locally with jumbo frames enabled. This proved to be
incorrect as demonstrated by the huge performance drop when the MTU was raised sufficiently high.

Relative ping

MTU 1500	Base	Wireguard	Tailscale	edgevpn
Base	100.00%	244.77%	410.11%	626.71%
Wireguard	40.86%	100.00%	167.55%	256.05%
Tailscale	24.38%	59.68%	100.00%	152.82%
edgevpn	15.96%	39.06%	65.44%	100.00%

MTU 9000	Base	Wireguard	edgevpn
Base	100.00%	143.91%	215.92%
Wireguard	69.49%	100.00%	150.04%
edgevpn	46.31%	66.65%	100.00%

These results are interesting because the increase in ping times appears to be roughly incremental for
wireguard/tailscale/edgevpn.
However, the reduction in this increment for jumbo frames is beyond my understanding.

Relative TX

MTU 1500	Base	Wireguard	Tailscale	edgevpn
Base	100.00%	95.30%	94.56%	11.31%
Wireguard	104.93%	100.00%	99.22%	11.87%
Tailscale	105.76%	100.79%	100.00%	11.96%
edgevpn	883.96%	824.45%	835.85%	100.00%

MTU 9000	Base	Wireguard	edgevpn
Base	100.00%	99.39%	0.87%
Wireguard	100.61%	100.00%	0.88%
edgevpn	11458.33%	11388.89%	100.00%

The similarity of throughputs for wireguard/tailscale is impressive given the difference in ping, as is the sudden drop
off from edgevpn.

Conclusions

It appears from casual observations that the poor performance of edgevpn is caused by being CPU bound whilst Wireguard
does not suffer from any such issues.
My hypothesis here is bottlenecks with go-libp2p implementation, and I wonder if using rust-libp2p³ may resolve these as
its possible.

My earlier assertion that a hybrid edgevpn/wireguard setup would be optimal for heavy workloads remains true.

Tl;dr

Wireguard achieves ~95% throughput of the host interface. edgevpn manages ~11%.

For survivability, edgevpn clearly wins as no coordination is required, but for throughput go with Wireguard.

1280 is the same MTU as tailscale networks ↩ ↩²
this was not expected to work, and I don't know why it did, but the results have been included out of
curiosity ↩ ↩²
this sounds like yet another fantastic way to procrastinate, rewrite edgevpn in rust with compatibility for the original. I will update with a reply if I have any findings to publish ↩

5 replies

infogulch Apr 6, 2024

I'm curious what the CPU usage is during during the bandwidth test. In units of CPU/Mbps.

Also it might be interesting to see a perf trace/flamegraph of tailscale/edgevpn (kernel mode may be more complicated to get perf data).

My hypothesis is that wireguard is just wildly more efficient.

tomtom5152 Apr 6, 2024

I didn’t measure it, but casual observation of top at the same time showed 200% CPU usage (so full dual core loading) by edgevpn process when running the above test, whilst WireGuard was around 1-2%. That was the case for both the 1500 MTU test and the 9000 one, with the wildly different throughput results.

I highly suspect you are correct hence starting to reimplement a minimal, compatible port of edgevpn in Rust to see if there is any appreciable speed difference that could go some way to explaining things as a proof of concept

infogulch Apr 7, 2024

Go is not 20000% slower than Rust, maybe 10%-30% at most if both have decent implementations. Note that tailscale is also implemented in Go and it only lagged the kernel throughput by a couple percent. I think the difference is not implementation language, but fundamental cryptography design. WireGuard was designed from the ground up using efficient primitives with the aim of encrypting traffic at near line rate with minimal CPU cost.

A better design would be to use libp2p as a network control plane to configure wireguard over which the actual traffic would be tunneled. Maybe it could be a backup if wireguard didn't work because of firewalls or something. In my unsolicited opinion, it would be a better use of time to pursue an architecture redesign rather than a language rewrite.

tomtom5152 Apr 7, 2024

My comment about the language was more a comment that go permits unclear/accidental memory copying, whilst rust’s strict borrow checking means you have to at least always been aware of if you are passing by reference or by value, not that rust is inherently orders of magnitude faster. This may be relevant at any point in the implementation down to something as mundane as string trimming, but if that mistake resulted in thousands of allocations/copies, it’d soon add up. Correct me if I’m wrong, but does tailscale not just defer to the system WireGuard implementation which I believe is usually the C one?
Writing it in a different language where the bulk is already done anyway would help eliminate those variables.

As for the fundamental architecture, I found edgevpn looking for “serverless tailscale”, so that idea had already crossed my mind. As for the idea, it’s certainly got legs to use libp2p as an alternative control server for WireGuard connections, I just don’t know if that should be a libp2p transport or something totally different. I also worry about relay traffic, is adding something like DERP ontop of DCUtR making even more back rods for future maintainers? I think these might be discussions best had over on libp2p itself.

infogulch Apr 7, 2024

I understand the benefits of Rust, but unless the Go implementation is comically awful then a few memory copies is not going to result in the difference observed. I think it's reasonable to assume that libp2p-go is moderately well written, though I have no evidence. You could be right that the Go implementation is extremely inefficient and produces an unreasonable amount of garbage as it tunnels packets, I think it's more likely that libp2p crypto is just not designed for low resource usage at high bandwidth and simply RIIR is not going to change that.

Yes tailscale will try to use the system wireguard if it is able, and it's likely that it did so, so you are correct that this does not help decide one way or another whether the issue is Go. Tailscale appears to have a flag to enable userspace networking mode: https://tailscale.com/kb/1112/userspace-networking#step-1-start-tailscaledtailscale-in-userspace-networking-mode

I think these might be discussions best had over on libp2p itself.

I agree, such a redesign has a bunch of open questions and the libp2p project is a better place to have them.

I did a few searches about performance & wireguard on the libp2p org:

Notably, the observable post from last August doesn't show a big difference between rust and go implementations, at least compared to line rate. (Also it doesn't show cpu usage.)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Does anyone have any stats comparing throughput of Wireguard vs Edgevpn using between same 2 Nodes? #157

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 5 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Does anyone have any stats comparing throughput of Wireguard vs Edgevpn using between same 2 Nodes? #157

Replies: 3 comments · 5 replies

Test criteria

Baseline tests

Wireguard

edgevpn

Conclusions

Tailscale (user-space Wireguard)

edgevpn take 2

Conclusions

Test overview

Standard 1500MTU

Baseline

Wireguard

Kernel mode

User mode (tailscale)

edgevpn

Jumbo frames

Baseline

Wireguard

Kernel mode

Usermode (tailscale)

edgevpn

Analysis

Relative ping

Relative TX

Conclusions

Tl;dr

Footnotes

Replies: 3 comments 5 replies