I need help troubleshooting slow wireguard tunnel (VPS to home network)
from stavefajl@feddit.dk to selfhosted@lemmy.world on 07 Oct 09:28
https://feddit.dk/post/16313088

Hello I hope one of you can help point me in the right direction.

I have a VPS with a static IP and a wireguard tunnel from VPS to home network (no bridging in the router, just point-to-point with specific devices).

I found an abysmal connection speed with bandwidth on the order of 50-100 kbps tested via iperf. Connection between the same devices outside the wireguard tunnel is 10-20 mbps, which is 100-400 times slower, which I don’t understand since wireguard usually has very little overhead.

I have tried different MTU settings on both VPS and devices on my home network (both cabled and via wi-fi) in the range from 1360 to 1460, and above speeds are the best I have reached with MTU 1420 and 1440. I have tried both with and without iptables rules setting the mss correspondingly.

The above speeds are acceptable for incremental backups and document synchronization, but completely unsuitable for media streaming.

Where would I start diagnosing the bottleneck?

Thanks in advance.

#selfhosted

threaded - newest

Brkdncr@lemmy.world on 07 Oct 09:46 next collapse

What iperf speeds are you getting with UDP?

On windows you can use the test-connection -MTUSize powershell command to figure out the correct mtu.

If your udp speeds are normal but your tcp speeds are not, it could be fragmentation, or out of order packets, or high latency.

stavefajl@feddit.dk on 07 Oct 09:59 next collapse

I just tested again:

  • no wireguard, tcp: 9.75 Mbits/s
  • no wireguard, udp: 1.05 Mbits/s
  • wireguard, tcp: 248 Kbits/s
  • wireguard, udp: 1.05 Mbits/s

So, I get the “full” udp speed, but I get some errors / warnings about ‘connection refused’ and ‘did not receive ack’. Obviously not correctly configured, when it is 10 times slower than tcp.

Brkdncr@lemmy.world on 07 Oct 10:16 collapse

What’s your mtu inside the vpn?

stavefajl@feddit.dk on 07 Oct 10:23 collapse

MTU is 1440.

stavefajl@feddit.dk on 07 Oct 10:02 collapse

I’m on linux on all devices. I tested packet fragmentation using:
ping -M do -s [packet size] [ip]
No fragmentation using packet size 1472 and below (before wireguard overhead).

stavefajl@feddit.dk on 07 Oct 10:46 next collapse

I am currently running mtr from multiple devices:

  • in wireguard tunnel: single hop 10.1.1.1, loss% 0,1, avg ping 27.4 ms
  • outside wg between same devices:
    • ISP supplied modem/router 55% loss, avg ping 1.6 ms
    • multiple hops without loss, avg ping 16-20 ms
    • random intermediary 30 % loss, avg ping 20 ms
    • endpoint, 5% loss, avg ping 25 ms

It looks like across the board my ISP modem / router is dropping 50-80 % of packets, and that packet loss is ramping up from 4% to 80 percent after a few minutes of running mtr.
It also looks like my VPS endpoint climbs to 20% packet loss over time (5-15 minutes of testing).

Can I use this information to probe further into the devices I have access to (ISP modem and VPS)?

Brkdncr@lemmy.world on 07 Oct 12:07 collapse

Some devices will drop icmp packets. Does iperf show the same amount of loss?

Can you try a different internet connection such as mobile hotspot just to see if you have similar results? That could help identify it as a vps issue.

stavefajl@feddit.dk on 07 Oct 20:01 collapse

I had to borrow a phone to set up a mobile hotspot.
It has the same speeds inside the wireguard tunnel as when I tested from my wired connection (250 kbps TCP, 170 kbps UDP).
The loss reported by iperf is dependent on the bandwidth that i test with. But as I increase bandwidth from the client the loss grows towards 100%.

I tried testing in reverse (sending from VPS to devices on different networks) with surprising results:

  • TCP, wireguard: 5-10 mbps
  • UDP, wireguard: 50 mbps
  • TCP, no wireguard: 45 mbps
  • UDP, no wireguard: 250 mbps (saturates download speed on client when compared to speedtest.net)
tychosmoose@lemmy.world on 07 Oct 13:13 collapse

Are you specifying bandwidth (-b) on the iperf UDP test? It defaults to 1M if I recall correctly, which would explain the result.

If not, try -b 10M or -b 0 for unlimited (the behavior used for TCP).

stavefajl@feddit.dk on 07 Oct 18:20 collapse

Thanks! I had not read the manpage close enough, I guess.
When specifying the bandwidth, I can saturate the connection using UDP to around 15 Mbits/s. (That this speed is much much lower than the 300-500 Mbits my connection and the VPS is capable of is a problem for a different time, I think).

What I also realized is, that I had not put the iperf server in UDP-mode, so my results reported in another comment are wrong.
I read the results from the client, but the server did not respond.
When running the iperf server in UDP-mode, I get 15 Mbits/s outside the wireguard tunnel and 180 Kbits/s inside the tunnel. With TCP-mode I get 10-15 Mbits/s outside the thunnel and 250 Kbits/s inside the tunnel.

tychosmoose@lemmy.world on 07 Oct 21:04 collapse

Just throwing out more ideas:

Is there a CPU spike on the VPS?

Anything weird about Wireguard on either end? Using kernel mode WG everywhere and not a user mode version, right?

As a test I would be inclined to try a very small mtu to see if it makes a difference. 1280 is a failsafe that I use when on unknown networks and trying to wg out.

Maybe try with a smaller packet size, like 1KB which I think is -l 1K