v15beta is not properly overriding DNS records Ubuntu 20.04

[acherifi]

Hi,

We are trying to use this client to connect to openvpn cloud.
We are using our custom dns nameservers on OpenVPN cloud configured through the DNS setting of OpenVPN cloud.

We can connect without any issue to the VPN but the DNS resolution is not working at all.
We can ping/ssh/curl our different services via cli with the IP without any issue but DNS resolution is not working. It seems that systemd-resolve is using the public DNS and IP not matter what.

The thing to notice is that it is properly working on the beta13.

Did we miss some configuration steps to make it work properly ?
You will find below the outputs from both beta15 and beta13

Output from beta15

systemd-resolv --status

Global
       LLMNR setting: no                  
MulticastDNS setting: no                  
  DNSOverTLS setting: no                  
      DNSSEC setting: no                  
    DNSSEC supported: no                  
          DNSSEC NTA: 10.in-addr.arpa     
                      16.172.in-addr.arpa 
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa 
                      18.172.in-addr.arpa 
                      19.172.in-addr.arpa 
                      20.172.in-addr.arpa 
                      21.172.in-addr.arpa 
                      22.172.in-addr.arpa 
                      23.172.in-addr.arpa 
                      24.172.in-addr.arpa 
                      25.172.in-addr.arpa 
                      26.172.in-addr.arpa 
                      27.172.in-addr.arpa 
                      28.172.in-addr.arpa 
                      29.172.in-addr.arpa 
                      30.172.in-addr.arpa 
                      31.172.in-addr.arpa 
                      corp                
                      d.f.ip6.arpa        
                      home                
                      internal            
                      intranet            
                      lan                 
                      local               
                      private             
                      test                
Link 34 (tun0)
      Current Scopes: DNS        
DefaultRoute setting: yes        
       LLMNR setting: yes        
MulticastDNS setting: no         
  DNSOverTLS setting: no         
      DNSSEC setting: no         
    DNSSEC supported: no         
  Current DNS Server: 100.96.1.33
         DNS Servers: 100.96.1.33
          DNS Domain: ~.         
Link 3 (wlo1)
      Current Scopes: DNS                    
DefaultRoute setting: yes                    
       LLMNR setting: yes                    
MulticastDNS setting: no                     
  DNSOverTLS setting: no                     
      DNSSEC setting: no                     
    DNSSEC supported: no                     
  Current DNS Server: 192.168.43.137         
         DNS Servers: 2a04:cec0:1122:8deb::54
                      192.168.43.137         
          DNS Domain: ~.                     
Link 2 (eno2)
      Current Scopes: none
DefaultRoute setting: no  
       LLMNR setting: yes 
MulticastDNS setting: no  
  DNSOverTLS setting: no  
      DNSSEC setting: no  
    DNSSEC supported: no

cat /etc/resolv.conf

# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 127.0.0.53
options edns0 trust-ad

Output from Beta13

systemd-resolve --status

Global
         Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: foreign
Current DNS Server: 100.96.1.33
       DNS Servers: 100.96.1.33 127.0.1.1
        DNS Domain: net

Link 2 (eno2)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (enx64c901d792c8)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.1.1
       DNS Servers: 192.168.1.1
        DNS Domain: net

Link 4 (wlo1)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 5 (docker0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 7 (tun1)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

cat /etc/resolv.conf

#
# Generated by OpenVPN 3 Linux (NetCfg::DNS::ResolvConfFile)
# Last updated: 2021-07-23 14:02:19 
#
search net

# OpenVPN defined name servers
nameserver 100.96.1.33

# System defined name servers
nameserver 127.0.1.1

# Other system settings
options edns0 trust-ad

[JorisVanEijden]

Same issue on Pop!_OS 20.04 LTS

v13 systemd-resolve --status

Link 4 (tun0)
      Current Scopes: none
DefaultRoute setting: no  
       LLMNR setting: yes 
MulticastDNS setting: no  
  DNSOverTLS setting: no  
      DNSSEC setting: no  
    DNSSEC supported: no  

v13 /etc/resolv.conf

# OpenVPN defined name servers
nameserver 192.168.16.254

# System defined name servers
nameserver 127.0.0.53

v15 systemd-resolve --status

Link 6 (tun0)
      Current Scopes: DNS          
DefaultRoute setting: yes          
       LLMNR setting: yes          
MulticastDNS setting: no           
  DNSOverTLS setting: no           
      DNSSEC setting: no           
    DNSSEC supported: no           
  Current DNS Server: 192.168.16.254
         DNS Servers: 192.168.16.254
          DNS Domain: ~.    

v15 /etc/resolv.conf

# System defined name servers
nameserver 127.0.0.53

That DNS Domain is not correct. It is the same as the default DNS config in systemd-resolv which causes it to use one of them at random.

OpenVPN As version 2.8.7
"DNS zones" empty since we want to use our DNS for everything.
"Default domain suffix" filled, but that is ignored.

*edit after reading how systemd-resolved works:

• we have a public internet presence at *.example.com (wildcard dns)
• we have a private network at *.private.example.com (only resolvable by our own dns server over the vpn)

on v13: only the vpn-defined DNS server is used, both public and private urls resolve.
on v15: both the standard as wel as the vpn-defined DNS servers are used and the 1st non-failing result is returned. Private urls only resolve if the vpn DNS returns a success before the standard dns does.

[slesru]

Hello!

Is it possible to force openvpn change resolv.conf as before?

Thank you!

[dsommers]

What does the hosts: line in your /etc/nsswitch.conf file contain?

In v14_beta/v15_beta openvpn3-llinux switched over to use systemd-resolved integration by default. But if the NSS configuration is incorrect, it will query the wrong resolver. On my RHEL-8 box, it looks like this:

 hosts:      files mdns4_minimal [NOTFOUND=return] resolve dns myhostname

You must ensure resolve (aka systemd-resolved) is enslisted before dns (aka /etc/resolv.conf)

To quickly switch back to modifying /etc/resolv.conf you can modify /usr/share/dbus-1/system-services/net.openvpn.v3.netcfg.service and replace the --systemd-resolved argument with --resolv-conf /etc/resolv.conf. But that is just a hackish workaround and it will be reset on the next upgrade.

[slesru]

Well, Ubuntu 20.04 is completely different from RHEL here, as I can see.

/etc/nsswitch.conf changed nothing here, it contains
hosts: files dns
in my installation
and adding resolve has no effect.

But just found this issue
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1774632
changed resolv.conf to symlink as suggested.
Looks like it works...

After connection using openvpn2 I see my internal nameserver in
/run/systemd/resolve/resolv.conf
but it is not removed after disconnect...

So I changed everything back to resolv.conf method...

Thank you!

[dsommers]

Okay, so the launchpad OP is actually configuring systemd-resolved in a different way, not fully utilizing the possibilities with systemd-resolved. To be honest, I have no idea why Ubuntu is not making use of the resolve support via nsswitch.conf, or why that is not working for you.

It is well spent time reading through the systemd-resolved(8) man page.

So when just re-linking /etc/resolv.conf to /run/systemd/resolved/resolv.conf, depending on the priority of devices systemd-resolved has, it may not put the VPN provided DNS servers first. On my RHEL system, I see the DNS server provided via the ethernet interface enlisted first and then at the end comes the VPN ones in that file. But using the hosts line you've seen above together with /etc/resolv.conf being a symlink to /run/systemd/resolve/stub-resolv.conf gives the expected behavior on for me.

Is this box running in some kind of VPS service? Or is it bare metal? I do know that several of the VPS providers and cloud providers may have default images which deviates from the way stock Ubuntu install is configuring the host. Several of them might even ignore the fact that the distro has moved towards using systemd-resolved for DNS resolving; which makes this quite more complicated.

[slesru]

/run/systemd/resolve/stub-resolv.conf is not changed in Ubuntu 20.04, nameserver is only added to /run/systemd/resolve/resolv.conf , and, as I wrote before, openvpn nameserver is not removed from /run/systemd/resolve/resolv.conf after disconnect.

my system is bare metal, this is home desktop, but, as I can see on fresh 20.04 install in VM there is no resolve in nsswitch.conf.

btw, what is interesting here - reverse zone resolution works OK.
I looked into tcpdump and I see that Ubuntu asks both nameservers- system ( let's call it this name) and openvpn, but prefer answer from system.
So, really, systemd-resolved is broken in Ubuntu , at least by default, so we need resolv.conf support...

Thank you!

[dsommers]

That's really interesting. Also because I have reports now from other channels that DNS resolution is finally working as expected on 20.04 - where modifying resolv.conf directly caused issues. So 20.04 is clearly a challenging case.

I'm a bit at loss here now, how to proceed, as the "group" of users which was unhappy with this change are now happy, and vice versa. I'll at least try to improve how to switch between resolv.conf and systemd-resolved.

[Kender2]

In a nutshell:
previously: dns servers are queried sequentially.
now: all dns servers are queried in parallel.
In both cases the first non-fail answer is used.

So if your regular DNS responds with a NXDOMAIN answer before the VPN-specified DNS server responds with an answer you're out of luck.

This is the default behaviour without "DNS Resolution Zones" (systemd-resolve --status shows the '~.' domain at both normal and vpn interface)
You can configure these "DNS Resolution Zones" in your OpenVPN server to force the vpn dns server to be used for certain domains.
You can not ensure the vpn dns is always used first before any other dns.
And this is the behaviour that people depended upon and which has now changed.

Ubuntu 20.04 indeed does not use resolve in nsswitch.
It uses /etc/resolv.conf and tries the nameservers listed there sequentially.
The last (and now the only) entry is 127.0.0.53 which is the systemd-resolve daemon.
That looks at the nameservers and domains specified for the interfaces and uses all that match the requested domain ('~.' indicates the nameserver should be tried for all domains.

[dsommers]

@Kender2 Thanks for this clarification. This is gold, as it also gives a reasonable explanation why some users are more happy while some not with this change - on many setups, the VPN response is fast enough to not cause any issues - or that the DNS server otherwise used is getting blocked due to routes pushed out by the VPN server, which either slows down or makes that DNS inaccessible - while the VPN provided DNS server is available.

I've been pondering on the next systemd-resolved improvement, to get closer to the old behavior of "use only VPN DNS servers". Should we attempt to loop through all the interfaces and remove the ~. entry for all interfaces except the VPN (most likely enabling through some configuration flag)? (And naturally restore it again when the VPN session closes) ... Any thoughts on this?

[Kender2]

That could work I think. You'd have to sure to only do that when not using "split dns". The ~. entry for the vpn interface only appears when no domain is filled in the OpenVPN server.
It'd also remove the fallback behaviour where the next dns server is tried when the vpn one doesn't respond (or responds with NoAnswer due to dns rebinding protection or such)
You'd better make a sheet/matrix with all the possible combinations of configurations and resulting behaviours first.
Quite a puzzle.

[slesru]

I'd like to add that v16 resolving is still broken on ubuntu 20.04 , I see no difference from v15...

[dsommers]

Thanks for testing that. We still have some work on improving the DNS configuration. What should be easier with v16_beta is to revert back to the "old way" of manipulating /etc/resolv.conf. This should only be considered a quick workaround and not an ideal final solution.

To switch back, run this command:
# openvpn3-admin netcfg-service --config-set resolv-conf /etc/resolv.conf
# openvpn3-admin netcfg-service --config-set systemd-resolved false

You will need to stop the openvpn3-service-netcfg service before these changes will be activated. Currently the only way to restart this service is to kill -INT the process, reboot the system or disconnect all VPN sessions and wait 10 minutes and verify that the process has stopped running.

[johantiden]

Same on regolith/focal. openvpn3 v17_beta.
Trying the workaround above works sometimes but it seems a bit random (or volatile at least).

[johantiden]

Thanks for testing that. We still have some work on improving the DNS configuration. What should be easier with v16_beta is to revert back to the "old way" of manipulating /etc/resolv.conf. This should only be considered a quick workaround and not an ideal final solution.

To switch back, run this command: # openvpn3-admin netcfg-service --config-set resolv-conf /etc/resolv-conf # openvpn3-admin netcfg-service --config-set systemd-resolved false

You will need to stop the openvpn3-service-netcfg service before these changes will be activated. Currently the only way to restart this service is to kill -INT the process, reboot the system or disconnect all VPN sessions and wait 10 minutes and verify that the process has stopped running.

Is that a typo in the filename /etc/resolv-conf? Shoudn't it be /etc/resolv.conf?

I don't really understand what I'm doing but from the output of running it, there seems to be a config file. There was none on my machine but there is nothing indicating failure.

$ openvpn3-admin netcfg-service --config-set resolv-conf /etc/resolv.conf
Loading configuration file: /var/lib/openvpn3/netcfg.json
Configuration file updated.  Changes will be activated next time openvpn3-service-netcfg restarts

$ openvpn3-admin netcfg-service --config-set systemd-resolved false
Loading configuration file: /var/lib/openvpn3/netcfg.json
Configuration file updated.  Changes will be activated next time openvpn3-service-netcfg restarts

There is still no config file

$ cat /var/lib/openvpn3/netcfg.json
cat: /var/lib/openvpn3/netcfg.json: No such file or directory

If I create the file manually, the config changes:

$ sudo touch /var/lib/openvpn3/netcfg.json
$ sudo chown "$USERNAME" /var/lib/openvpn3/netcfg.json
$ openvpn3-admin netcfg-service --config-set resolv-conf /etc/resolv.conf
Loading configuration file: /var/lib/openvpn3/netcfg.json
Configuration file updated.  Changes will be activated next time openvpn3-service-netcfg restarts
$ cat /var/lib/openvpn3/netcfg.json
{
	//  Option --resolv-conf :: resolv-conf file
	"resolv_conf_file" : "/etc/resolv.conf"
}

Now here comes the kicker:
The config is no longer allowed when running the second line(!):

$ openvpn3-admin netcfg-service --config-set systemd-resolved false
Loading configuration file: /var/lib/openvpn3/netcfg.json
Configuration NOT changed due to the following error:

 *** These options cannot be combined: resolv-conf, systemd-resolved

[dsommers]

Is that a typo in the filename /etc/resolv-conf? Shoudn't it be /etc/resolv.conf?

Oh, yes! That's a typo. I've fixed it. Thanks!

I don't really understand what I'm doing but from the output of running it, there seems to be a config file. There was none on my machine but there is nothing indicating failure.

$ openvpn3-admin netcfg-service --config-set resolv-conf /etc/resolv.conf
Loading configuration file: /var/lib/openvpn3/netcfg.json
Configuration file updated.  Changes will be activated next time openvpn3-service-netcfg restarts

Okay, that's unfortunate. It should have been an error here. openvpn3-admin is intended to be run as root. I'll file that as a bug internally.

If I create the file manually, the config changes:

$ sudo touch /var/lib/openvpn3/netcfg.json
$ sudo chown "$USERNAME" /var/lib/openvpn3/netcfg.json

This step should not be needed when running openvpn3-admin as root. The openvpn user normally needs to own these files.

$ openvpn3-admin netcfg-service --config-set resolv-conf /etc/resolv.conf
Loading configuration file: /var/lib/openvpn3/netcfg.json
Configuration file updated. Changes will be activated next time openvpn3-service-netcfg restarts
$ cat /var/lib/openvpn3/netcfg.json
{
// Option --resolv-conf :: resolv-conf file
"resolv_conf_file" : "/etc/resolv.conf"
}

Now here comes the kicker: The config is no longer allowed when running the second line(!):

$ openvpn3-admin netcfg-service --config-set systemd-resolved false
Loading configuration file: /var/lib/openvpn3/netcfg.json
Configuration NOT changed due to the following error:

*** These options cannot be combined: resolv-conf, systemd-resolved

Yikes ... this was intended to work. I'll double check this. This will also be filed as a bug internally.

Thanks again for thorough testing and documenting.

[johantiden]

I'll file that as a bug internally.

👍

[johantiden]

I figured out that I needed to add sudo, after a couple of hours of debugging. Some more feedback from openvpn3-admin would have been nice 👯. Just so summarize the two things there:

1. Warn if not running as root
2. Double check that the config file exists and was actually written. The log saying Configuration file updated is very much false.

[johantiden]

tl;dr:
Activate the workaround by running

# disconnect
openvpn3 session-manage --path $(openvpn3 sessions-list | head -n 2 | tail -n 1 | cut -c 15-) --disconnect

sudo openvpn3-admin netcfg-service --config-set resolv-conf /etc/resolv.conf
sudo openvpn3-admin netcfg-service --config-set systemd-resolved false   # Not working for me

# Make sure the config is reloaded. openvpn3-service-netcfg must be killed.
sudo killall -INT openvpn3-service-netcfg

[dsommers]

In regards to the error related to this command line:

  sudo openvpn3-admin netcfg-service --config-set systemd-resolved false   # Not working for me

This is actually not a bug. It's me who confused it with some prior implementations which I dropped. Instead it should say:

 sudo openvpn3-admin netcfg-service --config-unset systemd-resolved

And it should be called first, before the --config-set resolv-conf command line.

[dsommers]

I don't really understand what I'm doing but from the output of running it, there seems to be a config file. There was none on my machine but there is nothing indicating failure.

$ openvpn3-admin netcfg-service --config-set resolv-conf /etc/resolv.conf
Loading configuration file: /var/lib/openvpn3/netcfg.json
Configuration file updated.  Changes will be activated next time openvpn3-service-netcfg restarts

Okay, that's unfortunate. It should have been an error here. openvpn3-admin is intended to be run as root. I'll file that as a bug internally.

This should now be fixed with commit c40218d. The output now is still not optimal, but at least gives a pretty good idea:

$ openvpn3-admin netcfg-service --config-set resolv-conf /tmp/test
Loading configuration file: /var/lib/openvpn3/netcfg.json
** ERROR ** Configuration file error in /var/lib/openvpn3/netcfg.json: Error saving the configuration file

[walterbray]

I was having this same issue, I'm using Ubuntu 20.04, with OpenVPN 3/Linux v18_beta (openvpn3).

I followed the instructions above (thanks johantiden and dsommers) and resolved it by:

user@hostname:~$ openvpn3 session-manage --path $(openvpn3 sessions-list | head -n 2 | tail -n 1 | cut -c 15-) --disconnect
Initiated session shutdown.

Connection statistics:
     BYTES_IN...............119344058
     BYTES_OUT...............85034916
     PACKETS_IN................350732
     PACKETS_OUT...............651178
     TUN_BYTES_IN............66570523
     TUN_BYTES_OUT..........104480638
     TUN_PACKETS_IN............609700
     TUN_PACKETS_OUT...........470655
     KEEPALIVE_TIMEOUT..............2
     N_RECONNECT...................10

user@hostname:~$ sudo openvpn3-admin netcfg-service --config-unset systemd-resolved
Loading configuration file: /var/lib/openvpn3/netcfg.json

user@hostname:~$ sudo openvpn3-admin netcfg-service --config-set resolv-conf /etc/resolv.conf
Loading configuration file: /var/lib/openvpn3/netcfg.json

Configuration file updated.  Changes will be activated next time openvpn3-service-netcfg restarts
user@hostname:~$ sudo killall -INT openvpn3-service-netcfg

After that I connected the usual way and DNS behaved as expected (finally!). The proper behavior persists even after disconnect and reconnect, as well as after reboot.

I guess someday I will see if 22.04 handles this better.

[cgwhouse]

I was in the process of getting my machine set up, with a clean install (22.04), and ran into the same issue... @walterbray your steps worked well for me. Really learned from the discussion too as I'm still new to Linux. Thanks all! Just dropping in to say that this seems to affect 22.04 as well, or at least it did in my case.