Fixing: Acme script breaking during install on Ubuntu with Apache and with configured server FQDN

[cypherpork]

During install, when:

• ubuntu hostname = desired FQDN of the new Jitsi server, and
• Jitsi VurtualHost is not up yet on :443:

Apache's default host responds to the "FQDN call" and default settings supersede the Jitsi VirtualHost.

As a result, Let's Encrypt challenge is served with Apache default webpage from /var/www/html, which does not have the /var/www/html/.well-known/... response.

Proposed solution modifies install-letsencrypt-cert.sh to detect this condition and disable default Apache site while Let's Encrypt process is working for the first time.

Merging this will address issue #14510.

[jitsi-jenkins]

Hi, thanks for your contribution!
If you haven't already done so, could you please make sure you sign our CLA (https://jitsi.org/icla for individuals and https://jitsi.org/ccla for corporations)? We would unfortunately be unable to merge your patch unless we have that piece :(.

[damencho]

I don't think the jitsi-meet package should be touching that configuration, people maybe using it for some other stuff.
If you think that maybe a problem we should print some warning to inform the user, so the user can decide what to do.

[damencho]

Hum, you enable it back ... why is the 000-default.conf a problem actually?

[damencho]

I have a question about apache and let's encrypt does it work with default jitsi-meet config?
Do we need to define here before the redirect:

jitsi-meet/doc/debian/jitsi-meet/jitsi-meet.example-apache

Line 4 in f525fb1

Redirect permanent / https://jitsi-meet.example.com/

Something like:

jitsi-meet/doc/debian/jitsi-meet/jitsi-meet.example

Line 34 in f525fb1

location ^~ /.well-known/acme-challenge/ {

?

[damencho]

Or you just say that it doesn't work because the jitsi-meet cofig is still not active while running the installation?

[cypherpork]

Or you just say that it doesn't work because the jitsi-meet cofig is still not active while running the installation?

Looking back to confirm the answer, I started to have doubts about my earlier statement that the error depends on timing.

Revisiting:

Experiment 1.
Reproducing the bug, looking deeper in the Apache state.

root@j:~# hostname
j.myserver.com
root@j:~# ##############################
root@j:~#
root@j:~# apachectl -S
VirtualHost configuration:
*:80                   is a NameVirtualHost
         default server j.myserver.com (/etc/apache2/sites-enabled/000-default.conf:1)
         port 80 namevhost j.myserver.com (/etc/apache2/sites-enabled/000-default.conf:1)
         port 80 namevhost j.myserver.com (/etc/apache2/sites-enabled/j.myserver.com.conf:2)
*:443                  j.myserver.com (/etc/apache2/sites-enabled/j.myserver.com.conf:7)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex watchdog-callback: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="www-data" id=33
Group: name="www-data" id=33
root@j:~#

Checkout the config conflict here:

         port 80 namevhost j.myserver.com (/etc/apache2/sites-enabled/000-default.conf:1)
         port 80 namevhost j.myserver.com (/etc/apache2/sites-enabled/j.myserver.com.conf:2)

In that state, http://j.myserver.com lands me on the "It Works!" page; http://j.myserver.com/.well-known/... is 404ing. There is no redirect. https://j.myserver.com lands me on the familiar Jitsi page, with self-signed certs. https://j.myserver.com/.well-known/... works.

Experiment 2.

1. Rolled back the same VM to the state just before apt install jitsi-meet.
2. Pointed a different 3LD to my public IP: jitsimeet.myserver.com. Left the hostname intact: j.myserver.com.
3. apt install jitsi-meet.

This time Let's Encrypt succeeded.
Apache state:

root@j:~# apachectl -S
VirtualHost configuration:
*:80                   is a NameVirtualHost
         default server j.myserver.com (/etc/apache2/sites-enabled/000-default.conf:1)
         port 80 namevhost j.myserver.com (/etc/apache2/sites-enabled/000-default.conf:1)
         port 80 namevhost jitsimeet.myserver.com (/etc/apache2/sites-enabled/jitsimeet.myserver.com.conf:2)
*:443                  jitsimeet.myserver.com (/etc/apache2/sites-enabled/jitsimeet.myserver.com.conf:7)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default
Mutex watchdog-callback: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="www-data" id=33
Group: name="www-data" id=33
root@j:~#

Note there is no conflict:

         port 80 namevhost j.myserver.com (/etc/apache2/sites-enabled/000-default.conf:1)
         port 80 namevhost jitsimeet.myserver.com (/etc/apache2/sites-enabled/jitsimeet.myserver.com.conf:2)

Conclusion:
I am taking it back that the "timing" issue has a part in it.

It looks like an apache bug. I can not think of a reason why Apache should not resolve a configuration like that in favor of the non-default VirtualHost.

If I am right classifying it as an Apache bug, I do not know what is the right way for Jitsi to respond:

• not fix it, publish a workaround in the self-hosted guide (ln -s or something else), or
• compensate for it in the Jitsi install process.

If you choose to provide a fix – I tried several working ways to do it. Disabling the default host and then enabling it back seem the most gentle to me, because it does not create any persistent changes in the foreign territory. If anyone thinks of a better way – I will be interested to know.

Either way, I will be happy to do a modified pull request.

[damencho]

Do I understand correctly that the fact that your machine has a hostname j.myserver.com is the reason that this is added as a default domain by apache config when installing it?

[damencho]

Maybe the problem is that j.myserver.com:80 is handled in two places and which one will be handling it is random, so it work or not randomly :)
I remember that apache was not allowing to conflicting virtual hosts back in the days I was using it ...
I will give it a try these days and will merge it.

[cypherpork]

Do I understand correctly that the fact that your machine has a hostname j.myserver.com is the reason that this is added as a default domain by apache config when installing it?

Correct.

[cypherpork]

Maybe the problem is that j.myserver.com:80 is handled in two places and which one will be handling it is random, so it work or not randomly :) I remember that apache was not allowing to conflicting virtual hosts back in the days I was using it ... I will give it a try these days and will merge it.

I also have hard time believing that Apache would miss such a glaring thing. If I did not miss anything obvious on my side, and it is indeed apache misbehaving, I would expect it to be specific to some combination of versions, platforms and environments.

In my case it is Apache/2.4.52 on aarch64 ubuntu 22.04.4 running inside a parallels vm on macOS Ventura on a Mac Studio. I was able to reproduce it 3 out of 3 times. I will be setting up a fresh Intel host on Monday, will try to reproduce it there.