Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Unhelpful error message when importlib.metadata.version returns None (which it shouldn't do normally!) #12223

Closed
Videothek opened this issue Mar 14, 2022 · 34 comments
Assignees
Labels
X-Needs-Info This issue is blocked awaiting information from the reporter

Comments

@Videothek
Copy link

Videothek commented Mar 14, 2022

Description

I cant say it for sure but i just wanted to post it here just that you are aware of this problem. Today my Homeserver was running normally till it wasn't anymore. I got some python syntax errors. I tried to reinstall matrix-synapse-py3 but it hasn't worked. I reseted my homeserver.yaml to the default but the problem persisted. I checked for possible updates with apt but that wasn't the case either. I brought back my backup from yesterday and it worked again. After running apt update and apt list --upgradable i saw that matrix has an update. So i figured that it must be the unattended upgrade that installed the new version which somehow broke my server.

Steps to reproduce

  • run apt update
  • run apt upgrade

I expected it to work fine and it doesn't. I needed to bring up my backup.

Version information

  • Homeserver
  • Version: 1.44.0

  • python version: 3.8.10

  • Install method: package manager (apt)

  • Platform: Ubuntu Server 20.04 on a VM
@clokep
Copy link
Contributor

clokep commented Mar 14, 2022

@Videothek Any chance you have the logs from with the error in it?

@clokep clokep added the X-Needs-Info This issue is blocked awaiting information from the reporter label Mar 14, 2022
@TheLinuxNinja
Copy link

TheLinuxNinja commented Mar 15, 2022

I'm on Ubuntu 20.04.4 LTS. I ran the same upgrade to Synapse 1.54.0. I did not run into any issues.
All my homeserver.yaml settings are in /etc/matrix-synapse/conf.d instead of modifying the /etc/matrix-synapse/homeserver.yaml
I have a /etc/apt/sources.list.d/matrix-org.list that points to the upstream repo:
deb [signed-by=/usr/share/keyrings/matrix-org-archive-keyring.gpg] https://packages.matrix.org/debian/ focal main
I also use postgresql from their upstream /etc/apt/sources.list.d/pgdg.list :
deb http://apt.postgresql.org/pub/repos/apt/ focal-pgdg main

@Videothek
Copy link
Author

@Videothek Any chance you have the logs from with the error in it?

Unfortunately not because installed the complete backup. I will try to update to the new version today again and will keep this issue up to date. If the same error occurs i will post the error message and the logs here.

@Videothek
Copy link
Author

I'm on Ubuntu 20.04.4 LTS. I ran the same upgrade to Synapse 1.54.0. I did not run into any issues.

All my homeserver.yaml settings are in /etc/matrix-synapse/conf.d instead of modifying the /etc/matrix-synapse/homeserver.yaml

I have a /etc/apt/sources.list.d/matrix-org.list that points to the upstream repo:

deb [signed-by=/usr/share/keyrings/matrix-org-archive-keyring.gpg] https://packages.matrix.org/debian/ focal main

I also use postgresql from their upstream /etc/apt/sources.list.d/pgdg.list :

deb http://apt.postgresql.org/pub/repos/apt/ focal-pgdg main

Ok so this is good news because it means it is only a me problem. As mentioned i will try to install it again today. So far i was ok with having the config in my homeserver.yaml maybe its time to switch. Do you have any tutorial how to add the settings to /etc/matrix-synapse/conf.d ? I can also check my apt lists later, maybe i have the wrong repos installed which i dont think but you never know.

@squahtx
Copy link
Contributor

squahtx commented Mar 15, 2022

Possibly #12185. though it's hard to tell without the logs. If you're running on sqlite with enable_search off in the config, you can try temporarily turning that setting on to allow the upgrade to succeed.

@Videothek
Copy link
Author

Videothek commented Mar 15, 2022

Possibly #12185. though it's hard to tell without the logs. If you're running on sqlite with enable_search off in the config, you can try temporarily turning that setting on enable_search to allow the upgrade to succeed.

I looked at the errors and mine were diffrent as far as i remember correctly there wasnt anything regarding sqlite in the errors. I am aware that troubleshooting without logs is nearly impossible. But today i will try to upgrade again and will post the logs and errors if the same problem occures again.

@Videothek
Copy link
Author

Videothek commented Mar 15, 2022

Well as promised i upgraded matrix-synapse. It throws the errors again so i will just post them here maybe anybody can help me.

systemctl status matrix-synapse.service throws following:

Mar 15 17:05:56 matrix systemd[1]: matrix-synapse.service: Scheduled restart job, restart counter is at 13.
Mar 15 17:05:56 matrix systemd[1]: Stopped Synapse Matrix homeserver.
Mar 15 17:05:56 matrix systemd[1]: Starting Synapse Matrix homeserver...
Mar 15 17:05:57 matrix matrix-synapse[2252]: Traceback (most recent call last):
Mar 15 17:05:57 matrix matrix-synapse[2252]:   File "/usr/lib/python3.8/runpy.py", line 185, in _run_module_as_main
Mar 15 17:05:57 matrix matrix-synapse[2252]:     mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
Mar 15 17:05:57 matrix matrix-synapse[2252]:   File "/usr/lib/python3.8/runpy.py", line 111, in _get_module_details
Mar 15 17:05:57 matrix matrix-synapse[2252]:     __import__(pkg_name)
Mar 15 17:05:57 matrix matrix-synapse[2252]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/app/__init__.py", line 23, in <module>
Mar 15 17:05:57 matrix matrix-synapse[2252]:     check_dependencies.check_requirements()
Mar 15 17:05:57 matrix matrix-synapse[2252]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/util/check_dependencies.py", line 167, in check_requirements
Mar 15 17:05:57 matrix matrix-synapse[2252]:     if not requirement.specifier.contains(dist.version, prereleases=True):
Mar 15 17:05:57 matrix matrix-synapse[2252]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/packaging/specifiers.py", line 728, in contains
Mar 15 17:05:57 matrix matrix-synapse[2252]:     item = parse(item)
Mar 15 17:05:57 matrix matrix-synapse[2252]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/packaging/version.py", line 49, in parse
Mar 15 17:05:57 matrix matrix-synapse[2252]:     return Version(version)
Mar 15 17:05:57 matrix matrix-synapse[2252]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/packaging/version.py", line 264, in __init__
Mar 15 17:05:57 matrix matrix-synapse[2252]:     match = self._regex.search(version)
Mar 15 17:05:57 matrix matrix-synapse[2252]: TypeError: expected string or bytes-like object

And executing journalctl -xe throwes this:

Mar 15 17:08:48 matrix systemd[1]: Starting Synapse Matrix homeserver...
ubject: A start job for unit matrix-synapse.service has begun execution
Defined-By: systemd
Support: http://www.ubuntu.com/support

A start job for unit matrix-synapse.service has begun execution.
 
The job identifier is 5769.
Mar 15 17:08:48 matrix matrix-synapse[2557]: Traceback (most recent call last):
Mar 15 17:08:48 matrix matrix-synapse[2557]:   File "/usr/lib/python3.8/runpy.py", line 185, in _run_module_as_main
Mar 15 17:08:48 matrix matrix-synapse[2557]:     mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
Mar 15 17:08:48 matrix matrix-synapse[2557]:   File "/usr/lib/python3.8/runpy.py", line 111, in _get_module_details
Mar 15 17:08:48 matrix matrix-synapse[2557]:     __import__(pkg_name)
Mar 15 17:08:48 matrix matrix-synapse[2557]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/app/__init__.py", line 23, in <module>
Mar 15 17:08:48 matrix matrix-synapse[2557]:     check_dependencies.check_requirements()
Mar 15 17:08:48 matrix matrix-synapse[2557]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/util/check_dependencies.py", line 167, in check_requirements
Mar 15 17:08:48 matrix matrix-synapse[2557]:     if not requirement.specifier.contains(dist.version, prereleases=True):
Mar 15 17:08:48 matrix matrix-synapse[2557]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/packaging/specifiers.py", line 728, in contains
Mar 15 17:08:48 matrix matrix-synapse[2557]:     item = parse(item)
Mar 15 17:08:48 matrix matrix-synapse[2557]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/packaging/version.py", line 49, in parse
Mar 15 17:08:48 matrix matrix-synapse[2557]:     return Version(version)
Mar 15 17:08:48 matrix matrix-synapse[2557]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/packaging/version.py", line 264, in __init__
Mar 15 17:08:48 matrix matrix-synapse[2557]:     match = self._regex.search(version)
Mar 15 17:08:48 matrix matrix-synapse[2557]: TypeError: expected string or bytes-like object
Mar 15 17:08:49 matrix systemd[1]: matrix-synapse.service: Control process exited, code=exited, status=1/FAILURE
 Subject: Unit process exited
 Defined-By: systemd
Support: http://www.ubuntu.com/support

 An ExecStartPre= process belonging to unit matrix-synapse.service has exited.

The process' exit code is 'exited' and its exit status is 1.
Mar 15 17:08:49 matrix systemd[1]: matrix-synapse.service: Failed with result 'exit-code'.
Subject: Unit failed
Defined-By: systemd
Support: http://www.ubuntu.com/support

The unit matrix-synapse.service has entered the 'failed' state with result 'exit-code'.
Mar 15 17:08:49 matrix systemd[1]: Failed to start Synapse Matrix homeserver.
Subject: A start job for unit matrix-synapse.service has failed
Defined-By: systemd
Support: http://www.ubuntu.com/support

A start job for unit matrix-synapse.service has finished with a failure.

And the logs are especially useless with giving no Error or anything at all.

Maybe someone can help me because not upgrading to a new Version is never a good fix.

Just for clearance: I keept my modifyed homeserver.yaml when asked during updating.

Thanks in Advance.

@Videothek
Copy link
Author

I'm on Ubuntu 20.04.4 LTS. I ran the same upgrade to Synapse 1.54.0. I did not run into any issues. All my homeserver.yaml settings are in /etc/matrix-synapse/conf.d instead of modifying the /etc/matrix-synapse/homeserver.yaml I have a /etc/apt/sources.list.d/matrix-org.list that points to the upstream repo: deb [signed-by=/usr/share/keyrings/matrix-org-archive-keyring.gpg] https://packages.matrix.org/debian/ focal main I also use postgresql from their upstream /etc/apt/sources.list.d/pgdg.list : deb http://apt.postgresql.org/pub/repos/apt/ focal-pgdg main

So for matrix it is the same key and for the postgresql i dont have any special key.

@DMRobertson
Copy link
Contributor

I'm not able to reproduce this using an Ubuntu 20.04 docker image. Can you please run /opt/venvs/matrix-synapse/bin/pip freeze and attach the output here?

@Videothek
Copy link
Author

Videothek commented Mar 15, 2022

Ok weird seems like i have a specific problem.

@DMRobertson This is the output of /opt/venvs/matrix-synapse/bin/pip freeze

attrs==21.4.0
Authlib==0.15.5
Automat==20.2.0
bcrypt==3.2.0
bleach==4.1.0
canonicaljson==1.6.0
certifi==2021.10.8
cffi==1.15.0
charset-normalizer==2.0.12
constantly==15.1.0
cryptography==36.0.1
defusedxml==0.7.1
elementpath==2.5.0
frozendict==2.3.0
hiredis==2.0.0
hyperlink==21.0.0
idna==3.3
ijson==3.1.4
importlib-metadata==4.11.2
importlib-resources==5.4.0
incremental==21.3.0
jaeger-client==4.8.0
Jinja2==3.0.3
jsonschema==4.4.0
ldap3==2.9.1
lxml==4.8.0
MarkupSafe==2.1.0
matrix-common==1.1.0
matrix-synapse @ file:///synapse/build
matrix-synapse-ldap3==0.1.5
mock==4.0.3
msgpack==1.0.3
netaddr==0.8.0
opentracing==2.4.0
packaging==21.3
parameterized==0.8.1
phonenumbers==8.12.44
Pillow==9.0.1
pkg_resources==0.0.0
prometheus-client==0.13.1
psycopg2==2.9.3
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.21
PyJWT==2.3.0
pymacaroons==0.13.0
Pympler==1.0.1
PyNaCl==1.5.0
pyOpenSSL==22.0.0
pyparsing==3.0.7
pyrsistent==0.18.1
pysaml2==7.1.2
python-dateutil==2.8.2
pytz==2021.3
PyYAML==6.0
requests==2.27.1
sentry-sdk==1.5.7
service-identity==21.1.0
signedjson==1.1.1
simplejson==3.17.6
six==1.16.0
sortedcontainers==2.4.0
systemd-python==234
threadloop==1.0.2
thrift==0.15.0
tornado==6.1
treq==22.2.0
Twisted==22.2.0
txredisapi==1.4.7
typing_extensions==4.1.1
unpaddedbase64==2.1.0
urllib3==1.26.8
webencodings==0.5.1
xmlschema==1.10.0
zipp==3.7.0
zope.interface==5.4.0

@DMRobertson
Copy link
Contributor

That pip freeze list matches what I see.

You could try adding a line which prints out the version information before everything goes wrong. E.g. edit /opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/util/check_dependencies.py and add a print call as follows. Then try to start Synapse and show the us the logs just before your traceback.

--- /opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/util/check_dependencies_orig.py	2022-03-15 17:10:44.788970350 +0000
+++ /opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/util/check_dependencies.py	2022-03-15 17:11:05.672812181 +0000
@@ -164,6 +164,7 @@
                 errors.append(_not_installed(requirement, extra))
         else:
             # We specify prereleases=True to allow prereleases such as RCs.
+            print(f"DEBUG: {requirement}, {must_be_installed}, {dist}, {dist.version!r}")
             if not requirement.specifier.contains(dist.version, prereleases=True):
                 deps_unfulfilled.append(requirement.name)
                 errors.append(_incorrect_version(requirement, dist.version, extra))

@Videothek
Copy link
Author

Videothek commented Mar 15, 2022

@DMRobertson Ok so first of all thank you for your reply. Unfortunately it hasnt fixed anything.

I added the following to /opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/util/check_dependencies.py

--- /opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/util/check_dependencies_orig.py	2022-03-15 17:10:44.788970350 +0000
+++ /opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/util/check_dependencies.py	2022-03-15 17:11:05.672812181 +0000
@@ -164,6 +164,7 @@
                 errors.append(_not_installed(requirement, extra))
         else:
             # We specify prereleases=True to allow prereleases such as RCs.
+            print(f"DEBUG: {requirement}, {must_be_installed}, {dist}, {dist.version!r}")
             if not requirement.specifier.contains(dist.version, prereleases=True):
                 deps_unfulfilled.append(requirement.name)
                 errors.append(_incorrect_version(requirement, dist.version, extra))

No the error is this:

-- The job identifier is 265453.
Mar 15 20:22:50 matrix matrix-synapse[15539]: Traceback (most recent call last):
Mar 15 20:22:50 matrix matrix-synapse[15539]:   File "/usr/lib/python3.8/runpy.py", line 185, in _run_module_as_main
Mar 15 20:22:50 matrix matrix-synapse[15539]:     mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
Mar 15 20:22:50 matrix matrix-synapse[15539]:   File "/usr/lib/python3.8/runpy.py", line 111, in _get_module_details
Mar 15 20:22:50 matrix matrix-synapse[15539]:     __import__(pkg_name)
Mar 15 20:22:50 matrix matrix-synapse[15539]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/app/__init__.py", line 18, in <module>
Mar 15 20:22:50 matrix matrix-synapse[15539]:     from synapse.util import check_dependencies
Mar 15 20:22:50 matrix matrix-synapse[15539]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/util/check_dependencies.py", line 23
Mar 15 20:22:50 matrix matrix-synapse[15539]:     --- /opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/util/check_dependencies_orig.py        2022-03-15 17:10:44.788970350 +0000
Mar 15 20:22:50 matrix matrix-synapse[15539]:         ^
Mar 15 20:22:50 matrix matrix-synapse[15539]: SyntaxError: invalid syntax
Mar 15 20:22:50 matrix systemd[1]: matrix-synapse.service: Control process exited, code=exited, status=1/FAILURE

@Videothek
Copy link
Author

I changed the content of the file because i figured that the first two lines are unnecessary.

So it looks like this:

@@ -164,6 +164,7 @@
                 errors.append(_not_installed(requirement, extra))
         else:
             # We specify prereleases=True to allow prereleases such as RCs.
+            print(f"DEBUG: {requirement}, {must_be_installed}, {dist}, {dist.version!r}")
             if not requirement.specifier.contains(dist.version, prereleases=True):
                 deps_unfulfilled.append(requirement.name)
                 errors.append(_incorrect_version(requirement, dist.version, extra))

The error persits anyways but changed:

-- The job identifier is 315139.
Mar 15 20:58:53 matrix matrix-synapse[18512]: Traceback (most recent call last):
Mar 15 20:58:53 matrix matrix-synapse[18512]:   File "/usr/lib/python3.8/runpy.py", line 185, in _run_module_as_main
Mar 15 20:58:53 matrix matrix-synapse[18512]:     mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
Mar 15 20:58:53 matrix matrix-synapse[18512]:   File "/usr/lib/python3.8/runpy.py", line 111, in _get_module_details
Mar 15 20:58:53 matrix matrix-synapse[18512]:     __import__(pkg_name)
Mar 15 20:58:53 matrix matrix-synapse[18512]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/app/__init__.py", line 18, in <module>
Mar 15 20:58:53 matrix matrix-synapse[18512]:     from synapse.util import check_dependencies
Mar 15 20:58:53 matrix matrix-synapse[18512]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/util/check_dependencies.py", line 24
Mar 15 20:58:53 matrix matrix-synapse[18512]:     @@ -164,6 +164,7 @@
Mar 15 20:58:53 matrix matrix-synapse[18512]:      ^
Mar 15 20:58:53 matrix matrix-synapse[18512]: SyntaxError: invalid syntax
Mar 15 20:58:53 matrix systemd[1]: matrix-synapse.service: Control process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support

@DMRobertson
Copy link
Contributor

@DMRobertson Ok so first of all thank you for your reply. Unfortunately it hasnt fixed anything.

The point of this is to work out what your problem is; we can't provide a fix until we understand the root cause.

So it looks like this:

The example I gave was a "unified diff" to explain how to make the change (and give the nice highlighting on github). In a nutshell:

  • don't include the lines starting with +++ or --- or @@
  • do include the line with print, but don't include the initial +. It'll need to line up properly.

After doing so, the file should look like this:

                 errors.append(_not_installed(requirement, extra))
         else:
             # We specify prereleases=True to allow prereleases such as RCs.
             print(f"DEBUG: {requirement}, {must_be_installed}, {dist}, {dist.version!r}")
             if not requirement.specifier.contains(dist.version, prereleases=True):
                 deps_unfulfilled.append(requirement.name)
                 errors.append(_incorrect_version(requirement, dist.version, extra))

@Videothek
Copy link
Author

@DMRobertson Ok so first of all thank you for your reply. Unfortunately it hasnt fixed anything.

The point of this is to work out what your problem is; we can't provide a fix until we understand the root cause.

So it looks like this:

The example I gave was a "unified diff" to explain how to make the change (and give the nice highlighting on github). In a nutshell:

* don't include the lines starting with `+++` or `---` or `@@`

* do include the line with `print`, but don't include the initial `+`. It'll need to line up properly.

After doing so, the file should look like this:

                 errors.append(_not_installed(requirement, extra))
         else:
             # We specify prereleases=True to allow prereleases such as RCs.
             print(f"DEBUG: {requirement}, {must_be_installed}, {dist}, {dist.version!r}")
             if not requirement.specifier.contains(dist.version, prereleases=True):
                 deps_unfulfilled.append(requirement.name)
                 errors.append(_incorrect_version(requirement, dist.version, extra))

Yeah makes sense, just tell me what you need me to do to figure out the root cause.

Thank you for explaning that to me, i am not that familiar with coding.

With the correct lines in the code it spits out this:

-- The job identifier is 1276035.
Mar 16 08:47:59 matrix matrix-synapse[72273]: Traceback (most recent call last):
Mar 16 08:47:59 matrix matrix-synapse[72273]:   File "/usr/lib/python3.8/runpy.py", line 185, in _run_module_as_main
Mar 16 08:47:59 matrix matrix-synapse[72273]:     mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
Mar 16 08:47:59 matrix matrix-synapse[72273]:   File "/usr/lib/python3.8/runpy.py", line 111, in _get_module_details
Mar 16 08:47:59 matrix matrix-synapse[72273]:     __import__(pkg_name)
Mar 16 08:47:59 matrix matrix-synapse[72273]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/app/__init__.py", line 18, in <module>
Mar 16 08:47:59 matrix matrix-synapse[72273]:     from synapse.util import check_dependencies
Mar 16 08:47:59 matrix matrix-synapse[72273]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/util/check_dependencies.py", line 25
Mar 16 08:47:59 matrix matrix-synapse[72273]:     errors.append(_not_installed(requirement, extra))
Mar 16 08:47:59 matrix matrix-synapse[72273]:     ^
Mar 16 08:47:59 matrix matrix-synapse[72273]: IndentationError: unexpected indent
Mar 16 08:47:59 matrix systemd[1]: matrix-synapse.service: Control process exited, code=exited, status=1/FAILURE

Maybe i have done something wrong again so just let me know what you think is the problem.

Just that i said it, i copied your code to the beginning of the file, so there are outcommented lines above and the rest of the file comes after that.

Thank you in advance.

DMRobertson pushed a commit that referenced this issue Mar 16, 2022
@DMRobertson
Copy link
Contributor

Hi @Videothek, and thanks for bearing with me.

I think it might be easier to download a copy of the file which has my debug changes applied. It is available here. You can apply this to your server with the following command. I wouldn't normally recommend this, since this command will download and overwrite a file onto your system which will ultimately get run by Synapse. (In general, you should be extremely suspicious if someone asks you to run a command you don't understand.)

curl https://raw.githubusercontent.com/matrix-org/synapse/4c988172646877bbafa8284e0faa4deb35a1c7d1/synapse/util/check_dependencies.py -o /opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/util/check_dependencies.py

To undo this, run apt reinstall matrix-synapse-py3.

@Videothek
Copy link
Author

Hey @DMRobertson thank you for your patience.

I have replaced the file and it spits out errors again but diffrent once than last time:

-- The job identifier is 1969004.
Mar 16 17:15:02 matrix matrix-synapse[110772]: DEP CHECK: jsonschema>=3.0.0, True, <importlib.metadata.PathDistribution object at 0x7f1ed36df130>, '4.0.1'
Mar 16 17:15:02 matrix matrix-synapse[110772]: DEP CHECK: frozendict>=1, True, <importlib.metadata.PathDistribution object at 0x7f1ed36df040>, '2.3.0'
Mar 16 17:15:02 matrix matrix-synapse[110772]: DEP CHECK: unpaddedbase64>=1.1.0, True, <importlib.metadata.PathDistribution object at 0x7f1ed36df400>, '2.1.0'
Mar 16 17:15:02 matrix matrix-synapse[110772]: DEP CHECK: canonicaljson>=1.4.0, True, <importlib.metadata.PathDistribution object at 0x7f1ed36df640>, '1.6.0'
Mar 16 17:15:02 matrix matrix-synapse[110772]: DEP CHECK: signedjson>=1.1.0, True, <importlib.metadata.PathDistribution object at 0x7f1ed36df880>, '1.1.1'
Mar 16 17:15:02 matrix matrix-synapse[110772]: DEP CHECK: pynacl>=1.2.1, True, <importlib.metadata.PathDistribution object at 0x7f1ed36dfac0>, '1.5.0'
Mar 16 17:15:02 matrix matrix-synapse[110772]: DEP CHECK: idna>=2.5, True, <importlib.metadata.PathDistribution object at 0x7f1ed36dfd00>, None
Mar 16 17:15:02 matrix matrix-synapse[110772]: Traceback (most recent call last):
Mar 16 17:15:02 matrix matrix-synapse[110772]:   File "/usr/lib/python3.8/runpy.py", line 185, in _run_module_as_main
Mar 16 17:15:02 matrix matrix-synapse[110772]:     mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
Mar 16 17:15:02 matrix matrix-synapse[110772]:   File "/usr/lib/python3.8/runpy.py", line 111, in _get_module_details
Mar 16 17:15:02 matrix matrix-synapse[110772]:     __import__(pkg_name)
Mar 16 17:15:02 matrix matrix-synapse[110772]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/app/__init__.py", line 23, in <module>
Mar 16 17:15:02 matrix matrix-synapse[110772]:     check_dependencies.check_requirements()
Mar 16 17:15:02 matrix matrix-synapse[110772]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/util/check_dependencies.py", line 172, in check_requirements
Mar 16 17:15:02 matrix matrix-synapse[110772]:     if not requirement.specifier.contains(dist.version, prereleases=True):
Mar 16 17:15:02 matrix matrix-synapse[110772]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/packaging/specifiers.py", line 728, in contains
Mar 16 17:15:02 matrix matrix-synapse[110772]:     item = parse(item)
Mar 16 17:15:02 matrix matrix-synapse[110772]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/packaging/version.py", line 49, in parse
Mar 16 17:15:02 matrix matrix-synapse[110772]:     return Version(version)
Mar 16 17:15:02 matrix matrix-synapse[110772]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/packaging/version.py", line 264, in __init__
Mar 16 17:15:02 matrix matrix-synapse[110772]:     match = self._regex.search(version)
Mar 16 17:15:02 matrix matrix-synapse[110772]: TypeError: expected string or bytes-like object
Mar 16 17:15:02 matrix systemd[1]: matrix-synapse.service: Control process exited, code=exited, status=1/FAILURE

I really hope we are coming the solution closer.

I am considering installing a fresh ubuntu server and test if a clean install would fix it.

Is there a good way to backup messages and user settings to import them on a fresh install?

@DMRobertson
Copy link
Contributor

DMRobertson commented Mar 16, 2022

For some reason, idna has version None on your system. I'm not sure why. To confirm this, could you provide the output of these commands?

/opt/venvs/matrix-synapse/bin/python --version
/opt/venvs/matrix-synapse/bin/python -c "import importlib.metadata as m; print(repr(m.version('idna')))"
/opt/venvs/matrix-synapse/bin/python -c "import importlib.metadata as m; print(repr(m.metadata('idna').keys()))"

You could try

/opt/venvs/matrix-synapse/bin/pip uninstall idna && /opt/venvs/matrix-synapse/bin/pip install idna==3.3

to see if that helps. Failing that, I'd suggest apt reinstall matrix-synapse-py3 and see if that helps.

I am considering installing a fresh ubuntu server and test if a clean install would fix it.
Is there a good way to backup messages and user settings to import them on a fresh in
stall?

I'd save a clean install as a last resort.

This should all be kept in the database that Synapse uses. If you're going to do a clean install, you must back this up. You probably also wants to backup the configuration files for Synapse itself. #synapse:matrix.org would probably be able to better advise.

@DMRobertson
Copy link
Contributor

For some reason, idna has version None on your system.

Possibly due to python/importlib_metadata#291 (comment)?

@Videothek
Copy link
Author

Videothek commented Mar 16, 2022

For some reason, idna has version None on your system. I'm not sure why. To confirm this, could you provide the output of these commands?

/opt/venvs/matrix-synapse/bin/python --version
/opt/venvs/matrix-synapse/bin/python -c "import importlib.metadata as m; print(repr(m.version('idna')))"
/opt/venvs/matrix-synapse/bin/python -c "import importlib.metadata as m; print(repr(m.metadata('idna').keys()))"

You could try

/opt/venvs/matrix-synapse/bin/pip uninstall idna && /opt/venvs/matrix-synapse/bin/pip install idna==3.3

to see if that helps. Failing that, I'd suggest apt reinstall matrix-synapse-py3 and see if that helps.

I am considering installing a fresh ubuntu server and test if a clean install would fix it.
Is there a good way to backup messages and user settings to import them on a fresh in
stall?

I'd save a clean install as a last resort.

This should all be kept in the database that Synapse uses. If you're going to do a clean install, you must back this up. You probably also wants to backup the configuration files for Synapse itself. #synapse:matrix.org would probably be able to better advise.

ok weird, yeah sure.

/opt/venvs/matrix-synapse/bin/python --version returns: Python 3.8.10

/opt/venvs/matrix-synapse/bin/python -c "import importlib.metadata as m; print(repr(m.version('idna')))" returns: None

/opt/venvs/matrix-synapse/bin/python -c "import importlib.metadata as m; print(repr(m.metadata('idna').keys()))" returns: []

running /opt/venvs/matrix-synapse/bin/pip uninstall idna && /opt/venvs/matrix-synapse/bin/pip install idna==3.3 returns this after the uninstalling part:

Collecting idna==3.3
  Using cached idna-3.3-py3-none-any.whl (61 kB)
WARNING: Error parsing requirements for idna: [Errno 2] No such file or directory: '/opt/venvs/matrix-synapse/lib/python3.8/site-packages/idna-3.2.dist-info/METADATA'
Installing collected packages: idna
  Attempting uninstall: idna
    WARNING: No metadata found in /opt/venvs/matrix-synapse/lib/python3.8/site-packages
    Found existing installation: idna 3.2
    WARNING: No metadata found in /opt/venvs/matrix-synapse/lib/python3.8/site-packages
    WARNING: No metadata found in /opt/venvs/matrix-synapse/lib/python3.8/site-packages
    WARNING: No metadata found in /opt/venvs/matrix-synapse/lib/python3.8/site-packages
    WARNING: No metadata found in /opt/venvs/matrix-synapse/lib/python3.8/site-packages
    WARNING: No metadata found in /opt/venvs/matrix-synapse/lib/python3.8/site-packages
ERROR: Cannot uninstall idna 3.2, RECORD file not found. You might be able to recover from this via: 'pip install --force-reinstall --no-deps idna==3.2'.

I tried pip install --force-reinstall --no-deps idna==3.2 but this doesnt fix the issue.

I tried apt reinstall matrix-synapse-py3 yesterday but unfortunately hasnt helped.

Yeah i also would love to see this instance working again.

@reivilibre
Copy link
Contributor

I tried pip install --force-reinstall --no-deps idna==3.2 but this doesnt fix the issue.

Was this the full form?: /opt/venvs/matrix-synapse/bin/pip install --force-reinstall --no-deps idna==3.2

@DMRobertson
Copy link
Contributor

@reivilibre makes a good point --- see if that gets you anywhere.

Failing that, my only other suggestion would be to delete the leftover metadata directory, then try to start Synapse again.

  • rm -r /opt/venvs/matrix-synapse/lib/python3.8/site-packages/idna-3.2.dist-info

@Videothek
Copy link
Author

Videothek commented Mar 16, 2022

I tried pip install --force-reinstall --no-deps idna==3.2 but this doesnt fix the issue.

Was this the full form?: /opt/venvs/matrix-synapse/bin/pip install --force-reinstall --no-deps idna==3.2

No it was without the full path, i will try that later.

@Videothek
Copy link
Author

@reivilibre makes a good point --- see if that gets you anywhere.

Failing that, my only other suggestion would be to delete the leftover metadata directory, then try to start Synapse again.

  • rm -r /opt/venvs/matrix-synapse/lib/python3.8/site-packages/idna-3.2.dist-info

True, i will try that later i am currently not home but should be in one hour, than i will test this.

@Videothek
Copy link
Author

@DMRobertson @reivilibre Ok so after running rm -r /opt/venvs/matrix-synapse/lib/python3.8/site-packages/idna-3.2.dist-info i was able to install it with /opt/venvs/matrix-synapse/bin/pip install --force-reinstall --no-deps idna==3.2..

After trying to start matrix it throws py errors again.

So what is the next step?

@DMRobertson
Copy link
Contributor

After trying to start matrix it throws py errors again.

Can we see these errors please?

@Videothek
Copy link
Author

Videothek commented Mar 17, 2022

After trying to start matrix it throws py errors again.

Can we see these errors please?

@DMRobertson sure, sorry.

-- The job identifier is 3171338.
Mar 17 08:28:00 matrix matrix-synapse[171474]: Traceback (most recent call last):
Mar 17 08:28:00 matrix matrix-synapse[171474]:   File "/usr/lib/python3.8/runpy.py", line 185, in _run_module_as_main
Mar 17 08:28:00 matrix matrix-synapse[171474]:     mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
Mar 17 08:28:00 matrix matrix-synapse[171474]:   File "/usr/lib/python3.8/runpy.py", line 111, in _get_module_details
Mar 17 08:28:00 matrix matrix-synapse[171474]:     __import__(pkg_name)
Mar 17 08:28:00 matrix matrix-synapse[171474]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/app/__init__.py", line 23, in <module>
Mar 17 08:28:00 matrix matrix-synapse[171474]:     check_dependencies.check_requirements()
Mar 17 08:28:00 matrix matrix-synapse[171474]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/util/check_dependencies.py", line 167, in check_requirements
Mar 17 08:28:00 matrix matrix-synapse[171474]:     if not requirement.specifier.contains(dist.version, prereleases=True):
Mar 17 08:28:00 matrix matrix-synapse[171474]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/packaging/specifiers.py", line 728, in contains
Mar 17 08:28:00 matrix matrix-synapse[171474]:     item = parse(item)
Mar 17 08:28:00 matrix matrix-synapse[171474]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/packaging/version.py", line 49, in parse
Mar 17 08:28:00 matrix matrix-synapse[171474]:     return Version(version)
Mar 17 08:28:00 matrix matrix-synapse[171474]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/packaging/version.py", line 264, in __init__
Mar 17 08:28:00 matrix matrix-synapse[171474]:     match = self._regex.search(version)
Mar 17 08:28:00 matrix matrix-synapse[171474]: TypeError: expected string or bytes-like object
Mar 17 08:28:00 matrix systemd[1]: matrix-synapse.service: Control process exited, code=exited, status=1/FAILURE

@DMRobertson
Copy link
Contributor

Thanks. Can you please try:

@Videothek
Copy link
Author

Videothek commented Mar 17, 2022

Thanks. Can you please try:

* /opt/venvs/matrix-synapse/bin/pip uninstall idna && /opt/venvs/matrix-synapse/bin/pip install idna==3.3

* Rerun the curl command from here: [matrix-synapse-py3 homeserver breaks on ubuntu 20.04 after update to 1.54 #12223 (comment)](https://github.com/matrix-org/synapse/issues/12223#issuecomment-1069282184)

* rerun synapse

@DMRobertson i did it in the same order as you said.

The error output is as follows:

Mar 17 13:53:41 matrix matrix-synapse[194785]: DEP CHECK: jsonschema>=3.0.0, True, <importlib.metadata.PathDistribution object at 0x7f75f53e53a0>, '4.0.1'
Mar 17 13:53:41 matrix matrix-synapse[194785]: DEP CHECK: frozendict>=1, True, <importlib.metadata.PathDistribution object at 0x7f75f53e51f0>, '2.3.0'
Mar 17 13:53:41 matrix matrix-synapse[194785]: DEP CHECK: unpaddedbase64>=1.1.0, True, <importlib.metadata.PathDistribution object at 0x7f75f53e5640>, '2.1.0'
Mar 17 13:53:41 matrix matrix-synapse[194785]: DEP CHECK: canonicaljson>=1.4.0, True, <importlib.metadata.PathDistribution object at 0x7f75f53e5850>, '1.6.0'
Mar 17 13:53:41 matrix matrix-synapse[194785]: DEP CHECK: signedjson>=1.1.0, True, <importlib.metadata.PathDistribution object at 0x7f75f53e5ac0>, '1.1.1'
Mar 17 13:53:41 matrix matrix-synapse[194785]: DEP CHECK: pynacl>=1.2.1, True, <importlib.metadata.PathDistribution object at 0x7f75f53e5cd0>, '1.5.0'
Mar 17 13:53:41 matrix matrix-synapse[194785]: DEP CHECK: idna>=2.5, True, <importlib.metadata.PathDistribution object at 0x7f75f53e5f10>, '3.3'
Mar 17 13:53:41 matrix matrix-synapse[194785]: DEP CHECK: service-identity>=18.1.0, True, <importlib.metadata.PathDistribution object at 0x7f75f53fb1c0>, '21.1.0'
Mar 17 13:53:41 matrix matrix-synapse[194785]: DEP CHECK: Twisted>=18.9.0, True, <importlib.metadata.PathDistribution object at 0x7f75f53fb430>, '22.2.0'
Mar 17 13:53:41 matrix matrix-synapse[194785]: DEP CHECK: treq>=15.1, True, <importlib.metadata.PathDistribution object at 0x7f75f53e5fd0>, '22.2.0'
Mar 17 13:53:41 matrix matrix-synapse[194785]: DEP CHECK: pyopenssl>=16.0.0, True, <importlib.metadata.PathDistribution object at 0x7f75f53e5bb0>, '22.0.0'
Mar 17 13:53:41 matrix matrix-synapse[194785]: DEP CHECK: pyyaml>=3.11, True, <importlib.metadata.PathDistribution object at 0x7f75f53e5940>, '6.0'
Mar 17 13:53:41 matrix matrix-synapse[194785]: DEP CHECK: pyasn1>=0.1.9, True, <importlib.metadata.PathDistribution object at 0x7f75f53e5700>, '0.4.8'
Mar 17 13:53:41 matrix matrix-synapse[194785]: DEP CHECK: pyasn1-modules>=0.0.7, True, <importlib.metadata.PathDistribution object at 0x7f75f53e5610>, '0.2.8'
Mar 17 13:53:41 matrix matrix-synapse[194785]: DEP CHECK: bcrypt>=3.1.0, True, <importlib.metadata.PathDistribution object at 0x7f75f53fb070>, '3.2.0'
Mar 17 13:53:41 matrix matrix-synapse[194785]: DEP CHECK: pillow>=4.3.0, True, <importlib.metadata.PathDistribution object at 0x7f75f53fb190>, None
Mar 17 13:53:41 matrix matrix-synapse[194785]: Traceback (most recent call last):
Mar 17 13:53:41 matrix matrix-synapse[194785]:   File "/usr/lib/python3.8/runpy.py", line 185, in _run_module_as_main
Mar 17 13:53:41 matrix matrix-synapse[194785]:     mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
Mar 17 13:53:41 matrix matrix-synapse[194785]:   File "/usr/lib/python3.8/runpy.py", line 111, in _get_module_details
Mar 17 13:53:41 matrix matrix-synapse[194785]:     __import__(pkg_name)
Mar 17 13:53:41 matrix matrix-synapse[194785]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/app/__init__.py", line 23, in <module>
Mar 17 13:53:41 matrix matrix-synapse[194785]:     check_dependencies.check_requirements()
Mar 17 13:53:41 matrix matrix-synapse[194785]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/util/check_dependencies.py", line 172, in check_requirements
Mar 17 13:53:41 matrix matrix-synapse[194785]:     if not requirement.specifier.contains(dist.version, prereleases=True):
Mar 17 13:53:41 matrix matrix-synapse[194785]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/packaging/specifiers.py", line 728, in contains
Mar 17 13:53:41 matrix matrix-synapse[194785]:     item = parse(item)
Mar 17 13:53:41 matrix matrix-synapse[194785]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/packaging/version.py", line 49, in parse
Mar 17 13:53:41 matrix matrix-synapse[194785]:     return Version(version)
Mar 17 13:53:41 matrix matrix-synapse[194785]:   File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/packaging/version.py", line 264, in __init__
Mar 17 13:53:41 matrix matrix-synapse[194785]:     match = self._regex.search(version)
Mar 17 13:53:41 matrix matrix-synapse[194785]: TypeError: expected string or bytes-like object
Mar 17 13:53:41 matrix systemd[1]: matrix-synapse.service: Control process exited, code=exited, status=1/FAILURE

@DMRobertson
Copy link
Contributor

How odd. Something has also gone wrong with the pillow package. Let's try the same fix, but for the entire synapse virtualenv:

@Videothek
Copy link
Author

@DMRobertson it seems like you hit the nail on the head, thank you.

It works now after doing your steps.

  1. rm -r /opt/venvs/matrix-synapse/
  2. apt reinstall matrix-synapse-py3
  3. curl https://raw.githubusercontent.com/matrix-org/synapse/4c988172646877bbafa8284e0faa4deb35a1c7d1/synapse/util/check_dependencies.py -o /opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/util/check_dependencies.py
  4. systemctl restart matrix-synapse

Could you maybe summerize what the problem was and if there is any solutions without running your file.

(no offense but as you said running files you dont have any clue about with private messages on risk i dont feel very well)

@DMRobertson
Copy link
Contributor

Thanks for going through all of this. I'm glad to hear it's working now!

Could you maybe summerize what the problem was and if there is any solutions without running your file.

Certainly. Synapse contains code which will check that any packages it needs requires installed and running the right version. I recently changed this in the Synapse 1.54 release: it now uses a different mechanism (importlib.metadata instead of pkg_resources). The new way of doing things was confused by your Synapse installation. It seemed to have incomplete "leftover" old versions of packages (idna, pillow, possibly others) remaining in its virtual environment. I'm not sure how the these leftovers came to exist---maybe an incorrect restoration from backup, or a cancelled apt run? But either way, they were causing pain.

This isn't something you should normally see during an installation; I'm not sure why this happened and you shouldn't expect to have to do this in the future.

In any case, the fix was: delete the virtual environment (1), then reinstall a clean virtual environment (2). Step (3) overwrites the checking code with a version that includes extra debug information. I asked you to do this in case things still were broken. You can undo this change by running apt reinstall matrix-synapse.

Does that make sense?

In the meantime I'll change this checking code to detect this situation and provide a more useful error message.

@DMRobertson DMRobertson changed the title matrix-synapse-py3 homeserver breaks on ubuntu 20.04 after update to 1.54 Unhelpful error message when importlib.metadata.version returns None (which it shouldn't do normally!) Mar 17, 2022
@DMRobertson DMRobertson self-assigned this Mar 17, 2022
@richvdh
Copy link
Member

richvdh commented Mar 22, 2022

You can undo this change by running apt reinstall matrix-synapse.

minor correction: apt reinstall matrix-synapse-py3 (because #5215)

@DMRobertson
Copy link
Contributor

For completeness: I made noise about the surprising(?) behaviour of importlib.metadata.version here, here and here. The importlib_metadata maintainer is pondering how best to treat the issue.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
X-Needs-Info This issue is blocked awaiting information from the reporter
Projects
None yet
Development

No branches or pull requests

7 participants