I cannot send encrypted messages to matrix.org, but I can receive them?!?

[Dominion0815]

Description

On my home server, I can send and receive encrypted messages to and from different servers without any problems.

But I cannot send messages to matrix.org, receiving them works fine.
There is also no write notification when sending.

I also switched the Docker Synapse Worker setup back to Monolith for testing, but this did not bring any improvement.

I cannot find anything in the logs that directly indicates an encryption problem.

Is there anything else I can do to provide usable logs?

Steps to reproduce

I have no idea, from my other home server I can send and receive encrypted messages to matrix.org without any problems.

Homeserver

nope.chat

Synapse Version

1.104.0

Installation Method

Docker (matrixdotorg/synapse)

Database

PostgreSQL 14

Workers

Multiple workers

Platform

Debian 12.5
Docker Engine - Community Version: 26.0.1
Docker Compose version v2.26.1

Configuration

No response

Relevant log output

federation_transaction_transmission_loop-57 - TX [matrix.org] Failed to send transaction: Failed to send request: HttpResponseException: 520:

federation-reader-1-1  | 2024-04-14 13:31:27,645 - synapse.handlers.federation_event - 1621 - WARNING - _process_incoming_pdus_in_room_inner-204-$vFLgCEA_PHgyeB0Ena99jA2Ko2N2WtVKaN4f3RnTgTc-$IN2XvsDIUNwbE83Fd1oR6LWIjkh8_wrRppDcRMipeCk-$KPepzUH_IZuJcbnK3vkHmgixAundmJb4z80hUNGzhDQ-$2Lbm1lJsspYohzd6bVsBOm0BqMNgqvg7tmvuEbx1cWg - Server matrix.org didn't return event $2Lbm1lJsspYohzd6bVsBOm0BqMNgqvg7tmvuEbx1cWg

Anything else that would be useful to know?

No response

[Dominion0815]

One more thing, the clients receive this message after a while:

{
	"type": "m.room.message",
	"content" {
		"type": "m.bad.encrypted",
		"body": "** Unable to decrypt: DecryptionError: The sender's device has not sent us the keys for this message. **"
	}
}

[AlexPewMaster]

Some other homeservers are also experiencing the same issue. I've seen people complain about it in other public rooms.

[Dominion0815]

Problem still exist ... can anyone help?

[reivilibre]

• Is this all rooms or just one?
• Do you have any logs showing requests made to matrix.org? Error 520 sounds a bit suspicious; do you have any more extensive and recent logs?

Logs like this one from your snippet sound useful; if you have any recent ones please share:
federation_transaction_transmission_loop-57 - TX [matrix.org] Failed to send transaction: Failed to send request: HttpResponseException: 520:

Try to include all the log lines from the log message (I don't know if you are filtering it out), it'd be really useful to see the full URL requested from Matrix.org

Probably your server can't federate directly with Matrix.org but can with other servers; then other servers are relaying the encrypted message to Matrix.org but since you never federate directly, you can't send the required to-device messages for Matrix.org users to decrypt the messages. Does this sound plausible — are there users from other servers in the same rooms?

[Dominion0815]

The problem has only existed since the beginning of April. Before that, I was able to write to matrix.org people without any problems.

Since April it no longer works, regardless of whether it is in encrypted groups or 1-to-1 chats.

Unfortunately I don't have any more logs, when I grep matrix.org only these few lines come up.
I have over 1200 MAUs, so there are too many log files jumbled together.

[reivilibre]

With a busy log like that, usually the best approach is to first make a search based on what you're interested in, then grep for the request identifier, to get all the log lines for the related request.

See https://element-hq.github.io/synapse/latest/usage/administration/request_log.html

In the example from your first post federation_transaction_transmission_loop-57

[Dominion0815]

I find a lot of these errors now in the log.
The problem also occurs when I disable the event-persister. I think then the errors came to the Synapse Master.

What else can I do? What could be the reason?

event-persister-1      | KeyError: '$1676357201147842ZKMjV:matrix.org'
event-persister-1      | 2024-05-31 08:37:46,636 - synapse.http.server - 146 - ERROR - POST-255334 - Failed handle request via 'ReplicationFederationSendEventsRestServlet': <SynapseRequest at 0x7f77a203d950 method='POST' uri='/_synapse/replication/fed_send_events/BXWjfcqyPF' clientproto='HTTP/1.1' site='8034'>
--
event-persister-1      | KeyError: '$1676357201147842ZKMjV:matrix.org'
event-persister-1      | 2024-05-31 08:37:54,073 - synapse.http.server - 146 - ERROR - POST-255346 - Failed handle request via 'ReplicationFederationSendEventsRestServlet': <SynapseRequest at 0x7f77a0248a10 method='POST' uri='/_synapse/replication/fed_send_events/cBLpHBmRpz' clientproto='HTTP/1.1' site='8034'>
--
event-persister-1      | KeyError: '$1676357201147842ZKMjV:matrix.org'
event-persister-1      | 2024-05-31 08:38:00,954 - synapse.http.server - 146 - ERROR - POST-255356 - Failed handle request via 'ReplicationFederationSendEventsRestServlet': <SynapseRequest at 0x7f77a69e0c50 method='POST' uri='/_synapse/replication/fed_send_events/cIKfxKBpRF' clientproto='HTTP/1.1' site='8034'>
--
event-persister-1      | KeyError: '$1676357201147842ZKMjV:matrix.org'
event-persister-1      | 2024-05-31 08:38:07,439 - synapse.http.server - 146 - ERROR - POST-255368 - Failed handle request via 'ReplicationFederationSendEventsRestServlet': <SynapseRequest at 0x7f77b0959950 method='POST' uri='/_synapse/replication/fed_send_events/IwZbfzpTOS' clientproto='HTTP/1.1' site='8034'>
--
event-persister-1      | KeyError: '$1676357201147842ZKMjV:matrix.org'
event-persister-1      | 2024-05-31 08:38:24,367 - synapse.http.server - 146 - ERROR - POST-255388 - Failed handle request via 'ReplicationFederationSendEventsRestServlet': <SynapseRequest at 0x7f77764e2790 method='POST' uri='/_synapse/replication/fed_send_events/XLmTqKDJXo' clientproto='HTTP/1.1' site='8034'>
--
event-persister-1      | KeyError: '$1676357201147842ZKMjV:matrix.org'
event-persister-1      | 2024-05-31 08:41:49,881 - synapse.http.server - 146 - ERROR - POST-255553 - Failed handle request via 'ReplicationFederationSendEventsRestServlet': <SynapseRequest at 0x7f77aa02cb10 method='POST' uri='/_synapse/replication/fed_send_events/JXCdjAOcZA' clientproto='HTTP/1.1' site='8034'>
--
event-persister-1      | KeyError: '$169114881771282OrWup:matrix.org'
event-persister-1      | 2024-05-31 08:43:11,479 - synapse.http.server - 146 - ERROR - POST-255633 - Failed handle request via 'ReplicationFederationSendEventsRestServlet': <SynapseRequest at 0x7f77aa4ffdd0 method='POST' uri='/_synapse/replication/fed_send_events/hJMTpydOGt' clientproto='HTTP/1.1' site='8034'>
--
event-persister-1      | KeyError: '$1676357201147842ZKMjV:matrix.org'
event-persister-1      | 2024-05-31 08:43:55,032 - synapse.http.server - 146 - ERROR - POST-255677 - Failed handle request via 'ReplicationFederationSendEventsRestServlet': <SynapseRequest at 0x7f77c69bf4d0 method='POST' uri='/_synapse/replication/fed_send_events/HOvqIuvkxe' clientproto='HTTP/1.1' site='8034'>
--
event-persister-1      | KeyError: '$1679950803144130oYdON:matrix.org'
event-persister-1      | 2024-05-31 08:52:21,123 - synapse.http.server - 146 - ERROR - POST-256139 - Failed handle request via 'ReplicationFederationSendEventsRestServlet': <SynapseRequest at 0x7f77acfd2d90 method='POST' uri='/_synapse/replication/fed_send_events/QfekQXpsaH' clientproto='HTTP/1.1' site='8034'>
--
event-persister-1      | KeyError: '$1679950803144130oYdON:matrix.org'
event-persister-1      | 2024-05-31 09:00:21,120 - synapse.http.server - 146 - ERROR - POST-257063 - Failed handle request via 'ReplicationFederationSendEventsRestServlet': <SynapseRequest at 0x7f779a1e6410 method='POST' uri='/_synapse/replication/fed_send_events/GhZYrnsNTj' clientproto='HTTP/1.1' site='8034'>
--
event-persister-1      | KeyError: '$1679950803144130oYdON:matrix.org'
event-persister-1      | 2024-05-31 09:00:28,501 - synapse.http.server - 146 - ERROR - POST-257073 - Failed handle request via 'ReplicationFederationSendEventsRestServlet': <SynapseRequest at 0x7f77a0305450 method='POST' uri='/_synapse/replication/fed_send_events/BuhhmbnQYt' clientproto='HTTP/1.1' site='8034'>
--
event-persister-1      | KeyError: '$1679950803144130oYdON:matrix.org'
event-persister-1      | 2024-05-31 09:07:59,216 - synapse.http.server - 146 - ERROR - POST-257538 - Failed handle request via 'ReplicationFederationSendEventsRestServlet': <SynapseRequest at 0x7f77a2a80a50 method='POST' uri='/_synapse/replication/fed_send_events/IkQxMLCGBE' clientproto='HTTP/1.1' site='8034'>
--
event-persister-1      | KeyError: '$1679950803144130oYdON:matrix.org'
event-persister-1      | 2024-05-31 09:08:39,972 - synapse.http.server - 146 - ERROR - POST-257595 - Failed handle request via 'ReplicationFederationSendEventsRestServlet': <SynapseRequest at 0x7f779fded290 method='POST' uri='/_synapse/replication/fed_send_events/GyyfRpJwuZ' clientproto='HTTP/1.1' site='8034'>

[reivilibre]

I think we need more of the surrounding log lines.

But if anything an error in /fed_send_events means you'll struggle to receive messages rather than send them.

Can you dig up some log lines that fail when you are sending to matrix.org?

federation_transaction_transmission_loop-57 - TX [matrix.org] Failed to send transaction: Failed to send request: HttpResponseException: 520: is the kind of thing that is on the right lines — it'll be in your federation sender or master if you don't have one.

something like grep -E 'federation_transaction_transmission_loop.*matrix.org.*Failed' should help find the first line, then
get the related log lines by grepping for the identifier federation_transaction_transmission_loop-57 (57 might be different in your case)

[Dominion0815]

Unfortunately, I can't find anything in the log.

But when I restart Synapse, I get the missing messages in my matrix.org account, but it says “encrypted” instead of a text.
After the restart, the log says the following about matrix.org:

federation-reader-1-1  | 2024-06-03 13:02:01,438 - synapse.handlers.federation_event - 1621 - WARNING - _process_incoming_pdus_in_room_inner-17-$_zm_tGM3TkuO-7dsFKzzILWaP_SURce2gb6V2Yp0Gc4-$fhNp4Bn0myMZ_qUYUqE29KqOLn9KjZSxK0F9vDnQZzg-$xeDolob5OUQUNdaZEveUJuysssHfbg-Gb184ggXoEeY-$rE8XPez4s8i8UOK5VwKgZTQNFK258sK80ZVneXDLdks - Server matrix.org didn't return event $rE8XPez4s8i8UOK5VwKgZTQNFK258sK80ZVneXDLdks
federation-reader-1-1  | 2024-06-03 13:02:54,293 - synapse.federation.federation_server - 1281 - ERROR - _process_incoming_pdus_in_room_inner-41 - Failed to handle PDU $171741253820441fZoKG:matrix.org
event-persister-1      | KeyError: '$169114881771282OrWup:matrix.org'
federation-reader-1-1  | Exception: Error fetching missing prev_events for $171741253820441fZoKG:matrix.org: 500: Internal server error
event-persister-1      | KeyError: '$1676357201147842ZKMjV:matrix.org'
federation-reader-1-1  | 2024-06-03 13:07:28,398 - synapse.federation.federation_server - 1281 - ERROR - _process_incoming_pdus_in_room_inner-190 - Failed to handle PDU $171741283919406oXfBR:matrix.org
federation-reader-1-1  | Exception: Error fetching missing prev_events for $171741283919406oXfBR:matrix.org: 500: Internal server error
federation-reader-1-1  | 2024-06-03 13:08:21,898 - synapse.handlers.federation_event - 1621 - WARNING - _process_incoming_pdus_in_room_inner-182-$yIpIC33yRmWmtIclGDQorz2-mtUspsCzlgS2lIcosDc-$P2mZiAfcQHe0Mr2ugx-RjjKrhpkjlFznatWaOeeEinI-$XaxYWrDLhYKorzAZPn6wC0c7p7tns9fYFZ9KtE_wo00-$CBs2q9KJtPwPzBuQ02UxWLqly0o8ip7H5o6dTYc0HLM - Server matrix.org didn't return event $CBs2q9KJtPwPzBuQ02UxWLqly0o8ip7H5o6dTYc0HLM
federation-reader-1-1  | 2024-06-03 13:08:22,200 - synapse.handlers.federation_event - 1621 - WARNING - _process_incoming_pdus_in_room_inner-182-$yIpIC33yRmWmtIclGDQorz2-mtUspsCzlgS2lIcosDc-$P2mZiAfcQHe0Mr2ugx-RjjKrhpkjlFznatWaOeeEinI-$XaxYWrDLhYKorzAZPn6wC0c7p7tns9fYFZ9KtE_wo00-$2MfimG5f6cC9ITz3DNc32YP9krexb7YoTAbOkPqXyqo - Server matrix.org didn't return event $2MfimG5f6cC9ITz3DNc32YP9krexb7YoTAbOkPqXyqo
federation-reader-1-1  | 2024-06-03 13:08:22,265 - synapse.handlers.federation_event - 1621 - WARNING - _process_incoming_pdus_in_room_inner-182-$yIpIC33yRmWmtIclGDQorz2-mtUspsCzlgS2lIcosDc-$P2mZiAfcQHe0Mr2ugx-RjjKrhpkjlFznatWaOeeEinI-$XaxYWrDLhYKorzAZPn6wC0c7p7tns9fYFZ9KtE_wo00-$ipRSU8yt-3CZ3Koeekvq33MFcbvlshR8P1MZAGY4GhI - Server matrix.org didn't return event $ipRSU8yt-3CZ3Koeekvq33MFcbvlshR8P1MZAGY4GhI
federation-reader-1-1  | 2024-06-03 13:08:31,757 - synapse.handlers.federation_event - 1621 - WARNING - _process_incoming_pdus_in_room_inner-182-$yIpIC33yRmWmtIclGDQorz2-mtUspsCzlgS2lIcosDc-$P2mZiAfcQHe0Mr2ugx-RjjKrhpkjlFznatWaOeeEinI-$XaxYWrDLhYKorzAZPn6wC0c7p7tns9fYFZ9KtE_wo00-$rmslrJf0U0zu3aE26TEeb-NBLDM6uCnzNZ0t-jvyLAQ - Server matrix.org didn't return event $rmslrJf0U0zu3aE26TEeb-NBLDM6uCnzNZ0t-jvyLAQ
federation-reader-1-1  | 2024-06-03 13:08:31,804 - synapse.handlers.federation_event - 1621 - WARNING - _process_incoming_pdus_in_room_inner-182-$yIpIC33yRmWmtIclGDQorz2-mtUspsCzlgS2lIcosDc-$P2mZiAfcQHe0Mr2ugx-RjjKrhpkjlFznatWaOeeEinI-$XaxYWrDLhYKorzAZPn6wC0c7p7tns9fYFZ9KtE_wo00-$EKdD3DyR8j00JD-gfXJqcHaLec7kpbKEIr65cXjJ4a0 - Server matrix.org didn't return event $EKdD3DyR8j00JD-gfXJqcHaLec7kpbKEIr65cXjJ4a0
federation-reader-1-1  | 2024-06-03 13:08:31,959 - synapse.handlers.federation_event - 1621 - WARNING - _process_incoming_pdus_in_room_inner-182-$yIpIC33yRmWmtIclGDQorz2-mtUspsCzlgS2lIcosDc-$P2mZiAfcQHe0Mr2ugx-RjjKrhpkjlFznatWaOeeEinI-$XaxYWrDLhYKorzAZPn6wC0c7p7tns9fYFZ9KtE_wo00-$TbYSHMmWeYcT5A5n4wx7eJPqSCkJBM6BfMZ2Prz3VkE - Server matrix.org didn't return event $TbYSHMmWeYcT5A5n4wx7eJPqSCkJBM6BfMZ2Prz3VkE