federation issues and "This server is using an older version of Matrix. Upgrade to Matrix v1.5 to use Element without errors."

[r3k2]

Updated to last stable git version of dendrite mid december
version: v0. 13.15 https://github.com/matrix-org/dendrite/releases/tag/v0.13.5

People is reporting they can't see msgs from matrix.org or even other homeservers (this was working last month just fine )
since all the report from my users, at least 6 of them for the last 2 weeks I decided to log using element ( I use iamb and gomuks )
and the first weird thing I notice is :

This server is using an older version of Matrix. Upgrade to Matrix v1.5 to use Element without errors.

I have checked postgresql I see no errors
I need to enable debug for dendrite since the regular stdout I see no errors neither
posting this now so I do not forget, since I work a lot and always push things, and then I end up forgetting and my users keep asking me to fix this.

• The server does use ipv6 and ipv4
• latest git stable version https://github.com/matrix-org/dendrite/releases/tag/v0.13.5
• everyone in our homeserver can talk to each other just fine, sometimes I see msg's from other homeservers but for example in the #rust channel that is super active the last one I got was from Sunday, I find that very weird...

Hopefully this is a known issue and has a fast fix/workaround.
let me know what other info people will need when I enable this weekend debug ( I have a lot of work now ) I will post the results as well.
is possible we are block? how can we check that?

[edit] msgs work from our homeserver to matrix.org is the other way around, I had family members write to me and friends on matrix.org and I never got their msg's since Sunday. I saw them because I used my matrix.org test account and can see matrix.hispagatos.org msgs on matrix.org but not the other way around :(
[edit2] this is what I get to stdout

[root@matrix ~]# journalctl -fu dendrite
Jan 16 19:41:05 matrix systemd[1]: Started Dendrite (Matrix Homeserver).
Jan 16 20:38:45 matrix dendrite[437]: 2024/01/16 20:38:45 Transport: unhandled response frame type *http.http2UnknownFrame

thats about it

[idanoo]

Also got this opening Element after upgrading client today.

[ekbaker7]

Having the same issue using the docker version of dendrite, and taking the image down and restarting to get latest actually made my instance completely unreachable by Element.

[Mic92]

How can I check what Matrix version a server supports?

Update:

Is it this one?

curl -v https://matrix.yourdomain.com/_matrix/client/versions | jq
{
  "versions": [
    "r0.0.1",
    "r0.1.0",
    "r0.2.0",
    "r0.3.0",
    "r0.4.0",
    "r0.5.0",
    "r0.6.1",
    "v1.0",
    "v1.1",
    "v1.2"
  ],
  "unstable_features": {
    "org.matrix.e2e_cross_signing": true,
    "org.matrix.msc2285.stable": true,
    "org.matrix.msc2836": true,
    "org.matrix.msc2946": true
  }
}

Guess we have to pin now to older Element versions...

[Mic92]

This got introduced in

Element version: 1.11.54
Crypto version: Olm 3.2.15

Element 1.11.53 is still fine.

[tommie]

How can I check what Matrix version a server supports?

And the Dendrite source agrees:

dendrite/clientapi/routing/routing.go

Lines 140 to 149 in dae1ef2

	"r0.0.1",
	"r0.1.0",
	"r0.2.0",
	"r0.3.0",
	"r0.4.0",
	"r0.5.0",
	"r0.6.1",
	"v1.0",
	"v1.1",
	"v1.2",

Only up to v1.2 announced.

[S7evinK]

The Matrix 1.5 thingy was a change in the matrix-js-sdk, and unfortunately totally went past us and now we are here.

As for the federation issue, are you able to update to latest main? This would contain #3288 which likely fixes a few issues with incoming events.

[tommie]

Related context: conduit-rs announces up to v1.4, so that'll also trigger the warning: https://gitlab.com/famedly/conduit/-/blob/next/src/api/client_server/unversioned.rs?ref_type=heads#L28

Suggesting Synapse is the only(?) server that is not causing warnings in Element right now.

[r3k2]

@S7evinK I just updated to main, do I need to delete jetstream or something? Maybe is trying to work on all the late msg's and getting stuck?
[edit] I just turned on info log and I keep getting spam with this:

time="2024-01-17T15:51:28.900171071Z" level=error msg="Failed to manually update device lists for user" func="github.com/matrix-org/dendrite/userapi/internal.(*UserInternalAPI).queryRemoteKeysOnServer" file="/home/dendrite/dendrite/userapi/internal/key_api.go:535" error="DeviceKeysForUser @josef:netzgemeinde.eu returned no keys but wanted all keys, falling back to remote" server=netzgemeinde.eu user_id="@josef:netzgemeinde.eu"
time="2024-01-17T15:51:28.900206962Z" level=error msg="Failed to manually update device lists for user" func="github.com/matrix-org/dendrite/userapi/internal.(*UserInternalAPI).queryRemoteKeysOnServer" file="/home/dendrite/dendrite/userapi/internal/key_api.go:535" error="DeviceKeysForUser @ghost_of_nikiforova:privacytools.io returned no keys but wanted all keys, falling back to remote" server=privacytools.io user_id="@ghost_of_nikiforova:privacytools.io"
time="2024-01-17T15:51:39.007150042Z" level=error msg="Failed to manually update device lists for user" func="github.com/matrix-org/dendrite/userapi/internal.(*UserInternalAPI).queryRemoteKeysOnServer" file="/home/dendrite/dendrite/userapi/internal/key_api.go:535" error="DeviceKeysForUser @karl:matrix.kiwifarms.net returned no keys but wanted all keys, falling back to remote" server=matrix.kiwifarms.net user_id="@karl:matrix.kiwifarms.net"
time="2024-01-17T15:51:39.007185314Z" level=error msg="Failed to manually update device lists for user" func="github.com/matrix-org/dendrite/userapi/internal.(*UserInternalAPI).queryRemoteKeysOnServer" file="/home/dendrite/dendrite/userapi/internal/key_api.go:535" error="DeviceKeysForUser @gbenz:madrot.com returned no keys but wanted all keys, falling back to remote" server=madrot.com user_id="@gbenz:madrot.com"
time="2024-01-17T15:51:39.007382986Z" level=error msg="Failed to manually update device lists for user" func="github.com/matrix-org/dendrite/userapi/internal.(*UserInternalAPI).queryRemoteKeysOnServer" file="/home/dendrite/dendrite/userapi/internal/key_api.go:535" error="DeviceKeysForUser @ghost_of_nikiforova:privacytools.io returned no keys but wanted all keys, falling back to remote" server=privacytools.io user_id="@ghost_of_nikiforova:privacytools.io"
time="2024-01-17T15:51:39.007663699Z" level=error msg="Failed to manually update device lists for user" func="github.com/matrix-org/dendrite/userapi/internal.(*UserInternalAPI).queryRemoteKeysOnServer" file="/home/dendrite/dendrite/userapi/internal/key_api.go:535" error="DeviceKeysForUser @josef:netzgemeinde.eu returned no keys but wanted all keys, falling back to remote" server=netzgemeinde.eu user_id="@josef:netzgemeinde.eu"
time="2024-01-17T15:51:49.039960088Z" level=error msg="Failed to manually update device lists for user" func="github.com/matrix-org/dendrite/userapi/internal.(*UserInternalAPI).queryRemoteKeysOnServer" file="/home/dendrite/dendrite/userapi/internal/key_api.go:535" error="DeviceKeysForUser @karl:matrix.kiwifarms.net returned no keys but wanted all keys, falling back to remote" server=matrix.kiwifarms.net user_id="@karl:matrix.kiwifarms.net"
time="2024-01-17T15:51:49.040497894Z" level=error msg="Failed to manually update device lists for user" func="github.com/matrix-org/dendrite/userapi/internal.(*UserInternalAPI).queryRemoteKeysOnServer" file="/home/dendrite/dendrite/userapi/internal/key_api.go:535" error="DeviceKeysForUser @ghost_of_nikiforova:privacytools.io returned no keys but wanted all keys, falling back to remote" server=privacytools.io user_id="@ghost_of_nikiforova:privacytools.io"
time="2024-01-17T15:51:49.040563515Z" level=error msg="Failed to manually update device lists for user" func="github.com/matrix-org/dendrite/userapi/internal.(*UserInternalAPI).queryRemoteKeysOnServer" file="/home/dendrite/dendrite/userapi/internal/key_api.go:535" error="DeviceKeysForUser @josef:netzgemeinde.eu returned no keys but wanted all keys, falling back to remote" server=netzgemeinde.eu user_id="@josef:netzgemeinde.eu"
time="2024-01-17T15:51:49.040584404Z" level=error msg="Failed to manually update device lists for user" func="github.com/matrix-org/dendrite/userapi/internal.(*UserInternalAPI).queryRemoteKeysOnServer" file="/home/dendrite/dendrite/userapi/internal/key_api.go:535" error="DeviceKeysForUser @gbenz:madrot.com returned no keys but wanted all keys, falling back to remote" server=madrot.com user_id="@gbenz:madrot.com"

is not always the same user this is just a snippet.

Also this is interesting about the MSC goin to dissable it and see if it helps
also something about a jetstream payload..

● dendrite.service - Dendrite (Matrix Homeserver)
     Loaded: loaded (/etc/systemd/system/dendrite.service; enabled; preset: disabled)
     Active: active (running) since Wed 2024-01-17 15:50:53 UTC; 7s ago
   Main PID: 41491 (dendrite)
      Tasks: 11 (limit: 19176)
     Memory: 213.0M (peak: 213.9M)
        CPU: 2.151s
     CGroup: /system.slice/dendrite.service
             └─41491 /opt/dendrite/bin/dendrite

Jan 17 15:50:53 matrix systemd[1]: Started Dendrite (Matrix Homeserver).
Jan 17 15:50:53 matrix dendrite[41491]: time="2024-01-17T15:50:53.245551467Z" level=warning msg="Maximum payloads over 8.00 MB are generally discouraged and could lead to poor performance" component=jetstream
Jan 17 15:50:58 matrix dendrite[41491]: time="2024-01-17T15:50:58.628068176Z" level=warning msg="EnableMSC: unknown MSC 'msc2946', this MSC is either not supported or is natively supported by Dendrite"
Jan 17 15:50:58 matrix dendrite[41491]: time="2024-01-17T15:50:58.711555623Z" level=error msg="Failed to manually update device lists for user" error="DeviceKeysForUser @ghost_of_nikiforova:privacytools.io returned no keys but wanted all keys, falling back to remote" server=privacytools.io user_id="@ghost_of_nikifor>
Jan 17 15:50:58 matrix dendrite[41491]: time="2024-01-17T15:50:58.713383815Z" level=error msg="Failed to manually update device lists for user" error="DeviceKeysForUser @gbenz:madrot.com returned no keys but wanted all keys, falling back to remote" server=madrot.com user_id="@gbenz:madrot.com"
Jan 17 15:50:58 matrix dendrite[41491]: time="2024-01-17T15:50:58.715687890Z" level=error msg="Failed to manually update device lists for user" error="DeviceKeysForUser @josef:netzgemeinde.eu returned no keys but wanted all keys, falling back to remote" server=netzgemeinde.eu user_id="@josef:netzgemeinde.eu"

[edit2]
I enable debug and I keep getting spam with an error related to one of our users over and over non-stop always same user.

ime="2024-01-17T16:04:10.408006905Z" level=info msg=Responding func=github.com/matrix-org/dendrite/syncapi/routing.OnIncomingMessagesRequest file="/home/dendrite/dendrite/syncapi/routing/messages.go:289" backfilled=false backwards=true limit=100 req.id=bZrOyeJPs1CE req.method=GET req.path="/_matrix/client/r0/rooms/!CDTRSRRRFMQJADZO:hispagatos.org/messages" request_from=t3887_6436048 request_to="" response_end=t3787_6435572 response_start=t3887_6436048 user_id="@real-changeling:hispagatos.org"
time="2024-01-17T16:04:10.503519962Z" level=info msg="Fetched 1 events locally" func="github.com/matrix-org/dendrite/syncapi/routing.(*messagesReq).retrieveEvents" file="/home/dendrite/dendrite/syncapi/routing/messages.go:361" backwards=true end="" req.id=aRUrnPCk6DYK req.method=GET req.path="/_matrix/client/r0/rooms/!r1qC3DLPDkZOXURu:hispagatos.org/messages" start=t1_8023652 user_id="@real-changeling:hispagatos.org"
time="2024-01-17T16:04:10.504066148Z" level=info msg=Responding func=github.com/matrix-org/dendrite/syncapi/routing.OnIncomingMessagesRequest file="/home/dendrite/dendrite/syncapi/routing/messages.go:289" backfilled=false backwards=true limit=100 req.id=aRUrnPCk6DYK req.method=GET req.path="/_matrix/client/r0/rooms/!r1qC3DLPDkZOXURu:hispagatos.org/messages" request_from=t1_8023652 request_to="" response_end="" response_start=t1_8023652 user_id="@real-changeling:hispagatos.org"
time="2024-01-17T16:04:10.868949251Z" level=info msg="Fetched 100 events locally" func="github.com/matrix-org/dendrite/syncapi/routing.(*messagesReq).retrieveEvents" file="/home/dendrite/dendrite/syncapi/routing/messages.go:361" backwards=true end="" req.id=BtQ7DOj0pnfE req.method=GET req.path="/_matrix/client/r0/rooms/!CDTRSRRRFMQJADZO:hispagatos.org/messages" start=t3787_6435572 user_id="@real-changeling:hispagatos.org"

[S7evinK]

At least all of the mentioned servers don't seem to really reply to requests at all and should be blacklisted at some point. (tested with https://federationtester.matrix.org/)
The MSC was for Spaces (iirc) and is now natively supported.

And for the initial question: No, no need to delete Jetstream. It may be possible that there are thousands of messages in the streams already and are now being processed. So the interesting part would be right after start up, how many messages are being restored by NATS (most importantly for the DendriteInputRoomEvent stream).

[r3k2]

@S7evinK to late, I was getting spam like 200 msg a sec from that same msg regarding user real-changeling so had to stop and do how we used to do when we first added jetstream, and deleted the folder ( we do not mind if we lose old msg's) now seems like the spam has stop, also have removed the MSC for the config and that is gone... local msg's are working good again, not sure about federation so far is not, going to let it sit for 1-2 hours. Regarding that msg about 8m for jetstream how can we fix/improve that?
[edit] is there a way to see if matrix.org has blocked us or anything related? Just in case, I come back in 2-3 hours and federation still not working, to check and rule it out.

[S7evinK]

Unsure about the 8MB warning, we used to pump quite a bit of data (iirc, 20MB+) through Kafka before switching to JetStream, not sure how this looks at the moment. But this shouldn't be a problem.

And if matrix.org is receiving your messages, I don't think your server is blocked or anything.

[TruncatedDinoSour]

same here, https://matrix.ari.lt/
looking forward to the fix

[r3k2]

@S7evinK I just check on the federation test thing, as domain hispagatos.org as usuall and says is having issues... we have not changed anything in our side in years regarding this, did we change the way we point a domain to a home server? our homeserver is matrix.hispagatos.org but our matrix domain is hispagatos.org
https://matrix.hispagatos.org/_matrix/federation/v1/version <--- this works but..
https://federationtester.matrix.org/#hispagatos.org this has a warning and a red issue ... like is not changing it to matrix.hispagatos.org like it used to ? I remember we used to have it the very old way with cname

_matrix._tcp.hispagatos.org
	SRV 	
10 0 443 matrix.hispagatos.org
	1h

and also on the nginx server..
going to check the wiki see if I still have it configure right. Posting in case something changed regarding this.
[edit] yes I just checked and still is in my nginx config.

worker_processes auto;

events {
 worker_connections 2048;
}


http {
   client_max_body_size 80M;
   types_hash_max_size 4096;
   include mime.types;

#change IP to location of monolith server
upstream monolith{
    server 127.0.0.1:8008;
}

server {
    listen 443 ssl; # IPv4
    listen [::]:443 ssl; # IPv6
    server_name matrix.hispagatos.org;

    ssl_certificate /etc/letsencrypt/live/matrix.hispagatos.org-0001/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/matrix.hispagatos.org-0001/privkey.pem;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams-2048.pem;

    proxy_set_header Host      $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_read_timeout         600;


    location / {
      root /usr/share/nginx/html;
      index index.html;
    }

    location /online {
 	alias /usr/share/nginx/online;
	index index.html;
    }
    
    location /usenet {
 	alias /usr/share/nginx/usenet;
	index index.html;
    }

    location /_matrix/federation/v1/user/devices/@rek2:matrix.hispagatos.org {
	return 404;
    }

    location /.well-known/matrix/server {
        return 200 '{ "m.server": "matrix.hispagatos.org:443" }';
    }

    location /.well-known/matrix/client {
        # If your sever_name here doesn't match your matrix homeserver URL
        # (e.g. hostname.com as server_name and matrix.hostname.com as homeserver URL)
        add_header Access-Control-Allow-Origin '*';
        return 200 '{ "m.homeserver": { "base_url": "https://matrix.hispagatos.org" } }';
    }

    location /_matrix {
        proxy_pass http://monolith;
    }
}

you can see the .well-known/matrix/server and client are present.

• https://matrix.hispagatos.org/.well-known/matrix/client
• https://matrix.hispagatos.org/.well-known/matrix/server

[S7evinK]

Right, so: both well-known files should be served on https://hispagatos.org instead of https://matrix.hispagatos.org

If you need to use SRV records, hispagatos.org needs a valid TLS certificate. (this is not recommended, though).

So my course of action would be:

• Move the .well-known files to be served at
  • https://hispagatos.org/.well-known/matrix/client
  • https://hispagatos.org/.well-known/matrix/server
• Check the federation tester
  • if server name/.well-known result contains explicit port number: no SRV lookup done in the results, you're good for the next step
• Remove the SRV record

[r3k2]

ok wait this rings a bell now! @S7evinK so hispagatos.org our blog where I also have those .well-known was down because of a DDOS attack for the whole week last week to sourcehut sr.ht ... and now they have some broken git pages ..... this now does make sense because... is the only thing really that has happened between last week and this week since it was working 100% fine.
let me see what I can do regarding sr.ht
[edit] so they temp changed the ip I need to point the domain to https://srht.site/custom-domains
[edit2] ok this i fixed should propagate now and everything should get back to how it was 1 week a go.
[edit3] we getting federation now!!

@S7evinK thanks like always I appreciate your time. and was able to fix the spamming I had looking into the logs etc so win win situation, the issue with the matrix version is still there but seems is not creating any issues to our server.

[65a]

Completely broken with latest dendrite-monolith:latest container for element clients. Cannot register a new account: "Your homeserve is too old and does not support the minimum API version required..."

[S7evinK]

Going to close this here, as there is currently nothing we can do - it is already reported to the Element Web team.
In the mean time, you can use https://riots.im and specifically https://riots.im/1.11.53/ to connect to Dendrite.

[65a]

This appears to be fixed in element nightly now

[tommie]

Nice. Here's the "partial revert": matrix-org/matrix-js-sdk#4014