Broken encryption (?): matrix-js-sdk 0.14.2 + Olm 3.0.0 + synapse 0.34.0

[zhaytee]

Hi folks, encountering an issue which may be similar to #731

Unencrypted rooms work perfectly fine with the sdk, but as soon as encryption is turned on, things stop working.

Encryption is, as far as I can tell, set up properly in my program. Olm 3.0.0 is being used with sdk 0.14.2, everything is being require()'d and initialized in the correct order, and the library never complains about crypto being unavailable or anything like that.

Below is some relevant logging output. Earlier in the log, device keys were coming in fine, but it seems that the sdk is not getting the room keys it's looking for? Not sure. Happy to provide more detail to assist in debugging this issue.

The homeserver being used is the latest synapse as of this writing.

Thank you!

Enabling encryption in !JhItAEKvfODTYTHOmG:r0x0rz.net; starting to track device lists for all users therein
Starting to track devices for room !JhItAEKvfODTYTHOmG:r0x0rz.net ...
info: sync state: SYNCING
Saving device tracking data at token s70731_3061240_1195_45775_31409_6_4606_5230_1
info: sync state: SYNCING
info: sync state: SYNCING
Error decrypting event (id=undefined): DecryptionError[msg: Not included in recipients]
New session already forced with device @zhaytee:r0x0rz.net:b5R1MdZ68i7484KCouSOvy96kIt8eLewTErRkPl/dSg at 1546806561017: not forcing another
info: sync state: SYNCING
enqueueing key request for !JhItAEKvfODTYTHOmG:r0x0rz.net / KJrwmzoefpDzQKl+RX8OvjAOebUsHDoD+lzKceVQEOo
Error decrypting event (id=$1546806619150Xzvsb:r0x0rz.net): DecryptionError[msg: The sender's device has not sent us the keys for this message., session: b5R1MdZ68i7484KCouSOvy96kIt8eLewTErRkPl/dSg|KJrwmzoefpDzQKl+RX8OvjAOebUsHDoD+lzKceVQEOo]
info: sync state: SYNCING
Looking for queued outgoing room key requests
Requesting keys for !JhItAEKvfODTYTHOmG:r0x0rz.net / KJrwmzoefpDzQKl+RX8OvjAOebUsHDoD+lzKceVQEOo from [@seb:r0x0rz.net:*,@zhaytee:r0x0rz.net:GYRFJEDMAV](id m1546806620094.17)
Looking for queued outgoing room key requests
No more outgoing room key requests
m.room_key_request from @seb:r0x0rz.net:QDMJBTSEAE for !JhItAEKvfODTYTHOmG:r0x0rz.net / KJrwmzoefpDzQKl+RX8OvjAOebUsHDoD+lzKceVQEOo (id m1546806620094.17)
info: sync state: SYNCING
room key request for unknown session !JhItAEKvfODTYTHOmG:r0x0rz.net / KJrwmzoefpDzQKl+RX8OvjAOebUsHDoD+lzKceVQEOo
Error decrypting event (id=undefined): DecryptionError[msg: Not included in recipients]
New session already forced with device @zhaytee:r0x0rz.net:b5R1MdZ68i7484KCouSOvy96kIt8eLewTErRkPl/dSg at 1546806561017: not forcing another
info: sync state: SYNCING
info: sync state: SYNCING
info: sync state: SYNCING
already have key request outstanding for !JhItAEKvfODTYTHOmG:r0x0rz.net / KJrwmzoefpDzQKl+RX8OvjAOebUsHDoD+lzKceVQEOo: not sending another
Error decrypting event (id=$1546806628151SMFcu:r0x0rz.net): DecryptionError[msg: The sender's device has not sent us the keys for this message., session: b5R1MdZ68i7484KCouSOvy96kIt8eLewTErRkPl/dSg|KJrwmzoefpDzQKl+RX8OvjAOebUsHDoD+lzKceVQEOo]

[zhaytee]

I just tried this with a completely fresh instance of synapse (new accounts and rooms), and the result was still the same. 😭

[zhaytee]

Sorry, I left out one critical piece of information: I'm not using the SDK in a browser, I'm using it via nodejs.

I suspect the issue is related to session store persistence. I'm using a module called node-local-storage to emulate browser local storage, which seems to be working fine on its own, but I notice that the matrix sdk doesn't seem to actually be saving anything to the store. 🧐

[zhaytee]

Possibly related: #437

[t3chguy]

You could take a look at the js half of github.com/matrix-org/matrix-search as that successfully uses e2e via this sdk in node js.

[t3chguy]

Area around https://github.com/matrix-org/matrix-search/blob/master/js_fetcher/index.ts#L66 looks relevant. (i too had many issues running e2e in node at the time)

[zhaytee]

Thanks for pointing me to those examples, @t3chguy ! I was able to determine that the cause of my issues were indeed related to the availability of a working LocalStorage implementation, and making sure that it's present in all the spots it needs to be.

In case anyone else happens upon this issue: feel free to take a look at the source to my project which initially caused me to open this issue in the first place. It's an example of a working nodejs Matrix client coded against matrix-js-sdk (including end-to-end encryption and all).

https://github.com/zhaytee/matrix-rpc-js

Closing this for now!

[n8fr8]

Just a follow-up on this. First, thanks for the great samples and examples @t3chguy and @zhaytee . Very helpful.

I am building a crypto-enabled Matrix chatbot here: https://github.com/guardianproject/ractive/blob/master/bot.js

It currently works sans-crypto, but when I enable it, I can't decrypt any messages, and the client exits.

Any thoughts, insights, something I missed? Thanks!

---

Getting saved sync token...
Getting push rules...
Got saved sync token
Getting saved sync...
Got reply from saved sync, exists? false
RIVEBOT succesfully loaded directory
Got push rules
Checking lazy load status...
Checking whether lazy loading has changed in store...
Storing client options...
Stored client options
Getting filter...
Sending first sync request...
Waiting for saved sync before starting sync processing...
Starting sync request processing...
Storing sync data...
Sync data stored
Processing sync response...
Crypto: initialising roomlist...
Starting sync request processing...
Crypto: initialising crypto object...
Crypto: initialising Olm...
Crypto: initialising Olm device...
Crypto: loading device list...
Crypto: fetching own devices...
Crypto: checking for key backup...
Checking key backup status...
Key backup is absent or missing required data
No usable key backup: not enabling key backup
Storing sync data...
Sync data stored
Processing sync response...
Starting sync request processing...
Storing sync data...
Sync data stored
Processing sync response...
Starting sync request processing...
Storing sync data...
Sync data stored
Processing sync response...
enqueueing key request for ****
Error decrypting event (id=*****): DecryptionError[msg: The sender's device has not sent us the keys for this message., session: ****
Decryption failure: [object Object]
Starting sync request processing...

[jryans]

@n8fr8 I think it will be easier to discuss this in a separate issue. I have created #958 for that.

[menturion]

I am experiencing the same issue.

matrix-js-sdk: v8.5.0
synapse: v1.21.2

I am still trying to get e2ee running. In my scenario the users are using web clients (based on the JS SDK) only.
For instance, decrypting messages in a direct 1-to-1 conversation only works if both participants are online while sending/receiving encrypted messages. In case the receiver is offline while the sender is sending a message the below error is shown:

logger.js:50 Error decrypting event DecryptionError[msg: The sender's device has not sent us the keys for this message., session:

All keys are downloaded, secret storage as well as cross-signing are bootstrapped. All involved devices are getting verified.

Do I have to explicitly upload any keys when sending a message?

[n8fr8]

By the way, I gave up and switched to https://github.com/matrix-org/pantalaimon

My life is much better now :)

[menturion]

@n8fr8

I have implemented a JS client. So unfortunately I can't switch and have to know the "missing link" to solve the above issue.

In case the receiver is offline while the sender is sending a message the below error is shown:

logger.js:50 Error decrypting event DecryptionError[msg: The sender's device has not sent us the keys for this message., session:

[n8fr8]

Understood. I will keep at it here, in truth, and hopefully we can figure out what the missing link in the code or our understanding is.

[jryans]

If there's something here you'd like the core team to examine, please open a new issue with details of your use case.

[menturion]

@jryans

Many thanks for your reply.

I already filed the issue here ... matrix-org/synapse#8624.
The solution (device dehydration/rehydration) is obviously already in progress.