App got completely stuck sending messages

[ara4n]

Steps to reproduce

I've had two instances today of trying to send messages into (different) rooms, where sending got completely wedged.

Looks like the E2EE sync loop is getting stuck somehow, perhaps calling /keys/query on a device whose server is down, or a bug in rust-sdk-crypto grinding to a halt.

Outcome

What did you expect?

E2EE that works, and messages whose sending don't get stuck solid.

What happened instead?

Stuck send msgs, which i can't recover from.

Your phone model

No response

Operating system version

No response

Application version

292

Homeserver

No response

Will you send logs?

Yes

[manuroe]

@bnjbvr can you have a look at those logs, please? Kegan's analysis on the last rageshake is interesting.

[bnjbvr]

Looking.

[bnjbvr]

4/6 of the above crash reports include the following panic in Rust code:

2023-07-27_161951-ZBIH5VQT/console.3_1.log
258:thread 'tokio-runtime-worker' panicked at 'Session wasn't created nor shared', /Users/mauroromito/Documents/Xworkspace/matrix-rust-sdk/crates/matrix-sdk-crypto/src/session_manager/group_sessions.rs:205:50
260:2023-07-27T00:07:33.309181Z ERROR root:send_queued_messages{room_id="XXX:matrix.org"}: matrix_sdk_ui::timeline::queue: crates/matrix-sdk-ui/src/timeline/queue.rs:144: Message-sending task failed: task 283504 panicked

This seems to match the symptoms (not being able to send messages). @poljar and I debugged this, and we think it's caused by a combination of two issues:

• on the one hand, when the olm machine is getting regenerated because it detected some other apps took the cross-process lock, it will invalidate its internal caches. Assuming a perfect timing of events, this could in theory happen in the middle of sending messages (between sharing a room key and encrypting a message with that room key), leading to a bad race condition. The encryption not seeing the cached room key data would assume invalid state and panic; this will get fixed with fix(crypto): don't rely on outgoing sessions being in cache when encrypting a message matrix-org/matrix-rust-sdk#2348
• on the other hand, the olm machine gets invalidated way more than it should (i.e. every time we attempt to take the lock), and this is getting fixed in fix(crypto): store the generation counter in the Client, not the OlmMachine matrix-org/matrix-rust-sdk#2347.

My hope is that, despite the panic not appearing in the two other rageshakes, this is the same issue, or that the second PR will make it a non-issue.

Assuming the fixes go in quickly today, we can consider whether to close this issue in a week or so, if there are no new instances of this problem after the fix is in.

[bnjbvr]

I claim this has been fixed by the two PRs mentioned in the previous comment. If new issues appear, we can fix them later.