1
0
Commit Graph

24250 Commits

Author SHA1 Message Date
Erik Johnston 1226bb4de2 Merge branch 'madlittlemods/sliding-sync-pre-populate-room-meta-data' into erikj/ss_room_lists_new_tables 2024-08-28 15:38:29 +01:00
Erik Johnston 90d0e035dd Fix port script tests by handling empty DBs correctly 2024-08-28 14:39:25 +01:00
Erik Johnston ab414f2ab8 Use event_auth table to get previous membership 2024-08-28 14:23:32 +01:00
Erik Johnston 6f9932d146 Handle old rows with null event_stream_ordering column 2024-08-28 14:23:06 +01:00
Erik Johnston bb905cd02c Only run the sliding sync background updates on the main database 2024-08-28 11:44:56 +01:00
Erik Johnston 7c9c62051c Remove all rooms pulled out from the queue 2024-08-28 11:31:19 +01:00
Erik Johnston 42f43a8ddf Move computing interested rooms to room list 2024-08-28 10:45:39 +01:00
Eric Eastwood da463fb102 Add unique index right away for sliding_sync_joined_rooms_to_recalculate
This makes it so we can always `upsert` to avoid duplicates otherwise
I'm not sure of how to not insert duplicates in certain situations
(see FIXME in the diff) which would cause problems down the line
for the unique index being added later.
2024-08-28 00:50:33 -05:00
Eric Eastwood 8468401a97 Adapt to using sliding_sync_joined_rooms_to_recalculate table 2024-08-28 00:42:14 -05:00
Eric Eastwood 53b7309f6c Add sliding_sync_joined_rooms_to_recalculate table 2024-08-27 20:48:56 -05:00
Eric Eastwood 94e1a54687 get_events(...) will omit events from unknown room versions
Thanks @erikjohnston
2024-08-27 20:01:55 -05:00
Eric Eastwood a507f152c9 Use stream_id of some point before we fetch the current state
This is simpler and some rooms are so old that they don't have
`current_state_delta_stream` yet. It's easier if we just get a
general max `stream_id` of the whole table than the max `stream_id`
for the specific room anyway.

Thanks @erikjohnston
2024-08-27 19:45:50 -05:00
Eric Eastwood 9d08bc2157 Remove debug logs 2024-08-27 19:35:05 -05:00
Eric Eastwood 56a4c0ba6e Round out tests 2024-08-27 19:34:16 -05:00
Eric Eastwood 85a60c3132 More tests 2024-08-27 19:27:24 -05:00
Eric Eastwood e5e7269998 Add more tests 2024-08-27 18:49:53 -05:00
Eric Eastwood c8e17f7479 Add test when no rooms 2024-08-27 18:21:25 -05:00
Eric Eastwood 4dc9e268e6 Add test for catch-up background update 2024-08-27 18:08:17 -05:00
Eric Eastwood 9a7d8c2be4 Start catch-up if nothing written yet 2024-08-27 17:28:07 -05:00
Eric Eastwood c51a309da5 Maybe: always start background update 2024-08-27 17:11:51 -05:00
Eric Eastwood 9764f626ea Fix query in Postgres 2024-08-27 12:09:00 -05:00
Erik Johnston 5592a9e6cb Port room lists to new file 2024-08-27 15:35:02 +01:00
Erik Johnston defd4aca67 Speed up fetching latest stream positions via cache (#17606)
The idea is to engineer it so that the vast majority of the rooms can
stay in the cache, so we can just ignore them.
2024-08-27 11:03:56 +00:00
Erik Johnston b4d95409fb Fix @tag_args for non-methods (#17604)
The decorator assumed we were always wrapping function methods
2024-08-27 11:47:28 +01:00
Eric Eastwood 7a0c281028 Add placeholder tests 2024-08-26 19:43:52 -05:00
Eric Eastwood 7fe5d31e20 Note down caveat about forgotten 2024-08-26 18:52:10 -05:00
Eric Eastwood 53473a0eb4 Adapt sliding_sync_joined_rooms background update to use event_stream_ordering for progress
This way we can re-use it for the catch-up background process
2024-08-26 18:35:49 -05:00
Eric Eastwood eb3c84cf45 Kick-off background update for out-of-date snapshots 2024-08-26 17:31:26 -05:00
Eric Eastwood 6a44686dc3 Why it matters 2024-08-26 16:32:59 -05:00
Eric Eastwood a94c1dd62c Add more context for why 2024-08-26 16:27:20 -05:00
Eric Eastwood 8bddbe23bd Clear out-of-date rows 2024-08-26 16:19:47 -05:00
Eric Eastwood addb91485f Split test cases 2024-08-26 16:11:56 -05:00
Eric Eastwood 9795556052 Update comment 2024-08-26 14:42:55 -05:00
dependabot[bot] f1a1c7fc53 Bump types-setuptools from 71.1.0.20240726 to 71.1.0.20240818 (#17586) 2024-08-23 09:53:14 +01:00
dependabot[bot] cb9fa062b7 Bump sentry-sdk from 2.12.0 to 2.13.0 (#17585) 2024-08-23 09:53:06 +01:00
dependabot[bot] 74b75cfd54 Bump cryptography from 42.0.8 to 43.0.0 (#17584) 2024-08-23 09:52:53 +01:00
dependabot[bot] 87d13fd143 Bump types-jsonschema from 4.23.0.20240712 to 4.23.0.20240813 (#17583) 2024-08-23 09:52:34 +01:00
dependabot[bot] ad2cd9aefd Bump serde_json from 1.0.124 to 1.0.125 (#17582) 2024-08-23 09:52:26 +01:00
dependabot[bot] ad0ee53993 Bump serde from 1.0.206 to 1.0.208 (#17581) 2024-08-23 09:52:16 +01:00
Eric Eastwood a57d47b778 Use simple_upsert_txn for sliding_sync_membership_snapshots in background update 2024-08-22 23:59:06 -05:00
Eric Eastwood b6a7d2bf6c Use simple_upsert_txn for sliding_sync_joined_rooms in background update 2024-08-22 23:30:00 -05:00
Eric Eastwood f8926d07df Fix partial-stated room re-syncing state but nothing has changed
Fixes failing test in CI: `tests.handlers.test_federation.PartialJoinTestCase.test_failed_partial_join_is_clean`
```
2024-08-22 18:57:22-0500 [-] synapse.metrics.background_process_metrics - 253 - ERROR - sync_partial_state_room-0 - Background process 'sync_partial_state_room' threw an exception
	Traceback (most recent call last):
	  File "synapse/synapse/metrics/background_process_metrics.py", line 251, in run
	    return await func(*args, **kwargs)
	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "synapse/synapse/handlers/federation.py", line 1842, in _sync_partial_state_room_wrapper
	    await self._sync_partial_state_room(
	  File "synapse/synapse/handlers/federation.py", line 1933, in _sync_partial_state_room
	    await self.state_handler.update_current_state(room_id)
	  File "synapse/synapse/state/__init__.py", line 554, in update_current_state
	    await self._storage_controllers.persistence.update_current_state(room_id)
	  File "synapse/synapse/storage/controllers/persist_events.py", line 491, in update_current_state
	    await self._event_persist_queue.add_to_queue(
	  File "synapse/synapse/storage/controllers/persist_events.py", line 245, in add_to_queue
	    res = await make_deferred_yieldable(end_item.deferred.observe())
	          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "synapse/synapse/storage/controllers/persist_events.py", line 288, in handle_queue_loop
	    ret = await self._per_item_callback(room_id, item.task)
	          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "synapse/synapse/storage/controllers/persist_events.py", line 370, in _process_event_persist_queue_task
	    await self._update_current_state(room_id, task)
	  File "synapse/synapse/storage/controllers/persist_events.py", line 507, in _update_current_state
	    await self.persist_events_store._calculate_sliding_sync_table_changes(
	  File "synapse/synapse/storage/databases/main/events.py", line 624, in _calculate_sliding_sync_table_changes
	    assert most_recent_event_pos_results, (
	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	AssertionError: We should not be seeing `None` here because we are still in the room (!room:example.com) and it should at-least have a join membership event that's keeping us here.
```
2024-08-22 19:06:04 -05:00
Eric Eastwood 21cc97ba9d Use simple_upsert_many_txn for sliding_sync_membership_snapshots
See https://github.com/element-hq/synapse/pull/17512#discussion_r1726820006
2024-08-22 18:52:05 -05:00
Eric Eastwood fdb8b5931f Correct comment 2024-08-22 18:43:51 -05:00
Eric Eastwood 4b866c4fca Simplify what we need to think about to grab the best effort value 2024-08-22 18:36:43 -05:00
Eric Eastwood 088a4c7cf0 Use simple_upsert_txn to update sliding_sync_joined_rooms
See https://github.com/element-hq/synapse/pull/17512#discussion_r1726817206
2024-08-22 18:26:37 -05:00
Eric Eastwood 0726a6d58b Derive best effort stream_ordering outside of the transaction
See https://github.com/element-hq/synapse/pull/17512#discussion_r1727995882
2024-08-22 18:13:00 -05:00
Eric Eastwood 6edc4c78ce Allow for no bump_stamp (fix portdb CI job)
See https://github.com/element-hq/synapse/pull/17512#discussion_r1725998219
2024-08-22 17:09:43 -05:00
Eric Eastwood 44432e2118 Move tests to dedicated file
See https://github.com/element-hq/synapse/pull/17512#discussion_r1726849798
2024-08-22 16:56:09 -05:00
Eric Eastwood bcba8cccfe No need for transaction
See https://github.com/element-hq/synapse/pull/17512#discussion_r1726844107
2024-08-22 16:52:31 -05:00