1
0
Commit Graph

44 Commits

Author SHA1 Message Date
Shay
1c95ddd09b Batch up storing state groups when creating new room (#14918) 2023-02-24 13:15:29 -08:00
dependabot[bot]
9bb2eac719 Bump black from 22.12.0 to 23.1.0 (#15103) 2023-02-22 15:29:09 -05:00
Shay
03bccd542b Add a class UnpersistedEventContext to allow for the batching up of storing state groups (#14675)
* add class UnpersistedEventContext

* modify create new client event to create unpersistedeventcontexts

* persist event contexts after creation

* fix tests to persist unpersisted event contexts

* cleanup

* misc lints + cleanup

* changelog + fix comments

* lints

* fix batch insertion?

* reduce redundant calculation

* add unpersisted event classes

* rework compute_event_context, split into function that returns unpersisted event context and then persists it

* use calculate_context_info to create unpersisted event contexts

* update typing

* $%#^&*

* black

* fix comments and consolidate classes, use attr.s for class

* requested changes

* lint

* requested changes

* requested changes

* refactor to be stupidly explicit

* clearer renaming and flow

* make partial state non-optional

* update docstrings

---------

Co-authored-by: Erik Johnston <erik@matrix.org>
2023-02-09 13:05:02 -08:00
David Robertson
e2a1adbf5d Allow selecting "prejoin" events by state keys (#14642)
* Declare new config

* Parse new config

* Read new config

* Don't use trial/our TestCase where it's not needed

Before:

```
$ time trial tests/events/test_utils.py > /dev/null

real	0m2.277s
user	0m2.186s
sys	0m0.083s
```

After:
```
$ time trial tests/events/test_utils.py > /dev/null

real	0m0.566s
user	0m0.508s
sys	0m0.056s
```

* Helper to upsert to event fields

without exceeding size limits.

* Use helper when adding invite/knock state

Now that we allow admins to include events in prejoin room state with
arbitrary state keys, be a good Matrix citizen and ensure they don't
accidentally create an oversized event.

* Changelog

* Move StateFilter tests

should have done this in #14668

* Add extra methods to StateFilter

* Use StateFilter

* Ensure test file enforces typed defs; alphabetise

* Workaround surprising get_current_state_ids

* Whoops, fix mypy
2022-12-13 00:54:46 +00:00
David Robertson
b5b5f66084 Move StateFilter to synapse.types (#14668)
* Move `StateFilter` to `synapse.types`

* Changelog
2022-12-12 16:19:30 +00:00
Patrick Cloke
3ac412b4e2 Require types in tests.storage. (#14646)
Adds missing type hints to `tests.storage` package
and does not allow untyped definitions.
2022-12-09 12:36:32 -05:00
Erik Johnston
0b87eb8e0c Make DictionaryCache have better expiry properties (#13292) 2022-07-21 17:13:44 +01:00
Erik Johnston
1e453053cb Rename storage classes (#12913) 2022-05-31 12:17:50 +00:00
Richard van der Hoff
e24ff8ebe3 Remove HomeServer.get_datastore() (#12031)
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.

Part of #11733
2022-02-23 11:04:02 +00:00
reivilibre
eb609c65d0 Fix bug in StateFilter.return_expanded() and add some tests. (#12016) 2022-02-18 14:54:31 +00:00
reivilibre
406f7bfa17 Add an approximate difference method to StateFilters (#10825) 2021-10-12 10:44:59 +01:00
reivilibre
8eb7cb2e0d Make StateFilter frozen so we can hash it (#10816)
Also enables Mypy for related tests.
2021-09-14 16:35:53 +01:00
Jonathan de Jong
4b965c862d Remove redundant "coding: utf-8" lines (#9786)
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Patrick Cloke
e7b769aea1 Convert storage test cases to HomeserverTestCase. (#9736) 2021-04-06 07:21:02 -04:00
Patrick Cloke
01dd90b0f0 Add type hints to DictionaryCache and TTLCache. (#9442) 2021-03-29 12:15:33 -04:00
Patrick Cloke
b3a97d6dac Convert some of the data store to async. (#7976) 2020-07-30 07:20:41 -04:00
Patrick Cloke
3345c166a4 Convert storage layer to async/await. (#7963) 2020-07-28 16:09:53 -04:00
Patrick Cloke
cc9bb3dc3f Convert the message handler to async/await. (#7884) 2020-07-22 12:29:15 -04:00
Patrick Cloke
509e381afa Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
Erik Johnston
8df862e45d Add rooms.room_version column (#6729)
This is so that we don't have to rely on pulling it out from `current_state_events` table.
2020-01-27 14:30:57 +00:00
Erik Johnston
75d8f26ac8 Split state groups into a separate data store (#6296) 2019-12-20 10:48:24 +00:00
Erik Johnston
69f0054ce6 Port to use state storage 2019-10-30 14:46:54 +00:00
Erik Johnston
3ca4c7c516 Use new EventPersistenceStore 2019-10-23 16:15:03 +01:00
Amber Brown
4806651744 Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00
Amber Brown
32e7c9e7f2 Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Amber Brown
b36c82576e Run Black on the tests again (#5170) 2019-05-10 00:12:11 -05:00
Richard van der Hoff
54a87a7b08 Collect room-version variations into one place (#4969)
Collect all the things that make room-versions different to one another into
one place, so that it's easier to define new room versions.
2019-04-01 10:24:38 +01:00
Amber Brown
3f189c902e Fix flake8 (#4519) 2019-01-30 10:53:17 +00:00
Erik Johnston
9770ed91c2 Fix tests 2019-01-25 10:32:26 +00:00
Erik Johnston
be6a7e47fa Revert "Require event format version to parse or create events" 2019-01-25 10:23:51 +00:00
Erik Johnston
e79ba9eb34 Fix tests 2019-01-24 09:28:16 +00:00
Erik Johnston
cb53ce9d64 Refactor state group lookup to reduce DB hits (#4011)
Currently when fetching state groups from the data store we make two
hits two the database: once for members and once for non-members (unless
request is filtered to one or the other). This adds needless load to the
datbase, so this PR refactors the lookup to make only a single database
hit.
2018-10-25 17:49:55 +01:00
Richard van der Hoff
ae6ad4cf41 docstrings and unittests for storage.state (#3958)
I spent ages trying to figure out how I was going mad...
2018-09-27 11:22:25 +01:00
Amber Brown
52ec6e9dfa Port tests/ to Python 3 (#3808) 2018-09-07 02:58:18 +10:00
Matthew Hodgson
bb81e78ec6 Split the state_group_cache in two (#3726)
Splits the state_group_cache in two.

One half contains normal state events; the other contains member events.

The idea is that the lazyloading common case of: "I want a subset of member events plus all of the other state" can be accomplished efficiently by splitting the cache into two, and asking for "all events" from the non-members cache, and "just these keys" from the members cache.  This means we can avoid having to make DictionaryCache aware of these sort of complicated queries, whilst letting LL requests benefit from the caching.

Previously we were unable to sensibly use the caching and had to pull all state from the DB irrespective of the filtering, which made things slow.  Hopefully fixes https://github.com/matrix-org/synapse/issues/3720.
2018-08-22 00:56:37 +02:00
Matthew Hodgson
2f78f432c4 speed up /members and add at= and membership params (#3568) 2018-08-15 16:35:22 +01:00
Amber Brown
99dd975dae Run tests under PostgreSQL (#3423) 2018-08-13 16:47:46 +10:00
black
8b3d9b6b19 Run black. 2018-08-10 23:54:09 +10:00
Amber Brown
2511f3f8a0 Test fixes for Python 3 (#3647) 2018-08-09 12:22:01 +10:00
Matthew Hodgson
0620d27f4d flake8 2018-07-25 17:21:17 +01:00
Matthew Hodgson
0a7ee0ab8b add tests for _get_some_state_from_cache 2018-07-25 16:33:52 +01:00
Matthew Hodgson
cd241d6bda incorporate more review 2018-07-24 12:39:40 +01:00
Matthew Hodgson
650daf5628 make test work 2018-07-19 20:49:44 +01:00
Matthew Hodgson
1fa4f7e03e first cut of a UT for testing state store (untested) 2018-07-19 20:19:32 +01:00