kaslo/synapse
1
0
Fork 0
Code Wiki Activity
20,455 Commits 1,047 Branches 805 Tags
ae01a7edd32c8a4650700294c50a37385fa07984
Commit Graph

20455 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Richard van der Hoff
ae01a7edd3 Update type annotations for compatiblity with prometheus_client 0.14 (#12389) 
Principally, `prometheus_client.REGISTRY.register` now requires its argument to
extend `prometheus_client.Collector`.

Additionally, `Gauge.set` is now annotated so that passing `Optional[int]`
causes an error.
 2022-04-06 12:59:04 +00:00
Eric Eastwood
793d03e2c5 Generate historic pagination token for /messages when no ?from token provided (#12370) 2022-04-06 11:40:28 +01:00
David Robertson
573cd0f92f Add missing dependency on importlib_metadata (#12384) 2022-04-05 18:25:56 +00:00
David Robertson
7ec9b06303 Poetry: initial pyproject.toml and poetry.lock (#12381) 
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
 2022-04-05 18:40:56 +01:00
Sean Quah
fd1e7d0fc2 Post 1.56.0 release tidy up (#12379) 2022-04-05 17:35:20 +01:00
Andrew Morgan
163fd686b5 Update mdbook to 0.4.17 (#12339) 
Update mdbook (the tool used to render the documentation website) to version 0.4.17.
 2022-04-05 17:21:58 +01:00
Sean Quah
79e7c2c426 Fix edge case where a Linearizer could get stuck (#12358) 
Just after a task acquires a contended `Linearizer` lock, it sleeps.
If the task is cancelled during this sleep, we need to release the lock.

Signed-off-by: Sean Quah <seanq@element.io>
 2022-04-05 17:19:16 +01:00
Sean Quah
31c1209c50 Make StreamToken and RoomStreamToken methods propagate cancellations (#12366) 
`StreamToken.from_string` and `RoomStreamToken.parse` are both async
methods that could be cancelled. These methods must not replace
`CancelledError`s with `SynapseError`s.

Signed-off-by: Sean Quah <seanq@element.io>
 2022-04-05 16:56:52 +01:00
Sean Quah
9c4c49991d Update docstrings for ReadWriteLock tests (#12354) 
Signed-off-by: Sean Quah <seanq@element.io>
 2022-04-05 16:54:40 +01:00
Sean Quah
800ba87cc8 Refactor and convert Linearizer to async (#12357) 
Refactor and convert `Linearizer` to async. This makes a `Linearizer`
cancellation bug easier to fix.

Also refactor to use an async context manager, which eliminates an
unlikely footgun where code that doesn't immediately use the context
manager could forget to release the lock.

Signed-off-by: Sean Quah <seanq@element.io>
 2022-04-05 15:43:52 +01:00
Sean Quah
ab3fdcf960 Merge branch 'master' into develop 2022-04-05 15:14:53 +01:00
Sean Quah
41b5f72677 Convert Linearizer tests from inlineCallbacks to async (#12353) 
Signed-off-by: Sean Quah <seanq@element.io>
 2022-04-05 14:56:09 +01:00
Erik Johnston
66053b6bfb Prefill more stream change caches. (#12372) 2022-04-05 14:26:41 +01:00
Dirk Klimpel
d666fc02fa Add type hints to some tests files (#12371) 2022-04-05 13:54:41 +01:00
Sean Quah
ac80bfba42 Tweak CHANGES.md v1.56.0 2022-04-05 12:55:15 +01:00
reivilibre
42d8710f38 Fix a spec compliance issue where requests to the /publicRooms federation API would specify limit as a string. (#12364) 2022-04-05 12:45:36 +01:00
reivilibre
708d88b1a2 Allow specifying the Postgres database's port when running unit tests with Postgres. (#12376) 2022-04-05 12:44:48 +01:00
Sean Quah
7a95e80418 1.56.0 2022-04-05 12:44:00 +01:00
Richard van der Hoff
efdbcfd6af Merge tag 'v1.56.0rc1' into develop 
Synapse 1.56.0rc1 (2022-03-29)
==============================

Features
--------

- Allow modules to store already existing 3PID associations. ([\#12195](https://github.com/matrix-org/synapse/issues/12195))
- Allow registering server administrators using the module API. Contributed by Famedly. ([\#12250](https://github.com/matrix-org/synapse/issues/12250))

Bugfixes
--------

- Fix a long-standing bug which caused the `/_matrix/federation/v1/state` and `/_matrix/federation/v1/state_ids` endpoints to return incorrect or invalid data when called for an event which we have stored as an "outlier". ([\#12087](https://github.com/matrix-org/synapse/issues/12087))
- Fix a long-standing bug where events from ignored users would still be considered for relations. ([\#12227](https://github.com/matrix-org/synapse/issues/12227), [\#12232](https://github.com/matrix-org/synapse/issues/12232), [\#12285](https://github.com/matrix-org/synapse/issues/12285))
- Fix a bug introduced in Synapse 1.53.0 where an unnecessary query could be performed when fetching bundled aggregations for threads. ([\#12228](https://github.com/matrix-org/synapse/issues/12228))
- Fix a bug introduced in Synapse 1.52.0 where admins could not deactivate and GDPR-erase a user if Synapse was configured with limits on avatars. ([\#12261](https://github.com/matrix-org/synapse/issues/12261))

Improved Documentation
----------------------

- Fix the link to the module documentation in the legacy spam checker warning message. ([\#12231](https://github.com/matrix-org/synapse/issues/12231))
- Remove incorrect prefixes in the worker documentation for some endpoints. ([\#12243](https://github.com/matrix-org/synapse/issues/12243))
- Correct `check_username_for_spam` annotations and docs. ([\#12246](https://github.com/matrix-org/synapse/issues/12246))
- Correct Authentik OpenID typo, and add notes on troubleshooting. Contributed by @IronTooch. ([\#12275](https://github.com/matrix-org/synapse/issues/12275))
- HAProxy reverse proxy guide update to stop sending IPv4-mapped address to homeserver. Contributed by @villepeh. ([\#12279](https://github.com/matrix-org/synapse/issues/12279))

Internal Changes
----------------

- Rename `shared_rooms` to `mutual_rooms` ([MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666)), as per proposal changes. ([\#12036](https://github.com/matrix-org/synapse/issues/12036))
- Remove check on `update_user_directory` for shared rooms handler ([MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666)), and update/expand documentation. ([\#12038](https://github.com/matrix-org/synapse/issues/12038))
- Refactor `create_new_client_event` to use a new parameter, `state_event_ids`, which accurately describes the usage with [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) instead of abusing `auth_event_ids`. ([\#12083](https://github.com/matrix-org/synapse/issues/12083), [\#12304](https://github.com/matrix-org/synapse/issues/12304))
- Refuse to start if registration is enabled without email, captcha, or token-based verification unless the new config flag `enable_registration_without_verification` is set. ([\#12091](https://github.com/matrix-org/synapse/issues/12091))
- Add tests for database transaction callbacks. ([\#12198](https://github.com/matrix-org/synapse/issues/12198))
- Handle cancellation in `DatabasePool.runInteraction`. ([\#12199](https://github.com/matrix-org/synapse/issues/12199))
- Add missing type hints for cache storage. ([\#12216](https://github.com/matrix-org/synapse/issues/12216))
- Add missing type hints for storage. ([\#12248](https://github.com/matrix-org/synapse/issues/12248), [\#12255](https://github.com/matrix-org/synapse/issues/12255))
- Add type hints to tests files. ([\#12224](https://github.com/matrix-org/synapse/issues/12224), [\#12240](https://github.com/matrix-org/synapse/issues/12240), [\#12256](https://github.com/matrix-org/synapse/issues/12256))
- Use type stubs for `psycopg2`. ([\#12269](https://github.com/matrix-org/synapse/issues/12269))
- Improve type annotations for `execute_values`. ([\#12311](https://github.com/matrix-org/synapse/issues/12311))
- Clean-up logic around rebasing URLs for URL image previews. ([\#12219](https://github.com/matrix-org/synapse/issues/12219))
- Use the `ignored_users` table in additional places instead of re-parsing the account data. ([\#12225](https://github.com/matrix-org/synapse/issues/12225))
- Refactor the relations endpoints to add a `RelationsHandler`. ([\#12237](https://github.com/matrix-org/synapse/issues/12237))
- Generate announcement links in the release script. ([\#12242](https://github.com/matrix-org/synapse/issues/12242))
- Improve error message when dependencies check finds a broken installation. ([\#12244](https://github.com/matrix-org/synapse/issues/12244))
- Compress metrics HTTP resource when enabled. Contributed by Nick @ Beeper. ([\#12258](https://github.com/matrix-org/synapse/issues/12258))
- Refuse to start if the PostgreSQL database has a non-`C` locale, unless the config flag `allow_unsafe_db_locale` is set to true. ([\#12262](https://github.com/matrix-org/synapse/issues/12262), [\#12288](https://github.com/matrix-org/synapse/issues/12288))
- Optionally include account validity expiration information to experimental [MSC3720](https://github.com/matrix-org/matrix-doc/pull/3720) account status responses. ([\#12266](https://github.com/matrix-org/synapse/issues/12266))
- Add a new cache `_get_membership_from_event_id` to speed up push rule calculations in large rooms. ([\#12272](https://github.com/matrix-org/synapse/issues/12272))
- Re-enable Complement concurrency in CI. ([\#12283](https://github.com/matrix-org/synapse/issues/12283))
- Remove unused test utilities. ([\#12291](https://github.com/matrix-org/synapse/issues/12291))
- Enhance logging for inbound federation events. ([\#12301](https://github.com/matrix-org/synapse/issues/12301))
- Fix compatibility with the recently-released Jinja 3.1. ([\#12313](https://github.com/matrix-org/synapse/issues/12313))
- Avoid trying to calculate the state at outlier events. ([\#12314](https://github.com/matrix-org/synapse/issues/12314))
 2022-04-05 12:26:20 +01:00
Sean Quah
ca7e34cb57 Pin signedjson to <= 1.1.1 as a temporary workaround for #12324 
To be reverted after the Synapse 1.56 release.

Signed-off-by: Sean Quah <seanq@element.io>
 2022-04-05 12:13:44 +01:00
David Robertson
a7293ef16f Bump black and click versions (#12320) 2022-04-05 11:04:28 +01:00
Eric Eastwood
5218fe7670 Explain how to decipher live and historic pagination tokens (#12317) 2022-04-05 10:57:09 +01:00
Shay
f608e6c8cf Update broken link in workers.md (#12369) 2022-04-04 17:40:31 -07:00
Eric Eastwood
9633eb2162 Allow non-member state sent in room batch to resolve for historic events (MSC2716) (#12329) 
Part of https://github.com/matrix-org/synapse/issues/12110

Complement test: https://github.com/matrix-org/complement/pull/354

Previously, they didn't resolve because async `filter_events_for_client`
removes all outlier state except for out-of-band membership.

And fundamentally, we have the state at these events so they shouldn't be marked as outliers.
 2022-04-04 14:54:50 -05:00
Erik Johnston
b446c99ac9 Prefill the device_list_stream_cache (#12367) 
* Prefill the device_list_stream_cache

* Newsfile

* Newsfile
 2022-04-04 20:12:25 +01:00
Erik Johnston
5c9e39e619 Track device list updates per room. (#12321) 
This is a first step in dealing with #7721.

The idea is basically that rather than calculating the full set of users a device list update needs to be sent to up front, we instead simply record the rooms the user was in at the time of the change. This will allow a few things:

1. we can defer calculating the set of remote servers that need to be poked about the change; and
2. during `/sync` and `/keys/changes` we can avoid also avoid calculating users who share rooms with other users, and instead just look at the rooms that have changed.

However, care needs to be taken to correctly handle server downgrades. As such this PR writes to both `device_lists_changes_in_room` and the `device_lists_outbound_pokes` table synchronously. In a future release we can then bump the database schema compat version to `69` and then we can assume that the new `device_lists_changes_in_room` exists and is handled.

There is a temporary option to disable writing to `device_lists_outbound_pokes` synchronously, allowing us to test the new code path does work (and by implication upgrading to a future release and downgrading to this one will work correctly).

Note: Ideally we'd do the calculation of room to servers on a worker (e.g. the background worker), but currently only master can write to the `device_list_outbound_pokes` table.
 2022-04-04 15:25:20 +01:00
David Robertson
80839a44f1 Remove more dead/broken dev scripts (#12355) 2022-04-03 21:21:08 +01:00
Richard van der Hoff
f0b03186d9 Add type hints for tests/unittest.py. (#12347) 
In particular, add type hints for get_success and friends, which are then helpful in a bunch of places.
 2022-04-01 16:04:16 +00:00
Richard van der Hoff
33ebee47e4 Remove redundant get_success calls in test code (#12346) 
There are a bunch of places we call get_success on an immediate value, which is unnecessary. Let's rip them out, and remove the redundant functionality in get_success and friends.
 2022-04-01 16:10:31 +01:00
reivilibre
c4cf916ed7 Default to private room visibility rather than public when a client does not specify one, according to spec. (#12350) 2022-04-01 15:55:09 +01:00
David Robertson
336bff1104 Burn check_signature dev script. (#12351) 2022-04-01 13:41:42 +00:00
Nick Mills-Barrett
993d90f82b Use a sequence to generate AS transaction IDs, drop last_txn AS state (#12209) 
Switching to a sequence means there's no need to track `last_txn` on the
AS state table to generate new TXN IDs. This also means that there is
no longer contention between the AS scheduler and AS handler on updates
to the `application_services_state` table, which will prevent serialization
errors during the complete AS txn transaction.
 2022-04-01 13:33:25 +00:00
David Robertson
21351820e0 Remove list_url_patterns dev script (#12349) 2022-04-01 13:05:21 +00:00
Richard van der Hoff
b7762b0c9f Move single-use methods out of TestCase (#12348) 
These methods are only used by a single testcase, so they shouldn't be
cluttering up the base `TestCase` class.
 2022-04-01 12:48:08 +00:00
reivilibre
f871222880 Move update_client_ip background job from the main process to the background worker. (#12251) 2022-04-01 13:08:55 +01:00
Richard van der Hoff
319a805cd3 Raise an exception when getting state at an outlier (#12191) 
It seems like calling `_get_state_group_for_events` for an event where the
state is unknown is an error. Accordingly, let's raise an exception rather than
silently returning an empty result.
 2022-04-01 13:01:49 +01:00
Richard van der Hoff
9b43df1f7b Optimise _get_state_after_missing_prev_event: use /state (#12040) 
If we're missing most of the events in the room state, then we may as well call the /state endpoint, instead of individually requesting each and every event.
 2022-04-01 12:53:42 +01:00
Brendan Abolivier
e4409301ba Add a module callback to react to account data changes (#12327) 
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
 2022-04-01 11:22:48 +02:00
Amanda Graven
4e900ece42 Add set_user_admin function to the module API (#12341) 2022-04-01 09:31:30 +01:00
Patrick Cloke
bebf994ee8 Move MSC2654 support behind an experimental configuration flag. (#12295) 
To match the current thinking on disabling experimental features by default.
 2022-03-31 15:05:13 -04:00
Richard van der Hoff
6927d87254 Handle outliers in /federation/v1/event (#12332) 
The intention here is to avoid doing state lookups for outliers in
`/_matrix/federation/v1/event`. Unfortunately that's expanded into something of
a rewrite of `filter_events_for_server`, which ended up trying to do that
operation in a couple of places.
 2022-03-31 17:39:34 +00:00
Patrick Cloke
11df4ec6c2 Add more type hints to the main state store. (#12267) 2022-03-31 13:38:09 -04:00
Brendan Abolivier
5e88143dff Add a callback to react to 3PID associations (#12302) 2022-03-31 18:27:21 +02:00
Richard van der Hoff
34a8370d7b README-testing.md: fix minor error 2022-03-31 13:08:03 +01:00
Patrick Cloke
adbf975623 Remove an unnecessary class from the relations code. (#12338) 
The PaginationChunk class attempted to bundle some properties
together, but really just caused callers to jump through hoops and
hid implementation details.
 2022-03-31 07:13:49 -04:00
David Robertson
15cdcf8f30 Remove dockerfile-pgtests (#12336) 2022-03-31 11:30:07 +01:00
David Robertson
5a32ec59b2 Ignore .envrc for direnv users (#12335) 2022-03-31 11:03:07 +01:00
David Robertson
9a3f1f5383 Nuke the tox packaging job (#12334) 2022-03-31 11:01:26 +01:00
Brendan Abolivier
f96b85eca8 Ensure the type of URL attributes is always str when matching against preview blacklist (#12333) 2022-03-31 11:49:49 +02:00
Patrick Cloke
c31c1091d4 Remove the unused and unstable /aggregations endpoint. (#12293) 
This endpoint was removed from MSC2675 before it was approved.
It is currently unspecified (even in any MSCs) and therefore subject to
removal. It is not implemented by any known clients.

This also changes the bundled aggregation format for `m.annotation`,
which previously included pagination tokens for the `/aggregations`
endpoint, which are no longer useful.
 2022-03-30 11:45:32 -04:00