Modify the tests to work with the new flow; clean up tests

This ended up ballooning much larger than intended :/ Part of the reason for the large amount of changes was the elimination of the _ServiceQueuer.enqueue_{event,ephemeral} methods and replacing them with ApplicationServiceScheduler.enqueue_for_appservice. That meant we had to change all of the tests that mock'd enqueue_{event,ephemeral} to instead look lower-level, at the calls to _TransactionController.send. Other than that, there was just the normal amount of adding checks that to_device_messages as an argument to enqueue_for_appservice is filled out correctly. The change to scheduler means that we don't have to keep adding another [] every time we check a call to _TransactionController.send.
fix type hint; make AppServiceTransaction instantiation a bit easier
2021-12-01 19:34:39 +00:00 · 2021-12-01 17:13:49 +00:00 · 2021-12-01 17:13:47 +00:00 · 2021-11-24 16:20:41 +00:00 · 2021-11-24 16:08:49 +00:00 · 2021-11-24 15:19:57 +00:00
245 changed files with 6034 additions and 1527 deletions
@@ -1,12 +1,13 @@
 ### Pull Request Checklist

-<!-- Please read CONTRIBUTING.md before submitting your pull request -->
+<!-- Please read https://matrix-org.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request -->

 * [ ] Pull request is based on the develop branch
-* [ ] Pull request includes a [changelog file](https://github.com/matrix-org/synapse/blob/master/CONTRIBUTING.md#changelog). The entry should:
+* [ ] Pull request includes a [changelog file](https://matrix-org.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should:
  - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
  - Use markdown where necessary, mostly for `code blocks`.
  - End with either a period (.) or an exclamation mark (!).
  - Start with a capital letter.
-* [ ] Pull request includes a [sign off](https://github.com/matrix-org/synapse/blob/master/CONTRIBUTING.md#sign-off)
-* [ ] Code style is correct (run the [linters](https://github.com/matrix-org/synapse/blob/master/CONTRIBUTING.md#code-style))
+* [ ] Pull request includes a [sign off](https://matrix-org.github.io/synapse/latest/development/contributing_guide.html#sign-off)
+* [ ] [Code style](https://matrix-org.github.io/synapse/latest/code_style.html) is correct
+  (run the [linters](https://matrix-org.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
@@ -1,8 +1,111 @@
-Synapse 1.46.0rc1 (2021-10-27)
+Synapse 1.47.0rc2 (2021-11-10)
 ==============================

+This fixes an issue with publishing the Debian packages for 1.47.0rc1.
+It is otherwise identical to 1.47.0rc1.
+
+
+Synapse 1.47.0rc1 (2021-11-09)
+==============================
+
+Deprecations and Removals
+-------------------------
+
+- The `user_may_create_room_with_invites` module callback is now deprecated. Please refer to the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1470) for more information. ([\#11206](https://github.com/matrix-org/synapse/issues/11206))
+- Remove deprecated admin API to delete rooms (`POST /_synapse/admin/v1/rooms/<room_id>/delete`). ([\#11213](https://github.com/matrix-org/synapse/issues/11213))
+
+
+Features
+--------
+
+- Advertise support for Client-Server API r0.6.1. ([\#11097](https://github.com/matrix-org/synapse/issues/11097))
+- Add search by room ID and room alias to the List Room admin API. ([\#11099](https://github.com/matrix-org/synapse/issues/11099))
+- Add an `on_new_event` third-party rules callback to allow Synapse modules to act after an event has been sent into a room. ([\#11126](https://github.com/matrix-org/synapse/issues/11126))
+- Add a module API method to update a user's membership in a room. ([\#11147](https://github.com/matrix-org/synapse/issues/11147))
+- Add metrics for thread pool usage. ([\#11178](https://github.com/matrix-org/synapse/issues/11178))
+- Support the stable room type field for [MSC3288](https://github.com/matrix-org/matrix-doc/pull/3288). ([\#11187](https://github.com/matrix-org/synapse/issues/11187))
+- Add a module API method to retrieve the current state of a room. ([\#11204](https://github.com/matrix-org/synapse/issues/11204))
+- Calculate a default value for `public_baseurl` based on `server_name`. ([\#11210](https://github.com/matrix-org/synapse/issues/11210))
+- Add support for serving `/.well-known/matrix/server` files, to redirect federation traffic to port 443. ([\#11211](https://github.com/matrix-org/synapse/issues/11211))
+- Add admin APIs to pause, start and check the status of background updates. ([\#11263](https://github.com/matrix-org/synapse/issues/11263))
+
+
+Bugfixes
+--------
+
+- Fix a long-standing bug which allowed hidden devices to receive to-device messages, resulting in unnecessary database bloat. ([\#10097](https://github.com/matrix-org/synapse/issues/10097))
+- Fix a long-standing bug where messages in the `device_inbox` table for deleted devices would persist indefinitely. Contributed by @dklimpel and @JohannesKleine. ([\#10969](https://github.com/matrix-org/synapse/issues/10969), [\#11212](https://github.com/matrix-org/synapse/issues/11212))
+- Do not accept events if a third-party rule `check_event_allowed` callback raises an exception. ([\#11033](https://github.com/matrix-org/synapse/issues/11033))
+- Fix long-standing bug where verification requests could fail in certain cases if a federation whitelist was in place but did not include your own homeserver. ([\#11129](https://github.com/matrix-org/synapse/issues/11129))
+- Allow an empty list of `state_events_at_start` to be sent when using the [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint and the author of the historical messages is already part of the current room state at the given `?prev_event_id`. ([\#11188](https://github.com/matrix-org/synapse/issues/11188))
+- Fix a bug introduced in Synapse 1.45.0 which prevented the `synapse_review_recent_signups` script from running. Contributed by @samuel-p. ([\#11191](https://github.com/matrix-org/synapse/issues/11191))
+- Delete `to_device` messages for hidden devices that will never be read, reducing database size. ([\#11199](https://github.com/matrix-org/synapse/issues/11199))
+- Fix a long-standing bug wherein a missing `Content-Type` header when downloading remote media would cause Synapse to throw an error. ([\#11200](https://github.com/matrix-org/synapse/issues/11200))
+- Fix a long-standing bug which could result in serialization errors and potentially duplicate transaction data when sending ephemeral events to application services. Contributed by @Fizzadar at Beeper. ([\#11207](https://github.com/matrix-org/synapse/issues/11207))
+- Fix a bug introduced in Synapse 1.35.0 which made it impossible to join rooms that return a `send_join` response containing floats. ([\#11217](https://github.com/matrix-org/synapse/issues/11217))
+- Fix long-standing bug where cross signing keys were not included in the response to `/r0/keys/query` the first time a remote user was queried. ([\#11234](https://github.com/matrix-org/synapse/issues/11234))
+- Fix a long-standing bug where all requests that read events from the database could get stuck as a result of losing the database connection. ([\#11240](https://github.com/matrix-org/synapse/issues/11240))
+- Fix a bug preventing Synapse from being rolled back to an earlier version when using workers. ([\#11255](https://github.com/matrix-org/synapse/issues/11255), [\#11276](https://github.com/matrix-org/synapse/issues/11276))
+- Fix a bug introduced in Synapse 1.37.1 which caused a remote event being processed by a worker to not get processed on restart if the worker was killed. ([\#11262](https://github.com/matrix-org/synapse/issues/11262))
+- Only allow old Element/Riot Android clients to send read receipts without a request body. All other clients must include a request body as required by the specification. Contributed by @rogersheu. ([\#11157](https://github.com/matrix-org/synapse/issues/11157))
+
+
+Updates to the Docker image
+---------------------------
+
+- Avoid changing user ID when started as a non-root user, and no explicit `UID` is set. ([\#11209](https://github.com/matrix-org/synapse/issues/11209))
+
+
+Improved Documentation
+----------------------
+
+- Improve example HAProxy config in the docs to properly handle HTTP `Host` headers with port information. This is required for federation over port 443 to work correctly. ([\#11128](https://github.com/matrix-org/synapse/issues/11128))
+- Add documentation for using Authentik as an OpenID Connect Identity Provider. Contributed by @samip5. ([\#11151](https://github.com/matrix-org/synapse/issues/11151))
+- Clarify lack of support for Windows. ([\#11198](https://github.com/matrix-org/synapse/issues/11198))
+- Improve code formatting and fix a few typos in docs. Contributed by @sumnerevans at Beeper. ([\#11221](https://github.com/matrix-org/synapse/issues/11221))
+- Add documentation for using LemonLDAP as an OpenID Connect Identity Provider. Contributed by @l00ptr. ([\#11257](https://github.com/matrix-org/synapse/issues/11257))
+
+
+Internal Changes
+----------------
+
+- Add type annotations for the `log_function` decorator. ([\#10943](https://github.com/matrix-org/synapse/issues/10943))
+- Add type hints to `synapse.events`. ([\#11098](https://github.com/matrix-org/synapse/issues/11098))
+- Remove and document unnecessary `RoomStreamToken` checks in application service ephemeral event code. ([\#11137](https://github.com/matrix-org/synapse/issues/11137))
+- Add type hints so that `synapse.http` passes `mypy` checks. ([\#11164](https://github.com/matrix-org/synapse/issues/11164))
+- Update scripts to pass Shellcheck lints. ([\#11166](https://github.com/matrix-org/synapse/issues/11166))
+- Add knock information in admin export. Contributed by Rafael Gonçalves. ([\#11171](https://github.com/matrix-org/synapse/issues/11171))
+- Add tests to check that `ClientIpStore.get_last_client_ip_by_device` and `get_user_ip_and_agents` combine database and in-memory data correctly. ([\#11179](https://github.com/matrix-org/synapse/issues/11179))
+- Refactor `Filter` to check different fields depending on the data type. ([\#11194](https://github.com/matrix-org/synapse/issues/11194))
+- Improve type hints for the relations datastore. ([\#11205](https://github.com/matrix-org/synapse/issues/11205))
+- Replace outdated links in the pull request checklist with links to the rendered documentation. ([\#11225](https://github.com/matrix-org/synapse/issues/11225))
+- Fix a bug in unit test `test_block_room_and_not_purge`. ([\#11226](https://github.com/matrix-org/synapse/issues/11226))
+- In `ObservableDeferred`, run observers in the order they were registered. ([\#11229](https://github.com/matrix-org/synapse/issues/11229))
+- Minor speed up to start up times and getting updates for groups by adding missing index to `local_group_updates.stream_id`. ([\#11231](https://github.com/matrix-org/synapse/issues/11231))
+- Add `twine` and `towncrier` as dev dependencies, as they're used by the release script. ([\#11233](https://github.com/matrix-org/synapse/issues/11233))
+- Allow `stream_writers.typing` config to be a list of one worker. ([\#11237](https://github.com/matrix-org/synapse/issues/11237))
+- Remove debugging statement in tests. ([\#11239](https://github.com/matrix-org/synapse/issues/11239))
+- Fix [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) historical messages backfilling in random order on remote homeservers. ([\#11244](https://github.com/matrix-org/synapse/issues/11244))
+- Add an additional test for the `cachedList` method decorator. ([\#11246](https://github.com/matrix-org/synapse/issues/11246))
+- Make minor correction to the type of `auth_checkers` callbacks. ([\#11253](https://github.com/matrix-org/synapse/issues/11253))
+- Clean up trivial aspects of the Debian package build tooling. ([\#11269](https://github.com/matrix-org/synapse/issues/11269), [\#11273](https://github.com/matrix-org/synapse/issues/11273))
+- Blacklist new SyTest that checks that key uploads are valid pending the validation being implemented in Synapse. ([\#11270](https://github.com/matrix-org/synapse/issues/11270))
+
+
+Synapse 1.46.0 (2021-11-02)
+===========================
+
 The cause of the [performance regression affecting Synapse 1.44](https://github.com/matrix-org/synapse/issues/11049) has been identified and fixed. ([\#11177](https://github.com/matrix-org/synapse/issues/11177))

+Bugfixes
+--------
+
+- Fix a bug introduced in v1.46.0rc1 where URL previews of some XML documents would fail. ([\#11196](https://github.com/matrix-org/synapse/issues/11196))
+
+
+Synapse 1.46.0rc1 (2021-10-27)
+==============================
+
 Features
 --------

@@ -8575,14 +8678,14 @@ General:

 Federation:

-   Add key distribution mechanisms for fetching public keys of unavailable remote home servers. See [Retrieving Server Keys](https://github.com/matrix-org/matrix-doc/blob/6f2698/specification/30_server_server_api.rst#retrieving-server-keys) in the spec.
+-   Add key distribution mechanisms for fetching public keys of unavailable remote homeservers. See [Retrieving Server Keys](https://github.com/matrix-org/matrix-doc/blob/6f2698/specification/30_server_server_api.rst#retrieving-server-keys) in the spec.

 Configuration:

 -   Add support for multiple config files.
 -   Add support for dictionaries in config files.
 -   Remove support for specifying config options on the command line, except for:
-    -   `--daemonize` - Daemonize the home server.
+    -   `--daemonize` - Daemonize the homeserver.
    -   `--manhole` - Turn on the twisted telnet manhole service on the given port.
    -   `--database-path` - The path to a sqlite database to use.
    -   `--verbose` - The verbosity level.
@@ -8787,7 +8890,7 @@ This version adds support for using a TURN server. See docs/turn-howto.rst on ho
 Homeserver:

 -   Add support for redaction of messages.
-   Fix bug where inviting a user on a remote home server could take up to 20-30s.
+-   Fix bug where inviting a user on a remote homeserver could take up to 20-30s.
 -   Implement a get current room state API.
 -   Add support specifying and retrieving turn server configuration.

@@ -8877,7 +8980,7 @@ Changes in synapse 0.2.3 (2014-09-12)

 Homeserver:

-   Fix bug where we stopped sending events to remote home servers if a user from that home server left, even if there were some still in the room.
+-   Fix bug where we stopped sending events to remote homeservers if a user from that homeserver left, even if there were some still in the room.
 -   Fix bugs in the state conflict resolution where it was incorrectly rejecting events.

 Webclient:
@@ -1 +0,0 @@
-Fix a long-standing bug which allowed hidden devices to receive to-device messages, resulting in unnecessary database bloat.
@@ -1 +0,0 @@
-Add type annotations for the `log_function` decorator.
@@ -1 +0,0 @@
-Fix a long-standing bug where messages in the `device_inbox` table for deleted devices would persist indefinitely. Contributed by @dklimpel and @JohannesKleine.
@@ -1 +0,0 @@
-Do not accept events if a third-party rule module API callback raises an exception.
@@ -1 +0,0 @@
-Advertise support for Client-Server API r0.6.1.
@@ -1 +0,0 @@
-Add an `on_new_event` third-party rules callback to allow Synapse modules to act after an event has been sent into a room.
@@ -1 +0,0 @@
-Improve example HAProxy config in the docs to properly handle host headers with port information. This is required for federation over port 443 to work correctly.
@@ -1 +0,0 @@
-Fix long-standing bug where verification requests could fail in certain cases if whitelist was in place but did not include your own homeserver.
@@ -1 +0,0 @@
-Add a module API method to update a user's membership in a room.
@@ -1 +0,0 @@
-Add documentation for using Authentik as an OpenID Connect Identity Provider. Contributed by @samip5.
@@ -1 +0,0 @@
-Add type hints so that `synapse.http` passes `mypy` checks.
@@ -1 +0,0 @@
-Update scripts to pass Shellcheck lints.
@@ -1 +0,0 @@
-Add knock information in admin export. Contributed by Rafael Gonçalves.
@@ -1 +0,0 @@
-Add metrics for thread pool usage.
@@ -1 +0,0 @@
-Add tests to check that `ClientIpStore.get_last_client_ip_by_device` and `get_user_ip_and_agents` combine database and in-memory data correctly.
@@ -1 +0,0 @@
-Support the stable room type field for [MSC3288](https://github.com/matrix-org/matrix-doc/pull/3288).
@@ -1 +0,0 @@
-Fix a bug introduced in Synapse 1.45.0 which prevented the `synapse_review_recent_signups` script from running. Contributed by @samuel-p.
@@ -1 +0,0 @@
-Refactor `Filter` to check different fields depending on the data type.
@@ -1 +0,0 @@
-Clarify lack of support for Windows.
@@ -1 +0,0 @@
-Add a module API method to retrieve the current state of a room.
@@ -1 +0,0 @@
-Improve type hints for the relations datastore.
@@ -1 +0,0 @@
-The `user_may_create_room_with_invites` module callback is now deprecated. Please refer to the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1470) for more information.
@@ -1 +0,0 @@
-Avoid changing userid when started as a non-root user, and no explicit `UID` is set.
@@ -1 +0,0 @@
-Add support for serving `/.well-known/matrix/server` files, to redirect federation traffic to port 443.
@@ -1 +0,0 @@
-Fix a long-standing bug where messages in the `device_inbox` table for deleted devices would persist indefinitely. Contributed by @dklimpel and @JohannesKleine.
@@ -1 +0,0 @@
-Remove deprecated admin API to delete rooms (`POST /_synapse/admin/v1/rooms/<room_id>/delete`).
@@ -0,0 +1 @@
+Add experimental support for sending to-device messages to application services, as specified by [MSC2409](https://github.com/matrix-org/matrix-doc/pull/2409). Disabled by default.
@@ -1 +0,0 @@
-Improve code formatting and fix a few typos in docs. Contributed by @sumnerevans at Beeper.
@@ -0,0 +1 @@
+Add a new version of delete room admin API `DELETE /_synapse/admin/v2/rooms/<room_id>` to run it in background. Contributed by @dklimpel.
@@ -1 +0,0 @@
-Fix a bug in unit test `test_block_room_and_not_purge`.
@@ -0,0 +1 @@
+Allow the admin [Delete Room API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#delete-room-api) to block a room without the need to join it.
@@ -0,0 +1,2 @@
+Fix a long-standing bug wherein display names or avatar URLs containing null bytes cause an internal server error
+when stored in the DB.
@@ -0,0 +1 @@
+Support filtering by relation senders & types per [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440).
@@ -0,0 +1 @@
+Split out federated PDU retrieval function into a non-cached version.
@@ -0,0 +1 @@
+Clean up code relating to to-device messages and sending ephemeral events to application services.
@@ -0,0 +1 @@
+Fix a small typo in the error response when a relation type other than 'm.annotation' is passed to `GET /rooms/{room_id}/aggregations/{event_id}`.
@@ -0,0 +1 @@
+Drop unused db tables `room_stats_historical` and `user_stats_historical`.
@@ -0,0 +1 @@
+Suggest users of the Debian packages add configuration to `/etc/matrix-synapse/conf.d/` to prevent, upon upgrade, being asked to choose between their configuration and the maintainer's.
@@ -0,0 +1 @@
+Require all files in synapse/ and tests/ to pass mypy unless specifically excluded.
@@ -0,0 +1 @@
+Require all files in synapse/ and tests/ to pass mypy unless specifically excluded.
@@ -0,0 +1 @@
+Fix typo in the word `available` and fix HTTP method (should be `GET`) for the `username_available` admin API. Contributed by Stanislav Motylkov.
@@ -0,0 +1 @@
+Add missing type hints to `synapse.app`.
@@ -0,0 +1 @@
+Fix a long-standing bug where uploading extremely thin images (e.g. 1000x1) would fail. Contributed by @Neeeflix.
@@ -0,0 +1 @@
+Remove unused parameters on `FederationEventHandler._check_event_auth`.
@@ -0,0 +1 @@
+Add type hints to `synapse._scripts`.
@@ -0,0 +1 @@
+Add Single Sign-On, SAML and CAS pages to the documentation.
@@ -0,0 +1 @@
+Fix an issue which prevented the 'remove deleted devices from device_inbox column' background process from running when updating from a recent Synapse version.
@@ -0,0 +1 @@
+Add type hints to storage classes.
@@ -0,0 +1 @@
+Add type hints to storage classes.
@@ -0,0 +1 @@
+Add type hints to storage classes.
@@ -0,0 +1 @@
+Add type hints to storage classes.
@@ -0,0 +1 @@
+Add type hints to storage classes.
@@ -0,0 +1 @@
+Add type hints to storage classes.
@@ -0,0 +1 @@
+Add type hints to storage classes.
@@ -0,0 +1 @@
+Changed the word 'Home server' as one word 'homeserver' in documentation.
@@ -0,0 +1 @@
+Add type hints to `synapse.util`.
@@ -0,0 +1 @@
+Add type hints to storage classes.
@@ -0,0 +1 @@
+Improve type annotations in Synapse's test suite.
@@ -0,0 +1 @@
+Test that room alias deletion works as intended.
@@ -0,0 +1 @@
+Improve type annotations in Synapse's test suite.
@@ -0,0 +1 @@
+Add type hints to storage classes.
@@ -0,0 +1 @@
+Support the stable version of [MSC2778](https://github.com/matrix-org/matrix-doc/pull/2778): the `m.login.application_service` login type. Contributed by @tulir.
@@ -0,0 +1 @@
+Add type hints to storage classes.
@@ -0,0 +1 @@
+Add type hints to storage classes.
@@ -0,0 +1 @@
+Add admin API to un-shadow-ban a user.
@@ -40,6 +40,7 @@ dh_virtualenv \
    --upgrade-pip \
    --preinstall="lxml" \
    --preinstall="mock" \
+    --preinstall="wheel" \
    --extra-pip-arg="--no-cache-dir" \
    --extra-pip-arg="--compile" \
    --extras="all,systemd,test"
@@ -1,8 +1,26 @@
-matrix-synapse-py3 (1.47.0+nmu1) UNRELEASED; urgency=medium
+matrix-synapse-py3 (1.47.0~rc2) stable; urgency=medium

+  [ Dan Callahan ]
  * Update scripts to pass Shellcheck lints.
+  * Remove unused Vagrant scripts from debian/ directory.
+  * Allow building Debian packages for any architecture, not just amd64.
+  * Preinstall the "wheel" package when building virtualenvs.
+  * Do not error if /etc/default/matrix-synapse is missing.

- -- root <root@cae79a6e79d7>  Fri, 22 Oct 2021 22:20:31 +0000
+  [ Synapse Packaging team ]
+  * New synapse release 1.47.0~rc2.
+
+ -- Synapse Packaging team <packages@matrix.org>  Wed, 10 Nov 2021 09:41:01 +0000
+
+matrix-synapse-py3 (1.46.0) stable; urgency=medium
+
+  [ Richard van der Hoff ]
+  * Compress debs with xz, to fix incompatibility of impish debs with reprepro.
+
+  [ Synapse Packaging team ]
+  * New synapse release 1.46.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 02 Nov 2021 13:22:53 +0000

 matrix-synapse-py3 (1.46.0~rc1) stable; urgency=medium

@@ -19,7 +19,7 @@ Standards-Version: 3.9.8
 Homepage: https://github.com/matrix-org/synapse

 Package: matrix-synapse-py3
-Architecture: amd64
+Architecture: any
 Provides: matrix-synapse
 Conflicts:
 matrix-synapse (<< 0.34.0.1-0matrix2),
@@ -5,7 +5,7 @@ Description=Synapse Matrix homeserver
 Type=notify
 User=matrix-synapse
 WorkingDirectory=/var/lib/matrix-synapse
-EnvironmentFile=/etc/default/matrix-synapse
+EnvironmentFile=-/etc/default/matrix-synapse
 ExecStartPre=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys
 ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/
 ExecReload=/bin/kill -HUP $MAINPID
@@ -51,5 +51,11 @@ override_dh_shlibdeps:
 override_dh_virtualenv:
 	./debian/build_virtualenv

+override_dh_builddeb:
+        # force the compression to xzip, to stop dpkg-deb on impish defaulting to zstd
+        # (which requires reprepro 5.3.0-1.3, which is currently only in 'experimental' in Debian:
+        # https://metadata.ftp-master.debian.org/changelogs/main/r/reprepro/reprepro_5.3.0-1.3_changelog)
+	dh_builddeb -- -Zxz
+
 %:
 	dh $@ --with python-virtualenv
@@ -1,2 +0,0 @@
-.vagrant
-*.log
@@ -1,24 +0,0 @@
-#!/bin/bash
-#
-# provisioning script for vagrant boxes for testing the matrix-synapse debs.
-#
-# Will install the most recent matrix-synapse-py3 deb for this platform from
-# the /debs directory.
-
-set -e
-
-apt-get update
-apt-get install -y lsb-release
-
-deb=$(find /debs -name "matrix-synapse-py3_*+$(lsb_release -cs)*.deb" | sort | tail -n1)
-
-debconf-set-selections <<EOF
-matrix-synapse matrix-synapse/report-stats boolean false
-matrix-synapse matrix-synapse/server-name string localhost:18448
-EOF
-
-dpkg -i "$deb"
-
-sed -i -e 's/port: 8448$/port: 18448/; s/port: 8008$/port: 18008' /etc/matrix-synapse/homeserver.yaml
-echo 'registration_shared_secret: secret' >> /etc/matrix-synapse/homeserver.yaml
-systemctl restart matrix-synapse
@@ -1,13 +0,0 @@
-# -*- mode: ruby -*-
-# vi: set ft=ruby :
-
-ver = `cd ../../..; dpkg-parsechangelog -S Version`.strip()
-
-Vagrant.configure("2") do |config|
-  config.vm.box = "debian/stretch64"
-
-  config.vm.synced_folder ".", "/vagrant", disabled: true
-  config.vm.synced_folder "../../../../debs", "/debs", type: "nfs"
-
-  config.vm.provision "shell", path: "../provision.sh"
-end
@@ -1,10 +0,0 @@
-# -*- mode: ruby -*-
-# vi: set ft=ruby :
-
-Vagrant.configure("2") do |config|
-  config.vm.box = "ubuntu/xenial64"
-
-  config.vm.synced_folder ".", "/vagrant", disabled: true
-  config.vm.synced_folder "../../../../debs", "/debs"
-  config.vm.provision "shell", path: "../provision.sh"
-end
@@ -23,10 +23,10 @@
    - [Structured Logging](structured_logging.md)
    - [Templates](templates.md)
    - [User Authentication](usage/configuration/user_authentication/README.md)
-      - [Single-Sign On]()
+      - [Single-Sign On](usage/configuration/user_authentication/single_sign_on/README.md)
        - [OpenID Connect](openid.md)
-        - [SAML]()
-        - [CAS]()
+        - [SAML](usage/configuration/user_authentication/single_sign_on/saml.md)
+        - [CAS](usage/configuration/user_authentication/single_sign_on/cas.md)
        - [SSO Mapping Providers](sso_mapping_providers.md)
      - [Password Auth Providers](password_auth_providers.md)
      - [JSON Web Tokens](jwt.md)
@@ -51,6 +51,7 @@
  - [Administration](usage/administration/README.md)
    - [Admin API](usage/administration/admin_api/README.md)
      - [Account Validity](admin_api/account_validity.md)
+      - [Background Updates](usage/administration/admin_api/background_updates.md)
      - [Delete Group](admin_api/delete_group.md)
      - [Event Reports](admin_api/event_reports.md)
      - [Media](admin_api/media_admin_api.md)
@@ -70,6 +70,8 @@ This API returns a JSON body like the following:

 The status will be one of `active`, `complete`, or `failed`.

+If `status` is `failed` there will be a string `error` with the error message.
+
 ## Reclaim disk space (Postgres)

 To reclaim the disk space and return it to the operating system, you need to run
@@ -4,6 +4,9 @@
 - [Room Members API](#room-members-api)
 - [Room State API](#room-state-api)
 - [Delete Room API](#delete-room-api)
+  * [Version 1 (old version)](#version-1-old-version)
+  * [Version 2 (new version)](#version-2-new-version)
+  * [Status of deleting rooms](#status-of-deleting-rooms)
  * [Undoing room shutdowns](#undoing-room-shutdowns)
 - [Make Room Admin API](#make-room-admin-api)
 - [Forward Extremities Admin API](#forward-extremities-admin-api)
@@ -38,9 +41,14 @@ The following query parameters are available:
  - `history_visibility` - Rooms are ordered alphabetically by visibility of history of the room.
  - `state_events` - Rooms are ordered by number of state events. Largest to smallest.
 * `dir` - Direction of room order. Either `f` for forwards or `b` for backwards. Setting
-          this value to `b` will reverse the above sort order. Defaults to `f`.
-* `search_term` - Filter rooms by their room name. Search term can be contained in any
-                  part of the room name. Defaults to no filtering.
+  this value to `b` will reverse the above sort order. Defaults to `f`.
+* `search_term` - Filter rooms by their room name, canonical alias and room id.
+  Specifically, rooms are selected if the search term is contained in
+  - the room's name,
+  - the local part of the room's canonical alias, or
+  - the complete (local and server part) room's id (case sensitive).
+
+  Defaults to no filtering.

 **Response**

@@ -380,7 +388,7 @@ A response body like the following is returned:

 # Delete Room API

-The Delete Room admin API allows server admins to remove rooms from server
+The Delete Room admin API allows server admins to remove rooms from the server
 and block these rooms.

 Shuts down a room. Moves all local users and room aliases automatically to a
@@ -391,18 +399,33 @@ The new room will be created with the user specified by the `new_room_user_id` p
 as room administrator and will contain a message explaining what happened. Users invited
 to the new room will have power level `-10` by default, and thus be unable to speak.

-If `block` is `True` it prevents new joins to the old room.
+If `block` is `true`, users will be prevented from joining the old room.
+This option can in [Version 1](#version-1-old-version) also be used to pre-emptively
+block a room, even if it's unknown to this homeserver. In this case, the room will be
+blocked, and no further action will be taken. If `block` is `false`, attempting to
+delete an unknown room is invalid and will be rejected as a bad request.

 This API will remove all trace of the old room from your database after removing
 all local users. If `purge` is `true` (the default), all traces of the old room will
 be removed from your database after removing all local users. If you do not want
 this to happen, set `purge` to `false`.
-Depending on the amount of history being purged a call to the API may take
+Depending on the amount of history being purged, a call to the API may take
 several minutes or longer.

 The local server will only have the power to move local user and room aliases to
 the new room. Users on other servers will be unaffected.

+To use it, you will need to authenticate by providing an ``access_token`` for a
+server admin: see [Admin API](../usage/administration/admin_api).
+
+## Version 1 (old version)
+
+This version works synchronously. That means you only get the response once the server has
+finished the action, which may take a long time. If you request the same action
+a second time, and the server has not finished the first one, the second request will block.
+This is fixed in version 2 of this API. The parameters are the same in both APIs.
+This API will become deprecated in the future.
+
 The API is:

 ```
@@ -421,9 +444,6 @@ with a body of:
 }
 ```

-To use it, you will need to authenticate by providing an ``access_token`` for a
-server admin: see [Admin API](../usage/administration/admin_api).
-
 A response body like the following is returned:

 ```json
@@ -440,6 +460,44 @@ A response body like the following is returned:
 }
 ```

+The parameters and response values have the same format as
+[version 2](#version-2-new-version) of the API.
+
+## Version 2 (new version)
+
+**Note**: This API is new, experimental and "subject to change".
+
+This version works asynchronously, meaning you get the response from server immediately
+while the server works on that task in background. You can then request the status of the action
+to check if it has completed.
+
+The API is:
+
+```
+DELETE /_synapse/admin/v2/rooms/<room_id>
+```
+
+with a body of:
+
+```json
+{
+    "new_room_user_id": "@someuser:example.com",
+    "room_name": "Content Violation Notification",
+    "message": "Bad Room has been shutdown due to content violations on this server. Please review our Terms of Service.",
+    "block": true,
+    "purge": true
+}
+```
+
+The API starts the shut down and purge running, and returns immediately with a JSON body with
+a purge id:
+
+```json
+{
+    "delete_id": "<opaque id>"
+}
+```
+
 **Parameters**

 The following parameters should be set in the URL:
@@ -459,8 +517,10 @@ The following JSON body parameters are available:
              `new_room_user_id` in the new room. Ideally this will clearly convey why the
               original room was shut down. Defaults to `Sharing illegal content on this server
               is not permitted and rooms in violation will be blocked.`
-* `block` - Optional. If set to `true`, this room will be added to a blocking list, preventing
-            future attempts to join the room. Defaults to `false`.
+* `block` - Optional. If set to `true`, this room will be added to a blocking list,
+            preventing future attempts to join the room. Rooms can be blocked
+            even if they're not yet known to the homeserver (only with
+            [Version 1](#version-1-old-version) of the API). Defaults to `false`.
 * `purge` - Optional. If set to `true`, it will remove all traces of the room from your database.
            Defaults to `true`.
 * `force_purge` - Optional, and ignored unless `purge` is `true`. If set to `true`, it
@@ -470,16 +530,124 @@ The following JSON body parameters are available:

 The JSON body must not be empty. The body must be at least `{}`.

-**Response**
+## Status of deleting rooms
+
+**Note**: This API is new, experimental and "subject to change".
+
+It is possible to query the status of the background task for deleting rooms.
+The status can be queried up to 24 hours after completion of the task,
+or until Synapse is restarted (whichever happens first).
+
+### Query by `room_id`
+
+With this API you can get the status of all active deletion tasks, and all those completed in the last 24h,
+for the given `room_id`.
+
+The API is:
+
+```
+GET /_synapse/admin/v2/rooms/<room_id>/delete_status
+```
+
+A response body like the following is returned:
+
+```json
+{
+    "results": [
+        {
+            "delete_id": "delete_id1",
+            "status": "failed",
+            "error": "error message",
+            "shutdown_room": {
+                "kicked_users": [],
+                "failed_to_kick_users": [],
+                "local_aliases": [],
+                "new_room_id": null
+            }
+        }, {
+            "delete_id": "delete_id2",
+            "status": "purging",
+            "shutdown_room": {
+                "kicked_users": [
+                    "@foobar:example.com"
+                ],
+                "failed_to_kick_users": [],
+                "local_aliases": [
+                    "#badroom:example.com",
+                    "#evilsaloon:example.com"
+                ],
+                "new_room_id": "!newroomid:example.com"
+            }
+        }
+    ]
+}
+```
+
+**Parameters**
+
+The following parameters should be set in the URL:
+
+* `room_id` - The ID of the room.
+
+### Query by `delete_id`
+
+With this API you can get the status of one specific task by `delete_id`.
+
+The API is:
+
+```
+GET /_synapse/admin/v2/rooms/delete_status/<delete_id>
+```
+
+A response body like the following is returned:
+
+```json
+{
+    "status": "purging",
+    "shutdown_room": {
+        "kicked_users": [
+            "@foobar:example.com"
+        ],
+        "failed_to_kick_users": [],
+        "local_aliases": [
+            "#badroom:example.com",
+            "#evilsaloon:example.com"
+        ],
+        "new_room_id": "!newroomid:example.com"
+    }
+}
+```
+
+**Parameters**
+
+The following parameters should be set in the URL:
+
+* `delete_id` - The ID for this delete.
+
+### Response

 The following fields are returned in the JSON response body:

-* `kicked_users` - An array of users (`user_id`) that were kicked.
-* `failed_to_kick_users` - An array of users (`user_id`) that that were not kicked.
-* `local_aliases` - An array of strings representing the local aliases that were migrated from
-                    the old room to the new.
-* `new_room_id` - A string representing the room ID of the new room.
-
+- `results` - An array of objects, each containing information about one task.
+  This field is omitted from the result when you query by `delete_id`.
+  Task objects contain the following fields:
+  - `delete_id` - The ID for this purge if you query by `room_id`.
+  - `status` - The status will be one of:
+    - `shutting_down` - The process is removing users from the room.
+    - `purging` - The process is purging the room and event data from database.
+    - `complete` - The process has completed successfully.
+    - `failed` - The process is aborted, an error has occurred.
+  - `error` - A string that shows an error message if `status` is `failed`.
+    Otherwise this field is hidden.
+  - `shutdown_room` - An object containing information about the result of shutting down the room.
+    *Note:* The result is shown after removing the room members.
+    The delete process can still be running. Please pay attention to the `status`.
+    - `kicked_users` - An array of users (`user_id`) that were kicked.
+    - `failed_to_kick_users` - An array of users (`user_id`) that that were not kicked.
+    - `local_aliases` - An array of strings representing the local aliases that were
+      migrated from the old room to the new.
+    - `new_room_id` - A string representing the room ID of the new room, or `null` if
+      no such room was created.

 ## Undoing room deletions

@@ -948,7 +948,7 @@ The following fields are returned in the JSON response body:
 See also the
 [Client-Server API Spec on pushers](https://matrix.org/docs/spec/client_server/latest#get-matrix-client-r0-pushers).

-## Shadow-banning users
+## Controlling whether a user is shadow-banned

 Shadow-banning is a useful tool for moderating malicious or egregiously abusive users.
 A shadow-banned users receives successful responses to their client-server API requests,
@@ -961,16 +961,22 @@ or broken behaviour for the client. A shadow-banned user will not receive any
 notification and it is generally more appropriate to ban or kick abusive users.
 A shadow-banned user will be unable to contact anyone on the server.

-The API is:
+To shadow-ban a user the API is:

 ```
 POST /_synapse/admin/v1/users/<user_id>/shadow_ban
 ```

+To un-shadow-ban a user the API is:
+
+```
+DELETE /_synapse/admin/v1/users/<user_id>/shadow_ban
+```
+
 To use it, you will need to authenticate by providing an `access_token` for a
 server admin: [Admin API](../usage/administration/admin_api)

-An empty JSON dict is returned.
+An empty JSON dict is returned in both cases.

 **Parameters**

@@ -1107,7 +1113,7 @@ This endpoint will work even if registration is disabled on the server, unlike
 The API is:

 ```
-POST /_synapse/admin/v1/username_availabile?username=$localpart
+GET /_synapse/admin/v1/username_available?username=$localpart
 ```

 The request and response format is the same as the [/_matrix/client/r0/register/available](https://matrix.org/docs/spec/client_server/r0.6.0#get-matrix-client-r0-register-available) API.
@@ -7,7 +7,7 @@

 ## Server to Server Stack

-To use the server to server stack, home servers should only need to
+To use the server to server stack, homeservers should only need to
 interact with the Messaging layer.

 The server to server side of things is designed into 4 distinct layers:
@@ -23,7 +23,7 @@ Server with a domain specific API.

 1. **Messaging Layer**

-    This is what the rest of the Home Server hits to send messages, join rooms,
+    This is what the rest of the homeserver hits to send messages, join rooms,
    etc. It also allows you to register callbacks for when it get's notified by
    lower levels that e.g. a new message has been received.

@@ -45,7 +45,7 @@ Server with a domain specific API.

    For incoming PDUs, it has to check the PDUs it references to see
    if we have missed any. If we have go and ask someone (another
-    home server) for it.
+    homeserver) for it.

 3. **Transaction Layer**

@@ -11,7 +11,7 @@ registered by using the Module API's `register_password_auth_provider_callbacks`
 _First introduced in Synapse v1.46.0_

 ```python
- auth_checkers: Dict[Tuple[str,Tuple], Callable]
+auth_checkers: Dict[Tuple[str, Tuple[str, ...]], Callable]
 ```

 A dict mapping from tuples of a login type identifier (such as `m.login.password`) and a
@@ -22,6 +22,7 @@ such as [Github][github-idp].
 [google-idp]: https://developers.google.com/identity/protocols/oauth2/openid-connect
 [auth0]: https://auth0.com/
 [authentik]: https://goauthentik.io/
+[lemonldap]: https://lemonldap-ng.org/
 [okta]: https://www.okta.com/
 [dex-idp]: https://github.com/dexidp/dex
 [keycloak-idp]: https://www.keycloak.org/docs/latest/server_admin/#sso-protocols
@@ -243,6 +244,43 @@ oidc_providers:
        display_name_template: "{{ user.preferred_username|capitalize }}" # TO BE FILLED: If your users have names in Authentik and you want those in Synapse, this should be replaced with user.name|capitalize.
 ```

+### LemonLDAP
+
+[LemonLDAP::NG][lemonldap] is an open-source IdP solution.
+
+1. Create an OpenID Connect Relying Parties in LemonLDAP::NG
+2. The parameters are:
+- Client ID under the basic menu of the new Relying Parties (`Options > Basic >
+  Client ID`)
+- Client secret (`Options > Basic > Client secret`)
+- JWT Algorithm: RS256 within the security menu of the new Relying Parties
+  (`Options > Security > ID Token signature algorithm` and `Options > Security >
+  Access Token signature algorithm`)
+- Scopes: OpenID, Email and Profile
+- Allowed redirection addresses for login (`Options > Basic > Allowed
+  redirection addresses for login` ) :
+  `[synapse public baseurl]/_synapse/client/oidc/callback`
+
+Synapse config:
+```yaml
+oidc_providers:
+  - idp_id: lemonldap
+    idp_name: lemonldap
+    discover: true
+    issuer: "https://auth.example.org/" # TO BE FILLED: replace with your domain
+    client_id: "your client id" # TO BE FILLED
+    client_secret: "your client secret" # TO BE FILLED
+    scopes:
+      - "openid"
+      - "profile"
+      - "email"
+    user_mapping_provider:
+      config:
+        localpart_template: "{{ user.preferred_username }}}"
+        # TO BE FILLED: If your users have names in LemonLDAP::NG and you want those in Synapse, this should be replaced with user.name|capitalize or any valid filter.
+        display_name_template: "{{ user.preferred_username|capitalize }}"
+```
+
 ### GitHub

 [GitHub][github-idp] is a bit special as it is not an OpenID Connect compliant provider, but
@@ -91,6 +91,8 @@ pid_file: DATADIR/homeserver.pid
 # Otherwise, it should be the URL to reach Synapse's client HTTP listener (see
 # 'listeners' below).
 #
+# Defaults to 'https://<server_name>/'.
+#
 #public_baseurl: https://example.com/

 # Uncomment the following to tell other servers to send federation traffic on
@@ -1265,7 +1267,7 @@ oembed:
 # in on this server.
 #
 # (By default, no suggestion is made, so it is left up to the client.
-# This setting is ignored unless public_baseurl is also set.)
+# This setting is ignored unless public_baseurl is also explicitly set.)
 #
 #default_identity_server: https://matrix.org

@@ -1290,8 +1292,6 @@ oembed:
 # by the Matrix Identity Service API specification:
 # https://matrix.org/docs/spec/identity_service/latest
 #
-# If a delegate is specified, the config option public_baseurl must also be filled out.
-#
 account_threepid_delegates:
    #email: https://example.com     # Delegate email sending to example.com
    #msisdn: http://localhost:8090  # Delegate SMS sending to this local process
@@ -1981,11 +1981,10 @@ sso:
    # phishing attacks from evil.site. To avoid this, include a slash after the
    # hostname: "https://my.client/".
    #
-    # If public_baseurl is set, then the login fallback page (used by clients
-    # that don't natively support the required login flows) is whitelisted in
-    # addition to any URLs in this list.
+    # The login fallback page (used by clients that don't natively support the
+    # required login flows) is whitelisted in addition to any URLs in this list.
    #
-    # By default, this list is empty.
+    # By default, this list contains only the login fallback page.
    #
    #client_whitelist:
    #  - https://riot.im/develop
@@ -76,6 +76,12 @@ The fingerprint of the repository signing key (as shown by `gpg
 /usr/share/keyrings/matrix-org-archive-keyring.gpg`) is
 `AAF9AE843A7584B5A3E4CD2BCF45A512DE2DA058`.

+When installing with Debian packages, you might prefer to place files in
+`/etc/matrix-synapse/conf.d/` to override your configuration without editing
+the main configuration file at `/etc/matrix-synapse/homeserver.yaml`.
+By doing that, you won't be asked if you want to replace your configuration
+file when you upgrade the Debian package to a later version.
+
 ##### Downstream Debian packages

 We do not recommend using the packages from the default Debian `buster`
@@ -15,7 +15,7 @@ Type=notify
 NotifyAccess=main
 User=matrix-synapse
 WorkingDirectory=/var/lib/matrix-synapse
-EnvironmentFile=/etc/default/matrix-synapse
+EnvironmentFile=-/etc/default/matrix-synapse
 ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.generic_worker --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --config-path=/etc/matrix-synapse/workers/%i.yaml
 ExecReload=/bin/kill -HUP $MAINPID
 Restart=always
@@ -10,7 +10,7 @@ Type=notify
 NotifyAccess=main
 User=matrix-synapse
 WorkingDirectory=/var/lib/matrix-synapse
-EnvironmentFile=/etc/default/matrix-synapse
+EnvironmentFile=-/etc/default/matrix-synapse
 ExecStartPre=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys
 ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/
 ExecReload=/bin/kill -HUP $MAINPID
@@ -1,12 +1,12 @@
 # Overview

-This document explains how to enable VoIP relaying on your Home Server with
+This document explains how to enable VoIP relaying on your homeserver with
 TURN.

-The synapse Matrix Home Server supports integration with TURN server via the
+The synapse Matrix homeserver supports integration with TURN server via the
 [TURN server REST API](<https://tools.ietf.org/html/draft-uberti-behave-turn-rest-00>). This
-allows the Home Server to generate credentials that are valid for use on the
-TURN server through the use of a secret shared between the Home Server and the
+allows the homeserver to generate credentials that are valid for use on the
+TURN server through the use of a secret shared between the homeserver and the
 TURN server.

 The following sections describe how to install [coturn](<https://github.com/coturn/coturn>) (which implements the TURN REST API) and integrate it with synapse.
@@ -165,18 +165,18 @@ This will install and start a systemd service called `coturn`.

 ## Synapse setup

-Your home server configuration file needs the following extra keys:
+Your homeserver configuration file needs the following extra keys:

 1.  "`turn_uris`": This needs to be a yaml list of public-facing URIs
    for your TURN server to be given out to your clients. Add separate
    entries for each transport your TURN server supports.
 2.  "`turn_shared_secret`": This is the secret shared between your
-    Home server and your TURN server, so you should set it to the same
+    homeserver and your TURN server, so you should set it to the same
    string you used in turnserver.conf.
 3.  "`turn_user_lifetime`": This is the amount of time credentials
-    generated by your Home Server are valid for (in milliseconds).
+    generated by your homeserver are valid for (in milliseconds).
    Shorter times offer less potential for abuse at the expense of
-    increased traffic between web clients and your home server to
+    increased traffic between web clients and your homeserver to
    refresh credentials. The TURN REST API specification recommends
    one day (86400000).
 4.  "`turn_allow_guests`": Whether to allow guest users to use the
@@ -0,0 +1,84 @@
+# Background Updates API
+
+This API allows a server administrator to manage the background updates being
+run against the database.
+
+## Status
+
+This API gets the current status of the background updates.
+
+
+The API is:
+
+```
+GET /_synapse/admin/v1/background_updates/status
+```
+
+Returning:
+
+```json
+{
+    "enabled": true,
+    "current_updates": {
+        "<db_name>": {
+            "name": "<background_update_name>",
+            "total_item_count": 50,
+            "total_duration_ms": 10000.0,
+            "average_items_per_ms": 2.2,
+        },
+    }
+}
+```
+
+`enabled` whether the background updates are enabled or disabled.
+
+`db_name` the database name (usually Synapse is configured with a single database named 'master').
+
+For each update:
+
+`name` the name of the update.
+`total_item_count` total number of "items" processed (the meaning of 'items' depends on the update in question).
+`total_duration_ms` how long the background process has been running, not including time spent sleeping.
+`average_items_per_ms` how many items are processed per millisecond based on an exponential average.
+
+
+
+## Enabled
+
+This API allow pausing background updates.
+
+Background updates should *not* be paused for significant periods of time, as
+this can affect the performance of Synapse.
+
+*Note*: This won't persist over restarts.
+
+*Note*: This won't cancel any update query that is currently running. This is
+usually fine since most queries are short lived, except for `CREATE INDEX`
+background updates which won't be cancelled once started.
+
+
+The API is:
+
+```
+POST /_synapse/admin/v1/background_updates/enabled
+```
+
+with the following body:
+
+```json
+{
+    "enabled": false
+}
+```
+
+`enabled` sets whether the background updates are enabled or disabled.
+
+The API returns the `enabled` param.
+
+```json
+{
+    "enabled": false
+}
+```
+
+There is also a `GET` version which returns the `enabled` state.
@@ -0,0 +1,5 @@
+# Single Sign-On
+
+Synapse supports single sign-on through the SAML, Open ID Connect or CAS protocols. 
+LDAP and other login methods are supported through first and third-party password
+auth provider modules.
@@ -0,0 +1,8 @@
+# CAS
+
+Synapse supports authenticating users via the [Central Authentication
+Service protocol](https://en.wikipedia.org/wiki/Central_Authentication_Service)
+(CAS) natively.
+
+Please see the `cas_config` and `sso` sections of the [Synapse configuration
+file](../../../configuration/homeserver_sample_config.md) for more details.
@@ -0,0 +1,8 @@
+# SAML
+
+Synapse supports authenticating users via the [Security Assertion
+Markup Language](https://en.wikipedia.org/wiki/Security_Assertion_Markup_Language)
+(SAML) protocol natively.
+
+Please see the `saml2_config` and `sso` sections of the [Synapse configuration
+file](../../../configuration/homeserver_sample_config.md) for more details.
@@ -10,92 +10,147 @@ warn_unreachable = True
 local_partial_types = True
 no_implicit_optional = True

-# To find all folders that pass mypy you run:
-#
-#   find synapse/* -type d -not -name __pycache__ -exec bash -c "mypy '{}' > /dev/null"  \; -print
-
 files =
  scripts-dev/sign_json,
-  synapse/__init__.py,
-  synapse/api,
-  synapse/appservice,
-  synapse/config,
-  synapse/crypto,
-  synapse/event_auth.py,
-  synapse/events/builder.py,
-  synapse/events/presence_router.py,
-  synapse/events/snapshot.py,
-  synapse/events/spamcheck.py,
-  synapse/events/third_party_rules.py,
-  synapse/events/utils.py,
-  synapse/events/validator.py,
-  synapse/federation,
-  synapse/groups,
-  synapse/handlers,
-  synapse/http,
-  synapse/logging,
-  synapse/metrics,
-  synapse/module_api,
-  synapse/notifier.py,
-  synapse/push,
-  synapse/replication,
-  synapse/rest,
-  synapse/server.py,
-  synapse/server_notices,
-  synapse/spam_checker_api,
-  synapse/state,
-  synapse/storage/__init__.py,
-  synapse/storage/_base.py,
-  synapse/storage/background_updates.py,
-  synapse/storage/databases/main/appservice.py,
-  synapse/storage/databases/main/client_ips.py,
-  synapse/storage/databases/main/events.py,
-  synapse/storage/databases/main/keys.py,
-  synapse/storage/databases/main/pusher.py,
-  synapse/storage/databases/main/registration.py,
-  synapse/storage/databases/main/relations.py,
-  synapse/storage/databases/main/session.py,
-  synapse/storage/databases/main/stream.py,
-  synapse/storage/databases/main/ui_auth.py,
-  synapse/storage/databases/state,
-  synapse/storage/database.py,
-  synapse/storage/engines,
-  synapse/storage/keys.py,
-  synapse/storage/persist_events.py,
-  synapse/storage/prepare_database.py,
-  synapse/storage/purge_events.py,
-  synapse/storage/push_rule.py,
-  synapse/storage/relations.py,
-  synapse/storage/roommember.py,
-  synapse/storage/state.py,
-  synapse/storage/types.py,
-  synapse/storage/util,
-  synapse/streams,
-  synapse/types.py,
-  synapse/util,
-  synapse/visibility.py,
-  tests/replication,
-  tests/test_event_auth.py,
-  tests/test_utils,
-  tests/handlers/test_password_providers.py,
-  tests/handlers/test_room.py,
-  tests/handlers/test_room_summary.py,
-  tests/handlers/test_send_email.py,
-  tests/handlers/test_sync.py,
-  tests/handlers/test_user_directory.py,
-  tests/rest/client/test_login.py,
-  tests/rest/client/test_auth.py,
-  tests/rest/client/test_relations.py,
-  tests/rest/media/v1/test_filepath.py,
-  tests/rest/media/v1/test_oembed.py,
-  tests/storage/test_state.py,
-  tests/storage/test_user_directory.py,
-  tests/util/test_itertools.py,
-  tests/util/test_stream_change_cache.py
+  setup.py,
+  synapse/,
+  tests/
+
+# Note: Better exclusion syntax coming in mypy > 0.910
+# https://github.com/python/mypy/pull/11329
+#
+# For now, set the (?x) flag enable "verbose" regexes
+# https://docs.python.org/3/library/re.html#re.X
+exclude = (?x)
+  ^(
+   |synapse/storage/databases/__init__.py
+   |synapse/storage/databases/main/__init__.py
+   |synapse/storage/databases/main/account_data.py
+   |synapse/storage/databases/main/cache.py
+   |synapse/storage/databases/main/devices.py
+   |synapse/storage/databases/main/e2e_room_keys.py
+   |synapse/storage/databases/main/end_to_end_keys.py
+   |synapse/storage/databases/main/event_federation.py
+   |synapse/storage/databases/main/event_push_actions.py
+   |synapse/storage/databases/main/events_bg_updates.py
+   |synapse/storage/databases/main/events_worker.py
+   |synapse/storage/databases/main/group_server.py
+   |synapse/storage/databases/main/metrics.py
+   |synapse/storage/databases/main/monthly_active_users.py
+   |synapse/storage/databases/main/presence.py
+   |synapse/storage/databases/main/purge_events.py
+   |synapse/storage/databases/main/push_rule.py
+   |synapse/storage/databases/main/receipts.py
+   |synapse/storage/databases/main/room.py
+   |synapse/storage/databases/main/roommember.py
+   |synapse/storage/databases/main/search.py
+   |synapse/storage/databases/main/state.py
+   |synapse/storage/databases/main/stats.py
+   |synapse/storage/databases/main/transactions.py
+   |synapse/storage/databases/main/user_directory.py
+   |synapse/storage/schema/
+
+   |tests/api/test_auth.py
+   |tests/api/test_ratelimiting.py
+   |tests/app/test_openid_listener.py
+   |tests/appservice/test_scheduler.py
+   |tests/config/test_cache.py
+   |tests/config/test_tls.py
+   |tests/crypto/test_keyring.py
+   |tests/events/test_presence_router.py
+   |tests/events/test_utils.py
+   |tests/federation/test_federation_catch_up.py
+   |tests/federation/test_federation_sender.py
+   |tests/federation/test_federation_server.py
+   |tests/federation/transport/test_knocking.py
+   |tests/federation/transport/test_server.py
+   |tests/handlers/test_cas.py
+   |tests/handlers/test_directory.py
+   |tests/handlers/test_e2e_keys.py
+   |tests/handlers/test_federation.py
+   |tests/handlers/test_oidc.py
+   |tests/handlers/test_presence.py
+   |tests/handlers/test_profile.py
+   |tests/handlers/test_saml.py
+   |tests/handlers/test_typing.py
+   |tests/http/federation/test_matrix_federation_agent.py
+   |tests/http/federation/test_srv_resolver.py
+   |tests/http/test_fedclient.py
+   |tests/http/test_proxyagent.py
+   |tests/http/test_servlet.py
+   |tests/http/test_site.py
+   |tests/logging/__init__.py
+   |tests/logging/test_terse_json.py
+   |tests/module_api/test_api.py
+   |tests/push/test_email.py
+   |tests/push/test_http.py
+   |tests/push/test_presentable_names.py
+   |tests/push/test_push_rule_evaluator.py
+   |tests/rest/admin/test_admin.py
+   |tests/rest/admin/test_device.py
+   |tests/rest/admin/test_media.py
+   |tests/rest/admin/test_server_notice.py
+   |tests/rest/admin/test_user.py
+   |tests/rest/admin/test_username_available.py
+   |tests/rest/client/test_account.py
+   |tests/rest/client/test_events.py
+   |tests/rest/client/test_filter.py
+   |tests/rest/client/test_groups.py
+   |tests/rest/client/test_register.py
+   |tests/rest/client/test_report_event.py
+   |tests/rest/client/test_rooms.py
+   |tests/rest/client/test_third_party_rules.py
+   |tests/rest/client/test_transactions.py
+   |tests/rest/client/test_typing.py
+   |tests/rest/client/utils.py
+   |tests/rest/key/v2/test_remote_key_resource.py
+   |tests/rest/media/v1/test_base.py
+   |tests/rest/media/v1/test_media_storage.py
+   |tests/rest/media/v1/test_url_preview.py
+   |tests/scripts/test_new_matrix_user.py
+   |tests/server.py
+   |tests/server_notices/test_resource_limits_server_notices.py
+   |tests/state/test_v2.py
+   |tests/storage/test_account_data.py
+   |tests/storage/test_appservice.py
+   |tests/storage/test_background_update.py
+   |tests/storage/test_base.py
+   |tests/storage/test_client_ips.py
+   |tests/storage/test_database.py
+   |tests/storage/test_event_federation.py
+   |tests/storage/test_id_generators.py
+   |tests/storage/test_roommember.py
+   |tests/test_metrics.py
+   |tests/test_phone_home.py
+   |tests/test_server.py
+   |tests/test_state.py
+   |tests/test_terms_auth.py
+   |tests/test_visibility.py
+   |tests/unittest.py
+   |tests/util/caches/test_cached_call.py
+   |tests/util/caches/test_deferred_cache.py
+   |tests/util/caches/test_descriptors.py
+   |tests/util/caches/test_response_cache.py
+   |tests/util/caches/test_ttlcache.py
+   |tests/util/test_async_helpers.py
+   |tests/util/test_batching_queue.py
+   |tests/util/test_dict_cache.py
+   |tests/util/test_expiring_cache.py
+   |tests/util/test_file_consumer.py
+   |tests/util/test_linearizer.py
+   |tests/util/test_logcontext.py
+   |tests/util/test_lrucache.py
+   |tests/util/test_rwlock.py
+   |tests/util/test_wheel_timer.py
+   |tests/utils.py
+   )$

 [mypy-synapse.api.*]
 disallow_untyped_defs = True

+[mypy-synapse.app.*]
+disallow_untyped_defs = True
+
 [mypy-synapse.crypto.*]
 disallow_untyped_defs = True

@@ -120,6 +175,21 @@ disallow_untyped_defs = True
 [mypy-synapse.storage.databases.main.client_ips]
 disallow_untyped_defs = True

+[mypy-synapse.storage.databases.main.directory]
+disallow_untyped_defs = True
+
+[mypy-synapse.storage.databases.main.room_batch]
+disallow_untyped_defs = True
+
+[mypy-synapse.storage.databases.main.profile]
+disallow_untyped_defs = True
+
+[mypy-synapse.storage.databases.main.state_deltas]
+disallow_untyped_defs = True
+
+[mypy-synapse.storage.databases.main.user_erasure_store]
+disallow_untyped_defs = True
+
 [mypy-synapse.storage.util.*]
 disallow_untyped_defs = True

@@ -216,9 +286,15 @@ disallow_untyped_defs = True
 [mypy-tests.handlers.test_user_directory]
 disallow_untyped_defs = True

+[mypy-tests.storage.test_profile]
+disallow_untyped_defs = True
+
 [mypy-tests.storage.test_user_directory]
 disallow_untyped_defs = True

+[mypy-tests.rest.client.test_directory]
+disallow_untyped_defs = True
+
 ;; Dependencies without annotations
 ;; Before ignoring a module, check to see if type stubs are available.
 ;; The `typeshed` project maintains stubs here:
@@ -278,6 +354,9 @@ ignore_missing_imports = True
 [mypy-opentracing]
 ignore_missing_imports = True

+[mypy-parameterized.*]
+ignore_missing_imports = True
+
 [mypy-phonenumbers.*]
 ignore_missing_imports = True

@@ -43,6 +43,7 @@ from synapse.storage.databases.main.end_to_end_keys import EndToEndKeyBackground
 from synapse.storage.databases.main.events_bg_updates import (
    EventsBackgroundUpdatesStore,
 )
+from synapse.storage.databases.main.group_server import GroupServerWorkerStore
 from synapse.storage.databases.main.media_repository import (
    MediaRepositoryBackgroundUpdateStore,
 )
@@ -181,6 +182,7 @@ class Store(
    StatsStore,
    PusherWorkerStore,
    PresenceBackgroundUpdateStore,
+    GroupServerWorkerStore,
 ):
    def execute(self, f, *args, **kwargs):
        return self.db_pool.runInteraction(f.__name__, f, *args, **kwargs)
@@ -17,6 +17,7 @@
 # limitations under the License.
 import glob
 import os
+from typing import Any, Dict

 from setuptools import Command, find_packages, setup

@@ -49,8 +50,6 @@ here = os.path.abspath(os.path.dirname(__file__))
 # [1]: http://tox.readthedocs.io/en/2.5.0/example/basic.html#integration-with-setup-py-test-command
 # [2]: https://pypi.python.org/pypi/setuptools_trial
 class TestCommand(Command):
-    user_options = []
-
    def initialize_options(self):
        pass

@@ -75,7 +74,7 @@ def read_file(path_segments):

 def exec_file(path_segments):
    """Execute a single python file to get the variables defined in it"""
-    result = {}
+    result: Dict[str, Any] = {}
    code = read_file(path_segments)
    exec(code, result)
    return result
@@ -111,6 +110,7 @@ CONDITIONAL_REQUIREMENTS["mypy"] = [
    "types-Pillow>=8.3.4",
    "types-pyOpenSSL>=20.0.7",
    "types-PyYAML>=5.4.10",
+    "types-requests>=2.26.0",
    "types-setuptools>=57.4.0",
 ]

@@ -132,6 +132,9 @@ CONDITIONAL_REQUIREMENTS["dev"] = (
        "GitPython==3.1.14",
        "commonmark==0.9.1",
        "pygithub==1.55",
+        # The following are executed as commands by the release script.
+        "twine",
+        "towncrier",
    ]
 )

@@ -47,7 +47,7 @@ try:
 except ImportError:
    pass

-__version__ = "1.46.0rc1"
+__version__ = "1.47.0rc2"

 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
    # We import here so that we don't have to install a bunch of deps when
@@ -1,5 +1,6 @@
 # Copyright 2015, 2016 OpenMarket Ltd
 # Copyright 2018 New Vector
+# Copyright 2021 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -19,22 +20,23 @@ import hashlib
 import hmac
 import logging
 import sys
+from typing import Callable, Optional

 import requests as _requests
 import yaml


 def request_registration(
-    user,
-    password,
-    server_location,
-    shared_secret,
-    admin=False,
-    user_type=None,
+    user: str,
+    password: str,
+    server_location: str,
+    shared_secret: str,
+    admin: bool = False,
+    user_type: Optional[str] = None,
    requests=_requests,
-    _print=print,
-    exit=sys.exit,
-):
+    _print: Callable[[str], None] = print,
+    exit: Callable[[int], None] = sys.exit,
+) -> None:

    url = "%s/_synapse/admin/v1/register" % (server_location.rstrip("/"),)

@@ -65,13 +67,13 @@ def request_registration(
        mac.update(b"\x00")
        mac.update(user_type.encode("utf8"))

-    mac = mac.hexdigest()
+    hex_mac = mac.hexdigest()

    data = {
        "nonce": nonce,
        "username": user,
        "password": password,
-        "mac": mac,
+        "mac": hex_mac,
        "admin": admin,
        "user_type": user_type,
    }
@@ -91,10 +93,17 @@ def request_registration(
    _print("Success!")


-def register_new_user(user, password, server_location, shared_secret, admin, user_type):
+def register_new_user(
+    user: str,
+    password: str,
+    server_location: str,
+    shared_secret: str,
+    admin: Optional[bool],
+    user_type: Optional[str],
+) -> None:
    if not user:
        try:
-            default_user = getpass.getuser()
+            default_user: Optional[str] = getpass.getuser()
        except Exception:
            default_user = None

@@ -123,8 +132,8 @@ def register_new_user(user, password, server_location, shared_secret, admin, use
            sys.exit(1)

    if admin is None:
-        admin = input("Make admin [no]: ")
-        if admin in ("y", "yes", "true"):
+        admin_inp = input("Make admin [no]: ")
+        if admin_inp in ("y", "yes", "true"):
            admin = True
        else:
            admin = False
@@ -134,7 +143,7 @@ def register_new_user(user, password, server_location, shared_secret, admin, use
    )


-def main():
+def main() -> None:

    logging.captureWarnings(True)

@@ -92,7 +92,7 @@ def get_recent_users(txn: LoggingTransaction, since_ms: int) -> List[UserInfo]:
    return user_infos


-def main():
+def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "-c",
@@ -142,7 +142,8 @@ def main():
    engine = create_engine(database_config.config)

    with make_conn(database_config, engine, "review_recent_signups") as db_conn:
-        user_infos = get_recent_users(db_conn.cursor(), since_ms)
+        # This generates a type of Cursor, not LoggingTransaction.
+        user_infos = get_recent_users(db_conn.cursor(), since_ms)  # type: ignore[arg-type]

    for user_info in user_infos:
        if exclude_users_with_email and user_info.emails:
@@ -1,7 +1,7 @@
 # Copyright 2015, 2016 OpenMarket Ltd
 # Copyright 2017 Vector Creations Ltd
 # Copyright 2018-2019 New Vector Ltd
-# Copyright 2019 The Matrix.org Foundation C.I.C.
+# Copyright 2019-2021 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -86,6 +86,9 @@ ROOM_EVENT_FILTER_SCHEMA = {
        # cf https://github.com/matrix-org/matrix-doc/pull/2326
        "org.matrix.labels": {"type": "array", "items": {"type": "string"}},
        "org.matrix.not_labels": {"type": "array", "items": {"type": "string"}},
+        # MSC3440, filtering by event relations.
+        "io.element.relation_senders": {"type": "array", "items": {"type": "string"}},
+        "io.element.relation_types": {"type": "array", "items": {"type": "string"}},
    },
 }

@@ -146,14 +149,16 @@ def matrix_user_id_validator(user_id_str: str) -> UserID:

 class Filtering:
    def __init__(self, hs: "HomeServer"):
-        super().__init__()
+        self._hs = hs
        self.store = hs.get_datastore()

+        self.DEFAULT_FILTER_COLLECTION = FilterCollection(hs, {})
+
    async def get_user_filter(
        self, user_localpart: str, filter_id: Union[int, str]
    ) -> "FilterCollection":
        result = await self.store.get_user_filter(user_localpart, filter_id)
-        return FilterCollection(result)
+        return FilterCollection(self._hs, result)

    def add_user_filter(
        self, user_localpart: str, user_filter: JsonDict
@@ -191,21 +196,22 @@ FilterEvent = TypeVar("FilterEvent", EventBase, UserPresenceState, JsonDict)


 class FilterCollection:
-    def __init__(self, filter_json: JsonDict):
+    def __init__(self, hs: "HomeServer", filter_json: JsonDict):
        self._filter_json = filter_json

        room_filter_json = self._filter_json.get("room", {})

        self._room_filter = Filter(
-            {k: v for k, v in room_filter_json.items() if k in ("rooms", "not_rooms")}
+            hs,
+            {k: v for k, v in room_filter_json.items() if k in ("rooms", "not_rooms")},
        )

-        self._room_timeline_filter = Filter(room_filter_json.get("timeline", {}))
-        self._room_state_filter = Filter(room_filter_json.get("state", {}))
-        self._room_ephemeral_filter = Filter(room_filter_json.get("ephemeral", {}))
-        self._room_account_data = Filter(room_filter_json.get("account_data", {}))
-        self._presence_filter = Filter(filter_json.get("presence", {}))
-        self._account_data = Filter(filter_json.get("account_data", {}))
+        self._room_timeline_filter = Filter(hs, room_filter_json.get("timeline", {}))
+        self._room_state_filter = Filter(hs, room_filter_json.get("state", {}))
+        self._room_ephemeral_filter = Filter(hs, room_filter_json.get("ephemeral", {}))
+        self._room_account_data = Filter(hs, room_filter_json.get("account_data", {}))
+        self._presence_filter = Filter(hs, filter_json.get("presence", {}))
+        self._account_data = Filter(hs, filter_json.get("account_data", {}))

        self.include_leave = filter_json.get("room", {}).get("include_leave", False)
        self.event_fields = filter_json.get("event_fields", [])
@@ -232,25 +238,37 @@ class FilterCollection:
    def include_redundant_members(self) -> bool:
        return self._room_state_filter.include_redundant_members

-    def filter_presence(
+    async def filter_presence(
        self, events: Iterable[UserPresenceState]
    ) -> List[UserPresenceState]:
-        return self._presence_filter.filter(events)
+        return await self._presence_filter.filter(events)

-    def filter_account_data(self, events: Iterable[JsonDict]) -> List[JsonDict]:
-        return self._account_data.filter(events)
+    async def filter_account_data(self, events: Iterable[JsonDict]) -> List[JsonDict]:
+        return await self._account_data.filter(events)

-    def filter_room_state(self, events: Iterable[EventBase]) -> List[EventBase]:
-        return self._room_state_filter.filter(self._room_filter.filter(events))
+    async def filter_room_state(self, events: Iterable[EventBase]) -> List[EventBase]:
+        return await self._room_state_filter.filter(
+            await self._room_filter.filter(events)
+        )

-    def filter_room_timeline(self, events: Iterable[EventBase]) -> List[EventBase]:
-        return self._room_timeline_filter.filter(self._room_filter.filter(events))
+    async def filter_room_timeline(
+        self, events: Iterable[EventBase]
+    ) -> List[EventBase]:
+        return await self._room_timeline_filter.filter(
+            await self._room_filter.filter(events)
+        )

-    def filter_room_ephemeral(self, events: Iterable[JsonDict]) -> List[JsonDict]:
-        return self._room_ephemeral_filter.filter(self._room_filter.filter(events))
+    async def filter_room_ephemeral(self, events: Iterable[JsonDict]) -> List[JsonDict]:
+        return await self._room_ephemeral_filter.filter(
+            await self._room_filter.filter(events)
+        )

-    def filter_room_account_data(self, events: Iterable[JsonDict]) -> List[JsonDict]:
-        return self._room_account_data.filter(self._room_filter.filter(events))
+    async def filter_room_account_data(
+        self, events: Iterable[JsonDict]
+    ) -> List[JsonDict]:
+        return await self._room_account_data.filter(
+            await self._room_filter.filter(events)
+        )

    def blocks_all_presence(self) -> bool:
        return (
@@ -274,7 +292,9 @@ class FilterCollection:


 class Filter:
-    def __init__(self, filter_json: JsonDict):
+    def __init__(self, hs: "HomeServer", filter_json: JsonDict):
+        self._hs = hs
+        self._store = hs.get_datastore()
        self.filter_json = filter_json

        self.limit = filter_json.get("limit", 10)
@@ -297,6 +317,20 @@ class Filter:
        self.labels = filter_json.get("org.matrix.labels", None)
        self.not_labels = filter_json.get("org.matrix.not_labels", [])

+        # Ideally these would be rejected at the endpoint if they were provided
+        # and not supported, but that would involve modifying the JSON schema
+        # based on the homeserver configuration.
+        if hs.config.experimental.msc3440_enabled:
+            self.relation_senders = self.filter_json.get(
+                "io.element.relation_senders", None
+            )
+            self.relation_types = self.filter_json.get(
+                "io.element.relation_types", None
+            )
+        else:
+            self.relation_senders = None
+            self.relation_types = None
+
    def filters_all_types(self) -> bool:
        return "*" in self.not_types

@@ -306,7 +340,7 @@ class Filter:
    def filters_all_rooms(self) -> bool:
        return "*" in self.not_rooms

-    def check(self, event: FilterEvent) -> bool:
+    def _check(self, event: FilterEvent) -> bool:
        """Checks whether the filter matches the given event.

        Args:
@@ -420,8 +454,30 @@ class Filter:

        return room_ids

-    def filter(self, events: Iterable[FilterEvent]) -> List[FilterEvent]:
-        return list(filter(self.check, events))
+    async def _check_event_relations(
+        self, events: Iterable[FilterEvent]
+    ) -> List[FilterEvent]:
+        # The event IDs to check, mypy doesn't understand the ifinstance check.
+        event_ids = [event.event_id for event in events if isinstance(event, EventBase)]  # type: ignore[attr-defined]
+        event_ids_to_keep = set(
+            await self._store.events_have_relations(
+                event_ids, self.relation_senders, self.relation_types
+            )
+        )
+
+        return [
+            event
+            for event in events
+            if not isinstance(event, EventBase) or event.event_id in event_ids_to_keep
+        ]
+
+    async def filter(self, events: Iterable[FilterEvent]) -> List[FilterEvent]:
+        result = [event for event in events if self._check(event)]
+
+        if self.relation_senders or self.relation_types:
+            return await self._check_event_relations(result)
+
+        return result

    def with_room_ids(self, room_ids: Iterable[str]) -> "Filter":
        """Returns a new filter with the given room IDs appended.
@@ -433,7 +489,7 @@ class Filter:
            filter: A new filter including the given rooms and the old
                    filter's rooms.
        """
-        newFilter = Filter(self.filter_json)
+        newFilter = Filter(self._hs, self.filter_json)
        newFilter.rooms += room_ids
        return newFilter

@@ -444,6 +500,3 @@ def _matches_wildcard(actual_value: Optional[str], filter_value: str) -> bool:
        return actual_value.startswith(type_prefix)
    else:
        return actual_value == filter_value
-
-
-DEFAULT_FILTER_COLLECTION = FilterCollection({})
@@ -38,9 +38,6 @@ class ConsentURIBuilder:
    def __init__(self, hs_config: HomeServerConfig):
        if hs_config.key.form_secret is None:
            raise ConfigError("form_secret not set in config")
-        if hs_config.server.public_baseurl is None:
-            raise ConfigError("public_baseurl not set in config")
-
        self._hmac_secret = hs_config.key.form_secret.encode("utf-8")
        self._public_baseurl = hs_config.server.public_baseurl

--- a/Show More
+++ b/Show More
				`@@ -1 +0,0 @@`
				`Fix a long-standing bug which allowed hidden devices to receive to-device messages, resulting in unnecessary database bloat.`
				`@@ -1 +0,0 @@`
				Add type annotations for the `log_function` decorator.
				`@@ -1 +0,0 @@`
				Fix a long-standing bug where messages in the `device_inbox` table for deleted devices would persist indefinitely. Contributed by @dklimpel and @JohannesKleine.
				`@@ -1 +0,0 @@`
				`Do not accept events if a third-party rule module API callback raises an exception.`
				`@@ -1 +0,0 @@`
				`Advertise support for Client-Server API r0.6.1.`
				`@@ -1 +0,0 @@`
				Add an `on_new_event` third-party rules callback to allow Synapse modules to act after an event has been sent into a room.
				`@@ -1 +0,0 @@`
				`Improve example HAProxy config in the docs to properly handle host headers with port information. This is required for federation over port 443 to work correctly.`
				`@@ -1 +0,0 @@`
				`Fix long-standing bug where verification requests could fail in certain cases if whitelist was in place but did not include your own homeserver.`
				`@@ -1 +0,0 @@`
				`Add a module API method to update a user's membership in a room.`
				`@@ -1 +0,0 @@`
				`Add documentation for using Authentik as an OpenID Connect Identity Provider. Contributed by @samip5.`