Merge branch 'shay/super_docs' of https://github.com/matrix-org/synapse into shay/super_docs

CHANGES.md

@@ -1,105 +1,8 @@
-Synapse 1.59.0rc1 (2022-05-10)
-==============================
+Synapse 1.59.0
+==============
 
-This release makes several changes that server administrators should be aware of:
+The non-standard `m.login.jwt` login type has been removed from Synapse. It can be replaced with `org.matrix.login.jwt` for identical behaviour. This is only used if `jwt_config.enabled` is set to `true` in the configuration.
 
-- Device name lookup over federation is now disabled by default. ([\#12616](https://github.com/matrix-org/synapse/issues/12616))
-- The `synapse.app.appservice` and `synapse.app.user_dir` worker application types are now deprecated. ([\#12452](https://github.com/matrix-org/synapse/issues/12452), [\#12654](https://github.com/matrix-org/synapse/issues/12654))
-
-See [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1590) for more details.
-
-Additionally, this release removes the non-standard `m.login.jwt` login type from Synapse. It can be replaced with `org.matrix.login.jwt` for identical behaviour. This is only used if `jwt_config.enabled` is set to `true` in the configuration. ([\#12597](https://github.com/matrix-org/synapse/issues/12597))
-
-Features
---------
-
-- Support [MSC3266](https://github.com/matrix-org/matrix-doc/pull/3266) room summaries over federation. ([\#11507](https://github.com/matrix-org/synapse/issues/11507))
-- Implement [changes](https://github.com/matrix-org/matrix-spec-proposals/pull/2285/commits/4a77139249c2e830aec3c7d6bd5501a514d1cc27) to [MSC2285 (hidden read receipts)](https://github.com/matrix-org/matrix-spec-proposals/pull/2285). Contributed by @SimonBrandner. ([\#12168](https://github.com/matrix-org/synapse/issues/12168), [\#12635](https://github.com/matrix-org/synapse/issues/12635), [\#12636](https://github.com/matrix-org/synapse/issues/12636), [\#12670](https://github.com/matrix-org/synapse/issues/12670))
-- Extend the [module API](https://github.com/matrix-org/synapse/blob/release-v1.59/synapse/module_api/__init__.py) to allow modules to change actions for existing push rules of local users. ([\#12406](https://github.com/matrix-org/synapse/issues/12406))
-- Add the `notify_appservices_from_worker` configuration option (superseding `notify_appservices`) to allow a generic worker to be designated as the worker to send traffic to Application Services. ([\#12452](https://github.com/matrix-org/synapse/issues/12452))
-- Add the `update_user_directory_from_worker` configuration option (superseding `update_user_directory`) to allow a generic worker to be designated as the worker to update the user directory. ([\#12654](https://github.com/matrix-org/synapse/issues/12654))
-- Add new `enable_registration_token_3pid_bypass` configuration option to allow registrations via token as an alternative to verifying a 3pid. ([\#12526](https://github.com/matrix-org/synapse/issues/12526))
-- Implement [MSC3786](https://github.com/matrix-org/matrix-spec-proposals/pull/3786): Add a default push rule to ignore `m.room.server_acl` events. ([\#12601](https://github.com/matrix-org/synapse/issues/12601))
-- Add new `mau_appservice_trial_days` configuration option to specify a different trial period for users registered via an appservice. ([\#12619](https://github.com/matrix-org/synapse/issues/12619))
-
-
-Bugfixes
---------
-
-- Fix a bug introduced in Synapse 1.48.0 where the latest thread reply provided failed to include the proper bundled aggregations. ([\#12273](https://github.com/matrix-org/synapse/issues/12273))
-- Fix a bug introduced in Synapse 1.22.0 where attempting to send a large amount of read receipts to an application service all at once would result in duplicate content and abnormally high memory usage. Contributed by Brad & Nick @ Beeper. ([\#12544](https://github.com/matrix-org/synapse/issues/12544))
-- Fix a bug introduced in Synapse 1.57.0 which could cause `Failed to calculate hosts in room` errors to be logged for outbound federation. ([\#12570](https://github.com/matrix-org/synapse/issues/12570))
-- Fix a long-standing bug where status codes would almost always get logged as `200!`, irrespective of the actual status code, when clients disconnect before a request has finished processing. ([\#12580](https://github.com/matrix-org/synapse/issues/12580))
-- Fix race when persisting an event and deleting a room that could lead to outbound federation breaking. ([\#12594](https://github.com/matrix-org/synapse/issues/12594))
-- Fix a bug introduced in Synapse 1.53.0 where bundled aggregations for annotations/edits were incorrectly calculated. ([\#12633](https://github.com/matrix-org/synapse/issues/12633))
-- Fix a long-standing bug where rooms containing power levels with string values could not be upgraded. ([\#12657](https://github.com/matrix-org/synapse/issues/12657))
-- Prevent memory leak from reoccurring when presence is disabled. ([\#12656](https://github.com/matrix-org/synapse/issues/12656))
-
-
-Updates to the Docker image
----------------------------
-
-- Explicitly opt-in to using [BuildKit-specific features](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md) in the Dockerfile. This fixes issues with building images in some GitLab CI environments. ([\#12541](https://github.com/matrix-org/synapse/issues/12541))
-- Update the "Build docker images" GitHub Actions workflow to use `docker/metadata-action` to generate docker image tags, instead of a custom shell script. Contributed by @henryclw. ([\#12573](https://github.com/matrix-org/synapse/issues/12573))
-
-
-Improved Documentation
-----------------------
-
-- Update SQL statements and replace use of old table `user_stats_historical` in docs for Synapse Admins. ([\#12536](https://github.com/matrix-org/synapse/issues/12536))
-- Add missing linebreak to `pipx` install instructions. ([\#12579](https://github.com/matrix-org/synapse/issues/12579))
-- Add information about the TCP replication module to docs. ([\#12621](https://github.com/matrix-org/synapse/issues/12621))
-- Fixes to the formatting of `README.rst`. ([\#12627](https://github.com/matrix-org/synapse/issues/12627))
-- Fix docs on how to run specific Complement tests using the `complement.sh` test runner. ([\#12664](https://github.com/matrix-org/synapse/issues/12664))
-
-
-Deprecations and Removals
--------------------------
-
-- Remove unstable identifiers from [MSC3069](https://github.com/matrix-org/matrix-doc/pull/3069). ([\#12596](https://github.com/matrix-org/synapse/issues/12596))
-- Remove the unspecified `m.login.jwt` login type and the unstable `uk.half-shot.msc2778.login.application_service` from 
-  [MSC2778](https://github.com/matrix-org/matrix-doc/pull/2778). ([\#12597](https://github.com/matrix-org/synapse/issues/12597))
-- Synapse now requires at least Python 3.7.1 (up from 3.7.0), for compatibility with the latest Twisted trunk. ([\#12613](https://github.com/matrix-org/synapse/issues/12613))
-
-
-Internal Changes
-----------------
-
-- Use supervisord to supervise Postgres and Caddy in the Complement image to reduce restart time. ([\#12480](https://github.com/matrix-org/synapse/issues/12480))
-- Immediately retry any requests that have backed off when a server comes back online. ([\#12500](https://github.com/matrix-org/synapse/issues/12500))
-- Use `make_awaitable` instead of `defer.succeed` for return values of mocks in tests. ([\#12505](https://github.com/matrix-org/synapse/issues/12505))
-- Consistently check if an object is a `frozendict`. ([\#12564](https://github.com/matrix-org/synapse/issues/12564))
-- Protect module callbacks with read semantics against cancellation. ([\#12568](https://github.com/matrix-org/synapse/issues/12568))
-- Improve comments and error messages around access tokens. ([\#12577](https://github.com/matrix-org/synapse/issues/12577))
-- Improve docstrings for the receipts store. ([\#12581](https://github.com/matrix-org/synapse/issues/12581))
-- Use constants for read-receipts in tests. ([\#12582](https://github.com/matrix-org/synapse/issues/12582))
-- Log status code of cancelled requests as 499 and avoid logging stack traces for them. ([\#12587](https://github.com/matrix-org/synapse/issues/12587), [\#12663](https://github.com/matrix-org/synapse/issues/12663))
-- Remove special-case for `twisted` logger from default log config. ([\#12589](https://github.com/matrix-org/synapse/issues/12589))
-- Use `getClientAddress` instead of the deprecated `getClientIP`. ([\#12599](https://github.com/matrix-org/synapse/issues/12599))
-- Add link to documentation in Grafana Dashboard. ([\#12602](https://github.com/matrix-org/synapse/issues/12602))
-- Reduce log spam when running multiple event persisters. ([\#12610](https://github.com/matrix-org/synapse/issues/12610))
-- Add extra debug logging to federation sender. ([\#12614](https://github.com/matrix-org/synapse/issues/12614))
-- Prevent remote homeservers from requesting local user device names by default. ([\#12616](https://github.com/matrix-org/synapse/issues/12616))
-- Add a consistency check on events which we read from the database. ([\#12620](https://github.com/matrix-org/synapse/issues/12620))
-- Remove use of the `constantly` library and switch to enums for `EventRedactBehaviour`. Contributed by @andrewdoh. ([\#12624](https://github.com/matrix-org/synapse/issues/12624))
-- Remove unused code related to receipts. ([\#12632](https://github.com/matrix-org/synapse/issues/12632))
-- Minor improvements to the scripts for running Synapse in worker mode under Complement. ([\#12637](https://github.com/matrix-org/synapse/issues/12637))
-- Move `pympler` back in to the `all` extras. ([\#12652](https://github.com/matrix-org/synapse/issues/12652))
-- Fix spelling of `M_UNRECOGNIZED` in comments. ([\#12665](https://github.com/matrix-org/synapse/issues/12665))
-- Release script: confirm the commit to be tagged before tagging. ([\#12556](https://github.com/matrix-org/synapse/issues/12556))
-- Fix a typo in the announcement text generated by the Synapse release development script. ([\#12612](https://github.com/matrix-org/synapse/issues/12612))
-
-### Typechecking
-
-- Fix scripts-dev to pass typechecking. ([\#12356](https://github.com/matrix-org/synapse/issues/12356))
-- Add some type hints to datastore. ([\#12485](https://github.com/matrix-org/synapse/issues/12485))
-- Remove unused `# type: ignore`s. ([\#12531](https://github.com/matrix-org/synapse/issues/12531))
-- Allow unused `# type: ignore` comments in bleeding edge CI jobs. ([\#12576](https://github.com/matrix-org/synapse/issues/12576))
-- Remove redundant lines of config from `mypy.ini`. ([\#12608](https://github.com/matrix-org/synapse/issues/12608))
-- Update to mypy 0.950. ([\#12650](https://github.com/matrix-org/synapse/issues/12650))
-- Use `Concatenate` to better annotate `_do_execute`. ([\#12666](https://github.com/matrix-org/synapse/issues/12666))
-- Use `ParamSpec` to refine type hints. ([\#12667](https://github.com/matrix-org/synapse/issues/12667))
-- Fix mypy against latest pillow stubs. ([\#12671](https://github.com/matrix-org/synapse/issues/12671))
 
 Synapse 1.58.1 (2022-05-05)
 ===========================

changelog.d/11507.feature

@@ -0,0 +1 @@
+Support [MSC3266](https://github.com/matrix-org/matrix-doc/pull/3266) room summaries over federation.

changelog.d/12168.feature

@@ -0,0 +1 @@
+Implement [changes](https://github.com/matrix-org/matrix-spec-proposals/pull/2285/commits/4a77139249c2e830aec3c7d6bd5501a514d1cc27) to [MSC2285 (hidden read receipts)](https://github.com/matrix-org/matrix-spec-proposals/pull/2285). Contributed by @SimonBrandner.

changelog.d/12273.bugfix

@@ -0,0 +1 @@
+Fix a bug introduced in Synapse v1.48.0 where latest thread reply provided failed to include the proper bundled aggregations.

changelog.d/12356.misc

@@ -0,0 +1 @@
+Fix scripts-dev to pass typechecking.

changelog.d/12406.feature

@@ -0,0 +1 @@
+Add a module API to allow modules to change actions for existing push rules of local users.

changelog.d/12452.feature

@@ -0,0 +1 @@
+Add the `notify_appservices_from_worker` configuration option (superseding `notify_appservices`) to allow a generic worker to be designated as the worker to send traffic to Application Services.

changelog.d/12480.misc

@@ -0,0 +1 @@
+Use supervisord to supervise Postgres and Caddy in the Complement image to reduce restart time.

changelog.d/12505.misc

@@ -0,0 +1 @@
+Use `make_awaitable` instead of `defer.succeed` for return values of mocks in tests.

changelog.d/12526.feature

@@ -0,0 +1 @@
+Add new `enable_registration_token_3pid_bypass` configuration option to allow registrations via token as an alternative to verifying a 3pid.

changelog.d/12531.misc

@@ -0,0 +1 @@
+Remove unused `# type: ignore`s.

changelog.d/12541.docker

@@ -0,0 +1 @@
+Explicitly opt-in to using [BuildKit-specific features](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md) in the Dockerfile. This fixes issues with building images in some GitLab CI environments.

changelog.d/12544.bugfix

@@ -0,0 +1 @@
+Fix a bug where attempting to send a large amount of read receipts to an application service all at once would result in duplicate content and abnormally high memory usage. Contributed by Brad & Nick @ Beeper.

changelog.d/12556.misc

@@ -0,0 +1 @@
+Release script: confirm the commit to be tagged before tagging.

changelog.d/12564.misc

@@ -0,0 +1 @@
+Consistently check if an object is a `frozendict`.

changelog.d/12568.misc

@@ -0,0 +1 @@
+Protect module callbacks with read semantics against cancellation.

changelog.d/12570.bugfix

@@ -0,0 +1 @@
+Fix a bug introduced in Synapse 1.57 which could cause `Failed to calculate hosts in room` errors to be logged for outbound federation.

changelog.d/12573.docker

@@ -0,0 +1 @@
+Update the "Build docker images" GitHub Actions workflow to use `docker/metadata-action` to generate docker image tags, instead of a custom shell script. Contributed by henryclw.

changelog.d/12576.misc

@@ -0,0 +1 @@
+Allow unused `#type: ignore` comments in bleeding edge CI jobs.

changelog.d/12577.misc

@@ -0,0 +1 @@
+Improve comments and error messages around access tokens.

changelog.d/12579.doc

@@ -0,0 +1 @@
+Add missing linebreak to pipx install instructions.

changelog.d/12580.bugfix

@@ -0,0 +1 @@
+Fix a long standing bug where status codes would almost always get logged as 200!, irrespective of the actual status code, when clients disconnect before a request has finished processing.

changelog.d/12581.misc

@@ -0,0 +1 @@
+Improve docstrings for the receipts store.

changelog.d/12582.misc

@@ -0,0 +1 @@
+Use constants for read-receipts in tests.

changelog.d/12586.misc

@@ -1 +0,0 @@
-Add `@cancellable` decorator, for use on endpoint methods that can be cancelled when clients disconnect.

changelog.d/12587.misc

@@ -0,0 +1 @@
+Log status code of cancelled requests as 499 and avoid logging stack traces for them.

changelog.d/12588.misc

@@ -1 +0,0 @@
-Add ability to cancel disconnected requests to `SynapseRequest`.

changelog.d/12589.misc

@@ -0,0 +1 @@
+Remove special-case for `twisted` logger from default log config.

changelog.d/12594.bugfix

@@ -0,0 +1 @@
+Fix race when persisting an event and deleting a room that could lead to outbound federation breaking.

changelog.d/12596.removal

@@ -0,0 +1 @@
+Remove unstable identifiers from [MSC3069](https://github.com/matrix-org/matrix-doc/pull/3069).

changelog.d/12597.removal

@@ -0,0 +1,2 @@
+Remove the unspecified `m.login.jwt` login type and the unstable `uk.half-shot.msc2778.login.application_service` from 
+[MSC2778](https://github.com/matrix-org/matrix-doc/pull/2778).

changelog.d/12599.misc

@@ -0,0 +1 @@
+Use `getClientAddress` instead of the deprecated `getClientIP`.

changelog.d/12602.misc

@@ -0,0 +1 @@
+Add link to documentation in Grafana Dashboard.

changelog.d/12608.misc

@@ -0,0 +1 @@
+Remove redundant lines of config from `mypy.ini`.

changelog.d/12610.misc

@@ -0,0 +1 @@
+Reduce log spam when running multiple event persisters.

changelog.d/12612.bugfix

@@ -0,0 +1 @@
+Fix a typo in the announcement text generated by the Synapse release development script.

changelog.d/12613.removal

@@ -0,0 +1 @@
+Synapse now requires at least Python 3.7.1 (up from 3.7.0), for compatibility with the latest Twisted trunk.

changelog.d/12614.misc

@@ -0,0 +1 @@
+Add extra debug logging to federation sender.

changelog.d/12616.misc

@@ -0,0 +1 @@
+Prevent remote homeservers from requesting local user device names by default.

changelog.d/12619.feature

@@ -0,0 +1 @@
+Add new `mau_appservice_trial_days` configuration option to specify a different trial period for users registered via an appservice.

changelog.d/12620.misc

@@ -0,0 +1 @@
+Add a consistency check on events which we read from the database.

changelog.d/12624.misc

@@ -0,0 +1 @@
+Remove use of constantly library and switch to enums for EventRedactBehaviour. Contributed by @andrewdoh.

changelog.d/12627.doc

@@ -0,0 +1 @@
+Fixes to the formatting of README.rst.

changelog.d/12630.misc

@@ -1 +0,0 @@
-Add a helper class for testing request cancellation.

changelog.d/12632.misc

@@ -0,0 +1 @@
+Remove unused code related to receipts.

changelog.d/12633.bugfix

@@ -0,0 +1 @@
+Fix a bug introduced in Synapse v1.53.0 where bundled aggregations for annotations/edits were incorrectly calculated.

changelog.d/12635.feature

@@ -0,0 +1 @@
+Implement [changes](https://github.com/matrix-org/matrix-spec-proposals/pull/2285/commits/4a77139249c2e830aec3c7d6bd5501a514d1cc27) to [MSC2285 (hidden read receipts)](https://github.com/matrix-org/matrix-spec-proposals/pull/2285). Contributed by @SimonBrandner.

changelog.d/12636.feature

@@ -0,0 +1 @@
+Implement [changes](https://github.com/matrix-org/matrix-spec-proposals/pull/2285/commits/4a77139249c2e830aec3c7d6bd5501a514d1cc27) to [MSC2285 (hidden read receipts)](https://github.com/matrix-org/matrix-spec-proposals/pull/2285). Contributed by @SimonBrandner.

changelog.d/12639.bugfix

@@ -0,0 +1 @@
+Add new `enable_registration_token_3pid_bypass` configuration option to allow registrations via token as an alternative to verifying a 3pid.

changelog.d/12650.misc

@@ -0,0 +1 @@
+Update to mypy 0.950.

changelog.d/12652.misc

@@ -0,0 +1 @@
+Move `pympler` back in to the `all` extras.

changelog.d/12656.misc

@@ -0,0 +1 @@
+Prevent memory leak from reoccurring when presence is disabled.

changelog.d/12657.bugfix

@@ -0,0 +1 @@
+Fix a long-standing bug where rooms containing power levels with string values could not be upgraded.

changelog.d/12663.misc

@@ -0,0 +1 @@
+Log status code of cancelled requests as 499 and avoid logging stack traces for them.

changelog.d/12664.doc

@@ -0,0 +1 @@
+Fix docs on how to run specific Complement tests using the `complement.sh` test runner.

changelog.d/12665.misc

@@ -0,0 +1 @@
+Fix spelling of `M_UNRECOGNIZED` in comments.

changelog.d/12666.misc

@@ -0,0 +1 @@
+Use `Concatenate` to better annotate `_do_execute`.

changelog.d/12667.misc

@@ -0,0 +1 @@
+Use `ParamSpec` to refine type hints.

changelog.d/12670.feature

@@ -0,0 +1 @@
+Implement [changes](https://github.com/matrix-org/matrix-spec-proposals/pull/2285/commits/4a77139249c2e830aec3c7d6bd5501a514d1cc27) to [MSC2285 (hidden read receipts)](https://github.com/matrix-org/matrix-spec-proposals/pull/2285). Contributed by @SimonBrandner.

changelog.d/12671.misc

@@ -0,0 +1 @@
+Fix mypy against latest pillow stubs.

changelog.d/12676.misc

@@ -1 +0,0 @@
-Improve documentation of the `synapse.push` module.

changelog.d/12677.misc

@@ -1 +0,0 @@
-Refactor functions to on `PushRuleEvaluatorForEvent`.

changelog.d/12679.misc

@@ -1 +0,0 @@
-Preparation for database schema simplifications: stop writing to `event_reference_hashes`.

changelog.d/12681.doc

@@ -0,0 +1 @@
+Add topics to synapse documentation.

changelog.d/12683.bugfix

@@ -1 +0,0 @@
-Fix a bug introduced in Synapse 1.57.0 where `/messages` would throw a 500 error when querying for a non-existent room.

changelog.d/12689.misc

@@ -1 +0,0 @@
-Refactor `EventContext` class.

changelog.d/12694.misc

@@ -1 +0,0 @@
-Capture the `Deferred` for request cancellation in `_AsyncResource`.

changelog.d/12695.misc

@@ -1 +0,0 @@
-Fixes an incorrect type hint for `Filter._check_event_relations`.

debian/changelog

@@ -1,11 +1,10 @@
-matrix-synapse-py3 (1.59.0~rc1) stable; urgency=medium
+matrix-synapse-py3 (1.58.2) UNRELEASED; urgency=medium
 
   * Adjust how the `exported-requirements.txt` file is generated as part of
     the process of building these packages. This affects the package
     maintainers only; end-users are unaffected.
-  * New Synapse release 1.59.0rc1.
 
- -- Synapse Packaging team <packages@matrix.org>  Tue, 10 May 2022 10:45:08 +0100
+ -- Synapse Packaging team <packages@matrix.org>  Fri, 06 May 2022 13:49:29 +0100
 
 matrix-synapse-py3 (1.58.1) stable; urgency=medium
 

docker/conf-workers/supervisord.conf.j2

@@ -9,7 +9,7 @@ user=root
 files = /etc/supervisor/conf.d/*.conf
 
 [program:nginx]
-command=/usr/local/bin/prefix-log /usr/sbin/nginx -g "daemon off;"
+command=/usr/sbin/nginx -g "daemon off;"
 priority=500
 stdout_logfile=/dev/stdout
 stdout_logfile_maxbytes=0
@@ -19,7 +19,7 @@ username=www-data
 autorestart=true
 
 [program:redis]
-command=/usr/local/bin/prefix-log /usr/bin/redis-server /etc/redis/redis.conf --daemonize no
+command=/usr/bin/redis-server /etc/redis/redis.conf --daemonize no
 priority=1
 stdout_logfile=/dev/stdout
 stdout_logfile_maxbytes=0
@@ -29,7 +29,7 @@ username=redis
 autorestart=true
 
 [program:synapse_main]
-command=/usr/local/bin/prefix-log /usr/local/bin/python -m synapse.app.homeserver --config-path="{{ main_config_path }}" --config-path=/conf/workers/shared.yaml
+command=/usr/local/bin/python -m synapse.app.homeserver --config-path="{{ main_config_path }}" --config-path=/conf/workers/shared.yaml
 priority=10
 # Log startup failures to supervisord's stdout/err
 # Regular synapse logs will still go in the configured data directory

docs/SUMMARY.md

@@ -88,19 +88,158 @@
   - [OpenTracing](opentracing.md)
   - [Database Schemas](development/database_schema.md)
   - [Experimental features](development/experimental_features.md)
-  - [Synapse Architecture]()
-    - [Log Contexts](log_contexts.md)
-    - [Replication](replication.md)
-    - [TCP Replication](tcp_replication.md)
-  - [Internal Documentation](development/internal_documentation/README.md)
+  - [Synapse Architecture](development/synapse_architechture.md)
+    - [Overview]()
+      - Flows of data through Synapse
+        - Starts with a Request/BG Job/Module
+        - Processing
+        - Maybe Response
+      - Config
+      - Request Handling
+        - Clients
+        - Federation
+        - Internal Processing - there could be an exception!
+      - Storage
+      - Federation
+      - External Entities: Remote homeservers, Appservices, Pushers, Identity Servers
+      - Modules
+      - Workers/Replication/Redis/Organization of a large homeserver instance
+  - [Configuration]()
+    - Experimental config options
+      - Define how long they should live after the corresponding msc has merged
+  - [Logging]()
+    - combine with [Log Contexts](log_contexts.md)
+  - [Storage]()
+    - What is current database schema?
+      - Start by linking script that can compile full schema
+      - Is there a tool that can visualize a postgres db? - Shay to check
+    - [Database Backends]()
+    - [Database Migrations]()
+      - Background [Schema] Updates
+    - [Caching]()
+      - Cache tuning
+      - How to check sizes
+        - Nice to have a metric for max cache sizes
+      - How to add a cache to a [storage] function
+        - Why would you
+      - How to invalidate (with support for workers)
+  - [Request Listening]()
+    - Mention that we use Twisted
+    - [Servlets and Resources]()
+      - When to use one over the other (always use a servlet?)
+      - Unstable/release-based endpoints
+      - Don't put too many servlets on the same resource
+    - [Handlers]()
+      - Machinery behind the servlet that does all the processing
+  - [Background Jobs]()
+  - [Federation]()
+    - Federation Catchup
+  - [Data Types]()
+    - [Users]()
+      - Allowed username characters
+    - [Devices]()
+    - [Events]()
+      - What is an event?
+      - Different event formats depending on room version
+      - Outlier events, rejected events, dropped events
+    - [Rooms]()
+      - [Room DAG concepts](development/room-dag-concepts.md)
+  - [Sync]()
+  - [State Resolution]()
+    - [The Auth Chain Difference Algorithm](auth_chain_difference_algorithm.md)
+  - [User Authentication]()
     - [Single Sign-On]()
       - [SAML](development/saml.md)
       - [CAS](development/cas.md)
-    - [Room DAG concepts](development/room-dag-concepts.md)
-    - [State Resolution]()
-      - [The Auth Chain Difference Algorithm](auth_chain_difference_algorithm.md)
-    - [Media Repository](media_repository.md)
-    - [Room and User Statistics](room_and_user_statistics.md)
+    - [User-Interactive Auth]()
+    - [Password-based]()
+    - [Password Auth Modules]()
+    - [Token-based]()
+  - [Rate Limiting]()
+    - Different rate limiting classes (Ratelimiter vs RequestRatelimiter)
+    - How does our rate limiting work
+    - When to use/add rate limiting
+    - Rate limiting for federation traffic
+  - [Media Repository](media_repository.md)
+  - [Email and HTML Templating]()
+  - [Presence]()
+    - What is Presence?
+    - How is it designed to work currently
+      - Why it is recommended to be disabled
+      - Why is this resource intensive
+        - Links and issues on how to fix this
+      - How it's implemented
+  - [Application Services]()
+  - [Push Notifications]()
+  - [Synapse Admin API]()
+  - [Synapse Modules]()
+  - [Workers]()
+    - Things to be mindful of in order to make your feature work with workers
+    - How to test a feature on workers
+  - [Replication](replication.md)
+    - [TCP Replication](tcp_replication.md)
+  - [Sources]()
+    - TypingSource, ReceiptSource, PresenceSource, etc
+    - Classes containing methods for storing and getting different types of data
+  - [Streams]()
+    - Stream Types (the ID of the stream)
+    - Stream Token: each entity added to the stream gets a unique, incremental token
+    - StreamReplication classes and clients
+    - How this interacts with sync?
+    - Stream token generators: Single Writer Only (sqlite), MultiWriterStreamIDGenerator (Postgres)
+  - [Notifier]()
+  - [Monitoring and Metrics]() # opentracing, phone home and metrics
+    - [Opentracing]()
+      - How does this work?
+      - How is this helpful?
+      - What's jaeger?
+      - How do I create and mutate a span?
+      - How are spans sent between homeservers?
+        - Only between whitelisted homeservers
+      - What areas are currently traced
+        - Database layer
+        - Servlets
+        - Crypto stuff
+    - [Metrics]()
+      - Prometheus
+      - How to make a new metrics / good practices
+        - What are all the current metrics and what each one means
+        - How to view in grafana
+          - Here's a quick example of making a new panel to visualize metrics
+        - Btw it would be nice to have a tooltip with an explainer for every graph in graphana
+      - [Room and User Statistics](room_and_user_statistics.md)  # TODO: This page is currently useless    
+  - [Testing]()
+    - [Unit Tests]()
+      - How to run the tests (partially in the contributing guide)
+        - IDE-specific examples
+        - How to run one test / one test case / run in parallel
+        - Documenting test environment variables: SYNAPSE_TEST_LOG_LEVEL, SYNAPSE_TEST_PERSISTENT_SQLITE_DB, SYNAPSE_POSTGRES etc.
+      - Test case versus test method
+        - prepare, make_homeserver, default_config, @override_config decorator, @parameterized decorator, pump/reactor.advance
+        - Test utilities: fake homeserver, federation, workers, channels, requests, HomeServerTestCase.helper
+          - How to use them in your tests at a high level
+          - How they are implemented
+      - Different inherited TestCase classes: HomeServerTestCase, FederatedHomeServerTestCase(sp?), TestCase
+      - Test pattern: How to start a request, check the state of the homeserver, respond to the request
+      - Ensuring the right servlets for your test are registered
+      - Good practice: use storage methods vs. tampering with the database directly (database schema changes)
+      - Dealing with deferreds/async (get_success/get_failure).
+      - Mocking - async mocks
+      - Organization of tests
+      - Skipping tests
+    - [Integration Tests]()
+        - [Complement]()
+            - Point to Complement's documentation on its structure and how to write a test
+            - How to run Complement against Synapse
+            - Document Synapse's Complement images and how they're built
+            - Which Complement build flags does Synapse support
+                - Unstable features
+                - Synapse's Complement Blacklist
+        - [Sytest]()
+            - You should prefer Complement :)
+            - Helper functions
+            - Dealing with race conditions
+            - Link to that one perl helper doc Matthew wrote
   - [Scripts]()
 
 # Other

docs/modules/spam_checker_callbacks.md

@@ -18,17 +18,6 @@ async def check_event_for_spam(event: "synapse.events.EventBase") -> Union[bool,
 
 Called when receiving an event from a client or via federation. The callback must return
 either:
-  - on `Decision.ALLOW`, the action is permitted.
-  - on `Decision.DENY`, the action is rejected with a default error message/code.
-  - on `Codes`, the action is rejected with a specific error message/code. In case
-      of doubt, use `Codes.FORBIDDEN`.
-  - (deprecated) on `False`, behave as `Decision.ALLOW`. Deprecated as methods in
-      this API are inconsistent, some expect `True` for `ALLOW` and others `True`
-      for `DENY`.
-  - (deprecated) on `True`, behave as `Decision.DENY`. Deprecated as methods in
-      this API are inconsistent, some expect `True` for `ALLOW` and others `True`
-      for `DENY`.
-
 - an error message string, to indicate the event must be rejected because of spam and 
   give a rejection reason to forward to clients;
 - the boolean `True`, to indicate that the event is spammy, but not provide further details; or

docs/replication.md

@@ -35,8 +35,3 @@ See [the TCP replication documentation](tcp_replication.md).
 There are read-only version of the synapse storage layer in
 `synapse/replication/slave/storage` that use the response of the
 replication API to invalidate their caches.
-
-### The TCP Replication Module
-Information about how the tcp replication module is structured, including how
-the classes interact, can be found in
-`synapse/replication/tcp/__init__.py`

docs/upgrade.md

@@ -101,36 +101,29 @@ To re-enable this functionality, set the
 homeserver config option to `true`.
 
 
-## Deprecation of the `synapse.app.appservice` and `synapse.app.user_dir` worker application types
+## Deprecation of the `synapse.app.appservice` worker application type
 
 The `synapse.app.appservice` worker application type allowed you to configure a
 single worker to use to notify application services of new events, as long
 as this functionality was disabled on the main process with `notify_appservices: False`.
-Further, the `synapse.app.user_dir` worker application type allowed you to configure
-a single worker to be responsible for updating the user directory, as long as this
-was disabled on the main process with `update_user_directory: False`.
 
 To unify Synapse's worker types, the `synapse.app.appservice` worker application
 type and the `notify_appservices` configuration option have been deprecated.
-The `synapse.app.user_dir` worker application type and `update_user_directory`
-configuration option have also been deprecated.
 
-To get the same functionality as was provided by the deprecated options, it's now recommended that the `synapse.app.generic_worker`
-worker application type is used and that the `notify_appservices_from_worker` and/or
-`update_user_directory_from_worker` options are set to the name of a worker.
+To get the same functionality, it's now recommended that the `synapse.app.generic_worker`
+worker application type is used and that the `notify_appservices_from_worker` option
+is set to the name of a worker.
 
-For the time being, the old options can be used alongside the new options to make
-it easier to transition between the two configurations, however please note that:
+For the time being, `notify_appservices_from_worker` can be used alongside
+`synapse.app.appservice` and `notify_appservices` to make it easier to transition
+between the two configurations, however please note that:
 
 - the options must not contradict each other (otherwise Synapse won't start); and
-- the `notify_appservices` and `update_user_directory` options will be removed in a future release of Synapse.
+- the `notify_appservices` option will be removed in a future release of Synapse.
 
-Please see the [*Notifying Application Services*][v1_59_notify_ases_from] and
-[*Updating the User Directory*][v1_59_update_user_dir] sections of the worker
-documentation for more information.
+Please see [the relevant section of the worker documentation][v1_59_notify_ases_from] for more information.
 
 [v1_59_notify_ases_from]: workers.md#notifying-application-services
-[v1_59_update_user_dir]: workers.md#updating-the-user-directory
 
 
 # Upgrading to v1.58.0

docs/usage/administration/useful_sql_for_admins.md

@@ -1,10 +1,7 @@
 ## Some useful SQL queries for Synapse Admins
 
 ## Size of full matrix db
-```sql
-SELECT pg_size_pretty( pg_database_size( 'matrix' ) );
-```
-
+`SELECT pg_size_pretty( pg_database_size( 'matrix' ) );`
 ### Result example:
 ``` 
 pg_size_pretty 
@@ -12,19 +9,39 @@ pg_size_pretty
  6420 MB
 (1 row)
 ```
+## Show top 20 larger rooms by state events count
+```sql
+SELECT r.name, s.room_id, s.current_state_events
+  FROM room_stats_current s
+  LEFT JOIN room_stats_state r USING (room_id)
+  ORDER BY current_state_events DESC
+  LIMIT 20;
+```
+
+and by state_group_events count:
+```sql
+SELECT rss.name, s.room_id, count(s.room_id) FROM state_groups_state s
+LEFT JOIN room_stats_state rss USING (room_id)
+GROUP BY s.room_id, rss.name        
+ORDER BY count(s.room_id) DESC
+LIMIT 20;
+```
+plus same, but with join removed for performance reasons:
+```sql
+SELECT s.room_id, count(s.room_id) FROM state_groups_state s
+GROUP BY s.room_id        
+ORDER BY count(s.room_id) DESC
+LIMIT 20;
+```
 
 ## Show top 20 larger tables by row count
 ```sql
-SELECT relname, n_live_tup AS "rows"
-  FROM pg_stat_user_tables
+SELECT relname, n_live_tup as rows
+  FROM pg_stat_user_tables 
   ORDER BY n_live_tup DESC
   LIMIT 20;
 ```
-This query is quick, but may be very approximate, for exact number of rows use:
-```sql
-SELECT COUNT(*) FROM <table_name>;
-```
-
+This query is quick, but may be very approximate, for exact number of rows use `SELECT COUNT(*) FROM <table_name>`.
 ### Result example:
 ```
 state_groups_state - 161687170
@@ -49,19 +66,46 @@ device_lists_stream - 326903
 user_directory_search - 316433
 ```
 
+## Show top 20 rooms by new events count in last 1 day:
+```sql
+SELECT e.room_id, r.name, COUNT(e.event_id) cnt FROM events e
+LEFT JOIN room_stats_state r USING (room_id)
+WHERE e.origin_server_ts >= DATE_PART('epoch', NOW() - INTERVAL '1 day') * 1000 GROUP BY e.room_id, r.name ORDER BY cnt DESC LIMIT 20;
+```
+
+## Show top 20 users on homeserver by sent events (messages) at last month:
+```sql
+SELECT user_id, SUM(total_events) 
+   FROM user_stats_historical
+   WHERE TO_TIMESTAMP(end_ts/1000) AT TIME ZONE 'UTC' > date_trunc('day', now() - interval '1 month')
+   GROUP BY user_id
+   ORDER BY SUM(total_events) DESC 
+   LIMIT 20;
+```
+
+## Show last 100 messages from needed user, with room names:
+```sql
+SELECT e.room_id, r.name, e.event_id, e.type, e.content, j.json FROM events e
+  LEFT JOIN event_json j USING (room_id)
+  LEFT JOIN room_stats_state r USING (room_id)
+  WHERE sender = '@LOGIN:example.com'
+  AND e.type = 'm.room.message'
+  ORDER BY stream_ordering DESC
+  LIMIT 100;
+```
+
 ## Show top 20 larger tables by storage size
 ```sql
 SELECT nspname || '.' || relname AS "relation",
-    pg_size_pretty(pg_total_relation_size(c.oid)) AS "total_size"
-  FROM pg_class c
-  LEFT JOIN pg_namespace n ON (n.oid = c.relnamespace)
+    pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size"
+  FROM pg_class C
+  LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
   WHERE nspname NOT IN ('pg_catalog', 'information_schema')
-    AND c.relkind <> 'i'
+    AND C.relkind <> 'i'
     AND nspname !~ '^pg_toast'
-  ORDER BY pg_total_relation_size(c.oid) DESC
+  ORDER BY pg_total_relation_size(C.oid) DESC
   LIMIT 20;
 ```
-
 ### Result example:
 ```
 public.state_groups_state - 27 GB
@@ -86,93 +130,8 @@ public.device_lists_remote_cache - 124 MB
 public.state_group_edges - 122 MB
 ```
 
-## Show top 20 larger rooms by state events count
-You get the same information when you use the
-[admin API](../../admin_api/rooms.md#list-room-api)
-and set parameter `order_by=state_events`.
-
-```sql
-SELECT r.name, s.room_id, s.current_state_events
-  FROM room_stats_current s
-  LEFT JOIN room_stats_state r USING (room_id)
-  ORDER BY current_state_events DESC
-  LIMIT 20;
-```
-
-and by state_group_events count:
-```sql
-SELECT rss.name, s.room_id, COUNT(s.room_id)
-  FROM state_groups_state s
-  LEFT JOIN room_stats_state rss USING (room_id)
-  GROUP BY s.room_id, rss.name
-  ORDER BY COUNT(s.room_id) DESC
-  LIMIT 20;
-```
-
-plus same, but with join removed for performance reasons:
-```sql
-SELECT s.room_id, COUNT(s.room_id)
-  FROM state_groups_state s
-  GROUP BY s.room_id 
-  ORDER BY COUNT(s.room_id) DESC
-  LIMIT 20;
-```
-
-## Show top 20 rooms by new events count in last 1 day:
-```sql
-SELECT e.room_id, r.name, COUNT(e.event_id) cnt
-  FROM events e
-  LEFT JOIN room_stats_state r USING (room_id)
-  WHERE e.origin_server_ts >= DATE_PART('epoch', NOW() - INTERVAL '1 day') * 1000
-  GROUP BY e.room_id, r.name 
-  ORDER BY cnt DESC
-  LIMIT 20;
-```
-
-## Show top 20 users on homeserver by sent events (messages) at last month:
-Caution. This query does not use any indexes, can be slow and create load on the database.
-```sql
-SELECT COUNT(*), sender
-  FROM events
-  WHERE (type = 'm.room.encrypted' OR type = 'm.room.message')
-    AND origin_server_ts >= DATE_PART('epoch', NOW() - INTERVAL '1 month') * 1000
-  GROUP BY sender
-  ORDER BY COUNT(*) DESC
-  LIMIT 20;
-```
-
-## Show last 100 messages from needed user, with room names:
-```sql
-SELECT e.room_id, r.name, e.event_id, e.type, e.content, j.json
-  FROM events e
-  LEFT JOIN event_json j USING (room_id)
-  LEFT JOIN room_stats_state r USING (room_id)
-  WHERE sender = '@LOGIN:example.com'
-    AND e.type = 'm.room.message'
-  ORDER BY stream_ordering DESC
-  LIMIT 100;
-```
-
 ## Show rooms with names, sorted by events in this rooms
-
-**Sort and order with bash**
-```bash
-echo "SELECT event_json.room_id, room_stats_state.name FROM event_json, room_stats_state \
-WHERE room_stats_state.room_id = event_json.room_id" | psql -d synapse -h localhost -U synapse_user -t \
-| sort | uniq -c | sort -n
-```
-Documentation for `psql` command line parameters: https://www.postgresql.org/docs/current/app-psql.html
-
-**Sort and order with SQL**
-```sql
-SELECT COUNT(*), event_json.room_id, room_stats_state.name
-  FROM event_json, room_stats_state
-  WHERE room_stats_state.room_id = event_json.room_id
-  GROUP BY event_json.room_id, room_stats_state.name
-  ORDER BY COUNT(*) DESC
-  LIMIT 50;
-```
-
+`echo "select event_json.room_id,room_stats_state.name from event_json,room_stats_state where room_stats_state.room_id=event_json.room_id" | psql synapse | sort | uniq -c | sort -n`
 ### Result example:
 ```
    9459  !FPUfgzXYWTKgIrwKxW:matrix.org              | This Week in Matrix
@@ -186,22 +145,12 @@ SELECT COUNT(*), event_json.room_id, room_stats_state.name
 ```
 
 ## Lookup room state info by list of room_id
-You get the same information when you use the
-[admin API](../../admin_api/rooms.md#room-details-api).
 ```sql
-SELECT rss.room_id, rss.name, rss.canonical_alias, rss.topic, rss.encryption,
-    rsc.joined_members, rsc.local_users_in_room, rss.join_rules
-  FROM room_stats_state rss
-  LEFT JOIN room_stats_current rsc USING (room_id)
-  WHERE room_id IN ( WHERE room_id IN (
-    '!OGEhHVWSdvArJzumhm:matrix.org',
-    '!YTvKGNlinIzlkMTVRl:matrix.org' 
-  );
-```
-
-## Show users and devices that have not been online for a while
-```sql
-SELECT user_id, device_id, user_agent, TO_TIMESTAMP(last_seen / 1000) AS "last_seen"
-  FROM devices
-  WHERE last_seen < DATE_PART('epoch', NOW() - INTERVAL '3 month') * 1000;
-```
+SELECT rss.room_id, rss.name, rss.canonical_alias, rss.topic, rss.encryption, rsc.joined_members, rsc.local_users_in_room, rss.join_rules
+FROM room_stats_state rss
+LEFT JOIN room_stats_current rsc USING (room_id)
+WHERE room_id IN (WHERE room_id IN (
+ '!OGEhHVWSdvArJzumhm:matrix.org',
+ '!YTvKGNlinIzlkMTVRl:matrix.org'
+)
+```

docs/workers.md

@@ -426,7 +426,7 @@ the shared configuration would include:
 run_background_tasks_on: background_worker
 ```
 
-You might also wish to investigate the `update_user_directory_from_worker` and
+You might also wish to investigate the `update_user_directory` and
 `media_instance_running_background_jobs` settings.
 
 An example for a dedicated background worker instance:
@@ -435,26 +435,9 @@ An example for a dedicated background worker instance:
 {{#include systemd-with-workers/workers/background_worker.yaml}}
 ```
 
-#### Updating the User Directory
-
-You can designate one generic worker to update the user directory.
-
-Specify its name in the shared configuration as follows:
-
-```yaml
-update_user_directory_from_worker: worker_name
-```
-
-This work cannot be load-balanced; please ensure the main process is restarted
-after setting this option in the shared configuration!
-
-This style of configuration supersedes the legacy `synapse.app.user_dir`
-worker application type.
-
-
 #### Notifying Application Services
 
-You can designate one generic worker to send output traffic to Application Services.
+You can designate one worker to send output traffic to Application Services.
 
 Specify its name in the shared configuration as follows:
 
@@ -487,7 +470,7 @@ pusher_instances:
 
 ### `synapse.app.appservice`
 
-**Deprecated as of Synapse v1.59.** [Use `synapse.app.generic_worker` with the
+**Deprecated as of Synapse v1.58.** [Use `synapse.app.generic_worker` with the
 `notify_appservices_from_worker` option instead.](#notifying-application-services)
 
 Handles sending output traffic to Application Services. Doesn't handle any
@@ -557,9 +540,6 @@ Note that if a reverse proxy is used , then `/_matrix/media/` must be routed for
 
 ### `synapse.app.user_dir`
 
-**Deprecated as of Synapse v1.59.** [Use `synapse.app.generic_worker` with the
-`update_user_directory_from_worker` option instead.](#updating-the-user-directory)
-
 Handles searches in the user directory. It can handle REST endpoints matching
 the following regular expressions:
 

pyproject.toml

@@ -54,7 +54,7 @@ skip_gitignore = true
 
 [tool.poetry]
 name = "matrix-synapse"
-version = "1.59.0rc1"
+version = "1.58.1"
 description = "Homeserver for the Matrix decentralised comms protocol"
 authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
 license = "Apache-2.0"

scripts-dev/complement.sh

@@ -43,8 +43,6 @@ fi
 # Build the base Synapse image from the local checkout
 docker build -t matrixdotorg/synapse -f "docker/Dockerfile" .
 
-extra_test_args=()
-
 # If we're using workers, modify the docker files slightly.
 if [[ -n "$WORKERS" ]]; then
   # Build the workers docker image (from the base Synapse image).
@@ -54,14 +52,7 @@ if [[ -n "$WORKERS" ]]; then
   COMPLEMENT_DOCKERFILE=SynapseWorkers.Dockerfile
 
   # And provide some more configuration to complement.
-
-  # It can take quite a while to spin up a worker-mode Synapse for the first
-  # time (the main problem is that we start 14 python processes for each test,
-  # and complement likes to do two of them in parallel).
-  export COMPLEMENT_SPAWN_HS_TIMEOUT_SECS=120
-
-  # ... and it takes longer than 10m to run the whole suite.
-  extra_test_args+=("-timeout=60m")
+  export COMPLEMENT_SPAWN_HS_TIMEOUT_SECS=60
 else
   export COMPLEMENT_BASE_IMAGE=complement-synapse
   COMPLEMENT_DOCKERFILE=Dockerfile
@@ -73,4 +64,4 @@ docker build -t $COMPLEMENT_BASE_IMAGE -f "docker/complement/$COMPLEMENT_DOCKERF
 # Run the tests!
 echo "Images built; running complement"
 cd "$COMPLEMENT_DIR"
-go test -v -tags synapse_blacklist,msc2716,msc3030,faster_joins -count=1 "${extra_test_args[@]}" "$@" ./tests/...
+go test -v -tags synapse_blacklist,msc2716,msc3030,faster_joins -count=1 "$@" ./tests/...

synapse/api/filtering.py

@@ -19,7 +19,6 @@ from typing import (
     TYPE_CHECKING,
     Awaitable,
     Callable,
-    Collection,
     Dict,
     Iterable,
     List,
@@ -445,9 +444,9 @@ class Filter:
         return room_ids
 
     async def _check_event_relations(
-        self, events: Collection[FilterEvent]
+        self, events: Iterable[FilterEvent]
     ) -> List[FilterEvent]:
-        # The event IDs to check, mypy doesn't understand the isinstance check.
+        # The event IDs to check, mypy doesn't understand the ifinstance check.
         event_ids = [event.event_id for event in events if isinstance(event, EventBase)]  # type: ignore[attr-defined]
         event_ids_to_keep = set(
             await self._store.events_have_relations(

synapse/app/admin_cmd.py

@@ -210,7 +210,7 @@ def start(config_options: List[str]) -> None:
         config.logging.no_redirect_stdio = True
 
     # Explicitly disable background processes
-    config.worker.should_update_user_directory = False
+    config.server.update_user_directory = False
     config.worker.run_background_tasks = False
     config.worker.start_pushers = False
     config.worker.pusher_shard_config.instances = []

synapse/app/generic_worker.py

@@ -441,6 +441,22 @@ def start(config_options: List[str]) -> None:
         "synapse.app.user_dir",
     )
 
+    if config.worker.worker_app == "synapse.app.user_dir":
+        if config.server.update_user_directory:
+            sys.stderr.write(
+                "\nThe update_user_directory must be disabled in the main synapse process"
+                "\nbefore they can be run in a separate worker."
+                "\nPlease add ``update_user_directory: false`` to the main config"
+                "\n"
+            )
+            sys.exit(1)
+
+        # Force the pushers to start since they will be disabled in the main config
+        config.server.update_user_directory = True
+    else:
+        # For other worker types we force this to off.
+        config.server.update_user_directory = False
+
     synapse.events.USE_FROZEN_DICTS = config.server.use_frozen_dicts
     synapse.util.caches.TRACK_MEMORY_USAGE = config.caches.track_memory_usage
 

synapse/config/experimental.py

@@ -81,6 +81,3 @@ class ExperimentalConfig(Config):
 
         # MSC2815 (allow room moderators to view redacted event content)
         self.msc2815_enabled: bool = experimental.get("msc2815_enabled", False)
-
-        # MSC3786 (Add a default push rule to ignore m.room.server_acl events)
-        self.msc3786_enabled: bool = experimental.get("msc3786_enabled", False)

synapse/config/server.py

@@ -319,6 +319,10 @@ class ServerConfig(Config):
                 self.presence_router_config,
             ) = load_module(presence_router_config, ("presence", "presence_router"))
 
+        # Whether to update the user directory or not. This should be set to
+        # false only if we are updating the user directory in a worker
+        self.update_user_directory = config.get("update_user_directory", True)
+
         # whether to enable the media repository endpoints. This should be set
         # to false if the media repository is running as a separate endpoint;
         # doing so ensures that we will not run cache cleanup jobs on the

synapse/config/workers.py

@@ -311,13 +311,6 @@ class WorkerConfig(Config):
             new_option_name="notify_appservices_from_worker",
         )
 
-        self.should_update_user_directory = self._should_this_worker_perform_duty(
-            config,
-            legacy_master_option_name="update_user_directory",
-            legacy_worker_app_name="synapse.app.user_dir",
-            new_option_name="update_user_directory_from_worker",
-        )
-
     def _should_this_worker_perform_duty(
         self,
         config: Dict[str, Any],

synapse/events/snapshot.py

@@ -15,10 +15,12 @@ from typing import TYPE_CHECKING, List, Optional, Tuple, Union
 
 import attr
 from frozendict import frozendict
-from typing_extensions import Literal
+
+from twisted.internet.defer import Deferred
 
 from synapse.appservice import ApplicationService
 from synapse.events import EventBase
+from synapse.logging.context import make_deferred_yieldable, run_in_background
 from synapse.types import JsonDict, StateMap
 
 if TYPE_CHECKING:
@@ -58,9 +60,6 @@ class EventContext:
             If ``state_group`` is None (ie, the event is an outlier),
             ``state_group_before_event`` will always also be ``None``.
 
-        state_delta_due_to_event: If `state_group` and `state_group_before_event` are not None
-            then this is the delta of the state between the two groups.
-
         prev_group: If it is known, ``state_group``'s prev_group. Note that this being
             None does not necessarily mean that ``state_group`` does not have
             a prev_group!
@@ -79,47 +78,73 @@ class EventContext:
         app_service: If this event is being sent by a (local) application service, that
             app service.
 
+        _current_state_ids: The room state map, including this event - ie, the state
+            in ``state_group``.
+
+            (type, state_key) -> event_id
+
+            For an outlier, this is {}
+
+            Note that this is a private attribute: it should be accessed via
+            ``get_current_state_ids``. _AsyncEventContext impl calculates this
+            on-demand: it will be None until that happens.
+
+        _prev_state_ids: The room state map, excluding this event - ie, the state
+            in ``state_group_before_event``. For a non-state
+            event, this will be the same as _current_state_events.
+
+            Note that it is a completely different thing to prev_group!
+
+            (type, state_key) -> event_id
+
+            For an outlier, this is {}
+
+            As with _current_state_ids, this is a private attribute. It should be
+            accessed via get_prev_state_ids.
+
         partial_state: if True, we may be storing this event with a temporary,
             incomplete state.
     """
 
-    _storage: "Storage"
-    rejected: Union[Literal[False], str] = False
+    rejected: Union[bool, str] = False
     _state_group: Optional[int] = None
     state_group_before_event: Optional[int] = None
-    _state_delta_due_to_event: Optional[StateMap[str]] = None
     prev_group: Optional[int] = None
     delta_ids: Optional[StateMap[str]] = None
     app_service: Optional[ApplicationService] = None
 
+    _current_state_ids: Optional[StateMap[str]] = None
+    _prev_state_ids: Optional[StateMap[str]] = None
+
     partial_state: bool = False
 
     @staticmethod
     def with_state(
-        storage: "Storage",
         state_group: Optional[int],
         state_group_before_event: Optional[int],
-        state_delta_due_to_event: Optional[StateMap[str]],
+        current_state_ids: Optional[StateMap[str]],
+        prev_state_ids: Optional[StateMap[str]],
         partial_state: bool,
         prev_group: Optional[int] = None,
         delta_ids: Optional[StateMap[str]] = None,
     ) -> "EventContext":
         return EventContext(
-            storage=storage,
+            current_state_ids=current_state_ids,
+            prev_state_ids=prev_state_ids,
             state_group=state_group,
             state_group_before_event=state_group_before_event,
-            state_delta_due_to_event=state_delta_due_to_event,
             prev_group=prev_group,
             delta_ids=delta_ids,
             partial_state=partial_state,
         )
 
     @staticmethod
-    def for_outlier(
-        storage: "Storage",
-    ) -> "EventContext":
+    def for_outlier() -> "EventContext":
         """Return an EventContext instance suitable for persisting an outlier event"""
-        return EventContext(storage=storage)
+        return EventContext(
+            current_state_ids={},
+            prev_state_ids={},
+        )
 
     async def serialize(self, event: EventBase, store: "DataStore") -> JsonDict:
         """Converts self to a type that can be serialized as JSON, and then
@@ -132,14 +157,24 @@ class EventContext:
             The serialized event.
         """
 
+        # We don't serialize the full state dicts, instead they get pulled out
+        # of the DB on the other side. However, the other side can't figure out
+        # the prev_state_ids, so if we're a state event we include the event
+        # id that we replaced in the state.
+        if event.is_state():
+            prev_state_ids = await self.get_prev_state_ids()
+            prev_state_id = prev_state_ids.get((event.type, event.state_key))
+        else:
+            prev_state_id = None
+
         return {
+            "prev_state_id": prev_state_id,
+            "event_type": event.type,
+            "event_state_key": event.get_state_key(),
             "state_group": self._state_group,
             "state_group_before_event": self.state_group_before_event,
             "rejected": self.rejected,
             "prev_group": self.prev_group,
-            "state_delta_due_to_event": _encode_state_dict(
-                self._state_delta_due_to_event
-            ),
             "delta_ids": _encode_state_dict(self.delta_ids),
             "app_service_id": self.app_service.id if self.app_service else None,
             "partial_state": self.partial_state,
@@ -157,16 +192,16 @@ class EventContext:
         Returns:
             The event context.
         """
-        context = EventContext(
+        context = _AsyncEventContextImpl(
             # We use the state_group and prev_state_id stuff to pull the
             # current_state_ids out of the DB and construct prev_state_ids.
             storage=storage,
+            prev_state_id=input["prev_state_id"],
+            event_type=input["event_type"],
+            event_state_key=input["event_state_key"],
             state_group=input["state_group"],
             state_group_before_event=input["state_group_before_event"],
             prev_group=input["prev_group"],
-            state_delta_due_to_event=_decode_state_dict(
-                input["state_delta_due_to_event"]
-            ),
             delta_ids=_decode_state_dict(input["delta_ids"]),
             rejected=input["rejected"],
             partial_state=input.get("partial_state", False),
@@ -214,15 +249,8 @@ class EventContext:
         if self.rejected:
             raise RuntimeError("Attempt to access state_ids of rejected event")
 
-        assert self._state_delta_due_to_event is not None
-
-        prev_state_ids = await self.get_prev_state_ids()
-
-        if self._state_delta_due_to_event:
-            prev_state_ids = dict(prev_state_ids)
-            prev_state_ids.update(self._state_delta_due_to_event)
-
-        return prev_state_ids
+        await self._ensure_fetched()
+        return self._current_state_ids
 
     async def get_prev_state_ids(self) -> StateMap[str]:
         """
@@ -237,10 +265,94 @@ class EventContext:
             Maps a (type, state_key) to the event ID of the state event matching
             this tuple.
         """
-        assert self.state_group_before_event is not None
-        return await self._storage.state.get_state_ids_for_group(
-            self.state_group_before_event
+        await self._ensure_fetched()
+        # There *should* be previous state IDs now.
+        assert self._prev_state_ids is not None
+        return self._prev_state_ids
+
+    def get_cached_current_state_ids(self) -> Optional[StateMap[str]]:
+        """Gets the current state IDs if we have them already cached.
+
+        It is an error to access this for a rejected event, since rejected state should
+        not make it into the room state. This method will raise an exception if
+        ``rejected`` is set.
+
+        Returns:
+            Returns None if we haven't cached the state or if state_group is None
+            (which happens when the associated event is an outlier).
+
+            Otherwise, returns the the current state IDs.
+        """
+        if self.rejected:
+            raise RuntimeError("Attempt to access state_ids of rejected event")
+
+        return self._current_state_ids
+
+    async def _ensure_fetched(self) -> None:
+        return None
+
+
+@attr.s(slots=True)
+class _AsyncEventContextImpl(EventContext):
+    """
+    An implementation of EventContext which fetches _current_state_ids and
+    _prev_state_ids from the database on demand.
+
+    Attributes:
+
+        _storage
+
+        _fetching_state_deferred: Resolves when *_state_ids have been calculated.
+            None if we haven't started calculating yet
+
+        _event_type: The type of the event the context is associated with.
+
+        _event_state_key: The state_key of the event the context is associated with.
+
+        _prev_state_id: If the event associated with the context is a state event,
+            then `_prev_state_id` is the event_id of the state that was replaced.
+    """
+
+    # This needs to have a default as we're inheriting
+    _storage: "Storage" = attr.ib(default=None)
+    _prev_state_id: Optional[str] = attr.ib(default=None)
+    _event_type: str = attr.ib(default=None)
+    _event_state_key: Optional[str] = attr.ib(default=None)
+    _fetching_state_deferred: Optional["Deferred[None]"] = attr.ib(default=None)
+
+    async def _ensure_fetched(self) -> None:
+        if not self._fetching_state_deferred:
+            self._fetching_state_deferred = run_in_background(self._fill_out_state)
+
+        await make_deferred_yieldable(self._fetching_state_deferred)
+
+    async def _fill_out_state(self) -> None:
+        """Called to populate the _current_state_ids and _prev_state_ids
+        attributes by loading from the database.
+        """
+        if self.state_group is None:
+            # No state group means the event is an outlier. Usually the state_ids dicts are also
+            # pre-set to empty dicts, but they get reset when the context is serialized, so set
+            # them to empty dicts again here.
+            self._current_state_ids = {}
+            self._prev_state_ids = {}
+            return
+
+        current_state_ids = await self._storage.state.get_state_ids_for_group(
+            self.state_group
         )
+        # Set this separately so mypy knows current_state_ids is not None.
+        self._current_state_ids = current_state_ids
+        if self._event_state_key is not None:
+            self._prev_state_ids = dict(current_state_ids)
+
+            key = (self._event_type, self._event_state_key)
+            if self._prev_state_id:
+                self._prev_state_ids[key] = self._prev_state_id
+            else:
+                self._prev_state_ids.pop(key, None)
+        else:
+            self._prev_state_ids = current_state_ids
 
 
 def _encode_state_dict(

synapse/events/spamcheck.py

@@ -27,10 +27,9 @@ from typing import (
     Union,
 )
 
-from synapse.api.errors import Codes
 from synapse.rest.media.v1._base import FileInfo
 from synapse.rest.media.v1.media_storage import ReadableFileWrapper
-from synapse.spam_checker_api import ALLOW, Decision, RegistrationBehaviour
+from synapse.spam_checker_api import RegistrationBehaviour
 from synapse.types import RoomAlias, UserProfile
 from synapse.util.async_helpers import delay_cancellation, maybe_awaitable
 
@@ -40,34 +39,17 @@ if TYPE_CHECKING:
 
 logger = logging.getLogger(__name__)
 
-
-DEPRECATED_BOOL = bool
-
 CHECK_EVENT_FOR_SPAM_CALLBACK = Callable[
     ["synapse.events.EventBase"],
-    Awaitable[Union[ALLOW, Codes, str, DEPRECATED_BOOL]],
-]
-USER_MAY_JOIN_ROOM_CALLBACK = Callable[
-    [str, str, bool], Awaitable[Union[ALLOW, Codes, DEPRECATED_BOOL]]
-]
-USER_MAY_INVITE_CALLBACK = Callable[
-    [str, str, str], Awaitable[Union[ALLOW, Codes, DEPRECATED_BOOL]]
-]
-USER_MAY_SEND_3PID_INVITE_CALLBACK = Callable[
-    [str, str, str, str], Awaitable[Union[ALLOW, Codes, DEPRECATED_BOOL]]
-]
-USER_MAY_CREATE_ROOM_CALLBACK = Callable[
-    [str], Awaitable[Union[ALLOW, Codes, DEPRECATED_BOOL]]
-]
-USER_MAY_CREATE_ROOM_ALIAS_CALLBACK = Callable[
-    [str, RoomAlias], Awaitable[Union[ALLOW, Codes, DEPRECATED_BOOL]]
-]
-USER_MAY_PUBLISH_ROOM_CALLBACK = Callable[
-    [str, str], Awaitable[Union[ALLOW, Codes, DEPRECATED_BOOL]]
-]
-CHECK_USERNAME_FOR_SPAM_CALLBACK = Callable[
-    [UserProfile], Awaitable[Union[ALLOW, Codes, DEPRECATED_BOOL]]
+    Awaitable[Union[bool, str]],
 ]
+USER_MAY_JOIN_ROOM_CALLBACK = Callable[[str, str, bool], Awaitable[bool]]
+USER_MAY_INVITE_CALLBACK = Callable[[str, str, str], Awaitable[bool]]
+USER_MAY_SEND_3PID_INVITE_CALLBACK = Callable[[str, str, str, str], Awaitable[bool]]
+USER_MAY_CREATE_ROOM_CALLBACK = Callable[[str], Awaitable[bool]]
+USER_MAY_CREATE_ROOM_ALIAS_CALLBACK = Callable[[str, RoomAlias], Awaitable[bool]]
+USER_MAY_PUBLISH_ROOM_CALLBACK = Callable[[str, str], Awaitable[bool]]
+CHECK_USERNAME_FOR_SPAM_CALLBACK = Callable[[UserProfile], Awaitable[bool]]
 LEGACY_CHECK_REGISTRATION_FOR_SPAM_CALLBACK = Callable[
     [
         Optional[dict],
@@ -83,11 +65,11 @@ CHECK_REGISTRATION_FOR_SPAM_CALLBACK = Callable[
         Collection[Tuple[str, str]],
         Optional[str],
     ],
-    Awaitable[Union[RegistrationBehaviour, Codes]],
+    Awaitable[RegistrationBehaviour],
 ]
 CHECK_MEDIA_FILE_FOR_SPAM_CALLBACK = Callable[
     [ReadableFileWrapper, FileInfo],
-    Awaitable[Union[ALLOW, Codes, DEPRECATED_BOOL]],
+    Awaitable[bool],
 ]
 
 
@@ -258,7 +240,7 @@ class SpamChecker:
 
     async def check_event_for_spam(
         self, event: "synapse.events.EventBase"
-    ) -> Union[ALLOW, Codes, str]:
+    ) -> Union[bool, str]:
         """Checks if a given event is considered "spammy" by this server.
 
         If the server considers an event spammy, then it will be rejected if
@@ -269,29 +251,19 @@ class SpamChecker:
             event: the event to be checked
 
         Returns:
-            - on `ALLOW`, the event is considered good (non-spammy) and should
-                be let through. Other spamcheck filters may still reject it.
-            - on `Codes`, the event is considered spammy and is rejected with a specific
-                error message/code.
-            - on `str`, the event is considered spammy and the string is used as error
-                message.
+            True or a string if the event is spammy. If a string is returned it
+            will be used as the error message returned to the user.
         """
         for callback in self._check_event_for_spam_callbacks:
-            res: Union[ALLOW, Codes, str, DEPRECATED_BOOL] = await delay_cancellation(
-                callback(event)
-            )
-            if res is False or res is ALLOW:
-                continue
-            elif res is True:
-                return Codes.FORBIDDEN
-            else:
+            res: Union[bool, str] = await delay_cancellation(callback(event))
+            if res:
                 return res
 
-        return ALLOW
+        return False
 
     async def user_may_join_room(
         self, user_id: str, room_id: str, is_invited: bool
-    ) -> Decision:
+    ) -> bool:
         """Checks if a given users is allowed to join a room.
         Not called when a user creates a room.
 
@@ -301,54 +273,48 @@ class SpamChecker:
             is_invited: Whether the user is invited into the room
 
         Returns:
-            - on `ALLOW`, the action is permitted.
-            - on `Codes`, the action is rejected with a specific error message/code.
+            Whether the user may join the room
         """
         for callback in self._user_may_join_room_callbacks:
             may_join_room = await delay_cancellation(
                 callback(user_id, room_id, is_invited)
             )
-            if may_join_room is True or may_join_room is ALLOW:
-                continue
-            elif may_join_room is False:
-                return Codes.FORBIDDEN
-            else:
-                return may_join_room
+            if may_join_room is False:
+                return False
 
-        return ALLOW
+        return True
 
     async def user_may_invite(
         self, inviter_userid: str, invitee_userid: str, room_id: str
-    ) -> Decision:
+    ) -> bool:
         """Checks if a given user may send an invite
 
+        If this method returns false, the invite will be rejected.
+
         Args:
             inviter_userid: The user ID of the sender of the invitation
             invitee_userid: The user ID targeted in the invitation
             room_id: The room ID
 
         Returns:
-            - on `ALLOW`, the action is permitted.
-            - on `Codes`, the action is rejected with a specific error message/code.
+            True if the user may send an invite, otherwise False
         """
         for callback in self._user_may_invite_callbacks:
             may_invite = await delay_cancellation(
                 callback(inviter_userid, invitee_userid, room_id)
             )
-            if may_invite is True or may_invite is ALLOW:
-                continue
-            elif may_invite is False:
-                return Codes.FORBIDDEN
-            else:
-                return may_invite
+            if may_invite is False:
+                return False
 
-        return ALLOW
+        return True
 
     async def user_may_send_3pid_invite(
         self, inviter_userid: str, medium: str, address: str, room_id: str
-    ) -> Decision:
+    ) -> bool:
         """Checks if a given user may invite a given threepid into the room
 
+        If this method returns false, the threepid invite will be rejected.
+
         Note that if the threepid is already associated with a Matrix user ID, Synapse
         will call user_may_invite with said user ID instead.
 
@@ -359,94 +325,78 @@ class SpamChecker:
             room_id: The room ID
 
         Returns:
-            - on `ALLOW`, the action is permitted.
-            - on `Codes`, the action is rejected with a specific error message/code.
+            True if the user may send the invite, otherwise False
         """
         for callback in self._user_may_send_3pid_invite_callbacks:
             may_send_3pid_invite = await delay_cancellation(
                 callback(inviter_userid, medium, address, room_id)
             )
-            if may_send_3pid_invite is True or may_send_3pid_invite is ALLOW:
-                continue
-            elif may_send_3pid_invite is False:
-                return Codes.FORBIDDEN
-            else:
-                return may_send_3pid_invite
+            if may_send_3pid_invite is False:
+                return False
 
-        return ALLOW
+        return True
 
-    async def user_may_create_room(self, userid: str) -> Decision:
+    async def user_may_create_room(self, userid: str) -> bool:
         """Checks if a given user may create a room
 
+        If this method returns false, the creation request will be rejected.
+
         Args:
             userid: The ID of the user attempting to create a room
 
         Returns:
-            - on `ALLOW`, the action is permitted.
-            - on `Codes`, the action is rejected with a specific error message/code.
+            True if the user may create a room, otherwise False
         """
         for callback in self._user_may_create_room_callbacks:
             may_create_room = await delay_cancellation(callback(userid))
-            if may_create_room is True or may_create_room is ALLOW:
-                continue
-            elif may_create_room is False:
-                return Codes.FORBIDDEN
-            else:
-                return may_create_room
+            if may_create_room is False:
+                return False
 
-        return ALLOW
+        return True
 
     async def user_may_create_room_alias(
         self, userid: str, room_alias: RoomAlias
-    ) -> Decision:
+    ) -> bool:
         """Checks if a given user may create a room alias
 
+        If this method returns false, the association request will be rejected.
+
         Args:
             userid: The ID of the user attempting to create a room alias
             room_alias: The alias to be created
 
         Returns:
-            - on `ALLOW`, the action is permitted.
-            - on `Codes`, the action is rejected with a specific error message/code.
+            True if the user may create a room alias, otherwise False
         """
         for callback in self._user_may_create_room_alias_callbacks:
             may_create_room_alias = await delay_cancellation(
                 callback(userid, room_alias)
             )
-            if may_create_room_alias is True or may_create_room_alias is ALLOW:
-                continue
-            elif may_create_room_alias is False:
-                return Codes.FORBIDDEN
-            else:
-                return may_create_room_alias
+            if may_create_room_alias is False:
+                return False
 
-        return ALLOW
+        return True
 
-    async def user_may_publish_room(
-        self, userid: str, room_id: str
-    ) -> Union[ALLOW, Codes, DEPRECATED_BOOL]:
+    async def user_may_publish_room(self, userid: str, room_id: str) -> bool:
         """Checks if a given user may publish a room to the directory
 
+        If this method returns false, the publish request will be rejected.
+
         Args:
             userid: The user ID attempting to publish the room
             room_id: The ID of the room that would be published
 
         Returns:
-            - on `ALLOW`, the action is permitted.
-            - on `Codes`, the action is rejected with a specific error message/code.
+            True if the user may publish the room, otherwise False
         """
         for callback in self._user_may_publish_room_callbacks:
             may_publish_room = await delay_cancellation(callback(userid, room_id))
-            if may_publish_room is True or may_publish_room is ALLOW:
-                continue
-            elif may_publish_room is False:
-                return Codes.FORBIDDEN
-            else:
-                return may_publish_room
+            if may_publish_room is False:
+                return False
 
-        return ALLOW
+        return True
 
-    async def check_username_for_spam(self, user_profile: UserProfile) -> Decision:
+    async def check_username_for_spam(self, user_profile: UserProfile) -> bool:
         """Checks if a user ID or display name are considered "spammy" by this server.
 
         If the server considers a username spammy, then it will not be included in
@@ -459,21 +409,15 @@ class SpamChecker:
                 * avatar_url
 
         Returns:
-            - on `ALLOW`, the action is permitted.
-            - on `Codes`, the action is rejected with a specific error message/code.
+            True if the user is spammy.
         """
         for callback in self._check_username_for_spam_callbacks:
             # Make a copy of the user profile object to ensure the spam checker cannot
             # modify it.
-            is_spam = await delay_cancellation(callback(user_profile.copy()))
-            if is_spam is False or is_spam is ALLOW:
-                continue
-            elif is_spam is True:
-                return Codes.FORBIDDEN
-            else:
-                return is_spam
+            if await delay_cancellation(callback(user_profile.copy())):
+                return True
 
-        return ALLOW
+        return False
 
     async def check_registration_for_spam(
         self,
@@ -501,8 +445,6 @@ class SpamChecker:
             behaviour = await delay_cancellation(
                 callback(email_threepid, username, request_info, auth_provider_id)
             )
-            if isinstance(behaviour, Codes):
-                return behaviour
             assert isinstance(behaviour, RegistrationBehaviour)
             if behaviour != RegistrationBehaviour.ALLOW:
                 return behaviour
@@ -511,7 +453,7 @@ class SpamChecker:
 
     async def check_media_file_for_spam(
         self, file_wrapper: ReadableFileWrapper, file_info: FileInfo
-    ) -> Decision:
+    ) -> bool:
         """Checks if a piece of newly uploaded media should be blocked.
 
         This will be called for local uploads, downloads of remote media, each
@@ -533,22 +475,19 @@ class SpamChecker:
 
                 return False
 
+
         Args:
             file: An object that allows reading the contents of the media.
             file_info: Metadata about the file.
 
         Returns:
-            - on `ALLOW`, the action is permitted.
-            - on `Codes`, the action is rejected with a specific error message/code.
+            True if the media should be blocked or False if it should be
+            allowed.
         """
 
         for callback in self._check_media_file_for_spam_callbacks:
-            is_spam = await delay_cancellation(callback(file_wrapper, file_info))
-            if is_spam is False or is_spam is ALLOW:
-                continue
-            elif is_spam is True:
-                return Codes.FORBIDDEN
-            else:
-                return is_spam
+            spam = await delay_cancellation(callback(file_wrapper, file_info))
+            if spam:
+                return True
 
-        return ALLOW
+        return False

synapse/federation/federation_base.py

@@ -15,7 +15,6 @@
 import logging
 from typing import TYPE_CHECKING
 
-import synapse
 from synapse.api.constants import MAX_DEPTH, EventContentFields, EventTypes, Membership
 from synapse.api.errors import Codes, SynapseError
 from synapse.api.room_versions import EventFormatVersions, RoomVersion
@@ -99,9 +98,9 @@ class FederationBase:
                 )
             return redacted_event
 
-        spam_check = await self.spam_checker.check_event_for_spam(pdu)
+        result = await self.spam_checker.check_event_for_spam(pdu)
 
-        if spam_check is not synapse.spam_checker_api.ALLOW:
+        if result:
             logger.warning("Event contains spam, soft-failing %s", pdu.event_id)
             # we redact (to save disk space) as well as soft-failing (to stop
             # using the event in prev_events).

synapse/handlers/directory.py

@@ -16,7 +16,6 @@ import logging
 import string
 from typing import TYPE_CHECKING, Iterable, List, Optional
 
-import synapse
 from synapse.api.constants import MAX_ALIAS_LENGTH, EventTypes
 from synapse.api.errors import (
     AuthError,
@@ -138,13 +137,10 @@ class DirectoryHandler:
                         403, "You must be in the room to create an alias for it"
                     )
 
-            spam_check = await self.spam_checker.user_may_create_room_alias(
+            if not await self.spam_checker.user_may_create_room_alias(
                 user_id, room_alias
-            )
-            if spam_check is not synapse.spam_checker_api.ALLOW:
-                raise AuthError(
-                    403, "This alias creation request has been rejected", spam_check
-                )
+            ):
+                raise AuthError(403, "This user is not permitted to create this alias")
 
             if not self.config.roomdirectory.is_alias_creation_allowed(
                 user_id, room_id, room_alias_str
@@ -430,12 +426,9 @@ class DirectoryHandler:
         """
         user_id = requester.user.to_string()
 
-        spam_check = await self.spam_checker.user_may_publish_room(user_id, room_id)
-        if spam_check is not synapse.spam_checker_api.ALLOW:
+        if not await self.spam_checker.user_may_publish_room(user_id, room_id):
             raise AuthError(
-                403,
-                "This request to publish a room to the room list has been rejected",
-                spam_check,
+                403, "This user is not permitted to publish rooms to the room list"
             )
 
         if requester.is_guest:

synapse/handlers/federation.py

@@ -27,7 +27,6 @@ from signedjson.key import decode_verify_key_bytes
 from signedjson.sign import verify_signed_json
 from unpaddedbase64 import decode_base64
 
-import synapse
 from synapse import event_auth
 from synapse.api.constants import EventContentFields, EventTypes, Membership
 from synapse.api.errors import (
@@ -660,7 +659,7 @@ class FederationHandler:
         # in the invitee's sync stream. It is stripped out for all other local users.
         event.unsigned["knock_room_state"] = stripped_room_state["knock_state_events"]
 
-        context = EventContext.for_outlier(self.storage)
+        context = EventContext.for_outlier()
         stream_id = await self._federation_event_handler.persist_events_and_notify(
             event.room_id, [(event, context)]
         )
@@ -800,14 +799,11 @@ class FederationHandler:
         if self.hs.config.server.block_non_admin_invites:
             raise SynapseError(403, "This server does not accept room invites")
 
-        spam_check = await self.spam_checker.user_may_invite(
+        if not await self.spam_checker.user_may_invite(
             event.sender, event.state_key, event.room_id
-        )
-        if spam_check is not synapse.spam_checker_api.ALLOW:
+        ):
             raise SynapseError(
-                403,
-                "This user is not permitted to send invites to this server/user",
-                spam_check,
+                403, "This user is not permitted to send invites to this server/user"
             )
 
         membership = event.content.get("membership")
@@ -852,7 +848,7 @@ class FederationHandler:
             )
         )
 
-        context = EventContext.for_outlier(self.storage)
+        context = EventContext.for_outlier()
         await self._federation_event_handler.persist_events_and_notify(
             event.room_id, [(event, context)]
         )
@@ -881,7 +877,7 @@ class FederationHandler:
 
         await self.federation_client.send_leave(host_list, event)
 
-        context = EventContext.for_outlier(self.storage)
+        context = EventContext.for_outlier()
         stream_id = await self._federation_event_handler.persist_events_and_notify(
             event.room_id, [(event, context)]
         )

synapse/handlers/federation_event.py

@@ -1423,7 +1423,7 @@ class FederationEventHandler:
                 # we're not bothering about room state, so flag the event as an outlier.
                 event.internal_metadata.outlier = True
 
-                context = EventContext.for_outlier(self._storage)
+                context = EventContext.for_outlier()
                 try:
                     validate_event_for_room_version(room_version_obj, event)
                     check_auth_rules_for_event(room_version_obj, event, auth)
@@ -1874,10 +1874,10 @@ class FederationEventHandler:
         )
 
         return EventContext.with_state(
-            storage=self._storage,
             state_group=state_group,
             state_group_before_event=context.state_group_before_event,
-            state_delta_due_to_event=state_updates,
+            current_state_ids=current_state_ids,
+            prev_state_ids=prev_state_ids,
             prev_group=prev_group,
             delta_ids=state_updates,
             partial_state=context.partial_state,

synapse/handlers/message.py

@@ -23,7 +23,6 @@ from canonicaljson import encode_canonical_json
 
 from twisted.internet.interfaces import IDelayedCall
 
-import synapse
 from synapse import event_auth
 from synapse.api.constants import (
     EventContentFields,
@@ -758,10 +757,6 @@ class EventCreationHandler:
             The previous version of the event is returned, if it is found in the
             event context. Otherwise, None is returned.
         """
-        if event.internal_metadata.is_outlier():
-            # This can happen due to out of band memberships
-            return None
-
         prev_state_ids = await context.get_prev_state_ids()
         prev_event_id = prev_state_ids.get((event.type, event.state_key))
         if not prev_event_id:
@@ -882,11 +877,11 @@ class EventCreationHandler:
                 event.sender,
             )
 
-            spam_check = await self.spam_checker.check_event_for_spam(event)
-            if spam_check is not synapse.spam_checker_api.ALLOW:
-                raise SynapseError(
-                    403, "This message had been rejected as probable spam", spam_check
-                )
+            spam_error = await self.spam_checker.check_event_for_spam(event)
+            if spam_error:
+                if not isinstance(spam_error, str):
+                    spam_error = "Spam is not permitted here"
+                raise SynapseError(403, spam_error, Codes.FORBIDDEN)
 
             ev = await self.handle_new_client_event(
                 requester=requester,
@@ -1006,7 +1001,7 @@ class EventCreationHandler:
         # after it is created
         if builder.internal_metadata.outlier:
             event.internal_metadata.outlier = True
-            context = EventContext.for_outlier(self.storage)
+            context = EventContext.for_outlier()
         elif (
             event.type == EventTypes.MSC2716_INSERTION
             and state_event_ids

synapse/handlers/pagination.py

@@ -448,7 +448,7 @@ class PaginationHandler:
             )
             # We expect `/messages` to use historic pagination tokens by default but
             # `/messages` should still works with live tokens when manually provided.
-            assert from_token.room_key.topological is not None
+            assert from_token.room_key.topological
 
         if pagin_config.limit is None:
             # This shouldn't happen as we've set a default limit before this

synapse/handlers/room.py

@@ -33,7 +33,6 @@ from typing import (
 import attr
 from typing_extensions import TypedDict
 
-import synapse
 from synapse.api.constants import (
     EventContentFields,
     EventTypes,
@@ -408,10 +407,9 @@ class RoomCreationHandler:
         """
         user_id = requester.user.to_string()
 
-        spam_check = await self.spam_checker.user_may_create_room(user_id)
-        if spam_check is not synapse.spam_checker_api.ALLOW:
+        if not await self.spam_checker.user_may_create_room(user_id):
             raise SynapseError(
-                403, "This room creation request has been rejected", spam_check
+                403, "You are not permitted to create rooms", Codes.FORBIDDEN
             )
 
         creation_content: JsonDict = {

synapse/handlers/room_member.py

@@ -18,7 +18,6 @@ import random
 from http import HTTPStatus
 from typing import TYPE_CHECKING, Iterable, List, Optional, Set, Tuple
 
-import synapse
 from synapse import types
 from synapse.api.constants import (
     AccountDataTypes,
@@ -680,6 +679,8 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):
             if target_id == self._server_notices_mxid:
                 raise SynapseError(HTTPStatus.FORBIDDEN, "Cannot invite this user")
 
+            block_invite = False
+
             if (
                 self._server_notices_mxid is not None
                 and requester.user.to_string() == self._server_notices_mxid
@@ -696,18 +697,16 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):
                         "Blocking invite: user is not admin and non-admin "
                         "invites disabled"
                     )
-                    raise SynapseError(403, "Invites have been disabled on this server")
+                    block_invite = True
 
-                spam_check = await self.spam_checker.user_may_invite(
+                if not await self.spam_checker.user_may_invite(
                     requester.user.to_string(), target_id, room_id
-                )
-                if spam_check is not synapse.spam_checker_api.ALLOW:
+                ):
                     logger.info("Blocking invite due to spam checker")
-                    raise SynapseError(
-                        403,
-                        "This invite has been rejected as probable spam",
-                        spam_check,
-                    )
+                    block_invite = True
+
+            if block_invite:
+                raise SynapseError(403, "Invites have been disabled on this server")
 
         # An empty prev_events list is allowed as long as the auth_event_ids are present
         if prev_event_ids is not None:
@@ -815,14 +814,11 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):
                 # We assume that if the spam checker allowed the user to create
                 # a room then they're allowed to join it.
                 and not new_room
-            ):
-                spam_check = await self.spam_checker.user_may_join_room(
+                and not await self.spam_checker.user_may_join_room(
                     target.to_string(), room_id, is_invited=inviter is not None
                 )
-                if spam_check is not synapse.spam_checker_api.ALLOW:
-                    raise SynapseError(
-                        403, "This request to join room has been rejected", spam_check
-                    )
+            ):
+                raise SynapseError(403, "Not allowed to join this room")
 
             # Check if a remote join should be performed.
             remote_join, remote_room_hosts = await self._should_perform_remote_join(
@@ -1376,14 +1372,13 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):
             )
         else:
             # Check if the spamchecker(s) allow this invite to go through.
-            spam_check = await self.spam_checker.user_may_send_3pid_invite(
+            if not await self.spam_checker.user_may_send_3pid_invite(
                 inviter_userid=requester.user.to_string(),
                 medium=medium,
                 address=address,
                 room_id=room_id,
-            )
-            if spam_check is not synapse.spam_checker_api.ALLOW:
-                raise SynapseError(403, "Cannot send threepid invite", spam_check)
+            ):
+                raise SynapseError(403, "Cannot send threepid invite")
 
             stream_id = await self._make_and_store_3pid_invite(
                 requester,

synapse/handlers/user_directory.py

@@ -60,7 +60,7 @@ class UserDirectoryHandler(StateDeltasHandler):
         self.clock = hs.get_clock()
         self.notifier = hs.get_notifier()
         self.is_mine_id = hs.is_mine_id
-        self.update_user_directory = hs.config.worker.should_update_user_directory
+        self.update_user_directory = hs.config.server.update_user_directory
         self.search_all_users = hs.config.userdirectory.user_directory_search_all_users
         self.spam_checker = hs.get_spam_checker()
         # The current position in the current_state_delta stream
@@ -100,8 +100,7 @@ class UserDirectoryHandler(StateDeltasHandler):
         # Remove any spammy users from the results.
         non_spammy_users = []
         for user in results["results"]:
-            spam_check = await self.spam_checker.check_username_for_spam(user)
-            if spam_check is synapse.spam_checker_api.ALLOW:
+            if not await self.spam_checker.check_username_for_spam(user):
                 non_spammy_users.append(user)
         results["results"] = non_spammy_users
 

synapse/http/matrixfederationclient.py

@@ -73,7 +73,7 @@ from synapse.logging.context import make_deferred_yieldable, run_in_background
 from synapse.logging.opentracing import set_tag, start_active_span, tags
 from synapse.types import JsonDict
 from synapse.util import json_decoder
-from synapse.util.async_helpers import AwakenableSleeper, timeout_deferred
+from synapse.util.async_helpers import timeout_deferred
 from synapse.util.metrics import Measure
 
 if TYPE_CHECKING:
@@ -353,13 +353,6 @@ class MatrixFederationHttpClient:
 
         self._cooperator = Cooperator(scheduler=schedule)
 
-        self._sleeper = AwakenableSleeper(self.reactor)
-
-    def wake_destination(self, destination: str) -> None:
-        """Called when the remote server may have come back online."""
-
-        self._sleeper.wake(destination)
-
     async def _send_request_with_optional_trailing_slash(
         self,
         request: MatrixFederationRequest,
@@ -481,8 +474,6 @@ class MatrixFederationHttpClient:
             self._store,
             backoff_on_404=backoff_on_404,
             ignore_backoff=ignore_backoff,
-            notifier=self.hs.get_notifier(),
-            replication_client=self.hs.get_replication_command_handler(),
         )
 
         method_bytes = request.method.encode("ascii")
@@ -673,9 +664,7 @@ class MatrixFederationHttpClient:
                             delay,
                         )
 
-                        # Sleep for the calculated delay, or wake up immediately
-                        # if we get notified that the server is back up.
-                        await self._sleeper.sleep(request.destination, delay * 1000)
+                        await self.clock.sleep(delay)
                         retries_left -= 1
                     else:
                         raise

synapse/http/server.py

@@ -33,7 +33,6 @@ from typing import (
     Optional,
     Pattern,
     Tuple,
-    TypeVar,
     Union,
 )
 
@@ -93,66 +92,6 @@ HTML_ERROR_TEMPLATE = """<!DOCTYPE html>
 HTTP_STATUS_REQUEST_CANCELLED = 499
 
 
-F = TypeVar("F", bound=Callable[..., Any])
-
-
-_cancellable_method_names = frozenset(
-    {
-        # `RestServlet`, `BaseFederationServlet` and `BaseFederationServerServlet`
-        # methods
-        "on_GET",
-        "on_PUT",
-        "on_POST",
-        "on_DELETE",
-        # `_AsyncResource`, `DirectServeHtmlResource` and `DirectServeJsonResource`
-        # methods
-        "_async_render_GET",
-        "_async_render_PUT",
-        "_async_render_POST",
-        "_async_render_DELETE",
-        "_async_render_OPTIONS",
-        # `ReplicationEndpoint` methods
-        "_handle_request",
-    }
-)
-
-
-def cancellable(method: F) -> F:
-    """Marks a servlet method as cancellable.
-
-    Methods with this decorator will be cancelled if the client disconnects before we
-    finish processing the request.
-
-    During cancellation, `Deferred.cancel()` will be invoked on the `Deferred` wrapping
-    the method. The `cancel()` call will propagate down to the `Deferred` that is
-    currently being waited on. That `Deferred` will raise a `CancelledError`, which will
-    propagate up, as per normal exception handling.
-
-    Before applying this decorator to a new endpoint, you MUST recursively check
-    that all `await`s in the function are on `async` functions or `Deferred`s that
-    handle cancellation cleanly, otherwise a variety of bugs may occur, ranging from
-    premature logging context closure, to stuck requests, to database corruption.
-
-    Usage:
-        class SomeServlet(RestServlet):
-            @cancellable
-            async def on_GET(self, request: SynapseRequest) -> ...:
-                ...
-    """
-    if method.__name__ not in _cancellable_method_names:
-        raise ValueError(
-            "@cancellable decorator can only be applied to servlet methods."
-        )
-
-    method.cancellable = True  # type: ignore[attr-defined]
-    return method
-
-
-def is_method_cancellable(method: Callable[..., Any]) -> bool:
-    """Checks whether a servlet method has the `@cancellable` flag."""
-    return getattr(method, "cancellable", False)
-
-
 def return_json_error(f: failure.Failure, request: SynapseRequest) -> None:
     """Sends a JSON error response to clients."""
 
@@ -344,9 +283,7 @@ class _AsyncResource(resource.Resource, metaclass=abc.ABCMeta):
 
     def render(self, request: SynapseRequest) -> int:
         """This gets called by twisted every time someone sends us a request."""
-        request.render_deferred = defer.ensureDeferred(
-            self._async_render_wrapper(request)
-        )
+        defer.ensureDeferred(self._async_render_wrapper(request))
         return NOT_DONE_YET
 
     @wrap_async_request_handler

synapse/http/site.py

@@ -19,7 +19,6 @@ from typing import TYPE_CHECKING, Any, Generator, Optional, Tuple, Union
 import attr
 from zope.interface import implementer
 
-from twisted.internet.defer import Deferred
 from twisted.internet.interfaces import IAddress, IReactorTime
 from twisted.python.failure import Failure
 from twisted.web.http import HTTPChannel
@@ -92,14 +91,6 @@ class SynapseRequest(Request):
         # we can't yet create the logcontext, as we don't know the method.
         self.logcontext: Optional[LoggingContext] = None
 
-        # The `Deferred` to cancel if the client disconnects early and
-        # `is_render_cancellable` is set. Expected to be set by `Resource.render`.
-        self.render_deferred: Optional["Deferred[None]"] = None
-        # A boolean indicating whether `render_deferred` should be cancelled if the
-        # client disconnects early. Expected to be set by the coroutine started by
-        # `Resource.render`, if rendering is asynchronous.
-        self.is_render_cancellable = False
-
         global _next_request_seq
         self.request_seq = _next_request_seq
         _next_request_seq += 1
@@ -366,21 +357,7 @@ class SynapseRequest(Request):
                     {"event": "client connection lost", "reason": str(reason.value)}
                 )
 
-            if self._is_processing:
-                if self.is_render_cancellable:
-                    if self.render_deferred is not None:
-                        # Throw a cancellation into the request processing, in the hope
-                        # that it will finish up sooner than it normally would.
-                        # The `self.processing()` context manager will call
-                        # `_finished_processing()` when done.
-                        with PreserveLoggingContext():
-                            self.render_deferred.cancel()
-                    else:
-                        logger.error(
-                            "Connection from client lost, but have no Deferred to "
-                            "cancel even though the request is marked as cancellable."
-                        )
-            else:
+            if not self._is_processing:
                 self._finished_processing()
 
     def _started_processing(self, servlet_name: str) -> None:

synapse/notifier.py

@@ -228,7 +228,9 @@ class Notifier:
         # Called when there are new things to stream over replication
         self.replication_callbacks: List[Callable[[], None]] = []
 
-        self._federation_client = hs.get_federation_http_client()
+        # Called when remote servers have come back online after having been
+        # down.
+        self.remote_server_up_callbacks: List[Callable[[str], None]] = []
 
         self._third_party_rules = hs.get_third_party_event_rules()
 
@@ -729,7 +731,3 @@ class Notifier:
         # circular dependencies.
         if self.federation_sender:
             self.federation_sender.wake_destination(server)
-
-        # Tell the federation client about the fact the server is back up, so
-        # that any in flight requests can be immediately retried.
-        self._federation_client.wake_destination(server)