Merge branch 'shay/super_docs' of https://github.com/matrix-org/synapse into shay/super_docs

CHANGES.md

@@ -1,105 +1,8 @@
-Synapse 1.59.0rc1 (2022-05-10)
-==============================
+Synapse 1.59.0
+==============
 
-This release makes several changes that server administrators should be aware of:
+The non-standard `m.login.jwt` login type has been removed from Synapse. It can be replaced with `org.matrix.login.jwt` for identical behaviour. This is only used if `jwt_config.enabled` is set to `true` in the configuration.
 
-- Device name lookup over federation is now disabled by default. ([\#12616](https://github.com/matrix-org/synapse/issues/12616))
-- The `synapse.app.appservice` and `synapse.app.user_dir` worker application types are now deprecated. ([\#12452](https://github.com/matrix-org/synapse/issues/12452), [\#12654](https://github.com/matrix-org/synapse/issues/12654))
-
-See [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1590) for more details.
-
-Additionally, this release removes the non-standard `m.login.jwt` login type from Synapse. It can be replaced with `org.matrix.login.jwt` for identical behaviour. This is only used if `jwt_config.enabled` is set to `true` in the configuration. ([\#12597](https://github.com/matrix-org/synapse/issues/12597))
-
-Features
---------
-
-- Support [MSC3266](https://github.com/matrix-org/matrix-doc/pull/3266) room summaries over federation. ([\#11507](https://github.com/matrix-org/synapse/issues/11507))
-- Implement [changes](https://github.com/matrix-org/matrix-spec-proposals/pull/2285/commits/4a77139249c2e830aec3c7d6bd5501a514d1cc27) to [MSC2285 (hidden read receipts)](https://github.com/matrix-org/matrix-spec-proposals/pull/2285). Contributed by @SimonBrandner. ([\#12168](https://github.com/matrix-org/synapse/issues/12168), [\#12635](https://github.com/matrix-org/synapse/issues/12635), [\#12636](https://github.com/matrix-org/synapse/issues/12636), [\#12670](https://github.com/matrix-org/synapse/issues/12670))
-- Extend the [module API](https://github.com/matrix-org/synapse/blob/release-v1.59/synapse/module_api/__init__.py) to allow modules to change actions for existing push rules of local users. ([\#12406](https://github.com/matrix-org/synapse/issues/12406))
-- Add the `notify_appservices_from_worker` configuration option (superseding `notify_appservices`) to allow a generic worker to be designated as the worker to send traffic to Application Services. ([\#12452](https://github.com/matrix-org/synapse/issues/12452))
-- Add the `update_user_directory_from_worker` configuration option (superseding `update_user_directory`) to allow a generic worker to be designated as the worker to update the user directory. ([\#12654](https://github.com/matrix-org/synapse/issues/12654))
-- Add new `enable_registration_token_3pid_bypass` configuration option to allow registrations via token as an alternative to verifying a 3pid. ([\#12526](https://github.com/matrix-org/synapse/issues/12526))
-- Implement [MSC3786](https://github.com/matrix-org/matrix-spec-proposals/pull/3786): Add a default push rule to ignore `m.room.server_acl` events. ([\#12601](https://github.com/matrix-org/synapse/issues/12601))
-- Add new `mau_appservice_trial_days` configuration option to specify a different trial period for users registered via an appservice. ([\#12619](https://github.com/matrix-org/synapse/issues/12619))
-
-
-Bugfixes
---------
-
-- Fix a bug introduced in Synapse 1.48.0 where the latest thread reply provided failed to include the proper bundled aggregations. ([\#12273](https://github.com/matrix-org/synapse/issues/12273))
-- Fix a bug introduced in Synapse 1.22.0 where attempting to send a large amount of read receipts to an application service all at once would result in duplicate content and abnormally high memory usage. Contributed by Brad & Nick @ Beeper. ([\#12544](https://github.com/matrix-org/synapse/issues/12544))
-- Fix a bug introduced in Synapse 1.57.0 which could cause `Failed to calculate hosts in room` errors to be logged for outbound federation. ([\#12570](https://github.com/matrix-org/synapse/issues/12570))
-- Fix a long-standing bug where status codes would almost always get logged as `200!`, irrespective of the actual status code, when clients disconnect before a request has finished processing. ([\#12580](https://github.com/matrix-org/synapse/issues/12580))
-- Fix race when persisting an event and deleting a room that could lead to outbound federation breaking. ([\#12594](https://github.com/matrix-org/synapse/issues/12594))
-- Fix a bug introduced in Synapse 1.53.0 where bundled aggregations for annotations/edits were incorrectly calculated. ([\#12633](https://github.com/matrix-org/synapse/issues/12633))
-- Fix a long-standing bug where rooms containing power levels with string values could not be upgraded. ([\#12657](https://github.com/matrix-org/synapse/issues/12657))
-- Prevent memory leak from reoccurring when presence is disabled. ([\#12656](https://github.com/matrix-org/synapse/issues/12656))
-
-
-Updates to the Docker image
----------------------------
-
-- Explicitly opt-in to using [BuildKit-specific features](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md) in the Dockerfile. This fixes issues with building images in some GitLab CI environments. ([\#12541](https://github.com/matrix-org/synapse/issues/12541))
-- Update the "Build docker images" GitHub Actions workflow to use `docker/metadata-action` to generate docker image tags, instead of a custom shell script. Contributed by @henryclw. ([\#12573](https://github.com/matrix-org/synapse/issues/12573))
-
-
-Improved Documentation
-----------------------
-
-- Update SQL statements and replace use of old table `user_stats_historical` in docs for Synapse Admins. ([\#12536](https://github.com/matrix-org/synapse/issues/12536))
-- Add missing linebreak to `pipx` install instructions. ([\#12579](https://github.com/matrix-org/synapse/issues/12579))
-- Add information about the TCP replication module to docs. ([\#12621](https://github.com/matrix-org/synapse/issues/12621))
-- Fixes to the formatting of `README.rst`. ([\#12627](https://github.com/matrix-org/synapse/issues/12627))
-- Fix docs on how to run specific Complement tests using the `complement.sh` test runner. ([\#12664](https://github.com/matrix-org/synapse/issues/12664))
-
-
-Deprecations and Removals
--------------------------
-
-- Remove unstable identifiers from [MSC3069](https://github.com/matrix-org/matrix-doc/pull/3069). ([\#12596](https://github.com/matrix-org/synapse/issues/12596))
-- Remove the unspecified `m.login.jwt` login type and the unstable `uk.half-shot.msc2778.login.application_service` from 
-  [MSC2778](https://github.com/matrix-org/matrix-doc/pull/2778). ([\#12597](https://github.com/matrix-org/synapse/issues/12597))
-- Synapse now requires at least Python 3.7.1 (up from 3.7.0), for compatibility with the latest Twisted trunk. ([\#12613](https://github.com/matrix-org/synapse/issues/12613))
-
-
-Internal Changes
-----------------
-
-- Use supervisord to supervise Postgres and Caddy in the Complement image to reduce restart time. ([\#12480](https://github.com/matrix-org/synapse/issues/12480))
-- Immediately retry any requests that have backed off when a server comes back online. ([\#12500](https://github.com/matrix-org/synapse/issues/12500))
-- Use `make_awaitable` instead of `defer.succeed` for return values of mocks in tests. ([\#12505](https://github.com/matrix-org/synapse/issues/12505))
-- Consistently check if an object is a `frozendict`. ([\#12564](https://github.com/matrix-org/synapse/issues/12564))
-- Protect module callbacks with read semantics against cancellation. ([\#12568](https://github.com/matrix-org/synapse/issues/12568))
-- Improve comments and error messages around access tokens. ([\#12577](https://github.com/matrix-org/synapse/issues/12577))
-- Improve docstrings for the receipts store. ([\#12581](https://github.com/matrix-org/synapse/issues/12581))
-- Use constants for read-receipts in tests. ([\#12582](https://github.com/matrix-org/synapse/issues/12582))
-- Log status code of cancelled requests as 499 and avoid logging stack traces for them. ([\#12587](https://github.com/matrix-org/synapse/issues/12587), [\#12663](https://github.com/matrix-org/synapse/issues/12663))
-- Remove special-case for `twisted` logger from default log config. ([\#12589](https://github.com/matrix-org/synapse/issues/12589))
-- Use `getClientAddress` instead of the deprecated `getClientIP`. ([\#12599](https://github.com/matrix-org/synapse/issues/12599))
-- Add link to documentation in Grafana Dashboard. ([\#12602](https://github.com/matrix-org/synapse/issues/12602))
-- Reduce log spam when running multiple event persisters. ([\#12610](https://github.com/matrix-org/synapse/issues/12610))
-- Add extra debug logging to federation sender. ([\#12614](https://github.com/matrix-org/synapse/issues/12614))
-- Prevent remote homeservers from requesting local user device names by default. ([\#12616](https://github.com/matrix-org/synapse/issues/12616))
-- Add a consistency check on events which we read from the database. ([\#12620](https://github.com/matrix-org/synapse/issues/12620))
-- Remove use of the `constantly` library and switch to enums for `EventRedactBehaviour`. Contributed by @andrewdoh. ([\#12624](https://github.com/matrix-org/synapse/issues/12624))
-- Remove unused code related to receipts. ([\#12632](https://github.com/matrix-org/synapse/issues/12632))
-- Minor improvements to the scripts for running Synapse in worker mode under Complement. ([\#12637](https://github.com/matrix-org/synapse/issues/12637))
-- Move `pympler` back in to the `all` extras. ([\#12652](https://github.com/matrix-org/synapse/issues/12652))
-- Fix spelling of `M_UNRECOGNIZED` in comments. ([\#12665](https://github.com/matrix-org/synapse/issues/12665))
-- Release script: confirm the commit to be tagged before tagging. ([\#12556](https://github.com/matrix-org/synapse/issues/12556))
-- Fix a typo in the announcement text generated by the Synapse release development script. ([\#12612](https://github.com/matrix-org/synapse/issues/12612))
-
-### Typechecking
-
-- Fix scripts-dev to pass typechecking. ([\#12356](https://github.com/matrix-org/synapse/issues/12356))
-- Add some type hints to datastore. ([\#12485](https://github.com/matrix-org/synapse/issues/12485))
-- Remove unused `# type: ignore`s. ([\#12531](https://github.com/matrix-org/synapse/issues/12531))
-- Allow unused `# type: ignore` comments in bleeding edge CI jobs. ([\#12576](https://github.com/matrix-org/synapse/issues/12576))
-- Remove redundant lines of config from `mypy.ini`. ([\#12608](https://github.com/matrix-org/synapse/issues/12608))
-- Update to mypy 0.950. ([\#12650](https://github.com/matrix-org/synapse/issues/12650))
-- Use `Concatenate` to better annotate `_do_execute`. ([\#12666](https://github.com/matrix-org/synapse/issues/12666))
-- Use `ParamSpec` to refine type hints. ([\#12667](https://github.com/matrix-org/synapse/issues/12667))
-- Fix mypy against latest pillow stubs. ([\#12671](https://github.com/matrix-org/synapse/issues/12671))
 
 Synapse 1.58.1 (2022-05-05)
 ===========================

changelog.d/11507.feature

@@ -0,0 +1 @@
+Support [MSC3266](https://github.com/matrix-org/matrix-doc/pull/3266) room summaries over federation.

changelog.d/12168.feature

@@ -0,0 +1 @@
+Implement [changes](https://github.com/matrix-org/matrix-spec-proposals/pull/2285/commits/4a77139249c2e830aec3c7d6bd5501a514d1cc27) to [MSC2285 (hidden read receipts)](https://github.com/matrix-org/matrix-spec-proposals/pull/2285). Contributed by @SimonBrandner.

changelog.d/12273.bugfix

@@ -0,0 +1 @@
+Fix a bug introduced in Synapse v1.48.0 where latest thread reply provided failed to include the proper bundled aggregations.

changelog.d/12356.misc

@@ -0,0 +1 @@
+Fix scripts-dev to pass typechecking.

changelog.d/12406.feature

@@ -0,0 +1 @@
+Add a module API to allow modules to change actions for existing push rules of local users.

changelog.d/12452.feature

@@ -0,0 +1 @@
+Add the `notify_appservices_from_worker` configuration option (superseding `notify_appservices`) to allow a generic worker to be designated as the worker to send traffic to Application Services.

changelog.d/12480.misc

@@ -0,0 +1 @@
+Use supervisord to supervise Postgres and Caddy in the Complement image to reduce restart time.

changelog.d/12485.misc

@@ -0,0 +1 @@
+Add some type hints to datastore.

changelog.d/12505.misc

@@ -0,0 +1 @@
+Use `make_awaitable` instead of `defer.succeed` for return values of mocks in tests.

changelog.d/12526.feature

@@ -0,0 +1 @@
+Add new `enable_registration_token_3pid_bypass` configuration option to allow registrations via token as an alternative to verifying a 3pid.

changelog.d/12531.misc

@@ -0,0 +1 @@
+Remove unused `# type: ignore`s.

changelog.d/12541.docker

@@ -0,0 +1 @@
+Explicitly opt-in to using [BuildKit-specific features](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md) in the Dockerfile. This fixes issues with building images in some GitLab CI environments.

changelog.d/12544.bugfix

@@ -0,0 +1 @@
+Fix a bug where attempting to send a large amount of read receipts to an application service all at once would result in duplicate content and abnormally high memory usage. Contributed by Brad & Nick @ Beeper.

changelog.d/12556.misc

@@ -0,0 +1 @@
+Release script: confirm the commit to be tagged before tagging.

changelog.d/12564.misc

@@ -0,0 +1 @@
+Consistently check if an object is a `frozendict`.

changelog.d/12568.misc

@@ -0,0 +1 @@
+Protect module callbacks with read semantics against cancellation.

changelog.d/12570.bugfix

@@ -0,0 +1 @@
+Fix a bug introduced in Synapse 1.57 which could cause `Failed to calculate hosts in room` errors to be logged for outbound federation.

changelog.d/12573.docker

@@ -0,0 +1 @@
+Update the "Build docker images" GitHub Actions workflow to use `docker/metadata-action` to generate docker image tags, instead of a custom shell script. Contributed by henryclw.

changelog.d/12576.misc

@@ -0,0 +1 @@
+Allow unused `#type: ignore` comments in bleeding edge CI jobs.

changelog.d/12577.misc

@@ -0,0 +1 @@
+Improve comments and error messages around access tokens.

changelog.d/12579.doc

@@ -0,0 +1 @@
+Add missing linebreak to pipx install instructions.

changelog.d/12580.bugfix

@@ -0,0 +1 @@
+Fix a long standing bug where status codes would almost always get logged as 200!, irrespective of the actual status code, when clients disconnect before a request has finished processing.

changelog.d/12581.misc

@@ -0,0 +1 @@
+Improve docstrings for the receipts store.

changelog.d/12582.misc

@@ -0,0 +1 @@
+Use constants for read-receipts in tests.

changelog.d/12587.misc

@@ -0,0 +1 @@
+Log status code of cancelled requests as 499 and avoid logging stack traces for them.

changelog.d/12589.misc

@@ -0,0 +1 @@
+Remove special-case for `twisted` logger from default log config.

changelog.d/12594.bugfix

@@ -0,0 +1 @@
+Fix race when persisting an event and deleting a room that could lead to outbound federation breaking.

changelog.d/12596.removal

@@ -0,0 +1 @@
+Remove unstable identifiers from [MSC3069](https://github.com/matrix-org/matrix-doc/pull/3069).

changelog.d/12597.removal

@@ -0,0 +1,2 @@
+Remove the unspecified `m.login.jwt` login type and the unstable `uk.half-shot.msc2778.login.application_service` from 
+[MSC2778](https://github.com/matrix-org/matrix-doc/pull/2778).

changelog.d/12599.misc

@@ -0,0 +1 @@
+Use `getClientAddress` instead of the deprecated `getClientIP`.

changelog.d/12602.misc

@@ -0,0 +1 @@
+Add link to documentation in Grafana Dashboard.

changelog.d/12608.misc

@@ -0,0 +1 @@
+Remove redundant lines of config from `mypy.ini`.

changelog.d/12610.misc

@@ -0,0 +1 @@
+Reduce log spam when running multiple event persisters.

changelog.d/12612.bugfix

@@ -0,0 +1 @@
+Fix a typo in the announcement text generated by the Synapse release development script.

changelog.d/12613.removal

@@ -0,0 +1 @@
+Synapse now requires at least Python 3.7.1 (up from 3.7.0), for compatibility with the latest Twisted trunk.

changelog.d/12614.misc

@@ -0,0 +1 @@
+Add extra debug logging to federation sender.

changelog.d/12616.misc

@@ -0,0 +1 @@
+Prevent remote homeservers from requesting local user device names by default.

changelog.d/12619.feature

@@ -0,0 +1 @@
+Add new `mau_appservice_trial_days` configuration option to specify a different trial period for users registered via an appservice.

changelog.d/12620.misc

@@ -0,0 +1 @@
+Add a consistency check on events which we read from the database.

changelog.d/12624.misc

@@ -0,0 +1 @@
+Remove use of constantly library and switch to enums for EventRedactBehaviour. Contributed by @andrewdoh.

changelog.d/12627.doc

@@ -0,0 +1 @@
+Fixes to the formatting of README.rst.

changelog.d/12632.misc

@@ -0,0 +1 @@
+Remove unused code related to receipts.

changelog.d/12633.bugfix

@@ -0,0 +1 @@
+Fix a bug introduced in Synapse v1.53.0 where bundled aggregations for annotations/edits were incorrectly calculated.

changelog.d/12635.feature

@@ -0,0 +1 @@
+Implement [changes](https://github.com/matrix-org/matrix-spec-proposals/pull/2285/commits/4a77139249c2e830aec3c7d6bd5501a514d1cc27) to [MSC2285 (hidden read receipts)](https://github.com/matrix-org/matrix-spec-proposals/pull/2285). Contributed by @SimonBrandner.

changelog.d/12636.feature

@@ -0,0 +1 @@
+Implement [changes](https://github.com/matrix-org/matrix-spec-proposals/pull/2285/commits/4a77139249c2e830aec3c7d6bd5501a514d1cc27) to [MSC2285 (hidden read receipts)](https://github.com/matrix-org/matrix-spec-proposals/pull/2285). Contributed by @SimonBrandner.

changelog.d/12639.bugfix

@@ -0,0 +1 @@
+Add new `enable_registration_token_3pid_bypass` configuration option to allow registrations via token as an alternative to verifying a 3pid.

changelog.d/12650.misc

@@ -0,0 +1 @@
+Update to mypy 0.950.

changelog.d/12652.misc

@@ -0,0 +1 @@
+Move `pympler` back in to the `all` extras.

changelog.d/12656.misc

@@ -0,0 +1 @@
+Prevent memory leak from reoccurring when presence is disabled.

changelog.d/12657.bugfix

@@ -0,0 +1 @@
+Fix a long-standing bug where rooms containing power levels with string values could not be upgraded.

changelog.d/12663.misc

@@ -0,0 +1 @@
+Log status code of cancelled requests as 499 and avoid logging stack traces for them.

changelog.d/12664.doc

@@ -0,0 +1 @@
+Fix docs on how to run specific Complement tests using the `complement.sh` test runner.

changelog.d/12665.misc

@@ -0,0 +1 @@
+Fix spelling of `M_UNRECOGNIZED` in comments.

changelog.d/12666.misc

@@ -0,0 +1 @@
+Use `Concatenate` to better annotate `_do_execute`.

changelog.d/12667.misc

@@ -0,0 +1 @@
+Use `ParamSpec` to refine type hints.

changelog.d/12670.feature

@@ -0,0 +1 @@
+Implement [changes](https://github.com/matrix-org/matrix-spec-proposals/pull/2285/commits/4a77139249c2e830aec3c7d6bd5501a514d1cc27) to [MSC2285 (hidden read receipts)](https://github.com/matrix-org/matrix-spec-proposals/pull/2285). Contributed by @SimonBrandner.

changelog.d/12671.misc

@@ -0,0 +1 @@
+Fix mypy against latest pillow stubs.

changelog.d/12681.doc

@@ -0,0 +1 @@
+Add topics to synapse documentation.

debian/changelog

@@ -1,11 +1,10 @@
-matrix-synapse-py3 (1.59.0~rc1) stable; urgency=medium
+matrix-synapse-py3 (1.58.2) UNRELEASED; urgency=medium
 
   * Adjust how the `exported-requirements.txt` file is generated as part of
     the process of building these packages. This affects the package
     maintainers only; end-users are unaffected.
-  * New Synapse release 1.59.0rc1.
 
- -- Synapse Packaging team <packages@matrix.org>  Tue, 10 May 2022 10:45:08 +0100
+ -- Synapse Packaging team <packages@matrix.org>  Fri, 06 May 2022 13:49:29 +0100
 
 matrix-synapse-py3 (1.58.1) stable; urgency=medium
 

docker/conf-workers/supervisord.conf.j2

@@ -9,7 +9,7 @@ user=root
 files = /etc/supervisor/conf.d/*.conf
 
 [program:nginx]
-command=/usr/local/bin/prefix-log /usr/sbin/nginx -g "daemon off;"
+command=/usr/sbin/nginx -g "daemon off;"
 priority=500
 stdout_logfile=/dev/stdout
 stdout_logfile_maxbytes=0
@@ -19,7 +19,7 @@ username=www-data
 autorestart=true
 
 [program:redis]
-command=/usr/local/bin/prefix-log /usr/bin/redis-server /etc/redis/redis.conf --daemonize no
+command=/usr/bin/redis-server /etc/redis/redis.conf --daemonize no
 priority=1
 stdout_logfile=/dev/stdout
 stdout_logfile_maxbytes=0
@@ -29,7 +29,7 @@ username=redis
 autorestart=true
 
 [program:synapse_main]
-command=/usr/local/bin/prefix-log /usr/local/bin/python -m synapse.app.homeserver --config-path="{{ main_config_path }}" --config-path=/conf/workers/shared.yaml
+command=/usr/local/bin/python -m synapse.app.homeserver --config-path="{{ main_config_path }}" --config-path=/conf/workers/shared.yaml
 priority=10
 # Log startup failures to supervisord's stdout/err
 # Regular synapse logs will still go in the configured data directory

docs/SUMMARY.md

@@ -88,19 +88,158 @@
   - [OpenTracing](opentracing.md)
   - [Database Schemas](development/database_schema.md)
   - [Experimental features](development/experimental_features.md)
-  - [Synapse Architecture]()
-    - [Log Contexts](log_contexts.md)
-    - [Replication](replication.md)
-    - [TCP Replication](tcp_replication.md)
-  - [Internal Documentation](development/internal_documentation/README.md)
+  - [Synapse Architecture](development/synapse_architechture.md)
+    - [Overview]()
+      - Flows of data through Synapse
+        - Starts with a Request/BG Job/Module
+        - Processing
+        - Maybe Response
+      - Config
+      - Request Handling
+        - Clients
+        - Federation
+        - Internal Processing - there could be an exception!
+      - Storage
+      - Federation
+      - External Entities: Remote homeservers, Appservices, Pushers, Identity Servers
+      - Modules
+      - Workers/Replication/Redis/Organization of a large homeserver instance
+  - [Configuration]()
+    - Experimental config options
+      - Define how long they should live after the corresponding msc has merged
+  - [Logging]()
+    - combine with [Log Contexts](log_contexts.md)
+  - [Storage]()
+    - What is current database schema?
+      - Start by linking script that can compile full schema
+      - Is there a tool that can visualize a postgres db? - Shay to check
+    - [Database Backends]()
+    - [Database Migrations]()
+      - Background [Schema] Updates
+    - [Caching]()
+      - Cache tuning
+      - How to check sizes
+        - Nice to have a metric for max cache sizes
+      - How to add a cache to a [storage] function
+        - Why would you
+      - How to invalidate (with support for workers)
+  - [Request Listening]()
+    - Mention that we use Twisted
+    - [Servlets and Resources]()
+      - When to use one over the other (always use a servlet?)
+      - Unstable/release-based endpoints
+      - Don't put too many servlets on the same resource
+    - [Handlers]()
+      - Machinery behind the servlet that does all the processing
+  - [Background Jobs]()
+  - [Federation]()
+    - Federation Catchup
+  - [Data Types]()
+    - [Users]()
+      - Allowed username characters
+    - [Devices]()
+    - [Events]()
+      - What is an event?
+      - Different event formats depending on room version
+      - Outlier events, rejected events, dropped events
+    - [Rooms]()
+      - [Room DAG concepts](development/room-dag-concepts.md)
+  - [Sync]()
+  - [State Resolution]()
+    - [The Auth Chain Difference Algorithm](auth_chain_difference_algorithm.md)
+  - [User Authentication]()
     - [Single Sign-On]()
       - [SAML](development/saml.md)
       - [CAS](development/cas.md)
-    - [Room DAG concepts](development/room-dag-concepts.md)
-    - [State Resolution]()
-      - [The Auth Chain Difference Algorithm](auth_chain_difference_algorithm.md)
-    - [Media Repository](media_repository.md)
-    - [Room and User Statistics](room_and_user_statistics.md)
+    - [User-Interactive Auth]()
+    - [Password-based]()
+    - [Password Auth Modules]()
+    - [Token-based]()
+  - [Rate Limiting]()
+    - Different rate limiting classes (Ratelimiter vs RequestRatelimiter)
+    - How does our rate limiting work
+    - When to use/add rate limiting
+    - Rate limiting for federation traffic
+  - [Media Repository](media_repository.md)
+  - [Email and HTML Templating]()
+  - [Presence]()
+    - What is Presence?
+    - How is it designed to work currently
+      - Why it is recommended to be disabled
+      - Why is this resource intensive
+        - Links and issues on how to fix this
+      - How it's implemented
+  - [Application Services]()
+  - [Push Notifications]()
+  - [Synapse Admin API]()
+  - [Synapse Modules]()
+  - [Workers]()
+    - Things to be mindful of in order to make your feature work with workers
+    - How to test a feature on workers
+  - [Replication](replication.md)
+    - [TCP Replication](tcp_replication.md)
+  - [Sources]()
+    - TypingSource, ReceiptSource, PresenceSource, etc
+    - Classes containing methods for storing and getting different types of data
+  - [Streams]()
+    - Stream Types (the ID of the stream)
+    - Stream Token: each entity added to the stream gets a unique, incremental token
+    - StreamReplication classes and clients
+    - How this interacts with sync?
+    - Stream token generators: Single Writer Only (sqlite), MultiWriterStreamIDGenerator (Postgres)
+  - [Notifier]()
+  - [Monitoring and Metrics]() # opentracing, phone home and metrics
+    - [Opentracing]()
+      - How does this work?
+      - How is this helpful?
+      - What's jaeger?
+      - How do I create and mutate a span?
+      - How are spans sent between homeservers?
+        - Only between whitelisted homeservers
+      - What areas are currently traced
+        - Database layer
+        - Servlets
+        - Crypto stuff
+    - [Metrics]()
+      - Prometheus
+      - How to make a new metrics / good practices
+        - What are all the current metrics and what each one means
+        - How to view in grafana
+          - Here's a quick example of making a new panel to visualize metrics
+        - Btw it would be nice to have a tooltip with an explainer for every graph in graphana
+      - [Room and User Statistics](room_and_user_statistics.md)  # TODO: This page is currently useless    
+  - [Testing]()
+    - [Unit Tests]()
+      - How to run the tests (partially in the contributing guide)
+        - IDE-specific examples
+        - How to run one test / one test case / run in parallel
+        - Documenting test environment variables: SYNAPSE_TEST_LOG_LEVEL, SYNAPSE_TEST_PERSISTENT_SQLITE_DB, SYNAPSE_POSTGRES etc.
+      - Test case versus test method
+        - prepare, make_homeserver, default_config, @override_config decorator, @parameterized decorator, pump/reactor.advance
+        - Test utilities: fake homeserver, federation, workers, channels, requests, HomeServerTestCase.helper
+          - How to use them in your tests at a high level
+          - How they are implemented
+      - Different inherited TestCase classes: HomeServerTestCase, FederatedHomeServerTestCase(sp?), TestCase
+      - Test pattern: How to start a request, check the state of the homeserver, respond to the request
+      - Ensuring the right servlets for your test are registered
+      - Good practice: use storage methods vs. tampering with the database directly (database schema changes)
+      - Dealing with deferreds/async (get_success/get_failure).
+      - Mocking - async mocks
+      - Organization of tests
+      - Skipping tests
+    - [Integration Tests]()
+        - [Complement]()
+            - Point to Complement's documentation on its structure and how to write a test
+            - How to run Complement against Synapse
+            - Document Synapse's Complement images and how they're built
+            - Which Complement build flags does Synapse support
+                - Unstable features
+                - Synapse's Complement Blacklist
+        - [Sytest]()
+            - You should prefer Complement :)
+            - Helper functions
+            - Dealing with race conditions
+            - Link to that one perl helper doc Matthew wrote
   - [Scripts]()
 
 # Other

docs/replication.md

@@ -35,8 +35,3 @@ See [the TCP replication documentation](tcp_replication.md).
 There are read-only version of the synapse storage layer in
 `synapse/replication/slave/storage` that use the response of the
 replication API to invalidate their caches.
-
-### The TCP Replication Module
-Information about how the tcp replication module is structured, including how
-the classes interact, can be found in
-`synapse/replication/tcp/__init__.py`

docs/upgrade.md

@@ -101,36 +101,29 @@ To re-enable this functionality, set the
 homeserver config option to `true`.
 
 
-## Deprecation of the `synapse.app.appservice` and `synapse.app.user_dir` worker application types
+## Deprecation of the `synapse.app.appservice` worker application type
 
 The `synapse.app.appservice` worker application type allowed you to configure a
 single worker to use to notify application services of new events, as long
 as this functionality was disabled on the main process with `notify_appservices: False`.
-Further, the `synapse.app.user_dir` worker application type allowed you to configure
-a single worker to be responsible for updating the user directory, as long as this
-was disabled on the main process with `update_user_directory: False`.
 
 To unify Synapse's worker types, the `synapse.app.appservice` worker application
 type and the `notify_appservices` configuration option have been deprecated.
-The `synapse.app.user_dir` worker application type and `update_user_directory`
-configuration option have also been deprecated.
 
-To get the same functionality as was provided by the deprecated options, it's now recommended that the `synapse.app.generic_worker`
-worker application type is used and that the `notify_appservices_from_worker` and/or
-`update_user_directory_from_worker` options are set to the name of a worker.
+To get the same functionality, it's now recommended that the `synapse.app.generic_worker`
+worker application type is used and that the `notify_appservices_from_worker` option
+is set to the name of a worker.
 
-For the time being, the old options can be used alongside the new options to make
-it easier to transition between the two configurations, however please note that:
+For the time being, `notify_appservices_from_worker` can be used alongside
+`synapse.app.appservice` and `notify_appservices` to make it easier to transition
+between the two configurations, however please note that:
 
 - the options must not contradict each other (otherwise Synapse won't start); and
-- the `notify_appservices` and `update_user_directory` options will be removed in a future release of Synapse.
+- the `notify_appservices` option will be removed in a future release of Synapse.
 
-Please see the [*Notifying Application Services*][v1_59_notify_ases_from] and
-[*Updating the User Directory*][v1_59_update_user_dir] sections of the worker
-documentation for more information.
+Please see [the relevant section of the worker documentation][v1_59_notify_ases_from] for more information.
 
 [v1_59_notify_ases_from]: workers.md#notifying-application-services
-[v1_59_update_user_dir]: workers.md#updating-the-user-directory
 
 
 # Upgrading to v1.58.0

docs/usage/administration/useful_sql_for_admins.md

@@ -1,10 +1,7 @@
 ## Some useful SQL queries for Synapse Admins
 
 ## Size of full matrix db
-```sql
-SELECT pg_size_pretty( pg_database_size( 'matrix' ) );
-```
-
+`SELECT pg_size_pretty( pg_database_size( 'matrix' ) );`
 ### Result example:
 ``` 
 pg_size_pretty 
@@ -12,19 +9,39 @@ pg_size_pretty
  6420 MB
 (1 row)
 ```
+## Show top 20 larger rooms by state events count
+```sql
+SELECT r.name, s.room_id, s.current_state_events
+  FROM room_stats_current s
+  LEFT JOIN room_stats_state r USING (room_id)
+  ORDER BY current_state_events DESC
+  LIMIT 20;
+```
+
+and by state_group_events count:
+```sql
+SELECT rss.name, s.room_id, count(s.room_id) FROM state_groups_state s
+LEFT JOIN room_stats_state rss USING (room_id)
+GROUP BY s.room_id, rss.name        
+ORDER BY count(s.room_id) DESC
+LIMIT 20;
+```
+plus same, but with join removed for performance reasons:
+```sql
+SELECT s.room_id, count(s.room_id) FROM state_groups_state s
+GROUP BY s.room_id        
+ORDER BY count(s.room_id) DESC
+LIMIT 20;
+```
 
 ## Show top 20 larger tables by row count
 ```sql
-SELECT relname, n_live_tup AS "rows"
-  FROM pg_stat_user_tables
+SELECT relname, n_live_tup as rows
+  FROM pg_stat_user_tables 
   ORDER BY n_live_tup DESC
   LIMIT 20;
 ```
-This query is quick, but may be very approximate, for exact number of rows use:
-```sql
-SELECT COUNT(*) FROM <table_name>;
-```
-
+This query is quick, but may be very approximate, for exact number of rows use `SELECT COUNT(*) FROM <table_name>`.
 ### Result example:
 ```
 state_groups_state - 161687170
@@ -49,19 +66,46 @@ device_lists_stream - 326903
 user_directory_search - 316433
 ```
 
+## Show top 20 rooms by new events count in last 1 day:
+```sql
+SELECT e.room_id, r.name, COUNT(e.event_id) cnt FROM events e
+LEFT JOIN room_stats_state r USING (room_id)
+WHERE e.origin_server_ts >= DATE_PART('epoch', NOW() - INTERVAL '1 day') * 1000 GROUP BY e.room_id, r.name ORDER BY cnt DESC LIMIT 20;
+```
+
+## Show top 20 users on homeserver by sent events (messages) at last month:
+```sql
+SELECT user_id, SUM(total_events) 
+   FROM user_stats_historical
+   WHERE TO_TIMESTAMP(end_ts/1000) AT TIME ZONE 'UTC' > date_trunc('day', now() - interval '1 month')
+   GROUP BY user_id
+   ORDER BY SUM(total_events) DESC 
+   LIMIT 20;
+```
+
+## Show last 100 messages from needed user, with room names:
+```sql
+SELECT e.room_id, r.name, e.event_id, e.type, e.content, j.json FROM events e
+  LEFT JOIN event_json j USING (room_id)
+  LEFT JOIN room_stats_state r USING (room_id)
+  WHERE sender = '@LOGIN:example.com'
+  AND e.type = 'm.room.message'
+  ORDER BY stream_ordering DESC
+  LIMIT 100;
+```
+
 ## Show top 20 larger tables by storage size
 ```sql
 SELECT nspname || '.' || relname AS "relation",
-    pg_size_pretty(pg_total_relation_size(c.oid)) AS "total_size"
-  FROM pg_class c
-  LEFT JOIN pg_namespace n ON (n.oid = c.relnamespace)
+    pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size"
+  FROM pg_class C
+  LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
   WHERE nspname NOT IN ('pg_catalog', 'information_schema')
-    AND c.relkind <> 'i'
+    AND C.relkind <> 'i'
     AND nspname !~ '^pg_toast'
-  ORDER BY pg_total_relation_size(c.oid) DESC
+  ORDER BY pg_total_relation_size(C.oid) DESC
   LIMIT 20;
 ```
-
 ### Result example:
 ```
 public.state_groups_state - 27 GB
@@ -86,93 +130,8 @@ public.device_lists_remote_cache - 124 MB
 public.state_group_edges - 122 MB
 ```
 
-## Show top 20 larger rooms by state events count
-You get the same information when you use the
-[admin API](../../admin_api/rooms.md#list-room-api)
-and set parameter `order_by=state_events`.
-
-```sql
-SELECT r.name, s.room_id, s.current_state_events
-  FROM room_stats_current s
-  LEFT JOIN room_stats_state r USING (room_id)
-  ORDER BY current_state_events DESC
-  LIMIT 20;
-```
-
-and by state_group_events count:
-```sql
-SELECT rss.name, s.room_id, COUNT(s.room_id)
-  FROM state_groups_state s
-  LEFT JOIN room_stats_state rss USING (room_id)
-  GROUP BY s.room_id, rss.name
-  ORDER BY COUNT(s.room_id) DESC
-  LIMIT 20;
-```
-
-plus same, but with join removed for performance reasons:
-```sql
-SELECT s.room_id, COUNT(s.room_id)
-  FROM state_groups_state s
-  GROUP BY s.room_id 
-  ORDER BY COUNT(s.room_id) DESC
-  LIMIT 20;
-```
-
-## Show top 20 rooms by new events count in last 1 day:
-```sql
-SELECT e.room_id, r.name, COUNT(e.event_id) cnt
-  FROM events e
-  LEFT JOIN room_stats_state r USING (room_id)
-  WHERE e.origin_server_ts >= DATE_PART('epoch', NOW() - INTERVAL '1 day') * 1000
-  GROUP BY e.room_id, r.name 
-  ORDER BY cnt DESC
-  LIMIT 20;
-```
-
-## Show top 20 users on homeserver by sent events (messages) at last month:
-Caution. This query does not use any indexes, can be slow and create load on the database.
-```sql
-SELECT COUNT(*), sender
-  FROM events
-  WHERE (type = 'm.room.encrypted' OR type = 'm.room.message')
-    AND origin_server_ts >= DATE_PART('epoch', NOW() - INTERVAL '1 month') * 1000
-  GROUP BY sender
-  ORDER BY COUNT(*) DESC
-  LIMIT 20;
-```
-
-## Show last 100 messages from needed user, with room names:
-```sql
-SELECT e.room_id, r.name, e.event_id, e.type, e.content, j.json
-  FROM events e
-  LEFT JOIN event_json j USING (room_id)
-  LEFT JOIN room_stats_state r USING (room_id)
-  WHERE sender = '@LOGIN:example.com'
-    AND e.type = 'm.room.message'
-  ORDER BY stream_ordering DESC
-  LIMIT 100;
-```
-
 ## Show rooms with names, sorted by events in this rooms
-
-**Sort and order with bash**
-```bash
-echo "SELECT event_json.room_id, room_stats_state.name FROM event_json, room_stats_state \
-WHERE room_stats_state.room_id = event_json.room_id" | psql -d synapse -h localhost -U synapse_user -t \
-| sort | uniq -c | sort -n
-```
-Documentation for `psql` command line parameters: https://www.postgresql.org/docs/current/app-psql.html
-
-**Sort and order with SQL**
-```sql
-SELECT COUNT(*), event_json.room_id, room_stats_state.name
-  FROM event_json, room_stats_state
-  WHERE room_stats_state.room_id = event_json.room_id
-  GROUP BY event_json.room_id, room_stats_state.name
-  ORDER BY COUNT(*) DESC
-  LIMIT 50;
-```
-
+`echo "select event_json.room_id,room_stats_state.name from event_json,room_stats_state where room_stats_state.room_id=event_json.room_id" | psql synapse | sort | uniq -c | sort -n`
 ### Result example:
 ```
    9459  !FPUfgzXYWTKgIrwKxW:matrix.org              | This Week in Matrix
@@ -186,22 +145,12 @@ SELECT COUNT(*), event_json.room_id, room_stats_state.name
 ```
 
 ## Lookup room state info by list of room_id
-You get the same information when you use the
-[admin API](../../admin_api/rooms.md#room-details-api).
 ```sql
-SELECT rss.room_id, rss.name, rss.canonical_alias, rss.topic, rss.encryption,
-    rsc.joined_members, rsc.local_users_in_room, rss.join_rules
-  FROM room_stats_state rss
-  LEFT JOIN room_stats_current rsc USING (room_id)
-  WHERE room_id IN ( WHERE room_id IN (
-    '!OGEhHVWSdvArJzumhm:matrix.org',
-    '!YTvKGNlinIzlkMTVRl:matrix.org' 
-  );
-```
-
-## Show users and devices that have not been online for a while
-```sql
-SELECT user_id, device_id, user_agent, TO_TIMESTAMP(last_seen / 1000) AS "last_seen"
-  FROM devices
-  WHERE last_seen < DATE_PART('epoch', NOW() - INTERVAL '3 month') * 1000;
-```
+SELECT rss.room_id, rss.name, rss.canonical_alias, rss.topic, rss.encryption, rsc.joined_members, rsc.local_users_in_room, rss.join_rules
+FROM room_stats_state rss
+LEFT JOIN room_stats_current rsc USING (room_id)
+WHERE room_id IN (WHERE room_id IN (
+ '!OGEhHVWSdvArJzumhm:matrix.org',
+ '!YTvKGNlinIzlkMTVRl:matrix.org'
+)
+```

docs/workers.md

@@ -426,7 +426,7 @@ the shared configuration would include:
 run_background_tasks_on: background_worker
 ```
 
-You might also wish to investigate the `update_user_directory_from_worker` and
+You might also wish to investigate the `update_user_directory` and
 `media_instance_running_background_jobs` settings.
 
 An example for a dedicated background worker instance:
@@ -435,26 +435,9 @@ An example for a dedicated background worker instance:
 {{#include systemd-with-workers/workers/background_worker.yaml}}
 ```
 
-#### Updating the User Directory
-
-You can designate one generic worker to update the user directory.
-
-Specify its name in the shared configuration as follows:
-
-```yaml
-update_user_directory_from_worker: worker_name
-```
-
-This work cannot be load-balanced; please ensure the main process is restarted
-after setting this option in the shared configuration!
-
-This style of configuration supersedes the legacy `synapse.app.user_dir`
-worker application type.
-
-
 #### Notifying Application Services
 
-You can designate one generic worker to send output traffic to Application Services.
+You can designate one worker to send output traffic to Application Services.
 
 Specify its name in the shared configuration as follows:
 
@@ -487,7 +470,7 @@ pusher_instances:
 
 ### `synapse.app.appservice`
 
-**Deprecated as of Synapse v1.59.** [Use `synapse.app.generic_worker` with the
+**Deprecated as of Synapse v1.58.** [Use `synapse.app.generic_worker` with the
 `notify_appservices_from_worker` option instead.](#notifying-application-services)
 
 Handles sending output traffic to Application Services. Doesn't handle any
@@ -557,9 +540,6 @@ Note that if a reverse proxy is used , then `/_matrix/media/` must be routed for
 
 ### `synapse.app.user_dir`
 
-**Deprecated as of Synapse v1.59.** [Use `synapse.app.generic_worker` with the
-`update_user_directory_from_worker` option instead.](#updating-the-user-directory)
-
 Handles searches in the user directory. It can handle REST endpoints matching
 the following regular expressions:
 

pyproject.toml

@@ -54,7 +54,7 @@ skip_gitignore = true
 
 [tool.poetry]
 name = "matrix-synapse"
-version = "1.59.0rc1"
+version = "1.58.1"
 description = "Homeserver for the Matrix decentralised comms protocol"
 authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
 license = "Apache-2.0"

scripts-dev/complement.sh

@@ -43,8 +43,6 @@ fi
 # Build the base Synapse image from the local checkout
 docker build -t matrixdotorg/synapse -f "docker/Dockerfile" .
 
-extra_test_args=()
-
 # If we're using workers, modify the docker files slightly.
 if [[ -n "$WORKERS" ]]; then
   # Build the workers docker image (from the base Synapse image).
@@ -54,14 +52,7 @@ if [[ -n "$WORKERS" ]]; then
   COMPLEMENT_DOCKERFILE=SynapseWorkers.Dockerfile
 
   # And provide some more configuration to complement.
-
-  # It can take quite a while to spin up a worker-mode Synapse for the first
-  # time (the main problem is that we start 14 python processes for each test,
-  # and complement likes to do two of them in parallel).
-  export COMPLEMENT_SPAWN_HS_TIMEOUT_SECS=120
-
-  # ... and it takes longer than 10m to run the whole suite.
-  extra_test_args+=("-timeout=60m")
+  export COMPLEMENT_SPAWN_HS_TIMEOUT_SECS=60
 else
   export COMPLEMENT_BASE_IMAGE=complement-synapse
   COMPLEMENT_DOCKERFILE=Dockerfile
@@ -73,4 +64,4 @@ docker build -t $COMPLEMENT_BASE_IMAGE -f "docker/complement/$COMPLEMENT_DOCKERF
 # Run the tests!
 echo "Images built; running complement"
 cd "$COMPLEMENT_DIR"
-go test -v -tags synapse_blacklist,msc2716,msc3030,faster_joins -count=1 "${extra_test_args[@]}" "$@" ./tests/...
+go test -v -tags synapse_blacklist,msc2716,msc3030,faster_joins -count=1 "$@" ./tests/...

synapse/app/admin_cmd.py

@@ -210,7 +210,7 @@ def start(config_options: List[str]) -> None:
         config.logging.no_redirect_stdio = True
 
     # Explicitly disable background processes
-    config.worker.should_update_user_directory = False
+    config.server.update_user_directory = False
     config.worker.run_background_tasks = False
     config.worker.start_pushers = False
     config.worker.pusher_shard_config.instances = []

synapse/app/generic_worker.py

@@ -441,6 +441,22 @@ def start(config_options: List[str]) -> None:
         "synapse.app.user_dir",
     )
 
+    if config.worker.worker_app == "synapse.app.user_dir":
+        if config.server.update_user_directory:
+            sys.stderr.write(
+                "\nThe update_user_directory must be disabled in the main synapse process"
+                "\nbefore they can be run in a separate worker."
+                "\nPlease add ``update_user_directory: false`` to the main config"
+                "\n"
+            )
+            sys.exit(1)
+
+        # Force the pushers to start since they will be disabled in the main config
+        config.server.update_user_directory = True
+    else:
+        # For other worker types we force this to off.
+        config.server.update_user_directory = False
+
     synapse.events.USE_FROZEN_DICTS = config.server.use_frozen_dicts
     synapse.util.caches.TRACK_MEMORY_USAGE = config.caches.track_memory_usage
 

synapse/config/experimental.py

@@ -81,6 +81,3 @@ class ExperimentalConfig(Config):
 
         # MSC2815 (allow room moderators to view redacted event content)
         self.msc2815_enabled: bool = experimental.get("msc2815_enabled", False)
-
-        # MSC3786 (Add a default push rule to ignore m.room.server_acl events)
-        self.msc3786_enabled: bool = experimental.get("msc3786_enabled", False)

synapse/config/server.py

@@ -319,6 +319,10 @@ class ServerConfig(Config):
                 self.presence_router_config,
             ) = load_module(presence_router_config, ("presence", "presence_router"))
 
+        # Whether to update the user directory or not. This should be set to
+        # false only if we are updating the user directory in a worker
+        self.update_user_directory = config.get("update_user_directory", True)
+
         # whether to enable the media repository endpoints. This should be set
         # to false if the media repository is running as a separate endpoint;
         # doing so ensures that we will not run cache cleanup jobs on the

synapse/config/workers.py

@@ -311,13 +311,6 @@ class WorkerConfig(Config):
             new_option_name="notify_appservices_from_worker",
         )
 
-        self.should_update_user_directory = self._should_this_worker_perform_duty(
-            config,
-            legacy_master_option_name="update_user_directory",
-            legacy_worker_app_name="synapse.app.user_dir",
-            new_option_name="update_user_directory_from_worker",
-        )
-
     def _should_this_worker_perform_duty(
         self,
         config: Dict[str, Any],

synapse/handlers/user_directory.py

@@ -60,7 +60,7 @@ class UserDirectoryHandler(StateDeltasHandler):
         self.clock = hs.get_clock()
         self.notifier = hs.get_notifier()
         self.is_mine_id = hs.is_mine_id
-        self.update_user_directory = hs.config.worker.should_update_user_directory
+        self.update_user_directory = hs.config.server.update_user_directory
         self.search_all_users = hs.config.userdirectory.user_directory_search_all_users
         self.spam_checker = hs.get_spam_checker()
         # The current position in the current_state_delta stream

synapse/http/matrixfederationclient.py

@@ -73,7 +73,7 @@ from synapse.logging.context import make_deferred_yieldable, run_in_background
 from synapse.logging.opentracing import set_tag, start_active_span, tags
 from synapse.types import JsonDict
 from synapse.util import json_decoder
-from synapse.util.async_helpers import AwakenableSleeper, timeout_deferred
+from synapse.util.async_helpers import timeout_deferred
 from synapse.util.metrics import Measure
 
 if TYPE_CHECKING:
@@ -353,13 +353,6 @@ class MatrixFederationHttpClient:
 
         self._cooperator = Cooperator(scheduler=schedule)
 
-        self._sleeper = AwakenableSleeper(self.reactor)
-
-    def wake_destination(self, destination: str) -> None:
-        """Called when the remote server may have come back online."""
-
-        self._sleeper.wake(destination)
-
     async def _send_request_with_optional_trailing_slash(
         self,
         request: MatrixFederationRequest,
@@ -481,8 +474,6 @@ class MatrixFederationHttpClient:
             self._store,
             backoff_on_404=backoff_on_404,
             ignore_backoff=ignore_backoff,
-            notifier=self.hs.get_notifier(),
-            replication_client=self.hs.get_replication_command_handler(),
         )
 
         method_bytes = request.method.encode("ascii")
@@ -673,9 +664,7 @@ class MatrixFederationHttpClient:
                             delay,
                         )
 
-                        # Sleep for the calculated delay, or wake up immediately
-                        # if we get notified that the server is back up.
-                        await self._sleeper.sleep(request.destination, delay * 1000)
+                        await self.clock.sleep(delay)
                         retries_left -= 1
                     else:
                         raise

synapse/notifier.py

@@ -228,7 +228,9 @@ class Notifier:
         # Called when there are new things to stream over replication
         self.replication_callbacks: List[Callable[[], None]] = []
 
-        self._federation_client = hs.get_federation_http_client()
+        # Called when remote servers have come back online after having been
+        # down.
+        self.remote_server_up_callbacks: List[Callable[[str], None]] = []
 
         self._third_party_rules = hs.get_third_party_event_rules()
 
@@ -729,7 +731,3 @@ class Notifier:
         # circular dependencies.
         if self.federation_sender:
             self.federation_sender.wake_destination(server)
-
-        # Tell the federation client about the fact the server is back up, so
-        # that any in flight requests can be immediately retried.
-        self._federation_client.wake_destination(server)

synapse/push/baserules.py

@@ -277,21 +277,6 @@ BASE_APPEND_OVERRIDE_RULES: List[Dict[str, Any]] = [
         ],
         "actions": ["dont_notify"],
     },
-    # XXX: This is an experimental rule that is only enabled if msc3786_enabled
-    # is enabled, if it is not the rule gets filtered out in _load_rules() in
-    # PushRulesWorkerStore
-    {
-        "rule_id": "global/override/.org.matrix.msc3786.rule.room.server_acl",
-        "conditions": [
-            {
-                "kind": "event_match",
-                "key": "type",
-                "pattern": "m.room.server_acl",
-                "_cache_key": "_room_server_acl",
-            }
-        ],
-        "actions": ["dont_notify"],
-    },
 ]
 
 

synapse/replication/tcp/__init__.py

@@ -15,7 +15,7 @@
 """This module implements the TCP replication protocol used by synapse to
 communicate between the master process and its workers (when they're enabled).
 
-Further details can be found in docs/tcp_replication.md
+Further details can be found in docs/tcp_replication.rst
 
 
 Structure of the module:

synapse/storage/databases/main/push_rule.py

@@ -17,7 +17,6 @@ import logging
 from typing import TYPE_CHECKING, Dict, List, Tuple, Union
 
 from synapse.api.errors import StoreError
-from synapse.config.homeserver import ExperimentalConfig
 from synapse.push.baserules import list_with_base_rules
 from synapse.replication.slave.storage._slaved_id_tracker import SlavedIdTracker
 from synapse.storage._base import SQLBaseStore, db_to_json
@@ -43,21 +42,7 @@ if TYPE_CHECKING:
 logger = logging.getLogger(__name__)
 
 
-def _is_experimental_rule_enabled(
-    rule_id: str, experimental_config: ExperimentalConfig
-) -> bool:
-    """Used by `_load_rules` to filter out experimental rules when they
-    have not been enabled.
-    """
-    if (
-        rule_id == "global/override/.org.matrix.msc3786.rule.room.server_acl"
-        and not experimental_config.msc3786_enabled
-    ):
-        return False
-    return True
-
-
-def _load_rules(rawrules, enabled_map, experimental_config: ExperimentalConfig):
+def _load_rules(rawrules, enabled_map):
     ruleslist = []
     for rawrule in rawrules:
         rule = dict(rawrule)
@@ -66,26 +51,17 @@ def _load_rules(rawrules, enabled_map, experimental_config: ExperimentalConfig):
         rule["default"] = False
         ruleslist.append(rule)
 
-    # We're going to be mutating this a lot, so copy it. We also filter out
-    # any experimental default push rules that aren't enabled.
-    rules = [
-        rule
-        for rule in list_with_base_rules(ruleslist)
-        if _is_experimental_rule_enabled(rule["rule_id"], experimental_config)
-    ]
+    # We're going to be mutating this a lot, so do a deep copy
+    rules = list(list_with_base_rules(ruleslist))
 
     for i, rule in enumerate(rules):
         rule_id = rule["rule_id"]
-
-        if rule_id not in enabled_map:
-            continue
-        if rule.get("enabled", True) == bool(enabled_map[rule_id]):
-            continue
-
-        # Rules are cached across users.
-        rule = dict(rule)
-        rule["enabled"] = bool(enabled_map[rule_id])
-        rules[i] = rule
+        if rule_id in enabled_map:
+            if rule.get("enabled", True) != bool(enabled_map[rule_id]):
+                # Rules are cached across users.
+                rule = dict(rule)
+                rule["enabled"] = bool(enabled_map[rule_id])
+                rules[i] = rule
 
     return rules
 
@@ -165,7 +141,7 @@ class PushRulesWorkerStore(
 
         enabled_map = await self.get_push_rules_enabled_for_user(user_id)
 
-        return _load_rules(rows, enabled_map, self.hs.config.experimental)
+        return _load_rules(rows, enabled_map)
 
     @cached(max_entries=5000)
     async def get_push_rules_enabled_for_user(self, user_id) -> Dict[str, bool]:
@@ -224,9 +200,7 @@ class PushRulesWorkerStore(
         enabled_map_by_user = await self.bulk_get_push_rules_enabled(user_ids)
 
         for user_id, rules in results.items():
-            results[user_id] = _load_rules(
-                rules, enabled_map_by_user.get(user_id, {}), self.hs.config.experimental
-            )
+            results[user_id] = _load_rules(rules, enabled_map_by_user.get(user_id, {}))
 
         return results
 

synapse/util/async_helpers.py

@@ -778,60 +778,3 @@ def delay_cancellation(awaitable: Awaitable[T]) -> Awaitable[T]:
     new_deferred: "defer.Deferred[T]" = defer.Deferred(handle_cancel)
     deferred.chainDeferred(new_deferred)
     return new_deferred
-
-
-class AwakenableSleeper:
-    """Allows explicitly waking up deferreds related to an entity that are
-    currently sleeping.
-    """
-
-    def __init__(self, reactor: IReactorTime) -> None:
-        self._streams: Dict[str, Set[defer.Deferred[None]]] = {}
-        self._reactor = reactor
-
-    def wake(self, name: str) -> None:
-        """Wake everything related to `name` that is currently sleeping."""
-        stream_set = self._streams.pop(name, set())
-        for deferred in stream_set:
-            try:
-                with PreserveLoggingContext():
-                    deferred.callback(None)
-            except Exception:
-                pass
-
-    async def sleep(self, name: str, delay_ms: int) -> None:
-        """Sleep for the given number of milliseconds, or return if the given
-        `name` is explicitly woken up.
-        """
-
-        # Create a deferred that gets called in N seconds
-        sleep_deferred: "defer.Deferred[None]" = defer.Deferred()
-        call = self._reactor.callLater(delay_ms / 1000, sleep_deferred.callback, None)
-
-        # Create a deferred that will get called if `wake` is called with
-        # the same `name`.
-        stream_set = self._streams.setdefault(name, set())
-        notify_deferred: "defer.Deferred[None]" = defer.Deferred()
-        stream_set.add(notify_deferred)
-
-        try:
-            # Wait for either the delay or for `wake` to be called.
-            await make_deferred_yieldable(
-                defer.DeferredList(
-                    [sleep_deferred, notify_deferred],
-                    fireOnOneCallback=True,
-                    fireOnOneErrback=True,
-                    consumeErrors=True,
-                )
-            )
-        finally:
-            # Clean up the state
-            curr_stream_set = self._streams.get(name)
-            if curr_stream_set is not None:
-                curr_stream_set.discard(notify_deferred)
-                if len(curr_stream_set) == 0:
-                    self._streams.pop(name)
-
-            # Cancel the sleep if we were woken up
-            if call.active():
-                call.cancel()

synapse/util/retryutils.py

@@ -14,17 +14,13 @@
 import logging
 import random
 from types import TracebackType
-from typing import TYPE_CHECKING, Any, Optional, Type
+from typing import Any, Optional, Type
 
 import synapse.logging.context
 from synapse.api.errors import CodeMessageException
 from synapse.storage import DataStore
 from synapse.util import Clock
 
-if TYPE_CHECKING:
-    from synapse.notifier import Notifier
-    from synapse.replication.tcp.handler import ReplicationCommandHandler
-
 logger = logging.getLogger(__name__)
 
 # the initial backoff, after the first transaction fails
@@ -135,8 +131,6 @@ class RetryDestinationLimiter:
         retry_interval: int,
         backoff_on_404: bool = False,
         backoff_on_failure: bool = True,
-        notifier: Optional["Notifier"] = None,
-        replication_client: Optional["ReplicationCommandHandler"] = None,
     ):
         """Marks the destination as "down" if an exception is thrown in the
         context, except for CodeMessageException with code < 500.
@@ -166,9 +160,6 @@ class RetryDestinationLimiter:
         self.backoff_on_404 = backoff_on_404
         self.backoff_on_failure = backoff_on_failure
 
-        self.notifier = notifier
-        self.replication_client = replication_client
-
     def __enter__(self) -> None:
         pass
 
@@ -248,19 +239,6 @@ class RetryDestinationLimiter:
                     retry_last_ts,
                     self.retry_interval,
                 )
-
-                if self.notifier:
-                    # Inform the relevant places that the remote server is back up.
-                    self.notifier.notify_remote_server_up(self.destination)
-
-                if self.replication_client:
-                    # If we're on a worker we try and inform master about this. The
-                    # replication client doesn't hook into the notifier to avoid
-                    # infinite loops where we send a `REMOTE_SERVER_UP` command to
-                    # master, which then echoes it back to us which in turn pokes
-                    # the notifier.
-                    self.replication_client.send_remote_server_up(self.destination)
-
             except Exception:
                 logger.exception("Failed to store destination_retry_timings")
 

tests/config/test_workers.py

@@ -286,30 +286,3 @@ class WorkerDutyConfigTestCase(TestCase):
                 "notify_appservices_from_worker",
             )
         )
-
-    def test_worker_duty_configs(self) -> None:
-        """
-        Additional tests for the worker duties
-        """
-
-        worker1_config = self._make_worker_config(
-            worker_app="synapse.app.generic_worker",
-            worker_name="worker1",
-            extras={
-                "notify_appservices_from_worker": "worker2",
-                "update_user_directory_from_worker": "worker1",
-            },
-        )
-        self.assertFalse(worker1_config.should_notify_appservices)
-        self.assertTrue(worker1_config.should_update_user_directory)
-
-        worker2_config = self._make_worker_config(
-            worker_app="synapse.app.generic_worker",
-            worker_name="worker2",
-            extras={
-                "notify_appservices_from_worker": "worker2",
-                "update_user_directory_from_worker": "worker1",
-            },
-        )
-        self.assertTrue(worker2_config.should_notify_appservices)
-        self.assertFalse(worker2_config.should_update_user_directory)

tests/util/test_async_helpers.py

@@ -28,7 +28,6 @@ from synapse.logging.context import (
     make_deferred_yieldable,
 )
 from synapse.util.async_helpers import (
-    AwakenableSleeper,
     ObservableDeferred,
     concurrently_execute,
     delay_cancellation,
@@ -36,7 +35,6 @@ from synapse.util.async_helpers import (
     timeout_deferred,
 )
 
-from tests.server import get_clock
 from tests.unittest import TestCase
 
 
@@ -498,81 +496,3 @@ class DelayCancellationTests(TestCase):
         # logging context.
         blocking_d.callback(None)
         self.successResultOf(d)
-
-
-class AwakenableSleeperTests(TestCase):
-    "Tests AwakenableSleeper"
-
-    def test_sleep(self):
-        reactor, _ = get_clock()
-        sleeper = AwakenableSleeper(reactor)
-
-        d = defer.ensureDeferred(sleeper.sleep("name", 1000))
-
-        reactor.pump([0.0])
-        self.assertFalse(d.called)
-
-        reactor.advance(0.5)
-        self.assertFalse(d.called)
-
-        reactor.advance(0.6)
-        self.assertTrue(d.called)
-
-    def test_explicit_wake(self):
-        reactor, _ = get_clock()
-        sleeper = AwakenableSleeper(reactor)
-
-        d = defer.ensureDeferred(sleeper.sleep("name", 1000))
-
-        reactor.pump([0.0])
-        self.assertFalse(d.called)
-
-        reactor.advance(0.5)
-        self.assertFalse(d.called)
-
-        sleeper.wake("name")
-        self.assertTrue(d.called)
-
-        reactor.advance(0.6)
-
-    def test_multiple_sleepers_timeout(self):
-        reactor, _ = get_clock()
-        sleeper = AwakenableSleeper(reactor)
-
-        d1 = defer.ensureDeferred(sleeper.sleep("name", 1000))
-
-        reactor.advance(0.6)
-        self.assertFalse(d1.called)
-
-        # Add another sleeper
-        d2 = defer.ensureDeferred(sleeper.sleep("name", 1000))
-
-        # Only the first sleep should time out now.
-        reactor.advance(0.6)
-        self.assertTrue(d1.called)
-        self.assertFalse(d2.called)
-
-        reactor.advance(0.6)
-        self.assertTrue(d2.called)
-
-    def test_multiple_sleepers_wake(self):
-        reactor, _ = get_clock()
-        sleeper = AwakenableSleeper(reactor)
-
-        d1 = defer.ensureDeferred(sleeper.sleep("name", 1000))
-
-        reactor.advance(0.5)
-        self.assertFalse(d1.called)
-
-        # Add another sleeper
-        d2 = defer.ensureDeferred(sleeper.sleep("name", 1000))
-
-        # Neither should fire yet
-        reactor.advance(0.3)
-        self.assertFalse(d1.called)
-        self.assertFalse(d2.called)
-
-        # Explicitly waking both up works
-        sleeper.wake("name")
-        self.assertTrue(d1.called)
-        self.assertTrue(d2.called)