Merge remote-tracking branch 'origin/release-v1.28.0' into toml/keycloak_hints

.buildkite/scripts/test_old_deps.sh

@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/bin/bash
 
 # this script is run by buildkite in a plain `xenial` container; it installs the
 # minimal requirements for tox and hands over to the py35-old tox environment.

.buildkite/scripts/test_synapse_port_db.sh

@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/bin/bash
 #
 # Test script for 'synapse_port_db', which creates a virtualenv, installs Synapse along
 # with additional dependencies needed for the test (such as coverage or the PostgreSQL

.git-blame-ignore-revs

@@ -1,8 +0,0 @@
-# Black reformatting (#5482).
-32e7c9e7f20b57dd081023ac42d6931a8da9b3a3
-
-# Target Python 3.5 with black (#8664).
-aff1eb7c671b0a3813407321d2702ec46c71fa56
-
-# Update black to 20.8b1 (#9381).
-0a00b7ff14890987f09112a2ae696c61001e6cf1

.gitignore

@@ -6,14 +6,13 @@
 *.egg
 *.egg-info
 *.lock
-*.py[cod]
+*.pyc
 *.snap
 *.tac
 _trial_temp/
 _trial_temp*/
 /out
 .DS_Store
-__pycache__/
 
 # stuff that is likely to exist when you run a server locally
 /*.db

CHANGES.md

@@ -1,265 +1,3 @@
-Synapse 1.31.0 (2021-04-06)
-===========================
-
-**Note:** As announced in v1.25.0, and in line with the deprecation policy for platform dependencies, this is the last release to support Python 3.5 and PostgreSQL 9.5. Future versions of Synapse will require Python 3.6+ and PostgreSQL 9.6+, as per our [deprecation policy](docs/deprecation_policy.md).
-
-This is also the last release that the Synapse team will be publishing packages for Debian Stretch and Ubuntu Xenial.
-
-
-Improved Documentation
-----------------------
-
-- Add a document describing the deprecation policy for platform dependencies. ([\#9723](https://github.com/matrix-org/synapse/issues/9723))
-
-
-Internal Changes
-----------------
-
-- Revert using `dmypy run` in lint script. ([\#9720](https://github.com/matrix-org/synapse/issues/9720))
-- Pin flake8-bugbear's version. ([\#9734](https://github.com/matrix-org/synapse/issues/9734))
-
-
-Synapse 1.31.0rc1 (2021-03-30)
-==============================
-
-Features
---------
-
-- Add support to OpenID Connect login for requiring attributes on the `userinfo` response. Contributed by Hubbe King. ([\#9609](https://github.com/matrix-org/synapse/issues/9609))
-- Add initial experimental support for a "space summary" API. ([\#9643](https://github.com/matrix-org/synapse/issues/9643), [\#9652](https://github.com/matrix-org/synapse/issues/9652), [\#9653](https://github.com/matrix-org/synapse/issues/9653))
-- Add support for the busy presence state as described in [MSC3026](https://github.com/matrix-org/matrix-doc/pull/3026). ([\#9644](https://github.com/matrix-org/synapse/issues/9644))
-- Add support for credentials for proxy authentication in the `HTTPS_PROXY` environment variable. ([\#9657](https://github.com/matrix-org/synapse/issues/9657))
-
-
-Bugfixes
---------
-
-- Fix a longstanding bug that could cause issues when editing a reply to a message. ([\#9585](https://github.com/matrix-org/synapse/issues/9585))
-- Fix the `/capabilities` endpoint to return `m.change_password` as disabled if the local password database is not used for authentication. Contributed by @dklimpel. ([\#9588](https://github.com/matrix-org/synapse/issues/9588))
-- Check if local passwords are enabled before setting them for the user. ([\#9636](https://github.com/matrix-org/synapse/issues/9636))
-- Fix a bug where federation sending can stall due to `concurrent access` database exceptions when it falls behind. ([\#9639](https://github.com/matrix-org/synapse/issues/9639))
-- Fix a bug introduced in Synapse 1.30.1 which meant the suggested `pip` incantation to install an updated `cryptography` was incorrect. ([\#9699](https://github.com/matrix-org/synapse/issues/9699))
-
-
-Updates to the Docker image
----------------------------
-
-- Speed up Docker builds and make it nicer to test against Complement while developing (install all dependencies before copying the project). ([\#9610](https://github.com/matrix-org/synapse/issues/9610))
-- Include [opencontainers labels](https://github.com/opencontainers/image-spec/blob/master/annotations.md#pre-defined-annotation-keys) in the Docker image. ([\#9612](https://github.com/matrix-org/synapse/issues/9612))
-
-
-Improved Documentation
-----------------------
-
-- Clarify that `register_new_matrix_user` is present also when installed via non-pip package. ([\#9074](https://github.com/matrix-org/synapse/issues/9074))
-- Update source install documentation to mention platform prerequisites before the source install steps. ([\#9667](https://github.com/matrix-org/synapse/issues/9667))
-- Improve worker documentation for fallback/web auth endpoints. ([\#9679](https://github.com/matrix-org/synapse/issues/9679))
-- Update the sample configuration for OIDC authentication. ([\#9695](https://github.com/matrix-org/synapse/issues/9695))
-
-
-Internal Changes
-----------------
-
-- Preparatory steps for removing redundant `outlier` data from `event_json.internal_metadata` column. ([\#9411](https://github.com/matrix-org/synapse/issues/9411))
-- Add type hints to the caching module. ([\#9442](https://github.com/matrix-org/synapse/issues/9442))
-- Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9499](https://github.com/matrix-org/synapse/issues/9499), [\#9659](https://github.com/matrix-org/synapse/issues/9659))
-- Add additional type hints to the Homeserver object. ([\#9631](https://github.com/matrix-org/synapse/issues/9631), [\#9638](https://github.com/matrix-org/synapse/issues/9638), [\#9675](https://github.com/matrix-org/synapse/issues/9675), [\#9681](https://github.com/matrix-org/synapse/issues/9681))
-- Only save remote cross-signing and device keys if they're different from the current ones. ([\#9634](https://github.com/matrix-org/synapse/issues/9634))
-- Rename storage function to fix spelling and not conflict with another function's name. ([\#9637](https://github.com/matrix-org/synapse/issues/9637))
-- Improve performance of federation catch up by sending the latest events in the room to the remote, rather than just the last event sent by the local server. ([\#9640](https://github.com/matrix-org/synapse/issues/9640), [\#9664](https://github.com/matrix-org/synapse/issues/9664))
-- In the `federation_client` commandline client, stop automatically adding the URL prefix, so that servlets on other prefixes can be tested. ([\#9645](https://github.com/matrix-org/synapse/issues/9645))
-- In the `federation_client` commandline client, handle inline `signing_key`s in `homeserver.yaml`. ([\#9647](https://github.com/matrix-org/synapse/issues/9647))
-- Fixed some antipattern issues to improve code quality. ([\#9649](https://github.com/matrix-org/synapse/issues/9649))
-- Add a storage method for pulling all current user presence state from the database. ([\#9650](https://github.com/matrix-org/synapse/issues/9650))
-- Import `HomeServer` from the proper module. ([\#9665](https://github.com/matrix-org/synapse/issues/9665))
-- Increase default join ratelimiting burst rate. ([\#9674](https://github.com/matrix-org/synapse/issues/9674))
-- Add type hints to third party event rules and visibility modules. ([\#9676](https://github.com/matrix-org/synapse/issues/9676))
-- Bump mypy-zope to 0.2.13 to fix "Cannot determine consistent method resolution order (MRO)" errors when running mypy a second time. ([\#9678](https://github.com/matrix-org/synapse/issues/9678))
-- Use interpreter from `$PATH` via `/usr/bin/env` instead of absolute paths in various scripts. ([\#9689](https://github.com/matrix-org/synapse/issues/9689))
-- Make it possible to use `dmypy`. ([\#9692](https://github.com/matrix-org/synapse/issues/9692))
-- Suppress "CryptographyDeprecationWarning: int_from_bytes is deprecated". ([\#9698](https://github.com/matrix-org/synapse/issues/9698))
-- Use `dmypy run` in lint script for improved performance in type-checking while developing. ([\#9701](https://github.com/matrix-org/synapse/issues/9701))
-- Fix undetected mypy error when using Python 3.6. ([\#9703](https://github.com/matrix-org/synapse/issues/9703))
-- Fix type-checking CI on develop. ([\#9709](https://github.com/matrix-org/synapse/issues/9709))
-
-
-Synapse 1.30.1 (2021-03-26)
-===========================
-
-This release is identical to Synapse 1.30.0, with the exception of explicitly
-setting a minimum version of Python's Cryptography library to ensure that users
-of Synapse are protected from the recent [OpenSSL security advisories](https://mta.openssl.org/pipermail/openssl-announce/2021-March/000198.html),
-especially CVE-2021-3449.
-
-Note that Cryptography defaults to bundling its own statically linked copy of
-OpenSSL, which means that you may not be protected by your operating system's
-security updates.
-
-It's also worth noting that Cryptography no longer supports Python 3.5, so
-admins deploying to older environments may not be protected against this or
-future vulnerabilities. Synapse will be dropping support for Python 3.5 at the
-end of March.
-
-
-Updates to the Docker image
----------------------------
-
-- Ensure that the docker container has up to date versions of openssl. ([\#9697](https://github.com/matrix-org/synapse/issues/9697))
-
-
-Internal Changes
-----------------
-
-- Enforce that `cryptography` dependency is up to date to ensure it has the most recent openssl patches. ([\#9697](https://github.com/matrix-org/synapse/issues/9697))
-
-
-Synapse 1.30.0 (2021-03-22)
-===========================
-
-Note that this release deprecates the ability for appservices to
-call `POST /_matrix/client/r0/register`  without the body parameter `type`. Appservice
-developers should use a `type` value of `m.login.application_service` as
-per [the spec](https://matrix.org/docs/spec/application_service/r0.1.2#server-admin-style-permissions).
-In future releases, calling this endpoint with an access token - but without a `m.login.application_service`
-type - will fail.
-
-
-No significant changes.
-
-
-Synapse 1.30.0rc1 (2021-03-16)
-==============================
-
-Features
---------
-
-- Add prometheus metrics for number of users successfully registering and logging in. ([\#9510](https://github.com/matrix-org/synapse/issues/9510), [\#9511](https://github.com/matrix-org/synapse/issues/9511), [\#9573](https://github.com/matrix-org/synapse/issues/9573))
-- Add `synapse_federation_last_sent_pdu_time` and `synapse_federation_last_received_pdu_time` prometheus metrics, which monitor federation delays by reporting the timestamps of messages sent and received to a set of remote servers. ([\#9540](https://github.com/matrix-org/synapse/issues/9540))
-- Add support for generating JSON Web Tokens dynamically for use as OIDC client secrets. ([\#9549](https://github.com/matrix-org/synapse/issues/9549))
-- Optimise handling of incomplete room history for incoming federation. ([\#9601](https://github.com/matrix-org/synapse/issues/9601))
-- Finalise support for allowing clients to pick an SSO Identity Provider ([MSC2858](https://github.com/matrix-org/matrix-doc/pull/2858)). ([\#9617](https://github.com/matrix-org/synapse/issues/9617))
-- Tell spam checker modules about the SSO IdP a user registered through if one was used. ([\#9626](https://github.com/matrix-org/synapse/issues/9626))
-
-
-Bugfixes
---------
-
-- Fix long-standing bug when generating thumbnails for some images with transparency: `TypeError: cannot unpack non-iterable int object`. ([\#9473](https://github.com/matrix-org/synapse/issues/9473))
-- Purge chain cover indexes for events that were purged prior to Synapse v1.29.0. ([\#9542](https://github.com/matrix-org/synapse/issues/9542), [\#9583](https://github.com/matrix-org/synapse/issues/9583))
-- Fix bug where federation requests were not correctly retried on 5xx responses. ([\#9567](https://github.com/matrix-org/synapse/issues/9567))
-- Fix re-activating an account via the admin API when local passwords are disabled. ([\#9587](https://github.com/matrix-org/synapse/issues/9587))
-- Fix a bug introduced in Synapse 1.20 which caused incoming federation transactions to stack up, causing slow recovery from outages. ([\#9597](https://github.com/matrix-org/synapse/issues/9597))
-- Fix a bug introduced in v1.28.0 where the OpenID Connect callback endpoint could error with a `MacaroonInitException`. ([\#9620](https://github.com/matrix-org/synapse/issues/9620))
-- Fix Internal Server Error on `GET /_synapse/client/saml2/authn_response` request. ([\#9623](https://github.com/matrix-org/synapse/issues/9623))
-
-
-Updates to the Docker image
----------------------------
-
-- Make use of an improved malloc implementation (`jemalloc`) in the docker image. ([\#8553](https://github.com/matrix-org/synapse/issues/8553))
-
-
-Improved Documentation
-----------------------
-
-- Add relayd entry to reverse proxy example configurations. ([\#9508](https://github.com/matrix-org/synapse/issues/9508))
-- Improve the SAML2 upgrade notes for 1.27.0. ([\#9550](https://github.com/matrix-org/synapse/issues/9550))
-- Link to the "List user's media" admin API from the media admin API docs. ([\#9571](https://github.com/matrix-org/synapse/issues/9571))
-- Clarify the spam checker modules documentation example to mention that `parse_config` is a required method. ([\#9580](https://github.com/matrix-org/synapse/issues/9580))
-- Clarify the sample configuration for `stats` settings. ([\#9604](https://github.com/matrix-org/synapse/issues/9604))
-
-
-Deprecations and Removals
--------------------------
-
-- The `synapse_federation_last_sent_pdu_age` and `synapse_federation_last_received_pdu_age` prometheus metrics have been removed. They are replaced by `synapse_federation_last_sent_pdu_time` and `synapse_federation_last_received_pdu_time`. ([\#9540](https://github.com/matrix-org/synapse/issues/9540))
-- Registering an Application Service user without using the `m.login.application_service` login type will be unsupported in an upcoming Synapse release. ([\#9559](https://github.com/matrix-org/synapse/issues/9559))
-
-
-Internal Changes
-----------------
-
-- Add tests to ResponseCache. ([\#9458](https://github.com/matrix-org/synapse/issues/9458))
-- Add type hints to purge room and server notice admin API. ([\#9520](https://github.com/matrix-org/synapse/issues/9520))
-- Add extra logging to ObservableDeferred when callbacks throw exceptions. ([\#9523](https://github.com/matrix-org/synapse/issues/9523))
-- Fix incorrect type hints. ([\#9528](https://github.com/matrix-org/synapse/issues/9528), [\#9543](https://github.com/matrix-org/synapse/issues/9543), [\#9591](https://github.com/matrix-org/synapse/issues/9591), [\#9608](https://github.com/matrix-org/synapse/issues/9608), [\#9618](https://github.com/matrix-org/synapse/issues/9618))
-- Add an additional test for purging a room. ([\#9541](https://github.com/matrix-org/synapse/issues/9541))
-- Add a `.git-blame-ignore-revs` file with the hashes of auto-formatting. ([\#9560](https://github.com/matrix-org/synapse/issues/9560))
-- Increase the threshold before which outbound federation to a server goes into "catch up" mode, which is expensive for the remote server to handle. ([\#9561](https://github.com/matrix-org/synapse/issues/9561))
-- Fix spurious errors reported by the `config-lint.sh` script. ([\#9562](https://github.com/matrix-org/synapse/issues/9562))
-- Fix type hints and tests for BlacklistingAgentWrapper and BlacklistingReactorWrapper. ([\#9563](https://github.com/matrix-org/synapse/issues/9563))
-- Do not have mypy ignore type hints from unpaddedbase64. ([\#9568](https://github.com/matrix-org/synapse/issues/9568))
-- Improve efficiency of calculating the auth chain in large rooms. ([\#9576](https://github.com/matrix-org/synapse/issues/9576))
-- Convert `synapse.types.Requester` to an `attrs` class. ([\#9586](https://github.com/matrix-org/synapse/issues/9586))
-- Add logging for redis connection setup. ([\#9590](https://github.com/matrix-org/synapse/issues/9590))
-- Improve logging when processing incoming transactions. ([\#9596](https://github.com/matrix-org/synapse/issues/9596))
-- Remove unused `stats.retention` setting, and emit a warning if stats are disabled. ([\#9604](https://github.com/matrix-org/synapse/issues/9604))
-- Prevent attempting to bundle aggregations for state events in /context APIs. ([\#9619](https://github.com/matrix-org/synapse/issues/9619))
-
-
-Synapse 1.29.0 (2021-03-08)
-===========================
-
-Note that synapse now expects an `X-Forwarded-Proto` header when used with a reverse proxy. Please see [UPGRADE.rst](UPGRADE.rst#upgrading-to-v1290) for more details on this change.
-
-
-No significant changes.
-
-
-Synapse 1.29.0rc1 (2021-03-04)
-==============================
-
-Features
---------
-
-- Add rate limiters to cross-user key sharing requests. ([\#8957](https://github.com/matrix-org/synapse/issues/8957))
-- Add `order_by` to the admin API `GET /_synapse/admin/v1/users/<user_id>/media`. Contributed by @dklimpel. ([\#8978](https://github.com/matrix-org/synapse/issues/8978))
-- Add some configuration settings to make users' profile data more private. ([\#9203](https://github.com/matrix-org/synapse/issues/9203))
-- The `no_proxy` and `NO_PROXY` environment variables are now respected in proxied HTTP clients with the lowercase form taking precedence if both are present. Additionally, the lowercase `https_proxy` environment variable is now respected in proxied HTTP clients on top of existing support for the uppercase `HTTPS_PROXY` form and takes precedence if both are present. Contributed by Timothy Leung. ([\#9372](https://github.com/matrix-org/synapse/issues/9372))
-- Add a configuration option, `user_directory.prefer_local_users`, which when enabled will make it more likely for users on the same server as you to appear above other users. ([\#9383](https://github.com/matrix-org/synapse/issues/9383), [\#9385](https://github.com/matrix-org/synapse/issues/9385))
-- Add support for regenerating thumbnails if they have been deleted but the original image is still stored. ([\#9438](https://github.com/matrix-org/synapse/issues/9438))
-- Add support for `X-Forwarded-Proto` header when using a reverse proxy. ([\#9472](https://github.com/matrix-org/synapse/issues/9472), [\#9501](https://github.com/matrix-org/synapse/issues/9501), [\#9512](https://github.com/matrix-org/synapse/issues/9512), [\#9539](https://github.com/matrix-org/synapse/issues/9539))
-
-
-Bugfixes
---------
-
-- Fix a bug where users' pushers were not all deleted when they deactivated their account. ([\#9285](https://github.com/matrix-org/synapse/issues/9285), [\#9516](https://github.com/matrix-org/synapse/issues/9516))
-- Fix a bug where a lot of unnecessary presence updates were sent when joining a room. ([\#9402](https://github.com/matrix-org/synapse/issues/9402))
-- Fix a bug that caused multiple calls to the experimental `shared_rooms` endpoint to return stale results. ([\#9416](https://github.com/matrix-org/synapse/issues/9416))
-- Fix a bug in single sign-on which could cause a "No session cookie found" error. ([\#9436](https://github.com/matrix-org/synapse/issues/9436))
-- Fix bug introduced in v1.27.0 where allowing a user to choose their own username when logging in via single sign-on did not work unless an `idp_icon` was defined. ([\#9440](https://github.com/matrix-org/synapse/issues/9440))
-- Fix a bug introduced in v1.26.0 where some sequences were not properly configured when running `synapse_port_db`. ([\#9449](https://github.com/matrix-org/synapse/issues/9449))
-- Fix deleting pushers when using sharded pushers. ([\#9465](https://github.com/matrix-org/synapse/issues/9465), [\#9466](https://github.com/matrix-org/synapse/issues/9466), [\#9479](https://github.com/matrix-org/synapse/issues/9479), [\#9536](https://github.com/matrix-org/synapse/issues/9536))
-- Fix missing startup checks for the consistency of certain PostgreSQL sequences. ([\#9470](https://github.com/matrix-org/synapse/issues/9470))
-- Fix a long-standing bug where the media repository could leak file descriptors while previewing media. ([\#9497](https://github.com/matrix-org/synapse/issues/9497))
-- Properly purge the event chain cover index when purging history. ([\#9498](https://github.com/matrix-org/synapse/issues/9498))
-- Fix missing chain cover index due to a schema delta not being applied correctly. Only affected servers that ran development versions. ([\#9503](https://github.com/matrix-org/synapse/issues/9503))
-- Fix a bug introduced in v1.25.0 where `/_synapse/admin/join/` would fail when given a room alias. ([\#9506](https://github.com/matrix-org/synapse/issues/9506))
-- Prevent presence background jobs from running when presence is disabled. ([\#9530](https://github.com/matrix-org/synapse/issues/9530))
-- Fix rare edge case that caused a background update to fail if the server had rejected an event that had duplicate auth events. ([\#9537](https://github.com/matrix-org/synapse/issues/9537))
-
-
-Improved Documentation
-----------------------
-
-- Update the example systemd config to propagate reloads to individual units. ([\#9463](https://github.com/matrix-org/synapse/issues/9463))
-
-
-Internal Changes
-----------------
-
-- Add documentation and type hints to `parse_duration`. ([\#9432](https://github.com/matrix-org/synapse/issues/9432))
-- Remove vestiges of `uploads_path` configuration setting. ([\#9462](https://github.com/matrix-org/synapse/issues/9462))
-- Add a comment about systemd-python. ([\#9464](https://github.com/matrix-org/synapse/issues/9464))
-- Test that we require validated email for email pushers. ([\#9496](https://github.com/matrix-org/synapse/issues/9496))
-- Allow python to generate bytecode for synapse. ([\#9502](https://github.com/matrix-org/synapse/issues/9502))
-- Fix incorrect type hints. ([\#9515](https://github.com/matrix-org/synapse/issues/9515), [\#9518](https://github.com/matrix-org/synapse/issues/9518))
-- Add type hints to device and event report admin API. ([\#9519](https://github.com/matrix-org/synapse/issues/9519))
-- Add type hints to user admin API. ([\#9521](https://github.com/matrix-org/synapse/issues/9521))
-- Bump the versions of mypy and mypy-zope used for static type checking. ([\#9529](https://github.com/matrix-org/synapse/issues/9529))
-
-
 Synapse 1.28.0 (2021-02-25)
 ===========================
 

INSTALL.md

@@ -6,7 +6,7 @@ There are 3 steps to follow under **Installation Instructions**.
   - [Choosing your server name](#choosing-your-server-name)
   - [Installing Synapse](#installing-synapse)
     - [Installing from source](#installing-from-source)
-      - [Platform-specific prerequisites](#platform-specific-prerequisites)
+      - [Platform-Specific Instructions](#platform-specific-instructions)
         - [Debian/Ubuntu/Raspbian](#debianubunturaspbian)
         - [ArchLinux](#archlinux)
         - [CentOS/Fedora](#centosfedora)
@@ -38,7 +38,6 @@ There are 3 steps to follow under **Installation Instructions**.
     - [URL previews](#url-previews)
     - [Troubleshooting Installation](#troubleshooting-installation)
 
-
 ## Choosing your server name
 
 It is important to choose the name for your server before you install Synapse,
@@ -61,14 +60,17 @@ that your email address is probably `user@example.com` rather than
 
 (Prebuilt packages are available for some platforms - see [Prebuilt packages](#prebuilt-packages).)
 
-When installing from source please make sure that the [Platform-specific prerequisites](#platform-specific-prerequisites) are already installed.
-
 System requirements:
 
 - POSIX-compliant system (tested on Linux & OS X)
 - Python 3.5.2 or later, up to Python 3.9.
 - At least 1GB of free RAM if you want to join large public rooms like #matrix:matrix.org
 
+Synapse is written in Python but some of the libraries it uses are written in
+C. So before we can install Synapse itself we need a working C compiler and the
+header files for Python C extensions. See [Platform-Specific
+Instructions](#platform-specific-instructions) for information on installing
+these on various platforms.
 
 To install the Synapse homeserver run:
 
@@ -126,11 +128,7 @@ source env/bin/activate
 synctl start
 ```
 
-#### Platform-specific prerequisites
-
-Synapse is written in Python but some of the libraries it uses are written in
-C. So before we can install Synapse itself we need a working C compiler and the
-header files for Python C extensions.
+#### Platform-Specific Instructions
 
 ##### Debian/Ubuntu/Raspbian
 
@@ -528,24 +526,14 @@ email will be disabled.
 
 The easiest way to create a new user is to do so from a client like [Element](https://element.io/).
 
-Alternatively, you can do so from the command line. This can be done as follows:
+Alternatively you can do so from the command line if you have installed via pip.
 
- 1. If synapse was installed via pip, activate the virtualenv as follows (if Synapse was
-    installed via a prebuilt package, `register_new_matrix_user` should already be
-    on the search path):
-    ```sh
-    cd ~/synapse
-    source env/bin/activate
-    synctl start # if not already running
-    ```
- 2. Run the following command:
-    ```sh
-    register_new_matrix_user -c homeserver.yaml http://localhost:8008
-    ```
+This can be done as follows:
 
-This will prompt you to add details for the new user, and will then connect to
-the running Synapse to create the new user. For example:
-```
+```sh
+$ source ~/synapse/env/bin/activate
+$ synctl start # if not already running
+$ register_new_matrix_user -c homeserver.yaml http://localhost:8008
 New user localpart: erikj
 Password:
 Confirm password:

MANIFEST.in

@@ -20,10 +20,9 @@ recursive-include scripts *
 recursive-include scripts-dev *
 recursive-include synapse *.pyi
 recursive-include tests *.py
-recursive-include tests *.pem
-recursive-include tests *.p8
-recursive-include tests *.crt
-recursive-include tests *.key
+include tests/http/ca.crt
+include tests/http/ca.key
+include tests/http/server.key
 
 recursive-include synapse/res *
 recursive-include synapse/static *.css

README.rst

@@ -183,9 +183,8 @@ Using a reverse proxy with Synapse
 It is recommended to put a reverse proxy such as
 `nginx <https://nginx.org/en/docs/http/ngx_http_proxy_module.html>`_,
 `Apache <https://httpd.apache.org/docs/current/mod/mod_proxy_http.html>`_,
-`Caddy <https://caddyserver.com/docs/quick-starts/reverse-proxy>`_,
-`HAProxy <https://www.haproxy.org/>`_ or
-`relayd <https://man.openbsd.org/relayd.8>`_ in front of Synapse. One advantage of
+`Caddy <https://caddyserver.com/docs/quick-starts/reverse-proxy>`_ or
+`HAProxy <https://www.haproxy.org/>`_ in front of Synapse. One advantage of
 doing so is that it means that you can expose the default https port (443) to
 Matrix clients without needing to run Synapse with root privileges.
 
@@ -314,15 +313,6 @@ Testing with SyTest is recommended for verifying that changes related to the
 Client-Server API are functioning correctly. See the `installation instructions
 <https://github.com/matrix-org/sytest#installing>`_ for details.
 
-
-Platform dependencies
-=====================
-
-Synapse uses a number of platform dependencies such as Python and PostgreSQL,
-and aims to follow supported upstream versions. See the
-`<docs/deprecation_policy.md>`_ document for more details.
-
-
 Troubleshooting
 ===============
 
@@ -398,7 +388,7 @@ likely cause. The misbehavior can be worked around by setting
 People can't accept room invitations from me
 --------------------------------------------
 
-The typical failure mode here is that you send an invitation to someone
+The typical failure mode here is that you send an invitation to someone 
 to join a room or direct chat, but when they go to accept it, they get an
 error (typically along the lines of "Invalid signature"). They might see
 something like the following in their logs::

UPGRADE.rst

@@ -85,29 +85,6 @@ for example:
      wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb
      dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
 
-Upgrading to v1.29.0
-====================
-
-Requirement for X-Forwarded-Proto header
-----------------------------------------
-
-When using Synapse with a reverse proxy (in particular, when using the
-`x_forwarded` option on an HTTP listener), Synapse now expects to receive an
-`X-Forwarded-Proto` header on incoming HTTP requests. If it is not set, Synapse
-will log a warning on each received request.
-
-To avoid the warning, administrators using a reverse proxy should ensure that
-the reverse proxy sets `X-Forwarded-Proto` header to `https` or `http` to
-indicate the protocol used by the client.
-
-Synapse also requires the `Host` header to be preserved.
-
-See the `reverse proxy documentation <docs/reverse_proxy.md>`_, where the
-example configurations have been updated to show how to set these headers.
-
-(Users of `Caddy <https://caddyserver.com/>`_ are unaffected, since we believe it
-sets `X-Forwarded-Proto` by default.)
-
 Upgrading to v1.27.0
 ====================
 
@@ -127,13 +104,6 @@ This version changes the URI used for callbacks from OAuth2 and SAML2 identity p
   need to add ``[synapse public baseurl]/_synapse/client/saml2/authn_response`` as a permitted
   "ACS location" (also known as "allowed callback URLs") at the identity provider.
 
-  The "Issuer" in the "AuthnRequest" to the SAML2 identity provider is also updated to
-  ``[synapse public baseurl]/_synapse/client/saml2/metadata.xml``. If your SAML2 identity
-  provider uses this property to validate or otherwise identify Synapse, its configuration
-  will need to be updated to use the new URL. Alternatively you could create a new, separate
-  "EntityDescriptor" in your SAML2 identity provider with the new URLs and leave the URLs in
-  the existing "EntityDescriptor" as they were.
-
 Changes to HTML templates
 -------------------------
 

contrib/purge_api/purge_history.sh

@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/bin/bash
 
 # this script will use the api:
 #    https://github.com/matrix-org/synapse/blob/master/docs/admin_api/purge_history_api.rst

contrib/purge_api/purge_remote_media.sh

@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/bin/bash
 
 DOMAIN=yourserver.tld
 # add this user as admin in your home server:

debian/build_virtualenv

@@ -58,10 +58,10 @@ trap "rm -r $tmpdir" EXIT
 cp -r tests "$tmpdir"
 
 PYTHONPATH="$tmpdir" \
-    "${TARGET_PYTHON}" -m twisted.trial --reporter=text -j2 tests
+    "${TARGET_PYTHON}" -B -m twisted.trial --reporter=text -j2 tests
 
 # build the config file
-"${TARGET_PYTHON}" "${VIRTUALENV_DIR}/bin/generate_config" \
+"${TARGET_PYTHON}" -B "${VIRTUALENV_DIR}/bin/generate_config" \
         --config-dir="/etc/matrix-synapse" \
         --data-dir="/var/lib/matrix-synapse" |
     perl -pe '
@@ -87,7 +87,7 @@ PYTHONPATH="$tmpdir" \
 ' > "${PACKAGE_BUILD_DIR}/etc/matrix-synapse/homeserver.yaml"
 
 # build the log config file
-"${TARGET_PYTHON}" "${VIRTUALENV_DIR}/bin/generate_log_config" \
+"${TARGET_PYTHON}" -B "${VIRTUALENV_DIR}/bin/generate_log_config" \
         --output-file="${PACKAGE_BUILD_DIR}/etc/matrix-synapse/log.yaml"
 
 # add a dependency on the right version of python to substvars.

debian/changelog

@@ -1,31 +1,3 @@
-matrix-synapse-py3 (1.31.0) stable; urgency=medium
-
-  * New synapse release 1.31.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 06 Apr 2021 13:08:29 +0100
-
-matrix-synapse-py3 (1.30.1) stable; urgency=medium
-
-  * New synapse release 1.30.1.
-
- -- Synapse Packaging team <packages@matrix.org>  Fri, 26 Mar 2021 12:01:28 +0000
-
-matrix-synapse-py3 (1.30.0) stable; urgency=medium
-
-  * New synapse release 1.30.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Mon, 22 Mar 2021 13:15:34 +0000
-
-matrix-synapse-py3 (1.29.0) stable; urgency=medium
-
-  [ Jonathan de Jong ]
-  * Remove the python -B flag (don't generate bytecode) in scripts and documentation.
-
-  [ Synapse Packaging team ]
-  * New synapse release 1.29.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Mon, 08 Mar 2021 13:51:50 +0000
-
 matrix-synapse-py3 (1.28.0) stable; urgency=medium
 
   * New synapse release 1.28.0.

debian/synctl.1

@@ -44,7 +44,7 @@ Configuration file may be generated as follows:
 .
 .nf
 
-$ python \-m synapse\.app\.homeserver \-c config\.yaml \-\-generate\-config \-\-server\-name=<server name>
+$ python \-B \-m synapse\.app\.homeserver \-c config\.yaml \-\-generate\-config \-\-server\-name=<server name>
 .
 .fi
 .

debian/synctl.ronn

@@ -41,7 +41,7 @@ process.
 
 Configuration file may be generated as follows:
 
-    $ python -m synapse.app.homeserver -c config.yaml --generate-config --server-name=<server name>
+    $ python -B -m synapse.app.homeserver -c config.yaml --generate-config --server-name=<server name>
 
 ## ENVIRONMENT
 

demo/clean.sh

@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/bin/bash
 
 set -e
 

demo/start.sh

@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/bin/bash
 
 DIR="$( cd "$( dirname "$0" )" && pwd )"
 

demo/stop.sh

@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/bin/bash
 
 DIR="$( cd "$( dirname "$0" )" && pwd )"
 

docker/Dockerfile

@@ -18,11 +18,6 @@ ARG PYTHON_VERSION=3.8
 ###
 FROM docker.io/python:${PYTHON_VERSION}-slim as builder
 
-LABEL org.opencontainers.image.url='https://matrix.org/docs/projects/server/synapse'
-LABEL org.opencontainers.image.documentation='https://github.com/matrix-org/synapse/blob/master/docker/README.md'
-LABEL org.opencontainers.image.source='https://github.com/matrix-org/synapse.git'
-LABEL org.opencontainers.image.licenses='Apache-2.0'
-
 # install the OS build deps
 RUN apt-get update && apt-get install -y \
     build-essential \
@@ -33,32 +28,33 @@ RUN apt-get update && apt-get install -y \
     libwebp-dev \
     libxml++2.6-dev \
     libxslt1-dev \
-    openssl \
     rustc \
     zlib1g-dev \
-    && rm -rf /var/lib/apt/lists/*
+ && rm -rf /var/lib/apt/lists/*
 
-# Copy just what we need to pip install
+# Build dependencies that are not available as wheels, to speed up rebuilds
+RUN pip install --prefix="/install" --no-warn-script-location \
+        cryptography \
+        frozendict \
+        jaeger-client \
+        opentracing \
+        # Match the version constraints of Synapse
+        "prometheus_client>=0.4.0" \
+        psycopg2 \
+        pycparser \
+        pyrsistent \
+        pyyaml \
+        simplejson \
+        threadloop \
+        thrift
+
+# now install synapse and all of the python deps to /install.
+COPY synapse /synapse/synapse/
 COPY scripts /synapse/scripts/
 COPY MANIFEST.in README.rst setup.py synctl /synapse/
-COPY synapse/__init__.py /synapse/synapse/__init__.py
-COPY synapse/python_dependencies.py /synapse/synapse/python_dependencies.py
 
-# To speed up rebuilds, install all of the dependencies before we copy over
-# the whole synapse project so that we this layer in the Docker cache can be
-# used while you develop on the source
-#
-# This is aiming at installing the `install_requires` and `extras_require` from `setup.py`
 RUN pip install --prefix="/install" --no-warn-script-location \
-    /synapse[all]
-
-# Copy over the rest of the project
-COPY synapse /synapse/synapse/
-
-# Install the synapse package itself and all of its children packages.
-#
-# This is aiming at installing only the `packages=find_packages(...)` from `setup.py
-RUN pip install --prefix="/install" --no-deps --no-warn-script-location /synapse
+        /synapse[all]
 
 ###
 ### Stage 1: runtime
@@ -73,10 +69,7 @@ RUN apt-get update && apt-get install -y \
     libpq5 \
     libwebp6 \
     xmlsec1 \
-    libjemalloc2 \
-    libssl-dev \
-    openssl \
-    && rm -rf /var/lib/apt/lists/*
+ && rm -rf /var/lib/apt/lists/*
 
 COPY --from=builder /install /usr/local
 COPY ./docker/start.py /start.py
@@ -89,4 +82,4 @@ EXPOSE 8008/tcp 8009/tcp 8448/tcp
 ENTRYPOINT ["/start.py"]
 
 HEALTHCHECK --interval=1m --timeout=5s \
-    CMD curl -fSs http://localhost:8008/health || exit 1
+  CMD curl -fSs http://localhost:8008/health || exit 1

docker/README.md

@@ -11,6 +11,7 @@ The image also does *not* provide a TURN server.
 By default, the image expects a single volume, located at ``/data``, that will hold:
 
 * configuration files;
+* temporary files during uploads;
 * uploaded media and thumbnails;
 * the SQLite database if you do not configure postgres;
 * the appservices configuration.
@@ -204,8 +205,3 @@ healthcheck:
   timeout: 10s
   retries: 3
 ```
-
-## Using jemalloc
-
-Jemalloc is embedded in the image and will be used instead of the default allocator.
-You can read about jemalloc by reading the Synapse [README](../README.md)

docker/build_debian.sh

@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/bin/bash
 
 # The script to build the Debian package, as ran inside the Docker image.
 

docker/conf/homeserver.yaml

@@ -89,6 +89,7 @@ federation_rc_concurrent: 3
 ## Files ##
 
 media_store_path: "/data/media"
+uploads_path: "/data/uploads"
 max_upload_size: "{{ SYNAPSE_MAX_UPLOAD_SIZE or "50M" }}"
 max_image_pixels: "32M"
 dynamic_thumbnails: false

docker/run_pg_tests.sh

@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/bin/bash
 
 # This script runs the PostgreSQL tests inside a Docker container. It expects
 # the relevant source files to be mounted into /src (done automatically by the

docker/start.py

@@ -3,7 +3,6 @@
 import codecs
 import glob
 import os
-import platform
 import subprocess
 import sys
 
@@ -214,13 +213,6 @@ def main(args, environ):
     if "-m" not in args:
         args = ["-m", synapse_worker] + args
 
-    jemallocpath = "/usr/lib/%s-linux-gnu/libjemalloc.so.2" % (platform.machine(),)
-
-    if os.path.isfile(jemallocpath):
-        environ["LD_PRELOAD"] = jemallocpath
-    else:
-        log("Could not find %s, will not use" % (jemallocpath,))
-
     # if there are no config files passed to synapse, try adding the default file
     if not any(p.startswith("--config-path") or p.startswith("-c") for p in args):
         config_dir = environ.get("SYNAPSE_CONFIG_DIR", "/data")
@@ -256,9 +248,9 @@ running with 'migrate_config'. See the README for more details.
     args = ["python"] + args
     if ownership is not None:
         args = ["gosu", ownership] + args
-        os.execve("/usr/sbin/gosu", args, environ)
+        os.execv("/usr/sbin/gosu", args)
     else:
-        os.execve("/usr/local/bin/python", args, environ)
+        os.execv("/usr/local/bin/python", args)
 
 
 if __name__ == "__main__":

docs/admin_api/media_admin_api.md

@@ -1,7 +1,5 @@
 # Contents
-- [Querying media](#querying-media)
-  * [List all media in a room](#list-all-media-in-a-room)
-  * [List all media uploaded by a user](#list-all-media-uploaded-by-a-user)
+- [List all media in a room](#list-all-media-in-a-room)
 - [Quarantine media](#quarantine-media)
   * [Quarantining media by ID](#quarantining-media-by-id)
   * [Quarantining media in a room](#quarantining-media-in-a-room)
@@ -12,11 +10,7 @@
   * [Delete local media by date or size](#delete-local-media-by-date-or-size)
 - [Purge Remote Media API](#purge-remote-media-api)
 
-# Querying media
-
-These APIs allow extracting media information from the homeserver.
-
-## List all media in a room
+# List all media in a room
 
 This API gets a list of known media in a room.
 However, it only shows media from unencrypted events or rooms.
@@ -42,12 +36,6 @@ The API returns a JSON body like the following:
 }
 ```
 
-## List all media uploaded by a user
-
-Listing all media that has been uploaded by a local user can be achieved through
-the use of the [List media of a user](user_admin_api.rst#list-media-of-a-user)
-Admin API.
-
 # Quarantine media
 
 Quarantining media means that it is marked as inaccessible by users. It applies

docs/admin_api/user_admin_api.rst

@@ -379,12 +379,11 @@ The following fields are returned in the JSON response body:
 - ``total`` - Number of rooms.
 
 
-List media of a user
-====================
+List media of an user
+================================
 Gets a list of all local media that a specific ``user_id`` has created.
-By default, the response is ordered by descending creation date and ascending media ID.
-The newest media is on top. You can change the order with parameters
-``order_by`` and ``dir``.
+The response is ordered by creation date descending and media ID descending.
+The newest media is on top.
 
 The API is::
 
@@ -441,35 +440,6 @@ The following parameters should be set in the URL:
   denoting the offset in the returned results. This should be treated as an opaque value and
   not explicitly set to anything other than the return value of ``next_token`` from a previous call.
   Defaults to ``0``.
-- ``order_by`` - The method by which to sort the returned list of media.
-  If the ordered field has duplicates, the second order is always by ascending ``media_id``,
-  which guarantees a stable ordering. Valid values are:
-
-  - ``media_id`` - Media are ordered alphabetically by ``media_id``.
-  - ``upload_name`` - Media are ordered alphabetically by name the media was uploaded with.
-  - ``created_ts`` - Media are ordered by when the content was uploaded in ms.
-    Smallest to largest. This is the default.
-  - ``last_access_ts`` - Media are ordered by when the content was last accessed in ms.
-    Smallest to largest.
-  - ``media_length`` - Media are ordered by length of the media in bytes.
-    Smallest to largest.
-  - ``media_type`` - Media are ordered alphabetically by MIME-type.
-  - ``quarantined_by`` - Media are ordered alphabetically by the user ID that
-    initiated the quarantine request for this media.
-  - ``safe_from_quarantine`` - Media are ordered by the status if this media is safe
-    from quarantining.
-
-- ``dir`` - Direction of media order. Either ``f`` for forwards or ``b`` for backwards.
-  Setting this value to ``b`` will reverse the above sort order. Defaults to ``f``.
-
-If neither ``order_by`` nor ``dir`` is set, the default order is newest media on top
-(corresponds to ``order_by`` = ``created_ts`` and ``dir`` = ``b``).
-
-Caution. The database only has indexes on the columns ``media_id``,
-``user_id`` and ``created_ts``. This means that if a different sort order is used
-(``upload_name``, ``last_access_ts``, ``media_length``, ``media_type``,
-``quarantined_by`` or ``safe_from_quarantine``), this can cause a large load on the
-database, especially for large environments.
 
 **Response**
 

docs/deprecation_policy.md

@@ -1,33 +0,0 @@
-Deprecation Policy for Platform Dependencies
-============================================
-
-Synapse has a number of platform dependencies, including Python and PostgreSQL.
-This document outlines the policy towards which versions we support, and when we
-drop support for versions in the future.
-
-
-Policy
-------
-
-Synapse follows the upstream support life cycles for Python and PostgreSQL,
-i.e. when a version reaches End of Life Synapse will withdraw support for that
-version in future releases.
-
-Details on the upstream support life cycles for Python and PostgreSQL are
-documented at https://endoflife.date/python and
-https://endoflife.date/postgresql.
-
-
-Context
--------
-
-It is important for system admins to have a clear understanding of the platform
-requirements of Synapse and its deprecation policies so that they can
-effectively plan upgrading their infrastructure ahead of time. This is
-especially important in contexts where upgrading the infrastructure requires
-auditing and approval from a security team, or where otherwise upgrading is a
-long process.
-
-By following the upstream support life cycles Synapse can ensure that its
-dependencies continue to get security patches, while not requiring system admins
-to constantly update their platform dependencies to the latest versions.

docs/openid.md

@@ -226,7 +226,7 @@ Synapse config:
 oidc_providers:
   - idp_id: github
     idp_name: Github
-    idp_brand: "github"  # optional: styling hint for clients
+    idp_brand: "org.matrix.github"  # optional: styling hint for clients
     discover: false
     issuer: "https://github.com/"
     client_id: "your-client-id" # TO BE FILLED
@@ -252,7 +252,7 @@ oidc_providers:
    oidc_providers:
      - idp_id: google
        idp_name: Google
-       idp_brand: "google"  # optional: styling hint for clients
+       idp_brand: "org.matrix.google"  # optional: styling hint for clients
        issuer: "https://accounts.google.com/"
        client_id: "your-client-id" # TO BE FILLED
        client_secret: "your-client-secret" # TO BE FILLED
@@ -299,7 +299,7 @@ Synapse config:
 oidc_providers:
   - idp_id: gitlab
     idp_name: Gitlab
-    idp_brand: "gitlab"  # optional: styling hint for clients
+    idp_brand: "org.matrix.gitlab"  # optional: styling hint for clients
     issuer: "https://gitlab.com/"
     client_id: "your-client-id" # TO BE FILLED
     client_secret: "your-client-secret" # TO BE FILLED
@@ -334,7 +334,7 @@ Synapse config:
 ```yaml
   - idp_id: facebook
     idp_name: Facebook
-    idp_brand: "facebook"  # optional: styling hint for clients
+    idp_brand: "org.matrix.facebook"  # optional: styling hint for clients
     discover: false
     issuer: "https://facebook.com"
     client_id: "your-client-id" # TO BE FILLED
@@ -386,7 +386,7 @@ oidc_providers:
       config:
         subject_claim: "id"
         localpart_template: "{{ user.login }}"
-        display_name_template: "{{ user.full_name }}"
+        display_name_template: "{{ user.full_name }}" 
 ```
 
 ### XWiki
@@ -401,7 +401,8 @@ oidc_providers:
     idp_name: "XWiki"
     issuer: "https://myxwikihost/xwiki/oidc/"
     client_id: "your-client-id" # TO BE FILLED
-    client_auth_method: none
+    # Needed until https://github.com/matrix-org/synapse/issues/9212 is fixed
+    client_secret: "dontcare"
     scopes: ["openid", "profile"]
     user_profile_method: "userinfo_endpoint"
     user_mapping_provider:
@@ -409,40 +410,3 @@ oidc_providers:
         localpart_template: "{{ user.preferred_username }}"
         display_name_template: "{{ user.name }}"
 ```
-
-## Apple
-
-Configuring "Sign in with Apple" (SiWA) requires an Apple Developer account.
-
-You will need to create a new "Services ID" for SiWA, and create and download a
-private key with "SiWA" enabled.
-
-As well as the private key file, you will need:
- * Client ID: the "identifier" you gave the "Services ID"
- * Team ID: a 10-character ID associated with your developer account.
- * Key ID: the 10-character identifier for the key.
-
-https://help.apple.com/developer-account/?lang=en#/dev77c875b7e has more
-documentation on setting up SiWA.
-
-The synapse config will look like this:
-
-```yaml
-  - idp_id: apple
-    idp_name: Apple
-    issuer: "https://appleid.apple.com"
-    client_id: "your-client-id" # Set to the "identifier" for your "ServicesID"
-    client_auth_method: "client_secret_post"
-    client_secret_jwt_key:
-      key_file: "/path/to/AuthKey_KEYIDCODE.p8"  # point to your key file
-      jwt_header:
-        alg: ES256
-        kid: "KEYIDCODE"   # Set to the 10-char Key ID
-      jwt_payload:
-        iss: TEAMIDCODE    # Set to the 10-char Team ID
-    scopes: ["name", "email", "openid"]
-    authorization_endpoint: https://appleid.apple.com/auth/authorize?response_mode=form_post
-    user_mapping_provider:
-      config:
-        email_template: "{{ user.email }}"
-```

docs/reverse_proxy.md

@@ -3,31 +3,30 @@
 It is recommended to put a reverse proxy such as
 [nginx](https://nginx.org/en/docs/http/ngx_http_proxy_module.html),
 [Apache](https://httpd.apache.org/docs/current/mod/mod_proxy_http.html),
-[Caddy](https://caddyserver.com/docs/quick-starts/reverse-proxy),
-[HAProxy](https://www.haproxy.org/) or
-[relayd](https://man.openbsd.org/relayd.8) in front of Synapse. One advantage
+[Caddy](https://caddyserver.com/docs/quick-starts/reverse-proxy) or
+[HAProxy](https://www.haproxy.org/) in front of Synapse. One advantage
 of doing so is that it means that you can expose the default https port
 (443) to Matrix clients without needing to run Synapse with root
 privileges.
 
-You should configure your reverse proxy to forward requests to `/_matrix` or
-`/_synapse/client` to Synapse, and have it set the `X-Forwarded-For` and
-`X-Forwarded-Proto` request headers.
-
-You should remember that Matrix clients and other Matrix servers do not
-necessarily need to connect to your server via the same server name or
-port. Indeed, clients will use port 443 by default, whereas servers default to
-port 8448. Where these are different, we refer to the 'client port' and the
-'federation port'. See [the Matrix
-specification](https://matrix.org/docs/spec/server_server/latest#resolving-server-names)
-for more details of the algorithm used for federation connections, and
-[delegate.md](<delegate.md>) for instructions on setting up delegation.
-
 **NOTE**: Your reverse proxy must not `canonicalise` or `normalise`
 the requested URI in any way (for example, by decoding `%xx` escapes).
 Beware that Apache *will* canonicalise URIs unless you specify
 `nocanon`.
 
+When setting up a reverse proxy, remember that Matrix clients and other
+Matrix servers do not necessarily need to connect to your server via the
+same server name or port. Indeed, clients will use port 443 by default,
+whereas servers default to port 8448. Where these are different, we
+refer to the 'client port' and the 'federation port'. See [the Matrix
+specification](https://matrix.org/docs/spec/server_server/latest#resolving-server-names)
+for more details of the algorithm used for federation connections, and
+[delegate.md](<delegate.md>) for instructions on setting up delegation.
+
+Endpoints that are part of the standardised Matrix specification are
+located under `/_matrix`, whereas endpoints specific to Synapse are
+located under `/_synapse/client`.
+
 Let's assume that we expect clients to connect to our server at
 `https://matrix.example.com`, and other servers to connect at
 `https://example.com:8448`.  The following sections detail the configuration of
@@ -53,9 +52,6 @@ server {
     location ~* ^(\/_matrix|\/_synapse\/client) {
         proxy_pass http://localhost:8008;
         proxy_set_header X-Forwarded-For $remote_addr;
-        proxy_set_header X-Forwarded-Proto $scheme;
-        proxy_set_header Host $host;
-
         # Nginx by default only allows file uploads up to 1M in size
         # Increase client_max_body_size to match max_upload_size defined in homeserver.yaml
         client_max_body_size 50M;
@@ -104,11 +100,9 @@ example.com:8448 {
 ```
 <VirtualHost *:443>
     SSLEngine on
-    ServerName matrix.example.com
+    ServerName matrix.example.com;
 
-    RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
     AllowEncodedSlashes NoDecode
-    ProxyPreserveHost on
     ProxyPass /_matrix http://127.0.0.1:8008/_matrix nocanon
     ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix
     ProxyPass /_synapse/client http://127.0.0.1:8008/_synapse/client nocanon
@@ -117,9 +111,8 @@ example.com:8448 {
 
 <VirtualHost *:8448>
     SSLEngine on
-    ServerName example.com
+    ServerName example.com;
 
-    RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
     AllowEncodedSlashes NoDecode
     ProxyPass /_matrix http://127.0.0.1:8008/_matrix nocanon
     ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix
@@ -136,16 +129,11 @@ example.com:8448 {
 </IfModule>
 ```
 
-**NOTE 3**: Missing `ProxyPreserveHost on` can lead to a redirect loop.
-
 ### HAProxy
 
 ```
 frontend https
   bind :::443 v4v6 ssl crt /etc/ssl/haproxy/ strict-sni alpn h2,http/1.1
-  http-request set-header X-Forwarded-Proto https if { ssl_fc }
-  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
-  http-request set-header X-Forwarded-For %[src]
 
   # Matrix client traffic
   acl matrix-host hdr(host) -i matrix.example.com
@@ -156,62 +144,12 @@ frontend https
 
 frontend matrix-federation
   bind :::8448 v4v6 ssl crt /etc/ssl/haproxy/synapse.pem alpn h2,http/1.1
-  http-request set-header X-Forwarded-Proto https if { ssl_fc }
-  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
-  http-request set-header X-Forwarded-For %[src]
-
   default_backend matrix
 
 backend matrix
   server matrix 127.0.0.1:8008
 ```
 
-### Relayd
-
-```
-table <webserver>    { 127.0.0.1 }
-table <matrixserver> { 127.0.0.1 }
-
-http protocol "https" {
-    tls { no tlsv1.0, ciphers "HIGH" }
-    tls keypair "example.com"
-    match header set "X-Forwarded-For"   value "$REMOTE_ADDR"
-    match header set "X-Forwarded-Proto" value "https"
-
-    # set CORS header for .well-known/matrix/server, .well-known/matrix/client
-    # httpd does not support setting headers, so do it here
-    match request path "/.well-known/matrix/*" tag "matrix-cors"
-    match response tagged "matrix-cors" header set "Access-Control-Allow-Origin" value "*"
-
-    pass quick path "/_matrix/*"         forward to <matrixserver>
-    pass quick path "/_synapse/client/*" forward to <matrixserver>
-
-    # pass on non-matrix traffic to webserver
-    pass                                 forward to <webserver>
-}
-
-relay "https_traffic" {
-    listen on egress port 443 tls
-    protocol "https"
-    forward to <matrixserver> port 8008 check tcp
-    forward to <webserver>    port 8080 check tcp
-}
-
-http protocol "matrix" {
-    tls { no tlsv1.0, ciphers "HIGH" }
-    tls keypair "example.com"
-    block
-    pass quick path "/_matrix/*"         forward to <matrixserver>
-    pass quick path "/_synapse/client/*" forward to <matrixserver>
-}
-
-relay "matrix_federation" {
-    listen on egress port 8448 tls
-    protocol "matrix"
-    forward to <matrixserver> port 8008 check tcp
-}
-```
-
 ## Homeserver Configuration
 
 You will also want to set `bind_addresses: ['127.0.0.1']` and

docs/sample_config.yaml

@@ -89,7 +89,8 @@ pid_file: DATADIR/homeserver.pid
 # Whether to require authentication to retrieve profile data (avatars,
 # display names) of other users through the client API. Defaults to
 # 'false'. Note that profile data is also available via the federation
-# API, unless allow_profile_lookup_over_federation is set to false.
+# API, so this setting is of limited value if federation is enabled on
+# the server.
 #
 #require_auth_for_profile_requests: true
 
@@ -100,14 +101,6 @@ pid_file: DATADIR/homeserver.pid
 #
 #limit_profile_requests_to_users_who_share_rooms: true
 
-# Uncomment to prevent a user's profile data from being retrieved and
-# displayed in a room until they have joined it. By default, a user's
-# profile data is included in an invite event, regardless of the values
-# of the above two settings, and whether or not the users share a server.
-# Defaults to 'true'.
-#
-#include_profile_data_on_invite: false
-
 # If set to 'true', removes the need for authentication to access the server's
 # public rooms directory through the client API, meaning that anyone can
 # query the room directory. Defaults to 'false'.
@@ -706,12 +699,6 @@ acme:
 #  - matrix.org
 #  - example.com
 
-# Uncomment to disable profile lookup over federation. By default, the
-# Federation API allows other homeservers to obtain profile data of any user
-# on this homeserver. Defaults to 'true'.
-#
-#allow_profile_lookup_over_federation: false
-
 
 ## Caching ##
 
@@ -869,10 +856,10 @@ log_config: "CONFDIR/SERVERNAME.log.config"
 #rc_joins:
 #  local:
 #    per_second: 0.1
-#    burst_count: 10
+#    burst_count: 3
 #  remote:
 #    per_second: 0.01
-#    burst_count: 10
+#    burst_count: 3
 #
 #rc_3pid_validation:
 #  per_second: 0.003
@@ -1758,9 +1745,6 @@ saml2_config:
 #       Note that, if this is changed, users authenticating via that provider
 #       will no longer be recognised as the same user!
 #
-#       (Use "oidc" here if you are migrating from an old "oidc_config"
-#       configuration.)
-#
 #   idp_name: A user-facing name for this identity provider, which is used to
 #       offer the user a choice of login mechanisms.
 #
@@ -1782,26 +1766,7 @@ saml2_config:
 #
 #   client_id: Required. oauth2 client id to use.
 #
-#   client_secret: oauth2 client secret to use. May be omitted if
-#        client_secret_jwt_key is given, or if client_auth_method is 'none'.
-#
-#   client_secret_jwt_key: Alternative to client_secret: details of a key used
-#      to create a JSON Web Token to be used as an OAuth2 client secret. If
-#      given, must be a dictionary with the following properties:
-#
-#          key: a pem-encoded signing key. Must be a suitable key for the
-#              algorithm specified. Required unless 'key_file' is given.
-#
-#          key_file: the path to file containing a pem-encoded signing key file.
-#              Required unless 'key' is given.
-#
-#          jwt_header: a dictionary giving properties to include in the JWT
-#              header. Must include the key 'alg', giving the algorithm used to
-#              sign the JWT, such as "ES256", using the JWA identifiers in
-#              RFC7518.
-#
-#          jwt_payload: an optional dictionary giving properties to include in
-#              the JWT payload. Normally this should include an 'iss' key.
+#   client_secret: Required. oauth2 client secret to use.
 #
 #   client_auth_method: auth method to use when exchanging the token. Valid
 #       values are 'client_secret_basic' (default), 'client_secret_post' and
@@ -1876,24 +1841,6 @@ saml2_config:
 #           which is set to the claims returned by the UserInfo Endpoint and/or
 #           in the ID Token.
 #
-#   It is possible to configure Synapse to only allow logins if certain attributes
-#   match particular values in the OIDC userinfo. The requirements can be listed under
-#   `attribute_requirements` as shown below. All of the listed attributes must
-#   match for the login to be permitted. Additional attributes can be added to
-#   userinfo by expanding the `scopes` section of the OIDC config to retrieve
-#   additional information from the OIDC provider.
-#
-#   If the OIDC claim is a list, then the attribute must match any value in the list.
-#   Otherwise, it must exactly match the value of the claim. Using the example
-#   below, the `family_name` claim MUST be "Stephensson", but the `groups`
-#   claim MUST contain "admin".
-#
-#   attribute_requirements:
-#     - attribute: family_name
-#       value: "Stephensson"
-#     - attribute: groups
-#       value: "admin"
-#
 # See https://github.com/matrix-org/synapse/blob/master/docs/openid.md
 # for information on how to configure these options.
 #
@@ -1926,9 +1873,34 @@ oidc_providers:
   #      localpart_template: "{{ user.login }}"
   #      display_name_template: "{{ user.name }}"
   #      email_template: "{{ user.email }}"
-  #  attribute_requirements:
-  #    - attribute: userGroup
-  #      value: "synapseUsers"
+
+  # For use with Keycloak
+  #
+  #- idp_id: keycloak
+  #  idp_name: Keycloak
+  #  issuer: "https://127.0.0.1:8443/auth/realms/my_realm_name"
+  #  client_id: "synapse"
+  #  client_secret: "copy secret generated in Keycloak UI"
+  #  scopes: ["openid", "profile"]
+
+  # For use with Github
+  #
+  #- idp_id: github
+  #  idp_name: Github
+  #  idp_brand: org.matrix.github
+  #  discover: false
+  #  issuer: "https://github.com/"
+  #  client_id: "your-client-id" # TO BE FILLED
+  #  client_secret: "your-client-secret" # TO BE FILLED
+  #  authorization_endpoint: "https://github.com/login/oauth/authorize"
+  #  token_endpoint: "https://github.com/login/oauth/access_token"
+  #  userinfo_endpoint: "https://api.github.com/user"
+  #  scopes: ["read:user"]
+  #  user_mapping_provider:
+  #    config:
+  #      subject_claim: "id"
+  #      localpart_template: "{{ user.login }}"
+  #      display_name_template: "{{ user.name }}"
 
 
 # Enable Central Authentication Service (CAS) for registration and login.
@@ -2558,35 +2530,19 @@ spam_checker:
 
 # User Directory configuration
 #
-user_directory:
-    # Defines whether users can search the user directory. If false then
-    # empty responses are returned to all queries. Defaults to true.
-    #
-    # Uncomment to disable the user directory.
-    #
-    #enabled: false
-
-    # Defines whether to search all users visible to your HS when searching
-    # the user directory, rather than limiting to users visible in public
-    # rooms. Defaults to false.
-    #
-    # If you set it true, you'll have to rebuild the user_directory search
-    # indexes, see:
-    # https://github.com/matrix-org/synapse/blob/master/docs/user_directory.md
-    #
-    # Uncomment to return search results containing all known users, even if that
-    # user does not share a room with the requester.
-    #
-    #search_all_users: true
-
-    # Defines whether to prefer local users in search query results.
-    # If True, local users are more likely to appear above remote users
-    # when searching the user directory. Defaults to false.
-    #
-    # Uncomment to prefer local over remote users in user directory search
-    # results.
-    #
-    #prefer_local_users: true
+# 'enabled' defines whether users can search the user directory. If
+# false then empty responses are returned to all queries. Defaults to
+# true.
+#
+# 'search_all_users' defines whether to search all users visible to your HS
+# when searching the user directory, rather than limiting to users visible
+# in public rooms.  Defaults to false.  If you set it True, you'll have to
+# rebuild the user_directory search indexes, see
+# https://github.com/matrix-org/synapse/blob/master/docs/user_directory.md
+#
+#user_directory:
+#  enabled: true
+#  search_all_users: false
 
 
 # User Consent configuration
@@ -2641,20 +2597,19 @@ user_directory:
 
 
 
-# Settings for local room and user statistics collection. See
-# docs/room_and_user_statistics.md.
+# Local statistics collection. Used in populating the room directory.
 #
-stats:
-  # Uncomment the following to disable room and user statistics. Note that doing
-  # so may cause certain features (such as the room directory) not to work
-  # correctly.
-  #
-  #enabled: false
-
-  # The size of each timeslice in the room_stats_historical and
-  # user_stats_historical tables, as a time period. Defaults to "1d".
-  #
-  #bucket_size: 1h
+# 'bucket_size' controls how large each statistics timeslice is. It can
+# be defined in a human readable short form -- e.g. "1d", "1y".
+#
+# 'retention' controls how long historical statistics will be kept for.
+# It can be defined in a human readable short form -- e.g. "1d", "1y".
+#
+#
+#stats:
+#   enabled: true
+#   bucket_size: 1d
+#   retention: 1y
 
 
 # Server Notices room configuration

docs/spam_checker.md

@@ -14,7 +14,6 @@ The Python class is instantiated with two objects:
 * An instance of `synapse.module_api.ModuleApi`.
 
 It then implements methods which return a boolean to alter behavior in Synapse.
-All the methods must be defined.
 
 There's a generic method for checking every event (`check_event_for_spam`), as
 well as some specific methods:
@@ -25,18 +24,13 @@ well as some specific methods:
 * `user_may_publish_room`
 * `check_username_for_spam`
 * `check_registration_for_spam`
-* `check_media_file_for_spam`
 
-The details of each of these methods (as well as their inputs and outputs)
+The details of the each of these methods (as well as their inputs and outputs)
 are documented in the `synapse.events.spamcheck.SpamChecker` class.
 
 The `ModuleApi` class provides a way for the custom spam checker class to
 call back into the homeserver internals.
 
-Additionally, a `parse_config` method is mandatory and receives the plugin config
-dictionary. After parsing, It must return an object which will be
-passed to `__init__` later.
-
 ### Example
 
 ```python
@@ -47,10 +41,6 @@ class ExampleSpamChecker:
         self.config = config
         self.api = api
 
-    @staticmethod
-    def parse_config(config):
-        return config
-        
     async def check_event_for_spam(self, foo):
         return False  # allow all events
 
@@ -69,13 +59,7 @@ class ExampleSpamChecker:
     async def check_username_for_spam(self, user_profile):
         return False  # allow all usernames
 
-    async def check_registration_for_spam(
-        self,
-        email_threepid,
-        username,
-        request_info,
-        auth_provider_id,
-    ):
+    async def check_registration_for_spam(self, email_threepid, username, request_info):
         return RegistrationBehaviour.ALLOW  # allow all registrations
 
     async def check_media_file_for_spam(self, file_wrapper, file_info):

docs/systemd-with-workers/system/matrix-synapse-worker@.service

@@ -4,7 +4,6 @@ AssertPathExists=/etc/matrix-synapse/workers/%i.yaml
 
 # This service should be restarted when the synapse target is restarted.
 PartOf=matrix-synapse.target
-ReloadPropagatedFrom=matrix-synapse.target
 
 # if this is started at the same time as the main, let the main process start
 # first, to initialise the database schema.

docs/systemd-with-workers/system/matrix-synapse.service

@@ -3,7 +3,6 @@ Description=Synapse master
 
 # This service should be restarted when the synapse target is restarted.
 PartOf=matrix-synapse.target
-ReloadPropagatedFrom=matrix-synapse.target
 
 [Service]
 Type=notify

docs/tcp_replication.md

@@ -220,6 +220,10 @@ Asks the server for the current position of all streams.
 
    Acknowledge receipt of some federation data
 
+#### REMOVE_PUSHER (C)
+
+   Inform the server a pusher should be removed
+
 ### REMOTE_SERVER_UP (S, C)
 
    Inform other processes that a remote server may have come back online.

docs/workers.md

@@ -232,6 +232,7 @@ expressions:
     # Registration/login requests
     ^/_matrix/client/(api/v1|r0|unstable)/login$
     ^/_matrix/client/(r0|unstable)/register$
+    ^/_matrix/client/(r0|unstable)/auth/.*/fallback/web$
 
     # Event sending requests
     ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/redact
@@ -275,7 +276,7 @@ using):
 
 Ensure that all SSO logins go to a single process.
 For multiple workers not handling the SSO endpoints properly, see
-[#7530](https://github.com/matrix-org/synapse/issues/7530) and
+[#7530](https://github.com/matrix-org/synapse/issues/7530) and 
 [#9427](https://github.com/matrix-org/synapse/issues/9427).
 
 Note that a HTTP listener with `client` and `federation` resources must be

mypy.ini

@@ -1,13 +1,12 @@
 [mypy]
 namespace_packages = True
 plugins = mypy_zope:plugin, scripts-dev/mypy_synapse_plugin.py
-follow_imports = normal
+follow_imports = silent
 check_untyped_defs = True
 show_error_codes = True
 show_traceback = True
 mypy_path = stubs
 warn_unreachable = True
-local_partial_types = True
 
 # To find all folders that pass mypy you run:
 #
@@ -21,9 +20,8 @@ files =
   synapse/crypto,
   synapse/event_auth.py,
   synapse/events/builder.py,
-  synapse/events/spamcheck.py,
-  synapse/events/third_party_rules.py,
   synapse/events/validator.py,
+  synapse/events/spamcheck.py,
   synapse/federation,
   synapse/groups,
   synapse/handlers,
@@ -40,7 +38,6 @@ files =
   synapse/push,
   synapse/replication,
   synapse/rest,
-  synapse/secrets.py,
   synapse/server.py,
   synapse/server_notices,
   synapse/spam_checker_api,
@@ -72,9 +69,7 @@ files =
   synapse/util/async_helpers.py,
   synapse/util/caches,
   synapse/util/metrics.py,
-  synapse/util/macaroons.py,
   synapse/util/stringutils.py,
-  synapse/visibility.py,
   tests/replication,
   tests/test_utils,
   tests/handlers/test_password_providers.py,
@@ -121,6 +116,9 @@ ignore_missing_imports = True
 [mypy-saml2.*]
 ignore_missing_imports = True
 
+[mypy-unpaddedbase64]
+ignore_missing_imports = True
+
 [mypy-canonicaljson]
 ignore_missing_imports = True
 

scripts-dev/check-newsfragment

@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/bin/bash
 #
 # A script which checks that an appropriate news file has been added on this
 # branch.

scripts-dev/config-lint.sh

@@ -1,15 +1,10 @@
-#!/usr/bin/env bash
+#!/bin/bash
 # Find linting errors in Synapse's default config file.
 # Exits with 0 if there are no problems, or another code otherwise.
 
-# cd to the root of the repository
-cd `dirname $0`/..
-
-# Restore backup of sample config upon script exit
-trap "mv docs/sample_config.yaml.bak docs/sample_config.yaml" EXIT
-
 # Fix non-lowercase true/false values
 sed -i.bak -E "s/: +True/: true/g; s/: +False/: false/g;" docs/sample_config.yaml
+rm docs/sample_config.yaml.bak
 
 # Check if anything changed
-diff docs/sample_config.yaml docs/sample_config.yaml.bak
+git diff --exit-code docs/sample_config.yaml

scripts-dev/federation_client.py

@@ -22,8 +22,8 @@ import sys
 from typing import Any, Optional
 from urllib import parse as urlparse
 
+import nacl.signing
 import requests
-import signedjson.key
 import signedjson.types
 import srvlookup
 import yaml
@@ -44,6 +44,18 @@ def encode_base64(input_bytes):
     return output_string
 
 
+def decode_base64(input_string):
+    """Decode a base64 string to bytes inferring padding from the length of the
+    string."""
+
+    input_bytes = input_string.encode("ascii")
+    input_len = len(input_bytes)
+    padding = b"=" * (3 - ((input_len + 3) % 4))
+    output_len = 3 * ((input_len + 2) // 4) + (input_len + 2) % 4 - 2
+    output_bytes = base64.b64decode(input_bytes + padding)
+    return output_bytes[:output_len]
+
+
 def encode_canonical_json(value):
     return json.dumps(
         value,
@@ -76,6 +88,42 @@ def sign_json(
     return json_object
 
 
+NACL_ED25519 = "ed25519"
+
+
+def decode_signing_key_base64(algorithm, version, key_base64):
+    """Decode a base64 encoded signing key
+    Args:
+        algorithm (str): The algorithm the key is for (currently "ed25519").
+        version (str): Identifies this key out of the keys for this entity.
+        key_base64 (str): Base64 encoded bytes of the key.
+    Returns:
+        A SigningKey object.
+    """
+    if algorithm == NACL_ED25519:
+        key_bytes = decode_base64(key_base64)
+        key = nacl.signing.SigningKey(key_bytes)
+        key.version = version
+        key.alg = NACL_ED25519
+        return key
+    else:
+        raise ValueError("Unsupported algorithm %s" % (algorithm,))
+
+
+def read_signing_keys(stream):
+    """Reads a list of keys from a stream
+    Args:
+        stream : A stream to iterate for keys.
+    Returns:
+        list of SigningKey objects.
+    """
+    keys = []
+    for line in stream:
+        algorithm, version, key_base64 = line.split()
+        keys.append(decode_signing_key_base64(algorithm, version, key_base64))
+    return keys
+
+
 def request(
     method: Optional[str],
     origin_name: str,
@@ -175,28 +223,23 @@ def main():
     parser.add_argument("--body", help="Data to send as the body of the HTTP request")
 
     parser.add_argument(
-        "path", help="request path, including the '/_matrix/federation/...' prefix."
+        "path", help="request path. We will add '/_matrix/federation/v1/' to this."
     )
 
     args = parser.parse_args()
 
-    args.signing_key = None
-    if args.signing_key_path:
-        with open(args.signing_key_path) as f:
-            args.signing_key = f.readline()
-
-    if not args.server_name or not args.signing_key:
+    if not args.server_name or not args.signing_key_path:
         read_args_from_config(args)
 
-    algorithm, version, key_base64 = args.signing_key.split()
-    key = signedjson.key.decode_signing_key_base64(algorithm, version, key_base64)
+    with open(args.signing_key_path) as f:
+        key = read_signing_keys(f)[0]
 
     result = request(
         args.method,
         args.server_name,
         key,
         args.destination,
-        args.path,
+        "/_matrix/federation/v1/" + args.path,
         content=args.body,
     )
 
@@ -212,16 +255,10 @@ def main():
 def read_args_from_config(args):
     with open(args.config, "r") as fh:
         config = yaml.safe_load(fh)
-
         if not args.server_name:
             args.server_name = config["server_name"]
-
-        if not args.signing_key:
-            if "signing_key" in config:
-                args.signing_key = config["signing_key"]
-            else:
-                with open(config["signing_key_path"]) as f:
-                    args.signing_key = f.readline()
+        if not args.signing_key_path:
+            args.signing_key_path = config["signing_key_path"]
 
 
 class MatrixConnectionAdapter(HTTPAdapter):

scripts-dev/generate_sample_config

@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/bin/bash
 #
 # Update/check the docs/sample_config.yaml
 

scripts-dev/lint.sh

@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/bin/bash
 #
 # Runs linting scripts over the local Synapse checkout
 # isort - sorts import statements

scripts-dev/make_full_schema.sh

@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/bin/bash
 #
 # This script generates SQL files for creating a brand new Synapse DB with the latest
 # schema, on both SQLite3 and Postgres.

scripts-dev/next_github_number.sh

@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/bin/bash
 
 set -e
 
@@ -6,4 +6,4 @@ set -e
 # next PR number.
 CURRENT_NUMBER=`curl -s "https://api.github.com/repos/matrix-org/synapse/issues?state=all&per_page=1" | jq -r ".[0].number"`
 CURRENT_NUMBER=$((CURRENT_NUMBER+1))
-echo $CURRENT_NUMBER
+echo $CURRENT_NUMBER

scripts/move_remote_media_to_new_store.py

@@ -51,7 +51,7 @@ def main(src_repo, dest_repo):
         parts = line.split("|")
         if len(parts) != 2:
             print("Unable to parse input line %s" % line, file=sys.stderr)
-            sys.exit(1)
+            exit(1)
 
         move_media(parts[0], parts[1], src_paths, dest_paths)
 

scripts/synapse_port_db

@@ -22,7 +22,7 @@ import logging
 import sys
 import time
 import traceback
-from typing import Dict, Iterable, Optional, Set
+from typing import Dict, Optional, Set
 
 import yaml
 
@@ -47,7 +47,6 @@ from synapse.storage.databases.main.events_bg_updates import (
 from synapse.storage.databases.main.media_repository import (
     MediaRepositoryBackgroundUpdateStore,
 )
-from synapse.storage.databases.main.pusher import PusherWorkerStore
 from synapse.storage.databases.main.registration import (
     RegistrationBackgroundUpdateStore,
     find_max_generated_user_id_localpart,
@@ -178,7 +177,6 @@ class Store(
     UserDirectoryBackgroundUpdateStore,
     EndToEndKeyBackgroundStore,
     StatsStore,
-    PusherWorkerStore,
 ):
     def execute(self, f, *args, **kwargs):
         return self.db_pool.runInteraction(f.__name__, f, *args, **kwargs)
@@ -631,13 +629,7 @@ class Porter(object):
             await self._setup_state_group_id_seq()
             await self._setup_user_id_seq()
             await self._setup_events_stream_seqs()
-            await self._setup_sequence(
-                "device_inbox_sequence", ("device_inbox", "device_federation_outbox")
-            )
-            await self._setup_sequence(
-                "account_data_sequence", ("room_account_data", "room_tags_revisions", "account_data"))
-            await self._setup_sequence("receipts_sequence", ("receipts_linearized", ))
-            await self._setup_auth_chain_sequence()
+            await self._setup_device_inbox_seq()
 
             # Step 3. Get tables.
             self.progress.set_state("Fetching tables")
@@ -862,7 +854,7 @@ class Porter(object):
 
         return done, remaining + done
 
-    async def _setup_state_group_id_seq(self) -> None:
+    async def _setup_state_group_id_seq(self):
         curr_id = await self.sqlite_store.db_pool.simple_select_one_onecol(
             table="state_groups", keyvalues={}, retcol="MAX(id)", allow_none=True
         )
@@ -876,7 +868,7 @@ class Porter(object):
 
         await self.postgres_store.db_pool.runInteraction("setup_state_group_id_seq", r)
 
-    async def _setup_user_id_seq(self) -> None:
+    async def _setup_user_id_seq(self):
         curr_id = await self.sqlite_store.db_pool.runInteraction(
             "setup_user_id_seq", find_max_generated_user_id_localpart
         )
@@ -885,9 +877,9 @@ class Porter(object):
             next_id = curr_id + 1
             txn.execute("ALTER SEQUENCE user_id_seq RESTART WITH %s", (next_id,))
 
-        await self.postgres_store.db_pool.runInteraction("setup_user_id_seq", r)
+        return self.postgres_store.db_pool.runInteraction("setup_user_id_seq", r)
 
-    async def _setup_events_stream_seqs(self) -> None:
+    async def _setup_events_stream_seqs(self):
         """Set the event stream sequences to the correct values.
         """
 
@@ -916,46 +908,35 @@ class Porter(object):
                 (curr_backward_id + 1,),
             )
 
-        await self.postgres_store.db_pool.runInteraction(
+        return await self.postgres_store.db_pool.runInteraction(
             "_setup_events_stream_seqs", _setup_events_stream_seqs_set_pos,
         )
 
-    async def _setup_sequence(self, sequence_name: str, stream_id_tables: Iterable[str]) -> None:
-        """Set a sequence to the correct value.
+    async def _setup_device_inbox_seq(self):
+        """Set the device inbox sequence to the correct value.
         """
-        current_stream_ids = []
-        for stream_id_table in stream_id_tables:
-            max_stream_id = await self.sqlite_store.db_pool.simple_select_one_onecol(
-                table=stream_id_table,
-                keyvalues={},
-                retcol="COALESCE(MAX(stream_id), 1)",
-                allow_none=True,
-            )
-            current_stream_ids.append(max_stream_id)
-
-        next_id = max(current_stream_ids) + 1
-
-        def r(txn):
-            sql = "ALTER SEQUENCE %s RESTART WITH" % (sequence_name, )
-            txn.execute(sql + " %s", (next_id, ))
-
-        await self.postgres_store.db_pool.runInteraction("_setup_%s" % (sequence_name,), r)
-
-    async def _setup_auth_chain_sequence(self) -> None:
-        curr_chain_id = await self.sqlite_store.db_pool.simple_select_one_onecol(
-            table="event_auth_chains", keyvalues={}, retcol="MAX(chain_id)", allow_none=True
+        curr_local_id = await self.sqlite_store.db_pool.simple_select_one_onecol(
+            table="device_inbox",
+            keyvalues={},
+            retcol="COALESCE(MAX(stream_id), 1)",
+            allow_none=True,
         )
 
+        curr_federation_id = await self.sqlite_store.db_pool.simple_select_one_onecol(
+            table="device_federation_outbox",
+            keyvalues={},
+            retcol="COALESCE(MAX(stream_id), 1)",
+            allow_none=True,
+        )
+
+        next_id = max(curr_local_id, curr_federation_id) + 1
+
         def r(txn):
             txn.execute(
-                "ALTER SEQUENCE event_auth_chain_id RESTART WITH %s",
-                (curr_chain_id,),
+                "ALTER SEQUENCE device_inbox_sequence RESTART WITH %s", (next_id,)
             )
 
-        await self.postgres_store.db_pool.runInteraction(
-            "_setup_event_auth_chain_id", r,
-        )
-
+        return self.postgres_store.db_pool.runInteraction("_setup_device_inbox_seq", r)
 
 
 ##############################################

setup.cfg

@@ -3,7 +3,6 @@ test_suite = tests
 
 [check-manifest]
 ignore =
-    .git-blame-ignore-revs
     contrib
     contrib/*
     docs/*
@@ -18,8 +17,7 @@ ignore =
 #  E203: whitespace before ':' (which is contrary to pep8?)
 #  E731: do not assign a lambda expression, use a def
 #  E501: Line too long (black enforces this for us)
-#  B00*: Subsection of the bugbear suite (TODO: add in remaining fixes)
-ignore=W503,W504,E203,E731,E501,B006,B007,B008
+ignore=W503,W504,E203,E731,E501
 
 [isort]
 line_length = 88

setup.py

@@ -99,11 +99,10 @@ CONDITIONAL_REQUIREMENTS["lint"] = [
     "isort==5.7.0",
     "black==20.8b1",
     "flake8-comprehensions",
-    "flake8-bugbear==21.3.2",
     "flake8",
 ]
 
-CONDITIONAL_REQUIREMENTS["mypy"] = ["mypy==0.812", "mypy-zope==0.2.13"]
+CONDITIONAL_REQUIREMENTS["mypy"] = ["mypy==0.790", "mypy-zope==0.2.8"]
 
 # Dependencies which are exclusively required by unit test code. This is
 # NOT a list of all modules that are necessary to run the unit tests.

stubs/txredisapi.pyi

@@ -17,9 +17,7 @@
 """
 from typing import Any, List, Optional, Type, Union
 
-from twisted.internet import protocol
-
-class RedisProtocol(protocol.Protocol):
+class RedisProtocol:
     def publish(self, channel: str, message: bytes): ...
     async def ping(self) -> None: ...
     async def set(
@@ -54,7 +52,7 @@ def lazyConnection(
 
 class ConnectionHandler: ...
 
-class RedisFactory(protocol.ReconnectingClientFactory):
+class RedisFactory:
     continueTrying: bool
     handler: RedisProtocol
     pool: List[RedisProtocol]

synapse/__init__.py

@@ -48,7 +48,7 @@ try:
 except ImportError:
     pass
 
-__version__ = "1.31.0"
+__version__ = "1.28.0"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when

synapse/api/auth.py

@@ -39,7 +39,6 @@ from synapse.logging import opentracing as opentracing
 from synapse.storage.databases.main.registration import TokenLookupResult
 from synapse.types import StateMap, UserID
 from synapse.util.caches.lrucache import LruCache
-from synapse.util.macaroons import get_value_from_macaroon, satisfy_expiry
 from synapse.util.metrics import Measure
 
 logger = logging.getLogger(__name__)
@@ -164,7 +163,7 @@ class Auth:
 
     async def get_user_by_req(
         self,
-        request: SynapseRequest,
+        request: Request,
         allow_guest: bool = False,
         rights: str = "access",
         allow_expired: bool = False,
@@ -409,7 +408,7 @@ class Auth:
             raise _InvalidMacaroonException()
 
         try:
-            user_id = get_value_from_macaroon(macaroon, "user_id")
+            user_id = self.get_user_id_from_macaroon(macaroon)
 
             guest = False
             for caveat in macaroon.caveats:
@@ -417,12 +416,7 @@ class Auth:
                     guest = True
 
             self.validate_macaroon(macaroon, rights, user_id=user_id)
-        except (
-            pymacaroons.exceptions.MacaroonException,
-            KeyError,
-            TypeError,
-            ValueError,
-        ):
+        except (pymacaroons.exceptions.MacaroonException, TypeError, ValueError):
             raise InvalidClientTokenError("Invalid macaroon passed.")
 
         if rights == "access":
@@ -430,6 +424,27 @@ class Auth:
 
         return user_id, guest
 
+    def get_user_id_from_macaroon(self, macaroon):
+        """Retrieve the user_id given by the caveats on the macaroon.
+
+        Does *not* validate the macaroon.
+
+        Args:
+            macaroon (pymacaroons.Macaroon): The macaroon to validate
+
+        Returns:
+            (str) user id
+
+        Raises:
+            InvalidClientCredentialsError if there is no user_id caveat in the
+                macaroon
+        """
+        user_prefix = "user_id = "
+        for caveat in macaroon.caveats:
+            if caveat.caveat_id.startswith(user_prefix):
+                return caveat.caveat_id[len(user_prefix) :]
+        raise InvalidClientTokenError("No user caveat in macaroon")
+
     def validate_macaroon(self, macaroon, type_string, user_id):
         """
         validate that a Macaroon is understood by and was signed by this server.
@@ -450,13 +465,21 @@ class Auth:
         v.satisfy_exact("type = " + type_string)
         v.satisfy_exact("user_id = %s" % user_id)
         v.satisfy_exact("guest = true")
-        satisfy_expiry(v, self.clock.time_msec)
+        v.satisfy_general(self._verify_expiry)
 
         # access_tokens include a nonce for uniqueness: any value is acceptable
         v.satisfy_general(lambda c: c.startswith("nonce = "))
 
         v.verify(macaroon, self._macaroon_secret_key)
 
+    def _verify_expiry(self, caveat):
+        prefix = "time < "
+        if not caveat.startswith(prefix):
+            return False
+        expiry = int(caveat[len(prefix) :])
+        now = self.hs.get_clock().time_msec()
+        return now < expiry
+
     def get_appservice_by_req(self, request: SynapseRequest) -> ApplicationService:
         token = self.get_access_token_from_request(request)
         service = self.store.get_app_service_by_token(token)
@@ -558,9 +581,6 @@ class Auth:
         Returns:
             bool: False if no access_token was given, True otherwise.
         """
-        # This will always be set by the time Twisted calls us.
-        assert request.args is not None
-
         query_params = request.args.get(b"access_token")
         auth_headers = request.requestHeaders.getRawHeaders(b"Authorization")
         return bool(query_params) or bool(auth_headers)
@@ -577,8 +597,6 @@ class Auth:
             MissingClientTokenError: If there isn't a single access_token in the
                 request
         """
-        # This will always be set by the time Twisted calls us.
-        assert request.args is not None
 
         auth_headers = request.requestHeaders.getRawHeaders(b"Authorization")
         query_params = request.args.get(b"access_token")

synapse/api/constants.py

@@ -51,7 +51,6 @@ class PresenceState:
     OFFLINE = "offline"
     UNAVAILABLE = "unavailable"
     ONLINE = "online"
-    BUSY = "org.matrix.msc3026.busy"
 
 
 class JoinRules:
@@ -99,15 +98,9 @@ class EventTypes:
 
     Retention = "m.room.retention"
 
-    Dummy = "org.matrix.dummy_event"
-
-    MSC1772_SPACE_CHILD = "org.matrix.msc1772.space.child"
-    MSC1772_SPACE_PARENT = "org.matrix.msc1772.space.parent"
-
-
-class EduTypes:
     Presence = "m.presence"
-    RoomKeyRequest = "m.room_key_request"
+
+    Dummy = "org.matrix.dummy_event"
 
 
 class RejectedReason:
@@ -164,9 +157,6 @@ class EventContentFields:
     # cf https://github.com/matrix-org/matrix-doc/pull/2228
     SELF_DESTRUCT_AFTER = "org.matrix.self_destruct_after"
 
-    # cf https://github.com/matrix-org/matrix-doc/pull/1772
-    MSC1772_ROOM_TYPE = "org.matrix.msc1772.type"
-
 
 class RoomEncryptionAlgorithms:
     MEGOLM_V1_AES_SHA2 = "m.megolm.v1.aes-sha2"

synapse/api/ratelimiting.py

@@ -14,7 +14,7 @@
 # limitations under the License.
 
 from collections import OrderedDict
-from typing import Hashable, Optional, Tuple
+from typing import Any, Optional, Tuple
 
 from synapse.api.errors import LimitExceededError
 from synapse.types import Requester
@@ -42,9 +42,7 @@ class Ratelimiter:
         #   * How many times an action has occurred since a point in time
         #   * The point in time
         #   * The rate_hz of this particular entry. This can vary per request
-        self.actions = (
-            OrderedDict()
-        )  # type: OrderedDict[Hashable, Tuple[float, int, float]]
+        self.actions = OrderedDict()  # type: OrderedDict[Any, Tuple[float, int, float]]
 
     def can_requester_do_action(
         self,
@@ -84,7 +82,7 @@ class Ratelimiter:
 
     def can_do_action(
         self,
-        key: Hashable,
+        key: Any,
         rate_hz: Optional[float] = None,
         burst_count: Optional[int] = None,
         update: bool = True,
@@ -177,7 +175,7 @@ class Ratelimiter:
 
     def ratelimit(
         self,
-        key: Hashable,
+        key: Any,
         rate_hz: Optional[float] = None,
         burst_count: Optional[int] = None,
         update: bool = True,

synapse/app/__init__.py

@@ -17,14 +17,14 @@ import sys
 
 from synapse import python_dependencies  # noqa: E402
 
+sys.dont_write_bytecode = True
+
 logger = logging.getLogger(__name__)
 
 try:
     python_dependencies.check_requirements()
 except python_dependencies.DependencyException as e:
-    sys.stderr.writelines(
-        e.message  # noqa: B306, DependencyException.message is a property
-    )
+    sys.stderr.writelines(e.message)
     sys.exit(1)
 
 

synapse/app/_base.py

@@ -21,10 +21,8 @@ import signal
 import socket
 import sys
 import traceback
-import warnings
 from typing import Awaitable, Callable, Iterable
 
-from cryptography.utils import CryptographyDeprecationWarning
 from typing_extensions import NoReturn
 
 from twisted.internet import defer, error, reactor
@@ -197,25 +195,6 @@ def listen_metrics(bind_addresses, port):
         start_http_server(port, addr=host, registry=RegistryProxy)
 
 
-def listen_manhole(bind_addresses: Iterable[str], port: int, manhole_globals: dict):
-    # twisted.conch.manhole 21.1.0 uses "int_from_bytes", which produces a confusing
-    # warning. It's fixed by https://github.com/twisted/twisted/pull/1522), so
-    # suppress the warning for now.
-    warnings.filterwarnings(
-        action="ignore",
-        category=CryptographyDeprecationWarning,
-        message="int_from_bytes is deprecated",
-    )
-
-    from synapse.util.manhole import manhole
-
-    listen_tcp(
-        bind_addresses,
-        port,
-        manhole(username="matrix", password="rabbithole", globals=manhole_globals),
-    )
-
-
 def listen_tcp(bind_addresses, port, factory, reactor=reactor, backlog=50):
     """
     Create a TCP socket for a port and several addresses

synapse/app/admin_cmd.py

@@ -210,9 +210,7 @@ def start(config_options):
     config.update_user_directory = False
     config.run_background_tasks = False
     config.start_pushers = False
-    config.pusher_shard_config.instances = []
     config.send_federation = False
-    config.federation_shard_config.instances = []
 
     synapse.events.USE_FROZEN_DICTS = config.use_frozen_dicts
 

synapse/app/generic_worker.py

@@ -23,7 +23,6 @@ from typing_extensions import ContextManager
 
 from twisted.internet import address
 from twisted.web.resource import IResource
-from twisted.web.server import Request
 
 import synapse
 import synapse.events
@@ -147,6 +146,7 @@ from synapse.storage.databases.main.user_directory import UserDirectoryStore
 from synapse.types import ReadReceipt
 from synapse.util.async_helpers import Linearizer
 from synapse.util.httpresourcetree import create_resource_tree
+from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
 
 logger = logging.getLogger("synapse.app.generic_worker")
@@ -190,7 +190,7 @@ class KeyUploadServlet(RestServlet):
         self.http_client = hs.get_simple_http_client()
         self.main_uri = hs.config.worker_main_http_uri
 
-    async def on_POST(self, request: Request, device_id: Optional[str]):
+    async def on_POST(self, request, device_id):
         requester = await self.auth.get_user_by_req(request, allow_guest=True)
         user_id = requester.user.to_string()
         body = parse_json_object_from_request(request)
@@ -223,12 +223,10 @@ class KeyUploadServlet(RestServlet):
                 header: request.requestHeaders.getRawHeaders(header, [])
                 for header in (b"Authorization", b"User-Agent")
             }
-            # Add the previous hop to the X-Forwarded-For header.
+            # Add the previous hop the the X-Forwarded-For header.
             x_forwarded_for = request.requestHeaders.getRawHeaders(
                 b"X-Forwarded-For", []
             )
-            # we use request.client here, since we want the previous hop, not the
-            # original client (as returned by request.getClientAddress()).
             if isinstance(request.client, (address.IPv4Address, address.IPv6Address)):
                 previous_host = request.client.host.encode("ascii")
                 # If the header exists, add to the comma-separated list of the first
@@ -241,14 +239,6 @@ class KeyUploadServlet(RestServlet):
                     x_forwarded_for = [previous_host]
             headers[b"X-Forwarded-For"] = x_forwarded_for
 
-            # Replicate the original X-Forwarded-Proto header. Note that
-            # XForwardedForRequest overrides isSecure() to give us the original protocol
-            # used by the client, as opposed to the protocol used by our upstream proxy
-            # - which is what we want here.
-            headers[b"X-Forwarded-Proto"] = [
-                b"https" if request.isSecure() else b"http"
-            ]
-
             try:
                 result = await self.http_client.post_json_get_json(
                     self.main_uri + request.uri.decode("ascii"), body, headers=headers
@@ -301,8 +291,6 @@ class GenericWorkerPresence(BasePresenceHandler):
             self.send_stop_syncing, UPDATE_SYNCING_USERS_MS
         )
 
-        self._busy_presence_enabled = hs.config.experimental.msc3026_enabled
-
         hs.get_reactor().addSystemEventTrigger(
             "before",
             "shutdown",
@@ -440,12 +428,8 @@ class GenericWorkerPresence(BasePresenceHandler):
             PresenceState.ONLINE,
             PresenceState.UNAVAILABLE,
             PresenceState.OFFLINE,
-            PresenceState.BUSY,
         )
-
-        if presence not in valid_presence or (
-            presence == PresenceState.BUSY and not self._busy_presence_enabled
-        ):
+        if presence not in valid_presence:
             raise SynapseError(400, "Invalid presence state")
 
         user_id = target_user.to_string()
@@ -639,8 +623,12 @@ class GenericWorkerServer(HomeServer):
             if listener.type == "http":
                 self._listen_http(listener)
             elif listener.type == "manhole":
-                _base.listen_manhole(
-                    listener.bind_addresses, listener.port, manhole_globals={"hs": self}
+                _base.listen_tcp(
+                    listener.bind_addresses,
+                    listener.port,
+                    manhole(
+                        username="matrix", password="rabbithole", globals={"hs": self}
+                    ),
                 )
             elif listener.type == "metrics":
                 if not self.get_config().enable_metrics:
@@ -657,6 +645,9 @@ class GenericWorkerServer(HomeServer):
 
         self.get_tcp_replication().start_replication(self)
 
+    async def remove_pusher(self, app_id, push_key, user_id):
+        self.get_tcp_replication().send_remove_pusher(app_id, push_key, user_id)
+
     @cache_in_self
     def get_replication_data_handler(self):
         return GenericWorkerReplicationHandler(self)
@@ -787,6 +778,13 @@ class FederationSenderHandler:
 
         self._fed_position_linearizer = Linearizer(name="_fed_position_linearizer")
 
+    def on_start(self):
+        # There may be some events that are persisted but haven't been sent,
+        # so send them now.
+        self.federation_sender.notify_new_events(
+            self.store.get_room_max_stream_ordering()
+        )
+
     def wake_destination(self, server: str):
         self.federation_sender.wake_destination(server)
 
@@ -924,6 +922,22 @@ def start(config_options):
         # For other worker types we force this to off.
         config.appservice.notify_appservices = False
 
+    if config.worker_app == "synapse.app.pusher":
+        if config.server.start_pushers:
+            sys.stderr.write(
+                "\nThe pushers must be disabled in the main synapse process"
+                "\nbefore they can be run in a separate worker."
+                "\nPlease add ``start_pushers: false`` to the main config"
+                "\n"
+            )
+            sys.exit(1)
+
+        # Force the pushers to start since they will be disabled in the main config
+        config.server.start_pushers = True
+    else:
+        # For other worker types we force this to off.
+        config.server.start_pushers = False
+
     if config.worker_app == "synapse.app.user_dir":
         if config.server.update_user_directory:
             sys.stderr.write(
@@ -940,6 +954,22 @@ def start(config_options):
         # For other worker types we force this to off.
         config.server.update_user_directory = False
 
+    if config.worker_app == "synapse.app.federation_sender":
+        if config.worker.send_federation:
+            sys.stderr.write(
+                "\nThe send_federation must be disabled in the main synapse process"
+                "\nbefore they can be run in a separate worker."
+                "\nPlease add ``send_federation: false`` to the main config"
+                "\n"
+            )
+            sys.exit(1)
+
+        # Force the pushers to start since they will be disabled in the main config
+        config.worker.send_federation = True
+    else:
+        # For other worker types we force this to off.
+        config.worker.send_federation = False
+
     synapse.events.USE_FROZEN_DICTS = config.use_frozen_dicts
 
     hs = GenericWorkerServer(

synapse/app/homeserver.py

@@ -67,6 +67,7 @@ from synapse.storage import DataStore
 from synapse.storage.engines import IncorrectDatabaseSetup
 from synapse.storage.prepare_database import UpgradeDatabaseException
 from synapse.util.httpresourcetree import create_resource_tree
+from synapse.util.manhole import manhole
 from synapse.util.module_loader import load_module
 from synapse.util.versionstring import get_version_string
 
@@ -287,8 +288,12 @@ class SynapseHomeServer(HomeServer):
             if listener.type == "http":
                 self._listening_services.extend(self._listener_http(config, listener))
             elif listener.type == "manhole":
-                _base.listen_manhole(
-                    listener.bind_addresses, listener.port, manhole_globals={"hs": self}
+                listen_tcp(
+                    listener.bind_addresses,
+                    listener.port,
+                    manhole(
+                        username="matrix", password="rabbithole", globals={"hs": self}
+                    ),
                 )
             elif listener.type == "replication":
                 services = listen_tcp(

synapse/appservice/api.py

@@ -90,7 +90,7 @@ class ApplicationServiceApi(SimpleHttpClient):
         self.clock = hs.get_clock()
 
         self.protocol_meta_cache = ResponseCache(
-            hs.get_clock(), "as_protocol_meta", timeout_ms=HOUR_IN_MS
+            hs, "as_protocol_meta", timeout_ms=HOUR_IN_MS
         )  # type: ResponseCache[Tuple[str, str]]
 
     async def query_user(self, service, user_id):

synapse/config/_base.py

@@ -21,7 +21,7 @@ import os
 from collections import OrderedDict
 from hashlib import sha256
 from textwrap import dedent
-from typing import Any, Iterable, List, MutableMapping, Optional, Union
+from typing import Any, Iterable, List, MutableMapping, Optional
 
 import attr
 import jinja2
@@ -147,20 +147,7 @@ class Config:
         return int(value) * size
 
     @staticmethod
-    def parse_duration(value: Union[str, int]) -> int:
-        """Convert a duration as a string or integer to a number of milliseconds.
-
-        If an integer is provided it is treated as milliseconds and is unchanged.
-
-        String durations can have a suffix of 's', 'm', 'h', 'd', 'w', or 'y'.
-        No suffix is treated as milliseconds.
-
-        Args:
-            value: The duration to parse.
-
-        Returns:
-            The number of milliseconds in the duration.
-        """
+    def parse_duration(value):
         if isinstance(value, int):
             return value
         second = 1000
@@ -212,8 +199,9 @@ class Config:
 
     @classmethod
     def read_file(cls, file_path, config_name):
-        """Deprecated: call read_file directly"""
-        return read_file(file_path, (config_name,))
+        cls.check_file(file_path, config_name)
+        with open(file_path) as file_stream:
+            return file_stream.read()
 
     def read_template(self, filename: str) -> jinja2.Template:
         """Load a template file from disk.
@@ -843,23 +831,22 @@ class ShardedWorkerHandlingConfig:
 
     def should_handle(self, instance_name: str, key: str) -> bool:
         """Whether this instance is responsible for handling the given key."""
-        # If no instances are defined we assume some other worker is handling
-        # this.
-        if not self.instances:
-            return False
+        # If multiple instances are not defined we always return true
+        if not self.instances or len(self.instances) == 1:
+            return True
 
-        return self._get_instance(key) == instance_name
+        return self.get_instance(key) == instance_name
 
-    def _get_instance(self, key: str) -> str:
+    def get_instance(self, key: str) -> str:
         """Get the instance responsible for handling the given key.
 
-        Note: For federation sending and pushers the config for which instance
-        is sending is known only to the sender instance, so we don't expose this
-        method by default.
+        Note: For things like federation sending the config for which instance
+        is sending is known only to the sender instance if there is only one.
+        Therefore `should_handle` should be used where possible.
         """
 
         if not self.instances:
-            raise Exception("Unknown worker")
+            return "master"
 
         if len(self.instances) == 1:
             return self.instances[0]
@@ -876,52 +863,4 @@ class ShardedWorkerHandlingConfig:
         return self.instances[remainder]
 
 
-@attr.s
-class RoutableShardedWorkerHandlingConfig(ShardedWorkerHandlingConfig):
-    """A version of `ShardedWorkerHandlingConfig` that is used for config
-    options where all instances know which instances are responsible for the
-    sharded work.
-    """
-
-    def __attrs_post_init__(self):
-        # We require that `self.instances` is non-empty.
-        if not self.instances:
-            raise Exception("Got empty list of instances for shard config")
-
-    def get_instance(self, key: str) -> str:
-        """Get the instance responsible for handling the given key."""
-        return self._get_instance(key)
-
-
-def read_file(file_path: Any, config_path: Iterable[str]) -> str:
-    """Check the given file exists, and read it into a string
-
-    If it does not, emit an error indicating the problem
-
-    Args:
-        file_path: the file to be read
-        config_path: where in the configuration file_path came from, so that a useful
-           error can be emitted if it does not exist.
-    Returns:
-        content of the file.
-    Raises:
-        ConfigError if there is a problem reading the file.
-    """
-    if not isinstance(file_path, str):
-        raise ConfigError("%r is not a string", config_path)
-
-    try:
-        os.stat(file_path)
-        with open(file_path) as file_stream:
-            return file_stream.read()
-    except OSError as e:
-        raise ConfigError("Error accessing file %r" % (file_path,), config_path) from e
-
-
-__all__ = [
-    "Config",
-    "RootConfig",
-    "ShardedWorkerHandlingConfig",
-    "RoutableShardedWorkerHandlingConfig",
-    "read_file",
-]
+__all__ = ["Config", "RootConfig", "ShardedWorkerHandlingConfig"]

synapse/config/_base.pyi

@@ -149,8 +149,4 @@ class ShardedWorkerHandlingConfig:
     instances: List[str]
     def __init__(self, instances: List[str]) -> None: ...
     def should_handle(self, instance_name: str, key: str) -> bool: ...
-
-class RoutableShardedWorkerHandlingConfig(ShardedWorkerHandlingConfig):
     def get_instance(self, key: str) -> str: ...
-
-def read_file(file_path: Any, config_path: Iterable[str]) -> str: ...

synapse/config/cache.py

@@ -24,7 +24,7 @@ from ._base import Config, ConfigError
 _CACHE_PREFIX = "SYNAPSE_CACHE_FACTOR"
 
 # Map from canonicalised cache name to cache.
-_CACHES = {}  # type: Dict[str, Callable[[float], None]]
+_CACHES = {}
 
 # a lock on the contents of _CACHES
 _CACHES_LOCK = threading.Lock()
@@ -59,9 +59,7 @@ def _canonicalise_cache_name(cache_name: str) -> str:
     return cache_name.lower()
 
 
-def add_resizable_cache(
-    cache_name: str, cache_resize_callback: Callable[[float], None]
-):
+def add_resizable_cache(cache_name: str, cache_resize_callback: Callable):
     """Register a cache that's size can dynamically change
 
     Args:

synapse/config/experimental.py

@@ -27,7 +27,3 @@ class ExperimentalConfig(Config):
 
         # MSC2858 (multiple SSO identity providers)
         self.msc2858_enabled = experimental.get("msc2858_enabled", False)  # type: bool
-        # Spaces (MSC1772, MSC2946, etc)
-        self.spaces_enabled = experimental.get("spaces_enabled", False)  # type: bool
-        # MSC3026 (busy presence state)
-        self.msc3026_enabled = experimental.get("msc3026_enabled", False)  # type: bool

synapse/config/federation.py

@@ -41,10 +41,6 @@ class FederationConfig(Config):
         )
         self.federation_metrics_domains = set(federation_metrics_domains)
 
-        self.allow_profile_lookup_over_federation = config.get(
-            "allow_profile_lookup_over_federation", True
-        )
-
     def generate_config_section(self, config_dir_path, server_name, **kwargs):
         return """\
         ## Federation ##
@@ -70,12 +66,6 @@ class FederationConfig(Config):
         #federation_metrics_domains:
         #  - matrix.org
         #  - example.com
-
-        # Uncomment to disable profile lookup over federation. By default, the
-        # Federation API allows other homeservers to obtain profile data of any user
-        # on this homeserver. Defaults to 'true'.
-        #
-        #allow_profile_lookup_over_federation: false
         """
 
 

synapse/config/key.py

@@ -404,11 +404,7 @@ def _parse_key_servers(key_servers, federation_verify_certificates):
     try:
         jsonschema.validate(key_servers, TRUSTED_KEY_SERVERS_SCHEMA)
     except jsonschema.ValidationError as e:
-        raise ConfigError(
-            "Unable to parse 'trusted_key_servers': {}".format(
-                e.message  # noqa: B306, jsonschema.ValidationError.message is a valid attribute
-            )
-        )
+        raise ConfigError("Unable to parse 'trusted_key_servers': " + e.message)
 
     for server in key_servers:
         server_name = server["server_name"]

synapse/config/logger.py

@@ -21,10 +21,8 @@ import threading
 from string import Template
 
 import yaml
-from zope.interface import implementer
 
 from twisted.logger import (
-    ILogObserver,
     LogBeginner,
     STDLibLogObserver,
     eventAsText,
@@ -229,8 +227,7 @@ def _setup_stdlib_logging(config, log_config_path, logBeginner: LogBeginner) ->
 
     threadlocal = threading.local()
 
-    @implementer(ILogObserver)
-    def _log(event: dict) -> None:
+    def _log(event):
         if "log_text" in event:
             if event["log_text"].startswith("DNSDatagramProtocol starting on "):
                 return

synapse/config/metrics.py

@@ -56,9 +56,7 @@ class MetricsConfig(Config):
             try:
                 check_requirements("sentry")
             except DependencyException as e:
-                raise ConfigError(
-                    e.message  # noqa: B306, DependencyException.message is a property
-                )
+                raise ConfigError(e.message)
 
             self.sentry_dsn = config["sentry"].get("dsn")
             if not self.sentry_dsn:

synapse/config/oidc_config.py

@@ -15,18 +15,17 @@
 # limitations under the License.
 
 from collections import Counter
-from typing import Iterable, List, Mapping, Optional, Tuple, Type
+from typing import Iterable, Optional, Tuple, Type
 
 import attr
 
 from synapse.config._util import validate_config
-from synapse.config.sso import SsoAttributeRequirement
 from synapse.python_dependencies import DependencyException, check_requirements
 from synapse.types import Collection, JsonDict
 from synapse.util.module_loader import load_module
 from synapse.util.stringutils import parse_and_validate_mxc_uri
 
-from ._base import Config, ConfigError, read_file
+from ._base import Config, ConfigError
 
 DEFAULT_USER_MAPPING_PROVIDER = "synapse.handlers.oidc_handler.JinjaOidcMappingProvider"
 
@@ -42,9 +41,7 @@ class OIDCConfig(Config):
         try:
             check_requirements("oidc")
         except DependencyException as e:
-            raise ConfigError(
-                e.message  # noqa: B306, DependencyException.message is a property
-            ) from e
+            raise ConfigError(e.message) from e
 
         # check we don't have any duplicate idp_ids now. (The SSO handler will also
         # check for duplicates when the REST listeners get registered, but that happens
@@ -79,9 +76,6 @@ class OIDCConfig(Config):
         #       Note that, if this is changed, users authenticating via that provider
         #       will no longer be recognised as the same user!
         #
-        #       (Use "oidc" here if you are migrating from an old "oidc_config"
-        #       configuration.)
-        #
         #   idp_name: A user-facing name for this identity provider, which is used to
         #       offer the user a choice of login mechanisms.
         #
@@ -103,26 +97,7 @@ class OIDCConfig(Config):
         #
         #   client_id: Required. oauth2 client id to use.
         #
-        #   client_secret: oauth2 client secret to use. May be omitted if
-        #        client_secret_jwt_key is given, or if client_auth_method is 'none'.
-        #
-        #   client_secret_jwt_key: Alternative to client_secret: details of a key used
-        #      to create a JSON Web Token to be used as an OAuth2 client secret. If
-        #      given, must be a dictionary with the following properties:
-        #
-        #          key: a pem-encoded signing key. Must be a suitable key for the
-        #              algorithm specified. Required unless 'key_file' is given.
-        #
-        #          key_file: the path to file containing a pem-encoded signing key file.
-        #              Required unless 'key' is given.
-        #
-        #          jwt_header: a dictionary giving properties to include in the JWT
-        #              header. Must include the key 'alg', giving the algorithm used to
-        #              sign the JWT, such as "ES256", using the JWA identifiers in
-        #              RFC7518.
-        #
-        #          jwt_payload: an optional dictionary giving properties to include in
-        #              the JWT payload. Normally this should include an 'iss' key.
+        #   client_secret: Required. oauth2 client secret to use.
         #
         #   client_auth_method: auth method to use when exchanging the token. Valid
         #       values are 'client_secret_basic' (default), 'client_secret_post' and
@@ -197,24 +172,6 @@ class OIDCConfig(Config):
         #           which is set to the claims returned by the UserInfo Endpoint and/or
         #           in the ID Token.
         #
-        #   It is possible to configure Synapse to only allow logins if certain attributes
-        #   match particular values in the OIDC userinfo. The requirements can be listed under
-        #   `attribute_requirements` as shown below. All of the listed attributes must
-        #   match for the login to be permitted. Additional attributes can be added to
-        #   userinfo by expanding the `scopes` section of the OIDC config to retrieve
-        #   additional information from the OIDC provider.
-        #
-        #   If the OIDC claim is a list, then the attribute must match any value in the list.
-        #   Otherwise, it must exactly match the value of the claim. Using the example
-        #   below, the `family_name` claim MUST be "Stephensson", but the `groups`
-        #   claim MUST contain "admin".
-        #
-        #   attribute_requirements:
-        #     - attribute: family_name
-        #       value: "Stephensson"
-        #     - attribute: groups
-        #       value: "admin"
-        #
         # See https://github.com/matrix-org/synapse/blob/master/docs/openid.md
         # for information on how to configure these options.
         #
@@ -247,9 +204,34 @@ class OIDCConfig(Config):
           #      localpart_template: "{{{{ user.login }}}}"
           #      display_name_template: "{{{{ user.name }}}}"
           #      email_template: "{{{{ user.email }}}}"
-          #  attribute_requirements:
-          #    - attribute: userGroup
-          #      value: "synapseUsers"
+
+          # For use with Keycloak
+          #
+          #- idp_id: keycloak
+          #  idp_name: Keycloak
+          #  issuer: "https://127.0.0.1:8443/auth/realms/my_realm_name"
+          #  client_id: "synapse"
+          #  client_secret: "copy secret generated in Keycloak UI"
+          #  scopes: ["openid", "profile"]
+
+          # For use with Github
+          #
+          #- idp_id: github
+          #  idp_name: Github
+          #  idp_brand: org.matrix.github
+          #  discover: false
+          #  issuer: "https://github.com/"
+          #  client_id: "your-client-id" # TO BE FILLED
+          #  client_secret: "your-client-secret" # TO BE FILLED
+          #  authorization_endpoint: "https://github.com/login/oauth/authorize"
+          #  token_endpoint: "https://github.com/login/oauth/access_token"
+          #  userinfo_endpoint: "https://api.github.com/user"
+          #  scopes: ["read:user"]
+          #  user_mapping_provider:
+          #    config:
+          #      subject_claim: "id"
+          #      localpart_template: "{{{{ user.login }}}}"
+          #      display_name_template: "{{{{ user.name }}}}"
         """.format(
             mapping_provider=DEFAULT_USER_MAPPING_PROVIDER
         )
@@ -258,7 +240,7 @@ class OIDCConfig(Config):
 # jsonschema definition of the configuration settings for an oidc identity provider
 OIDC_PROVIDER_CONFIG_SCHEMA = {
     "type": "object",
-    "required": ["issuer", "client_id"],
+    "required": ["issuer", "client_id", "client_secret"],
     "properties": {
         "idp_id": {
             "type": "string",
@@ -271,12 +253,7 @@ OIDC_PROVIDER_CONFIG_SCHEMA = {
         "idp_icon": {"type": "string"},
         "idp_brand": {
             "type": "string",
-            "minLength": 1,
-            "maxLength": 255,
-            "pattern": "^[a-z][a-z0-9_.-]*$",
-        },
-        "idp_unstable_brand": {
-            "type": "string",
+            # MSC2758-style namespaced identifier
             "minLength": 1,
             "maxLength": 255,
             "pattern": "^[a-z][a-z0-9_.-]*$",
@@ -285,30 +262,6 @@ OIDC_PROVIDER_CONFIG_SCHEMA = {
         "issuer": {"type": "string"},
         "client_id": {"type": "string"},
         "client_secret": {"type": "string"},
-        "client_secret_jwt_key": {
-            "type": "object",
-            "required": ["jwt_header"],
-            "oneOf": [
-                {"required": ["key"]},
-                {"required": ["key_file"]},
-            ],
-            "properties": {
-                "key": {"type": "string"},
-                "key_file": {"type": "string"},
-                "jwt_header": {
-                    "type": "object",
-                    "required": ["alg"],
-                    "properties": {
-                        "alg": {"type": "string"},
-                    },
-                    "additionalProperties": {"type": "string"},
-                },
-                "jwt_payload": {
-                    "type": "object",
-                    "additionalProperties": {"type": "string"},
-                },
-            },
-        },
         "client_auth_method": {
             "type": "string",
             # the following list is the same as the keys of
@@ -328,10 +281,6 @@ OIDC_PROVIDER_CONFIG_SCHEMA = {
         },
         "allow_existing_users": {"type": "boolean"},
         "user_mapping_provider": {"type": ["object", "null"]},
-        "attribute_requirements": {
-            "type": "array",
-            "items": SsoAttributeRequirement.JSON_SCHEMA,
-        },
     },
 }
 
@@ -455,36 +404,15 @@ def _parse_oidc_config_dict(
                 "idp_icon must be a valid MXC URI", config_path + ("idp_icon",)
             ) from e
 
-    client_secret_jwt_key_config = oidc_config.get("client_secret_jwt_key")
-    client_secret_jwt_key = None  # type: Optional[OidcProviderClientSecretJwtKey]
-    if client_secret_jwt_key_config is not None:
-        keyfile = client_secret_jwt_key_config.get("key_file")
-        if keyfile:
-            key = read_file(keyfile, config_path + ("client_secret_jwt_key",))
-        else:
-            key = client_secret_jwt_key_config["key"]
-        client_secret_jwt_key = OidcProviderClientSecretJwtKey(
-            key=key,
-            jwt_header=client_secret_jwt_key_config["jwt_header"],
-            jwt_payload=client_secret_jwt_key_config.get("jwt_payload", {}),
-        )
-    # parse attribute_requirements from config (list of dicts) into a list of SsoAttributeRequirement
-    attribute_requirements = [
-        SsoAttributeRequirement(**x)
-        for x in oidc_config.get("attribute_requirements", [])
-    ]
-
     return OidcProviderConfig(
         idp_id=idp_id,
         idp_name=oidc_config.get("idp_name", "OIDC"),
         idp_icon=idp_icon,
         idp_brand=oidc_config.get("idp_brand"),
-        unstable_idp_brand=oidc_config.get("unstable_idp_brand"),
         discover=oidc_config.get("discover", True),
         issuer=oidc_config["issuer"],
         client_id=oidc_config["client_id"],
-        client_secret=oidc_config.get("client_secret"),
-        client_secret_jwt_key=client_secret_jwt_key,
+        client_secret=oidc_config["client_secret"],
         client_auth_method=oidc_config.get("client_auth_method", "client_secret_basic"),
         scopes=oidc_config.get("scopes", ["openid"]),
         authorization_endpoint=oidc_config.get("authorization_endpoint"),
@@ -496,22 +424,9 @@ def _parse_oidc_config_dict(
         allow_existing_users=oidc_config.get("allow_existing_users", False),
         user_mapping_provider_class=user_mapping_provider_class,
         user_mapping_provider_config=user_mapping_provider_config,
-        attribute_requirements=attribute_requirements,
     )
 
 
-@attr.s(slots=True, frozen=True)
-class OidcProviderClientSecretJwtKey:
-    # a pem-encoded signing key
-    key = attr.ib(type=str)
-
-    # properties to include in the JWT header
-    jwt_header = attr.ib(type=Mapping[str, str])
-
-    # properties to include in the JWT payload.
-    jwt_payload = attr.ib(type=Mapping[str, str])
-
-
 @attr.s(slots=True, frozen=True)
 class OidcProviderConfig:
     # a unique identifier for this identity provider. Used in the 'user_external_ids'
@@ -527,9 +442,6 @@ class OidcProviderConfig:
     # Optional brand identifier for this IdP.
     idp_brand = attr.ib(type=Optional[str])
 
-    # Optional brand identifier for the unstable API (see MSC2858).
-    unstable_idp_brand = attr.ib(type=Optional[str])
-
     # whether the OIDC discovery mechanism is used to discover endpoints
     discover = attr.ib(type=bool)
 
@@ -540,13 +452,8 @@ class OidcProviderConfig:
     # oauth2 client id to use
     client_id = attr.ib(type=str)
 
-    # oauth2 client secret to use. if `None`, use client_secret_jwt_key to generate
-    # a secret.
-    client_secret = attr.ib(type=Optional[str])
-
-    # key to use to construct a JWT to use as a client secret. May be `None` if
-    # `client_secret` is set.
-    client_secret_jwt_key = attr.ib(type=Optional[OidcProviderClientSecretJwtKey])
+    # oauth2 client secret to use
+    client_secret = attr.ib(type=str)
 
     # auth method to use when exchanging the token.
     # Valid values are 'client_secret_basic', 'client_secret_post' and
@@ -586,6 +493,3 @@ class OidcProviderConfig:
 
     # the config of the user mapping provider
     user_mapping_provider_config = attr.ib()
-
-    # required attributes to require in userinfo to allow login/registration
-    attribute_requirements = attr.ib(type=List[SsoAttributeRequirement])

synapse/config/push.py

@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from ._base import Config
+from ._base import Config, ShardedWorkerHandlingConfig
 
 
 class PushConfig(Config):
@@ -27,6 +27,9 @@ class PushConfig(Config):
             "group_unread_count_by_room", True
         )
 
+        pusher_instances = config.get("pusher_instances") or []
+        self.pusher_shard_config = ShardedWorkerHandlingConfig(pusher_instances)
+
         # There was a a 'redact_content' setting but mistakenly read from the
         # 'email'section'. Check for the flag in the 'push' section, and log,
         # but do not honour it to avoid nasty surprises when people upgrade.

synapse/config/ratelimiting.py

@@ -95,21 +95,11 @@ class RatelimitConfig(Config):
 
         self.rc_joins_local = RateLimitConfig(
             config.get("rc_joins", {}).get("local", {}),
-            defaults={"per_second": 0.1, "burst_count": 10},
+            defaults={"per_second": 0.1, "burst_count": 3},
         )
         self.rc_joins_remote = RateLimitConfig(
             config.get("rc_joins", {}).get("remote", {}),
-            defaults={"per_second": 0.01, "burst_count": 10},
-        )
-
-        # Ratelimit cross-user key requests:
-        # * For local requests this is keyed by the sending device.
-        # * For requests received over federation this is keyed by the origin.
-        #
-        # Note that this isn't exposed in the configuration as it is obscure.
-        self.rc_key_requests = RateLimitConfig(
-            config.get("rc_key_requests", {}),
-            defaults={"per_second": 20, "burst_count": 100},
+            defaults={"per_second": 0.01, "burst_count": 3},
         )
 
         self.rc_3pid_validation = RateLimitConfig(
@@ -187,10 +177,10 @@ class RatelimitConfig(Config):
         #rc_joins:
         #  local:
         #    per_second: 0.1
-        #    burst_count: 10
+        #    burst_count: 3
         #  remote:
         #    per_second: 0.01
-        #    burst_count: 10
+        #    burst_count: 3
         #
         #rc_3pid_validation:
         #  per_second: 0.003

synapse/config/repository.py

@@ -176,9 +176,7 @@ class ContentRepositoryConfig(Config):
                 check_requirements("url_preview")
 
             except DependencyException as e:
-                raise ConfigError(
-                    e.message  # noqa: B306, DependencyException.message is a property
-                )
+                raise ConfigError(e.message)
 
             if "url_preview_ip_range_blacklist" not in config:
                 raise ConfigError(
@@ -208,6 +206,7 @@ class ContentRepositoryConfig(Config):
 
     def generate_config_section(self, data_dir_path, **kwargs):
         media_store = os.path.join(data_dir_path, "media_store")
+        uploads_path = os.path.join(data_dir_path, "uploads")
 
         formatted_thumbnail_sizes = "".join(
             THUMBNAIL_SIZE_YAML % s for s in DEFAULT_THUMBNAIL_SIZES

synapse/config/saml2_config.py

@@ -76,9 +76,7 @@ class SAML2Config(Config):
         try:
             check_requirements("saml2")
         except DependencyException as e:
-            raise ConfigError(
-                e.message  # noqa: B306, DependencyException.message is a property
-            )
+            raise ConfigError(e.message)
 
         self.saml2_enabled = True
 

synapse/config/server.py

@@ -263,12 +263,6 @@ class ServerConfig(Config):
             False,
         )
 
-        # Whether to retrieve and display profile data for a user when they
-        # are invited to a room
-        self.include_profile_data_on_invite = config.get(
-            "include_profile_data_on_invite", True
-        )
-
         if "restrict_public_rooms_to_local_users" in config and (
             "allow_public_rooms_without_auth" in config
             or "allow_public_rooms_over_federation" in config
@@ -397,6 +391,7 @@ class ServerConfig(Config):
         if self.public_baseurl is not None:
             if self.public_baseurl[-1] != "/":
                 self.public_baseurl += "/"
+        self.start_pushers = config.get("start_pushers", True)
 
         # (undocumented) option for torturing the worker-mode replication a bit,
         # for testing. The value defines the number of milliseconds to pause before
@@ -841,7 +836,8 @@ class ServerConfig(Config):
         # Whether to require authentication to retrieve profile data (avatars,
         # display names) of other users through the client API. Defaults to
         # 'false'. Note that profile data is also available via the federation
-        # API, unless allow_profile_lookup_over_federation is set to false.
+        # API, so this setting is of limited value if federation is enabled on
+        # the server.
         #
         #require_auth_for_profile_requests: true
 
@@ -852,14 +848,6 @@ class ServerConfig(Config):
         #
         #limit_profile_requests_to_users_who_share_rooms: true
 
-        # Uncomment to prevent a user's profile data from being retrieved and
-        # displayed in a room until they have joined it. By default, a user's
-        # profile data is included in an invite event, regardless of the values
-        # of the above two settings, and whether or not the users share a server.
-        # Defaults to 'true'.
-        #
-        #include_profile_data_on_invite: false
-
         # If set to 'true', removes the need for authentication to access the server's
         # public rooms directory through the client API, meaning that anyone can
         # query the room directory. Defaults to 'false'.

synapse/config/stats.py

@@ -13,22 +13,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import logging
+import sys
 
 from ._base import Config
 
-ROOM_STATS_DISABLED_WARN = """\
-WARNING: room/user statistics have been disabled via the stats.enabled
-configuration setting. This means that certain features (such as the room
-directory) will not operate correctly. Future versions of Synapse may ignore
-this setting.
-
-To fix this warning, remove the stats.enabled setting from your configuration
-file.
---------------------------------------------------------------------------------"""
-
-logger = logging.getLogger(__name__)
-
 
 class StatsConfig(Config):
     """Stats Configuration
@@ -40,29 +28,30 @@ class StatsConfig(Config):
     def read_config(self, config, **kwargs):
         self.stats_enabled = True
         self.stats_bucket_size = 86400 * 1000
+        self.stats_retention = sys.maxsize
         stats_config = config.get("stats", None)
         if stats_config:
             self.stats_enabled = stats_config.get("enabled", self.stats_enabled)
             self.stats_bucket_size = self.parse_duration(
                 stats_config.get("bucket_size", "1d")
             )
-        if not self.stats_enabled:
-            logger.warning(ROOM_STATS_DISABLED_WARN)
+            self.stats_retention = self.parse_duration(
+                stats_config.get("retention", "%ds" % (sys.maxsize,))
+            )
 
     def generate_config_section(self, config_dir_path, server_name, **kwargs):
         return """
-        # Settings for local room and user statistics collection. See
-        # docs/room_and_user_statistics.md.
+        # Local statistics collection. Used in populating the room directory.
         #
-        stats:
-          # Uncomment the following to disable room and user statistics. Note that doing
-          # so may cause certain features (such as the room directory) not to work
-          # correctly.
-          #
-          #enabled: false
-
-          # The size of each timeslice in the room_stats_historical and
-          # user_stats_historical tables, as a time period. Defaults to "1d".
-          #
-          #bucket_size: 1h
+        # 'bucket_size' controls how large each statistics timeslice is. It can
+        # be defined in a human readable short form -- e.g. "1d", "1y".
+        #
+        # 'retention' controls how long historical statistics will be kept for.
+        # It can be defined in a human readable short form -- e.g. "1d", "1y".
+        #
+        #
+        #stats:
+        #   enabled: true
+        #   bucket_size: 1d
+        #   retention: 1y
         """

synapse/config/tracer.py

@@ -39,9 +39,7 @@ class TracerConfig(Config):
         try:
             check_requirements("opentracing")
         except DependencyException as e:
-            raise ConfigError(
-                e.message  # noqa: B306, DependencyException.message is a property
-            )
+            raise ConfigError(e.message)
 
         # The tracer is enabled so sanitize the config
 

synapse/config/user_directory.py

@@ -24,46 +24,32 @@ class UserDirectoryConfig(Config):
     section = "userdirectory"
 
     def read_config(self, config, **kwargs):
-        user_directory_config = config.get("user_directory") or {}
-        self.user_directory_search_enabled = user_directory_config.get("enabled", True)
-        self.user_directory_search_all_users = user_directory_config.get(
-            "search_all_users", False
-        )
-        self.user_directory_search_prefer_local_users = user_directory_config.get(
-            "prefer_local_users", False
-        )
+        self.user_directory_search_enabled = True
+        self.user_directory_search_all_users = False
+        user_directory_config = config.get("user_directory", None)
+        if user_directory_config:
+            self.user_directory_search_enabled = user_directory_config.get(
+                "enabled", True
+            )
+            self.user_directory_search_all_users = user_directory_config.get(
+                "search_all_users", False
+            )
 
     def generate_config_section(self, config_dir_path, server_name, **kwargs):
         return """
         # User Directory configuration
         #
-        user_directory:
-            # Defines whether users can search the user directory. If false then
-            # empty responses are returned to all queries. Defaults to true.
-            #
-            # Uncomment to disable the user directory.
-            #
-            #enabled: false
-
-            # Defines whether to search all users visible to your HS when searching
-            # the user directory, rather than limiting to users visible in public
-            # rooms. Defaults to false.
-            #
-            # If you set it true, you'll have to rebuild the user_directory search
-            # indexes, see:
-            # https://github.com/matrix-org/synapse/blob/master/docs/user_directory.md
-            #
-            # Uncomment to return search results containing all known users, even if that
-            # user does not share a room with the requester.
-            #
-            #search_all_users: true
-
-            # Defines whether to prefer local users in search query results.
-            # If True, local users are more likely to appear above remote users
-            # when searching the user directory. Defaults to false.
-            #
-            # Uncomment to prefer local over remote users in user directory search
-            # results.
-            #
-            #prefer_local_users: true
+        # 'enabled' defines whether users can search the user directory. If
+        # false then empty responses are returned to all queries. Defaults to
+        # true.
+        #
+        # 'search_all_users' defines whether to search all users visible to your HS
+        # when searching the user directory, rather than limiting to users visible
+        # in public rooms.  Defaults to false.  If you set it True, you'll have to
+        # rebuild the user_directory search indexes, see
+        # https://github.com/matrix-org/synapse/blob/master/docs/user_directory.md
+        #
+        #user_directory:
+        #  enabled: true
+        #  search_all_users: false
         """

synapse/config/workers.py

@@ -17,28 +17,9 @@ from typing import List, Union
 
 import attr
 
-from ._base import (
-    Config,
-    ConfigError,
-    RoutableShardedWorkerHandlingConfig,
-    ShardedWorkerHandlingConfig,
-)
+from ._base import Config, ConfigError, ShardedWorkerHandlingConfig
 from .server import ListenerConfig, parse_listener_def
 
-_FEDERATION_SENDER_WITH_SEND_FEDERATION_ENABLED_ERROR = """
-The send_federation config option must be disabled in the main
-synapse process before they can be run in a separate worker.
-
-Please add ``send_federation: false`` to the main config
-"""
-
-_PUSHER_WITH_START_PUSHERS_ENABLED_ERROR = """
-The start_pushers config option must be disabled in the main
-synapse process before they can be run in a separate worker.
-
-Please add ``start_pushers: false`` to the main config
-"""
-
 
 def _instance_to_list_converter(obj: Union[str, List[str]]) -> List[str]:
     """Helper for allowing parsing a string or list of strings to a config
@@ -122,7 +103,6 @@ class WorkerConfig(Config):
         self.worker_replication_secret = config.get("worker_replication_secret", None)
 
         self.worker_name = config.get("worker_name", self.worker_app)
-        self.instance_name = self.worker_name or "master"
 
         self.worker_main_http_uri = config.get("worker_main_http_uri", None)
 
@@ -138,41 +118,12 @@ class WorkerConfig(Config):
                 )
             )
 
-        # Handle federation sender configuration.
-        #
-        # There are two ways of configuring which instances handle federation
-        # sending:
-        #   1. The old way where "send_federation" is set to false and running a
-        #      `synapse.app.federation_sender` worker app.
-        #   2. Specifying the workers sending federation in
-        #      `federation_sender_instances`.
-        #
+        # Whether to send federation traffic out in this process. This only
+        # applies to some federation traffic, and so shouldn't be used to
+        # "disable" federation
+        self.send_federation = config.get("send_federation", True)
 
-        send_federation = config.get("send_federation", True)
-
-        federation_sender_instances = config.get("federation_sender_instances")
-        if federation_sender_instances is None:
-            # Default to an empty list, which means "another, unknown, worker is
-            # responsible for it".
-            federation_sender_instances = []
-
-            # If no federation sender instances are set we check if
-            # `send_federation` is set, which means use master
-            if send_federation:
-                federation_sender_instances = ["master"]
-
-            if self.worker_app == "synapse.app.federation_sender":
-                if send_federation:
-                    # If we're running federation senders, and not using
-                    # `federation_sender_instances`, then we should have
-                    # explicitly set `send_federation` to false.
-                    raise ConfigError(
-                        _FEDERATION_SENDER_WITH_SEND_FEDERATION_ENABLED_ERROR
-                    )
-
-                federation_sender_instances = [self.worker_name]
-
-        self.send_federation = self.instance_name in federation_sender_instances
+        federation_sender_instances = config.get("federation_sender_instances") or []
         self.federation_shard_config = ShardedWorkerHandlingConfig(
             federation_sender_instances
         )
@@ -213,37 +164,7 @@ class WorkerConfig(Config):
                 "Must only specify one instance to handle `receipts` messages."
             )
 
-        if len(self.writers.events) == 0:
-            raise ConfigError("Must specify at least one instance to handle `events`.")
-
-        self.events_shard_config = RoutableShardedWorkerHandlingConfig(
-            self.writers.events
-        )
-
-        # Handle sharded push
-        start_pushers = config.get("start_pushers", True)
-        pusher_instances = config.get("pusher_instances")
-        if pusher_instances is None:
-            # Default to an empty list, which means "another, unknown, worker is
-            # responsible for it".
-            pusher_instances = []
-
-            # If no pushers instances are set we check if `start_pushers` is
-            # set, which means use master
-            if start_pushers:
-                pusher_instances = ["master"]
-
-            if self.worker_app == "synapse.app.pusher":
-                if start_pushers:
-                    # If we're running pushers, and not using
-                    # `pusher_instances`, then we should have explicitly set
-                    # `start_pushers` to false.
-                    raise ConfigError(_PUSHER_WITH_START_PUSHERS_ENABLED_ERROR)
-
-                pusher_instances = [self.instance_name]
-
-        self.start_pushers = self.instance_name in pusher_instances
-        self.pusher_shard_config = ShardedWorkerHandlingConfig(pusher_instances)
+        self.events_shard_config = ShardedWorkerHandlingConfig(self.writers.events)
 
         # Whether this worker should run background tasks or not.
         #

synapse/crypto/context_factory.py

@@ -191,7 +191,7 @@ def _context_info_cb(ssl_connection, where, ret):
         # ... we further assume that SSLClientConnectionCreator has set the
         # '_synapse_tls_verifier' attribute to a ConnectionVerifier object.
         tls_protocol._synapse_tls_verifier.verify_context_info_cb(ssl_connection, where)
-    except BaseException:  # taken from the twisted implementation
+    except:  # noqa: E722, taken from the twisted implementation
         logger.exception("Error during info_callback")
         f = Failure()
         tls_protocol.failVerification(f)
@@ -219,7 +219,7 @@ class SSLClientConnectionCreator:
         # ... and we also gut-wrench a '_synapse_tls_verifier' attribute into the
         # tls_protocol so that the SSL context's info callback has something to
         # call to do the cert verification.
-        tls_protocol._synapse_tls_verifier = self._verifier
+        setattr(tls_protocol, "_synapse_tls_verifier", self._verifier)
         return connection
 
 

synapse/crypto/keyring.py

@@ -57,7 +57,7 @@ from synapse.util.metrics import Measure
 from synapse.util.retryutils import NotRetryingDestination
 
 if TYPE_CHECKING:
-    from synapse.server import HomeServer
+    from synapse.app.homeserver import HomeServer
 
 logger = logging.getLogger(__name__)
 

synapse/events/__init__.py

@@ -98,7 +98,7 @@ class DefaultDictProperty(DictProperty):
 
 
 class _EventInternalMetadata:
-    __slots__ = ["_dict", "stream_ordering", "outlier"]
+    __slots__ = ["_dict", "stream_ordering"]
 
     def __init__(self, internal_metadata_dict: JsonDict):
         # we have to copy the dict, because it turns out that the same dict is
@@ -108,10 +108,7 @@ class _EventInternalMetadata:
         # the stream ordering of this event. None, until it has been persisted.
         self.stream_ordering = None  # type: Optional[int]
 
-        # whether this event is an outlier (ie, whether we have the state at that point
-        # in the DAG)
-        self.outlier = False
-
+    outlier = DictProperty("outlier")  # type: bool
     out_of_band_membership = DictProperty("out_of_band_membership")  # type: bool
     send_on_behalf_of = DictProperty("send_on_behalf_of")  # type: str
     recheck_redaction = DictProperty("recheck_redaction")  # type: bool
@@ -132,7 +129,7 @@ class _EventInternalMetadata:
         return dict(self._dict)
 
     def is_outlier(self) -> bool:
-        return self.outlier
+        return self._dict.get("outlier", False)
 
     def is_out_of_band_membership(self) -> bool:
         """Whether this is an out of band membership, like an invite or an invite

synapse/events/spamcheck.py

@@ -15,7 +15,6 @@
 # limitations under the License.
 
 import inspect
-import logging
 from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
 
 from synapse.rest.media.v1._base import FileInfo
@@ -28,8 +27,6 @@ if TYPE_CHECKING:
     import synapse.events
     import synapse.server
 
-logger = logging.getLogger(__name__)
-
 
 class SpamChecker:
     def __init__(self, hs: "synapse.server.HomeServer"):
@@ -193,7 +190,6 @@ class SpamChecker:
         email_threepid: Optional[dict],
         username: Optional[str],
         request_info: Collection[Tuple[str, str]],
-        auth_provider_id: Optional[str] = None,
     ) -> RegistrationBehaviour:
         """Checks if we should allow the given registration request.
 
@@ -202,9 +198,6 @@ class SpamChecker:
             username: The request user name, if any
             request_info: List of tuples of user agent and IP that
                 were used during the registration process.
-            auth_provider_id: The SSO IdP the user used, e.g "oidc", "saml",
-                "cas". If any. Note this does not include users registered
-                via a password provider.
 
         Returns:
             Enum for how the request should be handled
@@ -215,25 +208,9 @@ class SpamChecker:
             # spam checker
             checker = getattr(spam_checker, "check_registration_for_spam", None)
             if checker:
-                # Provide auth_provider_id if the function supports it
-                checker_args = inspect.signature(checker)
-                if len(checker_args.parameters) == 4:
-                    d = checker(
-                        email_threepid,
-                        username,
-                        request_info,
-                        auth_provider_id,
-                    )
-                elif len(checker_args.parameters) == 3:
-                    d = checker(email_threepid, username, request_info)
-                else:
-                    logger.error(
-                        "Invalid signature for %s.check_registration_for_spam. Denying registration",
-                        spam_checker.__module__,
-                    )
-                    return RegistrationBehaviour.DENY
-
-                behaviour = await maybe_awaitable(d)
+                behaviour = await maybe_awaitable(
+                    checker(email_threepid, username, request_info)
+                )
                 assert isinstance(behaviour, RegistrationBehaviour)
                 if behaviour != RegistrationBehaviour.ALLOW:
                     return behaviour

synapse/events/third_party_rules.py

@@ -13,15 +13,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from typing import TYPE_CHECKING, Union
+from typing import Callable, Union
 
 from synapse.events import EventBase
 from synapse.events.snapshot import EventContext
 from synapse.types import Requester, StateMap
 
-if TYPE_CHECKING:
-    from synapse.server import HomeServer
-
 
 class ThirdPartyEventRules:
     """Allows server admins to provide a Python module implementing an extra
@@ -31,7 +28,7 @@ class ThirdPartyEventRules:
     behaviours.
     """
 
-    def __init__(self, hs: "HomeServer"):
+    def __init__(self, hs):
         self.third_party_rules = None
 
         self.store = hs.get_datastore()
@@ -98,9 +95,10 @@ class ThirdPartyEventRules:
         if self.third_party_rules is None:
             return True
 
-        return await self.third_party_rules.on_create_room(
+        ret = await self.third_party_rules.on_create_room(
             requester, config, is_requester_admin
         )
+        return ret
 
     async def check_threepid_can_be_invited(
         self, medium: str, address: str, room_id: str
@@ -121,9 +119,10 @@ class ThirdPartyEventRules:
 
         state_events = await self._get_state_map_for_room(room_id)
 
-        return await self.third_party_rules.check_threepid_can_be_invited(
+        ret = await self.third_party_rules.check_threepid_can_be_invited(
             medium, address, state_events
         )
+        return ret
 
     async def check_visibility_can_be_modified(
         self, room_id: str, new_visibility: str
@@ -144,7 +143,7 @@ class ThirdPartyEventRules:
         check_func = getattr(
             self.third_party_rules, "check_visibility_can_be_modified", None
         )
-        if not check_func or not callable(check_func):
+        if not check_func or not isinstance(check_func, Callable):
             return True
 
         state_events = await self._get_state_map_for_room(room_id)

synapse/events/utils.py

@@ -22,7 +22,6 @@ from synapse.api.constants import EventTypes, RelationTypes
 from synapse.api.errors import Codes, SynapseError
 from synapse.api.room_versions import RoomVersion
 from synapse.util.async_helpers import yieldable_gather_results
-from synapse.util.frozenutils import unfreeze
 
 from . import EventBase
 
@@ -55,8 +54,6 @@ def prune_event(event: EventBase) -> EventBase:
         event.internal_metadata.stream_ordering
     )
 
-    pruned_event.internal_metadata.outlier = event.internal_metadata.outlier
-
     # Mark the event as redacted
     pruned_event.internal_metadata.redacted = True
 
@@ -403,19 +400,10 @@ class EventClientSerializer:
                 # If there is an edit replace the content, preserving existing
                 # relations.
 
-                # Ensure we take copies of the edit content, otherwise we risk modifying
-                # the original event.
-                edit_content = edit.content.copy()
-
-                # Unfreeze the event content if necessary, so that we may modify it below
-                edit_content = unfreeze(edit_content)
-                serialized_event["content"] = edit_content.get("m.new_content", {})
-
-                # Check for existing relations
                 relations = event.content.get("m.relates_to")
+                serialized_event["content"] = edit.content.get("m.new_content", {})
                 if relations:
-                    # Keep the relations, ensuring we use a dict copy of the original
-                    serialized_event["content"]["m.relates_to"] = relations.copy()
+                    serialized_event["content"]["m.relates_to"] = relations
                 else:
                     serialized_event["content"].pop("m.relates_to", None)
 

synapse/federation/federation_client.py

@@ -27,13 +27,11 @@ from typing import (
     List,
     Mapping,
     Optional,
-    Sequence,
     Tuple,
     TypeVar,
     Union,
 )
 
-import attr
 from prometheus_client import Counter
 
 from twisted.internet import defer
@@ -64,7 +62,7 @@ from synapse.util.caches.expiringcache import ExpiringCache
 from synapse.util.retryutils import NotRetryingDestination
 
 if TYPE_CHECKING:
-    from synapse.server import HomeServer
+    from synapse.app.homeserver import HomeServer
 
 logger = logging.getLogger(__name__)
 
@@ -457,7 +455,6 @@ class FederationClient(FederationBase):
         description: str,
         destinations: Iterable[str],
         callback: Callable[[str], Awaitable[T]],
-        failover_on_unknown_endpoint: bool = False,
     ) -> T:
         """Try an operation on a series of servers, until it succeeds
 
@@ -477,10 +474,6 @@ class FederationClient(FederationBase):
                 next server tried. Normally the stacktrace is logged but this is
                 suppressed if the exception is an InvalidResponseError.
 
-            failover_on_unknown_endpoint: if True, we will try other servers if it looks
-                like a server doesn't support the endpoint. This is typically useful
-                if the endpoint in question is new or experimental.
-
         Returns:
             The result of callback, if it succeeds
 
@@ -500,31 +493,16 @@ class FederationClient(FederationBase):
             except UnsupportedRoomVersionError:
                 raise
             except HttpResponseException as e:
-                synapse_error = e.to_synapse_error()
-                failover = False
-
-                if 500 <= e.code < 600:
-                    failover = True
-
-                elif failover_on_unknown_endpoint:
-                    # there is no good way to detect an "unknown" endpoint. Dendrite
-                    # returns a 404 (with no body); synapse returns a 400
-                    # with M_UNRECOGNISED.
-                    if e.code == 404 or (
-                        e.code == 400 and synapse_error.errcode == Codes.UNRECOGNIZED
-                    ):
-                        failover = True
-
-                if not failover:
-                    raise synapse_error from e
-
-                logger.warning(
-                    "Failed to %s via %s: %i %s",
-                    description,
-                    destination,
-                    e.code,
-                    e.args[0],
-                )
+                if not 500 <= e.code < 600:
+                    raise e.to_synapse_error()
+                else:
+                    logger.warning(
+                        "Failed to %s via %s: %i %s",
+                        description,
+                        destination,
+                        e.code,
+                        e.args[0],
+                    )
             except Exception:
                 logger.warning(
                     "Failed to %s via %s", description, destination, exc_info=True
@@ -1064,141 +1042,3 @@ class FederationClient(FederationBase):
         # If we don't manage to find it, return None. It's not an error if a
         # server doesn't give it to us.
         return None
-
-    async def get_space_summary(
-        self,
-        destinations: Iterable[str],
-        room_id: str,
-        suggested_only: bool,
-        max_rooms_per_space: Optional[int],
-        exclude_rooms: List[str],
-    ) -> "FederationSpaceSummaryResult":
-        """
-        Call other servers to get a summary of the given space
-
-
-        Args:
-            destinations: The remote servers. We will try them in turn, omitting any
-                that have been blacklisted.
-
-            room_id: ID of the space to be queried
-
-            suggested_only:  If true, ask the remote server to only return children
-                with the "suggested" flag set
-
-            max_rooms_per_space: A limit on the number of children to return for each
-                space
-
-            exclude_rooms: A list of room IDs to tell the remote server to skip
-
-        Returns:
-            a parsed FederationSpaceSummaryResult
-
-        Raises:
-            SynapseError if we were unable to get a valid summary from any of the
-               remote servers
-        """
-
-        async def send_request(destination: str) -> FederationSpaceSummaryResult:
-            res = await self.transport_layer.get_space_summary(
-                destination=destination,
-                room_id=room_id,
-                suggested_only=suggested_only,
-                max_rooms_per_space=max_rooms_per_space,
-                exclude_rooms=exclude_rooms,
-            )
-
-            try:
-                return FederationSpaceSummaryResult.from_json_dict(res)
-            except ValueError as e:
-                raise InvalidResponseError(str(e))
-
-        return await self._try_destination_list(
-            "fetch space summary",
-            destinations,
-            send_request,
-            failover_on_unknown_endpoint=True,
-        )
-
-
-@attr.s(frozen=True, slots=True)
-class FederationSpaceSummaryEventResult:
-    """Represents a single event in the result of a successful get_space_summary call.
-
-    It's essentially just a serialised event object, but we do a bit of parsing and
-    validation in `from_json_dict` and store some of the validated properties in
-    object attributes.
-    """
-
-    event_type = attr.ib(type=str)
-    state_key = attr.ib(type=str)
-    via = attr.ib(type=Sequence[str])
-
-    # the raw data, including the above keys
-    data = attr.ib(type=JsonDict)
-
-    @classmethod
-    def from_json_dict(cls, d: JsonDict) -> "FederationSpaceSummaryEventResult":
-        """Parse an event within the result of a /spaces/ request
-
-        Args:
-            d: json object to be parsed
-
-        Raises:
-            ValueError if d is not a valid event
-        """
-
-        event_type = d.get("type")
-        if not isinstance(event_type, str):
-            raise ValueError("Invalid event: 'event_type' must be a str")
-
-        state_key = d.get("state_key")
-        if not isinstance(state_key, str):
-            raise ValueError("Invalid event: 'state_key' must be a str")
-
-        content = d.get("content")
-        if not isinstance(content, dict):
-            raise ValueError("Invalid event: 'content' must be a dict")
-
-        via = content.get("via")
-        if not isinstance(via, Sequence):
-            raise ValueError("Invalid event: 'via' must be a list")
-        if any(not isinstance(v, str) for v in via):
-            raise ValueError("Invalid event: 'via' must be a list of strings")
-
-        return cls(event_type, state_key, via, d)
-
-
-@attr.s(frozen=True, slots=True)
-class FederationSpaceSummaryResult:
-    """Represents the data returned by a successful get_space_summary call."""
-
-    rooms = attr.ib(type=Sequence[JsonDict])
-    events = attr.ib(type=Sequence[FederationSpaceSummaryEventResult])
-
-    @classmethod
-    def from_json_dict(cls, d: JsonDict) -> "FederationSpaceSummaryResult":
-        """Parse the result of a /spaces/ request
-
-        Args:
-            d: json object to be parsed
-
-        Raises:
-            ValueError if d is not a valid /spaces/ response
-        """
-        rooms = d.get("rooms")
-        if not isinstance(rooms, Sequence):
-            raise ValueError("'rooms' must be a list")
-        if any(not isinstance(r, dict) for r in rooms):
-            raise ValueError("Invalid room in 'rooms' list")
-
-        events = d.get("events")
-        if not isinstance(events, Sequence):
-            raise ValueError("'events' must be a list")
-        if any(not isinstance(e, dict) for e in events):
-            raise ValueError("Invalid event in 'events' list")
-        parsed_events = [
-            FederationSpaceSummaryEventResult.from_json_dict(e) for e in events
-        ]
-
-        return cls(rooms, parsed_events)

synapse/federation/federation_server.py

@@ -22,7 +22,6 @@ from typing import (
     Awaitable,
     Callable,
     Dict,
-    Iterable,
     List,
     Optional,
     Tuple,
@@ -35,7 +34,7 @@ from twisted.internet import defer
 from twisted.internet.abstract import isIPAddress
 from twisted.python import failure
 
-from synapse.api.constants import EduTypes, EventTypes
+from synapse.api.constants import EventTypes, Membership
 from synapse.api.errors import (
     AuthError,
     Codes,
@@ -45,7 +44,6 @@ from synapse.api.errors import (
     SynapseError,
     UnsupportedRoomVersionError,
 )
-from synapse.api.ratelimiting import Ratelimiter
 from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
 from synapse.events import EventBase
 from synapse.federation.federation_base import FederationBase, event_from_pdu_json
@@ -63,7 +61,7 @@ from synapse.replication.http.federation import (
     ReplicationFederationSendEduRestServlet,
     ReplicationGetQueryRestServlet,
 )
-from synapse.types import JsonDict
+from synapse.types import JsonDict, get_domain_from_id
 from synapse.util import glob_to_regex, json_decoder, unwrapFirstError
 from synapse.util.async_helpers import Linearizer, concurrently_execute
 from synapse.util.caches.response_cache import ResponseCache
@@ -91,15 +89,16 @@ pdu_process_time = Histogram(
     "Time taken to process an event",
 )
 
-last_pdu_ts_metric = Gauge(
-    "synapse_federation_last_received_pdu_time",
-    "The timestamp of the last PDU which was successfully received from the given domain",
+
+last_pdu_age_metric = Gauge(
+    "synapse_federation_last_received_pdu_age",
+    "The age (in seconds) of the last PDU successfully received from the given domain",
     labelnames=("server_name",),
 )
 
 
 class FederationServer(FederationBase):
-    def __init__(self, hs: "HomeServer"):
+    def __init__(self, hs):
         super().__init__(hs)
 
         self.auth = hs.get_auth()
@@ -112,15 +111,14 @@ class FederationServer(FederationBase):
         # with FederationHandlerRegistry.
         hs.get_directory_handler()
 
-        self._server_linearizer = Linearizer("fed_server")
+        self._federation_ratelimiter = hs.get_federation_ratelimiter()
 
-        # origins that we are currently processing a transaction from.
-        # a dict from origin to txn id.
-        self._active_transactions = {}  # type: Dict[str, str]
+        self._server_linearizer = Linearizer("fed_server")
+        self._transaction_linearizer = Linearizer("fed_txn_handler")
 
         # We cache results for transaction with the same ID
         self._transaction_resp_cache = ResponseCache(
-            hs.get_clock(), "fed_txn_handler", timeout_ms=30000
+            hs, "fed_txn_handler", timeout_ms=30000
         )  # type: ResponseCache[Tuple[str, str]]
 
         self.transaction_actions = TransactionActions(self.store)
@@ -130,10 +128,10 @@ class FederationServer(FederationBase):
         # We cache responses to state queries, as they take a while and often
         # come in waves.
         self._state_resp_cache = ResponseCache(
-            hs.get_clock(), "state_resp", timeout_ms=30000
+            hs, "state_resp", timeout_ms=30000
         )  # type: ResponseCache[Tuple[str, str]]
         self._state_ids_resp_cache = ResponseCache(
-            hs.get_clock(), "state_ids_resp", timeout_ms=30000
+            hs, "state_ids_resp", timeout_ms=30000
         )  # type: ResponseCache[Tuple[str, str]]
 
         self._federation_metrics_domains = (
@@ -170,33 +168,6 @@ class FederationServer(FederationBase):
 
         logger.debug("[%s] Got transaction", transaction_id)
 
-        # Reject malformed transactions early: reject if too many PDUs/EDUs
-        if len(transaction.pdus) > 50 or (  # type: ignore
-            hasattr(transaction, "edus") and len(transaction.edus) > 100  # type: ignore
-        ):
-            logger.info("Transaction PDU or EDU count too large. Returning 400")
-            return 400, {}
-
-        # we only process one transaction from each origin at a time. We need to do
-        # this check here, rather than in _on_incoming_transaction_inner so that we
-        # don't cache the rejection in _transaction_resp_cache (so that if the txn
-        # arrives again later, we can process it).
-        current_transaction = self._active_transactions.get(origin)
-        if current_transaction and current_transaction != transaction_id:
-            logger.warning(
-                "Received another txn %s from %s while still processing %s",
-                transaction_id,
-                origin,
-                current_transaction,
-            )
-            return 429, {
-                "errcode": Codes.UNKNOWN,
-                "error": "Too many concurrent transactions",
-            }
-
-        # CRITICAL SECTION: we must now not await until we populate _active_transactions
-        # in _on_incoming_transaction_inner.
-
         # We wrap in a ResponseCache so that we de-duplicate retried
         # transactions.
         return await self._transaction_resp_cache.wrap(
@@ -210,18 +181,26 @@ class FederationServer(FederationBase):
     async def _on_incoming_transaction_inner(
         self, origin: str, transaction: Transaction, request_time: int
     ) -> Tuple[int, Dict[str, Any]]:
-        # CRITICAL SECTION: the first thing we must do (before awaiting) is
-        # add an entry to _active_transactions.
-        assert origin not in self._active_transactions
-        self._active_transactions[origin] = transaction.transaction_id  # type: ignore
+        # Use a linearizer to ensure that transactions from a remote are
+        # processed in order.
+        with await self._transaction_linearizer.queue(origin):
+            # We rate limit here *after* we've queued up the incoming requests,
+            # so that we don't fill up the ratelimiter with blocked requests.
+            #
+            # This is important as the ratelimiter allows N concurrent requests
+            # at a time, and only starts ratelimiting if there are more requests
+            # than that being processed at a time. If we queued up requests in
+            # the linearizer/response cache *after* the ratelimiting then those
+            # queued up requests would count as part of the allowed limit of N
+            # concurrent requests.
+            with self._federation_ratelimiter.ratelimit(origin) as d:
+                await d
 
-        try:
-            result = await self._handle_incoming_transaction(
-                origin, transaction, request_time
-            )
-            return result
-        finally:
-            del self._active_transactions[origin]
+                result = await self._handle_incoming_transaction(
+                    origin, transaction, request_time
+                )
+
+        return result
 
     async def _handle_incoming_transaction(
         self, origin: str, transaction: Transaction, request_time: int
@@ -247,6 +226,19 @@ class FederationServer(FederationBase):
 
         logger.debug("[%s] Transaction is new", transaction.transaction_id)  # type: ignore
 
+        # Reject if PDU count > 50 or EDU count > 100
+        if len(transaction.pdus) > 50 or (  # type: ignore
+            hasattr(transaction, "edus") and len(transaction.edus) > 100  # type: ignore
+        ):
+
+            logger.info("Transaction PDU or EDU count too large. Returning 400")
+
+            response = {}
+            await self.transaction_actions.set_response(
+                origin, transaction, 400, response
+            )
+            return 400, response
+
         # We process PDUs and EDUs in parallel. This is important as we don't
         # want to block things like to device messages from reaching clients
         # behind the potentially expensive handling of PDUs.
@@ -342,48 +334,42 @@ class FederationServer(FederationBase):
         # impose a limit to avoid going too crazy with ram/cpu.
 
         async def process_pdus_for_room(room_id: str):
-            with nested_logging_context(room_id):
-                logger.debug("Processing PDUs for %s", room_id)
-
-                try:
-                    await self.check_server_matches_acl(origin_host, room_id)
-                except AuthError as e:
-                    logger.warning(
-                        "Ignoring PDUs for room %s from banned server", room_id
-                    )
-                    for pdu in pdus_by_room[room_id]:
-                        event_id = pdu.event_id
-                        pdu_results[event_id] = e.error_dict()
-                    return
-
+            logger.debug("Processing PDUs for %s", room_id)
+            try:
+                await self.check_server_matches_acl(origin_host, room_id)
+            except AuthError as e:
+                logger.warning("Ignoring PDUs for room %s from banned server", room_id)
                 for pdu in pdus_by_room[room_id]:
-                    pdu_results[pdu.event_id] = await process_pdu(pdu)
+                    event_id = pdu.event_id
+                    pdu_results[event_id] = e.error_dict()
+                return
 
-        async def process_pdu(pdu: EventBase) -> JsonDict:
-            event_id = pdu.event_id
-            with pdu_process_time.time():
-                with nested_logging_context(event_id):
-                    try:
-                        await self._handle_received_pdu(origin, pdu)
-                        return {}
-                    except FederationError as e:
-                        logger.warning("Error handling PDU %s: %s", event_id, e)
-                        return {"error": str(e)}
-                    except Exception as e:
-                        f = failure.Failure()
-                        logger.error(
-                            "Failed to handle PDU %s",
-                            event_id,
-                            exc_info=(f.type, f.value, f.getTracebackObject()),  # type: ignore
-                        )
-                        return {"error": str(e)}
+            for pdu in pdus_by_room[room_id]:
+                event_id = pdu.event_id
+                with pdu_process_time.time():
+                    with nested_logging_context(event_id):
+                        try:
+                            await self._handle_received_pdu(origin, pdu)
+                            pdu_results[event_id] = {}
+                        except FederationError as e:
+                            logger.warning("Error handling PDU %s: %s", event_id, e)
+                            pdu_results[event_id] = {"error": str(e)}
+                        except Exception as e:
+                            f = failure.Failure()
+                            pdu_results[event_id] = {"error": str(e)}
+                            logger.error(
+                                "Failed to handle PDU %s",
+                                event_id,
+                                exc_info=(f.type, f.value, f.getTracebackObject()),
+                            )
 
         await concurrently_execute(
             process_pdus_for_room, pdus_by_room.keys(), TRANSACTION_CONCURRENCY_LIMIT
         )
 
         if newest_pdu_ts and origin in self._federation_metrics_domains:
-            last_pdu_ts_metric.labels(server_name=origin).set(newest_pdu_ts / 1000)
+            newest_pdu_age = self._clock.time_msec() - newest_pdu_ts
+            last_pdu_age_metric.labels(server_name=origin).set(newest_pdu_age / 1000)
 
         return pdu_results
 
@@ -461,22 +447,18 @@ class FederationServer(FederationBase):
 
     async def _on_state_ids_request_compute(self, room_id, event_id):
         state_ids = await self.handler.get_state_ids_for_pdu(room_id, event_id)
-        auth_chain_ids = await self.store.get_auth_chain_ids(room_id, state_ids)
+        auth_chain_ids = await self.store.get_auth_chain_ids(state_ids)
         return {"pdu_ids": state_ids, "auth_chain_ids": auth_chain_ids}
 
     async def _on_context_state_request_compute(
         self, room_id: str, event_id: str
     ) -> Dict[str, list]:
         if event_id:
-            pdus = await self.handler.get_state_for_pdu(
-                room_id, event_id
-            )  # type: Iterable[EventBase]
+            pdus = await self.handler.get_state_for_pdu(room_id, event_id)
         else:
             pdus = (await self.state.get_current_state(room_id)).values()
 
-        auth_chain = await self.store.get_auth_chain(
-            room_id, [pdu.event_id for pdu in pdus]
-        )
+        auth_chain = await self.store.get_auth_chain([pdu.event_id for pdu in pdus])
 
         return {
             "pdus": [pdu.get_pdu_json() for pdu in pdus],
@@ -727,6 +709,27 @@ class FederationServer(FederationBase):
             if the event was unacceptable for any other reason (eg, too large,
             too many prev_events, couldn't find the prev_events)
         """
+        # check that it's actually being sent from a valid destination to
+        # workaround bug #1753 in 0.18.5 and 0.18.6
+        if origin != get_domain_from_id(pdu.sender):
+            # We continue to accept join events from any server; this is
+            # necessary for the federation join dance to work correctly.
+            # (When we join over federation, the "helper" server is
+            # responsible for sending out the join event, rather than the
+            # origin. See bug #1893. This is also true for some third party
+            # invites).
+            if not (
+                pdu.type == "m.room.member"
+                and pdu.content
+                and pdu.content.get("membership", None)
+                in (Membership.JOIN, Membership.INVITE)
+            ):
+                logger.info(
+                    "Discarding PDU %s from invalid origin %s", pdu.event_id, origin
+                )
+                return
+            else:
+                logger.info("Accepting join PDU %s from %s", pdu.event_id, origin)
 
         # We've already checked that we know the room version by this point
         room_version = await self.store.get_room_version(pdu.room_id)
@@ -859,22 +862,13 @@ class FederationHandlerRegistry:
         self.edu_handlers = (
             {}
         )  # type: Dict[str, Callable[[str, dict], Awaitable[None]]]
-        self.query_handlers = (
-            {}
-        )  # type: Dict[str, Callable[[dict], Awaitable[JsonDict]]]
+        self.query_handlers = {}  # type: Dict[str, Callable[[dict], Awaitable[None]]]
 
         # Map from type to instance names that we should route EDU handling to.
         # We randomly choose one instance from the list to route to for each new
         # EDU received.
         self._edu_type_to_instance = {}  # type: Dict[str, List[str]]
 
-        # A rate limiter for incoming room key requests per origin.
-        self._room_key_request_rate_limiter = Ratelimiter(
-            clock=self.clock,
-            rate_hz=self.config.rc_key_requests.per_second,
-            burst_count=self.config.rc_key_requests.burst_count,
-        )
-
     def register_edu_handler(
         self, edu_type: str, handler: Callable[[str, JsonDict], Awaitable[None]]
     ):
@@ -895,7 +889,7 @@ class FederationHandlerRegistry:
         self.edu_handlers[edu_type] = handler
 
     def register_query_handler(
-        self, query_type: str, handler: Callable[[dict], Awaitable[JsonDict]]
+        self, query_type: str, handler: Callable[[dict], defer.Deferred]
     ):
         """Sets the handler callable that will be used to handle an incoming
         federation query of the given type.
@@ -923,15 +917,7 @@ class FederationHandlerRegistry:
         self._edu_type_to_instance[edu_type] = instance_names
 
     async def on_edu(self, edu_type: str, origin: str, content: dict):
-        if not self.config.use_presence and edu_type == EduTypes.Presence:
-            return
-
-        # If the incoming room key requests from a particular origin are over
-        # the limit, drop them.
-        if (
-            edu_type == EduTypes.RoomKeyRequest
-            and not self._room_key_request_rate_limiter.can_do_action(origin)
-        ):
+        if not self.config.use_presence and edu_type == "m.presence":
             return
 
         # Check if we have a handler on this instance
@@ -968,7 +954,7 @@ class FederationHandlerRegistry:
         # Oh well, let's just log and move on.
         logger.warning("No handler registered for EDU type %s", edu_type)
 
-    async def on_query(self, query_type: str, args: dict) -> JsonDict:
+    async def on_query(self, query_type: str, args: dict):
         handler = self.query_handlers.get(query_type)
         if handler:
             return await handler(args)

synapse/federation/send_queue.py

@@ -31,39 +31,25 @@ Events are replicated via a separate events stream.
 
 import logging
 from collections import namedtuple
-from typing import (
-    TYPE_CHECKING,
-    Dict,
-    Hashable,
-    Iterable,
-    List,
-    Optional,
-    Sized,
-    Tuple,
-    Type,
-)
+from typing import Dict, List, Tuple, Type
 
 from sortedcontainers import SortedDict
 
+from twisted.internet import defer
+
 from synapse.api.presence import UserPresenceState
-from synapse.federation.sender import AbstractFederationSender, FederationSender
 from synapse.metrics import LaterGauge
-from synapse.replication.tcp.streams.federation import FederationStream
-from synapse.types import JsonDict, ReadReceipt, RoomStreamToken
 from synapse.util.metrics import Measure
 
 from .units import Edu
 
-if TYPE_CHECKING:
-    from synapse.server import HomeServer
-
 logger = logging.getLogger(__name__)
 
 
-class FederationRemoteSendQueue(AbstractFederationSender):
+class FederationRemoteSendQueue:
     """A drop in replacement for FederationSender"""
 
-    def __init__(self, hs: "HomeServer"):
+    def __init__(self, hs):
         self.server_name = hs.hostname
         self.clock = hs.get_clock()
         self.notifier = hs.get_notifier()
@@ -72,7 +58,7 @@ class FederationRemoteSendQueue(AbstractFederationSender):
         # We may have multiple federation sender instances, so we need to track
         # their positions separately.
         self._sender_instances = hs.config.worker.federation_shard_config.instances
-        self._sender_positions = {}  # type: Dict[str, int]
+        self._sender_positions = {}
 
         # Pending presence map user_id -> UserPresenceState
         self.presence_map = {}  # type: Dict[str, UserPresenceState]
@@ -85,7 +71,7 @@ class FederationRemoteSendQueue(AbstractFederationSender):
         # Stream position -> (user_id, destinations)
         self.presence_destinations = (
             SortedDict()
-        )  # type: SortedDict[int, Tuple[str, Iterable[str]]]
+        )  # type: SortedDict[int, Tuple[str, List[str]]]
 
         # (destination, key) -> EDU
         self.keyed_edu = {}  # type: Dict[Tuple[str, tuple], Edu]
@@ -108,7 +94,7 @@ class FederationRemoteSendQueue(AbstractFederationSender):
         # we make a new function, so we need to make a new function so the inner
         # lambda binds to the queue rather than to the name of the queue which
         # changes. ARGH.
-        def register(name: str, queue: Sized) -> None:
+        def register(name, queue):
             LaterGauge(
                 "synapse_federation_send_queue_%s_size" % (queue_name,),
                 "",
@@ -129,13 +115,13 @@ class FederationRemoteSendQueue(AbstractFederationSender):
 
         self.clock.looping_call(self._clear_queue, 30 * 1000)
 
-    def _next_pos(self) -> int:
+    def _next_pos(self):
         pos = self.pos
         self.pos += 1
         self.pos_time[self.clock.time_msec()] = pos
         return pos
 
-    def _clear_queue(self) -> None:
+    def _clear_queue(self):
         """Clear the queues for anything older than N minutes"""
 
         FIVE_MINUTES_AGO = 5 * 60 * 1000
@@ -152,7 +138,7 @@ class FederationRemoteSendQueue(AbstractFederationSender):
 
         self._clear_queue_before_pos(position_to_delete)
 
-    def _clear_queue_before_pos(self, position_to_delete: int) -> None:
+    def _clear_queue_before_pos(self, position_to_delete):
         """Clear all the queues from before a given position"""
         with Measure(self.clock, "send_queue._clear"):
             # Delete things out of presence maps
@@ -202,18 +188,13 @@ class FederationRemoteSendQueue(AbstractFederationSender):
             for key in keys[:i]:
                 del self.edus[key]
 
-    def notify_new_events(self, max_token: RoomStreamToken) -> None:
+    def notify_new_events(self, max_token):
         """As per FederationSender"""
-        # This should never get called.
-        raise NotImplementedError()
+        # We don't need to replicate this as it gets sent down a different
+        # stream.
+        pass
 
-    def build_and_send_edu(
-        self,
-        destination: str,
-        edu_type: str,
-        content: JsonDict,
-        key: Optional[Hashable] = None,
-    ) -> None:
+    def build_and_send_edu(self, destination, edu_type, content, key=None):
         """As per FederationSender"""
         if destination == self.server_name:
             logger.info("Not sending EDU to ourselves")
@@ -237,39 +218,38 @@ class FederationRemoteSendQueue(AbstractFederationSender):
 
         self.notifier.on_new_replication_data()
 
-    async def send_read_receipt(self, receipt: ReadReceipt) -> None:
+    def send_read_receipt(self, receipt):
         """As per FederationSender
 
         Args:
-            receipt:
+            receipt (synapse.types.ReadReceipt):
         """
         # nothing to do here: the replication listener will handle it.
+        return defer.succeed(None)
 
-    def send_presence(self, states: List[UserPresenceState]) -> None:
+    def send_presence(self, states):
         """As per FederationSender
 
         Args:
-            states
+            states (list(UserPresenceState))
         """
         pos = self._next_pos()
 
         # We only want to send presence for our own users, so lets always just
         # filter here just in case.
-        local_states = [s for s in states if self.is_mine_id(s.user_id)]
+        local_states = list(filter(lambda s: self.is_mine_id(s.user_id), states))
 
         self.presence_map.update({state.user_id: state for state in local_states})
         self.presence_changed[pos] = [state.user_id for state in local_states]
 
         self.notifier.on_new_replication_data()
 
-    def send_presence_to_destinations(
-        self, states: Iterable[UserPresenceState], destinations: Iterable[str]
-    ) -> None:
+    def send_presence_to_destinations(self, states, destinations):
         """As per FederationSender
 
         Args:
-            states
-            destinations
+            states (list[UserPresenceState])
+            destinations (list[str])
         """
         for state in states:
             pos = self._next_pos()
@@ -278,18 +258,15 @@ class FederationRemoteSendQueue(AbstractFederationSender):
 
         self.notifier.on_new_replication_data()
 
-    def send_device_messages(self, destination: str) -> None:
+    def send_device_messages(self, destination):
         """As per FederationSender"""
         # We don't need to replicate this as it gets sent down a different
         # stream.
 
-    def wake_destination(self, server: str) -> None:
-        pass
-
-    def get_current_token(self) -> int:
+    def get_current_token(self):
         return self.pos - 1
 
-    def federation_ack(self, instance_name: str, token: int) -> None:
+    def federation_ack(self, instance_name, token):
         if self._sender_instances:
             # If we have configured multiple federation sender instances we need
             # to track their positions separately, and only clear the queue up
@@ -527,16 +504,13 @@ ParsedFederationStreamData = namedtuple(
 )
 
 
-def process_rows_for_federation(
-    transaction_queue: FederationSender,
-    rows: List[FederationStream.FederationStreamRow],
-) -> None:
+def process_rows_for_federation(transaction_queue, rows):
     """Parse a list of rows from the federation stream and put them in the
     transaction queue ready for sending to the relevant homeservers.
 
     Args:
-        transaction_queue
-        rows
+        transaction_queue (FederationSender)
+        rows (list(synapse.replication.tcp.streams.federation.FederationStream.FederationStreamRow))
     """
 
     # The federation stream contains a bunch of different types of

synapse/federation/sender/__init__.py

@@ -13,14 +13,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import abc
 import logging
-from typing import TYPE_CHECKING, Dict, Hashable, Iterable, List, Optional, Set, Tuple
+from typing import Dict, Hashable, Iterable, List, Optional, Set, Tuple
 
 from prometheus_client import Counter
 
 from twisted.internet import defer
 
+import synapse
 import synapse.metrics
 from synapse.api.presence import UserPresenceState
 from synapse.events import EventBase
@@ -40,12 +40,9 @@ from synapse.metrics import (
     events_processed_counter,
 )
 from synapse.metrics.background_process_metrics import run_as_background_process
-from synapse.types import JsonDict, ReadReceipt, RoomStreamToken
+from synapse.types import ReadReceipt, RoomStreamToken
 from synapse.util.metrics import Measure, measure_func
 
-if TYPE_CHECKING:
-    from synapse.server import HomeServer
-
 logger = logging.getLogger(__name__)
 
 sent_pdus_destination_dist_count = Counter(
@@ -68,91 +65,8 @@ CATCH_UP_STARTUP_DELAY_SEC = 15
 CATCH_UP_STARTUP_INTERVAL_SEC = 5
 
 
-class AbstractFederationSender(metaclass=abc.ABCMeta):
-    @abc.abstractmethod
-    def notify_new_events(self, max_token: RoomStreamToken) -> None:
-        """This gets called when we have some new events we might want to
-        send out to other servers.
-        """
-        raise NotImplementedError()
-
-    @abc.abstractmethod
-    async def send_read_receipt(self, receipt: ReadReceipt) -> None:
-        """Send a RR to any other servers in the room
-
-        Args:
-            receipt: receipt to be sent
-        """
-        raise NotImplementedError()
-
-    @abc.abstractmethod
-    def send_presence(self, states: List[UserPresenceState]) -> None:
-        """Send the new presence states to the appropriate destinations.
-
-        This actually queues up the presence states ready for sending and
-        triggers a background task to process them and send out the transactions.
-        """
-        raise NotImplementedError()
-
-    @abc.abstractmethod
-    def send_presence_to_destinations(
-        self, states: Iterable[UserPresenceState], destinations: Iterable[str]
-    ) -> None:
-        """Send the given presence states to the given destinations.
-
-        Args:
-            destinations:
-        """
-        raise NotImplementedError()
-
-    @abc.abstractmethod
-    def build_and_send_edu(
-        self,
-        destination: str,
-        edu_type: str,
-        content: JsonDict,
-        key: Optional[Hashable] = None,
-    ) -> None:
-        """Construct an Edu object, and queue it for sending
-
-        Args:
-            destination: name of server to send to
-            edu_type: type of EDU to send
-            content: content of EDU
-            key: clobbering key for this edu
-        """
-        raise NotImplementedError()
-
-    @abc.abstractmethod
-    def send_device_messages(self, destination: str) -> None:
-        raise NotImplementedError()
-
-    @abc.abstractmethod
-    def wake_destination(self, destination: str) -> None:
-        """Called when we want to retry sending transactions to a remote.
-
-        This is mainly useful if the remote server has been down and we think it
-        might have come back.
-        """
-        raise NotImplementedError()
-
-    @abc.abstractmethod
-    def get_current_token(self) -> int:
-        raise NotImplementedError()
-
-    @abc.abstractmethod
-    def federation_ack(self, instance_name: str, token: int) -> None:
-        raise NotImplementedError()
-
-    @abc.abstractmethod
-    async def get_replication_rows(
-        self, instance_name: str, from_token: int, to_token: int, target_row_count: int
-    ) -> Tuple[List[Tuple[int, Tuple]], int, bool]:
-        raise NotImplementedError()
-
-
-class FederationSender(AbstractFederationSender):
-    def __init__(self, hs: "HomeServer"):
+class FederationSender:
+    def __init__(self, hs: "synapse.server.HomeServer"):
         self.hs = hs
         self.server_name = hs.hostname
 
@@ -518,7 +432,7 @@ class FederationSender(AbstractFederationSender):
             queue.flush_read_receipts_for_room(room_id)
 
     @preserve_fn  # the caller should not yield on this
-    async def send_presence(self, states: List[UserPresenceState]) -> None:
+    async def send_presence(self, states: List[UserPresenceState]):
         """Send the new presence states to the appropriate destinations.
 
         This actually queues up the presence states ready for sending and
@@ -560,7 +474,7 @@ class FederationSender(AbstractFederationSender):
             self._processing_pending_presence = False
 
     def send_presence_to_destinations(
-        self, states: Iterable[UserPresenceState], destinations: Iterable[str]
+        self, states: List[UserPresenceState], destinations: List[str]
     ) -> None:
         """Send the given presence states to the given destinations.
         destinations (list[str])
@@ -580,7 +494,7 @@ class FederationSender(AbstractFederationSender):
             self._get_per_destination_queue(destination).send_presence(states)
 
     @measure_func("txnqueue._process_presence")
-    async def _process_presence_inner(self, states: List[UserPresenceState]) -> None:
+    async def _process_presence_inner(self, states: List[UserPresenceState]):
         """Given a list of states populate self.pending_presence_by_dest and
         poke to send a new transaction to each destination
         """
@@ -602,9 +516,9 @@ class FederationSender(AbstractFederationSender):
         self,
         destination: str,
         edu_type: str,
-        content: JsonDict,
+        content: dict,
         key: Optional[Hashable] = None,
-    ) -> None:
+    ):
         """Construct an Edu object, and queue it for sending
 
         Args:
@@ -631,7 +545,7 @@ class FederationSender(AbstractFederationSender):
 
         self.send_edu(edu, key)
 
-    def send_edu(self, edu: Edu, key: Optional[Hashable]) -> None:
+    def send_edu(self, edu: Edu, key: Optional[Hashable]):
         """Queue an EDU for sending
 
         Args:
@@ -649,7 +563,7 @@ class FederationSender(AbstractFederationSender):
         else:
             queue.send_edu(edu)
 
-    def send_device_messages(self, destination: str) -> None:
+    def send_device_messages(self, destination: str):
         if destination == self.server_name:
             logger.warning("Not sending device update to ourselves")
             return
@@ -661,7 +575,7 @@ class FederationSender(AbstractFederationSender):
 
         self._get_per_destination_queue(destination).attempt_new_transaction()
 
-    def wake_destination(self, destination: str) -> None:
+    def wake_destination(self, destination: str):
         """Called when we want to retry sending transactions to a remote.
 
         This is mainly useful if the remote server has been down and we think it
@@ -685,10 +599,6 @@ class FederationSender(AbstractFederationSender):
         # to a worker.
         return 0
 
-    def federation_ack(self, instance_name: str, token: int) -> None:
-        # It is not expected that this gets called on FederationSender.
-        raise NotImplementedError()
-
     @staticmethod
     async def get_replication_rows(
         instance_name: str, from_token: int, to_token: int, target_row_count: int
@@ -697,7 +607,7 @@ class FederationSender(AbstractFederationSender):
         # to a worker.
         return [], 0, False
 
-    async def _wake_destinations_needing_catchup(self) -> None:
+    async def _wake_destinations_needing_catchup(self):
         """
         Wakes up destinations that need catch-up and are not currently being
         backed off from.

synapse/federation/sender/per_destination_queue.py

@@ -15,9 +15,8 @@
 # limitations under the License.
 import datetime
 import logging
-from typing import TYPE_CHECKING, Dict, Hashable, Iterable, List, Optional, Tuple
+from typing import TYPE_CHECKING, Dict, Hashable, Iterable, List, Optional, Tuple, cast
 
-import attr
 from prometheus_client import Counter
 
 from synapse.api.errors import (
@@ -77,7 +76,6 @@ class PerDestinationQueue:
         self._transaction_manager = transaction_manager
         self._instance_name = hs.get_instance_name()
         self._federation_shard_config = hs.config.worker.federation_shard_config
-        self._state = hs.get_state_handler()
 
         self._should_send_on_this_instance = True
         if not self._federation_shard_config.should_handle(
@@ -95,10 +93,6 @@ class PerDestinationQueue:
         self._destination = destination
         self.transmission_loop_running = False
 
-        # Flag to signal to any running transmission loop that there is new data
-        # queued up to be sent.
-        self._new_data_to_send = False
-
         # True whilst we are sending events that the remote homeserver missed
         # because it was unreachable. We start in this state so we can perform
         # catch-up at startup.
@@ -114,7 +108,7 @@ class PerDestinationQueue:
         # destination (we are the only updater so this is safe)
         self._last_successful_stream_ordering = None  # type: Optional[int]
 
-        # a queue of pending PDUs
+        # a list of pending PDUs
         self._pending_pdus = []  # type: List[EventBase]
 
         # XXX this is never actually used: see
@@ -214,10 +208,6 @@ class PerDestinationQueue:
         transaction in the background.
         """
 
-        # Mark that we (may) have new things to send, so that any running
-        # transmission loop will recheck whether there is stuff to send.
-        self._new_data_to_send = True
-
         if self.transmission_loop_running:
             # XXX: this can get stuck on by a never-ending
             # request at which point pending_pdus just keeps growing.
@@ -260,41 +250,125 @@ class PerDestinationQueue:
 
             pending_pdus = []
             while True:
-                self._new_data_to_send = False
+                # We have to keep 2 free slots for presence and rr_edus
+                limit = MAX_EDUS_PER_TRANSACTION - 2
 
-                async with _TransactionQueueManager(self) as (
-                    pending_pdus,
-                    pending_edus,
-                ):
-                    if not pending_pdus and not pending_edus:
-                        logger.debug("TX [%s] Nothing to send", self._destination)
+                device_update_edus, dev_list_id = await self._get_device_update_edus(
+                    limit
+                )
 
-                        # If we've gotten told about new things to send during
-                        # checking for things to send, we try looking again.
-                        # Otherwise new PDUs or EDUs might arrive in the meantime,
-                        # but not get sent because we hold the
-                        # `transmission_loop_running` flag.
-                        if self._new_data_to_send:
-                            continue
-                        else:
-                            return
+                limit -= len(device_update_edus)
 
-                    if pending_pdus:
-                        logger.debug(
-                            "TX [%s] len(pending_pdus_by_dest[dest]) = %d",
-                            self._destination,
-                            len(pending_pdus),
+                (
+                    to_device_edus,
+                    device_stream_id,
+                ) = await self._get_to_device_message_edus(limit)
+
+                pending_edus = device_update_edus + to_device_edus
+
+                # BEGIN CRITICAL SECTION
+                #
+                # In order to avoid a race condition, we need to make sure that
+                # the following code (from popping the queues up to the point
+                # where we decide if we actually have any pending messages) is
+                # atomic - otherwise new PDUs or EDUs might arrive in the
+                # meantime, but not get sent because we hold the
+                # transmission_loop_running flag.
+
+                pending_pdus = self._pending_pdus
+
+                # We can only include at most 50 PDUs per transactions
+                pending_pdus, self._pending_pdus = pending_pdus[:50], pending_pdus[50:]
+
+                pending_edus.extend(self._get_rr_edus(force_flush=False))
+                pending_presence = self._pending_presence
+                self._pending_presence = {}
+                if pending_presence:
+                    pending_edus.append(
+                        Edu(
+                            origin=self._server_name,
+                            destination=self._destination,
+                            edu_type="m.presence",
+                            content={
+                                "push": [
+                                    format_user_presence_state(
+                                        presence, self._clock.time_msec()
+                                    )
+                                    for presence in pending_presence.values()
+                                ]
+                            },
                         )
-
-                    await self._transaction_manager.send_new_transaction(
-                        self._destination, pending_pdus, pending_edus
                     )
 
+                pending_edus.extend(
+                    self._pop_pending_edus(MAX_EDUS_PER_TRANSACTION - len(pending_edus))
+                )
+                while (
+                    len(pending_edus) < MAX_EDUS_PER_TRANSACTION
+                    and self._pending_edus_keyed
+                ):
+                    _, val = self._pending_edus_keyed.popitem()
+                    pending_edus.append(val)
+
+                if pending_pdus:
+                    logger.debug(
+                        "TX [%s] len(pending_pdus_by_dest[dest]) = %d",
+                        self._destination,
+                        len(pending_pdus),
+                    )
+
+                if not pending_pdus and not pending_edus:
+                    logger.debug("TX [%s] Nothing to send", self._destination)
+                    self._last_device_stream_id = device_stream_id
+                    return
+
+                # if we've decided to send a transaction anyway, and we have room, we
+                # may as well send any pending RRs
+                if len(pending_edus) < MAX_EDUS_PER_TRANSACTION:
+                    pending_edus.extend(self._get_rr_edus(force_flush=True))
+
+                # END CRITICAL SECTION
+
+                success = await self._transaction_manager.send_new_transaction(
+                    self._destination, pending_pdus, pending_edus
+                )
+                if success:
                     sent_transactions_counter.inc()
                     sent_edus_counter.inc(len(pending_edus))
                     for edu in pending_edus:
                         sent_edus_by_type.labels(edu.edu_type).inc()
+                    # Remove the acknowledged device messages from the database
+                    # Only bother if we actually sent some device messages
+                    if to_device_edus:
+                        await self._store.delete_device_msgs_for_remote(
+                            self._destination, device_stream_id
+                        )
 
+                    # also mark the device updates as sent
+                    if device_update_edus:
+                        logger.info(
+                            "Marking as sent %r %r", self._destination, dev_list_id
+                        )
+                        await self._store.mark_as_sent_devices_by_remote(
+                            self._destination, dev_list_id
+                        )
+
+                    self._last_device_stream_id = device_stream_id
+                    self._last_device_list_stream_id = dev_list_id
+
+                    if pending_pdus:
+                        # we sent some PDUs and it was successful, so update our
+                        # last_successful_stream_ordering in the destinations table.
+                        final_pdu = pending_pdus[-1]
+                        last_successful_stream_ordering = (
+                            final_pdu.internal_metadata.stream_ordering
+                        )
+                        assert last_successful_stream_ordering
+                        await self._store.set_destination_last_successful_stream_ordering(
+                            self._destination, last_successful_stream_ordering
+                        )
+                else:
+                    break
         except NotRetryingDestination as e:
             logger.debug(
                 "TX [%s] not ready for retry yet (next retry at %s) - "
@@ -327,7 +401,7 @@ class PerDestinationQueue:
                 self._pending_presence = {}
                 self._pending_rrs = {}
 
-                self._start_catching_up()
+            self._start_catching_up()
         except FederationDeniedError as e:
             logger.info(e)
         except HttpResponseException as e:
@@ -338,6 +412,7 @@ class PerDestinationQueue:
                 e,
             )
 
+            self._start_catching_up()
         except RequestSendFailed as e:
             logger.warning(
                 "TX [%s] Failed to send transaction: %s", self._destination, e
@@ -347,12 +422,16 @@ class PerDestinationQueue:
                 logger.info(
                     "Failed to send event %s to %s", p.event_id, self._destination
                 )
+
+            self._start_catching_up()
         except Exception:
             logger.exception("TX [%s] Failed to send transaction", self._destination)
             for p in pending_pdus:
                 logger.info(
                     "Failed to send event %s to %s", p.event_id, self._destination
                 )
+
+            self._start_catching_up()
         finally:
             # We want to be *very* sure we clear this after we stop processing
             self.transmission_loop_running = False
@@ -416,97 +495,25 @@ class PerDestinationQueue:
                     "This should not happen." % event_ids
                 )
 
-            # We send transactions with events from one room only, as its likely
-            # that the remote will have to do additional processing, which may
-            # take some time. It's better to give it small amounts of work
-            # rather than risk the request timing out and repeatedly being
-            # retried, and not making any progress.
-            #
-            # Note: `catchup_pdus` will have exactly one PDU per room.
-            for pdu in catchup_pdus:
-                # The PDU from the DB will be the last PDU in the room from
-                # *this server* that wasn't sent to the remote. However, other
-                # servers may have sent lots of events since then, and we want
-                # to try and tell the remote only about the *latest* events in
-                # the room. This is so that it doesn't get inundated by events
-                # from various parts of the DAG, which all need to be processed.
-                #
-                # Note: this does mean that in large rooms a server coming back
-                # online will get sent the same events from all the different
-                # servers, but the remote will correctly deduplicate them and
-                # handle it only once.
+            if logger.isEnabledFor(logging.INFO):
+                rooms = [p.room_id for p in catchup_pdus]
+                logger.info("Catching up rooms to %s: %r", self._destination, rooms)
 
-                # Step 1, fetch the current extremities
-                extrems = await self._store.get_prev_events_for_room(pdu.room_id)
+            success = await self._transaction_manager.send_new_transaction(
+                self._destination, catchup_pdus, []
+            )
 
-                if pdu.event_id in extrems:
-                    # If the event is in the extremities, then great! We can just
-                    # use that without having to do further checks.
-                    room_catchup_pdus = [pdu]
-                else:
-                    # If not, fetch the extremities and figure out which we can
-                    # send.
-                    extrem_events = await self._store.get_events_as_list(extrems)
+            if not success:
+                return
 
-                    new_pdus = []
-                    for p in extrem_events:
-                        # We pulled this from the DB, so it'll be non-null
-                        assert p.internal_metadata.stream_ordering
-
-                        # Filter out events that happened before the remote went
-                        # offline
-                        if (
-                            p.internal_metadata.stream_ordering
-                            < self._last_successful_stream_ordering
-                        ):
-                            continue
-
-                        # Filter out events where the server is not in the room,
-                        # e.g. it may have left/been kicked. *Ideally* we'd pull
-                        # out the kick and send that, but it's a rare edge case
-                        # so we don't bother for now (the server that sent the
-                        # kick should send it out if its online).
-                        hosts = await self._state.get_hosts_in_room_at_events(
-                            p.room_id, [p.event_id]
-                        )
-                        if self._destination not in hosts:
-                            continue
-
-                        new_pdus.append(p)
-
-                    # If we've filtered out all the extremities, fall back to
-                    # sending the original event. This should ensure that the
-                    # server gets at least some of missed events (especially if
-                    # the other sending servers are up).
-                    if new_pdus:
-                        room_catchup_pdus = new_pdus
-                    else:
-                        room_catchup_pdus = [pdu]
-
-                logger.info(
-                    "Catching up rooms to %s: %r", self._destination, pdu.room_id
-                )
-
-                await self._transaction_manager.send_new_transaction(
-                    self._destination, room_catchup_pdus, []
-                )
-
-                sent_transactions_counter.inc()
-
-                # We pulled this from the DB, so it'll be non-null
-                assert pdu.internal_metadata.stream_ordering
-
-                # Note that we mark the last successful stream ordering as that
-                # from the *original* PDU, rather than the PDU(s) we actually
-                # send. This is because we use it to mark our position in the
-                # queue of missed PDUs to process.
-                self._last_successful_stream_ordering = (
-                    pdu.internal_metadata.stream_ordering
-                )
-
-                await self._store.set_destination_last_successful_stream_ordering(
-                    self._destination, self._last_successful_stream_ordering
-                )
+            sent_transactions_counter.inc()
+            final_pdu = catchup_pdus[-1]
+            self._last_successful_stream_ordering = cast(
+                int, final_pdu.internal_metadata.stream_ordering
+            )
+            await self._store.set_destination_last_successful_stream_ordering(
+                self._destination, self._last_successful_stream_ordering
+            )
 
     def _get_rr_edus(self, force_flush: bool) -> Iterable[Edu]:
         if not self._pending_rrs:
@@ -577,135 +584,3 @@ class PerDestinationQueue:
         """
         self._catching_up = True
         self._pending_pdus = []
-
-
-@attr.s(slots=True)
-class _TransactionQueueManager:
-    """A helper async context manager for pulling stuff off the queues and
-    tracking what was last successfully sent, etc.
-    """
-
-    queue = attr.ib(type=PerDestinationQueue)
-
-    _device_stream_id = attr.ib(type=Optional[int], default=None)
-    _device_list_id = attr.ib(type=Optional[int], default=None)
-    _last_stream_ordering = attr.ib(type=Optional[int], default=None)
-    _pdus = attr.ib(type=List[EventBase], factory=list)
-
-    async def __aenter__(self) -> Tuple[List[EventBase], List[Edu]]:
-        # First we calculate the EDUs we want to send, if any.
-
-        # We start by fetching device related EDUs, i.e device updates and to
-        # device messages. We have to keep 2 free slots for presence and rr_edus.
-        limit = MAX_EDUS_PER_TRANSACTION - 2
-
-        device_update_edus, dev_list_id = await self.queue._get_device_update_edus(
-            limit
-        )
-
-        if device_update_edus:
-            self._device_list_id = dev_list_id
-        else:
-            self.queue._last_device_list_stream_id = dev_list_id
-
-        limit -= len(device_update_edus)
-
-        (
-            to_device_edus,
-            device_stream_id,
-        ) = await self.queue._get_to_device_message_edus(limit)
-
-        if to_device_edus:
-            self._device_stream_id = device_stream_id
-        else:
-            self.queue._last_device_stream_id = device_stream_id
-
-        pending_edus = device_update_edus + to_device_edus
-
-        # Now add the read receipt EDU.
-        pending_edus.extend(self.queue._get_rr_edus(force_flush=False))
-
-        # And presence EDU.
-        if self.queue._pending_presence:
-            pending_edus.append(
-                Edu(
-                    origin=self.queue._server_name,
-                    destination=self.queue._destination,
-                    edu_type="m.presence",
-                    content={
-                        "push": [
-                            format_user_presence_state(
-                                presence, self.queue._clock.time_msec()
-                            )
-                            for presence in self.queue._pending_presence.values()
-                        ]
-                    },
-                )
-            )
-            self.queue._pending_presence = {}
-
-        # Finally add any other types of EDUs if there is room.
-        pending_edus.extend(
-            self.queue._pop_pending_edus(MAX_EDUS_PER_TRANSACTION - len(pending_edus))
-        )
-        while (
-            len(pending_edus) < MAX_EDUS_PER_TRANSACTION
-            and self.queue._pending_edus_keyed
-        ):
-            _, val = self.queue._pending_edus_keyed.popitem()
-            pending_edus.append(val)
-
-        # Now we look for any PDUs to send, by getting up to 50 PDUs from the
-        # queue
-        self._pdus = self.queue._pending_pdus[:50]
-
-        if not self._pdus and not pending_edus:
-            return [], []
-
-        # if we've decided to send a transaction anyway, and we have room, we
-        # may as well send any pending RRs
-        if len(pending_edus) < MAX_EDUS_PER_TRANSACTION:
-            pending_edus.extend(self.queue._get_rr_edus(force_flush=True))
-
-        if self._pdus:
-            self._last_stream_ordering = self._pdus[
-                -1
-            ].internal_metadata.stream_ordering
-            assert self._last_stream_ordering
-
-        return self._pdus, pending_edus
-
-    async def __aexit__(self, exc_type, exc, tb):
-        if exc_type is not None:
-            # Failed to send transaction, so we bail out.
-            return
-
-        # Successfully sent transactions, so we remove pending PDUs from the queue
-        if self._pdus:
-            self.queue._pending_pdus = self.queue._pending_pdus[len(self._pdus) :]
-
-        # Succeeded to send the transaction so we record where we have sent up
-        # to in the various streams
-
-        if self._device_stream_id:
-            await self.queue._store.delete_device_msgs_for_remote(
-                self.queue._destination, self._device_stream_id
-            )
-            self.queue._last_device_stream_id = self._device_stream_id
-
-        # also mark the device updates as sent
-        if self._device_list_id:
-            logger.info(
-                "Marking as sent %r %r", self.queue._destination, self._device_list_id
-            )
-            await self.queue._store.mark_as_sent_devices_by_remote(
-                self.queue._destination, self._device_list_id
-            )
-            self.queue._last_device_list_stream_id = self._device_list_id
-
-        if self._last_stream_ordering:
-            # we sent some PDUs and it was successful, so update our
-            # last_successful_stream_ordering in the destinations table.
-            await self.queue._store.set_destination_last_successful_stream_ordering(
-                self.queue._destination, self._last_stream_ordering
-            )

synapse/federation/sender/transaction_manager.py

@@ -36,9 +36,9 @@ if TYPE_CHECKING:
 
 logger = logging.getLogger(__name__)
 
-last_pdu_ts_metric = Gauge(
-    "synapse_federation_last_sent_pdu_time",
-    "The timestamp of the last PDU which was successfully sent to the given domain",
+last_pdu_age_metric = Gauge(
+    "synapse_federation_last_sent_pdu_age",
+    "The age (in seconds) of the last PDU successfully sent to the given domain",
     labelnames=("server_name",),
 )
 
@@ -69,12 +69,15 @@ class TransactionManager:
         destination: str,
         pdus: List[EventBase],
         edus: List[Edu],
-    ) -> None:
+    ) -> bool:
         """
         Args:
             destination: The destination to send to (e.g. 'example.org')
             pdus: In-order list of PDUs to send
             edus: List of EDUs to send
+
+        Returns:
+            True iff the transaction was successful
         """
 
         # Make a transaction-sending opentracing span. This span follows on from
@@ -93,6 +96,8 @@ class TransactionManager:
                 edu.strip_context()
 
         with start_active_span_follows_from("send_transaction", span_contexts):
+            success = True
+
             logger.debug("TX [%s] _attempt_new_transaction", destination)
 
             txn_id = str(self._next_txn_id)
@@ -147,29 +152,45 @@ class TransactionManager:
                 response = await self._transport_layer.send_transaction(
                     transaction, json_data_cb
                 )
+                code = 200
             except HttpResponseException as e:
                 code = e.code
                 response = e.response
 
-                set_tag(tags.ERROR, True)
+                if e.code in (401, 404, 429) or 500 <= e.code:
+                    logger.info(
+                        "TX [%s] {%s} got %d response", destination, txn_id, code
+                    )
+                    raise e
 
-                logger.info("TX [%s] {%s} got %d response", destination, txn_id, code)
-                raise
+            logger.info("TX [%s] {%s} got %d response", destination, txn_id, code)
 
-            logger.info("TX [%s] {%s} got 200 response", destination, txn_id)
-
-            for e_id, r in response.get("pdus", {}).items():
-                if "error" in r:
+            if code == 200:
+                for e_id, r in response.get("pdus", {}).items():
+                    if "error" in r:
+                        logger.warning(
+                            "TX [%s] {%s} Remote returned error for %s: %s",
+                            destination,
+                            txn_id,
+                            e_id,
+                            r,
+                        )
+            else:
+                for p in pdus:
                     logger.warning(
-                        "TX [%s] {%s} Remote returned error for %s: %s",
+                        "TX [%s] {%s} Failed to send event %s",
                         destination,
                         txn_id,
-                        e_id,
-                        r,
+                        p.event_id,
                     )
+                success = False
 
-            if pdus and destination in self._federation_metrics_domains:
+            if success and pdus and destination in self._federation_metrics_domains:
                 last_pdu = pdus[-1]
-                last_pdu_ts_metric.labels(server_name=destination).set(
-                    last_pdu.origin_server_ts / 1000
+                last_pdu_age = self.clock.time_msec() - last_pdu.origin_server_ts
+                last_pdu_age_metric.labels(server_name=destination).set(
+                    last_pdu_age / 1000
                 )
+
+            set_tag(tags.ERROR, not success)
+            return success

synapse/federation/transport/client.py

@@ -16,7 +16,7 @@
 
 import logging
 import urllib
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, Optional
 
 from synapse.api.constants import Membership
 from synapse.api.errors import Codes, HttpResponseException, SynapseError
@@ -26,7 +26,6 @@ from synapse.api.urls import (
     FEDERATION_V2_PREFIX,
 )
 from synapse.logging.utils import log_function
-from synapse.types import JsonDict
 
 logger = logging.getLogger(__name__)
 
@@ -979,38 +978,6 @@ class TransportLayerClient:
 
         return self.client.get_json(destination=destination, path=path)
 
-    async def get_space_summary(
-        self,
-        destination: str,
-        room_id: str,
-        suggested_only: bool,
-        max_rooms_per_space: Optional[int],
-        exclude_rooms: List[str],
-    ) -> JsonDict:
-        """
-        Args:
-            destination: The remote server
-            room_id: The room ID to ask about.
-            suggested_only: if True, only suggested rooms will be returned
-            max_rooms_per_space: an optional limit to the number of children to be
-               returned per space
-            exclude_rooms: a list of any rooms we can skip
-        """
-        path = _create_path(
-            FEDERATION_UNSTABLE_PREFIX, "/org.matrix.msc2946/spaces/%s", room_id
-        )
-
-        params = {
-            "suggested_only": suggested_only,
-            "exclude_rooms": exclude_rooms,
-        }
-        if max_rooms_per_space is not None:
-            params["max_rooms_per_space"] = max_rooms_per_space
-
-        return await self.client.post_json(
-            destination=destination, path=path, data=params
-        )
-
 
 def _create_path(federation_prefix, path, *args):
     """

synapse/federation/transport/server.py

@@ -18,7 +18,7 @@
 import functools
 import logging
 import re
-from typing import Container, Mapping, Optional, Sequence, Tuple, Type
+from typing import Optional, Tuple, Type
 
 import synapse
 from synapse.api.constants import MAX_GROUP_CATEGORYID_LENGTH, MAX_GROUP_ROLEID_LENGTH
@@ -29,7 +29,7 @@ from synapse.api.urls import (
     FEDERATION_V1_PREFIX,
     FEDERATION_V2_PREFIX,
 )
-from synapse.http.server import HttpServer, JsonResource
+from synapse.http.server import JsonResource
 from synapse.http.servlet import (
     parse_boolean_from_args,
     parse_integer_from_args,
@@ -44,8 +44,7 @@ from synapse.logging.opentracing import (
     whitelisted_homeserver,
 )
 from synapse.server import HomeServer
-from synapse.types import JsonDict, ThirdPartyInstanceID, get_domain_from_id
-from synapse.util.ratelimitutils import FederationRateLimiter
+from synapse.types import ThirdPartyInstanceID, get_domain_from_id
 from synapse.util.stringutils import parse_and_validate_server_name
 from synapse.util.versionstring import get_version_string
 
@@ -485,9 +484,10 @@ class FederationQueryServlet(BaseFederationServlet):
 
     # This is when we receive a server-server Query
     async def on_GET(self, origin, content, query, query_type):
-        args = {k.decode("utf8"): v[0].decode("utf-8") for k, v in query.items()}
-        args["origin"] = origin
-        return await self.handler.on_query_request(query_type, args)
+        return await self.handler.on_query_request(
+            query_type,
+            {k.decode("utf8"): v[0].decode("utf-8") for k, v in query.items()},
+        )
 
 
 class FederationMakeJoinServlet(BaseFederationServlet):
@@ -1377,40 +1377,6 @@ class FederationGroupsSettingJoinPolicyServlet(BaseFederationServlet):
         return 200, new_content
 
 
-class FederationSpaceSummaryServlet(BaseFederationServlet):
-    PREFIX = FEDERATION_UNSTABLE_PREFIX + "/org.matrix.msc2946"
-    PATH = "/spaces/(?P<room_id>[^/]*)"
-
-    async def on_POST(
-        self,
-        origin: str,
-        content: JsonDict,
-        query: Mapping[bytes, Sequence[bytes]],
-        room_id: str,
-    ) -> Tuple[int, JsonDict]:
-        suggested_only = content.get("suggested_only", False)
-        if not isinstance(suggested_only, bool):
-            raise SynapseError(
-                400, "'suggested_only' must be a boolean", Codes.BAD_JSON
-            )
-
-        exclude_rooms = content.get("exclude_rooms", [])
-        if not isinstance(exclude_rooms, list) or any(
-            not isinstance(x, str) for x in exclude_rooms
-        ):
-            raise SynapseError(400, "bad value for 'exclude_rooms'", Codes.BAD_JSON)
-
-        max_rooms_per_space = content.get("max_rooms_per_space")
-        if max_rooms_per_space is not None and not isinstance(max_rooms_per_space, int):
-            raise SynapseError(
-                400, "bad value for 'max_rooms_per_space'", Codes.BAD_JSON
-            )
-
-        return 200, await self.handler.federation_space_summary(
-            room_id, suggested_only, max_rooms_per_space, exclude_rooms
-        )
-
-
 class RoomComplexityServlet(BaseFederationServlet):
     """
     Indicates to other servers how complex (and therefore likely
@@ -1509,24 +1475,18 @@ DEFAULT_SERVLET_GROUPS = (
 )
 
 
-def register_servlets(
-    hs: HomeServer,
-    resource: HttpServer,
-    authenticator: Authenticator,
-    ratelimiter: FederationRateLimiter,
-    servlet_groups: Optional[Container[str]] = None,
-):
+def register_servlets(hs, resource, authenticator, ratelimiter, servlet_groups=None):
     """Initialize and register servlet classes.
 
     Will by default register all servlets. For custom behaviour, pass in
     a list of servlet_groups to register.
 
     Args:
-        hs: homeserver
-        resource: resource class to register to
-        authenticator: authenticator to use
-        ratelimiter: ratelimiter to use
-        servlet_groups: List of servlet groups to register.
+        hs (synapse.server.HomeServer): homeserver
+        resource (JsonResource): resource class to register to
+        authenticator (Authenticator): authenticator to use
+        ratelimiter (util.ratelimitutils.FederationRateLimiter): ratelimiter to use
+        servlet_groups (list[str], optional): List of servlet groups to register.
             Defaults to ``DEFAULT_SERVLET_GROUPS``.
     """
     if not servlet_groups:
@@ -1541,14 +1501,6 @@ def register_servlets(
                 server_name=hs.hostname,
             ).register(resource)
 
-        if hs.config.experimental.spaces_enabled:
-            FederationSpaceSummaryServlet(
-                handler=hs.get_space_summary_handler(),
-                authenticator=authenticator,
-                ratelimiter=ratelimiter,
-                server_name=hs.hostname,
-            ).register(resource)
-
     if "openid" in servlet_groups:
         for servletclass in OPENID_SERVLET_CLASSES:
             servletclass(

synapse/groups/attestations.py

@@ -46,7 +46,7 @@ from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.types import JsonDict, get_domain_from_id
 
 if TYPE_CHECKING:
-    from synapse.server import HomeServer
+    from synapse.app.homeserver import HomeServer
 
 logger = logging.getLogger(__name__)
 

synapse/groups/groups_server.py

@@ -25,7 +25,7 @@ from synapse.types import GroupID, JsonDict, RoomID, UserID, get_domain_from_id
 from synapse.util.async_helpers import concurrently_execute
 
 if TYPE_CHECKING:
-    from synapse.server import HomeServer
+    from synapse.app.homeserver import HomeServer
 
 logger = logging.getLogger(__name__)
 

synapse/handlers/_base.py

@@ -24,7 +24,7 @@ from synapse.api.ratelimiting import Ratelimiter
 from synapse.types import UserID
 
 if TYPE_CHECKING:
-    from synapse.server import HomeServer
+    from synapse.app.homeserver import HomeServer
 
 logger = logging.getLogger(__name__)
 

synapse/handlers/account_data.py

@@ -25,7 +25,7 @@ from synapse.replication.http.account_data import (
 from synapse.types import JsonDict, UserID
 
 if TYPE_CHECKING:
-    from synapse.server import HomeServer
+    from synapse.app.homeserver import HomeServer
 
 
 class AccountDataHandler:

synapse/handlers/account_validity.py

@@ -27,7 +27,7 @@ from synapse.types import UserID
 from synapse.util import stringutils
 
 if TYPE_CHECKING:
-    from synapse.server import HomeServer
+    from synapse.app.homeserver import HomeServer
 
 logger = logging.getLogger(__name__)
 

synapse/handlers/acme.py

@@ -24,7 +24,7 @@ from twisted.web.resource import Resource
 from synapse.app import check_bind_error
 
 if TYPE_CHECKING:
-    from synapse.server import HomeServer
+    from synapse.app.homeserver import HomeServer
 
 logger = logging.getLogger(__name__)
 
@@ -73,9 +73,7 @@ class AcmeHandler:
                 "Listening for ACME requests on %s:%i", host, self.hs.config.acme_port
             )
             try:
-                self.reactor.listenTCP(
-                    self.hs.config.acme_port, srv, backlog=50, interface=host
-                )
+                self.reactor.listenTCP(self.hs.config.acme_port, srv, interface=host)
             except twisted.internet.error.CannotListenError as e:
                 check_bind_error(e, host, bind_addresses)
 

synapse/handlers/admin.py

@@ -25,7 +25,7 @@ from synapse.visibility import filter_events_for_client
 from ._base import BaseHandler
 
 if TYPE_CHECKING:
-    from synapse.server import HomeServer
+    from synapse.app.homeserver import HomeServer
 
 logger = logging.getLogger(__name__)
 

synapse/handlers/appservice.py

@@ -38,7 +38,7 @@ from synapse.types import Collection, JsonDict, RoomAlias, RoomStreamToken, User
 from synapse.util.metrics import Measure
 
 if TYPE_CHECKING:
-    from synapse.server import HomeServer
+    from synapse.app.homeserver import HomeServer
 
 logger = logging.getLogger(__name__)
 

synapse/handlers/auth.py

@@ -36,7 +36,7 @@ import attr
 import bcrypt
 import pymacaroons
 
-from twisted.web.server import Request
+from twisted.web.http import Request
 
 from synapse.api.constants import LoginType
 from synapse.api.errors import (
@@ -65,12 +65,11 @@ from synapse.storage.roommember import ProfileInfo
 from synapse.types import JsonDict, Requester, UserID
 from synapse.util import stringutils as stringutils
 from synapse.util.async_helpers import maybe_awaitable
-from synapse.util.macaroons import get_value_from_macaroon, satisfy_expiry
 from synapse.util.msisdn import phone_number_to_msisdn
 from synapse.util.threepids import canonicalise_email
 
 if TYPE_CHECKING:
-    from synapse.server import HomeServer
+    from synapse.app.homeserver import HomeServer
 
 logger = logging.getLogger(__name__)
 
@@ -171,16 +170,6 @@ class SsoLoginExtraAttributes:
     extra_attributes = attr.ib(type=JsonDict)
 
 
-@attr.s(slots=True, frozen=True)
-class LoginTokenAttributes:
-    """Data we store in a short-term login token"""
-
-    user_id = attr.ib(type=str)
-
-    # the SSO Identity Provider that the user authenticated with, to get this token
-    auth_provider_id = attr.ib(type=str)
-
-
 class AuthHandler(BaseHandler):
     SESSION_EXPIRE_MS = 48 * 60 * 60 * 1000
 
@@ -337,8 +326,7 @@ class AuthHandler(BaseHandler):
                 user is too high to proceed
 
         """
-        if not requester.access_token_id:
-            raise ValueError("Cannot validate a user without an access token")
+
         if self._ui_auth_session_timeout:
             last_validated = await self.store.get_access_token_last_validated(
                 requester.access_token_id
@@ -493,7 +481,7 @@ class AuthHandler(BaseHandler):
             sid = authdict["session"]
 
         # Convert the URI and method to strings.
-        uri = request.uri.decode("utf-8")  # type: ignore
+        uri = request.uri.decode("utf-8")
         method = request.method.decode("utf-8")
 
         # If there's no session ID, create a new session.
@@ -886,19 +874,6 @@ class AuthHandler(BaseHandler):
             )
         return result
 
-    def can_change_password(self) -> bool:
-        """Get whether users on this server are allowed to change or set a password.
-
-        Both `config.password_enabled` and `config.password_localdb_enabled` must be true.
-
-        Note that any account (even SSO accounts) are allowed to add passwords if the above
-        is true.
-
-        Returns:
-            Whether users on this server are allowed to change or set a password
-        """
-        return self._password_enabled and self._password_localdb_enabled
-
     def get_supported_login_types(self) -> Iterable[str]:
         """Get a the login types supported for the /login API
 
@@ -1189,16 +1164,18 @@ class AuthHandler(BaseHandler):
             return None
         return user_id
 
-    async def validate_short_term_login_token(
-        self, login_token: str
-    ) -> LoginTokenAttributes:
+    async def validate_short_term_login_token_and_get_user_id(self, login_token: str):
+        auth_api = self.hs.get_auth()
+        user_id = None
         try:
-            res = self.macaroon_gen.verify_short_term_login_token(login_token)
+            macaroon = pymacaroons.Macaroon.deserialize(login_token)
+            user_id = auth_api.get_user_id_from_macaroon(macaroon)
+            auth_api.validate_macaroon(macaroon, "login", user_id)
         except Exception:
             raise AuthError(403, "Invalid token", errcode=Codes.FORBIDDEN)
 
-        await self.auth.check_auth_blocking(res.user_id)
-        return res
+        await self.auth.check_auth_blocking(user_id)
+        return user_id
 
     async def delete_access_token(self, access_token: str):
         """Invalidate a single access token
@@ -1227,7 +1204,7 @@ class AuthHandler(BaseHandler):
     async def delete_access_tokens_for_user(
         self,
         user_id: str,
-        except_token_id: Optional[int] = None,
+        except_token_id: Optional[str] = None,
         device_id: Optional[str] = None,
     ):
         """Invalidate access tokens belonging to a user
@@ -1420,7 +1397,6 @@ class AuthHandler(BaseHandler):
     async def complete_sso_login(
         self,
         registered_user_id: str,
-        auth_provider_id: str,
         request: Request,
         client_redirect_url: str,
         extra_attributes: Optional[JsonDict] = None,
@@ -1430,9 +1406,6 @@ class AuthHandler(BaseHandler):
 
         Args:
             registered_user_id: The registered user ID to complete SSO login for.
-            auth_provider_id: The id of the SSO Identity provider that was used for
-                login. This will be stored in the login token for future tracking in
-                prometheus metrics.
             request: The request to complete.
             client_redirect_url: The URL to which to redirect the user at the end of the
                 process.
@@ -1454,7 +1427,6 @@ class AuthHandler(BaseHandler):
 
         self._complete_sso_login(
             registered_user_id,
-            auth_provider_id,
             request,
             client_redirect_url,
             extra_attributes,
@@ -1465,7 +1437,6 @@ class AuthHandler(BaseHandler):
     def _complete_sso_login(
         self,
         registered_user_id: str,
-        auth_provider_id: str,
         request: Request,
         client_redirect_url: str,
         extra_attributes: Optional[JsonDict] = None,
@@ -1492,7 +1463,7 @@ class AuthHandler(BaseHandler):
 
         # Create a login token
         login_token = self.macaroon_gen.generate_short_term_login_token(
-            registered_user_id, auth_provider_id=auth_provider_id
+            registered_user_id
         )
 
         # Append the login token to the original redirect URL (i.e. with its query
@@ -1598,48 +1569,15 @@ class MacaroonGenerator:
         return macaroon.serialize()
 
     def generate_short_term_login_token(
-        self,
-        user_id: str,
-        auth_provider_id: str,
-        duration_in_ms: int = (2 * 60 * 1000),
+        self, user_id: str, duration_in_ms: int = (2 * 60 * 1000)
     ) -> str:
         macaroon = self._generate_base_macaroon(user_id)
         macaroon.add_first_party_caveat("type = login")
         now = self.hs.get_clock().time_msec()
         expiry = now + duration_in_ms
         macaroon.add_first_party_caveat("time < %d" % (expiry,))
-        macaroon.add_first_party_caveat("auth_provider_id = %s" % (auth_provider_id,))
         return macaroon.serialize()
 
-    def verify_short_term_login_token(self, token: str) -> LoginTokenAttributes:
-        """Verify a short-term-login macaroon
-
-        Checks that the given token is a valid, unexpired short-term-login token
-        minted by this server.
-
-        Args:
-            token: the login token to verify
-
-        Returns:
-            the user_id that this token is valid for
-
-        Raises:
-            MacaroonVerificationFailedException if the verification failed
-        """
-        macaroon = pymacaroons.Macaroon.deserialize(token)
-        user_id = get_value_from_macaroon(macaroon, "user_id")
-        auth_provider_id = get_value_from_macaroon(macaroon, "auth_provider_id")
-
-        v = pymacaroons.Verifier()
-        v.satisfy_exact("gen = 1")
-        v.satisfy_exact("type = login")
-        v.satisfy_general(lambda c: c.startswith("user_id = "))
-        v.satisfy_general(lambda c: c.startswith("auth_provider_id = "))
-        satisfy_expiry(v, self.hs.get_clock().time_msec)
-        v.verify(macaroon, self.hs.config.key.macaroon_secret_key)
-
-        return LoginTokenAttributes(user_id=user_id, auth_provider_id=auth_provider_id)
-
     def generate_delete_pusher_token(self, user_id: str) -> str:
         macaroon = self._generate_base_macaroon(user_id)
         macaroon.add_first_party_caveat("type = delete_pusher")

synapse/handlers/cas_handler.py

@@ -27,7 +27,7 @@ from synapse.http.site import SynapseRequest
 from synapse.types import UserID, map_username_to_mxid_localpart
 
 if TYPE_CHECKING:
-    from synapse.server import HomeServer
+    from synapse.app.homeserver import HomeServer
 
 logger = logging.getLogger(__name__)
 
@@ -83,7 +83,6 @@ class CasHandler:
         # the SsoIdentityProvider protocol type.
         self.idp_icon = None
         self.idp_brand = None
-        self.unstable_idp_brand = None
 
         self._sso_handler = hs.get_sso_handler()