Persited up to command

Wire up token
Reduce usages of RoomStreamToken constructor
2020-09-29 14:45:42 +01:00 · 2020-09-29 14:43:28 +01:00 · 2020-09-29 14:43:28 +01:00 · 2020-09-29 11:06:11 +01:00 · 2020-09-29 10:29:21 +01:00 · 2020-09-28 18:00:30 +01:00
305 changed files with 4886 additions and 1559 deletions
@@ -1,3 +1,89 @@
+Synapse 1.20.1 (2020-09-24)
+===========================
+
+Bugfixes
+--------
+
+- Fix a bug introduced in v1.20.0 which caused the `synapse_port_db` script to fail. ([\#8386](https://github.com/matrix-org/synapse/issues/8386))
+- Fix a bug introduced in v1.20.0 which caused variables to be incorrectly escaped in Jinja2 templates. ([\#8394](https://github.com/matrix-org/synapse/issues/8394))
+
+
+Synapse 1.20.0 (2020-09-22)
+===========================
+
+No significant changes since v1.20.0rc5.
+
+Removal warning
+---------------
+
+Historically, the [Synapse Admin
+API](https://github.com/matrix-org/synapse/tree/master/docs) has been
+accessible under the `/_matrix/client/api/v1/admin`,
+`/_matrix/client/unstable/admin`, `/_matrix/client/r0/admin` and
+`/_synapse/admin` prefixes. In a future release, we will be dropping support
+for accessing Synapse's Admin API using the `/_matrix/client/*` prefixes. This
+makes it easier for homeserver admins to lock down external access to the Admin
+API endpoints.
+
+Synapse 1.20.0rc5 (2020-09-18)
+==============================
+
+In addition to the below, Synapse 1.20.0rc5 also includes the bug fix that was included in 1.19.3.
+
+Features
+--------
+
+- Add flags to the `/versions` endpoint for whether new rooms default to using E2EE. ([\#8343](https://github.com/matrix-org/synapse/issues/8343))
+
+
+Bugfixes
+--------
+
+- Fix rate limiting of federation `/send` requests. ([\#8342](https://github.com/matrix-org/synapse/issues/8342))
+- Fix a longstanding bug where back pagination over federation could get stuck if it failed to handle a received event. ([\#8349](https://github.com/matrix-org/synapse/issues/8349))
+
+
+Internal Changes
+----------------
+
+- Blacklist [MSC2753](https://github.com/matrix-org/matrix-doc/pull/2753) SyTests until it is implemented. ([\#8285](https://github.com/matrix-org/synapse/issues/8285))
+
+
+Synapse 1.19.3 (2020-09-18)
+===========================
+
+Bugfixes
+--------
+
+- Partially mitigate bug where newly joined servers couldn't get past events in a room when there is a malformed event. ([\#8350](https://github.com/matrix-org/synapse/issues/8350))
+
+
+Synapse 1.20.0rc4 (2020-09-16)
+==============================
+
+Synapse 1.20.0rc4 is identical to 1.20.0rc3, with the addition of the security fix that was included in 1.19.2.
+
+
+Synapse 1.19.2 (2020-09-16)
+===========================
+
+Due to the issue below server admins are encouraged to upgrade as soon as possible.
+
+Bugfixes
+--------
+
+- Fix joining rooms over federation that include malformed events. ([\#8324](https://github.com/matrix-org/synapse/issues/8324))
+
+
+Synapse 1.20.0rc3 (2020-09-11)
+==============================
+
+Bugfixes
+--------
+
+- Fix a bug introduced in v1.20.0rc1 where the wrong exception was raised when invalid JSON data is encountered. ([\#8291](https://github.com/matrix-org/synapse/issues/8291))
+
+
 Synapse 1.20.0rc2 (2020-09-09)
 ==============================

@@ -17,9 +17,9 @@ https://help.github.com/articles/using-pull-requests/) to ask us to pull your
 changes into our repo.

 Some other points to follow:
- 
+
 * Please base your changes on the `develop` branch.
-  
+
 * Please follow the [code style requirements](#code-style).

 * Please include a [changelog entry](#changelog) with each PR.
@@ -46,7 +46,7 @@ locally. You'll need python 3.6 or later, and to install a number of tools:

 ```
 # Install the dependencies
-pip install -U black flake8 flake8-comprehensions isort
+pip install -e ".[lint]"

 # Run the linter script
 ./scripts-dev/lint.sh
@@ -1,16 +1,3 @@
-Upgrading to v1.20.0
-====================
-
-Shared rooms endpoint (MSC2666)
-------------------------------
-
-This release contains a new unstable endpoint `/_matrix/client/unstable/uk.half-shot.msc2666/user/shared_rooms/.*`
-for fetching rooms one user has in common with another. This feature requires the
-`update_user_directory` config flag to be `True`. If you are you are using a `synapse.app.user_dir`
-worker, requests to this endpoint must be handled by that worker.
-See `docs/workers.md <docs/workers.md>`_ for more details.
-
-
 Upgrading Synapse
 =================

@@ -127,6 +114,20 @@ request to
 with the query parameters from the original link, presented as a URL-encoded form. See the file
 itself for more details.

+Updated Single Sign-on HTML Templates
+-------------------------------------
+
+The ``saml_error.html`` template was removed from Synapse and replaced with the
+``sso_error.html`` template. If your Synapse is configured to use SAML and a
+custom ``sso_redirect_confirm_template_dir`` configuration then any customisations
+of the ``saml_error.html`` template will need to be merged into the ``sso_error.html``
+template. These templates are similar, but the parameters are slightly different:
+
+* The ``msg`` parameter should be renamed to ``error_description``.
+* There is no longer a ``code`` parameter for the response code.
+* A string ``error`` parameter is available that includes a short hint of why a
+  user is seeing the error page.
+
 Upgrading to v1.18.0
 ====================

@@ -0,0 +1 @@
+Fix tests on distros which disable TLSv1.0. Contributed by @danc86.
@@ -0,0 +1 @@
+Add an admin API `GET /_synapse/admin/v1/event_reports` to read entries of table `event_reports`. Contributed by @dklimpel.
@@ -0,0 +1 @@
+Fix messages over federation being lost until an event is sent into the same room.
@@ -1 +0,0 @@
-Track the latest event for every destination and room for catch-up after federation outage.
@@ -0,0 +1 @@
+Fix messages over federation being lost until an event is sent into the same room.
@@ -1 +0,0 @@
-Track the `stream_ordering` of the last successfully-sent event to every destination, so we can use this information to 'catch up' a remote server after an outage.
@@ -0,0 +1 @@
+Consolidate the SSO error template across all configuration.
@@ -0,0 +1 @@
+Fix messages over federation being lost until an event is sent into the same room.
@@ -1 +0,0 @@
-Track the `stream_ordering` of the last successfully-sent event to every destination, so we can use this information to 'catch up' a remote server after an outage.
@@ -0,0 +1 @@
+Fix messages over federation being lost until an event is sent into the same room.
@@ -0,0 +1 @@
+Change `StreamToken.room_key` to be a `RoomStreamToken` instance.
@@ -1 +0,0 @@
-Blacklist [MSC2753](https://github.com/matrix-org/matrix-doc/pull/2753) SyTests until it is implemented.
@@ -0,0 +1 @@
+Add experimental support for sharding event persister.
@@ -0,0 +1 @@
+Use slotted classes where possible.
@@ -0,0 +1 @@
+Add the room topic and avatar to the room details admin API.
@@ -0,0 +1 @@
+Add an admin API for querying rooms where a user is a member. Contributed by @dklimpel.
@@ -0,0 +1 @@
+Add `uk.half-shot.msc2778.login.application_service` login type to allow appservices to login.
@@ -0,0 +1 @@
+Fix messages over federation being lost until an event is sent into the same room.
@@ -0,0 +1 @@
+Fix fetching events from remote servers that are malformed.
@@ -0,0 +1 @@
+Update outdated usages of `metaclass` to python 3 syntax.
@@ -0,0 +1 @@
+Fix UnboundLocalError from occuring when appservices send malformed register request.
@@ -0,0 +1 @@
+Move lint-related dependencies to package-extra field, update CONTRIBUTING.md to utilise this.
@@ -0,0 +1 @@
+Use the `admin_patterns` helper in additional locations.
@@ -0,0 +1 @@
+Fix test logging to allow braces in log output.
@@ -0,0 +1 @@
+Remove `__future__` imports related to Python 2 compatibility.
@@ -0,0 +1 @@
+Simplify `super()` calls to Python 3 syntax.
@@ -0,0 +1 @@
+Add a configuration option that allows existing users to log in with OpenID Connect. Contributed by @BBBSnowball and @OmmyZhang.
@@ -0,0 +1 @@
+Don't send push notifications to expired user accounts.
@@ -0,0 +1 @@
+Fix bad merge from `release-v1.20.0` branch to `develop`.
@@ -0,0 +1 @@
+Fixed a regression in v1.19.0 with reactivating users through the admin API.
@@ -0,0 +1,2 @@
+Fix a bug where during device registration the length of the device name wasn't
+limited.
@@ -0,0 +1 @@
+Factor out a `_send_dummy_event_for_room` method.
@@ -0,0 +1 @@
+Improve logging of state resolution.
@@ -0,0 +1 @@
+Add type annotations to `SimpleHttpClient`.
@@ -0,0 +1 @@
+Include `guest_access` in the fields that are checked for null bytes when updating `room_stats_state`. Broke in v1.7.2.
@@ -0,0 +1 @@
+Fix theoretical race condition where events are not sent down `/sync` if the synchrotron worker is restarted without restarting other workers.
@@ -0,0 +1 @@
+Add note to the reverse proxy settings documentation about disabling Apache's mod_security2. Contributed by Julian Fietkau (@jfietkau).
@@ -0,0 +1 @@
+Move lint-related dependencies to package-extra field, update CONTRIBUTING.md to utilise this.
@@ -0,0 +1 @@
+Refactor ID generators to use `async with` syntax.
@@ -0,0 +1 @@
+Fix a bug which could cause errors in rooms with malformed membership events, on servers using sqlite.
@@ -0,0 +1 @@
+Fix a bug introduced in v1.20.0 which caused the `synapse_port_db` script to fail.
@@ -0,0 +1 @@
+Add experimental support for sharding event persister.
@@ -0,0 +1 @@
+Add `EventStreamPosition` type.
@@ -0,0 +1 @@
+Add experimental support for sharding event persister.
@@ -0,0 +1 @@
+Fix "Re-starting finished log context" warning when receiving an event we already had over federation.
@@ -0,0 +1 @@
+Create a mechanism for marking tests "logcontext clean".
@@ -0,0 +1 @@
+Fix incorrect handling of timeouts on outgoing HTTP requests.
@@ -0,0 +1 @@
+A pair of tiny cleanups in the federation request code.
@@ -0,0 +1 @@
+Add checks on startup that PostgreSQL sequences are consistent with their associated tables.
@@ -0,0 +1 @@
+Consolidate the SSO error template across all configuration.
@@ -0,0 +1 @@
+Add prometheus metrics for replication requests.
@@ -0,0 +1 @@
+Fix a v1.20.0 regression in the `synapse_port_db` script regarding the `ui_auth_sessions_ips` table.
@@ -15,8 +15,6 @@
 # limitations under the License.

 """ Starts a synapse client console. """
-from __future__ import print_function
-
 import argparse
 import cmd
 import getpass
@@ -13,8 +13,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-from __future__ import print_function
-
 import json
 import urllib
 from pprint import pformat
@@ -1,5 +1,3 @@
-from __future__ import print_function
-
 import argparse
 import cgi
 import datetime
@@ -1,5 +1,3 @@
-from __future__ import print_function
-
 import argparse
 import cgi
 import datetime
@@ -10,8 +10,6 @@ the bridge.
 Requires:
 npm install jquery jsdom
 """
-from __future__ import print_function
-
 import json
 import subprocess
 import time
@@ -1,5 +1,4 @@
 #!/usr/bin/env python
-from __future__ import print_function

 import json
 import sys
@@ -8,11 +7,6 @@ from argparse import ArgumentParser

 import requests

-try:
-    raw_input
-except NameError:  # Python 3
-    raw_input = input
-

 def _mkurl(template, kws):
    for key in kws:
@@ -58,7 +52,7 @@ def main(hs, room_id, access_token, user_id_prefix, why):
    print("The following user IDs will be kicked from %s" % room_name)
    for uid in kick_list:
        print(uid)
-    doit = raw_input("Continue? [Y]es\n")
+    doit = input("Continue? [Y]es\n")
    if len(doit) > 0 and doit.lower() == "y":
        print("Kicking members...")
        # encode them all
@@ -1,8 +1,30 @@
-matrix-synapse-py3 (1.19.0ubuntu1) UNRELEASED; urgency=medium
+matrix-synapse-py3 (1.20.1) stable; urgency=medium

+  * New synapse release 1.20.1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Thu, 24 Sep 2020 16:25:22 +0100
+
+matrix-synapse-py3 (1.20.0) stable; urgency=medium
+
+  [ Synapse Packaging team ]
+  * New synapse release 1.20.0.
+
+  [ Dexter Chua ]
  * Use Type=notify in systemd service

- -- Dexter Chua <dec41@srcf.net>  Wed, 26 Aug 2020 12:41:36 +0000
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 22 Sep 2020 15:19:32 +0100
+
+matrix-synapse-py3 (1.19.3) stable; urgency=medium
+
+  * New synapse release 1.19.3.
+
+ -- Synapse Packaging team <packages@matrix.org>  Fri, 18 Sep 2020 14:59:30 +0100
+
+matrix-synapse-py3 (1.19.2) stable; urgency=medium
+
+  * New synapse release 1.19.2.
+
+ -- Synapse Packaging team <packages@matrix.org>  Wed, 16 Sep 2020 12:50:30 +0100

 matrix-synapse-py3 (1.19.1) stable; urgency=medium

@@ -0,0 +1,129 @@
+Show reported events
+====================
+
+This API returns information about reported events.
+
+The api is::
+
+    GET /_synapse/admin/v1/event_reports?from=0&limit=10
+
+To use it, you will need to authenticate by providing an ``access_token`` for a
+server admin: see `README.rst <README.rst>`_.
+
+It returns a JSON body like the following:
+
+.. code:: jsonc
+
+    {
+        "event_reports": [
+            {
+                "content": {
+                    "reason": "foo",
+                    "score": -100
+                },
+                "event_id": "$bNUFCwGzWca1meCGkjp-zwslF-GfVcXukvRLI1_FaVY",
+                "event_json": {
+                    "auth_events": [
+                        "$YK4arsKKcc0LRoe700pS8DSjOvUT4NDv0HfInlMFw2M",
+                        "$oggsNXxzPFRE3y53SUNd7nsj69-QzKv03a1RucHu-ws"
+                    ],
+                    "content": {
+                        "body": "matrix.org: This Week in Matrix",
+                        "format": "org.matrix.custom.html",
+                        "formatted_body": "<strong>matrix.org</strong>:<br><a href=\"https://matrix.org/blog/\"><strong>This Week in Matrix</strong></a>",
+                        "msgtype": "m.notice"
+                    },
+                    "depth": 546,
+                    "hashes": {
+                        "sha256": "xK1//xnmvHJIOvbgXlkI8eEqdvoMmihVDJ9J4SNlsAw"
+                    },
+                    "origin": "matrix.org",
+                    "origin_server_ts": 1592291711430,
+                    "prev_events": [
+                        "$YK4arsKKcc0LRoe700pS8DSjOvUT4NDv0HfInlMFw2M"
+                    ],
+                    "prev_state": [],
+                    "room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
+                    "sender": "@foobar:matrix.org",
+                    "signatures": {
+                        "matrix.org": {
+                            "ed25519:a_JaEG": "cs+OUKW/iHx5pEidbWxh0UiNNHwe46Ai9LwNz+Ah16aWDNszVIe2gaAcVZfvNsBhakQTew51tlKmL2kspXk/Dg"
+                        }
+                    },
+                    "type": "m.room.message",
+                    "unsigned": {
+                        "age_ts": 1592291711430,
+                    }
+                },
+                "id": 2,
+                "reason": "foo",
+                "received_ts": 1570897107409,
+                "room_alias": "#alias1:matrix.org",
+                "room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
+                "sender": "@foobar:matrix.org",
+                "user_id": "@foo:matrix.org"
+            },
+            {
+                "content": {
+                    "reason": "bar",
+                    "score": -100
+                },
+                "event_id": "$3IcdZsDaN_En-S1DF4EMCy3v4gNRKeOJs8W5qTOKj4I",
+                "event_json": {
+                    // hidden items
+                    // see above
+                },
+                "id": 3,
+                "reason": "bar",
+                "received_ts": 1598889612059,
+                "room_alias": "#alias2:matrix.org",
+                "room_id": "!eGvUQuTCkHGVwNMOjv:matrix.org",
+                "sender": "@foobar:matrix.org",
+                "user_id": "@bar:matrix.org"
+            }
+        ],
+        "next_token": 2,
+        "total": 4
+    }
+
+To paginate, check for ``next_token`` and if present, call the endpoint again
+with ``from`` set to the value of ``next_token``. This will return a new page.
+
+If the endpoint does not return a ``next_token`` then there are no more
+reports to paginate through.
+
+**URL parameters:**
+
+- ``limit``: integer - Is optional but is used for pagination,
+  denoting the maximum number of items to return in this call. Defaults to ``100``.
+- ``from``: integer - Is optional but used for pagination,
+  denoting the offset in the returned results. This should be treated as an opaque value and
+  not explicitly set to anything other than the return value of ``next_token`` from a previous call.
+  Defaults to ``0``.
+- ``dir``: string - Direction of event report order. Whether to fetch the most recent first (``b``) or the
+  oldest first (``f``). Defaults to ``b``.
+- ``user_id``: string - Is optional and filters to only return users with user IDs that contain this value.
+  This is the user who reported the event and wrote the reason.
+- ``room_id``: string - Is optional and filters to only return rooms with room IDs that contain this value.
+
+**Response**
+
+The following fields are returned in the JSON response body:
+
+- ``id``: integer - ID of event report.
+- ``received_ts``: integer - The timestamp (in milliseconds since the unix epoch) when this report was sent.
+- ``room_id``: string - The ID of the room in which the event being reported is located.
+- ``event_id``: string - The ID of the reported event.
+- ``user_id``: string - This is the user who reported the event and wrote the reason.
+- ``reason``: string - Comment made by the ``user_id`` in this report. May be blank.
+- ``content``: object - Content of reported event.
+
+  - ``reason``: string - Comment made by the ``user_id`` in this report. May be blank.
+  - ``score``: integer - Content is reported based upon a negative score, where -100 is "most offensive" and 0 is "inoffensive".
+
+- ``sender``: string - This is the ID of the user who sent the original message/event that was reported.
+- ``room_alias``: string - The alias of the room. ``null`` if the room does not have a canonical alias set.
+- ``event_json``: object - Details of the original event that was reported.
+- ``next_token``: integer - Indication for pagination. See above.
+- ``total``: integer - Total number of event reports related to the query (``user_id`` and ``room_id``).
+
@@ -275,6 +275,8 @@ The following fields are possible in the JSON response body:

 * `room_id` - The ID of the room.
 * `name` - The name of the room.
+* `topic` - The topic of the room.
+* `avatar` - The `mxc` URI to the avatar of the room.
 * `canonical_alias` - The canonical (main) alias address of the room.
 * `joined_members` - How many users are currently in the room.
 * `joined_local_members` - How many local users are currently in the room.
@@ -304,6 +306,8 @@ Response:
 {
  "room_id": "!mscvqgqpHYjBGDxNym:matrix.org",
  "name": "Music Theory",
+  "avatar": "mxc://matrix.org/AQDaVFlbkQoErdOgqWRgiGSV",
+  "topic": "Theory, Composition, Notation, Analysis",
  "canonical_alias": "#musictheory:matrix.org",
  "joined_members": 127
  "joined_local_members": 2,
@@ -304,6 +304,43 @@ To use it, you will need to authenticate by providing an ``access_token`` for a
 server admin: see `README.rst <README.rst>`_.


+List room memberships of an user
+================================
+Gets a list of all ``room_id`` that a specific ``user_id`` is member.
+
+The API is::
+
+  GET /_synapse/admin/v1/users/<user_id>/joined_rooms
+
+To use it, you will need to authenticate by providing an ``access_token`` for a
+server admin: see `README.rst <README.rst>`_.
+
+A response body like the following is returned:
+
+.. code:: json
+
+    {
+        "joined_rooms": [
+            "!DuGcnbhHGaSZQoNQR:matrix.org",
+            "!ZtSaPCawyWtxfWiIy:matrix.org"
+        ],
+        "total": 2
+    }
+
+**Parameters**
+
+The following parameters should be set in the URL:
+
+- ``user_id`` - fully qualified: for example, ``@user:server.com``.
+
+**Response**
+
+The following fields are returned in the JSON response body:
+
+- ``joined_rooms`` - An array of ``room_id``.
+- ``total`` - Number of rooms.
+
+
 User devices
 ============

@@ -106,6 +106,17 @@ Note that the above may fail with an error about duplicate rows if corruption
 has already occurred, and such duplicate rows will need to be manually removed.


+## Fixing inconsistent sequences error
+
+Synapse uses Postgres sequences to generate IDs for various tables. A sequence
+and associated table can get out of sync if, for example, Synapse has been
+downgraded and then upgraded again.
+
+To fix the issue shut down Synapse (including any and all workers) and run the
+SQL command included in the error message. Once done Synapse should start
+successfully.
+
+
 ## Tuning Postgres

 The default settings should be fine for most deployments. For larger
@@ -121,6 +121,14 @@ example.com:8448 {

 **NOTE**: ensure the  `nocanon` options are included.

+**NOTE 2**: It appears that Synapse is currently incompatible with the ModSecurity module for Apache (`mod_security2`). If you need it enabled for other services on your web server, you can disable it for Synapse's two VirtualHosts by including the following lines before each of the two `</VirtualHost>` above:
+
+```
+<IfModule security2_module>
+    SecRuleEngine off
+</IfModule>
+```
+
 ### HAProxy

 ```
@@ -1485,11 +1485,14 @@ trusted_key_servers:
 # At least one of `sp_config` or `config_path` must be set in this section to
 # enable SAML login.
 #
-# (You will probably also want to set the following options to `false` to
+# You will probably also want to set the following options to `false` to
 # disable the regular login/registration flows:
 #   * enable_registration
 #   * password_config.enabled
 #
+# You will also want to investigate the settings under the "sso" configuration
+# section below.
+#
 # Once SAML support is enabled, a metadata file will be exposed at
 # https://<server>:<port>/_matrix/saml2/metadata.xml, which you may be able to
 # use to configure your SAML IdP with. Alternatively, you can manually configure
@@ -1612,31 +1615,6 @@ saml2_config:
  #  - attribute: department
  #    value: "sales"

-  # Directory in which Synapse will try to find the template files below.
-  # If not set, default templates from within the Synapse package will be used.
-  #
-  # DO NOT UNCOMMENT THIS SETTING unless you want to customise the templates.
-  # If you *do* uncomment it, you will need to make sure that all the templates
-  # below are in the directory.
-  #
-  # Synapse will look for the following templates in this directory:
-  #
-  # * HTML page to display to users if something goes wrong during the
-  #   authentication process: 'saml_error.html'.
-  #
-  #   When rendering, this template is given the following variables:
-  #     * code: an HTML error code corresponding to the error that is being
-  #       returned (typically 400 or 500)
-  #
-  #     * msg: a textual message describing the error.
-  #
-  #   The variables will automatically be HTML-escaped.
-  #
-  # You can see the default templates at:
-  # https://github.com/matrix-org/synapse/tree/master/synapse/res/templates
-  #
-  #template_dir: "res/templates"
-

 # OpenID Connect integration. The following settings can be used to make Synapse
 # use an OpenID Connect Provider for authentication, instead of its internal
@@ -1711,6 +1689,11 @@ oidc_config:
  #
  #skip_verification: true

+  # Uncomment to allow a user logging in via OIDC to match a pre-existing account instead
+  # of failing. This could be used if switching from password logins to OIDC. Defaults to false.
+  #
+  #allow_existing_users: true
+
  # An external module can be provided here as a custom solution to mapping
  # attributes returned from a OIDC provider onto a matrix user.
  #
@@ -381,7 +381,6 @@ Handles searches in the user directory. It can handle REST endpoints matching
 the following regular expressions:

    ^/_matrix/client/(api/v1|r0|unstable)/user_directory/search$
-    ^/_matrix/client/unstable/uk.half-shot.msc2666/user/shared_rooms/.*$

 When using this worker you must also set `update_user_directory: False` in the
 shared configuration file to stop the main synapse running background
@@ -46,10 +46,12 @@ files =
  synapse/server_notices,
  synapse/spam_checker_api,
  synapse/state,
+  synapse/storage/databases/main/events.py,
  synapse/storage/databases/main/stream.py,
  synapse/storage/databases/main/ui_auth.py,
  synapse/storage/database.py,
  synapse/storage/engines,
+  synapse/storage/persist_events.py,
  synapse/storage/state.py,
  synapse/storage/util,
  synapse/streams,
@@ -1,7 +1,5 @@
 #! /usr/bin/python

-from __future__ import print_function
-
 import argparse
 import ast
 import os
@@ -13,7 +11,7 @@ import yaml

 class DefinitionVisitor(ast.NodeVisitor):
    def __init__(self):
-        super(DefinitionVisitor, self).__init__()
+        super().__init__()
        self.functions = {}
        self.classes = {}
        self.names = {}
@@ -1,7 +1,5 @@
 #!/usr/bin/env python2

-from __future__ import print_function
-
 import sys

 import pymacaroons
@@ -15,8 +15,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-from __future__ import print_function
-
 import argparse
 import base64
 import json
@@ -323,7 +321,7 @@ class MatrixConnectionAdapter(HTTPAdapter):
        url = urlparse.urlunparse(
            ("https", netloc, parsed.path, parsed.params, parsed.query, parsed.fragment)
        )
-        return super(MatrixConnectionAdapter, self).get_connection(url, proxies)
+        return super().get_connection(url, proxies)


 if __name__ == "__main__":
@@ -1,5 +1,3 @@
-from __future__ import print_function
-
 import sqlite3
 import sys

@@ -32,8 +32,6 @@ To use, pipe the above into::
    PYTHON_PATH=. ./scripts/move_remote_media_to_new_store.py <source repo> <dest repo>
 """

-from __future__ import print_function
-
 import argparse
 import logging
 import os
@@ -14,8 +14,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-from __future__ import print_function
-
 from synapse._scripts.register_new_matrix_user import main

 if __name__ == "__main__":
@@ -89,6 +89,7 @@ BOOLEAN_COLUMNS = {
    "redactions": ["have_censored"],
    "room_stats_state": ["is_federatable"],
    "local_media_repository": ["safe_from_quarantine"],
+    "users": ["shadow_banned"],
 }


@@ -144,6 +145,7 @@ IGNORED_TABLES = {
    # the sessions are transient anyway, so ignore them.
    "ui_auth_sessions",
    "ui_auth_sessions_credentials",
+    "ui_auth_sessions_ips",
 }


@@ -627,6 +629,7 @@ class Porter(object):
            self.progress.set_state("Setting up sequence generators")
            await self._setup_state_group_id_seq()
            await self._setup_user_id_seq()
+            await self._setup_events_stream_seqs()

            self.progress.done()
        except Exception as e:
@@ -803,6 +806,29 @@ class Porter(object):

        return self.postgres_store.db_pool.runInteraction("setup_user_id_seq", r)

+    def _setup_events_stream_seqs(self):
+        def r(txn):
+            txn.execute("SELECT MAX(stream_ordering) FROM events")
+            curr_id = txn.fetchone()[0]
+            if curr_id:
+                next_id = curr_id + 1
+                txn.execute(
+                    "ALTER SEQUENCE events_stream_seq RESTART WITH %s", (next_id,)
+                )
+
+            txn.execute("SELECT -MIN(stream_ordering) FROM events")
+            curr_id = txn.fetchone()[0]
+            if curr_id:
+                next_id = curr_id + 1
+                txn.execute(
+                    "ALTER SEQUENCE events_backfill_stream_seq RESTART WITH %s",
+                    (next_id,),
+                )
+
+        return self.postgres_store.db_pool.runInteraction(
+            "_setup_events_stream_seqs", r
+        )
+

 ##############################################
 # The following is simply UI stuff
@@ -94,6 +94,22 @@ ALL_OPTIONAL_REQUIREMENTS = dependencies["ALL_OPTIONAL_REQUIREMENTS"]
 # Make `pip install matrix-synapse[all]` install all the optional dependencies.
 CONDITIONAL_REQUIREMENTS["all"] = list(ALL_OPTIONAL_REQUIREMENTS)

+# Developer dependencies should not get included in "all".
+#
+# We pin black so that our tests don't start failing on new releases.
+CONDITIONAL_REQUIREMENTS["lint"] = [
+    "isort==5.0.3",
+    "black==19.10b0",
+    "flake8-comprehensions",
+    "flake8",
+]
+
+# Dependencies which are exclusively required by unit test code. This is
+# NOT a list of all modules that are necessary to run the unit tests.
+# Tests assume that all optional dependencies are installed.
+#
+# parameterized_class decorator was introduced in parameterized 0.7.0
+CONDITIONAL_REQUIREMENTS["test"] = ["mock>=2.0", "parameterized>=0.7.0"]

 setup(
    name="matrix-synapse",
@@ -48,7 +48,7 @@ try:
 except ImportError:
    pass

-__version__ = "1.20.0rc2"
+__version__ = "1.20.1"

 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
    # We import here so that we don't have to install a bunch of deps when
@@ -14,8 +14,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-from __future__ import print_function
-
 import argparse
 import getpass
 import hashlib
@@ -218,11 +218,7 @@ class Auth:
            # Deny the request if the user account has expired.
            if self._account_validity.enabled and not allow_expired:
                user_id = user.to_string()
-                expiration_ts = await self.store.get_expiration_ts_for_user(user_id)
-                if (
-                    expiration_ts is not None
-                    and self.clock.time_msec() >= expiration_ts
-                ):
+                if await self.store.is_account_expired(user_id, self.clock.time_msec()):
                    raise AuthError(
                        403, "User account has expired", errcode=Codes.EXPIRED_ACCOUNT
                    )
@@ -87,7 +87,7 @@ class CodeMessageException(RuntimeError):
    """

    def __init__(self, code: Union[int, HTTPStatus], msg: str):
-        super(CodeMessageException, self).__init__("%d: %s" % (code, msg))
+        super().__init__("%d: %s" % (code, msg))

        # Some calls to this method pass instances of http.HTTPStatus for `code`.
        # While HTTPStatus is a subclass of int, it has magic __str__ methods
@@ -138,7 +138,7 @@ class SynapseError(CodeMessageException):
            msg: The human-readable error message.
            errcode: The matrix error code e.g 'M_FORBIDDEN'
        """
-        super(SynapseError, self).__init__(code, msg)
+        super().__init__(code, msg)
        self.errcode = errcode

    def error_dict(self):
@@ -159,7 +159,7 @@ class ProxiedRequestError(SynapseError):
        errcode: str = Codes.UNKNOWN,
        additional_fields: Optional[Dict] = None,
    ):
-        super(ProxiedRequestError, self).__init__(code, msg, errcode)
+        super().__init__(code, msg, errcode)
        if additional_fields is None:
            self._additional_fields = {}  # type: Dict
        else:
@@ -181,7 +181,7 @@ class ConsentNotGivenError(SynapseError):
            msg: The human-readable error message
            consent_url: The URL where the user can give their consent
        """
-        super(ConsentNotGivenError, self).__init__(
+        super().__init__(
            code=HTTPStatus.FORBIDDEN, msg=msg, errcode=Codes.CONSENT_NOT_GIVEN
        )
        self._consent_uri = consent_uri
@@ -201,7 +201,7 @@ class UserDeactivatedError(SynapseError):
        Args:
            msg: The human-readable error message
        """
-        super(UserDeactivatedError, self).__init__(
+        super().__init__(
            code=HTTPStatus.FORBIDDEN, msg=msg, errcode=Codes.USER_DEACTIVATED
        )

@@ -225,7 +225,7 @@ class FederationDeniedError(SynapseError):

        self.destination = destination

-        super(FederationDeniedError, self).__init__(
+        super().__init__(
            code=403,
            msg="Federation denied with %s." % (self.destination,),
            errcode=Codes.FORBIDDEN,
@@ -244,9 +244,7 @@ class InteractiveAuthIncompleteError(Exception):
    """

    def __init__(self, session_id: str, result: "JsonDict"):
-        super(InteractiveAuthIncompleteError, self).__init__(
-            "Interactive auth not yet complete"
-        )
+        super().__init__("Interactive auth not yet complete")
        self.session_id = session_id
        self.result = result

@@ -261,14 +259,14 @@ class UnrecognizedRequestError(SynapseError):
            message = "Unrecognized request"
        else:
            message = args[0]
-        super(UnrecognizedRequestError, self).__init__(400, message, **kwargs)
+        super().__init__(400, message, **kwargs)


 class NotFoundError(SynapseError):
    """An error indicating we can't find the thing you asked for"""

    def __init__(self, msg: str = "Not found", errcode: str = Codes.NOT_FOUND):
-        super(NotFoundError, self).__init__(404, msg, errcode=errcode)
+        super().__init__(404, msg, errcode=errcode)


 class AuthError(SynapseError):
@@ -279,7 +277,7 @@ class AuthError(SynapseError):
    def __init__(self, *args, **kwargs):
        if "errcode" not in kwargs:
            kwargs["errcode"] = Codes.FORBIDDEN
-        super(AuthError, self).__init__(*args, **kwargs)
+        super().__init__(*args, **kwargs)


 class InvalidClientCredentialsError(SynapseError):
@@ -335,7 +333,7 @@ class ResourceLimitError(SynapseError):
    ):
        self.admin_contact = admin_contact
        self.limit_type = limit_type
-        super(ResourceLimitError, self).__init__(code, msg, errcode=errcode)
+        super().__init__(code, msg, errcode=errcode)

    def error_dict(self):
        return cs_error(
@@ -352,7 +350,7 @@ class EventSizeError(SynapseError):
    def __init__(self, *args, **kwargs):
        if "errcode" not in kwargs:
            kwargs["errcode"] = Codes.TOO_LARGE
-        super(EventSizeError, self).__init__(413, *args, **kwargs)
+        super().__init__(413, *args, **kwargs)


 class EventStreamError(SynapseError):
@@ -361,7 +359,7 @@ class EventStreamError(SynapseError):
    def __init__(self, *args, **kwargs):
        if "errcode" not in kwargs:
            kwargs["errcode"] = Codes.BAD_PAGINATION
-        super(EventStreamError, self).__init__(*args, **kwargs)
+        super().__init__(*args, **kwargs)


 class LoginError(SynapseError):
@@ -384,7 +382,7 @@ class InvalidCaptchaError(SynapseError):
        error_url: Optional[str] = None,
        errcode: str = Codes.CAPTCHA_INVALID,
    ):
-        super(InvalidCaptchaError, self).__init__(code, msg, errcode)
+        super().__init__(code, msg, errcode)
        self.error_url = error_url

    def error_dict(self):
@@ -402,7 +400,7 @@ class LimitExceededError(SynapseError):
        retry_after_ms: Optional[int] = None,
        errcode: str = Codes.LIMIT_EXCEEDED,
    ):
-        super(LimitExceededError, self).__init__(code, msg, errcode)
+        super().__init__(code, msg, errcode)
        self.retry_after_ms = retry_after_ms

    def error_dict(self):
@@ -418,9 +416,7 @@ class RoomKeysVersionError(SynapseError):
        Args:
            current_version: the current version of the store they should have used
        """
-        super(RoomKeysVersionError, self).__init__(
-            403, "Wrong room_keys version", Codes.WRONG_ROOM_KEYS_VERSION
-        )
+        super().__init__(403, "Wrong room_keys version", Codes.WRONG_ROOM_KEYS_VERSION)
        self.current_version = current_version


@@ -429,7 +425,7 @@ class UnsupportedRoomVersionError(SynapseError):
    not support."""

    def __init__(self, msg: str = "Homeserver does not support this room version"):
-        super(UnsupportedRoomVersionError, self).__init__(
+        super().__init__(
            code=400, msg=msg, errcode=Codes.UNSUPPORTED_ROOM_VERSION,
        )

@@ -440,7 +436,7 @@ class ThreepidValidationError(SynapseError):
    def __init__(self, *args, **kwargs):
        if "errcode" not in kwargs:
            kwargs["errcode"] = Codes.FORBIDDEN
-        super(ThreepidValidationError, self).__init__(*args, **kwargs)
+        super().__init__(*args, **kwargs)


 class IncompatibleRoomVersionError(SynapseError):
@@ -451,7 +447,7 @@ class IncompatibleRoomVersionError(SynapseError):
    """

    def __init__(self, room_version: str):
-        super(IncompatibleRoomVersionError, self).__init__(
+        super().__init__(
            code=400,
            msg="Your homeserver does not support the features required to "
            "join this room",
@@ -473,7 +469,7 @@ class PasswordRefusedError(SynapseError):
        msg: str = "This password doesn't comply with the server's policy",
        errcode: str = Codes.WEAK_PASSWORD,
    ):
-        super(PasswordRefusedError, self).__init__(
+        super().__init__(
            code=400, msg=msg, errcode=errcode,
        )

@@ -488,7 +484,7 @@ class RequestSendFailed(RuntimeError):
    """

    def __init__(self, inner_exception, can_retry):
-        super(RequestSendFailed, self).__init__(
+        super().__init__(
            "Failed to send request: %s: %s"
            % (type(inner_exception).__name__, inner_exception)
        )
@@ -542,7 +538,7 @@ class FederationError(RuntimeError):
        self.source = source

        msg = "%s %s: %s" % (level, code, reason)
-        super(FederationError, self).__init__(msg)
+        super().__init__(msg)

    def get_dict(self):
        return {
@@ -570,7 +566,7 @@ class HttpResponseException(CodeMessageException):
            msg: reason phrase from HTTP response status line
            response: body of response
        """
-        super(HttpResponseException, self).__init__(code, msg)
+        super().__init__(code, msg)
        self.response = response

    def to_synapse_error(self):
@@ -132,7 +132,7 @@ def matrix_user_id_validator(user_id_str):

 class Filtering:
    def __init__(self, hs):
-        super(Filtering, self).__init__()
+        super().__init__()
        self.store = hs.get_datastore()

    async def get_user_filter(self, user_localpart, filter_id):
@@ -152,7 +152,7 @@ class PresenceStatusStubServlet(RestServlet):
    PATTERNS = client_patterns("/presence/(?P<user_id>[^/]*)/status")

    def __init__(self, hs):
-        super(PresenceStatusStubServlet, self).__init__()
+        super().__init__()
        self.auth = hs.get_auth()

    async def on_GET(self, request, user_id):
@@ -176,7 +176,7 @@ class KeyUploadServlet(RestServlet):
        Args:
            hs (synapse.server.HomeServer): server
        """
-        super(KeyUploadServlet, self).__init__()
+        super().__init__()
        self.auth = hs.get_auth()
        self.store = hs.get_datastore()
        self.http_client = hs.get_simple_http_client()
@@ -646,7 +646,7 @@ class GenericWorkerServer(HomeServer):

 class GenericWorkerReplicationHandler(ReplicationDataHandler):
    def __init__(self, hs):
-        super(GenericWorkerReplicationHandler, self).__init__(hs)
+        super().__init__(hs)

        self.store = hs.get_datastore()
        self.presence_handler = hs.get_presence_handler()  # type: GenericWorkerPresence
@@ -15,8 +15,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-from __future__ import print_function
-
 import gc
 import logging
 import math
@@ -88,7 +88,7 @@ class ApplicationServiceApi(SimpleHttpClient):
    """

    def __init__(self, hs):
-        super(ApplicationServiceApi, self).__init__(hs)
+        super().__init__(hs)
        self.clock = hs.get_clock()

        self.protocol_meta_cache = ResponseCache(
@@ -178,7 +178,7 @@ class ApplicationServiceApi(SimpleHttpClient):
                urllib.parse.quote(protocol),
            )
            try:
-                info = await self.get_json(uri, {})
+                info = await self.get_json(uri)

                if not _is_valid_3pe_metadata(info):
                    logger.warning(
@@ -194,7 +194,10 @@ class Config:
            return file_stream.read()

    def read_templates(
-        self, filenames: List[str], custom_template_directory: Optional[str] = None,
+        self,
+        filenames: List[str],
+        custom_template_directory: Optional[str] = None,
+        autoescape: bool = False,
    ) -> List[jinja2.Template]:
        """Load a list of template files from disk using the given variables.

@@ -210,6 +213,9 @@ class Config:
            custom_template_directory: A directory to try to look for the templates
                before using the default Synapse template directory instead.

+            autoescape: Whether to autoescape variables before inserting them into the
+                template.
+
        Raises:
            ConfigError: if the file's path is incorrect or otherwise cannot be read.

@@ -233,7 +239,7 @@ class Config:
            search_directories.insert(0, custom_template_directory)

        loader = jinja2.FileSystemLoader(search_directories)
-        env = jinja2.Environment(loader=loader, autoescape=True)
+        env = jinja2.Environment(loader=loader, autoescape=autoescape)

        # Update the environment with our custom filters
        env.filters.update(
@@ -832,11 +838,26 @@ class ShardedWorkerHandlingConfig:
    def should_handle(self, instance_name: str, key: str) -> bool:
        """Whether this instance is responsible for handling the given key.
        """
-
-        # If multiple instances are not defined we always return true.
+        # If multiple instances are not defined we always return true
        if not self.instances or len(self.instances) == 1:
            return True

+        return self.get_instance(key) == instance_name
+
+    def get_instance(self, key: str) -> str:
+        """Get the instance responsible for handling the given key.
+
+        Note: For things like federation sending the config for which instance
+        is sending is known only to the sender instance if there is only one.
+        Therefore `should_handle` should be used where possible.
+        """
+
+        if not self.instances:
+            return "master"
+
+        if len(self.instances) == 1:
+            return self.instances[0]
+
        # We shard by taking the hash, modulo it by the number of instances and
        # then checking whether this instance matches the instance at that
        # index.
@@ -846,7 +867,7 @@ class ShardedWorkerHandlingConfig:
        dest_hash = sha256(key.encode("utf8")).digest()
        dest_int = int.from_bytes(dest_hash, byteorder="little")
        remainder = dest_int % (len(self.instances))
-        return self.instances[remainder] == instance_name
+        return self.instances[remainder]


 __all__ = ["Config", "RootConfig", "ShardedWorkerHandlingConfig"]
@@ -142,3 +142,4 @@ class ShardedWorkerHandlingConfig:
    instances: List[str]
    def __init__(self, instances: List[str]) -> None: ...
    def should_handle(self, instance_name: str, key: str) -> bool: ...
+    def get_instance(self, key: str) -> str: ...
@@ -77,7 +77,7 @@ class ConsentConfig(Config):
    section = "consent"

    def __init__(self, *args):
-        super(ConsentConfig, self).__init__(*args)
+        super().__init__(*args)

        self.user_consent_version = None
        self.user_consent_template_dir = None
@@ -14,7 +14,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-from __future__ import print_function

 # This file can't be called email.py because if it is, we cannot:
 import email.utils
@@ -56,6 +56,7 @@ class OIDCConfig(Config):
        self.oidc_userinfo_endpoint = oidc_config.get("userinfo_endpoint")
        self.oidc_jwks_uri = oidc_config.get("jwks_uri")
        self.oidc_skip_verification = oidc_config.get("skip_verification", False)
+        self.oidc_allow_existing_users = oidc_config.get("allow_existing_users", False)

        ump_config = oidc_config.get("user_mapping_provider", {})
        ump_config.setdefault("module", DEFAULT_USER_MAPPING_PROVIDER)
@@ -158,6 +159,11 @@ class OIDCConfig(Config):
          #
          #skip_verification: true

+          # Uncomment to allow a user logging in via OIDC to match a pre-existing account instead
+          # of failing. This could be used if switching from password logins to OIDC. Defaults to false.
+          #
+          #allow_existing_users: true
+
          # An external module can be provided here as a custom solution to mapping
          # attributes returned from a OIDC provider onto a matrix user.
          #
@@ -30,7 +30,7 @@ class AccountValidityConfig(Config):
    def __init__(self, config, synapse_config):
        if config is None:
            return
-        super(AccountValidityConfig, self).__init__()
+        super().__init__()
        self.enabled = config.get("enabled", False)
        self.renew_by_email_enabled = "renew_at" in config

@@ -169,10 +169,6 @@ class SAML2Config(Config):
            saml2_config.get("saml_session_lifetime", "15m")
        )

-        self.saml2_error_html_template = self.read_templates(
-            ["saml_error.html"], saml2_config.get("template_dir")
-        )[0]
-
    def _default_saml_config_dict(
        self, required_attributes: set, optional_attributes: set
    ):
@@ -225,11 +221,14 @@ class SAML2Config(Config):
        # At least one of `sp_config` or `config_path` must be set in this section to
        # enable SAML login.
        #
-        # (You will probably also want to set the following options to `false` to
+        # You will probably also want to set the following options to `false` to
        # disable the regular login/registration flows:
        #   * enable_registration
        #   * password_config.enabled
        #
+        # You will also want to investigate the settings under the "sso" configuration
+        # section below.
+        #
        # Once SAML support is enabled, a metadata file will be exposed at
        # https://<server>:<port>/_matrix/saml2/metadata.xml, which you may be able to
        # use to configure your SAML IdP with. Alternatively, you can manually configure
@@ -351,31 +350,6 @@ class SAML2Config(Config):
          #    value: "staff"
          #  - attribute: department
          #    value: "sales"
-
-          # Directory in which Synapse will try to find the template files below.
-          # If not set, default templates from within the Synapse package will be used.
-          #
-          # DO NOT UNCOMMENT THIS SETTING unless you want to customise the templates.
-          # If you *do* uncomment it, you will need to make sure that all the templates
-          # below are in the directory.
-          #
-          # Synapse will look for the following templates in this directory:
-          #
-          # * HTML page to display to users if something goes wrong during the
-          #   authentication process: 'saml_error.html'.
-          #
-          #   When rendering, this template is given the following variables:
-          #     * code: an HTML error code corresponding to the error that is being
-          #       returned (typically 400 or 500)
-          #
-          #     * msg: a textual message describing the error.
-          #
-          #   The variables will automatically be HTML-escaped.
-          #
-          # You can see the default templates at:
-          # https://github.com/matrix-org/synapse/tree/master/synapse/res/templates
-          #
-          #template_dir: "res/templates"
        """ % {
            "config_dir_path": config_dir_path
        }
@@ -62,7 +62,7 @@ class ServerNoticesConfig(Config):
    section = "servernotices"

    def __init__(self, *args):
-        super(ServerNoticesConfig, self).__init__(*args)
+        super().__init__(*args)
        self.server_notices_mxid = None
        self.server_notices_mxid_display_name = None
        self.server_notices_mxid_avatar_url = None
@@ -13,8 +13,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-from __future__ import division
-
 import sys

 from ._base import Config
@@ -13,12 +13,24 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

+from typing import List, Union
+
 import attr

 from ._base import Config, ConfigError, ShardedWorkerHandlingConfig
 from .server import ListenerConfig, parse_listener_def


+def _instance_to_list_converter(obj: Union[str, List[str]]) -> List[str]:
+    """Helper for allowing parsing a string or list of strings to a config
+    option expecting a list of strings.
+    """
+
+    if isinstance(obj, str):
+        return [obj]
+    return obj
+
+
@attr.s
 class InstanceLocationConfig:
    """The host and port to talk to an instance via HTTP replication.
@@ -33,11 +45,13 @@ class WriterLocations:
    """Specifies the instances that write various streams.

    Attributes:
-        events: The instance that writes to the event and backfill streams.
-        events: The instance that writes to the typing stream.
+        events: The instances that write to the event and backfill streams.
+        typing: The instance that writes to the typing stream.
    """

-    events = attr.ib(default="master", type=str)
+    events = attr.ib(
+        default=["master"], type=List[str], converter=_instance_to_list_converter
+    )
    typing = attr.ib(default="master", type=str)


@@ -105,15 +119,18 @@ class WorkerConfig(Config):
        writers = config.get("stream_writers") or {}
        self.writers = WriterLocations(**writers)

-        # Check that the configured writer for events and typing also appears in
+        # Check that the configured writers for events and typing also appears in
        # `instance_map`.
        for stream in ("events", "typing"):
-            instance = getattr(self.writers, stream)
-            if instance != "master" and instance not in self.instance_map:
-                raise ConfigError(
-                    "Instance %r is configured to write %s but does not appear in `instance_map` config."
-                    % (instance, stream)
-                )
+            instances = _instance_to_list_converter(getattr(self.writers, stream))
+            for instance in instances:
+                if instance != "master" and instance not in self.instance_map:
+                    raise ConfigError(
+                        "Instance %r is configured to write %s but does not appear in `instance_map` config."
+                        % (instance, stream)
+                    )
+
+        self.events_shard_config = ShardedWorkerHandlingConfig(self.writers.events)

    def generate_config_section(self, config_dir_path, server_name, **kwargs):
        return """\
@@ -45,7 +45,11 @@ _TLS_VERSION_MAP = {

 class ServerContextFactory(ContextFactory):
    """Factory for PyOpenSSL SSL contexts that are used to handle incoming
-    connections."""
+    connections.
+
+    TODO: replace this with an implementation of IOpenSSLServerConnectionCreator,
+    per https://github.com/matrix-org/synapse/issues/1691
+    """

    def __init__(self, config):
        # TODO: once pyOpenSSL exposes TLS_METHOD and SSL_CTX_set_min_proto_version,
@@ -42,7 +42,6 @@ from synapse.api.errors import (
 )
 from synapse.logging.context import (
    PreserveLoggingContext,
-    current_context,
    make_deferred_yieldable,
    preserve_fn,
    run_in_background,
@@ -233,8 +232,6 @@ class Keyring:
        """

        try:
-            ctx = current_context()
-
            # map from server name to a set of outstanding request ids
            server_to_request_ids = {}

@@ -265,12 +262,8 @@ class Keyring:

                # if there are no more requests for this server, we can drop the lock.
                if not server_requests:
-                    with PreserveLoggingContext(ctx):
-                        logger.debug("Releasing key lookup lock on %s", server_name)
-
-                    # ... but not immediately, as that can cause stack explosions if
-                    # we get a long queue of lookups.
-                    self.clock.call_later(0, drop_server_lock, server_name)
+                    logger.debug("Releasing key lookup lock on %s", server_name)
+                    drop_server_lock(server_name)

                return res

@@ -335,20 +328,32 @@ class Keyring:
                        )

                    # look for any requests which weren't satisfied
-                    with PreserveLoggingContext():
-                        for verify_request in remaining_requests:
-                            verify_request.key_ready.errback(
-                                SynapseError(
-                                    401,
-                                    "No key for %s with ids in %s (min_validity %i)"
-                                    % (
-                                        verify_request.server_name,
-                                        verify_request.key_ids,
-                                        verify_request.minimum_valid_until_ts,
-                                    ),
-                                    Codes.UNAUTHORIZED,
-                                )
+                    while remaining_requests:
+                        verify_request = remaining_requests.pop()
+                        rq_str = (
+                            "VerifyJsonRequest(server=%s, key_ids=%s, min_valid=%i)"
+                            % (
+                                verify_request.server_name,
+                                verify_request.key_ids,
+                                verify_request.minimum_valid_until_ts,
                            )
+                        )
+
+                        # If we run the errback immediately, it may cancel our
+                        # loggingcontext while we are still in it, so instead we
+                        # schedule it for the next time round the reactor.
+                        #
+                        # (this also ensures that we don't get a stack overflow if we
+                        # has a massive queue of lookups waiting for this server).
+                        self.clock.call_later(
+                            0,
+                            verify_request.key_ready.errback,
+                            SynapseError(
+                                401,
+                                "Failed to find any key to satisfy %s" % (rq_str,),
+                                Codes.UNAUTHORIZED,
+                            ),
+                        )
            except Exception as err:
                # we don't really expect to get here, because any errors should already
                # have been caught and logged. But if we do, let's log the error and make
@@ -410,10 +415,23 @@ class Keyring:
                    # key was not valid at this point
                    continue

-                with PreserveLoggingContext():
-                    verify_request.key_ready.callback(
-                        (server_name, key_id, fetch_key_result.verify_key)
-                    )
+                # we have a valid key for this request. If we run the callback
+                # immediately, it may cancel our loggingcontext while we are still in
+                # it, so instead we schedule it for the next time round the reactor.
+                #
+                # (this also ensures that we don't get a stack overflow if we had
+                # a massive queue of lookups waiting for this server).
+                logger.debug(
+                    "Found key %s:%s for %s",
+                    server_name,
+                    key_id,
+                    verify_request.request_name,
+                )
+                self.clock.call_later(
+                    0,
+                    verify_request.key_ready.callback,
+                    (server_name, key_id, fetch_key_result.verify_key),
+                )
                completed.append(verify_request)
                break

@@ -558,7 +576,7 @@ class PerspectivesKeyFetcher(BaseV2KeyFetcher):
    """KeyFetcher impl which fetches keys from the "perspectives" servers"""

    def __init__(self, hs):
-        super(PerspectivesKeyFetcher, self).__init__(hs)
+        super().__init__(hs)
        self.clock = hs.get_clock()
        self.client = hs.get_http_client()
        self.key_servers = self.config.key_servers
@@ -728,7 +746,7 @@ class ServerKeyFetcher(BaseV2KeyFetcher):
    """KeyFetcher impl which fetches keys from the origin servers"""

    def __init__(self, hs):
-        super(ServerKeyFetcher, self).__init__(hs)
+        super().__init__(hs)
        self.clock = hs.get_clock()
        self.client = hs.get_http_client()

@@ -54,7 +54,7 @@ from synapse.events import EventBase, builder
 from synapse.federation.federation_base import FederationBase, event_from_pdu_json
 from synapse.logging.context import make_deferred_yieldable, preserve_fn
 from synapse.logging.utils import log_function
-from synapse.types import JsonDict
+from synapse.types import JsonDict, get_domain_from_id
 from synapse.util import unwrapFirstError
 from synapse.util.caches.expiringcache import ExpiringCache
 from synapse.util.retryutils import NotRetryingDestination
@@ -79,7 +79,7 @@ class InvalidResponseError(RuntimeError):

 class FederationClient(FederationBase):
    def __init__(self, hs):
-        super(FederationClient, self).__init__(hs)
+        super().__init__(hs)

        self.pdu_destination_tried = {}
        self._clock.looping_call(self._clear_tried_cache, 60 * 1000)
@@ -217,11 +217,9 @@ class FederationClient(FederationBase):
            for p in transaction_data["pdus"]
        ]

-        # FIXME: We should handle signature failures more gracefully.
-        pdus[:] = await make_deferred_yieldable(
-            defer.gatherResults(
-                self._check_sigs_and_hashes(room_version, pdus), consumeErrors=True,
-            ).addErrback(unwrapFirstError)
+        # Check signatures and hash of pdus, removing any from the list that fail checks
+        pdus[:] = await self._check_sigs_and_hash_and_fetch(
+            dest, pdus, outlier=True, room_version=room_version
        )

        return pdus
@@ -386,10 +384,11 @@ class FederationClient(FederationBase):
                    pdu.event_id, allow_rejected=True, allow_none=True
                )

-            if not res and pdu.origin != origin:
+            pdu_origin = get_domain_from_id(pdu.sender)
+            if not res and pdu_origin != origin:
                try:
                    res = await self.get_pdu(
-                        destinations=[pdu.origin],
+                        destinations=[pdu_origin],
                        event_id=pdu.event_id,
                        room_version=room_version,
                        outlier=outlier,
--- a/Show More
+++ b/Show More
				`@@ -0,0 +1 @@`
				`Fix tests on distros which disable TLSv1.0. Contributed by @danc86.`
				`@@ -0,0 +1 @@`
				Add an admin API `GET /_synapse/admin/v1/event_reports` to read entries of table `event_reports`. Contributed by @dklimpel.
				`@@ -0,0 +1 @@`
				`Fix messages over federation being lost until an event is sent into the same room.`
				`@@ -1 +0,0 @@`
				`Track the latest event for every destination and room for catch-up after federation outage.`
				`@@ -1 +0,0 @@`
				Track the `stream_ordering` of the last successfully-sent event to every destination, so we can use this information to 'catch up' a remote server after an outage.
				`@@ -0,0 +1 @@`
				`Consolidate the SSO error template across all configuration.`
				`@@ -0,0 +1 @@`
				Change `StreamToken.room_key` to be a `RoomStreamToken` instance.
				`@@ -1 +0,0 @@`
				`Blacklist [MSC2753](https://github.com/matrix-org/matrix-doc/pull/2753) SyTests until it is implemented.`
				`@@ -0,0 +1 @@`
				`Add experimental support for sharding event persister.`
				`@@ -0,0 +1 @@`
				`Add the room topic and avatar to the room details admin API.`