Add rudimentary API for promoting/demoting other people in a group

For https://github.com/matrix-org/synapse/issues/2855 (initial)

CHANGES.md

@@ -1,111 +1,15 @@
-Synapse 1.20.0rc2 (2020-09-09)
-==============================
-
-Bugfixes
---------
-
-- Fix a bug introduced in v1.20.0rc1 causing some features related to notifications to misbehave following the implementation of unread counts. ([\#8280](https://github.com/matrix-org/synapse/issues/8280))
-
-
-Synapse 1.20.0rc1 (2020-09-08)
-==============================
+For the next release
+====================
 
 Removal warning
 ---------------
 
-Some older clients used a [disallowed character](https://matrix.org/docs/spec/client_server/r0.6.1#post-matrix-client-r0-register-email-requesttoken) (`:`) in the `client_secret` parameter of various endpoints. The incorrect behaviour was allowed for backwards compatibility, but is now being removed from Synapse as most users have updated their client. Further context can be found at [\#6766](https://github.com/matrix-org/synapse/issues/6766).
-
-Features
---------
-
-- Add an endpoint to query your shared rooms with another user as an implementation of [MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666). ([\#7785](https://github.com/matrix-org/synapse/issues/7785))
-- Iteratively encode JSON to avoid blocking the reactor. ([\#8013](https://github.com/matrix-org/synapse/issues/8013), [\#8116](https://github.com/matrix-org/synapse/issues/8116))
-- Add support for shadow-banning users (ignoring any message send requests). ([\#8034](https://github.com/matrix-org/synapse/issues/8034), [\#8092](https://github.com/matrix-org/synapse/issues/8092), [\#8095](https://github.com/matrix-org/synapse/issues/8095), [\#8142](https://github.com/matrix-org/synapse/issues/8142), [\#8152](https://github.com/matrix-org/synapse/issues/8152), [\#8157](https://github.com/matrix-org/synapse/issues/8157), [\#8158](https://github.com/matrix-org/synapse/issues/8158), [\#8176](https://github.com/matrix-org/synapse/issues/8176))
-- Use the default template file when its equivalent is not found in a custom template directory. ([\#8037](https://github.com/matrix-org/synapse/issues/8037), [\#8107](https://github.com/matrix-org/synapse/issues/8107), [\#8252](https://github.com/matrix-org/synapse/issues/8252))
-- Add unread messages count to sync responses, as specified in [MSC2654](https://github.com/matrix-org/matrix-doc/pull/2654). ([\#8059](https://github.com/matrix-org/synapse/issues/8059), [\#8254](https://github.com/matrix-org/synapse/issues/8254), [\#8270](https://github.com/matrix-org/synapse/issues/8270), [\#8274](https://github.com/matrix-org/synapse/issues/8274))
-- Optimise `/federation/v1/user/devices/` API by only returning devices with encryption keys. ([\#8198](https://github.com/matrix-org/synapse/issues/8198))
-
-
-Bugfixes
---------
-
-- Fix a memory leak by limiting the length of time that messages will be queued for a remote server that has been unreachable. ([\#7864](https://github.com/matrix-org/synapse/issues/7864))
-- Fix `Re-starting finished log context PUT-nnnn` warning when event persistence failed. ([\#8081](https://github.com/matrix-org/synapse/issues/8081))
-- Synapse now correctly enforces the valid characters in the `client_secret` parameter used in various endpoints. ([\#8101](https://github.com/matrix-org/synapse/issues/8101))
-- Fix a bug introduced in v1.7.2 impacting message retention policies that would allow federated homeservers to dictate a retention period that's lower than the configured minimum allowed duration in the configuration file. ([\#8104](https://github.com/matrix-org/synapse/issues/8104))
-- Fix a long-standing bug where invalid JSON would be accepted by Synapse. ([\#8106](https://github.com/matrix-org/synapse/issues/8106))
-- Fix a bug introduced in Synapse v1.12.0 which could cause `/sync` requests to fail with a 404 if you had a very old outstanding room invite. ([\#8110](https://github.com/matrix-org/synapse/issues/8110))
-- Return a proper error code when the rooms of an invalid group are requested. ([\#8129](https://github.com/matrix-org/synapse/issues/8129))
-- Fix a bug which could cause a leaked postgres connection if synapse was set to daemonize. ([\#8131](https://github.com/matrix-org/synapse/issues/8131))
-- Clarify the error code if a user tries to register with a numeric ID. This bug was introduced in v1.15.0. ([\#8135](https://github.com/matrix-org/synapse/issues/8135))
-- Fix a bug where appservices with ratelimiting disabled would still be ratelimited when joining rooms. This bug was introduced in v1.19.0. ([\#8139](https://github.com/matrix-org/synapse/issues/8139))
-- Fix logging in via OpenID Connect with a provider that uses integer user IDs. ([\#8190](https://github.com/matrix-org/synapse/issues/8190))
-- Fix a longstanding bug where user directory updates could break when unexpected profile data was included in events. ([\#8223](https://github.com/matrix-org/synapse/issues/8223))
-- Fix a longstanding bug where stats updates could break when unexpected profile data was included in events. ([\#8226](https://github.com/matrix-org/synapse/issues/8226))
-- Fix slow start times for large servers by removing a table scan of the `users` table from startup code. ([\#8271](https://github.com/matrix-org/synapse/issues/8271))
-
-
-Updates to the Docker image
----------------------------
-
-- Fix builds of the Docker image on non-x86 platforms. ([\#8144](https://github.com/matrix-org/synapse/issues/8144))
-- Added curl for healthcheck support and readme updates for the change. Contributed by @maquis196. ([\#8147](https://github.com/matrix-org/synapse/issues/8147))
-
-
-Improved Documentation
-----------------------
-
-- Link to matrix-synapse-rest-password-provider in the password provider documentation. ([\#8111](https://github.com/matrix-org/synapse/issues/8111))
-- Updated documentation to note that Synapse does not follow `HTTP 308` redirects due to an upstream library not supporting them. Contributed by Ryan Cole. ([\#8120](https://github.com/matrix-org/synapse/issues/8120))
-- Explain better what GDPR-erased means when deactivating a user. ([\#8189](https://github.com/matrix-org/synapse/issues/8189))
-
-
-Internal Changes
-----------------
-
-- Add filter `name` to the `/users` admin API, which filters by user ID or displayname. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#7377](https://github.com/matrix-org/synapse/issues/7377), [\#8163](https://github.com/matrix-org/synapse/issues/8163))
-- Reduce run times of some unit tests by advancing the reactor a fewer number of times. ([\#7757](https://github.com/matrix-org/synapse/issues/7757))
-- Don't fail `/submit_token` requests on incorrect session ID if `request_token_inhibit_3pid_errors` is turned on. ([\#7991](https://github.com/matrix-org/synapse/issues/7991))
-- Convert various parts of the codebase to async/await. ([\#8071](https://github.com/matrix-org/synapse/issues/8071), [\#8072](https://github.com/matrix-org/synapse/issues/8072), [\#8074](https://github.com/matrix-org/synapse/issues/8074), [\#8075](https://github.com/matrix-org/synapse/issues/8075), [\#8076](https://github.com/matrix-org/synapse/issues/8076), [\#8087](https://github.com/matrix-org/synapse/issues/8087), [\#8100](https://github.com/matrix-org/synapse/issues/8100), [\#8119](https://github.com/matrix-org/synapse/issues/8119), [\#8121](https://github.com/matrix-org/synapse/issues/8121), [\#8133](https://github.com/matrix-org/synapse/issues/8133), [\#8156](https://github.com/matrix-org/synapse/issues/8156), [\#8162](https://github.com/matrix-org/synapse/issues/8162), [\#8166](https://github.com/matrix-org/synapse/issues/8166), [\#8168](https://github.com/matrix-org/synapse/issues/8168), [\#8173](https://github.com/matrix-org/synapse/issues/8173), [\#8191](https://github.com/matrix-org/synapse/issues/8191), [\#8192](https://github.com/matrix-org/synapse/issues/8192), [\#8193](https://github.com/matrix-org/synapse/issues/8193), [\#8194](https://github.com/matrix-org/synapse/issues/8194), [\#8195](https://github.com/matrix-org/synapse/issues/8195), [\#8197](https://github.com/matrix-org/synapse/issues/8197), [\#8199](https://github.com/matrix-org/synapse/issues/8199), [\#8200](https://github.com/matrix-org/synapse/issues/8200), [\#8201](https://github.com/matrix-org/synapse/issues/8201), [\#8202](https://github.com/matrix-org/synapse/issues/8202), [\#8207](https://github.com/matrix-org/synapse/issues/8207), [\#8213](https://github.com/matrix-org/synapse/issues/8213), [\#8214](https://github.com/matrix-org/synapse/issues/8214))
-- Remove some unused database functions. ([\#8085](https://github.com/matrix-org/synapse/issues/8085))
-- Add type hints to various parts of the codebase. ([\#8090](https://github.com/matrix-org/synapse/issues/8090), [\#8127](https://github.com/matrix-org/synapse/issues/8127), [\#8187](https://github.com/matrix-org/synapse/issues/8187), [\#8241](https://github.com/matrix-org/synapse/issues/8241), [\#8140](https://github.com/matrix-org/synapse/issues/8140), [\#8183](https://github.com/matrix-org/synapse/issues/8183), [\#8232](https://github.com/matrix-org/synapse/issues/8232), [\#8235](https://github.com/matrix-org/synapse/issues/8235), [\#8237](https://github.com/matrix-org/synapse/issues/8237), [\#8244](https://github.com/matrix-org/synapse/issues/8244))
-- Return the previous stream token if a non-member event is a duplicate. ([\#8093](https://github.com/matrix-org/synapse/issues/8093), [\#8112](https://github.com/matrix-org/synapse/issues/8112))
-- Separate `get_current_token` into two since there are two different use cases for it. ([\#8113](https://github.com/matrix-org/synapse/issues/8113))
-- Remove `ChainedIdGenerator`. ([\#8123](https://github.com/matrix-org/synapse/issues/8123))
-- Reduce the amount of whitespace in JSON stored and sent in responses. ([\#8124](https://github.com/matrix-org/synapse/issues/8124))
-- Update the test federation client to handle streaming responses. ([\#8130](https://github.com/matrix-org/synapse/issues/8130))
-- Micro-optimisations to `get_auth_chain_ids`. ([\#8132](https://github.com/matrix-org/synapse/issues/8132))
-- Refactor `StreamIdGenerator` and `MultiWriterIdGenerator` to have the same interface. ([\#8161](https://github.com/matrix-org/synapse/issues/8161))
-- Add functions to `MultiWriterIdGen` used by events stream. ([\#8164](https://github.com/matrix-org/synapse/issues/8164), [\#8179](https://github.com/matrix-org/synapse/issues/8179))
-- Fix tests that were broken due to the merge of 1.19.1. ([\#8167](https://github.com/matrix-org/synapse/issues/8167))
-- Make `SlavedIdTracker.advance` have the same interface as `MultiWriterIDGenerator`. ([\#8171](https://github.com/matrix-org/synapse/issues/8171))
-- Remove unused `is_guest` parameter from, and add safeguard to, `MessageHandler.get_room_data`. ([\#8174](https://github.com/matrix-org/synapse/issues/8174), [\#8181](https://github.com/matrix-org/synapse/issues/8181))
-- Standardize the mypy configuration. ([\#8175](https://github.com/matrix-org/synapse/issues/8175))
-- Refactor some of `LoginRestServlet`'s helper methods, and move them to `AuthHandler` for easier reuse. ([\#8182](https://github.com/matrix-org/synapse/issues/8182))
-- Fix `wait_for_stream_position` to allow multiple waiters on same stream ID. ([\#8196](https://github.com/matrix-org/synapse/issues/8196))
-- Make `MultiWriterIDGenerator` work for streams that use negative values. ([\#8203](https://github.com/matrix-org/synapse/issues/8203))
-- Refactor queries for device keys and cross-signatures. ([\#8204](https://github.com/matrix-org/synapse/issues/8204), [\#8205](https://github.com/matrix-org/synapse/issues/8205), [\#8222](https://github.com/matrix-org/synapse/issues/8222), [\#8224](https://github.com/matrix-org/synapse/issues/8224), [\#8225](https://github.com/matrix-org/synapse/issues/8225), [\#8231](https://github.com/matrix-org/synapse/issues/8231), [\#8233](https://github.com/matrix-org/synapse/issues/8233), [\#8234](https://github.com/matrix-org/synapse/issues/8234))
-- Fix type hints for functions decorated with `@cached`. ([\#8240](https://github.com/matrix-org/synapse/issues/8240))
-- Remove obsolete `order` field from federation send queues. ([\#8245](https://github.com/matrix-org/synapse/issues/8245))
-- Stop sub-classing from object. ([\#8249](https://github.com/matrix-org/synapse/issues/8249))
-- Add more logging to debug slow startup. ([\#8264](https://github.com/matrix-org/synapse/issues/8264))
-- Do not attempt to upgrade database schema on worker processes. ([\#8266](https://github.com/matrix-org/synapse/issues/8266), [\#8276](https://github.com/matrix-org/synapse/issues/8276))
-
-
-Synapse 1.19.1 (2020-08-27)
-===========================
-
-No significant changes.
-
-
-Synapse 1.19.1rc1 (2020-08-25)
-==============================
-
-Bugfixes
---------
-
-- Fix a bug introduced in v1.19.0 where appservices with ratelimiting disabled would still be ratelimited when joining rooms. ([\#8139](https://github.com/matrix-org/synapse/issues/8139))
-- Fix a bug introduced in v1.19.0 that would cause e.g. profile updates to fail due to incorrect application of rate limits on join requests. ([\#8153](https://github.com/matrix-org/synapse/issues/8153))
+Some older clients used a
+[disallowed character](https://matrix.org/docs/spec/client_server/r0.6.1#post-matrix-client-r0-register-email-requesttoken)
+(`:`) in the `client_secret` parameter of various endpoints. The incorrect
+behaviour was allowed for backwards compatibility, but is now being removed
+from Synapse as most users have updated their client. Further context can be
+found at [\#6766](https://github.com/matrix-org/synapse/issues/6766).
 
 
 Synapse 1.19.0 (2020-08-17)

UPGRADE.rst

@@ -1,16 +1,3 @@
-Upgrading to v1.20.0
-====================
-
-Shared rooms endpoint (MSC2666)
--------------------------------
-
-This release contains a new unstable endpoint `/_matrix/client/unstable/uk.half-shot.msc2666/user/shared_rooms/.*`
-for fetching rooms one user has in common with another. This feature requires the
-`update_user_directory` config flag to be `True`. If you are you are using a `synapse.app.user_dir`
-worker, requests to this endpoint must be handled by that worker. 
-See `docs/workers.md <docs/workers.md>`_ for more details.
-
-
 Upgrading Synapse
 =================
 

changelog.d/7864.bugfix

@@ -0,0 +1 @@
+Fix a memory leak by limiting the length of time that messages will be queued for a remote server that has been unreachable.

changelog.d/8013.feature

@@ -0,0 +1 @@
+Iteratively encode JSON to avoid blocking the reactor.

changelog.d/8037.feature

@@ -0,0 +1 @@
+Use the default template file when its equivalent is not found in a custom template directory.

changelog.d/8072.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8074.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8075.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8076.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8081.bugfix

@@ -0,0 +1 @@
+Fix `Re-starting finished log context PUT-nnnn` warning when event persistence failed.

changelog.d/8085.misc

@@ -0,0 +1 @@
+Remove some unused database functions.

changelog.d/8087.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8090.misc

@@ -0,0 +1 @@
+Add type hints to `synapse.handlers.room`.

changelog.d/8092.feature

@@ -0,0 +1 @@
+Add support for shadow-banning users (ignoring any message send requests).

changelog.d/8093.misc

@@ -0,0 +1 @@
+Return the previous stream token if a non-member event is a duplicate.

changelog.d/8100.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8101.bugfix

@@ -0,0 +1 @@
+Synapse now correctly enforces the valid characters in the `client_secret` parameter used in various endpoints.

changelog.d/8107.feature

@@ -0,0 +1 @@
+Use the default template file when its equivalent is not found in a custom template directory.

changelog.d/8111.doc

@@ -0,0 +1 @@
+Link to matrix-synapse-rest-password-provider in the password provider documentation.

changelog.d/8112.misc

@@ -0,0 +1 @@
+Return the previous stream token if a non-member event is a duplicate.

contrib/cmdclient/http.py

@@ -24,7 +24,7 @@ from twisted.web.client import Agent, readBody
 from twisted.web.http_headers import Headers
 
 
-class HttpClient:
+class HttpClient(object):
     """ Interface for talking json over http
     """
 
@@ -169,7 +169,7 @@ class TwistedHttpClient(HttpClient):
         return d
 
 
-class _RawProducer:
+class _RawProducer(object):
     def __init__(self, data):
         self.data = data
         self.body = data
@@ -186,7 +186,7 @@ class _RawProducer:
         pass
 
 
-class _JsonProducer:
+class _JsonProducer(object):
     """ Used by the twisted http client to create the HTTP body from json
     """
 

contrib/experiments/cursesio.py

@@ -141,7 +141,7 @@ class CursesStdIO:
         curses.endwin()
 
 
-class Callback:
+class Callback(object):
     def __init__(self, stdio):
         self.stdio = stdio
 

contrib/experiments/test_messaging.py

@@ -55,7 +55,7 @@ def excpetion_errback(failure):
     logging.exception(failure)
 
 
-class InputOutput:
+class InputOutput(object):
     """ This is responsible for basic I/O so that a user can interact with
     the example app.
     """
@@ -132,7 +132,7 @@ class IOLoggerHandler(logging.Handler):
         self.io.print_log(msg)
 
 
-class Room:
+class Room(object):
     """ Used to store (in memory) the current membership state of a room, and
     which home servers we should send PDUs associated with the room to.
     """

debian/changelog

@@ -1,15 +1,3 @@
-matrix-synapse-py3 (1.19.0ubuntu1) UNRELEASED; urgency=medium
-
-  * Use Type=notify in systemd service
-
- -- Dexter Chua <dec41@srcf.net>  Wed, 26 Aug 2020 12:41:36 +0000
-
-matrix-synapse-py3 (1.19.1) stable; urgency=medium
-
-  * New synapse release 1.19.1.
-
- -- Synapse Packaging team <packages@matrix.org>  Thu, 27 Aug 2020 10:50:19 +0100
-
 matrix-synapse-py3 (1.19.0) stable; urgency=medium
 
   [ Synapse Packaging team ]

debian/matrix-synapse.service

@@ -2,7 +2,7 @@
 Description=Synapse Matrix homeserver
 
 [Service]
-Type=notify
+Type=simple
 User=matrix-synapse
 WorkingDirectory=/var/lib/matrix-synapse
 EnvironmentFile=/etc/default/matrix-synapse

docker/Dockerfile

@@ -19,16 +19,11 @@ ARG PYTHON_VERSION=3.7
 FROM docker.io/python:${PYTHON_VERSION}-slim as builder
 
 # install the OS build deps
+
+
 RUN apt-get update && apt-get install -y \
     build-essential \
-    libffi-dev \
-    libjpeg-dev \
     libpq-dev \
-    libssl-dev \
-    libwebp-dev \
-    libxml++2.6-dev \
-    libxslt1-dev \
-    zlib1g-dev \
  && rm -rf /var/lib/apt/lists/*
 
 # Build dependencies that are not available as wheels, to speed up rebuilds
@@ -60,12 +55,9 @@ RUN pip install --prefix="/install" --no-warn-script-location \
 FROM docker.io/python:${PYTHON_VERSION}-slim
 
 RUN apt-get update && apt-get install -y \
-    curl \
-    gosu \
-    libjpeg62-turbo \
     libpq5 \
-    libwebp6 \
     xmlsec1 \
+    gosu \
  && rm -rf /var/lib/apt/lists/*
 
 COPY --from=builder /install /usr/local
@@ -77,6 +69,3 @@ VOLUME ["/data"]
 EXPOSE 8008/tcp 8009/tcp 8448/tcp
 
 ENTRYPOINT ["/start.py"]
-
-HEALTHCHECK --interval=1m --timeout=5s \
-  CMD curl -fSs http://localhost:8008/health || exit 1

docker/README.md

@@ -162,32 +162,3 @@ docker build -t matrixdotorg/synapse -f docker/Dockerfile .
 
 You can choose to build a different docker image by changing the value of the `-f` flag to
 point to another Dockerfile.
-
-## Disabling the healthcheck
-
-If you are using a non-standard port or tls inside docker you can disable the healthcheck
-whilst running the above `docker run` commands. 
-
-```
-   --no-healthcheck
-```
-## Setting custom healthcheck on docker run
-
-If you wish to point the healthcheck at a different port with docker command, add the following
-
-```
-  --health-cmd 'curl -fSs http://localhost:1234/health'
-```
-
-## Setting the healthcheck in docker-compose file
-
-You can add the following to set a custom healthcheck in a docker compose file.
-You will need version >2.1 for this to work. 
-
-```
-healthcheck:
-  test: ["CMD", "curl", "-fSs", "http://localhost:8008/health"]
-  interval: 1m
-  timeout: 10s
-  retries: 3
-```

docs/admin_api/user_admin_api.rst

@@ -108,7 +108,7 @@ The api is::
 
     GET /_synapse/admin/v2/users?from=0&limit=10&guests=false
 
-To use it, you will need to authenticate by providing an ``access_token`` for a
+To use it, you will need to authenticate by providing an `access_token` for a
 server admin: see `README.rst <README.rst>`_.
 
 The parameter ``from`` is optional but used for pagination, denoting the
@@ -119,11 +119,8 @@ from a previous call.
 The parameter ``limit`` is optional but is used for pagination, denoting the
 maximum number of items to return in this call. Defaults to ``100``.
 
-The parameter ``user_id`` is optional and filters to only return users with user IDs
-that contain this value. This parameter is ignored when using the ``name`` parameter.
-
-The parameter ``name`` is optional and filters to only return users with user ID localparts
-**or** displaynames that contain this value.
+The parameter ``user_id`` is optional and filters to only users with user IDs
+that contain this value.
 
 The parameter ``guests`` is optional and if ``false`` will **exclude** guest users.
 Defaults to ``true`` to include guest users.
@@ -214,11 +211,9 @@ Deactivate Account
 
 This API deactivates an account. It removes active access tokens, resets the
 password, and deletes third-party IDs (to prevent the user requesting a
-password reset).
-
-It can also mark the user as GDPR-erased. This means messages sent by the
-user will still be visible by anyone that was in the room when these messages
-were sent, but hidden from users joining the room afterwards.
+password reset). It can also mark the user as GDPR-erased (stopping their data
+from distributed further, and deleting it entirely if there are no other
+references to it).
 
 The api is::
 

docs/federate.md

@@ -47,18 +47,6 @@ you invite them to. This can be caused by an incorrectly-configured reverse
 proxy: see [reverse_proxy.md](<reverse_proxy.md>) for instructions on how to correctly
 configure a reverse proxy.
 
-### Known issues
-
-**HTTP `308 Permanent Redirect` redirects are not followed**: Due to missing features
-in the HTTP library used by Synapse, 308 redirects are currently not followed by
-federating servers, which can cause `M_UNKNOWN` or `401 Unauthorized` errors. This
-may affect users who are redirecting apex-to-www (e.g. `example.com` -> `www.example.com`),
-and especially users of the Kubernetes *Nginx Ingress* module, which uses 308 redirect
-codes by default. For those Kubernetes users, [this Stackoverflow post](https://stackoverflow.com/a/52617528/5096871) 
-might be helpful. For other users, switching to a `301 Moved Permanently` code may be
-an option. 308 redirect codes will be supported properly in a future
-release of Synapse.
-
 ## Running a demo federation of Synapses
 
 If you want to get up and running quickly with a trio of homeservers in a

docs/sample_config.yaml

@@ -378,10 +378,11 @@ retention:
   #  min_lifetime: 1d
   #  max_lifetime: 1y
 
-  # Retention policy limits. If set, and the state of a room contains a
-  # 'm.room.retention' event in its state which contains a 'min_lifetime' or a
-  # 'max_lifetime' that's out of these bounds, Synapse will cap the room's policy
-  # to these limits when running purge jobs.
+  # Retention policy limits. If set, a user won't be able to send a
+  # 'm.room.retention' event which features a 'min_lifetime' or a 'max_lifetime'
+  # that's not within this range. This is especially useful in closed federations,
+  # in which server admins can make sure every federating server applies the same
+  # rules.
   #
   #allowed_lifetime_min: 1d
   #allowed_lifetime_max: 1y
@@ -407,19 +408,12 @@ retention:
   # (e.g. every 12h), but not want that purge to be performed by a job that's
   # iterating over every room it knows, which could be heavy on the server.
   #
-  # If any purge job is configured, it is strongly recommended to have at least
-  # a single job with neither 'shortest_max_lifetime' nor 'longest_max_lifetime'
-  # set, or one job without 'shortest_max_lifetime' and one job without
-  # 'longest_max_lifetime' set. Otherwise some rooms might be ignored, even if
-  # 'allowed_lifetime_min' and 'allowed_lifetime_max' are set, because capping a
-  # room's policy to these values is done after the policies are retrieved from
-  # Synapse's database (which is done using the range specified in a purge job's
-  # configuration).
-  #
   #purge_jobs:
-  #  - longest_max_lifetime: 3d
+  #  - shortest_max_lifetime: 1d
+  #    longest_max_lifetime: 3d
   #    interval: 12h
   #  - shortest_max_lifetime: 3d
+  #    longest_max_lifetime: 1y
   #    interval: 1d
 
 # Inhibits the /requestToken endpoints from returning an error that might leak

docs/systemd-with-workers/system/matrix-synapse-worker@.service

@@ -1,14 +1,9 @@
 [Unit]
 Description=Synapse %i
 AssertPathExists=/etc/matrix-synapse/workers/%i.yaml
-
 # This service should be restarted when the synapse target is restarted.
 PartOf=matrix-synapse.target
 
-# if this is started at the same time as the main, let the main process start
-# first, to initialise the database schema.
-After=matrix-synapse.service
-
 [Service]
 Type=notify
 NotifyAccess=main

docs/workers.md

@@ -380,7 +380,6 @@ Handles searches in the user directory. It can handle REST endpoints matching
 the following regular expressions:
 
     ^/_matrix/client/(api/v1|r0|unstable)/user_directory/search$
-    ^/_matrix/client/unstable/uk.half-shot.msc2666/user/shared_rooms/.*$
 
 When using this worker you must also set `update_user_directory: False` in the
 shared configuration file to stop the main synapse running background

mypy.ini

@@ -1,66 +1,11 @@
 [mypy]
 namespace_packages = True
-plugins = mypy_zope:plugin, scripts-dev/mypy_synapse_plugin.py
+plugins = mypy_zope:plugin
 follow_imports = silent
 check_untyped_defs = True
 show_error_codes = True
 show_traceback = True
 mypy_path = stubs
-files =
-  synapse/api,
-  synapse/appservice,
-  synapse/config,
-  synapse/event_auth.py,
-  synapse/events/builder.py,
-  synapse/events/spamcheck.py,
-  synapse/federation,
-  synapse/handlers/auth.py,
-  synapse/handlers/cas_handler.py,
-  synapse/handlers/directory.py,
-  synapse/handlers/events.py,
-  synapse/handlers/federation.py,
-  synapse/handlers/identity.py,
-  synapse/handlers/initial_sync.py,
-  synapse/handlers/message.py,
-  synapse/handlers/oidc_handler.py,
-  synapse/handlers/pagination.py,
-  synapse/handlers/presence.py,
-  synapse/handlers/room.py,
-  synapse/handlers/room_member.py,
-  synapse/handlers/room_member_worker.py,
-  synapse/handlers/saml_handler.py,
-  synapse/handlers/sync.py,
-  synapse/handlers/ui_auth,
-  synapse/http/federation/well_known_resolver.py,
-  synapse/http/server.py,
-  synapse/http/site.py,
-  synapse/logging/,
-  synapse/metrics,
-  synapse/module_api,
-  synapse/notifier.py,
-  synapse/push/pusherpool.py,
-  synapse/push/push_rule_evaluator.py,
-  synapse/replication,
-  synapse/rest,
-  synapse/server.py,
-  synapse/server_notices,
-  synapse/spam_checker_api,
-  synapse/state,
-  synapse/storage/databases/main/stream.py,
-  synapse/storage/databases/main/ui_auth.py,
-  synapse/storage/database.py,
-  synapse/storage/engines,
-  synapse/storage/state.py,
-  synapse/storage/util,
-  synapse/streams,
-  synapse/types.py,
-  synapse/util/caches/descriptors.py,
-  synapse/util/caches/stream_change_cache.py,
-  synapse/util/metrics.py,
-  tests/replication,
-  tests/test_utils,
-  tests/rest/client/v2_alpha/test_auth.py,
-  tests/util/test_stream_change_cache.py
 
 [mypy-pymacaroons.*]
 ignore_missing_imports = True

scripts-dev/federation_client.py

@@ -21,12 +21,10 @@ import argparse
 import base64
 import json
 import sys
-from typing import Any, Optional
 from urllib import parse as urlparse
 
 import nacl.signing
 import requests
-import signedjson.types
 import srvlookup
 import yaml
 from requests.adapters import HTTPAdapter
@@ -71,9 +69,7 @@ def encode_canonical_json(value):
     ).encode("UTF-8")
 
 
-def sign_json(
-    json_object: Any, signing_key: signedjson.types.SigningKey, signing_name: str
-) -> Any:
+def sign_json(json_object, signing_key, signing_name):
     signatures = json_object.pop("signatures", {})
     unsigned = json_object.pop("unsigned", None)
 
@@ -126,14 +122,7 @@ def read_signing_keys(stream):
     return keys
 
 
-def request(
-    method: Optional[str],
-    origin_name: str,
-    origin_key: signedjson.types.SigningKey,
-    destination: str,
-    path: str,
-    content: Optional[str],
-) -> requests.Response:
+def request_json(method, origin_name, origin_key, destination, path, content):
     if method is None:
         if content is None:
             method = "GET"
@@ -170,14 +159,11 @@ def request(
     if method == "POST":
         headers["Content-Type"] = "application/json"
 
-    return s.request(
-        method=method,
-        url=dest,
-        headers=headers,
-        verify=False,
-        data=content,
-        stream=True,
+    result = s.request(
+        method=method, url=dest, headers=headers, verify=False, data=content
     )
+    sys.stderr.write("Status Code: %d\n" % (result.status_code,))
+    return result.json()
 
 
 def main():
@@ -236,7 +222,7 @@ def main():
     with open(args.signing_key_path) as f:
         key = read_signing_keys(f)[0]
 
-    result = request(
+    result = request_json(
         args.method,
         args.server_name,
         key,
@@ -245,12 +231,7 @@ def main():
         content=args.body,
     )
 
-    sys.stderr.write("Status Code: %d\n" % (result.status_code,))
-
-    for chunk in result.iter_content():
-        # we write raw utf8 to stdout.
-        sys.stdout.buffer.write(chunk)
-
+    json.dump(result, sys.stdout)
     print("")
 
 

scripts-dev/hash_history.py

@@ -15,7 +15,7 @@ from synapse.storage.pdu import PduStore
 from synapse.storage.signatures import SignatureStore
 
 
-class Store:
+class Store(object):
     _get_pdu_tuples = PduStore.__dict__["_get_pdu_tuples"]
     _get_pdu_content_hashes_txn = SignatureStore.__dict__["_get_pdu_content_hashes_txn"]
     _get_prev_pdu_hashes_txn = SignatureStore.__dict__["_get_prev_pdu_hashes_txn"]

scripts-dev/mypy_synapse_plugin.py

@@ -1,85 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2020 The Matrix.org Foundation C.I.C.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""This is a mypy plugin for Synpase to deal with some of the funky typing that
-can crop up, e.g the cache descriptors.
-"""
-
-from typing import Callable, Optional
-
-from mypy.plugin import MethodSigContext, Plugin
-from mypy.typeops import bind_self
-from mypy.types import CallableType
-
-
-class SynapsePlugin(Plugin):
-    def get_method_signature_hook(
-        self, fullname: str
-    ) -> Optional[Callable[[MethodSigContext], CallableType]]:
-        if fullname.startswith(
-            "synapse.util.caches.descriptors._CachedFunction.__call__"
-        ):
-            return cached_function_method_signature
-        return None
-
-
-def cached_function_method_signature(ctx: MethodSigContext) -> CallableType:
-    """Fixes the `_CachedFunction.__call__` signature to be correct.
-
-    It already has *almost* the correct signature, except:
-
-        1. the `self` argument needs to be marked as "bound"; and
-        2. any `cache_context` argument should be removed.
-    """
-
-    # First we mark this as a bound function signature.
-    signature = bind_self(ctx.default_signature)
-
-    # Secondly, we remove any "cache_context" args.
-    #
-    # Note: We should be only doing this if `cache_context=True` is set, but if
-    # it isn't then the code will raise an exception when its called anyway, so
-    # its not the end of the world.
-    context_arg_index = None
-    for idx, name in enumerate(signature.arg_names):
-        if name == "cache_context":
-            context_arg_index = idx
-            break
-
-    if context_arg_index:
-        arg_types = list(signature.arg_types)
-        arg_types.pop(context_arg_index)
-
-        arg_names = list(signature.arg_names)
-        arg_names.pop(context_arg_index)
-
-        arg_kinds = list(signature.arg_kinds)
-        arg_kinds.pop(context_arg_index)
-
-        signature = signature.copy_modified(
-            arg_types=arg_types, arg_names=arg_names, arg_kinds=arg_kinds,
-        )
-
-    return signature
-
-
-def plugin(version: str):
-    # This is the entry point of the plugin, and let's us deal with the fact
-    # that the mypy plugin interface is *not* stable by looking at the version
-    # string.
-    #
-    # However, since we pin the version of mypy Synapse uses in CI, we don't
-    # really care.
-    return SynapsePlugin

stubs/frozendict.pyi

@@ -1,47 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2020 The Matrix.org Foundation C.I.C.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Stub for frozendict.
-
-from typing import (
-    Any,
-    Hashable,
-    Iterable,
-    Iterator,
-    Mapping,
-    overload,
-    Tuple,
-    TypeVar,
-)
-
-_KT = TypeVar("_KT", bound=Hashable)  # Key type.
-_VT = TypeVar("_VT")  # Value type.
-
-class frozendict(Mapping[_KT, _VT]):
-    @overload
-    def __init__(self, **kwargs: _VT) -> None: ...
-    @overload
-    def __init__(self, __map: Mapping[_KT, _VT], **kwargs: _VT) -> None: ...
-    @overload
-    def __init__(
-        self, __iterable: Iterable[Tuple[_KT, _VT]], **kwargs: _VT
-    ) -> None: ...
-    def __getitem__(self, key: _KT) -> _VT: ...
-    def __contains__(self, key: Any) -> bool: ...
-    def copy(self, **add_or_replace: Any) -> frozendict: ...
-    def __iter__(self) -> Iterator[_KT]: ...
-    def __len__(self) -> int: ...
-    def __repr__(self) -> str: ...
-    def __hash__(self) -> int: ...

synapse/__init__.py

@@ -48,7 +48,7 @@ try:
 except ImportError:
     pass
 
-__version__ = "1.20.0rc2"
+__version__ = "1.19.0"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when

synapse/api/auth.py

@@ -58,7 +58,7 @@ class _InvalidMacaroonException(Exception):
     pass
 
 
-class Auth:
+class Auth(object):
     """
     FIXME: This class contains a mix of functions for authenticating users
     of our client-server API and authenticating events added to room graphs.

synapse/api/auth_blocking.py

@@ -22,7 +22,7 @@ from synapse.config.server import is_threepid_reserved
 logger = logging.getLogger(__name__)
 
 
-class AuthBlocking:
+class AuthBlocking(object):
     def __init__(self, hs):
         self.store = hs.get_datastore()
 

synapse/api/constants.py

@@ -28,7 +28,7 @@ MAX_ALIAS_LENGTH = 255
 MAX_USERID_LENGTH = 255
 
 
-class Membership:
+class Membership(object):
 
     """Represents the membership states of a user in a room."""
 
@@ -40,7 +40,7 @@ class Membership:
     LIST = (INVITE, JOIN, KNOCK, LEAVE, BAN)
 
 
-class PresenceState:
+class PresenceState(object):
     """Represents the presence state of a user."""
 
     OFFLINE = "offline"
@@ -48,14 +48,14 @@ class PresenceState:
     ONLINE = "online"
 
 
-class JoinRules:
+class JoinRules(object):
     PUBLIC = "public"
     KNOCK = "knock"
     INVITE = "invite"
     PRIVATE = "private"
 
 
-class LoginType:
+class LoginType(object):
     PASSWORD = "m.login.password"
     EMAIL_IDENTITY = "m.login.email.identity"
     MSISDN = "m.login.msisdn"
@@ -65,7 +65,7 @@ class LoginType:
     DUMMY = "m.login.dummy"
 
 
-class EventTypes:
+class EventTypes(object):
     Member = "m.room.member"
     Create = "m.room.create"
     Tombstone = "m.room.tombstone"
@@ -96,17 +96,17 @@ class EventTypes:
     Presence = "m.presence"
 
 
-class RejectedReason:
+class RejectedReason(object):
     AUTH_ERROR = "auth_error"
 
 
-class RoomCreationPreset:
+class RoomCreationPreset(object):
     PRIVATE_CHAT = "private_chat"
     PUBLIC_CHAT = "public_chat"
     TRUSTED_PRIVATE_CHAT = "trusted_private_chat"
 
 
-class ThirdPartyEntityKind:
+class ThirdPartyEntityKind(object):
     USER = "user"
     LOCATION = "location"
 
@@ -115,7 +115,7 @@ ServerNoticeMsgType = "m.server_notice"
 ServerNoticeLimitReached = "m.server_notice.usage_limit_reached"
 
 
-class UserTypes:
+class UserTypes(object):
     """Allows for user type specific behaviour. With the benefit of hindsight
     'admin' and 'guest' users should also be UserTypes. Normal users are type None
     """
@@ -125,7 +125,7 @@ class UserTypes:
     ALL_USER_TYPES = (SUPPORT, BOT)
 
 
-class RelationTypes:
+class RelationTypes(object):
     """The types of relations known to this server.
     """
 
@@ -134,14 +134,14 @@ class RelationTypes:
     REFERENCE = "m.reference"
 
 
-class LimitBlockingTypes:
+class LimitBlockingTypes(object):
     """Reasons that a server may be blocked"""
 
     MONTHLY_ACTIVE_USER = "monthly_active_user"
     HS_DISABLED = "hs_disabled"
 
 
-class EventContentFields:
+class EventContentFields(object):
     """Fields found in events' content, regardless of type."""
 
     # Labels for the event, cf https://github.com/matrix-org/matrix-doc/pull/2326
@@ -152,6 +152,6 @@ class EventContentFields:
     SELF_DESTRUCT_AFTER = "org.matrix.self_destruct_after"
 
 
-class RoomEncryptionAlgorithms:
+class RoomEncryptionAlgorithms(object):
     MEGOLM_V1_AES_SHA2 = "m.megolm.v1.aes-sha2"
     DEFAULT = MEGOLM_V1_AES_SHA2

synapse/api/errors.py

@@ -21,9 +21,9 @@ import typing
 from http import HTTPStatus
 from typing import Dict, List, Optional, Union
 
-from twisted.web import http
+from canonicaljson import json
 
-from synapse.util import json_decoder
+from twisted.web import http
 
 if typing.TYPE_CHECKING:
     from synapse.types import JsonDict
@@ -31,7 +31,7 @@ if typing.TYPE_CHECKING:
 logger = logging.getLogger(__name__)
 
 
-class Codes:
+class Codes(object):
     UNRECOGNIZED = "M_UNRECOGNIZED"
     UNAUTHORIZED = "M_UNAUTHORIZED"
     FORBIDDEN = "M_FORBIDDEN"
@@ -593,7 +593,7 @@ class HttpResponseException(CodeMessageException):
         # try to parse the body as json, to get better errcode/msg, but
         # default to M_UNKNOWN with the HTTP status as the error text
         try:
-            j = json_decoder.decode(self.response.decode("utf-8"))
+            j = json.loads(self.response.decode("utf-8"))
         except ValueError:
             j = {}
 
@@ -604,11 +604,3 @@ class HttpResponseException(CodeMessageException):
         errmsg = j.pop("error", self.msg)
 
         return ProxiedRequestError(self.code, errmsg, errcode, j)
-
-
-class ShadowBanError(Exception):
-    """
-    Raised when a shadow-banned user attempts to perform an action.
-
-    This should be caught and a proper "fake" success response sent to the user.
-    """

synapse/api/filtering.py

@@ -130,7 +130,7 @@ def matrix_user_id_validator(user_id_str):
     return UserID.from_string(user_id_str)
 
 
-class Filtering:
+class Filtering(object):
     def __init__(self, hs):
         super(Filtering, self).__init__()
         self.store = hs.get_datastore()
@@ -168,7 +168,7 @@ class Filtering:
             raise SynapseError(400, str(e))
 
 
-class FilterCollection:
+class FilterCollection(object):
     def __init__(self, filter_json):
         self._filter_json = filter_json
 
@@ -249,7 +249,7 @@ class FilterCollection:
         )
 
 
-class Filter:
+class Filter(object):
     def __init__(self, filter_json):
         self.filter_json = filter_json
 

synapse/api/ratelimiting.py

@@ -17,11 +17,10 @@ from collections import OrderedDict
 from typing import Any, Optional, Tuple
 
 from synapse.api.errors import LimitExceededError
-from synapse.types import Requester
 from synapse.util import Clock
 
 
-class Ratelimiter:
+class Ratelimiter(object):
     """
     Ratelimit actions marked by arbitrary keys.
 
@@ -44,42 +43,6 @@ class Ratelimiter:
         #   * The rate_hz of this particular entry. This can vary per request
         self.actions = OrderedDict()  # type: OrderedDict[Any, Tuple[float, int, float]]
 
-    def can_requester_do_action(
-        self,
-        requester: Requester,
-        rate_hz: Optional[float] = None,
-        burst_count: Optional[int] = None,
-        update: bool = True,
-        _time_now_s: Optional[int] = None,
-    ) -> Tuple[bool, float]:
-        """Can the requester perform the action?
-
-        Args:
-            requester: The requester to key off when rate limiting. The user property
-                will be used.
-            rate_hz: The long term number of actions that can be performed in a second.
-                Overrides the value set during instantiation if set.
-            burst_count: How many actions that can be performed before being limited.
-                Overrides the value set during instantiation if set.
-            update: Whether to count this check as performing the action
-            _time_now_s: The current time. Optional, defaults to the current time according
-                to self.clock. Only used by tests.
-
-        Returns:
-            A tuple containing:
-                * A bool indicating if they can perform the action now
-                * The reactor timestamp for when the action can be performed next.
-                  -1 if rate_hz is less than or equal to zero
-        """
-        # Disable rate limiting of users belonging to any AS that is configured
-        # not to be rate limited in its registration file (rate_limited: true|false).
-        if requester.app_service and not requester.app_service.is_rate_limited():
-            return True, -1.0
-
-        return self.can_do_action(
-            requester.user.to_string(), rate_hz, burst_count, update, _time_now_s
-        )
-
     def can_do_action(
         self,
         key: Any,

synapse/api/room_versions.py

@@ -18,7 +18,7 @@ from typing import Dict
 import attr
 
 
-class EventFormatVersions:
+class EventFormatVersions(object):
     """This is an internal enum for tracking the version of the event format,
     independently from the room version.
     """
@@ -35,20 +35,20 @@ KNOWN_EVENT_FORMAT_VERSIONS = {
 }
 
 
-class StateResolutionVersions:
+class StateResolutionVersions(object):
     """Enum to identify the state resolution algorithms"""
 
     V1 = 1  # room v1 state res
     V2 = 2  # MSC1442 state res: room v2 and later
 
 
-class RoomDisposition:
+class RoomDisposition(object):
     STABLE = "stable"
     UNSTABLE = "unstable"
 
 
 @attr.s(slots=True, frozen=True)
-class RoomVersion:
+class RoomVersion(object):
     """An object which describes the unique attributes of a room version."""
 
     identifier = attr.ib()  # str; the identifier for this version
@@ -69,7 +69,7 @@ class RoomVersion:
     limit_notifications_power_levels = attr.ib(type=bool)
 
 
-class RoomVersions:
+class RoomVersions(object):
     V1 = RoomVersion(
         "1",
         RoomDisposition.STABLE,

synapse/api/urls.py

@@ -33,7 +33,7 @@ MEDIA_PREFIX = "/_matrix/media/r0"
 LEGACY_MEDIA_PREFIX = "/_matrix/media/v1"
 
 
-class ConsentURIBuilder:
+class ConsentURIBuilder(object):
     def __init__(self, hs_config):
         """
         Args:

synapse/app/_base.py

@@ -334,13 +334,6 @@ def install_dns_limiter(reactor, max_dns_requests_in_flight=100):
     This is to workaround https://twistedmatrix.com/trac/ticket/9620, where we
     can run out of file descriptors and infinite loop if we attempt to do too
     many DNS queries at once
-
-    XXX: I'm confused by this. reactor.nameResolver does not use twisted.names unless
-    you explicitly install twisted.names as the resolver; rather it uses a GAIResolver
-    backed by the reactor's default threadpool (which is limited to 10 threads). So
-    (a) I don't understand why twisted ticket 9620 is relevant, and (b) I don't
-    understand why we would run out of FDs if we did too many lookups at once.
-    -- richvdh 2020/08/29
     """
     new_resolver = _LimitedHostnameResolver(
         reactor.nameResolver, max_dns_requests_in_flight
@@ -349,7 +342,7 @@ def install_dns_limiter(reactor, max_dns_requests_in_flight=100):
     reactor.installNameResolver(new_resolver)
 
 
-class _LimitedHostnameResolver:
+class _LimitedHostnameResolver(object):
     """Wraps a IHostnameResolver, limiting the number of in-flight DNS lookups.
     """
 
@@ -409,7 +402,7 @@ class _LimitedHostnameResolver:
             yield deferred
 
 
-class _DeferredResolutionReceiver:
+class _DeferredResolutionReceiver(object):
     """Wraps a IResolutionReceiver and simply resolves the given deferred when
     resolution is complete
     """

synapse/app/admin_cmd.py

@@ -79,7 +79,8 @@ class AdminCmdServer(HomeServer):
         pass
 
 
-async def export_data_command(hs, args):
+@defer.inlineCallbacks
+def export_data_command(hs, args):
     """Export data for a user.
 
     Args:
@@ -90,8 +91,10 @@ async def export_data_command(hs, args):
     user_id = args.user_id
     directory = args.output_directory
 
-    res = await hs.get_handlers().admin_handler.export_user_data(
-        user_id, FileExfiltrationWriter(user_id, directory=directory)
+    res = yield defer.ensureDeferred(
+        hs.get_handlers().admin_handler.export_user_data(
+            user_id, FileExfiltrationWriter(user_id, directory=directory)
+        )
     )
     print(res)
 
@@ -229,15 +232,14 @@ def start(config_options):
     # We also make sure that `_base.start` gets run before we actually run the
     # command.
 
-    async def run():
+    @defer.inlineCallbacks
+    def run(_reactor):
         with LoggingContext("command"):
-            _base.start(ss, [])
-            await args.func(ss, args)
+            yield _base.start(ss, [])
+            yield args.func(ss, args)
 
     _base.start_worker_reactor(
-        "synapse-admin-cmd",
-        config,
-        run_command=lambda: task.react(lambda _reactor: defer.ensureDeferred(run())),
+        "synapse-admin-cmd", config, run_command=lambda: task.react(run)
     )
 
 

synapse/app/generic_worker.py

@@ -745,7 +745,7 @@ class GenericWorkerReplicationHandler(ReplicationDataHandler):
             self.send_handler.wake_destination(server)
 
 
-class FederationSenderHandler:
+class FederationSenderHandler(object):
     """Processes the fedration replication stream
 
     This class is only instantiate on the worker responsible for sending outbound

synapse/app/homeserver.py

@@ -411,24 +411,26 @@ def setup(config_options):
 
         return provision
 
-    async def reprovision_acme():
+    @defer.inlineCallbacks
+    def reprovision_acme():
         """
         Provision a certificate from ACME, if required, and reload the TLS
         certificate if it's renewed.
         """
-        reprovisioned = await do_acme()
+        reprovisioned = yield defer.ensureDeferred(do_acme())
         if reprovisioned:
             _base.refresh_certificate(hs)
 
-    async def start():
+    @defer.inlineCallbacks
+    def start():
         try:
             # Run the ACME provisioning code, if it's enabled.
             if hs.config.acme_enabled:
                 acme = hs.get_acme_handler()
                 # Start up the webservices which we will respond to ACME
                 # challenges with, and then provision.
-                await acme.start_listening()
-                await do_acme()
+                yield defer.ensureDeferred(acme.start_listening())
+                yield defer.ensureDeferred(do_acme())
 
                 # Check if it needs to be reprovisioned every day.
                 hs.get_clock().looping_call(reprovision_acme, 24 * 60 * 60 * 1000)
@@ -437,8 +439,8 @@ def setup(config_options):
             if hs.config.oidc_enabled:
                 oidc = hs.get_oidc_handler()
                 # Loading the provider metadata also ensures the provider config is valid.
-                await oidc.load_metadata()
-                await oidc.load_jwks()
+                yield defer.ensureDeferred(oidc.load_metadata())
+                yield defer.ensureDeferred(oidc.load_jwks())
 
             _base.start(hs, config.listeners)
 
@@ -454,7 +456,7 @@ def setup(config_options):
                 reactor.stop()
             sys.exit(1)
 
-    reactor.callWhenRunning(lambda: defer.ensureDeferred(start()))
+    reactor.callWhenRunning(start)
 
     return hs
 

synapse/appservice/__init__.py

@@ -14,25 +14,20 @@
 # limitations under the License.
 import logging
 import re
-from typing import TYPE_CHECKING
 
 from synapse.api.constants import EventTypes
-from synapse.appservice.api import ApplicationServiceApi
 from synapse.types import GroupID, get_domain_from_id
 from synapse.util.caches.descriptors import cached
 
-if TYPE_CHECKING:
-    from synapse.storage.databases.main import DataStore
-
 logger = logging.getLogger(__name__)
 
 
-class ApplicationServiceState:
+class ApplicationServiceState(object):
     DOWN = "down"
     UP = "up"
 
 
-class AppServiceTransaction:
+class AppServiceTransaction(object):
     """Represents an application service transaction."""
 
     def __init__(self, service, id, events):
@@ -40,19 +35,19 @@ class AppServiceTransaction:
         self.id = id
         self.events = events
 
-    async def send(self, as_api: ApplicationServiceApi) -> bool:
+    def send(self, as_api):
         """Sends this transaction using the provided AS API interface.
 
         Args:
-            as_api: The API to use to send.
+            as_api(ApplicationServiceApi): The API to use to send.
         Returns:
-            True if the transaction was sent.
+            An Awaitable which resolves to True if the transaction was sent.
         """
-        return await as_api.push_bulk(
+        return as_api.push_bulk(
             service=self.service, events=self.events, txn_id=self.id
         )
 
-    async def complete(self, store: "DataStore") -> None:
+    def complete(self, store):
         """Completes this transaction as successful.
 
         Marks this transaction ID on the application service and removes the
@@ -60,11 +55,13 @@ class AppServiceTransaction:
 
         Args:
             store: The database store to operate on.
+        Returns:
+            A Deferred which resolves to True if the transaction was completed.
         """
-        await store.complete_appservice_txn(service=self.service, txn_id=self.id)
+        return store.complete_appservice_txn(service=self.service, txn_id=self.id)
 
 
-class ApplicationService:
+class ApplicationService(object):
     """Defines an application service. This definition is mostly what is
     provided to the /register AS API.
 

synapse/appservice/api.py

@@ -14,20 +14,18 @@
 # limitations under the License.
 import logging
 import urllib
-from typing import TYPE_CHECKING, Optional
 
 from prometheus_client import Counter
 
+from twisted.internet import defer
+
 from synapse.api.constants import EventTypes, ThirdPartyEntityKind
 from synapse.api.errors import CodeMessageException
 from synapse.events.utils import serialize_event
 from synapse.http.client import SimpleHttpClient
-from synapse.types import JsonDict, ThirdPartyInstanceID
+from synapse.types import ThirdPartyInstanceID
 from synapse.util.caches.response_cache import ResponseCache
 
-if TYPE_CHECKING:
-    from synapse.appservice import ApplicationService
-
 logger = logging.getLogger(__name__)
 
 sent_transactions_counter = Counter(
@@ -165,20 +163,19 @@ class ApplicationServiceApi(SimpleHttpClient):
             logger.warning("query_3pe to %s threw exception %s", uri, ex)
             return []
 
-    async def get_3pe_protocol(
-        self, service: "ApplicationService", protocol: str
-    ) -> Optional[JsonDict]:
+    def get_3pe_protocol(self, service, protocol):
         if service.url is None:
             return {}
 
-        async def _get() -> Optional[JsonDict]:
+        @defer.inlineCallbacks
+        def _get():
             uri = "%s%s/thirdparty/protocol/%s" % (
                 service.url,
                 APP_SERVICE_PREFIX,
                 urllib.parse.quote(protocol),
             )
             try:
-                info = await self.get_json(uri, {})
+                info = yield defer.ensureDeferred(self.get_json(uri, {}))
 
                 if not _is_valid_3pe_metadata(info):
                     logger.warning(
@@ -199,7 +196,7 @@ class ApplicationServiceApi(SimpleHttpClient):
                 return None
 
         key = (service.id, protocol)
-        return await self.protocol_meta_cache.wrap(key, _get)
+        return self.protocol_meta_cache.wrap(key, _get)
 
     async def push_bulk(self, service, events, txn_id=None):
         if service.url is None:

synapse/appservice/scheduler.py

@@ -57,7 +57,7 @@ from synapse.metrics.background_process_metrics import run_as_background_process
 logger = logging.getLogger(__name__)
 
 
-class ApplicationServiceScheduler:
+class ApplicationServiceScheduler(object):
     """ Public facing API for this module. Does the required DI to tie the
     components together. This also serves as the "event_pool", which in this
     case is a simple array.
@@ -86,7 +86,7 @@ class ApplicationServiceScheduler:
         self.queuer.enqueue(service, event)
 
 
-class _ServiceQueuer:
+class _ServiceQueuer(object):
     """Queue of events waiting to be sent to appservices.
 
     Groups events into transactions per-appservice, and sends them on to the
@@ -133,7 +133,7 @@ class _ServiceQueuer:
             self.requests_in_flight.discard(service.id)
 
 
-class _TransactionController:
+class _TransactionController(object):
     """Transaction manager.
 
     Builds AppServiceTransactions and runs their lifecycle. Also starts a Recoverer
@@ -209,7 +209,7 @@ class _TransactionController:
         return state == ApplicationServiceState.UP or state is None
 
 
-class _Recoverer:
+class _Recoverer(object):
     """Manages retries and backoff for a DOWN appservice.
 
     We have one of these for each appservice which is currently considered DOWN.

synapse/config/_base.py

@@ -88,7 +88,7 @@ def path_exists(file_path):
         return False
 
 
-class Config:
+class Config(object):
     """
     A configuration section, containing configuration keys and values.
 
@@ -283,7 +283,7 @@ def _create_mxc_to_http_filter(public_baseurl: str) -> Callable:
     return mxc_to_http_filter
 
 
-class RootConfig:
+class RootConfig(object):
     """
     Holder of an application's configuration.
 

synapse/config/cache.py

@@ -33,7 +33,7 @@ _DEFAULT_FACTOR_SIZE = 0.5
 _DEFAULT_EVENT_CACHE_SIZE = "10K"
 
 
-class CacheProperties:
+class CacheProperties(object):
     def __init__(self):
         # The default factor size for all caches
         self.default_factor_size = float(

synapse/config/key.py

@@ -82,7 +82,7 @@ logger = logging.getLogger(__name__)
 
 
 @attr.s
-class TrustedKeyServer:
+class TrustedKeyServer(object):
     # string: name of the server.
     server_name = attr.ib()
 

synapse/config/metrics.py

@@ -22,7 +22,7 @@ from ._base import Config, ConfigError
 
 
 @attr.s
-class MetricsFlags:
+class MetricsFlags(object):
     known_servers = attr.ib(default=False, validator=attr.validators.instance_of(bool))
 
     @classmethod

synapse/config/ratelimiting.py

@@ -17,7 +17,7 @@ from typing import Dict
 from ._base import Config
 
 
-class RateLimitConfig:
+class RateLimitConfig(object):
     def __init__(
         self,
         config: Dict[str, float],
@@ -27,7 +27,7 @@ class RateLimitConfig:
         self.burst_count = config.get("burst_count", defaults["burst_count"])
 
 
-class FederationRateLimitConfig:
+class FederationRateLimitConfig(object):
     _items_and_default = {
         "window_size": 1000,
         "sleep_limit": 10,

synapse/config/room.py

@@ -22,7 +22,7 @@ from ._base import Config, ConfigError
 logger = logging.Logger(__name__)
 
 
-class RoomDefaultEncryptionTypes:
+class RoomDefaultEncryptionTypes(object):
     """Possible values for the encryption_enabled_by_default_for_room_type config option"""
 
     ALL = "all"

synapse/config/room_directory.py

@@ -149,7 +149,7 @@ class RoomDirectoryConfig(Config):
         return False
 
 
-class _RoomDirectoryRule:
+class _RoomDirectoryRule(object):
     """Helper class to test whether a room directory action is allowed, like
     creating an alias or publishing a room.
     """

synapse/config/saml2_config.py

@@ -171,7 +171,7 @@ class SAML2Config(Config):
 
         self.saml2_error_html_template = self.read_templates(
             ["saml_error.html"], saml2_config.get("template_dir")
-        )[0]
+        )
 
     def _default_saml_config_dict(
         self, required_attributes: set, optional_attributes: set

synapse/config/server.py

@@ -424,7 +424,7 @@ class ServerConfig(Config):
         self.gc_thresholds = read_gc_thresholds(config.get("gc_thresholds", None))
 
         @attr.s
-        class LimitRemoteRoomsConfig:
+        class LimitRemoteRoomsConfig(object):
             enabled = attr.ib(
                 validator=attr.validators.instance_of(bool), default=False
             )
@@ -961,10 +961,11 @@ class ServerConfig(Config):
           #  min_lifetime: 1d
           #  max_lifetime: 1y
 
-          # Retention policy limits. If set, and the state of a room contains a
-          # 'm.room.retention' event in its state which contains a 'min_lifetime' or a
-          # 'max_lifetime' that's out of these bounds, Synapse will cap the room's policy
-          # to these limits when running purge jobs.
+          # Retention policy limits. If set, a user won't be able to send a
+          # 'm.room.retention' event which features a 'min_lifetime' or a 'max_lifetime'
+          # that's not within this range. This is especially useful in closed federations,
+          # in which server admins can make sure every federating server applies the same
+          # rules.
           #
           #allowed_lifetime_min: 1d
           #allowed_lifetime_max: 1y
@@ -990,19 +991,12 @@ class ServerConfig(Config):
           # (e.g. every 12h), but not want that purge to be performed by a job that's
           # iterating over every room it knows, which could be heavy on the server.
           #
-          # If any purge job is configured, it is strongly recommended to have at least
-          # a single job with neither 'shortest_max_lifetime' nor 'longest_max_lifetime'
-          # set, or one job without 'shortest_max_lifetime' and one job without
-          # 'longest_max_lifetime' set. Otherwise some rooms might be ignored, even if
-          # 'allowed_lifetime_min' and 'allowed_lifetime_max' are set, because capping a
-          # room's policy to these values is done after the policies are retrieved from
-          # Synapse's database (which is done using the range specified in a purge job's
-          # configuration).
-          #
           #purge_jobs:
-          #  - longest_max_lifetime: 3d
+          #  - shortest_max_lifetime: 1d
+          #    longest_max_lifetime: 3d
           #    interval: 12h
           #  - shortest_max_lifetime: 3d
+          #    longest_max_lifetime: 1y
           #    interval: 1d
 
         # Inhibits the /requestToken endpoints from returning an error that might leak

synapse/crypto/context_factory.py

@@ -83,7 +83,7 @@ class ServerContextFactory(ContextFactory):
 
 
 @implementer(IPolicyForHTTPS)
-class FederationPolicyForHTTPS:
+class FederationPolicyForHTTPS(object):
     """Factory for Twisted SSLClientConnectionCreators that are used to make connections
     to remote servers for federation.
 
@@ -152,7 +152,7 @@ class FederationPolicyForHTTPS:
 
 
 @implementer(IPolicyForHTTPS)
-class RegularPolicyForHTTPS:
+class RegularPolicyForHTTPS(object):
     """Factory for Twisted SSLClientConnectionCreators that are used to make connections
     to remote servers, for other than federation.
 
@@ -189,7 +189,7 @@ def _context_info_cb(ssl_connection, where, ret):
 
 
 @implementer(IOpenSSLClientConnectionCreator)
-class SSLClientConnectionCreator:
+class SSLClientConnectionCreator(object):
     """Creates openssl connection objects for client connections.
 
     Replaces twisted.internet.ssl.ClientTLSOptions
@@ -214,7 +214,7 @@ class SSLClientConnectionCreator:
         return connection
 
 
-class ConnectionVerifier:
+class ConnectionVerifier(object):
     """Set the SNI, and do cert verification
 
     This is a thing which is attached to the TLSMemoryBIOProtocol, and is called by

synapse/crypto/keyring.py

@@ -57,7 +57,7 @@ logger = logging.getLogger(__name__)
 
 
 @attr.s(slots=True, cmp=False)
-class VerifyJsonRequest:
+class VerifyJsonRequest(object):
     """
     A request to verify a JSON object.
 
@@ -96,7 +96,7 @@ class KeyLookupError(ValueError):
     pass
 
 
-class Keyring:
+class Keyring(object):
     def __init__(self, hs, key_fetchers=None):
         self.clock = hs.get_clock()
 
@@ -420,7 +420,7 @@ class Keyring:
         remaining_requests.difference_update(completed)
 
 
-class KeyFetcher:
+class KeyFetcher(object):
     async def get_keys(self, keys_to_fetch):
         """
         Args:
@@ -456,7 +456,7 @@ class StoreKeyFetcher(KeyFetcher):
         return keys
 
 
-class BaseV2KeyFetcher:
+class BaseV2KeyFetcher(object):
     def __init__(self, hs):
         self.store = hs.get_datastore()
         self.config = hs.get_config()
@@ -757,8 +757,9 @@ class ServerKeyFetcher(BaseV2KeyFetcher):
             except Exception:
                 logger.exception("Error getting keys %s from %s", key_ids, server_name)
 
-        await yieldable_gather_results(get_key, keys_to_fetch.items())
-        return results
+        return await yieldable_gather_results(
+            get_key, keys_to_fetch.items()
+        ).addCallback(lambda _: results)
 
     async def get_server_verify_key_v2_direct(self, server_name, key_ids):
         """
@@ -768,7 +769,7 @@ class ServerKeyFetcher(BaseV2KeyFetcher):
             key_ids (iterable[str]):
 
         Returns:
-            dict[str, FetchKeyResult]: map from key ID to lookup result
+            Deferred[dict[str, FetchKeyResult]]: map from key ID to lookup result
 
         Raises:
             KeyLookupError if there was a problem making the lookup

synapse/event_auth.py

@@ -47,7 +47,7 @@ def check(
     Args:
         room_version_obj: the version of the room
         event: the event being checked.
-        auth_events: the existing room state.
+        auth_events (dict: event-key -> event): the existing room state.
 
     Raises:
         AuthError if the checks fail

synapse/events/__init__.py

@@ -18,7 +18,7 @@
 import abc
 import os
 from distutils.util import strtobool
-from typing import Dict, Optional, Tuple, Type
+from typing import Dict, Optional, Type
 
 from unpaddedbase64 import encode_base64
 
@@ -96,7 +96,7 @@ class DefaultDictProperty(DictProperty):
         return instance._dict.get(self.key, self.default)
 
 
-class _EventInternalMetadata:
+class _EventInternalMetadata(object):
     __slots__ = ["_dict"]
 
     def __init__(self, internal_metadata_dict: JsonDict):
@@ -120,7 +120,7 @@ class _EventInternalMetadata:
     # be here
     before = DictProperty("before")  # type: str
     after = DictProperty("after")  # type: str
-    order = DictProperty("order")  # type: Tuple[int, int]
+    order = DictProperty("order")  # type: int
 
     def get_dict(self) -> JsonDict:
         return dict(self._dict)
@@ -133,8 +133,6 @@ class _EventInternalMetadata:
         rejection. This is needed as those events are marked as outliers, but
         they still need to be processed as if they're new events (e.g. updating
         invite state in the database, relaying to clients, etc).
-
-        (Added in synapse 0.99.0, so may be unreliable for events received before that)
         """
         return self._dict.get("out_of_band_membership", False)
 

synapse/events/builder.py

@@ -12,7 +12,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-from typing import Any, Dict, List, Optional, Tuple, Union
+from typing import Optional
 
 import attr
 from nacl.signing import SigningKey
@@ -36,7 +36,7 @@ from synapse.util.stringutils import random_string
 
 
 @attr.s(slots=True, cmp=False, frozen=True)
-class EventBuilder:
+class EventBuilder(object):
     """A format independent event builder used to build up the event content
     before signing the event.
 
@@ -97,14 +97,14 @@ class EventBuilder:
     def is_state(self):
         return self._state_key is not None
 
-    async def build(self, prev_event_ids: List[str]) -> EventBase:
+    async def build(self, prev_event_ids):
         """Transform into a fully signed and hashed event
 
         Args:
-            prev_event_ids: The event IDs to use as the prev events
+            prev_event_ids (list[str]): The event IDs to use as the prev events
 
         Returns:
-            The signed and hashed event.
+            FrozenEvent
         """
 
         state_ids = await self._state.get_current_state_ids(
@@ -114,13 +114,8 @@ class EventBuilder:
 
         format_version = self.room_version.event_format
         if format_version == EventFormatVersions.V1:
-            # The types of auth/prev events changes between event versions.
-            auth_events = await self._store.add_event_hashes(
-                auth_ids
-            )  # type: Union[List[str], List[Tuple[str, Dict[str, str]]]]
-            prev_events = await self._store.add_event_hashes(
-                prev_event_ids
-            )  # type: Union[List[str], List[Tuple[str, Dict[str, str]]]]
+            auth_events = await self._store.add_event_hashes(auth_ids)
+            prev_events = await self._store.add_event_hashes(prev_event_ids)
         else:
             auth_events = auth_ids
             prev_events = prev_event_ids
@@ -143,7 +138,7 @@ class EventBuilder:
             "unsigned": self.unsigned,
             "depth": depth,
             "prev_state": [],
-        }  # type: Dict[str, Any]
+        }
 
         if self.is_state():
             event_dict["state_key"] = self._state_key
@@ -164,7 +159,7 @@ class EventBuilder:
         )
 
 
-class EventBuilderFactory:
+class EventBuilderFactory(object):
     def __init__(self, hs):
         self.clock = hs.get_clock()
         self.hostname = hs.hostname

synapse/events/spamcheck.py

@@ -15,17 +15,16 @@
 # limitations under the License.
 
 import inspect
-from typing import Any, Dict, List, Optional, Tuple
+from typing import Any, Dict, List
 
-from synapse.spam_checker_api import RegistrationBehaviour, SpamCheckerApi
-from synapse.types import Collection
+from synapse.spam_checker_api import SpamCheckerApi
 
 MYPY = False
 if MYPY:
     import synapse.server
 
 
-class SpamChecker:
+class SpamChecker(object):
     def __init__(self, hs: "synapse.server.HomeServer"):
         self.spam_checkers = []  # type: List[Any]
 
@@ -161,33 +160,3 @@ class SpamChecker:
                     return True
 
         return False
-
-    def check_registration_for_spam(
-        self,
-        email_threepid: Optional[dict],
-        username: Optional[str],
-        request_info: Collection[Tuple[str, str]],
-    ) -> RegistrationBehaviour:
-        """Checks if we should allow the given registration request.
-
-        Args:
-            email_threepid: The email threepid used for registering, if any
-            username: The request user name, if any
-            request_info: List of tuples of user agent and IP that
-                were used during the registration process.
-
-        Returns:
-            Enum for how the request should be handled
-        """
-
-        for spam_checker in self.spam_checkers:
-            # For backwards compatibility, only run if the method exists on the
-            # spam checker
-            checker = getattr(spam_checker, "check_registration_for_spam", None)
-            if checker:
-                behaviour = checker(email_threepid, username, request_info)
-                assert isinstance(behaviour, RegistrationBehaviour)
-                if behaviour != RegistrationBehaviour.ALLOW:
-                    return behaviour
-
-        return RegistrationBehaviour.ALLOW

synapse/events/third_party_rules.py

@@ -18,7 +18,7 @@ from synapse.events.snapshot import EventContext
 from synapse.types import Requester
 
 
-class ThirdPartyEventRules:
+class ThirdPartyEventRules(object):
     """Allows server admins to provide a Python module implementing an extra
     set of rules to apply when processing events.
 

synapse/events/utils.py

@@ -322,7 +322,7 @@ def serialize_event(
     return d
 
 
-class EventClientSerializer:
+class EventClientSerializer(object):
     """Serializes events that are to be sent to clients.
 
     This is used for bundling extra information with any events to be sent to

synapse/events/validator.py

@@ -20,7 +20,7 @@ from synapse.events.utils import validate_canonicaljson
 from synapse.types import EventID, RoomID, UserID
 
 
-class EventValidator:
+class EventValidator(object):
     def validate_new(self, event, config):
         """Validates the event has roughly the right format
 
@@ -74,14 +74,15 @@ class EventValidator:
                         )
 
         if event.type == EventTypes.Retention:
-            self._validate_retention(event)
+            self._validate_retention(event, config)
 
-    def _validate_retention(self, event):
+    def _validate_retention(self, event, config):
         """Checks that an event that defines the retention policy for a room respects the
-        format enforced by the spec.
+        boundaries imposed by the server's administrator.
 
         Args:
             event (FrozenEvent): The event to validate.
+            config (Config): The homeserver's configuration.
         """
         min_lifetime = event.content.get("min_lifetime")
         max_lifetime = event.content.get("max_lifetime")
@@ -94,6 +95,32 @@ class EventValidator:
                     errcode=Codes.BAD_JSON,
                 )
 
+            if (
+                config.retention_allowed_lifetime_min is not None
+                and min_lifetime < config.retention_allowed_lifetime_min
+            ):
+                raise SynapseError(
+                    code=400,
+                    msg=(
+                        "'min_lifetime' can't be lower than the minimum allowed"
+                        " value enforced by the server's administrator"
+                    ),
+                    errcode=Codes.BAD_JSON,
+                )
+
+            if (
+                config.retention_allowed_lifetime_max is not None
+                and min_lifetime > config.retention_allowed_lifetime_max
+            ):
+                raise SynapseError(
+                    code=400,
+                    msg=(
+                        "'min_lifetime' can't be greater than the maximum allowed"
+                        " value enforced by the server's administrator"
+                    ),
+                    errcode=Codes.BAD_JSON,
+                )
+
         if max_lifetime is not None:
             if not isinstance(max_lifetime, int):
                 raise SynapseError(
@@ -102,6 +129,32 @@ class EventValidator:
                     errcode=Codes.BAD_JSON,
                 )
 
+            if (
+                config.retention_allowed_lifetime_min is not None
+                and max_lifetime < config.retention_allowed_lifetime_min
+            ):
+                raise SynapseError(
+                    code=400,
+                    msg=(
+                        "'max_lifetime' can't be lower than the minimum allowed value"
+                        " enforced by the server's administrator"
+                    ),
+                    errcode=Codes.BAD_JSON,
+                )
+
+            if (
+                config.retention_allowed_lifetime_max is not None
+                and max_lifetime > config.retention_allowed_lifetime_max
+            ):
+                raise SynapseError(
+                    code=400,
+                    msg=(
+                        "'max_lifetime' can't be greater than the maximum allowed"
+                        " value enforced by the server's administrator"
+                    ),
+                    errcode=Codes.BAD_JSON,
+                )
+
         if (
             min_lifetime is not None
             and max_lifetime is not None

synapse/federation/federation_base.py

@@ -39,7 +39,7 @@ from synapse.types import JsonDict, get_domain_from_id
 logger = logging.getLogger(__name__)
 
 
-class FederationBase:
+class FederationBase(object):
     def __init__(self, hs):
         self.hs = hs
 

synapse/federation/federation_server.py

@@ -28,6 +28,7 @@ from typing import (
     Union,
 )
 
+from canonicaljson import json
 from prometheus_client import Counter, Histogram
 
 from twisted.internet import defer
@@ -62,7 +63,7 @@ from synapse.replication.http.federation import (
     ReplicationGetQueryRestServlet,
 )
 from synapse.types import JsonDict, get_domain_from_id
-from synapse.util import glob_to_regex, json_decoder, unwrapFirstError
+from synapse.util import glob_to_regex, unwrapFirstError
 from synapse.util.async_helpers import Linearizer, concurrently_execute
 from synapse.util.caches.response_cache import ResponseCache
 
@@ -550,7 +551,7 @@ class FederationServer(FederationBase):
             for device_id, keys in device_keys.items():
                 for key_id, json_str in keys.items():
                     json_result.setdefault(user_id, {})[device_id] = {
-                        key_id: json_decoder.decode(json_str)
+                        key_id: json.loads(json_str)
                     }
 
         logger.info(
@@ -785,7 +786,7 @@ def _acl_entry_matches(server_name: str, acl_entry: str) -> Match:
     return regex.match(server_name)
 
 
-class FederationHandlerRegistry:
+class FederationHandlerRegistry(object):
     """Allows classes to register themselves as handlers for a given EDU or
     query type for incoming federation traffic.
     """

synapse/federation/persistence.py

@@ -20,16 +20,13 @@ These actions are mostly only used by the :py:mod:`.replication` module.
 """
 
 import logging
-from typing import Optional, Tuple
 
-from synapse.federation.units import Transaction
 from synapse.logging.utils import log_function
-from synapse.types import JsonDict
 
 logger = logging.getLogger(__name__)
 
 
-class TransactionActions:
+class TransactionActions(object):
     """ Defines persistence actions that relate to handling Transactions.
     """
 
@@ -37,32 +34,30 @@ class TransactionActions:
         self.store = datastore
 
     @log_function
-    async def have_responded(
-        self, origin: str, transaction: Transaction
-    ) -> Optional[Tuple[int, JsonDict]]:
-        """Have we already responded to a transaction with the same id and
+    def have_responded(self, origin, transaction):
+        """ Have we already responded to a transaction with the same id and
         origin?
 
         Returns:
-            `None` if we have not previously responded to this transaction or a
-            2-tuple of `(int, dict)` representing the response code and response body.
+            Deferred: Results in `None` if we have not previously responded to
+            this transaction or a 2-tuple of `(int, dict)` representing the
+            response code and response body.
         """
-        transaction_id = transaction.transaction_id  # type: ignore
-        if not transaction_id:
+        if not transaction.transaction_id:
             raise RuntimeError("Cannot persist a transaction with no transaction_id")
 
-        return await self.store.get_received_txn_response(transaction_id, origin)
+        return self.store.get_received_txn_response(transaction.transaction_id, origin)
 
     @log_function
-    async def set_response(
-        self, origin: str, transaction: Transaction, code: int, response: JsonDict
-    ) -> None:
-        """Persist how we responded to a transaction.
+    def set_response(self, origin, transaction, code, response):
+        """ Persist how we responded to a transaction.
+
+        Returns:
+            Deferred
         """
-        transaction_id = transaction.transaction_id  # type: ignore
-        if not transaction_id:
+        if not transaction.transaction_id:
             raise RuntimeError("Cannot persist a transaction with no transaction_id")
 
-        await self.store.set_received_txn_response(
-            transaction_id, origin, code, response
+        return self.store.set_received_txn_response(
+            transaction.transaction_id, origin, code, response
         )

synapse/federation/send_queue.py

@@ -46,7 +46,7 @@ from .units import Edu
 logger = logging.getLogger(__name__)
 
 
-class FederationRemoteSendQueue:
+class FederationRemoteSendQueue(object):
     """A drop in replacement for FederationSender"""
 
     def __init__(self, hs):
@@ -365,7 +365,7 @@ class FederationRemoteSendQueue:
         )
 
 
-class BaseFederationRow:
+class BaseFederationRow(object):
     """Base class for rows to be sent in the federation stream.
 
     Specifies how to identify, serialize and deserialize the different types.

synapse/federation/sender/__init__.py

@@ -56,7 +56,7 @@ sent_pdus_destination_dist_total = Counter(
 )
 
 
-class FederationSender:
+class FederationSender(object):
     def __init__(self, hs: "synapse.server.HomeServer"):
         self.hs = hs
         self.server_name = hs.hostname
@@ -108,6 +108,8 @@ class FederationSender:
             ),
         )
 
+        self._order = 1
+
         self._is_processing = False
         self._last_poked_id = -1
 
@@ -270,6 +272,9 @@ class FederationSender:
         # a transaction in progress. If we do, stick it in the pending_pdus
         # table and we'll get back to it later.
 
+        order = self._order
+        self._order += 1
+
         destinations = set(destinations)
         destinations.discard(self.server_name)
         logger.debug("Sending to: %s", str(destinations))
@@ -281,7 +286,7 @@ class FederationSender:
         sent_pdus_destination_dist_count.inc()
 
         for destination in destinations:
-            self._get_per_destination_queue(destination).send_pdu(pdu)
+            self._get_per_destination_queue(destination).send_pdu(pdu, order)
 
     async def send_read_receipt(self, receipt: ReadReceipt) -> None:
         """Send a RR to any other servers in the room
@@ -324,10 +329,10 @@ class FederationSender:
         room_id = receipt.room_id
 
         # Work out which remote servers should be poked and poke them.
-        domains_set = await self.state.get_current_hosts_in_room(room_id)
+        domains = await self.state.get_current_hosts_in_room(room_id)
         domains = [
             d
-            for d in domains_set
+            for d in domains
             if d != self.server_name
             and self._federation_shard_config.should_handle(self._instance_name, d)
         ]

synapse/federation/sender/per_destination_queue.py

@@ -53,7 +53,7 @@ sent_edus_by_type = Counter(
 )
 
 
-class PerDestinationQueue:
+class PerDestinationQueue(object):
     """
     Manages the per-destination transmission queues.
 
@@ -92,8 +92,8 @@ class PerDestinationQueue:
         self._destination = destination
         self.transmission_loop_running = False
 
-        # a list of pending PDUs
-        self._pending_pdus = []  # type: List[EventBase]
+        # a list of tuples of (pending pdu, order)
+        self._pending_pdus = []  # type: List[Tuple[EventBase, int]]
 
         # XXX this is never actually used: see
         # https://github.com/matrix-org/synapse/issues/7549
@@ -132,13 +132,14 @@ class PerDestinationQueue:
             + len(self._pending_edus_keyed)
         )
 
-    def send_pdu(self, pdu: EventBase) -> None:
+    def send_pdu(self, pdu: EventBase, order: int) -> None:
         """Add a PDU to the queue, and start the transmission loop if necessary
 
         Args:
             pdu: pdu to send
+            order
         """
-        self._pending_pdus.append(pdu)
+        self._pending_pdus.append((pdu, order))
         self.attempt_new_transaction()
 
     def send_presence(self, states: Iterable[UserPresenceState]) -> None:
@@ -184,7 +185,7 @@ class PerDestinationQueue:
         returns immediately. Otherwise kicks off the process of sending a
         transaction in the background.
         """
-
+        # list of (pending_pdu, deferred, order)
         if self.transmission_loop_running:
             # XXX: this can get stuck on by a never-ending
             # request at which point pending_pdus just keeps growing.
@@ -209,7 +210,7 @@ class PerDestinationQueue:
         )
 
     async def _transaction_transmission_loop(self) -> None:
-        pending_pdus = []  # type: List[EventBase]
+        pending_pdus = []  # type: List[Tuple[EventBase, int]]
         try:
             self.transmission_loop_running = True
 
@@ -372,13 +373,13 @@ class PerDestinationQueue:
                 "TX [%s] Failed to send transaction: %s", self._destination, e
             )
 
-            for p in pending_pdus:
+            for p, _ in pending_pdus:
                 logger.info(
                     "Failed to send event %s to %s", p.event_id, self._destination
                 )
         except Exception:
             logger.exception("TX [%s] Failed to send transaction", self._destination)
-            for p in pending_pdus:
+            for p, _ in pending_pdus:
                 logger.info(
                     "Failed to send event %s to %s", p.event_id, self._destination
                 )

synapse/federation/sender/transaction_manager.py

@@ -13,7 +13,9 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import logging
-from typing import TYPE_CHECKING, List
+from typing import TYPE_CHECKING, List, Tuple
+
+from canonicaljson import json
 
 from synapse.api.errors import HttpResponseException
 from synapse.events import EventBase
@@ -26,7 +28,6 @@ from synapse.logging.opentracing import (
     tags,
     whitelisted_homeserver,
 )
-from synapse.util import json_decoder
 from synapse.util.metrics import measure_func
 
 if TYPE_CHECKING:
@@ -35,7 +36,7 @@ if TYPE_CHECKING:
 logger = logging.getLogger(__name__)
 
 
-class TransactionManager:
+class TransactionManager(object):
     """Helper class which handles building and sending transactions
 
     shared between PerDestinationQueue objects
@@ -53,17 +54,11 @@ class TransactionManager:
 
     @measure_func("_send_new_transaction")
     async def send_new_transaction(
-        self, destination: str, pdus: List[EventBase], edus: List[Edu],
-    ) -> bool:
-        """
-        Args:
-            destination: The destination to send to (e.g. 'example.org')
-            pdus: In-order list of PDUs to send
-            edus: List of EDUs to send
-
-        Returns:
-            True iff the transaction was successful
-        """
+        self,
+        destination: str,
+        pending_pdus: List[Tuple[EventBase, int]],
+        pending_edus: List[Edu],
+    ):
 
         # Make a transaction-sending opentracing span. This span follows on from
         # all the edus in that transaction. This needs to be done since there is
@@ -73,14 +68,20 @@ class TransactionManager:
         span_contexts = []
         keep_destination = whitelisted_homeserver(destination)
 
-        for edu in edus:
+        for edu in pending_edus:
             context = edu.get_context()
             if context:
-                span_contexts.append(extract_text_map(json_decoder.decode(context)))
+                span_contexts.append(extract_text_map(json.loads(context)))
             if keep_destination:
                 edu.strip_context()
 
         with start_active_span_follows_from("send_transaction", span_contexts):
+
+            # Sort based on the order field
+            pending_pdus.sort(key=lambda t: t[1])
+            pdus = [x[0] for x in pending_pdus]
+            edus = pending_edus
+
             success = True
 
             logger.debug("TX [%s] _attempt_new_transaction", destination)

synapse/federation/transport/client.py

@@ -30,7 +30,7 @@ from synapse.logging.utils import log_function
 logger = logging.getLogger(__name__)
 
 
-class TransportLayerClient:
+class TransportLayerClient(object):
     """Sends federation HTTP requests to other servers"""
 
     def __init__(self, hs):

synapse/federation/transport/server.py

@@ -100,7 +100,7 @@ class NoAuthenticationError(AuthenticationError):
     pass
 
 
-class Authenticator:
+class Authenticator(object):
     def __init__(self, hs: HomeServer):
         self._clock = hs.get_clock()
         self.keyring = hs.get_keyring()
@@ -228,7 +228,7 @@ def _parse_auth_header(header_bytes):
         )
 
 
-class BaseFederationServlet:
+class BaseFederationServlet(object):
     """Abstract base class for federation servlet classes.
 
     The servlet object should have a PATH attribute which takes the form of a regexp to

synapse/federation/units.py

@@ -107,7 +107,9 @@ class Transaction(JsonEncodedObject):
         if "edus" in kwargs and not kwargs["edus"]:
             del kwargs["edus"]
 
-        super().__init__(transaction_id=transaction_id, pdus=pdus, **kwargs)
+        super(Transaction, self).__init__(
+            transaction_id=transaction_id, pdus=pdus, **kwargs
+        )
 
     @staticmethod
     def create_new(pdus, **kwargs):

synapse/groups/attestations.py

@@ -60,7 +60,7 @@ DEFAULT_ATTESTATION_JITTER = (0.9, 1.3)
 UPDATE_ATTESTATION_TIME_MS = 1 * 24 * 60 * 60 * 1000
 
 
-class GroupAttestationSigning:
+class GroupAttestationSigning(object):
     """Creates and verifies group attestations.
     """
 
@@ -124,7 +124,7 @@ class GroupAttestationSigning:
         )
 
 
-class GroupAttestionRenewer:
+class GroupAttestionRenewer(object):
     """Responsible for sending and receiving attestation updates.
     """
 

synapse/groups/groups_server.py

@@ -32,7 +32,7 @@ logger = logging.getLogger(__name__)
 # TODO: Flairs
 
 
-class GroupsServerWorkerHandler:
+class GroupsServerWorkerHandler(object):
     def __init__(self, hs):
         self.hs = hs
         self.store = hs.get_datastore()
@@ -719,6 +719,27 @@ class GroupsServerHandler(GroupsServerWorkerHandler):
 
         raise NotImplementedError()
 
+    async def change_user_admin_in_group(
+        self, group_id, user_id, want_admin, requester_user_id, content
+    ):
+        """Promotes or demotes a user in a group.
+        """
+
+        await self.check_group_is_ours(group_id, requester_user_id, and_exists=True)
+
+        if requester_user_id == user_id:
+            raise SynapseError(400, "User cannot target themselves")
+
+        is_admin = await self.store.is_user_admin_in_group(
+            group_id, requester_user_id
+        )
+        if not is_admin:
+            raise SynapseError(403, "User is not admin in group")
+
+        await self.store.change_user_admin_in_group(group_id, user_id, want_admin)
+
+        return {}
+
     async def remove_user_from_group(
         self, group_id, user_id, requester_user_id, content
     ):

synapse/handlers/__init__.py

@@ -20,7 +20,7 @@ from .identity import IdentityHandler
 from .search import SearchHandler
 
 
-class Handlers:
+class Handlers(object):
 
     """ Deprecated. A collection of handlers.
 

synapse/handlers/_base.py

@@ -25,7 +25,7 @@ from synapse.types import UserID
 logger = logging.getLogger(__name__)
 
 
-class BaseHandler:
+class BaseHandler(object):
     """
     Common base class for the event handlers.
     """

synapse/handlers/account_data.py

@@ -14,7 +14,7 @@
 # limitations under the License.
 
 
-class AccountDataEventSource:
+class AccountDataEventSource(object):
     def __init__(self, hs):
         self.store = hs.get_datastore()
 

synapse/handlers/account_validity.py

@@ -29,7 +29,7 @@ from synapse.util import stringutils
 logger = logging.getLogger(__name__)
 
 
-class AccountValidityHandler:
+class AccountValidityHandler(object):
     def __init__(self, hs):
         self.hs = hs
         self.config = hs.config

synapse/handlers/acme.py

@@ -34,7 +34,7 @@ solutions, please read https://github.com/matrix-org/synapse/blob/master/docs/AC
 --------------------------------------------------------------------------------"""
 
 
-class AcmeHandler:
+class AcmeHandler(object):
     def __init__(self, hs):
         self.hs = hs
         self.reactor = hs.get_reactor()

synapse/handlers/acme_issuing_service.py

@@ -78,7 +78,7 @@ def create_issuing_service(reactor, acme_url, account_key_file, well_known_resou
 
 @attr.s
 @implementer(ICertificateStore)
-class ErsatzStore:
+class ErsatzStore(object):
     """
     A store that only stores in memory.
     """

synapse/handlers/admin.py

@@ -197,7 +197,7 @@ class AdminHandler(BaseHandler):
         return writer.finished()
 
 
-class ExfiltrationWriter:
+class ExfiltrationWriter(object):
     """Interface used to specify how to write exported data.
     """
 

synapse/handlers/appservice.py

@@ -34,7 +34,7 @@ logger = logging.getLogger(__name__)
 events_processed_counter = Counter("synapse_handlers_appservice_events_processed", "")
 
 
-class ApplicationServicesHandler:
+class ApplicationServicesHandler(object):
     def __init__(self, hs):
         self.store = hs.get_datastore()
         self.is_mine_id = hs.is_mine_id

synapse/handlers/auth.py

@@ -42,9 +42,8 @@ from synapse.http.site import SynapseRequest
 from synapse.logging.context import defer_to_thread
 from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.module_api import ModuleApi
-from synapse.types import JsonDict, Requester, UserID
+from synapse.types import Requester, UserID
 from synapse.util import stringutils as stringutils
-from synapse.util.msisdn import phone_number_to_msisdn
 from synapse.util.threepids import canonicalise_email
 
 from ._base import BaseHandler
@@ -52,91 +51,6 @@ from ._base import BaseHandler
 logger = logging.getLogger(__name__)
 
 
-def convert_client_dict_legacy_fields_to_identifier(
-    submission: JsonDict,
-) -> Dict[str, str]:
-    """
-    Convert a legacy-formatted login submission to an identifier dict.
-
-    Legacy login submissions (used in both login and user-interactive authentication)
-    provide user-identifying information at the top-level instead.
-
-    These are now deprecated and replaced with identifiers:
-    https://matrix.org/docs/spec/client_server/r0.6.1#identifier-types
-
-    Args:
-        submission: The client dict to convert
-
-    Returns:
-        The matching identifier dict
-
-    Raises:
-        SynapseError: If the format of the client dict is invalid
-    """
-    identifier = submission.get("identifier", {})
-
-    # Generate an m.id.user identifier if "user" parameter is present
-    user = submission.get("user")
-    if user:
-        identifier = {"type": "m.id.user", "user": user}
-
-    # Generate an m.id.thirdparty identifier if "medium" and "address" parameters are present
-    medium = submission.get("medium")
-    address = submission.get("address")
-    if medium and address:
-        identifier = {
-            "type": "m.id.thirdparty",
-            "medium": medium,
-            "address": address,
-        }
-
-    # We've converted valid, legacy login submissions to an identifier. If the
-    # submission still doesn't have an identifier, it's invalid
-    if not identifier:
-        raise SynapseError(400, "Invalid login submission", Codes.INVALID_PARAM)
-
-    # Ensure the identifier has a type
-    if "type" not in identifier:
-        raise SynapseError(
-            400, "'identifier' dict has no key 'type'", errcode=Codes.MISSING_PARAM,
-        )
-
-    return identifier
-
-
-def login_id_phone_to_thirdparty(identifier: JsonDict) -> Dict[str, str]:
-    """
-    Convert a phone login identifier type to a generic threepid identifier.
-
-    Args:
-        identifier: Login identifier dict of type 'm.id.phone'
-
-    Returns:
-        An equivalent m.id.thirdparty identifier dict
-    """
-    if "country" not in identifier or (
-        # The specification requires a "phone" field, while Synapse used to require a "number"
-        # field. Accept both for backwards compatibility.
-        "phone" not in identifier
-        and "number" not in identifier
-    ):
-        raise SynapseError(
-            400, "Invalid phone-type identifier", errcode=Codes.INVALID_PARAM
-        )
-
-    # Accept both "phone" and "number" as valid keys in m.id.phone
-    phone_number = identifier.get("phone", identifier["number"])
-
-    # Convert user-provided phone number to a consistent representation
-    msisdn = phone_number_to_msisdn(identifier["country"], phone_number)
-
-    return {
-        "type": "m.id.thirdparty",
-        "medium": "msisdn",
-        "address": msisdn,
-    }
-
-
 class AuthHandler(BaseHandler):
     SESSION_EXPIRE_MS = 48 * 60 * 60 * 1000
 
@@ -450,14 +364,6 @@ class AuthHandler(BaseHandler):
             # authentication flow.
             await self.store.set_ui_auth_clientdict(sid, clientdict)
 
-        user_agent = request.requestHeaders.getRawHeaders(b"User-Agent", default=[b""])[
-            0
-        ].decode("ascii", "surrogateescape")
-
-        await self.store.add_user_agent_ip_to_ui_auth_session(
-            session.session_id, user_agent, clientip
-        )
-
         if not authdict:
             raise InteractiveAuthIncompleteError(
                 session.session_id, self._auth_dict_for_flows(flows, session.session_id)
@@ -1236,7 +1142,7 @@ class AuthHandler(BaseHandler):
 
 
 @attr.s
-class MacaroonGenerator:
+class MacaroonGenerator(object):
 
     hs = attr.ib()
 

synapse/handlers/cas_handler.py

@@ -35,7 +35,6 @@ class CasHandler:
     """
 
     def __init__(self, hs):
-        self.hs = hs
         self._hostname = hs.hostname
         self._auth_handler = hs.get_auth_handler()
         self._registration_handler = hs.get_registration_handler()
@@ -211,16 +210,8 @@ class CasHandler:
 
         else:
             if not registered_user_id:
-                # Pull out the user-agent and IP from the request.
-                user_agent = request.requestHeaders.getRawHeaders(
-                    b"User-Agent", default=[b""]
-                )[0].decode("ascii", "surrogateescape")
-                ip_address = self.hs.get_ip_from_request(request)
-
                 registered_user_id = await self._registration_handler.register_user(
-                    localpart=localpart,
-                    default_display_name=user_display_name,
-                    user_agent_ips=(user_agent, ip_address),
+                    localpart=localpart, default_display_name=user_display_name
                 )
 
             await self._auth_handler.complete_sso_login(

synapse/handlers/device.py

@@ -234,9 +234,7 @@ class DeviceWorkerHandler(BaseHandler):
         return result
 
     async def on_federation_query_user_devices(self, user_id):
-        stream_id, devices = await self.store.get_e2e_device_keys_for_federation_query(
-            user_id
-        )
+        stream_id, devices = await self.store.get_devices_with_keys_by_user(user_id)
         master_key = await self.store.get_e2e_cross_signing_key(user_id, "master")
         self_signing_key = await self.store.get_e2e_cross_signing_key(
             user_id, "self_signing"
@@ -497,7 +495,7 @@ def _update_device_from_client_ips(device, client_ips):
     device.update({"last_seen_ts": ip.get("last_seen"), "last_seen_ip": ip.get("ip")})
 
 
-class DeviceListUpdater:
+class DeviceListUpdater(object):
     "Handles incoming device list updates from federation and updates the DB"
 
     def __init__(self, hs, device_handler):

synapse/handlers/devicemessage.py

@@ -16,6 +16,8 @@
 import logging
 from typing import Any, Dict
 
+from canonicaljson import json
+
 from synapse.api.errors import SynapseError
 from synapse.logging.context import run_in_background
 from synapse.logging.opentracing import (
@@ -25,13 +27,12 @@ from synapse.logging.opentracing import (
     start_active_span,
 )
 from synapse.types import UserID, get_domain_from_id
-from synapse.util import json_encoder
 from synapse.util.stringutils import random_string
 
 logger = logging.getLogger(__name__)
 
 
-class DeviceMessageHandler:
+class DeviceMessageHandler(object):
     def __init__(self, hs):
         """
         Args:
@@ -173,7 +174,7 @@ class DeviceMessageHandler:
                     "sender": sender_user_id,
                     "type": message_type,
                     "message_id": message_id,
-                    "org.matrix.opentracing_context": json_encoder.encode(context),
+                    "org.matrix.opentracing_context": json.dumps(context),
                 }
 
         log_kv({"local_messages": local_messages})

synapse/handlers/directory.py

@@ -23,7 +23,6 @@ from synapse.api.errors import (
     CodeMessageException,
     Codes,
     NotFoundError,
-    ShadowBanError,
     StoreError,
     SynapseError,
 )
@@ -200,8 +199,6 @@ class DirectoryHandler(BaseHandler):
 
         try:
             await self._update_canonical_alias(requester, user_id, room_id, room_alias)
-        except ShadowBanError as e:
-            logger.info("Failed to update alias events due to shadow-ban: %s", e)
         except AuthError as e:
             logger.info("Failed to update alias events: %s", e)
 
@@ -295,9 +292,6 @@ class DirectoryHandler(BaseHandler):
         """
         Send an updated canonical alias event if the removed alias was set as
         the canonical alias or listed in the alt_aliases field.
-
-        Raises:
-            ShadowBanError if the requester has been shadow-banned.
         """
         alias_event = await self.state.get_current_state(
             room_id, EventTypes.CanonicalAlias, ""

synapse/handlers/e2e_keys.py

@@ -19,7 +19,7 @@ import logging
 from typing import Dict, List, Optional, Tuple
 
 import attr
-from canonicaljson import encode_canonical_json
+from canonicaljson import encode_canonical_json, json
 from signedjson.key import VerifyKey, decode_verify_key_bytes
 from signedjson.sign import SignatureVerifyException, verify_signed_json
 from unpaddedbase64 import decode_base64
@@ -35,7 +35,7 @@ from synapse.types import (
     get_domain_from_id,
     get_verify_key_from_cross_signing_key,
 )
-from synapse.util import json_decoder, unwrapFirstError
+from synapse.util import unwrapFirstError
 from synapse.util.async_helpers import Linearizer
 from synapse.util.caches.expiringcache import ExpiringCache
 from synapse.util.retryutils import NotRetryingDestination
@@ -43,7 +43,7 @@ from synapse.util.retryutils import NotRetryingDestination
 logger = logging.getLogger(__name__)
 
 
-class E2eKeysHandler:
+class E2eKeysHandler(object):
     def __init__(self, hs):
         self.store = hs.get_datastore()
         self.federation = hs.get_federation_client()
@@ -353,7 +353,7 @@ class E2eKeysHandler:
             # make sure that each queried user appears in the result dict
             result_dict[user_id] = {}
 
-        results = await self.store.get_e2e_device_keys_for_cs_api(local_query)
+        results = await self.store.get_e2e_device_keys(local_query)
 
         # Build the result structure
         for user_id, device_keys in results.items():
@@ -404,7 +404,7 @@ class E2eKeysHandler:
             for device_id, keys in device_keys.items():
                 for key_id, json_bytes in keys.items():
                     json_result.setdefault(user_id, {})[device_id] = {
-                        key_id: json_decoder.decode(json_bytes)
+                        key_id: json.loads(json_bytes)
                     }
 
         @trace
@@ -734,7 +734,7 @@ class E2eKeysHandler:
             # fetch our stored devices.  This is used to 1. verify
             # signatures on the master key, and 2. to compare with what
             # was sent if the device was signed
-            devices = await self.store.get_e2e_device_keys_for_cs_api([(user_id, None)])
+            devices = await self.store.get_e2e_device_keys([(user_id, None)])
 
             if user_id not in devices:
                 raise NotFoundError("No device keys found")
@@ -1186,7 +1186,7 @@ def _exception_to_failure(e):
 
 
 def _one_time_keys_match(old_key_json, new_key):
-    old_key = json_decoder.decode(old_key_json)
+    old_key = json.loads(old_key_json)
 
     # if either is a string rather than an object, they must match exactly
     if not isinstance(old_key, dict) or not isinstance(new_key, dict):
@@ -1212,7 +1212,7 @@ class SignatureListItem:
     signature = attr.ib()
 
 
-class SigningKeyEduUpdater:
+class SigningKeyEduUpdater(object):
     """Handles incoming signing key updates from federation and updates the DB"""
 
     def __init__(self, hs, e2e_keys_handler):

synapse/handlers/e2e_room_keys.py

@@ -29,7 +29,7 @@ from synapse.util.async_helpers import Linearizer
 logger = logging.getLogger(__name__)
 
 
-class E2eRoomKeysHandler:
+class E2eRoomKeysHandler(object):
     """
     Implements an optional realtime backup mechanism for encrypted E2E megolm room keys.
     This gives a way for users to store and recover their megolm keys if they lose all

synapse/handlers/events.py

@@ -15,30 +15,29 @@
 
 import logging
 import random
-from typing import TYPE_CHECKING, Iterable, List, Optional
 
 from synapse.api.constants import EventTypes, Membership
 from synapse.api.errors import AuthError, SynapseError
 from synapse.events import EventBase
 from synapse.handlers.presence import format_user_presence_state
 from synapse.logging.utils import log_function
-from synapse.streams.config import PaginationConfig
-from synapse.types import JsonDict, UserID
+from synapse.types import UserID
 from synapse.visibility import filter_events_for_client
 
 from ._base import BaseHandler
 
-if TYPE_CHECKING:
-    from synapse.server import HomeServer
-
-
 logger = logging.getLogger(__name__)
 
 
 class EventStreamHandler(BaseHandler):
-    def __init__(self, hs: "HomeServer"):
+    def __init__(self, hs):
         super(EventStreamHandler, self).__init__(hs)
 
+        # Count of active streams per user
+        self._streams_per_user = {}
+        # Grace timers per user to delay the "stopped" signal
+        self._stop_timer_per_user = {}
+
         self.distributor = hs.get_distributor()
         self.distributor.declare("started_user_eventstream")
         self.distributor.declare("stopped_user_eventstream")
@@ -53,14 +52,14 @@ class EventStreamHandler(BaseHandler):
     @log_function
     async def get_stream(
         self,
-        auth_user_id: str,
-        pagin_config: PaginationConfig,
-        timeout: int = 0,
-        as_client_event: bool = True,
-        affect_presence: bool = True,
-        room_id: Optional[str] = None,
-        is_guest: bool = False,
-    ) -> JsonDict:
+        auth_user_id,
+        pagin_config,
+        timeout=0,
+        as_client_event=True,
+        affect_presence=True,
+        room_id=None,
+        is_guest=False,
+    ):
         """Fetches the events stream for a given user.
         """
 
@@ -99,7 +98,7 @@ class EventStreamHandler(BaseHandler):
 
             # When the user joins a new room, or another user joins a currently
             # joined room, we need to send down presence for those users.
-            to_add = []  # type: List[JsonDict]
+            to_add = []
             for event in events:
                 if not isinstance(event, EventBase):
                     continue
@@ -111,7 +110,7 @@ class EventStreamHandler(BaseHandler):
                         # Send down presence for everyone in the room.
                         users = await self.state.get_current_users_in_room(
                             event.room_id
-                        )  # type: Iterable[str]
+                        )
                     else:
                         users = [event.state_key]
 
@@ -145,22 +144,20 @@ class EventStreamHandler(BaseHandler):
 
 
 class EventHandler(BaseHandler):
-    def __init__(self, hs: "HomeServer"):
+    def __init__(self, hs):
         super(EventHandler, self).__init__(hs)
         self.storage = hs.get_storage()
 
-    async def get_event(
-        self, user: UserID, room_id: Optional[str], event_id: str
-    ) -> Optional[EventBase]:
+    async def get_event(self, user, room_id, event_id):
         """Retrieve a single specified event.
 
         Args:
-            user: The user requesting the event
-            room_id: The expected room id. We'll return None if the
+            user (synapse.types.UserID): The user requesting the event
+            room_id (str|None): The expected room id. We'll return None if the
                 event's room does not match.
-            event_id: The event ID to obtain.
+            event_id (str): The event ID to obtain.
         Returns:
-            An event, or None if there is no event matching this ID.
+            dict: An event, or None if there is no event matching this ID.
         Raises:
             SynapseError if there was a problem retrieving this event, or
             AuthError if the user does not have the rights to inspect this

synapse/handlers/federation.py

@@ -72,13 +72,7 @@ from synapse.replication.http.federation import (
 from synapse.replication.http.membership import ReplicationUserJoinedLeftRoomRestServlet
 from synapse.state import StateResolutionStore, resolve_events_with_store
 from synapse.storage.databases.main.events_worker import EventRedactBehaviour
-from synapse.types import (
-    JsonDict,
-    MutableStateMap,
-    StateMap,
-    UserID,
-    get_domain_from_id,
-)
+from synapse.types import JsonDict, StateMap, UserID, get_domain_from_id
 from synapse.util.async_helpers import Linearizer, concurrently_execute
 from synapse.util.distributor import user_joined_room
 from synapse.util.retryutils import NotRetryingDestination
@@ -102,7 +96,7 @@ class _NewEventInfo:
 
     event = attr.ib(type=EventBase)
     state = attr.ib(type=Optional[Sequence[EventBase]], default=None)
-    auth_events = attr.ib(type=Optional[MutableStateMap[EventBase]], default=None)
+    auth_events = attr.ib(type=Optional[StateMap[EventBase]], default=None)
 
 
 class FederationHandler(BaseHandler):
@@ -440,11 +434,11 @@ class FederationHandler(BaseHandler):
         if not prevs - seen:
             return
 
-        latest_list = await self.store.get_latest_event_ids_in_room(room_id)
+        latest = await self.store.get_latest_event_ids_in_room(room_id)
 
         # We add the prev events that we have seen to the latest
         # list to ensure the remote server doesn't give them to us
-        latest = set(latest_list)
+        latest = set(latest)
         latest |= seen
 
         logger.info(
@@ -781,7 +775,7 @@ class FederationHandler(BaseHandler):
                     # keys across all devices.
                     current_keys = [
                         key
-                        for device in cached_devices.values()
+                        for device in cached_devices
                         for key in device.get("keys", {}).get("keys", {}).values()
                     ]
 
@@ -1783,7 +1777,9 @@ class FederationHandler(BaseHandler):
         """Returns the state at the event. i.e. not including said event.
         """
 
-        event = await self.store.get_event(event_id, check_room_id=room_id)
+        event = await self.store.get_event(
+            event_id, allow_none=False, check_room_id=room_id
+        )
 
         state_groups = await self.state_store.get_state_groups(room_id, [event_id])
 
@@ -1809,7 +1805,9 @@ class FederationHandler(BaseHandler):
     async def get_state_ids_for_pdu(self, room_id: str, event_id: str) -> List[str]:
         """Returns the state at the event. i.e. not including said event.
         """
-        event = await self.store.get_event(event_id, check_room_id=room_id)
+        event = await self.store.get_event(
+            event_id, allow_none=False, check_room_id=room_id
+        )
 
         state_groups = await self.state_store.get_state_groups_ids(room_id, [event_id])
 
@@ -1879,8 +1877,8 @@ class FederationHandler(BaseHandler):
         else:
             return None
 
-    async def get_min_depth_for_context(self, context):
-        return await self.store.get_min_depth(context)
+    def get_min_depth_for_context(self, context):
+        return self.store.get_min_depth(context)
 
     async def _handle_new_event(
         self, origin, event, state=None, auth_events=None, backfilled=False
@@ -2059,7 +2057,7 @@ class FederationHandler(BaseHandler):
         origin: str,
         event: EventBase,
         state: Optional[Iterable[EventBase]],
-        auth_events: Optional[MutableStateMap[EventBase]],
+        auth_events: Optional[StateMap[EventBase]],
         backfilled: bool,
     ) -> EventContext:
         context = await self.state_handler.compute_event_context(event, old_state=state)
@@ -2109,8 +2107,8 @@ class FederationHandler(BaseHandler):
         if backfilled or event.internal_metadata.is_outlier():
             return
 
-        extrem_ids_list = await self.store.get_latest_event_ids_in_room(event.room_id)
-        extrem_ids = set(extrem_ids_list)
+        extrem_ids = await self.store.get_latest_event_ids_in_room(event.room_id)
+        extrem_ids = set(extrem_ids)
         prev_event_ids = set(event.prev_event_ids())
 
         if extrem_ids == prev_event_ids:
@@ -2140,12 +2138,10 @@ class FederationHandler(BaseHandler):
             )
             state_sets = list(state_sets.values())
             state_sets.append(state)
-            current_states = await self.state_handler.resolve_events(
+            current_state_ids = await self.state_handler.resolve_events(
                 room_version, state_sets, event
             )
-            current_state_ids = {
-                k: e.event_id for k, e in current_states.items()
-            }  # type: StateMap[str]
+            current_state_ids = {k: e.event_id for k, e in current_state_ids.items()}
         else:
             current_state_ids = await self.state_handler.get_current_state_ids(
                 event.room_id, latest_event_ids=extrem_ids
@@ -2157,13 +2153,11 @@ class FederationHandler(BaseHandler):
 
         # Now check if event pass auth against said current state
         auth_types = auth_types_for_event(event)
-        current_state_ids_list = [
-            e for k, e in current_state_ids.items() if k in auth_types
-        ]
+        current_state_ids = [e for k, e in current_state_ids.items() if k in auth_types]
 
-        auth_events_map = await self.store.get_events(current_state_ids_list)
+        current_auth_events = await self.store.get_events(current_state_ids)
         current_auth_events = {
-            (e.type, e.state_key): e for e in auth_events_map.values()
+            (e.type, e.state_key): e for e in current_auth_events.values()
         }
 
         try:
@@ -2179,7 +2173,9 @@ class FederationHandler(BaseHandler):
         if not in_room:
             raise AuthError(403, "Host not in room.")
 
-        event = await self.store.get_event(event_id, check_room_id=room_id)
+        event = await self.store.get_event(
+            event_id, allow_none=False, check_room_id=room_id
+        )
 
         # Just go through and process each event in `remote_auth_chain`. We
         # don't want to fall into the trap of `missing` being wrong.
@@ -2231,7 +2227,7 @@ class FederationHandler(BaseHandler):
         origin: str,
         event: EventBase,
         context: EventContext,
-        auth_events: MutableStateMap[EventBase],
+        auth_events: StateMap[EventBase],
     ) -> EventContext:
         """
 
@@ -2282,7 +2278,7 @@ class FederationHandler(BaseHandler):
         origin: str,
         event: EventBase,
         context: EventContext,
-        auth_events: MutableStateMap[EventBase],
+        auth_events: StateMap[EventBase],
     ) -> EventContext:
         """Helper for do_auth. See there for docs.