custom edu draft

Fix another jsonschema typecheck error (#11830 )
Similar to #11817. In `_create_power_level_validator` we - retrieve `validator`. This is a class implementing the `jsonschema.protocols.Validator` interface. In other words, `validator: Type[jsonschema.protocols.Validator]`. - we then create an second validator class by modifying the original `validator`. We return that class, which is also of type `Type[jsonschema.protocols.Validator]`. So the original annotation was incorrect: it claimed we were returning an instance of jsonSchema.Draft7Validator, not the class (or a subclass) itself. (Strictly speaking this is incorrect, because `POWER_LEVELS_SCHEMA` isn't pinned to a particular version of JSON Schema. But there are other complications with the type stubs if you try to fix this; I felt like the change herein was a decent compromise that better expresses intent). (I suspect/hope the typeshed project would welcome an effort to improve the jsonschema stubs. Let's see if I get some spare time.)
2022-02-01 01:18:46 +00:00 · 2022-01-25 15:29:28 -05:00 · 2022-01-25 10:35:18 -08:00 · 2022-01-25 16:11:40 +00:00 · 2022-01-25 15:05:22 +00:00 · 2022-01-25 15:35:35 +01:00
198 changed files with 5224 additions and 1913 deletions
@@ -1,12 +1,14 @@
 #!/usr/bin/env bash
-
-# this script is run by GitHub Actions in a plain `bionic` container; it installs the
+# this script is run by GitHub Actions in a plain `focal` container; it installs the
 # minimal requirements for tox and hands over to the py3-old tox environment.

+# Prevent tzdata from asking for user input
+export DEBIAN_FRONTEND=noninteractive
+
 set -ex

 apt-get update
-apt-get install -y python3 python3-dev python3-pip libxml2-dev libxslt-dev xmlsec1 zlib1g-dev tox
+apt-get install -y python3 python3-dev python3-pip libxml2-dev libxslt-dev xmlsec1 zlib1g-dev tox libjpeg-dev libwebp-dev

 export LANG="C.UTF-8"

@@ -8,6 +8,7 @@
  - Use markdown where necessary, mostly for `code blocks`.
  - End with either a period (.) or an exclamation mark (!).
  - Start with a capital letter.
+  - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
 * [ ] Pull request includes a [sign off](https://matrix-org.github.io/synapse/latest/development/contributing_guide.html#sign-off)
 * [ ] [Code style](https://matrix-org.github.io/synapse/latest/code_style.html) is correct
  (run the [linters](https://matrix-org.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
@@ -34,6 +34,8 @@ jobs:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

+      # TODO: consider using https://github.com/docker/metadata-action instead of this
+      # custom magic
      - name: Calculate docker image tag
        id: set-tag
        run: |
@@ -53,18 +55,6 @@ jobs:
          esac
          echo "::set-output name=tag::$tag"

-        # for release builds, we want to get the amd64 image out asap, so first
-        # we do an amd64-only build, before following up with a multiarch build.
-      - name: Build and push amd64
-        uses: docker/build-push-action@v2
-        if: "${{ startsWith(github.ref, 'refs/tags/v') }}"
-        with:
-          push: true
-          labels: "gitsha1=${{ github.sha }}"
-          tags: "matrixdotorg/synapse:${{ steps.set-tag.outputs.tag }}"
-          file: "docker/Dockerfile"
-          platforms: linux/amd64
-
      - name: Build and push all platforms
        uses: docker/build-push-action@v2
        with:
@@ -141,7 +141,7 @@ jobs:
    steps:
      - uses: actions/checkout@v2
      - name: Test with old deps
-        uses: docker://ubuntu:bionic # For old python and sqlite
+        uses: docker://ubuntu:focal # For old python and sqlite
        with:
          workdir: /github/workspace
          entrypoint: .ci/scripts/test_old_deps.sh
@@ -213,15 +213,15 @@ jobs:
      fail-fast: false
      matrix:
        include:
-          - sytest-tag: bionic
+          - sytest-tag: focal

-          - sytest-tag: bionic
+          - sytest-tag: focal
            postgres: postgres

          - sytest-tag: testing
            postgres: postgres

-          - sytest-tag: bionic
+          - sytest-tag: focal
            postgres: multi-postgres
            workers: workers

@@ -323,17 +323,22 @@ jobs:
    if: ${{ !failure() && !cancelled() }}
    needs: linting-done
    runs-on: ubuntu-latest
-    container:
-      # https://github.com/matrix-org/complement/blob/master/dockerfiles/ComplementCIBuildkite.Dockerfile
-      image: matrixdotorg/complement:latest
-      env:
-        CI: true
-      ports:
-        - 8448:8448
-      volumes:
-        - /var/run/docker.sock:/var/run/docker.sock

    steps:
+      # The path is set via a file given by $GITHUB_PATH. We need both Go 1.17 and GOPATH on the path to run Complement.
+      # See https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#adding-a-system-path
+      - name: "Set Go Version"
+        run: |
+          # Add Go 1.17 to the PATH: see https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu2004-Readme.md#environment-variables-2
+          echo "$GOROOT_1_17_X64/bin" >> $GITHUB_PATH
+          # Add the Go path to the PATH: We need this so we can call gotestfmt
+          echo "~/go/bin" >> $GITHUB_PATH
+
+      - name: "Install Complement Dependencies"
+        run: |
+          sudo apt-get update && sudo apt-get install -y libolm3 libolm-dev
+          go get -v github.com/haveyoudebuggedit/gotestfmt/v2/cmd/gotestfmt@latest
+
      - name: Run actions/checkout@v2 for synapse
        uses: actions/checkout@v2
        with:
@@ -366,6 +371,8 @@ jobs:
      # Build initial Synapse image
      - run: docker build -t matrixdotorg/synapse:latest -f docker/Dockerfile .
        working-directory: synapse
+        env:
+          DOCKER_BUILDKIT: 1

      # Build a ready-to-run Synapse image based on the initial image above.
      # This new image includes a config file, keys for signing and TLS, and
@@ -374,7 +381,11 @@ jobs:
        working-directory: complement/dockerfiles

      # Run Complement
-      - run: go test -v -tags synapse_blacklist,msc2403 ./tests/...
+      - run: |
+          set -o pipefail
+          go test -v -json -tags synapse_blacklist,msc2403 ./tests/... 2>&1 | gotestfmt
+        shell: bash
+        name: Run Complement Tests
        env:
          COMPLEMENT_BASE_IMAGE: complement-synapse:latest
        working-directory: complement
@@ -25,7 +25,7 @@ jobs:
      - run: sudo apt-get -qq install xmlsec1
      - uses: actions/setup-python@v2
        with:
-          python-version: 3.6
+          python-version: 3.7
      - run: .ci/patch_for_twisted_trunk.sh
      - run: pip install tox
      - run: tox -e py
@@ -50,3 +50,7 @@ __pycache__/

 # docs
 book/
+
+# complement
+/complement-*
+/master.tar.gz
@@ -1,11 +1,133 @@
+Synapse 1.51.0 (2022-01-25)
+===========================
+
+No significant changes since 1.51.0rc2.
+
+Synapse 1.51.0 deprecates `webclient` listeners and non-HTTP(S) `web_client_location`s. Support for these will be removed in Synapse 1.53.0, at which point Synapse will not be capable of directly serving a web client for Matrix. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1510).
+
+Synapse 1.51.0rc2 (2022-01-24)
+==============================
+
+Bugfixes
+--------
+
+- Fix a bug introduced in Synapse 1.40.0 that caused Synapse to fail to process incoming federation traffic after handling a large amount of events in a v1 room. ([\#11806](https://github.com/matrix-org/synapse/issues/11806))
+
+
+Synapse 1.50.2 (2022-01-24)
+===========================
+
+This release includes the same bugfix as Synapse 1.51.0rc2.
+
+Bugfixes
+--------
+
+- Fix a bug introduced in Synapse 1.40.0 that caused Synapse to fail to process incoming federation traffic after handling a large amount of events in a v1 room. ([\#11806](https://github.com/matrix-org/synapse/issues/11806))
+
+
+Synapse 1.51.0rc1 (2022-01-21)
+==============================
+
+Features
+--------
+
+- Add `track_puppeted_user_ips` config flag to record client IP addresses against puppeted users, and include the puppeted users in monthly active user counts. ([\#11561](https://github.com/matrix-org/synapse/issues/11561), [\#11749](https://github.com/matrix-org/synapse/issues/11749), [\#11757](https://github.com/matrix-org/synapse/issues/11757))
+- Include whether the requesting user has participated in a thread when generating a summary for [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440). ([\#11577](https://github.com/matrix-org/synapse/issues/11577))
+- Return an `M_FORBIDDEN` error code instead of `M_UNKNOWN` when a spam checker module prevents a user from creating a room. ([\#11672](https://github.com/matrix-org/synapse/issues/11672))
+- Add a flag to the `synapse_review_recent_signups` script to ignore and filter appservice users. ([\#11675](https://github.com/matrix-org/synapse/issues/11675), [\#11770](https://github.com/matrix-org/synapse/issues/11770))
+
+
+Bugfixes
+--------
+
+- Fix a long-standing issue which could cause Synapse to incorrectly accept data in the unsigned field of events
+  received over federation. ([\#11530](https://github.com/matrix-org/synapse/issues/11530))
+- Fix a long-standing bug where Synapse wouldn't cache a response indicating that a remote user has no devices. ([\#11587](https://github.com/matrix-org/synapse/issues/11587))
+- Fix an error that occurs whilst trying to get the federation status of a destination server that was working normally. This admin API was newly introduced in Synapse v1.49.0. ([\#11593](https://github.com/matrix-org/synapse/issues/11593))
+- Fix bundled aggregations not being included in the `/sync` response, per [MSC2675](https://github.com/matrix-org/matrix-doc/pull/2675). ([\#11612](https://github.com/matrix-org/synapse/issues/11612), [\#11659](https://github.com/matrix-org/synapse/issues/11659), [\#11791](https://github.com/matrix-org/synapse/issues/11791))
+- Fix the `/_matrix/client/v1/room/{roomId}/hierarchy` endpoint returning incorrect fields which have been present since Synapse 1.49.0. ([\#11667](https://github.com/matrix-org/synapse/issues/11667))
+- Fix preview of some GIF URLs (like tenor.com). Contributed by Philippe Daouadi. ([\#11669](https://github.com/matrix-org/synapse/issues/11669))
+- Fix a bug where only the first 50 rooms from a space were returned from the `/hierarchy` API. This has existed since the introduction of the API in Synapse v1.41.0. ([\#11695](https://github.com/matrix-org/synapse/issues/11695))
+- Fix a bug introduced in Synapse v1.18.0 where password reset and address validation emails would not be sent if their subject was configured to use the 'app' template variable. Contributed by @br4nnigan. ([\#11710](https://github.com/matrix-org/synapse/issues/11710), [\#11745](https://github.com/matrix-org/synapse/issues/11745))
+- Make the 'List Rooms' Admin API sort stable. Contributed by Daniël Sonck. ([\#11737](https://github.com/matrix-org/synapse/issues/11737))
+- Fix a long-standing bug where space hierarchy over federation would only work correctly some of the time. ([\#11775](https://github.com/matrix-org/synapse/issues/11775))
+- Fix a bug introduced in Synapse v1.46.0 that prevented `on_logged_out` module callbacks from being correctly awaited by Synapse. ([\#11786](https://github.com/matrix-org/synapse/issues/11786))
+
+
+Improved Documentation
+----------------------
+
+- Warn against using a Let's Encrypt certificate for TLS/DTLS TURN server client connections, and suggest using ZeroSSL certificate instead. This works around client-side connectivity errors caused by WebRTC libraries that reject Let's Encrypt certificates. Contibuted by @AndrewFerr. ([\#11686](https://github.com/matrix-org/synapse/issues/11686))
+- Document the new `SYNAPSE_TEST_PERSIST_SQLITE_DB` environment variable in the contributing guide. ([\#11715](https://github.com/matrix-org/synapse/issues/11715))
+- Document that the minimum supported PostgreSQL version is now 10. ([\#11725](https://github.com/matrix-org/synapse/issues/11725))
+- Fix typo in demo docs: differnt. ([\#11735](https://github.com/matrix-org/synapse/issues/11735))
+- Update room spec URL in config files. ([\#11739](https://github.com/matrix-org/synapse/issues/11739))
+- Mention `python3-venv` and `libpq-dev` dependencies in the contribution guide. ([\#11740](https://github.com/matrix-org/synapse/issues/11740))
+- Update documentation for configuring login with Facebook. ([\#11755](https://github.com/matrix-org/synapse/issues/11755))
+- Update installation instructions to note that Python 3.6 is no longer supported. ([\#11781](https://github.com/matrix-org/synapse/issues/11781))
+
+
+Deprecations and Removals
+-------------------------
+
+- Remove the unstable `/send_relation` endpoint. ([\#11682](https://github.com/matrix-org/synapse/issues/11682))
+- Remove `python_twisted_reactor_pending_calls` Prometheus metric. ([\#11724](https://github.com/matrix-org/synapse/issues/11724))
+- Remove the `password_hash` field from the response dictionaries of the [Users Admin API](https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html). ([\#11576](https://github.com/matrix-org/synapse/issues/11576))
+- **Deprecate support for `webclient` listeners and non-HTTP(S) `web_client_location` configuration. ([\#11774](https://github.com/matrix-org/synapse/issues/11774), [\#11783](https://github.com/matrix-org/synapse/issues/11783))**
+
+
+Internal Changes
+----------------
+
+- Run `pyupgrade --py37-plus --keep-percent-format` on Synapse. ([\#11685](https://github.com/matrix-org/synapse/issues/11685))
+- Use buildkit's cache feature to speed up docker builds. ([\#11691](https://github.com/matrix-org/synapse/issues/11691))
+- Use `auto_attribs` and native type hints for attrs classes. ([\#11692](https://github.com/matrix-org/synapse/issues/11692), [\#11768](https://github.com/matrix-org/synapse/issues/11768))
+- Remove debug logging for #4422, which has been closed since Synapse 0.99. ([\#11693](https://github.com/matrix-org/synapse/issues/11693))
+- Remove fallback code for Python 2. ([\#11699](https://github.com/matrix-org/synapse/issues/11699))
+- Add a test for [an edge case](https://github.com/matrix-org/synapse/pull/11532#discussion_r769104461) in the `/sync` logic. ([\#11701](https://github.com/matrix-org/synapse/issues/11701))
+- Add the option to write SQLite test dbs to disk when running tests. ([\#11702](https://github.com/matrix-org/synapse/issues/11702))
+- Improve Complement test output for Gitub Actions. ([\#11707](https://github.com/matrix-org/synapse/issues/11707))
+- Fix docstring on `add_account_data_for_user`. ([\#11716](https://github.com/matrix-org/synapse/issues/11716))
+- Complement environment variable name change and update `.gitignore`. ([\#11718](https://github.com/matrix-org/synapse/issues/11718))
+- Simplify calculation of Prometheus metrics for garbage collection. ([\#11723](https://github.com/matrix-org/synapse/issues/11723))
+- Improve accuracy of `python_twisted_reactor_tick_time` Prometheus metric. ([\#11724](https://github.com/matrix-org/synapse/issues/11724), [\#11771](https://github.com/matrix-org/synapse/issues/11771))
+- Minor efficiency improvements when inserting many values into the database. ([\#11742](https://github.com/matrix-org/synapse/issues/11742))
+- Invite PR authors to give themselves credit in the changelog. ([\#11744](https://github.com/matrix-org/synapse/issues/11744))
+- Add optional debugging to investigate [issue 8631](https://github.com/matrix-org/synapse/issues/8631). ([\#11760](https://github.com/matrix-org/synapse/issues/11760))
+- Remove `log_function` utility function and its uses. ([\#11761](https://github.com/matrix-org/synapse/issues/11761))
+- Add a unit test that checks both `client` and `webclient` resources will function when simultaneously enabled. ([\#11765](https://github.com/matrix-org/synapse/issues/11765))
+- Allow overriding complement commit using `COMPLEMENT_REF`. ([\#11766](https://github.com/matrix-org/synapse/issues/11766))
+- Add some comments and type annotations for `_update_outliers_txn`. ([\#11776](https://github.com/matrix-org/synapse/issues/11776))
+
+
+Synapse 1.50.1 (2022-01-18)
+===========================
+
+This release fixes a bug in Synapse 1.50.0 that could prevent clients from being able to connect to Synapse if the `webclient` resource was enabled. Further details are available in [this issue](https://github.com/matrix-org/synapse/issues/11763).
+
+Bugfixes
+--------
+
+- Fix a bug introduced in Synapse 1.50.0rc1 that could cause Matrix clients to be unable to connect to Synapse instances with the `webclient` resource enabled. ([\#11764](https://github.com/matrix-org/synapse/issues/11764))
+
+
+Synapse 1.50.0 (2022-01-18)
+===========================
+
+**This release contains a critical bug that may prevent clients from being able to connect. 
+As such, it is not recommended to upgrade to 1.50.0. Instead, please upgrade straight to
+to 1.50.1. Further details are available in [this issue](https://github.com/matrix-org/synapse/issues/11763).**
+
+Please note that we now only support Python 3.7+ and PostgreSQL 10+ (if applicable), because Python 3.6 and PostgreSQL 9.6 have reached end-of-life.
+
+No significant changes since 1.50.0rc2.
+
+
 Synapse 1.50.0rc2 (2022-01-14)
 ==============================

 This release candidate fixes a federation-breaking regression introduced in Synapse 1.50.0rc1.

-Please note that we now only support Python 3.7+ and PostgreSQL 10+ (if applicable), because Python 3.6 and PostgreSQL 9.6 have reached end-of-life.
-
-
 Bugfixes
 --------

@@ -0,0 +1 @@
+Include the bundled aggregations in the `/sync` response, per [MSC2675](https://github.com/matrix-org/matrix-doc/pull/2675).
@@ -0,0 +1 @@
+Remove account data (including client config, push rules and ignored users) upon user deactivation.
@@ -0,0 +1 @@
+Add admin API to reset connection timeouts for remote server.
@@ -0,0 +1 @@
+Add an admin API to get a list of rooms that federate with a given remote homeserver.
@@ -0,0 +1 @@
+Drop support for Python 3.6, which is EOL.
@@ -0,0 +1 @@
+Fix a long-standing bug when previewing Reddit URLs which do not contain an image.
@@ -0,0 +1 @@
+Fix a long-standing bug that media streams could cause long-lived connections when generating URL previews.
@@ -0,0 +1 @@
+Remove account data (including client config, push rules and ignored users) upon user deactivation.
@@ -0,0 +1 @@
+Remove account data (including client config, push rules and ignored users) upon user deactivation.
@@ -0,0 +1 @@
+Preparation for database schema simplifications: add `state_key` and `rejection_reason` columns to `events` table.
@@ -0,0 +1 @@
+Add `FrozenEvent.get_state_key` and use it in a couple of places.
@@ -0,0 +1 @@
+Preparation for database schema simplifications: stop reading from `event_reference_hashes`.
@@ -0,0 +1 @@
+Drop unused table `public_room_list_stream`.
@@ -0,0 +1 @@
+Preparation for reducing Postgres serialization errors: allow setting transaction isolation level. Contributed by Nick @ Beeper.
@@ -0,0 +1 @@
+Docker: skip the initial amd64-only build and go straight to multiarch.
@@ -0,0 +1 @@
+Run Complement on the Github Actions VM and not inside a Docker container.
@@ -0,0 +1 @@
+Log module names at startup.
@@ -0,0 +1 @@
+Drop support for Python 3.6, which is EOL.
@@ -0,0 +1 @@
+Correct a type annotation in the event validation logic.
@@ -0,0 +1 @@
+Add missing steps to the contribution submission process in the documentation.  Contributed by @sequentialread.
@@ -0,0 +1 @@
+Minor updates and documentation for database schema delta files.
@@ -0,0 +1 @@
+Correct a type annotation in the event validation logic.
@@ -92,22 +92,6 @@ new PromConsole.Graph({
 })
 </script>

-<h3>Pending calls per tick</h3>
-<div id="reactor_pending_calls"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#reactor_pending_calls"),
-  expr: "rate(python_twisted_reactor_pending_calls_sum[30s]) / rate(python_twisted_reactor_pending_calls_count[30s])",
-  name: "[[job]]-[[index]]",
-  min: 0,
-  renderer: "line",
-  height: 150,
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yTitle: "Pending Calls"
-})
-</script>
-
 <h1>Storage</h1>

 <h3>Queries</h3>
@@ -1,3 +1,39 @@
+matrix-synapse-py3 (1.51.0) stable; urgency=medium
+
+  * New synapse release 1.51.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 25 Jan 2022 11:28:51 +0000
+
+matrix-synapse-py3 (1.51.0~rc2) stable; urgency=medium
+
+  * New synapse release 1.51.0~rc2.
+
+ -- Synapse Packaging team <packages@matrix.org>  Mon, 24 Jan 2022 12:25:00 +0000
+
+matrix-synapse-py3 (1.51.0~rc1) stable; urgency=medium
+
+  * New synapse release 1.51.0~rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Fri, 21 Jan 2022 10:46:02 +0000
+
+matrix-synapse-py3 (1.50.2) stable; urgency=medium
+
+  * New synapse release 1.50.2.
+
+ -- Synapse Packaging team <packages@matrix.org>  Mon, 24 Jan 2022 13:37:11 +0000
+
+matrix-synapse-py3 (1.50.1) stable; urgency=medium
+
+  * New synapse release 1.50.1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 18 Jan 2022 16:06:26 +0000
+
+matrix-synapse-py3 (1.50.0) stable; urgency=medium
+
+  * New synapse release 1.50.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 18 Jan 2022 10:40:38 +0000
+
 matrix-synapse-py3 (1.50.0~rc2) stable; urgency=medium

  * New synapse release 1.50.0~rc2.
@@ -22,5 +22,5 @@ Logs and sqlitedb will be stored in demo/808{0,1,2}.{log,db}



-Also note that when joining a public room on a differnt HS via "#foo:bar.net", then you are (in the current impl) joining a room with room_id "foo". This means that it won't work if your HS already has a room with that name.
+Also note that when joining a public room on a different HS via "#foo:bar.net", then you are (in the current impl) joining a room with room_id "foo". This means that it won't work if your HS already has a room with that name.

@@ -1,14 +1,17 @@
 # Dockerfile to build the matrixdotorg/synapse docker images.
 #
+# Note that it uses features which are only available in BuildKit - see
+# https://docs.docker.com/go/buildkit/ for more information.
+#
 # To build the image, run `docker build` command from the root of the
 # synapse repository:
 #
-#    docker build -f docker/Dockerfile .
+#    DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile .
 #
 # There is an optional PYTHON_VERSION build argument which sets the
 # version of python to build against: for example:
 #
-#    docker build -f docker/Dockerfile --build-arg PYTHON_VERSION=3.6 .
+#    DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile --build-arg PYTHON_VERSION=3.9 .
 #

 ARG PYTHON_VERSION=3.8
@@ -19,7 +22,16 @@ ARG PYTHON_VERSION=3.8
 FROM docker.io/python:${PYTHON_VERSION}-slim as builder

 # install the OS build deps
-RUN apt-get update && apt-get install -y \
+#
+# RUN --mount is specific to buildkit and is documented at
+# https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md#build-mounts-run---mount.
+# Here we use it to set up a cache for apt, to improve rebuild speeds on
+# slow connections.
+#
+RUN \
+   --mount=type=cache,target=/var/cache/apt,sharing=locked \
+   --mount=type=cache,target=/var/lib/apt,sharing=locked \
+ apt-get update && apt-get install -y \
    build-essential \
    libffi-dev \
    libjpeg-dev \
@@ -44,7 +56,8 @@ COPY synapse/python_dependencies.py /synapse/synapse/python_dependencies.py
 # used while you develop on the source
 #
 # This is aiming at installing the `install_requires` and `extras_require` from `setup.py`
-RUN pip install --prefix="/install" --no-warn-script-location \
+RUN --mount=type=cache,target=/root/.cache/pip \
+  pip install --prefix="/install" --no-warn-script-location \
    /synapse[all]

 # Copy over the rest of the project
@@ -66,7 +79,10 @@ LABEL org.opencontainers.image.documentation='https://github.com/matrix-org/syna
 LABEL org.opencontainers.image.source='https://github.com/matrix-org/synapse.git'
 LABEL org.opencontainers.image.licenses='Apache-2.0'

-RUN apt-get update && apt-get install -y \
+RUN \
+   --mount=type=cache,target=/var/cache/apt,sharing=locked \
+   --mount=type=cache,target=/var/lib/apt,sharing=locked \
+  apt-get update && apt-get install -y \
    curl \
    gosu \
    libjpeg62-turbo \
@@ -1,6 +1,6 @@
 # Use the Sytest image that comes with a lot of the build dependencies
 # pre-installed
-FROM matrixdotorg/sytest:bionic
+FROM matrixdotorg/sytest:focal

 # The Sytest image doesn't come with python, so install that
 RUN apt-get update && apt-get -qq install -y python3 python3-dev python3-pip
@@ -16,4 +16,4 @@ sudo -u postgres /usr/lib/postgresql/10/bin/pg_ctl -w -D /var/lib/postgresql/dat
 # Run the tests
 cd /src
 export TRIAL_FLAGS="-j 4"
-tox --workdir=./.tox-pg-container -e py36-postgres "$@"
+tox --workdir=./.tox-pg-container -e py37-postgres "$@"
@@ -15,9 +15,10 @@ server admin: [Admin API](../usage/administration/admin_api)

 It returns a JSON body like the following:

-```json
+```jsonc
 {
-    "displayname": "User",
+    "name": "@user:example.com",
+    "displayname": "User", // can be null if not set
    "threepids": [
        {
            "medium": "email",
@@ -32,11 +33,11 @@ It returns a JSON body like the following:
            "validated_at": 1586458409743
        }
    ],
-    "avatar_url": "<avatar_url>",
+    "avatar_url": "<avatar_url>",  // can be null if not set
+    "is_guest": 0,
    "admin": 0,
    "deactivated": 0,
    "shadow_banned": 0,
-    "password_hash": "$2b$12$p9B4GkqYdRTPGD",
    "creation_ts": 1560432506,
    "appservice_id": null,
    "consent_server_notice_sent": null,
@@ -352,6 +353,11 @@ The following actions are performed when deactivating an user:
 - Remove the user from the user directory
 - Reject all pending invites
 - Remove all account validity information related to the user
+- Remove the arbitrary data store known as *account data*. For example, this includes:
+    - list of ignored users;
+    - push rules;
+    - secret storage keys; and
+    - cross-signing keys.

 The following additional actions are performed during deactivation if `erase`
 is set to `true`:
@@ -365,7 +371,6 @@ The following actions are **NOT** performed. The list may be incomplete.
 - Remove mappings of SSO IDs
 - [Delete media uploaded](#delete-media-uploaded-by-a-user) by user (included avatar images)
 - Delete sent and received messages
- Delete E2E cross-signing keys
 - Remove the user's creation (registration) timestamp
 - [Remove rate limit overrides](#override-ratelimiting-for-users)
 - Remove from monthly active users
@@ -16,6 +16,6 @@ It returns a JSON body like the following:
 ```json
 {
    "server_version": "0.99.2rc1 (b=develop, abcdef123)",
-    "python_version": "3.6.8"
+    "python_version": "3.7.8"
 }
 ```
@@ -20,7 +20,9 @@ recommended for development. More information about WSL can be found at
 <https://docs.microsoft.com/en-us/windows/wsl/install>. Running Synapse natively
 on Windows is not officially supported.

-The code of Synapse is written in Python 3. To do pretty much anything, you'll need [a recent version of Python 3](https://wiki.python.org/moin/BeginnersGuide/Download).
+The code of Synapse is written in Python 3. To do pretty much anything, you'll need [a recent version of Python 3](https://www.python.org/downloads/). Your Python also needs support for [virtual environments](https://docs.python.org/3/library/venv.html). This is usually built-in, but some Linux distributions like Debian and Ubuntu split it out into its own package. Running `sudo apt install python3-venv` should be enough.
+
+Synapse can connect to PostgreSQL via the [psycopg2](https://pypi.org/project/psycopg2/) Python library. Building this library from source requires access to PostgreSQL's C header files. On Debian or Ubuntu Linux, these can be installed with `sudo apt install libpq-dev`.

 The source code of Synapse is hosted on GitHub. You will also need [a recent version of git](https://github.com/git-guides/install-git).

@@ -53,6 +55,7 @@ setup a *virtualenv*, as follows:
 cd path/where/you/have/cloned/the/repository
 python3 -m venv ./env
 source ./env/bin/activate
+pip install wheel
 pip install -e ".[all,dev]"
 pip install tox
 ```
@@ -114,7 +117,7 @@ The linters look at your code and do two things:
 - ensure that your code follows the coding style adopted by the project;
 - catch a number of errors in your code.

-They're pretty fast, don't hesitate!
+The linter takes no time at all to run as soon as you've [downloaded the dependencies into your python virtual environment](#4-install-the-dependencies).

 ```sh
 source ./env/bin/activate
@@ -169,6 +172,27 @@ To increase the log level for the tests, set `SYNAPSE_TEST_LOG_LEVEL`:
 SYNAPSE_TEST_LOG_LEVEL=DEBUG trial tests
 ```

+By default, tests will use an in-memory SQLite database for test data. For additional
+help with debugging, one can use an on-disk SQLite database file instead, in order to
+review database state during and after running tests. This can be done by setting
+the `SYNAPSE_TEST_PERSIST_SQLITE_DB` environment variable. Doing so will cause the
+database state to be stored in a file named `test.db` under the trial process'
+working directory. Typically, this ends up being `_trial_temp/test.db`. For example:
+
+```sh
+SYNAPSE_TEST_PERSIST_SQLITE_DB=1 trial tests
+```
+
+The database file can then be inspected with:
+
+```sh
+sqlite3 _trial_temp/test.db
+```
+
+Note that the database file is cleared at the beginning of each test run. Thus it 
+will always only contain the data generated by the *last run test*. Though generally
+when debugging, one is only running a single test anyway.
+
 ### Running tests under PostgreSQL

 Invoking `trial` as above will use an in-memory SQLite database. This is great for
@@ -96,6 +96,60 @@ Ensure postgres is installed, then run:
 NB at the time of writing, this script predates the split into separate `state`/`main`
 databases so will require updates to handle that correctly.

+## Delta files
+
+Delta files define the steps required to upgrade the database from an earlier version.
+They can be written as either a file containing a series of SQL statements, or a Python
+module.
+
+Synapse remembers which delta files it has applied to a database (they are stored in the
+`applied_schema_deltas` table) and will not re-apply them (even if a given file is
+subsequently updated).
+
+Delta files should be placed in a directory named `synapse/storage/schema/<database>/delta/<version>/`.
+They are applied in alphanumeric order, so  by convention the first two characters
+of the filename should be an integer such as `01`, to put the file in the right order.
+
+### SQL delta files
+
+These should be named `*.sql`, or —  for changes which should only be applied for a
+given database engine — `*.sql.posgres` or `*.sql.sqlite`. For example, a delta which
+adds a new column to the `foo` table might be called `01add_bar_to_foo.sql`.
+
+Note that our SQL parser is a bit simple - it understands comments (`--` and `/*...*/`),
+but complex statements which require a `;` in the middle of them (such as `CREATE
+TRIGGER`) are beyond it and you'll have to use a Python delta file.
+
+### Python delta files
+
+For more flexibility, a delta file can take the form of a python module. These should
+be named `*.py`. Note that database-engine-specific modules are not supported here –
+instead you can write `if isinstance(database_engine, PostgresEngine)` or similar.
+
+A Python delta module should define either or both of the following functions:
+
+```python
+import synapse.config.homeserver
+import synapse.storage.engines
+import synapse.storage.types
+
+
+def run_create(
+    cur: synapse.storage.types.Cursor,
+    database_engine: synapse.storage.engines.BaseDatabaseEngine,
+) -> None:
+    """Called whenever an existing or new database is to be upgraded"""
+    ...
+
+def run_upgrade(
+    cur: synapse.storage.types.Cursor,
+    database_engine: synapse.storage.engines.BaseDatabaseEngine,
+    config: synapse.config.homeserver.HomeServerConfig,
+) -> None:
+    """Called whenever an existing database is to be upgraded."""
+    ...
+```
+
 ## Boolean columns

 Boolean columns require special treatment, since SQLite treats booleans the
@@ -35,7 +35,12 @@ When Synapse is asked to preview a URL it does the following:
   5. If the media is HTML:
      1. Decodes the HTML via the stored file.
      2. Generates an Open Graph response from the HTML.
-      3. If an image exists in the Open Graph response:
+      3. If a JSON oEmbed URL was found in the HTML via autodiscovery:
+         1. Downloads the URL and stores it into a file via the media storage provider
+            and saves the local media metadata.
+         2. Convert the oEmbed response to an Open Graph response.
+         3. Override any Open Graph data from the HTML with data from oEmbed.
+      4. If an image exists in the Open Graph response:
         1. Downloads the URL and stores it into a file via the media storage
            provider and saves the local media metadata.
         2. Generates thumbnails.
@@ -390,9 +390,6 @@ oidc_providers:

 ### Facebook

-Like Github, Facebook provide a custom OAuth2 API rather than an OIDC-compliant
-one so requires a little more configuration.
-
 0. You will need a Facebook developer account. You can register for one
   [here](https://developers.facebook.com/async/registration/).
 1. On the [apps](https://developers.facebook.com/apps/) page of the developer
@@ -412,24 +409,28 @@ Synapse config:
    idp_name: Facebook
    idp_brand: "facebook"  # optional: styling hint for clients
    discover: false
-    issuer: "https://facebook.com"
+    issuer: "https://www.facebook.com"
    client_id: "your-client-id" # TO BE FILLED
    client_secret: "your-client-secret" # TO BE FILLED
    scopes: ["openid", "email"]
-    authorization_endpoint: https://facebook.com/dialog/oauth
-    token_endpoint: https://graph.facebook.com/v9.0/oauth/access_token
-    user_profile_method: "userinfo_endpoint"
-    userinfo_endpoint: "https://graph.facebook.com/v9.0/me?fields=id,name,email,picture"
+    authorization_endpoint: "https://facebook.com/dialog/oauth"
+    token_endpoint: "https://graph.facebook.com/v9.0/oauth/access_token"
+    jwks_uri: "https://www.facebook.com/.well-known/oauth/openid/jwks/"
    user_mapping_provider:
      config:
-        subject_claim: "id"
        display_name_template: "{{ user.name }}"
+        email_template: "{{ '{{ user.email }}' }}"
 ```

 Relevant documents:
- * https://developers.facebook.com/docs/facebook-login/manually-build-a-login-flow
- * Using Facebook's Graph API: https://developers.facebook.com/docs/graph-api/using-graph-api/
- * Reference to the User endpoint: https://developers.facebook.com/docs/graph-api/reference/user
+ * [Manually Build a Login Flow](https://developers.facebook.com/docs/facebook-login/manually-build-a-login-flow)
+ * [Using Facebook's Graph API](https://developers.facebook.com/docs/graph-api/using-graph-api/)
+ * [Reference to the User endpoint](https://developers.facebook.com/docs/graph-api/reference/user)
+
+Facebook do have an [OIDC discovery endpoint](https://www.facebook.com/.well-known/openid-configuration),
+but it has a `response_types_supported` which excludes "code" (which we rely on, and
+is even mentioned in their [documentation](https://developers.facebook.com/docs/facebook-login/manually-build-a-login-flow#login)),
+so we have to disable discovery and configure the URIs manually.

 ### Gitea

@@ -74,13 +74,7 @@ server_name: "SERVERNAME"
 #
 pid_file: DATADIR/homeserver.pid

-# The absolute URL to the web client which /_matrix/client will redirect
-# to if 'webclient' is configured under the 'listeners' configuration.
-#
-# This option can be also set to the filesystem path to the web client
-# which will be served at /_matrix/client/ if 'webclient' is configured
-# under the 'listeners' configuration, however this is a security risk:
-# https://github.com/matrix-org/synapse#security-note
+# The absolute URL to the web client which / will redirect to.
 #
 #web_client_location: https://riot.example.com/

@@ -164,7 +158,7 @@ presence:
 # The default room version for newly created rooms.
 #
 # Known room versions are listed here:
-# https://matrix.org/docs/spec/#complete-list-of-room-versions
+# https://spec.matrix.org/latest/rooms/#complete-list-of-room-versions
 #
 # For example, for room version 1, default_room_version should be set
 # to "1".
@@ -310,8 +304,6 @@ presence:
 #   static: static resources under synapse/static (/_matrix/static). (Mostly
 #       useful for 'fallback authentication'.)
 #
-#   webclient: A web client. Requires web_client_location to be set.
-#
 listeners:
  # TLS-enabled listener: for when matrix traffic is sent directly to synapse.
  #
@@ -1503,6 +1495,21 @@ room_prejoin_state:
   #additional_event_types:
   #  - org.example.custom.event.type

+# We record the IP address of clients used to access the API for various
+# reasons, including displaying it to the user in the "Where you're signed in"
+# dialog.
+#
+# By default, when puppeting another user via the admin API, the client IP
+# address is recorded against the user who created the access token (ie, the
+# admin user), and *not* the puppeted user.
+#
+# Uncomment the following to also record the IP address against the puppeted
+# user. (This also means that the puppeted user will count as an "active" user
+# for the purpose of monthly active user tracking - see 'limit_usage_by_mau' etc
+# above.)
+#
+#track_puppeted_user_ips: true
+

 # A list of application service config files to use
 #
@@ -1870,10 +1877,13 @@ saml2_config:
 #       Defaults to false. Avoid this in production.
 #
 #   user_profile_method: Whether to fetch the user profile from the userinfo
-#       endpoint. Valid values are: 'auto' or 'userinfo_endpoint'.
+#       endpoint, or to rely on the data returned in the id_token from the
+#       token_endpoint.
 #
-#       Defaults to 'auto', which fetches the userinfo endpoint if 'openid' is
-#       included in 'scopes'. Set to 'userinfo_endpoint' to always fetch the
+#       Valid values are: 'auto' or 'userinfo_endpoint'.
+#
+#       Defaults to 'auto', which uses the userinfo endpoint if 'openid' is
+#       not included in 'scopes'. Set to 'userinfo_endpoint' to always use the
 #       userinfo endpoint.
 #
 #   allow_existing_users: set to 'true' to allow a user logging in via OIDC to
@@ -194,7 +194,7 @@ When following this route please make sure that the [Platform-specific prerequis
 System requirements:

 - POSIX-compliant system (tested on Linux & OS X)
- Python 3.6 or later, up to Python 3.9.
+- Python 3.7 or later, up to Python 3.9.
 - At least 1GB of free RAM if you want to join large public rooms like #matrix:matrix.org

 To install the Synapse homeserver run:
@@ -137,6 +137,10 @@ This will install and start a systemd service called `coturn`.

    # TLS private key file
    pkey=/path/to/privkey.pem
+
+    # Ensure the configuration lines that disable TLS/DTLS are commented-out or removed
+    #no-tls
+    #no-dtls
    ```

    In this case, replace the `turn:` schemes in the `turn_uris` settings below
@@ -145,6 +149,14 @@ This will install and start a systemd service called `coturn`.
    We recommend that you only try to set up TLS/DTLS once you have set up a
    basic installation and got it working.

+    NB: If your TLS certificate was provided by Let's Encrypt, TLS/DTLS will
+    not work with any Matrix client that uses Chromium's WebRTC library. This
+    currently includes Element Android & iOS; for more details, see their
+    [respective](https://github.com/vector-im/element-android/issues/1533)
+    [issues](https://github.com/vector-im/element-ios/issues/2712) as well as the underlying
+    [WebRTC issue](https://bugs.chromium.org/p/webrtc/issues/detail?id=11710).
+    Consider using a ZeroSSL certificate for your TURN server as a working alternative.
+
 1.  Ensure your firewall allows traffic into the TURN server on the ports
    you've configured it to listen on (By default: 3478 and 5349 for TURN
    traffic (remember to allow both TCP and UDP traffic), and ports 49152-65535
@@ -250,6 +262,10 @@ Here are a few things to try:
 * Check that you have opened your firewall to allow UDP traffic to the UDP
   relay ports (49152-65535 by default).

+ * Try disabling `coturn`'s TLS/DTLS listeners and enable only its (unencrypted)
+   TCP/UDP listeners. (This will only leave signaling traffic unencrypted;
+   voice & video WebRTC traffic is always encrypted.)
+
 * Some WebRTC implementations (notably, that of Google Chrome) appear to get
   confused by TURN servers which are reachable over IPv6 (this appears to be
   an unexpected side-effect of its handling of multiple IP addresses as
@@ -85,6 +85,17 @@ process, for example:
    dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
    ```

+# Upgrading to v1.51.0
+
+## Deprecation of `webclient` listeners and non-HTTP(S) `web_client_location`
+
+Listeners of type  `webclient` are deprecated and scheduled to be removed in
+Synapse v1.53.0.
+
+Similarly, a non-HTTP(S) `web_client_location` configuration is deprecated and
+will become a configuration error in Synapse v1.53.0.
+
+
 # Upgrading to v1.50.0

 ## Dropping support for old Python and Postgres versions
@@ -86,7 +86,7 @@ The following fields are returned in the JSON response body:
 - `next_token`: string representing a positive integer - Indication for pagination. See above.
 - `total` - integer - Total number of destinations.

-# Destination Details API
+## Destination Details API

 This API gets the retry timing info for a specific remote server.

@@ -108,7 +108,105 @@ A response body like the following is returned:
 }
 ```

+**Parameters**
+
+The following parameters should be set in the URL:
+
+- `destination` - Name of the remote server.
+
 **Response**

 The response fields are the same like in the `destinations` array in
 [List of destinations](#list-of-destinations) response.
+
+## Destination rooms
+
+This API gets the rooms that federate with a specific remote server.
+
+The API is:
+
+```
+GET /_synapse/admin/v1/federation/destinations/<destination>/rooms
+```
+
+A response body like the following is returned:
+
+```json
+{
+   "rooms":[
+      {
+         "room_id": "!OGEhHVWSdvArJzumhm:matrix.org",
+         "stream_ordering": 8326
+      },
+      {
+         "room_id": "!xYvNcQPhnkrdUmYczI:matrix.org",
+         "stream_ordering": 93534
+      }
+   ],
+   "total": 2
+}
+```
+
+To paginate, check for `next_token` and if present, call the endpoint again
+with `from` set to the value of `next_token`. This will return a new page.
+
+If the endpoint does not return a `next_token` then there are no more destinations
+to paginate through.
+
+**Parameters**
+
+The following parameters should be set in the URL:
+
+- `destination` - Name of the remote server.
+
+The following query parameters are available:
+
+- `from` - Offset in the returned list. Defaults to `0`.
+- `limit` - Maximum amount of destinations to return. Defaults to `100`.
+- `dir` - Direction of room order by `room_id`. Either `f` for forwards or `b` for
+  backwards. Defaults to `f`.
+
+**Response**
+
+The following fields are returned in the JSON response body:
+
+- `rooms` - An array of objects, each containing information about a room.
+  Room objects contain the following fields:
+  - `room_id` - string - The ID of the room.
+  - `stream_ordering` - integer -  The stream ordering of the most recent
+    successfully-sent [PDU](understanding_synapse_through_grafana_graphs.md#federation)
+    to this destination in this room.
+- `next_token`: string representing a positive integer - Indication for pagination. See above.
+- `total` - integer - Total number of destinations.
+
+## Reset connection timeout
+
+Synapse makes federation requests to other homeservers. If a federation request fails,
+Synapse will mark the destination homeserver as offline, preventing any future requests
+to that server for a "cooldown" period. This period grows over time if the server
+continues to fail its responses
+([exponential backoff](https://en.wikipedia.org/wiki/Exponential_backoff)).
+
+Admins can cancel the cooldown period with this API.
+
+This API resets the retry timing for a specific remote server and tries to connect to
+the remote server again. It does not wait for the next `retry_interval`.
+The connection must have previously run into an error and `retry_last_ts`
+([Destination Details API](#destination-details-api)) must not be equal to `0`.
+
+The connection attempt is carried out in the background and can take a while
+even if the API already returns the http status 200.
+
+The API is:
+
+```
+POST /_synapse/admin/v1/federation/destinations/<destination>/reset_connection
+
+{}
+```
+
+**Parameters**
+
+The following parameters should be set in the URL:
+
+- `destination` - Name of the remote server.
@@ -8,7 +8,8 @@
 # By default the script will fetch the latest Complement master branch and
 # run tests with that. This can be overridden to use a custom Complement
 # checkout by setting the COMPLEMENT_DIR environment variable to the
-# filepath of a local Complement checkout.
+# filepath of a local Complement checkout or by setting the COMPLEMENT_REF
+# environment variable to pull a different branch or commit.
 #
 # By default Synapse is run in monolith mode. This can be overridden by
 # setting the WORKERS environment variable.
@@ -23,16 +24,20 @@
 # Exit if a line returns a non-zero exit code
 set -e

+# enable buildkit for the docker builds
+export DOCKER_BUILDKIT=1
+
 # Change to the repository root
 cd "$(dirname $0)/.."

 # Check for a user-specified Complement checkout
 if [[ -z "$COMPLEMENT_DIR" ]]; then
-  echo "COMPLEMENT_DIR not set. Fetching the latest Complement checkout..."
-  wget -Nq https://github.com/matrix-org/complement/archive/master.tar.gz
-  tar -xzf master.tar.gz
-  COMPLEMENT_DIR=complement-master
-  echo "Checkout available at 'complement-master'"
+  COMPLEMENT_REF=${COMPLEMENT_REF:-master}
+  echo "COMPLEMENT_DIR not set. Fetching Complement checkout from ${COMPLEMENT_REF}..."
+  wget -Nq https://github.com/matrix-org/complement/archive/${COMPLEMENT_REF}.tar.gz
+  tar -xzf ${COMPLEMENT_REF}.tar.gz
+  COMPLEMENT_DIR=complement-${COMPLEMENT_REF}
+  echo "Checkout available at 'complement-${COMPLEMENT_REF}'"
 fi

 # Build the base Synapse image from the local checkout
@@ -47,7 +52,7 @@ if [[ -n "$WORKERS" ]]; then
  COMPLEMENT_DOCKERFILE=SynapseWorkers.Dockerfile
  # And provide some more configuration to complement.
  export COMPLEMENT_CA=true
-  export COMPLEMENT_VERSION_CHECK_ITERATIONS=500
+  export COMPLEMENT_SPAWN_HS_TIMEOUT_SECS=25
 else
  export COMPLEMENT_BASE_IMAGE=complement-synapse
  COMPLEMENT_DOCKERFILE=Synapse.Dockerfile
@@ -65,4 +70,5 @@ if [[ -n "$1" ]]; then
 fi

 # Run the tests!
+echo "Images built; running complement"
 go test -v -tags synapse_blacklist,msc2403 -count=1 $EXTRA_COMPLEMENT_ARGS ./tests/...
@@ -150,7 +150,7 @@ setup(
    zip_safe=False,
    long_description=long_description,
    long_description_content_type="text/x-rst",
-    python_requires="~=3.6",
+    python_requires="~=3.7",
    entry_points={
        "console_scripts": [
            "synapse_homeserver = synapse.app.homeserver:main",
@@ -21,8 +21,8 @@ import os
 import sys

 # Check that we're not running on an unsupported Python version.
-if sys.version_info < (3, 6):
-    print("Synapse requires Python 3.6 or above.")
+if sys.version_info < (3, 7):
+    print("Synapse requires Python 3.7 or above.")
    sys.exit(1)

 # Twisted and canonicaljson will fail to import when this file is executed to
@@ -47,7 +47,7 @@ try:
 except ImportError:
    pass

-__version__ = "1.50.0rc2"
+__version__ = "1.51.0"

 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
    # We import here so that we don't have to install a bunch of deps when
@@ -46,7 +46,9 @@ class UserInfo:
    ips: List[str] = attr.Factory(list)


-def get_recent_users(txn: LoggingTransaction, since_ms: int) -> List[UserInfo]:
+def get_recent_users(
+    txn: LoggingTransaction, since_ms: int, exclude_app_service: bool
+) -> List[UserInfo]:
    """Fetches recently registered users and some info on them."""

    sql = """
@@ -56,6 +58,9 @@ def get_recent_users(txn: LoggingTransaction, since_ms: int) -> List[UserInfo]:
            AND deactivated = 0
    """

+    if exclude_app_service:
+        sql += " AND appservice_id IS NULL"
+
    txn.execute(sql, (since_ms / 1000,))

    user_infos = [UserInfo(user_id, creation_ts) for user_id, creation_ts in txn]
@@ -113,7 +118,7 @@ def main() -> None:
        "-e",
        "--exclude-emails",
        action="store_true",
-        help="Exclude users that have validated email addresses",
+        help="Exclude users that have validated email addresses.",
    )
    parser.add_argument(
        "-u",
@@ -121,6 +126,12 @@ def main() -> None:
        action="store_true",
        help="Only print user IDs that match.",
    )
+    parser.add_argument(
+        "-a",
+        "--exclude-app-service",
+        help="Exclude appservice users.",
+        action="store_true",
+    )

    config = ReviewConfig()

@@ -133,6 +144,7 @@ def main() -> None:

    since_ms = time.time() * 1000 - Config.parse_duration(config_args.since)
    exclude_users_with_email = config_args.exclude_emails
+    exclude_users_with_appservice = config_args.exclude_app_service
    include_context = not config_args.only_users

    for database_config in config.database.databases:
@@ -143,7 +155,7 @@ def main() -> None:

    with make_conn(database_config, engine, "review_recent_signups") as db_conn:
        # This generates a type of Cursor, not LoggingTransaction.
-        user_infos = get_recent_users(db_conn.cursor(), since_ms)  # type: ignore[arg-type]
+        user_infos = get_recent_users(db_conn.cursor(), since_ms, exclude_users_with_appservice)  # type: ignore[arg-type]

    for user_info in user_infos:
        if exclude_users_with_email and user_info.emails:
@@ -71,6 +71,7 @@ class Auth:
        self._auth_blocking = AuthBlocking(self.hs)

        self._track_appservice_user_ips = hs.config.appservice.track_appservice_user_ips
+        self._track_puppeted_user_ips = hs.config.api.track_puppeted_user_ips
        self._macaroon_secret_key = hs.config.key.macaroon_secret_key
        self._force_tracing_for_users = hs.config.tracing.force_tracing_for_users

@@ -246,6 +247,18 @@ class Auth:
                    user_agent=user_agent,
                    device_id=device_id,
                )
+                # Track also the puppeted user client IP if enabled and the user is puppeting
+                if (
+                    user_info.user_id != user_info.token_owner
+                    and self._track_puppeted_user_ips
+                ):
+                    await self.store.insert_client_ip(
+                        user_id=user_info.user_id,
+                        access_token=access_token,
+                        ip=ip_addr,
+                        user_agent=user_agent,
+                        device_id=device_id,
+                    )

            if is_guest and not allow_guest:
                raise AuthError(
@@ -46,41 +46,41 @@ class RoomDisposition:
    UNSTABLE = "unstable"


-@attr.s(slots=True, frozen=True)
+@attr.s(slots=True, frozen=True, auto_attribs=True)
 class RoomVersion:
    """An object which describes the unique attributes of a room version."""

-    identifier = attr.ib(type=str)  # the identifier for this version
-    disposition = attr.ib(type=str)  # one of the RoomDispositions
-    event_format = attr.ib(type=int)  # one of the EventFormatVersions
-    state_res = attr.ib(type=int)  # one of the StateResolutionVersions
-    enforce_key_validity = attr.ib(type=bool)
+    identifier: str  # the identifier for this version
+    disposition: str  # one of the RoomDispositions
+    event_format: int  # one of the EventFormatVersions
+    state_res: int  # one of the StateResolutionVersions
+    enforce_key_validity: bool

    # Before MSC2432, m.room.aliases had special auth rules and redaction rules
-    special_case_aliases_auth = attr.ib(type=bool)
+    special_case_aliases_auth: bool
    # Strictly enforce canonicaljson, do not allow:
    # * Integers outside the range of [-2 ^ 53 + 1, 2 ^ 53 - 1]
    # * Floats
    # * NaN, Infinity, -Infinity
-    strict_canonicaljson = attr.ib(type=bool)
+    strict_canonicaljson: bool
    # MSC2209: Check 'notifications' key while verifying
    # m.room.power_levels auth rules.
-    limit_notifications_power_levels = attr.ib(type=bool)
+    limit_notifications_power_levels: bool
    # MSC2174/MSC2176: Apply updated redaction rules algorithm.
-    msc2176_redaction_rules = attr.ib(type=bool)
+    msc2176_redaction_rules: bool
    # MSC3083: Support the 'restricted' join_rule.
-    msc3083_join_rules = attr.ib(type=bool)
+    msc3083_join_rules: bool
    # MSC3375: Support for the proper redaction rules for MSC3083. This mustn't
    #          be enabled if MSC3083 is not.
-    msc3375_redaction_rules = attr.ib(type=bool)
+    msc3375_redaction_rules: bool
    # MSC2403: Allows join_rules to be set to 'knock', changes auth rules to allow sending
    # m.room.membership event with membership 'knock'.
-    msc2403_knocking = attr.ib(type=bool)
+    msc2403_knocking: bool
    # MSC2716: Adds m.room.power_levels -> content.historical field to control
    # whether "insertion", "chunk", "marker" events can be sent
-    msc2716_historical = attr.ib(type=bool)
+    msc2716_historical: bool
    # MSC2716: Adds support for redacting "insertion", "chunk", and "marker" events
-    msc2716_redactions = attr.ib(type=bool)
+    msc2716_redactions: bool


 class RoomVersions:
@@ -16,7 +16,6 @@ import atexit
 import gc
 import logging
 import os
-import platform
 import signal
 import socket
 import sys
@@ -60,7 +59,7 @@ from synapse.events.spamcheck import load_legacy_spam_checkers
 from synapse.events.third_party_rules import load_legacy_third_party_event_rules
 from synapse.handlers.auth import load_legacy_password_auth_providers
 from synapse.logging.context import PreserveLoggingContext
-from synapse.metrics import register_threadpool
+from synapse.metrics import install_gc_manager, register_threadpool
 from synapse.metrics.background_process_metrics import wrap_as_background_process
 from synapse.metrics.jemalloc import setup_jemalloc_stats
 from synapse.types import ISynapseReactor
@@ -159,6 +158,7 @@ def start_reactor(
        change_resource_limit(soft_file_limit)
        if gc_thresholds:
            gc.set_threshold(*gc_thresholds)
+        install_gc_manager()
        run_command()

    # make sure that we run the reactor with the sentinel log context,
@@ -435,7 +435,8 @@ async def start(hs: "HomeServer") -> None:
    # before we start the listeners.
    module_api = hs.get_module_api()
    for module, config in hs.config.modules.loaded_modules:
-        module(config=config, api=module_api)
+        m = module(config=config, api=module_api)
+        logger.info("Loaded module %s", m)

    load_legacy_spam_checkers(hs)
    load_legacy_third_party_event_rules(hs)
@@ -467,15 +468,13 @@ async def start(hs: "HomeServer") -> None:
    # everything currently allocated are things that will be used for the
    # rest of time. Doing so means less work each GC (hopefully).
    #
-    # This only works on Python 3.7
-    if platform.python_implementation() == "CPython" and sys.version_info >= (3, 7):
+    # PyPy does not (yet?) implement gc.freeze()
+    if hasattr(gc, "freeze"):
        gc.collect()
        gc.freeze()

-    # Speed up shutdowns by freezing all allocated objects. This moves everything
-    # into the permanent generation and excludes them from the final GC.
-    # Unfortunately only works on Python 3.7
-    if platform.python_implementation() == "CPython" and sys.version_info >= (3, 7):
+        # Speed up shutdowns by freezing all allocated objects. This moves everything
+        # into the permanent generation and excludes them from the final GC.
        atexit.register(gc.freeze)


@@ -27,7 +27,6 @@ import synapse
 import synapse.config.logger
 from synapse import events
 from synapse.api.urls import (
-    CLIENT_API_PREFIX,
    FEDERATION_PREFIX,
    LEGACY_MEDIA_PREFIX,
    MEDIA_R0_PREFIX,
@@ -132,9 +131,18 @@ class SynapseHomeServer(HomeServer):
        resources.update(self._module_web_resources)
        self._module_web_resources_consumed = True

-        # try to find something useful to redirect '/' to
-        if WEB_CLIENT_PREFIX in resources:
-            root_resource: Resource = RootOptionsRedirectResource(WEB_CLIENT_PREFIX)
+        # Try to find something useful to serve at '/':
+        #
+        # 1. Redirect to the web client if it is an HTTP(S) URL.
+        # 2. Redirect to the web client served via Synapse.
+        # 3. Redirect to the static "Synapse is running" page.
+        # 4. Do not redirect and use a blank resource.
+        if self.config.server.web_client_location_is_redirect:
+            root_resource: Resource = RootOptionsRedirectResource(
+                self.config.server.web_client_location
+            )
+        elif WEB_CLIENT_PREFIX in resources:
+            root_resource = RootOptionsRedirectResource(WEB_CLIENT_PREFIX)
        elif STATIC_PREFIX in resources:
            root_resource = RootOptionsRedirectResource(STATIC_PREFIX)
        else:
@@ -193,7 +201,13 @@ class SynapseHomeServer(HomeServer):

            resources.update(
                {
-                    CLIENT_API_PREFIX: client_resource,
+                    "/_matrix/client/api/v1": client_resource,
+                    "/_matrix/client/r0": client_resource,
+                    "/_matrix/client/v1": client_resource,
+                    "/_matrix/client/v3": client_resource,
+                    "/_matrix/client/unstable": client_resource,
+                    "/_matrix/client/v2_alpha": client_resource,
+                    "/_matrix/client/versions": client_resource,
                    "/.well-known": well_known_resource(self),
                    "/_synapse/admin": AdminRestResource(self),
                    **build_synapse_client_resource_tree(self),
@@ -257,15 +271,15 @@ class SynapseHomeServer(HomeServer):
            resources[SERVER_KEY_V2_PREFIX] = KeyApiV2Resource(self)

        if name == "webclient":
+            # webclient listeners are deprecated as of Synapse v1.51.0, remove it
+            # in > v1.53.0.
            webclient_loc = self.config.server.web_client_location

            if webclient_loc is None:
                logger.warning(
                    "Not enabling webclient resource, as web_client_location is unset."
                )
-            elif webclient_loc.startswith("http://") or webclient_loc.startswith(
-                "https://"
-            ):
+            elif self.config.server.web_client_location_is_redirect:
                resources[WEB_CLIENT_PREFIX] = RootRedirect(webclient_loc)
            else:
                logger.warning(
@@ -29,6 +29,7 @@ class ApiConfig(Config):
    def read_config(self, config: JsonDict, **kwargs):
        validate_config(_MAIN_SCHEMA, config, ())
        self.room_prejoin_state = list(self._get_prejoin_state_types(config))
+        self.track_puppeted_user_ips = config.get("track_puppeted_user_ips", False)

    def generate_config_section(cls, **kwargs) -> str:
        formatted_default_state_types = "\n".join(
@@ -59,6 +60,21 @@ class ApiConfig(Config):
           #
           #additional_event_types:
           #  - org.example.custom.event.type
+
+        # We record the IP address of clients used to access the API for various
+        # reasons, including displaying it to the user in the "Where you're signed in"
+        # dialog.
+        #
+        # By default, when puppeting another user via the admin API, the client IP
+        # address is recorded against the user who created the access token (ie, the
+        # admin user), and *not* the puppeted user.
+        #
+        # Uncomment the following to also record the IP address against the puppeted
+        # user. (This also means that the puppeted user will count as an "active" user
+        # for the purpose of monthly active user tracking - see 'limit_usage_by_mau' etc
+        # above.)
+        #
+        #track_puppeted_user_ips: true
        """ % {
            "formatted_default_state_types": formatted_default_state_types
        }
@@ -138,5 +154,8 @@ _MAIN_SCHEMA = {
    "properties": {
        "room_prejoin_state": _ROOM_PREJOIN_STATE_CONFIG_SCHEMA,
        "room_invite_state_types": _ROOM_INVITE_STATE_TYPES_SCHEMA,
+        "track_puppeted_user_ips": {
+            "type": "boolean",
+        },
    },
 }
@@ -55,19 +55,19 @@ https://matrix-org.github.io/synapse/latest/templates.html
 ---------------------------------------------------------------------------------------"""


-@attr.s(slots=True, frozen=True)
+@attr.s(slots=True, frozen=True, auto_attribs=True)
 class EmailSubjectConfig:
-    message_from_person_in_room = attr.ib(type=str)
-    message_from_person = attr.ib(type=str)
-    messages_from_person = attr.ib(type=str)
-    messages_in_room = attr.ib(type=str)
-    messages_in_room_and_others = attr.ib(type=str)
-    messages_from_person_and_others = attr.ib(type=str)
-    invite_from_person = attr.ib(type=str)
-    invite_from_person_to_room = attr.ib(type=str)
-    invite_from_person_to_space = attr.ib(type=str)
-    password_reset = attr.ib(type=str)
-    email_validation = attr.ib(type=str)
+    message_from_person_in_room: str
+    message_from_person: str
+    messages_from_person: str
+    messages_in_room: str
+    messages_in_room_and_others: str
+    messages_from_person_and_others: str
+    invite_from_person: str
+    invite_from_person_to_room: str
+    invite_from_person_to_space: str
+    password_reset: str
+    email_validation: str


 class EmailConfig(Config):
@@ -148,10 +148,13 @@ class OIDCConfig(Config):
        #       Defaults to false. Avoid this in production.
        #
        #   user_profile_method: Whether to fetch the user profile from the userinfo
-        #       endpoint. Valid values are: 'auto' or 'userinfo_endpoint'.
+        #       endpoint, or to rely on the data returned in the id_token from the
+        #       token_endpoint.
        #
-        #       Defaults to 'auto', which fetches the userinfo endpoint if 'openid' is
-        #       included in 'scopes'. Set to 'userinfo_endpoint' to always fetch the
+        #       Valid values are: 'auto' or 'userinfo_endpoint'.
+        #
+        #       Defaults to 'auto', which uses the userinfo endpoint if 'openid' is
+        #       not included in 'scopes'. Set to 'userinfo_endpoint' to always use the
        #       userinfo endpoint.
        #
        #   allow_existing_users: set to 'true' to allow a user logging in via OIDC to
@@ -200,8 +200,8 @@ class HttpListenerConfig:
    """Object describing the http-specific parts of the config of a listener"""

    x_forwarded: bool = False
-    resources: List[HttpResourceConfig] = attr.ib(factory=list)
-    additional_resources: Dict[str, dict] = attr.ib(factory=dict)
+    resources: List[HttpResourceConfig] = attr.Factory(list)
+    additional_resources: Dict[str, dict] = attr.Factory(dict)
    tag: Optional[str] = None


@@ -259,7 +259,6 @@ class ServerConfig(Config):
            raise ConfigError(str(e))

        self.pid_file = self.abspath(config.get("pid_file"))
-        self.web_client_location = config.get("web_client_location", None)
        self.soft_file_limit = config.get("soft_file_limit", 0)
        self.daemonize = config.get("daemonize")
        self.print_pidfile = config.get("print_pidfile")
@@ -506,8 +505,17 @@ class ServerConfig(Config):
                    l2.append(listener)
            self.listeners = l2

-        if not self.web_client_location:
-            _warn_if_webclient_configured(self.listeners)
+        self.web_client_location = config.get("web_client_location", None)
+        self.web_client_location_is_redirect = self.web_client_location and (
+            self.web_client_location.startswith("http://")
+            or self.web_client_location.startswith("https://")
+        )
+        # A non-HTTP(S) web client location is deprecated.
+        if self.web_client_location and not self.web_client_location_is_redirect:
+            logger.warning(NO_MORE_NONE_HTTP_WEB_CLIENT_LOCATION_WARNING)
+
+        # Warn if webclient is configured for a worker.
+        _warn_if_webclient_configured(self.listeners)

        self.gc_thresholds = read_gc_thresholds(config.get("gc_thresholds", None))
        self.gc_seconds = self.read_gc_intervals(config.get("gc_min_interval", None))
@@ -793,13 +801,7 @@ class ServerConfig(Config):
        #
        pid_file: %(pid_file)s

-        # The absolute URL to the web client which /_matrix/client will redirect
-        # to if 'webclient' is configured under the 'listeners' configuration.
-        #
-        # This option can be also set to the filesystem path to the web client
-        # which will be served at /_matrix/client/ if 'webclient' is configured
-        # under the 'listeners' configuration, however this is a security risk:
-        # https://github.com/matrix-org/synapse#security-note
+        # The absolute URL to the web client which / will redirect to.
        #
        #web_client_location: https://riot.example.com/

@@ -883,7 +885,7 @@ class ServerConfig(Config):
        # The default room version for newly created rooms.
        #
        # Known room versions are listed here:
-        # https://matrix.org/docs/spec/#complete-list-of-room-versions
+        # https://spec.matrix.org/latest/rooms/#complete-list-of-room-versions
        #
        # For example, for room version 1, default_room_version should be set
        # to "1".
@@ -1011,8 +1013,6 @@ class ServerConfig(Config):
        #   static: static resources under synapse/static (/_matrix/static). (Mostly
        #       useful for 'fallback authentication'.)
        #
-        #   webclient: A web client. Requires web_client_location to be set.
-        #
        listeners:
          # TLS-enabled listener: for when matrix traffic is sent directly to synapse.
          #
@@ -1349,9 +1349,15 @@ def parse_listener_def(listener: Any) -> ListenerConfig:
    return ListenerConfig(port, bind_addresses, listener_type, tls, http_config)


+NO_MORE_NONE_HTTP_WEB_CLIENT_LOCATION_WARNING = """
+Synapse no longer supports serving a web client. To remove this warning,
+configure 'web_client_location' with an HTTP(S) URL.
+"""
+
+
 NO_MORE_WEB_CLIENT_WARNING = """
-Synapse no longer includes a web client. To enable a web client, configure
-web_client_location. To remove this warning, remove 'webclient' from the 'listeners'
+Synapse no longer includes a web client. To redirect the root resource to a web client, configure
+'web_client_location'. To remove this warning, remove 'webclient' from the 'listeners'
 configuration.
 """

@@ -51,12 +51,12 @@ def _instance_to_list_converter(obj: Union[str, List[str]]) -> List[str]:
    return obj


-@attr.s
+@attr.s(auto_attribs=True)
 class InstanceLocationConfig:
    """The host and port to talk to an instance via HTTP replication."""

-    host = attr.ib(type=str)
-    port = attr.ib(type=int)
+    host: str
+    port: int


@attr.s
@@ -77,34 +77,28 @@ class WriterLocations:
            can only be a single instance.
    """

-    events = attr.ib(
+    events: List[str] = attr.ib(
        default=["master"],
-        type=List[str],
        converter=_instance_to_list_converter,
    )
-    typing = attr.ib(
+    typing: List[str] = attr.ib(
        default=["master"],
-        type=List[str],
        converter=_instance_to_list_converter,
    )
-    to_device = attr.ib(
+    to_device: List[str] = attr.ib(
        default=["master"],
-        type=List[str],
        converter=_instance_to_list_converter,
    )
-    account_data = attr.ib(
+    account_data: List[str] = attr.ib(
        default=["master"],
-        type=List[str],
        converter=_instance_to_list_converter,
    )
-    receipts = attr.ib(
+    receipts: List[str] = attr.ib(
        default=["master"],
-        type=List[str],
        converter=_instance_to_list_converter,
    )
-    presence = attr.ib(
+    presence: List[str] = attr.ib(
        default=["master"],
-        type=List[str],
        converter=_instance_to_list_converter,
    )

@@ -58,7 +58,7 @@ if TYPE_CHECKING:
 logger = logging.getLogger(__name__)


-@attr.s(slots=True, cmp=False)
+@attr.s(slots=True, frozen=True, cmp=False, auto_attribs=True)
 class VerifyJsonRequest:
    """
    A request to verify a JSON object.
@@ -78,10 +78,10 @@ class VerifyJsonRequest:
        key_ids: The set of key_ids to that could be used to verify the JSON object
    """

-    server_name = attr.ib(type=str)
-    get_json_object = attr.ib(type=Callable[[], JsonDict])
-    minimum_valid_until_ts = attr.ib(type=int)
-    key_ids = attr.ib(type=List[str])
+    server_name: str
+    get_json_object: Callable[[], JsonDict]
+    minimum_valid_until_ts: int
+    key_ids: List[str]

    @staticmethod
    def from_json_object(
@@ -124,7 +124,7 @@ class KeyLookupError(ValueError):
    pass


-@attr.s(slots=True)
+@attr.s(slots=True, frozen=True, auto_attribs=True)
 class _FetchKeyRequest:
    """A request for keys for a given server.

@@ -138,9 +138,9 @@ class _FetchKeyRequest:
        key_ids: The IDs of the keys to attempt to fetch
    """

-    server_name = attr.ib(type=str)
-    minimum_valid_until_ts = attr.ib(type=int)
-    key_ids = attr.ib(type=List[str])
+    server_name: str
+    minimum_valid_until_ts: int
+    key_ids: List[str]


 class Keyring:
@@ -315,10 +315,11 @@ class EventBase(metaclass=abc.ABCMeta):
    redacts: DefaultDictProperty[Optional[str]] = DefaultDictProperty("redacts", None)
    room_id: DictProperty[str] = DictProperty("room_id")
    sender: DictProperty[str] = DictProperty("sender")
-    # TODO state_key should be Optional[str], this is generally asserted in Synapse
-    # by calling is_state() first (which ensures this), but it is hard (not possible?)
+    # TODO state_key should be Optional[str]. This is generally asserted in Synapse
+    # by calling is_state() first (which ensures it is not None), but it is hard (not possible?)
    # to properly annotate that calling is_state() asserts that state_key exists
-    # and is non-None.
+    # and is non-None. It would be better to replace such direct references with
+    # get_state_key() (and a check for None).
    state_key: DictProperty[str] = DictProperty("state_key")
    type: DictProperty[str] = DictProperty("type")
    user_id: DictProperty[str] = DictProperty("sender")
@@ -332,7 +333,11 @@ class EventBase(metaclass=abc.ABCMeta):
        return self.content["membership"]

    def is_state(self) -> bool:
-        return hasattr(self, "state_key") and self.state_key is not None
+        return self.get_state_key() is not None
+
+    def get_state_key(self) -> Optional[str]:
+        """Get the state key of this event, or None if it's not a state event"""
+        return self._dict.get("state_key")

    def get_dict(self) -> JsonDict:
        d = dict(self._dict)
@@ -28,7 +28,7 @@ if TYPE_CHECKING:
    from synapse.storage.databases.main import DataStore


-@attr.s(slots=True)
+@attr.s(slots=True, auto_attribs=True)
 class EventContext:
    """
    Holds information relevant to persisting an event
@@ -103,15 +103,15 @@ class EventContext:
            accessed via get_prev_state_ids.
    """

-    rejected = attr.ib(default=False, type=Union[bool, str])
-    _state_group = attr.ib(default=None, type=Optional[int])
-    state_group_before_event = attr.ib(default=None, type=Optional[int])
-    prev_group = attr.ib(default=None, type=Optional[int])
-    delta_ids = attr.ib(default=None, type=Optional[StateMap[str]])
-    app_service = attr.ib(default=None, type=Optional[ApplicationService])
+    rejected: Union[bool, str] = False
+    _state_group: Optional[int] = None
+    state_group_before_event: Optional[int] = None
+    prev_group: Optional[int] = None
+    delta_ids: Optional[StateMap[str]] = None
+    app_service: Optional[ApplicationService] = None

-    _current_state_ids = attr.ib(default=None, type=Optional[StateMap[str]])
-    _prev_state_ids = attr.ib(default=None, type=Optional[StateMap[str]])
+    _current_state_ids: Optional[StateMap[str]] = None
+    _prev_state_ids: Optional[StateMap[str]] = None

    @staticmethod
    def with_state(
@@ -163,7 +163,7 @@ class EventContext:
        return {
            "prev_state_id": prev_state_id,
            "event_type": event.type,
-            "event_state_key": event.state_key if event.is_state() else None,
+            "event_state_key": event.get_state_key(),
            "state_group": self._state_group,
            "state_group_before_event": self.state_group_before_event,
            "rejected": self.rejected,
@@ -14,17 +14,7 @@
 # limitations under the License.
 import collections.abc
 import re
-from typing import (
-    TYPE_CHECKING,
-    Any,
-    Callable,
-    Dict,
-    Iterable,
-    List,
-    Mapping,
-    Optional,
-    Union,
-)
+from typing import Any, Callable, Dict, Iterable, List, Mapping, Optional, Union

 from frozendict import frozendict

@@ -32,14 +22,10 @@ from synapse.api.constants import EventContentFields, EventTypes, RelationTypes
 from synapse.api.errors import Codes, SynapseError
 from synapse.api.room_versions import RoomVersion
 from synapse.types import JsonDict
-from synapse.util.async_helpers import yieldable_gather_results
 from synapse.util.frozenutils import unfreeze

 from . import EventBase

-if TYPE_CHECKING:
-    from synapse.server import HomeServer
-
 # Split strings on "." but not "\." This uses a negative lookbehind assertion for '\'
 # (?<!stuff) matches if the current position in the string is not preceded
 # by a match for 'stuff'.
@@ -385,17 +371,12 @@ class EventClientSerializer:
    clients.
    """

-    def __init__(self, hs: "HomeServer"):
-        self.store = hs.get_datastore()
-        self._msc1849_enabled = hs.config.experimental.msc1849_enabled
-        self._msc3440_enabled = hs.config.experimental.msc3440_enabled
-
-    async def serialize_event(
+    def serialize_event(
        self,
        event: Union[JsonDict, EventBase],
        time_now: int,
        *,
-        bundle_aggregations: bool = False,
+        bundle_aggregations: Optional[Dict[str, JsonDict]] = None,
        **kwargs: Any,
    ) -> JsonDict:
        """Serializes a single event.
@@ -418,66 +399,41 @@ class EventClientSerializer:
        serialized_event = serialize_event(event, time_now, **kwargs)

        # Check if there are any bundled aggregations to include with the event.
-        #
-        # Do not bundle aggregations if any of the following at true:
-        #
-        # * Support is disabled via the configuration or the caller.
-        # * The event is a state event.
-        # * The event has been redacted.
-        if (
-            self._msc1849_enabled
-            and bundle_aggregations
-            and not event.is_state()
-            and not event.internal_metadata.is_redacted()
-        ):
-            await self._injected_bundled_aggregations(event, time_now, serialized_event)
+        if bundle_aggregations:
+            event_aggregations = bundle_aggregations.get(event.event_id)
+            if event_aggregations:
+                self._inject_bundled_aggregations(
+                    event,
+                    time_now,
+                    bundle_aggregations[event.event_id],
+                    serialized_event,
+                )

        return serialized_event

-    async def _injected_bundled_aggregations(
-        self, event: EventBase, time_now: int, serialized_event: JsonDict
+    def _inject_bundled_aggregations(
+        self,
+        event: EventBase,
+        time_now: int,
+        aggregations: JsonDict,
+        serialized_event: JsonDict,
    ) -> None:
        """Potentially injects bundled aggregations into the unsigned portion of the serialized event.

        Args:
            event: The event being serialized.
            time_now: The current time in milliseconds
+            aggregations: The bundled aggregation to serialize.
            serialized_event: The serialized event which may be modified.

        """
-        # Do not bundle aggregations for an event which represents an edit or an
-        # annotation. It does not make sense for them to have related events.
-        relates_to = event.content.get("m.relates_to")
-        if isinstance(relates_to, (dict, frozendict)):
-            relation_type = relates_to.get("rel_type")
-            if relation_type in (RelationTypes.ANNOTATION, RelationTypes.REPLACE):
-                return
+        # Make a copy in-case the object is cached.
+        aggregations = aggregations.copy()

-        event_id = event.event_id
-        room_id = event.room_id
-
-        # The bundled aggregations to include.
-        aggregations = {}
-
-        annotations = await self.store.get_aggregation_groups_for_event(
-            event_id, room_id
-        )
-        if annotations.chunk:
-            aggregations[RelationTypes.ANNOTATION] = annotations.to_dict()
-
-        references = await self.store.get_relations_for_event(
-            event_id, room_id, RelationTypes.REFERENCE, direction="f"
-        )
-        if references.chunk:
-            aggregations[RelationTypes.REFERENCE] = references.to_dict()
-
-        edit = None
-        if event.type == EventTypes.Message:
-            edit = await self.store.get_applicable_edit(event_id, room_id)
-
-        if edit:
+        if RelationTypes.REPLACE in aggregations:
            # If there is an edit replace the content, preserving existing
            # relations.
+            edit = aggregations[RelationTypes.REPLACE]

            # Ensure we take copies of the edit content, otherwise we risk modifying
            # the original event.
@@ -502,27 +458,19 @@ class EventClientSerializer:
            }

        # If this event is the start of a thread, include a summary of the replies.
-        if self._msc3440_enabled:
-            (
-                thread_count,
-                latest_thread_event,
-            ) = await self.store.get_thread_summary(event_id, room_id)
-            if latest_thread_event:
-                aggregations[RelationTypes.THREAD] = {
-                    # Don't bundle aggregations as this could recurse forever.
-                    "latest_event": await self.serialize_event(
-                        latest_thread_event, time_now, bundle_aggregations=False
-                    ),
-                    "count": thread_count,
-                }
+        if RelationTypes.THREAD in aggregations:
+            # Serialize the latest thread event.
+            latest_thread_event = aggregations[RelationTypes.THREAD]["latest_event"]

-        # If any bundled aggregations were found, include them.
-        if aggregations:
-            serialized_event["unsigned"].setdefault("m.relations", {}).update(
-                aggregations
+            # Don't bundle aggregations as this could recurse forever.
+            aggregations[RelationTypes.THREAD]["latest_event"] = self.serialize_event(
+                latest_thread_event, time_now, bundle_aggregations=None
            )

-    async def serialize_events(
+        # Include the bundled aggregations in the event.
+        serialized_event["unsigned"].setdefault("m.relations", {}).update(aggregations)
+
+    def serialize_events(
        self, events: Iterable[Union[JsonDict, EventBase]], time_now: int, **kwargs: Any
    ) -> List[JsonDict]:
        """Serializes multiple events.
@@ -535,9 +483,9 @@ class EventClientSerializer:
        Returns:
            The list of serialized events
        """
-        return await yieldable_gather_results(
-            self.serialize_event, events, time_now=time_now, **kwargs
-        )
+        return [
+            self.serialize_event(event, time_now=time_now, **kwargs) for event in events
+        ]


 def copy_power_levels_contents(
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import collections.abc
-from typing import Iterable, Union
+from typing import Iterable, Type, Union

 import jsonschema

@@ -246,7 +246,7 @@ POWER_LEVELS_SCHEMA = {

 # This could return something newer than Draft 7, but that's the current "latest"
 # validator.
-def _create_power_level_validator() -> jsonschema.Draft7Validator:
+def _create_power_level_validator() -> Type[jsonschema.Draft7Validator]:
    validator = jsonschema.validators.validator_for(POWER_LEVELS_SCHEMA)

    # by default jsonschema does not consider a frozendict to be an object so
@@ -230,6 +230,10 @@ def event_from_pdu_json(pdu_json: JsonDict, room_version: RoomVersion) -> EventB
    # origin, etc etc)
    assert_params_in_dict(pdu_json, ("type", "depth"))

+    # Strip any unauthorized values from "unsigned" if they exist
+    if "unsigned" in pdu_json:
+        _strip_unsigned_values(pdu_json)
+
    depth = pdu_json["depth"]
    if not isinstance(depth, int):
        raise SynapseError(400, "Depth %r not an intger" % (depth,), Codes.BAD_JSON)
@@ -245,3 +249,24 @@ def event_from_pdu_json(pdu_json: JsonDict, room_version: RoomVersion) -> EventB

    event = make_event_from_dict(pdu_json, room_version)
    return event
+
+
+def _strip_unsigned_values(pdu_dict: JsonDict) -> None:
+    """
+    Strip any unsigned values unless specifically allowed, as defined by the whitelist.
+
+    pdu: the json dict to strip values from. Note that the dict is mutated by this
+    function
+    """
+    unsigned = pdu_dict["unsigned"]
+
+    if not isinstance(unsigned, dict):
+        pdu_dict["unsigned"] = {}
+
+    if pdu_dict["type"] == "m.room.member":
+        whitelist = ["knock_room_state", "invite_room_state", "age"]
+    else:
+        whitelist = ["age"]
+
+    filtered_unsigned = {k: v for k, v in unsigned.items() if k in whitelist}
+    pdu_dict["unsigned"] = filtered_unsigned
@@ -56,7 +56,6 @@ from synapse.api.room_versions import (
 from synapse.events import EventBase, builder
 from synapse.federation.federation_base import FederationBase, event_from_pdu_json
 from synapse.federation.transport.client import SendJoinResponse
-from synapse.logging.utils import log_function
 from synapse.types import JsonDict, get_domain_from_id
 from synapse.util.async_helpers import concurrently_execute
 from synapse.util.caches.expiringcache import ExpiringCache
@@ -119,7 +118,8 @@ class FederationClient(FederationBase):
        # It is a map of (room ID, suggested-only) -> the response of
        # get_room_hierarchy.
        self._get_room_hierarchy_cache: ExpiringCache[
-            Tuple[str, bool], Tuple[JsonDict, Sequence[JsonDict], Sequence[str]]
+            Tuple[str, bool],
+            Tuple[JsonDict, Sequence[JsonDict], Sequence[JsonDict], Sequence[str]],
        ] = ExpiringCache(
            cache_name="get_room_hierarchy_cache",
            clock=self._clock,
@@ -144,7 +144,6 @@ class FederationClient(FederationBase):
            if destination_dict:
                self.pdu_destination_tried[event_id] = destination_dict

-    @log_function
    async def make_query(
        self,
        destination: str,
@@ -178,7 +177,6 @@ class FederationClient(FederationBase):
            ignore_backoff=ignore_backoff,
        )

-    @log_function
    async def query_client_keys(
        self, destination: str, content: JsonDict, timeout: int
    ) -> JsonDict:
@@ -196,7 +194,6 @@ class FederationClient(FederationBase):
            destination, content, timeout
        )

-    @log_function
    async def query_user_devices(
        self, destination: str, user_id: str, timeout: int = 30000
    ) -> JsonDict:
@@ -208,7 +205,6 @@ class FederationClient(FederationBase):
            destination, user_id, timeout
        )

-    @log_function
    async def claim_client_keys(
        self, destination: str, content: JsonDict, timeout: int
    ) -> JsonDict:
@@ -1338,7 +1334,7 @@ class FederationClient(FederationBase):
        destinations: Iterable[str],
        room_id: str,
        suggested_only: bool,
-    ) -> Tuple[JsonDict, Sequence[JsonDict], Sequence[str]]:
+    ) -> Tuple[JsonDict, Sequence[JsonDict], Sequence[JsonDict], Sequence[str]]:
        """
        Call other servers to get a hierarchy of the given room.

@@ -1353,7 +1349,8 @@ class FederationClient(FederationBase):

        Returns:
            A tuple of:
-                The room as a JSON dictionary.
+                The room as a JSON dictionary, without a "children_state" key.
+                A list of `m.space.child` state events.
                A list of children rooms, as JSON dictionaries.
                A list of inaccessible children room IDs.

@@ -1368,7 +1365,7 @@ class FederationClient(FederationBase):

        async def send_request(
            destination: str,
-        ) -> Tuple[JsonDict, Sequence[JsonDict], Sequence[str]]:
+        ) -> Tuple[JsonDict, Sequence[JsonDict], Sequence[JsonDict], Sequence[str]]:
            try:
                res = await self.transport_layer.get_room_hierarchy(
                    destination=destination,
@@ -1397,7 +1394,7 @@ class FederationClient(FederationBase):
                raise InvalidResponseError("'room' must be a dict")

            # Validate children_state of the room.
-            children_state = room.get("children_state", [])
+            children_state = room.pop("children_state", [])
            if not isinstance(children_state, Sequence):
                raise InvalidResponseError("'room.children_state' must be a list")
            if any(not isinstance(e, dict) for e in children_state):
@@ -1426,7 +1423,7 @@ class FederationClient(FederationBase):
                    "Invalid room ID in 'inaccessible_children' list"
                )

-            return room, children, inaccessible_children
+            return room, children_state, children, inaccessible_children

        try:
            result = await self._try_destination_list(
@@ -1474,8 +1471,6 @@ class FederationClient(FederationBase):
                if event.room_id == room_id:
                    children_events.append(event.data)
                    children_room_ids.add(event.state_key)
-            # And add them under the requested room.
-            requested_room["children_state"] = children_events

            # Find the children rooms.
            children = []
@@ -1485,7 +1480,7 @@ class FederationClient(FederationBase):

            # It isn't clear from the response whether some of the rooms are
            # not accessible.
-            result = (requested_room, children, ())
+            result = (requested_room, children_events, children, ())

        # Cache the result to avoid fetching data over federation every time.
        self._get_room_hierarchy_cache[(room_id, suggested_only)] = result
@@ -58,7 +58,6 @@ from synapse.logging.context import (
    run_in_background,
 )
 from synapse.logging.opentracing import log_kv, start_active_span_from_edu, trace
-from synapse.logging.utils import log_function
 from synapse.metrics.background_process_metrics import wrap_as_background_process
 from synapse.replication.http.federation import (
    ReplicationFederationSendEduRestServlet,
@@ -859,7 +858,6 @@ class FederationServer(FederationBase):
            res = {"auth_chain": [a.get_pdu_json(time_now) for a in auth_pdus]}
        return 200, res

-    @log_function
    async def on_query_client_keys(
        self, origin: str, content: Dict[str, str]
    ) -> Tuple[int, Dict[str, Any]]:
@@ -940,7 +938,6 @@ class FederationServer(FederationBase):

        return {"events": [ev.get_pdu_json(time_now) for ev in missing_events]}

-    @log_function
    async def on_openid_userinfo(self, token: str) -> Optional[str]:
        ts_now_ms = self._clock.time_msec()
        return await self.store.get_user_id_for_open_id_token(token, ts_now_ms)
@@ -23,7 +23,6 @@ import logging
 from typing import Optional, Tuple

 from synapse.federation.units import Transaction
-from synapse.logging.utils import log_function
 from synapse.storage.databases.main import DataStore
 from synapse.types import JsonDict

@@ -36,7 +35,6 @@ class TransactionActions:
    def __init__(self, datastore: DataStore):
        self.store = datastore

-    @log_function
    async def have_responded(
        self, origin: str, transaction: Transaction
    ) -> Optional[Tuple[int, JsonDict]]:
@@ -53,7 +51,6 @@ class TransactionActions:

        return await self.store.get_received_txn_response(transaction_id, origin)

-    @log_function
    async def set_response(
        self, origin: str, transaction: Transaction, code: int, response: JsonDict
    ) -> None:
@@ -607,18 +607,18 @@ class PerDestinationQueue:
        self._pending_pdus = []


-@attr.s(slots=True)
+@attr.s(slots=True, auto_attribs=True)
 class _TransactionQueueManager:
    """A helper async context manager for pulling stuff off the queues and
    tracking what was last successfully sent, etc.
    """

-    queue = attr.ib(type=PerDestinationQueue)
+    queue: PerDestinationQueue

-    _device_stream_id = attr.ib(type=Optional[int], default=None)
-    _device_list_id = attr.ib(type=Optional[int], default=None)
-    _last_stream_ordering = attr.ib(type=Optional[int], default=None)
-    _pdus = attr.ib(type=List[EventBase], factory=list)
+    _device_stream_id: Optional[int] = None
+    _device_list_id: Optional[int] = None
+    _last_stream_ordering: Optional[int] = None
+    _pdus: List[EventBase] = attr.Factory(list)

    async def __aenter__(self) -> Tuple[List[EventBase], List[Edu]]:
        # First we calculate the EDUs we want to send, if any.
@@ -35,6 +35,7 @@ if TYPE_CHECKING:
    import synapse.server

 logger = logging.getLogger(__name__)
+issue_8631_logger = logging.getLogger("synapse.8631_debug")

 last_pdu_ts_metric = Gauge(
    "synapse_federation_last_sent_pdu_time",
@@ -124,6 +125,17 @@ class TransactionManager:
                len(pdus),
                len(edus),
            )
+            if issue_8631_logger.isEnabledFor(logging.DEBUG):
+                DEVICE_UPDATE_EDUS = {"m.device_list_update", "m.signing_key_update"}
+                device_list_updates = [
+                    edu.content for edu in edus if edu.edu_type in DEVICE_UPDATE_EDUS
+                ]
+                if device_list_updates:
+                    issue_8631_logger.debug(
+                        "about to send txn [%s] including device list updates: %s",
+                        transaction.transaction_id,
+                        device_list_updates,
+                    )

            # Actually send the transaction

@@ -44,7 +44,6 @@ from synapse.api.urls import (
 from synapse.events import EventBase, make_event_from_dict
 from synapse.federation.units import Transaction
 from synapse.http.matrixfederationclient import ByteParser
-from synapse.logging.utils import log_function
 from synapse.types import JsonDict

 logger = logging.getLogger(__name__)
@@ -62,7 +61,6 @@ class TransportLayerClient:
        self.server_name = hs.hostname
        self.client = hs.get_federation_http_client()

-    @log_function
    async def get_room_state_ids(
        self, destination: str, room_id: str, event_id: str
    ) -> JsonDict:
@@ -88,7 +86,6 @@ class TransportLayerClient:
            try_trailing_slash_on_400=True,
        )

-    @log_function
    async def get_event(
        self, destination: str, event_id: str, timeout: Optional[int] = None
    ) -> JsonDict:
@@ -111,7 +108,6 @@ class TransportLayerClient:
            destination, path=path, timeout=timeout, try_trailing_slash_on_400=True
        )

-    @log_function
    async def backfill(
        self, destination: str, room_id: str, event_tuples: Collection[str], limit: int
    ) -> Optional[JsonDict]:
@@ -149,7 +145,6 @@ class TransportLayerClient:
            destination, path=path, args=args, try_trailing_slash_on_400=True
        )

-    @log_function
    async def timestamp_to_event(
        self, destination: str, room_id: str, timestamp: int, direction: str
    ) -> Union[JsonDict, List]:
@@ -185,7 +180,6 @@ class TransportLayerClient:

        return remote_response

-    @log_function
    async def send_transaction(
        self,
        transaction: Transaction,
@@ -234,7 +228,6 @@ class TransportLayerClient:
            try_trailing_slash_on_400=True,
        )

-    @log_function
    async def make_query(
        self,
        destination: str,
@@ -254,7 +247,6 @@ class TransportLayerClient:
            ignore_backoff=ignore_backoff,
        )

-    @log_function
    async def make_membership_event(
        self,
        destination: str,
@@ -317,7 +309,6 @@ class TransportLayerClient:
            ignore_backoff=ignore_backoff,
        )

-    @log_function
    async def send_join_v1(
        self,
        room_version: RoomVersion,
@@ -336,7 +327,6 @@ class TransportLayerClient:
            max_response_size=MAX_RESPONSE_SIZE_SEND_JOIN,
        )

-    @log_function
    async def send_join_v2(
        self,
        room_version: RoomVersion,
@@ -355,7 +345,6 @@ class TransportLayerClient:
            max_response_size=MAX_RESPONSE_SIZE_SEND_JOIN,
        )

-    @log_function
    async def send_leave_v1(
        self, destination: str, room_id: str, event_id: str, content: JsonDict
    ) -> Tuple[int, JsonDict]:
@@ -372,7 +361,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def send_leave_v2(
        self, destination: str, room_id: str, event_id: str, content: JsonDict
    ) -> JsonDict:
@@ -389,7 +377,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def send_knock_v1(
        self,
        destination: str,
@@ -423,7 +410,6 @@ class TransportLayerClient:
            destination=destination, path=path, data=content
        )

-    @log_function
    async def send_invite_v1(
        self, destination: str, room_id: str, event_id: str, content: JsonDict
    ) -> Tuple[int, JsonDict]:
@@ -433,7 +419,6 @@ class TransportLayerClient:
            destination=destination, path=path, data=content, ignore_backoff=True
        )

-    @log_function
    async def send_invite_v2(
        self, destination: str, room_id: str, event_id: str, content: JsonDict
    ) -> JsonDict:
@@ -443,7 +428,6 @@ class TransportLayerClient:
            destination=destination, path=path, data=content, ignore_backoff=True
        )

-    @log_function
    async def get_public_rooms(
        self,
        remote_server: str,
@@ -516,7 +500,6 @@ class TransportLayerClient:

        return response

-    @log_function
    async def exchange_third_party_invite(
        self, destination: str, room_id: str, event_dict: JsonDict
    ) -> JsonDict:
@@ -526,7 +509,6 @@ class TransportLayerClient:
            destination=destination, path=path, data=event_dict
        )

-    @log_function
    async def get_event_auth(
        self, destination: str, room_id: str, event_id: str
    ) -> JsonDict:
@@ -534,7 +516,6 @@ class TransportLayerClient:

        return await self.client.get_json(destination=destination, path=path)

-    @log_function
    async def query_client_keys(
        self, destination: str, query_content: JsonDict, timeout: int
    ) -> JsonDict:
@@ -576,7 +557,6 @@ class TransportLayerClient:
            destination=destination, path=path, data=query_content, timeout=timeout
        )

-    @log_function
    async def query_user_devices(
        self, destination: str, user_id: str, timeout: int
    ) -> JsonDict:
@@ -616,7 +596,6 @@ class TransportLayerClient:
            destination=destination, path=path, timeout=timeout
        )

-    @log_function
    async def claim_client_keys(
        self, destination: str, query_content: JsonDict, timeout: int
    ) -> JsonDict:
@@ -655,7 +634,6 @@ class TransportLayerClient:
            destination=destination, path=path, data=query_content, timeout=timeout
        )

-    @log_function
    async def get_missing_events(
        self,
        destination: str,
@@ -680,7 +658,6 @@ class TransportLayerClient:
            timeout=timeout,
        )

-    @log_function
    async def get_group_profile(
        self, destination: str, group_id: str, requester_user_id: str
    ) -> JsonDict:
@@ -694,7 +671,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def update_group_profile(
        self, destination: str, group_id: str, requester_user_id: str, content: JsonDict
    ) -> JsonDict:
@@ -716,7 +692,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def get_group_summary(
        self, destination: str, group_id: str, requester_user_id: str
    ) -> JsonDict:
@@ -730,7 +705,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def get_rooms_in_group(
        self, destination: str, group_id: str, requester_user_id: str
    ) -> JsonDict:
@@ -798,7 +772,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def get_users_in_group(
        self, destination: str, group_id: str, requester_user_id: str
    ) -> JsonDict:
@@ -812,7 +785,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def get_invited_users_in_group(
        self, destination: str, group_id: str, requester_user_id: str
    ) -> JsonDict:
@@ -826,7 +798,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def accept_group_invite(
        self, destination: str, group_id: str, user_id: str, content: JsonDict
    ) -> JsonDict:
@@ -837,7 +808,6 @@ class TransportLayerClient:
            destination=destination, path=path, data=content, ignore_backoff=True
        )

-    @log_function
    def join_group(
        self, destination: str, group_id: str, user_id: str, content: JsonDict
    ) -> Awaitable[JsonDict]:
@@ -848,7 +818,6 @@ class TransportLayerClient:
            destination=destination, path=path, data=content, ignore_backoff=True
        )

-    @log_function
    async def invite_to_group(
        self,
        destination: str,
@@ -868,7 +837,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def invite_to_group_notification(
        self, destination: str, group_id: str, user_id: str, content: JsonDict
    ) -> JsonDict:
@@ -882,7 +850,6 @@ class TransportLayerClient:
            destination=destination, path=path, data=content, ignore_backoff=True
        )

-    @log_function
    async def remove_user_from_group(
        self,
        destination: str,
@@ -902,7 +869,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def remove_user_from_group_notification(
        self, destination: str, group_id: str, user_id: str, content: JsonDict
    ) -> JsonDict:
@@ -916,7 +882,6 @@ class TransportLayerClient:
            destination=destination, path=path, data=content, ignore_backoff=True
        )

-    @log_function
    async def renew_group_attestation(
        self, destination: str, group_id: str, user_id: str, content: JsonDict
    ) -> JsonDict:
@@ -930,7 +895,6 @@ class TransportLayerClient:
            destination=destination, path=path, data=content, ignore_backoff=True
        )

-    @log_function
    async def update_group_summary_room(
        self,
        destination: str,
@@ -959,7 +923,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def delete_group_summary_room(
        self,
        destination: str,
@@ -986,7 +949,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def get_group_categories(
        self, destination: str, group_id: str, requester_user_id: str
    ) -> JsonDict:
@@ -1000,7 +962,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def get_group_category(
        self, destination: str, group_id: str, requester_user_id: str, category_id: str
    ) -> JsonDict:
@@ -1014,7 +975,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def update_group_category(
        self,
        destination: str,
@@ -1034,7 +994,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def delete_group_category(
        self, destination: str, group_id: str, requester_user_id: str, category_id: str
    ) -> JsonDict:
@@ -1048,7 +1007,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def get_group_roles(
        self, destination: str, group_id: str, requester_user_id: str
    ) -> JsonDict:
@@ -1062,7 +1020,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def get_group_role(
        self, destination: str, group_id: str, requester_user_id: str, role_id: str
    ) -> JsonDict:
@@ -1076,7 +1033,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def update_group_role(
        self,
        destination: str,
@@ -1096,7 +1052,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def delete_group_role(
        self, destination: str, group_id: str, requester_user_id: str, role_id: str
    ) -> JsonDict:
@@ -1110,7 +1065,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def update_group_summary_user(
        self,
        destination: str,
@@ -1136,7 +1090,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def set_group_join_policy(
        self, destination: str, group_id: str, requester_user_id: str, content: JsonDict
    ) -> JsonDict:
@@ -1151,7 +1104,6 @@ class TransportLayerClient:
            ignore_backoff=True,
        )

-    @log_function
    async def delete_group_summary_user(
        self,
        destination: str,
@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import logging
-from typing import Dict, Iterable, List, Optional, Tuple, Type
+from typing import TYPE_CHECKING, Dict, Iterable, List, Optional, Tuple, Type

 from typing_extensions import Literal

@@ -36,17 +36,19 @@ from synapse.http.servlet import (
    parse_integer_from_args,
    parse_string_from_args,
 )
-from synapse.server import HomeServer
 from synapse.types import JsonDict, ThirdPartyInstanceID
 from synapse.util.ratelimitutils import FederationRateLimiter

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
 logger = logging.getLogger(__name__)


 class TransportLayerServer(JsonResource):
    """Handles incoming federation HTTP requests"""

-    def __init__(self, hs: HomeServer, servlet_groups: Optional[List[str]] = None):
+    def __init__(self, hs: "HomeServer", servlet_groups: Optional[List[str]] = None):
        """Initialize the TransportLayerServer

        Will by default register all servlets. For custom behaviour, pass in
@@ -113,7 +115,7 @@ class PublicRoomList(BaseFederationServlet):

    def __init__(
        self,
-        hs: HomeServer,
+        hs: "HomeServer",
        authenticator: Authenticator,
        ratelimiter: FederationRateLimiter,
        server_name: str,
@@ -203,7 +205,7 @@ class FederationGroupsRenewAttestaionServlet(BaseFederationServlet):

    def __init__(
        self,
-        hs: HomeServer,
+        hs: "HomeServer",
        authenticator: Authenticator,
        ratelimiter: FederationRateLimiter,
        server_name: str,
@@ -251,7 +253,7 @@ class OpenIdUserInfo(BaseFederationServlet):

    def __init__(
        self,
-        hs: HomeServer,
+        hs: "HomeServer",
        authenticator: Authenticator,
        ratelimiter: FederationRateLimiter,
        server_name: str,
@@ -297,7 +299,7 @@ DEFAULT_SERVLET_GROUPS: Dict[str, Iterable[Type[BaseFederationServlet]]] = {


 def register_servlets(
-    hs: HomeServer,
+    hs: "HomeServer",
    resource: HttpServer,
    authenticator: Authenticator,
    ratelimiter: FederationRateLimiter,
@@ -15,7 +15,7 @@
 import functools
 import logging
 import re
-from typing import Any, Awaitable, Callable, Optional, Tuple, cast
+from typing import TYPE_CHECKING, Any, Awaitable, Callable, Optional, Tuple, cast

 from synapse.api.errors import Codes, FederationDeniedError, SynapseError
 from synapse.api.urls import FEDERATION_V1_PREFIX
@@ -29,11 +29,13 @@ from synapse.logging.opentracing import (
    start_active_span_follows_from,
    whitelisted_homeserver,
 )
-from synapse.server import HomeServer
 from synapse.types import JsonDict
 from synapse.util.ratelimitutils import FederationRateLimiter
 from synapse.util.stringutils import parse_and_validate_server_name

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
 logger = logging.getLogger(__name__)


@@ -46,7 +48,7 @@ class NoAuthenticationError(AuthenticationError):


 class Authenticator:
-    def __init__(self, hs: HomeServer):
+    def __init__(self, hs: "HomeServer"):
        self._clock = hs.get_clock()
        self.keyring = hs.get_keyring()
        self.server_name = hs.hostname
@@ -114,11 +116,11 @@ class Authenticator:
        # alive
        retry_timings = await self.store.get_destination_retry_timings(origin)
        if retry_timings and retry_timings.retry_last_ts:
-            run_in_background(self._reset_retry_timings, origin)
+            run_in_background(self.reset_retry_timings, origin)

        return origin

-    async def _reset_retry_timings(self, origin: str) -> None:
+    async def reset_retry_timings(self, origin: str) -> None:
        try:
            logger.info("Marking origin %r as up", origin)
            await self.store.set_destination_retry_timings(origin, None, 0, 0)
@@ -227,7 +229,7 @@ class BaseFederationServlet:

    def __init__(
        self,
-        hs: HomeServer,
+        hs: "HomeServer",
        authenticator: Authenticator,
        ratelimiter: FederationRateLimiter,
        server_name: str,
@@ -12,7 +12,17 @@
 #  See the License for the specific language governing permissions and
 #  limitations under the License.
 import logging
-from typing import Dict, List, Mapping, Optional, Sequence, Tuple, Type, Union
+from typing import (
+    TYPE_CHECKING,
+    Dict,
+    List,
+    Mapping,
+    Optional,
+    Sequence,
+    Tuple,
+    Type,
+    Union,
+)

 from typing_extensions import Literal

@@ -30,12 +40,15 @@ from synapse.http.servlet import (
    parse_string_from_args,
    parse_strings_from_args,
 )
-from synapse.server import HomeServer
 from synapse.types import JsonDict
 from synapse.util.ratelimitutils import FederationRateLimiter
 from synapse.util.versionstring import get_version_string

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+
 logger = logging.getLogger(__name__)
+issue_8631_logger = logging.getLogger("synapse.8631_debug")


 class BaseFederationServerServlet(BaseFederationServlet):
@@ -46,7 +59,7 @@ class BaseFederationServerServlet(BaseFederationServlet):

    def __init__(
        self,
-        hs: HomeServer,
+        hs: "HomeServer",
        authenticator: Authenticator,
        ratelimiter: FederationRateLimiter,
        server_name: str,
@@ -95,6 +108,20 @@ class FederationSendServlet(BaseFederationServerServlet):
                len(transaction_data.get("edus", [])),
            )

+            if issue_8631_logger.isEnabledFor(logging.DEBUG):
+                DEVICE_UPDATE_EDUS = {"m.device_list_update", "m.signing_key_update"}
+                device_list_updates = [
+                    edu.content
+                    for edu in transaction_data.get("edus", [])
+                    if edu.edu_type in DEVICE_UPDATE_EDUS
+                ]
+                if device_list_updates:
+                    issue_8631_logger.debug(
+                        "received transaction [%s] including device list updates: %s",
+                        transaction_id,
+                        device_list_updates,
+                    )
+
        except Exception as e:
            logger.exception(e)
            return 400, {"error": "Invalid transaction"}
@@ -581,7 +608,7 @@ class FederationSpaceSummaryServlet(BaseFederationServlet):

    def __init__(
        self,
-        hs: HomeServer,
+        hs: "HomeServer",
        authenticator: Authenticator,
        ratelimiter: FederationRateLimiter,
        server_name: str,
@@ -655,7 +682,7 @@ class FederationRoomHierarchyServlet(BaseFederationServlet):

    def __init__(
        self,
-        hs: HomeServer,
+        hs: "HomeServer",
        authenticator: Authenticator,
        ratelimiter: FederationRateLimiter,
        server_name: str,
@@ -691,7 +718,7 @@ class RoomComplexityServlet(BaseFederationServlet):

    def __init__(
        self,
-        hs: HomeServer,
+        hs: "HomeServer",
        authenticator: Authenticator,
        ratelimiter: FederationRateLimiter,
        server_name: str,
@@ -11,7 +11,7 @@
 #  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 #  See the License for the specific language governing permissions and
 #  limitations under the License.
-from typing import Dict, List, Tuple, Type
+from typing import TYPE_CHECKING, Dict, List, Tuple, Type

 from synapse.api.errors import SynapseError
 from synapse.federation.transport.server._base import (
@@ -19,10 +19,12 @@ from synapse.federation.transport.server._base import (
    BaseFederationServlet,
 )
 from synapse.handlers.groups_local import GroupsLocalHandler
-from synapse.server import HomeServer
 from synapse.types import JsonDict, get_domain_from_id
 from synapse.util.ratelimitutils import FederationRateLimiter

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+

 class BaseGroupsLocalServlet(BaseFederationServlet):
    """Abstract base class for federation servlet classes which provides a groups local handler.
@@ -32,7 +34,7 @@ class BaseGroupsLocalServlet(BaseFederationServlet):

    def __init__(
        self,
-        hs: HomeServer,
+        hs: "HomeServer",
        authenticator: Authenticator,
        ratelimiter: FederationRateLimiter,
        server_name: str,
@@ -11,7 +11,7 @@
 #  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 #  See the License for the specific language governing permissions and
 #  limitations under the License.
-from typing import Dict, List, Tuple, Type
+from typing import TYPE_CHECKING, Dict, List, Tuple, Type

 from typing_extensions import Literal

@@ -22,10 +22,12 @@ from synapse.federation.transport.server._base import (
    BaseFederationServlet,
 )
 from synapse.http.servlet import parse_string_from_args
-from synapse.server import HomeServer
 from synapse.types import JsonDict, get_domain_from_id
 from synapse.util.ratelimitutils import FederationRateLimiter

+if TYPE_CHECKING:
+    from synapse.server import HomeServer
+

 class BaseGroupsServerServlet(BaseFederationServlet):
    """Abstract base class for federation servlet classes which provides a groups server handler.
@@ -35,7 +37,7 @@ class BaseGroupsServerServlet(BaseFederationServlet):

    def __init__(
        self,
-        hs: HomeServer,
+        hs: "HomeServer",
        authenticator: Authenticator,
        ratelimiter: FederationRateLimiter,
        server_name: str,
@@ -77,7 +77,7 @@ class AccountDataHandler:
    async def add_account_data_for_user(
        self, user_id: str, account_data_type: str, content: JsonDict
    ) -> int:
-        """Add some account_data to a room for a user.
+        """Add some global account_data for a user.

        Args:
            user_id: The user to add a tag for.
@@ -55,21 +55,47 @@ class AdminHandler:

    async def get_user(self, user: UserID) -> Optional[JsonDict]:
        """Function to get user details"""
-        ret = await self.store.get_user_by_id(user.to_string())
-        if ret:
-            profile = await self.store.get_profileinfo(user.localpart)
-            threepids = await self.store.user_get_threepids(user.to_string())
-            external_ids = [
-                ({"auth_provider": auth_provider, "external_id": external_id})
-                for auth_provider, external_id in await self.store.get_external_ids_by_user(
-                    user.to_string()
-                )
-            ]
-            ret["displayname"] = profile.display_name
-            ret["avatar_url"] = profile.avatar_url
-            ret["threepids"] = threepids
-            ret["external_ids"] = external_ids
-        return ret
+        user_info_dict = await self.store.get_user_by_id(user.to_string())
+        if user_info_dict is None:
+            return None
+
+        # Restrict returned information to a known set of fields. This prevents additional
+        # fields added to get_user_by_id from modifying Synapse's external API surface.
+        user_info_to_return = {
+            "name",
+            "admin",
+            "deactivated",
+            "shadow_banned",
+            "creation_ts",
+            "appservice_id",
+            "consent_server_notice_sent",
+            "consent_version",
+            "user_type",
+            "is_guest",
+        }
+
+        # Restrict returned keys to a known set.
+        user_info_dict = {
+            key: value
+            for key, value in user_info_dict.items()
+            if key in user_info_to_return
+        }
+
+        # Add additional user metadata
+        profile = await self.store.get_profileinfo(user.localpart)
+        threepids = await self.store.user_get_threepids(user.to_string())
+        external_ids = [
+            ({"auth_provider": auth_provider, "external_id": external_id})
+            for auth_provider, external_id in await self.store.get_external_ids_by_user(
+                user.to_string()
+            )
+        ]
+        user_info_dict["displayname"] = profile.display_name
+        user_info_dict["avatar_url"] = profile.avatar_url
+        user_info_dict["threepids"] = threepids
+        user_info_dict["external_ids"] = external_ids
+
+        return user_info_dict

    async def export_user_data(self, user_id: str, writer: "ExfiltrationWriter") -> Any:
        """Write all data we have on the user to the given writer.
@@ -199,8 +199,9 @@ class ApplicationServicesHandler:
        Args:
            stream_key: The stream the event came from.

-                `stream_key` can be "typing_key", "receipt_key" or "presence_key". Any other
-                value for `stream_key` will cause this function to return early.
+                `stream_key` can be "typing_key", "receipt_key", "edu_key"
+                or "presence_key". Any other value for `stream_key` will
+                cause this function to return early.

                Ephemeral events will only be pushed to appservices that have opted into
                receiving them by setting `push_ephemeral` to true in their registration
@@ -217,7 +218,7 @@ class ApplicationServicesHandler:
            return

        # Ignore any unsupported streams
-        if stream_key not in ("typing_key", "receipt_key", "presence_key"):
+        if stream_key not in ("typing_key", "receipt_key", "presence_key", "edu_key"):
            return

        # Assert that new_token is an integer (and not a RoomStreamToken).
@@ -307,6 +308,18 @@ class ApplicationServicesHandler:
                            service, "presence", new_token
                        )

+                    elif stream_key == "edu_key":
+                        events = await self._handle_edus(service, new_token)
+                        if events:
+                            self.scheduler.submit_ephemeral_events_for_as(
+                                service, events
+                            )
+
+                        # Persist the latest handled stream token for this appservice
+                        await self.store.set_type_stream_id_for_appservice(
+                            service, "edu", new_token
+                        )
+
    async def _handle_typing(
        self, service: ApplicationService, new_token: int
    ) -> List[JsonDict]:
@@ -378,6 +391,42 @@ class ApplicationServicesHandler:
        )
        return receipts

+    async def _handle_edus(
+        self, service: ApplicationService, new_token: Optional[int]
+    ) -> List[JsonDict]:
+        """
+        Return the latest custom EDUs that the given application service should receive.
+
+        First fetch all custom EDUs between the last EDU stream token that this
+        application service should have previously received (non-inclusive) and the
+        latest EDU stream token (inclusive). Then from that set, return only
+        those custom EDUs that the given application service may be interested in.
+
+        Args:
+            service: The application service to check for which events it should receive.
+            new_token: A receipts event stream token. Purely used to double-check that the
+                from_token we pull from the database isn't greater than or equal to this
+                token. Prevents accidentally duplicating work.
+
+        Returns:
+            A list of JSON dictionaries containing data derived from the custom EDUs that
+            should be sent to the given application service.
+        """
+        from_key = await self.store.get_type_stream_id_for_appservice(
+            service, "edu"
+        )
+        if new_token is not None and new_token <= from_key:
+            logger.debug(
+                "Rejecting token lower than or equal to stored: %s" % (new_token,)
+            )
+            return []
+
+        edus_source = self.event_sources.sources.edus
+        edus, _ = await edus_source.get_new_events_as(
+            service=service, from_key=from_key
+        )
+        return edus
+
    async def _handle_presence(
        self,
        service: ApplicationService,
@@ -168,25 +168,25 @@ def login_id_phone_to_thirdparty(identifier: JsonDict) -> Dict[str, str]:
    }


-@attr.s(slots=True)
+@attr.s(slots=True, auto_attribs=True)
 class SsoLoginExtraAttributes:
    """Data we track about SAML2 sessions"""

    # time the session was created, in milliseconds
-    creation_time = attr.ib(type=int)
-    extra_attributes = attr.ib(type=JsonDict)
+    creation_time: int
+    extra_attributes: JsonDict


-@attr.s(slots=True, frozen=True)
+@attr.s(slots=True, frozen=True, auto_attribs=True)
 class LoginTokenAttributes:
    """Data we store in a short-term login token"""

-    user_id = attr.ib(type=str)
+    user_id: str

-    auth_provider_id = attr.ib(type=str)
+    auth_provider_id: str
    """The SSO Identity Provider that the user authenticated with, to get this token."""

-    auth_provider_session_id = attr.ib(type=Optional[str])
+    auth_provider_session_id: Optional[str]
    """The session ID advertised by the SSO Identity Provider."""


@@ -2281,7 +2281,7 @@ class PasswordAuthProvider:
        # call all of the on_logged_out callbacks
        for callback in self.on_logged_out_callbacks:
            try:
-                callback(user_id, device_id, access_token)
+                await callback(user_id, device_id, access_token)
            except Exception as e:
                logger.warning("Failed to run module API callback %s: %s", callback, e)
                continue
@@ -157,6 +157,9 @@ class DeactivateAccountHandler:
        # Mark the user as deactivated.
        await self.store.set_user_deactivated_status(user_id, True)

+        # Remove account data (including ignored users and push rules).
+        await self.store.purge_account_data_for_user(user_id)
+
        return identity_server_supports_unbinding

    async def _reject_pending_invites_for_user(self, user_id: str) -> None:
@@ -948,8 +948,16 @@ class DeviceListUpdater:
            devices = []
            ignore_devices = True
        else:
+            prev_stream_id = await self.store.get_device_list_last_stream_id_for_remote(
+                user_id
+            )
            cached_devices = await self.store.get_cached_devices_for_user(user_id)
-            if cached_devices == {d["device_id"]: d for d in devices}:
+
+            # To ensure that a user with no devices is cached, we skip the resync only
+            # if we have a stream_id from previously writing a cache entry.
+            if prev_stream_id is not None and cached_devices == {
+                d["device_id"]: d for d in devices
+            }:
                logging.info(
                    "Skipping device list resync for %s, as our cache matches already",
                    user_id,
@@ -1321,14 +1321,14 @@ def _one_time_keys_match(old_key_json: str, new_key: JsonDict) -> bool:
    return old_key == new_key_copy


-@attr.s(slots=True)
+@attr.s(slots=True, auto_attribs=True)
 class SignatureListItem:
    """An item in the signature list as used by upload_signatures_for_device_keys."""

-    signing_key_id = attr.ib(type=str)
-    target_user_id = attr.ib(type=str)
-    target_device_id = attr.ib(type=str)
-    signature = attr.ib(type=JsonDict)
+    signing_key_id: str
+    target_user_id: str
+    target_device_id: str
+    signature: JsonDict


 class SigningKeyEduUpdater:
@@ -20,7 +20,6 @@ from synapse.api.constants import EduTypes, EventTypes, Membership
 from synapse.api.errors import AuthError, SynapseError
 from synapse.events import EventBase
 from synapse.handlers.presence import format_user_presence_state
-from synapse.logging.utils import log_function
 from synapse.streams.config import PaginationConfig
 from synapse.types import JsonDict, UserID
 from synapse.visibility import filter_events_for_client
@@ -43,7 +42,6 @@ class EventStreamHandler:
        self._server_notices_sender = hs.get_server_notices_sender()
        self._event_serializer = hs.get_event_client_serializer()

-    @log_function
    async def get_stream(
        self,
        auth_user_id: str,
@@ -119,7 +117,7 @@ class EventStreamHandler:

            events.extend(to_add)

-            chunks = await self._event_serializer.serialize_events(
+            chunks = self._event_serializer.serialize_events(
                events,
                time_now,
                as_client_event=as_client_event,
@@ -51,7 +51,6 @@ from synapse.logging.context import (
    preserve_fn,
    run_in_background,
 )
-from synapse.logging.utils import log_function
 from synapse.replication.http.federation import (
    ReplicationCleanRoomRestServlet,
    ReplicationStoreRoomOnOutlierMembershipRestServlet,
@@ -556,7 +555,6 @@ class FederationHandler:

            run_in_background(self._handle_queued_pdus, room_queue)

-    @log_function
    async def do_knock(
        self,
        target_hosts: List[str],
@@ -928,7 +926,6 @@ class FederationHandler:

        return event

-    @log_function
    async def on_make_knock_request(
        self, origin: str, room_id: str, user_id: str
    ) -> EventBase:
@@ -1039,7 +1036,6 @@ class FederationHandler:
        else:
            return []

-    @log_function
    async def on_backfill_request(
        self, origin: str, room_id: str, pdu_list: List[str], limit: int
    ) -> List[EventBase]:
@@ -1056,7 +1052,6 @@ class FederationHandler:

        return events

-    @log_function
    async def get_persisted_pdu(
        self, origin: str, event_id: str
    ) -> Optional[EventBase]:
@@ -1118,7 +1113,6 @@ class FederationHandler:

        return missing_events

-    @log_function
    async def exchange_third_party_invite(
        self, sender_user_id: str, target_user_id: str, room_id: str, signed: JsonDict
    ) -> None:
@@ -56,7 +56,6 @@ from synapse.events import EventBase
 from synapse.events.snapshot import EventContext
 from synapse.federation.federation_client import InvalidResponseError
 from synapse.logging.context import nested_logging_context, run_in_background
-from synapse.logging.utils import log_function
 from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.replication.http.devices import ReplicationUserDevicesResyncRestServlet
 from synapse.replication.http.federation import (
@@ -275,7 +274,6 @@ class FederationEventHandler:

        await self._process_received_pdu(origin, pdu, state=None)

-    @log_function
    async def on_send_membership_event(
        self, origin: str, event: EventBase
    ) -> Tuple[EventBase, EventContext]:
@@ -472,7 +470,6 @@ class FederationEventHandler:

            return await self.persist_events_and_notify(room_id, [(event, context)])

-    @log_function
    async def backfill(
        self, dest: str, room_id: str, limit: int, extremities: Collection[str]
    ) -> None:
@@ -170,7 +170,7 @@ class InitialSyncHandler:
                d["inviter"] = event.sender

                invite_event = await self.store.get_event(event.event_id)
-                d["invite"] = await self._event_serializer.serialize_event(
+                d["invite"] = self._event_serializer.serialize_event(
                    invite_event,
                    time_now,
                    as_client_event=as_client_event,
@@ -222,7 +222,7 @@ class InitialSyncHandler:

                d["messages"] = {
                    "chunk": (
-                        await self._event_serializer.serialize_events(
+                        self._event_serializer.serialize_events(
                            messages,
                            time_now=time_now,
                            as_client_event=as_client_event,
@@ -232,7 +232,7 @@ class InitialSyncHandler:
                    "end": await end_token.to_string(self.store),
                }

-                d["state"] = await self._event_serializer.serialize_events(
+                d["state"] = self._event_serializer.serialize_events(
                    current_state.values(),
                    time_now=time_now,
                    as_client_event=as_client_event,
@@ -376,16 +376,14 @@ class InitialSyncHandler:
            "messages": {
                "chunk": (
                    # Don't bundle aggregations as this is a deprecated API.
-                    await self._event_serializer.serialize_events(messages, time_now)
+                    self._event_serializer.serialize_events(messages, time_now)
                ),
                "start": await start_token.to_string(self.store),
                "end": await end_token.to_string(self.store),
            },
            "state": (
                # Don't bundle aggregations as this is a deprecated API.
-                await self._event_serializer.serialize_events(
-                    room_state.values(), time_now
-                )
+                self._event_serializer.serialize_events(room_state.values(), time_now)
            ),
            "presence": [],
            "receipts": [],
@@ -404,7 +402,7 @@ class InitialSyncHandler:
        # TODO: These concurrently
        time_now = self.clock.time_msec()
        # Don't bundle aggregations as this is a deprecated API.
-        state = await self._event_serializer.serialize_events(
+        state = self._event_serializer.serialize_events(
            current_state.values(), time_now
        )

@@ -480,7 +478,7 @@ class InitialSyncHandler:
            "messages": {
                "chunk": (
                    # Don't bundle aggregations as this is a deprecated API.
-                    await self._event_serializer.serialize_events(messages, time_now)
+                    self._event_serializer.serialize_events(messages, time_now)
                ),
                "start": await start_token.to_string(self.store),
                "end": await end_token.to_string(self.store),
@@ -1,7 +1,7 @@
 # Copyright 2014-2016 OpenMarket Ltd
 # Copyright 2017-2018 New Vector Ltd
-# Copyright 2019-2020 The Matrix.org Foundation C.I.C.
 # Copyrignt 2020 Sorunome
+# Copyright 2019-2022 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -80,6 +80,10 @@ class MessageHandler:
        self.state_store = self.storage.state
        self._event_serializer = hs.get_event_client_serializer()
        self._ephemeral_events_enabled = hs.config.server.enable_ephemeral_messages
+        self.federation = None
+        if hs.should_send_federation():
+            self.federation = hs.get_federation_sender()
+        self.notifier = hs.get_notifier()

        # The scheduled call to self._expire_event. None if no call is currently
        # scheduled.
@@ -246,7 +250,7 @@ class MessageHandler:
                room_state = room_state_events[membership_event_id]

        now = self.clock.time_msec()
-        events = await self._event_serializer.serialize_events(room_state.values(), now)
+        events = self._event_serializer.serialize_events(room_state.values(), now)
        return events

    async def get_joined_members(self, requester: Requester, room_id: str) -> dict:
@@ -760,6 +764,86 @@ class EventCreationHandler:
                return prev_event
        return None

+    async def _push_remote_edu(
+        self,
+        event_dict: dict,
+    ) -> None:
+        if not self.federation:
+            return
+
+        try:
+            users = await self.store.get_users_in_room(event_dict, room_id)
+
+            for domain in {get_domain_from_id(u) for u in users}:
+                if domain != self.server_name:
+                    logger.debug("sending custom EDU to %s", domain)
+                    self.federation.build_and_send_edu(
+                        destination=domain,
+                        edu_type=event_dict.type,
+                        content=event_dict,
+                    )
+        except Exception:
+            logger.exception("Error pushing custom EDU to remotes")
+
+    async def send_ephemeral_event(
+        self,
+        requester: Requester,
+        event_dict: dict,
+        ratelimit: bool = True,
+        txn_id: Optional[str] = None,
+        ignore_shadow_ban: bool = False,
+    ) -> None:
+        """
+        Creates an event, then sends it.
+
+        See self.create_event and self.handle_new_client_event.
+
+        Args:
+            requester: The requester sending the event.
+            event_dict: An entire ephemeral event dict.
+            ratelimit: Whether to rate limit this send.
+            txn_id: The transaction ID.
+            ignore_shadow_ban: True if shadow-banned users should be allowed to
+                send this event.
+
+        Raises:
+            ShadowBanError if the requester has been shadow-banned.
+        """
+
+        if not self._ephemeral_events_enabled:
+            return
+
+        if not ignore_shadow_ban and requester.shadow_banned:
+            # We randomly sleep a bit just to annoy the requester.
+            await self.clock.sleep(random.randint(1, 10))
+            raise ShadowBanError()
+
+        assert self.hs.is_mine_id(event.sender), "User must be our own: %s" % (
+            event.sender,
+        )
+
+        # TODO: spam check the EDU
+
+        # TODO: store it in the DB so it's persisted nicely
+
+        # send it remotely
+        run_as_background_process(
+            "message._push_remote_edu", self._push_remote_edu, event_dict
+        )
+
+        # send it locally
+        async def _notify() -> None:
+            try:
+                await self.notifier.on_new_event(
+                    "ephemeral_key", self._latest_room_serial, rooms=[room_id]
+                )
+            except Exception:
+                logger.exception(
+                    "Error notifying about new custom EDU"
+                )
+
+        run_in_background(_notify);
+
    async def create_and_send_nonmember_event(
        self,
        requester: Requester,
@@ -1789,3 +1873,70 @@ class EventCreationHandler:
        # delta_ids might need an update.
        context = await self.state.compute_event_context(event)
        return event, context
+
+class EduEventSource(EventSource[int, JsonDict]):
+    def __init__(self, hs: "HomeServer"):
+        self.store = hs.get_datastore()
+        self.config = hs.config
+
+    async def get_new_events(
+        self,
+        user: UserID,
+        from_key: int,
+        limit: Optional[int],
+        room_ids: Iterable[str],
+        is_guest: bool,
+        explicit_room_id: Optional[str] = None,
+    ) -> Tuple[List[JsonDict], int]:
+        from_key = int(from_key)
+        to_key = self.get_current_key()
+
+        if from_key == to_key:
+            return [], to_key
+
+        events = await self.store.get_edus_for_rooms(
+            room_ids, from_key=from_key, to_key=to_key
+        )
+
+        return events, to_key
+
+    async def get_new_events_as(
+        self, from_key: int, service: ApplicationService
+    ) -> Tuple[List[JsonDict], int]:
+        """Returns a set of new custom EDUs that an appservice
+        may be interested in.
+
+        Args:
+            from_key: the stream position at which events should be fetched from
+            service: The appservice which may be interested
+
+        Returns:
+            A two-tuple containing the following:
+                * A list of json dictionaries derived from read receipts that the
+                  appservice may be interested in.
+                * The current read receipt stream token.
+        """
+        from_key = int(from_key)
+        to_key = self.get_current_key()
+
+        if from_key == to_key:
+            return [], to_key
+
+        # Fetch all custom EDUs for all rooms, up to a limit of 100. This is ordered
+        # by most recent.
+        rooms_to_events = await self.store.get_edus_for_all_rooms(
+            from_key=from_key, to_key=to_key
+        )
+
+        # Then filter down to rooms that the AS can read
+        events = []
+        for room_id, event in rooms_to_events.items():
+            if not await service.matches_user_in_member_list(room_id, self.store):
+                continue
+
+            events.append(event)
+
+        return events, to_key
+
+    def get_current_key(self, direction: str = "f") -> int:
+        return self.store.get_max_edus_stream_id()
@@ -537,14 +537,16 @@ class PaginationHandler:
                state_dict = await self.store.get_events(list(state_ids.values()))
                state = state_dict.values()

+        aggregations = await self.store.get_bundled_aggregations(events, user_id)
+
        time_now = self.clock.time_msec()

        chunk = {
            "chunk": (
-                await self._event_serializer.serialize_events(
+                self._event_serializer.serialize_events(
                    events,
                    time_now,
-                    bundle_aggregations=True,
+                    bundle_aggregations=aggregations,
                    as_client_event=as_client_event,
                )
            ),
@@ -553,7 +555,7 @@ class PaginationHandler:
        }

        if state:
-            chunk["state"] = await self._event_serializer.serialize_events(
+            chunk["state"] = self._event_serializer.serialize_events(
                state, time_now, as_client_event=as_client_event
            )

@@ -55,7 +55,6 @@ from synapse.api.presence import UserPresenceState
 from synapse.appservice import ApplicationService
 from synapse.events.presence_router import PresenceRouter
 from synapse.logging.context import run_in_background
-from synapse.logging.utils import log_function
 from synapse.metrics import LaterGauge
 from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.replication.http.presence import (
@@ -1542,7 +1541,6 @@ class PresenceEventSource(EventSource[int, UserPresenceState]):
        self.clock = hs.get_clock()
        self.store = hs.get_datastore()

-    @log_function
    async def get_new_events(
        self,
        user: UserID,
@@ -393,7 +393,9 @@ class RoomCreationHandler:
        user_id = requester.user.to_string()

        if not await self.spam_checker.user_may_create_room(user_id):
-            raise SynapseError(403, "You are not permitted to create rooms")
+            raise SynapseError(
+                403, "You are not permitted to create rooms", Codes.FORBIDDEN
+            )

        creation_content: JsonDict = {
            "room_version": new_room_version.identifier,
@@ -685,7 +687,9 @@ class RoomCreationHandler:
                invite_3pid_list,
            )
        ):
-            raise SynapseError(403, "You are not permitted to create rooms")
+            raise SynapseError(
+                403, "You are not permitted to create rooms", Codes.FORBIDDEN
+            )

        if ratelimit:
            await self.request_ratelimiter.ratelimit(requester)
@@ -1177,6 +1181,22 @@ class RoomContextHandler:
        # `filtered` rather than the event we retrieved from the datastore.
        results["event"] = filtered[0]

+        # Fetch the aggregations.
+        aggregations = await self.store.get_bundled_aggregations(
+            [results["event"]], user.to_string()
+        )
+        aggregations.update(
+            await self.store.get_bundled_aggregations(
+                results["events_before"], user.to_string()
+            )
+        )
+        aggregations.update(
+            await self.store.get_bundled_aggregations(
+                results["events_after"], user.to_string()
+            )
+        )
+        results["aggregations"] = aggregations
+
        if results["events_after"]:
            last_event_id = results["events_after"][-1].event_id
        else:
@@ -153,6 +153,9 @@ class RoomSummaryHandler:
        rooms_result: List[JsonDict] = []
        events_result: List[JsonDict] = []

+        if max_rooms_per_space is None or max_rooms_per_space > MAX_ROOMS_PER_SPACE:
+            max_rooms_per_space = MAX_ROOMS_PER_SPACE
+
        while room_queue and len(rooms_result) < MAX_ROOMS:
            queue_entry = room_queue.popleft()
            room_id = queue_entry.room_id
@@ -167,7 +170,7 @@ class RoomSummaryHandler:
            # The client-specified max_rooms_per_space limit doesn't apply to the
            # room_id specified in the request, so we ignore it if this is the
            # first room we are processing.
-            max_children = max_rooms_per_space if processed_rooms else None
+            max_children = max_rooms_per_space if processed_rooms else MAX_ROOMS

            if is_in_room:
                room_entry = await self._summarize_local_room(
@@ -209,7 +212,7 @@ class RoomSummaryHandler:
                        # Before returning to the client, remove the allowed_room_ids
                        # and allowed_spaces keys.
                        room.pop("allowed_room_ids", None)
-                        room.pop("allowed_spaces", None)
+                        room.pop("allowed_spaces", None)  # historical

                        rooms_result.append(room)
                        events.extend(room_entry.children_state_events)
@@ -395,7 +398,7 @@ class RoomSummaryHandler:
                    None,
                    room_id,
                    suggested_only,
-                    # TODO Handle max children.
+                    # Do not limit the maximum children.
                    max_children=None,
                )

@@ -525,6 +528,10 @@ class RoomSummaryHandler:
        rooms_result: List[JsonDict] = []
        events_result: List[JsonDict] = []

+        # Set a limit on the number of rooms to return.
+        if max_rooms_per_space is None or max_rooms_per_space > MAX_ROOMS_PER_SPACE:
+            max_rooms_per_space = MAX_ROOMS_PER_SPACE
+
        while room_queue and len(rooms_result) < MAX_ROOMS:
            room_id = room_queue.popleft()
            if room_id in processed_rooms:
@@ -583,7 +590,9 @@ class RoomSummaryHandler:

        # Iterate through each child and potentially add it, but not its children,
        # to the response.
-        for child_room in root_room_entry.children_state_events:
+        for child_room in itertools.islice(
+            root_room_entry.children_state_events, MAX_ROOMS_PER_SPACE
+        ):
            room_id = child_room.get("state_key")
            assert isinstance(room_id, str)
            # If the room is unknown, skip it.
@@ -633,8 +642,8 @@ class RoomSummaryHandler:
            suggested_only: True if only suggested children should be returned.
                Otherwise, all children are returned.
            max_children:
-                The maximum number of children rooms to include. This is capped
-                to a server-set limit.
+                The maximum number of children rooms to include. A value of None
+                means no limit.

        Returns:
            A room entry if the room should be returned. None, otherwise.
@@ -656,8 +665,13 @@ class RoomSummaryHandler:
            # we only care about suggested children
            child_events = filter(_is_suggested_child_event, child_events)

-        if max_children is None or max_children > MAX_ROOMS_PER_SPACE:
-            max_children = MAX_ROOMS_PER_SPACE
+        # TODO max_children is legacy code for the /spaces endpoint.
+        if max_children is not None:
+            child_iter: Iterable[EventBase] = itertools.islice(
+                child_events, max_children
+            )
+        else:
+            child_iter = child_events

        stripped_events: List[JsonDict] = [
            {
@@ -668,7 +682,7 @@ class RoomSummaryHandler:
                "sender": e.sender,
                "origin_server_ts": e.origin_server_ts,
            }
-            for e in itertools.islice(child_events, max_children)
+            for e in child_iter
        ]
        return _RoomEntry(room_id, room_entry, stripped_events)

@@ -766,6 +780,7 @@ class RoomSummaryHandler:
        try:
            (
                room_response,
+                children_state_events,
                children,
                inaccessible_children,
            ) = await self._federation_client.get_room_hierarchy(
@@ -790,7 +805,7 @@ class RoomSummaryHandler:
        }

        return (
-            _RoomEntry(room_id, room_response, room_response.pop("children_state", ())),
+            _RoomEntry(room_id, room_response, children_state_events),
            children_by_room_id,
            set(inaccessible_children),
        )
@@ -988,12 +1003,14 @@ class RoomSummaryHandler:
            "canonical_alias": stats["canonical_alias"],
            "num_joined_members": stats["joined_members"],
            "avatar_url": stats["avatar"],
+            # plural join_rules is a documentation error but kept for historical
+            # purposes. Should match /publicRooms.
            "join_rules": stats["join_rules"],
+            "join_rule": stats["join_rules"],
            "world_readable": (
                stats["history_visibility"] == HistoryVisibility.WORLD_READABLE
            ),
            "guest_can_join": stats["guest_access"] == "can_join",
-            "creation_ts": create_event.origin_server_ts,
            "room_type": create_event.content.get(EventContentFields.ROOM_TYPE),
        }

@@ -420,10 +420,10 @@ class SearchHandler:
        time_now = self.clock.time_msec()

        for context in contexts.values():
-            context["events_before"] = await self._event_serializer.serialize_events(
+            context["events_before"] = self._event_serializer.serialize_events(
                context["events_before"], time_now
            )
-            context["events_after"] = await self._event_serializer.serialize_events(
+            context["events_after"] = self._event_serializer.serialize_events(
                context["events_after"], time_now
            )

@@ -441,9 +441,7 @@ class SearchHandler:
            results.append(
                {
                    "rank": rank_map[e.event_id],
-                    "result": (
-                        await self._event_serializer.serialize_event(e, time_now)
-                    ),
+                    "result": self._event_serializer.serialize_event(e, time_now),
                    "context": contexts.get(e.event_id, {}),
                }
            )
@@ -457,7 +455,7 @@ class SearchHandler:
        if state_results:
            s = {}
            for room_id, state_events in state_results.items():
-                s[room_id] = await self._event_serializer.serialize_events(
+                s[room_id] = self._event_serializer.serialize_events(
                    state_events, time_now
                )

@@ -126,45 +126,45 @@ class SsoIdentityProvider(Protocol):
        raise NotImplementedError()


-@attr.s
+@attr.s(auto_attribs=True)
 class UserAttributes:
    # the localpart of the mxid that the mapper has assigned to the user.
    # if `None`, the mapper has not picked a userid, and the user should be prompted to
    # enter one.
-    localpart = attr.ib(type=Optional[str])
-    display_name = attr.ib(type=Optional[str], default=None)
-    emails = attr.ib(type=Collection[str], default=attr.Factory(list))
+    localpart: Optional[str]
+    display_name: Optional[str] = None
+    emails: Collection[str] = attr.Factory(list)


-@attr.s(slots=True)
+@attr.s(slots=True, auto_attribs=True)
 class UsernameMappingSession:
    """Data we track about SSO sessions"""

    # A unique identifier for this SSO provider, e.g.  "oidc" or "saml".
-    auth_provider_id = attr.ib(type=str)
+    auth_provider_id: str

    # user ID on the IdP server
-    remote_user_id = attr.ib(type=str)
+    remote_user_id: str

    # attributes returned by the ID mapper
-    display_name = attr.ib(type=Optional[str])
-    emails = attr.ib(type=Collection[str])
+    display_name: Optional[str]
+    emails: Collection[str]

    # An optional dictionary of extra attributes to be provided to the client in the
    # login response.
-    extra_login_attributes = attr.ib(type=Optional[JsonDict])
+    extra_login_attributes: Optional[JsonDict]

    # where to redirect the client back to
-    client_redirect_url = attr.ib(type=str)
+    client_redirect_url: str

    # expiry time for the session, in milliseconds
-    expiry_time_ms = attr.ib(type=int)
+    expiry_time_ms: int

    # choices made by the user
-    chosen_localpart = attr.ib(type=Optional[str], default=None)
-    use_display_name = attr.ib(type=bool, default=True)
-    emails_to_use = attr.ib(type=Collection[str], default=())
-    terms_accepted_version = attr.ib(type=Optional[str], default=None)
+    chosen_localpart: Optional[str] = None
+    use_display_name: bool = True
+    emails_to_use: Collection[str] = ()
+    terms_accepted_version: Optional[str] = None


 # the HTTP cookie used to track the mapping session id
@@ -60,10 +60,6 @@ if TYPE_CHECKING:

 logger = logging.getLogger(__name__)

-# Debug logger for https://github.com/matrix-org/synapse/issues/4422
-issue4422_logger = logging.getLogger("synapse.handler.sync.4422_debug")
-
-
 # Counts the number of times we returned a non-empty sync. `type` is one of
 # "initial_sync", "full_state_sync" or "incremental_sync", `lazy_loaded` is
 # "true" or "false" depending on if the request asked for lazy loaded members or
@@ -102,6 +98,9 @@ class TimelineBatch:
    prev_batch: StreamToken
    events: List[EventBase]
    limited: bool
+    # A mapping of event ID to the bundled aggregations for the above events.
+    # This is only calculated if limited is true.
+    bundled_aggregations: Optional[Dict[str, Dict[str, Any]]] = None

    def __bool__(self) -> bool:
        """Make the result appear empty if there are no updates. This is used
@@ -500,6 +499,22 @@ class SyncHandler:
                event_copy = {k: v for (k, v) in event.items() if k != "room_id"}
                ephemeral_by_room.setdefault(room_id, []).append(event_copy)

+            edus_source = self.event_sources.sources.edus
+            edus, edu_key = await edus_source.get_new_events(
+                user=sync_config.user,
+                from_key=receipt_key,
+                limit=sync_config.filter_collection.ephemeral_limit(),
+                room_ids=room_ids,
+                is_guest=sync_config.is_guest,
+            )
+            now_token = now_token.copy_and_replace("edu_key", edu_key)
+
+            for event in edus:
+                room_id = event["room_id"]
+                # exclude room id, as above
+                event_copy = {k: v for (k, v) in event.items() if k != "room_id"}
+                ephemeral_by_room.setdefault(room_id, []).append(event_copy)
+
        return now_token, ephemeral_by_room

    async def _load_filtered_recents(
@@ -634,10 +649,19 @@ class SyncHandler:

            prev_batch_token = now_token.copy_and_replace("room_key", room_key)

+        # Don't bother to bundle aggregations if the timeline is unlimited,
+        # as clients will have all the necessary information.
+        bundled_aggregations = None
+        if limited or newly_joined_room:
+            bundled_aggregations = await self.store.get_bundled_aggregations(
+                recents, sync_config.user.to_string()
+            )
+
        return TimelineBatch(
            events=recents,
            prev_batch=prev_batch_token,
            limited=limited or newly_joined_room,
+            bundled_aggregations=bundled_aggregations,
        )

    async def get_state_after_event(
@@ -1161,13 +1185,8 @@ class SyncHandler:

        num_events = 0

-        # debug for https://github.com/matrix-org/synapse/issues/4422
+        # debug for https://github.com/matrix-org/synapse/issues/9424
        for joined_room in sync_result_builder.joined:
-            room_id = joined_room.room_id
-            if room_id in newly_joined_rooms:
-                issue4422_logger.debug(
-                    "Sync result for newly joined room %s: %r", room_id, joined_room
-                )
            num_events += len(joined_room.timeline.events)

        log_kv(
@@ -1616,7 +1635,7 @@ class SyncHandler:
        # TODO: Can we `SELECT ignored_user_id FROM ignored_users WHERE ignorer_user_id=?;` instead?
        ignored_account_data = (
            await self.store.get_global_account_data_by_type_for_user(
-                AccountDataTypes.IGNORED_USER_LIST, user_id=user_id
+                user_id=user_id, data_type=AccountDataTypes.IGNORED_USER_LIST
            )
        )

@@ -1740,18 +1759,6 @@ class SyncHandler:
                        old_mem_ev_id, allow_none=True
                    )

-                # debug for #4422
-                if has_join:
-                    prev_membership = None
-                    if old_mem_ev:
-                        prev_membership = old_mem_ev.membership
-                    issue4422_logger.debug(
-                        "Previous membership for room %s with join: %s (event %s)",
-                        room_id,
-                        prev_membership,
-                        old_mem_ev_id,
-                    )
-
                if not old_mem_ev or old_mem_ev.membership != Membership.JOIN:
                    newly_joined_rooms.append(room_id)

@@ -1893,13 +1900,6 @@ class SyncHandler:
                    upto_token=since_token,
                )

-            if newly_joined:
-                # debugging for https://github.com/matrix-org/synapse/issues/4422
-                issue4422_logger.debug(
-                    "RoomSyncResultBuilder events for newly joined room %s: %r",
-                    room_id,
-                    entry.events,
-                )
            room_entries.append(entry)

        return _RoomChanges(
@@ -2077,14 +2077,6 @@ class SyncHandler:
            # `_load_filtered_recents` can't find any events the user should see
            # (e.g. due to having ignored the sender of the last 50 events).

-            if newly_joined:
-                # debug for https://github.com/matrix-org/synapse/issues/4422
-                issue4422_logger.debug(
-                    "Timeline events after filtering in newly-joined room %s: %r",
-                    room_id,
-                    batch,
-                )
-
            # When we join the room (or the client requests full_state), we should
            # send down any existing tags. Usually the user won't have tags in a
            # newly joined room, unless either a) they've joined before or b) the
@@ -731,15 +731,24 @@ class SimpleHttpClient:
        # straight back in again

        try:
-            length = await make_deferred_yieldable(
-                read_body_with_max_size(response, output_stream, max_size)
-            )
+            d = read_body_with_max_size(response, output_stream, max_size)
+
+            # Ensure that the body is not read forever.
+            d = timeout_deferred(d, 30, self.hs.get_reactor())
+
+            length = await make_deferred_yieldable(d)
        except BodyExceededMaxSize:
            raise SynapseError(
                HTTPStatus.BAD_GATEWAY,
                "Requested file is too large > %r bytes" % (max_size,),
                Codes.TOO_LARGE,
            )
+        except defer.TimeoutError:
+            raise SynapseError(
+                HTTPStatus.BAD_GATEWAY,
+                "Requested file took too long to download",
+                Codes.TOO_LARGE,
+            )
        except Exception as e:
            raise SynapseError(
                HTTPStatus.BAD_GATEWAY, ("Failed to download remote body: %s" % e)
@@ -32,9 +32,9 @@ class ProxyConnectError(ConnectError):
    pass


-@attr.s
+@attr.s(auto_attribs=True)
 class ProxyCredentials:
-    username_password = attr.ib(type=bytes)
+    username_password: bytes

    def as_proxy_authorization_value(self) -> bytes:
        """
@@ -123,37 +123,37 @@ class ByteParser(ByteWriteable, Generic[T], abc.ABC):
        pass


-@attr.s(slots=True, frozen=True)
+@attr.s(slots=True, frozen=True, auto_attribs=True)
 class MatrixFederationRequest:
-    method = attr.ib(type=str)
+    method: str
    """HTTP method
    """

-    path = attr.ib(type=str)
+    path: str
    """HTTP path
    """

-    destination = attr.ib(type=str)
+    destination: str
    """The remote server to send the HTTP request to.
    """

-    json = attr.ib(default=None, type=Optional[JsonDict])
+    json: Optional[JsonDict] = None
    """JSON to send in the body.
    """

-    json_callback = attr.ib(default=None, type=Optional[Callable[[], JsonDict]])
+    json_callback: Optional[Callable[[], JsonDict]] = None
    """A callback to generate the JSON.
    """

-    query = attr.ib(default=None, type=Optional[dict])
+    query: Optional[dict] = None
    """Query arguments.
    """

-    txn_id = attr.ib(default=None, type=Optional[str])
+    txn_id: Optional[str] = None
    """Unique ID for this request (for logging)
    """

-    uri = attr.ib(init=False, type=bytes)
+    uri: bytes = attr.ib(init=False)
    """The URI of this request
    """

@@ -534,9 +534,9 @@ class XForwardedForRequest(SynapseRequest):


@implementer(IAddress)
-@attr.s(frozen=True, slots=True)
+@attr.s(frozen=True, slots=True, auto_attribs=True)
 class _XForwardedForAddress:
-    host = attr.ib(type=str)
+    host: str


 class SynapseSite(Site):
@@ -39,7 +39,7 @@ from twisted.python.failure import Failure
 logger = logging.getLogger(__name__)


-@attr.s
+@attr.s(slots=True, auto_attribs=True)
@implementer(IPushProducer)
 class LogProducer:
    """
@@ -54,10 +54,10 @@ class LogProducer:

    # This is essentially ITCPTransport, but that is missing certain fields
    # (connected and registerProducer) which are part of the implementation.
-    transport = attr.ib(type=Connection)
-    _format = attr.ib(type=Callable[[logging.LogRecord], str])
-    _buffer = attr.ib(type=deque)
-    _paused = attr.ib(default=False, type=bool, init=False)
+    transport: Connection
+    _format: Callable[[logging.LogRecord], str]
+    _buffer: Deque[logging.LogRecord]
+    _paused: bool = attr.ib(default=False, init=False)

    def pauseProducing(self):
        self._paused = True
--- a/Show More
+++ b/Show More
				`@@ -0,0 +1 @@`
				Include the bundled aggregations in the `/sync` response, per [MSC2675](https://github.com/matrix-org/matrix-doc/pull/2675).
				`@@ -0,0 +1 @@`
				`Remove account data (including client config, push rules and ignored users) upon user deactivation.`
				`@@ -0,0 +1 @@`
				`Add admin API to reset connection timeouts for remote server.`
				`@@ -0,0 +1 @@`
				`Add an admin API to get a list of rooms that federate with a given remote homeserver.`
				`@@ -0,0 +1 @@`
				`Fix a long-standing bug when previewing Reddit URLs which do not contain an image.`
				`@@ -0,0 +1 @@`
				`Fix a long-standing bug that media streams could cause long-lived connections when generating URL previews.`
				`@@ -0,0 +1 @@`
				Preparation for database schema simplifications: add `state_key` and `rejection_reason` columns to `events` table.
				`@@ -0,0 +1 @@`
				Add `FrozenEvent.get_state_key` and use it in a couple of places.
				`@@ -0,0 +1 @@`
				`Preparation for reducing Postgres serialization errors: allow setting transaction isolation level. Contributed by Nick @ Beeper.`
				`@@ -0,0 +1 @@`
				`Docker: skip the initial amd64-only build and go straight to multiarch.`