disallow-untyped-defs for synapse.replication.http._base

Merge branch 'develop' into dmr/typing/wip
Pin mypy versions
2021-10-07 19:06:45 +01:00 · 2021-10-07 17:20:15 +01:00 · 2021-10-07 14:41:41 +01:00 · 2021-10-07 09:38:31 -04:00 · 2021-10-07 13:26:11 +01:00 · 2021-10-07 11:43:25 +00:00
376 changed files with 7072 additions and 3839 deletions
@@ -25,7 +25,7 @@ python -m synapse.app.homeserver --generate-keys -c .ci/sqlite-config.yaml
 echo "--- Prepare test database"

 # Make sure the SQLite3 database is using the latest schema and has no pending background update.
-scripts-dev/update_database --database-config .ci/sqlite-config.yaml
+scripts/update_synapse_database --database-config .ci/sqlite-config.yaml --run-background-updates

 # Create the PostgreSQL database.
 .ci/scripts/postgres_exec.py "CREATE DATABASE synapse"
@@ -46,7 +46,7 @@ echo "--- Prepare empty SQLite database"
 # we do this by deleting the sqlite db, and then doing the same again.
 rm .ci/test_db.db

-scripts-dev/update_database --database-config .ci/sqlite-config.yaml
+scripts/update_synapse_database --database-config .ci/sqlite-config.yaml --run-background-updates

 # re-create the PostgreSQL database.
 .ci/scripts/postgres_exec.py \
@@ -0,0 +1,2 @@
+# Automatically request reviews from the synapse-core team when a pull request comes in.
+* @matrix-org/synapse-core
@@ -61,6 +61,5 @@ jobs:
        uses: peaceiris/actions-gh-pages@068dc23d9710f1ba62e86896f84735d869951305 # v3.8.0
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
-          keep_files: true
          publish_dir: ./book
          destination_dir: ./${{ steps.vars.outputs.branch-version }}
@@ -76,11 +76,11 @@ jobs:
    runs-on: ubuntu-latest
    strategy:
      matrix:
-        python-version: ["3.6", "3.7", "3.8", "3.9"]
+        python-version: ["3.6", "3.7", "3.8", "3.9", "3.10"]
        database: ["sqlite"]
        include:
          # Newest Python without optional deps
-          - python-version: "3.9"
+          - python-version: "3.10"
            toxenv: "py-noextras,combine"

          # Oldest Python with PostgreSQL
@@ -88,10 +88,10 @@ jobs:
            database: "postgres"
            postgres-version: "9.6"

-          # Newest Python with PostgreSQL
-          - python-version: "3.9"
+          # Newest Python with newest PostgreSQL
+          - python-version: "3.10"
            database: "postgres"
-            postgres-version: "13"
+            postgres-version: "14"

    steps:
      - uses: actions/checkout@v2
@@ -192,6 +192,7 @@ jobs:
      volumes:
        - ${{ github.workspace }}:/src
      env:
+        SYTEST_BRANCH: ${{ github.head_ref }}
        POSTGRES: ${{ matrix.postgres && 1}}
        MULTI_POSTGRES: ${{ (matrix.postgres == 'multi-postgres') && 1}}
        WORKERS: ${{ matrix.workers && 1 }}
@@ -255,8 +256,8 @@ jobs:
          - python-version: "3.6"
            postgres-version: "9.6"

-          - python-version: "3.9"
-            postgres-version: "13"
+          - python-version: "3.10"
+            postgres-version: "14"

    services:
      postgres:
@@ -40,6 +40,7 @@ __pycache__/
 /.coverage*
 /.mypy_cache/
 /.tox
+/.tox-pg-container
 /build/
 /coverage.*
 /dist/
@@ -1,7 +1,127 @@
-Synapse 1.43.0rc1 (2021-09-14)
+Synapse 1.44.0 (2021-10-05)
+===========================
+
+No significant changes since 1.44.0rc3.
+
+
+Synapse 1.44.0rc3 (2021-10-04)
 ==============================

-This release drops support for the deprecated, unstable API for [MSC2858](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix), as well as the undocumented `experimental.msc2858_enabled` config option. Client authors should update their clients to use the stable API, available since Synapse 1.30.
+Bugfixes
+--------
+
+- Fix a bug introduced in Synapse v1.40.0 where changing a user's display name or avatar in a restricted room would cause an authentication error. ([\#10933](https://github.com/matrix-org/synapse/issues/10933))
+- Fix `/admin/whois/{user_id}` endpoint, which was broken in v1.44.0rc1. ([\#10968](https://github.com/matrix-org/synapse/issues/10968))
+
+
+Synapse 1.44.0rc2 (2021-09-30)
+==============================
+
+Bugfixes
+--------
+
+- Fix a bug introduced in v1.44.0rc1 which caused the experimental [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint to return a 500 error. ([\#10938](https://github.com/matrix-org/synapse/issues/10938))
+- Fix a bug introduced in v1.44.0rc1 which prevented sending presence events to application services. ([\#10944](https://github.com/matrix-org/synapse/issues/10944))
+
+
+Improved Documentation
+----------------------
+
+- Minor updates to the installation instructions. ([\#10919](https://github.com/matrix-org/synapse/issues/10919))
+
+
+Synapse 1.44.0rc1 (2021-09-29)
+==============================
+
+Features
+--------
+
+- Only allow the [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send?chunk_id=xxx` endpoint to connect to an already existing insertion event. ([\#10776](https://github.com/matrix-org/synapse/issues/10776))
+- Improve oEmbed URL previews by processing the author name, photo, and video information. ([\#10814](https://github.com/matrix-org/synapse/issues/10814), [\#10819](https://github.com/matrix-org/synapse/issues/10819))
+- Speed up responding with large JSON objects to requests. ([\#10868](https://github.com/matrix-org/synapse/issues/10868), [\#10905](https://github.com/matrix-org/synapse/issues/10905))
+- Add a `user_may_create_room_with_invites` spam checker callback to allow modules to allow or deny a room creation request based on the invites and/or 3PID invites it includes. ([\#10898](https://github.com/matrix-org/synapse/issues/10898))
+
+
+Bugfixes
+--------
+
+- Fix a long-standing bug that caused an `AssertionError` when purging history in certain rooms. Contributed by @Kokokokoka. ([\#10690](https://github.com/matrix-org/synapse/issues/10690))
+- Fix a long-standing bug which caused deactivated users that were later reactivated to be missing from the user directory. ([\#10782](https://github.com/matrix-org/synapse/issues/10782))
+- Fix a long-standing bug that caused unbanning a user by sending a membership event to fail. Contributed by @aaronraimist. ([\#10807](https://github.com/matrix-org/synapse/issues/10807))
+- Fix a long-standing bug where logging contexts would go missing when federation requests time out. ([\#10810](https://github.com/matrix-org/synapse/issues/10810))
+- Fix a long-standing bug causing an error in the deprecated `/initialSync` endpoint when using the undocumented `from` and `to` parameters. ([\#10827](https://github.com/matrix-org/synapse/issues/10827))
+- Fix a bug causing the `remove_stale_pushers` background job to repeatedly fail and log errors. This bug affected Synapse servers that had been upgraded from version 1.28 or older and are using SQLite. ([\#10843](https://github.com/matrix-org/synapse/issues/10843))
+- Fix a long-standing bug in Unicode support of the room search admin API breaking search for rooms with non-ASCII characters. ([\#10859](https://github.com/matrix-org/synapse/issues/10859))
+- Fix a bug introduced in Synapse 1.37.0 which caused `knock` membership events which we sent to remote servers to be incorrectly stored in the local database. ([\#10873](https://github.com/matrix-org/synapse/issues/10873))
+- Fix invalidating one-time key count cache after claiming keys. The bug was introduced in Synapse v1.41.0. Contributed by Tulir at Beeper. ([\#10875](https://github.com/matrix-org/synapse/issues/10875))
+- Fix a long-standing bug causing application service users to be subject to MAU blocking if the MAU limit had been reached, even if configured not to be blocked. ([\#10881](https://github.com/matrix-org/synapse/issues/10881))
+- Fix a long-standing bug which could cause events pulled over federation to be incorrectly rejected. ([\#10907](https://github.com/matrix-org/synapse/issues/10907))
+- Fix a long-standing bug causing URL cache files to be stored in storage providers. Server admins may safely delete the `url_cache/` and `url_cache_thumbnails/` directories from any configured storage providers to reclaim space. ([\#10911](https://github.com/matrix-org/synapse/issues/10911))
+- Fix a long-standing bug leading to race conditions when creating media store and config directories. ([\#10913](https://github.com/matrix-org/synapse/issues/10913))
+
+
+Improved Documentation
+----------------------
+
+- Fix some crashes in the Module API example code, by adding JSON encoding/decoding. ([\#10845](https://github.com/matrix-org/synapse/issues/10845))
+- Add developer documentation about experimental configuration flags. ([\#10865](https://github.com/matrix-org/synapse/issues/10865))
+- Properly remove deleted files from GitHub pages when generating the documentation. ([\#10869](https://github.com/matrix-org/synapse/issues/10869))
+
+
+Internal Changes
+----------------
+
+- Fix GitHub Actions config so we can run sytest on synapse from parallel branches. ([\#10659](https://github.com/matrix-org/synapse/issues/10659))
+- Split out [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) meta events to their own fields in the `/batch_send` response. ([\#10777](https://github.com/matrix-org/synapse/issues/10777))
+- Add missing type hints to REST servlets. ([\#10785](https://github.com/matrix-org/synapse/issues/10785), [\#10817](https://github.com/matrix-org/synapse/issues/10817))
+- Simplify the internal logic which maintains the user directory database tables. ([\#10796](https://github.com/matrix-org/synapse/issues/10796))
+- Use direct references to config flags. ([\#10812](https://github.com/matrix-org/synapse/issues/10812), [\#10885](https://github.com/matrix-org/synapse/issues/10885), [\#10893](https://github.com/matrix-org/synapse/issues/10893), [\#10897](https://github.com/matrix-org/synapse/issues/10897))
+- Specify the type of token in generic "Invalid token" error messages. ([\#10815](https://github.com/matrix-org/synapse/issues/10815))
+- Make `StateFilter` frozen so it is hashable. ([\#10816](https://github.com/matrix-org/synapse/issues/10816))
+- Fix a long-standing bug where an `m.room.message` event containing a null byte would cause an internal server error. ([\#10820](https://github.com/matrix-org/synapse/issues/10820))
+- Add type hints to the state database. ([\#10823](https://github.com/matrix-org/synapse/issues/10823))
+- Opt out of cache expiry for `get_users_who_share_room_with_user`, to hopefully improve `/sync` performance when you
+  haven't synced recently. ([\#10826](https://github.com/matrix-org/synapse/issues/10826))
+- Track cache eviction rates more finely in Prometheus's monitoring. ([\#10829](https://github.com/matrix-org/synapse/issues/10829))
+- Add missing type hints to `synapse.handlers`. ([\#10831](https://github.com/matrix-org/synapse/issues/10831), [\#10856](https://github.com/matrix-org/synapse/issues/10856))
+- Extend the Module API to let plug-ins check whether an ID is local and to access IP + User Agent data. ([\#10833](https://github.com/matrix-org/synapse/issues/10833))
+- Factor out PNG image data to a constant to be used in several tests. ([\#10834](https://github.com/matrix-org/synapse/issues/10834))
+- Add a test to ensure state events sent by modules get persisted correctly. ([\#10835](https://github.com/matrix-org/synapse/issues/10835))
+- Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) fields and event types from `chunk` to `batch` to match the `/batch_send` endpoint. ([\#10838](https://github.com/matrix-org/synapse/issues/10838))
+- Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` query parameter from `?prev_event` to more obvious usage with `?prev_event_id`. ([\#10839](https://github.com/matrix-org/synapse/issues/10839))
+- Add type hints to `synapse.http.site`. ([\#10867](https://github.com/matrix-org/synapse/issues/10867))
+- Include outlier status when we log V2 or V3 events. ([\#10879](https://github.com/matrix-org/synapse/issues/10879))
+- Break down Grafana's cache expiry time series based on reason for eviction, c.f. [\#10829](https://github.com/matrix-org/synapse/issues/10829). ([\#10880](https://github.com/matrix-org/synapse/issues/10880))
+- Clean up some of the federation event authentication code for clarity. ([\#10883](https://github.com/matrix-org/synapse/issues/10883), [\#10884](https://github.com/matrix-org/synapse/issues/10884), [\#10896](https://github.com/matrix-org/synapse/issues/10896), [\#10901](https://github.com/matrix-org/synapse/issues/10901))
+- Allow the `.` and `~` characters when creating registration tokens as per the change to [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231). ([\#10887](https://github.com/matrix-org/synapse/issues/10887))
+- Clean up some unnecessary parentheses in places around the codebase. ([\#10889](https://github.com/matrix-org/synapse/issues/10889))
+- Improve type hinting in the user directory code. ([\#10891](https://github.com/matrix-org/synapse/issues/10891))
+- Update development testing script `test_postgresql.sh` to use a supported Python version and make re-runs quicker. ([\#10906](https://github.com/matrix-org/synapse/issues/10906))
+- Document and summarize changes in schema version `61` – `64`. ([\#10917](https://github.com/matrix-org/synapse/issues/10917))
+- Update release script to sign the newly created git tags. ([\#10925](https://github.com/matrix-org/synapse/issues/10925))
+- Fix Debian builds due to `dh-virtualenv` no longer being able to build their docs. ([\#10931](https://github.com/matrix-org/synapse/issues/10931))
+
+
+Synapse 1.43.0 (2021-09-21)
+===========================
+
+This release drops support for the deprecated, unstable API for [MSC2858 (Multiple SSO Identity Providers)](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix), as well as the undocumented `experimental.msc2858_enabled` config option. Client authors should update their clients to use the stable API, available since Synapse 1.30.
+
+The documentation has been updated with configuration for routing `/spaces`, `/hierarchy` and `/summary` to workers. See [the upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.43/docs/upgrade.md#upgrading-to-v1430) for more details.
+
+No significant changes since 1.43.0rc2.
+
+Synapse 1.43.0rc2 (2021-09-17)
+==============================
+
+Bugfixes
+--------
+
+- Added opentracing logging to help debug [\#9424](https://github.com/matrix-org/synapse/issues/9424). ([\#10828](https://github.com/matrix-org/synapse/issues/10828))
+
+
+Synapse 1.43.0rc1 (2021-09-14)
+==============================

 Features
 --------
@@ -55,11 +55,8 @@ solutions. The hope is for Matrix to act as the building blocks for a new
 generation of fully open and interoperable messaging and VoIP apps for the
 internet.

-Synapse is a reference "homeserver" implementation of Matrix from the core
-development team at matrix.org, written in Python/Twisted.  It is intended to
-showcase the concept of Matrix and let folks see the spec in the context of a
-codebase and let you run your own homeserver and generally help bootstrap the
-ecosystem.
+Synapse is a Matrix "homeserver" implementation developed by the matrix.org core 
+team, written in Python 3/Twisted.

 In Matrix, every user runs one or more Matrix clients, which connect through to
 a Matrix homeserver. The homeserver stores all their personal chat history and
@@ -288,7 +285,7 @@ Quick start

 Before setting up a development environment for synapse, make sure you have the
 system dependencies (such as the python header files) installed - see
-`Installing from source <https://matrix-org.github.io/synapse/latest/setup/installation.html#installing-from-source>`_.
+`Platform-specific prerequisites <https://matrix-org.github.io/synapse/latest/setup/installation.html#platform-specific-prerequisites>`_.

 To check out a synapse for development, clone the git repo into a working
 directory of your choice::
@@ -0,0 +1 @@
+Ensure `(room_id, next_batch_id)` is unique across [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) insertion events in rooms to avoid cross-talk/conflicts between batches.
@@ -0,0 +1 @@
+Improve type hinting in `synapse.util`.
@@ -0,0 +1 @@
+Add a `user_may_send_3pid_invite` spam checker callback for modules to allow or deny 3PID invites.
@@ -0,0 +1 @@
+Fix type hints to be compatible with an upcoming change to Twisted.
@@ -0,0 +1 @@
+Update utility code to handle C implementations of frozendict.
@@ -0,0 +1 @@
+Drop old functionality which maintained database compatibility with Synapse versions before 1.31.
@@ -0,0 +1 @@
+Add a spam checker callback to allow or deny room joins.
@@ -0,0 +1 @@
+Clean-up configuration helper classes for the `ServerConfig` class.
@@ -0,0 +1 @@
+Use direct references to config flags.
@@ -0,0 +1 @@
+Fix a minor bug in the response to `/_matrix/client/r0/voip/turnServer`. Contributed by @lukaslihotzki.
@@ -0,0 +1 @@
+Fix a bug where empty `yyyy-mm-dd/` directories would be left behind in the media store's `url_cache_thumbnails/` directory.
@@ -0,0 +1 @@
+Clean up some of the federation event authentication code for clarity.
@@ -0,0 +1 @@
+Fix a bug introduced in Synapse v1.40.0 where the signature checks for room version 8/9 could be applied to earlier room versions in some situations.
@@ -0,0 +1 @@
+Refactor various parts of the codebase to use `RoomVersion` objects instead of room version identifier strings.
@@ -0,0 +1 @@
+Refactor user directory tests in preparation for upcoming changes.
@@ -0,0 +1 @@
+Include the event id in the logcontext when handling PDUs received over federation.
@@ -0,0 +1 @@
+Fix logged errors in unit tests.
@@ -0,0 +1 @@
+Clean up some of the federation event authentication code for clarity.
@@ -0,0 +1 @@
+Fix a broken test to ensure that consent configuration works during registration.
@@ -0,0 +1 @@
+Fixes a long-standing bug wherin deactivated users still count towards the mau limit.
@@ -0,0 +1 @@
+Include an `update_synapse_database` script in the distribution. Contributed by @Fizzadar at Beeper.
@@ -0,0 +1 @@
+Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state.
@@ -0,0 +1 @@
+Add type hints to filtering classes.
@@ -0,0 +1 @@
+Use direct references to config flags.
@@ -0,0 +1 @@
+Fix a long-standing bug where rebuilding the user directory wouldn't exclude support and disabled users.
@@ -0,0 +1 @@
+Add type-hint to `HomeserverTestcase.setup_test_homeserver`.
@@ -0,0 +1 @@
+Fix [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint rejecting subsequent batches with unknown batch ID error in existing room versions from the room creator.
@@ -0,0 +1 @@
+Fix the test utility function `create_room_as` so that `is_public=True` will explicitly set the `visibility` parameter of room creation requests to `public`. Contributed by @AndrewFerr.
@@ -0,0 +1 @@
+Change wording ("reference homeserver") in Synapse repository documentation. Contributed by @maxkratz.
@@ -0,0 +1 @@
+Fix a dead URL in development documentation (SAML) and change wording from "Riot" to "Element". Contributed by @maxkratz.
@@ -0,0 +1 @@
+Fix a bug that could leak local users' per-room nicknames and avatars when the user directory is rebuilt.
@@ -0,0 +1 @@
+Fix a long-standing bug where the remainder of a batch of user directory changes would be silently dropped if the server left a room early in the batch.
@@ -0,0 +1 @@
+Log stack traces when a missing opentracing span is detected.
@@ -0,0 +1 @@
+Use direct references to config flags.
@@ -0,0 +1 @@
+Clean up some of the federation event authentication code for clarity.
@@ -0,0 +1 @@
+Clean up some of the federation event authentication code for clarity.
@@ -0,0 +1 @@
+Clean up some of the federation event authentication code for clarity.
@@ -0,0 +1 @@
+Add additional content to the Welcome and Overview page of the documentation.
@@ -0,0 +1 @@
+Update links to MSCs in documentation. Contributed by @dklimpel.
@@ -0,0 +1 @@
+Update GHA config to run tests against Python 3.10 and PostgreSQL 14.
@@ -0,0 +1 @@
+Add a `CODEOWNERS` file to automatically request reviews from the `@matrix-org/synapse-core` team on new pull requests.
@@ -0,0 +1 @@
+Correct a bugfix introduced in Synapse v1.44.0 that wouldn't catch every error of the connection breaks before a response could be written to it.
@@ -0,0 +1 @@
+Fix a long-standing bug where local users' per-room nicknames/avatars were visible to anyone who could see you in the user_directory.
@@ -0,0 +1 @@
+Add further type hints to `synapse.state`.
@@ -0,0 +1 @@
+Use type stubs for jsonschema, pyOpenSSL and Pillow when running mypy in CI.
@@ -0,0 +1 @@
+Clean up some of the federation event authentication code for clarity.
@@ -0,0 +1 @@
+Clean up some of the federation event authentication code for clarity.
@@ -0,0 +1 @@
+Ensure that cache config tests do not share state.
@@ -0,0 +1 @@
+Add [MSC3069](https://github.com/matrix-org/matrix-doc/pull/3069) support to `/account/whoami`.
@@ -6785,7 +6785,7 @@
              "expr": "rate(synapse_util_caches_cache:evicted_size{instance=\"$instance\",job=~\"$job\",index=~\"$index\"}[$bucket_size])",
              "format": "time_series",
              "intervalFactor": 1,
-              "legendFormat": "{{name}} {{job}}-{{index}}",
+              "legendFormat": "{{name}} ({{reason}}) {{job}}-{{index}}",
              "refId": "A"
            }
          ],
@@ -10888,5 +10888,5 @@
  "timezone": "",
  "title": "Synapse",
  "uid": "000000012",
-  "version": 99
+  "version": 100
 }
@@ -1,3 +1,46 @@
+matrix-synapse-py3 (1.44.0~rc2+nmu1) UNRELEASED; urgency=medium
+
+  [ Nick @ Beeper ]
+  * Include an `update_synapse_database` script in the distribution.
+
+ -- root <root@f7b8a71098d3>  Mon, 04 Oct 2021 13:29:26 +0000
+
+matrix-synapse-py3 (1.44.0) stable; urgency=medium
+
+  * New synapse release 1.44.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 05 Oct 2021 13:43:57 +0100
+
+matrix-synapse-py3 (1.44.0~rc3) stable; urgency=medium
+
+  * New synapse release 1.44.0~rc3.
+
+ -- Synapse Packaging team <packages@matrix.org>  Mon, 04 Oct 2021 14:57:22 +0100
+
+matrix-synapse-py3 (1.44.0~rc2) stable; urgency=medium
+
+  * New synapse release 1.44.0~rc2.
+
+ -- Synapse Packaging team <packages@matrix.org>  Thu, 30 Sep 2021 12:39:10 +0100
+
+matrix-synapse-py3 (1.44.0~rc1) stable; urgency=medium
+
+  * New synapse release 1.44.0~rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 28 Sep 2021 13:41:28 +0100
+
+matrix-synapse-py3 (1.43.0) stable; urgency=medium
+
+  * New synapse release 1.43.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 21 Sep 2021 11:49:05 +0100
+
+matrix-synapse-py3 (1.43.0~rc2) stable; urgency=medium
+
+  * New synapse release 1.43.0~rc2.
+
+ -- Synapse Packaging team <packages@matrix.org>  Fri, 17 Sep 2021 10:43:21 +0100
+
 matrix-synapse-py3 (1.43.0~rc1) stable; urgency=medium

  * New synapse release 1.43.0~rc1.
@@ -3,3 +3,4 @@ opt/venvs/matrix-synapse/bin/register_new_matrix_user usr/bin/register_new_matri
 opt/venvs/matrix-synapse/bin/synapse_port_db usr/bin/synapse_port_db
 opt/venvs/matrix-synapse/bin/synapse_review_recent_signups usr/bin/synapse_review_recent_signups
 opt/venvs/matrix-synapse/bin/synctl usr/bin/synctl
+opt/venvs/matrix-synapse/bin/update_synapse_database usr/bin/update_synapse_database
@@ -47,8 +47,9 @@ RUN apt-get update -qq -o Acquire::Languages=none \
    && cd /dh-virtualenv \
    && env DEBIAN_FRONTEND=noninteractive mk-build-deps -ri -t "apt-get -y --no-install-recommends"

-# build it
-RUN cd /dh-virtualenv && dpkg-buildpackage -us -uc -b
+# Build it. Note that building the docs doesn't work due to differences in
+# Sphinx APIs across versions/distros.
+RUN cd /dh-virtualenv && DEB_BUILD_OPTIONS=nodoc dpkg-buildpackage -us -uc -b

 ###
 ### Stage 1
@@ -1,6 +1,6 @@
 # Use the Sytest image that comes with a lot of the build dependencies
 # pre-installed
-FROM matrixdotorg/sytest:latest
+FROM matrixdotorg/sytest:bionic

 # The Sytest image doesn't come with python, so install that
 RUN apt-get update && apt-get -qq install -y python3 python3-dev python3-pip
@@ -8,5 +8,23 @@ RUN apt-get update && apt-get -qq install -y python3 python3-dev python3-pip
 # We need tox to run the tests in run_pg_tests.sh
 RUN python3 -m pip install tox

-ADD run_pg_tests.sh /pg_tests.sh
-ENTRYPOINT /pg_tests.sh
+# Initialise the db
+RUN su -c '/usr/lib/postgresql/10/bin/initdb -D /var/lib/postgresql/data -E "UTF-8" --lc-collate="C.UTF-8" --lc-ctype="C.UTF-8" --username=postgres' postgres
+
+# Add a user with our UID and GID so that files get created on the host owned
+# by us, not root.
+ARG UID
+ARG GID
+RUN groupadd --gid $GID user
+RUN useradd --uid $UID --gid $GID --groups sudo --no-create-home user
+
+# Ensure we can start postgres by sudo-ing as the postgres user.
+RUN apt-get update && apt-get -qq install -y sudo
+RUN echo "user ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
+
+ADD run_pg_tests.sh /run_pg_tests.sh
+# Use the "exec form" of ENTRYPOINT (https://docs.docker.com/engine/reference/builder/#entrypoint)
+# so that we can `docker run` this container and pass arguments to pg_tests.sh
+ENTRYPOINT ["/run_pg_tests.sh"]
+
+USER user
@@ -10,11 +10,10 @@ set -e
 # Set PGUSER so Synapse's tests know what user to connect to the database with
 export PGUSER=postgres

-# Initialise & start the database
-su -c '/usr/lib/postgresql/9.6/bin/initdb -D /var/lib/postgresql/data -E "UTF-8" --lc-collate="en_US.UTF-8" --lc-ctype="en_US.UTF-8" --username=postgres' postgres
-su -c '/usr/lib/postgresql/9.6/bin/pg_ctl -w -D /var/lib/postgresql/data start' postgres
+# Start the database
+sudo -u postgres /usr/lib/postgresql/10/bin/pg_ctl -w -D /var/lib/postgresql/data start

 # Run the tests
 cd /src
 export TRIAL_FLAGS="-j 4"
-tox --workdir=/tmp -e py35-postgres
+tox --workdir=./.tox-pg-container -e py36-postgres "$@"
@@ -3,7 +3,7 @@
 ## Historical Note
 This document was originally written to guide server admins through the upgrade
 path towards Synapse 1.0. Specifically,
-[MSC1711](https://github.com/matrix-org/matrix-doc/blob/master/proposals/1711-x509-for-federation.md)
+[MSC1711](https://github.com/matrix-org/matrix-doc/blob/main/proposals/1711-x509-for-federation.md)
 required that all servers present valid TLS certificates on their federation
 API. Admins were encouraged to achieve compliance from version 0.99.0 (released
 in February 2019) ahead of version 1.0 (released June 2019) enforcing the
@@ -282,7 +282,7 @@ coffin of the Perspectives project (which was already pretty dead). So, the
 Spec Core Team decided that a better approach would be to mandate valid TLS
 certificates for federation alongside the rest of the Web. More details can be
 found in
-[MSC1711](https://github.com/matrix-org/matrix-doc/blob/master/proposals/1711-x509-for-federation.md#background-the-failure-of-the-perspectives-approach).
+[MSC1711](https://github.com/matrix-org/matrix-doc/blob/main/proposals/1711-x509-for-federation.md#background-the-failure-of-the-perspectives-approach).

 This results in a breaking change, which is disruptive, but absolutely critical
 for the security model. However, the existence of Let's Encrypt as a trivial
@@ -6,9 +6,9 @@ Please update any links to point to the new website instead.
 ## About

 This directory currently holds a series of markdown files documenting how to install, use
-and develop Synapse, the reference Matrix homeserver. The documentation is readable directly
-from this repository, but it is recommended to instead browse through the
-[website](https://matrix-org.github.io/synapse) for easier discoverability.
+and develop Synapse. The documentation is readable directly from this repository, but it is 
+recommended to instead browse through the [website](https://matrix-org.github.io/synapse) for 
+easier discoverability.

 ## Adding to the documentation

@@ -74,6 +74,7 @@
  - [Testing]()
  - [OpenTracing](opentracing.md)
  - [Database Schemas](development/database_schema.md)
+  - [Experimental features](development/experimental_features.md)
  - [Synapse Architecture]()
    - [Log Contexts](log_contexts.md)
    - [Replication](replication.md)
@@ -63,7 +63,7 @@ TBD

 # 5. Get in touch.

-Join our developer community on Matrix: #synapse-dev:matrix.org !
+Join our developer community on Matrix: [#synapse-dev:matrix.org](https://matrix.to/#/#synapse-dev:matrix.org)!


 # 6. Pick an issue.
@@ -170,6 +170,53 @@ To increase the log level for the tests, set `SYNAPSE_TEST_LOG_LEVEL`:
 SYNAPSE_TEST_LOG_LEVEL=DEBUG trial tests
 ```

+### Running tests under PostgreSQL
+
+Invoking `trial` as above will use an in-memory SQLite database. This is great for
+quick development and testing. However, we recommend using a PostgreSQL database
+in production (and indeed, we have some code paths specific to each database).
+This means that we need to run our unit tests against PostgreSQL too. Our CI does
+this automatically for pull requests and release candidates, but it's sometimes
+useful to reproduce this locally.
+
+To do so, [configure Postgres](../postgres.md) and run `trial` with the
+following environment variables matching your configuration:
+
+- `SYNAPSE_POSTGRES` to anything nonempty
+- `SYNAPSE_POSTGRES_HOST`
+- `SYNAPSE_POSTGRES_USER`
+- `SYNAPSE_POSTGRES_PASSWORD`
+
+For example:
+
+```shell
+export SYNAPSE_POSTGRES=1
+export SYNAPSE_POSTGRES_HOST=localhost
+export SYNAPSE_POSTGRES_USER=postgres
+export SYNAPSE_POSTGRES_PASSWORD=mydevenvpassword
+trial
+```
+
+#### Prebuilt container
+
+Since configuring PostgreSQL can be fiddly, we can make use of a pre-made
+Docker container to set up PostgreSQL and run our tests for us. To do so, run
+
+```shell
+scripts-dev/test_postgresql.sh
+```
+
+Any extra arguments to the script will be passed to `tox` and then to `trial`,
+so we can run a specific test in this container with e.g.
+
+```shell
+scripts-dev/test_postgresql.sh tests.replication.test_sharded_event_persister.EventPersisterShardTestCase
+```
+
+The container creates a folder in your Synapse checkout called
+`.tox-pg-container` and uses this as a tox environment. The output of any
+`trial` runs goes into `_trial_temp` in your synapse source directory — the same
+as running `trial` directly on your host machine.

 ## Run the integration tests ([Sytest](https://github.com/matrix-org/sytest)).

@@ -0,0 +1,37 @@
+# Implementing experimental features in Synapse
+
+It can be desirable to implement "experimental" features which are disabled by
+default and must be explicitly enabled via the Synapse configuration. This is
+applicable for features which:
+
+* Are unstable in the Matrix spec (e.g. those defined by an MSC that has not yet been merged).
+* Developers are not confident in their use by general Synapse administrators/users
+  (e.g. a feature is incomplete, buggy, performs poorly, or needs further testing).
+
+Note that this only really applies to features which are expected to be desirable
+to a broad audience. The [module infrastructure](../modules/index.md) should
+instead be investigated for non-standard features.
+
+Guarding experimental features behind configuration flags should help with some
+of the following scenarios:
+
+* Ensure that clients do not assume that unstable features exist (failing
+  gracefully if they do not).
+* Unstable features do not become de-facto standards and can be removed
+  aggressively (since only those who have opted-in will be affected).
+* Ease finding the implementation of unstable features in Synapse (for future
+  removal or stabilization).
+* Ease testing a feature (or removal of feature) due to enabling/disabling without
+  code changes. It also becomes possible to ask for wider testing, if desired.
+
+Experimental configuration flags should be disabled by default (requiring Synapse
+administrators to explicitly opt-in), although there are situations where it makes
+sense (from a product point-of-view) to enable features by default. This is
+expected and not an issue.
+
+It is not a requirement for experimental features to be behind a configuration flag,
+but one should be used if unsure.
+
+New experimental configuration flags should be added under the `experimental`
+configuration key (see the `synapse.config.experimental` file) and either explain
+(briefly) what is being enabled, or include the MSC number.
@@ -1,10 +1,9 @@
 # How to test SAML as a developer without a server

-https://capriza.github.io/samling/samling.html (https://github.com/capriza/samling) is a great
-resource for being able to tinker with the SAML options within Synapse without needing to
-deploy and configure a complicated software stack.
+https://fujifish.github.io/samling/samling.html (https://github.com/fujifish/samling) is a great resource for being able to tinker with the 
+SAML options within Synapse without needing to deploy and configure a complicated software stack.

-To make Synapse (and therefore Riot) use it:
+To make Synapse (and therefore Element) use it:

 1. Use the samling.html URL above or deploy your own and visit the IdP Metadata tab.
 2. Copy the XML to your clipboard.
@@ -26,9 +25,9 @@ To make Synapse (and therefore Riot) use it:
   the dependencies are installed and ready to go.
 7. Restart Synapse.

-Then in Riot:
+Then in Element:

-1. Visit the login page with a Riot pointing at your homeserver.
+1. Visit the login page and point Element towards your homeserver using the `public_baseurl` above.
 2. Click the Single Sign-On button.
 3. On the samling page, enter a Name Identifier and add a SAML Attribute for `uid=your_localpart`.
   The response must also be signed.
@@ -25,16 +25,14 @@ When Synapse is asked to preview a URL it does the following:
 3. Kicks off a background process to generate a preview:
   1. Checks the database cache by URL and timestamp and returns the result if it
      has not expired and was successful (a 2xx return code).
-   2. Checks if the URL matches an oEmbed pattern. If it does, fetch the oEmbed
-      response. If this is an image, replace the URL to fetch and continue. If
-      if it is HTML content, use the HTML as the document and continue.
-   3. If it doesn't match an oEmbed pattern, downloads the URL and stores it
-      into a file via the media storage provider and saves the local media
-      metadata.
-   5. If the media is an image:
+   2. Checks if the URL matches an [oEmbed](https://oembed.com/) pattern. If it
+      does, update the URL to download.
+   3. Downloads the URL and stores it into a file via the media storage provider
+      and saves the local media metadata.
+   4. If the media is an image:
      1. Generates thumbnails.
      2. Generates an Open Graph response based on image properties.
-   6. If the media is HTML:
+   5. If the media is HTML:
      1. Decodes the HTML via the stored file.
      2. Generates an Open Graph response from the HTML.
      3. If an image exists in the Open Graph response:
@@ -42,6 +40,13 @@ When Synapse is asked to preview a URL it does the following:
            provider and saves the local media metadata.
         2. Generates thumbnails.
         3. Updates the Open Graph response based on image properties.
+   6. If the media is JSON and an oEmbed URL was found:
+      1. Convert the oEmbed response to an Open Graph response.
+      2. If a thumbnail or image is in the oEmbed response:
+         1. Downloads the URL and stores it into a file via the media storage
+            provider and saves the local media metadata.
+         2. Generates thumbnails.
+         3. Updates the Open Graph response based on image properties.
   7. Stores the result in the database cache.
 4. Returns the result.

@@ -19,6 +19,21 @@ either a `bool` to indicate whether the event must be rejected because of spam,
 to indicate the event must be rejected because of spam and to give a rejection reason to
 forward to clients.

+### `user_may_join_room`
+
+```python
+async def user_may_join_room(user: str, room: str, is_invited: bool) -> bool
+```
+
+Called when a user is trying to join a room. The module must return a `bool` to indicate
+whether the user can join the room. The user is represented by their Matrix user ID (e.g.
+`@alice:example.com`) and the room is represented by its Matrix ID (e.g.
+`!room:example.com`). The module is also given a boolean to indicate whether the user
+currently has a pending invite in the room.
+
+This callback isn't called if the join is performed by a server administrator, or in the
+context of a room creation.
+
 ### `user_may_invite`

 ```python
@@ -29,6 +44,41 @@ Called when processing an invitation. The module must return a `bool` indicating
 the inviter can invite the invitee to the given room. Both inviter and invitee are
 represented by their Matrix user ID (e.g. `@alice:example.com`).

+### `user_may_send_3pid_invite`
+
+```python
+async def user_may_send_3pid_invite(
+    inviter: str,
+    medium: str,
+    address: str,
+    room_id: str,
+) -> bool
+```
+
+Called when processing an invitation using a third-party identifier (also called a 3PID,
+e.g. an email address or a phone number). The module must return a `bool` indicating
+whether the inviter can invite the invitee to the given room.
+
+The inviter is represented by their Matrix user ID (e.g. `@alice:example.com`), and the
+invitee is represented by its medium (e.g. "email") and its address
+(e.g. `alice@example.com`). See [the Matrix specification](https://matrix.org/docs/spec/appendices#pid-types)
+for more information regarding third-party identifiers.
+
+For example, a call to this callback to send an invitation to the email address
+`alice@example.com` would look like this:
+
+```python
+await user_may_send_3pid_invite(
+    "@bob:example.com",  # The inviter's user ID
+    "email",  # The medium of the 3PID to invite
+    "alice@example.com",  # The address of the 3PID to invite
+    "!some_room:example.com",  # The ID of the room to send the invite into
+)
+```
+
+**Note**: If the third-party identifier is already associated with a matrix user ID,
+[`user_may_invite`](#user_may_invite) will be used instead.
+
 ### `user_may_create_room`

 ```python
@@ -38,6 +88,35 @@ async def user_may_create_room(user: str) -> bool
 Called when processing a room creation request. The module must return a `bool` indicating
 whether the given user (represented by their Matrix user ID) is allowed to create a room.

+### `user_may_create_room_with_invites`
+
+```python
+async def user_may_create_room_with_invites(
+    user: str,
+    invites: List[str],
+    threepid_invites: List[Dict[str, str]],
+) -> bool
+```
+
+Called when processing a room creation request (right after `user_may_create_room`).
+The module is given the Matrix user ID of the user trying to create a room, as well as a
+list of Matrix users to invite and a list of third-party identifiers (3PID, e.g. email
+addresses) to invite.
+
+An invited Matrix user to invite is represented by their Matrix user IDs, and an invited
+3PIDs is represented by a dict that includes the 3PID medium (e.g. "email") through its
+`medium` key and its address (e.g. "alice@example.com") through its `address` key.
+
+See [the Matrix specification](https://matrix.org/docs/spec/appendices#pid-types) for more
+information regarding third-party identifiers.
+
+If no invite and/or 3PID invite were specified in the room creation request, the
+corresponding list(s) will be empty.
+
+**Note**: This callback is not called when a room is cloned (e.g. during a room upgrade)
+since no invites are sent when cloning a room. To cover this case, modules also need to
+implement `user_may_create_room`.
+
 ### `user_may_create_room_alias`

 ```python
@@ -136,9 +215,9 @@ class IsUserEvilResource(Resource):
        self.evil_users = config.get("evil_users") or []

    def render_GET(self, request: Request):
-        user = request.args.get(b"user")[0]
+        user = request.args.get(b"user")[0].decode()
        request.setHeader(b"Content-Type", b"application/json")
-        return json.dumps({"evil": user in self.evil_users})
+        return json.dumps({"evil": user in self.evil_users}).encode()


 class ListSpamChecker:
@@ -2362,12 +2362,16 @@ user_directory:
    #enabled: false

    # Defines whether to search all users visible to your HS when searching
-    # the user directory, rather than limiting to users visible in public
-    # rooms. Defaults to false.
+    # the user directory. If false, search results will only contain users
+    # visible in public rooms and users sharing a room with the requester.
+    # Defaults to false.
    #
-    # If you set it true, you'll have to rebuild the user_directory search
-    # indexes, see:
-    # https://matrix-org.github.io/synapse/latest/user_directory.html
+    # NB. If you set this to true, and the last time the user_directory search
+    # indexes were (re)built was before Synapse 1.44, you'll have to
+    # rebuild the indexes in order to search through all known users.
+    # These indexes are built the first time Synapse starts; admins can
+    # manually trigger a rebuild following the instructions at
+    #     https://matrix-org.github.io/synapse/latest/user_directory.html
    #
    # Uncomment to return search results containing all known users, even if that
    # user does not share a room with the requester.
@@ -18,19 +18,179 @@ that your email address is probably `user@example.com` rather than

 ## Installing Synapse

-### Installing from source
+### Prebuilt packages

-(Prebuilt packages are available for some platforms - see [Prebuilt packages](#prebuilt-packages).)
+Prebuilt packages are available for a number of platforms. These are recommended
+for most users.

-When installing from source please make sure that the [Platform-specific prerequisites](#platform-specific-prerequisites) are already installed.
+#### Docker images and Ansible playbooks
+
+There is an official synapse image available at
+<https://hub.docker.com/r/matrixdotorg/synapse> which can be used with
+the docker-compose file available at
+[contrib/docker](https://github.com/matrix-org/synapse/tree/develop/contrib/docker).
+Further information on this including configuration options is available in the README
+on hub.docker.com.
+
+Alternatively, Andreas Peters (previously Silvio Fricke) has contributed a
+Dockerfile to automate a synapse server in a single Docker image, at
+<https://hub.docker.com/r/avhost/docker-matrix/tags/>
+
+Slavi Pantaleev has created an Ansible playbook,
+which installs the offical Docker image of Matrix Synapse
+along with many other Matrix-related services (Postgres database, Element, coturn,
+ma1sd, SSL support, etc.).
+For more details, see
+<https://github.com/spantaleev/matrix-docker-ansible-deploy>
+
+#### Debian/Ubuntu
+
+##### Matrix.org packages
+
+Matrix.org provides Debian/Ubuntu packages of Synapse, for the amd64
+architecture via <https://packages.matrix.org/debian/>.  
+
+To install the latest release:
+
+```sh
+sudo apt install -y lsb-release wget apt-transport-https
+sudo wget -O /usr/share/keyrings/matrix-org-archive-keyring.gpg https://packages.matrix.org/debian/matrix-org-archive-keyring.gpg
+echo "deb [signed-by=/usr/share/keyrings/matrix-org-archive-keyring.gpg] https://packages.matrix.org/debian/ $(lsb_release -cs) main" |
+    sudo tee /etc/apt/sources.list.d/matrix-org.list
+sudo apt update
+sudo apt install matrix-synapse-py3
+```
+
+Packages are also published for release candidates. To enable the prerelease
+channel, add `prerelease` to the `sources.list` line. For example:
+
+```sh
+sudo wget -O /usr/share/keyrings/matrix-org-archive-keyring.gpg https://packages.matrix.org/debian/matrix-org-archive-keyring.gpg
+echo "deb [signed-by=/usr/share/keyrings/matrix-org-archive-keyring.gpg] https://packages.matrix.org/debian/ $(lsb_release -cs) main prerelease" |
+    sudo tee /etc/apt/sources.list.d/matrix-org.list
+sudo apt update
+sudo apt install matrix-synapse-py3
+```
+
+The fingerprint of the repository signing key (as shown by `gpg
+/usr/share/keyrings/matrix-org-archive-keyring.gpg`) is
+`AAF9AE843A7584B5A3E4CD2BCF45A512DE2DA058`.
+
+##### Downstream Debian packages
+
+We do not recommend using the packages from the default Debian `buster`
+repository at this time, as they are old and suffer from known security
+vulnerabilities. You can install the latest version of Synapse from
+[our repository](#matrixorg-packages) or from `buster-backports`. Please
+see the [Debian documentation](https://backports.debian.org/Instructions/)
+for information on how to use backports.
+
+If you are using Debian `sid` or testing, Synapse is available in the default
+repositories and it should be possible to install it simply with:
+
+```sh
+sudo apt install matrix-synapse
+```
+
+##### Downstream Ubuntu packages
+
+We do not recommend using the packages in the default Ubuntu repository
+at this time, as they are old and suffer from known security vulnerabilities.
+The latest version of Synapse can be installed from [our repository](#matrixorg-packages).
+
+#### Fedora
+
+Synapse is in the Fedora repositories as `matrix-synapse`:
+
+```sh
+sudo dnf install matrix-synapse
+```
+
+Oleg Girko provides Fedora RPMs at
+<https://obs.infoserver.lv/project/monitor/matrix-synapse>
+
+#### OpenSUSE
+
+Synapse is in the OpenSUSE repositories as `matrix-synapse`:
+
+```sh
+sudo zypper install matrix-synapse
+```
+
+#### SUSE Linux Enterprise Server
+
+Unofficial package are built for SLES 15 in the openSUSE:Backports:SLE-15 repository at
+<https://download.opensuse.org/repositories/openSUSE:/Backports:/SLE-15/standard/>
+
+#### ArchLinux
+
+The quickest way to get up and running with ArchLinux is probably with the community package
+<https://www.archlinux.org/packages/community/any/matrix-synapse/>, which should pull in most of
+the necessary dependencies.
+
+pip may be outdated (6.0.7-1 and needs to be upgraded to 6.0.8-1 ):
+
+```sh
+sudo pip install --upgrade pip
+```
+
+If you encounter an error with lib bcrypt causing an Wrong ELF Class:
+ELFCLASS32 (x64 Systems), you may need to reinstall py-bcrypt to correctly
+compile it under the right architecture. (This should not be needed if
+installing under virtualenv):
+
+```sh
+sudo pip uninstall py-bcrypt
+sudo pip install py-bcrypt
+```
+
+#### Void Linux
+
+Synapse can be found in the void repositories as 'synapse':
+
+```sh
+xbps-install -Su
+xbps-install -S synapse
+```
+
+#### FreeBSD
+
+Synapse can be installed via FreeBSD Ports or Packages contributed by Brendan Molloy from:
+
+- Ports: `cd /usr/ports/net-im/py-matrix-synapse && make install clean`
+- Packages: `pkg install py37-matrix-synapse`
+
+#### OpenBSD
+
+As of OpenBSD 6.7 Synapse is available as a pre-compiled binary. The filesystem
+underlying the homeserver directory (defaults to `/var/synapse`) has to be
+mounted with `wxallowed` (cf. `mount(8)`), so creating a separate filesystem
+and mounting it to `/var/synapse` should be taken into consideration.
+
+Installing Synapse:
+
+```sh
+doas pkg_add synapse
+```
+
+#### NixOS
+
+Robin Lambertz has packaged Synapse for NixOS at:
+<https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/misc/matrix-synapse.nix>
+
+
+### Installing as a Python module from PyPI
+
+It's also possible to install Synapse as a Python module from PyPI.
+
+When following this route please make sure that the [Platform-specific prerequisites](#platform-specific-prerequisites) are already installed.

 System requirements:

 - POSIX-compliant system (tested on Linux & OS X)
- Python 3.5.2 or later, up to Python 3.9.
+- Python 3.6 or later, up to Python 3.9.
 - At least 1GB of free RAM if you want to join large public rooms like #matrix:matrix.org

-
 To install the Synapse homeserver run:

 ```sh
@@ -203,164 +363,6 @@ be found at <https://docs.microsoft.com/en-us/windows/wsl/install-win10> for
 Windows 10 and <https://docs.microsoft.com/en-us/windows/wsl/install-on-server>
 for Windows Server.

-### Prebuilt packages
-
-As an alternative to installing from source, prebuilt packages are available
-for a number of platforms.
-
-#### Docker images and Ansible playbooks
-
-There is an official synapse image available at
-<https://hub.docker.com/r/matrixdotorg/synapse> which can be used with
-the docker-compose file available at
-[contrib/docker](https://github.com/matrix-org/synapse/tree/develop/contrib/docker).
-Further information on this including configuration options is available in the README
-on hub.docker.com.
-
-Alternatively, Andreas Peters (previously Silvio Fricke) has contributed a
-Dockerfile to automate a synapse server in a single Docker image, at
-<https://hub.docker.com/r/avhost/docker-matrix/tags/>
-
-Slavi Pantaleev has created an Ansible playbook,
-which installs the offical Docker image of Matrix Synapse
-along with many other Matrix-related services (Postgres database, Element, coturn,
-ma1sd, SSL support, etc.).
-For more details, see
-<https://github.com/spantaleev/matrix-docker-ansible-deploy>
-
-#### Debian/Ubuntu
-
-##### Matrix.org packages
-
-Matrix.org provides Debian/Ubuntu packages of Synapse via
-<https://packages.matrix.org/debian/>.  To install the latest release:
-
-```sh
-sudo apt install -y lsb-release wget apt-transport-https
-sudo wget -O /usr/share/keyrings/matrix-org-archive-keyring.gpg https://packages.matrix.org/debian/matrix-org-archive-keyring.gpg
-echo "deb [signed-by=/usr/share/keyrings/matrix-org-archive-keyring.gpg] https://packages.matrix.org/debian/ $(lsb_release -cs) main" |
-    sudo tee /etc/apt/sources.list.d/matrix-org.list
-sudo apt update
-sudo apt install matrix-synapse-py3
-```
-
-Packages are also published for release candidates. To enable the prerelease
-channel, add `prerelease` to the `sources.list` line. For example:
-
-```sh
-sudo wget -O /usr/share/keyrings/matrix-org-archive-keyring.gpg https://packages.matrix.org/debian/matrix-org-archive-keyring.gpg
-echo "deb [signed-by=/usr/share/keyrings/matrix-org-archive-keyring.gpg] https://packages.matrix.org/debian/ $(lsb_release -cs) main prerelease" |
-    sudo tee /etc/apt/sources.list.d/matrix-org.list
-sudo apt update
-sudo apt install matrix-synapse-py3
-```
-
-The fingerprint of the repository signing key (as shown by `gpg
-/usr/share/keyrings/matrix-org-archive-keyring.gpg`) is
-`AAF9AE843A7584B5A3E4CD2BCF45A512DE2DA058`.
-
-##### Downstream Debian packages
-
-We do not recommend using the packages from the default Debian `buster`
-repository at this time, as they are old and suffer from known security
-vulnerabilities. You can install the latest version of Synapse from
-[our repository](#matrixorg-packages) or from `buster-backports`. Please
-see the [Debian documentation](https://backports.debian.org/Instructions/)
-for information on how to use backports.
-
-If you are using Debian `sid` or testing, Synapse is available in the default
-repositories and it should be possible to install it simply with:
-
-```sh
-sudo apt install matrix-synapse
-```
-
-##### Downstream Ubuntu packages
-
-We do not recommend using the packages in the default Ubuntu repository
-at this time, as they are old and suffer from known security vulnerabilities.
-The latest version of Synapse can be installed from [our repository](#matrixorg-packages).
-
-#### Fedora
-
-Synapse is in the Fedora repositories as `matrix-synapse`:
-
-```sh
-sudo dnf install matrix-synapse
-```
-
-Oleg Girko provides Fedora RPMs at
-<https://obs.infoserver.lv/project/monitor/matrix-synapse>
-
-#### OpenSUSE
-
-Synapse is in the OpenSUSE repositories as `matrix-synapse`:
-
-```sh
-sudo zypper install matrix-synapse
-```
-
-#### SUSE Linux Enterprise Server
-
-Unofficial package are built for SLES 15 in the openSUSE:Backports:SLE-15 repository at
-<https://download.opensuse.org/repositories/openSUSE:/Backports:/SLE-15/standard/>
-
-#### ArchLinux
-
-The quickest way to get up and running with ArchLinux is probably with the community package
-<https://www.archlinux.org/packages/community/any/matrix-synapse/>, which should pull in most of
-the necessary dependencies.
-
-pip may be outdated (6.0.7-1 and needs to be upgraded to 6.0.8-1 ):
-
-```sh
-sudo pip install --upgrade pip
-```
-
-If you encounter an error with lib bcrypt causing an Wrong ELF Class:
-ELFCLASS32 (x64 Systems), you may need to reinstall py-bcrypt to correctly
-compile it under the right architecture. (This should not be needed if
-installing under virtualenv):
-
-```sh
-sudo pip uninstall py-bcrypt
-sudo pip install py-bcrypt
-```
-
-#### Void Linux
-
-Synapse can be found in the void repositories as 'synapse':
-
-```sh
-xbps-install -Su
-xbps-install -S synapse
-```
-
-#### FreeBSD
-
-Synapse can be installed via FreeBSD Ports or Packages contributed by Brendan Molloy from:
-
- Ports: `cd /usr/ports/net-im/py-matrix-synapse && make install clean`
- Packages: `pkg install py37-matrix-synapse`
-
-#### OpenBSD
-
-As of OpenBSD 6.7 Synapse is available as a pre-compiled binary. The filesystem
-underlying the homeserver directory (defaults to `/var/synapse`) has to be
-mounted with `wxallowed` (cf. `mount(8)`), so creating a separate filesystem
-and mounting it to `/var/synapse` should be taken into consideration.
-
-Installing Synapse:
-
-```sh
-doas pkg_add synapse
-```
-
-#### NixOS
-
-Robin Lambertz has packaged Synapse for NixOS at:
-<https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/misc/matrix-synapse.nix>
-
 ## Setting up Synapse

 Once you have installed synapse as above, you will need to configure it.
@@ -85,6 +85,13 @@ process, for example:
    dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
    ```

+# Upgrading to v1.44.0
+
+## The URL preview cache is no longer mirrored to storage providers
+The `url_cache/` and `url_cache_thumbnails/` directories in the media store are
+no longer mirrored to storage providers. These two directories can be safely
+deleted from any configured storage providers to reclaim space.
+
 # Upgrading to v1.43.0

 ## The spaces summary APIs can now be handled by workers
@@ -1,7 +1,8 @@
 # Registration Tokens

 This API allows you to manage tokens which can be used to authenticate
-registration requests, as proposed in [MSC3231](https://github.com/govynnus/matrix-doc/blob/token-registration/proposals/3231-token-authenticated-registration.md).
+registration requests, as proposed in
+[MSC3231](https://github.com/matrix-org/matrix-doc/blob/main/proposals/3231-token-authenticated-registration.md).
 To use it, you will need to enable the `registration_requires_token` config
 option, and authenticate by providing an `access_token` for a server admin:
 see [Admin API](../../usage/administration/admin_api).
@@ -1,4 +1,79 @@
 # Introduction

-Welcome to the documentation repository for Synapse, the reference
-[Matrix](https://matrix.org) homeserver implementation.
+Welcome to the documentation repository for Synapse, a 
+[Matrix](https://matrix.org) homeserver implementation developed by the matrix.org core 
+team.
+
+## Installing and using Synapse
+
+This documentation covers topics for **installation**, **configuration** and
+**maintainence** of your Synapse process:
+
+* Learn how to [install](setup/installation.md) and
+  [configure](usage/configuration/index.html) your own instance, perhaps with [Single
+  Sign-On](usage/configuration/user_authentication/index.html).
+
+* See how to [upgrade](upgrade.md) between Synapse versions.
+
+* Administer your instance using the [Admin
+  API](usage/administration/admin_api/index.html), installing [pluggable
+  modules](modules/index.html), or by accessing the [manhole](manhole.md).
+
+* Learn how to [read log lines](usage/administration/request_log.md), configure
+  [logging](usage/configuration/logging_sample_config.md) or set up [structured
+  logging](structured_logging.md).
+
+* Scale Synapse through additional [worker processes](workers.md).
+
+* Set up [monitoring and metrics](metrics-howto.md) to keep an eye on your
+  Synapse instance's performance.
+
+## Developing on Synapse
+
+Contributions are welcome! Synapse is primarily written in
+[Python](https://python.org). As a developer, you may be interested in the
+following documentation:
+
+* Read the [Contributing Guide](development/contributing_guide.md). It is meant
+  to walk new contributors through the process of developing and submitting a
+  change to the Synapse codebase (which is [hosted on
+  GitHub](https://github.com/matrix-org/synapse)).
+
+* Set up your [development
+  environment](development/contributing_guide.md#2-what-do-i-need), then learn
+  how to [lint](development/contributing_guide.md#run-the-linters) and
+  [test](development/contributing_guide.md#8-test-test-test) your code.
+
+* Look at [the issue tracker](https://github.com/matrix-org/synapse/issues) for
+  bugs to fix or features to add. If you're new, it may be best to start with
+  those labeled [good first
+  issue](https://github.com/matrix-org/synapse/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
+
+* Understand [how Synapse is
+  built](development/internal_documentation/index.html), how to [migrate
+  database schemas](development/database_schema.md), learn about
+  [federation](federate.md) and how to [set up a local
+  federation](federate.md#running-a-demo-federation-of-synapses) for development.
+
+* We like to keep our `git` history clean. [Learn](development/git.md) how to
+  do so!
+
+* And finally, contribute to this documentation! The source for which is
+  [located here](https://github.com/matrix-org/synapse/tree/develop/docs).
+
+## Donating to Synapse development
+
+Want to help keep Synapse going but don't know how to code? Synapse is a
+[Matrix.org Foundation](https://matrix.org) project. Consider becoming a
+supportor on [Liberapay](https://liberapay.com/matrixdotorg),
+[Patreon](https://patreon.com/matrixdotorg) or through
+[PayPal](https://paypal.me/matrixdotorg) via a one-time donation.
+
+If you are an organisation or enterprise and would like to sponsor development,
+reach out to us over email at: support (at) matrix.org
+
+## Reporting a security vulnerability
+
+If you've found a security issue in Synapse or any other Matrix.org Foundation
+project, please report it to us in accordance with our [Security Disclosure
+Policy](https://www.matrix.org/security-disclosure-policy/). Thank you!
@@ -60,6 +60,7 @@ files =
  synapse/storage/databases/main/session.py,
  synapse/storage/databases/main/stream.py,
  synapse/storage/databases/main/ui_auth.py,
+  synapse/storage/databases/state,
  synapse/storage/database.py,
  synapse/storage/engines,
  synapse/storage/keys.py,
@@ -84,20 +85,50 @@ files =
  tests/handlers/test_room_summary.py,
  tests/handlers/test_send_email.py,
  tests/handlers/test_sync.py,
+  tests/handlers/test_user_directory.py,
  tests/rest/client/test_login.py,
  tests/rest/client/test_auth.py,
+  tests/storage/test_state.py,
+  tests/storage/test_user_directory.py,
  tests/util/test_itertools.py,
  tests/util/test_stream_change_cache.py

-[mypy-synapse.rest.client.*]
+[mypy-synapse.handlers.*]
+disallow_untyped_defs = True
+
+[mypy-synapse.rest.*]
+disallow_untyped_defs = True
+
+[mypy-synapse.replication.http._base]
+disallow_untyped_defs = True
+
+[mypy-synapse.state.*]
 disallow_untyped_defs = True

 [mypy-synapse.util.batching_queue]
 disallow_untyped_defs = True

+[mypy-synapse.util.caches.cached_call]
+disallow_untyped_defs = True
+
 [mypy-synapse.util.caches.dictionary_cache]
 disallow_untyped_defs = True

+[mypy-synapse.util.caches.lrucache]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.caches.response_cache]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.caches.stream_change_cache]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.caches.ttl_cache]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.daemonize]
+disallow_untyped_defs = True
+
 [mypy-synapse.util.file_consumer]
 disallow_untyped_defs = True

@@ -134,6 +165,9 @@ disallow_untyped_defs = True
 [mypy-synapse.util.msisdn]
 disallow_untyped_defs = True

+[mypy-synapse.util.patch_inline_callbacks]
+disallow_untyped_defs = True
+
 [mypy-synapse.util.ratelimitutils]
 disallow_untyped_defs = True

@@ -155,98 +189,106 @@ disallow_untyped_defs = True
 [mypy-synapse.util.wheel_timer]
 disallow_untyped_defs = True

-[mypy-pymacaroons.*]
-ignore_missing_imports = True
+[mypy-synapse.util.versionstring]
+disallow_untyped_defs = True

-[mypy-zope]
+[mypy-tests.handlers.test_user_directory]
+disallow_untyped_defs = True
+
+[mypy-tests.storage.test_user_directory]
+disallow_untyped_defs = True
+
+;; Dependencies without annotations
+;; Before ignoring a module, check to see if type stubs are available.
+;; The `typeshed` project maintains stubs here:
+;;     https://github.com/python/typeshed/tree/master/stubs
+;; and for each package `foo` there's a corresponding `types-foo` package on PyPI,
+;; which we can pull in as a dev dependency by adding to `setup.py`'s
+;; `CONDITIONAL_REQUIREMENTS["mypy"]` list.
+
+[mypy-authlib.*]
 ignore_missing_imports = True

 [mypy-bcrypt]
 ignore_missing_imports = True

-[mypy-constantly]
-ignore_missing_imports = True
-
-[mypy-twisted.*]
-ignore_missing_imports = True
-
-[mypy-treq.*]
-ignore_missing_imports = True
-
-[mypy-hyperlink]
-ignore_missing_imports = True
-
-[mypy-h11]
-ignore_missing_imports = True
-
-[mypy-msgpack]
-ignore_missing_imports = True
-
-[mypy-opentracing]
-ignore_missing_imports = True
-
-[mypy-OpenSSL.*]
-ignore_missing_imports = True
-
-[mypy-netaddr]
-ignore_missing_imports = True
-
-[mypy-saml2.*]
-ignore_missing_imports = True
-
 [mypy-canonicaljson]
 ignore_missing_imports = True

-[mypy-jaeger_client.*]
-ignore_missing_imports = True
-
-[mypy-jsonschema]
-ignore_missing_imports = True
-
-[mypy-signedjson.*]
-ignore_missing_imports = True
-
-[mypy-prometheus_client.*]
-ignore_missing_imports = True
-
-[mypy-service_identity.*]
+[mypy-constantly]
 ignore_missing_imports = True

 [mypy-daemonize]
 ignore_missing_imports = True

-[mypy-sentry_sdk]
-ignore_missing_imports = True
-
-[mypy-PIL.*]
-ignore_missing_imports = True
-
-[mypy-lxml]
-ignore_missing_imports = True
-
-[mypy-jwt.*]
-ignore_missing_imports = True
-
-[mypy-authlib.*]
-ignore_missing_imports = True
-
-[mypy-rust_python_jaeger_reporter.*]
-ignore_missing_imports = True
-
-[mypy-nacl.*]
+[mypy-h11]
 ignore_missing_imports = True

 [mypy-hiredis]
 ignore_missing_imports = True

+[mypy-hyperlink]
+ignore_missing_imports = True
+
+[mypy-ijson.*]
+ignore_missing_imports = True
+
+[mypy-jaeger_client.*]
+ignore_missing_imports = True
+
 [mypy-josepy.*]
 ignore_missing_imports = True

-[mypy-pympler.*]
+[mypy-jwt.*]
+ignore_missing_imports = True
+
+[mypy-lxml]
+ignore_missing_imports = True
+
+[mypy-msgpack]
+ignore_missing_imports = True
+
+[mypy-nacl.*]
+ignore_missing_imports = True
+
+[mypy-netaddr]
+ignore_missing_imports = True
+
+[mypy-opentracing]
 ignore_missing_imports = True

 [mypy-phonenumbers.*]
 ignore_missing_imports = True

-[mypy-ijson.*]
+[mypy-prometheus_client.*]
+ignore_missing_imports = True
+
+[mypy-pymacaroons.*]
+ignore_missing_imports = True
+
+[mypy-pympler.*]
+ignore_missing_imports = True
+
+[mypy-rust_python_jaeger_reporter.*]
+ignore_missing_imports = True
+
+[mypy-saml2.*]
+ignore_missing_imports = True
+
+[mypy-sentry_sdk]
+ignore_missing_imports = True
+
+[mypy-service_identity.*]
+ignore_missing_imports = True
+
+[mypy-signedjson.*]
+ignore_missing_imports = True
+
+[mypy-treq.*]
+ignore_missing_imports = True
+
+[mypy-twisted.*]
+ignore_missing_imports = True
+
+[mypy-zope]
 ignore_missing_imports = True
@@ -90,10 +90,10 @@ else
          "scripts/hash_password"
          "scripts/register_new_matrix_user"
          "scripts/synapse_port_db"
+          "scripts/update_synapse_database"
          "scripts-dev"
          "scripts-dev/build_debian_packages"
          "scripts-dev/sign_json"
-          "scripts-dev/update_database"
          "contrib" "synctl" "setup.py" "synmark" "stubs" ".ci"
      )
  fi
@@ -147,7 +147,7 @@ python -m synapse.app.homeserver --generate-keys -c "$SQLITE_CONFIG"

 # Make sure the SQLite3 database is using the latest schema and has no pending background update.
 echo "Running db background jobs..."
-scripts-dev/update_database --database-config "$SQLITE_CONFIG"
+scripts/update_synapse_database --database-config --run-background-updates "$SQLITE_CONFIG"

 # Create the PostgreSQL database.
 echo "Creating postgres database..."
@@ -276,7 +276,7 @@ def tag(gh_token: Optional[str]):
    if click.confirm("Edit text?", default=False):
        changes = click.edit(changes, require_save=False)

-    repo.create_tag(tag_name, message=changes)
+    repo.create_tag(tag_name, message=changes, sign=True)

    if not click.confirm("Push tag to GitHub?", default=True):
        print("")
@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+
+# This script builds the Docker image to run the PostgreSQL tests, and then runs
+# the tests. It uses a dedicated tox environment so that we don't have to
+# rebuild it each time.
+
+# Command line arguments to this script are forwarded to "tox" and then to "trial".
+
+set -e
+
+# Build, and tag
+docker build docker/ \
+  --build-arg "UID=$(id -u)" \
+  --build-arg "GID=$(id -g)" \
+  -f docker/Dockerfile-pgtests \
+  -t synapsepgtests
+
+# Run, mounting the current directory into /src
+docker run --rm -it -v "$(pwd):/src" -v synapse-pg-test-tox:/tox synapsepgtests "$@"
@@ -215,7 +215,7 @@ class MockHomeserver:
    def __init__(self, config):
        self.clock = Clock(reactor)
        self.config = config
-        self.hostname = config.server_name
+        self.hostname = config.server.server_name
        self.version_string = "Synapse/" + get_version_string(synapse)

    def get_clock(self):
@@ -583,7 +583,7 @@ class Porter(object):
                return

            self.postgres_store = self.build_db_store(
-                self.hs_config.get_single_database()
+                self.hs_config.database.get_single_database()
            )

            await self.run_background_updates_on_postgres()
@@ -36,16 +36,35 @@ class MockHomeserver(HomeServer):

    def __init__(self, config, **kwargs):
        super(MockHomeserver, self).__init__(
-            config.server_name, reactor=reactor, config=config, **kwargs
+            config.server.server_name, reactor=reactor, config=config, **kwargs
        )

        self.version_string = "Synapse/" + get_version_string(synapse)


-if __name__ == "__main__":
+def run_background_updates(hs):
+    store = hs.get_datastore()
+
+    async def run_background_updates():
+        await store.db_pool.updates.run_background_updates(sleep=False)
+        # Stop the reactor to exit the script once every background update is run.
+        reactor.stop()
+
+    def run():
+        # Apply all background updates on the database.
+        defer.ensureDeferred(
+            run_as_background_process("background_updates", run_background_updates)
+        )
+
+    reactor.callWhenRunning(run)
+
+    reactor.run()
+
+
+def main():
    parser = argparse.ArgumentParser(
        description=(
-            "Updates a synapse database to the latest schema and runs background updates"
+            "Updates a synapse database to the latest schema and optionally runs background updates"
            " on it."
        )
    )
@@ -54,7 +73,13 @@ if __name__ == "__main__":
        "--database-config",
        type=argparse.FileType("r"),
        required=True,
-        help="A database config file for either a SQLite3 database or a PostgreSQL one.",
+        help="Synapse configuration file, giving the details of the database to be updated",
+    )
+    parser.add_argument(
+        "--run-background-updates",
+        action="store_true",
+        required=False,
+        help="run background updates after upgrading the database schema",
    )

    args = parser.parse_args()
@@ -82,19 +107,10 @@ if __name__ == "__main__":
    # Setup instantiates the store within the homeserver object and updates the
    # DB.
    hs.setup()
-    store = hs.get_datastore()

-    async def run_background_updates():
-        await store.db_pool.updates.run_background_updates(sleep=False)
-        # Stop the reactor to exit the script once every background update is run.
-        reactor.stop()
+    if args.run_background_updates:
+        run_background_updates(hs)

-    def run():
-        # Apply all background updates on the database.
-        defer.ensureDeferred(
-            run_as_background_process("background_updates", run_background_updates)
-        )

-    reactor.callWhenRunning(run)
-
-    reactor.run()
+if __name__ == "__main__":
+    main()
@@ -112,7 +112,16 @@ CONDITIONAL_REQUIREMENTS["dev"] = CONDITIONAL_REQUIREMENTS["lint"] + [
    "pygithub==1.55",
 ]

-CONDITIONAL_REQUIREMENTS["mypy"] = ["mypy==0.812", "mypy-zope==0.2.13"]
+CONDITIONAL_REQUIREMENTS["mypy"] = [
+    "mypy==0.910",
+    "mypy-zope==0.3.2",
+    "types-bleach>=4.1.0",
+    "types-jsonschema>=3.2.0",
+    "types-Pillow>=8.3.4",
+    "types-pyOpenSSL>=20.0.7",
+    "types-PyYAML>=5.4.10",
+    "types-setuptools>=57.4.0",
+]

 # Dependencies which are exclusively required by unit test code. This is
 # NOT a list of all modules that are necessary to run the unit tests.
@@ -47,7 +47,7 @@ try:
 except ImportError:
    pass

-__version__ = "1.43.0rc1"
+__version__ = "1.44.0"

 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
    # We import here so that we don't have to install a bunch of deps when
@@ -70,8 +70,8 @@ class Auth:

        self._auth_blocking = AuthBlocking(self.hs)

-        self._track_appservice_user_ips = hs.config.track_appservice_user_ips
-        self._macaroon_secret_key = hs.config.macaroon_secret_key
+        self._track_appservice_user_ips = hs.config.appservice.track_appservice_user_ips
+        self._macaroon_secret_key = hs.config.key.macaroon_secret_key
        self._force_tracing_for_users = hs.config.tracing.force_tracing_for_users

    async def check_user_in_room(
@@ -30,13 +30,15 @@ class AuthBlocking:
    def __init__(self, hs: "HomeServer"):
        self.store = hs.get_datastore()

-        self._server_notices_mxid = hs.config.server_notices_mxid
-        self._hs_disabled = hs.config.hs_disabled
-        self._hs_disabled_message = hs.config.hs_disabled_message
-        self._admin_contact = hs.config.admin_contact
-        self._max_mau_value = hs.config.max_mau_value
-        self._limit_usage_by_mau = hs.config.limit_usage_by_mau
-        self._mau_limits_reserved_threepids = hs.config.mau_limits_reserved_threepids
+        self._server_notices_mxid = hs.config.servernotices.server_notices_mxid
+        self._hs_disabled = hs.config.server.hs_disabled
+        self._hs_disabled_message = hs.config.server.hs_disabled_message
+        self._admin_contact = hs.config.server.admin_contact
+        self._max_mau_value = hs.config.server.max_mau_value
+        self._limit_usage_by_mau = hs.config.server.limit_usage_by_mau
+        self._mau_limits_reserved_threepids = (
+            hs.config.server.mau_limits_reserved_threepids
+        )
        self._server_name = hs.hostname
        self._track_appservice_user_ips = hs.config.appservice.track_appservice_user_ips

@@ -79,7 +81,7 @@ class AuthBlocking:
                # We never block the server from doing actions on behalf of
                # users.
                return
-            elif requester.app_service and not self._track_appservice_user_ips:
+            if requester.app_service and not self._track_appservice_user_ips:
                # If we're authenticated as an appservice then we only block
                # auth if `track_appservice_user_ips` is set, as that option
                # implicitly means that application services are part of MAU
@@ -121,7 +121,7 @@ class EventTypes:
    SpaceParent = "m.space.parent"

    MSC2716_INSERTION = "org.matrix.msc2716.insertion"
-    MSC2716_CHUNK = "org.matrix.msc2716.chunk"
+    MSC2716_BATCH = "org.matrix.msc2716.batch"
    MSC2716_MARKER = "org.matrix.msc2716.marker"


@@ -209,14 +209,17 @@ class EventContentFields:

    # Used on normal messages to indicate they were historically imported after the fact
    MSC2716_HISTORICAL = "org.matrix.msc2716.historical"
-    # For "insertion" events to indicate what the next chunk ID should be in
+    # For "insertion" events to indicate what the next batch ID should be in
    # order to connect to it
-    MSC2716_NEXT_CHUNK_ID = "org.matrix.msc2716.next_chunk_id"
-    # Used on "chunk" events to indicate which insertion event it connects to
-    MSC2716_CHUNK_ID = "org.matrix.msc2716.chunk_id"
+    MSC2716_NEXT_BATCH_ID = "org.matrix.msc2716.next_batch_id"
+    # Used on "batch" events to indicate which insertion event it connects to
+    MSC2716_BATCH_ID = "org.matrix.msc2716.batch_id"
    # For "marker" events
    MSC2716_MARKER_INSERTION = "org.matrix.msc2716.marker.insertion"

+    # The authorising user for joining a restricted room.
+    AUTHORISING_USER = "join_authorised_via_users_server"
+

 class RoomTypes:
    """Understood values of the room_type field of m.room.create events."""
@@ -15,7 +15,17 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import json
-from typing import List
+from typing import (
+    TYPE_CHECKING,
+    Awaitable,
+    Container,
+    Iterable,
+    List,
+    Optional,
+    Set,
+    TypeVar,
+    Union,
+)

 import jsonschema
 from jsonschema import FormatChecker
@@ -23,7 +33,11 @@ from jsonschema import FormatChecker
 from synapse.api.constants import EventContentFields
 from synapse.api.errors import SynapseError
 from synapse.api.presence import UserPresenceState
-from synapse.types import RoomID, UserID
+from synapse.events import EventBase
+from synapse.types import JsonDict, RoomID, UserID
+
+if TYPE_CHECKING:
+    from synapse.server import HomeServer

 FILTER_SCHEMA = {
    "additionalProperties": False,
@@ -120,25 +134,29 @@ USER_FILTER_SCHEMA = {


@FormatChecker.cls_checks("matrix_room_id")
-def matrix_room_id_validator(room_id_str):
+def matrix_room_id_validator(room_id_str: str) -> RoomID:
    return RoomID.from_string(room_id_str)


@FormatChecker.cls_checks("matrix_user_id")
-def matrix_user_id_validator(user_id_str):
+def matrix_user_id_validator(user_id_str: str) -> UserID:
    return UserID.from_string(user_id_str)


 class Filtering:
-    def __init__(self, hs):
+    def __init__(self, hs: "HomeServer"):
        super().__init__()
        self.store = hs.get_datastore()

-    async def get_user_filter(self, user_localpart, filter_id):
+    async def get_user_filter(
+        self, user_localpart: str, filter_id: Union[int, str]
+    ) -> "FilterCollection":
        result = await self.store.get_user_filter(user_localpart, filter_id)
        return FilterCollection(result)

-    def add_user_filter(self, user_localpart, user_filter):
+    def add_user_filter(
+        self, user_localpart: str, user_filter: JsonDict
+    ) -> Awaitable[int]:
        self.check_valid_filter(user_filter)
        return self.store.add_user_filter(user_localpart, user_filter)

@@ -146,13 +164,13 @@ class Filtering:
    #   replace_user_filter at some point? There's no REST API specified for
    #   them however

-    def check_valid_filter(self, user_filter_json):
+    def check_valid_filter(self, user_filter_json: JsonDict) -> None:
        """Check if the provided filter is valid.

        This inspects all definitions contained within the filter.

        Args:
-            user_filter_json(dict): The filter
+            user_filter_json: The filter
        Raises:
            SynapseError: If the filter is not valid.
        """
@@ -167,8 +185,12 @@ class Filtering:
            raise SynapseError(400, str(e))


+# Filters work across events, presence EDUs, and account data.
+FilterEvent = TypeVar("FilterEvent", EventBase, UserPresenceState, JsonDict)
+
+
 class FilterCollection:
-    def __init__(self, filter_json):
+    def __init__(self, filter_json: JsonDict):
        self._filter_json = filter_json

        room_filter_json = self._filter_json.get("room", {})
@@ -188,25 +210,25 @@ class FilterCollection:
        self.event_fields = filter_json.get("event_fields", [])
        self.event_format = filter_json.get("event_format", "client")

-    def __repr__(self):
+    def __repr__(self) -> str:
        return "<FilterCollection %s>" % (json.dumps(self._filter_json),)

-    def get_filter_json(self):
+    def get_filter_json(self) -> JsonDict:
        return self._filter_json

-    def timeline_limit(self):
+    def timeline_limit(self) -> int:
        return self._room_timeline_filter.limit()

-    def presence_limit(self):
+    def presence_limit(self) -> int:
        return self._presence_filter.limit()

-    def ephemeral_limit(self):
+    def ephemeral_limit(self) -> int:
        return self._room_ephemeral_filter.limit()

-    def lazy_load_members(self):
+    def lazy_load_members(self) -> bool:
        return self._room_state_filter.lazy_load_members()

-    def include_redundant_members(self):
+    def include_redundant_members(self) -> bool:
        return self._room_state_filter.include_redundant_members()

    def filter_presence(self, events):
@@ -218,29 +240,31 @@ class FilterCollection:
    def filter_room_state(self, events):
        return self._room_state_filter.filter(self._room_filter.filter(events))

-    def filter_room_timeline(self, events):
+    def filter_room_timeline(self, events: Iterable[FilterEvent]) -> List[FilterEvent]:
        return self._room_timeline_filter.filter(self._room_filter.filter(events))

-    def filter_room_ephemeral(self, events):
+    def filter_room_ephemeral(self, events: Iterable[FilterEvent]) -> List[FilterEvent]:
        return self._room_ephemeral_filter.filter(self._room_filter.filter(events))

-    def filter_room_account_data(self, events):
+    def filter_room_account_data(
+        self, events: Iterable[FilterEvent]
+    ) -> List[FilterEvent]:
        return self._room_account_data.filter(self._room_filter.filter(events))

-    def blocks_all_presence(self):
+    def blocks_all_presence(self) -> bool:
        return (
            self._presence_filter.filters_all_types()
            or self._presence_filter.filters_all_senders()
        )

-    def blocks_all_room_ephemeral(self):
+    def blocks_all_room_ephemeral(self) -> bool:
        return (
            self._room_ephemeral_filter.filters_all_types()
            or self._room_ephemeral_filter.filters_all_senders()
            or self._room_ephemeral_filter.filters_all_rooms()
        )

-    def blocks_all_room_timeline(self):
+    def blocks_all_room_timeline(self) -> bool:
        return (
            self._room_timeline_filter.filters_all_types()
            or self._room_timeline_filter.filters_all_senders()
@@ -249,7 +273,7 @@ class FilterCollection:


 class Filter:
-    def __init__(self, filter_json):
+    def __init__(self, filter_json: JsonDict):
        self.filter_json = filter_json

        self.types = self.filter_json.get("types", None)
@@ -266,20 +290,20 @@ class Filter:
        self.labels = self.filter_json.get("org.matrix.labels", None)
        self.not_labels = self.filter_json.get("org.matrix.not_labels", [])

-    def filters_all_types(self):
+    def filters_all_types(self) -> bool:
        return "*" in self.not_types

-    def filters_all_senders(self):
+    def filters_all_senders(self) -> bool:
        return "*" in self.not_senders

-    def filters_all_rooms(self):
+    def filters_all_rooms(self) -> bool:
        return "*" in self.not_rooms

-    def check(self, event):
+    def check(self, event: FilterEvent) -> bool:
        """Checks whether the filter matches the given event.

        Returns:
-            bool: True if the event matches
+            True if the event matches
        """
        # We usually get the full "events" as dictionaries coming through,
        # except for presence which actually gets passed around as its own
@@ -305,18 +329,25 @@ class Filter:
            room_id = event.get("room_id", None)
            ev_type = event.get("type", None)

-            content = event.get("content", {})
+            content = event.get("content") or {}
            # check if there is a string url field in the content for filtering purposes
            contains_url = isinstance(content.get("url"), str)
            labels = content.get(EventContentFields.LABELS, [])

        return self.check_fields(room_id, sender, ev_type, labels, contains_url)

-    def check_fields(self, room_id, sender, event_type, labels, contains_url):
+    def check_fields(
+        self,
+        room_id: Optional[str],
+        sender: Optional[str],
+        event_type: Optional[str],
+        labels: Container[str],
+        contains_url: bool,
+    ) -> bool:
        """Checks whether the filter matches the given event fields.

        Returns:
-            bool: True if the event fields match
+            True if the event fields match
        """
        literal_keys = {
            "rooms": lambda v: room_id == v,
@@ -343,14 +374,14 @@ class Filter:

        return True

-    def filter_rooms(self, room_ids):
+    def filter_rooms(self, room_ids: Iterable[str]) -> Set[str]:
        """Apply the 'rooms' filter to a given list of rooms.

        Args:
-            room_ids (list): A list of room_ids.
+            room_ids: A list of room_ids.

        Returns:
-            list: A list of room_ids that match the filter
+            A list of room_ids that match the filter
        """
        room_ids = set(room_ids)

@@ -363,23 +394,23 @@ class Filter:

        return room_ids

-    def filter(self, events):
+    def filter(self, events: Iterable[FilterEvent]) -> List[FilterEvent]:
        return list(filter(self.check, events))

-    def limit(self):
+    def limit(self) -> int:
        return self.filter_json.get("limit", 10)

-    def lazy_load_members(self):
+    def lazy_load_members(self) -> bool:
        return self.filter_json.get("lazy_load_members", False)

-    def include_redundant_members(self):
+    def include_redundant_members(self) -> bool:
        return self.filter_json.get("include_redundant_members", False)

-    def with_room_ids(self, room_ids):
+    def with_room_ids(self, room_ids: Iterable[str]) -> "Filter":
        """Returns a new filter with the given room IDs appended.

        Args:
-            room_ids (iterable[unicode]): The room_ids to add
+            room_ids: The room_ids to add

        Returns:
            filter: A new filter including the given rooms and the old
@@ -390,8 +421,8 @@ class Filter:
        return newFilter


-def _matches_wildcard(actual_value, filter_value):
-    if filter_value.endswith("*"):
+def _matches_wildcard(actual_value: Optional[str], filter_value: str) -> bool:
+    if filter_value.endswith("*") and isinstance(actual_value, str):
        type_prefix = filter_value[:-1]
        return actual_value.startswith(type_prefix)
    else:
@@ -244,24 +244,8 @@ class RoomVersions:
        msc2716_historical=False,
        msc2716_redactions=False,
    )
-    MSC2716 = RoomVersion(
-        "org.matrix.msc2716",
-        RoomDisposition.UNSTABLE,
-        EventFormatVersions.V3,
-        StateResolutionVersions.V2,
-        enforce_key_validity=True,
-        special_case_aliases_auth=False,
-        strict_canonicaljson=True,
-        limit_notifications_power_levels=True,
-        msc2176_redaction_rules=False,
-        msc3083_join_rules=False,
-        msc3375_redaction_rules=False,
-        msc2403_knocking=True,
-        msc2716_historical=True,
-        msc2716_redactions=False,
-    )
-    MSC2716v2 = RoomVersion(
-        "org.matrix.msc2716v2",
+    MSC2716v3 = RoomVersion(
+        "org.matrix.msc2716v3",
        RoomDisposition.UNSTABLE,
        EventFormatVersions.V3,
        StateResolutionVersions.V2,
@@ -289,9 +273,9 @@ KNOWN_ROOM_VERSIONS: Dict[str, RoomVersion] = {
        RoomVersions.V6,
        RoomVersions.MSC2176,
        RoomVersions.V7,
-        RoomVersions.MSC2716,
        RoomVersions.V8,
        RoomVersions.V9,
+        RoomVersions.MSC2716v3,
    )
 }

@@ -39,12 +39,12 @@ class ConsentURIBuilder:
        Args:
            hs_config (synapse.config.homeserver.HomeServerConfig):
        """
-        if hs_config.form_secret is None:
+        if hs_config.key.form_secret is None:
            raise ConfigError("form_secret not set in config")
        if hs_config.server.public_baseurl is None:
            raise ConfigError("public_baseurl not set in config")

-        self._hmac_secret = hs_config.form_secret.encode("utf-8")
+        self._hmac_secret = hs_config.key.form_secret.encode("utf-8")
        self._public_baseurl = hs_config.server.public_baseurl

    def build_user_consent_uri(self, user_id):
@@ -86,11 +86,11 @@ def start_worker_reactor(appname, config, run_command=reactor.run):

    start_reactor(
        appname,
-        soft_file_limit=config.soft_file_limit,
-        gc_thresholds=config.gc_thresholds,
-        pid_file=config.worker_pid_file,
-        daemonize=config.worker_daemonize,
-        print_pidfile=config.print_pidfile,
+        soft_file_limit=config.server.soft_file_limit,
+        gc_thresholds=config.server.gc_thresholds,
+        pid_file=config.worker.worker_pid_file,
+        daemonize=config.worker.worker_daemonize,
+        print_pidfile=config.server.print_pidfile,
        logger=logger,
        run_command=run_command,
    )
@@ -298,10 +298,10 @@ def refresh_certificate(hs):
    Refresh the TLS certificates that Synapse is using by re-reading them from
    disk and updating the TLS context factories to use them.
    """
-    if not hs.config.has_tls_listener():
+    if not hs.config.server.has_tls_listener():
        return

-    hs.config.read_certificate_from_disk()
+    hs.config.tls.read_certificate_from_disk()
    hs.tls_server_context_factory = context_factory.ServerContextFactory(hs.config)

    if hs._listening_services:
@@ -424,12 +424,14 @@ def setup_sentry(hs):
        hs (synapse.server.HomeServer)
    """

-    if not hs.config.sentry_enabled:
+    if not hs.config.metrics.sentry_enabled:
        return

    import sentry_sdk

-    sentry_sdk.init(dsn=hs.config.sentry_dsn, release=get_version_string(synapse))
+    sentry_sdk.init(
+        dsn=hs.config.metrics.sentry_dsn, release=get_version_string(synapse)
+    )

    # We set some default tags that give some context to this instance
    with sentry_sdk.configure_scope() as scope:
@@ -186,23 +186,23 @@ def start(config_options):
    config.worker.worker_app = "synapse.app.admin_cmd"

    if (
-        not config.worker_daemonize
-        and not config.worker_log_file
-        and not config.worker_log_config
+        not config.worker.worker_daemonize
+        and not config.worker.worker_log_file
+        and not config.worker.worker_log_config
    ):
        # Since we're meant to be run as a "command" let's not redirect stdio
        # unless we've actually set log config.
-        config.no_redirect_stdio = True
+        config.logging.no_redirect_stdio = True

    # Explicitly disable background processes
-    config.update_user_directory = False
+    config.server.update_user_directory = False
    config.worker.run_background_tasks = False
-    config.start_pushers = False
+    config.worker.start_pushers = False
    config.pusher_shard_config.instances = []
-    config.send_federation = False
+    config.worker.send_federation = False
    config.federation_shard_config.instances = []

-    synapse.events.USE_FROZEN_DICTS = config.use_frozen_dicts
+    synapse.events.USE_FROZEN_DICTS = config.server.use_frozen_dicts

    ss = AdminCmdServer(
        config.server.server_name,
@@ -140,7 +140,7 @@ class KeyUploadServlet(RestServlet):
        self.auth = hs.get_auth()
        self.store = hs.get_datastore()
        self.http_client = hs.get_simple_http_client()
-        self.main_uri = hs.config.worker_main_http_uri
+        self.main_uri = hs.config.worker.worker_main_http_uri

    async def on_POST(self, request: Request, device_id: Optional[str]):
        requester = await self.auth.get_user_by_req(request, allow_guest=True)
@@ -321,7 +321,7 @@ class GenericWorkerServer(HomeServer):
                elif name == "federation":
                    resources.update({FEDERATION_PREFIX: TransportLayerServer(self)})
                elif name == "media":
-                    if self.config.can_load_media_repo:
+                    if self.config.media.can_load_media_repo:
                        media_repo = self.get_media_repository_resource()

                        # We need to serve the admin servlets for media on the
@@ -384,7 +384,7 @@ class GenericWorkerServer(HomeServer):
        logger.info("Synapse worker now listening on port %d", port)

    def start_listening(self):
-        for listener in self.config.worker_listeners:
+        for listener in self.config.worker.worker_listeners:
            if listener.type == "http":
                self._listen_http(listener)
            elif listener.type == "manhole":
@@ -395,7 +395,7 @@ class GenericWorkerServer(HomeServer):
                    manhole_globals={"hs": self},
                )
            elif listener.type == "metrics":
-                if not self.config.enable_metrics:
+                if not self.config.metrics.enable_metrics:
                    logger.warning(
                        "Metrics listener configured, but "
                        "enable_metrics is not True!"
@@ -462,7 +462,7 @@ def start(config_options):
        # For other worker types we force this to off.
        config.server.update_user_directory = False

-    synapse.events.USE_FROZEN_DICTS = config.use_frozen_dicts
+    synapse.events.USE_FROZEN_DICTS = config.server.use_frozen_dicts
    synapse.util.caches.TRACK_MEMORY_USAGE = config.caches.track_memory_usage

    if config.server.gc_seconds:
@@ -488,7 +488,7 @@ def start(config_options):
    register_start(_base.start, hs)

    # redirect stdio to the logs, if configured.
-    if not hs.config.no_redirect_stdio:
+    if not hs.config.logging.no_redirect_stdio:
        redirect_stdio_to_logs()

    _base.start_worker_reactor("synapse-generic-worker", config)
@@ -195,7 +195,7 @@ class SynapseHomeServer(HomeServer):
                }
            )

-            if self.config.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
+            if self.config.email.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
                from synapse.rest.synapse.client.password_reset import (
                    PasswordResetSubmitTokenResource,
                )
@@ -234,7 +234,7 @@ class SynapseHomeServer(HomeServer):
            )

        if name in ["media", "federation", "client"]:
-            if self.config.enable_media_repo:
+            if self.config.server.enable_media_repo:
                media_repo = self.get_media_repository_resource()
                resources.update(
                    {MEDIA_PREFIX: media_repo, LEGACY_MEDIA_PREFIX: media_repo}
@@ -248,7 +248,7 @@ class SynapseHomeServer(HomeServer):
            resources[SERVER_KEY_V2_PREFIX] = KeyApiV2Resource(self)

        if name == "webclient":
-            webclient_loc = self.config.web_client_location
+            webclient_loc = self.config.server.web_client_location

            if webclient_loc is None:
                logger.warning(
@@ -269,7 +269,7 @@ class SynapseHomeServer(HomeServer):
                # https://twistedmatrix.com/trac/ticket/7678
                resources[WEB_CLIENT_PREFIX] = File(webclient_loc)

-        if name == "metrics" and self.config.enable_metrics:
+        if name == "metrics" and self.config.metrics.enable_metrics:
            resources[METRICS_PREFIX] = MetricsResource(RegistryProxy)

        if name == "replication":
@@ -278,7 +278,7 @@ class SynapseHomeServer(HomeServer):
        return resources

    def start_listening(self):
-        if self.config.redis_enabled:
+        if self.config.redis.redis_enabled:
            # If redis is enabled we connect via the replication command handler
            # in the same way as the workers (since we're effectively a client
            # rather than a server).
@@ -305,7 +305,7 @@ class SynapseHomeServer(HomeServer):
                for s in services:
                    reactor.addSystemEventTrigger("before", "shutdown", s.stopListening)
            elif listener.type == "metrics":
-                if not self.config.enable_metrics:
+                if not self.config.metrics.enable_metrics:
                    logger.warning(
                        "Metrics listener configured, but "
                        "enable_metrics is not True!"
@@ -343,7 +343,7 @@ def setup(config_options):
        # generating config files and shouldn't try to continue.
        sys.exit(0)

-    events.USE_FROZEN_DICTS = config.use_frozen_dicts
+    events.USE_FROZEN_DICTS = config.server.use_frozen_dicts
    synapse.util.caches.TRACK_MEMORY_USAGE = config.caches.track_memory_usage

    if config.server.gc_seconds:
@@ -366,7 +366,7 @@ def setup(config_options):

    async def start():
        # Load the OIDC provider metadatas, if OIDC is enabled.
-        if hs.config.oidc_enabled:
+        if hs.config.oidc.oidc_enabled:
            oidc = hs.get_oidc_handler()
            # Loading the provider metadata also ensures the provider config is valid.
            await oidc.load_metadata()
@@ -439,11 +439,11 @@ def run(hs):

    _base.start_reactor(
        "synapse-homeserver",
-        soft_file_limit=hs.config.soft_file_limit,
-        gc_thresholds=hs.config.gc_thresholds,
-        pid_file=hs.config.pid_file,
-        daemonize=hs.config.daemonize,
-        print_pidfile=hs.config.print_pidfile,
+        soft_file_limit=hs.config.server.soft_file_limit,
+        gc_thresholds=hs.config.server.gc_thresholds,
+        pid_file=hs.config.server.pid_file,
+        daemonize=hs.config.server.daemonize,
+        print_pidfile=hs.config.server.print_pidfile,
        logger=logger,
    )

@@ -455,7 +455,7 @@ def main():
        hs = setup(sys.argv[1:])

        # redirect stdio to the logs, if configured.
-        if not hs.config.no_redirect_stdio:
+        if not hs.config.logging.no_redirect_stdio:
            redirect_stdio_to_logs()

        run(hs)
@@ -74,7 +74,7 @@ async def phone_stats_home(hs, stats, stats_process=_stats_process):
    store = hs.get_datastore()

    stats["homeserver"] = hs.config.server.server_name
-    stats["server_context"] = hs.config.server_context
+    stats["server_context"] = hs.config.server.server_context
    stats["timestamp"] = now
    stats["uptime_seconds"] = uptime
    version = sys.version_info
@@ -131,10 +131,12 @@ async def phone_stats_home(hs, stats, stats_process=_stats_process):
    log_level = synapse_logger.getEffectiveLevel()
    stats["log_level"] = logging.getLevelName(log_level)

-    logger.info("Reporting stats to %s: %s" % (hs.config.report_stats_endpoint, stats))
+    logger.info(
+        "Reporting stats to %s: %s" % (hs.config.metrics.report_stats_endpoint, stats)
+    )
    try:
        await hs.get_proxied_http_client().put_json(
-            hs.config.report_stats_endpoint, stats
+            hs.config.metrics.report_stats_endpoint, stats
        )
    except Exception as e:
        logger.warning("Error reporting stats: %s", e)
@@ -169,7 +171,7 @@ def start_phone_stats_home(hs):
        current_mau_count_by_service = {}
        reserved_users = ()
        store = hs.get_datastore()
-        if hs.config.limit_usage_by_mau or hs.config.mau_stats_only:
+        if hs.config.server.limit_usage_by_mau or hs.config.server.mau_stats_only:
            current_mau_count = await store.get_monthly_active_count()
            current_mau_count_by_service = (
                await store.get_monthly_active_count_by_service()
@@ -181,14 +183,14 @@ def start_phone_stats_home(hs):
            current_mau_by_service_gauge.labels(app_service).set(float(count))

        registered_reserved_users_mau_gauge.set(float(len(reserved_users)))
-        max_mau_gauge.set(float(hs.config.max_mau_value))
+        max_mau_gauge.set(float(hs.config.server.max_mau_value))

-    if hs.config.limit_usage_by_mau or hs.config.mau_stats_only:
+    if hs.config.server.limit_usage_by_mau or hs.config.server.mau_stats_only:
        generate_monthly_active_users()
        clock.looping_call(generate_monthly_active_users, 5 * 60 * 1000)
    # End of monthly active user settings

-    if hs.config.report_stats:
+    if hs.config.metrics.report_stats:
        logger.info("Scheduling stats reporting for 3 hour intervals")
        clock.looping_call(phone_stats_home, 3 * 60 * 60 * 1000, hs, stats)

@@ -118,21 +118,6 @@ class Config:
            "synapse", "res/templates"
        )

-    def __getattr__(self, item: str) -> Any:
-        """
-        Try and fetch a configuration option that does not exist on this class.
-
-        This is so that existing configs that rely on `self.value`, where value
-        is actually from a different config section, continue to work.
-        """
-        if item in ["generate_config_section", "read_config"]:
-            raise AttributeError(item)
-
-        if self.root is None:
-            raise AttributeError(item)
-        else:
-            return self.root._get_unclassed_config(self.section, item)
-
    @staticmethod
    def parse_size(value):
        if isinstance(value, int):
@@ -200,11 +185,7 @@ class Config:
    @classmethod
    def ensure_directory(cls, dir_path):
        dir_path = cls.abspath(dir_path)
-        try:
-            os.makedirs(dir_path)
-        except OSError as e:
-            if e.errno != errno.EEXIST:
-                raise
+        os.makedirs(dir_path, exist_ok=True)
        if not os.path.isdir(dir_path):
            raise ConfigError("%s is not a directory" % (dir_path,))
        return dir_path
@@ -293,7 +274,9 @@ class Config:
        env.filters.update(
            {
                "format_ts": _format_ts_filter,
-                "mxc_to_http": _create_mxc_to_http_filter(self.public_baseurl),
+                "mxc_to_http": _create_mxc_to_http_filter(
+                    self.root.server.public_baseurl
+                ),
            }
        )

@@ -315,8 +298,6 @@ class RootConfig:
    config_classes = []

    def __init__(self):
-        self._configs = OrderedDict()
-
        for config_class in self.config_classes:
            if config_class.section is None:
                raise ValueError("%r requires a section name" % (config_class,))
@@ -325,42 +306,7 @@ class RootConfig:
                conf = config_class(self)
            except Exception as e:
                raise Exception("Failed making %s: %r" % (config_class.section, e))
-            self._configs[config_class.section] = conf
-
-    def __getattr__(self, item: str) -> Any:
-        """
-        Redirect lookups on this object either to config objects, or values on
-        config objects, so that `config.tls.blah` works, as well as legacy uses
-        of things like `config.server_name`. It will first look up the config
-        section name, and then values on those config classes.
-        """
-        if item in self._configs.keys():
-            return self._configs[item]
-
-        return self._get_unclassed_config(None, item)
-
-    def _get_unclassed_config(self, asking_section: Optional[str], item: str):
-        """
-        Fetch a config value from one of the instantiated config classes that
-        has not been fetched directly.
-
-        Args:
-            asking_section: If this check is coming from a Config child, which
-                one? This section will not be asked if it has the value.
-            item: The configuration value key.
-
-        Raises:
-            AttributeError if no config classes have the config key. The body
-                will contain what sections were checked.
-        """
-        for key, val in self._configs.items():
-            if key == asking_section:
-                continue
-
-            if item in dir(val):
-                return getattr(val, item)
-
-        raise AttributeError(item, "not found in %s" % (list(self._configs.keys()),))
+            setattr(self, config_class.section, conf)

    def invoke_all(self, func_name: str, *args, **kwargs) -> MutableMapping[str, Any]:
        """
@@ -377,9 +323,11 @@ class RootConfig:
        """
        res = OrderedDict()

-        for name, config in self._configs.items():
+        for config_class in self.config_classes:
+            config = getattr(self, config_class.section)
+
            if hasattr(config, func_name):
-                res[name] = getattr(config, func_name)(*args, **kwargs)
+                res[config_class.section] = getattr(config, func_name)(*args, **kwargs)

        return res

@@ -693,8 +641,7 @@ class RootConfig:
                    open_private_ports=config_args.open_private_ports,
                )

-                if not path_exists(config_dir_path):
-                    os.makedirs(config_dir_path)
+                os.makedirs(config_dir_path, exist_ok=True)
                with open(config_path, "w") as config_file:
                    config_file.write(config_str)
                    config_file.write("\n\n# vim:ft=yaml")
@@ -76,7 +76,7 @@ class AccountValidityConfig(Config):
            )

        if self.account_validity_renew_by_email_enabled:
-            if not self.public_baseurl:
+            if not self.root.server.public_baseurl:
                raise ConfigError("Can't send renewal emails without 'public_baseurl'")

        # Load account validity templates.
@@ -37,7 +37,7 @@ class CasConfig(Config):

            # The public baseurl is required because it is used by the redirect
            # template.
-            public_baseurl = self.public_baseurl
+            public_baseurl = self.root.server.public_baseurl
            if not public_baseurl:
                raise ConfigError("cas_config requires a public_baseurl to be set")

@@ -13,6 +13,7 @@
 # limitations under the License.

 from os import path
+from typing import Optional

 from synapse.config import ConfigError

@@ -78,8 +79,8 @@ class ConsentConfig(Config):
    def __init__(self, *args):
        super().__init__(*args)

-        self.user_consent_version = None
-        self.user_consent_template_dir = None
+        self.user_consent_version: Optional[str] = None
+        self.user_consent_template_dir: Optional[str] = None
        self.user_consent_server_notice_content = None
        self.user_consent_server_notice_to_guests = False
        self.block_events_without_consent_error = None
@@ -94,7 +95,9 @@ class ConsentConfig(Config):
            return
        self.user_consent_version = str(consent_config["version"])
        self.user_consent_template_dir = self.abspath(consent_config["template_dir"])
-        if not path.isdir(self.user_consent_template_dir):
+        if not isinstance(self.user_consent_template_dir, str) or not path.isdir(
+            self.user_consent_template_dir
+        ):
            raise ConfigError(
                "Could not find template directory '%s'"
                % (self.user_consent_template_dir,)
@@ -19,7 +19,6 @@ import email.utils
 import logging
 import os
 from enum import Enum
-from typing import Optional

 import attr

@@ -135,7 +134,7 @@ class EmailConfig(Config):
            # msisdn is currently always remote while Synapse does not support any method of
            # sending SMS messages
            ThreepidBehaviour.REMOTE
-            if self.account_threepid_delegate_email
+            if self.root.registration.account_threepid_delegate_email
            else ThreepidBehaviour.LOCAL
        )
        # Prior to Synapse v1.4.0, there was another option that defined whether Synapse would
@@ -144,7 +143,7 @@ class EmailConfig(Config):
        # identity server in the process.
        self.using_identity_server_from_trusted_list = False
        if (
-            not self.account_threepid_delegate_email
+            not self.root.registration.account_threepid_delegate_email
            and config.get("trust_identity_server_for_password_resets", False) is True
        ):
            # Use the first entry in self.trusted_third_party_id_servers instead
@@ -156,7 +155,7 @@ class EmailConfig(Config):

                # trusted_third_party_id_servers does not contain a scheme whereas
                # account_threepid_delegate_email is expected to. Presume https
-                self.account_threepid_delegate_email: Optional[str] = (
+                self.root.registration.account_threepid_delegate_email = (
                    "https://" + first_trusted_identity_server
                )
                self.using_identity_server_from_trusted_list = True
@@ -335,7 +334,7 @@ class EmailConfig(Config):
                "client_base_url", email_config.get("riot_base_url", None)
            )

-        if self.account_validity_renew_by_email_enabled:
+        if self.root.account_validity.account_validity_renew_by_email_enabled:
            expiry_template_html = email_config.get(
                "expiry_template_html", "notice_expiry.html"
            )
--- a/Show More
+++ b/Show More
				`@@ -0,0 +1 @@`
				Ensure `(room_id, next_batch_id)` is unique across [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) insertion events in rooms to avoid cross-talk/conflicts between batches.
				`@@ -0,0 +1 @@`
				Add a `user_may_send_3pid_invite` spam checker callback for modules to allow or deny 3PID invites.
				`@@ -0,0 +1 @@`
				`Fix type hints to be compatible with an upcoming change to Twisted.`
				`@@ -0,0 +1 @@`
				`Update utility code to handle C implementations of frozendict.`
				`@@ -0,0 +1 @@`
				`Drop old functionality which maintained database compatibility with Synapse versions before 1.31.`
				`@@ -0,0 +1 @@`
				`Add a spam checker callback to allow or deny room joins.`
				`@@ -0,0 +1 @@`
				Clean-up configuration helper classes for the `ServerConfig` class.
				`@@ -0,0 +1 @@`
				Fix a minor bug in the response to `/_matrix/client/r0/voip/turnServer`. Contributed by @lukaslihotzki.
				`@@ -0,0 +1 @@`
				Fix a bug where empty `yyyy-mm-dd/` directories would be left behind in the media store's `url_cache_thumbnails/` directory.
				`@@ -0,0 +1 @@`
				`Clean up some of the federation event authentication code for clarity.`