moved more saml imports behind checks

(tried everything with pysaml2 uninstalled so should now work!)
Added ANOTHER check for saml being enabled before loading it
2021-08-25 11:49:21 +01:00 · 2021-08-25 11:22:04 +01:00 · 2021-08-25 10:48:47 +01:00 · 2021-08-24 16:42:03 +01:00 · 2021-08-24 16:29:07 +01:00 · 2021-08-24 14:38:22 +01:00
195 changed files with 5389 additions and 1715 deletions
@@ -0,0 +1,8 @@
+#!/bin/sh
+
+# replaces the dependency on Twisted in `python_dependencies` with trunk.
+
+set -e
+cd "$(dirname "$0")"/..
+
+sed -i -e 's#"Twisted.*"#"Twisted @ git+https://github.com/twisted/twisted"#' synapse/python_dependencies.py
@@ -0,0 +1,4 @@
+---
+title: CI run against Twisted trunk is failing
+---
+See https://github.com/{{env.GITHUB_REPOSITORY}}/actions/runs/{{env.GITHUB_RUN_ID}}
@@ -0,0 +1,90 @@
+name: Twisted Trunk
+
+on:
+  schedule:
+    - cron: 0 8 * * *
+
+  workflow_dispatch:
+
+jobs:
+  mypy:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+      - run: .ci/patch_for_twisted_trunk.sh
+      - run: pip install tox
+      - run: tox -e mypy
+
+  trial:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v2
+      - run: sudo apt-get -qq install xmlsec1
+      - uses: actions/setup-python@v2
+        with:
+          python-version: 3.6
+      - run: .ci/patch_for_twisted_trunk.sh
+      - run: pip install tox
+      - run: tox -e py
+        env:
+          TRIAL_FLAGS: "--jobs=2"
+
+      - name: Dump logs
+        # Note: Dumps to workflow logs instead of using actions/upload-artifact
+        #       This keeps logs colocated with failing jobs
+        #       It also ignores find's exit code; this is a best effort affair
+        run: >-
+          find _trial_temp -name '*.log'
+          -exec echo "::group::{}" \;
+          -exec cat {} \;
+          -exec echo "::endgroup::" \;
+          || true
+
+  sytest:
+    runs-on: ubuntu-latest
+    container:
+      image: matrixdotorg/sytest-synapse:buster
+      volumes:
+        - ${{ github.workspace }}:/src
+
+    steps:
+      - uses: actions/checkout@v2
+      - name: Patch dependencies
+        run: .ci/patch_for_twisted_trunk.sh
+        working-directory: /src
+      - name: Run SyTest
+        run: /bootstrap.sh synapse
+        working-directory: /src
+      - name: Summarise results.tap
+        if: ${{ always() }}
+        run: /sytest/scripts/tap_to_gha.pl /logs/results.tap
+      - name: Upload SyTest logs
+        uses: actions/upload-artifact@v2
+        if: ${{ always() }}
+        with:
+          name: Sytest Logs - ${{ job.status }} - (${{ join(matrix.*, ', ') }})
+          path: |
+            /logs/results.tap
+            /logs/**/*.log*
+
+  # open an issue if the build fails, so we know about it.
+  open-issue:
+    if: failure()
+    needs:
+      - mypy
+      - trial
+      - sytest
+
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v2
+      - uses: JasonEtco/create-an-issue@5d9504915f79f9cc6d791934b8ef34f2353dd74d # v2.5.0, 2020-12-06
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        with:
+          update_existing: true
+          filename: .ci/twisted_trunk_build_failed_issue_template.md
@@ -1,3 +1,82 @@
+Synapse 1.41.0rc1 (2021-08-18)
+==============================
+
+Features
+--------
+
+- Add `get_userinfo_by_id` method to ModuleApi. ([\#9581](https://github.com/matrix-org/synapse/issues/9581))
+- Initial local support for [MSC3266](https://github.com/matrix-org/synapse/pull/10394), Room Summary over the unstable `/rooms/{roomIdOrAlias}/summary` API. ([\#10394](https://github.com/matrix-org/synapse/issues/10394))
+- Experimental support for [MSC3288](https://github.com/matrix-org/matrix-doc/pull/3288), sending `room_type` to the identity server for 3pid invites over the `/store-invite` API. ([\#10435](https://github.com/matrix-org/synapse/issues/10435))
+- Add support for sending federation requests through a proxy. Contributed by @Bubu and @dklimpel. ([\#10475](https://github.com/matrix-org/synapse/issues/10475))
+- Add support for "marker" events which makes historical events discoverable for servers that already have all of the scrollback history (part of [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)). ([\#10498](https://github.com/matrix-org/synapse/issues/10498))
+- Add a configuration setting for the time a `/sync` response is cached for. ([\#10513](https://github.com/matrix-org/synapse/issues/10513))
+- The default logging handler for new installations is now `PeriodicallyFlushingMemoryHandler`, a buffered logging handler which periodically flushes itself. ([\#10518](https://github.com/matrix-org/synapse/issues/10518))
+- Add support for new redaction rules for historical events specified in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716). ([\#10538](https://github.com/matrix-org/synapse/issues/10538))
+- Add a setting to disable TLS when sending email. ([\#10546](https://github.com/matrix-org/synapse/issues/10546))
+- Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946). ([\#10549](https://github.com/matrix-org/synapse/issues/10549), [\#10560](https://github.com/matrix-org/synapse/issues/10560), [\#10569](https://github.com/matrix-org/synapse/issues/10569), [\#10574](https://github.com/matrix-org/synapse/issues/10574), [\#10575](https://github.com/matrix-org/synapse/issues/10575), [\#10579](https://github.com/matrix-org/synapse/issues/10579), [\#10583](https://github.com/matrix-org/synapse/issues/10583))
+- Admin API to delete several media for a specific user. Contributed by @dklimpel. ([\#10558](https://github.com/matrix-org/synapse/issues/10558), [\#10628](https://github.com/matrix-org/synapse/issues/10628))
+- Add support for routing `/createRoom` to workers. ([\#10564](https://github.com/matrix-org/synapse/issues/10564))
+- Update the Synapse Grafana dashboard. ([\#10570](https://github.com/matrix-org/synapse/issues/10570))
+- Add an admin API (`GET /_synapse/admin/username_available`) to check if a username is available (regardless of registration settings). ([\#10578](https://github.com/matrix-org/synapse/issues/10578))
+- Allow editing a user's `external_ids` via the "Edit User" admin API. Contributed by @dklimpel. ([\#10598](https://github.com/matrix-org/synapse/issues/10598))
+- The Synapse manhole no longer needs coroutines to be wrapped in `defer.ensureDeferred`. ([\#10602](https://github.com/matrix-org/synapse/issues/10602))
+- Add option to allow modules to run periodic tasks on all instances, rather than just the one configured to run background tasks. ([\#10638](https://github.com/matrix-org/synapse/issues/10638))
+
+
+Bugfixes
+--------
+
+- Add some clarification to the sample config file. Contributed by @Kentokamoto. ([\#10129](https://github.com/matrix-org/synapse/issues/10129))
+- Fix a long-standing bug where protocols which are not implemented by any appservices were incorrectly returned via `GET /_matrix/client/r0/thirdparty/protocols`. ([\#10532](https://github.com/matrix-org/synapse/issues/10532))
+- Fix exceptions in logs when failing to get remote room list. ([\#10541](https://github.com/matrix-org/synapse/issues/10541))
+- Fix longstanding bug which caused the user "status" to be reset when the user went offline. Contributed by @dklimpel. ([\#10550](https://github.com/matrix-org/synapse/issues/10550))
+- Allow public rooms to be previewed in the spaces summary APIs from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946). ([\#10580](https://github.com/matrix-org/synapse/issues/10580))
+- Fix a bug introduced in v1.37.1 where an error could occur in the asynchronous processing of PDUs when the queue was empty. ([\#10592](https://github.com/matrix-org/synapse/issues/10592))
+- Fix errors on /sync when read receipt data is a string. Only affects homeservers with the experimental flag for [MSC2285](https://github.com/matrix-org/matrix-doc/pull/2285) enabled. Contributed by @SimonBrandner. ([\#10606](https://github.com/matrix-org/synapse/issues/10606))
+- Additional validation for the spaces summary API to avoid errors like `ValueError: Stop argument for islice() must be None or an integer`. The missing validation has existed since v1.31.0. ([\#10611](https://github.com/matrix-org/synapse/issues/10611))
+- Revert behaviour introduced in v1.38.0 that strips `org.matrix.msc2732.device_unused_fallback_key_types` from `/sync` when its value is empty. This field should instead always be present according to [MSC2732](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2732-olm-fallback-keys.md). ([\#10623](https://github.com/matrix-org/synapse/issues/10623))
+
+
+Improved Documentation
+----------------------
+
+- Add documentation for configuration a forward proxy. ([\#10443](https://github.com/matrix-org/synapse/issues/10443))
+- Updated the reverse proxy documentation to highlight the homserver configuration that is needed to make Synapse aware that is is intentionally reverse proxied. ([\#10551](https://github.com/matrix-org/synapse/issues/10551))
+- Update CONTRIBUTING.md to fix index links and the instructions for SyTest in docker. ([\#10599](https://github.com/matrix-org/synapse/issues/10599))
+
+
+Deprecations and Removals
+-------------------------
+
+- No longer build `.deb` packages for Ubuntu 20.10 LTS Groovy Gorilla, which has now EOLed. ([\#10588](https://github.com/matrix-org/synapse/issues/10588))
+- The `template_dir` configuration settings in the `sso`, `account_validity` and `email` sections of the configuration file are now deprecated in favour of the global `templates.custom_template_directory` setting. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html) for more information. ([\#10596](https://github.com/matrix-org/synapse/issues/10596))
+
+
+Internal Changes
+----------------
+
+- Improve event caching mechanism to avoid having multiple copies of an event in memory at a time. ([\#10119](https://github.com/matrix-org/synapse/issues/10119))
+- Reduce errors in PostgreSQL logs due to concurrent serialization errors. ([\#10504](https://github.com/matrix-org/synapse/issues/10504))
+- Include room ID in ignored EDU log messages. Contributed by @ilmari. ([\#10507](https://github.com/matrix-org/synapse/issues/10507))
+- Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946). ([\#10527](https://github.com/matrix-org/synapse/issues/10527), [\#10530](https://github.com/matrix-org/synapse/issues/10530))
+- Fix CI to not break when run against branches rather than pull requests. ([\#10529](https://github.com/matrix-org/synapse/issues/10529))
+- Mark all events stemming from the [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint as historical. ([\#10537](https://github.com/matrix-org/synapse/issues/10537))
+- Clean up some of the federation event authentication code for clarity. ([\#10539](https://github.com/matrix-org/synapse/issues/10539), [\#10591](https://github.com/matrix-org/synapse/issues/10591))
+- Convert `Transaction` and `Edu` objects to attrs. ([\#10542](https://github.com/matrix-org/synapse/issues/10542))
+- Update `/batch_send` endpoint to only return `state_events` created by the `state_events_from_before` passed in. ([\#10552](https://github.com/matrix-org/synapse/issues/10552))
+- Update contributing.md to warn against rebasing an open PR. ([\#10563](https://github.com/matrix-org/synapse/issues/10563))
+- Remove the unused public rooms replication stream. ([\#10565](https://github.com/matrix-org/synapse/issues/10565))
+- Clarify error message when failing to join a restricted room. ([\#10572](https://github.com/matrix-org/synapse/issues/10572))
+- Remove references to BuildKite in favour of GitHub Actions. ([\#10573](https://github.com/matrix-org/synapse/issues/10573))
+- Move `/batch_send` endpoint defined by [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) to the `/v2_alpha` directory. ([\#10576](https://github.com/matrix-org/synapse/issues/10576))
+- Allow multiple custom directories in `read_templates`. ([\#10587](https://github.com/matrix-org/synapse/issues/10587))
+- Re-organize the `synapse.federation.transport.server` module to create smaller files. ([\#10590](https://github.com/matrix-org/synapse/issues/10590))
+- Flatten the `synapse.rest.client` package by moving the contents of `v1` and `v2_alpha` into the parent. ([\#10600](https://github.com/matrix-org/synapse/issues/10600))
+- Build Debian packages for Debian 12 (Bookworm). ([\#10612](https://github.com/matrix-org/synapse/issues/10612))
+- Fix up a couple of links to the database schema documentation. ([\#10620](https://github.com/matrix-org/synapse/issues/10620))
+- Fix a broken link to the upgrade notes. ([\#10631](https://github.com/matrix-org/synapse/issues/10631))
+
+
 Synapse 1.40.0 (2021-08-10)
 ===========================

@@ -1,7 +1,7 @@
 Upgrading Synapse
 =================

-This document has moved to the `Synapse documentation website <https://matrix-org.github.io/synapse/latest/upgrading>`_.
+This document has moved to the `Synapse documentation website <https://matrix-org.github.io/synapse/latest/upgrade>`_.
 Please update your links.

 The markdown source is available in `docs/upgrade.md <docs/upgrade.md>`_.
@@ -1 +0,0 @@
-Improve event caching mechanism to avoid having multiple copies of an event in memory at a time.
@@ -1 +0,0 @@
-Add some clarification to the sample config file. Contributed by @Kentokamoto.
@@ -0,0 +1 @@
+Add support for [MSC3231 - Token authenticated registration](https://github.com/matrix-org/matrix-doc/pull/3231). Users can be required to submit a token during registration to authenticate themselves. Contributed by Callum Brown.
@@ -0,0 +1 @@
+Add documentation on how to connect Django with synapse using oidc and django-oauth-toolkit. Contributed by @HugoDelval.
@@ -1 +0,0 @@
-Initial local support for [MSC3266](https://github.com/matrix-org/synapse/pull/10394), Room Summary over the unstable `/rooms/{roomIdOrAlias}/summary` API.
@@ -1 +0,0 @@
-Experimental support for [MSC3288](https://github.com/matrix-org/matrix-doc/pull/3288), sending `room_type` to the identity server for 3pid invites over the `/store-invite` API.
@@ -1 +0,0 @@
-Add documentation for configuration a forward proxy.
@@ -0,0 +1 @@
+Add support for [MSC3283](https://github.com/matrix-org/matrix-doc/pull/3283): Expose enable_set_displayname in capabilities.
@@ -1 +0,0 @@
-Add support for sending federation requests through a proxy. Contributed by @Bubu and @dklimpel.
@@ -1 +0,0 @@
-Add support for "marker" events which makes historical events discoverable for servers that already have all of the scrollback history (part of MSC2716).
@@ -1 +0,0 @@
-Reduce errors in PostgreSQL logs due to concurrent serialization errors.
@@ -1 +0,0 @@
-Include room ID in ignored EDU log messages. Contributed by @ilmari.
@@ -1 +0,0 @@
-Add a configuration setting for the time a `/sync` response is cached for.
@@ -1 +0,0 @@
-Fix CI to not break when run against branches rather than pull requests.
@@ -1 +0,0 @@
-Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).
@@ -1 +0,0 @@
-Fix a long-standing bug where protocols which are not implemented by any appservices were incorrectly returned via `GET /_matrix/client/r0/thirdparty/protocols`.
@@ -1 +0,0 @@
-Mark all events stemming from the MSC2716 `/batch_send` endpoint as historical.
@@ -1 +0,0 @@
-Add support for new redaction rules for historical events specified in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716).
@@ -1 +0,0 @@
-Fix exceptions in logs when failing to get remote room list.
@@ -1 +0,0 @@
-Convert `Transaction` and `Edu` objects to attrs.
@@ -1 +0,0 @@
-Add a setting to disable TLS when sending email.
@@ -1 +0,0 @@
-Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).
@@ -1 +0,0 @@
-Fix longstanding bug which caused the user "status" to be reset when the user went offline. Contributed by @dklimpel.
@@ -1 +0,0 @@
-Updated the reverse proxy documentation to highlight the homserver configuration that is needed to make Synapse aware that is is intentionally reverse proxied.
@@ -1 +0,0 @@
-Update `/batch_send` endpoint to only return `state_events` created by the `state_events_from_before` passed in.
@@ -1 +0,0 @@
-Admin API to delete several media for a specific user. Contributed by @dklimpel.
@@ -1 +0,0 @@
-Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).
@@ -1 +0,0 @@
-Update contributing.md to warn against rebasing an open PR.
@@ -1 +0,0 @@
-Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).
@@ -1 +0,0 @@
-Update the Synapse Grafana dashboard.
@@ -1 +0,0 @@
-Clarify error message when failing to join a restricted room.
@@ -1 +0,0 @@
-Remove references to BuildKite in favour of GitHub Actions.
@@ -1 +0,0 @@
-Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).
@@ -1 +0,0 @@
-Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).
@@ -1 +0,0 @@
-Move `/batch_send` endpoint defined by [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) to the `/v2_alpha` directory.
@@ -1 +0,0 @@
-Add an admin API (`GET /_synapse/admin/username_available`) to check if a username is available (regardless of registration settings).
@@ -1 +0,0 @@
-Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).
@@ -1 +0,0 @@
-Allow public rooms to be previewed in the spaces summary APIs from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).
@@ -1 +0,0 @@
-Allow multiple custom directories in `read_templates`.
@@ -1 +0,0 @@
-No longer build `.dev` packages for Ubuntu 20.10 LTS Groovy Gorilla, which has now EOLed.
@@ -1 +0,0 @@
-Re-organize the `synapse.federation.transport.server` module to create smaller files.
@@ -1 +0,0 @@
-Fix a bug introduced in v1.37.1 where an error could occur in the asyncronous processing of PDUs when the queue was empty.
@@ -0,0 +1 @@
+Reject Client-Server `/keys/query` requests which provide `device_ids` incorrectly.
@@ -1 +0,0 @@
-Allow editing a user's `external_ids` via the "Edit User" admin API. Contributed by @dklimpel.
@@ -1 +0,0 @@
-Update CONTRIBUTING.md to fix index links and the instructions for SyTest in docker.
@@ -1 +0,0 @@
-Flatten the `synapse.rest.client` package by moving the contents of `v1` and `v2_alpha` into the parent.
@@ -1 +0,0 @@
-The Synapse manhole no longer needs coroutines to be wrapped in `defer.ensureDeferred`.
@@ -1 +0,0 @@
-Fix errors on /sync when read receipt data is a string. Only affects homeservers with the experimental flag for [MSC2285](https://github.com/matrix-org/matrix-doc/pull/2285) enabled. Contributed by @SimonBrandner.
@@ -0,0 +1 @@
+Improve type hints for the proxy agent and SRV resolver modules. Contributed by @dklimpel.
@@ -1 +0,0 @@
-Additional validation for the spaces summary API to avoid errors like `ValueError: Stop argument for islice() must be None or an integer`. The missing validation has existed since v1.31.0.
@@ -1 +0,0 @@
-Fix up a couple of links to the database schema documentation.
@@ -1 +0,0 @@
-Revert behaviour introduced in v1.38.0 that strips `org.matrix.msc2732.device_unused_fallback_key_types` from `/sync` when its value is empty. This field should instead always be present according to [MSC2732](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2732-olm-fallback-keys.md).
@@ -0,0 +1 @@
+Remove not needed database updates in modify user admin API.
@@ -0,0 +1 @@
+Fix some of the titles not rendering in the OIDC documentation.
@@ -0,0 +1 @@
+Convert room member storage tuples to `attrs` classes.
@@ -0,0 +1 @@
+Rooms with unsupported room versions are no longer returned via `/sync`.
@@ -0,0 +1 @@
+Run a nightly CI build against Twisted trunk.
@@ -0,0 +1 @@
+Enforce the maximum length for per-room display names and avatar URLs.
@@ -0,0 +1 @@
+Do not print out stack traces for network errors when fetching data over federation.
@@ -0,0 +1 @@
+Simplify tests for device admin rest API.
@@ -0,0 +1 @@
+Add missing type hints to REST servlets.
@@ -0,0 +1 @@
+Add missing type hints to REST servlets.
@@ -0,0 +1 @@
+Flatten the `tests.synapse.rests` package by moving the contents of `v1` and `v2_alpha` into the parent.
@@ -0,0 +1 @@
+Run a nightly CI build against Twisted trunk.
@@ -0,0 +1 @@
+Fix a bug which caused the `synapse_user_logins_total` Prometheus metric not to be correctly initialised on restart.
@@ -0,0 +1 @@
+Port the SAML user mapping providers module interface to the new generic interface.
@@ -1 +0,0 @@
-Add `get_userinfo_by_id` method to ModuleApi.
@@ -1,3 +1,9 @@
+matrix-synapse-py3 (1.41.0~rc1) stable; urgency=medium
+
+  * New synapse release 1.41.0~rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Wed, 18 Aug 2021 15:52:00 +0100
+
 matrix-synapse-py3 (1.40.0) stable; urgency=medium

  * New synapse release 1.40.0.
@@ -18,18 +18,31 @@ handlers:
    backupCount: 6  # Does not include the current log file.
    encoding: utf8

-  # Default to buffering writes to log file for efficiency. This means that
-  # there will be a delay for INFO/DEBUG logs to get written, but WARNING/ERROR
-  # logs will still be flushed immediately.
+  # Default to buffering writes to log file for efficiency.
+  # WARNING/ERROR logs will still be flushed immediately, but there will be a
+  # delay (of up to `period` seconds, or until the buffer is full with
+  # `capacity` messages) before INFO/DEBUG logs get written.
  buffer:
-    class: logging.handlers.MemoryHandler
+    class: synapse.logging.handlers.PeriodicallyFlushingMemoryHandler
    target: file
-    # The capacity is the number of log lines that are buffered before
-    # being written to disk. Increasing this will lead to better
+
+    # The capacity is the maximum number of log lines that are buffered
+    # before being written to disk. Increasing this will lead to better
    # performance, at the expensive of it taking longer for log lines to
    # be written to disk.
+    # This parameter is required.
    capacity: 10
-    flushLevel: 30  # Flush for WARNING logs as well
+
+    # Logs with a level at or above the flush level will cause the buffer to
+    # be flushed immediately.
+    # Default value: 40 (ERROR)
+    # Other values: 50 (CRITICAL), 30 (WARNING), 20 (INFO), 10 (DEBUG)
+    flushLevel: 30  # Flush immediately for WARNING logs and higher
+
+    # The period of time, in seconds, between forced flushes.
+    # Messages will not be delayed for longer than this time.
+    # Default value: 5 seconds
+    period: 5
 {% endif %}

  console:
@@ -21,6 +21,7 @@
    - [Homeserver Sample Config File](usage/configuration/homeserver_sample_config.md)
    - [Logging Sample Config File](usage/configuration/logging_sample_config.md)
    - [Structured Logging](structured_logging.md)
+    - [Templates](templates.md)
    - [User Authentication](usage/configuration/user_authentication/README.md)
      - [Single-Sign On]()
        - [OpenID Connect](openid.md)
@@ -52,6 +53,7 @@
      - [Media](admin_api/media_admin_api.md)
      - [Purge History](admin_api/purge_history_api.md)
      - [Register Users](admin_api/register_api.md)
+      - [Registration Tokens](usage/administration/admin_api/registration_tokens.md)
      - [Manipulate Room Membership](admin_api/room_membership.md)
      - [Rooms](admin_api/rooms.md)
      - [Server Notices](admin_api/server_notices.md)
@@ -21,11 +21,15 @@ It returns a JSON body like the following:
    "threepids": [
        {
            "medium": "email",
-            "address": "<user_mail_1>"
+            "address": "<user_mail_1>",
+            "added_at": 1586458409743,
+            "validated_at": 1586458409743
        },
        {
            "medium": "email",
-            "address": "<user_mail_2>"
+            "address": "<user_mail_2>",
+            "added_at": 1586458409743,
+            "validated_at": 1586458409743
        }
    ],
    "avatar_url": "<avatar_url>",
@@ -329,6 +329,76 @@ For example, if the user `@alice:example.org` is passed to this method, and the
 should receive presence updates sent by Bob and Charlie, regardless of whether these users 
 share a room.

+#### Saml User Mapping Provider Callbacks
+
+Saml user mapping provider callbacks are used to work out how to map
+attributes of an SSO response to Matrix-specific user attributes.
+
+As an example, a SSO service may return the email address
+"john.smith@example.com" for a user and Synapse will need to figure out how
+to turn that into a displayname when creating a Matrix user for this individual.
+It may choose `John Smith`, or `Smith, John [Example.com]` or any number of
+variations.
+
+A module with mapping provider functionality must register all of the following:
+
+```python
+saml_attributes: Tuple[Set[str], Set[str]]
+```
+A tuple of two sets, the first being the SAML auth response attributes that are
+required for the module to function, and the second set being the attributes which
+can be used if available, but are not necessary.
+
+```python
+async def get_remote_user_id(
+        self,
+        saml_response: "saml2.response.AuthnResponse",
+        client_redirect_url: Optional[str],
+    ) -> str
+```
+This callback is used to extract the *remote* user id for a user. It is provided with a SAML
+auth response object to extract the information from, and the URL that the client will
+be redirected to after authentication - which may be None. It should return an
+immutable identifier for the user (Commonly the `uid` field of the response). The module should
+return a unique identifier for each user. If no mapping can be made then it should raise a 
+`synapse.ModuleApi.errors.MappingException`.
+
+```python
+async def saml_response_to_user_attributes(
+        self,
+        saml_response: "saml2.response.AuthnResponse",
+        failures: int,
+        client_redirect_url: str,
+    ) -> dict
+```
+This callback is used to extract certain attributes for a new user. 
+It is provided with a SAML auth response object to extract the information from, a number representing
+the number of times the returned matrix user id mapping has failed and the URL that the client will
+be redirected to after authentication.
+
+It should return a dict which will be used by Synapse to build a new user. 
+The following keys are allowed:  
+* `mxid_localpart` - A string, the local part of the matrix user ID for the new user. 
+                     If this is `None`, the user is prompted to pick their own username.
+                     This is only used during a user's first login. Once a localpart has 
+                     been associated with a remote user ID (see `get_remote_user_id`) it
+                     cannot be updated.  
+* `displayname` - The displayname of the new user. If not provided, it will default to 
+                  the value of `mxid_localpart`.  
+* `emails` - A list of emails for the new user. If not provided, it will default to an empty list.
+
+For example, if this method returns `john.doe` as the value of `mxid_localpart` in the returned
+dict, and that is already taken on the homeserver, this method will be called again with the
+same parameters but with `failures=1`. The method should then return a different `mxid_localpart`
+value, such as `john.doe1`.
+
+If no mapping can be made then it should raise a `synapse.ModuleApi.errors.MappingException`.
+
+Alternatively it may raise a `synapse.ModuleApi.errors.RedirectException` to redirect the user to another
+page which prompts for additional information. After which, it is the module's responsibility
+to either redirect back to `client_redirect_url` (including any additional information)
+or to complete registration using methods from the ModuleApi. TODO: explain what this means in more detail
+
 ### Porting an existing module that uses the old interface

 In order to port a module that uses Synapse's old module interface, its author needs to:
@@ -79,7 +79,7 @@ oidc_providers:
        display_name_template: "{{ user.name }}"
 ```

-### [Dex][dex-idp]
+### Dex

 [Dex][dex-idp] is a simple, open-source, certified OpenID Connect Provider.
 Although it is designed to help building a full-blown provider with an
@@ -117,7 +117,7 @@ oidc_providers:
        localpart_template: "{{ user.name }}"
        display_name_template: "{{ user.name|capitalize }}"
 ```
-### [Keycloak][keycloak-idp]
+### Keycloak

 [Keycloak][keycloak-idp] is an opensource IdP maintained by Red Hat.

@@ -166,7 +166,9 @@ oidc_providers:
        localpart_template: "{{ user.preferred_username }}"
        display_name_template: "{{ user.name }}"
 ```
-### [Auth0][auth0]
+### Auth0
+
+[Auth0][auth0] is a hosted SaaS IdP solution.

 1. Create a regular web application for Synapse
 2. Set the Allowed Callback URLs to `[synapse public baseurl]/_synapse/client/oidc/callback`
@@ -209,7 +211,7 @@ oidc_providers:

 ### GitHub

-GitHub is a bit special as it is not an OpenID Connect compliant provider, but
+[GitHub][github-idp] is a bit special as it is not an OpenID Connect compliant provider, but
 just a regular OAuth2 provider.

 The [`/user` API endpoint](https://developer.github.com/v3/users/#get-the-authenticated-user)
@@ -242,11 +244,13 @@ oidc_providers:
        display_name_template: "{{ user.name }}"
 ```

-### [Google][google-idp]
+### Google
+
+[Google][google-idp] is an OpenID certified authentication and authorisation provider.

 1. Set up a project in the Google API Console (see
   https://developers.google.com/identity/protocols/oauth2/openid-connect#appsetup).
-2. add an "OAuth Client ID" for a Web Application under "Credentials".
+2. Add an "OAuth Client ID" for a Web Application under "Credentials".
 3. Copy the Client ID and Client Secret, and add the following to your synapse config:
   ```yaml
   oidc_providers:
@@ -446,3 +450,51 @@ The synapse config will look like this:
      config:
        email_template: "{{ user.email }}"
 ```
+
+## Django OAuth Toolkit
+
+[django-oauth-toolkit](https://github.com/jazzband/django-oauth-toolkit) is a
+Django application providing out of the box all the endpoints, data and logic
+needed to add OAuth2 capabilities to your Django projects. It supports
+[OpenID Connect too](https://django-oauth-toolkit.readthedocs.io/en/latest/oidc.html).
+
+Configuration on Django's side:
+
+1. Add an application: https://example.com/admin/oauth2_provider/application/add/ and choose parameters like this:
+* `Redirect uris`: https://synapse.example.com/_synapse/client/oidc/callback
+* `Client type`: `Confidential`
+* `Authorization grant type`: `Authorization code`
+* `Algorithm`: `HMAC with SHA-2 256`
+2. You can [customize the claims](https://django-oauth-toolkit.readthedocs.io/en/latest/oidc.html#customizing-the-oidc-responses) Django gives to synapse (optional):
+   <details>
+    <summary>Code sample</summary>
+
+    ```python
+    class CustomOAuth2Validator(OAuth2Validator):
+
+        def get_additional_claims(self, request):
+            return {
+                "sub": request.user.email,
+                "email": request.user.email,
+                "first_name": request.user.first_name,
+                "last_name": request.user.last_name,
+            }
+    ```
+   </details>
+Your synapse config is then:
+
+```yaml
+oidc_providers:
+  - idp_id: django_example
+    idp_name: "Django Example"
+    issuer: "https://example.com/o/"
+    client_id: "your-client-id"  # CHANGE ME
+    client_secret: "your-client-secret"  # CHANGE ME
+    scopes: ["openid"]
+    user_profile_method: "userinfo_endpoint"  # needed because oauth-toolkit does not include user information in the authorization response
+    user_mapping_provider:
+      config:
+        localpart_template: "{{ user.email.split('@')[0] }}"
+        display_name_template: "{{ user.first_name }} {{ user.last_name }}"
+        email_template: "{{ user.email }}"
+```
@@ -551,6 +551,19 @@ retention:
 #
 #next_link_domain_whitelist: ["matrix.org"]

+# Templates to use when generating email or HTML page contents.
+#
+templates:
+  # Directory in which Synapse will try to find template files to use to generate
+  # email or HTML page contents.
+  # If not set, or a file is not found within the template directory, a default
+  # template from within the Synapse package will be used.
+  #
+  # See https://matrix-org.github.io/synapse/latest/templates.html for more
+  # information about using custom templates.
+  #
+  #custom_template_directory: /path/to/custom/templates/
+

 ## TLS ##

@@ -780,6 +793,8 @@ log_config: "CONFDIR/SERVERNAME.log.config"
 #     is using
 #   - one for registration that ratelimits registration requests based on the
 #     client's IP address.
+#   - one for checking the validity of registration tokens that ratelimits
+#     requests based on the client's IP address.
 #   - one for login that ratelimits login requests based on the client's IP
 #     address.
 #   - one for login that ratelimits login requests based on the account the
@@ -808,6 +823,10 @@ log_config: "CONFDIR/SERVERNAME.log.config"
 #  per_second: 0.17
 #  burst_count: 3
 #
+#rc_registration_token_validity:
+#  per_second: 0.1
+#  burst_count: 5
+#
 #rc_login:
 #  address:
 #    per_second: 0.17
@@ -1156,6 +1175,15 @@ url_preview_accept_language:
 #
 #enable_3pid_lookup: true

+# Require users to submit a token during registration.
+# Tokens can be managed using the admin API:
+# https://matrix-org.github.io/synapse/latest/usage/administration/admin_api/registration_tokens.html
+# Note that `enable_registration` must be set to `true`.
+# Disabling this option will not delete any tokens previously generated.
+# Defaults to false. Uncomment the following to require tokens:
+#
+#registration_requires_token: true
+
 # If set, allows registration of standard or admin accounts by anyone who
 # has the shared secret, even if registration is otherwise disabled.
 #
@@ -1516,7 +1544,9 @@ saml2_config:
  #
  # Default values will be used for the 'entityid' and 'service' settings,
  # so it is not normally necessary to specify them unless you need to
-  # override them.
+  # override them. Note that setting 'service.sp.required_attributes' or
+  # 'service.sp.optional_attributes' here will override anything configured
+  # by a module that registers saml2 user mapping provider callbacks
  #
  sp_config:
    # Point this to the IdP's metadata. You must provide either a local
@@ -1594,18 +1624,14 @@ saml2_config:
  #
  #saml_session_lifetime: 5m

-  # An external module can be provided here as a custom solution to
-  # mapping attributes returned from a saml provider onto a matrix user.
+  # Setting for the default mapping provider which maps attributes returned
+  # from a saml provider onto a matrix user. Custom solutions can be used by
+  # adding a module that provides these features to the 'modules' config
+  # section, in which case the following section will be ignored.
  #
  user_mapping_provider:
-    # The custom module's class. Uncomment to use a custom module.
-    #
-    #module: mapping_provider.SamlMappingProvider
-
    # Custom configuration values for the module. Below options are
-    # intended for the built-in provider, they should be changed if
-    # using a custom module. This section will be passed as a Python
-    # dictionary to the module's `parse_config` method.
+    # intended for the built-in provider.
    #
    config:
      # The SAML attribute (after mapping via the attribute maps) to use
@@ -1881,6 +1907,9 @@ cas_config:
 # Additional settings to use with single-sign on systems such as OpenID Connect,
 # SAML2 and CAS.
 #
+# Server admins can configure custom templates for pages related to SSO. See
+# https://matrix-org.github.io/synapse/latest/templates.html for more information.
+#
 sso:
    # A list of client URLs which are whitelisted so that the user does not
    # have to confirm giving access to their account to the URL. Any client
@@ -1913,169 +1942,6 @@ sso:
    #
    #update_profile_information: true

-    # Directory in which Synapse will try to find the template files below.
-    # If not set, or the files named below are not found within the template
-    # directory, default templates from within the Synapse package will be used.
-    #
-    # Synapse will look for the following templates in this directory:
-    #
-    # * HTML page to prompt the user to choose an Identity Provider during
-    #   login: 'sso_login_idp_picker.html'.
-    #
-    #   This is only used if multiple SSO Identity Providers are configured.
-    #
-    #   When rendering, this template is given the following variables:
-    #     * redirect_url: the URL that the user will be redirected to after
-    #       login.
-    #
-    #     * server_name: the homeserver's name.
-    #
-    #     * providers: a list of available Identity Providers. Each element is
-    #       an object with the following attributes:
-    #
-    #         * idp_id: unique identifier for the IdP
-    #         * idp_name: user-facing name for the IdP
-    #         * idp_icon: if specified in the IdP config, an MXC URI for an icon
-    #              for the IdP
-    #         * idp_brand: if specified in the IdP config, a textual identifier
-    #              for the brand of the IdP
-    #
-    #   The rendered HTML page should contain a form which submits its results
-    #   back as a GET request, with the following query parameters:
-    #
-    #     * redirectUrl: the client redirect URI (ie, the `redirect_url` passed
-    #       to the template)
-    #
-    #     * idp: the 'idp_id' of the chosen IDP.
-    #
-    # * HTML page to prompt new users to enter a userid and confirm other
-    #   details: 'sso_auth_account_details.html'. This is only shown if the
-    #   SSO implementation (with any user_mapping_provider) does not return
-    #   a localpart.
-    #
-    #   When rendering, this template is given the following variables:
-    #
-    #     * server_name: the homeserver's name.
-    #
-    #     * idp: details of the SSO Identity Provider that the user logged in
-    #       with: an object with the following attributes:
-    #
-    #         * idp_id: unique identifier for the IdP
-    #         * idp_name: user-facing name for the IdP
-    #         * idp_icon: if specified in the IdP config, an MXC URI for an icon
-    #              for the IdP
-    #         * idp_brand: if specified in the IdP config, a textual identifier
-    #              for the brand of the IdP
-    #
-    #     * user_attributes: an object containing details about the user that
-    #       we received from the IdP. May have the following attributes:
-    #
-    #         * display_name: the user's display_name
-    #         * emails: a list of email addresses
-    #
-    #   The template should render a form which submits the following fields:
-    #
-    #     * username: the localpart of the user's chosen user id
-    #
-    # * HTML page allowing the user to consent to the server's terms and
-    #   conditions. This is only shown for new users, and only if
-    #   `user_consent.require_at_registration` is set.
-    #
-    #   When rendering, this template is given the following variables:
-    #
-    #     * server_name: the homeserver's name.
-    #
-    #     * user_id: the user's matrix proposed ID.
-    #
-    #     * user_profile.display_name: the user's proposed display name, if any.
-    #
-    #     * consent_version: the version of the terms that the user will be
-    #       shown
-    #
-    #     * terms_url: a link to the page showing the terms.
-    #
-    #   The template should render a form which submits the following fields:
-    #
-    #     * accepted_version: the version of the terms accepted by the user
-    #       (ie, 'consent_version' from the input variables).
-    #
-    # * HTML page for a confirmation step before redirecting back to the client
-    #   with the login token: 'sso_redirect_confirm.html'.
-    #
-    #   When rendering, this template is given the following variables:
-    #
-    #     * redirect_url: the URL the user is about to be redirected to.
-    #
-    #     * display_url: the same as `redirect_url`, but with the query
-    #                    parameters stripped. The intention is to have a
-    #                    human-readable URL to show to users, not to use it as
-    #                    the final address to redirect to.
-    #
-    #     * server_name: the homeserver's name.
-    #
-    #     * new_user: a boolean indicating whether this is the user's first time
-    #          logging in.
-    #
-    #     * user_id: the user's matrix ID.
-    #
-    #     * user_profile.avatar_url: an MXC URI for the user's avatar, if any.
-    #           None if the user has not set an avatar.
-    #
-    #     * user_profile.display_name: the user's display name. None if the user
-    #           has not set a display name.
-    #
-    # * HTML page which notifies the user that they are authenticating to confirm
-    #   an operation on their account during the user interactive authentication
-    #   process: 'sso_auth_confirm.html'.
-    #
-    #   When rendering, this template is given the following variables:
-    #     * redirect_url: the URL the user is about to be redirected to.
-    #
-    #     * description: the operation which the user is being asked to confirm
-    #
-    #     * idp: details of the Identity Provider that we will use to confirm
-    #       the user's identity: an object with the following attributes:
-    #
-    #         * idp_id: unique identifier for the IdP
-    #         * idp_name: user-facing name for the IdP
-    #         * idp_icon: if specified in the IdP config, an MXC URI for an icon
-    #              for the IdP
-    #         * idp_brand: if specified in the IdP config, a textual identifier
-    #              for the brand of the IdP
-    #
-    # * HTML page shown after a successful user interactive authentication session:
-    #   'sso_auth_success.html'.
-    #
-    #   Note that this page must include the JavaScript which notifies of a successful authentication
-    #   (see https://matrix.org/docs/spec/client_server/r0.6.0#fallback).
-    #
-    #   This template has no additional variables.
-    #
-    # * HTML page shown after a user-interactive authentication session which
-    #   does not map correctly onto the expected user: 'sso_auth_bad_user.html'.
-    #
-    #   When rendering, this template is given the following variables:
-    #     * server_name: the homeserver's name.
-    #     * user_id_to_verify: the MXID of the user that we are trying to
-    #       validate.
-    #
-    # * HTML page shown during single sign-on if a deactivated user (according to Synapse's database)
-    #   attempts to login: 'sso_account_deactivated.html'.
-    #
-    #   This template has no additional variables.
-    #
-    # * HTML page to display to users if something goes wrong during the
-    #   OpenID Connect authentication process: 'sso_error.html'.
-    #
-    #   When rendering, this template is given two variables:
-    #     * error: the technical name of the error
-    #     * error_description: a human-readable message for the error
-    #
-    # You can see the default templates at:
-    # https://github.com/matrix-org/synapse/tree/master/synapse/res/templates
-    #
-    #template_dir: "res/templates"
-

 # JSON web token integration. The following settings can be used to make
 # Synapse JSON web tokens for authentication, instead of its internal
@@ -2206,6 +2072,9 @@ ui_auth:

 # Configuration for sending emails from Synapse.
 #
+# Server admins can configure custom templates for email content. See
+# https://matrix-org.github.io/synapse/latest/templates.html for more information.
+#
 email:
  # The hostname of the outgoing SMTP server to use. Defaults to 'localhost'.
  #
@@ -2282,49 +2151,6 @@ email:
  #
  #invite_client_location: https://app.element.io

-  # Directory in which Synapse will try to find the template files below.
-  # If not set, or the files named below are not found within the template
-  # directory, default templates from within the Synapse package will be used.
-  #
-  # Synapse will look for the following templates in this directory:
-  #
-  # * The contents of email notifications of missed events: 'notif_mail.html' and
-  #   'notif_mail.txt'.
-  #
-  # * The contents of account expiry notice emails: 'notice_expiry.html' and
-  #   'notice_expiry.txt'.
-  #
-  # * The contents of password reset emails sent by the homeserver:
-  #   'password_reset.html' and 'password_reset.txt'
-  #
-  # * An HTML page that a user will see when they follow the link in the password
-  #   reset email. The user will be asked to confirm the action before their
-  #   password is reset: 'password_reset_confirmation.html'
-  #
-  # * HTML pages for success and failure that a user will see when they confirm
-  #   the password reset flow using the page above: 'password_reset_success.html'
-  #   and 'password_reset_failure.html'
-  #
-  # * The contents of address verification emails sent during registration:
-  #   'registration.html' and 'registration.txt'
-  #
-  # * HTML pages for success and failure that a user will see when they follow
-  #   the link in an address verification email sent during registration:
-  #   'registration_success.html' and 'registration_failure.html'
-  #
-  # * The contents of address verification emails sent when an address is added
-  #   to a Matrix account: 'add_threepid.html' and 'add_threepid.txt'
-  #
-  # * HTML pages for success and failure that a user will see when they follow
-  #   the link in an address verification email sent when an address is added
-  #   to a Matrix account: 'add_threepid_success.html' and
-  #   'add_threepid_failure.html'
-  #
-  # You can see the default templates at:
-  # https://github.com/matrix-org/synapse/tree/master/synapse/res/templates
-  #
-  #template_dir: "res/templates"
-
  # Subjects to use when sending emails from Synapse.
  #
  # The placeholder '%(app)s' will be replaced with the value of the 'app_name'
@@ -24,18 +24,31 @@ handlers:
        backupCount: 3  # Does not include the current log file.
        encoding: utf8

-    # Default to buffering writes to log file for efficiency. This means that
-    # will be a delay for INFO/DEBUG logs to get written, but WARNING/ERROR
-    # logs will still be flushed immediately.
+    # Default to buffering writes to log file for efficiency.
+    # WARNING/ERROR logs will still be flushed immediately, but there will be a
+    # delay (of up to `period` seconds, or until the buffer is full with
+    # `capacity` messages) before INFO/DEBUG logs get written.
    buffer:
-        class: logging.handlers.MemoryHandler
+        class: synapse.logging.handlers.PeriodicallyFlushingMemoryHandler
        target: file
-        # The capacity is the number of log lines that are buffered before
-        # being written to disk. Increasing this will lead to better
+
+        # The capacity is the maximum number of log lines that are buffered
+        # before being written to disk. Increasing this will lead to better
        # performance, at the expensive of it taking longer for log lines to
        # be written to disk.
+        # This parameter is required.
        capacity: 10
-        flushLevel: 30  # Flush for WARNING logs as well
+
+        # Logs with a level at or above the flush level will cause the buffer to
+        # be flushed immediately.
+        # Default value: 40 (ERROR)
+        # Other values: 50 (CRITICAL), 30 (WARNING), 20 (INFO), 10 (DEBUG)
+        flushLevel: 30  # Flush immediately for WARNING logs and higher
+
+        # The period of time, in seconds, between forced flushes.
+        # Messages will not be delayed for longer than this time.
+        # Default value: 5 seconds
+        period: 5

    # A handler that writes logs to stderr. Unused by default, but can be used
    # instead of "buffer" and "file" in the logger handlers.
@@ -1,3 +1,9 @@
+<h2 style="color:red">
+Parts of this section of the Synapse documentation are now deprecated. For up to date
+documentation on setting up or writing a saml mapping provider module, please 
+see <a href="modules.md">this page</a>.
+</h2>
+
 # SSO Mapping Providers

 A mapping provider is a Python class (loaded via a Python module) that
@@ -0,0 +1,239 @@
+# Templates
+
+Synapse uses parametrised templates to generate the content of emails it sends and
+webpages it shows to users.
+
+By default, Synapse will use the templates listed [here](https://github.com/matrix-org/synapse/tree/master/synapse/res/templates).
+Server admins can configure an additional directory for Synapse to look for templates
+in, allowing them to specify custom templates:
+
+```yaml
+templates:
+  custom_templates_directory: /path/to/custom/templates/
+```
+
+If this setting is not set, or the files named below are not found within the directory,
+default templates from within the Synapse package will be used.
+
+Templates that are given variables when being rendered are rendered using [Jinja 2](https://jinja.palletsprojects.com/en/2.11.x/).
+Templates rendered by Jinja 2 can also access two functions on top of the functions
+already available as part of Jinja 2:
+
+```python
+format_ts(value: int, format: str) -> str
+```
+
+Formats a timestamp in milliseconds.
+
+Example: `reason.last_sent_ts|format_ts("%c")`
+
+```python
+mxc_to_http(value: str, width: int, height: int, resize_method: str = "crop") -> str
+```
+
+Turns a `mxc://` URL for media content into an HTTP(S) one using the homeserver's
+`public_baseurl` configuration setting as the URL's base.
+
+Example: `message.sender_avatar_url|mxc_to_http(32,32)`
+
+
+## Email templates
+
+Below are the templates Synapse will look for when generating the content of an email:
+
+* `notif_mail.html` and `notif_mail.txt`: The contents of email notifications of missed
+  events.
+  When rendering, this template is given the following variables:
+    * `user_display_name`: the display name for the user receiving the notification
+    * `unsubscribe_link`: the link users can click to unsubscribe from email notifications
+    * `summary_text`: a summary of the notification(s). The text used can be customised
+      by configuring the various settings in the `email.subjects` section of the
+      configuration file.
+    * `rooms`: a list of rooms containing events to include in the email. Each element is
+      an object with the following attributes:
+        * `title`: a human-readable name for the room
+        * `hash`: a hash of the ID of the room
+        * `invite`: a boolean, which is `True` if the room is an invite the user hasn't
+          accepted yet, `False` otherwise
+        * `notifs`: a list of events, or an empty list if `invite` is `True`. Each element
+          is an object with the following attributes:
+            * `link`: a `matrix.to` link to the event
+            * `ts`: the time in milliseconds at which the event was received
+            * `messages`: a list of messages containing one message before the event, the
+              message in the event, and one message after the event. Each element is an
+              object with the following attributes:
+                * `event_type`: the type of the event
+                * `is_historical`: a boolean, which is `False` if the message is the one
+                  that triggered the notification, `True` otherwise
+                * `id`: the ID of the event
+                * `ts`: the time in milliseconds at which the event was sent
+                * `sender_name`: the display name for the event's sender
+                * `sender_avatar_url`: the avatar URL (as a `mxc://` URL) for the event's
+                  sender
+                * `sender_hash`: a hash of the user ID of the sender
+        * `link`: a `matrix.to` link to the room
+    * `reason`: information on the event that triggered the email to be sent. It's an
+      object with the following attributes:
+        * `room_id`: the ID of the room the event was sent in
+        * `room_name`: a human-readable name for the room the event was sent in
+        * `now`: the current time in milliseconds
+        * `received_at`: the time in milliseconds at which the event was received
+        * `delay_before_mail_ms`: the amount of time in milliseconds Synapse always waits
+          before ever emailing about a notification (to give the user a chance to respond
+          to other push or notice the window)
+        * `last_sent_ts`: the time in milliseconds at which a notification was last sent
+          for an event in this room
+        * `throttle_ms`: the minimum amount of time in milliseconds between two
+          notifications can be sent for this room
+* `password_reset.html` and `password_reset.txt`: The contents of password reset emails
+  sent by the homeserver.
+  When rendering, these templates are given a `link` variable which contains the link the
+  user must click in order to reset their password.
+* `registration.html` and `registration.txt`: The contents of address verification emails
+  sent during registration.
+  When rendering, these templates are given a `link` variable which contains the link the
+  user must click in order to validate their email address.
+* `add_threepid.html` and `add_threepid.txt`: The contents of address verification emails
+  sent when an address is added to a Matrix account.
+  When rendering, these templates are given a `link` variable which contains the link the
+  user must click in order to validate their email address.
+
+
+## HTML page templates for registration and password reset
+
+Below are the templates Synapse will look for when generating pages related to
+registration and password reset:
+
+* `password_reset_confirmation.html`: An HTML page that a user will see when they follow
+  the link in the password reset email. The user will be asked to confirm the action
+  before their password is reset.
+  When rendering, this template is given the following variables:
+    * `sid`: the session ID for the password reset
+    * `token`: the token for the password reset
+    * `client_secret`: the client secret for the password reset
+* `password_reset_success.html` and `password_reset_failure.html`: HTML pages for success
+  and failure that a user will see when they confirm the password reset flow using the
+  page above.
+  When rendering, `password_reset_success.html` is given no variable, and
+  `password_reset_failure.html` is given a `failure_reason`, which contains the reason
+  for the password reset failure. 
+* `registration_success.html` and `registration_failure.html`: HTML pages for success and
+  failure that a user will see when they follow the link in an address verification email
+  sent during registration.
+  When rendering, `registration_success.html` is given no variable, and
+  `registration_failure.html` is given a `failure_reason`, which contains the reason
+  for the registration failure.
+* `add_threepid_success.html` and `add_threepid_failure.html`: HTML pages for success and
+  failure that a user will see when they follow the link in an address verification email
+  sent when an address is added to a Matrix account.
+  When rendering, `add_threepid_success.html` is given no variable, and
+  `add_threepid_failure.html` is given a `failure_reason`, which contains the reason
+  for the registration failure.
+
+
+## HTML page templates for Single Sign-On (SSO)
+
+Below are the templates Synapse will look for when generating pages related to SSO:
+
+* `sso_login_idp_picker.html`: HTML page to prompt the user to choose an
+  Identity Provider during login.
+  This is only used if multiple SSO Identity Providers are configured.
+  When rendering, this template is given the following variables:
+    * `redirect_url`: the URL that the user will be redirected to after
+      login.
+    * `server_name`: the homeserver's name.
+    * `providers`: a list of available Identity Providers. Each element is
+      an object with the following attributes:
+        * `idp_id`: unique identifier for the IdP
+        * `idp_name`: user-facing name for the IdP
+        * `idp_icon`: if specified in the IdP config, an MXC URI for an icon
+             for the IdP
+        * `idp_brand`: if specified in the IdP config, a textual identifier
+             for the brand of the IdP
+  The rendered HTML page should contain a form which submits its results
+  back as a GET request, with the following query parameters:
+    * `redirectUrl`: the client redirect URI (ie, the `redirect_url` passed
+      to the template)
+    * `idp`: the 'idp_id' of the chosen IDP.
+* `sso_auth_account_details.html`: HTML page to prompt new users to enter a
+  userid and confirm other details. This is only shown if the
+  SSO implementation (with any `user_mapping_provider`) does not return
+  a localpart.
+  When rendering, this template is given the following variables:
+    * `server_name`: the homeserver's name.
+    * `idp`: details of the SSO Identity Provider that the user logged in
+      with: an object with the following attributes:
+        * `idp_id`: unique identifier for the IdP
+        * `idp_name`: user-facing name for the IdP
+        * `idp_icon`: if specified in the IdP config, an MXC URI for an icon
+             for the IdP
+        * `idp_brand`: if specified in the IdP config, a textual identifier
+             for the brand of the IdP
+    * `user_attributes`: an object containing details about the user that
+      we received from the IdP. May have the following attributes:
+        * display_name: the user's display_name
+        * emails: a list of email addresses
+  The template should render a form which submits the following fields:
+    * `username`: the localpart of the user's chosen user id
+* `sso_new_user_consent.html`: HTML page allowing the user to consent to the
+  server's terms and conditions. This is only shown for new users, and only if
+  `user_consent.require_at_registration` is set.
+  When rendering, this template is given the following variables:
+    * `server_name`: the homeserver's name.
+    * `user_id`: the user's matrix proposed ID.
+    * `user_profile.display_name`: the user's proposed display name, if any.
+    * consent_version: the version of the terms that the user will be
+      shown
+    * `terms_url`: a link to the page showing the terms.
+  The template should render a form which submits the following fields:
+    * `accepted_version`: the version of the terms accepted by the user
+      (ie, 'consent_version' from the input variables).
+* `sso_redirect_confirm.html`: HTML page for a confirmation step before redirecting back
+  to the client with the login token.
+  When rendering, this template is given the following variables:
+    * `redirect_url`: the URL the user is about to be redirected to.
+    * `display_url`: the same as `redirect_url`, but with the query
+                   parameters stripped. The intention is to have a
+                   human-readable URL to show to users, not to use it as
+                   the final address to redirect to.
+    * `server_name`: the homeserver's name.
+    * `new_user`: a boolean indicating whether this is the user's first time
+         logging in.
+    * `user_id`: the user's matrix ID.
+    * `user_profile.avatar_url`: an MXC URI for the user's avatar, if any.
+          `None` if the user has not set an avatar.
+    * `user_profile.display_name`: the user's display name. `None` if the user
+          has not set a display name.
+* `sso_auth_confirm.html`: HTML page which notifies the user that they are authenticating
+  to confirm an operation on their account during the user interactive authentication
+  process.
+  When rendering, this template is given the following variables:
+    * `redirect_url`: the URL the user is about to be redirected to.
+    * `description`: the operation which the user is being asked to confirm
+    * `idp`: details of the Identity Provider that we will use to confirm
+      the user's identity: an object with the following attributes:
+        * `idp_id`: unique identifier for the IdP
+        * `idp_name`: user-facing name for the IdP
+        * `idp_icon`: if specified in the IdP config, an MXC URI for an icon
+             for the IdP
+        * `idp_brand`: if specified in the IdP config, a textual identifier
+             for the brand of the IdP
+* `sso_auth_success.html`: HTML page shown after a successful user interactive
+  authentication session.
+  Note that this page must include the JavaScript which notifies of a successful
+  authentication (see https://matrix.org/docs/spec/client_server/r0.6.0#fallback).
+  This template has no additional variables.
+* `sso_auth_bad_user.html`: HTML page shown after a user-interactive authentication
+  session which does not map correctly onto the expected user.
+  When rendering, this template is given the following variables:
+    * `server_name`: the homeserver's name.
+    * `user_id_to_verify`: the MXID of the user that we are trying to
+      validate.
+* `sso_account_deactivated.html`: HTML page shown during single sign-on if a deactivated
+  user (according to Synapse's database) attempts to login.
+  This template has no additional variables.
+* `sso_error.html`: HTML page to display to users if something goes wrong during the
+  OpenID Connect authentication process.
+  When rendering, this template is given two variables:
+    * `error`: the technical name of the error
+    * `error_description`: a human-readable message for the error
@@ -95,11 +95,20 @@ The following admin APIs were deprecated in [Synapse 1.25](https://github.com/ma
 -   `POST /_synapse/admin/v1/purge_room`
 -   `POST /_synapse/admin/v1/shutdown_room/<room_id>`

-Any scripts still using the above APIs should be converted to use the 
+Any scripts still using the above APIs should be converted to use the
 [Delete Room API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#delete-room-api).

+## User-interactive authentication fallback templates can now display errors

-# Upgrading to v1.xx.0
+This may affect you if you make use of custom HTML templates for the
+[reCAPTCHA](../synapse/res/templates/recaptcha.html) or
+[terms](../synapse/res/templates/terms.html) fallback pages.
+
+The template is now provided an `error` variable if the authentication
+process failed. See the default templates linked above for an example.
+
+
+# Upgrading to v1.41.0

 ## Add support for routing outbound HTTP requests via a proxy for federation

@@ -125,14 +134,23 @@ environment variable.
 See [using a forward proxy with Synapse documentation](setup/forward_proxy.md) for
 details.

-## User-interactive authentication fallback templates can now display errors
+## Deprecation of `template_dir`

-This may affect you if you make use of custom HTML templates for the
-[reCAPTCHA](../synapse/res/templates/recaptcha.html) or
-[terms](../synapse/res/templates/terms.html) fallback pages.
+The `template_dir` settings in the `sso`, `account_validity` and `email` sections of the
+configuration file are now deprecated. Server admins should use the new
+`templates.custom_template_directory` setting in the configuration file and use one single
+custom template directory for all aforementioned features. Template file names remain
+unchanged. See [the related documentation](https://matrix-org.github.io/synapse/latest/templates.html)
+for more information and examples.

-The template is now provided an `error` variable if the authentication
-process failed. See the default templates linked above for an example.
+We plan to remove support for these settings in October 2021.
+
+## `/_synapse/admin/v1/users/{userId}/media` must be handled by media workers
+
+The [media repository worker documentation](https://matrix-org.github.io/synapse/latest/workers.html#synapseappmedia_repository)
+has been updated to reflect that calls to `/_synapse/admin/v1/users/{userId}/media`
+must now be handled by media repository workers. This is due to the new `DELETE` method
+of this endpoint modifying the media store.

 # Upgrading to v1.39.0

@@ -0,0 +1,295 @@
+# Registration Tokens
+
+This API allows you to manage tokens which can be used to authenticate
+registration requests, as proposed in [MSC3231](https://github.com/govynnus/matrix-doc/blob/token-registration/proposals/3231-token-authenticated-registration.md).
+To use it, you will need to enable the `registration_requires_token` config
+option, and authenticate by providing an `access_token` for a server admin:
+see [Admin API](../../usage/administration/admin_api).
+Note that this API is still experimental; not all clients may support it yet.
+
+
+## Registration token objects
+
+Most endpoints make use of JSON objects that contain details about tokens.
+These objects have the following fields:
+- `token`: The token which can be used to authenticate registration.
+- `uses_allowed`: The number of times the token can be used to complete a
+  registration before it becomes invalid.
+- `pending`: The number of pending uses the token has. When someone uses
+  the token to authenticate themselves, the pending counter is incremented
+  so that the token is not used more than the permitted number of times.
+  When the person completes registration the pending counter is decremented,
+  and the completed counter is incremented.
+- `completed`: The number of times the token has been used to successfully
+  complete a registration.
+- `expiry_time`: The latest time the token is valid. Given as the number of
+  milliseconds since 1970-01-01 00:00:00 UTC (the start of the Unix epoch).
+  To convert this into a human-readable form you can remove the milliseconds
+  and use the `date` command. For example, `date -d '@1625394937'`.
+
+
+## List all tokens
+
+Lists all tokens and details about them. If the request is successful, the top
+level JSON object will have a `registration_tokens` key which is an array of
+registration token objects.
+
+```
+GET /_synapse/admin/v1/registration_tokens
+```
+
+Optional query parameters:
+- `valid`: `true` or `false`. If `true`, only valid tokens are returned.
+  If `false`, only tokens that have expired or have had all uses exhausted are
+  returned. If omitted, all tokens are returned regardless of validity.
+
+Example:
+
+```
+GET /_synapse/admin/v1/registration_tokens
+```
+```
+200 OK
+
+{
+    "registration_tokens": [
+        {
+            "token": "abcd",
+            "uses_allowed": 3,
+            "pending": 0,
+            "completed": 1,
+            "expiry_time": null
+        },
+        {
+            "token": "pqrs",
+            "uses_allowed": 2,
+            "pending": 1,
+            "completed": 1,
+            "expiry_time": null
+        },
+        {
+            "token": "wxyz",
+            "uses_allowed": null,
+            "pending": 0,
+            "completed": 9,
+            "expiry_time": 1625394937000    // 2021-07-04 10:35:37 UTC
+        }
+    ]
+}
+```
+
+Example using the `valid` query parameter:
+
+```
+GET /_synapse/admin/v1/registration_tokens?valid=false
+```
+```
+200 OK
+
+{
+    "registration_tokens": [
+        {
+            "token": "pqrs",
+            "uses_allowed": 2,
+            "pending": 1,
+            "completed": 1,
+            "expiry_time": null
+        },
+        {
+            "token": "wxyz",
+            "uses_allowed": null,
+            "pending": 0,
+            "completed": 9,
+            "expiry_time": 1625394937000    // 2021-07-04 10:35:37 UTC
+        }
+    ]
+}
+```
+
+
+## Get one token
+
+Get details about a single token. If the request is successful, the response
+body will be a registration token object.
+
+```
+GET /_synapse/admin/v1/registration_tokens/<token>
+```
+
+Path parameters:
+- `token`: The registration token to return details of.
+
+Example:
+
+```
+GET /_synapse/admin/v1/registration_tokens/abcd
+```
+```
+200 OK
+
+{
+    "token": "abcd",
+    "uses_allowed": 3,
+    "pending": 0,
+    "completed": 1,
+    "expiry_time": null
+}
+```
+
+
+## Create token
+
+Create a new registration token. If the request is successful, the newly created
+token will be returned as a registration token object in the response body.
+
+```
+POST /_synapse/admin/v1/registration_tokens/new
+```
+
+The request body must be a JSON object and can contain the following fields:
+- `token`: The registration token. A string of no more than 64 characters that
+  consists only of characters matched by the regex `[A-Za-z0-9-_]`.
+  Default: randomly generated.
+- `uses_allowed`: The integer number of times the token can be used to complete
+  a registration before it becomes invalid.
+  Default: `null` (unlimited uses).
+- `expiry_time`: The latest time the token is valid. Given as the number of
+  milliseconds since 1970-01-01 00:00:00 UTC (the start of the Unix epoch).
+  You could use, for example, `date '+%s000' -d 'tomorrow'`.
+  Default: `null` (token does not expire).
+- `length`: The length of the token randomly generated if `token` is not
+  specified. Must be between 1 and 64 inclusive. Default: `16`.
+
+If a field is omitted the default is used.
+
+Example using defaults:
+
+```
+POST /_synapse/admin/v1/registration_tokens/new
+
+{}
+```
+```
+200 OK
+
+{
+    "token": "0M-9jbkf2t_Tgiw1",
+    "uses_allowed": null,
+    "pending": 0,
+    "completed": 0,
+    "expiry_time": null
+}
+```
+
+Example specifying some fields:
+
+```
+POST /_synapse/admin/v1/registration_tokens/new
+
+{
+    "token": "defg",
+    "uses_allowed": 1
+}
+```
+```
+200 OK
+
+{
+    "token": "defg",
+    "uses_allowed": 1,
+    "pending": 0,
+    "completed": 0,
+    "expiry_time": null
+}
+```
+
+
+## Update token
+
+Update the number of allowed uses or expiry time of a token. If the request is
+successful, the updated token will be returned as a registration token object
+in the response body.
+
+```
+PUT /_synapse/admin/v1/registration_tokens/<token>
+```
+
+Path parameters:
+- `token`: The registration token to update.
+
+The request body must be a JSON object and can contain the following fields:
+- `uses_allowed`: The integer number of times the token can be used to complete
+  a registration before it becomes invalid. By setting `uses_allowed` to `0`
+  the token can be easily made invalid without deleting it.
+  If `null` the token will have an unlimited number of uses.
+- `expiry_time`: The latest time the token is valid. Given as the number of
+  milliseconds since 1970-01-01 00:00:00 UTC (the start of the Unix epoch).
+  If `null` the token will not expire.
+
+If a field is omitted its value is not modified.
+
+Example:
+
+```
+PUT /_synapse/admin/v1/registration_tokens/defg
+
+{
+    "expiry_time": 4781243146000    // 2121-07-06 11:05:46 UTC
+}
+```
+```
+200 OK
+
+{
+    "token": "defg",
+    "uses_allowed": 1,
+    "pending": 0,
+    "completed": 0,
+    "expiry_time": 4781243146000
+}
+```
+
+
+## Delete token
+
+Delete a registration token. If the request is successful, the response body
+will be an empty JSON object.
+
+```
+DELETE /_synapse/admin/v1/registration_tokens/<token>
+```
+
+Path parameters:
+- `token`: The registration token to delete.
+
+Example:
+
+```
+DELETE /_synapse/admin/v1/registration_tokens/wxyz
+```
+```
+200 OK
+
+{}
+```
+
+
+## Errors
+
+If a request fails a "standard error response" will be returned as defined in
+the [Matrix Client-Server API specification](https://matrix.org/docs/spec/client_server/r0.6.1#api-standards).
+
+For example, if the token specified in a path parameter does not exist a
+`404 Not Found` error will be returned.
+
+```
+GET /_synapse/admin/v1/registration_tokens/1234
+```
+```
+404 Not Found
+
+{
+    "errcode": "M_NOT_FOUND",
+    "error": "No such registration token: 1234"
+}
+```
@@ -214,6 +214,7 @@ expressions:
    ^/_matrix/federation/v1/send/

    # Client API requests
+    ^/_matrix/client/(api/v1|r0|unstable)/createRoom$
    ^/_matrix/client/(api/v1|r0|unstable)/publicRooms$
    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/joined_members$
    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/context/.*$
@@ -235,6 +236,7 @@ expressions:
    # Registration/login requests
    ^/_matrix/client/(api/v1|r0|unstable)/login$
    ^/_matrix/client/(r0|unstable)/register$
+    ^/_matrix/client/unstable/org.matrix.msc3231/register/org.matrix.msc3231.login.registration_token/validity$

    # Event sending requests
    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/redact
@@ -425,10 +427,12 @@ Handles the media repository. It can handle all endpoints starting with:
    ^/_synapse/admin/v1/user/.*/media.*$
    ^/_synapse/admin/v1/media/.*$
    ^/_synapse/admin/v1/quarantine_media/.*$
+    ^/_synapse/admin/v1/users/.*/media$

 You should also set `enable_media_repo: False` in the shared configuration
 file to stop the main synapse running background jobs related to managing the
-media repository.
+media repository. Note that doing so will prevent the main process from being
+able to handle the above endpoints.

 In the `media_repository` worker configuration file, configure the http listener to
 expose the `media` resource. For example:
@@ -28,10 +28,13 @@ files =
  synapse/federation,
  synapse/groups,
  synapse/handlers,
+  synapse/http/additional_resource.py,
  synapse/http/client.py,
  synapse/http/federation/matrix_federation_agent.py,
+  synapse/http/federation/srv_resolver.py,
  synapse/http/federation/well_known_resolver.py,
  synapse/http/matrixfederationclient.py,
+  synapse/http/proxyagent.py,
  synapse/http/servlet.py,
  synapse/http/server.py,
  synapse/http/site.py,
@@ -54,6 +57,7 @@ files =
  synapse/storage/databases/main/keys.py,
  synapse/storage/databases/main/pusher.py,
  synapse/storage/databases/main/registration.py,
+  synapse/storage/databases/main/session.py,
  synapse/storage/databases/main/stream.py,
  synapse/storage/databases/main/ui_auth.py,
  synapse/storage/database.py,
@@ -87,8 +91,9 @@ files =
  tests/test_utils,
  tests/handlers/test_password_providers.py,
  tests/handlers/test_room_summary.py,
-  tests/rest/client/v1/test_login.py,
-  tests/rest/client/v2_alpha/test_auth.py,
+  tests/handlers/test_sync.py,
+  tests/rest/client/test_login.py,
+  tests/rest/client/test_auth.py,
  tests/util/test_itertools.py,
  tests/util/test_stream_change_cache.py

@@ -20,8 +20,9 @@ from concurrent.futures import ThreadPoolExecutor
 from typing import Optional, Sequence

 DISTS = (
-    "debian:buster",
+    "debian:buster",  # oldstable: EOL 2022-08
    "debian:bullseye",
+    "debian:bookworm",
    "debian:sid",
    "ubuntu:bionic",  # 18.04 LTS (our EOL forced by Py36 on 2021-12-23)
    "ubuntu:focal",  # 20.04 LTS (our EOL forced by Py38 on 2024-10-14)
@@ -47,7 +47,7 @@ try:
 except ImportError:
    pass

-__version__ = "1.40.0"
+__version__ = "1.41.0rc1"

 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
    # We import here so that we don't have to install a bunch of deps when
@@ -79,6 +79,7 @@ class LoginType:
    TERMS = "m.login.terms"
    SSO = "m.login.sso"
    DUMMY = "m.login.dummy"
+    REGISTRATION_TOKEN = "org.matrix.msc3231.login.registration_token"


 # This is used in the `type` parameter for /register when called by
@@ -147,6 +147,14 @@ class SynapseError(CodeMessageException):
        return cs_error(self.msg, self.errcode)


+class InvalidAPICallError(SynapseError):
+    """You called an existing API endpoint, but fed that endpoint
+    invalid or incomplete data."""
+
+    def __init__(self, msg: str):
+        super().__init__(HTTPStatus.BAD_REQUEST, msg, Codes.BAD_JSON)
+
+
 class ProxiedRequestError(SynapseError):
    """An error from a general matrix endpoint, eg. from a proxied Matrix API call.

@@ -372,6 +372,16 @@ async def start(hs: "HomeServer"):
    load_legacy_spam_checkers(hs)
    load_legacy_third_party_event_rules(hs)
    load_legacy_presence_router(hs)
+    # 'module_has_registered' is true if a module calls 'register_saml2_user_mapping_provider_callbacks'
+    # Only one mapping provider can be set, so only load default (or legacy configured one) if this is
+    # still false
+    if (
+        hs.config.saml2.saml2_enabled
+        and not hs.get_saml2_user_mapping_provider().module_has_registered
+    ):
+        from synapse.handlers.saml import load_default_or_legacy_saml2_mapping_provider
+
+        load_default_or_legacy_saml2_mapping_provider(hs)

    # If we've configured an expiry time for caches, start the background job now.
    setup_expire_lru_cache_entries(hs)
@@ -38,7 +38,6 @@ from synapse.replication.slave.storage.groups import SlavedGroupServerStore
 from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
 from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
 from synapse.replication.slave.storage.registration import SlavedRegistrationStore
-from synapse.replication.slave.storage.room import RoomStore
 from synapse.server import HomeServer
 from synapse.util.logcontext import LoggingContext
 from synapse.util.versionstring import get_version_string
@@ -58,7 +57,6 @@ class AdminCmdSlavedStore(
    SlavedPushRuleStore,
    SlavedEventStore,
    SlavedClientIpStore,
-    RoomStore,
    BaseSlavedStore,
 ):
    pass
@@ -64,7 +64,6 @@ from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
 from synapse.replication.slave.storage.pushers import SlavedPusherStore
 from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
 from synapse.replication.slave.storage.registration import SlavedRegistrationStore
-from synapse.replication.slave.storage.room import RoomStore
 from synapse.rest.admin import register_servlets_for_media_repo
 from synapse.rest.client import (
    account_data,
@@ -96,7 +95,10 @@ from synapse.rest.client.profile import (
    ProfileRestServlet,
 )
 from synapse.rest.client.push_rule import PushRuleRestServlet
-from synapse.rest.client.register import RegisterRestServlet
+from synapse.rest.client.register import (
+    RegisterRestServlet,
+    RegistrationTokenValidityRestServlet,
+)
 from synapse.rest.client.sendtodevice import SendToDeviceRestServlet
 from synapse.rest.client.versions import VersionsRestServlet
 from synapse.rest.client.voip import VoipRestServlet
@@ -114,7 +116,9 @@ from synapse.storage.databases.main.monthly_active_users import (
    MonthlyActiveUsersWorkerStore,
 )
 from synapse.storage.databases.main.presence import PresenceStore
+from synapse.storage.databases.main.room import RoomWorkerStore
 from synapse.storage.databases.main.search import SearchStore
+from synapse.storage.databases.main.session import SessionStore
 from synapse.storage.databases.main.stats import StatsStore
 from synapse.storage.databases.main.transactions import TransactionWorkerStore
 from synapse.storage.databases.main.ui_auth import UIAuthWorkerStore
@@ -237,7 +241,7 @@ class GenericWorkerSlavedStore(
    ClientIpWorkerStore,
    SlavedEventStore,
    SlavedKeyStore,
-    RoomStore,
+    RoomWorkerStore,
    DirectoryStore,
    SlavedApplicationServiceStore,
    SlavedRegistrationStore,
@@ -250,6 +254,7 @@ class GenericWorkerSlavedStore(
    SearchStore,
    TransactionWorkerStore,
    LockStore,
+    SessionStore,
    BaseSlavedStore,
 ):
    pass
@@ -279,6 +284,7 @@ class GenericWorkerServer(HomeServer):
                    resource = JsonResource(self, canonical_json=False)

                    RegisterRestServlet(self).register(resource)
+                    RegistrationTokenValidityRestServlet(self).register(resource)
                    login.register_servlets(self, resource)
                    ThreepidRestServlet(self).register(resource)
                    DevicesRestServlet(self).register(resource)
@@ -78,6 +78,11 @@ class AccountValidityConfig(Config):
        )

        # Read and store template content
+        custom_template_directories = (
+            self.root.server.custom_template_directory,
+            account_validity_template_dir,
+        )
+
        (
            self.account_validity_account_renewed_template,
            self.account_validity_account_previously_renewed_template,
@@ -88,5 +93,5 @@ class AccountValidityConfig(Config):
                "account_previously_renewed.html",
                invalid_token_template_filename,
            ],
-            (td for td in (account_validity_template_dir,) if td),
+            (td for td in custom_template_directories if td),
        )
@@ -258,7 +258,12 @@ class EmailConfig(Config):
                    add_threepid_template_success_html,
                ],
                (
-                    td for td in (template_dir,) if td
+                    td
+                    for td in (
+                        self.root.server.custom_template_directory,
+                        template_dir,
+                    )
+                    if td
                ),  # Filter out template_dir if not provided
            )

@@ -299,7 +304,14 @@ class EmailConfig(Config):
                self.email_notif_template_text,
            ) = self.read_templates(
                [notif_template_html, notif_template_text],
-                (td for td in (template_dir,) if td),
+                (
+                    td
+                    for td in (
+                        self.root.server.custom_template_directory,
+                        template_dir,
+                    )
+                    if td
+                ),  # Filter out template_dir if not provided
            )

            self.email_notif_for_new_users = email_config.get(
@@ -322,7 +334,14 @@ class EmailConfig(Config):
                self.account_validity_template_text,
            ) = self.read_templates(
                [expiry_template_html, expiry_template_text],
-                (td for td in (template_dir,) if td),
+                (
+                    td
+                    for td in (
+                        self.root.server.custom_template_directory,
+                        template_dir,
+                    )
+                    if td
+                ),  # Filter out template_dir if not provided
            )

        subjects_config = email_config.get("subjects", {})
@@ -354,6 +373,9 @@ class EmailConfig(Config):
            """\
        # Configuration for sending emails from Synapse.
        #
+        # Server admins can configure custom templates for email content. See
+        # https://matrix-org.github.io/synapse/latest/templates.html for more information.
+        #
        email:
          # The hostname of the outgoing SMTP server to use. Defaults to 'localhost'.
          #
@@ -430,49 +452,6 @@ class EmailConfig(Config):
          #
          #invite_client_location: https://app.element.io

-          # Directory in which Synapse will try to find the template files below.
-          # If not set, or the files named below are not found within the template
-          # directory, default templates from within the Synapse package will be used.
-          #
-          # Synapse will look for the following templates in this directory:
-          #
-          # * The contents of email notifications of missed events: 'notif_mail.html' and
-          #   'notif_mail.txt'.
-          #
-          # * The contents of account expiry notice emails: 'notice_expiry.html' and
-          #   'notice_expiry.txt'.
-          #
-          # * The contents of password reset emails sent by the homeserver:
-          #   'password_reset.html' and 'password_reset.txt'
-          #
-          # * An HTML page that a user will see when they follow the link in the password
-          #   reset email. The user will be asked to confirm the action before their
-          #   password is reset: 'password_reset_confirmation.html'
-          #
-          # * HTML pages for success and failure that a user will see when they confirm
-          #   the password reset flow using the page above: 'password_reset_success.html'
-          #   and 'password_reset_failure.html'
-          #
-          # * The contents of address verification emails sent during registration:
-          #   'registration.html' and 'registration.txt'
-          #
-          # * HTML pages for success and failure that a user will see when they follow
-          #   the link in an address verification email sent during registration:
-          #   'registration_success.html' and 'registration_failure.html'
-          #
-          # * The contents of address verification emails sent when an address is added
-          #   to a Matrix account: 'add_threepid.html' and 'add_threepid.txt'
-          #
-          # * HTML pages for success and failure that a user will see when they follow
-          #   the link in an address verification email sent when an address is added
-          #   to a Matrix account: 'add_threepid_success.html' and
-          #   'add_threepid_failure.html'
-          #
-          # You can see the default templates at:
-          # https://github.com/matrix-org/synapse/tree/master/synapse/res/templates
-          #
-          #template_dir: "res/templates"
-
          # Subjects to use when sending emails from Synapse.
          #
          # The placeholder '%%(app)s' will be replaced with the value of the 'app_name'
@@ -39,5 +39,8 @@ class ExperimentalConfig(Config):
        # MSC3244 (room version capabilities)
        self.msc3244_enabled: bool = experimental.get("msc3244_enabled", False)

+        # MSC3283 (set displayname, avatar_url and change 3pid capabilities)
+        self.msc3283_enabled: bool = experimental.get("msc3283_enabled", False)
+
        # MSC3266 (room summary api)
        self.msc3266_enabled: bool = experimental.get("msc3266_enabled", False)
@@ -67,18 +67,31 @@ handlers:
        backupCount: 3  # Does not include the current log file.
        encoding: utf8

-    # Default to buffering writes to log file for efficiency. This means that
-    # will be a delay for INFO/DEBUG logs to get written, but WARNING/ERROR
-    # logs will still be flushed immediately.
+    # Default to buffering writes to log file for efficiency.
+    # WARNING/ERROR logs will still be flushed immediately, but there will be a
+    # delay (of up to `period` seconds, or until the buffer is full with
+    # `capacity` messages) before INFO/DEBUG logs get written.
    buffer:
-        class: logging.handlers.MemoryHandler
+        class: synapse.logging.handlers.PeriodicallyFlushingMemoryHandler
        target: file
-        # The capacity is the number of log lines that are buffered before
-        # being written to disk. Increasing this will lead to better
+
+        # The capacity is the maximum number of log lines that are buffered
+        # before being written to disk. Increasing this will lead to better
        # performance, at the expensive of it taking longer for log lines to
        # be written to disk.
+        # This parameter is required.
        capacity: 10
-        flushLevel: 30  # Flush for WARNING logs as well
+
+        # Logs with a level at or above the flush level will cause the buffer to
+        # be flushed immediately.
+        # Default value: 40 (ERROR)
+        # Other values: 50 (CRITICAL), 30 (WARNING), 20 (INFO), 10 (DEBUG)
+        flushLevel: 30  # Flush immediately for WARNING logs and higher
+
+        # The period of time, in seconds, between forced flushes.
+        # Messages will not be delayed for longer than this time.
+        # Default value: 5 seconds
+        period: 5

    # A handler that writes logs to stderr. Unused by default, but can be used
    # instead of "buffer" and "file" in the logger handlers.
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Azrenbeth	862d820d44	moved more saml imports behind checks (tried everything with pysaml2 uninstalled so should now work!)	2021-08-25 11:49:21 +01:00
Azrenbeth	e7d6c061e4	Added ANOTHER check for saml being enabled before loading it	2021-08-25 11:22:04 +01:00
Azrenbeth	08d386a37a	no longer assert default was loaded, and better test for if using default	2021-08-25 10:48:47 +01:00
Azrenbeth	d9e48a4caa	Don't import saml stuff if saml not enabled	2021-08-24 16:42:03 +01:00
Azrenbeth	057628b018	Added (unneccesary?) parentheses to try and make python3.6 happy	2021-08-24 16:29:07 +01:00
Azrenbeth	d1a0a27056	created changelog	2021-08-24 14:38:22 +01:00
Azrenbeth	a0ace792e0	Wrote docs for saml user mapping provider callbacks	2021-08-24 14:38:22 +01:00
Azrenbeth	162738feb6	Updated tests to use new module system	2021-08-24 14:38:22 +01:00
Azrenbeth	2b3e4e856f	Port the saml mapping providers to new module interface	2021-08-24 14:38:22 +01:00
Patrick Cloke	d12ba52f17	Persist room hierarchy pagination sessions to the database. (#10613 )	2021-08-24 08:14:03 -04:00
Richard van der Hoff	15db8b7c7f	Correctly initialise the `synapse_user_logins` metric. (#10677 ) Fix a bug where the prometheus metrics for SSO logins wouldn't be initialised until the first user logged in with a given auth provider.	2021-08-24 09:17:51 +00:00
Hugo DELVAL	86415f162d	doc: add django-oauth-toolkit to oidc doc (#10192 ) Signed-off-by: Hugo Delval <hugo.delval@gmail.com>	2021-08-23 17:12:36 +00:00
Azrenbeth	0c1d6f65d7	Enforce the max length for per-room display names / avatar URLs. (#10654 ) To match the maximum lengths allowed for profile data.	2021-08-23 11:25:33 -04:00
Andrew Morgan	3e83f97154	Fix the titles in the OIDC documentation (#10639 ) * Fix the titles in the OIDC documentation Having them as links broke the table-of-contents rendering in mdbook. Plus there's no reason for only some of the provider titles to be links. * Changelog * Add link to google idp docs	2021-08-23 14:58:31 +01:00
Dan Callahan	2efc838f05	Avoid duplicate issues from Twisted trunk failures (#10672 ) Setting `update_existing: true` in the `create-an-issue` GitHub Action will avoid opening duplicate issues if an open issue already exists with an identical title. If no open issues match the title, then a new issue will be created. This helps avoid spamming our issue tracker should there be a failure when testing against Twisted's trunk. This PR also pins the SHA of the `create-an-issue` action to mitigate the risk of a malicious actor gaining access to JasonEtco's account. See GitHub's page on security hardening third party actions for more: https://docs.github.com/en/actions/learn-github-actions/security-hardening-for-github-actions#using-third-party-actions Signed-off-by: Dan Callahan <danc@element.io>	2021-08-23 13:06:49 +00:00
Patrick Cloke	bd7d398b05	Additional type hints for the sync REST servlet. (#10666 )	2021-08-23 08:14:42 -04:00
Patrick Cloke	2af6d31b78	Addtional type hints for the REST servlets. (#10665 )	2021-08-23 08:14:17 -04:00
Patrick Cloke	31dac7ffee	Do not include stack traces for known exceptions when trying multiple federation destinations. (#10662 )	2021-08-23 08:00:25 -04:00
Richard van der Hoff	4db65f911a	Run a nightly CI build against Twisted trunk. (#10651 ) This creates a GHA workflow which runs at 8am every day, and runs mypy, trial and sytest against Twisted's current trunk. If any of the jobs fail, it opens an issue.	2021-08-23 11:12:45 +01:00
Callum Brown	947dbbdfd1	Implement MSC3231: Token authenticated registration (#10142 ) Signed-off-by: Callum Brown <callum@calcuode.com> This is part of my GSoC project implementing [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231).	2021-08-21 22:14:43 +01:00
David Robertson	ecd823d766	Flatten tests/rest/client/{v1,v2_alpha} too (#10667 )	2021-08-20 17:50:44 +01:00
Dirk Klimpel	f499dc38bc	Simplify tests for the device admin rest API. (#10664 ) By replacing duplicated code with parameterized tests and avoiding unnecessary dumping of JSON data.	2021-08-20 15:43:26 +00:00
David Robertson	7862d704fd	Follow-up: format changelog, add licence (#10593 ) Merged before approval; these comments from @clokep on that PR.	2021-08-20 16:33:52 +01:00
David Robertson	ee3b2ac59a	Validate device_keys for C-S /keys/query requests (#10593 ) * Validate device_keys for C-S /keys/query requests Closes #10354 A small, not particularly critical fix. I'm interested in seeing if we can find a more systematic approach though. #8445 is the place for any discussion.	2021-08-20 15:47:03 +01:00
Richard van der Hoff	e81d62009e	Split `on_receive_pdu` in half (#10640 ) Here we split on_receive_pdu into two functions (on_receive_pdu and process_pulled_event), rather than having both cases in the same method. There's a tiny bit of overlap, but not that much.	2021-08-19 17:05:12 +00:00
Richard van der Hoff	50af1efe4b	Extract `_resolve_state_at_missing_prevs` (#10624 ) This is a follow-up to #10615: it takes the code that constructs the state at a backwards extremity, and extracts it to a separate method.	2021-08-19 17:31:40 +01:00
Patrick Cloke	000aa89be6	Do not include rooms with an unknown room version in a sync response. (#10644 ) A user will still see this room if it is in a local cache, but it will not reappear if clearing the cache and reloading.	2021-08-19 11:12:55 -04:00
Dirk Klimpel	b5fef6054a	Support MSC3283: Expose `enable_set_displayname` in capabilities (#10452 )	2021-08-19 09:40:40 +00:00
Dirk Klimpel	220f901229	Remove not needed database updates in modify user admin API (#10627 )	2021-08-19 10:25:05 +01:00
Dirk Klimpel	0c3565da4c	Additional type hints for the proxy agent and SRV resolver modules. (#10608 )	2021-08-18 13:53:20 -04:00
Erik Johnston	78a70a2e0b	Merge branch 'release-v1.41' into develop	2021-08-18 17:02:47 +01:00
Erik Johnston	b9c35586a4	Update docs/upgrade.md with new version	2021-08-18 16:59:36 +01:00
Patrick Cloke	d9856d9150	Fix weakref_slot parameter for room member storage attrs. (#10642 ) Follow-up to #10629 which set it to true, not false.	2021-08-18 15:00:37 +00:00
Erik Johnston	e328d8ffd9	Update changelog	2021-08-18 15:56:32 +01:00
Erik Johnston	49cb7eae97	1.41.0rc1	2021-08-18 15:52:11 +01:00
Andrew Morgan	3692f7fd33	Mount /_synapse/admin/v1/users/{userId}/media admin API on media workers only (#10628 ) Co-authored-by: Patrick Cloke <patrickc@matrix.org>	2021-08-18 13:25:12 +01:00
Richard van der Hoff	eea2873595	fix broken link to upgrade notes (#10631 )	2021-08-18 12:38:37 +01:00
Erik Johnston	5581dd7bf7	Allow modules to run looping call on all instances (#10638 ) By default the calls only ran on the worker configured to run background tasks.	2021-08-18 10:21:11 +00:00
Andrew Morgan	703e3a9e85	Allow /createRoom to be run on workers (#10564 ) Fixes https://github.com/matrix-org/synapse/issues/7867	2021-08-17 14:33:16 +01:00
Andrew Morgan	84469bdac7	Remove the unused public_room_list_stream (#10565 ) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>	2021-08-17 14:02:50 +01:00
Richard van der Hoff	c8132f4a31	Build debs for bookworm (#10612 )	2021-08-17 13:48:59 +01:00
Brendan Abolivier	5639759980	Centralise the custom template directory (#10596 ) Several configuration sections are using separate settings for custom template directories, which can be confusing. This PR adds a new top-level configuration for a custom template directory which is then used for every module. The only exception is the consent templates, since the consent template directory require a specific hierarchy, so it's probably better that it stays separate from everything else.	2021-08-17 14:45:24 +02:00
Patrick Cloke	c4cf0c0473	Attempt to pull from the legacy spaces summary API over federation. (#10583 ) If the new /hierarchy API does not exist on all destinations, fallback to querying the /spaces API and translating the results. This is a backwards compatibility hack since not all of the federated homeservers will update at the same time.	2021-08-17 08:19:12 -04:00
reivilibre	5f7b1e1f27	Make `PeriodicallyFlushingMemoryHandler` the default logging handler. (#10518 )	2021-08-17 13:13:11 +01:00
				`@@ -1 +0,0 @@`
				`Improve event caching mechanism to avoid having multiple copies of an event in memory at a time.`
				`@@ -1 +0,0 @@`
				`Add some clarification to the sample config file. Contributed by @Kentokamoto.`
				`@@ -0,0 +1 @@`
				`Add support for [MSC3231 - Token authenticated registration](https://github.com/matrix-org/matrix-doc/pull/3231). Users can be required to submit a token during registration to authenticate themselves. Contributed by Callum Brown.`
				`@@ -0,0 +1 @@`
				`Add documentation on how to connect Django with synapse using oidc and django-oauth-toolkit. Contributed by @HugoDelval.`
				`@@ -1 +0,0 @@`
				Initial local support for [MSC3266](https://github.com/matrix-org/synapse/pull/10394), Room Summary over the unstable `/rooms/{roomIdOrAlias}/summary` API.
				`@@ -1 +0,0 @@`
				Experimental support for [MSC3288](https://github.com/matrix-org/matrix-doc/pull/3288), sending `room_type` to the identity server for 3pid invites over the `/store-invite` API.
				`@@ -1 +0,0 @@`
				`Add documentation for configuration a forward proxy.`
				`@@ -0,0 +1 @@`
				`Add support for [MSC3283](https://github.com/matrix-org/matrix-doc/pull/3283): Expose enable_set_displayname in capabilities.`
				`@@ -1 +0,0 @@`
				`Add support for sending federation requests through a proxy. Contributed by @Bubu and @dklimpel.`