WIP: Type TreeCache

Comment TreeCache
Comment LruCache
2022-07-18 13:05:07 +01:00 · 2022-07-18 10:49:12 +01:00 · 2022-07-18 10:48:24 +01:00 · 2022-07-18 10:40:36 +01:00 · 2022-07-17 11:34:34 +01:00 · 2022-07-15 16:52:16 +01:00
172 changed files with 3229 additions and 2025 deletions
--- a/.editorconfig
+++ b/.editorconfig
@@ -7,3 +7,4 @@ root = true
 [*.py]
 indent_style = space
 indent_size = 4
+max_line_length = 88
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -328,9 +328,6 @@ jobs:
          - arrangement: monolith
            database: Postgres

-          - arrangement: workers
-            database: Postgres
-
    steps:
      - name: Run actions/checkout@v2 for synapse
        uses: actions/checkout@v2
@@ -346,6 +343,30 @@ jobs:
        shell: bash
        name: Run Complement Tests

+  # XXX When complement with workers is stable, move this back into the standard
+  #     "complement" matrix above.
+  #
+  # See https://github.com/matrix-org/synapse/issues/13161
+  complement-workers:
+    if: "${{ !failure() && !cancelled() }}"
+    needs: linting-done
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Run actions/checkout@v2 for synapse
+        uses: actions/checkout@v2
+        with:
+          path: synapse
+
+      - name: Prepare Complement's Prerequisites
+        run: synapse/.ci/scripts/setup_complement_prerequisites.sh
+
+      - run: |
+          set -o pipefail
+          POSTGRES=1 WORKERS=1 COMPLEMENT_DIR=`pwd`/complement synapse/scripts-dev/complement.sh -json 2>&1 | gotestfmt
+        shell: bash
+        name: Run Complement Tests
+
  # a job which marks all the other jobs as complete, thus allowing PRs to be merged.
  tests-done:
    if: ${{ always() }}
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,3 +1,83 @@
+Synapse vNext
+=============
+
+As of this release, Synapse no longer allows the tasks of verifying email address ownership, and password reset confirmation, to be delegated to an identity server. For more information, see the [upgrade notes](https://matrix-org.github.io/synapse/v1.64/upgrade.html#upgrading-to-v1640).
+
+Synapse 1.63.0rc1 (2022-07-12)
+==============================
+
+Features
+--------
+
+- Add a rate limit for local users sending invites. ([\#13125](https://github.com/matrix-org/synapse/issues/13125))
+- Implement [MSC3827](https://github.com/matrix-org/matrix-spec-proposals/pull/3827): Filtering of `/publicRooms` by room type. ([\#13031](https://github.com/matrix-org/synapse/issues/13031))
+- Improve validation logic in Synapse's REST endpoints. ([\#13148](https://github.com/matrix-org/synapse/issues/13148))
+
+
+Bugfixes
+--------
+
+- Fix a long-standing bug where application services were not able to join remote federated rooms without a profile. ([\#13131](https://github.com/matrix-org/synapse/issues/13131))
+- Fix a long-standing bug where `_get_state_map_for_room` might raise errors when third party event rules callbacks are present. ([\#13174](https://github.com/matrix-org/synapse/issues/13174))
+- Fix a long-standing bug where the `synapse_port_db` script could fail to copy rows with negative row ids. ([\#13226](https://github.com/matrix-org/synapse/issues/13226))
+- Fix a bug introduced in 1.54.0 where appservices would not receive room-less EDUs, like presence, when both [MSC2409](https://github.com/matrix-org/matrix-spec-proposals/pull/2409) and [MSC3202](https://github.com/matrix-org/matrix-spec-proposals/pull/3202) are enabled. ([\#13236](https://github.com/matrix-org/synapse/issues/13236))
+- Fix a bug introduced in 1.62.0 where rows were not deleted from `event_push_actions` table on large servers. ([\#13194](https://github.com/matrix-org/synapse/issues/13194))
+- Fix a bug introduced in 1.62.0 where notification counts would get stuck after a highlighted message. ([\#13223](https://github.com/matrix-org/synapse/issues/13223))
+- Fix exception when using experimental [MSC3030](https://github.com/matrix-org/matrix-spec-proposals/pull/3030) `/timestamp_to_event` endpoint to look for remote federated imported events before room creation. ([\#13197](https://github.com/matrix-org/synapse/issues/13197))
+- Fix [MSC3202](https://github.com/matrix-org/matrix-spec-proposals/pull/3202)-enabled appservices not receiving to-device messages, preventing messages from being decrypted. ([\#13235](https://github.com/matrix-org/synapse/issues/13235))
+
+
+Updates to the Docker image
+---------------------------
+
+- Bump the version of `lxml` in matrix.org Docker images Debian packages from 4.8.0 to 4.9.1. ([\#13207](https://github.com/matrix-org/synapse/issues/13207))
+
+
+Improved Documentation
+----------------------
+
+- Add an explanation of the `--report-stats` argument to the docs. ([\#13029](https://github.com/matrix-org/synapse/issues/13029))
+- Add a helpful example bash script to the contrib directory for creating multiple worker configuration files of the same type. Contributed by @villepeh. ([\#13032](https://github.com/matrix-org/synapse/issues/13032))
+- Add missing links to config options. ([\#13166](https://github.com/matrix-org/synapse/issues/13166))
+- Add documentation for anonymised homeserver statistics collection. ([\#13086](https://github.com/matrix-org/synapse/issues/13086))
+- Add documentation for the existing `databases` option in the homeserver configuration manual. ([\#13212](https://github.com/matrix-org/synapse/issues/13212))
+- Clean up references to sample configuration and redirect users to the configuration manual instead. ([\#13077](https://github.com/matrix-org/synapse/issues/13077), [\#13139](https://github.com/matrix-org/synapse/issues/13139))
+- Document how the Synapse team does reviews. ([\#13132](https://github.com/matrix-org/synapse/issues/13132))
+- Fix wrong section header for `allow_public_rooms_over_federation` in the homeserver config documentation. ([\#13116](https://github.com/matrix-org/synapse/issues/13116))
+
+
+Deprecations and Removals
+-------------------------
+
+- Remove obsolete and for 8 years unused `RoomEventsStoreTestCase`. Contributed by @arkamar. ([\#13200](https://github.com/matrix-org/synapse/issues/13200))
+
+
+Internal Changes
+----------------
+
+- Add type annotations to `synapse.logging`, `tests.server` and `tests.utils`. ([\#13028](https://github.com/matrix-org/synapse/issues/13028), [\#13103](https://github.com/matrix-org/synapse/issues/13103), [\#13159](https://github.com/matrix-org/synapse/issues/13159), [\#13136](https://github.com/matrix-org/synapse/issues/13136))
+- Enforce type annotations for `tests.test_server`. ([\#13135](https://github.com/matrix-org/synapse/issues/13135))
+- Support temporary experimental return values for spam checker module callbacks. ([\#13044](https://github.com/matrix-org/synapse/issues/13044))
+- Add support to `complement.sh` for skipping the docker build. ([\#13143](https://github.com/matrix-org/synapse/issues/13143), [\#13158](https://github.com/matrix-org/synapse/issues/13158))
+- Add support to `complement.sh` for setting the log level using the `SYNAPSE_TEST_LOG_LEVEL` environment variable. ([\#13152](https://github.com/matrix-org/synapse/issues/13152))
+- Enable Complement testing in the 'Twisted Trunk' CI runs. ([\#13079](https://github.com/matrix-org/synapse/issues/13079), [\#13157](https://github.com/matrix-org/synapse/issues/13157))
+- Improve startup times in Complement test runs against workers, particularly in CPU-constrained environments. ([\#13127](https://github.com/matrix-org/synapse/issues/13127))
+- Update config used by Complement to allow device name lookup over federation. ([\#13167](https://github.com/matrix-org/synapse/issues/13167))
+- Faster room joins: handle race between persisting an event and un-partial stating a room. ([\#13100](https://github.com/matrix-org/synapse/issues/13100))
+- Faster room joins: fix race in recalculation of current room state. ([\#13151](https://github.com/matrix-org/synapse/issues/13151))
+- Faster room joins: skip waiting for full state when processing incoming events over federation. ([\#13144](https://github.com/matrix-org/synapse/issues/13144))
+- Raise a `DependencyError` on missing dependencies instead of a `ConfigError`. ([\#13113](https://github.com/matrix-org/synapse/issues/13113))
+- Avoid stripping line breaks from SQL sent to the database. ([\#13129](https://github.com/matrix-org/synapse/issues/13129))
+- Apply ratelimiting earlier in processing of `/send` requests. ([\#13134](https://github.com/matrix-org/synapse/issues/13134))
+- Improve exception handling when processing events received over federation. ([\#13145](https://github.com/matrix-org/synapse/issues/13145))
+- Check that `auto_vacuum` is disabled when porting a SQLite database to Postgres, as `VACUUM`s must not be performed between runs of the script. ([\#13195](https://github.com/matrix-org/synapse/issues/13195))
+- Reduce DB usage of `/sync` when a large number of unread messages have recently been sent in a room. ([\#13119](https://github.com/matrix-org/synapse/issues/13119), [\#13153](https://github.com/matrix-org/synapse/issues/13153))
+- Reduce memory consumption when processing incoming events in large rooms. ([\#13078](https://github.com/matrix-org/synapse/issues/13078), [\#13222](https://github.com/matrix-org/synapse/issues/13222))
+- Reduce number of queries used to get profile information. Contributed by Nick @ Beeper (@fizzadar). ([\#13209](https://github.com/matrix-org/synapse/issues/13209))
+- Reduce number of events queried during room creation. Contributed by Nick @ Beeper (@fizzadar). ([\#13210](https://github.com/matrix-org/synapse/issues/13210))
+- More aggressively rotate push actions. ([\#13211](https://github.com/matrix-org/synapse/issues/13211))
+- Add `max_line_length` setting for Python files to the `.editorconfig`. Contributed by @sumnerevans @ Beeper. ([\#13228](https://github.com/matrix-org/synapse/issues/13228))
+
 Synapse 1.62.0 (2022-07-05)
 ===========================

@@ -5,7 +85,6 @@ No significant changes since 1.62.0rc3.

 Authors of spam-checker plugins should consult the [upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.62/docs/upgrade.md#upgrading-to-v1620) to learn about the enriched signatures for spam checker callbacks, which are supported with this release of Synapse.

-
 Synapse 1.62.0rc3 (2022-07-04)
 ==============================

--- a/changelog.d/12943.misc
+++ b/changelog.d/12943.misc
@@ -0,0 +1 @@
+Remove code which incorrectly attempted to reconcile state with remote servers when processing incoming events.
--- a/changelog.d/12967.removal
+++ b/changelog.d/12967.removal
@@ -0,0 +1 @@
+Drop tables used for groups/communities.
--- a/changelog.d/13028.misc
+++ b/changelog.d/13028.misc
@@ -1 +0,0 @@
-Add type annotations to `tests.utils`.
--- a/changelog.d/13029.doc
+++ b/changelog.d/13029.doc
@@ -1 +0,0 @@
-Add an explanation of the `--report-stats` argument to the docs.
--- a/changelog.d/13031.feature
+++ b/changelog.d/13031.feature
@@ -1 +0,0 @@
-Implement [MSC3827](https://github.com/matrix-org/matrix-spec-proposals/pull/3827): Filtering of /publicRooms by room type.
--- a/changelog.d/13077.doc
+++ b/changelog.d/13077.doc
@@ -1,3 +0,0 @@
-Clean up references to sample configuration and redirect users to the configuration manual instead.
-
-
--- a/changelog.d/13079.misc
+++ b/changelog.d/13079.misc
@@ -1 +0,0 @@
-Enable Complement testing in the 'Twisted Trunk' CI runs.
--- a/changelog.d/13086.doc
+++ b/changelog.d/13086.doc
@@ -1 +0,0 @@
-Add documentation for anonymised homeserver statistics collection.
--- a/changelog.d/13094.misc
+++ b/changelog.d/13094.misc
@@ -0,0 +1 @@
+Make the AS login method call `Auth.get_user_by_req` for checking the AS token.
--- a/changelog.d/13100.misc
+++ b/changelog.d/13100.misc
@@ -1 +0,0 @@
-Faster room joins: Handle race between persisting an event and un-partial stating a room.
--- a/changelog.d/13103.misc
+++ b/changelog.d/13103.misc
@@ -1 +0,0 @@
-Add missing type hints to `synapse.logging`.
--- a/changelog.d/13113.misc
+++ b/changelog.d/13113.misc
@@ -1 +0,0 @@
-Raise a `DependencyError` on missing dependencies instead of a `ConfigError`.
--- a/changelog.d/13116.doc
+++ b/changelog.d/13116.doc
@@ -1 +0,0 @@
-Fix wrong section header for `allow_public_rooms_over_federation` in the homeserver config documentation.
--- a/changelog.d/13119.misc
+++ b/changelog.d/13119.misc
@@ -1 +0,0 @@
-Reduce DB usage of `/sync` when a large number of unread messages have recently been sent in a room.
--- a/changelog.d/13125.feature
+++ b/changelog.d/13125.feature
@@ -1 +0,0 @@
-Add a rate limit for local users sending invites.
--- a/changelog.d/13127.misc
+++ b/changelog.d/13127.misc
@@ -1 +0,0 @@
-Improve startup times in Complement test runs against workers, particularly in CPU-constrained environments.
--- a/changelog.d/13129.misc
+++ b/changelog.d/13129.misc
@@ -1 +0,0 @@
-Only one-line SQL statements for logging and tracing.
--- a/changelog.d/13131.bugfix
+++ b/changelog.d/13131.bugfix
@@ -1 +0,0 @@
-Fix application service not being able to join remote federated room without a profile set.
--- a/changelog.d/13132.doc
+++ b/changelog.d/13132.doc
@@ -1 +0,0 @@
-Document how the Synapse team does reviews.
--- a/changelog.d/13134.misc
+++ b/changelog.d/13134.misc
@@ -1 +0,0 @@
-Apply ratelimiting earlier in processing of /send request.
--- a/changelog.d/13135.misc
+++ b/changelog.d/13135.misc
@@ -1 +0,0 @@
-Enforce type annotations for `tests.test_server`.
--- a/changelog.d/13136.misc
+++ b/changelog.d/13136.misc
@@ -1 +0,0 @@
-Add type annotations to `tests.server`.
--- a/changelog.d/13139.doc
+++ b/changelog.d/13139.doc
@@ -1 +0,0 @@
-Add a link to the configuration manual from the homeserver sample config documentation.
--- a/changelog.d/13143.misc
+++ b/changelog.d/13143.misc
@@ -1 +0,0 @@
-Add support to `complement.sh` for skipping the docker build.
--- a/changelog.d/13144.misc
+++ b/changelog.d/13144.misc
@@ -1 +0,0 @@
-Faster joins: skip waiting for full state when processing incoming events over federation.
--- a/changelog.d/13145.misc
+++ b/changelog.d/13145.misc
@@ -1 +0,0 @@
-Improve exception handling when processing events received over federation.
--- a/changelog.d/13148.feature
+++ b/changelog.d/13148.feature
@@ -1 +0,0 @@
-Improve validation logic in Synapse's REST endpoints.
--- a/changelog.d/13151.misc
+++ b/changelog.d/13151.misc
@@ -1 +0,0 @@
-Faster room joins: fix race in recalculation of current room state.
--- a/changelog.d/13152.misc
+++ b/changelog.d/13152.misc
@@ -1 +0,0 @@
-Add the ability to set the log level using the `SYNAPSE_TEST_LOG_LEVEL` environment when using `complement.sh`.
--- a/changelog.d/13153.misc
+++ b/changelog.d/13153.misc
@@ -1 +0,0 @@
-Reduce DB usage of `/sync` when a large number of unread messages have recently been sent in a room.
--- a/changelog.d/13157.misc
+++ b/changelog.d/13157.misc
@@ -1 +0,0 @@
-Enable Complement testing in the 'Twisted Trunk' CI runs.
--- a/changelog.d/13158.misc
+++ b/changelog.d/13158.misc
@@ -1 +0,0 @@
-Add support to `complement.sh` for skipping the docker build.
--- a/changelog.d/13159.misc
+++ b/changelog.d/13159.misc
@@ -1 +0,0 @@
-Improve and fix type hints.
--- a/changelog.d/13166.doc
+++ b/changelog.d/13166.doc
@@ -1 +0,0 @@
-Add missing links to config options.
--- a/changelog.d/13167.misc
+++ b/changelog.d/13167.misc
@@ -1 +0,0 @@
-Update config used by Complement to allow device name lookup over federation.
--- a/changelog.d/13174.bugfix
+++ b/changelog.d/13174.bugfix
@@ -1 +0,0 @@
-Make use of the more robust `get_current_state` in `_get_state_map_for_room` to avoid breakages.
--- a/changelog.d/13175.misc
+++ b/changelog.d/13175.misc
@@ -0,0 +1 @@
+Add prometheus counters for ephemeral events and to device messages pushed to app services. Contributed by Brad @ Beeper.
--- a/changelog.d/13192.removal
+++ b/changelog.d/13192.removal
@@ -0,0 +1 @@
+Drop support for delegating email verification to an external server.
--- a/changelog.d/13194.bugfix
+++ b/changelog.d/13194.bugfix
@@ -1 +0,0 @@
-Fix bug where rows were not deleted from `event_push_actions` table on large servers. Introduced in v1.62.0.
--- a/changelog.d/13195.misc
+++ b/changelog.d/13195.misc
@@ -1 +0,0 @@
-Check that `auto_vacuum` is disabled when porting a SQLite database to Postgres, as `VACUUM`s must not be performed between runs of the script.
--- a/changelog.d/13198.misc
+++ b/changelog.d/13198.misc
@@ -0,0 +1 @@
+Refactor receipts servlet logic to avoid duplicated code.
--- a/changelog.d/13200.removal
+++ b/changelog.d/13200.removal
@@ -1 +0,0 @@
-Remove obsolete and for 8 years unused `RoomEventsStoreTestCase`. Contributed by @arkamar.
--- a/changelog.d/13207.docker
+++ b/changelog.d/13207.docker
@@ -1 +0,0 @@
-Bump the version of `lxml` in matrix.org Docker images Debian packages from 4.8.0 to 4.9.1.
--- a/changelog.d/13208.feature
+++ b/changelog.d/13208.feature
@@ -0,0 +1 @@
+Add a `room_type` field in the responses for the list room and room details admin API. Contributed by @andrewdoh.
--- a/changelog.d/13209.misc
+++ b/changelog.d/13209.misc
@@ -1 +0,0 @@
-Reduce number of queries used to get profile information. Contributed by Nick @ Beeper (@fizzadar).
--- a/changelog.d/13215.misc
+++ b/changelog.d/13215.misc
@@ -0,0 +1 @@
+Preparation for database schema simplifications: populate `state_key` and `rejection_reason` for existing rows in the `events` table.
--- a/changelog.d/13218.misc
+++ b/changelog.d/13218.misc
@@ -0,0 +1 @@
+Remove unused database table `event_reference_hashes`.
--- a/changelog.d/13220.feature
+++ b/changelog.d/13220.feature
@@ -0,0 +1 @@
+Add support for room version 10.
--- a/changelog.d/13224.misc
+++ b/changelog.d/13224.misc
@@ -0,0 +1 @@
+Further reduce queries used sending events when creating new rooms. Contributed by Nick @ Beeper (@fizzadar).
--- a/changelog.d/13231.doc
+++ b/changelog.d/13231.doc
@@ -0,0 +1 @@
+Provide an example of using the Admin API. Contributed by @jejo86.
--- a/changelog.d/13233.doc
+++ b/changelog.d/13233.doc
@@ -0,0 +1 @@
+Move the documentation for how URL previews work to the URL preview module.
--- a/changelog.d/13239.removal
+++ b/changelog.d/13239.removal
@@ -0,0 +1 @@
+Drop support for calling `/_matrix/client/v3/account/3pid/bind` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu.
--- a/changelog.d/13240.misc
+++ b/changelog.d/13240.misc
@@ -0,0 +1 @@
+Call the v2 identity service `/3pid/unbind` endpoint, rather than v1.
--- a/changelog.d/13242.misc
+++ b/changelog.d/13242.misc
@@ -0,0 +1 @@
+Use an asynchronous cache wrapper for the get event cache. Contributed by Nick @ Beeper (@fizzadar).
--- a/changelog.d/13251.misc
+++ b/changelog.d/13251.misc
@@ -0,0 +1 @@
+Optimise federation sender and appservice pusher event stream processing queries. Contributed by Nick @ Beeper (@fizzadar).
--- a/changelog.d/13253.misc
+++ b/changelog.d/13253.misc
@@ -0,0 +1 @@
+Preparatory work for a per-room rate limiter on joins.
--- a/changelog.d/13254.misc
+++ b/changelog.d/13254.misc
@@ -0,0 +1 @@
+Preparatory work for a per-room rate limiter on joins.
--- a/changelog.d/13255.misc
+++ b/changelog.d/13255.misc
@@ -0,0 +1 @@
+Preparatory work for a per-room rate limiter on joins.
--- a/changelog.d/13257.misc
+++ b/changelog.d/13257.misc
@@ -0,0 +1 @@
+Log the stack when waiting for an entire room to be un-partial stated.
--- a/changelog.d/13260.misc
+++ b/changelog.d/13260.misc
@@ -0,0 +1 @@
+Clean-up tests for notifications.
--- a/changelog.d/13261.doc
+++ b/changelog.d/13261.doc
@@ -0,0 +1 @@
+Move the documentation for how URL previews work to the URL preview module.
--- a/changelog.d/13263.bugfix
+++ b/changelog.d/13263.bugfix
@@ -0,0 +1 @@
+Fix a bug introduced in Synapse 1.15.0 where adding a user through the Synapse Admin API with a phone number would fail if the "enable_email_notifs" and "email_notifs_for_new_users" options were enabled. Contributed by @thomasweston12.
--- a/changelog.d/13266.misc
+++ b/changelog.d/13266.misc
@@ -0,0 +1 @@
+Do not fail build if complement with workers fails.
--- a/changelog.d/13267.misc
+++ b/changelog.d/13267.misc
@@ -0,0 +1 @@
+Don't pull out state in `compute_event_context` for unconflicted state.
--- a/changelog.d/13270.bugfix
+++ b/changelog.d/13270.bugfix
@@ -0,0 +1 @@
+Fix a bug introduced in Synapse 1.40 where a user invited to a restricted room would be briefly unable to join.
--- a/changelog.d/13274.misc
+++ b/changelog.d/13274.misc
@@ -0,0 +1 @@
+Don't pull out state in `compute_event_context` for unconflicted state.
--- a/changelog.d/13278.bugfix
+++ b/changelog.d/13278.bugfix
@@ -0,0 +1 @@
+Fix long-standing bug where in rare instances Synapse could store the incorrect state for a room after a state resolution.
--- a/changelog.d/13279.misc
+++ b/changelog.d/13279.misc
@@ -0,0 +1 @@
+Reduce the rebuild time for the complement-synapse docker image.
--- a/changelog.d/13284.misc
+++ b/changelog.d/13284.misc
@@ -0,0 +1 @@
+Update locked version of `frozendict` to 2.3.2, which has a fix for a memory leak.
--- a/changelog.d/13292.misc
+++ b/changelog.d/13292.misc
@@ -0,0 +1 @@
+Make `DictionaryCache` expire full entries if they haven't been queried in a while, even if specific keys have been queried recently.
--- a/contrib/workers-bash-scripts/create-multiple-workers.md
+++ b/contrib/workers-bash-scripts/create-multiple-workers.md
@@ -0,0 +1,31 @@
+# Creating multiple workers with a bash script
+
+Setting up multiple worker configuration files manually can be time-consuming.
+You can alternatively create multiple worker configuration files with a simple `bash` script. For example:
+
+```sh
+#!/bin/bash
+for i in {1..5}
+do
+cat << EOF >> generic_worker$i.yaml
+worker_app: synapse.app.generic_worker
+worker_name: generic_worker$i
+
+# The replication listener on the main synapse process.
+worker_replication_host: 127.0.0.1
+worker_replication_http_port: 9093
+
+worker_listeners:
+  - type: http
+    port: 808$i
+    resources:
+      - names: [client, federation]
+
+worker_log_config: /etc/matrix-synapse/generic-worker-log.yaml
+EOF
+done
+```
+
+This would create five generic workers with a unique `worker_name` field in each file and listening on ports 8081-8085.
+
+Customise the script to your needs.
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+matrix-synapse-py3 (1.63.0~rc1) stable; urgency=medium
+
+  * New Synapse release 1.63.0rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 12 Jul 2022 11:26:02 +0100
+
 matrix-synapse-py3 (1.62.0) stable; urgency=medium

  * New Synapse release 1.62.0.
--- a/docker/complement/Dockerfile
+++ b/docker/complement/Dockerfile
@@ -4,42 +4,58 @@
 #
 # Instructions for building this image from those it depends on is detailed in this guide:
 # https://github.com/matrix-org/synapse/blob/develop/docker/README-testing.md#testing-with-postgresql-and-single-or-multi-process-synapse
+
 ARG SYNAPSE_VERSION=latest
+
+# first of all, we create a base image with a postgres server and database,
+# which we can copy into the target image. For repeated rebuilds, this is
+# much faster than apt installing postgres each time.
+#
+# This trick only works because (a) the Synapse image happens to have all the
+# shared libraries that postgres wants, (b) we use a postgres image based on
+# the same debian version as Synapse's docker image (so the versions of the
+# shared libraries match).
+
+FROM postgres:13-bullseye AS postgres_base
+    # initialise the database cluster in /var/lib/postgresql
+    RUN gosu postgres initdb --locale=C --encoding=UTF-8 --auth-host password
+
+    # Configure a password and create a database for Synapse
+    RUN echo "ALTER USER postgres PASSWORD 'somesecret'" | gosu postgres postgres --single
+    RUN echo "CREATE DATABASE synapse" | gosu postgres postgres --single
+
+# now build the final image, based on the Synapse image.
+
 FROM matrixdotorg/synapse-workers:$SYNAPSE_VERSION
+    # copy the postgres installation over from the image we built above
+    RUN adduser --system --uid 999 postgres --home /var/lib/postgresql
+    COPY --from=postgres_base /var/lib/postgresql /var/lib/postgresql
+    COPY --from=postgres_base /usr/lib/postgresql /usr/lib/postgresql
+    COPY --from=postgres_base /usr/share/postgresql /usr/share/postgresql
+    RUN mkdir /var/run/postgresql && chown postgres /var/run/postgresql
+    ENV PATH="${PATH}:/usr/lib/postgresql/13/bin"
+    ENV PGDATA=/var/lib/postgresql/data

-# Install postgresql
-RUN apt-get update && \
-  DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -yqq postgresql-13
+    # Extend the shared homeserver config to disable rate-limiting,
+    # set Complement's static shared secret, enable registration, amongst other
+    # tweaks to get Synapse ready for testing.
+    # To do this, we copy the old template out of the way and then include it
+    # with Jinja2.
+    RUN mv /conf/shared.yaml.j2 /conf/shared-orig.yaml.j2
+    COPY conf/workers-shared-extra.yaml.j2 /conf/shared.yaml.j2

-# Configure a user and create a database for Synapse
-RUN pg_ctlcluster 13 main start &&  su postgres -c "echo \
- \"ALTER USER postgres PASSWORD 'somesecret'; \
- CREATE DATABASE synapse \
-  ENCODING 'UTF8' \
-  LC_COLLATE='C' \
-  LC_CTYPE='C' \
-  template=template0;\" | psql" && pg_ctlcluster 13 main stop
+    WORKDIR /data

-# Extend the shared homeserver config to disable rate-limiting,
-# set Complement's static shared secret, enable registration, amongst other
-# tweaks to get Synapse ready for testing.
-# To do this, we copy the old template out of the way and then include it
-# with Jinja2.
-RUN mv /conf/shared.yaml.j2 /conf/shared-orig.yaml.j2
-COPY conf/workers-shared-extra.yaml.j2 /conf/shared.yaml.j2
+    COPY conf/postgres.supervisord.conf /etc/supervisor/conf.d/postgres.conf

-WORKDIR /data
+    # Copy the entrypoint
+    COPY conf/start_for_complement.sh /

-COPY conf/postgres.supervisord.conf /etc/supervisor/conf.d/postgres.conf
+    # Expose nginx's listener ports
+    EXPOSE 8008 8448

-# Copy the entrypoint
-COPY conf/start_for_complement.sh /
+    ENTRYPOINT ["/start_for_complement.sh"]

-# Expose nginx's listener ports
-EXPOSE 8008 8448
-
-ENTRYPOINT ["/start_for_complement.sh"]
-
-# Update the healthcheck to have a shorter check interval
-HEALTHCHECK --start-period=5s --interval=1s --timeout=1s \
-    CMD /bin/sh /healthcheck.sh
+    # Update the healthcheck to have a shorter check interval
+    HEALTHCHECK --start-period=5s --interval=1s --timeout=1s \
+        CMD /bin/sh /healthcheck.sh
--- a/docker/complement/conf/postgres.supervisord.conf
+++ b/docker/complement/conf/postgres.supervisord.conf
@@ -1,5 +1,5 @@
 [program:postgres]
-command=/usr/local/bin/prefix-log /usr/bin/pg_ctlcluster 13 main start --foreground
+command=/usr/local/bin/prefix-log gosu postgres postgres

 # Only start if START_POSTGRES=1
 autostart=%(ENV_START_POSTGRES)s
--- a/docs/SUMMARY.md
+++ b/docs/SUMMARY.md
@@ -35,7 +35,6 @@
    - [Application Services](application_services.md)
    - [Server Notices](server_notices.md)
    - [Consent Tracking](consent_tracking.md)
-    - [URL Previews](development/url_previews.md)
    - [User Directory](user_directory.md)
    - [Message Retention Policies](message_retention_policies.md)
    - [Pluggable Modules](modules/index.md)
--- a/docs/admin_api/rooms.md
+++ b/docs/admin_api/rooms.md
@@ -59,6 +59,7 @@ The following fields are possible in the JSON response body:
    - `guest_access` - Whether guests can join the room. One of: ["can_join", "forbidden"].
    - `history_visibility` - Who can see the room history. One of: ["invited", "joined", "shared", "world_readable"].
    - `state_events` - Total number of state_events of a room. Complexity of the room.
+    - `room_type` - The type of the room taken from the room's creation event; for example "m.space" if the room is a space. If the room does not define a type, the value will be `null`.
 * `offset` - The current pagination offset in rooms. This parameter should be
             used instead of `next_token` for room offset as `next_token` is
             not intended to be parsed.
@@ -101,7 +102,8 @@ A response body like the following is returned:
      "join_rules": "invite",
      "guest_access": null,
      "history_visibility": "shared",
-      "state_events": 93534
+      "state_events": 93534,
+      "room_type": "m.space"
    },
    ... (8 hidden items) ...
    {
@@ -118,7 +120,8 @@ A response body like the following is returned:
      "join_rules": "invite",
      "guest_access": null,
      "history_visibility": "shared",
-      "state_events": 8345
+      "state_events": 8345,
+      "room_type": null
    }
  ],
  "offset": 0,
@@ -151,7 +154,8 @@ A response body like the following is returned:
      "join_rules": "invite",
      "guest_access": null,
      "history_visibility": "shared",
-      "state_events": 8
+      "state_events": 8,
+      "room_type": null
    }
  ],
  "offset": 0,
@@ -184,7 +188,8 @@ A response body like the following is returned:
      "join_rules": "invite",
      "guest_access": null,
      "history_visibility": "shared",
-      "state_events": 93534
+      "state_events": 93534,
+      "room_type": null
    },
    ... (98 hidden items) ...
    {
@@ -201,7 +206,8 @@ A response body like the following is returned:
      "join_rules": "invite",
      "guest_access": null,
      "history_visibility": "shared",
-      "state_events": 8345
+      "state_events": 8345,
+      "room_type": "m.space"
    }
  ],
  "offset": 0,
@@ -238,7 +244,9 @@ A response body like the following is returned:
      "join_rules": "invite",
      "guest_access": null,
      "history_visibility": "shared",
-      "state_events": 93534
+      "state_events": 93534,
+      "room_type": "m.space"
+
    },
    ... (48 hidden items) ...
    {
@@ -255,7 +263,9 @@ A response body like the following is returned:
      "join_rules": "invite",
      "guest_access": null,
      "history_visibility": "shared",
-      "state_events": 8345
+      "state_events": 8345,
+      "room_type": null
+
    }
  ],
  "offset": 100,
@@ -290,6 +300,8 @@ The following fields are possible in the JSON response body:
 * `guest_access` - Whether guests can join the room. One of: ["can_join", "forbidden"].
 * `history_visibility` - Who can see the room history. One of: ["invited", "joined", "shared", "world_readable"].
 * `state_events` - Total number of state_events of a room. Complexity of the room.
+* `room_type` - The type of the room taken from the room's creation event; for example "m.space" if the room is a space.
+  If the room does not define a type, the value will be `null`.

 The API is:

@@ -317,7 +329,8 @@ A response body like the following is returned:
  "join_rules": "invite",
  "guest_access": null,
  "history_visibility": "shared",
-  "state_events": 93534
+  "state_events": 93534,
+  "room_type": "m.space"
 }
 ```

--- a/docs/admin_api/user_admin_api.md
+++ b/docs/admin_api/user_admin_api.md
@@ -544,7 +544,7 @@ Gets a list of all local media that a specific `user_id` has created.
 These are media that the user has uploaded themselves
 ([local media](../media_repository.md#local-media)), as well as
 [URL preview images](../media_repository.md#url-previews) requested by the user if the
-[feature is enabled](../development/url_previews.md).
+[feature is enabled](../usage/configuration/config_documentation.md#url_preview_enabled).

 By default, the response is ordered by descending creation date and ascending media ID.
 The newest media is on top. You can change the order with parameters
--- a/docs/development/url_previews.md
+++ b/docs/development/url_previews.md
@@ -1,61 +0,0 @@
-URL Previews
-============
-
-The `GET /_matrix/media/r0/preview_url` endpoint provides a generic preview API
-for URLs which outputs [Open Graph](https://ogp.me/) responses (with some Matrix
-specific additions).
-
-This does have trade-offs compared to other designs:
-
-* Pros:
-  * Simple and flexible; can be used by any clients at any point
-* Cons:
-  * If each homeserver provides one of these independently, all the HSes in a
-    room may needlessly DoS the target URI
-  * The URL metadata must be stored somewhere, rather than just using Matrix
-    itself to store the media.
-  * Matrix cannot be used to distribute the metadata between homeservers.
-
-When Synapse is asked to preview a URL it does the following:
-
-1. Checks against a URL blacklist (defined as `url_preview_url_blacklist` in the
-   config).
-2. Checks the in-memory cache by URLs and returns the result if it exists. (This
-   is also used to de-duplicate processing of multiple in-flight requests at once.)
-3. Kicks off a background process to generate a preview:
-   1. Checks the database cache by URL and timestamp and returns the result if it
-      has not expired and was successful (a 2xx return code).
-   2. Checks if the URL matches an [oEmbed](https://oembed.com/) pattern. If it
-      does, update the URL to download.
-   3. Downloads the URL and stores it into a file via the media storage provider
-      and saves the local media metadata.
-   4. If the media is an image:
-      1. Generates thumbnails.
-      2. Generates an Open Graph response based on image properties.
-   5. If the media is HTML:
-      1. Decodes the HTML via the stored file.
-      2. Generates an Open Graph response from the HTML.
-      3. If a JSON oEmbed URL was found in the HTML via autodiscovery:
-         1. Downloads the URL and stores it into a file via the media storage provider
-            and saves the local media metadata.
-         2. Convert the oEmbed response to an Open Graph response.
-         3. Override any Open Graph data from the HTML with data from oEmbed.
-      4. If an image exists in the Open Graph response:
-         1. Downloads the URL and stores it into a file via the media storage
-            provider and saves the local media metadata.
-         2. Generates thumbnails.
-         3. Updates the Open Graph response based on image properties.
-   6. If the media is JSON and an oEmbed URL was found:
-      1. Convert the oEmbed response to an Open Graph response.
-      2. If a thumbnail or image is in the oEmbed response:
-         1. Downloads the URL and stores it into a file via the media storage
-            provider and saves the local media metadata.
-         2. Generates thumbnails.
-         3. Updates the Open Graph response based on image properties.
-   7. Stores the result in the database cache.
-4. Returns the result.
-
-The in-memory cache expires after 1 hour.
-
-Expired entries in the database cache (and their associated media files) are
-deleted every 10 seconds. The default expiration time is 1 hour from download.
--- a/docs/media_repository.md
+++ b/docs/media_repository.md
@@ -7,8 +7,7 @@ The media repository
   users.
 * caches avatars, attachments and their thumbnails for media uploaded by remote
   users.
- * caches resources and thumbnails used for
-   [URL previews](development/url_previews.md).
+ * caches resources and thumbnails used for URL previews.

 All media in Matrix can be identified by a unique
 [MXC URI](https://spec.matrix.org/latest/client-server-api/#matrix-content-mxc-uris),
@@ -59,8 +58,6 @@ remote_thumbnail/matrix.org/aa/bb/cccccccccccccccccccc/128-96-image-jpeg
 Note that `remote_thumbnail/` does not have an `s`.

 ## URL Previews
-See [URL Previews](development/url_previews.md) for documentation on the URL preview
-process.

 When generating previews for URLs, Synapse may download and cache various
 resources, including images. These resources are assigned temporary media IDs
--- a/docs/upgrade.md
+++ b/docs/upgrade.md
@@ -89,6 +89,21 @@ process, for example:
    dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
    ```

+# Upgrading to v1.64.0
+
+## Delegation of email validation no longer supported
+
+As of this version, Synapse no longer allows the tasks of verifying email address
+ownership, and password reset confirmation, to be delegated to an identity server.
+
+To continue to allow users to add email addresses to their homeserver accounts,
+and perform password resets, make sure that Synapse is configured with a
+working email server in the `email` configuration section (including, at a
+minimum, a `notif_from` setting.)
+
+Specifying an `email` setting under `account_threepid_delegates` will now cause
+an error at startup.
+
 # Upgrading to v1.62.0

 ## New signatures for spam checker callbacks
--- a/docs/usage/administration/admin_api/README.md
+++ b/docs/usage/administration/admin_api/README.md
@@ -18,6 +18,11 @@ already on your `$PATH` depending on how Synapse was installed.
 Finding your user's `access_token` is client-dependent, but will usually be shown in the client's settings.

 ## Making an Admin API request
+For security reasons, we [recommend](reverse_proxy.md#synapse-administration-endpoints)
+that the Admin API (`/_synapse/admin/...`) should be hidden from public view using a
+reverse proxy. This means you should typically query the Admin API from a terminal on
+the machine which runs Synapse.
+
 Once you have your `access_token`, you will need to authenticate each request to an Admin API endpoint by
 providing the token as either a query parameter or a request header. To add it as a request header in cURL:

@@ -25,5 +30,17 @@ providing the token as either a query parameter or a request header. To add it a
 curl --header "Authorization: Bearer <access_token>" <the_rest_of_your_API_request>
 ```

+For example, suppose we want to
+[query the account](user_admin_api.md#query-user-account) of the user
+`@foo:bar.com`. We need an admin access token (e.g.
+`syt_AjfVef2_L33JNpafeif_0feKJfeaf0CQpoZk`), and we need to know which port
+Synapse's [`client` listener](config_documentation.md#listeners) is listening
+on (e.g. `8008`). Then we can use the following command to request the account
+information from the Admin API.
+
+```sh
+curl --header "Authorization: Bearer syt_AjfVef2_L33JNpafeif_0feKJfeaf0CQpoZk" -X GET http://127.0.0.1:8008/_synapse/admin/v2/users/@foo:bar.com
+```
+
 For more details on access tokens in Matrix, please refer to the complete
 [matrix spec documentation](https://matrix.org/docs/spec/client_server/r0.6.1#using-access-tokens).
--- a/docs/usage/configuration/config_documentation.md
+++ b/docs/usage/configuration/config_documentation.md
@@ -1257,6 +1257,98 @@ database:
    cp_max: 10
 ```
 ---
+### `databases`
+
+The `databases` option allows specifying a mapping between certain database tables and
+database host details, spreading the load of a single Synapse instance across multiple
+database backends. This is often referred to as "database sharding". This option is only
+supported for PostgreSQL database backends.
+
+**Important note:** This is a supported option, but is not currently used in production by the
+Matrix.org Foundation. Proceed with caution and always make backups.
+
+`databases` is a dictionary of arbitrarily-named database entries. Each entry is equivalent
+to the value of the `database` homeserver config option (see above), with the addition of
+a `data_stores` key. `data_stores` is an array of strings that specifies the data store(s)
+(a defined label for a set of tables) that should be stored on the associated database
+backend entry.
+
+The currently defined values for `data_stores` are:
+
+* `"state"`: Database that relates to state groups will be stored in this database.
+
+  Specifically, that means the following tables:
+  * `state_groups`
+  * `state_group_edges`
+  * `state_groups_state`
+
+  And the following sequences:
+  * `state_groups_seq_id`
+
+* `"main"`: All other database tables and sequences.
+
+All databases will end up with additional tables used for tracking database schema migrations
+and any pending background updates. Synapse will create these automatically on startup when checking for
+and/or performing database schema migrations.
+
+To migrate an existing database configuration (e.g. all tables on a single database) to a different
+configuration (e.g. the "main" data store on one database, and "state" on another), do the following:
+
+1. Take a backup of your existing database. Things can and do go wrong and database corruption is no joke!
+2. Ensure all pending database migrations have been applied and background updates have run. The simplest
+   way to do this is to use the `update_synapse_database` script supplied with your Synapse installation.
+
+   ```sh
+   update_synapse_database --database-config homeserver.yaml --run-background-updates
+   ```
+
+3. Copy over the necessary tables and sequences from one database to the other. Tables relating to database
+   migrations, schemas, schema versions and background updates should **not** be copied.
+
+   As an example, say that you'd like to split out the "state" data store from an existing database which
+   currently contains all data stores.
+
+   Simply copy the tables and sequences defined above for the "state" datastore from the existing database
+   to the secondary database. As noted above, additional tables will be created in the secondary database
+   when Synapse is started.
+
+4. Modify/create the `databases` option in your `homeserver.yaml` to match the desired database configuration.
+5. Start Synapse. Check that it starts up successfully and that things generally seem to be working.
+6. Drop the old tables that were copied in step 3.
+
+Only one of the options `database` or `databases` may be specified in your config, but not both.
+
+Example configuration:
+
+```yaml
+databases:
+  basement_box:
+    name: psycopg2
+    txn_limit: 10000
+    data_stores: ["main"]
+    args:
+      user: synapse_user
+      password: secretpassword
+      database: synapse_main
+      host: localhost
+      port: 5432
+      cp_min: 5
+      cp_max: 10
+
+  my_other_database:
+    name: psycopg2
+    txn_limit: 10000
+    data_stores: ["state"]
+    args:
+      user: synapse_user
+      password: secretpassword
+      database: synapse_state
+      host: localhost
+      port: 5432
+      cp_min: 5
+      cp_max: 10
+```
+---
 ## Logging ##
 Config options related to logging. 

@@ -2076,30 +2168,26 @@ default_identity_server: https://matrix.org
 ---
 ### `account_threepid_delegates`

-Handle threepid (email/phone etc) registration and password resets through a set of
-*trusted* identity servers. Note that this allows the configured identity server to
-reset passwords for accounts!
+Delegate verification of phone numbers to an identity server.

-Be aware that if `email` is not set, and SMTP options have not been
-configured in the email config block, registration and user password resets via
-email will be globally disabled.
+When a user wishes to add a phone number to their account, we need to verify that they
+actually own that phone number, which requires sending them a text message (SMS).
+Currently Synapse does not support sending those texts itself and instead delegates the
+task to an identity server. The base URI for the identity server to be used is
+specified by the `account_threepid_delegates.msisdn` option.

-Additionally, if `msisdn` is not set, registration and password resets via msisdn
-will be disabled regardless, and users will not be able to associate an msisdn
-identifier to their account. This is due to Synapse currently not supporting
-any method of sending SMS messages on its own.
+If this is left unspecified, Synapse will not allow users to add phone numbers to
+their account.

-To enable using an identity server for operations regarding a particular third-party
-identifier type, set the value to the URL of that identity server as shown in the
-examples below.
+(Servers handling the these requests must answer the `/requestToken` endpoints defined
+by the Matrix Identity Service API
+[specification](https://matrix.org/docs/spec/identity_service/latest).)

-Servers handling the these requests must answer the `/requestToken` endpoints defined
-by the Matrix Identity Service API [specification](https://matrix.org/docs/spec/identity_service/latest).
+*Updated in Synapse 1.64.0*: No longer accepts an `email` option.

 Example configuration:
 ```yaml
 account_threepid_delegates:
-    email: https://example.com     # Delegate email sending to example.com
    msisdn: http://localhost:8090  # Delegate SMS sending to this local process
 ```
 ---
--- a/poetry.lock
+++ b/poetry.lock
@@ -290,7 +290,7 @@ importlib-metadata = {version = "*", markers = "python_version < \"3.8\""}

 [[package]]
 name = "frozendict"
-version = "2.3.0"
+version = "2.3.2"
 description = "A simple immutable dictionary"
 category = "main"
 optional = false
@@ -1753,23 +1753,23 @@ flake8-comprehensions = [
    {file = "flake8_comprehensions-3.8.0-py3-none-any.whl", hash = "sha256:9406314803abe1193c064544ab14fdc43c58424c0882f6ff8a581eb73fc9bb58"},
 ]
 frozendict = [
-    {file = "frozendict-2.3.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:e18e2abd144a9433b0a8334582843b2aa0d3b9ac8b209aaa912ad365115fe2e1"},
-    {file = "frozendict-2.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:96dc7a02e78da5725e5e642269bb7ae792e0c9f13f10f2e02689175ebbfedb35"},
-    {file = "frozendict-2.3.0-cp310-cp310-win_amd64.whl", hash = "sha256:752a6dcfaf9bb20a7ecab24980e4dbe041f154509c989207caf185522ef85461"},
-    {file = "frozendict-2.3.0-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:5346d9fc1c936c76d33975a9a9f1a067342963105d9a403a99e787c939cc2bb2"},
-    {file = "frozendict-2.3.0-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:60dd2253f1bacb63a7c486ec541a968af4f985ffb06602ee8954a3d39ec6bd2e"},
-    {file = "frozendict-2.3.0-cp36-cp36m-win_amd64.whl", hash = "sha256:b2e044602ce17e5cd86724add46660fb9d80169545164e763300a3b839cb1b79"},
-    {file = "frozendict-2.3.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:a27a69b1ac3591e4258325108aee62b53c0eeb6ad0a993ae68d3c7eaea980420"},
-    {file = "frozendict-2.3.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4f45ef5f6b184d84744fff97b61f6b9a855e24d36b713ea2352fc723a047afa5"},
-    {file = "frozendict-2.3.0-cp37-cp37m-win_amd64.whl", hash = "sha256:2d3f5016650c0e9a192f5024e68fb4d63f670d0ee58b099ed3f5b4c62ea30ecb"},
-    {file = "frozendict-2.3.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:6cf605916f50aabaaba5624c81eb270200f6c2c466c46960237a125ec8fe3ae0"},
-    {file = "frozendict-2.3.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f6da06e44904beae4412199d7e49be4f85c6cc168ab06b77c735ea7da5ce3454"},
-    {file = "frozendict-2.3.0-cp38-cp38-win_amd64.whl", hash = "sha256:1f34793fb409c4fa70ffd25bea87b01f3bd305fb1c6b09e7dff085b126302206"},
-    {file = "frozendict-2.3.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:fd72494a559bdcd28aa71f4aa81860269cd0b7c45fff3e2614a0a053ecfd2a13"},
-    {file = "frozendict-2.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:00ea9166aa68cc5feed05986206fdbf35e838a09cb3feef998cf35978ff8a803"},
-    {file = "frozendict-2.3.0-cp39-cp39-win_amd64.whl", hash = "sha256:9ffaf440648b44e0bc694c1a4701801941378ba3ba6541e17750ae4b4aeeb116"},
-    {file = "frozendict-2.3.0-py3-none-any.whl", hash = "sha256:8578fe06815fcdcc672bd5603eebc98361a5317c1c3a13b28c6c810f6ea3b323"},
-    {file = "frozendict-2.3.0.tar.gz", hash = "sha256:da4231adefc5928e7810da2732269d3ad7b5616295b3e693746392a8205ea0b5"},
+    {file = "frozendict-2.3.2-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:4fb171d1e84d17335365877e19d17440373b47ca74a73c06f65ac0b16d01e87f"},
+    {file = "frozendict-2.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c0a3640e9d7533d164160b758351aa49d9e85bbe0bd76d219d4021e90ffa6a52"},
+    {file = "frozendict-2.3.2-cp310-cp310-win_amd64.whl", hash = "sha256:87cfd00fafbc147d8cd2590d1109b7db8ac8d7d5bdaa708ba46caee132b55d4d"},
+    {file = "frozendict-2.3.2-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:fb09761e093cfabb2f179dbfdb2521e1ec5701df714d1eb5c51fa7849027be19"},
+    {file = "frozendict-2.3.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:82176dc7adf01cf8f0193e909401939415a230a1853f4a672ec1629a06ceae18"},
+    {file = "frozendict-2.3.2-cp36-cp36m-win_amd64.whl", hash = "sha256:c1c70826aa4a50fa283fe161834ac4a3ac7c753902c980bb8b595b0998a38ddb"},
+    {file = "frozendict-2.3.2-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:1db5035ddbed995badd1a62c4102b5e207b5aeb24472df2c60aba79639d7996b"},
+    {file = "frozendict-2.3.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4246fc4cb1413645ba4d3513939b90d979a5bae724be605a10b2b26ee12f839c"},
+    {file = "frozendict-2.3.2-cp37-cp37m-win_amd64.whl", hash = "sha256:680cd42fb0a255da1ce45678ccbd7f69da750d5243809524ebe8f45b2eda6e6b"},
+    {file = "frozendict-2.3.2-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:6a7f3a181d6722c92a9fab12d0c5c2b006a18ca5666098531f316d1e1c8984e3"},
+    {file = "frozendict-2.3.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b1cb866eabb3c1384a7fe88e1e1033e2b6623073589012ab637c552bf03f6364"},
+    {file = "frozendict-2.3.2-cp38-cp38-win_amd64.whl", hash = "sha256:952c5e5e664578c5c2ce8489ee0ab6a1855da02b58ef593ee728fc10d672641a"},
+    {file = "frozendict-2.3.2-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:608b77904cd0117cd816df605a80d0043a5326ee62529327d2136c792165a823"},
+    {file = "frozendict-2.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0eed41fd326f0bcc779837d8d9e1374da1bc9857fe3b9f2910195bbd5fff3aeb"},
+    {file = "frozendict-2.3.2-cp39-cp39-win_amd64.whl", hash = "sha256:bde28db6b5868dd3c45b3555f9d1dc5a1cca6d93591502fa5dcecce0dde6a335"},
+    {file = "frozendict-2.3.2-py3-none-any.whl", hash = "sha256:6882a9bbe08ab9b5ff96ce11bdff3fe40b114b9813bc6801261e2a7b45e20012"},
+    {file = "frozendict-2.3.2.tar.gz", hash = "sha256:7fac4542f0a13fbe704db4942f41ba3abffec5af8b100025973e59dff6a09d0d"},
 ]
 gitdb = [
    {file = "gitdb-4.0.9-py3-none-any.whl", hash = "sha256:8033ad4e853066ba6ca92050b9df2f89301b8fc8bf7e9324d412a63f8bf1a8fd"},
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -54,7 +54,7 @@ skip_gitignore = true

 [tool.poetry]
 name = "matrix-synapse"
-version = "1.62.0"
+version = "1.63.0rc1"
 description = "Homeserver for the Matrix decentralised comms protocol"
 authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
 license = "Apache-2.0"
--- a/synapse/_scripts/synapse_port_db.py
+++ b/synapse/_scripts/synapse_port_db.py
@@ -166,22 +166,6 @@ IGNORED_TABLES = {
    "ui_auth_sessions",
    "ui_auth_sessions_credentials",
    "ui_auth_sessions_ips",
-    # Groups/communities is no longer supported.
-    "group_attestations_remote",
-    "group_attestations_renewals",
-    "group_invites",
-    "group_roles",
-    "group_room_categories",
-    "group_rooms",
-    "group_summary_roles",
-    "group_summary_room_categories",
-    "group_summary_rooms",
-    "group_summary_users",
-    "group_users",
-    "groups",
-    "local_group_membership",
-    "local_group_updates",
-    "remote_profile_cache",
 }


@@ -418,12 +402,15 @@ class Porter:
            self.progress.update(table, table_size)  # Mark table as done
            return

+        # We sweep over rowids in two directions: one forwards (rowids 1, 2, 3, ...)
+        # and another backwards (rowids 0, -1, -2, ...).
        forward_select = (
            "SELECT rowid, * FROM %s WHERE rowid >= ? ORDER BY rowid LIMIT ?" % (table,)
        )

        backward_select = (
-            "SELECT rowid, * FROM %s WHERE rowid <= ? ORDER BY rowid LIMIT ?" % (table,)
+            "SELECT rowid, * FROM %s WHERE rowid <= ? ORDER BY rowid DESC LIMIT ?"
+            % (table,)
        )

        do_forward = [True]
--- a/synapse/api/errors.py
+++ b/synapse/api/errors.py
@@ -297,8 +297,14 @@ class AuthError(SynapseError):
    other poorly-defined times.
    """

-    def __init__(self, code: int, msg: str, errcode: str = Codes.FORBIDDEN):
-        super().__init__(code, msg, errcode)
+    def __init__(
+        self,
+        code: int,
+        msg: str,
+        errcode: str = Codes.FORBIDDEN,
+        additional_fields: Optional[dict] = None,
+    ):
+        super().__init__(code, msg, errcode, additional_fields)


 class InvalidClientCredentialsError(SynapseError):
--- a/synapse/api/ratelimiting.py
+++ b/synapse/api/ratelimiting.py
@@ -27,6 +27,33 @@ class Ratelimiter:
    """
    Ratelimit actions marked by arbitrary keys.

+    (Note that the source code speaks of "actions" and "burst_count" rather than
+    "tokens" and a "bucket_size".)
+
+    This is a "leaky bucket as a meter". For each key to be tracked there is a bucket
+    containing some number 0 <= T <= `burst_count` of tokens corresponding to previously
+    permitted requests for that key. Each bucket starts empty, and gradually leaks
+    tokens at a rate of `rate_hz`.
+
+    Upon an incoming request, we must determine:
+    - the key that this request falls under (which bucket to inspect), and
+    - the cost C of this request in tokens.
+    Then, if there is room in the bucket for C tokens (T + C <= `burst_count`),
+    the request is permitted and `cost` tokens are added to the bucket.
+    Otherwise the request is denied, and the bucket continues to hold T tokens.
+
+    This means that the limiter enforces an average request frequency of `rate_hz`,
+    while accumulating a buffer of up to `burst_count` requests which can be consumed
+    instantaneously.
+
+    The tricky bit is the leaking. We do not want to have a periodic process which
+    leaks every bucket! Instead, we track
+    - the time point when the bucket was last completely empty, and
+    - how many tokens have added to the bucket permitted since then.
+    Then for each incoming request, we can calculate how many tokens have leaked
+    since this time point, and use that to decide if we should accept or reject the
+    request.
+
    Args:
        clock: A homeserver clock, for retrieving the current time
        rate_hz: The long term number of actions that can be performed in a second.
@@ -41,14 +68,30 @@ class Ratelimiter:
        self.burst_count = burst_count
        self.store = store

-        # A ordered dictionary keeping track of actions, when they were last
-        # performed and how often. Each entry is a mapping from a key of arbitrary type
-        # to a tuple representing:
-        #   * How many times an action has occurred since a point in time
-        #   * The point in time
-        #   * The rate_hz of this particular entry. This can vary per request
+        # An ordered dictionary representing the token buckets tracked by this rate
+        # limiter. Each entry maps a key of arbitrary type to a tuple representing:
+        #   * The number of tokens currently in the bucket,
+        #   * The time point when the bucket was last completely empty, and
+        #   * The rate_hz (leak rate) of this particular bucket.
        self.actions: OrderedDict[Hashable, Tuple[float, float, float]] = OrderedDict()

+    def _get_key(
+        self, requester: Optional[Requester], key: Optional[Hashable]
+    ) -> Hashable:
+        """Use the requester's MXID as a fallback key if no key is provided."""
+        if key is None:
+            if not requester:
+                raise ValueError("Must supply at least one of `requester` or `key`")
+
+            key = requester.user.to_string()
+        return key
+
+    def _get_action_counts(
+        self, key: Hashable, time_now_s: float
+    ) -> Tuple[float, float, float]:
+        """Retrieve the action counts, with a fallback representing an empty bucket."""
+        return self.actions.get(key, (0.0, time_now_s, 0.0))
+
    async def can_do_action(
        self,
        requester: Optional[Requester],
@@ -88,11 +131,7 @@ class Ratelimiter:
                * The reactor timestamp for when the action can be performed next.
                  -1 if rate_hz is less than or equal to zero
        """
-        if key is None:
-            if not requester:
-                raise ValueError("Must supply at least one of `requester` or `key`")
-
-            key = requester.user.to_string()
+        key = self._get_key(requester, key)

        if requester:
            # Disable rate limiting of users belonging to any AS that is configured
@@ -121,7 +160,7 @@ class Ratelimiter:
        self._prune_message_counts(time_now_s)

        # Check if there is an existing count entry for this key
-        action_count, time_start, _ = self.actions.get(key, (0.0, time_now_s, 0.0))
+        action_count, time_start, _ = self._get_action_counts(key, time_now_s)

        # Check whether performing another action is allowed
        time_delta = time_now_s - time_start
@@ -164,6 +203,37 @@ class Ratelimiter:

        return allowed, time_allowed

+    def record_action(
+        self,
+        requester: Optional[Requester],
+        key: Optional[Hashable] = None,
+        n_actions: int = 1,
+        _time_now_s: Optional[float] = None,
+    ) -> None:
+        """Record that an action(s) took place, even if they violate the rate limit.
+
+        This is useful for tracking the frequency of events that happen across
+        federation which we still want to impose local rate limits on. For instance, if
+        we are alice.com monitoring a particular room, we cannot prevent bob.com
+        from joining users to that room. However, we can track the number of recent
+        joins in the room and refuse to serve new joins ourselves if there have been too
+        many in the room across both homeservers.
+
+        Args:
+            requester: The requester that is doing the action, if any.
+            key: An arbitrary key used to classify an action. Defaults to the
+                requester's user ID.
+            n_actions: The number of times the user wants to do this action. If the user
+                cannot do all of the actions, the user's action count is not incremented
+                at all.
+            _time_now_s: The current time. Optional, defaults to the current time according
+                to self.clock. Only used by tests.
+        """
+        key = self._get_key(requester, key)
+        time_now_s = _time_now_s if _time_now_s is not None else self.clock.time()
+        action_count, time_start, rate_hz = self._get_action_counts(key, time_now_s)
+        self.actions[key] = (action_count + n_actions, time_start, rate_hz)
+
    def _prune_message_counts(self, time_now_s: float) -> None:
        """Remove message count entries that have not exceeded their defined
        rate_hz limit
--- a/synapse/api/room_versions.py
+++ b/synapse/api/room_versions.py
@@ -84,6 +84,8 @@ class RoomVersion:
    # MSC3787: Adds support for a `knock_restricted` join rule, mixing concepts of
    # knocks and restricted join rules into the same join condition.
    msc3787_knock_restricted_join_rule: bool
+    # MSC3667: Enforce integer power levels
+    msc3667_int_only_power_levels: bool


 class RoomVersions:
@@ -103,6 +105,7 @@ class RoomVersions:
        msc2716_historical=False,
        msc2716_redactions=False,
        msc3787_knock_restricted_join_rule=False,
+        msc3667_int_only_power_levels=False,
    )
    V2 = RoomVersion(
        "2",
@@ -120,6 +123,7 @@ class RoomVersions:
        msc2716_historical=False,
        msc2716_redactions=False,
        msc3787_knock_restricted_join_rule=False,
+        msc3667_int_only_power_levels=False,
    )
    V3 = RoomVersion(
        "3",
@@ -137,6 +141,7 @@ class RoomVersions:
        msc2716_historical=False,
        msc2716_redactions=False,
        msc3787_knock_restricted_join_rule=False,
+        msc3667_int_only_power_levels=False,
    )
    V4 = RoomVersion(
        "4",
@@ -154,6 +159,7 @@ class RoomVersions:
        msc2716_historical=False,
        msc2716_redactions=False,
        msc3787_knock_restricted_join_rule=False,
+        msc3667_int_only_power_levels=False,
    )
    V5 = RoomVersion(
        "5",
@@ -171,6 +177,7 @@ class RoomVersions:
        msc2716_historical=False,
        msc2716_redactions=False,
        msc3787_knock_restricted_join_rule=False,
+        msc3667_int_only_power_levels=False,
    )
    V6 = RoomVersion(
        "6",
@@ -188,6 +195,7 @@ class RoomVersions:
        msc2716_historical=False,
        msc2716_redactions=False,
        msc3787_knock_restricted_join_rule=False,
+        msc3667_int_only_power_levels=False,
    )
    MSC2176 = RoomVersion(
        "org.matrix.msc2176",
@@ -205,6 +213,7 @@ class RoomVersions:
        msc2716_historical=False,
        msc2716_redactions=False,
        msc3787_knock_restricted_join_rule=False,
+        msc3667_int_only_power_levels=False,
    )
    V7 = RoomVersion(
        "7",
@@ -222,6 +231,7 @@ class RoomVersions:
        msc2716_historical=False,
        msc2716_redactions=False,
        msc3787_knock_restricted_join_rule=False,
+        msc3667_int_only_power_levels=False,
    )
    V8 = RoomVersion(
        "8",
@@ -239,6 +249,7 @@ class RoomVersions:
        msc2716_historical=False,
        msc2716_redactions=False,
        msc3787_knock_restricted_join_rule=False,
+        msc3667_int_only_power_levels=False,
    )
    V9 = RoomVersion(
        "9",
@@ -256,6 +267,7 @@ class RoomVersions:
        msc2716_historical=False,
        msc2716_redactions=False,
        msc3787_knock_restricted_join_rule=False,
+        msc3667_int_only_power_levels=False,
    )
    MSC2716v3 = RoomVersion(
        "org.matrix.msc2716v3",
@@ -273,6 +285,7 @@ class RoomVersions:
        msc2716_historical=True,
        msc2716_redactions=True,
        msc3787_knock_restricted_join_rule=False,
+        msc3667_int_only_power_levels=False,
    )
    MSC3787 = RoomVersion(
        "org.matrix.msc3787",
@@ -290,6 +303,25 @@ class RoomVersions:
        msc2716_historical=False,
        msc2716_redactions=False,
        msc3787_knock_restricted_join_rule=True,
+        msc3667_int_only_power_levels=False,
+    )
+    V10 = RoomVersion(
+        "10",
+        RoomDisposition.STABLE,
+        EventFormatVersions.V3,
+        StateResolutionVersions.V2,
+        enforce_key_validity=True,
+        special_case_aliases_auth=False,
+        strict_canonicaljson=True,
+        limit_notifications_power_levels=True,
+        msc2176_redaction_rules=False,
+        msc3083_join_rules=True,
+        msc3375_redaction_rules=True,
+        msc2403_knocking=True,
+        msc2716_historical=False,
+        msc2716_redactions=False,
+        msc3787_knock_restricted_join_rule=True,
+        msc3667_int_only_power_levels=True,
    )


@@ -308,6 +340,7 @@ KNOWN_ROOM_VERSIONS: Dict[str, RoomVersion] = {
        RoomVersions.V9,
        RoomVersions.MSC2716v3,
        RoomVersions.MSC3787,
+        RoomVersions.V10,
    )
 }

--- a/synapse/app/admin_cmd.py
+++ b/synapse/app/admin_cmd.py
@@ -39,6 +39,7 @@ from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
 from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
 from synapse.replication.slave.storage.registration import SlavedRegistrationStore
 from synapse.server import HomeServer
+from synapse.storage.database import DatabasePool, LoggingDatabaseConnection
 from synapse.storage.databases.main.room import RoomWorkerStore
 from synapse.types import StateMap
 from synapse.util import SYNAPSE_VERSION
@@ -60,7 +61,17 @@ class AdminCmdSlavedStore(
    BaseSlavedStore,
    RoomWorkerStore,
 ):
-    pass
+    def __init__(
+        self,
+        database: DatabasePool,
+        db_conn: LoggingDatabaseConnection,
+        hs: "HomeServer",
+    ):
+        super().__init__(database, db_conn, hs)
+
+        # Annoyingly `filter_events_for_client` assumes that this exists. We
+        # should refactor it to take a `Clock` directly.
+        self.clock = hs.get_clock()


 class AdminCmdServer(HomeServer):
--- a/synapse/app/homeserver.py
+++ b/synapse/app/homeserver.py
@@ -44,7 +44,6 @@ from synapse.app._base import (
    register_start,
 )
 from synapse.config._base import ConfigError, format_config_error
-from synapse.config.emailconfig import ThreepidBehaviour
 from synapse.config.homeserver import HomeServerConfig
 from synapse.config.server import ListenerConfig
 from synapse.federation.transport.server import TransportLayerServer
@@ -202,7 +201,7 @@ class SynapseHomeServer(HomeServer):
                }
            )

-            if self.config.email.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
+            if self.config.email.can_verify_email:
                from synapse.rest.synapse.client.password_reset import (
                    PasswordResetSubmitTokenResource,
                )
--- a/synapse/appservice/api.py
+++ b/synapse/appservice/api.py
@@ -53,6 +53,18 @@ sent_events_counter = Counter(
    "synapse_appservice_api_sent_events", "Number of events sent to the AS", ["service"]
 )

+sent_ephemeral_counter = Counter(
+    "synapse_appservice_api_sent_ephemeral",
+    "Number of ephemeral events sent to the AS",
+    ["service"],
+)
+
+sent_todevice_counter = Counter(
+    "synapse_appservice_api_sent_todevice",
+    "Number of todevice messages sent to the AS",
+    ["service"],
+)
+
 HOUR_IN_MS = 60 * 60 * 1000


@@ -310,6 +322,8 @@ class ApplicationServiceApi(SimpleHttpClient):
                )
            sent_transactions_counter.labels(service.id).inc()
            sent_events_counter.labels(service.id).inc(len(serialized_events))
+            sent_ephemeral_counter.labels(service.id).inc(len(ephemeral))
+            sent_todevice_counter.labels(service.id).inc(len(to_device_messages))
            return True
        except CodeMessageException as e:
            logger.warning(
--- a/synapse/appservice/scheduler.py
+++ b/synapse/appservice/scheduler.py
@@ -319,7 +319,9 @@ class _ServiceQueuer:
        rooms_of_interesting_users.update(event.room_id for event in events)
        # EDUs
        rooms_of_interesting_users.update(
-            ephemeral["room_id"] for ephemeral in ephemerals
+            ephemeral["room_id"]
+            for ephemeral in ephemerals
+            if ephemeral.get("room_id") is not None
        )

        # Look up the AS users in those rooms
@@ -329,8 +331,9 @@ class _ServiceQueuer:
            )

        # Add recipients of to-device messages.
-        # device_message["user_id"] is the ID of the recipient.
-        users.update(device_message["user_id"] for device_message in to_device_messages)
+        users.update(
+            device_message["to_user_id"] for device_message in to_device_messages
+        )

        # Compute and return the counts / fallback key usage states
        otk_counts = await self._store.count_bulk_e2e_one_time_keys_for_as(users)
--- a/synapse/config/emailconfig.py
+++ b/synapse/config/emailconfig.py
@@ -18,7 +18,6 @@
 import email.utils
 import logging
 import os
-from enum import Enum
 from typing import Any

 import attr
@@ -131,41 +130,22 @@ class EmailConfig(Config):

        self.email_enable_notifs = email_config.get("enable_notifs", False)

-        self.threepid_behaviour_email = (
-            # Have Synapse handle the email sending if account_threepid_delegates.email
-            # is not defined
-            # msisdn is currently always remote while Synapse does not support any method of
-            # sending SMS messages
-            ThreepidBehaviour.REMOTE
-            if self.root.registration.account_threepid_delegate_email
-            else ThreepidBehaviour.LOCAL
-        )
-
        if config.get("trust_identity_server_for_password_resets"):
            raise ConfigError(
                'The config option "trust_identity_server_for_password_resets" '
-                'has been replaced by "account_threepid_delegate". '
-                "Please consult the configuration manual at docs/usage/configuration/config_documentation.md for "
-                "details and update your config file."
+                "is no longer supported. Please remove it from the config file."
            )

-        self.local_threepid_handling_disabled_due_to_email_config = False
-        if (
-            self.threepid_behaviour_email == ThreepidBehaviour.LOCAL
-            and email_config == {}
-        ):
-            # We cannot warn the user this has happened here
-            # Instead do so when a user attempts to reset their password
-            self.local_threepid_handling_disabled_due_to_email_config = True
-
-            self.threepid_behaviour_email = ThreepidBehaviour.OFF
+        # If we have email config settings, assume that we can verify ownership of
+        # email addresses.
+        self.can_verify_email = email_config != {}

        # Get lifetime of a validation token in milliseconds
        self.email_validation_token_lifetime = self.parse_duration(
            email_config.get("validation_token_lifetime", "1h")
        )

-        if self.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
+        if self.can_verify_email:
            missing = []
            if not self.email_notif_from:
                missing.append("email.notif_from")
@@ -356,18 +336,3 @@ class EmailConfig(Config):
                    "Config option email.invite_client_location must be a http or https URL",
                    path=("email", "invite_client_location"),
                )
-
-
-class ThreepidBehaviour(Enum):
-    """
-    Enum to define the behaviour of Synapse with regards to when it contacts an identity
-    server for 3pid registration and password resets
-
-    REMOTE = use an external server to send tokens
-    LOCAL = send tokens ourselves
-    OFF = disable registration via 3pid and password resets
-    """
-
-    REMOTE = "remote"
-    LOCAL = "local"
-    OFF = "off"
--- a/synapse/config/registration.py
+++ b/synapse/config/registration.py
@@ -20,6 +20,13 @@ from synapse.config._base import Config, ConfigError
 from synapse.types import JsonDict, RoomAlias, UserID
 from synapse.util.stringutils import random_string_with_symbols, strtobool

+NO_EMAIL_DELEGATE_ERROR = """\
+Delegation of email verification to an identity server is no longer supported. To
+continue to allow users to add email addresses to their accounts, and use them for
+password resets, configure Synapse with an SMTP server via the `email` setting, and
+remove `account_threepid_delegates.email`.
+"""
+

 class RegistrationConfig(Config):
    section = "registration"
@@ -51,7 +58,9 @@ class RegistrationConfig(Config):
        self.bcrypt_rounds = config.get("bcrypt_rounds", 12)

        account_threepid_delegates = config.get("account_threepid_delegates") or {}
-        self.account_threepid_delegate_email = account_threepid_delegates.get("email")
+        if "email" in account_threepid_delegates:
+            raise ConfigError(NO_EMAIL_DELEGATE_ERROR)
+        # self.account_threepid_delegate_email = account_threepid_delegates.get("email")
        self.account_threepid_delegate_msisdn = account_threepid_delegates.get("msisdn")
        self.default_identity_server = config.get("default_identity_server")
        self.allow_guest_access = config.get("allow_guest_access", False)
--- a/synapse/event_auth.py
+++ b/synapse/event_auth.py
@@ -740,6 +740,32 @@ def _check_power_levels(
        except Exception:
            raise SynapseError(400, "Not a valid power level: %s" % (v,))

+    # Reject events with stringy power levels if required by room version
+    if (
+        event.type == EventTypes.PowerLevels
+        and room_version_obj.msc3667_int_only_power_levels
+    ):
+        for k, v in event.content.items():
+            if k in {
+                "users_default",
+                "events_default",
+                "state_default",
+                "ban",
+                "redact",
+                "kick",
+                "invite",
+            }:
+                if not isinstance(v, int):
+                    raise SynapseError(400, f"{v!r} must be an integer.")
+            if k in {"events", "notifications", "users"}:
+                if not isinstance(v, dict) or not all(
+                    isinstance(v, int) for v in v.values()
+                ):
+                    raise SynapseError(
+                        400,
+                        f"{v!r} must be a dict wherein all the values are integers.",
+                    )
+
    key = (event.type, event.state_key)
    current_state = auth_events.get(key)

--- a/synapse/events/builder.py
+++ b/synapse/events/builder.py
@@ -120,7 +120,7 @@ class EventBuilder:
            The signed and hashed event.
        """
        if auth_event_ids is None:
-            state_ids = await self._state.get_current_state_ids(
+            state_ids = await self._state.compute_state_after_events(
                self.room_id, prev_event_ids
            )
            auth_event_ids = self._event_auth_handler.compute_auth_events(
--- a/synapse/events/spamcheck.py
+++ b/synapse/events/spamcheck.py
@@ -21,7 +21,6 @@ from typing import (
    Awaitable,
    Callable,
    Collection,
-    Dict,
    List,
    Optional,
    Tuple,
@@ -32,10 +31,11 @@ from typing import (
 from typing_extensions import Literal

 import synapse
+from synapse.api.errors import Codes
 from synapse.rest.media.v1._base import FileInfo
 from synapse.rest.media.v1.media_storage import ReadableFileWrapper
 from synapse.spam_checker_api import RegistrationBehaviour
-from synapse.types import RoomAlias, UserProfile
+from synapse.types import JsonDict, RoomAlias, UserProfile
 from synapse.util.async_helpers import delay_cancellation, maybe_awaitable
 from synapse.util.metrics import Measure

@@ -50,12 +50,12 @@ CHECK_EVENT_FOR_SPAM_CALLBACK = Callable[
    Awaitable[
        Union[
            str,
-            "synapse.api.errors.Codes",
+            Codes,
            # Highly experimental, not officially part of the spamchecker API, may
            # disappear without warning depending on the results of ongoing
            # experiments.
            # Use this to return additional information as part of an error.
-            Tuple["synapse.api.errors.Codes", Dict],
+            Tuple[Codes, JsonDict],
            # Deprecated
            bool,
        ]
@@ -70,7 +70,12 @@ USER_MAY_JOIN_ROOM_CALLBACK = Callable[
    Awaitable[
        Union[
            Literal["NOT_SPAM"],
-            "synapse.api.errors.Codes",
+            Codes,
+            # Highly experimental, not officially part of the spamchecker API, may
+            # disappear without warning depending on the results of ongoing
+            # experiments.
+            # Use this to return additional information as part of an error.
+            Tuple[Codes, JsonDict],
            # Deprecated
            bool,
        ]
@@ -81,7 +86,12 @@ USER_MAY_INVITE_CALLBACK = Callable[
    Awaitable[
        Union[
            Literal["NOT_SPAM"],
-            "synapse.api.errors.Codes",
+            Codes,
+            # Highly experimental, not officially part of the spamchecker API, may
+            # disappear without warning depending on the results of ongoing
+            # experiments.
+            # Use this to return additional information as part of an error.
+            Tuple[Codes, JsonDict],
            # Deprecated
            bool,
        ]
@@ -92,7 +102,12 @@ USER_MAY_SEND_3PID_INVITE_CALLBACK = Callable[
    Awaitable[
        Union[
            Literal["NOT_SPAM"],
-            "synapse.api.errors.Codes",
+            Codes,
+            # Highly experimental, not officially part of the spamchecker API, may
+            # disappear without warning depending on the results of ongoing
+            # experiments.
+            # Use this to return additional information as part of an error.
+            Tuple[Codes, JsonDict],
            # Deprecated
            bool,
        ]
@@ -103,7 +118,12 @@ USER_MAY_CREATE_ROOM_CALLBACK = Callable[
    Awaitable[
        Union[
            Literal["NOT_SPAM"],
-            "synapse.api.errors.Codes",
+            Codes,
+            # Highly experimental, not officially part of the spamchecker API, may
+            # disappear without warning depending on the results of ongoing
+            # experiments.
+            # Use this to return additional information as part of an error.
+            Tuple[Codes, JsonDict],
            # Deprecated
            bool,
        ]
@@ -114,7 +134,12 @@ USER_MAY_CREATE_ROOM_ALIAS_CALLBACK = Callable[
    Awaitable[
        Union[
            Literal["NOT_SPAM"],
-            "synapse.api.errors.Codes",
+            Codes,
+            # Highly experimental, not officially part of the spamchecker API, may
+            # disappear without warning depending on the results of ongoing
+            # experiments.
+            # Use this to return additional information as part of an error.
+            Tuple[Codes, JsonDict],
            # Deprecated
            bool,
        ]
@@ -125,7 +150,12 @@ USER_MAY_PUBLISH_ROOM_CALLBACK = Callable[
    Awaitable[
        Union[
            Literal["NOT_SPAM"],
-            "synapse.api.errors.Codes",
+            Codes,
+            # Highly experimental, not officially part of the spamchecker API, may
+            # disappear without warning depending on the results of ongoing
+            # experiments.
+            # Use this to return additional information as part of an error.
+            Tuple[Codes, JsonDict],
            # Deprecated
            bool,
        ]
@@ -154,7 +184,12 @@ CHECK_MEDIA_FILE_FOR_SPAM_CALLBACK = Callable[
    Awaitable[
        Union[
            Literal["NOT_SPAM"],
-            "synapse.api.errors.Codes",
+            Codes,
+            # Highly experimental, not officially part of the spamchecker API, may
+            # disappear without warning depending on the results of ongoing
+            # experiments.
+            # Use this to return additional information as part of an error.
+            Tuple[Codes, JsonDict],
            # Deprecated
            bool,
        ]
@@ -345,7 +380,7 @@ class SpamChecker:

    async def check_event_for_spam(
        self, event: "synapse.events.EventBase"
-    ) -> Union[Tuple["synapse.api.errors.Codes", Dict], str]:
+    ) -> Union[Tuple[Codes, JsonDict], str]:
        """Checks if a given event is considered "spammy" by this server.

        If the server considers an event spammy, then it will be rejected if
@@ -376,7 +411,16 @@ class SpamChecker:
                elif res is True:
                    # This spam-checker rejects the event with deprecated
                    # return value `True`
-                    return (synapse.api.errors.Codes.FORBIDDEN, {})
+                    return synapse.api.errors.Codes.FORBIDDEN, {}
+                elif (
+                    isinstance(res, tuple)
+                    and len(res) == 2
+                    and isinstance(res[0], synapse.api.errors.Codes)
+                    and isinstance(res[1], dict)
+                ):
+                    return res
+                elif isinstance(res, synapse.api.errors.Codes):
+                    return res, {}
                elif not isinstance(res, str):
                    # mypy complains that we can't reach this code because of the
                    # return type in CHECK_EVENT_FOR_SPAM_CALLBACK, but we don't know
@@ -422,7 +466,7 @@ class SpamChecker:

    async def user_may_join_room(
        self, user_id: str, room_id: str, is_invited: bool
-    ) -> Union["synapse.api.errors.Codes", Literal["NOT_SPAM"]]:
+    ) -> Union[Tuple[Codes, JsonDict], Literal["NOT_SPAM"]]:
        """Checks if a given users is allowed to join a room.
        Not called when a user creates a room.

@@ -432,7 +476,7 @@ class SpamChecker:
            is_invited: Whether the user is invited into the room

        Returns:
-            NOT_SPAM if the operation is permitted, Codes otherwise.
+            NOT_SPAM if the operation is permitted, [Codes, Dict] otherwise.
        """
        for callback in self._user_may_join_room_callbacks:
            with Measure(
@@ -443,21 +487,28 @@ class SpamChecker:
                if res is True or res is self.NOT_SPAM:
                    continue
                elif res is False:
-                    return synapse.api.errors.Codes.FORBIDDEN
+                    return synapse.api.errors.Codes.FORBIDDEN, {}
                elif isinstance(res, synapse.api.errors.Codes):
+                    return res, {}
+                elif (
+                    isinstance(res, tuple)
+                    and len(res) == 2
+                    and isinstance(res[0], synapse.api.errors.Codes)
+                    and isinstance(res[1], dict)
+                ):
                    return res
                else:
                    logger.warning(
                        "Module returned invalid value, rejecting join as spam"
                    )
-                    return synapse.api.errors.Codes.FORBIDDEN
+                    return synapse.api.errors.Codes.FORBIDDEN, {}

        # No spam-checker has rejected the request, let it pass.
        return self.NOT_SPAM

    async def user_may_invite(
        self, inviter_userid: str, invitee_userid: str, room_id: str
-    ) -> Union["synapse.api.errors.Codes", Literal["NOT_SPAM"]]:
+    ) -> Union[Tuple[Codes, dict], Literal["NOT_SPAM"]]:
        """Checks if a given user may send an invite

        Args:
@@ -479,21 +530,28 @@ class SpamChecker:
                if res is True or res is self.NOT_SPAM:
                    continue
                elif res is False:
-                    return synapse.api.errors.Codes.FORBIDDEN
+                    return synapse.api.errors.Codes.FORBIDDEN, {}
                elif isinstance(res, synapse.api.errors.Codes):
+                    return res, {}
+                elif (
+                    isinstance(res, tuple)
+                    and len(res) == 2
+                    and isinstance(res[0], synapse.api.errors.Codes)
+                    and isinstance(res[1], dict)
+                ):
                    return res
                else:
                    logger.warning(
                        "Module returned invalid value, rejecting invite as spam"
                    )
-                    return synapse.api.errors.Codes.FORBIDDEN
+                    return synapse.api.errors.Codes.FORBIDDEN, {}

        # No spam-checker has rejected the request, let it pass.
        return self.NOT_SPAM

    async def user_may_send_3pid_invite(
        self, inviter_userid: str, medium: str, address: str, room_id: str
-    ) -> Union["synapse.api.errors.Codes", Literal["NOT_SPAM"]]:
+    ) -> Union[Tuple[Codes, dict], Literal["NOT_SPAM"]]:
        """Checks if a given user may invite a given threepid into the room

        Note that if the threepid is already associated with a Matrix user ID, Synapse
@@ -519,20 +577,27 @@ class SpamChecker:
                if res is True or res is self.NOT_SPAM:
                    continue
                elif res is False:
-                    return synapse.api.errors.Codes.FORBIDDEN
+                    return synapse.api.errors.Codes.FORBIDDEN, {}
                elif isinstance(res, synapse.api.errors.Codes):
+                    return res, {}
+                elif (
+                    isinstance(res, tuple)
+                    and len(res) == 2
+                    and isinstance(res[0], synapse.api.errors.Codes)
+                    and isinstance(res[1], dict)
+                ):
                    return res
                else:
                    logger.warning(
                        "Module returned invalid value, rejecting 3pid invite as spam"
                    )
-                    return synapse.api.errors.Codes.FORBIDDEN
+                    return synapse.api.errors.Codes.FORBIDDEN, {}

        return self.NOT_SPAM

    async def user_may_create_room(
        self, userid: str
-    ) -> Union["synapse.api.errors.Codes", Literal["NOT_SPAM"]]:
+    ) -> Union[Tuple[Codes, dict], Literal["NOT_SPAM"]]:
        """Checks if a given user may create a room

        Args:
@@ -546,20 +611,27 @@ class SpamChecker:
                if res is True or res is self.NOT_SPAM:
                    continue
                elif res is False:
-                    return synapse.api.errors.Codes.FORBIDDEN
+                    return synapse.api.errors.Codes.FORBIDDEN, {}
                elif isinstance(res, synapse.api.errors.Codes):
+                    return res, {}
+                elif (
+                    isinstance(res, tuple)
+                    and len(res) == 2
+                    and isinstance(res[0], synapse.api.errors.Codes)
+                    and isinstance(res[1], dict)
+                ):
                    return res
                else:
                    logger.warning(
                        "Module returned invalid value, rejecting room creation as spam"
                    )
-                    return synapse.api.errors.Codes.FORBIDDEN
+                    return synapse.api.errors.Codes.FORBIDDEN, {}

        return self.NOT_SPAM

    async def user_may_create_room_alias(
        self, userid: str, room_alias: RoomAlias
-    ) -> Union["synapse.api.errors.Codes", Literal["NOT_SPAM"]]:
+    ) -> Union[Tuple[Codes, dict], Literal["NOT_SPAM"]]:
        """Checks if a given user may create a room alias

        Args:
@@ -575,20 +647,27 @@ class SpamChecker:
                if res is True or res is self.NOT_SPAM:
                    continue
                elif res is False:
-                    return synapse.api.errors.Codes.FORBIDDEN
+                    return synapse.api.errors.Codes.FORBIDDEN, {}
                elif isinstance(res, synapse.api.errors.Codes):
+                    return res, {}
+                elif (
+                    isinstance(res, tuple)
+                    and len(res) == 2
+                    and isinstance(res[0], synapse.api.errors.Codes)
+                    and isinstance(res[1], dict)
+                ):
                    return res
                else:
                    logger.warning(
                        "Module returned invalid value, rejecting room create as spam"
                    )
-                    return synapse.api.errors.Codes.FORBIDDEN
+                    return synapse.api.errors.Codes.FORBIDDEN, {}

        return self.NOT_SPAM

    async def user_may_publish_room(
        self, userid: str, room_id: str
-    ) -> Union["synapse.api.errors.Codes", Literal["NOT_SPAM"]]:
+    ) -> Union[Tuple[Codes, dict], Literal["NOT_SPAM"]]:
        """Checks if a given user may publish a room to the directory

        Args:
@@ -603,14 +682,21 @@ class SpamChecker:
                if res is True or res is self.NOT_SPAM:
                    continue
                elif res is False:
-                    return synapse.api.errors.Codes.FORBIDDEN
+                    return synapse.api.errors.Codes.FORBIDDEN, {}
                elif isinstance(res, synapse.api.errors.Codes):
+                    return res, {}
+                elif (
+                    isinstance(res, tuple)
+                    and len(res) == 2
+                    and isinstance(res[0], synapse.api.errors.Codes)
+                    and isinstance(res[1], dict)
+                ):
                    return res
                else:
                    logger.warning(
                        "Module returned invalid value, rejecting room publication as spam"
                    )
-                    return synapse.api.errors.Codes.FORBIDDEN
+                    return synapse.api.errors.Codes.FORBIDDEN, {}

        return self.NOT_SPAM

@@ -678,7 +764,7 @@ class SpamChecker:

    async def check_media_file_for_spam(
        self, file_wrapper: ReadableFileWrapper, file_info: FileInfo
-    ) -> Union["synapse.api.errors.Codes", Literal["NOT_SPAM"]]:
+    ) -> Union[Tuple[Codes, dict], Literal["NOT_SPAM"]]:
        """Checks if a piece of newly uploaded media should be blocked.

        This will be called for local uploads, downloads of remote media, each
@@ -715,13 +801,20 @@ class SpamChecker:
                if res is False or res is self.NOT_SPAM:
                    continue
                elif res is True:
-                    return synapse.api.errors.Codes.FORBIDDEN
+                    return synapse.api.errors.Codes.FORBIDDEN, {}
                elif isinstance(res, synapse.api.errors.Codes):
+                    return res, {}
+                elif (
+                    isinstance(res, tuple)
+                    and len(res) == 2
+                    and isinstance(res[0], synapse.api.errors.Codes)
+                    and isinstance(res[1], dict)
+                ):
                    return res
                else:
                    logger.warning(
                        "Module returned invalid value, rejecting media file as spam"
                    )
-                    return synapse.api.errors.Codes.FORBIDDEN
+                    return synapse.api.errors.Codes.FORBIDDEN, {}

        return self.NOT_SPAM
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Erik Johnston	5de571987e	WIP: Type TreeCache	2022-07-18 13:05:07 +01:00
Erik Johnston	129691f190	Comment TreeCache	2022-07-18 10:49:12 +01:00
Erik Johnston	462db2a171	Comment LruCache	2022-07-18 10:48:24 +01:00
Erik Johnston	7f7b36d56d	Comment LruCache	2022-07-18 10:40:36 +01:00
Erik Johnston	057ae8b61c	Comments	2022-07-17 11:34:34 +01:00
Erik Johnston	7aceec3ed9	Fix up	2022-07-15 16:52:16 +01:00
Erik Johnston	cad555f07c	Better stuff	2022-07-15 16:31:53 +01:00
Erik Johnston	23c2f394a5	Fix mypy	2022-07-15 16:30:56 +01:00
Erik Johnston	602a81f5a2	don't update access	2022-07-15 16:24:10 +01:00
Erik Johnston	f046366d2a	Fix test	2022-07-15 15:54:01 +01:00
Erik Johnston	a22716c5c5	Fix literal	2022-07-15 15:31:57 +01:00
Erik Johnston	40a8fba5f6	Newsfile	2022-07-15 15:27:03 +01:00
Erik Johnston	326a175987	Make DictionaryCache have better expiry properties	2022-07-15 15:26:02 +01:00
Erik Johnston	0731e0829c	Don't pull out the full state when storing state (#13274 )	2022-07-15 12:59:45 +00:00
Patrick Cloke	3343035a06	Use a real room in the notification rotation tests. (#13260 ) Instead of manually inserting fake data. This fixes some issues with having to manually calculate stream orderings and other oddities.	2022-07-15 08:22:43 -04:00
David Robertson	7281591f4c	Use state before join to determine if we `_should_perform_remote_join` (#13270 ) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2022-07-15 12:20:47 +00:00
Sean Quah	d765ada84f	Update locked frozendict version to 2.3.2 (#13284 ) `frozendict` 2.3.2 includes a fix for a memory leak in `frozendict.__hash__`. This likely has no impact outside of the deprecated `/initialSync` endpoint, which uses `StreamToken`s, containing `RoomStreamToken`s, containing `frozendict`s, as cache keys. Signed-off-by: Sean Quah <seanq@matrix.org>	2022-07-15 13:18:51 +01:00
Richard van der Hoff	b116d3ce00	Bg update to populate new `events` table columns (#13215 ) These columns were added back in Synapse 1.52, and have been populated for new events since then. It's now (beyond) time to back-populate them for existing events.	2022-07-15 12:47:26 +01:00
Erik Johnston	7be954f59b	Fix a bug which could lead to incorrect state (#13278 ) There are two fixes here: 1. A long-standing bug where we incorrectly calculated `delta_ids`; and 2. A bug introduced in #13267 where we got current state incorrect.	2022-07-15 11:06:41 +00:00
Richard van der Hoff	512486bbeb	Docker: copy postgres from base image (#13279 ) When building the docker images for complement testing, copy a preinstalled complement over from a base image, rather than apt installing it. This avoids network traffic and is much faster.	2022-07-15 11:13:40 +01:00
Nick Mills-Barrett	cc21a431f3	Async get event cache prep (#13242 ) Some experimental prep work to enable external event caching based on #9379 & #12955. Doesn't actually move the cache at all, just lays the groundwork for async implemented caches. Signed off by Nick @ Beeper (@Fizzadar)	2022-07-15 09:30:46 +00:00
Nick Mills-Barrett	21eeacc995	Federation Sender & Appservice Pusher Stream Optimisations (#13251 ) * Replace `get_new_events_for_appservice` with `get_all_new_events_stream` The functions were near identical and this brings the AS worker closer to the way federation senders work which can allow for multiple workers to handle AS traffic. * Pull received TS alongside events when processing the stream This avoids an extra query -per event- when both federation sender and appservice pusher process events.	2022-07-15 09:36:56 +01:00
Richard van der Hoff	fe15a865a5	Rip out auth-event reconciliation code (#12943 ) There is a corner in `_check_event_auth` (long known as "the weird corner") where, if we get an event with auth_events which don't match those we were expecting, we attempt to resolve the diffence between our state and the remote's with a state resolution. This isn't specced, and there's general agreement we shouldn't be doing it. However, it turns out that the faster-joins code was relying on it, so we need to introduce something similar (but rather simpler) for that.	2022-07-14 21:52:26 +00:00
Richard van der Hoff	df55b377be	CHANGES.md: fix link to upgrade notes	2022-07-14 15:07:52 +01:00
Erik Johnston	0ca4172b5d	Don't pull out state in `compute_event_context` for unconflicted state (#13267 )	2022-07-14 13:57:02 +00:00
David Robertson	599c403d99	Allow rate limiters to passively record actions they cannot limit (#13253 ) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>	2022-07-13 19:09:42 +00:00
David Robertson	0eb7e69768	Notifier: accept callbacks to fire on room joins (#13254 )	2022-07-13 19:48:24 +01:00
Jacek Kuśnierz	cc1071598a	Call the v2 identity service `/3pid/unbind` endpoint, rather than v1. (#13240 ) * Drop support for v1 unbind Signed-off-by: Jacek Kusnierz <jacek.kusnierz@tum.de> * Add changelog Signed-off-by: Jacek Kusnierz <jacek.kusnierz@tum.de> * Update changelog.d/13240.misc	2022-07-13 19:43:17 +01:00
Shay	ad5761b65c	Add support for room version 10 (#13220 )	2022-07-13 11:36:02 -07:00
jejo86	2341032cf2	Document advising against publicly exposing the Admin API and provide a usage example (#13231 ) * Admin API request explanation improved Pointed out, that the Admin API is not accessible by default from any remote computer, but only from the PC `matrix-synapse` is running on. Added a full, working example, making sure to include the cURL flag `-X`, which needs to be prepended to `GET`, `POST`, `PUT` etc. and listing the full query string including protocol, IP address and port. * Admin API request explanation improved * Apply suggestions from code review Update changelog. Reword prose. Co-authored-by: David Robertson <david.m.robertson1@gmail.com>	2022-07-13 19:33:33 +01:00
Nick Mills-Barrett	982fe29655	Optimise room creation event lookups part 2 (#13224 )	2022-07-13 19:32:46 +01:00
Patrick Cloke	1d5c80b161	Reduce duplicate code in receipts servlets. (#13198 )	2022-07-13 13:23:16 -04:00
Brad Murray	3371e1abcb	Add prometheus counters for content types other than events (#13175 )	2022-07-13 15:18:20 +01:00
Patrick Cloke	4db7862e0f	Drop unused tables from groups/communities. (#12967 ) These tables have been unused since Synapse v1.61.0, although schema version 72 was added in Synapse v1.62.0.	2022-07-13 09:55:14 -04:00
Patrick Cloke	90e9b4fa1e	Do not fail build if complement with workers fails. (#13266 )	2022-07-13 08:30:42 -04:00
Thomas Weston	0312ff44c6	Fix "add user" admin api error when request contains a "msisdn" threepid (#13263 ) Co-authored-by: Thomas Weston <thomas.weston@clearspancloud.com> Co-authored-by: David Robertson <david.m.robertson1@gmail.com>	2022-07-13 11:33:21 +01:00
Patrick Cloke	1381563988	Inline URL preview documentation. (#13261 ) Inline URL preview documentation near the implementation.	2022-07-12 15:01:58 -04:00
Richard van der Hoff	a366b75b72	Drop unused table `event_reference_hashes` (#13218 ) This is unused since Synapse 1.60.0 (#12679). It's time for it to go.	2022-07-12 18:52:06 +00:00
Jacek Kuśnierz	7218a0ca18	Drop support for calling `/_matrix/client/v3/account/3pid/bind` without an `id_access_token` (#13239 ) Fixes #13201 Signed-off-by: Jacek Kusnierz jacek.kusnierz@tum.de	2022-07-12 18:48:29 +00:00
David Robertson	52a0c8f2f7	Rename test case method to `add_hashes_and_signatures_from_other_server` (#13255 )	2022-07-12 18:46:32 +00:00
Richard van der Hoff	fa71bb18b5	Drop support for delegating email validation (#13192 ) * Drop support for delegating email validation Delegating email validation to an IS is insecure (since it allows the owner of the IS to do a password reset on your HS), and has long been deprecated. It will now cause a config error at startup. * Update unit test which checks for email verification Give it an `email` config instead of a threepid delegate * Remove unused method `requestEmailToken` * Simplify config handling for email verification Rather than an enum and a boolean, all we need here is a single bool, which says whether we are or are not doing email verification. * update docs * changelog * upgrade.md: fix typo * update version number this will be in 1.64, not 1.63 * update version number this one too	2022-07-12 19:18:53 +01:00
Sean Quah	3f178332d6	Log the stack when waiting for an entire room to be un-partial stated (#13257 ) The stack is already logged when waiting for an event to be un-partial stated. Log the stack for rooms as well, to aid in debugging.	2022-07-12 18:57:38 +01:00
Shay	6f30eb5b8e	Add info about configuration in the url preview docs (#13233 ) Cross-link doc pages for easier navigation.	2022-07-12 13:48:47 -04:00
Quentin Gliech	b19060a29b	Make the AS login method call `Auth.get_user_by_req` for checking the AS token. (#13094 ) This gets rid of another usage of get_appservice_by_req, with all the benefits, including correctly tracking the appservice IP and setting the tracing attributes correctly. Signed-off-by: Quentin Gliech <quenting@element.io>	2022-07-12 18:06:29 +01:00
andrew do	2d82cdafd2	expose whether a room is a space in the Admin API (#13208 )	2022-07-12 15:30:53 +01:00
Sean Quah	f14c632134	Update changelog once more	2022-07-12 13:01:42 +01:00
Sean Quah	ac7aec0cd3	Reorder and tidy up changelog	2022-07-12 12:52:47 +01:00
Sean Quah	6173d585df	1.63.0rc1	2022-07-12 11:26:25 +01:00
Erik Johnston	e5716b631c	Don't pull out the full state when calculating push actions (#13078 )	2022-07-11 20:08:39 +00:00
villepeh	bc8eefc1e1	Add a sample bash script to docs for creating multiple worker files (#13032 ) Signed-off-by: Ville Petteri Huh.	2022-07-11 18:33:53 +01:00
Nick Mills-Barrett	92202ce867	Reduce event lookups during room creation by passing known event IDs (#13210 ) Inspired by the room batch handler, this uses previous event inserts to pre-populate prev events during room creation, reducing the number of queries required to create a room. Signed off by Nick @ Beeper (@Fizzadar)	2022-07-11 18:00:12 +01:00
David Teller	11f811470f	Uniformize spam-checker API, part 5: expand other spam-checker callbacks to return `Tuple[Codes, dict]` (#13044 ) Signed-off-by: David Teller <davidt@element.io> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>	2022-07-11 16:52:10 +00:00
Travis Ralston	d736d5cfad	Fix to-device messages not being sent to MSC3202-enabled appservices (#13235 ) The field name was simply incorrect, leading to errors.	2022-07-11 17:22:17 +01:00
Erik Johnston	f1711e1f5c	Remove delay when rotating event push actions (#13211 ) We want to be as up to date as possible, and sleeping doesn't help here and can mean we fall behind.	2022-07-11 16:51:30 +01:00
Andrew Morgan	5ef2f87569	Document the 'databases' homeserver config option (#13212 )	2022-07-11 14:05:24 +00:00
Erik Johnston	e610128c50	Add a `filter_event_for_clients_with_state` function (#13222 )	2022-07-11 14:14:09 +01:00
Travis Ralston	a113011794	Fix appservice EDUs failing to send if the EDU doesn't have a room ID (#13236 ) * Fix appservice EDUs failing to send if the EDU doesn't have a room ID As is in the case of presence. * changelog * linter * fix linter again	2022-07-11 14:12:28 +01:00
David Robertson	28d96cb2b4	Ensure portdb selects _all_ rows with negative rowids (#13226 )	2022-07-11 10:36:18 +01:00
Sumner Evans	739adf1551	editorconfig: add max_line_length for Python files (#13228 ) See the documentation for the property here: https://github.com/editorconfig/editorconfig/wiki/EditorConfig-Properties#max_line_length Signed-off-by: Sumner Evans <me@sumnerevans.com>	2022-07-08 16:40:25 +00:00
Erik Johnston	757bc0caef	Fix notification count after a highlighted message (#13223 ) Fixes #13196 Broke by #13005	2022-07-08 14:00:29 +01:00
Eric Eastwood	a962c5a56d	Fix exception when using MSC3030 to look for remote federated events before room creation (#13197 ) Complement tests: https://github.com/matrix-org/complement/pull/405 This happens when you have some messages imported before the room is created. Then use MSC3030 to look backwards before the room creation from a remote federated server. The server won't find anything locally, but will ask over federation which will have the remote event. The previous logic would choke on not having the local event assigned. ``` Failed to fetch /timestamp_to_event from hs2 because of exception(UnboundLocalError) local variable 'local_event' referenced before assignment args=("local variable 'local_event' referenced before assignment",) ```	2022-07-07 11:52:45 -05:00
				`@@ -0,0 +1 @@`
				`Remove code which incorrectly attempted to reconcile state with remote servers when processing incoming events.`
				`@@ -1 +0,0 @@`
				Add an explanation of the `--report-stats` argument to the docs.
				`@@ -1 +0,0 @@`
				`Implement [MSC3827](https://github.com/matrix-org/matrix-spec-proposals/pull/3827): Filtering of /publicRooms by room type.`
				`@@ -1,3 +0,0 @@`
				`Clean up references to sample configuration and redirect users to the configuration manual instead.`
				`@@ -1 +0,0 @@`
				`Enable Complement testing in the 'Twisted Trunk' CI runs.`
				`@@ -1 +0,0 @@`
				`Add documentation for anonymised homeserver statistics collection.`
				`@@ -0,0 +1 @@`
				Make the AS login method call `Auth.get_user_by_req` for checking the AS token.
				`@@ -1 +0,0 @@`
				`Faster room joins: Handle race between persisting an event and un-partial stating a room.`
				`@@ -1 +0,0 @@`
				Raise a `DependencyError` on missing dependencies instead of a `ConfigError`.