Update synapse/storage/databases/main/deviceinbox.py

Co-authored-by: Erik Johnston <erik@matrix.org>
Use org.matrix.unstable prefix
2022-08-03 14:47:26 +02:00 · 2022-08-03 14:46:46 +02:00 · 2022-08-02 15:23:52 +02:00 · 2022-08-02 15:18:57 +02:00 · 2022-08-02 15:08:12 +02:00 · 2022-08-02 15:08:12 +02:00
182 changed files with 2721 additions and 1279 deletions
@@ -135,11 +135,42 @@ jobs:
            /logs/**/*.log*


-  # TODO: run complement (as with twisted trunk, see #12473).
+  complement:
+    if: "${{ !failure() && !cancelled() }}"
+    runs-on: ubuntu-latest

-  # open an issue if the build fails, so we know about it.
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - arrangement: monolith
+            database: SQLite
+
+          - arrangement: monolith
+            database: Postgres
+
+          - arrangement: workers
+            database: Postgres
+
+    steps:
+      - name: Run actions/checkout@v2 for synapse
+        uses: actions/checkout@v2
+        with:
+          path: synapse
+
+      - name: Prepare Complement's Prerequisites
+        run: synapse/.ci/scripts/setup_complement_prerequisites.sh
+
+      - run: |
+          set -o pipefail
+          TEST_ONLY_IGNORE_POETRY_LOCKFILE=1 POSTGRES=${{ (matrix.database == 'Postgres') && 1 || '' }} WORKERS=${{ (matrix.arrangement == 'workers') && 1 || '' }} COMPLEMENT_DIR=`pwd`/complement synapse/scripts-dev/complement.sh -json 2>&1 | gotestfmt
+        shell: bash
+        name: Run Complement Tests
+
+  # Open an issue if the build fails, so we know about it.
+  # Only do this if we're not experimenting with this action in a PR.
  open-issue:
-    if: failure()
+    if: "failure() && github.event_name != 'push' && github.event_name != 'pull_request'"
    needs:
      # TODO: should mypy be included here? It feels more brittle than the other two.
      - mypy
@@ -328,6 +328,9 @@ jobs:
          - arrangement: monolith
            database: Postgres

+          - arrangement: workers
+            database: Postgres
+
    steps:
      - name: Run actions/checkout@v2 for synapse
        uses: actions/checkout@v2
@@ -343,30 +346,6 @@ jobs:
        shell: bash
        name: Run Complement Tests

-  # XXX When complement with workers is stable, move this back into the standard
-  #     "complement" matrix above.
-  #
-  # See https://github.com/matrix-org/synapse/issues/13161
-  complement-workers:
-    if: "${{ !failure() && !cancelled() }}"
-    needs: linting-done
-    runs-on: ubuntu-latest
-
-    steps:
-      - name: Run actions/checkout@v2 for synapse
-        uses: actions/checkout@v2
-        with:
-          path: synapse
-
-      - name: Prepare Complement's Prerequisites
-        run: synapse/.ci/scripts/setup_complement_prerequisites.sh
-
-      - run: |
-          set -o pipefail
-          POSTGRES=1 WORKERS=1 COMPLEMENT_DIR=`pwd`/complement synapse/scripts-dev/complement.sh -json 2>&1 | gotestfmt
-        shell: bash
-        name: Run Complement Tests
-
  # a job which marks all the other jobs as complete, thus allowing PRs to be merged.
  tests-done:
    if: ${{ always() }}
@@ -1,7 +1,124 @@
-Synapse vNext
-=============
+Synapse 1.64.0 (2022-08-02)
+===========================
+
+No significant changes since 1.64.0rc2.
+
+
+Deprecation Warning
+-------------------
+
+Synapse v1.66.0 will remove the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server.
+
+If you require your homeserver to verify e-mail addresses or to support password resets via e-mail, please configure your homeserver with SMTP access so that it can send e-mails on its own behalf.
+[Consult the configuration documentation for more information.](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#email)
+
+
+Synapse 1.64.0rc2 (2022-07-29)
+==============================
+
+This RC reintroduces support for `account_threepid_delegates.email`, which was removed in 1.64.0rc1. It remains deprecated and will be removed altogether in Synapse v1.66.0. ([\#13406](https://github.com/matrix-org/synapse/issues/13406))
+
+
+Synapse 1.64.0rc1 (2022-07-26)
+==============================
+
+This RC removed the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server.
+
+We have also stopped building `.deb` packages for Ubuntu 21.10 as it is no longer an active version of Ubuntu.
+
+
+Features
+--------
+
+- Improve error messages when media thumbnails cannot be served. ([\#13038](https://github.com/matrix-org/synapse/issues/13038))
+- Allow pagination from remote event after discovering it from [MSC3030](https://github.com/matrix-org/matrix-spec-proposals/pull/3030) `/timestamp_to_event`. ([\#13205](https://github.com/matrix-org/synapse/issues/13205))
+- Add a `room_type` field in the responses for the list room and room details admin APIs. Contributed by @andrewdoh. ([\#13208](https://github.com/matrix-org/synapse/issues/13208))
+- Add support for room version 10. ([\#13220](https://github.com/matrix-org/synapse/issues/13220))
+- Add per-room rate limiting for room joins. For each room, Synapse now monitors the rate of join events in that room, and throttles additional joins if that rate grows too large. ([\#13253](https://github.com/matrix-org/synapse/issues/13253), [\#13254](https://github.com/matrix-org/synapse/issues/13254), [\#13255](https://github.com/matrix-org/synapse/issues/13255), [\#13276](https://github.com/matrix-org/synapse/issues/13276))
+- Support Implicit TLS (TLS without using a STARTTLS upgrade, typically on port 465) for sending emails, enabled by the new option `force_tls`. Contributed by Jan Schär. ([\#13317](https://github.com/matrix-org/synapse/issues/13317))
+
+
+Bugfixes
+--------
+
+- Fix a bug introduced in Synapse 1.15.0 where adding a user through the Synapse Admin API with a phone number would fail if the `enable_email_notifs` and `email_notifs_for_new_users` options were enabled. Contributed by @thomasweston12. ([\#13263](https://github.com/matrix-org/synapse/issues/13263))
+- Fix a bug introduced in Synapse 1.40.0 where a user invited to a restricted room would be briefly unable to join. ([\#13270](https://github.com/matrix-org/synapse/issues/13270))
+- Fix a long-standing bug where, in rare instances, Synapse could store the incorrect state for a room after a state resolution. ([\#13278](https://github.com/matrix-org/synapse/issues/13278))
+- Fix a bug introduced in v1.18.0 where the `synapse_pushers` metric would overcount pushers when they are replaced. ([\#13296](https://github.com/matrix-org/synapse/issues/13296))
+- Disable autocorrection and autocapitalisation on the username text field shown during registration when using SSO. ([\#13350](https://github.com/matrix-org/synapse/issues/13350))
+- Update locked version of `frozendict` to 2.3.3, which has fixes for memory leaks affecting `/sync`. ([\#13284](https://github.com/matrix-org/synapse/issues/13284), [\#13352](https://github.com/matrix-org/synapse/issues/13352))
+
+
+Improved Documentation
+----------------------
+
+- Provide an example of using the Admin API. Contributed by @jejo86. ([\#13231](https://github.com/matrix-org/synapse/issues/13231))
+- Move the documentation for how URL previews work to the URL preview module. ([\#13233](https://github.com/matrix-org/synapse/issues/13233), [\#13261](https://github.com/matrix-org/synapse/issues/13261))
+- Add another `contrib` script to help set up worker processes. Contributed by @villepeh. ([\#13271](https://github.com/matrix-org/synapse/issues/13271))
+- Document that certain config options were added or changed in Synapse 1.62. Contributed by @behrmann. ([\#13314](https://github.com/matrix-org/synapse/issues/13314))
+- Document the new `rc_invites.per_issuer` throttling option added in Synapse 1.63. ([\#13333](https://github.com/matrix-org/synapse/issues/13333))
+- Mention that BuildKit is needed when building Docker images for tests. ([\#13338](https://github.com/matrix-org/synapse/issues/13338))
+- Improve Caddy reverse proxy documentation. ([\#13344](https://github.com/matrix-org/synapse/issues/13344))
+
+
+Deprecations and Removals
+-------------------------
+
+- Drop tables that were formerly used for groups/communities. ([\#12967](https://github.com/matrix-org/synapse/issues/12967))
+- Drop support for delegating email verification to an external server. ([\#13192](https://github.com/matrix-org/synapse/issues/13192))
+- Drop support for calling `/_matrix/client/v3/account/3pid/bind` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu. ([\#13239](https://github.com/matrix-org/synapse/issues/13239))
+- Stop building `.deb` packages for Ubuntu 21.10 (Impish Indri), which has reached end of life. ([\#13326](https://github.com/matrix-org/synapse/issues/13326))
+
+
+Internal Changes
+----------------
+
+- Use lower transaction isolation level when purging rooms to avoid serialization errors. Contributed by Nick @ Beeper. ([\#12942](https://github.com/matrix-org/synapse/issues/12942))
+- Remove code which incorrectly attempted to reconcile state with remote servers when processing incoming events. ([\#12943](https://github.com/matrix-org/synapse/issues/12943))
+- Make the AS login method call `Auth.get_user_by_req` for checking the AS token. ([\#13094](https://github.com/matrix-org/synapse/issues/13094))
+- Always use a version of canonicaljson that supports the C implementation of frozendict. ([\#13172](https://github.com/matrix-org/synapse/issues/13172))
+- Add prometheus counters for ephemeral events and to device messages pushed to app services. Contributed by Brad @ Beeper. ([\#13175](https://github.com/matrix-org/synapse/issues/13175))
+- Refactor receipts servlet logic to avoid duplicated code. ([\#13198](https://github.com/matrix-org/synapse/issues/13198))
+- Preparation for database schema simplifications: populate `state_key` and `rejection_reason` for existing rows in the `events` table. ([\#13215](https://github.com/matrix-org/synapse/issues/13215))
+- Remove unused database table `event_reference_hashes`. ([\#13218](https://github.com/matrix-org/synapse/issues/13218))
+- Further reduce queries used sending events when creating new rooms. Contributed by Nick @ Beeper (@fizzadar). ([\#13224](https://github.com/matrix-org/synapse/issues/13224))
+- Call the v2 identity service `/3pid/unbind` endpoint, rather than v1. Contributed by @Vetchu. ([\#13240](https://github.com/matrix-org/synapse/issues/13240))
+- Use an asynchronous cache wrapper for the get event cache. Contributed by Nick @ Beeper (@fizzadar). ([\#13242](https://github.com/matrix-org/synapse/issues/13242), [\#13308](https://github.com/matrix-org/synapse/issues/13308))
+- Optimise federation sender and appservice pusher event stream processing queries. Contributed by Nick @ Beeper (@fizzadar). ([\#13251](https://github.com/matrix-org/synapse/issues/13251))
+- Log the stack when waiting for an entire room to be un-partial stated. ([\#13257](https://github.com/matrix-org/synapse/issues/13257))
+- Fix spurious warning when fetching state after a missing prev event. ([\#13258](https://github.com/matrix-org/synapse/issues/13258))
+- Clean-up tests for notifications. ([\#13260](https://github.com/matrix-org/synapse/issues/13260))
+- Do not fail build if complement with workers fails. ([\#13266](https://github.com/matrix-org/synapse/issues/13266))
+- Don't pull out state in `compute_event_context` for unconflicted state. ([\#13267](https://github.com/matrix-org/synapse/issues/13267), [\#13274](https://github.com/matrix-org/synapse/issues/13274))
+- Reduce the rebuild time for the complement-synapse docker image. ([\#13279](https://github.com/matrix-org/synapse/issues/13279))
+- Don't pull out the full state when creating an event. ([\#13281](https://github.com/matrix-org/synapse/issues/13281), [\#13307](https://github.com/matrix-org/synapse/issues/13307))
+- Upgrade from Poetry 1.1.12 to 1.1.14, to fix bugs when locking packages. ([\#13285](https://github.com/matrix-org/synapse/issues/13285))
+- Make `DictionaryCache` expire full entries if they haven't been queried in a while, even if specific keys have been queried recently. ([\#13292](https://github.com/matrix-org/synapse/issues/13292))
+- Use `HTTPStatus` constants in place of literals in tests. ([\#13297](https://github.com/matrix-org/synapse/issues/13297))
+- Improve performance of query  `_get_subset_users_in_room_with_profiles`. ([\#13299](https://github.com/matrix-org/synapse/issues/13299))
+- Up batch size of `bulk_get_push_rules` and `_get_joined_profiles_from_event_ids`. ([\#13300](https://github.com/matrix-org/synapse/issues/13300))
+- Remove unnecessary `json.dumps` from tests. ([\#13303](https://github.com/matrix-org/synapse/issues/13303))
+- Reduce memory usage of sending dummy events. ([\#13310](https://github.com/matrix-org/synapse/issues/13310))
+- Prevent formatting changes of [#3679](https://github.com/matrix-org/synapse/pull/3679) from appearing in `git blame`. ([\#13311](https://github.com/matrix-org/synapse/issues/13311))
+- Change `get_users_in_room` and `get_rooms_for_user` caches to enable pruning of old entries. ([\#13313](https://github.com/matrix-org/synapse/issues/13313))
+- Validate federation destinations and log an error if a destination is invalid. ([\#13318](https://github.com/matrix-org/synapse/issues/13318))
+- Fix `FederationClient.get_pdu()` returning events from the cache as `outliers` instead of original events we saw over federation. ([\#13320](https://github.com/matrix-org/synapse/issues/13320))
+- Reduce memory usage of state caches. ([\#13323](https://github.com/matrix-org/synapse/issues/13323))
+- Reduce the amount of state we store in the `state_cache`. ([\#13324](https://github.com/matrix-org/synapse/issues/13324))
+- Add missing type hints to open tracing module. ([\#13328](https://github.com/matrix-org/synapse/issues/13328), [\#13345](https://github.com/matrix-org/synapse/issues/13345), [\#13362](https://github.com/matrix-org/synapse/issues/13362))
+- Remove old base slaved store and de-duplicate cache ID generators. Contributed by Nick @ Beeper (@fizzadar). ([\#13329](https://github.com/matrix-org/synapse/issues/13329), [\#13349](https://github.com/matrix-org/synapse/issues/13349))
+- When reporting metrics is enabled, use ~8x less data to describe DB transaction metrics. ([\#13342](https://github.com/matrix-org/synapse/issues/13342))
+- Faster room joins: skip soft fail checks while Synapse only has partial room state, since the current membership of event senders may not be accurately known. ([\#13354](https://github.com/matrix-org/synapse/issues/13354))
+
+
+Synapse 1.63.1 (2022-07-20)
+===========================
+
+Bugfixes
+--------
+
+- Fix a bug introduced in Synapse 1.63.0 where push actions were incorrectly calculated for appservice users. This caused performance issues on servers with large numbers of appservices. ([\#13332](https://github.com/matrix-org/synapse/issues/13332))

-As of this release, Synapse no longer allows the tasks of verifying email address ownership, and password reset confirmation, to be delegated to an identity server. For more information, see the [upgrade notes](https://matrix-org.github.io/synapse/v1.64/upgrade.html#upgrading-to-v1640).

 Synapse 1.63.0 (2022-07-19)
 ===========================
@@ -133,7 +250,7 @@ Bugfixes
 - Update [MSC3786](https://github.com/matrix-org/matrix-spec-proposals/pull/3786) implementation to check `state_key`. ([\#12939](https://github.com/matrix-org/synapse/issues/12939))
 - Fix a bug introduced in Synapse 1.58 where Synapse would not report full version information when installed from a git checkout. This is a best-effort affair and not guaranteed to be stable. ([\#12973](https://github.com/matrix-org/synapse/issues/12973))
 - Fix a bug introduced in Synapse 1.60 where Synapse would fail to start if the `sqlite3` module was not available. ([\#12979](https://github.com/matrix-org/synapse/issues/12979))
- Fix a bug where non-standard information was required when requesting the `/hierarchy` API over federation. Introduced 
+- Fix a bug where non-standard information was required when requesting the `/hierarchy` API over federation. Introduced
  in Synapse v1.41.0. ([\#12991](https://github.com/matrix-org/synapse/issues/12991))
 - Fix a long-standing bug which meant that rate limiting was not restrictive enough in some cases. ([\#13018](https://github.com/matrix-org/synapse/issues/13018))
 - Fix a bug introduced in Synapse 1.58 where profile requests for a malformed user ID would ccause an internal error. Synapse now returns 400 Bad Request in this situation. ([\#13041](https://github.com/matrix-org/synapse/issues/13041))
@@ -1 +0,0 @@
-Use lower isolation level when purging rooms to avoid serialization errors. Contributed by Nick @ Beeper.
@@ -1 +0,0 @@
-Remove code which incorrectly attempted to reconcile state with remote servers when processing incoming events.
@@ -1 +0,0 @@
-Drop tables used for groups/communities.
@@ -0,0 +1 @@
+Extend the release script to automatically push a new SyTest branch, rather than having that be a manual process.
@@ -1 +0,0 @@
-Provide more info why we don't have any thumbnails to serve.
@@ -1 +0,0 @@
-Make the AS login method call `Auth.get_user_by_req` for checking the AS token.
@@ -0,0 +1 @@
+Make minor clarifications to the error messages given when we fail to join a room via any server.
@@ -1 +0,0 @@
-Always use a version of canonicaljson that supports the C implementation of frozendict.
@@ -1 +0,0 @@
-Add prometheus counters for ephemeral events and to device messages pushed to app services. Contributed by Brad @ Beeper.
@@ -1 +0,0 @@
-Drop support for delegating email verification to an external server.
@@ -1 +0,0 @@
-Refactor receipts servlet logic to avoid duplicated code.
@@ -1 +0,0 @@
-Add a `room_type` field in the responses for the list room and room details admin API. Contributed by @andrewdoh.
@@ -0,0 +1 @@
+Enable Complement CI tests in the 'latest deps' test run.
@@ -1 +0,0 @@
-Preparation for database schema simplifications: populate `state_key` and `rejection_reason` for existing rows in the `events` table.
@@ -1 +0,0 @@
-Remove unused database table `event_reference_hashes`.
@@ -1 +0,0 @@
-Add support for room version 10.
@@ -0,0 +1 @@
+Document which HTTP resources support gzip compression.
@@ -1 +0,0 @@
-Further reduce queries used sending events when creating new rooms. Contributed by Nick @ Beeper (@fizzadar).
@@ -1 +0,0 @@
-Provide an example of using the Admin API. Contributed by @jejo86.
@@ -1 +0,0 @@
-Move the documentation for how URL previews work to the URL preview module.
@@ -1 +0,0 @@
-Drop support for calling `/_matrix/client/v3/account/3pid/bind` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu.
@@ -1 +0,0 @@
-Call the v2 identity service `/3pid/unbind` endpoint, rather than v1.
@@ -1 +0,0 @@
-Use an asynchronous cache wrapper for the get event cache. Contributed by Nick @ Beeper (@fizzadar).
@@ -1 +0,0 @@
-Optimise federation sender and appservice pusher event stream processing queries. Contributed by Nick @ Beeper (@fizzadar).
@@ -1 +0,0 @@
-Preparatory work for a per-room rate limiter on joins.
@@ -1 +0,0 @@
-Preparatory work for a per-room rate limiter on joins.
@@ -1 +0,0 @@
-Preparatory work for a per-room rate limiter on joins.
@@ -1 +0,0 @@
-Log the stack when waiting for an entire room to be un-partial stated.
@@ -1 +0,0 @@
-Fix spurious warning when fetching state after a missing prev event.
@@ -1 +0,0 @@
-Clean-up tests for notifications.
@@ -1 +0,0 @@
-Move the documentation for how URL previews work to the URL preview module.
@@ -1 +0,0 @@
-Fix a bug introduced in Synapse 1.15.0 where adding a user through the Synapse Admin API with a phone number would fail if the "enable_email_notifs" and "email_notifs_for_new_users" options were enabled. Contributed by @thomasweston12.
@@ -1 +0,0 @@
-Do not fail build if complement with workers fails.
@@ -1 +0,0 @@
-Don't pull out state in `compute_event_context` for unconflicted state.
@@ -1 +0,0 @@
-Fix a bug introduced in Synapse 1.40 where a user invited to a restricted room would be briefly unable to join.
@@ -1 +0,0 @@
-Add another `contrib` script to help set up worker processes. Contributed by @villepeh.
@@ -1 +0,0 @@
-Don't pull out state in `compute_event_context` for unconflicted state.
@@ -1 +0,0 @@
-Add per-room rate limiting for room joins. For each room, Synapse now monitors the rate of join events in that room, and throttle additional joins if that rate grows too large.
@@ -1 +0,0 @@
-Fix long-standing bug where in rare instances Synapse could store the incorrect state for a room after a state resolution.
@@ -1 +0,0 @@
-Reduce the rebuild time for the complement-synapse docker image.
@@ -1 +0,0 @@
-Don't pull out the full state when creating an event.
@@ -1 +0,0 @@
-Update locked version of `frozendict` to 2.3.2, which has a fix for a memory leak.
@@ -1 +0,0 @@
-Upgrade from Poetry 1.1.14 to 1.1.12, to fix bugs when locking packages.
@@ -1 +0,0 @@
-Fix a bug introduced in v1.18.0 where the `synapse_pushers` metric would overcount pushers when they are replaced.
@@ -1 +0,0 @@
-Use `HTTPStatus` constants in place of literals in tests.
@@ -1 +0,0 @@
-Improve performance of query  `_get_subset_users_in_room_with_profiles`.
@@ -1 +0,0 @@
-Up batch size of `bulk_get_push_rules` and `_get_joined_profiles_from_event_ids`.
@@ -1 +0,0 @@
-Remove unnecessary `json.dumps` from tests.
@@ -1 +0,0 @@
-Don't pull out the full state when creating an event.
@@ -1 +0,0 @@
-Use an asynchronous cache wrapper for the get event cache. Contributed by Nick @ Beeper (@fizzadar).
@@ -1 +0,0 @@
-Reduce memory usage of sending dummy events.
@@ -1 +0,0 @@
-Prevent formatting changes of [#3679](https://github.com/matrix-org/synapse/pull/3679) from appearing in `git blame`.
@@ -1 +0,0 @@
-Add notes when config options where changed. Contributed by @behrmann.
@@ -1 +0,0 @@
-Reduce memory usage of state caches.
@@ -1 +0,0 @@
-Stop builindg `.deb` packages for Ubuntu 21.10 (Impish Indri), which has reached end of life.
@@ -1 +0,0 @@
-Add type hints to `trace` decorator.
@@ -0,0 +1 @@
+Add new unstable error codes `ORG.MATRIX.MSC3848.ALREADY_JOINED`, `ORG.MATRIX.MSC3848.NOT_JOINED`, and `ORG.MATRIX.MSC3848.INSUFFICIENT_POWER` described in MSC3848.
@@ -0,0 +1 @@
+Fix long-standing bugged logic which was never hit in `get_pdu` asking every remote destination even after it finds an event.
@@ -0,0 +1 @@
+Fix a bug in the experimental faster-room-joins support which could cause it to get stuck in an infinite loop.
@@ -0,0 +1 @@
+Faster room joins: avoid blocking when pulling events with partially missing prev events.
@@ -0,0 +1 @@
+Fix a bug introduced in Synapse v1.41.0 where the `/hierarchy` API returned non-standard information (a `room_id` field under each entry in `children_state`).
@@ -0,0 +1 @@
+Use stable prefixes for [MSC3827](https://github.com/matrix-org/matrix-spec-proposals/pull/3827).
@@ -0,0 +1 @@
+Remove an unused argument to `get_relations_for_event`.
@@ -0,0 +1 @@
+Fix bug in handling of typing events for appservices. Contributed by Nick @ Beeper (@fizzadar).
@@ -0,0 +1 @@
+Adding missing type hints to tests.
@@ -0,0 +1 @@
+Faster Room Joins: don't leave a stuck room partial state flag if the join fails.
@@ -0,0 +1 @@
+Refactor `_resolve_state_at_missing_prevs` to compute an `EventContext` instead.
@@ -0,0 +1 @@
+Add `org.matrix.unstable.to_device_limit` field to sync filter. This is an experiment to see if it improves sync performance.
@@ -0,0 +1 @@
+Faster room joins: fix a bug which caused rejected events to become un-rejected during state syncing.
@@ -0,0 +1 @@
+Re-enable running Complement tests against Synapse with workers.
@@ -0,0 +1 @@
+Refactor `_resolve_state_at_missing_prevs` to compute an `EventContext` instead.
@@ -0,0 +1 @@
+Faster room joins: Fix error when running out of servers to sync partial state with, so that Synapse raises the intended error instead.
@@ -1,3 +1,27 @@
+matrix-synapse-py3 (1.64.0) stable; urgency=medium
+
+  * New Synapse release 1.64.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 02 Aug 2022 10:32:30 +0100
+
+matrix-synapse-py3 (1.64.0~rc2) stable; urgency=medium
+
+  * New Synapse release 1.64.0rc2.
+
+ -- Synapse Packaging team <packages@matrix.org>  Fri, 29 Jul 2022 12:22:53 +0100
+
+matrix-synapse-py3 (1.64.0~rc1) stable; urgency=medium
+
+  * New Synapse release 1.64.0rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 26 Jul 2022 12:11:49 +0100
+
+matrix-synapse-py3 (1.63.1) stable; urgency=medium
+
+  * New Synapse release 1.63.1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Wed, 20 Jul 2022 13:36:52 +0100
+
 matrix-synapse-py3 (1.63.0) stable; urgency=medium

  * Clarify that homeserver server names are included in the data reported
@@ -68,7 +68,18 @@ COPY pyproject.toml poetry.lock /synapse/
 # reason, such as when a git repository is used directly as a dependency.
 ARG TEST_ONLY_SKIP_DEP_HASH_VERIFICATION

-RUN /root/.local/bin/poetry export --extras all -o /synapse/requirements.txt ${TEST_ONLY_SKIP_DEP_HASH_VERIFICATION:+--without-hashes}
+# If specified, we won't use the Poetry lockfile.
+# Instead, we'll just install what a regular `pip install` would from PyPI.
+ARG TEST_ONLY_IGNORE_POETRY_LOCKFILE
+
+# Export the dependencies, but only if we're actually going to use the Poetry lockfile.
+# Otherwise, just create an empty requirements file so that the Dockerfile can
+# proceed.
+RUN if [ -z "$TEST_ONLY_IGNORE_POETRY_LOCKFILE" ]; then \
+    /root/.local/bin/poetry export --extras all -o /synapse/requirements.txt ${TEST_ONLY_SKIP_DEP_HASH_VERIFICATION:+--without-hashes}; \
+  else \
+    touch /synapse/requirements.txt; \
+  fi

 ###
 ### Stage 1: builder
@@ -108,8 +119,17 @@ COPY synapse /synapse/synapse/
 # ... and what we need to `pip install`.
 COPY pyproject.toml README.rst /synapse/

+# Repeat of earlier build argument declaration, as this is a new build stage.
+ARG TEST_ONLY_IGNORE_POETRY_LOCKFILE
+
 # Install the synapse package itself.
-RUN pip install --prefix="/install" --no-deps --no-warn-script-location /synapse
+# If we have populated requirements.txt, we don't install any dependencies
+# as we should already have those from the previous `pip install` step.
+RUN if [ -z "$TEST_ONLY_IGNORE_POETRY_LOCKFILE" ]; then \
+    pip install --prefix="/install" --no-deps --no-warn-script-location /synapse[all]; \
+  else \
+    pip install --prefix="/install" --no-warn-script-location /synapse[all]; \
+  fi

 ###
 ### Stage 2: runtime
@@ -1,3 +1,4 @@
+# syntax=docker/dockerfile:1
 # Inherit from the official Synapse docker image
 ARG SYNAPSE_VERSION=latest
 FROM matrixdotorg/synapse:$SYNAPSE_VERSION
@@ -22,6 +22,10 @@ Consult the [contributing guide][guideComplementSh] for instructions on how to u
 Under some circumstances, you may wish to build the images manually.
 The instructions below will lead you to doing that.

+Note that these images can only be built using [BuildKit](https://docs.docker.com/develop/develop-images/build_enhancements/),
+therefore BuildKit needs to be enabled when calling `docker build`. This can be done by
+setting `DOCKER_BUILDKIT=1` in your environment.
+
 Start by building the base Synapse docker image. If you wish to run tests with the latest
 release of Synapse, instead of your current checkout, you can skip this step. From the
 root of the repository:
@@ -1,3 +1,4 @@
+# syntax=docker/dockerfile:1
 # This dockerfile builds on top of 'docker/Dockerfile-workers' in matrix-org/synapse
 # by including a built-in postgres instance, as well as setting up the homeserver so
 # that it is ready for testing via Complement.
@@ -79,63 +79,32 @@ server {
 }
 ```

-### Caddy v1
-
-```
-matrix.example.com {
-  proxy /_matrix http://localhost:8008 {
-    transparent
-  }
-
-  proxy /_synapse/client http://localhost:8008 {
-    transparent
-  }
-}
-
-example.com:8448 {
-  proxy / http://localhost:8008 {
-    transparent
-  }
-}
-```
-
 ### Caddy v2

 ```
 matrix.example.com {
-  reverse_proxy /_matrix/* http://localhost:8008
-  reverse_proxy /_synapse/client/* http://localhost:8008
+  reverse_proxy /_matrix/* localhost:8008
+  reverse_proxy /_synapse/client/* localhost:8008
 }

 example.com:8448 {
-  reverse_proxy http://localhost:8008
+  reverse_proxy localhost:8008
 }
 ```
+
 [Delegation](delegate.md) example:
+
 ```
-(matrix-well-known-header) {
-    # Headers
-    header Access-Control-Allow-Origin "*"
-    header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
-    header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
-    header Content-Type "application/json"
-}
-
 example.com {
-    handle /.well-known/matrix/server {
-        import matrix-well-known-header
-        respond `{"m.server":"matrix.example.com:443"}`
-    }
-
-    handle /.well-known/matrix/client {
-        import matrix-well-known-header
-        respond `{"m.homeserver":{"base_url":"https://matrix.example.com"},"m.identity_server":{"base_url":"https://identity.example.com"}}`
-    }
+	header /.well-known/matrix/* Content-Type application/json
+	header /.well-known/matrix/* Access-Control-Allow-Origin *
+	respond /.well-known/matrix/server `{"m.server": "matrix.example.com:443"}`
+	respond /.well-known/matrix/client `{"m.homeserver":{"base_url":"https://matrix.example.com"},"m.identity_server":{"base_url":"https://identity.example.com"}}`
 }

 matrix.example.com {
-    reverse_proxy /_matrix/* http://localhost:8008
-    reverse_proxy /_synapse/client/* http://localhost:8008
+    reverse_proxy /_matrix/* localhost:8008
+    reverse_proxy /_synapse/client/* localhost:8008
 }
 ```

@@ -91,18 +91,15 @@ process, for example:

 # Upgrading to v1.64.0

-## Delegation of email validation no longer supported
+## Deprecation of the ability to delegate e-mail verification to identity servers

-As of this version, Synapse no longer allows the tasks of verifying email address
-ownership, and password reset confirmation, to be delegated to an identity server.
+Synapse v1.66.0 will remove the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server.

-To continue to allow users to add email addresses to their homeserver accounts,
-and perform password resets, make sure that Synapse is configured with a
-working email server in the `email` configuration section (including, at a
-minimum, a `notif_from` setting.)
+If you require your homeserver to verify e-mail addresses or to support password resets via e-mail, please configure your homeserver with SMTP access so that it can send e-mails on its own behalf.
+[Consult the configuration documentation for more information.](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#email)
+
+The option that will be removed is `account_threepid_delegates.email`.

-Specifying an `email` setting under `account_threepid_delegates` will now cause
-an error at startup.

 ## Changes to the event replication streams

@@ -114,6 +111,15 @@ vice versa.
 Once all workers are upgraded to v1.64 (or downgraded to v1.63), event
 replication will resume as normal.

+## frozendict release
+
+[frozendict 2.3.3](https://github.com/Marco-Sulla/python-frozendict/releases/tag/v2.3.3)
+has recently been released, which fixes a memory leak that occurs during `/sync`
+requests. We advise server administrators who installed Synapse via pip to upgrade
+frozendict with `pip install --upgrade frozendict`. The Docker image
+`matrixdotorg/synapse` and the Debian packages from `packages.matrix.org` already
+include the updated library.
+
 # Upgrading to v1.62.0

 ## New signatures for spam checker callbacks
@@ -84,9 +84,6 @@ disallow_untyped_defs = False
 [mypy-synapse.http.matrixfederationclient]
 disallow_untyped_defs = False

-[mypy-synapse.logging.opentracing]
-disallow_untyped_defs = False
-
 [mypy-synapse.metrics._reactor_metrics]
 disallow_untyped_defs = False
 # This module imports select.epoll. That exists on Linux, but doesn't on macOS.
@@ -290,7 +290,7 @@ importlib-metadata = {version = "*", markers = "python_version < \"3.8\""}

 [[package]]
 name = "frozendict"
-version = "2.3.2"
+version = "2.3.3"
 description = "A simple immutable dictionary"
 category = "main"
 optional = false
@@ -1753,23 +1753,23 @@ flake8-comprehensions = [
    {file = "flake8_comprehensions-3.8.0-py3-none-any.whl", hash = "sha256:9406314803abe1193c064544ab14fdc43c58424c0882f6ff8a581eb73fc9bb58"},
 ]
 frozendict = [
-    {file = "frozendict-2.3.2-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:4fb171d1e84d17335365877e19d17440373b47ca74a73c06f65ac0b16d01e87f"},
-    {file = "frozendict-2.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c0a3640e9d7533d164160b758351aa49d9e85bbe0bd76d219d4021e90ffa6a52"},
-    {file = "frozendict-2.3.2-cp310-cp310-win_amd64.whl", hash = "sha256:87cfd00fafbc147d8cd2590d1109b7db8ac8d7d5bdaa708ba46caee132b55d4d"},
-    {file = "frozendict-2.3.2-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:fb09761e093cfabb2f179dbfdb2521e1ec5701df714d1eb5c51fa7849027be19"},
-    {file = "frozendict-2.3.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:82176dc7adf01cf8f0193e909401939415a230a1853f4a672ec1629a06ceae18"},
-    {file = "frozendict-2.3.2-cp36-cp36m-win_amd64.whl", hash = "sha256:c1c70826aa4a50fa283fe161834ac4a3ac7c753902c980bb8b595b0998a38ddb"},
-    {file = "frozendict-2.3.2-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:1db5035ddbed995badd1a62c4102b5e207b5aeb24472df2c60aba79639d7996b"},
-    {file = "frozendict-2.3.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4246fc4cb1413645ba4d3513939b90d979a5bae724be605a10b2b26ee12f839c"},
-    {file = "frozendict-2.3.2-cp37-cp37m-win_amd64.whl", hash = "sha256:680cd42fb0a255da1ce45678ccbd7f69da750d5243809524ebe8f45b2eda6e6b"},
-    {file = "frozendict-2.3.2-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:6a7f3a181d6722c92a9fab12d0c5c2b006a18ca5666098531f316d1e1c8984e3"},
-    {file = "frozendict-2.3.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b1cb866eabb3c1384a7fe88e1e1033e2b6623073589012ab637c552bf03f6364"},
-    {file = "frozendict-2.3.2-cp38-cp38-win_amd64.whl", hash = "sha256:952c5e5e664578c5c2ce8489ee0ab6a1855da02b58ef593ee728fc10d672641a"},
-    {file = "frozendict-2.3.2-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:608b77904cd0117cd816df605a80d0043a5326ee62529327d2136c792165a823"},
-    {file = "frozendict-2.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0eed41fd326f0bcc779837d8d9e1374da1bc9857fe3b9f2910195bbd5fff3aeb"},
-    {file = "frozendict-2.3.2-cp39-cp39-win_amd64.whl", hash = "sha256:bde28db6b5868dd3c45b3555f9d1dc5a1cca6d93591502fa5dcecce0dde6a335"},
-    {file = "frozendict-2.3.2-py3-none-any.whl", hash = "sha256:6882a9bbe08ab9b5ff96ce11bdff3fe40b114b9813bc6801261e2a7b45e20012"},
-    {file = "frozendict-2.3.2.tar.gz", hash = "sha256:7fac4542f0a13fbe704db4942f41ba3abffec5af8b100025973e59dff6a09d0d"},
+    {file = "frozendict-2.3.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:39942914c1217a5a49c7551495a103b3dbd216e19413687e003b859c6b0ebc12"},
+    {file = "frozendict-2.3.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5589256058b31f2b91419fa30b8dc62dbdefe7710e688a3fd5b43849161eecc9"},
+    {file = "frozendict-2.3.3-cp310-cp310-win_amd64.whl", hash = "sha256:35eb7e59e287c41f4f712d4d3d2333354175b155d217b97c99c201d2d8920790"},
+    {file = "frozendict-2.3.3-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:310aaf81793abf4f471895e6fe65e0e74a28a2aaf7b25c2ba6ccd4e35af06842"},
+    {file = "frozendict-2.3.3-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c353c11010a986566a0cb37f9a783c560ffff7d67d5e7fd52221fb03757cdc43"},
+    {file = "frozendict-2.3.3-cp36-cp36m-win_amd64.whl", hash = "sha256:15b5f82aad108125336593cec1b6420c638bf45f449c57e50949fc7654ea5a41"},
+    {file = "frozendict-2.3.3-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:a4737e5257756bd6b877504ff50185b705db577b5330d53040a6cf6417bb3cdb"},
+    {file = "frozendict-2.3.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:80a14c11e33e8b0bc09e07bba3732c77a502c39edb8c3959fd9a0e490e031158"},
+    {file = "frozendict-2.3.3-cp37-cp37m-win_amd64.whl", hash = "sha256:027952d1698ac9c766ef43711226b178cdd49d2acbdff396936639ad1d2a5615"},
+    {file = "frozendict-2.3.3-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:ef818d66c85098a37cf42509545a4ba7dd0c4c679d6262123a8dc14cc474bab7"},
+    {file = "frozendict-2.3.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:812279f2b270c980112dc4e367b168054f937108f8044eced4199e0ab2945a37"},
+    {file = "frozendict-2.3.3-cp38-cp38-win_amd64.whl", hash = "sha256:c1fb7efbfebc2075f781be3d9774e4ba6ce4fc399148b02097f68d4b3c4bc00a"},
+    {file = "frozendict-2.3.3-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:a0b46d4bf95bce843c0151959d54c3e5b8d0ce29cb44794e820b3ec980d63eee"},
+    {file = "frozendict-2.3.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:38c4660f37fcc70a32ff997fe58e40b3fcc60b2017b286e33828efaa16b01308"},
+    {file = "frozendict-2.3.3-cp39-cp39-win_amd64.whl", hash = "sha256:919e3609844fece11ab18bcbf28a3ed20f8108ad4149d7927d413687f281c6c9"},
+    {file = "frozendict-2.3.3-py3-none-any.whl", hash = "sha256:f988b482d08972a196664718167a993a61c9e9f6fe7b0ca2443570b5f20ca44a"},
+    {file = "frozendict-2.3.3.tar.gz", hash = "sha256:398539c52af3c647d103185bbaa1291679f0507ad035fe3bab2a8b0366d52cf1"},
 ]
 gitdb = [
    {file = "gitdb-4.0.9-py3-none-any.whl", hash = "sha256:8033ad4e853066ba6ca92050b9df2f89301b8fc8bf7e9324d412a63f8bf1a8fd"},
@@ -54,7 +54,7 @@ skip_gitignore = true

 [tool.poetry]
 name = "matrix-synapse"
-version = "1.63.0"
+version = "1.64.0"
 description = "Homeserver for the Matrix decentralised comms protocol"
 authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
 license = "Apache-2.0"
@@ -101,6 +101,7 @@ if [ -z "$skip_docker_build" ]; then
    echo_if_github "::group::Build Docker image: matrixdotorg/synapse"
    docker build -t matrixdotorg/synapse \
      --build-arg TEST_ONLY_SKIP_DEP_HASH_VERIFICATION \
+      --build-arg TEST_ONLY_IGNORE_POETRY_LOCKFILE \
      -f "docker/Dockerfile" .
    echo_if_github "::endgroup::"

@@ -55,9 +55,12 @@ def run_until_successful(
 def cli() -> None:
    """An interactive script to walk through the parts of creating a release.

-    Requires the dev dependencies be installed, which can be done via:
+    Requirements:
+      - The dev dependencies be installed, which can be done via:

-        pip install -e .[dev]
+            pip install -e .[dev]
+
+      - A checkout of the sytest repository at ../sytest

    Then to use:

@@ -89,10 +92,12 @@ def prepare() -> None:
    """

    # Make sure we're in a git repo.
-    repo = get_repo_and_check_clean_checkout()
+    synapse_repo = get_repo_and_check_clean_checkout()
+    sytest_repo = get_repo_and_check_clean_checkout("../sytest", "sytest")

-    click.secho("Updating git repo...")
-    repo.remote().fetch()
+    click.secho("Updating Synapse and Sytest git repos...")
+    synapse_repo.remote().fetch()
+    sytest_repo.remote().fetch()

    # Get the current version and AST from root Synapse module.
    current_version = get_package_version()
@@ -166,12 +171,12 @@ def prepare() -> None:
    assert not parsed_new_version.is_postrelease

    release_branch_name = get_release_branch_name(parsed_new_version)
-    release_branch = find_ref(repo, release_branch_name)
+    release_branch = find_ref(synapse_repo, release_branch_name)
    if release_branch:
        if release_branch.is_remote():
            # If the release branch only exists on the remote we check it out
            # locally.
-            repo.git.checkout(release_branch_name)
+            synapse_repo.git.checkout(release_branch_name)
    else:
        # If a branch doesn't exist we create one. We ask which one branch it
        # should be based off, defaulting to sensible values depending on the
@@ -187,25 +192,34 @@ def prepare() -> None:
            "Which branch should the release be based on?", default=default
        )

-        base_branch = find_ref(repo, branch_name)
-        if not base_branch:
-            print(f"Could not find base branch {branch_name}!")
-            click.get_current_context().abort()
+        for repo_name, repo in {"synapse": synapse_repo, "sytest": sytest_repo}.items():
+            base_branch = find_ref(repo, branch_name)
+            if not base_branch:
+                print(f"Could not find base branch {branch_name} for {repo_name}!")
+                click.get_current_context().abort()

-        # Check out the base branch and ensure it's up to date
-        repo.head.set_reference(base_branch, "check out the base branch")
-        repo.head.reset(index=True, working_tree=True)
-        if not base_branch.is_remote():
-            update_branch(repo)
+            # Check out the base branch and ensure it's up to date
+            repo.head.set_reference(
+                base_branch, f"check out the base branch for {repo_name}"
+            )
+            repo.head.reset(index=True, working_tree=True)
+            if not base_branch.is_remote():
+                update_branch(repo)

-        # Create the new release branch
-        # Type ignore will no longer be needed after GitPython 3.1.28.
-        # See https://github.com/gitpython-developers/GitPython/pull/1419
-        repo.create_head(release_branch_name, commit=base_branch)  # type: ignore[arg-type]
+            # Create the new release branch
+            # Type ignore will no longer be needed after GitPython 3.1.28.
+            # See https://github.com/gitpython-developers/GitPython/pull/1419
+            repo.create_head(release_branch_name, commit=base_branch)  # type: ignore[arg-type]
+
+        # Special-case SyTest: we don't actually prepare any files so we may
+        # as well push it now (and only when we create a release branch;
+        # not on subsequent RCs or full releases).
+        if click.confirm("Push new SyTest branch?", default=True):
+            sytest_repo.git.push("-u", sytest_repo.remote().name, release_branch_name)

    # Switch to the release branch and ensure it's up to date.
-    repo.git.checkout(release_branch_name)
-    update_branch(repo)
+    synapse_repo.git.checkout(release_branch_name)
+    update_branch(synapse_repo)

    # Update the version specified in pyproject.toml.
    subprocess.check_output(["poetry", "version", new_version])
@@ -230,15 +244,15 @@ def prepare() -> None:
    run_until_successful('dch -M -r -D stable ""', shell=True)

    # Show the user the changes and ask if they want to edit the change log.
-    repo.git.add("-u")
+    synapse_repo.git.add("-u")
    subprocess.run("git diff --cached", shell=True)

    if click.confirm("Edit changelog?", default=False):
        click.edit(filename="CHANGES.md")

    # Commit the changes.
-    repo.git.add("-u")
-    repo.git.commit("-m", new_version)
+    synapse_repo.git.add("-u")
+    synapse_repo.git.commit("-m", new_version)

    # We give the option to bail here in case the user wants to make sure things
    # are OK before pushing.
@@ -246,17 +260,21 @@ def prepare() -> None:
        print("")
        print("Run when ready to push:")
        print("")
-        print(f"\tgit push -u {repo.remote().name} {repo.active_branch.name}")
+        print(
+            f"\tgit push -u {synapse_repo.remote().name} {synapse_repo.active_branch.name}"
+        )
        print("")
        sys.exit(0)

    # Otherwise, push and open the changelog in the browser.
-    repo.git.push("-u", repo.remote().name, repo.active_branch.name)
+    synapse_repo.git.push(
+        "-u", synapse_repo.remote().name, synapse_repo.active_branch.name
+    )

    print("Opening the changelog in your browser...")
    print("Please ask others to give it a check.")
    click.launch(
-        f"https://github.com/matrix-org/synapse/blob/{repo.active_branch.name}/CHANGES.md"
+        f"https://github.com/matrix-org/synapse/blob/{synapse_repo.active_branch.name}/CHANGES.md"
    )


@@ -469,14 +487,18 @@ def get_release_branch_name(version_number: version.Version) -> str:
    return f"release-v{version_number.major}.{version_number.minor}"


-def get_repo_and_check_clean_checkout() -> git.Repo:
+def get_repo_and_check_clean_checkout(
+    path: str = ".", name: str = "synapse"
+) -> git.Repo:
    """Get the project repo and check it's not got any uncommitted changes."""
    try:
-        repo = git.Repo()
+        repo = git.Repo(path=path)
    except git.InvalidGitRepositoryError:
-        raise click.ClickException("Not in Synapse repo.")
+        raise click.ClickException(
+            f"{path} is not a git repository (expecting a {name} repository)."
+        )
    if repo.is_dirty():
-        raise click.ClickException("Uncommitted changes exist.")
+        raise click.ClickException(f"Uncommitted changes exist in {path}.")
    return repo


@@ -26,6 +26,7 @@ from synapse.api.errors import (
    Codes,
    InvalidClientTokenError,
    MissingClientTokenError,
+    UnstableSpecAuthError,
 )
 from synapse.appservice import ApplicationService
 from synapse.http import get_request_user_agent
@@ -106,8 +107,11 @@ class Auth:
                forgot = await self.store.did_forget(user_id, room_id)
                if not forgot:
                    return membership, member_event_id
-
-        raise AuthError(403, "User %s not in room %s" % (user_id, room_id))
+        raise UnstableSpecAuthError(
+            403,
+            "User %s not in room %s" % (user_id, room_id),
+            errcode=Codes.NOT_JOINED,
+        )

    async def get_user_by_req(
        self,
@@ -600,8 +604,9 @@ class Auth:
                == HistoryVisibility.WORLD_READABLE
            ):
                return Membership.JOIN, None
-            raise AuthError(
+            raise UnstableSpecAuthError(
                403,
                "User %s not in room %s, and room previews are disabled"
                % (user_id, room_id),
+                errcode=Codes.NOT_JOINED,
            )
@@ -268,4 +268,4 @@ class PublicRoomsFilterFields:
    """

    GENERIC_SEARCH_TERM: Final = "generic_search_term"
-    ROOM_TYPES: Final = "org.matrix.msc3827.room_types"
+    ROOM_TYPES: Final = "room_types"
@@ -26,6 +26,7 @@ from twisted.web import http
 from synapse.util import json_decoder

 if typing.TYPE_CHECKING:
+    from synapse.config.homeserver import HomeServerConfig
    from synapse.types import JsonDict

 logger = logging.getLogger(__name__)
@@ -80,6 +81,12 @@ class Codes(str, Enum):
    INVALID_SIGNATURE = "M_INVALID_SIGNATURE"
    USER_DEACTIVATED = "M_USER_DEACTIVATED"

+    # Part of MSC3848
+    # https://github.com/matrix-org/matrix-spec-proposals/pull/3848
+    ALREADY_JOINED = "ORG.MATRIX.MSC3848.ALREADY_JOINED"
+    NOT_JOINED = "ORG.MATRIX.MSC3848.NOT_JOINED"
+    INSUFFICIENT_POWER = "ORG.MATRIX.MSC3848.INSUFFICIENT_POWER"
+
    # The account has been suspended on the server.
    # By opposition to `USER_DEACTIVATED`, this is a reversible measure
    # that can possibly be appealed and reverted.
@@ -167,7 +174,7 @@ class SynapseError(CodeMessageException):
        else:
            self._additional_fields = dict(additional_fields)

-    def error_dict(self) -> "JsonDict":
+    def error_dict(self, config: Optional["HomeServerConfig"]) -> "JsonDict":
        return cs_error(self.msg, self.errcode, **self._additional_fields)


@@ -213,7 +220,7 @@ class ConsentNotGivenError(SynapseError):
        )
        self._consent_uri = consent_uri

-    def error_dict(self) -> "JsonDict":
+    def error_dict(self, config: Optional["HomeServerConfig"]) -> "JsonDict":
        return cs_error(self.msg, self.errcode, consent_uri=self._consent_uri)


@@ -307,6 +314,37 @@ class AuthError(SynapseError):
        super().__init__(code, msg, errcode, additional_fields)


+class UnstableSpecAuthError(AuthError):
+    """An error raised when a new error code is being proposed to replace a previous one.
+    This error will return a "org.matrix.unstable.errcode" property with the new error code,
+    with the previous error code still being defined in the "errcode" property.
+
+    This error will include `org.matrix.msc3848.unstable.errcode` in the C-S error body.
+    """
+
+    def __init__(
+        self,
+        code: int,
+        msg: str,
+        errcode: str,
+        previous_errcode: str = Codes.FORBIDDEN,
+        additional_fields: Optional[dict] = None,
+    ):
+        self.previous_errcode = previous_errcode
+        super().__init__(code, msg, errcode, additional_fields)
+
+    def error_dict(self, config: Optional["HomeServerConfig"]) -> "JsonDict":
+        fields = {}
+        if config is not None and config.experimental.msc3848_enabled:
+            fields["org.matrix.msc3848.unstable.errcode"] = self.errcode
+        return cs_error(
+            self.msg,
+            self.previous_errcode,
+            **fields,
+            **self._additional_fields,
+        )
+
+
 class InvalidClientCredentialsError(SynapseError):
    """An error raised when there was a problem with the authorisation credentials
    in a client request.
@@ -338,8 +376,8 @@ class InvalidClientTokenError(InvalidClientCredentialsError):
        super().__init__(msg=msg, errcode="M_UNKNOWN_TOKEN")
        self._soft_logout = soft_logout

-    def error_dict(self) -> "JsonDict":
-        d = super().error_dict()
+    def error_dict(self, config: Optional["HomeServerConfig"]) -> "JsonDict":
+        d = super().error_dict(config)
        d["soft_logout"] = self._soft_logout
        return d

@@ -362,7 +400,7 @@ class ResourceLimitError(SynapseError):
        self.limit_type = limit_type
        super().__init__(code, msg, errcode=errcode)

-    def error_dict(self) -> "JsonDict":
+    def error_dict(self, config: Optional["HomeServerConfig"]) -> "JsonDict":
        return cs_error(
            self.msg,
            self.errcode,
@@ -397,7 +435,7 @@ class InvalidCaptchaError(SynapseError):
        super().__init__(code, msg, errcode)
        self.error_url = error_url

-    def error_dict(self) -> "JsonDict":
+    def error_dict(self, config: Optional["HomeServerConfig"]) -> "JsonDict":
        return cs_error(self.msg, self.errcode, error_url=self.error_url)


@@ -414,7 +452,7 @@ class LimitExceededError(SynapseError):
        super().__init__(code, msg, errcode)
        self.retry_after_ms = retry_after_ms

-    def error_dict(self) -> "JsonDict":
+    def error_dict(self, config: Optional["HomeServerConfig"]) -> "JsonDict":
        return cs_error(self.msg, self.errcode, retry_after_ms=self.retry_after_ms)


@@ -429,7 +467,7 @@ class RoomKeysVersionError(SynapseError):
        super().__init__(403, "Wrong room_keys version", Codes.WRONG_ROOM_KEYS_VERSION)
        self.current_version = current_version

-    def error_dict(self) -> "JsonDict":
+    def error_dict(self, config: Optional["HomeServerConfig"]) -> "JsonDict":
        return cs_error(self.msg, self.errcode, current_version=self.current_version)


@@ -469,7 +507,7 @@ class IncompatibleRoomVersionError(SynapseError):

        self._room_version = room_version

-    def error_dict(self) -> "JsonDict":
+    def error_dict(self, config: Optional["HomeServerConfig"]) -> "JsonDict":
        return cs_error(self.msg, self.errcode, room_version=self._room_version)


@@ -515,7 +553,7 @@ class UnredactedContentDeletedError(SynapseError):
        )
        self.content_keep_ms = content_keep_ms

-    def error_dict(self) -> "JsonDict":
+    def error_dict(self, config: Optional["HomeServerConfig"]) -> "JsonDict":
        extra = {}
        if self.content_keep_ms is not None:
            extra = {"fi.mau.msc2815.content_keep_ms": self.content_keep_ms}
@@ -134,6 +134,9 @@ USER_FILTER_SCHEMA = {
                "pattern": r"^((?!\\\\).)*$",
            },
        },
+        # This is an experiment, a MSC will follow if it happens to be useful
+        # for clients sync performance
+        "org.matrix.unstable.to_device_limit": {"type": "number"},
    },
    "additionalProperties": False,
 }
@@ -219,6 +222,15 @@ class FilterCollection:
        self.event_fields = filter_json.get("event_fields", [])
        self.event_format = filter_json.get("event_format", "client")

+        self.to_device_limit = 100
+        if hs.config.experimental.to_device_limit_enabled:
+            self.to_device_limit = filter_json.get(
+                "org.matrix.unstable.to_device_limit", 100
+            )
+            # We don't want to overload the server so let's limit it to under a thousand
+            if self.to_device_limit > 1000:
+                self.to_device_limit = 1000
+
    def __repr__(self) -> str:
        return "<FilterCollection %s>" % (json.dumps(self._filter_json),)

@@ -28,19 +28,22 @@ from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
 from synapse.events import EventBase
 from synapse.handlers.admin import ExfiltrationWriter
-from synapse.replication.slave.storage._base import BaseSlavedStore
-from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
-from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
-from synapse.replication.slave.storage.deviceinbox import SlavedDeviceInboxStore
 from synapse.replication.slave.storage.devices import SlavedDeviceStore
 from synapse.replication.slave.storage.events import SlavedEventStore
 from synapse.replication.slave.storage.filtering import SlavedFilteringStore
 from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
-from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
-from synapse.replication.slave.storage.registration import SlavedRegistrationStore
 from synapse.server import HomeServer
 from synapse.storage.database import DatabasePool, LoggingDatabaseConnection
+from synapse.storage.databases.main.account_data import AccountDataWorkerStore
+from synapse.storage.databases.main.appservice import (
+    ApplicationServiceTransactionWorkerStore,
+    ApplicationServiceWorkerStore,
+)
+from synapse.storage.databases.main.deviceinbox import DeviceInboxWorkerStore
+from synapse.storage.databases.main.receipts import ReceiptsWorkerStore
+from synapse.storage.databases.main.registration import RegistrationWorkerStore
 from synapse.storage.databases.main.room import RoomWorkerStore
+from synapse.storage.databases.main.tags import TagsWorkerStore
 from synapse.types import StateMap
 from synapse.util import SYNAPSE_VERSION
 from synapse.util.logcontext import LoggingContext
@@ -49,16 +52,17 @@ logger = logging.getLogger("synapse.app.admin_cmd")


 class AdminCmdSlavedStore(
-    SlavedReceiptsStore,
-    SlavedAccountDataStore,
-    SlavedApplicationServiceStore,
-    SlavedRegistrationStore,
    SlavedFilteringStore,
-    SlavedDeviceInboxStore,
    SlavedDeviceStore,
    SlavedPushRuleStore,
    SlavedEventStore,
-    BaseSlavedStore,
+    TagsWorkerStore,
+    DeviceInboxWorkerStore,
+    AccountDataWorkerStore,
+    ApplicationServiceTransactionWorkerStore,
+    ApplicationServiceWorkerStore,
+    RegistrationWorkerStore,
+    ReceiptsWorkerStore,
    RoomWorkerStore,
 ):
    def __init__(
@@ -48,20 +48,12 @@ from synapse.http.site import SynapseRequest, SynapseSite
 from synapse.logging.context import LoggingContext
 from synapse.metrics import METRICS_PREFIX, MetricsResource, RegistryProxy
 from synapse.replication.http import REPLICATION_PREFIX, ReplicationRestResource
-from synapse.replication.slave.storage._base import BaseSlavedStore
-from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
-from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
-from synapse.replication.slave.storage.deviceinbox import SlavedDeviceInboxStore
 from synapse.replication.slave.storage.devices import SlavedDeviceStore
-from synapse.replication.slave.storage.directory import DirectoryStore
 from synapse.replication.slave.storage.events import SlavedEventStore
 from synapse.replication.slave.storage.filtering import SlavedFilteringStore
 from synapse.replication.slave.storage.keys import SlavedKeyStore
-from synapse.replication.slave.storage.profile import SlavedProfileStore
 from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
 from synapse.replication.slave.storage.pushers import SlavedPusherStore
-from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
-from synapse.replication.slave.storage.registration import SlavedRegistrationStore
 from synapse.rest.admin import register_servlets_for_media_repo
 from synapse.rest.client import (
    account_data,
@@ -100,8 +92,15 @@ from synapse.rest.key.v2 import KeyApiV2Resource
 from synapse.rest.synapse.client import build_synapse_client_resource_tree
 from synapse.rest.well_known import well_known_resource
 from synapse.server import HomeServer
+from synapse.storage.databases.main.account_data import AccountDataWorkerStore
+from synapse.storage.databases.main.appservice import (
+    ApplicationServiceTransactionWorkerStore,
+    ApplicationServiceWorkerStore,
+)
 from synapse.storage.databases.main.censor_events import CensorEventsStore
 from synapse.storage.databases.main.client_ips import ClientIpWorkerStore
+from synapse.storage.databases.main.deviceinbox import DeviceInboxWorkerStore
+from synapse.storage.databases.main.directory import DirectoryWorkerStore
 from synapse.storage.databases.main.e2e_room_keys import EndToEndRoomKeyStore
 from synapse.storage.databases.main.lock import LockStore
 from synapse.storage.databases.main.media_repository import MediaRepositoryStore
@@ -110,11 +109,15 @@ from synapse.storage.databases.main.monthly_active_users import (
    MonthlyActiveUsersWorkerStore,
 )
 from synapse.storage.databases.main.presence import PresenceStore
+from synapse.storage.databases.main.profile import ProfileWorkerStore
+from synapse.storage.databases.main.receipts import ReceiptsWorkerStore
+from synapse.storage.databases.main.registration import RegistrationWorkerStore
 from synapse.storage.databases.main.room import RoomWorkerStore
 from synapse.storage.databases.main.room_batch import RoomBatchStore
 from synapse.storage.databases.main.search import SearchStore
 from synapse.storage.databases.main.session import SessionStore
 from synapse.storage.databases.main.stats import StatsStore
+from synapse.storage.databases.main.tags import TagsWorkerStore
 from synapse.storage.databases.main.transactions import TransactionWorkerStore
 from synapse.storage.databases.main.ui_auth import UIAuthWorkerStore
 from synapse.storage.databases.main.user_directory import UserDirectoryStore
@@ -227,11 +230,11 @@ class GenericWorkerSlavedStore(
    UIAuthWorkerStore,
    EndToEndRoomKeyStore,
    PresenceStore,
-    SlavedDeviceInboxStore,
+    DeviceInboxWorkerStore,
    SlavedDeviceStore,
-    SlavedReceiptsStore,
    SlavedPushRuleStore,
-    SlavedAccountDataStore,
+    TagsWorkerStore,
+    AccountDataWorkerStore,
    SlavedPusherStore,
    CensorEventsStore,
    ClientIpWorkerStore,
@@ -239,19 +242,20 @@ class GenericWorkerSlavedStore(
    SlavedKeyStore,
    RoomWorkerStore,
    RoomBatchStore,
-    DirectoryStore,
-    SlavedApplicationServiceStore,
-    SlavedRegistrationStore,
-    SlavedProfileStore,
+    DirectoryWorkerStore,
+    ApplicationServiceTransactionWorkerStore,
+    ApplicationServiceWorkerStore,
+    ProfileWorkerStore,
    SlavedFilteringStore,
    MonthlyActiveUsersWorkerStore,
    MediaRepositoryStore,
    ServerMetricsStore,
+    ReceiptsWorkerStore,
+    RegistrationWorkerStore,
    SearchStore,
    TransactionWorkerStore,
    LockStore,
    SessionStore,
-    BaseSlavedStore,
 ):
    # Properties that multiple storage classes define. Tell mypy what the
    # expected type is.
@@ -44,6 +44,7 @@ from synapse.app._base import (
    register_start,
 )
 from synapse.config._base import ConfigError, format_config_error
+from synapse.config.emailconfig import ThreepidBehaviour
 from synapse.config.homeserver import HomeServerConfig
 from synapse.config.server import ListenerConfig
 from synapse.federation.transport.server import TransportLayerServer
@@ -201,7 +202,7 @@ class SynapseHomeServer(HomeServer):
                }
            )

-            if self.config.email.can_verify_email:
+            if self.config.email.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
                from synapse.rest.synapse.client.password_reset import (
                    PasswordResetSubmitTokenResource,
                )
@@ -18,6 +18,7 @@
 import email.utils
 import logging
 import os
+from enum import Enum
 from typing import Any

 import attr
@@ -85,14 +86,19 @@ class EmailConfig(Config):
        if email_config is None:
            email_config = {}

+        self.force_tls = email_config.get("force_tls", False)
        self.email_smtp_host = email_config.get("smtp_host", "localhost")
-        self.email_smtp_port = email_config.get("smtp_port", 25)
+        self.email_smtp_port = email_config.get(
+            "smtp_port", 465 if self.force_tls else 25
+        )
        self.email_smtp_user = email_config.get("smtp_user", None)
        self.email_smtp_pass = email_config.get("smtp_pass", None)
        self.require_transport_security = email_config.get(
            "require_transport_security", False
        )
        self.enable_smtp_tls = email_config.get("enable_tls", True)
+        if self.force_tls and not self.enable_smtp_tls:
+            raise ConfigError("email.force_tls requires email.enable_tls to be true")
        if self.require_transport_security and not self.enable_smtp_tls:
            raise ConfigError(
                "email.require_transport_security requires email.enable_tls to be true"
@@ -130,22 +136,40 @@ class EmailConfig(Config):

        self.email_enable_notifs = email_config.get("enable_notifs", False)

+        self.threepid_behaviour_email = (
+            # Have Synapse handle the email sending if account_threepid_delegates.email
+            # is not defined
+            # msisdn is currently always remote while Synapse does not support any method of
+            # sending SMS messages
+            ThreepidBehaviour.REMOTE
+            if self.root.registration.account_threepid_delegate_email
+            else ThreepidBehaviour.LOCAL
+        )
+
        if config.get("trust_identity_server_for_password_resets"):
            raise ConfigError(
-                'The config option "trust_identity_server_for_password_resets" '
-                "is no longer supported. Please remove it from the config file."
+                'The config option "trust_identity_server_for_password_resets" has been removed.'
+                "Please consult the configuration manual at docs/usage/configuration/config_documentation.md for "
+                "details and update your config file."
            )

-        # If we have email config settings, assume that we can verify ownership of
-        # email addresses.
-        self.can_verify_email = email_config != {}
+        self.local_threepid_handling_disabled_due_to_email_config = False
+        if (
+            self.threepid_behaviour_email == ThreepidBehaviour.LOCAL
+            and email_config == {}
+        ):
+            # We cannot warn the user this has happened here
+            # Instead do so when a user attempts to reset their password
+            self.local_threepid_handling_disabled_due_to_email_config = True
+
+            self.threepid_behaviour_email = ThreepidBehaviour.OFF

        # Get lifetime of a validation token in milliseconds
        self.email_validation_token_lifetime = self.parse_duration(
            email_config.get("validation_token_lifetime", "1h")
        )

-        if self.can_verify_email:
+        if self.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
            missing = []
            if not self.email_notif_from:
                missing.append("email.notif_from")
@@ -336,3 +360,18 @@ class EmailConfig(Config):
                    "Config option email.invite_client_location must be a http or https URL",
                    path=("email", "invite_client_location"),
                )
+
+
+class ThreepidBehaviour(Enum):
+    """
+    Enum to define the behaviour of Synapse with regards to when it contacts an identity
+    server for 3pid registration and password resets
+
+    REMOTE = use an external server to send tokens
+    LOCAL = send tokens ourselves
+    OFF = disable registration via 3pid and password resets
+    """
+
+    REMOTE = "remote"
+    LOCAL = "local"
+    OFF = "off"
@@ -88,5 +88,11 @@ class ExperimentalConfig(Config):
        # MSC3715: dir param on /relations.
        self.msc3715_enabled: bool = experimental.get("msc3715_enabled", False)

-        # MSC3827: Filtering of /publicRooms by room type
-        self.msc3827_enabled: bool = experimental.get("msc3827_enabled", False)
+        # MSC3848: Introduce errcodes for specific event sending failures
+        self.msc3848_enabled: bool = experimental.get("msc3848_enabled", False)
+
+        # Experimental feature to optimize client sync performance
+        # Will become a proper MSC if it appears to be useful
+        self.to_device_limit_enabled: bool = experimental.get(
+            "to_device_limit_enabled", False
+        )
@@ -13,6 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import argparse
+import logging
 from typing import Any, Optional

 from synapse.api.constants import RoomCreationPreset
@@ -20,11 +21,15 @@ from synapse.config._base import Config, ConfigError
 from synapse.types import JsonDict, RoomAlias, UserID
 from synapse.util.stringutils import random_string_with_symbols, strtobool

-NO_EMAIL_DELEGATE_ERROR = """\
-Delegation of email verification to an identity server is no longer supported. To
+logger = logging.getLogger(__name__)
+
+LEGACY_EMAIL_DELEGATE_WARNING = """\
+Delegation of email verification to an identity server is now deprecated. To
 continue to allow users to add email addresses to their accounts, and use them for
 password resets, configure Synapse with an SMTP server via the `email` setting, and
 remove `account_threepid_delegates.email`.
+
+This will be an error in a future version.
 """


@@ -59,8 +64,9 @@ class RegistrationConfig(Config):

        account_threepid_delegates = config.get("account_threepid_delegates") or {}
        if "email" in account_threepid_delegates:
-            raise ConfigError(NO_EMAIL_DELEGATE_ERROR)
-        # self.account_threepid_delegate_email = account_threepid_delegates.get("email")
+            logger.warning(LEGACY_EMAIL_DELEGATE_WARNING)
+
+        self.account_threepid_delegate_email = account_threepid_delegates.get("email")
        self.account_threepid_delegate_msisdn = account_threepid_delegates.get("msisdn")
        self.default_identity_server = config.get("default_identity_server")
        self.allow_guest_access = config.get("allow_guest_access", False)
@@ -30,7 +30,13 @@ from synapse.api.constants import (
    JoinRules,
    Membership,
 )
-from synapse.api.errors import AuthError, EventSizeError, SynapseError
+from synapse.api.errors import (
+    AuthError,
+    Codes,
+    EventSizeError,
+    SynapseError,
+    UnstableSpecAuthError,
+)
 from synapse.api.room_versions import (
    KNOWN_ROOM_VERSIONS,
    EventFormatVersions,
@@ -291,7 +297,11 @@ def check_state_dependent_auth_rules(
        invite_level = get_named_level(auth_dict, "invite", 0)

        if user_level < invite_level:
-            raise AuthError(403, "You don't have permission to invite users")
+            raise UnstableSpecAuthError(
+                403,
+                "You don't have permission to invite users",
+                errcode=Codes.INSUFFICIENT_POWER,
+            )
        else:
            logger.debug("Allowing! %s", event)
            return
@@ -474,7 +484,11 @@ def _is_membership_change_allowed(
            return

        if not caller_in_room:  # caller isn't joined
-            raise AuthError(403, "%s not in room %s." % (event.user_id, event.room_id))
+            raise UnstableSpecAuthError(
+                403,
+                "%s not in room %s." % (event.user_id, event.room_id),
+                errcode=Codes.NOT_JOINED,
+            )

    if Membership.INVITE == membership:
        # TODO (erikj): We should probably handle this more intelligently
@@ -484,10 +498,18 @@ def _is_membership_change_allowed(
        if target_banned:
            raise AuthError(403, "%s is banned from the room" % (target_user_id,))
        elif target_in_room:  # the target is already in the room.
-            raise AuthError(403, "%s is already in the room." % target_user_id)
+            raise UnstableSpecAuthError(
+                403,
+                "%s is already in the room." % target_user_id,
+                errcode=Codes.ALREADY_JOINED,
+            )
        else:
            if user_level < invite_level:
-                raise AuthError(403, "You don't have permission to invite users")
+                raise UnstableSpecAuthError(
+                    403,
+                    "You don't have permission to invite users",
+                    errcode=Codes.INSUFFICIENT_POWER,
+                )
    elif Membership.JOIN == membership:
        # Joins are valid iff caller == target and:
        # * They are not banned.
@@ -549,15 +571,27 @@ def _is_membership_change_allowed(
    elif Membership.LEAVE == membership:
        # TODO (erikj): Implement kicks.
        if target_banned and user_level < ban_level:
-            raise AuthError(403, "You cannot unban user %s." % (target_user_id,))
+            raise UnstableSpecAuthError(
+                403,
+                "You cannot unban user %s." % (target_user_id,),
+                errcode=Codes.INSUFFICIENT_POWER,
+            )
        elif target_user_id != event.user_id:
            kick_level = get_named_level(auth_events, "kick", 50)

            if user_level < kick_level or user_level <= target_level:
-                raise AuthError(403, "You cannot kick user %s." % target_user_id)
+                raise UnstableSpecAuthError(
+                    403,
+                    "You cannot kick user %s." % target_user_id,
+                    errcode=Codes.INSUFFICIENT_POWER,
+                )
    elif Membership.BAN == membership:
        if user_level < ban_level or user_level <= target_level:
-            raise AuthError(403, "You don't have permission to ban")
+            raise UnstableSpecAuthError(
+                403,
+                "You don't have permission to ban",
+                errcode=Codes.INSUFFICIENT_POWER,
+            )
    elif room_version.msc2403_knocking and Membership.KNOCK == membership:
        if join_rule != JoinRules.KNOCK and (
            not room_version.msc3787_knock_restricted_join_rule
@@ -567,7 +601,11 @@ def _is_membership_change_allowed(
        elif target_user_id != event.user_id:
            raise AuthError(403, "You cannot knock for other users")
        elif target_in_room:
-            raise AuthError(403, "You cannot knock on a room you are already in")
+            raise UnstableSpecAuthError(
+                403,
+                "You cannot knock on a room you are already in",
+                errcode=Codes.ALREADY_JOINED,
+            )
        elif caller_invited:
            raise AuthError(403, "You are already invited to this room")
        elif target_banned:
@@ -638,10 +676,11 @@ def _can_send_event(event: "EventBase", auth_events: StateMap["EventBase"]) -> b
    user_level = get_user_power_level(event.user_id, auth_events)

    if user_level < send_level:
-        raise AuthError(
+        raise UnstableSpecAuthError(
            403,
            "You don't have permission to post that to the room. "
            + "user_level (%d) < send_level (%d)" % (user_level, send_level),
+            errcode=Codes.INSUFFICIENT_POWER,
        )

    # Check state_key
@@ -716,9 +755,10 @@ def check_historical(
    historical_level = get_named_level(auth_events, "historical", 100)

    if user_level < historical_level:
-        raise AuthError(
+        raise UnstableSpecAuthError(
            403,
            'You don\'t have permission to send send historical related events ("insertion", "batch", and "marker")',
+            errcode=Codes.INSUFFICIENT_POWER,
        )


@@ -53,7 +53,7 @@ from synapse.api.room_versions import (
    RoomVersion,
    RoomVersions,
 )
-from synapse.events import EventBase, builder
+from synapse.events import EventBase, builder, make_event_from_dict
 from synapse.federation.federation_base import (
    FederationBase,
    InvalidEventSignatureError,
@@ -299,7 +299,8 @@ class FederationClient(FederationBase):
                moving to the next destination. None indicates no timeout.

        Returns:
-            The requested PDU, or None if we were unable to find it.
+            A copy of the requested PDU that is safe to modify, or None if we
+            were unable to find it.

        Raises:
            SynapseError, NotRetryingDestination, FederationDeniedError
@@ -309,7 +310,7 @@ class FederationClient(FederationBase):
        )

        logger.debug(
-            "retrieved event id %s from %s: %r",
+            "get_pdu_from_destination_raw: retrieved event id %s from %s: %r",
            event_id,
            destination,
            transaction_data,
@@ -358,54 +359,92 @@ class FederationClient(FederationBase):
            The requested PDU, or None if we were unable to find it.
        """

+        logger.debug(
+            "get_pdu: event_id=%s from destinations=%s", event_id, destinations
+        )
+
        # TODO: Rate limit the number of times we try and get the same event.

-        ev = self._get_pdu_cache.get(event_id)
-        if ev:
-            return ev
+        # We might need the same event multiple times in quick succession (before
+        # it gets persisted to the database), so we cache the results of the lookup.
+        # Note that this is separate to the regular get_event cache which caches
+        # events once they have been persisted.
+        event = self._get_pdu_cache.get(event_id)

-        pdu_attempts = self.pdu_destination_tried.setdefault(event_id, {})
+        # If we don't see the event in the cache, go try to fetch it from the
+        # provided remote federated destinations
+        if not event:
+            pdu_attempts = self.pdu_destination_tried.setdefault(event_id, {})

-        signed_pdu = None
-        for destination in destinations:
-            now = self._clock.time_msec()
-            last_attempt = pdu_attempts.get(destination, 0)
-            if last_attempt + PDU_RETRY_TIME_MS > now:
-                continue
+            for destination in destinations:
+                now = self._clock.time_msec()
+                last_attempt = pdu_attempts.get(destination, 0)
+                if last_attempt + PDU_RETRY_TIME_MS > now:
+                    logger.debug(
+                        "get_pdu: skipping destination=%s because we tried it recently last_attempt=%s and we only check every %s (now=%s)",
+                        destination,
+                        last_attempt,
+                        PDU_RETRY_TIME_MS,
+                        now,
+                    )
+                    continue

-            try:
-                signed_pdu = await self.get_pdu_from_destination_raw(
-                    destination=destination,
-                    event_id=event_id,
-                    room_version=room_version,
-                    timeout=timeout,
-                )
+                try:
+                    event = await self.get_pdu_from_destination_raw(
+                        destination=destination,
+                        event_id=event_id,
+                        room_version=room_version,
+                        timeout=timeout,
+                    )

-                pdu_attempts[destination] = now
+                    pdu_attempts[destination] = now

-            except SynapseError as e:
-                logger.info(
-                    "Failed to get PDU %s from %s because %s", event_id, destination, e
-                )
-                continue
-            except NotRetryingDestination as e:
-                logger.info(str(e))
-                continue
-            except FederationDeniedError as e:
-                logger.info(str(e))
-                continue
-            except Exception as e:
-                pdu_attempts[destination] = now
+                    if event:
+                        # Prime the cache
+                        self._get_pdu_cache[event.event_id] = event

-                logger.info(
-                    "Failed to get PDU %s from %s because %s", event_id, destination, e
-                )
-                continue
+                        # Now that we have an event, we can break out of this
+                        # loop and stop asking other destinations.
+                        break

-        if signed_pdu:
-            self._get_pdu_cache[event_id] = signed_pdu
+                except SynapseError as e:
+                    logger.info(
+                        "Failed to get PDU %s from %s because %s",
+                        event_id,
+                        destination,
+                        e,
+                    )
+                    continue
+                except NotRetryingDestination as e:
+                    logger.info(str(e))
+                    continue
+                except FederationDeniedError as e:
+                    logger.info(str(e))
+                    continue
+                except Exception as e:
+                    pdu_attempts[destination] = now

-        return signed_pdu
+                    logger.info(
+                        "Failed to get PDU %s from %s because %s",
+                        event_id,
+                        destination,
+                        e,
+                    )
+                    continue
+
+        if not event:
+            return None
+
+        # `event` now refers to an object stored in `get_pdu_cache`. Our
+        # callers may need to modify the returned object (eg to set
+        # `event.internal_metadata.outlier = true`), so we return a copy
+        # rather than the original object.
+        event_copy = make_event_from_dict(
+            event.get_pdu_json(),
+            event.room_version,
+        )
+
+        return event_copy

    async def get_room_state_ids(
        self, destination: str, room_id: str, event_id: str
@@ -686,6 +725,12 @@ class FederationClient(FederationBase):
        if failover_errcodes is None:
            failover_errcodes = ()

+        if not destinations:
+            # Give a bit of a clearer message if no servers were specified at all.
+            raise SynapseError(
+                502, f"Failed to {description} via any server: No servers specified."
+            )
+
        for destination in destinations:
            if destination == self.server_name:
                continue
@@ -735,7 +780,7 @@ class FederationClient(FederationBase):
                    "Failed to %s via %s", description, destination, exc_info=True
                )

-        raise SynapseError(502, "Failed to %s via any server" % (description,))
+        raise SynapseError(502, f"Failed to {description} via any server")

    async def make_membership_event(
        self,
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Mathieu Velten	7af1d98df0	Update synapse/storage/databases/main/deviceinbox.py Co-authored-by: Erik Johnston <erik@matrix.org>	2022-08-03 14:47:26 +02:00
Mathieu Velten	4cc2f3e990	Use org.matrix.unstable prefix	2022-08-03 14:46:46 +02:00
Mathieu Velten	ea8cf6edcf	Type mistake	2022-08-02 15:23:52 +02:00
Mathieu Velten	5de1f166f9	Address comment	2022-08-02 15:18:57 +02:00
Mathieu Velten	f14bb8768a	Apply suggestions from code review Co-authored-by: Erik Johnston <erik@matrix.org>	2022-08-02 15:08:12 +02:00
Mathieu Velten	dfca02b6de	Add do_not_use_to_device_limit to sync filter	2022-08-02 15:08:12 +02:00
Sean Quah	8d317f6da5	Fix error when out of servers to sync partial state with (#13432 ) so that we raise the intended error instead. Signed-off-by: Sean Quah <seanq@matrix.org>	2022-08-02 12:12:44 +01:00
Olivier Wilkinson (reivilibre)	a2a867b521	Merge branch 'master' into develop	2022-08-02 11:56:02 +01:00
Olivier Wilkinson (reivilibre)	c2f4871226	Mention specific version in rc2 notes	2022-08-02 11:19:32 +01:00
Olivier Wilkinson (reivilibre)	cb209638ea	Add upgrade notes	2022-08-02 11:10:26 +01:00
Olivier Wilkinson (reivilibre)	4e80ca2243	1.64.0	2022-08-02 11:04:08 +01:00
reivilibre	e17e5c97e0	Faster Room Joins: don't leave a stuck room partial state flag if the join fails. (#13403 )	2022-08-01 16:45:39 +00:00
Patrick Cloke	f8e7a9418a	Fix missing import in `federation_event` handler. (#13431 ) #13404 removed an import of `Optional` which was still needed due to #13413 added more usages.	2022-08-01 14:14:29 +00:00
Sean Quah	224d792dd7	Refactor `_resolve_state_at_missing_prevs` to return an `EventContext` (#13404 ) Previously, `_resolve_state_at_missing_prevs` returned the resolved state before an event and a partial state flag. These were unwieldy to carry around would only ever be used to build an event context. Build the event context directly instead. Signed-off-by: Sean Quah <seanq@matrix.org>	2022-08-01 13:53:56 +01:00
reivilibre	05aeeb3a80	Enable Complement CI tests in the 'latest deps' test run. (#13213 ) Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>	2022-08-01 10:55:31 +00:00
reivilibre	b817574be7	Re-enable running Complement tests against Synapse with workers. (#13420 )	2022-08-01 11:51:44 +01:00
Richard van der Hoff	23768ccb4d	Faster joins: fix rejected events becoming un-rejected during resync (#13413 ) Make sure that we re-check the auth rules during state resync, otherwise rejected events get un-rejected.	2022-08-01 11:20:05 +01:00
Richard van der Hoff	d548d8f18d	Merge tag 'v1.64.0rc2' into develop Synapse 1.64.0rc2 (2022-07-29) ============================== This RC reintroduces support for `account_threepid_delegates.email`, which was removed in 1.64.0rc1. It remains deprecated and will be removed altogether in a future release. ([\#13406](https://github.com/matrix-org/synapse/issues/13406))	2022-07-29 15:15:21 +01:00
Richard van der Hoff	979d94de29	update changelog	2022-07-29 12:27:23 +01:00
Richard van der Hoff	6b4fd8b430	1.64.0rc2	2022-07-29 12:23:13 +01:00
3nprob	98fb610cc0	Revert "Drop support for delegating email validation (#13192 )" (#13406 ) Reverts commit `fa71bb18b5`, and tweaks documentation. Signed-off-by: 3nprob <git@3n.anonaddy.com>	2022-07-29 10:29:23 +00:00
Brendan Abolivier	24ef1460f6	Explicitly mention which resources support compression in the config guide (#13221 )	2022-07-29 09:09:57 +00:00
Šimon Brandner	583f22780f	Use stable prefixes for MSC3827: filtering of `/publicRooms` by room type (#13370 ) Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>	2022-07-27 19:46:57 +01:00
Patrick Cloke	922b771337	Add missing type hints for tests.unittest. (#13397 )	2022-07-27 17:18:41 +00:00
Will Hunt	502f075e96	Implement MSC3848: Introduce errcodes for specific event sending failures (#13343 ) Implements MSC3848	2022-07-27 13:44:40 +01:00
reivilibre	39be5bc550	Make minor clarifications to the error messages given when we fail to join a room via any server. (#13160 )	2022-07-27 10:37:50 +00:00
Eric Eastwood	4f3082d6bf	Fix `get_pdu` asking every remote destination even after it finds an event (#13346 )	2022-07-27 10:40:04 +01:00
Nick Mills-Barrett	bf3115584c	Copy room serials before handling in `get_new_events_as` (#13392 )	2022-07-26 17:45:27 +00:00
reivilibre	543dc9c93e	Extend the release script to automatically push a new SyTest branch, rather than having that be a manual process. (#12978 ) Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2022-07-26 18:08:14 +01:00
Olivier Wilkinson (reivilibre)	6236afc621	Merge branch 'release-v1.64' into develop	2022-07-26 16:26:30 +01:00
Patrick Cloke	57d334a13d	Remove the unspecced `room_id` field in the `/hierarchy` response. (#13365 ) The `room_id` field represented the parent space for each room and was made redundant by changes in the API shape where the `children_state` is now nested underneath each `room`. The room ID of each child is in the `state_key` field and is still available.	2022-07-26 08:02:34 -04:00
Olivier Wilkinson (reivilibre)	33788a07ee	Explain less-known term 'Implicit TLS'	2022-07-26 12:56:24 +01:00
Richard van der Hoff	ca3db044a3	Fix infinite loop in partial-state resync (#13353 ) Make sure that we only pull out events from the db once they have no prev-events with partial state.	2022-07-26 11:47:31 +00:00
Olivier Wilkinson (reivilibre)	5d7e2b0195	Tweak changelog in response to review	2022-07-26 12:45:19 +01:00
Sean Quah	335ebb21cc	Faster room joins: avoid blocking when pulling events with missing prevs (#13355 ) Avoid blocking on full state in `_resolve_state_at_missing_prevs` and return a new flag indicating whether the resolved state is partial. Thread that flag around so that it makes it into the event context. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2022-07-26 12:39:23 +01:00
Olivier Wilkinson (reivilibre)	f765a40f69	Tweak changelog	2022-07-26 12:26:36 +01:00
Patrick Cloke	8b603299bf	Remove unused argument for get_relations_for_event. (#13383 )	2022-07-26 07:19:20 -04:00
Olivier Wilkinson (reivilibre)	641412decd	1.64.0rc1	2022-07-26 12:12:22 +01:00
Doug	549c55606a	Disable autocorrect and autocaptialisation when entering username for SSO registration. (#13350 ) When registering a new account via SSO on iOS, the text field becomes pretty annoying as it autocapitalises and autocorrects your input. This PR fixes that (although I have only tested the raw HTML file on the simulator, I'm not sure how to get the complete setup available for testing in the flow).	2022-07-26 08:08:20 +00:00
Matt Holt	935e73efed	Update Caddy reverse proxy documentation (#13344 ) Improve/simplify Caddy examples. Remove Caddy v1 (has long been EOL'ed) Signed-off-by: Matthew Holt <mholt@users.noreply.github.com>	2022-07-25 16:07:26 +00:00
Jan Schär	e8519e0ed2	Support Implicit TLS for sending emails (#13317 ) Previously, TLS could only be used with STARTTLS. Add a new option `force_tls`, where TLS is used from the start. Implicit TLS is recommended over STARTLS, see https://datatracker.ietf.org/doc/html/rfc8314 Fixes #8046. Signed-off-by: Jan Schär <jan@jschaer.ch>	2022-07-25 16:27:19 +01:00
Patrick Cloke	908aeac44a	Additional fixes for opentracing type hints. (#13362 )	2022-07-25 08:34:06 -04:00
Erik Johnston	43adf2521c	Refactor presence so we can prune user in room caches (#13313 ) See #10826 and #10786 for context as to why we had to disable pruning on those caches. Now that `get_users_who_share_room_with_user` is called frequently only for presence, we just need to make calls to it less frequent and then we can remove the various levels of caching that is going on.	2022-07-25 09:21:06 +00:00
Eric Eastwood	357561c1a2	Backfill remote event fetched by MSC3030 so we can paginate from it later (#13205 ) Depends on https://github.com/matrix-org/synapse/pull/13320 Complement tests: https://github.com/matrix-org/complement/pull/406 We could use the same method to backfill for `/context` as well in the future, see https://github.com/matrix-org/synapse/issues/3848	2022-07-22 16:00:11 -05:00
Richard van der Hoff	c7c84b81e3	Update config_documentation.md (#13364 ) "changed in" goes before the example	2022-07-22 13:50:20 +01:00
Sean Quah	0fa41a7b17	Update locked frozendict version to 2.3.3 (#13352 ) frozendict 2.3.3 includes fixes for memory leaks that get triggered during `/sync`.	2022-07-22 10:26:09 +01:00
Sean Quah	158782c3ce	Skip soft fail checks for rooms with partial state (#13354 ) When a room has the partial state flag, we may not have an accurate `m.room.member` event for event senders in the room's current state, and so cannot perform soft fail checks correctly. Skip the soft fail check entirely in this case. As an alternative, we could block until we have full state, but that would prevent us from receiving incoming events over federation, which is undesirable. Signed-off-by: Sean Quah <seanq@matrix.org>	2022-07-22 10:13:01 +01:00
Nick Mills-Barrett	86e366a46e	Remove old empty/redundant slaved stores. (#13349 )	2022-07-21 17:56:45 +00:00
Erik Johnston	0b87eb8e0c	Make DictionaryCache have better expiry properties (#13292 )	2022-07-21 17:13:44 +01:00
Erik Johnston	13341dde5a	Don't hold onto full state in state cache (#13324 )	2022-07-21 16:02:02 +01:00
Brendan Abolivier	10e4093839	Call out buildkit is required when building test docker images (#13338 ) Co-authored-by: David Robertson <davidr@element.io>	2022-07-21 14:29:58 +02:00
David Robertson	34949ead1f	Track DB txn times w/ two counters, not histogram (#13342 )	2022-07-21 13:23:05 +01:00
Patrick Cloke	50122754c8	Add missing types to opentracing. (#13345 ) After this change `synapse.logging` is fully typed.	2022-07-21 12:01:52 +00:00
Nick Mills-Barrett	190f49d8ab	Use cache store remove base slaved (#13329 ) This comes from two identical definitions in each of the base stores, and means the base slaved store is now empty and can be removed.	2022-07-21 11:51:30 +01:00
David Robertson	4f57ef0b18	Merge branch 'master' into develop	2022-07-21 11:27:08 +01:00
David Teller	b909d5327b	Document `rc_invites.per_issuer`, added in v1.63. Resolves #13330. Missed in #13125. Signed-off-by: David Teller <davidt@element.io>	2022-07-21 11:26:34 +01:00
Eric Eastwood	0f971ca68e	Update `get_pdu` to return the original, pristine `EventBase` (#13320 ) Update `get_pdu` to return the untouched, pristine `EventBase` as it was originally seen over federation (no metadata added). Previously, we returned the same `event` reference that we stored in the cache which downstream code modified in place and added metadata like setting it as an `outlier` and essentially poisoned our cache. Now we always return a copy of the `event` so the original can stay pristine in our cache and re-used for the next cache call. Split out from https://github.com/matrix-org/synapse/pull/13205 As discussed at: - https://github.com/matrix-org/synapse/pull/13205#discussion_r918365746 - https://github.com/matrix-org/synapse/pull/13205#discussion_r918366125 Related to https://github.com/matrix-org/synapse/issues/12584. This PR doesn't fix that issue because it hits [`get_event` which exists from the local database before it tries to `get_pdu`](https://github.com/matrix-org/synapse/blob/7864f33e286dec22368dc0b11c06eebb1462a51e/synapse/federation/federation_client.py#L581-L594).	2022-07-20 15:58:51 -05:00
Shay	a1b62af2af	Validate federation destinations and log an error if server name is invalid. (#13318 )	2022-07-20 11:17:26 -07:00
Erik Johnston	d3995049a8	Merge remote-tracking branch 'origin/master' into develop	2022-07-20 14:59:43 +01:00
Erik Johnston	93740cae57	1.63.1	2022-07-20 13:37:00 +01:00
Erik Johnston	b4ae3b0d44	Don't include appservice users when calculating push rules (#13332 ) This can cause a lot of extra load on servers with lots of appservice users. Introduced in #13078	2022-07-20 12:06:13 +01:00
				`@@ -1 +0,0 @@`
				`Use lower isolation level when purging rooms to avoid serialization errors. Contributed by Nick @ Beeper.`
				`@@ -1 +0,0 @@`
				`Remove code which incorrectly attempted to reconcile state with remote servers when processing incoming events.`
				`@@ -0,0 +1 @@`
				`Extend the release script to automatically push a new SyTest branch, rather than having that be a manual process.`
				`@@ -1 +0,0 @@`
				`Provide more info why we don't have any thumbnails to serve.`
				`@@ -1 +0,0 @@`
				Make the AS login method call `Auth.get_user_by_req` for checking the AS token.
				`@@ -0,0 +1 @@`
				`Make minor clarifications to the error messages given when we fail to join a room via any server.`
				`@@ -1 +0,0 @@`
				`Always use a version of canonicaljson that supports the C implementation of frozendict.`
				`@@ -1 +0,0 @@`
				`Add prometheus counters for ephemeral events and to device messages pushed to app services. Contributed by Brad @ Beeper.`
				`@@ -1 +0,0 @@`
				`Drop support for delegating email verification to an external server.`