Changelog

Avoid state lookups for events we'll ignore
Track a set of strings, not EventBases
2023-03-08 19:25:41 +00:00 · 2023-03-08 19:21:44 +00:00 · 2023-03-08 19:21:43 +00:00 · 2023-03-08 19:21:43 +00:00 · 2023-03-08 19:21:29 +00:00 · 2023-03-08 19:19:46 +00:00
100 changed files with 1093 additions and 827 deletions
@@ -109,11 +109,26 @@ sytest_tests = [
        "postgres": "multi-postgres",
        "workers": "workers",
    },
+    {
+        "sytest-tag": "focal",
+        "postgres": "multi-postgres",
+        "workers": "workers",
+        "reactor": "asyncio",
+    },
 ]

 if not IS_PR:
    sytest_tests.extend(
        [
+            {
+                "sytest-tag": "focal",
+                "reactor": "asyncio",
+            },
+            {
+                "sytest-tag": "focal",
+                "postgres": "postgres",
+                "reactor": "asyncio",
+            },
            {
                "sytest-tag": "testing",
                "postgres": "postgres",
@@ -156,7 +156,8 @@ jobs:
        # We pin to a specific commit for paranoia's sake.
        uses: dtolnay/rust-toolchain@e12eda571dc9a5ee5d58eecf4738ec291c66f295
        with:
-          toolchain: 1.58.1
+          # We use nightly so that it correctly groups together imports
+          toolchain: nightly-2022-12-01
          components: rustfmt
      - uses: Swatinem/rust-cache@v2

@@ -368,6 +369,7 @@ jobs:
        SYTEST_BRANCH: ${{ github.head_ref }}
        POSTGRES: ${{ matrix.job.postgres && 1}}
        MULTI_POSTGRES: ${{ (matrix.job.postgres == 'multi-postgres') && 1}}
+        ASYNCIO_REACTOR: ${{ (matrix.job.reactor == 'asyncio') && 1 }}
        WORKERS: ${{ matrix.job.workers && 1 }}
        BLACKLIST: ${{ matrix.job.workers && 'synapse-blacklist-with-workers' }}
        TOP: ${{ github.workspace }}
@@ -1,3 +1,100 @@
+Synapse 1.79.0rc1 (2023-03-07)
+==============================
+
+Features
+--------
+
+- Add two new Third Party Rules module API callbacks: [`on_add_user_third_party_identifier`](https://matrix-org.github.io/synapse/v1.79/modules/third_party_rules_callbacks.html#on_add_user_third_party_identifier) and [`on_remove_user_third_party_identifier`](https://matrix-org.github.io/synapse/v1.79/modules/third_party_rules_callbacks.html#on_remove_user_third_party_identifier). ([\#15044](https://github.com/matrix-org/synapse/issues/15044))
+- Experimental support for [MSC3967](https://github.com/matrix-org/matrix-spec-proposals/pull/3967) to not require UIA for setting up cross-signing on first use. ([\#15077](https://github.com/matrix-org/synapse/issues/15077))
+- Add media information to the command line [user data export tool](https://matrix-org.github.io/synapse/v1.79/usage/administration/admin_faq.html#how-can-i-export-user-data). ([\#15107](https://github.com/matrix-org/synapse/issues/15107))
+- Add an [admin API](https://matrix-org.github.io/synapse/latest/usage/administration/admin_api/index.html) to delete a [specific event report](https://spec.matrix.org/v1.6/client-server-api/#reporting-content). ([\#15116](https://github.com/matrix-org/synapse/issues/15116))
+- Add support for knocking to workers. ([\#15133](https://github.com/matrix-org/synapse/issues/15133))
+- Allow use of the `/filter` Client-Server APIs on workers. ([\#15134](https://github.com/matrix-org/synapse/issues/15134))
+- Update support for [MSC2677](https://github.com/matrix-org/matrix-spec-proposals/pull/2677): remove support for server-side aggregation of reactions. ([\#15172](https://github.com/matrix-org/synapse/issues/15172))
+- Stabilise support for [MSC3758](https://github.com/matrix-org/matrix-spec-proposals/pull/3758): `event_property_is` push condition. ([\#15185](https://github.com/matrix-org/synapse/issues/15185))
+
+
+Bugfixes
+--------
+
+- Fix a bug introduced in Synapse 1.75 that caused experimental support for deleting account data to raise an internal server error while using an account data writer worker. ([\#14869](https://github.com/matrix-org/synapse/issues/14869))
+- Fix a long-standing bug where Synapse handled an unspecced field on push rules. ([\#15088](https://github.com/matrix-org/synapse/issues/15088))
+- Fix a long-standing bug where a URL preview would break if the discovered oEmbed failed to download. ([\#15092](https://github.com/matrix-org/synapse/issues/15092))
+- Fix a long-standing bug where an initial sync would not respond to changes to the list of ignored users if there was an initial sync cached. ([\#15163](https://github.com/matrix-org/synapse/issues/15163))
+- Add the `transaction_id` in the events included in many endpoints' responses. ([\#15174](https://github.com/matrix-org/synapse/issues/15174))
+- Fix a bug introduced in Synapse 1.78.0 where requests to claim dehydrated devices would fail with a `405` error. ([\#15180](https://github.com/matrix-org/synapse/issues/15180))
+- Stop applying edits when bundling aggregations, per [MSC3925](https://github.com/matrix-org/matrix-spec-proposals/pull/3925). ([\#15193](https://github.com/matrix-org/synapse/issues/15193))
+- Fix a long-standing bug where the user directory search was not case-insensitive for accented characters. ([\#15143](https://github.com/matrix-org/synapse/issues/15143))
+
+
+Updates to the Docker image
+---------------------------
+
+- Improve startup logging in the with-workers Docker image. ([\#15186](https://github.com/matrix-org/synapse/issues/15186))
+
+
+Improved Documentation
+----------------------
+
+- Document how to use caches in a module. ([\#14026](https://github.com/matrix-org/synapse/issues/14026))
+- Clarify which worker processes the ThirdPartyRules' [`on_new_event`](https://matrix-org.github.io/synapse/v1.78/modules/third_party_rules_callbacks.html#on_new_event) module API callback runs on. ([\#15071](https://github.com/matrix-org/synapse/issues/15071))
+- Document using [Shibboleth](https://www.shibboleth.net/) as an OpenID Provider. ([\#15112](https://github.com/matrix-org/synapse/issues/15112))
+- Correct reference to `federation_verify_certificates` in configuration documentation. ([\#15139](https://github.com/matrix-org/synapse/issues/15139))
+- Correct small documentation errors in some `MatrixFederationHttpClient` methods. ([\#15148](https://github.com/matrix-org/synapse/issues/15148))
+- Correct the description of the behavior of `registration_shared_secret_path` on startup. ([\#15168](https://github.com/matrix-org/synapse/issues/15168))
+
+
+Deprecations and Removals
+-------------------------
+
+- Deprecate the `on_threepid_bind` module callback, to be replaced by [`on_add_user_third_party_identifier`](https://matrix-org.github.io/synapse/v1.79/modules/third_party_rules_callbacks.html#on_add_user_third_party_identifier). See [upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.79/docs/upgrade.md#upgrading-to-v1790). ([\#15044](https://github.com/matrix-org/synapse/issues/15044))
+- Remove the unspecced `room_alias` field from the [`/createRoom`](https://spec.matrix.org/v1.6/client-server-api/#post_matrixclientv3createroom) response. ([\#15093](https://github.com/matrix-org/synapse/issues/15093))
+- Remove the unspecced `PUT` on the `/knock/{roomIdOrAlias}` endpoint. ([\#15189](https://github.com/matrix-org/synapse/issues/15189))
+- Remove the undocumented and unspecced `type` parameter to the `/thumbnail` endpoint. ([\#15137](https://github.com/matrix-org/synapse/issues/15137))
+- Remove unspecced and buggy `PUT` method on the unstable `/rooms/<room_id>/batch_send` endpoint. ([\#15199](https://github.com/matrix-org/synapse/issues/15199))
+
+
+Internal Changes
+----------------
+
+- Run the integration test suites with the asyncio reactor enabled in CI. ([\#14101](https://github.com/matrix-org/synapse/issues/14101))
+- Batch up storing state groups when creating a new room. ([\#14918](https://github.com/matrix-org/synapse/issues/14918))
+- Update [MSC3952](https://github.com/matrix-org/matrix-spec-proposals/pull/3952) support based on changes to the MSC. ([\#15051](https://github.com/matrix-org/synapse/issues/15051))
+- Refactor writing json data in `FileExfiltrationWriter`. ([\#15095](https://github.com/matrix-org/synapse/issues/15095))
+- Tighten the login ratelimit defaults. ([\#15135](https://github.com/matrix-org/synapse/issues/15135))
+- Fix a typo in an experimental config setting. ([\#15138](https://github.com/matrix-org/synapse/issues/15138))
+- Refactor the media modules. ([\#15146](https://github.com/matrix-org/synapse/issues/15146), [\#15175](https://github.com/matrix-org/synapse/issues/15175))
+- Improve type hints. ([\#15164](https://github.com/matrix-org/synapse/issues/15164))
+- Move `get_event_report` and `get_event_reports_paginate` from `RoomStore` to `RoomWorkerStore`. ([\#15165](https://github.com/matrix-org/synapse/issues/15165))
+- Remove dangling reference to being a reference implementation in docstring. ([\#15167](https://github.com/matrix-org/synapse/issues/15167))
+- Add an option to force a rebuild of the "editable" complement image. ([\#15184](https://github.com/matrix-org/synapse/issues/15184))
+- Use nightly rustfmt in CI. ([\#15188](https://github.com/matrix-org/synapse/issues/15188))
+- Add a `get_next_txn` method to `StreamIdGenerator` to match `MultiWriterIdGenerator`. ([\#15191](https://github.com/matrix-org/synapse/issues/15191))
+- Combine `AbstractStreamIdTracker` and `AbstractStreamIdGenerator`. ([\#15192](https://github.com/matrix-org/synapse/issues/15192))
+- Automatically fix errors with `ruff`. ([\#15194](https://github.com/matrix-org/synapse/issues/15194))
+- Refactor database transaction for query users' devices to reduce database pool contention. ([\#15215](https://github.com/matrix-org/synapse/issues/15215))
+- Correct `test_icu_word_boundary_punctuation` so that it passes with the ICU versions available in Alpine and macOS. ([\#15177](https://github.com/matrix-org/synapse/issues/15177))
+
+<details><summary>Locked dependency updates</summary>
+
+  - Bump actions/checkout from 2 to 3. ([\#15155](https://github.com/matrix-org/synapse/issues/15155))
+  - Bump black from 22.12.0 to 23.1.0. ([\#15103](https://github.com/matrix-org/synapse/issues/15103))
+  - Bump dawidd6/action-download-artifact from 2.25.0 to 2.26.0. ([\#15152](https://github.com/matrix-org/synapse/issues/15152))
+  - Bump docker/login-action from 1 to 2. ([\#15154](https://github.com/matrix-org/synapse/issues/15154))
+  - Bump matrix-org/backend-meta from 1 to 2. ([\#15156](https://github.com/matrix-org/synapse/issues/15156))
+  - Bump ruff from 0.0.237 to 0.0.252. ([\#15159](https://github.com/matrix-org/synapse/issues/15159))
+  - Bump serde_json from 1.0.93 to 1.0.94. ([\#15214](https://github.com/matrix-org/synapse/issues/15214))
+  - Bump types-commonmark from 0.9.2.1 to 0.9.2.2. ([\#15209](https://github.com/matrix-org/synapse/issues/15209))
+  - Bump types-opentracing from 2.4.10.1 to 2.4.10.3. ([\#15158](https://github.com/matrix-org/synapse/issues/15158))
+  - Bump types-pillow from 9.4.0.13 to 9.4.0.17. ([\#15211](https://github.com/matrix-org/synapse/issues/15211))
+  - Bump types-psycopg2 from 2.9.21.4 to 2.9.21.8. ([\#15210](https://github.com/matrix-org/synapse/issues/15210))
+  - Bump types-pyopenssl from 22.1.0.2 to 23.0.0.4. ([\#15213](https://github.com/matrix-org/synapse/issues/15213))
+  - Bump types-setuptools from 67.3.0.1 to 67.4.0.3. ([\#15160](https://github.com/matrix-org/synapse/issues/15160))
+  - Bump types-setuptools from 67.4.0.3 to 67.5.0.0. ([\#15212](https://github.com/matrix-org/synapse/issues/15212))
+  - Bump typing-extensions from 4.4.0 to 4.5.0. ([\#15157](https://github.com/matrix-org/synapse/issues/15157))
+</details>
+
+
 Synapse 1.78.0 (2023-02-28)
 ===========================

@@ -343,9 +343,9 @@ dependencies = [

 [[package]]
 name = "serde_json"
-version = "1.0.93"
+version = "1.0.94"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "cad406b69c91885b5107daf2c29572f6c8cdb3c66826821e286c533490c0bc76"
+checksum = "1c533a59c9d8a93a09c6ab31f0fd5e5f4dd1b8fc9434804029839884765d04ea"
 dependencies = [
 "itoa",
 "ryu",
@@ -1 +0,0 @@
-Document how to use caches in a module.
@@ -1 +0,0 @@
-Batch up storing state groups when creating a new room.
@@ -1 +0,0 @@
-Add two new Third Party Rules module API callbacks: [`on_add_user_third_party_identifier`](https://matrix-org.github.io/synapse/v1.79/modules/third_party_rules_callbacks.html#on_add_user_third_party_identifier) and [`on_remove_user_third_party_identifier`](https://matrix-org.github.io/synapse/v1.79/modules/third_party_rules_callbacks.html#on_remove_user_third_party_identifier).
@@ -1 +0,0 @@
-Clarify which worker processes the ThirdPartyRules' [`on_new_event`](https://matrix-org.github.io/synapse/v1.78/modules/third_party_rules_callbacks.html#on_new_event) module API callback runs on.
@@ -1 +0,0 @@
-Fix a long-standing bug where Synapse handled an unspecced field on push rules.
@@ -1 +0,0 @@
-Fix a long-standing bug where a URL preview would break if the discovered oEmbed failed to download.
@@ -1 +0,0 @@
-Remove the unspecced `room_alias` field from the [`/createRoom`](https://spec.matrix.org/v1.6/client-server-api/#post_matrixclientv3createroom) response.
@@ -1 +0,0 @@
-Refactor writing json data in `FileExfiltrationWriter`.
@@ -1 +0,0 @@
-Bump black from 22.12.0 to 23.1.0.
@@ -1 +0,0 @@
-Add media information to the command line [user data export tool](https://matrix-org.github.io/synapse/v1.79/usage/administration/admin_faq.html#how-can-i-export-user-data).
@@ -1 +0,0 @@
-Document using [Shibboleth](https://www.shibboleth.net/) as an OpenID Provider.
@@ -1 +0,0 @@
-Add an [admin API](https://matrix-org.github.io/synapse/latest/usage/administration/admin_api/index.html) to delete a [specific event report](https://spec.matrix.org/v1.6/client-server-api/#reporting-content).
@@ -1 +0,0 @@
-Allow use of the `/filter` Client-Server APIs on workers.
@@ -1 +0,0 @@
-Tighten the login ratelimit defaults.
@@ -1 +0,0 @@
-Remove the undocumented and unspecced `type` parameter to the `/thumbnail` endpoint.
@@ -1 +0,0 @@
-Fix a typo in an experimental config setting.
@@ -1 +0,0 @@
-Correct reference to `federation_verify_certificates` in configuration documentation.
@@ -1 +0,0 @@
-Fix a long-standing bug where the user directory search was not case-insensitive for accented characters.
@@ -1 +0,0 @@
-Refactor the media modules.
@@ -1 +0,0 @@
-Correct small documentation errors in some `MatrixFederationHttpClient` methods.
@@ -1 +0,0 @@
-Bump dawidd6/action-download-artifact from 2.25.0 to 2.26.0.
@@ -1 +0,0 @@
-Bump docker/login-action from 1 to 2.
@@ -1 +0,0 @@
-Bump actions/checkout from 2 to 3.
@@ -1 +0,0 @@
-Bump matrix-org/backend-meta from 1 to 2.
@@ -1 +0,0 @@
-Bump typing-extensions from 4.4.0 to 4.5.0.
@@ -1 +0,0 @@
-Bump types-opentracing from 2.4.10.1 to 2.4.10.3.
@@ -1 +0,0 @@
-Bump ruff from 0.0.237 to 0.0.252.
@@ -1 +0,0 @@
-Bump types-setuptools from 67.3.0.1 to 67.4.0.3.
@@ -1 +0,0 @@
-Fix a long-standing bug where an initial sync would not respond to changes to the list of ignored users if there was an initial sync cached.
@@ -1 +0,0 @@
-Improve type hints.
@@ -1 +0,0 @@
-Remove dangling reference to being a reference implementation in docstring.
@@ -1 +0,0 @@
-Correct the description of the behavior of `registration_shared_secret_path` on startup.
@@ -1 +0,0 @@
-Remove support for server-side aggregation of reactions.
@@ -1 +0,0 @@
-Refactor the media modules.
@@ -0,0 +1 @@
+Temp changelog entry (TODO).
@@ -1,3 +1,9 @@
+matrix-synapse-py3 (1.79.0~rc1) stable; urgency=medium
+
+  * New Synapse release 1.79.0rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 07 Mar 2023 12:03:49 +0000
+
 matrix-synapse-py3 (1.78.0) stable; urgency=medium

  * New Synapse release 1.78.0.
@@ -205,6 +205,7 @@ WORKERS_CONFIG: Dict[str, Dict[str, Any]] = {
            "^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/send",
            "^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$",
            "^/_matrix/client/(api/v1|r0|v3|unstable)/join/",
+            "^/_matrix/client/(api/v1|r0|v3|unstable)/knock/",
            "^/_matrix/client/(api/v1|r0|v3|unstable)/profile/",
            "^/_matrix/client/(v1|unstable/org.matrix.msc2716)/rooms/.*/batch_send",
        ],
@@ -675,17 +676,21 @@ def main(args: List[str], environ: MutableMapping[str, str]) -> None:
    if not os.path.exists(config_path):
        log("Generating base homeserver config")
        generate_base_homeserver_config()
-
+    else:
+        log("Base homeserver config exists—not regenerating")
    # This script may be run multiple times (mostly by Complement, see note at top of file).
    # Don't re-configure workers in this instance.
    mark_filepath = "/conf/workers_have_been_configured"
    if not os.path.exists(mark_filepath):
        # Always regenerate all other config files
+        log("Generating worker config files")
        generate_worker_files(environ, config_path, data_dir)

        # Mark workers as being configured
        with open(mark_filepath, "w") as f:
            f.write("")
+    else:
+        log("Worker config exists—not regenerating")

    # Lifted right out of start.py
    jemallocpath = "/usr/lib/%s-linux-gnu/libjemalloc.so.2" % (platform.machine(),)
@@ -252,6 +252,7 @@ information.
    ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state/
    ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$
    ^/_matrix/client/(api/v1|r0|v3|unstable)/join/
+    ^/_matrix/client/(api/v1|r0|v3|unstable)/knock/
    ^/_matrix/client/(api/v1|r0|v3|unstable)/profile/

    # Account data requests
@@ -2575,54 +2575,14 @@ files = [

 [[package]]
 name = "types-commonmark"
-version = "0.9.2.1"
+version = "0.9.2.2"
 description = "Typing stubs for commonmark"
 category = "dev"
 optional = false
 python-versions = "*"
 files = [
-    {file = "types-commonmark-0.9.2.1.tar.gz", hash = "sha256:db8277e6aeb83429265eccece98a24954a9a502dde7bc7cf840a8741abd96b86"},
-    {file = "types_commonmark-0.9.2.1-py3-none-any.whl", hash = "sha256:9d5f500cb7eced801bde728137b0a10667bd853d328db641d03141f189e3aab4"},
-]
-
-[[package]]
-name = "types-cryptography"
-version = "3.3.15"
-description = "Typing stubs for cryptography"
-category = "dev"
-optional = false
-python-versions = "*"
-files = [
-    {file = "types-cryptography-3.3.15.tar.gz", hash = "sha256:a7983a75a7b88a18f88832008f0ef140b8d1097888ec1a0824ec8fb7e105273b"},
-    {file = "types_cryptography-3.3.15-py3-none-any.whl", hash = "sha256:d9b0dd5465d7898d400850e7f35e5518aa93a7e23d3e11757cd81b4777089046"},
-]
-
-[package.dependencies]
-types-enum34 = "*"
-types-ipaddress = "*"
-
-[[package]]
-name = "types-enum34"
-version = "1.1.8"
-description = "Typing stubs for enum34"
-category = "dev"
-optional = false
-python-versions = "*"
-files = [
-    {file = "types-enum34-1.1.8.tar.gz", hash = "sha256:6f9c769641d06d73a55e11c14d38ac76fcd37eb545ce79cebb6eec9d50a64110"},
-    {file = "types_enum34-1.1.8-py3-none-any.whl", hash = "sha256:05058c7a495f6bfaaca0be4aeac3cce5cdd80a2bad2aab01fd49a20bf4a0209d"},
-]
-
-[[package]]
-name = "types-ipaddress"
-version = "1.0.8"
-description = "Typing stubs for ipaddress"
-category = "dev"
-optional = false
-python-versions = "*"
-files = [
-    {file = "types-ipaddress-1.0.8.tar.gz", hash = "sha256:a03df3be5935e50ba03fa843daabff539a041a28e73e0fce2c5705bee54d3841"},
-    {file = "types_ipaddress-1.0.8-py3-none-any.whl", hash = "sha256:4933b74da157ba877b1a705d64f6fa7742745e9ffd65e51011f370c11ebedb55"},
+    {file = "types-commonmark-0.9.2.2.tar.gz", hash = "sha256:f3259350634c2ce68ae503398430482f7cf44e5cae3d344995e916fbf453b4be"},
+    {file = "types_commonmark-0.9.2.2-py3-none-any.whl", hash = "sha256:d3d878692615e7fbe47bf19ba67497837b135812d665012a3d42219c1f2c3a61"},
 ]

 [[package]]
@@ -2651,42 +2611,42 @@ files = [

 [[package]]
 name = "types-pillow"
-version = "9.4.0.13"
+version = "9.4.0.17"
 description = "Typing stubs for Pillow"
 category = "dev"
 optional = false
 python-versions = "*"
 files = [
-    {file = "types-Pillow-9.4.0.13.tar.gz", hash = "sha256:4510aa98a28947bf63f2b29edebbd11b7cff8647d90b867cec9b3674c0a8c321"},
-    {file = "types_Pillow-9.4.0.13-py3-none-any.whl", hash = "sha256:14a8a19021b8fe569a9fef9edc64a8d8a4aef340e38669d4fb3dc05cfd941130"},
+    {file = "types-Pillow-9.4.0.17.tar.gz", hash = "sha256:7f0e871d2d46fbb6bc7deca3e02dc552cf9c1e8b49deb9595509551be3954e49"},
+    {file = "types_Pillow-9.4.0.17-py3-none-any.whl", hash = "sha256:f8b848a05f17cb4d53d245c59bf560372b9778d4cfaf9705f6245009bf9f65f3"},
 ]

 [[package]]
 name = "types-psycopg2"
-version = "2.9.21.4"
+version = "2.9.21.8"
 description = "Typing stubs for psycopg2"
 category = "dev"
 optional = false
 python-versions = "*"
 files = [
-    {file = "types-psycopg2-2.9.21.4.tar.gz", hash = "sha256:d43dda166a70d073ddac40718e06539836b5844c99b58ef8d4489a8df2edf5c0"},
-    {file = "types_psycopg2-2.9.21.4-py3-none-any.whl", hash = "sha256:6a05dca0856996aa37d7abe436751803bf47ec006cabbefea092e057f23bc95d"},
+    {file = "types-psycopg2-2.9.21.8.tar.gz", hash = "sha256:b629440ffcfdebd742fab07f777ff69aefdd19394a138c18e921a1964c3cf5f6"},
+    {file = "types_psycopg2-2.9.21.8-py3-none-any.whl", hash = "sha256:e747fbec6e0e2502b625bc7686d13cc62fc170e8ae920e5ba27fac946778eeb9"},
 ]

 [[package]]
 name = "types-pyopenssl"
-version = "22.1.0.2"
+version = "23.0.0.4"
 description = "Typing stubs for pyOpenSSL"
 category = "dev"
 optional = false
 python-versions = "*"
 files = [
-    {file = "types-pyOpenSSL-22.1.0.2.tar.gz", hash = "sha256:7a350e29e55bc3ee4571f996b4b1c18c4e4098947db45f7485b016eaa35b44bc"},
-    {file = "types_pyOpenSSL-22.1.0.2-py3-none-any.whl", hash = "sha256:54606a6afb203eb261e0fca9b7f75fa6c24d5ff71e13903c162ffb951c2c64c6"},
+    {file = "types-pyOpenSSL-23.0.0.4.tar.gz", hash = "sha256:8b3550b6e19d51ce78aabd724b0d8ebd962081a5fce95e7f85a592dfcdbc16bf"},
+    {file = "types_pyOpenSSL-23.0.0.4-py3-none-any.whl", hash = "sha256:ad49e15bb8bb2f251b8fc24776f414d877629e44b1b049240063ab013b5a6a7d"},
 ]

 [package.dependencies]
-types-cryptography = "*"
+cryptography = ">=35.0.0"

 [[package]]
 name = "types-pyyaml"
@@ -2717,14 +2677,14 @@ types-urllib3 = "<1.27"

 [[package]]
 name = "types-setuptools"
-version = "67.4.0.3"
+version = "67.5.0.0"
 description = "Typing stubs for setuptools"
 category = "dev"
 optional = false
 python-versions = "*"
 files = [
-    {file = "types-setuptools-67.4.0.3.tar.gz", hash = "sha256:19e958dfdbf1c5a628e54c2a7ee84935051afb7278d0c1cdb08ac194757ee3b1"},
-    {file = "types_setuptools-67.4.0.3-py3-none-any.whl", hash = "sha256:3c83c3a6363dd3ddcdd054796705605f0fa8b8e5a39390e07a05e5f7af054978"},
+    {file = "types-setuptools-67.5.0.0.tar.gz", hash = "sha256:fa6f231eeb27e86b1d6e8260f73de300e91f99c205b9a5e21debd49f3726a849"},
+    {file = "types_setuptools-67.5.0.0-py3-none-any.whl", hash = "sha256:f7f4bf4ab777e88631d3a387bbfdd4d480a2a4693ca896130f8ef738370377b8"},
 ]

 [[package]]
@@ -89,7 +89,7 @@ manifest-path = "rust/Cargo.toml"

 [tool.poetry]
 name = "matrix-synapse"
-version = "1.78.0"
+version = "1.79.0rc1"
 description = "Homeserver for the Matrix decentralised comms protocol"
 authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
 license = "Apache-2.0"
@@ -14,6 +14,7 @@

 #![feature(test)]
 use std::collections::BTreeSet;
+
 use synapse::push::{
    evaluator::PushRuleEvaluator, Condition, EventMatchCondition, FilteredPushRules, JsonValue,
    PushRules, SimpleJsonValue,
@@ -44,7 +45,6 @@ fn bench_match_exact(b: &mut Bencher) {
    let eval = PushRuleEvaluator::py_new(
        flattened_keys,
        false,
-        BTreeSet::new(),
        10,
        Some(0),
        Default::default(),
@@ -53,7 +53,6 @@ fn bench_match_exact(b: &mut Bencher) {
        vec![],
        false,
        false,
-        false,
    )
    .unwrap();

@@ -92,7 +91,6 @@ fn bench_match_word(b: &mut Bencher) {
    let eval = PushRuleEvaluator::py_new(
        flattened_keys,
        false,
-        BTreeSet::new(),
        10,
        Some(0),
        Default::default(),
@@ -101,7 +99,6 @@ fn bench_match_word(b: &mut Bencher) {
        vec![],
        false,
        false,
-        false,
    )
    .unwrap();

@@ -140,7 +137,6 @@ fn bench_match_word_miss(b: &mut Bencher) {
    let eval = PushRuleEvaluator::py_new(
        flattened_keys,
        false,
-        BTreeSet::new(),
        10,
        Some(0),
        Default::default(),
@@ -149,7 +145,6 @@ fn bench_match_word_miss(b: &mut Bencher) {
        vec![],
        false,
        false,
-        false,
    )
    .unwrap();

@@ -188,7 +183,6 @@ fn bench_eval_message(b: &mut Bencher) {
    let eval = PushRuleEvaluator::py_new(
        flattened_keys,
        false,
-        BTreeSet::new(),
        10,
        Some(0),
        Default::default(),
@@ -197,7 +191,6 @@ fn bench_eval_message(b: &mut Bencher) {
        vec![],
        false,
        false,
-        false,
    )
    .unwrap();

@@ -21,13 +21,13 @@ use lazy_static::lazy_static;
 use serde_json::Value;

 use super::KnownCondition;
-use crate::push::PushRule;
 use crate::push::RelatedEventMatchTypeCondition;
 use crate::push::SetTweak;
 use crate::push::TweakValue;
-use crate::push::{Action, ExactEventMatchCondition, SimpleJsonValue};
+use crate::push::{Action, EventPropertyIsCondition, SimpleJsonValue};
 use crate::push::{Condition, EventMatchTypeCondition};
 use crate::push::{EventMatchCondition, EventMatchPatternType};
+use crate::push::{EventPropertyIsTypeCondition, PushRule};

 const HIGHLIGHT_ACTION: Action = Action::SetTweak(SetTweak {
    set_tweak: Cow::Borrowed("highlight"),
@@ -144,7 +144,12 @@ pub const BASE_APPEND_OVERRIDE_RULES: &[PushRule] = &[
    PushRule {
        rule_id: Cow::Borrowed(".org.matrix.msc3952.is_user_mention"),
        priority_class: 5,
-        conditions: Cow::Borrowed(&[Condition::Known(KnownCondition::IsUserMention)]),
+        conditions: Cow::Borrowed(&[Condition::Known(
+            KnownCondition::ExactEventPropertyContainsType(EventPropertyIsTypeCondition {
+                key: Cow::Borrowed("content.org.matrix.msc3952.mentions.user_ids"),
+                value_type: Cow::Borrowed(&EventMatchPatternType::UserId),
+            }),
+        )]),
        actions: Cow::Borrowed(&[Action::Notify, HIGHLIGHT_ACTION, SOUND_ACTION]),
        default: true,
        default_enabled: true,
@@ -161,7 +166,7 @@ pub const BASE_APPEND_OVERRIDE_RULES: &[PushRule] = &[
        rule_id: Cow::Borrowed(".org.matrix.msc3952.is_room_mention"),
        priority_class: 5,
        conditions: Cow::Borrowed(&[
-            Condition::Known(KnownCondition::ExactEventMatch(ExactEventMatchCondition {
+            Condition::Known(KnownCondition::EventPropertyIs(EventPropertyIsCondition {
                key: Cow::Borrowed("content.org.matrix.msc3952.mentions.room"),
                value: Cow::Borrowed(&SimpleJsonValue::Bool(true)),
            })),
@@ -13,9 +13,8 @@
 // limitations under the License.

 use std::borrow::Cow;
-use std::collections::{BTreeMap, BTreeSet};
+use std::collections::BTreeMap;

-use crate::push::{EventMatchPatternType, JsonValue};
 use anyhow::{Context, Error};
 use lazy_static::lazy_static;
 use log::warn;
@@ -24,9 +23,10 @@ use regex::Regex;

 use super::{
    utils::{get_glob_matcher, get_localpart_from_id, GlobMatchType},
-    Action, Condition, ExactEventMatchCondition, FilteredPushRules, KnownCondition,
+    Action, Condition, EventPropertyIsCondition, FilteredPushRules, KnownCondition,
    SimpleJsonValue,
 };
+use crate::push::{EventMatchPatternType, JsonValue};

 lazy_static! {
    /// Used to parse the `is` clause in the room member count condition.
@@ -72,8 +72,6 @@ pub struct PushRuleEvaluator {

    /// True if the event has a mentions property and MSC3952 support is enabled.
    has_mentions: bool,
-    /// The user mentions that were part of the message.
-    user_mentions: BTreeSet<String>,

    /// The number of users in the room.
    room_member_count: u64,
@@ -99,9 +97,6 @@ pub struct PushRuleEvaluator {
    /// flag as MSC1767 (extensible events core).
    msc3931_enabled: bool,

-    /// If MSC3758 (exact_event_match push rule condition) is enabled.
-    msc3758_exact_event_match: bool,
-
    /// If MSC3966 (exact_event_property_contains push rule condition) is enabled.
    msc3966_exact_event_property_contains: bool,
 }
@@ -114,7 +109,6 @@ impl PushRuleEvaluator {
    pub fn py_new(
        flattened_keys: BTreeMap<String, JsonValue>,
        has_mentions: bool,
-        user_mentions: BTreeSet<String>,
        room_member_count: u64,
        sender_power_level: Option<i64>,
        notification_power_levels: BTreeMap<String, i64>,
@@ -122,7 +116,6 @@ impl PushRuleEvaluator {
        related_event_match_enabled: bool,
        room_version_feature_flags: Vec<String>,
        msc3931_enabled: bool,
-        msc3758_exact_event_match: bool,
        msc3966_exact_event_property_contains: bool,
    ) -> Result<Self, Error> {
        let body = match flattened_keys.get("content.body") {
@@ -134,7 +127,6 @@ impl PushRuleEvaluator {
            flattened_keys,
            body,
            has_mentions,
-            user_mentions,
            room_member_count,
            notification_power_levels,
            sender_power_level,
@@ -142,7 +134,6 @@ impl PushRuleEvaluator {
            related_event_match_enabled,
            room_version_feature_flags,
            msc3931_enabled,
-            msc3758_exact_event_match,
            msc3966_exact_event_property_contains,
        })
    }
@@ -279,8 +270,8 @@ impl PushRuleEvaluator {

                self.match_event_match(&self.flattened_keys, &event_match.key, pattern)?
            }
-            KnownCondition::ExactEventMatch(exact_event_match) => {
-                self.match_exact_event_match(exact_event_match)?
+            KnownCondition::EventPropertyIs(event_property_is) => {
+                self.match_event_property_is(event_property_is)?
            }
            KnownCondition::RelatedEventMatch(event_match) => self.match_related_event_match(
                &event_match.rel_type.clone(),
@@ -310,15 +301,30 @@ impl PushRuleEvaluator {
                    Some(Cow::Borrowed(pattern)),
                )?
            }
-            KnownCondition::ExactEventPropertyContains(exact_event_match) => {
-                self.match_exact_event_property_contains(exact_event_match)?
-            }
-            KnownCondition::IsUserMention => {
-                if let Some(uid) = user_id {
-                    self.user_mentions.contains(uid)
+            KnownCondition::ExactEventPropertyContains(event_property_is) => self
+                .match_exact_event_property_contains(
+                    event_property_is.key.clone(),
+                    event_property_is.value.clone(),
+                )?,
+            KnownCondition::ExactEventPropertyContainsType(exact_event_match) => {
+                // The `pattern_type` can either be "user_id" or "user_localpart",
+                // either way if we don't have a `user_id` then the condition can't
+                // match.
+                let user_id = if let Some(user_id) = user_id {
+                    user_id
                } else {
-                    false
-                }
+                    return Ok(false);
+                };
+
+                let pattern = match &*exact_event_match.value_type {
+                    EventMatchPatternType::UserId => user_id,
+                    EventMatchPatternType::UserLocalpart => get_localpart_from_id(user_id)?,
+                };
+
+                self.match_exact_event_property_contains(
+                    exact_event_match.key.clone(),
+                    Cow::Borrowed(&SimpleJsonValue::Str(pattern.to_string())),
+                )?
            }
            KnownCondition::ContainsDisplayName => {
                if let Some(dn) = display_name {
@@ -394,20 +400,15 @@ impl PushRuleEvaluator {
        compiled_pattern.is_match(haystack)
    }

-    /// Evaluates a `exact_event_match` condition. (MSC3758)
-    fn match_exact_event_match(
+    /// Evaluates a `event_property_is` condition.
+    fn match_event_property_is(
        &self,
-        exact_event_match: &ExactEventMatchCondition,
+        event_property_is: &EventPropertyIsCondition,
    ) -> Result<bool, Error> {
-        // First check if the feature is enabled.
-        if !self.msc3758_exact_event_match {
-            return Ok(false);
-        }
-
-        let value = &exact_event_match.value;
+        let value = &event_property_is.value;

        let haystack = if let Some(JsonValue::Value(haystack)) =
-            self.flattened_keys.get(&*exact_event_match.key)
+            self.flattened_keys.get(&*event_property_is.key)
        {
            haystack
        } else {
@@ -453,27 +454,24 @@ impl PushRuleEvaluator {
        }
    }

-    /// Evaluates a `exact_event_property_contains` condition. (MSC3758)
+    /// Evaluates a `exact_event_property_contains` condition. (MSC3966)
    fn match_exact_event_property_contains(
        &self,
-        exact_event_match: &ExactEventMatchCondition,
+        key: Cow<str>,
+        value: Cow<SimpleJsonValue>,
    ) -> Result<bool, Error> {
        // First check if the feature is enabled.
        if !self.msc3966_exact_event_property_contains {
            return Ok(false);
        }

-        let value = &exact_event_match.value;
-
-        let haystack = if let Some(JsonValue::Array(haystack)) =
-            self.flattened_keys.get(&*exact_event_match.key)
-        {
+        let haystack = if let Some(JsonValue::Array(haystack)) = self.flattened_keys.get(&*key) {
            haystack
        } else {
            return Ok(false);
        };

-        Ok(haystack.contains(&**value))
+        Ok(haystack.contains(&value))
    }

    /// Match the member count against an 'is' condition
@@ -510,7 +508,6 @@ fn push_rule_evaluator() {
    let evaluator = PushRuleEvaluator::py_new(
        flattened_keys,
        false,
-        BTreeSet::new(),
        10,
        Some(0),
        BTreeMap::new(),
@@ -519,7 +516,6 @@ fn push_rule_evaluator() {
        vec![],
        true,
        true,
-        true,
    )
    .unwrap();

@@ -542,7 +538,6 @@ fn test_requires_room_version_supports_condition() {
    let evaluator = PushRuleEvaluator::py_new(
        flattened_keys,
        false,
-        BTreeSet::new(),
        10,
        Some(0),
        BTreeMap::new(),
@@ -551,7 +546,6 @@ fn test_requires_room_version_supports_condition() {
        flags,
        true,
        true,
-        true,
    )
    .unwrap();

@@ -331,17 +331,20 @@ pub enum KnownCondition {
    // Identical to event_match but gives predefined patterns. Cannot be added by users.
    #[serde(skip_deserializing, rename = "event_match")]
    EventMatchType(EventMatchTypeCondition),
-    #[serde(rename = "com.beeper.msc3758.exact_event_match")]
-    ExactEventMatch(ExactEventMatchCondition),
+    EventPropertyIs(EventPropertyIsCondition),
    #[serde(rename = "im.nheko.msc3664.related_event_match")]
    RelatedEventMatch(RelatedEventMatchCondition),
    // Identical to related_event_match but gives predefined patterns. Cannot be added by users.
    #[serde(skip_deserializing, rename = "im.nheko.msc3664.related_event_match")]
    RelatedEventMatchType(RelatedEventMatchTypeCondition),
    #[serde(rename = "org.matrix.msc3966.exact_event_property_contains")]
-    ExactEventPropertyContains(ExactEventMatchCondition),
-    #[serde(rename = "org.matrix.msc3952.is_user_mention")]
-    IsUserMention,
+    ExactEventPropertyContains(EventPropertyIsCondition),
+    // Identical to exact_event_property_contains but gives predefined patterns. Cannot be added by users.
+    #[serde(
+        skip_deserializing,
+        rename = "org.matrix.msc3966.exact_event_property_contains"
+    )]
+    ExactEventPropertyContainsType(EventPropertyIsTypeCondition),
    ContainsDisplayName,
    RoomMemberCount {
        #[serde(skip_serializing_if = "Option::is_none")]
@@ -391,13 +394,22 @@ pub struct EventMatchTypeCondition {
    pub pattern_type: Cow<'static, EventMatchPatternType>,
 }

-/// The body of a [`Condition::ExactEventMatch`]
+/// The body of a [`Condition::EventPropertyIs`]
 #[derive(Serialize, Deserialize, Debug, Clone)]
-pub struct ExactEventMatchCondition {
+pub struct EventPropertyIsCondition {
    pub key: Cow<'static, str>,
    pub value: Cow<'static, SimpleJsonValue>,
 }

+/// The body of a [`Condition::EventPropertyIs`] that uses user_id or user_localpart as a pattern.
+#[derive(Serialize, Debug, Clone)]
+pub struct EventPropertyIsTypeCondition {
+    pub key: Cow<'static, str>,
+    // During serialization, the pattern_type property gets replaced with a
+    // pattern property of the correct value in synapse.push.clientformat.format_push_rules_for_user.
+    pub value_type: Cow<'static, EventMatchPatternType>,
+}
+
 /// The body of a [`Condition::RelatedEventMatch`]
 #[derive(Serialize, Deserialize, Debug, Clone)]
 pub struct RelatedEventMatchCondition {
@@ -698,55 +710,41 @@ fn test_deserialize_unstable_msc3931_condition() {
 }

 #[test]
-fn test_deserialize_unstable_msc3758_condition() {
+fn test_deserialize_event_property_is_condition() {
    // A string condition should work.
-    let json =
-        r#"{"kind":"com.beeper.msc3758.exact_event_match","key":"content.value","value":"foo"}"#;
+    let json = r#"{"kind":"event_property_is","key":"content.value","value":"foo"}"#;

    let condition: Condition = serde_json::from_str(json).unwrap();
    assert!(matches!(
        condition,
-        Condition::Known(KnownCondition::ExactEventMatch(_))
+        Condition::Known(KnownCondition::EventPropertyIs(_))
    ));

    // A boolean condition should work.
-    let json =
-        r#"{"kind":"com.beeper.msc3758.exact_event_match","key":"content.value","value":true}"#;
+    let json = r#"{"kind":"event_property_is","key":"content.value","value":true}"#;

    let condition: Condition = serde_json::from_str(json).unwrap();
    assert!(matches!(
        condition,
-        Condition::Known(KnownCondition::ExactEventMatch(_))
+        Condition::Known(KnownCondition::EventPropertyIs(_))
    ));

    // An integer condition should work.
-    let json = r#"{"kind":"com.beeper.msc3758.exact_event_match","key":"content.value","value":1}"#;
+    let json = r#"{"kind":"event_property_is","key":"content.value","value":1}"#;

    let condition: Condition = serde_json::from_str(json).unwrap();
    assert!(matches!(
        condition,
-        Condition::Known(KnownCondition::ExactEventMatch(_))
+        Condition::Known(KnownCondition::EventPropertyIs(_))
    ));

    // A null condition should work
-    let json =
-        r#"{"kind":"com.beeper.msc3758.exact_event_match","key":"content.value","value":null}"#;
+    let json = r#"{"kind":"event_property_is","key":"content.value","value":null}"#;

    let condition: Condition = serde_json::from_str(json).unwrap();
    assert!(matches!(
        condition,
-        Condition::Known(KnownCondition::ExactEventMatch(_))
-    ));
-}
-
-#[test]
-fn test_deserialize_unstable_msc3952_user_condition() {
-    let json = r#"{"kind":"org.matrix.msc3952.is_user_mention"}"#;
-
-    let condition: Condition = serde_json::from_str(json).unwrap();
-    assert!(matches!(
-        condition,
-        Condition::Known(KnownCondition::IsUserMention)
+        Condition::Known(KnownCondition::EventPropertyIs(_))
    ));
 }

@@ -59,6 +59,11 @@ Run the complement test suite on Synapse.
        is important.
        Not suitable for use in CI in case the editable environment is impure.

+  --rebuild-editable
+        Force a rebuild of the editable build of Synapse.
+        This is occasionally useful if the built-in rebuild detection with
+        --editable fails, e.g. when changing configure_workers_and_start.py.
+
 For help on arguments to 'go test', run 'go help testflag'.
 EOF
 }
@@ -82,6 +87,9 @@ while [ $# -ge 1 ]; do
        "-e"|"--editable")
            use_editable_synapse=1
            ;;
+        "--rebuild-editable")
+            rebuild_editable_synapse=1
+            ;;
        *)
            # unknown arg: presumably an argument to gotest. break the loop.
            break
@@ -116,7 +124,9 @@ if [ -n "$use_editable_synapse" ]; then
    fi

    editable_mount="$(realpath .):/editable-src:z"
-    if docker inspect complement-synapse-editable &>/dev/null; then
+    if [ -n "$rebuild_editable_synapse" ]; then
+        unset skip_docker_build
+    elif docker inspect complement-synapse-editable &>/dev/null; then
        # complement-synapse-editable already exists: see if we can still use it:
        # - The Rust module must still be importable; it will fail to import if the Rust source has changed.
        # - The Poetry lock file must be the same (otherwise we assume dependencies have changed)
@@ -112,7 +112,7 @@ python3 -m black "${files[@]}"

 # Catch any common programming mistakes in Python code.
 # --quiet suppresses the update check.
-ruff --quiet "${files[@]}"
+ruff --quiet --fix "${files[@]}"

 # Catch any common programming mistakes in Rust code.
 #
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-from typing import Any, Collection, Dict, Mapping, Optional, Sequence, Set, Tuple, Union
+from typing import Any, Collection, Dict, Mapping, Optional, Sequence, Tuple, Union

 from synapse.types import JsonDict, JsonValue

@@ -58,7 +58,6 @@ class PushRuleEvaluator:
        self,
        flattened_keys: Mapping[str, JsonValue],
        has_mentions: bool,
-        user_mentions: Set[str],
        room_member_count: int,
        sender_power_level: Optional[int],
        notification_power_levels: Mapping[str, int],
@@ -66,7 +65,6 @@ class PushRuleEvaluator:
        related_event_match_enabled: bool,
        room_version_feature_flags: Tuple[str, ...],
        msc3931_enabled: bool,
-        msc3758_exact_event_match: bool,
        msc3966_exact_event_property_contains: bool,
    ): ...
    def run(
@@ -166,23 +166,20 @@ class ExperimentalConfig(Config):
        # MSC3391: Removing account data.
        self.msc3391_enabled = experimental.get("msc3391_enabled", False)

-        # MSC3925: do not replace events with their edits
-        self.msc3925_inhibit_edit = experimental.get("msc3925_inhibit_edit", False)
-
-        # MSC3758: exact_event_match push rule condition
-        self.msc3758_exact_event_match = experimental.get(
-            "msc3758_exact_event_match", False
-        )
-
        # MSC3873: Disambiguate event_match keys.
        self.msc3873_escape_event_match_key = experimental.get(
            "msc3873_escape_event_match_key", False
        )

-        # MSC3952: Intentional mentions, this depends on MSC3758.
+        # MSC3966: exact_event_property_contains push rule condition.
+        self.msc3966_exact_event_property_contains = experimental.get(
+            "msc3966_exact_event_property_contains", False
+        )
+
+        # MSC3952: Intentional mentions, this depends on MSC3966.
        self.msc3952_intentional_mentions = (
            experimental.get("msc3952_intentional_mentions", False)
-            and self.msc3758_exact_event_match
+            and self.msc3966_exact_event_property_contains
        )

        # MSC3959: Do not generate notifications for edits.
@@ -194,3 +191,6 @@ class ExperimentalConfig(Config):
        self.msc3966_exact_event_property_contains = experimental.get(
            "msc3966_exact_event_property_contains", False
        )
+
+        # MSC3967: Do not require UIA when first uploading cross signing keys
+        self.msc3967_enabled = experimental.get("msc3967_enabled", False)
@@ -168,21 +168,13 @@ async def check_state_independent_auth_rules(
        return

    # 2. Reject if event has auth_events that: ...
+    auth_events = await store.get_events(
+        event.auth_event_ids(),
+        redact_behaviour=EventRedactBehaviour.as_is,
+        allow_rejected=True,
+    )
    if batched_auth_events:
-        auth_event_ids = event.auth_event_ids()
-        auth_events = dict(batched_auth_events)
-        if set(auth_event_ids) - batched_auth_events.keys():
-            auth_events.update(
-                await store.get_events(
-                    set(auth_event_ids) - batched_auth_events.keys()
-                )
-            )
-    else:
-        auth_events = await store.get_events(
-            event.auth_event_ids(),
-            redact_behaviour=EventRedactBehaviour.as_is,
-            allow_rejected=True,
-        )
+        auth_events.update(batched_auth_events)

    room_id = event.room_id
    auth_dict: MutableStateMap[str] = {}
@@ -135,8 +135,6 @@ class EventContext(UnpersistedEventContextBase):
    delta_ids: Optional[StateMap[str]] = None
    app_service: Optional[ApplicationService] = None

-    _state_map_before_event: Optional[StateMap[str]] = None
-
    partial_state: bool = False

    @staticmethod
@@ -295,11 +293,6 @@ class EventContext(UnpersistedEventContextBase):
            Maps a (type, state_key) to the event ID of the state event matching
            this tuple.
        """
-        if self._state_map_before_event is not None:
-            if state_filter is not None:
-                return state_filter.filter_state(self._state_map_before_event)
-            return self._state_map_before_event
-
        assert self.state_group_before_event is not None
        return await self._storage.state.get_state_ids_for_group(
            self.state_group_before_event, state_filter
@@ -381,16 +374,26 @@ class UnpersistedEventContext(UnpersistedEventContextBase):

        events_and_persisted_context = []
        for event, unpersisted_context in amended_events_and_context:
-            context = EventContext(
-                storage=unpersisted_context._storage,
-                state_group=unpersisted_context.state_group_after_event,
-                state_group_before_event=unpersisted_context.state_group_before_event,
-                state_delta_due_to_event=unpersisted_context.state_delta_due_to_event,
-                partial_state=unpersisted_context.partial_state,
-                prev_group=unpersisted_context.prev_group_for_state_group_before_event,
-                delta_ids=unpersisted_context.delta_ids_to_state_group_before_event,
-                state_map_before_event=unpersisted_context.state_map_before_event,
-            )
+            if event.is_state():
+                context = EventContext(
+                    storage=unpersisted_context._storage,
+                    state_group=unpersisted_context.state_group_after_event,
+                    state_group_before_event=unpersisted_context.state_group_before_event,
+                    state_delta_due_to_event=unpersisted_context.state_delta_due_to_event,
+                    partial_state=unpersisted_context.partial_state,
+                    prev_group=unpersisted_context.state_group_before_event,
+                    delta_ids=unpersisted_context.state_delta_due_to_event,
+                )
+            else:
+                context = EventContext(
+                    storage=unpersisted_context._storage,
+                    state_group=unpersisted_context.state_group_after_event,
+                    state_group_before_event=unpersisted_context.state_group_before_event,
+                    state_delta_due_to_event=unpersisted_context.state_delta_due_to_event,
+                    partial_state=unpersisted_context.partial_state,
+                    prev_group=unpersisted_context.prev_group_for_state_group_before_event,
+                    delta_ids=unpersisted_context.delta_ids_to_state_group_before_event,
+                )
            events_and_persisted_context.append((event, context))
        return events_and_persisted_context

@@ -38,8 +38,7 @@ from synapse.api.constants import (
 )
 from synapse.api.errors import Codes, SynapseError
 from synapse.api.room_versions import RoomVersion
-from synapse.types import JsonDict
-from synapse.util.frozenutils import unfreeze
+from synapse.types import JsonDict, Requester

 from . import EventBase

@@ -317,8 +316,9 @@ class SerializeEventConfig:
    as_client_event: bool = True
    # Function to convert from federation format to client format
    event_format: Callable[[JsonDict], JsonDict] = format_event_for_client_v1
-    # ID of the user's auth token - used for namespacing of transaction IDs
-    token_id: Optional[int] = None
+    # The entity that requested the event. This is used to determine whether to include
+    # the transaction_id in the unsigned section of the event.
+    requester: Optional[Requester] = None
    # List of event fields to include. If empty, all fields will be returned.
    only_event_fields: Optional[List[str]] = None
    # Some events can have stripped room state stored in the `unsigned` field.
@@ -368,11 +368,24 @@ def serialize_event(
            e.unsigned["redacted_because"], time_now_ms, config=config
        )

-    if config.token_id is not None:
-        if config.token_id == getattr(e.internal_metadata, "token_id", None):
-            txn_id = getattr(e.internal_metadata, "txn_id", None)
-            if txn_id is not None:
-                d["unsigned"]["transaction_id"] = txn_id
+    # If we have a txn_id saved in the internal_metadata, we should include it in the
+    # unsigned section of the event if it was sent by the same session as the one
+    # requesting the event.
+    # There is a special case for guests, because they only have one access token
+    # without associated access_token_id, so we always include the txn_id for events
+    # they sent.
+    txn_id = getattr(e.internal_metadata, "txn_id", None)
+    if txn_id is not None and config.requester is not None:
+        event_token_id = getattr(e.internal_metadata, "token_id", None)
+        if config.requester.user.to_string() == e.sender and (
+            (
+                event_token_id is not None
+                and config.requester.access_token_id is not None
+                and event_token_id == config.requester.access_token_id
+            )
+            or config.requester.is_guest
+        ):
+            d["unsigned"]["transaction_id"] = txn_id

    # invite_room_state and knock_room_state are a list of stripped room state events
    # that are meant to provide metadata about a room to an invitee/knocker. They are
@@ -403,14 +416,6 @@ class EventClientSerializer:
    clients.
    """

-    def __init__(self, inhibit_replacement_via_edits: bool = False):
-        """
-        Args:
-            inhibit_replacement_via_edits: If this is set to True, then events are
-               never replaced by their edits.
-        """
-        self._inhibit_replacement_via_edits = inhibit_replacement_via_edits
-
    def serialize_event(
        self,
        event: Union[JsonDict, EventBase],
@@ -418,7 +423,6 @@ class EventClientSerializer:
        *,
        config: SerializeEventConfig = _DEFAULT_SERIALIZE_EVENT_CONFIG,
        bundle_aggregations: Optional[Dict[str, "BundledAggregations"]] = None,
-        apply_edits: bool = True,
    ) -> JsonDict:
        """Serializes a single event.

@@ -428,10 +432,7 @@ class EventClientSerializer:
            config: Event serialization config
            bundle_aggregations: A map from event_id to the aggregations to be bundled
               into the event.
-            apply_edits: Whether the content of the event should be modified to reflect
-               any replacement in `bundle_aggregations[<event_id>].replace`.
-               See also the `inhibit_replacement_via_edits` constructor arg: if that is
-               set to True, then this argument is ignored.
+
        Returns:
            The serialized event
        """
@@ -450,38 +451,10 @@ class EventClientSerializer:
                    config,
                    bundle_aggregations,
                    serialized_event,
-                    apply_edits=apply_edits,
                )

        return serialized_event

-    def _apply_edit(
-        self, orig_event: EventBase, serialized_event: JsonDict, edit: EventBase
-    ) -> None:
-        """Replace the content, preserving existing relations of the serialized event.
-
-        Args:
-            orig_event: The original event.
-            serialized_event: The original event, serialized. This is modified.
-            edit: The event which edits the above.
-        """
-
-        # Ensure we take copies of the edit content, otherwise we risk modifying
-        # the original event.
-        edit_content = edit.content.copy()
-
-        # Unfreeze the event content if necessary, so that we may modify it below
-        edit_content = unfreeze(edit_content)
-        serialized_event["content"] = edit_content.get("m.new_content", {})
-
-        # Check for existing relations
-        relates_to = orig_event.content.get("m.relates_to")
-        if relates_to:
-            # Keep the relations, ensuring we use a dict copy of the original
-            serialized_event["content"]["m.relates_to"] = relates_to.copy()
-        else:
-            serialized_event["content"].pop("m.relates_to", None)
-
    def _inject_bundled_aggregations(
        self,
        event: EventBase,
@@ -489,7 +462,6 @@ class EventClientSerializer:
        config: SerializeEventConfig,
        bundled_aggregations: Dict[str, "BundledAggregations"],
        serialized_event: JsonDict,
-        apply_edits: bool,
    ) -> None:
        """Potentially injects bundled aggregations into the unsigned portion of the serialized event.

@@ -504,9 +476,6 @@ class EventClientSerializer:
                While serializing the bundled aggregations this map may be searched
                again for additional events in a recursive manner.
            serialized_event: The serialized event which may be modified.
-            apply_edits: Whether the content of the event should be modified to reflect
-               any replacement in `aggregations.replace` (subject to the
-               `inhibit_replacement_via_edits` constructor arg).
        """

        # We have already checked that aggregations exist for this event.
@@ -522,11 +491,6 @@ class EventClientSerializer:
            ] = event_aggregations.references

        if event_aggregations.replace:
-            # If there is an edit, optionally apply it to the event.
-            edit = event_aggregations.replace
-            if apply_edits and not self._inhibit_replacement_via_edits:
-                self._apply_edit(event, serialized_event, edit)
-
            # Include information about it in the relations dict.
            #
            # Matrix spec v1.5 (https://spec.matrix.org/v1.5/client-server-api/#server-side-aggregation-of-mreplace-relationships)
@@ -534,10 +498,7 @@ class EventClientSerializer:
            # `sender` of the edit; however MSC3925 proposes extending it to the whole
            # of the edit, which is what we do here.
            serialized_aggregations[RelationTypes.REPLACE] = self.serialize_event(
-                edit,
-                time_now,
-                config=config,
-                apply_edits=False,
+                event_aggregations.replace, time_now, config=config
            )

        # Include any threaded replies to this event.
@@ -155,9 +155,6 @@ class AccountDataHandler:
            max_stream_id = await self._store.remove_account_data_for_room(
                user_id, room_id, account_data_type
            )
-            if max_stream_id is None:
-                # The referenced account data did not exist, so no delete occurred.
-                return None

            self._notifier.on_new_event(
                StreamKeyType.ACCOUNT_DATA, max_stream_id, users=[user_id]
@@ -230,9 +227,6 @@ class AccountDataHandler:
            max_stream_id = await self._store.remove_account_data_for_user(
                user_id, account_data_type
            )
-            if max_stream_id is None:
-                # The referenced account data did not exist, so no delete occurred.
-                return None

            self._notifier.on_new_event(
                StreamKeyType.ACCOUNT_DATA, max_stream_id, users=[user_id]
@@ -248,7 +242,6 @@ class AccountDataHandler:
                instance_name=random.choice(self._account_data_writers),
                user_id=user_id,
                account_data_type=account_data_type,
-                content={},
            )
            return response["max_stream_id"]

@@ -1301,6 +1301,20 @@ class E2eKeysHandler:

        return desired_key_data

+    async def is_cross_signing_set_up_for_user(self, user_id: str) -> bool:
+        """Checks if the user has cross-signing set up
+
+        Args:
+            user_id: The user to check
+
+        Returns:
+            True if the user has cross-signing set up, False otherwise
+        """
+        existing_master_key = await self.store.get_e2e_cross_signing_key(
+            user_id, "master"
+        )
+        return existing_master_key is not None
+

 def _check_cross_signing_key(
    key: JsonDict, user_id: str, key_type: str, signing_key: Optional[VerifyKey] = None
@@ -63,19 +63,9 @@ class EventAuthHandler:
            self._store, event, batched_auth_events
        )
        auth_event_ids = event.auth_event_ids()
-        logger.info("Batched auth events %s", list(batched_auth_events.keys()))
-        logger.info("auth events %s", auth_event_ids)
+        auth_events_by_id = await self._store.get_events(auth_event_ids)
        if batched_auth_events:
-            auth_events_by_id = dict(batched_auth_events)
-            if set(auth_event_ids) - set(batched_auth_events):
-                logger.info("fetching %s", set(auth_event_ids) - set(batched_auth_events))
-                auth_events_by_id.update(
-                    await self._store.get_events(
-                        set(auth_event_ids) - set(batched_auth_events)
-                    )
-                )
-        else:
-            auth_events_by_id = await self._store.get_events(auth_event_ids)
+            auth_events_by_id.update(batched_auth_events)
        check_state_dependent_auth_rules(event, auth_events_by_id.values())

    def compute_auth_events(
@@ -23,7 +23,7 @@ from synapse.events.utils import SerializeEventConfig
 from synapse.handlers.presence import format_user_presence_state
 from synapse.storage.databases.main.events_worker import EventRedactBehaviour
 from synapse.streams.config import PaginationConfig
-from synapse.types import JsonDict, UserID
+from synapse.types import JsonDict, Requester, UserID
 from synapse.visibility import filter_events_for_client

 if TYPE_CHECKING:
@@ -46,13 +46,12 @@ class EventStreamHandler:

    async def get_stream(
        self,
-        auth_user_id: str,
+        requester: Requester,
        pagin_config: PaginationConfig,
        timeout: int = 0,
        as_client_event: bool = True,
        affect_presence: bool = True,
        room_id: Optional[str] = None,
-        is_guest: bool = False,
    ) -> JsonDict:
        """Fetches the events stream for a given user."""

@@ -62,13 +61,12 @@ class EventStreamHandler:
                raise SynapseError(403, "This room has been blocked on this server")

        # send any outstanding server notices to the user.
-        await self._server_notices_sender.on_user_syncing(auth_user_id)
+        await self._server_notices_sender.on_user_syncing(requester.user.to_string())

-        auth_user = UserID.from_string(auth_user_id)
        presence_handler = self.hs.get_presence_handler()

        context = await presence_handler.user_syncing(
-            auth_user_id,
+            requester.user.to_string(),
            affect_presence=affect_presence,
            presence_state=PresenceState.ONLINE,
        )
@@ -82,10 +80,10 @@ class EventStreamHandler:
                timeout = random.randint(int(timeout * 0.9), int(timeout * 1.1))

            stream_result = await self.notifier.get_events_for(
-                auth_user,
+                requester.user,
                pagin_config,
                timeout,
-                is_guest=is_guest,
+                is_guest=requester.is_guest,
                explicit_room_id=room_id,
            )
            events = stream_result.events
@@ -102,7 +100,7 @@ class EventStreamHandler:
                    if event.membership != Membership.JOIN:
                        continue
                    # Send down presence.
-                    if event.state_key == auth_user_id:
+                    if event.state_key == requester.user.to_string():
                        # Send down presence for everyone in the room.
                        users: Iterable[str] = await self.store.get_users_in_room(
                            event.room_id
@@ -124,7 +122,9 @@ class EventStreamHandler:
            chunks = self._event_serializer.serialize_events(
                events,
                time_now,
-                config=SerializeEventConfig(as_client_event=as_client_event),
+                config=SerializeEventConfig(
+                    as_client_event=as_client_event, requester=requester
+                ),
            )

            chunk = {
@@ -392,7 +392,7 @@ class FederationHandler:
                get_prev_content=False,
            )

-            # We set `check_history_visibility_only` as we might otherwise get false
+            # We unset `filter_out_erased_senders` as we might otherwise get false
            # positives from users having been erased.
            filtered_extremities = await filter_events_for_server(
                self._storage_controllers,
@@ -400,7 +400,7 @@ class FederationHandler:
                self.server_name,
                events_to_check,
                redact=False,
-                check_history_visibility_only=True,
+                filter_out_erased_senders=False,
            )
            if filtered_extremities:
                extremities_to_request.append(bp.event_id)
@@ -318,11 +318,9 @@ class InitialSyncHandler:
        )
        is_peeking = member_event_id is None

-        user_id = requester.user.to_string()
-
        if membership == Membership.JOIN:
            result = await self._room_initial_sync_joined(
-                user_id, room_id, pagin_config, membership, is_peeking
+                requester, room_id, pagin_config, membership, is_peeking
            )
        elif membership == Membership.LEAVE:
            # The member_event_id will always be available if membership is set
@@ -330,10 +328,16 @@ class InitialSyncHandler:
            assert member_event_id

            result = await self._room_initial_sync_parted(
-                user_id, room_id, pagin_config, membership, member_event_id, is_peeking
+                requester,
+                room_id,
+                pagin_config,
+                membership,
+                member_event_id,
+                is_peeking,
            )

        account_data_events = []
+        user_id = requester.user.to_string()
        tags = await self.store.get_tags_for_room(user_id, room_id)
        if tags:
            account_data_events.append(
@@ -350,7 +354,7 @@ class InitialSyncHandler:

    async def _room_initial_sync_parted(
        self,
-        user_id: str,
+        requester: Requester,
        room_id: str,
        pagin_config: PaginationConfig,
        membership: str,
@@ -369,13 +373,17 @@ class InitialSyncHandler:
        )

        messages = await filter_events_for_client(
-            self._storage_controllers, user_id, messages, is_peeking=is_peeking
+            self._storage_controllers,
+            requester.user.to_string(),
+            messages,
+            is_peeking=is_peeking,
        )

        start_token = StreamToken.START.copy_and_replace(StreamKeyType.ROOM, token)
        end_token = StreamToken.START.copy_and_replace(StreamKeyType.ROOM, stream_token)

        time_now = self.clock.time_msec()
+        serialize_options = SerializeEventConfig(requester=requester)

        return {
            "membership": membership,
@@ -383,14 +391,18 @@ class InitialSyncHandler:
            "messages": {
                "chunk": (
                    # Don't bundle aggregations as this is a deprecated API.
-                    self._event_serializer.serialize_events(messages, time_now)
+                    self._event_serializer.serialize_events(
+                        messages, time_now, config=serialize_options
+                    )
                ),
                "start": await start_token.to_string(self.store),
                "end": await end_token.to_string(self.store),
            },
            "state": (
                # Don't bundle aggregations as this is a deprecated API.
-                self._event_serializer.serialize_events(room_state.values(), time_now)
+                self._event_serializer.serialize_events(
+                    room_state.values(), time_now, config=serialize_options
+                )
            ),
            "presence": [],
            "receipts": [],
@@ -398,7 +410,7 @@ class InitialSyncHandler:

    async def _room_initial_sync_joined(
        self,
-        user_id: str,
+        requester: Requester,
        room_id: str,
        pagin_config: PaginationConfig,
        membership: str,
@@ -410,9 +422,12 @@ class InitialSyncHandler:

        # TODO: These concurrently
        time_now = self.clock.time_msec()
+        serialize_options = SerializeEventConfig(requester=requester)
        # Don't bundle aggregations as this is a deprecated API.
        state = self._event_serializer.serialize_events(
-            current_state.values(), time_now
+            current_state.values(),
+            time_now,
+            config=serialize_options,
        )

        now_token = self.hs.get_event_sources().get_current_token()
@@ -450,7 +465,10 @@ class InitialSyncHandler:
            if not receipts:
                return []

-            return ReceiptEventSource.filter_out_private_receipts(receipts, user_id)
+            return ReceiptEventSource.filter_out_private_receipts(
+                receipts,
+                requester.user.to_string(),
+            )

        presence, receipts, (messages, token) = await make_deferred_yieldable(
            gather_results(
@@ -469,20 +487,23 @@ class InitialSyncHandler:
        )

        messages = await filter_events_for_client(
-            self._storage_controllers, user_id, messages, is_peeking=is_peeking
+            self._storage_controllers,
+            requester.user.to_string(),
+            messages,
+            is_peeking=is_peeking,
        )

        start_token = now_token.copy_and_replace(StreamKeyType.ROOM, token)
        end_token = now_token

-        time_now = self.clock.time_msec()
-
        ret = {
            "room_id": room_id,
            "messages": {
                "chunk": (
                    # Don't bundle aggregations as this is a deprecated API.
-                    self._event_serializer.serialize_events(messages, time_now)
+                    self._event_serializer.serialize_events(
+                        messages, time_now, config=serialize_options
+                    )
                ),
                "start": await start_token.to_string(self.store),
                "end": await end_token.to_string(self.store),
@@ -50,7 +50,7 @@ from synapse.event_auth import validate_event_for_room_version
 from synapse.events import EventBase, relation_from_event
 from synapse.events.builder import EventBuilder
 from synapse.events.snapshot import EventContext, UnpersistedEventContextBase
-from synapse.events.utils import maybe_upsert_event_field
+from synapse.events.utils import SerializeEventConfig, maybe_upsert_event_field
 from synapse.events.validator import EventValidator
 from synapse.handlers.directory import DirectoryHandler
 from synapse.logging import opentracing
@@ -245,8 +245,11 @@ class MessageHandler:
                )
                room_state = room_state_events[membership_event_id]

-        now = self.clock.time_msec()
-        events = self._event_serializer.serialize_events(room_state.values(), now)
+        events = self._event_serializer.serialize_events(
+            room_state.values(),
+            self.clock.time_msec(),
+            config=SerializeEventConfig(requester=requester),
+        )
        return events

    async def _user_can_see_state_at_event(
@@ -579,7 +579,9 @@ class PaginationHandler:

        time_now = self.clock.time_msec()

-        serialize_options = SerializeEventConfig(as_client_event=as_client_event)
+        serialize_options = SerializeEventConfig(
+            as_client_event=as_client_event, requester=requester
+        )

        chunk = {
            "chunk": (
@@ -20,6 +20,7 @@ import attr
 from synapse.api.constants import Direction, EventTypes, RelationTypes
 from synapse.api.errors import SynapseError
 from synapse.events import EventBase, relation_from_event
+from synapse.events.utils import SerializeEventConfig
 from synapse.logging.context import make_deferred_yieldable, run_in_background
 from synapse.logging.opentracing import trace
 from synapse.storage.databases.main.relations import ThreadsNextBatch, _RelatedEvent
@@ -151,16 +152,23 @@ class RelationsHandler:
        )

        now = self._clock.time_msec()
+        serialize_options = SerializeEventConfig(requester=requester)
        return_value: JsonDict = {
            "chunk": self._event_serializer.serialize_events(
-                events, now, bundle_aggregations=aggregations
+                events,
+                now,
+                bundle_aggregations=aggregations,
+                config=serialize_options,
            ),
        }
        if include_original_event:
            # Do not bundle aggregations when retrieving the original event because
            # we want the content before relations are applied to it.
            return_value["original_event"] = self._event_serializer.serialize_event(
-                event, now, bundle_aggregations=None
+                event,
+                now,
+                bundle_aggregations=None,
+                config=serialize_options,
            )

        if next_token:
@@ -1123,7 +1123,7 @@ class RoomCreationHandler:
                event_dict,
                prev_event_ids=prev_event,
                depth=depth,
-                state_map=dict(state_map),
+                state_map=state_map,
                for_batch=for_batch,
            )

@@ -207,6 +207,7 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):
    @abc.abstractmethod
    async def remote_knock(
        self,
+        requester: Requester,
        remote_room_hosts: List[str],
        room_id: str,
        user: UserID,
@@ -1073,7 +1074,7 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):
                    )

                return await self.remote_knock(
-                    remote_room_hosts, room_id, target, content
+                    requester, remote_room_hosts, room_id, target, content
                )

        return await self._local_membership_update(
@@ -1984,6 +1985,7 @@ class RoomMemberMasterHandler(RoomMemberHandler):

    async def remote_knock(
        self,
+        requester: Requester,
        remote_room_hosts: List[str],
        room_id: str,
        user: UserID,
@@ -113,6 +113,7 @@ class RoomMemberWorkerHandler(RoomMemberHandler):

    async def remote_knock(
        self,
+        requester: Requester,
        remote_room_hosts: List[str],
        room_id: str,
        user: UserID,
@@ -123,9 +124,10 @@ class RoomMemberWorkerHandler(RoomMemberHandler):
        Implements RoomMemberHandler.remote_knock
        """
        ret = await self._remote_knock_client(
+            requester=requester,
            remote_room_hosts=remote_room_hosts,
            room_id=room_id,
-            user=user,
+            user_id=user.to_string(),
            content=content,
        )
        return ret["event_id"], ret["stream_id"]
@@ -23,7 +23,8 @@ from synapse.api.constants import EventTypes, Membership
 from synapse.api.errors import NotFoundError, SynapseError
 from synapse.api.filtering import Filter
 from synapse.events import EventBase
-from synapse.types import JsonDict, StrCollection, StreamKeyType, UserID
+from synapse.events.utils import SerializeEventConfig
+from synapse.types import JsonDict, Requester, StrCollection, StreamKeyType, UserID
 from synapse.types.state import StateFilter
 from synapse.visibility import filter_events_for_client

@@ -109,12 +110,12 @@ class SearchHandler:
        return historical_room_ids

    async def search(
-        self, user: UserID, content: JsonDict, batch: Optional[str] = None
+        self, requester: Requester, content: JsonDict, batch: Optional[str] = None
    ) -> JsonDict:
        """Performs a full text search for a user.

        Args:
-            user: The user performing the search.
+            requester: The user performing the search.
            content: Search parameters
            batch: The next_batch parameter. Used for pagination.

@@ -199,7 +200,7 @@ class SearchHandler:
            )

        return await self._search(
-            user,
+            requester,
            batch_group,
            batch_group_key,
            batch_token,
@@ -217,7 +218,7 @@ class SearchHandler:

    async def _search(
        self,
-        user: UserID,
+        requester: Requester,
        batch_group: Optional[str],
        batch_group_key: Optional[str],
        batch_token: Optional[str],
@@ -235,7 +236,7 @@ class SearchHandler:
        """Performs a full text search for a user.

        Args:
-            user: The user performing the search.
+            requester: The user performing the search.
            batch_group: Pagination information.
            batch_group_key: Pagination information.
            batch_token: Pagination information.
@@ -269,7 +270,7 @@ class SearchHandler:

        # TODO: Search through left rooms too
        rooms = await self.store.get_rooms_for_local_user_where_membership_is(
-            user.to_string(),
+            requester.user.to_string(),
            membership_list=[Membership.JOIN],
            # membership_list=[Membership.JOIN, Membership.LEAVE, Membership.Ban],
        )
@@ -303,13 +304,13 @@ class SearchHandler:

        if order_by == "rank":
            search_result, sender_group = await self._search_by_rank(
-                user, room_ids, search_term, keys, search_filter
+                requester.user, room_ids, search_term, keys, search_filter
            )
            # Unused return values for rank search.
            global_next_batch = None
        elif order_by == "recent":
            search_result, global_next_batch = await self._search_by_recent(
-                user,
+                requester.user,
                room_ids,
                search_term,
                keys,
@@ -334,7 +335,7 @@ class SearchHandler:
            assert after_limit is not None

            contexts = await self._calculate_event_contexts(
-                user,
+                requester.user,
                search_result.allowed_events,
                before_limit,
                after_limit,
@@ -363,27 +364,37 @@ class SearchHandler:
                # The returned events.
                search_result.allowed_events,
            ),
-            user.to_string(),
+            requester.user.to_string(),
        )

        # We're now about to serialize the events. We should not make any
        # blocking calls after this. Otherwise, the 'age' will be wrong.

        time_now = self.clock.time_msec()
+        serialize_options = SerializeEventConfig(requester=requester)

        for context in contexts.values():
            context["events_before"] = self._event_serializer.serialize_events(
-                context["events_before"], time_now, bundle_aggregations=aggregations
+                context["events_before"],
+                time_now,
+                bundle_aggregations=aggregations,
+                config=serialize_options,
            )
            context["events_after"] = self._event_serializer.serialize_events(
-                context["events_after"], time_now, bundle_aggregations=aggregations
+                context["events_after"],
+                time_now,
+                bundle_aggregations=aggregations,
+                config=serialize_options,
            )

        results = [
            {
                "rank": search_result.rank_map[e.event_id],
                "result": self._event_serializer.serialize_event(
-                    e, time_now, bundle_aggregations=aggregations
+                    e,
+                    time_now,
+                    bundle_aggregations=aggregations,
+                    config=serialize_options,
                ),
                "context": contexts.get(e.event_id, {}),
            }
@@ -398,7 +409,9 @@ class SearchHandler:

        if state_results:
            rooms_cat_res["state"] = {
-                room_id: self._event_serializer.serialize_events(state_events, time_now)
+                room_id: self._event_serializer.serialize_events(
+                    state_events, time_now, config=serialize_options
+                )
                for room_id, state_events in state_results.items()
            }

@@ -23,7 +23,6 @@ from typing import (
    Mapping,
    Optional,
    Sequence,
-    Set,
    Tuple,
    Union,
 )
@@ -396,18 +395,10 @@ class BulkPushRuleEvaluator:
                        del notification_levels[key]

        # Pull out any user and room mentions.
-        mentions = event.content.get(EventContentFields.MSC3952_MENTIONS)
-        has_mentions = self._intentional_mentions_enabled and isinstance(mentions, dict)
-        user_mentions: Set[str] = set()
-        if has_mentions:
-            # mypy seems to have lost the type even though it must be a dict here.
-            assert isinstance(mentions, dict)
-            # Remove out any non-string items and convert to a set.
-            user_mentions_raw = mentions.get("user_ids")
-            if isinstance(user_mentions_raw, list):
-                user_mentions = set(
-                    filter(lambda item: isinstance(item, str), user_mentions_raw)
-                )
+        has_mentions = (
+            self._intentional_mentions_enabled
+            and EventContentFields.MSC3952_MENTIONS in event.content
+        )

        evaluator = PushRuleEvaluator(
            _flatten_dict(
@@ -415,7 +406,6 @@ class BulkPushRuleEvaluator:
                msc3873_escape_event_match_key=self.hs.config.experimental.msc3873_escape_event_match_key,
            ),
            has_mentions,
-            user_mentions,
            room_member_count,
            sender_power_level,
            notification_levels,
@@ -423,7 +413,6 @@ class BulkPushRuleEvaluator:
            self._related_event_match_enabled,
            event.room_version.msc3931_push_features,
            self.hs.config.experimental.msc1767_enabled,  # MSC3931 flag
-            self.hs.config.experimental.msc3758_exact_event_match,
            self.hs.config.experimental.msc3966_exact_event_property_contains,
        )

@@ -41,11 +41,12 @@ def format_push_rules_for_user(

        rulearray.append(template_rule)

-        pattern_type = template_rule.pop("pattern_type", None)
-        if pattern_type == "user_id":
-            template_rule["pattern"] = user.to_string()
-        elif pattern_type == "user_localpart":
-            template_rule["pattern"] = user.localpart
+        for type_key in ("pattern", "value"):
+            type_value = template_rule.pop(f"{type_key}_type", None)
+            if type_value == "user_id":
+                template_rule[type_key] = user.to_string()
+            elif type_value == "user_localpart":
+                template_rule[type_key] = user.localpart

        template_rule["enabled"] = enabled

@@ -142,17 +142,12 @@ class ReplicationRemoteKnockRestServlet(ReplicationEndpoint):
        }

    async def _handle_request(  # type: ignore[override]
-        self,
-        request: SynapseRequest,
-        content: JsonDict,
-        room_id: str,
-        user_id: str,
+        self, request: SynapseRequest, content: JsonDict, room_id: str, user_id: str
    ) -> Tuple[int, JsonDict]:
        remote_room_hosts = content["remote_room_hosts"]
        event_content = content["content"]

        requester = Requester.deserialize(self.store, content["requester"])
-
        request.requester = requester

        logger.debug("remote_knock: %s on room: %s", user_id, room_id)
@@ -277,16 +272,12 @@ class ReplicationRemoteRescindKnockRestServlet(ReplicationEndpoint):
        }

    async def _handle_request(  # type: ignore[override]
-        self,
-        request: SynapseRequest,
-        content: JsonDict,
-        knock_event_id: str,
+        self, request: SynapseRequest, content: JsonDict, knock_event_id: str
    ) -> Tuple[int, JsonDict]:
        txn_id = content["txn_id"]
        event_content = content["content"]

        requester = Requester.deserialize(self.store, content["requester"])
-
        request.requester = requester

        # hopefully we're now on the master, so this won't recurse!
@@ -363,3 +354,5 @@ def register_servlets(hs: "HomeServer", http_server: HttpServer) -> None:
    ReplicationRemoteJoinRestServlet(hs).register(http_server)
    ReplicationRemoteRejectInviteRestServlet(hs).register(http_server)
    ReplicationUserJoinedLeftRoomRestServlet(hs).register(http_server)
+    ReplicationRemoteKnockRestServlet(hs).register(http_server)
+    ReplicationRemoteRescindKnockRestServlet(hs).register(http_server)
@@ -139,7 +139,7 @@ class ClientRestResource(JsonResource):
        relations.register_servlets(hs, client_resource)
        if is_main_process:
            password_policy.register_servlets(hs, client_resource)
-            knock.register_servlets(hs, client_resource)
+        knock.register_servlets(hs, client_resource)

        # moving to /_synapse/admin
        if is_main_process:
@@ -255,7 +255,7 @@ class DehydratedDeviceServlet(RestServlet):

    """

-    PATTERNS = client_patterns("/org.matrix.msc2697.v2/dehydrated_device", releases=())
+    PATTERNS = client_patterns("/org.matrix.msc2697.v2/dehydrated_device$", releases=())

    def __init__(self, hs: "HomeServer"):
        super().__init__()
@@ -17,6 +17,7 @@ import logging
 from typing import TYPE_CHECKING, Dict, List, Tuple, Union

 from synapse.api.errors import SynapseError
+from synapse.events.utils import SerializeEventConfig
 from synapse.http.server import HttpServer
 from synapse.http.servlet import RestServlet, parse_string
 from synapse.http.site import SynapseRequest
@@ -43,9 +44,8 @@ class EventStreamRestServlet(RestServlet):

    async def on_GET(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
        requester = await self.auth.get_user_by_req(request, allow_guest=True)
-        is_guest = requester.is_guest
        args: Dict[bytes, List[bytes]] = request.args  # type: ignore
-        if is_guest:
+        if requester.is_guest:
            if b"room_id" not in args:
                raise SynapseError(400, "Guest users must specify room_id param")
        room_id = parse_string(request, "room_id")
@@ -63,13 +63,12 @@ class EventStreamRestServlet(RestServlet):
        as_client_event = b"raw" not in args

        chunk = await self.event_stream_handler.get_stream(
-            requester.user.to_string(),
+            requester,
            pagin_config,
            timeout=timeout,
            as_client_event=as_client_event,
-            affect_presence=(not is_guest),
+            affect_presence=(not requester.is_guest),
            room_id=room_id,
-            is_guest=is_guest,
        )

        return 200, chunk
@@ -91,9 +90,12 @@ class EventRestServlet(RestServlet):
        requester = await self.auth.get_user_by_req(request)
        event = await self.event_handler.get_event(requester.user, None, event_id)

-        time_now = self.clock.time_msec()
        if event:
-            result = self._event_serializer.serialize_event(event, time_now)
+            result = self._event_serializer.serialize_event(
+                event,
+                self.clock.time_msec(),
+                config=SerializeEventConfig(requester=requester),
+            )
            return 200, result
        else:
            return 404, "Event not found."
@@ -312,15 +312,29 @@ class SigningKeyUploadServlet(RestServlet):
        user_id = requester.user.to_string()
        body = parse_json_object_from_request(request)

-        await self.auth_handler.validate_user_via_ui_auth(
-            requester,
-            request,
-            body,
-            "add a device signing key to your account",
-            # Allow skipping of UI auth since this is frequently called directly
-            # after login and it is silly to ask users to re-auth immediately.
-            can_skip_ui_auth=True,
-        )
+        if self.hs.config.experimental.msc3967_enabled:
+            if await self.e2e_keys_handler.is_cross_signing_set_up_for_user(user_id):
+                # If we already have a master key then cross signing is set up and we require UIA to reset
+                await self.auth_handler.validate_user_via_ui_auth(
+                    requester,
+                    request,
+                    body,
+                    "reset the device signing key on your account",
+                    # Do not allow skipping of UIA auth.
+                    can_skip_ui_auth=False,
+                )
+            # Otherwise we don't require UIA since we are setting up cross signing for first time
+        else:
+            # Previous behaviour is to always require UIA but allow it to be skipped
+            await self.auth_handler.validate_user_via_ui_auth(
+                requester,
+                request,
+                body,
+                "add a device signing key to your account",
+                # Allow skipping of UI auth since this is frequently called directly
+                # after login and it is silly to ask users to re-auth immediately.
+                can_skip_ui_auth=True,
+            )

        result = await self.e2e_keys_handler.upload_signing_keys_for_user(user_id, body)
        return 200, result
@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import logging
-from typing import TYPE_CHECKING, Awaitable, Dict, List, Optional, Tuple
+from typing import TYPE_CHECKING, Dict, List, Tuple

 from synapse.api.constants import Membership
 from synapse.api.errors import SynapseError
@@ -24,8 +24,6 @@ from synapse.http.servlet import (
    parse_strings_from_args,
 )
 from synapse.http.site import SynapseRequest
-from synapse.logging.opentracing import set_tag
-from synapse.rest.client.transactions import HttpTransactionCache
 from synapse.types import JsonDict, RoomAlias, RoomID

 if TYPE_CHECKING:
@@ -45,7 +43,6 @@ class KnockRoomAliasServlet(RestServlet):

    def __init__(self, hs: "HomeServer"):
        super().__init__()
-        self.txns = HttpTransactionCache(hs)
        self.room_member_handler = hs.get_room_member_handler()
        self.auth = hs.get_auth()

@@ -53,7 +50,6 @@ class KnockRoomAliasServlet(RestServlet):
        self,
        request: SynapseRequest,
        room_identifier: str,
-        txn_id: Optional[str] = None,
    ) -> Tuple[int, JsonDict]:
        requester = await self.auth.get_user_by_req(request)

@@ -67,7 +63,6 @@ class KnockRoomAliasServlet(RestServlet):

            # twisted.web.server.Request.args is incorrectly defined as Optional[Any]
            args: Dict[bytes, List[bytes]] = request.args  # type: ignore
-
            remote_room_hosts = parse_strings_from_args(
                args, "server_name", required=False
            )
@@ -86,7 +81,6 @@ class KnockRoomAliasServlet(RestServlet):
            target=requester.user,
            room_id=room_id,
            action=Membership.KNOCK,
-            txn_id=txn_id,
            third_party_signed=None,
            remote_room_hosts=remote_room_hosts,
            content=event_content,
@@ -94,15 +88,6 @@ class KnockRoomAliasServlet(RestServlet):

        return 200, {"room_id": room_id}

-    def on_PUT(
-        self, request: SynapseRequest, room_identifier: str, txn_id: str
-    ) -> Awaitable[Tuple[int, JsonDict]]:
-        set_tag("txn_id", txn_id)
-
-        return self.txns.fetch_or_execute_request(
-            request, self.on_POST, request, room_identifier, txn_id
-        )
-

 def register_servlets(hs: "HomeServer", http_server: HttpServer) -> None:
    KnockRoomAliasServlet(hs).register(http_server)
@@ -72,6 +72,12 @@ class NotificationsServlet(RestServlet):

        next_token = None

+        serialize_options = SerializeEventConfig(
+            event_format=format_event_for_client_v2_without_room_id,
+            requester=requester,
+        )
+        now = self.clock.time_msec()
+
        for pa in push_actions:
            returned_pa = {
                "room_id": pa.room_id,
@@ -81,10 +87,8 @@ class NotificationsServlet(RestServlet):
                "event": (
                    self._event_serializer.serialize_event(
                        notif_events[pa.event_id],
-                        self.clock.time_msec(),
-                        config=SerializeEventConfig(
-                            event_format=format_event_for_client_v2_without_room_id
-                        ),
+                        now,
+                        config=serialize_options,
                    )
                ),
            }
@@ -37,7 +37,7 @@ from synapse.api.errors import (
    UnredactedContentDeletedError,
 )
 from synapse.api.filtering import Filter
-from synapse.events.utils import format_event_for_client_v2
+from synapse.events.utils import SerializeEventConfig, format_event_for_client_v2
 from synapse.http.server import HttpServer
 from synapse.http.servlet import (
    ResolveRoomIdMixin,
@@ -814,11 +814,13 @@ class RoomEventServlet(RestServlet):
                [event], requester.user.to_string()
            )

-            time_now = self.clock.time_msec()
            # per MSC2676, /rooms/{roomId}/event/{eventId}, should return the
            # *original* event, rather than the edited version
            event_dict = self._event_serializer.serialize_event(
-                event, time_now, bundle_aggregations=aggregations, apply_edits=False
+                event,
+                self.clock.time_msec(),
+                bundle_aggregations=aggregations,
+                config=SerializeEventConfig(requester=requester),
            )
            return 200, event_dict

@@ -863,24 +865,30 @@ class RoomEventContextServlet(RestServlet):
            raise SynapseError(404, "Event not found.", errcode=Codes.NOT_FOUND)

        time_now = self.clock.time_msec()
+        serializer_options = SerializeEventConfig(requester=requester)
        results = {
            "events_before": self._event_serializer.serialize_events(
                event_context.events_before,
                time_now,
                bundle_aggregations=event_context.aggregations,
+                config=serializer_options,
            ),
            "event": self._event_serializer.serialize_event(
                event_context.event,
                time_now,
                bundle_aggregations=event_context.aggregations,
+                config=serializer_options,
            ),
            "events_after": self._event_serializer.serialize_events(
                event_context.events_after,
                time_now,
                bundle_aggregations=event_context.aggregations,
+                config=serializer_options,
            ),
            "state": self._event_serializer.serialize_events(
-                event_context.state, time_now
+                event_context.state,
+                time_now,
+                config=serializer_options,
            ),
            "start": event_context.start,
            "end": event_context.end,
@@ -926,7 +934,7 @@ class RoomMembershipRestServlet(TransactionRestServlet):
        self.auth = hs.get_auth()

    def register(self, http_server: HttpServer) -> None:
-        # /rooms/$roomid/[invite|join|leave]
+        # /rooms/$roomid/[join|invite|leave|ban|unban|kick]
        PATTERNS = (
            "/rooms/(?P<room_id>[^/]*)/"
            "(?P<membership_action>join|invite|leave|ban|unban|kick)"
@@ -1192,7 +1200,7 @@ class SearchRestServlet(RestServlet):
        content = parse_json_object_from_request(request)

        batch = parse_string(request, "next_batch")
-        results = await self.search_handler.search(requester.user, content, batch)
+        results = await self.search_handler.search(requester, content, batch)

        return 200, results

@@ -15,9 +15,7 @@
 import logging
 import re
 from http import HTTPStatus
-from typing import TYPE_CHECKING, Awaitable, Tuple
-
-from twisted.web.server import Request
+from typing import TYPE_CHECKING, Tuple

 from synapse.api.constants import EventContentFields
 from synapse.api.errors import AuthError, Codes, SynapseError
@@ -30,7 +28,6 @@ from synapse.http.servlet import (
    parse_strings_from_args,
 )
 from synapse.http.site import SynapseRequest
-from synapse.rest.client.transactions import HttpTransactionCache
 from synapse.types import JsonDict

 if TYPE_CHECKING:
@@ -79,7 +76,6 @@ class RoomBatchSendEventRestServlet(RestServlet):
        self.event_creation_handler = hs.get_event_creation_handler()
        self.auth = hs.get_auth()
        self.room_batch_handler = hs.get_room_batch_handler()
-        self.txns = HttpTransactionCache(hs)

    async def on_POST(
        self, request: SynapseRequest, room_id: str
@@ -249,16 +245,6 @@ class RoomBatchSendEventRestServlet(RestServlet):

        return HTTPStatus.OK, response_dict

-    def on_GET(self, request: Request, room_id: str) -> Tuple[int, str]:
-        return HTTPStatus.NOT_IMPLEMENTED, "Not implemented"
-
-    def on_PUT(
-        self, request: SynapseRequest, room_id: str
-    ) -> Awaitable[Tuple[int, JsonDict]]:
-        return self.txns.fetch_or_execute_request(
-            request, self.on_POST, request, room_id
-        )
-

 def register_servlets(hs: "HomeServer", http_server: HttpServer) -> None:
    msc2716_enabled = hs.config.experimental.msc2716_enabled
@@ -38,7 +38,7 @@ from synapse.http.server import HttpServer
 from synapse.http.servlet import RestServlet, parse_boolean, parse_integer, parse_string
 from synapse.http.site import SynapseRequest
 from synapse.logging.opentracing import trace_with_opname
-from synapse.types import JsonDict, StreamToken
+from synapse.types import JsonDict, Requester, StreamToken
 from synapse.util import json_decoder

 from ._base import client_patterns, set_timeline_upper_limit
@@ -226,7 +226,7 @@ class SyncRestServlet(RestServlet):
        # We know that the the requester has an access token since appservices
        # cannot use sync.
        response_content = await self.encode_response(
-            time_now, sync_result, requester.access_token_id, filter_collection
+            time_now, sync_result, requester, filter_collection
        )

        logger.debug("Event formatting complete")
@@ -237,7 +237,7 @@ class SyncRestServlet(RestServlet):
        self,
        time_now: int,
        sync_result: SyncResult,
-        access_token_id: Optional[int],
+        requester: Requester,
        filter: FilterCollection,
    ) -> JsonDict:
        logger.debug("Formatting events in sync response")
@@ -250,12 +250,12 @@ class SyncRestServlet(RestServlet):

        serialize_options = SerializeEventConfig(
            event_format=event_formatter,
-            token_id=access_token_id,
+            requester=requester,
            only_event_fields=filter.event_fields,
        )
        stripped_serialize_options = SerializeEventConfig(
            event_format=event_formatter,
-            token_id=access_token_id,
+            requester=requester,
            include_stripped_room_state=True,
        )

@@ -743,7 +743,7 @@ class HomeServer(metaclass=abc.ABCMeta):

    @cache_in_self
    def get_event_client_serializer(self) -> EventClientSerializer:
-        return EventClientSerializer(self.config.experimental.msc3925_inhibit_edit)
+        return EventClientSerializer()

    @cache_in_self
    def get_password_policy_handler(self) -> PasswordPolicyHandler:
@@ -672,7 +672,15 @@ class DatabasePool:
            f = cast(types.FunctionType, func)  # type: ignore[redundant-cast]
            if f.__closure__:
                for i, cell in enumerate(f.__closure__):
-                    if inspect.isgenerator(cell.cell_contents):
+                    try:
+                        contents = cell.cell_contents
+                    except ValueError:
+                        # cell.cell_contents can raise if the "cell" is empty,
+                        # which indicates that the variable is currently
+                        # unbound.
+                        continue
+
+                    if inspect.isgenerator(contents):
                        logger.error(
                            "Programming error: function %s references generator %s "
                            "via its closure",
@@ -40,7 +40,6 @@ from synapse.storage.databases.main.push_rule import PushRulesWorkerStore
 from synapse.storage.engines import PostgresEngine
 from synapse.storage.util.id_generators import (
    AbstractStreamIdGenerator,
-    AbstractStreamIdTracker,
    MultiWriterIdGenerator,
    StreamIdGenerator,
 )
@@ -64,14 +63,12 @@ class AccountDataWorkerStore(PushRulesWorkerStore, CacheInvalidationWorkerStore)
    ):
        super().__init__(database, db_conn, hs)

-        # `_can_write_to_account_data` indicates whether the current worker is allowed
-        # to write account data. A value of `True` implies that `_account_data_id_gen`
-        # is an `AbstractStreamIdGenerator` and not just a tracker.
-        self._account_data_id_gen: AbstractStreamIdTracker
        self._can_write_to_account_data = (
            self._instance_name in hs.config.worker.writers.account_data
        )

+        self._account_data_id_gen: AbstractStreamIdGenerator
+
        if isinstance(database.engine, PostgresEngine):
            self._account_data_id_gen = MultiWriterIdGenerator(
                db_conn=db_conn,
@@ -558,7 +555,6 @@ class AccountDataWorkerStore(PushRulesWorkerStore, CacheInvalidationWorkerStore)
            The maximum stream ID.
        """
        assert self._can_write_to_account_data
-        assert isinstance(self._account_data_id_gen, AbstractStreamIdGenerator)

        content_json = json_encoder.encode(content)

@@ -585,7 +581,7 @@ class AccountDataWorkerStore(PushRulesWorkerStore, CacheInvalidationWorkerStore)

    async def remove_account_data_for_room(
        self, user_id: str, room_id: str, account_data_type: str
-    ) -> Optional[int]:
+    ) -> int:
        """Delete the room account data for the user of a given type.

        Args:
@@ -598,7 +594,6 @@ class AccountDataWorkerStore(PushRulesWorkerStore, CacheInvalidationWorkerStore)
            data to delete.
        """
        assert self._can_write_to_account_data
-        assert isinstance(self._account_data_id_gen, AbstractStreamIdGenerator)

        def _remove_account_data_for_room_txn(
            txn: LoggingTransaction, next_id: int
@@ -637,15 +632,13 @@ class AccountDataWorkerStore(PushRulesWorkerStore, CacheInvalidationWorkerStore)
                next_id,
            )

-            if not row_updated:
-                return None
-
-            self._account_data_stream_cache.entity_has_changed(user_id, next_id)
-            self.get_room_account_data_for_user.invalidate((user_id,))
-            self.get_account_data_for_room.invalidate((user_id, room_id))
-            self.get_account_data_for_room_and_type.prefill(
-                (user_id, room_id, account_data_type), {}
-            )
+            if row_updated:
+                self._account_data_stream_cache.entity_has_changed(user_id, next_id)
+                self.get_room_account_data_for_user.invalidate((user_id,))
+                self.get_account_data_for_room.invalidate((user_id, room_id))
+                self.get_account_data_for_room_and_type.prefill(
+                    (user_id, room_id, account_data_type), {}
+                )

        return self._account_data_id_gen.get_current_token()

@@ -663,7 +656,6 @@ class AccountDataWorkerStore(PushRulesWorkerStore, CacheInvalidationWorkerStore)
            The maximum stream ID.
        """
        assert self._can_write_to_account_data
-        assert isinstance(self._account_data_id_gen, AbstractStreamIdGenerator)

        async with self._account_data_id_gen.get_next() as next_id:
            await self.db_pool.runInteraction(
@@ -753,7 +745,7 @@ class AccountDataWorkerStore(PushRulesWorkerStore, CacheInvalidationWorkerStore)
        self,
        user_id: str,
        account_data_type: str,
-    ) -> Optional[int]:
+    ) -> int:
        """
        Delete a single piece of user account data by type.

@@ -770,7 +762,6 @@ class AccountDataWorkerStore(PushRulesWorkerStore, CacheInvalidationWorkerStore)
            to delete.
        """
        assert self._can_write_to_account_data
-        assert isinstance(self._account_data_id_gen, AbstractStreamIdGenerator)

        def _remove_account_data_for_user_txn(
            txn: LoggingTransaction, next_id: int
@@ -840,14 +831,12 @@ class AccountDataWorkerStore(PushRulesWorkerStore, CacheInvalidationWorkerStore)
                next_id,
            )

-            if not row_updated:
-                return None
-
-            self._account_data_stream_cache.entity_has_changed(user_id, next_id)
-            self.get_global_account_data_for_user.invalidate((user_id,))
-            self.get_global_account_data_by_type_for_user.prefill(
-                (user_id, account_data_type), {}
-            )
+            if row_updated:
+                self._account_data_stream_cache.entity_has_changed(user_id, next_id)
+                self.get_global_account_data_for_user.invalidate((user_id,))
+                self.get_global_account_data_by_type_for_user.prefill(
+                    (user_id, account_data_type), {}
+                )

        return self._account_data_id_gen.get_current_token()

@@ -52,7 +52,6 @@ from synapse.storage.databases.main.roommember import RoomMemberWorkerStore
 from synapse.storage.types import Cursor
 from synapse.storage.util.id_generators import (
    AbstractStreamIdGenerator,
-    AbstractStreamIdTracker,
    StreamIdGenerator,
 )
 from synapse.types import JsonDict, StrCollection, get_verify_key_from_cross_signing_key
@@ -91,7 +90,7 @@ class DeviceWorkerStore(RoomMemberWorkerStore, EndToEndKeyWorkerStore):

        # In the worker store this is an ID tracker which we overwrite in the non-worker
        # class below that is used on the main process.
-        self._device_list_id_gen: AbstractStreamIdTracker = StreamIdGenerator(
+        self._device_list_id_gen = StreamIdGenerator(
            db_conn,
            hs.get_replication_notifier(),
            "device_lists_stream",
@@ -712,9 +711,7 @@ class DeviceWorkerStore(RoomMemberWorkerStore, EndToEndKeyWorkerStore):
            The new stream ID.
        """

-        # TODO: this looks like it's _writing_. Should this be on DeviceStore rather
-        #  than DeviceWorkerStore?
-        async with self._device_list_id_gen.get_next() as stream_id:  # type: ignore[attr-defined]
+        async with self._device_list_id_gen.get_next() as stream_id:
            await self.db_pool.runInteraction(
                "add_user_sig_change_to_streams",
                self._add_user_signature_change_txn,
@@ -244,9 +244,7 @@ class EndToEndKeyWorkerStore(EndToEndKeyBackgroundStore, CacheInvalidationWorker
        set_tag("include_all_devices", include_all_devices)
        set_tag("include_deleted_devices", include_deleted_devices)

-        result = await self.db_pool.runInteraction(
-            "get_e2e_device_keys",
-            self._get_e2e_device_keys_txn,
+        result = await self._get_e2e_device_keys(
            query_list,
            include_all_devices,
            include_deleted_devices,
@@ -285,9 +283,8 @@ class EndToEndKeyWorkerStore(EndToEndKeyBackgroundStore, CacheInvalidationWorker
        log_kv(result)
        return result

-    def _get_e2e_device_keys_txn(
+    async def _get_e2e_device_keys(
        self,
-        txn: LoggingTransaction,
        query_list: Collection[Tuple[str, Optional[str]]],
        include_all_devices: bool = False,
        include_deleted_devices: bool = False,
@@ -319,7 +316,7 @@ class EndToEndKeyWorkerStore(EndToEndKeyBackgroundStore, CacheInvalidationWorker

        if user_list:
            user_id_in_list_clause, user_args = make_in_list_sql_clause(
-                txn.database_engine, "user_id", user_list
+                self.database_engine, "user_id", user_list
            )
            query_clauses.append(user_id_in_list_clause)
            query_params_list.append(user_args)
@@ -332,13 +329,16 @@ class EndToEndKeyWorkerStore(EndToEndKeyBackgroundStore, CacheInvalidationWorker
                    user_device_id_in_list_clause,
                    user_device_args,
                ) = make_tuple_in_list_sql_clause(
-                    txn.database_engine, ("user_id", "device_id"), user_device_batch
+                    self.database_engine, ("user_id", "device_id"), user_device_batch
                )
                query_clauses.append(user_device_id_in_list_clause)
                query_params_list.append(user_device_args)

        result: Dict[str, Dict[str, Optional[DeviceKeyLookupResult]]] = {}
-        for query_clause, query_params in zip(query_clauses, query_params_list):
+
+        def get_e2e_device_keys_txn(
+            txn: LoggingTransaction, query_clause: str, query_params: list
+        ) -> None:
            sql = (
                "SELECT user_id, device_id, "
                "    d.display_name, "
@@ -361,6 +361,14 @@ class EndToEndKeyWorkerStore(EndToEndKeyBackgroundStore, CacheInvalidationWorker
                    display_name, db_to_json(key_json) if key_json else None
                )

+        for query_clause, query_params in zip(query_clauses, query_params_list):
+            await self.db_pool.runInteraction(
+                "_get_e2e_device_keys",
+                get_e2e_device_keys_txn,
+                query_clause,
+                query_params,
+            )
+
        if include_deleted_devices:
            for user_id, device_id in deleted_devices:
                if device_id is None:
@@ -72,7 +72,6 @@ from synapse.storage.engines import PostgresEngine
 from synapse.storage.types import Cursor
 from synapse.storage.util.id_generators import (
    AbstractStreamIdGenerator,
-    AbstractStreamIdTracker,
    MultiWriterIdGenerator,
    StreamIdGenerator,
 )
@@ -187,8 +186,8 @@ class EventsWorkerStore(SQLBaseStore):
    ):
        super().__init__(database, db_conn, hs)

-        self._stream_id_gen: AbstractStreamIdTracker
-        self._backfill_id_gen: AbstractStreamIdTracker
+        self._stream_id_gen: AbstractStreamIdGenerator
+        self._backfill_id_gen: AbstractStreamIdGenerator
        if isinstance(database.engine, PostgresEngine):
            # If we're using Postgres than we can use `MultiWriterIdGenerator`
            # regardless of whether this process writes to the streams or not.
@@ -46,7 +46,6 @@ from synapse.storage.engines import PostgresEngine, Sqlite3Engine
 from synapse.storage.push_rule import InconsistentRuleException, RuleNotFoundException
 from synapse.storage.util.id_generators import (
    AbstractStreamIdGenerator,
-    AbstractStreamIdTracker,
    IdGenerator,
    StreamIdGenerator,
 )
@@ -118,7 +117,7 @@ class PushRulesWorkerStore(

        # In the worker store this is an ID tracker which we overwrite in the non-worker
        # class below that is used on the main process.
-        self._push_rules_stream_id_gen: AbstractStreamIdTracker = StreamIdGenerator(
+        self._push_rules_stream_id_gen = StreamIdGenerator(
            db_conn,
            hs.get_replication_notifier(),
            "push_rules_stream",
@@ -36,7 +36,6 @@ from synapse.storage.database import (
 )
 from synapse.storage.util.id_generators import (
    AbstractStreamIdGenerator,
-    AbstractStreamIdTracker,
    StreamIdGenerator,
 )
 from synapse.types import JsonDict
@@ -60,7 +59,7 @@ class PusherWorkerStore(SQLBaseStore):

        # In the worker store this is an ID tracker which we overwrite in the non-worker
        # class below that is used on the main process.
-        self._pushers_id_gen: AbstractStreamIdTracker = StreamIdGenerator(
+        self._pushers_id_gen = StreamIdGenerator(
            db_conn,
            hs.get_replication_notifier(),
            "pushers",
@@ -39,7 +39,7 @@ from synapse.storage.database import (
 from synapse.storage.engines import PostgresEngine
 from synapse.storage.engines._base import IsolationLevel
 from synapse.storage.util.id_generators import (
-    AbstractStreamIdTracker,
+    AbstractStreamIdGenerator,
    MultiWriterIdGenerator,
    StreamIdGenerator,
 )
@@ -65,7 +65,7 @@ class ReceiptsWorkerStore(SQLBaseStore):

        # In the worker store this is an ID tracker which we overwrite in the non-worker
        # class below that is used on the main process.
-        self._receipts_id_gen: AbstractStreamIdTracker
+        self._receipts_id_gen: AbstractStreamIdGenerator

        if isinstance(database.engine, PostgresEngine):
            self._can_write_to_receipts = (
@@ -768,7 +768,7 @@ class ReceiptsWorkerStore(SQLBaseStore):
                "insert_receipt_conv", self._graph_to_linear, room_id, event_ids
            )

-        async with self._receipts_id_gen.get_next() as stream_id:  # type: ignore[attr-defined]
+        async with self._receipts_id_gen.get_next() as stream_id:
            event_ts = await self.db_pool.runInteraction(
                "insert_linearized_receipt",
                self._insert_linearized_receipt_txn,
@@ -1417,6 +1417,183 @@ class RoomWorkerStore(CacheInvalidationWorkerStore):
            get_un_partial_stated_rooms_from_stream_txn,
        )

+    async def get_event_report(self, report_id: int) -> Optional[Dict[str, Any]]:
+        """Retrieve an event report
+
+        Args:
+            report_id: ID of reported event in database
+        Returns:
+            JSON dict of information from an event report or None if the
+            report does not exist.
+        """
+
+        def _get_event_report_txn(
+            txn: LoggingTransaction, report_id: int
+        ) -> Optional[Dict[str, Any]]:
+            sql = """
+                SELECT
+                    er.id,
+                    er.received_ts,
+                    er.room_id,
+                    er.event_id,
+                    er.user_id,
+                    er.content,
+                    events.sender,
+                    room_stats_state.canonical_alias,
+                    room_stats_state.name,
+                    event_json.json AS event_json
+                FROM event_reports AS er
+                LEFT JOIN events
+                    ON events.event_id = er.event_id
+                JOIN event_json
+                    ON event_json.event_id = er.event_id
+                JOIN room_stats_state
+                    ON room_stats_state.room_id = er.room_id
+                WHERE er.id = ?
+            """
+
+            txn.execute(sql, [report_id])
+            row = txn.fetchone()
+
+            if not row:
+                return None
+
+            event_report = {
+                "id": row[0],
+                "received_ts": row[1],
+                "room_id": row[2],
+                "event_id": row[3],
+                "user_id": row[4],
+                "score": db_to_json(row[5]).get("score"),
+                "reason": db_to_json(row[5]).get("reason"),
+                "sender": row[6],
+                "canonical_alias": row[7],
+                "name": row[8],
+                "event_json": db_to_json(row[9]),
+            }
+
+            return event_report
+
+        return await self.db_pool.runInteraction(
+            "get_event_report", _get_event_report_txn, report_id
+        )
+
+    async def get_event_reports_paginate(
+        self,
+        start: int,
+        limit: int,
+        direction: Direction = Direction.BACKWARDS,
+        user_id: Optional[str] = None,
+        room_id: Optional[str] = None,
+    ) -> Tuple[List[Dict[str, Any]], int]:
+        """Retrieve a paginated list of event reports
+
+        Args:
+            start: event offset to begin the query from
+            limit: number of rows to retrieve
+            direction: Whether to fetch the most recent first (backwards) or the
+                oldest first (forwards)
+            user_id: search for user_id. Ignored if user_id is None
+            room_id: search for room_id. Ignored if room_id is None
+        Returns:
+            Tuple of:
+                json list of event reports
+                total number of event reports matching the filter criteria
+        """
+
+        def _get_event_reports_paginate_txn(
+            txn: LoggingTransaction,
+        ) -> Tuple[List[Dict[str, Any]], int]:
+            filters = []
+            args: List[object] = []
+
+            if user_id:
+                filters.append("er.user_id LIKE ?")
+                args.extend(["%" + user_id + "%"])
+            if room_id:
+                filters.append("er.room_id LIKE ?")
+                args.extend(["%" + room_id + "%"])
+
+            if direction == Direction.BACKWARDS:
+                order = "DESC"
+            else:
+                order = "ASC"
+
+            where_clause = "WHERE " + " AND ".join(filters) if len(filters) > 0 else ""
+
+            # We join on room_stats_state despite not using any columns from it
+            # because the join can influence the number of rows returned;
+            # e.g. a room that doesn't have state, maybe because it was deleted.
+            # The query returning the total count should be consistent with
+            # the query returning the results.
+            sql = """
+                SELECT COUNT(*) as total_event_reports
+                FROM event_reports AS er
+                JOIN room_stats_state ON room_stats_state.room_id = er.room_id
+                {}
+                """.format(
+                where_clause
+            )
+            txn.execute(sql, args)
+            count = cast(Tuple[int], txn.fetchone())[0]
+
+            sql = """
+                SELECT
+                    er.id,
+                    er.received_ts,
+                    er.room_id,
+                    er.event_id,
+                    er.user_id,
+                    er.content,
+                    events.sender,
+                    room_stats_state.canonical_alias,
+                    room_stats_state.name
+                FROM event_reports AS er
+                LEFT JOIN events
+                    ON events.event_id = er.event_id
+                JOIN room_stats_state
+                    ON room_stats_state.room_id = er.room_id
+                {where_clause}
+                ORDER BY er.received_ts {order}
+                LIMIT ?
+                OFFSET ?
+            """.format(
+                where_clause=where_clause,
+                order=order,
+            )
+
+            args += [limit, start]
+            txn.execute(sql, args)
+
+            event_reports = []
+            for row in txn:
+                try:
+                    s = db_to_json(row[5]).get("score")
+                    r = db_to_json(row[5]).get("reason")
+                except Exception:
+                    logger.error("Unable to parse json from event_reports: %s", row[0])
+                    continue
+                event_reports.append(
+                    {
+                        "id": row[0],
+                        "received_ts": row[1],
+                        "room_id": row[2],
+                        "event_id": row[3],
+                        "user_id": row[4],
+                        "score": s,
+                        "reason": r,
+                        "sender": row[6],
+                        "canonical_alias": row[7],
+                        "name": row[8],
+                    }
+                )
+
+            return event_reports, count
+
+        return await self.db_pool.runInteraction(
+            "get_event_reports_paginate", _get_event_reports_paginate_txn
+        )
+
    async def delete_event_report(self, report_id: int) -> bool:
        """Remove an event report from database.

@@ -2189,183 +2366,6 @@ class RoomStore(RoomBackgroundUpdateStore, RoomWorkerStore):
        )
        return next_id

-    async def get_event_report(self, report_id: int) -> Optional[Dict[str, Any]]:
-        """Retrieve an event report
-
-        Args:
-            report_id: ID of reported event in database
-        Returns:
-            JSON dict of information from an event report or None if the
-            report does not exist.
-        """
-
-        def _get_event_report_txn(
-            txn: LoggingTransaction, report_id: int
-        ) -> Optional[Dict[str, Any]]:
-            sql = """
-                SELECT
-                    er.id,
-                    er.received_ts,
-                    er.room_id,
-                    er.event_id,
-                    er.user_id,
-                    er.content,
-                    events.sender,
-                    room_stats_state.canonical_alias,
-                    room_stats_state.name,
-                    event_json.json AS event_json
-                FROM event_reports AS er
-                LEFT JOIN events
-                    ON events.event_id = er.event_id
-                JOIN event_json
-                    ON event_json.event_id = er.event_id
-                JOIN room_stats_state
-                    ON room_stats_state.room_id = er.room_id
-                WHERE er.id = ?
-            """
-
-            txn.execute(sql, [report_id])
-            row = txn.fetchone()
-
-            if not row:
-                return None
-
-            event_report = {
-                "id": row[0],
-                "received_ts": row[1],
-                "room_id": row[2],
-                "event_id": row[3],
-                "user_id": row[4],
-                "score": db_to_json(row[5]).get("score"),
-                "reason": db_to_json(row[5]).get("reason"),
-                "sender": row[6],
-                "canonical_alias": row[7],
-                "name": row[8],
-                "event_json": db_to_json(row[9]),
-            }
-
-            return event_report
-
-        return await self.db_pool.runInteraction(
-            "get_event_report", _get_event_report_txn, report_id
-        )
-
-    async def get_event_reports_paginate(
-        self,
-        start: int,
-        limit: int,
-        direction: Direction = Direction.BACKWARDS,
-        user_id: Optional[str] = None,
-        room_id: Optional[str] = None,
-    ) -> Tuple[List[Dict[str, Any]], int]:
-        """Retrieve a paginated list of event reports
-
-        Args:
-            start: event offset to begin the query from
-            limit: number of rows to retrieve
-            direction: Whether to fetch the most recent first (backwards) or the
-                oldest first (forwards)
-            user_id: search for user_id. Ignored if user_id is None
-            room_id: search for room_id. Ignored if room_id is None
-        Returns:
-            Tuple of:
-                json list of event reports
-                total number of event reports matching the filter criteria
-        """
-
-        def _get_event_reports_paginate_txn(
-            txn: LoggingTransaction,
-        ) -> Tuple[List[Dict[str, Any]], int]:
-            filters = []
-            args: List[object] = []
-
-            if user_id:
-                filters.append("er.user_id LIKE ?")
-                args.extend(["%" + user_id + "%"])
-            if room_id:
-                filters.append("er.room_id LIKE ?")
-                args.extend(["%" + room_id + "%"])
-
-            if direction == Direction.BACKWARDS:
-                order = "DESC"
-            else:
-                order = "ASC"
-
-            where_clause = "WHERE " + " AND ".join(filters) if len(filters) > 0 else ""
-
-            # We join on room_stats_state despite not using any columns from it
-            # because the join can influence the number of rows returned;
-            # e.g. a room that doesn't have state, maybe because it was deleted.
-            # The query returning the total count should be consistent with
-            # the query returning the results.
-            sql = """
-                SELECT COUNT(*) as total_event_reports
-                FROM event_reports AS er
-                JOIN room_stats_state ON room_stats_state.room_id = er.room_id
-                {}
-                """.format(
-                where_clause
-            )
-            txn.execute(sql, args)
-            count = cast(Tuple[int], txn.fetchone())[0]
-
-            sql = """
-                SELECT
-                    er.id,
-                    er.received_ts,
-                    er.room_id,
-                    er.event_id,
-                    er.user_id,
-                    er.content,
-                    events.sender,
-                    room_stats_state.canonical_alias,
-                    room_stats_state.name
-                FROM event_reports AS er
-                LEFT JOIN events
-                    ON events.event_id = er.event_id
-                JOIN room_stats_state
-                    ON room_stats_state.room_id = er.room_id
-                {where_clause}
-                ORDER BY er.received_ts {order}
-                LIMIT ?
-                OFFSET ?
-            """.format(
-                where_clause=where_clause,
-                order=order,
-            )
-
-            args += [limit, start]
-            txn.execute(sql, args)
-
-            event_reports = []
-            for row in txn:
-                try:
-                    s = db_to_json(row[5]).get("score")
-                    r = db_to_json(row[5]).get("reason")
-                except Exception:
-                    logger.error("Unable to parse json from event_reports: %s", row[0])
-                    continue
-                event_reports.append(
-                    {
-                        "id": row[0],
-                        "received_ts": row[1],
-                        "room_id": row[2],
-                        "event_id": row[3],
-                        "user_id": row[4],
-                        "score": s,
-                        "reason": r,
-                        "sender": row[6],
-                        "canonical_alias": row[7],
-                        "name": row[8],
-                    }
-                )
-
-            return event_reports, count
-
-        return await self.db_pool.runInteraction(
-            "get_event_reports_paginate", _get_event_reports_paginate_txn
-        )
-
    async def block_room(self, room_id: str, user_id: str) -> None:
        """Marks the room as blocked.

@@ -93,8 +93,11 @@ def _load_current_id(
    return res


-class AbstractStreamIdTracker(metaclass=abc.ABCMeta):
-    """Tracks the "current" stream ID of a stream that may have multiple writers.
+class AbstractStreamIdGenerator(metaclass=abc.ABCMeta):
+    """Generates or tracks stream IDs for a stream that may have multiple writers.
+
+    Each stream ID represents a write transaction, whose completion is tracked
+    so that the "current" stream ID of the stream can be determined.

    Stream IDs are monotonically increasing or decreasing integers representing write
    transactions. The "current" stream ID is the stream ID such that all transactions
@@ -130,16 +133,6 @@ class AbstractStreamIdTracker(metaclass=abc.ABCMeta):
        """
        raise NotImplementedError()

-
-class AbstractStreamIdGenerator(AbstractStreamIdTracker):
-    """Generates stream IDs for a stream that may have multiple writers.
-
-    Each stream ID represents a write transaction, whose completion is tracked
-    so that the "current" stream ID of the stream can be determined.
-
-    See `AbstractStreamIdTracker` for more details.
-    """
-
    @abc.abstractmethod
    def get_next(self) -> AsyncContextManager[int]:
        """
@@ -158,6 +151,15 @@ class AbstractStreamIdGenerator(AbstractStreamIdTracker):
        """
        raise NotImplementedError()

+    @abc.abstractmethod
+    def get_next_txn(self, txn: LoggingTransaction) -> int:
+        """
+        Usage:
+            stream_id_gen.get_next_txn(txn)
+            # ... persist events ...
+        """
+        raise NotImplementedError()
+

 class StreamIdGenerator(AbstractStreamIdGenerator):
    """Generates and tracks stream IDs for a stream with a single writer.
@@ -263,6 +265,40 @@ class StreamIdGenerator(AbstractStreamIdGenerator):

        return _AsyncCtxManagerWrapper(manager())

+    def get_next_txn(self, txn: LoggingTransaction) -> int:
+        """
+        Retrieve the next stream ID from within a database transaction.
+
+        Clean-up functions will be called when the transaction finishes.
+
+        Args:
+            txn: The database transaction object.
+
+        Returns:
+            The next stream ID.
+        """
+        if not self._is_writer:
+            raise Exception("Tried to allocate stream ID on non-writer")
+
+        # Get the next stream ID.
+        with self._lock:
+            self._current += self._step
+            next_id = self._current
+
+            self._unfinished_ids[next_id] = next_id
+
+        def clear_unfinished_id(id_to_clear: int) -> None:
+            """A function to mark processing this ID as finished"""
+            with self._lock:
+                self._unfinished_ids.pop(id_to_clear)
+
+        # Mark this ID as finished once the database transaction itself finishes.
+        txn.call_after(clear_unfinished_id, next_id)
+        txn.call_on_exception(clear_unfinished_id, next_id)
+
+        # Return the new ID.
+        return next_id
+
    def get_current_token(self) -> int:
        if not self._is_writer:
            return self._current
@@ -568,7 +604,7 @@ class MultiWriterIdGenerator(AbstractStreamIdGenerator):
        """
        Usage:

-            stream_id = stream_id_gen.get_next(txn)
+            stream_id = stream_id_gen.get_next_txn(txn)
            # ... persist event ...
        """

@@ -205,7 +205,7 @@ class LocalSequenceGenerator(SequenceGenerator):
        """
        Args:
            get_first_callback: a callback which is called on the first call to
-                 get_next_id_txn; should return the curreent maximum id
+                 get_next_id_txn; should return the current maximum id
        """
        # the callback. this is cleared after it is called, so that it can be GCed.
        self._callback: Optional[GetFirstCallbackType] = get_first_callback
@@ -14,7 +14,17 @@
 # limitations under the License.
 import logging
 from enum import Enum, auto
-from typing import Collection, Dict, FrozenSet, List, Optional, Tuple
+from typing import (
+    Collection,
+    Dict,
+    FrozenSet,
+    List,
+    Mapping,
+    Optional,
+    Sequence,
+    Set,
+    Tuple,
+)

 import attr
 from typing_extensions import Final
@@ -565,21 +575,32 @@ async def filter_events_for_server(
    storage: StorageControllers,
    target_server_name: str,
    local_server_name: str,
-    events: List[EventBase],
+    events: Sequence[EventBase],
    redact: bool = True,
-    check_history_visibility_only: bool = False,
+    filter_out_erased_senders: bool = True,
 ) -> List[EventBase]:
-    """Filter a list of events based on whether given server is allowed to
+    """Filter a list of events based on whether the target server is allowed to
    see them.

+    For a fully stated room, the target server is allowed to see an event E if:
+      - the state at E has world readable or shared history vis, OR
+      - the state at E says that the target server is in the room.
+
+    For a partially stated room, the target server is allowed to see E if:
+      - E was created by this homeserver, AND:
+          - the partial state at E has world readable or shared history vis, OR
+          - the partial state at E says that the target server is in the room.
+
+    TODO: state before or state after?
+
    Args:
        storage
-        server_name
+        target_server_name
+        local_server_name
        events
-        redact: Whether to return a redacted version of the event, or
-            to filter them out entirely.
-        check_history_visibility_only: Whether to only check the
-            history visibility, rather than things like if the sender has been
+        redact: Controls what to do with events which have been filtered out.
+            If True, include their redacted forms; if False, omit them entirely.
+        filter_out_erased_senders: If true, also filter out events whose sender has been
            erased. This is used e.g. during pagination to decide whether to
            backfill or not.

@@ -587,7 +608,7 @@ async def filter_events_for_server(
        The filtered events.
    """

-    def is_sender_erased(event: EventBase, erased_senders: Dict[str, bool]) -> bool:
+    def is_sender_erased(event: EventBase, erased_senders: Mapping[str, bool]) -> bool:
        if erased_senders and erased_senders[event.sender]:
            logger.info("Sender of %s has been erased, redacting", event.event_id)
            return True
@@ -616,7 +637,7 @@ async def filter_events_for_server(
        # server has no users in the room: redact
        return False

-    if not check_history_visibility_only:
+    if filter_out_erased_senders:
        erased_senders = await storage.main.are_users_erased(e.sender for e in events)
    else:
        # We don't want to check whether users are erased, which is equivalent
@@ -631,44 +652,49 @@ async def filter_events_for_server(
    # otherwise a room could be fully joined after we retrieve those, which would then bypass
    # this check but would base the filtering on an outdated view of the membership events.

-    partial_state_invisible_events = set()
-    if not check_history_visibility_only:
-        for e in events:
-            sender_domain = get_domain_from_id(e.sender)
-            if (
-                sender_domain != local_server_name
-                and await storage.main.is_partial_state_room(e.room_id)
-            ):
-                partial_state_invisible_events.add(e)
+    partial_state_invisible_event_ids: Set[str] = set()
+    maybe_visible_events: List[EventBase] = []
+    for e in events:
+        sender_domain = get_domain_from_id(e.sender)
+        if (
+            sender_domain != local_server_name
+            and await storage.main.is_partial_state_room(e.room_id)
+        ):
+            partial_state_invisible_event_ids.add(e.event_id)
+        else:
+            maybe_visible_events.append(e)

    # Let's check to see if all the events have a history visibility
    # of "shared" or "world_readable". If that's the case then we don't
    # need to check membership (as we know the server is in the room).
-    event_to_history_vis = await _event_to_history_vis(storage, events)
+    event_to_history_vis = await _event_to_history_vis(storage, maybe_visible_events)

    # for any with restricted vis, we also need the memberships
    event_to_memberships = await _event_to_memberships(
        storage,
        [
            e
-            for e in events
+            for e in maybe_visible_events
            if event_to_history_vis[e.event_id]
            not in (HistoryVisibility.SHARED, HistoryVisibility.WORLD_READABLE)
        ],
        target_server_name,
    )

-    to_return = []
-    for e in events:
+    def include_event_in_output(e: EventBase) -> bool:
+        if e.event_id in partial_state_invisible_event_ids:
+            return False
+
        erased = is_sender_erased(e, erased_senders)
        visible = check_event_is_visible(
            event_to_history_vis[e.event_id], event_to_memberships.get(e.event_id, {})
        )

-        if e in partial_state_invisible_events:
-            visible = False
+        return visible and not erased

-        if visible and not erased:
+    to_return = []
+    for e in events:
+        if include_event_in_output(e):
            to_return.append(e)
        elif redact:
            to_return.append(prune_event(e))
@@ -1,4 +1,5 @@
-from typing import Callable, List, Optional, Tuple
+from typing import Callable, Collection, List, Optional, Tuple
+from unittest import mock
 from unittest.mock import Mock

 from twisted.test.proto_helpers import MemoryReactor
@@ -500,3 +501,77 @@ class FederationCatchUpTestCases(FederatingHomeserverTestCase):
        self.assertEqual(len(sent_pdus), 1)
        self.assertEqual(sent_pdus[0].event_id, event_2.event_id)
        self.assertFalse(per_dest_queue._catching_up)
+
+    def test_catch_up_is_not_blocked_by_partial_state_room(self) -> None:
+        """Detects (part of?) https://github.com/matrix-org/synapse/issues/15220."""
+        # ARRANGE:
+        # - a local user (u1)
+        # - a room with which contains u1 and two remote users, @u2:host2 and @u3:other
+        # - events in that room such that
+        #   - history visibility is restricted
+        #   - u1 sent message events
+        #   - afterwards, u3 sent a remote event
+        # - catchup to begin for host2
+        per_dest_queue, sent_pdus = self.make_fake_destination_queue()
+
+        self.register_user("u1", "you the one")
+        u1_token = self.login("u1", "you the one")
+        room = self.helper.create_room_as("u1", tok=u1_token)
+        self.helper.send_state(
+            room_id=room,
+            event_type="m.room.history_visibility",
+            body={"history_visibility": "joined"},
+            tok=u1_token,
+        )
+        self.get_success(
+            event_injection.inject_member_event(self.hs, room, "@u2:host2", "join")
+        )
+        self.get_success(
+            event_injection.inject_member_event(self.hs, room, "@u3:other", "join")
+        )
+
+        # create some events
+        event_id_1 = self.helper.send(room, "hello", tok=u1_token)["event_id"]
+        event_id_2 = self.helper.send(room, "world", tok=u1_token)["event_id"]
+        # pretend that u3 changes their displayname
+        event_id_3 = self.get_success(
+            event_injection.inject_member_event(self.hs, room, "@u3:other", "join")
+        ).event_id
+
+        # destination_rooms should already be populated, but let us pretend that we already
+        # sent (successfully) up to and including event id 1
+        event_1 = self.get_success(self.hs.get_datastores().main.get_event(event_id_1))
+        assert event_1.internal_metadata.stream_ordering is not None
+        self.get_success(
+            self.hs.get_datastores().main.set_destination_last_successful_stream_ordering(
+                "host2", event_1.internal_metadata.stream_ordering
+            )
+        )
+
+        # Mock event 3 as having partial state
+        self.get_success(
+            event_injection.mark_event_as_partial_state(self.hs, event_id_3, room)
+        )
+
+        # Fail the test if we block on full state for event 3.
+        async def mock_await_full_state(event_ids: Collection[str]) -> None:
+            if event_id_3 in event_ids:
+                raise AssertionError("Tried to await full state for event_id_3")
+
+        # ACT
+        with mock.patch.object(
+            self.hs.get_storage_controllers().state._partial_state_events_tracker,
+            "await_full_state",
+            mock_await_full_state,
+        ):
+            self.get_success(per_dest_queue._catch_up_transmission_loop())
+
+        # ASSERT
+        # We should have:
+        # - not sent event 3: it's not ours, and the room is partial stated
+        # - fallen back to sending event 2: it's the most recent event in the room
+        #   we tried to send to host2
+        # - completed catch-up
+        self.assertEqual(len(sent_pdus), 1)
+        self.assertEqual(sent_pdus[0].event_id, event_id_2)
+        self.assertFalse(per_dest_queue._catching_up)
@@ -231,8 +231,8 @@ class TestBulkPushRuleEvaluator(HomeserverTestCase):
    @override_config(
        {
            "experimental_features": {
-                "msc3758_exact_event_match": True,
                "msc3952_intentional_mentions": True,
+                "msc3966_exact_event_property_contains": True,
            }
        }
    )
@@ -334,8 +334,8 @@ class TestBulkPushRuleEvaluator(HomeserverTestCase):
    @override_config(
        {
            "experimental_features": {
-                "msc3758_exact_event_match": True,
                "msc3952_intentional_mentions": True,
+                "msc3966_exact_event_property_contains": True,
            }
        }
    )
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-from typing import Any, Dict, List, Optional, Set, Union, cast
+from typing import Any, Dict, List, Optional, Union, cast

 import frozendict

@@ -147,8 +147,6 @@ class PushRuleEvaluatorTestCase(unittest.TestCase):
        self,
        content: JsonMapping,
        *,
-        has_mentions: bool = False,
-        user_mentions: Optional[Set[str]] = None,
        related_events: Optional[JsonDict] = None,
    ) -> PushRuleEvaluator:
        event = FrozenEvent(
@@ -167,8 +165,7 @@ class PushRuleEvaluatorTestCase(unittest.TestCase):
        power_levels: Dict[str, Union[int, Dict[str, int]]] = {}
        return PushRuleEvaluator(
            _flatten_dict(event),
-            has_mentions,
-            user_mentions or set(),
+            False,
            room_member_count,
            sender_power_level,
            cast(Dict[str, int], power_levels.get("notifications", {})),
@@ -176,7 +173,6 @@ class PushRuleEvaluatorTestCase(unittest.TestCase):
            related_event_match_enabled=True,
            room_version_feature_flags=event.room_version.msc3931_push_features,
            msc3931_enabled=True,
-            msc3758_exact_event_match=True,
            msc3966_exact_event_property_contains=True,
        )

@@ -204,32 +200,6 @@ class PushRuleEvaluatorTestCase(unittest.TestCase):
        # A display name with spaces should work fine.
        self.assertTrue(evaluator.matches(condition, "@user:test", "foo bar"))

-    def test_user_mentions(self) -> None:
-        """Check for user mentions."""
-        condition = {"kind": "org.matrix.msc3952.is_user_mention"}
-
-        # No mentions shouldn't match.
-        evaluator = self._get_evaluator({}, has_mentions=True)
-        self.assertFalse(evaluator.matches(condition, "@user:test", None))
-
-        # An empty set shouldn't match
-        evaluator = self._get_evaluator({}, has_mentions=True, user_mentions=set())
-        self.assertFalse(evaluator.matches(condition, "@user:test", None))
-
-        # The Matrix ID appearing anywhere in the mentions list should match
-        evaluator = self._get_evaluator(
-            {}, has_mentions=True, user_mentions={"@user:test"}
-        )
-        self.assertTrue(evaluator.matches(condition, "@user:test", None))
-
-        evaluator = self._get_evaluator(
-            {}, has_mentions=True, user_mentions={"@another:test", "@user:test"}
-        )
-        self.assertTrue(evaluator.matches(condition, "@user:test", None))
-
-        # Note that invalid data is tested at tests.push.test_bulk_push_rule_evaluator.TestBulkPushRuleEvaluator.test_mentions
-        # since the BulkPushRuleEvaluator is what handles data sanitisation.
-
    def _assert_matches(
        self, condition: JsonDict, content: JsonMapping, msg: Optional[str] = None
    ) -> None:
@@ -433,7 +403,7 @@ class PushRuleEvaluatorTestCase(unittest.TestCase):

        # Test against a string value.
        condition = {
-            "kind": "com.beeper.msc3758.exact_event_match",
+            "kind": "event_property_is",
            "key": "content.value",
            "value": "foobaz",
        }
@@ -471,11 +441,7 @@ class PushRuleEvaluatorTestCase(unittest.TestCase):
        """Check that exact_event_match conditions work as expected for booleans."""

        # Test against a True boolean value.
-        condition = {
-            "kind": "com.beeper.msc3758.exact_event_match",
-            "key": "content.value",
-            "value": True,
-        }
+        condition = {"kind": "event_property_is", "key": "content.value", "value": True}
        self._assert_matches(
            condition,
            {"value": True},
@@ -495,7 +461,7 @@ class PushRuleEvaluatorTestCase(unittest.TestCase):

        # Test against a False boolean value.
        condition = {
-            "kind": "com.beeper.msc3758.exact_event_match",
+            "kind": "event_property_is",
            "key": "content.value",
            "value": False,
        }
@@ -520,11 +486,7 @@ class PushRuleEvaluatorTestCase(unittest.TestCase):
    def test_exact_event_match_null(self) -> None:
        """Check that exact_event_match conditions work as expected for null."""

-        condition = {
-            "kind": "com.beeper.msc3758.exact_event_match",
-            "key": "content.value",
-            "value": None,
-        }
+        condition = {"kind": "event_property_is", "key": "content.value", "value": None}
        self._assert_matches(
            condition,
            {"value": None},
@@ -540,11 +502,7 @@ class PushRuleEvaluatorTestCase(unittest.TestCase):
    def test_exact_event_match_integer(self) -> None:
        """Check that exact_event_match conditions work as expected for integers."""

-        condition = {
-            "kind": "com.beeper.msc3758.exact_event_match",
-            "key": "content.value",
-            "value": 1,
-        }
+        condition = {"kind": "event_property_is", "key": "content.value", "value": 1}
        self._assert_matches(
            condition,
            {"value": 1},
@@ -14,12 +14,21 @@

 from http import HTTPStatus

+from signedjson.key import (
+    encode_verify_key_base64,
+    generate_signing_key,
+    get_verify_key,
+)
+from signedjson.sign import sign_json
+
 from synapse.api.errors import Codes
 from synapse.rest import admin
 from synapse.rest.client import keys, login
+from synapse.types import JsonDict

 from tests import unittest
 from tests.http.server._base import make_request_with_cancellation_test
+from tests.unittest import override_config


 class KeyQueryTestCase(unittest.HomeserverTestCase):
@@ -118,3 +127,135 @@ class KeyQueryTestCase(unittest.HomeserverTestCase):

        self.assertEqual(200, channel.code, msg=channel.result["body"])
        self.assertIn(bob, channel.json_body["device_keys"])
+
+    def make_device_keys(self, user_id: str, device_id: str) -> JsonDict:
+        # We only generate a master key to simplify the test.
+        master_signing_key = generate_signing_key(device_id)
+        master_verify_key = encode_verify_key_base64(get_verify_key(master_signing_key))
+
+        return {
+            "master_key": sign_json(
+                {
+                    "user_id": user_id,
+                    "usage": ["master"],
+                    "keys": {"ed25519:" + master_verify_key: master_verify_key},
+                },
+                user_id,
+                master_signing_key,
+            ),
+        }
+
+    def test_device_signing_with_uia(self) -> None:
+        """Device signing key upload requires UIA."""
+        password = "wonderland"
+        device_id = "ABCDEFGHI"
+        alice_id = self.register_user("alice", password)
+        alice_token = self.login("alice", password, device_id=device_id)
+
+        content = self.make_device_keys(alice_id, device_id)
+
+        channel = self.make_request(
+            "POST",
+            "/_matrix/client/v3/keys/device_signing/upload",
+            content,
+            alice_token,
+        )
+
+        self.assertEqual(channel.code, HTTPStatus.UNAUTHORIZED, channel.result)
+        # Grab the session
+        session = channel.json_body["session"]
+        # Ensure that flows are what is expected.
+        self.assertIn({"stages": ["m.login.password"]}, channel.json_body["flows"])
+
+        # add UI auth
+        content["auth"] = {
+            "type": "m.login.password",
+            "identifier": {"type": "m.id.user", "user": alice_id},
+            "password": password,
+            "session": session,
+        }
+
+        channel = self.make_request(
+            "POST",
+            "/_matrix/client/v3/keys/device_signing/upload",
+            content,
+            alice_token,
+        )
+
+        self.assertEqual(channel.code, HTTPStatus.OK, channel.result)
+
+    @override_config({"ui_auth": {"session_timeout": "15m"}})
+    def test_device_signing_with_uia_session_timeout(self) -> None:
+        """Device signing key upload requires UIA buy passes with grace period."""
+        password = "wonderland"
+        device_id = "ABCDEFGHI"
+        alice_id = self.register_user("alice", password)
+        alice_token = self.login("alice", password, device_id=device_id)
+
+        content = self.make_device_keys(alice_id, device_id)
+
+        channel = self.make_request(
+            "POST",
+            "/_matrix/client/v3/keys/device_signing/upload",
+            content,
+            alice_token,
+        )
+
+        self.assertEqual(channel.code, HTTPStatus.OK, channel.result)
+
+    @override_config(
+        {
+            "experimental_features": {"msc3967_enabled": True},
+            "ui_auth": {"session_timeout": "15s"},
+        }
+    )
+    def test_device_signing_with_msc3967(self) -> None:
+        """Device signing key follows MSC3967 behaviour when enabled."""
+        password = "wonderland"
+        device_id = "ABCDEFGHI"
+        alice_id = self.register_user("alice", password)
+        alice_token = self.login("alice", password, device_id=device_id)
+
+        keys1 = self.make_device_keys(alice_id, device_id)
+
+        # Initial request should succeed as no existing keys are present.
+        channel = self.make_request(
+            "POST",
+            "/_matrix/client/v3/keys/device_signing/upload",
+            keys1,
+            alice_token,
+        )
+        self.assertEqual(channel.code, HTTPStatus.OK, channel.result)
+
+        keys2 = self.make_device_keys(alice_id, device_id)
+
+        # Subsequent request should require UIA as keys already exist even though session_timeout is set.
+        channel = self.make_request(
+            "POST",
+            "/_matrix/client/v3/keys/device_signing/upload",
+            keys2,
+            alice_token,
+        )
+        self.assertEqual(channel.code, HTTPStatus.UNAUTHORIZED, channel.result)
+
+        # Grab the session
+        session = channel.json_body["session"]
+        # Ensure that flows are what is expected.
+        self.assertIn({"stages": ["m.login.password"]}, channel.json_body["flows"])
+
+        # add UI auth
+        keys2["auth"] = {
+            "type": "m.login.password",
+            "identifier": {"type": "m.id.user", "user": alice_id},
+            "password": password,
+            "session": session,
+        }
+
+        # Request should complete
+        channel = self.make_request(
+            "POST",
+            "/_matrix/client/v3/keys/device_signing/upload",
+            keys2,
+            alice_token,
+        )
+        self.assertEqual(channel.code, HTTPStatus.OK, channel.result)
@@ -30,7 +30,6 @@ from tests import unittest
 from tests.server import FakeChannel
 from tests.test_utils import make_awaitable
 from tests.test_utils.event_injection import inject_event
-from tests.unittest import override_config


 class BaseRelationsTestCase(unittest.HomeserverTestCase):
@@ -403,7 +402,7 @@ class RelationsTestCase(BaseRelationsTestCase):

    def test_edit(self) -> None:
        """Test that a simple edit works."""
-
+        orig_body = {"body": "Hi!", "msgtype": "m.text"}
        new_body = {"msgtype": "m.text", "body": "I've been edited!"}
        edit_event_content = {
            "msgtype": "m.text",
@@ -424,9 +423,7 @@ class RelationsTestCase(BaseRelationsTestCase):
            access_token=self.user_token,
        )
        self.assertEqual(200, channel.code, channel.json_body)
-        self.assertEqual(
-            channel.json_body["content"], {"body": "Hi!", "msgtype": "m.text"}
-        )
+        self.assertEqual(channel.json_body["content"], orig_body)
        self._assert_edit_bundle(channel.json_body, edit_event_id, edit_event_content)

        # Request the room messages.
@@ -443,7 +440,7 @@ class RelationsTestCase(BaseRelationsTestCase):
        )

        # Request the room context.
-        # /context should return the edited event.
+        # /context should return the event.
        channel = self.make_request(
            "GET",
            f"/rooms/{self.room}/context/{self.parent_id}",
@@ -453,7 +450,7 @@ class RelationsTestCase(BaseRelationsTestCase):
        self._assert_edit_bundle(
            channel.json_body["event"], edit_event_id, edit_event_content
        )
-        self.assertEqual(channel.json_body["event"]["content"], new_body)
+        self.assertEqual(channel.json_body["event"]["content"], orig_body)

        # Request sync, but limit the timeline so it becomes limited (and includes
        # bundled aggregations).
@@ -491,45 +488,11 @@ class RelationsTestCase(BaseRelationsTestCase):
            edit_event_content,
        )

-    @override_config({"experimental_features": {"msc3925_inhibit_edit": True}})
-    def test_edit_inhibit_replace(self) -> None:
-        """
-        If msc3925_inhibit_edit is enabled, then the original event should not be
-        replaced.
-        """
-
-        new_body = {"msgtype": "m.text", "body": "I've been edited!"}
-        edit_event_content = {
-            "msgtype": "m.text",
-            "body": "foo",
-            "m.new_content": new_body,
-        }
-        channel = self._send_relation(
-            RelationTypes.REPLACE,
-            "m.room.message",
-            content=edit_event_content,
-        )
-        edit_event_id = channel.json_body["event_id"]
-
-        # /context should return the *original* event.
-        channel = self.make_request(
-            "GET",
-            f"/rooms/{self.room}/context/{self.parent_id}",
-            access_token=self.user_token,
-        )
-        self.assertEqual(200, channel.code, channel.json_body)
-        self.assertEqual(
-            channel.json_body["event"]["content"], {"body": "Hi!", "msgtype": "m.text"}
-        )
-        self._assert_edit_bundle(
-            channel.json_body["event"], edit_event_id, edit_event_content
-        )
-
    def test_multi_edit(self) -> None:
        """Test that multiple edits, including attempts by people who
        shouldn't be allowed, are correctly handled.
        """
-
+        orig_body = orig_body = {"body": "Hi!", "msgtype": "m.text"}
        self._send_relation(
            RelationTypes.REPLACE,
            "m.room.message",
@@ -570,7 +533,7 @@ class RelationsTestCase(BaseRelationsTestCase):
        )
        self.assertEqual(200, channel.code, channel.json_body)

-        self.assertEqual(channel.json_body["event"]["content"], new_body)
+        self.assertEqual(channel.json_body["event"]["content"], orig_body)
        self._assert_edit_bundle(
            channel.json_body["event"], edit_event_id, edit_event_content
        )
@@ -642,6 +605,7 @@ class RelationsTestCase(BaseRelationsTestCase):

    def test_edit_edit(self) -> None:
        """Test that an edit cannot be edited."""
+        orig_body = {"body": "Hi!", "msgtype": "m.text"}
        new_body = {"msgtype": "m.text", "body": "Initial edit"}
        edit_event_content = {
            "msgtype": "m.text",
@@ -675,14 +639,12 @@ class RelationsTestCase(BaseRelationsTestCase):
            access_token=self.user_token,
        )
        self.assertEqual(200, channel.code, channel.json_body)
-        self.assertEqual(
-            channel.json_body["content"], {"body": "Hi!", "msgtype": "m.text"}
-        )
+        self.assertEqual(channel.json_body["content"], orig_body)

        # The relations information should not include the edit to the edit.
        self._assert_edit_bundle(channel.json_body, edit_event_id, edit_event_content)

-        # /context should return the event updated for the *first* edit
+        # /context should return the bundled edit for the *first* edit
        # (The edit to the edit should be ignored.)
        channel = self.make_request(
            "GET",
@@ -690,7 +652,7 @@ class RelationsTestCase(BaseRelationsTestCase):
            access_token=self.user_token,
        )
        self.assertEqual(200, channel.code, channel.json_body)
-        self.assertEqual(channel.json_body["event"]["content"], new_body)
+        self.assertEqual(channel.json_body["event"]["content"], orig_body)
        self._assert_edit_bundle(
            channel.json_body["event"], edit_event_id, edit_event_content
        )
@@ -1287,7 +1249,6 @@ class BundledAggregationsTestCase(BaseRelationsTestCase):
        thread_summary = relations_dict[RelationTypes.THREAD]
        self.assertIn("latest_event", thread_summary)
        latest_event_in_thread = thread_summary["latest_event"]
-        self.assertEqual(latest_event_in_thread["content"]["body"], "I've been edited!")
        # The latest event in the thread should have the edit appear under the
        # bundled aggregations.
        self.assertDictContainsSubset(
@@ -696,6 +696,8 @@ class UserDirectoryICUTestCase(HomeserverTestCase):
                ["lazy'fox", "jumped", "over", "the", "dog"],
                # ICU 70 on Ubuntu 22.04
                ["lazy'fox", "jumped:over", "the.dog"],
+                # pyicu 2.10.2 on Alpine edge / macOS
+                ["lazy'fox", "jumped", "over", "the.dog"],
            ),
        )

@@ -102,3 +102,34 @@ async def create_event(
    context = await unpersisted_context.persist(event)

    return event, context
+
+
+async def mark_event_as_partial_state(
+    hs: synapse.server.HomeServer,
+    event_id: str,
+    room_id: str,
+) -> None:
+    """
+    (Falsely) mark an event as having partial state.
+
+    Naughty, but occasionally useful when checking that partial state doesn't
+    block something from happening.
+
+    If the event already has partial state, this insert will fail (event_id is unique
+    in this table).
+    """
+    store = hs.get_datastores().main
+    await store.db_pool.simple_upsert(
+        table="partial_state_rooms",
+        keyvalues={"room_id": room_id},
+        values={},
+        insertion_values={"room_id": room_id},
+    )
+
+    await store.db_pool.simple_insert(
+        table="partial_state_events",
+        values={
+            "room_id": room_id,
+            "event_id": event_id,
+        },
+    )
Author	SHA1	Message	Date
David Robertson	31f2b152d2	Changelog	2023-03-08 19:25:41 +00:00
David Robertson	f1392475ae	Avoid state lookups for events we'll ignore	2023-03-08 19:21:44 +00:00
David Robertson	1cb55e90be	Track a set of strings, not EventBases	2023-03-08 19:21:43 +00:00
David Robertson	7f977832d6	Separate decision from action	2023-03-08 19:21:43 +00:00
David Robertson	0aa0201452	Unconditionally omit remote events in partial state rooms I can't see why erased senders makes a difference here?	2023-03-08 19:21:29 +00:00
David Robertson	c813f89de6	Flip logic and provide better name	2023-03-08 19:19:46 +00:00
David Robertson	bebd7d29fc	Tweak docstring and type hint	2023-03-08 19:19:46 +00:00
David Robertson	6b70d44470	Test	2023-03-08 19:19:45 +00:00
David Robertson	9418344db4	Fix typo in changelog	2023-03-07 18:14:51 +00:00
David Robertson	2af1a982c1	Remove duplicate entry from changelog	2023-03-07 13:34:06 +00:00
David Robertson	8314646cd3	Update changelog	2023-03-07 13:30:47 +00:00
David Robertson	506e24ffc4	1.79.0rc1	2023-03-07 12:11:15 +00:00
David Robertson	c0854ce65a	Hack to rebuild the complement editable image (#15184 ) * Hack to rebuild the complement editable image * Changelog	2023-03-07 11:51:18 +00:00
dependabot[bot]	869ef75cb7	Bump types-pyopenssl from 22.1.0.2 to 23.0.0.4 (#15213 )	2023-03-07 10:14:21 +00:00
dependabot[bot]	2a869d257f	Bump types-pillow from 9.4.0.13 to 9.4.0.17 (#15211 )	2023-03-07 10:14:05 +00:00
dependabot[bot]	a9478e436e	Bump types-setuptools from 67.4.0.3 to 67.5.0.0 (#15212 )	2023-03-07 10:13:51 +00:00
dependabot[bot]	89ae8ce7ca	Bump types-psycopg2 from 2.9.21.4 to 2.9.21.8 (#15210 )	2023-03-07 10:13:24 +00:00
dependabot[bot]	c114befd6b	Bump types-commonmark from 0.9.2.1 to 0.9.2.2 (#15209 )	2023-03-07 10:13:10 +00:00
Erik Johnston	c69aae94cd	Split up txn for fetching device keys (#15215 ) We look up keys in batches, but we should do that outside of the transaction to avoid starving the database pool.	2023-03-07 08:51:34 +00:00
Quentin Gliech	41f127e068	Pass the requester during event serialization. (#15174 ) This allows Synapse to properly include the transaction ID in the unsigned data of events.	2023-03-06 16:08:39 +00:00
Patrick Cloke	05e0a4089a	Stop applying edits to event contents (MSC3925). (#15193 ) Enables MSC3925 support by default, which: * Includes the full edit event in the bundled aggregations of an edited event. * Stops modifying the original event's content to return the new content from the edit event. This is a backwards-incompatible change that is considered to be "correct" by the spec.	2023-03-06 09:43:01 -05:00
Patrick Cloke	fd9cadcf53	Stabilize support for MSC3758: event_property_is push condition (#15185 ) This removes the configuration flag & updates the identifiers to use the stable version.	2023-03-06 08:38:01 -05:00
dependabot[bot]	95876cf5f1	Bump serde_json from 1.0.93 to 1.0.94 (#15214 )Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com> * Bump serde_json from 1.0.93 to 1.0.94 Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.93 to 1.0.94. - [Release notes](https://github.com/serde-rs/json/releases) - [Commits](https://github.com/serde-rs/json/compare/v1.0.93...v1.0.94) --- updated-dependencies: - dependency-name: serde_json dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Changelog --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>	2023-03-06 12:01:05 +00:00
Erik Johnston	242d2a27ce	Use nightly rustfmt in CI (#15188 ) As we use some nightly only options, e.g. to group and sort imports consistently.	2023-03-03 14:26:14 +00:00
6543	6b6e91e610	Fix ICU tests on alpine / macOS. (#15177 ) The word boundary behaviour is slightly different, consider it acceptable for the tests.	2023-03-03 14:22:06 +00:00
Patrick Cloke	02f74f3a99	Combine AbstractStreamIdTracker and AbstractStreamIdGenerator. (#15192 ) AbstractStreamIdTracker (now) has only a single sub-class: AbstractStreamIdGenerator, combine them to simplify some code and remove any direct references to AbstractStreamIdTracker.	2023-03-03 08:13:37 -05:00
Quentin Gliech	848f7e3d5f	Remove unspecced and buggy `PUT` method on the unstable `/rooms/<room_id>/batch_send` endpoint. (#15199 )	2023-03-03 12:22:49 +00:00
Patrick Cloke	7ae4f7236a	Configure ruff to automatically fix issues. (#15194 )	2023-03-03 07:13:03 -05:00
Andrew Morgan	15e975f68f	Experimental MSC3890 Implementation: Fix deleting account data when using an account data writer worker (#14869 )	2023-03-03 10:51:57 +00:00
Andrew Morgan	1eea662780	Add a `get_next_txn` method to `StreamIdGenerator` to match `MultiWriterIdGenerator` (#15191	2023-03-02 18:27:00 +00:00
Dirk Klimpel	ecbe0ddbe7	Add support for knocking to workers. (#15133 )	2023-03-02 12:59:53 -05:00
Quentin Gliech	c8665dd25d	Remove the unspecced and bugged PUT /knock/{roomIdOrAlias} endpoint (#15189 )	2023-03-02 17:16:54 +00:00
David Robertson	c4f4dc35cd	Dockerfile-workers: spell out when config isn't generated (#15186 ) * Complement: Spell out when config isn't generated * Changelog	2023-03-02 15:55:26 +00:00
Patrick Cloke	8ef324ea6f	Update intentional mentions (MSC3952) to depend on `exact_event_property_contains` (MSC3966). (#15051 ) This replaces the specific `is_user_mention` push rule condition used in MSC3952 with the generic `exact_event_property_contains` push rule condition from MSC3966.	2023-03-02 08:30:51 -05:00
Patrick Cloke	33a85cf08c	Fix conflicting URLs for dehydrated devices. (#15180 )	2023-03-02 07:24:29 -05:00
Quentin Gliech	7ec1f096d3	Add Sytest jobs with the asyncio reactor enabled (#14101 )	2023-03-02 11:14:44 +00:00
Dirk Klimpel	65f10afb64	Move event_reports to `RoomWorkerStore` (#15165 )	2023-03-02 10:38:46 +00:00
Hugh Nimmo-Smith	916b8061d2	Implementation of MSC3967: Don't require UIA for initial upload of cross signing keys (#15077 )	2023-03-02 10:34:59 +00:00
				`@@ -1 +0,0 @@`
				`Batch up storing state groups when creating a new room.`
				`@@ -1 +0,0 @@`
				Add two new Third Party Rules module API callbacks: [`on_add_user_third_party_identifier`](https://matrix-org.github.io/synapse/v1.79/modules/third_party_rules_callbacks.html#on_add_user_third_party_identifier) and [`on_remove_user_third_party_identifier`](https://matrix-org.github.io/synapse/v1.79/modules/third_party_rules_callbacks.html#on_remove_user_third_party_identifier).
				`@@ -1 +0,0 @@`
				Clarify which worker processes the ThirdPartyRules' [`on_new_event`](https://matrix-org.github.io/synapse/v1.78/modules/third_party_rules_callbacks.html#on_new_event) module API callback runs on.
				`@@ -1 +0,0 @@`
				`Fix a long-standing bug where Synapse handled an unspecced field on push rules.`
				`@@ -1 +0,0 @@`
				`Fix a long-standing bug where a URL preview would break if the discovered oEmbed failed to download.`
				`@@ -1 +0,0 @@`
				Remove the unspecced `room_alias` field from the [`/createRoom`](https://spec.matrix.org/v1.6/client-server-api/#post_matrixclientv3createroom) response.
				`@@ -1 +0,0 @@`
				Refactor writing json data in `FileExfiltrationWriter`.
				`@@ -1 +0,0 @@`
				`Add media information to the command line [user data export tool](https://matrix-org.github.io/synapse/v1.79/usage/administration/admin_faq.html#how-can-i-export-user-data).`
				`@@ -1 +0,0 @@`
				`Document using [Shibboleth](https://www.shibboleth.net/) as an OpenID Provider.`