1.120.0

In a worker-mode deployment, the `E2eKeysHandler` is not necessarily
loaded, which means the handler for the `delete_old_otks` task will not
be registered. Make sure we load the handler.

Introduced in https://github.com/element-hq/synapse/pull/17934

CHANGES.md

@@ -1,3 +1,59 @@
+# Synapse 1.120.0 (2024-11-26)
+
+### Bugfixes
+
+- Fix a bug introduced in Synapse v1.120rc1 which would cause the newly-introduced `delete_old_otks` job to fail in worker-mode deployments. ([\#17960](https://github.com/element-hq/synapse/issues/17960))
+
+
+
+
+# Synapse 1.120.0rc1 (2024-11-20)
+
+This release enables the enforcement of authenticated media by default, with exemptions for media that is already present in the
+homeserver's media store.
+
+Most homeservers operating in the public federation will not be impacted by this change, given that
+the large homeserver `matrix.org` enabled this in September 2024 and therefore most clients and servers
+will already have updated as a result.
+
+Some server administrators may still wish to disable this enforcement for the time being, in the interest of compatibility with older clients
+and older federated homeservers.
+See the [upgrade notes](https://element-hq.github.io/synapse/v1.120/upgrade.html#authenticated-media-is-now-enforced-by-default) for more information.
+
+### Features
+
+- Enforce authenticated media by default. Administrators can revert this by configuring `enable_authenticated_media` to `false`. In a future release of Synapse, this option will be removed and become always-on. ([\#17889](https://github.com/element-hq/synapse/issues/17889))
+- Add a one-off task to delete old One-Time Keys, to guard against us having old OTKs in the database that the client has long forgotten about. ([\#17934](https://github.com/element-hq/synapse/issues/17934))
+
+### Improved Documentation
+
+- Clarify the semantics of the `enable_authenticated_media` configuration option. ([\#17913](https://github.com/element-hq/synapse/issues/17913))
+- Add documentation about backing up Synapse. ([\#17931](https://github.com/element-hq/synapse/issues/17931))
+
+### Deprecations and Removals
+
+- Remove support for [MSC3886: Simple client rendezvous capability](https://github.com/matrix-org/matrix-spec-proposals/pull/3886), which has been superseded by [MSC4108](https://github.com/matrix-org/matrix-spec-proposals/pull/4108) and therefore closed. ([\#17638](https://github.com/element-hq/synapse/issues/17638))
+
+### Internal Changes
+
+- Addressed some typos in docs and returned error message for unknown MXC ID. ([\#17865](https://github.com/element-hq/synapse/issues/17865))
+- Unpin the upload release GHA action. ([\#17923](https://github.com/element-hq/synapse/issues/17923))
+- Bump macOS version used to build wheels during release, as current version used is end-of-life. ([\#17924](https://github.com/element-hq/synapse/issues/17924))
+- Move server event filtering logic to Rust. ([\#17928](https://github.com/element-hq/synapse/issues/17928))
+- Support new package name of PyPI package `python-multipart` 0.0.13 so that distro packagers do not need to work around name conflict with PyPI package `multipart`. ([\#17932](https://github.com/element-hq/synapse/issues/17932))
+- Speed up slow initial sliding syncs on large servers. ([\#17946](https://github.com/element-hq/synapse/issues/17946))
+
+### Updates to locked dependencies
+
+* Bump anyhow from 1.0.92 to 1.0.93. ([\#17920](https://github.com/element-hq/synapse/issues/17920))
+* Bump bleach from 6.1.0 to 6.2.0. ([\#17918](https://github.com/element-hq/synapse/issues/17918))
+* Bump immutabledict from 4.2.0 to 4.2.1. ([\#17941](https://github.com/element-hq/synapse/issues/17941))
+* Bump packaging from 24.1 to 24.2. ([\#17940](https://github.com/element-hq/synapse/issues/17940))
+* Bump phonenumbers from 8.13.49 to 8.13.50. ([\#17942](https://github.com/element-hq/synapse/issues/17942))
+* Bump pygithub from 2.4.0 to 2.5.0. ([\#17917](https://github.com/element-hq/synapse/issues/17917))
+* Bump ruff from 0.7.2 to 0.7.3. ([\#17919](https://github.com/element-hq/synapse/issues/17919))
+* Bump serde from 1.0.214 to 1.0.215. ([\#17938](https://github.com/element-hq/synapse/issues/17938))
+
 # Synapse 1.119.0 (2024-11-13)
 
 No significant changes since 1.119.0rc2.

Cargo.lock

@@ -505,9 +505,9 @@ dependencies = [
 
 [[package]]
 name = "serde_json"
-version = "1.0.133"
+version = "1.0.132"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c7fceb2473b9166b2294ef05efcb65a3db80803f0b03ef86a5fc88a2b85ee377"
+checksum = "d726bfaff4b320266d395898905d0eba0345aae23b54aee3a737e260fd46db03"
 dependencies = [
  "itoa",
  "memchr",

changelog.d/17638.removal

@@ -1 +0,0 @@
-Remove support for closed [MSC3886](https://github.com/matrix-org/matrix-spec-proposals/pull/3886).

changelog.d/17865.misc

@@ -1 +0,0 @@
-Addressed some typos in docs and returned error message for unknown MXC ID.

changelog.d/17889.feature

@@ -1 +0,0 @@
-Enforce authenticated media by default. Administrators can revert this by configuring `enable_authenticated_media` to `false`. In a future release of Synapse, this option will be removed and become always-on.

changelog.d/17913.doc

@@ -1 +0,0 @@
-Clarify the semantics of the `enable_authenticated_media` configuration option.

changelog.d/17923.misc

@@ -1 +0,0 @@
-Unpin the upload release GHA action.

changelog.d/17924.misc

@@ -1 +0,0 @@
-Bump macos version used to build wheels during release, as current version used is end-of-life.

changelog.d/17928.misc

@@ -1 +0,0 @@
-Move server event filtering logic to rust.

changelog.d/17931.doc

@@ -1 +0,0 @@
-Add documentation about backing up Synapse.

changelog.d/17932.misc

@@ -1 +0,0 @@
-Support new package name of PyPI package `python-multipart` 0.0.13 so that distro packagers do not need to work around name conflict with PyPI package `multipart`.

changelog.d/17934.feature

@@ -1 +0,0 @@
-Add a one-off task to delete old one-time-keys, to guard against us having old OTKs in the database that the client has long forgotten about.

changelog.d/17944.misc

@@ -1 +0,0 @@
-Raise setuptools_rust version cap to 1.10.2.

changelog.d/17945.misc

@@ -1 +0,0 @@
-Enable encrypted appservice related experimental features in the complement docker image.

changelog.d/17946.misc

@@ -1 +0,0 @@
-Speed up slow initial sliding syncs on large servers.

changelog.d/17951.feature

@@ -1 +0,0 @@
-Add experimental option to pass through unsigned data in `/keys/query` responses.

debian/changelog

@@ -1,3 +1,15 @@
+matrix-synapse-py3 (1.120.0) stable; urgency=medium
+
+  * New synapse release 1.120.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 26 Nov 2024 13:10:23 +0000
+
+matrix-synapse-py3 (1.120.0~rc1) stable; urgency=medium
+
+  * New Synapse release 1.120.0rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Wed, 20 Nov 2024 15:02:21 +0000
+
 matrix-synapse-py3 (1.119.0) stable; urgency=medium
 
   * New Synapse release 1.119.0.

docker/complement/conf/workers-shared-extra.yaml.j2

@@ -104,18 +104,6 @@ experimental_features:
   msc3967_enabled: true
   # Expose a room summary for public rooms
   msc3266_enabled: true
-  # Send to-device messages to application services
-  msc2409_to_device_messages_enabled: true
-  # Allow application services to masquerade devices
-  msc3202_device_masquerading: true
-  # Sending device list changes, one-time key counts and fallback key usage to application services
-  msc3202_transaction_extensions: true
-  # Proxy OTK claim requests to exclusive ASes
-  msc3983_appservice_otk_claims: true
-  # Proxy key queries to exclusive ASes
-  msc3984_appservice_key_query: true
-  # Pass through unsigned device data in /keys/query
-  msc4229_enabled: true
 
 server_notices:
   system_mxid_localpart: _server

pyproject.toml

@@ -97,7 +97,7 @@ module-name = "synapse.synapse_rust"
 
 [tool.poetry]
 name = "matrix-synapse"
-version = "1.119.0"
+version = "1.120.0"
 description = "Homeserver for the Matrix decentralised comms protocol"
 authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
 license = "AGPL-3.0-or-later"
@@ -370,7 +370,7 @@ tomli = ">=1.2.3"
 # runtime errors caused by build system changes.
 # We are happy to raise these upper bounds upon request,
 # provided we check that it's safe to do so (i.e. that CI passes).
-requires = ["poetry-core>=1.1.0,<=1.9.1", "setuptools_rust>=1.3,<=1.10.2"]
+requires = ["poetry-core>=1.1.0,<=1.9.1", "setuptools_rust>=1.3,<=1.8.1"]
 build-backend = "poetry.core.masonry.api"
 
 

scripts-dev/complement.sh

@@ -225,7 +225,6 @@ test_packages=(
     ./tests/msc3902
     ./tests/msc3967
     ./tests/msc4140
-    ./tests/msc4229
 )
 
 # Enable dirty runs, so tests will reuse the same container where possible.

synapse/config/experimental.py

@@ -448,6 +448,3 @@ class ExperimentalConfig(Config):
 
         # MSC4222: Adding `state_after` to sync v2
         self.msc4222_enabled: bool = experimental.get("msc4222_enabled", False)
-
-        # MSC4229: Pass through `unsigned` data from `/keys/upload` to `/keys/query`
-        self.msc4229_enabled: bool = experimental.get("msc4229_enabled", False)

synapse/handlers/e2e_keys.py

@@ -542,9 +542,7 @@ class E2eKeysHandler:
             result_dict[user_id] = {}
 
         results = await self.store.get_e2e_device_keys_for_cs_api(
-            local_query,
-            include_displaynames,
-            include_uploaded_unsigned_data=self.config.experimental.msc4229_enabled,
+            local_query, include_displaynames
         )
 
         # Check if the application services have any additional results.

synapse/server.py

@@ -254,6 +254,7 @@ class HomeServer(metaclass=abc.ABCMeta):
         "auth",
         "deactivate_account",
         "delayed_events",
+        "e2e_keys",  # for the `delete_old_otks` scheduled-task handler
         "message",
         "pagination",
         "profile",

synapse/storage/databases/main/end_to_end_keys.py

@@ -220,15 +220,12 @@ class EndToEndKeyWorkerStore(EndToEndKeyBackgroundStore, CacheInvalidationWorker
         self,
         query_list: Collection[Tuple[str, Optional[str]]],
         include_displaynames: bool = True,
-        include_uploaded_unsigned_data: bool = False,
     ) -> Dict[str, Dict[str, JsonDict]]:
         """Fetch a list of device keys, formatted suitably for the C/S API.
         Args:
             query_list: List of pairs of user_ids and device_ids.
             include_displaynames: Whether to include the displayname of returned devices
                 (if one exists).
-            include_uploaded_unsigned_data: Whether to include uploaded `unsigned` data
-                in the response
         Returns:
             Dict mapping from user-id to dict mapping from device_id to
             key data.  The key data will be a dict in the same format as the
@@ -250,13 +247,7 @@ class EndToEndKeyWorkerStore(EndToEndKeyBackgroundStore, CacheInvalidationWorker
                 if r is None:
                     continue
 
-                # If there was already an `unsigned` dict in the uploaded key, keep it.
-                # Otherwise, create a new one.
-                if not include_uploaded_unsigned_data or not isinstance(
-                    r.get("unsigned"), dict
-                ):
-                    r["unsigned"] = {}
-
+                r["unsigned"] = {}
                 if include_displaynames:
                     # Include the device's display name in the "unsigned" dictionary
                     display_name = device_info.display_name

tests/rest/client/test_keys.py

@@ -19,7 +19,6 @@
 #
 #
 import urllib.parse
-from copy import deepcopy
 from http import HTTPStatus
 from unittest.mock import patch
 
@@ -206,141 +205,6 @@ class KeyQueryTestCase(unittest.HomeserverTestCase):
         self.assertEqual(channel.code, HTTPStatus.OK, channel.result)
 
 
-class UnsignedKeyDataTestCase(unittest.HomeserverTestCase):
-    servlets = [
-        keys.register_servlets,
-        admin.register_servlets_for_client_rest_resource,
-        login.register_servlets,
-    ]
-
-    def default_config(self) -> JsonDict:
-        config = super().default_config()
-        config["experimental_features"] = {"msc4229_enabled": True}
-        return config
-
-    def make_key_data(self, user_id: str, device_id: str) -> JsonDict:
-        return {
-            "algorithms": ["m.olm.v1.curve25519-aes-sha2", "m.megolm.v1.aes-sha2"],
-            "device_id": device_id,
-            "keys": {
-                f"curve25519:{device_id}": "keykeykey",
-                f"ed25519:{device_id}": "keykeykey",
-            },
-            "signatures": {user_id: {f"ed25519:{device_id}": "sigsigsig"}},
-            "user_id": user_id,
-        }
-
-    def test_unsigned_uploaded_data_returned_in_keys_query(self) -> None:
-        password = "wonderland"
-        device_id = "ABCDEFGHI"
-        alice_id = self.register_user("alice", password)
-        alice_token = self.login(
-            "alice",
-            password,
-            device_id=device_id,
-            additional_request_fields={"initial_device_display_name": "mydevice"},
-        )
-
-        # Alice uploads some keys, with a bit of unsigned data
-        keys1 = self.make_key_data(alice_id, device_id)
-        keys1["unsigned"] = {"a": "b"}
-
-        channel = self.make_request(
-            "POST",
-            "/_matrix/client/v3/keys/upload",
-            {"device_keys": keys1},
-            alice_token,
-        )
-        self.assertEqual(channel.code, HTTPStatus.OK, channel.result)
-
-        # /keys/query should return the unsigned data, with the device display name merged in.
-        channel = self.make_request(
-            "POST",
-            "/_matrix/client/v3/keys/query",
-            {"device_keys": {alice_id: []}},
-            alice_token,
-        )
-        self.assertEqual(channel.code, HTTPStatus.OK, channel.result)
-        device_response = channel.json_body["device_keys"][alice_id][device_id]
-        expected_device_response = deepcopy(keys1)
-        expected_device_response["unsigned"]["device_display_name"] = "mydevice"
-        self.assertEqual(device_response, expected_device_response)
-
-        # /_matrix/federation/v1/user/devices/{userId} should return the unsigned data too
-        fed_response = self.get_success(
-            self.hs.get_device_handler().on_federation_query_user_devices(alice_id)
-        )
-        self.assertEqual(
-            fed_response["devices"][0],
-            {"device_id": device_id, "keys": keys1},
-        )
-
-        # so should /_matrix/federation/v1/user/keys/query
-        fed_response = self.get_success(
-            self.hs.get_e2e_keys_handler().on_federation_query_client_keys(
-                {"device_keys": {alice_id: []}}
-            )
-        )
-        fed_device_response = fed_response["device_keys"][alice_id][device_id]
-        self.assertEqual(fed_device_response, keys1)
-
-    def test_non_dict_unsigned_is_ignored(self) -> None:
-        password = "wonderland"
-        device_id = "ABCDEFGHI"
-        alice_id = self.register_user("alice", password)
-        alice_token = self.login(
-            "alice",
-            password,
-            device_id=device_id,
-            additional_request_fields={"initial_device_display_name": "mydevice"},
-        )
-
-        # Alice uploads some keys, with a malformed unsigned data
-        keys1 = self.make_key_data(alice_id, device_id)
-        keys1["unsigned"] = ["a", "b"]  # a list!
-
-        channel = self.make_request(
-            "POST",
-            "/_matrix/client/v3/keys/upload",
-            {"device_keys": keys1},
-            alice_token,
-        )
-        self.assertEqual(channel.code, HTTPStatus.OK, channel.result)
-
-        # /keys/query should return the unsigned data, with the device display name merged in.
-        channel = self.make_request(
-            "POST",
-            "/_matrix/client/v3/keys/query",
-            {"device_keys": {alice_id: []}},
-            alice_token,
-        )
-        self.assertEqual(channel.code, HTTPStatus.OK, channel.result)
-        device_response = channel.json_body["device_keys"][alice_id][device_id]
-        expected_device_response = deepcopy(keys1)
-        expected_device_response["unsigned"] = {"device_display_name": "mydevice"}
-        self.assertEqual(device_response, expected_device_response)
-
-        # /_matrix/federation/v1/user/devices/{userId} should return the unsigned data too
-        fed_response = self.get_success(
-            self.hs.get_device_handler().on_federation_query_user_devices(alice_id)
-        )
-        self.assertEqual(
-            fed_response["devices"][0],
-            {"device_id": device_id, "keys": keys1},
-        )
-
-        # so should /_matrix/federation/v1/user/keys/query
-        fed_response = self.get_success(
-            self.hs.get_e2e_keys_handler().on_federation_query_client_keys(
-                {"device_keys": {alice_id: []}}
-            )
-        )
-        fed_device_response = fed_response["device_keys"][alice_id][device_id]
-        expected_device_response = deepcopy(keys1)
-        expected_device_response["unsigned"] = {}
-        self.assertEqual(fed_device_response, expected_device_response)
-
-
 class SigningKeyUploadServletTestCase(unittest.HomeserverTestCase):
     servlets = [
         admin.register_servlets,