changelog

We'd like a mechanism by which a client can add "unsigned" data to their device
keys, and have it be accessible by other clients involved in E2EE discussions.

Most of this actually already works; the bit that doesn't is that *client-side*
`/keys/query` strips out any "unsigned" data from the `/keys/upload`
body. (Server-side `/keys/query` follows a different codepath and is fine).

This commit adds an experimental option which modifies client-side
`/keys/query` so that `unsigned` data is preserved.

.github/workflows/release-artifacts.yml

@@ -5,7 +5,7 @@ name: Build release artifacts
 on:
   # we build on PRs and develop to (hopefully) get early warning
   # of things breaking (but only build one set of debs). PRs skip
-  # building wheels on ARM.
+  # building wheels on macOS & ARM.
   pull_request:
   push:
     branches: ["develop", "release-*"]
@@ -111,7 +111,7 @@ jobs:
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
-        os: [ubuntu-22.04]
+        os: [ubuntu-22.04, macos-13]
         arch: [x86_64, aarch64]
         # is_pr is a flag used to exclude certain jobs from the matrix on PRs.
         # It is not read by the rest of the workflow.
@@ -119,6 +119,12 @@ jobs:
           - ${{ startsWith(github.ref, 'refs/pull/') }}
 
         exclude:
+          # Don't build macos wheels on PR CI.
+          - is_pr: true
+            os: "macos-13"
+          # Don't build aarch64 wheels on mac.
+          - os: "macos-13"
+            arch: aarch64
           # Don't build aarch64 wheels on PR CI.
           - is_pr: true
             arch: aarch64
@@ -206,8 +212,7 @@ jobs:
           mv debs*/* debs/
           tar -cvJf debs.tar.xz debs
       - name: Attach to release
-        # Pinned to work around https://github.com/softprops/action-gh-release/issues/445
-        uses: softprops/action-gh-release@v2.0.5
+        uses: softprops/action-gh-release@v2
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         with:

CHANGES.md

@@ -1,109 +1,3 @@
-# Synapse 1.120.2 (2024-12-03)
-
-This version has building of wheels for macOS disabled.
-It is functionally identical to 1.120.1, which contains multiple security fixes.
-If you are already using 1.120.1, there is no need to upgrade to this version.
-
-
-
-# Synapse 1.120.1 (2024-12-03)
-
-This patch release fixes multiple security vulnerabilities, some affecting all prior versions of Synapse. Server administrators are encouraged to update Synapse as soon as possible. We are not aware of these vulnerabilities being exploited in the wild.
-
-Administrators who are unable to update Synapse may use the workarounds described in the linked GitHub Security Advisory below.
-
-### Security advisory
-
-The following issues are fixed in 1.120.1.
-
-- [GHSA-rfq8-j7rh-8hf2](https://github.com/element-hq/synapse/security/advisories/GHSA-rfq8-j7rh-8hf2) / [CVE-2024-52805](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-52805): **Unsupported content types can lead to memory exhaustion**
-
-  Synapse instances which have a high `max_upload_size` and which don't have a reverse proxy in front of them that would otherwise limit upload size are affected.
-
-  Fixed by [4b7154c58501b4bf5e1c2d6c11ebef96529f2fdf](https://github.com/element-hq/synapse/commit/4b7154c58501b4bf5e1c2d6c11ebef96529f2fdf).
-
-- [GHSA-f3r3-h2mq-hx2h](https://github.com/element-hq/synapse/security/advisories/GHSA-f3r3-h2mq-hx2h) / [CVE-2024-52815](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-52815): **Malicious invites via federation can break a user's sync**
-
-  Fixed by [d82e1ed357b7ee21dff83d06cba7a67840cfd464](https://github.com/element-hq/synapse/commit/d82e1ed357b7ee21dff83d06cba7a67840cfd464).
-
-- [GHSA-vp6v-whfm-rv3g](https://github.com/element-hq/synapse/security/advisories/GHSA-vp6v-whfm-rv3g) / [CVE-2024-53863](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-53863): **Synapse can be forced to thumbnail unexpected file formats, invoking potentially untrustworthy decoders**
-
-  Synapse instances can disable dynamic thumbnailing by setting `dynamic_thumbnails` to `false` in the configuration file.
-
-  Fixed by [b64a4e5fbbbf119b6c65aedf0d999b4237d55503](https://github.com/element-hq/synapse/commit/b64a4e5fbbbf119b6c65aedf0d999b4237d55503).
-
-- [GHSA-56w4-5538-8v8h](https://github.com/element-hq/synapse/security/advisories/GHSA-56w4-5538-8v8h) / [CVE-2024-53867](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-53867): **The Sliding Sync feature on Synapse versions between 1.113.0rc1 and 1.120.0 can leak partial room state changes to users no longer in a room**
-
-  Non-state events, like messages, are unaffected.
-
-  Synapse instances can disable the Sliding Sync feature by setting `experimental_features.msc3575_enabled` to `false` in the configuration file.
-
-  Fixed by [4daa533e82f345ce87b9495d31781af570ba3ead](https://github.com/element-hq/synapse/commit/4daa533e82f345ce87b9495d31781af570ba3ead).
-
-See the advisories for more details. If you have any questions, email [security at element.io](mailto:security@element.io).
-
-### Bugfixes
-
-- Fix release process to not create duplicate releases. ([\#17970](https://github.com/element-hq/synapse/issues/17970))
-
-
-
-# Synapse 1.120.0 (2024-11-26)
-
-### Bugfixes
-
-- Fix a bug introduced in Synapse v1.120rc1 which would cause the newly-introduced `delete_old_otks` job to fail in worker-mode deployments. ([\#17960](https://github.com/element-hq/synapse/issues/17960))
-
-
-
-
-# Synapse 1.120.0rc1 (2024-11-20)
-
-This release enables the enforcement of authenticated media by default, with exemptions for media that is already present in the
-homeserver's media store.
-
-Most homeservers operating in the public federation will not be impacted by this change, given that
-the large homeserver `matrix.org` enabled this in September 2024 and therefore most clients and servers
-will already have updated as a result.
-
-Some server administrators may still wish to disable this enforcement for the time being, in the interest of compatibility with older clients
-and older federated homeservers.
-See the [upgrade notes](https://element-hq.github.io/synapse/v1.120/upgrade.html#authenticated-media-is-now-enforced-by-default) for more information.
-
-### Features
-
-- Enforce authenticated media by default. Administrators can revert this by configuring `enable_authenticated_media` to `false`. In a future release of Synapse, this option will be removed and become always-on. ([\#17889](https://github.com/element-hq/synapse/issues/17889))
-- Add a one-off task to delete old One-Time Keys, to guard against us having old OTKs in the database that the client has long forgotten about. ([\#17934](https://github.com/element-hq/synapse/issues/17934))
-
-### Improved Documentation
-
-- Clarify the semantics of the `enable_authenticated_media` configuration option. ([\#17913](https://github.com/element-hq/synapse/issues/17913))
-- Add documentation about backing up Synapse. ([\#17931](https://github.com/element-hq/synapse/issues/17931))
-
-### Deprecations and Removals
-
-- Remove support for [MSC3886: Simple client rendezvous capability](https://github.com/matrix-org/matrix-spec-proposals/pull/3886), which has been superseded by [MSC4108](https://github.com/matrix-org/matrix-spec-proposals/pull/4108) and therefore closed. ([\#17638](https://github.com/element-hq/synapse/issues/17638))
-
-### Internal Changes
-
-- Addressed some typos in docs and returned error message for unknown MXC ID. ([\#17865](https://github.com/element-hq/synapse/issues/17865))
-- Unpin the upload release GHA action. ([\#17923](https://github.com/element-hq/synapse/issues/17923))
-- Bump macOS version used to build wheels during release, as current version used is end-of-life. ([\#17924](https://github.com/element-hq/synapse/issues/17924))
-- Move server event filtering logic to Rust. ([\#17928](https://github.com/element-hq/synapse/issues/17928))
-- Support new package name of PyPI package `python-multipart` 0.0.13 so that distro packagers do not need to work around name conflict with PyPI package `multipart`. ([\#17932](https://github.com/element-hq/synapse/issues/17932))
-- Speed up slow initial sliding syncs on large servers. ([\#17946](https://github.com/element-hq/synapse/issues/17946))
-
-### Updates to locked dependencies
-
-* Bump anyhow from 1.0.92 to 1.0.93. ([\#17920](https://github.com/element-hq/synapse/issues/17920))
-* Bump bleach from 6.1.0 to 6.2.0. ([\#17918](https://github.com/element-hq/synapse/issues/17918))
-* Bump immutabledict from 4.2.0 to 4.2.1. ([\#17941](https://github.com/element-hq/synapse/issues/17941))
-* Bump packaging from 24.1 to 24.2. ([\#17940](https://github.com/element-hq/synapse/issues/17940))
-* Bump phonenumbers from 8.13.49 to 8.13.50. ([\#17942](https://github.com/element-hq/synapse/issues/17942))
-* Bump pygithub from 2.4.0 to 2.5.0. ([\#17917](https://github.com/element-hq/synapse/issues/17917))
-* Bump ruff from 0.7.2 to 0.7.3. ([\#17919](https://github.com/element-hq/synapse/issues/17919))
-* Bump serde from 1.0.214 to 1.0.215. ([\#17938](https://github.com/element-hq/synapse/issues/17938))
-
 # Synapse 1.119.0 (2024-11-13)
 
 No significant changes since 1.119.0rc2.

Cargo.lock

@@ -505,9 +505,9 @@ dependencies = [
 
 [[package]]
 name = "serde_json"
-version = "1.0.132"
+version = "1.0.133"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d726bfaff4b320266d395898905d0eba0345aae23b54aee3a737e260fd46db03"
+checksum = "c7fceb2473b9166b2294ef05efcb65a3db80803f0b03ef86a5fc88a2b85ee377"
 dependencies = [
  "itoa",
  "memchr",

changelog.d/17638.removal

@@ -0,0 +1 @@
+Remove support for closed [MSC3886](https://github.com/matrix-org/matrix-spec-proposals/pull/3886).

changelog.d/17865.misc

@@ -0,0 +1 @@
+Addressed some typos in docs and returned error message for unknown MXC ID.

changelog.d/17889.feature

@@ -0,0 +1 @@
+Enforce authenticated media by default. Administrators can revert this by configuring `enable_authenticated_media` to `false`. In a future release of Synapse, this option will be removed and become always-on.

changelog.d/17913.doc

@@ -0,0 +1 @@
+Clarify the semantics of the `enable_authenticated_media` configuration option.

changelog.d/17923.misc

@@ -0,0 +1 @@
+Unpin the upload release GHA action.

changelog.d/17924.misc

@@ -0,0 +1 @@
+Bump macos version used to build wheels during release, as current version used is end-of-life.

changelog.d/17928.misc

@@ -0,0 +1 @@
+Move server event filtering logic to rust.

changelog.d/17931.doc

@@ -0,0 +1 @@
+Add documentation about backing up Synapse.

changelog.d/17932.misc

@@ -0,0 +1 @@
+Support new package name of PyPI package `python-multipart` 0.0.13 so that distro packagers do not need to work around name conflict with PyPI package `multipart`.

changelog.d/17934.feature

@@ -0,0 +1 @@
+Add a one-off task to delete old one-time-keys, to guard against us having old OTKs in the database that the client has long forgotten about.

changelog.d/17944.misc

@@ -0,0 +1 @@
+Raise setuptools_rust version cap to 1.10.2.

changelog.d/17945.misc

@@ -0,0 +1 @@
+Enable encrypted appservice related experimental features in the complement docker image.

changelog.d/17946.misc

@@ -0,0 +1 @@
+Speed up slow initial sliding syncs on large servers.

changelog.d/17951.feature

@@ -0,0 +1 @@
+Add experimental option to pass through unsigned data in `/keys/query` responses.

debian/changelog

@@ -1,27 +1,3 @@
-matrix-synapse-py3 (1.120.2) stable; urgency=medium
-
-  * New synapse release 1.120.2.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 03 Dec 2024 15:43:37 +0000
-
-matrix-synapse-py3 (1.120.1) stable; urgency=medium
-
-  * New synapse release 1.120.1.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 03 Dec 2024 09:07:57 +0000
-
-matrix-synapse-py3 (1.120.0) stable; urgency=medium
-
-  * New synapse release 1.120.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 26 Nov 2024 13:10:23 +0000
-
-matrix-synapse-py3 (1.120.0~rc1) stable; urgency=medium
-
-  * New Synapse release 1.120.0rc1.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 20 Nov 2024 15:02:21 +0000
-
 matrix-synapse-py3 (1.119.0) stable; urgency=medium
 
   * New Synapse release 1.119.0.

docker/complement/conf/workers-shared-extra.yaml.j2

@@ -104,6 +104,18 @@ experimental_features:
   msc3967_enabled: true
   # Expose a room summary for public rooms
   msc3266_enabled: true
+  # Send to-device messages to application services
+  msc2409_to_device_messages_enabled: true
+  # Allow application services to masquerade devices
+  msc3202_device_masquerading: true
+  # Sending device list changes, one-time key counts and fallback key usage to application services
+  msc3202_transaction_extensions: true
+  # Proxy OTK claim requests to exclusive ASes
+  msc3983_appservice_otk_claims: true
+  # Proxy key queries to exclusive ASes
+  msc3984_appservice_key_query: true
+  # Pass through unsigned device data in /keys/query
+  msc4229_enabled: true
 
 server_notices:
   system_mxid_localpart: _server

pyproject.toml

@@ -97,7 +97,7 @@ module-name = "synapse.synapse_rust"
 
 [tool.poetry]
 name = "matrix-synapse"
-version = "1.120.2"
+version = "1.119.0"
 description = "Homeserver for the Matrix decentralised comms protocol"
 authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
 license = "AGPL-3.0-or-later"
@@ -370,7 +370,7 @@ tomli = ">=1.2.3"
 # runtime errors caused by build system changes.
 # We are happy to raise these upper bounds upon request,
 # provided we check that it's safe to do so (i.e. that CI passes).
-requires = ["poetry-core>=1.1.0,<=1.9.1", "setuptools_rust>=1.3,<=1.8.1"]
+requires = ["poetry-core>=1.1.0,<=1.9.1", "setuptools_rust>=1.3,<=1.10.2"]
 build-backend = "poetry.core.masonry.api"
 
 

scripts-dev/complement.sh

@@ -225,6 +225,7 @@ test_packages=(
     ./tests/msc3902
     ./tests/msc3967
     ./tests/msc4140
+    ./tests/msc4229
 )
 
 # Enable dirty runs, so tests will reuse the same container where possible.

synapse/config/experimental.py

@@ -448,3 +448,6 @@ class ExperimentalConfig(Config):
 
         # MSC4222: Adding `state_after` to sync v2
         self.msc4222_enabled: bool = experimental.get("msc4222_enabled", False)
+
+        # MSC4229: Pass through `unsigned` data from `/keys/upload` to `/keys/query`
+        self.msc4229_enabled: bool = experimental.get("msc4229_enabled", False)

synapse/federation/transport/server/federation.py

@@ -509,9 +509,6 @@ class FederationV2InviteServlet(BaseFederationServerServlet):
         event = content["event"]
         invite_room_state = content.get("invite_room_state", [])
 
-        if not isinstance(invite_room_state, list):
-            invite_room_state = []
-
         # Synapse expects invite_room_state to be in unsigned, as it is in v1
         # API
 

synapse/handlers/e2e_keys.py

@@ -542,7 +542,9 @@ class E2eKeysHandler:
             result_dict[user_id] = {}
 
         results = await self.store.get_e2e_device_keys_for_cs_api(
-            local_query, include_displaynames
+            local_query,
+            include_displaynames,
+            include_uploaded_unsigned_data=self.config.experimental.msc4229_enabled,
         )
 
         # Check if the application services have any additional results.

synapse/handlers/federation.py

@@ -880,9 +880,6 @@ class FederationHandler:
         if stripped_room_state is None:
             raise KeyError("Missing 'knock_room_state' field in send_knock response")
 
-        if not isinstance(stripped_room_state, list):
-            raise TypeError("'knock_room_state' has wrong type")
-
         event.unsigned["knock_room_state"] = stripped_room_state
 
         context = EventContext.for_outlier(self._storage_controllers)

synapse/handlers/sliding_sync/__init__.py

@@ -39,7 +39,6 @@ from synapse.logging.opentracing import (
     trace,
 )
 from synapse.storage.databases.main.roommember import extract_heroes_from_room_summary
-from synapse.storage.databases.main.state_deltas import StateDelta
 from synapse.storage.databases.main.stream import PaginateFunction
 from synapse.storage.roommember import (
     MemberSummary,
@@ -49,7 +48,6 @@ from synapse.types import (
     MutableStateMap,
     PersistedEventPosition,
     Requester,
-    RoomStreamToken,
     SlidingSyncStreamToken,
     StateMap,
     StrCollection,
@@ -472,64 +470,6 @@ class SlidingSyncHandler:
 
         return state_map
 
-    @trace
-    async def get_current_state_deltas_for_room(
-        self,
-        room_id: str,
-        room_membership_for_user_at_to_token: RoomsForUserType,
-        from_token: RoomStreamToken,
-        to_token: RoomStreamToken,
-    ) -> List[StateDelta]:
-        """
-        Get the state deltas between two tokens taking into account the user's
-        membership. If the user is LEAVE/BAN, we will only get the state deltas up to
-        their LEAVE/BAN event (inclusive).
-
-        (> `from_token` and <= `to_token`)
-        """
-        membership = room_membership_for_user_at_to_token.membership
-        # We don't know how to handle `membership` values other than these. The
-        # code below would need to be updated.
-        assert membership in (
-            Membership.JOIN,
-            Membership.INVITE,
-            Membership.KNOCK,
-            Membership.LEAVE,
-            Membership.BAN,
-        )
-
-        # People shouldn't see past their leave/ban event
-        if membership in (
-            Membership.LEAVE,
-            Membership.BAN,
-        ):
-            to_bound = (
-                room_membership_for_user_at_to_token.event_pos.to_room_stream_token()
-            )
-        # If we are participating in the room, we can get the latest current state in
-        # the room
-        elif membership == Membership.JOIN:
-            to_bound = to_token
-        # We can only rely on the stripped state included in the invite/knock event
-        # itself so there will never be any state deltas to send down.
-        elif membership in (Membership.INVITE, Membership.KNOCK):
-            return []
-        else:
-            # We don't know how to handle this type of membership yet
-            #
-            # FIXME: We should use `assert_never` here but for some reason
-            # the exhaustive matching doesn't recognize the `Never` here.
-            # assert_never(membership)
-            raise AssertionError(
-                f"Unexpected membership {membership} that we don't know how to handle yet"
-            )
-
-        return await self.store.get_current_state_deltas_for_room(
-            room_id=room_id,
-            from_token=from_token,
-            to_token=to_bound,
-        )
-
     @trace
     async def get_room_sync_data(
         self,
@@ -815,19 +755,13 @@ class SlidingSyncHandler:
 
             stripped_state = []
             if invite_or_knock_event.membership == Membership.INVITE:
-                invite_state = invite_or_knock_event.unsigned.get(
-                    "invite_room_state", []
+                stripped_state.extend(
+                    invite_or_knock_event.unsigned.get("invite_room_state", [])
                 )
-                if not isinstance(invite_state, list):
-                    invite_state = []
-
-                stripped_state.extend(invite_state)
             elif invite_or_knock_event.membership == Membership.KNOCK:
-                knock_state = invite_or_knock_event.unsigned.get("knock_room_state", [])
-                if not isinstance(knock_state, list):
-                    knock_state = []
-
-                stripped_state.extend(knock_state)
+                stripped_state.extend(
+                    invite_or_knock_event.unsigned.get("knock_room_state", [])
+                )
 
             stripped_state.append(strip_event(invite_or_knock_event))
 
@@ -856,9 +790,8 @@ class SlidingSyncHandler:
             # TODO: Limit the number of state events we're about to send down
             # the room, if its too many we should change this to an
             # `initial=True`?
-            deltas = await self.get_current_state_deltas_for_room(
+            deltas = await self.store.get_current_state_deltas_for_room(
                 room_id=room_id,
-                room_membership_for_user_at_to_token=room_membership_for_user_at_to_token,
                 from_token=from_bound,
                 to_token=to_token.room_key,
             )

synapse/http/site.py

@@ -21,7 +21,6 @@
 import contextlib
 import logging
 import time
-from http import HTTPStatus
 from typing import TYPE_CHECKING, Any, Generator, Optional, Tuple, Union
 
 import attr
@@ -140,41 +139,6 @@ class SynapseRequest(Request):
             self.synapse_site.site_tag,
         )
 
-    # Twisted machinery: this method is called by the Channel once the full request has
-    # been received, to dispatch the request to a resource.
-    #
-    # We're patching Twisted to bail/abort early when we see someone trying to upload
-    # `multipart/form-data` so we can avoid Twisted parsing the entire request body into
-    # in-memory (specific problem of this specific `Content-Type`). This protects us
-    # from an attacker uploading something bigger than the available RAM and crashing
-    # the server with a `MemoryError`, or carefully block just enough resources to cause
-    # all other requests to fail.
-    #
-    # FIXME: This can be removed once we Twisted releases a fix and we update to a
-    # version that is patched
-    def requestReceived(self, command: bytes, path: bytes, version: bytes) -> None:
-        if command == b"POST":
-            ctype = self.requestHeaders.getRawHeaders(b"content-type")
-            if ctype and b"multipart/form-data" in ctype[0]:
-                self.method, self.uri = command, path
-                self.clientproto = version
-                self.code = HTTPStatus.UNSUPPORTED_MEDIA_TYPE.value
-                self.code_message = bytes(
-                    HTTPStatus.UNSUPPORTED_MEDIA_TYPE.phrase, "ascii"
-                )
-                self.responseHeaders.setRawHeaders(b"content-length", [b"0"])
-
-                logger.warning(
-                    "Aborting connection from %s because `content-type: multipart/form-data` is unsupported: %s %s",
-                    self.client,
-                    command,
-                    path,
-                )
-                self.write(b"")
-                self.loseConnection()
-                return
-        return super().requestReceived(command, path, version)
-
     def handleContentChunk(self, data: bytes) -> None:
         # we should have a `content` by now.
         assert self.content, "handleContentChunk() called before gotLength()"

synapse/media/thumbnailer.py

@@ -67,11 +67,6 @@ class ThumbnailError(Exception):
 class Thumbnailer:
     FORMATS = {"image/jpeg": "JPEG", "image/png": "PNG"}
 
-    # Which image formats we allow Pillow to open.
-    # This should intentionally be kept restrictive, because the decoder of any
-    # format in this list becomes part of our trusted computing base.
-    PILLOW_FORMATS = ("jpeg", "png", "webp", "gif")
-
     @staticmethod
     def set_limits(max_image_pixels: int) -> None:
         Image.MAX_IMAGE_PIXELS = max_image_pixels
@@ -81,7 +76,7 @@ class Thumbnailer:
         self._closed = False
 
         try:
-            self.image = Image.open(input_path, formats=self.PILLOW_FORMATS)
+            self.image = Image.open(input_path)
         except OSError as e:
             # If an error occurs opening the image, a thumbnail won't be able to
             # be generated.

synapse/push/push_tools.py

@@ -74,13 +74,9 @@ async def get_context_for_event(
 
         room_state = []
         if ev.content.get("membership") == Membership.INVITE:
-            invite_room_state = ev.unsigned.get("invite_room_state", [])
-            if isinstance(invite_room_state, list):
-                room_state = invite_room_state
+            room_state = ev.unsigned.get("invite_room_state", [])
         elif ev.content.get("membership") == Membership.KNOCK:
-            knock_room_state = ev.unsigned.get("knock_room_state", [])
-            if isinstance(knock_room_state, list):
-                room_state = knock_room_state
+            room_state = ev.unsigned.get("knock_room_state", [])
 
         # Ideally we'd reuse the logic in `calculate_room_name`, but that gets
         # complicated to handle partial events vs pulling events from the DB.

synapse/rest/client/sync.py

@@ -436,12 +436,7 @@ class SyncRestServlet(RestServlet):
             )
             unsigned = dict(invite.get("unsigned", {}))
             invite["unsigned"] = unsigned
-
-            invited_state = unsigned.pop("invite_room_state", [])
-            if not isinstance(invited_state, list):
-                invited_state = []
-
-            invited_state = list(invited_state)
+            invited_state = list(unsigned.pop("invite_room_state", []))
             invited_state.append(invite)
             invited[room.room_id] = {"invite_state": {"events": invited_state}}
 
@@ -481,10 +476,7 @@ class SyncRestServlet(RestServlet):
             # Extract the stripped room state from the unsigned dict
             # This is for clients to get a little bit of information about
             # the room they've knocked on, without revealing any sensitive information
-            knocked_state = unsigned.pop("knock_room_state", [])
-            if not isinstance(knocked_state, list):
-                knocked_state = []
-            knocked_state = list(knocked_state)
+            knocked_state = list(unsigned.pop("knock_room_state", []))
 
             # Append the actual knock membership event itself as well. This provides
             # the client with:

synapse/server.py

@@ -254,7 +254,6 @@ class HomeServer(metaclass=abc.ABCMeta):
         "auth",
         "deactivate_account",
         "delayed_events",
-        "e2e_keys",  # for the `delete_old_otks` scheduled-task handler
         "message",
         "pagination",
         "profile",

synapse/storage/databases/main/end_to_end_keys.py

@@ -220,12 +220,15 @@ class EndToEndKeyWorkerStore(EndToEndKeyBackgroundStore, CacheInvalidationWorker
         self,
         query_list: Collection[Tuple[str, Optional[str]]],
         include_displaynames: bool = True,
+        include_uploaded_unsigned_data: bool = False,
     ) -> Dict[str, Dict[str, JsonDict]]:
         """Fetch a list of device keys, formatted suitably for the C/S API.
         Args:
             query_list: List of pairs of user_ids and device_ids.
             include_displaynames: Whether to include the displayname of returned devices
                 (if one exists).
+            include_uploaded_unsigned_data: Whether to include uploaded `unsigned` data
+                in the response
         Returns:
             Dict mapping from user-id to dict mapping from device_id to
             key data.  The key data will be a dict in the same format as the
@@ -247,7 +250,13 @@ class EndToEndKeyWorkerStore(EndToEndKeyBackgroundStore, CacheInvalidationWorker
                 if r is None:
                     continue
 
-                r["unsigned"] = {}
+                # If there was already an `unsigned` dict in the uploaded key, keep it.
+                # Otherwise, create a new one.
+                if not include_uploaded_unsigned_data or not isinstance(
+                    r.get("unsigned"), dict
+                ):
+                    r["unsigned"] = {}
+
                 if include_displaynames:
                     # Include the device's display name in the "unsigned" dictionary
                     display_name = device_info.display_name

synapse/storage/databases/main/state_deltas.py

@@ -243,13 +243,6 @@ class StateDeltasStore(SQLBaseStore):
 
         (> `from_token` and <= `to_token`)
         """
-        # We can bail early if the `from_token` is after the `to_token`
-        if (
-            to_token is not None
-            and from_token is not None
-            and to_token.is_before_or_eq(from_token)
-        ):
-            return []
 
         if (
             from_token is not None

tests/http/test_site.py

@@ -90,56 +90,3 @@ class SynapseRequestTestCase(HomeserverTestCase):
         # default max upload size is 50M, so it should drop on the next buffer after
         # that.
         self.assertEqual(sent, 50 * 1024 * 1024 + 1024)
-
-    def test_content_type_multipart(self) -> None:
-        """HTTP POST requests with `content-type: multipart/form-data` should be rejected"""
-        self.hs.start_listening()
-
-        # find the HTTP server which is configured to listen on port 0
-        (port, factory, _backlog, interface) = self.reactor.tcpServers[0]
-        self.assertEqual(interface, "::")
-        self.assertEqual(port, 0)
-
-        # as a control case, first send a regular request.
-
-        # complete the connection and wire it up to a fake transport
-        client_address = IPv6Address("TCP", "::1", 2345)
-        protocol = factory.buildProtocol(client_address)
-        transport = StringTransport()
-        protocol.makeConnection(transport)
-
-        protocol.dataReceived(
-            b"POST / HTTP/1.1\r\n"
-            b"Connection: close\r\n"
-            b"Transfer-Encoding: chunked\r\n"
-            b"\r\n"
-            b"0\r\n"
-            b"\r\n"
-        )
-
-        while not transport.disconnecting:
-            self.reactor.advance(1)
-
-        # we should get a 404
-        self.assertRegex(transport.value().decode(), r"^HTTP/1\.1 404 ")
-
-        # now send request with content-type header
-        protocol = factory.buildProtocol(client_address)
-        transport = StringTransport()
-        protocol.makeConnection(transport)
-
-        protocol.dataReceived(
-            b"POST / HTTP/1.1\r\n"
-            b"Connection: close\r\n"
-            b"Transfer-Encoding: chunked\r\n"
-            b"Content-Type: multipart/form-data\r\n"
-            b"\r\n"
-            b"0\r\n"
-            b"\r\n"
-        )
-
-        while not transport.disconnecting:
-            self.reactor.advance(1)
-
-        # we should get a 415
-        self.assertRegex(transport.value().decode(), r"^HTTP/1\.1 415 ")

tests/rest/client/sliding_sync/test_rooms_required_state.py

@@ -751,10 +751,9 @@ class SlidingSyncRoomsRequiredStateTestCase(SlidingSyncBase):
         self.assertIsNone(response_body["rooms"][room_id1].get("invite_state"))
 
     @parameterized.expand([(Membership.LEAVE,), (Membership.BAN,)])
-    def test_rooms_required_state_leave_ban_initial(self, stop_membership: str) -> None:
+    def test_rooms_required_state_leave_ban(self, stop_membership: str) -> None:
         """
-        Test `rooms.required_state` should not return state past a leave/ban event when
-        it's the first "initial" time the room is being sent down the connection.
+        Test `rooms.required_state` should not return state past a leave/ban event.
         """
         user1_id = self.register_user("user1", "pass")
         user1_tok = self.login(user1_id, "pass")
@@ -789,13 +788,6 @@ class SlidingSyncRoomsRequiredStateTestCase(SlidingSyncBase):
             body={"foo": "bar"},
             tok=user2_tok,
         )
-        self.helper.send_state(
-            room_id1,
-            event_type="org.matrix.bar_state",
-            state_key="",
-            body={"bar": "bar"},
-            tok=user2_tok,
-        )
 
         if stop_membership == Membership.LEAVE:
             # User 1 leaves
@@ -804,8 +796,6 @@ class SlidingSyncRoomsRequiredStateTestCase(SlidingSyncBase):
             # User 1 is banned
             self.helper.ban(room_id1, src=user2_id, targ=user1_id, tok=user2_tok)
 
-        # Get the state_map before we change the state as this is the final state we
-        # expect User1 to be able to see
         state_map = self.get_success(
             self.storage_controllers.state.get_current_state(room_id1)
         )
@@ -818,36 +808,12 @@ class SlidingSyncRoomsRequiredStateTestCase(SlidingSyncBase):
             body={"foo": "qux"},
             tok=user2_tok,
         )
-        self.helper.send_state(
-            room_id1,
-            event_type="org.matrix.bar_state",
-            state_key="",
-            body={"bar": "qux"},
-            tok=user2_tok,
-        )
         self.helper.leave(room_id1, user3_id, tok=user3_tok)
 
         # Make an incremental Sliding Sync request
-        #
-        # Also expand the required state to include the `org.matrix.bar_state` event.
-        # This is just an extra complication of the test.
-        sync_body = {
-            "lists": {
-                "foo-list": {
-                    "ranges": [[0, 1]],
-                    "required_state": [
-                        [EventTypes.Create, ""],
-                        [EventTypes.Member, "*"],
-                        ["org.matrix.foo_state", ""],
-                        ["org.matrix.bar_state", ""],
-                    ],
-                    "timeline_limit": 3,
-                }
-            }
-        }
         response_body, _ = self.do_sync(sync_body, since=from_token, tok=user1_tok)
 
-        # We should only see the state up to the leave/ban event
+        # Only user2 and user3 sent events in the 3 events we see in the `timeline`
         self._assertRequiredStateIncludes(
             response_body["rooms"][room_id1]["required_state"],
             {
@@ -856,126 +822,6 @@ class SlidingSyncRoomsRequiredStateTestCase(SlidingSyncBase):
                 state_map[(EventTypes.Member, user2_id)],
                 state_map[(EventTypes.Member, user3_id)],
                 state_map[("org.matrix.foo_state", "")],
-                state_map[("org.matrix.bar_state", "")],
-            },
-            exact=True,
-        )
-        self.assertIsNone(response_body["rooms"][room_id1].get("invite_state"))
-
-    @parameterized.expand([(Membership.LEAVE,), (Membership.BAN,)])
-    def test_rooms_required_state_leave_ban_incremental(
-        self, stop_membership: str
-    ) -> None:
-        """
-        Test `rooms.required_state` should not return state past a leave/ban event on
-        incremental sync.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-        user3_id = self.register_user("user3", "pass")
-        user3_tok = self.login(user3_id, "pass")
-
-        room_id1 = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.join(room_id1, user1_id, tok=user1_tok)
-        self.helper.join(room_id1, user3_id, tok=user3_tok)
-
-        self.helper.send_state(
-            room_id1,
-            event_type="org.matrix.foo_state",
-            state_key="",
-            body={"foo": "bar"},
-            tok=user2_tok,
-        )
-        self.helper.send_state(
-            room_id1,
-            event_type="org.matrix.bar_state",
-            state_key="",
-            body={"bar": "bar"},
-            tok=user2_tok,
-        )
-
-        sync_body = {
-            "lists": {
-                "foo-list": {
-                    "ranges": [[0, 1]],
-                    "required_state": [
-                        [EventTypes.Create, ""],
-                        [EventTypes.Member, "*"],
-                        ["org.matrix.foo_state", ""],
-                    ],
-                    "timeline_limit": 3,
-                }
-            }
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        if stop_membership == Membership.LEAVE:
-            # User 1 leaves
-            self.helper.leave(room_id1, user1_id, tok=user1_tok)
-        elif stop_membership == Membership.BAN:
-            # User 1 is banned
-            self.helper.ban(room_id1, src=user2_id, targ=user1_id, tok=user2_tok)
-
-        # Get the state_map before we change the state as this is the final state we
-        # expect User1 to be able to see
-        state_map = self.get_success(
-            self.storage_controllers.state.get_current_state(room_id1)
-        )
-
-        # Change the state after user 1 leaves
-        self.helper.send_state(
-            room_id1,
-            event_type="org.matrix.foo_state",
-            state_key="",
-            body={"foo": "qux"},
-            tok=user2_tok,
-        )
-        self.helper.send_state(
-            room_id1,
-            event_type="org.matrix.bar_state",
-            state_key="",
-            body={"bar": "qux"},
-            tok=user2_tok,
-        )
-        self.helper.leave(room_id1, user3_id, tok=user3_tok)
-
-        # Make an incremental Sliding Sync request
-        #
-        # Also expand the required state to include the `org.matrix.bar_state` event.
-        # This is just an extra complication of the test.
-        sync_body = {
-            "lists": {
-                "foo-list": {
-                    "ranges": [[0, 1]],
-                    "required_state": [
-                        [EventTypes.Create, ""],
-                        [EventTypes.Member, "*"],
-                        ["org.matrix.foo_state", ""],
-                        ["org.matrix.bar_state", ""],
-                    ],
-                    "timeline_limit": 3,
-                }
-            }
-        }
-        response_body, _ = self.do_sync(sync_body, since=from_token, tok=user1_tok)
-
-        # User1 should only see the state up to the leave/ban event
-        self._assertRequiredStateIncludes(
-            response_body["rooms"][room_id1]["required_state"],
-            {
-                # User1 should see their leave/ban membership
-                state_map[(EventTypes.Member, user1_id)],
-                state_map[("org.matrix.bar_state", "")],
-                # The commented out state events were already returned in the initial
-                # sync so we shouldn't see them again on the incremental sync. And we
-                # shouldn't see the state events that changed after the leave/ban event.
-                #
-                # state_map[(EventTypes.Create, "")],
-                # state_map[(EventTypes.Member, user2_id)],
-                # state_map[(EventTypes.Member, user3_id)],
-                # state_map[("org.matrix.foo_state", "")],
             },
             exact=True,
         )

tests/rest/client/test_keys.py

@@ -19,6 +19,7 @@
 #
 #
 import urllib.parse
+from copy import deepcopy
 from http import HTTPStatus
 from unittest.mock import patch
 
@@ -205,6 +206,141 @@ class KeyQueryTestCase(unittest.HomeserverTestCase):
         self.assertEqual(channel.code, HTTPStatus.OK, channel.result)
 
 
+class UnsignedKeyDataTestCase(unittest.HomeserverTestCase):
+    servlets = [
+        keys.register_servlets,
+        admin.register_servlets_for_client_rest_resource,
+        login.register_servlets,
+    ]
+
+    def default_config(self) -> JsonDict:
+        config = super().default_config()
+        config["experimental_features"] = {"msc4229_enabled": True}
+        return config
+
+    def make_key_data(self, user_id: str, device_id: str) -> JsonDict:
+        return {
+            "algorithms": ["m.olm.v1.curve25519-aes-sha2", "m.megolm.v1.aes-sha2"],
+            "device_id": device_id,
+            "keys": {
+                f"curve25519:{device_id}": "keykeykey",
+                f"ed25519:{device_id}": "keykeykey",
+            },
+            "signatures": {user_id: {f"ed25519:{device_id}": "sigsigsig"}},
+            "user_id": user_id,
+        }
+
+    def test_unsigned_uploaded_data_returned_in_keys_query(self) -> None:
+        password = "wonderland"
+        device_id = "ABCDEFGHI"
+        alice_id = self.register_user("alice", password)
+        alice_token = self.login(
+            "alice",
+            password,
+            device_id=device_id,
+            additional_request_fields={"initial_device_display_name": "mydevice"},
+        )
+
+        # Alice uploads some keys, with a bit of unsigned data
+        keys1 = self.make_key_data(alice_id, device_id)
+        keys1["unsigned"] = {"a": "b"}
+
+        channel = self.make_request(
+            "POST",
+            "/_matrix/client/v3/keys/upload",
+            {"device_keys": keys1},
+            alice_token,
+        )
+        self.assertEqual(channel.code, HTTPStatus.OK, channel.result)
+
+        # /keys/query should return the unsigned data, with the device display name merged in.
+        channel = self.make_request(
+            "POST",
+            "/_matrix/client/v3/keys/query",
+            {"device_keys": {alice_id: []}},
+            alice_token,
+        )
+        self.assertEqual(channel.code, HTTPStatus.OK, channel.result)
+        device_response = channel.json_body["device_keys"][alice_id][device_id]
+        expected_device_response = deepcopy(keys1)
+        expected_device_response["unsigned"]["device_display_name"] = "mydevice"
+        self.assertEqual(device_response, expected_device_response)
+
+        # /_matrix/federation/v1/user/devices/{userId} should return the unsigned data too
+        fed_response = self.get_success(
+            self.hs.get_device_handler().on_federation_query_user_devices(alice_id)
+        )
+        self.assertEqual(
+            fed_response["devices"][0],
+            {"device_id": device_id, "keys": keys1},
+        )
+
+        # so should /_matrix/federation/v1/user/keys/query
+        fed_response = self.get_success(
+            self.hs.get_e2e_keys_handler().on_federation_query_client_keys(
+                {"device_keys": {alice_id: []}}
+            )
+        )
+        fed_device_response = fed_response["device_keys"][alice_id][device_id]
+        self.assertEqual(fed_device_response, keys1)
+
+    def test_non_dict_unsigned_is_ignored(self) -> None:
+        password = "wonderland"
+        device_id = "ABCDEFGHI"
+        alice_id = self.register_user("alice", password)
+        alice_token = self.login(
+            "alice",
+            password,
+            device_id=device_id,
+            additional_request_fields={"initial_device_display_name": "mydevice"},
+        )
+
+        # Alice uploads some keys, with a malformed unsigned data
+        keys1 = self.make_key_data(alice_id, device_id)
+        keys1["unsigned"] = ["a", "b"]  # a list!
+
+        channel = self.make_request(
+            "POST",
+            "/_matrix/client/v3/keys/upload",
+            {"device_keys": keys1},
+            alice_token,
+        )
+        self.assertEqual(channel.code, HTTPStatus.OK, channel.result)
+
+        # /keys/query should return the unsigned data, with the device display name merged in.
+        channel = self.make_request(
+            "POST",
+            "/_matrix/client/v3/keys/query",
+            {"device_keys": {alice_id: []}},
+            alice_token,
+        )
+        self.assertEqual(channel.code, HTTPStatus.OK, channel.result)
+        device_response = channel.json_body["device_keys"][alice_id][device_id]
+        expected_device_response = deepcopy(keys1)
+        expected_device_response["unsigned"] = {"device_display_name": "mydevice"}
+        self.assertEqual(device_response, expected_device_response)
+
+        # /_matrix/federation/v1/user/devices/{userId} should return the unsigned data too
+        fed_response = self.get_success(
+            self.hs.get_device_handler().on_federation_query_user_devices(alice_id)
+        )
+        self.assertEqual(
+            fed_response["devices"][0],
+            {"device_id": device_id, "keys": keys1},
+        )
+
+        # so should /_matrix/federation/v1/user/keys/query
+        fed_response = self.get_success(
+            self.hs.get_e2e_keys_handler().on_federation_query_client_keys(
+                {"device_keys": {alice_id: []}}
+            )
+        )
+        fed_device_response = fed_response["device_keys"][alice_id][device_id]
+        expected_device_response = deepcopy(keys1)
+        expected_device_response["unsigned"] = {}
+        self.assertEqual(fed_device_response, expected_device_response)
+
+
 class SigningKeyUploadServletTestCase(unittest.HomeserverTestCase):
     servlets = [
         admin.register_servlets,