WIP: MAS in complement

CHANGES.md

@@ -1,54 +1,3 @@
-# Synapse 1.113.0 (2024-08-13)
-
-No significant changes since 1.113.0rc1.
-
-
-
-
-# Synapse 1.113.0rc1 (2024-08-06)
-
-### Features
-
-- Track which rooms have been sent to clients in the experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint. ([\#17447](https://github.com/element-hq/synapse/issues/17447))
-- Add Account Data extension support to experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint. ([\#17477](https://github.com/element-hq/synapse/issues/17477))
-- Add receipts extension support to experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint. ([\#17489](https://github.com/element-hq/synapse/issues/17489))
-- Add typing notification extension support to experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint. ([\#17505](https://github.com/element-hq/synapse/issues/17505))
-
-### Bugfixes
-
-- Update experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint to handle invite/knock rooms when filtering. ([\#17450](https://github.com/element-hq/synapse/issues/17450))
-- Fix a bug introduced in v1.110.0 which caused `/keys/query` to return incomplete results, leading to high network activity and CPU usage on Matrix clients. ([\#17499](https://github.com/element-hq/synapse/issues/17499))
-
-### Improved Documentation
-
-- Update the [`allowed_local_3pids`](https://element-hq.github.io/synapse/v1.112/usage/configuration/config_documentation.html#allowed_local_3pids) config option's msisdn address to a working example. ([\#17476](https://github.com/element-hq/synapse/issues/17476))
-
-### Internal Changes
-
-- Change sliding sync to use their own token format in preparation for storing per-connection state. ([\#17452](https://github.com/element-hq/synapse/issues/17452))
-- Ensure we don't send down negative `bump_stamp` in experimental sliding sync endpoint. ([\#17478](https://github.com/element-hq/synapse/issues/17478))
-- Do not send down empty room entries down experimental sliding sync endpoint. ([\#17479](https://github.com/element-hq/synapse/issues/17479))
-- Refactor Sliding Sync tests to better utilize the `SlidingSyncBase`. ([\#17481](https://github.com/element-hq/synapse/issues/17481), [\#17482](https://github.com/element-hq/synapse/issues/17482))
-- Add some opentracing tags and logging to the experimental sliding sync implementation. ([\#17501](https://github.com/element-hq/synapse/issues/17501))
-- Split and move Sliding Sync tests so we have some more sane test file sizes. ([\#17504](https://github.com/element-hq/synapse/issues/17504))
-- Update the `limited` field description in the Sliding Sync response to accurately describe what it actually represents. ([\#17507](https://github.com/element-hq/synapse/issues/17507))
-- Easier to understand `timeline` assertions in Sliding Sync tests. ([\#17511](https://github.com/element-hq/synapse/issues/17511))
-- Reset the sliding sync connection if we don't recognize the per-connection state position. ([\#17529](https://github.com/element-hq/synapse/issues/17529))
-
-
-
-### Updates to locked dependencies
-
-* Bump bcrypt from 4.1.3 to 4.2.0. ([\#17495](https://github.com/element-hq/synapse/issues/17495))
-* Bump black from 24.4.2 to 24.8.0. ([\#17522](https://github.com/element-hq/synapse/issues/17522))
-* Bump phonenumbers from 8.13.39 to 8.13.42. ([\#17521](https://github.com/element-hq/synapse/issues/17521))
-* Bump ruff from 0.5.4 to 0.5.5. ([\#17494](https://github.com/element-hq/synapse/issues/17494))
-* Bump serde_json from 1.0.120 to 1.0.121. ([\#17493](https://github.com/element-hq/synapse/issues/17493))
-* Bump serde_json from 1.0.121 to 1.0.122. ([\#17525](https://github.com/element-hq/synapse/issues/17525))
-* Bump towncrier from 23.11.0 to 24.7.1. ([\#17523](https://github.com/element-hq/synapse/issues/17523))
-* Bump types-pyopenssl from 24.1.0.20240425 to 24.1.0.20240722. ([\#17496](https://github.com/element-hq/synapse/issues/17496))
-* Bump types-setuptools from 70.1.0.20240627 to 71.1.0.20240726. ([\#17497](https://github.com/element-hq/synapse/issues/17497))
-
 # Synapse 1.112.0 (2024-07-30)
 
 This security release is to update our locked dependency on Twisted to 24.7.0rc1, which includes a security fix for [CVE-2024-41671 / GHSA-c8m8-j448-xjx7: Disordered HTTP pipeline response in twisted.web, again](https://github.com/twisted/twisted/security/advisories/GHSA-c8m8-j448-xjx7).

Cargo.lock

@@ -505,9 +505,9 @@ dependencies = [
 
 [[package]]
 name = "serde_json"
-version = "1.0.122"
+version = "1.0.121"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "784b6203951c57ff748476b126ccb5e8e2959a5c19e5c617ab1956be3dbc68da"
+checksum = "4ab380d7d9f22ef3f21ad3e6c1ebe8e4fc7a2000ccba2e4d71fc96f15b2cb609"
 dependencies = [
  "itoa",
  "memchr",

changelog.d/17447.feature

@@ -0,0 +1 @@
+Track which rooms have been sent to clients in the experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint.

changelog.d/17450.bugfix

@@ -0,0 +1 @@
+Update experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint to handle invite/knock rooms when filtering.

changelog.d/17452.misc

@@ -0,0 +1 @@
+Change sliding sync to use their own token format in preparation for storing per-connection state.

changelog.d/17476.doc

@@ -0,0 +1 @@
+Update the [`allowed_local_3pids`](https://element-hq.github.io/synapse/v1.112/usage/configuration/config_documentation.html#allowed_local_3pids) config option's msisdn address to a working example.

changelog.d/17477.feature

@@ -0,0 +1 @@
+Add Account Data extension support to experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint.

changelog.d/17478.misc

@@ -0,0 +1 @@
+Ensure we don't send down negative `bump_stamp` in experimental sliding sync endpoint.

changelog.d/17479.misc

@@ -0,0 +1 @@
+Do not send down empty room entries down experimental sliding sync endpoint.

changelog.d/17481.misc

@@ -0,0 +1 @@
+Refactor Sliding Sync tests to better utilize the `SlidingSyncBase`.

changelog.d/17482.misc

@@ -0,0 +1 @@
+Refactor Sliding Sync tests to better utilize the `SlidingSyncBase`.

changelog.d/17489.feature

@@ -0,0 +1 @@
+Add receipts extension support to experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint.

changelog.d/17499.bugfix

@@ -0,0 +1 @@
+Fix a bug introduced in v1.110.0 which caused `/keys/query` to return incomplete results, leading to high network activity and CPU usage on Matrix clients.

changelog.d/17501.misc

@@ -0,0 +1 @@
+Add some opentracing tags and logging to the experimental sliding sync implementation.

changelog.d/17504.misc

@@ -0,0 +1 @@
+Split and move Sliding Sync tests so we have some more sane test file sizes.

changelog.d/17505.feature

@@ -0,0 +1 @@
+Add typing notification extension support to experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint.

changelog.d/17507.misc

@@ -0,0 +1 @@
+Update the `limited` field description in the Sliding Sync response to accurately describe what it actually represents.

debian/changelog

@@ -1,15 +1,3 @@
-matrix-synapse-py3 (1.113.0) stable; urgency=medium
-
-  * New Synapse release 1.113.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 13 Aug 2024 14:36:56 +0100
-
-matrix-synapse-py3 (1.113.0~rc1) stable; urgency=medium
-
-  * New Synapse release 1.113.0rc1.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 06 Aug 2024 12:23:23 +0100
-
 matrix-synapse-py3 (1.112.0) stable; urgency=medium
 
   * New Synapse release 1.112.0.

docker/Dockerfile-workers

@@ -1,67 +1,62 @@
 # syntax=docker/dockerfile:1
 
 ARG SYNAPSE_VERSION=latest
-ARG FROM=matrixdotorg/synapse:$SYNAPSE_VERSION
+ARG SYNAPSE_IMAGE=docker.io/matrixdotorg/synapse:$SYNAPSE_VERSION
 
-# first of all, we create a base image with an nginx which we can copy into the
-# target image. For repeated rebuilds, this is much faster than apt installing
-# each time.
+ARG MAS_VERSION=latest
+ARG MAS_IMAGE=ghcr.io/matrix-org/matrix-authentication-service:$MAS_VERSION
 
-FROM docker.io/library/debian:bookworm-slim AS deps_base
-    RUN \
-       --mount=type=cache,target=/var/cache/apt,sharing=locked \
-       --mount=type=cache,target=/var/lib/apt,sharing=locked \
-      apt-get update -qq && \
-      DEBIAN_FRONTEND=noninteractive apt-get install -yqq --no-install-recommends \
-          redis-server nginx-light
+ARG REDIS_VERSION=7.4.0
+ARG REDIS_IMAGE=docker.io/library/redis:$REDIS_VERSION-bookworm
 
-# Similarly, a base to copy the redis server from.
-#
-# The redis docker image has fewer dynamic libraries than the debian package,
-# which makes it much easier to copy (but we need to make sure we use an image
-# based on the same debian version as the synapse image, to make sure we get
-# the expected version of libc.
-FROM docker.io/library/redis:7-bookworm AS redis_base
+ARG NGINX_VERSION=1.26.1
+ARG NGINX_IMAGE=docker.io/library/nginx:$NGINX_VERSION-bookworm
+
+FROM $NGINX_IMAGE AS nginx
+FROM $REDIS_IMAGE AS redis
+FROM $MAS_IMAGE AS mas
 
 # now build the final image, based on the the regular Synapse docker image
-FROM $FROM
+FROM $SYNAPSE_IMAGE
 
-    # Install supervisord with pip instead of apt, to avoid installing a second
-    # copy of python.
-    RUN --mount=type=cache,target=/root/.cache/pip \
-        pip install supervisor~=4.2
-    RUN mkdir -p /etc/supervisor/conf.d
+# Install supervisord with pip instead of apt, to avoid installing a second
+# copy of python.
+RUN --mount=type=cache,target=/root/.cache/pip \
+    pip install supervisor~=4.2
+RUN mkdir -p /etc/supervisor/conf.d
 
-    # Copy over redis and nginx
-    COPY --from=redis_base /usr/local/bin/redis-server /usr/local/bin
+# Copy over redis, nginx and matrix-authentication-service
+COPY --from=redis /usr/local/bin/redis-server /usr/local/bin
 
-    COPY --from=deps_base /usr/sbin/nginx /usr/sbin
-    COPY --from=deps_base /usr/share/nginx /usr/share/nginx
-    COPY --from=deps_base /usr/lib/nginx /usr/lib/nginx
-    COPY --from=deps_base /etc/nginx /etc/nginx
-    RUN rm /etc/nginx/sites-enabled/default
-    RUN mkdir /var/log/nginx /var/lib/nginx
-    RUN chown www-data /var/lib/nginx
+COPY --from=nginx /usr/sbin/nginx /usr/sbin
+COPY --from=nginx /usr/share/nginx /usr/share/nginx
+COPY --from=nginx /usr/lib/nginx /usr/lib/nginx
+COPY --from=nginx /etc/nginx /etc/nginx
+RUN mkdir /var/log/nginx /var/lib/nginx
+RUN chown www-data /var/lib/nginx
 
-    # have nginx log to stderr/out
-    RUN ln -sf /dev/stdout /var/log/nginx/access.log
-    RUN ln -sf /dev/stderr /var/log/nginx/error.log
+# have nginx log to stderr/out
+RUN ln -sf /dev/stdout /var/log/nginx/access.log
+RUN ln -sf /dev/stderr /var/log/nginx/error.log
 
-    # Copy Synapse worker, nginx and supervisord configuration template files
-    COPY ./docker/conf-workers/* /conf/
+COPY --from=mas /usr/local/bin/mas-cli /usr/local/bin
+COPY --from=mas /usr/local/share/mas-cli /usr/local/share
 
-    # Copy a script to prefix log lines with the supervisor program name
-    COPY ./docker/prefix-log /usr/local/bin/
+# Copy Synapse worker, nginx and supervisord configuration template files
+COPY ./docker/conf-workers/* /conf/
 
-    # Expose nginx listener port
-    EXPOSE 8080/tcp
+# Copy a script to prefix log lines with the supervisor program name
+COPY ./docker/prefix-log /usr/local/bin/
 
-    # A script to read environment variables and create the necessary
-    # files to run the desired worker configuration. Will start supervisord.
-    COPY ./docker/configure_workers_and_start.py /configure_workers_and_start.py
-    ENTRYPOINT ["/configure_workers_and_start.py"]
+# Expose nginx listener port
+EXPOSE 8080/tcp
 
-    # Replace the healthcheck with one which checks *all* the workers. The script
-    # is generated by configure_workers_and_start.py.
-    HEALTHCHECK --start-period=5s --interval=15s --timeout=5s \
-        CMD /bin/sh /healthcheck.sh
+# A script to read environment variables and create the necessary
+# files to run the desired worker configuration. Will start supervisord.
+COPY ./docker/configure_workers_and_start.py /configure_workers_and_start.py
+ENTRYPOINT ["/configure_workers_and_start.py"]
+
+# Replace the healthcheck with one which checks *all* the workers. The script
+# is generated by configure_workers_and_start.py.
+HEALTHCHECK --start-period=5s --interval=15s --timeout=5s \
+    CMD /bin/sh /healthcheck.sh

docker/complement/Dockerfile

@@ -6,11 +6,17 @@
 # Instructions for building this image from those it depends on is detailed in this guide:
 # https://github.com/element-hq/synapse/blob/develop/docker/README-testing.md#testing-with-postgresql-and-single-or-multi-process-synapse
 
-ARG SYNAPSE_VERSION=latest
 # This is an intermediate image, to be built locally (not pulled from a registry).
-ARG FROM=matrixdotorg/synapse-workers:$SYNAPSE_VERSION
+ARG SYNAPSE_WORKERS_IMAGE=synapse-workers
+
+ARG POSTGRES_VERSION=13
+ARG POSTGRES_IMAGE=docker.io/library/postgres:$POSTGRES_VERSION-bookworm
+
+# Save the Postgres image for later
+FROM $POSTGRES_IMAGE AS postgres
+
+FROM $SYNAPSE_WORKERS_IMAGE
 
-FROM $FROM
 # First of all, we copy postgres server from the official postgres image,
 # since for repeated rebuilds, this is much faster than apt installing
 # postgres each time.
@@ -20,8 +26,8 @@ FROM $FROM
 # the same debian version as Synapse's docker image (so the versions of the
 # shared libraries match).
 RUN adduser --system --uid 999 postgres --home /var/lib/postgresql
-COPY --from=docker.io/library/postgres:13-bookworm /usr/lib/postgresql /usr/lib/postgresql
-COPY --from=docker.io/library/postgres:13-bookworm /usr/share/postgresql /usr/share/postgresql
+COPY --from=postgres /usr/lib/postgresql /usr/lib/postgresql
+COPY --from=postgres /usr/share/postgresql /usr/share/postgresql
 RUN mkdir /var/run/postgresql && chown postgres /var/run/postgresql
 ENV PATH="${PATH}:/usr/lib/postgresql/13/bin"
 ENV PGDATA=/var/lib/postgresql/data
@@ -29,9 +35,10 @@ ENV PGDATA=/var/lib/postgresql/data
 # We also initialize the database at build time, rather than runtime, so that it's faster to spin up the image.
 RUN gosu postgres initdb --locale=C --encoding=UTF-8 --auth-host password
 
-# Configure a password and create a database for Synapse
+# Configure a password and create a database for Synapse and MAS
 RUN echo "ALTER USER postgres PASSWORD 'somesecret'" | gosu postgres postgres --single
 RUN echo "CREATE DATABASE synapse" | gosu postgres postgres --single
+RUN echo "CREATE DATABASE mas" | gosu postgres postgres --single
 
 # Extend the shared homeserver config to disable rate-limiting,
 # set Complement's static shared secret, enable registration, amongst other

docker/conf-workers/shared.yaml.j2

@@ -20,4 +20,15 @@ app_service_config_files:
 {%- endfor %}
 {%- endif %}
 
+{% if enable_mas %}
+experimental_features:
+  msc3861:
+    enabled: true
+    issuer: "http://localhost:8008/"
+    client_id: "0000000000000000000SYNAPSE"
+    client_auth_method: client_secret_basic
+    client_secret: choozia3ThiefahZaofeiveish1kahr0
+    admin_token: eeShoo4ceebae4Lo4Che1hoofoophaiz
+{% endif %}
+
 {{ shared_worker_config }}

docker/conf-workers/supervisord.conf.j2

@@ -35,3 +35,12 @@ autorestart=true
 # Redis can be disabled if the image is being used without workers
 autostart={{ enable_redis }}
 
+[program:mas]
+comamnd=/usr/local/bin/prefix-log /usr/local/bin/mas-cli --config /conf/mas.yaml
+stdout_logfile=/dev/stdout
+stdout_logfile_maxbytes=0
+stderr_logfile=/dev/stderr
+stderr_logfile_maxbytes=0
+autorestart=unexpected
+
+autostart={{ enable_mas }}

docker/configure_workers_and_start.py

@@ -959,6 +959,7 @@ def generate_worker_files(
         shared_worker_config=yaml.dump(shared_config),
         appservice_registrations=appservice_registrations,
         enable_redis=workers_in_use,
+        enable_mas=False,
         workers_in_use=workers_in_use,
         using_unix_sockets=using_unix_sockets,
     )
@@ -981,6 +982,7 @@ def generate_worker_files(
         "/etc/supervisor/supervisord.conf",
         main_config_path=config_path,
         enable_redis=workers_in_use,
+        enable_mas=False,
         using_unix_sockets=using_unix_sockets,
     )
 

poetry.lock

@@ -1,4 +1,4 @@
-# This file is automatically @generated by Poetry 1.8.3 and should not be changed by hand.
+# This file is automatically @generated by Poetry 1.5.1 and should not be changed by hand.
 
 [[package]]
 name = "annotated-types"
@@ -107,33 +107,33 @@ typecheck = ["mypy"]
 
 [[package]]
 name = "black"
-version = "24.8.0"
+version = "24.4.2"
 description = "The uncompromising code formatter."
 optional = false
 python-versions = ">=3.8"
 files = [
-    {file = "black-24.8.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:09cdeb74d494ec023ded657f7092ba518e8cf78fa8386155e4a03fdcc44679e6"},
-    {file = "black-24.8.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:81c6742da39f33b08e791da38410f32e27d632260e599df7245cccee2064afeb"},
-    {file = "black-24.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:707a1ca89221bc8a1a64fb5e15ef39cd755633daa672a9db7498d1c19de66a42"},
-    {file = "black-24.8.0-cp310-cp310-win_amd64.whl", hash = "sha256:d6417535d99c37cee4091a2f24eb2b6d5ec42b144d50f1f2e436d9fe1916fe1a"},
-    {file = "black-24.8.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:fb6e2c0b86bbd43dee042e48059c9ad7830abd5c94b0bc518c0eeec57c3eddc1"},
-    {file = "black-24.8.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:837fd281f1908d0076844bc2b801ad2d369c78c45cf800cad7b61686051041af"},
-    {file = "black-24.8.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:62e8730977f0b77998029da7971fa896ceefa2c4c4933fcd593fa599ecbf97a4"},
-    {file = "black-24.8.0-cp311-cp311-win_amd64.whl", hash = "sha256:72901b4913cbac8972ad911dc4098d5753704d1f3c56e44ae8dce99eecb0e3af"},
-    {file = "black-24.8.0-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:7c046c1d1eeb7aea9335da62472481d3bbf3fd986e093cffd35f4385c94ae368"},
-    {file = "black-24.8.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:649f6d84ccbae73ab767e206772cc2d7a393a001070a4c814a546afd0d423aed"},
-    {file = "black-24.8.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2b59b250fdba5f9a9cd9d0ece6e6d993d91ce877d121d161e4698af3eb9c1018"},
-    {file = "black-24.8.0-cp312-cp312-win_amd64.whl", hash = "sha256:6e55d30d44bed36593c3163b9bc63bf58b3b30e4611e4d88a0c3c239930ed5b2"},
-    {file = "black-24.8.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:505289f17ceda596658ae81b61ebbe2d9b25aa78067035184ed0a9d855d18afd"},
-    {file = "black-24.8.0-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:b19c9ad992c7883ad84c9b22aaa73562a16b819c1d8db7a1a1a49fb7ec13c7d2"},
-    {file = "black-24.8.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1f13f7f386f86f8121d76599114bb8c17b69d962137fc70efe56137727c7047e"},
-    {file = "black-24.8.0-cp38-cp38-win_amd64.whl", hash = "sha256:f490dbd59680d809ca31efdae20e634f3fae27fba3ce0ba3208333b713bc3920"},
-    {file = "black-24.8.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:eab4dd44ce80dea27dc69db40dab62d4ca96112f87996bca68cd75639aeb2e4c"},
-    {file = "black-24.8.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:3c4285573d4897a7610054af5a890bde7c65cb466040c5f0c8b732812d7f0e5e"},
-    {file = "black-24.8.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9e84e33b37be070ba135176c123ae52a51f82306def9f7d063ee302ecab2cf47"},
-    {file = "black-24.8.0-cp39-cp39-win_amd64.whl", hash = "sha256:73bbf84ed136e45d451a260c6b73ed674652f90a2b3211d6a35e78054563a9bb"},
-    {file = "black-24.8.0-py3-none-any.whl", hash = "sha256:972085c618ee94f402da1af548a4f218c754ea7e5dc70acb168bfaca4c2542ed"},
-    {file = "black-24.8.0.tar.gz", hash = "sha256:2500945420b6784c38b9ee885af039f5e7471ef284ab03fa35ecdde4688cd83f"},
+    {file = "black-24.4.2-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:dd1b5a14e417189db4c7b64a6540f31730713d173f0b63e55fabd52d61d8fdce"},
+    {file = "black-24.4.2-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:8e537d281831ad0e71007dcdcbe50a71470b978c453fa41ce77186bbe0ed6021"},
+    {file = "black-24.4.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:eaea3008c281f1038edb473c1aa8ed8143a5535ff18f978a318f10302b254063"},
+    {file = "black-24.4.2-cp310-cp310-win_amd64.whl", hash = "sha256:7768a0dbf16a39aa5e9a3ded568bb545c8c2727396d063bbaf847df05b08cd96"},
+    {file = "black-24.4.2-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:257d724c2c9b1660f353b36c802ccece186a30accc7742c176d29c146df6e474"},
+    {file = "black-24.4.2-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:bdde6f877a18f24844e381d45e9947a49e97933573ac9d4345399be37621e26c"},
+    {file = "black-24.4.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e151054aa00bad1f4e1f04919542885f89f5f7d086b8a59e5000e6c616896ffb"},
+    {file = "black-24.4.2-cp311-cp311-win_amd64.whl", hash = "sha256:7e122b1c4fb252fd85df3ca93578732b4749d9be076593076ef4d07a0233c3e1"},
+    {file = "black-24.4.2-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:accf49e151c8ed2c0cdc528691838afd217c50412534e876a19270fea1e28e2d"},
+    {file = "black-24.4.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:88c57dc656038f1ab9f92b3eb5335ee9b021412feaa46330d5eba4e51fe49b04"},
+    {file = "black-24.4.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:be8bef99eb46d5021bf053114442914baeb3649a89dc5f3a555c88737e5e98fc"},
+    {file = "black-24.4.2-cp312-cp312-win_amd64.whl", hash = "sha256:415e686e87dbbe6f4cd5ef0fbf764af7b89f9057b97c908742b6008cc554b9c0"},
+    {file = "black-24.4.2-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:bf10f7310db693bb62692609b397e8d67257c55f949abde4c67f9cc574492cc7"},
+    {file = "black-24.4.2-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:98e123f1d5cfd42f886624d84464f7756f60ff6eab89ae845210631714f6db94"},
+    {file = "black-24.4.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:48a85f2cb5e6799a9ef05347b476cce6c182d6c71ee36925a6c194d074336ef8"},
+    {file = "black-24.4.2-cp38-cp38-win_amd64.whl", hash = "sha256:b1530ae42e9d6d5b670a34db49a94115a64596bc77710b1d05e9801e62ca0a7c"},
+    {file = "black-24.4.2-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:37aae07b029fa0174d39daf02748b379399b909652a806e5708199bd93899da1"},
+    {file = "black-24.4.2-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:da33a1a5e49c4122ccdfd56cd021ff1ebc4a1ec4e2d01594fef9b6f267a9e741"},
+    {file = "black-24.4.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ef703f83fc32e131e9bcc0a5094cfe85599e7109f896fe8bc96cc402f3eb4b6e"},
+    {file = "black-24.4.2-cp39-cp39-win_amd64.whl", hash = "sha256:b9176b9832e84308818a99a561e90aa479e73c523b3f77afd07913380ae2eab7"},
+    {file = "black-24.4.2-py3-none-any.whl", hash = "sha256:d36ed1124bb81b32f8614555b34cc4259c3fbc7eec17870e8ff8ded335b58d8c"},
+    {file = "black-24.4.2.tar.gz", hash = "sha256:c872b53057f000085da66a19c55d68f6f8ddcac2642392ad3a355878406fbd4d"},
 ]
 
 [package.dependencies]
@@ -1516,13 +1516,13 @@ files = [
 
 [[package]]
 name = "phonenumbers"
-version = "8.13.42"
+version = "8.13.39"
 description = "Python version of Google's common library for parsing, formatting, storing and validating international phone numbers."
 optional = false
 python-versions = "*"
 files = [
-    {file = "phonenumbers-8.13.42-py2.py3-none-any.whl", hash = "sha256:18acc22ee03116d27b26e990f53806a1770a3e05f05e1620bc09ad187f889456"},
-    {file = "phonenumbers-8.13.42.tar.gz", hash = "sha256:7137904f2db3b991701e853174ce8e1cb8f540b8bfdf27617540de04c0b7bed5"},
+    {file = "phonenumbers-8.13.39-py2.py3-none-any.whl", hash = "sha256:3ad2d086fa71e7eef409001b9195ac54bebb0c6e3e752209b558ca192c9229a0"},
+    {file = "phonenumbers-8.13.39.tar.gz", hash = "sha256:db7ca4970d206b2056231105300753b1a5b229f43416f8c2b3010e63fbb68d77"},
 ]
 
 [[package]]
@@ -2649,24 +2649,24 @@ files = [
 
 [[package]]
 name = "towncrier"
-version = "24.7.1"
+version = "23.11.0"
 description = "Building newsfiles for your project."
 optional = false
 python-versions = ">=3.8"
 files = [
-    {file = "towncrier-24.7.1-py3-none-any.whl", hash = "sha256:685e2a94335b5dc47537b4d3b449a25b18571ea85b07dcf6e8df31ba40f692dd"},
-    {file = "towncrier-24.7.1.tar.gz", hash = "sha256:57a057faedabcadf1a62f6f9bad726ae566c1f31a411338ddb8316993f583b3d"},
+    {file = "towncrier-23.11.0-py3-none-any.whl", hash = "sha256:2e519ca619426d189e3c98c99558fe8be50c9ced13ea1fc20a4a353a95d2ded7"},
+    {file = "towncrier-23.11.0.tar.gz", hash = "sha256:13937c247e3f8ae20ac44d895cf5f96a60ad46cfdcc1671759530d7837d9ee5d"},
 ]
 
 [package.dependencies]
 click = "*"
-importlib-metadata = {version = ">=4.6", markers = "python_version < \"3.10\""}
 importlib-resources = {version = ">=5", markers = "python_version < \"3.10\""}
+incremental = "*"
 jinja2 = "*"
 tomli = {version = "*", markers = "python_version < \"3.11\""}
 
 [package.extras]
-dev = ["furo (>=2024.05.06)", "nox", "packaging", "sphinx (>=5)", "twisted"]
+dev = ["furo", "packaging", "sphinx (>=5)", "twisted"]
 
 [[package]]
 name = "treq"
@@ -3196,4 +3196,4 @@ user-search = ["pyicu"]
 [metadata]
 lock-version = "2.0"
 python-versions = "^3.8.0"
-content-hash = "c165cdc1f6612c9f1b5bfd8063c23e2d595d717dd8ac1a468519e902be2cdf93"
+content-hash = "5f458ce53b7469844af2e0c5a9c5ef720736de5f080c4eb8d3a0e60286424f44"

pyproject.toml

@@ -97,7 +97,7 @@ module-name = "synapse.synapse_rust"
 
 [tool.poetry]
 name = "matrix-synapse"
-version = "1.113.0"
+version = "1.112.0"
 description = "Homeserver for the Matrix decentralised comms protocol"
 authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
 license = "AGPL-3.0-or-later"

scripts-dev/complement.sh

@@ -167,11 +167,11 @@ if [ -z "$skip_docker_build" ]; then
             -f "docker/editable.Dockerfile" .
 
         $CONTAINER_RUNTIME build -t synapse-workers-editable \
-            --build-arg FROM=synapse-editable \
+            --build-arg SYNAPSE_IMAGE=synapse-editable \
             -f "docker/Dockerfile-workers" .
 
         $CONTAINER_RUNTIME build -t complement-synapse-editable \
-            --build-arg FROM=synapse-workers-editable \
+            --build-arg SUNAPSE_WORKERS_IMAGE=synapse-workers-editable \
             -f "docker/complement/Dockerfile" "docker/complement"
 
         # Prepare the Rust module
@@ -180,21 +180,24 @@ if [ -z "$skip_docker_build" ]; then
     else
 
         # Build the base Synapse image from the local checkout
-        echo_if_github "::group::Build Docker image: matrixdotorg/synapse"
-        $CONTAINER_RUNTIME build -t matrixdotorg/synapse \
-        --build-arg TEST_ONLY_SKIP_DEP_HASH_VERIFICATION \
-        --build-arg TEST_ONLY_IGNORE_POETRY_LOCKFILE \
-        -f "docker/Dockerfile" .
+        echo_if_github "::group::Build Docker image: synapse"
+        $CONTAINER_RUNTIME build -t synapse \
+            --build-arg TEST_ONLY_SKIP_DEP_HASH_VERIFICATION \
+            --build-arg TEST_ONLY_IGNORE_POETRY_LOCKFILE \
+            -f "docker/Dockerfile" .
         echo_if_github "::endgroup::"
 
         # Build the workers docker image (from the base Synapse image we just built).
-        echo_if_github "::group::Build Docker image: matrixdotorg/synapse-workers"
-        $CONTAINER_RUNTIME build -t matrixdotorg/synapse-workers -f "docker/Dockerfile-workers" .
+        echo_if_github "::group::Build Docker image: synapse-workers"
+        $CONTAINER_RUNTIME build -t synapse-workers \
+            --build-arg SYNAPSE_IMAGE=synapse \
+            -f "docker/Dockerfile-workers" .
         echo_if_github "::endgroup::"
 
         # Build the unified Complement image (from the worker Synapse image we just built).
-        echo_if_github "::group::Build Docker image: complement/Dockerfile"
+        echo_if_github "::group::Build Docker image: complement-synapse"
         $CONTAINER_RUNTIME build -t complement-synapse \
+            --build-arg SYNAPSE_WORKERS_IMAGE=synapse-workers \
             -f "docker/complement/Dockerfile" "docker/complement"
         echo_if_github "::endgroup::"
 

synapse/api/errors.py

@@ -128,10 +128,6 @@ class Codes(str, Enum):
     # MSC2677
     DUPLICATE_ANNOTATION = "M_DUPLICATE_ANNOTATION"
 
-    # MSC3575 we are telling the client they need to reset their sliding sync
-    # connection.
-    UNKNOWN_POS = "M_UNKNOWN_POS"
-
 
 class CodeMessageException(RuntimeError):
     """An exception with integer code, a message string attributes and optional headers.
@@ -851,17 +847,3 @@ class PartialStateConflictError(SynapseError):
             msg=PartialStateConflictError.message(),
             errcode=Codes.UNKNOWN,
         )
-
-
-class SlidingSyncUnknownPosition(SynapseError):
-    """An error that Synapse can return to signal to the client to expire their
-    sliding sync connection (i.e. send a new request without a `?since=`
-    param).
-    """
-
-    def __init__(self) -> None:
-        super().__init__(
-            HTTPStatus.BAD_REQUEST,
-            msg="Unknown position",
-            errcode=Codes.UNKNOWN_POS,
-        )

synapse/handlers/sliding_sync.py

@@ -47,7 +47,6 @@ from synapse.api.constants import (
     EventTypes,
     Membership,
 )
-from synapse.api.errors import SlidingSyncUnknownPosition
 from synapse.events import EventBase, StrippedStateEvent
 from synapse.events.utils import parse_stripped_state_event, strip_event
 from synapse.handlers.relations import BundledAggregations
@@ -492,22 +491,6 @@ class SlidingSyncHandler:
             # See https://github.com/matrix-org/matrix-doc/issues/1144
             raise NotImplementedError()
 
-        if from_token:
-            # Check that we recognize the connection position, if not tell the
-            # clients that they need to start again.
-            #
-            # If we don't do this and the client asks for the full range of
-            # rooms, we end up sending down all rooms and their state from
-            # scratch (which can be very slow). By expiring the connection we
-            # allow the client a chance to do an initial request with a smaller
-            # range of rooms to get them some results sooner but will end up
-            # taking the same amount of time (more with round-trips and
-            # re-processing) in the end to get everything again.
-            if not await self.connection_store.is_valid_token(
-                sync_config, from_token.connection_position
-            ):
-                raise SlidingSyncUnknownPosition()
-
         await self.connection_store.mark_token_seen(
             sync_config=sync_config,
             from_token=from_token,
@@ -2838,16 +2821,6 @@ class SlidingSyncConnectionStore:
         attr.Factory(dict)
     )
 
-    async def is_valid_token(
-        self, sync_config: SlidingSyncConfig, connection_token: int
-    ) -> bool:
-        """Return whether the connection token is valid/recognized"""
-        if connection_token == 0:
-            return True
-
-        conn_key = self._get_connection_key(sync_config)
-        return connection_token in self._connections.get(conn_key, {})
-
     async def have_sent_room(
         self, sync_config: SlidingSyncConfig, connection_token: int, room_id: str
     ) -> HaveSentRoom:

tests/rest/client/sliding_sync/test_rooms_required_state.py

@@ -161,10 +161,10 @@ class SlidingSyncRoomsRequiredStateTestCase(SlidingSyncBase):
         self.assertIsNone(response_body["rooms"][room_id1].get("required_state"))
         self.assertIsNone(response_body["rooms"][room_id1].get("invite_state"))
 
-    def test_rooms_incremental_sync_restart(self) -> None:
+    def test_rooms_required_state_incremental_sync_restart(self) -> None:
         """
-        Test that after a restart (and so the in memory caches are reset) that
-        we correctly return an `M_UNKNOWN_POS`
+        Test `rooms.required_state` returns requested state events in the room during an
+        incremental sync, after a restart (and so the in memory caches are reset).
         """
 
         user1_id = self.register_user("user1", "pass")
@@ -195,16 +195,22 @@ class SlidingSyncRoomsRequiredStateTestCase(SlidingSyncBase):
         self.hs.get_sliding_sync_handler().connection_store._connections.clear()
 
         # Make the Sliding Sync request
-        channel = self.make_request(
-            method="POST",
-            path=self.sync_endpoint + f"?pos={from_token}",
-            content=sync_body,
-            access_token=user1_tok,
+        response_body, _ = self.do_sync(sync_body, since=from_token, tok=user1_tok)
+
+        # If the cache has been cleared then we do expect the state to come down
+        state_map = self.get_success(
+            self.storage_controllers.state.get_current_state(room_id1)
         )
-        self.assertEqual(channel.code, 400, channel.json_body)
-        self.assertEqual(
-            channel.json_body["errcode"], "M_UNKNOWN_POS", channel.json_body
+
+        self._assertRequiredStateIncludes(
+            response_body["rooms"][room_id1]["required_state"],
+            {
+                state_map[(EventTypes.Create, "")],
+                state_map[(EventTypes.RoomHistoryVisibility, "")],
+            },
+            exact=True,
         )
+        self.assertIsNone(response_body["rooms"][room_id1].get("invite_state"))
 
     def test_rooms_required_state_wildcard(self) -> None:
         """

tests/rest/client/sliding_sync/test_rooms_timeline.py

@@ -12,14 +12,13 @@
 # <https://www.gnu.org/licenses/agpl-3.0.html>.
 #
 import logging
-from typing import List, Optional
 
 from twisted.test.proto_helpers import MemoryReactor
 
 import synapse.rest.admin
 from synapse.rest.client import login, room, sync
 from synapse.server import HomeServer
-from synapse.types import StreamToken, StrSequence
+from synapse.types import StreamToken
 from synapse.util import Clock
 
 from tests.rest.client.sliding_sync.test_sliding_sync import SlidingSyncBase
@@ -43,82 +42,6 @@ class SlidingSyncRoomsTimelineTestCase(SlidingSyncBase):
         self.store = hs.get_datastores().main
         self.storage_controllers = hs.get_storage_controllers()
 
-    def _assertListEqual(
-        self,
-        actual_items: StrSequence,
-        expected_items: StrSequence,
-        message: Optional[str] = None,
-    ) -> None:
-        """
-        Like `self.assertListEqual(...)` but with an actually understandable diff message.
-        """
-
-        if actual_items == expected_items:
-            return
-
-        expected_lines: List[str] = []
-        for expected_item in expected_items:
-            is_expected_in_actual = expected_item in actual_items
-            expected_lines.append(
-                "{}  {}".format(" " if is_expected_in_actual else "?", expected_item)
-            )
-
-        actual_lines: List[str] = []
-        for actual_item in actual_items:
-            is_actual_in_expected = actual_item in expected_items
-            actual_lines.append(
-                "{}  {}".format("+" if is_actual_in_expected else " ", actual_item)
-            )
-
-        newline = "\n"
-        expected_string = f"Expected items to be in actual ('?' = missing expected items):\n [\n{newline.join(expected_lines)}\n ]"
-        actual_string = f"Actual ('+' = found expected items):\n [\n{newline.join(actual_lines)}\n ]"
-        first_message = "Items must"
-        diff_message = f"{first_message}\n{expected_string}\n{actual_string}"
-
-        self.fail(f"{diff_message}\n{message}")
-
-    def _assertTimelineEqual(
-        self,
-        *,
-        room_id: str,
-        actual_event_ids: List[str],
-        expected_event_ids: List[str],
-        message: Optional[str] = None,
-    ) -> None:
-        """
-        Like `self.assertListEqual(...)` for event IDs in a room but will give a nicer
-        output with context for what each event_id is (type, stream_ordering, content,
-        etc).
-        """
-        if actual_event_ids == expected_event_ids:
-            return
-
-        event_id_set = set(actual_event_ids + expected_event_ids)
-        events = self.get_success(self.store.get_events(event_id_set))
-
-        def event_id_to_string(event_id: str) -> str:
-            event = events.get(event_id)
-            if event:
-                state_key = event.get_state_key()
-                state_key_piece = f", {state_key}" if state_key is not None else ""
-                return (
-                    f"({event.internal_metadata.stream_ordering: >2}, {event.internal_metadata.instance_name}) "
-                    + f"{event.event_id} ({event.type}{state_key_piece}) {event.content.get('membership', '')}{event.content.get('body', '')}"
-                )
-
-            return f"{event_id} <event not found in room_id={room_id}>"
-
-        self._assertListEqual(
-            actual_items=[
-                event_id_to_string(event_id) for event_id in actual_event_ids
-            ],
-            expected_items=[
-                event_id_to_string(event_id) for event_id in expected_event_ids
-            ],
-            message=message,
-        )
-
     def test_rooms_limited_initial_sync(self) -> None:
         """
         Test that we mark `rooms` as `limited=True` when we saturate the `timeline_limit`
@@ -162,18 +85,17 @@ class SlidingSyncRoomsTimelineTestCase(SlidingSyncBase):
             response_body["rooms"][room_id1],
         )
         # Check to make sure the latest events are returned
-        self._assertTimelineEqual(
-            room_id=room_id1,
-            actual_event_ids=[
+        self.assertEqual(
+            [
                 event["event_id"]
                 for event in response_body["rooms"][room_id1]["timeline"]
             ],
-            expected_event_ids=[
+            [
                 event_response4["event_id"],
                 event_response5["event_id"],
                 user1_join_response["event_id"],
             ],
-            message=str(response_body["rooms"][room_id1]["timeline"]),
+            response_body["rooms"][room_id1]["timeline"],
         )
 
         # Check to make sure the `prev_batch` points at the right place
@@ -305,17 +227,16 @@ class SlidingSyncRoomsTimelineTestCase(SlidingSyncBase):
             + str(response_body["rooms"][room_id1]),
         )
         # Check to make sure the latest events are returned
-        self._assertTimelineEqual(
-            room_id=room_id1,
-            actual_event_ids=[
+        self.assertEqual(
+            [
                 event["event_id"]
                 for event in response_body["rooms"][room_id1]["timeline"]
             ],
-            expected_event_ids=[
+            [
                 event_response2["event_id"],
                 event_response3["event_id"],
             ],
-            message=str(response_body["rooms"][room_id1]["timeline"]),
+            response_body["rooms"][room_id1]["timeline"],
         )
 
         # All events are "live"
@@ -382,19 +303,18 @@ class SlidingSyncRoomsTimelineTestCase(SlidingSyncBase):
             + str(response_body["rooms"][room_id1]),
         )
         # Check to make sure that the "live" and historical events are returned
-        self._assertTimelineEqual(
-            room_id=room_id1,
-            actual_event_ids=[
+        self.assertEqual(
+            [
                 event["event_id"]
                 for event in response_body["rooms"][room_id1]["timeline"]
             ],
-            expected_event_ids=[
+            [
                 event_response2["event_id"],
                 user1_join_response["event_id"],
                 event_response3["event_id"],
                 event_response4["event_id"],
             ],
-            message=str(response_body["rooms"][room_id1]["timeline"]),
+            response_body["rooms"][room_id1]["timeline"],
         )
 
         # Only events after the `from_token` are "live" (join, event3, event4)
@@ -441,18 +361,17 @@ class SlidingSyncRoomsTimelineTestCase(SlidingSyncBase):
         response_body, _ = self.do_sync(sync_body, tok=user1_tok)
 
         # We should see events before the ban but not after
-        self._assertTimelineEqual(
-            room_id=room_id1,
-            actual_event_ids=[
+        self.assertEqual(
+            [
                 event["event_id"]
                 for event in response_body["rooms"][room_id1]["timeline"]
             ],
-            expected_event_ids=[
+            [
                 event_response3["event_id"],
                 event_response4["event_id"],
                 user1_ban_response["event_id"],
             ],
-            message=str(response_body["rooms"][room_id1]["timeline"]),
+            response_body["rooms"][room_id1]["timeline"],
         )
         # No "live" events in an initial sync (no `from_token` to define the "live"
         # range)
@@ -509,18 +428,17 @@ class SlidingSyncRoomsTimelineTestCase(SlidingSyncBase):
         response_body, _ = self.do_sync(sync_body, since=from_token, tok=user1_tok)
 
         # We should see events before the ban but not after
-        self._assertTimelineEqual(
-            room_id=room_id1,
-            actual_event_ids=[
+        self.assertEqual(
+            [
                 event["event_id"]
                 for event in response_body["rooms"][room_id1]["timeline"]
             ],
-            expected_event_ids=[
+            [
                 event_response3["event_id"],
                 event_response4["event_id"],
                 user1_ban_response["event_id"],
             ],
-            message=str(response_body["rooms"][room_id1]["timeline"]),
+            response_body["rooms"][room_id1]["timeline"],
         )
         # All live events in the incremental sync
         self.assertEqual(