temp

Attempt to re-connect better.
2022-03-11 11:26:03 -05:00 · 2022-03-11 10:35:23 -05:00 · 2022-03-11 10:35:22 -05:00 · 2022-03-11 10:34:59 -05:00 · 2022-03-11 10:34:58 -05:00 · 2022-03-11 10:33:58 -05:00
159 changed files with 1985 additions and 977 deletions
@@ -7,6 +7,5 @@
 !MANIFEST.in
 !README.rst
 !setup.py
-!synctl

 **/__pycache__
@@ -112,7 +112,8 @@ jobs:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          files: |
-            python-dist/*
+            Sdist/*
+            Wheel/*
            debs.tar.xz
          # if it's not already published, keep the release as a draft.
          draft: true
@@ -17,6 +17,7 @@ jobs:
      - uses: actions/setup-python@v2
      - run: pip install -e .
      - run: scripts-dev/generate_sample_config.sh --check
+      - run: scripts-dev/config-lint.sh

  lint:
    runs-on: ubuntu-latest
@@ -387,34 +388,22 @@ jobs:
  tests-done:
    if: ${{ always() }}
    needs:
+      - check-sampleconfig
      - lint
      - lint-crlf
      - lint-newsfile
      - trial
      - trial-olddeps
      - sytest
+      - export-data
      - portdb
      - complement
    runs-on: ubuntu-latest
    steps:
-      - name: Set build result
-        env:
-          NEEDS_CONTEXT: ${{ toJSON(needs) }}
-        # the `jq` incantation dumps out a series of "<job> <result>" lines.
-        # we set it to an intermediate variable to avoid a pipe, which makes it
-        # hard to set $rc.
-        run: |
-          rc=0
-          results=$(jq -r 'to_entries[] | [.key,.value.result] | join(" ")' <<< $NEEDS_CONTEXT)
-          while read job result ; do
-              # The newsfile lint may be skipped on non PR builds
-              if [ $result == "skipped" ] && [ $job == "lint-newsfile" ]; then
-                continue
-              fi
+      - uses: matrix-org/done-action@v2
+        with:
+          needs: ${{ toJSON(needs) }}

-              if [ "$result" != "success" ]; then
-                  echo "::set-failed ::Job $job returned $result"
-                  rc=1
-              fi
-          done <<< $results
-          exit $rc
+          # The newsfile lint may be skipped on non PR builds
+          skippable:
+            lint-newsfile
@@ -1,10 +1,28 @@
-Synapse 1.54.0rc1 (2022-03-02)
-==============================
+Synapse 1.54.0 (2022-03-08)
+===========================

 Please note that this will be the last release of Synapse that is compatible with Mjolnir 1.3.1 and earlier.
 Administrators of servers which have the Mjolnir module installed are advised to upgrade Mjolnir to version 1.3.2 or later.


+Bugfixes
+--------
+
+- Fix a bug introduced in Synapse 1.54.0rc1 preventing the new module callbacks introduced in this release from being registered by modules. ([\#12141](https://github.com/matrix-org/synapse/issues/12141))
+- Fix a bug introduced in Synapse 1.54.0rc1 where runtime dependency version checks would mistakenly check development dependencies if they were present and would not accept pre-release versions of dependencies. ([\#12129](https://github.com/matrix-org/synapse/issues/12129), [\#12177](https://github.com/matrix-org/synapse/issues/12177))
+
+
+Internal Changes
+----------------
+
+- Update release script to insert the previous version when writing "No significant changes" line in the changelog. ([\#12127](https://github.com/matrix-org/synapse/issues/12127))
+- Relax the version guard for "packaging" added in [\#12088](https://github.com/matrix-org/synapse/issues/12088). ([\#12166](https://github.com/matrix-org/synapse/issues/12166))
+
+
+Synapse 1.54.0rc1 (2022-03-02)
+==============================
+
+
 Features
 --------

@@ -1,4 +1,3 @@
-include synctl
 include LICENSE
 include VERSION
 include *.rst
@@ -312,6 +312,9 @@ We recommend using the demo which starts 3 federated instances running on ports

 (to stop, you can use `./demo/stop.sh`)

+See the [demo documentation](https://matrix-org.github.io/synapse/develop/development/demo.html)
+for more information.
+
 If you just want to start a single instance of the app and run it directly::

    # Create the homeserver.yaml config once
@@ -0,0 +1 @@
+Remove workaround introduced in Synapse 1.50.0 for Mjolnir compatibility. Breaks compatibility with Mjolnir 1.3.1 and earlier.
@@ -0,0 +1 @@
+Add third-party rules rules callbacks `check_can_shutdown_room` and `check_can_deactivate_user`.
@@ -0,0 +1 @@
+Correct type hints for txredis.
@@ -1 +1 @@
-Add type hints to `tests/rest/client`.
+Add type hints to tests files.
@@ -0,0 +1 @@
+Fix a long-standing bug when redacting events with relations.
@@ -1 +0,0 @@
-Refactor the tests for event relations.
@@ -0,0 +1 @@
+Fix a long-standing bug when redacting events with relations.
@@ -0,0 +1 @@
+Fix a long-standing bug when redacting events with relations.
@@ -0,0 +1 @@
+Fix CI not attaching source distributions and wheels to the GitHub releases.
@@ -0,0 +1 @@
+Improve performance of logging in for large accounts.
@@ -0,0 +1 @@
+Add experimental env var `SYNAPSE_ASYNC_IO_REACTOR` that causes Synapse to use the asyncio reactor for Twisted.
@@ -0,0 +1 @@
+Remove backwards compatibilty with pagination tokens from the `/relations` and `/aggregations` endpoints generated from Synapse < v1.52.0.
@@ -0,0 +1 @@
+Move `synctl` into `synapse._scripts` and expose as an entry point.
@@ -0,0 +1 @@
+Improve documentation for demo scripts.
@@ -1 +1 @@
-Add type hints to `tests/rest`.
+Add type hints to tests files.
@@ -0,0 +1 @@
+Add test for `ObservableDeferred`'s cancellation behaviour.
@@ -0,0 +1 @@
+Use `ParamSpec` in type hints for `synapse.logging.context`.
@@ -0,0 +1 @@
+Support the stable identifiers from [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440): threads.
@@ -0,0 +1 @@
+Move CI checks out of tox, to facilitate a move to using poetry.
@@ -0,0 +1 @@
+Fix a bug introduced in #4864 whereby background updates are never run with the default background batch size.
@@ -0,0 +1 @@
+Add type hints for `ObservableDeferred` attributes.
@@ -0,0 +1 @@
+Use a prebuilt Action for the `tests-done` CI job.
@@ -0,0 +1 @@
+Reduce number of DB queries made during processing of `/sync`.
@@ -0,0 +1 @@
+Avoid trying to calculate the state at outlier events.
@@ -0,0 +1 @@
+Fix a bug where non-standard information was returned from the `/hierarchy` API. Introduced in Synapse v1.41.0.
@@ -0,0 +1 @@
+Updates to the Room DAG concepts development document.
@@ -0,0 +1 @@
+Retry HTTP replication failures, this should prevent 502's when restarting stateful workers (main, event persisters, stream writers). Contributed by Nick @ Beeper.
@@ -0,0 +1 @@
+Remove unused variables.
@@ -0,0 +1 @@
+Fix a long-standing bug when redacting events with relations.
@@ -0,0 +1 @@
+Rename `HomeServer.get_tcp_replication` to `get_replication_command_handler`.
@@ -0,0 +1 @@
+Document that the `typing`, `to_device`, `account_data`, `receipts`, and `presence` stream writer can only be used on a single worker.
@@ -0,0 +1 @@
+Remove some dead code.
@@ -0,0 +1 @@
+Avoid trying to calculate the state at outlier events.
@@ -0,0 +1 @@
+Fix a misleading comment in the function `check_event_for_spam`.
@@ -0,0 +1 @@
+Document that contributors can sign off privately by email.
@@ -0,0 +1 @@
+Remove unnecessary `pass` statements.
@@ -0,0 +1 @@
+Add type hints to tests files.
@@ -0,0 +1 @@
+Add type hints to tests files.
@@ -0,0 +1 @@
+Update the SSO username picker template to comply with SIWA guidelines.
@@ -1,3 +1,9 @@
+matrix-synapse-py3 (1.54.0) stable; urgency=medium
+
+  * New synapse release 1.54.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 08 Mar 2022 10:54:52 +0000
+
 matrix-synapse-py3 (1.54.0~rc1) stable; urgency=medium

  * New synapse release 1.54.0~rc1.
@@ -1,7 +1,4 @@
-*.db
-*.log
-*.log.*
-*.pid
-
-/media_store.*
-/etc
+# Ignore all the temporary files from the demo servers.
+8080/
+8081/
+8082/
@@ -1,26 +0,0 @@
-DO NOT USE THESE DEMO SERVERS IN PRODUCTION
-
-Requires you to have done:
-    python setup.py develop
-
-
-The demo start.sh will start three synapse servers on ports 8080, 8081 and 8082, with host names localhost:$port. This can be easily changed to `hostname`:$port in start.sh if required.
-
-To enable the servers to communicate untrusted ssl certs are used. In order to do this the servers do not check the certs
-and are configured in a highly insecure way. Do not use these configuration files in production.
-
-stop.sh will stop the synapse servers and the webclient.
-
-clean.sh will delete the databases and log files.
-
-To start a completely new set of servers, run:
-
-    ./demo/stop.sh; ./demo/clean.sh && ./demo/start.sh
-
-
-Logs and sqlitedb will be stored in demo/808{0,1,2}.{log,db}
-
-
-
-Also note that when joining a public room on a different HS via "#foo:bar.net", then you are (in the current impl) joining a room with room_id "foo". This means that it won't work if your HS already has a room with that name.
-
@@ -4,6 +4,9 @@ set -e

 DIR="$( cd "$( dirname "$0" )" && pwd )"

+# Ensure that the servers are stopped.
+$DIR/stop.sh
+
 PID_FILE="$DIR/servers.pid"

 if [ -f "$PID_FILE" ]; then
@@ -6,8 +6,6 @@ CWD=$(pwd)

 cd "$DIR/.." || exit

-mkdir -p demo/etc
-
 PYTHONPATH=$(readlink -f "$(pwd)")
 export PYTHONPATH

@@ -21,22 +19,26 @@ for port in 8080 8081 8082; do
    mkdir -p demo/$port
    pushd demo/$port || exit

-    #rm $DIR/etc/$port.config
+    # Generate the configuration for the homeserver at localhost:848x.
    python3 -m synapse.app.homeserver \
        --generate-config \
-        -H "localhost:$https_port" \
-        --config-path "$DIR/etc/$port.config" \
+        --server-name "localhost:$port" \
+        --config-path "$port.config" \
        --report-stats no

-    if ! grep -F "Customisation made by demo/start.sh" -q "$DIR/etc/$port.config"; then
-        # Generate tls keys
-        openssl req -x509 -newkey rsa:4096 -keyout "$DIR/etc/localhost:$https_port.tls.key" -out "$DIR/etc/localhost:$https_port.tls.crt" -days 365 -nodes -subj "/O=matrix"
+    if ! grep -F "Customisation made by demo/start.sh" -q "$port.config"; then
+        # Generate TLS keys.
+        openssl req -x509 -newkey rsa:4096 \
+          -keyout "localhost:$port.tls.key" \
+          -out "localhost:$port.tls.crt" \
+          -days 365 -nodes -subj "/O=matrix"

-        # Regenerate configuration
+        # Add customisations to the configuration.
        {
-            printf '\n\n# Customisation made by demo/start.sh\n'
+            printf '\n\n# Customisation made by demo/start.sh\n\n'
            echo "public_baseurl: http://localhost:$port/"
            echo 'enable_registration: true'
+            echo ''

 			# Warning, this heredoc depends on the interaction of tabs and spaces.
 			# Please don't accidentaly bork me with your fancy settings.
@@ -63,38 +65,34 @@ for port in 8080 8081 8082; do

            echo "${listeners}"

-            # Disable tls for the servers
-            printf '\n\n# Disable tls on the servers.'
+            # Disable TLS for the servers
+            printf '\n\n# Disable TLS for the servers.'
            echo '# DO NOT USE IN PRODUCTION'
            echo 'use_insecure_ssl_client_just_for_testing_do_not_use: true'
            echo 'federation_verify_certificates: false'

-            # Set tls paths
-            echo "tls_certificate_path: \"$DIR/etc/localhost:$https_port.tls.crt\""
-            echo "tls_private_key_path: \"$DIR/etc/localhost:$https_port.tls.key\""
+            # Set paths for the TLS certificates.
+            echo "tls_certificate_path: \"$DIR/$port/localhost:$port.tls.crt\""
+            echo "tls_private_key_path: \"$DIR/$port/localhost:$port.tls.key\""

            # Ignore keys from the trusted keys server
            echo '# Ignore keys from the trusted keys server'
            echo 'trusted_key_servers:'
            echo '  - server_name: "matrix.org"'
            echo '    accept_keys_insecurely: true'
+            echo ''

-			# Reduce the blacklist
-			blacklist=$(cat <<-BLACK
-			# Set the blacklist so that it doesn't include 127.0.0.1, ::1
-			federation_ip_range_blacklist:
-			  - '10.0.0.0/8'
-			  - '172.16.0.0/12'
-			  - '192.168.0.0/16'
-			  - '100.64.0.0/10'
-			  - '169.254.0.0/16'
-			  - 'fe80::/64'
-			  - 'fc00::/7'
-			BLACK
+			# Allow the servers to communicate over localhost.
+			allow_list=$(cat <<-ALLOW_LIST
+			# Allow the servers to communicate over localhost.
+			ip_range_whitelist:
+			  - '127.0.0.1/8'
+			  - '::1/128'
+			ALLOW_LIST
 			)

-            echo "${blacklist}"
-        } >> "$DIR/etc/$port.config"
+            echo "${allow_list}"
+        } >> "$port.config"
    fi

    # Check script parameters
@@ -141,19 +139,18 @@ for port in 8080 8081 8082; do
 			    burst_count: 1000
 			RC
 			)
-            echo "${ratelimiting}" >> "$DIR/etc/$port.config"
+            echo "${ratelimiting}" >> "$port.config"
        fi
    fi

-    if ! grep -F "full_twisted_stacktraces" -q  "$DIR/etc/$port.config"; then
-        echo "full_twisted_stacktraces: true" >> "$DIR/etc/$port.config"
-    fi
-    if ! grep -F "report_stats" -q  "$DIR/etc/$port.config" ; then
-        echo "report_stats: false" >> "$DIR/etc/$port.config"
+    # Always disable reporting of stats if the option is not there.
+    if ! grep -F "report_stats" -q  "$port.config" ; then
+        echo "report_stats: false" >> "$port.config"
    fi

+    # Run the homeserver in the background.
    python3 -m synapse.app.homeserver \
-        --config-path "$DIR/etc/$port.config" \
+        --config-path "$port.config" \
        -D \

    popd || exit
@@ -46,7 +46,7 @@ RUN \
    && rm -rf /var/lib/apt/lists/*

 # Copy just what we need to pip install
-COPY MANIFEST.in README.rst setup.py synctl /synapse/
+COPY MANIFEST.in README.rst setup.py /synapse/
 COPY synapse/__init__.py /synapse/synapse/__init__.py
 COPY synapse/python_dependencies.py /synapse/synapse/python_dependencies.py

@@ -82,6 +82,7 @@
  - [Release Cycle](development/releases.md)
  - [Git Usage](development/git.md)
  - [Testing]()
+    - [Demo scripts](development/demo.md)
  - [OpenTracing](opentracing.md)
  - [Database Schemas](development/database_schema.md)
  - [Experimental features](development/experimental_features.md)
@@ -458,6 +458,17 @@ Git allows you to add this signoff automatically when using the `-s`
 flag to `git commit`, which uses the name and email set in your
 `user.name` and `user.email` git configs.

+### Private Sign off
+
+If you would like to provide your legal name privately to the Matrix.org
+Foundation (instead of in a public commit or comment), you can do so
+by emailing your legal name and a link to the pull request to
+[dco@matrix.org](mailto:dco@matrix.org?subject=Private%20sign%20off).
+It helps to include "sign off" or similar in the subject line. You will then
+be instructed further.
+
+Once private sign off is complete, doing so for future contributions will not
+be required.

 # 10. Turn feedback into better code.

@@ -0,0 +1,41 @@
+# Synapse demo setup
+
+**DO NOT USE THESE DEMO SERVERS IN PRODUCTION**
+
+Requires you to have a [Synapse development environment setup](https://matrix-org.github.io/synapse/develop/development/contributing_guide.html#4-install-the-dependencies).
+
+The demo setup allows running three federation Synapse servers, with server
+names `localhost:8080`, `localhost:8081`, and `localhost:8082`.
+
+You can access them via any Matrix client over HTTP at `localhost:8080`,
+`localhost:8081`, and `localhost:8082` or over HTTPS at `localhost:8480`,
+`localhost:8481`, and `localhost:8482`.
+
+To enable the servers to communicate, self-signed SSL certificates are generated
+and the servers are configured in a highly insecure way, including:
+
+* Not checking certificates over federation.
+* Not verifying keys.
+
+The servers are configured to store their data under `demo/8080`, `demo/8081`, and
+`demo/8082`. This includes configuration, logs, SQLite databases, and media.
+
+Note that when joining a public room on a different HS via "#foo:bar.net", then
+you are (in the current impl) joining a room with room_id "foo". This means that
+it won't work if your HS already has a room with that name.
+
+## Using the demo scripts
+
+There's three main scripts with straightforward purposes:
+
+* `start.sh` will start the Synapse servers, generating any missing configuration.
+  * This accepts a single parameter `--no-rate-limit` to "disable" rate limits
+    (they actually still exist, but are very high).
+* `stop.sh` will stop the Synapse servers.
+* `clean.sh` will delete the configuration, databases, log files, etc.
+
+To start a completely new set of servers, run:
+
+```sh
+./demo/stop.sh; ./demo/clean.sh && ./demo/start.sh
+```
@@ -30,13 +30,57 @@ rather than skipping any that arrived late; whereas if you're looking at a
 historical section of timeline (i.e. `/messages`), you want to see the best
 representation of the state of the room as others were seeing it at the time.

+## Outliers
+
+We mark an event as an `outlier` when we haven't figured out the state for the
+room at that point in the DAG yet. They are "floating" events that we haven't
+yet correlated to the DAG.
+
+Outliers typically arise when we fetch the auth chain or state for a given
+event. When that happens, we just grab the events in the state/auth chain,
+without calculating the state at those events, or backfilling their
+`prev_events`.
+
+So, typically, we won't have the `prev_events` of an `outlier` in the database,
+(though it's entirely possible that we *might* have them for some other
+reason). Other things that make outliers different from regular events:
+
+ * We don't have state for them, so there should be no entry in
+   `event_to_state_groups` for an outlier. (In practice this isn't always
+   the case, though I'm not sure why: see https://github.com/matrix-org/synapse/issues/12201).
+
+ * We don't record entries for them in the `event_edges`,
+   `event_forward_extremeties` or `event_backward_extremities` tables.
+
+Since outliers are not tied into the DAG, they do not normally form part of the
+timeline sent down to clients via `/sync` or `/messages`; however there is an
+exception:
+
+### Out-of-band membership events
+
+A special case of outlier events are some membership events for federated rooms
+that we aren't full members of. For example:
+
+ * invites received over federation, before we join the room
+ * *rejections* for said invites
+ * knock events for rooms that we would like to join but have not yet joined.
+
+In all the above cases, we don't have the state for the room, which is why they
+are treated as outliers. They are a bit special though, in that they are
+proactively sent to clients via `/sync`.

 ## Forward extremity

-Most-recent-in-time events in the DAG which are not referenced by any other events' `prev_events` yet.
+Most-recent-in-time events in the DAG which are not referenced by any other
+events' `prev_events` yet. (In this definition, outliers, rejected events, and
+soft-failed events don't count.)

-The forward extremities of a room are used as the `prev_events` when the next event is sent.
+The forward extremities of a room (or at least, a subset of them, if there are
+more than ten) are used as the `prev_events` when the next event is sent.

+The "current state" of a room (ie: the state which would be used if we
+generated a new event) is, therefore, the resolution of the room states
+at each of the forward extremities.

 ## Backward extremity

@@ -44,23 +88,14 @@ The current marker of where we have backfilled up to and will generally be the
 `prev_events` of the oldest-in-time events we have in the DAG. This gives a starting point when
 backfilling history.

-When we persist a non-outlier event, we clear it as a backward extremity and set
-all of its `prev_events` as the new backward extremities if they aren't already
-persisted in the `events` table.
-
-
-## Outliers
-
-We mark an event as an `outlier` when we haven't figured out the state for the
-room at that point in the DAG yet.
-
-We won't *necessarily* have the `prev_events` of an `outlier` in the database,
-but it's entirely possible that we *might*.
-
-For example, when we fetch the event auth chain or state for a given event, we
-mark all of those claimed auth events as outliers because we haven't done the
-state calculation ourself.
+Note that, unlike forward extremities, we typically don't have any backward
+extremity events themselves in the database - or, if we do, they will be "outliers" (see
+above). Either way, we don't expect to have the room state at a backward extremity.

+When we persist a non-outlier event, if it was previously a backward extremity,
+we clear it as a backward extremity and set all of its `prev_events` as the new
+backward extremities if they aren't already persisted as non-outliers. This
+therefore keeps the backward extremities up-to-date.

 ## State groups

@@ -63,4 +63,5 @@ release of Synapse.

 If you want to get up and running quickly with a trio of homeservers in a
 private federation, there is a script in the `demo` directory. This is mainly
-useful just for development purposes. See [demo/README](https://github.com/matrix-org/synapse/tree/develop/demo/).
+useful just for development purposes. See
+[demo scripts](https://matrix-org.github.io/synapse/develop/development/demo.html).
@@ -148,6 +148,49 @@ deny an incoming event, see [`check_event_for_spam`](spam_checker_callbacks.md#c

 If multiple modules implement this callback, Synapse runs them all in order.

+### `check_can_shutdown_room`
+
+_First introduced in Synapse v1.55.0_
+
+```python
+async def check_can_shutdown_room(
+    user_id: str, room_id: str,
+) -> bool:
+```
+
+Called when an admin user requests the shutdown of a room. The module must return a
+boolean indicating whether the shutdown can go through. If the callback returns `False`,
+the shutdown will not proceed and the caller will see a `M_FORBIDDEN` error.
+
+If multiple modules implement this callback, they will be considered in order. If a
+callback returns `True`, Synapse falls through to the next one. The value of the first
+callback that does not return `True` will be used. If this happens, Synapse will not call
+any of the subsequent implementations of this callback.
+
+### `check_can_deactivate_user`
+
+_First introduced in Synapse v1.55.0_
+
+```python
+async def check_can_deactivate_user(
+    user_id: str, by_admin: bool,
+) -> bool:
+```
+
+Called when the deactivation of a user is requested. User deactivation can be
+performed by an admin or the user themselves, so developers are encouraged to check the
+requester when implementing this callback. The module must return a
+boolean indicating whether the deactivation can go through. If the callback returns `False`,
+the deactivation will not proceed and the caller will see a `M_FORBIDDEN` error.
+
+The module is passed two parameters, `user_id` which is the ID of the user being deactivated, and `by_admin` which is `True` if the request is made by a serve admin, and `False` otherwise.
+
+If multiple modules implement this callback, they will be considered in order. If a
+callback returns `True`, Synapse falls through to the next one. The value of the first
+callback that does not return `True` will be used. If this happens, Synapse will not call
+any of the subsequent implementations of this callback.
+
+
 ### `on_profile_update`

 _First introduced in Synapse v1.54.0_
@@ -153,9 +153,9 @@ database file (typically `homeserver.db`) to another location. Once the
 copy is complete, restart synapse. For instance:

 ```sh
-./synctl stop
+synctl stop
 cp homeserver.db homeserver.db.snapshot
-./synctl start
+synctl start
 ```

 Copy the old config file into a new config file:
@@ -192,10 +192,10 @@ Once that has completed, change the synapse config to point at the
 PostgreSQL database configuration file `homeserver-postgres.yaml`:

 ```sh
-./synctl stop
+synctl stop
 mv homeserver.yaml homeserver-old-sqlite.yaml
 mv homeserver-postgres.yaml homeserver.yaml
-./synctl start
+synctl start
 ```

 Synapse should now be running against PostgreSQL.
@@ -1947,8 +1947,13 @@ saml2_config:
 #
 #             localpart_template: Jinja2 template for the localpart of the MXID.
 #                 If this is not set, the user will be prompted to choose their
-#                 own username (see 'sso_auth_account_details.html' in the 'sso'
-#                 section of this file).
+#                 own username (see the documentation for the
+#                 'sso_auth_account_details.html' template).
+#
+#             confirm_localpart: Whether to prompt the user to validate (or
+#                 change) the generated localpart (see the documentation for the
+#                 'sso_auth_account_details.html' template), instead of
+#                 registering the account right away.
 #
 #             display_name_template: Jinja2 template for the display name to set
 #                 on first login. If unset, no displayname will be set.
@@ -176,8 +176,11 @@ Below are the templates Synapse will look for when generating pages related to S
             for the brand of the IdP
    * `user_attributes`: an object containing details about the user that
      we received from the IdP. May have the following attributes:
-        * display_name: the user's display_name
-        * emails: a list of email addresses
+        * `display_name`: the user's display name
+        * `emails`: a list of email addresses
+        * `localpart`: the local part of the Matrix user ID to register,
+          if `localpart_template` is set in the mapping provider configuration (empty
+          string if not)
  The template should render a form which submits the following fields:
    * `username`: the localpart of the user's chosen user id
 * `sso_new_user_consent.html`: HTML page allowing the user to consent to the
@@ -238,8 +238,9 @@ After updating the homeserver configuration, you must restart synapse:

  * If you use synctl:
    ```sh
-    cd /where/you/run/synapse
-    ./synctl restart
+    # Depending on how Synapse is installed, synctl may already be on
+    # your PATH. If not, you may need to activate a virtual environment.
+    synctl restart
    ```
  * If you use systemd:
    ```sh
@@ -47,7 +47,7 @@ this document.
    3.  Restart Synapse:

        ```bash
-        ./synctl restart
+        synctl restart
        ```

 To check whether your update was successful, you can check the running
@@ -85,6 +85,35 @@ process, for example:
    dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
    ```

+# Upgrading to v1.55.0
+
+## `synctl` script has been moved
+
+The `synctl` script
+[has been made](https://github.com/matrix-org/synapse/pull/12140) an
+[entry point](https://packaging.python.org/en/latest/specifications/entry-points/)
+and no longer exists at the root of Synapse's source tree. If you wish to use
+`synctl` to manage your homeserver, you should invoke `synctl` directly, e.g. 
+`synctl start` instead of `./synctl start` or `/path/to/synctl start`. 
+
+You will need to ensure `synctl` is on your `PATH`.
+  - This is automatically the case when using
+    [Debian packages](https://packages.matrix.org/debian/) or
+    [docker images](https://hub.docker.com/r/matrixdotorg/synapse)
+    provided by Matrix.org.
+  - When installing from a wheel, sdist, or PyPI, a `synctl` executable is added 
+    to your Python installation's `bin`. This should be on your `PATH`
+    automatically, though you might need to activate a virtual environment
+    depending on how you installed Synapse.
+
+
+## Compatibility dropped for Mjolnir 1.3.1 and earlier
+
+Synapse v1.55.0 drops support for Mjolnir 1.3.1 and earlier.
+If you use the Mjolnir module to moderate your homeserver,
+please upgrade Mjolnir to version 1.3.2 or later before upgrading Synapse.
+
+
 # Upgrading to v1.54.0

 ## Legacy structured logging configuration removal
@@ -351,8 +351,11 @@ is only supported with Redis-based replication.)

 To enable this, the worker must have a HTTP replication listener configured,
 have a `worker_name` and be listed in the `instance_map` config. The same worker
-can handle multiple streams. For example, to move event persistence off to a
-dedicated worker, the shared configuration would include:
+can handle multiple streams, but unless otherwise documented, each stream can only
+have a single writer.
+
+For example, to move event persistence off to a dedicated worker, the shared
+configuration would include:

 ```yaml
 instance_map:
@@ -370,8 +373,8 @@ streams and the endpoints associated with them:

 ##### The `events` stream

-The `events` stream also experimentally supports having multiple writers, where
-work is sharded between them by room ID. Note that you *must* restart all worker
+The `events` stream experimentally supports having multiple writers, where work
+is sharded between them by room ID. Note that you *must* restart all worker
 instances when adding or removing event persisters. An example `stream_writers`
 configuration with multiple writers:

@@ -384,38 +387,38 @@ stream_writers:

 ##### The `typing` stream

-The following endpoints should be routed directly to the workers configured as
-stream writers for the `typing` stream:
+The following endpoints should be routed directly to the worker configured as
+the stream writer for the `typing` stream:

    ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/typing

 ##### The `to_device` stream

-The following endpoints should be routed directly to the workers configured as
-stream writers for the `to_device` stream:
+The following endpoints should be routed directly to the worker configured as
+the stream writer for the `to_device` stream:

    ^/_matrix/client/(api/v1|r0|v3|unstable)/sendToDevice/

 ##### The `account_data` stream

-The following endpoints should be routed directly to the workers configured as
-stream writers for the `account_data` stream:
+The following endpoints should be routed directly to the worker configured as
+the stream writer for the `account_data` stream:

    ^/_matrix/client/(api/v1|r0|v3|unstable)/.*/tags
    ^/_matrix/client/(api/v1|r0|v3|unstable)/.*/account_data

 ##### The `receipts` stream

-The following endpoints should be routed directly to the workers configured as
-stream writers for the `receipts` stream:
+The following endpoints should be routed directly to the worker configured as
+the stream writer for the `receipts` stream:

    ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/receipt
    ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/read_markers

 ##### The `presence` stream

-The following endpoints should be routed directly to the workers configured as
-stream writers for the `presence` stream:
+The following endpoints should be routed directly to the worker configured as
+the stream writer for the `presence` stream:

    ^/_matrix/client/(api/v1|r0|v3|unstable)/presence/

@@ -90,7 +90,6 @@ exclude = (?x)
   |tests/push/test_push_rule_evaluator.py
   |tests/rest/client/test_transactions.py
   |tests/rest/media/v1/test_media_storage.py
-   |tests/rest/media/v1/test_url_preview.py
   |tests/scripts/test_new_matrix_user.py
   |tests/server.py
   |tests/server_notices/test_resource_limits_server_notices.py
@@ -353,3 +352,6 @@ ignore_missing_imports = True

 [mypy-zope]
 ignore_missing_imports = True
+
+[mypy-incremental.*]
+ignore_missing_imports = True
@@ -85,7 +85,7 @@ else
          "synapse" "docker" "tests"
          # annoyingly, black doesn't find these so we have to list them
          "scripts-dev"
-          "contrib" "synctl" "setup.py" "synmark" "stubs" ".ci"
+          "contrib" "setup.py" "synmark" "stubs" ".ci"
      )
  fi
 fi
@@ -17,6 +17,8 @@
 """An interactive script for doing a release. See `cli()` below.
 """

+import glob
+import os
 import re
 import subprocess
 import sys
@@ -209,8 +211,8 @@ def prepare():
    with open("synapse/__init__.py", "w") as f:
        f.write(parsed_synapse_ast.dumps())

-    # Generate changelogs
-    run_until_successful("python3 -m towncrier", shell=True)
+    # Generate changelogs.
+    generate_and_write_changelog(current_version)

    # Generate debian changelogs
    if parsed_new_version.pre is not None:
@@ -523,5 +525,29 @@ def get_changes_for_version(wanted_version: version.Version) -> str:
    return "\n".join(version_changelog)


+def generate_and_write_changelog(current_version: version.Version):
+    # We do this by getting a draft so that we can edit it before writing to the
+    # changelog.
+    result = run_until_successful(
+        "python3 -m towncrier --draft", shell=True, capture_output=True
+    )
+    new_changes = result.stdout.decode("utf-8")
+    new_changes = new_changes.replace(
+        "No significant changes.", f"No significant changes since {current_version}."
+    )
+
+    # Prepend changes to changelog
+    with open("CHANGES.md", "r+") as f:
+        existing_content = f.read()
+        f.seek(0, 0)
+        f.write(new_changes)
+        f.write("\n")
+        f.write(existing_content)
+
+    # Remove all the news fragments
+    for f in glob.iglob("changelog.d/*.*"):
+        os.remove(f)
+
+
 if __name__ == "__main__":
    cli()
@@ -155,6 +155,7 @@ setup(
            # Application
            "synapse_homeserver = synapse.app.homeserver:main",
            "synapse_worker = synapse.app.generic_worker:main",
+            "synctl = synapse._scripts.synctl:main",
            # Scripts
            "export_signing_key = synapse._scripts.export_signing_key:main",
            "generate_config = synapse._scripts.generate_config:main",
@@ -177,6 +178,5 @@ setup(
        "Programming Language :: Python :: 3.9",
        "Programming Language :: Python :: 3.10",
    ],
-    scripts=["synctl"],
    cmdclass={"test": TestCommand},
 )
@@ -20,7 +20,7 @@ from twisted.internet import protocol
 from twisted.internet.defer import Deferred

 class RedisProtocol(protocol.Protocol):
-    def publish(self, channel: str, message: bytes): ...
+    def publish(self, channel: str, message: bytes) -> "Deferred[None]": ...
    def ping(self) -> "Deferred[None]": ...
    def set(
        self,
@@ -52,11 +52,14 @@ def lazyConnection(
    convertNumbers: bool = ...,
 ) -> RedisProtocol: ...

-class ConnectionHandler: ...
+# ConnectionHandler doesn't actually inherit from RedisProtocol, but it proxies
+# most methods to it via ConnectionHandler.__getattr__.
+class ConnectionHandler(RedisProtocol):
+    def disconnect(self) -> "Deferred[None]": ...

 class RedisFactory(protocol.ReconnectingClientFactory):
    continueTrying: bool
-    handler: RedisProtocol
+    handler: ConnectionHandler
    pool: List[RedisProtocol]
    replyTimeout: Optional[int]
    def __init__(
@@ -25,6 +25,27 @@ if sys.version_info < (3, 7):
    print("Synapse requires Python 3.7 or above.")
    sys.exit(1)

+# Allow using the asyncio reactor via env var.
+if bool(os.environ.get("SYNAPSE_ASYNC_IO_REACTOR", False)):
+    try:
+        from incremental import Version
+
+        import twisted
+
+        # We need a bugfix that is included in Twisted 21.2.0:
+        # https://twistedmatrix.com/trac/ticket/9787
+        if twisted.version < Version("Twisted", 21, 2, 0):
+            print("Using asyncio reactor requires Twisted>=21.2.0")
+            sys.exit(1)
+
+        import asyncio
+
+        from twisted.internet import asyncioreactor
+
+        asyncioreactor.install(asyncio.get_event_loop())
+    except ImportError:
+        pass
+
 # Twisted and canonicaljson will fail to import when this file is executed to
 # get the __version__ during a fresh install. That's OK and subsequent calls to
 # actually start Synapse will import these libraries fine.
@@ -47,7 +68,7 @@ try:
 except ImportError:
    pass

-__version__ = "1.54.0rc1"
+__version__ = "1.54.0"

 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
    # We import here so that we don't have to install a bunch of deps when
@@ -178,7 +178,9 @@ class RelationTypes:
    ANNOTATION: Final = "m.annotation"
    REPLACE: Final = "m.replace"
    REFERENCE: Final = "m.reference"
-    THREAD: Final = "io.element.thread"
+    THREAD: Final = "m.thread"
+    # TODO Remove this in Synapse >= v1.57.0.
+    UNSTABLE_THREAD: Final = "io.element.thread"


 class LimitBlockingTypes:
@@ -88,7 +88,9 @@ ROOM_EVENT_FILTER_SCHEMA = {
        "org.matrix.labels": {"type": "array", "items": {"type": "string"}},
        "org.matrix.not_labels": {"type": "array", "items": {"type": "string"}},
        # MSC3440, filtering by event relations.
+        "related_by_senders": {"type": "array", "items": {"type": "string"}},
        "io.element.relation_senders": {"type": "array", "items": {"type": "string"}},
+        "related_by_rel_types": {"type": "array", "items": {"type": "string"}},
        "io.element.relation_types": {"type": "array", "items": {"type": "string"}},
    },
 }
@@ -318,19 +320,18 @@ class Filter:
        self.labels = filter_json.get("org.matrix.labels", None)
        self.not_labels = filter_json.get("org.matrix.not_labels", [])

-        # Ideally these would be rejected at the endpoint if they were provided
-        # and not supported, but that would involve modifying the JSON schema
-        # based on the homeserver configuration.
+        self.related_by_senders = self.filter_json.get("related_by_senders", None)
+        self.related_by_rel_types = self.filter_json.get("related_by_rel_types", None)
+
+        # Fallback to the unstable prefix if the stable version is not given.
        if hs.config.experimental.msc3440_enabled:
-            self.relation_senders = self.filter_json.get(
+            self.related_by_senders = self.related_by_senders or self.filter_json.get(
                "io.element.relation_senders", None
            )
-            self.relation_types = self.filter_json.get(
-                "io.element.relation_types", None
+            self.related_by_rel_types = (
+                self.related_by_rel_types
+                or self.filter_json.get("io.element.relation_types", None)
            )
-        else:
-            self.relation_senders = None
-            self.relation_types = None

    def filters_all_types(self) -> bool:
        return "*" in self.not_types
@@ -461,7 +462,7 @@ class Filter:
        event_ids = [event.event_id for event in events if isinstance(event, EventBase)]  # type: ignore[attr-defined]
        event_ids_to_keep = set(
            await self._store.events_have_relations(
-                event_ids, self.relation_senders, self.relation_types
+                event_ids, self.related_by_senders, self.related_by_rel_types
            )
        )

@@ -474,7 +475,7 @@ class Filter:
    async def filter(self, events: Iterable[FilterEvent]) -> List[FilterEvent]:
        result = [event for event in events if self._check(event)]

-        if self.relation_senders or self.relation_types:
+        if self.related_by_senders or self.related_by_rel_types:
            return await self._check_event_relations(result)

        return result
@@ -417,7 +417,7 @@ class GenericWorkerServer(HomeServer):
            else:
                logger.warning("Unsupported listener type: %s", listener.type)

-        self.get_tcp_replication().start_replication(self)
+        self.get_replication_command_handler().start_replication(self)


 def start(config_options: List[str]) -> None:
@@ -273,7 +273,7 @@ class SynapseHomeServer(HomeServer):
            # If redis is enabled we connect via the replication command handler
            # in the same way as the workers (since we're effectively a client
            # rather than a server).
-            self.get_tcp_replication().start_replication(self)
+            self.get_replication_command_handler().start_replication(self)

        for listener in self.config.server.listeners:
            if listener.type == "http":
@@ -182,8 +182,13 @@ class OIDCConfig(Config):
        #
        #             localpart_template: Jinja2 template for the localpart of the MXID.
        #                 If this is not set, the user will be prompted to choose their
-        #                 own username (see 'sso_auth_account_details.html' in the 'sso'
-        #                 section of this file).
+        #                 own username (see the documentation for the
+        #                 'sso_auth_account_details.html' template).
+        #
+        #             confirm_localpart: Whether to prompt the user to validate (or
+        #                 change) the generated localpart (see the documentation for the
+        #                 'sso_auth_account_details.html' template), instead of
+        #                 registering the account right away.
        #
        #             display_name_template: Jinja2 template for the display name to set
        #                 on first login. If unset, no displayname will be set.
@@ -245,8 +245,8 @@ class SpamChecker:
        """Checks if a given event is considered "spammy" by this server.

        If the server considers an event spammy, then it will be rejected if
-        sent by a local user. If it is sent by a user on another server, then
-        users receive a blank event.
+        sent by a local user. If it is sent by a user on another server, the
+        event is soft-failed.

        Args:
            event: the event to be checked
@@ -38,6 +38,8 @@ CHECK_VISIBILITY_CAN_BE_MODIFIED_CALLBACK = Callable[
    [str, StateMap[EventBase], str], Awaitable[bool]
 ]
 ON_NEW_EVENT_CALLBACK = Callable[[EventBase, StateMap[EventBase]], Awaitable]
+CHECK_CAN_SHUTDOWN_ROOM_CALLBACK = Callable[[str, str], Awaitable[bool]]
+CHECK_CAN_DEACTIVATE_USER_CALLBACK = Callable[[str, bool], Awaitable[bool]]
 ON_PROFILE_UPDATE_CALLBACK = Callable[[str, ProfileInfo, bool, bool], Awaitable]
 ON_USER_DEACTIVATION_STATUS_CHANGED_CALLBACK = Callable[[str, bool, bool], Awaitable]

@@ -157,6 +159,12 @@ class ThirdPartyEventRules:
            CHECK_VISIBILITY_CAN_BE_MODIFIED_CALLBACK
        ] = []
        self._on_new_event_callbacks: List[ON_NEW_EVENT_CALLBACK] = []
+        self._check_can_shutdown_room_callbacks: List[
+            CHECK_CAN_SHUTDOWN_ROOM_CALLBACK
+        ] = []
+        self._check_can_deactivate_user_callbacks: List[
+            CHECK_CAN_DEACTIVATE_USER_CALLBACK
+        ] = []
        self._on_profile_update_callbacks: List[ON_PROFILE_UPDATE_CALLBACK] = []
        self._on_user_deactivation_status_changed_callbacks: List[
            ON_USER_DEACTIVATION_STATUS_CHANGED_CALLBACK
@@ -173,8 +181,12 @@ class ThirdPartyEventRules:
            CHECK_VISIBILITY_CAN_BE_MODIFIED_CALLBACK
        ] = None,
        on_new_event: Optional[ON_NEW_EVENT_CALLBACK] = None,
+        check_can_shutdown_room: Optional[CHECK_CAN_SHUTDOWN_ROOM_CALLBACK] = None,
+        check_can_deactivate_user: Optional[CHECK_CAN_DEACTIVATE_USER_CALLBACK] = None,
        on_profile_update: Optional[ON_PROFILE_UPDATE_CALLBACK] = None,
-        on_deactivation: Optional[ON_USER_DEACTIVATION_STATUS_CHANGED_CALLBACK] = None,
+        on_user_deactivation_status_changed: Optional[
+            ON_USER_DEACTIVATION_STATUS_CHANGED_CALLBACK
+        ] = None,
    ) -> None:
        """Register callbacks from modules for each hook."""
        if check_event_allowed is not None:
@@ -196,11 +208,18 @@ class ThirdPartyEventRules:
        if on_new_event is not None:
            self._on_new_event_callbacks.append(on_new_event)

+        if check_can_shutdown_room is not None:
+            self._check_can_shutdown_room_callbacks.append(check_can_shutdown_room)
+
+        if check_can_deactivate_user is not None:
+            self._check_can_deactivate_user_callbacks.append(check_can_deactivate_user)
        if on_profile_update is not None:
            self._on_profile_update_callbacks.append(on_profile_update)

-        if on_deactivation is not None:
-            self._on_user_deactivation_status_changed_callbacks.append(on_deactivation)
+        if on_user_deactivation_status_changed is not None:
+            self._on_user_deactivation_status_changed_callbacks.append(
+                on_user_deactivation_status_changed,
+            )

    async def check_event_allowed(
        self, event: EventBase, context: EventContext
@@ -365,6 +384,46 @@ class ThirdPartyEventRules:
                    "Failed to run module API callback %s: %s", callback, e
                )

+    async def check_can_shutdown_room(self, user_id: str, room_id: str) -> bool:
+        """Intercept requests to shutdown a room. If `False` is returned, the
+         room must not be shut down.
+
+        Args:
+            requester: The ID of the user requesting the shutdown.
+            room_id: The ID of the room.
+        """
+        for callback in self._check_can_shutdown_room_callbacks:
+            try:
+                if await callback(user_id, room_id) is False:
+                    return False
+            except Exception as e:
+                logger.exception(
+                    "Failed to run module API callback %s: %s", callback, e
+                )
+        return True
+
+    async def check_can_deactivate_user(
+        self,
+        user_id: str,
+        by_admin: bool,
+    ) -> bool:
+        """Intercept requests to deactivate a user. If `False` is returned, the
+        user should not be deactivated.
+
+        Args:
+            requester
+            user_id: The ID of the room.
+        """
+        for callback in self._check_can_deactivate_user_callbacks:
+            try:
+                if await callback(user_id, by_admin) is False:
+                    return False
+            except Exception as e:
+                logger.exception(
+                    "Failed to run module API callback %s: %s", callback, e
+                )
+        return True
+
    async def _get_state_map_for_room(self, room_id: str) -> StateMap[EventBase]:
        """Given a room ID, return the state events of that room.

@@ -38,6 +38,7 @@ from synapse.util.frozenutils import unfreeze
 from . import EventBase

 if TYPE_CHECKING:
+    from synapse.server import HomeServer
    from synapse.storage.databases.main.relations import BundledAggregations


@@ -395,6 +396,9 @@ class EventClientSerializer:
    clients.
    """

+    def __init__(self, hs: "HomeServer"):
+        self._msc3440_enabled = hs.config.experimental.msc3440_enabled
+
    def serialize_event(
        self,
        event: Union[JsonDict, EventBase],
@@ -515,11 +519,14 @@ class EventClientSerializer:
                    thread.latest_event, serialized_latest_event, thread.latest_edit
                )

-            serialized_aggregations[RelationTypes.THREAD] = {
+            thread_summary = {
                "latest_event": serialized_latest_event,
                "count": thread.count,
                "current_user_participated": thread.current_user_participated,
            }
+            serialized_aggregations[RelationTypes.THREAD] = thread_summary
+            if self._msc3440_enabled:
+                serialized_aggregations[RelationTypes.UNSTABLE_THREAD] = thread_summary

        # Include the bundled aggregations in the event.
        if serialized_aggregations:
@@ -244,7 +244,7 @@ class FederationRemoteSendQueue(AbstractFederationSender):

        self.notifier.on_new_replication_data()

-    def send_device_messages(self, destination: str) -> None:
+    def send_device_messages(self, destination: str, immediate: bool = False) -> None:
        """As per FederationSender"""
        # We don't need to replicate this as it gets sent down a different
        # stream.
@@ -118,7 +118,12 @@ class AbstractFederationSender(metaclass=abc.ABCMeta):
        raise NotImplementedError()

    @abc.abstractmethod
-    def send_device_messages(self, destination: str) -> None:
+    def send_device_messages(self, destination: str, immediate: bool = True) -> None:
+        """Tells the sender that a new device message is ready to be sent to the
+        destination. The `immediate` flag specifies whether the messages should
+        be tried to be sent immediately, or whether it can be delayed for a
+        short while (to aid performance).
+        """
        raise NotImplementedError()

    @abc.abstractmethod
@@ -146,9 +151,8 @@ class AbstractFederationSender(metaclass=abc.ABCMeta):


@attr.s
-class _PresenceQueue:
-    """A queue of destinations that need to be woken up due to new presence
-    updates.
+class _DestinationWakeupQueue:
+    """A queue of destinations that need to be woken up due to new updates.

    Staggers waking up of per destination queues to ensure that we don't attempt
    to start TLS connections with many hosts all at once, leading to pinned CPU.
@@ -175,7 +179,7 @@ class _PresenceQueue:
        if not self.processing:
            self._handle()

-    @wrap_as_background_process("_PresenceQueue.handle")
+    @wrap_as_background_process("_DestinationWakeupQueue.handle")
    async def _handle(self) -> None:
        """Background process to drain the queue."""

@@ -297,7 +301,7 @@ class FederationSender(AbstractFederationSender):

        self._external_cache = hs.get_external_cache()

-        self._presence_queue = _PresenceQueue(self, self.clock)
+        self._destination_wakeup_queue = _DestinationWakeupQueue(self, self.clock)

    def _get_per_destination_queue(self, destination: str) -> PerDestinationQueue:
        """Get or create a PerDestinationQueue for the given destination
@@ -614,7 +618,7 @@ class FederationSender(AbstractFederationSender):
                states, start_loop=False
            )

-            self._presence_queue.add_to_queue(destination)
+            self._destination_wakeup_queue.add_to_queue(destination)

    def build_and_send_edu(
        self,
@@ -667,7 +671,7 @@ class FederationSender(AbstractFederationSender):
        else:
            queue.send_edu(edu)

-    def send_device_messages(self, destination: str) -> None:
+    def send_device_messages(self, destination: str, immediate: bool = False) -> None:
        if destination == self.server_name:
            logger.warning("Not sending device update to ourselves")
            return
@@ -677,7 +681,11 @@ class FederationSender(AbstractFederationSender):
        ):
            return

-        self._get_per_destination_queue(destination).attempt_new_transaction()
+        if immediate:
+            self._get_per_destination_queue(destination).attempt_new_transaction()
+        else:
+            self._get_per_destination_queue(destination).mark_new_data()
+            self._destination_wakeup_queue.add_to_queue(destination)

    def wake_destination(self, destination: str) -> None:
        """Called when we want to retry sending transactions to a remote.
@@ -219,6 +219,16 @@ class PerDestinationQueue:
        self._pending_edus.append(edu)
        self.attempt_new_transaction()

+    def mark_new_data(self) -> None:
+        """Marks that the destination has new data to send, without starting a
+        new transaction.
+
+        If a transaction loop is already in progress then a new transcation will
+        be attempted when the current one finishes.
+        """
+
+        self._new_data_to_send = True
+
    def attempt_new_transaction(self) -> None:
        """Try to start a new transaction to this destination

@@ -63,7 +63,7 @@ class Authenticator:

        self.replication_client = None
        if hs.config.worker.worker_app:
-            self.replication_client = hs.get_tcp_replication()
+            self.replication_client = hs.get_replication_command_handler()

    # A method just so we can pass 'self' as the authenticator to the Servlets
    async def authenticate_request(
@@ -17,7 +17,7 @@ from typing import TYPE_CHECKING, Optional

 from synapse.api.errors import SynapseError
 from synapse.metrics.background_process_metrics import run_as_background_process
-from synapse.types import Requester, UserID, create_requester
+from synapse.types import Codes, Requester, UserID, create_requester

 if TYPE_CHECKING:
    from synapse.server import HomeServer
@@ -42,6 +42,7 @@ class DeactivateAccountHandler:

        # Flag that indicates whether the process to part users from rooms is running
        self._user_parter_running = False
+        self._third_party_rules = hs.get_third_party_event_rules()

        # Start the user parter loop so it can resume parting users from rooms where
        # it left off (if it has work left to do).
@@ -74,6 +75,15 @@ class DeactivateAccountHandler:
        Returns:
            True if identity server supports removing threepids, otherwise False.
        """
+
+        # Check if this user can be deactivated
+        if not await self._third_party_rules.check_can_deactivate_user(
+            user_id, by_admin
+        ):
+            raise SynapseError(
+                403, "Deactivation of this user is forbidden", Codes.FORBIDDEN
+            )
+
        # FIXME: Theoretically there is a race here wherein user resets
        # password using threepid.

@@ -371,7 +371,6 @@ class DeviceHandler(DeviceWorkerHandler):
                log_kv(
                    {"reason": "User doesn't have device id.", "device_id": device_id}
                )
-                pass
            else:
                raise

@@ -414,7 +413,6 @@ class DeviceHandler(DeviceWorkerHandler):
                # no match
                set_tag("error", True)
                set_tag("reason", "User doesn't have that device id.")
-                pass
            else:
                raise

@@ -506,7 +504,7 @@ class DeviceHandler(DeviceWorkerHandler):
                "Sending device list update notif for %r to: %r", user_id, hosts
            )
            for host in hosts:
-                self.federation_sender.send_device_messages(host)
+                self.federation_sender.send_device_messages(host, immediate=False)
                log_kv({"message": "sent device update to host", "host": host})

    async def notify_user_signature_update(
@@ -23,8 +23,6 @@ from signedjson.key import decode_verify_key_bytes
 from signedjson.sign import verify_signed_json
 from unpaddedbase64 import decode_base64

-from twisted.internet import defer
-
 from synapse import event_auth
 from synapse.api.constants import EventContentFields, EventTypes, Membership
 from synapse.api.errors import (
@@ -45,11 +43,7 @@ from synapse.events.snapshot import EventContext
 from synapse.events.validator import EventValidator
 from synapse.federation.federation_client import InvalidResponseError
 from synapse.http.servlet import assert_params_in_dict
-from synapse.logging.context import (
-    make_deferred_yieldable,
-    nested_logging_context,
-    preserve_fn,
-)
+from synapse.logging.context import nested_logging_context
 from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.replication.http.federation import (
    ReplicationCleanRoomRestServlet,
@@ -355,56 +349,8 @@ class FederationHandler:
        if success:
            return True

-        # Huh, well *those* domains didn't work out. Lets try some domains
-        # from the time.
-
-        tried_domains = set(likely_domains)
-        tried_domains.add(self.server_name)
-
-        event_ids = list(extremities.keys())
-
-        logger.debug("calling resolve_state_groups in _maybe_backfill")
-        resolve = preserve_fn(self.state_handler.resolve_state_groups_for_events)
-        states_list = await make_deferred_yieldable(
-            defer.gatherResults(
-                [resolve(room_id, [e]) for e in event_ids], consumeErrors=True
-            )
-        )
-
-        # A map from event_id to state map of event_ids.
-        state_ids: Dict[str, StateMap[str]] = dict(
-            zip(event_ids, [s.state for s in states_list])
-        )
-
-        state_map = await self.store.get_events(
-            [e_id for ids in state_ids.values() for e_id in ids.values()],
-            get_prev_content=False,
-        )
-
-        # A map from event_id to state map of events.
-        state_events: Dict[str, StateMap[EventBase]] = {
-            key: {
-                k: state_map[e_id]
-                for k, e_id in state_dict.items()
-                if e_id in state_map
-            }
-            for key, state_dict in state_ids.items()
-        }
-
-        for e_id in event_ids:
-            likely_extremeties_domains = get_domains_from_state(state_events[e_id])
-
-            success = await try_backfill(
-                [
-                    dom
-                    for dom, _ in likely_extremeties_domains
-                    if dom not in tried_domains
-                ]
-            )
-            if success:
-                return True
-
-            tried_domains.update(dom for dom, _ in likely_extremeties_domains)
+        # TODO: we could also try servers which were previously in the room, but
+        #   are no longer.

        return False

@@ -153,8 +153,9 @@ class InitialSyncHandler:

        public_room_ids = await self.store.get_public_room_ids()

-        limit = pagin_config.limit
-        if limit is None:
+        if pagin_config.limit is not None:
+            limit = pagin_config.limit
+        else:
            limit = 10

        serializer_options = SerializeEventConfig(as_client_event=as_client_event)
@@ -1079,7 +1079,10 @@ class EventCreationHandler:
                raise SynapseError(400, "Can't send same reaction twice")

        # Don't attempt to start a thread if the parent event is a relation.
-        elif relation_type == RelationTypes.THREAD:
+        elif (
+            relation_type == RelationTypes.THREAD
+            or relation_type == RelationTypes.UNSTABLE_THREAD
+        ):
            if await self.store.event_includes_relation(relates_to):
                raise SynapseError(
                    400, "Cannot start threads from an event with a relation"
@@ -1228,6 +1228,7 @@ class OidcSessionData:

 class UserAttributeDict(TypedDict):
    localpart: Optional[str]
+    confirm_localpart: bool
    display_name: Optional[str]
    emails: List[str]

@@ -1316,6 +1317,7 @@ class JinjaOidcMappingConfig:
    display_name_template: Optional[Template]
    email_template: Optional[Template]
    extra_attributes: Dict[str, Template]
+    confirm_localpart: bool = False


 class JinjaOidcMappingProvider(OidcMappingProvider[JinjaOidcMappingConfig]):
@@ -1357,12 +1359,17 @@ class JinjaOidcMappingProvider(OidcMappingProvider[JinjaOidcMappingConfig]):
                        "invalid jinja template", path=["extra_attributes", key]
                    ) from e

+        confirm_localpart = config.get("confirm_localpart") or False
+        if not isinstance(confirm_localpart, bool):
+            raise ConfigError("must be a bool", path=["confirm_localpart"])
+
        return JinjaOidcMappingConfig(
            subject_claim=subject_claim,
            localpart_template=localpart_template,
            display_name_template=display_name_template,
            email_template=email_template,
            extra_attributes=extra_attributes,
+            confirm_localpart=confirm_localpart,
        )

    def get_remote_user_id(self, userinfo: UserInfo) -> str:
@@ -1398,7 +1405,10 @@ class JinjaOidcMappingProvider(OidcMappingProvider[JinjaOidcMappingConfig]):
            emails.append(email)

        return UserAttributeDict(
-            localpart=localpart, display_name=display_name, emails=emails
+            localpart=localpart,
+            display_name=display_name,
+            emails=emails,
+            confirm_localpart=self._config.confirm_localpart,
        )

    async def get_extra_attributes(self, userinfo: UserInfo, token: Token) -> JsonDict:
@@ -267,7 +267,6 @@ class BasePresenceHandler(abc.ABC):
            is_syncing: Whether or not the user is now syncing
            sync_time_msec: Time in ms when the user was last syncing
        """
-        pass

    async def update_external_syncs_clear(self, process_id: str) -> None:
        """Marks all users that had been marked as syncing by a given process
@@ -277,7 +276,6 @@ class BasePresenceHandler(abc.ABC):

        This is a no-op when presence is handled by a different worker.
        """
-        pass

    async def process_replication_rows(
        self, stream_name: str, instance_name: str, token: int, rows: list
@@ -424,13 +422,13 @@ class WorkerPresenceHandler(BasePresenceHandler):

    async def _on_shutdown(self) -> None:
        if self._presence_enabled:
-            self.hs.get_tcp_replication().send_command(
+            self.hs.get_replication_command_handler().send_command(
                ClearUserSyncsCommand(self.instance_id)
            )

    def send_user_sync(self, user_id: str, is_syncing: bool, last_sync_ms: int) -> None:
        if self._presence_enabled:
-            self.hs.get_tcp_replication().send_user_sync(
+            self.hs.get_replication_command_handler().send_user_sync(
                self.instance_id, user_id, is_syncing, last_sync_ms
            )

@@ -1475,6 +1475,7 @@ class RoomShutdownHandler:
        self.room_member_handler = hs.get_room_member_handler()
        self._room_creation_handler = hs.get_room_creation_handler()
        self._replication = hs.get_replication_data_handler()
+        self._third_party_rules = hs.get_third_party_event_rules()
        self.event_creation_handler = hs.get_event_creation_handler()
        self.store = hs.get_datastores().main

@@ -1548,6 +1549,13 @@ class RoomShutdownHandler:
        if not RoomID.is_valid(room_id):
            raise SynapseError(400, "%s is not a legal room ID" % (room_id,))

+        if not await self._third_party_rules.check_can_shutdown_room(
+            requester_user_id, room_id
+        ):
+            raise SynapseError(
+                403, "Shutdown of this room is forbidden", Codes.FORBIDDEN
+            )
+
        # Action the block first (even if the room doesn't exist yet)
        if block:
            # This will work even if the room is already blocked, but that is
@@ -295,7 +295,7 @@ class RoomSummaryHandler:
            # inaccessible to the requesting user.
            if room_entry:
                # Add the room (including the stripped m.space.child events).
-                rooms_result.append(room_entry.as_json())
+                rooms_result.append(room_entry.as_json(for_client=True))

                # If this room is not at the max-depth, check if there are any
                # children to process.
@@ -843,14 +843,25 @@ class _RoomEntry:
    # This may not include all children.
    children_state_events: Sequence[JsonDict] = ()

-    def as_json(self) -> JsonDict:
+    def as_json(self, for_client: bool = False) -> JsonDict:
        """
        Returns a JSON dictionary suitable for the room hierarchy endpoint.

        It returns the room summary including the stripped m.space.child events
        as a sub-key.
+
+        Args:
+            for_client: If true, any server-server only fields are stripped from
+                the result.
+
        """
        result = dict(self.room)
+
+        # Before returning to the client, remove the allowed_room_ids key, if it
+        # exists.
+        if for_client:
+            result.pop("allowed_room_ids", False)
+
        result["children_state"] = self.children_state_events
        return result

@@ -132,6 +132,7 @@ class UserAttributes:
    # if `None`, the mapper has not picked a userid, and the user should be prompted to
    # enter one.
    localpart: Optional[str]
+    confirm_localpart: bool = False
    display_name: Optional[str] = None
    emails: Collection[str] = attr.Factory(list)

@@ -561,9 +562,10 @@ class SsoHandler:
        # Must provide either attributes or session, not both
        assert (attributes is not None) != (session is not None)

-        if (attributes and attributes.localpart is None) or (
-            session and session.chosen_localpart is None
-        ):
+        if (
+            attributes
+            and (attributes.localpart is None or attributes.confirm_localpart is True)
+        ) or (session and session.chosen_localpart is None):
            return b"/_synapse/client/pick_username/account_details"
        elif self._consent_at_registration and not (
            session and session.terms_accepted_version
@@ -120,7 +120,6 @@ class ByteParser(ByteWriteable, Generic[T], abc.ABC):
        """Called when response has finished streaming and the parser should
        return the final result (or error).
        """
-        pass


@attr.s(slots=True, frozen=True, auto_attribs=True)
@@ -601,7 +600,6 @@ class MatrixFederationHttpClient:
                            response.code,
                            response_phrase,
                        )
-                        pass
                    else:
                        logger.info(
                            "{%s} [%s] Got response headers: %d %s",
@@ -233,7 +233,6 @@ class HttpServer(Protocol):
            servlet_classname (str): The name of the handler to be used in prometheus
                and opentracing logs.
        """
-        pass


 class _AsyncResource(resource.Resource, metaclass=abc.ABCMeta):
@@ -29,7 +29,6 @@ import warnings
 from types import TracebackType
 from typing import (
    TYPE_CHECKING,
-    Any,
    Awaitable,
    Callable,
    Optional,
@@ -41,7 +40,7 @@ from typing import (
 )

 import attr
-from typing_extensions import Literal
+from typing_extensions import Literal, ParamSpec

 from twisted.internet import defer, threads
 from twisted.python.threadpool import ThreadPool
@@ -719,32 +718,33 @@ def nested_logging_context(suffix: str) -> LoggingContext:
    )


+P = ParamSpec("P")
 R = TypeVar("R")


@overload
 def preserve_fn(  # type: ignore[misc]
-    f: Callable[..., Awaitable[R]],
-) -> Callable[..., "defer.Deferred[R]"]:
+    f: Callable[P, Awaitable[R]],
+) -> Callable[P, "defer.Deferred[R]"]:
    # The `type: ignore[misc]` above suppresses
    # "Overloaded function signatures 1 and 2 overlap with incompatible return types"
    ...


@overload
-def preserve_fn(f: Callable[..., R]) -> Callable[..., "defer.Deferred[R]"]:
+def preserve_fn(f: Callable[P, R]) -> Callable[P, "defer.Deferred[R]"]:
    ...


 def preserve_fn(
    f: Union[
-        Callable[..., R],
-        Callable[..., Awaitable[R]],
+        Callable[P, R],
+        Callable[P, Awaitable[R]],
    ]
-) -> Callable[..., "defer.Deferred[R]"]:
+) -> Callable[P, "defer.Deferred[R]"]:
    """Function decorator which wraps the function with run_in_background"""

-    def g(*args: Any, **kwargs: Any) -> "defer.Deferred[R]":
+    def g(*args: P.args, **kwargs: P.kwargs) -> "defer.Deferred[R]":
        return run_in_background(f, *args, **kwargs)

    return g
@@ -752,7 +752,7 @@ def preserve_fn(

@overload
 def run_in_background(  # type: ignore[misc]
-    f: Callable[..., Awaitable[R]], *args: Any, **kwargs: Any
+    f: Callable[P, Awaitable[R]], *args: P.args, **kwargs: P.kwargs
 ) -> "defer.Deferred[R]":
    # The `type: ignore[misc]` above suppresses
    # "Overloaded function signatures 1 and 2 overlap with incompatible return types"
@@ -761,18 +761,22 @@ def run_in_background(  # type: ignore[misc]

@overload
 def run_in_background(
-    f: Callable[..., R], *args: Any, **kwargs: Any
+    f: Callable[P, R], *args: P.args, **kwargs: P.kwargs
 ) -> "defer.Deferred[R]":
    ...


-def run_in_background(
+def run_in_background(  # type: ignore[misc]
+    # The `type: ignore[misc]` above suppresses
+    # "Overloaded function implementation does not accept all possible arguments of signature 1"
+    # "Overloaded function implementation does not accept all possible arguments of signature 2"
+    # which seems like a bug in mypy.
    f: Union[
-        Callable[..., R],
-        Callable[..., Awaitable[R]],
+        Callable[P, R],
+        Callable[P, Awaitable[R]],
    ],
-    *args: Any,
-    **kwargs: Any,
+    *args: P.args,
+    **kwargs: P.kwargs,
 ) -> "defer.Deferred[R]":
    """Calls a function, ensuring that the current context is restored after
    return from the function, and that the sentinel context is set once the
@@ -872,7 +876,7 @@ def _set_context_cb(result: ResultT, context: LoggingContext) -> ResultT:


 def defer_to_thread(
-    reactor: "ISynapseReactor", f: Callable[..., R], *args: Any, **kwargs: Any
+    reactor: "ISynapseReactor", f: Callable[P, R], *args: P.args, **kwargs: P.kwargs
 ) -> "defer.Deferred[R]":
    """
    Calls the function `f` using a thread from the reactor's default threadpool and
@@ -908,9 +912,9 @@ def defer_to_thread(
 def defer_to_threadpool(
    reactor: "ISynapseReactor",
    threadpool: ThreadPool,
-    f: Callable[..., R],
-    *args: Any,
-    **kwargs: Any,
+    f: Callable[P, R],
+    *args: P.args,
+    **kwargs: P.kwargs,
 ) -> "defer.Deferred[R]":
    """
    A wrapper for twisted.internet.threads.deferToThreadpool, which handles
@@ -54,11 +54,15 @@ from synapse.events.spamcheck import (
    USER_MAY_SEND_3PID_INVITE_CALLBACK,
 )
 from synapse.events.third_party_rules import (
+    CHECK_CAN_DEACTIVATE_USER_CALLBACK,
+    CHECK_CAN_SHUTDOWN_ROOM_CALLBACK,
    CHECK_EVENT_ALLOWED_CALLBACK,
    CHECK_THREEPID_CAN_BE_INVITED_CALLBACK,
    CHECK_VISIBILITY_CAN_BE_MODIFIED_CALLBACK,
    ON_CREATE_ROOM_CALLBACK,
    ON_NEW_EVENT_CALLBACK,
+    ON_PROFILE_UPDATE_CALLBACK,
+    ON_USER_DEACTIVATION_STATUS_CHANGED_CALLBACK,
 )
 from synapse.handlers.account_validity import (
    IS_USER_EXPIRED_CALLBACK,
@@ -281,6 +285,12 @@ class ModuleApi:
            CHECK_VISIBILITY_CAN_BE_MODIFIED_CALLBACK
        ] = None,
        on_new_event: Optional[ON_NEW_EVENT_CALLBACK] = None,
+        check_can_shutdown_room: Optional[CHECK_CAN_SHUTDOWN_ROOM_CALLBACK] = None,
+        check_can_deactivate_user: Optional[CHECK_CAN_DEACTIVATE_USER_CALLBACK] = None,
+        on_profile_update: Optional[ON_PROFILE_UPDATE_CALLBACK] = None,
+        on_user_deactivation_status_changed: Optional[
+            ON_USER_DEACTIVATION_STATUS_CHANGED_CALLBACK
+        ] = None,
    ) -> None:
        """Registers callbacks for third party event rules capabilities.

@@ -292,6 +302,10 @@ class ModuleApi:
            check_threepid_can_be_invited=check_threepid_can_be_invited,
            check_visibility_can_be_modified=check_visibility_can_be_modified,
            on_new_event=on_new_event,
+            check_can_shutdown_room=check_can_shutdown_room,
+            check_can_deactivate_user=check_can_deactivate_user,
+            on_profile_update=on_profile_update,
+            on_user_deactivation_status_changed=on_user_deactivation_status_changed,
        )

    def register_presence_router_callbacks(
@@ -76,15 +76,16 @@ REQUIREMENTS = [
    "netaddr>=0.7.18",
    "Jinja2>=2.9",
    "bleach>=1.4.3",
-    "typing-extensions>=3.7.4",
+    # We use `ParamSpec`, which was added in `typing-extensions` 3.10.0.0.
+    "typing-extensions>=3.10.0",
    # We enforce that we have a `cryptography` version that bundles an `openssl`
    # with the latest security patches.
    "cryptography>=3.4.7",
    # ijson 3.1.4 fixes a bug with "." in property names
    "ijson>=3.1.4",
    "matrix-common~=1.1.0",
-    # For runtime introspection of our dependencies
-    "packaging~=21.3",
+    # We need packaging.requirements.Requirement, added in 16.1.
+    "packaging>=16.1",
 ]

 CONDITIONAL_REQUIREMENTS = {
@@ -21,6 +21,7 @@ from typing import TYPE_CHECKING, Any, Awaitable, Callable, Dict, List, Tuple

 from prometheus_client import Counter, Gauge

+from twisted.internet.error import ConnectError, DNSLookupError
 from twisted.web.server import Request

 from synapse.api.errors import HttpResponseException, SynapseError
@@ -87,6 +88,10 @@ class ReplicationEndpoint(metaclass=abc.ABCMeta):
            `_handle_request` must return a Deferred.
        RETRY_ON_TIMEOUT(bool): Whether or not to retry the request when a 504
            is received.
+        RETRY_ON_CONNECT_ERROR (bool): Whether or not to retry the request when
+            a connection error is received.
+        RETRY_ON_CONNECT_ERROR_ATTEMPTS (int): Number of attempts to retry when
+            receiving connection errors, each will backoff exponentially longer.
    """

    NAME: str = abc.abstractproperty()  # type: ignore
@@ -94,6 +99,8 @@ class ReplicationEndpoint(metaclass=abc.ABCMeta):
    METHOD = "POST"
    CACHE = True
    RETRY_ON_TIMEOUT = True
+    RETRY_ON_CONNECT_ERROR = True
+    RETRY_ON_CONNECT_ERROR_ATTEMPTS = 5  # =63s (2^6-1)

    def __init__(self, hs: "HomeServer"):
        if self.CACHE:
@@ -236,18 +243,20 @@ class ReplicationEndpoint(metaclass=abc.ABCMeta):
                    "/".join(url_args),
                )

+                headers: Dict[bytes, List[bytes]] = {}
+                # Add an authorization header, if configured.
+                if replication_secret:
+                    headers[b"Authorization"] = [b"Bearer " + replication_secret]
+                opentracing.inject_header_dict(headers, check_destination=False)
+
                try:
+                    # Keep track of attempts made so we can bail if we don't manage to
+                    # connect to the target after N tries.
+                    attempts = 0
                    # We keep retrying the same request for timeouts. This is so that we
                    # have a good idea that the request has either succeeded or failed
                    # on the master, and so whether we should clean up or not.
                    while True:
-                        headers: Dict[bytes, List[bytes]] = {}
-                        # Add an authorization header, if configured.
-                        if replication_secret:
-                            headers[b"Authorization"] = [
-                                b"Bearer " + replication_secret
-                            ]
-                        opentracing.inject_header_dict(headers, check_destination=False)
                        try:
                            result = await request_func(uri, data, headers=headers)
                            break
@@ -255,11 +264,27 @@ class ReplicationEndpoint(metaclass=abc.ABCMeta):
                            if not cls.RETRY_ON_TIMEOUT:
                                raise

-                        logger.warning("%s request timed out; retrying", cls.NAME)
+                            logger.warning("%s request timed out; retrying", cls.NAME)

-                        # If we timed out we probably don't need to worry about backing
-                        # off too much, but lets just wait a little anyway.
-                        await clock.sleep(1)
+                            # If we timed out we probably don't need to worry about backing
+                            # off too much, but lets just wait a little anyway.
+                            await clock.sleep(1)
+                        except (ConnectError, DNSLookupError) as e:
+                            if not cls.RETRY_ON_CONNECT_ERROR:
+                                raise
+                            if attempts > cls.RETRY_ON_CONNECT_ERROR_ATTEMPTS:
+                                raise
+
+                            delay = 2 ** attempts
+                            logger.warning(
+                                "%s request connection failed; retrying in %ds: %r",
+                                cls.NAME,
+                                delay,
+                                e,
+                            )
+
+                            await clock.sleep(delay)
+                            attempts += 1
                except HttpResponseException as e:
                    # We convert to SynapseError as we know that it was a SynapseError
                    # on the main process that we should send to the client. (And
@@ -54,6 +54,6 @@ class SlavedClientIpStore(BaseSlavedStore):

        self.client_ip_last_seen.set(key, now)

-        self.hs.get_tcp_replication().send_user_ip(
+        self.hs.get_replication_command_handler().send_user_ip(
            user_id, access_token, ip, user_agent, device_id, now
        )
@@ -380,7 +380,7 @@ class FederationSenderHandler:
            # changes.
            hosts = {row.entity for row in rows if not row.entity.startswith("@")}
            for host in hosts:
-                self.federation_sender.send_device_messages(host)
+                self.federation_sender.send_device_messages(host, immediate=False)

        elif stream_name == ToDeviceStream.NAME:
            # The to_device stream includes stuff to be pushed to both local
@@ -462,6 +462,8 @@ class FederationSenderHandler:

                # We ACK this token over replication so that the master can drop
                # its in memory queues
-                self._hs.get_tcp_replication().send_federation_ack(current_position)
+                self._hs.get_replication_command_handler().send_federation_ack(
+                    current_position
+                )
        except Exception:
            logger.exception("Error updating federation stream position")
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Patrick Cloke	e0b60a9b4e	temp	2022-03-11 11:26:03 -05:00
Patrick Cloke	b32bb82bee	temp	2022-03-11 10:35:23 -05:00
Patrick Cloke	829139c3d5	Attempt to re-connect better.	2022-03-11 10:35:22 -05:00
Patrick Cloke	7375bd4828	More robust-ness against dying connections.	2022-03-11 10:34:59 -05:00
Erik Johnston	fd491969a6	Attempt some progress	2022-03-11 10:34:58 -05:00
Patrick Cloke	0f3798dac7	Rip out TCP replication bits for tests and hook up Redis replication.	2022-03-11 10:33:58 -05:00
Patrick Cloke	9e1dfc68fd	Use redis for all replication tests.	2022-03-11 10:33:58 -05:00
Patrick Cloke	b7d7a1b0a8	Respond to Redis PING messages.	2022-03-11 10:33:58 -05:00
Patrick Cloke	e545948eef	Use the reactor from the HomeServer.	2022-03-11 10:33:58 -05:00
Brendan Abolivier	003cc6910a	Update the SSO username picker template to comply with SIWA guidelines (#12210 ) Fixes https://github.com/matrix-org/synapse/issues/12205	2022-03-11 13:20:00 +00:00
Dirk Klimpel	32c828d0f7	Add type hints to `tests/rest`. (#12208 ) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>	2022-03-11 12:42:22 +00:00
Patrick Cloke	e10a2fe0c2	Add some type hints to the tests.handlers module. (#12207 )	2022-03-11 07:07:15 -05:00
Patrick Cloke	bc9dff1d95	Remove unnecessary pass statements. (#12206 )	2022-03-11 07:06:21 -05:00
Andrew Morgan	3b12f6d61b	Note that contributors can sign off privately (#12204 ) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>	2022-03-11 11:10:20 +00:00
Richard van der Hoff	483f2aa2ec	Retention test: avoid relying on state at purged events (#12202 ) This test was relying on poking events which weren't in the database into filter_events_for_client.	2022-03-11 10:33:49 +00:00
~creme	7577894bec	Document that most streams can only have a single writer. (#12196 ) This includes the `typing`, `to_device`, `account_data`, `receipts`, and `presence` streams (really anything except the `events` stream).	2022-03-10 18:15:19 +00:00
Shay	ed9aea42fa	fix misleading comment in `check_events_for_spam` (#12203 )	2022-03-10 09:40:07 -08:00
reivilibre	72e7f1c420	Remove workaround introduced in Synapse v1.50.0rc1 for Mjolnir compatibility. Breaks compatibility with Mjolnir v1.3.1 and earlier. (#11700 )	2022-03-10 15:53:23 +00:00
Patrick Cloke	ea27528b5d	Support stable identifiers for MSC3440: Threading (#12151 ) The unstable identifiers are still supported if the experimental configuration flag is enabled. The unstable identifiers will be removed in a future release.	2022-03-10 15:36:13 +00:00
Richard van der Hoff	52a947dc46	Updates to the Room DAG concepts development document (#12179 ) Some stuff that came up while we were talking about #12173.	2022-03-10 15:18:31 +00:00
Patrick Cloke	88cd6f9378	Allow retrieving the relations of a redacted event. (#12130 ) This is allowed per MSC2675, although the original implementation did not allow for it and would return an empty chunk / not bundle aggregations. The main thing to improve is that the various caches get cleared properly when an event is redacted, and that edits must not leak if the original event is redacted (as that would presumably leak something similar to the original event content).	2022-03-10 09:03:59 -05:00
Patrick Cloke	3e4af36bc8	Rename get_tcp_replication to get_replication_command_handler. (#12192 ) Since the object it returns is a ReplicationCommandHandler. This is clean-up from adding support to Redis where the command handler was added as an additional layer of abstraction from the TCP protocol.	2022-03-10 13:01:56 +00:00
Sean Quah	a4c1fdb44a	Remove dead code in `tests/storage/test_database.py` (#12197 ) Signed-off-by: Sean Quah <seanq@element.io>	2022-03-09 18:45:21 +00:00
Will Hunt	15382b1afa	Add third_party module callbacks to check if a user can delete a room and deactivate a user (#12028 ) * Add check_can_deactivate_user * Add check_can_shutdown_rooms * Documentation * callbacks, not functions * Various suggested tweaks * Add tests for test_check_can_shutdown_room and test_check_can_deactivate_user * Update check_can_deactivate_user to not take a Requester * Fix check_can_shutdown_room docs * Renegade and use `by_admin` instead of `admin_user_id` * fix lint * Update docs/modules/third_party_rules_callbacks.md Co-authored-by: Brendan Abolivier <babolivier@matrix.org> * Update docs/modules/third_party_rules_callbacks.md Co-authored-by: Brendan Abolivier <babolivier@matrix.org> * Update docs/modules/third_party_rules_callbacks.md Co-authored-by: Brendan Abolivier <babolivier@matrix.org> * Update docs/modules/third_party_rules_callbacks.md Co-authored-by: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>	2022-03-09 18:23:57 +00:00
Patrick Cloke	690cb4f3b3	Allow for ignoring some arguments when caching. (#12189 ) * `@cached` can now take an `uncached_args` which is an iterable of names to not use in the cache key. * Requires `@cached`, @cachedList` and `@lru_cache` to use keyword arguments for clarity. * Asserts that keyword-only arguments in cached functions are not accepted. (I tested this briefly and I don't believe this works properly.)	2022-03-09 18:07:41 +00:00
Patrick Cloke	032688854b	Remove some unused variables/parameters. (#12187 )	2022-03-09 15:29:39 +00:00
Nick Mills-Barrett	180d8ff0d4	Retry some http replication failures (#12182 ) This allows for the target process to be down for around a minute which provides time for restarts during synapse upgrades/config updates. Closes: #12178 Signed off by Nick Mills-Barrett nick@beeper.com	2022-03-09 14:53:28 +00:00
Richard van der Hoff	dc8d825ef2	Skip attempt to get state at backwards-extremities (#12173 ) We don't have the state at a backwards-extremity, so this is never going to do anything useful.	2022-03-09 11:00:48 +00:00
Patrick Cloke	9a0172d49f	Clean-up demo scripts & documentation (#12143 ) * Rewrites the demo documentation to be clearer, accurate, and moves it to our documentation tree. * Improvements to the demo scripts: * `clean.sh` now runs `stop.sh` first to avoid zombie processes. * Uses more modern Synapse configuration (and removes some obsolete configuration). * Consistently use the HTTP ports for server name, etc. * Remove the `demo/etc` directory and place everything into the `demo/808x` directories.	2022-03-08 15:02:59 -05:00
Sean Quah	5627182788	Use `ParamSpec` in type hints for `synapse.logging.context` (#12150 ) Signed-off-by: Sean Quah <seanq@element.io>	2022-03-08 15:58:14 +00:00
Olivier Wilkinson (reivilibre)	0dc9c5653c	Merge branch 'master' into develop	2022-03-08 15:37:35 +00:00
reivilibre	bfa7d6b035	Fix CI not attaching source distributions and wheels to the GitHub releases. (#12131 )	2022-03-08 15:11:50 +00:00
Olivier Wilkinson (reivilibre)	b1989ced00	Fix silly markdown typo	2022-03-08 14:01:19 +00:00
Olivier Wilkinson (reivilibre)	65e02b3e6d	Tweak changelog formatting	2022-03-08 14:00:16 +00:00
Erik Johnston	2ce27a24fe	Add experimental environment variable to enable asyncio reactor (#12135 )	2022-03-08 13:23:18 +00:00
Patrick Cloke	ca9234a9eb	Do not return allowed_room_ids from /hierarchy response. (#12175 ) This field is only to be used in the Server-Server API, and not the Client-Server API, but was being leaked when a federation response was used in the /hierarchy API.	2022-03-08 08:09:11 -05:00
Patrick Cloke	d8bab6793c	Fix incorrect type hints for txredis. (#12042 ) Some properties were marked as RedisProtocol instead of ConnectionHandler, which wraps RedisProtocol instance(s).	2022-03-08 07:26:05 -05:00
Olivier Wilkinson (reivilibre)	094802e04e	Shift up warning about Mjolnir	2022-03-08 10:58:10 +00:00
Olivier Wilkinson (reivilibre)	ea992adf86	1.54.0	2022-03-08 10:55:26 +00:00
reivilibre	2eef234ae3	Fix a bug introduced in 1.54.0rc1 which meant that Synapse would refuse to start if pre-release versions of dependencies were installed. (#12177 ) * Add failing test to characterise the regression #12176 * Permit pre-release versions of specified packages * Newsfile (bugfix) Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>	2022-03-08 10:47:28 +00:00
Shay	26211fec24	Fix a bug in background updates wherein background updates are never run using the default batch size (#12157 )	2022-03-07 09:44:33 -08:00
Patrick Cloke	f63bedef07	Invalidate caches when an event with a relation is redacted. (#12121 ) The caches for the target of the relation must be cleared so that the bundled aggregations are re-calculated after the redaction is processed.	2022-03-07 14:00:05 +00:00
Richard van der Hoff	0211f18d65	Switch the `tests-done` job to an Action (#12161 ) I've factored it out for easier use in other workflows.	2022-03-07 12:24:06 +00:00
Richard van der Hoff	00a67f831a	Merge remote-tracking branch 'origin/release-v1.54' into develop	2022-03-04 22:40:51 +00:00
David Robertson	d2ef1a79cf	Relax version guard for packaging (#12166 ) It’s just occurred to me that #12088 pulled in the “packaging” package (~=21.3). I pulled in the newest version I had at the time. I only use it for packaging.requirements.Requirements. Which was added in packaging 16.1: https://github.com/pypa/packaging/releases/tag/16.1 https://pkgs.org/download/python3-packaging suggests that the oldest version we care about is 17.1 in Ubuntu Bionic. So I think with this bound we're hunky dory.	2022-03-04 22:40:24 +00:00
Erik Johnston	0752ab7a36	Reduce to-device queries for /sync. (#12163 )	2022-03-04 17:57:27 +00:00
Sean Quah	75574726a7	Add type hints for `ObservableDeferred` attributes (#12159 ) Signed-off-by: Sean Quah <seanq@element.io>	2022-03-04 15:37:02 +00:00
Sean Quah	158e0937eb	Add test for `ObservableDeferred`'s cancellation behaviour (#12149 ) Signed-off-by: Sean Quah <seanq@element.io>	2022-03-04 13:10:05 +00:00
Patrick Cloke	cd1ae3d0b4	Remove backwards compatibility with RelationPaginationToken. (#12138 )	2022-03-04 07:10:10 -05:00
David Robertson	36071d39f7	Changelog (#12153 )	2022-03-04 12:01:51 +00:00
David Robertson	4aeb00ca20	Move synctl into `synapse._scripts` and expose as an entrypoint (#12140 )	2022-03-04 11:58:49 +00:00
Erik Johnston	423cca9efe	Spread out sending device lists to remote hosts (#12132 )	2022-03-04 11:48:15 +00:00
David Robertson	cea1b58c4a	Don't impose version checks on dev extras at runtime (#12129 ) * Fix incorrect argument in test case * Add copyright header * Docstring and __all__ * Exclude dev depenencies * Use changelog from #12088 * Include version in error messages This will hopefully distinguish between the version of the source code and the version of the distribution package that is installed. * Linter script is your friend	2022-03-03 12:47:55 +00:00
Brendan Abolivier	ae8a616b49	Correctly register deactivation and profile update module callbacks (#12141 )	2022-03-03 11:39:58 +01:00
Erik Johnston	6d282a9c89	Make release script write correct no-op changelog (#12127 ) As we want to include the previous version in the "No new changes..." string.	2022-03-02 14:28:18 +00:00
				`@@ -0,0 +1 @@`
				`Remove workaround introduced in Synapse 1.50.0 for Mjolnir compatibility. Breaks compatibility with Mjolnir 1.3.1 and earlier.`
				`@@ -0,0 +1 @@`
				Add third-party rules rules callbacks `check_can_shutdown_room` and `check_can_deactivate_user`.
				`@@ -0,0 +1 @@`
				`Fix a long-standing bug when redacting events with relations.`
				`@@ -0,0 +1 @@`
				`Fix CI not attaching source distributions and wheels to the GitHub releases.`
				`@@ -0,0 +1 @@`
				`Improve performance of logging in for large accounts.`
				`@@ -0,0 +1 @@`
				Add experimental env var `SYNAPSE_ASYNC_IO_REACTOR` that causes Synapse to use the asyncio reactor for Twisted.
				`@@ -0,0 +1 @@`
				Remove backwards compatibilty with pagination tokens from the `/relations` and `/aggregations` endpoints generated from Synapse < v1.52.0.
				`@@ -0,0 +1 @@`
				Move `synctl` into `synapse._scripts` and expose as an entry point.
				`@@ -0,0 +1 @@`
				Add test for `ObservableDeferred`'s cancellation behaviour.