Merge branch 'shay/super_docs' of https://github.com/matrix-org/synapse into shay/super_docs

fix newsfragment
Merge branch 'develop' into shay/super_docs
2022-05-09 13:51:08 -07:00 · 2022-05-09 13:50:54 -07:00 · 2022-05-09 13:46:12 -07:00 · 2022-05-09 13:39:57 -07:00 · 2022-05-09 13:31:42 -07:00 · 2022-05-09 20:29:07 +00:00
213 changed files with 3442 additions and 1178 deletions
--- a/.github/workflows/docker.yml
+++ b/.github/workflows/docker.yml
@@ -34,32 +34,24 @@ jobs:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

-      # TODO: consider using https://github.com/docker/metadata-action instead of this
-      # custom magic
      - name: Calculate docker image tag
        id: set-tag
-        run: |
-          case "${GITHUB_REF}" in
-              refs/heads/develop)
-                  tag=develop
-                  ;;
-              refs/heads/master|refs/heads/main)
-                  tag=latest
-                  ;;
-              refs/tags/*)
-                  tag=${GITHUB_REF#refs/tags/}
-                  ;;
-              *)
-                  tag=${GITHUB_SHA}
-                  ;;
-          esac
-          echo "::set-output name=tag::$tag"
+        uses: docker/metadata-action@master
+        with:
+          images: matrixdotorg/synapse
+          flavor: |
+            latest=false
+          tags: |
+            type=raw,value=develop,enable=${{ github.ref == 'refs/heads/develop' }}
+            type=raw,value=latest,enable=${{ github.ref == 'refs/heads/master' }}
+            type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' }}
+            type=pep440,pattern={{raw}}

      - name: Build and push all platforms
        uses: docker/build-push-action@v2
        with:
          push: true
          labels: "gitsha1=${{ github.sha }}"
-          tags: "matrixdotorg/synapse:${{ steps.set-tag.outputs.tag }}"
+          tags: "${{ steps.set-tag.outputs.tags }}"
          file: "docker/Dockerfile"
          platforms: linux/amd64,linux/arm64
--- a/.github/workflows/latest_deps.yml
+++ b/.github/workflows/latest_deps.yml
@@ -32,12 +32,15 @@ jobs:
        with:
          python-version: "3.x"
          poetry-version: "1.2.0b1"
+          extras: "all"
      # Dump installed versions for debugging.
      - run: poetry run pip list > before.txt
      # Upgrade all runtime dependencies only. This is intended to mimic a fresh
      # `pip install matrix-synapse[all]` as closely as possible.
      - run: poetry update --no-dev
      - run: poetry run pip list > after.txt && (diff -u before.txt after.txt || true)
+      - name: Remove warn_unused_ignores from mypy config
+        run: sed '/warn_unused_ignores = True/d' -i mypy.ini
      - run: poetry run mypy
  trial:
    runs-on: ubuntu-latest
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -20,13 +20,9 @@ jobs:
      - run: scripts-dev/config-lint.sh

  lint:
-    # This does a vanilla `poetry install` - no extras. I'm slightly anxious
-    # that we might skip some typechecks on code that uses extras. However,
-    # I think the right way to fix this is to mark any extras needed for
-    # typechecking as development dependencies. To detect this, we ought to
-    # turn up mypy's strictness: disallow unknown imports and be accept fewer
-    # uses of `Any`.
    uses: "matrix-org/backend-meta/.github/workflows/python-poetry-ci.yml@v1"
+    with:
+      typechecking-extras: "all"

  lint-crlf:
    runs-on: ubuntu-latest
--- a/.github/workflows/twisted_trunk.yml
+++ b/.github/workflows/twisted_trunk.yml
@@ -24,6 +24,8 @@ jobs:
          poetry remove twisted
          poetry add --extras tls git+https://github.com/twisted/twisted.git#trunk
          poetry install --no-interaction --extras "all test"
+      - name: Remove warn_unused_ignores from mypy config
+        run: sed '/warn_unused_ignores = True/d' -i mypy.ini
      - run: poetry run mypy

  trial:
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,3 +1,29 @@
+Synapse 1.59.0
+==============
+
+The non-standard `m.login.jwt` login type has been removed from Synapse. It can be replaced with `org.matrix.login.jwt` for identical behaviour. This is only used if `jwt_config.enabled` is set to `true` in the configuration.
+
+
+Synapse 1.58.1 (2022-05-05)
+===========================
+
+This patch release includes a fix to the Debian packages, installing the
+`systemd` and `cache_memory` extra package groups, which were incorrectly
+omitted in v1.58.0. This primarily prevented Synapse from starting
+when the `systemd.journal.JournalHandler` log handler was configured.
+See [#12631](https://github.com/matrix-org/synapse/issues/12631) for further information.
+
+Otherwise, no significant changes since 1.58.0.
+
+
+Synapse 1.58.0 (2022-05-03)
+===========================
+
+As of this release, the groups/communities feature in Synapse is now disabled by default. See [\#11584](https://github.com/matrix-org/synapse/issues/11584) for details. As mentioned in [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1580), this feature will be removed in Synapse 1.61.
+
+No significant changes since 1.58.0rc2.
+
+
 Synapse 1.58.0rc2 (2022-04-26)
 ==============================

@@ -19,8 +45,6 @@ Internal Changes
 Synapse 1.58.0rc1 (2022-04-26)
 ==============================

-As of this release, the groups/communities feature in Synapse is now disabled by default. See [\#11584](https://github.com/matrix-org/synapse/issues/11584) for details. As mentioned in [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1580), this feature will be removed in Synapse 1.61.
-
 Features
 --------

--- a/README.rst
+++ b/README.rst
@@ -55,7 +55,7 @@ solutions. The hope is for Matrix to act as the building blocks for a new
 generation of fully open and interoperable messaging and VoIP apps for the
 internet.

-Synapse is a Matrix "homeserver" implementation developed by the matrix.org core 
+Synapse is a Matrix "homeserver" implementation developed by the matrix.org core
 team, written in Python 3/Twisted.

 In Matrix, every user runs one or more Matrix clients, which connect through to
@@ -294,13 +294,13 @@ directory of your choice::
    cd synapse

 Synapse has a number of external dependencies. We maintain a fixed development
-environment using [poetry](https://python-poetry.org/). First, install poetry. We recommend
+environment using `Poetry <https://python-poetry.org/>`_. First, install poetry. We recommend::

    pip install --user pipx
    pipx install poetry

 as described `here <https://python-poetry.org/docs/#installing-with-pipx>`_.
-(See `poetry's installation docs <https://python-poetry.org/docs/#installation>`
+(See `poetry's installation docs <https://python-poetry.org/docs/#installation>`_
 for other installation methods.) Then ask poetry to create a virtual environment
 from the project and install Synapse's dependencies::

@@ -309,11 +309,11 @@ from the project and install Synapse's dependencies::
 This will run a process of downloading and installing all the needed
 dependencies into a virtual env.

-We recommend using the demo which starts 3 federated instances running on ports `8080` - `8082`
+We recommend using the demo which starts 3 federated instances running on ports `8080` - `8082`::

    poetry run ./demo/start.sh

-(to stop, you can use `poetry run ./demo/stop.sh`)
+(to stop, you can use ``poetry run ./demo/stop.sh``)

 See the `demo documentation <https://matrix-org.github.io/synapse/develop/development/demo.html>`_
 for more information.
--- a/changelog.d/11507.feature
+++ b/changelog.d/11507.feature
@@ -0,0 +1 @@
+Support [MSC3266](https://github.com/matrix-org/matrix-doc/pull/3266) room summaries over federation.
--- a/changelog.d/12168.feature
+++ b/changelog.d/12168.feature
@@ -0,0 +1 @@
+Implement [changes](https://github.com/matrix-org/matrix-spec-proposals/pull/2285/commits/4a77139249c2e830aec3c7d6bd5501a514d1cc27) to [MSC2285 (hidden read receipts)](https://github.com/matrix-org/matrix-spec-proposals/pull/2285). Contributed by @SimonBrandner.
--- a/changelog.d/12273.bugfix
+++ b/changelog.d/12273.bugfix
@@ -0,0 +1 @@
+Fix a bug introduced in Synapse v1.48.0 where latest thread reply provided failed to include the proper bundled aggregations.
--- a/changelog.d/12356.misc
+++ b/changelog.d/12356.misc
@@ -0,0 +1 @@
+Fix scripts-dev to pass typechecking.
--- a/changelog.d/12406.feature
+++ b/changelog.d/12406.feature
@@ -0,0 +1 @@
+Add a module API to allow modules to change actions for existing push rules of local users.
--- a/changelog.d/12452.feature
+++ b/changelog.d/12452.feature
@@ -0,0 +1 @@
+Add the `notify_appservices_from_worker` configuration option (superseding `notify_appservices`) to allow a generic worker to be designated as the worker to send traffic to Application Services.
--- a/changelog.d/12480.misc
+++ b/changelog.d/12480.misc
@@ -0,0 +1 @@
+Use supervisord to supervise Postgres and Caddy in the Complement image to reduce restart time.
--- a/changelog.d/12505.misc
+++ b/changelog.d/12505.misc
@@ -0,0 +1 @@
+Use `make_awaitable` instead of `defer.succeed` for return values of mocks in tests.
--- a/changelog.d/12526.feature
+++ b/changelog.d/12526.feature
@@ -0,0 +1 @@
+Add new `enable_registration_token_3pid_bypass` configuration option to allow registrations via token as an alternative to verifying a 3pid.
--- a/changelog.d/12531.misc
+++ b/changelog.d/12531.misc
@@ -0,0 +1 @@
+Remove unused `# type: ignore`s.
--- a/changelog.d/12556.misc
+++ b/changelog.d/12556.misc
@@ -0,0 +1 @@
+Release script: confirm the commit to be tagged before tagging.
--- a/changelog.d/12564.misc
+++ b/changelog.d/12564.misc
@@ -0,0 +1 @@
+Consistently check if an object is a `frozendict`.
--- a/changelog.d/12568.misc
+++ b/changelog.d/12568.misc
@@ -0,0 +1 @@
+Protect module callbacks with read semantics against cancellation.
--- a/changelog.d/12570.bugfix
+++ b/changelog.d/12570.bugfix
@@ -0,0 +1 @@
+Fix a bug introduced in Synapse 1.57 which could cause `Failed to calculate hosts in room` errors to be logged for outbound federation.
--- a/changelog.d/12573.docker
+++ b/changelog.d/12573.docker
@@ -0,0 +1 @@
+Update the "Build docker images" GitHub Actions workflow to use `docker/metadata-action` to generate docker image tags, instead of a custom shell script. Contributed by henryclw.
--- a/changelog.d/12576.misc
+++ b/changelog.d/12576.misc
@@ -0,0 +1 @@
+Allow unused `#type: ignore` comments in bleeding edge CI jobs.
--- a/changelog.d/12577.misc
+++ b/changelog.d/12577.misc
@@ -0,0 +1 @@
+Improve comments and error messages around access tokens.
--- a/changelog.d/12579.doc
+++ b/changelog.d/12579.doc
@@ -0,0 +1 @@
+Add missing linebreak to pipx install instructions.
--- a/changelog.d/12580.bugfix
+++ b/changelog.d/12580.bugfix
@@ -0,0 +1 @@
+Fix a long standing bug where status codes would almost always get logged as 200!, irrespective of the actual status code, when clients disconnect before a request has finished processing.
--- a/changelog.d/12581.misc
+++ b/changelog.d/12581.misc
@@ -0,0 +1 @@
+Improve docstrings for the receipts store.
--- a/changelog.d/12582.misc
+++ b/changelog.d/12582.misc
@@ -0,0 +1 @@
+Use constants for read-receipts in tests.
--- a/changelog.d/12587.misc
+++ b/changelog.d/12587.misc
@@ -0,0 +1 @@
+Log status code of cancelled requests as 499 and avoid logging stack traces for them.
--- a/changelog.d/12589.misc
+++ b/changelog.d/12589.misc
@@ -0,0 +1 @@
+Remove special-case for `twisted` logger from default log config.
--- a/changelog.d/12594.bugfix
+++ b/changelog.d/12594.bugfix
@@ -0,0 +1 @@
+Fix race when persisting an event and deleting a room that could lead to outbound federation breaking.
--- a/changelog.d/12596.removal
+++ b/changelog.d/12596.removal
@@ -0,0 +1 @@
+Remove unstable identifiers from [MSC3069](https://github.com/matrix-org/matrix-doc/pull/3069).
--- a/changelog.d/12597.removal
+++ b/changelog.d/12597.removal
@@ -0,0 +1,2 @@
+Remove the unspecified `m.login.jwt` login type and the unstable `uk.half-shot.msc2778.login.application_service` from 
+[MSC2778](https://github.com/matrix-org/matrix-doc/pull/2778).
--- a/changelog.d/12599.misc
+++ b/changelog.d/12599.misc
@@ -0,0 +1 @@
+Use `getClientAddress` instead of the deprecated `getClientIP`.
--- a/changelog.d/12602.misc
+++ b/changelog.d/12602.misc
@@ -0,0 +1 @@
+Add link to documentation in Grafana Dashboard.
--- a/changelog.d/12608.misc
+++ b/changelog.d/12608.misc
@@ -0,0 +1 @@
+Remove redundant lines of config from `mypy.ini`.
--- a/changelog.d/12610.misc
+++ b/changelog.d/12610.misc
@@ -0,0 +1 @@
+Reduce log spam when running multiple event persisters.
--- a/changelog.d/12612.bugfix
+++ b/changelog.d/12612.bugfix
@@ -0,0 +1 @@
+Fix a typo in the announcement text generated by the Synapse release development script.
--- a/changelog.d/12613.removal
+++ b/changelog.d/12613.removal
@@ -0,0 +1 @@
+Synapse now requires at least Python 3.7.1 (up from 3.7.0), for compatibility with the latest Twisted trunk.
--- a/changelog.d/12614.misc
+++ b/changelog.d/12614.misc
@@ -0,0 +1 @@
+Add extra debug logging to federation sender.
--- a/changelog.d/12616.misc
+++ b/changelog.d/12616.misc
@@ -0,0 +1 @@
+Prevent remote homeservers from requesting local user device names by default.
--- a/changelog.d/12619.feature
+++ b/changelog.d/12619.feature
@@ -0,0 +1 @@
+Add new `mau_appservice_trial_days` configuration option to specify a different trial period for users registered via an appservice.
--- a/changelog.d/12620.misc
+++ b/changelog.d/12620.misc
@@ -0,0 +1 @@
+Add a consistency check on events which we read from the database.
--- a/changelog.d/12624.misc
+++ b/changelog.d/12624.misc
@@ -0,0 +1 @@
+Remove use of constantly library and switch to enums for EventRedactBehaviour. Contributed by @andrewdoh.
--- a/changelog.d/12627.doc
+++ b/changelog.d/12627.doc
@@ -0,0 +1 @@
+Fixes to the formatting of README.rst.
--- a/changelog.d/12632.misc
+++ b/changelog.d/12632.misc
@@ -0,0 +1 @@
+Remove unused code related to receipts.
--- a/changelog.d/12633.bugfix
+++ b/changelog.d/12633.bugfix
@@ -0,0 +1 @@
+Fix a bug introduced in Synapse v1.53.0 where bundled aggregations for annotations/edits were incorrectly calculated.
--- a/changelog.d/12635.feature
+++ b/changelog.d/12635.feature
@@ -0,0 +1 @@
+Implement [changes](https://github.com/matrix-org/matrix-spec-proposals/pull/2285/commits/4a77139249c2e830aec3c7d6bd5501a514d1cc27) to [MSC2285 (hidden read receipts)](https://github.com/matrix-org/matrix-spec-proposals/pull/2285). Contributed by @SimonBrandner.
--- a/changelog.d/12636.feature
+++ b/changelog.d/12636.feature
@@ -0,0 +1 @@
+Implement [changes](https://github.com/matrix-org/matrix-spec-proposals/pull/2285/commits/4a77139249c2e830aec3c7d6bd5501a514d1cc27) to [MSC2285 (hidden read receipts)](https://github.com/matrix-org/matrix-spec-proposals/pull/2285). Contributed by @SimonBrandner.
--- a/changelog.d/12639.bugfix
+++ b/changelog.d/12639.bugfix
@@ -0,0 +1 @@
+Add new `enable_registration_token_3pid_bypass` configuration option to allow registrations via token as an alternative to verifying a 3pid.
--- a/changelog.d/12650.misc
+++ b/changelog.d/12650.misc
@@ -0,0 +1 @@
+Update to mypy 0.950.
--- a/changelog.d/12652.misc
+++ b/changelog.d/12652.misc
@@ -0,0 +1 @@
+Move `pympler` back in to the `all` extras.
--- a/changelog.d/12656.misc
+++ b/changelog.d/12656.misc
@@ -0,0 +1 @@
+Prevent memory leak from reoccurring when presence is disabled.
--- a/changelog.d/12657.bugfix
+++ b/changelog.d/12657.bugfix
@@ -0,0 +1 @@
+Fix a long-standing bug where rooms containing power levels with string values could not be upgraded.
--- a/changelog.d/12663.misc
+++ b/changelog.d/12663.misc
@@ -0,0 +1 @@
+Log status code of cancelled requests as 499 and avoid logging stack traces for them.
--- a/changelog.d/12664.doc
+++ b/changelog.d/12664.doc
@@ -0,0 +1 @@
+Fix docs on how to run specific Complement tests using the `complement.sh` test runner.
--- a/changelog.d/12665.misc
+++ b/changelog.d/12665.misc
@@ -0,0 +1 @@
+Fix spelling of `M_UNRECOGNIZED` in comments.
--- a/changelog.d/12666.misc
+++ b/changelog.d/12666.misc
@@ -0,0 +1 @@
+Use `Concatenate` to better annotate `_do_execute`.
--- a/changelog.d/12667.misc
+++ b/changelog.d/12667.misc
@@ -0,0 +1 @@
+Use `ParamSpec` to refine type hints.
--- a/changelog.d/12670.feature
+++ b/changelog.d/12670.feature
@@ -0,0 +1 @@
+Implement [changes](https://github.com/matrix-org/matrix-spec-proposals/pull/2285/commits/4a77139249c2e830aec3c7d6bd5501a514d1cc27) to [MSC2285 (hidden read receipts)](https://github.com/matrix-org/matrix-spec-proposals/pull/2285). Contributed by @SimonBrandner.
--- a/changelog.d/12671.misc
+++ b/changelog.d/12671.misc
@@ -0,0 +1 @@
+Fix mypy against latest pillow stubs.
--- a/changelog.d/12681.doc
+++ b/changelog.d/12681.doc
@@ -0,0 +1 @@
+Add topics to synapse documentation.
--- a/contrib/grafana/synapse.json
+++ b/contrib/grafana/synapse.json
@@ -66,6 +66,18 @@
      ],
      "title": "Dashboards",
      "type": "dashboards"
+    },
+    {
+      "asDropdown": false,
+      "icon": "external link",
+      "includeVars": false,
+      "keepTime": false,
+      "tags": [],
+      "targetBlank": true,
+      "title": "Synapse Documentation",
+      "tooltip": "Open Documentation",
+      "type": "link",
+      "url": "https://matrix-org.github.io/synapse/latest/"
    }
  ],
  "panels": [
@@ -10889,4 +10901,4 @@
  "title": "Synapse",
  "uid": "000000012",
  "version": 100
-}
+}
--- a/debian/build_virtualenv
+++ b/debian/build_virtualenv
@@ -37,7 +37,11 @@ python3 -m venv "$TEMP_VENV"
 source "$TEMP_VENV/bin/activate"
 pip install -U pip
 pip install poetry==1.2.0b1
-poetry export --extras all --extras test -o exported_requirements.txt
+poetry export \
+    --extras all \
+    --extras test \
+    --extras systemd \
+    -o exported_requirements.txt
 deactivate
 rm -rf "$TEMP_VENV"

--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,25 @@
+matrix-synapse-py3 (1.58.2) UNRELEASED; urgency=medium
+
+  * Adjust how the `exported-requirements.txt` file is generated as part of
+    the process of building these packages. This affects the package
+    maintainers only; end-users are unaffected.
+
+ -- Synapse Packaging team <packages@matrix.org>  Fri, 06 May 2022 13:49:29 +0100
+
+matrix-synapse-py3 (1.58.1) stable; urgency=medium
+
+  * Include python dependencies from the `systemd` and `cache_memory` extras package groups, which
+    were incorrectly omitted from the 1.58.0 package.
+  * New Synapse release 1.58.1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Thu, 05 May 2022 14:58:23 +0100
+
+matrix-synapse-py3 (1.58.0) stable; urgency=medium
+
+  * New Synapse release 1.58.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 03 May 2022 10:52:58 +0100
+
 matrix-synapse-py3 (1.58.0~rc2) stable; urgency=medium

  * New Synapse release 1.58.0rc2.
--- a/docker/Dockerfile-workers
+++ b/docker/Dockerfile-workers
@@ -20,6 +20,9 @@ RUN rm /etc/nginx/sites-enabled/default
 # Copy Synapse worker, nginx and supervisord configuration template files
 COPY ./docker/conf-workers/* /conf/

+# Copy a script to prefix log lines with the supervisor program name
+COPY ./docker/prefix-log /usr/local/bin/
+
 # Expose nginx listener port
 EXPOSE 8080/tcp

--- a/docker/complement/SynapseWorkers.Dockerfile
+++ b/docker/complement/SynapseWorkers.Dockerfile
@@ -34,13 +34,16 @@ WORKDIR /data
 # Copy the caddy config
 COPY conf-workers/caddy.complement.json /root/caddy.json

+COPY conf-workers/postgres.supervisord.conf /etc/supervisor/conf.d/postgres.conf
+COPY conf-workers/caddy.supervisord.conf /etc/supervisor/conf.d/caddy.conf
+
 # Copy the entrypoint
 COPY conf-workers/start-complement-synapse-workers.sh /

 # Expose caddy's listener ports
 EXPOSE 8008 8448

-ENTRYPOINT /start-complement-synapse-workers.sh
+ENTRYPOINT ["/start-complement-synapse-workers.sh"]

 # Update the healthcheck to have a shorter check interval
 HEALTHCHECK --start-period=5s --interval=1s --timeout=1s \
--- a/docker/complement/conf-workers/caddy.supervisord.conf
+++ b/docker/complement/conf-workers/caddy.supervisord.conf
@@ -0,0 +1,7 @@
+[program:caddy]
+command=/usr/local/bin/prefix-log /root/caddy run --config /root/caddy.json
+autorestart=unexpected
+stdout_logfile=/dev/stdout
+stdout_logfile_maxbytes=0
+stderr_logfile=/dev/stderr
+stderr_logfile_maxbytes=0
--- a/docker/complement/conf-workers/postgres.supervisord.conf
+++ b/docker/complement/conf-workers/postgres.supervisord.conf
@@ -0,0 +1,16 @@
+[program:postgres]
+command=/usr/local/bin/prefix-log /usr/bin/pg_ctlcluster 13 main start --foreground
+
+# Lower priority number = starts first
+priority=1
+
+autorestart=unexpected
+stdout_logfile=/dev/stdout
+stdout_logfile_maxbytes=0
+stderr_logfile=/dev/stderr
+stderr_logfile_maxbytes=0
+
+# Use 'Fast Shutdown' mode which aborts current transactions and closes connections quickly.
+# (Default (TERM) is 'Smart Shutdown' which stops accepting new connections but
+#  lets existing connections close gracefully.)
+stopsignal=INT
--- a/docker/complement/conf-workers/start-complement-synapse-workers.sh
+++ b/docker/complement/conf-workers/start-complement-synapse-workers.sh
@@ -12,12 +12,6 @@ function log {
 # Replace the server name in the caddy config
 sed -i "s/{{ server_name }}/${SERVER_NAME}/g" /root/caddy.json

-log "starting postgres"
-pg_ctlcluster 13 main start
-
-log "starting caddy"
-/root/caddy start --config /root/caddy.json
-
 # Set the server name of the homeserver
 export SYNAPSE_SERVER_NAME=${SERVER_NAME}

--- a/docker/conf/log.config
+++ b/docker/conf/log.config
@@ -2,11 +2,7 @@ version: 1

 formatters:
  precise:
-{% if worker_name %}
-    format: '%(asctime)s - worker:{{ worker_name }} - %(name)s - %(lineno)d - %(levelname)s - %(request)s - %(message)s'
-{% else %}
    format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s - %(message)s'
-{% endif %}

 handlers:
 {% if LOG_FILE_PATH %}
--- a/docker/configure_workers_and_start.py
+++ b/docker/configure_workers_and_start.py
@@ -69,10 +69,10 @@ WORKERS_CONFIG: Dict[str, Dict[str, Any]] = {
        "worker_extra_conf": "enable_media_repo: true",
    },
    "appservice": {
-        "app": "synapse.app.appservice",
+        "app": "synapse.app.generic_worker",
        "listener_resources": [],
        "endpoint_patterns": [],
-        "shared_extra_conf": {"notify_appservices": False},
+        "shared_extra_conf": {"notify_appservices_from_worker": "appservice"},
        "worker_extra_conf": "",
    },
    "federation_sender": {
@@ -171,7 +171,7 @@ WORKERS_CONFIG: Dict[str, Dict[str, Any]] = {
 # Templates for sections that may be inserted multiple times in config files
 SUPERVISORD_PROCESS_CONFIG_BLOCK = """
 [program:synapse_{name}]
-command=/usr/local/bin/python -m {app} \
+command=/usr/local/bin/prefix-log /usr/local/bin/python -m {app} \
    --config-path="{config_path}" \
    --config-path=/conf/workers/shared.yaml \
    --config-path=/conf/workers/{name}.yaml
--- a/docker/prefix-log
+++ b/docker/prefix-log
@@ -0,0 +1,12 @@
+#!/bin/bash
+#
+# Prefixes all lines on stdout and stderr with the process name (as determined by
+# the SUPERVISOR_PROCESS_NAME env var, which is automatically set by Supervisor).
+#
+# Usage:
+#   prefix-log command [args...]
+#
+
+exec 1> >(awk '{print "'"${SUPERVISOR_PROCESS_NAME}"' | "$0}' >&1)
+exec 2> >(awk '{print "'"${SUPERVISOR_PROCESS_NAME}"' | "$0}' >&2)
+exec "$@"
--- a/docs/SUMMARY.md
+++ b/docs/SUMMARY.md
@@ -88,19 +88,158 @@
  - [OpenTracing](opentracing.md)
  - [Database Schemas](development/database_schema.md)
  - [Experimental features](development/experimental_features.md)
-  - [Synapse Architecture]()
-    - [Log Contexts](log_contexts.md)
-    - [Replication](replication.md)
-    - [TCP Replication](tcp_replication.md)
-  - [Internal Documentation](development/internal_documentation/README.md)
+  - [Synapse Architecture](development/synapse_architechture.md)
+    - [Overview]()
+      - Flows of data through Synapse
+        - Starts with a Request/BG Job/Module
+        - Processing
+        - Maybe Response
+      - Config
+      - Request Handling
+        - Clients
+        - Federation
+        - Internal Processing - there could be an exception!
+      - Storage
+      - Federation
+      - External Entities: Remote homeservers, Appservices, Pushers, Identity Servers
+      - Modules
+      - Workers/Replication/Redis/Organization of a large homeserver instance
+  - [Configuration]()
+    - Experimental config options
+      - Define how long they should live after the corresponding msc has merged
+  - [Logging]()
+    - combine with [Log Contexts](log_contexts.md)
+  - [Storage]()
+    - What is current database schema?
+      - Start by linking script that can compile full schema
+      - Is there a tool that can visualize a postgres db? - Shay to check
+    - [Database Backends]()
+    - [Database Migrations]()
+      - Background [Schema] Updates
+    - [Caching]()
+      - Cache tuning
+      - How to check sizes
+        - Nice to have a metric for max cache sizes
+      - How to add a cache to a [storage] function
+        - Why would you
+      - How to invalidate (with support for workers)
+  - [Request Listening]()
+    - Mention that we use Twisted
+    - [Servlets and Resources]()
+      - When to use one over the other (always use a servlet?)
+      - Unstable/release-based endpoints
+      - Don't put too many servlets on the same resource
+    - [Handlers]()
+      - Machinery behind the servlet that does all the processing
+  - [Background Jobs]()
+  - [Federation]()
+    - Federation Catchup
+  - [Data Types]()
+    - [Users]()
+      - Allowed username characters
+    - [Devices]()
+    - [Events]()
+      - What is an event?
+      - Different event formats depending on room version
+      - Outlier events, rejected events, dropped events
+    - [Rooms]()
+      - [Room DAG concepts](development/room-dag-concepts.md)
+  - [Sync]()
+  - [State Resolution]()
+    - [The Auth Chain Difference Algorithm](auth_chain_difference_algorithm.md)
+  - [User Authentication]()
    - [Single Sign-On]()
      - [SAML](development/saml.md)
      - [CAS](development/cas.md)
-    - [Room DAG concepts](development/room-dag-concepts.md)
-    - [State Resolution]()
-      - [The Auth Chain Difference Algorithm](auth_chain_difference_algorithm.md)
-    - [Media Repository](media_repository.md)
-    - [Room and User Statistics](room_and_user_statistics.md)
+    - [User-Interactive Auth]()
+    - [Password-based]()
+    - [Password Auth Modules]()
+    - [Token-based]()
+  - [Rate Limiting]()
+    - Different rate limiting classes (Ratelimiter vs RequestRatelimiter)
+    - How does our rate limiting work
+    - When to use/add rate limiting
+    - Rate limiting for federation traffic
+  - [Media Repository](media_repository.md)
+  - [Email and HTML Templating]()
+  - [Presence]()
+    - What is Presence?
+    - How is it designed to work currently
+      - Why it is recommended to be disabled
+      - Why is this resource intensive
+        - Links and issues on how to fix this
+      - How it's implemented
+  - [Application Services]()
+  - [Push Notifications]()
+  - [Synapse Admin API]()
+  - [Synapse Modules]()
+  - [Workers]()
+    - Things to be mindful of in order to make your feature work with workers
+    - How to test a feature on workers
+  - [Replication](replication.md)
+    - [TCP Replication](tcp_replication.md)
+  - [Sources]()
+    - TypingSource, ReceiptSource, PresenceSource, etc
+    - Classes containing methods for storing and getting different types of data
+  - [Streams]()
+    - Stream Types (the ID of the stream)
+    - Stream Token: each entity added to the stream gets a unique, incremental token
+    - StreamReplication classes and clients
+    - How this interacts with sync?
+    - Stream token generators: Single Writer Only (sqlite), MultiWriterStreamIDGenerator (Postgres)
+  - [Notifier]()
+  - [Monitoring and Metrics]() # opentracing, phone home and metrics
+    - [Opentracing]()
+      - How does this work?
+      - How is this helpful?
+      - What's jaeger?
+      - How do I create and mutate a span?
+      - How are spans sent between homeservers?
+        - Only between whitelisted homeservers
+      - What areas are currently traced
+        - Database layer
+        - Servlets
+        - Crypto stuff
+    - [Metrics]()
+      - Prometheus
+      - How to make a new metrics / good practices
+        - What are all the current metrics and what each one means
+        - How to view in grafana
+          - Here's a quick example of making a new panel to visualize metrics
+        - Btw it would be nice to have a tooltip with an explainer for every graph in graphana
+      - [Room and User Statistics](room_and_user_statistics.md)  # TODO: This page is currently useless    
+  - [Testing]()
+    - [Unit Tests]()
+      - How to run the tests (partially in the contributing guide)
+        - IDE-specific examples
+        - How to run one test / one test case / run in parallel
+        - Documenting test environment variables: SYNAPSE_TEST_LOG_LEVEL, SYNAPSE_TEST_PERSISTENT_SQLITE_DB, SYNAPSE_POSTGRES etc.
+      - Test case versus test method
+        - prepare, make_homeserver, default_config, @override_config decorator, @parameterized decorator, pump/reactor.advance
+        - Test utilities: fake homeserver, federation, workers, channels, requests, HomeServerTestCase.helper
+          - How to use them in your tests at a high level
+          - How they are implemented
+      - Different inherited TestCase classes: HomeServerTestCase, FederatedHomeServerTestCase(sp?), TestCase
+      - Test pattern: How to start a request, check the state of the homeserver, respond to the request
+      - Ensuring the right servlets for your test are registered
+      - Good practice: use storage methods vs. tampering with the database directly (database schema changes)
+      - Dealing with deferreds/async (get_success/get_failure).
+      - Mocking - async mocks
+      - Organization of tests
+      - Skipping tests
+    - [Integration Tests]()
+        - [Complement]()
+            - Point to Complement's documentation on its structure and how to write a test
+            - How to run Complement against Synapse
+            - Document Synapse's Complement images and how they're built
+            - Which Complement build flags does Synapse support
+                - Unstable features
+                - Synapse's Complement Blacklist
+        - [Sytest]()
+            - You should prefer Complement :)
+            - Helper functions
+            - Dealing with race conditions
+            - Link to that one perl helper doc Matthew wrote
  - [Scripts]()

 # Other
--- a/docs/development/contributing_guide.md
+++ b/docs/development/contributing_guide.md
@@ -270,13 +270,13 @@ COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh
 To run a specific test file, you can pass the test name at the end of the command. The name passed comes from the naming structure in your Complement tests. If you're unsure of the name, you can do a full run and copy it from the test output:

 ```sh
-COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestBackfillingHistory
+COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -run TestImportHistoricalMessages
 ```

 To run a specific test, you can specify the whole name structure:

 ```sh
-COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestBackfillingHistory/parallel/Backfilled_historical_events_resolve_with_proper_state_in_correct_order
+COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -run TestImportHistoricalMessages/parallel/Historical_events_resolve_in_the_correct_order
 ```


--- a/docs/jwt.md
+++ b/docs/jwt.md
@@ -17,9 +17,6 @@ follows:
 }
 ```

-Note that the login type of `m.login.jwt` is supported, but is deprecated. This
-will be removed in a future version of Synapse.
-
 The `token` field should include the JSON web token with the following claims:

 * A claim that encodes the local part of the user ID is required. By default,
--- a/docs/sample_config.yaml
+++ b/docs/sample_config.yaml
@@ -407,6 +407,11 @@ manhole_settings:
 # sign up in a short space of time never to return after their initial
 # session.
 #
+# The option `mau_appservice_trial_days` is similar to `mau_trial_days`, but
+# applies a different trial number if the user was registered by an appservice.
+# A value of 0 means no trial days are applied. Appservices not listed in this
+# dictionary use the value of `mau_trial_days` instead.
+#
 # 'mau_limit_alerting' is a means of limiting client side alerting
 # should the mau limit be reached. This is useful for small instances
 # where the admin has 5 mau seats (say) for 5 specific people and no
@@ -417,6 +422,8 @@ manhole_settings:
 #max_mau_value: 50
 #mau_trial_days: 2
 #mau_limit_alerting: false
+#mau_appservice_trial_days:
+#  "appservice-id": 1

 # If enabled, the metrics for the number of monthly active users will
 # be populated, however no one will be limited. If limit_usage_by_mau
@@ -709,11 +716,11 @@ retention:
 #
 #allow_profile_lookup_over_federation: false

-# Uncomment to disable device display name lookup over federation. By default, the
-# Federation API allows other homeservers to obtain device display names of any user
-# on this homeserver. Defaults to 'true'.
+# Uncomment to allow device display name lookup over federation. By default, the
+# Federation API prevents other homeservers from obtaining the display names of
+# user devices on this homeserver. Defaults to 'false'.
 #
-#allow_device_name_lookup_over_federation: false
+#allow_device_name_lookup_over_federation: true


 ## Caching ##
@@ -1323,6 +1330,12 @@ oembed:
 #
 #registration_requires_token: true

+# Allow users to submit a token during registration to bypass any required 3pid
+# steps configured in `registrations_require_3pid`.
+# Defaults to false, requiring that registration tokens (if enabled) complete a 3pid flow.
+#
+#enable_registration_token_3pid_bypass: false
+
 # If set, allows registration of standard or admin accounts by anyone who
 # has the shared secret, even if registration is otherwise disabled.
 #
--- a/docs/sample_log_config.yaml
+++ b/docs/sample_log_config.yaml
@@ -62,13 +62,6 @@ loggers:
        # information such as access tokens.
        level: INFO

-    twisted:
-        # We send the twisted logging directly to the file handler,
-        # to work around https://github.com/matrix-org/synapse/issues/3471
-        # when using "buffer" logger. Use "console" to log to stderr instead.
-        handlers: [file]
-        propagate: false
-
 root:
    level: INFO

--- a/docs/upgrade.md
+++ b/docs/upgrade.md
@@ -89,6 +89,43 @@ process, for example:
    dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
    ```

+# Upgrading to v1.59.0
+
+## Device name lookup over federation has been disabled by default
+
+The names of user devices are no longer visible to users on other homeservers by default.
+Device IDs are unaffected, as these are necessary to facilitate end-to-end encryption.
+
+To re-enable this functionality, set the
+[`allow_device_name_lookup_over_federation`](https://matrix-org.github.io/synapse/v1.59/usage/configuration/config_documentation.html#federation)
+homeserver config option to `true`.
+
+
+## Deprecation of the `synapse.app.appservice` worker application type
+
+The `synapse.app.appservice` worker application type allowed you to configure a
+single worker to use to notify application services of new events, as long
+as this functionality was disabled on the main process with `notify_appservices: False`.
+
+To unify Synapse's worker types, the `synapse.app.appservice` worker application
+type and the `notify_appservices` configuration option have been deprecated.
+
+To get the same functionality, it's now recommended that the `synapse.app.generic_worker`
+worker application type is used and that the `notify_appservices_from_worker` option
+is set to the name of a worker.
+
+For the time being, `notify_appservices_from_worker` can be used alongside
+`synapse.app.appservice` and `notify_appservices` to make it easier to transition
+between the two configurations, however please note that:
+
+- the options must not contradict each other (otherwise Synapse won't start); and
+- the `notify_appservices` option will be removed in a future release of Synapse.
+
+Please see [the relevant section of the worker documentation][v1_59_notify_ases_from] for more information.
+
+[v1_59_notify_ases_from]: workers.md#notifying-application-services
+
+
 # Upgrading to v1.58.0

 ## Groups/communities feature has been disabled by default
@@ -96,6 +133,7 @@ process, for example:
 The non-standard groups/communities feature in Synapse has been disabled by default
 and will be removed in Synapse v1.61.0.

+
 # Upgrading to v1.57.0

 ## Changes to database schema for application services
--- a/docs/usage/administration/request_log.md
+++ b/docs/usage/administration/request_log.md
@@ -28,7 +28,7 @@ See the following for how to decode the dense data available from the default lo
 | NNNN  | Total time waiting for response to DB queries across all parallel DB work from this request |
 | OOOO  | Count of DB transactions performed |
 | PPPP  | Response body size |
-| QQQQ  | Response status code (prefixed with ! if the socket was closed before the response was generated) |
+| QQQQ  | Response status code<br/>Suffixed with `!` if the socket was closed before the response was generated.<br/>A `499!` status code indicates that Synapse also cancelled request processing after the socket was closed.<br/> |
 | RRRR  | Request |
 | SSSS  | User-agent |
 | TTTT  | Events fetched from DB to service this request (note that this does not include events fetched from the cache) |
--- a/docs/usage/configuration/config_documentation.md
+++ b/docs/usage/configuration/config_documentation.md
@@ -627,6 +627,20 @@ Example configuration:
 mau_trial_days: 5
 ```
 ---
+Config option: `mau_appservice_trial_days`
+
+The option `mau_appservice_trial_days` is similar to `mau_trial_days`, but applies a different
+trial number if the user was registered by an appservice. A value
+of 0 means no trial days are applied. Appservices not listed in this dictionary
+use the value of `mau_trial_days` instead.
+
+Example configuration:
+```yaml
+mau_appservice_trial_days: 
+  my_appservice_id: 3
+  another_appservice_id: 6
+```
+---
 Config option: `mau_limit_alerting`

 The option `mau_limit_alerting` is a means of limiting client-side alerting
@@ -1035,13 +1049,13 @@ allow_profile_lookup_over_federation: false
 ---
 Config option: `allow_device_name_lookup_over_federation`

-Set this option to false to disable device display name lookup over federation. By default, the
-Federation API allows other homeservers to obtain device display names of any user
+Set this option to true to allow device display name lookup over federation. By default, the
+Federation API prevents other homeservers from obtaining the display names of any user devices
 on this homeserver.

 Example configuration:
 ```yaml
-allow_device_name_lookup_over_federation: false
+allow_device_name_lookup_over_federation: true
 ```
 ---
 ## Caching ##
--- a/docs/workers.md
+++ b/docs/workers.md
@@ -435,6 +435,23 @@ An example for a dedicated background worker instance:
 {{#include systemd-with-workers/workers/background_worker.yaml}}
 ```

+#### Notifying Application Services
+
+You can designate one worker to send output traffic to Application Services.
+
+Specify its name in the shared configuration as follows:
+
+```yaml
+notify_appservices_from_worker: worker_name
+```
+
+This work cannot be load-balanced; please ensure the main process is restarted
+after setting this option in the shared configuration!
+
+This style of configuration supersedes the legacy `synapse.app.appservice`
+worker application type.
+
+
 ### `synapse.app.pusher`

 Handles sending push notifications to sygnal and email. Doesn't handle any
@@ -453,6 +470,9 @@ pusher_instances:

 ### `synapse.app.appservice`

+**Deprecated as of Synapse v1.58.** [Use `synapse.app.generic_worker` with the
+`notify_appservices_from_worker` option instead.](#notifying-application-services)
+
 Handles sending output traffic to Application Services. Doesn't handle any
 REST endpoints itself, but you should set `notify_appservices: False` in the
 shared configuration file to stop the main synapse sending appservice notifications.
--- a/mypy.ini
+++ b/mypy.ini
@@ -7,6 +7,7 @@ show_error_codes = True
 show_traceback = True
 mypy_path = stubs
 warn_unreachable = True
+warn_unused_ignores = True
 local_partial_types = True
 no_implicit_optional = True

@@ -23,10 +24,6 @@ files =
 # https://docs.python.org/3/library/re.html#re.X
 exclude = (?x)
  ^(
-   |scripts-dev/build_debian_packages.py
-   |scripts-dev/federation_client.py
-   |scripts-dev/release.py
-
   |synapse/storage/databases/__init__.py
   |synapse/storage/databases/main/cache.py
   |synapse/storage/databases/main/devices.py
@@ -134,6 +131,11 @@ disallow_untyped_defs = True
 [mypy-synapse.metrics.*]
 disallow_untyped_defs = True

+[mypy-synapse.metrics._reactor_metrics]
+# This module imports select.epoll. That exists on Linux, but doesn't on macOS.
+# See https://github.com/matrix-org/synapse/pull/11771.
+warn_unused_ignores = False
+
 [mypy-synapse.module_api.*]
 disallow_untyped_defs = True

@@ -239,63 +241,26 @@ disallow_untyped_defs = True
 [mypy-authlib.*]
 ignore_missing_imports = True

-[mypy-bcrypt]
-ignore_missing_imports = True
-
 [mypy-canonicaljson]
 ignore_missing_imports = True

-[mypy-constantly]
-ignore_missing_imports = True
-
-[mypy-daemonize]
-ignore_missing_imports = True
-
-[mypy-h11]
-ignore_missing_imports = True
-
-[mypy-hiredis]
-ignore_missing_imports = True
-
-[mypy-hyperlink]
-ignore_missing_imports = True
-
 [mypy-ijson.*]
 ignore_missing_imports = True

-[mypy-importlib_metadata.*]
-ignore_missing_imports = True
-
-[mypy-jaeger_client.*]
-ignore_missing_imports = True
-
-[mypy-josepy.*]
-ignore_missing_imports = True
-
-[mypy-jwt.*]
-ignore_missing_imports = True
-
 [mypy-lxml]
 ignore_missing_imports = True

 [mypy-msgpack]
 ignore_missing_imports = True

-[mypy-nacl.*]
-ignore_missing_imports = True
-
+# Note: WIP stubs available at
+# https://github.com/microsoft/python-type-stubs/tree/64934207f523ad6b611e6cfe039d85d7175d7d0d/netaddr
 [mypy-netaddr]
 ignore_missing_imports = True

 [mypy-parameterized.*]
 ignore_missing_imports = True

-[mypy-phonenumbers.*]
-ignore_missing_imports = True
-
-[mypy-prometheus_client.*]
-ignore_missing_imports = True
-
 [mypy-pymacaroons.*]
 ignore_missing_imports = True

@@ -308,23 +273,14 @@ ignore_missing_imports = True
 [mypy-saml2.*]
 ignore_missing_imports = True

-[mypy-sentry_sdk]
-ignore_missing_imports = True
-
 [mypy-service_identity.*]
 ignore_missing_imports = True

-[mypy-signedjson.*]
+[mypy-srvlookup.*]
 ignore_missing_imports = True

 [mypy-treq.*]
 ignore_missing_imports = True

-[mypy-twisted.*]
-ignore_missing_imports = True
-
-[mypy-zope]
-ignore_missing_imports = True
-
 [mypy-incremental.*]
 ignore_missing_imports = True
--- a/poetry.lock
+++ b/poetry.lock
@@ -309,14 +309,15 @@ smmap = ">=3.0.1,<6"

 [[package]]
 name = "gitpython"
-version = "3.1.14"
-description = "Python Git Library"
+version = "3.1.27"
+description = "GitPython is a python library used to interact with Git repositories"
 category = "dev"
 optional = false
-python-versions = ">=3.4"
+python-versions = ">=3.7"

 [package.dependencies]
 gitdb = ">=4.0.1,<5"
+typing-extensions = {version = ">=3.7.4.3", markers = "python_version < \"3.8\""}

 [[package]]
 name = "hiredis"
@@ -571,7 +572,7 @@ python-versions = "*"

 [[package]]
 name = "mypy"
-version = "0.931"
+version = "0.950"
 description = "Optional static typing for Python"
 category = "dev"
 optional = false
@@ -579,13 +580,14 @@ python-versions = ">=3.6"

 [package.dependencies]
 mypy-extensions = ">=0.4.3"
-tomli = ">=1.1.0"
+tomli = {version = ">=1.1.0", markers = "python_version < \"3.11\""}
 typed-ast = {version = ">=1.4.0,<2", markers = "python_version < \"3.8\""}
 typing-extensions = ">=3.10"

 [package.extras]
 dmypy = ["psutil (>=4.0)"]
 python2 = ["typed-ast (>=1.4.0,<2)"]
+reports = ["lxml"]

 [[package]]
 name = "mypy-extensions"
@@ -597,14 +599,14 @@ python-versions = "*"

 [[package]]
 name = "mypy-zope"
-version = "0.3.5"
+version = "0.3.7"
 description = "Plugin for mypy to support zope interfaces"
 category = "dev"
 optional = false
 python-versions = "*"

 [package.dependencies]
-mypy = "0.931"
+mypy = "0.950"
 "zope.interface" = "*"
 "zope.schema" = "*"

@@ -1019,7 +1021,7 @@ jeepney = ">=0.6"

 [[package]]
 name = "sentry-sdk"
-version = "1.5.7"
+version = "1.5.11"
 description = "Python client for Sentry (https://sentry.io)"
 category = "main"
 optional = true
@@ -1315,6 +1317,14 @@ category = "dev"
 optional = false
 python-versions = "*"

+[[package]]
+name = "types-commonmark"
+version = "0.9.2"
+description = "Typing stubs for commonmark"
+category = "dev"
+optional = false
+python-versions = "*"
+
 [[package]]
 name = "types-cryptography"
 version = "3.3.15"
@@ -1361,7 +1371,7 @@ python-versions = "*"

 [[package]]
 name = "types-pillow"
-version = "9.0.6"
+version = "9.0.15"
 description = "Typing stubs for Pillow"
 category = "dev"
 optional = false
@@ -1536,7 +1546,7 @@ docs = ["sphinx", "repoze.sphinx.autointerface"]
 test = ["zope.i18nmessageid", "zope.testing", "zope.testrunner"]

 [extras]
-all = ["matrix-synapse-ldap3", "psycopg2", "psycopg2cffi", "psycopg2cffi-compat", "pysaml2", "authlib", "lxml", "sentry-sdk", "jaeger-client", "opentracing", "pyjwt", "txredisapi", "hiredis"]
+all = ["matrix-synapse-ldap3", "psycopg2", "psycopg2cffi", "psycopg2cffi-compat", "pysaml2", "authlib", "lxml", "sentry-sdk", "jaeger-client", "opentracing", "pyjwt", "txredisapi", "hiredis", "Pympler"]
 cache_memory = ["Pympler"]
 jwt = ["pyjwt"]
 matrix-synapse-ldap3 = ["matrix-synapse-ldap3"]
@@ -1552,8 +1562,8 @@ url_preview = ["lxml"]

 [metadata]
 lock-version = "1.1"
-python-versions = "^3.7"
-content-hash = "f482a4f594a165dfe01ce253a22510d5faf38647ab0dcebc35789350cafd9bf0"
+python-versions = "^3.7.1"
+content-hash = "d39d5ac5d51c014581186b7691999b861058b569084c525523baf70b77f292b1"

 [metadata.files]
 attrs = [
@@ -1766,8 +1776,8 @@ gitdb = [
    {file = "gitdb-4.0.9.tar.gz", hash = "sha256:bac2fd45c0a1c9cf619e63a90d62bdc63892ef92387424b855792a6cabe789aa"},
 ]
 gitpython = [
-    {file = "GitPython-3.1.14-py3-none-any.whl", hash = "sha256:3283ae2fba31c913d857e12e5ba5f9a7772bbc064ae2bb09efafa71b0dd4939b"},
-    {file = "GitPython-3.1.14.tar.gz", hash = "sha256:be27633e7509e58391f10207cd32b2a6cf5b908f92d9cd30da2e514e1137af61"},
+    {file = "GitPython-3.1.27-py3-none-any.whl", hash = "sha256:5b68b000463593e05ff2b261acff0ff0972df8ab1b70d3cdbd41b546c8b8fc3d"},
+    {file = "GitPython-3.1.27.tar.gz", hash = "sha256:1c885ce809e8ba2d88a29befeb385fcea06338d3640712b59ca623c220bb5704"},
 ]
 hiredis = [
    {file = "hiredis-2.0.0-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:b4c8b0bc5841e578d5fb32a16e0c305359b987b850a06964bd5a62739d688048"},
@@ -2080,34 +2090,37 @@ msgpack = [
    {file = "msgpack-1.0.3.tar.gz", hash = "sha256:51fdc7fb93615286428ee7758cecc2f374d5ff363bdd884c7ea622a7a327a81e"},
 ]
 mypy = [
-    {file = "mypy-0.931-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:3c5b42d0815e15518b1f0990cff7a705805961613e701db60387e6fb663fe78a"},
-    {file = "mypy-0.931-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:c89702cac5b302f0c5d33b172d2b55b5df2bede3344a2fbed99ff96bddb2cf00"},
-    {file = "mypy-0.931-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:300717a07ad09525401a508ef5d105e6b56646f7942eb92715a1c8d610149714"},
-    {file = "mypy-0.931-cp310-cp310-win_amd64.whl", hash = "sha256:7b3f6f557ba4afc7f2ce6d3215d5db279bcf120b3cfd0add20a5d4f4abdae5bc"},
-    {file = "mypy-0.931-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:1bf752559797c897cdd2c65f7b60c2b6969ffe458417b8d947b8340cc9cec08d"},
-    {file = "mypy-0.931-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:4365c60266b95a3f216a3047f1d8e3f895da6c7402e9e1ddfab96393122cc58d"},
-    {file = "mypy-0.931-cp36-cp36m-win_amd64.whl", hash = "sha256:1b65714dc296a7991000b6ee59a35b3f550e0073411ac9d3202f6516621ba66c"},
-    {file = "mypy-0.931-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:e839191b8da5b4e5d805f940537efcaa13ea5dd98418f06dc585d2891d228cf0"},
-    {file = "mypy-0.931-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:50c7346a46dc76a4ed88f3277d4959de8a2bd0a0fa47fa87a4cde36fe247ac05"},
-    {file = "mypy-0.931-cp37-cp37m-win_amd64.whl", hash = "sha256:d8f1ff62f7a879c9fe5917b3f9eb93a79b78aad47b533911b853a757223f72e7"},
-    {file = "mypy-0.931-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:f9fe20d0872b26c4bba1c1be02c5340de1019530302cf2dcc85c7f9fc3252ae0"},
-    {file = "mypy-0.931-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:1b06268df7eb53a8feea99cbfff77a6e2b205e70bf31743e786678ef87ee8069"},
-    {file = "mypy-0.931-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:8c11003aaeaf7cc2d0f1bc101c1cc9454ec4cc9cb825aef3cafff8a5fdf4c799"},
-    {file = "mypy-0.931-cp38-cp38-win_amd64.whl", hash = "sha256:d9d2b84b2007cea426e327d2483238f040c49405a6bf4074f605f0156c91a47a"},
-    {file = "mypy-0.931-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:ff3bf387c14c805ab1388185dd22d6b210824e164d4bb324b195ff34e322d166"},
-    {file = "mypy-0.931-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:5b56154f8c09427bae082b32275a21f500b24d93c88d69a5e82f3978018a0266"},
-    {file = "mypy-0.931-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:8ca7f8c4b1584d63c9a0f827c37ba7a47226c19a23a753d52e5b5eddb201afcd"},
-    {file = "mypy-0.931-cp39-cp39-win_amd64.whl", hash = "sha256:74f7eccbfd436abe9c352ad9fb65872cc0f1f0a868e9d9c44db0893440f0c697"},
-    {file = "mypy-0.931-py3-none-any.whl", hash = "sha256:1171f2e0859cfff2d366da2c7092b06130f232c636a3f7301e3feb8b41f6377d"},
-    {file = "mypy-0.931.tar.gz", hash = "sha256:0038b21890867793581e4cb0d810829f5fd4441aa75796b53033af3aa30430ce"},
+    {file = "mypy-0.950-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:cf9c261958a769a3bd38c3e133801ebcd284ffb734ea12d01457cb09eacf7d7b"},
+    {file = "mypy-0.950-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:b5b5bd0ffb11b4aba2bb6d31b8643902c48f990cc92fda4e21afac658044f0c0"},
+    {file = "mypy-0.950-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:5e7647df0f8fc947388e6251d728189cfadb3b1e558407f93254e35abc026e22"},
+    {file = "mypy-0.950-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:eaff8156016487c1af5ffa5304c3e3fd183edcb412f3e9c72db349faf3f6e0eb"},
+    {file = "mypy-0.950-cp310-cp310-win_amd64.whl", hash = "sha256:563514c7dc504698fb66bb1cf897657a173a496406f1866afae73ab5b3cdb334"},
+    {file = "mypy-0.950-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:dd4d670eee9610bf61c25c940e9ade2d0ed05eb44227275cce88701fee014b1f"},
+    {file = "mypy-0.950-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:ca75ecf2783395ca3016a5e455cb322ba26b6d33b4b413fcdedfc632e67941dc"},
+    {file = "mypy-0.950-cp36-cp36m-win_amd64.whl", hash = "sha256:6003de687c13196e8a1243a5e4bcce617d79b88f83ee6625437e335d89dfebe2"},
+    {file = "mypy-0.950-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:4c653e4846f287051599ed8f4b3c044b80e540e88feec76b11044ddc5612ffed"},
+    {file = "mypy-0.950-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:e19736af56947addedce4674c0971e5dceef1b5ec7d667fe86bcd2b07f8f9075"},
+    {file = "mypy-0.950-cp37-cp37m-win_amd64.whl", hash = "sha256:ef7beb2a3582eb7a9f37beaf38a28acfd801988cde688760aea9e6cc4832b10b"},
+    {file = "mypy-0.950-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:0112752a6ff07230f9ec2f71b0d3d4e088a910fdce454fdb6553e83ed0eced7d"},
+    {file = "mypy-0.950-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:ee0a36edd332ed2c5208565ae6e3a7afc0eabb53f5327e281f2ef03a6bc7687a"},
+    {file = "mypy-0.950-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:77423570c04aca807508a492037abbd72b12a1fb25a385847d191cd50b2c9605"},
+    {file = "mypy-0.950-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:5ce6a09042b6da16d773d2110e44f169683d8cc8687e79ec6d1181a72cb028d2"},
+    {file = "mypy-0.950-cp38-cp38-win_amd64.whl", hash = "sha256:5b231afd6a6e951381b9ef09a1223b1feabe13625388db48a8690f8daa9b71ff"},
+    {file = "mypy-0.950-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:0384d9f3af49837baa92f559d3fa673e6d2652a16550a9ee07fc08c736f5e6f8"},
+    {file = "mypy-0.950-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:1fdeb0a0f64f2a874a4c1f5271f06e40e1e9779bf55f9567f149466fc7a55038"},
+    {file = "mypy-0.950-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:61504b9a5ae166ba5ecfed9e93357fd51aa693d3d434b582a925338a2ff57fd2"},
+    {file = "mypy-0.950-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:a952b8bc0ae278fc6316e6384f67bb9a396eb30aced6ad034d3a76120ebcc519"},
+    {file = "mypy-0.950-cp39-cp39-win_amd64.whl", hash = "sha256:eaea21d150fb26d7b4856766e7addcf929119dd19fc832b22e71d942835201ef"},
+    {file = "mypy-0.950-py3-none-any.whl", hash = "sha256:a4d9898f46446bfb6405383b57b96737dcfd0a7f25b748e78ef3e8c576bba3cb"},
+    {file = "mypy-0.950.tar.gz", hash = "sha256:1b333cfbca1762ff15808a0ef4f71b5d3eed8528b23ea1c3fb50543c867d68de"},
 ]
 mypy-extensions = [
    {file = "mypy_extensions-0.4.3-py2.py3-none-any.whl", hash = "sha256:090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d"},
    {file = "mypy_extensions-0.4.3.tar.gz", hash = "sha256:2d82818f5bb3e369420cb3c4060a7970edba416647068eb4c5343488a6c604a8"},
 ]
 mypy-zope = [
-    {file = "mypy-zope-0.3.5.tar.gz", hash = "sha256:489e7da1c2af887f2cfe3496995fc247f296512b495b57817edddda9d22308f3"},
-    {file = "mypy_zope-0.3.5-py3-none-any.whl", hash = "sha256:3bd0cc9a3e5933b02931af4b214ba32a4f4ff98adb30c979ce733857db91a18b"},
+    {file = "mypy-zope-0.3.7.tar.gz", hash = "sha256:9da171e78e8ef7ac8922c86af1a62f1b7f3244f121020bd94a2246bc3f33c605"},
+    {file = "mypy_zope-0.3.7-py3-none-any.whl", hash = "sha256:9c7637d066e4d1bafa0651abc091c752009769098043b236446e6725be2bc9c2"},
 ]
 netaddr = [
    {file = "netaddr-0.8.0-py2.py3-none-any.whl", hash = "sha256:9666d0232c32d2656e5e5f8d735f58fd6c7457ce52fc21c98d45f2af78f990ac"},
@@ -2377,8 +2390,8 @@ secretstorage = [
    {file = "SecretStorage-3.3.1.tar.gz", hash = "sha256:fd666c51a6bf200643495a04abb261f83229dcb6fd8472ec393df7ffc8b6f195"},
 ]
 sentry-sdk = [
-    {file = "sentry-sdk-1.5.7.tar.gz", hash = "sha256:aa52da941c56b5a76fd838f8e9e92a850bf893a9eb1e33ffce6c21431d07ee30"},
-    {file = "sentry_sdk-1.5.7-py2.py3-none-any.whl", hash = "sha256:411a8495bd18cf13038e5749e4710beb4efa53da6351f67b4c2f307c2d9b6d49"},
+    {file = "sentry-sdk-1.5.11.tar.gz", hash = "sha256:6c01d9d0b65935fd275adc120194737d1df317dce811e642cbf0394d0d37a007"},
+    {file = "sentry_sdk-1.5.11-py2.py3-none-any.whl", hash = "sha256:c17179183cac614e900cbd048dab03f49a48e2820182ec686c25e7ce46f8548f"},
 ]
 service-identity = [
    {file = "service-identity-21.1.0.tar.gz", hash = "sha256:6e6c6086ca271dc11b033d17c3a8bea9f24ebff920c587da090afc9519419d34"},
@@ -2588,6 +2601,10 @@ types-bleach = [
    {file = "types-bleach-4.1.4.tar.gz", hash = "sha256:2d30c2c4fb6854088ac636471352c9a51bf6c089289800d2a8060820a01cd43a"},
    {file = "types_bleach-4.1.4-py3-none-any.whl", hash = "sha256:edffe173ed6d7b6f3543036a96204a9319c3bf6c3645917b14274e43f000cc9b"},
 ]
+types-commonmark = [
+    {file = "types-commonmark-0.9.2.tar.gz", hash = "sha256:b894b67750c52fd5abc9a40a9ceb9da4652a391d75c1b480bba9cef90f19fc86"},
+    {file = "types_commonmark-0.9.2-py3-none-any.whl", hash = "sha256:56f20199a1f9a2924443211a0ef97f8b15a8a956a7f4e9186be6950bf38d6d02"},
+]
 types-cryptography = [
    {file = "types-cryptography-3.3.15.tar.gz", hash = "sha256:a7983a75a7b88a18f88832008f0ef140b8d1097888ec1a0824ec8fb7e105273b"},
    {file = "types_cryptography-3.3.15-py3-none-any.whl", hash = "sha256:d9b0dd5465d7898d400850e7f35e5518aa93a7e23d3e11757cd81b4777089046"},
@@ -2609,8 +2626,8 @@ types-opentracing = [
    {file = "types_opentracing-2.4.7-py3-none-any.whl", hash = "sha256:861fb8103b07cf717f501dd400cb274ca9992552314d4d6c7a824b11a215e512"},
 ]
 types-pillow = [
-    {file = "types-Pillow-9.0.6.tar.gz", hash = "sha256:79b350b1188c080c27558429f1e119e69c9f020b877a82df761d9283070e0185"},
-    {file = "types_Pillow-9.0.6-py3-none-any.whl", hash = "sha256:bd1e0a844fc718398aa265bf50fcad550fc520cc54f80e5ffeb7b3226b3cc507"},
+    {file = "types-Pillow-9.0.15.tar.gz", hash = "sha256:d2e385fe5c192e75970f18accce69f5c2a9f186f3feb578a9b91cd6fdf64211d"},
+    {file = "types_Pillow-9.0.15-py3-none-any.whl", hash = "sha256:c9646595dfafdf8b63d4b1443292ead17ee0fc7b18a143e497b68e0ea2dc1eb6"},
 ]
 types-psycopg2 = [
    {file = "types-psycopg2-2.9.9.tar.gz", hash = "sha256:4f9d4d52eeb343dc00fd5ed4f1513a8a5c18efba0a072eb82706d15cf4f20a2e"},
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -54,7 +54,7 @@ skip_gitignore = true

 [tool.poetry]
 name = "matrix-synapse"
-version = "1.58.0rc2"
+version = "1.58.1"
 description = "Homeserver for the Matrix decentralised comms protocol"
 authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
 license = "Apache-2.0"
@@ -100,7 +100,7 @@ synapse_review_recent_signups = "synapse._scripts.review_recent_signups:main"
 update_synapse_database = "synapse._scripts.update_synapse_database:main"

 [tool.poetry.dependencies]
-python = "^3.7"
+python = "^3.7.1"

 # Mandatory Dependencies
 # ----------------------
@@ -142,8 +142,10 @@ netaddr = ">=0.7.18"
 # add a lower bound to the Jinja2 dependency.
 Jinja2 = ">=3.0"
 bleach = ">=1.4.3"
-# We use `ParamSpec`, which was added in `typing-extensions` 3.10.0.0.
-typing-extensions = ">=3.10.0"
+# We use `ParamSpec` and `Concatenate`, which were added in `typing-extensions` 3.10.0.0.
+# Additionally we need https://github.com/python/typing/pull/817 to allow types to be
+# generic over ParamSpecs.
+typing-extensions = ">=3.10.0.1"
 # We enforce that we have a `cryptography` version that bundles an `openssl`
 # with the latest security patches.
 cryptography = ">=3.4.7"
@@ -231,10 +233,11 @@ all = [
    "jaeger-client", "opentracing",
    # jwt
    "pyjwt",
-    #redis
-    "txredisapi", "hiredis"
+    # redis
+    "txredisapi", "hiredis",
+    # cache_memory
+    "pympler",
    # omitted:
-    #   - cache_memory: this is an experimental option
    #   - test: it's useful to have this separate from dev deps in the olddeps job
    #   - systemd: this is a system-based requirement
 ]
@@ -248,9 +251,10 @@ flake8-bugbear = "==21.3.2"
 flake8 = "*"

 # Typechecking
-mypy = "==0.931"
-mypy-zope = "==0.3.5"
+mypy = "*"
+mypy-zope = "*"
 types-bleach = ">=4.1.0"
+types-commonmark = ">=0.9.2"
 types-jsonschema = ">=3.2.0"
 types-opentracing = ">=2.4.2"
 types-Pillow = ">=8.3.4"
@@ -270,7 +274,8 @@ idna = ">=2.5"

 # The following are used by the release script
 click = "==8.1.0"
-GitPython = "==3.1.14"
+# GitPython was == 3.1.14; bumped to 3.1.20, the first release with type hints.
+GitPython = ">=3.1.20"
 commonmark = "==0.9.1"
 pygithub = "==1.55"
 # The following are executed as commands by the release script.
--- a/scripts-dev/build_debian_packages.py
+++ b/scripts-dev/build_debian_packages.py
@@ -17,7 +17,8 @@ import subprocess
 import sys
 import threading
 from concurrent.futures import ThreadPoolExecutor
-from typing import Optional, Sequence
+from types import FrameType
+from typing import Collection, Optional, Sequence, Set

 DISTS = (
    "debian:buster",  # oldstable: EOL 2022-08
@@ -41,15 +42,17 @@ projdir = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))

 class Builder(object):
    def __init__(
-        self, redirect_stdout=False, docker_build_args: Optional[Sequence[str]] = None
+        self,
+        redirect_stdout: bool = False,
+        docker_build_args: Optional[Sequence[str]] = None,
    ):
        self.redirect_stdout = redirect_stdout
        self._docker_build_args = tuple(docker_build_args or ())
-        self.active_containers = set()
+        self.active_containers: Set[str] = set()
        self._lock = threading.Lock()
        self._failed = False

-    def run_build(self, dist, skip_tests=False):
+    def run_build(self, dist: str, skip_tests: bool = False) -> None:
        """Build deb for a single distribution"""

        if self._failed:
@@ -63,7 +66,7 @@ class Builder(object):
            self._failed = True
            raise

-    def _inner_build(self, dist, skip_tests=False):
+    def _inner_build(self, dist: str, skip_tests: bool = False) -> None:
        tag = dist.split(":", 1)[1]

        # Make the dir where the debs will live.
@@ -138,7 +141,7 @@ class Builder(object):
            stdout.close()
            print("Completed build of %s" % (dist,))

-    def kill_containers(self):
+    def kill_containers(self) -> None:
        with self._lock:
            active = list(self.active_containers)

@@ -156,8 +159,10 @@ class Builder(object):
                self.active_containers.remove(c)


-def run_builds(builder, dists, jobs=1, skip_tests=False):
-    def sig(signum, _frame):
+def run_builds(
+    builder: Builder, dists: Collection[str], jobs: int = 1, skip_tests: bool = False
+) -> None:
+    def sig(signum: int, _frame: Optional[FrameType]) -> None:
        print("Caught SIGINT")
        builder.kill_containers()

--- a/scripts-dev/federation_client.py
+++ b/scripts-dev/federation_client.py
@@ -38,7 +38,7 @@ import argparse
 import base64
 import json
 import sys
-from typing import Any, Optional
+from typing import Any, Dict, Optional, Tuple
 from urllib import parse as urlparse

 import requests
@@ -47,13 +47,14 @@ import signedjson.types
 import srvlookup
 import yaml
 from requests.adapters import HTTPAdapter
+from urllib3 import HTTPConnectionPool

 # uncomment the following to enable debug logging of http requests
 # from httplib import HTTPConnection
 # HTTPConnection.debuglevel = 1


-def encode_base64(input_bytes):
+def encode_base64(input_bytes: bytes) -> str:
    """Encode bytes as a base64 string without any padding."""

    input_len = len(input_bytes)
@@ -63,7 +64,7 @@ def encode_base64(input_bytes):
    return output_string


-def encode_canonical_json(value):
+def encode_canonical_json(value: object) -> bytes:
    return json.dumps(
        value,
        # Encode code-points outside of ASCII as UTF-8 rather than \u escapes
@@ -130,7 +131,7 @@ def request(
            sig,
            destination,
        )
-        authorization_headers.append(header.encode("ascii"))
+        authorization_headers.append(header)
        print("Authorization: %s" % header, file=sys.stderr)

    dest = "matrix://%s%s" % (destination, path)
@@ -139,7 +140,10 @@ def request(
    s = requests.Session()
    s.mount("matrix://", MatrixConnectionAdapter())

-    headers = {"Host": destination, "Authorization": authorization_headers[0]}
+    headers: Dict[str, str] = {
+        "Host": destination,
+        "Authorization": authorization_headers[0],
+    }

    if method == "POST":
        headers["Content-Type"] = "application/json"
@@ -154,7 +158,7 @@ def request(
    )


-def main():
+def main() -> None:
    parser = argparse.ArgumentParser(
        description="Signs and sends a federation request to a matrix homeserver"
    )
@@ -212,6 +216,7 @@ def main():
    if not args.server_name or not args.signing_key:
        read_args_from_config(args)

+    assert isinstance(args.signing_key, str)
    algorithm, version, key_base64 = args.signing_key.split()
    key = signedjson.key.decode_signing_key_base64(algorithm, version, key_base64)

@@ -233,7 +238,7 @@ def main():
    print("")


-def read_args_from_config(args):
+def read_args_from_config(args: argparse.Namespace) -> None:
    with open(args.config, "r") as fh:
        config = yaml.safe_load(fh)

@@ -250,7 +255,7 @@ def read_args_from_config(args):

 class MatrixConnectionAdapter(HTTPAdapter):
    @staticmethod
-    def lookup(s, skip_well_known=False):
+    def lookup(s: str, skip_well_known: bool = False) -> Tuple[str, int]:
        if s[-1] == "]":
            # ipv6 literal (with no port)
            return s, 8448
@@ -276,7 +281,7 @@ class MatrixConnectionAdapter(HTTPAdapter):
            return s, 8448

    @staticmethod
-    def get_well_known(server_name):
+    def get_well_known(server_name: str) -> Optional[str]:
        uri = "https://%s/.well-known/matrix/server" % (server_name,)
        print("fetching %s" % (uri,), file=sys.stderr)

@@ -299,7 +304,9 @@ class MatrixConnectionAdapter(HTTPAdapter):
            print("Invalid response from %s: %s" % (uri, e), file=sys.stderr)
        return None

-    def get_connection(self, url, proxies=None):
+    def get_connection(
+        self, url: str, proxies: Optional[Dict[str, str]] = None
+    ) -> HTTPConnectionPool:
        parsed = urlparse.urlparse(url)

        (host, port) = self.lookup(parsed.netloc)
--- a/scripts-dev/mypy_synapse_plugin.py
+++ b/scripts-dev/mypy_synapse_plugin.py
@@ -16,7 +16,7 @@
 can crop up, e.g the cache descriptors.
 """

-from typing import Callable, Optional
+from typing import Callable, Optional, Type

 from mypy.nodes import ARG_NAMED_OPT
 from mypy.plugin import MethodSigContext, Plugin
@@ -94,7 +94,7 @@ def cached_function_method_signature(ctx: MethodSigContext) -> CallableType:
    return signature


-def plugin(version: str):
+def plugin(version: str) -> Type[SynapsePlugin]:
    # This is the entry point of the plugin, and let's us deal with the fact
    # that the mypy plugin interface is *not* stable by looking at the version
    # string.
--- a/scripts-dev/release.py
+++ b/scripts-dev/release.py
@@ -25,7 +25,7 @@ import sys
 import urllib.request
 from os import path
 from tempfile import TemporaryDirectory
-from typing import List, Optional
+from typing import Any, List, Optional, cast

 import attr
 import click
@@ -36,7 +36,9 @@ from github import Github
 from packaging import version


-def run_until_successful(command, *args, **kwargs):
+def run_until_successful(
+    command: str, *args: Any, **kwargs: Any
+) -> subprocess.CompletedProcess:
    while True:
        completed_process = subprocess.run(command, *args, **kwargs)
        exit_code = completed_process.returncode
@@ -50,7 +52,7 @@ def run_until_successful(command, *args, **kwargs):


@click.group()
-def cli():
+def cli() -> None:
    """An interactive script to walk through the parts of creating a release.

    Requires the dev dependencies be installed, which can be done via:
@@ -81,19 +83,13 @@ def cli():


@cli.command()
-def prepare():
+def prepare() -> None:
    """Do the initial stages of creating a release, including creating release
    branch, updating changelog and pushing to GitHub.
    """

    # Make sure we're in a git repo.
-    try:
-        repo = git.Repo()
-    except git.InvalidGitRepositoryError:
-        raise click.ClickException("Not in Synapse repo.")
-
-    if repo.is_dirty():
-        raise click.ClickException("Uncommitted changes exist.")
+    repo = get_repo_and_check_clean_checkout()

    click.secho("Updating git repo...")
    repo.remote().fetch()
@@ -161,22 +157,21 @@ def prepare():
        click.get_current_context().abort()

    # Switch to the release branch.
-    parsed_new_version: version.Version = version.parse(new_version)
+    # Cast safety: parse() won't return a version.LegacyVersion from our
+    # version string format.
+    parsed_new_version = cast(version.Version, version.parse(new_version))

    # We assume for debian changelogs that we only do RCs or full releases.
    assert not parsed_new_version.is_devrelease
    assert not parsed_new_version.is_postrelease

-    release_branch_name = (
-        f"release-v{parsed_new_version.major}.{parsed_new_version.minor}"
-    )
+    release_branch_name = get_release_branch_name(parsed_new_version)
    release_branch = find_ref(repo, release_branch_name)
    if release_branch:
        if release_branch.is_remote():
            # If the release branch only exists on the remote we check it out
            # locally.
            repo.git.checkout(release_branch_name)
-            release_branch = repo.active_branch
    else:
        # If a branch doesn't exist we create one. We ask which one branch it
        # should be based off, defaulting to sensible values depending on the
@@ -198,13 +193,15 @@ def prepare():
            click.get_current_context().abort()

        # Check out the base branch and ensure it's up to date
-        repo.head.reference = base_branch
+        repo.head.set_reference(base_branch, "check out the base branch")
        repo.head.reset(index=True, working_tree=True)
        if not base_branch.is_remote():
            update_branch(repo)

        # Create the new release branch
-        release_branch = repo.create_head(release_branch_name, commit=base_branch)
+        # Type ignore will no longer be needed after GitPython 3.1.28.
+        # See https://github.com/gitpython-developers/GitPython/pull/1419
+        repo.create_head(release_branch_name, commit=base_branch)  # type: ignore[arg-type]

    # Switch to the release branch and ensure it's up to date.
    repo.git.checkout(release_branch_name)
@@ -265,17 +262,11 @@ def prepare():

@cli.command()
@click.option("--gh-token", envvar=["GH_TOKEN", "GITHUB_TOKEN"])
-def tag(gh_token: Optional[str]):
+def tag(gh_token: Optional[str]) -> None:
    """Tags the release and generates a draft GitHub release"""

    # Make sure we're in a git repo.
-    try:
-        repo = git.Repo()
-    except git.InvalidGitRepositoryError:
-        raise click.ClickException("Not in Synapse repo.")
-
-    if repo.is_dirty():
-        raise click.ClickException("Uncommitted changes exist.")
+    repo = get_repo_and_check_clean_checkout()

    click.secho("Updating git repo...")
    repo.remote().fetch()
@@ -288,12 +279,26 @@ def tag(gh_token: Optional[str]):
    if tag_name in repo.tags:
        raise click.ClickException(f"Tag {tag_name} already exists!\n")

+    # Check we're on the right release branch
+    release_branch = get_release_branch_name(current_version)
+    if repo.active_branch.name != release_branch:
+        click.echo(
+            f"Need to be on the release branch ({release_branch}) before tagging. "
+            f"Currently on ({repo.active_branch.name})."
+        )
+        click.get_current_context().abort()
+
    # Get the appropriate changelogs and tag.
    changes = get_changes_for_version(current_version)

    click.echo_via_pager(changes)
    if click.confirm("Edit text?", default=False):
-        changes = click.edit(changes, require_save=False)
+        edited_changes = click.edit(changes, require_save=False)
+        # This assert is for mypy's benefit. click's docs are a little unclear, but
+        # when `require_save=False`, not saving the temp file in the editor returns
+        # the original string.
+        assert edited_changes is not None
+        changes = edited_changes

    repo.create_tag(tag_name, message=changes, sign=True)

@@ -347,22 +352,16 @@ def tag(gh_token: Optional[str]):

@cli.command()
@click.option("--gh-token", envvar=["GH_TOKEN", "GITHUB_TOKEN"], required=True)
-def publish(gh_token: str):
-    """Publish release."""
+def publish(gh_token: str) -> None:
+    """Publish release on GitHub."""

    # Make sure we're in a git repo.
-    try:
-        repo = git.Repo()
-    except git.InvalidGitRepositoryError:
-        raise click.ClickException("Not in Synapse repo.")
-
-    if repo.is_dirty():
-        raise click.ClickException("Uncommitted changes exist.")
+    get_repo_and_check_clean_checkout()

    current_version = get_package_version()
    tag_name = f"v{current_version}"

-    if not click.confirm(f"Publish {tag_name}?", default=True):
+    if not click.confirm(f"Publish release {tag_name} on GitHub?", default=True):
        return

    # Publish the draft release
@@ -390,12 +389,19 @@ def publish(gh_token: str):


@cli.command()
-def upload():
+def upload() -> None:
    """Upload release to pypi."""

    current_version = get_package_version()
    tag_name = f"v{current_version}"

+    # Check we have the right tag checked out.
+    repo = get_repo_and_check_clean_checkout()
+    tag = repo.tag(f"refs/tags/{tag_name}")
+    if repo.head.commit != tag.commit:
+        click.echo("Tag {tag_name} (tag.commit) is not currently checked out!")
+        click.get_current_context().abort()
+
    pypi_asset_names = [
        f"matrix_synapse-{current_version}-py3-none-any.whl",
        f"matrix-synapse-{current_version}.tar.gz",
@@ -418,7 +424,7 @@ def upload():


@cli.command()
-def announce():
+def announce() -> None:
    """Generate markdown to announce the release."""

    current_version = get_package_version()
@@ -428,7 +434,7 @@ def announce():
        f"""
 Hi everyone. Synapse {current_version} has just been released.

-[notes](https://github.com/matrix-org/synapse/releases/tag/{tag_name}) |\
+[notes](https://github.com/matrix-org/synapse/releases/tag/{tag_name}) | \
 [docker](https://hub.docker.com/r/matrixdotorg/synapse/tags?name={tag_name}) | \
 [debs](https://packages.matrix.org/debian/) | \
 [pypi](https://pypi.org/project/matrix-synapse/{current_version}/)"""
@@ -459,20 +465,36 @@ def get_package_version() -> version.Version:
    return version.Version(version_string)


+def get_release_branch_name(version_number: version.Version) -> str:
+    return f"release-v{version_number.major}.{version_number.minor}"
+
+
+def get_repo_and_check_clean_checkout() -> git.Repo:
+    """Get the project repo and check it's not got any uncommitted changes."""
+    try:
+        repo = git.Repo()
+    except git.InvalidGitRepositoryError:
+        raise click.ClickException("Not in Synapse repo.")
+    if repo.is_dirty():
+        raise click.ClickException("Uncommitted changes exist.")
+    return repo
+
+
 def find_ref(repo: git.Repo, ref_name: str) -> Optional[git.HEAD]:
    """Find the branch/ref, looking first locally then in the remote."""
-    if ref_name in repo.refs:
-        return repo.refs[ref_name]
+    if ref_name in repo.references:
+        return repo.references[ref_name]
    elif ref_name in repo.remote().refs:
        return repo.remote().refs[ref_name]
    else:
        return None


-def update_branch(repo: git.Repo):
+def update_branch(repo: git.Repo) -> None:
    """Ensure branch is up to date if it has a remote"""
-    if repo.active_branch.tracking_branch():
-        repo.git.merge(repo.active_branch.tracking_branch().name)
+    tracking_branch = repo.active_branch.tracking_branch()
+    if tracking_branch:
+        repo.git.merge(tracking_branch.name)


 def get_changes_for_version(wanted_version: version.Version) -> str:
@@ -536,7 +558,9 @@ def get_changes_for_version(wanted_version: version.Version) -> str:
    return "\n".join(version_changelog)


-def generate_and_write_changelog(current_version: version.Version, new_version: str):
+def generate_and_write_changelog(
+    current_version: version.Version, new_version: str
+) -> None:
    # We do this by getting a draft so that we can edit it before writing to the
    # changelog.
    result = run_until_successful(
@@ -558,8 +582,8 @@ def generate_and_write_changelog(current_version: version.Version, new_version:
        f.write(existing_content)

    # Remove all the news fragments
-    for f in glob.iglob("changelog.d/*.*"):
-        os.remove(f)
+    for filename in glob.iglob("changelog.d/*.*"):
+        os.remove(filename)


 if __name__ == "__main__":
--- a/scripts-dev/sign_json.py
+++ b/scripts-dev/sign_json.py
@@ -27,7 +27,7 @@ from synapse.crypto.event_signing import add_hashes_and_signatures
 from synapse.util import json_encoder


-def main():
+def main() -> None:
    parser = argparse.ArgumentParser(
        description="""Adds a signature to a JSON object.

--- a/stubs/sortedcontainers/sorteddict.pyi
+++ b/stubs/sortedcontainers/sorteddict.pyi
@@ -85,12 +85,19 @@ class SortedDict(Dict[_KT, _VT]):
    def popitem(self, index: int = ...) -> Tuple[_KT, _VT]: ...
    def peekitem(self, index: int = ...) -> Tuple[_KT, _VT]: ...
    def setdefault(self, key: _KT, default: Optional[_VT] = ...) -> _VT: ...
-    @overload
-    def update(self, __map: Mapping[_KT, _VT], **kwargs: _VT) -> None: ...
-    @overload
-    def update(self, __iterable: Iterable[Tuple[_KT, _VT]], **kwargs: _VT) -> None: ...
-    @overload
-    def update(self, **kwargs: _VT) -> None: ...
+    # Mypy now reports the first overload as an error, because typeshed widened the type
+    # of `__map` to its internal `_typeshed.SupportsKeysAndGetItem` type in
+    #   https://github.com/python/typeshed/pull/6653
+    # Since sorteddicts don't change the signature of `update` from that of `dict`, we
+    # let the stubs for `update` inherit from the stubs for `dict`. (I suspect we could
+    # do the same for many othe methods.) We leave the stubs commented to better track
+    # how this file has evolved from the original stubs.
+    # @overload
+    # def update(self, __map: Mapping[_KT, _VT], **kwargs: _VT) -> None: ...
+    # @overload
+    # def update(self, __iterable: Iterable[Tuple[_KT, _VT]], **kwargs: _VT) -> None: ...
+    # @overload
+    # def update(self, **kwargs: _VT) -> None: ...
    def __reduce__(
        self,
    ) -> Tuple[
@@ -115,9 +122,7 @@ class SortedKeysView(KeysView[_KT_co], Sequence[_KT_co]):
    def __getitem__(self, index: slice) -> List[_KT_co]: ...
    def __delitem__(self, index: Union[int, slice]) -> None: ...

-class SortedItemsView(  # type: ignore
-    ItemsView[_KT_co, _VT_co], Sequence[Tuple[_KT_co, _VT_co]]
-):
+class SortedItemsView(ItemsView[_KT_co, _VT_co], Sequence[Tuple[_KT_co, _VT_co]]):
    def __iter__(self) -> Iterator[Tuple[_KT_co, _VT_co]]: ...
    @overload
    def __getitem__(self, index: int) -> Tuple[_KT_co, _VT_co]: ...
--- a/synapse/api/auth.py
+++ b/synapse/api/auth.py
@@ -187,7 +187,7 @@ class Auth:
        Once get_user_by_req has set up the opentracing span, this does the actual work.
        """
        try:
-            ip_addr = request.getClientIP()
+            ip_addr = request.getClientAddress().host
            user_agent = get_request_user_agent(request)

            access_token = self.get_access_token_from_request(request)
@@ -356,7 +356,7 @@ class Auth:
            return None, None, None

        if app_service.ip_range_whitelist:
-            ip_address = IPAddress(request.getClientIP())
+            ip_address = IPAddress(request.getClientAddress().host)
            if ip_address not in app_service.ip_range_whitelist:
                return None, None, None

@@ -417,7 +417,8 @@ class Auth:
        """

        if rights == "access":
-            # first look in the database
+            # First look in the database to see if the access token is present
+            # as an opaque token.
            r = await self.store.get_user_by_access_token(token)
            if r:
                valid_until_ms = r.valid_until_ms
@@ -434,7 +435,8 @@ class Auth:

                return r

-        # otherwise it needs to be a valid macaroon
+        # If the token isn't found in the database, then it could still be a
+        # macaroon, so we check that here.
        try:
            user_id, guest = self._parse_and_validate_macaroon(token, rights)

@@ -482,8 +484,12 @@ class Auth:
            TypeError,
            ValueError,
        ) as e:
-            logger.warning("Invalid macaroon in auth: %s %s", type(e), e)
-            raise InvalidClientTokenError("Invalid macaroon passed.")
+            logger.warning(
+                "Invalid access token in auth: %s %s.",
+                type(e),
+                e,
+            )
+            raise InvalidClientTokenError("Invalid access token passed.")

    def _parse_and_validate_macaroon(
        self, token: str, rights: str = "access"
@@ -504,10 +510,7 @@ class Auth:
        try:
            macaroon = pymacaroons.Macaroon.deserialize(token)
        except Exception:  # deserialize can throw more-or-less anything
-            # doesn't look like a macaroon: treat it as an opaque token which
-            # must be in the database.
-            # TODO: it would be nice to get rid of this, but apparently some
-            # people use access tokens which aren't macaroons
+            # The access token doesn't look like a macaroon.
            raise _InvalidMacaroonException()

        try:
--- a/synapse/api/constants.py
+++ b/synapse/api/constants.py
@@ -255,7 +255,5 @@ class GuestAccess:

 class ReceiptTypes:
    READ: Final = "m.read"
-
-
-class ReadReceiptEventFields:
-    MSC2285_HIDDEN: Final = "org.matrix.msc2285.hidden"
+    READ_PRIVATE: Final = "org.matrix.msc2285.read.private"
+    FULLY_READ: Final = "m.fully_read"
--- a/synapse/app/_base.py
+++ b/synapse/app/_base.py
@@ -38,6 +38,7 @@ from typing import (

 from cryptography.utils import CryptographyDeprecationWarning
 from matrix_common.versionstring import get_distribution_version_string
+from typing_extensions import ParamSpec

 import twisted
 from twisted.internet import defer, error, reactor as _reactor
@@ -48,7 +49,6 @@ from twisted.logger import LoggingFile, LogLevel
 from twisted.protocols.tls import TLSMemoryBIOFactory
 from twisted.python.threadpool import ThreadPool

-import synapse
 from synapse.api.constants import MAX_PDU_SIZE
 from synapse.app import check_bind_error
 from synapse.app.phone_stats_home import start_phone_stats_home
@@ -60,6 +60,7 @@ from synapse.events.spamcheck import load_legacy_spam_checkers
 from synapse.events.third_party_rules import load_legacy_third_party_event_rules
 from synapse.handlers.auth import load_legacy_password_auth_providers
 from synapse.logging.context import PreserveLoggingContext
+from synapse.logging.opentracing import init_tracer
 from synapse.metrics import install_gc_manager, register_threadpool
 from synapse.metrics.background_process_metrics import wrap_as_background_process
 from synapse.metrics.jemalloc import setup_jemalloc_stats
@@ -81,11 +82,12 @@ logger = logging.getLogger(__name__)

 # list of tuples of function, args list, kwargs dict
 _sighup_callbacks: List[
-    Tuple[Callable[..., None], Tuple[Any, ...], Dict[str, Any]]
+    Tuple[Callable[..., None], Tuple[object, ...], Dict[str, object]]
 ] = []
+P = ParamSpec("P")


-def register_sighup(func: Callable[..., None], *args: Any, **kwargs: Any) -> None:
+def register_sighup(func: Callable[P, None], *args: P.args, **kwargs: P.kwargs) -> None:
    """
    Register a function to be called when a SIGHUP occurs.

@@ -93,7 +95,9 @@ def register_sighup(func: Callable[..., None], *args: Any, **kwargs: Any) -> Non
        func: Function to be called when sent a SIGHUP signal.
        *args, **kwargs: args and kwargs to be passed to the target function.
    """
-    _sighup_callbacks.append((func, args, kwargs))
+    # This type-ignore should be redundant once we use a mypy release with
+    # https://github.com/python/mypy/pull/12668.
+    _sighup_callbacks.append((func, args, kwargs))  # type: ignore[arg-type]


 def start_worker_reactor(
@@ -214,7 +218,9 @@ def redirect_stdio_to_logs() -> None:
    print("Redirected stdout/stderr to logs")


-def register_start(cb: Callable[..., Awaitable], *args: Any, **kwargs: Any) -> None:
+def register_start(
+    cb: Callable[P, Awaitable], *args: P.args, **kwargs: P.kwargs
+) -> None:
    """Register a callback with the reactor, to be called once it is running

    This can be used to initialise parts of the system which require an asynchronous
@@ -431,7 +437,7 @@ async def start(hs: "HomeServer") -> None:
    refresh_certificate(hs)

    # Start the tracer
-    synapse.logging.opentracing.init_tracer(hs)  # type: ignore[attr-defined] # noqa
+    init_tracer(hs)  # noqa

    # Instantiate the modules so they can register their web resources to the module API
    # before we start the listeners.
--- a/synapse/app/generic_worker.py
+++ b/synapse/app/generic_worker.py
@@ -441,22 +441,6 @@ def start(config_options: List[str]) -> None:
        "synapse.app.user_dir",
    )

-    if config.worker.worker_app == "synapse.app.appservice":
-        if config.appservice.notify_appservices:
-            sys.stderr.write(
-                "\nThe appservices must be disabled in the main synapse process"
-                "\nbefore they can be run in a separate worker."
-                "\nPlease add ``notify_appservices: false`` to the main config"
-                "\n"
-            )
-            sys.exit(1)
-
-        # Force the appservice to start since they will be disabled in the main config
-        config.appservice.notify_appservices = True
-    else:
-        # For other worker types we force this to off.
-        config.appservice.notify_appservices = False
-
    if config.worker.worker_app == "synapse.app.user_dir":
        if config.server.update_user_directory:
            sys.stderr.write(
--- a/synapse/appservice/api.py
+++ b/synapse/appservice/api.py
@@ -17,6 +17,7 @@ import urllib.parse
 from typing import TYPE_CHECKING, Dict, Iterable, List, Optional, Tuple

 from prometheus_client import Counter
+from typing_extensions import TypeGuard

 from synapse.api.constants import EventTypes, Membership, ThirdPartyEntityKind
 from synapse.api.errors import CodeMessageException
@@ -66,7 +67,7 @@ def _is_valid_3pe_metadata(info: JsonDict) -> bool:
    return True


-def _is_valid_3pe_result(r: JsonDict, field: str) -> bool:
+def _is_valid_3pe_result(r: object, field: str) -> TypeGuard[JsonDict]:
    if not isinstance(r, dict):
        return False

--- a/synapse/config/appservice.py
+++ b/synapse/config/appservice.py
@@ -33,7 +33,6 @@ class AppServiceConfig(Config):

    def read_config(self, config: JsonDict, **kwargs: Any) -> None:
        self.app_service_config_files = config.get("app_service_config_files", [])
-        self.notify_appservices = config.get("notify_appservices", True)
        self.track_appservice_user_ips = config.get("track_appservice_user_ips", False)

    def generate_config_section(cls, **kwargs: Any) -> str:
@@ -56,7 +55,8 @@ def load_appservices(
 ) -> List[ApplicationService]:
    """Returns a list of Application Services from the config files."""
    if not isinstance(config_files, list):
-        logger.warning("Expected %s to be a list of AS config files.", config_files)
+        # type-ignore: this function gets arbitrary json value; we do use this path.
+        logger.warning("Expected %s to be a list of AS config files.", config_files)  # type: ignore[unreachable]
        return []

    # Dicts of value -> filename
--- a/synapse/config/experimental.py
+++ b/synapse/config/experimental.py
@@ -32,7 +32,7 @@ class ExperimentalConfig(Config):
        # MSC2716 (importing historical messages)
        self.msc2716_enabled: bool = experimental.get("msc2716_enabled", False)

-        # MSC2285 (hidden read receipts)
+        # MSC2285 (private read receipts)
        self.msc2285_enabled: bool = experimental.get("msc2285_enabled", False)

        # MSC3244 (room version capabilities)
--- a/synapse/config/federation.py
+++ b/synapse/config/federation.py
@@ -46,7 +46,7 @@ class FederationConfig(Config):
        )

        self.allow_device_name_lookup_over_federation = config.get(
-            "allow_device_name_lookup_over_federation", True
+            "allow_device_name_lookup_over_federation", False
        )

    def generate_config_section(self, **kwargs: Any) -> str:
@@ -81,11 +81,11 @@ class FederationConfig(Config):
        #
        #allow_profile_lookup_over_federation: false

-        # Uncomment to disable device display name lookup over federation. By default, the
-        # Federation API allows other homeservers to obtain device display names of any user
-        # on this homeserver. Defaults to 'true'.
+        # Uncomment to allow device display name lookup over federation. By default, the
+        # Federation API prevents other homeservers from obtaining the display names of
+        # user devices on this homeserver. Defaults to 'false'.
        #
-        #allow_device_name_lookup_over_federation: false
+        #allow_device_name_lookup_over_federation: true
        """


--- a/synapse/config/logger.py
+++ b/synapse/config/logger.py
@@ -110,13 +110,6 @@ loggers:
        # information such as access tokens.
        level: INFO

-    twisted:
-        # We send the twisted logging directly to the file handler,
-        # to work around https://github.com/matrix-org/synapse/issues/3471
-        # when using "buffer" logger. Use "console" to log to stderr instead.
-        handlers: [file]
-        propagate: false
-
 root:
    level: INFO

--- a/synapse/config/registration.py
+++ b/synapse/config/registration.py
@@ -43,6 +43,9 @@ class RegistrationConfig(Config):
        self.registration_requires_token = config.get(
            "registration_requires_token", False
        )
+        self.enable_registration_token_3pid_bypass = config.get(
+            "enable_registration_token_3pid_bypass", False
+        )
        self.registration_shared_secret = config.get("registration_shared_secret")

        self.bcrypt_rounds = config.get("bcrypt_rounds", 12)
@@ -309,6 +312,12 @@ class RegistrationConfig(Config):
        #
        #registration_requires_token: true

+        # Allow users to submit a token during registration to bypass any required 3pid
+        # steps configured in `registrations_require_3pid`.
+        # Defaults to false, requiring that registration tokens (if enabled) complete a 3pid flow.
+        #
+        #enable_registration_token_3pid_bypass: false
+
        # If set, allows registration of standard or admin accounts by anyone who
        # has the shared secret, even if registration is otherwise disabled.
        #
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
H. Shay	a6f6076c22	Merge branch 'shay/super_docs' of https://github.com/matrix-org/synapse into shay/super_docs	2022-05-09 13:51:08 -07:00
H. Shay	b9d5322c8a	fix newsfragment	2022-05-09 13:50:54 -07:00
Shay	35b96a09d5	Merge branch 'develop' into shay/super_docs	2022-05-09 13:46:12 -07:00
H. Shay	7dbedbeb1a	newsfragement	2022-05-09 13:39:57 -07:00
H. Shay	116a66e3af	outline new docs skeleton	2022-05-09 13:31:42 -07:00
Val Lorentz	bf0c3ca20a	Fix inconsistent spelling of 'M_UNRECOGNIZED'. (#12665 )	2022-05-09 20:29:07 +00:00
Sean Quah	a00462dd99	Implement cancellation support/protection for module callbacks (#12568 ) There's no guarantee that module callbacks will handle cancellation appropriately. Protect module callbacks with read semantics from cancellation and avoid swallowing `CancelledError`s that arise. Other module callbacks, such as the `on_*` callbacks, are presumed to live on code paths that involve writes and aren't cancellation-friendly. These module callbacks have been left alone. Signed-off-by: Sean Quah <seanq@element.io>	2022-05-09 12:31:14 +01:00
David Robertson	8de0facaae	Fix mypy against latest pillow stubs (#12671 )	2022-05-09 10:48:14 +00:00
Sean Quah	41a882e62d	Update changelog for #12587 to be more accurate (#12663 ) #12587 has fallen on the wrong side of the release cutoff to the rest of the related PRs. Signed-off-by: Sean Quah <seanq@element.io>	2022-05-09 11:34:39 +01:00
David Robertson	fa0eab9c8e	Use `ParamSpec` in a few places (#12667 )	2022-05-09 10:27:39 +00:00
Erik Johnston	c5969b346d	Don't error on unknown receipt types (#12670 ) Fixes #12669	2022-05-09 11:09:19 +01:00
Sheogorath	77258b6725	docs(contrib): Add link to documentation in dashboard (#12602 )	2022-05-09 10:08:31 +00:00
Eric Eastwood	18d6c18aa1	Fix docs on how to run specific Complement tests after recent `complement.sh` change (#12664 )	2022-05-09 10:38:32 +01:00
David Robertson	26c1ad71c5	Use `Concatenate` to annotate `do_execute` (#12666 )	2022-05-09 10:28:38 +01:00
David Robertson	0ce2201932	Move `pympler` back into the `all` extras (#12652 ) * Move `pympler` back into the `all` extras Undoes a change I made in #12381. I can't fully remember my reasoning, but this changed the contents of the debian packages in a backwards incompatible way. We're not aware of anyone who's been bitten by this, but we still want to fix it. To the reviewer: please be convinced that the debian packages will still contain pympler after this change. * Debian changelog entry to keep the linter happy	2022-05-07 13:40:58 +01:00
David Robertson	051a1c3f22	Convert stringy power levels to integers on room upgrade (#12657 )	2022-05-07 13:37:29 +01:00
Erik Johnston	4337d33a73	Prevent memory leak from reoccurring when presence is disabled. (#12656 )	2022-05-06 16:41:57 +00:00
David Robertson	2607b3e181	Update mypy to 0.950 and fix complaints (#12650 )	2022-05-06 12:35:20 +00:00
reivilibre	c2d50e9f6c	Add the `notify_appservices_from_worker` configuration option (superseding `notify_appservices`) to allow a generic worker to be designated as the worker to send traffic to Application Services. (#12452 )	2022-05-06 11:43:53 +01:00
Andrew Morgan	f1fbf75cfc	Merge branch 'master' into develop	2022-05-05 17:43:27 +01:00
DeepBlueV7.X	a377a43386	Support MSC3266 room summaries over federation (#11507 ) Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>	2022-05-05 15:25:00 +01:00
Andrew Morgan	3a8ee22911	Update v1.58.1 changelog entry with more familiar language	2022-05-05 15:15:32 +01:00
Andrew Morgan	bc149a18f6	link to relevant bug report in v1.58.1 changelog	2022-05-05 15:10:24 +01:00
Andrew Morgan	d2784b6567	Minor wording change to v1.58.1 release notes	2022-05-05 15:06:39 +01:00
Andrew Morgan	6a17a291a6	1.58.1	2022-05-05 15:05:58 +01:00
Andrew Morgan	e923fc20bd	Include extra dependency groups 'systemd' and 'cache_memory' in debian packages (#12640 ) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2022-05-05 13:51:15 +00:00
Šimon Brandner	ef86cf3d28	Update `_on_new_receipts()` to work with MSC2285 changes. (#12636 )	2022-05-05 13:25:51 +00:00
reivilibre	07fa53ec40	Improve comments and error messages around access tokens. (#12577 )	2022-05-05 13:39:59 +01:00
Henry	b8fa24b022	Use `docker/metadata-action` to generate docker image tags (#12573 ) Update the "Build docker images" GitHub Actions workflow to use `docker/metadata-action` to generate docker image tags, instead of a custom shell script. Signed-off-by: Henry <97804910+henryclw@users.noreply.github.com>	2022-05-05 12:36:42 +00:00
Šimon Brandner	9ae0253f4e	Use `private` instead of `hidden` in MSC2285 related code. (#12635 )	2022-05-05 12:31:25 +00:00
Patrick Cloke	f90d381c7b	Edits/annotations should not have any bundled aggregations calculated. (#12633 ) Fixes a regression from `8b309adb43` (#11660) and `b65acead42` (#11752) where events which themselves were an edit or an annotation could have bundled aggregations calculated, which is not allowed.	2022-05-05 08:15:12 -04:00
Patrick Cloke	ddc8bba00f	Remove unused receipt datastore methods. (#12632 ) The last usage was removed in `5a1dd297c3` (#8059).	2022-05-05 07:51:19 -04:00
Will Hunt	cc7656099d	Fix typo in some instances of enable_registration_token_3pid_bypass. (#12639 )	2022-05-05 07:11:52 -04:00
Erik Johnston	c0379d6e5b	Reduce log spam when running multiple event persisters (#12610 )	2022-05-05 10:20:23 +01:00
Will Hunt	2d74a8c178	Add `mau_appservice_trial_days` config (#12619 ) * Add mau_appservice_trial_days * Add a test * Tweaks * changelog * Ensure we sync after the delay * Fix types * Add config statement * Fix test * Reinstate logging that got removed * Fix feature name	2022-05-04 19:33:26 +01:00
Patrick Cloke	7fbf42499d	Use `getClientAddress` instead of `getClientIP`. (#12599 ) getClientIP was deprecated in Twisted 18.4.0, which also added getClientAddress. The Synapse minimum version for Twisted is currently 18.9.0, so all supported versions have the new API.	2022-05-04 14:11:21 -04:00
Šimon Brandner	116a4c8340	Implement changes to MSC2285 (hidden read receipts) (#12168 ) * Changes hidden read receipts to be a separate receipt type (instead of a field on `m.read`). * Updates the `/receipts` endpoint to accept `m.fully_read`.	2022-05-04 11:59:22 -04:00
Andrew Morgan	332cce8dcf	Disable device name lookup over federation by default (#12616 )	2022-05-04 16:41:40 +01:00
Patrick Cloke	ba3fd54bad	Remove unstable/unspecced login types. (#12597 ) * `m.login.jwt`, which was never specced and has been deprecated since Synapse 1.16.0. (`org.matrix.login.jwt` can be used instead.) * `uk.half-shot.msc2778.login.application_service`, which was stabilized as part of the Matrix spec v1.2 release.	2022-05-04 13:53:21 +00:00
Sean Quah	b2df0716bc	Improve logging for cancelled requests (#12587 ) Don't log stack traces for cancelled requests and use a custom HTTP status code of 499. Signed-off-by: Sean Quah <seanq@element.io>	2022-05-04 13:38:55 +01:00
Patrick Cloke	75dff3dc98	Include bundled aggregations for the latest event in a thread. (#12273 ) The `latest_event` field of the bundled aggregations for `m.thread` relations did not include bundled aggregations itself. This resulted in clients needing to immediately request the event from the server (and thus making it useless that the latest event itself was serialized instead of just including an event ID).	2022-05-04 08:38:18 -04:00
andrew do	01e625513a	remove constantly lib use and switch to enums. (#12624 )	2022-05-04 11:26:11 +00:00
Richard van der Hoff	873d467976	Fixes to the formatting of README.rst (#12627 ) Fixes a couple of formatting errors which were introduced in #12475.	2022-05-04 11:02:19 +01:00
Richard van der Hoff	96e0cdbc5a	Add a consistency check on events read from the database (#12620 ) I've seen a few errors which can only plausibly be explained by the calculated event id for an event being different from the ID of the event in the database. It should be cheap to check this, so let's do so and raise an exception.	2022-05-03 21:27:52 +01:00
David Robertson	9ce51a47f6	Bump Synapse minimum Python version to 3.7.1 (#12613 )	2022-05-03 19:22:06 +01:00
Patrick Cloke	aa5f5ede33	Remove unstable identifiers for MSC3069. (#12596 )	2022-05-03 12:43:12 -04:00
Richard van der Hoff	d66d68f917	Add extra debug logging to federation sender (#12614 ) ... in order to debug some problems we've been having with certain events not being sent when expected.	2022-05-03 16:32:40 +01:00
Andrew Morgan	c4514b97db	Add missing space before 'docker' link in release announcement script (#12612 )	2022-05-03 14:46:42 +00:00
Richard van der Hoff	77dee1b451	fix imports broken in `5938928` :-S	2022-05-03 13:59:28 +01:00
Richard van der Hoff	5938928c59	minor wording fix in docstring	2022-05-03 13:50:50 +01:00
Richard van der Hoff	db2edf5a65	Exclude OOB memberships from the federation sender (#12570 ) As the comment says, there is no need to process such events, and indeed we need to avoid doing so. Fixes #12509.	2022-05-03 12:47:56 +00:00
Andrew Morgan	13e4386710	Merge branch 'master' into develop	2022-05-03 11:51:24 +01:00
David Robertson	bf2fea8f7d	Add sanity checks to the release script (#12556 ) Check we're on the right branch before tagging, and on the right tag before uploading * Abort if we're on the wrong branch * Check we have the right tag checked out * Clarify that `publish` only releases to GitHub	2022-05-03 10:50:03 +00:00
Erik Johnston	ae7858f184	Fix race when persisting an event and deleting a room (#12594 ) This works by taking a row level lock on the `rooms` table at the start of both transactions, ensuring that they don't run at the same time. In the event persistence transaction we also check that there is an entry still in the `rooms` table. I can't figure out how to do this in SQLite. I was just going to lock the table, but it seems that we don't support that in SQLite either, so I'm really confused as to how we maintain integrity in SQLite when using `lock_table`....	2022-05-03 11:47:21 +01:00
David Robertson	01dcf7532d	Prune mypy ignore_missing_imports list (#12608 )	2022-05-03 11:03:20 +01:00
Andrew Morgan	7e6598bcf6	Move groups/communities deprecation notice to 1.58.0 heading	2022-05-03 10:54:20 +01:00
Andrew Morgan	8f5d2823df	1.58.0	2022-05-03 10:53:09 +01:00
Richard van der Hoff	8d156ec0ba	Remove special-case for `twisted` logger (#12589 ) This was originally added when we first added a `MemoryHandler` to the default log config back in https://github.com/matrix-org/synapse/pull/8040, to ensure that we didn't explode with an infinite loop if there was an error formatting the logs. Since then, we made additional improvements to logging which make this workaround redundant. In particular: * we no longer attempt to log un-UTF8-decodable byte sequences, which were the most likely cause of an error in the first place. * https://github.com/matrix-org/synapse/pull/8268 ensures that in the unlikely case that there is an error, it won't cause an infinite loop.	2022-04-29 22:05:18 +01:00
David Robertson	57fac2a234	Allow unused ignores in "bleeding edge" CI (#12576 ) * Allow unused ignores in "bleeding edge" CI Where "bleeding edge" means the Twisted Trunk and Latest Deps jobs. Follow up from #12531. Resolves #12574. * Use `--extras all` in latest deps mypy CI Twisted trunk job already does this. Missed in #12531. * changelog	2022-04-29 17:57:23 +01:00
Patrick Cloke	3ae56d125c	Improve the docstrings for the receipts store. (#12581 )	2022-04-28 17:58:58 +00:00
Šimon Brandner	0d9eaa19fd	Use constants for receipt types in tests. (#12582 )	2022-04-28 13:34:33 -04:00
Sean Quah	0b684b59e5	Fix logging of incorrect status codes for disconnected requests (#12580 ) The status code of requests must always be set, regardless of client disconnection, otherwise they will always be logged as 200!. Broken for `respond_with_json` in `f48792eec4`. Broken for `respond_with_json_bytes` in `3e58ce72b4`. Broken for `respond_with_html_bytes` in `ea26e9a98b`. Signed-off-by: Sean Quah <seanq@element.io>	2022-04-28 15:49:50 +00:00
DeepBlueV7.X	629aa51743	Add linebreak to pipx install quote in README (#12579 )	2022-04-28 13:54:46 +01:00
David Robertson	5d3509dfda	Revert accidental direct-to-develop commits. This reverts commit `5a320baa45`. This reverts commit `f282d5fc11`. This reverts commit `ce6ecdd4b4`.	2022-04-28 11:33:05 +01:00
David Robertson	5a320baa45	changelog	2022-04-28 11:31:26 +01:00
David Robertson	f282d5fc11	Use `--extras all` in latest deps mypy CI Twisted trunk job already does this. Missed in #12531.	2022-04-28 11:29:13 +01:00
David Robertson	ce6ecdd4b4	Allow unused ignores in "bleeding edge" CI Where "bleeding edge" means the Twisted Trunk and Latest Deps jobs. Follow up from #12531. Resolves #12574.	2022-04-28 11:28:22 +01:00
Sean Quah	78b99de7c2	Prefer `make_awaitable` over `defer.succeed` in tests (#12505 ) When configuring the return values of mocks, prefer awaitables from `make_awaitable` over `defer.succeed`. `Deferred`s are only awaitable once, so it is inappropriate for a mock to return the same `Deferred` multiple times. Also update `run_in_background` to support functions that return arbitrary awaitables. Signed-off-by: Sean Quah <seanq@element.io>	2022-04-27 14:58:26 +01:00
Brendan Abolivier	5ef673de4f	Add a module API to allow modules to edit push rule actions (#12406 ) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2022-04-27 13:55:33 +00:00
reivilibre	d743b25c8f	Use supervisord to supervise Postgres and Caddy in the Complement image. (#12480 ) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2022-04-27 14:39:41 +01:00
David Robertson	30c8e7e408	Make `scripts-dev` pass `mypy --disallow-untyped-defs` (#12356 ) Not enforced in config yet. One day.	2022-04-27 13:10:31 +00:00
David Robertson	6463244375	Remove unused `# type: ignore`s (#12531 ) Over time we've begun to use newer versions of mypy, typeshed, stub packages---and of course we've improved our own annotations. This makes some type ignore comments no longer necessary. I have removed them. There was one exception: a module that imports `select.epoll`. The ignore is redundant on Linux, but I've kept it ignored for those of us who work on the source tree using not-Linux. (#11771) I'm more interested in the config line which enforces this. I want unused ignores to be reported, because I think it's useful feedback when annotating to know when you've fixed a problem you had to previously ignore. * Installing extras before typechecking Lacking an easy way to install all extras generically, let's bite the bullet and make install the hand-maintained `all` extra before typechecking. Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to the release/v1 branch.	2022-04-27 14:03:44 +01:00
Patrick Cloke	8a23bde823	Consistently use collections.abc.Mapping to check frozendict. (#12564 )	2022-04-27 09:00:07 -04:00
Will Hunt	e8d1ec0e92	Add option to enable token registration without requiring 3pids (#12526 )	2022-04-27 12:57:53 +00:00
				`@@ -0,0 +1 @@`
				`Support [MSC3266](https://github.com/matrix-org/matrix-doc/pull/3266) room summaries over federation.`
				`@@ -0,0 +1 @@`
				`Implement [changes](https://github.com/matrix-org/matrix-spec-proposals/pull/2285/commits/4a77139249c2e830aec3c7d6bd5501a514d1cc27) to [MSC2285 (hidden read receipts)](https://github.com/matrix-org/matrix-spec-proposals/pull/2285). Contributed by @SimonBrandner.`
				`@@ -0,0 +1 @@`
				`Fix a bug introduced in Synapse v1.48.0 where latest thread reply provided failed to include the proper bundled aggregations.`
				`@@ -0,0 +1 @@`
				`Add a module API to allow modules to change actions for existing push rules of local users.`
				`@@ -0,0 +1 @@`
				Add the `notify_appservices_from_worker` configuration option (superseding `notify_appservices`) to allow a generic worker to be designated as the worker to send traffic to Application Services.
				`@@ -0,0 +1 @@`
				`Use supervisord to supervise Postgres and Caddy in the Complement image to reduce restart time.`
				`@@ -0,0 +1 @@`
				Use `make_awaitable` instead of `defer.succeed` for return values of mocks in tests.
				`@@ -0,0 +1 @@`
				Add new `enable_registration_token_3pid_bypass` configuration option to allow registrations via token as an alternative to verifying a 3pid.
				`@@ -0,0 +1 @@`
				`Release script: confirm the commit to be tagged before tagging.`