Note that AS interest via room ID or alias isn't respected

Typing and read receipt events (those that are associated with a room) should be sent to an AS that has registered interest in that room, according to MSC2409: https://github.com/Sorunome/matrix-doc/blob/231084da132af6518ecefa2a42c37937dfe373d2/proposals/2409-appservice-edus.md#expectations-of-when-an-edu-should-be-pushed-to-an-appservice
newsfile
2021-10-20 18:23:56 +01:00 · 2021-10-20 18:04:44 +01:00 · 2021-10-20 17:58:53 +01:00 · 2021-10-20 17:58:51 +01:00 · 2021-10-20 17:42:43 +01:00 · 2021-10-20 17:11:45 +01:00
735 changed files with 17319 additions and 43829 deletions
@@ -3,7 +3,7 @@
 # Test for the export-data admin command against sqlite and postgres

 set -xe
-cd "$(dirname "$0")/../.."
+cd `dirname $0`/../..

 echo "--- Install dependencies"

@@ -21,7 +21,7 @@ python -m synapse.app.homeserver --generate-keys -c .ci/sqlite-config.yaml
 echo "--- Prepare test database"

 # Make sure the SQLite3 database is using the latest schema and has no pending background update.
-update_synapse_database --database-config .ci/sqlite-config.yaml --run-background-updates
+scripts/update_synapse_database --database-config .ci/sqlite-config.yaml --run-background-updates

 # Run the export-data command on the sqlite test database
 python -m synapse.app.admin_cmd -c .ci/sqlite-config.yaml  export-data @anon-20191002_181700-832:localhost:8800 \
@@ -41,7 +41,7 @@ fi

 # Port the SQLite databse to postgres so we can check command works against postgres
 echo "+++ Port SQLite3 databse to postgres"
-synapse_port_db --sqlite-database .ci/test_db.db --postgres-config .ci/postgres-config.yaml
+scripts/synapse_port_db --sqlite-database .ci/test_db.db --postgres-config .ci/postgres-config.yaml

 # Run the export-data command on postgres database
 python -m synapse.app.admin_cmd -c .ci/postgres-config.yaml  export-data @anon-20191002_181700-832:localhost:8800 \
@@ -1,20 +1,16 @@
 #!/usr/bin/env bash
-# this script is run by GitHub Actions in a plain `focal` container; it installs the
-# minimal requirements for tox and hands over to the py3-old tox environment.

-# Prevent tzdata from asking for user input
-export DEBIAN_FRONTEND=noninteractive
+# this script is run by GitHub Actions in a plain `bionic` container; it installs the
+# minimal requirements for tox and hands over to the py3-old tox environment.

 set -ex

 apt-get update
-apt-get install -y \
-        python3 python3-dev python3-pip python3-venv \
-        libxml2-dev libxslt-dev xmlsec1 zlib1g-dev tox libjpeg-dev libwebp-dev
+apt-get install -y python3 python3-dev python3-pip libxml2-dev libxslt-dev xmlsec1 zlib1g-dev tox

 export LANG="C.UTF-8"

 # Prevent virtualenv from auto-updating pip to an incompatible version
 export VIRTUALENV_NO_DOWNLOAD=1

-exec tox -e py3-old
+exec tox -e py3-old,combine
@@ -7,7 +7,7 @@


 set -xe
-cd "$(dirname "$0")/../.."
+cd `dirname $0`/../..

 echo "--- Install dependencies"

@@ -25,19 +25,17 @@ python -m synapse.app.homeserver --generate-keys -c .ci/sqlite-config.yaml
 echo "--- Prepare test database"

 # Make sure the SQLite3 database is using the latest schema and has no pending background update.
-update_synapse_database --database-config .ci/sqlite-config.yaml --run-background-updates
+scripts/update_synapse_database --database-config .ci/sqlite-config.yaml --run-background-updates

 # Create the PostgreSQL database.
 .ci/scripts/postgres_exec.py "CREATE DATABASE synapse"

 echo "+++ Run synapse_port_db against test database"
-# TODO: this invocation of synapse_port_db (and others below) used to be prepended with `coverage run`,
-# but coverage seems unable to find the entrypoints installed by `pip install -e .`.
-synapse_port_db --sqlite-database .ci/test_db.db --postgres-config .ci/postgres-config.yaml
+coverage run scripts/synapse_port_db --sqlite-database .ci/test_db.db --postgres-config .ci/postgres-config.yaml

 # We should be able to run twice against the same database.
 echo "+++ Run synapse_port_db a second time"
-synapse_port_db --sqlite-database .ci/test_db.db --postgres-config .ci/postgres-config.yaml
+coverage run scripts/synapse_port_db --sqlite-database .ci/test_db.db --postgres-config .ci/postgres-config.yaml

 #####

@@ -48,7 +46,7 @@ echo "--- Prepare empty SQLite database"
 # we do this by deleting the sqlite db, and then doing the same again.
 rm .ci/test_db.db

-update_synapse_database --database-config .ci/sqlite-config.yaml --run-background-updates
+scripts/update_synapse_database --database-config .ci/sqlite-config.yaml --run-background-updates

 # re-create the PostgreSQL database.
 .ci/scripts/postgres_exec.py \
@@ -56,4 +54,4 @@ update_synapse_database --database-config .ci/sqlite-config.yaml --run-backgroun
  "CREATE DATABASE synapse"

 echo "+++ Run synapse_port_db against empty database"
-synapse_port_db --sqlite-database .ci/test_db.db --postgres-config .ci/postgres-config.yaml
+coverage run scripts/synapse_port_db --sqlite-database .ci/test_db.db --postgres-config .ci/postgres-config.yaml
@@ -3,9 +3,11 @@

 # things to include
 !docker
+!scripts
 !synapse
 !MANIFEST.in
 !README.rst
 !setup.py
+!synctl

 **/__pycache__
@@ -1,11 +0,0 @@
-# TODO: incorporate this into pyproject.toml if flake8 supports it in the future.
-# See https://github.com/PyCQA/flake8/issues/234
-[flake8]
-# see https://pycodestyle.readthedocs.io/en/latest/intro.html#error-codes
-# for error codes. The ones we ignore are:
-#  W503: line break before binary operator
-#  W504: line break after binary operator
-#  E203: whitespace before ':' (which is contrary to pep8?)
-#  E731: do not assign a lambda expression, use a def
-#  E501: Line too long (black enforces this for us)
-ignore=W503,W504,E203,E731,E501
@@ -1,14 +1,12 @@
 ### Pull Request Checklist

-<!-- Please read https://matrix-org.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request -->
+<!-- Please read CONTRIBUTING.md before submitting your pull request -->

 * [ ] Pull request is based on the develop branch
-* [ ] Pull request includes a [changelog file](https://matrix-org.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should:
+* [ ] Pull request includes a [changelog file](https://github.com/matrix-org/synapse/blob/master/CONTRIBUTING.md#changelog). The entry should:
  - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
  - Use markdown where necessary, mostly for `code blocks`.
  - End with either a period (.) or an exclamation mark (!).
  - Start with a capital letter.
-  - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
-* [ ] Pull request includes a [sign off](https://matrix-org.github.io/synapse/latest/development/contributing_guide.html#sign-off)
-* [ ] [Code style](https://matrix-org.github.io/synapse/latest/code_style.html) is correct
-  (run the [linters](https://matrix-org.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
+* [ ] Pull request includes a [sign off](https://github.com/matrix-org/synapse/blob/master/CONTRIBUTING.md#sign-off)
+* [ ] Code style is correct (run the [linters](https://github.com/matrix-org/synapse/blob/master/CONTRIBUTING.md#code-style))
@@ -5,7 +5,7 @@ name: Build docker images
 on:
  push:
    tags: ["v*"]
-    branches: [ master, main, develop ]
+    branches: [ master, main ]
  workflow_dispatch:

 permissions:
@@ -34,15 +34,10 @@ jobs:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

-      # TODO: consider using https://github.com/docker/metadata-action instead of this
-      # custom magic
      - name: Calculate docker image tag
        id: set-tag
        run: |
          case "${GITHUB_REF}" in
-              refs/heads/develop)
-                  tag=develop
-                  ;;
              refs/heads/master|refs/heads/main)
                  tag=latest
                  ;;
@@ -55,6 +50,18 @@ jobs:
          esac
          echo "::set-output name=tag::$tag"

+        # for release builds, we want to get the amd64 image out asap, so first
+        # we do an amd64-only build, before following up with a multiarch build.
+      - name: Build and push amd64
+        uses: docker/build-push-action@v2
+        if: "${{ startsWith(github.ref, 'refs/tags/v') }}"
+        with:
+          push: true
+          labels: "gitsha1=${{ github.sha }}"
+          tags: "matrixdotorg/synapse:${{ steps.set-tag.outputs.tag }}"
+          file: "docker/Dockerfile"
+          platforms: linux/amd64
+
      - name: Build and push all platforms
        uses: docker/build-push-action@v2
        with:
@@ -7,7 +7,7 @@ on:
  # of things breaking (but only build one set of debs)
  pull_request:
  push:
-    branches: ["develop", "release-*"]
+    branches: ["develop"]

    # we do the full build on tags.
    tags: ["v*"]
@@ -31,7 +31,7 @@ jobs:
          # if we're running from a tag, get the full list of distros; otherwise just use debian:sid
          dists='["debian:sid"]'
          if [[ $GITHUB_REF == refs/tags/* ]]; then
-              dists=$(scripts-dev/build_debian_packages.py --show-dists-json)
+              dists=$(scripts-dev/build_debian_packages --show-dists-json)
          fi
          echo "::set-output name=distros::$dists"
    # map the step outputs to job outputs
@@ -74,7 +74,7 @@ jobs:
        # see https://github.com/docker/build-push-action/issues/252
        # for the cache magic here
        run: |
-          ./src/scripts-dev/build_debian_packages.py \
+          ./src/scripts-dev/build_debian_packages \
            --docker-build-arg=--cache-from=type=local,src=/tmp/.buildx-cache \
            --docker-build-arg=--cache-to=type=local,mode=max,dest=/tmp/.buildx-cache-new \
            --docker-build-arg=--progress=plain \
@@ -91,7 +91,17 @@ jobs:

  build-sdist:
    name: "Build pypi distribution files"
-    uses: "matrix-org/backend-meta/.github/workflows/packaging.yml@v1"
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+      - run: pip install wheel
+      - run: |
+          python setup.py sdist bdist_wheel
+      - uses: actions/upload-artifact@v2
+        with:
+          name: python-dist
+          path: dist/*

  # if it's a tag, create a release and attach the artifacts to it
  attach-assets:
@@ -112,8 +122,7 @@ jobs:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          files: |
-            Sdist/*
-            Wheel/*
+            python-dist/*
            debs.tar.xz
          # if it's not already published, keep the release as a draft.
          draft: true
@@ -10,20 +10,12 @@ concurrency:
  cancel-in-progress: true

 jobs:
-  check-sampleconfig:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v2
-      - uses: actions/setup-python@v2
-      - run: pip install -e .
-      - run: scripts-dev/generate_sample_config.sh --check
-      - run: scripts-dev/config-lint.sh
-
  lint:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        toxenv:
+          - "check-sampleconfig"
          - "check_codestyle"
          - "check_isort"
          - "mypy"
@@ -51,15 +43,29 @@ jobs:
          ref: ${{ github.event.pull_request.head.sha }}
          fetch-depth: 0
      - uses: actions/setup-python@v2
-      - run: "pip install 'towncrier>=18.6.0rc1'"
-      - run: scripts-dev/check-newsfragment.sh
+      - run: pip install tox
+      - run: scripts-dev/check-newsfragment
        env:
          PULL_REQUEST_NUMBER: ${{ github.event.number }}

+  lint-sdist:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+        with:
+          python-version: "3.x"
+      - run: pip install wheel
+      - run: python setup.py sdist bdist_wheel
+      - uses: actions/upload-artifact@v2
+        with:
+          name: Python Distributions
+          path: dist/*
+
  # Dummy step to gate other tests on without repeating the whole list
  linting-done:
    if: ${{ !cancelled() }} # Run this even if prior jobs were skipped
-    needs: [lint, lint-crlf, lint-newsfile, check-sampleconfig]
+    needs: [lint, lint-crlf, lint-newsfile, lint-sdist]
    runs-on: ubuntu-latest
    steps:
      - run: "true"
@@ -70,7 +76,7 @@ jobs:
    runs-on: ubuntu-latest
    strategy:
      matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10"]
+        python-version: ["3.6", "3.7", "3.8", "3.9", "3.10"]
        database: ["sqlite"]
        toxenv: ["py"]
        include:
@@ -79,9 +85,9 @@ jobs:
            toxenv: "py-noextras"

          # Oldest Python with PostgreSQL
-          - python-version: "3.7"
+          - python-version: "3.6"
            database: "postgres"
-            postgres-version: "10"
+            postgres-version: "9.6"
            toxenv: "py"

          # Newest Python with newest PostgreSQL
@@ -135,7 +141,7 @@ jobs:
    steps:
      - uses: actions/checkout@v2
      - name: Test with old deps
-        uses: docker://ubuntu:focal # For old python and sqlite
+        uses: docker://ubuntu:bionic # For old python and sqlite
        with:
          workdir: /github/workspace
          entrypoint: .ci/scripts/test_old_deps.sh
@@ -161,7 +167,7 @@ jobs:
    runs-on: ubuntu-latest
    strategy:
      matrix:
-        python-version: ["pypy-3.7"]
+        python-version: ["pypy-3.6"]

    steps:
      - uses: actions/checkout@v2
@@ -207,15 +213,15 @@ jobs:
      fail-fast: false
      matrix:
        include:
-          - sytest-tag: focal
+          - sytest-tag: bionic

-          - sytest-tag: focal
+          - sytest-tag: bionic
            postgres: postgres

          - sytest-tag: testing
            postgres: postgres

-          - sytest-tag: focal
+          - sytest-tag: bionic
            postgres: multi-postgres
            workers: workers

@@ -285,8 +291,8 @@ jobs:
    strategy:
      matrix:
        include:
-          - python-version: "3.7"
-            postgres-version: "10"
+          - python-version: "3.6"
+            postgres-version: "9.6"

          - python-version: "3.10"
            postgres-version: "14"
@@ -317,29 +323,24 @@ jobs:
    if: ${{ !failure() && !cancelled() }}
    needs: linting-done
    runs-on: ubuntu-latest
+    container:
+      # https://github.com/matrix-org/complement/blob/master/dockerfiles/ComplementCIBuildkite.Dockerfile
+      image: matrixdotorg/complement:latest
+      env:
+        CI: true
+      ports:
+        - 8448:8448
+      volumes:
+        - /var/run/docker.sock:/var/run/docker.sock

    steps:
-      # The path is set via a file given by $GITHUB_PATH. We need both Go 1.17 and GOPATH on the path to run Complement.
-      # See https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#adding-a-system-path
-      - name: "Set Go Version"
-        run: |
-          # Add Go 1.17 to the PATH: see https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu2004-Readme.md#environment-variables-2
-          echo "$GOROOT_1_17_X64/bin" >> $GITHUB_PATH
-          # Add the Go path to the PATH: We need this so we can call gotestfmt
-          echo "~/go/bin" >> $GITHUB_PATH
-
-      - name: "Install Complement Dependencies"
-        run: |
-          sudo apt-get update && sudo apt-get install -y libolm3 libolm-dev
-          go get -v github.com/haveyoudebuggedit/gotestfmt/v2/cmd/gotestfmt@latest
-
      - name: Run actions/checkout@v2 for synapse
        uses: actions/checkout@v2
        with:
          path: synapse

      # Attempt to check out the same branch of Complement as the PR. If it
-      # doesn't exist, fallback to HEAD.
+      # doesn't exist, fallback to master.
      - name: Checkout complement
        shell: bash
        run: |
@@ -352,8 +353,8 @@ jobs:
          #    for pull requests, otherwise GITHUB_REF).
          # 2. Attempt to use the base branch, e.g. when merging into release-vX.Y
          #    (GITHUB_BASE_REF for pull requests).
-          # 3. Use the default complement branch ("HEAD").
-          for BRANCH_NAME in "$GITHUB_HEAD_REF" "$GITHUB_BASE_REF" "${GITHUB_REF#refs/heads/}" "HEAD"; do
+          # 3. Use the default complement branch ("master").
+          for BRANCH_NAME in "$GITHUB_HEAD_REF" "$GITHUB_BASE_REF" "${GITHUB_REF#refs/heads/}" "master"; do
            # Skip empty branch names and merge commits.
            if [[ -z "$BRANCH_NAME" || $BRANCH_NAME =~ ^refs/pull/.* ]]; then
              continue
@@ -365,8 +366,6 @@ jobs:
      # Build initial Synapse image
      - run: docker build -t matrixdotorg/synapse:latest -f docker/Dockerfile .
        working-directory: synapse
-        env:
-          DOCKER_BUILDKIT: 1

      # Build a ready-to-run Synapse image based on the initial image above.
      # This new image includes a config file, keys for signing and TLS, and
@@ -375,11 +374,7 @@ jobs:
        working-directory: complement/dockerfiles

      # Run Complement
-      - run: |
-          set -o pipefail
-          go test -v -json -p 1 -tags synapse_blacklist,msc2403,msc2716,msc3030 ./tests/... 2>&1 | gotestfmt
-        shell: bash
-        name: Run Complement Tests
+      - run: go test -v -tags synapse_blacklist,msc2403,msc2946,msc3083 ./tests/...
        env:
          COMPLEMENT_BASE_IMAGE: complement-synapse:latest
        working-directory: complement
@@ -388,22 +383,35 @@ jobs:
  tests-done:
    if: ${{ always() }}
    needs:
-      - check-sampleconfig
      - lint
      - lint-crlf
      - lint-newsfile
+      - lint-sdist
      - trial
      - trial-olddeps
      - sytest
-      - export-data
      - portdb
      - complement
    runs-on: ubuntu-latest
    steps:
-      - uses: matrix-org/done-action@v2
-        with:
-          needs: ${{ toJSON(needs) }}
+      - name: Set build result
+        env:
+          NEEDS_CONTEXT: ${{ toJSON(needs) }}
+        # the `jq` incantation dumps out a series of "<job> <result>" lines.
+        # we set it to an intermediate variable to avoid a pipe, which makes it
+        # hard to set $rc.
+        run: |
+          rc=0
+          results=$(jq -r 'to_entries[] | [.key,.value.result] | join(" ")' <<< $NEEDS_CONTEXT)
+          while read job result ; do
+              # The newsfile lint may be skipped on non PR builds
+              if [ $result == "skipped" ] && [ $job == "lint-newsfile" ]; then
+                continue
+              fi

-          # The newsfile lint may be skipped on non PR builds
-          skippable:
-            lint-newsfile
+              if [ "$result" != "success" ]; then
+                  echo "::set-failed ::Job $job returned $result"
+                  rc=1
+              fi
+          done <<< $results
+          exit $rc
@@ -25,7 +25,7 @@ jobs:
      - run: sudo apt-get -qq install xmlsec1
      - uses: actions/setup-python@v2
        with:
-          python-version: 3.7
+          python-version: 3.6
      - run: .ci/patch_for_twisted_trunk.sh
      - run: pip install tox
      - run: tox -e py
@@ -50,7 +50,3 @@ __pycache__/

 # docs
 book/
-
-# complement
-/complement-*
-/master.tar.gz
@@ -1,3 +1,4 @@
+include synctl
 include LICENSE
 include VERSION
 include *.rst
@@ -16,6 +17,7 @@ recursive-include synapse/storage *.txt
 recursive-include synapse/storage *.md

 recursive-include docs *
+recursive-include scripts *
 recursive-include scripts-dev *
 recursive-include synapse *.pyi
 recursive-include tests *.py
@@ -43,7 +45,6 @@ include book.toml
 include pyproject.toml
 recursive-include changelog.d *

-include .flake8
 prune .circleci
 prune .github
 prune .ci
@@ -51,4 +52,5 @@ prune contrib
 prune debian
 prune demo/etc
 prune docker
+prune snap
 prune stubs
@@ -246,7 +246,7 @@ Password reset
 ==============

 Users can reset their password through their client. Alternatively, a server admin
-can reset a users password using the `admin API <docs/admin_api/user_admin_api.md#reset-password>`_
+can reset a users password using the `admin API <docs/admin_api/user_admin_api.rst#reset-password>`_
 or by directly editing the database as shown below.

 First calculate the hash of the new password::
@@ -312,9 +312,6 @@ We recommend using the demo which starts 3 federated instances running on ports

 (to stop, you can use `./demo/stop.sh`)

-See the [demo documentation](https://matrix-org.github.io/synapse/develop/development/demo.html)
-for more information.
-
 If you just want to start a single instance of the app and run it directly::

    # Create the homeserver.yaml config once
@@ -0,0 +1 @@
+Port the Password Auth Providers module interface to the new generic interface.
@@ -0,0 +1 @@
+Add type hints to `synapse.storage.databases.main.client_ips`.
@@ -0,0 +1 @@
+Resolve and share `state_groups` for all [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) historical events in batch.
@@ -0,0 +1 @@
+Fix spurious warnings about losing the logging context on the `ReplicationCommandHandler` when losing the replication connection.
@@ -0,0 +1 @@
+ Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state.
@@ -0,0 +1 @@
+Include rejected status when we log events.
@@ -0,0 +1 @@
+Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state.
@@ -0,0 +1 @@
+Add some extra logging to the event persistence code.
@@ -0,0 +1 @@
+Add support for Ubuntu 21.10 "Impish Indri".
@@ -0,0 +1 @@
+Fix 500 error on `/messages` when the server accumulates more than 5 backwards extremities at a given depth for a room.
@@ -0,0 +1 @@
+Rearrange the internal workings of the incremental user directory updates.
@@ -0,0 +1 @@
+Simplify the user admin API tests.
@@ -0,0 +1 @@
+Mark the Synapse package as containing type annotations and fix export declarations so that Synapse pluggable modules may be type checked against Synapse.
@@ -0,0 +1 @@
+Improve type hints for `_wrap_in_base_path` decorator used by `MediaFilePaths`.
@@ -0,0 +1 @@
+Remove dead code from `MediaFilePaths`.
@@ -0,0 +1 @@
+Add tests for `MediaFilePaths` class.
@@ -0,0 +1 @@
+Be more lenient when parsing oEmbed response versions.
@@ -0,0 +1 @@
+Add type hints to `synapse.events`.
@@ -0,0 +1 @@
+Always dump logs from unit tests during CI runs.
@@ -0,0 +1 @@
+Fix broken links relating to module API deprecation in the upgrade notes.
@@ -0,0 +1 @@
+Create a separate module for the retention configuration.
@@ -0,0 +1 @@
+Add a test for the workaround introduced in [\#11042](https://github.com/matrix-org/synapse/pull/11042) concerning the behaviour of third-party rule modules and `SynapseError`s.
@@ -0,0 +1 @@
+Fix a long-standing bug where users excluded from the user directory were added into the directory if they belonged to a room which became public or private.
@@ -0,0 +1 @@
+Fix type hints in the relations tests.
@@ -0,0 +1 @@
+Fix a long-standing bug when attempting to preview URLs which are in the `windows-1252` character encoding.
@@ -0,0 +1 @@
+Fix broken export-data admin command and add test script checking the command to CI.
@@ -0,0 +1 @@
+Fix a long-standing bug when attempting to preview URLs which are in the `windows-1252` character encoding.
@@ -0,0 +1 @@
+Clarify the the sample log config can be copied from the documentation without issue.
@@ -0,0 +1 @@
+Update the admin API documentation with an updated list of the characters allowed in registration tokens.
@@ -0,0 +1 @@
+Document Synapse's behaviour when dealing with multiple modules registering the same callbacks and/or handlers for the same HTTP endpoints.
@@ -0,0 +1 @@
+Fix local users who left all their rooms being removed from the user directory, even if the "search_all_users" config option was enabled.
@@ -0,0 +1 @@
+Add missing type hints to `synapse.api` module.
@@ -0,0 +1 @@
+Clean up some of the federation event authentication code for clarity.
@@ -0,0 +1 @@
+Clean up some of the federation event authentication code for clarity.
@@ -0,0 +1 @@
+Fix instances of `[example]{.title-ref}` in the upgrade documentation as a result of prior RST to Markdown conversion.
@@ -0,0 +1 @@
+Add type hints for event fetching.
@@ -0,0 +1 @@
+Document the version of Synapse each module callback was introduced in.
@@ -0,0 +1 @@
+Add docstrings and comments to the application service ephemeral event sending code.
@@ -14,7 +14,6 @@ services:
    # failure
    restart: unless-stopped
    # See the readme for a full documentation of the environment settings
-    # NOTE: You must edit homeserver.yaml to use postgres, it defaults to sqlite
    environment:
      - SYNAPSE_CONFIG_PATH=/data/homeserver.yaml
    volumes:
@@ -92,6 +92,22 @@ new PromConsole.Graph({
 })
 </script>

+<h3>Pending calls per tick</h3>
+<div id="reactor_pending_calls"></div>
+<script>
+new PromConsole.Graph({
+  node: document.querySelector("#reactor_pending_calls"),
+  expr: "rate(python_twisted_reactor_pending_calls_sum[30s]) / rate(python_twisted_reactor_pending_calls_count[30s])",
+  name: "[[job]]-[[index]]",
+  min: 0,
+  renderer: "line",
+  height: 150,
+  yAxisFormatter: PromConsole.NumberFormatter.humanize,
+  yHoverFormatter: PromConsole.NumberFormatter.humanize,
+  yTitle: "Pending Calls"
+})
+</script>
+
 <h1>Storage</h1>

 <h3>Queries</h3>
@@ -84,9 +84,7 @@ AUTH="Authorization: Bearer $TOKEN"
 ###################################################################################################
 # finally start pruning the room:
 ###################################################################################################
-# this will really delete local events, so the messages in the room really
-# disappear unless they are restored by remote federation. This is because
-# we pass {"delete_local_events":true} to the curl invocation below.
+POSTDATA='{"delete_local_events":"true"}' # this will really delete local events, so the messages in the room really disappear unless they are restored by remote federation

 for ROOM in "${ROOMS_ARRAY[@]}"; do
    echo "########################################### $(date) ################# "
@@ -106,7 +104,7 @@ for ROOM in "${ROOMS_ARRAY[@]}"; do
      SLEEP=2
      set -x
      # call purge
-      OUT=$(curl --header "$AUTH" -s -d '{"delete_local_events":true}' POST "$API_URL/admin/purge_history/$ROOM/$EVENT_ID")
+      OUT=$(curl --header "$AUTH" -s -d $POSTDATA POST "$API_URL/admin/purge_history/$ROOM/$EVENT_ID")
      PURGE_ID=$(echo "$OUT" |grep purge_id|cut -d'"' -f4 )
      if [ "$PURGE_ID" == "" ]; then
        # probably the history purge is already in progress for $ROOM
@@ -15,7 +15,7 @@ export DH_VIRTUALENV_INSTALL_ROOT=/opt/venvs
 # python won't look in the right directory. At least this way, the error will
 # be a *bit* more obvious.
 #
-SNAKE=$(readlink -e /usr/bin/python3)
+SNAKE=`readlink -e /usr/bin/python3`

 # try to set the CFLAGS so any compiled C extensions are compiled with the most
 # generic as possible x64 instructions, so that compiling it on a new Intel chip
@@ -24,7 +24,7 @@ SNAKE=$(readlink -e /usr/bin/python3)
 # TODO: add similar things for non-amd64, or figure out a more generic way to
 # do this.

-case $(dpkg-architecture -q DEB_HOST_ARCH) in
+case `dpkg-architecture -q DEB_HOST_ARCH` in
    amd64)
        export CFLAGS=-march=x86-64
        ;;
@@ -40,7 +40,6 @@ dh_virtualenv \
    --upgrade-pip \
    --preinstall="lxml" \
    --preinstall="mock" \
-    --preinstall="wheel" \
    --extra-pip-arg="--no-cache-dir" \
    --extra-pip-arg="--compile" \
    --extras="all,systemd,test"
@@ -57,8 +56,8 @@ case "$DEB_BUILD_OPTIONS" in
    *)
        # Copy tests to a temporary directory so that we can put them on the
        # PYTHONPATH without putting the uninstalled synapse on the pythonpath.
-        tmpdir=$(mktemp -d)
-        trap 'rm -r $tmpdir' EXIT
+        tmpdir=`mktemp -d`
+        trap "rm -r $tmpdir" EXIT

        cp -r tests "$tmpdir"

@@ -99,7 +98,7 @@ esac
        --output-file="${PACKAGE_BUILD_DIR}/etc/matrix-synapse/log.yaml"

 # add a dependency on the right version of python to substvars.
-PYPKG=$(basename "$SNAKE")
+PYPKG=`basename $SNAKE`
 echo "synapse:pydepends=$PYPKG" >> debian/matrix-synapse-py3.substvars


@@ -1,189 +1,3 @@
-matrix-synapse-py3 (1.55.1) stable; urgency=medium
-
-  * New synapse release 1.55.1.
-
- -- Synapse Packaging team <packages@matrix.org>  Thu, 24 Mar 2022 17:44:23 +0000
-
-matrix-synapse-py3 (1.55.0) stable; urgency=medium
-
-  * New synapse release 1.55.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 22 Mar 2022 13:59:26 +0000
-
-matrix-synapse-py3 (1.55.0~rc1) stable; urgency=medium
-
-  * New synapse release 1.55.0~rc1.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 15 Mar 2022 10:59:31 +0000
-
-matrix-synapse-py3 (1.54.0) stable; urgency=medium
-
-  * New synapse release 1.54.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 08 Mar 2022 10:54:52 +0000
-
-matrix-synapse-py3 (1.54.0~rc1) stable; urgency=medium
-
-  * New synapse release 1.54.0~rc1.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 02 Mar 2022 10:43:22 +0000
-
-matrix-synapse-py3 (1.53.0) stable; urgency=medium
-
-  * New synapse release 1.53.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 22 Feb 2022 11:32:06 +0000
-
-matrix-synapse-py3 (1.53.0~rc1) stable; urgency=medium
-
-  * New synapse release 1.53.0~rc1.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 15 Feb 2022 10:40:50 +0000
-
-matrix-synapse-py3 (1.52.0) stable; urgency=medium
-
-  * New synapse release 1.52.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 08 Feb 2022 11:34:54 +0000
-
-matrix-synapse-py3 (1.52.0~rc1) stable; urgency=medium
-
-  * New synapse release 1.52.0~rc1.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 01 Feb 2022 11:04:09 +0000
-
-matrix-synapse-py3 (1.51.0) stable; urgency=medium
-
-  * New synapse release 1.51.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 25 Jan 2022 11:28:51 +0000
-
-matrix-synapse-py3 (1.51.0~rc2) stable; urgency=medium
-
-  * New synapse release 1.51.0~rc2.
-
- -- Synapse Packaging team <packages@matrix.org>  Mon, 24 Jan 2022 12:25:00 +0000
-
-matrix-synapse-py3 (1.51.0~rc1) stable; urgency=medium
-
-  * New synapse release 1.51.0~rc1.
-
- -- Synapse Packaging team <packages@matrix.org>  Fri, 21 Jan 2022 10:46:02 +0000
-
-matrix-synapse-py3 (1.50.2) stable; urgency=medium
-
-  * New synapse release 1.50.2.
-
- -- Synapse Packaging team <packages@matrix.org>  Mon, 24 Jan 2022 13:37:11 +0000
-
-matrix-synapse-py3 (1.50.1) stable; urgency=medium
-
-  * New synapse release 1.50.1.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 18 Jan 2022 16:06:26 +0000
-
-matrix-synapse-py3 (1.50.0) stable; urgency=medium
-
-  * New synapse release 1.50.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 18 Jan 2022 10:40:38 +0000
-
-matrix-synapse-py3 (1.50.0~rc2) stable; urgency=medium
-
-  * New synapse release 1.50.0~rc2.
-
- -- Synapse Packaging team <packages@matrix.org>  Fri, 14 Jan 2022 11:18:06 +0000
-
-matrix-synapse-py3 (1.50.0~rc1) stable; urgency=medium
-
-  * New synapse release 1.50.0~rc1.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 05 Jan 2022 12:36:17 +0000
-
-matrix-synapse-py3 (1.49.2) stable; urgency=medium
-
-  * New synapse release 1.49.2.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 21 Dec 2021 17:31:03 +0000
-
-matrix-synapse-py3 (1.49.1) stable; urgency=medium
-
-  * New synapse release 1.49.1.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 21 Dec 2021 11:07:30 +0000
-
-matrix-synapse-py3 (1.49.0) stable; urgency=medium
-
-  * New synapse release 1.49.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 14 Dec 2021 12:39:46 +0000
-
-matrix-synapse-py3 (1.49.0~rc1) stable; urgency=medium
-
-  * New synapse release 1.49.0~rc1.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 07 Dec 2021 13:52:21 +0000
-
-matrix-synapse-py3 (1.48.0) stable; urgency=medium
-
-  * New synapse release 1.48.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 30 Nov 2021 11:24:15 +0000
-
-matrix-synapse-py3 (1.48.0~rc1) stable; urgency=medium
-
-  * New synapse release 1.48.0~rc1.
-
- -- Synapse Packaging team <packages@matrix.org>  Thu, 25 Nov 2021 15:56:03 +0000
-
-matrix-synapse-py3 (1.47.1) stable; urgency=medium
-
-  * New synapse release 1.47.1.
-
- -- Synapse Packaging team <packages@matrix.org>  Fri, 19 Nov 2021 13:44:32 +0000
-
-matrix-synapse-py3 (1.47.0) stable; urgency=medium
-
-  * New synapse release 1.47.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 17 Nov 2021 13:09:43 +0000
-
-matrix-synapse-py3 (1.47.0~rc3) stable; urgency=medium
-
-  * New synapse release 1.47.0~rc3.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 16 Nov 2021 14:32:47 +0000
-
-matrix-synapse-py3 (1.47.0~rc2) stable; urgency=medium
-
-  [ Dan Callahan ]
-  * Update scripts to pass Shellcheck lints.
-  * Remove unused Vagrant scripts from debian/ directory.
-  * Allow building Debian packages for any architecture, not just amd64.
-  * Preinstall the "wheel" package when building virtualenvs.
-  * Do not error if /etc/default/matrix-synapse is missing.
-
-  [ Synapse Packaging team ]
-  * New synapse release 1.47.0~rc2.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 10 Nov 2021 09:41:01 +0000
-
-matrix-synapse-py3 (1.46.0) stable; urgency=medium
-
-  [ Richard van der Hoff ]
-  * Compress debs with xz, to fix incompatibility of impish debs with reprepro.
-
-  [ Synapse Packaging team ]
-  * New synapse release 1.46.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 02 Nov 2021 13:22:53 +0000
-
-matrix-synapse-py3 (1.46.0~rc1) stable; urgency=medium
-
-  * New synapse release 1.46.0~rc1.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 26 Oct 2021 14:04:04 +0100
-
 matrix-synapse-py3 (1.45.1) stable; urgency=medium

  * New synapse release 1.45.1.
@@ -19,7 +19,7 @@ Standards-Version: 3.9.8
 Homepage: https://github.com/matrix-org/synapse

 Package: matrix-synapse-py3
-Architecture: any
+Architecture: amd64
 Provides: matrix-synapse
 Conflicts:
 matrix-synapse (<< 0.34.0.1-0matrix2),
@@ -2,7 +2,6 @@

 set -e

-# shellcheck disable=SC1091
 . /usr/share/debconf/confmodule

 # try to update the debconf db according to whatever is in the config files
@@ -1,6 +1,5 @@
 #!/bin/sh -e

-# shellcheck disable=SC1091
 . /usr/share/debconf/confmodule

 CONFIGFILE_SERVERNAME="/etc/matrix-synapse/conf.d/server_name.yaml"
@@ -5,7 +5,7 @@ Description=Synapse Matrix homeserver
 Type=notify
 User=matrix-synapse
 WorkingDirectory=/var/lib/matrix-synapse
-EnvironmentFile=-/etc/default/matrix-synapse
+EnvironmentFile=/etc/default/matrix-synapse
 ExecStartPre=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys
 ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/
 ExecReload=/bin/kill -HUP $MAINPID
@@ -51,11 +51,5 @@ override_dh_shlibdeps:
 override_dh_virtualenv:
 	./debian/build_virtualenv

-override_dh_builddeb:
-        # force the compression to xzip, to stop dpkg-deb on impish defaulting to zstd
-        # (which requires reprepro 5.3.0-1.3, which is currently only in 'experimental' in Debian:
-        # https://metadata.ftp-master.debian.org/changelogs/main/r/reprepro/reprepro_5.3.0-1.3_changelog)
-	dh_builddeb -- -Zxz
-
 %:
 	dh $@ --with python-virtualenv
@@ -0,0 +1,2 @@
+.vagrant
+*.log
@@ -0,0 +1,23 @@
+#!/bin/bash
+#
+# provisioning script for vagrant boxes for testing the matrix-synapse debs.
+#
+# Will install the most recent matrix-synapse-py3 deb for this platform from
+# the /debs directory.
+
+set -e
+
+apt-get update
+apt-get install -y lsb-release
+
+deb=`ls /debs/matrix-synapse-py3_*+$(lsb_release -cs)*.deb | sort | tail -n1`
+
+debconf-set-selections <<EOF
+matrix-synapse matrix-synapse/report-stats boolean false
+matrix-synapse matrix-synapse/server-name string localhost:18448
+EOF
+
+dpkg -i "$deb"
+
+sed -i -e '/port: 8...$/{s/8448/18448/; s/8008/18008/}' -e '$aregistration_shared_secret: secret' /etc/matrix-synapse/homeserver.yaml
+systemctl restart matrix-synapse
@@ -0,0 +1,13 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+ver = `cd ../../..; dpkg-parsechangelog -S Version`.strip()
+
+Vagrant.configure("2") do |config|
+  config.vm.box = "debian/stretch64"
+
+  config.vm.synced_folder ".", "/vagrant", disabled: true
+  config.vm.synced_folder "../../../../debs", "/debs", type: "nfs"
+
+  config.vm.provision "shell", path: "../provision.sh"
+end
@@ -0,0 +1,10 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+Vagrant.configure("2") do |config|
+  config.vm.box = "ubuntu/xenial64"
+
+  config.vm.synced_folder ".", "/vagrant", disabled: true
+  config.vm.synced_folder "../../../../debs", "/debs"
+  config.vm.provision "shell", path: "../provision.sh"
+end
@@ -1,4 +1,7 @@
-# Ignore all the temporary files from the demo servers.
-8080/
-8081/
-8082/
+*.db
+*.log
+*.log.*
+*.pid
+
+/media_store.*
+/etc
@@ -0,0 +1,26 @@
+DO NOT USE THESE DEMO SERVERS IN PRODUCTION
+
+Requires you to have done:
+    python setup.py develop
+
+
+The demo start.sh will start three synapse servers on ports 8080, 8081 and 8082, with host names localhost:$port. This can be easily changed to `hostname`:$port in start.sh if required.
+
+To enable the servers to communicate untrusted ssl certs are used. In order to do this the servers do not check the certs
+and are configured in a highly insecure way. Do not use these configuration files in production.
+
+stop.sh will stop the synapse servers and the webclient.
+
+clean.sh will delete the databases and log files.
+
+To start a completely new set of servers, run:
+
+    ./demo/stop.sh; ./demo/clean.sh && ./demo/start.sh
+
+
+Logs and sqlitedb will be stored in demo/808{0,1,2}.{log,db}
+
+
+
+Also note that when joining a public room on a differnt HS via "#foo:bar.net", then you are (in the current impl) joining a room with room_id "foo". This means that it won't work if your HS already has a room with that name.
+
@@ -4,19 +4,16 @@ set -e

 DIR="$( cd "$( dirname "$0" )" && pwd )"

-# Ensure that the servers are stopped.
-$DIR/stop.sh
-
 PID_FILE="$DIR/servers.pid"

-if [ -f "$PID_FILE" ]; then
+if [ -f $PID_FILE ]; then
    echo "servers.pid exists!"
    exit 1
 fi

 for port in 8080 8081 8082; do
-    rm -rf "${DIR:?}/$port"
-    rm -rf "$DIR/media_store.$port"
+    rm -rf $DIR/$port
+    rm -rf $DIR/media_store.$port
 done

-rm -rf "${DIR:?}/etc"
+rm -rf $DIR/etc
@@ -4,100 +4,98 @@ DIR="$( cd "$( dirname "$0" )" && pwd )"

 CWD=$(pwd)

-cd "$DIR/.." || exit
+cd "$DIR/.."

-PYTHONPATH=$(readlink -f "$(pwd)")
-export PYTHONPATH
+mkdir -p demo/etc
+
+export PYTHONPATH=$(readlink -f $(pwd))


-echo "$PYTHONPATH"
+echo $PYTHONPATH

 for port in 8080 8081 8082; do
    echo "Starting server on port $port... "

    https_port=$((port + 400))
    mkdir -p demo/$port
-    pushd demo/$port || exit
+    pushd demo/$port

-    # Generate the configuration for the homeserver at localhost:848x.
+    #rm $DIR/etc/$port.config
    python3 -m synapse.app.homeserver \
        --generate-config \
-        --server-name "localhost:$port" \
-        --config-path "$port.config" \
+        -H "localhost:$https_port" \
+        --config-path "$DIR/etc/$port.config" \
        --report-stats no

-    if ! grep -F "Customisation made by demo/start.sh" -q "$port.config"; then
-        # Generate TLS keys.
-        openssl req -x509 -newkey rsa:4096 \
-          -keyout "localhost:$port.tls.key" \
-          -out "localhost:$port.tls.crt" \
-          -days 365 -nodes -subj "/O=matrix"
+    if ! grep -F "Customisation made by demo/start.sh" -q  $DIR/etc/$port.config; then
+        printf '\n\n# Customisation made by demo/start.sh\n' >> $DIR/etc/$port.config

-        # Add customisations to the configuration.
-        {
-            printf '\n\n# Customisation made by demo/start.sh\n\n'
-            echo "public_baseurl: http://localhost:$port/"
-            echo 'enable_registration: true'
-            echo ''
+        echo "public_baseurl: http://localhost:$port/" >> $DIR/etc/$port.config

-			# Warning, this heredoc depends on the interaction of tabs and spaces.
-			# Please don't accidentaly bork me with your fancy settings.
-			listeners=$(cat <<-PORTLISTENERS
-			# Configure server to listen on both $https_port and $port
-			# This overides some of the default settings above
-			listeners:
-			  - port: $https_port
-			    type: http
-			    tls: true
-			    resources:
-			      - names: [client, federation]
+        echo 'enable_registration: true' >> $DIR/etc/$port.config

-			  - port: $port
-			    tls: false
-			    bind_addresses: ['::1', '127.0.0.1']
-			    type: http
-			    x_forwarded: true
-			    resources:
-			      - names: [client, federation]
-			        compress: false
-			PORTLISTENERS
-			)
+        # Warning, this heredoc depends on the interaction of tabs and spaces. Please don't
+        # accidentaly bork me with your fancy settings.
+		listeners=$(cat <<-PORTLISTENERS
+		# Configure server to listen on both $https_port and $port
+		# This overides some of the default settings above
+		listeners:
+		  - port: $https_port
+		    type: http
+		    tls: true
+		    resources:
+		      - names: [client, federation]

-            echo "${listeners}"
+		  - port: $port
+		    tls: false
+		    bind_addresses: ['::1', '127.0.0.1']
+		    type: http
+		    x_forwarded: true
+		    resources:
+		      - names: [client, federation]
+		        compress: false
+		PORTLISTENERS
+		)
+        echo "${listeners}" >> $DIR/etc/$port.config

-            # Disable TLS for the servers
-            printf '\n\n# Disable TLS for the servers.'
-            echo '# DO NOT USE IN PRODUCTION'
-            echo 'use_insecure_ssl_client_just_for_testing_do_not_use: true'
-            echo 'federation_verify_certificates: false'
+        # Disable tls for the servers
+        printf '\n\n# Disable tls on the servers.' >> $DIR/etc/$port.config
+        echo '# DO NOT USE IN PRODUCTION' >> $DIR/etc/$port.config
+        echo 'use_insecure_ssl_client_just_for_testing_do_not_use: true' >> $DIR/etc/$port.config
+        echo 'federation_verify_certificates: false' >> $DIR/etc/$port.config

-            # Set paths for the TLS certificates.
-            echo "tls_certificate_path: \"$DIR/$port/localhost:$port.tls.crt\""
-            echo "tls_private_key_path: \"$DIR/$port/localhost:$port.tls.key\""
+        # Set tls paths
+        echo "tls_certificate_path: \"$DIR/etc/localhost:$https_port.tls.crt\"" >> $DIR/etc/$port.config
+        echo "tls_private_key_path: \"$DIR/etc/localhost:$https_port.tls.key\"" >> $DIR/etc/$port.config

-            # Ignore keys from the trusted keys server
-            echo '# Ignore keys from the trusted keys server'
-            echo 'trusted_key_servers:'
-            echo '  - server_name: "matrix.org"'
-            echo '    accept_keys_insecurely: true'
-            echo ''
+        # Generate tls keys
+        openssl req -x509 -newkey rsa:4096 -keyout $DIR/etc/localhost\:$https_port.tls.key -out $DIR/etc/localhost\:$https_port.tls.crt -days 365 -nodes -subj "/O=matrix"

-			# Allow the servers to communicate over localhost.
-			allow_list=$(cat <<-ALLOW_LIST
-			# Allow the servers to communicate over localhost.
-			ip_range_whitelist:
-			  - '127.0.0.1/8'
-			  - '::1/128'
-			ALLOW_LIST
-			)
+        # Ignore keys from the trusted keys server
+        echo '# Ignore keys from the trusted keys server' >> $DIR/etc/$port.config
+        echo 'trusted_key_servers:' >> $DIR/etc/$port.config
+        echo '  - server_name: "matrix.org"' >> $DIR/etc/$port.config
+        echo '    accept_keys_insecurely: true' >> $DIR/etc/$port.config

-            echo "${allow_list}"
-        } >> "$port.config"
+        # Reduce the blacklist
+        blacklist=$(cat <<-BLACK
+		# Set the blacklist so that it doesn't include 127.0.0.1, ::1
+		federation_ip_range_blacklist:
+		  - '10.0.0.0/8'
+		  - '172.16.0.0/12'
+		  - '192.168.0.0/16'
+		  - '100.64.0.0/10'
+		  - '169.254.0.0/16'
+		  - 'fe80::/64'
+		  - 'fc00::/7'
+		BLACK
+		)
+        echo "${blacklist}" >> $DIR/etc/$port.config
    fi

    # Check script parameters
    if [ $# -eq 1 ]; then
-        if [ "$1" = "--no-rate-limit" ]; then
+        if [ $1 = "--no-rate-limit" ]; then

            # Disable any rate limiting
            ratelimiting=$(cat <<-RC
@@ -139,21 +137,22 @@ for port in 8080 8081 8082; do
 			    burst_count: 1000
 			RC
 			)
-            echo "${ratelimiting}" >> "$port.config"
+            echo "${ratelimiting}" >> $DIR/etc/$port.config
        fi
    fi

-    # Always disable reporting of stats if the option is not there.
-    if ! grep -F "report_stats" -q  "$port.config" ; then
-        echo "report_stats: false" >> "$port.config"
+    if ! grep -F "full_twisted_stacktraces" -q  $DIR/etc/$port.config; then
+        echo "full_twisted_stacktraces: true" >> $DIR/etc/$port.config
+    fi
+    if ! grep -F "report_stats" -q  $DIR/etc/$port.config ; then
+        echo "report_stats: false" >> $DIR/etc/$port.config
    fi

-    # Run the homeserver in the background.
    python3 -m synapse.app.homeserver \
-        --config-path "$port.config" \
+        --config-path "$DIR/etc/$port.config" \
        -D \

-    popd || exit
+    popd
 done

-cd "$CWD" || exit
+cd "$CWD"
@@ -8,7 +8,7 @@ for pid_file in $FILES; do
    pid=$(cat "$pid_file")
    if [[ $pid ]]; then
        echo "Killing $pid_file with $pid"
-        kill "$pid"
+        kill $pid
    fi
 done

@@ -1,20 +1,17 @@
 # Dockerfile to build the matrixdotorg/synapse docker images.
 #
-# Note that it uses features which are only available in BuildKit - see
-# https://docs.docker.com/go/buildkit/ for more information.
-#
 # To build the image, run `docker build` command from the root of the
 # synapse repository:
 #
-#    DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile .
+#    docker build -f docker/Dockerfile .
 #
 # There is an optional PYTHON_VERSION build argument which sets the
 # version of python to build against: for example:
 #
-#    DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile --build-arg PYTHON_VERSION=3.10 .
+#    docker build -f docker/Dockerfile --build-arg PYTHON_VERSION=3.6 .
 #

-ARG PYTHON_VERSION=3.9
+ARG PYTHON_VERSION=3.8

 ###
 ### Stage 0: builder
@@ -22,16 +19,7 @@ ARG PYTHON_VERSION=3.9
 FROM docker.io/python:${PYTHON_VERSION}-slim as builder

 # install the OS build deps
-#
-# RUN --mount is specific to buildkit and is documented at
-# https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md#build-mounts-run---mount.
-# Here we use it to set up a cache for apt, to improve rebuild speeds on
-# slow connections.
-#
-RUN \
-   --mount=type=cache,target=/var/cache/apt,sharing=locked \
-   --mount=type=cache,target=/var/lib/apt,sharing=locked \
- apt-get update && apt-get install -y \
+RUN apt-get update && apt-get install -y \
    build-essential \
    libffi-dev \
    libjpeg-dev \
@@ -46,7 +34,8 @@ RUN \
    && rm -rf /var/lib/apt/lists/*

 # Copy just what we need to pip install
-COPY MANIFEST.in README.rst setup.py /synapse/
+COPY scripts /synapse/scripts/
+COPY MANIFEST.in README.rst setup.py synctl /synapse/
 COPY synapse/__init__.py /synapse/synapse/__init__.py
 COPY synapse/python_dependencies.py /synapse/synapse/python_dependencies.py

@@ -55,8 +44,7 @@ COPY synapse/python_dependencies.py /synapse/synapse/python_dependencies.py
 # used while you develop on the source
 #
 # This is aiming at installing the `install_requires` and `extras_require` from `setup.py`
-RUN --mount=type=cache,target=/root/.cache/pip \
-  pip install --prefix="/install" --no-warn-script-location \
+RUN pip install --prefix="/install" --no-warn-script-location \
    /synapse[all]

 # Copy over the rest of the project
@@ -78,10 +66,7 @@ LABEL org.opencontainers.image.documentation='https://github.com/matrix-org/syna
 LABEL org.opencontainers.image.source='https://github.com/matrix-org/synapse.git'
 LABEL org.opencontainers.image.licenses='Apache-2.0'

-RUN \
-   --mount=type=cache,target=/var/cache/apt,sharing=locked \
-   --mount=type=cache,target=/var/lib/apt,sharing=locked \
-  apt-get update && apt-get install -y \
+RUN apt-get update && apt-get install -y \
    curl \
    gosu \
    libjpeg62-turbo \
@@ -97,6 +82,8 @@ COPY --from=builder /install /usr/local
 COPY ./docker/start.py /start.py
 COPY ./docker/conf /conf

+VOLUME ["/data"]
+
 EXPOSE 8008/tcp 8009/tcp 8448/tcp

 ENTRYPOINT ["/start.py"]
@@ -16,7 +16,7 @@ ARG distro=""
 ### Stage 0: build a dh-virtualenv
 ###

-# This is only really needed on focal, since other distributions we
+# This is only really needed on bionic and focal, since other distributions we
 # care about have a recent version of dh-virtualenv by default. Unfortunately,
 # it looks like focal is going to be with us for a while.
 #
@@ -36,8 +36,9 @@ RUN env DEBIAN_FRONTEND=noninteractive apt-get install \
        wget

 # fetch and unpack the package
+# TODO: Upgrade to 1.2.2 once bionic is dropped (1.2.2 requires debhelper 12; bionic has only 11)
 RUN mkdir /dh-virtualenv
-RUN wget -q -O /dh-virtualenv.tar.gz https://github.com/spotify/dh-virtualenv/archive/refs/tags/1.2.2.tar.gz
+RUN wget -q -O /dh-virtualenv.tar.gz https://github.com/spotify/dh-virtualenv/archive/ac6e1b1.tar.gz
 RUN tar -xv --strip-components=1 -C /dh-virtualenv -f /dh-virtualenv.tar.gz

 # install its build deps. We do another apt-cache-update here, because we might
@@ -85,12 +86,12 @@ RUN apt-get update -qq -o Acquire::Languages=none \
        libpq-dev \
        xmlsec1

-COPY --from=builder /dh-virtualenv_1.2.2-1_all.deb /
+COPY --from=builder /dh-virtualenv_1.2~dev-1_all.deb /

 # install dhvirtualenv. Update the apt cache again first, in case we got a
 # cached cache from docker the first time.
 RUN apt-get update -qq -o Acquire::Languages=none \
-    && apt-get install -yq /dh-virtualenv_1.2.2-1_all.deb
+    && apt-get install -yq /dh-virtualenv_1.2~dev-1_all.deb

 WORKDIR /synapse/source
 ENTRYPOINT ["bash","/synapse/source/docker/build_debian.sh"]
@@ -1,6 +1,6 @@
 # Use the Sytest image that comes with a lot of the build dependencies
 # pre-installed
-FROM matrixdotorg/sytest:focal
+FROM matrixdotorg/sytest:bionic

 # The Sytest image doesn't come with python, so install that
 RUN apt-get update && apt-get -qq install -y python3 python3-dev python3-pip
@@ -21,6 +21,3 @@ VOLUME ["/data"]
 # files to run the desired worker configuration. Will start supervisord.
 COPY ./docker/configure_workers_and_start.py /configure_workers_and_start.py
 ENTRYPOINT ["/configure_workers_and_start.py"]
-
-HEALTHCHECK --start-period=5s --interval=15s --timeout=5s \
-    CMD /bin/sh /healthcheck.sh
@@ -65,12 +65,7 @@ The following environment variables are supported in `generate` mode:
 * `SYNAPSE_DATA_DIR`: where the generated config will put persistent data
  such as the database and media store. Defaults to `/data`.
 * `UID`, `GID`: the user id and group id to use for creating the data
-  directories. If unset, and no user is set via `docker run --user`, defaults
-  to `991`, `991`.
-
-## Postgres
-
-By default the config will use SQLite. See the [docs on using Postgres](https://github.com/matrix-org/synapse/blob/develop/docs/postgres.md) for more info on how to use Postgres. Until this section is improved [this issue](https://github.com/matrix-org/synapse/issues/8304) may provide useful information.
+  directories. Defaults to `991`, `991`.

 ## Running synapse

@@ -102,9 +97,7 @@ The following environment variables are supported in `run` mode:
  `<SYNAPSE_CONFIG_DIR>/homeserver.yaml`.
 * `SYNAPSE_WORKER`: module to execute, used when running synapse with workers.
   Defaults to `synapse.app.homeserver`, which is suitable for non-worker mode.
-* `UID`, `GID`: the user and group id to run Synapse as. If unset, and no user
-  is set via `docker run --user`, defaults to `991`, `991`. Note that this user
-  must have permission to read the config files, and write to the data directories.
+* `UID`, `GID`: the user and group id to run Synapse as. Defaults to `991`, `991`.
 * `TZ`: the [timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) the container will run with. Defaults to `UTC`.

 For more complex setups (e.g. for workers) you can also pass your args directly to synapse using `run` mode. For example like this:
@@ -193,7 +186,7 @@ point to another Dockerfile.
 ## Disabling the healthcheck

 If you are using a non-standard port or tls inside docker you can disable the healthcheck
-whilst running the above `docker run` commands.
+whilst running the above `docker run` commands. 

 ```
   --no-healthcheck
@@ -219,7 +212,7 @@ If you wish to point the healthcheck at a different port with docker command, ad
 ## Setting the healthcheck in docker-compose file

 You can add the following to set a custom healthcheck in a docker compose file.
-You will need docker-compose version >2.1 for this to work.
+You will need docker-compose version >2.1 for this to work. 

 ```
 healthcheck:
@@ -233,5 +226,4 @@ healthcheck:
 ## Using jemalloc

 Jemalloc is embedded in the image and will be used instead of the default allocator.
-You can read about jemalloc by reading the Synapse
-[README](https://github.com/matrix-org/synapse/blob/HEAD/README.rst#help-synapse-is-slow-and-eats-all-my-ram-cpu).
+You can read about jemalloc by reading the Synapse [README](../README.rst).
@@ -5,7 +5,7 @@
 set -ex

 # Get the codename from distro env
-DIST=$(cut -d ':' -f2 <<< "${distro:?}")
+DIST=`cut -d ':' -f2 <<< $distro`

 # we get a read-only copy of the source: make a writeable copy
 cp -aT /synapse/source /synapse/build
@@ -17,7 +17,7 @@ cd /synapse/build
 # Section to determine which "component" it should go into (see
 # https://manpages.debian.org/stretch/reprepro/reprepro.1.en.html#GUESSING)

-DEB_VERSION=$(dpkg-parsechangelog -SVersion)
+DEB_VERSION=`dpkg-parsechangelog -SVersion`
 case $DEB_VERSION in
    *~rc*|*~a*|*~b*|*~c*)
        sed -ie '/^Section:/c\Section: prerelease' debian/control
@@ -1,6 +0,0 @@
-#!/bin/sh
-# This healthcheck script is designed to return OK when every 
-# host involved returns OK
-{%- for healthcheck_url in healthcheck_urls %}
-curl -fSs {{ healthcheck_url }} || exit 1
-{%- endfor %}
@@ -148,6 +148,14 @@ bcrypt_rounds: 12
 allow_guest_access: {{ "True" if SYNAPSE_ALLOW_GUEST else "False" }}
 enable_group_creation: true

+# The list of identity servers trusted to verify third party
+# identifiers by this server.
+#
+# Also defines the ID server which will be called when an account is
+# deactivated (one will be picked arbitrarily).
+trusted_third_party_id_servers:
+    - matrix.org
+    - vector.im

 ## Metrics ###

@@ -48,7 +48,7 @@ WORKERS_CONFIG = {
        "app": "synapse.app.user_dir",
        "listener_resources": ["client"],
        "endpoint_patterns": [
-            "^/_matrix/client/(api/v1|r0|v3|unstable)/user_directory/search$"
+            "^/_matrix/client/(api/v1|r0|unstable)/user_directory/search$"
        ],
        "shared_extra_conf": {"update_user_directory": False},
        "worker_extra_conf": "",
@@ -85,10 +85,10 @@ WORKERS_CONFIG = {
        "app": "synapse.app.generic_worker",
        "listener_resources": ["client"],
        "endpoint_patterns": [
-            "^/_matrix/client/(v2_alpha|r0|v3)/sync$",
-            "^/_matrix/client/(api/v1|v2_alpha|r0|v3)/events$",
-            "^/_matrix/client/(api/v1|r0|v3)/initialSync$",
-            "^/_matrix/client/(api/v1|r0|v3)/rooms/[^/]+/initialSync$",
+            "^/_matrix/client/(v2_alpha|r0)/sync$",
+            "^/_matrix/client/(api/v1|v2_alpha|r0)/events$",
+            "^/_matrix/client/(api/v1|r0)/initialSync$",
+            "^/_matrix/client/(api/v1|r0)/rooms/[^/]+/initialSync$",
        ],
        "shared_extra_conf": {},
        "worker_extra_conf": "",
@@ -146,11 +146,11 @@ WORKERS_CONFIG = {
        "app": "synapse.app.generic_worker",
        "listener_resources": ["client"],
        "endpoint_patterns": [
-            "^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/redact",
-            "^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/send",
-            "^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$",
-            "^/_matrix/client/(api/v1|r0|v3|unstable)/join/",
-            "^/_matrix/client/(api/v1|r0|v3|unstable)/profile/",
+            "^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/redact",
+            "^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/send",
+            "^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$",
+            "^/_matrix/client/(api/v1|r0|unstable)/join/",
+            "^/_matrix/client/(api/v1|r0|unstable)/profile/",
        ],
        "shared_extra_conf": {},
        "worker_extra_conf": "",
@@ -158,7 +158,7 @@ WORKERS_CONFIG = {
    "frontend_proxy": {
        "app": "synapse.app.frontend_proxy",
        "listener_resources": ["client", "replication"],
-        "endpoint_patterns": ["^/_matrix/client/(api/v1|r0|v3|unstable)/keys/upload"],
+        "endpoint_patterns": ["^/_matrix/client/(api/v1|r0|unstable)/keys/upload"],
        "shared_extra_conf": {},
        "worker_extra_conf": (
            "worker_main_http_uri: http://127.0.0.1:%d"
@@ -474,16 +474,10 @@ def generate_worker_files(environ, config_path: str, data_dir: str):

    # Determine the load-balancing upstreams to configure
    nginx_upstream_config = ""
-
-    # At the same time, prepare a list of internal endpoints to healthcheck
-    # starting with the main process which exists even if no workers do.
-    healthcheck_urls = ["http://localhost:8080/health"]
-
    for upstream_worker_type, upstream_worker_ports in nginx_upstreams.items():
        body = ""
        for port in upstream_worker_ports:
            body += "    server localhost:%d;\n" % (port,)
-            healthcheck_urls.append("http://localhost:%d/health" % (port,))

        # Add to the list of configured upstreams
        nginx_upstream_config += NGINX_UPSTREAM_CONFIG_BLOCK.format(
@@ -516,13 +510,6 @@ def generate_worker_files(environ, config_path: str, data_dir: str):
        worker_config=supervisord_config,
    )

-    # healthcheck config
-    convert(
-        "/conf/healthcheck.sh.j2",
-        "/healthcheck.sh",
-        healthcheck_urls=healthcheck_urls,
-    )
-
    # Ensure the logging directory exists
    log_dir = data_dir + "/logs"
    if not os.path.exists(log_dir):
@@ -16,4 +16,4 @@ sudo -u postgres /usr/lib/postgresql/10/bin/pg_ctl -w -D /var/lib/postgresql/dat
 # Run the tests
 cd /src
 export TRIAL_FLAGS="-j 4"
-tox --workdir=./.tox-pg-container -e py37-postgres "$@"
+tox --workdir=./.tox-pg-container -e py36-postgres "$@"
@@ -120,7 +120,6 @@ def generate_config_from_template(config_dir, config_path, environ, ownership):
    ]

    if ownership is not None:
-        log(f"Setting ownership on /data to {ownership}")
        subprocess.check_output(["chown", "-R", ownership, "/data"])
        args = ["gosu", ownership] + args

@@ -145,18 +144,12 @@ def run_generate_config(environ, ownership):
    config_path = environ.get("SYNAPSE_CONFIG_PATH", config_dir + "/homeserver.yaml")
    data_dir = environ.get("SYNAPSE_DATA_DIR", "/data")

-    if ownership is not None:
-        # make sure that synapse has perms to write to the data dir.
-        log(f"Setting ownership on {data_dir} to {ownership}")
-        subprocess.check_output(["chown", ownership, data_dir])
-
    # create a suitable log config from our template
    log_config_file = "%s/%s.log.config" % (config_dir, server_name)
    if not os.path.exists(log_config_file):
        log("Creating log config %s" % (log_config_file,))
        convert("/conf/log.config", log_config_file, environ)

-    # generate the main config file, and a signing key.
    args = [
        "python",
        "-m",
@@ -175,23 +168,29 @@ def run_generate_config(environ, ownership):
        "--open-private-ports",
    ]
    # log("running %s" % (args, ))
-    os.execv("/usr/local/bin/python", args)
+
+    if ownership is not None:
+        # make sure that synapse has perms to write to the data dir.
+        subprocess.check_output(["chown", ownership, data_dir])
+
+        args = ["gosu", ownership] + args
+        os.execv("/usr/sbin/gosu", args)
+    else:
+        os.execv("/usr/local/bin/python", args)


 def main(args, environ):
    mode = args[1] if len(args) > 1 else "run"
-
-    # if we were given an explicit user to switch to, do so
-    ownership = None
-    if "UID" in environ:
-        desired_uid = int(environ["UID"])
-        desired_gid = int(environ.get("GID", "991"))
-        ownership = f"{desired_uid}:{desired_gid}"
-    elif os.getuid() == 0:
-        # otherwise, if we are running as root, use user 991
-        ownership = "991:991"
-
+    desired_uid = int(environ.get("UID", "991"))
+    desired_gid = int(environ.get("GID", "991"))
    synapse_worker = environ.get("SYNAPSE_WORKER", "synapse.app.homeserver")
+    if (desired_uid == os.getuid()) and (desired_gid == os.getgid()):
+        ownership = None
+    else:
+        ownership = "{}:{}".format(desired_uid, desired_gid)
+
+    if ownership is None:
+        log("Will not perform chmod/gosu as UserID already matches request")

    # In generate mode, generate a configuration and missing keys, then exit
    if mode == "generate":
@@ -15,12 +15,12 @@ in `homeserver.yaml`, to the list of authorized domains. If you have not set
 1. Agree to the terms of service and submit.
 1. Copy your site key and secret key and add them to your `homeserver.yaml`
 configuration file
-    ```yaml
+    ```
    recaptcha_public_key: YOUR_SITE_KEY
    recaptcha_private_key: YOUR_SECRET_KEY
    ```
 1. Enable the CAPTCHA for new registrations
-    ```yaml
+    ```
    enable_registration_captcha: true
    ```
 1. Go to the settings page for the CAPTCHA you just created
@@ -0,0 +1,335 @@
+# MSC1711 Certificates FAQ
+
+## Historical Note
+This document was originally written to guide server admins through the upgrade
+path towards Synapse 1.0. Specifically,
+[MSC1711](https://github.com/matrix-org/matrix-doc/blob/main/proposals/1711-x509-for-federation.md)
+required that all servers present valid TLS certificates on their federation
+API. Admins were encouraged to achieve compliance from version 0.99.0 (released
+in February 2019) ahead of version 1.0 (released June 2019) enforcing the
+certificate checks.
+
+Much of what follows is now outdated since most admins will have already
+upgraded, however it may be of use to those with old installs returning to the
+project.
+
+If you are setting up a server from scratch you almost certainly should look at
+the [installation guide](setup/installation.md) instead.
+
+## Introduction
+The goal of Synapse 0.99.0 is to act as a stepping stone to Synapse 1.0.0. It
+supports the r0.1 release of the server to server specification, but is
+compatible with both the legacy Matrix federation behaviour (pre-r0.1) as well
+as post-r0.1 behaviour, in order to allow for a smooth upgrade across the
+federation.
+
+The most important thing to know is that Synapse 1.0.0 will require a valid TLS
+certificate on federation endpoints. Self signed certificates will not be
+sufficient.
+
+Synapse 0.99.0 makes it easy to configure TLS certificates and will
+interoperate with both >= 1.0.0 servers as well as existing servers yet to
+upgrade.
+
+**It is critical that all admins upgrade to 0.99.0 and configure a valid TLS
+certificate.** Admins will have 1 month to do so, after which 1.0.0 will be
+released and those servers without a valid certificate will not longer be able
+to federate with >= 1.0.0 servers.
+
+Full details on how to carry out this configuration change is given
+[below](#configuring-certificates-for-compatibility-with-synapse-100). A
+timeline and some frequently asked questions are also given below.
+
+For more details and context on the release of the r0.1 Server/Server API and
+imminent Matrix 1.0 release, you can also see our
+[main talk from FOSDEM 2019](https://matrix.org/blog/2019/02/04/matrix-at-fosdem-2019/).
+
+## Contents
+* Timeline
+* Configuring certificates for compatibility with Synapse 1.0
+* FAQ
+  * Synapse 0.99.0 has just been released, what do I need to do right now?
+  * How do I upgrade?
+  * What will happen if I do not set up a valid federation certificate
+    immediately?
+  * What will happen if I do nothing at all?
+  * When do I need a SRV record or .well-known URI?
+  * Can I still use an SRV record?
+  * I have created a .well-known URI. Do I still need an SRV record?
+  * It used to work just fine, why are you breaking everything?
+  * Can I manage my own certificates rather than having Synapse renew
+    certificates itself?
+  * Do you still recommend against using a reverse proxy on the federation port?
+  * Do I still need to give my TLS certificates to Synapse if I am using a
+    reverse proxy?
+  * Do I need the same certificate for the client and federation port?
+  * How do I tell Synapse to reload my keys/certificates after I replace them?
+
+## Timeline
+
+**5th Feb 2019  - Synapse 0.99.0 is released.**
+
+All server admins are encouraged to upgrade.
+
+0.99.0:
+
+-   provides support for ACME to make setting up Let's Encrypt certs easy, as
+    well as .well-known support.
+
+-   does not enforce that a valid CA cert is present on the federation API, but
+    rather makes it easy to set one up.
+
+-   provides support for .well-known
+
+Admins should upgrade and configure a valid CA cert. Homeservers that require a
+.well-known entry (see below), should retain their SRV record and use it
+alongside their .well-known record.
+
+**10th June 2019  - Synapse 1.0.0 is released**
+
+1.0.0 is scheduled for release on 10th June. In
+accordance with the the [S2S spec](https://matrix.org/docs/spec/server_server/r0.1.0.html)
+1.0.0 will enforce certificate validity. This means that any homeserver without a
+valid certificate after this point will no longer be able to federate with
+1.0.0 servers.
+
+## Configuring certificates for compatibility with Synapse 1.0.0
+
+### If you do not currently have an SRV record
+
+In this case, your `server_name` points to the host where your Synapse is
+running. There is no need to create a `.well-known` URI or an SRV record, but
+you will need to give Synapse a valid, signed, certificate.
+
+### If you do have an SRV record currently
+
+If you are using an SRV record, your matrix domain (`server_name`) may not
+point to the same host that your Synapse is running on (the 'target
+domain'). (If it does, you can follow the recommendation above; otherwise, read
+on.)
+
+Let's assume that your `server_name` is `example.com`, and your Synapse is
+hosted at a target domain of `customer.example.net`. Currently you should have
+an SRV record which looks like:
+
+```
+_matrix._tcp.example.com. IN SRV 10 5 8000 customer.example.net.
+```
+
+In this situation, you have three choices for how to proceed:
+
+#### Option 1: give Synapse a certificate for your matrix domain
+
+Synapse 1.0 will expect your server to present a TLS certificate for your
+`server_name` (`example.com` in the above example). You can achieve this by acquiring a
+certificate for the `server_name` yourself (for example, using `certbot`), and giving it
+and the key to Synapse via `tls_certificate_path` and `tls_private_key_path`.
+
+#### Option 2: run Synapse behind a reverse proxy
+
+If you have an existing reverse proxy set up with correct TLS certificates for
+your domain, you can simply route all traffic through the reverse proxy by
+updating the SRV record appropriately (or removing it, if the proxy listens on
+8448).
+
+See [the reverse proxy documentation](reverse_proxy.md) for information on setting up a
+reverse proxy.
+
+#### Option 3: add a .well-known file to delegate your matrix traffic
+
+This will allow you to keep Synapse on a separate domain, without having to
+give it a certificate for the matrix domain.
+
+You can do this with a `.well-known` file as follows:
+
+ 1. Keep the SRV record in place - it is needed for backwards compatibility
+    with Synapse 0.34 and earlier.
+
+  2. Give Synapse a certificate corresponding to the target domain
+    (`customer.example.net` in the above example). You can do this by acquire a 
+    certificate for the target domain and giving it to Synapse via `tls_certificate_path`
+    and `tls_private_key_path`.
+
+ 3. Restart Synapse to ensure the new certificate is loaded.
+
+ 4. Arrange for a `.well-known` file at
+    `https://<server_name>/.well-known/matrix/server` with contents:
+
+    ```json
+    {"m.server": "<target server name>"}
+    ```
+
+    where the target server name is resolved as usual (i.e. SRV lookup, falling
+    back to talking to port 8448).
+
+    In the above example, where synapse is listening on port 8000,
+    `https://example.com/.well-known/matrix/server` should have `m.server` set to one of:
+
+    1. `customer.example.net` ─ with a SRV record on
+       `_matrix._tcp.customer.example.com` pointing to port 8000, or:
+
+    2. `customer.example.net` ─ updating synapse to listen on the default port
+       8448, or:
+
+    3. `customer.example.net:8000` ─ ensuring that if there is a reverse proxy
+       on `customer.example.net:8000` it correctly handles HTTP requests with
+       Host header set to `customer.example.net:8000`.
+
+## FAQ
+
+### Synapse 0.99.0 has just been released, what do I need to do right now?
+
+Upgrade as soon as you can in preparation for Synapse 1.0.0, and update your
+TLS certificates as [above](#configuring-certificates-for-compatibility-with-synapse-100).
+
+### What will happen if I do not set up a valid federation certificate immediately?
+
+Nothing initially, but once 1.0.0 is in the wild it will not be possible to
+federate with 1.0.0 servers.
+
+### What will happen if I do nothing at all?
+
+If the admin takes no action at all, and remains on a Synapse < 0.99.0 then the
+homeserver will be unable to federate with those who have implemented
+.well-known. Then, as above, once the month upgrade window has expired the
+homeserver will not be able to federate with any Synapse >= 1.0.0
+
+### When do I need a SRV record or .well-known URI?
+
+If your homeserver listens on the default federation port (8448), and your
+`server_name` points to the host that your homeserver runs on, you do not need an
+SRV record or `.well-known/matrix/server` URI.
+
+For instance, if you registered `example.com` and pointed its DNS A record at a
+fresh Upcloud VPS or similar, you could install Synapse 0.99 on that host,
+giving it a server_name of `example.com`, and it would automatically generate a
+valid TLS certificate for you via Let's Encrypt and no SRV record or
+`.well-known` URI would be needed.
+
+This is the common case, although you can add an SRV record or
+`.well-known/matrix/server` URI for completeness if you wish.
+
+**However**, if your server does not listen on port 8448, or if your `server_name`
+does not point to the host that your homeserver runs on, you will need to let
+other servers know how to find it.
+
+In this case, you should see ["If you do have an SRV record
+currently"](#if-you-do-have-an-srv-record-currently) above.
+
+### Can I still use an SRV record?
+
+Firstly, if you didn't need an SRV record before (because your server is
+listening on port 8448 of your server_name), you certainly don't need one now:
+the defaults are still the same.
+
+If you previously had an SRV record, you can keep using it provided you are
+able to give Synapse a TLS certificate corresponding to your server name. For
+example, suppose you had the following SRV record, which directs matrix traffic
+for example.com to matrix.example.com:443:
+
+```
+_matrix._tcp.example.com. IN SRV 10 5 443 matrix.example.com
+```
+
+In this case, Synapse must be given a certificate for example.com - or be
+configured to acquire one from Let's Encrypt.
+
+If you are unable to give Synapse a certificate for your server_name, you will
+also need to use a .well-known URI instead. However, see also "I have created a
+.well-known URI. Do I still need an SRV record?".
+
+### I have created a .well-known URI. Do I still need an SRV record?
+
+As of Synapse 0.99, Synapse will first check for the existence of a `.well-known`
+URI and follow any delegation it suggests. It will only then check for the
+existence of an SRV record.
+
+That means that the SRV record will often be redundant. However, you should
+remember that there may still be older versions of Synapse in the federation
+which do not understand `.well-known` URIs, so if you removed your SRV record you
+would no longer be able to federate with them.
+
+It is therefore best to leave the SRV record in place for now. Synapse 0.34 and
+earlier will follow the SRV record (and not care about the invalid
+certificate). Synapse 0.99 and later will follow the .well-known URI, with the
+correct certificate chain.
+
+### It used to work just fine, why are you breaking everything?
+
+We have always wanted Matrix servers to be as easy to set up as possible, and
+so back when we started federation in 2014 we didn't want admins to have to go
+through the cumbersome process of buying a valid TLS certificate to run a
+server. This was before Let's Encrypt came along and made getting a free and
+valid TLS certificate straightforward. So instead, we adopted a system based on
+[Perspectives](https://en.wikipedia.org/wiki/Convergence_(SSL)): an approach
+where you check a set of "notary servers" (in practice, homeservers) to vouch
+for the validity of a certificate rather than having it signed by a CA. As long
+as enough different notaries agree on the certificate's validity, then it is
+trusted.
+
+However, in practice this has never worked properly. Most people only use the
+default notary server (matrix.org), leading to inadvertent centralisation which
+we want to eliminate. Meanwhile, we never implemented the full consensus
+algorithm to query the servers participating in a room to determine consensus
+on whether a given certificate is valid. This is fiddly to get right
+(especially in face of sybil attacks), and we found ourselves questioning
+whether it was worth the effort to finish the work and commit to maintaining a
+secure certificate validation system as opposed to focusing on core Matrix
+development.
+
+Meanwhile, Let's Encrypt came along in 2016, and put the final nail in the
+coffin of the Perspectives project (which was already pretty dead). So, the
+Spec Core Team decided that a better approach would be to mandate valid TLS
+certificates for federation alongside the rest of the Web. More details can be
+found in
+[MSC1711](https://github.com/matrix-org/matrix-doc/blob/main/proposals/1711-x509-for-federation.md#background-the-failure-of-the-perspectives-approach).
+
+This results in a breaking change, which is disruptive, but absolutely critical
+for the security model. However, the existence of Let's Encrypt as a trivial
+way to replace the old self-signed certificates with valid CA-signed ones helps
+smooth things over massively, especially as Synapse can now automate Let's
+Encrypt certificate generation if needed.
+
+### Can I manage my own certificates rather than having Synapse renew certificates itself?
+
+Yes, you are welcome to manage your certificates yourself. Synapse will only
+attempt to obtain certificates from Let's Encrypt if you configure it to do
+so.The only requirement is that there is a valid TLS cert present for
+federation end points.
+
+### Do you still recommend against using a reverse proxy on the federation port?
+
+We no longer actively recommend against using a reverse proxy. Many admins will
+find it easier to direct federation traffic to a reverse proxy and manage their
+own TLS certificates, and this is a supported configuration.
+
+See [the reverse proxy documentation](reverse_proxy.md) for information on setting up a
+reverse proxy.
+
+### Do I still need to give my TLS certificates to Synapse if I am using a reverse proxy?
+
+Practically speaking, this is no longer necessary.
+
+If you are using a reverse proxy for all of your TLS traffic, then you can set
+`no_tls: True`. In that case, the only reason Synapse needs the certificate is
+to populate a legacy 'tls_fingerprints' field in the federation API. This is
+ignored by Synapse 0.99.0 and later, and the only time pre-0.99 Synapses will
+check it is when attempting to fetch the server keys - and generally this is
+delegated via `matrix.org`, which is on 0.99.0.
+
+However, there is a bug in Synapse 0.99.0
+[4554](<https://github.com/matrix-org/synapse/issues/4554>) which prevents
+Synapse from starting if you do not give it a TLS certificate. To work around
+this, you can give it any TLS certificate at all. This will be fixed soon.
+
+### Do I need the same certificate for the client and federation port?
+
+No. There is nothing stopping you from using different certificates,
+particularly if you are using a reverse proxy. However, Synapse will use the
+same certificate on any ports where TLS is configured.
+
+### How do I tell Synapse to reload my keys/certificates after I replace them?
+
+Synapse will reload the keys and certificates when it receives a SIGHUP - for
+example `kill -HUP $(cat homeserver.pid)`. Alternatively, simply restart
+Synapse, though this will result in downtime while it restarts.
@@ -50,10 +50,8 @@ build the documentation with:
 mdbook build
 ```

-The rendered contents will be outputted to a new `book/` directory at the root of the repository. Please note that 
-index.html is not built by default, it is created by copying over the file `welcome_and_overview.html` to `index.html`
-during deployment. Thus, when running `mdbook serve` locally the book will initially show a 404 in place of the index
-due to the above. Do not be alarmed!
+The rendered contents will be outputted to a new `book/` directory at the root of the repository. You can
+browse the book by opening `book/index.html` in a web browser.

 You can also have mdbook host the docs on a local webserver with hot-reload functionality via:

@@ -13,6 +13,7 @@

 # Upgrading
  - [Upgrading between Synapse Versions](upgrade.md)
+  - [Upgrading from pre-Synapse 1.0](MSC1711_certificates_FAQ.md)

 # Usage
  - [Federation](federate.md)
@@ -22,14 +23,13 @@
    - [Structured Logging](structured_logging.md)
    - [Templates](templates.md)
    - [User Authentication](usage/configuration/user_authentication/README.md)
-      - [Single-Sign On](usage/configuration/user_authentication/single_sign_on/README.md)
+      - [Single-Sign On]()
        - [OpenID Connect](openid.md)
-        - [SAML](usage/configuration/user_authentication/single_sign_on/saml.md)
-        - [CAS](usage/configuration/user_authentication/single_sign_on/cas.md)
+        - [SAML]()
+        - [CAS]()
        - [SSO Mapping Providers](sso_mapping_providers.md)
      - [Password Auth Providers](password_auth_providers.md)
      - [JSON Web Tokens](jwt.md)
-      - [Refresh Tokens](usage/configuration/user_authentication/refresh_tokens.md)
    - [Registration Captcha](CAPTCHA_SETUP.md)
    - [Application Services](application_services.md)
    - [Server Notices](server_notices.md)
@@ -44,7 +44,6 @@
        - [Presence router callbacks](modules/presence_router_callbacks.md)
        - [Account validity callbacks](modules/account_validity_callbacks.md)
        - [Password auth provider callbacks](modules/password_auth_provider_callbacks.md)
-        - [Background update controller callbacks](modules/background_update_controller_callbacks.md)
        - [Porting a legacy module to the new interface](modules/porting_legacy_module.md)
    - [Workers](workers.md)
      - [Using `synctl` with Workers](synctl_workers.md)
@@ -52,7 +51,6 @@
  - [Administration](usage/administration/README.md)
    - [Admin API](usage/administration/admin_api/README.md)
      - [Account Validity](admin_api/account_validity.md)
-      - [Background Updates](usage/administration/admin_api/background_updates.md)
      - [Delete Group](admin_api/delete_group.md)
      - [Event Reports](admin_api/event_reports.md)
      - [Media](admin_api/media_admin_api.md)
@@ -65,24 +63,16 @@
      - [Statistics](admin_api/statistics.md)
      - [Users](admin_api/user_admin_api.md)
      - [Server Version](admin_api/version_api.md)
-      - [Federation](usage/administration/admin_api/federation.md)
    - [Manhole](manhole.md)
    - [Monitoring](metrics-howto.md)
-    - [Understanding Synapse Through Grafana Graphs](usage/administration/understanding_synapse_through_grafana_graphs.md)
-    - [Useful SQL for Admins](usage/administration/useful_sql_for_admins.md)
-    - [Database Maintenance Tools](usage/administration/database_maintenance_tools.md)
-    - [State Groups](usage/administration/state_groups.md)
    - [Request log format](usage/administration/request_log.md)
-    - [Admin FAQ](usage/administration/admin_faq.md)
    - [Scripts]()

 # Development
  - [Contributing Guide](development/contributing_guide.md)
  - [Code Style](code_style.md)
-  - [Release Cycle](development/releases.md)
  - [Git Usage](development/git.md)
  - [Testing]()
-    - [Demo scripts](development/demo.md)
  - [OpenTracing](opentracing.md)
  - [Database Schemas](development/database_schema.md)
  - [Experimental features](development/experimental_features.md)
@@ -103,4 +93,3 @@

 # Other
  - [Dependency Deprecation Policy](deprecation_policy.md)
-  - [Running Synapse on a Single-Board Computer](other/running_synapse_on_single_board_computers.md)
@@ -4,9 +4,6 @@ This API allows a server administrator to manage the validity of an account. To
 use it, you must enable the account validity feature (under
 `account_validity`) in Synapse's configuration.

-To use it, you will need to authenticate by providing an `access_token`
-for a server admin: see [Admin API](../usage/administration/admin_api).
-
 ## Renew account

 This API extends the validity of an account by as much time as configured in the
@@ -4,11 +4,11 @@ This API lets a server admin delete a local group. Doing so will kick all
 users out of the group so that their clients will correctly handle the group
 being deleted.

-To use it, you will need to authenticate by providing an `access_token`
-for a server admin: see [Admin API](../usage/administration/admin_api).
-
 The API is:

 ```
 POST /_synapse/admin/v1/delete_group/<group_id>
 ```
+
+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: see [Admin API](../usage/administration/admin_api).
@@ -2,13 +2,12 @@

 This API returns information about reported events.

-To use it, you will need to authenticate by providing an `access_token`
-for a server admin: see [Admin API](../usage/administration/admin_api).
-
 The api is:
 ```
 GET /_synapse/admin/v1/event_reports?from=0&limit=10
 ```
+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: see [Admin API](../usage/administration/admin_api).

 It returns a JSON body like the following:

@@ -95,10 +94,12 @@ The api is:
 ```
 GET /_synapse/admin/v1/event_reports/<report_id>
 ```
+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: see [Admin API](../usage/administration/admin_api).

 It returns a JSON body like the following:

-```json
+```jsonc
 {
    "event_id": "$bNUFCwGzWca1meCGkjp-zwslF-GfVcXukvRLI1_FaVY",
    "event_json": {
@@ -131,7 +132,7 @@ It returns a JSON body like the following:
        },
        "type": "m.room.message",
        "unsigned": {
-            "age_ts": 1592291711430
+            "age_ts": 1592291711430,
        }
    },
    "id": <report_id>,
@@ -1,13 +1,24 @@
+# Contents
+- [Querying media](#querying-media)
+  * [List all media in a room](#list-all-media-in-a-room)
+  * [List all media uploaded by a user](#list-all-media-uploaded-by-a-user)
+- [Quarantine media](#quarantine-media)
+  * [Quarantining media by ID](#quarantining-media-by-id)
+  * [Remove media from quarantine by ID](#remove-media-from-quarantine-by-id)
+  * [Quarantining media in a room](#quarantining-media-in-a-room)
+  * [Quarantining all media of a user](#quarantining-all-media-of-a-user)
+  * [Protecting media from being quarantined](#protecting-media-from-being-quarantined)
+  * [Unprotecting media from being quarantined](#unprotecting-media-from-being-quarantined)
+- [Delete local media](#delete-local-media)
+  * [Delete a specific local media](#delete-a-specific-local-media)
+  * [Delete local media by date or size](#delete-local-media-by-date-or-size)
+  * [Delete media uploaded by a user](#delete-media-uploaded-by-a-user)
+- [Purge Remote Media API](#purge-remote-media-api)
+
 # Querying media

 These APIs allow extracting media information from the homeserver.

-Details about the format of the `media_id` and storage of the media in the file system
-are documented under [media repository](../media_repository.md).
-
-To use it, you will need to authenticate by providing an `access_token`
-for a server admin: see [Admin API](../usage/administration/admin_api).
-
 ## List all media in a room

 This API gets a list of known media in a room.
@@ -17,6 +28,8 @@ The API is:
 ```
 GET /_synapse/admin/v1/room/<room_id>/media
 ```
+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: see [Admin API](../usage/administration/admin_api).

 The API returns a JSON body like the following:
 ```json
@@ -244,9 +257,9 @@ POST /_synapse/admin/v1/media/<server_name>/delete?before_ts=<before_ts>
 URL Parameters

 * `server_name`: string - The name of your local server (e.g `matrix.org`).
-* `before_ts`: string representing a positive integer - Unix timestamp in milliseconds.
+* `before_ts`: string representing a positive integer - Unix timestamp in ms.
 Files that were last used before this timestamp will be deleted. It is the timestamp of
-last access, not the timestamp when the file was created.
+last access and not the timestamp creation.
 * `size_gt`: Optional - string representing a positive integer - Size of the media in bytes.
 Files that are larger will be deleted. Defaults to `0`.
 * `keep_profiles`: Optional - string representing a boolean - Switch to also delete files
@@ -289,7 +302,7 @@ POST /_synapse/admin/v1/purge_media_cache?before_ts=<unix_timestamp_in_ms>

 URL Parameters

-* `unix_timestamp_in_ms`: string representing a positive integer - Unix timestamp in milliseconds.
+* `unix_timestamp_in_ms`: string representing a positive integer - Unix timestamp in ms.
 All cached media that was last accessed before this timestamp will be removed.

 Response:
@@ -304,5 +317,8 @@ The following fields are returned in the JSON response body:

 * `deleted`: integer - The number of media items successfully deleted

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: see [Admin API](../usage/administration/admin_api).
+
 If the user re-requests purged remote media, synapse will re-request the media
 from the originating server.
@@ -10,15 +10,15 @@ paginate further back in the room from the point being purged from.
 Note that Synapse requires at least one message in each room, so it will never
 delete the last message in a room.

-To use it, you will need to authenticate by providing an `access_token`
-for a server admin: see [Admin API](../usage/administration/admin_api).
-
 The API is:

 ```
 POST /_synapse/admin/v1/purge_history/<room_id>[/<event_id>]
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
 By default, events sent by local users are not deleted, as they may represent
 the only copies of this content in existence. (Events sent by remote users are
 deleted.)
@@ -27,7 +27,7 @@ Room state data (such as joins, leaves, topic) is always preserved.

 To delete local message events as well, set `delete_local_events` in the body:

-```json
+```
 {
   "delete_local_events": true
 }
@@ -57,6 +57,9 @@ It is possible to poll for updates on recent purges with a second API;
 GET /_synapse/admin/v1/purge_history_status/<purge_id>
 ```

+Again, you will need to authenticate by providing an `access_token` for a
+server admin.
+
 This API returns a JSON body like the following:

 ```json
@@ -67,8 +70,6 @@ This API returns a JSON body like the following:

 The status will be one of `active`, `complete`, or `failed`.

-If `status` is `failed` there will be a string `error` with the error message.
-
 ## Reclaim disk space (Postgres)

 To reclaim the disk space and return it to the operating system, you need to run
@@ -5,9 +5,6 @@ to a room with a given `room_id_or_alias`. You can only modify the membership of
 local users. The server administrator must be in the room and have permission to
 invite users.

-To use it, you will need to authenticate by providing an `access_token`
-for a server admin: see [Admin API](../usage/administration/admin_api).
-
 ## Parameters

 The following parameters are available:
@@ -26,9 +23,12 @@ POST /_synapse/admin/v1/join/<room_id_or_alias>
 }
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: see [Admin API](../usage/administration/admin_api).
+
 Response:

-```json
+```
 {
  "room_id": "!636q39766251:server.com"
 }
@@ -1,12 +1,20 @@
+# Contents
+- [List Room API](#list-room-api)
+- [Room Details API](#room-details-api)
+- [Room Members API](#room-members-api)
+- [Room State API](#room-state-api)
+- [Delete Room API](#delete-room-api)
+  * [Undoing room shutdowns](#undoing-room-shutdowns)
+- [Make Room Admin API](#make-room-admin-api)
+- [Forward Extremities Admin API](#forward-extremities-admin-api)
+- [Event Context API](#event-context-api)
+
 # List Room API

 The List Room admin API allows server admins to get a list of rooms on their
 server. There are various parameters available that allow for filtering and
 sorting the returned list. This API supports pagination.

-To use it, you will need to authenticate by providing an `access_token`
-for a server admin: see [Admin API](../usage/administration/admin_api).
-
 **Parameters**

 The following query parameters are available:
@@ -30,14 +38,9 @@ The following query parameters are available:
  - `history_visibility` - Rooms are ordered alphabetically by visibility of history of the room.
  - `state_events` - Rooms are ordered by number of state events. Largest to smallest.
 * `dir` - Direction of room order. Either `f` for forwards or `b` for backwards. Setting
-  this value to `b` will reverse the above sort order. Defaults to `f`.
-* `search_term` - Filter rooms by their room name, canonical alias and room id.
-  Specifically, rooms are selected if the search term is contained in
-  - the room's name,
-  - the local part of the room's canonical alias, or
-  - the complete (local and server part) room's id (case sensitive).
-
-  Defaults to no filtering.
+          this value to `b` will reverse the above sort order. Defaults to `f`.
+* `search_term` - Filter rooms by their room name. Search term can be contained in any
+                  part of the room name. Defaults to no filtering.

 **Response**

@@ -84,7 +87,7 @@ GET /_synapse/admin/v1/rooms

 A response body like the following is returned:

-```json
+```jsonc
 {
  "rooms": [
    {
@@ -167,7 +170,7 @@ GET /_synapse/admin/v1/rooms?order_by=size

 A response body like the following is returned:

-```json
+```jsonc
 {
  "rooms": [
    {
@@ -205,7 +208,7 @@ A response body like the following is returned:
    }
  ],
  "offset": 0,
-  "total_rooms": 150,
+  "total_rooms": 150
  "next_token": 100
 }
 ```
@@ -221,7 +224,7 @@ GET /_synapse/admin/v1/rooms?order_by=size&from=100

 A response body like the following is returned:

-```json
+```jsonc
 {
  "rooms": [
    {
@@ -375,86 +378,9 @@ A response body like the following is returned:
 }
 ```

-# Block Room API
-The Block Room admin API allows server admins to block and unblock rooms,
-and query to see if a given room is blocked.
-This API can be used to pre-emptively block a room, even if it's unknown to this
-homeserver. Users will be prevented from joining a blocked room.
-
-## Block or unblock a room
-
-The API is:
-
-```
-PUT /_synapse/admin/v1/rooms/<room_id>/block
-```
-
-with a body of:
-
-```json
-{
-    "block": true
-}
-```
-
-A response body like the following is returned:
-
-```json
-{
-    "block": true
-}
-```
-
-**Parameters**
-
-The following parameters should be set in the URL:
-
- `room_id` - The ID of the room.
-
-The following JSON body parameters are available:
-
- `block` - If `true` the room will be blocked and if `false` the room will be unblocked.
-
-**Response**
-
-The following fields are possible in the JSON response body:
-
- `block` - A boolean. `true` if the room is blocked, otherwise `false`
-
-## Get block status
-
-The API is:
-
-```
-GET /_synapse/admin/v1/rooms/<room_id>/block
-```
-
-A response body like the following is returned:
-
-```json
-{
-    "block": true,
-    "user_id": "<user_id>"
-}
-```
-
-**Parameters**
-
-The following parameters should be set in the URL:
-
- `room_id` - The ID of the room.
-
-**Response**
-
-The following fields are possible in the JSON response body:
-
- `block` - A boolean. `true` if the room is blocked, otherwise `false`
- `user_id` - An optional string. If the room is blocked (`block` is `true`) shows
-  the user who has add the room to blocking list. Otherwise it is not displayed.
-
 # Delete Room API

-The Delete Room admin API allows server admins to remove rooms from the server
+The Delete Room admin API allows server admins to remove rooms from server
 and block these rooms.

 Shuts down a room. Moves all local users and room aliases automatically to a
@@ -465,30 +391,18 @@ The new room will be created with the user specified by the `new_room_user_id` p
 as room administrator and will contain a message explaining what happened. Users invited
 to the new room will have power level `-10` by default, and thus be unable to speak.

-If `block` is `true`, users will be prevented from joining the old room.
-This option can in [Version 1](#version-1-old-version) also be used to pre-emptively
-block a room, even if it's unknown to this homeserver. In this case, the room will be
-blocked, and no further action will be taken. If `block` is `false`, attempting to
-delete an unknown room is invalid and will be rejected as a bad request.
+If `block` is `True` it prevents new joins to the old room.

 This API will remove all trace of the old room from your database after removing
 all local users. If `purge` is `true` (the default), all traces of the old room will
 be removed from your database after removing all local users. If you do not want
 this to happen, set `purge` to `false`.
-Depending on the amount of history being purged, a call to the API may take
+Depending on the amount of history being purged a call to the API may take
 several minutes or longer.

 The local server will only have the power to move local user and room aliases to
 the new room. Users on other servers will be unaffected.

-## Version 1 (old version)
-
-This version works synchronously. That means you only get the response once the server has
-finished the action, which may take a long time. If you request the same action
-a second time, and the server has not finished the first one, the second request will block.
-This is fixed in version 2 of this API. The parameters are the same in both APIs.
-This API will become deprecated in the future.
-
 The API is:

 ```
@@ -507,6 +421,9 @@ with a body of:
 }
 ```

+To use it, you will need to authenticate by providing an ``access_token`` for a
+server admin: see [Admin API](../usage/administration/admin_api).
+
 A response body like the following is returned:

 ```json
@@ -523,44 +440,6 @@ A response body like the following is returned:
 }
 ```

-The parameters and response values have the same format as
-[version 2](#version-2-new-version) of the API.
-
-## Version 2 (new version)
-
-**Note**: This API is new, experimental and "subject to change".
-
-This version works asynchronously, meaning you get the response from server immediately
-while the server works on that task in background. You can then request the status of the action
-to check if it has completed.
-
-The API is:
-
-```
-DELETE /_synapse/admin/v2/rooms/<room_id>
-```
-
-with a body of:
-
-```json
-{
-    "new_room_user_id": "@someuser:example.com",
-    "room_name": "Content Violation Notification",
-    "message": "Bad Room has been shutdown due to content violations on this server. Please review our Terms of Service.",
-    "block": true,
-    "purge": true
-}
-```
-
-The API starts the shut down and purge running, and returns immediately with a JSON body with
-a purge id:
-
-```json
-{
-    "delete_id": "<opaque id>"
-}
-```
-
 **Parameters**

 The following parameters should be set in the URL:
@@ -580,10 +459,8 @@ The following JSON body parameters are available:
              `new_room_user_id` in the new room. Ideally this will clearly convey why the
               original room was shut down. Defaults to `Sharing illegal content on this server
               is not permitted and rooms in violation will be blocked.`
-* `block` - Optional. If set to `true`, this room will be added to a blocking list,
-            preventing future attempts to join the room. Rooms can be blocked
-            even if they're not yet known to the homeserver (only with
-            [Version 1](#version-1-old-version) of the API). Defaults to `false`.
+* `block` - Optional. If set to `true`, this room will be added to a blocking list, preventing
+            future attempts to join the room. Defaults to `false`.
 * `purge` - Optional. If set to `true`, it will remove all traces of the room from your database.
            Defaults to `true`.
 * `force_purge` - Optional, and ignored unless `purge` is `true`. If set to `true`, it
@@ -593,124 +470,16 @@ The following JSON body parameters are available:

 The JSON body must not be empty. The body must be at least `{}`.

-## Status of deleting rooms
-
-**Note**: This API is new, experimental and "subject to change".
-
-It is possible to query the status of the background task for deleting rooms.
-The status can be queried up to 24 hours after completion of the task,
-or until Synapse is restarted (whichever happens first).
-
-### Query by `room_id`
-
-With this API you can get the status of all active deletion tasks, and all those completed in the last 24h,
-for the given `room_id`.
-
-The API is:
-
-```
-GET /_synapse/admin/v2/rooms/<room_id>/delete_status
-```
-
-A response body like the following is returned:
-
-```json
-{
-    "results": [
-        {
-            "delete_id": "delete_id1",
-            "status": "failed",
-            "error": "error message",
-            "shutdown_room": {
-                "kicked_users": [],
-                "failed_to_kick_users": [],
-                "local_aliases": [],
-                "new_room_id": null
-            }
-        }, {
-            "delete_id": "delete_id2",
-            "status": "purging",
-            "shutdown_room": {
-                "kicked_users": [
-                    "@foobar:example.com"
-                ],
-                "failed_to_kick_users": [],
-                "local_aliases": [
-                    "#badroom:example.com",
-                    "#evilsaloon:example.com"
-                ],
-                "new_room_id": "!newroomid:example.com"
-            }
-        }
-    ]
-}
-```
-
-**Parameters**
-
-The following parameters should be set in the URL:
-
-* `room_id` - The ID of the room.
-
-### Query by `delete_id`
-
-With this API you can get the status of one specific task by `delete_id`.
-
-The API is:
-
-```
-GET /_synapse/admin/v2/rooms/delete_status/<delete_id>
-```
-
-A response body like the following is returned:
-
-```json
-{
-    "status": "purging",
-    "shutdown_room": {
-        "kicked_users": [
-            "@foobar:example.com"
-        ],
-        "failed_to_kick_users": [],
-        "local_aliases": [
-            "#badroom:example.com",
-            "#evilsaloon:example.com"
-        ],
-        "new_room_id": "!newroomid:example.com"
-    }
-}
-```
-
-**Parameters**
-
-The following parameters should be set in the URL:
-
-* `delete_id` - The ID for this delete.
-
-### Response
+**Response**

 The following fields are returned in the JSON response body:

- `results` - An array of objects, each containing information about one task.
-  This field is omitted from the result when you query by `delete_id`.
-  Task objects contain the following fields:
-  - `delete_id` - The ID for this purge if you query by `room_id`.
-  - `status` - The status will be one of:
-    - `shutting_down` - The process is removing users from the room.
-    - `purging` - The process is purging the room and event data from database.
-    - `complete` - The process has completed successfully.
-    - `failed` - The process is aborted, an error has occurred.
-  - `error` - A string that shows an error message if `status` is `failed`.
-    Otherwise this field is hidden.
-  - `shutdown_room` - An object containing information about the result of shutting down the room.
-    *Note:* The result is shown after removing the room members.
-    The delete process can still be running. Please pay attention to the `status`.
-    - `kicked_users` - An array of users (`user_id`) that were kicked.
-    - `failed_to_kick_users` - An array of users (`user_id`) that that were not kicked.
-    - `local_aliases` - An array of strings representing the local aliases that were
-      migrated from the old room to the new.
-    - `new_room_id` - A string representing the room ID of the new room, or `null` if
-      no such room was created.
+* `kicked_users` - An array of users (`user_id`) that were kicked.
+* `failed_to_kick_users` - An array of users (`user_id`) that that were not kicked.
+* `local_aliases` - An array of strings representing the local aliases that were migrated from
+                    the old room to the new.
+* `new_room_id` - A string representing the room ID of the new room.
+

 ## Undoing room deletions

@@ -751,6 +520,16 @@ With all that being said, if you still want to try and recover the room:
 4. If `new_room_user_id` was given, a 'Content Violation' will have been
   created. Consider whether you want to delete that roomm.

+## Deprecated endpoint
+
+The previous deprecated API will be removed in a future release, it was:
+
+```
+POST /_synapse/admin/v1/rooms/<room_id>/delete
+```
+
+It behaves the same way than the current endpoint except the path and the method.
+
 # Make Room Admin API

 Grants another user the highest power available to a local user who is in the room.
@@ -3,15 +3,15 @@
 Returns information about all local media usage of users. Gives the
 possibility to filter them by time and user.

-To use it, you will need to authenticate by providing an `access_token`
-for a server admin: see [Admin API](../usage/administration/admin_api).
-
 The API is:

 ```
 GET /_synapse/admin/v1/statistics/users/media
 ```

+To use it, you will need to authenticate by providing an `access_token`
+for a server admin: see [Admin API](../usage/administration/admin_api).
+
 A response body like the following is returned:

 ```json
@@ -1,8 +1,5 @@
 # User Admin API

-To use it, you will need to authenticate by providing an `access_token`
-for a server admin: see [Admin API](../usage/administration/admin_api).
-
 ## Query User Account

 This API returns information about a specific user account.
@@ -13,12 +10,14 @@ The api is:
 GET /_synapse/admin/v2/users/<user_id>
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
 It returns a JSON body like the following:

-```jsonc
+```json
 {
-    "name": "@user:example.com",
-    "displayname": "User", // can be null if not set
+    "displayname": "User",
    "threepids": [
        {
            "medium": "email",
@@ -33,11 +32,11 @@ It returns a JSON body like the following:
            "validated_at": 1586458409743
        }
    ],
-    "avatar_url": "<avatar_url>",  // can be null if not set
-    "is_guest": 0,
+    "avatar_url": "<avatar_url>",
    "admin": 0,
    "deactivated": 0,
    "shadow_banned": 0,
+    "password_hash": "$2b$12$p9B4GkqYdRTPGD",
    "creation_ts": 1560432506,
    "appservice_id": null,
    "consent_server_notice_sent": null,
@@ -51,8 +50,7 @@ It returns a JSON body like the following:
            "auth_provider": "<provider2>",
            "external_id": "<user_id_provider_2>"
        }
-    ],
-    "user_type": null
+    ]
 }
 ```

@@ -99,11 +97,13 @@ with a body of:
    ],
    "avatar_url": "<avatar_url>",
    "admin": false,
-    "deactivated": false,
-    "user_type": null
+    "deactivated": false
 }
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
 Returns HTTP status code:
 - `201` - When a new user object was created.
 - `200` - When a user was modified.
@@ -126,8 +126,7 @@ Body parameters:
  [Sample Configuration File](../usage/configuration/homeserver_sample_config.html)
  section `sso` and `oidc_providers`.
  - `auth_provider` - string. ID of the external identity provider. Value of `idp_id`
-    in the homeserver configuration. Note that no error is raised if the provided
-    value is not in the homeserver configuration.
+    in homeserver configuration.
  - `external_id` - string, user ID in the external identity provider.
 - `avatar_url` - string, optional, must be a
  [MXC URI](https://matrix.org/docs/spec/client_server/r0.6.0#matrix-content-mxc-uris).
@@ -136,9 +135,6 @@ Body parameters:
  unchanged on existing accounts and set to `false` for new accounts.
  A user cannot be erased by deactivating with this API. For details on
  deactivating users see [Deactivate Account](#deactivate-account).
- `user_type` - string or null, optional. If provided, the user type will be
-  adjusted. If `null` given, the user type will be cleared. Other 
-  allowed options are: `bot` and `support`.

 If the user already exists then optional parameters default to the current value.

@@ -154,6 +150,9 @@ By default, the response is ordered by ascending user ID.
 GET /_synapse/admin/v2/users?from=0&limit=10&guests=false
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
 A response body like the following is returned:

 ```json
@@ -273,6 +272,9 @@ GET /_matrix/client/r0/admin/whois/<userId>
 See also: [Client Server
 API Whois](https://matrix.org/docs/spec/client_server/r0.6.1#get-matrix-client-r0-admin-whois-userid).

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
 It returns a JSON body like the following:

 ```json
@@ -327,26 +329,23 @@ with a body of:
 }
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
 The erase parameter is optional and defaults to `false`.
 An empty body may be passed for backwards compatibility.

 The following actions are performed when deactivating an user:

- Try to unbind 3PIDs from the identity server
+- Try to unpind 3PIDs from the identity server
 - Remove all 3PIDs from the homeserver
 - Delete all devices and E2EE keys
 - Delete all access tokens
- Delete all pushers
 - Delete the password hash
 - Removal from all rooms the user is a member of
 - Remove the user from the user directory
 - Reject all pending invites
 - Remove all account validity information related to the user
- Remove the arbitrary data store known as *account data*. For example, this includes:
-    - list of ignored users;
-    - push rules;
-    - secret storage keys; and
-    - cross-signing keys.

 The following additional actions are performed during deactivation if `erase`
 is set to `true`:
@@ -355,14 +354,6 @@ is set to `true`:
 - Remove the user's avatar URL
 - Mark the user as erased

-The following actions are **NOT** performed. The list may be incomplete.
-
- Remove mappings of SSO IDs
- [Delete media uploaded](#delete-media-uploaded-by-a-user) by user (included avatar images)
- Delete sent and received messages
- Remove the user's creation (registration) timestamp
- [Remove rate limit overrides](#override-ratelimiting-for-users)
- Remove from monthly active users

 ## Reset password

@@ -383,6 +374,9 @@ with a body of:
 }
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
 The parameter `new_password` is required.
 The parameter `logout_devices` is optional and defaults to `true`.

@@ -395,6 +389,9 @@ The api is:
 GET /_synapse/admin/v1/users/<user_id>/admin
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
 A response body like the following is returned:

 ```json
@@ -422,6 +419,10 @@ with a body of:
 }
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
+
 ## List room memberships of a user

 Gets a list of all `room_id` that a specific `user_id` is member.
@@ -432,6 +433,9 @@ The API is:
 GET /_synapse/admin/v1/users/<user_id>/joined_rooms
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
 A response body like the following is returned:

 ```json
@@ -461,90 +465,10 @@ The following fields are returned in the JSON response body:
 - `joined_rooms` - An array of `room_id`.
 - `total` - Number of rooms.

-## Account Data
-Gets information about account data for a specific `user_id`.
-
-The API is:
-
-```
-GET /_synapse/admin/v1/users/<user_id>/accountdata
-```
-
-A response body like the following is returned:
-
-```json
-{
-    "account_data": {
-        "global": {
-            "m.secret_storage.key.LmIGHTg5W": {
-                "algorithm": "m.secret_storage.v1.aes-hmac-sha2",
-                "iv": "fwjNZatxg==",
-                "mac": "eWh9kNnLWZUNOgnc="
-            },
-            "im.vector.hide_profile": {
-                "hide_profile": true
-            },
-            "org.matrix.preview_urls": {
-                "disable": false
-            },
-            "im.vector.riot.breadcrumb_rooms": {
-                "rooms": [
-                    "!LxcBDAsDUVAfJDEo:matrix.org",
-                    "!MAhRxqasbItjOqxu:matrix.org"
-                ]
-            },
-            "m.accepted_terms": {
-                "accepted": [
-                    "https://example.org/somewhere/privacy-1.2-en.html",
-                    "https://example.org/somewhere/terms-2.0-en.html"
-                ]
-            },
-            "im.vector.setting.breadcrumbs": {
-                "recent_rooms": [
-                    "!MAhRxqasbItqxuEt:matrix.org",
-                    "!ZtSaPCawyWtxiImy:matrix.org"
-                ]
-            }
-        },
-        "rooms": {
-            "!GUdfZSHUJibpiVqHYd:matrix.org": {
-                "m.fully_read": {
-                    "event_id": "$156334540fYIhZ:matrix.org"
-                }
-            },
-            "!tOZwOOiqwCYQkLhV:matrix.org": {
-                "m.fully_read": {
-                    "event_id": "$xjsIyp4_NaVl2yPvIZs_k1Jl8tsC_Sp23wjqXPno"
-                }
-            }
-        }
-    }
-}
-```
-
-**Parameters**
-
-The following parameters should be set in the URL:
-
- `user_id` - fully qualified: for example, `@user:server.com`.
-
-**Response**
-
-The following fields are returned in the JSON response body:
-
- `account_data` - A map containing the account data for the user
-  - `global` - A map containing the global account data for the user
-  - `rooms` - A map containing the account data per room for the user
-
 ## User media

 ### List media uploaded by a user
 Gets a list of all local media that a specific `user_id` has created.
-These are media that the user has uploaded themselves
-([local media](../media_repository.md#local-media)), as well as
-[URL preview images](../media_repository.md#url-previews) requested by the user if the
-[feature is enabled](../development/url_previews.md).
-
 By default, the response is ordered by descending creation date and ascending media ID.
 The newest media is on top. You can change the order with parameters
 `order_by` and `dir`.
@@ -555,6 +479,9 @@ The API is:
 GET /_synapse/admin/v1/users/<user_id>/media
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
 A response body like the following is returned:

 ```json
@@ -641,9 +568,7 @@ The following fields are returned in the JSON response body:
  Media objects contain the following fields:
  - `created_ts` - integer - Timestamp when the content was uploaded in ms.
  - `last_access_ts` - integer - Timestamp when the content was last accessed in ms.
-  - `media_id` - string - The id used to refer to the media. Details about the format
-    are documented under
-    [media repository](../media_repository.md).
+  - `media_id` - string - The id used to refer to the media.
  - `media_length` - integer - Length of the media in bytes.
  - `media_type` - string - The MIME-type of the media.
  - `quarantined_by` - string - The user ID that initiated the quarantine request
@@ -671,6 +596,9 @@ The API is:
 DELETE /_synapse/admin/v1/users/<user_id>/media
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
 A response body like the following is returned:

 ```json
@@ -743,6 +671,9 @@ The API is:
 GET /_synapse/admin/v2/users/<user_id>/devices
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
 A response body like the following is returned:

 ```json
@@ -808,6 +739,9 @@ POST /_synapse/admin/v2/users/<user_id>/delete_devices
 }
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
 An empty JSON dict is returned.

 **Parameters**
@@ -829,6 +763,9 @@ The API is:
 GET /_synapse/admin/v2/users/<user_id>/devices/<device_id>
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
 A response body like the following is returned:

 ```json
@@ -874,6 +811,9 @@ PUT /_synapse/admin/v2/users/<user_id>/devices/<device_id>
 }
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
 An empty JSON dict is returned.

 **Parameters**
@@ -900,6 +840,9 @@ DELETE /_synapse/admin/v2/users/<user_id>/devices/<device_id>
 {}
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
 An empty JSON dict is returned.

 **Parameters**
@@ -918,6 +861,9 @@ The API is:
 GET /_synapse/admin/v1/users/<user_id>/pushers
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
 A response body like the following is returned:

 ```json
@@ -987,7 +933,7 @@ The following fields are returned in the JSON response body:
 See also the
 [Client-Server API Spec on pushers](https://matrix.org/docs/spec/client_server/latest#get-matrix-client-r0-pushers).

-## Controlling whether a user is shadow-banned
+## Shadow-banning users

 Shadow-banning is a useful tool for moderating malicious or egregiously abusive users.
 A shadow-banned users receives successful responses to their client-server API requests,
@@ -1000,19 +946,16 @@ or broken behaviour for the client. A shadow-banned user will not receive any
 notification and it is generally more appropriate to ban or kick abusive users.
 A shadow-banned user will be unable to contact anyone on the server.

-To shadow-ban a user the API is:
+The API is:

 ```
 POST /_synapse/admin/v1/users/<user_id>/shadow_ban
 ```

-To un-shadow-ban a user the API is:
+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)

-```
-DELETE /_synapse/admin/v1/users/<user_id>/shadow_ban
-```
-
-An empty JSON dict is returned in both cases.
+An empty JSON dict is returned.

 **Parameters**

@@ -1034,6 +977,9 @@ The API is:
 GET /_synapse/admin/v1/users/<user_id>/override_ratelimit
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
 A response body like the following is returned:

 ```json
@@ -1073,6 +1019,9 @@ The API is:
 POST /_synapse/admin/v1/users/<user_id>/override_ratelimit
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
 A response body like the following is returned:

 ```json
@@ -1115,6 +1064,9 @@ The API is:
 DELETE /_synapse/admin/v1/users/<user_id>/override_ratelimit
 ```

+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
+
 An empty JSON dict is returned.

 ```json
@@ -1140,8 +1092,10 @@ This endpoint will work even if registration is disabled on the server, unlike
 The API is:

 ```
-GET /_synapse/admin/v1/username_available?username=$localpart
+POST /_synapse/admin/v1/username_availabile?username=$localpart
 ```

-The request and response format is the same as the
-[/_matrix/client/r0/register/available](https://matrix.org/docs/spec/client_server/r0.6.0#get-matrix-client-r0-register-available) API.
+The request and response format is the same as the [/_matrix/client/r0/register/available](https://matrix.org/docs/spec/client_server/r0.6.0#get-matrix-client-r0-register-available) API.
+
+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: [Admin API](../usage/administration/admin_api)
@@ -16,6 +16,6 @@ It returns a JSON body like the following:
 ```json
 {
    "server_version": "0.99.2rc1 (b=develop, abcdef123)",
-    "python_version": "3.7.8"
+    "python_version": "3.6.8"
 }
 ```
@@ -7,7 +7,7 @@

 ## Server to Server Stack

-To use the server to server stack, homeservers should only need to
+To use the server to server stack, home servers should only need to
 interact with the Messaging layer.

 The server to server side of things is designed into 4 distinct layers:
@@ -23,7 +23,7 @@ Server with a domain specific API.

 1. **Messaging Layer**

-    This is what the rest of the homeserver hits to send messages, join rooms,
+    This is what the rest of the Home Server hits to send messages, join rooms,
    etc. It also allows you to register callbacks for when it get's notified by
    lower levels that e.g. a new message has been received.

@@ -45,7 +45,7 @@ Server with a domain specific API.

    For incoming PDUs, it has to check the PDUs it references to see
    if we have missed any. If we have go and ask someone (another
-    homeserver) for it.
+    home server) for it.

 3. **Transaction Layer**

@@ -10,9 +10,7 @@ The necessary tools are detailed below.

 First install them with:

-```sh
-pip install -e ".[lint,mypy]"
-```
+    pip install -e ".[lint,mypy]"

 -   **black**

@@ -23,9 +21,7 @@ pip install -e ".[lint,mypy]"
    Have `black` auto-format your code (it shouldn't change any
    functionality) with:

-    ```sh
-    black . --exclude="\.tox|build|env"
-    ```
+        black . --exclude="\.tox|build|env"

 -   **flake8**

@@ -34,9 +30,7 @@ pip install -e ".[lint,mypy]"

    Check all application and test code with:

-    ```sh
-    flake8 synapse tests
-    ```
+        flake8 synapse tests

 -   **isort**

@@ -45,9 +39,7 @@ pip install -e ".[lint,mypy]"

    Auto-fix imports with:

-    ```sh
-    isort -rc synapse tests
-    ```
+        isort -rc synapse tests

    `-rc` means to recursively search the given directories.

@@ -74,19 +66,15 @@ save as it takes a while and is very resource intensive.

        Example:

-        ```python
-        from synapse.types import UserID
-        ...
-        user_id = UserID(local, server)
-        ```
+            from synapse.types import UserID
+            ...
+            user_id = UserID(local, server)

        is preferred over:

-        ```python
-        from synapse import types
-        ...
-        user_id = types.UserID(local, server)
-        ```
+            from synapse import types
+            ...
+            user_id = types.UserID(local, server)

        (or any other variant).

@@ -146,32 +134,30 @@ Some guidelines follow:

 Example:

-```yaml
-## Frobnication ##
+    ## Frobnication ##

-# The frobnicator will ensure that all requests are fully frobnicated.
-# To enable it, uncomment the following.
-#
-#frobnicator_enabled: true
+    # The frobnicator will ensure that all requests are fully frobnicated.
+    # To enable it, uncomment the following.
+    #
+    #frobnicator_enabled: true

-# By default, the frobnicator will frobnicate with the default frobber.
-# The following will make it use an alternative frobber.
-#
-#frobincator_frobber: special_frobber
+    # By default, the frobnicator will frobnicate with the default frobber.
+    # The following will make it use an alternative frobber.
+    #
+    #frobincator_frobber: special_frobber

-# Settings for the frobber
-#
-frobber:
-  # frobbing speed. Defaults to 1.
-  #
-  #speed: 10
+    # Settings for the frobber
+    #
+    frobber:
+      # frobbing speed. Defaults to 1.
+      #
+      #speed: 10

-  # frobbing distance. Defaults to 1000.
-  #
-  #distance: 100
-```
+      # frobbing distance. Defaults to 1000.
+      #
+      #distance: 100

 Note that the sample configuration is generated from the synapse code
-and is maintained by a script, `scripts-dev/generate_sample_config.sh`.
+and is maintained by a script, `scripts-dev/generate_sample_config`.
 Making sure that the output from this script matches the desired format
 is left as an exercise for the reader!
@@ -99,7 +99,7 @@ construct URIs where users can give their consent.
   see if an unauthenticated user is viewing the page. This is typically
   wrapped around the form that would be used to actually agree to the document:

-   ```html
+   ```
   {% if not public_version %}
     <!-- The variables used here are only provided when the 'u' param is given to the homeserver -->
     <form method="post" action="consent">
@@ -1,8 +1,4 @@
-# Delegation of incoming federation traffic
-
-In the following documentation, we use the term `server_name` to refer to that setting
-in your homeserver configuration file. It appears at the ends of user ids, and tells
-other homeservers where they can find your server.
+# Delegation

 By default, other homeservers will expect to be able to reach yours via
 your `server_name`, on port 8448. For example, if you set your `server_name`
@@ -16,21 +12,13 @@ to a different server and/or port (e.g. `synapse.example.com:443`).

 ## .well-known delegation

-To use this method, you need to be able to configure the server at
-`https://<server_name>` to serve a file at
-`https://<server_name>/.well-known/matrix/server`.  There are two ways to do this, shown below.
+To use this method, you need to be able to alter the
+`server_name` 's https server to serve the `/.well-known/matrix/server`
+URL. Having an active server (with a valid TLS certificate) serving your
+`server_name` domain is out of the scope of this documentation.

-Note that the `.well-known` file is hosted on the default port for `https` (port 443).
-
-### External server
-
-For maximum flexibility, you need to configure an external server such as nginx, Apache
-or HAProxy to serve the `https://<server_name>/.well-known/matrix/server` file. Setting
-up such a server is out of the scope of this documentation, but note that it is often
-possible to configure your [reverse proxy](reverse_proxy.md) for this.
-
-The URL `https://<server_name>/.well-known/matrix/server` should be configured
-return a JSON structure containing the key `m.server` like this:
+The URL `https://<server_name>/.well-known/matrix/server` should
+return a JSON structure containing the key `m.server` like so:

 ```json
 {
@@ -38,9 +26,8 @@ return a JSON structure containing the key `m.server` like this:
 }
 ```

-In our example (where we want federation traffic to be routed to
-`https://synapse.example.com`, on port 443), this would mean that
-`https://example.com/.well-known/matrix/server` should return:
+In our example, this would mean that URL `https://example.com/.well-known/matrix/server`
+should return:

 ```json
 {
@@ -51,29 +38,16 @@ In our example (where we want federation traffic to be routed to
 Note, specifying a port is optional. If no port is specified, then it defaults
 to 8448.

-### Serving a `.well-known/matrix/server` file with Synapse
-
-If you are able to set up your domain so that `https://<server_name>` is routed to
-Synapse (i.e., the only change needed is to direct federation traffic to port 443
-instead of port 8448), then it is possible to configure Synapse to serve a suitable
-`.well-known/matrix/server` file. To do so, add the following to your `homeserver.yaml`
-file:
-
-```yaml
-serve_server_wellknown: true
-```
-
-**Note**: this *only* works if `https://<server_name>` is routed to Synapse, so is
-generally not suitable if Synapse is hosted at a subdomain such as
-`https://synapse.example.com`.
+With .well-known delegation, federating servers will check for a valid TLS
+certificate for the delegated hostname (in our example: `synapse.example.com`).

 ## SRV DNS record delegation

-It is also possible to do delegation using a SRV DNS record. However, that is generally
-not recommended, as it can be difficult to configure the TLS certificates correctly in
-this case, and it offers little advantage over `.well-known` delegation.
+It is also possible to do delegation using a SRV DNS record. However, that is
+considered an advanced topic since it's a bit complex to set up, and `.well-known`
+delegation is already enough in most cases.

-However, if you really need it, you can find some documentation on what such a
+However, if you really need it, you can find some documentation on how such a
 record should look like and how Synapse will use it in [the Matrix
 specification](https://matrix.org/docs/spec/server_server/latest#resolving-server-names).

@@ -94,9 +68,27 @@ wouldn't need any delegation set up.
 domain `server_name` points to, you will need to let other servers know how to
 find it using delegation.

-### Should I use a reverse proxy for federation traffic?
+### Do you still recommend against using a reverse proxy on the federation port?

-Generally, using a reverse proxy for both the federation and client traffic is a good
-idea, since it saves handling TLS traffic in Synapse. See
-[the reverse proxy documentation](reverse_proxy.md) for information on setting up a
+We no longer actively recommend against using a reverse proxy. Many admins will
+find it easier to direct federation traffic to a reverse proxy and manage their
+own TLS certificates, and this is a supported configuration.
+
+See [the reverse proxy documentation](reverse_proxy.md) for information on setting up a
 reverse proxy.
+
+### Do I still need to give my TLS certificates to Synapse if I am using a reverse proxy?
+
+This is no longer necessary. If you are using a reverse proxy for all of your
+TLS traffic, then you can set `no_tls: True` in the Synapse config.
+
+In that case, the only reason Synapse needs the certificate is to populate a legacy
+`tls_fingerprints` field in the federation API. This is ignored by Synapse 0.99.0
+and later, and the only time pre-0.99 Synapses will check it is when attempting to
+fetch the server keys - and generally this is delegated via `matrix.org`, which
+is running a modern version of Synapse.
+
+### Do I need the same certificate for the client and federation port?
+
+No. There is nothing stopping you from using different certificates,
+particularly if you are using a reverse proxy.
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Andrew Morgan	95a49727a6	Note that AS interest via room ID or alias isn't respected Typing and read receipt events (those that are associated with a room) should be sent to an AS that has registered interest in that room, according to MSC2409: https://github.com/Sorunome/matrix-doc/blob/231084da132af6518ecefa2a42c37937dfe373d2/proposals/2409-appservice-edus.md#expectations-of-when-an-edu-should-be-pushed-to-an-appservice	2021-10-20 18:23:56 +01:00
Andrew Morgan	1df714c21c	newsfile	2021-10-20 18:04:44 +01:00
Andrew Morgan	e423a94b28	Improve docstring for format_user_presence_state	2021-10-20 17:58:53 +01:00
Andrew Morgan	0ce33e58fc	Add comments to _handle* methods and those methods they call	2021-10-20 17:58:51 +01:00
Andrew Morgan	2fd99fefed	Add comments to notify_interested_services_ephemeral	2021-10-20 17:42:43 +01:00
Andrew Morgan	1e5a0f2fea	Add docstring to _notify_app_services_ephemeral	2021-10-20 17:11:45 +01:00
Andrew Morgan	795d0584f6	Add docstring to on_new_event	2021-10-20 17:11:45 +01:00
				`@@ -0,0 +1 @@`
				`Port the Password Auth Providers module interface to the new generic interface.`
				`@@ -0,0 +1 @@`
				Add type hints to `synapse.storage.databases.main.client_ips`.
				`@@ -0,0 +1 @@`
				Resolve and share `state_groups` for all [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) historical events in batch.
				`@@ -0,0 +1 @@`
				Fix spurious warnings about losing the logging context on the `ReplicationCommandHandler` when losing the replication connection.
				`@@ -0,0 +1 @@`
				`Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state.`
				`@@ -0,0 +1 @@`
				`Include rejected status when we log events.`
				`@@ -0,0 +1 @@`
				`Add some extra logging to the event persistence code.`
				`@@ -0,0 +1 @@`
				`Add support for Ubuntu 21.10 "Impish Indri".`
				`@@ -0,0 +1 @@`
				Fix 500 error on `/messages` when the server accumulates more than 5 backwards extremities at a given depth for a room.