Changelog

Fixup broken links
``` dmr on titan in synapse/book on  dmr/docs-tidy via  v16.14.0 via 🐍 v3.10.7 (matrix-synapse-py3.10) via 🐏 12GiB/15GiB | 5GiB/8GiB took 8s 2022-10-24 15:15:53 ✔ $ linkchecker **.html INFO linkcheck.cmdline 2022-10-24 15:16:33,785 MainThread Checking intern URLs only; use --check-extern to check extern URLs. LinkChecker 10.1.0 Copyright (C) 2000-2016 Bastian Kleineidam, 2010-2021 LinkChecker Authors LinkChecker comes with ABSOLUTELY NO WARRANTY! This is free software, and you are welcome to redistribute it under certain conditions. Look at the file `LICENSE' within this distribution. Read the documentation at https://linkchecker.github.io/linkchecker/ Write comments and bugs to https://github.com/linkchecker/linkchecker/issues Start checking at 2022-10-24 15:16:33+001 /usr/lib/python3.10/site-packages/bs4/__init__.py:435: MarkupResemblesLocatorWarning: The input looks more like a filename than markup. You may want to open this file and pass the filehandle into Beautiful Soup. warnings.warn( 10 threads active, 88 links queued, 241 links in 339 URLs checked, runtime 1 seconds Statistics: Downloaded: 4.04MB. Content types: 7 image, 106 text, 0 video, 0 audio, 22 application, 3 mail and 508 other. URL lengths: min=16, max=256, avg=67. That's it. 646 links in 646 URLs checked. 0 warnings found. 0 errors found. Stopped checking at 2022-10-24 15:16:39+001 (5 seconds) ```
2022-10-24 16:17:37 +01:00 · 2022-10-24 15:16:45 +01:00 · 2022-10-24 14:50:40 +01:00 · 2022-10-24 14:14:10 +01:00 · 2022-10-24 14:12:59 +01:00 · 2022-10-24 14:12:59 +01:00
289 changed files with 6054 additions and 3013 deletions
@@ -18,6 +18,13 @@
 import json
 import os

+
+def set_output(key: str, value: str):
+    # See https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#setting-an-output-parameter
+    with open(os.environ["GITHUB_OUTPUT"], "at") as f:
+        print(f"{key}={value}", file=f)
+
+
 IS_PR = os.environ["GITHUB_REF"].startswith("refs/pull/")

 # First calculate the various trial jobs.
@@ -81,7 +88,7 @@ print("::endgroup::")
 test_matrix = json.dumps(
    trial_sqlite_tests + trial_postgres_tests + trial_no_extra_tests
 )
-print(f"::set-output name=trial_test_matrix::{test_matrix}")
+set_output("trial_test_matrix", test_matrix)


 # First calculate the various sytest jobs.
@@ -125,4 +132,4 @@ print(json.dumps(sytest_tests, indent=4))
 print("::endgroup::")

 test_matrix = json.dumps(sytest_tests)
-print(f"::set-output name=sytest_test_matrix::{test_matrix}")
+set_output("sytest_test_matrix", test_matrix)
@@ -21,7 +21,7 @@ endblock

 block Install Complement Dependencies
  sudo apt-get -qq update && sudo apt-get install -qqy libolm3 libolm-dev
-  go get -v github.com/haveyoudebuggedit/gotestfmt/v2/cmd/gotestfmt@latest
+  go get -v github.com/gotesttools/gotestfmt/v2/cmd/gotestfmt@latest
 endblock

 block Install custom gotestfmt template
@@ -8,4 +8,11 @@
 #  E203: whitespace before ':' (which is contrary to pep8?)
 #  E731: do not assign a lambda expression, use a def
 #  E501: Line too long (black enforces this for us)
-ignore=W503,W504,E203,E731,E501
+#
+# flake8-bugbear runs extra checks. Its error codes are described at
+# https://github.com/PyCQA/flake8-bugbear#list-of-warnings
+#  B019: Use of functools.lru_cache or functools.cache on methods can lead to memory leaks
+#  B023: Functions defined inside a loop must not use variables redefined in the loop
+#  B024: Abstract base class with no abstract method.
+
+ignore=W503,W504,E203,E731,E501,B019,B023,B024
@@ -15,3 +15,8 @@ updates:
    directory: "/"
    schedule:
      interval: "weekly"
+
+  - package-ecosystem: "cargo"
+    directory: "/"
+    schedule:
+      interval: "weekly"
@@ -27,7 +27,7 @@ jobs:

      - name: Inspect builder
        run: docker buildx inspect
-          
+
      - name: Log in to DockerHub
        uses: docker/login-action@v2
        with:
@@ -55,3 +55,8 @@ jobs:
          tags: "${{ steps.set-tag.outputs.tags }}"
          file: "docker/Dockerfile"
          platforms: linux/amd64,linux/arm64
+
+          # arm64 builds OOM without the git fetch setting. c.f.
+          # https://github.com/rust-lang/cargo/issues/10583
+          build-args: |
+            CARGO_NET_GIT_FETCH_WITH_CLI=true
@@ -20,7 +20,7 @@ jobs:
      - uses: actions/checkout@v3

      - name: Setup mdbook
-        uses: peaceiris/actions-mdbook@4b5ef36b314c2599664ca107bb8c02412548d79d # v1.1.14
+        uses: peaceiris/actions-mdbook@adeb05db28a0c0004681db83893d56c0388ea9ea # v1.2.0
        with:
          mdbook-version: '0.4.17'

@@ -54,11 +54,11 @@ jobs:
          esac

          # finally, set the 'branch-version' var.
-          echo "::set-output name=branch-version::$branch"
+          echo "branch-version=$branch" >> "$GITHUB_OUTPUT"
          
      # Deploy to the target directory.
      - name: Deploy to gh pages
-        uses: peaceiris/actions-gh-pages@068dc23d9710f1ba62e86896f84735d869951305 # v3.8.0
+        uses: peaceiris/actions-gh-pages@de7ea6f8efb354206b205ef54722213d99067935 # v3.9.0
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: ./book
@@ -76,7 +76,7 @@ jobs:
            -e POSTGRES_PASSWORD=postgres \
            -e POSTGRES_INITDB_ARGS="--lc-collate C --lc-ctype C --encoding UTF8" \
            postgres:${{ matrix.postgres-version }}
-      - uses: actions/setup-python@v2
+      - uses: actions/setup-python@v4
        with:
          python-version: "3.x"
      - run: pip install .[all,test]
@@ -26,7 +26,9 @@ jobs:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
-      - uses: actions/setup-python@v2
+      - uses: actions/setup-python@v4
+        with:
+          python-version: '3.x'
      - id: set-distros
        run: |
          # if we're running from a tag, get the full list of distros; otherwise just use debian:sid
@@ -34,7 +36,7 @@ jobs:
          if [[ $GITHUB_REF == refs/tags/* ]]; then
              dists=$(scripts-dev/build_debian_packages.py --show-dists-json)
          fi
-          echo "::set-output name=distros::$dists"
+          echo "distros=$dists" >> "$GITHUB_OUTPUT"
    # map the step outputs to job outputs
    outputs:
      distros: ${{ steps.set-distros.outputs.distros }}
@@ -69,7 +71,9 @@ jobs:
            ${{ runner.os }}-buildx-

      - name: Set up python
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.x'

      - name: Build the packages
        # see https://github.com/docker/build-push-action/issues/252
@@ -91,32 +95,53 @@ jobs:
          path: debs/*

  build-wheels:
-    name: Build wheels on ${{ matrix.os }}
+    name: Build wheels on ${{ matrix.os }} for ${{ matrix.arch }}
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
-        os: [ubuntu-20.04, macos-10.15]
+        os: [ubuntu-20.04, macos-11]
+        arch: [x86_64, aarch64]
+        # is_pr is a flag used to exclude certain jobs from the matrix on PRs.
+        # It is not read by the rest of the workflow.
        is_pr:
          - ${{ startsWith(github.ref, 'refs/pull/') }}

        exclude:
          # Don't build macos wheels on PR CI.
          - is_pr: true
-            os: "macos-10.15"
+            os: "macos-11"
+          # Don't build aarch64 wheels on mac.
+          - os: "macos-11"
+            arch: aarch64
+          # Don't build aarch64 wheels on PR CI.
+          - is_pr: true
+            arch: aarch64

    steps:
      - uses: actions/checkout@v3

-      - uses: actions/setup-python@v3
+      - uses: actions/setup-python@v4
+        with:
+          # setup-python@v4 doesn't impose a default python version. Need to use 3.x
+          # here, because `python` on osx points to Python 2.7.
+          python-version: "3.x"

      - name: Install cibuildwheel
        run: python -m pip install cibuildwheel==2.9.0 poetry==1.2.0

-      # Only build a single wheel in CI.
-      - name: Set env vars.
-        run: |
-          echo "CIBW_BUILD="cp37-manylinux_x86_64"" >> $GITHUB_ENV
+      - name: Set up QEMU to emulate aarch64
+        if: matrix.arch == 'aarch64'
+        uses: docker/setup-qemu-action@v2
+        with:
+          platforms: arm64
+
+      - name: Build aarch64 wheels
+        if: matrix.arch == 'aarch64'
+        run: echo 'CIBW_ARCHS_LINUX=aarch64' >> $GITHUB_ENV
+
+      - name: Only build a single wheel on PR
        if: startsWith(github.ref, 'refs/pull/')
+        run: echo "CIBW_BUILD="cp37-manylinux_${{ matrix.arch }}"" >> $GITHUB_ENV

      - name: Build wheels
        run: python -m cibuildwheel --output-dir wheelhouse
@@ -124,6 +149,9 @@ jobs:
          # Skip testing for platforms which various libraries don't have wheels
          # for, and so need extra build deps.
          CIBW_TEST_SKIP: pp39-* *i686* *musl* pp37-macosx*
+          # Fix Rust OOM errors on emulated aarch64: https://github.com/rust-lang/cargo/issues/10583
+          CARGO_NET_GIT_FETCH_WITH_CLI: true
+          CIBW_ENVIRONMENT_PASS_LINUX: CARGO_NET_GIT_FETCH_WITH_CLI

      - uses: actions/upload-artifact@v3
        with:
@@ -32,7 +32,7 @@ jobs:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
-      - uses: actions/setup-python@v2
+      - uses: actions/setup-python@v4
      - uses: matrix-org/setup-python-poetry@v1
        with:
          extras: "all"
@@ -43,7 +43,7 @@ jobs:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
-      - uses: actions/setup-python@v2
+      - uses: actions/setup-python@v4
      - run: "pip install 'click==8.1.1' 'GitPython>=3.1.20'"
      - run: scripts-dev/check_schema_delta.py --force-colors

@@ -67,7 +67,7 @@ jobs:
        with:
          ref: ${{ github.event.pull_request.head.sha }}
          fetch-depth: 0
-      - uses: actions/setup-python@v2
+      - uses: actions/setup-python@v4
      - run: "pip install 'towncrier>=18.6.0rc1'"
      - run: scripts-dev/check-newsfragment.sh
        env:
@@ -142,7 +142,7 @@ jobs:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
-      - uses: actions/setup-python@v2
+      - uses: actions/setup-python@v4
      - id: get-matrix
        run: .ci/scripts/calculate_jobs.py
    outputs:
@@ -5,24 +5,11 @@ on:
    types: [ opened ]

 jobs:
-  add_new_issues:
-    name: Add new issues to the triage board
-    runs-on: ubuntu-latest
-    steps:
-      - uses: octokit/graphql-action@v2.x
-        id: add_to_project
-        with:
-          headers: '{"GraphQL-Features": "projects_next_graphql"}'
-          query: |
-              mutation add_to_project($projectid:ID!,$contentid:ID!) {
-                addProjectV2ItemById(input: {projectId: $projectid contentId: $contentid}) {
-                item {
-                  id
-                  }
-                }
-              }
-          projectid: ${{ env.PROJECT_ID }}
-          contentid: ${{ github.event.issue.node_id }}
-        env:
-          PROJECT_ID: "PVT_kwDOAIB0Bs4AFDdZ"
-          GITHUB_TOKEN: ${{ secrets.ELEMENT_BOT_TOKEN }}
+  triage:
+    uses: matrix-org/backend-meta/.github/workflows/triage-incoming.yml@v1
+    with: 
+      project_id: 'PVT_kwDOAIB0Bs4AFDdZ'
+      content_id: ${{ github.event.issue.node_id }}
+    secrets: 
+      github_access_token: ${{ secrets.ELEMENT_BOT_TOKEN }}
+          
@@ -1,11 +1,128 @@
-Synapse 1.69.0rc3 (2022-10-12)
+Synapse 1.70.0rc1 (2022-10-19)
 ==============================

+Features
+--------
+
+- Support for [MSC3856](https://github.com/matrix-org/matrix-spec-proposals/pull/3856): threads list API. ([\#13394](https://github.com/matrix-org/synapse/issues/13394), [\#14171](https://github.com/matrix-org/synapse/issues/14171), [\#14175](https://github.com/matrix-org/synapse/issues/14175))
+- Support for thread-specific notifications & receipts ([MSC3771](https://github.com/matrix-org/matrix-spec-proposals/pull/3771) and [MSC3773](https://github.com/matrix-org/matrix-spec-proposals/pull/3773)). ([\#13776](https://github.com/matrix-org/synapse/issues/13776), [\#13824](https://github.com/matrix-org/synapse/issues/13824), [\#13877](https://github.com/matrix-org/synapse/issues/13877), [\#13878](https://github.com/matrix-org/synapse/issues/13878), [\#14050](https://github.com/matrix-org/synapse/issues/14050), [\#14140](https://github.com/matrix-org/synapse/issues/14140), [\#14159](https://github.com/matrix-org/synapse/issues/14159), [\#14163](https://github.com/matrix-org/synapse/issues/14163), [\#14174](https://github.com/matrix-org/synapse/issues/14174), [\#14222](https://github.com/matrix-org/synapse/issues/14222))
+- Stop fetching missing `prev_events` after we already know their signature is invalid. ([\#13816](https://github.com/matrix-org/synapse/issues/13816))
+- Send application service access tokens as a header (and query parameter). Implements [MSC2832](https://github.com/matrix-org/matrix-spec-proposals/pull/2832). ([\#13996](https://github.com/matrix-org/synapse/issues/13996))
+- Ignore server ACL changes when generating pushes. Implements [MSC3786](https://github.com/matrix-org/matrix-spec-proposals/pull/3786). ([\#13997](https://github.com/matrix-org/synapse/issues/13997))
+- Experimental support for redirecting to an implementation of a [MSC3886](https://github.com/matrix-org/matrix-spec-proposals/pull/3886) HTTP rendezvous service. ([\#14018](https://github.com/matrix-org/synapse/issues/14018))
+- The `/relations` endpoint can now be used on workers. ([\#14028](https://github.com/matrix-org/synapse/issues/14028))
+- Advertise support for Matrix 1.3 and 1.4 on `/_matrix/client/versions`. ([\#14032](https://github.com/matrix-org/synapse/issues/14032), [\#14184](https://github.com/matrix-org/synapse/issues/14184))
+- Improve validation of request bodies for the [Device Management](https://spec.matrix.org/v1.4/client-server-api/#device-management) and [MSC2697 Device Dehyrdation](https://github.com/matrix-org/matrix-spec-proposals/pull/2697) client-server API endpoints. ([\#14054](https://github.com/matrix-org/synapse/issues/14054))
+- Experimental support for [MSC3874](https://github.com/matrix-org/matrix-spec-proposals/pull/3874). ([\#14148](https://github.com/matrix-org/synapse/issues/14148))
+- Improve the validation of the following PUT endpoints: [`/directory/room/{roomAlias}`](https://spec.matrix.org/v1.4/client-server-api/#put_matrixclientv3directoryroomroomalias), [`/directory/list/room/{roomId}`](https://spec.matrix.org/v1.4/client-server-api/#put_matrixclientv3directorylistroomroomid) and [`/directory/list/appservice/{networkId}/{roomId}`](https://spec.matrix.org/v1.4/application-service-api/#put_matrixclientv3directorylistappservicenetworkidroomid). ([\#14179](https://github.com/matrix-org/synapse/issues/14179))
+- Build and publish binary wheels for `aarch64` platforms. ([\#14212](https://github.com/matrix-org/synapse/issues/14212))
+
+
+Bugfixes
+--------
+
+- Prevent device names from appearing in device list updates when `allow_device_name_lookup_over_federation` is `false`. ([\#10015](https://github.com/matrix-org/synapse/issues/10015))
+- Fix a long-standing bug where redactions were not being sent over federation if we did not have the original event. ([\#13813](https://github.com/matrix-org/synapse/issues/13813))
+- Fix a long-standing bug where edits of non-`m.room.message` events would not be correctly bundled or have their new content applied. ([\#14034](https://github.com/matrix-org/synapse/issues/14034))
+- Fix a bug introduced in Synapse 1.53.0 when querying `/publicRooms` with both a `room_type` filter and a `third_party_instance_id`. ([\#14053](https://github.com/matrix-org/synapse/issues/14053))
+- Fix a bug introduced in Synapse 1.35.0 where errors parsing a `/send_join` or `/state` response would produce excessive, low-quality Sentry events. ([\#14065](https://github.com/matrix-org/synapse/issues/14065))
+- Fix a long-standing bug where Synapse would error on the optional 'invite_room_state' field not being provided to [`PUT /_matrix/federation/v2/invite/{roomId}/{eventId}`](https://spec.matrix.org/v1.4/server-server-api/#put_matrixfederationv2inviteroomideventid). ([\#14083](https://github.com/matrix-org/synapse/issues/14083))
+- Fix a bug where invalid oEmbed fields would cause the entire response to be discarded. Introduced in Synapse 1.18.0. ([\#14089](https://github.com/matrix-org/synapse/issues/14089))
+- Fix a bug introduced in Synapse 1.37.0 in which an incorrect key name was used for sending and receiving room metadata when knocking on a room. ([\#14102](https://github.com/matrix-org/synapse/issues/14102))
+- Fix a bug introduced in v1.69.0rc1 where the joined hosts for a given event were not being properly cached. ([\#14125](https://github.com/matrix-org/synapse/issues/14125))
+- Fix a bug introduced in Synapse 1.30.0 where purging and rejoining a room without restarting in-between would result in a broken room. ([\#14161](https://github.com/matrix-org/synapse/issues/14161), [\#14164](https://github.com/matrix-org/synapse/issues/14164))
+- Fix [MSC3030](https://github.com/matrix-org/matrix-spec-proposals/pull/3030) `/timestamp_to_event` endpoint returning potentially inaccurate closest events with `outliers` present. ([\#14215](https://github.com/matrix-org/synapse/issues/14215))
+
+
+Updates to the Docker image
+---------------------------
+
+- Update the version of frozendict in Docker images and Debian packages from 2.3.3 to 2.3.4, which may fix memory leak problems. ([\#13955](https://github.com/matrix-org/synapse/issues/13955))
+- Use the `minimal` Rust profile when building Synapse. ([\#14141](https://github.com/matrix-org/synapse/issues/14141))
+- Prevent a class of database sharding errors when using `Dockerfile-workers` to spawn multiple instances of the same worker. Contributed by Jason Little. ([\#14165](https://github.com/matrix-org/synapse/issues/14165))
+- Set `LD_PRELOAD` to use jemalloc memory allocator in Dockerfile-workers. ([\#14182](https://github.com/matrix-org/synapse/issues/14182))
+- Fix pre-startup logging being lost when using the `Dockerfile-workers` image. ([\#14195](https://github.com/matrix-org/synapse/issues/14195))
+
+
+Improved Documentation
+----------------------
+
+- Add sample worker files for `pusher` and `federation_sender`. ([\#14077](https://github.com/matrix-org/synapse/issues/14077))
+- Improve the listener example on the metrics documentation. ([\#14078](https://github.com/matrix-org/synapse/issues/14078))
+- Expand Google OpenID Connect example config to map email attribute. Contributed by @ptman. ([\#14081](https://github.com/matrix-org/synapse/issues/14081))
+- The changelog entry ending in a full stop or exclamation mark is not optional. ([\#14087](https://github.com/matrix-org/synapse/issues/14087))
+- Fix links to jemalloc documentation, which were broken in [#13491](https://github.com/matrix-org/synapse/pull/14124). ([\#14093](https://github.com/matrix-org/synapse/issues/14093))
+- Remove not needed `replication` listener in docker compose example. ([\#14107](https://github.com/matrix-org/synapse/issues/14107))
+- Fix name of `alias_creation_rules` option in the config manual documentation. ([\#14124](https://github.com/matrix-org/synapse/issues/14124))
+- Clarify comment on event contexts. ([\#14145](https://github.com/matrix-org/synapse/issues/14145))
+- Fix dead link to the [Admin Registration API](https://matrix-org.github.io/synapse/latest/admin_api/register_api.html). ([\#14189](https://github.com/matrix-org/synapse/issues/14189))
+
+
+Deprecations and Removals
+-------------------------
+
+- Remove the experimental implementation of [MSC3772](https://github.com/matrix-org/matrix-spec-proposals/pull/3772). ([\#14094](https://github.com/matrix-org/synapse/issues/14094))
+- Remove the unstable identifier for [MSC3715](https://github.com/matrix-org/matrix-doc/pull/3715). ([\#14106](https://github.com/matrix-org/synapse/issues/14106), [\#14146](https://github.com/matrix-org/synapse/issues/14146))
+
+
+Internal Changes
+----------------
+
+- Optimise queries used to get a users rooms during sync. Contributed by Nick @ Beeper (@fizzadar). ([\#13991](https://github.com/matrix-org/synapse/issues/13991))
+- Update authlib from 0.15.5 to 1.1.0. ([\#14006](https://github.com/matrix-org/synapse/issues/14006))
+- Make `parse_server_name` consistent in handling invalid server names. ([\#14007](https://github.com/matrix-org/synapse/issues/14007))
+- Don't repeatedly wake up the same users for batched events. ([\#14033](https://github.com/matrix-org/synapse/issues/14033))
+- Complement test image: capture logs from nginx. ([\#14063](https://github.com/matrix-org/synapse/issues/14063))
+- Don't create noisy Sentry events when a requester drops connection to the metrics server mid-request. ([\#14072](https://github.com/matrix-org/synapse/issues/14072))
+- Run the integration test suites with the asyncio reactor enabled in CI. ([\#14092](https://github.com/matrix-org/synapse/issues/14092))
+- Add debug logs to figure out why an event was filtered out of the client response. ([\#14095](https://github.com/matrix-org/synapse/issues/14095))
+- Indicate what endpoint came back with a JSON response we were unable to parse. ([\#14097](https://github.com/matrix-org/synapse/issues/14097))
+- Break up calls to fetch rooms for many users. Contributed by Nick @ Beeper (@fizzadar). ([\#14109](https://github.com/matrix-org/synapse/issues/14109))
+- Faster joins: prioritise the server we joined by when restarting a partial join resync. ([\#14126](https://github.com/matrix-org/synapse/issues/14126))
+- Cache Rust build cache when building docker images. ([\#14130](https://github.com/matrix-org/synapse/issues/14130))
+- Enable dependabot for Rust dependencies. ([\#14132](https://github.com/matrix-org/synapse/issues/14132))
+- Bump typing-extensions from 4.1.1 to 4.4.0. ([\#14134](https://github.com/matrix-org/synapse/issues/14134))
+- Use the `minimal` Rust profile when building Synapse. ([\#14141](https://github.com/matrix-org/synapse/issues/14141))
+- Remove unused configuration code. ([\#14142](https://github.com/matrix-org/synapse/issues/14142))
+- Prepare for the [`gotestfmt` repository move](https://github.com/GoTestTools/gotestfmt/discussions/46). ([\#14144](https://github.com/matrix-org/synapse/issues/14144))
+- Invalidate rooms for user caches on replicated event, fix sync cache race in synapse workers. Contributed by Nick @ Beeper (@fizzadar). ([\#14155](https://github.com/matrix-org/synapse/issues/14155))
+- Enable url previews when testing with complement. ([\#14198](https://github.com/matrix-org/synapse/issues/14198))
+- When authenticating batched events, check for auth events in batch as well as DB. ([\#14214](https://github.com/matrix-org/synapse/issues/14214))
+- Update CI config to avoid GitHub Actions deprecation warnings. ([\#14216](https://github.com/matrix-org/synapse/issues/14216), [\#14224](https://github.com/matrix-org/synapse/issues/14224))
+- Update dependency requirements to allow building with poetry-core 1.3.2. ([\#14217](https://github.com/matrix-org/synapse/issues/14217))
+- Rename the `cache_memory` extra to `cache-memory`, for compatability with poetry-core 1.3.0 and [PEP 685](https://peps.python.org/pep-0685/). From-source installations using this extra will need to install using the new name. ([\#14221](https://github.com/matrix-org/synapse/issues/14221))
+- Specify dev-dependencies using lower bounds, to reduce the likelihood of a dependabot merge conflict. The lockfile continues to pin to specific versions. ([\#14227](https://github.com/matrix-org/synapse/issues/14227))
+
+
+Synapse 1.69.0 (2022-10-17)
+===========================
+
 Please note that legacy Prometheus metric names are now deprecated and will be removed in Synapse 1.73.0.
 Server administrators should update their dashboards and alerting rules to avoid using the deprecated metric names.
 See the [upgrade notes](https://matrix-org.github.io/synapse/v1.69/upgrade.html#upgrading-to-v1690) for more details.


+No significant changes since 1.69.0rc4.
+
+
+Synapse 1.69.0rc4 (2022-10-14)
+==============================
+
+Bugfixes
+--------
+
+- Fix poor performance of the `event_push_backfill_thread_id` background update, which was introduced in Synapse 1.68.0rc1. ([\#14172](https://github.com/matrix-org/synapse/issues/14172), [\#14181](https://github.com/matrix-org/synapse/issues/14181))
+
+
+Updates to the Docker image
+---------------------------
+
+- Fix docker build OOMing in CI for arm64 builds. ([\#14173](https://github.com/matrix-org/synapse/issues/14173))
+
+
+Synapse 1.69.0rc3 (2022-10-12)
+==============================
+
 Bugfixes
 --------

@@ -13,9 +13,9 @@ dependencies = [

 [[package]]
 name = "anyhow"
-version = "1.0.65"
+version = "1.0.66"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "98161a4e3e2184da77bb14f02184cdd111e83bbbcc9979dfee3c44b9a85f5602"
+checksum = "216261ddc8289130e551ddcd5ce8a064710c0d064a4d2895c67151c92b5443f6"

 [[package]]
 name = "arc-swap"
@@ -104,9 +104,9 @@ checksum = "adab1eaa3408fb7f0c777a73e7465fd5656136fc93b670eb6df3c88c2c1344e3"

 [[package]]
 name = "itoa"
-version = "1.0.3"
+version = "1.0.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6c8af84674fe1f223a982c933a0ee1086ac4d4052aa0fb8060c12c6ad838e754"
+checksum = "4217ad341ebadf8d8e724e264f13e593e0648f5b3e94b3896a5df283be015ecc"

 [[package]]
 name = "lazy_static"
@@ -116,15 +116,15 @@ checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646"

 [[package]]
 name = "libc"
-version = "0.2.132"
+version = "0.2.135"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8371e4e5341c3a96db127eb2465ac681ced4c433e01dd0e938adbef26ba93ba5"
+checksum = "68783febc7782c6c5cb401fbda4de5a9898be1762314da0bb2c10ced61f18b0c"

 [[package]]
 name = "lock_api"
-version = "0.4.7"
+version = "0.4.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "327fa5b6a6940e4699ec49a9beae1ea4845c6bab9314e4f84ac68742139d8c53"
+checksum = "435011366fe56583b16cf956f9df0095b405b82d76425bc8981c0e22e60ec4df"
 dependencies = [
 "autocfg",
 "scopeguard",
@@ -156,9 +156,9 @@ dependencies = [

 [[package]]
 name = "once_cell"
-version = "1.13.1"
+version = "1.15.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "074864da206b4973b84eb91683020dbefd6a8c3f0f38e054d93954e891935e4e"
+checksum = "e82dad04139b71a90c080c8463fe0dc7902db5192d939bd0950f074d014339e1"

 [[package]]
 name = "parking_lot"
@@ -185,18 +185,18 @@ dependencies = [

 [[package]]
 name = "proc-macro2"
-version = "1.0.43"
+version = "1.0.46"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0a2ca2c61bc9f3d74d2886294ab7b9853abd9c1ad903a3ac7815c58989bb7bab"
+checksum = "94e2ef8dbfc347b10c094890f778ee2e36ca9bb4262e86dc99cd217e35f3470b"
 dependencies = [
 "unicode-ident",
 ]

 [[package]]
 name = "pyo3"
-version = "0.17.1"
+version = "0.17.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "12f72538a0230791398a0986a6518ebd88abc3fded89007b506ed072acc831e1"
+checksum = "201b6887e5576bf2f945fe65172c1fcbf3fcf285b23e4d71eb171d9736e38d32"
 dependencies = [
 "anyhow",
 "cfg-if",
@@ -212,9 +212,9 @@ dependencies = [

 [[package]]
 name = "pyo3-build-config"
-version = "0.17.1"
+version = "0.17.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fc4cf18c20f4f09995f3554e6bcf9b09bd5e4d6b67c562fdfaafa644526ba479"
+checksum = "bf0708c9ed01692635cbf056e286008e5a2927ab1a5e48cdd3aeb1ba5a6fef47"
 dependencies = [
 "once_cell",
 "target-lexicon",
@@ -222,9 +222,9 @@ dependencies = [

 [[package]]
 name = "pyo3-ffi"
-version = "0.17.1"
+version = "0.17.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a41877f28d8ebd600b6aa21a17b40c3b0fc4dfe73a27b6e81ab3d895e401b0e9"
+checksum = "90352dea4f486932b72ddf776264d293f85b79a1d214de1d023927b41461132d"
 dependencies = [
 "libc",
 "pyo3-build-config",
@@ -243,9 +243,9 @@ dependencies = [

 [[package]]
 name = "pyo3-macros"
-version = "0.17.1"
+version = "0.17.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2e81c8d4bcc2f216dc1b665412df35e46d12ee8d3d046b381aad05f1fcf30547"
+checksum = "7eb24b804a2d9e88bfcc480a5a6dd76f006c1e3edaf064e8250423336e2cd79d"
 dependencies = [
 "proc-macro2",
 "pyo3-macros-backend",
@@ -255,9 +255,9 @@ dependencies = [

 [[package]]
 name = "pyo3-macros-backend"
-version = "0.17.1"
+version = "0.17.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "85752a767ee19399a78272cc2ab625cd7d373b2e112b4b13db28de71fa892784"
+checksum = "f22bb49f6a7348c253d7ac67a6875f2dc65f36c2ae64a82c381d528972bea6d6"
 dependencies = [
 "proc-macro2",
 "quote",
@@ -323,18 +323,18 @@ checksum = "d29ab0c6d3fc0ee92fe66e2d99f700eab17a8d57d1c1d3b748380fb20baa78cd"

 [[package]]
 name = "serde"
-version = "1.0.145"
+version = "1.0.147"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "728eb6351430bccb993660dfffc5a72f91ccc1295abaa8ce19b27ebe4f75568b"
+checksum = "d193d69bae983fc11a79df82342761dfbf28a99fc8d203dca4c3c1b590948965"
 dependencies = [
 "serde_derive",
 ]

 [[package]]
 name = "serde_derive"
-version = "1.0.145"
+version = "1.0.147"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "81fa1584d3d1bcacd84c277a0dfe21f5b0f6accf4a23d04d4c6d61f1af522b4c"
+checksum = "4f1d362ca8fc9c3e3a7484440752472d68a6caa98f1ab81d99b5dfe517cec852"
 dependencies = [
 "proc-macro2",
 "quote",
@@ -343,9 +343,9 @@ dependencies = [

 [[package]]
 name = "serde_json"
-version = "1.0.85"
+version = "1.0.86"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e55a28e3aaef9d5ce0506d0a14dbba8054ddc7e499ef522dd8b26859ec9d4a44"
+checksum = "41feea4228a6f1cd09ec7a3593a682276702cd67b5273544757dae23c096f074"
 dependencies = [
 "itoa",
 "ryu",
@@ -354,9 +354,9 @@ dependencies = [

 [[package]]
 name = "smallvec"
-version = "1.9.0"
+version = "1.10.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2fd0db749597d91ff862fd1d55ea87f7855a744a8425a64695b6fca237d1dad1"
+checksum = "a507befe795404456341dfab10cef66ead4c041f62b8b11bbb92bffe5d0953e0"

 [[package]]
 name = "subtle"
@@ -366,9 +366,9 @@ checksum = "6bdef32e8150c2a081110b42772ffe7d7c9032b606bc226c8260fd97e0976601"

 [[package]]
 name = "syn"
-version = "1.0.99"
+version = "1.0.102"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "58dbef6ec655055e20b86b15a8cc6d439cca19b667537ac6a1369572d151ab13"
+checksum = "3fcd952facd492f9be3ef0d0b7032a6e442ee9b361d4acc2b1d0c4aaa5f613a1"
 dependencies = [
 "proc-macro2",
 "quote",
@@ -406,9 +406,9 @@ checksum = "dcf81ac59edc17cc8697ff311e8f5ef2d99fcbd9817b34cec66f90b6c3dfd987"

 [[package]]
 name = "unicode-ident"
-version = "1.0.3"
+version = "1.0.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c4f5b37a154999a8f3f98cc23a628d850e154479cd94decf3414696e12e31aaf"
+checksum = "6ceab39d59e4c9499d4e5a8ee0e2735b891bb7308ac83dfb4e80cad195c9f6f3"

 [[package]]
 name = "unindent"
@@ -0,0 +1 @@
+Fix a long-standing bug where the `update_synapse_database` script could not be run with multiple databases. Contributed by @thefinn93 @ Beeper.
@@ -0,0 +1 @@
+Improve aesthetics of HTML templates. Note that these changes do not retroactively apply to templates which have been [customised](https://matrix-org.github.io/synapse/latest/templates.html#templates) by server admins.
@@ -0,0 +1 @@
+Fix a long-standing bug where Synapse would count codepoints instead of bytes when validating the size of some fields.
@@ -0,0 +1 @@
+Bump flake8-bugbear from 21.3.2 to 22.9.23.
@@ -0,0 +1 @@
+ Fix a long-standing bug where Synapse would accidentally include extra information in the response to [`PUT /_matrix/federation/v2/invite/{roomId}/{eventId}`](https://spec.matrix.org/v1.4/server-server-api/#put_matrixfederationv2inviteroomideventid).
@@ -0,0 +1 @@
+Correct the name of the config option [`encryption_enabled_by_default_for_room_type`](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#encryption_enabled_by_default_for_room_type).
@@ -0,0 +1 @@
+Bump types-opentracing from 2.4.7 to 2.4.10.
@@ -0,0 +1 @@
+Update docstrings of `SynapseError` and `FederationError` to bettter describe what they are used for and the effects of using them are.
@@ -0,0 +1 @@
+Show erasure status when listing users in the Admin API.
@@ -0,0 +1 @@
+Add initial power level event to batch of bulk persisted events when creating a new room.
@@ -0,0 +1 @@
+Refactor `/key/` endpoints to use `RestServlet` classes.
@@ -0,0 +1 @@
+Switch to using the `matrix-org/backend-meta` version of `triage-incoming` for new issues in CI.
@@ -0,0 +1 @@
+Build wheels on macos 11, not 10.15.
@@ -0,0 +1,2 @@
+Fix a bug introduced in Synapse 1.60.0 which caused an error to be logged when Synapse received a SIGHUP signal, and debug logging was enabled.
+
@@ -0,0 +1 @@
+Add debugging to help diagnose lost device-list-update.
@@ -0,0 +1 @@
+Bump pysaml2 from 7.1.2 to 7.2.1.
@@ -0,0 +1 @@
+Bump jinja2 from 3.0.3 to 3.1.2.
@@ -0,0 +1 @@
+Bump types-requests from 2.28.11 to 2.28.11.2.
@@ -0,0 +1 @@
+Bump setuptools-rust from 1.5.1 to 1.5.2.
@@ -0,0 +1 @@
+Bump prometheus-client from 0.14.0 to 0.15.0.
@@ -0,0 +1 @@
+Bump peaceiris/actions-mdbook from 1.1.14 to 1.2.0.
@@ -0,0 +1 @@
+Bump peaceiris/actions-gh-pages from 3.8.0 to 3.9.0.
@@ -0,0 +1 @@
+Bump serde from 1.0.145 to 1.0.147.
@@ -0,0 +1 @@
+Bump anyhow from 1.0.65 to 1.0.66.
@@ -0,0 +1 @@
+Reorganise documentation source files and fix dead links.
@@ -94,20 +94,6 @@ worker_replication_host: synapse
 worker_replication_http_port: 9093
 ```

-### Add Workers to `instance_map`
-
-Locate the `instance_map` section of your `homeserver.yaml` and populate it with your workers:
-
-```yaml
-instance_map:
-  synapse-generic-worker-1:        # The worker_name setting in your worker configuration file
-    host: synapse-generic-worker-1 # The name of the worker service in your Docker Compose file
-    port: 8034                     # The port assigned to the replication listener in your worker config file
-  synapse-federation-sender-1:
-    host: synapse-federation-sender-1
-    port: 8034
-```
-
 ### Configure Federation Senders

 This section is applicable if you are using Federation senders (synapse.app.federation_sender). Locate the `send_federation` and `federation_sender_instances` settings in your `homeserver.yaml` and configure them:
@@ -122,4 +108,4 @@ federation_sender_instances:

 ## Other Worker types

-Using the concepts shown here it is possible to create other worker types in Docker Compose. See the [Workers](https://matrix-org.github.io/synapse/latest/workers.html#available-worker-applications) documentation for a list of available workers.
+Using the concepts shown here it is possible to create other worker types in Docker Compose. See the [Workers](https://matrix-org.github.io/synapse/latest/workers.html#available-worker-applications) documentation for a list of available workers.
@@ -5,10 +5,4 @@ worker_name: synapse-federation-sender-1
 worker_replication_host: synapse
 worker_replication_http_port: 9093

-worker_listeners:
-  - type: http
-    port: 8034
-    resources:
-      - names: [replication]
-
 worker_log_config: /data/federation_sender.log.config
@@ -6,10 +6,6 @@ worker_replication_host: synapse
 worker_replication_http_port: 9093

 worker_listeners:
-  - type: http
-    port: 8034
-    resources:
-      - names: [replication]
  - type: http
    port: 8081
    x_forwarded: true
@@ -1,3 +1,21 @@
+matrix-synapse-py3 (1.70.0~rc1) stable; urgency=medium
+
+  * New Synapse release 1.70.0rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Wed, 19 Oct 2022 14:11:57 +0100
+
+matrix-synapse-py3 (1.69.0) stable; urgency=medium
+
+  * New Synapse release 1.69.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Mon, 17 Oct 2022 11:31:03 +0100
+
+matrix-synapse-py3 (1.69.0~rc4) stable; urgency=medium
+
+  * New Synapse release 1.69.0rc4.
+
+ -- Synapse Packaging team <packages@matrix.org>  Fri, 14 Oct 2022 15:04:47 +0100
+
 matrix-synapse-py3 (1.69.0~rc3) stable; urgency=medium

  * New Synapse release 1.69.0rc3.
@@ -106,7 +106,13 @@ ENV CARGO_HOME=/cargo
 ENV PATH=/cargo/bin:/rust/bin:$PATH
 RUN mkdir /rust /cargo

-RUN curl -sSf https://sh.rustup.rs | sh -s -- -y --no-modify-path --default-toolchain stable
+RUN curl -sSf https://sh.rustup.rs | sh -s -- -y --no-modify-path --default-toolchain stable --profile minimal
+
+
+# arm64 builds consume a lot of memory if `CARGO_NET_GIT_FETCH_WITH_CLI` is not
+# set to true, so we expose it as a build-arg.
+ARG CARGO_NET_GIT_FETCH_WITH_CLI=false
+ENV CARGO_NET_GIT_FETCH_WITH_CLI=$CARGO_NET_GIT_FETCH_WITH_CLI

 # To speed up rebuilds, install all of the dependencies before we copy over
 # the whole synapse project, so that this layer in the Docker cache can be
@@ -129,7 +135,9 @@ ARG TEST_ONLY_IGNORE_POETRY_LOCKFILE
 # Install the synapse package itself.
 # If we have populated requirements.txt, we don't install any dependencies
 # as we should already have those from the previous `pip install` step.
-RUN if [ -z "$TEST_ONLY_IGNORE_POETRY_LOCKFILE" ]; then \
+RUN --mount=type=cache,target=/synapse/target,sharing=locked \
+  --mount=type=cache,target=${CARGO_HOME}/registry,sharing=locked \
+  if [ -z "$TEST_ONLY_IGNORE_POETRY_LOCKFILE" ]; then \
    pip install --prefix="/install" --no-deps --no-warn-script-location /synapse[all]; \
  else \
    pip install --prefix="/install" --no-warn-script-location /synapse[all]; \
@@ -92,7 +92,7 @@ ENV CARGO_HOME=/cargo
 ENV PATH=/cargo/bin:/rust/bin:$PATH
 RUN mkdir /rust /cargo

-RUN curl -sSf https://sh.rustup.rs | sh -s -- -y --no-modify-path --default-toolchain stable
+RUN curl -sSf https://sh.rustup.rs | sh -s -- -y --no-modify-path --default-toolchain stable --profile minimal


 COPY --from=builder /dh-virtualenv_1.2.2-1_all.deb /
@@ -40,7 +40,11 @@ FROM matrixdotorg/synapse:$SYNAPSE_VERSION
    COPY --from=deps_base /etc/nginx /etc/nginx
    RUN rm /etc/nginx/sites-enabled/default
    RUN mkdir /var/log/nginx /var/lib/nginx
-    RUN chown www-data /var/log/nginx /var/lib/nginx
+    RUN chown www-data /var/lib/nginx
+
+    # have nginx log to stderr/out
+    RUN ln -sf /dev/stdout /var/log/nginx/access.log
+    RUN ln -sf /dev/stderr /var/log/nginx/error.log

    # Copy Synapse worker, nginx and supervisord configuration template files
    COPY ./docker/conf-workers/* /conf/
@@ -98,7 +98,7 @@ Dockerfile makes use of to generate appropriate worker, nginx and supervisord co
 files.

 Sharding is supported for a subset of workers, in line with the
-[worker documentation](../docs/workers.md). To run multiple instances of a given worker
+[worker documentation](../docs/usage/configuration/workers.md). To run multiple instances of a given worker
 type, simply specify the type multiple times in `SYNAPSE_WORKER_TYPES`
 (e.g `SYNAPSE_WORKER_TYPES=event_creator,event_creator...`).

@@ -241,4 +241,4 @@ healthcheck:

 Jemalloc is embedded in the image and will be used instead of the default allocator.
 You can read about jemalloc by reading the Synapse
-[README](https://github.com/matrix-org/synapse/blob/HEAD/README.rst#help-synapse-is-slow-and-eats-all-my-ram-cpu).
+[Admin FAQ](https://matrix-org.github.io/synapse/latest/usage/administration/admin_faq.html#help-synapse-is-slow-and-eats-all-my-ramcpu).
@@ -57,6 +57,7 @@ if [[ -n "$SYNAPSE_COMPLEMENT_USE_WORKERS" ]]; then
      federation_reader, \
      federation_sender, \
      synchrotron, \
+      client_reader, \
      appservice, \
      pusher"

@@ -12,6 +12,8 @@ trusted_key_servers: []
 enable_registration: true
 enable_registration_without_verification: true
 bcrypt_rounds: 4
+url_preview_enabled: true
+url_preview_ip_range_blacklist: []

 ## Registration ##

@@ -39,6 +39,7 @@
 # continue to work if so.

 import os
+import platform
 import subprocess
 import sys
 from pathlib import Path
@@ -107,6 +108,34 @@ WORKERS_CONFIG: Dict[str, Dict[str, Any]] = {
        "shared_extra_conf": {},
        "worker_extra_conf": "",
    },
+    "client_reader": {
+        "app": "synapse.app.generic_worker",
+        "listener_resources": ["client"],
+        "endpoint_patterns": [
+            "^/_matrix/client/(api/v1|r0|v3|unstable)/publicRooms$",
+            "^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/joined_members$",
+            "^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/context/.*$",
+            "^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/members$",
+            "^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state$",
+            "^/_matrix/client/v1/rooms/.*/hierarchy$",
+            "^/_matrix/client/(v1|unstable)/rooms/.*/relations/",
+            "^/_matrix/client/v1/rooms/.*/threads$",
+            "^/_matrix/client/(api/v1|r0|v3|unstable)/login$",
+            "^/_matrix/client/(api/v1|r0|v3|unstable)/account/3pid$",
+            "^/_matrix/client/(api/v1|r0|v3|unstable)/account/whoami$",
+            "^/_matrix/client/versions$",
+            "^/_matrix/client/(api/v1|r0|v3|unstable)/voip/turnServer$",
+            "^/_matrix/client/(r0|v3|unstable)/register$",
+            "^/_matrix/client/(r0|v3|unstable)/auth/.*/fallback/web$",
+            "^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/messages$",
+            "^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/event",
+            "^/_matrix/client/(api/v1|r0|v3|unstable)/joined_rooms",
+            "^/_matrix/client/(api/v1|r0|v3|unstable/.*)/rooms/.*/aliases",
+            "^/_matrix/client/(api/v1|r0|v3|unstable)/search",
+        ],
+        "shared_extra_conf": {},
+        "worker_extra_conf": "",
+    },
    "federation_reader": {
        "app": "synapse.app.generic_worker",
        "listener_resources": ["federation"],
@@ -201,24 +230,19 @@ upstream {upstream_worker_type} {{

 # Utility functions
 def log(txt: str) -> None:
-    """Log something to the stdout.
-
-    Args:
-        txt: The text to log.
-    """
    print(txt)


 def error(txt: str) -> NoReturn:
-    """Log something and exit with an error code.
-
-    Args:
-        txt: The text to log in error.
-    """
-    log(txt)
+    print(txt, file=sys.stderr)
    sys.exit(2)


+def flush_buffers() -> None:
+    sys.stdout.flush()
+    sys.stderr.flush()
+
+
 def convert(src: str, dst: str, **template_vars: object) -> None:
    """Generate a file from a template

@@ -299,7 +323,7 @@ def generate_base_homeserver_config() -> None:
    # start.py already does this for us, so just call that.
    # note that this script is copied in in the official, monolith dockerfile
    os.environ["SYNAPSE_HTTP_PORT"] = str(MAIN_PROCESS_HTTP_LISTENER_PORT)
-    subprocess.check_output(["/usr/local/bin/python", "/start.py", "migrate_config"])
+    subprocess.run(["/usr/local/bin/python", "/start.py", "migrate_config"], check=True)


 def generate_worker_files(
@@ -373,8 +397,8 @@ def generate_worker_files(
        # No workers, just the main process
        worker_types = []
    else:
-        # Split type names by comma
-        worker_types = worker_types_env.split(",")
+        # Split type names by comma, ignoring whitespace.
+        worker_types = [x.strip() for x in worker_types_env.split(",")]

    # Create the worker configuration directory if it doesn't already exist
    os.makedirs("/conf/workers", exist_ok=True)
@@ -393,8 +417,6 @@ def generate_worker_files(

    # For each worker type specified by the user, create config values
    for worker_type in worker_types:
-        worker_type = worker_type.strip()
-
        worker_config = WORKERS_CONFIG.get(worker_type)
        if worker_config:
            worker_config = worker_config.copy()
@@ -604,14 +626,24 @@ def main(args: List[str], environ: MutableMapping[str, str]) -> None:
        with open(mark_filepath, "w") as f:
            f.write("")

+    # Lifted right out of start.py
+    jemallocpath = "/usr/lib/%s-linux-gnu/libjemalloc.so.2" % (platform.machine(),)
+
+    if os.path.isfile(jemallocpath):
+        environ["LD_PRELOAD"] = jemallocpath
+    else:
+        log("Could not find %s, will not use" % (jemallocpath,))
+
    # Start supervisord, which will start Synapse, all of the configured worker
    # processes, redis, nginx etc. according to the config we created above.
    log("Starting supervisord")
-    os.execl(
+    flush_buffers()
+    os.execle(
        "/usr/local/bin/supervisord",
        "supervisord",
        "-c",
        "/etc/supervisor/supervisord.conf",
+        environ,
    )


@@ -13,14 +13,19 @@ import jinja2

 # Utility functions
 def log(txt: str) -> None:
-    print(txt, file=sys.stderr)
+    print(txt)


 def error(txt: str) -> NoReturn:
-    log(txt)
+    print(txt, file=sys.stderr)
    sys.exit(2)


+def flush_buffers() -> None:
+    sys.stdout.flush()
+    sys.stderr.flush()
+
+
 def convert(src: str, dst: str, environ: Mapping[str, object]) -> None:
    """Generate a file from a template

@@ -131,10 +136,10 @@ def generate_config_from_template(

    if ownership is not None:
        log(f"Setting ownership on /data to {ownership}")
-        subprocess.check_output(["chown", "-R", ownership, "/data"])
+        subprocess.run(["chown", "-R", ownership, "/data"], check=True)
        args = ["gosu", ownership] + args

-    subprocess.check_output(args)
+    subprocess.run(args, check=True)


 def run_generate_config(environ: Mapping[str, str], ownership: Optional[str]) -> None:
@@ -158,7 +163,7 @@ def run_generate_config(environ: Mapping[str, str], ownership: Optional[str]) ->
    if ownership is not None:
        # make sure that synapse has perms to write to the data dir.
        log(f"Setting ownership on {data_dir} to {ownership}")
-        subprocess.check_output(["chown", ownership, data_dir])
+        subprocess.run(["chown", ownership, data_dir], check=True)

    # create a suitable log config from our template
    log_config_file = "%s/%s.log.config" % (config_dir, server_name)
@@ -185,6 +190,7 @@ def run_generate_config(environ: Mapping[str, str], ownership: Optional[str]) ->
        "--open-private-ports",
    ]
    # log("running %s" % (args, ))
+    flush_buffers()
    os.execv(sys.executable, args)


@@ -267,8 +273,10 @@ running with 'migrate_config'. See the README for more details.
    args = [sys.executable] + args
    if ownership is not None:
        args = ["gosu", ownership] + args
+        flush_buffers()
        os.execve("/usr/sbin/gosu", args, environ)
    else:
+        flush_buffers()
        os.execve(sys.executable, args, environ)


@@ -5,38 +5,38 @@

 # Setup
  - [Installation](setup/installation.md)
-  - [Using Postgres](postgres.md)
-  - [Configuring a Reverse Proxy](reverse_proxy.md)
+  - [Using Postgres](setup/postgres.md)
+  - [Configuring a Reverse Proxy](setup/reverse_proxy.md)
  - [Configuring a Forward/Outbound Proxy](setup/forward_proxy.md)
-  - [Configuring a Turn Server](turn-howto.md)
-  - [Delegation](delegate.md)
+  - [Configuring a Turn Server](setup/turn-howto.md)
+  - [Delegation](setup/delegate.md)

 # Upgrading
  - [Upgrading between Synapse Versions](upgrade.md)

 # Usage
-  - [Federation](federate.md)
+  - [Federation](usage/federate.md)
  - [Configuration](usage/configuration/README.md)
    - [Configuration Manual](usage/configuration/config_documentation.md) 
    - [Homeserver Sample Config File](usage/configuration/homeserver_sample_config.md)
    - [Logging Sample Config File](usage/configuration/logging_sample_config.md)
-    - [Structured Logging](structured_logging.md)
-    - [Templates](templates.md)
+    - [Structured Logging](usage/configuration/structured_logging.md)
+    - [Templates](usage/configuration/templates.md)
    - [User Authentication](usage/configuration/user_authentication/README.md)
      - [Single-Sign On](usage/configuration/user_authentication/single_sign_on/README.md)
-        - [OpenID Connect](openid.md)
+        - [OpenID Connect](usage/configuration/user_authentication/single_sign_on/openid.md)
        - [SAML](usage/configuration/user_authentication/single_sign_on/saml.md)
        - [CAS](usage/configuration/user_authentication/single_sign_on/cas.md)
-        - [SSO Mapping Providers](sso_mapping_providers.md)
-      - [Password Auth Providers](password_auth_providers.md)
-      - [JSON Web Tokens](jwt.md)
+        - [SSO Mapping Providers](usage/configuration/user_authentication/single_sign_on/sso_mapping_providers.md)
+      - [Password Auth Providers](usage/configuration/user_authentication/password_auth_providers.md)
+      - [JSON Web Tokens](usage/configuration/user_authentication/jwt.md)
      - [Refresh Tokens](usage/configuration/user_authentication/refresh_tokens.md)
-    - [Registration Captcha](CAPTCHA_SETUP.md)
-    - [Application Services](application_services.md)
-    - [Server Notices](server_notices.md)
-    - [Consent Tracking](consent_tracking.md)
-    - [User Directory](user_directory.md)
-    - [Message Retention Policies](message_retention_policies.md)
+    - [Registration Captcha](usage/configuration/CAPTCHA_SETUP.md)
+    - [Application Services](usage/configuration/application_services.md)
+    - [Server Notices](usage/configuration/server_notices.md)
+    - [Consent Tracking](usage/configuration/consent_tracking.md)
+    - [User Directory](usage/configuration/user_directory.md)
+    - [Message Retention Policies](usage/configuration/message_retention_policies.md)
    - [Pluggable Modules](modules/index.md)
      - [Writing a module](modules/writing_a_module.md)
        - [Spam checker callbacks](modules/spam_checker_callbacks.md)
@@ -47,65 +47,65 @@
        - [Background update controller callbacks](modules/background_update_controller_callbacks.md)
        - [Account data callbacks](modules/account_data_callbacks.md)
        - [Porting a legacy module to the new interface](modules/porting_legacy_module.md)
-    - [Workers](workers.md)
-      - [Using `synctl` with Workers](synctl_workers.md)
-      - [Systemd](systemd-with-workers/README.md)
-  - [Administration](usage/administration/README.md)
-    - [Admin API](usage/administration/admin_api/README.md)
-      - [Account Validity](admin_api/account_validity.md)
-      - [Background Updates](usage/administration/admin_api/background_updates.md)
-      - [Event Reports](admin_api/event_reports.md)
-      - [Media](admin_api/media_admin_api.md)
-      - [Purge History](admin_api/purge_history_api.md)
-      - [Register Users](admin_api/register_api.md)
-      - [Registration Tokens](usage/administration/admin_api/registration_tokens.md)
-      - [Manipulate Room Membership](admin_api/room_membership.md)
-      - [Rooms](admin_api/rooms.md)
-      - [Server Notices](admin_api/server_notices.md)
-      - [Statistics](admin_api/statistics.md)
-      - [Users](admin_api/user_admin_api.md)
-      - [Server Version](admin_api/version_api.md)
-      - [Federation](usage/administration/admin_api/federation.md)
-    - [Manhole](manhole.md)
-    - [Monitoring](metrics-howto.md)
-      - [Reporting Homeserver Usage Statistics](usage/administration/monitoring/reporting_homeserver_usage_statistics.md)
-    - [Monthly Active Users](usage/administration/monthly_active_users.md)
-    - [Understanding Synapse Through Grafana Graphs](usage/administration/understanding_synapse_through_grafana_graphs.md)
-    - [Useful SQL for Admins](usage/administration/useful_sql_for_admins.md)
-    - [Database Maintenance Tools](usage/administration/database_maintenance_tools.md)
-    - [State Groups](usage/administration/state_groups.md)
-    - [Request log format](usage/administration/request_log.md)
-    - [Admin FAQ](usage/administration/admin_faq.md)
+    - [Workers](usage/configuration/workers.md)
+      - [Using `synctl` with Workers](usage/configuration/synctl_workers.md)
+      - [Systemd](usage/configuration/systemd-with-workers/README.md)
+  - [Administration](administration/README.md)
+    - [Admin API](administration/admin_api/README.md)
+      - [Account Validity](administration/admin_api/account_validity.md)
+      - [Background Updates](administration/admin_api/background_updates.md)
+      - [Event Reports](administration/admin_api/event_reports.md)
+      - [Media](administration/admin_api/media_admin_api.md)
+      - [Purge History](administration/admin_api/purge_history_api.md)
+      - [Register Users](administration/admin_api/register_api.md)
+      - [Registration Tokens](administration/admin_api/registration_tokens.md)
+      - [Manipulate Room Membership](administration/admin_api/room_membership.md)
+      - [Rooms](administration/admin_api/rooms.md)
+      - [Server Notices](administration/admin_api/server_notices.md)
+      - [Statistics](administration/admin_api/statistics.md)
+      - [Users](administration/admin_api/user_admin_api.md)
+      - [Server Version](administration/admin_api/version_api.md)
+      - [Federation](administration/admin_api/federation.md)
+    - [Manhole](administration/manhole.md)
+    - [Monitoring](administration/metrics-howto.md)
+      - [Reporting Homeserver Usage Statistics](administration/reporting_homeserver_usage_statistics.md)
+    - [Monthly Active Users](administration/monthly_active_users.md)
+    - [Understanding Synapse Through Grafana Graphs](administration/understanding_synapse_through_grafana_graphs.md)
+    - [Useful SQL for Admins](administration/useful_sql_for_admins.md)
+    - [Database Maintenance Tools](administration/database_maintenance_tools.md)
+    - [State Groups](administration/state_groups.md)
+    - [Request log format](administration/request_log.md)
+    - [Admin FAQ](administration/admin_faq.md)
    - [Scripts]()

 # Development
  - [Contributing Guide](development/contributing_guide.md)
-  - [Code Style](code_style.md)
+  - [Code Style](development/code_style.md)
  - [Reviewing Code](development/reviews.md)
  - [Release Cycle](development/releases.md)
  - [Git Usage](development/git.md)
  - [Testing]()
    - [Demo scripts](development/demo.md)
-  - [OpenTracing](opentracing.md)
+  - [OpenTracing](development/opentracing.md)
  - [Database Schemas](development/database_schema.md)
  - [Experimental features](development/experimental_features.md)
  - [Dependency management](development/dependencies.md)
  - [Synapse Architecture]()
    - [Cancellation](development/synapse_architecture/cancellation.md)
-    - [Log Contexts](log_contexts.md)
-    - [Replication](replication.md)
-    - [TCP Replication](tcp_replication.md)
+    - [Log Contexts](development/synapse_architecture/log_contexts.md)
+    - [Replication](development/synapse_architecture/replication.md)
+    - [TCP Replication](development/synapse_architecture/tcp_replication.md)
  - [Internal Documentation](development/internal_documentation/README.md)
    - [Single Sign-On]()
-      - [SAML](development/saml.md)
-      - [CAS](development/cas.md)
-    - [Room DAG concepts](development/room-dag-concepts.md)
+      - [SAML](development/internal_documentation/saml.md)
+      - [CAS](development/internal_documentation/cas.md)
+    - [Room DAG concepts](development/internal_documentation/room-dag-concepts.md)
    - [State Resolution]()
-      - [The Auth Chain Difference Algorithm](auth_chain_difference_algorithm.md)
-    - [Media Repository](media_repository.md)
-    - [Room and User Statistics](room_and_user_statistics.md)
+      - [The Auth Chain Difference Algorithm](development/internal_documentation/auth_chain_difference_algorithm.md)
+    - [Media Repository](development/internal_documentation/media_repository.md)
+    - [Room and User Statistics](development/internal_documentation/room_and_user_statistics.md)
  - [Scripts]()

 # Other
-  - [Dependency Deprecation Policy](deprecation_policy.md)
+  - [Dependency Deprecation Policy](other/deprecation_policy.md)
  - [Running Synapse on a Single-Board Computer](other/running_synapse_on_single_board_computers.md)
@@ -7,7 +7,11 @@ server admin. (Note that a server admin is distinct from a room admin.)

 An existing user can be marked as a server admin by updating the database directly.

-Check your [database settings](config_documentation.md#database) in the configuration file, connect to the correct database using either `psql [database name]` (if using PostgreSQL) or `sqlite3 path/to/your/database.db` (if using SQLite) and elevate the user `@foo:bar.com` to administrator.
+Check your [database settings](../../usage/configuration/config_documentation.md#database)
+in the configuration file, connect to the correct database using either 
+`psql [database name]` (if using PostgreSQL) or 
+`sqlite3 path/to/your/database.db` (if using SQLite) and elevate the user 
+`@foo:bar.com` to administrator.
 ```sql
 UPDATE users SET admin = 1 WHERE name = '@foo:bar.com';
 ```
@@ -19,7 +23,7 @@ already on your `$PATH` depending on how Synapse was installed.
 Finding your user's `access_token` is client-dependent, but will usually be shown in the client's settings.

 ## Making an Admin API request
-For security reasons, we [recommend](reverse_proxy.md#synapse-administration-endpoints)
+For security reasons, we [recommend](../../setup/reverse_proxy.md#synapse-administration-endpoints)
 that the Admin API (`/_synapse/admin/...`) should be hidden from public view using a
 reverse proxy. This means you should typically query the Admin API from a terminal on
 the machine which runs Synapse.
@@ -35,7 +39,7 @@ For example, suppose we want to
 [query the account](user_admin_api.md#query-user-account) of the user
 `@foo:bar.com`. We need an admin access token (e.g.
 `syt_AjfVef2_L33JNpafeif_0feKJfeaf0CQpoZk`), and we need to know which port
-Synapse's [`client` listener](config_documentation.md#listeners) is listening
+Synapse's [`client` listener](../../usage/configuration/config_documentation.md#listeners) is listening
 on (e.g. `8008`). Then we can use the following command to request the account
 information from the Admin API.

@@ -5,7 +5,7 @@ use it, you must enable the account validity feature (under
 `account_validity`) in Synapse's configuration.

 To use it, you will need to authenticate by providing an `access_token`
-for a server admin: see [Admin API](../usage/administration/admin_api).
+for a server admin: see [Admin API](index.html).

 ## Renew account

@@ -106,4 +106,4 @@ The following JSON body parameters are available:

 - `job_name` - A string which job to run. Valid values are:
  - `populate_stats_process_rooms` - Recalculate the stats for all rooms.
-  - `regenerate_directory` - Recalculate the [user directory](../../../user_directory.md) if it is stale or out of sync.
+  - `regenerate_directory` - Recalculate the [user directory](../../usage/configuration/user_directory.md) if it is stale or out of sync.
@@ -3,7 +3,7 @@
 This API returns information about reported events.

 To use it, you will need to authenticate by providing an `access_token`
-for a server admin: see [Admin API](../usage/administration/admin_api).
+for a server admin: see [Admin API](index.html).

 The api is:
 ```
@@ -81,7 +81,7 @@ The following fields are returned in the JSON response body:
  - `failure_ts` - nullable integer - The first time Synapse tried and failed to reach the
    remote server, in ms. This is `null` if communication with the remote server has never failed.
  - `last_successful_stream_ordering` - nullable integer - The stream ordering of the most
-    recent successfully-sent [PDU](understanding_synapse_through_grafana_graphs.md#federation)
+    recent successfully-sent [PDU](../understanding_synapse_through_grafana_graphs.md#federation)
    to this destination, or `null` if this information has not been tracked yet.
 - `next_token`: string representing a positive integer - Indication for pagination. See above.
 - `total` - integer - Total number of destinations.
@@ -174,7 +174,7 @@ The following fields are returned in the JSON response body:
  Room objects contain the following fields:
  - `room_id` - string - The ID of the room.
  - `stream_ordering` - integer -  The stream ordering of the most recent
-    successfully-sent [PDU](understanding_synapse_through_grafana_graphs.md#federation)
+    successfully-sent [PDU](../understanding_synapse_through_grafana_graphs.md#federation)
    to this destination in this room.
 - `next_token`: string representing a positive integer - Indication for pagination. See above.
 - `total` - integer - Total number of destinations.
@@ -3,10 +3,10 @@
 These APIs allow extracting media information from the homeserver.

 Details about the format of the `media_id` and storage of the media in the file system
-are documented under [media repository](../media_repository.md).
+are documented under [media repository](../../development/internal_documentation/media_repository.md).

 To use it, you will need to authenticate by providing an `access_token`
-for a server admin: see [Admin API](../usage/administration/admin_api).
+for a server admin: see [Admin API](index.html).

 ## List all media in a room

@@ -11,7 +11,7 @@ Note that Synapse requires at least one message in each room, so it will never
 delete the last message in a room.

 To use it, you will need to authenticate by providing an `access_token`
-for a server admin: see [Admin API](../usage/administration/admin_api).
+for a server admin: see [Admin API](index.html).

 The API is:

@@ -5,7 +5,7 @@ non-interactive way. This is generally used for bootstrapping a Synapse
 instance with administrator accounts.

 To authenticate yourself to the server, you will need both the shared secret
-([`registration_shared_secret`](../usage/configuration/config_documentation.md#registration_shared_secret)
+([`registration_shared_secret`](../../usage/configuration/config_documentation.md#registration_shared_secret)
 in the homeserver configuration), and a one-time nonce. If the registration
 shared secret is not configured, this API is not enabled.

@@ -6,7 +6,7 @@ registration requests, as proposed in
 and stabilised in version 1.2 of the Matrix specification.
 To use it, you will need to enable the `registration_requires_token` config
 option, and authenticate by providing an `access_token` for a server admin:
-see [Admin API](../admin_api).
+see [Admin API](index.html).


 ## Registration token objects
@@ -6,7 +6,7 @@ local users. The server administrator must be in the room and have permission to
 invite users.

 To use it, you will need to authenticate by providing an `access_token`
-for a server admin: see [Admin API](../usage/administration/admin_api).
+for a server admin: see [Admin API](index.html).

 ## Parameters

@@ -5,7 +5,7 @@ server. There are various parameters available that allow for filtering and
 sorting the returned list. This API supports pagination.

 To use it, you will need to authenticate by providing an `access_token`
-for a server admin: see [Admin API](../usage/administration/admin_api).
+for a server admin: see [Admin API](index.html).

 **Parameters**

@@ -400,7 +400,7 @@ sent to a room in a given timeframe. There are various parameters available
 that allow for filtering and ordering the returned list. This API supports pagination.

 To use it, you will need to authenticate by providing an `access_token`
-for a server admin: see [Admin API](../usage/administration/admin_api).
+for a server admin: see [Admin API](index.html).

 This endpoint mirrors the [Matrix Spec defined Messages API](https://spec.matrix.org/v1.1/client-server-api/#get_matrixclientv3roomsroomidmessages).

@@ -45,4 +45,4 @@ Once the notice has been sent, the API will return the following response:
 ```

 Note that server notices must be enabled in `homeserver.yaml` before this API
-can be used. See [the server notices documentation](../server_notices.md) for more information.
+can be used. See [the server notices documentation](../../usage/configuration/server_notices.md) for more information.
@@ -4,7 +4,7 @@ Returns information about all local media usage of users. Gives the
 possibility to filter them by time and user.

 To use it, you will need to authenticate by providing an `access_token`
-for a server admin: see [Admin API](../usage/administration/admin_api).
+for a server admin: see [Admin API](index.html).

 The API is:

@@ -1,7 +1,7 @@
 # User Admin API

 To use it, you will need to authenticate by providing an `access_token`
-for a server admin: see [Admin API](../usage/administration/admin_api).
+for a server admin: see [Admin API](index.html).

 ## Query User Account

@@ -37,6 +37,7 @@ It returns a JSON body like the following:
    "is_guest": 0,
    "admin": 0,
    "deactivated": 0,
+    "erased": false,
    "shadow_banned": 0,
    "creation_ts": 1560432506,
    "appservice_id": null,
@@ -126,7 +127,7 @@ Body parameters:
  belonging to a user.
 - `external_ids` - array, optional. Allow setting the identifier of the external identity
  provider for SSO (Single sign-on). Details in the configuration manual under the
-  sections [sso](../usage/configuration/config_documentation.md#sso) and [oidc_providers](../usage/configuration/config_documentation.md#oidc_providers).
+  sections [sso](../../usage/configuration/config_documentation.md#sso) and [oidc_providers](../../usage/configuration/config_documentation.md#oidc_providers).
  - `auth_provider` - string. ID of the external identity provider. Value of `idp_id`
    in the homeserver configuration. Note that no error is raised if the provided
    value is not in the homeserver configuration.
@@ -167,6 +168,7 @@ A response body like the following is returned:
            "admin": 0,
            "user_type": null,
            "deactivated": 0,
+            "erased": false,
            "shadow_banned": 0,
            "displayname": "<User One>",
            "avatar_url": null,
@@ -177,6 +179,7 @@ A response body like the following is returned:
            "admin": 1,
            "user_type": null,
            "deactivated": 0,
+            "erased": false,
            "shadow_banned": 0,
            "displayname": "<User Two>",
            "avatar_url": "<avatar_url>",
@@ -247,6 +250,7 @@ The following fields are returned in the JSON response body:
  - `user_type` - string - Type of the user. Normal users are type `None`.
    This allows user type specific behaviour. There are also types `support` and `bot`. 
  - `deactivated` - bool - Status if that user has been marked as deactivated.
+  - `erased` - bool - Status if that user has been marked as erased.
  - `shadow_banned` - bool - Status if that user has been marked as shadow banned.
  - `displayname` - string - The user's display name if they have set one.
  - `avatar_url` - string -  The user's avatar URL if they have set one.
@@ -544,9 +548,9 @@ The following fields are returned in the JSON response body:
 ### List media uploaded by a user
 Gets a list of all local media that a specific `user_id` has created.
 These are media that the user has uploaded themselves
-([local media](../media_repository.md#local-media)), as well as
-[URL preview images](../media_repository.md#url-previews) requested by the user if the
-[feature is enabled](../usage/configuration/config_documentation.md#url_preview_enabled).
+([local media](../../development/internal_documentation/media_repository.md#local-media)), as well as
+[URL preview images](../../development/internal_documentation/media_repository.md#url-previews) requested by the user if the
+[feature is enabled](../../usage/configuration/config_documentation.md#url_preview_enabled).

 By default, the response is ordered by descending creation date and ascending media ID.
 The newest media is on top. You can change the order with parameters
@@ -646,7 +650,7 @@ The following fields are returned in the JSON response body:
  - `last_access_ts` - integer - Timestamp when the content was last accessed in ms.
  - `media_id` - string - The id used to refer to the media. Details about the format
    are documented under
-    [media repository](../media_repository.md).
+    [media repository](../../development/internal_documentation/media_repository.md).
  - `media_length` - integer - Length of the media in bytes.
  - `media_type` - string - The MIME-type of the media.
  - `quarantined_by` - string - The user ID that initiated the quarantine request
@@ -2,7 +2,7 @@

 How do I become a server admin?
 ---
-If your server already has an admin account you should use the [User Admin API](../../admin_api/user_admin_api.md#Change-whether-a-user-is-a-server-administrator-or-not) to promote other accounts to become admins.
+If your server already has an admin account you should use the [User Admin API](../../administration/admin_api/user_admin_api.md#Change-whether-a-user-is-a-server-administrator-or-not) to promote other accounts to become admins.

 If you don't have any admin accounts yet you won't be able to use the admin API, so you'll have to edit the database manually. Manually editing the database is generally not recommended so once you have an admin account: use the admin APIs to make further changes.

@@ -35,7 +35,7 @@ SELECT NAME from users;
 Manually resetting passwords
 ---
 Users can reset their password through their client. Alternatively, a server admin
-can reset a user's password using the [admin API](../../admin_api/user_admin_api.md#reset-password).
+can reset a user's password using the [admin API](../../administration/admin_api/user_admin_api.md#reset-password).


 I have a problem with my server. Can I just delete my database and start again?
@@ -101,7 +101,7 @@ ORDER BY num_rows desc
 LIMIT 10;
 ```

-You can also use the [List Room API](../../admin_api/rooms.md#list-room-api)
+You can also use the [List Room API](../../administration/admin_api/rooms.md#list-room-api)
 and `order_by` `state_events`.


@@ -115,14 +115,14 @@ something like the following in their logs:

    2019-09-11 19:32:04,271 - synapse.federation.transport.server - 288 - WARNING - GET-11752 - authenticate_request failed: 401: Invalid signature for server <server> with key ed25519:a_EqML: Unable to verify signature for <server>

-This is normally caused by a misconfiguration in your reverse-proxy. See [the reverse proxy docs](docs/reverse_proxy.md) and double-check that your settings are correct.
+This is normally caused by a misconfiguration in your reverse-proxy. See [the reverse proxy docs](../setup/reverse_proxy.md) and double-check that your settings are correct.


 Help!! Synapse is slow and eats all my RAM/CPU!
 -----------------------------------------------

 First, ensure you are running the latest version of Synapse, using Python 3
-with a [PostgreSQL database](../../postgres.md).
+with a [PostgreSQL database](../setup/postgres.md).

 Synapse's architecture is quite RAM hungry currently - we deliberately
 cache a lot of recent room data and metadata in RAM in order to speed up
@@ -157,7 +157,7 @@ massive excess of outgoing federation requests (see [discussion](https://github.
 indicate that your server is also issuing far more outgoing federation
 requests than can be accounted for by your users' activity, this is a
 likely cause. The misbehavior can be worked around by disabling presence
-in the Synapse config file: [see here](../configuration/config_documentation.md#presence).
+in the Synapse config file: [see here](../usage/configuration/config_documentation.md#presence).


 Running out of File Handles
@@ -2,13 +2,13 @@ This blog post by Victor Berger explains how to use many of the tools listed on

 # List of useful tools and scripts for maintenance Synapse database:

-## [Purge Remote Media API](../../admin_api/media_admin_api.md#purge-remote-media-api)
+## [Purge Remote Media API](../../administration/admin_api/media_admin_api.md#purge-remote-media-api)
 The purge remote media API allows server admins to purge old cached remote media.

-## [Purge Local Media API](../../admin_api/media_admin_api.md#delete-local-media)
+## [Purge Local Media API](../../administration/admin_api/media_admin_api.md#delete-local-media)
 This API deletes the *local* media from the disk of your own server.

-## [Purge History API](../../admin_api/purge_history_api.md)
+## [Purge History API](../../administration/admin_api/purge_history_api.md)
 The purge history API allows server admins to purge historic events from their database, reclaiming disk space.

 ## [synapse-compress-state](https://github.com/matrix-org/rust-synapse-compress-state)
@@ -15,7 +15,7 @@ environments where untrusted users have shell access.

 To enable it, first add the `manhole` listener configuration in your
 `homeserver.yaml`. You can find information on how to do that 
-in the [configuration manual](usage/configuration/config_documentation.md#manhole_settings).
+in the [configuration manual](../usage/configuration/config_documentation.md#manhole_settings).
 The configuration is slightly different if you're using docker.

 #### Docker config
@@ -16,14 +16,21 @@
    There are two methods of enabling the metrics endpoint in Synapse.

    The first serves the metrics as a part of the usual web server and
-    can be enabled by adding the \"metrics\" resource to the existing
-    listener as such:
+    can be enabled by adding the `metrics` resource to the existing
+    listener as such as in this example:

    ```yaml
-      resources:
-        - names:
-          - client
-          - metrics
+    listeners:
+      - port: 8008
+        tls: false
+        type: http
+        x_forwarded: true
+        bind_addresses: ['::1', '127.0.0.1']
+
+        resources:
+          # added "metrics" in this line
+          - names: [client, federation, metrics]
+            compress: false
    ```

    This provides a simple way of adding metrics to your Synapse
@@ -37,14 +44,24 @@
    to just internal networks easier. The served metrics are available
    over HTTP only, and will be available at `/_synapse/metrics`.

-    Add a new listener to homeserver.yaml:
+    Add a new listener to homeserver.yaml as in this example:

    ```yaml
-      listeners:
-        - type: metrics
-          port: 9000
-          bind_addresses:
-            - '0.0.0.0'
+    listeners:
+      - port: 8008
+        tls: false
+        type: http
+        x_forwarded: true
+        bind_addresses: ['::1', '127.0.0.1']
+
+        resources:
+          - names: [client, federation]
+            compress: false
+
+      # beginning of the new metrics listener
+      - port: 9000
+        type: metrics
+        bind_addresses: ['::1', '127.0.0.1']
    ```

 1.  Restart Synapse.
@@ -75,7 +92,7 @@

 ## Monitoring workers

-To monitor a Synapse installation using [workers](workers.md),
+To monitor a Synapse installation using [workers](../usage/configuration/workers.md),
 every worker needs to be monitored independently, in addition to
 the main homeserver process. This is because workers don't send
 their metrics to the main homeserver process, but expose them
@@ -3,11 +3,11 @@
 Synapse can be configured to record the number of monthly active users (also referred to as MAU) on a given homeserver.
 For clarity's sake, MAU only tracks local users.

-Please note that the metrics recorded by the [Homeserver Usage Stats](../../usage/administration/monitoring/reporting_homeserver_usage_statistics.md)
+Please note that the metrics recorded by the [Homeserver Usage Stats](reporting_homeserver_usage_statistics.md)
 are calculated differently. The `monthly_active_users` from the usage stats does not take into account any
 of the rules below, and counts any users who have made a request to the homeserver in the last 30 days.

-See the [configuration manual](../../usage/configuration/config_documentation.md#limit_usage_by_mau) for details on how to configure MAU.
+See the [configuration manual](../usage/configuration/config_documentation.md#limit_usage_by_mau) for details on how to configure MAU.

 ## Calculating active users

@@ -6,9 +6,9 @@ provide the foundation a glimpse into the number of Synapse homeservers
 participating in the network, as well as statistics such as the number of
 rooms being created and messages being sent. This feature is sometimes
 affectionately called "phone home" stats. Reporting
-[is optional](../../configuration/config_documentation.md#report_stats)
+[is optional](../usage/configuration/config_documentation.md#report_stats)
 and the reporting endpoint
-[can be configured](../../configuration/config_documentation.md#report_stats_endpoint),
+[can be configured](../usage/configuration/config_documentation.md#report_stats_endpoint),
 in case you would like to instead report statistics from a set of homeservers
 to your own infrastructure.

@@ -52,15 +52,15 @@ The following statistics are sent to the configured reporting endpoint:
 | `r30v2_users_ios`          | int    | The number of 30 day retained users, as defined above. Filtered only to clients with ("riot" or "element") and "ios" (case-insensitive) in the user agent string.                                                                                                                               |
 | `r30v2_users_electron`     | int    | The number of 30 day retained users, as defined above. Filtered only to clients with ("riot" or "element") and "electron" (case-insensitive) in the user agent string.                                                                                                                          |
 | `r30v2_users_web`          | int    | The number of 30 day retained users, as defined above. Filtered only to clients with "mozilla" or "gecko" (case-insensitive) in the user agent string.                                                                                                                                          |
-| `cache_factor`             | int    | The configured [`global factor`](../../configuration/config_documentation.md#caching) value for caching.                                                                                                                                                                                        |
-| `event_cache_size`         | int    | The configured [`event_cache_size`](../../configuration/config_documentation.md#caching) value for caching.                                                                                                                                                                                     |
+| `cache_factor`             | int    | The configured [`global factor`](../usage/configuration/config_documentation.md#caching) value for caching.                                                                                                                                                                                        |
+| `event_cache_size`         | int    | The configured [`event_cache_size`](../usage/configuration/config_documentation.md#caching) value for caching.                                                                                                                                                                                     |
 | `database_engine`          | string | The database engine that is in use. Either "psycopg2" meaning PostgreSQL is in use, or "sqlite3" for SQLite3.                                                                                                                                                                                   |
 | `database_server_version` | string | The version of the database server. Examples being "10.10" for PostgreSQL server version 10.0, and "3.38.5" for SQLite 3.38.5 installed on the system.                                                                                                                                          |
 | `log_level` | string | The log level in use. Examples are "INFO", "WARNING", "ERROR", "DEBUG", etc.                                                                                                                                                                                                                    |


 [^1]: Native matrix users and guests are always counted. If the
-[`track_puppeted_user_ips`](../../configuration/config_documentation.md#track_puppeted_user_ips)
+[`track_puppeted_user_ips`](../usage/configuration/config_documentation.md#track_puppeted_user_ips)
 option is set to `true`, "puppeted" users (users that an Application Service have performed
 [an action on behalf of](https://spec.matrix.org/v1.3/application-service-api/#identity-assertion))
 will also be counted. Note that an Application Service can "puppet" any user in their
@@ -71,7 +71,7 @@ will additionally be counted as a user (irrespective of `track_puppeted_user_ips
 ## Using a Custom Statistics Collection Server

 If statistics reporting is enabled, the endpoint that Synapse sends metrics to is configured by the
-[`report_stats_endpoint`](../../configuration/config_documentation.md#report_stats_endpoint) config
+[`report_stats_endpoint`](../usage/configuration/config_documentation.md#report_stats_endpoint) config
 option. By default, statistics are sent to Matrix.org.

 If you would like to set up your own statistics collection server and send metrics there, you may
@@ -1,6 +1,6 @@
 # Request log format

-HTTP request logs are written by synapse (see [`site.py`](../synapse/http/site.py) for details).
+HTTP request logs are written by synapse (see [`site.py`](https://github.com/matrix-org/synapse/blob/latest/synapse/http/site.py) for details).

 See the following for how to decode the dense data available from the default logging configuration.

@@ -2,7 +2,7 @@

 It is possible to monitor much of the internal state of Synapse using [Prometheus](https://prometheus.io) 
 metrics and [Grafana](https://grafana.com/). 
-A guide for configuring Synapse to provide metrics is available [here](../../metrics-howto.md) 
+A guide for configuring Synapse to provide metrics is available [here](../../administration/metrics-howto.md) 
 and information on setting up Grafana is [here](https://github.com/matrix-org/synapse/tree/master/contrib/grafana).
 In this setup, Prometheus will periodically scrape the information Synapse provides and
 store a record of it over time. Grafana is then used as an interface to query and
@@ -79,6 +79,6 @@ indicator of problems, and a symptom of other problems though, so check other gr

 If you're still having performance problems with your Synapse instance and you've 
 tried everything you can, it may just be a lack of system resources. Consider adding
-more CPU and RAM, and make use of [worker mode](../../workers.md) 
+more CPU and RAM, and make use of [worker mode](../usage/configuration/workers.md) 
 to make use of multiple CPU cores / multiple machines for your homeserver.

@@ -75,7 +75,7 @@ on save as they take a while and can be very resource intensive.
 When adding a configuration option to the code, if several settings are grouped into a single dict, ensure that your code
 correctly handles the top-level option being set to `None` (as it will be if no sub-options are enabled).

-The [configuration manual](usage/configuration/config_documentation.md) acts as a
+The [configuration manual](../usage/configuration/config_documentation.md) acts as a
 reference to Synapse's configuration options for server administrators.
 Remember that many readers will be unfamiliar with YAML and server
 administration in general, so it is important that when you add
@@ -103,9 +103,9 @@ Synapse developers.
 regarding Synapse's Admin API, which is used mostly by sysadmins and external
 service developers.

-Synapse's code style is documented [here](../code_style.md). Please follow
+Synapse's code style is documented [here](code_style.md). Please follow
 it, including the conventions for the [sample configuration
-file](../code_style.md#configuration-file-format).
+file](code_style.md#configuration-file-format).

 We welcome improvements and additions to our documentation itself! When
 writing new pages, please
@@ -333,7 +333,7 @@ SYNAPSE_TEST_LOG_LEVEL=DEBUG COMPLEMENT_DIR=../complement ./scripts-dev/compleme
 ### Prettier formatting with `gotestfmt`

 If you want to format the output of the tests the same way as it looks in CI,
-install [gotestfmt](https://github.com/haveyoudebuggedit/gotestfmt).
+install [gotestfmt](https://github.com/GoTestTools/gotestfmt).

 You can then use this incantation to format the tests appropriately:

@@ -390,7 +390,7 @@ This file will become part of our [changelog](
 https://github.com/matrix-org/synapse/blob/master/CHANGES.md) at the next
 release, so the content of the file should be a short description of your
 change in the same style as the rest of the changelog. The file can contain Markdown
-formatting, and should end with a full stop (.) or an exclamation mark (!) for
+formatting, and must end with a full stop (.) or an exclamation mark (!) for
 consistency.

 Adding credits to the changelog is encouraged, we value your
@@ -58,7 +58,7 @@ https://www.jaegertracing.io/docs/latest/getting-started.

 OpenTracing is not enabled by default. It must be enabled in the
 homeserver config by adding the `opentracing` option to your config file. You can find 
-documentation about how to do this in the [config manual under the header 'Opentracing'](usage/configuration/config_documentation.md#opentracing).
+documentation about how to do this in the [config manual under the header 'Opentracing'](../usage/configuration/config_documentation.md#opentracing).
 See below for an example Opentracing configuration: 

 ```yaml
@@ -20,7 +20,7 @@ Joining a "large", federated room will initially fail with the below message in

 ![Screenshot_2020-10-02_17-15-06](https://user-images.githubusercontent.com/71895/94945781-18771500-04d3-11eb-8419-83c2da73a341.png)

-This is [#1211](https://github.com/matrix-org/synapse/issues/1211) and will also hopefully be mitigated by peeking [matrix-org/matrix-doc#2753](https://github.com/matrix-org/matrix-doc/pull/2753) so at least you don't need to wait for a join to complete before finding out if it's the kind of room you want. Note that you should first disable presence, otherwise it'll just make the situation worse [#3120](https://github.com/matrix-org/synapse/issues/3120). There is a lot of database interaction too, so make sure you've [migrated your data](../postgres.md) from the default sqlite to postgresql. Personally, I recommend patience - once the initial join is complete there's rarely any issues with actually interacting with the room, but if you like you can just block "large" rooms entirely.
+This is [#1211](https://github.com/matrix-org/synapse/issues/1211) and will also hopefully be mitigated by peeking [matrix-org/matrix-doc#2753](https://github.com/matrix-org/matrix-doc/pull/2753) so at least you don't need to wait for a join to complete before finding out if it's the kind of room you want. Note that you should first disable presence, otherwise it'll just make the situation worse [#3120](https://github.com/matrix-org/synapse/issues/3120). There is a lot of database interaction too, so make sure you've [migrated your data](../setup/postgres.md) from the default sqlite to postgresql. Personally, I recommend patience - once the initial join is complete there's rarely any issues with actually interacting with the room, but if you like you can just block "large" rooms entirely.

 ### Sessions

@@ -36,7 +36,7 @@ presence to (for those users that the receiving user is considered interested in
 It does not include state for users who are currently offline, and it can only be
 called on workers that support sending federation. Additionally, this method must
 only be called from the process that has been configured to write to the
-the [presence stream](workers.md#stream-writers).
+the [presence stream](usage/configuration/workers.md#stream-writers).
 By default, this is the main process, but another worker can be configured to do
 so.

@@ -14,7 +14,7 @@ production-ready setup, you will probably want to specify your domain
 (`example.com`) rather than a matrix-specific hostname here (in the same way
 that your email address is probably `user@example.com` rather than
 `user@email.example.com`) - but doing so may require more advanced setup: see
-[Setting up Federation](../federate.md).
+[Setting up Federation](../usage/federate.md).

 ## Installing Synapse

@@ -392,7 +392,7 @@ instead. Advantages include:
 - allowing the DB to be run on separate hardware

 For information on how to install and use PostgreSQL in Synapse, please see
-[Using Postgres](../postgres.md)
+[Using Postgres](postgres.md)

 SQLite is only acceptable for testing purposes. SQLite should not be used in
 a production server. Synapse will perform poorly when using
@@ -407,7 +407,7 @@ over HTTPS.

 The recommended way to do so is to set up a reverse proxy on port
 `8448`. You can find documentation on doing so in
-[the reverse proxy documentation](../reverse_proxy.md).
+[the reverse proxy documentation](reverse_proxy.md).

 Alternatively, you can configure Synapse to expose an HTTPS port. To do
 so, you will need to edit `homeserver.yaml`, as follows:
@@ -436,7 +436,7 @@ listeners:
  `cert.pem`).

 For a more detailed guide to configuring your server for federation, see
-[Federation](../federate.md).
+[Federation](../usage/federate.md).

 ### Client Well-Known URI

@@ -553,7 +553,7 @@ it can register users, including admin accounts, on your server even if
 ### Setting up a TURN server

 For reliable VoIP calls to be routed via this homeserver, you MUST configure
-a TURN server. See [TURN setup](../turn-howto.md) for details.
+a TURN server. See [TURN setup](turn-howto.md) for details.

 ### URL previews

@@ -8,14 +8,14 @@ Synapse will require the python postgres client library in order to
 connect to a postgres database.

 -   If you are using the [matrix.org debian/ubuntu
-    packages](setup/installation.md#matrixorg-packages), the necessary python
+    packages](installation.md#matrixorg-packages), the necessary python
    library will already be installed, but you will need to ensure the
    low-level postgres library is installed, which you can do with
    `apt install libpq5`.
 -   For other pre-built packages, please consult the documentation from
    the relevant package.
 -   If you installed synapse [in a
-    virtualenv](setup/installation.md#installing-from-source), you can install
+    virtualenv](installation.md#installing-from-source), you can install
    the library with:

        ~/synapse/env/bin/pip install "matrix-synapse[postgres]"
--- a/Show More
+++ b/Show More
				`@@ -0,0 +1 @@`
				Fix a long-standing bug where the `update_synapse_database` script could not be run with multiple databases. Contributed by @thefinn93 @ Beeper.
				`@@ -0,0 +1 @@`
				`Improve aesthetics of HTML templates. Note that these changes do not retroactively apply to templates which have been [customised](https://matrix-org.github.io/synapse/latest/templates.html#templates) by server admins.`
				`@@ -0,0 +1 @@`
				`Fix a long-standing bug where Synapse would count codepoints instead of bytes when validating the size of some fields.`
				`@@ -0,0 +1 @@`
				`Bump flake8-bugbear from 21.3.2 to 22.9.23.`
				`@@ -0,0 +1 @@`
				Fix a long-standing bug where Synapse would accidentally include extra information in the response to [`PUT /_matrix/federation/v2/invite/{roomId}/{eventId}`](https://spec.matrix.org/v1.4/server-server-api/#put_matrixfederationv2inviteroomideventid).
				`@@ -0,0 +1 @@`
				Correct the name of the config option [`encryption_enabled_by_default_for_room_type`](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#encryption_enabled_by_default_for_room_type).
				`@@ -0,0 +1 @@`
				`Bump types-opentracing from 2.4.7 to 2.4.10.`
				`@@ -0,0 +1 @@`
				Update docstrings of `SynapseError` and `FederationError` to bettter describe what they are used for and the effects of using them are.
				`@@ -0,0 +1 @@`
				`Show erasure status when listing users in the Admin API.`
				`@@ -0,0 +1 @@`
				`Add initial power level event to batch of bulk persisted events when creating a new room.`