Basic release script

.buildkite/scripts/create_postgres_db.py

@@ -15,7 +15,6 @@
 # limitations under the License.
 
 import logging
-
 from synapse.storage.engines import create_engine
 
 logger = logging.getLogger("create_postgres_db")

.buildkite/scripts/test_old_deps.sh

@@ -6,11 +6,8 @@
 set -ex
 
 apt-get update
-apt-get install -y python3.5 python3.5-dev python3-pip libxml2-dev libxslt-dev xmlsec1 zlib1g-dev tox
+apt-get install -y python3.5 python3.5-dev python3-pip libxml2-dev libxslt-dev zlib1g-dev tox
 
 export LANG="C.UTF-8"
 
-# Prevent virtualenv from auto-updating pip to an incompatible version
-export VIRTUALENV_NO_DOWNLOAD=1
-
 exec tox -e py35-old,combine

.circleci/config.yml

@@ -5,29 +5,30 @@ jobs:
       - image: docker:git
     steps:
       - checkout
+      - setup_remote_docker
       - docker_prepare
       - run: docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
-      # for release builds, we want to get the amd64 image out asap, so first
-      # we do an amd64-only build, before following up with a multiarch build.
       - docker_build:
           tag: -t matrixdotorg/synapse:${CIRCLE_TAG}
           platforms: linux/amd64
       - docker_build:
           tag: -t matrixdotorg/synapse:${CIRCLE_TAG}
-          platforms: linux/amd64,linux/arm64
+          platforms: linux/amd64,linux/arm/v7,linux/arm64
 
   dockerhubuploadlatest:
     docker:
       - image: docker:git
     steps:
       - checkout
+      - setup_remote_docker
       - docker_prepare
       - run: docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
-      # for `latest`, we don't want the arm images to disappear, so don't update the tag
-      # until all of the platforms are built.
       - docker_build:
           tag: -t matrixdotorg/synapse:latest
-          platforms: linux/amd64,linux/arm64
+          platforms: linux/amd64
+      - docker_build:
+          tag: -t matrixdotorg/synapse:latest
+          platforms: linux/amd64,linux/arm/v7,linux/arm64
 
 workflows:
   build:
@@ -45,16 +46,12 @@ workflows:
 
 commands:
   docker_prepare:
-    description: Sets up a remote docker server, downloads the buildx cli plugin, and enables multiarch images
+    description: Downloads the buildx cli plugin and enables multiarch images
     parameters:
       buildx_version:
         type: string
         default: "v0.4.1"
     steps:
-      - setup_remote_docker:
-          # 19.03.13 was the most recent available on circleci at the time of
-          # writing.
-          version: 19.03.13
       - run: apk add --no-cache curl
       - run: mkdir -vp ~/.docker/cli-plugins/ ~/dockercache
       - run: curl --silent -L "https://github.com/docker/buildx/releases/download/<< parameters.buildx_version >>/buildx-<< parameters.buildx_version >>.linux-amd64" > ~/.docker/cli-plugins/docker-buildx

.git-blame-ignore-revs

@@ -1,8 +0,0 @@
-# Black reformatting (#5482).
-32e7c9e7f20b57dd081023ac42d6931a8da9b3a3
-
-# Target Python 3.5 with black (#8664).
-aff1eb7c671b0a3813407321d2702ec46c71fa56
-
-# Update black to 20.8b1 (#9381).
-0a00b7ff14890987f09112a2ae696c61001e6cf1

.gitignore

@@ -6,19 +6,16 @@
 *.egg
 *.egg-info
 *.lock
-*.py[cod]
+*.pyc
 *.snap
 *.tac
 _trial_temp/
 _trial_temp*/
 /out
-.DS_Store
-__pycache__/
 
 # stuff that is likely to exist when you run a server locally
 /*.db
 /*.log
-/*.log.*
 /*.log.config
 /*.pid
 /.python-version

CONTRIBUTING.md

@@ -1,31 +1,4 @@
-Welcome to Synapse
-
-This document aims to get you started with contributing to this repo! 
-
-- [1. Who can contribute to Synapse?](#1-who-can-contribute-to-synapse)
-- [2. What do I need?](#2-what-do-i-need)
-- [3. Get the source.](#3-get-the-source)
-- [4. Install the dependencies](#4-install-the-dependencies)
-  * [Under Unix (macOS, Linux, BSD, ...)](#under-unix-macos-linux-bsd-)
-  * [Under Windows](#under-windows)
-- [5. Get in touch.](#5-get-in-touch)
-- [6. Pick an issue.](#6-pick-an-issue)
-- [7. Turn coffee and documentation into code and documentation!](#7-turn-coffee-and-documentation-into-code-and-documentation)
-- [8. Test, test, test!](#8-test-test-test)
-  * [Run the linters.](#run-the-linters)
-  * [Run the unit tests.](#run-the-unit-tests)
-  * [Run the integration tests.](#run-the-integration-tests)
-- [9. Submit your patch.](#9-submit-your-patch)
-  * [Changelog](#changelog)
-    + [How do I know what to call the changelog file before I create the PR?](#how-do-i-know-what-to-call-the-changelog-file-before-i-create-the-pr)
-    + [Debian changelog](#debian-changelog)
-  * [Sign off](#sign-off)
-- [10. Turn feedback into better code.](#10-turn-feedback-into-better-code)
-- [11. Find a new issue.](#11-find-a-new-issue)
-- [Notes for maintainers on merging PRs etc](#notes-for-maintainers-on-merging-prs-etc)
-- [Conclusion](#conclusion)
-
-# 1. Who can contribute to Synapse?
+# Contributing code to Synapse
 
 Everyone is welcome to contribute code to [matrix.org
 projects](https://github.com/matrix-org), provided that they are willing to
@@ -36,179 +9,70 @@ license the code under the same terms as the project's overall 'outbound'
 license - in our case, this is almost always Apache Software License v2 (see
 [LICENSE](LICENSE)).
 
-# 2. What do I need?
-
-The code of Synapse is written in Python 3. To do pretty much anything, you'll need [a recent version of Python 3](https://wiki.python.org/moin/BeginnersGuide/Download).
-
-The source code of Synapse is hosted on GitHub. You will also need [a recent version of git](https://github.com/git-guides/install-git).
-
-For some tests, you will need [a recent version of Docker](https://docs.docker.com/get-docker/).
-
-
-# 3. Get the source.
+## How to contribute
 
 The preferred and easiest way to contribute changes is to fork the relevant
-project on GitHub, and then [create a pull request](
+project on github, and then [create a pull request](
 https://help.github.com/articles/using-pull-requests/) to ask us to pull your
 changes into our repo.
 
-Please base your changes on the `develop` branch.
+Some other points to follow:
 
-```sh
-git clone git@github.com:YOUR_GITHUB_USER_NAME/synapse.git
-git checkout develop
-```
+ * Please base your changes on the `develop` branch.
 
-If you need help getting started with git, this is beyond the scope of the document, but you
-can find many good git tutorials on the web.
+ * Please follow the [code style requirements](#code-style).
 
-# 4. Install the dependencies
+ * Please include a [changelog entry](#changelog) with each PR.
 
-## Under Unix (macOS, Linux, BSD, ...)
+ * Please [sign off](#sign-off) your contribution.
 
-Once you have installed Python 3 and added the source, please open a terminal and
-setup a *virtualenv*, as follows:
+ * Please keep an eye on the pull request for feedback from the [continuous
+   integration system](#continuous-integration-and-testing) and try to fix any
+   errors that come up.
 
-```sh
-cd path/where/you/have/cloned/the/repository
-python3 -m venv ./env
-source ./env/bin/activate
-pip install -e ".[all,lint,mypy,test]"
-pip install tox
-```
+ * If you need to [update your PR](#updating-your-pull-request), just add new
+   commits to your branch rather than rebasing.
 
-This will install the developer dependencies for the project.
-
-## Under Windows
-
-TBD
-
-
-# 5. Get in touch.
-
-Join our developer community on Matrix: #synapse-dev:matrix.org !
-
-
-# 6. Pick an issue.
-
-Fix your favorite problem or perhaps find a [Good First Issue](https://github.com/matrix-org/synapse/issues?q=is%3Aopen+is%3Aissue+label%3A%22Good+First+Issue%22)
-to work on.
-
-
-# 7. Turn coffee and documentation into code and documentation!
+## Code style
 
 Synapse's code style is documented [here](docs/code_style.md). Please follow
 it, including the conventions for the [sample configuration
 file](docs/code_style.md#configuration-file-format).
 
-There is a growing amount of documentation located in the [docs](docs)
-directory. This documentation is intended primarily for sysadmins running their
-own Synapse instance, as well as developers interacting externally with
-Synapse. [docs/dev](docs/dev) exists primarily to house documentation for
-Synapse developers. [docs/admin_api](docs/admin_api) houses documentation
-regarding Synapse's Admin API, which is used mostly by sysadmins and external
-service developers.
+Many of the conventions are enforced by scripts which are run as part of the
+[continuous integration system](#continuous-integration-and-testing). To help
+check if you have followed the code style, you can run `scripts-dev/lint.sh`
+locally. You'll need python 3.6 or later, and to install a number of tools:
 
-If you add new files added to either of these folders, please use [GitHub-Flavoured
-Markdown](https://guides.github.com/features/mastering-markdown/).
+```
+# Install the dependencies
+pip install -e ".[lint,mypy]"
 
-Some documentation also exists in [Synapse's GitHub
-Wiki](https://github.com/matrix-org/synapse/wiki), although this is primarily
-contributed to by community authors.
-
-
-# 8. Test, test, test!
-<a name="test-test-test"></a>
-
-While you're developing and before submitting a patch, you'll
-want to test your code.
-
-## Run the linters.
-
-The linters look at your code and do two things:
-
-- ensure that your code follows the coding style adopted by the project;
-- catch a number of errors in your code.
-
-They're pretty fast, don't hesitate!
-
-```sh
-source ./env/bin/activate
+# Run the linter script
 ./scripts-dev/lint.sh
 ```
 
-Note that this script *will modify your files* to fix styling errors.
-Make sure that you have saved all your files.
+**Note that the script does not just test/check, but also reformats code, so you
+may wish to ensure any new code is committed first**.
 
-If you wish to restrict the linters to only the files changed since the last commit
-(much faster!), you can instead run:
+By default, this script checks all files and can take some time; if you alter
+only certain files, you might wish to specify paths as arguments to reduce the
+run-time:
 
-```sh
-source ./env/bin/activate
-./scripts-dev/lint.sh -d
 ```
-
-Or if you know exactly which files you wish to lint, you can instead run:
-
-```sh
-source ./env/bin/activate
 ./scripts-dev/lint.sh path/to/file1.py path/to/file2.py path/to/folder
 ```
 
-## Run the unit tests.
+You can also provide the `-d` option, which will lint the files that have been
+changed since the last git commit. This will often be significantly faster than
+linting the whole codebase.
 
-The unit tests run parts of Synapse, including your changes, to see if anything
-was broken. They are slower than the linters but will typically catch more errors.
-
-```sh
-source ./env/bin/activate
-trial tests
-```
-
-If you wish to only run *some* unit tests, you may specify
-another module instead of `tests` - or a test class or a method:
-
-```sh
-source ./env/bin/activate
-trial tests.rest.admin.test_room tests.handlers.test_admin.ExfiltrateData.test_invite
-```
-
-If your tests fail, you may wish to look at the logs:
-
-```sh
-less _trial_temp/test.log
-```
-
-## Run the integration tests.
-
-The integration tests are a more comprehensive suite of tests. They
-run a full version of Synapse, including your changes, to check if
-anything was broken. They are slower than the unit tests but will
-typically catch more errors.
-
-The following command will let you run the integration test with the most common
-configuration:
-
-```sh
-$ docker run --rm -it -v /path/where/you/have/cloned/the/repository\:/src:ro -v /path/to/where/you/want/logs\:/logs matrixdotorg/sytest-synapse:py37
-```
-
-This configuration should generally cover  your needs. For more details about other configurations, see [documentation in the SyTest repo](https://github.com/matrix-org/sytest/blob/develop/docker/README.md).
-
-
-# 9. Submit your patch.
-
-Once you're happy with your patch, it's time to prepare a Pull Request.
-
-To prepare a Pull Request, please:
-
-1. verify that [all the tests pass](#test-test-test), including the coding style;
-2. [sign off](#sign-off) your contribution;
-3. `git push` your commit to your fork of Synapse;
-4. on GitHub, [create the Pull Request](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request);
-5. add a [changelog entry](#changelog) and push it to your Pull Request;
-6. for most contributors, that's all - however, if you are a member of the organization `matrix-org`, on GitHub, please request a review from `matrix.org / Synapse Core`.
+Before pushing new changes, ensure they don't produce linting errors. Commit any
+files that were corrected.
 
+Please ensure your changes match the cosmetic style of the existing project,
+and **never** mix cosmetic and functional changes in the same commit, as it
+makes it horribly hard to review otherwise.
 
 ## Changelog
 
@@ -358,36 +222,47 @@ Git allows you to add this signoff automatically when using the `-s`
 flag to `git commit`, which uses the name and email set in your
 `user.name` and `user.email` git configs.
 
+## Continuous integration and testing
 
-# 10. Turn feedback into better code.
+[Buildkite](https://buildkite.com/matrix-dot-org/synapse) will automatically
+run a series of checks and tests against any PR which is opened against the
+project; if your change breaks the build, this will be shown in GitHub, with
+links to the build results. If your build fails, please try to fix the errors
+and update your branch.
 
-Once the Pull Request is opened, you will see a few things:
+To run unit tests in a local development environment, you can use:
 
-1. our automated CI (Continuous Integration) pipeline will run (again) the linters, the unit tests, the integration tests and more;
-2. one or more of the developers will take a look at your Pull Request and offer feedback.
+- ``tox -e py35`` (requires tox to be installed by ``pip install tox``)
+  for SQLite-backed Synapse on Python 3.5.
+- ``tox -e py36`` for SQLite-backed Synapse on Python 3.6.
+- ``tox -e py36-postgres`` for PostgreSQL-backed Synapse on Python 3.6
+  (requires a running local PostgreSQL with access to create databases).
+- ``./test_postgresql.sh`` for PostgreSQL-backed Synapse on Python 3.5
+  (requires Docker). Entirely self-contained, recommended if you don't want to
+  set up PostgreSQL yourself.
 
-From this point, you should:
+Docker images are available for running the integration tests (SyTest) locally,
+see the [documentation in the SyTest repo](
+https://github.com/matrix-org/sytest/blob/develop/docker/README.md) for more
+information.
 
-1. Look at the results of the CI pipeline.
-   - If there is any error, fix the error.
-2. If a developer has requested changes, make these changes and let us know if it is ready for a developer to review again.
-3. Create a new commit with the changes.
-   - Please do NOT overwrite the history. New commits make the reviewer's life easier.
-   - Push this commits to your Pull Request.
-4. Back to 1.
+## Updating your pull request
 
-Once both the CI and the developers are happy, the patch will be merged into Synapse and released shortly!
+If you decide to make changes to your pull request - perhaps to address issues
+raised in a review, or to fix problems highlighted by [continuous
+integration](#continuous-integration-and-testing) - just add new commits to your
+branch, and push to GitHub. The pull request will automatically be updated.
 
-# 11. Find a new issue.
+Please **avoid** rebasing your branch, especially once the PR has been
+reviewed: doing so makes it very difficult for a reviewer to see what has
+changed since a previous review.
 
-By now, you know the drill!
-
-# Notes for maintainers on merging PRs etc
+## Notes for maintainers on merging PRs etc
 
 There are some notes for those with commit access to the project on how we
 manage git [here](docs/dev/git.md).
 
-# Conclusion
+## Conclusion
 
 That's it! Matrix is a very open and collaborative project as you might expect
 given our obsession with open communication. If we're going to successfully

INSTALL.md

@@ -1,44 +1,19 @@
-# Installation Instructions
+- [Choosing your server name](#choosing-your-server-name)
+- [Picking a database engine](#picking-a-database-engine)
+- [Installing Synapse](#installing-synapse)
+  - [Installing from source](#installing-from-source)
+    - [Platform-Specific Instructions](#platform-specific-instructions)
+  - [Prebuilt packages](#prebuilt-packages)
+- [Setting up Synapse](#setting-up-synapse)
+  - [TLS certificates](#tls-certificates)
+  - [Client Well-Known URI](#client-well-known-uri)
+  - [Email](#email)
+  - [Registering a user](#registering-a-user)
+  - [Setting up a TURN server](#setting-up-a-turn-server)
+  - [URL previews](#url-previews)
+- [Troubleshooting Installation](#troubleshooting-installation)
 
-There are 3 steps to follow under **Installation Instructions**.
-
-- [Installation Instructions](#installation-instructions)
-  - [Choosing your server name](#choosing-your-server-name)
-  - [Installing Synapse](#installing-synapse)
-    - [Installing from source](#installing-from-source)
-      - [Platform-Specific Instructions](#platform-specific-instructions)
-        - [Debian/Ubuntu/Raspbian](#debianubunturaspbian)
-        - [ArchLinux](#archlinux)
-        - [CentOS/Fedora](#centosfedora)
-        - [macOS](#macos)
-        - [OpenSUSE](#opensuse)
-        - [OpenBSD](#openbsd)
-        - [Windows](#windows)
-    - [Prebuilt packages](#prebuilt-packages)
-      - [Docker images and Ansible playbooks](#docker-images-and-ansible-playbooks)
-      - [Debian/Ubuntu](#debianubuntu)
-        - [Matrix.org packages](#matrixorg-packages)
-        - [Downstream Debian packages](#downstream-debian-packages)
-        - [Downstream Ubuntu packages](#downstream-ubuntu-packages)
-      - [Fedora](#fedora)
-      - [OpenSUSE](#opensuse-1)
-      - [SUSE Linux Enterprise Server](#suse-linux-enterprise-server)
-      - [ArchLinux](#archlinux-1)
-      - [Void Linux](#void-linux)
-      - [FreeBSD](#freebsd)
-      - [OpenBSD](#openbsd-1)
-      - [NixOS](#nixos)
-  - [Setting up Synapse](#setting-up-synapse)
-    - [Using PostgreSQL](#using-postgresql)
-    - [TLS certificates](#tls-certificates)
-    - [Client Well-Known URI](#client-well-known-uri)
-    - [Email](#email)
-    - [Registering a user](#registering-a-user)
-    - [Setting up a TURN server](#setting-up-a-turn-server)
-    - [URL previews](#url-previews)
-    - [Troubleshooting Installation](#troubleshooting-installation)
-
-## Choosing your server name
+# Choosing your server name
 
 It is important to choose the name for your server before you install Synapse,
 because it cannot be changed later.
@@ -54,9 +29,28 @@ that your email address is probably `user@example.com` rather than
 `user@email.example.com`) - but doing so may require more advanced setup: see
 [Setting up Federation](docs/federate.md).
 
-## Installing Synapse
+# Picking a database engine
 
-### Installing from source
+Synapse offers two database engines:
+ * [PostgreSQL](https://www.postgresql.org)
+ * [SQLite](https://sqlite.org/)
+
+Almost all installations should opt to use PostgreSQL. Advantages include:
+
+* significant performance improvements due to the superior threading and
+  caching model, smarter query optimiser
+* allowing the DB to be run on separate hardware
+
+For information on how to install and use PostgreSQL, please see
+[docs/postgres.md](docs/postgres.md)
+
+By default Synapse uses SQLite and in doing so trades performance for convenience.
+SQLite is only recommended in Synapse for testing purposes or for servers with
+light workloads.
+
+# Installing Synapse
+
+## Installing from source
 
 (Prebuilt packages are available for some platforms - see [Prebuilt packages](#prebuilt-packages).)
 
@@ -74,7 +68,7 @@ these on various platforms.
 
 To install the Synapse homeserver run:
 
-```sh
+```
 mkdir -p ~/synapse
 virtualenv -p python3 ~/synapse/env
 source ~/synapse/env/bin/activate
@@ -91,7 +85,7 @@ prefer.
 This Synapse installation can then be later upgraded by using pip again with the
 update flag:
 
-```sh
+```
 source ~/synapse/env/bin/activate
 pip install -U matrix-synapse
 ```
@@ -99,7 +93,7 @@ pip install -U matrix-synapse
 Before you can start Synapse, you will need to generate a configuration
 file. To do this, run (in your virtualenv, as before):
 
-```sh
+```
 cd ~/synapse
 python -m synapse.app.homeserver \
     --server-name my.domain.name \
@@ -117,54 +111,70 @@ wise to back them up somewhere safe. (If, for whatever reason, you do need to
 change your homeserver's keys, you may find that other homeserver have the
 old key cached. If you update the signing key, you should change the name of the
 key in the `<server name>.signing.key` file (the second word) to something
-different. See the [spec](https://matrix.org/docs/spec/server_server/latest.html#retrieving-server-keys) for more information on key management).
+different. See the
+[spec](https://matrix.org/docs/spec/server_server/latest.html#retrieving-server-keys)
+for more information on key management).
 
 To actually run your new homeserver, pick a working directory for Synapse to
 run (e.g. `~/synapse`), and:
 
-```sh
+```
 cd ~/synapse
 source env/bin/activate
 synctl start
 ```
 
-#### Platform-Specific Instructions
+### Platform-Specific Instructions
 
-##### Debian/Ubuntu/Raspbian
+#### Debian/Ubuntu/Raspbian
 
 Installing prerequisites on Ubuntu or Debian:
 
-```sh
-sudo apt install build-essential python3-dev libffi-dev \
+```
+sudo apt-get install build-essential python3-dev libffi-dev \
                      python3-pip python3-setuptools sqlite3 \
                      libssl-dev virtualenv libjpeg-dev libxslt1-dev
 ```
 
-##### ArchLinux
+#### ArchLinux
 
 Installing prerequisites on ArchLinux:
 
-```sh
+```
 sudo pacman -S base-devel python python-pip \
                python-setuptools python-virtualenv sqlite3
 ```
 
-##### CentOS/Fedora
+#### CentOS/Fedora
 
-Installing prerequisites on CentOS or Fedora Linux:
+Installing prerequisites on CentOS 8 or Fedora>26:
 
-```sh
+```
 sudo dnf install libtiff-devel libjpeg-devel libzip-devel freetype-devel \
-                 libwebp-devel libxml2-devel libxslt-devel libpq-devel \
-                 python3-virtualenv libffi-devel openssl-devel python3-devel
+                 libwebp-devel tk-devel redhat-rpm-config \
+                 python3-virtualenv libffi-devel openssl-devel
 sudo dnf groupinstall "Development Tools"
 ```
 
-##### macOS
+Installing prerequisites on CentOS 7 or Fedora<=25:
+
+```
+sudo yum install libtiff-devel libjpeg-devel libzip-devel freetype-devel \
+                 lcms2-devel libwebp-devel tcl-devel tk-devel redhat-rpm-config \
+                 python3-virtualenv libffi-devel openssl-devel
+sudo yum groupinstall "Development Tools"
+```
+
+Note that Synapse does not support versions of SQLite before 3.11, and CentOS 7
+uses SQLite 3.7. You may be able to work around this by installing a more
+recent SQLite version, but it is recommended that you instead use a Postgres
+database: see [docs/postgres.md](docs/postgres.md).
+
+#### macOS
 
 Installing prerequisites on macOS:
 
-```sh
+```
 xcode-select --install
 sudo easy_install pip
 sudo pip install virtualenv
@@ -174,23 +184,22 @@ brew install pkg-config libffi
 On macOS Catalina (10.15) you may need to explicitly install OpenSSL
 via brew and inform `pip` about it so that `psycopg2` builds:
 
-```sh
+```
 brew install openssl@1.1
-export LDFLAGS="-L/usr/local/opt/openssl/lib"
-export CPPFLAGS="-I/usr/local/opt/openssl/include"
+export LDFLAGS=-L/usr/local/Cellar/openssl\@1.1/1.1.1d/lib/
 ```
 
-##### OpenSUSE
+#### OpenSUSE
 
 Installing prerequisites on openSUSE:
 
-```sh
+```
 sudo zypper in -t pattern devel_basis
 sudo zypper in python-pip python-setuptools sqlite3 python-virtualenv \
                python-devel libffi-devel libopenssl-devel libjpeg62-devel
 ```
 
-##### OpenBSD
+#### OpenBSD
 
 A port of Synapse is available under `net/synapse`. The filesystem
 underlying the homeserver directory (defaults to `/var/synapse`) has to be
@@ -204,72 +213,73 @@ mounted with `wxallowed` (cf. `mount(8)`).
 Creating a `WRKOBJDIR` for building python under `/usr/local` (which on a
 default OpenBSD installation is mounted with `wxallowed`):
 
-```sh
+```
 doas mkdir /usr/local/pobj_wxallowed
 ```
 
 Assuming `PORTS_PRIVSEP=Yes` (cf. `bsd.port.mk(5)`) and `SUDO=doas` are
 configured in `/etc/mk.conf`:
 
-```sh
+```
 doas chown _pbuild:_pbuild /usr/local/pobj_wxallowed
 ```
 
 Setting the `WRKOBJDIR` for building python:
 
-```sh
+```
 echo WRKOBJDIR_lang/python/3.7=/usr/local/pobj_wxallowed  \\nWRKOBJDIR_lang/python/2.7=/usr/local/pobj_wxallowed >> /etc/mk.conf
 ```
 
 Building Synapse:
 
-```sh
+```
 cd /usr/ports/net/synapse
 make install
 ```
 
-##### Windows
+#### Windows
 
 If you wish to run or develop Synapse on Windows, the Windows Subsystem For
 Linux provides a Linux environment on Windows 10 which is capable of using the
 Debian, Fedora, or source installation methods. More information about WSL can
-be found at <https://docs.microsoft.com/en-us/windows/wsl/install-win10> for
-Windows 10 and <https://docs.microsoft.com/en-us/windows/wsl/install-on-server>
+be found at https://docs.microsoft.com/en-us/windows/wsl/install-win10 for
+Windows 10 and https://docs.microsoft.com/en-us/windows/wsl/install-on-server
 for Windows Server.
 
-### Prebuilt packages
+## Prebuilt packages
 
 As an alternative to installing from source, prebuilt packages are available
 for a number of platforms.
 
-#### Docker images and Ansible playbooks
+### Docker images and Ansible playbooks
 
-There is an official synapse image available at
-<https://hub.docker.com/r/matrixdotorg/synapse> which can be used with
+There is an offical synapse image available at
+https://hub.docker.com/r/matrixdotorg/synapse which can be used with
 the docker-compose file available at [contrib/docker](contrib/docker). Further
 information on this including configuration options is available in the README
 on hub.docker.com.
 
 Alternatively, Andreas Peters (previously Silvio Fricke) has contributed a
 Dockerfile to automate a synapse server in a single Docker image, at
-<https://hub.docker.com/r/avhost/docker-matrix/tags/>
+https://hub.docker.com/r/avhost/docker-matrix/tags/
 
 Slavi Pantaleev has created an Ansible playbook,
 which installs the offical Docker image of Matrix Synapse
 along with many other Matrix-related services (Postgres database, Element, coturn,
 ma1sd, SSL support, etc.).
 For more details, see
-<https://github.com/spantaleev/matrix-docker-ansible-deploy>
+https://github.com/spantaleev/matrix-docker-ansible-deploy
 
-#### Debian/Ubuntu
 
-##### Matrix.org packages
+### Debian/Ubuntu
+
+#### Matrix.org packages
 
 Matrix.org provides Debian/Ubuntu packages of the latest stable version of
-Synapse via <https://packages.matrix.org/debian/>. They are available for Debian
+Synapse via https://packages.matrix.org/debian/. They are available for Debian
 9 (Stretch), Ubuntu 16.04 (Xenial), and later. To use them:
 
-```sh
+```
 sudo apt install -y lsb-release wget apt-transport-https
 sudo wget -O /usr/share/keyrings/matrix-org-archive-keyring.gpg https://packages.matrix.org/debian/matrix-org-archive-keyring.gpg
 echo "deb [signed-by=/usr/share/keyrings/matrix-org-archive-keyring.gpg] https://packages.matrix.org/debian/ $(lsb_release -cs) main" |
@@ -289,7 +299,7 @@ The fingerprint of the repository signing key (as shown by `gpg
 /usr/share/keyrings/matrix-org-archive-keyring.gpg`) is
 `AAF9AE843A7584B5A3E4CD2BCF45A512DE2DA058`.
 
-##### Downstream Debian packages
+#### Downstream Debian packages
 
 We do not recommend using the packages from the default Debian `buster`
 repository at this time, as they are old and suffer from known security
@@ -301,49 +311,49 @@ for information on how to use backports.
 If you are using Debian `sid` or testing, Synapse is available in the default
 repositories and it should be possible to install it simply with:
 
-```sh
+```
 sudo apt install matrix-synapse
 ```
 
-##### Downstream Ubuntu packages
+#### Downstream Ubuntu packages
 
 We do not recommend using the packages in the default Ubuntu repository
 at this time, as they are old and suffer from known security vulnerabilities.
 The latest version of Synapse can be installed from [our repository](#matrixorg-packages).
 
-#### Fedora
+### Fedora
 
 Synapse is in the Fedora repositories as `matrix-synapse`:
 
-```sh
+```
 sudo dnf install matrix-synapse
 ```
 
 Oleg Girko provides Fedora RPMs at
-<https://obs.infoserver.lv/project/monitor/matrix-synapse>
+https://obs.infoserver.lv/project/monitor/matrix-synapse
 
-#### OpenSUSE
+### OpenSUSE
 
 Synapse is in the OpenSUSE repositories as `matrix-synapse`:
 
-```sh
+```
 sudo zypper install matrix-synapse
 ```
 
-#### SUSE Linux Enterprise Server
+### SUSE Linux Enterprise Server
 
 Unofficial package are built for SLES 15 in the openSUSE:Backports:SLE-15 repository at
-<https://download.opensuse.org/repositories/openSUSE:/Backports:/SLE-15/standard/>
+https://download.opensuse.org/repositories/openSUSE:/Backports:/SLE-15/standard/
 
-#### ArchLinux
+### ArchLinux
 
 The quickest way to get up and running with ArchLinux is probably with the community package
-<https://www.archlinux.org/packages/community/any/matrix-synapse/>, which should pull in most of
+https://www.archlinux.org/packages/community/any/matrix-synapse/, which should pull in most of
 the necessary dependencies.
 
 pip may be outdated (6.0.7-1 and needs to be upgraded to 6.0.8-1 ):
 
-```sh
+```
 sudo pip install --upgrade pip
 ```
 
@@ -352,28 +362,28 @@ ELFCLASS32 (x64 Systems), you may need to reinstall py-bcrypt to correctly
 compile it under the right architecture. (This should not be needed if
 installing under virtualenv):
 
-```sh
+```
 sudo pip uninstall py-bcrypt
 sudo pip install py-bcrypt
 ```
 
-#### Void Linux
+### Void Linux
 
 Synapse can be found in the void repositories as 'synapse':
 
-```sh
+```
 xbps-install -Su
 xbps-install -S synapse
 ```
 
-#### FreeBSD
+### FreeBSD
 
 Synapse can be installed via FreeBSD Ports or Packages contributed by Brendan Molloy from:
 
-- Ports: `cd /usr/ports/net-im/py-matrix-synapse && make install clean`
-- Packages: `pkg install py37-matrix-synapse`
+ - Ports: `cd /usr/ports/net-im/py-matrix-synapse && make install clean`
+ - Packages: `pkg install py37-matrix-synapse`
 
-#### OpenBSD
+### OpenBSD
 
 As of OpenBSD 6.7 Synapse is available as a pre-compiled binary. The filesystem
 underlying the homeserver directory (defaults to `/var/synapse`) has to be
@@ -382,35 +392,20 @@ and mounting it to `/var/synapse` should be taken into consideration.
 
 Installing Synapse:
 
-```sh
+```
 doas pkg_add synapse
 ```
 
-#### NixOS
+### NixOS
 
 Robin Lambertz has packaged Synapse for NixOS at:
-<https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/misc/matrix-synapse.nix>
+https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/misc/matrix-synapse.nix
 
-## Setting up Synapse
+# Setting up Synapse
 
 Once you have installed synapse as above, you will need to configure it.
 
-### Using PostgreSQL
-
-By default Synapse uses [SQLite](https://sqlite.org/) and in doing so trades performance for convenience.
-SQLite is only recommended in Synapse for testing purposes or for servers with
-very light workloads.
-
-Almost all installations should opt to use [PostgreSQL](https://www.postgresql.org). Advantages include:
-
-- significant performance improvements due to the superior threading and
-  caching model, smarter query optimiser
-- allowing the DB to be run on separate hardware
-
-For information on how to install and use PostgreSQL in Synapse, please see
-[docs/postgres.md](docs/postgres.md)
-
-### TLS certificates
+## TLS certificates
 
 The default configuration exposes a single HTTP port on the local
 interface: `http://localhost:8008`. It is suitable for local testing,
@@ -424,19 +419,19 @@ The recommended way to do so is to set up a reverse proxy on port
 Alternatively, you can configure Synapse to expose an HTTPS port. To do
 so, you will need to edit `homeserver.yaml`, as follows:
 
-- First, under the `listeners` section, uncomment the configuration for the
+* First, under the `listeners` section, uncomment the configuration for the
   TLS-enabled listener. (Remove the hash sign (`#`) at the start of
   each line). The relevant lines are like this:
 
-```yaml
-  - port: 8448
-    type: http
-    tls: true
-    resources:
-      - names: [client, federation]
+  ```
+    - port: 8448
+      type: http
+      tls: true
+      resources:
+        - names: [client, federation]
   ```
 
-- You will also need to uncomment the `tls_certificate_path` and
+* You will also need to uncomment the `tls_certificate_path` and
   `tls_private_key_path` lines under the `TLS` section. You will need to manage
   provisioning of these certificates yourself — Synapse had built-in ACME
   support, but the ACMEv1 protocol Synapse implements is deprecated, not
@@ -451,7 +446,7 @@ so, you will need to edit `homeserver.yaml`, as follows:
 For a more detailed guide to configuring your server for federation, see
 [federate.md](docs/federate.md).
 
-### Client Well-Known URI
+## Client Well-Known URI
 
 Setting up the client Well-Known URI is optional but if you set it up, it will
 allow users to enter their full username (e.g. `@user:<server_name>`) into clients
@@ -462,7 +457,7 @@ about the actual homeserver URL you are using.
 The URL `https://<server_name>/.well-known/matrix/client` should return JSON in
 the following format.
 
-```json
+```
 {
   "m.homeserver": {
     "base_url": "https://<matrix.example.com>"
@@ -472,7 +467,7 @@ the following format.
 
 It can optionally contain identity server information as well.
 
-```json
+```
 {
   "m.homeserver": {
     "base_url": "https://<matrix.example.com>"
@@ -489,11 +484,10 @@ Cross-Origin Resource Sharing (CORS) headers. A recommended value would be
 view it.
 
 In nginx this would be something like:
-
-```nginx
+```
 location /.well-known/matrix/client {
     return 200 '{"m.homeserver": {"base_url": "https://<matrix.example.com>"}}';
-    default_type application/json;
+    add_header Content-Type application/json;
     add_header Access-Control-Allow-Origin *;
 }
 ```
@@ -503,11 +497,11 @@ correctly. `public_baseurl` should be set to the URL that clients will use to
 connect to your server. This is the same URL you put for the `m.homeserver`
 `base_url` above.
 
-```yaml
+```
 public_baseurl: "https://<matrix.example.com>"
 ```
 
-### Email
+## Email
 
 It is desirable for Synapse to have the capability to send email. This allows
 Synapse to send password reset emails, send verifications when an email address
@@ -522,7 +516,7 @@ and `notif_from` fields filled out.  You may also need to set `smtp_user`,
 If email is not configured, password reset, registration and notifications via
 email will be disabled.
 
-### Registering a user
+## Registering a user
 
 The easiest way to create a new user is to do so from a client like [Element](https://element.io/).
 
@@ -530,7 +524,7 @@ Alternatively you can do so from the command line if you have installed via pip.
 
 This can be done as follows:
 
-```sh
+```
 $ source ~/synapse/env/bin/activate
 $ synctl start # if not already running
 $ register_new_matrix_user -c homeserver.yaml http://localhost:8008
@@ -548,12 +542,12 @@ value is generated by `--generate-config`), but it should be kept secret, as
 anyone with knowledge of it can register users, including admin accounts,
 on your server even if `enable_registration` is `false`.
 
-### Setting up a TURN server
+## Setting up a TURN server
 
 For reliable VoIP calls to be routed via this homeserver, you MUST configure
 a TURN server. See [docs/turn-howto.md](docs/turn-howto.md) for details.
 
-### URL previews
+## URL previews
 
 Synapse includes support for previewing URLs, which is disabled by default.  To
 turn it on you must enable the `url_preview_enabled: True` config parameter
@@ -563,18 +557,19 @@ This is critical from a security perspective to stop arbitrary Matrix users
 spidering 'internal' URLs on your network. At the very least we recommend that
 your loopback and RFC1918 IP addresses are blacklisted.
 
-This also requires the optional `lxml` python dependency to be  installed. This
-in turn requires the `libxml2` library to be available - on  Debian/Ubuntu this
-means `apt-get install libxml2-dev`, or equivalent for your OS.
+This also requires the optional `lxml` and `netaddr` python dependencies to be
+installed. This in turn requires the `libxml2` library to be available - on
+Debian/Ubuntu this means `apt-get install libxml2-dev`, or equivalent for
+your OS.
 
-### Troubleshooting Installation
+# Troubleshooting Installation
 
 `pip` seems to leak *lots* of memory during installation. For instance, a Linux
 host with 512MB of RAM may run out of memory whilst installing Twisted. If this
 happens, you will have to individually install the dependencies which are
 failing, e.g.:
 
-```sh
+```
 pip install twisted
 ```
 

MANIFEST.in

@@ -20,10 +20,9 @@ recursive-include scripts *
 recursive-include scripts-dev *
 recursive-include synapse *.pyi
 recursive-include tests *.py
-recursive-include tests *.pem
-recursive-include tests *.p8
-recursive-include tests *.crt
-recursive-include tests *.key
+include tests/http/ca.crt
+include tests/http/ca.key
+include tests/http/server.key
 
 recursive-include synapse/res *
 recursive-include synapse/static *.css

README.rst

@@ -183,9 +183,8 @@ Using a reverse proxy with Synapse
 It is recommended to put a reverse proxy such as
 `nginx <https://nginx.org/en/docs/http/ngx_http_proxy_module.html>`_,
 `Apache <https://httpd.apache.org/docs/current/mod/mod_proxy_http.html>`_,
-`Caddy <https://caddyserver.com/docs/quick-starts/reverse-proxy>`_,
-`HAProxy <https://www.haproxy.org/>`_ or
-`relayd <https://man.openbsd.org/relayd.8>`_ in front of Synapse. One advantage of
+`Caddy <https://caddyserver.com/docs/quick-starts/reverse-proxy>`_ or
+`HAProxy <https://www.haproxy.org/>`_ in front of Synapse. One advantage of
 doing so is that it means that you can expose the default https port (443) to
 Matrix clients without needing to run Synapse with root privileges.
 
@@ -244,8 +243,6 @@ Then update the ``users`` table in the database::
 Synapse Development
 ===================
 
-Join our developer community on Matrix: `#synapse-dev:matrix.org <https://matrix.to/#/#synapse-dev:matrix.org>`_
-
 Before setting up a development environment for synapse, make sure you have the
 system dependencies (such as the python header files) installed - see
 `Installing from source <INSTALL.md#installing-from-source>`_.
@@ -264,43 +261,18 @@ to install using pip and a virtualenv::
     pip install -e ".[all,test]"
 
 This will run a process of downloading and installing all the needed
-dependencies into a virtual env. If any dependencies fail to install,
-try installing the failing modules individually::
+dependencies into a virtual env.
 
-    pip install -e "module-name"
-
-Once this is done, you may wish to run Synapse's unit tests to
-check that everything is installed correctly::
+Once this is done, you may wish to run Synapse's unit tests, to
+check that everything is installed as it should be::
 
     python -m twisted.trial tests
 
-This should end with a 'PASSED' result (note that exact numbers will
-differ)::
-
-    Ran 1337 tests in 716.064s
-
-    PASSED (skips=15, successes=1322)
-
-We recommend using the demo which starts 3 federated instances running on ports `8080` - `8082`
-
-    ./demo/start.sh
-
-(to stop, you can use `./demo/stop.sh`)
-
-If you just want to start a single instance of the app and run it directly::
-
-    # Create the homeserver.yaml config once
-    python -m synapse.app.homeserver \
-      --server-name my.domain.name \
-      --config-path homeserver.yaml \
-      --generate-config \
-      --report-stats=[yes|no]
-
-    # Start the app
-    python -m synapse.app.homeserver --config-path homeserver.yaml
-
+This should end with a 'PASSED' result::
 
+    Ran 1266 tests in 643.930s
 
+    PASSED (skips=15, successes=1251)
 
 Running the Integration Tests
 =============================

UPGRADE.rst

@@ -5,16 +5,6 @@ Before upgrading check if any special steps are required to upgrade from the
 version you currently have installed to the current version of Synapse. The extra
 instructions that may be required are listed later in this document.
 
-* Check that your versions of Python and PostgreSQL are still supported.
-
-  Synapse follows upstream lifecycles for `Python`_ and `PostgreSQL`_, and
-  removes support for versions which are no longer maintained.
-
-  The website https://endoflife.date also offers convenient summaries.
-
-  .. _Python: https://devguide.python.org/devcycle/#end-of-life-branches
-  .. _PostgreSQL: https://www.postgresql.org/support/versioning/
-
 * If Synapse was installed using `prebuilt packages
   <INSTALL.md#prebuilt-packages>`_, you will need to follow the normal process
   for upgrading those packages.
@@ -85,221 +75,6 @@ for example:
      wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb
      dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
 
-Upgrading to v1.29.0
-====================
-
-Requirement for X-Forwarded-Proto header
-----------------------------------------
-
-When using Synapse with a reverse proxy (in particular, when using the
-`x_forwarded` option on an HTTP listener), Synapse now expects to receive an
-`X-Forwarded-Proto` header on incoming HTTP requests. If it is not set, Synapse
-will log a warning on each received request.
-
-To avoid the warning, administrators using a reverse proxy should ensure that
-the reverse proxy sets `X-Forwarded-Proto` header to `https` or `http` to
-indicate the protocol used by the client. See the `reverse proxy documentation
-<docs/reverse_proxy.md>`_, where the example configurations have been updated to
-show how to set this header.
-
-(Users of `Caddy <https://caddyserver.com/>`_ are unaffected, since we believe it
-sets `X-Forwarded-Proto` by default.)
-
-Upgrading to v1.27.0
-====================
-
-Changes to callback URI for OAuth2 / OpenID Connect and SAML2
--------------------------------------------------------------
-
-This version changes the URI used for callbacks from OAuth2 and SAML2 identity providers:
-
-* If your server is configured for single sign-on via an OpenID Connect or OAuth2 identity
-  provider, you will need to add ``[synapse public baseurl]/_synapse/client/oidc/callback``
-  to the list of permitted "redirect URIs" at the identity provider.
-
-  See `docs/openid.md <docs/openid.md>`_ for more information on setting up OpenID
-  Connect.
-
-* If your server is configured for single sign-on via a SAML2 identity provider, you will
-  need to add ``[synapse public baseurl]/_synapse/client/saml2/authn_response`` as a permitted
-  "ACS location" (also known as "allowed callback URLs") at the identity provider.
-
-  The "Issuer" in the "AuthnRequest" to the SAML2 identity provider is also updated to
-  ``[synapse public baseurl]/_synapse/client/saml2/metadata.xml``. If your SAML2 identity
-  provider uses this property to validate or otherwise identify Synapse, its configuration
-  will need to be updated to use the new URL. Alternatively you could create a new, separate
-  "EntityDescriptor" in your SAML2 identity provider with the new URLs and leave the URLs in
-  the existing "EntityDescriptor" as they were.
-
-Changes to HTML templates
--------------------------
-
-The HTML templates for SSO and email notifications now have `Jinja2's autoescape <https://jinja.palletsprojects.com/en/2.11.x/api/#autoescaping>`_
-enabled for files ending in ``.html``, ``.htm``, and ``.xml``. If you have customised
-these templates and see issues when viewing them you might need to update them.
-It is expected that most configurations will need no changes.
-
-If you have customised the templates *names* for these templates, it is recommended
-to verify they end in ``.html`` to ensure autoescape is enabled.
-
-The above applies to the following templates:
-
-* ``add_threepid.html``
-* ``add_threepid_failure.html``
-* ``add_threepid_success.html``
-* ``notice_expiry.html``
-* ``notice_expiry.html``
-* ``notif_mail.html`` (which, by default, includes ``room.html`` and ``notif.html``)
-* ``password_reset.html``
-* ``password_reset_confirmation.html``
-* ``password_reset_failure.html``
-* ``password_reset_success.html``
-* ``registration.html``
-* ``registration_failure.html``
-* ``registration_success.html``
-* ``sso_account_deactivated.html``
-* ``sso_auth_bad_user.html``
-* ``sso_auth_confirm.html``
-* ``sso_auth_success.html``
-* ``sso_error.html``
-* ``sso_login_idp_picker.html``
-* ``sso_redirect_confirm.html``
-
-Upgrading to v1.26.0
-====================
-
-Rolling back to v1.25.0 after a failed upgrade
-----------------------------------------------
-
-v1.26.0 includes a lot of large changes. If something problematic occurs, you
-may want to roll-back to a previous version of Synapse. Because v1.26.0 also
-includes a new database schema version, reverting that version is also required
-alongside the generic rollback instructions mentioned above. In short, to roll
-back to v1.25.0 you need to:
-
-1. Stop the server
-2. Decrease the schema version in the database:
-
-   .. code:: sql
-
-      UPDATE schema_version SET version = 58;
-
-3. Delete the ignored users & chain cover data:
-
-   .. code:: sql
-
-      DROP TABLE IF EXISTS ignored_users;
-      UPDATE rooms SET has_auth_chain_index = false;
-
-   For PostgreSQL run:
-
-   .. code:: sql
-
-      TRUNCATE event_auth_chain_links;
-      TRUNCATE event_auth_chains;
-
-   For SQLite run:
-
-   .. code:: sql
-
-      DELETE FROM event_auth_chain_links;
-      DELETE FROM event_auth_chains;
-
-4. Mark the deltas as not run (so they will re-run on upgrade).
-
-   .. code:: sql
-
-      DELETE FROM applied_schema_deltas WHERE version = 59 AND file = "59/01ignored_user.py";
-      DELETE FROM applied_schema_deltas WHERE version = 59 AND file = "59/06chain_cover_index.sql";
-
-5. Downgrade Synapse by following the instructions for your installation method
-   in the "Rolling back to older versions" section above.
-
-Upgrading to v1.25.0
-====================
-
-Last release supporting Python 3.5
-----------------------------------
-
-This is the last release of Synapse which guarantees support with Python 3.5,
-which passed its upstream End of Life date several months ago.
-
-We will attempt to maintain support through March 2021, but without guarantees.
-
-In the future, Synapse will follow upstream schedules for ending support of
-older versions of Python and PostgreSQL. Please upgrade to at least Python 3.6
-and PostgreSQL 9.6 as soon as possible.
-
-Blacklisting IP ranges
-----------------------
-
-Synapse v1.25.0 includes new settings, ``ip_range_blacklist`` and
-``ip_range_whitelist``, for controlling outgoing requests from Synapse for federation,
-identity servers, push, and for checking key validity for third-party invite events.
-The previous setting, ``federation_ip_range_blacklist``, is deprecated. The new
-``ip_range_blacklist`` defaults to private IP ranges if it is not defined.
-
-If you have never customised ``federation_ip_range_blacklist`` it is recommended
-that you remove that setting.
-
-If you have customised ``federation_ip_range_blacklist`` you should update the
-setting name to ``ip_range_blacklist``.
-
-If you have a custom push server that is reached via private IP space you may
-need to customise ``ip_range_blacklist`` or ``ip_range_whitelist``.
-
-Upgrading to v1.24.0
-====================
-
-Custom OpenID Connect mapping provider breaking change
-------------------------------------------------------
-
-This release allows the OpenID Connect mapping provider to perform normalisation
-of the localpart of the Matrix ID. This allows for the mapping provider to
-specify different algorithms, instead of the [default way](https://matrix.org/docs/spec/appendices#mapping-from-other-character-sets).
-
-If your Synapse configuration uses a custom mapping provider
-(`oidc_config.user_mapping_provider.module` is specified and not equal to
-`synapse.handlers.oidc_handler.JinjaOidcMappingProvider`) then you *must* ensure
-that `map_user_attributes` of the mapping provider performs some normalisation
-of the `localpart` returned. To match previous behaviour you can use the
-`map_username_to_mxid_localpart` function provided by Synapse. An example is
-shown below:
-
-.. code-block:: python
-
-  from synapse.types import map_username_to_mxid_localpart
-
-  class MyMappingProvider:
-      def map_user_attributes(self, userinfo, token):
-          # ... your custom logic ...
-          sso_user_id = ...
-          localpart = map_username_to_mxid_localpart(sso_user_id)
-
-          return {"localpart": localpart}
-
-Removal historical Synapse Admin API
-------------------------------------
-
-Historically, the Synapse Admin API has been accessible under:
-
-* ``/_matrix/client/api/v1/admin``
-* ``/_matrix/client/unstable/admin``
-* ``/_matrix/client/r0/admin``
-* ``/_synapse/admin/v1``
-
-The endpoints with ``/_matrix/client/*`` prefixes have been removed as of v1.24.0.
-The Admin API is now only accessible under:
-
-* ``/_synapse/admin/v1``
-
-The only exception is the `/admin/whois` endpoint, which is
-`also available via the client-server API <https://matrix.org/docs/spec/client_server/r0.6.1#get-matrix-client-r0-admin-whois-userid>`_.
-
-The deprecation of the old endpoints was announced with Synapse 1.20.0 (released
-on 2020-09-22) and makes it easier for homeserver admins to lock down external
-access to the Admin API endpoints.
-
 Upgrading to v1.23.0
 ====================
 
@@ -312,7 +87,7 @@ then it should be modified based on the `structured logging documentation
 <https://github.com/matrix-org/synapse/blob/master/docs/structured_logging.md>`_.
 
 The ``structured`` and ``drains`` logging options are now deprecated and should
-be replaced by standard logging configuration of ``handlers`` and ``formatters``.
+be replaced by standard logging configuration of ``handlers`` and ``formatters`.
 
 A future will release of Synapse will make using ``structured: true`` an error.
 

changelog.d/8455.bugfix

@@ -0,0 +1 @@
+Fix fetching of E2E cross signing keys over federation when only one of the master key and device signing key is cached already.

changelog.d/8519.feature

@@ -0,0 +1 @@
+Add an admin api to delete a single file or files were not used for a defined time from server. Contributed by @dklimpel.

changelog.d/8539.feature

@@ -0,0 +1 @@
+Split admin API for reported events (`GET /_synapse/admin/v1/event_reports`) into detail and list endpoints. This is a breaking change to #8217 which was introduced in Synapse v1.21.0. Those who already use this API should check their scripts. Contributed by @dklimpel.

changelog.d/8559.misc

@@ -0,0 +1 @@
+Optimise `/createRoom` with multiple invited users.

changelog.d/8580.bugfix

@@ -0,0 +1 @@
+Fix a bug where Synapse would blindly forward bad responses from federation to clients when retrieving profile information.

changelog.d/8582.doc

@@ -0,0 +1 @@
+Instructions for Azure AD in the OpenID Connect documentation. Contributed by peterk.

changelog.d/8595.misc

@@ -0,0 +1 @@
+Implement and use an @lru_cache decorator.

changelog.d/8607.feature

@@ -0,0 +1 @@
+Support generating structured logs via the standard logging configuration.

changelog.d/8610.feature

@@ -0,0 +1 @@
+Add an admin APIs to allow server admins to list users' pushers. Contributed by @dklimpel.

changelog.d/8614.misc

@@ -0,0 +1 @@
+Don't instansiate Requester directly.

changelog.d/8615.misc

@@ -0,0 +1 @@
+Type hints for `RegistrationStore`.

changelog.d/8616.misc

@@ -0,0 +1 @@
+Change schema to support access tokens belonging to one user but granting access to another.

changelog.d/8620.bugfix

@@ -0,0 +1 @@
+Fix a bug where the account validity endpoint would silently fail if the user ID did not have an expiration time. It now returns a 400 error.

changelog.d/8621.misc

@@ -0,0 +1 @@
+Remove unused OPTIONS handlers.

changelog.d/8627.bugfix

@@ -0,0 +1 @@
+Fix email notifications for invites without local state.

changelog.d/8628.bugfix

@@ -0,0 +1 @@
+Fix handling of invalid group IDs to return a 400 rather than log an exception and return a 500.

changelog.d/8632.bugfix

@@ -0,0 +1 @@
+Fix handling of User-Agent headers that are invalid UTF-8, which caused user agents of users to not get correctly recorded.

changelog.d/8633.misc

@@ -0,0 +1 @@
+Run `mypy` as part of the lint.sh script.

changelog.d/8634.misc

@@ -0,0 +1 @@
+Correct Synapse's PyPI package name in the OpenID Connect installation instructions.

changelog.d/8635.doc

@@ -0,0 +1 @@
+Improve the sample configuration for single sign-on providers.

changelog.d/8639.misc

@@ -0,0 +1 @@
+Fix typos and spelling errors in the code.

changelog.d/8640.misc

@@ -0,0 +1 @@
+Reduce number of OpenTracing spans started.

changelog.d/8643.bugfix

@@ -0,0 +1 @@
+Fix a bug in the `joined_rooms` admin API if the user has never joined any rooms. The bug was introduced, along with the API, in v1.21.0.

changelog.d/8644.misc

@@ -0,0 +1 @@
+Add field `total` to device list in admin API.

changelog.d/8647.feature

@@ -0,0 +1 @@
+Add an admin API `GET /_synapse/admin/v1/users/<user_id>/media` to get information about uploaded media. Contributed by @dklimpel.

changelog.d/8655.misc

@@ -0,0 +1 @@
+Add more type hints to the application services code.

changelog.d/8657.doc

@@ -0,0 +1 @@
+Fix the filepath of Dex's example config and the link to Dex's Getting Started guide in the OpenID Connect docs.

changelog.d/8664.misc

@@ -0,0 +1 @@
+Tell Black to format code for Python 3.5.

changelog.d/8665.doc

@@ -0,0 +1 @@
+Note support for Python 3.9.

changelog.d/8666.doc

@@ -0,0 +1 @@
+Minor updates to docs on running tests.

changelog.d/8667.doc

@@ -0,0 +1 @@
+Interlink prometheus/grafana documentation.

changelog.d/8668.misc

@@ -0,0 +1 @@
+Reduce number of OpenTracing spans started.

changelog.d/8669.misc

@@ -0,0 +1 @@
+Don't pull event from DB when handling replication traffic.

changelog.d/8670.misc

@@ -0,0 +1 @@
+Reduce number of OpenTracing spans started.

changelog.d/8671.misc

@@ -0,0 +1 @@
+Abstract some invite-related code in preparation for landing knocking.

changelog.d/8679.misc

@@ -0,0 +1 @@
+Clarify representation of events in logfiles.

changelog.d/8680.misc

@@ -0,0 +1 @@
+Don't require `hiredis` package to be installed to run unit tests.

changelog.d/8682.bugfix

@@ -0,0 +1 @@
+Fix exception during handling multiple concurrent requests for remote media when using multiple media repositories.

changelog.d/8684.misc

@@ -0,0 +1 @@
+Fix typing info on cache call signature to accept `on_invalidate`.

changelog.d/8685.feature

@@ -0,0 +1 @@
+Support generating structured logs via the standard logging configuration.

changelog.d/8688.misc

@@ -0,0 +1 @@
+Abstract some invite-related code in preparation for landing knocking.

changelog.d/8689.feature

@@ -0,0 +1 @@
+Add an admin APIs to allow server admins to list users' pushers. Contributed by @dklimpel.

changelog.d/8690.misc

@@ -0,0 +1 @@
+Fail tests if they do not await coroutines.

changelog.d/8693.misc

@@ -0,0 +1 @@
+Add more type hints to the application services code.

contrib/cmdclient/console.py

@@ -92,7 +92,7 @@ class SynapseCmd(cmd.Cmd):
         return self.config["user"].split(":")[1]
 
     def do_config(self, line):
-        """Show the config for this client: "config"
+        """ Show the config for this client: "config"
         Edit a key value mapping: "config key value" e.g. "config token 1234"
         Config variables:
             user: The username to auth with.
@@ -360,7 +360,7 @@ class SynapseCmd(cmd.Cmd):
             print(e)
 
     def do_topic(self, line):
-        """ "topic [set|get] <roomid> [<newtopic>]"
+        """"topic [set|get] <roomid> [<newtopic>]"
         Set the topic for a room: topic set <roomid> <newtopic>
         Get the topic for a room: topic get <roomid>
         """
@@ -690,7 +690,7 @@ class SynapseCmd(cmd.Cmd):
         self._do_presence_state(2, line)
 
     def _parse(self, line, keys, force_keys=False):
-        """Parses the given line.
+        """ Parses the given line.
 
         Args:
             line : The line to parse
@@ -721,7 +721,7 @@ class SynapseCmd(cmd.Cmd):
         query_params={"access_token": None},
         alt_text=None,
     ):
-        """Runs an HTTP request and pretty prints the output.
+        """ Runs an HTTP request and pretty prints the output.
 
         Args:
             method: HTTP method

contrib/cmdclient/http.py

@@ -23,10 +23,11 @@ from twisted.web.http_headers import Headers
 
 
 class HttpClient:
-    """Interface for talking json over http"""
+    """ Interface for talking json over http
+    """
 
     def put_json(self, url, data):
-        """Sends the specifed json data using PUT
+        """ Sends the specifed json data using PUT
 
         Args:
             url (str): The URL to PUT data to.
@@ -40,7 +41,7 @@ class HttpClient:
         pass
 
     def get_json(self, url, args=None):
-        """Gets some json from the given host homeserver and path
+        """ Gets some json from the given host homeserver and path
 
         Args:
             url (str): The URL to GET data from.
@@ -57,7 +58,7 @@ class HttpClient:
 
 
 class TwistedHttpClient(HttpClient):
-    """Wrapper around the twisted HTTP client api.
+    """ Wrapper around the twisted HTTP client api.
 
     Attributes:
         agent (twisted.web.client.Agent): The twisted Agent used to send the
@@ -86,7 +87,8 @@ class TwistedHttpClient(HttpClient):
         defer.returnValue(json.loads(body))
 
     def _create_put_request(self, url, json_data, headers_dict={}):
-        """Wrapper of _create_request to issue a PUT request"""
+        """ Wrapper of _create_request to issue a PUT request
+        """
 
         if "Content-Type" not in headers_dict:
             raise defer.error(RuntimeError("Must include Content-Type header for PUTs"))
@@ -96,7 +98,8 @@ class TwistedHttpClient(HttpClient):
         )
 
     def _create_get_request(self, url, headers_dict={}):
-        """Wrapper of _create_request to issue a GET request"""
+        """ Wrapper of _create_request to issue a GET request
+        """
         return self._create_request("GET", url, headers_dict=headers_dict)
 
     @defer.inlineCallbacks
@@ -124,7 +127,8 @@ class TwistedHttpClient(HttpClient):
 
     @defer.inlineCallbacks
     def _create_request(self, method, url, producer=None, headers_dict={}):
-        """Creates and sends a request to the given url"""
+        """ Creates and sends a request to the given url
+        """
         headers_dict["User-Agent"] = ["Synapse Cmd Client"]
 
         retries_left = 5
@@ -181,7 +185,8 @@ class _RawProducer:
 
 
 class _JsonProducer:
-    """Used by the twisted http client to create the HTTP body from json"""
+    """ Used by the twisted http client to create the HTTP body from json
+    """
 
     def __init__(self, jsn):
         self.data = jsn

contrib/experiments/cursesio.py

@@ -63,7 +63,8 @@ class CursesStdIO:
         self.redraw()
 
     def redraw(self):
-        """method for redisplaying lines based on internal list of lines"""
+        """ method for redisplaying lines
+            based on internal list of lines """
 
         self.stdscr.clear()
         self.paintStatus(self.statusText)

contrib/experiments/test_messaging.py

@@ -56,7 +56,7 @@ def excpetion_errback(failure):
 
 
 class InputOutput:
-    """This is responsible for basic I/O so that a user can interact with
+    """ This is responsible for basic I/O so that a user can interact with
     the example app.
     """
 
@@ -68,7 +68,8 @@ class InputOutput:
         self.server = server
 
     def on_line(self, line):
-        """This is where we process commands."""
+        """ This is where we process commands.
+        """
 
         try:
             m = re.match(r"^join (\S+)$", line)
@@ -132,7 +133,7 @@ class IOLoggerHandler(logging.Handler):
 
 
 class Room:
-    """Used to store (in memory) the current membership state of a room, and
+    """ Used to store (in memory) the current membership state of a room, and
     which home servers we should send PDUs associated with the room to.
     """
 
@@ -147,7 +148,8 @@ class Room:
         self.have_got_metadata = False
 
     def add_participant(self, participant):
-        """Someone has joined the room"""
+        """ Someone has joined the room
+        """
         self.participants.add(participant)
         self.invited.discard(participant)
 
@@ -158,13 +160,14 @@ class Room:
             self.oldest_server = server
 
     def add_invited(self, invitee):
-        """Someone has been invited to the room"""
+        """ Someone has been invited to the room
+        """
         self.invited.add(invitee)
         self.servers.add(origin_from_ucid(invitee))
 
 
 class HomeServer(ReplicationHandler):
-    """A very basic home server implentation that allows people to join a
+    """ A very basic home server implentation that allows people to join a
     room and then invite other people.
     """
 
@@ -178,7 +181,8 @@ class HomeServer(ReplicationHandler):
         self.output = output
 
     def on_receive_pdu(self, pdu):
-        """We just received a PDU"""
+        """ We just received a PDU
+        """
         pdu_type = pdu.pdu_type
 
         if pdu_type == "sy.room.message":
@@ -195,20 +199,23 @@ class HomeServer(ReplicationHandler):
             )
 
     def _on_message(self, pdu):
-        """We received a message"""
+        """ We received a message
+        """
         self.output.print_line(
             "#%s %s %s" % (pdu.context, pdu.content["sender"], pdu.content["body"])
         )
 
     def _on_join(self, context, joinee):
-        """Someone has joined a room, either a remote user or a local user"""
+        """ Someone has joined a room, either a remote user or a local user
+        """
         room = self._get_or_create_room(context)
         room.add_participant(joinee)
 
         self.output.print_line("#%s %s %s" % (context, joinee, "*** JOINED"))
 
     def _on_invite(self, origin, context, invitee):
-        """Someone has been invited"""
+        """ Someone has been invited
+        """
         room = self._get_or_create_room(context)
         room.add_invited(invitee)
 
@@ -221,7 +228,8 @@ class HomeServer(ReplicationHandler):
 
     @defer.inlineCallbacks
     def send_message(self, room_name, sender, body):
-        """Send a message to a room!"""
+        """ Send a message to a room!
+        """
         destinations = yield self.get_servers_for_context(room_name)
 
         try:
@@ -239,7 +247,8 @@ class HomeServer(ReplicationHandler):
 
     @defer.inlineCallbacks
     def join_room(self, room_name, sender, joinee):
-        """Join a room!"""
+        """ Join a room!
+        """
         self._on_join(room_name, joinee)
 
         destinations = yield self.get_servers_for_context(room_name)
@@ -260,7 +269,8 @@ class HomeServer(ReplicationHandler):
 
     @defer.inlineCallbacks
     def invite_to_room(self, room_name, sender, invitee):
-        """Invite someone to a room!"""
+        """ Invite someone to a room!
+        """
         self._on_invite(self.server_name, room_name, invitee)
 
         destinations = yield self.get_servers_for_context(room_name)

contrib/jitsimeetbridge/jitsimeetbridge.py

@@ -193,12 +193,15 @@ class TrivialXmppClient:
         time.sleep(7)
         print("SSRC spammer started")
         while self.running:
-            ssrcMsg = "<presence to='%(tojid)s' xmlns='jabber:client'><x xmlns='http://jabber.org/protocol/muc'/><c xmlns='http://jabber.org/protocol/caps' hash='sha-1' node='http://jitsi.org/jitsimeet' ver='0WkSdhFnAUxrz4ImQQLdB80GFlE='/><nick xmlns='http://jabber.org/protocol/nick'>%(nick)s</nick><stats xmlns='http://jitsi.org/jitmeet/stats'><stat name='bitrate_download' value='175'/><stat name='bitrate_upload' value='176'/><stat name='packetLoss_total' value='0'/><stat name='packetLoss_download' value='0'/><stat name='packetLoss_upload' value='0'/></stats><media xmlns='http://estos.de/ns/mjs'><source type='audio' ssrc='%(assrc)s' direction='sendre'/><source type='video' ssrc='%(vssrc)s' direction='sendre'/></media></presence>" % {
-                "tojid": "%s@%s/%s" % (ROOMNAME, ROOMDOMAIN, self.shortJid),
-                "nick": self.userId,
-                "assrc": self.ssrcs["audio"],
-                "vssrc": self.ssrcs["video"],
-            }
+            ssrcMsg = (
+                "<presence to='%(tojid)s' xmlns='jabber:client'><x xmlns='http://jabber.org/protocol/muc'/><c xmlns='http://jabber.org/protocol/caps' hash='sha-1' node='http://jitsi.org/jitsimeet' ver='0WkSdhFnAUxrz4ImQQLdB80GFlE='/><nick xmlns='http://jabber.org/protocol/nick'>%(nick)s</nick><stats xmlns='http://jitsi.org/jitmeet/stats'><stat name='bitrate_download' value='175'/><stat name='bitrate_upload' value='176'/><stat name='packetLoss_total' value='0'/><stat name='packetLoss_download' value='0'/><stat name='packetLoss_upload' value='0'/></stats><media xmlns='http://estos.de/ns/mjs'><source type='audio' ssrc='%(assrc)s' direction='sendre'/><source type='video' ssrc='%(vssrc)s' direction='sendre'/></media></presence>"
+                % {
+                    "tojid": "%s@%s/%s" % (ROOMNAME, ROOMDOMAIN, self.shortJid),
+                    "nick": self.userId,
+                    "assrc": self.ssrcs["audio"],
+                    "vssrc": self.ssrcs["video"],
+                }
+            )
             res = self.sendIq(ssrcMsg)
             print("reply from ssrc announce: ", res)
             time.sleep(10)

contrib/prometheus/README.md

@@ -20,7 +20,6 @@ Add a new job to the main prometheus.conf file:
 ```
 
 ### for Prometheus v2
-
 Add a new job to the main prometheus.yml file:
 
 ```yaml
@@ -30,17 +29,14 @@ Add a new job to the main prometheus.yml file:
     scheme: "https"
 
     static_configs:
-    - targets: ["my.server.here:port"]
+    - targets: ['SERVER.LOCATION:PORT']
 ```
 
-An example of a Prometheus configuration with workers can be found in
-[metrics-howto.md](https://github.com/matrix-org/synapse/blob/master/docs/metrics-howto.md).
-
 To use `synapse.rules` add
 
 ```yaml
-  rule_files:
-    - "/PATH/TO/synapse-v2.rules"
+    rule_files:
+      - "/PATH/TO/synapse-v2.rules"
 ```
 
 Metrics are disabled by default when running synapse; they must be enabled

contrib/prometheus/consoles/synapse.html

@@ -9,7 +9,7 @@
 new PromConsole.Graph({
   node: document.querySelector("#process_resource_utime"),
   expr: "rate(process_cpu_seconds_total[2m]) * 100",
-  name: "[[job]]-[[index]]",
+  name: "[[job]]",  
   min: 0,
   max: 100,
   renderer: "line",
@@ -22,12 +22,12 @@ new PromConsole.Graph({
 </script>
 
 <h3>Memory</h3>
-<div id="process_resident_memory_bytes"></div>
+<div id="process_resource_maxrss"></div>
 <script>
 new PromConsole.Graph({
-  node: document.querySelector("#process_resident_memory_bytes"),
-  expr: "process_resident_memory_bytes",
-  name: "[[job]]-[[index]]",
+  node: document.querySelector("#process_resource_maxrss"),
+  expr: "process_psutil_rss:max",
+  name: "Maxrss",
   min: 0,
   renderer: "line",
   height: 150,
@@ -43,8 +43,8 @@ new PromConsole.Graph({
 <script>
 new PromConsole.Graph({
   node: document.querySelector("#process_fds"),
-  expr: "process_open_fds",
-  name: "[[job]]-[[index]]",
+  expr: "process_open_fds{job='synapse'}",
+  name: "FDs",
   min: 0,
   renderer: "line",
   height: 150,
@@ -62,8 +62,8 @@ new PromConsole.Graph({
 <script>
 new PromConsole.Graph({
   node: document.querySelector("#reactor_total_time"),
-  expr: "rate(python_twisted_reactor_tick_time_sum[2m])",
-  name: "[[job]]-[[index]]",
+  expr: "rate(python_twisted_reactor_tick_time:total[2m]) / 1000",
+  name: "time",
   max: 1,
   min: 0,
   renderer: "area",
@@ -80,8 +80,8 @@ new PromConsole.Graph({
 <script>
 new PromConsole.Graph({
   node: document.querySelector("#reactor_average_time"),
-  expr: "rate(python_twisted_reactor_tick_time_sum[2m]) / rate(python_twisted_reactor_tick_time_count[2m])",
-  name: "[[job]]-[[index]]",
+  expr: "rate(python_twisted_reactor_tick_time:total[2m]) / rate(python_twisted_reactor_tick_time:count[2m]) / 1000",
+  name: "time",
   min: 0,
   renderer: "line",
   height: 150,
@@ -97,14 +97,14 @@ new PromConsole.Graph({
 <script>
 new PromConsole.Graph({
   node: document.querySelector("#reactor_pending_calls"),
-  expr: "rate(python_twisted_reactor_pending_calls_sum[30s]) / rate(python_twisted_reactor_pending_calls_count[30s])",
-  name: "[[job]]-[[index]]",
+  expr: "rate(python_twisted_reactor_pending_calls:total[30s])/rate(python_twisted_reactor_pending_calls:count[30s])",
+  name: "calls",
   min: 0,
   renderer: "line",
   height: 150,
   yAxisFormatter: PromConsole.NumberFormatter.humanize,
   yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yTitle: "Pending Calls"
+  yTitle: "Pending Cals"
 })
 </script>
 
@@ -115,7 +115,7 @@ new PromConsole.Graph({
 <script>
 new PromConsole.Graph({
   node: document.querySelector("#synapse_storage_query_time"),
-  expr: "sum(rate(synapse_storage_query_time_count[2m])) by (verb)",
+  expr: "rate(synapse_storage_query_time:count[2m])",
   name: "[[verb]]",
   yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
   yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
@@ -129,8 +129,8 @@ new PromConsole.Graph({
 <script>
 new PromConsole.Graph({
   node: document.querySelector("#synapse_storage_transaction_time"),
-  expr: "topk(10, rate(synapse_storage_transaction_time_count[2m]))",
-  name: "[[job]]-[[index]] [[desc]]",
+  expr: "rate(synapse_storage_transaction_time:count[2m])",
+  name: "[[desc]]",
   min: 0,
   yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
   yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
@@ -140,12 +140,12 @@ new PromConsole.Graph({
 </script>
 
 <h3>Transaction execution time</h3>
-<div id="synapse_storage_transactions_time_sec"></div>
+<div id="synapse_storage_transactions_time_msec"></div>
 <script>
 new PromConsole.Graph({
-  node: document.querySelector("#synapse_storage_transactions_time_sec"),
-  expr: "rate(synapse_storage_transaction_time_sum[2m])",
-  name: "[[job]]-[[index]] [[desc]]",
+  node: document.querySelector("#synapse_storage_transactions_time_msec"),
+  expr: "rate(synapse_storage_transaction_time:total[2m]) / 1000",
+  name: "[[desc]]",
   min: 0,
   yAxisFormatter: PromConsole.NumberFormatter.humanize,
   yHoverFormatter: PromConsole.NumberFormatter.humanize,
@@ -154,33 +154,34 @@ new PromConsole.Graph({
 })
 </script>
 
-<h3>Average time waiting for database connection</h3>
-<div id="synapse_storage_avg_waiting_time"></div>
+<h3>Database scheduling latency</h3>
+<div id="synapse_storage_schedule_time"></div>
 <script>
 new PromConsole.Graph({
-  node: document.querySelector("#synapse_storage_avg_waiting_time"),
-  expr: "rate(synapse_storage_schedule_time_sum[2m]) / rate(synapse_storage_schedule_time_count[2m])",
-  name: "[[job]]-[[index]]",
+  node: document.querySelector("#synapse_storage_schedule_time"),
+  expr: "rate(synapse_storage_schedule_time:total[2m]) / 1000",
+  name: "Total latency",
   min: 0,
   yAxisFormatter: PromConsole.NumberFormatter.humanize,
   yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "s",
-  yTitle: "Time"
+  yUnits: "s/s",
+  yTitle: "Usage"
 })
 </script>
 
-<h3>Cache request rate</h3>
-<div id="synapse_cache_request_rate"></div>
+<h3>Cache hit ratio</h3>
+<div id="synapse_cache_ratio"></div>
 <script>
 new PromConsole.Graph({
-  node: document.querySelector("#synapse_cache_request_rate"),
-  expr: "rate(synapse_util_caches_cache:total[2m])",
-  name: "[[job]]-[[index]] [[name]]",
+  node: document.querySelector("#synapse_cache_ratio"),
+  expr: "rate(synapse_util_caches_cache:total[2m]) * 100",
+  name: "[[name]]",
   min: 0,
+  max: 100,
   yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
   yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yUnits: "rps",
-  yTitle: "Cache request rate"
+  yUnits: "%",
+  yTitle: "Percentage"
 })
 </script>
 
@@ -190,7 +191,7 @@ new PromConsole.Graph({
 new PromConsole.Graph({
   node: document.querySelector("#synapse_cache_size"),
   expr: "synapse_util_caches_cache:size",
-  name: "[[job]]-[[index]] [[name]]",
+  name: "[[name]]",
   yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
   yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
   yUnits: "",
@@ -205,8 +206,8 @@ new PromConsole.Graph({
 <script>
 new PromConsole.Graph({
   node: document.querySelector("#synapse_http_server_request_count_servlet"),
-  expr: "rate(synapse_http_server_in_flight_requests_count[2m])",
-  name: "[[job]]-[[index]] [[method]] [[servlet]]",
+  expr: "rate(synapse_http_server_request_count:servlet[2m])",
+  name: "[[servlet]]",
   yAxisFormatter: PromConsole.NumberFormatter.humanize,
   yHoverFormatter: PromConsole.NumberFormatter.humanize,
   yUnits: "req/s",
@@ -218,8 +219,8 @@ new PromConsole.Graph({
 <script>
 new PromConsole.Graph({
   node: document.querySelector("#synapse_http_server_request_count_servlet_minus_events"),
-  expr: "rate(synapse_http_server_in_flight_requests_count{servlet!=\"EventStreamRestServlet\", servlet!=\"SyncRestServlet\"}[2m])",
-  name: "[[job]]-[[index]] [[method]] [[servlet]]",
+  expr: "rate(synapse_http_server_request_count:servlet{servlet!=\"EventStreamRestServlet\", servlet!=\"SyncRestServlet\"}[2m])",
+  name: "[[servlet]]",
   yAxisFormatter: PromConsole.NumberFormatter.humanize,
   yHoverFormatter: PromConsole.NumberFormatter.humanize,
   yUnits: "req/s",
@@ -232,8 +233,8 @@ new PromConsole.Graph({
 <script>
 new PromConsole.Graph({
   node: document.querySelector("#synapse_http_server_response_time_avg"),
-  expr: "rate(synapse_http_server_response_time_seconds_sum[2m]) / rate(synapse_http_server_response_count[2m])",
-  name: "[[job]]-[[index]] [[servlet]]",
+  expr: "rate(synapse_http_server_response_time_seconds[2m]) / rate(synapse_http_server_response_count[2m]) / 1000",
+  name: "[[servlet]]",
   yAxisFormatter: PromConsole.NumberFormatter.humanize,
   yHoverFormatter: PromConsole.NumberFormatter.humanize,
   yUnits: "s/req",
@@ -276,7 +277,7 @@ new PromConsole.Graph({
 new PromConsole.Graph({
   node: document.querySelector("#synapse_http_server_response_ru_utime"),
   expr: "rate(synapse_http_server_response_ru_utime_seconds[2m])",
-  name: "[[job]]-[[index]] [[servlet]]",
+  name: "[[servlet]]",
   yAxisFormatter: PromConsole.NumberFormatter.humanize,
   yHoverFormatter: PromConsole.NumberFormatter.humanize,
   yUnits: "s/s",
@@ -291,7 +292,7 @@ new PromConsole.Graph({
 new PromConsole.Graph({
   node: document.querySelector("#synapse_http_server_response_db_txn_duration"),
   expr: "rate(synapse_http_server_response_db_txn_duration_seconds[2m])",
-  name: "[[job]]-[[index]] [[servlet]]",
+  name: "[[servlet]]",
   yAxisFormatter: PromConsole.NumberFormatter.humanize,
   yHoverFormatter: PromConsole.NumberFormatter.humanize,
   yUnits: "s/s",
@@ -305,8 +306,8 @@ new PromConsole.Graph({
 <script>
 new PromConsole.Graph({
   node: document.querySelector("#synapse_http_server_send_time_avg"),
-  expr: "rate(synapse_http_server_response_time_seconds_sum{servlet='RoomSendEventRestServlet'}[2m]) / rate(synapse_http_server_response_count{servlet='RoomSendEventRestServlet'}[2m])",
-  name: "[[job]]-[[index]] [[servlet]]",
+  expr: "rate(synapse_http_server_response_time_second{servlet='RoomSendEventRestServlet'}[2m]) / rate(synapse_http_server_response_count{servlet='RoomSendEventRestServlet'}[2m]) / 1000",
+  name: "[[servlet]]",
   yAxisFormatter: PromConsole.NumberFormatter.humanize,
   yHoverFormatter: PromConsole.NumberFormatter.humanize,
   yUnits: "s/req",
@@ -322,7 +323,7 @@ new PromConsole.Graph({
 new PromConsole.Graph({
   node: document.querySelector("#synapse_federation_client_sent"),
   expr: "rate(synapse_federation_client_sent[2m])",
-  name: "[[job]]-[[index]] [[type]]",
+  name: "[[type]]",
   yAxisFormatter: PromConsole.NumberFormatter.humanize,
   yHoverFormatter: PromConsole.NumberFormatter.humanize,
   yUnits: "req/s",
@@ -336,7 +337,7 @@ new PromConsole.Graph({
 new PromConsole.Graph({
   node: document.querySelector("#synapse_federation_server_received"),
   expr: "rate(synapse_federation_server_received[2m])",
-  name: "[[job]]-[[index]] [[type]]",
+  name: "[[type]]",
   yAxisFormatter: PromConsole.NumberFormatter.humanize,
   yHoverFormatter: PromConsole.NumberFormatter.humanize,
   yUnits: "req/s",
@@ -366,7 +367,7 @@ new PromConsole.Graph({
 new PromConsole.Graph({
   node: document.querySelector("#synapse_notifier_listeners"),
   expr: "synapse_notifier_listeners",
-  name: "[[job]]-[[index]]",
+  name: "listeners",
   min: 0,
   yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
   yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
@@ -381,7 +382,7 @@ new PromConsole.Graph({
 new PromConsole.Graph({
   node: document.querySelector("#synapse_notifier_notified_events"),
   expr: "rate(synapse_notifier_notified_events[2m])",
-  name: "[[job]]-[[index]]",
+  name: "events",
   yAxisFormatter: PromConsole.NumberFormatter.humanize,
   yHoverFormatter: PromConsole.NumberFormatter.humanize,
   yUnits: "events/s",

contrib/prometheus/synapse-v2.rules

@@ -58,21 +58,3 @@ groups:
     labels:
       type: "PDU"
     expr: 'synapse_federation_transaction_queue_pending_pdus + 0'
-
-  - record: synapse_storage_events_persisted_by_source_type
-    expr: sum without(type, origin_type, origin_entity) (synapse_storage_events_persisted_events_sep{origin_type="remote"})
-    labels:
-      type: remote
-  - record: synapse_storage_events_persisted_by_source_type
-    expr: sum without(type, origin_type, origin_entity) (synapse_storage_events_persisted_events_sep{origin_entity="*client*",origin_type="local"})
-    labels:
-      type: local
-  - record: synapse_storage_events_persisted_by_source_type
-    expr: sum without(type, origin_type, origin_entity) (synapse_storage_events_persisted_events_sep{origin_entity!="*client*",origin_type="local"})
-    labels:
-      type: bridges
-  - record: synapse_storage_events_persisted_by_event_type
-    expr: sum without(origin_entity, origin_type) (synapse_storage_events_persisted_events_sep)
-  - record: synapse_storage_events_persisted_by_origin
-    expr: sum without(type) (synapse_storage_events_persisted_events_sep)
-

debian/build_virtualenv

@@ -33,13 +33,11 @@ esac
 # Use --builtin-venv to use the better `venv` module from CPython 3.4+ rather
 # than the 2/3 compatible `virtualenv`.
 
-# Pin pip to 20.3.4 to fix breakage in 21.0 on py3.5 (xenial)
-
 dh_virtualenv \
     --install-suffix "matrix-synapse" \
     --builtin-venv \
     --python "$SNAKE" \
-    --upgrade-pip-to="20.3.4" \
+    --upgrade-pip \
     --preinstall="lxml" \
     --preinstall="mock" \
     --extra-pip-arg="--no-cache-dir" \
@@ -58,10 +56,10 @@ trap "rm -r $tmpdir" EXIT
 cp -r tests "$tmpdir"
 
 PYTHONPATH="$tmpdir" \
-    "${TARGET_PYTHON}" -m twisted.trial --reporter=text -j2 tests
+    "${TARGET_PYTHON}" -B -m twisted.trial --reporter=text -j2 tests
 
 # build the config file
-"${TARGET_PYTHON}" "${VIRTUALENV_DIR}/bin/generate_config" \
+"${TARGET_PYTHON}" -B "${VIRTUALENV_DIR}/bin/generate_config" \
         --config-dir="/etc/matrix-synapse" \
         --data-dir="/var/lib/matrix-synapse" |
     perl -pe '
@@ -87,7 +85,7 @@ PYTHONPATH="$tmpdir" \
 ' > "${PACKAGE_BUILD_DIR}/etc/matrix-synapse/homeserver.yaml"
 
 # build the log config file
-"${TARGET_PYTHON}" "${VIRTUALENV_DIR}/bin/generate_log_config" \
+"${TARGET_PYTHON}" -B "${VIRTUALENV_DIR}/bin/generate_log_config" \
         --output-file="${PACKAGE_BUILD_DIR}/etc/matrix-synapse/log.yaml"
 
 # add a dependency on the right version of python to substvars.

debian/changelog

@@ -1,68 +1,3 @@
-matrix-synapse-py3 (1.29.0) stable; urgency=medium
-
-  [ Jonathan de Jong ]
-  * Remove the python -B flag (don't generate bytecode) in scripts and documentation.
-
-  [ Synapse Packaging team ]
-  * New synapse release 1.29.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Mon, 08 Mar 2021 13:51:50 +0000
-
-matrix-synapse-py3 (1.28.0) stable; urgency=medium
-
-  * New synapse release 1.28.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Thu, 25 Feb 2021 10:21:57 +0000
-
-matrix-synapse-py3 (1.27.0) stable; urgency=medium
-
-  [ Dan Callahan ]
-  * Fix build on Ubuntu 16.04 LTS (Xenial).
-
-  [ Synapse Packaging team ]
-  * New synapse release 1.27.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 16 Feb 2021 13:11:28 +0000
-
-matrix-synapse-py3 (1.26.0) stable; urgency=medium
-
-  [ Richard van der Hoff ]
-  * Remove dependency on `python3-distutils`.
-
-  [ Synapse Packaging team ]
-  * New synapse release 1.26.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 27 Jan 2021 12:43:35 -0500
-
-matrix-synapse-py3 (1.25.0) stable; urgency=medium
-
-  [ Dan Callahan ]
-  * Update dependencies to account for the removal of the transitional
-    dh-systemd package from Debian Bullseye.
-
-  [ Synapse Packaging team ]
-  * New synapse release 1.25.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 13 Jan 2021 10:14:55 +0000
-
-matrix-synapse-py3 (1.24.0) stable; urgency=medium
-
-  * New synapse release 1.24.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 09 Dec 2020 10:14:30 +0000
-
-matrix-synapse-py3 (1.23.1) stable; urgency=medium
-
-  * New synapse release 1.23.1.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 09 Dec 2020 10:40:39 +0000
-
-matrix-synapse-py3 (1.23.0) stable; urgency=medium
-
-  * New synapse release 1.23.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 18 Nov 2020 11:41:28 +0000
-
 matrix-synapse-py3 (1.22.1) stable; urgency=medium
 
   * New synapse release 1.22.1.

debian/control

@@ -3,11 +3,9 @@ Section: contrib/python
 Priority: extra
 Maintainer: Synapse Packaging team <packages@matrix.org>
 # keep this list in sync with the build dependencies in docker/Dockerfile-dhvirtualenv.
-# TODO: Remove the dependency on dh-systemd after dropping support for Ubuntu xenial
-#       On all other supported releases, it's merely a transitional package which
-#       does nothing but depends on debhelper (> 9.20160709)
 Build-Depends:
- debhelper (>= 9.20160709) | dh-systemd,
+ debhelper (>= 9),
+ dh-systemd,
  dh-virtualenv (>= 1.1),
  libsystemd-dev,
  libpq-dev,
@@ -31,6 +29,7 @@ Pre-Depends: dpkg (>= 1.16.1)
 Depends:
  adduser,
  debconf,
+ python3-distutils|libpython3-stdlib (<< 3.6),
  ${misc:Depends},
  ${shlibs:Depends},
  ${synapse:pydepends},

debian/synctl.1

@@ -44,7 +44,7 @@ Configuration file may be generated as follows:
 .
 .nf
 
-$ python \-m synapse\.app\.homeserver \-c config\.yaml \-\-generate\-config \-\-server\-name=<server name>
+$ python \-B \-m synapse\.app\.homeserver \-c config\.yaml \-\-generate\-config \-\-server\-name=<server name>
 .
 .fi
 .

debian/synctl.ronn

@@ -41,7 +41,7 @@ process.
 
 Configuration file may be generated as follows:
 
-    $ python -m synapse.app.homeserver -c config.yaml --generate-config --server-name=<server name>
+    $ python -B -m synapse.app.homeserver -c config.yaml --generate-config --server-name=<server name>
 
 ## ENVIRONMENT
 

demo/webserver.py

@@ -0,0 +1,59 @@
+import argparse
+import BaseHTTPServer
+import os
+import SimpleHTTPServer
+import cgi, logging
+
+from daemonize import Daemonize
+
+
+class SimpleHTTPRequestHandlerWithPOST(SimpleHTTPServer.SimpleHTTPRequestHandler):
+    UPLOAD_PATH = "upload"
+
+    """
+    Accept all post request as file upload
+    """
+
+    def do_POST(self):
+
+        path = os.path.join(self.UPLOAD_PATH, os.path.basename(self.path))
+        length = self.headers["content-length"]
+        data = self.rfile.read(int(length))
+
+        with open(path, "wb") as fh:
+            fh.write(data)
+
+        self.send_response(200)
+        self.send_header("Content-Type", "application/json")
+        self.end_headers()
+
+        # Return the absolute path of the uploaded file
+        self.wfile.write('{"url":"/%s"}' % path)
+
+
+def setup():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("directory")
+    parser.add_argument("-p", "--port", dest="port", type=int, default=8080)
+    parser.add_argument("-P", "--pid-file", dest="pid", default="web.pid")
+    args = parser.parse_args()
+
+    # Get absolute path to directory to serve, as daemonize changes to '/'
+    os.chdir(args.directory)
+    dr = os.getcwd()
+
+    httpd = BaseHTTPServer.HTTPServer(("", args.port), SimpleHTTPRequestHandlerWithPOST)
+
+    def run():
+        os.chdir(dr)
+        httpd.serve_forever()
+
+    daemon = Daemonize(
+        app="synapse-webclient", pid=args.pid, action=run, auto_close_fds=False
+    )
+
+    daemon.start()
+
+
+if __name__ == "__main__":
+    setup()

docker/Dockerfile

@@ -11,7 +11,7 @@
 #    docker build -f docker/Dockerfile --build-arg PYTHON_VERSION=3.6 .
 #
 
-ARG PYTHON_VERSION=3.8
+ARG PYTHON_VERSION=3.7
 
 ###
 ### Stage 0: builder
@@ -28,18 +28,15 @@ RUN apt-get update && apt-get install -y \
     libwebp-dev \
     libxml++2.6-dev \
     libxslt1-dev \
-    rustc \
     zlib1g-dev \
  && rm -rf /var/lib/apt/lists/*
 
 # Build dependencies that are not available as wheels, to speed up rebuilds
 RUN pip install --prefix="/install" --no-warn-script-location \
-        cryptography \
         frozendict \
         jaeger-client \
         opentracing \
-        # Match the version constraints of Synapse
-        "prometheus_client>=0.4.0" \
+        prometheus-client \
         psycopg2 \
         pycparser \
         pyrsistent \
@@ -69,7 +66,6 @@ RUN apt-get update && apt-get install -y \
     libpq5 \
     libwebp6 \
     xmlsec1 \
-    libjemalloc2 \
  && rm -rf /var/lib/apt/lists/*
 
 COPY --from=builder /install /usr/local

docker/Dockerfile-dhvirtualenv

@@ -27,7 +27,6 @@ RUN env DEBIAN_FRONTEND=noninteractive apt-get install \
         wget
 
 # fetch and unpack the package
-# TODO: Upgrade to 1.2.2 once xenial is dropped
 RUN mkdir /dh-virtualenv
 RUN wget -q -O /dh-virtualenv.tar.gz https://github.com/spotify/dh-virtualenv/archive/ac6e1b1.tar.gz
 RUN tar -xv --strip-components=1 -C /dh-virtualenv -f /dh-virtualenv.tar.gz
@@ -51,22 +50,17 @@ FROM ${distro}
 ARG distro=""
 ENV distro ${distro}
 
-# Python < 3.7 assumes LANG="C" means ASCII-only and throws on printing unicode
-# http://bugs.python.org/issue19846
-ENV LANG C.UTF-8
-
 # Install the build dependencies
 #
 # NB: keep this list in sync with the list of build-deps in debian/control
 # TODO: it would be nice to do that automatically.
-# TODO: Remove the dh-systemd stanza after dropping support for Ubuntu xenial
-#       it's a transitional package on all other, more recent releases
 RUN apt-get update -qq -o Acquire::Languages=none \
     && env DEBIAN_FRONTEND=noninteractive apt-get install \
         -yqq --no-install-recommends -o Dpkg::Options::=--force-unsafe-io \
         build-essential \
         debhelper \
         devscripts \
+        dh-systemd \
         libsystemd-dev \
         lsb-release \
         pkg-config \
@@ -75,11 +69,7 @@ RUN apt-get update -qq -o Acquire::Languages=none \
         python3-setuptools \
         python3-venv \
         sqlite3 \
-        libpq-dev \
-        xmlsec1 \
-    && ( env DEBIAN_FRONTEND=noninteractive apt-get install \
-         -yqq --no-install-recommends -o Dpkg::Options::=--force-unsafe-io \
-         dh-systemd || true )
+        libpq-dev
 
 COPY --from=builder /dh-virtualenv_1.2~dev-1_all.deb /
 

docker/README.md

@@ -11,6 +11,7 @@ The image also does *not* provide a TURN server.
 By default, the image expects a single volume, located at ``/data``, that will hold:
 
 * configuration files;
+* temporary files during uploads;
 * uploaded media and thumbnails;
 * the SQLite database if you do not configure postgres;
 * the appservices configuration.
@@ -204,8 +205,3 @@ healthcheck:
   timeout: 10s
   retries: 3
 ```
-
-## Using jemalloc
-
-Jemalloc is embedded in the image and will be used instead of the default allocator.
-You can read about jemalloc by reading the Synapse [README](../README.md)

docker/conf/homeserver.yaml

@@ -89,6 +89,7 @@ federation_rc_concurrent: 3
 ## Files ##
 
 media_store_path: "/data/media"
+uploads_path: "/data/uploads"
 max_upload_size: "{{ SYNAPSE_MAX_UPLOAD_SIZE or "50M" }}"
 max_image_pixels: "32M"
 dynamic_thumbnails: false
@@ -197,10 +198,12 @@ old_signing_keys: {}
 key_refresh_interval: "1d" # 1 Day.
 
 # The trusted servers to download signing keys from.
-trusted_key_servers:
-  - server_name: matrix.org
-    verify_keys:
-      "ed25519:auto": "Noi6WqcDj0QmPxCNQqgezwTlBKrfqehY1u2FyWP9uYw"
+perspectives:
+  servers:
+    "matrix.org":
+      verify_keys:
+        "ed25519:auto":
+          key: "Noi6WqcDj0QmPxCNQqgezwTlBKrfqehY1u2FyWP9uYw"
 
 password_config:
    enabled: true

docker/start.py

@@ -3,7 +3,6 @@
 import codecs
 import glob
 import os
-import platform
 import subprocess
 import sys
 
@@ -214,13 +213,6 @@ def main(args, environ):
     if "-m" not in args:
         args = ["-m", synapse_worker] + args
 
-    jemallocpath = "/usr/lib/%s-linux-gnu/libjemalloc.so.2" % (platform.machine(),)
-
-    if os.path.isfile(jemallocpath):
-        environ["LD_PRELOAD"] = jemallocpath
-    else:
-        log("Could not find %s, will not use" % (jemallocpath,))
-
     # if there are no config files passed to synapse, try adding the default file
     if not any(p.startswith("--config-path") or p.startswith("-c") for p in args):
         config_dir = environ.get("SYNAPSE_CONFIG_DIR", "/data")
@@ -256,9 +248,9 @@ running with 'migrate_config'. See the README for more details.
     args = ["python"] + args
     if ownership is not None:
         args = ["gosu", ownership] + args
-        os.execve("/usr/sbin/gosu", args, environ)
+        os.execv("/usr/sbin/gosu", args)
     else:
-        os.execve("/usr/local/bin/python", args, environ)
+        os.execv("/usr/local/bin/python", args)
 
 
 if __name__ == "__main__":

docs/admin_api/event_reports.md

@@ -1,172 +0,0 @@
-# Show reported events
-
-This API returns information about reported events.
-
-The api is:
-```
-GET /_synapse/admin/v1/event_reports?from=0&limit=10
-```
-To use it, you will need to authenticate by providing an `access_token` for a
-server admin: see [README.rst](README.rst).
-
-It returns a JSON body like the following:
-
-```json
-{
-    "event_reports": [
-        {
-            "event_id": "$bNUFCwGzWca1meCGkjp-zwslF-GfVcXukvRLI1_FaVY",
-            "id": 2,
-            "reason": "foo",
-            "score": -100,
-            "received_ts": 1570897107409,
-            "canonical_alias": "#alias1:matrix.org",
-            "room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
-            "name": "Matrix HQ",
-            "sender": "@foobar:matrix.org",
-            "user_id": "@foo:matrix.org"
-        },
-        {
-            "event_id": "$3IcdZsDaN_En-S1DF4EMCy3v4gNRKeOJs8W5qTOKj4I",
-            "id": 3,
-            "reason": "bar",
-            "score": -100,
-            "received_ts": 1598889612059,
-            "canonical_alias": "#alias2:matrix.org",
-            "room_id": "!eGvUQuTCkHGVwNMOjv:matrix.org",
-            "name": "Your room name here",
-            "sender": "@foobar:matrix.org",
-            "user_id": "@bar:matrix.org"
-        }
-    ],
-    "next_token": 2,
-    "total": 4
-}
-```
-
-To paginate, check for `next_token` and if present, call the endpoint again with `from`
-set to the value of `next_token`. This will return a new page.
-
-If the endpoint does not return a `next_token` then there are no more reports to
-paginate through.
-
-**URL parameters:**
-
-* `limit`: integer - Is optional but is used for pagination, denoting the maximum number
-  of items to return in this call. Defaults to `100`.
-* `from`: integer - Is optional but used for pagination, denoting the offset in the
-  returned results. This should be treated as an opaque value and not explicitly set to
-  anything other than the return value of `next_token` from a previous call. Defaults to `0`.
-* `dir`: string - Direction of event report order. Whether to fetch the most recent
-  first (`b`) or the oldest first (`f`). Defaults to `b`.
-* `user_id`: string - Is optional and filters to only return users with user IDs that
-  contain this value. This is the user who reported the event and wrote the reason.
-* `room_id`: string - Is optional and filters to only return rooms with room IDs that
-  contain this value.
-
-**Response**
-
-The following fields are returned in the JSON response body:
-
-* `id`: integer - ID of event report.
-* `received_ts`: integer - The timestamp (in milliseconds since the unix epoch) when this
-  report was sent.
-* `room_id`: string - The ID of the room in which the event being reported is located.
-* `name`: string - The name of the room.
-* `event_id`: string - The ID of the reported event.
-* `user_id`: string - This is the user who reported the event and wrote the reason.
-* `reason`: string - Comment made by the `user_id` in this report. May be blank.
-* `score`: integer - Content is reported based upon a negative score, where -100 is
-  "most offensive" and 0 is "inoffensive".
-* `sender`: string - This is the ID of the user who sent the original message/event that
-  was reported.
-* `canonical_alias`: string - The canonical alias of the room. `null` if the room does not
-  have a canonical alias set.
-* `next_token`: integer - Indication for pagination. See above.
-* `total`: integer - Total number of event reports related to the query
-  (`user_id` and `room_id`).
-
-# Show details of a specific event report
-
-This API returns information about a specific event report.
-
-The api is:
-```
-GET /_synapse/admin/v1/event_reports/<report_id>
-```
-To use it, you will need to authenticate by providing an `access_token` for a
-server admin: see [README.rst](README.rst).
-
-It returns a JSON body like the following:
-
-```jsonc
-{
-    "event_id": "$bNUFCwGzWca1meCGkjp-zwslF-GfVcXukvRLI1_FaVY",
-    "event_json": {
-        "auth_events": [
-            "$YK4arsKKcc0LRoe700pS8DSjOvUT4NDv0HfInlMFw2M",
-            "$oggsNXxzPFRE3y53SUNd7nsj69-QzKv03a1RucHu-ws"
-        ],
-        "content": {
-            "body": "matrix.org: This Week in Matrix",
-            "format": "org.matrix.custom.html",
-            "formatted_body": "<strong>matrix.org</strong>:<br><a href=\"https://matrix.org/blog/\"><strong>This Week in Matrix</strong></a>",
-            "msgtype": "m.notice"
-        },
-        "depth": 546,
-        "hashes": {
-            "sha256": "xK1//xnmvHJIOvbgXlkI8eEqdvoMmihVDJ9J4SNlsAw"
-        },
-        "origin": "matrix.org",
-        "origin_server_ts": 1592291711430,
-        "prev_events": [
-            "$YK4arsKKcc0LRoe700pS8DSjOvUT4NDv0HfInlMFw2M"
-        ],
-        "prev_state": [],
-        "room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
-        "sender": "@foobar:matrix.org",
-        "signatures": {
-            "matrix.org": {
-                "ed25519:a_JaEG": "cs+OUKW/iHx5pEidbWxh0UiNNHwe46Ai9LwNz+Ah16aWDNszVIe2gaAcVZfvNsBhakQTew51tlKmL2kspXk/Dg"
-            }
-        },
-        "type": "m.room.message",
-        "unsigned": {
-            "age_ts": 1592291711430,
-        }
-    },
-    "id": <report_id>,
-    "reason": "foo",
-    "score": -100,
-    "received_ts": 1570897107409,
-    "canonical_alias": "#alias1:matrix.org",
-    "room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
-    "name": "Matrix HQ",
-    "sender": "@foobar:matrix.org",
-    "user_id": "@foo:matrix.org"
-}
-```
-
-**URL parameters:**
-
-* `report_id`: string - The ID of the event report.
-
-**Response**
-
-The following fields are returned in the JSON response body:
-
-* `id`: integer - ID of event report.
-* `received_ts`: integer - The timestamp (in milliseconds since the unix epoch) when this
-  report was sent.
-* `room_id`: string - The ID of the room in which the event being reported is located.
-* `name`: string - The name of the room.
-* `event_id`: string - The ID of the reported event.
-* `user_id`: string - This is the user who reported the event and wrote the reason.
-* `reason`: string - Comment made by the `user_id` in this report. May be blank.
-* `score`: integer - Content is reported based upon a negative score, where -100 is
-  "most offensive" and 0 is "inoffensive".
-* `sender`: string - This is the ID of the user who sent the original message/event that
-  was reported.
-* `canonical_alias`: string - The canonical alias of the room. `null` if the room does not
-  have a canonical alias set.
-* `event_json`: object - Details of the original event that was reported.

docs/admin_api/event_reports.rst

@@ -0,0 +1,165 @@
+Show reported events
+====================
+
+This API returns information about reported events.
+
+The api is::
+
+    GET /_synapse/admin/v1/event_reports?from=0&limit=10
+
+To use it, you will need to authenticate by providing an ``access_token`` for a
+server admin: see `README.rst <README.rst>`_.
+
+It returns a JSON body like the following:
+
+.. code:: jsonc
+
+    {
+        "event_reports": [
+            {
+                "event_id": "$bNUFCwGzWca1meCGkjp-zwslF-GfVcXukvRLI1_FaVY",
+                "id": 2,
+                "reason": "foo",
+                "score": -100,
+                "received_ts": 1570897107409,
+                "canonical_alias": "#alias1:matrix.org",
+                "room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
+                "name": "Matrix HQ",
+                "sender": "@foobar:matrix.org",
+                "user_id": "@foo:matrix.org"
+            },
+            {
+                "event_id": "$3IcdZsDaN_En-S1DF4EMCy3v4gNRKeOJs8W5qTOKj4I",
+                "id": 3,
+                "reason": "bar",
+                "score": -100,
+                "received_ts": 1598889612059,
+                "canonical_alias": "#alias2:matrix.org",
+                "room_id": "!eGvUQuTCkHGVwNMOjv:matrix.org",
+                "name": "Your room name here",
+                "sender": "@foobar:matrix.org",
+                "user_id": "@bar:matrix.org"
+            }
+        ],
+        "next_token": 2,
+        "total": 4
+    }
+
+To paginate, check for ``next_token`` and if present, call the endpoint again
+with ``from`` set to the value of ``next_token``. This will return a new page.
+
+If the endpoint does not return a ``next_token`` then there are no more
+reports to paginate through.
+
+**URL parameters:**
+
+- ``limit``: integer - Is optional but is used for pagination,
+  denoting the maximum number of items to return in this call. Defaults to ``100``.
+- ``from``: integer - Is optional but used for pagination,
+  denoting the offset in the returned results. This should be treated as an opaque value and
+  not explicitly set to anything other than the return value of ``next_token`` from a previous call.
+  Defaults to ``0``.
+- ``dir``: string - Direction of event report order. Whether to fetch the most recent first (``b``) or the
+  oldest first (``f``). Defaults to ``b``.
+- ``user_id``: string - Is optional and filters to only return users with user IDs that contain this value.
+  This is the user who reported the event and wrote the reason.
+- ``room_id``: string - Is optional and filters to only return rooms with room IDs that contain this value.
+
+**Response**
+
+The following fields are returned in the JSON response body:
+
+- ``id``: integer - ID of event report.
+- ``received_ts``: integer - The timestamp (in milliseconds since the unix epoch) when this report was sent.
+- ``room_id``: string - The ID of the room in which the event being reported is located.
+- ``name``: string - The name of the room.
+- ``event_id``: string - The ID of the reported event.
+- ``user_id``: string - This is the user who reported the event and wrote the reason.
+- ``reason``: string - Comment made by the ``user_id`` in this report. May be blank.
+- ``score``: integer - Content is reported based upon a negative score, where -100 is "most offensive" and 0 is "inoffensive".
+- ``sender``: string - This is the ID of the user who sent the original message/event that was reported.
+- ``canonical_alias``: string - The canonical alias of the room. ``null`` if the room does not have a canonical alias set.
+- ``next_token``: integer - Indication for pagination. See above.
+- ``total``: integer - Total number of event reports related to the query (``user_id`` and ``room_id``).
+
+Show details of a specific event report
+=======================================
+
+This API returns information about a specific event report.
+
+The api is::
+
+    GET /_synapse/admin/v1/event_reports/<report_id>
+
+To use it, you will need to authenticate by providing an ``access_token`` for a
+server admin: see `README.rst <README.rst>`_.
+
+It returns a JSON body like the following:
+
+.. code:: jsonc
+
+    {
+        "event_id": "$bNUFCwGzWca1meCGkjp-zwslF-GfVcXukvRLI1_FaVY",
+        "event_json": {
+            "auth_events": [
+                "$YK4arsKKcc0LRoe700pS8DSjOvUT4NDv0HfInlMFw2M",
+                "$oggsNXxzPFRE3y53SUNd7nsj69-QzKv03a1RucHu-ws"
+            ],
+            "content": {
+                "body": "matrix.org: This Week in Matrix",
+                "format": "org.matrix.custom.html",
+                "formatted_body": "<strong>matrix.org</strong>:<br><a href=\"https://matrix.org/blog/\"><strong>This Week in Matrix</strong></a>",
+                "msgtype": "m.notice"
+            },
+            "depth": 546,
+            "hashes": {
+                "sha256": "xK1//xnmvHJIOvbgXlkI8eEqdvoMmihVDJ9J4SNlsAw"
+            },
+            "origin": "matrix.org",
+            "origin_server_ts": 1592291711430,
+            "prev_events": [
+                "$YK4arsKKcc0LRoe700pS8DSjOvUT4NDv0HfInlMFw2M"
+            ],
+            "prev_state": [],
+            "room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
+            "sender": "@foobar:matrix.org",
+            "signatures": {
+                "matrix.org": {
+                    "ed25519:a_JaEG": "cs+OUKW/iHx5pEidbWxh0UiNNHwe46Ai9LwNz+Ah16aWDNszVIe2gaAcVZfvNsBhakQTew51tlKmL2kspXk/Dg"
+                }
+            },
+            "type": "m.room.message",
+            "unsigned": {
+                "age_ts": 1592291711430,
+            }
+        },
+        "id": <report_id>,
+        "reason": "foo",
+        "score": -100,
+        "received_ts": 1570897107409,
+        "canonical_alias": "#alias1:matrix.org",
+        "room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
+        "name": "Matrix HQ",
+        "sender": "@foobar:matrix.org",
+        "user_id": "@foo:matrix.org"
+    }
+
+**URL parameters:**
+
+- ``report_id``: string - The ID of the event report.
+
+**Response**
+
+The following fields are returned in the JSON response body:
+
+- ``id``: integer - ID of event report.
+- ``received_ts``: integer - The timestamp (in milliseconds since the unix epoch) when this report was sent.
+- ``room_id``: string - The ID of the room in which the event being reported is located.
+- ``name``: string - The name of the room.
+- ``event_id``: string - The ID of the reported event.
+- ``user_id``: string - This is the user who reported the event and wrote the reason.
+- ``reason``: string - Comment made by the ``user_id`` in this report. May be blank.
+- ``score``: integer - Content is reported based upon a negative score, where -100 is "most offensive" and 0 is "inoffensive".
+- ``sender``: string - This is the ID of the user who sent the original message/event that was reported.
+- ``canonical_alias``: string - The canonical alias of the room. ``null`` if the room does not have a canonical alias set.
+- ``event_json``: object - Details of the original event that was reported.

docs/admin_api/media_admin_api.md

@@ -1,25 +1,6 @@
-# Contents
-- [Querying media](#querying-media)
-  * [List all media in a room](#list-all-media-in-a-room)
-  * [List all media uploaded by a user](#list-all-media-uploaded-by-a-user)
-- [Quarantine media](#quarantine-media)
-  * [Quarantining media by ID](#quarantining-media-by-id)
-  * [Quarantining media in a room](#quarantining-media-in-a-room)
-  * [Quarantining all media of a user](#quarantining-all-media-of-a-user)
-  * [Protecting media from being quarantined](#protecting-media-from-being-quarantined)
-- [Delete local media](#delete-local-media)
-  * [Delete a specific local media](#delete-a-specific-local-media)
-  * [Delete local media by date or size](#delete-local-media-by-date-or-size)
-- [Purge Remote Media API](#purge-remote-media-api)
-
-# Querying media
-
-These APIs allow extracting media information from the homeserver.
-
-## List all media in a room
+# List all media in a room
 
 This API gets a list of known media in a room.
-However, it only shows media from unencrypted events or rooms.
 
 The API is:
 ```
@@ -29,25 +10,19 @@ To use it, you will need to authenticate by providing an `access_token` for a
 server admin: see [README.rst](README.rst).
 
 The API returns a JSON body like the following:
-```json
+```
 {
-  "local": [
-    "mxc://localhost/xwvutsrqponmlkjihgfedcba",
-    "mxc://localhost/abcdefghijklmnopqrstuvwx"
-  ],
-  "remote": [
-    "mxc://matrix.org/xwvutsrqponmlkjihgfedcba",
-    "mxc://matrix.org/abcdefghijklmnopqrstuvwx"
-  ]
+    "local": [
+        "mxc://localhost/xwvutsrqponmlkjihgfedcba",
+        "mxc://localhost/abcdefghijklmnopqrstuvwx"
+    ],
+    "remote": [
+        "mxc://matrix.org/xwvutsrqponmlkjihgfedcba",
+        "mxc://matrix.org/abcdefghijklmnopqrstuvwx"
+    ]
 }
 ```
 
-## List all media uploaded by a user
-
-Listing all media that has been uploaded by a local user can be achieved through
-the use of the [List media of a user](user_admin_api.rst#list-media-of-a-user)
-Admin API.
-
 # Quarantine media
 
 Quarantining media means that it is marked as inaccessible by users. It applies
@@ -72,7 +47,7 @@ form of `abcdefg12345...`.
 
 Response:
 
-```json
+```
 {}
 ```
 
@@ -92,18 +67,14 @@ Where `room_id` is in the form of `!roomid12345:example.org`.
 
 Response:
 
-```json
+```
 {
-  "num_quarantined": 10
+  "num_quarantined": 10  # The number of media items successfully quarantined
 }
 ```
 
-The following fields are returned in the JSON response body:
-
-* `num_quarantined`: integer - The number of media items successfully quarantined
-
 Note that there is a legacy endpoint, `POST
-/_synapse/admin/v1/quarantine_media/<room_id>`, that operates the same.
+/_synapse/admin/v1/quarantine_media/<room_id >`, that operates the same.
 However, it is deprecated and may be removed in a future release.
 
 ## Quarantining all media of a user
@@ -120,52 +91,23 @@ POST /_synapse/admin/v1/user/<user_id>/media/quarantine
 {}
 ```
 
-URL Parameters
-
-* `user_id`: string - User ID in the form of `@bob:example.org`
+Where `user_id` is in the form of `@bob:example.org`.
 
 Response:
 
-```json
+```
 {
-  "num_quarantined": 10
+  "num_quarantined": 10  # The number of media items successfully quarantined
 }
 ```
 
-The following fields are returned in the JSON response body:
-
-* `num_quarantined`: integer - The number of media items successfully quarantined
-
-## Protecting media from being quarantined
-
-This API protects a single piece of local media from being quarantined using the
-above APIs. This is useful for sticker packs and other shared media which you do
-not want to get quarantined, especially when
-[quarantining media in a room](#quarantining-media-in-a-room).
-
-Request:
-
-```
-POST /_synapse/admin/v1/media/protect/<media_id>
-
-{}
-```
-
-Where `media_id` is in the  form of `abcdefg12345...`.
-
-Response:
-
-```json
-{}
-```
-
 # Delete local media
 This API deletes the *local* media from the disk of your own server.
 This includes any local thumbnails and copies of media downloaded from
 remote homeservers.
 This API will not affect media that has been uploaded to external
 media repositories (e.g https://github.com/turt2live/matrix-media-repo/).
-See also [Purge Remote Media API](#purge-remote-media-api).
+See also [purge_remote_media.rst](purge_remote_media.rst).
 
 ## Delete a specific local media
 Delete a specific `media_id`.
@@ -186,12 +128,12 @@ URL Parameters
 Response:
 
 ```json
-{
-  "deleted_media": [
-    "abcdefghijklmnopqrstuvwx"
-  ],
-  "total": 1
-}
+    {
+       "deleted_media": [
+          "abcdefghijklmnopqrstuvwx"
+       ],
+       "total": 1
+    }
 ```
 
 The following fields are returned in the JSON response body:
@@ -224,51 +166,16 @@ If `false` these files will be deleted. Defaults to `true`.
 Response:
 
 ```json
-{
-  "deleted_media": [
-    "abcdefghijklmnopqrstuvwx",
-    "abcdefghijklmnopqrstuvwz"
-  ],
-  "total": 2
-}
+    {
+       "deleted_media": [
+          "abcdefghijklmnopqrstuvwx",
+          "abcdefghijklmnopqrstuvwz"
+       ],
+       "total": 2
+    }
 ```
 
 The following fields are returned in the JSON response body:
 
 * `deleted_media`: an array of strings - List of deleted `media_id`
 * `total`: integer - Total number of deleted `media_id`
-
-# Purge Remote Media API
-
-The purge remote media API allows server admins to purge old cached remote media.
-
-The API is:
-
-```
-POST /_synapse/admin/v1/purge_media_cache?before_ts=<unix_timestamp_in_ms>
-
-{}
-```
-
-URL Parameters
-
-* `unix_timestamp_in_ms`: string representing a positive integer - Unix timestamp in ms.
-All cached media that was last accessed before this timestamp will be removed.
-
-Response:
-
-```json
-{
-  "deleted": 10
-}
-```
-
-The following fields are returned in the JSON response body:
-
-* `deleted`: integer - The number of media items successfully deleted
-
-To use it, you will need to authenticate by providing an `access_token` for a
-server admin: see [README.rst](README.rst).
-
-If the user re-requests purged remote media, synapse will re-request the media
-from the originating server.

docs/admin_api/purge_remote_media.rst

@@ -0,0 +1,20 @@
+Purge Remote Media API
+======================
+
+The purge remote media API allows server admins to purge old cached remote
+media.
+
+The API is::
+
+    POST /_synapse/admin/v1/purge_media_cache?before_ts=<unix_timestamp_in_ms>
+
+    {}
+
+\... which will remove all cached media that was last accessed before
+``<unix_timestamp_in_ms>``.
+
+To use it, you will need to authenticate by providing an ``access_token`` for a
+server admin: see `README.rst <README.rst>`_.
+
+If the user re-requests purged remote media, synapse will re-request the media
+from the originating server.

docs/admin_api/purge_room.md

@@ -1,13 +1,12 @@
-Deprecated: Purge room API
-==========================
-
-**The old Purge room API is deprecated and will be removed in a future release.
-See the new [Delete Room API](rooms.md#delete-room-api) for more details.**
+Purge room API
+==============
 
 This API will remove all trace of a room from your database.
 
 All local users must have left the room before it can be removed.
 
+See also: [Delete Room API](rooms.md#delete-room-api)
+
 The API is:
 
 ```

docs/admin_api/register_api.rst

@@ -18,8 +18,7 @@ To fetch the nonce, you need to request one from the API::
 
 Once you have the nonce, you can make a ``POST`` to the same URL with a JSON
 body containing the nonce, username, password, whether they are an admin
-(optional, False by default), and a HMAC digest of the content. Also you can
-set the displayname (optional, ``username`` by default).
+(optional, False by default), and a HMAC digest of the content.
 
 As an example::
 
@@ -27,7 +26,6 @@ As an example::
   > {
      "nonce": "thisisanonce",
      "username": "pepper_roni",
-     "displayname": "Pepper Roni",
      "password": "pizza",
      "admin": true,
      "mac": "mac_digest_here"

docs/admin_api/rooms.md

@@ -1,17 +1,3 @@
-# Contents
-- [List Room API](#list-room-api)
-  * [Parameters](#parameters)
-  * [Usage](#usage)
-- [Room Details API](#room-details-api)
-- [Room Members API](#room-members-api)
-- [Delete Room API](#delete-room-api)
-  * [Parameters](#parameters-1)
-  * [Response](#response)
-  * [Undoing room shutdowns](#undoing-room-shutdowns)
-- [Make Room Admin API](#make-room-admin-api)
-- [Forward Extremities Admin API](#forward-extremities-admin-api)
-- [Event Context API](#event-context-api)
-
 # List Room API
 
 The List Room admin API allows server admins to get a list of rooms on their
@@ -90,7 +76,7 @@ GET /_synapse/admin/v1/rooms
 
 Response:
 
-```jsonc
+```
 {
   "rooms": [
     {
@@ -142,7 +128,7 @@ GET /_synapse/admin/v1/rooms?search_term=TWIM
 
 Response:
 
-```json
+```
 {
   "rooms": [
     {
@@ -177,7 +163,7 @@ GET /_synapse/admin/v1/rooms?order_by=size
 
 Response:
 
-```jsonc
+```
 {
   "rooms": [
     {
@@ -233,14 +219,14 @@ GET /_synapse/admin/v1/rooms?order_by=size&from=100
 
 Response:
 
-```jsonc
+```
 {
   "rooms": [
     {
       "room_id": "!mscvqgqpHYjBGDxNym:matrix.org",
       "name": "Music Theory",
       "canonical_alias": "#musictheory:matrix.org",
-      "joined_members": 127,
+      "joined_members": 127
       "joined_local_members": 2,
       "version": "1",
       "creator": "@foo:matrix.org",
@@ -257,7 +243,7 @@ Response:
       "room_id": "!twcBhHVdZlQWuuxBhN:termina.org.uk",
       "name": "weechat-matrix",
       "canonical_alias": "#weechat-matrix:termina.org.uk",
-      "joined_members": 137,
+      "joined_members": 137
       "joined_local_members": 20,
       "version": "4",
       "creator": "@foo:termina.org.uk",
@@ -279,10 +265,12 @@ Response:
 Once the `next_token` parameter is no longer present, we know we've reached the
 end of the list.
 
-# Room Details API
+# DRAFT: Room Details API
 
 The Room Details admin API allows server admins to get all details of a room.
 
+This API is still a draft and details might change!
+
 The following fields are possible in the JSON response body:
 
 * `room_id` - The ID of the room.
@@ -292,7 +280,6 @@ The following fields are possible in the JSON response body:
 * `canonical_alias` - The canonical (main) alias address of the room.
 * `joined_members` - How many users are currently in the room.
 * `joined_local_members` - How many local users are currently in the room.
-* `joined_local_devices` - How many local devices are currently in the room.
 * `version` - The version of the room as a string.
 * `creator` - The `user_id` of the room creator.
 * `encryption` - Algorithm of end-to-end encryption of messages. Is `null` if encryption is not active.
@@ -315,16 +302,15 @@ GET /_synapse/admin/v1/rooms/<room_id>
 
 Response:
 
-```json
+```
 {
   "room_id": "!mscvqgqpHYjBGDxNym:matrix.org",
   "name": "Music Theory",
   "avatar": "mxc://matrix.org/AQDaVFlbkQoErdOgqWRgiGSV",
   "topic": "Theory, Composition, Notation, Analysis",
   "canonical_alias": "#musictheory:matrix.org",
-  "joined_members": 127,
+  "joined_members": 127
   "joined_local_members": 2,
-  "joined_local_devices": 2,
   "version": "1",
   "creator": "@foo:matrix.org",
   "encryption": null,
@@ -358,51 +344,23 @@ GET /_synapse/admin/v1/rooms/<room_id>/members
 
 Response:
 
-```json
+```
 {
   "members": [
     "@foo:matrix.org",
     "@bar:matrix.org",
-    "@foobar:matrix.org"
-  ],
+    "@foobar:matrix.org
+    ],
   "total": 3
 }
 ```
 
-# Room State API
-
-The Room State admin API allows server admins to get a list of all state events in a room.
-
-The response includes the following fields:
-
-* `state` - The current state of the room at the time of request.
-
-## Usage
-
-A standard request:
-
-```
-GET /_synapse/admin/v1/rooms/<room_id>/state
-
-{}
-```
-
-Response:
-
-```json
-{
-  "state": [
-    {"type": "m.room.create", "state_key": "", "etc": true},
-    {"type": "m.room.power_levels", "state_key": "", "etc": true},
-    {"type": "m.room.name", "state_key": "", "etc": true}
-  ]
-}
-```
-
 # Delete Room API
 
 The Delete Room admin API allows server admins to remove rooms from server
 and block these rooms.
+It is a combination and improvement of "[Shutdown room](shutdown_room.md)"
+and "[Purge room](purge_room.md)" API.
 
 Shuts down a room. Moves all local users and room aliases automatically to a
 new room if `new_room_user_id` is set. Otherwise local users only
@@ -426,7 +384,7 @@ the new room. Users on other servers will be unaffected.
 
 The API is:
 
-```
+```json
 POST /_synapse/admin/v1/rooms/<room_id>/delete
 ```
 
@@ -483,10 +441,6 @@ The following JSON body parameters are available:
             future attempts to join the room. Defaults to `false`.
 * `purge` - Optional. If set to `true`, it will remove all traces of the room from your database.
             Defaults to `true`.
-* `force_purge` - Optional, and ignored unless `purge` is `true`. If set to `true`, it
-  will force a purge to go ahead even if there are local users still in the room. Do not
-  use this unless a regular `purge` operation fails, as it could leave those users'
-  clients in a confused state.
 
 The JSON body must not be empty. The body must be at least `{}`.
 
@@ -499,217 +453,3 @@ The following fields are returned in the JSON response body:
 * `local_aliases` - An array of strings representing the local aliases that were migrated from
                     the old room to the new.
 * `new_room_id` - A string representing the room ID of the new room.
-
-
-## Undoing room shutdowns
-
-*Note*: This guide may be outdated by the time you read it. By nature of room shutdowns being performed at the database level,
-the structure can and does change without notice.
-
-First, it's important to understand that a room shutdown is very destructive. Undoing a shutdown is not as simple as pretending it
-never happened - work has to be done to move forward instead of resetting the past. In fact, in some cases it might not be possible
-to recover at all:
-
-* If the room was invite-only, your users will need to be re-invited.
-* If the room no longer has any members at all, it'll be impossible to rejoin.
-* The first user to rejoin will have to do so via an alias on a different server.
-
-With all that being said, if you still want to try and recover the room:
-
-1. For safety reasons, shut down Synapse.
-2. In the database, run `DELETE FROM blocked_rooms WHERE room_id = '!example:example.org';`
-   * For caution: it's recommended to run this in a transaction: `BEGIN; DELETE ...;`, verify you got 1 result, then `COMMIT;`.
-   * The room ID is the same one supplied to the shutdown room API, not the Content Violation room.
-3. Restart Synapse.
-
-You will have to manually handle, if you so choose, the following:
-
-* Aliases that would have been redirected to the Content Violation room.
-* Users that would have been booted from the room (and will have been force-joined to the Content Violation room).
-* Removal of the Content Violation room if desired.
-
-
-# Make Room Admin API
-
-Grants another user the highest power available to a local user who is in the room.
-If the user is not in the room, and it is not publicly joinable, then invite the user.
-
-By default the server admin (the caller) is granted power, but another user can
-optionally be specified, e.g.:
-
-```
-    POST /_synapse/admin/v1/rooms/<room_id_or_alias>/make_room_admin
-    {
-        "user_id": "@foo:example.com"
-    }
-```
-
-# Forward Extremities Admin API
-
-Enables querying and deleting forward extremities from rooms. When a lot of forward
-extremities accumulate in a room, performance can become degraded. For details, see 
-[#1760](https://github.com/matrix-org/synapse/issues/1760).
-
-## Check for forward extremities
-
-To check the status of forward extremities for a room:
-
-```
-    GET /_synapse/admin/v1/rooms/<room_id_or_alias>/forward_extremities
-```
-
-A response as follows will be returned:
-
-```json
-{
-  "count": 1,
-  "results": [
-    {
-      "event_id": "$M5SP266vsnxctfwFgFLNceaCo3ujhRtg_NiiHabcdefgh",
-      "state_group": 439,
-      "depth": 123,
-      "received_ts": 1611263016761
-    }
-  ]
-}    
-```
-
-## Deleting forward extremities
-
-**WARNING**: Please ensure you know what you're doing and have read 
-the related issue [#1760](https://github.com/matrix-org/synapse/issues/1760).
-Under no situations should this API be executed as an automated maintenance task!
-
-If a room has lots of forward extremities, the extra can be
-deleted as follows:
-
-```
-    DELETE /_synapse/admin/v1/rooms/<room_id_or_alias>/forward_extremities
-```
-
-A response as follows will be returned, indicating the amount of forward extremities
-that were deleted.
-
-```json
-{
-  "deleted": 1
-}
-```
-
-# Event Context API
-
-This API lets a client find the context of an event. This is designed primarily to investigate abuse reports.
-
-```
-GET /_synapse/admin/v1/rooms/<room_id>/context/<event_id>
-```
-
-This API mimmicks [GET /_matrix/client/r0/rooms/{roomId}/context/{eventId}](https://matrix.org/docs/spec/client_server/r0.6.1#get-matrix-client-r0-rooms-roomid-context-eventid). Please refer to the link for all details on parameters and reseponse.
-
-Example response:
-
-```json
-{
-  "end": "t29-57_2_0_2",
-  "events_after": [
-    {
-      "content": {
-        "body": "This is an example text message",
-        "msgtype": "m.text",
-        "format": "org.matrix.custom.html",
-        "formatted_body": "<b>This is an example text message</b>"
-      },
-      "type": "m.room.message",
-      "event_id": "$143273582443PhrSn:example.org",
-      "room_id": "!636q39766251:example.com",
-      "sender": "@example:example.org",
-      "origin_server_ts": 1432735824653,
-      "unsigned": {
-        "age": 1234
-      }
-    }
-  ],
-  "event": {
-    "content": {
-      "body": "filename.jpg",
-      "info": {
-        "h": 398,
-        "w": 394,
-        "mimetype": "image/jpeg",
-        "size": 31037
-      },
-      "url": "mxc://example.org/JWEIFJgwEIhweiWJE",
-      "msgtype": "m.image"
-    },
-    "type": "m.room.message",
-    "event_id": "$f3h4d129462ha:example.com",
-    "room_id": "!636q39766251:example.com",
-    "sender": "@example:example.org",
-    "origin_server_ts": 1432735824653,
-    "unsigned": {
-      "age": 1234
-    }
-  },
-  "events_before": [
-    {
-      "content": {
-        "body": "something-important.doc",
-        "filename": "something-important.doc",
-        "info": {
-          "mimetype": "application/msword",
-          "size": 46144
-        },
-        "msgtype": "m.file",
-        "url": "mxc://example.org/FHyPlCeYUSFFxlgbQYZmoEoe"
-      },
-      "type": "m.room.message",
-      "event_id": "$143273582443PhrSn:example.org",
-      "room_id": "!636q39766251:example.com",
-      "sender": "@example:example.org",
-      "origin_server_ts": 1432735824653,
-      "unsigned": {
-        "age": 1234
-      }
-    }
-  ],
-  "start": "t27-54_2_0_2",
-  "state": [
-    {
-      "content": {
-        "creator": "@example:example.org",
-        "room_version": "1",
-        "m.federate": true,
-        "predecessor": {
-          "event_id": "$something:example.org",
-          "room_id": "!oldroom:example.org"
-        }
-      },
-      "type": "m.room.create",
-      "event_id": "$143273582443PhrSn:example.org",
-      "room_id": "!636q39766251:example.com",
-      "sender": "@example:example.org",
-      "origin_server_ts": 1432735824653,
-      "unsigned": {
-        "age": 1234
-      },
-      "state_key": ""
-    },
-    {
-      "content": {
-        "membership": "join",
-        "avatar_url": "mxc://example.org/SEsfnsuifSDFSSEF",
-        "displayname": "Alice Margatroid"
-      },
-      "type": "m.room.member",
-      "event_id": "$143273582443PhrSn:example.org",
-      "room_id": "!636q39766251:example.com",
-      "sender": "@example:example.org",
-      "origin_server_ts": 1432735824653,
-      "unsigned": {
-        "age": 1234
-      },
-      "state_key": "@alice:example.org"
-    }
-  ]
-}
-```

docs/admin_api/shutdown_room.md

@@ -1,7 +1,4 @@
-# Deprecated: Shutdown room API
-
-**The old Shutdown room API is deprecated and will be removed in a future release.
-See the new [Delete Room API](rooms.md#delete-room-api) for more details.**
+# Shutdown room API
 
 Shuts down a room, preventing new joins and moves local users and room aliases automatically
 to a new room. The new room will be created with the user specified by the
@@ -13,6 +10,8 @@ disallow any further invites or joins.
 The local server will only have the power to move local user and room aliases to
 the new room. Users on other servers will be unaffected.
 
+See also: [Delete Room API](rooms.md#delete-room-api)
+
 ## API
 
 You will need to authenticate with an access token for an admin user.

docs/admin_api/statistics.md

@@ -1,83 +0,0 @@
-# Users' media usage statistics
-
-Returns information about all local media usage of users. Gives the
-possibility to filter them by time and user.
-
-The API is:
-
-```
-GET /_synapse/admin/v1/statistics/users/media
-```
-
-To use it, you will need to authenticate by providing an `access_token`
-for a server admin: see [README.rst](README.rst).
-
-A response body like the following is returned:
-
-```json
-{
-  "users": [
-    {
-      "displayname": "foo_user_0",
-      "media_count": 2,
-      "media_length": 134,
-      "user_id": "@foo_user_0:test"
-    },
-    {
-      "displayname": "foo_user_1",
-      "media_count": 2,
-      "media_length": 134,
-      "user_id": "@foo_user_1:test"
-    }
-  ],
-  "next_token": 3,
-  "total": 10
-}
-```
-
-To paginate, check for `next_token` and if present, call the endpoint
-again with `from` set to the value of `next_token`. This will return a new page.
-
-If the endpoint does not return a `next_token` then there are no more
-reports to paginate through.
-
-**Parameters**
-
-The following parameters should be set in the URL:
-
-* `limit`: string representing a positive integer - Is optional but is
-  used for pagination, denoting the maximum number of items to return
-  in this call. Defaults to `100`.
-* `from`: string representing a positive integer - Is optional but used for pagination,
-  denoting the offset in the returned results. This should be treated as an opaque value
-  and not explicitly set to anything other than the return value of `next_token` from a
-  previous call. Defaults to `0`.
-* `order_by` - string - The method in which to sort the returned list of users. Valid values are:
-  - `user_id` - Users are ordered alphabetically by `user_id`. This is the default.
-  - `displayname` - Users are ordered alphabetically by `displayname`.
-  - `media_length` - Users are ordered by the total size of uploaded media in bytes.
-    Smallest to largest.
-  - `media_count` - Users are ordered by number of uploaded media. Smallest to largest.
-* `from_ts` - string representing a positive integer - Considers only
-  files created at this timestamp or later. Unix timestamp in ms.
-* `until_ts` - string representing a positive integer - Considers only
-  files created at this timestamp or earlier. Unix timestamp in ms.
-* `search_term` - string - Filter users by their user ID localpart **or** displayname.
-  The search term can be found in any part of the string.
-  Defaults to no filtering.
-* `dir` - string - Direction of order. Either `f` for forwards or `b` for backwards.
-  Setting this value to `b` will reverse the above sort order. Defaults to `f`.
-
-
-**Response**
-
-The following fields are returned in the JSON response body:
-
-* `users` - An array of objects, each containing information
-  about the user and their local media. Objects contain the following fields:
-  - `displayname` - string - Displayname of this user.
-  - `media_count` - integer - Number of uploaded media by this user.
-  - `media_length` - integer - Size of uploaded media in bytes by this user.
-  - `user_id` - string - Fully-qualified user ID (ex. `@user:server.com`).
-* `next_token` - integer - Opaque value used for pagination. See above.
-* `total` - integer - Total number of users after filtering.

docs/admin_api/user_admin_api.rst

@@ -29,14 +29,8 @@ It returns a JSON body like the following:
             }
         ],
         "avatar_url": "<avatar_url>",
-        "admin": 0,
-        "deactivated": 0,
-        "shadow_banned": 0,
-        "password_hash": "$2b$12$p9B4GkqYdRTPGD",
-        "creation_ts": 1560432506,
-        "appservice_id": null,
-        "consent_server_notice_sent": null,
-        "consent_version": null
+        "admin": false,
+        "deactivated": false
     }
 
 URL parameters:
@@ -99,8 +93,6 @@ Body parameters:
 
 - ``deactivated``, optional. If unspecified, deactivation state will be left
   unchanged on existing accounts and set to ``false`` for new accounts.
-  A user cannot be erased by deactivating with this API. For details on deactivating users see
-  `Deactivate Account <#deactivate-account>`_.
 
 If the user already exists then optional parameters default to the current value.
 
@@ -147,20 +139,20 @@ A JSON body is returned with the following shape:
         "users": [
             {
                 "name": "<user_id1>",
+                "password_hash": "<password_hash1>",
                 "is_guest": 0,
                 "admin": 0,
                 "user_type": null,
                 "deactivated": 0,
-                "shadow_banned": 0,
                 "displayname": "<User One>",
                 "avatar_url": null
             }, {
                 "name": "<user_id2>",
+                "password_hash": "<password_hash2>",
                 "is_guest": 0,
                 "admin": 1,
                 "user_type": null,
                 "deactivated": 0,
-                "shadow_banned": 0,
                 "displayname": "<User Two>",
                 "avatar_url": "<avatar_url>"
             }
@@ -184,13 +176,6 @@ The api is::
 
     GET /_synapse/admin/v1/whois/<user_id>
 
-and::
-
-    GET /_matrix/client/r0/admin/whois/<userId>
-
-See also: `Client Server API Whois
-<https://matrix.org/docs/spec/client_server/r0.6.1#get-matrix-client-r0-admin-whois-userid>`_
-
 To use it, you will need to authenticate by providing an ``access_token`` for a
 server admin: see `README.rst <README.rst>`_.
 
@@ -253,25 +238,6 @@ server admin: see `README.rst <README.rst>`_.
 The erase parameter is optional and defaults to ``false``.
 An empty body may be passed for backwards compatibility.
 
-The following actions are performed when deactivating an user:
-
-- Try to unpind 3PIDs from the identity server
-- Remove all 3PIDs from the homeserver
-- Delete all devices and E2EE keys
-- Delete all access tokens
-- Delete the password hash
-- Removal from all rooms the user is a member of
-- Remove the user from the user directory
-- Reject all pending invites
-- Remove all account validity information related to the user
-
-The following additional actions are performed during deactivation if ``erase``
-is set to ``true``:
-
-- Remove the user's display name
-- Remove the user's avatar URL
-- Mark the user as erased
-
 
 Reset password
 ==============
@@ -288,7 +254,7 @@ with a body of:
 
    {
        "new_password": "<secret>",
-       "logout_devices": true
+       "logout_devices": true,
    }
 
 To use it, you will need to authenticate by providing an ``access_token`` for a
@@ -361,10 +327,6 @@ A response body like the following is returned:
         "total": 2
     }
 
-The server returns the list of rooms of which the user and the server
-are member. If the user is local, all the rooms of which the user is
-member are returned.
-
 **Parameters**
 
 The following parameters should be set in the URL:
@@ -379,12 +341,11 @@ The following fields are returned in the JSON response body:
 - ``total`` - Number of rooms.
 
 
-List media of a user
-====================
+List media of an user
+================================
 Gets a list of all local media that a specific ``user_id`` has created.
-By default, the response is ordered by descending creation date and ascending media ID.
-The newest media is on top. You can change the order with parameters
-``order_by`` and ``dir``.
+The response is ordered by creation date descending and media ID descending.
+The newest media is on top.
 
 The API is::
 
@@ -441,35 +402,6 @@ The following parameters should be set in the URL:
   denoting the offset in the returned results. This should be treated as an opaque value and
   not explicitly set to anything other than the return value of ``next_token`` from a previous call.
   Defaults to ``0``.
-- ``order_by`` - The method by which to sort the returned list of media.
-  If the ordered field has duplicates, the second order is always by ascending ``media_id``,
-  which guarantees a stable ordering. Valid values are:
-
-  - ``media_id`` - Media are ordered alphabetically by ``media_id``.
-  - ``upload_name`` - Media are ordered alphabetically by name the media was uploaded with.
-  - ``created_ts`` - Media are ordered by when the content was uploaded in ms.
-    Smallest to largest. This is the default.
-  - ``last_access_ts`` - Media are ordered by when the content was last accessed in ms.
-    Smallest to largest.
-  - ``media_length`` - Media are ordered by length of the media in bytes.
-    Smallest to largest.
-  - ``media_type`` - Media are ordered alphabetically by MIME-type.
-  - ``quarantined_by`` - Media are ordered alphabetically by the user ID that
-    initiated the quarantine request for this media.
-  - ``safe_from_quarantine`` - Media are ordered by the status if this media is safe
-    from quarantining.
-
-- ``dir`` - Direction of media order. Either ``f`` for forwards or ``b`` for backwards.
-  Setting this value to ``b`` will reverse the above sort order. Defaults to ``f``.
-
-If neither ``order_by`` nor ``dir`` is set, the default order is newest media on top
-(corresponds to ``order_by`` = ``created_ts`` and ``dir`` = ``b``).
-
-Caution. The database only has indexes on the columns ``media_id``,
-``user_id`` and ``created_ts``. This means that if a different sort order is used
-(``upload_name``, ``last_access_ts``, ``media_length``, ``media_type``,
-``quarantined_by`` or ``safe_from_quarantine``), this can cause a large load on the
-database, especially for large environments.
 
 **Response**
 
@@ -492,41 +424,6 @@ The following fields are returned in the JSON response body:
 - ``next_token``: integer - Indication for pagination. See above.
 - ``total`` - integer - Total number of media.
 
-Login as a user
-===============
-
-Get an access token that can be used to authenticate as that user. Useful for
-when admins wish to do actions on behalf of a user.
-
-The API is::
-
-    POST /_synapse/admin/v1/users/<user_id>/login
-    {}
-
-An optional ``valid_until_ms`` field can be specified in the request body as an
-integer timestamp that specifies when the token should expire. By default tokens
-do not expire.
-
-A response body like the following is returned:
-
-.. code:: json
-
-    {
-        "access_token": "<opaque_access_token_string>"
-    }
-
-
-This API does *not* generate a new device for the user, and so will not appear
-their ``/devices`` list, and in general the target user should not be able to
-tell they have been logged in as.
-
-To expire the token call the standard ``/logout`` API with the token.
-
-Note: The token will expire if the *admin* user calls ``/logout/all`` from any
-of their devices, but the token will *not* expire if the target user does the
-same.
-
-
 User devices
 ============
 
@@ -793,33 +690,3 @@ The following fields are returned in the JSON response body:
 - ``total`` - integer - Number of pushers.
 
 See also `Client-Server API Spec <https://matrix.org/docs/spec/client_server/latest#get-matrix-client-r0-pushers>`_
-
-Shadow-banning users
-====================
-
-Shadow-banning is a useful tool for moderating malicious or egregiously abusive users.
-A shadow-banned users receives successful responses to their client-server API requests,
-but the events are not propagated into rooms. This can be an effective tool as it
-(hopefully) takes longer for the user to realise they are being moderated before
-pivoting to another account.
-
-Shadow-banning a user should be used as a tool of last resort and may lead to confusing
-or broken behaviour for the client. A shadow-banned user will not receive any
-notification and it is generally more appropriate to ban or kick abusive users.
-A shadow-banned user will be unable to contact anyone on the server.
-
-The API is::
-
-  POST /_synapse/admin/v1/users/<user_id>/shadow_ban
-
-To use it, you will need to authenticate by providing an ``access_token`` for a
-server admin: see `README.rst <README.rst>`_.
-
-An empty JSON dict is returned.
-
-**Parameters**
-
-The following parameters should be set in the URL:
-
-- ``user_id`` - The fully qualified MXID: for example, ``@user:server.com``. The user must
-  be local.

docs/auth_chain_diff.dot

@@ -1,32 +0,0 @@
-digraph auth {
-    nodesep=0.5;
-    rankdir="RL";
-
-    C [label="Create (1,1)"];
-
-    BJ [label="Bob's Join (2,1)", color=red];
-    BJ2 [label="Bob's Join (2,2)", color=red];
-    BJ2 -> BJ [color=red, dir=none];
-
-    subgraph cluster_foo {
-        A1 [label="Alice's invite (4,1)", color=blue];
-        A2 [label="Alice's Join (4,2)", color=blue];
-        A3 [label="Alice's Join (4,3)", color=blue];
-        A3 -> A2 -> A1 [color=blue, dir=none];
-        color=none;
-    }
-
-    PL1 [label="Power Level (3,1)", color=darkgreen];
-    PL2 [label="Power Level (3,2)", color=darkgreen];
-    PL2 -> PL1 [color=darkgreen, dir=none];
-
-    {rank = same; C; BJ; PL1; A1;}
-
-    A1 -> C [color=grey];
-    A1 -> BJ [color=grey];
-    PL1 -> C [color=grey];
-    BJ2 -> PL1 [penwidth=2];
-
-    A3 -> PL2 [penwidth=2];
-    A1 -> PL1 -> BJ -> C [penwidth=2];
-}

docs/auth_chain_difference_algorithm.md

@@ -1,108 +0,0 @@
-# Auth Chain Difference Algorithm
-
-The auth chain difference algorithm is used by V2 state resolution, where a
-naive implementation can be a significant source of CPU and DB usage.
-
-### Definitions
-
-A *state set* is a set of state events; e.g. the input of a state resolution
-algorithm is a collection of state sets.
-
-The *auth chain* of a set of events are all the events' auth events and *their*
-auth events, recursively (i.e. the events reachable by walking the graph induced
-by an event's auth events links).
-
-The *auth chain difference* of a collection of state sets is the union minus the
-intersection of the sets of auth chains corresponding to the state sets, i.e an
-event is in the auth chain difference if it is reachable by walking the auth
-event graph from at least one of the state sets but not from *all* of the state
-sets.
-
-## Breadth First Walk Algorithm
-
-A way of calculating the auth chain difference without calculating the full auth
-chains for each state set is to do a parallel breadth first walk (ordered by
-depth) of each state set's auth chain. By tracking which events are reachable
-from each state set we can finish early if every pending event is reachable from
-every state set.
-
-This can work well for state sets that have a small auth chain difference, but
-can be very inefficient for larger differences. However, this algorithm is still
-used if we don't have a chain cover index for the room (e.g. because we're in
-the process of indexing it).
-
-## Chain Cover Index
-
-Synapse computes auth chain differences by pre-computing a "chain cover" index
-for the auth chain in a room, allowing efficient reachability queries like "is
-event A in the auth chain of event B". This is done by assigning every event a
-*chain ID* and *sequence number* (e.g. `(5,3)`), and having a map of *links*
-between chains (e.g. `(5,3) -> (2,4)`) such that A is reachable by B (i.e. `A`
-is in the auth chain of `B`) if and only if either:
-
-1. A and B have the same chain ID and `A`'s sequence number is less than `B`'s
-   sequence number; or
-2. there is a link `L` between `B`'s chain ID and `A`'s chain ID such that
-   `L.start_seq_no` <= `B.seq_no` and `A.seq_no` <= `L.end_seq_no`.
-
-There are actually two potential implementations, one where we store links from
-each chain to every other reachable chain (the transitive closure of the links
-graph), and one where we remove redundant links (the transitive reduction of the
-links graph) e.g. if we have chains `C3 -> C2 -> C1` then the link `C3 -> C1`
-would not be stored. Synapse uses the former implementations so that it doesn't
-need to recurse to test reachability between chains.
-
-### Example
-
-An example auth graph would look like the following, where chains have been
-formed based on type/state_key and are denoted by colour and are labelled with
-`(chain ID, sequence number)`. Links are denoted by the arrows (links in grey
-are those that would be remove in the second implementation described above).
-
-![Example](auth_chain_diff.dot.png)
-
-Note that we don't include all links between events and their auth events, as
-most of those links would be redundant. For example, all events point to the
-create event, but each chain only needs the one link from it's base to the
-create event.
-
-## Using the Index
-
-This index can be used to calculate the auth chain difference of the state sets
-by looking at the chain ID and sequence numbers reachable from each state set:
-
-1. For every state set lookup the chain ID/sequence numbers of each state event
-2. Use the index to find all chains and the maximum sequence number reachable
-   from each state set.
-3. The auth chain difference is then all events in each chain that have sequence
-   numbers between the maximum sequence number reachable from *any* state set and
-   the minimum reachable by *all* state sets (if any).
-
-Note that steps 2 is effectively calculating the auth chain for each state set
-(in terms of chain IDs and sequence numbers), and step 3 is calculating the
-difference between the union and intersection of the auth chains.
-
-### Worked Example
-
-For example, given the above graph, we can calculate the difference between
-state sets consisting of:
-
-1. `S1`: Alice's invite `(4,1)` and Bob's second join `(2,2)`; and
-2. `S2`: Alice's second join `(4,3)` and Bob's first join `(2,1)`.
-
-Using the index we see that the following auth chains are reachable from each
-state set:
-
-1. `S1`: `(1,1)`, `(2,2)`, `(3,1)` & `(4,1)`
-2. `S2`: `(1,1)`, `(2,1)`, `(3,2)` & `(4,3)`
-
-And so, for each the ranges that are in the auth chain difference:
-1. Chain 1: None, (since everything can reach the create event).
-2. Chain 2: The range `(1, 2]` (i.e. just `2`), as `1` is reachable by all state
-   sets and the maximum reachable is `2` (corresponding to Bob's second join).
-3. Chain 3: Similarly the range `(1, 2]` (corresponding to the second power
-   level).
-4. Chain 4: The range `(1, 3]` (corresponding to both of Alice's joins).
-
-So the final result is: Bob's second join `(2,2)`, the second power level
-`(3,2)` and both of Alice's joins `(4,2)` & `(4,3)`.

docs/code_style.md

@@ -8,16 +8,16 @@ errors in code.
 
 The necessary tools are detailed below.
 
-First install them with:
-
-    pip install -e ".[lint,mypy]"
-
 -   **black**
 
     The Synapse codebase uses [black](https://pypi.org/project/black/)
     as an opinionated code formatter, ensuring all comitted code is
     properly formatted.
 
+    First install `black` with:
+
+        pip install --upgrade black
+
     Have `black` auto-format your code (it shouldn't change any
     functionality) with:
 
@@ -28,6 +28,10 @@ First install them with:
     `flake8` is a code checking tool. We require code to pass `flake8`
     before being merged into the codebase.
 
+    Install `flake8` with:
+
+        pip install --upgrade flake8 flake8-comprehensions
+
     Check all application and test code with:
 
         flake8 synapse tests
@@ -37,6 +41,10 @@ First install them with:
     `isort` ensures imports are nicely formatted, and can suggest and
     auto-fix issues such as double-importing.
 
+    Install `isort` with:
+
+        pip install --upgrade isort
+
     Auto-fix imports with:
 
         isort -rc synapse tests

docs/dev/cas.md

@@ -31,7 +31,7 @@ easy to run CAS implementation built on top of Django.
 You should now have a Django project configured to serve CAS authentication with
 a single user created.
 
-## Configure Synapse (and Element) to use CAS
+## Configure Synapse (and Riot) to use CAS
 
 1. Modify your `homeserver.yaml` to enable CAS and point it to your locally
    running Django test server:
@@ -51,9 +51,9 @@ and that the CAS server is on port 8000, both on localhost.
 
 ## Testing the configuration
 
-Then in Element:
+Then in Riot:
 
-1. Visit the login page with a Element pointing at your homeserver.
+1. Visit the login page with a Riot pointing at your homeserver.
 2. Click the Single Sign-On button.
 3. Login using the credentials created with `createsuperuser`.
 4. You should be logged in.

docs/metrics-howto.md

@@ -13,12 +13,10 @@
     can be enabled by adding the \"metrics\" resource to the existing
     listener as such:
 
-    ```yaml
-      resources:
-        - names:
-          - client
-          - metrics
-    ```
+        resources:
+          - names:
+            - client
+            - metrics
 
     This provides a simple way of adding metrics to your Synapse
     installation, and serves under `/_synapse/metrics`. If you do not
@@ -33,13 +31,11 @@
 
     Add a new listener to homeserver.yaml:
 
-    ```yaml
-      listeners:
-        - type: metrics
-          port: 9000
-          bind_addresses:
-            - '0.0.0.0'
-    ```
+        listeners:
+          - type: metrics
+            port: 9000
+            bind_addresses:
+              - '0.0.0.0'
 
     For both options, you will need to ensure that `enable_metrics` is
     set to `True`.
@@ -51,13 +47,10 @@
     It needs to set the `metrics_path` to a non-default value (under
     `scrape_configs`):
 
-    ```yaml
-      - job_name: "synapse"
-        scrape_interval: 15s
-        metrics_path: "/_synapse/metrics"
-        static_configs:
-          - targets: ["my.server.here:port"]
-    ```
+        - job_name: "synapse"
+          metrics_path: "/_synapse/metrics"
+          static_configs:
+            - targets: ["my.server.here:port"]
 
     where `my.server.here` is the IP address of Synapse, and `port` is
     the listener port configured with the `metrics` resource.
@@ -67,8 +60,7 @@
 
 1.  Restart Prometheus.
 
-1.  Consider using the [grafana dashboard](https://github.com/matrix-org/synapse/tree/master/contrib/grafana/)
-    and required [recording rules](https://github.com/matrix-org/synapse/tree/master/contrib/prometheus/) 
+1.  Consider using the [grafana dashboard](https://github.com/matrix-org/synapse/tree/master/contrib/grafana/) and required [recording rules](https://github.com/matrix-org/synapse/tree/master/contrib/prometheus/) 
 
 ## Monitoring workers
 
@@ -84,9 +76,9 @@ To allow collecting metrics from a worker, you need to add a
 under `worker_listeners`:
 
 ```yaml
-  - type: metrics
-    bind_address: ''
-    port: 9101
+ - type: metrics
+   bind_address: ''
+   port: 9101
 ```
 
 The `bind_address` and `port` parameters should be set so that
@@ -95,38 +87,6 @@ don't clash with an existing worker.
 With this example, the worker's metrics would then be available
 on `http://127.0.0.1:9101`.
 
-Example Prometheus target for Synapse with workers:
-
-```yaml
-  - job_name: "synapse"
-    scrape_interval: 15s
-    metrics_path: "/_synapse/metrics"
-    static_configs:
-      - targets: ["my.server.here:port"]
-        labels:
-          instance: "my.server"
-          job: "master"
-          index: 1
-      - targets: ["my.workerserver.here:port"]
-        labels:
-          instance: "my.server"
-          job: "generic_worker"
-          index: 1
-      - targets: ["my.workerserver.here:port"]
-        labels:
-          instance: "my.server"
-          job: "generic_worker"
-          index: 2
-      - targets: ["my.workerserver.here:port"]
-        labels:
-          instance: "my.server"
-          job: "media_repository"
-          index: 1
-```
-
-Labels (`instance`, `job`, `index`) can be defined as anything.
-The labels are used to group graphs in grafana.
-
 ## Renaming of metrics & deprecation of old names in 1.2
 
 Synapse 1.2 updates the Prometheus metrics to match the naming

docs/openid.md

@@ -42,41 +42,40 @@ as follows:
  * For other installation mechanisms, see the documentation provided by the
    maintainer.
 
-To enable the OpenID integration, you should then add a section to the `oidc_providers`
-setting in your configuration file (or uncomment one of the existing examples).
-See [sample_config.yaml](./sample_config.yaml) for some sample settings, as well as
-the text below for example configurations for specific providers.
+To enable the OpenID integration, you should then add an `oidc_config` section
+to your configuration file (or uncomment the `enabled: true` line in the
+existing section). See [sample_config.yaml](./sample_config.yaml) for some
+sample settings, as well as the text below for example configurations for
+specific providers.
 
 ## Sample configs
 
 Here are a few configs for providers that should work with Synapse.
 
 ### Microsoft Azure Active Directory
-Azure AD can act as an OpenID Connect Provider. Register a new application under
+Azure AD can act as an OpenID Connect Provider. Register a new application under 
 *App registrations* in the Azure AD management console. The RedirectURI for your
-application should point to your matrix server:
-`[synapse public baseurl]/_synapse/client/oidc/callback`
+application should point to your matrix server: `[synapse public baseurl]/_synapse/oidc/callback`
 
-Go to *Certificates & secrets* and register a new client secret. Make note of your
+Go to *Certificates & secrets* and register a new client secret. Make note of your 
 Directory (tenant) ID as it will be used in the Azure links.
 Edit your Synapse config file and change the `oidc_config` section:
 
 ```yaml
-oidc_providers:
-  - idp_id: microsoft
-    idp_name: Microsoft
-    issuer: "https://login.microsoftonline.com/<tenant id>/v2.0"
-    client_id: "<client id>"
-    client_secret: "<client secret>"
-    scopes: ["openid", "profile"]
-    authorization_endpoint: "https://login.microsoftonline.com/<tenant id>/oauth2/v2.0/authorize"
-    token_endpoint: "https://login.microsoftonline.com/<tenant id>/oauth2/v2.0/token"
-    userinfo_endpoint: "https://graph.microsoft.com/oidc/userinfo"
+oidc_config:
+   enabled: true
+   issuer: "https://login.microsoftonline.com/<tenant id>/v2.0"
+   client_id: "<client id>"
+   client_secret: "<client secret>"
+   scopes: ["openid", "profile"]
+   authorization_endpoint: "https://login.microsoftonline.com/<tenant id>/oauth2/v2.0/authorize"
+   token_endpoint: "https://login.microsoftonline.com/<tenant id>/oauth2/v2.0/token"
+   userinfo_endpoint: "https://graph.microsoft.com/oidc/userinfo"
 
-    user_mapping_provider:
-      config:
-        localpart_template: "{{ user.preferred_username.split('@')[0] }}"
-        display_name_template: "{{ user.name }}"
+   user_mapping_provider:
+     config:
+       localpart_template: "{{ user.preferred_username.split('@')[0] }}"
+       display_name_template: "{{ user.name }}"
 ```
 
 ### [Dex][dex-idp]
@@ -95,7 +94,7 @@ staticClients:
 - id: synapse
   secret: secret
   redirectURIs:
-  - '[synapse public baseurl]/_synapse/client/oidc/callback'
+  - '[synapse public baseurl]/_synapse/oidc/callback'
   name: 'Synapse'
 ```
 
@@ -104,22 +103,21 @@ Run with `dex serve examples/config-dev.yaml`.
 Synapse config:
 
 ```yaml
-oidc_providers:
-  - idp_id: dex
-    idp_name: "My Dex server"
-    skip_verification: true # This is needed as Dex is served on an insecure endpoint
-    issuer: "http://127.0.0.1:5556/dex"
-    client_id: "synapse"
-    client_secret: "secret"
-    scopes: ["openid", "profile"]
-    user_mapping_provider:
-      config:
-        localpart_template: "{{ user.name }}"
-        display_name_template: "{{ user.name|capitalize }}"
+oidc_config:
+   enabled: true
+   skip_verification: true # This is needed as Dex is served on an insecure endpoint
+   issuer: "http://127.0.0.1:5556/dex"
+   client_id: "synapse"
+   client_secret: "secret"
+   scopes: ["openid", "profile"]
+   user_mapping_provider:
+     config:
+       localpart_template: "{{ user.name }}"
+       display_name_template: "{{ user.name|capitalize }}"
 ```
 ### [Keycloak][keycloak-idp]
 
-[Keycloak][keycloak-idp] is an opensource IdP maintained by Red Hat.
+[Keycloak][keycloak-idp] is an opensource IdP maintained by Red Hat. 
 
 Follow the [Getting Started Guide](https://www.keycloak.org/getting-started) to install Keycloak and set up a realm.
 
@@ -141,7 +139,7 @@ Follow the [Getting Started Guide](https://www.keycloak.org/getting-started) to
 | Enabled | `On` |
 | Client Protocol | `openid-connect` |
 | Access Type | `confidential` |
-| Valid Redirect URIs | `[synapse public baseurl]/_synapse/client/oidc/callback` |
+| Valid Redirect URIs | `[synapse public baseurl]/_synapse/oidc/callback` |
 
 5. Click `Save`
 6. On the Credentials tab, update the fields:
@@ -154,22 +152,17 @@ Follow the [Getting Started Guide](https://www.keycloak.org/getting-started) to
 8. Copy Secret
 
 ```yaml
-oidc_providers:
-  - idp_id: keycloak
-    idp_name: "My KeyCloak server"
-    issuer: "https://127.0.0.1:8443/auth/realms/{realm_name}"
-    client_id: "synapse"
-    client_secret: "copy secret generated from above"
-    scopes: ["openid", "profile"]
-    user_mapping_provider:
-      config:
-        localpart_template: "{{ user.preferred_username }}"
-        display_name_template: "{{ user.name }}"
+oidc_config:
+   enabled: true
+   issuer: "https://127.0.0.1:8443/auth/realms/{realm_name}"
+   client_id: "synapse"
+   client_secret: "copy secret generated from above"
+   scopes: ["openid", "profile"]
 ```
 ### [Auth0][auth0]
 
 1. Create a regular web application for Synapse
-2. Set the Allowed Callback URLs to `[synapse public baseurl]/_synapse/client/oidc/callback`
+2. Set the Allowed Callback URLs to `[synapse public baseurl]/_synapse/oidc/callback`
 3. Add a rule to add the `preferred_username` claim.
    <details>
     <summary>Code sample</summary>
@@ -194,17 +187,16 @@ oidc_providers:
 Synapse config:
 
 ```yaml
-oidc_providers:
-  - idp_id: auth0
-    idp_name: Auth0
-    issuer: "https://your-tier.eu.auth0.com/" # TO BE FILLED
-    client_id: "your-client-id" # TO BE FILLED
-    client_secret: "your-client-secret" # TO BE FILLED
-    scopes: ["openid", "profile"]
-    user_mapping_provider:
-      config:
-        localpart_template: "{{ user.preferred_username }}"
-        display_name_template: "{{ user.name }}"
+oidc_config:
+   enabled: true
+   issuer: "https://your-tier.eu.auth0.com/" # TO BE FILLED
+   client_id: "your-client-id" # TO BE FILLED
+   client_secret: "your-client-secret" # TO BE FILLED
+   scopes: ["openid", "profile"]
+   user_mapping_provider:
+     config:
+       localpart_template: "{{ user.preferred_username }}"
+       display_name_template: "{{ user.name }}"
 ```
 
 ### GitHub
@@ -213,33 +205,31 @@ GitHub is a bit special as it is not an OpenID Connect compliant provider, but
 just a regular OAuth2 provider.
 
 The [`/user` API endpoint](https://developer.github.com/v3/users/#get-the-authenticated-user)
-can be used to retrieve information on the authenticated user. As the Synapse
+can be used to retrieve information on the authenticated user. As the Synaspse
 login mechanism needs an attribute to uniquely identify users, and that endpoint
 does not return a `sub` property, an alternative `subject_claim` has to be set.
 
 1. Create a new OAuth application: https://github.com/settings/applications/new.
-2. Set the callback URL to `[synapse public baseurl]/_synapse/client/oidc/callback`.
+2. Set the callback URL to `[synapse public baseurl]/_synapse/oidc/callback`.
 
 Synapse config:
 
 ```yaml
-oidc_providers:
-  - idp_id: github
-    idp_name: Github
-    idp_brand: "github"  # optional: styling hint for clients
-    discover: false
-    issuer: "https://github.com/"
-    client_id: "your-client-id" # TO BE FILLED
-    client_secret: "your-client-secret" # TO BE FILLED
-    authorization_endpoint: "https://github.com/login/oauth/authorize"
-    token_endpoint: "https://github.com/login/oauth/access_token"
-    userinfo_endpoint: "https://api.github.com/user"
-    scopes: ["read:user"]
-    user_mapping_provider:
-      config:
-        subject_claim: "id"
-        localpart_template: "{{ user.login }}"
-        display_name_template: "{{ user.name }}"
+oidc_config:
+   enabled: true
+   discover: false
+   issuer: "https://github.com/"
+   client_id: "your-client-id" # TO BE FILLED
+   client_secret: "your-client-secret" # TO BE FILLED
+   authorization_endpoint: "https://github.com/login/oauth/authorize"
+   token_endpoint: "https://github.com/login/oauth/access_token"
+   userinfo_endpoint: "https://api.github.com/user"
+   scopes: ["read:user"]
+   user_mapping_provider:
+     config:
+       subject_claim: "id"
+       localpart_template: "{{ user.login }}"
+       display_name_template: "{{ user.name }}"
 ```
 
 ### [Google][google-idp]
@@ -249,200 +239,60 @@ oidc_providers:
 2. add an "OAuth Client ID" for a Web Application under "Credentials".
 3. Copy the Client ID and Client Secret, and add the following to your synapse config:
    ```yaml
-   oidc_providers:
-     - idp_id: google
-       idp_name: Google
-       idp_brand: "google"  # optional: styling hint for clients
-       issuer: "https://accounts.google.com/"
-       client_id: "your-client-id" # TO BE FILLED
-       client_secret: "your-client-secret" # TO BE FILLED
-       scopes: ["openid", "profile"]
-       user_mapping_provider:
-         config:
-           localpart_template: "{{ user.given_name|lower }}"
-           display_name_template: "{{ user.name }}"
+   oidc_config:
+     enabled: true
+     issuer: "https://accounts.google.com/"
+     client_id: "your-client-id" # TO BE FILLED
+     client_secret: "your-client-secret" # TO BE FILLED
+     scopes: ["openid", "profile"]
+     user_mapping_provider:
+       config:
+         localpart_template: "{{ user.given_name|lower }}"
+         display_name_template: "{{ user.name }}"
    ```
 4. Back in the Google console, add this Authorized redirect URI: `[synapse
-   public baseurl]/_synapse/client/oidc/callback`.
+   public baseurl]/_synapse/oidc/callback`.
 
 ### Twitch
 
 1. Setup a developer account on [Twitch](https://dev.twitch.tv/)
 2. Obtain the OAuth 2.0 credentials by [creating an app](https://dev.twitch.tv/console/apps/)
-3. Add this OAuth Redirect URL: `[synapse public baseurl]/_synapse/client/oidc/callback`
+3. Add this OAuth Redirect URL: `[synapse public baseurl]/_synapse/oidc/callback`
 
 Synapse config:
 
 ```yaml
-oidc_providers:
-  - idp_id: twitch
-    idp_name: Twitch
-    issuer: "https://id.twitch.tv/oauth2/"
-    client_id: "your-client-id" # TO BE FILLED
-    client_secret: "your-client-secret" # TO BE FILLED
-    client_auth_method: "client_secret_post"
-    user_mapping_provider:
-      config:
-        localpart_template: "{{ user.preferred_username }}"
-        display_name_template: "{{ user.name }}"
+oidc_config:
+  enabled: true
+  issuer: "https://id.twitch.tv/oauth2/"
+  client_id: "your-client-id" # TO BE FILLED
+  client_secret: "your-client-secret" # TO BE FILLED
+  client_auth_method: "client_secret_post"
+  user_mapping_provider:
+    config:
+      localpart_template: "{{ user.preferred_username }}"
+      display_name_template: "{{ user.name }}"
 ```
 
 ### GitLab
 
 1. Create a [new application](https://gitlab.com/profile/applications).
 2. Add the `read_user` and `openid` scopes.
-3. Add this Callback URL: `[synapse public baseurl]/_synapse/client/oidc/callback`
+3. Add this Callback URL: `[synapse public baseurl]/_synapse/oidc/callback`
 
 Synapse config:
 
 ```yaml
-oidc_providers:
-  - idp_id: gitlab
-    idp_name: Gitlab
-    idp_brand: "gitlab"  # optional: styling hint for clients
-    issuer: "https://gitlab.com/"
-    client_id: "your-client-id" # TO BE FILLED
-    client_secret: "your-client-secret" # TO BE FILLED
-    client_auth_method: "client_secret_post"
-    scopes: ["openid", "read_user"]
-    user_profile_method: "userinfo_endpoint"
-    user_mapping_provider:
-      config:
-        localpart_template: '{{ user.nickname }}'
-        display_name_template: '{{ user.name }}'
-```
-
-### Facebook
-
-Like Github, Facebook provide a custom OAuth2 API rather than an OIDC-compliant
-one so requires a little more configuration.
-
-0. You will need a Facebook developer account. You can register for one
-   [here](https://developers.facebook.com/async/registration/).
-1. On the [apps](https://developers.facebook.com/apps/) page of the developer
-   console, "Create App", and choose "Build Connected Experiences".
-2. Once the app is created, add "Facebook Login" and choose "Web". You don't
-   need to go through the whole form here.
-3. In the left-hand menu, open "Products"/"Facebook Login"/"Settings".
-   * Add `[synapse public baseurl]/_synapse/client/oidc/callback` as an OAuth Redirect
-     URL.
-4. In the left-hand menu, open "Settings/Basic". Here you can copy the "App ID"
-   and "App Secret" for use below.
-
-Synapse config:
-
-```yaml
-  - idp_id: facebook
-    idp_name: Facebook
-    idp_brand: "facebook"  # optional: styling hint for clients
-    discover: false
-    issuer: "https://facebook.com"
-    client_id: "your-client-id" # TO BE FILLED
-    client_secret: "your-client-secret" # TO BE FILLED
-    scopes: ["openid", "email"]
-    authorization_endpoint: https://facebook.com/dialog/oauth
-    token_endpoint: https://graph.facebook.com/v9.0/oauth/access_token
-    user_profile_method: "userinfo_endpoint"
-    userinfo_endpoint: "https://graph.facebook.com/v9.0/me?fields=id,name,email,picture"
-    user_mapping_provider:
-      config:
-        subject_claim: "id"
-        display_name_template: "{{ user.name }}"
-```
-
-Relevant documents:
- * https://developers.facebook.com/docs/facebook-login/manually-build-a-login-flow
- * Using Facebook's Graph API: https://developers.facebook.com/docs/graph-api/using-graph-api/
- * Reference to the User endpoint: https://developers.facebook.com/docs/graph-api/reference/user
-
-### Gitea
-
-Gitea is, like Github, not an OpenID provider, but just an OAuth2 provider.
-
-The [`/user` API endpoint](https://try.gitea.io/api/swagger#/user/userGetCurrent)
-can be used to retrieve information on the authenticated user. As the Synapse
-login mechanism needs an attribute to uniquely identify users, and that endpoint
-does not return a `sub` property, an alternative `subject_claim` has to be set.
-
-1. Create a new application.
-2. Add this Callback URL: `[synapse public baseurl]/_synapse/client/oidc/callback`
-
-Synapse config:
-
-```yaml
-oidc_providers:
-  - idp_id: gitea
-    idp_name: Gitea
-    discover: false
-    issuer: "https://your-gitea.com/"
-    client_id: "your-client-id" # TO BE FILLED
-    client_secret: "your-client-secret" # TO BE FILLED
-    client_auth_method: client_secret_post
-    scopes: [] # Gitea doesn't support Scopes
-    authorization_endpoint: "https://your-gitea.com/login/oauth/authorize"
-    token_endpoint: "https://your-gitea.com/login/oauth/access_token"
-    userinfo_endpoint: "https://your-gitea.com/api/v1/user"
-    user_mapping_provider:
-      config:
-        subject_claim: "id"
-        localpart_template: "{{ user.login }}"
-        display_name_template: "{{ user.full_name }}"
-```
-
-### XWiki
-
-Install [OpenID Connect Provider](https://extensions.xwiki.org/xwiki/bin/view/Extension/OpenID%20Connect/OpenID%20Connect%20Provider/) extension in your [XWiki](https://www.xwiki.org) instance.
-
-Synapse config:
-
-```yaml
-oidc_providers:
-  - idp_id: xwiki
-    idp_name: "XWiki"
-    issuer: "https://myxwikihost/xwiki/oidc/"
-    client_id: "your-client-id" # TO BE FILLED
-    client_auth_method: none
-    scopes: ["openid", "profile"]
-    user_profile_method: "userinfo_endpoint"
-    user_mapping_provider:
-      config:
-        localpart_template: "{{ user.preferred_username }}"
-        display_name_template: "{{ user.name }}"
-```
-
-## Apple
-
-Configuring "Sign in with Apple" (SiWA) requires an Apple Developer account.
-
-You will need to create a new "Services ID" for SiWA, and create and download a
-private key with "SiWA" enabled.
-
-As well as the private key file, you will need:
- * Client ID: the "identifier" you gave the "Services ID"
- * Team ID: a 10-character ID associated with your developer account.
- * Key ID: the 10-character identifier for the key.
-
-https://help.apple.com/developer-account/?lang=en#/dev77c875b7e has more
-documentation on setting up SiWA.
-
-The synapse config will look like this:
-
-```yaml
-  - idp_id: apple
-    idp_name: Apple
-    issuer: "https://appleid.apple.com"
-    client_id: "your-client-id" # Set to the "identifier" for your "ServicesID"
-    client_auth_method: "client_secret_post"
-    client_secret_jwt_key:
-      key_file: "/path/to/AuthKey_KEYIDCODE.p8"  # point to your key file
-      jwt_header:
-        alg: ES256
-        kid: "KEYIDCODE"   # Set to the 10-char Key ID
-      jwt_payload:
-        iss: TEAMIDCODE    # Set to the 10-char Team ID
-    scopes: ["name", "email", "openid"]
-    authorization_endpoint: https://appleid.apple.com/auth/authorize?response_mode=form_post
-    user_mapping_provider:
-      config:
-        email_template: "{{ user.email }}"
+oidc_config:
+  enabled: true
+  issuer: "https://gitlab.com/"
+  client_id: "your-client-id" # TO BE FILLED
+  client_secret: "your-client-secret" # TO BE FILLED
+  client_auth_method: "client_secret_post"
+  scopes: ["openid", "read_user"]
+  user_profile_method: "userinfo_endpoint"
+  user_mapping_provider:
+    config:
+      localpart_template: '{{ user.nickname }}'
+      display_name_template: '{{ user.name }}'
 ```

docs/password_auth_providers.md

@@ -26,7 +26,6 @@ Password auth provider classes must provide the following methods:
 
   It should perform any appropriate sanity checks on the provided
   configuration, and return an object which is then passed into
-  `__init__`.
 
   This method should have the `@staticmethod` decoration.
 

docs/postgres.md

@@ -18,7 +18,7 @@ connect to a postgres database.
     virtualenv](../INSTALL.md#installing-from-source), you can install
     the library with:
 
-        ~/synapse/env/bin/pip install "matrix-synapse[postgres]"
+        ~/synapse/env/bin/pip install matrix-synapse[postgres]
 
     (substituting the path to your virtualenv for `~/synapse/env`, if
     you used a different path). You will require the postgres

docs/reverse_proxy.md

@@ -3,31 +3,30 @@
 It is recommended to put a reverse proxy such as
 [nginx](https://nginx.org/en/docs/http/ngx_http_proxy_module.html),
 [Apache](https://httpd.apache.org/docs/current/mod/mod_proxy_http.html),
-[Caddy](https://caddyserver.com/docs/quick-starts/reverse-proxy),
-[HAProxy](https://www.haproxy.org/) or
-[relayd](https://man.openbsd.org/relayd.8) in front of Synapse. One advantage
+[Caddy](https://caddyserver.com/docs/quick-starts/reverse-proxy) or
+[HAProxy](https://www.haproxy.org/) in front of Synapse. One advantage
 of doing so is that it means that you can expose the default https port
 (443) to Matrix clients without needing to run Synapse with root
 privileges.
 
-You should configure your reverse proxy to forward requests to `/_matrix` or
-`/_synapse/client` to Synapse, and have it set the `X-Forwarded-For` and
-`X-Forwarded-Proto` request headers.
-
-You should remember that Matrix clients and other Matrix servers do not
-necessarily need to connect to your server via the same server name or
-port. Indeed, clients will use port 443 by default, whereas servers default to
-port 8448. Where these are different, we refer to the 'client port' and the
-'federation port'. See [the Matrix
-specification](https://matrix.org/docs/spec/server_server/latest#resolving-server-names)
-for more details of the algorithm used for federation connections, and
-[delegate.md](<delegate.md>) for instructions on setting up delegation.
-
 **NOTE**: Your reverse proxy must not `canonicalise` or `normalise`
 the requested URI in any way (for example, by decoding `%xx` escapes).
 Beware that Apache *will* canonicalise URIs unless you specify
 `nocanon`.
 
+When setting up a reverse proxy, remember that Matrix clients and other
+Matrix servers do not necessarily need to connect to your server via the
+same server name or port. Indeed, clients will use port 443 by default,
+whereas servers default to port 8448. Where these are different, we
+refer to the 'client port' and the 'federation port'. See [the Matrix
+specification](https://matrix.org/docs/spec/server_server/latest#resolving-server-names)
+for more details of the algorithm used for federation connections, and
+[delegate.md](<delegate.md>) for instructions on setting up delegation.
+
+Endpoints that are part of the standardised Matrix specification are
+located under `/_matrix`, whereas endpoints specific to Synapse are
+located under `/_synapse/client`.
+
 Let's assume that we expect clients to connect to our server at
 `https://matrix.example.com`, and other servers to connect at
 `https://example.com:8448`.  The following sections detail the configuration of
@@ -41,21 +40,18 @@ the reverse proxy and the homeserver.
 
 ```
 server {
-    listen 443 ssl http2;
-    listen [::]:443 ssl http2;
+    listen 443 ssl;
+    listen [::]:443 ssl;
 
     # For the federation port
-    listen 8448 ssl http2 default_server;
-    listen [::]:8448 ssl http2 default_server;
+    listen 8448 ssl default_server;
+    listen [::]:8448 ssl default_server;
 
     server_name matrix.example.com;
 
     location ~* ^(\/_matrix|\/_synapse\/client) {
         proxy_pass http://localhost:8008;
         proxy_set_header X-Forwarded-For $remote_addr;
-        proxy_set_header X-Forwarded-Proto $scheme;
-        proxy_set_header Host $host;
-
         # Nginx by default only allows file uploads up to 1M in size
         # Increase client_max_body_size to match max_upload_size defined in homeserver.yaml
         client_max_body_size 50M;
@@ -106,7 +102,6 @@ example.com:8448 {
     SSLEngine on
     ServerName matrix.example.com;
 
-    RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
     AllowEncodedSlashes NoDecode
     ProxyPass /_matrix http://127.0.0.1:8008/_matrix nocanon
     ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix
@@ -118,7 +113,6 @@ example.com:8448 {
     SSLEngine on
     ServerName example.com;
 
-    RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
     AllowEncodedSlashes NoDecode
     ProxyPass /_matrix http://127.0.0.1:8008/_matrix nocanon
     ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix
@@ -140,9 +134,6 @@ example.com:8448 {
 ```
 frontend https
   bind :::443 v4v6 ssl crt /etc/ssl/haproxy/ strict-sni alpn h2,http/1.1
-  http-request set-header X-Forwarded-Proto https if { ssl_fc }
-  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
-  http-request set-header X-Forwarded-For %[src]
 
   # Matrix client traffic
   acl matrix-host hdr(host) -i matrix.example.com
@@ -153,62 +144,12 @@ frontend https
 
 frontend matrix-federation
   bind :::8448 v4v6 ssl crt /etc/ssl/haproxy/synapse.pem alpn h2,http/1.1
-  http-request set-header X-Forwarded-Proto https if { ssl_fc }
-  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
-  http-request set-header X-Forwarded-For %[src]
-
   default_backend matrix
 
 backend matrix
   server matrix 127.0.0.1:8008
 ```
 
-### Relayd
-
-```
-table <webserver>    { 127.0.0.1 }
-table <matrixserver> { 127.0.0.1 }
-
-http protocol "https" {
-    tls { no tlsv1.0, ciphers "HIGH" }
-    tls keypair "example.com"
-    match header set "X-Forwarded-For"   value "$REMOTE_ADDR"
-    match header set "X-Forwarded-Proto" value "https"
-
-    # set CORS header for .well-known/matrix/server, .well-known/matrix/client
-    # httpd does not support setting headers, so do it here
-    match request path "/.well-known/matrix/*" tag "matrix-cors"
-    match response tagged "matrix-cors" header set "Access-Control-Allow-Origin" value "*"
-
-    pass quick path "/_matrix/*"         forward to <matrixserver>
-    pass quick path "/_synapse/client/*" forward to <matrixserver>
-
-    # pass on non-matrix traffic to webserver
-    pass                                 forward to <webserver>
-}
-
-relay "https_traffic" {
-    listen on egress port 443 tls
-    protocol "https"
-    forward to <matrixserver> port 8008 check tcp
-    forward to <webserver>    port 8080 check tcp
-}
-
-http protocol "matrix" {
-    tls { no tlsv1.0, ciphers "HIGH" }
-    tls keypair "example.com"
-    block
-    pass quick path "/_matrix/*"         forward to <matrixserver>
-    pass quick path "/_synapse/client/*" forward to <matrixserver>
-}
-
-relay "matrix_federation" {
-    listen on egress port 8448 tls
-    protocol "matrix"
-    forward to <matrixserver> port 8008 check tcp
-}
-```
-
 ## Homeserver Configuration
 
 You will also want to set `bind_addresses: ['127.0.0.1']` and

docs/sample_config.yaml

@@ -67,12 +67,11 @@ pid_file: DATADIR/homeserver.pid
 #
 #web_client_location: https://riot.example.com/
 
-# The public-facing base URL that clients use to access this Homeserver (not
-# including _matrix/...). This is the same URL a user might enter into the
-# 'Custom Homeserver URL' field on their client. If you use Synapse with a
-# reverse proxy, this should be the URL to reach Synapse via the proxy.
-# Otherwise, it should be the URL to reach Synapse's client HTTP listener (see
-# 'listeners' below).
+# The public-facing base URL that clients use to access this HS
+# (not including _matrix/...). This is the same URL a user would
+# enter into the 'custom HS URL' field on their client. If you
+# use synapse with a reverse proxy, this should be the URL to reach
+# synapse via the proxy.
 #
 #public_baseurl: https://example.com/
 
@@ -89,7 +88,8 @@ pid_file: DATADIR/homeserver.pid
 # Whether to require authentication to retrieve profile data (avatars,
 # display names) of other users through the client API. Defaults to
 # 'false'. Note that profile data is also available via the federation
-# API, unless allow_profile_lookup_over_federation is set to false.
+# API, so this setting is of limited value if federation is enabled on
+# the server.
 #
 #require_auth_for_profile_requests: true
 
@@ -100,14 +100,6 @@ pid_file: DATADIR/homeserver.pid
 #
 #limit_profile_requests_to_users_who_share_rooms: true
 
-# Uncomment to prevent a user's profile data from being retrieved and
-# displayed in a room until they have joined it. By default, a user's
-# profile data is included in an invite event, regardless of the values
-# of the above two settings, and whether or not the users share a server.
-# Defaults to 'true'.
-#
-#include_profile_data_on_invite: false
-
 # If set to 'true', removes the need for authentication to access the server's
 # public rooms directory through the client API, meaning that anyone can
 # query the room directory. Defaults to 'false'.
@@ -152,51 +144,6 @@ pid_file: DATADIR/homeserver.pid
 #
 #enable_search: false
 
-# Prevent outgoing requests from being sent to the following blacklisted IP address
-# CIDR ranges. If this option is not specified then it defaults to private IP
-# address ranges (see the example below).
-#
-# The blacklist applies to the outbound requests for federation, identity servers,
-# push servers, and for checking key validity for third-party invite events.
-#
-# (0.0.0.0 and :: are always blacklisted, whether or not they are explicitly
-# listed here, since they correspond to unroutable addresses.)
-#
-# This option replaces federation_ip_range_blacklist in Synapse v1.25.0.
-#
-#ip_range_blacklist:
-#  - '127.0.0.0/8'
-#  - '10.0.0.0/8'
-#  - '172.16.0.0/12'
-#  - '192.168.0.0/16'
-#  - '100.64.0.0/10'
-#  - '192.0.0.0/24'
-#  - '169.254.0.0/16'
-#  - '192.88.99.0/24'
-#  - '198.18.0.0/15'
-#  - '192.0.2.0/24'
-#  - '198.51.100.0/24'
-#  - '203.0.113.0/24'
-#  - '224.0.0.0/4'
-#  - '::1/128'
-#  - 'fe80::/10'
-#  - 'fc00::/7'
-#  - '2001:db8::/32'
-#  - 'ff00::/8'
-#  - 'fec0::/10'
-
-# List of IP address CIDR ranges that should be allowed for federation,
-# identity servers, push servers, and for checking key validity for
-# third-party invite events. This is useful for specifying exceptions to
-# wide-ranging blacklisted target IP ranges - e.g. for communication with
-# a push server only visible in your network.
-#
-# This whitelist overrides ip_range_blacklist and defaults to an empty
-# list.
-#
-#ip_range_whitelist:
-#   - '192.168.1.1'
-
 # List of ports that Synapse should listen on, their purpose and their
 # configuration.
 #
@@ -695,6 +642,27 @@ acme:
 #  - nyc.example.com
 #  - syd.example.com
 
+# Prevent federation requests from being sent to the following
+# blacklist IP address CIDR ranges. If this option is not specified, or
+# specified with an empty list, no ip range blacklist will be enforced.
+#
+# As of Synapse v1.4.0 this option also affects any outbound requests to identity
+# servers provided by user input.
+#
+# (0.0.0.0 and :: are always blacklisted, whether or not they are explicitly
+# listed here, since they correspond to unroutable addresses.)
+#
+federation_ip_range_blacklist:
+  - '127.0.0.0/8'
+  - '10.0.0.0/8'
+  - '172.16.0.0/12'
+  - '192.168.0.0/16'
+  - '100.64.0.0/10'
+  - '169.254.0.0/16'
+  - '::1/128'
+  - 'fe80::/64'
+  - 'fc00::/7'
+
 # Report prometheus metrics on the age of PDUs being sent to and received from
 # the following domains. This can be used to give an idea of "delay" on inbound
 # and outbound federation, though be aware that any delay can be due to problems
@@ -706,12 +674,6 @@ acme:
 #  - matrix.org
 #  - example.com
 
-# Uncomment to disable profile lookup over federation. By default, the
-# Federation API allows other homeservers to obtain profile data of any user
-# on this homeserver. Defaults to 'true'.
-#
-#allow_profile_lookup_over_federation: false
-
 
 ## Caching ##
 
@@ -837,9 +799,6 @@ log_config: "CONFDIR/SERVERNAME.log.config"
 #     users are joining rooms the server is already in (this is cheap) vs
 #     "remote" for when users are trying to join rooms not on the server (which
 #     can be more expensive)
-#   - one for ratelimiting how often a user or IP can attempt to validate a 3PID.
-#   - two for ratelimiting how often invites can be sent in a room or to a
-#     specific user.
 #
 # The defaults are as shown below.
 #
@@ -873,18 +832,7 @@ log_config: "CONFDIR/SERVERNAME.log.config"
 #  remote:
 #    per_second: 0.01
 #    burst_count: 3
-#
-#rc_3pid_validation:
-#  per_second: 0.003
-#  burst_count: 5
-#
-#rc_invites:
-#  per_room:
-#    per_second: 0.3
-#    burst_count: 10
-#  per_user:
-#    per_second: 0.003
-#    burst_count: 5
+
 
 # Ratelimiting settings for incoming federation
 #
@@ -1005,20 +953,10 @@ media_store_path: "DATADIR/media_store"
 #  - '172.16.0.0/12'
 #  - '192.168.0.0/16'
 #  - '100.64.0.0/10'
-#  - '192.0.0.0/24'
 #  - '169.254.0.0/16'
-#  - '192.88.99.0/24'
-#  - '198.18.0.0/15'
-#  - '192.0.2.0/24'
-#  - '198.51.100.0/24'
-#  - '203.0.113.0/24'
-#  - '224.0.0.0/4'
 #  - '::1/128'
-#  - 'fe80::/10'
+#  - 'fe80::/64'
 #  - 'fc00::/7'
-#  - '2001:db8::/32'
-#  - 'ff00::/8'
-#  - 'fec0::/10'
 
 # List of IP address CIDR ranges that the URL preview spider is allowed
 # to access even if they are specified in url_preview_ip_range_blacklist.
@@ -1292,9 +1230,8 @@ account_validity:
 # email will be globally disabled.
 #
 # Additionally, if `msisdn` is not set, registration and password resets via msisdn
-# will be disabled regardless, and users will not be able to associate an msisdn
-# identifier to their account. This is due to Synapse currently not supporting
-# any method of sending SMS messages on its own.
+# will be disabled regardless. This is due to Synapse currently not supporting any
+# method of sending SMS messages on its own.
 #
 # To enable using an identity server for operations regarding a particular third-party
 # identifier type, set the value to the URL of that identity server as shown in the
@@ -1339,8 +1276,6 @@ account_threepid_delegates:
 # By default, any room aliases included in this list will be created
 # as a publicly joinable room when the first user registers for the
 # homeserver. This behaviour can be customised with the settings below.
-# If the room already exists, make certain it is a publicly joinable
-# room. The join rule of the room must be set to 'public'.
 #
 #auto_join_rooms:
 #  - "#example:example.com"
@@ -1587,10 +1522,10 @@ trusted_key_servers:
 # enable SAML login.
 #
 # Once SAML support is enabled, a metadata file will be exposed at
-# https://<server>:<port>/_synapse/client/saml2/metadata.xml, which you may be able to
+# https://<server>:<port>/_matrix/saml2/metadata.xml, which you may be able to
 # use to configure your SAML IdP with. Alternatively, you can manually configure
 # the IdP to use an ACS location of
-# https://<server>:<port>/_synapse/client/saml2/authn_response.
+# https://<server>:<port>/_matrix/saml2/authn_response.
 #
 saml2_config:
   # `sp_config` is the configuration for the pysaml2 Service Provider.
@@ -1610,12 +1545,6 @@ saml2_config:
     #  remote:
     #    - url: https://our_idp/metadata.xml
 
-    # Allowed clock difference in seconds between the homeserver and IdP.
-    #
-    # Uncomment the below to increase the accepted time difference from 0 to 3 seconds.
-    #
-    #accepted_time_diff: 3
-
     # By default, the user has to go to our login page first. If you'd like
     # to allow IdP-initiated login, set 'allow_unsolicited: true' in a
     # 'service.sp' section:
@@ -1631,28 +1560,6 @@ saml2_config:
     #description: ["My awesome SP", "en"]
     #name: ["Test SP", "en"]
 
-    #ui_info:
-    #  display_name:
-    #    - lang: en
-    #      text: "Display Name is the descriptive name of your service."
-    #  description:
-    #    - lang: en
-    #      text: "Description should be a short paragraph explaining the purpose of the service."
-    #  information_url:
-    #    - lang: en
-    #      text: "https://example.com/terms-of-service"
-    #  privacy_statement_url:
-    #    - lang: en
-    #      text: "https://example.com/privacy-policy"
-    #  keywords:
-    #    - lang: en
-    #      text: ["Matrix", "Element"]
-    #  logo:
-    #    - lang: en
-    #      text: "https://example.com/logo.svg"
-    #      width: "200"
-    #      height: "80"
-
     #organization:
     #  name: Example com
     #  display_name:
@@ -1738,201 +1645,141 @@ saml2_config:
   #  - attribute: department
   #    value: "sales"
 
-  # If the metadata XML contains multiple IdP entities then the `idp_entityid`
-  # option must be set to the entity to redirect users to.
-  #
-  # Most deployments only have a single IdP entity and so should omit this
-  # option.
-  #
-  #idp_entityid: 'https://our_idp/entityid'
 
-
-# List of OpenID Connect (OIDC) / OAuth 2.0 identity providers, for registration
-# and login.
-#
-# Options for each entry include:
-#
-#   idp_id: a unique identifier for this identity provider. Used internally
-#       by Synapse; should be a single word such as 'github'.
-#
-#       Note that, if this is changed, users authenticating via that provider
-#       will no longer be recognised as the same user!
-#
-#   idp_name: A user-facing name for this identity provider, which is used to
-#       offer the user a choice of login mechanisms.
-#
-#   idp_icon: An optional icon for this identity provider, which is presented
-#       by clients and Synapse's own IdP picker page. If given, must be an
-#       MXC URI of the format mxc://<server-name>/<media-id>. (An easy way to
-#       obtain such an MXC URI is to upload an image to an (unencrypted) room
-#       and then copy the "url" from the source of the event.)
-#
-#   idp_brand: An optional brand for this identity provider, allowing clients
-#       to style the login flow according to the identity provider in question.
-#       See the spec for possible options here.
-#
-#   discover: set to 'false' to disable the use of the OIDC discovery mechanism
-#       to discover endpoints. Defaults to true.
-#
-#   issuer: Required. The OIDC issuer. Used to validate tokens and (if discovery
-#       is enabled) to discover the provider's endpoints.
-#
-#   client_id: Required. oauth2 client id to use.
-#
-#   client_secret: oauth2 client secret to use. May be omitted if
-#        client_secret_jwt_key is given, or if client_auth_method is 'none'.
-#
-#   client_secret_jwt_key: Alternative to client_secret: details of a key used
-#      to create a JSON Web Token to be used as an OAuth2 client secret. If
-#      given, must be a dictionary with the following properties:
-#
-#          key: a pem-encoded signing key. Must be a suitable key for the
-#              algorithm specified. Required unless 'key_file' is given.
-#
-#          key_file: the path to file containing a pem-encoded signing key file.
-#              Required unless 'key' is given.
-#
-#          jwt_header: a dictionary giving properties to include in the JWT
-#              header. Must include the key 'alg', giving the algorithm used to
-#              sign the JWT, such as "ES256", using the JWA identifiers in
-#              RFC7518.
-#
-#          jwt_payload: an optional dictionary giving properties to include in
-#              the JWT payload. Normally this should include an 'iss' key.
-#
-#   client_auth_method: auth method to use when exchanging the token. Valid
-#       values are 'client_secret_basic' (default), 'client_secret_post' and
-#       'none'.
-#
-#   scopes: list of scopes to request. This should normally include the "openid"
-#       scope. Defaults to ["openid"].
-#
-#   authorization_endpoint: the oauth2 authorization endpoint. Required if
-#       provider discovery is disabled.
-#
-#   token_endpoint: the oauth2 token endpoint. Required if provider discovery is
-#       disabled.
-#
-#   userinfo_endpoint: the OIDC userinfo endpoint. Required if discovery is
-#       disabled and the 'openid' scope is not requested.
-#
-#   jwks_uri: URI where to fetch the JWKS. Required if discovery is disabled and
-#       the 'openid' scope is used.
-#
-#   skip_verification: set to 'true' to skip metadata verification. Use this if
-#       you are connecting to a provider that is not OpenID Connect compliant.
-#       Defaults to false. Avoid this in production.
-#
-#   user_profile_method: Whether to fetch the user profile from the userinfo
-#       endpoint. Valid values are: 'auto' or 'userinfo_endpoint'.
-#
-#       Defaults to 'auto', which fetches the userinfo endpoint if 'openid' is
-#       included in 'scopes'. Set to 'userinfo_endpoint' to always fetch the
-#       userinfo endpoint.
-#
-#   allow_existing_users: set to 'true' to allow a user logging in via OIDC to
-#       match a pre-existing account instead of failing. This could be used if
-#       switching from password logins to OIDC. Defaults to false.
-#
-#   user_mapping_provider: Configuration for how attributes returned from a OIDC
-#       provider are mapped onto a matrix user. This setting has the following
-#       sub-properties:
-#
-#       module: The class name of a custom mapping module. Default is
-#           'synapse.handlers.oidc_handler.JinjaOidcMappingProvider'.
-#           See https://github.com/matrix-org/synapse/blob/master/docs/sso_mapping_providers.md#openid-mapping-providers
-#           for information on implementing a custom mapping provider.
-#
-#       config: Configuration for the mapping provider module. This section will
-#           be passed as a Python dictionary to the user mapping provider
-#           module's `parse_config` method.
-#
-#           For the default provider, the following settings are available:
-#
-#             subject_claim: name of the claim containing a unique identifier
-#                 for the user. Defaults to 'sub', which OpenID Connect
-#                 compliant providers should provide.
-#
-#             localpart_template: Jinja2 template for the localpart of the MXID.
-#                 If this is not set, the user will be prompted to choose their
-#                 own username (see 'sso_auth_account_details.html' in the 'sso'
-#                 section of this file).
-#
-#             display_name_template: Jinja2 template for the display name to set
-#                 on first login. If unset, no displayname will be set.
-#
-#             email_template: Jinja2 template for the email address of the user.
-#                 If unset, no email address will be added to the account.
-#
-#             extra_attributes: a map of Jinja2 templates for extra attributes
-#                 to send back to the client during login.
-#                 Note that these are non-standard and clients will ignore them
-#                 without modifications.
-#
-#           When rendering, the Jinja2 templates are given a 'user' variable,
-#           which is set to the claims returned by the UserInfo Endpoint and/or
-#           in the ID Token.
+# Enable OpenID Connect (OIDC) / OAuth 2.0 for registration and login.
 #
 # See https://github.com/matrix-org/synapse/blob/master/docs/openid.md
-# for information on how to configure these options.
+# for some example configurations.
 #
-# For backwards compatibility, it is also possible to configure a single OIDC
-# provider via an 'oidc_config' setting. This is now deprecated and admins are
-# advised to migrate to the 'oidc_providers' format. (When doing that migration,
-# use 'oidc' for the idp_id to ensure that existing users continue to be
-# recognised.)
-#
-oidc_providers:
-  # Generic example
+oidc_config:
+  # Uncomment the following to enable authorization against an OpenID Connect
+  # server. Defaults to false.
   #
-  #- idp_id: my_idp
-  #  idp_name: "My OpenID provider"
-  #  idp_icon: "mxc://example.com/mediaid"
-  #  discover: false
-  #  issuer: "https://accounts.example.com/"
-  #  client_id: "provided-by-your-issuer"
-  #  client_secret: "provided-by-your-issuer"
-  #  client_auth_method: client_secret_post
-  #  scopes: ["openid", "profile"]
-  #  authorization_endpoint: "https://accounts.example.com/oauth2/auth"
-  #  token_endpoint: "https://accounts.example.com/oauth2/token"
-  #  userinfo_endpoint: "https://accounts.example.com/userinfo"
-  #  jwks_uri: "https://accounts.example.com/.well-known/jwks.json"
-  #  skip_verification: true
-  #  user_mapping_provider:
-  #    config:
-  #      subject_claim: "id"
-  #      localpart_template: "{{ user.login }}"
-  #      display_name_template: "{{ user.name }}"
-  #      email_template: "{{ user.email }}"
+  #enabled: true
 
-  # For use with Keycloak
+  # Uncomment the following to disable use of the OIDC discovery mechanism to
+  # discover endpoints. Defaults to true.
   #
-  #- idp_id: keycloak
-  #  idp_name: Keycloak
-  #  issuer: "https://127.0.0.1:8443/auth/realms/my_realm_name"
-  #  client_id: "synapse"
-  #  client_secret: "copy secret generated in Keycloak UI"
-  #  scopes: ["openid", "profile"]
+  #discover: false
 
-  # For use with Github
+  # the OIDC issuer. Used to validate tokens and (if discovery is enabled) to
+  # discover the provider's endpoints.
   #
-  #- idp_id: github
-  #  idp_name: Github
-  #  idp_brand: github
-  #  discover: false
-  #  issuer: "https://github.com/"
-  #  client_id: "your-client-id" # TO BE FILLED
-  #  client_secret: "your-client-secret" # TO BE FILLED
-  #  authorization_endpoint: "https://github.com/login/oauth/authorize"
-  #  token_endpoint: "https://github.com/login/oauth/access_token"
-  #  userinfo_endpoint: "https://api.github.com/user"
-  #  scopes: ["read:user"]
-  #  user_mapping_provider:
-  #    config:
-  #      subject_claim: "id"
-  #      localpart_template: "{{ user.login }}"
-  #      display_name_template: "{{ user.name }}"
+  # Required if 'enabled' is true.
+  #
+  #issuer: "https://accounts.example.com/"
+
+  # oauth2 client id to use.
+  #
+  # Required if 'enabled' is true.
+  #
+  #client_id: "provided-by-your-issuer"
+
+  # oauth2 client secret to use.
+  #
+  # Required if 'enabled' is true.
+  #
+  #client_secret: "provided-by-your-issuer"
+
+  # auth method to use when exchanging the token.
+  # Valid values are 'client_secret_basic' (default), 'client_secret_post' and
+  # 'none'.
+  #
+  #client_auth_method: client_secret_post
+
+  # list of scopes to request. This should normally include the "openid" scope.
+  # Defaults to ["openid"].
+  #
+  #scopes: ["openid", "profile"]
+
+  # the oauth2 authorization endpoint. Required if provider discovery is disabled.
+  #
+  #authorization_endpoint: "https://accounts.example.com/oauth2/auth"
+
+  # the oauth2 token endpoint. Required if provider discovery is disabled.
+  #
+  #token_endpoint: "https://accounts.example.com/oauth2/token"
+
+  # the OIDC userinfo endpoint. Required if discovery is disabled and the
+  # "openid" scope is not requested.
+  #
+  #userinfo_endpoint: "https://accounts.example.com/userinfo"
+
+  # URI where to fetch the JWKS. Required if discovery is disabled and the
+  # "openid" scope is used.
+  #
+  #jwks_uri: "https://accounts.example.com/.well-known/jwks.json"
+
+  # Uncomment to skip metadata verification. Defaults to false.
+  #
+  # Use this if you are connecting to a provider that is not OpenID Connect
+  # compliant.
+  # Avoid this in production.
+  #
+  #skip_verification: true
+
+  # Whether to fetch the user profile from the userinfo endpoint. Valid
+  # values are: "auto" or "userinfo_endpoint".
+  #
+  # Defaults to "auto", which fetches the userinfo endpoint if "openid" is included
+  # in `scopes`. Uncomment the following to always fetch the userinfo endpoint.
+  #
+  #user_profile_method: "userinfo_endpoint"
+
+  # Uncomment to allow a user logging in via OIDC to match a pre-existing account instead
+  # of failing. This could be used if switching from password logins to OIDC. Defaults to false.
+  #
+  #allow_existing_users: true
+
+  # An external module can be provided here as a custom solution to mapping
+  # attributes returned from a OIDC provider onto a matrix user.
+  #
+  user_mapping_provider:
+    # The custom module's class. Uncomment to use a custom module.
+    # Default is 'synapse.handlers.oidc_handler.JinjaOidcMappingProvider'.
+    #
+    # See https://github.com/matrix-org/synapse/blob/master/docs/sso_mapping_providers.md#openid-mapping-providers
+    # for information on implementing a custom mapping provider.
+    #
+    #module: mapping_provider.OidcMappingProvider
+
+    # Custom configuration values for the module. This section will be passed as
+    # a Python dictionary to the user mapping provider module's `parse_config`
+    # method.
+    #
+    # The examples below are intended for the default provider: they should be
+    # changed if using a custom provider.
+    #
+    config:
+      # name of the claim containing a unique identifier for the user.
+      # Defaults to `sub`, which OpenID Connect compliant providers should provide.
+      #
+      #subject_claim: "sub"
+
+      # Jinja2 template for the localpart of the MXID.
+      #
+      # When rendering, this template is given the following variables:
+      #   * user: The claims returned by the UserInfo Endpoint and/or in the ID
+      #     Token
+      #
+      # This must be configured if using the default mapping provider.
+      #
+      localpart_template: "{{ user.preferred_username }}"
+
+      # Jinja2 template for the display name to set on first login.
+      #
+      # If unset, no displayname will be set.
+      #
+      #display_name_template: "{{ user.given_name }} {{ user.last_name }}"
+
+      # Jinja2 templates for extra attributes to send back to the client during
+      # login.
+      #
+      # Note that these are non-standard and clients will ignore them without modifications.
+      #
+      #extra_attributes:
+        #birthdate: "{{ user.birthdate }}"
+
 
 
 # Enable Central Authentication Service (CAS) for registration and login.
@@ -1947,6 +1794,10 @@ cas_config:
   #
   #server_url: "https://cas-server.com"
 
+  # The public URL of the homeserver.
+  #
+  #service_url: "https://homeserver.domain.com:8448"
+
   # The attribute of the CAS response to use as the display name.
   #
   # If unset, no displayname will be set.
@@ -1989,135 +1840,41 @@ sso:
     #  - https://my.custom.client/
 
     # Directory in which Synapse will try to find the template files below.
-    # If not set, or the files named below are not found within the template
-    # directory, default templates from within the Synapse package will be used.
+    # If not set, default templates from within the Synapse package will be used.
+    #
+    # DO NOT UNCOMMENT THIS SETTING unless you want to customise the templates.
+    # If you *do* uncomment it, you will need to make sure that all the templates
+    # below are in the directory.
     #
     # Synapse will look for the following templates in this directory:
     #
-    # * HTML page to prompt the user to choose an Identity Provider during
-    #   login: 'sso_login_idp_picker.html'.
-    #
-    #   This is only used if multiple SSO Identity Providers are configured.
-    #
-    #   When rendering, this template is given the following variables:
-    #     * redirect_url: the URL that the user will be redirected to after
-    #       login.
-    #
-    #     * server_name: the homeserver's name.
-    #
-    #     * providers: a list of available Identity Providers. Each element is
-    #       an object with the following attributes:
-    #
-    #         * idp_id: unique identifier for the IdP
-    #         * idp_name: user-facing name for the IdP
-    #         * idp_icon: if specified in the IdP config, an MXC URI for an icon
-    #              for the IdP
-    #         * idp_brand: if specified in the IdP config, a textual identifier
-    #              for the brand of the IdP
-    #
-    #   The rendered HTML page should contain a form which submits its results
-    #   back as a GET request, with the following query parameters:
-    #
-    #     * redirectUrl: the client redirect URI (ie, the `redirect_url` passed
-    #       to the template)
-    #
-    #     * idp: the 'idp_id' of the chosen IDP.
-    #
-    # * HTML page to prompt new users to enter a userid and confirm other
-    #   details: 'sso_auth_account_details.html'. This is only shown if the
-    #   SSO implementation (with any user_mapping_provider) does not return
-    #   a localpart.
-    #
-    #   When rendering, this template is given the following variables:
-    #
-    #     * server_name: the homeserver's name.
-    #
-    #     * idp: details of the SSO Identity Provider that the user logged in
-    #       with: an object with the following attributes:
-    #
-    #         * idp_id: unique identifier for the IdP
-    #         * idp_name: user-facing name for the IdP
-    #         * idp_icon: if specified in the IdP config, an MXC URI for an icon
-    #              for the IdP
-    #         * idp_brand: if specified in the IdP config, a textual identifier
-    #              for the brand of the IdP
-    #
-    #     * user_attributes: an object containing details about the user that
-    #       we received from the IdP. May have the following attributes:
-    #
-    #         * display_name: the user's display_name
-    #         * emails: a list of email addresses
-    #
-    #   The template should render a form which submits the following fields:
-    #
-    #     * username: the localpart of the user's chosen user id
-    #
-    # * HTML page allowing the user to consent to the server's terms and
-    #   conditions. This is only shown for new users, and only if
-    #   `user_consent.require_at_registration` is set.
-    #
-    #   When rendering, this template is given the following variables:
-    #
-    #     * server_name: the homeserver's name.
-    #
-    #     * user_id: the user's matrix proposed ID.
-    #
-    #     * user_profile.display_name: the user's proposed display name, if any.
-    #
-    #     * consent_version: the version of the terms that the user will be
-    #       shown
-    #
-    #     * terms_url: a link to the page showing the terms.
-    #
-    #   The template should render a form which submits the following fields:
-    #
-    #     * accepted_version: the version of the terms accepted by the user
-    #       (ie, 'consent_version' from the input variables).
-    #
     # * HTML page for a confirmation step before redirecting back to the client
     #   with the login token: 'sso_redirect_confirm.html'.
     #
-    #   When rendering, this template is given the following variables:
-    #
-    #     * redirect_url: the URL the user is about to be redirected to.
+    #   When rendering, this template is given three variables:
+    #     * redirect_url: the URL the user is about to be redirected to. Needs
+    #                     manual escaping (see
+    #                     https://jinja.palletsprojects.com/en/2.11.x/templates/#html-escaping).
     #
     #     * display_url: the same as `redirect_url`, but with the query
     #                    parameters stripped. The intention is to have a
     #                    human-readable URL to show to users, not to use it as
-    #                    the final address to redirect to.
+    #                    the final address to redirect to. Needs manual escaping
+    #                    (see https://jinja.palletsprojects.com/en/2.11.x/templates/#html-escaping).
     #
     #     * server_name: the homeserver's name.
     #
-    #     * new_user: a boolean indicating whether this is the user's first time
-    #          logging in.
-    #
-    #     * user_id: the user's matrix ID.
-    #
-    #     * user_profile.avatar_url: an MXC URI for the user's avatar, if any.
-    #           None if the user has not set an avatar.
-    #
-    #     * user_profile.display_name: the user's display name. None if the user
-    #           has not set a display name.
-    #
     # * HTML page which notifies the user that they are authenticating to confirm
     #   an operation on their account during the user interactive authentication
     #   process: 'sso_auth_confirm.html'.
     #
     #   When rendering, this template is given the following variables:
-    #     * redirect_url: the URL the user is about to be redirected to.
+    #     * redirect_url: the URL the user is about to be redirected to. Needs
+    #                     manual escaping (see
+    #                     https://jinja.palletsprojects.com/en/2.11.x/templates/#html-escaping).
     #
     #     * description: the operation which the user is being asked to confirm
     #
-    #     * idp: details of the Identity Provider that we will use to confirm
-    #       the user's identity: an object with the following attributes:
-    #
-    #         * idp_id: unique identifier for the IdP
-    #         * idp_name: user-facing name for the IdP
-    #         * idp_icon: if specified in the IdP config, an MXC URI for an icon
-    #              for the IdP
-    #         * idp_brand: if specified in the IdP config, a textual identifier
-    #              for the brand of the IdP
-    #
     # * HTML page shown after a successful user interactive authentication session:
     #   'sso_auth_success.html'.
     #
@@ -2126,14 +1883,6 @@ sso:
     #
     #   This template has no additional variables.
     #
-    # * HTML page shown after a user-interactive authentication session which
-    #   does not map correctly onto the expected user: 'sso_auth_bad_user.html'.
-    #
-    #   When rendering, this template is given the following variables:
-    #     * server_name: the homeserver's name.
-    #     * user_id_to_verify: the MXID of the user that we are trying to
-    #       validate.
-    #
     # * HTML page shown during single sign-on if a deactivated user (according to Synapse's database)
     #   attempts to login: 'sso_account_deactivated.html'.
     #
@@ -2259,21 +2008,6 @@ password_config:
       #
       #require_uppercase: true
 
-ui_auth:
-    # The amount of time to allow a user-interactive authentication session
-    # to be active.
-    #
-    # This defaults to 0, meaning the user is queried for their credentials
-    # before every action, but this can be overridden to allow a single
-    # validation to be re-used.  This weakens the protections afforded by
-    # the user-interactive authentication process, by allowing for multiple
-    # (and potentially different) operations to use the same validation session.
-    #
-    # Uncomment below to allow for credential validation to last for 15
-    # seconds.
-    #
-    #session_timeout: "15s"
-
 
 # Configuration for sending emails from Synapse.
 #
@@ -2339,15 +2073,10 @@ email:
   #
   #validation_token_lifetime: 15m
 
-  # The web client location to direct users to during an invite. This is passed
-  # to the identity server as the org.matrix.web_client_location key. Defaults
-  # to unset, giving no guidance to the identity server.
-  #
-  #invite_client_location: https://app.element.io
-
   # Directory in which Synapse will try to find the template files below.
-  # If not set, or the files named below are not found within the template
-  # directory, default templates from within the Synapse package will be used.
+  # If not set, default templates from within the Synapse package will be used.
+  #
+  # Do not uncomment this setting unless you want to customise the templates.
   #
   # Synapse will look for the following templates in this directory:
   #
@@ -2485,35 +2214,20 @@ password_providers:
 
 
 
-## Push ##
-
-push:
-  # Clients requesting push notifications can either have the body of
-  # the message sent in the notification poke along with other details
-  # like the sender, or just the event ID and room ID (`event_id_only`).
-  # If clients choose the former, this option controls whether the
-  # notification request includes the content of the event (other details
-  # like the sender are still included). For `event_id_only` push, it
-  # has no effect.
-  #
-  # For modern android devices the notification content will still appear
-  # because it is loaded by the app. iPhone, however will send a
-  # notification saying only that a message arrived and who it came from.
-  #
-  # The default value is "true" to include message details. Uncomment to only
-  # include the event ID and room ID in push notification payloads.
-  #
-  #include_content: false
-
-  # When a push notification is received, an unread count is also sent.
-  # This number can either be calculated as the number of unread messages
-  # for the user, or the number of *rooms* the user has unread messages in.
-  #
-  # The default value is "true", meaning push clients will see the number of
-  # rooms with unread messages in them. Uncomment to instead send the number
-  # of unread messages.
-  #
-  #group_unread_count_by_room: false
+# Clients requesting push notifications can either have the body of
+# the message sent in the notification poke along with other details
+# like the sender, or just the event ID and room ID (`event_id_only`).
+# If clients choose the former, this option controls whether the
+# notification request includes the content of the event (other details
+# like the sender are still included). For `event_id_only` push, it
+# has no effect.
+#
+# For modern android devices the notification content will still appear
+# because it is loaded by the app. iPhone, however will send a
+# notification saying only that a message arrived and who it came from.
+#
+#push:
+#  include_content: true
 
 
 # Spam checkers are third-party modules that can block specific actions
@@ -2556,41 +2270,25 @@ spam_checker:
 # If enabled, non server admins can only create groups with local parts
 # starting with this prefix
 #
-#group_creation_prefix: "unofficial_"
+#group_creation_prefix: "unofficial/"
 
 
 
 # User Directory configuration
 #
-user_directory:
-    # Defines whether users can search the user directory. If false then
-    # empty responses are returned to all queries. Defaults to true.
-    #
-    # Uncomment to disable the user directory.
-    #
-    #enabled: false
-
-    # Defines whether to search all users visible to your HS when searching
-    # the user directory, rather than limiting to users visible in public
-    # rooms. Defaults to false.
-    #
-    # If you set it true, you'll have to rebuild the user_directory search
-    # indexes, see:
-    # https://github.com/matrix-org/synapse/blob/master/docs/user_directory.md
-    #
-    # Uncomment to return search results containing all known users, even if that
-    # user does not share a room with the requester.
-    #
-    #search_all_users: true
-
-    # Defines whether to prefer local users in search query results.
-    # If True, local users are more likely to appear above remote users
-    # when searching the user directory. Defaults to false.
-    #
-    # Uncomment to prefer local over remote users in user directory search
-    # results.
-    #
-    #prefer_local_users: true
+# 'enabled' defines whether users can search the user directory. If
+# false then empty responses are returned to all queries. Defaults to
+# true.
+#
+# 'search_all_users' defines whether to search all users visible to your HS
+# when searching the user directory, rather than limiting to users visible
+# in public rooms.  Defaults to false.  If you set it True, you'll have to
+# rebuild the user_directory search indexes, see
+# https://github.com/matrix-org/synapse/blob/master/docs/user_directory.md
+#
+#user_directory:
+#  enabled: true
+#  search_all_users: false
 
 
 # User Consent configuration
@@ -2645,20 +2343,19 @@ user_directory:
 
 
 
-# Settings for local room and user statistics collection. See
-# docs/room_and_user_statistics.md.
+# Local statistics collection. Used in populating the room directory.
 #
-stats:
-  # Uncomment the following to disable room and user statistics. Note that doing
-  # so may cause certain features (such as the room directory) not to work
-  # correctly.
-  #
-  #enabled: false
-
-  # The size of each timeslice in the room_stats_historical and
-  # user_stats_historical tables, as a time period. Defaults to "1d".
-  #
-  #bucket_size: 1h
+# 'bucket_size' controls how large each statistics timeslice is. It can
+# be defined in a human readable short form -- e.g. "1d", "1y".
+#
+# 'retention' controls how long historical statistics will be kept for.
+# It can be defined in a human readable short form -- e.g. "1d", "1y".
+#
+#
+#stats:
+#   enabled: true
+#   bucket_size: 1d
+#   retention: 1y
 
 
 # Server Notices room configuration
@@ -2838,13 +2535,6 @@ opentracing:
 #
 #run_background_tasks_on: worker1
 
-# A shared secret used by the replication APIs to authenticate HTTP requests
-# from workers.
-#
-# By default this is unused and traffic is not authenticated.
-#
-#worker_replication_secret: ""
-
 
 # Configuration for Redis when using workers. This *must* be enabled when
 # using workers (unless using old style direct TCP configuration).

docs/spam_checker.md

@@ -14,7 +14,6 @@ The Python class is instantiated with two objects:
 * An instance of `synapse.module_api.ModuleApi`.
 
 It then implements methods which return a boolean to alter behavior in Synapse.
-All the methods must be defined.
 
 There's a generic method for checking every event (`check_event_for_spam`), as
 well as some specific methods:
@@ -23,63 +22,38 @@ well as some specific methods:
 * `user_may_create_room`
 * `user_may_create_room_alias`
 * `user_may_publish_room`
-* `check_username_for_spam`
-* `check_registration_for_spam`
-* `check_media_file_for_spam`
 
-The details of each of these methods (as well as their inputs and outputs)
+The details of the each of these methods (as well as their inputs and outputs)
 are documented in the `synapse.events.spamcheck.SpamChecker` class.
 
 The `ModuleApi` class provides a way for the custom spam checker class to
 call back into the homeserver internals.
 
-Additionally, a `parse_config` method is mandatory and receives the plugin config
-dictionary. After parsing, It must return an object which will be
-passed to `__init__` later.
-
 ### Example
 
 ```python
-from synapse.spam_checker_api import RegistrationBehaviour
-
 class ExampleSpamChecker:
     def __init__(self, config, api):
         self.config = config
         self.api = api
 
-    @staticmethod
-    def parse_config(config):
-        return config
-        
-    async def check_event_for_spam(self, foo):
+    def check_event_for_spam(self, foo):
         return False  # allow all events
 
-    async def user_may_invite(self, inviter_userid, invitee_userid, room_id):
+    def user_may_invite(self, inviter_userid, invitee_userid, room_id):
         return True  # allow all invites
 
-    async def user_may_create_room(self, userid):
+    def user_may_create_room(self, userid):
         return True  # allow all room creations
 
-    async def user_may_create_room_alias(self, userid, room_alias):
+    def user_may_create_room_alias(self, userid, room_alias):
         return True  # allow all room aliases
 
-    async def user_may_publish_room(self, userid, room_id):
+    def user_may_publish_room(self, userid, room_id):
         return True  # allow publishing of all rooms
 
-    async def check_username_for_spam(self, user_profile):
+    def check_username_for_spam(self, user_profile):
         return False  # allow all usernames
-
-    async def check_registration_for_spam(
-        self,
-        email_threepid,
-        username,
-        request_info,
-        auth_provider_id,
-    ):
-        return RegistrationBehaviour.ALLOW  # allow all registrations
-
-    async def check_media_file_for_spam(self, file_wrapper, file_info):
-        return False  # allow all media
 ```
 
 ## Configuration

docs/sso_mapping_providers.md

@@ -15,21 +15,8 @@ where SAML mapping providers come into play.
 SSO mapping providers are currently supported for OpenID and SAML SSO
 configurations. Please see the details below for how to implement your own.
 
-It is up to the mapping provider whether the user should be assigned a predefined
-Matrix ID based on the SSO attributes, or if the user should be allowed to
-choose their own username.
-
-In the first case - where users are automatically allocated a Matrix ID - it is
-the responsibility of the mapping provider to normalise the SSO attributes and
-map them to a valid Matrix ID. The [specification for Matrix
-IDs](https://matrix.org/docs/spec/appendices#user-identifiers) has some
-information about what is considered valid.
-
-If the mapping provider does not assign a Matrix ID, then Synapse will
-automatically serve an HTML page allowing the user to pick their own username.
-
 External mapping providers are provided to Synapse in the form of an external
-Python module. You can retrieve this module from [PyPI](https://pypi.org) or elsewhere,
+Python module. You can retrieve this module from [PyPi](https://pypi.org) or elsewhere,
 but it must be importable via Synapse (e.g. it must be in the same virtualenv
 as Synapse). The Synapse config is then modified to point to the mapping provider
 (and optionally provide additional configuration for it).
@@ -69,26 +56,16 @@ A custom mapping provider must specify the following methods:
                      information from.
     - This method must return a string, which is the unique identifier for the
       user. Commonly the ``sub`` claim of the response.
-* `map_user_attributes(self, userinfo, token, failures)`
+* `map_user_attributes(self, userinfo, token)`
     - This method must be async.
     - Arguments:
       - `userinfo` - A `authlib.oidc.core.claims.UserInfo` object to extract user
                      information from.
       - `token` - A dictionary which includes information necessary to make
                   further requests to the OpenID provider.
-      - `failures` - An `int` that represents the amount of times the returned
-                     mxid localpart mapping has failed.  This should be used
-                     to create a deduplicated mxid localpart which should be
-                     returned instead. For example, if this method returns
-                     `john.doe` as the value of `localpart` in the returned
-                     dict, and that is already taken on the homeserver, this
-                     method will be called again with the same parameters but
-                     with failures=1. The method should then return a different
-                     `localpart` value, such as `john.doe1`.
     - Returns a dictionary with two keys:
-      - `localpart`: A string, used to generate the Matrix ID. If this is
-        `None`, the user is prompted to pick their own username.
-      - `displayname`: An optional string, the display name for the user.
+      - localpart: A required string, used to generate the Matrix ID.
+      - displayname: An optional string, the display name for the user.
 * `get_extra_attributes(self, userinfo, token)`
     - This method must be async.
     - Arguments:
@@ -123,13 +100,11 @@ comment these options out and use those specified by the module instead.
 
 A custom mapping provider must specify the following methods:
 
-* `__init__(self, parsed_config, module_api)`
+* `__init__(self, parsed_config)`
    - Arguments:
      - `parsed_config` - A configuration object that is the return value of the
        `parse_config` method. You should set any configuration options needed by
        the module here.
-     - `module_api` - a `synapse.module_api.ModuleApi` object which provides the
-       stable API available for extension modules.
 * `parse_config(config)`
     - This method should have the `@staticmethod` decoration.
     - Arguments:
@@ -172,20 +147,12 @@ A custom mapping provider must specify the following methods:
                                 redirected to.
     - This method must return a dictionary, which will then be used by Synapse
       to build a new user. The following keys are allowed:
-       * `mxid_localpart` - The mxid localpart of the new user.  If this is
-         `None`, the user is prompted to pick their own username.
+       * `mxid_localpart` - Required. The mxid localpart of the new user.
        * `displayname` - The displayname of the new user. If not provided, will default to
                          the value of `mxid_localpart`.
        * `emails` - A list of emails for the new user. If not provided, will
                     default to an empty list.
 
-       Alternatively it can raise a `synapse.api.errors.RedirectException` to
-       redirect the user to another page. This is useful to prompt the user for
-       additional information, e.g. if you want them to provide their own username.
-       It is the responsibility of the mapping provider to either redirect back
-       to `client_redirect_url` (including any additional information) or to
-       complete registration using methods from the `ModuleApi`.
-
 ### Default SAML Mapping Provider
 
 Synapse has a built-in SAML mapping provider if a custom provider isn't

docs/systemd-with-workers/README.md

@@ -31,16 +31,16 @@ There is no need for a separate configuration file for the master process.
 1. Adjust synapse configuration files as above.
 1. Copy the `*.service` and `*.target` files in [system](system) to
 `/etc/systemd/system`.
-1. Run `systemctl daemon-reload` to tell systemd to load the new unit files.
+1. Run `systemctl deamon-reload` to tell systemd to load the new unit files.
 1. Run `systemctl enable matrix-synapse.service`. This will configure the
 synapse master process to be started as part of the `matrix-synapse.target`
 target.
 1. For each worker process to be enabled, run `systemctl enable
 matrix-synapse-worker@<worker_name>.service`. For each `<worker_name>`, there
-should be a corresponding configuration file.
+should be a corresponding configuration file
 `/etc/matrix-synapse/workers/<worker_name>.yaml`.
 1. Start all the synapse processes with `systemctl start matrix-synapse.target`.
-1. Tell systemd to start synapse on boot with `systemctl enable matrix-synapse.target`.
+1. Tell systemd to start synapse on boot with `systemctl enable matrix-synapse.target`/
 
 ## Usage
 

docs/systemd-with-workers/system/matrix-synapse-worker@.service

@@ -4,7 +4,6 @@ AssertPathExists=/etc/matrix-synapse/workers/%i.yaml
 
 # This service should be restarted when the synapse target is restarted.
 PartOf=matrix-synapse.target
-ReloadPropagatedFrom=matrix-synapse.target
 
 # if this is started at the same time as the main, let the main process start
 # first, to initialise the database schema.