Add some debugging

.github/ISSUE_TEMPLATE.md

@@ -1,5 +0,0 @@
-**If you are looking for support** please ask in **#synapse:matrix.org**
-(using a matrix.org account if necessary). We do not use GitHub issues for
-support.
-
-**If you want to report a security issue** please see https://matrix.org/security-disclosure-policy/

.github/ISSUE_TEMPLATE/BUG_REPORT.md

@@ -4,13 +4,11 @@ about: Create a report to help us improve
 
 ---
 
-**THIS IS NOT A SUPPORT CHANNEL!**
-**IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER**,
-please ask in **#synapse:matrix.org** (using a matrix.org account if necessary)
-
 <!--
 
-If you want to report a security issue, please see https://matrix.org/security-disclosure-policy/
+**IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER**:
+You will likely get better support more quickly if you ask in ** #synapse:matrix.org ** ;)
+
 
 This is a bug report template. By following the instructions below and
 filling out the sections with your information, you will help the us to get all

CHANGES.md

@@ -1,112 +1,20 @@
-Synapse 1.14.0 (2020-05-28)
-===========================
-
-No significant changes.
-
-
-Synapse 1.14.0rc2 (2020-05-27)
+Synapse 1.13.0rc2 (2020-05-14)
 ==============================
 
 Bugfixes
 --------
 
-- Fix cache config to not apply cache factor to event cache. Regression in v1.14.0rc1. ([\#7578](https://github.com/matrix-org/synapse/issues/7578))
-- Fix bug where `ReplicationStreamer` was not always started when replication was enabled. Bug introduced in v1.14.0rc1. ([\#7579](https://github.com/matrix-org/synapse/issues/7579))
-- Fix specifying individual cache factors for caches with special characters in their name. Regression in v1.14.0rc1. ([\#7580](https://github.com/matrix-org/synapse/issues/7580))
-
-
-Improved Documentation
-----------------------
-
-- Fix the OIDC `client_auth_method` value in the sample config. ([\#7581](https://github.com/matrix-org/synapse/issues/7581))
-
-
-Synapse 1.14.0rc1 (2020-05-26)
-==============================
-
-Features
---------
-
-- Synapse's cache factor can now be configured in `homeserver.yaml` by the `caches.global_factor` setting. Additionally, `caches.per_cache_factors` controls the cache factors for individual caches. ([\#6391](https://github.com/matrix-org/synapse/issues/6391))
-- Add OpenID Connect login/registration support. Contributed by Quentin Gliech, on behalf of [les Connecteurs](https://connecteu.rs). ([\#7256](https://github.com/matrix-org/synapse/issues/7256), [\#7457](https://github.com/matrix-org/synapse/issues/7457))
-- Add room details admin endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#7317](https://github.com/matrix-org/synapse/issues/7317))
-- Allow for using more than one spam checker module at once. ([\#7435](https://github.com/matrix-org/synapse/issues/7435))
-- Add additional authentication checks for `m.room.power_levels` event per [MSC2209](https://github.com/matrix-org/matrix-doc/pull/2209). ([\#7502](https://github.com/matrix-org/synapse/issues/7502))
-- Implement room version 6 per [MSC2240](https://github.com/matrix-org/matrix-doc/pull/2240). ([\#7506](https://github.com/matrix-org/synapse/issues/7506))
-- Add highly experimental option to move event persistence off master. ([\#7281](https://github.com/matrix-org/synapse/issues/7281), [\#7374](https://github.com/matrix-org/synapse/issues/7374), [\#7436](https://github.com/matrix-org/synapse/issues/7436), [\#7440](https://github.com/matrix-org/synapse/issues/7440), [\#7475](https://github.com/matrix-org/synapse/issues/7475), [\#7490](https://github.com/matrix-org/synapse/issues/7490), [\#7491](https://github.com/matrix-org/synapse/issues/7491), [\#7492](https://github.com/matrix-org/synapse/issues/7492), [\#7493](https://github.com/matrix-org/synapse/issues/7493), [\#7495](https://github.com/matrix-org/synapse/issues/7495), [\#7515](https://github.com/matrix-org/synapse/issues/7515), [\#7516](https://github.com/matrix-org/synapse/issues/7516), [\#7517](https://github.com/matrix-org/synapse/issues/7517), [\#7542](https://github.com/matrix-org/synapse/issues/7542))
-
-
-Bugfixes
---------
-
-- Fix a bug where event updates might not be sent over replication to worker processes after the stream falls behind. ([\#7384](https://github.com/matrix-org/synapse/issues/7384))
-- Allow expired user accounts to log out their device sessions. ([\#7443](https://github.com/matrix-org/synapse/issues/7443))
-- Fix a bug that would cause Synapse not to resync out-of-sync device lists. ([\#7453](https://github.com/matrix-org/synapse/issues/7453))
-- Prevent rooms with 0 members or with invalid version strings from breaking group queries. ([\#7465](https://github.com/matrix-org/synapse/issues/7465))
-- Workaround for an upstream Twisted bug that caused Synapse to become unresponsive after startup. ([\#7473](https://github.com/matrix-org/synapse/issues/7473))
-- Fix Redis reconnection logic that can result in missed updates over replication if master reconnects to Redis without restarting. ([\#7482](https://github.com/matrix-org/synapse/issues/7482))
-- When sending `m.room.member` events, omit `displayname` and `avatar_url` if they aren't set instead of setting them to `null`. Contributed by Aaron Raimist. ([\#7497](https://github.com/matrix-org/synapse/issues/7497))
-- Fix incorrect `method` label on `synapse_http_matrixfederationclient_{requests,responses}` prometheus metrics. ([\#7503](https://github.com/matrix-org/synapse/issues/7503))
-- Ignore incoming presence events from other homeservers if presence is disabled locally. ([\#7508](https://github.com/matrix-org/synapse/issues/7508))
-- Fix a long-standing bug that broke the update remote profile background process. ([\#7511](https://github.com/matrix-org/synapse/issues/7511))
-- Hash passwords as early as possible during password reset. ([\#7538](https://github.com/matrix-org/synapse/issues/7538))
-- Fix bug where a local user leaving a room could fail under rare circumstances. ([\#7548](https://github.com/matrix-org/synapse/issues/7548))
-- Fix "Missing RelayState parameter" error when using user interactive authentication with SAML for some SAML providers. ([\#7552](https://github.com/matrix-org/synapse/issues/7552))
-- Fix exception `'GenericWorkerReplicationHandler' object has no attribute 'send_federation_ack'`, introduced in v1.13.0. ([\#7564](https://github.com/matrix-org/synapse/issues/7564))
-- `synctl` now warns if it was unable to stop Synapse and will not attempt to start Synapse if nothing was stopped. Contributed by Romain Bouyé. ([\#6590](https://github.com/matrix-org/synapse/issues/6590))
-
-
-Updates to the Docker image
----------------------------
-
-- Update docker runtime image to Alpine v3.11. Contributed by @Starbix. ([\#7398](https://github.com/matrix-org/synapse/issues/7398))
-
-
-Improved Documentation
-----------------------
-
-- Update information about mapping providers for SAML and OpenID. ([\#7458](https://github.com/matrix-org/synapse/issues/7458))
-- Add additional reverse proxy example for Caddy v2. Contributed by Jeff Peeler. ([\#7463](https://github.com/matrix-org/synapse/issues/7463))
-- Fix copy-paste error in `ServerNoticesConfig` docstring. Contributed by @ptman. ([\#7477](https://github.com/matrix-org/synapse/issues/7477))
-- Improve the formatting of `reverse_proxy.md`. ([\#7514](https://github.com/matrix-org/synapse/issues/7514))
-- Change the systemd worker service to check that the worker config file exists instead of silently failing. Contributed by David Vo. ([\#7528](https://github.com/matrix-org/synapse/issues/7528))
-- Minor clarifications to the TURN docs. ([\#7533](https://github.com/matrix-org/synapse/issues/7533))
-
+- Fix a long-standing bug which could cause messages not to be sent over federation, when state events with state keys matching user IDs (such as custom user statuses) were received. ([\#7376](https://github.com/matrix-org/synapse/issues/7376))
+- Restore compatibility with non-compliant clients during the user interactive authentication process, fixing a problem introduced in v1.13.0rc1. ([\#7483](https://github.com/matrix-org/synapse/issues/7483))
 
 Internal Changes
 ----------------
 
-- Add typing annotations in `synapse.federation`. ([\#7382](https://github.com/matrix-org/synapse/issues/7382))
-- Convert the room handler to async/await. ([\#7396](https://github.com/matrix-org/synapse/issues/7396))
-- Improve performance of `get_e2e_cross_signing_key`. ([\#7428](https://github.com/matrix-org/synapse/issues/7428))
-- Improve performance of `mark_as_sent_devices_by_remote`. ([\#7429](https://github.com/matrix-org/synapse/issues/7429), [\#7562](https://github.com/matrix-org/synapse/issues/7562))
-- Add type hints to the SAML handler. ([\#7445](https://github.com/matrix-org/synapse/issues/7445))
-- Remove storage method `get_hosts_in_room` that is no longer called anywhere. ([\#7448](https://github.com/matrix-org/synapse/issues/7448))
-- Fix some typos in the `notice_expiry` templates. ([\#7449](https://github.com/matrix-org/synapse/issues/7449))
-- Convert the federation handler to async/await. ([\#7459](https://github.com/matrix-org/synapse/issues/7459))
-- Convert the search handler to async/await. ([\#7460](https://github.com/matrix-org/synapse/issues/7460))
-- Add type hints to `synapse.event_auth`. ([\#7505](https://github.com/matrix-org/synapse/issues/7505))
-- Convert the room member handler to async/await. ([\#7507](https://github.com/matrix-org/synapse/issues/7507))
-- Add type hints to room member handler. ([\#7513](https://github.com/matrix-org/synapse/issues/7513))
-- Fix typing annotations in `tests.replication`. ([\#7518](https://github.com/matrix-org/synapse/issues/7518))
-- Remove some redundant Python 2 support code. ([\#7519](https://github.com/matrix-org/synapse/issues/7519))
-- All endpoints now respond with a 200 OK for `OPTIONS` requests. ([\#7534](https://github.com/matrix-org/synapse/issues/7534), [\#7560](https://github.com/matrix-org/synapse/issues/7560))
-- Synapse now exports [detailed allocator statistics](https://doc.pypy.org/en/latest/gc_info.html#gc-get-stats) and basic GC timings as Prometheus metrics (`pypy_gc_time_seconds_total` and `pypy_memory_bytes`) when run under PyPy. Contributed by Ivan Shapovalov. ([\#7536](https://github.com/matrix-org/synapse/issues/7536))
-- Remove Ubuntu Cosmic and Disco from the list of distributions which we provide `.deb`s for, due to end-of-life. ([\#7539](https://github.com/matrix-org/synapse/issues/7539))
-- Make worker processes return a stubbed-out response to `GET /presence` requests. ([\#7545](https://github.com/matrix-org/synapse/issues/7545))
-- Optimise some references to `hs.config`. ([\#7546](https://github.com/matrix-org/synapse/issues/7546))
-- On upgrade room only send canonical alias once. ([\#7547](https://github.com/matrix-org/synapse/issues/7547))
-- Fix some indentation inconsistencies in the sample config. ([\#7550](https://github.com/matrix-org/synapse/issues/7550))
-- Include `synapse.http.site` in type checking. ([\#7553](https://github.com/matrix-org/synapse/issues/7553))
-- Fix some test code to not mangle stacktraces, to make it easier to debug errors. ([\#7554](https://github.com/matrix-org/synapse/issues/7554))
-- Refresh apt cache when building `dh_virtualenv` docker image. ([\#7555](https://github.com/matrix-org/synapse/issues/7555))
-- Stop logging some expected HTTP request errors as exceptions. ([\#7556](https://github.com/matrix-org/synapse/issues/7556), [\#7563](https://github.com/matrix-org/synapse/issues/7563))
-- Convert sending mail to async/await. ([\#7557](https://github.com/matrix-org/synapse/issues/7557))
-- Simplify `reap_monthly_active_users`. ([\#7558](https://github.com/matrix-org/synapse/issues/7558))
+- Fix linting errors in new version of Flake8. ([\#7470](https://github.com/matrix-org/synapse/issues/7470))
 
 
-Synapse 1.13.0 (2020-05-19)
-===========================
+Synapse 1.13.0rc1 (2020-05-11)
+==============================
 
 This release brings some potential changes necessary for certain
 configurations of Synapse:
@@ -126,53 +34,6 @@ configurations of Synapse:
 Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes
 and for general upgrade guidance.
 
-
-Notice of change to the default `git` branch for Synapse
---------------------------------------------------------
-
-With the release of Synapse 1.13.0, the default `git` branch for Synapse has
-changed to `develop`, which is the development tip. This is more consistent with
-common practice and modern `git` usage.
-
-The `master` branch, which tracks the latest release, is still available. It is
-recommended that developers and distributors who have scripts which run builds
-using the default branch of Synapse should therefore consider pinning their
-scripts to `master`.
-
-
-Internal Changes
-----------------
-
-- Update the version of dh-virtualenv we use to build debs, and add focal to the list of target distributions. ([\#7526](https://github.com/matrix-org/synapse/issues/7526))
-
-
-Synapse 1.13.0rc3 (2020-05-18)
-==============================
-
-Bugfixes
---------
-
-- Hash passwords as early as possible during registration. ([\#7523](https://github.com/matrix-org/synapse/issues/7523))
-
-
-Synapse 1.13.0rc2 (2020-05-14)
-==============================
-
-Bugfixes
---------
-
-- Fix a long-standing bug which could cause messages not to be sent over federation, when state events with state keys matching user IDs (such as custom user statuses) were received. ([\#7376](https://github.com/matrix-org/synapse/issues/7376))
-- Restore compatibility with non-compliant clients during the user interactive authentication process, fixing a problem introduced in v1.13.0rc1. ([\#7483](https://github.com/matrix-org/synapse/issues/7483))
-
-Internal Changes
-----------------
-
-- Fix linting errors in new version of Flake8. ([\#7470](https://github.com/matrix-org/synapse/issues/7470))
-
-
-Synapse 1.13.0rc1 (2020-05-11)
-==============================
-
 Features
 --------
 

CONTRIBUTING.md

@@ -1,48 +1,62 @@
-# Contributing code to Synapse
+# Contributing code to Matrix
 
-Everyone is welcome to contribute code to [matrix.org
-projects](https://github.com/matrix-org), provided that they are willing to
-license their contributions under the same license as the project itself. We
-follow a simple 'inbound=outbound' model for contributions: the act of
-submitting an 'inbound' contribution means that the contributor agrees to
-license the code under the same terms as the project's overall 'outbound'
-license - in our case, this is almost always Apache Software License v2 (see
-[LICENSE](LICENSE)).
+Everyone is welcome to contribute code to Matrix
+(https://github.com/matrix-org), provided that they are willing to license
+their contributions under the same license as the project itself. We follow a
+simple 'inbound=outbound' model for contributions: the act of submitting an
+'inbound' contribution means that the contributor agrees to license the code
+under the same terms as the project's overall 'outbound' license - in our
+case, this is almost always Apache Software License v2 (see [LICENSE](LICENSE)).
 
 ## How to contribute
 
-The preferred and easiest way to contribute changes is to fork the relevant
-project on github, and then [create a pull request](
-https://help.github.com/articles/using-pull-requests/) to ask us to pull your
-changes into our repo.
+The preferred and easiest way to contribute changes to Matrix is to fork the
+relevant project on github, and then [create a pull request](
+https://help.github.com/articles/using-pull-requests/) to ask us to pull
+your changes into our repo.
 
-Some other points to follow:
- 
- * Please base your changes on the `develop` branch.
-  
- * Please follow the [code style requirements](#code-style).
+**The single biggest thing you need to know is: please base your changes on
+the develop branch - *not* master.**
 
- * Please include a [changelog entry](#changelog) with each PR.
+We use the master branch to track the most recent release, so that folks who
+blindly clone the repo and automatically check out master get something that
+works. Develop is the unstable branch where all the development actually
+happens: the workflow is that contributors should fork the develop branch to
+make a 'feature' branch for a particular contribution, and then make a pull
+request to merge this back into the matrix.org 'official' develop branch. We
+use github's pull request workflow to review the contribution, and either ask
+you to make any refinements needed or merge it and make them ourselves. The
+changes will then land on master when we next do a release.
 
- * Please [sign off](#sign-off) your contribution.
+We use [Buildkite](https://buildkite.com/matrix-dot-org/synapse) for continuous
+integration. If your change breaks the build, this will be shown in GitHub, so
+please keep an eye on the pull request for feedback.
 
- * Please keep an eye on the pull request for feedback from the [continuous
-   integration system](#continuous-integration-and-testing) and try to fix any
-   errors that come up.
+To run unit tests in a local development environment, you can use:
 
- * If you need to [update your PR](#updating-your-pull-request), just add new
-   commits to your branch rather than rebasing.
+- ``tox -e py35`` (requires tox to be installed by ``pip install tox``)
+  for SQLite-backed Synapse on Python 3.5.
+- ``tox -e py36`` for SQLite-backed Synapse on Python 3.6.
+- ``tox -e py36-postgres`` for PostgreSQL-backed Synapse on Python 3.6
+  (requires a running local PostgreSQL with access to create databases).
+- ``./test_postgresql.sh`` for PostgreSQL-backed Synapse on Python 3.5
+  (requires Docker). Entirely self-contained, recommended if you don't want to
+  set up PostgreSQL yourself.
+
+Docker images are available for running the integration tests (SyTest) locally,
+see the [documentation in the SyTest repo](
+https://github.com/matrix-org/sytest/blob/develop/docker/README.md) for more
+information.
 
 ## Code style
 
-Synapse's code style is documented [here](docs/code_style.md). Please follow
-it, including the conventions for the [sample configuration
-file](docs/code_style.md#configuration-file-format).
+All Matrix projects have a well-defined code-style - and sometimes we've even
+got as far as documenting it... For instance, synapse's code style doc lives
+[here](docs/code_style.md).
 
-Many of the conventions are enforced by scripts which are run as part of the
-[continuous integration system](#continuous-integration-and-testing). To help
-check if you have followed the code style, you can run `scripts-dev/lint.sh`
-locally. You'll need python 3.6 or later, and to install a number of tools:
+To facilitate meeting these criteria you can run `scripts-dev/lint.sh`
+locally. Since this runs the tools listed in the above document, you'll need
+python 3.6 and to install each tool:
 
 ```
 # Install the dependencies
@@ -53,11 +67,9 @@ pip install -U black flake8 flake8-comprehensions isort
 ```
 
 **Note that the script does not just test/check, but also reformats code, so you
-may wish to ensure any new code is committed first**.
-
-By default, this script checks all files and can take some time; if you alter
-only certain files, you might wish to specify paths as arguments to reduce the
-run-time:
+may wish to ensure any new code is committed first**. By default this script
+checks all files and can take some time; if you alter only certain files, you
+might wish to specify paths as arguments to reduce the run-time:
 
 ```
 ./scripts-dev/lint.sh path/to/file1.py path/to/file2.py path/to/folder
@@ -70,6 +82,7 @@ Please ensure your changes match the cosmetic style of the existing project,
 and **never** mix cosmetic and functional changes in the same commit, as it
 makes it horribly hard to review otherwise.
 
+
 ## Changelog
 
 All changes, even minor ones, need a corresponding changelog / newsfragment
@@ -85,55 +98,24 @@ in the format of `PRnumber.type`. The type can be one of the following:
 * `removal` (also used for deprecations)
 * `misc` (for internal-only changes)
 
-This file will become part of our [changelog](
-https://github.com/matrix-org/synapse/blob/master/CHANGES.md) at the next
-release, so the content of the file should be a short description of your
-change in the same style as the rest of the changelog. The file can contain Markdown
-formatting, and should end with a full stop (.) or an exclamation mark (!) for
-consistency.
+The content of the file is your changelog entry, which should be a short
+description of your change in the same style as the rest of our [changelog](
+https://github.com/matrix-org/synapse/blob/master/CHANGES.md). The file can
+contain Markdown formatting, and should end with a full stop (.) or an
+exclamation mark (!) for consistency.
 
 Adding credits to the changelog is encouraged, we value your
 contributions and would like to have you shouted out in the release notes!
 
 For example, a fix in PR #1234 would have its changelog entry in
-`changelog.d/1234.bugfix`, and contain content like:
+`changelog.d/1234.bugfix`, and contain content like "The security levels of
+Florbs are now validated when received over federation. Contributed by Jane
+Matrix.".
 
-> The security levels of Florbs are now validated when received
-> via the `/federation/florb` endpoint. Contributed by Jane Matrix.
-
-If there are multiple pull requests involved in a single bugfix/feature/etc,
-then the content for each `changelog.d` file should be the same. Towncrier will
-merge the matching files together into a single changelog entry when we come to
-release.
-
-### How do I know what to call the changelog file before I create the PR?
-
-Obviously, you don't know if you should call your newsfile
-`1234.bugfix` or `5678.bugfix` until you create the PR, which leads to a
-chicken-and-egg problem.
-
-There are two options for solving this:
-
- 1. Open the PR without a changelog file, see what number you got, and *then*
-    add the changelog file to your branch (see [Updating your pull
-    request](#updating-your-pull-request)), or:
-
- 1. Look at the [list of all
-    issues/PRs](https://github.com/matrix-org/synapse/issues?q=), add one to the
-    highest number you see, and quickly open the PR before somebody else claims
-    your number.
-
-    [This
-    script](https://github.com/richvdh/scripts/blob/master/next_github_number.sh)
-    might be helpful if you find yourself doing this a lot.
-
-Sorry, we know it's a bit fiddly, but it's *really* helpful for us when we come
-to put together a release!
-
-### Debian changelog
+## Debian changelog
 
 Changes which affect the debian packaging files (in `debian`) are an
-exception to the rule that all changes require a `changelog.d` file.
+exception.
 
 In this case, you will need to add an entry to the debian changelog for the
 next release. For this, run the following command:
@@ -218,45 +200,19 @@ Git allows you to add this signoff automatically when using the `-s`
 flag to `git commit`, which uses the name and email set in your
 `user.name` and `user.email` git configs.
 
-## Continuous integration and testing
+## Merge Strategy
 
-[Buildkite](https://buildkite.com/matrix-dot-org/synapse) will automatically
-run a series of checks and tests against any PR which is opened against the
-project; if your change breaks the build, this will be shown in GitHub, with
-links to the build results. If your build fails, please try to fix the errors
-and update your branch.
+We use the commit history of develop/master extensively to identify
+when regressions were introduced and what changes have been made.
 
-To run unit tests in a local development environment, you can use:
+We aim to have a clean merge history, which means we normally squash-merge
+changes into develop. For small changes this means there is no need to rebase
+to clean up your PR before merging. Larger changes with an organised set of
+commits may be merged as-is, if the history is judged to be useful.
 
-- ``tox -e py35`` (requires tox to be installed by ``pip install tox``)
-  for SQLite-backed Synapse on Python 3.5.
-- ``tox -e py36`` for SQLite-backed Synapse on Python 3.6.
-- ``tox -e py36-postgres`` for PostgreSQL-backed Synapse on Python 3.6
-  (requires a running local PostgreSQL with access to create databases).
-- ``./test_postgresql.sh`` for PostgreSQL-backed Synapse on Python 3.5
-  (requires Docker). Entirely self-contained, recommended if you don't want to
-  set up PostgreSQL yourself.
-
-Docker images are available for running the integration tests (SyTest) locally,
-see the [documentation in the SyTest repo](
-https://github.com/matrix-org/sytest/blob/develop/docker/README.md) for more
-information.
-
-## Updating your pull request
-
-If you decide to make changes to your pull request - perhaps to address issues
-raised in a review, or to fix problems highlighted by [continuous
-integration](#continuous-integration-and-testing) - just add new commits to your
-branch, and push to GitHub. The pull request will automatically be updated.
-
-Please **avoid** rebasing your branch, especially once the PR has been
-reviewed: doing so makes it very difficult for a reviewer to see what has
-changed since a previous review.
-
-## Notes for maintainers on merging PRs etc
-
-There are some notes for those with commit access to the project on how we
-manage git [here](docs/dev/git.md).
+This use of squash-merging will mean PRs built on each other will be hard to 
+merge. We suggest avoiding these where possible, and if required, ensuring
+each PR has a tidy set of commits to ease merging.
 
 ## Conclusion
 

README.rst

@@ -1,11 +1,3 @@
-================
-Synapse |shield|
-================
-
-.. |shield| image:: https://img.shields.io/matrix/synapse:matrix.org?label=support&logo=matrix
-  :alt: (get support on #synapse:matrix.org)
-  :target: https://matrix.to/#/#synapse:matrix.org
-
 .. contents::
 
 Introduction
@@ -85,17 +77,6 @@ Thanks for using Matrix!
 [1] End-to-end encryption is currently in beta: `blog post <https://matrix.org/blog/2016/11/21/matrixs-olm-end-to-end-encryption-security-assessment-released-and-implemented-cross-platform-on-riot-at-last>`_.
 
 
-Support
-=======
-
-For support installing or managing Synapse, please join |room|_ (from a matrix.org
-account if necessary) and ask questions there. We do not use GitHub issues for
-support requests, only for bug reports and feature requests.
-
-.. |room| replace:: ``#synapse:matrix.org``
-.. _room: https://matrix.to/#/#synapse:matrix.org
-
-
 Synapse Installation
 ====================
 

UPGRADE.rst

@@ -75,16 +75,10 @@ for example:
      wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb
      dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
 
-Upgrading to v1.14.0
-====================
-
-This version includes a database update which is run as part of the upgrade,
-and which may take a couple of minutes in the case of a large server. Synapse
-will not respond to HTTP requests while this update is taking place.
-
 Upgrading to v1.13.0
 ====================
 
+
 Incorrect database migration in old synapse versions
 ----------------------------------------------------
 
@@ -142,12 +136,12 @@ back to v1.12.4 you need to:
 2. Decrease the schema version in the database:
 
    .. code:: sql
-
+    
       UPDATE schema_version SET version = 57;
 
 3. Downgrade Synapse by following the instructions for your installation method
    in the "Rolling back to older versions" section above.
-
+    
 
 Upgrading to v1.12.0
 ====================

changelog.d/6391.feature

@@ -0,0 +1 @@
+Synapse's cache factor can now be configured in `homeserver.yaml` by the `caches.global_factor` setting. Additionally, `caches.per_cache_factors` controls the cache factors for individual caches.

changelog.d/7256.feature

@@ -0,0 +1 @@
+Add OpenID Connect login/registration support. Contributed by Quentin Gliech, on behalf of [les Connecteurs](https://connecteu.rs).

changelog.d/7281.misc

@@ -0,0 +1 @@
+Add MultiWriterIdGenerator to support multiple concurrent writers of streams.

changelog.d/7317.feature

@@ -0,0 +1 @@
+Add room details admin endpoint. Contributed by Awesome Technologies Innovationslabor GmbH.

changelog.d/7374.misc

@@ -0,0 +1 @@
+Move catchup of replication streams logic to worker.

changelog.d/7382.misc

@@ -0,0 +1 @@
+Add typing annotations in `synapse.federation`.

changelog.d/7396.misc

@@ -0,0 +1 @@
+Convert the room handler to async/await.

changelog.d/7398.docker

@@ -0,0 +1 @@
+Update docker runtime image to Alpine v3.11. Contributed by @Starbix.

changelog.d/7428.misc

@@ -0,0 +1 @@
+Improve performance of `get_e2e_cross_signing_key`.

changelog.d/7429.misc

@@ -0,0 +1 @@
+Improve performance of `mark_as_sent_devices_by_remote`.

changelog.d/7435.feature

@@ -0,0 +1 @@
+Allow for using more than one spam checker module at once.

changelog.d/7436.misc

@@ -0,0 +1 @@
+Support any process writing to cache invalidation stream.

changelog.d/7440.misc

@@ -0,0 +1 @@
+Refactor event persistence database functions in preparation for allowing them to be run on non-master processes.

changelog.d/7445.misc

@@ -0,0 +1 @@
+Add type hints to the SAML handler.

changelog.d/7448.misc

@@ -0,0 +1 @@
+Remove storage method `get_hosts_in_room` that is no longer called anywhere.

changelog.d/7449.misc

@@ -0,0 +1 @@
+Fix some typos in the notice_expiry templates.

changelog.d/7458.doc

@@ -0,0 +1 @@
+Update information about mapping providers for SAML and OpenID.

changelog.d/7459.misc

@@ -0,0 +1 @@
+Convert the federation handler to async/await.

changelog.d/7460.misc

@@ -0,0 +1 @@
+Convert the search handler to async/await.

changelog.d/7470.misc

@@ -0,0 +1 @@
+Fix linting errors in new version of Flake8.

changelog.d/7475.misc

@@ -0,0 +1 @@
+Have all instance correctly respond to REPLICATE command.

changelog.d/7477.doc

@@ -0,0 +1 @@
+Fix copy-paste error in `ServerNoticesConfig` docstring. Contributed by @ptman.

changelog.d/7482.bugfix

@@ -0,0 +1 @@
+Fix Redis reconnection logic that can result in missed updates over replication if master reconnects to Redis without restarting.

changelog.d/7490.misc

@@ -0,0 +1 @@
+Clean up replication unit tests.

changelog.d/7491.misc

@@ -0,0 +1 @@
+Move event stream handling out of slave store.

changelog.d/7492.misc

@@ -0,0 +1 @@
+Allow censoring of events to happen on workers.

changelog.d/7493.misc

@@ -0,0 +1 @@
+Move EventStream handling into default ReplicationDataHandler.

changelog.d/7495.feature

@@ -0,0 +1 @@
+Add `instance_map` config and route replication calls.

debian/build_virtualenv

@@ -36,6 +36,7 @@ esac
 dh_virtualenv \
     --install-suffix "matrix-synapse" \
     --builtin-venv \
+    --setuptools \
     --python "$SNAKE" \
     --upgrade-pip \
     --preinstall="lxml" \

debian/changelog

@@ -1,24 +1,16 @@
-matrix-synapse-py3 (1.14.0) stable; urgency=medium
+<<<<<<< HEAD
+matrix-synapse-py3 (1.12.3ubuntu1) UNRELEASED; urgency=medium
 
-  * New synapse release 1.14.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Thu, 28 May 2020 10:37:27 +0000
-
-matrix-synapse-py3 (1.13.0) stable; urgency=medium
-
-  [ Patrick Cloke ]
   * Add information about .well-known files to Debian installation scripts.
 
-  [ Synapse Packaging team ]
-  * New synapse release 1.13.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 19 May 2020 09:16:56 -0400
-
+ -- Patrick Cloke <patrickc@matrix.org>  Mon, 06 Apr 2020 10:10:38 -0400
+=======
 matrix-synapse-py3 (1.12.4) stable; urgency=medium
 
   * New synapse release 1.12.4.
 
  -- Synapse Packaging team <packages@matrix.org>  Thu, 23 Apr 2020 10:58:14 -0400
+>>>>>>> master
 
 matrix-synapse-py3 (1.12.3) stable; urgency=medium
 

docker/Dockerfile-dhvirtualenv

@@ -27,18 +27,15 @@ RUN env DEBIAN_FRONTEND=noninteractive apt-get install \
         wget
 
 # fetch and unpack the package
-RUN mkdir /dh-virtualenv
-RUN wget -q -O /dh-virtualenv.tar.gz https://github.com/matrix-org/dh-virtualenv/archive/matrixorg-20200519.tar.gz
-RUN tar -xv --strip-components=1 -C /dh-virtualenv -f /dh-virtualenv.tar.gz
+RUN wget -q -O /dh-virtuenv-1.1.tar.gz https://github.com/spotify/dh-virtualenv/archive/1.1.tar.gz
+RUN tar xvf /dh-virtuenv-1.1.tar.gz
 
-# install its build deps. We do another apt-cache-update here, because we might
-# be using a stale cache from docker build.
-RUN apt-get update -qq -o Acquire::Languages=none \
-    && cd /dh-virtualenv \
-    && env DEBIAN_FRONTEND=noninteractive mk-build-deps -ri -t "apt-get -y --no-install-recommends"
+# install its build deps
+RUN cd dh-virtualenv-1.1/ \
+    && env DEBIAN_FRONTEND=noninteractive mk-build-deps -ri -t "apt-get -yqq --no-install-recommends"
 
 # build it
-RUN cd /dh-virtualenv && dpkg-buildpackage -us -uc -b
+RUN cd dh-virtualenv-1.1 && dpkg-buildpackage -us -uc -b
 
 ###
 ### Stage 1
@@ -71,12 +68,12 @@ RUN apt-get update -qq -o Acquire::Languages=none \
         sqlite3 \
         libpq-dev
 
-COPY --from=builder /dh-virtualenv_1.2~dev-1_all.deb /
+COPY --from=builder /dh-virtualenv_1.1-1_all.deb /
 
 # install dhvirtualenv. Update the apt cache again first, in case we got a
 # cached cache from docker the first time.
 RUN apt-get update -qq -o Acquire::Languages=none \
-    && apt-get install -yq /dh-virtualenv_1.2~dev-1_all.deb
+    && apt-get install -yq /dh-virtualenv_1.1-1_all.deb
 
 WORKDIR /synapse/source
 ENTRYPOINT ["bash","/synapse/source/docker/build_debian.sh"]

docs/admin_api/user_admin_api.rst

@@ -33,22 +33,21 @@ with a body of:
 
 including an ``access_token`` of a server admin.
 
-Parameters:
+The parameter ``displayname`` is optional and defaults to the value of
+``user_id``.
 
-- ``password``, optional. If provided, the user's password is updated and all
-  devices are logged out.
-  
-- ``displayname``, optional, defaults to the value of ``user_id``.
+The parameter ``threepids`` is optional and allows setting the third-party IDs
+(email, msisdn) belonging to a user.
 
-- ``threepids``, optional, allows setting the third-party IDs (email, msisdn)
-  belonging to a user.
+The parameter ``avatar_url`` is optional. Must be a [MXC
+URI](https://matrix.org/docs/spec/client_server/r0.6.0#matrix-content-mxc-uris).
 
-- ``avatar_url``, optional, must be a 
-  `MXC URI <https://matrix.org/docs/spec/client_server/r0.6.0#matrix-content-mxc-uris>`_.
+The parameter ``admin`` is optional and defaults to ``false``.
 
-- ``admin``, optional, defaults to ``false``.
+The parameter ``deactivated`` is optional and defaults to ``false``.
 
-- ``deactivated``, optional, defaults to ``false``.
+The parameter ``password`` is optional. If provided, the user's password is
+updated and all devices are logged out.
 
 If the user already exists then optional parameters default to the current value.
 

docs/dev/git.md

@@ -1,148 +0,0 @@
-Some notes on how we use git
-============================
-
-On keeping the commit history clean
------------------------------------
-
-In an ideal world, our git commit history would be a linear progression of
-commits each of which contains a single change building on what came
-before. Here, by way of an arbitrary example, is the top of `git log --graph
-b2dba0607`:
-
-<img src="git/clean.png" alt="clean git graph" width="500px">
-
-Note how the commit comment explains clearly what is changing and why. Also
-note the *absence* of merge commits, as well as the absence of commits called
-things like (to pick a few culprits):
-[“pep8”](https://github.com/matrix-org/synapse/commit/84691da6c), [“fix broken
-test”](https://github.com/matrix-org/synapse/commit/474810d9d),
-[“oops”](https://github.com/matrix-org/synapse/commit/c9d72e457),
-[“typo”](https://github.com/matrix-org/synapse/commit/836358823), or [“Who's
-the president?”](https://github.com/matrix-org/synapse/commit/707374d5d).
-
-There are a number of reasons why keeping a clean commit history is a good
-thing:
-
- * From time to time, after a change lands, it turns out to be necessary to
-   revert it, or to backport it to a release branch. Those operations are
-   *much* easier when the change is contained in a single commit.
-
- * Similarly, it's much easier to answer questions like “is the fix for
-   `/publicRooms` on the release branch?” if that change consists of a single
-   commit.
-
- * Likewise: “what has changed on this branch in the last week?” is much
-   clearer without merges and “pep8” commits everywhere.
-
- * Sometimes we need to figure out where a bug got introduced, or some
-   behaviour changed. One way of doing that is with `git bisect`: pick an
-   arbitrary commit between the known good point and the known bad point, and
-   see how the code behaves. However, that strategy fails if the commit you
-   chose is the middle of someone's epic branch in which they broke the world
-   before putting it back together again.
-
-One counterargument is that it is sometimes useful to see how a PR evolved as
-it went through review cycles. This is true, but that information is always
-available via the GitHub UI (or via the little-known [refs/pull
-namespace](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/checking-out-pull-requests-locally)).
-
-
-Of course, in reality, things are more complicated than that. We have release
-branches as well as `develop` and `master`, and we deliberately merge changes
-between them. Bugs often slip through and have to be fixed later. That's all
-fine: this not a cast-iron rule which must be obeyed, but an ideal to aim
-towards.
-
-Merges, squashes, rebases: wtf?
--------------------------------
-
-Ok, so that's what we'd like to achieve. How do we achieve it?
-
-The TL;DR is: when you come to merge a pull request, you *probably* want to
-“squash and merge”:
-
-![squash and merge](git/squash.png).
-
-(This applies whether you are merging your own PR, or that of another
-contributor.)
-
-“Squash and merge”<sup id="a1">[1](#f1)</sup> takes all of the changes in the
-PR, and bundles them into a single commit. GitHub gives you the opportunity to
-edit the commit message before you confirm, and normally you should do so,
-because the default will be useless (again: `* woops typo` is not a useful
-thing to keep in the historical record).
-
-The main problem with this approach comes when you have a series of pull
-requests which build on top of one another: as soon as you squash-merge the
-first PR, you'll end up with a stack of conflicts to resolve in all of the
-others. In general, it's best to avoid this situation in the first place by
-trying not to have multiple related PRs in flight at the same time. Still,
-sometimes that's not possible and doing a regular merge is the lesser evil.
-
-Another occasion in which a regular merge makes more sense is a PR where you've
-deliberately created a series of commits each of which makes sense in its own
-right. For example: [a PR which gradually propagates a refactoring operation
-through the codebase](https://github.com/matrix-org/synapse/pull/6837), or [a
-PR which is the culmination of several other
-PRs](https://github.com/matrix-org/synapse/pull/5987). In this case the ability
-to figure out when a particular change/bug was introduced could be very useful.
-
-Ultimately: **this is not a hard-and-fast-rule**. If in doubt, ask yourself “do
-each of the commits I am about to merge make sense in their own right”, but
-remember that we're just doing our best to balance “keeping the commit history
-clean” with other factors.
-
-Git branching model
--------------------
-
-A [lot](https://nvie.com/posts/a-successful-git-branching-model/)
-[of](http://scottchacon.com/2011/08/31/github-flow.html)
-[words](https://www.endoflineblog.com/gitflow-considered-harmful) have been
-written in the past about git branching models (no really, [a
-lot](https://martinfowler.com/articles/branching-patterns.html)). I tend to
-think the whole thing is overblown. Fundamentally, it's not that
-complicated. Here's how we do it.
-
-Let's start with a picture:
-
-![branching model](git/branches.jpg)
-
-It looks complicated, but it's really not. There's one basic rule: *anyone* is
-free to merge from *any* more-stable branch to *any* less-stable branch at
-*any* time<sup id="a2">[2](#f2)</sup>. (The principle behind this is that if a
-change is good enough for the more-stable branch, then it's also good enough go
-put in a less-stable branch.)
-
-Meanwhile, merging (or squashing, as per the above) from a less-stable to a
-more-stable branch is a deliberate action in which you want to publish a change
-or a set of changes to (some subset of) the world: for example, this happens
-when a PR is landed, or as part of our release process.
-
-So, what counts as a more- or less-stable branch? A little reflection will show
-that our active branches are ordered thus, from more-stable to less-stable:
-
- * `master` (tracks our last release).
- * `release-vX.Y.Z` (the branch where we prepare the next release)<sup
-   id="a3">[3](#f3)</sup>.
- * PR branches which are targeting the release.
- * `develop` (our "mainline" branch containing our bleeding-edge).
- * regular PR branches.
-
-The corollary is: if you have a bugfix that needs to land in both
-`release-vX.Y.Z` *and* `develop`, then you should base your PR on
-`release-vX.Y.Z`, get it merged there, and then merge from `release-vX.Y.Z` to
-`develop`. (If a fix lands in `develop` and we later need it in a
-release-branch, we can of course cherry-pick it, but landing it in the release
-branch first helps reduce the chance of annoying conflicts.)
-
----
-
-<b id="f1">[1]</b>: “Squash and merge” is GitHub's term for this
-operation. Given that there is no merge involved, I'm not convinced it's the
-most intuitive name. [^](#a1)
-
-<b id="f2">[2]</b>: Well, anyone with commit access.[^](#a2)
-
-<b id="f3">[3]</b>: Very, very occasionally (I think this has happened once in
-the history of Synapse), we've had two releases in flight at once. Obviously,
-`release-v1.2.3` is more-stable than `release-v1.3.0`. [^](#a3)

docs/reverse_proxy.md

@@ -9,7 +9,7 @@ of doing so is that it means that you can expose the default https port
 (443) to Matrix clients without needing to run Synapse with root
 privileges.
 
-**NOTE**: Your reverse proxy must not `canonicalise` or `normalise`
+> **NOTE**: Your reverse proxy must not `canonicalise` or `normalise`
 the requested URI in any way (for example, by decoding `%xx` escapes).
 Beware that Apache *will* canonicalise URIs unless you specifify
 `nocanon`.
@@ -18,7 +18,7 @@ When setting up a reverse proxy, remember that Matrix clients and other
 Matrix servers do not necessarily need to connect to your server via the
 same server name or port. Indeed, clients will use port 443 by default,
 whereas servers default to port 8448. Where these are different, we
-refer to the 'client port' and the 'federation port'. See [the Matrix
+refer to the 'client port' and the \'federation port\'. See [the Matrix
 specification](https://matrix.org/docs/spec/server_server/latest#resolving-server-names)
 for more details of the algorithm used for federation connections, and
 [delegate.md](<delegate.md>) for instructions on setting up delegation.
@@ -28,113 +28,93 @@ Let's assume that we expect clients to connect to our server at
 `https://example.com:8448`.  The following sections detail the configuration of
 the reverse proxy and the homeserver.
 
-## Reverse-proxy configuration examples
+## Webserver configuration examples
 
-**NOTE**: You only need one of these.
+> **NOTE**: You only need one of these.
 
 ### nginx
 
-```
-server {
-    listen 443 ssl;
-    listen [::]:443 ssl;
-    server_name matrix.example.com;
+        server {
+            listen 443 ssl;
+            listen [::]:443 ssl;
+            server_name matrix.example.com;
 
-    location /_matrix {
-        proxy_pass http://localhost:8008;
-        proxy_set_header X-Forwarded-For $remote_addr;
-        # Nginx by default only allows file uploads up to 1M in size
-        # Increase client_max_body_size to match max_upload_size defined in homeserver.yaml
-        client_max_body_size 10M;
-    }
-}
+            location /_matrix {
+                proxy_pass http://localhost:8008;
+                proxy_set_header X-Forwarded-For $remote_addr;
+                # Nginx by default only allows file uploads up to 1M in size
+                # Increase client_max_body_size to match max_upload_size defined in homeserver.yaml
+                client_max_body_size 10M;
+            }
+        }
 
-server {
-    listen 8448 ssl default_server;
-    listen [::]:8448 ssl default_server;
-    server_name example.com;
+        server {
+            listen 8448 ssl default_server;
+            listen [::]:8448 ssl default_server;
+            server_name example.com;
 
-    location / {
-        proxy_pass http://localhost:8008;
-        proxy_set_header X-Forwarded-For $remote_addr;
-    }
-}
-```
+            location / {
+                proxy_pass http://localhost:8008;
+                proxy_set_header X-Forwarded-For $remote_addr;
+            }
+        }
 
-**NOTE**: Do not add a path after the port in `proxy_pass`, otherwise nginx will
+> **NOTE**: Do not add a `/` after the port in `proxy_pass`, otherwise nginx will
 canonicalise/normalise the URI.
 
-### Caddy 1
+### Caddy
 
-```
-matrix.example.com {
-  proxy /_matrix http://localhost:8008 {
-    transparent
-  }
-}
+        matrix.example.com {
+          proxy /_matrix http://localhost:8008 {
+            transparent
+          }
+        }
 
-example.com:8448 {
-  proxy / http://localhost:8008 {
-    transparent
-  }
-}
-```
-
-### Caddy 2
-
-```
-matrix.example.com {
-  reverse_proxy /_matrix/* http://localhost:8008
-}
-
-example.com:8448 {
-  reverse_proxy http://localhost:8008
-}
-```
+        example.com:8448 {
+          proxy / http://localhost:8008 {
+            transparent
+          }
+        }
 
 ### Apache
 
-```
-<VirtualHost *:443>
-    SSLEngine on
-    ServerName matrix.example.com;
+        <VirtualHost *:443>
+            SSLEngine on
+            ServerName matrix.example.com;
 
-    AllowEncodedSlashes NoDecode
-    ProxyPass /_matrix http://127.0.0.1:8008/_matrix nocanon
-    ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix
-</VirtualHost>
+            AllowEncodedSlashes NoDecode
+            ProxyPass /_matrix http://127.0.0.1:8008/_matrix nocanon
+            ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix
+        </VirtualHost>
 
-<VirtualHost *:8448>
-    SSLEngine on
-    ServerName example.com;
+        <VirtualHost *:8448>
+            SSLEngine on
+            ServerName example.com;
 
-    AllowEncodedSlashes NoDecode
-    ProxyPass /_matrix http://127.0.0.1:8008/_matrix nocanon
-    ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix
-</VirtualHost>
-```
+            AllowEncodedSlashes NoDecode
+            ProxyPass /_matrix http://127.0.0.1:8008/_matrix nocanon
+            ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix
+        </VirtualHost>
 
-**NOTE**: ensure the  `nocanon` options are included.
+> **NOTE**: ensure the  `nocanon` options are included.
 
 ### HAProxy
 
-```
-frontend https
-  bind :::443 v4v6 ssl crt /etc/ssl/haproxy/ strict-sni alpn h2,http/1.1
+        frontend https
+          bind :::443 v4v6 ssl crt /etc/ssl/haproxy/ strict-sni alpn h2,http/1.1
 
-  # Matrix client traffic
-  acl matrix-host hdr(host) -i matrix.example.com
-  acl matrix-path path_beg /_matrix
+          # Matrix client traffic
+          acl matrix-host hdr(host) -i matrix.example.com
+          acl matrix-path path_beg /_matrix
 
-  use_backend matrix if matrix-host matrix-path
+          use_backend matrix if matrix-host matrix-path
 
-frontend matrix-federation
-  bind :::8448 v4v6 ssl crt /etc/ssl/haproxy/synapse.pem alpn h2,http/1.1
-  default_backend matrix
+        frontend matrix-federation
+          bind :::8448 v4v6 ssl crt /etc/ssl/haproxy/synapse.pem alpn h2,http/1.1
+          default_backend matrix
 
-backend matrix
-  server matrix 127.0.0.1:8008
-```
+        backend matrix
+          server matrix 127.0.0.1:8008
 
 ## Homeserver Configuration
 

docs/sample_config.yaml

@@ -322,27 +322,22 @@ listeners:
 # Used by phonehome stats to group together related servers.
 #server_context: context
 
-# Resource-constrained homeserver settings
+# Resource-constrained homeserver Settings
 #
-# When this is enabled, the room "complexity" will be checked before a user
-# joins a new remote room. If it is above the complexity limit, the server will
-# disallow joining, or will instantly leave.
+# If limit_remote_rooms.enabled is True, the room complexity will be
+# checked before a user joins a new remote room. If it is above
+# limit_remote_rooms.complexity, it will disallow joining or
+# instantly leave.
 #
-# Room complexity is an arbitrary measure based on factors such as the number of
-# users in the room.
+# limit_remote_rooms.complexity_error can be set to customise the text
+# displayed to the user when a room above the complexity threshold has
+# its join cancelled.
 #
-limit_remote_rooms:
-  # Uncomment to enable room complexity checking.
-  #
-  #enabled: true
-
-  # the limit above which rooms cannot be joined. The default is 1.0.
-  #
-  #complexity: 0.5
-
-  # override the error which is returned when the room is too complex.
-  #
-  #complexity_error: "This room is too complex."
+# Uncomment the below lines to enable:
+#limit_remote_rooms:
+#  enabled: true
+#  complexity: 1.0
+#  complexity_error: "This room is too complex."
 
 # Whether to require a user to be in the room to add an alias to it.
 # Defaults to 'true'.
@@ -643,12 +638,6 @@ caches:
    # takes priority over setting through the config file.
    # Ex. SYNAPSE_CACHE_FACTOR_GET_USERS_WHO_SHARE_ROOM_WITH_USER=2.0
    #
-   # Some caches have '*' and other characters that are not
-   # alphanumeric or underscores. These caches can be named with or
-   # without the special characters stripped. For example, to specify
-   # the cache factor for `*stateGroupCache*` via an environment
-   # variable would be `SYNAPSE_CACHE_FACTOR_STATEGROUPCACHE=2.0`.
-   #
    per_cache_factors:
      #get_users_who_share_room_with_user: 2.0
 
@@ -953,28 +942,25 @@ url_preview_accept_language:
 
 
 ## Captcha ##
-# See docs/CAPTCHA_SETUP.md for full details of configuring this.
+# See docs/CAPTCHA_SETUP for full details of configuring this.
 
-# This homeserver's ReCAPTCHA public key. Must be specified if
-# enable_registration_captcha is enabled.
+# This homeserver's ReCAPTCHA public key.
 #
 #recaptcha_public_key: "YOUR_PUBLIC_KEY"
 
-# This homeserver's ReCAPTCHA private key. Must be specified if
-# enable_registration_captcha is enabled.
+# This homeserver's ReCAPTCHA private key.
 #
 #recaptcha_private_key: "YOUR_PRIVATE_KEY"
 
-# Uncomment to enable ReCaptcha checks when registering, preventing signup
+# Enables ReCaptcha checks when registering, preventing signup
 # unless a captcha is answered. Requires a valid ReCaptcha
-# public/private key. Defaults to 'false'.
+# public/private key.
 #
-#enable_registration_captcha: true
+#enable_registration_captcha: false
 
 # The API endpoint to use for verifying m.login.recaptcha responses.
-# Defaults to "https://www.recaptcha.net/recaptcha/api/siteverify".
 #
-#recaptcha_siteverify_api: "https://my.recaptcha.site"
+#recaptcha_siteverify_api: "https://www.recaptcha.net/recaptcha/api/siteverify"
 
 
 ## TURN ##
@@ -1118,7 +1104,7 @@ account_validity:
 # If set, allows registration of standard or admin accounts by anyone who
 # has the shared secret, even if registration is otherwise disabled.
 #
-#registration_shared_secret: <PRIVATE STRING>
+# registration_shared_secret: <PRIVATE STRING>
 
 # Set the number of bcrypt rounds used to generate password hash.
 # Larger numbers increase the work factor needed to generate the hash.
@@ -1251,8 +1237,7 @@ metrics_flags:
     #known_servers: true
 
 # Whether or not to report anonymized homeserver usage statistics.
-#
-#report_stats: true|false
+# report_stats: true|false
 
 # The endpoint to report the anonymized homeserver usage statistics to.
 # Defaults to https://matrix.org/report-usage-stats/push
@@ -1288,13 +1273,13 @@ metrics_flags:
 # the registration_shared_secret is used, if one is given; otherwise,
 # a secret key is derived from the signing key.
 #
-#macaroon_secret_key: <PRIVATE STRING>
+# macaroon_secret_key: <PRIVATE STRING>
 
 # a secret which is used to calculate HMACs for form values, to stop
 # falsification of values. Must be specified for the User Consent
 # forms to work.
 #
-#form_secret: <PRIVATE STRING>
+# form_secret: <PRIVATE STRING>
 
 ## Signing Keys ##
 
@@ -1546,7 +1531,7 @@ oidc_config:
     # auth method to use when exchanging the token.
     # Valid values are "client_secret_basic" (default), "client_secret_post" and "none".
     #
-    #client_auth_method: "client_secret_basic"
+    #client_auth_method: "client_auth_basic"
 
     # list of scopes to ask. This should include the "openid" scope. Defaults to ["openid"].
     #
@@ -1779,8 +1764,8 @@ email:
   # Username/password for authentication to the SMTP server. By default, no
   # authentication is attempted.
   #
-  #smtp_user: "exampleusername"
-  #smtp_pass: "examplepassword"
+  # smtp_user: "exampleusername"
+  # smtp_pass: "examplepassword"
 
   # Uncomment the following to require TLS transport security for SMTP.
   # By default, Synapse will connect over plain text, and will then switch to

docs/systemd-with-workers/system/matrix-synapse-worker@.service

@@ -1,6 +1,6 @@
 [Unit]
 Description=Synapse %i
-AssertPathExists=/etc/matrix-synapse/workers/%i.yaml
+
 # This service should be restarted when the synapse target is restarted.
 PartOf=matrix-synapse.target
 

docs/turn-howto.md

@@ -18,7 +18,7 @@ For TURN relaying with `coturn` to work, it must be hosted on a server/endpoint
 Hosting TURN behind a NAT (even with appropriate port forwarding) is known to cause issues
 and to often not work.
 
-## `coturn` setup
+## `coturn` Setup
 
 ### Initial installation
 
@@ -26,13 +26,7 @@ The TURN daemon `coturn` is available from a variety of sources such as native p
 
 #### Debian installation
 
-Just install the debian package:
-
-```sh
-apt install coturn
-```
-
-This will install and start a systemd service called `coturn`.
+    # apt install coturn
 
 #### Source installation
 
@@ -69,52 +63,38 @@ This will install and start a systemd service called `coturn`.
 1.  Consider your security settings. TURN lets users request a relay which will
     connect to arbitrary IP addresses and ports. The following configuration is
     suggested as a minimum starting point:
-
+    
         # VoIP traffic is all UDP. There is no reason to let users connect to arbitrary TCP endpoints via the relay.
         no-tcp-relay
-
+        
         # don't let the relay ever try to connect to private IP address ranges within your network (if any)
         # given the turn server is likely behind your firewall, remember to include any privileged public IPs too.
         denied-peer-ip=10.0.0.0-10.255.255.255
         denied-peer-ip=192.168.0.0-192.168.255.255
         denied-peer-ip=172.16.0.0-172.31.255.255
-
+        
         # special case the turn server itself so that client->TURN->TURN->client flows work
         allowed-peer-ip=10.0.0.1
-
+        
         # consider whether you want to limit the quota of relayed streams per user (or total) to avoid risk of DoS.
         user-quota=12 # 4 streams per video call, so 12 streams = 3 simultaneous relayed calls per user.
         total-quota=1200
 
-1.  Also consider supporting TLS/DTLS. To do this, add the following settings
-    to `turnserver.conf`:
-
-        # TLS certificates, including intermediate certs.
-        # For Let's Encrypt certificates, use `fullchain.pem` here.
-        cert=/path/to/fullchain.pem
-
-        # TLS private key file
-        pkey=/path/to/privkey.pem
+    Ideally coturn should refuse to relay traffic which isn't SRTP; see
+    <https://github.com/matrix-org/synapse/issues/2009>
 
 1.  Ensure your firewall allows traffic into the TURN server on the ports
-    you've configured it to listen on (By default: 3478 and 5349 for the TURN(s)
-    traffic (remember to allow both TCP and UDP traffic), and ports 49152-65535
-    for the UDP relay.)
+    you've configured it to listen on (remember to allow both TCP and UDP TURN
+    traffic)
 
-1.  (Re)start the turn server:
+1.  If you've configured coturn to support TLS/DTLS, generate or import your
+    private key and certificate.
 
-    * If you used the Debian package (or have set up a systemd unit yourself):
-      ```sh
-      systemctl restart coturn
-      ```
+1.  Start the turn server:
 
-    * If you installed from source:
+         bin/turnserver -o
 
-      ```sh
-      bin/turnserver -o
-      ```
-
-## Synapse setup
+## synapse Setup
 
 Your home server configuration file needs the following extra keys:
 
@@ -146,14 +126,7 @@ As an example, here is the relevant section of the config file for matrix.org:
 
 After updating the homeserver configuration, you must restart synapse:
 
-  * If you use synctl:
-    ```sh
     cd /where/you/run/synapse
     ./synctl restart
-    ```
-  * If you use systemd:
-    ```
-    systemctl restart synapse.service
-    ```
 
 ..and your Home Server now supports VoIP relaying!

scripts-dev/build_debian_packages

@@ -24,8 +24,9 @@ DISTS = (
     "debian:sid",
     "ubuntu:xenial",
     "ubuntu:bionic",
+    "ubuntu:cosmic",
+    "ubuntu:disco",
     "ubuntu:eoan",
-    "ubuntu:focal",
 )
 
 DESC = '''\

scripts-dev/convert_server_keys.py

@@ -3,6 +3,8 @@ import json
 import sys
 import time
 
+import six
+
 import psycopg2
 import yaml
 from canonicaljson import encode_canonical_json
@@ -10,7 +12,10 @@ from signedjson.key import read_signing_keys
 from signedjson.sign import sign_json
 from unpaddedbase64 import encode_base64
 
-db_binary_type = memoryview
+if six.PY2:
+    db_type = six.moves.builtins.buffer
+else:
+    db_type = memoryview
 
 
 def select_v1_keys(connection):
@@ -67,7 +72,7 @@ def rows_v2(server, json):
     valid_until = json["valid_until_ts"]
     key_json = encode_canonical_json(json)
     for key_id in json["verify_keys"]:
-        yield (server, key_id, "-", valid_until, valid_until, db_binary_type(key_json))
+        yield (server, key_id, "-", valid_until, valid_until, db_type(key_json))
 
 
 def main():

scripts/synapse_port_db

@@ -196,9 +196,6 @@ class MockHomeserver:
     def get_reactor(self):
         return reactor
 
-    def get_instance_name(self):
-        return "master"
-
 
 class Porter(object):
     def __init__(self, **kwargs):

synapse/__init__.py

@@ -36,7 +36,7 @@ try:
 except ImportError:
     pass
 
-__version__ = "1.14.0"
+__version__ = "1.13.0rc2"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when

synapse/api/auth.py

@@ -22,7 +22,6 @@ import pymacaroons
 from netaddr import IPAddress
 
 from twisted.internet import defer
-from twisted.web.server import Request
 
 import synapse.logging.opentracing as opentracing
 import synapse.types
@@ -163,25 +162,19 @@ class Auth(object):
 
     @defer.inlineCallbacks
     def get_user_by_req(
-        self,
-        request: Request,
-        allow_guest: bool = False,
-        rights: str = "access",
-        allow_expired: bool = False,
+        self, request, allow_guest=False, rights="access", allow_expired=False
     ):
         """ Get a registered user's ID.
 
         Args:
-            request: An HTTP request with an access_token query parameter.
-            allow_guest: If False, will raise an AuthError if the user making the
-                request is a guest.
-            rights: The operation being performed; the access token must allow this
-            allow_expired: If True, allow the request through even if the account
-                is expired, or session token lifetime has ended. Note that
-                /login will deliver access tokens regardless of expiration.
-
+            request - An HTTP request with an access_token query parameter.
+            allow_expired - Whether to allow the request through even if the account is
+                expired. If true, Synapse will still require an access token to be
+                provided but won't check if the account it belongs to has expired. This
+                works thanks to /login delivering access tokens regardless of accounts'
+                expiration.
         Returns:
-            defer.Deferred: resolves to a `synapse.types.Requester` object
+            defer.Deferred: resolves to a ``synapse.types.Requester`` object
         Raises:
             InvalidClientCredentialsError if no user by that token exists or the token
                 is invalid.
@@ -212,9 +205,7 @@ class Auth(object):
 
                 return synapse.types.create_requester(user_id, app_service=app_service)
 
-            user_info = yield self.get_user_by_access_token(
-                access_token, rights, allow_expired=allow_expired
-            )
+            user_info = yield self.get_user_by_access_token(access_token, rights)
             user = user_info["user"]
             token_id = user_info["token_id"]
             is_guest = user_info["is_guest"]
@@ -289,17 +280,13 @@ class Auth(object):
         return user_id, app_service
 
     @defer.inlineCallbacks
-    def get_user_by_access_token(
-        self, token: str, rights: str = "access", allow_expired: bool = False,
-    ):
+    def get_user_by_access_token(self, token, rights="access"):
         """ Validate access token and get user_id from it
 
         Args:
-            token: The access token to get the user by
-            rights: The operation being performed; the access token must
-                allow this
-            allow_expired: If False, raises an InvalidClientTokenError
-                if the token is expired
+            token (str): The access token to get the user by.
+            rights (str): The operation being performed; the access token must
+                allow this.
         Returns:
             Deferred[dict]: dict that includes:
                `user` (UserID)
@@ -307,10 +294,8 @@ class Auth(object):
                `token_id` (int|None): access token id. May be None if guest
                `device_id` (str|None): device corresponding to access token
         Raises:
-            InvalidClientTokenError if a user by that token exists, but the token is
-                expired
             InvalidClientCredentialsError if no user by that token exists or the token
-                is invalid
+                is invalid.
         """
 
         if rights == "access":
@@ -319,8 +304,7 @@ class Auth(object):
             if r:
                 valid_until_ms = r["valid_until_ms"]
                 if (
-                    not allow_expired
-                    and valid_until_ms is not None
+                    valid_until_ms is not None
                     and valid_until_ms < self.clock.time_msec()
                 ):
                     # there was a valid access token, but it has expired.
@@ -591,7 +575,7 @@ class Auth(object):
         return user_level >= send_level
 
     @staticmethod
-    def has_access_token(request: Request):
+    def has_access_token(request):
         """Checks if the request has an access_token.
 
         Returns:
@@ -602,7 +586,7 @@ class Auth(object):
         return bool(query_params) or bool(auth_headers)
 
     @staticmethod
-    def get_access_token_from_request(request: Request):
+    def get_access_token_from_request(request):
         """Extracts the access_token from the request.
 
         Args:

synapse/api/room_versions.py

@@ -58,15 +58,7 @@ class RoomVersion(object):
     enforce_key_validity = attr.ib()  # bool
 
     # bool: before MSC2261/MSC2432, m.room.aliases had special auth rules and redaction rules
-    special_case_aliases_auth = attr.ib(type=bool)
-    # Strictly enforce canonicaljson, do not allow:
-    # * Integers outside the range of [-2 ^ 53 + 1, 2 ^ 53 - 1]
-    # * Floats
-    # * NaN, Infinity, -Infinity
-    strict_canonicaljson = attr.ib(type=bool)
-    # bool: MSC2209: Check 'notifications' key while verifying
-    # m.room.power_levels auth rules.
-    limit_notifications_power_levels = attr.ib(type=bool)
+    special_case_aliases_auth = attr.ib(type=bool, default=False)
 
 
 class RoomVersions(object):
@@ -77,8 +69,6 @@ class RoomVersions(object):
         StateResolutionVersions.V1,
         enforce_key_validity=False,
         special_case_aliases_auth=True,
-        strict_canonicaljson=False,
-        limit_notifications_power_levels=False,
     )
     V2 = RoomVersion(
         "2",
@@ -87,8 +77,6 @@ class RoomVersions(object):
         StateResolutionVersions.V2,
         enforce_key_validity=False,
         special_case_aliases_auth=True,
-        strict_canonicaljson=False,
-        limit_notifications_power_levels=False,
     )
     V3 = RoomVersion(
         "3",
@@ -97,8 +85,6 @@ class RoomVersions(object):
         StateResolutionVersions.V2,
         enforce_key_validity=False,
         special_case_aliases_auth=True,
-        strict_canonicaljson=False,
-        limit_notifications_power_levels=False,
     )
     V4 = RoomVersion(
         "4",
@@ -107,8 +93,6 @@ class RoomVersions(object):
         StateResolutionVersions.V2,
         enforce_key_validity=False,
         special_case_aliases_auth=True,
-        strict_canonicaljson=False,
-        limit_notifications_power_levels=False,
     )
     V5 = RoomVersion(
         "5",
@@ -117,18 +101,14 @@ class RoomVersions(object):
         StateResolutionVersions.V2,
         enforce_key_validity=True,
         special_case_aliases_auth=True,
-        strict_canonicaljson=False,
-        limit_notifications_power_levels=False,
     )
-    V6 = RoomVersion(
-        "6",
-        RoomDisposition.STABLE,
+    MSC2432_DEV = RoomVersion(
+        "org.matrix.msc2432",
+        RoomDisposition.UNSTABLE,
         EventFormatVersions.V3,
         StateResolutionVersions.V2,
         enforce_key_validity=True,
         special_case_aliases_auth=False,
-        strict_canonicaljson=True,
-        limit_notifications_power_levels=True,
     )
 
 
@@ -140,6 +120,6 @@ KNOWN_ROOM_VERSIONS = {
         RoomVersions.V3,
         RoomVersions.V4,
         RoomVersions.V5,
-        RoomVersions.V6,
+        RoomVersions.MSC2432_DEV,
     )
 }  # type: Dict[str, RoomVersion]

synapse/app/generic_worker.py

@@ -17,15 +17,16 @@
 import contextlib
 import logging
 import sys
-from typing import Dict, Iterable, Optional, Set
+from typing import Dict, Iterable
 
 from typing_extensions import ContextManager
 
 from twisted.internet import defer, reactor
+from twisted.web.resource import NoResource
 
 import synapse
 import synapse.events
-from synapse.api.errors import HttpResponseException, RequestSendFailed, SynapseError
+from synapse.api.errors import HttpResponseException, SynapseError
 from synapse.api.urls import (
     CLIENT_API_PREFIX,
     FEDERATION_PREFIX,
@@ -39,22 +40,14 @@ from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
 from synapse.federation import send_queue
 from synapse.federation.transport.server import TransportLayerServer
-from synapse.handlers.presence import (
-    BasePresenceHandler,
-    PresenceState,
-    get_interested_parties,
-)
-from synapse.http.server import JsonResource, OptionsResource
+from synapse.handlers.presence import BasePresenceHandler, get_interested_parties
+from synapse.http.server import JsonResource
 from synapse.http.servlet import RestServlet, parse_json_object_from_request
 from synapse.http.site import SynapseSite
 from synapse.logging.context import LoggingContext
 from synapse.metrics import METRICS_PREFIX, MetricsResource, RegistryProxy
 from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.replication.http import REPLICATION_PREFIX, ReplicationRestResource
-from synapse.replication.http.presence import (
-    ReplicationBumpPresenceActiveTime,
-    ReplicationPresenceSetState,
-)
 from synapse.replication.slave.storage._base import BaseSlavedStore
 from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
 from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
@@ -144,18 +137,31 @@ logger = logging.getLogger("synapse.app.generic_worker")
 
 class PresenceStatusStubServlet(RestServlet):
     """If presence is disabled this servlet can be used to stub out setting
-    presence status.
+    presence status, while proxying the getters to the master instance.
     """
 
     PATTERNS = client_patterns("/presence/(?P<user_id>[^/]*)/status")
 
     def __init__(self, hs):
         super(PresenceStatusStubServlet, self).__init__()
+        self.http_client = hs.get_simple_http_client()
         self.auth = hs.get_auth()
+        self.main_uri = hs.config.worker_main_http_uri
 
     async def on_GET(self, request, user_id):
-        await self.auth.get_user_by_req(request)
-        return 200, {"presence": "offline"}
+        # Pass through the auth headers, if any, in case the access token
+        # is there.
+        auth_headers = request.requestHeaders.getRawHeaders("Authorization", [])
+        headers = {"Authorization": auth_headers}
+
+        try:
+            result = await self.http_client.get_json(
+                self.main_uri + request.uri.decode("ascii"), headers=headers
+            )
+        except HttpResponseException as e:
+            raise e.to_synapse_error()
+
+        return 200, result
 
     async def on_PUT(self, request, user_id):
         await self.auth.get_user_by_req(request)
@@ -209,14 +215,9 @@ class KeyUploadServlet(RestServlet):
             # is there.
             auth_headers = request.requestHeaders.getRawHeaders(b"Authorization", [])
             headers = {"Authorization": auth_headers}
-            try:
-                result = await self.http_client.post_json_get_json(
-                    self.main_uri + request.uri.decode("ascii"), body, headers=headers
-                )
-            except HttpResponseException as e:
-                raise e.to_synapse_error() from e
-            except RequestSendFailed as e:
-                raise SynapseError(502, "Failed to talk to master") from e
+            result = await self.http_client.post_json_get_json(
+                self.main_uri + request.uri.decode("ascii"), body, headers=headers
+            )
 
             return 200, result
         else:
@@ -255,9 +256,6 @@ class GenericWorkerPresence(BasePresenceHandler):
         # but we haven't notified the master of that yet
         self.users_going_offline = {}
 
-        self._bump_active_client = ReplicationBumpPresenceActiveTime.make_client(hs)
-        self._set_state_client = ReplicationPresenceSetState.make_client(hs)
-
         self._send_stop_syncing_loop = self.clock.looping_call(
             self.send_stop_syncing, UPDATE_SYNCING_USERS_MS
         )
@@ -315,6 +313,10 @@ class GenericWorkerPresence(BasePresenceHandler):
                 self.users_going_offline.pop(user_id, None)
                 self.send_user_sync(user_id, False, last_sync_ms)
 
+    def set_state(self, user, state, ignore_status_msg=False):
+        # TODO Hows this supposed to work?
+        return defer.succeed(None)
+
     async def user_syncing(
         self, user_id: str, affect_presence: bool
     ) -> ContextManager[None]:
@@ -393,42 +395,6 @@ class GenericWorkerPresence(BasePresenceHandler):
             if count > 0
         ]
 
-    async def set_state(self, target_user, state, ignore_status_msg=False):
-        """Set the presence state of the user.
-        """
-        presence = state["presence"]
-
-        valid_presence = (
-            PresenceState.ONLINE,
-            PresenceState.UNAVAILABLE,
-            PresenceState.OFFLINE,
-        )
-        if presence not in valid_presence:
-            raise SynapseError(400, "Invalid presence state")
-
-        user_id = target_user.to_string()
-
-        # If presence is disabled, no-op
-        if not self.hs.config.use_presence:
-            return
-
-        # Proxy request to master
-        await self._set_state_client(
-            user_id=user_id, state=state, ignore_status_msg=ignore_status_msg
-        )
-
-    async def bump_presence_active_time(self, user):
-        """We've seen the user do something that indicates they're interacting
-        with the app.
-        """
-        # If presence is disabled, no-op
-        if not self.hs.config.use_presence:
-            return
-
-        # Proxy request to master
-        user_id = user.to_string()
-        await self._bump_active_client(user_id=user_id)
-
 
 class GenericWorkerTyping(object):
     def __init__(self, hs):
@@ -608,7 +574,7 @@ class GenericWorkerServer(HomeServer):
                 if name == "replication":
                     resources[REPLICATION_PREFIX] = ReplicationRestResource(self)
 
-        root_resource = create_resource_tree(resources, OptionsResource())
+        root_resource = create_resource_tree(resources, NoResource())
 
         _base.listen_tcp(
             bind_addresses,
@@ -677,9 +643,10 @@ class GenericWorkerReplicationHandler(ReplicationDataHandler):
         self.notify_pushers = hs.config.start_pushers
         self.pusher_pool = hs.get_pusherpool()
 
-        self.send_handler = None  # type: Optional[FederationSenderHandler]
         if hs.config.send_federation:
-            self.send_handler = FederationSenderHandler(hs)
+            self.send_handler = FederationSenderHandler(hs, self)
+        else:
+            self.send_handler = None
 
     async def on_rdata(self, stream_name, instance_name, token, rows):
         await super().on_rdata(stream_name, instance_name, token, rows)
@@ -717,7 +684,7 @@ class GenericWorkerReplicationHandler(ReplicationDataHandler):
                 if entities:
                     self.notifier.on_new_event("to_device_key", token, users=entities)
             elif stream_name == DeviceListsStream.NAME:
-                all_room_ids = set()  # type: Set[str]
+                all_room_ids = set()
                 for row in rows:
                     if row.entity.startswith("@"):
                         room_ids = await self.store.get_rooms_for_user(row.entity)
@@ -768,33 +735,24 @@ class GenericWorkerReplicationHandler(ReplicationDataHandler):
 
 
 class FederationSenderHandler(object):
-    """Processes the fedration replication stream
-
-    This class is only instantiate on the worker responsible for sending outbound
-    federation transactions. It receives rows from the replication stream and forwards
-    the appropriate entries to the FederationSender class.
+    """Processes the replication stream and forwards the appropriate entries
+    to the federation sender.
     """
 
-    def __init__(self, hs: GenericWorkerServer):
+    def __init__(self, hs: GenericWorkerServer, replication_client):
         self.store = hs.get_datastore()
         self._is_mine_id = hs.is_mine_id
         self.federation_sender = hs.get_federation_sender()
-        self._hs = hs
+        self.replication_client = replication_client
 
-        # if the worker is restarted, we want to pick up where we left off in
-        # the replication stream, so load the position from the database.
-        #
-        # XXX is this actually worthwhile? Whenever the master is restarted, we'll
-        # drop some rows anyway (which is mostly fine because we're only dropping
-        # typing and presence notifications). If the replication stream is
-        # unreliable, why do we do all this hoop-jumping to store the position in the
-        # database? See also https://github.com/matrix-org/synapse/issues/7535.
-        #
         self.federation_position = self.store.federation_out_pos_startup
-
         self._fed_position_linearizer = Linearizer(name="_fed_position_linearizer")
+
         self._last_ack = self.federation_position
 
+        self._room_serials = {}
+        self._room_typing = {}
+
     def on_start(self):
         # There may be some events that are persisted but haven't been sent,
         # so send them now.
@@ -857,34 +815,22 @@ class FederationSenderHandler(object):
             await self.federation_sender.send_read_receipt(receipt_info)
 
     async def update_token(self, token):
-        """Update the record of where we have processed to in the federation stream.
-
-        Called after we have processed a an update received over replication. Sends
-        a FEDERATION_ACK back to the master, and stores the token that we have processed
-         in `federation_stream_position` so that we can restart where we left off.
-        """
         try:
             self.federation_position = token
 
             # We linearize here to ensure we don't have races updating the token
-            #
-            # XXX this appears to be redundant, since the ReplicationCommandHandler
-            # has a linearizer which ensures that we only process one line of
-            # replication data at a time. Should we remove it, or is it doing useful
-            # service for robustness? Or could we replace it with an assertion that
-            # we're not being re-entered?
-
             with (await self._fed_position_linearizer.queue(None)):
-                await self.store.update_federation_out_pos(
-                    "federation", self.federation_position
-                )
+                if self._last_ack < self.federation_position:
+                    await self.store.update_federation_out_pos(
+                        "federation", self.federation_position
+                    )
 
-                # We ACK this token over replication so that the master can drop
-                # its in memory queues
-                self._hs.get_tcp_replication().send_federation_ack(
-                    self.federation_position
-                )
-                self._last_ack = self.federation_position
+                    # We ACK this token over replication so that the master can drop
+                    # its in memory queues
+                    self.replication_client.send_federation_ack(
+                        self.federation_position
+                    )
+                    self._last_ack = self.federation_position
         except Exception:
             logger.exception("Error updating federation stream position")
 

synapse/app/homeserver.py

@@ -31,7 +31,7 @@ from prometheus_client import Gauge
 from twisted.application import service
 from twisted.internet import defer, reactor
 from twisted.python.failure import Failure
-from twisted.web.resource import EncodingResourceWrapper, IResource
+from twisted.web.resource import EncodingResourceWrapper, IResource, NoResource
 from twisted.web.server import GzipEncoderFactory
 from twisted.web.static import File
 
@@ -52,11 +52,7 @@ from synapse.config._base import ConfigError
 from synapse.config.homeserver import HomeServerConfig
 from synapse.federation.transport.server import TransportLayerServer
 from synapse.http.additional_resource import AdditionalResource
-from synapse.http.server import (
-    OptionsResource,
-    RootOptionsRedirectResource,
-    RootRedirect,
-)
+from synapse.http.server import RootRedirect
 from synapse.http.site import SynapseSite
 from synapse.logging.context import LoggingContext
 from synapse.metrics import METRICS_PREFIX, MetricsResource, RegistryProxy
@@ -125,11 +121,11 @@ class SynapseHomeServer(HomeServer):
 
         # try to find something useful to redirect '/' to
         if WEB_CLIENT_PREFIX in resources:
-            root_resource = RootOptionsRedirectResource(WEB_CLIENT_PREFIX)
+            root_resource = RootRedirect(WEB_CLIENT_PREFIX)
         elif STATIC_PREFIX in resources:
-            root_resource = RootOptionsRedirectResource(STATIC_PREFIX)
+            root_resource = RootRedirect(STATIC_PREFIX)
         else:
-            root_resource = OptionsResource()
+            root_resource = NoResource()
 
         root_resource = create_resource_tree(resources, root_resource)
 

synapse/appservice/__init__.py

@@ -270,7 +270,7 @@ class ApplicationService(object):
     def is_exclusive_room(self, room_id):
         return self._is_exclusive(ApplicationService.NS_ROOMS, room_id)
 
-    def get_exclusive_user_regexes(self):
+    def get_exlusive_user_regexes(self):
         """Get the list of regexes used to determine if a user is exclusively
         registered by the AS
         """

synapse/config/cache.py

@@ -14,17 +14,13 @@
 # limitations under the License.
 
 import os
-import re
 from typing import Callable, Dict
 
 from ._base import Config, ConfigError
 
 # The prefix for all cache factor-related environment variables
-_CACHE_PREFIX = "SYNAPSE_CACHE_FACTOR"
-
-# Map from canonicalised cache name to cache.
 _CACHES = {}
-
+_CACHE_PREFIX = "SYNAPSE_CACHE_FACTOR"
 _DEFAULT_FACTOR_SIZE = 0.5
 _DEFAULT_EVENT_CACHE_SIZE = "10K"
 
@@ -41,20 +37,6 @@ class CacheProperties(object):
 properties = CacheProperties()
 
 
-def _canonicalise_cache_name(cache_name: str) -> str:
-    """Gets the canonical form of the cache name.
-
-    Since we specify cache names in config and environment variables we need to
-    ignore case and special characters. For example, some caches have asterisks
-    in their name to denote that they're not attached to a particular database
-    function, and these asterisks need to be stripped out
-    """
-
-    cache_name = re.sub(r"[^A-Za-z_1-9]", "", cache_name)
-
-    return cache_name.lower()
-
-
 def add_resizable_cache(cache_name: str, cache_resize_callback: Callable):
     """Register a cache that's size can dynamically change
 
@@ -63,10 +45,7 @@ def add_resizable_cache(cache_name: str, cache_resize_callback: Callable):
         cache_resize_callback: A callback function that will be ran whenever
             the cache needs to be resized
     """
-    # Some caches have '*' in them which we strip out.
-    cache_name = _canonicalise_cache_name(cache_name)
-
-    _CACHES[cache_name] = cache_resize_callback
+    _CACHES[cache_name.lower()] = cache_resize_callback
 
     # Ensure all loaded caches are sized appropriately
     #
@@ -126,12 +105,6 @@ class CacheConfig(Config):
            # takes priority over setting through the config file.
            # Ex. SYNAPSE_CACHE_FACTOR_GET_USERS_WHO_SHARE_ROOM_WITH_USER=2.0
            #
-           # Some caches have '*' and other characters that are not
-           # alphanumeric or underscores. These caches can be named with or
-           # without the special characters stripped. For example, to specify
-           # the cache factor for `*stateGroupCache*` via an environment
-           # variable would be `SYNAPSE_CACHE_FACTOR_STATEGROUPCACHE=2.0`.
-           #
            per_cache_factors:
              #get_users_who_share_room_with_user: 2.0
         """
@@ -157,17 +130,10 @@ class CacheConfig(Config):
         if not isinstance(individual_factors, dict):
             raise ConfigError("caches.per_cache_factors must be a dictionary")
 
-        # Canonicalise the cache names *before* updating with the environment
-        # variables.
-        individual_factors = {
-            _canonicalise_cache_name(key): val
-            for key, val in individual_factors.items()
-        }
-
         # Override factors from environment if necessary
         individual_factors.update(
             {
-                _canonicalise_cache_name(key[len(_CACHE_PREFIX) + 1 :]): float(val)
+                key[len(_CACHE_PREFIX) + 1 :].lower(): float(val)
                 for key, val in self._environ.items()
                 if key.startswith(_CACHE_PREFIX + "_")
             }
@@ -176,9 +142,9 @@ class CacheConfig(Config):
         for cache, factor in individual_factors.items():
             if not isinstance(factor, (int, float)):
                 raise ConfigError(
-                    "caches.per_cache_factors.%s must be a number" % (cache,)
+                    "caches.per_cache_factors.%s must be a number" % (cache.lower(),)
                 )
-            self.cache_factors[cache] = factor
+            self.cache_factors[cache.lower()] = factor
 
         # Resize all caches (if necessary) with the new factors we've loaded
         self.resize_all_caches()

synapse/config/captcha.py

@@ -32,26 +32,23 @@ class CaptchaConfig(Config):
     def generate_config_section(self, **kwargs):
         return """\
         ## Captcha ##
-        # See docs/CAPTCHA_SETUP.md for full details of configuring this.
+        # See docs/CAPTCHA_SETUP for full details of configuring this.
 
-        # This homeserver's ReCAPTCHA public key. Must be specified if
-        # enable_registration_captcha is enabled.
+        # This homeserver's ReCAPTCHA public key.
         #
         #recaptcha_public_key: "YOUR_PUBLIC_KEY"
 
-        # This homeserver's ReCAPTCHA private key. Must be specified if
-        # enable_registration_captcha is enabled.
+        # This homeserver's ReCAPTCHA private key.
         #
         #recaptcha_private_key: "YOUR_PRIVATE_KEY"
 
-        # Uncomment to enable ReCaptcha checks when registering, preventing signup
+        # Enables ReCaptcha checks when registering, preventing signup
         # unless a captcha is answered. Requires a valid ReCaptcha
-        # public/private key. Defaults to 'false'.
+        # public/private key.
         #
-        #enable_registration_captcha: true
+        #enable_registration_captcha: false
 
         # The API endpoint to use for verifying m.login.recaptcha responses.
-        # Defaults to "https://www.recaptcha.net/recaptcha/api/siteverify".
         #
-        #recaptcha_siteverify_api: "https://my.recaptcha.site"
+        #recaptcha_siteverify_api: "https://www.recaptcha.net/recaptcha/api/siteverify"
         """

synapse/config/emailconfig.py

@@ -311,8 +311,8 @@ class EmailConfig(Config):
           # Username/password for authentication to the SMTP server. By default, no
           # authentication is attempted.
           #
-          #smtp_user: "exampleusername"
-          #smtp_pass: "examplepassword"
+          # smtp_user: "exampleusername"
+          # smtp_pass: "examplepassword"
 
           # Uncomment the following to require TLS transport security for SMTP.
           # By default, Synapse will connect over plain text, and will then switch to

synapse/config/key.py

@@ -175,8 +175,8 @@ class KeyConfig(Config):
             )
             form_secret = 'form_secret: "%s"' % random_string_with_symbols(50)
         else:
-            macaroon_secret_key = "#macaroon_secret_key: <PRIVATE STRING>"
-            form_secret = "#form_secret: <PRIVATE STRING>"
+            macaroon_secret_key = "# macaroon_secret_key: <PRIVATE STRING>"
+            form_secret = "# form_secret: <PRIVATE STRING>"
 
         return (
             """\

synapse/config/metrics.py

@@ -93,11 +93,10 @@ class MetricsConfig(Config):
             #known_servers: true
 
         # Whether or not to report anonymized homeserver usage statistics.
-        #
         """
 
         if report_stats is None:
-            res += "#report_stats: true|false\n"
+            res += "# report_stats: true|false\n"
         else:
             res += "report_stats: %s\n" % ("true" if report_stats else "false")
 

synapse/config/oidc_config.py

@@ -112,7 +112,7 @@ class OIDCConfig(Config):
             # auth method to use when exchanging the token.
             # Valid values are "client_secret_basic" (default), "client_secret_post" and "none".
             #
-            #client_auth_method: "client_secret_basic"
+            #client_auth_method: "client_auth_basic"
 
             # list of scopes to ask. This should include the "openid" scope. Defaults to ["openid"].
             #

synapse/config/registration.py

@@ -148,7 +148,9 @@ class RegistrationConfig(Config):
                 random_string_with_symbols(50),
             )
         else:
-            registration_shared_secret = "#registration_shared_secret: <PRIVATE STRING>"
+            registration_shared_secret = (
+                "# registration_shared_secret: <PRIVATE STRING>"
+            )
 
         return (
             """\

synapse/config/server.py

@@ -434,7 +434,7 @@ class ServerConfig(Config):
             )
 
         self.limit_remote_rooms = LimitRemoteRoomsConfig(
-            **(config.get("limit_remote_rooms") or {})
+            **config.get("limit_remote_rooms", {})
         )
 
         bind_port = config.get("bind_port")
@@ -895,27 +895,22 @@ class ServerConfig(Config):
         # Used by phonehome stats to group together related servers.
         #server_context: context
 
-        # Resource-constrained homeserver settings
+        # Resource-constrained homeserver Settings
         #
-        # When this is enabled, the room "complexity" will be checked before a user
-        # joins a new remote room. If it is above the complexity limit, the server will
-        # disallow joining, or will instantly leave.
+        # If limit_remote_rooms.enabled is True, the room complexity will be
+        # checked before a user joins a new remote room. If it is above
+        # limit_remote_rooms.complexity, it will disallow joining or
+        # instantly leave.
         #
-        # Room complexity is an arbitrary measure based on factors such as the number of
-        # users in the room.
+        # limit_remote_rooms.complexity_error can be set to customise the text
+        # displayed to the user when a room above the complexity threshold has
+        # its join cancelled.
         #
-        limit_remote_rooms:
-          # Uncomment to enable room complexity checking.
-          #
-          #enabled: true
-
-          # the limit above which rooms cannot be joined. The default is 1.0.
-          #
-          #complexity: 0.5
-
-          # override the error which is returned when the room is too complex.
-          #
-          #complexity_error: "This room is too complex."
+        # Uncomment the below lines to enable:
+        #limit_remote_rooms:
+        #  enabled: true
+        #  complexity: 1.0
+        #  complexity_error: "This room is too complex."
 
         # Whether to require a user to be in the room to add an alias to it.
         # Defaults to 'true'.

synapse/config/workers.py

@@ -15,7 +15,7 @@
 
 import attr
 
-from ._base import Config, ConfigError
+from ._base import Config
 
 
 @attr.s
@@ -94,26 +94,15 @@ class WorkerConfig(Config):
                     bind_addresses.append("")
 
         # A map from instance name to host/port of their HTTP replication endpoint.
-        instance_map = config.get("instance_map") or {}
+        instance_map = config.get("instance_map", {}) or {}
         self.instance_map = {
             name: InstanceLocationConfig(**c) for name, c in instance_map.items()
         }
 
         # Map from type of streams to source, c.f. WriterLocations.
-        writers = config.get("stream_writers") or {}
+        writers = config.get("writers", {}) or {}
         self.writers = WriterLocations(**writers)
 
-        # Check that the configured writer for events also appears in
-        # `instance_map`.
-        if (
-            self.writers.events != "master"
-            and self.writers.events not in self.instance_map
-        ):
-            raise ConfigError(
-                "Instance %r is configured to write events but does not appear in `instance_map` config."
-                % (self.writers.events,)
-            )
-
     def read_arguments(self, args):
         # We support a bunch of command line arguments that override options in
         # the config. A lot of these options have a worker_* prefix when running

synapse/event_auth.py

@@ -15,7 +15,7 @@
 # limitations under the License.
 
 import logging
-from typing import List, Optional, Set, Tuple
+from typing import Set, Tuple
 
 from canonicaljson import encode_canonical_json
 from signedjson.key import decode_verify_key_bytes
@@ -29,19 +29,18 @@ from synapse.api.room_versions import (
     EventFormatVersions,
     RoomVersion,
 )
-from synapse.events import EventBase
-from synapse.types import StateMap, UserID, get_domain_from_id
+from synapse.types import UserID, get_domain_from_id
 
 logger = logging.getLogger(__name__)
 
 
 def check(
     room_version_obj: RoomVersion,
-    event: EventBase,
-    auth_events: StateMap[EventBase],
-    do_sig_check: bool = True,
-    do_size_check: bool = True,
-) -> None:
+    event,
+    auth_events,
+    do_sig_check=True,
+    do_size_check=True,
+):
     """ Checks if this event is correctly authed.
 
     Args:
@@ -182,7 +181,7 @@ def check(
     _can_send_event(event, auth_events)
 
     if event.type == EventTypes.PowerLevels:
-        _check_power_levels(room_version_obj, event, auth_events)
+        _check_power_levels(event, auth_events)
 
     if event.type == EventTypes.Redaction:
         check_redaction(room_version_obj, event, auth_events)
@@ -190,7 +189,7 @@ def check(
     logger.debug("Allowing! %s", event)
 
 
-def _check_size_limits(event: EventBase) -> None:
+def _check_size_limits(event):
     def too_big(field):
         raise EventSizeError("%s too large" % (field,))
 
@@ -208,18 +207,13 @@ def _check_size_limits(event: EventBase) -> None:
         too_big("event")
 
 
-def _can_federate(event: EventBase, auth_events: StateMap[EventBase]) -> bool:
+def _can_federate(event, auth_events):
     creation_event = auth_events.get((EventTypes.Create, ""))
-    # There should always be a creation event, but if not don't federate.
-    if not creation_event:
-        return False
 
     return creation_event.content.get("m.federate", True) is True
 
 
-def _is_membership_change_allowed(
-    event: EventBase, auth_events: StateMap[EventBase]
-) -> None:
+def _is_membership_change_allowed(event, auth_events):
     membership = event.content["membership"]
 
     # Check if this is the room creator joining:
@@ -345,25 +339,21 @@ def _is_membership_change_allowed(
         raise AuthError(500, "Unknown membership %s" % membership)
 
 
-def _check_event_sender_in_room(
-    event: EventBase, auth_events: StateMap[EventBase]
-) -> None:
+def _check_event_sender_in_room(event, auth_events):
     key = (EventTypes.Member, event.user_id)
     member_event = auth_events.get(key)
 
-    _check_joined_room(member_event, event.user_id, event.room_id)
+    return _check_joined_room(member_event, event.user_id, event.room_id)
 
 
-def _check_joined_room(member: Optional[EventBase], user_id: str, room_id: str) -> None:
+def _check_joined_room(member, user_id, room_id):
     if not member or member.membership != Membership.JOIN:
         raise AuthError(
             403, "User %s not in room %s (%s)" % (user_id, room_id, repr(member))
         )
 
 
-def get_send_level(
-    etype: str, state_key: Optional[str], power_levels_event: Optional[EventBase]
-) -> int:
+def get_send_level(etype, state_key, power_levels_event):
     """Get the power level required to send an event of a given type
 
     The federation spec [1] refers to this as "Required Power Level".
@@ -371,13 +361,13 @@ def get_send_level(
     https://matrix.org/docs/spec/server_server/unstable.html#definitions
 
     Args:
-        etype: type of event
-        state_key: state_key of state event, or None if it is not
+        etype (str): type of event
+        state_key (str|None): state_key of state event, or None if it is not
             a state event.
-        power_levels_event: power levels event
+        power_levels_event (synapse.events.EventBase|None): power levels event
             in force at this point in the room
     Returns:
-        power level required to send this event.
+        int: power level required to send this event.
     """
 
     if power_levels_event:
@@ -398,7 +388,7 @@ def get_send_level(
     return int(send_level)
 
 
-def _can_send_event(event: EventBase, auth_events: StateMap[EventBase]) -> bool:
+def _can_send_event(event, auth_events):
     power_levels_event = _get_power_level_event(auth_events)
 
     send_level = get_send_level(event.type, event.get("state_key"), power_levels_event)
@@ -420,9 +410,7 @@ def _can_send_event(event: EventBase, auth_events: StateMap[EventBase]) -> bool:
     return True
 
 
-def check_redaction(
-    room_version_obj: RoomVersion, event: EventBase, auth_events: StateMap[EventBase],
-) -> bool:
+def check_redaction(room_version_obj: RoomVersion, event, auth_events):
     """Check whether the event sender is allowed to redact the target event.
 
     Returns:
@@ -454,9 +442,7 @@ def check_redaction(
     raise AuthError(403, "You don't have permission to redact events")
 
 
-def _check_power_levels(
-    room_version_obj: RoomVersion, event: EventBase, auth_events: StateMap[EventBase],
-) -> None:
+def _check_power_levels(event, auth_events):
     user_list = event.content.get("users", {})
     # Validate users
     for k, v in user_list.items():
@@ -487,7 +473,7 @@ def _check_power_levels(
         ("redact", None),
         ("kick", None),
         ("invite", None),
-    ]  # type: List[Tuple[str, Optional[str]]]
+    ]
 
     old_list = current_state.content.get("users", {})
     for user in set(list(old_list) + list(user_list)):
@@ -498,14 +484,6 @@ def _check_power_levels(
     for ev_id in set(list(old_list) + list(new_list)):
         levels_to_check.append((ev_id, "events"))
 
-    # MSC2209 specifies these checks should also be done for the "notifications"
-    # key.
-    if room_version_obj.limit_notifications_power_levels:
-        old_list = current_state.content.get("notifications", {})
-        new_list = event.content.get("notifications", {})
-        for ev_id in set(list(old_list) + list(new_list)):
-            levels_to_check.append((ev_id, "notifications"))
-
     old_state = current_state.content
     new_state = event.content
 
@@ -517,12 +495,12 @@ def _check_power_levels(
             new_loc = new_loc.get(dir, {})
 
         if level_to_check in old_loc:
-            old_level = int(old_loc[level_to_check])  # type: Optional[int]
+            old_level = int(old_loc[level_to_check])
         else:
             old_level = None
 
         if level_to_check in new_loc:
-            new_level = int(new_loc[level_to_check])  # type: Optional[int]
+            new_level = int(new_loc[level_to_check])
         else:
             new_level = None
 
@@ -548,21 +526,21 @@ def _check_power_levels(
             )
 
 
-def _get_power_level_event(auth_events: StateMap[EventBase]) -> Optional[EventBase]:
+def _get_power_level_event(auth_events):
     return auth_events.get((EventTypes.PowerLevels, ""))
 
 
-def get_user_power_level(user_id: str, auth_events: StateMap[EventBase]) -> int:
+def get_user_power_level(user_id, auth_events):
     """Get a user's power level
 
     Args:
-        user_id: user's id to look up in power_levels
-        auth_events:
+        user_id (str): user's id to look up in power_levels
+        auth_events (dict[(str, str), synapse.events.EventBase]):
             state in force at this point in the room (or rather, a subset of
             it including at least the create event and power levels event.
 
     Returns:
-        the user's power level in this room.
+        int: the user's power level in this room.
     """
     power_level_event = _get_power_level_event(auth_events)
     if power_level_event:
@@ -588,7 +566,7 @@ def get_user_power_level(user_id: str, auth_events: StateMap[EventBase]) -> int:
             return 0
 
 
-def _get_named_level(auth_events: StateMap[EventBase], name: str, default: int) -> int:
+def _get_named_level(auth_events, name, default):
     power_level_event = _get_power_level_event(auth_events)
 
     if not power_level_event:
@@ -601,7 +579,7 @@ def _get_named_level(auth_events: StateMap[EventBase], name: str, default: int)
         return default
 
 
-def _verify_third_party_invite(event: EventBase, auth_events: StateMap[EventBase]):
+def _verify_third_party_invite(event, auth_events):
     """
     Validates that the invite event is authorized by a previous third-party invite.
 
@@ -676,7 +654,7 @@ def get_public_keys(invite_event):
     return public_keys
 
 
-def auth_types_for_event(event: EventBase) -> Set[Tuple[str, str]]:
+def auth_types_for_event(event) -> Set[Tuple[str, str]]:
     """Given an event, return a list of (EventType, StateKey) that may be
     needed to auth the event. The returned list may be a superset of what
     would actually be required depending on the full state of the room.

synapse/events/utils.py

@@ -14,7 +14,7 @@
 # limitations under the License.
 import collections
 import re
-from typing import Any, Mapping, Union
+from typing import Mapping, Union
 
 from six import string_types
 
@@ -23,7 +23,6 @@ from frozendict import frozendict
 from twisted.internet import defer
 
 from synapse.api.constants import EventTypes, RelationTypes
-from synapse.api.errors import Codes, SynapseError
 from synapse.api.room_versions import RoomVersion
 from synapse.util.async_helpers import yieldable_gather_results
 
@@ -450,35 +449,3 @@ def copy_power_levels_contents(
         raise TypeError("Invalid power_levels value for %s: %r" % (k, v))
 
     return power_levels
-
-
-def validate_canonicaljson(value: Any):
-    """
-    Ensure that the JSON object is valid according to the rules of canonical JSON.
-
-    See the appendix section 3.1: Canonical JSON.
-
-    This rejects JSON that has:
-    * An integer outside the range of [-2 ^ 53 + 1, 2 ^ 53 - 1]
-    * Floats
-    * NaN, Infinity, -Infinity
-    """
-    if isinstance(value, int):
-        if value <= -(2 ** 53) or 2 ** 53 <= value:
-            raise SynapseError(400, "JSON integer out of range", Codes.BAD_JSON)
-
-    elif isinstance(value, float):
-        # Note that Infinity, -Infinity, and NaN are also considered floats.
-        raise SynapseError(400, "Bad JSON value: float", Codes.BAD_JSON)
-
-    elif isinstance(value, (dict, frozendict)):
-        for v in value.values():
-            validate_canonicaljson(v)
-
-    elif isinstance(value, (list, tuple)):
-        for i in value:
-            validate_canonicaljson(i)
-
-    elif not isinstance(value, (bool, str)) and value is not None:
-        # Other potential JSON values (bool, None, str) are safe.
-        raise SynapseError(400, "Unknown JSON value", Codes.BAD_JSON)

synapse/events/validator.py

@@ -18,7 +18,6 @@ from six import integer_types, string_types
 from synapse.api.constants import MAX_ALIAS_LENGTH, EventTypes, Membership
 from synapse.api.errors import Codes, SynapseError
 from synapse.api.room_versions import EventFormatVersions
-from synapse.events.utils import validate_canonicaljson
 from synapse.types import EventID, RoomID, UserID
 
 
@@ -56,12 +55,6 @@ class EventValidator(object):
             if not isinstance(getattr(event, s), string_types):
                 raise SynapseError(400, "'%s' not a string type" % (s,))
 
-        # Depending on the room version, ensure the data is spec compliant JSON.
-        if event.room_version.strict_canonicaljson:
-            # Note that only the client controlled portion of the event is
-            # checked, since we trust the portions of the event we created.
-            validate_canonicaljson(event.content)
-
         if event.type == EventTypes.Aliases:
             if "aliases" in event.content:
                 for alias in event.content["aliases"]:

synapse/federation/federation_base.py

@@ -29,7 +29,7 @@ from synapse.api.room_versions import EventFormatVersions, RoomVersion
 from synapse.crypto.event_signing import check_event_content_hash
 from synapse.crypto.keyring import Keyring
 from synapse.events import EventBase, make_event_from_dict
-from synapse.events.utils import prune_event, validate_canonicaljson
+from synapse.events.utils import prune_event
 from synapse.http.servlet import assert_params_in_dict
 from synapse.logging.context import (
     PreserveLoggingContext,
@@ -302,10 +302,6 @@ def event_from_pdu_json(
     elif depth > MAX_DEPTH:
         raise SynapseError(400, "Depth too large", Codes.BAD_JSON)
 
-    # Validate that the JSON conforms to the specification.
-    if room_version.strict_canonicaljson:
-        validate_canonicaljson(pdu_json)
-
     event = make_event_from_dict(pdu_json, room_version)
     event.internal_metadata.outlier = outlier
 

synapse/federation/sender/per_destination_queue.py

@@ -80,9 +80,6 @@ class PerDestinationQueue(object):
 
         # a list of tuples of (pending pdu, order)
         self._pending_pdus = []  # type: List[Tuple[EventBase, int]]
-
-        # XXX this is never actually used: see
-        # https://github.com/matrix-org/synapse/issues/7549
         self._pending_edus = []  # type: List[Edu]
 
         # Pending EDUs by their "key". Keyed EDUs are EDUs that get clobbered

synapse/handlers/auth.py

@@ -80,9 +80,7 @@ class AuthHandler(BaseHandler):
         self.hs = hs  # FIXME better possibility to access registrationHandler later?
         self.macaroon_gen = hs.get_macaroon_generator()
         self._password_enabled = hs.config.password_enabled
-        self._sso_enabled = (
-            hs.config.cas_enabled or hs.config.saml2_enabled or hs.config.oidc_enabled
-        )
+        self._sso_enabled = hs.config.saml2_enabled or hs.config.cas_enabled
 
         # we keep this as a list despite the O(N^2) implication so that we can
         # keep PASSWORD first and avoid confusing clients which pick the first

synapse/handlers/device.py

@@ -29,7 +29,6 @@ from synapse.api.errors import (
     SynapseError,
 )
 from synapse.logging.opentracing import log_kv, set_tag, trace
-from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.types import RoomStreamToken, get_domain_from_id
 from synapse.util import stringutils
 from synapse.util.async_helpers import Linearizer
@@ -536,15 +535,6 @@ class DeviceListUpdater(object):
             iterable=True,
         )
 
-        # Attempt to resync out of sync device lists every 30s.
-        self._resync_retry_in_progress = False
-        self.clock.looping_call(
-            run_as_background_process,
-            30 * 1000,
-            func=self._maybe_retry_device_resync,
-            desc="_maybe_retry_device_resync",
-        )
-
     @trace
     @defer.inlineCallbacks
     def incoming_device_list_update(self, origin, edu_content):
@@ -689,50 +679,11 @@ class DeviceListUpdater(object):
         return False
 
     @defer.inlineCallbacks
-    def _maybe_retry_device_resync(self):
-        """Retry to resync device lists that are out of sync, except if another retry is
-        in progress.
-        """
-        if self._resync_retry_in_progress:
-            return
-
-        try:
-            # Prevent another call of this function to retry resyncing device lists so
-            # we don't send too many requests.
-            self._resync_retry_in_progress = True
-            # Get all of the users that need resyncing.
-            need_resync = yield self.store.get_user_ids_requiring_device_list_resync()
-            # Iterate over the set of user IDs.
-            for user_id in need_resync:
-                # Try to resync the current user's devices list. Exception handling
-                # isn't necessary here, since user_device_resync catches all instances
-                # of "Exception" that might be raised from the federation request. This
-                # means that if an exception is raised by this function, it must be
-                # because of a database issue, which means _maybe_retry_device_resync
-                # probably won't be able to go much further anyway.
-                result = yield self.user_device_resync(
-                    user_id=user_id, mark_failed_as_stale=False,
-                )
-                # user_device_resync only returns a result if it managed to successfully
-                # resync and update the database. Updating the table of users requiring
-                # resync isn't necessary here as user_device_resync already does it
-                # (through self.store.update_remote_device_list_cache).
-                if result:
-                    logger.debug(
-                        "Successfully resynced the device list for %s" % user_id,
-                    )
-        finally:
-            # Allow future calls to retry resyncinc out of sync device lists.
-            self._resync_retry_in_progress = False
-
-    @defer.inlineCallbacks
-    def user_device_resync(self, user_id, mark_failed_as_stale=True):
+    def user_device_resync(self, user_id):
         """Fetches all devices for a user and updates the device cache with them.
 
         Args:
             user_id (str): The user's id whose device_list will be updated.
-            mark_failed_as_stale (bool): Whether to mark the user's device list as stale
-                if the attempt to resync failed.
         Returns:
             Deferred[dict]: a dict with device info as under the "devices" in the result of this
             request:
@@ -743,23 +694,10 @@ class DeviceListUpdater(object):
         origin = get_domain_from_id(user_id)
         try:
             result = yield self.federation.query_user_devices(origin, user_id)
-        except NotRetryingDestination:
-            if mark_failed_as_stale:
-                # Mark the remote user's device list as stale so we know we need to retry
-                # it later.
-                yield self.store.mark_remote_user_device_cache_as_stale(user_id)
-
-            return
-        except (RequestSendFailed, HttpResponseException) as e:
-            logger.warning(
-                "Failed to handle device list update for %s: %s", user_id, e,
-            )
-
-            if mark_failed_as_stale:
-                # Mark the remote user's device list as stale so we know we need to retry
-                # it later.
-                yield self.store.mark_remote_user_device_cache_as_stale(user_id)
-
+        except (NotRetryingDestination, RequestSendFailed, HttpResponseException):
+            # TODO: Remember that we are now out of sync and try again
+            # later
+            logger.warning("Failed to handle device list update for %s", user_id)
             # We abort on exceptions rather than accepting the update
             # as otherwise synapse will 'forget' that its device list
             # is out of date. If we bail then we will retry the resync
@@ -773,17 +711,13 @@ class DeviceListUpdater(object):
             logger.info(e)
             return
         except Exception as e:
+            # TODO: Remember that we are now out of sync and try again
+            # later
             set_tag("error", True)
             log_kv(
                 {"message": "Exception raised by federation request", "exception": e}
             )
             logger.exception("Failed to handle device list update for %s", user_id)
-
-            if mark_failed_as_stale:
-                # Mark the remote user's device list as stale so we know we need to retry
-                # it later.
-                yield self.store.mark_remote_user_device_cache_as_stale(user_id)
-
             return
         log_kv({"result": result})
         stream_id = result["stream_id"]

synapse/handlers/federation.py

@@ -40,7 +40,6 @@ from synapse.api.errors import (
     Codes,
     FederationDeniedError,
     FederationError,
-    HttpResponseException,
     RequestSendFailed,
     SynapseError,
 )
@@ -127,7 +126,6 @@ class FederationHandler(BaseHandler):
         self.config = hs.config
         self.http_client = hs.get_simple_http_client()
         self._instance_name = hs.get_instance_name()
-        self._replication = hs.get_replication_data_handler()
 
         self._send_events = ReplicationFederationSendEventsRestServlet.make_client(hs)
         self._notify_user_membership_change = ReplicationUserJoinedLeftRoomRestServlet.make_client(
@@ -1037,12 +1035,6 @@ class FederationHandler(BaseHandler):
                     # TODO: We can probably do something more intelligent here.
                     return True
                 except SynapseError as e:
-                    logger.info("Failed to backfill from %s because %s", dom, e)
-                    continue
-                except HttpResponseException as e:
-                    if 400 <= e.code < 500:
-                        raise e.to_synapse_error()
-
                     logger.info("Failed to backfill from %s because %s", dom, e)
                     continue
                 except CodeMessageException as e:
@@ -1221,7 +1213,7 @@ class FederationHandler(BaseHandler):
 
     async def do_invite_join(
         self, target_hosts: Iterable[str], room_id: str, joinee: str, content: JsonDict
-    ) -> Tuple[str, int]:
+    ) -> None:
         """ Attempts to join the `joinee` to the room `room_id` via the
         servers contained in `target_hosts`.
 
@@ -1242,10 +1234,6 @@ class FederationHandler(BaseHandler):
 
             content: The event content to use for the join event.
         """
-        # TODO: We should be able to call this on workers, but the upgrading of
-        # room stuff after join currently doesn't work on workers.
-        assert self.config.worker.worker_app is None
-
         logger.debug("Joining %s to %s", joinee, room_id)
 
         origin, event, room_version_obj = await self._make_and_verify_event(
@@ -1308,23 +1296,15 @@ class FederationHandler(BaseHandler):
                 room_id=room_id, room_version=room_version_obj,
             )
 
-            max_stream_id = await self._persist_auth_tree(
+            await self._persist_auth_tree(
                 origin, auth_chain, state, event, room_version_obj
             )
 
-            # We wait here until this instance has seen the events come down
-            # replication (if we're using replication) as the below uses caches.
-            #
-            # TODO: Currently the events stream is written to from master
-            await self._replication.wait_for_stream_position(
-                self.config.worker.writers.events, "events", max_stream_id
-            )
-
             # Check whether this room is the result of an upgrade of a room we already know
             # about. If so, migrate over user information
             predecessor = await self.store.get_room_predecessor(room_id)
             if not predecessor or not isinstance(predecessor.get("room_id"), str):
-                return event.event_id, max_stream_id
+                return
             old_room_id = predecessor["room_id"]
             logger.debug(
                 "Found predecessor for %s during remote join: %s", room_id, old_room_id
@@ -1337,7 +1317,6 @@ class FederationHandler(BaseHandler):
             )
 
             logger.debug("Finished joining %s to %s", joinee, room_id)
-            return event.event_id, max_stream_id
         finally:
             room_queue = self.room_queues[room_id]
             del self.room_queues[room_id]
@@ -1567,7 +1546,7 @@ class FederationHandler(BaseHandler):
 
     async def do_remotely_reject_invite(
         self, target_hosts: Iterable[str], room_id: str, user_id: str, content: JsonDict
-    ) -> Tuple[EventBase, int]:
+    ) -> EventBase:
         origin, event, room_version = await self._make_and_verify_event(
             target_hosts, room_id, user_id, "leave", content=content
         )
@@ -1587,9 +1566,9 @@ class FederationHandler(BaseHandler):
         await self.federation_client.send_leave(target_hosts, event)
 
         context = await self.state_handler.compute_event_context(event)
-        stream_id = await self.persist_events_and_notify([(event, context)])
+        await self.persist_events_and_notify([(event, context)])
 
-        return event, stream_id
+        return event
 
     async def _make_and_verify_event(
         self,
@@ -1901,7 +1880,7 @@ class FederationHandler(BaseHandler):
         state: List[EventBase],
         event: EventBase,
         room_version: RoomVersion,
-    ) -> int:
+    ) -> None:
         """Checks the auth chain is valid (and passes auth checks) for the
         state and event. Then persists the auth chain and state atomically.
         Persists the event separately. Notifies about the persisted events
@@ -1995,7 +1974,7 @@ class FederationHandler(BaseHandler):
             event, old_state=state
         )
 
-        return await self.persist_events_and_notify([(event, new_event_context)])
+        await self.persist_events_and_notify([(event, new_event_context)])
 
     async def _prep_event(
         self,
@@ -2848,7 +2827,7 @@ class FederationHandler(BaseHandler):
         self,
         event_and_contexts: Sequence[Tuple[EventBase, EventContext]],
         backfilled: bool = False,
-    ) -> int:
+    ) -> None:
         """Persists events and tells the notifier/pushers about them, if
         necessary.
 
@@ -2858,13 +2837,12 @@ class FederationHandler(BaseHandler):
                 backfilling or not
         """
         if self.config.worker.writers.events != self._instance_name:
-            result = await self._send_events(
+            await self._send_events(
                 instance_name=self.config.worker.writers.events,
                 store=self.store,
                 event_and_contexts=event_and_contexts,
                 backfilled=backfilled,
             )
-            return result["max_stream_id"]
         else:
             max_stream_id = await self.storage.persistence.persist_events(
                 event_and_contexts, backfilled=backfilled
@@ -2879,8 +2857,6 @@ class FederationHandler(BaseHandler):
                 for event, _ in event_and_contexts:
                     await self._notify_persisted_event(event, max_stream_id)
 
-            return max_stream_id
-
     async def _notify_persisted_event(
         self, event: EventBase, max_stream_id: int
     ) -> None:

synapse/handlers/identity.py

@@ -290,7 +290,8 @@ class IdentityHandler(BaseHandler):
 
         return changed
 
-    async def send_threepid_validation(
+    @defer.inlineCallbacks
+    def send_threepid_validation(
         self,
         email_address,
         client_secret,
@@ -318,7 +319,7 @@ class IdentityHandler(BaseHandler):
         """
         # Check that this email/client_secret/send_attempt combo is new or
         # greater than what we've seen previously
-        session = await self.store.get_threepid_validation_session(
+        session = yield self.store.get_threepid_validation_session(
             "email", client_secret, address=email_address, validated=False
         )
 
@@ -352,7 +353,7 @@ class IdentityHandler(BaseHandler):
         # Send the mail with the link containing the token, client_secret
         # and session_id
         try:
-            await send_email_func(email_address, token, client_secret, session_id)
+            yield send_email_func(email_address, token, client_secret, session_id)
         except Exception:
             logger.exception(
                 "Error sending threepid validation email to %s", email_address
@@ -363,7 +364,7 @@ class IdentityHandler(BaseHandler):
             self.hs.clock.time_msec() + self.hs.config.email_validation_token_lifetime
         )
 
-        await self.store.start_or_continue_validation_session(
+        yield self.store.start_or_continue_validation_session(
             "email",
             email_address,
             session_id,

synapse/handlers/message.py

@@ -15,7 +15,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import logging
-from typing import Optional, Tuple
+from typing import Optional
 
 from six import iteritems, itervalues, string_types
 
@@ -42,7 +42,6 @@ from synapse.api.errors import (
 )
 from synapse.api.room_versions import KNOWN_ROOM_VERSIONS, RoomVersions
 from synapse.api.urls import ConsentURIBuilder
-from synapse.events import EventBase
 from synapse.events.validator import EventValidator
 from synapse.logging.context import run_in_background
 from synapse.metrics.background_process_metrics import run_as_background_process
@@ -366,9 +365,7 @@ class EventCreationHandler(object):
         self.notifier = hs.get_notifier()
         self.config = hs.config
         self.require_membership_for_aliases = hs.config.require_membership_for_aliases
-        self._is_event_writer = (
-            self.config.worker.writers.events == hs.get_instance_name()
-        )
+        self._instance_name = hs.get_instance_name()
 
         self.room_invite_state_types = self.hs.config.room_invite_state_types
 
@@ -488,13 +485,9 @@ class EventCreationHandler(object):
 
                 try:
                     if "displayname" not in content:
-                        displayname = yield profile.get_displayname(target)
-                        if displayname is not None:
-                            content["displayname"] = displayname
+                        content["displayname"] = yield profile.get_displayname(target)
                     if "avatar_url" not in content:
-                        avatar_url = yield profile.get_avatar_url(target)
-                        if avatar_url is not None:
-                            content["avatar_url"] = avatar_url
+                        content["avatar_url"] = yield profile.get_avatar_url(target)
                 except Exception as e:
                     logger.info(
                         "Failed to get profile information for %r: %s", target, e
@@ -634,9 +627,7 @@ class EventCreationHandler(object):
         msg = self._block_events_without_consent_error % {"consent_uri": consent_uri}
         raise ConsentNotGivenError(msg=msg, consent_uri=consent_uri)
 
-    async def send_nonmember_event(
-        self, requester, event, context, ratelimit=True
-    ) -> int:
+    async def send_nonmember_event(self, requester, event, context, ratelimit=True):
         """
         Persists and notifies local clients and federation of an event.
 
@@ -645,9 +636,6 @@ class EventCreationHandler(object):
             context (Context) the context of the event.
             ratelimit (bool): Whether to rate limit this send.
             is_guest (bool): Whether the sender is a guest.
-
-        Return:
-            The stream_id of the persisted event.
         """
         if event.type == EventTypes.Member:
             raise SynapseError(
@@ -668,7 +656,7 @@ class EventCreationHandler(object):
                 )
                 return prev_state
 
-        return await self.handle_new_client_event(
+        await self.handle_new_client_event(
             requester=requester, event=event, context=context, ratelimit=ratelimit
         )
 
@@ -697,7 +685,7 @@ class EventCreationHandler(object):
 
     async def create_and_send_nonmember_event(
         self, requester, event_dict, ratelimit=True, txn_id=None
-    ) -> Tuple[EventBase, int]:
+    ):
         """
         Creates an event, then sends it.
 
@@ -720,10 +708,10 @@ class EventCreationHandler(object):
                     spam_error = "Spam is not permitted here"
                 raise SynapseError(403, spam_error, Codes.FORBIDDEN)
 
-            stream_id = await self.send_nonmember_event(
+            await self.send_nonmember_event(
                 requester, event, context, ratelimit=ratelimit
             )
-        return event, stream_id
+        return event
 
     @measure_func("create_new_client_event")
     @defer.inlineCallbacks
@@ -783,7 +771,7 @@ class EventCreationHandler(object):
     @measure_func("handle_new_client_event")
     async def handle_new_client_event(
         self, requester, event, context, ratelimit=True, extra_users=[]
-    ) -> int:
+    ):
         """Processes a new event. This includes checking auth, persisting it,
         notifying users, sending to remote servers, etc.
 
@@ -796,9 +784,6 @@ class EventCreationHandler(object):
             context (EventContext)
             ratelimit (bool)
             extra_users (list(UserID)): Any extra users to notify about event
-
-        Return:
-            The stream_id of the persisted event.
         """
 
         if event.is_state() and (event.type, event.state_key) == (
@@ -838,8 +823,8 @@ class EventCreationHandler(object):
         success = False
         try:
             # If we're a worker we need to hit out to the master.
-            if not self._is_event_writer:
-                result = await self.send_event(
+            if self.config.worker.writers.events != self._instance_name:
+                await self.send_event(
                     instance_name=self.config.worker.writers.events,
                     event_id=event.event_id,
                     store=self.store,
@@ -849,17 +834,14 @@ class EventCreationHandler(object):
                     ratelimit=ratelimit,
                     extra_users=extra_users,
                 )
-                stream_id = result["stream_id"]
-                event.internal_metadata.stream_ordering = stream_id
                 success = True
-                return stream_id
+                return
 
-            stream_id = await self.persist_and_notify_client_event(
+            await self.persist_and_notify_client_event(
                 requester, event, context, ratelimit=ratelimit, extra_users=extra_users
             )
 
             success = True
-            return stream_id
         finally:
             if not success:
                 # Ensure that we actually remove the entries in the push actions
@@ -902,13 +884,13 @@ class EventCreationHandler(object):
 
     async def persist_and_notify_client_event(
         self, requester, event, context, ratelimit=True, extra_users=[]
-    ) -> int:
+    ):
         """Called when we have fully built the event, have already
         calculated the push actions for the event, and checked auth.
 
-        This should only be run on the instance in charge of persisting events.
+        This should only be run on master.
         """
-        assert self._is_event_writer
+        assert self.config.worker.writers.events == self._instance_name
 
         if ratelimit:
             # We check if this is a room admin redacting an event so that we
@@ -1092,8 +1074,6 @@ class EventCreationHandler(object):
             # matters as sometimes presence code can take a while.
             run_in_background(self._bump_active_time, requester.user)
 
-        return event_stream_id
-
     async def _bump_active_time(self, user):
         try:
             presence = self.hs.get_presence_handler()

synapse/handlers/oidc_handler.py

@@ -311,7 +311,7 @@ class OidcHandler:
         ``ClientAuth`` to authenticate with the client with its ID and secret.
 
         Args:
-            code: The authorization code we got from the callback.
+            code: The autorization code we got from the callback.
 
         Returns:
             A dict containing various tokens.
@@ -497,14 +497,11 @@ class OidcHandler:
         return UserInfo(claims)
 
     async def handle_redirect_request(
-        self,
-        request: SynapseRequest,
-        client_redirect_url: bytes,
-        ui_auth_session_id: Optional[str] = None,
-    ) -> str:
+        self, request: SynapseRequest, client_redirect_url: bytes
+    ) -> None:
         """Handle an incoming request to /login/sso/redirect
 
-        It returns a redirect to the authorization endpoint with a few
+        It redirects the browser to the authorization endpoint with a few
         parameters:
 
           - ``client_id``: the client ID set in ``oidc_config.client_id``
@@ -514,32 +511,24 @@ class OidcHandler:
           - ``state``: a random string
           - ``nonce``: a random string
 
-        In addition generating a redirect URL, we are setting a cookie with
+        In addition to redirecting the client, we are setting a cookie with
         a signed macaroon token containing the state, the nonce and the
         client_redirect_url params. Those are then checked when the client
         comes back from the provider.
 
+
         Args:
             request: the incoming request from the browser.
                 We'll respond to it with a redirect and a cookie.
             client_redirect_url: the URL that we should redirect the client to
                 when everything is done
-            ui_auth_session_id: The session ID of the ongoing UI Auth (or
-                None if this is a login).
-
-        Returns:
-            The redirect URL to the authorization endpoint.
-
         """
 
         state = generate_token()
         nonce = generate_token()
 
         cookie = self._generate_oidc_session_token(
-            state=state,
-            nonce=nonce,
-            client_redirect_url=client_redirect_url.decode(),
-            ui_auth_session_id=ui_auth_session_id,
+            state=state, nonce=nonce, client_redirect_url=client_redirect_url.decode(),
         )
         request.addCookie(
             SESSION_COOKIE_NAME,
@@ -552,7 +541,7 @@ class OidcHandler:
 
         metadata = await self.load_metadata()
         authorization_endpoint = metadata.get("authorization_endpoint")
-        return prepare_grant_uri(
+        uri = prepare_grant_uri(
             authorization_endpoint,
             client_id=self._client_auth.client_id,
             response_type="code",
@@ -561,6 +550,8 @@ class OidcHandler:
             state=state,
             nonce=nonce,
         )
+        request.redirect(uri)
+        finish_request(request)
 
     async def handle_oidc_callback(self, request: SynapseRequest) -> None:
         """Handle an incoming request to /_synapse/oidc/callback
@@ -634,11 +625,7 @@ class OidcHandler:
 
         # Deserialize the session token and verify it.
         try:
-            (
-                nonce,
-                client_redirect_url,
-                ui_auth_session_id,
-            ) = self._verify_oidc_session_token(session, state)
+            nonce, client_redirect_url = self._verify_oidc_session_token(session, state)
         except MacaroonDeserializationException as e:
             logger.exception("Invalid session")
             self._render_error(request, "invalid_session", str(e))
@@ -691,21 +678,15 @@ class OidcHandler:
             return
 
         # and finally complete the login
-        if ui_auth_session_id:
-            await self._auth_handler.complete_sso_ui_auth(
-                user_id, ui_auth_session_id, request
-            )
-        else:
-            await self._auth_handler.complete_sso_login(
-                user_id, request, client_redirect_url
-            )
+        await self._auth_handler.complete_sso_login(
+            user_id, request, client_redirect_url
+        )
 
     def _generate_oidc_session_token(
         self,
         state: str,
         nonce: str,
         client_redirect_url: str,
-        ui_auth_session_id: Optional[str],
         duration_in_ms: int = (60 * 60 * 1000),
     ) -> str:
         """Generates a signed token storing data about an OIDC session.
@@ -721,8 +702,6 @@ class OidcHandler:
             nonce: The ``nonce`` parameter passed to the OIDC provider.
             client_redirect_url: The URL the client gave when it initiated the
                 flow.
-            ui_auth_session_id: The session ID of the ongoing UI Auth (or
-                None if this is a login).
             duration_in_ms: An optional duration for the token in milliseconds.
                 Defaults to an hour.
 
@@ -739,19 +718,12 @@ class OidcHandler:
         macaroon.add_first_party_caveat(
             "client_redirect_url = %s" % (client_redirect_url,)
         )
-        if ui_auth_session_id:
-            macaroon.add_first_party_caveat(
-                "ui_auth_session_id = %s" % (ui_auth_session_id,)
-            )
         now = self._clock.time_msec()
         expiry = now + duration_in_ms
         macaroon.add_first_party_caveat("time < %d" % (expiry,))
-
         return macaroon.serialize()
 
-    def _verify_oidc_session_token(
-        self, session: str, state: str
-    ) -> Tuple[str, str, Optional[str]]:
+    def _verify_oidc_session_token(self, session: str, state: str) -> Tuple[str, str]:
         """Verifies and extract an OIDC session token.
 
         This verifies that a given session token was issued by this homeserver
@@ -762,7 +734,7 @@ class OidcHandler:
             state: The state the OIDC provider gave back
 
         Returns:
-            The nonce, client_redirect_url, and ui_auth_session_id for this session
+            The nonce and the client_redirect_url for this session
         """
         macaroon = pymacaroons.Macaroon.deserialize(session)
 
@@ -772,27 +744,17 @@ class OidcHandler:
         v.satisfy_exact("state = %s" % (state,))
         v.satisfy_general(lambda c: c.startswith("nonce = "))
         v.satisfy_general(lambda c: c.startswith("client_redirect_url = "))
-        # Sometimes there's a UI auth session ID, it seems to be OK to attempt
-        # to always satisfy this.
-        v.satisfy_general(lambda c: c.startswith("ui_auth_session_id = "))
         v.satisfy_general(self._verify_expiry)
 
         v.verify(macaroon, self._macaroon_secret_key)
 
-        # Extract the `nonce`, `client_redirect_url`, and maybe the
-        # `ui_auth_session_id` from the token.
+        # Extract the `nonce` and `client_redirect_url` from the token
         nonce = self._get_value_from_macaroon(macaroon, "nonce")
         client_redirect_url = self._get_value_from_macaroon(
             macaroon, "client_redirect_url"
         )
-        try:
-            ui_auth_session_id = self._get_value_from_macaroon(
-                macaroon, "ui_auth_session_id"
-            )  # type: Optional[str]
-        except ValueError:
-            ui_auth_session_id = None
 
-        return nonce, client_redirect_url, ui_auth_session_id
+        return nonce, client_redirect_url
 
     def _get_value_from_macaroon(self, macaroon: pymacaroons.Macaroon, key: str) -> str:
         """Extracts a caveat value from a macaroon token.
@@ -811,7 +773,7 @@ class OidcHandler:
         for caveat in macaroon.caveats:
             if caveat.caveat_id.startswith(prefix):
                 return caveat.caveat_id[len(prefix) :]
-        raise ValueError("No %s caveat in macaroon" % (key,))
+        raise Exception("No %s caveat in macaroon" % (key,))
 
     def _verify_expiry(self, caveat: str) -> bool:
         prefix = "time < "

synapse/handlers/presence.py

@@ -193,12 +193,6 @@ class BasePresenceHandler(abc.ABC):
     ) -> None:
         """Set the presence state of the user. """
 
-    @abc.abstractmethod
-    async def bump_presence_active_time(self, user: UserID):
-        """We've seen the user do something that indicates they're interacting
-        with the app.
-        """
-
 
 class PresenceHandler(BasePresenceHandler):
     def __init__(self, hs: "synapse.server.HomeServer"):
@@ -210,7 +204,6 @@ class PresenceHandler(BasePresenceHandler):
         self.notifier = hs.get_notifier()
         self.federation = hs.get_federation_sender()
         self.state = hs.get_state_handler()
-        self._presence_enabled = hs.config.use_presence
 
         federation_registry = hs.get_federation_registry()
 
@@ -683,14 +676,13 @@ class PresenceHandler(BasePresenceHandler):
     async def incoming_presence(self, origin, content):
         """Called when we receive a `m.presence` EDU from a remote server.
         """
-        if not self._presence_enabled:
-            return
-
         now = self.clock.time_msec()
         updates = []
         for push in content.get("push", []):
             # A "push" contains a list of presence that we are probably interested
             # in.
+            # TODO: Actually check if we're interested, rather than blindly
+            # accepting presence updates.
             user_id = push.get("user_id", None)
             if not user_id:
                 logger.info(

synapse/handlers/register.py

@@ -132,7 +132,7 @@ class RegistrationHandler(BaseHandler):
     def register_user(
         self,
         localpart=None,
-        password_hash=None,
+        password=None,
         guest_access_token=None,
         make_guest=False,
         admin=False,
@@ -147,7 +147,7 @@ class RegistrationHandler(BaseHandler):
         Args:
             localpart: The local part of the user ID to register. If None,
               one will be generated.
-            password_hash (str|None): The hashed password to assign to this user so they can
+            password (unicode): The password to assign to this user so they can
               login again. This can be None which means they cannot login again
               via a password (e.g. the user is an application service user).
             user_type (str|None): type of user. One of the values from
@@ -164,6 +164,11 @@ class RegistrationHandler(BaseHandler):
         yield self.check_registration_ratelimit(address)
 
         yield self.auth.check_auth_blocking(threepid=threepid)
+        password_hash = None
+        if password:
+            password_hash = yield defer.ensureDeferred(
+                self._auth_handler.hash(password)
+            )
 
         if localpart is not None:
             yield self.check_username(localpart, guest_access_token=guest_access_token)

synapse/handlers/room.py

@@ -22,7 +22,6 @@ import logging
 import math
 import string
 from collections import OrderedDict
-from typing import Tuple
 
 from six import iteritems, string_types
 
@@ -89,8 +88,6 @@ class RoomCreationHandler(BaseHandler):
         self.room_member_handler = hs.get_room_member_handler()
         self.config = hs.config
 
-        self._replication = hs.get_replication_data_handler()
-
         # linearizer to stop two upgrades happening at once
         self._upgrade_linearizer = Linearizer("room_upgrade_linearizer")
 
@@ -442,78 +439,73 @@ class RoomCreationHandler(BaseHandler):
         new_room_id: str,
         old_room_state: StateMap[str],
     ):
+        directory_handler = self.hs.get_handlers().directory_handler
+
+        aliases = await self.store.get_aliases_for_room(old_room_id)
+
         # check to see if we have a canonical alias.
         canonical_alias_event = None
         canonical_alias_event_id = old_room_state.get((EventTypes.CanonicalAlias, ""))
         if canonical_alias_event_id:
             canonical_alias_event = await self.store.get_event(canonical_alias_event_id)
 
-        await self.store.update_aliases_for_room(old_room_id, new_room_id)
+        # first we try to remove the aliases from the old room (we suppress sending
+        # the room_aliases event until the end).
+        #
+        # Note that we'll only be able to remove aliases that (a) aren't owned by an AS,
+        # and (b) unless the user is a server admin, which the user created.
+        #
+        # This is probably correct - given we don't allow such aliases to be deleted
+        # normally, it would be odd to allow it in the case of doing a room upgrade -
+        # but it makes the upgrade less effective, and you have to wonder why a room
+        # admin can't remove aliases that point to that room anyway.
+        # (cf https://github.com/matrix-org/synapse/issues/2360)
+        #
+        removed_aliases = []
+        for alias_str in aliases:
+            alias = RoomAlias.from_string(alias_str)
+            try:
+                await directory_handler.delete_association(requester, alias)
+                removed_aliases.append(alias_str)
+            except SynapseError as e:
+                logger.warning("Unable to remove alias %s from old room: %s", alias, e)
 
-        if not canonical_alias_event:
+        # if we didn't find any aliases, or couldn't remove anyway, we can skip the rest
+        # of this.
+        if not removed_aliases:
             return
 
-        # If there is a canonical alias we need to update the one in the old
-        # room and set one in the new one.
-        old_canonical_alias_content = dict(canonical_alias_event.content)
-        new_canonical_alias_content = {}
-
-        canonical = canonical_alias_event.content.get("alias")
-        if canonical and self.hs.is_mine_id(canonical):
-            new_canonical_alias_content["alias"] = canonical
-            old_canonical_alias_content.pop("alias", None)
-
-        # We convert to a list as it will be a Tuple.
-        old_alt_aliases = list(old_canonical_alias_content.get("alt_aliases", []))
-        if old_alt_aliases:
-            old_canonical_alias_content["alt_aliases"] = old_alt_aliases
-            new_alt_aliases = new_canonical_alias_content.setdefault("alt_aliases", [])
-            for alias in canonical_alias_event.content.get("alt_aliases", []):
-                try:
-                    if self.hs.is_mine_id(alias):
-                        new_alt_aliases.append(alias)
-                        old_alt_aliases.remove(alias)
-                except Exception:
-                    logger.info(
-                        "Invalid alias %s in canonical alias event %s",
-                        alias,
-                        canonical_alias_event_id,
-                    )
-
-            if not old_alt_aliases:
-                old_canonical_alias_content.pop("alt_aliases")
+        # we can now add any aliases we successfully removed to the new room.
+        for alias in removed_aliases:
+            try:
+                await directory_handler.create_association(
+                    requester,
+                    RoomAlias.from_string(alias),
+                    new_room_id,
+                    servers=(self.hs.hostname,),
+                    check_membership=False,
+                )
+                logger.info("Moved alias %s to new room", alias)
+            except SynapseError as e:
+                # I'm not really expecting this to happen, but it could if the spam
+                # checking module decides it shouldn't, or similar.
+                logger.error("Error adding alias %s to new room: %s", alias, e)
 
         # If a canonical alias event existed for the old room, fire a canonical
         # alias event for the new room with a copy of the information.
         try:
-            await self.event_creation_handler.create_and_send_nonmember_event(
-                requester,
-                {
-                    "type": EventTypes.CanonicalAlias,
-                    "state_key": "",
-                    "room_id": old_room_id,
-                    "sender": requester.user.to_string(),
-                    "content": old_canonical_alias_content,
-                },
-                ratelimit=False,
-            )
-        except SynapseError as e:
-            # again I'm not really expecting this to fail, but if it does, I'd rather
-            # we returned the new room to the client at this point.
-            logger.error("Unable to send updated alias events in old room: %s", e)
-
-        try:
-            await self.event_creation_handler.create_and_send_nonmember_event(
-                requester,
-                {
-                    "type": EventTypes.CanonicalAlias,
-                    "state_key": "",
-                    "room_id": new_room_id,
-                    "sender": requester.user.to_string(),
-                    "content": new_canonical_alias_content,
-                },
-                ratelimit=False,
-            )
+            if canonical_alias_event:
+                await self.event_creation_handler.create_and_send_nonmember_event(
+                    requester,
+                    {
+                        "type": EventTypes.CanonicalAlias,
+                        "state_key": "",
+                        "room_id": new_room_id,
+                        "sender": requester.user.to_string(),
+                        "content": canonical_alias_event.content,
+                    },
+                    ratelimit=False,
+                )
         except SynapseError as e:
             # again I'm not really expecting this to fail, but if it does, I'd rather
             # we returned the new room to the client at this point.
@@ -521,7 +513,7 @@ class RoomCreationHandler(BaseHandler):
 
     async def create_room(
         self, requester, config, ratelimit=True, creator_join_profile=None
-    ) -> Tuple[dict, int]:
+    ):
         """ Creates a new room.
 
         Args:
@@ -538,9 +530,9 @@ class RoomCreationHandler(BaseHandler):
                 `avatar_url` and/or `displayname`.
 
         Returns:
-                First, a dict containing the keys `room_id` and, if an alias
-                was, requested, `room_alias`. Secondly, the stream_id of the
-                last persisted event.
+            Deferred[dict]:
+                a dict containing the keys `room_id` and, if an alias was
+                requested, `room_alias`.
         Raises:
             SynapseError if the room ID couldn't be stored, or something went
             horribly wrong.
@@ -672,7 +664,7 @@ class RoomCreationHandler(BaseHandler):
         # override any attempt to set room versions via the creation_content
         creation_content["room_version"] = room_version.identifier
 
-        last_stream_id = await self._send_events_for_new_room(
+        await self._send_events_for_new_room(
             requester,
             room_id,
             preset_config=preset_config,
@@ -686,10 +678,7 @@ class RoomCreationHandler(BaseHandler):
 
         if "name" in config:
             name = config["name"]
-            (
-                _,
-                last_stream_id,
-            ) = await self.event_creation_handler.create_and_send_nonmember_event(
+            await self.event_creation_handler.create_and_send_nonmember_event(
                 requester,
                 {
                     "type": EventTypes.Name,
@@ -703,10 +692,7 @@ class RoomCreationHandler(BaseHandler):
 
         if "topic" in config:
             topic = config["topic"]
-            (
-                _,
-                last_stream_id,
-            ) = await self.event_creation_handler.create_and_send_nonmember_event(
+            await self.event_creation_handler.create_and_send_nonmember_event(
                 requester,
                 {
                     "type": EventTypes.Topic,
@@ -724,7 +710,7 @@ class RoomCreationHandler(BaseHandler):
             if is_direct:
                 content["is_direct"] = is_direct
 
-            _, last_stream_id = await self.room_member_handler.update_membership(
+            await self.room_member_handler.update_membership(
                 requester,
                 UserID.from_string(invitee),
                 room_id,
@@ -738,7 +724,7 @@ class RoomCreationHandler(BaseHandler):
             id_access_token = invite_3pid.get("id_access_token")  # optional
             address = invite_3pid["address"]
             medium = invite_3pid["medium"]
-            last_stream_id = await self.hs.get_room_member_handler().do_3pid_invite(
+            await self.hs.get_room_member_handler().do_3pid_invite(
                 room_id,
                 requester.user,
                 medium,
@@ -754,12 +740,7 @@ class RoomCreationHandler(BaseHandler):
         if room_alias:
             result["room_alias"] = room_alias.to_string()
 
-        # Always wait for room creation to progate before returning
-        await self._replication.wait_for_stream_position(
-            self.hs.config.worker.writers.events, "events", last_stream_id
-        )
-
-        return result, last_stream_id
+        return result
 
     async def _send_events_for_new_room(
         self,
@@ -772,13 +753,7 @@ class RoomCreationHandler(BaseHandler):
         room_alias=None,
         power_level_content_override=None,  # Doesn't apply when initial state has power level state event content
         creator_join_profile=None,
-    ) -> int:
-        """Sends the initial events into a new room.
-
-        Returns:
-            The stream_id of the last event persisted.
-        """
-
+    ):
         def create(etype, content, **kwargs):
             e = {"type": etype, "content": content}
 
@@ -787,16 +762,12 @@ class RoomCreationHandler(BaseHandler):
 
             return e
 
-        async def send(etype, content, **kwargs) -> int:
+        async def send(etype, content, **kwargs):
             event = create(etype, content, **kwargs)
             logger.debug("Sending %s in new room", etype)
-            (
-                _,
-                last_stream_id,
-            ) = await self.event_creation_handler.create_and_send_nonmember_event(
+            await self.event_creation_handler.create_and_send_nonmember_event(
                 creator, event, ratelimit=False
             )
-            return last_stream_id
 
         config = RoomCreationHandler.PRESETS_DICT[preset_config]
 
@@ -821,9 +792,7 @@ class RoomCreationHandler(BaseHandler):
         # of the first events that get sent into a room.
         pl_content = initial_state.pop((EventTypes.PowerLevels, ""), None)
         if pl_content is not None:
-            last_sent_stream_id = await send(
-                etype=EventTypes.PowerLevels, content=pl_content
-            )
+            await send(etype=EventTypes.PowerLevels, content=pl_content)
         else:
             power_level_content = {
                 "users": {creator_id: 100},
@@ -856,39 +825,33 @@ class RoomCreationHandler(BaseHandler):
             if power_level_content_override:
                 power_level_content.update(power_level_content_override)
 
-            last_sent_stream_id = await send(
-                etype=EventTypes.PowerLevels, content=power_level_content
-            )
+            await send(etype=EventTypes.PowerLevels, content=power_level_content)
 
         if room_alias and (EventTypes.CanonicalAlias, "") not in initial_state:
-            last_sent_stream_id = await send(
+            await send(
                 etype=EventTypes.CanonicalAlias,
                 content={"alias": room_alias.to_string()},
             )
 
         if (EventTypes.JoinRules, "") not in initial_state:
-            last_sent_stream_id = await send(
+            await send(
                 etype=EventTypes.JoinRules, content={"join_rule": config["join_rules"]}
             )
 
         if (EventTypes.RoomHistoryVisibility, "") not in initial_state:
-            last_sent_stream_id = await send(
+            await send(
                 etype=EventTypes.RoomHistoryVisibility,
                 content={"history_visibility": config["history_visibility"]},
             )
 
         if config["guest_can_join"]:
             if (EventTypes.GuestAccess, "") not in initial_state:
-                last_sent_stream_id = await send(
+                await send(
                     etype=EventTypes.GuestAccess, content={"guest_access": "can_join"}
                 )
 
         for (etype, state_key), content in initial_state.items():
-            last_sent_stream_id = await send(
-                etype=etype, state_key=state_key, content=content
-            )
-
-        return last_sent_stream_id
+            await send(etype=etype, state_key=state_key, content=content)
 
     async def _generate_room_id(
         self, creator_id: str, is_public: str, room_version: RoomVersion,

synapse/handlers/room_member.py

@@ -17,19 +17,15 @@
 
 import abc
 import logging
-from typing import Dict, Iterable, List, Optional, Tuple
 
 from six.moves import http_client
 
+from twisted.internet import defer
+
 from synapse import types
 from synapse.api.constants import EventTypes, Membership
 from synapse.api.errors import AuthError, Codes, SynapseError
-from synapse.events import EventBase
-from synapse.events.snapshot import EventContext
-from synapse.replication.http.membership import (
-    ReplicationLocallyRejectInviteRestServlet,
-)
-from synapse.types import Collection, Requester, RoomAlias, RoomID, UserID
+from synapse.types import Collection, RoomID, UserID
 from synapse.util.async_helpers import Linearizer
 from synapse.util.distributor import user_joined_room, user_left_room
 
@@ -47,6 +43,11 @@ class RoomMemberHandler(object):
     __metaclass__ = abc.ABCMeta
 
     def __init__(self, hs):
+        """
+
+        Args:
+            hs (synapse.server.HomeServer):
+        """
         self.hs = hs
         self.store = hs.get_datastore()
         self.auth = hs.get_auth()
@@ -69,117 +70,90 @@ class RoomMemberHandler(object):
         self._enable_lookup = hs.config.enable_3pid_lookup
         self.allow_per_room_profiles = self.config.allow_per_room_profiles
 
-        self._event_stream_writer_instance = hs.config.worker.writers.events
-        self._is_on_event_persistence_instance = (
-            self._event_stream_writer_instance == hs.get_instance_name()
-        )
-        if self._is_on_event_persistence_instance:
-            self.persist_event_storage = hs.get_storage().persistence
-        else:
-            self._locally_reject_client = ReplicationLocallyRejectInviteRestServlet.make_client(
-                hs
-            )
-
         # This is only used to get at ratelimit function, and
         # maybe_kick_guest_users. It's fine there are multiple of these as
         # it doesn't store state.
         self.base_handler = BaseHandler(hs)
 
     @abc.abstractmethod
-    async def _remote_join(
-        self,
-        requester: Requester,
-        remote_room_hosts: List[str],
-        room_id: str,
-        user: UserID,
-        content: dict,
-    ) -> Tuple[str, int]:
+    def _remote_join(self, requester, remote_room_hosts, room_id, user, content):
         """Try and join a room that this server is not in
 
         Args:
-            requester
-            remote_room_hosts: List of servers that can be used to join via.
-            room_id: Room that we are trying to join
-            user: User who is trying to join
-            content: A dict that should be used as the content of the join event.
+            requester (Requester)
+            remote_room_hosts (list[str]): List of servers that can be used
+                to join via.
+            room_id (str): Room that we are trying to join
+            user (UserID): User who is trying to join
+            content (dict): A dict that should be used as the content of the
+                join event.
+
+        Returns:
+            Deferred
         """
         raise NotImplementedError()
 
     @abc.abstractmethod
-    async def _remote_reject_invite(
-        self,
-        requester: Requester,
-        remote_room_hosts: List[str],
-        room_id: str,
-        target: UserID,
-        content: dict,
-    ) -> Tuple[Optional[str], int]:
+    def _remote_reject_invite(
+        self, requester, remote_room_hosts, room_id, target, content
+    ):
         """Attempt to reject an invite for a room this server is not in. If we
         fail to do so we locally mark the invite as rejected.
 
         Args:
-            requester
-            remote_room_hosts: List of servers to use to try and reject invite
-            room_id
-            target: The user rejecting the invite
-            content: The content for the rejection event
+            requester (Requester)
+            remote_room_hosts (list[str]): List of servers to use to try and
+                reject invite
+            room_id (str)
+            target (UserID): The user rejecting the invite
+            content (dict): The content for the rejection event
 
         Returns:
-            A dictionary to be returned to the client, may
+            Deferred[dict]: A dictionary to be returned to the client, may
             include event_id etc, or nothing if we locally rejected
         """
         raise NotImplementedError()
 
-    async def locally_reject_invite(self, user_id: str, room_id: str) -> int:
-        """Mark the invite has having been rejected even though we failed to
-        create a leave event for it.
-        """
-        if self._is_on_event_persistence_instance:
-            return await self.persist_event_storage.locally_reject_invite(
-                user_id, room_id
-            )
-        else:
-            result = await self._locally_reject_client(
-                instance_name=self._event_stream_writer_instance,
-                user_id=user_id,
-                room_id=room_id,
-            )
-            return result["stream_id"]
-
     @abc.abstractmethod
-    async def _user_joined_room(self, target: UserID, room_id: str) -> None:
+    def _user_joined_room(self, target, room_id):
         """Notifies distributor on master process that the user has joined the
         room.
 
         Args:
-            target
-            room_id
+            target (UserID)
+            room_id (str)
+
+        Returns:
+            Deferred|None
         """
         raise NotImplementedError()
 
     @abc.abstractmethod
-    async def _user_left_room(self, target: UserID, room_id: str) -> None:
+    def _user_left_room(self, target, room_id):
         """Notifies distributor on master process that the user has left the
         room.
 
         Args:
-            target
-            room_id
+            target (UserID)
+            room_id (str)
+
+        Returns:
+            Deferred|None
         """
         raise NotImplementedError()
 
     async def _local_membership_update(
         self,
-        requester: Requester,
-        target: UserID,
-        room_id: str,
-        membership: str,
+        requester,
+        target,
+        room_id,
+        membership,
         prev_event_ids: Collection[str],
-        txn_id: Optional[str] = None,
-        ratelimit: bool = True,
-        content: Optional[dict] = None,
-        require_consent: bool = True,
-    ) -> Tuple[str, int]:
+        txn_id=None,
+        ratelimit=True,
+        content=None,
+        require_consent=True,
+    ):
         user_id = target.to_string()
 
         if content is None:
@@ -212,10 +186,9 @@ class RoomMemberHandler(object):
         )
         if duplicate is not None:
             # Discard the new event since this membership change is a no-op.
-            _, stream_id = await self.store.get_event_ordering(duplicate.event_id)
-            return duplicate.event_id, stream_id
+            return duplicate
 
-        stream_id = await self.event_creation_handler.handle_new_client_event(
+        await self.event_creation_handler.handle_new_client_event(
             requester, event, context, extra_users=[target], ratelimit=ratelimit
         )
 
@@ -239,20 +212,22 @@ class RoomMemberHandler(object):
                 if prev_member_event.membership == Membership.JOIN:
                     await self._user_left_room(target, room_id)
 
-        return event.event_id, stream_id
+        return event
 
-    async def copy_room_tags_and_direct_to_room(
-        self, old_room_id, new_room_id, user_id
-    ) -> None:
+    @defer.inlineCallbacks
+    def copy_room_tags_and_direct_to_room(self, old_room_id, new_room_id, user_id):
         """Copies the tags and direct room state from one room to another.
 
         Args:
-            old_room_id: The room ID of the old room.
-            new_room_id: The room ID of the new room.
-            user_id: The user's ID.
+            old_room_id (str)
+            new_room_id (str)
+            user_id (str)
+
+        Returns:
+            Deferred[None]
         """
         # Retrieve user account data for predecessor room
-        user_account_data, _ = await self.store.get_account_data_for_user(user_id)
+        user_account_data, _ = yield self.store.get_account_data_for_user(user_id)
 
         # Copy direct message state if applicable
         direct_rooms = user_account_data.get("m.direct", {})
@@ -265,31 +240,31 @@ class RoomMemberHandler(object):
                     direct_rooms[key].append(new_room_id)
 
                     # Save back to user's m.direct account data
-                    await self.store.add_account_data_for_user(
+                    yield self.store.add_account_data_for_user(
                         user_id, "m.direct", direct_rooms
                     )
                     break
 
         # Copy room tags if applicable
-        room_tags = await self.store.get_tags_for_room(user_id, old_room_id)
+        room_tags = yield self.store.get_tags_for_room(user_id, old_room_id)
 
         # Copy each room tag to the new room
         for tag, tag_content in room_tags.items():
-            await self.store.add_tag_to_room(user_id, new_room_id, tag, tag_content)
+            yield self.store.add_tag_to_room(user_id, new_room_id, tag, tag_content)
 
     async def update_membership(
         self,
-        requester: Requester,
-        target: UserID,
-        room_id: str,
-        action: str,
-        txn_id: Optional[str] = None,
-        remote_room_hosts: Optional[List[str]] = None,
-        third_party_signed: Optional[dict] = None,
-        ratelimit: bool = True,
-        content: Optional[dict] = None,
-        require_consent: bool = True,
-    ) -> Tuple[Optional[str], int]:
+        requester,
+        target,
+        room_id,
+        action,
+        txn_id=None,
+        remote_room_hosts=None,
+        third_party_signed=None,
+        ratelimit=True,
+        content=None,
+        require_consent=True,
+    ):
         key = (room_id,)
 
         with (await self.member_linearizer.queue(key)):
@@ -310,17 +285,17 @@ class RoomMemberHandler(object):
 
     async def _update_membership(
         self,
-        requester: Requester,
-        target: UserID,
-        room_id: str,
-        action: str,
-        txn_id: Optional[str] = None,
-        remote_room_hosts: Optional[List[str]] = None,
-        third_party_signed: Optional[dict] = None,
-        ratelimit: bool = True,
-        content: Optional[dict] = None,
-        require_consent: bool = True,
-    ) -> Tuple[Optional[str], int]:
+        requester,
+        target,
+        room_id,
+        action,
+        txn_id=None,
+        remote_room_hosts=None,
+        third_party_signed=None,
+        ratelimit=True,
+        content=None,
+        require_consent=True,
+    ):
         content_specified = bool(content)
         if content is None:
             content = {}
@@ -424,13 +399,7 @@ class RoomMemberHandler(object):
                 same_membership = old_membership == effective_membership_state
                 same_sender = requester.user.to_string() == old_state.sender
                 if same_sender and same_membership and same_content:
-                    _, stream_id = await self.store.get_event_ordering(
-                        old_state.event_id
-                    )
-                    return (
-                        old_state.event_id,
-                        stream_id,
-                    )
+                    return old_state
 
             if old_membership in ["ban", "leave"] and action == "kick":
                 raise AuthError(403, "The target user is not in the room")
@@ -500,11 +469,12 @@ class RoomMemberHandler(object):
                 else:
                     # send the rejection to the inviter's HS.
                     remote_room_hosts = remote_room_hosts + [inviter.domain]
-                    return await self._remote_reject_invite(
+                    res = await self._remote_reject_invite(
                         requester, remote_room_hosts, room_id, target, content,
                     )
+                    return res
 
-        return await self._local_membership_update(
+        res = await self._local_membership_update(
             requester=requester,
             target=target,
             room_id=room_id,
@@ -515,10 +485,10 @@ class RoomMemberHandler(object):
             content=content,
             require_consent=require_consent,
         )
+        return res
 
-    async def transfer_room_state_on_room_upgrade(
-        self, old_room_id: str, room_id: str
-    ) -> None:
+    @defer.inlineCallbacks
+    def transfer_room_state_on_room_upgrade(self, old_room_id, room_id):
         """Upon our server becoming aware of an upgraded room, either by upgrading a room
         ourselves or joining one, we can transfer over information from the previous room.
 
@@ -526,44 +496,50 @@ class RoomMemberHandler(object):
         well as migrating the room directory state.
 
         Args:
-            old_room_id: The ID of the old room
-            room_id: The ID of the new room
+            old_room_id (str): The ID of the old room
+
+            room_id (str): The ID of the new room
+
+        Returns:
+            Deferred
         """
         logger.info("Transferring room state from %s to %s", old_room_id, room_id)
 
         # Find all local users that were in the old room and copy over each user's state
-        users = await self.store.get_users_in_room(old_room_id)
-        await self.copy_user_state_on_room_upgrade(old_room_id, room_id, users)
+        users = yield self.store.get_users_in_room(old_room_id)
+        yield self.copy_user_state_on_room_upgrade(old_room_id, room_id, users)
 
         # Add new room to the room directory if the old room was there
         # Remove old room from the room directory
-        old_room = await self.store.get_room(old_room_id)
+        old_room = yield self.store.get_room(old_room_id)
         if old_room and old_room["is_public"]:
-            await self.store.set_room_is_public(old_room_id, False)
-            await self.store.set_room_is_public(room_id, True)
+            yield self.store.set_room_is_public(old_room_id, False)
+            yield self.store.set_room_is_public(room_id, True)
 
         # Transfer alias mappings in the room directory
-        await self.store.update_aliases_for_room(old_room_id, room_id)
+        yield self.store.update_aliases_for_room(old_room_id, room_id)
 
         # Check if any groups we own contain the predecessor room
-        local_group_ids = await self.store.get_local_groups_for_room(old_room_id)
+        local_group_ids = yield self.store.get_local_groups_for_room(old_room_id)
         for group_id in local_group_ids:
             # Add new the new room to those groups
-            await self.store.add_room_to_group(group_id, room_id, old_room["is_public"])
+            yield self.store.add_room_to_group(group_id, room_id, old_room["is_public"])
 
             # Remove the old room from those groups
-            await self.store.remove_room_from_group(group_id, old_room_id)
+            yield self.store.remove_room_from_group(group_id, old_room_id)
 
-    async def copy_user_state_on_room_upgrade(
-        self, old_room_id: str, new_room_id: str, user_ids: Iterable[str]
-    ) -> None:
+    @defer.inlineCallbacks
+    def copy_user_state_on_room_upgrade(self, old_room_id, new_room_id, user_ids):
         """Copy user-specific information when they join a new room when that new room is the
         result of a room upgrade
 
         Args:
-            old_room_id: The ID of upgraded room
-            new_room_id: The ID of the new room
-            user_ids: User IDs to copy state for
+            old_room_id (str): The ID of upgraded room
+            new_room_id (str): The ID of the new room
+            user_ids (Iterable[str]): User IDs to copy state for
+
+        Returns:
+            Deferred
         """
 
         logger.debug(
@@ -576,11 +552,11 @@ class RoomMemberHandler(object):
         for user_id in user_ids:
             try:
                 # It is an upgraded room. Copy over old tags
-                await self.copy_room_tags_and_direct_to_room(
+                yield self.copy_room_tags_and_direct_to_room(
                     old_room_id, new_room_id, user_id
                 )
                 # Copy over push rules
-                await self.store.copy_push_rules_from_room_to_room_for_user(
+                yield self.store.copy_push_rules_from_room_to_room_for_user(
                     old_room_id, new_room_id, user_id
                 )
             except Exception:
@@ -593,23 +569,17 @@ class RoomMemberHandler(object):
                 )
                 continue
 
-    async def send_membership_event(
-        self,
-        requester: Requester,
-        event: EventBase,
-        context: EventContext,
-        ratelimit: bool = True,
-    ):
+    async def send_membership_event(self, requester, event, context, ratelimit=True):
         """
         Change the membership status of a user in a room.
 
         Args:
-            requester: The local user who requested the membership
+            requester (Requester): The local user who requested the membership
                 event. If None, certain checks, like whether this homeserver can
                 act as the sender, will be skipped.
-            event: The membership event.
+            event (SynapseEvent): The membership event.
             context: The context of the event.
-            ratelimit: Whether to rate limit this request.
+            ratelimit (bool): Whether to rate limit this request.
         Raises:
             SynapseError if there was a problem changing the membership.
         """
@@ -669,9 +639,8 @@ class RoomMemberHandler(object):
                 if prev_member_event.membership == Membership.JOIN:
                     await self._user_left_room(target_user, room_id)
 
-    async def _can_guest_join(
-        self, current_state_ids: Dict[Tuple[str, str], str]
-    ) -> bool:
+    @defer.inlineCallbacks
+    def _can_guest_join(self, current_state_ids):
         """
         Returns whether a guest can join a room based on its current state.
         """
@@ -679,7 +648,7 @@ class RoomMemberHandler(object):
         if not guest_access_id:
             return False
 
-        guest_access = await self.store.get_event(guest_access_id)
+        guest_access = yield self.store.get_event(guest_access_id)
 
         return (
             guest_access
@@ -688,14 +657,13 @@ class RoomMemberHandler(object):
             and guest_access.content["guest_access"] == "can_join"
         )
 
-    async def lookup_room_alias(
-        self, room_alias: RoomAlias
-    ) -> Tuple[RoomID, List[str]]:
+    @defer.inlineCallbacks
+    def lookup_room_alias(self, room_alias):
         """
         Get the room ID associated with a room alias.
 
         Args:
-            room_alias: The alias to look up.
+            room_alias (RoomAlias): The alias to look up.
         Returns:
             A tuple of:
                 The room ID as a RoomID object.
@@ -704,7 +672,7 @@ class RoomMemberHandler(object):
             SynapseError if room alias could not be found.
         """
         directory_handler = self.directory_handler
-        mapping = await directory_handler.get_association(room_alias)
+        mapping = yield directory_handler.get_association(room_alias)
 
         if not mapping:
             raise SynapseError(404, "No such room alias")
@@ -719,25 +687,25 @@ class RoomMemberHandler(object):
 
         return RoomID.from_string(room_id), servers
 
-    async def _get_inviter(self, user_id: str, room_id: str) -> Optional[UserID]:
-        invite = await self.store.get_invite_for_local_user_in_room(
+    @defer.inlineCallbacks
+    def _get_inviter(self, user_id, room_id):
+        invite = yield self.store.get_invite_for_local_user_in_room(
             user_id=user_id, room_id=room_id
         )
         if invite:
             return UserID.from_string(invite.sender)
-        return None
 
     async def do_3pid_invite(
         self,
-        room_id: str,
-        inviter: UserID,
-        medium: str,
-        address: str,
-        id_server: str,
-        requester: Requester,
-        txn_id: Optional[str],
-        id_access_token: Optional[str] = None,
-    ) -> int:
+        room_id,
+        inviter,
+        medium,
+        address,
+        id_server,
+        requester,
+        txn_id,
+        id_access_token=None,
+    ):
         if self.config.block_non_admin_invites:
             is_requester_admin = await self.auth.is_server_admin(requester.user)
             if not is_requester_admin:
@@ -769,11 +737,11 @@ class RoomMemberHandler(object):
         )
 
         if invitee:
-            _, stream_id = await self.update_membership(
+            await self.update_membership(
                 requester, UserID.from_string(invitee), room_id, "invite", txn_id=txn_id
             )
         else:
-            stream_id = await self._make_and_store_3pid_invite(
+            await self._make_and_store_3pid_invite(
                 requester,
                 id_server,
                 medium,
@@ -784,19 +752,17 @@ class RoomMemberHandler(object):
                 id_access_token=id_access_token,
             )
 
-        return stream_id
-
     async def _make_and_store_3pid_invite(
         self,
-        requester: Requester,
-        id_server: str,
-        medium: str,
-        address: str,
-        room_id: str,
-        user: UserID,
-        txn_id: Optional[str],
-        id_access_token: Optional[str] = None,
-    ) -> int:
+        requester,
+        id_server,
+        medium,
+        address,
+        room_id,
+        user,
+        txn_id,
+        id_access_token=None,
+    ):
         room_state = await self.state_handler.get_current_state(room_id)
 
         inviter_display_name = ""
@@ -851,10 +817,7 @@ class RoomMemberHandler(object):
             id_access_token=id_access_token,
         )
 
-        (
-            event,
-            stream_id,
-        ) = await self.event_creation_handler.create_and_send_nonmember_event(
+        await self.event_creation_handler.create_and_send_nonmember_event(
             requester,
             {
                 "type": EventTypes.ThirdPartyInvite,
@@ -872,11 +835,9 @@ class RoomMemberHandler(object):
             ratelimit=False,
             txn_id=txn_id,
         )
-        return stream_id
 
-    async def _is_host_in_room(
-        self, current_state_ids: Dict[Tuple[str, str], str]
-    ) -> bool:
+    @defer.inlineCallbacks
+    def _is_host_in_room(self, current_state_ids):
         # Have we just created the room, and is this about to be the very
         # first member event?
         create_event_id = current_state_ids.get(("m.room.create", ""))
@@ -889,7 +850,7 @@ class RoomMemberHandler(object):
                 continue
 
             event_id = current_state_ids[(etype, state_key)]
-            event = await self.store.get_event(event_id, allow_none=True)
+            event = yield self.store.get_event(event_id, allow_none=True)
             if not event:
                 continue
 
@@ -898,10 +859,11 @@ class RoomMemberHandler(object):
 
         return False
 
-    async def _is_server_notice_room(self, room_id: str) -> bool:
+    @defer.inlineCallbacks
+    def _is_server_notice_room(self, room_id):
         if self._server_notices_mxid is None:
             return False
-        user_ids = await self.store.get_users_in_room(room_id)
+        user_ids = yield self.store.get_users_in_room(room_id)
         return self._server_notices_mxid in user_ids
 
 
@@ -913,15 +875,13 @@ class RoomMemberMasterHandler(RoomMemberHandler):
         self.distributor.declare("user_joined_room")
         self.distributor.declare("user_left_room")
 
-    async def _is_remote_room_too_complex(
-        self, room_id: str, remote_room_hosts: List[str]
-    ) -> Optional[bool]:
+    async def _is_remote_room_too_complex(self, room_id, remote_room_hosts):
         """
         Check if complexity of a remote room is too great.
 
         Args:
-            room_id
-            remote_room_hosts
+            room_id (str)
+            remote_room_hosts (list[str])
 
         Returns: bool of whether the complexity is too great, or None
             if unable to be fetched
@@ -935,26 +895,22 @@ class RoomMemberMasterHandler(RoomMemberHandler):
             return complexity["v1"] > max_complexity
         return None
 
-    async def _is_local_room_too_complex(self, room_id: str) -> bool:
+    @defer.inlineCallbacks
+    def _is_local_room_too_complex(self, room_id):
         """
         Check if the complexity of a local room is too great.
 
         Args:
-            room_id: The room ID to check for complexity.
+            room_id (str)
+
+        Returns: bool
         """
         max_complexity = self.hs.config.limit_remote_rooms.complexity
-        complexity = await self.store.get_room_complexity(room_id)
+        complexity = yield self.store.get_room_complexity(room_id)
 
         return complexity["v1"] > max_complexity
 
-    async def _remote_join(
-        self,
-        requester: Requester,
-        remote_room_hosts: List[str],
-        room_id: str,
-        user: UserID,
-        content: dict,
-    ) -> Tuple[str, int]:
+    async def _remote_join(self, requester, remote_room_hosts, room_id, user, content):
         """Implements RoomMemberHandler._remote_join
         """
         # filter ourselves out of remote_room_hosts: do_invite_join ignores it
@@ -983,7 +939,7 @@ class RoomMemberMasterHandler(RoomMemberHandler):
         # join dance for now, since we're kinda implicitly checking
         # that we are allowed to join when we decide whether or not we
         # need to do the invite/join dance.
-        event_id, stream_id = await self.federation_handler.do_invite_join(
+        await self.federation_handler.do_invite_join(
             remote_room_hosts, room_id, user.to_string(), content
         )
         await self._user_joined_room(user, room_id)
@@ -993,14 +949,14 @@ class RoomMemberMasterHandler(RoomMemberHandler):
         if self.hs.config.limit_remote_rooms.enabled:
             if too_complex is False:
                 # We checked, and we're under the limit.
-                return event_id, stream_id
+                return
 
             # Check again, but with the local state events
             too_complex = await self._is_local_room_too_complex(room_id)
 
             if too_complex is False:
                 # We're under the limit.
-                return event_id, stream_id
+                return
 
             # The room is too large. Leave.
             requester = types.create_requester(user, None, False, None)
@@ -1013,24 +969,20 @@ class RoomMemberMasterHandler(RoomMemberHandler):
                 errcode=Codes.RESOURCE_LIMIT_EXCEEDED,
             )
 
-        return event_id, stream_id
-
-    async def _remote_reject_invite(
-        self,
-        requester: Requester,
-        remote_room_hosts: List[str],
-        room_id: str,
-        target: UserID,
-        content: dict,
-    ) -> Tuple[Optional[str], int]:
+    @defer.inlineCallbacks
+    def _remote_reject_invite(
+        self, requester, remote_room_hosts, room_id, target, content
+    ):
         """Implements RoomMemberHandler._remote_reject_invite
         """
         fed_handler = self.federation_handler
         try:
-            event, stream_id = await fed_handler.do_remotely_reject_invite(
-                remote_room_hosts, room_id, target.to_string(), content=content,
+            ret = yield defer.ensureDeferred(
+                fed_handler.do_remotely_reject_invite(
+                    remote_room_hosts, room_id, target.to_string(), content=content,
+                )
             )
-            return event.event_id, stream_id
+            return ret
         except Exception as e:
             # if we were unable to reject the exception, just mark
             # it as rejected on our end and plough ahead.
@@ -1040,23 +992,24 @@ class RoomMemberMasterHandler(RoomMemberHandler):
             #
             logger.warning("Failed to reject invite: %s", e)
 
-            stream_id = await self.locally_reject_invite(target.to_string(), room_id)
-            return None, stream_id
+            yield self.store.locally_reject_invite(target.to_string(), room_id)
+            return {}
 
-    async def _user_joined_room(self, target: UserID, room_id: str) -> None:
+    def _user_joined_room(self, target, room_id):
         """Implements RoomMemberHandler._user_joined_room
         """
-        user_joined_room(self.distributor, target, room_id)
+        return defer.succeed(user_joined_room(self.distributor, target, room_id))
 
-    async def _user_left_room(self, target: UserID, room_id: str) -> None:
+    def _user_left_room(self, target, room_id):
         """Implements RoomMemberHandler._user_left_room
         """
-        user_left_room(self.distributor, target, room_id)
+        return defer.succeed(user_left_room(self.distributor, target, room_id))
 
-    async def forget(self, user: UserID, room_id: str) -> None:
+    @defer.inlineCallbacks
+    def forget(self, user, room_id):
         user_id = user.to_string()
 
-        member = await self.state_handler.get_current_state(
+        member = yield self.state_handler.get_current_state(
             room_id=room_id, event_type=EventTypes.Member, state_key=user_id
         )
         membership = member.membership if member else None
@@ -1068,4 +1021,4 @@ class RoomMemberMasterHandler(RoomMemberHandler):
             raise SynapseError(400, "User %s in room %s" % (user_id, room_id))
 
         if membership:
-            await self.store.forget(user_id, room_id)
+            yield self.store.forget(user_id, room_id)

synapse/handlers/room_member_worker.py

@@ -14,7 +14,8 @@
 # limitations under the License.
 
 import logging
-from typing import List, Optional, Tuple
+
+from twisted.internet import defer
 
 from synapse.api.errors import SynapseError
 from synapse.handlers.room_member import RoomMemberHandler
@@ -23,7 +24,6 @@ from synapse.replication.http.membership import (
     ReplicationRemoteRejectInviteRestServlet as ReplRejectInvite,
     ReplicationUserJoinedLeftRoomRestServlet as ReplJoinedLeft,
 )
-from synapse.types import Requester, UserID
 
 logger = logging.getLogger(__name__)
 
@@ -36,20 +36,14 @@ class RoomMemberWorkerHandler(RoomMemberHandler):
         self._remote_reject_client = ReplRejectInvite.make_client(hs)
         self._notify_change_client = ReplJoinedLeft.make_client(hs)
 
-    async def _remote_join(
-        self,
-        requester: Requester,
-        remote_room_hosts: List[str],
-        room_id: str,
-        user: UserID,
-        content: dict,
-    ) -> Tuple[str, int]:
+    @defer.inlineCallbacks
+    def _remote_join(self, requester, remote_room_hosts, room_id, user, content):
         """Implements RoomMemberHandler._remote_join
         """
         if len(remote_room_hosts) == 0:
             raise SynapseError(404, "No known servers")
 
-        ret = await self._remote_join_client(
+        ret = yield self._remote_join_client(
             requester=requester,
             remote_room_hosts=remote_room_hosts,
             room_id=room_id,
@@ -57,39 +51,33 @@ class RoomMemberWorkerHandler(RoomMemberHandler):
             content=content,
         )
 
-        await self._user_joined_room(user, room_id)
+        yield self._user_joined_room(user, room_id)
 
-        return ret["event_id"], ret["stream_id"]
+        return ret
 
-    async def _remote_reject_invite(
-        self,
-        requester: Requester,
-        remote_room_hosts: List[str],
-        room_id: str,
-        target: UserID,
-        content: dict,
-    ) -> Tuple[Optional[str], int]:
+    def _remote_reject_invite(
+        self, requester, remote_room_hosts, room_id, target, content
+    ):
         """Implements RoomMemberHandler._remote_reject_invite
         """
-        ret = await self._remote_reject_client(
+        return self._remote_reject_client(
             requester=requester,
             remote_room_hosts=remote_room_hosts,
             room_id=room_id,
             user_id=target.to_string(),
             content=content,
         )
-        return ret["event_id"], ret["stream_id"]
 
-    async def _user_joined_room(self, target: UserID, room_id: str) -> None:
+    def _user_joined_room(self, target, room_id):
         """Implements RoomMemberHandler._user_joined_room
         """
-        await self._notify_change_client(
+        return self._notify_change_client(
             user_id=target.to_string(), room_id=room_id, change="joined"
         )
 
-    async def _user_left_room(self, target: UserID, room_id: str) -> None:
+    def _user_left_room(self, target, room_id):
         """Implements RoomMemberHandler._user_left_room
         """
-        await self._notify_change_client(
+        return self._notify_change_client(
             user_id=target.to_string(), room_id=room_id, change="left"
         )

synapse/handlers/set_password.py

@@ -35,13 +35,16 @@ class SetPasswordHandler(BaseHandler):
     async def set_password(
         self,
         user_id: str,
-        password_hash: str,
+        new_password: str,
         logout_devices: bool,
         requester: Optional[Requester] = None,
     ):
         if not self.hs.config.password_localdb_enabled:
             raise SynapseError(403, "Password change disabled", errcode=Codes.FORBIDDEN)
 
+        self._password_policy_handler.validate_password(new_password)
+        password_hash = await self._auth_handler.hash(new_password)
+
         try:
             await self.store.user_set_password_hash(user_id, password_hash)
         except StoreError as e:

synapse/http/matrixfederationclient.py

@@ -19,7 +19,7 @@ import random
 import sys
 from io import BytesIO
 
-from six import raise_from, string_types
+from six import PY3, raise_from, string_types
 from six.moves import urllib
 
 import attr
@@ -70,7 +70,11 @@ incoming_responses_counter = Counter(
 
 MAX_LONG_RETRIES = 10
 MAX_SHORT_RETRIES = 3
-MAXINT = sys.maxsize
+
+if PY3:
+    MAXINT = sys.maxsize
+else:
+    MAXINT = sys.maxint
 
 
 _next_id = 1
@@ -144,11 +148,6 @@ def _handle_json_response(reactor, timeout_sec, request, response):
         d = timeout_deferred(d, timeout=timeout_sec, reactor=reactor)
 
         body = yield make_deferred_yieldable(d)
-    except TimeoutError as e:
-        logger.warning(
-            "{%s} [%s] Timed out reading response", request.txn_id, request.destination,
-        )
-        raise RequestSendFailed(e, can_retry=True) from e
     except Exception as e:
         logger.warning(
             "{%s} [%s] Error reading response: %s",
@@ -409,7 +408,7 @@ class MatrixFederationHttpClient(object):
                         _sec_timeout,
                     )
 
-                    outgoing_requests_counter.labels(request.method).inc()
+                    outgoing_requests_counter.labels(method_bytes).inc()
 
                     try:
                         with Measure(self.clock, "outbound_request"):
@@ -429,17 +428,13 @@ class MatrixFederationHttpClient(object):
                             )
 
                             response = yield request_deferred
-                    except TimeoutError as e:
-                        raise RequestSendFailed(e, can_retry=True) from e
                     except DNSLookupError as e:
                         raise_from(RequestSendFailed(e, can_retry=retry_on_dns_fail), e)
                     except Exception as e:
                         logger.info("Failed to send request: %s", e)
                         raise_from(RequestSendFailed(e, can_retry=True), e)
 
-                    incoming_responses_counter.labels(
-                        request.method, response.code
-                    ).inc()
+                    incoming_responses_counter.labels(method_bytes, response.code).inc()
 
                     set_tag(tags.HTTP_STATUS_CODE, response.code)
 

synapse/http/server.py

@@ -350,6 +350,9 @@ class JsonResource(HttpServer, resource.Resource):
                 register_paths, so will return (possibly via Deferred) either
                 None, or a tuple of (http code, response body).
         """
+        if request.method == b"OPTIONS":
+            return _options_handler, "options_request_handler", {}
+
         request_path = request.path.decode("ascii")
 
         # Loop through all the registered callbacks to check if the method
@@ -445,26 +448,6 @@ class RootRedirect(resource.Resource):
         return resource.Resource.getChild(self, name, request)
 
 
-class OptionsResource(resource.Resource):
-    """Responds to OPTION requests for itself and all children."""
-
-    def render_OPTIONS(self, request):
-        code, response_json_object = _options_handler(request)
-
-        return respond_with_json(
-            request, code, response_json_object, send_cors=True, canonical_json=False,
-        )
-
-    def getChildWithDefault(self, path, request):
-        if request.method == b"OPTIONS":
-            return self  # select ourselves as the child to render
-        return resource.Resource.getChildWithDefault(self, path, request)
-
-
-class RootOptionsRedirectResource(OptionsResource, RootRedirect):
-    pass
-
-
 def respond_with_json(
     request,
     code,

synapse/http/site.py

@@ -14,9 +14,7 @@
 import contextlib
 import logging
 import time
-from typing import Optional
 
-from twisted.python.failure import Failure
 from twisted.web.server import Request, Site
 
 from synapse.http import redact_uri
@@ -46,7 +44,7 @@ class SynapseRequest(Request):
     request even after the client has disconnected.
 
     Attributes:
-        logcontext: the log context for this request
+        logcontext(LoggingContext) : the log context for this request
     """
 
     def __init__(self, channel, *args, **kw):
@@ -54,10 +52,10 @@ class SynapseRequest(Request):
         self.site = channel.site
         self._channel = channel  # this is used by the tests
         self.authenticated_entity = None
-        self.start_time = 0.0
+        self.start_time = 0
 
         # we can't yet create the logcontext, as we don't know the method.
-        self.logcontext = None  # type: Optional[LoggingContext]
+        self.logcontext = None
 
         global _next_request_seq
         self.request_seq = _next_request_seq
@@ -183,7 +181,6 @@ class SynapseRequest(Request):
         self.finish_time = time.time()
         Request.finish(self)
         if not self._is_processing:
-            assert self.logcontext is not None
             with PreserveLoggingContext(self.logcontext):
                 self._finished_processing()
 
@@ -193,12 +190,6 @@ class SynapseRequest(Request):
         Overrides twisted.web.server.Request.connectionLost to record the finish time and
         do logging.
         """
-        # There is a bug in Twisted where reason is not wrapped in a Failure object
-        # Detect this and wrap it manually as a workaround
-        # More information: https://github.com/matrix-org/synapse/issues/7441
-        if not isinstance(reason, Failure):
-            reason = Failure(reason)
-
         self.finish_time = time.time()
         Request.connectionLost(self, reason)
 
@@ -251,7 +242,6 @@ class SynapseRequest(Request):
     def _finished_processing(self):
         """Log the completion of this request and update the metrics
         """
-        assert self.logcontext is not None
         usage = self.logcontext.get_resource_usage()
 
         if self._processing_finished_time is None:

synapse/logging/utils.py

@@ -20,6 +20,8 @@ import time
 from functools import wraps
 from inspect import getcallargs
 
+from six import PY3
+
 _TIME_FUNC_ID = 0
 
 
@@ -28,8 +30,12 @@ def _log_debug_as_f(f, msg, msg_args):
     logger = logging.getLogger(name)
 
     if logger.isEnabledFor(logging.DEBUG):
-        lineno = f.__code__.co_firstlineno
-        pathname = f.__code__.co_filename
+        if PY3:
+            lineno = f.__code__.co_firstlineno
+            pathname = f.__code__.co_filename
+        else:
+            lineno = f.func_code.co_firstlineno
+            pathname = f.func_code.co_filename
 
         record = logging.LogRecord(
             name=name,

synapse/metrics/__init__.py

@@ -26,12 +26,7 @@ import six
 
 import attr
 from prometheus_client import Counter, Gauge, Histogram
-from prometheus_client.core import (
-    REGISTRY,
-    CounterMetricFamily,
-    GaugeMetricFamily,
-    HistogramMetricFamily,
-)
+from prometheus_client.core import REGISTRY, GaugeMetricFamily, HistogramMetricFamily
 
 from twisted.internet import reactor
 
@@ -343,78 +338,6 @@ class GCCounts(object):
 if not running_on_pypy:
     REGISTRY.register(GCCounts())
 
-
-#
-# PyPy GC / memory metrics
-#
-
-
-class PyPyGCStats(object):
-    def collect(self):
-
-        # @stats is a pretty-printer object with __str__() returning a nice table,
-        # plus some fields that contain data from that table.
-        # unfortunately, fields are pretty-printed themselves (i. e. '4.5MB').
-        stats = gc.get_stats(memory_pressure=False)  # type: ignore
-        # @s contains same fields as @stats, but as actual integers.
-        s = stats._s  # type: ignore
-
-        # also note that field naming is completely braindead
-        # and only vaguely correlates with the pretty-printed table.
-        # >>>> gc.get_stats(False)
-        # Total memory consumed:
-        #     GC used:            8.7MB (peak: 39.0MB)        # s.total_gc_memory, s.peak_memory
-        #        in arenas:            3.0MB                  # s.total_arena_memory
-        #        rawmalloced:          1.7MB                  # s.total_rawmalloced_memory
-        #        nursery:              4.0MB                  # s.nursery_size
-        #     raw assembler used: 31.0kB                      # s.jit_backend_used
-        #     -----------------------------
-        #     Total:              8.8MB                       # stats.memory_used_sum
-        #
-        #     Total memory allocated:
-        #     GC allocated:            38.7MB (peak: 41.1MB)  # s.total_allocated_memory, s.peak_allocated_memory
-        #        in arenas:            30.9MB                 # s.peak_arena_memory
-        #        rawmalloced:          4.1MB                  # s.peak_rawmalloced_memory
-        #        nursery:              4.0MB                  # s.nursery_size
-        #     raw assembler allocated: 1.0MB                  # s.jit_backend_allocated
-        #     -----------------------------
-        #     Total:                   39.7MB                 # stats.memory_allocated_sum
-        #
-        #     Total time spent in GC:  0.073                  # s.total_gc_time
-
-        pypy_gc_time = CounterMetricFamily(
-            "pypy_gc_time_seconds_total", "Total time spent in PyPy GC", labels=[],
-        )
-        pypy_gc_time.add_metric([], s.total_gc_time / 1000)
-        yield pypy_gc_time
-
-        pypy_mem = GaugeMetricFamily(
-            "pypy_memory_bytes",
-            "Memory tracked by PyPy allocator",
-            labels=["state", "class", "kind"],
-        )
-        # memory used by JIT assembler
-        pypy_mem.add_metric(["used", "", "jit"], s.jit_backend_used)
-        pypy_mem.add_metric(["allocated", "", "jit"], s.jit_backend_allocated)
-        # memory used by GCed objects
-        pypy_mem.add_metric(["used", "", "arenas"], s.total_arena_memory)
-        pypy_mem.add_metric(["allocated", "", "arenas"], s.peak_arena_memory)
-        pypy_mem.add_metric(["used", "", "rawmalloced"], s.total_rawmalloced_memory)
-        pypy_mem.add_metric(["allocated", "", "rawmalloced"], s.peak_rawmalloced_memory)
-        pypy_mem.add_metric(["used", "", "nursery"], s.nursery_size)
-        pypy_mem.add_metric(["allocated", "", "nursery"], s.nursery_size)
-        # totals
-        pypy_mem.add_metric(["used", "totals", "gc"], s.total_gc_memory)
-        pypy_mem.add_metric(["allocated", "totals", "gc"], s.total_allocated_memory)
-        pypy_mem.add_metric(["used", "totals", "gc_peak"], s.peak_memory)
-        pypy_mem.add_metric(["allocated", "totals", "gc_peak"], s.peak_allocated_memory)
-        yield pypy_mem
-
-
-if running_on_pypy:
-    REGISTRY.register(PyPyGCStats())
-
-
 #
 # Twisted reactor metrics
 #

synapse/push/emailpusher.py

@@ -15,6 +15,7 @@
 
 import logging
 
+from twisted.internet import defer
 from twisted.internet.error import AlreadyCalled, AlreadyCancelled
 
 from synapse.metrics.background_process_metrics import run_as_background_process
@@ -131,7 +132,8 @@ class EmailPusher(object):
         self._is_processing = False
         self._start_processing()
 
-    async def _process(self):
+    @defer.inlineCallbacks
+    def _process(self):
         # we should never get here if we are already processing
         assert not self._is_processing
 
@@ -140,7 +142,7 @@ class EmailPusher(object):
 
             if self.throttle_params is None:
                 # this is our first loop: load up the throttle params
-                self.throttle_params = await self.store.get_throttle_params_by_room(
+                self.throttle_params = yield self.store.get_throttle_params_by_room(
                     self.pusher_id
                 )
 
@@ -149,7 +151,7 @@ class EmailPusher(object):
             while True:
                 starting_max_ordering = self.max_stream_ordering
                 try:
-                    await self._unsafe_process()
+                    yield self._unsafe_process()
                 except Exception:
                     logger.exception("Exception processing notifs")
                 if self.max_stream_ordering == starting_max_ordering:
@@ -157,7 +159,8 @@ class EmailPusher(object):
         finally:
             self._is_processing = False
 
-    async def _unsafe_process(self):
+    @defer.inlineCallbacks
+    def _unsafe_process(self):
         """
         Main logic of the push loop without the wrapper function that sets
         up logging, measures and guards against multiple instances of it
@@ -165,12 +168,12 @@ class EmailPusher(object):
         """
         start = 0 if INCLUDE_ALL_UNREAD_NOTIFS else self.last_stream_ordering
         fn = self.store.get_unread_push_actions_for_user_in_range_for_email
-        unprocessed = await fn(self.user_id, start, self.max_stream_ordering)
+        unprocessed = yield fn(self.user_id, start, self.max_stream_ordering)
 
         soonest_due_at = None
 
         if not unprocessed:
-            await self.save_last_stream_ordering_and_success(self.max_stream_ordering)
+            yield self.save_last_stream_ordering_and_success(self.max_stream_ordering)
             return
 
         for push_action in unprocessed:
@@ -198,15 +201,15 @@ class EmailPusher(object):
                     "throttle_ms": self.get_room_throttle_ms(push_action["room_id"]),
                 }
 
-                await self.send_notification(unprocessed, reason)
+                yield self.send_notification(unprocessed, reason)
 
-                await self.save_last_stream_ordering_and_success(
+                yield self.save_last_stream_ordering_and_success(
                     max(ea["stream_ordering"] for ea in unprocessed)
                 )
 
                 # we update the throttle on all the possible unprocessed push actions
                 for ea in unprocessed:
-                    await self.sent_notif_update_throttle(ea["room_id"], ea)
+                    yield self.sent_notif_update_throttle(ea["room_id"], ea)
                 break
             else:
                 if soonest_due_at is None or should_notify_at < soonest_due_at:
@@ -224,13 +227,14 @@ class EmailPusher(object):
                 self.seconds_until(soonest_due_at), self.on_timer
             )
 
-    async def save_last_stream_ordering_and_success(self, last_stream_ordering):
+    @defer.inlineCallbacks
+    def save_last_stream_ordering_and_success(self, last_stream_ordering):
         if last_stream_ordering is None:
             # This happens if we haven't yet processed anything
             return
 
         self.last_stream_ordering = last_stream_ordering
-        pusher_still_exists = await self.store.update_pusher_last_stream_ordering_and_success(
+        pusher_still_exists = yield self.store.update_pusher_last_stream_ordering_and_success(
             self.app_id,
             self.email,
             self.user_id,
@@ -271,12 +275,13 @@ class EmailPusher(object):
         may_send_at = last_sent_ts + throttle_ms
         return may_send_at
 
-    async def sent_notif_update_throttle(self, room_id, notified_push_action):
+    @defer.inlineCallbacks
+    def sent_notif_update_throttle(self, room_id, notified_push_action):
         # We have sent a notification, so update the throttle accordingly.
         # If the event that triggered the notif happened more than
         # THROTTLE_RESET_AFTER_MS after the previous one that triggered a
         # notif, we release the throttle. Otherwise, the throttle is increased.
-        time_of_previous_notifs = await self.store.get_time_of_last_push_action_before(
+        time_of_previous_notifs = yield self.store.get_time_of_last_push_action_before(
             notified_push_action["stream_ordering"]
         )
 
@@ -305,13 +310,14 @@ class EmailPusher(object):
             "last_sent_ts": self.clock.time_msec(),
             "throttle_ms": new_throttle_ms,
         }
-        await self.store.set_throttle_params(
+        yield self.store.set_throttle_params(
             self.pusher_id, room_id, self.throttle_params[room_id]
         )
 
-    async def send_notification(self, push_actions, reason):
+    @defer.inlineCallbacks
+    def send_notification(self, push_actions, reason):
         logger.info("Sending notif email for user %r", self.user_id)
 
-        await self.mailer.send_notification_mail(
+        yield self.mailer.send_notification_mail(
             self.app_id, self.user_id, self.email, push_actions, reason
         )

synapse/push/httppusher.py

@@ -15,6 +15,8 @@
 # limitations under the License.
 import logging
 
+import six
+
 from prometheus_client import Counter
 
 from twisted.internet import defer
@@ -26,6 +28,9 @@ from synapse.push import PusherConfigException
 
 from . import push_rule_evaluator, push_tools
 
+if six.PY3:
+    long = int
+
 logger = logging.getLogger(__name__)
 
 http_push_processed_counter = Counter(
@@ -313,7 +318,7 @@ class HttpPusher(object):
                         {
                             "app_id": self.app_id,
                             "pushkey": self.pushkey,
-                            "pushkey_ts": int(self.pushkey_ts / 1000),
+                            "pushkey_ts": long(self.pushkey_ts / 1000),
                             "data": self.data_minus_url,
                         }
                     ],
@@ -342,7 +347,7 @@ class HttpPusher(object):
                     {
                         "app_id": self.app_id,
                         "pushkey": self.pushkey,
-                        "pushkey_ts": int(self.pushkey_ts / 1000),
+                        "pushkey_ts": long(self.pushkey_ts / 1000),
                         "data": self.data_minus_url,
                         "tweaks": tweaks,
                     }
@@ -404,7 +409,7 @@ class HttpPusher(object):
                     {
                         "app_id": self.app_id,
                         "pushkey": self.pushkey,
-                        "pushkey_ts": int(self.pushkey_ts / 1000),
+                        "pushkey_ts": long(self.pushkey_ts / 1000),
                         "data": self.data_minus_url,
                     }
                 ],

synapse/push/mailer.py

@@ -26,6 +26,8 @@ from six.moves import urllib
 import bleach
 import jinja2
 
+from twisted.internet import defer
+
 from synapse.api.constants import EventTypes
 from synapse.api.errors import StoreError
 from synapse.logging.context import make_deferred_yieldable
@@ -125,7 +127,8 @@ class Mailer(object):
 
         logger.info("Created Mailer for app_name %s" % app_name)
 
-    async def send_password_reset_mail(self, email_address, token, client_secret, sid):
+    @defer.inlineCallbacks
+    def send_password_reset_mail(self, email_address, token, client_secret, sid):
         """Send an email with a password reset link to a user
 
         Args:
@@ -146,13 +149,14 @@ class Mailer(object):
 
         template_vars = {"link": link}
 
-        await self.send_email(
+        yield self.send_email(
             email_address,
             "[%s] Password Reset" % self.hs.config.server_name,
             template_vars,
         )
 
-    async def send_registration_mail(self, email_address, token, client_secret, sid):
+    @defer.inlineCallbacks
+    def send_registration_mail(self, email_address, token, client_secret, sid):
         """Send an email with a registration confirmation link to a user
 
         Args:
@@ -173,13 +177,14 @@ class Mailer(object):
 
         template_vars = {"link": link}
 
-        await self.send_email(
+        yield self.send_email(
             email_address,
             "[%s] Register your Email Address" % self.hs.config.server_name,
             template_vars,
         )
 
-    async def send_add_threepid_mail(self, email_address, token, client_secret, sid):
+    @defer.inlineCallbacks
+    def send_add_threepid_mail(self, email_address, token, client_secret, sid):
         """Send an email with a validation link to a user for adding a 3pid to their account
 
         Args:
@@ -201,19 +206,20 @@ class Mailer(object):
 
         template_vars = {"link": link}
 
-        await self.send_email(
+        yield self.send_email(
             email_address,
             "[%s] Validate Your Email" % self.hs.config.server_name,
             template_vars,
         )
 
-    async def send_notification_mail(
+    @defer.inlineCallbacks
+    def send_notification_mail(
         self, app_id, user_id, email_address, push_actions, reason
     ):
         """Send email regarding a user's room notifications"""
         rooms_in_order = deduped_ordered_list([pa["room_id"] for pa in push_actions])
 
-        notif_events = await self.store.get_events(
+        notif_events = yield self.store.get_events(
             [pa["event_id"] for pa in push_actions]
         )
 
@@ -226,7 +232,7 @@ class Mailer(object):
         state_by_room = {}
 
         try:
-            user_display_name = await self.store.get_profile_displayname(
+            user_display_name = yield self.store.get_profile_displayname(
                 UserID.from_string(user_id).localpart
             )
             if user_display_name is None:
@@ -234,13 +240,14 @@ class Mailer(object):
         except StoreError:
             user_display_name = user_id
 
-        async def _fetch_room_state(room_id):
-            room_state = await self.store.get_current_state_ids(room_id)
+        @defer.inlineCallbacks
+        def _fetch_room_state(room_id):
+            room_state = yield self.store.get_current_state_ids(room_id)
             state_by_room[room_id] = room_state
 
         # Run at most 3 of these at once: sync does 10 at a time but email
         # notifs are much less realtime than sync so we can afford to wait a bit.
-        await concurrently_execute(_fetch_room_state, rooms_in_order, 3)
+        yield concurrently_execute(_fetch_room_state, rooms_in_order, 3)
 
         # actually sort our so-called rooms_in_order list, most recent room first
         rooms_in_order.sort(key=lambda r: -(notifs_by_room[r][-1]["received_ts"] or 0))
@@ -248,19 +255,19 @@ class Mailer(object):
         rooms = []
 
         for r in rooms_in_order:
-            roomvars = await self.get_room_vars(
+            roomvars = yield self.get_room_vars(
                 r, user_id, notifs_by_room[r], notif_events, state_by_room[r]
             )
             rooms.append(roomvars)
 
-        reason["room_name"] = await calculate_room_name(
+        reason["room_name"] = yield calculate_room_name(
             self.store,
             state_by_room[reason["room_id"]],
             user_id,
             fallback_to_members=True,
         )
 
-        summary_text = await self.make_summary_text(
+        summary_text = yield self.make_summary_text(
             notifs_by_room, state_by_room, notif_events, user_id, reason
         )
 
@@ -275,11 +282,12 @@ class Mailer(object):
             "reason": reason,
         }
 
-        await self.send_email(
+        yield self.send_email(
             email_address, "[%s] %s" % (self.app_name, summary_text), template_vars
         )
 
-    async def send_email(self, email_address, subject, template_vars):
+    @defer.inlineCallbacks
+    def send_email(self, email_address, subject, template_vars):
         """Send an email with the given information and template text"""
         try:
             from_string = self.hs.config.email_notif_from % {"app": self.app_name}
@@ -309,7 +317,7 @@ class Mailer(object):
 
         logger.info("Sending email to %s" % email_address)
 
-        await make_deferred_yieldable(
+        yield make_deferred_yieldable(
             self.sendmail(
                 self.hs.config.email_smtp_host,
                 raw_from,
@@ -324,14 +332,13 @@ class Mailer(object):
             )
         )
 
-    async def get_room_vars(
-        self, room_id, user_id, notifs, notif_events, room_state_ids
-    ):
+    @defer.inlineCallbacks
+    def get_room_vars(self, room_id, user_id, notifs, notif_events, room_state_ids):
         my_member_event_id = room_state_ids[("m.room.member", user_id)]
-        my_member_event = await self.store.get_event(my_member_event_id)
+        my_member_event = yield self.store.get_event(my_member_event_id)
         is_invite = my_member_event.content["membership"] == "invite"
 
-        room_name = await calculate_room_name(self.store, room_state_ids, user_id)
+        room_name = yield calculate_room_name(self.store, room_state_ids, user_id)
 
         room_vars = {
             "title": room_name,
@@ -343,7 +350,7 @@ class Mailer(object):
 
         if not is_invite:
             for n in notifs:
-                notifvars = await self.get_notif_vars(
+                notifvars = yield self.get_notif_vars(
                     n, user_id, notif_events[n["event_id"]], room_state_ids
                 )
 
@@ -370,8 +377,9 @@ class Mailer(object):
 
         return room_vars
 
-    async def get_notif_vars(self, notif, user_id, notif_event, room_state_ids):
-        results = await self.store.get_events_around(
+    @defer.inlineCallbacks
+    def get_notif_vars(self, notif, user_id, notif_event, room_state_ids):
+        results = yield self.store.get_events_around(
             notif["room_id"],
             notif["event_id"],
             before_limit=CONTEXT_BEFORE,
@@ -384,24 +392,25 @@ class Mailer(object):
             "messages": [],
         }
 
-        the_events = await filter_events_for_client(
+        the_events = yield filter_events_for_client(
             self.storage, user_id, results["events_before"]
         )
         the_events.append(notif_event)
 
         for event in the_events:
-            messagevars = await self.get_message_vars(notif, event, room_state_ids)
+            messagevars = yield self.get_message_vars(notif, event, room_state_ids)
             if messagevars is not None:
                 ret["messages"].append(messagevars)
 
         return ret
 
-    async def get_message_vars(self, notif, event, room_state_ids):
+    @defer.inlineCallbacks
+    def get_message_vars(self, notif, event, room_state_ids):
         if event.type != EventTypes.Message:
             return
 
         sender_state_event_id = room_state_ids[("m.room.member", event.sender)]
-        sender_state_event = await self.store.get_event(sender_state_event_id)
+        sender_state_event = yield self.store.get_event(sender_state_event_id)
         sender_name = name_from_member_event(sender_state_event)
         sender_avatar_url = sender_state_event.content.get("avatar_url")
 
@@ -451,7 +460,8 @@ class Mailer(object):
 
         return messagevars
 
-    async def make_summary_text(
+    @defer.inlineCallbacks
+    def make_summary_text(
         self, notifs_by_room, room_state_ids, notif_events, user_id, reason
     ):
         if len(notifs_by_room) == 1:
@@ -461,17 +471,17 @@ class Mailer(object):
             # If the room has some kind of name, use it, but we don't
             # want the generated-from-names one here otherwise we'll
             # end up with, "new message from Bob in the Bob room"
-            room_name = await calculate_room_name(
+            room_name = yield calculate_room_name(
                 self.store, room_state_ids[room_id], user_id, fallback_to_members=False
             )
 
             my_member_event_id = room_state_ids[room_id][("m.room.member", user_id)]
-            my_member_event = await self.store.get_event(my_member_event_id)
+            my_member_event = yield self.store.get_event(my_member_event_id)
             if my_member_event.content["membership"] == "invite":
                 inviter_member_event_id = room_state_ids[room_id][
                     ("m.room.member", my_member_event.sender)
                 ]
-                inviter_member_event = await self.store.get_event(
+                inviter_member_event = yield self.store.get_event(
                     inviter_member_event_id
                 )
                 inviter_name = name_from_member_event(inviter_member_event)
@@ -496,7 +506,7 @@ class Mailer(object):
                     state_event_id = room_state_ids[room_id][
                         ("m.room.member", event.sender)
                     ]
-                    state_event = await self.store.get_event(state_event_id)
+                    state_event = yield self.store.get_event(state_event_id)
                     sender_name = name_from_member_event(state_event)
 
                 if sender_name is not None and room_name is not None:
@@ -525,7 +535,7 @@ class Mailer(object):
                         }
                     )
 
-                    member_events = await self.store.get_events(
+                    member_events = yield self.store.get_events(
                         [
                             room_state_ids[room_id][("m.room.member", s)]
                             for s in sender_ids
@@ -557,7 +567,7 @@ class Mailer(object):
                     }
                 )
 
-                member_events = await self.store.get_events(
+                member_events = yield self.store.get_events(
                     [room_state_ids[room_id][("m.room.member", s)] for s in sender_ids]
                 )
 

synapse/replication/http/__init__.py

@@ -19,7 +19,6 @@ from synapse.replication.http import (
     federation,
     login,
     membership,
-    presence,
     register,
     send_event,
     streams,
@@ -36,11 +35,9 @@ class ReplicationRestResource(JsonResource):
     def register_servlets(self, hs):
         send_event.register_servlets(hs, self)
         federation.register_servlets(hs, self)
-        presence.register_servlets(hs, self)
-        membership.register_servlets(hs, self)
 
-        # The following can't currently be instantiated on workers.
         if hs.config.worker.worker_app is None:
+            membership.register_servlets(hs, self)
             login.register_servlets(hs, self)
             register.register_servlets(hs, self)
             devices.register_servlets(hs, self)

synapse/replication/http/federation.py

@@ -29,7 +29,7 @@ logger = logging.getLogger(__name__)
 
 class ReplicationFederationSendEventsRestServlet(ReplicationEndpoint):
     """Handles events newly received from federation, including persisting and
-    notifying. Returns the maximum stream ID of the persisted events.
+    notifying.
 
     The API looks like:
 
@@ -46,13 +46,6 @@ class ReplicationFederationSendEventsRestServlet(ReplicationEndpoint):
                 "context": { .. serialized event context .. },
             }],
             "backfilled": false
-        }
-
-        200 OK
-
-        {
-            "max_stream_id": 32443,
-        }
     """
 
     NAME = "fed_send_events"
@@ -122,11 +115,11 @@ class ReplicationFederationSendEventsRestServlet(ReplicationEndpoint):
 
         logger.info("Got %d events from federation", len(event_and_contexts))
 
-        max_stream_id = await self.federation_handler.persist_events_and_notify(
+        await self.federation_handler.persist_events_and_notify(
             event_and_contexts, backfilled
         )
 
-        return 200, {"max_stream_id": max_stream_id}
+        return 200, {}
 
 
 class ReplicationFederationSendEduRestServlet(ReplicationEndpoint):

synapse/replication/http/membership.py

@@ -14,16 +14,12 @@
 # limitations under the License.
 
 import logging
-from typing import TYPE_CHECKING
 
 from synapse.http.servlet import parse_json_object_from_request
 from synapse.replication.http._base import ReplicationEndpoint
 from synapse.types import Requester, UserID
 from synapse.util.distributor import user_joined_room, user_left_room
 
-if TYPE_CHECKING:
-    from synapse.server import HomeServer
-
 logger = logging.getLogger(__name__)
 
 
@@ -80,11 +76,11 @@ class ReplicationRemoteJoinRestServlet(ReplicationEndpoint):
 
         logger.info("remote_join: %s into room: %s", user_id, room_id)
 
-        event_id, stream_id = await self.federation_handler.do_invite_join(
+        await self.federation_handler.do_invite_join(
             remote_room_hosts, room_id, user_id, event_content
         )
 
-        return 200, {"event_id": event_id, "stream_id": stream_id}
+        return 200, {}
 
 
 class ReplicationRemoteRejectInviteRestServlet(ReplicationEndpoint):
@@ -110,7 +106,6 @@ class ReplicationRemoteRejectInviteRestServlet(ReplicationEndpoint):
         self.federation_handler = hs.get_handlers().federation_handler
         self.store = hs.get_datastore()
         self.clock = hs.get_clock()
-        self.member_handler = hs.get_room_member_handler()
 
     @staticmethod
     def _serialize_payload(requester, room_id, user_id, remote_room_hosts, content):
@@ -141,10 +136,10 @@ class ReplicationRemoteRejectInviteRestServlet(ReplicationEndpoint):
         logger.info("remote_reject_invite: %s out of room: %s", user_id, room_id)
 
         try:
-            event, stream_id = await self.federation_handler.do_remotely_reject_invite(
+            event = await self.federation_handler.do_remotely_reject_invite(
                 remote_room_hosts, room_id, user_id, event_content,
             )
-            event_id = event.event_id
+            ret = event.get_pdu_json()
         except Exception as e:
             # if we were unable to reject the exception, just mark
             # it as rejected on our end and plough ahead.
@@ -154,42 +149,10 @@ class ReplicationRemoteRejectInviteRestServlet(ReplicationEndpoint):
             #
             logger.warning("Failed to reject invite: %s", e)
 
-            stream_id = await self.member_handler.locally_reject_invite(
-                user_id, room_id
-            )
-            event_id = None
+            await self.store.locally_reject_invite(user_id, room_id)
+            ret = {}
 
-        return 200, {"event_id": event_id, "stream_id": stream_id}
-
-
-class ReplicationLocallyRejectInviteRestServlet(ReplicationEndpoint):
-    """Rejects the invite for the user and room locally.
-
-    Request format:
-
-        POST /_synapse/replication/locally_reject_invite/:room_id/:user_id
-
-        {}
-    """
-
-    NAME = "locally_reject_invite"
-    PATH_ARGS = ("room_id", "user_id")
-
-    def __init__(self, hs: "HomeServer"):
-        super().__init__(hs)
-
-        self.member_handler = hs.get_room_member_handler()
-
-    @staticmethod
-    def _serialize_payload(room_id, user_id):
-        return {}
-
-    async def _handle_request(self, request, room_id, user_id):
-        logger.info("locally_reject_invite: %s out of room: %s", user_id, room_id)
-
-        stream_id = await self.member_handler.locally_reject_invite(user_id, room_id)
-
-        return 200, {"stream_id": stream_id}
+        return 200, ret
 
 
 class ReplicationUserJoinedLeftRoomRestServlet(ReplicationEndpoint):
@@ -245,4 +208,3 @@ def register_servlets(hs, http_server):
     ReplicationRemoteJoinRestServlet(hs).register(http_server)
     ReplicationRemoteRejectInviteRestServlet(hs).register(http_server)
     ReplicationUserJoinedLeftRoomRestServlet(hs).register(http_server)
-    ReplicationLocallyRejectInviteRestServlet(hs).register(http_server)

synapse/replication/http/presence.py

@@ -1,116 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2020 The Matrix.org Foundation C.I.C.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import logging
-from typing import TYPE_CHECKING
-
-from synapse.http.servlet import parse_json_object_from_request
-from synapse.replication.http._base import ReplicationEndpoint
-from synapse.types import UserID
-
-if TYPE_CHECKING:
-    from synapse.server import HomeServer
-
-logger = logging.getLogger(__name__)
-
-
-class ReplicationBumpPresenceActiveTime(ReplicationEndpoint):
-    """We've seen the user do something that indicates they're interacting
-    with the app.
-
-    The POST looks like:
-
-        POST /_synapse/replication/bump_presence_active_time/<user_id>
-
-        200 OK
-
-        {}
-    """
-
-    NAME = "bump_presence_active_time"
-    PATH_ARGS = ("user_id",)
-    METHOD = "POST"
-    CACHE = False
-
-    def __init__(self, hs: "HomeServer"):
-        super().__init__(hs)
-
-        self._presence_handler = hs.get_presence_handler()
-
-    @staticmethod
-    def _serialize_payload(user_id):
-        return {}
-
-    async def _handle_request(self, request, user_id):
-        await self._presence_handler.bump_presence_active_time(
-            UserID.from_string(user_id)
-        )
-
-        return (
-            200,
-            {},
-        )
-
-
-class ReplicationPresenceSetState(ReplicationEndpoint):
-    """Set the presence state for a user.
-
-    The POST looks like:
-
-        POST /_synapse/replication/presence_set_state/<user_id>
-
-        {
-            "state": { ... },
-            "ignore_status_msg": false,
-        }
-
-        200 OK
-
-        {}
-    """
-
-    NAME = "presence_set_state"
-    PATH_ARGS = ("user_id",)
-    METHOD = "POST"
-    CACHE = False
-
-    def __init__(self, hs: "HomeServer"):
-        super().__init__(hs)
-
-        self._presence_handler = hs.get_presence_handler()
-
-    @staticmethod
-    def _serialize_payload(user_id, state, ignore_status_msg=False):
-        return {
-            "state": state,
-            "ignore_status_msg": ignore_status_msg,
-        }
-
-    async def _handle_request(self, request, user_id):
-        content = parse_json_object_from_request(request)
-
-        await self._presence_handler.set_state(
-            UserID.from_string(user_id), content["state"], content["ignore_status_msg"]
-        )
-
-        return (
-            200,
-            {},
-        )
-
-
-def register_servlets(hs, http_server):
-    ReplicationBumpPresenceActiveTime(hs).register(http_server)
-    ReplicationPresenceSetState(hs).register(http_server)

synapse/replication/http/send_event.py

@@ -119,11 +119,11 @@ class ReplicationSendEventRestServlet(ReplicationEndpoint):
             "Got event to send with ID: %s into room: %s", event.event_id, event.room_id
         )
 
-        stream_id = await self.event_creation_handler.persist_and_notify_client_event(
+        await self.event_creation_handler.persist_and_notify_client_event(
             requester, event, context, ratelimit=ratelimit, extra_users=extra_users
         )
 
-        return 200, {"stream_id": stream_id}
+        return 200, {}
 
 
 def register_servlets(hs, http_server):

synapse/replication/http/streams.py

@@ -51,7 +51,10 @@ class ReplicationGetStreamUpdates(ReplicationEndpoint):
         super().__init__(hs)
 
         self._instance_name = hs.get_instance_name()
-        self.streams = hs.get_replication_streams()
+
+        # We pull the streams from the replication handler (if we try and make
+        # them ourselves we end up in an import loop).
+        self.streams = hs.get_tcp_replication().get_streams()
 
     @staticmethod
     def _serialize_payload(stream_name, from_token, upto_token):

synapse/replication/slave/storage/_base.py

@@ -16,6 +16,8 @@
 import logging
 from typing import Optional
 
+import six
+
 from synapse.storage.data_stores.main.cache import CacheInvalidationWorkerStore
 from synapse.storage.database import Database
 from synapse.storage.engines import PostgresEngine
@@ -24,6 +26,13 @@ from synapse.storage.util.id_generators import MultiWriterIdGenerator
 logger = logging.getLogger(__name__)
 
 
+def __func__(inp):
+    if six.PY3:
+        return inp
+    else:
+        return inp.__func__
+
+
 class BaseSlavedStore(CacheInvalidationWorkerStore):
     def __init__(self, database: Database, db_conn, hs):
         super(BaseSlavedStore, self).__init__(database, db_conn, hs)

synapse/replication/slave/storage/presence.py

@@ -18,7 +18,7 @@ from synapse.storage.data_stores.main.presence import PresenceStore
 from synapse.storage.database import Database
 from synapse.util.caches.stream_change_cache import StreamChangeCache
 
-from ._base import BaseSlavedStore
+from ._base import BaseSlavedStore, __func__
 from ._slaved_id_tracker import SlavedIdTracker
 
 
@@ -27,14 +27,14 @@ class SlavedPresenceStore(BaseSlavedStore):
         super(SlavedPresenceStore, self).__init__(database, db_conn, hs)
         self._presence_id_gen = SlavedIdTracker(db_conn, "presence_stream", "stream_id")
 
-        self._presence_on_startup = self._get_active_presence(db_conn)  # type: ignore
+        self._presence_on_startup = self._get_active_presence(db_conn)
 
         self.presence_stream_cache = StreamChangeCache(
             "PresenceStreamChangeCache", self._presence_id_gen.get_current_token()
         )
 
-    _get_active_presence = DataStore._get_active_presence
-    take_presence_startup_info = DataStore.take_presence_startup_info
+    _get_active_presence = __func__(DataStore._get_active_presence)
+    take_presence_startup_info = __func__(DataStore.take_presence_startup_info)
     _get_presence_for_user = PresenceStore.__dict__["_get_presence_for_user"]
     get_presence_for_users = PresenceStore.__dict__["get_presence_for_users"]
 

synapse/replication/tcp/client.py

@@ -14,23 +14,19 @@
 # limitations under the License.
 """A replication client for use by synapse workers.
 """
-import heapq
-import logging
-from typing import TYPE_CHECKING, Dict, List, Tuple
 
-from twisted.internet.defer import Deferred
+import logging
+from typing import TYPE_CHECKING, Tuple
+
 from twisted.internet.protocol import ReconnectingClientFactory
 
 from synapse.api.constants import EventTypes
-from synapse.logging.context import PreserveLoggingContext, make_deferred_yieldable
 from synapse.replication.tcp.protocol import ClientReplicationStreamProtocol
 from synapse.replication.tcp.streams.events import (
     EventsStream,
     EventsStreamEventRow,
     EventsStreamRow,
 )
-from synapse.util.async_helpers import timeout_deferred
-from synapse.util.metrics import Measure
 
 if TYPE_CHECKING:
     from synapse.server import HomeServer
@@ -39,10 +35,6 @@ if TYPE_CHECKING:
 logger = logging.getLogger(__name__)
 
 
-# How long we allow callers to wait for replication updates before timing out.
-_WAIT_FOR_REPLICATION_TIMEOUT_SECONDS = 30
-
-
 class DirectTcpReplicationClientFactory(ReconnectingClientFactory):
     """Factory for building connections to the master. Will reconnect if the
     connection is lost.
@@ -100,16 +92,6 @@ class ReplicationDataHandler:
         self.store = hs.get_datastore()
         self.pusher_pool = hs.get_pusherpool()
         self.notifier = hs.get_notifier()
-        self._reactor = hs.get_reactor()
-        self._clock = hs.get_clock()
-        self._streams = hs.get_replication_streams()
-        self._instance_name = hs.get_instance_name()
-
-        # Map from stream to list of deferreds waiting for the stream to
-        # arrive at a particular position. The lists are sorted by stream position.
-        self._streams_to_waiters = (
-            {}
-        )  # type: Dict[str, List[Tuple[int, Deferred[None]]]]
 
     async def on_rdata(
         self, stream_name: str, instance_name: str, token: int, rows: list
@@ -149,76 +131,8 @@ class ReplicationDataHandler:
 
             await self.pusher_pool.on_new_notifications(token, token)
 
-        # Notify any waiting deferreds. The list is ordered by position so we
-        # just iterate through the list until we reach a position that is
-        # greater than the received row position.
-        waiting_list = self._streams_to_waiters.get(stream_name, [])
-
-        # Index of first item with a position after the current token, i.e we
-        # have called all deferreds before this index. If not overwritten by
-        # loop below means either a) no items in list so no-op or b) all items
-        # in list were called and so the list should be cleared. Setting it to
-        # `len(list)` works for both cases.
-        index_of_first_deferred_not_called = len(waiting_list)
-
-        for idx, (position, deferred) in enumerate(waiting_list):
-            if position <= token:
-                try:
-                    with PreserveLoggingContext():
-                        deferred.callback(None)
-                except Exception:
-                    # The deferred has been cancelled or timed out.
-                    pass
-            else:
-                # The list is sorted by position so we don't need to continue
-                # checking any futher entries in the list.
-                index_of_first_deferred_not_called = idx
-                break
-
-        # Drop all entries in the waiting list that were called in the above
-        # loop. (This maintains the order so no need to resort)
-        waiting_list[:] = waiting_list[index_of_first_deferred_not_called:]
-
     async def on_position(self, stream_name: str, instance_name: str, token: int):
         self.store.process_replication_rows(stream_name, instance_name, token, [])
 
     def on_remote_server_up(self, server: str):
         """Called when get a new REMOTE_SERVER_UP command."""
-
-    async def wait_for_stream_position(
-        self, instance_name: str, stream_name: str, position: int
-    ):
-        """Wait until this instance has received updates up to and including
-        the given stream position.
-        """
-
-        if instance_name == self._instance_name:
-            # We don't get told about updates written by this process, and
-            # anyway in that case we don't need to wait.
-            return
-
-        current_position = self._streams[stream_name].current_token(self._instance_name)
-        if position <= current_position:
-            # We're already past the position
-            return
-
-        # Create a new deferred that times out after N seconds, as we don't want
-        # to wedge here forever.
-        deferred = Deferred()
-        deferred = timeout_deferred(
-            deferred, _WAIT_FOR_REPLICATION_TIMEOUT_SECONDS, self._reactor
-        )
-
-        waiting_list = self._streams_to_waiters.setdefault(stream_name, [])
-
-        # We insert into the list using heapq as it is more efficient than
-        # pushing then resorting each time.
-        heapq.heappush(waiting_list, (position, deferred))
-
-        # We measure here to get in flight counts and average waiting time.
-        with Measure(self._clock, "repl.wait_for_stream_position"):
-            logger.info("Waiting for repl stream %r to reach %s", stream_name, position)
-            await make_deferred_yieldable(deferred)
-            logger.info(
-                "Finished waiting for repl stream %r to reach %s", stream_name, position
-            )

synapse/replication/tcp/handler.py

@@ -89,13 +89,11 @@ class ReplicationCommandHandler:
                 self._streams_to_replicate.append(stream)
                 continue
 
-            if isinstance(stream, (EventsStream, BackfillStream)):
-                # Only add EventStream and BackfillStream as a source on the
-                # instance in charge of event persistence.
-                if hs.config.worker.writers.events == hs.get_instance_name():
-                    self._streams_to_replicate.append(stream)
-
-                continue
+            if (
+                isinstance(stream, (EventsStream, BackfillStream))
+                and hs.config.worker.writers.events == hs.get_instance_name()
+            ):
+                self._streams_to_replicate.append(stream)
 
             # Only add any other streams if we're on master.
             if hs.config.worker_app is not None:
@@ -159,9 +157,6 @@ class ReplicationCommandHandler:
                 hs.config.redis_port,
             )
 
-            # First let's ensure that we have a ReplicationStreamer started.
-            hs.get_replication_streamer()
-
             # We need two connections to redis, one for the subscription stream and
             # one to send commands to (as you can't send further redis commands to a
             # connection after SUBSCRIBE is called).

synapse/replication/tcp/streams/_base.py

@@ -14,27 +14,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import heapq
 import logging
 from collections import namedtuple
-from typing import (
-    TYPE_CHECKING,
-    Any,
-    Awaitable,
-    Callable,
-    List,
-    Optional,
-    Tuple,
-    TypeVar,
-)
+from typing import Any, Awaitable, Callable, List, Optional, Tuple
 
 import attr
 
 from synapse.replication.http.streams import ReplicationGetStreamUpdates
 
-if TYPE_CHECKING:
-    import synapse.server
-
 logger = logging.getLogger(__name__)
 
 # the number of rows to request from an update_function.
@@ -50,7 +37,7 @@ Token = int
 # parsing with Stream.parse_row (which turns it into a `ROW_TYPE`). Normally it's
 # just a row from a database query, though this is dependent on the stream in question.
 #
-StreamRow = TypeVar("StreamRow", bound=Tuple)
+StreamRow = Tuple
 
 # The type returned by the update_function of a stream, as well as get_updates(),
 # get_updates_since, etc.
@@ -546,63 +533,32 @@ class AccountDataStream(Stream):
     """
 
     AccountDataStreamRow = namedtuple(
-        "AccountDataStream",
-        ("user_id", "room_id", "data_type"),  # str  # Optional[str]  # str
+        "AccountDataStream", ("user_id", "room_id", "data_type")  # str  # str  # str
     )
 
     NAME = "account_data"
     ROW_TYPE = AccountDataStreamRow
 
-    def __init__(self, hs: "synapse.server.HomeServer"):
+    def __init__(self, hs):
         self.store = hs.get_datastore()
         super().__init__(
             hs.get_instance_name(),
             current_token_without_instance(self.store.get_max_account_data_stream_id),
-            self._update_function,
+            db_query_to_update_function(self._update_function),
         )
 
-    async def _update_function(
-        self, instance_name: str, from_token: int, to_token: int, limit: int
-    ) -> StreamUpdateResult:
-        limited = False
-        global_results = await self.store.get_updated_global_account_data(
-            from_token, to_token, limit
+    async def _update_function(self, from_token, to_token, limit):
+        global_results, room_results = await self.store.get_all_updated_account_data(
+            from_token, from_token, to_token, limit
         )
 
-        # if the global results hit the limit, we'll need to limit the room results to
-        # the same stream token.
-        if len(global_results) >= limit:
-            to_token = global_results[-1][0]
-            limited = True
-
-        room_results = await self.store.get_updated_room_account_data(
-            from_token, to_token, limit
-        )
-
-        # likewise, if the room results hit the limit, limit the global results to
-        # the same stream token.
-        if len(room_results) >= limit:
-            to_token = room_results[-1][0]
-            limited = True
-
-        # convert the global results to the right format, and limit them to the to_token
-        # at the same time
-        global_rows = (
-            (stream_id, (user_id, None, account_data_type))
+        results = list(room_results)
+        results.extend(
+            (stream_id, user_id, None, account_data_type)
             for stream_id, user_id, account_data_type in global_results
-            if stream_id <= to_token
         )
 
-        # we know that the room_results are already limited to `to_token` so no need
-        # for a check on `stream_id` here.
-        room_rows = (
-            (stream_id, (user_id, room_id, account_data_type))
-            for stream_id, user_id, room_id, account_data_type in room_results
-        )
-
-        # we need to return a sorted list, so merge them together.
-        updates = list(heapq.merge(room_rows, global_rows))
-        return updates, to_token, limited
+        return results
 
 
 class GroupServerStream(Stream):