Avoid doing presence updates on replication reconnect

Presence is supposed to be disabled on matrix.org, so we shouldn't send a load of USER_SYNC commands every time the synchrotron reconnects to the master.
Disable auth on room_members for now
2018-02-15 09:51:09 +00:00 · 2018-02-15 09:51:09 +00:00 · 2018-02-15 09:51:09 +00:00 · 2018-02-15 09:51:09 +00:00 · 2018-02-15 09:51:09 +00:00 · 2018-02-15 09:51:09 +00:00
240 changed files with 4281 additions and 8170 deletions
@@ -1,5 +0,0 @@
-Dockerfile
-.travis.yml
-.gitignore
-demo/etc
-tox.ini
@@ -32,7 +32,6 @@ demo/media_store.*
 demo/etc

 uploads
-cache

 .idea/
 media_store/
@@ -1,22 +1,14 @@
 sudo: false
 language: python
+python: 2.7

 # tell travis to cache ~/.cache/pip
 cache: pip

-matrix:
-  include:
-  - python: 2.7
-    env: TOX_ENV=packaging
-
-  - python: 2.7
-    env: TOX_ENV=pep8
-
-  - python: 2.7
-    env: TOX_ENV=py27
-    
-  - python: 3.6
-    env: TOX_ENV=py36
+env:
+  - TOX_ENV=packaging
+  - TOX_ENV=pep8
+  - TOX_ENV=py27

 install:
  - pip install tox
@@ -60,6 +60,3 @@ Niklas Riekenbrauck <nikriek at gmail dot.com>

 Christoph Witzany <christoph at web.crofting.com>
 * Add LDAP support for authentication
-
-Pierre Jaury <pierre at jaury.eu>
-* Docker packaging
@@ -1,319 +1,11 @@
-Changes in synapse v0.29.1 (2018-05-17)
-==========================================
-Changes:
+Unreleased
+==========

-* Update docker documentation (PR #3222)
-
-Changes in synapse v0.29.0 (2018-05-16)
-===========================================
-Not changes since v0.29.0-rc1
-
-Changes in synapse v0.29.0-rc1 (2018-05-14)
-===========================================
-
-Notable changes, a docker file for running Synapse (Thanks to @kaiyou!) and a
-closed spec bug in the Client Server API. Additionally further prep for Python 3
-migration.
-
-Potentially breaking change:
-
-* Make Client-Server API return 401 for invalid token (PR #3161).
-
-  This changes the Client-server spec to return a 401 error code instead of 403
-  when the access token is unrecognised. This is the behaviour required by the
-  specification, but some clients may be relying on the old, incorrect
-  behaviour.
-
-  Thanks to @NotAFile for fixing this.
-
-Features:
-
-* Add a Dockerfile for synapse (PR #2846) Thanks to @kaiyou!
-
-Changes - General:
-
-* nuke-room-from-db.sh: added postgresql option and help (PR #2337) Thanks to @rubo77!
-* Part user from rooms on account deactivate (PR #3201)
-* Make 'unexpected logging context' into warnings (PR #3007)
-* Set Server header in SynapseRequest (PR #3208)
-* remove duplicates from groups tables (PR #3129)
-* Improve exception handling for background processes (PR #3138)
-* Add missing consumeErrors to improve exception handling (PR #3139)
-* reraise exceptions more carefully (PR #3142)
-* Remove redundant call to preserve_fn (PR #3143)
-* Trap exceptions thrown within run_in_background (PR #3144)
-
-Changes - Refactors:
-
-* Refactor /context to reuse pagination storage functions (PR #3193)
-* Refactor recent events func to use pagination func (PR #3195)
-* Refactor pagination DB API to return concrete type (PR #3196)
-* Refactor get_recent_events_for_room return type (PR #3198)
-* Refactor sync APIs to reuse pagination API (PR #3199)
-* Remove unused code path from member change DB func (PR #3200)
-* Refactor request handling wrappers (PR #3203)
-* transaction_id, destination defined twice (PR #3209) Thanks to @damir-manapov!
-* Refactor event storage to prepare for changes in state calculations (PR #3141)
-* Set Server header in SynapseRequest (PR #3208)
-* Use deferred.addTimeout instead of time_bound_deferred (PR #3127, #3178)
-* Use run_in_background in preference to preserve_fn (PR #3140)
-
-Changes - Python 3 migration:
-
-* Construct HMAC as bytes on py3 (PR #3156) Thanks to @NotAFile!
-* run config tests on py3 (PR #3159) Thanks to @NotAFile!
-* Open certificate files as bytes (PR #3084) Thanks to @NotAFile!
-* Open config file in non-bytes mode (PR #3085) Thanks to @NotAFile!
-* Make event properties raise AttributeError instead (PR #3102) Thanks to @NotAFile!
-* Use six.moves.urlparse (PR #3108) Thanks to @NotAFile!
-* Add py3 tests to tox with folders that work (PR #3145) Thanks to @NotAFile!
-* Don't yield in list comprehensions (PR #3150) Thanks to @NotAFile!
-* Move more xrange to six (PR #3151) Thanks to @NotAFile!
-* make imports local (PR #3152) Thanks to @NotAFile!
-* move httplib import to six (PR #3153) Thanks to @NotAFile!
-* Replace stringIO imports with six (PR #3154, #3168) Thanks to @NotAFile!
-* more bytes strings (PR #3155) Thanks to @NotAFile!
-
-Bug Fixes:
-
-* synapse fails to start under Twisted >= 18.4 (PR #3157)
-* Fix a class of logcontext leaks (PR #3170)
-* Fix a couple of logcontext leaks in unit tests (PR #3172)
-* Fix logcontext leak in media repo (PR #3174)
-* Escape label values in prometheus metrics (PR #3175, #3186)
-* Fix 'Unhandled Error' logs with Twisted 18.4 (PR #3182) Thanks to @Half-Shot!
-* Fix logcontext leaks in rate limiter (PR #3183)
-* notifications: Convert next_token to string according to the spec (PR #3190) Thanks to @mujx!
-* nuke-room-from-db.sh: fix deletion from search table (PR #3194) Thanks to @rubo77!
-* add guard for None on purge_history api (PR #3160) Thanks to @krombel!
-
-Changes in synapse v0.28.1 (2018-05-01)
-=======================================
-
-SECURITY UPDATE
-
-* Clamp the allowed values of event depth received over federation to be
-  [0, 2^63 - 1].  This mitigates an attack where malicious events
-  injected with depth = 2^63 - 1 render rooms unusable.  Depth is used to
-  determine the cosmetic ordering of events within a room, and so the ordering
-  of events in such a room will default to using stream_ordering rather than depth
-  (topological_ordering).
-
-  This is a temporary solution to mitigate abuse in the wild, whilst a long term solution
-  is being implemented to improve how the depth parameter is used.
-
-  Full details at
-  https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI
-
-* Pin Twisted to <18.4 until we stop using the private _OpenSSLECCurve API.
-
-
-Changes in synapse v0.28.0 (2018-04-26)
-=======================================
-
-Bug Fixes:
-
-* Fix quarantine media admin API and search reindex (PR #3130)
-* Fix media admin APIs (PR #3134)
-
-
-Changes in synapse v0.28.0-rc1 (2018-04-24)
-===========================================
-
-Minor performance improvement to federation sending and bug fixes.
-
-(Note: This release does not include the delta state resolution implementation discussed in matrix live)
-
-
-Features:
-
-* Add metrics for event processing lag (PR #3090)
-* Add metrics for ResponseCache (PR #3092)
-
-Changes:
-
-* Synapse on PyPy (PR #2760) Thanks to @Valodim!
-* move handling of auto_join_rooms to RegisterHandler (PR #2996) Thanks to @krombel!
-* Improve handling of SRV records for federation connections (PR #3016) Thanks to @silkeh!
-* Document the behaviour of ResponseCache (PR #3059)
-* Preparation for py3 (PR #3061, #3073, #3074, #3075, #3103, #3104, #3106, #3107, #3109, #3110) Thanks to @NotAFile!
-* update prometheus dashboard to use new metric names (PR #3069) Thanks to @krombel!
-* use python3-compatible prints (PR #3074) Thanks to @NotAFile!
-* Send federation events concurrently (PR #3078)
-* Limit concurrent event sends for a room (PR #3079)
-* Improve R30 stat definition (PR #3086)
-* Send events to ASes concurrently (PR #3088)
-* Refactor ResponseCache usage (PR #3093)
-* Clarify that SRV may not point to a CNAME (PR #3100) Thanks to @silkeh!
-* Use str(e) instead of e.message (PR #3103) Thanks to @NotAFile!
-* Use six.itervalues in some places (PR #3106) Thanks to @NotAFile!
-* Refactor store.have_events (PR #3117)
-
-Bug Fixes:
-
-* Return 401 for invalid access_token on logout (PR #2938) Thanks to @dklug!
-* Return a 404 rather than a 500 on rejoining empty rooms (PR #3080)
-* fix federation_domain_whitelist (PR #3099)
-* Avoid creating events with huge numbers of prev_events (PR #3113)
-* Reject events which have lots of prev_events (PR #3118)
-
-
-Changes in synapse v0.27.4 (2018-04-13)
-======================================
-
-Changes:
-
-* Update canonicaljson dependency (#3095)
-
-
-Changes in synapse v0.27.3 (2018-04-11)
-======================================
-
-Bug fixes:
-
-* URL quote path segments over federation (#3082)
-
-Changes in synapse v0.27.3-rc2 (2018-04-09)
-==========================================
-
-v0.27.3-rc1 used a stale version of the develop branch so the changelog overstates
-the functionality. v0.27.3-rc2 is up to date, rc1 should be ignored.
-
-Changes in synapse v0.27.3-rc1 (2018-04-09)
-=======================================
-
-Notable changes include API support for joinability of groups. Also new metrics
-and phone home stats. Phone home stats include better visibility of system usage
-so we can tweak synpase to work better for all users rather than our own experience
-with matrix.org. Also, recording 'r30' stat which is the measure we use to track
-overal growth of the Matrix ecosystem. It is defined as:-
-
-Counts the number of native 30 day retained users, defined as:-
-         * Users who have created their accounts more than 30 days
-         * Where last seen at most 30 days ago
-         * Where account creation and last_seen are > 30 days"
-
-
-Features:
-
-* Add joinability for groups (PR #3045)
-* Implement group join API (PR #3046)
-* Add counter metrics for calculating state delta (PR #3033)
-* R30 stats (PR #3041)
-* Measure time it takes to calculate state group ID (PR #3043)
-* Add basic performance statistics to phone home (PR #3044)
-* Add response size metrics (PR #3071)
-* phone home cache size configurations (PR #3063)
-
-Changes:
-
-* Add a blurb explaining the main synapse worker (PR #2886) Thanks to @turt2live!
-* Replace old style error catching with 'as' keyword (PR #3000) Thanks to @NotAFile!
-* Use .iter* to avoid copies in StateHandler (PR #3006)
-* Linearize calls to _generate_user_id (PR #3029)
-* Remove last usage of ujson (PR #3030)
-* Use simplejson throughout (PR #3048)
-* Use static JSONEncoders (PR #3049)
-* Remove uses of events.content (PR #3060)
-* Improve database cache performance (PR #3068)
-
-Bug fixes:
-
-* Add room_id to the response of `rooms/{roomId}/join` (PR #2986) Thanks to @jplatte!
-* Fix replication after switch to simplejson (PR #3015)
-* 404 correctly on missing paths via NoResource (PR #3022)
-* Fix error when claiming e2e keys from offline servers (PR #3034)
-* fix tests/storage/test_user_directory.py (PR #3042)
-* use PUT instead of POST for federating groups/m.join_policy (PR #3070) Thanks to @krombel!
-* postgres port script: fix state_groups_pkey error (PR #3072)
-
-
-Changes in synapse v0.27.2 (2018-03-26)
-=======================================
-
-Bug fixes:
-
-* Fix bug which broke TCP replication between workers (PR #3015)
-
-
-Changes in synapse v0.27.1 (2018-03-26)
-=======================================
-
-Meta release as v0.27.0 temporarily pointed to the wrong commit
-
-
-Changes in synapse v0.27.0 (2018-03-26)
-=======================================
-
-No changes since v0.27.0-rc2
-
-
-Changes in synapse v0.27.0-rc2 (2018-03-19)
-===========================================
-
-Pulls in v0.26.1
-
-Bug fixes:
-
-* Fix bug introduced in v0.27.0-rc1 that causes much increased memory usage in state cache (PR #3005)
-
-
-Changes in synapse v0.26.1 (2018-03-15)
-=======================================
-
-Bug fixes:
-
-* Fix bug where an invalid event caused server to stop functioning correctly,
-  due to parsing and serializing bugs in ujson library (PR #3008)
-
-
-Changes in synapse v0.27.0-rc1 (2018-03-14)
-===========================================
-
-The common case for running Synapse is not to run separate workers, but for those that do, be aware that synctl no longer starts the main synapse when using ``-a`` option with workers. A new worker file should be added with ``worker_app: synapse.app.homeserver``.
+synctl no longer starts the main synapse when using ``-a`` option with workers.
+A new worker file should be added with ``worker_app: synapse.app.homeserver``.

 This release also begins the process of renaming a number of the metrics
 reported to prometheus. See `docs/metrics-howto.rst <docs/metrics-howto.rst#block-and-response-metrics-renamed-for-0-27-0>`_.
-Note that the v0.28.0 release will remove the deprecated metric names.
-
-Features:
-
-* Add ability for ASes to override message send time (PR #2754)
-* Add support for custom storage providers for media repository (PR #2867, #2777, #2783, #2789, #2791, #2804, #2812, #2814, #2857, #2868, #2767)
-* Add purge API features, see `docs/admin_api/purge_history_api.rst <docs/admin_api/purge_history_api.rst>`_ for full details (PR #2858, #2867, #2882, #2946, #2962, #2943)
-* Add support for whitelisting 3PIDs that users can register. (PR #2813)
-* Add ``/room/{id}/event/{id}`` API (PR #2766)
-* Add an admin API to get all the media in a room (PR #2818) Thanks to @turt2live!
-* Add ``federation_domain_whitelist`` option (PR #2820, #2821)
-
-
-Changes:
-
-* Continue to factor out processing from main process and into worker processes. See updated `docs/workers.rst <docs/workers.rst>`_ (PR #2892 - #2904, #2913, #2920 - #2926, #2947, #2847, #2854, #2872, #2873, #2874, #2928, #2929, #2934, #2856, #2976 - #2984, #2987 - #2989, #2991 - #2993, #2995, #2784)
-* Ensure state cache is used when persisting events (PR #2864, #2871, #2802, #2835, #2836, #2841, #2842, #2849)
-* Change the default config to bind on both IPv4 and IPv6 on all platforms (PR #2435) Thanks to @silkeh!
-* No longer require a specific version of saml2 (PR #2695) Thanks to @okurz!
-* Remove ``verbosity``/``log_file`` from generated config (PR #2755)
-* Add and improve metrics and logging (PR #2770, #2778, #2785, #2786, #2787, #2793, #2794, #2795, #2809, #2810, #2833, #2834, #2844, #2965, #2927, #2975, #2790, #2796, #2838)
-* When using synctl with workers, don't start the main synapse automatically (PR #2774)
-* Minor performance improvements (PR #2773, #2792)
-* Use a connection pool for non-federation outbound connections (PR #2817)
-* Make it possible to run unit tests against postgres (PR #2829)
-* Update pynacl dependency to 1.2.1 or higher (PR #2888) Thanks to @bachp!
-* Remove ability for AS users to call /events and /sync (PR #2948)
-* Use bcrypt.checkpw (PR #2949) Thanks to @krombel!
-
-Bug fixes:
-
-* Fix broken ``ldap_config`` config option (PR #2683) Thanks to @seckrv!
-* Fix error message when user is not allowed to unban (PR #2761) Thanks to @turt2live!
-* Fix publicised groups GET API (singular) over federation (PR #2772)
-* Fix user directory when using ``user_directory_search_all_users`` config option (PR #2803, #2831)
-* Fix error on ``/publicRooms`` when no rooms exist (PR #2827)
-* Fix bug in quarantine_media (PR #2837)
-* Fix url_previews when no Content-Type is returned from URL (PR #2845)
-* Fix rare race in sync API when joining room (PR #2944)
-* Fix slow event search, switch back from GIST to GIN indexes (PR #2769, #2848)


 Changes in synapse v0.26.0 (2018-01-05)
@@ -30,12 +30,8 @@ use github's pull request workflow to review the contribution, and either ask
 you to make any refinements needed or merge it and make them ourselves. The
 changes will then land on master when we next do a release.

-We use `Jenkins <http://matrix.org/jenkins>`_ and 
-`Travis <https://travis-ci.org/matrix-org/synapse>`_ for continuous
-integration. All pull requests to synapse get automatically tested by Travis; 
-the Jenkins builds require an adminstrator to start them. If your change 
-breaks the build, this will be shown in github, so please keep an eye on the 
-pull request for feedback.
+We use Jenkins for continuous integration (http://matrix.org/jenkins), and
+typically all pull requests get automatically tested Jenkins: if your change breaks the build, Jenkins will yell about it in #matrix-dev:matrix.org so please lurk there and keep an eye open.

 Code style
 ~~~~~~~~~~
@@ -119,4 +115,4 @@ can't be accepted. Git makes this trivial - just use the -s flag when you do
 Conclusion
 ~~~~~~~~~~

-That's it!  Matrix is a very open and collaborative project as you might expect given our obsession with open communication.  If we're going to successfully matrix together all the fragmented communication technologies out there we are reliant on contributions and collaboration from the community to do so.  So please get involved - and we hope you have as much fun hacking on Matrix as we do!
+That's it!  Matrix is a very open and collaborative project as you might expect given our obsession with open communication.  If we're going to successfully matrix together all the fragmented communication technologies out there we are reliant on contributions and collaboration from the community to do so.  So please get involved - and we hope you have as much fun hacking on Matrix as we do!
@@ -1,19 +0,0 @@
-FROM docker.io/python:2-alpine3.7
-
-RUN apk add --no-cache --virtual .nacl_deps su-exec build-base libffi-dev zlib-dev libressl-dev libjpeg-turbo-dev linux-headers postgresql-dev
-
-COPY . /synapse
-
-# A wheel cache may be provided in ./cache for faster build
-RUN cd /synapse \
- && pip install --upgrade pip setuptools psycopg2 \
- && mkdir -p /synapse/cache \
- && pip install -f /synapse/cache --upgrade --process-dependency-links . \
- && mv /synapse/contrib/docker/start.py /synapse/contrib/docker/conf / \
- && rm -rf setup.py setup.cfg synapse
-
-VOLUME ["/data"]
-
-EXPOSE 8008/tcp 8448/tcp
-
-ENTRYPOINT ["/start.py"]
@@ -25,8 +25,6 @@ recursive-include synapse/static *.js
 exclude jenkins.sh
 exclude jenkins*.sh
 exclude jenkins*
-exclude Dockerfile
-exclude .dockerignore
 recursive-exclude jenkins *.sh

 prune .github
@@ -157,9 +157,8 @@ if you prefer.

 In case of problems, please see the _`Troubleshooting` section below.

-There is an offical synapse image available at https://hub.docker.com/r/matrixdotorg/synapse/tags/ which can be used with the docker-compose file available at `contrib/docker`. Further information on this including configuration options is available in `contrib/docker/README.md`.
-
-Alternatively, Andreas Peters (previously Silvio Fricke) has contributed a Dockerfile to automate a synapse server in a single Docker image, at https://hub.docker.com/r/avhost/docker-matrix/tags/
+Alternatively, Silvio Fricke has contributed a Dockerfile to automate the
+above in Docker at https://registry.hub.docker.com/u/silviof/docker-matrix/.

 Also, Martin Giess has created an auto-deployment process with vagrant/ansible,
 tested with VirtualBox/AWS/DigitalOcean - see https://github.com/EMnify/matrix-synapse-auto-deploy
@@ -355,10 +354,6 @@ https://matrix.org/docs/projects/try-matrix-now.html (or build your own with one
 Fedora
 ------

-Synapse is in the Fedora repositories as ``matrix-synapse``::
-
-    sudo dnf install matrix-synapse
-
 Oleg Girko provides Fedora RPMs at
 https://obs.infoserver.lv/project/monitor/matrix-synapse

@@ -615,9 +610,6 @@ should have the format ``_matrix._tcp.<yourdomain.com> <ttl> IN SRV 10 0 <port>
    $ dig -t srv _matrix._tcp.example.com
    _matrix._tcp.example.com. 3600    IN      SRV     10 0 8448 synapse.example.com.

-Note that the server hostname cannot be an alias (CNAME record): it has to point
-directly to the server hosting the synapse instance.
-
 You can then configure your homeserver to use ``<yourdomain.com>`` as the domain in
 its user-ids, by setting ``server_name``::

@@ -898,17 +890,6 @@ This should end with a 'PASSED' result::

    PASSED (successes=143)

-Running the Integration Tests
-=============================
-
-Synapse is accompanied by `SyTest <https://github.com/matrix-org/sytest>`_,
-a Matrix homeserver integration testing suite, which uses HTTP requests to
-access the API as a Matrix client would. It is able to run Synapse directly from
-the source tree, so installation of the server is not required.
-
-Testing with SyTest is recommended for verifying that changes related to the
-Client-Server API are functioning correctly. See the `installation instructions
-<https://github.com/matrix-org/sytest#installing>`_ for details.

 Building Internal API Documentation
 ===================================
@@ -48,18 +48,6 @@ returned by the Client-Server API:
    # configured on port 443.
    curl -kv https://<host.name>/_matrix/client/versions 2>&1 | grep "Server:"

-Upgrading to $NEXT_VERSION
-====================
-
-This release expands the anonymous usage stats sent if the opt-in
-``report_stats`` configuration is set to ``true``. We now capture RSS memory 
-and cpu use at a very coarse level. This requires administrators to install
-the optional ``psutil`` python module.
-
-We would appreciate it if you could assist by ensuring this module is available
-and ``report_stats`` is enabled. This will let us see if performance changes to
-synapse are having an impact to the general community.
-
 Upgrading to v0.15.0
 ====================

@@ -1,10 +0,0 @@
-Community Contributions
-=======================
-
-Everything in this directory are projects submitted by the community that may be useful
-to others. As such, the project maintainers cannot guarantee support, stability
-or backwards compatibility of these projects. 
-
-Files in this directory should *not* be relied on directly, as they may not
-continue to work or exist in future. If you wish to use any of these files then
-they should be copied to avoid them breaking from underneath you.
@@ -1,153 +0,0 @@
-# Synapse Docker
-
-The `matrixdotorg/synapse` Docker image will run Synapse as a single process. It does not provide a
-database server or a TURN server, you should run these separately.
-
-If you run a Postgres server, you should simply include it in the same Compose
-project or set the proper environment variables and the image will automatically
-use that server.
-
-## Build
-
-Build the docker image with the `docker build` command from the root of the synapse repository.
-
-```
-docker build -t docker.io/matrixdotorg/synapse .
-```
-
-The `-t` option sets the image tag. Official images are tagged `matrixdotorg/synapse:<version>` where `<version>` is the same as the release tag in the synapse git repository.
-
-You may have a local Python wheel cache available, in which case copy the relevant packages in the ``cache/`` directory at the root of the project.
-
-## Run
-
-This image is designed to run either with an automatically generated configuration
-file or with a custom configuration that requires manual edition.
-
-### Automated configuration
-
-It is recommended that you use Docker Compose to run your containers, including
-this image and a Postgres server. A sample ``docker-compose.yml`` is provided,
-including example labels for reverse proxying and other artifacts.
-
-Read the section about environment variables and set at least mandatory variables,
-then run the server:
-
-```
-docker-compose up -d
-```
-
-If secrets are not specified in the environment variables, they will be generated
-as part of the startup. Please ensure these secrets are kept between launches of the
-Docker container, as their loss may require users to log in again.
-
-### Manual configuration
-
-A sample ``docker-compose.yml`` is provided, including example labels for
-reverse proxying and other artifacts. The docker-compose file is an example,
-please comment/uncomment sections that are not suitable for your usecase.
-
-Specify a ``SYNAPSE_CONFIG_PATH``, preferably to a persistent path,
-to use manual configuration. To generate a fresh ``homeserver.yaml``, simply run:
-
-```
-docker-compose run --rm -e SYNAPSE_SERVER_NAME=my.matrix.host synapse generate
-```
-
-Then, customize your configuration and run the server:
-
-```
-docker-compose up -d
-```
-
-### Without Compose
-
-If you do not wish to use Compose, you may still run this image using plain
-Docker commands. Note that the following is just a guideline and you may need
-to add parameters to the docker run command to account for the network situation
-with your postgres database.
-
-```
-docker run \
-    -d \
-    --name synapse \
-    -v ${DATA_PATH}:/data \
-    -e SYNAPSE_SERVER_NAME=my.matrix.host \
-    -e SYNAPSE_REPORT_STATS=yes \
-    docker.io/matrixdotorg/synapse:latest
-```
-
-## Volumes
-
-The image expects a single volume, located at ``/data``, that will hold:
-
-* temporary files during uploads;
-* uploaded media and thumbnails;
-* the SQLite database if you do not configure postgres;
-* the appservices configuration.
-
-You are free to use separate volumes depending on storage endpoints at your
-disposal. For instance, ``/data/media`` coud be stored on a large but low
-performance hdd storage while other files could be stored on high performance
-endpoints.
-
-In order to setup an application service, simply create an ``appservices``
-directory in the data volume and write the application service Yaml
-configuration file there. Multiple application services are supported.
-
-## Environment
-
-Unless you specify a custom path for the configuration file, a very generic
-file will be generated, based on the following environment settings.
-These are a good starting point for setting up your own deployment.
-
-Global settings:
-
-* ``UID``, the user id Synapse will run as [default 991]
-* ``GID``, the group id Synapse will run as [default 991]
-* ``SYNAPSE_CONFIG_PATH``, path to a custom config file
-
-If ``SYNAPSE_CONFIG_PATH`` is set, you should generate a configuration file
-then customize it manually. No other environment variable is required.
-
-Otherwise, a dynamic configuration file will be used. The following environment
-variables are available for configuration:
-
-* ``SYNAPSE_SERVER_NAME`` (mandatory), the current server public hostname.
-* ``SYNAPSE_REPORT_STATS``, (mandatory, ``yes`` or ``no``), enable anonymous
-  statistics reporting back to the Matrix project which helps us to get funding.
-* ``SYNAPSE_NO_TLS``, set this variable to disable TLS in Synapse (use this if
-  you run your own TLS-capable reverse proxy).
-* ``SYNAPSE_ENABLE_REGISTRATION``, set this variable to enable registration on
-  the Synapse instance.
-* ``SYNAPSE_ALLOW_GUEST``, set this variable to allow guest joining this server.
-* ``SYNAPSE_EVENT_CACHE_SIZE``, the event cache size [default `10K`].
-* ``SYNAPSE_CACHE_FACTOR``, the cache factor [default `0.5`].
-* ``SYNAPSE_RECAPTCHA_PUBLIC_KEY``, set this variable to the recaptcha public
-  key in order to enable recaptcha upon registration.
-* ``SYNAPSE_RECAPTCHA_PRIVATE_KEY``, set this variable to the recaptcha private
-  key in order to enable recaptcha upon registration.
-* ``SYNAPSE_TURN_URIS``, set this variable to the coma-separated list of TURN
-  uris to enable TURN for this homeserver.
-* ``SYNAPSE_TURN_SECRET``, set this to the TURN shared secret if required.
-
-Shared secrets, that will be initialized to random values if not set:
-
-* ``SYNAPSE_REGISTRATION_SHARED_SECRET``, secret for registrering users if
-  registration is disable.
-* ``SYNAPSE_MACAROON_SECRET_KEY`` secret for signing access tokens
-  to the server.
-
-Database specific values (will use SQLite if not set):
-
-* `POSTGRES_DB` - The database name for the synapse postgres database. [default: `synapse`]
-* `POSTGRES_HOST` - The host of the postgres database if you wish to use postgresql instead of sqlite3. [default: `db` which is useful when using a container on the same docker network in a compose file where the postgres service is called `db`]
-* `POSTGRES_PASSWORD` - The password for the synapse postgres database. **If this is set then postgres will be used instead of sqlite3.** [default: none] **NOTE**: You are highly encouraged to use postgresql! Please use the compose file to make it easier to deploy.
-* `POSTGRES_USER` - The user for the synapse postgres database. [default: `matrix`]
-
-Mail server specific values (will not send emails if not set):
-
-* ``SYNAPSE_SMTP_HOST``, hostname to the mail server.
-* ``SYNAPSE_SMTP_PORT``, TCP port for accessing the mail server [default ``25``].
-* ``SYNAPSE_SMTP_USER``, username for authenticating against the mail server if any.
-* ``SYNAPSE_SMTP_PASSWORD``, password for authenticating against the mail server if any.
@@ -1,219 +0,0 @@
-# vim:ft=yaml
-
-## TLS ##
-
-tls_certificate_path: "/data/{{ SYNAPSE_SERVER_NAME }}.tls.crt"
-tls_private_key_path: "/data/{{ SYNAPSE_SERVER_NAME }}.tls.key"
-tls_dh_params_path: "/data/{{ SYNAPSE_SERVER_NAME }}.tls.dh"
-no_tls: {{ "True" if SYNAPSE_NO_TLS else "False" }}
-tls_fingerprints: []
-
-## Server ##
-
-server_name: "{{ SYNAPSE_SERVER_NAME }}"
-pid_file: /homeserver.pid
-web_client: False
-soft_file_limit: 0
-
-## Ports ##
-
-listeners:
-  {% if not SYNAPSE_NO_TLS %}
-  -
-    port: 8448
-    bind_addresses: ['0.0.0.0']
-    type: http
-    tls: true
-    x_forwarded: false
-    resources:
-      - names: [client]
-        compress: true
-      - names: [federation]  # Federation APIs
-        compress: false
-  {% endif %}
-
-  - port: 8008
-    tls: false
-    bind_addresses: ['0.0.0.0']
-    type: http
-    x_forwarded: false
-
-    resources:
-      - names: [client]
-        compress: true
-      - names: [federation]
-        compress: false
-
-## Database ##
-
-{% if POSTGRES_PASSWORD %}
-database:
-  name: "psycopg2"
-  args:
-    user: "{{ POSTGRES_USER or "synapse" }}"
-    password: "{{ POSTGRES_PASSWORD }}"
-    database: "{{ POSTGRES_DB or "synapse" }}"
-    host: "{{ POSTGRES_HOST or "db" }}"
-    port: "{{ POSTGRES_PORT or "5432" }}"
-    cp_min: 5
-    cp_max: 10
-{% else %}
-database:
-  name: "sqlite3"
-  args:
-    database: "/data/homeserver.db"
-{% endif %}
-
-## Performance ##
-
-event_cache_size: "{{ SYNAPSE_EVENT_CACHE_SIZE or "10K" }}"
-verbose: 0
-log_file: "/data/homeserver.log"
-log_config: "/compiled/log.config"
-
-## Ratelimiting ##
-
-rc_messages_per_second: 0.2
-rc_message_burst_count: 10.0
-federation_rc_window_size: 1000
-federation_rc_sleep_limit: 10
-federation_rc_sleep_delay: 500
-federation_rc_reject_limit: 50
-federation_rc_concurrent: 3
-
-## Files ##
-
-media_store_path: "/data/media"
-uploads_path: "/data/uploads"
-max_upload_size: "10M"
-max_image_pixels: "32M"
-dynamic_thumbnails: false
-
-# List of thumbnail to precalculate when an image is uploaded.
-thumbnail_sizes:
- width: 32
-  height: 32
-  method: crop
- width: 96
-  height: 96
-  method: crop
- width: 320
-  height: 240
-  method: scale
- width: 640
-  height: 480
-  method: scale
- width: 800
-  height: 600
-  method: scale
-
-url_preview_enabled: False
-max_spider_size: "10M"
-
-## Captcha ##
-
-{% if SYNAPSE_RECAPTCHA_PUBLIC_KEY %}
-recaptcha_public_key: "{{ SYNAPSE_RECAPTCHA_PUBLIC_KEY }}"
-recaptcha_private_key: "{{ SYNAPSE_RECAPTCHA_PRIVATE_KEY }}"
-enable_registration_captcha: True
-recaptcha_siteverify_api: "https://www.google.com/recaptcha/api/siteverify"
-{% else %}
-recaptcha_public_key: "YOUR_PUBLIC_KEY"
-recaptcha_private_key: "YOUR_PRIVATE_KEY"
-enable_registration_captcha: False
-recaptcha_siteverify_api: "https://www.google.com/recaptcha/api/siteverify"
-{% endif %}
-
-## Turn ##
-
-{% if SYNAPSE_TURN_URIS %}
-turn_uris:
-{% for uri in SYNAPSE_TURN_URIS.split(',') %}    - "{{ uri }}"
-{% endfor %}
-turn_shared_secret: "{{ SYNAPSE_TURN_SECRET }}"
-turn_user_lifetime: "1h"
-turn_allow_guests: True
-{% else %}
-turn_uris: []
-turn_shared_secret: "YOUR_SHARED_SECRET"
-turn_user_lifetime: "1h"
-turn_allow_guests: True
-{% endif %}
-
-## Registration ##
-
-enable_registration: {{ "True" if SYNAPSE_ENABLE_REGISTRATION else "False" }}
-registration_shared_secret: "{{ SYNAPSE_REGISTRATION_SHARED_SECRET }}"
-bcrypt_rounds: 12
-allow_guest_access: {{ "True" if SYNAPSE_ALLOW_GUEST else "False" }}
-enable_group_creation: true
-
-# The list of identity servers trusted to verify third party
-# identifiers by this server.
-trusted_third_party_id_servers:
-    - matrix.org
-    - vector.im
-    - riot.im
-
-## Metrics ###
-
-{% if SYNAPSE_REPORT_STATS.lower() == "yes" %}
-enable_metrics: True
-report_stats: True
-{% else %}
-enable_metrics: False
-report_stats: False
-{% endif %}
-
-## API Configuration ##
-
-room_invite_state_types:
-    - "m.room.join_rules"
-    - "m.room.canonical_alias"
-    - "m.room.avatar"
-    - "m.room.name"
-
-{% if SYNAPSE_APPSERVICES %}
-app_service_config_files:
-{% for appservice in SYNAPSE_APPSERVICES %}    - "{{ appservice }}"
-{% endfor %}
-{% else %}
-app_service_config_files: []
-{% endif %}
-
-macaroon_secret_key: "{{ SYNAPSE_MACAROON_SECRET_KEY }}"
-expire_access_token: False
-
-## Signing Keys ##
-
-signing_key_path: "/data/{{ SYNAPSE_SERVER_NAME }}.signing.key"
-old_signing_keys: {}
-key_refresh_interval: "1d" # 1 Day.
-
-# The trusted servers to download signing keys from.
-perspectives:
-  servers:
-    "matrix.org":
-      verify_keys:
-        "ed25519:auto":
-          key: "Noi6WqcDj0QmPxCNQqgezwTlBKrfqehY1u2FyWP9uYw"
-
-password_config:
-   enabled: true
-
-{% if SYNAPSE_SMTP_HOST %}
-email:
-   enable_notifs: false
-   smtp_host: "{{ SYNAPSE_SMTP_HOST }}"
-   smtp_port: {{ SYNAPSE_SMTP_PORT or "25" }}
-   smtp_user: "{{ SYNAPSE_SMTP_USER }}"
-   smtp_pass: "{{ SYNAPSE_SMTP_PASSWORD }}"
-   require_transport_security: False
-   notif_from: "{{ SYNAPSE_SMTP_FROM or "hostmaster@" + SYNAPSE_SERVER_NAME }}"
-   app_name: Matrix
-   template_dir: res/templates
-   notif_template_html: notif_mail.html
-   notif_template_text: notif_mail.txt
-   notif_for_new_users: True
-   riot_base_url: "https://{{ SYNAPSE_SERVER_NAME }}"
-{% endif %}
@@ -1,29 +0,0 @@
-version: 1
-
-formatters:
-  precise:
-   format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s- %(message)s'
-
-filters:
-  context:
-    (): synapse.util.logcontext.LoggingContextFilter
-    request: ""
-
-handlers:
-  console:
-    class: logging.StreamHandler
-    formatter: precise
-    filters: [context]
-
-loggers:
-    synapse:
-        level: {{ SYNAPSE_LOG_LEVEL or "WARNING" }}
-
-    synapse.storage.SQL:
-        # beware: increasing this to DEBUG will make synapse log sensitive
-        # information such as access tokens.
-        level: {{ SYNAPSE_LOG_LEVEL or "WARNING" }}
-
-root:
-    level: {{ SYNAPSE_LOG_LEVEL or "WARNING" }}
-    handlers: [console]
@@ -1,49 +0,0 @@
-# This compose file is compatible with Compose itself, it might need some
-# adjustments to run properly with stack.
-
-version: '3'
-
-services:
-
-  synapse:
-    image: docker.io/matrixdotorg/synapse:latest
-    # Since snyapse does not retry to connect to the database, restart upon
-    # failure
-    restart: unless-stopped
-    # See the readme for a full documentation of the environment settings
-    environment:
-      - SYNAPSE_SERVER_NAME=my.matrix.host
-      - SYNAPSE_REPORT_STATS=no
-      - SYNAPSE_ENABLE_REGISTRATION=yes
-      - SYNAPSE_LOG_LEVEL=INFO
-      - POSTGRES_PASSWORD=changeme
-    volumes:
-      # You may either store all the files in a local folder
-      - ./files:/data
-      # .. or you may split this between different storage points
-      # - ./files:/data
-      # - /path/to/ssd:/data/uploads
-      # - /path/to/large_hdd:/data/media
-    depends_on:
-      - db
-    # In order to expose Synapse, remove one of the following, you might for
-    # instance expose the TLS port directly:
-    ports:
-      - 8448:8448/tcp
-    # ... or use a reverse proxy, here is an example for traefik:
-    labels:
-      - traefik.enable=true
-      - traefik.frontend.rule=Host:my.matrix.Host
-      - traefik.port=8448
-
-  db:
-    image: docker.io/postgres:10-alpine
-    # Change that password, of course!
-    environment:
-      - POSTGRES_USER=synapse
-      - POSTGRES_PASSWORD=changeme
-    volumes:
-      # You may store the database tables in a local folder..
-      - ./schemas:/var/lib/postgresql/data
-      # .. or store them on some high performance storage for better results
-      # - /path/to/ssd/storage:/var/lib/postfesql/data
@@ -1,66 +0,0 @@
-#!/usr/local/bin/python
-
-import jinja2
-import os
-import sys
-import subprocess
-import glob
-
-# Utility functions
-convert = lambda src, dst, environ: open(dst, "w").write(jinja2.Template(open(src).read()).render(**environ))
-
-def check_arguments(environ, args):
-    for argument in args:
-        if argument not in environ:
-            print("Environment variable %s is mandatory, exiting." % argument)
-            sys.exit(2)
-
-def generate_secrets(environ, secrets):
-    for name, secret in secrets.items():
-        if secret not in environ:
-            filename = "/data/%s.%s.key" % (environ["SYNAPSE_SERVER_NAME"], name)
-            if os.path.exists(filename):
-                with open(filename) as handle: value = handle.read()
-            else:
-                print("Generating a random secret for {}".format(name))
-                value = os.urandom(32).encode("hex")
-                with open(filename, "w") as handle: handle.write(value)
-            environ[secret] = value
-
-# Prepare the configuration
-mode = sys.argv[1] if len(sys.argv) > 1 else None
-environ = os.environ.copy()
-ownership = "{}:{}".format(environ.get("UID", 991), environ.get("GID", 991))
-args = ["python", "-m", "synapse.app.homeserver"]
-
-# In generate mode, generate a configuration, missing keys, then exit
-if mode == "generate":
-    check_arguments(environ, ("SYNAPSE_SERVER_NAME", "SYNAPSE_REPORT_STATS", "SYNAPSE_CONFIG_PATH"))
-    args += [
-        "--server-name", environ["SYNAPSE_SERVER_NAME"],
-        "--report-stats", environ["SYNAPSE_REPORT_STATS"],
-        "--config-path", environ["SYNAPSE_CONFIG_PATH"],
-        "--generate-config"
-    ]
-    os.execv("/usr/local/bin/python", args)
-
-# In normal mode, generate missing keys if any, then run synapse
-else:
-    # Parse the configuration file
-    if "SYNAPSE_CONFIG_PATH" in environ:
-        args += ["--config-path", environ["SYNAPSE_CONFIG_PATH"]]
-    else:
-        check_arguments(environ, ("SYNAPSE_SERVER_NAME", "SYNAPSE_REPORT_STATS"))
-        generate_secrets(environ, {
-            "registration": "SYNAPSE_REGISTRATION_SHARED_SECRET",
-            "macaroon": "SYNAPSE_MACAROON_SECRET_KEY"
-        })
-        environ["SYNAPSE_APPSERVICES"] = glob.glob("/data/appservices/*.yaml")
-        if not os.path.exists("/compiled"): os.mkdir("/compiled")
-        convert("/conf/homeserver.yaml", "/compiled/homeserver.yaml", environ)
-        convert("/conf/log.config", "/compiled/log.config", environ)
-        subprocess.check_output(["chown", "-R", ownership, "/data"])
-        args += ["--config-path", "/compiled/homeserver.yaml"]
-    # Generate missing keys and start synapse
-    subprocess.check_output(args + ["--generate-keys"])
-    os.execv("/sbin/su-exec", ["su-exec", ownership] + args)
@@ -22,8 +22,6 @@ import argparse
 from synapse.events import FrozenEvent
 from synapse.util.frozenutils import unfreeze

-from six import string_types
-

 def make_graph(file_name, room_id, file_prefix, limit):
    print "Reading lines"
@@ -60,7 +58,7 @@ def make_graph(file_name, room_id, file_prefix, limit):
        for key, value in unfreeze(event.get_dict()["content"]).items():
            if value is None:
                value = "<null>"
-            elif isinstance(value, string_types):
+            elif isinstance(value, basestring):
                pass
            else:
                value = json.dumps(value)
@@ -202,11 +202,11 @@ new PromConsole.Graph({
 <h1>Requests</h1>

 <h3>Requests by Servlet</h3>
-<div id="synapse_http_server_request_count_servlet"></div>
+<div id="synapse_http_server_requests_servlet"></div>
 <script>
 new PromConsole.Graph({
-  node: document.querySelector("#synapse_http_server_request_count_servlet"),
-  expr: "rate(synapse_http_server_request_count:servlet[2m])",
+  node: document.querySelector("#synapse_http_server_requests_servlet"),
+  expr: "rate(synapse_http_server_requests:servlet[2m])",
  name: "[[servlet]]",
  yAxisFormatter: PromConsole.NumberFormatter.humanize,
  yHoverFormatter: PromConsole.NumberFormatter.humanize,
@@ -215,11 +215,11 @@ new PromConsole.Graph({
 })
 </script>
 <h4>&nbsp;(without <tt>EventStreamRestServlet</tt> or <tt>SyncRestServlet</tt>)</h4>
-<div id="synapse_http_server_request_count_servlet_minus_events"></div>
+<div id="synapse_http_server_requests_servlet_minus_events"></div>
 <script>
 new PromConsole.Graph({
-  node: document.querySelector("#synapse_http_server_request_count_servlet_minus_events"),
-  expr: "rate(synapse_http_server_request_count:servlet{servlet!=\"EventStreamRestServlet\", servlet!=\"SyncRestServlet\"}[2m])",
+  node: document.querySelector("#synapse_http_server_requests_servlet_minus_events"),
+  expr: "rate(synapse_http_server_requests:servlet{servlet!=\"EventStreamRestServlet\", servlet!=\"SyncRestServlet\"}[2m])",
  name: "[[servlet]]",
  yAxisFormatter: PromConsole.NumberFormatter.humanize,
  yHoverFormatter: PromConsole.NumberFormatter.humanize,
@@ -233,7 +233,7 @@ new PromConsole.Graph({
 <script>
 new PromConsole.Graph({
  node: document.querySelector("#synapse_http_server_response_time_avg"),
-  expr: "rate(synapse_http_server_response_time_seconds[2m]) / rate(synapse_http_server_response_count[2m]) / 1000",
+  expr: "rate(synapse_http_server_response_time:total[2m]) / rate(synapse_http_server_response_time:count[2m]) / 1000",
  name: "[[servlet]]",
  yAxisFormatter: PromConsole.NumberFormatter.humanize,
  yHoverFormatter: PromConsole.NumberFormatter.humanize,
@@ -276,7 +276,7 @@ new PromConsole.Graph({
 <script>
 new PromConsole.Graph({
  node: document.querySelector("#synapse_http_server_response_ru_utime"),
-  expr: "rate(synapse_http_server_response_ru_utime_seconds[2m])",
+  expr: "rate(synapse_http_server_response_ru_utime:total[2m])",
  name: "[[servlet]]",
  yAxisFormatter: PromConsole.NumberFormatter.humanize,
  yHoverFormatter: PromConsole.NumberFormatter.humanize,
@@ -291,7 +291,7 @@ new PromConsole.Graph({
 <script>
 new PromConsole.Graph({
  node: document.querySelector("#synapse_http_server_response_db_txn_duration"),
-  expr: "rate(synapse_http_server_response_db_txn_duration_seconds[2m])",
+  expr: "rate(synapse_http_server_response_db_txn_duration:total[2m])",
  name: "[[servlet]]",
  yAxisFormatter: PromConsole.NumberFormatter.humanize,
  yHoverFormatter: PromConsole.NumberFormatter.humanize,
@@ -306,7 +306,7 @@ new PromConsole.Graph({
 <script>
 new PromConsole.Graph({
  node: document.querySelector("#synapse_http_server_send_time_avg"),
-  expr: "rate(synapse_http_server_response_time_second{servlet='RoomSendEventRestServlet'}[2m]) / rate(synapse_http_server_response_count{servlet='RoomSendEventRestServlet'}[2m]) / 1000",
+  expr: "rate(synapse_http_server_response_time:total{servlet='RoomSendEventRestServlet'}[2m]) / rate(synapse_http_server_response_time:count{servlet='RoomSendEventRestServlet'}[2m]) / 1000",
  name: "[[servlet]]",
  yAxisFormatter: PromConsole.NumberFormatter.humanize,
  yHoverFormatter: PromConsole.NumberFormatter.humanize,
@@ -1,10 +1,10 @@
 synapse_federation_transaction_queue_pendingEdus:total = sum(synapse_federation_transaction_queue_pendingEdus or absent(synapse_federation_transaction_queue_pendingEdus)*0)
 synapse_federation_transaction_queue_pendingPdus:total = sum(synapse_federation_transaction_queue_pendingPdus or absent(synapse_federation_transaction_queue_pendingPdus)*0)

-synapse_http_server_request_count:method{servlet=""} = sum(synapse_http_server_request_count) by (method)
-synapse_http_server_request_count:servlet{method=""} = sum(synapse_http_server_request_count) by (servlet)
+synapse_http_server_requests:method{servlet=""} = sum(synapse_http_server_requests) by (method)
+synapse_http_server_requests:servlet{method=""} = sum(synapse_http_server_requests) by (servlet)

-synapse_http_server_request_count:total{servlet=""} = sum(synapse_http_server_request_count:by_method) by (servlet)
+synapse_http_server_requests:total{servlet=""} = sum(synapse_http_server_requests:by_method) by (servlet)

 synapse_cache:hit_ratio_5m = rate(synapse_util_caches_cache:hits[5m]) / rate(synapse_util_caches_cache:total[5m])
 synapse_cache:hit_ratio_30s = rate(synapse_util_caches_cache:hits[30s]) / rate(synapse_util_caches_cache:total[30s])
@@ -5,19 +5,19 @@ groups:
    expr: "sum(synapse_federation_transaction_queue_pendingEdus or absent(synapse_federation_transaction_queue_pendingEdus)*0)"
  - record: "synapse_federation_transaction_queue_pendingPdus:total"
    expr:   "sum(synapse_federation_transaction_queue_pendingPdus or absent(synapse_federation_transaction_queue_pendingPdus)*0)"
-  - record: 'synapse_http_server_request_count:method'
+  - record: 'synapse_http_server_requests:method'
    labels:
      servlet: ""
-    expr: "sum(synapse_http_server_request_count) by (method)"
-  - record: 'synapse_http_server_request_count:servlet'
+    expr: "sum(synapse_http_server_requests) by (method)"
+  - record: 'synapse_http_server_requests:servlet'
    labels:
      method: ""
-    expr: 'sum(synapse_http_server_request_count) by (servlet)'
+    expr: 'sum(synapse_http_server_requests) by (servlet)'

-  - record: 'synapse_http_server_request_count:total'
+  - record: 'synapse_http_server_requests:total'
    labels:
      servlet: ""
-    expr: 'sum(synapse_http_server_request_count:by_method) by (servlet)'
+    expr: 'sum(synapse_http_server_requests:by_method) by (servlet)'

  - record: 'synapse_cache:hit_ratio_5m'
    expr: 'rate(synapse_util_caches_cache:hits[5m]) / rate(synapse_util_caches_cache:total[5m])'
@@ -2,9 +2,6 @@
 # (e.g. https://www.archlinux.org/packages/community/any/matrix-synapse/ for ArchLinux)
 # rather than in a user home directory or similar under virtualenv.

-# **NOTE:** This is an example service file that may change in the future. If you
-# wish to use this please copy rather than symlink it.
-
 [Unit]
 Description=Synapse Matrix homeserver

@@ -15,7 +12,6 @@ Group=synapse
 WorkingDirectory=/var/lib/synapse
 ExecStart=/usr/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/synapse/homeserver.yaml
 ExecStop=/usr/bin/synctl stop /etc/synapse/homeserver.yaml
-# EnvironmentFile=-/etc/sysconfig/synapse  # Can be used to e.g. set SYNAPSE_CACHE_FACTOR 

 [Install]
 WantedBy=multi-user.target
@@ -8,56 +8,20 @@ Depending on the amount of history being purged a call to the API may take
 several minutes or longer. During this period users will not be able to
 paginate further back in the room from the point being purged from.

-The API is:
+The API is simply:

-``POST /_matrix/client/r0/admin/purge_history/<room_id>[/<event_id>]``
+``POST /_matrix/client/r0/admin/purge_history/<room_id>/<event_id>``

 including an ``access_token`` of a server admin.

 By default, events sent by local users are not deleted, as they may represent
 the only copies of this content in existence. (Events sent by remote users are
-deleted.)
+deleted, and room state data before the cutoff is always removed).

-Room state data (such as joins, leaves, topic) is always preserved.
-
-To delete local message events as well, set ``delete_local_events`` in the body:
+To delete local events as well, set ``delete_local_events`` in the body:

 .. code:: json

   {
       "delete_local_events": true
   }
-
-The caller must specify the point in the room to purge up to. This can be
-specified by including an event_id in the URI, or by setting a
-``purge_up_to_event_id`` or ``purge_up_to_ts`` in the request body. If an event
-id is given, that event (and others at the same graph depth) will be retained.
-If ``purge_up_to_ts`` is given, it should be a timestamp since the unix epoch,
-in milliseconds.
-
-The API starts the purge running, and returns immediately with a JSON body with
-a purge id:
-
-.. code:: json
-
-    {
-        "purge_id": "<opaque id>"
-    }
-
-Purge status query
------------------
-
-It is possible to poll for updates on recent purges with a second API;
-
-``GET /_matrix/client/r0/admin/purge_history_status/<purge_id>``
-
-(again, with a suitable ``access_token``). This API returns a JSON body like
-the following:
-
-.. code:: json
-
-    {
-        "status": "active"
-    }
-
-The status will be one of ``active``, ``complete``, or ``failed``.
@@ -279,9 +279,9 @@ Obviously that option means that the operations done in
 that might be fixed by setting a different logcontext via a ``with
 LoggingContext(...)`` in ``background_operation``).

-The second option is to use ``logcontext.run_in_background``, which wraps a
-function so that it doesn't reset the logcontext even when it returns an
-incomplete deferred, and adds a callback to the returned deferred to reset the
+The second option is to use ``logcontext.preserve_fn``, which wraps a function
+so that it doesn't reset the logcontext even when it returns an incomplete
+deferred, and adds a callback to the returned deferred to reset the
 logcontext. In other words, it turns a function that follows the Synapse rules
 about logcontexts and Deferreds into one which behaves more like an external
 function — the opposite operation to that described in the previous section.
@@ -293,7 +293,7 @@ It can be used like this:
    def do_request_handling():
        yield foreground_operation()

-        logcontext.run_in_background(background_operation)
+        logcontext.preserve_fn(background_operation)()

        # this will now be logged against the request context
        logger.debug("Request handling complete")
@@ -55,12 +55,7 @@ synapse process.)

 You then create a set of configs for the various worker processes.  These
 should be worker configuration files, and should be stored in a dedicated
-subdirectory, to allow synctl to manipulate them. An additional configuration
-for the master synapse process will need to be created because the process will
-not be started automatically. That configuration should look like this::
-
-    worker_app: synapse.app.homeserver
-    daemonize: true
+subdirectory, to allow synctl to manipulate them.

 Each worker configuration file inherits the configuration of the main homeserver
 configuration file.  You can then override configuration specific to that worker,
@@ -235,11 +230,9 @@ file. For example::
 ``synapse.app.event_creator``
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-Handles some event creation. It can handle REST endpoints matching::
+Handles non-state event creation. It can handle REST endpoints matching:

    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/send
-    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$
-    ^/_matrix/client/(api/v1|r0|unstable)/join/

 It will create events locally and then send them on to the main synapse
 instance to be persisted and handled.
@@ -1,7 +1,5 @@
 #! /bin/bash

-set -eux
-
 cd "`dirname $0`/.."

 TOX_DIR=$WORKSPACE/.tox
@@ -16,20 +14,7 @@ fi
 tox -e py27 --notest -v

 TOX_BIN=$TOX_DIR/py27/bin
-
-# cryptography 2.2 requires setuptools >= 18.5.
-#
-# older versions of virtualenv (?) give us a virtualenv with the same version
-# of setuptools as is installed on the system python (and tox runs virtualenv
-# under python3, so we get the version of setuptools that is installed on that).
-#
-# anyway, make sure that we have a recent enough setuptools.
-$TOX_BIN/pip install 'setuptools>=18.5'
-
-# we also need a semi-recent version of pip, because old ones fail to install
-# the "enum34" dependency of cryptography.
-$TOX_BIN/pip install 'pip>=10'
-
+$TOX_BIN/pip install setuptools
 { python synapse/python_dependencies.py
  echo lxml psycopg2
 } | xargs $TOX_BIN/pip install
@@ -6,19 +6,9 @@

 ## Do not run it lightly.

-set -e
-
-if [ "$1" == "-h" ] || [ "$1" == "" ]; then
-  echo "Call with ROOM_ID as first option and then pipe it into the database. So for instance you might run"
-  echo " nuke-room-from-db.sh <room_id> | sqlite3 homeserver.db"
-  echo "or"
-  echo " nuke-room-from-db.sh <room_id> | psql --dbname=synapse"
-  exit
-fi
-
 ROOMID="$1"

-cat <<EOF
+sqlite3 homeserver.db <<EOF
 DELETE FROM event_forward_extremities WHERE room_id = '$ROOMID';
 DELETE FROM event_backward_extremities WHERE room_id = '$ROOMID';
 DELETE FROM event_edges WHERE room_id = '$ROOMID';
@@ -39,7 +29,7 @@ DELETE FROM state_groups WHERE room_id = '$ROOMID';
 DELETE FROM state_groups_state WHERE room_id = '$ROOMID';
 DELETE FROM receipts_graph WHERE room_id = '$ROOMID';
 DELETE FROM receipts_linearized WHERE room_id = '$ROOMID';
-DELETE FROM event_search WHERE room_id = '$ROOMID'; 
+DELETE FROM event_search_content WHERE c1room_id = '$ROOMID';
 DELETE FROM guest_access WHERE room_id = '$ROOMID';
 DELETE FROM history_visibility WHERE room_id = '$ROOMID';
 DELETE FROM room_tags WHERE room_id = '$ROOMID';
@@ -1,7 +1,6 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 # Copyright 2015, 2016 OpenMarket Ltd
-# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -30,8 +29,6 @@ import time
 import traceback
 import yaml

-from six import string_types
-

 logger = logging.getLogger("synapse_port_db")

@@ -253,12 +250,6 @@ class Porter(object):
    @defer.inlineCallbacks
    def handle_table(self, table, postgres_size, table_size, forward_chunk,
                     backward_chunk):
-        logger.info(
-            "Table %s: %i/%i (rows %i-%i) already ported",
-            table, postgres_size, table_size,
-            backward_chunk+1, forward_chunk-1,
-        )
-
        if not table_size:
            return

@@ -476,10 +467,31 @@ class Porter(object):
            self.progress.set_state("Preparing PostgreSQL")
            self.setup_db(postgres_config, postgres_engine)

-            self.progress.set_state("Creating port tables")
+            # Step 2. Get tables.
+            self.progress.set_state("Fetching tables")
+            sqlite_tables = yield self.sqlite_store._simple_select_onecol(
+                table="sqlite_master",
+                keyvalues={
+                    "type": "table",
+                },
+                retcol="name",
+            )
+
+            postgres_tables = yield self.postgres_store._simple_select_onecol(
+                table="information_schema.tables",
+                keyvalues={},
+                retcol="distinct table_name",
+            )
+
+            tables = set(sqlite_tables) & set(postgres_tables)
+
+            self.progress.set_state("Creating tables")
+
+            logger.info("Found %d tables", len(tables))
+
            def create_port_table(txn):
                txn.execute(
-                    "CREATE TABLE IF NOT EXISTS port_from_sqlite3 ("
+                    "CREATE TABLE port_from_sqlite3 ("
                    " table_name varchar(100) NOT NULL UNIQUE,"
                    " forward_rowid bigint NOT NULL,"
                    " backward_rowid bigint NOT NULL"
@@ -505,33 +517,18 @@ class Porter(object):
                    "alter_table", alter_table
                )
            except Exception as e:
-                pass
+                logger.info("Failed to create port table: %s", e)

-            yield self.postgres_store.runInteraction(
-                "create_port_table", create_port_table
-            )
+            try:
+                yield self.postgres_store.runInteraction(
+                    "create_port_table", create_port_table
+                )
+            except Exception as e:
+                logger.info("Failed to create port table: %s", e)

-            # Step 2. Get tables.
-            self.progress.set_state("Fetching tables")
-            sqlite_tables = yield self.sqlite_store._simple_select_onecol(
-                table="sqlite_master",
-                keyvalues={
-                    "type": "table",
-                },
-                retcol="name",
-            )
+            self.progress.set_state("Setting up")

-            postgres_tables = yield self.postgres_store._simple_select_onecol(
-                table="information_schema.tables",
-                keyvalues={},
-                retcol="distinct table_name",
-            )
-
-            tables = set(sqlite_tables) & set(postgres_tables)
-            logger.info("Found %d tables", len(tables))
-
-            # Step 3. Figure out what still needs copying
-            self.progress.set_state("Checking on port progress")
+            # Set up tables.
            setup_res = yield defer.gatherResults(
                [
                    self.setup_table(table)
@@ -542,8 +539,7 @@ class Porter(object):
                consumeErrors=True,
            )

-            # Step 4. Do the copying.
-            self.progress.set_state("Copying to postgres")
+            # Process tables.
            yield defer.gatherResults(
                [
                    self.handle_table(*res)
@@ -552,9 +548,6 @@ class Porter(object):
                consumeErrors=True,
            )

-            # Step 5. Do final post-processing
-            yield self._setup_state_group_id_seq()
-
            self.progress.done()
        except:
            global end_error_exec_info
@@ -576,7 +569,7 @@ class Porter(object):
        def conv(j, col):
            if j in bool_cols:
                return bool(col)
-            elif isinstance(col, string_types) and "\0" in col:
+            elif isinstance(col, basestring) and "\0" in col:
                logger.warn("DROPPING ROW: NUL value in table %s col %s: %r", table, headers[j], col)
                raise BadValueException();
            return col
@@ -714,16 +707,6 @@ class Porter(object):

        defer.returnValue((done, remaining + done))

-    def _setup_state_group_id_seq(self):
-        def r(txn):
-            txn.execute("SELECT MAX(id) FROM state_groups")
-            next_id = txn.fetchone()[0]+1
-            txn.execute(
-                "ALTER SEQUENCE state_group_id_seq RESTART WITH %s",
-                (next_id,),
-            )
-        return self.postgres_store.runInteraction("setup_state_group_id_seq", r)
-

 ##############################################
 ###### The following is simply UI stuff ######
@@ -16,4 +16,4 @@
 """ This is a reference implementation of a Matrix home server.
 """

-__version__ = "0.29.1"
+__version__ = "0.26.0"
@@ -204,8 +204,8 @@ class Auth(object):

            ip_addr = self.hs.get_ip_from_request(request)
            user_agent = request.requestHeaders.getRawHeaders(
-                b"User-Agent",
-                default=[b""]
+                "User-Agent",
+                default=[""]
            )[0]
            if user and access_token and ip_addr:
                self.store.insert_client_ip(
@@ -672,7 +672,7 @@ def has_access_token(request):
        bool: False if no access_token was given, True otherwise.
    """
    query_params = request.args.get("access_token")
-    auth_headers = request.requestHeaders.getRawHeaders(b"Authorization")
+    auth_headers = request.requestHeaders.getRawHeaders("Authorization")
    return bool(query_params) or bool(auth_headers)


@@ -692,8 +692,8 @@ def get_access_token_from_request(request, token_not_found_http_status=401):
        AuthError: If there isn't an access_token in the request.
    """

-    auth_headers = request.requestHeaders.getRawHeaders(b"Authorization")
-    query_params = request.args.get(b"access_token")
+    auth_headers = request.requestHeaders.getRawHeaders("Authorization")
+    query_params = request.args.get("access_token")
    if auth_headers:
        # Try the get the access_token from a "Authorization: Bearer"
        # header
@@ -16,9 +16,6 @@

 """Contains constants from the specification."""

-# the "depth" field on events is limited to 2**63 - 1
-MAX_DEPTH = 2**63 - 1
-

 class Membership(object):

@@ -15,11 +15,9 @@

 """Contains exceptions and error codes."""

+import json
 import logging

-import simplejson as json
-from six import iteritems
-
 logger = logging.getLogger(__name__)


@@ -298,7 +296,7 @@ def cs_error(msg, code=Codes.UNKNOWN, **kwargs):
        A dict representing the error response JSON.
    """
    err = {"error": msg, "errcode": code}
-    for key, value in iteritems(kwargs):
+    for key, value in kwargs.iteritems():
        err[key] = value
    return err

@@ -17,7 +17,7 @@ from synapse.storage.presence import UserPresenceState
 from synapse.types import UserID, RoomID
 from twisted.internet import defer

-import simplejson as json
+import ujson as json
 import jsonschema
 from jsonschema import FormatChecker

@@ -32,11 +32,11 @@ from synapse.replication.tcp.client import ReplicationClientHandler
 from synapse.server import HomeServer
 from synapse.storage.engines import create_engine
 from synapse.util.httpresourcetree import create_resource_tree
-from synapse.util.logcontext import LoggingContext, run_in_background
+from synapse.util.logcontext import LoggingContext, preserve_fn
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
-from twisted.internet import reactor, defer
-from twisted.web.resource import NoResource
+from twisted.internet import reactor
+from twisted.web.resource import Resource

 logger = logging.getLogger("synapse.app.appservice")

@@ -64,7 +64,7 @@ class AppserviceServer(HomeServer):
                if name == "metrics":
                    resources[METRICS_PREFIX] = MetricsResource(self)

-        root_resource = create_resource_tree(resources, NoResource())
+        root_resource = create_resource_tree(resources, Resource())

        _base.listen_tcp(
            bind_addresses,
@@ -74,7 +74,6 @@ class AppserviceServer(HomeServer):
                site_tag,
                listener_config,
                root_resource,
-                self.version_string,
            )
        )

@@ -113,14 +112,9 @@ class ASReplicationHandler(ReplicationClientHandler):

        if stream_name == "events":
            max_stream_id = self.store.get_room_max_stream_ordering()
-            run_in_background(self._notify_app_services, max_stream_id)
-
-    @defer.inlineCallbacks
-    def _notify_app_services(self, room_stream_id):
-        try:
-            yield self.appservice_handler.notify_interested_services(room_stream_id)
-        except Exception:
-            logger.exception("Error notifying application services of event")
+            preserve_fn(
+                self.appservice_handler.notify_interested_services
+            )(max_stream_id)


 def start(config_options):
@@ -44,7 +44,7 @@ from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
 from twisted.internet import reactor
-from twisted.web.resource import NoResource
+from twisted.web.resource import Resource

 logger = logging.getLogger("synapse.app.client_reader")

@@ -88,7 +88,7 @@ class ClientReaderServer(HomeServer):
                        "/_matrix/client/api/v1": resource,
                    })

-        root_resource = create_resource_tree(resources, NoResource())
+        root_resource = create_resource_tree(resources, Resource())

        _base.listen_tcp(
            bind_addresses,
@@ -98,7 +98,6 @@ class ClientReaderServer(HomeServer):
                site_tag,
                listener_config,
                root_resource,
-                self.version_string,
            )
        )

@@ -157,6 +156,7 @@ def start(config_options):
    )

    ss.setup()
+    ss.get_handlers()
    ss.start_listening(config.worker_listeners)

    def start():
@@ -27,24 +27,14 @@ from synapse.http.server import JsonResource
 from synapse.http.site import SynapseSite
 from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
 from synapse.replication.slave.storage._base import BaseSlavedStore
-from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
 from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
 from synapse.replication.slave.storage.client_ips import SlavedClientIpStore
 from synapse.replication.slave.storage.devices import SlavedDeviceStore
-from synapse.replication.slave.storage.directory import DirectoryStore
 from synapse.replication.slave.storage.events import SlavedEventStore
-from synapse.replication.slave.storage.profile import SlavedProfileStore
-from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
-from synapse.replication.slave.storage.pushers import SlavedPusherStore
-from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
 from synapse.replication.slave.storage.registration import SlavedRegistrationStore
 from synapse.replication.slave.storage.room import RoomStore
-from synapse.replication.slave.storage.transactions import TransactionStore
 from synapse.replication.tcp.client import ReplicationClientHandler
-from synapse.rest.client.v1.room import (
-    RoomSendEventRestServlet, RoomMembershipRestServlet, RoomStateEventRestServlet,
-    JoinRoomAliasServlet,
-)
+from synapse.rest.client.v1.room import RoomSendEventRestServlet
 from synapse.server import HomeServer
 from synapse.storage.engines import create_engine
 from synapse.util.httpresourcetree import create_resource_tree
@@ -52,19 +42,12 @@ from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
 from twisted.internet import reactor
-from twisted.web.resource import NoResource
+from twisted.web.resource import Resource

 logger = logging.getLogger("synapse.app.event_creator")


 class EventCreatorSlavedStore(
-    DirectoryStore,
-    TransactionStore,
-    SlavedProfileStore,
-    SlavedAccountDataStore,
-    SlavedPusherStore,
-    SlavedReceiptsStore,
-    SlavedPushRuleStore,
    SlavedDeviceStore,
    SlavedClientIpStore,
    SlavedApplicationServiceStore,
@@ -94,9 +77,6 @@ class EventCreatorServer(HomeServer):
                elif name == "client":
                    resource = JsonResource(self, canonical_json=False)
                    RoomSendEventRestServlet(self).register(resource)
-                    RoomMembershipRestServlet(self).register(resource)
-                    RoomStateEventRestServlet(self).register(resource)
-                    JoinRoomAliasServlet(self).register(resource)
                    resources.update({
                        "/_matrix/client/r0": resource,
                        "/_matrix/client/unstable": resource,
@@ -104,7 +84,7 @@ class EventCreatorServer(HomeServer):
                        "/_matrix/client/api/v1": resource,
                    })

-        root_resource = create_resource_tree(resources, NoResource())
+        root_resource = create_resource_tree(resources, Resource())

        _base.listen_tcp(
            bind_addresses,
@@ -114,7 +94,6 @@ class EventCreatorServer(HomeServer):
                site_tag,
                listener_config,
                root_resource,
-                self.version_string,
            )
        )

@@ -174,6 +153,7 @@ def start(config_options):
    )

    ss.setup()
+    ss.get_handlers()
    ss.start_listening(config.worker_listeners)

    def start():
@@ -41,7 +41,7 @@ from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
 from twisted.internet import reactor
-from twisted.web.resource import NoResource
+from twisted.web.resource import Resource

 logger = logging.getLogger("synapse.app.federation_reader")

@@ -77,7 +77,7 @@ class FederationReaderServer(HomeServer):
                        FEDERATION_PREFIX: TransportLayerServer(self),
                    })

-        root_resource = create_resource_tree(resources, NoResource())
+        root_resource = create_resource_tree(resources, Resource())

        _base.listen_tcp(
            bind_addresses,
@@ -87,7 +87,6 @@ class FederationReaderServer(HomeServer):
                site_tag,
                listener_config,
                root_resource,
-                self.version_string,
            )
        )

@@ -145,6 +144,7 @@ def start(config_options):
    )

    ss.setup()
+    ss.get_handlers()
    ss.start_listening(config.worker_listeners)

    def start():
@@ -38,11 +38,11 @@ from synapse.server import HomeServer
 from synapse.storage.engines import create_engine
 from synapse.util.async import Linearizer
 from synapse.util.httpresourcetree import create_resource_tree
-from synapse.util.logcontext import LoggingContext, run_in_background
+from synapse.util.logcontext import LoggingContext, preserve_fn
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
 from twisted.internet import defer, reactor
-from twisted.web.resource import NoResource
+from twisted.web.resource import Resource

 logger = logging.getLogger("synapse.app.federation_sender")

@@ -91,7 +91,7 @@ class FederationSenderServer(HomeServer):
                if name == "metrics":
                    resources[METRICS_PREFIX] = MetricsResource(self)

-        root_resource = create_resource_tree(resources, NoResource())
+        root_resource = create_resource_tree(resources, Resource())

        _base.listen_tcp(
            bind_addresses,
@@ -101,7 +101,6 @@ class FederationSenderServer(HomeServer):
                site_tag,
                listener_config,
                root_resource,
-                self.version_string,
            )
        )

@@ -230,7 +229,7 @@ class FederationSenderHandler(object):
        # presence, typing, etc.
        if stream_name == "federation":
            send_queue.process_rows_for_federation(self.federation_sender, rows)
-            run_in_background(self.update_token, token)
+            preserve_fn(self.update_token)(token)

        # We also need to poke the federation sender when new events happen
        elif stream_name == "events":
@@ -238,22 +237,19 @@ class FederationSenderHandler(object):

    @defer.inlineCallbacks
    def update_token(self, token):
-        try:
-            self.federation_position = token
+        self.federation_position = token

-            # We linearize here to ensure we don't have races updating the token
-            with (yield self._fed_position_linearizer.queue(None)):
-                if self._last_ack < self.federation_position:
-                    yield self.store.update_federation_out_pos(
-                        "federation", self.federation_position
-                    )
+        # We linearize here to ensure we don't have races updating the token
+        with (yield self._fed_position_linearizer.queue(None)):
+            if self._last_ack < self.federation_position:
+                yield self.store.update_federation_out_pos(
+                    "federation", self.federation_position
+                )

-                    # We ACK this token over replication so that the master can drop
-                    # its in memory queues
-                    self.replication_client.send_federation_ack(self.federation_position)
-                    self._last_ack = self.federation_position
-        except Exception:
-            logger.exception("Error updating federation stream position")
+                # We ACK this token over replication so that the master can drop
+                # its in memory queues
+                self.replication_client.send_federation_ack(self.federation_position)
+                self._last_ack = self.federation_position


 if __name__ == '__main__':
@@ -36,6 +36,7 @@ from synapse.replication.slave.storage.client_ips import SlavedClientIpStore
 from synapse.replication.slave.storage.devices import SlavedDeviceStore
 from synapse.replication.slave.storage.registration import SlavedRegistrationStore
 from synapse.replication.tcp.client import ReplicationClientHandler
+from synapse.rest.client.v1.base import ClientV1RestServlet, client_path_patterns
 from synapse.rest.client.v2_alpha._base import client_v2_patterns
 from synapse.server import HomeServer
 from synapse.storage.engines import create_engine
@@ -44,11 +45,40 @@ from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
 from twisted.internet import defer, reactor
-from twisted.web.resource import NoResource
+from twisted.web.resource import Resource

 logger = logging.getLogger("synapse.app.frontend_proxy")


+class PresenceStatusStubServlet(ClientV1RestServlet):
+    PATTERNS = client_path_patterns("/presence/(?P<user_id>[^/]*)/status")
+
+    def __init__(self, hs):
+        super(PresenceStatusStubServlet, self).__init__(hs)
+        self.http_client = hs.get_simple_http_client()
+        self.auth = hs.get_auth()
+        self.main_uri = hs.config.worker_main_http_uri
+
+    @defer.inlineCallbacks
+    def on_GET(self, request, user_id):
+        # Pass through the auth headers, if any, in case the access token
+        # is there.
+        auth_headers = request.requestHeaders.getRawHeaders("Authorization", [])
+        headers = {
+            "Authorization": auth_headers,
+        }
+        result = yield self.http_client.get_json(
+            self.main_uri + request.uri,
+            headers=headers,
+        )
+        defer.returnValue((200, result))
+
+    @defer.inlineCallbacks
+    def on_PUT(self, request, user_id):
+        yield self.auth.get_user_by_req(request)
+        defer.returnValue((200, {}))
+
+
 class KeyUploadServlet(RestServlet):
    PATTERNS = client_v2_patterns("/keys/upload(/(?P<device_id>[^/]+))?$")

@@ -90,7 +120,7 @@ class KeyUploadServlet(RestServlet):
            # They're actually trying to upload something, proxy to main synapse.
            # Pass through the auth headers, if any, in case the access token
            # is there.
-            auth_headers = request.requestHeaders.getRawHeaders(b"Authorization", [])
+            auth_headers = request.requestHeaders.getRawHeaders("Authorization", [])
            headers = {
                "Authorization": auth_headers,
            }
@@ -135,6 +165,7 @@ class FrontendProxyServer(HomeServer):
                elif name == "client":
                    resource = JsonResource(self, canonical_json=False)
                    KeyUploadServlet(self).register(resource)
+                    PresenceStatusStubServlet(self).register(resource)
                    resources.update({
                        "/_matrix/client/r0": resource,
                        "/_matrix/client/unstable": resource,
@@ -142,7 +173,7 @@ class FrontendProxyServer(HomeServer):
                        "/_matrix/client/api/v1": resource,
                    })

-        root_resource = create_resource_tree(resources, NoResource())
+        root_resource = create_resource_tree(resources, Resource())

        _base.listen_tcp(
            bind_addresses,
@@ -152,7 +183,6 @@ class FrontendProxyServer(HomeServer):
                site_tag,
                listener_config,
                root_resource,
-                self.version_string,
            )
        )

@@ -212,6 +242,7 @@ def start(config_options):
    )

    ss.setup()
+    ss.get_handlers()
    ss.start_listening(config.worker_listeners)

    def start():
@@ -48,7 +48,6 @@ from synapse.server import HomeServer
 from synapse.storage import are_all_users_on_domain
 from synapse.storage.engines import IncorrectDatabaseSetup, create_engine
 from synapse.storage.prepare_database import UpgradeDatabaseException, prepare_database
-from synapse.util.caches import CACHE_SIZE_FACTOR
 from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
@@ -57,7 +56,7 @@ from synapse.util.rlimit import change_resource_limit
 from synapse.util.versionstring import get_version_string
 from twisted.application import service
 from twisted.internet import defer, reactor
-from twisted.web.resource import EncodingResourceWrapper, NoResource
+from twisted.web.resource import EncodingResourceWrapper, Resource
 from twisted.web.server import GzipEncoderFactory
 from twisted.web.static import File

@@ -127,7 +126,7 @@ class SynapseHomeServer(HomeServer):
        if WEB_CLIENT_PREFIX in resources:
            root_resource = RootRedirect(WEB_CLIENT_PREFIX)
        else:
-            root_resource = NoResource()
+            root_resource = Resource()

        root_resource = create_resource_tree(resources, root_resource)

@@ -140,7 +139,6 @@ class SynapseHomeServer(HomeServer):
                    site_tag,
                    listener_config,
                    root_resource,
-                    self.version_string,
                ),
                self.tls_server_context_factory,
            )
@@ -154,7 +152,6 @@ class SynapseHomeServer(HomeServer):
                    site_tag,
                    listener_config,
                    root_resource,
-                    self.version_string,
                )
            )
        logger.info("Synapse now listening on port %d", port)
@@ -351,7 +348,7 @@ def setup(config_options):
        hs.get_state_handler().start_caching()
        hs.get_datastore().start_profiling()
        hs.get_datastore().start_doing_background_updates()
-        hs.get_federation_client().start_get_pdu_cache()
+        hs.get_replication_layer().start_get_pdu_cache()

        register_memory_metrics(hs)

@@ -405,10 +402,6 @@ def run(hs):

    stats = {}

-    # Contains the list of processes we will be monitoring
-    # currently either 0 or 1
-    stats_process = []
-
    @defer.inlineCallbacks
    def phone_stats_home():
        logger.info("Gathering stats for reporting")
@@ -432,21 +425,8 @@ def run(hs):
        stats["daily_active_rooms"] = yield hs.get_datastore().count_daily_active_rooms()
        stats["daily_messages"] = yield hs.get_datastore().count_daily_messages()

-        r30_results = yield hs.get_datastore().count_r30_users()
-        for name, count in r30_results.iteritems():
-            stats["r30_users_" + name] = count
-
        daily_sent_messages = yield hs.get_datastore().count_daily_sent_messages()
        stats["daily_sent_messages"] = daily_sent_messages
-        stats["cache_factor"] = CACHE_SIZE_FACTOR
-        stats["event_cache_size"] = hs.config.event_cache_size
-
-        if len(stats_process) > 0:
-            stats["memory_rss"] = 0
-            stats["cpu_average"] = 0
-            for process in stats_process:
-                stats["memory_rss"] += process.memory_info().rss
-                stats["cpu_average"] += int(process.cpu_percent(interval=None))

        logger.info("Reporting stats to matrix.org: %s" % (stats,))
        try:
@@ -457,32 +437,10 @@ def run(hs):
        except Exception as e:
            logger.warn("Error reporting stats: %s", e)

-    def performance_stats_init():
-        try:
-            import psutil
-            process = psutil.Process()
-            # Ensure we can fetch both, and make the initial request for cpu_percent
-            # so the next request will use this as the initial point.
-            process.memory_info().rss
-            process.cpu_percent(interval=None)
-            logger.info("report_stats can use psutil")
-            stats_process.append(process)
-        except (ImportError, AttributeError):
-            logger.warn(
-                "report_stats enabled but psutil is not installed or incorrect version."
-                " Disabling reporting of memory/cpu stats."
-                " Ensuring psutil is available will help matrix.org track performance"
-                " changes across releases."
-            )
-
    if hs.config.report_stats:
        logger.info("Scheduling stats reporting for 3 hour intervals")
        clock.looping_call(phone_stats_home, 3 * 60 * 60 * 1000)

-        # We need to defer this init for the cases that we daemonize
-        # otherwise the process ID we get is that of the non-daemon process
-        clock.call_later(0, performance_stats_init)
-
        # We wait 5 minutes to send the first set of stats as the server can
        # be quite busy the first few minutes
        clock.call_later(5 * 60, phone_stats_home)
@@ -43,7 +43,7 @@ from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
 from twisted.internet import reactor
-from twisted.web.resource import NoResource
+from twisted.web.resource import Resource

 logger = logging.getLogger("synapse.app.media_repository")

@@ -84,7 +84,7 @@ class MediaRepositoryServer(HomeServer):
                        ),
                    })

-        root_resource = create_resource_tree(resources, NoResource())
+        root_resource = create_resource_tree(resources, Resource())

        _base.listen_tcp(
            bind_addresses,
@@ -94,7 +94,6 @@ class MediaRepositoryServer(HomeServer):
                site_tag,
                listener_config,
                root_resource,
-                self.version_string,
            )
        )

@@ -159,6 +158,7 @@ def start(config_options):
    )

    ss.setup()
+    ss.get_handlers()
    ss.start_listening(config.worker_listeners)

    def start():
@@ -32,12 +32,13 @@ from synapse.replication.tcp.client import ReplicationClientHandler
 from synapse.server import HomeServer
 from synapse.storage import DataStore
 from synapse.storage.engines import create_engine
+from synapse.storage.roommember import RoomMemberStore
 from synapse.util.httpresourcetree import create_resource_tree
-from synapse.util.logcontext import LoggingContext, run_in_background
+from synapse.util.logcontext import LoggingContext, preserve_fn
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
 from twisted.internet import defer, reactor
-from twisted.web.resource import NoResource
+from twisted.web.resource import Resource

 logger = logging.getLogger("synapse.app.pusher")

@@ -74,6 +75,10 @@ class PusherSlaveStore(
        DataStore.get_profile_displayname.__func__
    )

+    who_forgot_in_room = (
+        RoomMemberStore.__dict__["who_forgot_in_room"]
+    )
+

 class PusherServer(HomeServer):
    def setup(self):
@@ -94,7 +99,7 @@ class PusherServer(HomeServer):
                if name == "metrics":
                    resources[METRICS_PREFIX] = MetricsResource(self)

-        root_resource = create_resource_tree(resources, NoResource())
+        root_resource = create_resource_tree(resources, Resource())

        _base.listen_tcp(
            bind_addresses,
@@ -104,7 +109,6 @@ class PusherServer(HomeServer):
                site_tag,
                listener_config,
                root_resource,
-                self.version_string,
            )
        )

@@ -141,27 +145,24 @@ class PusherReplicationHandler(ReplicationClientHandler):

    def on_rdata(self, stream_name, token, rows):
        super(PusherReplicationHandler, self).on_rdata(stream_name, token, rows)
-        run_in_background(self.poke_pushers, stream_name, token, rows)
+        preserve_fn(self.poke_pushers)(stream_name, token, rows)

    @defer.inlineCallbacks
    def poke_pushers(self, stream_name, token, rows):
-        try:
-            if stream_name == "pushers":
-                for row in rows:
-                    if row.deleted:
-                        yield self.stop_pusher(row.user_id, row.app_id, row.pushkey)
-                    else:
-                        yield self.start_pusher(row.user_id, row.app_id, row.pushkey)
-            elif stream_name == "events":
-                yield self.pusher_pool.on_new_notifications(
-                    token, token,
-                )
-            elif stream_name == "receipts":
-                yield self.pusher_pool.on_new_receipts(
-                    token, token, set(row.room_id for row in rows)
-                )
-        except Exception:
-            logger.exception("Error poking pushers")
+        if stream_name == "pushers":
+            for row in rows:
+                if row.deleted:
+                    yield self.stop_pusher(row.user_id, row.app_id, row.pushkey)
+                else:
+                    yield self.start_pusher(row.user_id, row.app_id, row.pushkey)
+        elif stream_name == "events":
+            yield self.pusher_pool.on_new_notifications(
+                token, token,
+            )
+        elif stream_name == "receipts":
+            yield self.pusher_pool.on_new_receipts(
+                token, token, set(row.room_id for row in rows)
+            )

    def stop_pusher(self, user_id, app_id, pushkey):
        key = "%s:%s" % (app_id, pushkey)
@@ -51,19 +51,19 @@ from synapse.storage.engines import create_engine
 from synapse.storage.presence import UserPresenceState
 from synapse.storage.roommember import RoomMemberStore
 from synapse.util.httpresourcetree import create_resource_tree
-from synapse.util.logcontext import LoggingContext, run_in_background
+from synapse.util.logcontext import LoggingContext, preserve_fn
 from synapse.util.manhole import manhole
 from synapse.util.stringutils import random_string
 from synapse.util.versionstring import get_version_string
 from twisted.internet import defer, reactor
-from twisted.web.resource import NoResource
-
-from six import iteritems
+from twisted.web.resource import Resource

 logger = logging.getLogger("synapse.app.synchrotron")


 class SynchrotronSlavedStore(
+    SlavedPushRuleStore,
+    SlavedEventStore,
    SlavedReceiptsStore,
    SlavedAccountDataStore,
    SlavedApplicationServiceStore,
@@ -73,12 +73,14 @@ class SynchrotronSlavedStore(
    SlavedGroupServerStore,
    SlavedDeviceInboxStore,
    SlavedDeviceStore,
-    SlavedPushRuleStore,
-    SlavedEventStore,
    SlavedClientIpStore,
    RoomStore,
    BaseSlavedStore,
 ):
+    who_forgot_in_room = (
+        RoomMemberStore.__dict__["who_forgot_in_room"]
+    )
+
    did_forget = (
        RoomMemberStore.__dict__["did_forget"]
    )
@@ -115,6 +117,7 @@ class SynchrotronPresence(object):
        logger.info("Presence process_id is %r", self.process_id)

    def send_user_sync(self, user_id, is_syncing, last_sync_ms):
+        return
        self.hs.get_tcp_replication().send_user_sync(user_id, is_syncing, last_sync_ms)

    def mark_as_coming_online(self, user_id):
@@ -212,8 +215,10 @@ class SynchrotronPresence(object):
        yield self.notify_from_replication(states, stream_id)

    def get_currently_syncing_users(self):
+        # presence is disabled on matrix.org, so we return the empty set
+        return set()
        return [
-            user_id for user_id, count in iteritems(self.user_to_num_current_syncs)
+            user_id for user_id, count in self.user_to_num_current_syncs.iteritems()
            if count > 0
        ]

@@ -271,7 +276,7 @@ class SynchrotronServer(HomeServer):
                        "/_matrix/client/api/v1": resource,
                    })

-        root_resource = create_resource_tree(resources, NoResource())
+        root_resource = create_resource_tree(resources, Resource())

        _base.listen_tcp(
            bind_addresses,
@@ -281,7 +286,6 @@ class SynchrotronServer(HomeServer):
                site_tag,
                listener_config,
                root_resource,
-                self.version_string,
            )
        )

@@ -328,7 +332,8 @@ class SyncReplicationHandler(ReplicationClientHandler):

    def on_rdata(self, stream_name, token, rows):
        super(SyncReplicationHandler, self).on_rdata(stream_name, token, rows)
-        run_in_background(self.process_and_notify, stream_name, token, rows)
+
+        preserve_fn(self.process_and_notify)(stream_name, token, rows)

    def get_streams_to_replicate(self):
        args = super(SyncReplicationHandler, self).get_streams_to_replicate()
@@ -340,58 +345,55 @@ class SyncReplicationHandler(ReplicationClientHandler):

    @defer.inlineCallbacks
    def process_and_notify(self, stream_name, token, rows):
-        try:
-            if stream_name == "events":
-                # We shouldn't get multiple rows per token for events stream, so
-                # we don't need to optimise this for multiple rows.
-                for row in rows:
-                    event = yield self.store.get_event(row.event_id)
-                    extra_users = ()
-                    if event.type == EventTypes.Member:
-                        extra_users = (event.state_key,)
-                    max_token = self.store.get_room_max_stream_ordering()
-                    self.notifier.on_new_room_event(
-                        event, token, max_token, extra_users
-                    )
-            elif stream_name == "push_rules":
-                self.notifier.on_new_event(
-                    "push_rules_key", token, users=[row.user_id for row in rows],
+        if stream_name == "events":
+            # We shouldn't get multiple rows per token for events stream, so
+            # we don't need to optimise this for multiple rows.
+            for row in rows:
+                event = yield self.store.get_event(row.event_id)
+                extra_users = ()
+                if event.type == EventTypes.Member:
+                    extra_users = (event.state_key,)
+                max_token = self.store.get_room_max_stream_ordering()
+                self.notifier.on_new_room_event(
+                    event, token, max_token, extra_users
                )
-            elif stream_name in ("account_data", "tag_account_data",):
+        elif stream_name == "push_rules":
+            self.notifier.on_new_event(
+                "push_rules_key", token, users=[row.user_id for row in rows],
+            )
+        elif stream_name in ("account_data", "tag_account_data",):
+            self.notifier.on_new_event(
+                "account_data_key", token, users=[row.user_id for row in rows],
+            )
+        elif stream_name == "receipts":
+            self.notifier.on_new_event(
+                "receipt_key", token, rooms=[row.room_id for row in rows],
+            )
+        elif stream_name == "typing":
+            self.typing_handler.process_replication_rows(token, rows)
+            self.notifier.on_new_event(
+                "typing_key", token, rooms=[row.room_id for row in rows],
+            )
+        elif stream_name == "to_device":
+            entities = [row.entity for row in rows if row.entity.startswith("@")]
+            if entities:
                self.notifier.on_new_event(
-                    "account_data_key", token, users=[row.user_id for row in rows],
+                    "to_device_key", token, users=entities,
                )
-            elif stream_name == "receipts":
-                self.notifier.on_new_event(
-                    "receipt_key", token, rooms=[row.room_id for row in rows],
-                )
-            elif stream_name == "typing":
-                self.typing_handler.process_replication_rows(token, rows)
-                self.notifier.on_new_event(
-                    "typing_key", token, rooms=[row.room_id for row in rows],
-                )
-            elif stream_name == "to_device":
-                entities = [row.entity for row in rows if row.entity.startswith("@")]
-                if entities:
-                    self.notifier.on_new_event(
-                        "to_device_key", token, users=entities,
-                    )
-            elif stream_name == "device_lists":
-                all_room_ids = set()
-                for row in rows:
-                    room_ids = yield self.store.get_rooms_for_user(row.user_id)
-                    all_room_ids.update(room_ids)
-                self.notifier.on_new_event(
-                    "device_list_key", token, rooms=all_room_ids,
-                )
-            elif stream_name == "presence":
-                yield self.presence_handler.process_replication_rows(token, rows)
-            elif stream_name == "receipts":
-                self.notifier.on_new_event(
-                    "groups_key", token, users=[row.user_id for row in rows],
-                )
-        except Exception:
-            logger.exception("Error processing replication")
+        elif stream_name == "device_lists":
+            all_room_ids = set()
+            for row in rows:
+                room_ids = yield self.store.get_rooms_for_user(row.user_id)
+                all_room_ids.update(room_ids)
+            self.notifier.on_new_event(
+                "device_list_key", token, rooms=all_room_ids,
+            )
+        elif stream_name == "presence":
+            yield self.presence_handler.process_replication_rows(token, rows)
+        elif stream_name == "receipts":
+            self.notifier.on_new_event(
+                "groups_key", token, users=[row.user_id for row in rows],
+            )


 def start(config_options):
@@ -38,7 +38,7 @@ def pid_running(pid):
    try:
        os.kill(pid, 0)
        return True
-    except OSError as err:
+    except OSError, err:
        if err.errno == errno.EPERM:
            return True
        return False
@@ -98,7 +98,7 @@ def stop(pidfile, app):
        try:
            os.kill(pid, signal.SIGTERM)
            write("stopped %s" % (app,), colour=GREEN)
-        except OSError as err:
+        except OSError, err:
            if err.errno == errno.ESRCH:
                write("%s not running" % (app,), colour=YELLOW)
            elif err.errno == errno.EPERM:
@@ -252,7 +252,6 @@ def main():
            for running_pid in running_pids:
                while pid_running(running_pid):
                    time.sleep(0.2)
-            write("All processes exited; now restarting...")

    if action == "start" or action == "restart":
        if start_stop_synapse:
@@ -39,11 +39,11 @@ from synapse.storage.engines import create_engine
 from synapse.storage.user_directory import UserDirectoryStore
 from synapse.util.caches.stream_change_cache import StreamChangeCache
 from synapse.util.httpresourcetree import create_resource_tree
-from synapse.util.logcontext import LoggingContext, run_in_background
+from synapse.util.logcontext import LoggingContext, preserve_fn
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
-from twisted.internet import reactor, defer
-from twisted.web.resource import NoResource
+from twisted.internet import reactor
+from twisted.web.resource import Resource

 logger = logging.getLogger("synapse.app.user_dir")

@@ -116,7 +116,7 @@ class UserDirectoryServer(HomeServer):
                        "/_matrix/client/api/v1": resource,
                    })

-        root_resource = create_resource_tree(resources, NoResource())
+        root_resource = create_resource_tree(resources, Resource())

        _base.listen_tcp(
            bind_addresses,
@@ -126,7 +126,6 @@ class UserDirectoryServer(HomeServer):
                site_tag,
                listener_config,
                root_resource,
-                self.version_string,
            )
        )

@@ -165,14 +164,7 @@ class UserDirectoryReplicationHandler(ReplicationClientHandler):
            stream_name, token, rows
        )
        if stream_name == "current_state_deltas":
-            run_in_background(self._notify_directory)
-
-    @defer.inlineCallbacks
-    def _notify_directory(self):
-        try:
-            yield self.user_directory.notify_new_event()
-        except Exception:
-            logger.exception("Error notifiying user directory of state update")
+            preserve_fn(self.user_directory.notify_new_event)()


 def start(config_options):
@@ -21,8 +21,6 @@ from twisted.internet import defer
 import logging
 import re

-from six import string_types
-
 logger = logging.getLogger(__name__)


@@ -148,7 +146,7 @@ class ApplicationService(object):
                        )

                regex = regex_obj.get("regex")
-                if isinstance(regex, string_types):
+                if isinstance(regex, basestring):
                    regex_obj["regex"] = re.compile(regex)  # Pre-compile regex
                else:
                    raise ValueError(
@@ -18,6 +18,7 @@ from synapse.api.constants import ThirdPartyEntityKind
 from synapse.api.errors import CodeMessageException
 from synapse.http.client import SimpleHttpClient
 from synapse.events.utils import serialize_event
+from synapse.util.logcontext import preserve_fn, make_deferred_yieldable
 from synapse.util.caches.response_cache import ResponseCache
 from synapse.types import ThirdPartyInstanceID

@@ -72,8 +73,7 @@ class ApplicationServiceApi(SimpleHttpClient):
        super(ApplicationServiceApi, self).__init__(hs)
        self.clock = hs.get_clock()

-        self.protocol_meta_cache = ResponseCache(hs, "as_protocol_meta",
-                                                 timeout_ms=HOUR_IN_MS)
+        self.protocol_meta_cache = ResponseCache(hs, timeout_ms=HOUR_IN_MS)

    @defer.inlineCallbacks
    def query_user(self, service, user_id):
@@ -193,7 +193,12 @@ class ApplicationServiceApi(SimpleHttpClient):
                defer.returnValue(None)

        key = (service.id, protocol)
-        return self.protocol_meta_cache.wrap(key, _get)
+        result = self.protocol_meta_cache.get(key)
+        if not result:
+            result = self.protocol_meta_cache.set(
+                key, preserve_fn(_get)()
+            )
+        return make_deferred_yieldable(result)

    @defer.inlineCallbacks
    def push_bulk(self, service, events, txn_id=None):
@@ -51,7 +51,7 @@ components.
 from twisted.internet import defer

 from synapse.appservice import ApplicationServiceState
-from synapse.util.logcontext import run_in_background
+from synapse.util.logcontext import preserve_fn
 from synapse.util.metrics import Measure

 import logging
@@ -106,7 +106,7 @@ class _ServiceQueuer(object):
    def enqueue(self, service, event):
        # if this service isn't being sent something
        self.queued_events.setdefault(service.id, []).append(event)
-        run_in_background(self._send_request, service)
+        preserve_fn(self._send_request)(service)

    @defer.inlineCallbacks
    def _send_request(self, service):
@@ -152,10 +152,10 @@ class _TransactionController(object):
                if sent:
                    yield txn.complete(self.store)
                else:
-                    run_in_background(self._start_recoverer, service)
-        except Exception:
-            logger.exception("Error creating appservice transaction")
-            run_in_background(self._start_recoverer, service)
+                    preserve_fn(self._start_recoverer)(service)
+        except Exception as e:
+            logger.exception(e)
+            preserve_fn(self._start_recoverer)(service)

    @defer.inlineCallbacks
    def on_recovered(self, recoverer):
@@ -176,20 +176,17 @@ class _TransactionController(object):

    @defer.inlineCallbacks
    def _start_recoverer(self, service):
-        try:
-            yield self.store.set_appservice_state(
-                service,
-                ApplicationServiceState.DOWN
-            )
-            logger.info(
-                "Application service falling behind. Starting recoverer. AS ID %s",
-                service.id
-            )
-            recoverer = self.recoverer_fn(service, self.on_recovered)
-            self.add_recoverers([recoverer])
-            recoverer.recover()
-        except Exception:
-            logger.exception("Error starting AS recoverer")
+        yield self.store.set_appservice_state(
+            service,
+            ApplicationServiceState.DOWN
+        )
+        logger.info(
+            "Application service falling behind. Starting recoverer. AS ID %s",
+            service.id
+        )
+        recoverer = self.recoverer_fn(service, self.on_recovered)
+        self.add_recoverers([recoverer])
+        recoverer.recover()

    @defer.inlineCallbacks
    def _is_service_up(self, service):
@@ -19,8 +19,6 @@ import os
 import yaml
 from textwrap import dedent

-from six import integer_types
-

 class ConfigError(Exception):
    pass
@@ -51,7 +49,7 @@ Missing mandatory `server_name` config option.
 class Config(object):
    @staticmethod
    def parse_size(value):
-        if isinstance(value, integer_types):
+        if isinstance(value, int) or isinstance(value, long):
            return value
        sizes = {"K": 1024, "M": 1024 * 1024}
        size = 1
@@ -63,7 +61,7 @@ class Config(object):

    @staticmethod
    def parse_duration(value):
-        if isinstance(value, integer_types):
+        if isinstance(value, int) or isinstance(value, long):
            return value
        second = 1000
        minute = 60 * second
@@ -281,31 +279,31 @@ class Config(object):
                    )
                if not cls.path_exists(config_dir_path):
                    os.makedirs(config_dir_path)
-                with open(config_path, "w") as config_file:
-                    config_str, config = obj.generate_config(
+                with open(config_path, "wb") as config_file:
+                    config_bytes, config = obj.generate_config(
                        config_dir_path=config_dir_path,
                        server_name=server_name,
                        report_stats=(config_args.report_stats == "yes"),
                        is_generating_file=True
                    )
                    obj.invoke_all("generate_files", config)
-                    config_file.write(config_str)
-                print((
+                    config_file.write(config_bytes)
+                print (
                    "A config file has been generated in %r for server name"
                    " %r with corresponding SSL keys and self-signed"
                    " certificates. Please review this file and customise it"
                    " to your needs."
-                ) % (config_path, server_name))
-                print(
+                ) % (config_path, server_name)
+                print (
                    "If this server name is incorrect, you will need to"
                    " regenerate the SSL certificates"
                )
                return
            else:
-                print((
+                print (
                    "Config file %r already exists. Generating any missing key"
                    " files."
-                ) % (config_path,))
+                ) % (config_path,)
                generate_keys = True

        parser = argparse.ArgumentParser(
@@ -17,12 +17,10 @@ from ._base import Config, ConfigError
 from synapse.appservice import ApplicationService
 from synapse.types import UserID

+import urllib
 import yaml
 import logging

-from six import string_types
-from six.moves.urllib import parse as urlparse
-
 logger = logging.getLogger(__name__)


@@ -91,21 +89,21 @@ def _load_appservice(hostname, as_info, config_filename):
        "id", "as_token", "hs_token", "sender_localpart"
    ]
    for field in required_string_fields:
-        if not isinstance(as_info.get(field), string_types):
+        if not isinstance(as_info.get(field), basestring):
            raise KeyError("Required string field: '%s' (%s)" % (
                field, config_filename,
            ))

    # 'url' must either be a string or explicitly null, not missing
    # to avoid accidentally turning off push for ASes.
-    if (not isinstance(as_info.get("url"), string_types) and
+    if (not isinstance(as_info.get("url"), basestring) and
            as_info.get("url", "") is not None):
        raise KeyError(
            "Required string field or explicit null: 'url' (%s)" % (config_filename,)
        )

    localpart = as_info["sender_localpart"]
-    if urlparse.quote(localpart) != localpart:
+    if urllib.quote(localpart) != localpart:
        raise ValueError(
            "sender_localpart needs characters which are not URL encoded."
        )
@@ -130,7 +128,7 @@ def _load_appservice(hostname, as_info, config_filename):
                        "Expected namespace entry in %s to be an object,"
                        " but got %s", ns, regex_obj
                    )
-                if not isinstance(regex_obj.get("regex"), string_types):
+                if not isinstance(regex_obj.get("regex"), basestring):
                    raise ValueError(
                        "Missing/bad type 'regex' key in %s", regex_obj
                    )
@@ -117,7 +117,7 @@ class LoggingConfig(Config):
        log_config = config.get("log_config")
        if log_config and not os.path.exists(log_config):
            log_file = self.abspath("homeserver.log")
-            with open(log_config, "w") as log_config_file:
+            with open(log_config, "wb") as log_config_file:
                log_config_file.write(
                    DEFAULT_LOG_CONFIG.substitute(log_file=log_file)
                )
@@ -77,9 +77,7 @@ class RegistrationConfig(Config):

        # Set the number of bcrypt rounds used to generate password hash.
        # Larger numbers increase the work factor needed to generate the hash.
-        # The default number is 12 (which equates to 2^12 rounds).
-        # N.B. that increasing this will exponentially increase the time required
-        # to register or login - e.g. 24 => 2^24 rounds which will take >20 mins.
+        # The default number of rounds is 12.
        bcrypt_rounds: 12

        # Allows users to register as guests without a password/email/etc, and
@@ -133,7 +133,7 @@ class TlsConfig(Config):
        tls_dh_params_path = config["tls_dh_params_path"]

        if not self.path_exists(tls_private_key_path):
-            with open(tls_private_key_path, "wb") as private_key_file:
+            with open(tls_private_key_path, "w") as private_key_file:
                tls_private_key = crypto.PKey()
                tls_private_key.generate_key(crypto.TYPE_RSA, 2048)
                private_key_pem = crypto.dump_privatekey(
@@ -148,7 +148,7 @@ class TlsConfig(Config):
                )

        if not self.path_exists(tls_certificate_path):
-            with open(tls_certificate_path, "wb") as certificate_file:
+            with open(tls_certificate_path, "w") as certificate_file:
                cert = crypto.X509()
                subject = cert.get_subject()
                subject.CN = config["server_name"]
@@ -13,8 +13,8 @@
 # limitations under the License.

 from twisted.internet import ssl
-from OpenSSL import SSL, crypto
-from twisted.internet._sslverify import _defaultCurveName
+from OpenSSL import SSL
+from twisted.internet._sslverify import _OpenSSLECCurve, _defaultCurveName

 import logging

@@ -32,9 +32,8 @@ class ServerContextFactory(ssl.ContextFactory):
    @staticmethod
    def configure_context(context, config):
        try:
-            _ecCurve = crypto.get_elliptic_curve(_defaultCurveName)
-            context.set_tmp_ecdh(_ecCurve)
-
+            _ecCurve = _OpenSSLECCurve(_defaultCurveName)
+            _ecCurve.addECKeyToContext(context)
        except Exception:
            logger.exception("Failed to enable elliptic curve for TLS")
        context.set_options(SSL.OP_NO_SSLv2 | SSL.OP_NO_SSLv3)
@@ -19,8 +19,7 @@ from synapse.api.errors import SynapseError, Codes
 from synapse.util import unwrapFirstError, logcontext
 from synapse.util.logcontext import (
    PreserveLoggingContext,
-    preserve_fn,
-    run_in_background,
+    preserve_fn
 )
 from synapse.util.metrics import Measure

@@ -128,7 +127,7 @@ class Keyring(object):

            verify_requests.append(verify_request)

-        run_in_background(self._start_key_lookups, verify_requests)
+        preserve_fn(self._start_key_lookups)(verify_requests)

        # Pass those keys to handle_key_deferred so that the json object
        # signatures can be verified
@@ -147,56 +146,53 @@ class Keyring(object):
            verify_requests (List[VerifyKeyRequest]):
        """

-        try:
-            # create a deferred for each server we're going to look up the keys
-            # for; we'll resolve them once we have completed our lookups.
-            # These will be passed into wait_for_previous_lookups to block
-            # any other lookups until we have finished.
-            # The deferreds are called with no logcontext.
-            server_to_deferred = {
-                rq.server_name: defer.Deferred()
-                for rq in verify_requests
-            }
+        # create a deferred for each server we're going to look up the keys
+        # for; we'll resolve them once we have completed our lookups.
+        # These will be passed into wait_for_previous_lookups to block
+        # any other lookups until we have finished.
+        # The deferreds are called with no logcontext.
+        server_to_deferred = {
+            rq.server_name: defer.Deferred()
+            for rq in verify_requests
+        }

-            # We want to wait for any previous lookups to complete before
-            # proceeding.
-            yield self.wait_for_previous_lookups(
-                [rq.server_name for rq in verify_requests],
-                server_to_deferred,
+        # We want to wait for any previous lookups to complete before
+        # proceeding.
+        yield self.wait_for_previous_lookups(
+            [rq.server_name for rq in verify_requests],
+            server_to_deferred,
+        )
+
+        # Actually start fetching keys.
+        self._get_server_verify_keys(verify_requests)
+
+        # When we've finished fetching all the keys for a given server_name,
+        # resolve the deferred passed to `wait_for_previous_lookups` so that
+        # any lookups waiting will proceed.
+        #
+        # map from server name to a set of request ids
+        server_to_request_ids = {}
+
+        for verify_request in verify_requests:
+            server_name = verify_request.server_name
+            request_id = id(verify_request)
+            server_to_request_ids.setdefault(server_name, set()).add(request_id)
+
+        def remove_deferreds(res, verify_request):
+            server_name = verify_request.server_name
+            request_id = id(verify_request)
+            server_to_request_ids[server_name].discard(request_id)
+            if not server_to_request_ids[server_name]:
+                d = server_to_deferred.pop(server_name, None)
+                if d:
+                    d.callback(None)
+            return res
+
+        for verify_request in verify_requests:
+            verify_request.deferred.addBoth(
+                remove_deferreds, verify_request,
            )

-            # Actually start fetching keys.
-            self._get_server_verify_keys(verify_requests)
-
-            # When we've finished fetching all the keys for a given server_name,
-            # resolve the deferred passed to `wait_for_previous_lookups` so that
-            # any lookups waiting will proceed.
-            #
-            # map from server name to a set of request ids
-            server_to_request_ids = {}
-
-            for verify_request in verify_requests:
-                server_name = verify_request.server_name
-                request_id = id(verify_request)
-                server_to_request_ids.setdefault(server_name, set()).add(request_id)
-
-            def remove_deferreds(res, verify_request):
-                server_name = verify_request.server_name
-                request_id = id(verify_request)
-                server_to_request_ids[server_name].discard(request_id)
-                if not server_to_request_ids[server_name]:
-                    d = server_to_deferred.pop(server_name, None)
-                    if d:
-                        d.callback(None)
-                return res
-
-            for verify_request in verify_requests:
-                verify_request.deferred.addBoth(
-                    remove_deferreds, verify_request,
-                )
-        except Exception:
-            logger.exception("Error starting key lookups")
-
    @defer.inlineCallbacks
    def wait_for_previous_lookups(self, server_names, server_to_deferred):
        """Waits for any previous key lookups for the given servers to finish.
@@ -317,7 +313,7 @@ class Keyring(object):
                    if not verify_request.deferred.called:
                        verify_request.deferred.errback(err)

-        run_in_background(do_iterations).addErrback(on_err)
+        preserve_fn(do_iterations)().addErrback(on_err)

    @defer.inlineCallbacks
    def get_keys_from_store(self, server_name_and_key_ids):
@@ -333,9 +329,8 @@ class Keyring(object):
        """
        res = yield logcontext.make_deferred_yieldable(defer.gatherResults(
            [
-                run_in_background(
-                    self.store.get_server_verify_keys,
-                    server_name, key_ids,
+                preserve_fn(self.store.get_server_verify_keys)(
+                    server_name, key_ids
                ).addCallback(lambda ks, server: (server, ks), server_name)
                for server_name, key_ids in server_name_and_key_ids
            ],
@@ -357,13 +352,13 @@ class Keyring(object):
                logger.exception(
                    "Unable to get key from %r: %s %s",
                    perspective_name,
-                    type(e).__name__, str(e),
+                    type(e).__name__, str(e.message),
                )
                defer.returnValue({})

        results = yield logcontext.make_deferred_yieldable(defer.gatherResults(
            [
-                run_in_background(get_key, p_name, p_keys)
+                preserve_fn(get_key)(p_name, p_keys)
                for p_name, p_keys in self.perspective_servers.items()
            ],
            consumeErrors=True,
@@ -389,7 +384,7 @@ class Keyring(object):
                logger.info(
                    "Unable to get key %r for %r directly: %s %s",
                    key_ids, server_name,
-                    type(e).__name__, str(e),
+                    type(e).__name__, str(e.message),
                )

            if not keys:
@@ -403,7 +398,7 @@ class Keyring(object):

        results = yield logcontext.make_deferred_yieldable(defer.gatherResults(
            [
-                run_in_background(get_key, server_name, key_ids)
+                preserve_fn(get_key)(server_name, key_ids)
                for server_name, key_ids in server_name_and_key_ids
            ],
            consumeErrors=True,
@@ -486,8 +481,7 @@ class Keyring(object):

        yield logcontext.make_deferred_yieldable(defer.gatherResults(
            [
-                run_in_background(
-                    self.store_keys,
+                preserve_fn(self.store_keys)(
                    server_name=server_name,
                    from_server=perspective_name,
                    verify_keys=response_keys,
@@ -545,8 +539,7 @@ class Keyring(object):

        yield logcontext.make_deferred_yieldable(defer.gatherResults(
            [
-                run_in_background(
-                    self.store_keys,
+                preserve_fn(self.store_keys)(
                    server_name=key_server_name,
                    from_server=server_name,
                    verify_keys=verify_keys,
@@ -622,8 +615,7 @@ class Keyring(object):

        yield logcontext.make_deferred_yieldable(defer.gatherResults(
            [
-                run_in_background(
-                    self.store.store_server_keys_json,
+                preserve_fn(self.store.store_server_keys_json)(
                    server_name=server_name,
                    key_id=key_id,
                    from_server=server_name,
@@ -724,8 +716,7 @@ class Keyring(object):
        # TODO(markjh): Store whether the keys have expired.
        return logcontext.make_deferred_yieldable(defer.gatherResults(
            [
-                run_in_background(
-                    self.store.store_server_verify_key,
+                preserve_fn(self.store.store_server_verify_key)(
                    server_name, server_name, key.time_added, key
                )
                for key_id, key in verify_keys.items()
@@ -743,7 +734,7 @@ def _handle_key_deferred(verify_request):
    except IOError as e:
        logger.warn(
            "Got IOError when downloading keys for %s: %s %s",
-            server_name, type(e).__name__, str(e),
+            server_name, type(e).__name__, str(e.message),
        )
        raise SynapseError(
            502,
@@ -753,7 +744,7 @@ def _handle_key_deferred(verify_request):
    except Exception as e:
        logger.exception(
            "Got Exception when downloading keys for %s: %s %s",
-            server_name, type(e).__name__, str(e),
+            server_name, type(e).__name__, str(e.message),
        )
        raise SynapseError(
            401,
@@ -47,26 +47,14 @@ class _EventInternalMetadata(object):


 def _event_dict_property(key):
-    # We want to be able to use hasattr with the event dict properties.
-    # However, (on python3) hasattr expects AttributeError to be raised. Hence,
-    # we need to transform the KeyError into an AttributeError
    def getter(self):
-        try:
-            return self._event_dict[key]
-        except KeyError:
-            raise AttributeError(key)
+        return self._event_dict[key]

    def setter(self, v):
-        try:
-            self._event_dict[key] = v
-        except KeyError:
-            raise AttributeError(key)
+        self._event_dict[key] = v

    def delete(self):
-        try:
-            del self._event_dict[key]
-        except KeyError:
-            raise AttributeError(key)
+        del self._event_dict[key]

    return property(
        getter,
@@ -13,7 +13,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-from twisted.internet import defer

 from frozendict import frozendict

@@ -52,6 +51,7 @@ class EventContext(object):
        "prev_state_ids",
        "state_group",
        "rejected",
+        "push_actions",
        "prev_group",
        "delta_ids",
        "prev_state_events",
@@ -66,6 +66,7 @@ class EventContext(object):
        self.state_group = None

        self.rejected = False
+        self.push_actions = []

        # A previously persisted state group and a delta between that
        # and this state.
@@ -76,32 +77,19 @@ class EventContext(object):

        self.app_service = None

-    def serialize(self, event):
+    def serialize(self):
        """Converts self to a type that can be serialized as JSON, and then
        deserialized by `deserialize`

-        Args:
-            event (FrozenEvent): The event that this context relates to
-
        Returns:
            dict
        """
-
-        # We don't serialize the full state dicts, instead they get pulled out
-        # of the DB on the other side. However, the other side can't figure out
-        # the prev_state_ids, so if we're a state event we include the event
-        # id that we replaced in the state.
-        if event.is_state():
-            prev_state_id = self.prev_state_ids.get((event.type, event.state_key))
-        else:
-            prev_state_id = None
-
        return {
-            "prev_state_id": prev_state_id,
-            "event_type": event.type,
-            "event_state_key": event.state_key if event.is_state() else None,
+            "current_state_ids": _encode_state_dict(self.current_state_ids),
+            "prev_state_ids": _encode_state_dict(self.prev_state_ids),
            "state_group": self.state_group,
            "rejected": self.rejected,
+            "push_actions": self.push_actions,
            "prev_group": self.prev_group,
            "delta_ids": _encode_state_dict(self.delta_ids),
            "prev_state_events": self.prev_state_events,
@@ -109,7 +97,6 @@ class EventContext(object):
        }

    @staticmethod
-    @defer.inlineCallbacks
    def deserialize(store, input):
        """Converts a dict that was produced by `serialize` back into a
        EventContext.
@@ -122,32 +109,20 @@ class EventContext(object):
            EventContext
        """
        context = EventContext()
+        context.current_state_ids = _decode_state_dict(input["current_state_ids"])
+        context.prev_state_ids = _decode_state_dict(input["prev_state_ids"])
        context.state_group = input["state_group"]
        context.rejected = input["rejected"]
+        context.push_actions = input["push_actions"]
        context.prev_group = input["prev_group"]
        context.delta_ids = _decode_state_dict(input["delta_ids"])
        context.prev_state_events = input["prev_state_events"]

-        # We use the state_group and prev_state_id stuff to pull the
-        # current_state_ids out of the DB and construct prev_state_ids.
-        prev_state_id = input["prev_state_id"]
-        event_type = input["event_type"]
-        event_state_key = input["event_state_key"]
-
-        context.current_state_ids = yield store.get_state_ids_for_group(
-            context.state_group,
-        )
-        if prev_state_id and event_state_key:
-            context.prev_state_ids = dict(context.current_state_ids)
-            context.prev_state_ids[(event_type, event_state_key)] = prev_state_id
-        else:
-            context.prev_state_ids = context.current_state_ids
-
        app_service_id = input["app_service_id"]
        if app_service_id:
            context.app_service = store.get_app_service_by_id(app_service_id)

-        defer.returnValue(context)
+        return context


 def _encode_state_dict(state_dict):
@@ -15,3 +15,11 @@

 """ This package includes all the federation specific logic.
 """
+
+from .replication import ReplicationLayer
+
+
+def initialize_http_replication(hs):
+    transport = hs.get_federation_transport_client()
+
+    return ReplicationLayer(hs, transport)
@@ -14,10 +14,7 @@
 # limitations under the License.
 import logging

-import six
-
-from synapse.api.constants import MAX_DEPTH
-from synapse.api.errors import SynapseError, Codes
+from synapse.api.errors import SynapseError
 from synapse.crypto.event_signing import check_event_content_hash
 from synapse.events import FrozenEvent
 from synapse.events.utils import prune_event
@@ -30,13 +27,7 @@ logger = logging.getLogger(__name__)

 class FederationBase(object):
    def __init__(self, hs):
-        self.hs = hs
-
-        self.server_name = hs.hostname
-        self.keyring = hs.get_keyring()
        self.spam_checker = hs.get_spam_checker()
-        self.store = hs.get_datastore()
-        self._clock = hs.get_clock()

    @defer.inlineCallbacks
    def _check_sigs_and_hash_and_fetch(self, origin, pdus, outlier=False,
@@ -193,23 +184,11 @@ def event_from_pdu_json(pdu_json, outlier=False):
        FrozenEvent

    Raises:
-        SynapseError: if the pdu is missing required fields or is otherwise
-            not a valid matrix event
+        SynapseError: if the pdu is missing required fields
    """
    # we could probably enforce a bunch of other fields here (room_id, sender,
    # origin, etc etc)
-    assert_params_in_request(pdu_json, ('event_id', 'type', 'depth'))
-
-    depth = pdu_json['depth']
-    if not isinstance(depth, six.integer_types):
-        raise SynapseError(400, "Depth %r not an intger" % (depth, ),
-                           Codes.BAD_JSON)
-
-    if depth < 0:
-        raise SynapseError(400, "Depth too small", Codes.BAD_JSON)
-    elif depth > MAX_DEPTH:
-        raise SynapseError(400, "Depth too large", Codes.BAD_JSON)
-
+    assert_params_in_request(pdu_json, ('event_id', 'type'))
    event = FrozenEvent(
        pdu_json
    )
@@ -19,8 +19,6 @@ import itertools
 import logging
 import random

-from six.moves import range
-
 from twisted.internet import defer

 from synapse.api.constants import Membership
@@ -35,7 +33,7 @@ from synapse.federation.federation_base import (
 import synapse.metrics
 from synapse.util import logcontext, unwrapFirstError
 from synapse.util.caches.expiringcache import ExpiringCache
-from synapse.util.logcontext import make_deferred_yieldable, run_in_background
+from synapse.util.logcontext import make_deferred_yieldable, preserve_fn
 from synapse.util.logutils import log_function
 from synapse.util.retryutils import NotRetryingDestination

@@ -60,7 +58,6 @@ class FederationClient(FederationBase):
            self._clear_tried_cache, 60 * 1000,
        )
        self.state = hs.get_state_handler()
-        self.transport_layer = hs.get_federation_transport_client()

    def _clear_tried_cache(self):
        """Clear pdu_destination_tried cache"""
@@ -396,7 +393,7 @@ class FederationClient(FederationBase):
            seen_events = yield self.store.get_events(event_ids, allow_rejected=True)
            signed_events = seen_events.values()
        else:
-            seen_events = yield self.store.have_seen_events(event_ids)
+            seen_events = yield self.store.have_events(event_ids)
            signed_events = []

        failed_to_fetch = set()
@@ -415,12 +412,11 @@ class FederationClient(FederationBase):

        batch_size = 20
        missing_events = list(missing_events)
-        for i in range(0, len(missing_events), batch_size):
+        for i in xrange(0, len(missing_events), batch_size):
            batch = set(missing_events[i:i + batch_size])

            deferreds = [
-                run_in_background(
-                    self.get_pdu,
+                preserve_fn(self.get_pdu)(
                    destinations=random_server_list(),
                    event_id=e_id,
                )
@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2015, 2016 OpenMarket Ltd
-# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -18,23 +17,20 @@ import logging
 import simplejson as json
 from twisted.internet import defer

-from synapse.api.errors import AuthError, FederationError, SynapseError, NotFoundError
+from synapse.api.errors import AuthError, FederationError, SynapseError
 from synapse.crypto.event_signing import compute_event_signature
 from synapse.federation.federation_base import (
    FederationBase,
    event_from_pdu_json,
 )
-
-from synapse.federation.persistence import TransactionActions
 from synapse.federation.units import Edu, Transaction
 import synapse.metrics
 from synapse.types import get_domain_from_id
 from synapse.util import async
 from synapse.util.caches.response_cache import ResponseCache
+from synapse.util.logcontext import make_deferred_yieldable, preserve_fn
 from synapse.util.logutils import log_function

-from six import iteritems
-
 # when processing incoming transactions, we try to handle multiple rooms in
 # parallel, up to this limit.
 TRANSACTION_CONCURRENCY_LIMIT = 10
@@ -56,18 +52,49 @@ class FederationServer(FederationBase):
        super(FederationServer, self).__init__(hs)

        self.auth = hs.get_auth()
-        self.handler = hs.get_handlers().federation_handler

        self._server_linearizer = async.Linearizer("fed_server")
        self._transaction_linearizer = async.Linearizer("fed_txn_handler")

-        self.transaction_actions = TransactionActions(self.store)
-
-        self.registry = hs.get_federation_registry()
-
        # We cache responses to state queries, as they take a while and often
        # come in waves.
-        self._state_resp_cache = ResponseCache(hs, "state_resp", timeout_ms=30000)
+        self._state_resp_cache = ResponseCache(hs, timeout_ms=30000)
+
+    def set_handler(self, handler):
+        """Sets the handler that the replication layer will use to communicate
+        receipt of new PDUs from other home servers. The required methods are
+        documented on :py:class:`.ReplicationHandler`.
+        """
+        self.handler = handler
+
+    def register_edu_handler(self, edu_type, handler):
+        if edu_type in self.edu_handlers:
+            raise KeyError("Already have an EDU handler for %s" % (edu_type,))
+
+        self.edu_handlers[edu_type] = handler
+
+    def register_query_handler(self, query_type, handler):
+        """Sets the handler callable that will be used to handle an incoming
+        federation Query of the given type.
+
+        Args:
+            query_type (str): Category name of the query, which should match
+                the string used by make_query.
+            handler (callable): Invoked to handle incoming queries of this type
+
+        handler is invoked as:
+            result = handler(args)
+
+        where 'args' is a dict mapping strings to strings of the query
+          arguments. It should return a Deferred that will eventually yield an
+          object to encode as JSON.
+        """
+        if query_type in self.query_handlers:
+            raise KeyError(
+                "Already have a Query handler for %s" % (query_type,)
+            )
+
+        self.query_handlers[query_type] = handler

    @defer.inlineCallbacks
    @log_function
@@ -202,7 +229,16 @@ class FederationServer(FederationBase):
    @defer.inlineCallbacks
    def received_edu(self, origin, edu_type, content):
        received_edus_counter.inc()
-        yield self.registry.on_edu(edu_type, origin, content)
+
+        if edu_type in self.edu_handlers:
+            try:
+                yield self.edu_handlers[edu_type](origin, content)
+            except SynapseError as e:
+                logger.info("Failed to handle edu %r: %r", edu_type, e)
+            except Exception as e:
+                logger.exception("Failed to handle edu %r", edu_type)
+        else:
+            logger.warn("Received EDU of type %s with no handler", edu_type)

    @defer.inlineCallbacks
    @log_function
@@ -214,17 +250,16 @@ class FederationServer(FederationBase):
        if not in_room:
            raise AuthError(403, "Host not in room.")

-        # we grab the linearizer to protect ourselves from servers which hammer
-        # us. In theory we might already have the response to this query
-        # in the cache so we could return it without waiting for the linearizer
-        # - but that's non-trivial to get right, and anyway somewhat defeats
-        # the point of the linearizer.
-        with (yield self._server_linearizer.queue((origin, room_id))):
-            resp = yield self._state_resp_cache.wrap(
-                (room_id, event_id),
-                self._on_context_state_request_compute,
-                room_id, event_id,
-            )
+        result = self._state_resp_cache.get((room_id, event_id))
+        if not result:
+            with (yield self._server_linearizer.queue((origin, room_id))):
+                d = self._state_resp_cache.set(
+                    (room_id, event_id),
+                    preserve_fn(self._on_context_state_request_compute)(room_id, event_id)
+                )
+                resp = yield make_deferred_yieldable(d)
+        else:
+            resp = yield make_deferred_yieldable(result)

        defer.returnValue((200, resp))

@@ -293,8 +328,14 @@ class FederationServer(FederationBase):
    @defer.inlineCallbacks
    def on_query_request(self, query_type, args):
        received_queries_counter.inc(query_type)
-        resp = yield self.registry.on_query(query_type, args)
-        defer.returnValue((200, resp))
+
+        if query_type in self.query_handlers:
+            response = yield self.query_handlers[query_type](args)
+            defer.returnValue((200, response))
+        else:
+            defer.returnValue(
+                (404, "No handler for Query type '%s'" % (query_type,))
+            )

    @defer.inlineCallbacks
    def on_make_join_request(self, room_id, user_id):
@@ -428,9 +469,9 @@ class FederationServer(FederationBase):
            "Claimed one-time-keys: %s",
            ",".join((
                "%s for %s:%s" % (key_id, user_id, device_id)
-                for user_id, user_keys in iteritems(json_result)
-                for device_id, device_keys in iteritems(user_keys)
-                for key_id, _ in iteritems(device_keys)
+                for user_id, user_keys in json_result.iteritems()
+                for device_id, device_keys in user_keys.iteritems()
+                for key_id, _ in device_keys.iteritems()
            )),
        )

@@ -497,33 +538,13 @@ class FederationServer(FederationBase):
    def _handle_received_pdu(self, origin, pdu):
        """ Process a PDU received in a federation /send/ transaction.

-        If the event is invalid, then this method throws a FederationError.
-        (The error will then be logged and sent back to the sender (which
-        probably won't do anything with it), and other events in the
-        transaction will be processed as normal).
-
-        It is likely that we'll then receive other events which refer to
-        this rejected_event in their prev_events, etc.  When that happens,
-        we'll attempt to fetch the rejected event again, which will presumably
-        fail, so those second-generation events will also get rejected.
-
-        Eventually, we get to the point where there are more than 10 events
-        between any new events and the original rejected event. Since we
-        only try to backfill 10 events deep on received pdu, we then accept the
-        new event, possibly introducing a discontinuity in the DAG, with new
-        forward extremities, so normal service is approximately returned,
-        until we try to backfill across the discontinuity.
-
        Args:
            origin (str): server which sent the pdu
            pdu (FrozenEvent): received pdu

        Returns (Deferred): completes with None
-
-        Raises: FederationError if the signatures / hash do not match, or
-            if the event was unacceptable for any other reason (eg, too large,
-            too many prev_events, couldn't find the prev_events)
-        """
+        Raises: FederationError if the signatures / hash do not match
+    """
        # check that it's actually being sent from a valid destination to
        # workaround bug #1753 in 0.18.5 and 0.18.6
        if origin != get_domain_from_id(pdu.event_id):
@@ -586,66 +607,3 @@ class FederationServer(FederationBase):
            origin, room_id, event_dict
        )
        defer.returnValue(ret)
-
-
-class FederationHandlerRegistry(object):
-    """Allows classes to register themselves as handlers for a given EDU or
-    query type for incoming federation traffic.
-    """
-    def __init__(self):
-        self.edu_handlers = {}
-        self.query_handlers = {}
-
-    def register_edu_handler(self, edu_type, handler):
-        """Sets the handler callable that will be used to handle an incoming
-        federation EDU of the given type.
-
-        Args:
-            edu_type (str): The type of the incoming EDU to register handler for
-            handler (Callable[[str, dict]]): A callable invoked on incoming EDU
-                of the given type. The arguments are the origin server name and
-                the EDU contents.
-        """
-        if edu_type in self.edu_handlers:
-            raise KeyError("Already have an EDU handler for %s" % (edu_type,))
-
-        self.edu_handlers[edu_type] = handler
-
-    def register_query_handler(self, query_type, handler):
-        """Sets the handler callable that will be used to handle an incoming
-        federation query of the given type.
-
-        Args:
-            query_type (str): Category name of the query, which should match
-                the string used by make_query.
-            handler (Callable[[dict], Deferred[dict]]): Invoked to handle
-                incoming queries of this type. The return will be yielded
-                on and the result used as the response to the query request.
-        """
-        if query_type in self.query_handlers:
-            raise KeyError(
-                "Already have a Query handler for %s" % (query_type,)
-            )
-
-        self.query_handlers[query_type] = handler
-
-    @defer.inlineCallbacks
-    def on_edu(self, edu_type, origin, content):
-        handler = self.edu_handlers.get(edu_type)
-        if not handler:
-            logger.warn("No handler registered for EDU type %s", edu_type)
-
-        try:
-            yield handler(origin, content)
-        except SynapseError as e:
-            logger.info("Failed to handle edu %r: %r", edu_type, e)
-        except Exception as e:
-            logger.exception("Failed to handle edu %r", edu_type)
-
-    def on_query(self, query_type, args):
-        handler = self.query_handlers.get(query_type)
-        if not handler:
-            logger.warn("No handler registered for query type %s", query_type)
-            raise NotFoundError("No handler for Query type '%s'" % (query_type,))
-
-        return handler(args)
@@ -0,0 +1,73 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014-2016 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""This layer is responsible for replicating with remote home servers using
+a given transport.
+"""
+
+from .federation_client import FederationClient
+from .federation_server import FederationServer
+
+from .persistence import TransactionActions
+
+import logging
+
+
+logger = logging.getLogger(__name__)
+
+
+class ReplicationLayer(FederationClient, FederationServer):
+    """This layer is responsible for replicating with remote home servers over
+    the given transport. I.e., does the sending and receiving of PDUs to
+    remote home servers.
+
+    The layer communicates with the rest of the server via a registered
+    ReplicationHandler.
+
+    In more detail, the layer:
+        * Receives incoming data and processes it into transactions and pdus.
+        * Fetches any PDUs it thinks it might have missed.
+        * Keeps the current state for contexts up to date by applying the
+          suitable conflict resolution.
+        * Sends outgoing pdus wrapped in transactions.
+        * Fills out the references to previous pdus/transactions appropriately
+          for outgoing data.
+    """
+
+    def __init__(self, hs, transport_layer):
+        self.server_name = hs.hostname
+
+        self.keyring = hs.get_keyring()
+
+        self.transport_layer = transport_layer
+
+        self.federation_client = self
+
+        self.store = hs.get_datastore()
+
+        self.handler = None
+        self.edu_handlers = {}
+        self.query_handlers = {}
+
+        self._clock = hs.get_clock()
+
+        self.transaction_actions = TransactionActions(self.store)
+
+        self.hs = hs
+
+        super(ReplicationLayer, self).__init__(hs)
+
+    def __str__(self):
+        return "<ReplicationLayer(%s)>" % self.server_name
@@ -40,8 +40,6 @@ from collections import namedtuple

 import logging

-from six import itervalues, iteritems
-
 logger = logging.getLogger(__name__)


@@ -124,7 +122,7 @@ class FederationRemoteSendQueue(object):

            user_ids = set(
                user_id
-                for uids in itervalues(self.presence_changed)
+                for uids in self.presence_changed.itervalues()
                for user_id in uids
            )

@@ -278,7 +276,7 @@ class FederationRemoteSendQueue(object):
        # stream position.
        keyed_edus = {self.keyed_edu_changed[k]: k for k in keys[i:j]}

-        for ((destination, edu_key), pos) in iteritems(keyed_edus):
+        for ((destination, edu_key), pos) in keyed_edus.iteritems():
            rows.append((pos, KeyedEduRow(
                key=edu_key,
                edu=self.keyed_edu[(destination, edu_key)],
@@ -311,7 +309,7 @@ class FederationRemoteSendQueue(object):
        j = keys.bisect_right(to_token) + 1
        device_messages = {self.device_messages[k]: k for k in keys[i:j]}

-        for (destination, pos) in iteritems(device_messages):
+        for (destination, pos) in device_messages.iteritems():
            rows.append((pos, DeviceRow(
                destination=destination,
            )))
@@ -530,19 +528,19 @@ def process_rows_for_federation(transaction_queue, rows):
    if buff.presence:
        transaction_queue.send_presence(buff.presence)

-    for destination, edu_map in iteritems(buff.keyed_edus):
+    for destination, edu_map in buff.keyed_edus.iteritems():
        for key, edu in edu_map.items():
            transaction_queue.send_edu(
                edu.destination, edu.edu_type, edu.content, key=key,
            )

-    for destination, edu_list in iteritems(buff.edus):
+    for destination, edu_list in buff.edus.iteritems():
        for edu in edu_list:
            transaction_queue.send_edu(
                edu.destination, edu.edu_type, edu.content, key=None,
            )

-    for destination, failure_list in iteritems(buff.failures):
+    for destination, failure_list in buff.failures.iteritems():
        for failure in failure_list:
            transaction_queue.send_failure(destination, failure)

@@ -169,7 +169,7 @@ class TransactionQueue(object):
            while True:
                last_token = yield self.store.get_federation_out_pos("events")
                next_token, events = yield self.store.get_all_new_events_stream(
-                    last_token, self._last_poked_id, limit=100,
+                    last_token, self._last_poked_id, limit=20,
                )

                logger.debug("Handling %s -> %s", last_token, next_token)
@@ -177,13 +177,12 @@ class TransactionQueue(object):
                if not events and next_token >= self._last_poked_id:
                    break

-                @defer.inlineCallbacks
-                def handle_event(event):
+                for event in events:
                    # Only send events for this server.
                    send_on_behalf_of = event.internal_metadata.get_send_on_behalf_of()
                    is_mine = self.is_mine_id(event.event_id)
                    if not is_mine and send_on_behalf_of is None:
-                        return
+                        continue

                    try:
                        # Get the state from before the event.
@@ -198,11 +197,8 @@ class TransactionQueue(object):
                            ],
                        )
                    except Exception:
-                        logger.exception(
-                            "Failed to calculate hosts in room for event: %s",
-                            event.event_id,
-                        )
-                        return
+                        logger.exception("Failed to calculate hosts in room")
+                        continue

                    destinations = set(destinations)

@@ -216,44 +212,12 @@ class TransactionQueue(object):

                    self._send_pdu(event, destinations)

-                @defer.inlineCallbacks
-                def handle_room_events(events):
-                    for event in events:
-                        yield handle_event(event)
-
-                events_by_room = {}
-                for event in events:
-                    events_by_room.setdefault(event.room_id, []).append(event)
-
-                yield logcontext.make_deferred_yieldable(defer.gatherResults(
-                    [
-                        logcontext.run_in_background(handle_room_events, evs)
-                        for evs in events_by_room.itervalues()
-                    ],
-                    consumeErrors=True
-                ))
+                events_processed_counter.inc_by(len(events))

                yield self.store.update_federation_out_pos(
                    "events", next_token
                )

-                if events:
-                    now = self.clock.time_msec()
-                    ts = yield self.store.get_received_ts(events[-1].event_id)
-
-                    synapse.metrics.event_processing_lag.set(
-                        now - ts, "federation_sender",
-                    )
-                    synapse.metrics.event_processing_last_ts.set(
-                        ts, "federation_sender",
-                    )
-
-                events_processed_counter.inc_by(len(events))
-
-                synapse.metrics.event_processing_positions.set(
-                    next_token, "federation_sender",
-                )
-
        finally:
            self._is_processing = False

@@ -295,6 +259,7 @@ class TransactionQueue(object):
        Args:
            states (list(UserPresenceState))
        """
+        return

        # First we queue up the new presence by user ID, so multiple presence
        # updates in quick successtion are correctly handled
@@ -323,8 +288,6 @@ class TransactionQueue(object):
                    break

                yield self._process_presence_inner(states_map.values())
-        except Exception:
-            logger.exception("Error sending presence states to servers")
        finally:
            self._processing_pending_presence = False

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -21,7 +20,6 @@ from synapse.api.urls import FEDERATION_PREFIX as PREFIX
 from synapse.util.logutils import log_function

 import logging
-import urllib


 logger = logging.getLogger(__name__)
@@ -51,7 +49,7 @@ class TransportLayerClient(object):
        logger.debug("get_room_state dest=%s, room=%s",
                     destination, room_id)

-        path = _create_path(PREFIX, "/state/%s/", room_id)
+        path = PREFIX + "/state/%s/" % room_id
        return self.client.get_json(
            destination, path=path, args={"event_id": event_id},
        )
@@ -73,7 +71,7 @@ class TransportLayerClient(object):
        logger.debug("get_room_state_ids dest=%s, room=%s",
                     destination, room_id)

-        path = _create_path(PREFIX, "/state_ids/%s/", room_id)
+        path = PREFIX + "/state_ids/%s/" % room_id
        return self.client.get_json(
            destination, path=path, args={"event_id": event_id},
        )
@@ -95,7 +93,7 @@ class TransportLayerClient(object):
        logger.debug("get_pdu dest=%s, event_id=%s",
                     destination, event_id)

-        path = _create_path(PREFIX, "/event/%s/", event_id)
+        path = PREFIX + "/event/%s/" % (event_id, )
        return self.client.get_json(destination, path=path, timeout=timeout)

    @log_function
@@ -121,7 +119,7 @@ class TransportLayerClient(object):
            # TODO: raise?
            return

-        path = _create_path(PREFIX, "/backfill/%s/", room_id)
+        path = PREFIX + "/backfill/%s/" % (room_id,)

        args = {
            "v": event_tuples,
@@ -159,11 +157,9 @@ class TransportLayerClient(object):
        # generated by the json_data_callback.
        json_data = transaction.get_dict()

-        path = _create_path(PREFIX, "/send/%s/", transaction.transaction_id)
-
        response = yield self.client.put_json(
            transaction.destination,
-            path=path,
+            path=PREFIX + "/send/%s/" % transaction.transaction_id,
            data=json_data,
            json_data_callback=json_data_callback,
            long_retries=True,
@@ -181,7 +177,7 @@ class TransportLayerClient(object):
    @log_function
    def make_query(self, destination, query_type, args, retry_on_dns_fail,
                   ignore_backoff=False):
-        path = _create_path(PREFIX, "/query/%s", query_type)
+        path = PREFIX + "/query/%s" % query_type

        content = yield self.client.get_json(
            destination=destination,
@@ -226,7 +222,7 @@ class TransportLayerClient(object):
                "make_membership_event called with membership='%s', must be one of %s" %
                (membership, ",".join(valid_memberships))
            )
-        path = _create_path(PREFIX, "/make_%s/%s/%s", membership, room_id, user_id)
+        path = PREFIX + "/make_%s/%s/%s" % (membership, room_id, user_id)

        ignore_backoff = False
        retry_on_dns_fail = False
@@ -252,7 +248,7 @@ class TransportLayerClient(object):
    @defer.inlineCallbacks
    @log_function
    def send_join(self, destination, room_id, event_id, content):
-        path = _create_path(PREFIX, "/send_join/%s/%s", room_id, event_id)
+        path = PREFIX + "/send_join/%s/%s" % (room_id, event_id)

        response = yield self.client.put_json(
            destination=destination,
@@ -265,7 +261,7 @@ class TransportLayerClient(object):
    @defer.inlineCallbacks
    @log_function
    def send_leave(self, destination, room_id, event_id, content):
-        path = _create_path(PREFIX, "/send_leave/%s/%s", room_id, event_id)
+        path = PREFIX + "/send_leave/%s/%s" % (room_id, event_id)

        response = yield self.client.put_json(
            destination=destination,
@@ -284,7 +280,7 @@ class TransportLayerClient(object):
    @defer.inlineCallbacks
    @log_function
    def send_invite(self, destination, room_id, event_id, content):
-        path = _create_path(PREFIX, "/invite/%s/%s", room_id, event_id)
+        path = PREFIX + "/invite/%s/%s" % (room_id, event_id)

        response = yield self.client.put_json(
            destination=destination,
@@ -326,7 +322,7 @@ class TransportLayerClient(object):
    @defer.inlineCallbacks
    @log_function
    def exchange_third_party_invite(self, destination, room_id, event_dict):
-        path = _create_path(PREFIX, "/exchange_third_party_invite/%s", room_id,)
+        path = PREFIX + "/exchange_third_party_invite/%s" % (room_id,)

        response = yield self.client.put_json(
            destination=destination,
@@ -339,7 +335,7 @@ class TransportLayerClient(object):
    @defer.inlineCallbacks
    @log_function
    def get_event_auth(self, destination, room_id, event_id):
-        path = _create_path(PREFIX, "/event_auth/%s/%s", room_id, event_id)
+        path = PREFIX + "/event_auth/%s/%s" % (room_id, event_id)

        content = yield self.client.get_json(
            destination=destination,
@@ -351,7 +347,7 @@ class TransportLayerClient(object):
    @defer.inlineCallbacks
    @log_function
    def send_query_auth(self, destination, room_id, event_id, content):
-        path = _create_path(PREFIX, "/query_auth/%s/%s", room_id, event_id)
+        path = PREFIX + "/query_auth/%s/%s" % (room_id, event_id)

        content = yield self.client.post_json(
            destination=destination,
@@ -413,7 +409,7 @@ class TransportLayerClient(object):
        Returns:
            A dict containg the device keys.
        """
-        path = _create_path(PREFIX, "/user/devices/%s", user_id)
+        path = PREFIX + "/user/devices/" + user_id

        content = yield self.client.get_json(
            destination=destination,
@@ -463,7 +459,7 @@ class TransportLayerClient(object):
    @log_function
    def get_missing_events(self, destination, room_id, earliest_events,
                           latest_events, limit, min_depth, timeout):
-        path = _create_path(PREFIX, "/get_missing_events/%s", room_id,)
+        path = PREFIX + "/get_missing_events/%s" % (room_id,)

        content = yield self.client.post_json(
            destination=destination,
@@ -483,7 +479,7 @@ class TransportLayerClient(object):
    def get_group_profile(self, destination, group_id, requester_user_id):
        """Get a group profile
        """
-        path = _create_path(PREFIX, "/groups/%s/profile", group_id,)
+        path = PREFIX + "/groups/%s/profile" % (group_id,)

        return self.client.get_json(
            destination=destination,
@@ -502,7 +498,7 @@ class TransportLayerClient(object):
            requester_user_id (str)
            content (dict): The new profile of the group
        """
-        path = _create_path(PREFIX, "/groups/%s/profile", group_id,)
+        path = PREFIX + "/groups/%s/profile" % (group_id,)

        return self.client.post_json(
            destination=destination,
@@ -516,7 +512,7 @@ class TransportLayerClient(object):
    def get_group_summary(self, destination, group_id, requester_user_id):
        """Get a group summary
        """
-        path = _create_path(PREFIX, "/groups/%s/summary", group_id,)
+        path = PREFIX + "/groups/%s/summary" % (group_id,)

        return self.client.get_json(
            destination=destination,
@@ -529,7 +525,7 @@ class TransportLayerClient(object):
    def get_rooms_in_group(self, destination, group_id, requester_user_id):
        """Get all rooms in a group
        """
-        path = _create_path(PREFIX, "/groups/%s/rooms", group_id,)
+        path = PREFIX + "/groups/%s/rooms" % (group_id,)

        return self.client.get_json(
            destination=destination,
@@ -542,7 +538,7 @@ class TransportLayerClient(object):
                          content):
        """Add a room to a group
        """
-        path = _create_path(PREFIX, "/groups/%s/room/%s", group_id, room_id,)
+        path = PREFIX + "/groups/%s/room/%s" % (group_id, room_id,)

        return self.client.post_json(
            destination=destination,
@@ -556,10 +552,7 @@ class TransportLayerClient(object):
                             config_key, content):
        """Update room in group
        """
-        path = _create_path(
-            PREFIX, "/groups/%s/room/%s/config/%s",
-            group_id, room_id, config_key,
-        )
+        path = PREFIX + "/groups/%s/room/%s/config/%s" % (group_id, room_id, config_key,)

        return self.client.post_json(
            destination=destination,
@@ -572,7 +565,7 @@ class TransportLayerClient(object):
    def remove_room_from_group(self, destination, group_id, requester_user_id, room_id):
        """Remove a room from a group
        """
-        path = _create_path(PREFIX, "/groups/%s/room/%s", group_id, room_id,)
+        path = PREFIX + "/groups/%s/room/%s" % (group_id, room_id,)

        return self.client.delete_json(
            destination=destination,
@@ -585,7 +578,7 @@ class TransportLayerClient(object):
    def get_users_in_group(self, destination, group_id, requester_user_id):
        """Get users in a group
        """
-        path = _create_path(PREFIX, "/groups/%s/users", group_id,)
+        path = PREFIX + "/groups/%s/users" % (group_id,)

        return self.client.get_json(
            destination=destination,
@@ -598,7 +591,7 @@ class TransportLayerClient(object):
    def get_invited_users_in_group(self, destination, group_id, requester_user_id):
        """Get users that have been invited to a group
        """
-        path = _create_path(PREFIX, "/groups/%s/invited_users", group_id,)
+        path = PREFIX + "/groups/%s/invited_users" % (group_id,)

        return self.client.get_json(
            destination=destination,
@@ -611,23 +604,7 @@ class TransportLayerClient(object):
    def accept_group_invite(self, destination, group_id, user_id, content):
        """Accept a group invite
        """
-        path = _create_path(
-            PREFIX, "/groups/%s/users/%s/accept_invite",
-            group_id, user_id,
-        )
-
-        return self.client.post_json(
-            destination=destination,
-            path=path,
-            data=content,
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def join_group(self, destination, group_id, user_id, content):
-        """Attempts to join a group
-        """
-        path = _create_path(PREFIX, "/groups/%s/users/%s/join", group_id, user_id)
+        path = PREFIX + "/groups/%s/users/%s/accept_invite" % (group_id, user_id)

        return self.client.post_json(
            destination=destination,
@@ -640,7 +617,7 @@ class TransportLayerClient(object):
    def invite_to_group(self, destination, group_id, user_id, requester_user_id, content):
        """Invite a user to a group
        """
-        path = _create_path(PREFIX, "/groups/%s/users/%s/invite", group_id, user_id)
+        path = PREFIX + "/groups/%s/users/%s/invite" % (group_id, user_id)

        return self.client.post_json(
            destination=destination,
@@ -656,7 +633,7 @@ class TransportLayerClient(object):
        invited.
        """

-        path = _create_path(PREFIX, "/groups/local/%s/users/%s/invite", group_id, user_id)
+        path = PREFIX + "/groups/local/%s/users/%s/invite" % (group_id, user_id)

        return self.client.post_json(
            destination=destination,
@@ -670,7 +647,7 @@ class TransportLayerClient(object):
                               user_id, content):
        """Remove a user fron a group
        """
-        path = _create_path(PREFIX, "/groups/%s/users/%s/remove", group_id, user_id)
+        path = PREFIX + "/groups/%s/users/%s/remove" % (group_id, user_id)

        return self.client.post_json(
            destination=destination,
@@ -687,7 +664,7 @@ class TransportLayerClient(object):
        kicked from the group.
        """

-        path = _create_path(PREFIX, "/groups/local/%s/users/%s/remove", group_id, user_id)
+        path = PREFIX + "/groups/local/%s/users/%s/remove" % (group_id, user_id)

        return self.client.post_json(
            destination=destination,
@@ -702,7 +679,7 @@ class TransportLayerClient(object):
        the attestations
        """

-        path = _create_path(PREFIX, "/groups/%s/renew_attestation/%s", group_id, user_id)
+        path = PREFIX + "/groups/%s/renew_attestation/%s" % (group_id, user_id)

        return self.client.post_json(
            destination=destination,
@@ -717,12 +694,11 @@ class TransportLayerClient(object):
        """Update a room entry in a group summary
        """
        if category_id:
-            path = _create_path(
-                PREFIX, "/groups/%s/summary/categories/%s/rooms/%s",
+            path = PREFIX + "/groups/%s/summary/categories/%s/rooms/%s" % (
                group_id, category_id, room_id,
            )
        else:
-            path = _create_path(PREFIX, "/groups/%s/summary/rooms/%s", group_id, room_id,)
+            path = PREFIX + "/groups/%s/summary/rooms/%s" % (group_id, room_id,)

        return self.client.post_json(
            destination=destination,
@@ -738,12 +714,11 @@ class TransportLayerClient(object):
        """Delete a room entry in a group summary
        """
        if category_id:
-            path = _create_path(
-                PREFIX + "/groups/%s/summary/categories/%s/rooms/%s",
+            path = PREFIX + "/groups/%s/summary/categories/%s/rooms/%s" % (
                group_id, category_id, room_id,
            )
        else:
-            path = _create_path(PREFIX, "/groups/%s/summary/rooms/%s", group_id, room_id,)
+            path = PREFIX + "/groups/%s/summary/rooms/%s" % (group_id, room_id,)

        return self.client.delete_json(
            destination=destination,
@@ -756,7 +731,7 @@ class TransportLayerClient(object):
    def get_group_categories(self, destination, group_id, requester_user_id):
        """Get all categories in a group
        """
-        path = _create_path(PREFIX, "/groups/%s/categories", group_id,)
+        path = PREFIX + "/groups/%s/categories" % (group_id,)

        return self.client.get_json(
            destination=destination,
@@ -769,7 +744,7 @@ class TransportLayerClient(object):
    def get_group_category(self, destination, group_id, requester_user_id, category_id):
        """Get category info in a group
        """
-        path = _create_path(PREFIX, "/groups/%s/categories/%s", group_id, category_id,)
+        path = PREFIX + "/groups/%s/categories/%s" % (group_id, category_id,)

        return self.client.get_json(
            destination=destination,
@@ -783,7 +758,7 @@ class TransportLayerClient(object):
                              content):
        """Update a category in a group
        """
-        path = _create_path(PREFIX, "/groups/%s/categories/%s", group_id, category_id,)
+        path = PREFIX + "/groups/%s/categories/%s" % (group_id, category_id,)

        return self.client.post_json(
            destination=destination,
@@ -798,7 +773,7 @@ class TransportLayerClient(object):
                              category_id):
        """Delete a category in a group
        """
-        path = _create_path(PREFIX, "/groups/%s/categories/%s", group_id, category_id,)
+        path = PREFIX + "/groups/%s/categories/%s" % (group_id, category_id,)

        return self.client.delete_json(
            destination=destination,
@@ -811,7 +786,7 @@ class TransportLayerClient(object):
    def get_group_roles(self, destination, group_id, requester_user_id):
        """Get all roles in a group
        """
-        path = _create_path(PREFIX, "/groups/%s/roles", group_id,)
+        path = PREFIX + "/groups/%s/roles" % (group_id,)

        return self.client.get_json(
            destination=destination,
@@ -824,7 +799,7 @@ class TransportLayerClient(object):
    def get_group_role(self, destination, group_id, requester_user_id, role_id):
        """Get a roles info
        """
-        path = _create_path(PREFIX, "/groups/%s/roles/%s", group_id, role_id,)
+        path = PREFIX + "/groups/%s/roles/%s" % (group_id, role_id,)

        return self.client.get_json(
            destination=destination,
@@ -838,7 +813,7 @@ class TransportLayerClient(object):
                          content):
        """Update a role in a group
        """
-        path = _create_path(PREFIX, "/groups/%s/roles/%s", group_id, role_id,)
+        path = PREFIX + "/groups/%s/roles/%s" % (group_id, role_id,)

        return self.client.post_json(
            destination=destination,
@@ -852,7 +827,7 @@ class TransportLayerClient(object):
    def delete_group_role(self, destination, group_id, requester_user_id, role_id):
        """Delete a role in a group
        """
-        path = _create_path(PREFIX, "/groups/%s/roles/%s", group_id, role_id,)
+        path = PREFIX + "/groups/%s/roles/%s" % (group_id, role_id,)

        return self.client.delete_json(
            destination=destination,
@@ -867,12 +842,11 @@ class TransportLayerClient(object):
        """Update a users entry in a group
        """
        if role_id:
-            path = _create_path(
-                PREFIX, "/groups/%s/summary/roles/%s/users/%s",
+            path = PREFIX + "/groups/%s/summary/roles/%s/users/%s" % (
                group_id, role_id, user_id,
            )
        else:
-            path = _create_path(PREFIX, "/groups/%s/summary/users/%s", group_id, user_id,)
+            path = PREFIX + "/groups/%s/summary/users/%s" % (group_id, user_id,)

        return self.client.post_json(
            destination=destination,
@@ -882,33 +856,17 @@ class TransportLayerClient(object):
            ignore_backoff=True,
        )

-    @log_function
-    def set_group_join_policy(self, destination, group_id, requester_user_id,
-                              content):
-        """Sets the join policy for a group
-        """
-        path = _create_path(PREFIX, "/groups/%s/settings/m.join_policy", group_id,)
-
-        return self.client.put_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            data=content,
-            ignore_backoff=True,
-        )
-
    @log_function
    def delete_group_summary_user(self, destination, group_id, requester_user_id,
                                  user_id, role_id):
        """Delete a users entry in a group
        """
        if role_id:
-            path = _create_path(
-                PREFIX, "/groups/%s/summary/roles/%s/users/%s",
+            path = PREFIX + "/groups/%s/summary/roles/%s/users/%s" % (
                group_id, role_id, user_id,
            )
        else:
-            path = _create_path(PREFIX, "/groups/%s/summary/users/%s", group_id, user_id,)
+            path = PREFIX + "/groups/%s/summary/users/%s" % (group_id, user_id,)

        return self.client.delete_json(
            destination=destination,
@@ -931,22 +889,3 @@ class TransportLayerClient(object):
            data=content,
            ignore_backoff=True,
        )
-
-
-def _create_path(prefix, path, *args):
-    """Creates a path from the prefix, path template and args. Ensures that
-    all args are url encoded.
-
-    Example:
-
-        _create_path(PREFIX, "/event/%s/", event_id)
-
-    Args:
-        prefix (str)
-        path (str): String template for the path
-        args: ([str]): Args to insert into path. Each arg will be url encoded
-
-    Returns:
-        str
-    """
-    return prefix + path % tuple(urllib.quote(arg, "") for arg in args)
@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -25,7 +24,7 @@ from synapse.http.servlet import (
 )
 from synapse.util.ratelimitutils import FederationRateLimiter
 from synapse.util.versionstring import get_version_string
-from synapse.util.logcontext import run_in_background
+from synapse.util.logcontext import preserve_fn
 from synapse.types import ThirdPartyInstanceID, get_domain_from_id

 import functools
@@ -94,6 +93,12 @@ class Authenticator(object):
            "signatures": {},
        }

+        if (
+            self.federation_domain_whitelist is not None and
+            self.server_name not in self.federation_domain_whitelist
+        ):
+            raise FederationDeniedError(self.server_name)
+
        if content is not None:
            json_request["content"] = content

@@ -132,12 +137,6 @@ class Authenticator(object):
                json_request["origin"] = origin
                json_request["signatures"].setdefault(origin, {})[key] = sig

-        if (
-            self.federation_domain_whitelist is not None and
-            origin not in self.federation_domain_whitelist
-        ):
-            raise FederationDeniedError(origin)
-
        if not json_request["signatures"]:
            raise NoAuthenticationError(
                401, "Missing Authorization headers", Codes.UNAUTHORIZED,
@@ -152,18 +151,11 @@ class Authenticator(object):
        # alive
        retry_timings = yield self.store.get_destination_retry_timings(origin)
        if retry_timings and retry_timings["retry_last_ts"]:
-            run_in_background(self._reset_retry_timings, origin)
+            logger.info("Marking origin %r as up", origin)
+            preserve_fn(self.store.set_destination_retry_timings)(origin, 0, 0)

        defer.returnValue(origin)

-    @defer.inlineCallbacks
-    def _reset_retry_timings(self, origin):
-        try:
-            logger.info("Marking origin %r as up", origin)
-            yield self.store.set_destination_retry_timings(origin, 0, 0)
-        except Exception:
-            logger.exception("Error resetting retry timings on %s", origin)
-

 class BaseFederationServlet(object):
    REQUIRE_AUTH = True
@@ -810,23 +802,6 @@ class FederationGroupsAcceptInviteServlet(BaseFederationServlet):
        defer.returnValue((200, new_content))


-class FederationGroupsJoinServlet(BaseFederationServlet):
-    """Attempt to join a group
-    """
-    PATH = "/groups/(?P<group_id>[^/]*)/users/(?P<user_id>[^/]*)/join$"
-
-    @defer.inlineCallbacks
-    def on_POST(self, origin, content, query, group_id, user_id):
-        if get_domain_from_id(user_id) != origin:
-            raise SynapseError(403, "user_id doesn't match origin")
-
-        new_content = yield self.handler.join_group(
-            group_id, user_id, content,
-        )
-
-        defer.returnValue((200, new_content))
-
-
 class FederationGroupsRemoveUserServlet(BaseFederationServlet):
    """Leave or kick a user from the group
    """
@@ -1149,24 +1124,6 @@ class FederationGroupsBulkPublicisedServlet(BaseFederationServlet):
        defer.returnValue((200, resp))


-class FederationGroupsSettingJoinPolicyServlet(BaseFederationServlet):
-    """Sets whether a group is joinable without an invite or knock
-    """
-    PATH = "/groups/(?P<group_id>[^/]*)/settings/m.join_policy$"
-
-    @defer.inlineCallbacks
-    def on_PUT(self, origin, content, query, group_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        new_content = yield self.handler.set_group_join_policy(
-            group_id, requester_user_id, content
-        )
-
-        defer.returnValue((200, new_content))
-
-
 FEDERATION_SERVLET_CLASSES = (
    FederationSendServlet,
    FederationPullServlet,
@@ -1206,7 +1163,6 @@ GROUP_SERVER_SERVLET_CLASSES = (
    FederationGroupsInvitedUsersServlet,
    FederationGroupsInviteServlet,
    FederationGroupsAcceptInviteServlet,
-    FederationGroupsJoinServlet,
    FederationGroupsRemoveUserServlet,
    FederationGroupsSummaryRoomsServlet,
    FederationGroupsCategoriesServlet,
@@ -1216,7 +1172,6 @@ GROUP_SERVER_SERVLET_CLASSES = (
    FederationGroupsSummaryUsersServlet,
    FederationGroupsAddRoomsServlet,
    FederationGroupsAddRoomsConfigServlet,
-    FederationGroupsSettingJoinPolicyServlet,
 )


@@ -1235,7 +1190,7 @@ GROUP_ATTESTATION_SERVLET_CLASSES = (
 def register_servlets(hs, resource, authenticator, ratelimiter):
    for servletclass in FEDERATION_SERVLET_CLASSES:
        servletclass(
-            handler=hs.get_federation_server(),
+            handler=hs.get_replication_layer(),
            authenticator=authenticator,
            ratelimiter=ratelimiter,
            server_name=hs.hostname,
@@ -74,6 +74,8 @@ class Transaction(JsonEncodedObject):
        "previous_ids",
        "pdus",
        "edus",
+        "transaction_id",
+        "destination",
        "pdu_failures",
    ]

@@ -42,7 +42,7 @@ from twisted.internet import defer

 from synapse.api.errors import SynapseError
 from synapse.types import get_domain_from_id
-from synapse.util.logcontext import run_in_background
+from synapse.util.logcontext import preserve_fn

 from signedjson.sign import sign_json

@@ -165,35 +165,31 @@ class GroupAttestionRenewer(object):

        @defer.inlineCallbacks
        def _renew_attestation(group_id, user_id):
-            try:
-                if not self.is_mine_id(group_id):
-                    destination = get_domain_from_id(group_id)
-                elif not self.is_mine_id(user_id):
-                    destination = get_domain_from_id(user_id)
-                else:
-                    logger.warn(
-                        "Incorrectly trying to do attestations for user: %r in %r",
-                        user_id, group_id,
-                    )
-                    yield self.store.remove_attestation_renewal(group_id, user_id)
-                    return
-
-                attestation = self.attestations.create_attestation(group_id, user_id)
-
-                yield self.transport_client.renew_group_attestation(
-                    destination, group_id, user_id,
-                    content={"attestation": attestation},
+            if not self.is_mine_id(group_id):
+                destination = get_domain_from_id(group_id)
+            elif not self.is_mine_id(user_id):
+                destination = get_domain_from_id(user_id)
+            else:
+                logger.warn(
+                    "Incorrectly trying to do attestations for user: %r in %r",
+                    user_id, group_id,
                )
+                yield self.store.remove_attestation_renewal(group_id, user_id)
+                return

-                yield self.store.update_attestation_renewal(
-                    group_id, user_id, attestation
-                )
-            except Exception:
-                logger.exception("Error renewing attestation of %r in %r",
-                                 user_id, group_id)
+            attestation = self.attestations.create_attestation(group_id, user_id)
+
+            yield self.transport_client.renew_group_attestation(
+                destination, group_id, user_id,
+                content={"attestation": attestation},
+            )
+
+            yield self.store.update_attestation_renewal(
+                group_id, user_id, attestation
+            )

        for row in rows:
            group_id = row["group_id"]
            user_id = row["user_id"]

-            run_in_background(_renew_attestation, group_id, user_id)
+            preserve_fn(_renew_attestation)(group_id, user_id)
@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2017 Vector Creations Ltd
-# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -206,28 +205,6 @@ class GroupsServerHandler(object):

        defer.returnValue({})

-    @defer.inlineCallbacks
-    def set_group_join_policy(self, group_id, requester_user_id, content):
-        """Sets the group join policy.
-
-        Currently supported policies are:
-         - "invite": an invite must be received and accepted in order to join.
-         - "open": anyone can join.
-        """
-        yield self.check_group_is_ours(
-            group_id, requester_user_id, and_exists=True, and_is_admin=requester_user_id
-        )
-
-        join_policy = _parse_join_policy_from_contents(content)
-        if join_policy is None:
-            raise SynapseError(
-                400, "No value specified for 'm.join_policy'"
-            )
-
-        yield self.store.set_group_join_policy(group_id, join_policy=join_policy)
-
-        defer.returnValue({})
-
    @defer.inlineCallbacks
    def get_group_categories(self, group_id, requester_user_id):
        """Get all categories in a group (as seen by user)
@@ -404,16 +381,9 @@ class GroupsServerHandler(object):

        yield self.check_group_is_ours(group_id, requester_user_id)

-        group = yield self.store.get_group(group_id)
-
-        if group:
-            cols = [
-                "name", "short_description", "long_description",
-                "avatar_url", "is_public",
-            ]
-            group_description = {key: group[key] for key in cols}
-            group_description["is_openly_joinable"] = group["join_policy"] == "open"
+        group_description = yield self.store.get_group(group_id)

+        if group_description:
            defer.returnValue(group_description)
        else:
            raise SynapseError(404, "Unknown group")
@@ -684,40 +654,6 @@ class GroupsServerHandler(object):
        else:
            raise SynapseError(502, "Unknown state returned by HS")

-    @defer.inlineCallbacks
-    def _add_user(self, group_id, user_id, content):
-        """Add a user to a group based on a content dict.
-
-        See accept_invite, join_group.
-        """
-        if not self.hs.is_mine_id(user_id):
-            local_attestation = self.attestations.create_attestation(
-                group_id, user_id,
-            )
-
-            remote_attestation = content["attestation"]
-
-            yield self.attestations.verify_attestation(
-                remote_attestation,
-                user_id=user_id,
-                group_id=group_id,
-            )
-        else:
-            local_attestation = None
-            remote_attestation = None
-
-        is_public = _parse_visibility_from_contents(content)
-
-        yield self.store.add_user_to_group(
-            group_id, user_id,
-            is_admin=False,
-            is_public=is_public,
-            local_attestation=local_attestation,
-            remote_attestation=remote_attestation,
-        )
-
-        defer.returnValue(local_attestation)
-
    @defer.inlineCallbacks
    def accept_invite(self, group_id, requester_user_id, content):
        """User tries to accept an invite to the group.
@@ -734,27 +670,30 @@ class GroupsServerHandler(object):
        if not is_invited:
            raise SynapseError(403, "User not invited to group")

-        local_attestation = yield self._add_user(group_id, requester_user_id, content)
+        if not self.hs.is_mine_id(requester_user_id):
+            local_attestation = self.attestations.create_attestation(
+                group_id, requester_user_id,
+            )
+            remote_attestation = content["attestation"]

-        defer.returnValue({
-            "state": "join",
-            "attestation": local_attestation,
-        })
+            yield self.attestations.verify_attestation(
+                remote_attestation,
+                user_id=requester_user_id,
+                group_id=group_id,
+            )
+        else:
+            local_attestation = None
+            remote_attestation = None

-    @defer.inlineCallbacks
-    def join_group(self, group_id, requester_user_id, content):
-        """User tries to join the group.
+        is_public = _parse_visibility_from_contents(content)

-        This will error if the group requires an invite/knock to join
-        """
-
-        group_info = yield self.check_group_is_ours(
-            group_id, requester_user_id, and_exists=True
+        yield self.store.add_user_to_group(
+            group_id, requester_user_id,
+            is_admin=False,
+            is_public=is_public,
+            local_attestation=local_attestation,
+            remote_attestation=remote_attestation,
        )
-        if group_info['join_policy'] != "open":
-            raise SynapseError(403, "Group is not publicly joinable")
-
-        local_attestation = yield self._add_user(group_id, requester_user_id, content)

        defer.returnValue({
            "state": "join",
@@ -896,31 +835,6 @@ class GroupsServerHandler(object):
        })


-def _parse_join_policy_from_contents(content):
-    """Given a content for a request, return the specified join policy or None
-    """
-
-    join_policy_dict = content.get("m.join_policy")
-    if join_policy_dict:
-        return _parse_join_policy_dict(join_policy_dict)
-    else:
-        return None
-
-
-def _parse_join_policy_dict(join_policy_dict):
-    """Given a dict for the "m.join_policy" config return the join policy specified
-    """
-    join_policy_type = join_policy_dict.get("type")
-    if not join_policy_type:
-        return "invite"
-
-    if join_policy_type not in ("invite", "open"):
-        raise SynapseError(
-            400, "Synapse only supports 'invite'/'open' join rule"
-        )
-    return join_policy_type
-
-
 def _parse_visibility_from_contents(content):
    """Given a content for a request parse out whether the entity should be
    public or not
@@ -17,6 +17,7 @@ from .register import RegistrationHandler
 from .room import (
    RoomCreationHandler, RoomContextHandler,
 )
+from .room_member import RoomMemberHandler
 from .message import MessageHandler
 from .federation import FederationHandler
 from .directory import DirectoryHandler
@@ -48,6 +49,7 @@ class Handlers(object):
        self.registration_handler = RegistrationHandler(hs)
        self.message_handler = MessageHandler(hs)
        self.room_creation_handler = RoomCreationHandler(hs)
+        self.room_member_handler = RoomMemberHandler(hs)
        self.federation_handler = FederationHandler(hs)
        self.directory_handler = DirectoryHandler(hs)
        self.admin_handler = AdminHandler(hs)
@@ -158,7 +158,7 @@ class BaseHandler(object):
                # homeserver.
                requester = synapse.types.create_requester(
                    target_user, is_guest=True)
-                handler = self.hs.get_room_member_handler()
+                handler = self.hs.get_handlers().room_member_handler
                yield handler.update_membership(
                    requester,
                    target_user,
@@ -18,9 +18,7 @@ from twisted.internet import defer
 import synapse
 from synapse.api.constants import EventTypes
 from synapse.util.metrics import Measure
-from synapse.util.logcontext import (
-    make_deferred_yieldable, run_in_background,
-)
+from synapse.util.logcontext import make_deferred_yieldable, preserve_fn

 import logging

@@ -86,16 +84,11 @@ class ApplicationServicesHandler(object):
                    if not events:
                        break

-                    events_by_room = {}
                    for event in events:
-                        events_by_room.setdefault(event.room_id, []).append(event)
-
-                    @defer.inlineCallbacks
-                    def handle_event(event):
                        # Gather interested services
                        services = yield self._get_services_for_event(event)
                        if len(services) == 0:
-                            return  # no services need notifying
+                            continue  # no services need notifying

                        # Do we know this user exists? If not, poke the user
                        # query API for all services which match that user regex.
@@ -111,35 +104,13 @@ class ApplicationServicesHandler(object):

                        # Fork off pushes to these services
                        for service in services:
-                            self.scheduler.submit_event_for_as(service, event)
-
-                    @defer.inlineCallbacks
-                    def handle_room_events(events):
-                        for event in events:
-                            yield handle_event(event)
-
-                    yield make_deferred_yieldable(defer.gatherResults([
-                        run_in_background(handle_room_events, evs)
-                        for evs in events_by_room.itervalues()
-                    ], consumeErrors=True))
-
-                    yield self.store.set_appservice_last_pos(upper_bound)
-
-                    now = self.clock.time_msec()
-                    ts = yield self.store.get_received_ts(events[-1].event_id)
-
-                    synapse.metrics.event_processing_positions.set(
-                        upper_bound, "appservice_sender",
-                    )
+                            preserve_fn(self.scheduler.submit_event_for_as)(
+                                service, event
+                            )

                    events_processed_counter.inc_by(len(events))

-                    synapse.metrics.event_processing_lag.set(
-                        now - ts, "appservice_sender",
-                    )
-                    synapse.metrics.event_processing_last_ts.set(
-                        ts, "appservice_sender",
-                    )
+                    yield self.store.set_appservice_last_pos(upper_bound)
            finally:
                self.is_processing = False

@@ -196,10 +167,7 @@ class ApplicationServicesHandler(object):
        services = yield self._get_services_for_3pn(protocol)

        results = yield make_deferred_yieldable(defer.DeferredList([
-            run_in_background(
-                self.appservice_api.query_3pe,
-                service, kind, protocol, fields,
-            )
+            preserve_fn(self.appservice_api.query_3pe)(service, kind, protocol, fields)
            for service in services
        ], consumeErrors=True))

@@ -260,15 +228,11 @@ class ApplicationServicesHandler(object):
            event based on the service regex.
        """
        services = self.store.get_app_services()
-
-        # we can't use a list comprehension here. Since python 3, list
-        # comprehensions use a generator internally. This means you can't yield
-        # inside of a list comprehension anymore.
-        interested_list = []
-        for s in services:
-            if (yield s.is_interested(event, self.store)):
-                interested_list.append(s)
-
+        interested_list = [
+            s for s in services if (
+                yield s.is_interested(event, self.store)
+            )
+        ]
        defer.returnValue(interested_list)

    def _get_services_for_user(self, user_id):
@@ -863,10 +863,8 @@ class AuthHandler(BaseHandler):
        """

        def _do_validate_hash():
-            return bcrypt.checkpw(
-                password.encode('utf8') + self.hs.config.password_pepper,
-                stored_hash.encode('utf8')
-            )
+            return bcrypt.hashpw(password.encode('utf8') + self.hs.config.password_pepper,
+                                 stored_hash.encode('utf8')) == stored_hash

        if stored_hash:
            return make_deferred_yieldable(threads.deferToThread(_do_validate_hash))
@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2017, 2018 New Vector Ltd
+# Copyright 2017 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -12,11 +12,9 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-from twisted.internet import defer, reactor
+from twisted.internet import defer

 from ._base import BaseHandler
-from synapse.types import UserID, create_requester
-from synapse.util.logcontext import run_in_background

 import logging

@@ -29,14 +27,6 @@ class DeactivateAccountHandler(BaseHandler):
        super(DeactivateAccountHandler, self).__init__(hs)
        self._auth_handler = hs.get_auth_handler()
        self._device_handler = hs.get_device_handler()
-        self._room_member_handler = hs.get_room_member_handler()
-
-        # Flag that indicates whether the process to part users from rooms is running
-        self._user_parter_running = False
-
-        # Start the user parter loop so it can resume parting users from rooms where
-        # it left off (if it has work left to do).
-        reactor.callWhenRunning(self._start_user_parting)

    @defer.inlineCallbacks
    def deactivate_account(self, user_id):
@@ -60,70 +50,3 @@ class DeactivateAccountHandler(BaseHandler):

        yield self.store.user_delete_threepids(user_id)
        yield self.store.user_set_password_hash(user_id, None)
-
-        # Add the user to a table of users penpding deactivation (ie.
-        # removal from all the rooms they're a member of)
-        yield self.store.add_user_pending_deactivation(user_id)
-
-        # Now start the process that goes through that list and
-        # parts users from rooms (if it isn't already running)
-        self._start_user_parting()
-
-    def _start_user_parting(self):
-        """
-        Start the process that goes through the table of users
-        pending deactivation, if it isn't already running.
-
-        Returns:
-            None
-        """
-        if not self._user_parter_running:
-            run_in_background(self._user_parter_loop)
-
-    @defer.inlineCallbacks
-    def _user_parter_loop(self):
-        """Loop that parts deactivated users from rooms
-
-        Returns:
-            None
-        """
-        self._user_parter_running = True
-        logger.info("Starting user parter")
-        try:
-            while True:
-                user_id = yield self.store.get_user_pending_deactivation()
-                if user_id is None:
-                    break
-                logger.info("User parter parting %r", user_id)
-                yield self._part_user(user_id)
-                yield self.store.del_user_pending_deactivation(user_id)
-                logger.info("User parter finished parting %r", user_id)
-            logger.info("User parter finished: stopping")
-        finally:
-            self._user_parter_running = False
-
-    @defer.inlineCallbacks
-    def _part_user(self, user_id):
-        """Causes the given user_id to leave all the rooms they're joined to
-
-        Returns:
-            None
-        """
-        user = UserID.from_string(user_id)
-
-        rooms_for_user = yield self.store.get_rooms_for_user(user_id)
-        for room_id in rooms_for_user:
-            logger.info("User parter parting %r from %r", user_id, room_id)
-            try:
-                yield self._room_member_handler.update_membership(
-                    create_requester(user),
-                    user,
-                    room_id,
-                    "leave",
-                    ratelimit=False,
-                )
-            except Exception:
-                logger.exception(
-                    "Failed to part user %r from room %r: ignoring and continuing",
-                    user_id, room_id,
-                )
@@ -37,15 +37,14 @@ class DeviceHandler(BaseHandler):
        self.state = hs.get_state_handler()
        self._auth_handler = hs.get_auth_handler()
        self.federation_sender = hs.get_federation_sender()
+        self.federation = hs.get_replication_layer()

        self._edu_updater = DeviceListEduUpdater(hs, self)

-        federation_registry = hs.get_federation_registry()
-
-        federation_registry.register_edu_handler(
+        self.federation.register_edu_handler(
            "m.device_list_update", self._edu_updater.incoming_device_list_update,
        )
-        federation_registry.register_query_handler(
+        self.federation.register_query_handler(
            "user_devices", self.on_federation_query_user_devices,
        )

@@ -155,7 +154,7 @@ class DeviceHandler(BaseHandler):

        try:
            yield self.store.delete_device(user_id, device_id)
-        except errors.StoreError as e:
+        except errors.StoreError, e:
            if e.code == 404:
                # no match
                pass
@@ -204,7 +203,7 @@ class DeviceHandler(BaseHandler):

        try:
            yield self.store.delete_devices(user_id, device_ids)
-        except errors.StoreError as e:
+        except errors.StoreError, e:
            if e.code == 404:
                # no match
                pass
@@ -243,7 +242,7 @@ class DeviceHandler(BaseHandler):
                new_display_name=content.get("display_name")
            )
            yield self.notify_device_update(user_id, [device_id])
-        except errors.StoreError as e:
+        except errors.StoreError, e:
            if e.code == 404:
                raise errors.NotFoundError()
            else:
@@ -431,7 +430,7 @@ class DeviceListEduUpdater(object):

    def __init__(self, hs, device_handler):
        self.store = hs.get_datastore()
-        self.federation = hs.get_federation_client()
+        self.federation = hs.get_replication_layer()
        self.clock = hs.get_clock()
        self.device_handler = device_handler

@@ -37,7 +37,7 @@ class DeviceMessageHandler(object):
        self.is_mine = hs.is_mine
        self.federation = hs.get_federation_sender()

-        hs.get_federation_registry().register_edu_handler(
+        hs.get_replication_layer().register_edu_handler(
            "m.direct_to_device", self.on_direct_to_device_edu
        )

@@ -36,8 +36,8 @@ class DirectoryHandler(BaseHandler):
        self.appservice_handler = hs.get_application_service_handler()
        self.event_creation_handler = hs.get_event_creation_handler()

-        self.federation = hs.get_federation_client()
-        hs.get_federation_registry().register_query_handler(
+        self.federation = hs.get_replication_layer()
+        self.federation.register_query_handler(
            "directory", self.on_directory_query
        )

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2016 OpenMarket Ltd
-# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-import simplejson as json
+import ujson as json
 import logging

 from canonicaljson import encode_canonical_json
@@ -24,7 +23,7 @@ from synapse.api.errors import (
    SynapseError, CodeMessageException, FederationDeniedError,
 )
 from synapse.types import get_domain_from_id, UserID
-from synapse.util.logcontext import make_deferred_yieldable, run_in_background
+from synapse.util.logcontext import preserve_fn, make_deferred_yieldable
 from synapse.util.retryutils import NotRetryingDestination

 logger = logging.getLogger(__name__)
@@ -33,7 +32,7 @@ logger = logging.getLogger(__name__)
 class E2eKeysHandler(object):
    def __init__(self, hs):
        self.store = hs.get_datastore()
-        self.federation = hs.get_federation_client()
+        self.federation = hs.get_replication_layer()
        self.device_handler = hs.get_device_handler()
        self.is_mine = hs.is_mine
        self.clock = hs.get_clock()
@@ -41,7 +40,7 @@ class E2eKeysHandler(object):
        # doesn't really work as part of the generic query API, because the
        # query request requires an object POST, but we abuse the
        # "query handler" interface.
-        hs.get_federation_registry().register_query_handler(
+        self.federation.register_query_handler(
            "client_keys", self.on_federation_query_client_keys
        )

@@ -135,13 +134,28 @@ class E2eKeysHandler(object):
                    if user_id in destination_query:
                        results[user_id] = keys

+            except CodeMessageException as e:
+                failures[destination] = {
+                    "status": e.code, "message": e.message
+                }
+            except NotRetryingDestination as e:
+                failures[destination] = {
+                    "status": 503, "message": "Not ready for retry",
+                }
+            except FederationDeniedError as e:
+                failures[destination] = {
+                    "status": 403, "message": "Federation Denied",
+                }
            except Exception as e:
-                failures[destination] = _exception_to_failure(e)
+                # include ConnectionRefused and other errors
+                failures[destination] = {
+                    "status": 503, "message": e.message
+                }

        yield make_deferred_yieldable(defer.gatherResults([
-            run_in_background(do_remote_query, destination)
+            preserve_fn(do_remote_query)(destination)
            for destination in remote_queries_not_in_cache
-        ], consumeErrors=True))
+        ]))

        defer.returnValue({
            "device_keys": results, "failures": failures,
@@ -238,13 +252,24 @@ class E2eKeysHandler(object):
                for user_id, keys in remote_result["one_time_keys"].items():
                    if user_id in device_keys:
                        json_result[user_id] = keys
+            except CodeMessageException as e:
+                failures[destination] = {
+                    "status": e.code, "message": e.message
+                }
+            except NotRetryingDestination as e:
+                failures[destination] = {
+                    "status": 503, "message": "Not ready for retry",
+                }
            except Exception as e:
-                failures[destination] = _exception_to_failure(e)
+                # include ConnectionRefused and other errors
+                failures[destination] = {
+                    "status": 503, "message": e.message
+                }

        yield make_deferred_yieldable(defer.gatherResults([
-            run_in_background(claim_client_keys, destination)
+            preserve_fn(claim_client_keys)(destination)
            for destination in remote_queries
-        ], consumeErrors=True))
+        ]))

        logger.info(
            "Claimed one-time-keys: %s",
@@ -337,31 +362,6 @@ class E2eKeysHandler(object):
        )


-def _exception_to_failure(e):
-    if isinstance(e, CodeMessageException):
-        return {
-            "status": e.code, "message": e.message,
-        }
-
-    if isinstance(e, NotRetryingDestination):
-        return {
-            "status": 503, "message": "Not ready for retry",
-        }
-
-    if isinstance(e, FederationDeniedError):
-        return {
-            "status": 403, "message": "Federation Denied",
-        }
-
-    # include ConnectionRefused and other errors
-    #
-    # Note that some Exceptions (notably twisted's ResponseFailed etc) don't
-    # give a string for e.message, which simplejson then fails to serialize.
-    return {
-        "status": 503, "message": str(e.message),
-    }
-
-
 def _one_time_keys_match(old_key_json, new_key):
    old_key = json.loads(old_key_json)

@@ -15,16 +15,8 @@
 # limitations under the License.

 """Contains handlers for federation events."""
-
-import itertools
-import logging
-import sys
-
 from signedjson.key import decode_verify_key_bytes
 from signedjson.sign import verify_signed_json
-import six
-from six.moves import http_client
-from twisted.internet import defer
 from unpaddedbase64 import decode_base64

 from ._base import BaseHandler
@@ -51,6 +43,10 @@ from synapse.util.retryutils import NotRetryingDestination

 from synapse.util.distributor import user_joined_room

+from twisted.internet import defer
+
+import itertools
+import logging

 logger = logging.getLogger(__name__)

@@ -72,7 +68,7 @@ class FederationHandler(BaseHandler):
        self.hs = hs

        self.store = hs.get_datastore()
-        self.replication_layer = hs.get_federation_client()
+        self.replication_layer = hs.get_replication_layer()
        self.state_handler = hs.get_state_handler()
        self.server_name = hs.hostname
        self.keyring = hs.get_keyring()
@@ -82,6 +78,8 @@ class FederationHandler(BaseHandler):
        self.spam_checker = hs.get_spam_checker()
        self.event_creation_handler = hs.get_event_creation_handler()

+        self.replication_layer.set_handler(self)
+
        # When joining a room we need to queue any events for that room up
        self.room_queues = {}
        self._room_pdu_linearizer = Linearizer("fed_room_pdu")
@@ -119,19 +117,6 @@ class FederationHandler(BaseHandler):
            logger.debug("Already seen pdu %s", pdu.event_id)
            return

-        # do some initial sanity-checking of the event. In particular, make
-        # sure it doesn't have hundreds of prev_events or auth_events, which
-        # could cause a huge state resolution or cascade of event fetches.
-        try:
-            self._sanity_check_event(pdu)
-        except SynapseError as err:
-            raise FederationError(
-                "ERROR",
-                err.code,
-                err.msg,
-                affected=pdu.event_id,
-            )
-
        # If we are currently in the process of joining this room, then we
        # queue up events for later processing.
        if pdu.room_id in self.room_queues:
@@ -166,6 +151,10 @@ class FederationHandler(BaseHandler):

        auth_chain = []

+        have_seen = yield self.store.have_events(
+            [ev for ev, _ in pdu.prev_events]
+        )
+
        fetch_state = False

        # Get missing pdus if necessary.
@@ -181,7 +170,7 @@ class FederationHandler(BaseHandler):
            )

            prevs = {e_id for e_id, _ in pdu.prev_events}
-            seen = yield self.store.have_seen_events(prevs)
+            seen = set(have_seen.keys())

            if min_depth and pdu.depth < min_depth:
                # This is so that we don't notify the user about this
@@ -209,7 +198,8 @@ class FederationHandler(BaseHandler):

                        # Update the set of things we've seen after trying to
                        # fetch the missing stuff
-                        seen = yield self.store.have_seen_events(prevs)
+                        have_seen = yield self.store.have_events(prevs)
+                        seen = set(have_seen.iterkeys())

                        if not prevs - seen:
                            logger.info(
@@ -260,7 +250,8 @@ class FederationHandler(BaseHandler):
            min_depth (int): Minimum depth of events to return.
        """
        # We recalculate seen, since it may have changed.
-        seen = yield self.store.have_seen_events(prevs)
+        have_seen = yield self.store.have_events(prevs)
+        seen = set(have_seen.keys())

        if not prevs - seen:
            return
@@ -372,7 +363,9 @@ class FederationHandler(BaseHandler):
            if auth_chain:
                event_ids |= {e.event_id for e in auth_chain}

-            seen_ids = yield self.store.have_seen_events(event_ids)
+            seen_ids = set(
+                (yield self.store.have_events(event_ids)).keys()
+            )

            if state and auth_chain is not None:
                # If we have any state or auth_chain given to us by the replication
@@ -536,16 +529,9 @@ class FederationHandler(BaseHandler):
    def backfill(self, dest, room_id, limit, extremities):
        """ Trigger a backfill request to `dest` for the given `room_id`

-        This will attempt to get more events from the remote. If the other side
-        has no new events to offer, this will return an empty list.
-
-        As the events are received, we check their signatures, and also do some
-        sanity-checking on them. If any of the backfilled events are invalid,
-        this method throws a SynapseError.
-
-        TODO: make this more useful to distinguish failures of the remote
-        server from invalid events (there is probably no point in trying to
-        re-fetch invalid events from every other HS in the room.)
+        This will attempt to get more events from the remote. This may return
+        be successfull and still return no events if the other side has no new
+        events to offer.
        """
        if dest == self.server_name:
            raise SynapseError(400, "Can't backfill from self.")
@@ -557,16 +543,6 @@ class FederationHandler(BaseHandler):
            extremities=extremities,
        )

-        # ideally we'd sanity check the events here for excess prev_events etc,
-        # but it's hard to reject events at this point without completely
-        # breaking backfill in the same way that it is currently broken by
-        # events whose signature we cannot verify (#3121).
-        #
-        # So for now we accept the events anyway. #3124 tracks this.
-        #
-        # for ev in events:
-        #     self._sanity_check_event(ev)
-
        # Don't bother processing events we already have.
        seen_events = yield self.store.have_events_in_timeline(
            set(e.event_id for e in events)
@@ -639,8 +615,7 @@ class FederationHandler(BaseHandler):

                results = yield logcontext.make_deferred_yieldable(defer.gatherResults(
                    [
-                        logcontext.run_in_background(
-                            self.replication_layer.get_pdu,
+                        logcontext.preserve_fn(self.replication_layer.get_pdu)(
                            [dest],
                            event_id,
                            outlier=True,
@@ -660,7 +635,7 @@ class FederationHandler(BaseHandler):

                failed_to_fetch = missing_auth - set(auth_events)

-        seen_events = yield self.store.have_seen_events(
+        seen_events = yield self.store.have_events(
            set(auth_events.keys()) | set(state_events.keys())
        )

@@ -870,38 +845,6 @@ class FederationHandler(BaseHandler):

        defer.returnValue(False)

-    def _sanity_check_event(self, ev):
-        """
-        Do some early sanity checks of a received event
-
-        In particular, checks it doesn't have an excessive number of
-        prev_events or auth_events, which could cause a huge state resolution
-        or cascade of event fetches.
-
-        Args:
-            ev (synapse.events.EventBase): event to be checked
-
-        Returns: None
-
-        Raises:
-            SynapseError if the event does not pass muster
-        """
-        if len(ev.prev_events) > 20:
-            logger.warn("Rejecting event %s which has %i prev_events",
-                        ev.event_id, len(ev.prev_events))
-            raise SynapseError(
-                http_client.BAD_REQUEST,
-                "Too many prev_events",
-            )
-
-        if len(ev.auth_events) > 10:
-            logger.warn("Rejecting event %s which has %i auth_events",
-                        ev.event_id, len(ev.auth_events))
-            raise SynapseError(
-                http_client.BAD_REQUEST,
-                "Too many auth_events",
-            )
-
    @defer.inlineCallbacks
    def send_invite(self, target_host, event):
        """ Sends the invite to the remote server for signing.
@@ -1026,7 +969,7 @@ class FederationHandler(BaseHandler):
            # lots of requests for missing prev_events which we do actually
            # have. Hence we fire off the deferred, but don't wait for it.

-            logcontext.run_in_background(self._handle_queued_pdus, room_queue)
+            logcontext.preserve_fn(self._handle_queued_pdus)(room_queue)

        defer.returnValue(True)

@@ -1504,33 +1447,22 @@ class FederationHandler(BaseHandler):
            auth_events=auth_events,
        )

-        try:
-            if not event.internal_metadata.is_outlier() and not backfilled:
-                yield self.action_generator.handle_push_actions_for_event(
-                    event, context
-                )
-
-            event_stream_id, max_stream_id = yield self.store.persist_event(
-                event,
-                context=context,
-                backfilled=backfilled,
-            )
-        except:  # noqa: E722, as we reraise the exception this is fine.
-            tp, value, tb = sys.exc_info()
-
-            logcontext.run_in_background(
-                self.store.remove_push_actions_from_staging,
-                event.event_id,
+        if not event.internal_metadata.is_outlier() and not backfilled:
+            yield self.action_generator.handle_push_actions_for_event(
+                event, context
            )

-            six.reraise(tp, value, tb)
+        event_stream_id, max_stream_id = yield self.store.persist_event(
+            event,
+            context=context,
+            backfilled=backfilled,
+        )

        if not backfilled:
            # this intentionally does not yield: we don't care about the result
            # and don't need to wait for it.
-            logcontext.run_in_background(
-                self.pusher_pool.on_new_notifications,
-                event_stream_id, max_stream_id,
+            logcontext.preserve_fn(self.pusher_pool.on_new_notifications)(
+                event_stream_id, max_stream_id
            )

        defer.returnValue((context, event_stream_id, max_stream_id))
@@ -1544,8 +1476,7 @@ class FederationHandler(BaseHandler):
        """
        contexts = yield logcontext.make_deferred_yieldable(defer.gatherResults(
            [
-                logcontext.run_in_background(
-                    self._prep_event,
+                logcontext.preserve_fn(self._prep_event)(
                    origin,
                    ev_info["event"],
                    state=ev_info.get("state"),
@@ -1799,8 +1730,7 @@ class FederationHandler(BaseHandler):
            event_key = None

        if event_auth_events - current_state:
-            # TODO: can we use store.have_seen_events here instead?
-            have_events = yield self.store.get_seen_events_with_rejections(
+            have_events = yield self.store.have_events(
                event_auth_events - current_state
            )
        else:
@@ -1823,12 +1753,12 @@ class FederationHandler(BaseHandler):
                    origin, event.room_id, event.event_id
                )

-                seen_remotes = yield self.store.have_seen_events(
+                seen_remotes = yield self.store.have_events(
                    [e.event_id for e in remote_auth_chain]
                )

                for e in remote_auth_chain:
-                    if e.event_id in seen_remotes:
+                    if e.event_id in seen_remotes.keys():
                        continue

                    if e.event_id == event.event_id:
@@ -1855,7 +1785,7 @@ class FederationHandler(BaseHandler):
                    except AuthError:
                        pass

-                have_events = yield self.store.get_seen_events_with_rejections(
+                have_events = yield self.store.have_events(
                    [e_id for e_id, _ in event.auth_events]
                )
                seen_events = set(have_events.keys())
@@ -1874,8 +1804,7 @@ class FederationHandler(BaseHandler):

            different_events = yield logcontext.make_deferred_yieldable(
                defer.gatherResults([
-                    logcontext.run_in_background(
-                        self.store.get_event,
+                    logcontext.preserve_fn(self.store.get_event)(
                        d,
                        allow_none=True,
                        allow_rejected=False,
@@ -1941,13 +1870,13 @@ class FederationHandler(BaseHandler):
                        local_auth_chain,
                    )

-                    seen_remotes = yield self.store.have_seen_events(
+                    seen_remotes = yield self.store.have_events(
                        [e.event_id for e in result["auth_chain"]]
                    )

                    # 3. Process any remote auth chain events we haven't seen.
                    for ev in result["auth_chain"]:
-                        if ev.event_id in seen_remotes:
+                        if ev.event_id in seen_remotes.keys():
                            continue

                        if ev.event_id == event.event_id:
@@ -2216,7 +2145,7 @@ class FederationHandler(BaseHandler):
                raise e

            yield self._check_signature(event, context)
-            member_handler = self.hs.get_room_member_handler()
+            member_handler = self.hs.get_handlers().room_member_handler
            yield member_handler.send_membership_event(None, event, context)
        else:
            destinations = set(x.split(":", 1)[-1] for x in (sender_user_id, room_id))
@@ -2260,7 +2189,7 @@ class FederationHandler(BaseHandler):
        # TODO: Make sure the signatures actually are correct.
        event.signatures.update(returned_invite.signatures)

-        member_handler = self.hs.get_room_member_handler()
+        member_handler = self.hs.get_handlers().room_member_handler
        yield member_handler.send_membership_event(None, event, context)

    @defer.inlineCallbacks
@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2017 Vector Creations Ltd
-# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -91,8 +90,6 @@ class GroupsLocalHandler(object):
    get_group_role = _create_rerouter("get_group_role")
    get_group_roles = _create_rerouter("get_group_roles")

-    set_group_join_policy = _create_rerouter("set_group_join_policy")
-
    @defer.inlineCallbacks
    def get_group_summary(self, group_id, requester_user_id):
        """Get the group summary for a group.
@@ -229,45 +226,7 @@ class GroupsLocalHandler(object):
    def join_group(self, group_id, user_id, content):
        """Request to join a group
        """
-        if self.is_mine_id(group_id):
-            yield self.groups_server_handler.join_group(
-                group_id, user_id, content
-            )
-            local_attestation = None
-            remote_attestation = None
-        else:
-            local_attestation = self.attestations.create_attestation(group_id, user_id)
-            content["attestation"] = local_attestation
-
-            res = yield self.transport_client.join_group(
-                get_domain_from_id(group_id), group_id, user_id, content,
-            )
-
-            remote_attestation = res["attestation"]
-
-            yield self.attestations.verify_attestation(
-                remote_attestation,
-                group_id=group_id,
-                user_id=user_id,
-                server_name=get_domain_from_id(group_id),
-            )
-
-        # TODO: Check that the group is public and we're being added publically
-        is_publicised = content.get("publicise", False)
-
-        token = yield self.store.register_user_group_membership(
-            group_id, user_id,
-            membership="join",
-            is_admin=False,
-            local_attestation=local_attestation,
-            remote_attestation=remote_attestation,
-            is_publicised=is_publicised,
-        )
-        self.notifier.on_new_event(
-            "groups_key", token, users=[user_id],
-        )
-
-        defer.returnValue({})
+        raise NotImplementedError()  # TODO

    @defer.inlineCallbacks
    def accept_invite(self, group_id, user_id, content):
@@ -15,11 +15,6 @@
 # limitations under the License.

 """Utilities for interacting with Identity Servers"""
-
-import logging
-
-import simplejson as json
-
 from twisted.internet import defer

 from synapse.api.errors import (
@@ -29,6 +24,9 @@ from ._base import BaseHandler
 from synapse.util.async import run_on_reactor
 from synapse.api.errors import SynapseError, Codes

+import json
+import logging
+
 logger = logging.getLogger(__name__)


@@ -27,7 +27,7 @@ from synapse.types import (
 from synapse.util import unwrapFirstError
 from synapse.util.async import concurrently_execute
 from synapse.util.caches.snapshot_cache import SnapshotCache
-from synapse.util.logcontext import make_deferred_yieldable, run_in_background
+from synapse.util.logcontext import make_deferred_yieldable, preserve_fn
 from synapse.visibility import filter_events_for_client

 from ._base import BaseHandler
@@ -166,8 +166,7 @@ class InitialSyncHandler(BaseHandler):
                (messages, token), current_state = yield make_deferred_yieldable(
                    defer.gatherResults(
                        [
-                            run_in_background(
-                                self.store.get_recent_events_for_room,
+                            preserve_fn(self.store.get_recent_events_for_room)(
                                event.room_id,
                                limit=limit,
                                end_token=room_end_token,
@@ -181,8 +180,8 @@ class InitialSyncHandler(BaseHandler):
                    self.store, user_id, messages
                )

-                start_token = now_token.copy_and_replace("room_key", token)
-                end_token = now_token.copy_and_replace("room_key", room_end_token)
+                start_token = now_token.copy_and_replace("room_key", token[0])
+                end_token = now_token.copy_and_replace("room_key", token[1])
                time_now = self.clock.time_msec()

                d["messages"] = {
@@ -325,8 +324,8 @@ class InitialSyncHandler(BaseHandler):
            self.store, user_id, messages, is_peeking=is_peeking
        )

-        start_token = StreamToken.START.copy_and_replace("room_key", token)
-        end_token = StreamToken.START.copy_and_replace("room_key", stream_token)
+        start_token = StreamToken.START.copy_and_replace("room_key", token[0])
+        end_token = StreamToken.START.copy_and_replace("room_key", token[1])

        time_now = self.clock.time_msec()

@@ -373,6 +372,7 @@ class InitialSyncHandler(BaseHandler):

        @defer.inlineCallbacks
        def get_presence():
+            defer.returnValue([])
            states = yield presence_handler.get_states(
                [m.user_id for m in room_members],
                as_event=True,
@@ -392,10 +392,9 @@ class InitialSyncHandler(BaseHandler):

        presence, receipts, (messages, token) = yield defer.gatherResults(
            [
-                run_in_background(get_presence),
-                run_in_background(get_receipts),
-                run_in_background(
-                    self.store.get_recent_events_for_room,
+                preserve_fn(get_presence)(),
+                preserve_fn(get_receipts)(),
+                preserve_fn(self.store.get_recent_events_for_room)(
                    room_id,
                    limit=limit,
                    end_token=now_token.room_key,
@@ -408,8 +407,8 @@ class InitialSyncHandler(BaseHandler):
            self.store, user_id, messages, is_peeking=is_peeking,
        )

-        start_token = now_token.copy_and_replace("room_key", token)
-        end_token = now_token
+        start_token = now_token.copy_and_replace("room_key", token[0])
+        end_token = now_token.copy_and_replace("room_key", token[1])

        time_now = self.clock.time_msec()

@@ -13,16 +13,9 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
-import simplejson
-import sys
+from twisted.internet import defer

-from canonicaljson import encode_canonical_json
-import six
-from twisted.internet import defer, reactor
-from twisted.python.failure import Failure
-
-from synapse.api.constants import EventTypes, Membership, MAX_DEPTH
+from synapse.api.constants import EventTypes, Membership
 from synapse.api.errors import AuthError, Codes, SynapseError
 from synapse.crypto.event_signing import add_hashes_and_signatures
 from synapse.events.utils import serialize_event
@@ -31,48 +24,23 @@ from synapse.types import (
    UserID, RoomAlias, RoomStreamToken,
 )
 from synapse.util.async import run_on_reactor, ReadWriteLock, Limiter
-from synapse.util.logcontext import run_in_background
+from synapse.util.logcontext import preserve_fn
 from synapse.util.metrics import measure_func
-from synapse.util.frozenutils import frozendict_json_encoder
-from synapse.util.stringutils import random_string
+from synapse.util.frozenutils import unfreeze
 from synapse.visibility import filter_events_for_client
 from synapse.replication.http.send_event import send_event_to_master

 from ._base import BaseHandler

+from canonicaljson import encode_canonical_json
+
+import logging
+import random
+import ujson
+
 logger = logging.getLogger(__name__)


-class PurgeStatus(object):
-    """Object tracking the status of a purge request
-
-    This class contains information on the progress of a purge request, for
-    return by get_purge_status.
-
-    Attributes:
-        status (int): Tracks whether this request has completed. One of
-            STATUS_{ACTIVE,COMPLETE,FAILED}
-    """
-
-    STATUS_ACTIVE = 0
-    STATUS_COMPLETE = 1
-    STATUS_FAILED = 2
-
-    STATUS_TEXT = {
-        STATUS_ACTIVE: "active",
-        STATUS_COMPLETE: "complete",
-        STATUS_FAILED: "failed",
-    }
-
-    def __init__(self):
-        self.status = PurgeStatus.STATUS_ACTIVE
-
-    def asdict(self):
-        return {
-            "status": PurgeStatus.STATUS_TEXT[self.status]
-        }
-
-
 class MessageHandler(BaseHandler):

    def __init__(self, hs):
@@ -82,87 +50,18 @@ class MessageHandler(BaseHandler):
        self.clock = hs.get_clock()

        self.pagination_lock = ReadWriteLock()
-        self._purges_in_progress_by_room = set()
-        # map from purge id to PurgeStatus
-        self._purges_by_id = {}
-
-    def start_purge_history(self, room_id, topological_ordering,
-                            delete_local_events=False):
-        """Start off a history purge on a room.
-
-        Args:
-            room_id (str): The room to purge from
-
-            topological_ordering (int): minimum topo ordering to preserve
-            delete_local_events (bool): True to delete local events as well as
-                remote ones
-
-        Returns:
-            str: unique ID for this purge transaction.
-        """
-        if room_id in self._purges_in_progress_by_room:
-            raise SynapseError(
-                400,
-                "History purge already in progress for %s" % (room_id, ),
-            )
-
-        purge_id = random_string(16)
-
-        # we log the purge_id here so that it can be tied back to the
-        # request id in the log lines.
-        logger.info("[purge] starting purge_id %s", purge_id)
-
-        self._purges_by_id[purge_id] = PurgeStatus()
-        run_in_background(
-            self._purge_history,
-            purge_id, room_id, topological_ordering, delete_local_events,
-        )
-        return purge_id

    @defer.inlineCallbacks
-    def _purge_history(self, purge_id, room_id, topological_ordering,
-                       delete_local_events):
-        """Carry out a history purge on a room.
+    def purge_history(self, room_id, event_id, delete_local_events=False):
+        event = yield self.store.get_event(event_id)

-        Args:
-            purge_id (str): The id for this purge
-            room_id (str): The room to purge from
-            topological_ordering (int): minimum topo ordering to preserve
-            delete_local_events (bool): True to delete local events as well as
-                remote ones
+        if event.room_id != room_id:
+            raise SynapseError(400, "Event is for wrong room.")

-        Returns:
-            Deferred
-        """
-        self._purges_in_progress_by_room.add(room_id)
-        try:
-            with (yield self.pagination_lock.write(room_id)):
-                yield self.store.purge_history(
-                    room_id, topological_ordering, delete_local_events,
-                )
-            logger.info("[purge] complete")
-            self._purges_by_id[purge_id].status = PurgeStatus.STATUS_COMPLETE
-        except Exception:
-            logger.error("[purge] failed: %s", Failure().getTraceback().rstrip())
-            self._purges_by_id[purge_id].status = PurgeStatus.STATUS_FAILED
-        finally:
-            self._purges_in_progress_by_room.discard(room_id)
+        depth = event.depth

-            # remove the purge from the list 24 hours after it completes
-            def clear_purge():
-                del self._purges_by_id[purge_id]
-            reactor.callLater(24 * 3600, clear_purge)
-
-    def get_purge_status(self, purge_id):
-        """Get the current status of an active purge
-
-        Args:
-            purge_id (str): purge_id returned by start_purge_history
-
-        Returns:
-            PurgeStatus|None
-        """
-        return self._purges_by_id.get(purge_id)
+        with (yield self.pagination_lock.write(room_id)):
+            yield self.store.purge_history(room_id, depth, delete_local_events)

    @defer.inlineCallbacks
    def get_messages(self, requester, room_id=None, pagin_config=None,
@@ -384,7 +283,7 @@ class MessageHandler(BaseHandler):
        # If this is an AS, double check that they are allowed to see the members.
        # This can either be because the AS user is in the room or becuase there
        # is a user in the room that the AS is "interested in"
-        if requester.app_service and user_id not in users_with_profile:
+        if False and requester.app_service and user_id not in users_with_profile:
            for uid in users_with_profile:
                if requester.app_service.is_interested_in_user(uid):
                    break
@@ -433,7 +332,7 @@ class EventCreationHandler(object):

    @defer.inlineCallbacks
    def create_event(self, requester, event_dict, token_id=None, txn_id=None,
-                     prev_events_and_hashes=None):
+                     prev_event_ids=None):
        """
        Given a dict from a client, create a new event.

@@ -447,52 +346,47 @@ class EventCreationHandler(object):
            event_dict (dict): An entire event
            token_id (str)
            txn_id (str)
-
-            prev_events_and_hashes (list[(str, dict[str, str], int)]|None):
-                the forward extremities to use as the prev_events for the
-                new event. For each event, a tuple of (event_id, hashes, depth)
-                where *hashes* is a map from algorithm to hash.
-
-                If None, they will be requested from the database.
+            prev_event_ids (list): The prev event ids to use when creating the event

        Returns:
            Tuple of created event (FrozenEvent), Context
        """
        builder = self.event_builder_factory.new(event_dict)

-        self.validator.validate_new(builder)
+        with (yield self.limiter.queue(builder.room_id)):
+            self.validator.validate_new(builder)

-        if builder.type == EventTypes.Member:
-            membership = builder.content.get("membership", None)
-            target = UserID.from_string(builder.state_key)
+            if builder.type == EventTypes.Member:
+                membership = builder.content.get("membership", None)
+                target = UserID.from_string(builder.state_key)

-            if membership in {Membership.JOIN, Membership.INVITE}:
-                # If event doesn't include a display name, add one.
-                profile = self.profile_handler
-                content = builder.content
+                if membership in {Membership.JOIN, Membership.INVITE}:
+                    # If event doesn't include a display name, add one.
+                    profile = self.profile_handler
+                    content = builder.content

-                try:
-                    if "displayname" not in content:
-                        content["displayname"] = yield profile.get_displayname(target)
-                    if "avatar_url" not in content:
-                        content["avatar_url"] = yield profile.get_avatar_url(target)
-                except Exception as e:
-                    logger.info(
-                        "Failed to get profile information for %r: %s",
-                        target, e
-                    )
+                    try:
+                        if "displayname" not in content:
+                            content["displayname"] = yield profile.get_displayname(target)
+                        if "avatar_url" not in content:
+                            content["avatar_url"] = yield profile.get_avatar_url(target)
+                    except Exception as e:
+                        logger.info(
+                            "Failed to get profile information for %r: %s",
+                            target, e
+                        )

-        if token_id is not None:
-            builder.internal_metadata.token_id = token_id
+            if token_id is not None:
+                builder.internal_metadata.token_id = token_id

-        if txn_id is not None:
-            builder.internal_metadata.txn_id = txn_id
+            if txn_id is not None:
+                builder.internal_metadata.txn_id = txn_id

-        event, context = yield self.create_new_client_event(
-            builder=builder,
-            requester=requester,
-            prev_events_and_hashes=prev_events_and_hashes,
-        )
+            event, context = yield self.create_new_client_event(
+                builder=builder,
+                requester=requester,
+                prev_event_ids=prev_event_ids,
+            )

        defer.returnValue((event, context))

@@ -562,80 +456,64 @@ class EventCreationHandler(object):

        See self.create_event and self.send_nonmember_event.
        """
+        event, context = yield self.create_event(
+            requester,
+            event_dict,
+            token_id=requester.access_token_id,
+            txn_id=txn_id
+        )

-        # We limit the number of concurrent event sends in a room so that we
-        # don't fork the DAG too much. If we don't limit then we can end up in
-        # a situation where event persistence can't keep up, causing
-        # extremities to pile up, which in turn leads to state resolution
-        # taking longer.
-        with (yield self.limiter.queue(event_dict["room_id"])):
-            event, context = yield self.create_event(
-                requester,
-                event_dict,
-                token_id=requester.access_token_id,
-                txn_id=txn_id
+        spam_error = self.spam_checker.check_event_for_spam(event)
+        if spam_error:
+            if not isinstance(spam_error, basestring):
+                spam_error = "Spam is not permitted here"
+            raise SynapseError(
+                403, spam_error, Codes.FORBIDDEN
            )

-            spam_error = self.spam_checker.check_event_for_spam(event)
-            if spam_error:
-                if not isinstance(spam_error, basestring):
-                    spam_error = "Spam is not permitted here"
-                raise SynapseError(
-                    403, spam_error, Codes.FORBIDDEN
-                )
-
-            yield self.send_nonmember_event(
-                requester,
-                event,
-                context,
-                ratelimit=ratelimit,
-            )
+        yield self.send_nonmember_event(
+            requester,
+            event,
+            context,
+            ratelimit=ratelimit,
+        )
        defer.returnValue(event)

    @measure_func("create_new_client_event")
    @defer.inlineCallbacks
-    def create_new_client_event(self, builder, requester=None,
-                                prev_events_and_hashes=None):
-        """Create a new event for a local client
-
-        Args:
-            builder (EventBuilder):
-
-            requester (synapse.types.Requester|None):
-
-            prev_events_and_hashes (list[(str, dict[str, str], int)]|None):
-                the forward extremities to use as the prev_events for the
-                new event. For each event, a tuple of (event_id, hashes, depth)
-                where *hashes* is a map from algorithm to hash.
-
-                If None, they will be requested from the database.
-
-        Returns:
-            Deferred[(synapse.events.EventBase, synapse.events.snapshot.EventContext)]
-        """
-
-        if prev_events_and_hashes is not None:
-            assert len(prev_events_and_hashes) <= 10, \
-                "Attempting to create an event with %i prev_events" % (
-                    len(prev_events_and_hashes),
+    def create_new_client_event(self, builder, requester=None, prev_event_ids=None):
+        if prev_event_ids:
+            prev_events = yield self.store.add_event_hashes(prev_event_ids)
+            prev_max_depth = yield self.store.get_max_depth_of_events(prev_event_ids)
+            depth = prev_max_depth + 1
+        else:
+            latest_ret = yield self.store.get_latest_event_ids_and_hashes_in_room(
+                builder.room_id,
            )
-        else:
-            prev_events_and_hashes = \
-                yield self.store.get_prev_events_for_room(builder.room_id)

-        if prev_events_and_hashes:
-            depth = max([d for _, _, d in prev_events_and_hashes]) + 1
-            # we cap depth of generated events, to ensure that they are not
-            # rejected by other servers (and so that they can be persisted in
-            # the db)
-            depth = min(depth, MAX_DEPTH)
-        else:
-            depth = 1
+            # We want to limit the max number of prev events we point to in our
+            # new event
+            if len(latest_ret) > 10:
+                # Sort by reverse depth, so we point to the most recent.
+                latest_ret.sort(key=lambda a: -a[2])
+                new_latest_ret = latest_ret[:5]

-        prev_events = [
-            (event_id, prev_hashes)
-            for event_id, prev_hashes, _ in prev_events_and_hashes
-        ]
+                # We also randomly point to some of the older events, to make
+                # sure that we don't completely ignore the older events.
+                if latest_ret[5:]:
+                    sample_size = min(5, len(latest_ret[5:]))
+                    new_latest_ret.extend(random.sample(latest_ret[5:], sample_size))
+                latest_ret = new_latest_ret
+
+            if latest_ret:
+                depth = max([d for _, _, d in latest_ret]) + 1
+            else:
+                depth = 1
+
+            prev_events = [
+                (event_id, prev_hashes)
+                for event_id, prev_hashes, _ in latest_ret
+            ]

        builder.prev_events = prev_events
        builder.depth = depth
@@ -675,21 +553,24 @@ class EventCreationHandler(object):
        event,
        context,
        ratelimit=True,
-        extra_users=[],
+        extra_users=[]
    ):
-        """Processes a new event. This includes checking auth, persisting it,
-        notifying users, sending to remote servers, etc.
+        # We now need to go and hit out to wherever we need to hit out to.

-        If called from a worker will hit out to the master process for final
-        processing.
+        # If we're a worker we need to hit out to the master.
+        if self.config.worker_app:
+            yield send_event_to_master(
+                self.http_client,
+                host=self.config.worker_replication_host,
+                port=self.config.worker_replication_http_port,
+                requester=requester,
+                event=event,
+                context=context,
+            )
+            return

-        Args:
-            requester (Requester)
-            event (FrozenEvent)
-            context (EventContext)
-            ratelimit (bool)
-            extra_users (list(UserID)): Any extra users to notify about event
-        """
+        if ratelimit:
+            yield self.base_handler.ratelimit(requester)

        try:
            yield self.auth.check_from_context(event, context)
@@ -699,69 +580,12 @@ class EventCreationHandler(object):

        # Ensure that we can round trip before trying to persist in db
        try:
-            dump = frozendict_json_encoder.encode(event.content)
-            simplejson.loads(dump)
+            dump = ujson.dumps(unfreeze(event.content))
+            ujson.loads(dump)
        except Exception:
            logger.exception("Failed to encode content: %r", event.content)
            raise

-        yield self.action_generator.handle_push_actions_for_event(
-            event, context
-        )
-
-        try:
-            # If we're a worker we need to hit out to the master.
-            if self.config.worker_app:
-                yield send_event_to_master(
-                    self.http_client,
-                    host=self.config.worker_replication_host,
-                    port=self.config.worker_replication_http_port,
-                    requester=requester,
-                    event=event,
-                    context=context,
-                    ratelimit=ratelimit,
-                    extra_users=extra_users,
-                )
-                return
-
-            yield self.persist_and_notify_client_event(
-                requester,
-                event,
-                context,
-                ratelimit=ratelimit,
-                extra_users=extra_users,
-            )
-        except:  # noqa: E722, as we reraise the exception this is fine.
-            # Ensure that we actually remove the entries in the push actions
-            # staging area, if we calculated them.
-            tp, value, tb = sys.exc_info()
-
-            run_in_background(
-                self.store.remove_push_actions_from_staging,
-                event.event_id,
-            )
-
-            six.reraise(tp, value, tb)
-
-    @defer.inlineCallbacks
-    def persist_and_notify_client_event(
-        self,
-        requester,
-        event,
-        context,
-        ratelimit=True,
-        extra_users=[],
-    ):
-        """Called when we have fully built the event, have already
-        calculated the push actions for the event, and checked auth.
-
-        This should only be run on master.
-        """
-        assert not self.config.worker_app
-
-        if ratelimit:
-            yield self.base_handler.ratelimit(requester)
-
        yield self.base_handler.maybe_kick_guest_users(event, context)

        if event.type == EventTypes.CanonicalAlias:
@@ -855,39 +679,32 @@ class EventCreationHandler(object):
                "Changing the room create event is forbidden",
            )

+        yield self.action_generator.handle_push_actions_for_event(
+            event, context
+        )
+
        (event_stream_id, max_stream_id) = yield self.store.persist_event(
            event, context=context
        )

        # this intentionally does not yield: we don't care about the result
        # and don't need to wait for it.
-        run_in_background(
-            self.pusher_pool.on_new_notifications,
+        preserve_fn(self.pusher_pool.on_new_notifications)(
            event_stream_id, max_stream_id
        )

        @defer.inlineCallbacks
        def _notify():
            yield run_on_reactor()
-            try:
-                self.notifier.on_new_room_event(
-                    event, event_stream_id, max_stream_id,
-                    extra_users=extra_users
-                )
-            except Exception:
-                logger.exception("Error notifying about new room event")
+            self.notifier.on_new_room_event(
+                event, event_stream_id, max_stream_id,
+                extra_users=extra_users
+            )

-        run_in_background(_notify)
+        preserve_fn(_notify)()

        if event.type == EventTypes.Message:
+            presence = self.hs.get_presence_handler()
            # We don't want to block sending messages on any presence code. This
            # matters as sometimes presence code can take a while.
-            run_in_background(self._bump_active_time, requester.user)
-
-    @defer.inlineCallbacks
-    def _bump_active_time(self, user):
-        try:
-            presence = self.hs.get_presence_handler()
-            yield presence.bump_presence_active_time(user)
-        except Exception:
-            logger.exception("Error bumping presence active time")
+            preserve_fn(presence.bump_presence_active_time)(requester.user)
@@ -31,7 +31,7 @@ from synapse.storage.presence import UserPresenceState

 from synapse.util.caches.descriptors import cachedInlineCallbacks
 from synapse.util.async import Linearizer
-from synapse.util.logcontext import run_in_background
+from synapse.util.logcontext import preserve_fn
 from synapse.util.logutils import log_function
 from synapse.util.metrics import Measure
 from synapse.util.wheel_timer import WheelTimer
@@ -93,30 +93,29 @@ class PresenceHandler(object):
        self.store = hs.get_datastore()
        self.wheel_timer = WheelTimer()
        self.notifier = hs.get_notifier()
+        self.replication = hs.get_replication_layer()
        self.federation = hs.get_federation_sender()

        self.state = hs.get_state_handler()

-        federation_registry = hs.get_federation_registry()
-
-        federation_registry.register_edu_handler(
+        self.replication.register_edu_handler(
            "m.presence", self.incoming_presence
        )
-        federation_registry.register_edu_handler(
+        self.replication.register_edu_handler(
            "m.presence_invite",
            lambda origin, content: self.invite_presence(
                observed_user=UserID.from_string(content["observed_user"]),
                observer_user=UserID.from_string(content["observer_user"]),
            )
        )
-        federation_registry.register_edu_handler(
+        self.replication.register_edu_handler(
            "m.presence_accept",
            lambda origin, content: self.accept_presence(
                observed_user=UserID.from_string(content["observed_user"]),
                observer_user=UserID.from_string(content["observer_user"]),
            )
        )
-        federation_registry.register_edu_handler(
+        self.replication.register_edu_handler(
            "m.presence_deny",
            lambda origin, content: self.deny_presence(
                observed_user=UserID.from_string(content["observed_user"]),
@@ -254,14 +253,6 @@ class PresenceHandler(object):

        logger.info("Finished _persist_unpersisted_changes")

-    @defer.inlineCallbacks
-    def _update_states_and_catch_exception(self, new_states):
-        try:
-            res = yield self._update_states(new_states)
-            defer.returnValue(res)
-        except Exception:
-            logger.exception("Error updating presence")
-
    @defer.inlineCallbacks
    def _update_states(self, new_states):
        """Updates presence of users. Sets the appropriate timeouts. Pokes
@@ -372,7 +363,7 @@ class PresenceHandler(object):
                    now=now,
                )

-            run_in_background(self._update_states_and_catch_exception, changes)
+            preserve_fn(self._update_states)(changes)
        except Exception:
            logger.exception("Exception in _handle_timeouts loop")

@@ -381,6 +372,7 @@ class PresenceHandler(object):
        """We've seen the user do something that indicates they're interacting
        with the app.
        """
+        return
        user_id = user.to_string()

        bump_active_time_counter.inc()
@@ -410,6 +402,7 @@ class PresenceHandler(object):
                Useful for streams that are not associated with an actual
                client that is being used by a user.
        """
+        affect_presence = False
        if affect_presence:
            curr_sync = self.user_to_num_current_syncs.get(user_id, 0)
            self.user_to_num_current_syncs[user_id] = curr_sync + 1
@@ -430,23 +423,20 @@ class PresenceHandler(object):

        @defer.inlineCallbacks
        def _end():
-            try:
+            if affect_presence:
                self.user_to_num_current_syncs[user_id] -= 1

                prev_state = yield self.current_state_for_user(user_id)
                yield self._update_states([prev_state.copy_and_replace(
                    last_user_sync_ts=self.clock.time_msec(),
                )])
-            except Exception:
-                logger.exception("Error updating presence after sync")

        @contextmanager
        def _user_syncing():
            try:
                yield
            finally:
-                if affect_presence:
-                    run_in_background(_end)
+                preserve_fn(_end)()

        defer.returnValue(_user_syncing())

@@ -455,6 +445,8 @@ class PresenceHandler(object):
        Returns:
            set(str): A set of user_id strings.
        """
+        # presence is disabled on matrix.org, so we return the empty set
+        return set()
        syncing_user_ids = {
            user_id for user_id, count in self.user_to_num_current_syncs.items()
            if count
@@ -474,6 +466,7 @@ class PresenceHandler(object):
            syncing_user_ids(set(str)): The set of user_ids that are
                currently syncing on that server.
        """
+        return

        # Grab the previous list of user_ids that were syncing on that process
        prev_syncing_user_ids = (
@@ -31,17 +31,14 @@ class ProfileHandler(BaseHandler):
    def __init__(self, hs):
        super(ProfileHandler, self).__init__(hs)

-        self.federation = hs.get_federation_client()
-        hs.get_federation_registry().register_query_handler(
+        self.federation = hs.get_replication_layer()
+        self.federation.register_query_handler(
            "profile", self.on_profile_query
        )

        self.user_directory_handler = hs.get_user_directory_handler()

-        if hs.config.worker_app is None:
-            self.clock.looping_call(
-                self._update_remote_profile_cache, self.PROFILE_UPDATE_MS,
-            )
+        self.clock.looping_call(self._update_remote_profile_cache, self.PROFILE_UPDATE_MS)

    @defer.inlineCallbacks
    def get_profile(self, user_id):
@@ -236,7 +233,7 @@ class ProfileHandler(BaseHandler):
        )

        for room_id in room_ids:
-            handler = self.hs.get_room_member_handler()
+            handler = self.hs.get_handlers().room_member_handler
            try:
                # Assume the target_user isn't a guest,
                # because we don't let guests set profile or avatar data.
@@ -41,9 +41,9 @@ class ReadMarkerHandler(BaseHandler):
        """

        with (yield self.read_marker_linearizer.queue((room_id, user_id))):
-            existing_read_marker = yield self.store.get_account_data_for_room_and_type(
-                user_id, room_id, "m.fully_read",
-            )
+            account_data = yield self.store.get_account_data_for_room(user_id, room_id)
+
+            existing_read_marker = account_data.get("m.fully_read", None)

            should_update = True

@@ -35,7 +35,7 @@ class ReceiptsHandler(BaseHandler):
        self.store = hs.get_datastore()
        self.hs = hs
        self.federation = hs.get_federation_sender()
-        hs.get_federation_registry().register_edu_handler(
+        hs.get_replication_layer().register_edu_handler(
            "m.receipt", self._received_remote_receipt
        )
        self.clock = self.hs.get_clock()
@@ -135,40 +135,37 @@ class ReceiptsHandler(BaseHandler):
        """Given a list of receipts, works out which remote servers should be
        poked and pokes them.
        """
-        try:
-            # TODO: Some of this stuff should be coallesced.
-            for receipt in receipts:
-                room_id = receipt["room_id"]
-                receipt_type = receipt["receipt_type"]
-                user_id = receipt["user_id"]
-                event_ids = receipt["event_ids"]
-                data = receipt["data"]
+        # TODO: Some of this stuff should be coallesced.
+        for receipt in receipts:
+            room_id = receipt["room_id"]
+            receipt_type = receipt["receipt_type"]
+            user_id = receipt["user_id"]
+            event_ids = receipt["event_ids"]
+            data = receipt["data"]

-                users = yield self.state.get_current_user_in_room(room_id)
-                remotedomains = set(get_domain_from_id(u) for u in users)
-                remotedomains = remotedomains.copy()
-                remotedomains.discard(self.server_name)
+            users = yield self.state.get_current_user_in_room(room_id)
+            remotedomains = set(get_domain_from_id(u) for u in users)
+            remotedomains = remotedomains.copy()
+            remotedomains.discard(self.server_name)

-                logger.debug("Sending receipt to: %r", remotedomains)
+            logger.debug("Sending receipt to: %r", remotedomains)

-                for domain in remotedomains:
-                    self.federation.send_edu(
-                        destination=domain,
-                        edu_type="m.receipt",
-                        content={
-                            room_id: {
-                                receipt_type: {
-                                    user_id: {
-                                        "event_ids": event_ids,
-                                        "data": data,
-                                    }
+            for domain in remotedomains:
+                self.federation.send_edu(
+                    destination=domain,
+                    edu_type="m.receipt",
+                    content={
+                        room_id: {
+                            receipt_type: {
+                                user_id: {
+                                    "event_ids": event_ids,
+                                    "data": data,
                                }
-                            },
+                            }
                        },
-                        key=(room_id, receipt_type, user_id),
-                    )
-        except Exception:
-            logger.exception("Error pushing receipts to remote servers")
+                    },
+                    key=(room_id, receipt_type, user_id),
+                )

    @defer.inlineCallbacks
    def get_receipts_for_room(self, room_id, to_key):
@@ -23,8 +23,8 @@ from synapse.api.errors import (
 )
 from synapse.http.client import CaptchaServerHttpClient
 from synapse import types
-from synapse.types import UserID, create_requester, RoomID, RoomAlias
-from synapse.util.async import run_on_reactor, Linearizer
+from synapse.types import UserID
+from synapse.util.async import run_on_reactor
 from synapse.util.threepids import check_3pid_allowed
 from ._base import BaseHandler

@@ -46,10 +46,6 @@ class RegistrationHandler(BaseHandler):

        self.macaroon_gen = hs.get_macaroon_generator()

-        self._generate_user_id_linearizer = Linearizer(
-            name="_generate_user_id_linearizer",
-        )
-
    @defer.inlineCallbacks
    def check_username(self, localpart, guest_access_token=None,
                       assigned_user_id=None):
@@ -205,17 +201,10 @@ class RegistrationHandler(BaseHandler):
                    token = None
                    attempts += 1

-        # auto-join the user to any rooms we're supposed to dump them into
-        fake_requester = create_requester(user_id)
-        for r in self.hs.config.auto_join_rooms:
-            try:
-                yield self._join_user_to_room(fake_requester, r)
-            except Exception as e:
-                logger.error("Failed to join new user to %r: %r", r, e)
-
        # We used to generate default identicons here, but nowadays
        # we want clients to generate their own as part of their branding
        # rather than there being consistent matrix-wide ones, so we don't.
+
        defer.returnValue((user_id, token))

    @defer.inlineCallbacks
@@ -356,11 +345,9 @@ class RegistrationHandler(BaseHandler):
    @defer.inlineCallbacks
    def _generate_user_id(self, reseed=False):
        if reseed or self._next_generated_user_id is None:
-            with (yield self._generate_user_id_linearizer.queue(())):
-                if reseed or self._next_generated_user_id is None:
-                    self._next_generated_user_id = (
-                        yield self.store.find_next_generated_user_id_localpart()
-                    )
+            self._next_generated_user_id = (
+                yield self.store.find_next_generated_user_id_localpart()
+            )

        id = self._next_generated_user_id
        self._next_generated_user_id += 1
@@ -459,59 +446,16 @@ class RegistrationHandler(BaseHandler):
        return self.hs.get_auth_handler()

    @defer.inlineCallbacks
-    def get_or_register_3pid_guest(self, medium, address, inviter_user_id):
-        """Get a guest access token for a 3PID, creating a guest account if
-        one doesn't already exist.
-
-        Args:
-            medium (str)
-            address (str)
-            inviter_user_id (str): The user ID who is trying to invite the
-                3PID
-
-        Returns:
-            Deferred[(str, str)]: A 2-tuple of `(user_id, access_token)` of the
-            3PID guest account.
-        """
+    def guest_access_token_for(self, medium, address, inviter_user_id):
        access_token = yield self.store.get_3pid_guest_access_token(medium, address)
        if access_token:
-            user_info = yield self.auth.get_user_by_access_token(
-                access_token
-            )
+            defer.returnValue(access_token)

-            defer.returnValue((user_info["user"].to_string(), access_token))
-
-        user_id, access_token = yield self.register(
+        _, access_token = yield self.register(
            generate_token=True,
            make_guest=True
        )
        access_token = yield self.store.save_or_get_3pid_guest_access_token(
            medium, address, access_token, inviter_user_id
        )
-
-        defer.returnValue((user_id, access_token))
-
-    @defer.inlineCallbacks
-    def _join_user_to_room(self, requester, room_identifier):
-        room_id = None
-        room_member_handler = self.hs.get_room_member_handler()
-        if RoomID.is_valid(room_identifier):
-            room_id = room_identifier
-        elif RoomAlias.is_valid(room_identifier):
-            room_alias = RoomAlias.from_string(room_identifier)
-            room_id, remote_room_hosts = (
-                yield room_member_handler.lookup_room_alias(room_alias)
-            )
-            room_id = room_id.to_string()
-        else:
-            raise SynapseError(400, "%s was not legal room ID or room alias" % (
-                room_identifier,
-            ))
-
-        yield room_member_handler.update_membership(
-            requester=requester,
-            target=requester.user,
-            room_id=room_id,
-            remote_room_hosts=remote_room_hosts,
-            action="join",
-        )
+        defer.returnValue(access_token)
@@ -165,7 +165,7 @@ class RoomCreationHandler(BaseHandler):

        creation_content = config.get("creation_content", {})

-        room_member_handler = self.hs.get_room_member_handler()
+        room_member_handler = self.hs.get_handlers().room_member_handler

        yield self._send_events_for_new_room(
            requester,
@@ -224,7 +224,7 @@ class RoomCreationHandler(BaseHandler):
            id_server = invite_3pid["id_server"]
            address = invite_3pid["address"]
            medium = invite_3pid["medium"]
-            yield self.hs.get_room_member_handler().do_3pid_invite(
+            yield self.hs.get_handlers().room_member_handler.do_3pid_invite(
                room_id,
                requester.user,
                medium,
@@ -475,9 +475,12 @@ class RoomEventSource(object):
            user.to_string()
        )
        if app_service:
-            # We no longer support AS users using /sync directly.
-            # See https://github.com/matrix-org/matrix-doc/issues/1144
-            raise NotImplementedError()
+            events, end_key = yield self.store.get_appservice_room_stream(
+                service=app_service,
+                from_key=from_key,
+                to_key=to_key,
+                limit=limit,
+            )
        else:
            room_events = yield self.store.get_membership_changes_for_user(
                user.to_string(), from_key, to_key
@@ -15,13 +15,12 @@

 from twisted.internet import defer

-from six.moves import range
-
 from ._base import BaseHandler

 from synapse.api.constants import (
    EventTypes, JoinRules,
 )
+from synapse.util.logcontext import make_deferred_yieldable, preserve_fn
 from synapse.util.async import concurrently_execute
 from synapse.util.caches.descriptors import cachedInlineCallbacks
 from synapse.util.caches.response_cache import ResponseCache
@@ -45,9 +44,8 @@ EMTPY_THIRD_PARTY_ID = ThirdPartyInstanceID(None, None)
 class RoomListHandler(BaseHandler):
    def __init__(self, hs):
        super(RoomListHandler, self).__init__(hs)
-        self.response_cache = ResponseCache(hs, "room_list")
-        self.remote_response_cache = ResponseCache(hs, "remote_room_list",
-                                                   timeout_ms=30 * 1000)
+        self.response_cache = ResponseCache(hs, timeout_ms=10 * 60 * 1000)
+        self.remote_response_cache = ResponseCache(hs, timeout_ms=30 * 1000)

    def get_local_public_room_list(self, limit=None, since_token=None,
                                   search_filter=None,
@@ -79,11 +77,18 @@ class RoomListHandler(BaseHandler):
            )

        key = (limit, since_token, network_tuple)
-        return self.response_cache.wrap(
-            key,
-            self._get_public_room_list,
-            limit, since_token, network_tuple=network_tuple,
-        )
+        result = self.response_cache.get(key)
+        if not result:
+            logger.info("No cached result, calculating one.")
+            result = self.response_cache.set(
+                key,
+                preserve_fn(self._get_public_room_list)(
+                    limit, since_token, network_tuple=network_tuple
+                )
+            )
+        else:
+            logger.info("Using cached deferred result.")
+        return make_deferred_yieldable(result)

    @defer.inlineCallbacks
    def _get_public_room_list(self, limit=None, since_token=None,
@@ -202,7 +207,7 @@ class RoomListHandler(BaseHandler):
            step = len(rooms_to_scan) if len(rooms_to_scan) != 0 else 1

        chunk = []
-        for i in range(0, len(rooms_to_scan), step):
+        for i in xrange(0, len(rooms_to_scan), step):
            batch = rooms_to_scan[i:i + step]
            logger.info("Processing %i rooms for result", len(batch))
            yield concurrently_execute(
@@ -404,7 +409,7 @@ class RoomListHandler(BaseHandler):
    def _get_remote_list_cached(self, server_name, limit=None, since_token=None,
                                search_filter=None, include_all_networks=False,
                                third_party_instance_id=None,):
-        repl_layer = self.hs.get_federation_client()
+        repl_layer = self.hs.get_replication_layer()
        if search_filter:
            # We can't cache when asking for search
            return repl_layer.get_public_rooms(
@@ -417,14 +422,18 @@ class RoomListHandler(BaseHandler):
            server_name, limit, since_token, include_all_networks,
            third_party_instance_id,
        )
-        return self.remote_response_cache.wrap(
-            key,
-            repl_layer.get_public_rooms,
-            server_name, limit=limit, since_token=since_token,
-            search_filter=search_filter,
-            include_all_networks=include_all_networks,
-            third_party_instance_id=third_party_instance_id,
-        )
+        result = self.remote_response_cache.get(key)
+        if not result:
+            result = self.remote_response_cache.set(
+                key,
+                repl_layer.get_public_rooms(
+                    server_name, limit=limit, since_token=since_token,
+                    search_filter=search_filter,
+                    include_all_networks=include_all_networks,
+                    third_party_instance_id=third_party_instance_id,
+                )
+            )
+        return result


 class RoomListNextBatch(namedtuple("RoomListNextBatch", (
@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-import abc
+
 import logging

 from signedjson.key import decode_verify_key_bytes
@@ -28,128 +28,41 @@ from synapse.api.constants import (
 )
 from synapse.api.errors import AuthError, SynapseError, Codes
 from synapse.types import UserID, RoomID
-from synapse.util.async import Linearizer
+from synapse.util.async import Linearizer, Limiter
 from synapse.util.distributor import user_left_room, user_joined_room
-
+from ._base import BaseHandler

 logger = logging.getLogger(__name__)

 id_server_scheme = "https://"


-class RoomMemberHandler(object):
+class RoomMemberHandler(BaseHandler):
    # TODO(paul): This handler currently contains a messy conflation of
    #   low-level API that works on UserID objects and so on, and REST-level
    #   API that takes ID strings and returns pagination chunks. These concerns
    #   ought to be separated out a lot better.

-    __metaclass__ = abc.ABCMeta
-
    def __init__(self, hs):
-        self.hs = hs
-        self.store = hs.get_datastore()
-        self.auth = hs.get_auth()
-        self.state_handler = hs.get_state_handler()
-        self.config = hs.config
-        self.simple_http_client = hs.get_simple_http_client()
+        super(RoomMemberHandler, self).__init__(hs)

-        self.federation_handler = hs.get_handlers().federation_handler
-        self.directory_handler = hs.get_handlers().directory_handler
-        self.registration_handler = hs.get_handlers().registration_handler
        self.profile_handler = hs.get_profile_handler()
        self.event_creation_hander = hs.get_event_creation_handler()

        self.member_linearizer = Linearizer(name="member")
+        self.member_limiter = Limiter(3)

        self.clock = hs.get_clock()
        self.spam_checker = hs.get_spam_checker()

-    @abc.abstractmethod
-    def _remote_join(self, requester, remote_room_hosts, room_id, user, content):
-        """Try and join a room that this server is not in
-
-        Args:
-            requester (Requester)
-            remote_room_hosts (list[str]): List of servers that can be used
-                to join via.
-            room_id (str): Room that we are trying to join
-            user (UserID): User who is trying to join
-            content (dict): A dict that should be used as the content of the
-                join event.
-
-        Returns:
-            Deferred
-        """
-        raise NotImplementedError()
-
-    @abc.abstractmethod
-    def _remote_reject_invite(self, remote_room_hosts, room_id, target):
-        """Attempt to reject an invite for a room this server is not in. If we
-        fail to do so we locally mark the invite as rejected.
-
-        Args:
-            requester (Requester)
-            remote_room_hosts (list[str]): List of servers to use to try and
-                reject invite
-            room_id (str)
-            target (UserID): The user rejecting the invite
-
-        Returns:
-            Deferred[dict]: A dictionary to be returned to the client, may
-            include event_id etc, or nothing if we locally rejected
-        """
-        raise NotImplementedError()
-
-    @abc.abstractmethod
-    def get_or_register_3pid_guest(self, requester, medium, address, inviter_user_id):
-        """Get a guest access token for a 3PID, creating a guest account if
-        one doesn't already exist.
-
-        Args:
-            requester (Requester)
-            medium (str)
-            address (str)
-            inviter_user_id (str): The user ID who is trying to invite the
-                3PID
-
-        Returns:
-            Deferred[(str, str)]: A 2-tuple of `(user_id, access_token)` of the
-            3PID guest account.
-        """
-        raise NotImplementedError()
-
-    @abc.abstractmethod
-    def _user_joined_room(self, target, room_id):
-        """Notifies distributor on master process that the user has joined the
-        room.
-
-        Args:
-            target (UserID)
-            room_id (str)
-
-        Returns:
-            Deferred|None
-        """
-        raise NotImplementedError()
-
-    @abc.abstractmethod
-    def _user_left_room(self, target, room_id):
-        """Notifies distributor on master process that the user has left the
-        room.
-
-        Args:
-            target (UserID)
-            room_id (str)
-
-        Returns:
-            Deferred|None
-        """
-        raise NotImplementedError()
+        self.distributor = hs.get_distributor()
+        self.distributor.declare("user_joined_room")
+        self.distributor.declare("user_left_room")

    @defer.inlineCallbacks
    def _local_membership_update(
        self, requester, target, room_id, membership,
-        prev_events_and_hashes,
+        prev_event_ids,
        txn_id=None,
        ratelimit=True,
        content=None,
@@ -175,7 +88,7 @@ class RoomMemberHandler(object):
            },
            token_id=requester.access_token_id,
            txn_id=txn_id,
-            prev_events_and_hashes=prev_events_and_hashes,
+            prev_event_ids=prev_event_ids,
        )

        # Check if this event matches the previous membership event for the user.
@@ -208,15 +121,32 @@ class RoomMemberHandler(object):
                prev_member_event = yield self.store.get_event(prev_member_event_id)
                newly_joined = prev_member_event.membership != Membership.JOIN
            if newly_joined:
-                yield self._user_joined_room(target, room_id)
+                yield user_joined_room(self.distributor, target, room_id)
        elif event.membership == Membership.LEAVE:
            if prev_member_event_id:
                prev_member_event = yield self.store.get_event(prev_member_event_id)
                if prev_member_event.membership == Membership.JOIN:
-                    yield self._user_left_room(target, room_id)
+                    user_left_room(self.distributor, target, room_id)

        defer.returnValue(event)

+    @defer.inlineCallbacks
+    def remote_join(self, remote_room_hosts, room_id, user, content):
+        if len(remote_room_hosts) == 0:
+            raise SynapseError(404, "No known servers")
+
+        # We don't do an auth check if we are doing an invite
+        # join dance for now, since we're kinda implicitly checking
+        # that we are allowed to join when we decide whether or not we
+        # need to do the invite/join dance.
+        yield self.hs.get_handlers().federation_handler.do_invite_join(
+            remote_room_hosts,
+            room_id,
+            user.to_string(),
+            content,
+        )
+        yield user_joined_room(self.distributor, user, room_id)
+
    @defer.inlineCallbacks
    def update_membership(
            self,
@@ -232,18 +162,23 @@ class RoomMemberHandler(object):
    ):
        key = (room_id,)

-        with (yield self.member_linearizer.queue(key)):
-            result = yield self._update_membership(
-                requester,
-                target,
-                room_id,
-                action,
-                txn_id=txn_id,
-                remote_room_hosts=remote_room_hosts,
-                third_party_signed=third_party_signed,
-                ratelimit=ratelimit,
-                content=content,
-            )
+        as_id = object()
+        if requester.app_service:
+            as_id = requester.app_service.id
+
+        with (yield self.member_limiter.queue(as_id)):
+            with (yield self.member_linearizer.queue(key)):
+                result = yield self._update_membership(
+                    requester,
+                    target,
+                    room_id,
+                    action,
+                    txn_id=txn_id,
+                    remote_room_hosts=remote_room_hosts,
+                    third_party_signed=third_party_signed,
+                    ratelimit=ratelimit,
+                    content=content,
+                )

        defer.returnValue(result)

@@ -275,7 +210,8 @@ class RoomMemberHandler(object):
        # if this is a join with a 3pid signature, we may need to turn a 3pid
        # invite into a normal invite before we can handle the join.
        if third_party_signed is not None:
-            yield self.federation_handler.exchange_third_party_invite(
+            replication = self.hs.get_replication_layer()
+            yield replication.exchange_third_party_invite(
                third_party_signed["sender"],
                target.to_string(),
                room_id,
@@ -296,7 +232,7 @@ class RoomMemberHandler(object):
                requester.user,
            )
            if not is_requester_admin:
-                if self.config.block_non_admin_invites:
+                if self.hs.config.block_non_admin_invites:
                    logger.info(
                        "Blocking invite: user is not admin and non-admin "
                        "invites disabled"
@@ -314,12 +250,7 @@ class RoomMemberHandler(object):
                    403, "Invites have been disabled on this server",
                )

-        prev_events_and_hashes = yield self.store.get_prev_events_for_room(
-            room_id,
-        )
-        latest_event_ids = (
-            event_id for (event_id, _, _) in prev_events_and_hashes
-        )
+        latest_event_ids = yield self.store.get_latest_event_ids_in_room(room_id)
        current_state_ids = yield self.state_handler.get_current_state_ids(
            room_id, latest_event_ids=latest_event_ids,
        )
@@ -360,7 +291,7 @@ class RoomMemberHandler(object):
                    raise AuthError(403, "Guest access not allowed")

            if not is_host_in_room:
-                inviter = yield self._get_inviter(target.to_string(), room_id)
+                inviter = yield self.get_inviter(target.to_string(), room_id)
                if inviter and not self.hs.is_mine(inviter):
                    remote_room_hosts.append(inviter.domain)

@@ -374,15 +305,15 @@ class RoomMemberHandler(object):
                if requester.is_guest:
                    content["kind"] = "guest"

-                ret = yield self._remote_join(
-                    requester, remote_room_hosts, room_id, target, content
+                ret = yield self.remote_join(
+                    remote_room_hosts, room_id, target, content
                )
                defer.returnValue(ret)

        elif effective_membership_state == Membership.LEAVE:
            if not is_host_in_room:
                # perhaps we've been invited
-                inviter = yield self._get_inviter(target.to_string(), room_id)
+                inviter = yield self.get_inviter(target.to_string(), room_id)
                if not inviter:
                    raise SynapseError(404, "Not a known room")

@@ -396,10 +327,28 @@ class RoomMemberHandler(object):
                else:
                    # send the rejection to the inviter's HS.
                    remote_room_hosts = remote_room_hosts + [inviter.domain]
-                    res = yield self._remote_reject_invite(
-                        requester, remote_room_hosts, room_id, target,
-                    )
-                    defer.returnValue(res)
+                    fed_handler = self.hs.get_handlers().federation_handler
+                    try:
+                        ret = yield fed_handler.do_remotely_reject_invite(
+                            remote_room_hosts,
+                            room_id,
+                            target.to_string(),
+                        )
+                        defer.returnValue(ret)
+                    except Exception as e:
+                        # if we were unable to reject the exception, just mark
+                        # it as rejected on our end and plough ahead.
+                        #
+                        # The 'except' clause is very broad, but we need to
+                        # capture everything from DNS failures upwards
+                        #
+                        logger.warn("Failed to reject invite: %s", e)
+
+                        yield self.store.locally_reject_invite(
+                            target.to_string(), room_id
+                        )
+
+                        defer.returnValue({})

        res = yield self._local_membership_update(
            requester=requester,
@@ -408,7 +357,7 @@ class RoomMemberHandler(object):
            membership=effective_membership_state,
            txn_id=txn_id,
            ratelimit=ratelimit,
-            prev_events_and_hashes=prev_events_and_hashes,
+            prev_event_ids=latest_event_ids,
            content=content,
        )
        defer.returnValue(res)
@@ -495,12 +444,12 @@ class RoomMemberHandler(object):
                prev_member_event = yield self.store.get_event(prev_member_event_id)
                newly_joined = prev_member_event.membership != Membership.JOIN
            if newly_joined:
-                yield self._user_joined_room(target_user, room_id)
+                yield user_joined_room(self.distributor, target_user, room_id)
        elif event.membership == Membership.LEAVE:
            if prev_member_event_id:
                prev_member_event = yield self.store.get_event(prev_member_event_id)
                if prev_member_event.membership == Membership.JOIN:
-                    yield self._user_left_room(target_user, room_id)
+                    user_left_room(self.distributor, target_user, room_id)

    @defer.inlineCallbacks
    def _can_guest_join(self, current_state_ids):
@@ -534,7 +483,7 @@ class RoomMemberHandler(object):
        Raises:
            SynapseError if room alias could not be found.
        """
-        directory_handler = self.directory_handler
+        directory_handler = self.hs.get_handlers().directory_handler
        mapping = yield directory_handler.get_association(room_alias)

        if not mapping:
@@ -546,7 +495,7 @@ class RoomMemberHandler(object):
        defer.returnValue((RoomID.from_string(room_id), servers))

    @defer.inlineCallbacks
-    def _get_inviter(self, user_id, room_id):
+    def get_inviter(self, user_id, room_id):
        invite = yield self.store.get_invite_for_user_in_room(
            user_id=user_id,
            room_id=room_id,
@@ -565,7 +514,7 @@ class RoomMemberHandler(object):
            requester,
            txn_id
    ):
-        if self.config.block_non_admin_invites:
+        if self.hs.config.block_non_admin_invites:
            is_requester_admin = yield self.auth.is_server_admin(
                requester.user,
            )
@@ -612,7 +561,7 @@ class RoomMemberHandler(object):
            str: the matrix ID of the 3pid, or None if it is not recognized.
        """
        try:
-            data = yield self.simple_http_client.get_json(
+            data = yield self.hs.get_simple_http_client().get_json(
                "%s%s/_matrix/identity/api/v1/lookup" % (id_server_scheme, id_server,),
                {
                    "medium": medium,
@@ -623,7 +572,7 @@ class RoomMemberHandler(object):
            if "mxid" in data:
                if "signatures" not in data:
                    raise AuthError(401, "No signatures on 3pid binding")
-                yield self._verify_any_signature(data, id_server)
+                self.verify_any_signature(data, id_server)
                defer.returnValue(data["mxid"])

        except IOError as e:
@@ -631,11 +580,11 @@ class RoomMemberHandler(object):
            defer.returnValue(None)

    @defer.inlineCallbacks
-    def _verify_any_signature(self, data, server_hostname):
+    def verify_any_signature(self, data, server_hostname):
        if server_hostname not in data["signatures"]:
            raise AuthError(401, "No signature from server %s" % (server_hostname,))
        for key_name, signature in data["signatures"][server_hostname].items():
-            key_data = yield self.simple_http_client.get_json(
+            key_data = yield self.hs.get_simple_http_client().get_json(
                "%s%s/_matrix/identity/api/v1/pubkey/%s" %
                (id_server_scheme, server_hostname, key_name,),
            )
@@ -660,7 +609,7 @@ class RoomMemberHandler(object):
            user,
            txn_id
    ):
-        room_state = yield self.state_handler.get_current_state(room_id)
+        room_state = yield self.hs.get_state_handler().get_current_state(room_id)

        inviter_display_name = ""
        inviter_avatar_url = ""
@@ -691,7 +640,6 @@ class RoomMemberHandler(object):

        token, public_keys, fallback_public_key, display_name = (
            yield self._ask_id_server_for_third_party_invite(
-                requester=requester,
                id_server=id_server,
                medium=medium,
                address=address,
@@ -728,7 +676,6 @@ class RoomMemberHandler(object):
    @defer.inlineCallbacks
    def _ask_id_server_for_third_party_invite(
            self,
-            requester,
            id_server,
            medium,
            address,
@@ -745,7 +692,6 @@ class RoomMemberHandler(object):
        Asks an identity server for a third party invite.

        Args:
-            requester (Requester)
            id_server (str): hostname + optional port for the identity server.
            medium (str): The literal string "email".
            address (str): The third party address being invited.
@@ -787,20 +733,24 @@ class RoomMemberHandler(object):
            "sender_avatar_url": inviter_avatar_url,
        }

-        if self.config.invite_3pid_guest:
-            guest_access_token, guest_user_id = yield self.get_or_register_3pid_guest(
-                requester=requester,
+        if self.hs.config.invite_3pid_guest:
+            registration_handler = self.hs.get_handlers().registration_handler
+            guest_access_token = yield registration_handler.guest_access_token_for(
                medium=medium,
                address=address,
                inviter_user_id=inviter_user_id,
            )

+            guest_user_info = yield self.hs.get_auth().get_user_by_access_token(
+                guest_access_token
+            )
+
            invite_config.update({
                "guest_access_token": guest_access_token,
-                "guest_user_id": guest_user_id,
+                "guest_user_id": guest_user_info["user"].to_string(),
            })

-        data = yield self.simple_http_client.post_urlencoded_get_json(
+        data = yield self.hs.get_simple_http_client().post_urlencoded_get_json(
            is_url,
            invite_config
        )
@@ -822,6 +772,27 @@ class RoomMemberHandler(object):
        display_name = data["display_name"]
        defer.returnValue((token, public_keys, fallback_public_key, display_name))

+    @defer.inlineCallbacks
+    def forget(self, user, room_id):
+        user_id = user.to_string()
+
+        member = yield self.state_handler.get_current_state(
+            room_id=room_id,
+            event_type=EventTypes.Member,
+            state_key=user_id
+        )
+        membership = member.membership if member else None
+
+        if membership is not None and membership not in [
+            Membership.LEAVE, Membership.BAN
+        ]:
+            raise SynapseError(400, "User %s in room %s" % (
+                user_id, room_id
+            ))
+
+        if membership:
+            yield self.store.forget(user_id, room_id)
+
    @defer.inlineCallbacks
    def _is_host_in_room(self, current_state_ids):
        # Have we just created the room, and is this about to be the very
@@ -843,102 +814,3 @@ class RoomMemberHandler(object):
                defer.returnValue(True)

        defer.returnValue(False)
-
-
-class RoomMemberMasterHandler(RoomMemberHandler):
-    def __init__(self, hs):
-        super(RoomMemberMasterHandler, self).__init__(hs)
-
-        self.distributor = hs.get_distributor()
-        self.distributor.declare("user_joined_room")
-        self.distributor.declare("user_left_room")
-
-    @defer.inlineCallbacks
-    def _remote_join(self, requester, remote_room_hosts, room_id, user, content):
-        """Implements RoomMemberHandler._remote_join
-        """
-        # filter ourselves out of remote_room_hosts: do_invite_join ignores it
-        # and if it is the only entry we'd like to return a 404 rather than a
-        # 500.
-
-        remote_room_hosts = [
-            host for host in remote_room_hosts if host != self.hs.hostname
-        ]
-
-        if len(remote_room_hosts) == 0:
-            raise SynapseError(404, "No known servers")
-
-        # We don't do an auth check if we are doing an invite
-        # join dance for now, since we're kinda implicitly checking
-        # that we are allowed to join when we decide whether or not we
-        # need to do the invite/join dance.
-        yield self.federation_handler.do_invite_join(
-            remote_room_hosts,
-            room_id,
-            user.to_string(),
-            content,
-        )
-        yield self._user_joined_room(user, room_id)
-
-    @defer.inlineCallbacks
-    def _remote_reject_invite(self, requester, remote_room_hosts, room_id, target):
-        """Implements RoomMemberHandler._remote_reject_invite
-        """
-        fed_handler = self.federation_handler
-        try:
-            ret = yield fed_handler.do_remotely_reject_invite(
-                remote_room_hosts,
-                room_id,
-                target.to_string(),
-            )
-            defer.returnValue(ret)
-        except Exception as e:
-            # if we were unable to reject the exception, just mark
-            # it as rejected on our end and plough ahead.
-            #
-            # The 'except' clause is very broad, but we need to
-            # capture everything from DNS failures upwards
-            #
-            logger.warn("Failed to reject invite: %s", e)
-
-            yield self.store.locally_reject_invite(
-                target.to_string(), room_id
-            )
-            defer.returnValue({})
-
-    def get_or_register_3pid_guest(self, requester, medium, address, inviter_user_id):
-        """Implements RoomMemberHandler.get_or_register_3pid_guest
-        """
-        rg = self.registration_handler
-        return rg.get_or_register_3pid_guest(medium, address, inviter_user_id)
-
-    def _user_joined_room(self, target, room_id):
-        """Implements RoomMemberHandler._user_joined_room
-        """
-        return user_joined_room(self.distributor, target, room_id)
-
-    def _user_left_room(self, target, room_id):
-        """Implements RoomMemberHandler._user_left_room
-        """
-        return user_left_room(self.distributor, target, room_id)
-
-    @defer.inlineCallbacks
-    def forget(self, user, room_id):
-        user_id = user.to_string()
-
-        member = yield self.state_handler.get_current_state(
-            room_id=room_id,
-            event_type=EventTypes.Member,
-            state_key=user_id
-        )
-        membership = member.membership if member else None
-
-        if membership is not None and membership not in [
-            Membership.LEAVE, Membership.BAN
-        ]:
-            raise SynapseError(400, "User %s in room %s" % (
-                user_id, room_id
-            ))
-
-        if membership:
-            yield self.store.forget(user_id, room_id)
@@ -1,102 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2018 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import logging
-
-from twisted.internet import defer
-
-from synapse.api.errors import SynapseError
-from synapse.handlers.room_member import RoomMemberHandler
-from synapse.replication.http.membership import (
-    remote_join, remote_reject_invite, get_or_register_3pid_guest,
-    notify_user_membership_change,
-)
-
-
-logger = logging.getLogger(__name__)
-
-
-class RoomMemberWorkerHandler(RoomMemberHandler):
-    @defer.inlineCallbacks
-    def _remote_join(self, requester, remote_room_hosts, room_id, user, content):
-        """Implements RoomMemberHandler._remote_join
-        """
-        if len(remote_room_hosts) == 0:
-            raise SynapseError(404, "No known servers")
-
-        ret = yield remote_join(
-            self.simple_http_client,
-            host=self.config.worker_replication_host,
-            port=self.config.worker_replication_http_port,
-            requester=requester,
-            remote_room_hosts=remote_room_hosts,
-            room_id=room_id,
-            user_id=user.to_string(),
-            content=content,
-        )
-
-        yield self._user_joined_room(user, room_id)
-
-        defer.returnValue(ret)
-
-    def _remote_reject_invite(self, requester, remote_room_hosts, room_id, target):
-        """Implements RoomMemberHandler._remote_reject_invite
-        """
-        return remote_reject_invite(
-            self.simple_http_client,
-            host=self.config.worker_replication_host,
-            port=self.config.worker_replication_http_port,
-            requester=requester,
-            remote_room_hosts=remote_room_hosts,
-            room_id=room_id,
-            user_id=target.to_string(),
-        )
-
-    def _user_joined_room(self, target, room_id):
-        """Implements RoomMemberHandler._user_joined_room
-        """
-        return notify_user_membership_change(
-            self.simple_http_client,
-            host=self.config.worker_replication_host,
-            port=self.config.worker_replication_http_port,
-            user_id=target.to_string(),
-            room_id=room_id,
-            change="joined",
-        )
-
-    def _user_left_room(self, target, room_id):
-        """Implements RoomMemberHandler._user_left_room
-        """
-        return notify_user_membership_change(
-            self.simple_http_client,
-            host=self.config.worker_replication_host,
-            port=self.config.worker_replication_http_port,
-            user_id=target.to_string(),
-            room_id=room_id,
-            change="left",
-        )
-
-    def get_or_register_3pid_guest(self, requester, medium, address, inviter_user_id):
-        """Implements RoomMemberHandler.get_or_register_3pid_guest
-        """
-        return get_or_register_3pid_guest(
-            self.simple_http_client,
-            host=self.config.worker_replication_host,
-            port=self.config.worker_replication_http_port,
-            requester=requester,
-            medium=medium,
-            address=address,
-            inviter_user_id=inviter_user_id,
-        )
@@ -15,7 +15,7 @@

 from synapse.api.constants import Membership, EventTypes
 from synapse.util.async import concurrently_execute
-from synapse.util.logcontext import LoggingContext
+from synapse.util.logcontext import LoggingContext, make_deferred_yieldable, preserve_fn
 from synapse.util.metrics import Measure, measure_func
 from synapse.util.caches.response_cache import ResponseCache
 from synapse.push.clientformat import format_push_rules_for_user
@@ -52,7 +52,6 @@ class TimelineBatch(collections.namedtuple("TimelineBatch", [
        to tell if room needs to be part of the sync result.
        """
        return bool(self.events)
-    __bool__ = __nonzero__  # python3


 class JoinedSyncResult(collections.namedtuple("JoinedSyncResult", [
@@ -77,7 +76,6 @@ class JoinedSyncResult(collections.namedtuple("JoinedSyncResult", [
            # nb the notification count does not, er, count: if there's nothing
            # else in the result, we don't need to send it.
        )
-    __bool__ = __nonzero__  # python3


 class ArchivedSyncResult(collections.namedtuple("ArchivedSyncResult", [
@@ -97,7 +95,6 @@ class ArchivedSyncResult(collections.namedtuple("ArchivedSyncResult", [
            or self.state
            or self.account_data
        )
-    __bool__ = __nonzero__  # python3


 class InvitedSyncResult(collections.namedtuple("InvitedSyncResult", [
@@ -109,7 +106,6 @@ class InvitedSyncResult(collections.namedtuple("InvitedSyncResult", [
    def __nonzero__(self):
        """Invited rooms should always be reported to the client"""
        return True
-    __bool__ = __nonzero__  # python3


 class GroupsSyncResult(collections.namedtuple("GroupsSyncResult", [
@@ -121,7 +117,6 @@ class GroupsSyncResult(collections.namedtuple("GroupsSyncResult", [

    def __nonzero__(self):
        return bool(self.join or self.invite or self.leave)
-    __bool__ = __nonzero__  # python3


 class DeviceLists(collections.namedtuple("DeviceLists", [
@@ -132,7 +127,6 @@ class DeviceLists(collections.namedtuple("DeviceLists", [

    def __nonzero__(self):
        return bool(self.changed or self.left)
-    __bool__ = __nonzero__  # python3


 class SyncResult(collections.namedtuple("SyncResult", [
@@ -165,7 +159,6 @@ class SyncResult(collections.namedtuple("SyncResult", [
            self.device_lists or
            self.groups
        )
-    __bool__ = __nonzero__  # python3


 class SyncHandler(object):
@@ -176,7 +169,7 @@ class SyncHandler(object):
        self.presence_handler = hs.get_presence_handler()
        self.event_sources = hs.get_event_sources()
        self.clock = hs.get_clock()
-        self.response_cache = ResponseCache(hs, "sync")
+        self.response_cache = ResponseCache(hs)
        self.state = hs.get_state_handler()

    def wait_for_sync_for_user(self, sync_config, since_token=None, timeout=0,
@@ -187,11 +180,15 @@ class SyncHandler(object):
        Returns:
            A Deferred SyncResult.
        """
-        return self.response_cache.wrap(
-            sync_config.request_key,
-            self._wait_for_sync_for_user,
-            sync_config, since_token, timeout, full_state,
-        )
+        result = self.response_cache.get(sync_config.request_key)
+        if not result:
+            result = self.response_cache.set(
+                sync_config.request_key,
+                preserve_fn(self._wait_for_sync_for_user)(
+                    sync_config, since_token, timeout, full_state
+                )
+            )
+        return make_deferred_yieldable(result)

    @defer.inlineCallbacks
    def _wait_for_sync_for_user(self, sync_config, since_token, timeout,
@@ -238,10 +235,10 @@ class SyncHandler(object):
        defer.returnValue(rules)

    @defer.inlineCallbacks
-    def ephemeral_by_room(self, sync_result_builder, now_token, since_token=None):
+    def ephemeral_by_room(self, sync_config, now_token, since_token=None):
        """Get the ephemeral events for each room the user is in
        Args:
-            sync_result_builder(SyncResultBuilder)
+            sync_config (SyncConfig): The flags, filters and user for the sync.
            now_token (StreamToken): Where the server is currently up to.
            since_token (StreamToken): Where the server was when the client
                last synced.
@@ -251,12 +248,10 @@ class SyncHandler(object):
            typing events for that room.
        """

-        sync_config = sync_result_builder.sync_config
-
        with Measure(self.clock, "ephemeral_by_room"):
            typing_key = since_token.typing_key if since_token else "0"

-            room_ids = sync_result_builder.joined_room_ids
+            room_ids = yield self.store.get_rooms_for_user(sync_config.user.to_string())

            typing_source = self.event_sources.sources["typing"]
            typing, typing_key = yield typing_source.get_new_events(
@@ -354,24 +349,12 @@ class SyncHandler(object):
                since_key = since_token.room_key

            while limited and len(recents) < timeline_limit and max_repeat:
-                # If we have a since_key then we are trying to get any events
-                # that have happened since `since_key` up to `end_key`, so we
-                # can just use `get_room_events_stream_for_room`.
-                # Otherwise, we want to return the last N events in the room
-                # in toplogical ordering.
-                if since_key:
-                    events, end_key = yield self.store.get_room_events_stream_for_room(
-                        room_id,
-                        limit=load_limit + 1,
-                        from_key=since_key,
-                        to_key=end_key,
-                    )
-                else:
-                    events, end_key = yield self.store.get_recent_events_for_room(
-                        room_id,
-                        limit=load_limit + 1,
-                        end_token=end_key,
-                    )
+                events, end_key = yield self.store.get_room_events_stream_for_room(
+                    room_id,
+                    limit=load_limit + 1,
+                    from_key=since_key,
+                    to_key=end_key,
+                )
                loaded_recents = sync_config.filter_collection.filter_room_timeline(
                    events
                )
@@ -441,7 +424,7 @@ class SyncHandler(object):
        Returns:
            A Deferred map from ((type, state_key)->Event)
        """
-        last_events, _ = yield self.store.get_recent_events_for_room(
+        last_events, token = yield self.store.get_recent_events_for_room(
            room_id, end_token=stream_position.room_key, limit=1,
        )

@@ -582,22 +565,10 @@ class SyncHandler(object):
        # Always use the `now_token` in `SyncResultBuilder`
        now_token = yield self.event_sources.get_current_token()

-        user_id = sync_config.user.to_string()
-        app_service = self.store.get_app_service_by_user_id(user_id)
-        if app_service:
-            # We no longer support AS users using /sync directly.
-            # See https://github.com/matrix-org/matrix-doc/issues/1144
-            raise NotImplementedError()
-        else:
-            joined_room_ids = yield self.get_rooms_for_user_at(
-                user_id, now_token.room_stream_id,
-            )
-
        sync_result_builder = SyncResultBuilder(
            sync_config, full_state,
            since_token=since_token,
            now_token=now_token,
-            joined_room_ids=joined_room_ids,
        )

        account_data_by_room = yield self._generate_sync_entry_for_account_data(
@@ -614,7 +585,7 @@ class SyncHandler(object):
            since_token is None and
            sync_config.filter_collection.blocks_all_presence()
        )
-        if not block_all_presence_data:
+        if False and not block_all_presence_data:
            yield self._generate_sync_entry_for_presence(
                sync_result_builder, newly_joined_rooms, newly_joined_users
            )
@@ -632,6 +603,7 @@ class SyncHandler(object):
        device_id = sync_config.device_id
        one_time_key_counts = {}
        if device_id:
+            user_id = sync_config.user.to_string()
            one_time_key_counts = yield self.store.count_e2e_one_time_keys(
                user_id, device_id
            )
@@ -919,7 +891,7 @@ class SyncHandler(object):
            ephemeral_by_room = {}
        else:
            now_token, ephemeral_by_room = yield self.ephemeral_by_room(
-                sync_result_builder,
+                sync_result_builder.sync_config,
                now_token=sync_result_builder.now_token,
                since_token=sync_result_builder.since_token,
            )
@@ -1024,8 +996,15 @@ class SyncHandler(object):
        if rooms_changed:
            defer.returnValue(True)

+        app_service = self.store.get_app_service_by_user_id(user_id)
+        if app_service:
+            rooms = yield self.store.get_app_service_rooms(app_service)
+            joined_room_ids = set(r.room_id for r in rooms)
+        else:
+            joined_room_ids = yield self.store.get_rooms_for_user(user_id)
+
        stream_id = RoomStreamToken.parse_stream_token(since_token.room_key).stream
-        for room_id in sync_result_builder.joined_room_ids:
+        for room_id in joined_room_ids:
            if self.store.has_room_changed_since(room_id, stream_id):
                defer.returnValue(True)
        defer.returnValue(False)
@@ -1049,6 +1028,13 @@ class SyncHandler(object):

        assert since_token

+        app_service = self.store.get_app_service_by_user_id(user_id)
+        if app_service:
+            rooms = yield self.store.get_app_service_rooms(app_service)
+            joined_room_ids = set(r.room_id for r in rooms)
+        else:
+            joined_room_ids = yield self.store.get_rooms_for_user(user_id)
+
        # Get a list of membership change events that have happened.
        rooms_changed = yield self.store.get_membership_changes_for_user(
            user_id, since_token.room_key, now_token.room_key
@@ -1071,7 +1057,7 @@ class SyncHandler(object):
            # we do send down the room, and with full state, where necessary

            old_state_ids = None
-            if room_id in sync_result_builder.joined_room_ids and non_joins:
+            if room_id in joined_room_ids and non_joins:
                # Always include if the user (re)joined the room, especially
                # important so that device list changes are calculated correctly.
                # If there are non join member events, but we are still in the room,
@@ -1081,7 +1067,7 @@ class SyncHandler(object):
                # User is in the room so we don't need to do the invite/leave checks
                continue

-            if room_id in sync_result_builder.joined_room_ids or has_join:
+            if room_id in joined_room_ids or has_join:
                old_state_ids = yield self.get_state_at(room_id, since_token)
                old_mem_ev_id = old_state_ids.get((EventTypes.Member, user_id), None)
                old_mem_ev = None
@@ -1093,7 +1079,7 @@ class SyncHandler(object):
                    newly_joined_rooms.append(room_id)

            # If user is in the room then we don't need to do the invite/leave checks
-            if room_id in sync_result_builder.joined_room_ids:
+            if room_id in joined_room_ids:
                continue

            if not non_joins:
@@ -1160,7 +1146,7 @@ class SyncHandler(object):

        # Get all events for rooms we're currently joined to.
        room_to_events = yield self.store.get_room_events_stream_for_rooms(
-            room_ids=sync_result_builder.joined_room_ids,
+            room_ids=joined_room_ids,
            from_key=since_token.room_key,
            to_key=now_token.room_key,
            limit=timeline_limit + 1,
@@ -1168,7 +1154,7 @@ class SyncHandler(object):

        # We loop through all room ids, even if there are no new events, in case
        # there are non room events taht we need to notify about.
-        for room_id in sync_result_builder.joined_room_ids:
+        for room_id in joined_room_ids:
            room_entry = room_to_events.get(room_id, None)

            if room_entry:
@@ -1376,54 +1362,6 @@ class SyncHandler(object):
        else:
            raise Exception("Unrecognized rtype: %r", room_builder.rtype)

-    @defer.inlineCallbacks
-    def get_rooms_for_user_at(self, user_id, stream_ordering):
-        """Get set of joined rooms for a user at the given stream ordering.
-
-        The stream ordering *must* be recent, otherwise this may throw an
-        exception if older than a month. (This function is called with the
-        current token, which should be perfectly fine).
-
-        Args:
-            user_id (str)
-            stream_ordering (int)
-
-        ReturnValue:
-            Deferred[frozenset[str]]: Set of room_ids the user is in at given
-            stream_ordering.
-        """
-        joined_rooms = yield self.store.get_rooms_for_user_with_stream_ordering(
-            user_id,
-        )
-
-        joined_room_ids = set()
-
-        # We need to check that the stream ordering of the join for each room
-        # is before the stream_ordering asked for. This might not be the case
-        # if the user joins a room between us getting the current token and
-        # calling `get_rooms_for_user_with_stream_ordering`.
-        # If the membership's stream ordering is after the given stream
-        # ordering, we need to go and work out if the user was in the room
-        # before.
-        for room_id, membership_stream_ordering in joined_rooms:
-            if membership_stream_ordering <= stream_ordering:
-                joined_room_ids.add(room_id)
-                continue
-
-            logger.info("User joined room after current token: %s", room_id)
-
-            extrems = yield self.store.get_forward_extremeties_for_room(
-                room_id, stream_ordering,
-            )
-            users_in_room = yield self.state.get_current_user_in_room(
-                room_id, extrems,
-            )
-            if user_id in users_in_room:
-                joined_room_ids.add(room_id)
-
-        joined_room_ids = frozenset(joined_room_ids)
-        defer.returnValue(joined_room_ids)
-

 def _action_has_highlight(actions):
    for action in actions:
@@ -1473,8 +1411,7 @@ def _calculate_state(timeline_contains, timeline_start, previous, current):

 class SyncResultBuilder(object):
    "Used to help build up a new SyncResult for a user"
-    def __init__(self, sync_config, full_state, since_token, now_token,
-                 joined_room_ids):
+    def __init__(self, sync_config, full_state, since_token, now_token):
        """
        Args:
            sync_config(SyncConfig)
@@ -1486,7 +1423,6 @@ class SyncResultBuilder(object):
        self.full_state = full_state
        self.since_token = since_token
        self.now_token = now_token
-        self.joined_room_ids = joined_room_ids

        self.presence = []
        self.account_data = []
@@ -16,7 +16,7 @@
 from twisted.internet import defer

 from synapse.api.errors import SynapseError, AuthError
-from synapse.util.logcontext import run_in_background
+from synapse.util.logcontext import preserve_fn
 from synapse.util.metrics import Measure
 from synapse.util.wheel_timer import WheelTimer
 from synapse.types import UserID, get_domain_from_id
@@ -56,7 +56,7 @@ class TypingHandler(object):

        self.federation = hs.get_federation_sender()

-        hs.get_federation_registry().register_edu_handler("m.typing", self._recv_edu)
+        hs.get_replication_layer().register_edu_handler("m.typing", self._recv_edu)

        hs.get_distributor().observe("user_left_room", self.user_left_room)

@@ -97,8 +97,7 @@ class TypingHandler(object):
            if self.hs.is_mine_id(member.user_id):
                last_fed_poke = self._member_last_federation_poke.get(member, None)
                if not last_fed_poke or last_fed_poke + FEDERATION_PING_INTERVAL <= now:
-                    run_in_background(
-                        self._push_remote,
+                    preserve_fn(self._push_remote)(
                        member=member,
                        typing=True
                    )
@@ -197,7 +196,7 @@ class TypingHandler(object):
    def _push_update(self, member, typing):
        if self.hs.is_mine_id(member.user_id):
            # Only send updates for changes to our own users.
-            run_in_background(self._push_remote, member, typing)
+            preserve_fn(self._push_remote)(member, typing)

        self._push_update_local(
            member=member,
@@ -206,31 +205,28 @@ class TypingHandler(object):

    @defer.inlineCallbacks
    def _push_remote(self, member, typing):
-        try:
-            users = yield self.state.get_current_user_in_room(member.room_id)
-            self._member_last_federation_poke[member] = self.clock.time_msec()
+        users = yield self.state.get_current_user_in_room(member.room_id)
+        self._member_last_federation_poke[member] = self.clock.time_msec()

-            now = self.clock.time_msec()
-            self.wheel_timer.insert(
-                now=now,
-                obj=member,
-                then=now + FEDERATION_PING_INTERVAL,
-            )
+        now = self.clock.time_msec()
+        self.wheel_timer.insert(
+            now=now,
+            obj=member,
+            then=now + FEDERATION_PING_INTERVAL,
+        )

-            for domain in set(get_domain_from_id(u) for u in users):
-                if domain != self.server_name:
-                    self.federation.send_edu(
-                        destination=domain,
-                        edu_type="m.typing",
-                        content={
-                            "room_id": member.room_id,
-                            "user_id": member.user_id,
-                            "typing": typing,
-                        },
-                        key=member,
-                    )
-        except Exception:
-            logger.exception("Error pushing typing notif to remotes")
+        for domain in set(get_domain_from_id(u) for u in users):
+            if domain != self.server_name:
+                self.federation.send_edu(
+                    destination=domain,
+                    edu_type="m.typing",
+                    content={
+                        "room_id": member.room_id,
+                        "user_id": member.user_id,
+                        "typing": typing,
+                    },
+                    key=member,
+                )

    @defer.inlineCallbacks
    def _recv_edu(self, origin, content):
@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -13,24 +12,3 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-from twisted.internet.defer import CancelledError
-from twisted.python import failure
-
-from synapse.api.errors import SynapseError
-
-
-class RequestTimedOutError(SynapseError):
-    """Exception representing timeout of an outbound request"""
-    def __init__(self):
-        super(RequestTimedOutError, self).__init__(504, "Timed out")
-
-
-def cancelled_to_request_timed_out_error(value, timeout):
-    """Turns CancelledErrors into RequestTimedOutErrors.
-
-    For use with async.add_timeout_to_deferred
-    """
-    if isinstance(value, failure.Failure):
-        value.trap(CancelledError)
-        raise RequestTimedOutError()
-    return value
@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-from synapse.http.server import wrap_json_request_handler
+from synapse.http.server import wrap_request_handler
 from twisted.web.resource import Resource
 from twisted.web.server import NOT_DONE_YET

@@ -42,13 +42,14 @@ class AdditionalResource(Resource):
        Resource.__init__(self)
        self._handler = handler

-        # required by the request_handler wrapper
+        # these are required by the request_handler wrapper
+        self.version_string = hs.version_string
        self.clock = hs.get_clock()

    def render(self, request):
        self._async_render(request)
        return NOT_DONE_YET

-    @wrap_json_request_handler
+    @wrap_request_handler
    def _async_render(self, request):
        return self._handler(request)
@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -19,10 +18,9 @@ from OpenSSL.SSL import VERIFY_NONE
 from synapse.api.errors import (
    CodeMessageException, MatrixCodeMessageException, SynapseError, Codes,
 )
-from synapse.http import cancelled_to_request_timed_out_error
-from synapse.util.async import add_timeout_to_deferred
 from synapse.util.caches import CACHE_SIZE_FACTOR
 from synapse.util.logcontext import make_deferred_yieldable
+from synapse.util import logcontext
 import synapse.metrics
 from synapse.http.endpoint import SpiderEndpoint

@@ -40,7 +38,7 @@ from twisted.web.http import PotentialDataLoss
 from twisted.web.http_headers import Headers
 from twisted.web._newclient import ResponseDone

-from six import StringIO
+from StringIO import StringIO

 import simplejson as json
 import logging
@@ -97,17 +95,21 @@ class SimpleHttpClient(object):
        # counters to it
        outgoing_requests_counter.inc(method)

-        logger.info("Sending request %s %s", method, uri)
-
-        try:
+        def send_request():
            request_deferred = self.agent.request(
                method, uri, *args, **kwargs
            )
-            add_timeout_to_deferred(
+
+            return self.clock.time_bound_deferred(
                request_deferred,
-                60, cancelled_to_request_timed_out_error,
+                time_out=60,
            )
-            response = yield make_deferred_yieldable(request_deferred)
+
+        logger.info("Sending request %s %s", method, uri)
+
+        try:
+            with logcontext.PreserveLoggingContext():
+                response = yield send_request()

            incoming_responses_counter.inc(method, response.code)
            logger.info(
@@ -507,7 +509,7 @@ class SpiderHttpClient(SimpleHttpClient):
                    reactor,
                    SpiderEndpointFactory(hs)
                )
-            ), [(b'gzip', GzipDecoder)]
+            ), [('gzip', GzipDecoder)]
        )
        # We could look like Chrome:
        # self.user_agent = ("Mozilla/5.0 (%s) (KHTML, like Gecko)
@@ -12,6 +12,8 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import socket
+
 from twisted.internet.endpoints import HostnameEndpoint, wrapClientTLS
 from twisted.internet import defer, reactor
 from twisted.internet.error import ConnectError
@@ -31,7 +33,7 @@ SERVER_CACHE = {}

 # our record of an individual server which can be tried to reach a destination.
 #
-# "host" is the hostname acquired from the SRV record. Except when there's
+# "host" is actually a dotted-quad or ipv6 address string. Except when there's
 # no SRV record, in which case it is the original hostname.
 _Server = collections.namedtuple(
    "_Server", "priority weight host port expires"
@@ -115,15 +117,10 @@ class _WrappedConnection(object):
        if time.time() - self.last_request >= 2.5 * 60:
            self.abort()
            # Abort the underlying TLS connection. The abort() method calls
-            # loseConnection() on the TLS connection which tries to
+            # loseConnection() on the underlying TLS connection which tries to
            # shutdown the connection cleanly. We call abortConnection()
-            # since that will promptly close the TLS connection.
-            #
-            # In Twisted >18.4; the TLS connection will be None if it has closed
-            # which will make abortConnection() throw. Check that the TLS connection
-            # is not None before trying to close it.
-            if self.transport.getHandle() is not None:
-                self.transport.abortConnection()
+            # since that will promptly close the underlying TCP connection.
+            self.transport.abortConnection()

    def request(self, request):
        self.last_request = time.time()
@@ -291,7 +288,7 @@ def resolve_service(service_name, dns_client=client, cache=SERVER_CACHE, clock=t
        if (len(answers) == 1
                and answers[0].type == dns.SRV
                and answers[0].payload
-                and answers[0].payload.target == dns.Name(b'.')):
+                and answers[0].payload.target == dns.Name('.')):
            raise ConnectError("Service %s unavailable" % service_name)

        for answer in answers:
@@ -300,13 +297,20 @@ def resolve_service(service_name, dns_client=client, cache=SERVER_CACHE, clock=t

            payload = answer.payload

-            servers.append(_Server(
-                host=str(payload.target),
-                port=int(payload.port),
-                priority=int(payload.priority),
-                weight=int(payload.weight),
-                expires=int(clock.time()) + answer.ttl,
-            ))
+            hosts = yield _get_hosts_for_srv_record(
+                dns_client, str(payload.target)
+            )
+
+            for (ip, ttl) in hosts:
+                host_ttl = min(answer.ttl, ttl)
+
+                servers.append(_Server(
+                    host=ip,
+                    port=int(payload.port),
+                    priority=int(payload.priority),
+                    weight=int(payload.weight),
+                    expires=int(clock.time()) + host_ttl,
+                ))

        servers.sort()
        cache[service_name] = list(servers)
@@ -324,3 +328,81 @@ def resolve_service(service_name, dns_client=client, cache=SERVER_CACHE, clock=t
            raise e

    defer.returnValue(servers)
+
+
+@defer.inlineCallbacks
+def _get_hosts_for_srv_record(dns_client, host):
+    """Look up each of the hosts in a SRV record
+
+    Args:
+        dns_client (twisted.names.dns.IResolver):
+        host (basestring): host to look up
+
+    Returns:
+        Deferred[list[(str, int)]]: a list of (host, ttl) pairs
+
+    """
+    ip4_servers = []
+    ip6_servers = []
+
+    def cb(res):
+        # lookupAddress and lookupIP6Address return a three-tuple
+        # giving the answer, authority, and additional sections of the
+        # response.
+        #
+        # we only care about the answers.
+
+        return res[0]
+
+    def eb(res, record_type):
+        if res.check(DNSNameError):
+            return []
+        logger.warn("Error looking up %s for %s: %s", record_type, host, res)
+        return res
+
+    # no logcontexts here, so we can safely fire these off and gatherResults
+    d1 = dns_client.lookupAddress(host).addCallbacks(
+        cb, eb, errbackArgs=("A", ))
+    d2 = dns_client.lookupIPV6Address(host).addCallbacks(
+        cb, eb, errbackArgs=("AAAA", ))
+    results = yield defer.DeferredList(
+        [d1, d2], consumeErrors=True)
+
+    # if all of the lookups failed, raise an exception rather than blowing out
+    # the cache with an empty result.
+    if results and all(s == defer.FAILURE for (s, _) in results):
+        defer.returnValue(results[0][1])
+
+    for (success, result) in results:
+        if success == defer.FAILURE:
+            continue
+
+        for answer in result:
+            if not answer.payload:
+                continue
+
+            try:
+                if answer.type == dns.A:
+                    ip = answer.payload.dottedQuad()
+                    ip4_servers.append((ip, answer.ttl))
+                elif answer.type == dns.AAAA:
+                    ip = socket.inet_ntop(
+                        socket.AF_INET6, answer.payload.address,
+                    )
+                    ip6_servers.append((ip, answer.ttl))
+                else:
+                    # the most likely candidate here is a CNAME record.
+                    # rfc2782 says srvs may not point to aliases.
+                    logger.warn(
+                        "Ignoring unexpected DNS record type %s for %s",
+                        answer.type, host,
+                    )
+                    continue
+            except Exception as e:
+                logger.warn("Ignoring invalid DNS response for %s: %s",
+                            host, e)
+                continue
+
+    # keep the ipv4 results before the ipv6 results, mostly to match historical
+    # behaviour.
+    defer.returnValue(ip4_servers + ip6_servers)
@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -13,19 +12,17 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import synapse.util.retryutils
 from twisted.internet import defer, reactor, protocol
 from twisted.internet.error import DNSLookupError
 from twisted.web.client import readBody, HTTPConnectionPool, Agent
 from twisted.web.http_headers import Headers
 from twisted.web._newclient import ResponseDone

-from synapse.http import cancelled_to_request_timed_out_error
 from synapse.http.endpoint import matrix_federation_endpoint
-import synapse.metrics
-from synapse.util.async import sleep, add_timeout_to_deferred
+from synapse.util.async import sleep
 from synapse.util import logcontext
-from synapse.util.logcontext import make_deferred_yieldable
-import synapse.util.retryutils
+import synapse.metrics

 from canonicaljson import encode_canonical_json

@@ -41,7 +38,8 @@ import logging
 import random
 import sys
 import urllib
-from six.moves.urllib import parse as urlparse
+import urlparse
+

 logger = logging.getLogger(__name__)
 outbound_logger = logging.getLogger("synapse.http.outbound")
@@ -186,20 +184,21 @@ class MatrixFederationHttpClient(object):
                        producer = body_callback(method, http_url_bytes, headers_dict)

                    try:
-                        request_deferred = self.agent.request(
-                            method,
-                            url_bytes,
-                            Headers(headers_dict),
-                            producer
-                        )
-                        add_timeout_to_deferred(
-                            request_deferred,
-                            timeout / 1000. if timeout else 60,
-                            cancelled_to_request_timed_out_error,
-                        )
-                        response = yield make_deferred_yieldable(
-                            request_deferred,
-                        )
+                        def send_request():
+                            request_deferred = self.agent.request(
+                                method,
+                                url_bytes,
+                                Headers(headers_dict),
+                                producer
+                            )
+
+                            return self.clock.time_bound_deferred(
+                                request_deferred,
+                                time_out=timeout / 1000. if timeout else 60,
+                            )
+
+                        with logcontext.PreserveLoggingContext():
+                            response = yield send_request()

                        log_result = "%d %s" % (response.code, response.phrase,)
                        break
@@ -287,8 +286,7 @@ class MatrixFederationHttpClient(object):
        headers_dict[b"Authorization"] = auth_headers

    @defer.inlineCallbacks
-    def put_json(self, destination, path, args={}, data={},
-                 json_data_callback=None,
+    def put_json(self, destination, path, data={}, json_data_callback=None,
                 long_retries=False, timeout=None,
                 ignore_backoff=False,
                 backoff_on_404=False):
@@ -298,7 +296,6 @@ class MatrixFederationHttpClient(object):
            destination (str): The remote server to send the HTTP request
                to.
            path (str): The HTTP path.
-            args (dict): query params
            data (dict): A dict containing the data that will be used as
                the request body. This will be encoded as JSON.
            json_data_callback (callable): A callable returning the dict to
@@ -345,7 +342,6 @@ class MatrixFederationHttpClient(object):
            path,
            body_callback=body_callback,
            headers_dict={"Content-Type": ["application/json"]},
-            query_bytes=encode_query_args(args),
            long_retries=long_retries,
            timeout=timeout,
            ignore_backoff=ignore_backoff,
@@ -377,7 +373,6 @@ class MatrixFederationHttpClient(object):
                giving up. None indicates no timeout.
            ignore_backoff (bool): true to ignore the historical backoff data and
                try the request anyway.
-            args (dict): query params
        Returns:
            Deferred: Succeeds when we get a 2xx HTTP response. The result
            will be the decoded JSON body.
@@ -1,149 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2018 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import logging
-
-import synapse.metrics
-from synapse.util.logcontext import LoggingContext
-
-logger = logging.getLogger(__name__)
-
-metrics = synapse.metrics.get_metrics_for("synapse.http.server")
-
-# total number of responses served, split by method/servlet/tag
-response_count = metrics.register_counter(
-    "response_count",
-    labels=["method", "servlet", "tag"],
-    alternative_names=(
-        # the following are all deprecated aliases for the same metric
-        metrics.name_prefix + x for x in (
-            "_requests",
-            "_response_time:count",
-            "_response_ru_utime:count",
-            "_response_ru_stime:count",
-            "_response_db_txn_count:count",
-            "_response_db_txn_duration:count",
-        )
-    )
-)
-
-requests_counter = metrics.register_counter(
-    "requests_received",
-    labels=["method", "servlet", ],
-)
-
-outgoing_responses_counter = metrics.register_counter(
-    "responses",
-    labels=["method", "code"],
-)
-
-response_timer = metrics.register_counter(
-    "response_time_seconds",
-    labels=["method", "servlet", "tag"],
-    alternative_names=(
-        metrics.name_prefix + "_response_time:total",
-    ),
-)
-
-response_ru_utime = metrics.register_counter(
-    "response_ru_utime_seconds", labels=["method", "servlet", "tag"],
-    alternative_names=(
-        metrics.name_prefix + "_response_ru_utime:total",
-    ),
-)
-
-response_ru_stime = metrics.register_counter(
-    "response_ru_stime_seconds", labels=["method", "servlet", "tag"],
-    alternative_names=(
-        metrics.name_prefix + "_response_ru_stime:total",
-    ),
-)
-
-response_db_txn_count = metrics.register_counter(
-    "response_db_txn_count", labels=["method", "servlet", "tag"],
-    alternative_names=(
-        metrics.name_prefix + "_response_db_txn_count:total",
-    ),
-)
-
-# seconds spent waiting for db txns, excluding scheduling time, when processing
-# this request
-response_db_txn_duration = metrics.register_counter(
-    "response_db_txn_duration_seconds", labels=["method", "servlet", "tag"],
-    alternative_names=(
-        metrics.name_prefix + "_response_db_txn_duration:total",
-    ),
-)
-
-# seconds spent waiting for a db connection, when processing this request
-response_db_sched_duration = metrics.register_counter(
-    "response_db_sched_duration_seconds", labels=["method", "servlet", "tag"]
-)
-
-# size in bytes of the response written
-response_size = metrics.register_counter(
-    "response_size", labels=["method", "servlet", "tag"]
-)
-
-
-class RequestMetrics(object):
-    def start(self, time_msec, name):
-        self.start = time_msec
-        self.start_context = LoggingContext.current_context()
-        self.name = name
-
-    def stop(self, time_msec, request):
-        context = LoggingContext.current_context()
-
-        tag = ""
-        if context:
-            tag = context.tag
-
-            if context != self.start_context:
-                logger.warn(
-                    "Context have unexpectedly changed %r, %r",
-                    context, self.start_context
-                )
-                return
-
-        outgoing_responses_counter.inc(request.method, str(request.code))
-
-        response_count.inc(request.method, self.name, tag)
-
-        response_timer.inc_by(
-            time_msec - self.start, request.method,
-            self.name, tag
-        )
-
-        ru_utime, ru_stime = context.get_resource_usage()
-
-        response_ru_utime.inc_by(
-            ru_utime, request.method, self.name, tag
-        )
-        response_ru_stime.inc_by(
-            ru_stime, request.method, self.name, tag
-        )
-        response_db_txn_count.inc_by(
-            context.db_txn_count, request.method, self.name, tag
-        )
-        response_db_txn_duration.inc_by(
-            context.db_txn_duration_ms / 1000., request.method, self.name, tag
-        )
-        response_db_sched_duration.inc_by(
-            context.db_sched_duration_ms / 1000., request.method, self.name, tag
-        )
-
-        response_size.inc_by(request.sentLength, request.method, self.name, tag)
@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -18,9 +17,6 @@
 from synapse.api.errors import (
    cs_exception, SynapseError, CodeMessageException, UnrecognizedRequestError, Codes
 )
-from synapse.http.request_metrics import (
-    requests_counter,
-)
 from synapse.util.logcontext import LoggingContext, PreserveLoggingContext
 from synapse.util.caches import intern_dict
 from synapse.util.metrics import Measure
@@ -40,107 +36,170 @@ from twisted.web.util import redirectTo
 import collections
 import logging
 import urllib
-import simplejson
+import ujson

 logger = logging.getLogger(__name__)

+metrics = synapse.metrics.get_metrics_for(__name__)

-def wrap_json_request_handler(h):
-    """Wraps a request handler method with exception handling.
+# total number of responses served, split by method/servlet/tag
+response_count = metrics.register_counter(
+    "response_count",
+    labels=["method", "servlet", "tag"],
+    alternative_names=(
+        # the following are all deprecated aliases for the same metric
+        metrics.name_prefix + x for x in (
+            "_requests",
+            "_response_time:count",
+            "_response_ru_utime:count",
+            "_response_ru_stime:count",
+            "_response_db_txn_count:count",
+            "_response_db_txn_duration:count",
+        )
+    )
+)

-    Also adds logging as per wrap_request_handler_with_logging.
+outgoing_responses_counter = metrics.register_counter(
+    "responses",
+    labels=["method", "code"],
+)

-    The handler method must have a signature of "handle_foo(self, request)",
-    where "self" must have a "clock" attribute (and "request" must be a
-    SynapseRequest).
+response_timer = metrics.register_counter(
+    "response_time_seconds",
+    labels=["method", "servlet", "tag"],
+    alternative_names=(
+        metrics.name_prefix + "_response_time:total",
+    ),
+)

-    The handler must return a deferred. If the deferred succeeds we assume that
+response_ru_utime = metrics.register_counter(
+    "response_ru_utime_seconds", labels=["method", "servlet", "tag"],
+    alternative_names=(
+        metrics.name_prefix + "_response_ru_utime:total",
+    ),
+)
+
+response_ru_stime = metrics.register_counter(
+    "response_ru_stime_seconds", labels=["method", "servlet", "tag"],
+    alternative_names=(
+        metrics.name_prefix + "_response_ru_stime:total",
+    ),
+)
+
+response_db_txn_count = metrics.register_counter(
+    "response_db_txn_count", labels=["method", "servlet", "tag"],
+    alternative_names=(
+        metrics.name_prefix + "_response_db_txn_count:total",
+    ),
+)
+
+# seconds spent waiting for db txns, excluding scheduling time, when processing
+# this request
+response_db_txn_duration = metrics.register_counter(
+    "response_db_txn_duration_seconds", labels=["method", "servlet", "tag"],
+    alternative_names=(
+        metrics.name_prefix + "_response_db_txn_duration:total",
+    ),
+)
+
+# seconds spent waiting for a db connection, when processing this request
+response_db_sched_duration = metrics.register_counter(
+    "response_db_sched_duration_seconds", labels=["method", "servlet", "tag"]
+)
+
+_next_request_id = 0
+
+
+def request_handler(include_metrics=False):
+    """Decorator for ``wrap_request_handler``"""
+    return lambda request_handler: wrap_request_handler(request_handler, include_metrics)
+
+
+def wrap_request_handler(request_handler, include_metrics=False):
+    """Wraps a method that acts as a request handler with the necessary logging
+    and exception handling.
+
+    The method must have a signature of "handle_foo(self, request)". The
+    argument "self" must have "version_string" and "clock" attributes. The
+    argument "request" must be a twisted HTTP request.
+
+    The method must return a deferred. If the deferred succeeds we assume that
    a response has been sent. If the deferred fails with a SynapseError we use
    it to send a JSON response with the appropriate HTTP reponse code. If the
    deferred fails with any other type of error we send a 500 reponse.
+
+    We insert a unique request-id into the logging context for this request and
+    log the response and duration for this request.
    """

    @defer.inlineCallbacks
    def wrapped_request_handler(self, request):
-        try:
-            yield h(self, request)
-        except CodeMessageException as e:
-            code = e.code
-            if isinstance(e, SynapseError):
-                logger.info(
-                    "%s SynapseError: %s - %s", request, code, e.msg
-                )
-            else:
-                logger.exception(e)
-            respond_with_json(
-                request, code, cs_exception(e), send_cors=True,
-                pretty_print=_request_user_agent_is_curl(request),
-            )
+        global _next_request_id
+        request_id = "%s-%s" % (request.method, _next_request_id)
+        _next_request_id += 1

-        except Exception:
-            # failure.Failure() fishes the original Failure out
-            # of our stack, and thus gives us a sensible stack
-            # trace.
-            f = failure.Failure()
-            logger.error(
-                "Failed handle request via %r: %r: %s",
-                h,
-                request,
-                f.getTraceback().rstrip(),
-            )
-            respond_with_json(
-                request,
-                500,
-                {
-                    "error": "Internal server error",
-                    "errcode": Codes.UNKNOWN,
-                },
-                send_cors=True,
-                pretty_print=_request_user_agent_is_curl(request),
-            )
-
-    return wrap_request_handler_with_logging(wrapped_request_handler)
-
-
-def wrap_request_handler_with_logging(h):
-    """Wraps a request handler to provide logging and metrics
-
-    The handler method must have a signature of "handle_foo(self, request)",
-    where "self" must have a "clock" attribute (and "request" must be a
-    SynapseRequest).
-
-    As well as calling `request.processing` (which will log the response and
-    duration for this request), the wrapped request handler will insert the
-    request id into the logging context.
-    """
-    @defer.inlineCallbacks
-    def wrapped_request_handler(self, request):
-        """
-        Args:
-            self:
-            request (synapse.http.site.SynapseRequest):
-        """
-
-        request_id = request.get_request_id()
        with LoggingContext(request_id) as request_context:
-            request_context.request = request_id
            with Measure(self.clock, "wrapped_request_handler"):
+                request_metrics = RequestMetrics()
                # we start the request metrics timer here with an initial stab
                # at the servlet name. For most requests that name will be
                # JsonResource (or a subclass), and JsonResource._async_render
                # will update it once it picks a servlet.
-                servlet_name = self.__class__.__name__
-                with request.processing(servlet_name):
-                    with PreserveLoggingContext(request_context):
-                        d = h(self, request)
+                request_metrics.start(self.clock, name=self.__class__.__name__)

-                        # record the arrival of the request *after*
-                        # dispatching to the handler, so that the handler
-                        # can update the servlet name in the request
-                        # metrics
-                        requests_counter.inc(request.method,
-                                             request.request_metrics.name)
-                        yield d
+                request_context.request = request_id
+                with request.processing():
+                    try:
+                        with PreserveLoggingContext(request_context):
+                            if include_metrics:
+                                yield request_handler(self, request, request_metrics)
+                            else:
+                                yield request_handler(self, request)
+                    except CodeMessageException as e:
+                        code = e.code
+                        if isinstance(e, SynapseError):
+                            logger.info(
+                                "%s SynapseError: %s - %s", request, code, e.msg
+                            )
+                        else:
+                            logger.exception(e)
+                        outgoing_responses_counter.inc(request.method, str(code))
+                        respond_with_json(
+                            request, code, cs_exception(e), send_cors=True,
+                            pretty_print=_request_user_agent_is_curl(request),
+                            version_string=self.version_string,
+                        )
+                    except Exception:
+                        # failure.Failure() fishes the original Failure out
+                        # of our stack, and thus gives us a sensible stack
+                        # trace.
+                        f = failure.Failure()
+                        logger.error(
+                            "Failed handle request %s.%s on %r: %r: %s",
+                            request_handler.__module__,
+                            request_handler.__name__,
+                            self,
+                            request,
+                            f.getTraceback().rstrip(),
+                        )
+                        respond_with_json(
+                            request,
+                            500,
+                            {
+                                "error": "Internal server error",
+                                "errcode": Codes.UNKNOWN,
+                            },
+                            send_cors=True,
+                            pretty_print=_request_user_agent_is_curl(request),
+                            version_string=self.version_string,
+                        )
+                    finally:
+                        try:
+                            request_metrics.stop(
+                                self.clock, request
+                            )
+                        except Exception as e:
+                            logger.warn("Failed to stop metrics: %r", e)
    return wrapped_request_handler


@@ -170,7 +229,7 @@ class JsonResource(HttpServer, resource.Resource):
    """ This implements the HttpServer interface and provides JSON support for
    Resources.

-    Register callbacks via register_paths()
+    Register callbacks via register_path()

    Callbacks can return a tuple of status code and a dict in which case the
    the dict will automatically be sent to the client as a JSON object.
@@ -190,6 +249,7 @@ class JsonResource(HttpServer, resource.Resource):
        self.canonical_json = canonical_json
        self.clock = hs.get_clock()
        self.path_regexs = {}
+        self.version_string = hs.version_string
        self.hs = hs

    def register_paths(self, method, path_patterns, callback):
@@ -205,103 +265,120 @@ class JsonResource(HttpServer, resource.Resource):
        self._async_render(request)
        return server.NOT_DONE_YET

-    @wrap_json_request_handler
+    # Disable metric reporting because _async_render does its own metrics.
+    # It does its own metric reporting because _async_render dispatches to
+    # a callback and it's the class name of that callback we want to report
+    # against rather than the JsonResource itself.
+    @request_handler(include_metrics=True)
    @defer.inlineCallbacks
-    def _async_render(self, request):
+    def _async_render(self, request, request_metrics):
        """ This gets called from render() every time someone sends us a request.
            This checks if anyone has registered a callback for that method and
            path.
        """
-        callback, group_dict = self._get_handler_for_request(request)
-
-        servlet_instance = getattr(callback, "__self__", None)
-        if servlet_instance is not None:
-            servlet_classname = servlet_instance.__class__.__name__
-        else:
-            servlet_classname = "%r" % callback
-        request.request_metrics.name = servlet_classname
-
-        # Now trigger the callback. If it returns a response, we send it
-        # here. If it throws an exception, that is handled by the wrapper
-        # installed by @request_handler.
-
-        kwargs = intern_dict({
-            name: urllib.unquote(value).decode("UTF-8") if value else value
-            for name, value in group_dict.items()
-        })
-
-        callback_return = yield callback(request, **kwargs)
-        if callback_return is not None:
-            code, response = callback_return
-            self._send_response(request, code, response)
-
-    def _get_handler_for_request(self, request):
-        """Finds a callback method to handle the given request
-
-        Args:
-            request (twisted.web.http.Request):
-
-        Returns:
-            Tuple[Callable, dict[str, str]]: callback method, and the dict
-                mapping keys to path components as specified in the handler's
-                path match regexp.
-
-                The callback will normally be a method registered via
-                register_paths, so will return (possibly via Deferred) either
-                None, or a tuple of (http code, response body).
-        """
-        if request.method == b"OPTIONS":
-            return _options_handler, {}
+        if request.method == "OPTIONS":
+            self._send_response(request, 200, {})
+            return

        # Loop through all the registered callbacks to check if the method
        # and path regex match
        for path_entry in self.path_regexs.get(request.method, []):
            m = path_entry.pattern.match(request.path)
-            if m:
-                # We found a match!
-                return path_entry.callback, m.groupdict()
+            if not m:
+                continue
+
+            # We found a match! First update the metrics object to indicate
+            # which servlet is handling the request.
+
+            callback = path_entry.callback
+
+            servlet_instance = getattr(callback, "__self__", None)
+            if servlet_instance is not None:
+                servlet_classname = servlet_instance.__class__.__name__
+            else:
+                servlet_classname = "%r" % callback
+
+            request_metrics.name = servlet_classname
+
+            # Now trigger the callback. If it returns a response, we send it
+            # here. If it throws an exception, that is handled by the wrapper
+            # installed by @request_handler.
+
+            kwargs = intern_dict({
+                name: urllib.unquote(value).decode("UTF-8") if value else value
+                for name, value in m.groupdict().items()
+            })
+
+            callback_return = yield callback(request, **kwargs)
+            if callback_return is not None:
+                code, response = callback_return
+                self._send_response(request, code, response)
+
+            return

        # Huh. No one wanted to handle that? Fiiiiiine. Send 400.
-        return _unrecognised_request_handler, {}
+        request_metrics.name = self.__class__.__name__ + ".UnrecognizedRequest"
+        raise UnrecognizedRequestError()

    def _send_response(self, request, code, response_json_object,
                       response_code_message=None):
+        outgoing_responses_counter.inc(request.method, str(code))
+
        # TODO: Only enable CORS for the requests that need it.
        respond_with_json(
            request, code, response_json_object,
            send_cors=True,
            response_code_message=response_code_message,
            pretty_print=_request_user_agent_is_curl(request),
+            version_string=self.version_string,
            canonical_json=self.canonical_json,
        )


-def _options_handler(request):
-    """Request handler for OPTIONS requests
+class RequestMetrics(object):
+    def start(self, clock, name):
+        self.start = clock.time_msec()
+        self.start_context = LoggingContext.current_context()
+        self.name = name

-    This is a request handler suitable for return from
-    _get_handler_for_request. It returns a 200 and an empty body.
+    def stop(self, clock, request):
+        context = LoggingContext.current_context()

-    Args:
-        request (twisted.web.http.Request):
+        tag = ""
+        if context:
+            tag = context.tag

-    Returns:
-        Tuple[int, dict]: http code, response body.
-    """
-    return 200, {}
+            if context != self.start_context:
+                logger.warn(
+                    "Context have unexpectedly changed %r, %r",
+                    context, self.start_context
+                )
+                return

+        response_count.inc(request.method, self.name, tag)

-def _unrecognised_request_handler(request):
-    """Request handler for unrecognised requests
+        response_timer.inc_by(
+            clock.time_msec() - self.start, request.method,
+            self.name, tag
+        )

-    This is a request handler suitable for return from
-    _get_handler_for_request. It actually just raises an
-    UnrecognizedRequestError.
+        ru_utime, ru_stime = context.get_resource_usage()

-    Args:
-        request (twisted.web.http.Request):
-    """
-    raise UnrecognizedRequestError()
+        response_ru_utime.inc_by(
+            ru_utime, request.method, self.name, tag
+        )
+        response_ru_stime.inc_by(
+            ru_stime, request.method, self.name, tag
+        )
+        response_db_txn_count.inc_by(
+            context.db_txn_count, request.method, self.name, tag
+        )
+        response_db_txn_duration.inc_by(
+            context.db_txn_duration_ms / 1000., request.method, self.name, tag
+        )
+        response_db_sched_duration.inc_by(
+            context.db_sched_duration_ms / 1000., request.method, self.name, tag
+        )


 class RootRedirect(resource.Resource):
@@ -322,7 +399,7 @@ class RootRedirect(resource.Resource):

 def respond_with_json(request, code, json_object, send_cors=False,
                      response_code_message=None, pretty_print=False,
-                      canonical_json=True):
+                      version_string="", canonical_json=True):
    # could alternatively use request.notifyFinish() and flip a flag when
    # the Deferred fires, but since the flag is RIGHT THERE it seems like
    # a waste.
@@ -338,17 +415,19 @@ def respond_with_json(request, code, json_object, send_cors=False,
        if canonical_json or synapse.events.USE_FROZEN_DICTS:
            json_bytes = encode_canonical_json(json_object)
        else:
-            json_bytes = simplejson.dumps(json_object)
+            # ujson doesn't like frozen_dicts.
+            json_bytes = ujson.dumps(json_object, ensure_ascii=False)

    return respond_with_json_bytes(
        request, code, json_bytes,
        send_cors=send_cors,
        response_code_message=response_code_message,
+        version_string=version_string
    )


 def respond_with_json_bytes(request, code, json_bytes, send_cors=False,
-                            response_code_message=None):
+                            version_string="", response_code_message=None):
    """Sends encoded JSON in response to the given request.

    Args:
@@ -362,8 +441,8 @@ def respond_with_json_bytes(request, code, json_bytes, send_cors=False,

    request.setResponseCode(code, message=response_code_message)
    request.setHeader(b"Content-Type", b"application/json")
+    request.setHeader(b"Server", version_string)
    request.setHeader(b"Content-Length", b"%d" % (len(json_bytes),))
-    request.setHeader(b"Cache-Control", b"no-cache, no-store, must-revalidate")

    if send_cors:
        set_cors_headers(request)
@@ -411,9 +490,9 @@ def finish_request(request):

 def _request_user_agent_is_curl(request):
    user_agents = request.requestHeaders.getRawHeaders(
-        b"User-Agent", default=[]
+        "User-Agent", default=[]
    )
    for user_agent in user_agents:
-        if b"curl" in user_agent:
+        if "curl" in user_agent:
            return True
    return False
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Richard van der Hoff	b07a33f024	Avoid doing presence updates on replication reconnect Presence is supposed to be disabled on matrix.org, so we shouldn't send a load of USER_SYNC commands every time the synchrotron reconnects to the master.	2018-02-15 09:51:09 +00:00
hera	74539aa25b	Disable auth on room_members for now because the moznet bridge is broken (https://github.com/matrix-org/matrix-appservice-irc/issues/506)	2018-02-15 09:51:09 +00:00
Erik Johnston	8f25cd6627	Bump LAST_SEEN_GRANULARITY in client_ips	2018-02-15 09:51:09 +00:00
Erik Johnston	dc1299a4b0	Prefill client_ip_last_seen in replication	2018-02-15 09:51:09 +00:00
Erik Johnston	2c72d66cda	Move event sending to end in shutdown room admin api	2018-02-15 09:51:09 +00:00
Erik Johnston	cde90a89ed	Add dummy presence REST handler to frontend proxy The handler no-ops all requests as presence is disabled.	2018-02-15 09:51:09 +00:00
Erik Johnston	5aec53ad95	Don't intern type/state_keys in state store	2018-02-15 09:51:09 +00:00
Erik Johnston	d10d19f0ad	Increase store._state_group_cache cache size	2018-02-15 09:51:09 +00:00
Erik Johnston	daec1d77be	Make _get_joined_hosts_cache cache non-iterable	2018-02-15 09:51:09 +00:00
Erik Johnston	61885f7849	Increase MAX_EVENTS_BEHIND for replication clients	2018-02-15 09:51:09 +00:00
Erik Johnston	ba30d489d9	Disable presence in txn queue	2018-02-15 09:51:09 +00:00
Erik Johnston	5e2d0650df	Handle exceptions in get_hosts_for_room when sending events over federation	2018-02-15 09:51:09 +00:00
Erik Johnston	841bcbcafa	Limit concurrent AS joins	2018-02-15 09:51:09 +00:00
Erik Johnston	08a6b88e3d	Disable presence This reverts commit `0ebd376a53` and disables presence a bit more	2018-02-15 09:51:09 +00:00
Erik Johnston	aece8e73b1	Make push actions rotation configurable	2018-02-15 09:51:09 +00:00
Mark Haines	328bd35e00	Deleting from event_push_actions needs to use an index	2018-02-15 09:51:09 +00:00
Erik Johnston	7de9a28b8e	Disable auto search for prefixes in event search	2018-02-15 09:51:09 +00:00
Erik Johnston	2a9c3aea89	Add timeout to ResponseCache of /public_rooms	2018-02-15 09:51:09 +00:00