Require SQLite3 version 3.15 or above

This is primarily to allow tuple comparisons in queries, though a better query optimiser and other improvements mean that using newer versions of sqlite is highly recommended anyway.
2018-05-08 13:48:45 +01:00
130 changed files with 970 additions and 5248 deletions
@@ -1,5 +0,0 @@
-Dockerfile
-.travis.yml
-.gitignore
-demo/etc
-tox.ini
@@ -32,7 +32,6 @@ demo/media_store.*
 demo/etc

 uploads
-cache

 .idea/
 media_store/
@@ -49,4 +48,3 @@ env/
 *.config

 .vscode/
-.ropeproject/
@@ -60,6 +60,3 @@ Niklas Riekenbrauck <nikriek at gmail dot.com>

 Christoph Witzany <christoph at web.crofting.com>
 * Add LDAP support for authentication
-
-Pierre Jaury <pierre at jaury.eu>
-* Docker packaging
@@ -1,82 +1,5 @@
-Changes in <unreleased>
-=======================
-
-This release adds an index to the events table. This means that on first
-startup there will be an inceased amount of IO until the index is created, and
-an increase in disk usage.
-
-
-Changes in synapse v0.30.0 (2018-05-24)
-==========================================
-
-'Server Notices' are a new feature introduced in Synapse 0.30. They provide a
-channel whereby server administrators can send messages to users on the server.
-
-They are used as part of communication of the server policies (see ``docs/consent_tracking.md``),
-however the intention is that they may also find a use for features such
-as "Message of the day".
-
-This feature is specific to Synapse, but uses standard Matrix communication mechanisms,
-so should work with any Matrix client. For more details see ``docs/server_notices.md``
-
-Further Server Notices/Consent Tracking Support:
-
-* Allow overriding the server_notices user's avatar (PR #3273)
-* Use the localpart in the consent uri (PR #3272)
-* Support for putting %(consent_uri)s in messages (PR #3271)
-* Block attempts to send server notices to remote users (PR #3270)
-* Docs on consent bits (PR #3268)
-
-
-
-Changes in synapse v0.30.0-rc1 (2018-05-23)
-==========================================
-
-Server Notices/Consent Tracking Support:
-
-* ConsentResource to gather policy consent from users (PR #3213)
-* Move RoomCreationHandler out of synapse.handlers.Handlers (PR #3225)
-* Infrastructure for a server notices room (PR #3232)
-* Send users a server notice about consent (PR #3236)
-* Reject attempts to send event before privacy consent is given (PR #3257)
-* Add a 'has_consented' template var to consent forms (PR #3262)
-* Fix dependency on jinja2 (PR #3263)
-
-Features:
-
-* Cohort analytics (PR #3163, #3241, #3251)
-* Add lxml to docker image for web previews (PR #3239) Thanks to @ptman!
-* Add in flight request metrics (PR #3252)
-
-Changes:
-
-* Remove unused `update_external_syncs` (PR #3233)
-* Use stream rather depth ordering for push actions (PR #3212)
-* Make purge_history operate on tokens (PR #3221)
-* Don't support limitless pagination (PR #3265)
-
-Bug Fixes:
-
-* Fix logcontext resource usage tracking (PR #3258)
-* Fix error in handling receipts (PR #3235)
-* Stop the transaction cache caching failures (PR #3255)
-
-Changes in synapse v0.29.1 (2018-05-17)
-==========================================
-Changes:
-
-* Update docker documentation (PR #3222)
-
-Changes in synapse v0.29.0 (2018-05-16)
-===========================================
-Not changes since v0.29.0-rc1
-
-Changes in synapse v0.29.0-rc1 (2018-05-14)
-===========================================
-
-Notable changes, a docker file for running Synapse (Thanks to @kaiyou!) and a
-closed spec bug in the Client Server API. Additionally further prep for Python 3
-migration.
+Changes in synapse <unreleased>
+===============================

 Potentially breaking change:

@@ -89,66 +12,6 @@ Potentially breaking change:

  Thanks to @NotAFile for fixing this.

-Features:
-
-* Add a Dockerfile for synapse (PR #2846) Thanks to @kaiyou!
-
-Changes - General:
-
-* nuke-room-from-db.sh: added postgresql option and help (PR #2337) Thanks to @rubo77!
-* Part user from rooms on account deactivate (PR #3201)
-* Make 'unexpected logging context' into warnings (PR #3007)
-* Set Server header in SynapseRequest (PR #3208)
-* remove duplicates from groups tables (PR #3129)
-* Improve exception handling for background processes (PR #3138)
-* Add missing consumeErrors to improve exception handling (PR #3139)
-* reraise exceptions more carefully (PR #3142)
-* Remove redundant call to preserve_fn (PR #3143)
-* Trap exceptions thrown within run_in_background (PR #3144)
-
-Changes - Refactors:
-
-* Refactor /context to reuse pagination storage functions (PR #3193)
-* Refactor recent events func to use pagination func (PR #3195)
-* Refactor pagination DB API to return concrete type (PR #3196)
-* Refactor get_recent_events_for_room return type (PR #3198)
-* Refactor sync APIs to reuse pagination API (PR #3199)
-* Remove unused code path from member change DB func (PR #3200)
-* Refactor request handling wrappers (PR #3203)
-* transaction_id, destination defined twice (PR #3209) Thanks to @damir-manapov!
-* Refactor event storage to prepare for changes in state calculations (PR #3141)
-* Set Server header in SynapseRequest (PR #3208)
-* Use deferred.addTimeout instead of time_bound_deferred (PR #3127, #3178)
-* Use run_in_background in preference to preserve_fn (PR #3140)
-
-Changes - Python 3 migration:
-
-* Construct HMAC as bytes on py3 (PR #3156) Thanks to @NotAFile!
-* run config tests on py3 (PR #3159) Thanks to @NotAFile!
-* Open certificate files as bytes (PR #3084) Thanks to @NotAFile!
-* Open config file in non-bytes mode (PR #3085) Thanks to @NotAFile!
-* Make event properties raise AttributeError instead (PR #3102) Thanks to @NotAFile!
-* Use six.moves.urlparse (PR #3108) Thanks to @NotAFile!
-* Add py3 tests to tox with folders that work (PR #3145) Thanks to @NotAFile!
-* Don't yield in list comprehensions (PR #3150) Thanks to @NotAFile!
-* Move more xrange to six (PR #3151) Thanks to @NotAFile!
-* make imports local (PR #3152) Thanks to @NotAFile!
-* move httplib import to six (PR #3153) Thanks to @NotAFile!
-* Replace stringIO imports with six (PR #3154, #3168) Thanks to @NotAFile!
-* more bytes strings (PR #3155) Thanks to @NotAFile!
-
-Bug Fixes:
-
-* synapse fails to start under Twisted >= 18.4 (PR #3157)
-* Fix a class of logcontext leaks (PR #3170)
-* Fix a couple of logcontext leaks in unit tests (PR #3172)
-* Fix logcontext leak in media repo (PR #3174)
-* Escape label values in prometheus metrics (PR #3175, #3186)
-* Fix 'Unhandled Error' logs with Twisted 18.4 (PR #3182) Thanks to @Half-Shot!
-* Fix logcontext leaks in rate limiter (PR #3183)
-* notifications: Convert next_token to string according to the spec (PR #3190) Thanks to @mujx!
-* nuke-room-from-db.sh: fix deletion from search table (PR #3194) Thanks to @rubo77!
-* add guard for None on purge_history api (PR #3160) Thanks to @krombel!

 Changes in synapse v0.28.1 (2018-05-01)
 =======================================
@@ -1,19 +0,0 @@
-FROM docker.io/python:2-alpine3.7
-
-RUN apk add --no-cache --virtual .nacl_deps su-exec build-base libffi-dev zlib-dev libressl-dev libjpeg-turbo-dev linux-headers postgresql-dev libxslt-dev
-
-COPY . /synapse
-
-# A wheel cache may be provided in ./cache for faster build
-RUN cd /synapse \
- && pip install --upgrade pip setuptools psycopg2 lxml \
- && mkdir -p /synapse/cache \
- && pip install -f /synapse/cache --upgrade --process-dependency-links . \
- && mv /synapse/contrib/docker/start.py /synapse/contrib/docker/conf / \
- && rm -rf setup.py setup.cfg synapse
-
-VOLUME ["/data"]
-
-EXPOSE 8008/tcp 8448/tcp
-
-ENTRYPOINT ["/start.py"]
@@ -25,8 +25,6 @@ recursive-include synapse/static *.js
 exclude jenkins.sh
 exclude jenkins*.sh
 exclude jenkins*
-exclude Dockerfile
-exclude .dockerignore
 recursive-exclude jenkins *.sh

 prune .github
@@ -157,9 +157,8 @@ if you prefer.

 In case of problems, please see the _`Troubleshooting` section below.

-There is an offical synapse image available at https://hub.docker.com/r/matrixdotorg/synapse/tags/ which can be used with the docker-compose file available at `contrib/docker`. Further information on this including configuration options is available in `contrib/docker/README.md`.
-
-Alternatively, Andreas Peters (previously Silvio Fricke) has contributed a Dockerfile to automate a synapse server in a single Docker image, at https://hub.docker.com/r/avhost/docker-matrix/tags/
+Alternatively, Andreas Peters (previously Silvio Fricke) has contributed a Dockerfile to automate the
+above in Docker at https://hub.docker.com/r/avhost/docker-matrix/tags/

 Also, Martin Giess has created an auto-deployment process with vagrant/ansible,
 tested with VirtualBox/AWS/DigitalOcean - see https://github.com/EMnify/matrix-synapse-auto-deploy
@@ -1,153 +0,0 @@
-# Synapse Docker
-
-The `matrixdotorg/synapse` Docker image will run Synapse as a single process. It does not provide a
-database server or a TURN server, you should run these separately.
-
-If you run a Postgres server, you should simply include it in the same Compose
-project or set the proper environment variables and the image will automatically
-use that server.
-
-## Build
-
-Build the docker image with the `docker build` command from the root of the synapse repository.
-
-```
-docker build -t docker.io/matrixdotorg/synapse .
-```
-
-The `-t` option sets the image tag. Official images are tagged `matrixdotorg/synapse:<version>` where `<version>` is the same as the release tag in the synapse git repository.
-
-You may have a local Python wheel cache available, in which case copy the relevant packages in the ``cache/`` directory at the root of the project.
-
-## Run
-
-This image is designed to run either with an automatically generated configuration
-file or with a custom configuration that requires manual edition.
-
-### Automated configuration
-
-It is recommended that you use Docker Compose to run your containers, including
-this image and a Postgres server. A sample ``docker-compose.yml`` is provided,
-including example labels for reverse proxying and other artifacts.
-
-Read the section about environment variables and set at least mandatory variables,
-then run the server:
-
-```
-docker-compose up -d
-```
-
-If secrets are not specified in the environment variables, they will be generated
-as part of the startup. Please ensure these secrets are kept between launches of the
-Docker container, as their loss may require users to log in again.
-
-### Manual configuration
-
-A sample ``docker-compose.yml`` is provided, including example labels for
-reverse proxying and other artifacts. The docker-compose file is an example,
-please comment/uncomment sections that are not suitable for your usecase.
-
-Specify a ``SYNAPSE_CONFIG_PATH``, preferably to a persistent path,
-to use manual configuration. To generate a fresh ``homeserver.yaml``, simply run:
-
-```
-docker-compose run --rm -e SYNAPSE_SERVER_NAME=my.matrix.host synapse generate
-```
-
-Then, customize your configuration and run the server:
-
-```
-docker-compose up -d
-```
-
-### Without Compose
-
-If you do not wish to use Compose, you may still run this image using plain
-Docker commands. Note that the following is just a guideline and you may need
-to add parameters to the docker run command to account for the network situation
-with your postgres database.
-
-```
-docker run \
-    -d \
-    --name synapse \
-    -v ${DATA_PATH}:/data \
-    -e SYNAPSE_SERVER_NAME=my.matrix.host \
-    -e SYNAPSE_REPORT_STATS=yes \
-    docker.io/matrixdotorg/synapse:latest
-```
-
-## Volumes
-
-The image expects a single volume, located at ``/data``, that will hold:
-
-* temporary files during uploads;
-* uploaded media and thumbnails;
-* the SQLite database if you do not configure postgres;
-* the appservices configuration.
-
-You are free to use separate volumes depending on storage endpoints at your
-disposal. For instance, ``/data/media`` coud be stored on a large but low
-performance hdd storage while other files could be stored on high performance
-endpoints.
-
-In order to setup an application service, simply create an ``appservices``
-directory in the data volume and write the application service Yaml
-configuration file there. Multiple application services are supported.
-
-## Environment
-
-Unless you specify a custom path for the configuration file, a very generic
-file will be generated, based on the following environment settings.
-These are a good starting point for setting up your own deployment.
-
-Global settings:
-
-* ``UID``, the user id Synapse will run as [default 991]
-* ``GID``, the group id Synapse will run as [default 991]
-* ``SYNAPSE_CONFIG_PATH``, path to a custom config file
-
-If ``SYNAPSE_CONFIG_PATH`` is set, you should generate a configuration file
-then customize it manually. No other environment variable is required.
-
-Otherwise, a dynamic configuration file will be used. The following environment
-variables are available for configuration:
-
-* ``SYNAPSE_SERVER_NAME`` (mandatory), the current server public hostname.
-* ``SYNAPSE_REPORT_STATS``, (mandatory, ``yes`` or ``no``), enable anonymous
-  statistics reporting back to the Matrix project which helps us to get funding.
-* ``SYNAPSE_NO_TLS``, set this variable to disable TLS in Synapse (use this if
-  you run your own TLS-capable reverse proxy).
-* ``SYNAPSE_ENABLE_REGISTRATION``, set this variable to enable registration on
-  the Synapse instance.
-* ``SYNAPSE_ALLOW_GUEST``, set this variable to allow guest joining this server.
-* ``SYNAPSE_EVENT_CACHE_SIZE``, the event cache size [default `10K`].
-* ``SYNAPSE_CACHE_FACTOR``, the cache factor [default `0.5`].
-* ``SYNAPSE_RECAPTCHA_PUBLIC_KEY``, set this variable to the recaptcha public
-  key in order to enable recaptcha upon registration.
-* ``SYNAPSE_RECAPTCHA_PRIVATE_KEY``, set this variable to the recaptcha private
-  key in order to enable recaptcha upon registration.
-* ``SYNAPSE_TURN_URIS``, set this variable to the coma-separated list of TURN
-  uris to enable TURN for this homeserver.
-* ``SYNAPSE_TURN_SECRET``, set this to the TURN shared secret if required.
-
-Shared secrets, that will be initialized to random values if not set:
-
-* ``SYNAPSE_REGISTRATION_SHARED_SECRET``, secret for registrering users if
-  registration is disable.
-* ``SYNAPSE_MACAROON_SECRET_KEY`` secret for signing access tokens
-  to the server.
-
-Database specific values (will use SQLite if not set):
-
-* `POSTGRES_DB` - The database name for the synapse postgres database. [default: `synapse`]
-* `POSTGRES_HOST` - The host of the postgres database if you wish to use postgresql instead of sqlite3. [default: `db` which is useful when using a container on the same docker network in a compose file where the postgres service is called `db`]
-* `POSTGRES_PASSWORD` - The password for the synapse postgres database. **If this is set then postgres will be used instead of sqlite3.** [default: none] **NOTE**: You are highly encouraged to use postgresql! Please use the compose file to make it easier to deploy.
-* `POSTGRES_USER` - The user for the synapse postgres database. [default: `matrix`]
-
-Mail server specific values (will not send emails if not set):
-
-* ``SYNAPSE_SMTP_HOST``, hostname to the mail server.
-* ``SYNAPSE_SMTP_PORT``, TCP port for accessing the mail server [default ``25``].
-* ``SYNAPSE_SMTP_USER``, username for authenticating against the mail server if any.
-* ``SYNAPSE_SMTP_PASSWORD``, password for authenticating against the mail server if any.
@@ -1,219 +0,0 @@
-# vim:ft=yaml
-
-## TLS ##
-
-tls_certificate_path: "/data/{{ SYNAPSE_SERVER_NAME }}.tls.crt"
-tls_private_key_path: "/data/{{ SYNAPSE_SERVER_NAME }}.tls.key"
-tls_dh_params_path: "/data/{{ SYNAPSE_SERVER_NAME }}.tls.dh"
-no_tls: {{ "True" if SYNAPSE_NO_TLS else "False" }}
-tls_fingerprints: []
-
-## Server ##
-
-server_name: "{{ SYNAPSE_SERVER_NAME }}"
-pid_file: /homeserver.pid
-web_client: False
-soft_file_limit: 0
-
-## Ports ##
-
-listeners:
-  {% if not SYNAPSE_NO_TLS %}
-  -
-    port: 8448
-    bind_addresses: ['0.0.0.0']
-    type: http
-    tls: true
-    x_forwarded: false
-    resources:
-      - names: [client]
-        compress: true
-      - names: [federation]  # Federation APIs
-        compress: false
-  {% endif %}
-
-  - port: 8008
-    tls: false
-    bind_addresses: ['0.0.0.0']
-    type: http
-    x_forwarded: false
-
-    resources:
-      - names: [client]
-        compress: true
-      - names: [federation]
-        compress: false
-
-## Database ##
-
-{% if POSTGRES_PASSWORD %}
-database:
-  name: "psycopg2"
-  args:
-    user: "{{ POSTGRES_USER or "synapse" }}"
-    password: "{{ POSTGRES_PASSWORD }}"
-    database: "{{ POSTGRES_DB or "synapse" }}"
-    host: "{{ POSTGRES_HOST or "db" }}"
-    port: "{{ POSTGRES_PORT or "5432" }}"
-    cp_min: 5
-    cp_max: 10
-{% else %}
-database:
-  name: "sqlite3"
-  args:
-    database: "/data/homeserver.db"
-{% endif %}
-
-## Performance ##
-
-event_cache_size: "{{ SYNAPSE_EVENT_CACHE_SIZE or "10K" }}"
-verbose: 0
-log_file: "/data/homeserver.log"
-log_config: "/compiled/log.config"
-
-## Ratelimiting ##
-
-rc_messages_per_second: 0.2
-rc_message_burst_count: 10.0
-federation_rc_window_size: 1000
-federation_rc_sleep_limit: 10
-federation_rc_sleep_delay: 500
-federation_rc_reject_limit: 50
-federation_rc_concurrent: 3
-
-## Files ##
-
-media_store_path: "/data/media"
-uploads_path: "/data/uploads"
-max_upload_size: "10M"
-max_image_pixels: "32M"
-dynamic_thumbnails: false
-
-# List of thumbnail to precalculate when an image is uploaded.
-thumbnail_sizes:
- width: 32
-  height: 32
-  method: crop
- width: 96
-  height: 96
-  method: crop
- width: 320
-  height: 240
-  method: scale
- width: 640
-  height: 480
-  method: scale
- width: 800
-  height: 600
-  method: scale
-
-url_preview_enabled: False
-max_spider_size: "10M"
-
-## Captcha ##
-
-{% if SYNAPSE_RECAPTCHA_PUBLIC_KEY %}
-recaptcha_public_key: "{{ SYNAPSE_RECAPTCHA_PUBLIC_KEY }}"
-recaptcha_private_key: "{{ SYNAPSE_RECAPTCHA_PRIVATE_KEY }}"
-enable_registration_captcha: True
-recaptcha_siteverify_api: "https://www.google.com/recaptcha/api/siteverify"
-{% else %}
-recaptcha_public_key: "YOUR_PUBLIC_KEY"
-recaptcha_private_key: "YOUR_PRIVATE_KEY"
-enable_registration_captcha: False
-recaptcha_siteverify_api: "https://www.google.com/recaptcha/api/siteverify"
-{% endif %}
-
-## Turn ##
-
-{% if SYNAPSE_TURN_URIS %}
-turn_uris:
-{% for uri in SYNAPSE_TURN_URIS.split(',') %}    - "{{ uri }}"
-{% endfor %}
-turn_shared_secret: "{{ SYNAPSE_TURN_SECRET }}"
-turn_user_lifetime: "1h"
-turn_allow_guests: True
-{% else %}
-turn_uris: []
-turn_shared_secret: "YOUR_SHARED_SECRET"
-turn_user_lifetime: "1h"
-turn_allow_guests: True
-{% endif %}
-
-## Registration ##
-
-enable_registration: {{ "True" if SYNAPSE_ENABLE_REGISTRATION else "False" }}
-registration_shared_secret: "{{ SYNAPSE_REGISTRATION_SHARED_SECRET }}"
-bcrypt_rounds: 12
-allow_guest_access: {{ "True" if SYNAPSE_ALLOW_GUEST else "False" }}
-enable_group_creation: true
-
-# The list of identity servers trusted to verify third party
-# identifiers by this server.
-trusted_third_party_id_servers:
-    - matrix.org
-    - vector.im
-    - riot.im
-
-## Metrics ###
-
-{% if SYNAPSE_REPORT_STATS.lower() == "yes" %}
-enable_metrics: True
-report_stats: True
-{% else %}
-enable_metrics: False
-report_stats: False
-{% endif %}
-
-## API Configuration ##
-
-room_invite_state_types:
-    - "m.room.join_rules"
-    - "m.room.canonical_alias"
-    - "m.room.avatar"
-    - "m.room.name"
-
-{% if SYNAPSE_APPSERVICES %}
-app_service_config_files:
-{% for appservice in SYNAPSE_APPSERVICES %}    - "{{ appservice }}"
-{% endfor %}
-{% else %}
-app_service_config_files: []
-{% endif %}
-
-macaroon_secret_key: "{{ SYNAPSE_MACAROON_SECRET_KEY }}"
-expire_access_token: False
-
-## Signing Keys ##
-
-signing_key_path: "/data/{{ SYNAPSE_SERVER_NAME }}.signing.key"
-old_signing_keys: {}
-key_refresh_interval: "1d" # 1 Day.
-
-# The trusted servers to download signing keys from.
-perspectives:
-  servers:
-    "matrix.org":
-      verify_keys:
-        "ed25519:auto":
-          key: "Noi6WqcDj0QmPxCNQqgezwTlBKrfqehY1u2FyWP9uYw"
-
-password_config:
-   enabled: true
-
-{% if SYNAPSE_SMTP_HOST %}
-email:
-   enable_notifs: false
-   smtp_host: "{{ SYNAPSE_SMTP_HOST }}"
-   smtp_port: {{ SYNAPSE_SMTP_PORT or "25" }}
-   smtp_user: "{{ SYNAPSE_SMTP_USER }}"
-   smtp_pass: "{{ SYNAPSE_SMTP_PASSWORD }}"
-   require_transport_security: False
-   notif_from: "{{ SYNAPSE_SMTP_FROM or "hostmaster@" + SYNAPSE_SERVER_NAME }}"
-   app_name: Matrix
-   template_dir: res/templates
-   notif_template_html: notif_mail.html
-   notif_template_text: notif_mail.txt
-   notif_for_new_users: True
-   riot_base_url: "https://{{ SYNAPSE_SERVER_NAME }}"
-{% endif %}
@@ -1,29 +0,0 @@
-version: 1
-
-formatters:
-  precise:
-   format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s- %(message)s'
-
-filters:
-  context:
-    (): synapse.util.logcontext.LoggingContextFilter
-    request: ""
-
-handlers:
-  console:
-    class: logging.StreamHandler
-    formatter: precise
-    filters: [context]
-
-loggers:
-    synapse:
-        level: {{ SYNAPSE_LOG_LEVEL or "WARNING" }}
-
-    synapse.storage.SQL:
-        # beware: increasing this to DEBUG will make synapse log sensitive
-        # information such as access tokens.
-        level: {{ SYNAPSE_LOG_LEVEL or "WARNING" }}
-
-root:
-    level: {{ SYNAPSE_LOG_LEVEL or "WARNING" }}
-    handlers: [console]
@@ -1,49 +0,0 @@
-# This compose file is compatible with Compose itself, it might need some
-# adjustments to run properly with stack.
-
-version: '3'
-
-services:
-
-  synapse:
-    image: docker.io/matrixdotorg/synapse:latest
-    # Since snyapse does not retry to connect to the database, restart upon
-    # failure
-    restart: unless-stopped
-    # See the readme for a full documentation of the environment settings
-    environment:
-      - SYNAPSE_SERVER_NAME=my.matrix.host
-      - SYNAPSE_REPORT_STATS=no
-      - SYNAPSE_ENABLE_REGISTRATION=yes
-      - SYNAPSE_LOG_LEVEL=INFO
-      - POSTGRES_PASSWORD=changeme
-    volumes:
-      # You may either store all the files in a local folder
-      - ./files:/data
-      # .. or you may split this between different storage points
-      # - ./files:/data
-      # - /path/to/ssd:/data/uploads
-      # - /path/to/large_hdd:/data/media
-    depends_on:
-      - db
-    # In order to expose Synapse, remove one of the following, you might for
-    # instance expose the TLS port directly:
-    ports:
-      - 8448:8448/tcp
-    # ... or use a reverse proxy, here is an example for traefik:
-    labels:
-      - traefik.enable=true
-      - traefik.frontend.rule=Host:my.matrix.Host
-      - traefik.port=8448
-
-  db:
-    image: docker.io/postgres:10-alpine
-    # Change that password, of course!
-    environment:
-      - POSTGRES_USER=synapse
-      - POSTGRES_PASSWORD=changeme
-    volumes:
-      # You may store the database tables in a local folder..
-      - ./schemas:/var/lib/postgresql/data
-      # .. or store them on some high performance storage for better results
-      # - /path/to/ssd/storage:/var/lib/postfesql/data
@@ -1,66 +0,0 @@
-#!/usr/local/bin/python
-
-import jinja2
-import os
-import sys
-import subprocess
-import glob
-
-# Utility functions
-convert = lambda src, dst, environ: open(dst, "w").write(jinja2.Template(open(src).read()).render(**environ))
-
-def check_arguments(environ, args):
-    for argument in args:
-        if argument not in environ:
-            print("Environment variable %s is mandatory, exiting." % argument)
-            sys.exit(2)
-
-def generate_secrets(environ, secrets):
-    for name, secret in secrets.items():
-        if secret not in environ:
-            filename = "/data/%s.%s.key" % (environ["SYNAPSE_SERVER_NAME"], name)
-            if os.path.exists(filename):
-                with open(filename) as handle: value = handle.read()
-            else:
-                print("Generating a random secret for {}".format(name))
-                value = os.urandom(32).encode("hex")
-                with open(filename, "w") as handle: handle.write(value)
-            environ[secret] = value
-
-# Prepare the configuration
-mode = sys.argv[1] if len(sys.argv) > 1 else None
-environ = os.environ.copy()
-ownership = "{}:{}".format(environ.get("UID", 991), environ.get("GID", 991))
-args = ["python", "-m", "synapse.app.homeserver"]
-
-# In generate mode, generate a configuration, missing keys, then exit
-if mode == "generate":
-    check_arguments(environ, ("SYNAPSE_SERVER_NAME", "SYNAPSE_REPORT_STATS", "SYNAPSE_CONFIG_PATH"))
-    args += [
-        "--server-name", environ["SYNAPSE_SERVER_NAME"],
-        "--report-stats", environ["SYNAPSE_REPORT_STATS"],
-        "--config-path", environ["SYNAPSE_CONFIG_PATH"],
-        "--generate-config"
-    ]
-    os.execv("/usr/local/bin/python", args)
-
-# In normal mode, generate missing keys if any, then run synapse
-else:
-    # Parse the configuration file
-    if "SYNAPSE_CONFIG_PATH" in environ:
-        args += ["--config-path", environ["SYNAPSE_CONFIG_PATH"]]
-    else:
-        check_arguments(environ, ("SYNAPSE_SERVER_NAME", "SYNAPSE_REPORT_STATS"))
-        generate_secrets(environ, {
-            "registration": "SYNAPSE_REGISTRATION_SHARED_SECRET",
-            "macaroon": "SYNAPSE_MACAROON_SECRET_KEY"
-        })
-        environ["SYNAPSE_APPSERVICES"] = glob.glob("/data/appservices/*.yaml")
-        if not os.path.exists("/compiled"): os.mkdir("/compiled")
-        convert("/conf/homeserver.yaml", "/compiled/homeserver.yaml", environ)
-        convert("/conf/log.config", "/compiled/log.config", environ)
-        subprocess.check_output(["chown", "-R", ownership, "/data"])
-        args += ["--config-path", "/compiled/homeserver.yaml"]
-    # Generate missing keys and start synapse
-    subprocess.check_output(args + ["--generate-keys"])
-    os.execv("/sbin/su-exec", ["su-exec", ownership] + args)
@@ -1,160 +0,0 @@
-Support in Synapse for tracking agreement to server terms and conditions
-========================================================================
-
-Synapse 0.30 introduces support for tracking whether users have agreed to the
-terms and conditions set by the administrator of a server - and blocking access
-to the server until they have.
-
-There are several parts to this functionality; each requires some specific
-configuration in `homeserver.yaml` to be enabled.
-
-Note that various parts of the configuation and this document refer to the
-"privacy policy": agreement with a privacy policy is one particular use of this
-feature, but of course adminstrators can specify other terms and conditions
-unrelated to "privacy" per se.
-
-Collecting policy agreement from a user
---------------------------------------
-
-Synapse can be configured to serve the user a simple policy form with an
-"accept" button. Clicking "Accept" records the user's acceptance in the
-database and shows a success page.
-
-To enable this, first create templates for the policy and success pages.
-These should be stored on the local filesystem.
-
-These templates use the [Jinja2](http://jinja.pocoo.org) templating language,
-and [docs/privacy_policy_templates](privacy_policy_templates) gives
-examples of the sort of thing that can be done.
-
-Note that the templates must be stored under a name giving the language of the
-template - currently this must always be `en` (for "English");
-internationalisation support is intended for the future.
-
-The template for the policy itself should be versioned and named according to 
-the version: for example `1.0.html`. The version of the policy which the user
-has agreed to is stored in the database.
-
-Once the templates are in place, make the following changes to `homeserver.yaml`:
-
- 1. Add a `user_consent` section, which should look like:
-
-    ```yaml
-    user_consent:
-      template_dir: privacy_policy_templates
-      version: 1.0
-    ```
-
-    `template_dir` points to the directory containing the policy
-    templates. `version` defines the version of the policy which will be served
-    to the user. In the example above, Synapse will serve
-    `privacy_policy_templates/en/1.0.html`.
-
-
- 2. Add a `form_secret` setting at the top level:
-
-
-    ```yaml
-    form_secret: "<unique secret>"
-    ```
-
-    This should be set to an arbitrary secret string (try `pwgen -y 30` to
-    generate suitable secrets).
-
-    More on what this is used for below.
-
- 3. Add `consent` wherever the `client` resource is currently enabled in the
-    `listeners` configuration. For example:
-
-    ```yaml
-    listeners:
-      - port: 8008
-        resources:
-          - names:
-            - client
-            - consent
-    ```
-
-
-Finally, ensure that `jinja2` is installed. If you are using a virtualenv, this
-should be a matter of `pip install Jinja2`. On debian, try `apt-get install
-python-jinja2`.
-
-Once this is complete, and the server has been restarted, try visiting
-`https://<server>/_matrix/consent`. If correctly configured, this should give
-an error "Missing string query parameter 'u'". It is now possible to manually
-construct URIs where users can give their consent.
-
-### Constructing the consent URI
-
-It may be useful to manually construct the "consent URI" for a given user - for
-instance, in order to send them an email asking them to consent. To do this,
-take the base `https://<server>/_matrix/consent` URL and add the following
-query parameters:
-
- * `u`: the user id of the user. This can either be a full MXID
-   (`@user:server.com`) or just the localpart (`user`).
-
- * `h`: hex-encoded HMAC-SHA256 of `u` using the `form_secret` as a key. It is
-   possible to calculate this on the commandline with something like:
-
-   ```bash
-   echo -n '<user>' | openssl sha256 -hmac '<form_secret>'
-   ```
-
-   This should result in a URI which looks something like:
-   `https://<server>/_matrix/consent?u=<user>&h=68a152465a4d...`.
-
-
-Sending users a server notice asking them to agree to the policy
----------------------------------------------------------------
-
-It is possible to configure Synapse to send a [server
-notice](server_notices.md) to anybody who has not yet agreed to the current
-version of the policy. To do so:
-
- * ensure that the consent resource is configured, as in the previous section
-
- * ensure that server notices are configured, as in [server_notices.md](server_notices.md).
-
- * Add `server_notice_content` under `user_consent` in `homeserver.yaml`. For
-   example:
-
-   ```yaml
-   user_consent:
-     server_notice_content:
-       msgtype: m.text
-       body: >-
-         Please give your consent to the privacy policy at %(consent_uri)s.
-   ```
-
-   Synapse automatically replaces the placeholder `%(consent_uri)s` with the
-   consent uri for that user.
-
- * ensure that `public_baseurl` is set in `homeserver.yaml`, and gives the base
-   URI that clients use to connect to the server. (It is used to construct
-   `consent_uri` in the server notice.)
-
-
-Blocking users from using the server until they agree to the policy
-------------------------------------------------------------------
-
-Synapse can be configured to block any attempts to join rooms or send messages
-until the user has given their agreement to the policy. (Joining the server
-notices room is exempted from this).
-
-To enable this, add `block_events_error` under `user_consent`. For example:
-
-```yaml
-user_consent:
-  block_events_error: >-
-    You can't send any messages until you consent to the privacy policy at
-    %(consent_uri)s.
-```
-
-Synapse automatically replaces the placeholder `%(consent_uri)s` with the
-consent uri for that user.
-
-ensure that `public_baseurl` is set in `homeserver.yaml`, and gives the base
-URI that clients use to connect to the server. (It is used to construct
-`consent_uri` in the error.)
@@ -1,43 +0,0 @@
-Using the synapse manhole
-=========================
-
-The "manhole" allows server administrators to access a Python shell on a running
-Synapse installation. This is a very powerful mechanism for administration and
-debugging.
-
-To enable it, first uncomment the `manhole` listener configuration in
-`homeserver.yaml`:
-
-```yaml
-listeners:
-  - port: 9000
-    bind_addresses: ['::1', '127.0.0.1']
-    type: manhole
-```
-
-(`bind_addresses` in the above is important: it ensures that access to the
-manhole is only possible for local users).
-
-Note that this will give administrative access to synapse to **all users** with
-shell access to the server. It should therefore **not** be enabled in
-environments where untrusted users have shell access.
-
-Then restart synapse, and point an ssh client at port 9000 on localhost, using
-the username `matrix`:
-
-```bash
-ssh -p9000 matrix@localhost
-```
-
-The password is `rabbithole`.
-
-This gives a Python REPL in which `hs` gives access to the
-`synapse.server.HomeServer` object - which in turn gives access to many other
-parts of the process.
-
-As a simple example, retrieving an event from the database:
-
-```
->>> hs.get_datastore().get_event('$1416420717069yeQaw:matrix.org')
-<Deferred at 0x7ff253fc6998 current result: <FrozenEvent event_id='$1416420717069yeQaw:matrix.org', type='m.room.create', state_key=''>>
-```
@@ -6,13 +6,7 @@ Postgres version 9.4 or later is known to work.
 Set up database
 ===============

-Assuming your PostgreSQL database user is called ``postgres``, create a user
-``synapse_user`` with::
-
- su - postgres
- createuser --pwprompt synapse_user
-
-The PostgreSQL database used *must* have the correct encoding set, otherwise it
+The PostgreSQL database used *must* have the correct encoding set, otherwise
 would not be able to store UTF8 strings. To create a database with the correct
 encoding use, e.g.::

@@ -52,8 +46,8 @@ As with Debian/Ubuntu, postgres support depends on the postgres python connector
 Synapse config
 ==============

-When you are ready to start using PostgreSQL, edit the ``database`` section in
-your config file to match the following lines::
+When you are ready to start using PostgreSQL, add the following line to your
+config file::

    database:
        name: psycopg2
@@ -102,12 +96,9 @@ complete, restart synapse.  For instance::
    cp homeserver.db homeserver.db.snapshot
    ./synctl start

-Copy the old config file into a new config file::
-
-    cp homeserver.yaml homeserver-postgres.yaml
-
-Edit the database section as described in the section *Synapse config* above
-and with the SQLite snapshot located at ``homeserver.db.snapshot`` simply run::
+Assuming your new config file (as described in the section *Synapse config*)
+is named ``homeserver-postgres.yaml`` and the SQLite snapshot is at
+``homeserver.db.snapshot`` then simply run::

    synapse_port_db --sqlite-database homeserver.db.snapshot \
        --postgres-config homeserver-postgres.yaml
@@ -126,11 +117,6 @@ run::
        --postgres-config homeserver-postgres.yaml

 Once that has completed, change the synapse config to point at the PostgreSQL
-database configuration file ``homeserver-postgres.yaml``:
-
-    ./synctl stop
-    mv homeserver.yaml homeserver-old-sqlite.yaml 
-    mv homeserver-postgres.yaml homeserver.yaml 
-    ./synctl start
-
-Synapse should now be running against PostgreSQL.
+database configuration file ``homeserver-postgres.yaml`` (i.e. rename it to 
+``homeserver.yaml``) and restart synapse. Synapse should now be running against
+PostgreSQL.
@@ -1,23 +0,0 @@
-<!doctype html>
-<html lang="en">
-  <head>
-    <title>Matrix.org Privacy policy</title>
-  </head>
-  <body>
-  {% if has_consented %}
-    <p>
-      Your base already belong to us.
-    </p>
-  {% else %}
-    <p>
-      All your base are belong to us.
-    </p>
-    <form method="post" action="consent">
-      <input type="hidden" name="v" value="{{version}}"/>
-      <input type="hidden" name="u" value="{{user}}"/>
-      <input type="hidden" name="h" value="{{userhmac}}"/>
-      <input type="submit" value="Sure thing!"/>
-    </form>
-  {% endif %}
-  </body>
-</html>
@@ -1,11 +0,0 @@
-<!doctype html>
-<html lang="en">
-  <head>
-    <title>Matrix.org Privacy policy</title>
-  </head>
-  <body>
-    <p>
-      Sweet.
-    </p>
-  </body>
-</html>
@@ -1,71 +0,0 @@
-Server Notices
-==============
-
-'Server Notices' are a new feature introduced in Synapse 0.30. They provide a
-channel whereby server administrators can send messages to users on the server.
-
-They are used as part of communication of the server polices(see
-[consent_tracking.md](consent_tracking.md)), however the intention is that 
-they may also find a use for features such as "Message of the day".
-
-This is a feature specific to Synapse, but it uses standard Matrix
-communication mechanisms, so should work with any Matrix client.
-
-User experience
---------------
-
-When the user is first sent a server notice, they will get an invitation to a
-room (typically called 'Server Notices', though this is configurable in
-`homeserver.yaml`). They will be **unable to reject** this invitation -
-attempts to do so will receive an error.
-
-Once they accept the invitation, they will see the notice message in the room
-history; it will appear to have come from the 'server notices user' (see
-below).
-
-The user is prevented from sending any messages in this room by the power
-levels. They also cannot leave it.
-
-Synapse configuration
---------------------
-
-Server notices come from a specific user id on the server. Server
-administrators are free to choose the user id - something like `server` is
-suggested, meaning the notices will come from
-`@server:<your_server_name>`. Once the Server Notices user is configured, that
-user id becomes a special, privileged user, so administrators should ensure
-that **it is not already allocated**.
-
-In order to support server notices, it is necessary to add some configuration
-to the `homeserver.yaml` file. In particular, you should add a `server_notices`
-section, which should look like this:
-
-```yaml
-server_notices:
-   system_mxid_localpart: server
-   system_mxid_display_name: "Server Notices"
-   system_mxid_avatar_url: "mxc://server.com/oumMVlgDnLYFaPVkExemNVVZ"
-   room_name: "Server Notices"
-```
-
-The only compulsory setting is `system_mxid_localpart`, which defines the user
-id of the Server Notices user, as above. `room_name` defines the name of the
-room which will be created.
-
-`system_mxid_display_name` and `system_mxid_avatar_url` can be used to set the
-displayname and avatar of the Server Notices user.
-
-Sending notices
---------------
-
-As of the current version of synapse, there is no convenient interface for
-sending notices (other than the automated ones sent as part of consent
-tracking).
-
-In the meantime, it is possible to test this feature using the manhole. Having
-gone into the manhole as described in [manhole.md](manhole.md), a notice can be
-sent with something like:
-
-```
->>> hs.get_server_notices_manager().send_notice('@user:server.com', {'msgtype':'m.text', 'body':'foo'})
-```
@@ -6,19 +6,9 @@

 ## Do not run it lightly.

-set -e
-
-if [ "$1" == "-h" ] || [ "$1" == "" ]; then
-  echo "Call with ROOM_ID as first option and then pipe it into the database. So for instance you might run"
-  echo " nuke-room-from-db.sh <room_id> | sqlite3 homeserver.db"
-  echo "or"
-  echo " nuke-room-from-db.sh <room_id> | psql --dbname=synapse"
-  exit
-fi
-
 ROOMID="$1"

-cat <<EOF
+sqlite3 homeserver.db <<EOF
 DELETE FROM event_forward_extremities WHERE room_id = '$ROOMID';
 DELETE FROM event_backward_extremities WHERE room_id = '$ROOMID';
 DELETE FROM event_edges WHERE room_id = '$ROOMID';
@@ -39,7 +29,7 @@ DELETE FROM state_groups WHERE room_id = '$ROOMID';
 DELETE FROM state_groups_state WHERE room_id = '$ROOMID';
 DELETE FROM receipts_graph WHERE room_id = '$ROOMID';
 DELETE FROM receipts_linearized WHERE room_id = '$ROOMID';
-DELETE FROM event_search WHERE room_id = '$ROOMID'; 
+DELETE FROM event_search_content WHERE c1room_id = '$ROOMID';
 DELETE FROM guest_access WHERE room_id = '$ROOMID';
 DELETE FROM history_visibility WHERE room_id = '$ROOMID';
 DELETE FROM room_tags WHERE room_id = '$ROOMID';
@@ -16,4 +16,4 @@
 """ This is a reference implementation of a Matrix home server.
 """

-__version__ = "0.30.0"
+__version__ = "0.28.1"
@@ -19,7 +19,6 @@ import logging

 import simplejson as json
 from six import iteritems
-from six.moves import http_client

 logger = logging.getLogger(__name__)

@@ -52,8 +51,6 @@ class Codes(object):
    THREEPID_DENIED = "M_THREEPID_DENIED"
    INVALID_USERNAME = "M_INVALID_USERNAME"
    SERVER_NOT_TRUSTED = "M_SERVER_NOT_TRUSTED"
-    CONSENT_NOT_GIVEN = "M_CONSENT_NOT_GIVEN"
-    CANNOT_LEAVE_SERVER_NOTICE_ROOM = "M_CANNOT_LEAVE_SERVER_NOTICE_ROOM"


 class CodeMessageException(RuntimeError):
@@ -141,32 +138,6 @@ class SynapseError(CodeMessageException):
        return res


-class ConsentNotGivenError(SynapseError):
-    """The error returned to the client when the user has not consented to the
-    privacy policy.
-    """
-    def __init__(self, msg, consent_uri):
-        """Constructs a ConsentNotGivenError
-
-        Args:
-            msg (str): The human-readable error message
-            consent_url (str): The URL where the user can give their consent
-        """
-        super(ConsentNotGivenError, self).__init__(
-            code=http_client.FORBIDDEN,
-            msg=msg,
-            errcode=Codes.CONSENT_NOT_GIVEN
-        )
-        self._consent_uri = consent_uri
-
-    def error_dict(self):
-        return cs_error(
-            self.msg,
-            self.errcode,
-            consent_uri=self._consent_uri
-        )
-
-
 class RegistrationError(SynapseError):
    """An error raised when a registration event fails."""
    pass
@@ -321,7 +292,7 @@ def cs_error(msg, code=Codes.UNKNOWN, **kwargs):

    Args:
        msg (str): The error message.
-        code (str): The error code.
+        code (int): The error code.
        kwargs : Additional keys to add to the response.
    Returns:
        A dict representing the error response JSON.
@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2018 New Vector Ltd.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,12 +14,6 @@
 # limitations under the License.

 """Contains the URL paths to prefix various aspects of the server with. """
-from hashlib import sha256
-import hmac
-
-from six.moves.urllib.parse import urlencode
-
-from synapse.config import ConfigError

 CLIENT_PREFIX = "/_matrix/client/api/v1"
 CLIENT_V2_ALPHA_PREFIX = "/_matrix/client/v2_alpha"
@@ -32,46 +25,3 @@ SERVER_KEY_PREFIX = "/_matrix/key/v1"
 SERVER_KEY_V2_PREFIX = "/_matrix/key/v2"
 MEDIA_PREFIX = "/_matrix/media/r0"
 LEGACY_MEDIA_PREFIX = "/_matrix/media/v1"
-
-
-class ConsentURIBuilder(object):
-    def __init__(self, hs_config):
-        """
-        Args:
-            hs_config (synapse.config.homeserver.HomeServerConfig):
-        """
-        if hs_config.form_secret is None:
-            raise ConfigError(
-                "form_secret not set in config",
-            )
-        if hs_config.public_baseurl is None:
-            raise ConfigError(
-                "public_baseurl not set in config",
-            )
-
-        self._hmac_secret = hs_config.form_secret.encode("utf-8")
-        self._public_baseurl = hs_config.public_baseurl
-
-    def build_user_consent_uri(self, user_id):
-        """Build a URI which we can give to the user to do their privacy
-        policy consent
-
-        Args:
-            user_id (str): mxid or username of user
-
-        Returns
-            (str) the URI where the user can do consent
-        """
-        mac = hmac.new(
-            key=self._hmac_secret,
-            msg=user_id,
-            digestmod=sha256,
-        ).hexdigest()
-        consent_uri = "%s_matrix/consent?%s" % (
-            self._public_baseurl,
-            urlencode({
-                "u": user_id,
-                "h": mac
-            }),
-        )
-        return consent_uri
@@ -74,7 +74,6 @@ class AppserviceServer(HomeServer):
                site_tag,
                listener_config,
                root_resource,
-                self.version_string,
            )
        )

@@ -98,7 +98,6 @@ class ClientReaderServer(HomeServer):
                site_tag,
                listener_config,
                root_resource,
-                self.version_string,
            )
        )

@@ -114,7 +114,6 @@ class EventCreatorServer(HomeServer):
                site_tag,
                listener_config,
                root_resource,
-                self.version_string,
            )
        )

@@ -87,7 +87,6 @@ class FederationReaderServer(HomeServer):
                site_tag,
                listener_config,
                root_resource,
-                self.version_string,
            )
        )

@@ -101,7 +101,6 @@ class FederationSenderServer(HomeServer):
                site_tag,
                listener_config,
                root_resource,
-                self.version_string,
            )
        )

@@ -152,7 +152,6 @@ class FrontendProxyServer(HomeServer):
                site_tag,
                listener_config,
                root_resource,
-                self.version_string,
            )
        )

@@ -140,7 +140,6 @@ class SynapseHomeServer(HomeServer):
                    site_tag,
                    listener_config,
                    root_resource,
-                    self.version_string,
                ),
                self.tls_server_context_factory,
            )
@@ -154,7 +153,6 @@ class SynapseHomeServer(HomeServer):
                    site_tag,
                    listener_config,
                    root_resource,
-                    self.version_string,
                )
            )
        logger.info("Synapse now listening on port %d", port)
@@ -184,15 +182,6 @@ class SynapseHomeServer(HomeServer):
                "/_matrix/client/versions": client_resource,
            })

-        if name == "consent":
-            from synapse.rest.consent.consent_resource import ConsentResource
-            consent_resource = ConsentResource(self)
-            if compress:
-                consent_resource = gz_wrap(consent_resource)
-            resources.update({
-                "/_matrix/consent": consent_resource,
-            })
-
        if name == "federation":
            resources.update({
                FEDERATION_PREFIX: TransportLayerServer(self),
@@ -484,14 +473,6 @@ def run(hs):
                " changes across releases."
            )

-    def generate_user_daily_visit_stats():
-        hs.get_datastore().generate_user_daily_visits()
-
-    # Rather than update on per session basis, batch up the requests.
-    # If you increase the loop period, the accuracy of user_daily_visits
-    # table will decrease
-    clock.looping_call(generate_user_daily_visit_stats, 5 * 60 * 1000)
-
    if hs.config.report_stats:
        logger.info("Scheduling stats reporting for 3 hour intervals")
        clock.looping_call(phone_stats_home, 3 * 60 * 60 * 1000)
@@ -94,7 +94,6 @@ class MediaRepositoryServer(HomeServer):
                site_tag,
                listener_config,
                root_resource,
-                self.version_string,
            )
        )

@@ -104,7 +104,6 @@ class PusherServer(HomeServer):
                site_tag,
                listener_config,
                root_resource,
-                self.version_string,
            )
        )

@@ -281,7 +281,6 @@ class SynchrotronServer(HomeServer):
                site_tag,
                listener_config,
                root_resource,
-                self.version_string,
            )
        )

@@ -126,7 +126,6 @@ class UserDirectoryServer(HomeServer):
                site_tag,
                listener_config,
                root_resource,
-                self.version_string,
            )
        )

@@ -12,9 +12,3 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
-from ._base import ConfigError
-
-# export ConfigError if somebody does import *
-# this is largely a fudge to stop PEP8 moaning about the import
-__all__ = ["ConfigError"]
@@ -1,79 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2018 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from ._base import Config
-
-DEFAULT_CONFIG = """\
-# User Consent configuration
-#
-# Parts of this section are required if enabling the 'consent' resource under
-# 'listeners', in particular 'template_dir' and 'version'.
-#
-# 'template_dir' gives the location of the templates for the HTML forms.
-# This directory should contain one subdirectory per language (eg, 'en', 'fr'),
-# and each language directory should contain the policy document (named as
-# '<version>.html') and a success page (success.html).
-#
-# 'version' specifies the 'current' version of the policy document. It defines
-# the version to be served by the consent resource if there is no 'v'
-# parameter.
-#
-# 'server_notice_content', if enabled, will send a user a "Server Notice"
-# asking them to consent to the privacy policy. The 'server_notices' section
-# must also be configured for this to work.
-#
-# 'block_events_error', if set, will block any attempts to send events
-# until the user consents to the privacy policy. The value of the setting is
-# used as the text of the error.
-#
-# user_consent:
-#   template_dir: res/templates/privacy
-#   version: 1.0
-#   server_notice_content:
-#     msgtype: m.text
-#     body: >-
-#       To continue using this homeserver you must review and agree to the
-#       terms and conditions at %(consent_uri)s
-#   block_events_error: >-
-#     To continue using this homeserver you must review and agree to the
-#     terms and conditions at %(consent_uri)s
-#
-"""
-
-
-class ConsentConfig(Config):
-    def __init__(self):
-        super(ConsentConfig, self).__init__()
-
-        self.user_consent_version = None
-        self.user_consent_template_dir = None
-        self.user_consent_server_notice_content = None
-        self.block_events_without_consent_error = None
-
-    def read_config(self, config):
-        consent_config = config.get("user_consent")
-        if consent_config is None:
-            return
-        self.user_consent_version = str(consent_config["version"])
-        self.user_consent_template_dir = consent_config["template_dir"]
-        self.user_consent_server_notice_content = consent_config.get(
-            "server_notice_content",
-        )
-        self.block_events_without_consent_error = consent_config.get(
-            "block_events_error",
-        )
-
-    def default_config(self, **kwargs):
-        return DEFAULT_CONFIG
@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -13,6 +12,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
 from .tls import TlsConfig
 from .server import ServerConfig
 from .logger import LoggingConfig
@@ -37,8 +37,6 @@ from .push import PushConfig
 from .spam_checker import SpamCheckerConfig
 from .groups import GroupsConfig
 from .user_directory import UserDirectoryConfig
-from .consent_config import ConsentConfig
-from .server_notices_config import ServerNoticesConfig


 class HomeServerConfig(TlsConfig, ServerConfig, DatabaseConfig, LoggingConfig,
@@ -47,15 +45,12 @@ class HomeServerConfig(TlsConfig, ServerConfig, DatabaseConfig, LoggingConfig,
                       AppServiceConfig, KeyConfig, SAML2Config, CasConfig,
                       JWTConfig, PasswordConfig, EmailConfig,
                       WorkerConfig, PasswordAuthProviderConfig, PushConfig,
-                       SpamCheckerConfig, GroupsConfig, UserDirectoryConfig,
-                       ConsentConfig,
-                       ServerNoticesConfig,
-                       ):
+                       SpamCheckerConfig, GroupsConfig, UserDirectoryConfig,):
    pass


 if __name__ == '__main__':
    import sys
    sys.stdout.write(
-        HomeServerConfig().generate_config(sys.argv[1], sys.argv[2], True)[0]
+        HomeServerConfig().generate_config(sys.argv[1], sys.argv[2])[0]
    )
@@ -59,20 +59,14 @@ class KeyConfig(Config):

        self.expire_access_token = config.get("expire_access_token", False)

-        # a secret which is used to calculate HMACs for form values, to stop
-        # falsification of values
-        self.form_secret = config.get("form_secret", None)
-
    def default_config(self, config_dir_path, server_name, is_generating_file=False,
                       **kwargs):
        base_key_name = os.path.join(config_dir_path, server_name)

        if is_generating_file:
            macaroon_secret_key = random_string_with_symbols(50)
-            form_secret = '"%s"' % random_string_with_symbols(50)
        else:
            macaroon_secret_key = None
-            form_secret = 'null'

        return """\
        macaroon_secret_key: "%(macaroon_secret_key)s"
@@ -80,10 +74,6 @@ class KeyConfig(Config):
        # Used to enable access token expiration.
        expire_access_token: False

-        # a secret which is used to calculate HMACs for form values, to stop
-        # falsification of values
-        form_secret: %(form_secret)s
-
        ## Signing Keys ##

        # Path to the signing key to sign messages with
@@ -1,86 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2018 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-from ._base import Config
-from synapse.types import UserID
-
-DEFAULT_CONFIG = """\
-# Server Notices room configuration
-#
-# Uncomment this section to enable a room which can be used to send notices
-# from the server to users. It is a special room which cannot be left; notices
-# come from a special "notices" user id.
-#
-# If you uncomment this section, you *must* define the system_mxid_localpart
-# setting, which defines the id of the user which will be used to send the
-# notices.
-#
-# It's also possible to override the room name, the display name of the
-# "notices" user, and the avatar for the user.
-#
-# server_notices:
-#   system_mxid_localpart: notices
-#   system_mxid_display_name: "Server Notices"
-#   system_mxid_avatar_url: "mxc://server.com/oumMVlgDnLYFaPVkExemNVVZ"
-#   room_name: "Server Notices"
-"""
-
-
-class ServerNoticesConfig(Config):
-    """Configuration for the server notices room.
-
-    Attributes:
-        server_notices_mxid (str|None):
-            The MXID to use for server notices.
-            None if server notices are not enabled.
-
-        server_notices_mxid_display_name (str|None):
-            The display name to use for the server notices user.
-            None if server notices are not enabled.
-
-        server_notices_mxid_avatar_url (str|None):
-            The display name to use for the server notices user.
-            None if server notices are not enabled.
-
-        server_notices_room_name (str|None):
-            The name to use for the server notices room.
-            None if server notices are not enabled.
-    """
-    def __init__(self):
-        super(ServerNoticesConfig, self).__init__()
-        self.server_notices_mxid = None
-        self.server_notices_mxid_display_name = None
-        self.server_notices_mxid_avatar_url = None
-        self.server_notices_room_name = None
-
-    def read_config(self, config):
-        c = config.get("server_notices")
-        if c is None:
-            return
-
-        mxid_localpart = c['system_mxid_localpart']
-        self.server_notices_mxid = UserID(
-            mxid_localpart, self.server_name,
-        ).to_string()
-        self.server_notices_mxid_display_name = c.get(
-            'system_mxid_display_name', None,
-        )
-        self.server_notices_mxid_avatar_url = c.get(
-            'system_mxid_avatar_url', None,
-        )
-        # todo: i18n
-        self.server_notices_room_name = c.get('room_name', "Server Notices")
-
-    def default_config(self, **kwargs):
-        return DEFAULT_CONFIG
@@ -20,8 +20,6 @@ from frozendict import frozendict

 import re

-from six import string_types
-
 # Split strings on "." but not "\." This uses a negative lookbehind assertion for '\'
 # (?<!stuff) matches if the current position in the string is not preceded
 # by a match for 'stuff'.
@@ -279,7 +277,7 @@ def serialize_event(e, time_now_ms, as_client_event=True,

    if only_event_fields:
        if (not isinstance(only_event_fields, list) or
-                not all(isinstance(f, string_types) for f in only_event_fields)):
+                not all(isinstance(f, basestring) for f in only_event_fields)):
            raise TypeError("only_event_fields must be a list of strings")
        d = only_fields(d, only_event_fields)

@@ -17,8 +17,6 @@ from synapse.types import EventID, RoomID, UserID
 from synapse.api.errors import SynapseError
 from synapse.api.constants import EventTypes, Membership

-from six import string_types
-

 class EventValidator(object):

@@ -51,7 +49,7 @@ class EventValidator(object):
            strings.append("state_key")

        for s in strings:
-            if not isinstance(getattr(event, s), string_types):
+            if not isinstance(getattr(event, s), basestring):
                raise SynapseError(400, "Not '%s' a string type" % (s,))

        if event.type == EventTypes.Member:
@@ -90,5 +88,5 @@ class EventValidator(object):
        for s in keys:
            if s not in d:
                raise SynapseError(400, "'%s' not in content" % (s,))
-            if not isinstance(d[s], string_types):
+            if not isinstance(d[s], basestring):
                raise SynapseError(400, "Not '%s' a string type" % (s,))
@@ -74,6 +74,8 @@ class Transaction(JsonEncodedObject):
        "previous_ids",
        "pdus",
        "edus",
+        "transaction_id",
+        "destination",
        "pdu_failures",
    ]

@@ -20,8 +20,6 @@ from synapse.api.errors import SynapseError
 from synapse.types import GroupID, RoomID, UserID, get_domain_from_id
 from twisted.internet import defer

-from six import string_types
-
 logger = logging.getLogger(__name__)


@@ -433,7 +431,7 @@ class GroupsServerHandler(object):
                        "long_description"):
            if keyname in content:
                value = content[keyname]
-                if not isinstance(value, string_types):
+                if not isinstance(value, basestring):
                    raise SynapseError(400, "%r value is not a string" % (keyname,))
                profile[keyname] = value

@@ -14,7 +14,9 @@
 # limitations under the License.

 from .register import RegistrationHandler
-from .room import RoomContextHandler
+from .room import (
+    RoomCreationHandler, RoomContextHandler,
+)
 from .message import MessageHandler
 from .federation import FederationHandler
 from .directory import DirectoryHandler
@@ -45,6 +47,7 @@ class Handlers(object):
    def __init__(self, hs):
        self.registration_handler = RegistrationHandler(hs)
        self.message_handler = MessageHandler(hs)
+        self.room_creation_handler = RoomCreationHandler(hs)
        self.federation_handler = FederationHandler(hs)
        self.directory_handler = DirectoryHandler(hs)
        self.admin_handler = AdminHandler(hs)
@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2017, 2018 New Vector Ltd
+# Copyright 2017 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -12,11 +12,9 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-from twisted.internet import defer, reactor
+from twisted.internet import defer

 from ._base import BaseHandler
-from synapse.types import UserID, create_requester
-from synapse.util.logcontext import run_in_background

 import logging

@@ -29,15 +27,6 @@ class DeactivateAccountHandler(BaseHandler):
        super(DeactivateAccountHandler, self).__init__(hs)
        self._auth_handler = hs.get_auth_handler()
        self._device_handler = hs.get_device_handler()
-        self._room_member_handler = hs.get_room_member_handler()
-        self.user_directory_handler = hs.get_user_directory_handler()
-
-        # Flag that indicates whether the process to part users from rooms is running
-        self._user_parter_running = False
-
-        # Start the user parter loop so it can resume parting users from rooms where
-        # it left off (if it has work left to do).
-        reactor.callWhenRunning(self._start_user_parting)

    @defer.inlineCallbacks
    def deactivate_account(self, user_id):
@@ -61,73 +50,3 @@ class DeactivateAccountHandler(BaseHandler):

        yield self.store.user_delete_threepids(user_id)
        yield self.store.user_set_password_hash(user_id, None)
-
-        # Add the user to a table of users pending deactivation (ie.
-        # removal from all the rooms they're a member of)
-        yield self.store.add_user_pending_deactivation(user_id)
-
-        # delete from user directory
-        yield self.user_directory_handler.handle_user_deactivated(user_id)
-
-        # Now start the process that goes through that list and
-        # parts users from rooms (if it isn't already running)
-        self._start_user_parting()
-
-    def _start_user_parting(self):
-        """
-        Start the process that goes through the table of users
-        pending deactivation, if it isn't already running.
-
-        Returns:
-            None
-        """
-        if not self._user_parter_running:
-            run_in_background(self._user_parter_loop)
-
-    @defer.inlineCallbacks
-    def _user_parter_loop(self):
-        """Loop that parts deactivated users from rooms
-
-        Returns:
-            None
-        """
-        self._user_parter_running = True
-        logger.info("Starting user parter")
-        try:
-            while True:
-                user_id = yield self.store.get_user_pending_deactivation()
-                if user_id is None:
-                    break
-                logger.info("User parter parting %r", user_id)
-                yield self._part_user(user_id)
-                yield self.store.del_user_pending_deactivation(user_id)
-                logger.info("User parter finished parting %r", user_id)
-            logger.info("User parter finished: stopping")
-        finally:
-            self._user_parter_running = False
-
-    @defer.inlineCallbacks
-    def _part_user(self, user_id):
-        """Causes the given user_id to leave all the rooms they're joined to
-
-        Returns:
-            None
-        """
-        user = UserID.from_string(user_id)
-
-        rooms_for_user = yield self.store.get_rooms_for_user(user_id)
-        for room_id in rooms_for_user:
-            logger.info("User parter parting %r from %r", user_id, room_id)
-            try:
-                yield self._room_member_handler.update_membership(
-                    create_requester(user),
-                    user,
-                    room_id,
-                    "leave",
-                    ratelimit=False,
-                )
-            except Exception:
-                logger.exception(
-                    "Failed to part user %r from room %r: ignoring and continuing",
-                    user_id, room_id,
-                )
@@ -26,8 +26,6 @@ from ._base import BaseHandler

 import logging

-from six import itervalues, iteritems
-
 logger = logging.getLogger(__name__)


@@ -320,7 +318,7 @@ class DeviceHandler(BaseHandler):
            # The user may have left the room
            # TODO: Check if they actually did or if we were just invited.
            if room_id not in room_ids:
-                for key, event_id in iteritems(current_state_ids):
+                for key, event_id in current_state_ids.iteritems():
                    etype, state_key = key
                    if etype != EventTypes.Member:
                        continue
@@ -340,7 +338,7 @@ class DeviceHandler(BaseHandler):
            # special-case for an empty prev state: include all members
            # in the changed list
            if not event_ids:
-                for key, event_id in iteritems(current_state_ids):
+                for key, event_id in current_state_ids.iteritems():
                    etype, state_key = key
                    if etype != EventTypes.Member:
                        continue
@@ -356,10 +354,10 @@ class DeviceHandler(BaseHandler):

            # Check if we've joined the room? If so we just blindly add all the users to
            # the "possibly changed" users.
-            for state_dict in itervalues(prev_state_ids):
+            for state_dict in prev_state_ids.itervalues():
                member_event = state_dict.get((EventTypes.Member, user_id), None)
                if not member_event or member_event != current_member_id:
-                    for key, event_id in iteritems(current_state_ids):
+                    for key, event_id in current_state_ids.iteritems():
                        etype, state_key = key
                        if etype != EventTypes.Member:
                            continue
@@ -369,14 +367,14 @@ class DeviceHandler(BaseHandler):
            # If there has been any change in membership, include them in the
            # possibly changed list. We'll check if they are joined below,
            # and we're not toooo worried about spuriously adding users.
-            for key, event_id in iteritems(current_state_ids):
+            for key, event_id in current_state_ids.iteritems():
                etype, state_key = key
                if etype != EventTypes.Member:
                    continue

                # check if this member has changed since any of the extremities
                # at the stream_ordering, and add them to the list if so.
-                for state_dict in itervalues(prev_state_ids):
+                for state_dict in prev_state_ids.itervalues():
                    prev_event_id = state_dict.get(key, None)
                    if not prev_event_id or prev_event_id != event_id:
                        if state_key != user_id:
@@ -19,7 +19,6 @@ import logging

 from canonicaljson import encode_canonical_json
 from twisted.internet import defer
-from six import iteritems

 from synapse.api.errors import (
    SynapseError, CodeMessageException, FederationDeniedError,
@@ -93,7 +92,7 @@ class E2eKeysHandler(object):
        remote_queries_not_in_cache = {}
        if remote_queries:
            query_list = []
-            for user_id, device_ids in iteritems(remote_queries):
+            for user_id, device_ids in remote_queries.iteritems():
                if device_ids:
                    query_list.extend((user_id, device_id) for device_id in device_ids)
                else:
@@ -104,9 +103,9 @@ class E2eKeysHandler(object):
                    query_list
                )
            )
-            for user_id, devices in iteritems(remote_results):
+            for user_id, devices in remote_results.iteritems():
                user_devices = results.setdefault(user_id, {})
-                for device_id, device in iteritems(devices):
+                for device_id, device in devices.iteritems():
                    keys = device.get("keys", None)
                    device_display_name = device.get("device_display_name", None)
                    if keys:
@@ -251,9 +250,9 @@ class E2eKeysHandler(object):
            "Claimed one-time-keys: %s",
            ",".join((
                "%s for %s:%s" % (key_id, user_id, device_id)
-                for user_id, user_keys in iteritems(json_result)
-                for device_id, device_keys in iteritems(user_keys)
-                for key_id, _ in iteritems(device_keys)
+                for user_id, user_keys in json_result.iteritems()
+                for device_id, device_keys in user_keys.iteritems()
+                for key_id, _ in device_keys.iteritems()
            )),
        )

@@ -48,7 +48,6 @@ class EventStreamHandler(BaseHandler):

        self.notifier = hs.get_notifier()
        self.state = hs.get_state_handler()
-        self._server_notices_sender = hs.get_server_notices_sender()

    @defer.inlineCallbacks
    @log_function
@@ -59,10 +58,6 @@ class EventStreamHandler(BaseHandler):

        If `only_keys` is not None, events from keys will be sent down.
        """
-
-        # send any outstanding server notices to the user.
-        yield self._server_notices_sender.on_user_syncing(auth_user_id)
-
        auth_user = UserID.from_string(auth_user_id)
        presence_handler = self.hs.get_presence_handler()

@@ -24,7 +24,6 @@ from signedjson.key import decode_verify_key_bytes
 from signedjson.sign import verify_signed_json
 import six
 from six.moves import http_client
-from six import iteritems
 from twisted.internet import defer
 from unpaddedbase64 import decode_base64

@@ -82,7 +81,6 @@ class FederationHandler(BaseHandler):
        self.pusher_pool = hs.get_pusherpool()
        self.spam_checker = hs.get_spam_checker()
        self.event_creation_handler = hs.get_event_creation_handler()
-        self._server_notices_mxid = hs.config.server_notices_mxid

        # When joining a room we need to queue any events for that room up
        self.room_queues = {}
@@ -480,18 +478,18 @@ class FederationHandler(BaseHandler):
        # to get all state ids that we're interested in.
        event_map = yield self.store.get_events([
            e_id
-            for key_to_eid in event_to_state_ids.itervalues()
-            for key, e_id in key_to_eid.iteritems()
+            for key_to_eid in event_to_state_ids.values()
+            for key, e_id in key_to_eid.items()
            if key[0] != EventTypes.Member or check_match(key[1])
        ])

        event_to_state = {
            e_id: {
                key: event_map[inner_e_id]
-                for key, inner_e_id in key_to_eid.iteritems()
+                for key, inner_e_id in key_to_eid.items()
                if inner_e_id in event_map
            }
-            for e_id, key_to_eid in event_to_state_ids.iteritems()
+            for e_id, key_to_eid in event_to_state_ids.items()
        }

        def redact_disallowed(event, state):
@@ -506,7 +504,7 @@ class FederationHandler(BaseHandler):
                    # membership states for the requesting server to determine
                    # if the server is either in the room or has been invited
                    # into the room.
-                    for ev in state.itervalues():
+                    for ev in state.values():
                        if ev.type != EventTypes.Member:
                            continue
                        try:
@@ -752,19 +750,9 @@ class FederationHandler(BaseHandler):
        curr_state = yield self.state_handler.get_current_state(room_id)

        def get_domains_from_state(state):
-            """Get joined domains from state
-
-            Args:
-                state (dict[tuple, FrozenEvent]): State map from type/state
-                    key to event.
-
-            Returns:
-                list[tuple[str, int]]: Returns a list of servers with the
-                lowest depth of their joins. Sorted by lowest depth first.
-            """
            joined_users = [
                (state_key, int(event.depth))
-                for (e_type, state_key), event in state.iteritems()
+                for (e_type, state_key), event in state.items()
                if e_type == EventTypes.Member
                and event.membership == Membership.JOIN
            ]
@@ -781,7 +769,7 @@ class FederationHandler(BaseHandler):
                except Exception:
                    pass

-            return sorted(joined_domains.iteritems(), key=lambda d: d[1])
+            return sorted(joined_domains.items(), key=lambda d: d[1])

        curr_domains = get_domains_from_state(curr_state)

@@ -798,7 +786,7 @@ class FederationHandler(BaseHandler):
                    yield self.backfill(
                        dom, room_id,
                        limit=100,
-                        extremities=extremities,
+                        extremities=[e for e in extremities.keys()]
                    )
                    # If this succeeded then we probably already have the
                    # appropriate stuff.
@@ -844,7 +832,7 @@ class FederationHandler(BaseHandler):
        tried_domains = set(likely_domains)
        tried_domains.add(self.server_name)

-        event_ids = list(extremities.iterkeys())
+        event_ids = list(extremities.keys())

        logger.debug("calling resolve_state_groups in _maybe_backfill")
        resolve = logcontext.preserve_fn(
@@ -854,34 +842,31 @@ class FederationHandler(BaseHandler):
            [resolve(room_id, [e]) for e in event_ids],
            consumeErrors=True,
        ))
-
-        # dict[str, dict[tuple, str]], a map from event_id to state map of
-        # event_ids.
        states = dict(zip(event_ids, [s.state for s in states]))

        state_map = yield self.store.get_events(
-            [e_id for ids in states.itervalues() for e_id in ids.itervalues()],
+            [e_id for ids in states.values() for e_id in ids],
            get_prev_content=False
        )
        states = {
            key: {
                k: state_map[e_id]
-                for k, e_id in state_dict.iteritems()
+                for k, e_id in state_dict.items()
                if e_id in state_map
-            } for key, state_dict in states.iteritems()
+            } for key, state_dict in states.items()
        }

        for e_id, _ in sorted_extremeties_tuple:
            likely_domains = get_domains_from_state(states[e_id])

            success = yield try_backfill([
-                dom for dom, _ in likely_domains
+                dom for dom in likely_domains
                if dom not in tried_domains
            ])
            if success:
                defer.returnValue(True)

-            tried_domains.update(dom for dom, _ in likely_domains)
+            tried_domains.update(likely_domains)

        defer.returnValue(False)

@@ -1195,13 +1180,6 @@ class FederationHandler(BaseHandler):
        if not self.is_mine_id(event.state_key):
            raise SynapseError(400, "The invite event must be for this server")

-        # block any attempts to invite the server notices mxid
-        if event.state_key == self._server_notices_mxid:
-            raise SynapseError(
-                http_client.FORBIDDEN,
-                "Cannot invite this user",
-            )
-
        event.internal_metadata.outlier = True
        event.internal_metadata.invite_from_remote = True

@@ -1389,7 +1367,7 @@ class FederationHandler(BaseHandler):
        )

        if state_groups:
-            _, state = list(iteritems(state_groups)).pop()
+            _, state = state_groups.items().pop()
            results = {
                (e.type, e.state_key): e for e in state
            }
@@ -2035,7 +2013,7 @@ class FederationHandler(BaseHandler):
                this will not be included in the current_state in the context.
        """
        state_updates = {
-            k: a.event_id for k, a in iteritems(auth_events)
+            k: a.event_id for k, a in auth_events.iteritems()
            if k != event_key
        }
        context.current_state_ids = dict(context.current_state_ids)
@@ -2045,7 +2023,7 @@ class FederationHandler(BaseHandler):
            context.delta_ids.update(state_updates)
        context.prev_state_ids = dict(context.prev_state_ids)
        context.prev_state_ids.update({
-            k: a.event_id for k, a in iteritems(auth_events)
+            k: a.event_id for k, a in auth_events.iteritems()
        })
        context.state_group = yield self.store.store_state_group(
            event.event_id,
@@ -2097,7 +2075,7 @@ class FederationHandler(BaseHandler):

        def get_next(it, opt=None):
            try:
-                return next(it)
+                return it.next()
            except Exception:
                return opt

@@ -15,7 +15,6 @@
 # limitations under the License.

 from twisted.internet import defer
-from six import iteritems

 from synapse.api.errors import SynapseError
 from synapse.types import get_domain_from_id
@@ -450,7 +449,7 @@ class GroupsLocalHandler(object):

        results = {}
        failed_results = []
-        for destination, dest_user_ids in iteritems(destinations):
+        for destination, dest_user_ids in destinations.iteritems():
            try:
                r = yield self.transport_client.bulk_get_publicised_groups(
                    destination, list(dest_user_ids),
@@ -181,8 +181,8 @@ class InitialSyncHandler(BaseHandler):
                    self.store, user_id, messages
                )

-                start_token = now_token.copy_and_replace("room_key", token)
-                end_token = now_token.copy_and_replace("room_key", room_end_token)
+                start_token = now_token.copy_and_replace("room_key", token[0])
+                end_token = now_token.copy_and_replace("room_key", token[1])
                time_now = self.clock.time_msec()

                d["messages"] = {
@@ -325,8 +325,8 @@ class InitialSyncHandler(BaseHandler):
            self.store, user_id, messages, is_peeking=is_peeking
        )

-        start_token = StreamToken.START.copy_and_replace("room_key", token)
-        end_token = StreamToken.START.copy_and_replace("room_key", stream_token)
+        start_token = StreamToken.START.copy_and_replace("room_key", token[0])
+        end_token = StreamToken.START.copy_and_replace("room_key", token[1])

        time_now = self.clock.time_msec()

@@ -408,8 +408,8 @@ class InitialSyncHandler(BaseHandler):
            self.store, user_id, messages, is_peeking=is_peeking,
        )

-        start_token = now_token.copy_and_replace("room_key", token)
-        end_token = now_token
+        start_token = now_token.copy_and_replace("room_key", token[0])
+        end_token = now_token.copy_and_replace("room_key", token[1])

        time_now = self.clock.time_msec()

@@ -19,17 +19,11 @@ import sys

 from canonicaljson import encode_canonical_json
 import six
-from six import string_types, itervalues, iteritems
 from twisted.internet import defer, reactor
-from twisted.internet.defer import succeed
 from twisted.python.failure import Failure

 from synapse.api.constants import EventTypes, Membership, MAX_DEPTH
-from synapse.api.errors import (
-    AuthError, Codes, SynapseError,
-    ConsentNotGivenError,
-)
-from synapse.api.urls import ConsentURIBuilder
+from synapse.api.errors import AuthError, Codes, SynapseError
 from synapse.crypto.event_signing import add_hashes_and_signatures
 from synapse.events.utils import serialize_event
 from synapse.events.validator import EventValidator
@@ -92,14 +86,14 @@ class MessageHandler(BaseHandler):
        # map from purge id to PurgeStatus
        self._purges_by_id = {}

-    def start_purge_history(self, room_id, token,
+    def start_purge_history(self, room_id, topological_ordering,
                            delete_local_events=False):
        """Start off a history purge on a room.

        Args:
            room_id (str): The room to purge from

-            token (str): topological token to delete events before
+            topological_ordering (int): minimum topo ordering to preserve
            delete_local_events (bool): True to delete local events as well as
                remote ones

@@ -121,19 +115,19 @@ class MessageHandler(BaseHandler):
        self._purges_by_id[purge_id] = PurgeStatus()
        run_in_background(
            self._purge_history,
-            purge_id, room_id, token, delete_local_events,
+            purge_id, room_id, topological_ordering, delete_local_events,
        )
        return purge_id

    @defer.inlineCallbacks
-    def _purge_history(self, purge_id, room_id, token,
+    def _purge_history(self, purge_id, room_id, topological_ordering,
                       delete_local_events):
        """Carry out a history purge on a room.

        Args:
            purge_id (str): The id for this purge
            room_id (str): The room to purge from
-            token (str): topological token to delete events before
+            topological_ordering (int): minimum topo ordering to preserve
            delete_local_events (bool): True to delete local events as well as
                remote ones

@@ -144,7 +138,7 @@ class MessageHandler(BaseHandler):
        try:
            with (yield self.pagination_lock.write(room_id)):
                yield self.store.purge_history(
-                    room_id, token, delete_local_events,
+                    room_id, topological_ordering, delete_local_events,
                )
            logger.info("[purge] complete")
            self._purges_by_id[purge_id].status = PurgeStatus.STATUS_COMPLETE
@@ -211,19 +205,31 @@ class MessageHandler(BaseHandler):
            )

            if source_config.direction == 'b':
+                # if we're going backwards, we might need to backfill. This
+                # requires that we have a topo token.
+                if room_token.topological:
+                    max_topo = room_token.topological
+                else:
+                    max_topo = yield self.store.get_max_topological_token(
+                        room_id, room_token.stream
+                    )
+
                if membership == Membership.LEAVE:
                    # If they have left the room then clamp the token to be before
                    # they left the room, to save the effort of loading from the
                    # database.
-
                    leave_token = yield self.store.get_topological_token_for_event(
-                        member_event_id,
-                    )
-                    source_config.from_key = yield self.store.clamp_token_before(
-                        room_id, source_config.from_key, leave_token,
+                        member_event_id
                    )
+                    leave_token = RoomStreamToken.parse(leave_token)
+                    if leave_token.topological < max_topo:
+                        source_config.from_key = str(leave_token)

-            events, next_key, extremities = yield self.store.paginate_room_events(
+                yield self.hs.get_handlers().federation_handler.maybe_backfill(
+                    room_id, max_topo
+                )
+
+            events, next_key = yield self.store.paginate_room_events(
                room_id=room_id,
                from_key=source_config.from_key,
                to_key=source_config.to_key,
@@ -391,7 +397,7 @@ class MessageHandler(BaseHandler):
                "avatar_url": profile.avatar_url,
                "display_name": profile.display_name,
            }
-            for user_id, profile in iteritems(users_with_profile)
+            for user_id, profile in users_with_profile.iteritems()
        })


@@ -425,9 +431,6 @@ class EventCreationHandler(object):

        self.spam_checker = hs.get_spam_checker()

-        if self.config.block_events_without_consent_error is not None:
-            self._consent_uri_builder = ConsentURIBuilder(self.config)
-
    @defer.inlineCallbacks
    def create_event(self, requester, event_dict, token_id=None, txn_id=None,
                     prev_events_and_hashes=None):
@@ -479,10 +482,6 @@ class EventCreationHandler(object):
                        target, e
                    )

-        is_exempt = yield self._is_exempt_from_privacy_policy(builder)
-        if not is_exempt:
-            yield self.assert_accepted_privacy_policy(requester)
-
        if token_id is not None:
            builder.internal_metadata.token_id = token_id

@@ -497,83 +496,6 @@ class EventCreationHandler(object):

        defer.returnValue((event, context))

-    def _is_exempt_from_privacy_policy(self, builder):
-        """"Determine if an event to be sent is exempt from having to consent
-        to the privacy policy
-
-        Args:
-            builder (synapse.events.builder.EventBuilder): event being created
-
-        Returns:
-            Deferred[bool]: true if the event can be sent without the user
-                consenting
-        """
-        # the only thing the user can do is join the server notices room.
-        if builder.type == EventTypes.Member:
-            membership = builder.content.get("membership", None)
-            if membership == Membership.JOIN:
-                return self._is_server_notices_room(builder.room_id)
-        return succeed(False)
-
-    @defer.inlineCallbacks
-    def _is_server_notices_room(self, room_id):
-        if self.config.server_notices_mxid is None:
-            defer.returnValue(False)
-        user_ids = yield self.store.get_users_in_room(room_id)
-        defer.returnValue(self.config.server_notices_mxid in user_ids)
-
-    @defer.inlineCallbacks
-    def assert_accepted_privacy_policy(self, requester):
-        """Check if a user has accepted the privacy policy
-
-        Called when the given user is about to do something that requires
-        privacy consent. We see if the user is exempt and otherwise check that
-        they have given consent. If they have not, a ConsentNotGiven error is
-        raised.
-
-        Args:
-            requester (synapse.types.Requester):
-                The user making the request
-
-        Returns:
-            Deferred[None]: returns normally if the user has consented or is
-                exempt
-
-        Raises:
-            ConsentNotGivenError: if the user has not given consent yet
-        """
-        if self.config.block_events_without_consent_error is None:
-            return
-
-        # exempt AS users from needing consent
-        if requester.app_service is not None:
-            return
-
-        user_id = requester.user.to_string()
-
-        # exempt the system notices user
-        if (
-            self.config.server_notices_mxid is not None and
-            user_id == self.config.server_notices_mxid
-        ):
-            return
-
-        u = yield self.store.get_user_by_id(user_id)
-        assert u is not None
-        if u["consent_version"] == self.config.user_consent_version:
-            return
-
-        consent_uri = self._consent_uri_builder.build_user_consent_uri(
-            requester.user.localpart,
-        )
-        msg = self.config.block_events_without_consent_error % {
-            'consent_uri': consent_uri,
-        }
-        raise ConsentNotGivenError(
-            msg=msg,
-            consent_uri=consent_uri,
-        )
-
    @defer.inlineCallbacks
    def send_nonmember_event(self, requester, event, context, ratelimit=True):
        """
@@ -656,7 +578,7 @@ class EventCreationHandler(object):

            spam_error = self.spam_checker.check_event_for_spam(event)
            if spam_error:
-                if not isinstance(spam_error, string_types):
+                if not isinstance(spam_error, basestring):
                    spam_error = "Spam is not permitted here"
                raise SynapseError(
                    403, spam_error, Codes.FORBIDDEN
@@ -870,7 +792,7 @@ class EventCreationHandler(object):

                state_to_include_ids = [
                    e_id
-                    for k, e_id in iteritems(context.current_state_ids)
+                    for k, e_id in context.current_state_ids.iteritems()
                    if k[0] in self.hs.config.room_invite_state_types
                    or k == (EventTypes.Member, event.sender)
                ]
@@ -884,7 +806,7 @@ class EventCreationHandler(object):
                        "content": e.content,
                        "sender": e.sender,
                    }
-                    for e in itervalues(state_to_include)
+                    for e in state_to_include.itervalues()
                ]

                invitee = UserID.from_string(event.state_key)
@@ -25,8 +25,6 @@ The methods that define policy are:
 from twisted.internet import defer, reactor
 from contextlib import contextmanager

-from six import itervalues, iteritems
-
 from synapse.api.errors import SynapseError
 from synapse.api.constants import PresenceState
 from synapse.storage.presence import UserPresenceState
@@ -42,6 +40,7 @@ import synapse.metrics

 import logging

+
 logger = logging.getLogger(__name__)

 metrics = synapse.metrics.get_metrics_for(__name__)
@@ -88,11 +87,6 @@ assert LAST_ACTIVE_GRANULARITY < IDLE_TIMER
 class PresenceHandler(object):

    def __init__(self, hs):
-        """
-
-        Args:
-            hs (synapse.server.HomeServer):
-        """
        self.is_mine = hs.is_mine
        self.is_mine_id = hs.is_mine_id
        self.clock = hs.get_clock()
@@ -100,6 +94,7 @@ class PresenceHandler(object):
        self.wheel_timer = WheelTimer()
        self.notifier = hs.get_notifier()
        self.federation = hs.get_federation_sender()
+
        self.state = hs.get_state_handler()

        federation_registry = hs.get_federation_registry()
@@ -468,6 +463,61 @@ class PresenceHandler(object):
            syncing_user_ids.update(user_ids)
        return syncing_user_ids

+    @defer.inlineCallbacks
+    def update_external_syncs(self, process_id, syncing_user_ids):
+        """Update the syncing users for an external process
+
+        Args:
+            process_id(str): An identifier for the process the users are
+                syncing against. This allows synapse to process updates
+                as user start and stop syncing against a given process.
+            syncing_user_ids(set(str)): The set of user_ids that are
+                currently syncing on that server.
+        """
+
+        # Grab the previous list of user_ids that were syncing on that process
+        prev_syncing_user_ids = (
+            self.external_process_to_current_syncs.get(process_id, set())
+        )
+        # Grab the current presence state for both the users that are syncing
+        # now and the users that were syncing before this update.
+        prev_states = yield self.current_state_for_users(
+            syncing_user_ids | prev_syncing_user_ids
+        )
+        updates = []
+        time_now_ms = self.clock.time_msec()
+
+        # For each new user that is syncing check if we need to mark them as
+        # being online.
+        for new_user_id in syncing_user_ids - prev_syncing_user_ids:
+            prev_state = prev_states[new_user_id]
+            if prev_state.state == PresenceState.OFFLINE:
+                updates.append(prev_state.copy_and_replace(
+                    state=PresenceState.ONLINE,
+                    last_active_ts=time_now_ms,
+                    last_user_sync_ts=time_now_ms,
+                ))
+            else:
+                updates.append(prev_state.copy_and_replace(
+                    last_user_sync_ts=time_now_ms,
+                ))
+
+        # For each user that is still syncing or stopped syncing update the
+        # last sync time so that we will correctly apply the grace period when
+        # they stop syncing.
+        for old_user_id in prev_syncing_user_ids:
+            prev_state = prev_states[old_user_id]
+            updates.append(prev_state.copy_and_replace(
+                last_user_sync_ts=time_now_ms,
+            ))
+
+        yield self._update_states(updates)
+
+        # Update the last updated time for the process. We expire the entries
+        # if we don't receive an update in the given timeframe.
+        self.external_process_last_updated_ms[process_id] = self.clock.time_msec()
+        self.external_process_to_current_syncs[process_id] = syncing_user_ids
+
    @defer.inlineCallbacks
    def update_external_syncs_row(self, process_id, user_id, is_syncing, sync_time_msec):
        """Update the syncing users for an external process as a delta.
@@ -531,7 +581,7 @@ class PresenceHandler(object):
                prev_state.copy_and_replace(
                    last_user_sync_ts=time_now_ms,
                )
-                for prev_state in itervalues(prev_states)
+                for prev_state in prev_states.itervalues()
            ])
            self.external_process_last_updated_ms.pop(process_id, None)

@@ -554,14 +604,14 @@ class PresenceHandler(object):
            for user_id in user_ids
        }

-        missing = [user_id for user_id, state in iteritems(states) if not state]
+        missing = [user_id for user_id, state in states.iteritems() if not state]
        if missing:
            # There are things not in our in memory cache. Lets pull them out of
            # the database.
            res = yield self.store.get_presence_for_users(missing)
            states.update(res)

-            missing = [user_id for user_id, state in iteritems(states) if not state]
+            missing = [user_id for user_id, state in states.iteritems() if not state]
            if missing:
                new = {
                    user_id: UserPresenceState.default(user_id)
@@ -1049,7 +1099,7 @@ class PresenceEventSource(object):
            defer.returnValue((updates.values(), max_token))
        else:
            defer.returnValue(([
-                s for s in itervalues(updates)
+                s for s in updates.itervalues()
                if s.state != PresenceState.OFFLINE
            ], max_token))

@@ -1306,11 +1356,11 @@ def get_interested_remotes(store, states, state_handler):
    # hosts in those rooms.
    room_ids_to_states, users_to_states = yield get_interested_parties(store, states)

-    for room_id, states in iteritems(room_ids_to_states):
+    for room_id, states in room_ids_to_states.iteritems():
        hosts = yield state_handler.get_current_hosts_in_room(room_id)
        hosts_and_states.append((hosts, states))

-    for user_id, states in iteritems(users_to_states):
+    for user_id, states in users_to_states.iteritems():
        host = get_domain_from_id(user_id)
        hosts_and_states.append(([host], states))

@@ -34,11 +34,6 @@ logger = logging.getLogger(__name__)
 class RegistrationHandler(BaseHandler):

    def __init__(self, hs):
-        """
-
-        Args:
-            hs (synapse.server.HomeServer):
-        """
        super(RegistrationHandler, self).__init__(hs)

        self.auth = hs.get_auth()
@@ -54,7 +49,6 @@ class RegistrationHandler(BaseHandler):
        self._generate_user_id_linearizer = Linearizer(
            name="_generate_user_id_linearizer",
        )
-        self._server_notices_mxid = hs.config.server_notices_mxid

    @defer.inlineCallbacks
    def check_username(self, localpart, guest_access_token=None,
@@ -344,14 +338,6 @@ class RegistrationHandler(BaseHandler):
            yield identity_handler.bind_threepid(c, user_id)

    def check_user_id_not_appservice_exclusive(self, user_id, allowed_appservice=None):
-        # don't allow people to register the server notices mxid
-        if self._server_notices_mxid is not None:
-            if user_id == self._server_notices_mxid:
-                raise SynapseError(
-                    400, "This user ID is reserved.",
-                    errcode=Codes.EXCLUSIVE
-                )
-
        # valid user IDs must not clash with any user ID namespaces claimed by
        # application services.
        services = self.store.get_app_services()
@@ -68,27 +68,14 @@ class RoomCreationHandler(BaseHandler):
        self.event_creation_handler = hs.get_event_creation_handler()

    @defer.inlineCallbacks
-    def create_room(self, requester, config, ratelimit=True,
-                    creator_join_profile=None):
+    def create_room(self, requester, config, ratelimit=True):
        """ Creates a new room.

        Args:
-            requester (synapse.types.Requester):
-                The user who requested the room creation.
+            requester (Requester): The user who requested the room creation.
            config (dict) : A dict of configuration options.
-            ratelimit (bool): set to False to disable the rate limiter
-
-            creator_join_profile (dict|None):
-                Set to override the displayname and avatar for the creating
-                user in this room. If unset, displayname and avatar will be
-                derived from the user's profile. If set, should contain the
-                values to go in the body of the 'join' event (typically
-                `avatar_url` and/or `displayname`.
-
        Returns:
-            Deferred[dict]:
-                a dict containing the keys `room_id` and, if an alias was
-                requested, `room_alias`.
+            The new room ID.
        Raises:
            SynapseError if the room ID couldn't be stored, or something went
            horribly wrong.
@@ -126,10 +113,6 @@ class RoomCreationHandler(BaseHandler):
            except Exception:
                raise SynapseError(400, "Invalid user_id: %s" % (i,))

-        yield self.event_creation_handler.assert_accepted_privacy_policy(
-            requester,
-        )
-
        invite_3pid_list = config.get("invite_3pid", [])

        visibility = config.get("visibility", None)
@@ -193,8 +176,7 @@ class RoomCreationHandler(BaseHandler):
            initial_state=initial_state,
            creation_content=creation_content,
            room_alias=room_alias,
-            power_level_content_override=config.get("power_level_content_override", {}),
-            creator_join_profile=creator_join_profile,
+            power_level_content_override=config.get("power_level_content_override", {})
        )

        if "name" in config:
@@ -274,7 +256,6 @@ class RoomCreationHandler(BaseHandler):
            creation_content,
            room_alias,
            power_level_content_override,
-            creator_join_profile,
    ):
        def create(etype, content, **kwargs):
            e = {
@@ -318,7 +299,6 @@ class RoomCreationHandler(BaseHandler):
            room_id,
            "join",
            ratelimit=False,
-            content=creator_join_profile,
        )

        # We treat the power levels override specially as this needs to be one
@@ -514,8 +494,7 @@ class RoomEventSource(object):
            events = list(room_events)
            events.extend(e for evs, _ in room_to_events.values() for e in evs)

-            # Order by the stream ordering of the events.
-            events.sort(key=lambda e: e.internal_metadata.stream_ordering)
+            events.sort(key=lambda e: e.internal_metadata.order)

            if limit:
                events[:] = events[:limit]
@@ -535,7 +514,7 @@ class RoomEventSource(object):

    @defer.inlineCallbacks
    def get_pagination_rows(self, user, config, key):
-        events, next_key, _ = yield self.store.paginate_room_events(
+        events, next_key = yield self.store.paginate_room_events(
            room_id=key,
            from_key=config.from_key,
            to_key=config.to_key,
@@ -17,14 +17,11 @@
 import abc
 import logging

-from six.moves import http_client
-
 from signedjson.key import decode_verify_key_bytes
 from signedjson.sign import verify_signed_json
 from twisted.internet import defer
 from unpaddedbase64 import decode_base64

-import synapse.server
 import synapse.types
 from synapse.api.constants import (
    EventTypes, Membership,
@@ -49,11 +46,6 @@ class RoomMemberHandler(object):
    __metaclass__ = abc.ABCMeta

    def __init__(self, hs):
-        """
-
-        Args:
-            hs (synapse.server.HomeServer):
-        """
        self.hs = hs
        self.store = hs.get_datastore()
        self.auth = hs.get_auth()
@@ -71,7 +63,6 @@ class RoomMemberHandler(object):

        self.clock = hs.get_clock()
        self.spam_checker = hs.get_spam_checker()
-        self._server_notices_mxid = self.config.server_notices_mxid

    @abc.abstractmethod
    def _remote_join(self, requester, remote_room_hosts, room_id, user, content):
@@ -298,37 +289,12 @@ class RoomMemberHandler(object):
            is_blocked = yield self.store.is_room_blocked(room_id)
            if is_blocked:
                raise SynapseError(403, "This room has been blocked on this server")
-        else:
-            # we don't allow people to reject invites to, or leave, the
-            # server notice room.
-            is_blocked = yield self._is_server_notice_room(room_id)
-            if is_blocked:
-                raise SynapseError(
-                    http_client.FORBIDDEN,
-                    "You cannot leave this room",
-                    errcode=Codes.CANNOT_LEAVE_SERVER_NOTICE_ROOM,
-                )
-
-        if effective_membership_state == Membership.INVITE:
-            # block any attempts to invite the server notices mxid
-            if target.to_string() == self._server_notices_mxid:
-                raise SynapseError(
-                    http_client.FORBIDDEN,
-                    "Cannot invite this user",
-                )

+        if effective_membership_state == "invite":
            block_invite = False
-
-            if (self._server_notices_mxid is not None and
-                    requester.user.to_string() == self._server_notices_mxid):
-                # allow the server notices mxid to send invites
-                is_requester_admin = True
-
-            else:
-                is_requester_admin = yield self.auth.is_server_admin(
-                    requester.user,
-                )
-
+            is_requester_admin = yield self.auth.is_server_admin(
+                requester.user,
+            )
            if not is_requester_admin:
                if self.config.block_non_admin_invites:
                    logger.info(
@@ -878,13 +844,6 @@ class RoomMemberHandler(object):

        defer.returnValue(False)

-    @defer.inlineCallbacks
-    def _is_server_notice_room(self, room_id):
-        if self._server_notices_mxid is None:
-            defer.returnValue(False)
-        user_ids = yield self.store.get_users_in_room(room_id)
-        defer.returnValue(self._server_notices_mxid in user_ids)
-

 class RoomMemberMasterHandler(RoomMemberHandler):
    def __init__(self, hs):
@@ -28,8 +28,6 @@ import collections
 import logging
 import itertools

-from six import itervalues, iteritems
-
 logger = logging.getLogger(__name__)


@@ -277,7 +275,7 @@ class SyncHandler(object):
                # result returned by the event source is poor form (it might cache
                # the object)
                room_id = event["room_id"]
-                event_copy = {k: v for (k, v) in iteritems(event)
+                event_copy = {k: v for (k, v) in event.iteritems()
                              if k != "room_id"}
                ephemeral_by_room.setdefault(room_id, []).append(event_copy)

@@ -296,7 +294,7 @@ class SyncHandler(object):
            for event in receipts:
                room_id = event["room_id"]
                # exclude room id, as above
-                event_copy = {k: v for (k, v) in iteritems(event)
+                event_copy = {k: v for (k, v) in event.iteritems()
                              if k != "room_id"}
                ephemeral_by_room.setdefault(room_id, []).append(event_copy)

@@ -327,7 +325,7 @@ class SyncHandler(object):
                current_state_ids = frozenset()
                if any(e.is_state() for e in recents):
                    current_state_ids = yield self.state.get_current_state_ids(room_id)
-                    current_state_ids = frozenset(itervalues(current_state_ids))
+                    current_state_ids = frozenset(current_state_ids.itervalues())

                recents = yield filter_events_for_client(
                    self.store,
@@ -356,24 +354,12 @@ class SyncHandler(object):
                since_key = since_token.room_key

            while limited and len(recents) < timeline_limit and max_repeat:
-                # If we have a since_key then we are trying to get any events
-                # that have happened since `since_key` up to `end_key`, so we
-                # can just use `get_room_events_stream_for_room`.
-                # Otherwise, we want to return the last N events in the room
-                # in toplogical ordering.
-                if since_key:
-                    events, end_key = yield self.store.get_room_events_stream_for_room(
-                        room_id,
-                        limit=load_limit + 1,
-                        from_key=since_key,
-                        to_key=end_key,
-                    )
-                else:
-                    events, end_key = yield self.store.get_recent_events_for_room(
-                        room_id,
-                        limit=load_limit + 1,
-                        end_token=end_key,
-                    )
+                events, end_key = yield self.store.get_room_events_stream_for_room(
+                    room_id,
+                    limit=load_limit + 1,
+                    from_key=since_key,
+                    to_key=end_key,
+                )
                loaded_recents = sync_config.filter_collection.filter_room_timeline(
                    events
                )
@@ -384,7 +370,7 @@ class SyncHandler(object):
                current_state_ids = frozenset()
                if any(e.is_state() for e in loaded_recents):
                    current_state_ids = yield self.state.get_current_state_ids(room_id)
-                    current_state_ids = frozenset(itervalues(current_state_ids))
+                    current_state_ids = frozenset(current_state_ids.itervalues())

                loaded_recents = yield filter_events_for_client(
                    self.store,
@@ -443,7 +429,7 @@ class SyncHandler(object):
        Returns:
            A Deferred map from ((type, state_key)->Event)
        """
-        last_events, _ = yield self.store.get_recent_events_for_room(
+        last_events, token = yield self.store.get_recent_events_for_room(
            room_id, end_token=stream_position.room_key, limit=1,
        )

@@ -986,7 +972,7 @@ class SyncHandler(object):
        if since_token:
            for joined_sync in sync_result_builder.joined:
                it = itertools.chain(
-                    joined_sync.timeline.events, itervalues(joined_sync.state)
+                    joined_sync.timeline.events, joined_sync.state.itervalues()
                )
                for event in it:
                    if event.type == EventTypes.Member:
@@ -1064,7 +1050,7 @@ class SyncHandler(object):
        newly_left_rooms = []
        room_entries = []
        invited = []
-        for room_id, events in iteritems(mem_change_events_by_room_id):
+        for room_id, events in mem_change_events_by_room_id.iteritems():
            non_joins = [e for e in events if e.membership != Membership.JOIN]
            has_join = len(non_joins) != len(events)

@@ -22,7 +22,6 @@ from synapse.util.metrics import Measure
 from synapse.util.async import sleep
 from synapse.types import get_localpart_from_id

-from six import iteritems

 logger = logging.getLogger(__name__)

@@ -123,13 +122,6 @@ class UserDirectoryHandler(object):
            user_id, profile.display_name, profile.avatar_url, None,
        )

-    @defer.inlineCallbacks
-    def handle_user_deactivated(self, user_id):
-        """Called when a user ID is deactivated
-        """
-        yield self.store.remove_from_user_dir(user_id)
-        yield self.store.remove_from_user_in_public_room(user_id)
-
    @defer.inlineCallbacks
    def _unsafe_process(self):
        # If self.pos is None then means we haven't fetched it from DB
@@ -411,7 +403,7 @@ class UserDirectoryHandler(object):

        if change:
            users_with_profile = yield self.state.get_current_user_in_room(room_id)
-            for user_id, profile in iteritems(users_with_profile):
+            for user_id, profile in users_with_profile.iteritems():
                yield self._handle_new_user(room_id, user_id, profile)
        else:
            users = yield self.store.get_users_in_public_due_to_room(room_id)
@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-from synapse.http.server import wrap_json_request_handler
+from synapse.http.server import wrap_request_handler
 from twisted.web.resource import Resource
 from twisted.web.server import NOT_DONE_YET

@@ -42,13 +42,14 @@ class AdditionalResource(Resource):
        Resource.__init__(self)
        self._handler = handler

-        # required by the request_handler wrapper
+        # these are required by the request_handler wrapper
+        self.version_string = hs.version_string
        self.clock = hs.get_clock()

    def render(self, request):
        self._async_render(request)
        return NOT_DONE_YET

-    @wrap_json_request_handler
+    @wrap_request_handler
    def _async_render(self, request):
        return self._handler(request)
@@ -42,8 +42,6 @@ import random
 import sys
 import urllib
 from six.moves.urllib import parse as urlparse
-from six import string_types
-

 logger = logging.getLogger(__name__)
 outbound_logger = logging.getLogger("synapse.http.outbound")
@@ -555,7 +553,7 @@ class MatrixFederationHttpClient(object):

        encoded_args = {}
        for k, vs in args.items():
-            if isinstance(vs, string_types):
+            if isinstance(vs, basestring):
                vs = [vs]
            encoded_args[k] = [v.encode("UTF-8") for v in vs]

@@ -670,7 +668,7 @@ def check_content_type_is_json(headers):
        RuntimeError if the

    """
-    c_type = headers.getRawHeaders(b"Content-Type")
+    c_type = headers.getRawHeaders("Content-Type")
    if c_type is None:
        raise RuntimeError(
            "No Content-Type header"
@@ -687,7 +685,7 @@ def check_content_type_is_json(headers):
 def encode_query_args(args):
    encoded_args = {}
    for k, vs in args.items():
-        if isinstance(vs, string_types):
+        if isinstance(vs, basestring):
            vs = [vs]
        encoded_args[k] = [v.encode("UTF-8") for v in vs]

@@ -1,307 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2018 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import logging
-
-import synapse.metrics
-from synapse.util.logcontext import LoggingContext
-
-logger = logging.getLogger(__name__)
-
-metrics = synapse.metrics.get_metrics_for("synapse.http.server")
-
-# total number of responses served, split by method/servlet/tag
-response_count = metrics.register_counter(
-    "response_count",
-    labels=["method", "servlet", "tag"],
-    alternative_names=(
-        # the following are all deprecated aliases for the same metric
-        metrics.name_prefix + x for x in (
-            "_requests",
-            "_response_time:count",
-            "_response_ru_utime:count",
-            "_response_ru_stime:count",
-            "_response_db_txn_count:count",
-            "_response_db_txn_duration:count",
-        )
-    )
-)
-
-requests_counter = metrics.register_counter(
-    "requests_received",
-    labels=["method", "servlet", ],
-)
-
-outgoing_responses_counter = metrics.register_counter(
-    "responses",
-    labels=["method", "code"],
-)
-
-response_timer = metrics.register_counter(
-    "response_time_seconds",
-    labels=["method", "servlet", "tag"],
-    alternative_names=(
-        metrics.name_prefix + "_response_time:total",
-    ),
-)
-
-response_ru_utime = metrics.register_counter(
-    "response_ru_utime_seconds", labels=["method", "servlet", "tag"],
-    alternative_names=(
-        metrics.name_prefix + "_response_ru_utime:total",
-    ),
-)
-
-response_ru_stime = metrics.register_counter(
-    "response_ru_stime_seconds", labels=["method", "servlet", "tag"],
-    alternative_names=(
-        metrics.name_prefix + "_response_ru_stime:total",
-    ),
-)
-
-response_db_txn_count = metrics.register_counter(
-    "response_db_txn_count", labels=["method", "servlet", "tag"],
-    alternative_names=(
-        metrics.name_prefix + "_response_db_txn_count:total",
-    ),
-)
-
-# seconds spent waiting for db txns, excluding scheduling time, when processing
-# this request
-response_db_txn_duration = metrics.register_counter(
-    "response_db_txn_duration_seconds", labels=["method", "servlet", "tag"],
-    alternative_names=(
-        metrics.name_prefix + "_response_db_txn_duration:total",
-    ),
-)
-
-# seconds spent waiting for a db connection, when processing this request
-response_db_sched_duration = metrics.register_counter(
-    "response_db_sched_duration_seconds", labels=["method", "servlet", "tag"]
-)
-
-# size in bytes of the response written
-response_size = metrics.register_counter(
-    "response_size", labels=["method", "servlet", "tag"]
-)
-
-# In flight metrics are incremented while the requests are in flight, rather
-# than when the response was written.
-
-in_flight_requests_ru_utime = metrics.register_counter(
-    "in_flight_requests_ru_utime_seconds", labels=["method", "servlet"],
-)
-
-in_flight_requests_ru_stime = metrics.register_counter(
-    "in_flight_requests_ru_stime_seconds", labels=["method", "servlet"],
-)
-
-in_flight_requests_db_txn_count = metrics.register_counter(
-    "in_flight_requests_db_txn_count", labels=["method", "servlet"],
-)
-
-# seconds spent waiting for db txns, excluding scheduling time, when processing
-# this request
-in_flight_requests_db_txn_duration = metrics.register_counter(
-    "in_flight_requests_db_txn_duration_seconds", labels=["method", "servlet"],
-)
-
-# seconds spent waiting for a db connection, when processing this request
-in_flight_requests_db_sched_duration = metrics.register_counter(
-    "in_flight_requests_db_sched_duration_seconds", labels=["method", "servlet"]
-)
-
-
-# The set of all in flight requests, set[RequestMetrics]
-_in_flight_requests = set()
-
-
-def _collect_in_flight():
-    """Called just before metrics are collected, so we use it to update all
-    the in flight request metrics
-    """
-
-    for rm in _in_flight_requests:
-        rm.update_metrics()
-
-
-metrics.register_collector(_collect_in_flight)
-
-
-def _get_in_flight_counts():
-    """Returns a count of all in flight requests by (method, server_name)
-
-    Returns:
-        dict[tuple[str, str], int]
-    """
-
-    # Map from (method, name) -> int, the number of in flight requests of that
-    # type
-    counts = {}
-    for rm in _in_flight_requests:
-        key = (rm.method, rm.name,)
-        counts[key] = counts.get(key, 0) + 1
-
-    return counts
-
-
-metrics.register_callback(
-    "in_flight_requests_count",
-    _get_in_flight_counts,
-    labels=["method", "servlet"]
-)
-
-
-class RequestMetrics(object):
-    def start(self, time_msec, name, method):
-        self.start = time_msec
-        self.start_context = LoggingContext.current_context()
-        self.name = name
-        self.method = method
-
-        self._request_stats = _RequestStats.from_context(self.start_context)
-
-        _in_flight_requests.add(self)
-
-    def stop(self, time_msec, request):
-        _in_flight_requests.discard(self)
-
-        context = LoggingContext.current_context()
-
-        tag = ""
-        if context:
-            tag = context.tag
-
-            if context != self.start_context:
-                logger.warn(
-                    "Context have unexpectedly changed %r, %r",
-                    context, self.start_context
-                )
-                return
-
-        outgoing_responses_counter.inc(request.method, str(request.code))
-
-        response_count.inc(request.method, self.name, tag)
-
-        response_timer.inc_by(
-            time_msec - self.start, request.method,
-            self.name, tag
-        )
-
-        ru_utime, ru_stime = context.get_resource_usage()
-
-        response_ru_utime.inc_by(
-            ru_utime, request.method, self.name, tag
-        )
-        response_ru_stime.inc_by(
-            ru_stime, request.method, self.name, tag
-        )
-        response_db_txn_count.inc_by(
-            context.db_txn_count, request.method, self.name, tag
-        )
-        response_db_txn_duration.inc_by(
-            context.db_txn_duration_ms / 1000., request.method, self.name, tag
-        )
-        response_db_sched_duration.inc_by(
-            context.db_sched_duration_ms / 1000., request.method, self.name, tag
-        )
-
-        response_size.inc_by(request.sentLength, request.method, self.name, tag)
-
-        # We always call this at the end to ensure that we update the metrics
-        # regardless of whether a call to /metrics while the request was in
-        # flight.
-        self.update_metrics()
-
-    def update_metrics(self):
-        """Updates the in flight metrics with values from this request.
-        """
-
-        diff = self._request_stats.update(self.start_context)
-
-        in_flight_requests_ru_utime.inc_by(
-            diff.ru_utime, self.method, self.name,
-        )
-
-        in_flight_requests_ru_stime.inc_by(
-            diff.ru_stime, self.method, self.name,
-        )
-
-        in_flight_requests_db_txn_count.inc_by(
-            diff.db_txn_count, self.method, self.name,
-        )
-
-        in_flight_requests_db_txn_duration.inc_by(
-            diff.db_txn_duration_ms / 1000., self.method, self.name,
-        )
-
-        in_flight_requests_db_sched_duration.inc_by(
-            diff.db_sched_duration_ms / 1000., self.method, self.name,
-        )
-
-
-class _RequestStats(object):
-    """Keeps tracks of various metrics for an in flight request.
-    """
-
-    __slots__ = [
-        "ru_utime", "ru_stime",
-        "db_txn_count", "db_txn_duration_ms", "db_sched_duration_ms",
-    ]
-
-    def __init__(self, ru_utime, ru_stime, db_txn_count,
-                 db_txn_duration_ms, db_sched_duration_ms):
-        self.ru_utime = ru_utime
-        self.ru_stime = ru_stime
-        self.db_txn_count = db_txn_count
-        self.db_txn_duration_ms = db_txn_duration_ms
-        self.db_sched_duration_ms = db_sched_duration_ms
-
-    @staticmethod
-    def from_context(context):
-        ru_utime, ru_stime = context.get_resource_usage()
-
-        return _RequestStats(
-            ru_utime, ru_stime,
-            context.db_txn_count,
-            context.db_txn_duration_ms,
-            context.db_sched_duration_ms,
-        )
-
-    def update(self, context):
-        """Updates the current values and returns the difference between the
-        old and new values.
-
-        Returns:
-            _RequestStats: The difference between the old and new values
-        """
-        new = _RequestStats.from_context(context)
-
-        diff = _RequestStats(
-            new.ru_utime - self.ru_utime,
-            new.ru_stime - self.ru_stime,
-            new.db_txn_count - self.db_txn_count,
-            new.db_txn_duration_ms - self.db_txn_duration_ms,
-            new.db_sched_duration_ms - self.db_sched_duration_ms,
-        )
-
-        self.ru_utime = new.ru_utime
-        self.ru_stime = new.ru_stime
-        self.db_txn_count = new.db_txn_count
-        self.db_txn_duration_ms = new.db_txn_duration_ms
-        self.db_sched_duration_ms = new.db_sched_duration_ms
-
-        return diff
@@ -13,15 +13,11 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import cgi
-from six.moves import http_client
+

 from synapse.api.errors import (
    cs_exception, SynapseError, CodeMessageException, UnrecognizedRequestError, Codes
 )
-from synapse.http.request_metrics import (
-    requests_counter,
-)
 from synapse.util.logcontext import LoggingContext, PreserveLoggingContext
 from synapse.util.caches import intern_dict
 from synapse.util.metrics import Measure
@@ -45,174 +41,178 @@ import simplejson

 logger = logging.getLogger(__name__)

-HTML_ERROR_TEMPLATE = """<!DOCTYPE html>
-<html lang=en>
-  <head>
-    <meta charset="utf-8">
-    <title>Error {code}</title>
-  </head>
-  <body>
-     <p>{msg}</p>
-  </body>
-</html>
-"""
+metrics = synapse.metrics.get_metrics_for(__name__)
+
+# total number of responses served, split by method/servlet/tag
+response_count = metrics.register_counter(
+    "response_count",
+    labels=["method", "servlet", "tag"],
+    alternative_names=(
+        # the following are all deprecated aliases for the same metric
+        metrics.name_prefix + x for x in (
+            "_requests",
+            "_response_time:count",
+            "_response_ru_utime:count",
+            "_response_ru_stime:count",
+            "_response_db_txn_count:count",
+            "_response_db_txn_duration:count",
+        )
+    )
+)
+
+requests_counter = metrics.register_counter(
+    "requests_received",
+    labels=["method", "servlet", ],
+)
+
+outgoing_responses_counter = metrics.register_counter(
+    "responses",
+    labels=["method", "code"],
+)
+
+response_timer = metrics.register_counter(
+    "response_time_seconds",
+    labels=["method", "servlet", "tag"],
+    alternative_names=(
+        metrics.name_prefix + "_response_time:total",
+    ),
+)
+
+response_ru_utime = metrics.register_counter(
+    "response_ru_utime_seconds", labels=["method", "servlet", "tag"],
+    alternative_names=(
+        metrics.name_prefix + "_response_ru_utime:total",
+    ),
+)
+
+response_ru_stime = metrics.register_counter(
+    "response_ru_stime_seconds", labels=["method", "servlet", "tag"],
+    alternative_names=(
+        metrics.name_prefix + "_response_ru_stime:total",
+    ),
+)
+
+response_db_txn_count = metrics.register_counter(
+    "response_db_txn_count", labels=["method", "servlet", "tag"],
+    alternative_names=(
+        metrics.name_prefix + "_response_db_txn_count:total",
+    ),
+)
+
+# seconds spent waiting for db txns, excluding scheduling time, when processing
+# this request
+response_db_txn_duration = metrics.register_counter(
+    "response_db_txn_duration_seconds", labels=["method", "servlet", "tag"],
+    alternative_names=(
+        metrics.name_prefix + "_response_db_txn_duration:total",
+    ),
+)
+
+# seconds spent waiting for a db connection, when processing this request
+response_db_sched_duration = metrics.register_counter(
+    "response_db_sched_duration_seconds", labels=["method", "servlet", "tag"]
+)
+
+# size in bytes of the response written
+response_size = metrics.register_counter(
+    "response_size", labels=["method", "servlet", "tag"]
+)
+
+_next_request_id = 0


-def wrap_json_request_handler(h):
-    """Wraps a request handler method with exception handling.
+def request_handler(include_metrics=False):
+    """Decorator for ``wrap_request_handler``"""
+    return lambda request_handler: wrap_request_handler(request_handler, include_metrics)

-    Also adds logging as per wrap_request_handler_with_logging.

-    The handler method must have a signature of "handle_foo(self, request)",
-    where "self" must have a "clock" attribute (and "request" must be a
-    SynapseRequest).
+def wrap_request_handler(request_handler, include_metrics=False):
+    """Wraps a method that acts as a request handler with the necessary logging
+    and exception handling.

-    The handler must return a deferred. If the deferred succeeds we assume that
+    The method must have a signature of "handle_foo(self, request)". The
+    argument "self" must have "version_string" and "clock" attributes. The
+    argument "request" must be a twisted HTTP request.
+
+    The method must return a deferred. If the deferred succeeds we assume that
    a response has been sent. If the deferred fails with a SynapseError we use
    it to send a JSON response with the appropriate HTTP reponse code. If the
    deferred fails with any other type of error we send a 500 reponse.
+
+    We insert a unique request-id into the logging context for this request and
+    log the response and duration for this request.
    """

    @defer.inlineCallbacks
    def wrapped_request_handler(self, request):
-        try:
-            yield h(self, request)
-        except CodeMessageException as e:
-            code = e.code
-            if isinstance(e, SynapseError):
-                logger.info(
-                    "%s SynapseError: %s - %s", request, code, e.msg
-                )
-            else:
-                logger.exception(e)
-            respond_with_json(
-                request, code, cs_exception(e), send_cors=True,
-                pretty_print=_request_user_agent_is_curl(request),
-            )
+        global _next_request_id
+        request_id = "%s-%s" % (request.method, _next_request_id)
+        _next_request_id += 1

-        except Exception:
-            # failure.Failure() fishes the original Failure out
-            # of our stack, and thus gives us a sensible stack
-            # trace.
-            f = failure.Failure()
-            logger.error(
-                "Failed handle request via %r: %r: %s",
-                h,
-                request,
-                f.getTraceback().rstrip(),
-            )
-            respond_with_json(
-                request,
-                500,
-                {
-                    "error": "Internal server error",
-                    "errcode": Codes.UNKNOWN,
-                },
-                send_cors=True,
-                pretty_print=_request_user_agent_is_curl(request),
-            )
-
-    return wrap_request_handler_with_logging(wrapped_request_handler)
-
-
-def wrap_html_request_handler(h):
-    """Wraps a request handler method with exception handling.
-
-    Also adds logging as per wrap_request_handler_with_logging.
-
-    The handler method must have a signature of "handle_foo(self, request)",
-    where "self" must have a "clock" attribute (and "request" must be a
-    SynapseRequest).
-    """
-    def wrapped_request_handler(self, request):
-        d = defer.maybeDeferred(h, self, request)
-        d.addErrback(_return_html_error, request)
-        return d
-
-    return wrap_request_handler_with_logging(wrapped_request_handler)
-
-
-def _return_html_error(f, request):
-    """Sends an HTML error page corresponding to the given failure
-
-    Args:
-        f (twisted.python.failure.Failure):
-        request (twisted.web.iweb.IRequest):
-    """
-    if f.check(CodeMessageException):
-        cme = f.value
-        code = cme.code
-        msg = cme.msg
-
-        if isinstance(cme, SynapseError):
-            logger.info(
-                "%s SynapseError: %s - %s", request, code, msg
-            )
-        else:
-            logger.error(
-                "Failed handle request %r: %s",
-                request,
-                f.getTraceback().rstrip(),
-            )
-    else:
-        code = http_client.INTERNAL_SERVER_ERROR
-        msg = "Internal server error"
-
-        logger.error(
-            "Failed handle request %r: %s",
-            request,
-            f.getTraceback().rstrip(),
-        )
-
-    body = HTML_ERROR_TEMPLATE.format(
-        code=code, msg=cgi.escape(msg),
-    ).encode("utf-8")
-    request.setResponseCode(code)
-    request.setHeader(b"Content-Type", b"text/html; charset=utf-8")
-    request.setHeader(b"Content-Length", b"%i" % (len(body),))
-    request.write(body)
-    finish_request(request)
-
-
-def wrap_request_handler_with_logging(h):
-    """Wraps a request handler to provide logging and metrics
-
-    The handler method must have a signature of "handle_foo(self, request)",
-    where "self" must have a "clock" attribute (and "request" must be a
-    SynapseRequest).
-
-    As well as calling `request.processing` (which will log the response and
-    duration for this request), the wrapped request handler will insert the
-    request id into the logging context.
-    """
-    @defer.inlineCallbacks
-    def wrapped_request_handler(self, request):
-        """
-        Args:
-            self:
-            request (synapse.http.site.SynapseRequest):
-        """
-
-        request_id = request.get_request_id()
        with LoggingContext(request_id) as request_context:
-            request_context.request = request_id
            with Measure(self.clock, "wrapped_request_handler"):
+                request_metrics = RequestMetrics()
                # we start the request metrics timer here with an initial stab
                # at the servlet name. For most requests that name will be
                # JsonResource (or a subclass), and JsonResource._async_render
                # will update it once it picks a servlet.
                servlet_name = self.__class__.__name__
-                with request.processing(servlet_name):
-                    with PreserveLoggingContext(request_context):
-                        d = defer.maybeDeferred(h, self, request)
+                request_metrics.start(self.clock, name=servlet_name)

-                        # record the arrival of the request *after*
-                        # dispatching to the handler, so that the handler
-                        # can update the servlet name in the request
-                        # metrics
-                        requests_counter.inc(request.method,
-                                             request.request_metrics.name)
-                        yield d
+                request_context.request = request_id
+                with request.processing():
+                    try:
+                        with PreserveLoggingContext(request_context):
+                            if include_metrics:
+                                yield request_handler(self, request, request_metrics)
+                            else:
+                                requests_counter.inc(request.method, servlet_name)
+                                yield request_handler(self, request)
+                    except CodeMessageException as e:
+                        code = e.code
+                        if isinstance(e, SynapseError):
+                            logger.info(
+                                "%s SynapseError: %s - %s", request, code, e.msg
+                            )
+                        else:
+                            logger.exception(e)
+                        outgoing_responses_counter.inc(request.method, str(code))
+                        respond_with_json(
+                            request, code, cs_exception(e), send_cors=True,
+                            pretty_print=_request_user_agent_is_curl(request),
+                            version_string=self.version_string,
+                        )
+                    except Exception:
+                        # failure.Failure() fishes the original Failure out
+                        # of our stack, and thus gives us a sensible stack
+                        # trace.
+                        f = failure.Failure()
+                        logger.error(
+                            "Failed handle request %s.%s on %r: %r: %s",
+                            request_handler.__module__,
+                            request_handler.__name__,
+                            self,
+                            request,
+                            f.getTraceback().rstrip(),
+                        )
+                        respond_with_json(
+                            request,
+                            500,
+                            {
+                                "error": "Internal server error",
+                                "errcode": Codes.UNKNOWN,
+                            },
+                            send_cors=True,
+                            pretty_print=_request_user_agent_is_curl(request),
+                            version_string=self.version_string,
+                        )
+                    finally:
+                        try:
+                            request_metrics.stop(
+                                self.clock, request
+                            )
+                        except Exception as e:
+                            logger.warn("Failed to stop metrics: %r", e)
    return wrapped_request_handler


@@ -262,6 +262,7 @@ class JsonResource(HttpServer, resource.Resource):
        self.canonical_json = canonical_json
        self.clock = hs.get_clock()
        self.path_regexs = {}
+        self.version_string = hs.version_string
        self.hs = hs

    def register_paths(self, method, path_patterns, callback):
@@ -277,9 +278,13 @@ class JsonResource(HttpServer, resource.Resource):
        self._async_render(request)
        return server.NOT_DONE_YET

-    @wrap_json_request_handler
+    # Disable metric reporting because _async_render does its own metrics.
+    # It does its own metric reporting because _async_render dispatches to
+    # a callback and it's the class name of that callback we want to report
+    # against rather than the JsonResource itself.
+    @request_handler(include_metrics=True)
    @defer.inlineCallbacks
-    def _async_render(self, request):
+    def _async_render(self, request, request_metrics):
        """ This gets called from render() every time someone sends us a request.
            This checks if anyone has registered a callback for that method and
            path.
@@ -291,7 +296,9 @@ class JsonResource(HttpServer, resource.Resource):
            servlet_classname = servlet_instance.__class__.__name__
        else:
            servlet_classname = "%r" % callback
-        request.request_metrics.name = servlet_classname
+
+        request_metrics.name = servlet_classname
+        requests_counter.inc(request.method, servlet_classname)

        # Now trigger the callback. If it returns a response, we send it
        # here. If it throws an exception, that is handled by the wrapper
@@ -338,12 +345,15 @@ class JsonResource(HttpServer, resource.Resource):

    def _send_response(self, request, code, response_json_object,
                       response_code_message=None):
+        outgoing_responses_counter.inc(request.method, str(code))
+
        # TODO: Only enable CORS for the requests that need it.
        respond_with_json(
            request, code, response_json_object,
            send_cors=True,
            response_code_message=response_code_message,
            pretty_print=_request_user_agent_is_curl(request),
+            version_string=self.version_string,
            canonical_json=self.canonical_json,
        )

@@ -376,6 +386,54 @@ def _unrecognised_request_handler(request):
    raise UnrecognizedRequestError()


+class RequestMetrics(object):
+    def start(self, clock, name):
+        self.start = clock.time_msec()
+        self.start_context = LoggingContext.current_context()
+        self.name = name
+
+    def stop(self, clock, request):
+        context = LoggingContext.current_context()
+
+        tag = ""
+        if context:
+            tag = context.tag
+
+            if context != self.start_context:
+                logger.warn(
+                    "Context have unexpectedly changed %r, %r",
+                    context, self.start_context
+                )
+                return
+
+        response_count.inc(request.method, self.name, tag)
+
+        response_timer.inc_by(
+            clock.time_msec() - self.start, request.method,
+            self.name, tag
+        )
+
+        ru_utime, ru_stime = context.get_resource_usage()
+
+        response_ru_utime.inc_by(
+            ru_utime, request.method, self.name, tag
+        )
+        response_ru_stime.inc_by(
+            ru_stime, request.method, self.name, tag
+        )
+        response_db_txn_count.inc_by(
+            context.db_txn_count, request.method, self.name, tag
+        )
+        response_db_txn_duration.inc_by(
+            context.db_txn_duration_ms / 1000., request.method, self.name, tag
+        )
+        response_db_sched_duration.inc_by(
+            context.db_sched_duration_ms / 1000., request.method, self.name, tag
+        )
+
+        response_size.inc_by(request.sentLength, request.method, self.name, tag)
+
+
 class RootRedirect(resource.Resource):
    """Redirects the root '/' path to another path."""

@@ -394,7 +452,7 @@ class RootRedirect(resource.Resource):

 def respond_with_json(request, code, json_object, send_cors=False,
                      response_code_message=None, pretty_print=False,
-                      canonical_json=True):
+                      version_string="", canonical_json=True):
    # could alternatively use request.notifyFinish() and flip a flag when
    # the Deferred fires, but since the flag is RIGHT THERE it seems like
    # a waste.
@@ -416,11 +474,12 @@ def respond_with_json(request, code, json_object, send_cors=False,
        request, code, json_bytes,
        send_cors=send_cors,
        response_code_message=response_code_message,
+        version_string=version_string
    )


 def respond_with_json_bytes(request, code, json_bytes, send_cors=False,
-                            response_code_message=None):
+                            version_string="", response_code_message=None):
    """Sends encoded JSON in response to the given request.

    Args:
@@ -434,6 +493,7 @@ def respond_with_json_bytes(request, code, json_bytes, send_cors=False,

    request.setResponseCode(code, message=response_code_message)
    request.setHeader(b"Content-Type", b"application/json")
+    request.setHeader(b"Server", version_string)
    request.setHeader(b"Content-Length", b"%d" % (len(json_bytes),))
    request.setHeader(b"Cache-Control", b"no-cache, no-store, must-revalidate")

@@ -12,51 +12,27 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

+from synapse.util.logcontext import LoggingContext
+from twisted.web.server import Site, Request
+
 import contextlib
 import logging
 import re
 import time

-from twisted.web.server import Site, Request
-
-from synapse.http.request_metrics import RequestMetrics
-from synapse.util.logcontext import LoggingContext
-
-logger = logging.getLogger(__name__)
-
 ACCESS_TOKEN_RE = re.compile(br'(\?.*access(_|%5[Ff])token=)[^&]*(.*)$')

-_next_request_seq = 0
-

 class SynapseRequest(Request):
-    """Class which encapsulates an HTTP request to synapse.
-
-    All of the requests processed in synapse are of this type.
-
-    It extends twisted's twisted.web.server.Request, and adds:
-     * Unique request ID
-     * Redaction of access_token query-params in __repr__
-     * Logging at start and end
-     * Metrics to record CPU, wallclock and DB time by endpoint.
-
-    It provides a method `processing` which should be called by the Resource
-    which is handling the request, and returns a context manager.
-
-    """
    def __init__(self, site, *args, **kw):
        Request.__init__(self, *args, **kw)
        self.site = site
        self.authenticated_entity = None
        self.start_time = 0

-        global _next_request_seq
-        self.request_seq = _next_request_seq
-        _next_request_seq += 1
-
    def __repr__(self):
        # We overwrite this so that we don't log ``access_token``
-        return '<%s at 0x%x method=%r uri=%r clientproto=%r site=%r>' % (
+        return '<%s at 0x%x method=%s uri=%s clientproto=%s site=%s>' % (
            self.__class__.__name__,
            id(self),
            self.method,
@@ -65,9 +41,6 @@ class SynapseRequest(Request):
            self.site.site_tag,
        )

-    def get_request_id(self):
-        return "%s-%i" % (self.method, self.request_seq)
-
    def get_redacted_uri(self):
        return ACCESS_TOKEN_RE.sub(
            br'\1<redacted>\3',
@@ -77,18 +50,7 @@ class SynapseRequest(Request):
    def get_user_agent(self):
        return self.requestHeaders.getRawHeaders(b"User-Agent", [None])[-1]

-    def render(self, resrc):
-        # override the Server header which is set by twisted
-        self.setHeader("Server", self.site.server_version_string)
-        return Request.render(self, resrc)
-
-    def _started_processing(self, servlet_name):
-        self.start_time = int(time.time() * 1000)
-        self.request_metrics = RequestMetrics()
-        self.request_metrics.start(
-            self.start_time, name=servlet_name, method=self.method,
-        )
-
+    def started_processing(self):
        self.site.access_logger.info(
            "%s - %s - Received request: %s %s",
            self.getClientIP(),
@@ -96,8 +58,10 @@ class SynapseRequest(Request):
            self.method,
            self.get_redacted_uri()
        )
+        self.start_time = int(time.time() * 1000)
+
+    def finished_processing(self):

-    def _finished_processing(self):
        try:
            context = LoggingContext.current_context()
            ru_utime, ru_stime = context.get_resource_usage()
@@ -108,8 +72,6 @@ class SynapseRequest(Request):
            ru_utime, ru_stime = (0, 0)
            db_txn_count, db_txn_duration_ms = (0, 0)

-        end_time = int(time.time() * 1000)
-
        self.site.access_logger.info(
            "%s - %s - {%s}"
            " Processed request: %dms (%dms, %dms) (%dms/%dms/%d)"
@@ -117,7 +79,7 @@ class SynapseRequest(Request):
            self.getClientIP(),
            self.site.site_tag,
            self.authenticated_entity,
-            end_time - self.start_time,
+            int(time.time() * 1000) - self.start_time,
            int(ru_utime * 1000),
            int(ru_stime * 1000),
            db_sched_duration_ms,
@@ -131,38 +93,11 @@ class SynapseRequest(Request):
            self.get_user_agent(),
        )

-        try:
-            self.request_metrics.stop(end_time, self)
-        except Exception as e:
-            logger.warn("Failed to stop metrics: %r", e)
-
    @contextlib.contextmanager
-    def processing(self, servlet_name):
-        """Record the fact that we are processing this request.
-
-        Returns a context manager; the correct way to use this is:
-
-        @defer.inlineCallbacks
-        def handle_request(request):
-            with request.processing("FooServlet"):
-                yield really_handle_the_request()
-
-        This will log the request's arrival. Once the context manager is
-        closed, the completion of the request will be logged, and the various
-        metrics will be updated.
-
-        Args:
-            servlet_name (str): the name of the servlet which will be
-                processing this request. This is used in the metrics.
-
-                It is possible to update this afterwards by updating
-                self.request_metrics.servlet_name.
-        """
-        # TODO: we should probably just move this into render() and finish(),
-        # to save having to call a separate method.
-        self._started_processing(servlet_name)
+    def processing(self):
+        self.started_processing()
        yield
-        self._finished_processing()
+        self.finished_processing()


 class XForwardedForRequest(SynapseRequest):
@@ -200,8 +135,7 @@ class SynapseSite(Site):
    Subclass of a twisted http Site that does access logging with python's
    standard logging
    """
-    def __init__(self, logger_name, site_tag, config, resource,
-                 server_version_string, *args, **kwargs):
+    def __init__(self, logger_name, site_tag, config, resource, *args, **kwargs):
        Site.__init__(self, resource, *args, **kwargs)

        self.site_tag = site_tag
@@ -209,7 +143,6 @@ class SynapseSite(Site):
        proxied = config.get("x_forwarded", False)
        self.requestFactory = SynapseRequestFactory(self, proxied)
        self.access_logger = logging.getLogger(logger_name)
-        self.server_version_string = server_version_string

    def log(self, request):
        pass
@@ -71,8 +71,7 @@ class BaseMetric(object):
        """Render this metric for a single set of labels

        Args:
-            label_values (list[object]): values for each of the labels,
-                (which get stringified).
+            label_values (list[str]): values for each of the labels
            value: value of the metric at with these labels

        Returns:
@@ -325,4 +324,4 @@ def _escape_character(m):
 def _escape_label_value(value):
    """Takes a label value and escapes quotes, newlines and backslashes
    """
-    return re.sub(r"([\n\"\\])", _escape_character, str(value))
+    return re.sub(r"([\n\"\\])", _escape_character, value)
@@ -15,7 +15,6 @@

 import os

-from six import iteritems

 TICKS_PER_SEC = 100
 BYTES_PER_PAGE = 4096
@@ -56,7 +55,7 @@ def update_resource_metrics():
            # line is PID (command) more stats go here ...
            raw_stats = line.split(") ", 1)[1].split(" ")

-            for (name, index) in iteritems(STAT_FIELDS):
+            for (name, index) in STAT_FIELDS.iteritems():
                # subtract 3 from the index, because proc(5) is 1-based, and
                # we've lost the first two fields in PID and COMMAND above
                stats[name] = int(raw_stats[index - 3])
@@ -30,7 +30,6 @@ from synapse.state import POWER_KEY

 from collections import namedtuple

-from six import itervalues, iteritems

 logger = logging.getLogger(__name__)

@@ -127,7 +126,7 @@ class BulkPushRuleEvaluator(object):
            )
            auth_events = yield self.store.get_events(auth_events_ids)
            auth_events = {
-                (e.type, e.state_key): e for e in itervalues(auth_events)
+                (e.type, e.state_key): e for e in auth_events.itervalues()
            }

        sender_level = get_user_power_level(event.sender, auth_events)
@@ -161,7 +160,7 @@ class BulkPushRuleEvaluator(object):

        condition_cache = {}

-        for uid, rules in iteritems(rules_by_user):
+        for uid, rules in rules_by_user.iteritems():
            if event.sender == uid:
                continue

@@ -407,7 +406,7 @@ class RulesForRoom(object):
        # If the event is a join event then it will be in current state evnts
        # map but not in the DB, so we have to explicitly insert it.
        if event.type == EventTypes.Member:
-            for event_id in itervalues(member_event_ids):
+            for event_id in member_event_ids.itervalues():
                if event_id == event.event_id:
                    members[event_id] = (event.state_key, event.membership)

@@ -415,7 +414,7 @@ class RulesForRoom(object):
            logger.debug("Found members %r: %r", self.room_id, members.values())

        interested_in_user_ids = set(
-            user_id for user_id, membership in itervalues(members)
+            user_id for user_id, membership in members.itervalues()
            if membership == Membership.JOIN
        )

@@ -427,7 +426,7 @@ class RulesForRoom(object):
        )

        user_ids = set(
-            uid for uid, have_pusher in iteritems(if_users_with_pushers) if have_pusher
+            uid for uid, have_pusher in if_users_with_pushers.iteritems() if have_pusher
        )

        logger.debug("With pushers: %r", user_ids)
@@ -448,7 +447,7 @@ class RulesForRoom(object):
        )

        ret_rules_by_user.update(
-            item for item in iteritems(rules_by_user) if item[0] is not None
+            item for item in rules_by_user.iteritems() if item[0] is not None
        )

        self.update_cache(sequence, members, ret_rules_by_user, state_group)
@@ -21,8 +21,6 @@ from synapse.types import UserID
 from synapse.util.caches import CACHE_SIZE_FACTOR, register_cache
 from synapse.util.caches.lrucache import LruCache

-from six import string_types
-
 logger = logging.getLogger(__name__)


@@ -240,7 +238,7 @@ def _flatten_dict(d, prefix=[], result=None):
    if result is None:
        result = {}
    for key, value in d.items():
-        if isinstance(value, string_types):
+        if isinstance(value, basestring):
            result[".".join(prefix + [key])] = value.lower()
        elif hasattr(value, "items"):
            _flatten_dict(value, prefix=(prefix + [key]), result=result)
@@ -68,7 +68,6 @@ import synapse.metrics
 import struct
 import fcntl

-from six import iterkeys, iteritems

 metrics = synapse.metrics.get_metrics_for(__name__)

@@ -393,7 +392,7 @@ class ServerReplicationStreamProtocol(BaseReplicationStreamProtocol):

        if stream_name == "ALL":
            # Subscribe to all streams we're publishing to.
-            for stream in iterkeys(self.streamer.streams_by_name):
+            for stream in self.streamer.streams_by_name.iterkeys():
                self.subscribe_to_stream(stream, token)
        else:
            self.subscribe_to_stream(stream_name, token)
@@ -499,7 +498,7 @@ class ClientReplicationStreamProtocol(BaseReplicationStreamProtocol):
        BaseReplicationStreamProtocol.connectionMade(self)

        # Once we've connected subscribe to the necessary streams
-        for stream_name, token in iteritems(self.handler.get_streams_to_replicate()):
+        for stream_name, token in self.handler.get_streams_to_replicate().iteritems():
            self.replicate(stream_name, token)

        # Tell the server if we have any users currently syncing (should only
@@ -634,7 +633,7 @@ metrics.register_callback(
    lambda: {
        (k[0], p.name, p.conn_id): count
        for p in connected_connections
-        for k, count in iteritems(p.inbound_commands_counter.counts)
+        for k, count in p.inbound_commands_counter.counts.iteritems()
    },
    labels=["command", "name", "conn_id"],
 )
@@ -644,7 +643,7 @@ metrics.register_callback(
    lambda: {
        (k[0], p.name, p.conn_id): count
        for p in connected_connections
-        for k, count in iteritems(p.outbound_commands_counter.counts)
+        for k, count in p.outbound_commands_counter.counts.iteritems()
    },
    labels=["command", "name", "conn_id"],
 )
@@ -26,7 +26,6 @@ from synapse.util.metrics import Measure, measure_func
 import logging
 import synapse.metrics

-from six import itervalues

 metrics = synapse.metrics.get_metrics_for(__name__)
 stream_updates_counter = metrics.register_counter(
@@ -70,7 +69,6 @@ class ReplicationStreamer(object):
        self.presence_handler = hs.get_presence_handler()
        self.clock = hs.get_clock()
        self.notifier = hs.get_notifier()
-        self._server_notices_sender = hs.get_server_notices_sender()

        # Current connections.
        self.connections = []
@@ -81,7 +79,7 @@ class ReplicationStreamer(object):
        # We only support federation stream if federation sending hase been
        # disabled on the master.
        self.streams = [
-            stream(hs) for stream in itervalues(STREAMS_MAP)
+            stream(hs) for stream in STREAMS_MAP.itervalues()
            if stream != FederationStream or not hs.config.send_federation
        ]

@@ -255,7 +253,6 @@ class ReplicationStreamer(object):
        yield self.store.insert_client_ip(
            user_id, access_token, ip, user_agent, device_id, last_seen,
        )
-        yield self._server_notices_sender.on_user_ip(user_id)

    def send_sync_to_all_connections(self, data):
        """Sends a SYNC command to all clients.
@@ -19,7 +19,6 @@ import logging

 from synapse.api.auth import get_access_token_from_request
 from synapse.util.async import ObservableDeferred
-from synapse.util.logcontext import make_deferred_yieldable, run_in_background

 logger = logging.getLogger(__name__)

@@ -81,26 +80,27 @@ class HttpTransactionCache(object):
        Returns:
            Deferred which resolves to a tuple of (response_code, response_dict).
        """
-        if txn_key in self.transactions:
-            observable = self.transactions[txn_key][0]
-        else:
-            # execute the function instead.
-            deferred = run_in_background(fn, *args, **kwargs)
+        try:
+            return self.transactions[txn_key][0].observe()
+        except (KeyError, IndexError):
+            pass  # execute the function instead.

-            observable = ObservableDeferred(deferred)
-            self.transactions[txn_key] = (observable, self.clock.time_msec())
+        deferred = fn(*args, **kwargs)

-            # if the request fails with an exception, remove it
-            # from the transaction map. This is done to ensure that we don't
-            # cache transient errors like rate-limiting errors, etc.
-            def remove_from_map(err):
-                self.transactions.pop(txn_key, None)
-                # we deliberately do not propagate the error any further, as we
-                # expect the observers to have reported it.
+        # if the request fails with a Twisted failure, remove it
+        # from the transaction map. This is done to ensure that we don't
+        # cache transient errors like rate-limiting errors, etc.
+        def remove_from_map(err):
+            self.transactions.pop(txn_key, None)
+            return err
+        deferred.addErrback(remove_from_map)

-            deferred.addErrback(remove_from_map)
-
-        return make_deferred_yieldable(observable.observe())
+        # We don't add any other errbacks to the raw deferred, so we ask
+        # ObservableDeferred to swallow the error. This is fine as the error will
+        # still be reported to the observers.
+        observable = ObservableDeferred(deferred, consumeErrors=True)
+        self.transactions[txn_key] = (observable, self.clock.time_msec())
+        return observable.observe()

    def _cleanup(self):
        now = self.clock.time_msec()
@@ -151,11 +151,10 @@ class PurgeHistoryRestServlet(ClientV1RestServlet):
            if event.room_id != room_id:
                raise SynapseError(400, "Event is for wrong room.")

-            token = yield self.store.get_topological_token_for_event(event_id)
-
+            depth = event.depth
            logger.info(
-                "[purge] purging up to token %s (event_id %s)",
-                token, event_id,
+                "[purge] purging up to depth %i (event_id %s)",
+                depth, event_id,
            )
        elif 'purge_up_to_ts' in body:
            ts = body['purge_up_to_ts']
@@ -175,9 +174,7 @@ class PurgeHistoryRestServlet(ClientV1RestServlet):
                )
            )
            if room_event_after_stream_ordering:
-                token = yield self.store.get_topological_token_for_event(
-                    room_event_after_stream_ordering,
-                )
+                (_, depth, _) = room_event_after_stream_ordering
            else:
                logger.warn(
                    "[purge] purging events not possible: No event found "
@@ -190,9 +187,9 @@ class PurgeHistoryRestServlet(ClientV1RestServlet):
                    errcode=Codes.NOT_FOUND,
                )
            logger.info(
-                "[purge] purging up to token %d (received_ts %i => "
+                "[purge] purging up to depth %i (received_ts %i => "
                "stream_ordering %i)",
-                token, ts, stream_ordering,
+                depth, ts, stream_ordering,
            )
        else:
            raise SynapseError(
@@ -202,7 +199,7 @@ class PurgeHistoryRestServlet(ClientV1RestServlet):
            )

        purge_id = yield self.handlers.message_handler.start_purge_history(
-            room_id, token,
+            room_id, depth,
            delete_local_events=delete_local_events,
        )

@@ -276,8 +273,8 @@ class ShutdownRoomRestServlet(ClientV1RestServlet):
    def __init__(self, hs):
        super(ShutdownRoomRestServlet, self).__init__(hs)
        self.store = hs.get_datastore()
+        self.handlers = hs.get_handlers()
        self.state = hs.get_state_handler()
-        self._room_creation_handler = hs.get_room_creation_handler()
        self.event_creation_handler = hs.get_event_creation_handler()
        self.room_member_handler = hs.get_room_member_handler()

@@ -299,7 +296,7 @@ class ShutdownRoomRestServlet(ClientV1RestServlet):
        message = content.get("message", self.DEFAULT_MESSAGE)
        room_name = content.get("room_name", "Content Violation Notification")

-        info = yield self._room_creation_handler.create_room(
+        info = yield self.handlers.room_creation_handler.create_room(
            room_creator_requester,
            config={
                "preset": "public_chat",
@@ -23,8 +23,6 @@ from synapse.handlers.presence import format_user_presence_state
 from synapse.http.servlet import parse_json_object_from_request
 from .base import ClientV1RestServlet, client_path_patterns

-from six import string_types
-
 import logging

 logger = logging.getLogger(__name__)
@@ -73,7 +71,7 @@ class PresenceStatusRestServlet(ClientV1RestServlet):

            if "status_msg" in content:
                state["status_msg"] = content.pop("status_msg")
-                if not isinstance(state["status_msg"], string_types):
+                if not isinstance(state["status_msg"], basestring):
                    raise SynapseError(400, "status_msg must be a string.")

            if content:
@@ -131,7 +129,7 @@ class PresenceListRestServlet(ClientV1RestServlet):

        if "invite" in content:
            for u in content["invite"]:
-                if not isinstance(u, string_types):
+                if not isinstance(u, basestring):
                    raise SynapseError(400, "Bad invite value.")
                if len(u) == 0:
                    continue
@@ -142,7 +140,7 @@ class PresenceListRestServlet(ClientV1RestServlet):

        if "drop" in content:
            for u in content["drop"]:
-                if not isinstance(u, string_types):
+                if not isinstance(u, basestring):
                    raise SynapseError(400, "Bad drop value.")
                if len(u) == 0:
                    continue
@@ -176,6 +176,7 @@ class PushersRemoveRestServlet(RestServlet):

        request.setResponseCode(200)
        request.setHeader(b"Content-Type", b"text/html; charset=utf-8")
+        request.setHeader(b"Server", self.hs.version_string)
        request.setHeader(b"Content-Length", b"%d" % (
            len(PushersRemoveRestServlet.SUCCESS_HTML),
        ))
@@ -41,7 +41,7 @@ class RoomCreateRestServlet(ClientV1RestServlet):

    def __init__(self, hs):
        super(RoomCreateRestServlet, self).__init__(hs)
-        self._room_creation_handler = hs.get_room_creation_handler()
+        self.handlers = hs.get_handlers()

    def register(self, http_server):
        PATTERNS = "/createRoom"
@@ -64,7 +64,8 @@ class RoomCreateRestServlet(ClientV1RestServlet):
    def on_POST(self, request):
        requester = yield self.auth.get_user_by_req(request)

-        info = yield self._room_creation_handler.create_room(
+        handler = self.handlers.room_creation_handler
+        info = yield handler.create_room(
            requester, self.get_room_config(request)
        )

@@ -129,6 +129,7 @@ class AuthRestServlet(RestServlet):
            html_bytes = html.encode("utf8")
            request.setResponseCode(200)
            request.setHeader(b"Content-Type", b"text/html; charset=utf-8")
+            request.setHeader(b"Server", self.hs.version_string)
            request.setHeader(b"Content-Length", b"%d" % (len(html_bytes),))

            request.write(html_bytes)
@@ -174,6 +175,7 @@ class AuthRestServlet(RestServlet):
            html_bytes = html.encode("utf8")
            request.setResponseCode(200)
            request.setHeader(b"Content-Type", b"text/html; charset=utf-8")
+            request.setHeader(b"Server", self.hs.version_string)
            request.setHeader(b"Content-Length", b"%d" % (len(html_bytes),))

            request.write(html_bytes)
@@ -88,7 +88,7 @@ class NotificationsServlet(RestServlet):
                    pa["topological_ordering"], pa["stream_ordering"]
                )
            returned_push_actions.append(returned_pa)
-            next_token = str(pa["stream_ordering"])
+            next_token = pa["stream_ordering"]

        defer.returnValue((200, {
            "notifications": returned_push_actions,
@@ -85,7 +85,6 @@ class SyncRestServlet(RestServlet):
        self.clock = hs.get_clock()
        self.filtering = hs.get_filtering()
        self.presence_handler = hs.get_presence_handler()
-        self._server_notices_sender = hs.get_server_notices_sender()

    @defer.inlineCallbacks
    def on_GET(self, request):
@@ -150,9 +149,6 @@ class SyncRestServlet(RestServlet):
        else:
            since_token = None

-        # send any outstanding server notices to the user.
-        yield self._server_notices_sender.on_user_syncing(user.to_string())
-
        affect_presence = set_presence != PresenceState.OFFLINE

        if affect_presence:
@@ -1,222 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2018 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from hashlib import sha256
-import hmac
-import logging
-from os import path
-from six.moves import http_client
-
-import jinja2
-from jinja2 import TemplateNotFound
-from twisted.internet import defer
-from twisted.web.resource import Resource
-from twisted.web.server import NOT_DONE_YET
-
-from synapse.api.errors import NotFoundError, SynapseError, StoreError
-from synapse.config import ConfigError
-from synapse.http.server import (
-    finish_request,
-    wrap_html_request_handler,
-)
-from synapse.http.servlet import parse_string
-from synapse.types import UserID
-
-
-# language to use for the templates. TODO: figure this out from Accept-Language
-TEMPLATE_LANGUAGE = "en"
-
-logger = logging.getLogger(__name__)
-
-# use hmac.compare_digest if we have it (python 2.7.7), else just use equality
-if hasattr(hmac, "compare_digest"):
-    compare_digest = hmac.compare_digest
-else:
-    def compare_digest(a, b):
-        return a == b
-
-
-class ConsentResource(Resource):
-    """A twisted Resource to display a privacy policy and gather consent to it
-
-    When accessed via GET, returns the privacy policy via a template.
-
-    When accessed via POST, records the user's consent in the database and
-    displays a success page.
-
-    The config should include a template_dir setting which contains templates
-    for the HTML. The directory should contain one subdirectory per language
-    (eg, 'en', 'fr'), and each language directory should contain the policy
-    document (named as '<version>.html') and a success page (success.html).
-
-    Both forms take a set of parameters from the browser. For the POST form,
-    these are normally sent as form parameters (but may be query-params); for
-    GET requests they must be query params. These are:
-
-        u: the complete mxid, or the localpart of the user giving their
-           consent. Required for both GET (where it is used as an input to the
-           template) and for POST (where it is used to find the row in the db
-           to update).
-
-        h: hmac_sha256(secret, u), where 'secret' is the privacy_secret in the
-           config file. If it doesn't match, the request is 403ed.
-
-        v: the version of the privacy policy being agreed to.
-
-           For GET: optional, and defaults to whatever was set in the config
-           file. Used to choose the version of the policy to pick from the
-           templates directory.
-
-           For POST: required; gives the value to be recorded in the database
-           against the user.
-    """
-    def __init__(self, hs):
-        """
-        Args:
-            hs (synapse.server.HomeServer): homeserver
-        """
-        Resource.__init__(self)
-
-        self.hs = hs
-        self.store = hs.get_datastore()
-
-        # this is required by the request_handler wrapper
-        self.clock = hs.get_clock()
-
-        self._default_consent_version = hs.config.user_consent_version
-        if self._default_consent_version is None:
-            raise ConfigError(
-                "Consent resource is enabled but user_consent section is "
-                "missing in config file.",
-            )
-
-        # daemonize changes the cwd to /, so make the path absolute now.
-        consent_template_directory = path.abspath(
-            hs.config.user_consent_template_dir,
-        )
-        if not path.isdir(consent_template_directory):
-            raise ConfigError(
-                "Could not find template directory '%s'" % (
-                    consent_template_directory,
-                ),
-            )
-
-        loader = jinja2.FileSystemLoader(consent_template_directory)
-        self._jinja_env = jinja2.Environment(
-            loader=loader,
-            autoescape=jinja2.select_autoescape(['html', 'htm', 'xml']),
-        )
-
-        if hs.config.form_secret is None:
-            raise ConfigError(
-                "Consent resource is enabled but form_secret is not set in "
-                "config file. It should be set to an arbitrary secret string.",
-            )
-
-        self._hmac_secret = hs.config.form_secret.encode("utf-8")
-
-    def render_GET(self, request):
-        self._async_render_GET(request)
-        return NOT_DONE_YET
-
-    @wrap_html_request_handler
-    @defer.inlineCallbacks
-    def _async_render_GET(self, request):
-        """
-        Args:
-            request (twisted.web.http.Request):
-        """
-
-        version = parse_string(request, "v",
-                               default=self._default_consent_version)
-        username = parse_string(request, "u", required=True)
-        userhmac = parse_string(request, "h", required=True)
-
-        self._check_hash(username, userhmac)
-
-        if username.startswith('@'):
-            qualified_user_id = username
-        else:
-            qualified_user_id = UserID(username, self.hs.hostname).to_string()
-
-        u = yield self.store.get_user_by_id(qualified_user_id)
-        if u is None:
-            raise NotFoundError("Unknown user")
-
-        try:
-            self._render_template(
-                request, "%s.html" % (version,),
-                user=username, userhmac=userhmac, version=version,
-                has_consented=(u["consent_version"] == version),
-            )
-        except TemplateNotFound:
-            raise NotFoundError("Unknown policy version")
-
-    def render_POST(self, request):
-        self._async_render_POST(request)
-        return NOT_DONE_YET
-
-    @wrap_html_request_handler
-    @defer.inlineCallbacks
-    def _async_render_POST(self, request):
-        """
-        Args:
-            request (twisted.web.http.Request):
-        """
-        version = parse_string(request, "v", required=True)
-        username = parse_string(request, "u", required=True)
-        userhmac = parse_string(request, "h", required=True)
-
-        self._check_hash(username, userhmac)
-
-        if username.startswith('@'):
-            qualified_user_id = username
-        else:
-            qualified_user_id = UserID(username, self.hs.hostname).to_string()
-
-        try:
-            yield self.store.user_set_consent_version(qualified_user_id, version)
-        except StoreError as e:
-            if e.code != 404:
-                raise
-            raise NotFoundError("Unknown user")
-
-        try:
-            self._render_template(request, "success.html")
-        except TemplateNotFound:
-            raise NotFoundError("success.html not found")
-
-    def _render_template(self, request, template_name, **template_args):
-        # get_template checks for ".." so we don't need to worry too much
-        # about path traversal here.
-        template_html = self._jinja_env.get_template(
-            path.join(TEMPLATE_LANGUAGE, template_name)
-        )
-        html_bytes = template_html.render(**template_args).encode("utf8")
-
-        request.setHeader(b"Content-Type", b"text/html; charset=utf-8")
-        request.setHeader(b"Content-Length", b"%i" % len(html_bytes))
-        request.write(html_bytes)
-        finish_request(request)
-
-    def _check_hash(self, userid, userhmac):
-        want_mac = hmac.new(
-            key=self._hmac_secret,
-            msg=userid,
-            digestmod=sha256,
-        ).hexdigest()
-
-        if not compare_digest(want_mac, userhmac):
-            raise SynapseError(http_client.FORBIDDEN, "HMAC incorrect")
@@ -49,6 +49,7 @@ class LocalKey(Resource):
    """

    def __init__(self, hs):
+        self.version_string = hs.version_string
        self.response_body = encode_canonical_json(
            self.response_json_object(hs.config)
        )
@@ -83,6 +84,7 @@ class LocalKey(Resource):
    def render_GET(self, request):
        return respond_with_json_bytes(
            request, 200, self.response_body,
+            version_string=self.version_string
        )

    def getChild(self, name, request):
@@ -63,6 +63,7 @@ class LocalKey(Resource):
    isLeaf = True

    def __init__(self, hs):
+        self.version_string = hs.version_string
        self.config = hs.config
        self.clock = hs.clock
        self.update_response_body(self.clock.time_msec())
@@ -114,4 +115,5 @@ class LocalKey(Resource):
            self.update_response_body(time_now)
        return respond_with_json_bytes(
            request, 200, self.response_body,
+            version_string=self.version_string
        )
@@ -12,9 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-from synapse.http.server import (
-    respond_with_json_bytes, wrap_json_request_handler,
-)
+from synapse.http.server import request_handler, respond_with_json_bytes
 from synapse.http.servlet import parse_integer, parse_json_object_from_request
 from synapse.api.errors import SynapseError, Codes
 from synapse.crypto.keyring import KeyLookupError
@@ -93,6 +91,7 @@ class RemoteKey(Resource):
    def __init__(self, hs):
        self.keyring = hs.get_keyring()
        self.store = hs.get_datastore()
+        self.version_string = hs.version_string
        self.clock = hs.get_clock()
        self.federation_domain_whitelist = hs.config.federation_domain_whitelist

@@ -100,7 +99,7 @@ class RemoteKey(Resource):
        self.async_render_GET(request)
        return NOT_DONE_YET

-    @wrap_json_request_handler
+    @request_handler()
    @defer.inlineCallbacks
    def async_render_GET(self, request):
        if len(request.postpath) == 1:
@@ -125,7 +124,7 @@ class RemoteKey(Resource):
        self.async_render_POST(request)
        return NOT_DONE_YET

-    @wrap_json_request_handler
+    @request_handler()
    @defer.inlineCallbacks
    def async_render_POST(self, request):
        content = parse_json_object_from_request(request)
@@ -241,4 +240,5 @@ class RemoteKey(Resource):

            respond_with_json_bytes(
                request, 200, result_io.getvalue(),
+                version_string=self.version_string
            )
@@ -12,18 +12,16 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
-
-from twisted.internet import defer
-from twisted.web.resource import Resource
-from twisted.web.server import NOT_DONE_YET
-
-from synapse.http.server import (
-    set_cors_headers,
-    wrap_json_request_handler,
-)
 import synapse.http.servlet
+
 from ._base import parse_media_id, respond_404
+from twisted.web.resource import Resource
+from synapse.http.server import request_handler, set_cors_headers
+
+from twisted.web.server import NOT_DONE_YET
+from twisted.internet import defer
+
+import logging

 logger = logging.getLogger(__name__)

@@ -37,14 +35,15 @@ class DownloadResource(Resource):
        self.media_repo = media_repo
        self.server_name = hs.hostname

-        # this is expected by @wrap_json_request_handler
+        # Both of these are expected by @request_handler()
        self.clock = hs.get_clock()
+        self.version_string = hs.version_string

    def render_GET(self, request):
        self._async_render_GET(request)
        return NOT_DONE_YET

-    @wrap_json_request_handler
+    @request_handler()
    @defer.inlineCallbacks
    def _async_render_GET(self, request):
        set_cors_headers(request)
@@ -48,7 +48,6 @@ import shutil
 import cgi
 import logging
 from six.moves.urllib import parse as urlparse
-from six import iteritems

 logger = logging.getLogger(__name__)

@@ -604,7 +603,7 @@ class MediaRepository(object):
                thumbnails[(t_width, t_height, r_type)] = r_method

        # Now we generate the thumbnails for each dimension, store it
-        for (t_width, t_height, t_type), t_method in iteritems(thumbnails):
+        for (t_width, t_height, t_type), t_method in thumbnails.iteritems():
            # Generate the thumbnail
            if t_method == "crop":
                t_byte_source = yield make_deferred_yieldable(threads.deferToThread(
@@ -24,9 +24,7 @@ import shutil
 import sys
 import traceback
 import simplejson as json
-
-from six.moves import urllib_parse as urlparse
-from six import string_types
+import urlparse

 from twisted.web.server import NOT_DONE_YET
 from twisted.internet import defer
@@ -42,9 +40,8 @@ from synapse.util.stringutils import random_string
 from synapse.util.caches.expiringcache import ExpiringCache
 from synapse.http.client import SpiderHttpClient
 from synapse.http.server import (
-    respond_with_json_bytes,
+    request_handler, respond_with_json_bytes,
    respond_with_json,
-    wrap_json_request_handler,
 )
 from synapse.util.async import ObservableDeferred
 from synapse.util.stringutils import is_ascii
@@ -60,6 +57,7 @@ class PreviewUrlResource(Resource):

        self.auth = hs.get_auth()
        self.clock = hs.get_clock()
+        self.version_string = hs.version_string
        self.filepaths = media_repo.filepaths
        self.max_spider_size = hs.config.max_spider_size
        self.server_name = hs.hostname
@@ -92,7 +90,7 @@ class PreviewUrlResource(Resource):
        self._async_render_GET(request)
        return NOT_DONE_YET

-    @wrap_json_request_handler
+    @request_handler()
    @defer.inlineCallbacks
    def _async_render_GET(self, request):

@@ -592,8 +590,8 @@ def _iterate_over_text(tree, *tags_to_ignore):
    # to be returned.
    elements = iter([tree])
    while True:
-        el = next(elements)
-        if isinstance(el, string_types):
+        el = elements.next()
+        if isinstance(el, basestring):
            yield el
        elif el is not None and el.tag not in tags_to_ignore:
            # el.text is the text before the first child, so we can immediately
@@ -14,21 +14,18 @@
 # limitations under the License.


-import logging
-
-from twisted.internet import defer
-from twisted.web.resource import Resource
-from twisted.web.server import NOT_DONE_YET
-
-from synapse.http.server import (
-    set_cors_headers,
-    wrap_json_request_handler,
-)
-from synapse.http.servlet import parse_integer, parse_string
 from ._base import (
-    FileInfo, parse_media_id, respond_404, respond_with_file,
+    parse_media_id, respond_404, respond_with_file, FileInfo,
    respond_with_responder,
 )
+from twisted.web.resource import Resource
+from synapse.http.servlet import parse_string, parse_integer
+from synapse.http.server import request_handler, set_cors_headers
+
+from twisted.web.server import NOT_DONE_YET
+from twisted.internet import defer
+
+import logging

 logger = logging.getLogger(__name__)

@@ -44,13 +41,14 @@ class ThumbnailResource(Resource):
        self.media_storage = media_storage
        self.dynamic_thumbnails = hs.config.dynamic_thumbnails
        self.server_name = hs.hostname
+        self.version_string = hs.version_string
        self.clock = hs.get_clock()

    def render_GET(self, request):
        self._async_render_GET(request)
        return NOT_DONE_YET

-    @wrap_json_request_handler
+    @request_handler()
    @defer.inlineCallbacks
    def _async_render_GET(self, request):
        set_cors_headers(request)
@@ -13,17 +13,16 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-import logging
-
-from twisted.internet import defer
-from twisted.web.resource import Resource
-from twisted.web.server import NOT_DONE_YET
+from synapse.http.server import respond_with_json, request_handler

 from synapse.api.errors import SynapseError
-from synapse.http.server import (
-    respond_with_json,
-    wrap_json_request_handler,
-)
+
+from twisted.web.server import NOT_DONE_YET
+from twisted.internet import defer
+
+from twisted.web.resource import Resource
+
+import logging

 logger = logging.getLogger(__name__)

@@ -41,6 +40,7 @@ class UploadResource(Resource):
        self.server_name = hs.hostname
        self.auth = hs.get_auth()
        self.max_upload_size = hs.config.max_upload_size
+        self.version_string = hs.version_string
        self.clock = hs.get_clock()

    def render_POST(self, request):
@@ -51,7 +51,7 @@ class UploadResource(Resource):
        respond_with_json(request, 200, {}, send_cors=True)
        return NOT_DONE_YET

-    @wrap_json_request_handler
+    @request_handler()
    @defer.inlineCallbacks
    def _async_render_POST(self, request):
        requester = yield self.auth.get_user_by_req(request)
@@ -46,7 +46,6 @@ from synapse.handlers.devicemessage import DeviceMessageHandler
 from synapse.handlers.device import DeviceHandler
 from synapse.handlers.e2e_keys import E2eKeysHandler
 from synapse.handlers.presence import PresenceHandler
-from synapse.handlers.room import RoomCreationHandler
 from synapse.handlers.room_list import RoomListHandler
 from synapse.handlers.room_member import RoomMemberMasterHandler
 from synapse.handlers.room_member_worker import RoomMemberWorkerHandler
@@ -72,11 +71,6 @@ from synapse.rest.media.v1.media_repository import (
    MediaRepository,
    MediaRepositoryResource,
 )
-from synapse.server_notices.server_notices_manager import ServerNoticesManager
-from synapse.server_notices.server_notices_sender import ServerNoticesSender
-from synapse.server_notices.worker_server_notices_sender import (
-    WorkerServerNoticesSender,
-)
 from synapse.state import StateHandler, StateResolutionHandler
 from synapse.storage import DataStore
 from synapse.streams.events import EventSources
@@ -103,9 +97,6 @@ class HomeServer(object):
    which must be implemented by the subclass. This code may call any of the
    required "get" methods on the instance to obtain the sub-dependencies that
    one requires.
-
-    Attributes:
-        config (synapse.config.homeserver.HomeserverConfig):
    """

    DEPENDENCIES = [
@@ -115,7 +106,6 @@ class HomeServer(object):
        'federation_server',
        'handlers',
        'auth',
-        'room_creation_handler',
        'state_handler',
        'state_resolution_handler',
        'presence_handler',
@@ -161,8 +151,6 @@ class HomeServer(object):
        'spam_checker',
        'room_member_handler',
        'federation_registry',
-        'server_notices_manager',
-        'server_notices_sender',
    ]

    def __init__(self, hostname, **kwargs):
@@ -236,9 +224,6 @@ class HomeServer(object):
    def build_simple_http_client(self):
        return SimpleHttpClient(self)

-    def build_room_creation_handler(self):
-        return RoomCreationHandler(self)
-
    def build_state_handler(self):
        return StateHandler(self)

@@ -405,16 +390,6 @@ class HomeServer(object):
    def build_federation_registry(self):
        return FederationHandlerRegistry()

-    def build_server_notices_manager(self):
-        if self.config.worker_app:
-            raise Exception("Workers cannot send server notices")
-        return ServerNoticesManager(self)
-
-    def build_server_notices_sender(self):
-        if self.config.worker_app:
-            return WorkerServerNoticesSender(self)
-        return ServerNoticesSender(self)
-
    def remove_pusher(self, app_id, push_key, user_id):
        return self.get_pusherpool().remove_pusher(app_id, push_key, user_id)

@@ -1,5 +1,4 @@
 import synapse.api.auth
-import synapse.config.homeserver
 import synapse.federation.transaction_queue
 import synapse.federation.transport.client
 import synapse.handlers
@@ -9,17 +8,11 @@ import synapse.handlers.device
 import synapse.handlers.e2e_keys
 import synapse.handlers.set_password
 import synapse.rest.media.v1.media_repository
-import synapse.server_notices.server_notices_manager
-import synapse.server_notices.server_notices_sender
 import synapse.state
 import synapse.storage


 class HomeServer(object):
-    @property
-    def config(self) -> synapse.config.homeserver.HomeServerConfig:
-        pass
-
    def get_auth(self) -> synapse.api.auth.Auth:
        pass

@@ -47,12 +40,6 @@ class HomeServer(object):
    def get_deactivate_account_handler(self) -> synapse.handlers.deactivate_account.DeactivateAccountHandler:
        pass

-    def get_room_creation_handler(self) -> synapse.handlers.room.RoomCreationHandler:
-        pass
-
-    def get_event_creation_handler(self) -> synapse.handlers.message.EventCreationHandler:
-        pass
-
    def get_set_password_handler(self) -> synapse.handlers.set_password.SetPasswordHandler:
        pass

@@ -67,9 +54,3 @@ class HomeServer(object):

    def get_media_repository(self) -> synapse.rest.media.v1.media_repository.MediaRepository:
        pass
-
-    def get_server_notices_manager(self) -> synapse.server_notices.server_notices_manager.ServerNoticesManager:
-        pass
-
-    def get_server_notices_sender(self) -> synapse.server_notices.server_notices_sender.ServerNoticesSender:
-        pass
@@ -1,132 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2018 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-import logging
-
-from six import (iteritems, string_types)
-from twisted.internet import defer
-
-from synapse.api.errors import SynapseError
-from synapse.api.urls import ConsentURIBuilder
-from synapse.config import ConfigError
-from synapse.types import get_localpart_from_id
-
-logger = logging.getLogger(__name__)
-
-
-class ConsentServerNotices(object):
-    """Keeps track of whether we need to send users server_notices about
-    privacy policy consent, and sends one if we do.
-    """
-    def __init__(self, hs):
-        """
-
-        Args:
-            hs (synapse.server.HomeServer):
-        """
-        self._server_notices_manager = hs.get_server_notices_manager()
-        self._store = hs.get_datastore()
-
-        self._users_in_progress = set()
-
-        self._current_consent_version = hs.config.user_consent_version
-        self._server_notice_content = hs.config.user_consent_server_notice_content
-
-        if self._server_notice_content is not None:
-            if not self._server_notices_manager.is_enabled():
-                raise ConfigError(
-                    "user_consent configuration requires server notices, but "
-                    "server notices are not enabled.",
-                )
-            if 'body' not in self._server_notice_content:
-                raise ConfigError(
-                    "user_consent server_notice_consent must contain a 'body' "
-                    "key.",
-                )
-
-            self._consent_uri_builder = ConsentURIBuilder(hs.config)
-
-    @defer.inlineCallbacks
-    def maybe_send_server_notice_to_user(self, user_id):
-        """Check if we need to send a notice to this user, and does so if so
-
-        Args:
-            user_id (str): user to check
-
-        Returns:
-            Deferred
-        """
-        if self._server_notice_content is None:
-            # not enabled
-            return
-
-        # make sure we don't send two messages to the same user at once
-        if user_id in self._users_in_progress:
-            return
-        self._users_in_progress.add(user_id)
-        try:
-            u = yield self._store.get_user_by_id(user_id)
-
-            if u["consent_version"] == self._current_consent_version:
-                # user has already consented
-                return
-
-            if u["consent_server_notice_sent"] == self._current_consent_version:
-                # we've already sent a notice to the user
-                return
-
-            # need to send a message.
-            try:
-                consent_uri = self._consent_uri_builder.build_user_consent_uri(
-                    get_localpart_from_id(user_id),
-                )
-                content = copy_with_str_subst(
-                    self._server_notice_content, {
-                        'consent_uri': consent_uri,
-                    },
-                )
-                yield self._server_notices_manager.send_notice(
-                    user_id, content,
-                )
-                yield self._store.user_set_consent_server_notice_sent(
-                    user_id, self._current_consent_version,
-                )
-            except SynapseError as e:
-                logger.error("Error sending server notice about user consent: %s", e)
-        finally:
-            self._users_in_progress.remove(user_id)
-
-
-def copy_with_str_subst(x, substitutions):
-    """Deep-copy a structure, carrying out string substitions on any strings
-
-    Args:
-        x (object): structure to be copied
-        substitutions (object): substitutions to be made - passed into the
-            string '%' operator
-
-    Returns:
-        copy of x
-    """
-    if isinstance(x, string_types):
-        return x % substitutions
-    if isinstance(x, dict):
-        return {
-            k: copy_with_str_subst(v, substitutions) for (k, v) in iteritems(x)
-        }
-    if isinstance(x, (list, tuple)):
-        return [copy_with_str_subst(y) for y in x]
-
-    # assume it's uninterested and can be shallow-copied.
-    return x
@@ -1,146 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2018 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-import logging
-
-from twisted.internet import defer
-
-from synapse.api.constants import EventTypes, Membership, RoomCreationPreset
-from synapse.types import create_requester
-from synapse.util.caches.descriptors import cachedInlineCallbacks
-
-logger = logging.getLogger(__name__)
-
-
-class ServerNoticesManager(object):
-    def __init__(self, hs):
-        """
-
-        Args:
-            hs (synapse.server.HomeServer):
-        """
-
-        self._store = hs.get_datastore()
-        self._config = hs.config
-        self._room_creation_handler = hs.get_room_creation_handler()
-        self._event_creation_handler = hs.get_event_creation_handler()
-        self._is_mine_id = hs.is_mine_id
-
-    def is_enabled(self):
-        """Checks if server notices are enabled on this server.
-
-        Returns:
-            bool
-        """
-        return self._config.server_notices_mxid is not None
-
-    @defer.inlineCallbacks
-    def send_notice(self, user_id, event_content):
-        """Send a notice to the given user
-
-        Creates the server notices room, if none exists.
-
-        Args:
-            user_id (str): mxid of user to send event to.
-            event_content (dict): content of event to send
-
-        Returns:
-            Deferred[None]
-        """
-        room_id = yield self.get_notice_room_for_user(user_id)
-
-        system_mxid = self._config.server_notices_mxid
-        requester = create_requester(system_mxid)
-
-        logger.info("Sending server notice to %s", user_id)
-
-        yield self._event_creation_handler.create_and_send_nonmember_event(
-            requester, {
-                "type": EventTypes.Message,
-                "room_id": room_id,
-                "sender": system_mxid,
-                "content": event_content,
-            },
-            ratelimit=False,
-        )
-
-    @cachedInlineCallbacks()
-    def get_notice_room_for_user(self, user_id):
-        """Get the room for notices for a given user
-
-        If we have not yet created a notice room for this user, create it
-
-        Args:
-            user_id (str): complete user id for the user we want a room for
-
-        Returns:
-            str: room id of notice room.
-        """
-        if not self.is_enabled():
-            raise Exception("Server notices not enabled")
-
-        assert self._is_mine_id(user_id), \
-            "Cannot send server notices to remote users"
-
-        rooms = yield self._store.get_rooms_for_user_where_membership_is(
-            user_id, [Membership.INVITE, Membership.JOIN],
-        )
-        system_mxid = self._config.server_notices_mxid
-        for room in rooms:
-            # it's worth noting that there is an asymmetry here in that we
-            # expect the user to be invited or joined, but the system user must
-            # be joined. This is kinda deliberate, in that if somebody somehow
-            # manages to invite the system user to a room, that doesn't make it
-            # the server notices room.
-            user_ids = yield self._store.get_users_in_room(room.room_id)
-            if system_mxid in user_ids:
-                # we found a room which our user shares with the system notice
-                # user
-                logger.info("Using room %s", room.room_id)
-                defer.returnValue(room.room_id)
-
-        # apparently no existing notice room: create a new one
-        logger.info("Creating server notices room for %s", user_id)
-
-        # see if we want to override the profile info for the server user.
-        # note that if we want to override either the display name or the
-        # avatar, we have to use both.
-        join_profile = None
-        if (
-            self._config.server_notices_mxid_display_name is not None or
-            self._config.server_notices_mxid_avatar_url is not None
-        ):
-            join_profile = {
-                "displayname": self._config.server_notices_mxid_display_name,
-                "avatar_url": self._config.server_notices_mxid_avatar_url,
-            }
-
-        requester = create_requester(system_mxid)
-        info = yield self._room_creation_handler.create_room(
-            requester,
-            config={
-                "preset": RoomCreationPreset.PRIVATE_CHAT,
-                "name": self._config.server_notices_room_name,
-                "power_level_content_override": {
-                    "users_default": -10,
-                },
-                "invite": (user_id,)
-            },
-            ratelimit=False,
-            creator_join_profile=join_profile,
-        )
-        room_id = info['room_id']
-
-        logger.info("Created server notices room %s for %s", room_id, user_id)
-        defer.returnValue(room_id)
@@ -1,58 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2018 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-from synapse.server_notices.consent_server_notices import ConsentServerNotices
-
-
-class ServerNoticesSender(object):
-    """A centralised place which sends server notices automatically when
-    Certain Events take place
-    """
-    def __init__(self, hs):
-        """
-
-        Args:
-            hs (synapse.server.HomeServer):
-        """
-        # todo: it would be nice to make this more dynamic
-        self._consent_server_notices = ConsentServerNotices(hs)
-
-    def on_user_syncing(self, user_id):
-        """Called when the user performs a sync operation.
-
-        Args:
-            user_id (str): mxid of user who synced
-
-        Returns:
-            Deferred
-        """
-        return self._consent_server_notices.maybe_send_server_notice_to_user(
-            user_id,
-        )
-
-    def on_user_ip(self, user_id):
-        """Called on the master when a worker process saw a client request.
-
-        Args:
-            user_id (str): mxid
-
-        Returns:
-            Deferred
-        """
-        # The synchrotrons use a stubbed version of ServerNoticesSender, so
-        # we check for notices to send to the user in on_user_ip as well as
-        # in on_user_syncing
-        return self._consent_server_notices.maybe_send_server_notice_to_user(
-            user_id,
-        )
@@ -1,46 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2018 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-from twisted.internet import defer
-
-
-class WorkerServerNoticesSender(object):
-    """Stub impl of ServerNoticesSender which does nothing"""
-    def __init__(self, hs):
-        """
-        Args:
-            hs (synapse.server.HomeServer):
-        """
-
-    def on_user_syncing(self, user_id):
-        """Called when the user performs a sync operation.
-
-        Args:
-            user_id (str): mxid of user who synced
-
-        Returns:
-            Deferred
-        """
-        return defer.succeed(None)
-
-    def on_user_ip(self, user_id):
-        """Called on the master when a worker process saw a client request.
-
-        Args:
-            user_id (str): mxid
-
-        Returns:
-            Deferred
-        """
-        raise AssertionError("on_user_ip unexpectedly called on worker")
@@ -32,8 +32,6 @@ from frozendict import frozendict
 import logging
 import hashlib

-from six import iteritems, itervalues
-
 logger = logging.getLogger(__name__)


@@ -134,7 +132,7 @@ class StateHandler(object):

        state_map = yield self.store.get_events(state.values(), get_prev_content=False)
        state = {
-            key: state_map[e_id] for key, e_id in iteritems(state) if e_id in state_map
+            key: state_map[e_id] for key, e_id in state.iteritems() if e_id in state_map
        }

        defer.returnValue(state)
@@ -340,7 +338,7 @@ class StateHandler(object):
        )

        if len(state_groups_ids) == 1:
-            name, state_list = list(state_groups_ids.items()).pop()
+            name, state_list = state_groups_ids.items().pop()

            prev_group, delta_ids = yield self.store.get_state_group_delta(name)

@@ -380,7 +378,7 @@ class StateHandler(object):
            new_state = resolve_events_with_state_map(state_set_ids, state_map)

        new_state = {
-            key: state_map[ev_id] for key, ev_id in iteritems(new_state)
+            key: state_map[ev_id] for key, ev_id in new_state.iteritems()
        }

        return new_state
@@ -460,15 +458,15 @@ class StateResolutionHandler(object):
            # build a map from state key to the event_ids which set that state.
            # dict[(str, str), set[str])
            state = {}
-            for st in itervalues(state_groups_ids):
-                for key, e_id in iteritems(st):
+            for st in state_groups_ids.itervalues():
+                for key, e_id in st.iteritems():
                    state.setdefault(key, set()).add(e_id)

            # build a map from state key to the event_ids which set that state,
            # including only those where there are state keys in conflict.
            conflicted_state = {
                k: list(v)
-                for k, v in iteritems(state)
+                for k, v in state.iteritems()
                if len(v) > 1
            }

@@ -476,13 +474,13 @@ class StateResolutionHandler(object):
                logger.info("Resolving conflicted state for %r", room_id)
                with Measure(self.clock, "state._resolve_events"):
                    new_state = yield resolve_events_with_factory(
-                        list(state_groups_ids.values()),
+                        state_groups_ids.values(),
                        event_map=event_map,
                        state_map_factory=state_map_factory,
                    )
            else:
                new_state = {
-                    key: e_ids.pop() for key, e_ids in iteritems(state)
+                    key: e_ids.pop() for key, e_ids in state.iteritems()
                }

            with Measure(self.clock, "state.create_group_ids"):
@@ -491,8 +489,8 @@ class StateResolutionHandler(object):
                # which will be used as a cache key for future resolutions, but
                # not get persisted.
                state_group = None
-                new_state_event_ids = frozenset(itervalues(new_state))
-                for sg, events in iteritems(state_groups_ids):
+                new_state_event_ids = frozenset(new_state.itervalues())
+                for sg, events in state_groups_ids.iteritems():
                    if new_state_event_ids == frozenset(e_id for e_id in events):
                        state_group = sg
                        break
@@ -503,11 +501,11 @@ class StateResolutionHandler(object):

                prev_group = None
                delta_ids = None
-                for old_group, old_ids in iteritems(state_groups_ids):
+                for old_group, old_ids in state_groups_ids.iteritems():
                    if not set(new_state) - set(old_ids):
                        n_delta_ids = {
                            k: v
-                            for k, v in iteritems(new_state)
+                            for k, v in new_state.iteritems()
                            if old_ids.get(k) != v
                        }
                        if not delta_ids or len(n_delta_ids) < len(delta_ids):
@@ -529,7 +527,7 @@ class StateResolutionHandler(object):

 def _ordered_events(events):
    def key_func(e):
-        return -int(e.depth), hashlib.sha1(e.event_id.encode()).hexdigest()
+        return -int(e.depth), hashlib.sha1(e.event_id).hexdigest()

    return sorted(events, key=key_func)

@@ -586,7 +584,7 @@ def _seperate(state_sets):
    conflicted_state = {}

    for state_set in state_sets[1:]:
-        for key, value in iteritems(state_set):
+        for key, value in state_set.iteritems():
            # Check if there is an unconflicted entry for the state key.
            unconflicted_value = unconflicted_state.get(key)
            if unconflicted_value is None:
@@ -642,7 +640,7 @@ def resolve_events_with_factory(state_sets, event_map, state_map_factory):

    needed_events = set(
        event_id
-        for event_ids in itervalues(conflicted_state)
+        for event_ids in conflicted_state.itervalues()
        for event_id in event_ids
    )
    if event_map is not None:
@@ -664,7 +662,7 @@ def resolve_events_with_factory(state_sets, event_map, state_map_factory):
        unconflicted_state, conflicted_state, state_map
    )

-    new_needed_events = set(itervalues(auth_events))
+    new_needed_events = set(auth_events.itervalues())
    new_needed_events -= needed_events
    if event_map is not None:
        new_needed_events -= set(event_map.iterkeys())
@@ -681,7 +679,7 @@ def resolve_events_with_factory(state_sets, event_map, state_map_factory):

 def _create_auth_events_from_maps(unconflicted_state, conflicted_state, state_map):
    auth_events = {}
-    for event_ids in itervalues(conflicted_state):
+    for event_ids in conflicted_state.itervalues():
        for event_id in event_ids:
            if event_id in state_map:
                keys = event_auth.auth_types_for_event(state_map[event_id])
@@ -696,7 +694,7 @@ def _create_auth_events_from_maps(unconflicted_state, conflicted_state, state_ma
 def _resolve_with_state(unconflicted_state_ids, conflicted_state_ds, auth_event_ids,
                        state_map):
    conflicted_state = {}
-    for key, event_ids in iteritems(conflicted_state_ds):
+    for key, event_ids in conflicted_state_ds.iteritems():
        events = [state_map[ev_id] for ev_id in event_ids if ev_id in state_map]
        if len(events) > 1:
            conflicted_state[key] = events
@@ -705,7 +703,7 @@ def _resolve_with_state(unconflicted_state_ids, conflicted_state_ds, auth_event_

    auth_events = {
        key: state_map[ev_id]
-        for key, ev_id in iteritems(auth_event_ids)
+        for key, ev_id in auth_event_ids.iteritems()
        if ev_id in state_map
    }

@@ -718,7 +716,7 @@ def _resolve_with_state(unconflicted_state_ids, conflicted_state_ds, auth_event_
        raise

    new_state = unconflicted_state_ids
-    for key, event in iteritems(resolved_state):
+    for key, event in resolved_state.iteritems():
        new_state[key] = event.event_id

    return new_state
@@ -743,7 +741,7 @@ def _resolve_state_events(conflicted_state, auth_events):

    auth_events.update(resolved_state)

-    for key, events in iteritems(conflicted_state):
+    for key, events in conflicted_state.iteritems():
        if key[0] == EventTypes.JoinRules:
            logger.debug("Resolving conflicted join rules %r", events)
            resolved_state[key] = _resolve_auth_events(
@@ -753,7 +751,7 @@ def _resolve_state_events(conflicted_state, auth_events):

    auth_events.update(resolved_state)

-    for key, events in iteritems(conflicted_state):
+    for key, events in conflicted_state.iteritems():
        if key[0] == EventTypes.Member:
            logger.debug("Resolving conflicted member lists %r", events)
            resolved_state[key] = _resolve_auth_events(
@@ -763,7 +761,7 @@ def _resolve_state_events(conflicted_state, auth_events):

    auth_events.update(resolved_state)

-    for key, events in iteritems(conflicted_state):
+    for key, events in conflicted_state.iteritems():
        if key not in resolved_state:
            logger.debug("Resolving conflicted state %r:%r", key, events)
            resolved_state[key] = _resolve_normal_events(
@@ -14,11 +14,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-import datetime
-from dateutil import tz
-import time
-import logging
-
 from synapse.storage.devices import DeviceStore
 from .appservice import (
    ApplicationServiceStore, ApplicationServiceTransactionStore
@@ -60,6 +55,10 @@ from .engines import PostgresEngine
 from synapse.api.constants import PresenceState
 from synapse.util.caches.stream_change_cache import StreamChangeCache

+
+import logging
+
+
 logger = logging.getLogger(__name__)


@@ -131,7 +130,6 @@ class DataStore(RoomMemberStore, RoomStore,
        self._group_updates_id_gen = StreamIdGenerator(
            db_conn, "local_group_updates", "stream_id",
        )
-        self._chunk_id_gen = IdGenerator(db_conn, "events", "chunk_id")

        if isinstance(self.database_engine, PostgresEngine):
            self._cache_id_gen = StreamIdGenerator(
@@ -215,9 +213,6 @@ class DataStore(RoomMemberStore, RoomStore,
        self._stream_order_on_start = self.get_room_max_stream_ordering()
        self._min_stream_order_on_start = self.get_room_min_stream_ordering()

-        # Used in _generate_user_daily_visits to keep track of progress
-        self._last_user_visit_update = self._get_start_of_day()
-
        super(DataStore, self).__init__(db_conn, hs)

    def take_presence_startup_info(self):
@@ -352,69 +347,6 @@ class DataStore(RoomMemberStore, RoomStore,

        return self.runInteraction("count_r30_users", _count_r30_users)

-    def _get_start_of_day(self):
-        """
-        Returns millisecond unixtime for start of UTC day.
-        """
-        now = datetime.datetime.utcnow()
-        today_start = datetime.datetime(now.year, now.month,
-                                        now.day, tzinfo=tz.tzutc())
-        return int(time.mktime(today_start.timetuple())) * 1000
-
-    def generate_user_daily_visits(self):
-        """
-        Generates daily visit data for use in cohort/ retention analysis
-        """
-        def _generate_user_daily_visits(txn):
-            logger.info("Calling _generate_user_daily_visits")
-            today_start = self._get_start_of_day()
-            a_day_in_milliseconds = 24 * 60 * 60 * 1000
-            now = self.clock.time_msec()
-
-            sql = """
-                INSERT INTO user_daily_visits (user_id, device_id, timestamp)
-                    SELECT u.user_id, u.device_id, ?
-                    FROM user_ips AS u
-                    LEFT JOIN (
-                      SELECT user_id, device_id, timestamp FROM user_daily_visits
-                      WHERE timestamp = ?
-                    ) udv
-                    ON u.user_id = udv.user_id AND u.device_id=udv.device_id
-                    INNER JOIN users ON users.name=u.user_id
-                    WHERE last_seen > ? AND last_seen <= ?
-                    AND udv.timestamp IS NULL AND users.is_guest=0
-                    AND users.appservice_id IS NULL
-                    GROUP BY u.user_id, u.device_id
-            """
-
-            # This means that the day has rolled over but there could still
-            # be entries from the previous day. There is an edge case
-            # where if the user logs in at 23:59 and overwrites their
-            # last_seen at 00:01 then they will not be counted in the
-            # previous day's stats - it is important that the query is run
-            # often to minimise this case.
-            if today_start > self._last_user_visit_update:
-                yesterday_start = today_start - a_day_in_milliseconds
-                txn.execute(sql, (
-                    yesterday_start, yesterday_start,
-                    self._last_user_visit_update, today_start
-                ))
-                self._last_user_visit_update = today_start
-
-            txn.execute(sql, (
-                today_start, today_start,
-                self._last_user_visit_update,
-                now
-            ))
-            # Update _last_user_visit_update to now. The reason to do this
-            # rather just clamping to the beginning of the day is to limit
-            # the size of the join - meaning that the query can be run more
-            # frequently
-            self._last_user_visit_update = now
-
-        return self.runInteraction("generate_user_daily_visits",
-                                   _generate_user_daily_visits)
-
    def get_users(self):
        """Function to reterive a list of users in users table.

@@ -27,17 +27,9 @@ import sys
 import time
 import threading

-from six import itervalues, iterkeys, iteritems
-from six.moves import intern, range

 logger = logging.getLogger(__name__)

-try:
-    MAX_TXN_ID = sys.maxint - 1
-except AttributeError:
-    # python 3 does not have a maximum int value
-    MAX_TXN_ID = 2**63 - 1
-
 sql_logger = logging.getLogger("synapse.storage.SQL")
 transaction_logger = logging.getLogger("synapse.storage.txn")
 perf_logger = logging.getLogger("synapse.storage.TIME")
@@ -145,7 +137,7 @@ class PerformanceCounters(object):

    def interval(self, interval_duration, limit=3):
        counters = []
-        for name, (count, cum_time) in iteritems(self.current_counters):
+        for name, (count, cum_time) in self.current_counters.iteritems():
            prev_count, prev_time = self.previous_counters.get(name, (0, 0))
            counters.append((
                (cum_time - prev_time) / interval_duration,
@@ -230,7 +222,7 @@ class SQLBaseStore(object):

        # We don't really need these to be unique, so lets stop it from
        # growing really large.
-        self._TXN_ID = (self._TXN_ID + 1) % (MAX_TXN_ID)
+        self._TXN_ID = (self._TXN_ID + 1) % (sys.maxint - 1)

        name = "%s-%x" % (desc, txn_id, )

@@ -551,7 +543,7 @@ class SQLBaseStore(object):
            ", ".join("%s = ?" % (k,) for k in values),
            " AND ".join("%s = ?" % (k,) for k in keyvalues)
        )
-        sqlargs = list(values.values()) + list(keyvalues.values())
+        sqlargs = values.values() + keyvalues.values()

        txn.execute(sql, sqlargs)
        if txn.rowcount > 0:
@@ -569,7 +561,7 @@ class SQLBaseStore(object):
            ", ".join(k for k in allvalues),
            ", ".join("?" for _ in allvalues)
        )
-        txn.execute(sql, list(allvalues.values()))
+        txn.execute(sql, allvalues.values())
        # successfully inserted
        return True

@@ -637,8 +629,8 @@ class SQLBaseStore(object):
        }

        if keyvalues:
-            sql += " WHERE %s" % " AND ".join("%s = ?" % k for k in iterkeys(keyvalues))
-            txn.execute(sql, list(keyvalues.values()))
+            sql += " WHERE %s" % " AND ".join("%s = ?" % k for k in keyvalues.iterkeys())
+            txn.execute(sql, keyvalues.values())
        else:
            txn.execute(sql)

@@ -702,7 +694,7 @@ class SQLBaseStore(object):
                table,
                " AND ".join("%s = ?" % (k, ) for k in keyvalues)
            )
-            txn.execute(sql, list(keyvalues.values()))
+            txn.execute(sql, keyvalues.values())
        else:
            sql = "SELECT %s FROM %s" % (
                ", ".join(retcols),
@@ -733,12 +725,9 @@ class SQLBaseStore(object):
        if not iterable:
            defer.returnValue(results)

-        # iterables can not be sliced, so convert it to a list first
-        it_list = list(iterable)
-
        chunks = [
-            it_list[i:i + batch_size]
-            for i in range(0, len(it_list), batch_size)
+            iterable[i:i + batch_size]
+            for i in xrange(0, len(iterable), batch_size)
        ]
        for chunk in chunks:
            rows = yield self.runInteraction(
@@ -778,7 +767,7 @@ class SQLBaseStore(object):
        )
        values.extend(iterable)

-        for key, value in iteritems(keyvalues):
+        for key, value in keyvalues.iteritems():
            clauses.append("%s = ?" % (key,))
            values.append(value)

@@ -801,7 +790,7 @@ class SQLBaseStore(object):
    @staticmethod
    def _simple_update_txn(txn, table, keyvalues, updatevalues):
        if keyvalues:
-            where = "WHERE %s" % " AND ".join("%s = ?" % k for k in iterkeys(keyvalues))
+            where = "WHERE %s" % " AND ".join("%s = ?" % k for k in keyvalues.iterkeys())
        else:
            where = ""

@@ -813,7 +802,7 @@ class SQLBaseStore(object):

        txn.execute(
            update_sql,
-            list(updatevalues.values()) + list(keyvalues.values())
+            updatevalues.values() + keyvalues.values()
        )

        return txn.rowcount
@@ -861,7 +850,7 @@ class SQLBaseStore(object):
            " AND ".join("%s = ?" % (k,) for k in keyvalues)
        )

-        txn.execute(select_sql, list(keyvalues.values()))
+        txn.execute(select_sql, keyvalues.values())

        row = txn.fetchone()
        if not row:
@@ -899,7 +888,7 @@ class SQLBaseStore(object):
            " AND ".join("%s = ?" % (k, ) for k in keyvalues)
        )

-        txn.execute(sql, list(keyvalues.values()))
+        txn.execute(sql, keyvalues.values())
        if txn.rowcount == 0:
            raise StoreError(404, "No row found")
        if txn.rowcount > 1:
@@ -917,7 +906,7 @@ class SQLBaseStore(object):
            " AND ".join("%s = ?" % (k, ) for k in keyvalues)
        )

-        return txn.execute(sql, list(keyvalues.values()))
+        return txn.execute(sql, keyvalues.values())

    def _simple_delete_many(self, table, column, iterable, keyvalues, desc):
        return self.runInteraction(
@@ -949,7 +938,7 @@ class SQLBaseStore(object):
        )
        values.extend(iterable)

-        for key, value in iteritems(keyvalues):
+        for key, value in keyvalues.iteritems():
            clauses.append("%s = ?" % (key,))
            values.append(value)

@@ -989,7 +978,7 @@ class SQLBaseStore(object):
        txn.close()

        if cache:
-            min_val = min(itervalues(cache))
+            min_val = min(cache.itervalues())
        else:
            min_val = max_value

@@ -1104,7 +1093,7 @@ class SQLBaseStore(object):
                " AND ".join("%s = ?" % (k,) for k in keyvalues),
                " ? ASC LIMIT ? OFFSET ?"
            )
-            txn.execute(sql, list(keyvalues.values()) + list(pagevalues))
+            txn.execute(sql, keyvalues.values() + pagevalues)
        else:
            sql = "SELECT %s FROM %s ORDER BY %s" % (
                ", ".join(retcols),
@@ -1,319 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2018 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import math
-import logging
-
-from synapse.storage._base import SQLBaseStore
-from synapse.util.katriel_bodlaender import OrderedListStore
-from synapse.util.metrics import Measure
-
-import synapse.metrics
-
-metrics = synapse.metrics.get_metrics_for(__name__)
-rebalance_counter = metrics.register_counter("rebalances")
-
-
-logger = logging.getLogger(__name__)
-
-
-class ChunkDBOrderedListStore(OrderedListStore):
-    """Used as the list store for room chunks, efficiently maintaining them in
-    topological order on updates.
-
-    A room chunk is a connected portion of the room events DAG. Chunks are
-    constructed so that they have the additional property that for all events in
-    the chunk, either all of their prev_events are in that chunk or none of them
-    are. This ensures that no event that is subsequently received needs to be
-    inserted into the middle of a chunk, since it cannot both reference an event
-    in the chunk and be referenced by an event in the chunk (assuming no
-    cycles).
-
-    As such the set of chunks in a room inherits a DAG, i.e. if an event in one
-    chunk references an event in a second chunk, then we say that the first
-    chunk references the second, and thus forming a DAG. (This means that chunks
-    start off disconnected until an event is received that connects the two
-    chunks.)
-
-    We can therefore end up with multiple chunks in a room when the server
-    misses some events, e.g. due to the server being offline for a time.
-
-    The server may only have a subset of all events in a room, in which case
-    its possible for the server to have chunks that are unconnected from each
-    other. The ordering between unconnected chunks is arbitrary.
-
-    The class is designed for use inside transactions and so takes a
-    transaction object in the constructor. This means that it needs to be
-    re-instantiated in each transaction, so all state needs to be stored
-    in the database.
-
-    Internally the ordering is implemented using floats, and the average is
-    taken when a node is inserted between other nodes. To avoid precision
-    errors a minimum difference between sucessive orderings is attempted to be
-    kept; whenever the difference is too small we attempt to rebalance. See
-    the `_rebalance` function for implementation details.
-
-    Note that OrderedListStore orders nodes such that source of an edge
-    comes before the target. This is counter intuitive when edges represent
-    causality, so for the purposes of ordering algorithm we invert the edge
-    directions, i.e. if chunk A has a prev chunk of B then we say that the
-    edge is from B to A. This ensures that newer chunks get inserted at the
-    end (rather than the start).
-
-    Note: Calls to `add_node` and `add_edge` cannot overlap for the same room,
-    and so callers should perform some form of per-room locking when using
-    this class.
-
-    Args:
-        txn
-        room_id (str)
-        clock
-        rebalance_digits (int): When a rebalance is triggered we rebalance
-            in a range around the node, where the bounds are rounded to this
-            number of digits.
-        min_difference (int): A rebalance is triggered when the difference
-            between two successive orderings is less than the reciprocal of
-            this.
-    """
-    def __init__(self,
-                 txn, room_id, clock,
-                 rebalance_digits=3,
-                 min_difference=1000000):
-        self.txn = txn
-        self.room_id = room_id
-        self.clock = clock
-
-        self.rebalance_digits = rebalance_digits
-        self.min_difference = 1. / min_difference
-
-    def is_before(self, a, b):
-        """Implements OrderedListStore"""
-        return self._get_order(a) < self._get_order(b)
-
-    def get_prev(self, node_id):
-        """Implements OrderedListStore"""
-        order = self._get_order(node_id)
-
-        sql = """
-            SELECT chunk_id FROM chunk_linearized
-            WHERE ordering < ? AND room_id = ?
-            ORDER BY ordering DESC
-            LIMIT 1
-        """
-
-        self.txn.execute(sql, (order, self.room_id,))
-
-        row = self.txn.fetchone()
-        if row:
-            return row[0]
-        return None
-
-    def get_next(self, node_id):
-        """Implements OrderedListStore"""
-        order = self._get_order(node_id)
-
-        sql = """
-            SELECT chunk_id FROM chunk_linearized
-            WHERE ordering > ? AND room_id = ?
-            ORDER BY ordering ASC
-            LIMIT 1
-        """
-
-        self.txn.execute(sql, (order, self.room_id,))
-
-        row = self.txn.fetchone()
-        if row:
-            return row[0]
-        return None
-
-    def _insert_before(self, node_id, target_id):
-        """Implements OrderedListStore"""
-
-        rebalance = False  # Set to true if we need to trigger a rebalance
-
-        if target_id:
-            target_order = self._get_order(target_id)
-            before_id = self.get_prev(target_id)
-
-            if before_id:
-                before_order = self._get_order(before_id)
-                new_order = (target_order + before_order) / 2.
-
-                rebalance = math.fabs(target_order - before_order) < self.min_difference
-            else:
-                new_order = math.floor(target_order) - 1
-        else:
-            # If target_id is None then we insert at the end.
-            self.txn.execute("""
-                SELECT COALESCE(MAX(ordering), 0) + 1
-                FROM chunk_linearized
-                WHERE room_id = ?
-            """, (self.room_id,))
-
-            new_order, = self.txn.fetchone()
-
-        self._insert(node_id, new_order)
-
-        if rebalance:
-            self._rebalance(node_id)
-
-    def _insert_after(self, node_id, target_id):
-        """Implements OrderedListStore"""
-
-        rebalance = False  # Set to true if we need to trigger a rebalance
-
-        if target_id:
-            target_order = self._get_order(target_id)
-            after_id = self.get_next(target_id)
-            if after_id:
-                after_order = self._get_order(after_id)
-                new_order = (target_order + after_order) / 2.
-
-                rebalance = math.fabs(target_order - after_order) < self.min_difference
-            else:
-                new_order = math.ceil(target_order) + 1
-        else:
-            # If target_id is None then we insert at the start.
-            self.txn.execute("""
-                SELECT COALESCE(MIN(ordering), 0) - 1
-                FROM chunk_linearized
-                WHERE room_id = ?
-            """, (self.room_id,))
-
-            new_order, = self.txn.fetchone()
-
-        self._insert(node_id, new_order)
-
-        if rebalance:
-            self._rebalance(node_id)
-
-    def get_nodes_with_edges_to(self, node_id):
-        """Implements OrderedListStore"""
-
-        # Note that we use the inverse relation here
-        sql = """
-            SELECT l.ordering, l.chunk_id FROM chunk_graph AS g
-            INNER JOIN chunk_linearized AS l ON g.prev_id = l.chunk_id
-            WHERE g.chunk_id = ?
-        """
-        self.txn.execute(sql, (node_id,))
-        return self.txn.fetchall()
-
-    def get_nodes_with_edges_from(self, node_id):
-        """Implements OrderedListStore"""
-
-        # Note that we use the inverse relation here
-        sql = """
-            SELECT l.ordering, l.chunk_id FROM chunk_graph AS g
-            INNER JOIN chunk_linearized AS l ON g.chunk_id = l.chunk_id
-            WHERE g.prev_id = ?
-        """
-        self.txn.execute(sql, (node_id,))
-        return self.txn.fetchall()
-
-    def _delete_ordering(self, node_id):
-        """Implements OrderedListStore"""
-
-        SQLBaseStore._simple_delete_txn(
-            self.txn,
-            table="chunk_linearized",
-            keyvalues={"chunk_id": node_id},
-        )
-
-    def _add_edge_to_graph(self, source_id, target_id):
-        """Implements OrderedListStore"""
-
-        # Note that we use the inverse relation
-        SQLBaseStore._simple_insert_txn(
-            self.txn,
-            table="chunk_graph",
-            values={"chunk_id": target_id, "prev_id": source_id}
-        )
-
-    def _insert(self, node_id, order):
-        """Inserts the node with the given ordering.
-        """
-        SQLBaseStore._simple_insert_txn(
-            self.txn,
-            table="chunk_linearized",
-            values={
-                "chunk_id": node_id,
-                "room_id": self.room_id,
-                "ordering": order,
-            }
-        )
-
-    def _get_order(self, node_id):
-        """Get the ordering of the given node.
-        """
-
-        return SQLBaseStore._simple_select_one_onecol_txn(
-            self.txn,
-            table="chunk_linearized",
-            keyvalues={"chunk_id": node_id},
-            retcol="ordering"
-        )
-
-    def _rebalance(self, node_id):
-        """Rebalances the list around the given node to ensure that the
-        ordering floats don't get too small.
-
-        This works by finding a range that includes the given node, and
-        recalculating the ordering floats such that they're equidistant in
-        that range.
-        """
-
-        logger.info("Rebalancing room %s, chunk %s", self.room_id, node_id)
-
-        with Measure(self.clock, "chunk_rebalance"):
-            # We pick the interval to try and minimise the number of decimal
-            # places, i.e. we round to nearest float with `rebalance_digits` and
-            # use that as one side of the interval
-            order = self._get_order(node_id)
-            a = round(order, self.rebalance_digits)
-            min_order = a - 10 ** -self.rebalance_digits
-            max_order = a + 10 ** -self.rebalance_digits
-
-            # Now we get all the nodes in the range. We add the minimum difference
-            # to the bounds to ensure that we don't accidentally move a node to be
-            # within the minimum difference of a node outside the range.
-            sql = """
-                SELECT chunk_id FROM chunk_linearized
-                WHERE ordering >= ? AND ordering <= ? AND room_id = ?
-            """
-            self.txn.execute(sql, (
-                min_order - self.min_difference,
-                max_order + self.min_difference,
-                self.room_id,
-            ))
-
-            chunk_ids = [c for c, in self.txn]
-
-            sql = """
-                UPDATE chunk_linearized
-                SET ordering = ?
-                WHERE chunk_id = ?
-            """
-
-            step = (max_order - min_order) / len(chunk_ids)
-            self.txn.executemany(
-                sql,
-                (
-                    ((idx * step + min_order), chunk_id)
-                    for idx, chunk_id in enumerate(chunk_ids)
-                )
-            )
-
-            rebalance_counter.inc()
@@ -22,8 +22,6 @@ from . import background_updates

 from synapse.util.caches import CACHE_SIZE_FACTOR

-from six import iteritems
-

 logger = logging.getLogger(__name__)

@@ -57,13 +55,6 @@ class ClientIpStore(background_updates.BackgroundUpdateStore):
            columns=["user_id", "last_seen"],
        )

-        self.register_background_index_update(
-            "user_ips_last_seen_only_index",
-            index_name="user_ips_last_seen_only",
-            table="user_ips",
-            columns=["last_seen"],
-        )
-
        # (user_id, access_token, ip) -> (user_agent, device_id, last_seen)
        self._batch_row_update = {}

@@ -101,7 +92,7 @@ class ClientIpStore(background_updates.BackgroundUpdateStore):
    def _update_client_ips_batch_txn(self, txn, to_update):
        self.database_engine.lock_table(txn, "user_ips")

-        for entry in iteritems(to_update):
+        for entry in to_update.iteritems():
            (user_id, access_token, ip), (user_agent, device_id, last_seen) = entry

            self._simple_upsert_txn(
@@ -233,5 +224,5 @@ class ClientIpStore(background_updates.BackgroundUpdateStore):
                "user_agent": user_agent,
                "last_seen": last_seen,
            }
-            for (access_token, ip), (user_agent, last_seen) in iteritems(results)
+            for (access_token, ip), (user_agent, last_seen) in results.iteritems()
        ))
@@ -21,7 +21,6 @@ from synapse.api.errors import StoreError
 from ._base import SQLBaseStore, Cache
 from synapse.util.caches.descriptors import cached, cachedList, cachedInlineCallbacks

-from six import itervalues, iteritems

 logger = logging.getLogger(__name__)

@@ -361,7 +360,7 @@ class DeviceStore(SQLBaseStore):
            return (now_stream_id, [])

        if len(query_map) >= 20:
-            now_stream_id = max(stream_id for stream_id in itervalues(query_map))
+            now_stream_id = max(stream_id for stream_id in query_map.itervalues())

        devices = self._get_e2e_device_keys_txn(
            txn, query_map.keys(), include_all_devices=True
@@ -374,13 +373,13 @@ class DeviceStore(SQLBaseStore):
        """

        results = []
-        for user_id, user_devices in iteritems(devices):
+        for user_id, user_devices in devices.iteritems():
            # The prev_id for the first row is always the last row before
            # `from_stream_id`
            txn.execute(prev_sent_id_sql, (destination, user_id, from_stream_id))
            rows = txn.fetchall()
            prev_id = rows[0][0]
-            for device_id, device in iteritems(user_devices):
+            for device_id, device in user_devices.iteritems():
                stream_id = query_map[(user_id, device_id)]
                result = {
                    "user_id": user_id,
@@ -484,7 +483,7 @@ class DeviceStore(SQLBaseStore):
        if devices:
            user_devices = devices[user_id]
            results = []
-            for device_id, device in iteritems(user_devices):
+            for device_id, device in user_devices.iteritems():
                result = {
                    "device_id": device_id,
                }
@@ -21,8 +21,6 @@ import simplejson as json

 from ._base import SQLBaseStore

-from six import iteritems
-

 class EndToEndKeyStore(SQLBaseStore):
    def set_e2e_device_keys(self, user_id, device_id, time_now, device_keys):
@@ -83,8 +81,8 @@ class EndToEndKeyStore(SQLBaseStore):
            query_list, include_all_devices,
        )

-        for user_id, device_keys in iteritems(results):
-            for device_id, device_info in iteritems(device_keys):
+        for user_id, device_keys in results.iteritems():
+            for device_id, device_info in device_keys.iteritems():
                device_info["keys"] = json.loads(device_info.pop("key_json"))

        defer.returnValue(results)
--- a/Show More
+++ b/Show More