Support pagination for tokens without chunk part

Implement pagination using chunks

.gitignore

@@ -1,6 +1,5 @@
 *.pyc
 .*.swp
-*~
 
 .DS_Store
 _trial_temp/
@@ -14,7 +13,6 @@ docs/build/
 cmdclient_config.json
 homeserver*.db
 homeserver*.log
-homeserver*.log.*
 homeserver*.pid
 homeserver*.yaml
 
@@ -42,8 +40,6 @@ media_store/
 *.tac
 
 build/
-venv/
-venv*/
 
 localhost-800*/
 static/client/register/register_config.js

.travis.yml

@@ -4,12 +4,7 @@ language: python
 # tell travis to cache ~/.cache/pip
 cache: pip
 
-before_script:
-  - git remote set-branches --add origin develop
-  - git fetch origin develop
-
 matrix:
-  fast_finish: true
   include:
   - python: 2.7
     env: TOX_ENV=packaging
@@ -19,16 +14,10 @@ matrix:
 
   - python: 2.7
     env: TOX_ENV=py27
-
+    
   - python: 3.6
     env: TOX_ENV=py36
 
-  - python: 3.6
-    env: TOX_ENV=check_isort
-
-  - python: 3.6
-    env: TOX_ENV=check-newsfragment
-
 install:
   - pip install tox
 

CHANGES.rst

@@ -1,196 +1,9 @@
-Synapse 0.33.1 (2018-08-02)
-===========================
+Changes in <unreleased>
+=======================
 
-SECURITY FIXES
---------------
-
-- Fix a potential issue where servers could request events for rooms they have not joined. (`#3641 <https://github.com/matrix-org/synapse/issues/3641>`_)
-- Fix a potential issue where users could see events in private rooms before they joined. (`#3642 <https://github.com/matrix-org/synapse/issues/3642>`_)
-
-
-Synapse 0.33.0 (2018-07-19)
-===========================
-
-Bugfixes
---------
-
-- Disable a noisy warning about logcontexts. (`#3561 <https://github.com/matrix-org/synapse/issues/3561>`_)
-
-
-Synapse 0.33.0rc1 (2018-07-18)
-==============================
-
-Features
---------
-
-- Enforce the specified API for report_event. (`#3316 <https://github.com/matrix-org/synapse/issues/3316>`_)
-- Include CPU time from database threads in request/block metrics. (`#3496 <https://github.com/matrix-org/synapse/issues/3496>`_, `#3501 <https://github.com/matrix-org/synapse/issues/3501>`_)
-- Add CPU metrics for _fetch_event_list. (`#3497 <https://github.com/matrix-org/synapse/issues/3497>`_)
-- Optimisation to make handling incoming federation requests more efficient. (`#3541 <https://github.com/matrix-org/synapse/issues/3541>`_)
-
-
-Bugfixes
---------
-
-- Fix a significant performance regression in /sync. (`#3505 <https://github.com/matrix-org/synapse/issues/3505>`_, `#3521 <https://github.com/matrix-org/synapse/issues/3521>`_, `#3530 <https://github.com/matrix-org/synapse/issues/3530>`_, `#3544 <https://github.com/matrix-org/synapse/issues/3544>`_)
-- Use more portable syntax in our use of the attrs package, widening the supported versions. (`#3498 <https://github.com/matrix-org/synapse/issues/3498>`_)
-- Fix queued federation requests being processed in the wrong order. (`#3533 <https://github.com/matrix-org/synapse/issues/3533>`_)
-- Ensure that erasure requests are correctly honoured for publicly accessible rooms when accessed over federation. (`#3546 <https://github.com/matrix-org/synapse/issues/3546>`_)
-
-
-Misc
-----
-
-- Refactoring to improve testability. (`#3351 <https://github.com/matrix-org/synapse/issues/3351>`_, `#3499 <https://github.com/matrix-org/synapse/issues/3499>`_)
-- Use ``isort`` to sort imports. (`#3463 <https://github.com/matrix-org/synapse/issues/3463>`_, `#3464 <https://github.com/matrix-org/synapse/issues/3464>`_, `#3540 <https://github.com/matrix-org/synapse/issues/3540>`_)
-- Use parse and asserts from http.servlet. (`#3534 <https://github.com/matrix-org/synapse/issues/3534>`_, `#3535 <https://github.com/matrix-org/synapse/issues/3535>`_).
-
-
-Synapse 0.32.2 (2018-07-07)
-===========================
-
-Bugfixes
---------
-
-- Amend the Python dependencies to depend on attrs from PyPI, not attr (`#3492 <https://github.com/matrix-org/synapse/issues/3492>`_)
-
-
-Synapse 0.32.1 (2018-07-06)
-===========================
-
-Bugfixes
---------
-
-- Add explicit dependency on netaddr (`#3488 <https://github.com/matrix-org/synapse/issues/3488>`_)
-
-
-Changes in synapse v0.32.0 (2018-07-06)
-===========================================
-No changes since 0.32.0rc1
-
-Synapse 0.32.0rc1 (2018-07-05)
-==============================
-
-Features
---------
-
-- Add blacklist & whitelist of servers allowed to send events to a room via ``m.room.server_acl`` event.
-- Cache factor override system for specific caches (`#3334 <https://github.com/matrix-org/synapse/issues/3334>`_)
-- Add metrics to track appservice transactions (`#3344 <https://github.com/matrix-org/synapse/issues/3344>`_)
-- Try to log more helpful info when a sig verification fails (`#3372 <https://github.com/matrix-org/synapse/issues/3372>`_)
-- Synapse now uses the best performing JSON encoder/decoder according to your runtime (simplejson on CPython, stdlib json on PyPy). (`#3462 <https://github.com/matrix-org/synapse/issues/3462>`_)
-- Add optional ip_range_whitelist param to AS registration files to lock AS IP access (`#3465 <https://github.com/matrix-org/synapse/issues/3465>`_)
-- Reject invalid server names in federation requests (`#3480 <https://github.com/matrix-org/synapse/issues/3480>`_)
-- Reject invalid server names in homeserver.yaml (`#3483 <https://github.com/matrix-org/synapse/issues/3483>`_)
-
-
-Bugfixes
---------
-
-- Strip access_token from outgoing requests (`#3327 <https://github.com/matrix-org/synapse/issues/3327>`_)
-- Redact AS tokens in logs (`#3349 <https://github.com/matrix-org/synapse/issues/3349>`_)
-- Fix federation backfill from SQLite servers (`#3355 <https://github.com/matrix-org/synapse/issues/3355>`_)
-- Fix event-purge-by-ts admin API (`#3363 <https://github.com/matrix-org/synapse/issues/3363>`_)
-- Fix event filtering in get_missing_events handler (`#3371 <https://github.com/matrix-org/synapse/issues/3371>`_)
-- Synapse is now stricter regarding accepting events which it cannot retrieve the prev_events for. (`#3456 <https://github.com/matrix-org/synapse/issues/3456>`_)
-- Fix bug where synapse would explode when receiving unicode in HTTP User-Agent header (`#3470 <https://github.com/matrix-org/synapse/issues/3470>`_)
-- Invalidate cache on correct thread to avoid race (`#3473 <https://github.com/matrix-org/synapse/issues/3473>`_)
-
-
-Improved Documentation
-----------------------
-
-- ``doc/postgres.rst``: fix display of the last command block. Thanks to @ArchangeGabriel! (`#3340 <https://github.com/matrix-org/synapse/issues/3340>`_)
-
-
-Deprecations and Removals
--------------------------
-
-- Remove was_forgotten_at (`#3324 <https://github.com/matrix-org/synapse/issues/3324>`_)
-
-
-Misc
-----
-
-- `#3332 <https://github.com/matrix-org/synapse/issues/3332>`_, `#3341 <https://github.com/matrix-org/synapse/issues/3341>`_, `#3347 <https://github.com/matrix-org/synapse/issues/3347>`_, `#3348 <https://github.com/matrix-org/synapse/issues/3348>`_, `#3356 <https://github.com/matrix-org/synapse/issues/3356>`_, `#3385 <https://github.com/matrix-org/synapse/issues/3385>`_, `#3446 <https://github.com/matrix-org/synapse/issues/3446>`_, `#3447 <https://github.com/matrix-org/synapse/issues/3447>`_, `#3467 <https://github.com/matrix-org/synapse/issues/3467>`_, `#3474 <https://github.com/matrix-org/synapse/issues/3474>`_
-
-
-Changes in synapse v0.31.2 (2018-06-14)
-=======================================
-
-SECURITY UPDATE: Prevent unauthorised users from setting state events in a room
-when there is no ``m.room.power_levels`` event in force in the room. (PR #3397)
-
-Discussion around the Matrix Spec change proposal for this change can be
-followed at https://github.com/matrix-org/matrix-doc/issues/1304.
-
-Changes in synapse v0.31.1 (2018-06-08)
-=======================================
-
-v0.31.1 fixes a security bug in the ``get_missing_events`` federation API
-where event visibility rules were not applied correctly.
-
-We are not aware of it being actively exploited but please upgrade asap.
-
-Bug Fixes:
-
-* Fix event filtering in get_missing_events handler (PR #3371)
-
-Changes in synapse v0.31.0 (2018-06-06)
-=======================================
-
-Most notable change from v0.30.0 is to switch to the python prometheus library to improve system
-stats reporting. WARNING: this changes a number of prometheus metrics in a
-backwards-incompatible manner. For more details, see
-`docs/metrics-howto.rst <docs/metrics-howto.rst#removal-of-deprecated-metrics--time-based-counters-becoming-histograms-in-0310>`_.
-
-Bug Fixes:
-
-* Fix metric documentation tables (PR #3341)
-* Fix LaterGauge error handling (694968f)
-* Fix replication metrics (b7e7fd2)
-
-Changes in synapse v0.31.0-rc1 (2018-06-04)
-==========================================
-
-Features:
-
-* Switch to the Python Prometheus library (PR #3256, #3274)
-* Let users leave the server notice room after joining (PR #3287)
-
-
-Changes:
-
-* daily user type phone home stats (PR #3264)
-* Use iter* methods for _filter_events_for_server (PR #3267)
-* Docs on consent bits (PR #3268)
-* Remove users from user directory on deactivate (PR #3277)
-* Avoid sending consent notice to guest users (PR #3288)
-* disable CPUMetrics if no /proc/self/stat (PR #3299)
-* Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (PR #3307)
-* Add private IPv6 addresses to example config for url preview blacklist (PR #3317) Thanks to @thegcat!
-* Reduce stuck read-receipts: ignore depth when updating (PR #3318)
-* Put python's logs into Trial when running unit tests (PR #3319)
-
-Changes, python 3 migration:
-
-* Replace some more comparisons with six (PR #3243) Thanks to @NotAFile!
-* replace some iteritems with six (PR #3244) Thanks to @NotAFile!
-* Add batch_iter to utils (PR #3245) Thanks to @NotAFile!
-* use repr, not str (PR #3246) Thanks to @NotAFile!
-* Misc Python3 fixes (PR #3247) Thanks to @NotAFile!
-* Py3 storage/_base.py (PR #3278) Thanks to @NotAFile!
-* more six iteritems (PR #3279) Thanks to @NotAFile!
-* More Misc. py3 fixes (PR #3280) Thanks to @NotAFile!
-* remaining isintance fixes (PR #3281) Thanks to @NotAFile!
-* py3-ize state.py (PR #3283) Thanks to @NotAFile!
-* extend tox testing for py3 to avoid regressions (PR #3302) Thanks to @krombel!
-* use memoryview in py3 (PR #3303) Thanks to @NotAFile!
-
-Bugs:
-
-* Fix federation backfill bugs (PR #3261)
-* federation: fix LaterGauge usage (PR #3328) Thanks to @intelfx!
+This release adds an index to the events table. This means that on first
+startup there will be an inceased amount of IO until the index is created, and
+an increase in disk usage.
 
 
 Changes in synapse v0.30.0 (2018-05-24)
@@ -248,7 +61,6 @@ Bug Fixes:
 * Fix error in handling receipts (PR #3235)
 * Stop the transaction cache caching failures (PR #3255)
 
-
 Changes in synapse v0.29.1 (2018-05-17)
 ==========================================
 Changes:

CONTRIBUTING.rst

@@ -48,26 +48,6 @@ Please ensure your changes match the cosmetic style of the existing project,
 and **never** mix cosmetic and functional changes in the same commit, as it
 makes it horribly hard to review otherwise.
 
-Changelog
-~~~~~~~~~
-
-All changes, even minor ones, need a corresponding changelog
-entry. These are managed by Towncrier
-(https://github.com/hawkowl/towncrier).
-
-To create a changelog entry, make a new file in the ``changelog.d``
-file named in the format of ``issuenumberOrPR.type``. The type can be
-one of ``feature``, ``bugfix``, ``removal`` (also used for
-deprecations), or ``misc`` (for internal-only changes). The content of
-the file is your changelog entry, which can contain RestructuredText
-formatting. A note of contributors is welcomed in changelogs for
-non-misc changes (the content of misc changes is not displayed).
-
-For example, a fix for a bug reported in #1234 would have its
-changelog entry in ``changelog.d/1234.bugfix``, and contain content
-like "The security levels of Florbs are now validated when
-recieved over federation. Contributed by Jane Matrix".
-
 Attribution
 ~~~~~~~~~~~
 
@@ -130,15 +110,11 @@ If you agree to this for your contribution, then all that's needed is to
 include the line in your commit or pull request comment::
 
     Signed-off-by: Your Name <your@email.example.org>
-
-We accept contributions under a legally identifiable name, such as
-your name on government documentation or common-law names (names
-claimed by legitimate usage or repute). Unfortunately, we cannot
-accept anonymous contributions at this time.
-
-Git allows you to add this signoff automatically when using the ``-s``
-flag to ``git commit``, which uses the name and email set in your
-``user.name`` and ``user.email`` git configs.
+    
+...using your real name; unfortunately pseudonyms and anonymous contributions
+can't be accepted. Git makes this trivial - just use the -s flag when you do
+``git commit``, having first set ``user.name`` and ``user.email`` git configs
+(which you should have done anyway :)
 
 Conclusion
 ~~~~~~~~~~

MANIFEST.in

@@ -29,8 +29,5 @@ exclude Dockerfile
 exclude .dockerignore
 recursive-exclude jenkins *.sh
 
-include pyproject.toml
-recursive-include changelog.d *
-
 prune .github
 prune demo/etc

changelog.d/.gitignore

@@ -1 +0,0 @@
-!.gitignore

docs/admin_api/user_admin_api.rst

@@ -44,26 +44,13 @@ Deactivate Account
 
 This API deactivates an account. It removes active access tokens, resets the
 password, and deletes third-party IDs (to prevent the user requesting a
-password reset). It can also mark the user as GDPR-erased (stopping their data
-from distributed further, and deleting it entirely if there are no other
-references to it).
+password reset).
 
 The api is::
 
     POST /_matrix/client/r0/admin/deactivate/<user_id>
 
-with a body of:
-
-.. code:: json
-
-    {
-        "erase": true
-    }
-
-including an ``access_token`` of a server admin.
-
-The erase parameter is optional and defaults to 'false'.
-An empty body may be passed for backwards compatibility.
+including an ``access_token`` of a server admin, and an empty request body.
 
 
 Reset password

docs/code_style.rst

@@ -16,7 +16,7 @@
       print("I am a fish %s" %
             "moo")
 
-    and this::
+      and this::
 
         print(
             "I am a fish %s" %

docs/metrics-howto.rst

@@ -1,47 +1,25 @@
 How to monitor Synapse metrics using Prometheus
 ===============================================
 
-1. Install Prometheus:
+1. Install prometheus:
 
    Follow instructions at http://prometheus.io/docs/introduction/install/
 
-2. Enable Synapse metrics:
+2. Enable synapse metrics:
 
-   There are two methods of enabling metrics in Synapse.
+   Simply setting a (local) port number will enable it. Pick a port.
+   prometheus itself defaults to 9090, so starting just above that for
+   locally monitored services seems reasonable. E.g. 9092:
 
-   The first serves the metrics as a part of the usual web server and can be
-   enabled by adding the "metrics" resource to the existing listener as such::
+   Add to homeserver.yaml::
 
-     resources:
-       - names:
-         - client
-         - metrics
+     metrics_port: 9092
 
-   This provides a simple way of adding metrics to your Synapse installation,
-   and serves under ``/_synapse/metrics``. If you do not wish your metrics be
-   publicly exposed, you will need to either filter it out at your load
-   balancer, or use the second method.
+   Also ensure that ``enable_metrics`` is set to ``True``.
 
-   The second method runs the metrics server on a different port, in a
-   different thread to Synapse. This can make it more resilient to heavy load
-   meaning metrics cannot be retrieved, and can be exposed to just internal
-   networks easier. The served metrics are available over HTTP only, and will
-   be available at ``/``.
+   Restart synapse.
 
-   Add a new listener to homeserver.yaml::
-
-     listeners:
-       - type: metrics
-         port: 9000
-         bind_addresses:
-           - '0.0.0.0'
-
-   For both options, you will need to ensure that ``enable_metrics`` is set to
-   ``True``.
-
-   Restart Synapse.
-
-3. Add a Prometheus target for Synapse.
+3. Add a prometheus target for synapse.
 
    It needs to set the ``metrics_path`` to a non-default value (under ``scrape_configs``)::
 
@@ -53,50 +31,7 @@ How to monitor Synapse metrics using Prometheus
    If your prometheus is older than 1.5.2, you will need to replace
    ``static_configs`` in the above with ``target_groups``.
 
-   Restart Prometheus.
-
-
-Removal of deprecated metrics & time based counters becoming histograms in 0.31.0
----------------------------------------------------------------------------------
-
-The duplicated metrics deprecated in Synapse 0.27.0 have been removed.
-
-All time duration-based metrics have been changed to be seconds. This affects:
-
-+----------------------------------+
-| msec -> sec metrics              |
-+==================================+
-| python_gc_time                   |
-+----------------------------------+
-| python_twisted_reactor_tick_time |
-+----------------------------------+
-| synapse_storage_query_time       |
-+----------------------------------+
-| synapse_storage_schedule_time    |
-+----------------------------------+
-| synapse_storage_transaction_time |
-+----------------------------------+
-
-Several metrics have been changed to be histograms, which sort entries into
-buckets and allow better analysis. The following metrics are now histograms:
-
-+-------------------------------------------+
-| Altered metrics                           |
-+===========================================+
-| python_gc_time                            |
-+-------------------------------------------+
-| python_twisted_reactor_pending_calls      |
-+-------------------------------------------+
-| python_twisted_reactor_tick_time          |
-+-------------------------------------------+
-| synapse_http_server_response_time_seconds |
-+-------------------------------------------+
-| synapse_storage_query_time                |
-+-------------------------------------------+
-| synapse_storage_schedule_time             |
-+-------------------------------------------+
-| synapse_storage_transaction_time          |
-+-------------------------------------------+
+   Restart prometheus.
 
 
 Block and response metrics renamed for 0.27.0

docs/postgres.rst

@@ -9,19 +9,19 @@ Set up database
 Assuming your PostgreSQL database user is called ``postgres``, create a user
 ``synapse_user`` with::
 
-   su - postgres
-   createuser --pwprompt synapse_user
+ su - postgres
+ createuser --pwprompt synapse_user
 
 The PostgreSQL database used *must* have the correct encoding set, otherwise it
 would not be able to store UTF8 strings. To create a database with the correct
 encoding use, e.g.::
 
-   CREATE DATABASE synapse
-    ENCODING 'UTF8'
-    LC_COLLATE='C'
-    LC_CTYPE='C'
-    template=template0
-    OWNER synapse_user;
+ CREATE DATABASE synapse
+  ENCODING 'UTF8'
+  LC_COLLATE='C'
+  LC_CTYPE='C'
+  template=template0
+  OWNER synapse_user;
 
 This would create an appropriate database named ``synapse`` owned by the
 ``synapse_user`` user (which must already exist).
@@ -126,7 +126,7 @@ run::
         --postgres-config homeserver-postgres.yaml
 
 Once that has completed, change the synapse config to point at the PostgreSQL
-database configuration file ``homeserver-postgres.yaml``::
+database configuration file ``homeserver-postgres.yaml``:
 
     ./synctl stop
     mv homeserver.yaml homeserver-old-sqlite.yaml 

docs/server_notices.md

@@ -5,7 +5,7 @@ Server Notices
 channel whereby server administrators can send messages to users on the server.
 
 They are used as part of communication of the server polices(see
-[consent_tracking.md](consent_tracking.md)), however the intention is that
+[consent_tracking.md](consent_tracking.md)), however the intention is that 
 they may also find a use for features such as "Message of the day".
 
 This is a feature specific to Synapse, but it uses standard Matrix
@@ -24,10 +24,7 @@ history; it will appear to have come from the 'server notices user' (see
 below).
 
 The user is prevented from sending any messages in this room by the power
-levels.
-
-Having joined the room, the user can leave the room if they want. Subsequent
-server notices will then cause a new room to be created.
+levels. They also cannot leave it.
 
 Synapse configuration
 ---------------------

pyproject.toml

@@ -1,5 +0,0 @@
-[tool.towncrier]
-    package = "synapse"
-    filename = "CHANGES.rst"
-    directory = "changelog.d"
-    issue_format = "`#{issue} <https://github.com/matrix-org/synapse/issues/{issue}>`_"

scripts-dev/federation_client.py

@@ -18,22 +18,14 @@
 from __future__ import print_function
 
 import argparse
-from urlparse import urlparse, urlunparse
-
 import nacl.signing
 import json
 import base64
 import requests
 import sys
-
-from requests.adapters import HTTPAdapter
 import srvlookup
 import yaml
 
-# uncomment the following to enable debug logging of http requests
-#from httplib import HTTPConnection
-#HTTPConnection.debuglevel = 1
-
 def encode_base64(input_bytes):
     """Encode bytes as a base64 string without any padding."""
 
@@ -121,6 +113,17 @@ def read_signing_keys(stream):
     return keys
 
 
+def lookup(destination, path):
+    if ":" in destination:
+        return "https://%s%s" % (destination, path)
+    else:
+        try:
+            srv = srvlookup.lookup("matrix", "tcp", destination)[0]
+            return "https://%s:%d%s" % (srv.host, srv.port, path)
+        except:
+            return "https://%s:%d%s" % (destination, 8448, path)
+
+
 def request_json(method, origin_name, origin_key, destination, path, content):
     if method is None:
         if content is None:
@@ -149,19 +152,13 @@ def request_json(method, origin_name, origin_key, destination, path, content):
         authorization_headers.append(bytes(header))
         print ("Authorization: %s" % header, file=sys.stderr)
 
-    dest = "matrix://%s%s" % (destination, path)
+    dest = lookup(destination, path)
     print ("Requesting %s" % dest, file=sys.stderr)
 
-    s = requests.Session()
-    s.mount("matrix://", MatrixConnectionAdapter())
-
-    result = s.request(
+    result = requests.request(
         method=method,
         url=dest,
-        headers={
-            "Host": destination,
-            "Authorization": authorization_headers[0]
-        },
+        headers={"Authorization": authorization_headers[0]},
         verify=False,
         data=content,
     )
@@ -245,39 +242,5 @@ def read_args_from_config(args):
             args.signing_key_path = config['signing_key_path']
 
 
-class MatrixConnectionAdapter(HTTPAdapter):
-    @staticmethod
-    def lookup(s):
-        if s[-1] == ']':
-            # ipv6 literal (with no port)
-            return s, 8448
-
-        if ":" in s:
-            out = s.rsplit(":",1)
-            try:
-                port = int(out[1])
-            except ValueError:
-                raise ValueError("Invalid host:port '%s'" % s)
-            return out[0], port
-
-        try:
-            srv = srvlookup.lookup("matrix", "tcp", s)[0]
-            return srv.host, srv.port
-        except:
-            return s, 8448
-
-    def get_connection(self, url, proxies=None):
-        parsed = urlparse(url)
-
-        (host, port) = self.lookup(parsed.netloc)
-        netloc = "%s:%d" % (host, port)
-        print("Connecting to %s" % (netloc,), file=sys.stderr)
-        url = urlunparse((
-            "https", netloc, parsed.path, parsed.params, parsed.query,
-            parsed.fragment,
-        ))
-        return super(MatrixConnectionAdapter, self).get_connection(url, proxies)
-
-
 if __name__ == "__main__":
     main()

setup.cfg

@@ -17,17 +17,4 @@ ignore =
 [flake8]
 max-line-length = 90
 #  W503 requires that binary operators be at the end, not start, of lines. Erik doesn't like it.
-#  E203 is contrary to PEP8.
-ignore = W503,E203
-
-[isort]
-line_length = 89
-not_skip = __init__.py
-sections=FUTURE,STDLIB,COMPAT,THIRDPARTY,TWISTED,FIRSTPARTY,TESTS,LOCALFOLDER
-default_section=THIRDPARTY
-known_first_party = synapse
-known_tests=tests
-known_compat = mock,six
-known_twisted=twisted,OpenSSL
-multi_line_output=3
-include_trailing_comma=true
+ignore = W503

synapse/__init__.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -17,4 +16,4 @@
 """ This is a reference implementation of a Matrix home server.
 """
 
-__version__ = "0.33.1"
+__version__ = "0.30.0"

synapse/api/auth.py

@@ -15,19 +15,15 @@
 
 import logging
 
-from six import itervalues
-
 import pymacaroons
-from netaddr import IPAddress
-
 from twisted.internet import defer
 
 import synapse.types
 from synapse import event_auth
-from synapse.api.constants import EventTypes, JoinRules, Membership
+from synapse.api.constants import EventTypes, Membership, JoinRules
 from synapse.api.errors import AuthError, Codes
 from synapse.types import UserID
-from synapse.util.caches import CACHE_SIZE_FACTOR, register_cache
+from synapse.util.caches import register_cache, CACHE_SIZE_FACTOR
 from synapse.util.caches.lrucache import LruCache
 from synapse.util.metrics import Measure
 
@@ -61,7 +57,7 @@ class Auth(object):
         self.TOKEN_NOT_FOUND_HTTP_STATUS = 401
 
         self.token_cache = LruCache(CACHE_SIZE_FACTOR * 10000)
-        register_cache("cache", "token_cache", self.token_cache)
+        register_cache("token_cache", self.token_cache)
 
     @defer.inlineCallbacks
     def check_from_context(self, event, context, do_sig_check=True):
@@ -70,7 +66,7 @@ class Auth(object):
         )
         auth_events = yield self.store.get_events(auth_events_ids)
         auth_events = {
-            (e.type, e.state_key): e for e in itervalues(auth_events)
+            (e.type, e.state_key): e for e in auth_events.values()
         }
         self.check(event, auth_events=auth_events, do_sig_check=do_sig_check)
 
@@ -193,7 +189,7 @@ class Auth(object):
                     synapse.types.create_requester(user_id, app_service=app_service)
                 )
 
-            access_token = self.get_access_token_from_request(
+            access_token = get_access_token_from_request(
                 request, self.TOKEN_NOT_FOUND_HTTP_STATUS
             )
 
@@ -239,18 +235,13 @@ class Auth(object):
     @defer.inlineCallbacks
     def _get_appservice_user_id(self, request):
         app_service = self.store.get_app_service_by_token(
-            self.get_access_token_from_request(
+            get_access_token_from_request(
                 request, self.TOKEN_NOT_FOUND_HTTP_STATUS
             )
         )
         if app_service is None:
             defer.returnValue((None, None))
 
-        if app_service.ip_range_whitelist:
-            ip_address = IPAddress(self.hs.get_ip_from_request(request))
-            if ip_address not in app_service.ip_range_whitelist:
-                defer.returnValue((None, None))
-
         if "user_id" not in request.args:
             defer.returnValue((app_service.sender, app_service))
 
@@ -495,7 +486,7 @@ class Auth(object):
     def _look_up_user_by_access_token(self, token):
         ret = yield self.store.get_user_by_access_token(token)
         if not ret:
-            logger.warn("Unrecognised access token - not in store.")
+            logger.warn("Unrecognised access token - not in store: %s" % (token,))
             raise AuthError(
                 self.TOKEN_NOT_FOUND_HTTP_STATUS, "Unrecognised access token.",
                 errcode=Codes.UNKNOWN_TOKEN
@@ -513,12 +504,12 @@ class Auth(object):
 
     def get_appservice_by_req(self, request):
         try:
-            token = self.get_access_token_from_request(
+            token = get_access_token_from_request(
                 request, self.TOKEN_NOT_FOUND_HTTP_STATUS
             )
             service = self.store.get_app_service_by_token(token)
             if not service:
-                logger.warn("Unrecognised appservice access token.")
+                logger.warn("Unrecognised appservice access token: %s" % (token,))
                 raise AuthError(
                     self.TOKEN_NOT_FOUND_HTTP_STATUS,
                     "Unrecognised access token.",
@@ -662,7 +653,7 @@ class Auth(object):
             auth_events[(EventTypes.PowerLevels, "")] = power_level_event
 
         send_level = event_auth.get_send_level(
-            EventTypes.Aliases, "", power_level_event,
+            EventTypes.Aliases, "", auth_events
         )
         user_level = event_auth.get_user_power_level(user_id, auth_events)
 
@@ -673,67 +664,67 @@ class Auth(object):
                 " edit its room list entry"
             )
 
-    @staticmethod
-    def has_access_token(request):
-        """Checks if the request has an access_token.
 
-        Returns:
-            bool: False if no access_token was given, True otherwise.
-        """
-        query_params = request.args.get("access_token")
-        auth_headers = request.requestHeaders.getRawHeaders(b"Authorization")
-        return bool(query_params) or bool(auth_headers)
+def has_access_token(request):
+    """Checks if the request has an access_token.
 
-    @staticmethod
-    def get_access_token_from_request(request, token_not_found_http_status=401):
-        """Extracts the access_token from the request.
+    Returns:
+        bool: False if no access_token was given, True otherwise.
+    """
+    query_params = request.args.get("access_token")
+    auth_headers = request.requestHeaders.getRawHeaders(b"Authorization")
+    return bool(query_params) or bool(auth_headers)
 
-        Args:
-            request: The http request.
-            token_not_found_http_status(int): The HTTP status code to set in the
-                AuthError if the token isn't found. This is used in some of the
-                legacy APIs to change the status code to 403 from the default of
-                401 since some of the old clients depended on auth errors returning
-                403.
-        Returns:
-            str: The access_token
-        Raises:
-            AuthError: If there isn't an access_token in the request.
-        """
 
-        auth_headers = request.requestHeaders.getRawHeaders(b"Authorization")
-        query_params = request.args.get(b"access_token")
-        if auth_headers:
-            # Try the get the access_token from a "Authorization: Bearer"
-            # header
-            if query_params is not None:
-                raise AuthError(
-                    token_not_found_http_status,
-                    "Mixing Authorization headers and access_token query parameters.",
-                    errcode=Codes.MISSING_TOKEN,
-                )
-            if len(auth_headers) > 1:
-                raise AuthError(
-                    token_not_found_http_status,
-                    "Too many Authorization headers.",
-                    errcode=Codes.MISSING_TOKEN,
-                )
-            parts = auth_headers[0].split(" ")
-            if parts[0] == "Bearer" and len(parts) == 2:
-                return parts[1]
-            else:
-                raise AuthError(
-                    token_not_found_http_status,
-                    "Invalid Authorization header.",
-                    errcode=Codes.MISSING_TOKEN,
-                )
+def get_access_token_from_request(request, token_not_found_http_status=401):
+    """Extracts the access_token from the request.
+
+    Args:
+        request: The http request.
+        token_not_found_http_status(int): The HTTP status code to set in the
+            AuthError if the token isn't found. This is used in some of the
+            legacy APIs to change the status code to 403 from the default of
+            401 since some of the old clients depended on auth errors returning
+            403.
+    Returns:
+        str: The access_token
+    Raises:
+        AuthError: If there isn't an access_token in the request.
+    """
+
+    auth_headers = request.requestHeaders.getRawHeaders(b"Authorization")
+    query_params = request.args.get(b"access_token")
+    if auth_headers:
+        # Try the get the access_token from a "Authorization: Bearer"
+        # header
+        if query_params is not None:
+            raise AuthError(
+                token_not_found_http_status,
+                "Mixing Authorization headers and access_token query parameters.",
+                errcode=Codes.MISSING_TOKEN,
+            )
+        if len(auth_headers) > 1:
+            raise AuthError(
+                token_not_found_http_status,
+                "Too many Authorization headers.",
+                errcode=Codes.MISSING_TOKEN,
+            )
+        parts = auth_headers[0].split(" ")
+        if parts[0] == "Bearer" and len(parts) == 2:
+            return parts[1]
         else:
-            # Try to get the access_token from the query params.
-            if not query_params:
-                raise AuthError(
-                    token_not_found_http_status,
-                    "Missing access token.",
-                    errcode=Codes.MISSING_TOKEN
-                )
+            raise AuthError(
+                token_not_found_http_status,
+                "Invalid Authorization header.",
+                errcode=Codes.MISSING_TOKEN,
+            )
+    else:
+        # Try to get the access_token from the query params.
+        if not query_params:
+            raise AuthError(
+                token_not_found_http_status,
+                "Missing access token.",
+                errcode=Codes.MISSING_TOKEN
+            )
 
-            return query_params[0]
+        return query_params[0]

synapse/api/constants.py

@@ -76,8 +76,6 @@ class EventTypes(object):
     Topic = "m.room.topic"
     Name = "m.room.name"
 
-    ServerACL = "m.room.server_acl"
-
 
 class RejectedReason(object):
     AUTH_ERROR = "auth_error"

synapse/api/errors.py

@@ -17,11 +17,10 @@
 
 import logging
 
+import simplejson as json
 from six import iteritems
 from six.moves import http_client
 
-from canonicaljson import json
-
 logger = logging.getLogger(__name__)
 
 

synapse/api/filtering.py

@@ -12,15 +12,14 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import jsonschema
-from canonicaljson import json
-from jsonschema import FormatChecker
-
-from twisted.internet import defer
-
 from synapse.api.errors import SynapseError
 from synapse.storage.presence import UserPresenceState
-from synapse.types import RoomID, UserID
+from synapse.types import UserID, RoomID
+from twisted.internet import defer
+
+import simplejson as json
+import jsonschema
+from jsonschema import FormatChecker
 
 FILTER_SCHEMA = {
     "additionalProperties": False,
@@ -412,7 +411,7 @@ class Filter(object):
         return room_ids
 
     def filter(self, events):
-        return list(filter(self.check, events))
+        return filter(self.check, events)
 
     def limit(self):
         return self.filter_json.get("limit", 10)

synapse/api/urls.py

@@ -15,8 +15,8 @@
 # limitations under the License.
 
 """Contains the URL paths to prefix various aspects of the server with. """
-import hmac
 from hashlib import sha256
+import hmac
 
 from six.moves.urllib.parse import urlencode
 

synapse/app/__init__.py

@@ -14,11 +14,9 @@
 # limitations under the License.
 
 import sys
-
-from synapse import python_dependencies  # noqa: E402
-
 sys.dont_write_bytecode = True
 
+from synapse import python_dependencies   # noqa: E402
 
 try:
     python_dependencies.check_requirements()

synapse/app/_base.py

@@ -17,18 +17,15 @@ import gc
 import logging
 import sys
 
-from daemonize import Daemonize
-
-from twisted.internet import error, reactor
-
-from synapse.util import PreserveLoggingContext
-from synapse.util.rlimit import change_resource_limit
-
 try:
     import affinity
 except Exception:
     affinity = None
 
+from daemonize import Daemonize
+from synapse.util import PreserveLoggingContext
+from synapse.util.rlimit import change_resource_limit
+from twisted.internet import error, reactor
 
 logger = logging.getLogger(__name__)
 
@@ -127,19 +124,6 @@ def quit_with_error(error_string):
     sys.exit(1)
 
 
-def listen_metrics(bind_addresses, port):
-    """
-    Start Prometheus metrics server.
-    """
-    from synapse.metrics import RegistryProxy
-    from prometheus_client import start_http_server
-
-    for host in bind_addresses:
-        reactor.callInThread(start_http_server, int(port),
-                             addr=host, registry=RegistryProxy)
-        logger.info("Metrics now reporting on %s:%d", host, port)
-
-
 def listen_tcp(bind_addresses, port, factory, backlog=50):
     """
     Create a TCP socket for a port and several addresses

synapse/app/appservice.py

@@ -16,9 +16,6 @@
 import logging
 import sys
 
-from twisted.internet import defer, reactor
-from twisted.web.resource import NoResource
-
 import synapse
 from synapse import events
 from synapse.app import _base
@@ -26,7 +23,6 @@ from synapse.config._base import ConfigError
 from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
 from synapse.http.site import SynapseSite
-from synapse.metrics import RegistryProxy
 from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
 from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
 from synapse.replication.slave.storage.directory import DirectoryStore
@@ -39,6 +35,8 @@ from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext, run_in_background
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
+from twisted.internet import reactor, defer
+from twisted.web.resource import NoResource
 
 logger = logging.getLogger("synapse.app.appservice")
 
@@ -64,7 +62,7 @@ class AppserviceServer(HomeServer):
         for res in listener_config["resources"]:
             for name in res["names"]:
                 if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(RegistryProxy)
+                    resources[METRICS_PREFIX] = MetricsResource(self)
 
         root_resource = create_resource_tree(resources, NoResource())
 
@@ -96,13 +94,6 @@ class AppserviceServer(HomeServer):
                         globals={"hs": self},
                     )
                 )
-            elif listener["type"] == "metrics":
-                if not self.get_config().enable_metrics:
-                    logger.warn(("Metrics listener configured, but "
-                                 "enable_metrics is not True!"))
-                else:
-                    _base.listen_metrics(listener["bind_addresses"],
-                                         listener["port"])
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 

synapse/app/client_reader.py

@@ -16,9 +16,6 @@
 import logging
 import sys
 
-from twisted.internet import reactor
-from twisted.web.resource import NoResource
-
 import synapse
 from synapse import events
 from synapse.app import _base
@@ -28,7 +25,6 @@ from synapse.config.logger import setup_logging
 from synapse.crypto import context_factory
 from synapse.http.server import JsonResource
 from synapse.http.site import SynapseSite
-from synapse.metrics import RegistryProxy
 from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
 from synapse.replication.slave.storage._base import BaseSlavedStore
 from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
@@ -47,6 +43,8 @@ from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
+from twisted.internet import reactor
+from twisted.web.resource import NoResource
 
 logger = logging.getLogger("synapse.app.client_reader")
 
@@ -79,7 +77,7 @@ class ClientReaderServer(HomeServer):
         for res in listener_config["resources"]:
             for name in res["names"]:
                 if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(RegistryProxy)
+                    resources[METRICS_PREFIX] = MetricsResource(self)
                 elif name == "client":
                     resource = JsonResource(self, canonical_json=False)
                     PublicRoomListRestServlet(self).register(resource)
@@ -120,13 +118,7 @@ class ClientReaderServer(HomeServer):
                         globals={"hs": self},
                     )
                 )
-            elif listener["type"] == "metrics":
-                if not self.get_config().enable_metrics:
-                    logger.warn(("Metrics listener configured, but "
-                                 "enable_metrics is not True!"))
-                else:
-                    _base.listen_metrics(listener["bind_addresses"],
-                                         listener["port"])
+
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 

synapse/app/event_creator.py

@@ -16,9 +16,6 @@
 import logging
 import sys
 
-from twisted.internet import reactor
-from twisted.web.resource import NoResource
-
 import synapse
 from synapse import events
 from synapse.app import _base
@@ -28,7 +25,6 @@ from synapse.config.logger import setup_logging
 from synapse.crypto import context_factory
 from synapse.http.server import JsonResource
 from synapse.http.site import SynapseSite
-from synapse.metrics import RegistryProxy
 from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
 from synapse.replication.slave.storage._base import BaseSlavedStore
 from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
@@ -46,10 +42,8 @@ from synapse.replication.slave.storage.room import RoomStore
 from synapse.replication.slave.storage.transactions import TransactionStore
 from synapse.replication.tcp.client import ReplicationClientHandler
 from synapse.rest.client.v1.room import (
+    RoomSendEventRestServlet, RoomMembershipRestServlet, RoomStateEventRestServlet,
     JoinRoomAliasServlet,
-    RoomMembershipRestServlet,
-    RoomSendEventRestServlet,
-    RoomStateEventRestServlet,
 )
 from synapse.server import HomeServer
 from synapse.storage.engines import create_engine
@@ -57,6 +51,8 @@ from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
+from twisted.internet import reactor
+from twisted.web.resource import NoResource
 
 logger = logging.getLogger("synapse.app.event_creator")
 
@@ -94,7 +90,7 @@ class EventCreatorServer(HomeServer):
         for res in listener_config["resources"]:
             for name in res["names"]:
                 if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(RegistryProxy)
+                    resources[METRICS_PREFIX] = MetricsResource(self)
                 elif name == "client":
                     resource = JsonResource(self, canonical_json=False)
                     RoomSendEventRestServlet(self).register(resource)
@@ -138,13 +134,6 @@ class EventCreatorServer(HomeServer):
                         globals={"hs": self},
                     )
                 )
-            elif listener["type"] == "metrics":
-                if not self.get_config().enable_metrics:
-                    logger.warn(("Metrics listener configured, but "
-                                 "enable_metrics is not True!"))
-                else:
-                    _base.listen_metrics(listener["bind_addresses"],
-                                         listener["port"])
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 

synapse/app/federation_reader.py

@@ -16,9 +16,6 @@
 import logging
 import sys
 
-from twisted.internet import reactor
-from twisted.web.resource import NoResource
-
 import synapse
 from synapse import events
 from synapse.api.urls import FEDERATION_PREFIX
@@ -29,7 +26,6 @@ from synapse.config.logger import setup_logging
 from synapse.crypto import context_factory
 from synapse.federation.transport.server import TransportLayerServer
 from synapse.http.site import SynapseSite
-from synapse.metrics import RegistryProxy
 from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
 from synapse.replication.slave.storage._base import BaseSlavedStore
 from synapse.replication.slave.storage.directory import DirectoryStore
@@ -44,6 +40,8 @@ from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
+from twisted.internet import reactor
+from twisted.web.resource import NoResource
 
 logger = logging.getLogger("synapse.app.federation_reader")
 
@@ -73,7 +71,7 @@ class FederationReaderServer(HomeServer):
         for res in listener_config["resources"]:
             for name in res["names"]:
                 if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(RegistryProxy)
+                    resources[METRICS_PREFIX] = MetricsResource(self)
                 elif name == "federation":
                     resources.update({
                         FEDERATION_PREFIX: TransportLayerServer(self),
@@ -109,13 +107,6 @@ class FederationReaderServer(HomeServer):
                         globals={"hs": self},
                     )
                 )
-            elif listener["type"] == "metrics":
-                if not self.get_config().enable_metrics:
-                    logger.warn(("Metrics listener configured, but "
-                                 "enable_metrics is not True!"))
-                else:
-                    _base.listen_metrics(listener["bind_addresses"],
-                                         listener["port"])
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 

synapse/app/federation_sender.py

@@ -16,9 +16,6 @@
 import logging
 import sys
 
-from twisted.internet import defer, reactor
-from twisted.web.resource import NoResource
-
 import synapse
 from synapse import events
 from synapse.app import _base
@@ -28,7 +25,6 @@ from synapse.config.logger import setup_logging
 from synapse.crypto import context_factory
 from synapse.federation import send_queue
 from synapse.http.site import SynapseSite
-from synapse.metrics import RegistryProxy
 from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
 from synapse.replication.slave.storage.deviceinbox import SlavedDeviceInboxStore
 from synapse.replication.slave.storage.devices import SlavedDeviceStore
@@ -45,6 +41,8 @@ from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext, run_in_background
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
+from twisted.internet import defer, reactor
+from twisted.web.resource import NoResource
 
 logger = logging.getLogger("synapse.app.federation_sender")
 
@@ -91,7 +89,7 @@ class FederationSenderServer(HomeServer):
         for res in listener_config["resources"]:
             for name in res["names"]:
                 if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(RegistryProxy)
+                    resources[METRICS_PREFIX] = MetricsResource(self)
 
         root_resource = create_resource_tree(resources, NoResource())
 
@@ -123,13 +121,6 @@ class FederationSenderServer(HomeServer):
                         globals={"hs": self},
                     )
                 )
-            elif listener["type"] == "metrics":
-                if not self.get_config().enable_metrics:
-                    logger.warn(("Metrics listener configured, but "
-                                 "enable_metrics is not True!"))
-                else:
-                    _base.listen_metrics(listener["bind_addresses"],
-                                         listener["port"])
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 

synapse/app/frontend_proxy.py

@@ -16,9 +16,6 @@
 import logging
 import sys
 
-from twisted.internet import defer, reactor
-from twisted.web.resource import NoResource
-
 import synapse
 from synapse import events
 from synapse.api.errors import SynapseError
@@ -28,9 +25,10 @@ from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
 from synapse.crypto import context_factory
 from synapse.http.server import JsonResource
-from synapse.http.servlet import RestServlet, parse_json_object_from_request
+from synapse.http.servlet import (
+    RestServlet, parse_json_object_from_request,
+)
 from synapse.http.site import SynapseSite
-from synapse.metrics import RegistryProxy
 from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
 from synapse.replication.slave.storage._base import BaseSlavedStore
 from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
@@ -45,6 +43,8 @@ from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
+from twisted.internet import defer, reactor
+from twisted.web.resource import NoResource
 
 logger = logging.getLogger("synapse.app.frontend_proxy")
 
@@ -131,7 +131,7 @@ class FrontendProxyServer(HomeServer):
         for res in listener_config["resources"]:
             for name in res["names"]:
                 if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(RegistryProxy)
+                    resources[METRICS_PREFIX] = MetricsResource(self)
                 elif name == "client":
                     resource = JsonResource(self, canonical_json=False)
                     KeyUploadServlet(self).register(resource)
@@ -172,13 +172,6 @@ class FrontendProxyServer(HomeServer):
                         globals={"hs": self},
                     )
                 )
-            elif listener["type"] == "metrics":
-                if not self.get_config().enable_metrics:
-                    logger.warn(("Metrics listener configured, but "
-                                 "enable_metrics is not True!"))
-                else:
-                    _base.listen_metrics(listener["bind_addresses"],
-                                         listener["port"])
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 

synapse/app/homeserver.py

@@ -18,39 +18,27 @@ import logging
 import os
 import sys
 
-from twisted.application import service
-from twisted.internet import defer, reactor
-from twisted.web.resource import EncodingResourceWrapper, NoResource
-from twisted.web.server import GzipEncoderFactory
-from twisted.web.static import File
-
 import synapse
 import synapse.config.logger
 from synapse import events
-from synapse.api.urls import (
-    CONTENT_REPO_PREFIX,
-    FEDERATION_PREFIX,
-    LEGACY_MEDIA_PREFIX,
-    MEDIA_PREFIX,
-    SERVER_KEY_PREFIX,
-    SERVER_KEY_V2_PREFIX,
-    STATIC_PREFIX,
-    WEB_CLIENT_PREFIX,
-)
+from synapse.api.urls import CONTENT_REPO_PREFIX, FEDERATION_PREFIX, \
+    LEGACY_MEDIA_PREFIX, MEDIA_PREFIX, SERVER_KEY_PREFIX, SERVER_KEY_V2_PREFIX, \
+    STATIC_PREFIX, WEB_CLIENT_PREFIX
 from synapse.app import _base
-from synapse.app._base import listen_ssl, listen_tcp, quit_with_error
+from synapse.app._base import quit_with_error, listen_ssl, listen_tcp
 from synapse.config._base import ConfigError
 from synapse.config.homeserver import HomeServerConfig
 from synapse.crypto import context_factory
 from synapse.federation.transport.server import TransportLayerServer
+from synapse.module_api import ModuleApi
 from synapse.http.additional_resource import AdditionalResource
 from synapse.http.server import RootRedirect
 from synapse.http.site import SynapseSite
-from synapse.metrics import RegistryProxy
+from synapse.metrics import register_memory_metrics
 from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
-from synapse.module_api import ModuleApi
-from synapse.python_dependencies import CONDITIONAL_REQUIREMENTS, check_requirements
-from synapse.replication.http import REPLICATION_PREFIX, ReplicationRestResource
+from synapse.python_dependencies import CONDITIONAL_REQUIREMENTS, \
+    check_requirements
+from synapse.replication.http import ReplicationRestResource, REPLICATION_PREFIX
 from synapse.replication.tcp.resource import ReplicationStreamProtocolFactory
 from synapse.rest import ClientRestResource
 from synapse.rest.key.v1.server_key_resource import LocalKey
@@ -67,6 +55,11 @@ from synapse.util.manhole import manhole
 from synapse.util.module_loader import load_module
 from synapse.util.rlimit import change_resource_limit
 from synapse.util.versionstring import get_version_string
+from twisted.application import service
+from twisted.internet import defer, reactor
+from twisted.web.resource import EncodingResourceWrapper, NoResource
+from twisted.web.server import GzipEncoderFactory
+from twisted.web.static import File
 
 logger = logging.getLogger("synapse.app.homeserver")
 
@@ -237,7 +230,7 @@ class SynapseHomeServer(HomeServer):
             resources[WEB_CLIENT_PREFIX] = build_resource_for_web_client(self)
 
         if name == "metrics" and self.get_config().enable_metrics:
-            resources[METRICS_PREFIX] = MetricsResource(RegistryProxy)
+            resources[METRICS_PREFIX] = MetricsResource(self)
 
         if name == "replication":
             resources[REPLICATION_PREFIX] = ReplicationRestResource(self)
@@ -270,13 +263,6 @@ class SynapseHomeServer(HomeServer):
                     reactor.addSystemEventTrigger(
                         "before", "shutdown", server_listener.stopListening,
                     )
-            elif listener["type"] == "metrics":
-                if not self.get_config().enable_metrics:
-                    logger.warn(("Metrics listener configured, but "
-                                 "enable_metrics is not True!"))
-                else:
-                    _base.listen_metrics(listener["bind_addresses"],
-                                         listener["port"])
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 
@@ -325,6 +311,11 @@ def setup(config_options):
     # check any extra requirements we have now we have a config
     check_requirements(config)
 
+    version_string = "Synapse/" + get_version_string(synapse)
+
+    logger.info("Server hostname: %s", config.server_name)
+    logger.info("Server version: %s", version_string)
+
     events.USE_FROZEN_DICTS = config.use_frozen_dicts
 
     tls_server_context_factory = context_factory.ServerContextFactory(config)
@@ -337,7 +328,7 @@ def setup(config_options):
         db_config=config.database_config,
         tls_server_context_factory=tls_server_context_factory,
         config=config,
-        version_string="Synapse/" + get_version_string(synapse),
+        version_string=version_string,
         database_engine=database_engine,
     )
 
@@ -371,6 +362,8 @@ def setup(config_options):
         hs.get_datastore().start_doing_background_updates()
         hs.get_federation_client().start_get_pdu_cache()
 
+        register_memory_metrics(hs)
+
     reactor.callWhenRunning(start)
 
     return hs
@@ -441,10 +434,6 @@ def run(hs):
         total_nonbridged_users = yield hs.get_datastore().count_nonbridged_users()
         stats["total_nonbridged_users"] = total_nonbridged_users
 
-        daily_user_type_results = yield hs.get_datastore().count_daily_user_type()
-        for name, count in daily_user_type_results.iteritems():
-            stats["daily_user_type_" + name] = count
-
         room_count = yield hs.get_datastore().get_room_count()
         stats["total_room_count"] = room_count
 

synapse/app/media_repository.py

@@ -16,19 +16,17 @@
 import logging
 import sys
 
-from twisted.internet import reactor
-from twisted.web.resource import NoResource
-
 import synapse
 from synapse import events
-from synapse.api.urls import CONTENT_REPO_PREFIX, LEGACY_MEDIA_PREFIX, MEDIA_PREFIX
+from synapse.api.urls import (
+    CONTENT_REPO_PREFIX, LEGACY_MEDIA_PREFIX, MEDIA_PREFIX
+)
 from synapse.app import _base
 from synapse.config._base import ConfigError
 from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
 from synapse.crypto import context_factory
 from synapse.http.site import SynapseSite
-from synapse.metrics import RegistryProxy
 from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
 from synapse.replication.slave.storage._base import BaseSlavedStore
 from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
@@ -44,6 +42,8 @@ from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
+from twisted.internet import reactor
+from twisted.web.resource import NoResource
 
 logger = logging.getLogger("synapse.app.media_repository")
 
@@ -73,7 +73,7 @@ class MediaRepositoryServer(HomeServer):
         for res in listener_config["resources"]:
             for name in res["names"]:
                 if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(RegistryProxy)
+                    resources[METRICS_PREFIX] = MetricsResource(self)
                 elif name == "media":
                     media_repo = self.get_media_repository_resource()
                     resources.update({
@@ -114,13 +114,6 @@ class MediaRepositoryServer(HomeServer):
                         globals={"hs": self},
                     )
                 )
-            elif listener["type"] == "metrics":
-                if not self.get_config().enable_metrics:
-                    logger.warn(("Metrics listener configured, but "
-                                 "enable_metrics is not True!"))
-                else:
-                    _base.listen_metrics(listener["bind_addresses"],
-                                         listener["port"])
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 

synapse/app/pusher.py

@@ -16,9 +16,6 @@
 import logging
 import sys
 
-from twisted.internet import defer, reactor
-from twisted.web.resource import NoResource
-
 import synapse
 from synapse import events
 from synapse.app import _base
@@ -26,7 +23,6 @@ from synapse.config._base import ConfigError
 from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
 from synapse.http.site import SynapseSite
-from synapse.metrics import RegistryProxy
 from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
 from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
 from synapse.replication.slave.storage.events import SlavedEventStore
@@ -40,6 +36,8 @@ from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext, run_in_background
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
+from twisted.internet import defer, reactor
+from twisted.web.resource import NoResource
 
 logger = logging.getLogger("synapse.app.pusher")
 
@@ -94,7 +92,7 @@ class PusherServer(HomeServer):
         for res in listener_config["resources"]:
             for name in res["names"]:
                 if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(RegistryProxy)
+                    resources[METRICS_PREFIX] = MetricsResource(self)
 
         root_resource = create_resource_tree(resources, NoResource())
 
@@ -126,13 +124,6 @@ class PusherServer(HomeServer):
                         globals={"hs": self},
                     )
                 )
-            elif listener["type"] == "metrics":
-                if not self.get_config().enable_metrics:
-                    logger.warn(("Metrics listener configured, but "
-                                 "enable_metrics is not True!"))
-                else:
-                    _base.listen_metrics(listener["bind_addresses"],
-                                         listener["port"])
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 

synapse/app/synchrotron.py

@@ -17,11 +17,6 @@ import contextlib
 import logging
 import sys
 
-from six import iteritems
-
-from twisted.internet import defer, reactor
-from twisted.web.resource import NoResource
-
 import synapse
 from synapse.api.constants import EventTypes
 from synapse.app import _base
@@ -31,7 +26,6 @@ from synapse.config.logger import setup_logging
 from synapse.handlers.presence import PresenceHandler, get_interested_parties
 from synapse.http.server import JsonResource
 from synapse.http.site import SynapseSite
-from synapse.metrics import RegistryProxy
 from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
 from synapse.replication.slave.storage._base import BaseSlavedStore
 from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
@@ -41,12 +35,12 @@ from synapse.replication.slave.storage.deviceinbox import SlavedDeviceInboxStore
 from synapse.replication.slave.storage.devices import SlavedDeviceStore
 from synapse.replication.slave.storage.events import SlavedEventStore
 from synapse.replication.slave.storage.filtering import SlavedFilteringStore
-from synapse.replication.slave.storage.groups import SlavedGroupServerStore
 from synapse.replication.slave.storage.presence import SlavedPresenceStore
 from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
 from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
 from synapse.replication.slave.storage.registration import SlavedRegistrationStore
 from synapse.replication.slave.storage.room import RoomStore
+from synapse.replication.slave.storage.groups import SlavedGroupServerStore
 from synapse.replication.tcp.client import ReplicationClientHandler
 from synapse.rest.client.v1 import events
 from synapse.rest.client.v1.initial_sync import InitialSyncRestServlet
@@ -61,6 +55,10 @@ from synapse.util.logcontext import LoggingContext, run_in_background
 from synapse.util.manhole import manhole
 from synapse.util.stringutils import random_string
 from synapse.util.versionstring import get_version_string
+from twisted.internet import defer, reactor
+from twisted.web.resource import NoResource
+
+from six import iteritems
 
 logger = logging.getLogger("synapse.app.synchrotron")
 
@@ -259,7 +257,7 @@ class SynchrotronServer(HomeServer):
         for res in listener_config["resources"]:
             for name in res["names"]:
                 if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(RegistryProxy)
+                    resources[METRICS_PREFIX] = MetricsResource(self)
                 elif name == "client":
                     resource = JsonResource(self, canonical_json=False)
                     sync.register_servlets(self, resource)
@@ -303,13 +301,6 @@ class SynchrotronServer(HomeServer):
                         globals={"hs": self},
                     )
                 )
-            elif listener["type"] == "metrics":
-                if not self.get_config().enable_metrics:
-                    logger.warn(("Metrics listener configured, but "
-                                 "enable_metrics is not True!"))
-                else:
-                    _base.listen_metrics(listener["bind_addresses"],
-                                         listener["port"])
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 

synapse/app/synctl.py

@@ -16,16 +16,15 @@
 
 import argparse
 import collections
-import errno
 import glob
 import os
 import os.path
 import signal
 import subprocess
 import sys
-import time
-
 import yaml
+import errno
+import time
 
 SYNAPSE = [sys.executable, "-B", "-m", "synapse.app.homeserver"]
 
@@ -172,10 +171,6 @@ def main():
     if cache_factor:
         os.environ["SYNAPSE_CACHE_FACTOR"] = str(cache_factor)
 
-    cache_factors = config.get("synctl_cache_factors", {})
-    for cache_name, factor in cache_factors.iteritems():
-        os.environ["SYNAPSE_CACHE_FACTOR_" + cache_name.upper()] = str(factor)
-
     worker_configfiles = []
     if options.worker:
         start_stop_synapse = False

synapse/app/user_dir.py

@@ -17,9 +17,6 @@
 import logging
 import sys
 
-from twisted.internet import defer, reactor
-from twisted.web.resource import NoResource
-
 import synapse
 from synapse import events
 from synapse.app import _base
@@ -29,7 +26,6 @@ from synapse.config.logger import setup_logging
 from synapse.crypto import context_factory
 from synapse.http.server import JsonResource
 from synapse.http.site import SynapseSite
-from synapse.metrics import RegistryProxy
 from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
 from synapse.replication.slave.storage._base import BaseSlavedStore
 from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
@@ -46,6 +42,8 @@ from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext, run_in_background
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
+from twisted.internet import reactor, defer
+from twisted.web.resource import NoResource
 
 logger = logging.getLogger("synapse.app.user_dir")
 
@@ -107,7 +105,7 @@ class UserDirectoryServer(HomeServer):
         for res in listener_config["resources"]:
             for name in res["names"]:
                 if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(RegistryProxy)
+                    resources[METRICS_PREFIX] = MetricsResource(self)
                 elif name == "client":
                     resource = JsonResource(self, canonical_json=False)
                     user_directory.register_servlets(self, resource)
@@ -148,13 +146,6 @@ class UserDirectoryServer(HomeServer):
                         globals={"hs": self},
                     )
                 )
-            elif listener["type"] == "metrics":
-                if not self.get_config().enable_metrics:
-                    logger.warn(("Metrics listener configured, but "
-                                 "enable_metrics is not True!"))
-                else:
-                    _base.listen_metrics(listener["bind_addresses"],
-                                         listener["port"])
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 

synapse/appservice/__init__.py

@@ -12,17 +12,17 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+from synapse.api.constants import EventTypes
+from synapse.util.caches.descriptors import cachedInlineCallbacks
+from synapse.types import GroupID, get_domain_from_id
+
+from twisted.internet import defer
+
 import logging
 import re
 
 from six import string_types
 
-from twisted.internet import defer
-
-from synapse.api.constants import EventTypes
-from synapse.types import GroupID, get_domain_from_id
-from synapse.util.caches.descriptors import cachedInlineCallbacks
-
 logger = logging.getLogger(__name__)
 
 
@@ -85,8 +85,7 @@ class ApplicationService(object):
     NS_LIST = [NS_USERS, NS_ALIASES, NS_ROOMS]
 
     def __init__(self, token, hostname, url=None, namespaces=None, hs_token=None,
-                 sender=None, id=None, protocols=None, rate_limited=True,
-                 ip_range_whitelist=None):
+                 sender=None, id=None, protocols=None, rate_limited=True):
         self.token = token
         self.url = url
         self.hs_token = hs_token
@@ -94,7 +93,6 @@ class ApplicationService(object):
         self.server_name = hostname
         self.namespaces = self._check_namespaces(namespaces)
         self.id = id
-        self.ip_range_whitelist = ip_range_whitelist
 
         if "|" in self.id:
             raise Exception("application service ID cannot contain '|' character")
@@ -294,8 +292,4 @@ class ApplicationService(object):
         return self.rate_limited
 
     def __str__(self):
-        # copy dictionary and redact token fields so they don't get logged
-        dict_copy = self.__dict__.copy()
-        dict_copy["token"] = "<redacted>"
-        dict_copy["hs_token"] = "<redacted>"
-        return "ApplicationService: %s" % (dict_copy,)
+        return "ApplicationService: %s" % (self.__dict__,)

synapse/appservice/api.py

@@ -12,39 +12,20 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
-import urllib
-
-from prometheus_client import Counter
-
 from twisted.internet import defer
 
 from synapse.api.constants import ThirdPartyEntityKind
 from synapse.api.errors import CodeMessageException
-from synapse.events.utils import serialize_event
 from synapse.http.client import SimpleHttpClient
-from synapse.types import ThirdPartyInstanceID
+from synapse.events.utils import serialize_event
 from synapse.util.caches.response_cache import ResponseCache
+from synapse.types import ThirdPartyInstanceID
+
+import logging
+import urllib
 
 logger = logging.getLogger(__name__)
 
-sent_transactions_counter = Counter(
-    "synapse_appservice_api_sent_transactions",
-    "Number of /transactions/ requests sent",
-    ["service"]
-)
-
-failed_transactions_counter = Counter(
-    "synapse_appservice_api_failed_transactions",
-    "Number of /transactions/ requests that failed to send",
-    ["service"]
-)
-
-sent_events_counter = Counter(
-    "synapse_appservice_api_sent_events",
-    "Number of events sent to the AS",
-    ["service"]
-)
 
 HOUR_IN_MS = 60 * 60 * 1000
 
@@ -238,15 +219,12 @@ class ApplicationServiceApi(SimpleHttpClient):
                 args={
                     "access_token": service.hs_token
                 })
-            sent_transactions_counter.labels(service.id).inc()
-            sent_events_counter.labels(service.id).inc(len(events))
             defer.returnValue(True)
             return
         except CodeMessageException as e:
             logger.warning("push_bulk to %s received %s", uri, e.code)
         except Exception as ex:
             logger.warning("push_bulk to %s threw exception %s", uri, ex)
-        failed_transactions_counter.labels(service.id).inc()
         defer.returnValue(False)
 
     def _serialize(self, events):

synapse/appservice/scheduler.py

@@ -48,14 +48,14 @@ UP & quit           +---------- YES                       SUCCESS
 This is all tied together by the AppServiceScheduler which DIs the required
 components.
 """
-import logging
-
 from twisted.internet import defer
 
 from synapse.appservice import ApplicationServiceState
 from synapse.util.logcontext import run_in_background
 from synapse.util.metrics import Measure
 
+import logging
+
 logger = logging.getLogger(__name__)
 
 

synapse/config/_base.py

@@ -16,12 +16,11 @@
 import argparse
 import errno
 import os
+import yaml
 from textwrap import dedent
 
 from six import integer_types
 
-import yaml
-
 
 class ConfigError(Exception):
     pass

synapse/config/api.py

@@ -12,10 +12,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from synapse.api.constants import EventTypes
-
 from ._base import Config
 
+from synapse.api.constants import EventTypes
+
 
 class ApiConfig(Config):
 

synapse/config/appservice.py

@@ -12,18 +12,16 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import logging
-
-from six import string_types
-from six.moves.urllib import parse as urlparse
-
-import yaml
-from netaddr import IPSet
+from ._base import Config, ConfigError
 
 from synapse.appservice import ApplicationService
 from synapse.types import UserID
 
-from ._base import Config, ConfigError
+import yaml
+import logging
+
+from six import string_types
+from six.moves.urllib import parse as urlparse
 
 logger = logging.getLogger(__name__)
 
@@ -156,13 +154,6 @@ def _load_appservice(hostname, as_info, config_filename):
             " will not receive events or queries.",
             config_filename,
         )
-
-    ip_range_whitelist = None
-    if as_info.get('ip_range_whitelist'):
-        ip_range_whitelist = IPSet(
-            as_info.get('ip_range_whitelist')
-        )
-
     return ApplicationService(
         token=as_info["as_token"],
         hostname=hostname,
@@ -172,6 +163,5 @@ def _load_appservice(hostname, as_info, config_filename):
         sender=user_id,
         id=as_info["id"],
         protocols=protocols,
-        rate_limited=rate_limited,
-        ip_range_whitelist=ip_range_whitelist,
+        rate_limited=rate_limited
     )

synapse/config/consent_config.py

@@ -18,9 +18,6 @@ from ._base import Config
 DEFAULT_CONFIG = """\
 # User Consent configuration
 #
-# for detailed instructions, see
-# https://github.com/matrix-org/synapse/blob/master/docs/consent_tracking.md
-#
 # Parts of this section are required if enabling the 'consent' resource under
 # 'listeners', in particular 'template_dir' and 'version'.
 #
@@ -35,8 +32,7 @@ DEFAULT_CONFIG = """\
 #
 # 'server_notice_content', if enabled, will send a user a "Server Notice"
 # asking them to consent to the privacy policy. The 'server_notices' section
-# must also be configured for this to work. Notices will *not* be sent to
-# guest users unless 'send_server_notice_to_guests' is set to true.
+# must also be configured for this to work.
 #
 # 'block_events_error', if set, will block any attempts to send events
 # until the user consents to the privacy policy. The value of the setting is
@@ -50,7 +46,6 @@ DEFAULT_CONFIG = """\
 #     body: >-
 #       To continue using this homeserver you must review and agree to the
 #       terms and conditions at %(consent_uri)s
-#   send_server_notice_to_guests: True
 #   block_events_error: >-
 #     To continue using this homeserver you must review and agree to the
 #     terms and conditions at %(consent_uri)s
@@ -65,7 +60,6 @@ class ConsentConfig(Config):
         self.user_consent_version = None
         self.user_consent_template_dir = None
         self.user_consent_server_notice_content = None
-        self.user_consent_server_notice_to_guests = False
         self.block_events_without_consent_error = None
 
     def read_config(self, config):
@@ -80,9 +74,6 @@ class ConsentConfig(Config):
         self.block_events_without_consent_error = consent_config.get(
             "block_events_error",
         )
-        self.user_consent_server_notice_to_guests = bool(consent_config.get(
-            "send_server_notice_to_guests", False,
-        ))
 
     def default_config(self, **kwargs):
         return DEFAULT_CONFIG

synapse/config/homeserver.py

@@ -13,32 +13,32 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+from .tls import TlsConfig
+from .server import ServerConfig
+from .logger import LoggingConfig
+from .database import DatabaseConfig
+from .ratelimiting import RatelimitConfig
+from .repository import ContentRepositoryConfig
+from .captcha import CaptchaConfig
+from .voip import VoipConfig
+from .registration import RegistrationConfig
+from .metrics import MetricsConfig
 from .api import ApiConfig
 from .appservice import AppServiceConfig
-from .captcha import CaptchaConfig
-from .cas import CasConfig
-from .consent_config import ConsentConfig
-from .database import DatabaseConfig
-from .emailconfig import EmailConfig
-from .groups import GroupsConfig
-from .jwt import JWTConfig
 from .key import KeyConfig
-from .logger import LoggingConfig
-from .metrics import MetricsConfig
-from .password import PasswordConfig
-from .password_auth_providers import PasswordAuthProviderConfig
-from .push import PushConfig
-from .ratelimiting import RatelimitConfig
-from .registration import RegistrationConfig
-from .repository import ContentRepositoryConfig
 from .saml2 import SAML2Config
-from .server import ServerConfig
-from .server_notices_config import ServerNoticesConfig
-from .spam_checker import SpamCheckerConfig
-from .tls import TlsConfig
-from .user_directory import UserDirectoryConfig
-from .voip import VoipConfig
+from .cas import CasConfig
+from .password import PasswordConfig
+from .jwt import JWTConfig
+from .password_auth_providers import PasswordAuthProviderConfig
+from .emailconfig import EmailConfig
 from .workers import WorkerConfig
+from .push import PushConfig
+from .spam_checker import SpamCheckerConfig
+from .groups import GroupsConfig
+from .user_directory import UserDirectoryConfig
+from .consent_config import ConsentConfig
+from .server_notices_config import ServerNoticesConfig
 
 
 class HomeServerConfig(TlsConfig, ServerConfig, DatabaseConfig, LoggingConfig,

synapse/config/jwt.py

@@ -15,6 +15,7 @@
 
 from ._base import Config, ConfigError
 
+
 MISSING_JWT = (
     """Missing jwt library. This is required for jwt login.
 

synapse/config/key.py

@@ -13,24 +13,21 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import hashlib
-import logging
-import os
+from ._base import Config, ConfigError
 
+from synapse.util.stringutils import random_string
 from signedjson.key import (
-    NACL_ED25519,
-    decode_signing_key_base64,
-    decode_verify_key_bytes,
-    generate_signing_key,
-    is_signing_algorithm_supported,
-    read_signing_keys,
-    write_signing_keys,
+    generate_signing_key, is_signing_algorithm_supported,
+    decode_signing_key_base64, decode_verify_key_bytes,
+    read_signing_keys, write_signing_keys, NACL_ED25519
 )
 from unpaddedbase64 import decode_base64
+from synapse.util.stringutils import random_string_with_symbols
 
-from synapse.util.stringutils import random_string, random_string_with_symbols
+import os
+import hashlib
+import logging
 
-from ._base import Config, ConfigError
 
 logger = logging.getLogger(__name__)
 

synapse/config/logger.py

@@ -12,22 +12,17 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
-import logging.config
-import os
-import signal
-import sys
-from string import Template
-
-import yaml
-
-from twisted.logger import STDLibLogObserver, globalLogBeginner
-
-import synapse
-from synapse.util.logcontext import LoggingContextFilter
-from synapse.util.versionstring import get_version_string
 
 from ._base import Config
+from synapse.util.logcontext import LoggingContextFilter
+from twisted.logger import globalLogBeginner, STDLibLogObserver
+import logging
+import logging.config
+import yaml
+from string import Template
+import os
+import signal
+
 
 DEFAULT_LOG_CONFIG = Template("""
 version: 1
@@ -207,15 +202,6 @@ def setup_logging(config, use_worker_options=False):
     if getattr(signal, "SIGHUP"):
         signal.signal(signal.SIGHUP, sighup)
 
-    # make sure that the first thing we log is a thing we can grep backwards
-    # for
-    logging.warn("***** STARTING SERVER *****")
-    logging.warn(
-        "Server %s version %s",
-        sys.argv[0], get_version_string(synapse),
-    )
-    logging.info("Server hostname: %s", config.server_name)
-
     # It's critical to point twisted's internal logging somewhere, otherwise it
     # stacks up and leaks kup to 64K object;
     # see: https://twistedmatrix.com/trac/ticket/8164

synapse/config/password_auth_providers.py

@@ -13,10 +13,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from synapse.util.module_loader import load_module
-
 from ._base import Config
 
+from synapse.util.module_loader import load_module
+
 LDAP_PROVIDER = 'ldap_auth_provider.LdapAuthProvider'
 
 

synapse/config/registration.py

@@ -13,11 +13,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from distutils.util import strtobool
+from ._base import Config
 
 from synapse.util.stringutils import random_string_with_symbols
 
-from ._base import Config
+from distutils.util import strtobool
 
 
 class RegistrationConfig(Config):

synapse/config/repository.py

@@ -13,11 +13,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from ._base import Config, ConfigError
 from collections import namedtuple
 
 from synapse.util.module_loader import load_module
 
-from ._base import Config, ConfigError
 
 MISSING_NETADDR = (
     "Missing netaddr library. This is required for URL preview API."
@@ -250,9 +250,6 @@ class ContentRepositoryConfig(Config):
         # - '192.168.0.0/16'
         # - '100.64.0.0/10'
         # - '169.254.0.0/16'
-        # - '::1/128'
-        # - 'fe80::/64'
-        # - 'fc00::/7'
         #
         # List of IP address CIDR ranges that the URL preview spider is allowed
         # to access even if they are specified in url_preview_ip_range_blacklist.

synapse/config/server.py

@@ -14,25 +14,13 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import logging
-
-from synapse.http.endpoint import parse_and_validate_server_name
-
 from ._base import Config, ConfigError
 
-logger = logging.Logger(__name__)
-
 
 class ServerConfig(Config):
 
     def read_config(self, config):
         self.server_name = config["server_name"]
-
-        try:
-            parse_and_validate_server_name(self.server_name)
-        except ValueError as e:
-            raise ConfigError(str(e))
-
         self.pid_file = self.abspath(config.get("pid_file"))
         self.web_client = config["web_client"]
         self.web_client_location = config.get("web_client_location", None)
@@ -150,12 +138,6 @@ class ServerConfig(Config):
 
         metrics_port = config.get("metrics_port")
         if metrics_port:
-            logger.warn(
-                ("The metrics_port configuration option is deprecated in Synapse 0.31 "
-                 "in favour of a listener. Please see "
-                 "http://github.com/matrix-org/synapse/blob/master/docs/metrics-howto.rst"
-                 " on how to configure the new listener."))
-
             self.listeners.append({
                 "port": metrics_port,
                 "bind_addresses": [config.get("metrics_bind_host", "127.0.0.1")],
@@ -170,8 +152,8 @@ class ServerConfig(Config):
             })
 
     def default_config(self, server_name, **kwargs):
-        _, bind_port = parse_and_validate_server_name(server_name)
-        if bind_port is not None:
+        if ":" in server_name:
+            bind_port = int(server_name.split(":")[1])
             unsecure_port = bind_port - 400
         else:
             bind_port = 8448

synapse/config/server_notices_config.py

@@ -12,9 +12,8 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-from synapse.types import UserID
-
 from ._base import Config
+from synapse.types import UserID
 
 DEFAULT_CONFIG = """\
 # Server Notices room configuration

synapse/config/tls.py

@@ -13,15 +13,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import os
-import subprocess
-from hashlib import sha256
-
-from unpaddedbase64 import encode_base64
+from ._base import Config
 
 from OpenSSL import crypto
+import subprocess
+import os
 
-from ._base import Config
+from hashlib import sha256
+from unpaddedbase64 import encode_base64
 
 GENERATE_DH_PARAMS = False
 

synapse/crypto/context_factory.py

@@ -12,12 +12,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import logging
-
-from OpenSSL import SSL, crypto
 from twisted.internet import ssl
+from OpenSSL import SSL, crypto
 from twisted.internet._sslverify import _defaultCurveName
 
+import logging
+
 logger = logging.getLogger(__name__)
 
 

synapse/crypto/event_signing.py

@@ -15,15 +15,15 @@
 # limitations under the License.
 
 
-import hashlib
-import logging
+from synapse.api.errors import SynapseError, Codes
+from synapse.events.utils import prune_event
 
 from canonicaljson import encode_canonical_json
+from unpaddedbase64 import encode_base64, decode_base64
 from signedjson.sign import sign_json
-from unpaddedbase64 import decode_base64, encode_base64
 
-from synapse.api.errors import Codes, SynapseError
-from synapse.events.utils import prune_event
+import hashlib
+import logging
 
 logger = logging.getLogger(__name__)
 

synapse/crypto/keyclient.py

@@ -13,16 +13,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from synapse.util import logcontext
+from twisted.web.http import HTTPClient
+from twisted.internet.protocol import Factory
+from twisted.internet import defer, reactor
+from synapse.http.endpoint import matrix_federation_endpoint
+import simplejson as json
 import logging
 
-from canonicaljson import json
-
-from twisted.internet import defer, reactor
-from twisted.internet.protocol import Factory
-from twisted.web.http import HTTPClient
-
-from synapse.http.endpoint import matrix_federation_endpoint
-from synapse.util import logcontext
 
 logger = logging.getLogger(__name__)
 

synapse/crypto/keyring.py

@@ -14,31 +14,9 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import hashlib
-import logging
-import urllib
-from collections import namedtuple
-
-from signedjson.key import (
-    decode_verify_key_bytes,
-    encode_verify_key_base64,
-    is_signing_algorithm_supported,
-)
-from signedjson.sign import (
-    SignatureVerifyException,
-    encode_canonical_json,
-    sign_json,
-    signature_ids,
-    verify_signed_json,
-)
-from unpaddedbase64 import decode_base64, encode_base64
-
-from OpenSSL import crypto
-from twisted.internet import defer
-
-from synapse.api.errors import Codes, SynapseError
 from synapse.crypto.keyclient import fetch_server_key
-from synapse.util import logcontext, unwrapFirstError
+from synapse.api.errors import SynapseError, Codes
+from synapse.util import unwrapFirstError, logcontext
 from synapse.util.logcontext import (
     PreserveLoggingContext,
     preserve_fn,
@@ -46,6 +24,24 @@ from synapse.util.logcontext import (
 )
 from synapse.util.metrics import Measure
 
+from twisted.internet import defer
+
+from signedjson.sign import (
+    verify_signed_json, signature_ids, sign_json, encode_canonical_json
+)
+from signedjson.key import (
+    is_signing_algorithm_supported, decode_verify_key_bytes
+)
+from unpaddedbase64 import decode_base64, encode_base64
+
+from OpenSSL import crypto
+
+from collections import namedtuple
+import urllib
+import hashlib
+import logging
+
+
 logger = logging.getLogger(__name__)
 
 
@@ -60,7 +56,7 @@ Attributes:
     key_ids(set(str)): The set of key_ids to that could be used to verify the
         JSON object
     json_object(dict): The JSON object to verify.
-    deferred(Deferred[str, str, nacl.signing.VerifyKey]):
+    deferred(twisted.internet.defer.Deferred):
         A deferred (server_name, key_id, verify_key) tuple that resolves when
         a verify key has been fetched. The deferreds' callbacks are run with no
         logcontext.
@@ -740,17 +736,6 @@ class Keyring(object):
 
 @defer.inlineCallbacks
 def _handle_key_deferred(verify_request):
-    """Waits for the key to become available, and then performs a verification
-
-    Args:
-        verify_request (VerifyKeyRequest):
-
-    Returns:
-        Deferred[None]
-
-    Raises:
-        SynapseError if there was a problem performing the verification
-    """
     server_name = verify_request.server_name
     try:
         with PreserveLoggingContext():
@@ -783,17 +768,11 @@ def _handle_key_deferred(verify_request):
     ))
     try:
         verify_signed_json(json_object, server_name, verify_key)
-    except SignatureVerifyException as e:
-        logger.debug(
-            "Error verifying signature for %s:%s:%s with key %s: %s",
-            server_name, verify_key.alg, verify_key.version,
-            encode_verify_key_base64(verify_key),
-            str(e),
-        )
+    except Exception:
         raise SynapseError(
             401,
-            "Invalid signature for server %s with key %s:%s: %s" % (
-                server_name, verify_key.alg, verify_key.version, str(e),
+            "Invalid signature for server %s with key %s:%s" % (
+                server_name, verify_key.alg, verify_key.version
             ),
             Codes.UNAUTHORIZED,
         )

synapse/event_auth.py

@@ -17,11 +17,11 @@ import logging
 
 from canonicaljson import encode_canonical_json
 from signedjson.key import decode_verify_key_bytes
-from signedjson.sign import SignatureVerifyException, verify_signed_json
+from signedjson.sign import verify_signed_json, SignatureVerifyException
 from unpaddedbase64 import decode_base64
 
-from synapse.api.constants import EventTypes, JoinRules, Membership
-from synapse.api.errors import AuthError, EventSizeError, SynapseError
+from synapse.api.constants import EventTypes, Membership, JoinRules
+from synapse.api.errors import AuthError, SynapseError, EventSizeError
 from synapse.types import UserID, get_domain_from_id
 
 logger = logging.getLogger(__name__)
@@ -34,11 +34,9 @@ def check(event, auth_events, do_sig_check=True, do_size_check=True):
         event: the event being checked.
         auth_events (dict: event-key -> event): the existing room state.
 
-    Raises:
-        AuthError if the checks fail
 
     Returns:
-         if the auth checks pass.
+        True if the auth checks pass.
     """
     if do_size_check:
         _check_size_limits(event)
@@ -73,10 +71,9 @@ def check(event, auth_events, do_sig_check=True, do_size_check=True):
         # Oh, we don't know what the state of the room was, so we
         # are trusting that this is allowed (at least for now)
         logger.warn("Trusting event: %s", event.event_id)
-        return
+        return True
 
     if event.type == EventTypes.Create:
-        sender_domain = get_domain_from_id(event.sender)
         room_id_domain = get_domain_from_id(event.room_id)
         if room_id_domain != sender_domain:
             raise AuthError(
@@ -84,8 +81,7 @@ def check(event, auth_events, do_sig_check=True, do_size_check=True):
                 "Creation event's room_id domain does not match sender's"
             )
         # FIXME
-        logger.debug("Allowing! %s", event)
-        return
+        return True
 
     creation_event = auth_events.get((EventTypes.Create, ""), None)
 
@@ -122,8 +118,7 @@ def check(event, auth_events, do_sig_check=True, do_size_check=True):
                 403,
                 "Alias event's state_key does not match sender's domain"
             )
-        logger.debug("Allowing! %s", event)
-        return
+        return True
 
     if logger.isEnabledFor(logging.DEBUG):
         logger.debug(
@@ -132,9 +127,14 @@ def check(event, auth_events, do_sig_check=True, do_size_check=True):
         )
 
     if event.type == EventTypes.Member:
-        _is_membership_change_allowed(event, auth_events)
-        logger.debug("Allowing! %s", event)
-        return
+        allowed = _is_membership_change_allowed(
+            event, auth_events
+        )
+        if allowed:
+            logger.debug("Allowing! %s", event)
+        else:
+            logger.debug("Denying! %s", event)
+        return allowed
 
     _check_event_sender_in_room(event, auth_events)
 
@@ -153,8 +153,7 @@ def check(event, auth_events, do_sig_check=True, do_size_check=True):
                 )
             )
         else:
-            logger.debug("Allowing! %s", event)
-            return
+            return True
 
     _can_send_event(event, auth_events)
 
@@ -201,7 +200,7 @@ def _is_membership_change_allowed(event, auth_events):
         create = auth_events.get(key)
         if create and event.prev_events[0][0] == create.event_id:
             if create.content["creator"] == event.state_key:
-                return
+                return True
 
     target_user_id = event.state_key
 
@@ -266,13 +265,13 @@ def _is_membership_change_allowed(event, auth_events):
             raise AuthError(
                 403, "%s is banned from the room" % (target_user_id,)
             )
-        return
+        return True
 
     if Membership.JOIN != membership:
         if (caller_invited
                 and Membership.LEAVE == membership
                 and target_user_id == event.user_id):
-            return
+            return True
 
         if not caller_in_room:  # caller isn't joined
             raise AuthError(
@@ -335,6 +334,8 @@ def _is_membership_change_allowed(event, auth_events):
     else:
         raise AuthError(500, "Unknown membership %s" % membership)
 
+    return True
+
 
 def _check_event_sender_in_room(event, auth_events):
     key = (EventTypes.Member, event.user_id, )
@@ -354,46 +355,35 @@ def _check_joined_room(member, user_id, room_id):
         ))
 
 
-def get_send_level(etype, state_key, power_levels_event):
-    """Get the power level required to send an event of a given type
+def get_send_level(etype, state_key, auth_events):
+    key = (EventTypes.PowerLevels, "", )
+    send_level_event = auth_events.get(key)
+    send_level = None
+    if send_level_event:
+        send_level = send_level_event.content.get("events", {}).get(
+            etype
+        )
+        if send_level is None:
+            if state_key is not None:
+                send_level = send_level_event.content.get(
+                    "state_default", 50
+                )
+            else:
+                send_level = send_level_event.content.get(
+                    "events_default", 0
+                )
 
-    The federation spec [1] refers to this as "Required Power Level".
-
-    https://matrix.org/docs/spec/server_server/unstable.html#definitions
-
-    Args:
-        etype (str): type of event
-        state_key (str|None): state_key of state event, or None if it is not
-            a state event.
-        power_levels_event (synapse.events.EventBase|None): power levels event
-            in force at this point in the room
-    Returns:
-        int: power level required to send this event.
-    """
-
-    if power_levels_event:
-        power_levels_content = power_levels_event.content
+    if send_level:
+        send_level = int(send_level)
     else:
-        power_levels_content = {}
+        send_level = 0
 
-    # see if we have a custom level for this event type
-    send_level = power_levels_content.get("events", {}).get(etype)
-
-    # otherwise, fall back to the state_default/events_default.
-    if send_level is None:
-        if state_key is not None:
-            send_level = power_levels_content.get("state_default", 50)
-        else:
-            send_level = power_levels_content.get("events_default", 0)
-
-    return int(send_level)
+    return send_level
 
 
 def _can_send_event(event, auth_events):
-    power_levels_event = _get_power_level_event(auth_events)
-
     send_level = get_send_level(
-        event.type, event.get("state_key"), power_levels_event,
+        event.type, event.get("state_key", None), auth_events
     )
     user_level = get_user_power_level(event.user_id, auth_events)
 
@@ -481,14 +471,14 @@ def _check_power_levels(event, auth_events):
     ]
 
     old_list = current_state.content.get("users", {})
-    for user in set(list(old_list) + list(user_list)):
+    for user in set(old_list.keys() + user_list.keys()):
         levels_to_check.append(
             (user, "users")
         )
 
     old_list = current_state.content.get("events", {})
     new_list = event.content.get("events", {})
-    for ev_id in set(list(old_list) + list(new_list)):
+    for ev_id in set(old_list.keys() + new_list.keys()):
         levels_to_check.append(
             (ev_id, "events")
         )
@@ -525,11 +515,7 @@ def _check_power_levels(event, auth_events):
                     "to your own"
                 )
 
-        # Check if the old and new levels are greater than the user level
-        # (if defined)
-        old_level_too_big = old_level is not None and old_level > user_level
-        new_level_too_big = new_level is not None and new_level > user_level
-        if old_level_too_big or new_level_too_big:
+        if old_level > user_level or new_level > user_level:
             raise AuthError(
                 403,
                 "You don't have permission to add ops level greater "
@@ -538,22 +524,13 @@ def _check_power_levels(event, auth_events):
 
 
 def _get_power_level_event(auth_events):
-    return auth_events.get((EventTypes.PowerLevels, ""))
+    key = (EventTypes.PowerLevels, "", )
+    return auth_events.get(key)
 
 
 def get_user_power_level(user_id, auth_events):
-    """Get a user's power level
-
-    Args:
-        user_id (str): user's id to look up in power_levels
-        auth_events (dict[(str, str), synapse.events.EventBase]):
-            state in force at this point in the room (or rather, a subset of
-            it including at least the create event and power levels event.
-
-    Returns:
-        int: the user's power level in this room.
-    """
     power_level_event = _get_power_level_event(auth_events)
+
     if power_level_event:
         level = power_level_event.content.get("users", {}).get(user_id)
         if not level:
@@ -564,11 +541,6 @@ def get_user_power_level(user_id, auth_events):
         else:
             return int(level)
     else:
-        # if there is no power levels event, the creator gets 100 and everyone
-        # else gets 0.
-
-        # some things which call this don't pass the create event: hack around
-        # that.
         key = (EventTypes.Create, "", )
         create_event = auth_events.get(key)
         if (create_event is not None and

synapse/events/__init__.py

@@ -13,8 +13,9 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from synapse.util.caches import intern_dict
 from synapse.util.frozenutils import freeze
+from synapse.util.caches import intern_dict
+
 
 # Whether we should use frozen_dict in FrozenEvent. Using frozen_dicts prevents
 # bugs where we accidentally share e.g. signature dicts. However, converting
@@ -145,7 +146,7 @@ class EventBase(object):
         return field in self._event_dict
 
     def items(self):
-        return list(self._event_dict.items())
+        return self._event_dict.items()
 
 
 class FrozenEvent(EventBase):

synapse/events/builder.py

@@ -13,12 +13,13 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import copy
+from . import EventBase, FrozenEvent, _event_dict_property
 
 from synapse.types import EventID
+
 from synapse.util.stringutils import random_string
 
-from . import EventBase, FrozenEvent, _event_dict_property
+import copy
 
 
 class EventBuilder(EventBase):

synapse/events/snapshot.py

@@ -13,10 +13,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from frozendict import frozendict
-
 from twisted.internet import defer
 
+from frozendict import frozendict
+
 
 class EventContext(object):
     """

synapse/events/utils.py

@@ -13,15 +13,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import re
-
-from six import string_types
+from synapse.api.constants import EventTypes
+from . import EventBase
 
 from frozendict import frozendict
 
-from synapse.api.constants import EventTypes
+import re
 
-from . import EventBase
+from six import string_types
 
 # Split strings on "." but not "\." This uses a negative lookbehind assertion for '\'
 # (?<!stuff) matches if the current position in the string is not preceded

synapse/events/validator.py

@@ -13,11 +13,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from six import string_types
-
-from synapse.api.constants import EventTypes, Membership
-from synapse.api.errors import SynapseError
 from synapse.types import EventID, RoomID, UserID
+from synapse.api.errors import SynapseError
+from synapse.api.constants import EventTypes, Membership
+
+from six import string_types
 
 
 class EventValidator(object):

synapse/federation/federation_base.py

@@ -16,15 +16,14 @@ import logging
 
 import six
 
-from twisted.internet import defer
-
 from synapse.api.constants import MAX_DEPTH
-from synapse.api.errors import Codes, SynapseError
+from synapse.api.errors import SynapseError, Codes
 from synapse.crypto.event_signing import check_event_content_hash
 from synapse.events import FrozenEvent
 from synapse.events.utils import prune_event
-from synapse.http.servlet import assert_params_in_dict
-from synapse.util import logcontext, unwrapFirstError
+from synapse.http.servlet import assert_params_in_request
+from synapse.util import unwrapFirstError, logcontext
+from twisted.internet import defer
 
 logger = logging.getLogger(__name__)
 
@@ -199,7 +198,7 @@ def event_from_pdu_json(pdu_json, outlier=False):
     """
     # we could probably enforce a bunch of other fields here (room_id, sender,
     # origin, etc etc)
-    assert_params_in_dict(pdu_json, ('event_id', 'type', 'depth'))
+    assert_params_in_request(pdu_json, ('event_id', 'type', 'depth'))
 
     depth = pdu_json['depth']
     if not isinstance(depth, six.integer_types):

synapse/federation/federation_client.py

@@ -21,19 +21,18 @@ import random
 
 from six.moves import range
 
-from prometheus_client import Counter
-
 from twisted.internet import defer
 
 from synapse.api.constants import Membership
 from synapse.api.errors import (
-    CodeMessageException,
-    FederationDeniedError,
-    HttpResponseException,
-    SynapseError,
+    CodeMessageException, HttpResponseException, SynapseError, FederationDeniedError
 )
 from synapse.events import builder
-from synapse.federation.federation_base import FederationBase, event_from_pdu_json
+from synapse.federation.federation_base import (
+    FederationBase,
+    event_from_pdu_json,
+)
+import synapse.metrics
 from synapse.util import logcontext, unwrapFirstError
 from synapse.util.caches.expiringcache import ExpiringCache
 from synapse.util.logcontext import make_deferred_yieldable, run_in_background
@@ -42,7 +41,11 @@ from synapse.util.retryutils import NotRetryingDestination
 
 logger = logging.getLogger(__name__)
 
-sent_queries_counter = Counter("synapse_federation_client_sent_queries", "", ["type"])
+
+# synapse.federation.federation_client is a silly name
+metrics = synapse.metrics.get_metrics_for("synapse.federation.client")
+
+sent_queries_counter = metrics.register_counter("sent_queries", labels=["type"])
 
 
 PDU_RETRY_TIME_MS = 1 * 60 * 1000
@@ -105,7 +108,7 @@ class FederationClient(FederationBase):
             a Deferred which will eventually yield a JSON object from the
             response
         """
-        sent_queries_counter.labels(query_type).inc()
+        sent_queries_counter.inc(query_type)
 
         return self.transport_layer.make_query(
             destination, query_type, args, retry_on_dns_fail=retry_on_dns_fail,
@@ -124,7 +127,7 @@ class FederationClient(FederationBase):
             a Deferred which will eventually yield a JSON object from the
             response
         """
-        sent_queries_counter.labels("client_device_keys").inc()
+        sent_queries_counter.inc("client_device_keys")
         return self.transport_layer.query_client_keys(
             destination, content, timeout
         )
@@ -134,7 +137,7 @@ class FederationClient(FederationBase):
         """Query the device keys for a list of user ids hosted on a remote
         server.
         """
-        sent_queries_counter.labels("user_devices").inc()
+        sent_queries_counter.inc("user_devices")
         return self.transport_layer.query_user_devices(
             destination, user_id, timeout
         )
@@ -151,7 +154,7 @@ class FederationClient(FederationBase):
             a Deferred which will eventually yield a JSON object from the
             response
         """
-        sent_queries_counter.labels("client_one_time_keys").inc()
+        sent_queries_counter.inc("client_one_time_keys")
         return self.transport_layer.claim_client_keys(
             destination, content, timeout
         )
@@ -391,7 +394,7 @@ class FederationClient(FederationBase):
         """
         if return_local:
             seen_events = yield self.store.get_events(event_ids, allow_rejected=True)
-            signed_events = list(seen_events.values())
+            signed_events = seen_events.values()
         else:
             seen_events = yield self.store.have_seen_events(event_ids)
             signed_events = []
@@ -589,7 +592,7 @@ class FederationClient(FederationBase):
                 }
 
                 valid_pdus = yield self._check_sigs_and_hash_and_fetch(
-                    destination, list(pdus.values()),
+                    destination, pdus.values(),
                     outlier=True,
                 )
 

synapse/federation/federation_server.py

@@ -14,46 +14,44 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import logging
-import re
-
-import six
-from six import iteritems
-
-from canonicaljson import json
-from prometheus_client import Counter
 
+import simplejson as json
 from twisted.internet import defer
-from twisted.internet.abstract import isIPAddress
 
-from synapse.api.constants import EventTypes
-from synapse.api.errors import AuthError, FederationError, NotFoundError, SynapseError
+from synapse.api.errors import AuthError, FederationError, SynapseError, NotFoundError
 from synapse.crypto.event_signing import compute_event_signature
-from synapse.federation.federation_base import FederationBase, event_from_pdu_json
+from synapse.federation.federation_base import (
+    FederationBase,
+    event_from_pdu_json,
+)
+
 from synapse.federation.persistence import TransactionActions
 from synapse.federation.units import Edu, Transaction
-from synapse.http.endpoint import parse_server_name
+import synapse.metrics
 from synapse.types import get_domain_from_id
 from synapse.util import async
 from synapse.util.caches.response_cache import ResponseCache
 from synapse.util.logutils import log_function
 
+from six import iteritems
+
 # when processing incoming transactions, we try to handle multiple rooms in
 # parallel, up to this limit.
 TRANSACTION_CONCURRENCY_LIMIT = 10
 
 logger = logging.getLogger(__name__)
 
-received_pdus_counter = Counter("synapse_federation_server_received_pdus", "")
+# synapse.federation.federation_server is a silly name
+metrics = synapse.metrics.get_metrics_for("synapse.federation.server")
 
-received_edus_counter = Counter("synapse_federation_server_received_edus", "")
+received_pdus_counter = metrics.register_counter("received_pdus")
 
-received_queries_counter = Counter(
-    "synapse_federation_server_received_queries", "", ["type"]
-)
+received_edus_counter = metrics.register_counter("received_edus")
+
+received_queries_counter = metrics.register_counter("received_queries", labels=["type"])
 
 
 class FederationServer(FederationBase):
-
     def __init__(self, hs):
         super(FederationServer, self).__init__(hs)
 
@@ -75,9 +73,6 @@ class FederationServer(FederationBase):
     @log_function
     def on_backfill_request(self, origin, room_id, versions, limit):
         with (yield self._server_linearizer.queue((origin, room_id))):
-            origin_host, _ = parse_server_name(origin)
-            yield self.check_server_matches_acl(origin_host, room_id)
-
             pdus = yield self.handler.on_backfill_request(
                 origin, room_id, versions, limit
             )
@@ -136,9 +131,7 @@ class FederationServer(FederationBase):
 
         logger.debug("[%s] Transaction is new", transaction.transaction_id)
 
-        received_pdus_counter.inc(len(transaction.pdus))
-
-        origin_host, _ = parse_server_name(transaction.origin)
+        received_pdus_counter.inc_by(len(transaction.pdus))
 
         pdus_by_room = {}
 
@@ -160,21 +153,9 @@ class FederationServer(FederationBase):
         # we can process different rooms in parallel (which is useful if they
         # require callouts to other servers to fetch missing events), but
         # impose a limit to avoid going too crazy with ram/cpu.
-
         @defer.inlineCallbacks
         def process_pdus_for_room(room_id):
             logger.debug("Processing PDUs for %s", room_id)
-            try:
-                yield self.check_server_matches_acl(origin_host, room_id)
-            except AuthError as e:
-                logger.warn(
-                    "Ignoring PDUs for room %s from banned server", room_id,
-                )
-                for pdu in pdus_by_room[room_id]:
-                    event_id = pdu.event_id
-                    pdu_results[event_id] = e.error_dict()
-                return
-
             for pdu in pdus_by_room[room_id]:
                 event_id = pdu.event_id
                 try:
@@ -229,9 +210,6 @@ class FederationServer(FederationBase):
         if not event_id:
             raise NotImplementedError("Specify an event")
 
-        origin_host, _ = parse_server_name(origin)
-        yield self.check_server_matches_acl(origin_host, room_id)
-
         in_room = yield self.auth.check_host_in_room(room_id, origin)
         if not in_room:
             raise AuthError(403, "Host not in room.")
@@ -255,9 +233,6 @@ class FederationServer(FederationBase):
         if not event_id:
             raise NotImplementedError("Specify an event")
 
-        origin_host, _ = parse_server_name(origin)
-        yield self.check_server_matches_acl(origin_host, room_id)
-
         in_room = yield self.auth.check_host_in_room(room_id, origin)
         if not in_room:
             raise AuthError(403, "Host not in room.")
@@ -301,7 +276,7 @@ class FederationServer(FederationBase):
     @defer.inlineCallbacks
     @log_function
     def on_pdu_request(self, origin, event_id):
-        pdu = yield self.handler.get_persisted_pdu(origin, event_id)
+        pdu = yield self._get_persisted_pdu(origin, event_id)
 
         if pdu:
             defer.returnValue(
@@ -317,14 +292,12 @@ class FederationServer(FederationBase):
 
     @defer.inlineCallbacks
     def on_query_request(self, query_type, args):
-        received_queries_counter.labels(query_type).inc()
+        received_queries_counter.inc(query_type)
         resp = yield self.registry.on_query(query_type, args)
         defer.returnValue((200, resp))
 
     @defer.inlineCallbacks
-    def on_make_join_request(self, origin, room_id, user_id):
-        origin_host, _ = parse_server_name(origin)
-        yield self.check_server_matches_acl(origin_host, room_id)
+    def on_make_join_request(self, room_id, user_id):
         pdu = yield self.handler.on_make_join_request(room_id, user_id)
         time_now = self._clock.time_msec()
         defer.returnValue({"event": pdu.get_pdu_json(time_now)})
@@ -332,8 +305,6 @@ class FederationServer(FederationBase):
     @defer.inlineCallbacks
     def on_invite_request(self, origin, content):
         pdu = event_from_pdu_json(content)
-        origin_host, _ = parse_server_name(origin)
-        yield self.check_server_matches_acl(origin_host, pdu.room_id)
         ret_pdu = yield self.handler.on_invite_request(origin, pdu)
         time_now = self._clock.time_msec()
         defer.returnValue((200, {"event": ret_pdu.get_pdu_json(time_now)}))
@@ -342,10 +313,6 @@ class FederationServer(FederationBase):
     def on_send_join_request(self, origin, content):
         logger.debug("on_send_join_request: content: %s", content)
         pdu = event_from_pdu_json(content)
-
-        origin_host, _ = parse_server_name(origin)
-        yield self.check_server_matches_acl(origin_host, pdu.room_id)
-
         logger.debug("on_send_join_request: pdu sigs: %s", pdu.signatures)
         res_pdus = yield self.handler.on_send_join_request(origin, pdu)
         time_now = self._clock.time_msec()
@@ -357,9 +324,7 @@ class FederationServer(FederationBase):
         }))
 
     @defer.inlineCallbacks
-    def on_make_leave_request(self, origin, room_id, user_id):
-        origin_host, _ = parse_server_name(origin)
-        yield self.check_server_matches_acl(origin_host, room_id)
+    def on_make_leave_request(self, room_id, user_id):
         pdu = yield self.handler.on_make_leave_request(room_id, user_id)
         time_now = self._clock.time_msec()
         defer.returnValue({"event": pdu.get_pdu_json(time_now)})
@@ -368,10 +333,6 @@ class FederationServer(FederationBase):
     def on_send_leave_request(self, origin, content):
         logger.debug("on_send_leave_request: content: %s", content)
         pdu = event_from_pdu_json(content)
-
-        origin_host, _ = parse_server_name(origin)
-        yield self.check_server_matches_acl(origin_host, pdu.room_id)
-
         logger.debug("on_send_leave_request: pdu sigs: %s", pdu.signatures)
         yield self.handler.on_send_leave_request(origin, pdu)
         defer.returnValue((200, {}))
@@ -379,9 +340,6 @@ class FederationServer(FederationBase):
     @defer.inlineCallbacks
     def on_event_auth(self, origin, room_id, event_id):
         with (yield self._server_linearizer.queue((origin, room_id))):
-            origin_host, _ = parse_server_name(origin)
-            yield self.check_server_matches_acl(origin_host, room_id)
-
             time_now = self._clock.time_msec()
             auth_pdus = yield self.handler.on_event_auth(event_id)
             res = {
@@ -410,9 +368,6 @@ class FederationServer(FederationBase):
             Deferred: Results in `dict` with the same format as `content`
         """
         with (yield self._server_linearizer.queue((origin, room_id))):
-            origin_host, _ = parse_server_name(origin)
-            yield self.check_server_matches_acl(origin_host, room_id)
-
             auth_chain = [
                 event_from_pdu_json(e)
                 for e in content["auth_chain"]
@@ -425,7 +380,6 @@ class FederationServer(FederationBase):
             ret = yield self.handler.on_query_auth(
                 origin,
                 event_id,
-                room_id,
                 signed_auth,
                 content.get("rejects", []),
                 content.get("missing", []),
@@ -487,9 +441,6 @@ class FederationServer(FederationBase):
     def on_get_missing_events(self, origin, room_id, earliest_events,
                               latest_events, limit, min_depth):
         with (yield self._server_linearizer.queue((origin, room_id))):
-            origin_host, _ = parse_server_name(origin)
-            yield self.check_server_matches_acl(origin_host, room_id)
-
             logger.info(
                 "on_get_missing_events: earliest_events: %r, latest_events: %r,"
                 " limit: %d, min_depth: %d",
@@ -518,6 +469,17 @@ class FederationServer(FederationBase):
         ts_now_ms = self._clock.time_msec()
         return self.store.get_user_id_for_open_id_token(token, ts_now_ms)
 
+    @log_function
+    def _get_persisted_pdu(self, origin, event_id, do_auth=True):
+        """ Get a PDU from the database with given origin and id.
+
+        Returns:
+            Deferred: Results in a `Pdu`.
+        """
+        return self.handler.get_persisted_pdu(
+            origin, event_id, do_auth=do_auth
+        )
+
     def _transaction_from_pdus(self, pdu_list):
         """Returns a new Transaction containing the given PDUs suitable for
         transmission.
@@ -597,9 +559,7 @@ class FederationServer(FederationBase):
                 affected=pdu.event_id,
             )
 
-        yield self.handler.on_receive_pdu(
-            origin, pdu, get_missing=True, sent_to_us_directly=True,
-        )
+        yield self.handler.on_receive_pdu(origin, pdu, get_missing=True)
 
     def __str__(self):
         return "<ReplicationLayer(%s)>" % self.server_name
@@ -627,101 +587,6 @@ class FederationServer(FederationBase):
         )
         defer.returnValue(ret)
 
-    @defer.inlineCallbacks
-    def check_server_matches_acl(self, server_name, room_id):
-        """Check if the given server is allowed by the server ACLs in the room
-
-        Args:
-            server_name (str): name of server, *without any port part*
-            room_id (str): ID of the room to check
-
-        Raises:
-            AuthError if the server does not match the ACL
-        """
-        state_ids = yield self.store.get_current_state_ids(room_id)
-        acl_event_id = state_ids.get((EventTypes.ServerACL, ""))
-
-        if not acl_event_id:
-            return
-
-        acl_event = yield self.store.get_event(acl_event_id)
-        if server_matches_acl_event(server_name, acl_event):
-            return
-
-        raise AuthError(code=403, msg="Server is banned from room")
-
-
-def server_matches_acl_event(server_name, acl_event):
-    """Check if the given server is allowed by the ACL event
-
-    Args:
-        server_name (str): name of server, without any port part
-        acl_event (EventBase): m.room.server_acl event
-
-    Returns:
-        bool: True if this server is allowed by the ACLs
-    """
-    logger.debug("Checking %s against acl %s", server_name, acl_event.content)
-
-    # first of all, check if literal IPs are blocked, and if so, whether the
-    # server name is a literal IP
-    allow_ip_literals = acl_event.content.get("allow_ip_literals", True)
-    if not isinstance(allow_ip_literals, bool):
-        logger.warn("Ignorning non-bool allow_ip_literals flag")
-        allow_ip_literals = True
-    if not allow_ip_literals:
-        # check for ipv6 literals. These start with '['.
-        if server_name[0] == '[':
-            return False
-
-        # check for ipv4 literals. We can just lift the routine from twisted.
-        if isIPAddress(server_name):
-            return False
-
-    # next,  check the deny list
-    deny = acl_event.content.get("deny", [])
-    if not isinstance(deny, (list, tuple)):
-        logger.warn("Ignorning non-list deny ACL %s", deny)
-        deny = []
-    for e in deny:
-        if _acl_entry_matches(server_name, e):
-            # logger.info("%s matched deny rule %s", server_name, e)
-            return False
-
-    # then the allow list.
-    allow = acl_event.content.get("allow", [])
-    if not isinstance(allow, (list, tuple)):
-        logger.warn("Ignorning non-list allow ACL %s", allow)
-        allow = []
-    for e in allow:
-        if _acl_entry_matches(server_name, e):
-            # logger.info("%s matched allow rule %s", server_name, e)
-            return True
-
-    # everything else should be rejected.
-    # logger.info("%s fell through", server_name)
-    return False
-
-
-def _acl_entry_matches(server_name, acl_entry):
-    if not isinstance(acl_entry, six.string_types):
-        logger.warn("Ignoring non-str ACL entry '%s' (is %s)", acl_entry, type(acl_entry))
-        return False
-    regex = _glob_to_regex(acl_entry)
-    return regex.match(server_name)
-
-
-def _glob_to_regex(glob):
-    res = ''
-    for c in glob:
-        if c == '*':
-            res = res + '.*'
-        elif c == '?':
-            res = res + '.'
-        else:
-            res = res + re.escape(c)
-    return re.compile(res + "\\Z", re.IGNORECASE)
-
 
 class FederationHandlerRegistry(object):
     """Allows classes to register themselves as handlers for a given EDU or

synapse/federation/persistence.py

@@ -19,12 +19,13 @@ package.
 These actions are mostly only used by the :py:mod:`.replication` module.
 """
 
-import logging
-
 from twisted.internet import defer
 
 from synapse.util.logutils import log_function
 
+import logging
+
+
 logger = logging.getLogger(__name__)
 
 

synapse/federation/send_queue.py

@@ -29,22 +29,25 @@ dead worker doesn't cause the queues to grow limitlessly.
 Events are replicated via a separate events stream.
 """
 
-import logging
-from collections import namedtuple
-
-from six import iteritems, itervalues
-
-from sortedcontainers import SortedDict
-
-from synapse.metrics import LaterGauge
-from synapse.storage.presence import UserPresenceState
-from synapse.util.metrics import Measure
-
 from .units import Edu
 
+from synapse.storage.presence import UserPresenceState
+from synapse.util.metrics import Measure
+import synapse.metrics
+
+from blist import sorteddict
+from collections import namedtuple
+
+import logging
+
+from six import itervalues, iteritems
+
 logger = logging.getLogger(__name__)
 
 
+metrics = synapse.metrics.get_metrics_for(__name__)
+
+
 class FederationRemoteSendQueue(object):
     """A drop in replacement for TransactionQueue"""
 
@@ -55,27 +58,29 @@ class FederationRemoteSendQueue(object):
         self.is_mine_id = hs.is_mine_id
 
         self.presence_map = {}  # Pending presence map user_id -> UserPresenceState
-        self.presence_changed = SortedDict()  # Stream position -> user_id
+        self.presence_changed = sorteddict()  # Stream position -> user_id
 
         self.keyed_edu = {}  # (destination, key) -> EDU
-        self.keyed_edu_changed = SortedDict()  # stream position -> (destination, key)
+        self.keyed_edu_changed = sorteddict()  # stream position -> (destination, key)
 
-        self.edus = SortedDict()  # stream position -> Edu
+        self.edus = sorteddict()  # stream position -> Edu
 
-        self.failures = SortedDict()  # stream position -> (destination, Failure)
+        self.failures = sorteddict()  # stream position -> (destination, Failure)
 
-        self.device_messages = SortedDict()  # stream position -> destination
+        self.device_messages = sorteddict()  # stream position -> destination
 
         self.pos = 1
-        self.pos_time = SortedDict()
+        self.pos_time = sorteddict()
 
         # EVERYTHING IS SAD. In particular, python only makes new scopes when
         # we make a new function, so we need to make a new function so the inner
         # lambda binds to the queue rather than to the name of the queue which
         # changes. ARGH.
         def register(name, queue):
-            LaterGauge("synapse_federation_send_queue_%s_size" % (queue_name,),
-                       "", [], lambda: len(queue))
+            metrics.register_callback(
+                queue_name + "_size",
+                lambda: len(queue),
+            )
 
         for queue_name in [
             "presence_map", "presence_changed", "keyed_edu", "keyed_edu_changed",
@@ -98,7 +103,7 @@ class FederationRemoteSendQueue(object):
         now = self.clock.time_msec()
 
         keys = self.pos_time.keys()
-        time = self.pos_time.bisect_left(now - FIVE_MINUTES_AGO)
+        time = keys.bisect_left(now - FIVE_MINUTES_AGO)
         if not keys[:time]:
             return
 
@@ -113,7 +118,7 @@ class FederationRemoteSendQueue(object):
         with Measure(self.clock, "send_queue._clear"):
             # Delete things out of presence maps
             keys = self.presence_changed.keys()
-            i = self.presence_changed.bisect_left(position_to_delete)
+            i = keys.bisect_left(position_to_delete)
             for key in keys[:i]:
                 del self.presence_changed[key]
 
@@ -131,7 +136,7 @@ class FederationRemoteSendQueue(object):
 
             # Delete things out of keyed edus
             keys = self.keyed_edu_changed.keys()
-            i = self.keyed_edu_changed.bisect_left(position_to_delete)
+            i = keys.bisect_left(position_to_delete)
             for key in keys[:i]:
                 del self.keyed_edu_changed[key]
 
@@ -145,19 +150,19 @@ class FederationRemoteSendQueue(object):
 
             # Delete things out of edu map
             keys = self.edus.keys()
-            i = self.edus.bisect_left(position_to_delete)
+            i = keys.bisect_left(position_to_delete)
             for key in keys[:i]:
                 del self.edus[key]
 
             # Delete things out of failure map
             keys = self.failures.keys()
-            i = self.failures.bisect_left(position_to_delete)
+            i = keys.bisect_left(position_to_delete)
             for key in keys[:i]:
                 del self.failures[key]
 
             # Delete things out of device map
             keys = self.device_messages.keys()
-            i = self.device_messages.bisect_left(position_to_delete)
+            i = keys.bisect_left(position_to_delete)
             for key in keys[:i]:
                 del self.device_messages[key]
 
@@ -197,7 +202,7 @@ class FederationRemoteSendQueue(object):
 
         # We only want to send presence for our own users, so lets always just
         # filter here just in case.
-        local_states = list(filter(lambda s: self.is_mine_id(s.user_id), states))
+        local_states = filter(lambda s: self.is_mine_id(s.user_id), states)
 
         self.presence_map.update({state.user_id: state for state in local_states})
         self.presence_changed[pos] = [state.user_id for state in local_states]
@@ -250,12 +255,13 @@ class FederationRemoteSendQueue(object):
             self._clear_queue_before_pos(federation_ack)
 
         # Fetch changed presence
-        i = self.presence_changed.bisect_right(from_token)
-        j = self.presence_changed.bisect_right(to_token) + 1
+        keys = self.presence_changed.keys()
+        i = keys.bisect_right(from_token)
+        j = keys.bisect_right(to_token) + 1
         dest_user_ids = [
             (pos, user_id)
-            for pos, user_id_list in self.presence_changed.items()[i:j]
-            for user_id in user_id_list
+            for pos in keys[i:j]
+            for user_id in self.presence_changed[pos]
         ]
 
         for (key, user_id) in dest_user_ids:
@@ -264,12 +270,13 @@ class FederationRemoteSendQueue(object):
             )))
 
         # Fetch changes keyed edus
-        i = self.keyed_edu_changed.bisect_right(from_token)
-        j = self.keyed_edu_changed.bisect_right(to_token) + 1
+        keys = self.keyed_edu_changed.keys()
+        i = keys.bisect_right(from_token)
+        j = keys.bisect_right(to_token) + 1
         # We purposefully clobber based on the key here, python dict comprehensions
         # always use the last value, so this will correctly point to the last
         # stream position.
-        keyed_edus = {v: k for k, v in self.keyed_edu_changed.items()[i:j]}
+        keyed_edus = {self.keyed_edu_changed[k]: k for k in keys[i:j]}
 
         for ((destination, edu_key), pos) in iteritems(keyed_edus):
             rows.append((pos, KeyedEduRow(
@@ -278,17 +285,19 @@ class FederationRemoteSendQueue(object):
             )))
 
         # Fetch changed edus
-        i = self.edus.bisect_right(from_token)
-        j = self.edus.bisect_right(to_token) + 1
-        edus = self.edus.items()[i:j]
+        keys = self.edus.keys()
+        i = keys.bisect_right(from_token)
+        j = keys.bisect_right(to_token) + 1
+        edus = ((k, self.edus[k]) for k in keys[i:j])
 
         for (pos, edu) in edus:
             rows.append((pos, EduRow(edu)))
 
         # Fetch changed failures
-        i = self.failures.bisect_right(from_token)
-        j = self.failures.bisect_right(to_token) + 1
-        failures = self.failures.items()[i:j]
+        keys = self.failures.keys()
+        i = keys.bisect_right(from_token)
+        j = keys.bisect_right(to_token) + 1
+        failures = ((k, self.failures[k]) for k in keys[i:j])
 
         for (pos, (destination, failure)) in failures:
             rows.append((pos, FailureRow(
@@ -297,9 +306,10 @@ class FederationRemoteSendQueue(object):
             )))
 
         # Fetch changed device messages
-        i = self.device_messages.bisect_right(from_token)
-        j = self.device_messages.bisect_right(to_token) + 1
-        device_messages = {v: k for k, v in self.device_messages.items()[i:j]}
+        keys = self.device_messages.keys()
+        i = keys.bisect_right(from_token)
+        j = keys.bisect_right(to_token) + 1
+        device_messages = {self.device_messages[k]: k for k in keys[i:j]}
 
         for (destination, pos) in iteritems(device_messages):
             rows.append((pos, DeviceRow(

synapse/federation/transaction_queue.py

@@ -13,38 +13,36 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import datetime
-import logging
-
-from six import itervalues
-
-from prometheus_client import Counter
 
 from twisted.internet import defer
 
-import synapse.metrics
-from synapse.api.errors import FederationDeniedError, HttpResponseException
-from synapse.handlers.presence import format_user_presence_state, get_interested_remotes
-from synapse.metrics import (
-    LaterGauge,
-    events_processed_counter,
-    sent_edus_counter,
-    sent_transactions_counter,
-)
-from synapse.util import PreserveLoggingContext, logcontext
-from synapse.util.metrics import measure_func
-from synapse.util.retryutils import NotRetryingDestination, get_retry_limiter
-
 from .persistence import TransactionActions
-from .units import Edu, Transaction
+from .units import Transaction, Edu
+
+from synapse.api.errors import HttpResponseException, FederationDeniedError
+from synapse.util import logcontext, PreserveLoggingContext
+from synapse.util.async import run_on_reactor
+from synapse.util.retryutils import NotRetryingDestination, get_retry_limiter
+from synapse.util.metrics import measure_func
+from synapse.handlers.presence import format_user_presence_state, get_interested_remotes
+import synapse.metrics
+
+import logging
+
 
 logger = logging.getLogger(__name__)
 
-sent_pdus_destination_dist_count = Counter(
-    "synapse_federation_client_sent_pdu_destinations:count", ""
-)
-sent_pdus_destination_dist_total = Counter(
-    "synapse_federation_client_sent_pdu_destinations:total", ""
+metrics = synapse.metrics.get_metrics_for(__name__)
+
+client_metrics = synapse.metrics.get_metrics_for("synapse.federation.client")
+sent_pdus_destination_dist = client_metrics.register_distribution(
+    "sent_pdu_destinations"
 )
+sent_edus_counter = client_metrics.register_counter("sent_edus")
+
+sent_transactions_counter = client_metrics.register_counter("sent_transactions")
+
+events_processed_counter = client_metrics.register_counter("events_processed")
 
 
 class TransactionQueue(object):
@@ -71,10 +69,8 @@ class TransactionQueue(object):
         # done
         self.pending_transactions = {}
 
-        LaterGauge(
-            "synapse_federation_transaction_queue_pending_destinations",
-            "",
-            [],
+        metrics.register_callback(
+            "pending_destinations",
             lambda: len(self.pending_transactions),
         )
 
@@ -98,16 +94,12 @@ class TransactionQueue(object):
         # Map of destination -> (edu_type, key) -> Edu
         self.pending_edus_keyed_by_dest = edus_keyed = {}
 
-        LaterGauge(
-            "synapse_federation_transaction_queue_pending_pdus",
-            "",
-            [],
+        metrics.register_callback(
+            "pending_pdus",
             lambda: sum(map(len, pdus.values())),
         )
-        LaterGauge(
-            "synapse_federation_transaction_queue_pending_edus",
-            "",
-            [],
+        metrics.register_callback(
+            "pending_edus",
             lambda: (
                 sum(map(len, edus.values()))
                 + sum(map(len, presence.values()))
@@ -236,7 +228,7 @@ class TransactionQueue(object):
                 yield logcontext.make_deferred_yieldable(defer.gatherResults(
                     [
                         logcontext.run_in_background(handle_room_events, evs)
-                        for evs in itervalues(events_by_room)
+                        for evs in events_by_room.itervalues()
                     ],
                     consumeErrors=True
                 ))
@@ -249,15 +241,18 @@ class TransactionQueue(object):
                     now = self.clock.time_msec()
                     ts = yield self.store.get_received_ts(events[-1].event_id)
 
-                    synapse.metrics.event_processing_lag.labels(
-                        "federation_sender").set(now - ts)
-                    synapse.metrics.event_processing_last_ts.labels(
-                        "federation_sender").set(ts)
+                    synapse.metrics.event_processing_lag.set(
+                        now - ts, "federation_sender",
+                    )
+                    synapse.metrics.event_processing_last_ts.set(
+                        ts, "federation_sender",
+                    )
 
-                events_processed_counter.inc(len(events))
+                events_processed_counter.inc_by(len(events))
 
-                synapse.metrics.event_processing_positions.labels(
-                    "federation_sender").set(next_token)
+                synapse.metrics.event_processing_positions.set(
+                    next_token, "federation_sender",
+                )
 
         finally:
             self._is_processing = False
@@ -280,8 +275,7 @@ class TransactionQueue(object):
         if not destinations:
             return
 
-        sent_pdus_destination_dist_total.inc(len(destinations))
-        sent_pdus_destination_dist_count.inc()
+        sent_pdus_destination_dist.inc_by(len(destinations))
 
         for destination in destinations:
             self.pending_pdus_by_dest.setdefault(destination, []).append(
@@ -328,7 +322,7 @@ class TransactionQueue(object):
                 if not states_map:
                     break
 
-                yield self._process_presence_inner(list(states_map.values()))
+                yield self._process_presence_inner(states_map.values())
         except Exception:
             logger.exception("Error sending presence states to servers")
         finally:
@@ -452,6 +446,9 @@ class TransactionQueue(object):
             # hence why we throw the result away.
             yield get_retry_limiter(destination, self.clock, self.store)
 
+            # XXX: what's this for?
+            yield run_on_reactor()
+
             pending_pdus = []
             while True:
                 device_message_edus, device_stream_id, dev_list_id = (

synapse/federation/transport/client.py

@@ -14,14 +14,15 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from twisted.internet import defer
+from synapse.api.constants import Membership
+
+from synapse.api.urls import FEDERATION_PREFIX as PREFIX
+from synapse.util.logutils import log_function
+
 import logging
 import urllib
 
-from twisted.internet import defer
-
-from synapse.api.constants import Membership
-from synapse.api.urls import FEDERATION_PREFIX as PREFIX
-from synapse.util.logutils import log_function
 
 logger = logging.getLogger(__name__)
 

synapse/federation/transport/server.py

@@ -14,27 +14,25 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from twisted.internet import defer
+
+from synapse.api.urls import FEDERATION_PREFIX as PREFIX
+from synapse.api.errors import Codes, SynapseError, FederationDeniedError
+from synapse.http.server import JsonResource
+from synapse.http.servlet import (
+    parse_json_object_from_request, parse_integer_from_args, parse_string_from_args,
+    parse_boolean_from_args,
+)
+from synapse.util.ratelimitutils import FederationRateLimiter
+from synapse.util.versionstring import get_version_string
+from synapse.util.logcontext import run_in_background
+from synapse.types import ThirdPartyInstanceID, get_domain_from_id
+
 import functools
 import logging
 import re
-
-from twisted.internet import defer
-
 import synapse
-from synapse.api.errors import Codes, FederationDeniedError, SynapseError
-from synapse.api.urls import FEDERATION_PREFIX as PREFIX
-from synapse.http.endpoint import parse_and_validate_server_name
-from synapse.http.server import JsonResource
-from synapse.http.servlet import (
-    parse_boolean_from_args,
-    parse_integer_from_args,
-    parse_json_object_from_request,
-    parse_string_from_args,
-)
-from synapse.types import ThirdPartyInstanceID, get_domain_from_id
-from synapse.util.logcontext import run_in_background
-from synapse.util.ratelimitutils import FederationRateLimiter
-from synapse.util.versionstring import get_version_string
+
 
 logger = logging.getLogger(__name__)
 
@@ -101,6 +99,26 @@ class Authenticator(object):
 
         origin = None
 
+        def parse_auth_header(header_str):
+            try:
+                params = auth.split(" ")[1].split(",")
+                param_dict = dict(kv.split("=") for kv in params)
+
+                def strip_quotes(value):
+                    if value.startswith("\""):
+                        return value[1:-1]
+                    else:
+                        return value
+
+                origin = strip_quotes(param_dict["origin"])
+                key = strip_quotes(param_dict["key"])
+                sig = strip_quotes(param_dict["sig"])
+                return (origin, key, sig)
+            except Exception:
+                raise AuthenticationError(
+                    400, "Malformed Authorization header", Codes.UNAUTHORIZED
+                )
+
         auth_headers = request.requestHeaders.getRawHeaders(b"Authorization")
 
         if not auth_headers:
@@ -109,8 +127,8 @@ class Authenticator(object):
             )
 
         for auth in auth_headers:
-            if auth.startswith(b"X-Matrix"):
-                (origin, key, sig) = _parse_auth_header(auth)
+            if auth.startswith("X-Matrix"):
+                (origin, key, sig) = parse_auth_header(auth)
                 json_request["origin"] = origin
                 json_request["signatures"].setdefault(origin, {})[key] = sig
 
@@ -147,48 +165,6 @@ class Authenticator(object):
             logger.exception("Error resetting retry timings on %s", origin)
 
 
-def _parse_auth_header(header_bytes):
-    """Parse an X-Matrix auth header
-
-    Args:
-        header_bytes (bytes): header value
-
-    Returns:
-        Tuple[str, str, str]: origin, key id, signature.
-
-    Raises:
-        AuthenticationError if the header could not be parsed
-    """
-    try:
-        header_str = header_bytes.decode('utf-8')
-        params = header_str.split(" ")[1].split(",")
-        param_dict = dict(kv.split("=") for kv in params)
-
-        def strip_quotes(value):
-            if value.startswith(b"\""):
-                return value[1:-1]
-            else:
-                return value
-
-        origin = strip_quotes(param_dict["origin"])
-
-        # ensure that the origin is a valid server name
-        parse_and_validate_server_name(origin)
-
-        key = strip_quotes(param_dict["key"])
-        sig = strip_quotes(param_dict["sig"])
-        return origin, key, sig
-    except Exception as e:
-        logger.warn(
-            "Error parsing auth header '%s': %s",
-            header_bytes.decode('ascii', 'replace'),
-            e,
-        )
-        raise AuthenticationError(
-            400, "Malformed Authorization header", Codes.UNAUTHORIZED,
-        )
-
-
 class BaseFederationServlet(object):
     REQUIRE_AUTH = True
 
@@ -386,9 +362,7 @@ class FederationMakeJoinServlet(BaseFederationServlet):
 
     @defer.inlineCallbacks
     def on_GET(self, origin, content, query, context, user_id):
-        content = yield self.handler.on_make_join_request(
-            origin, context, user_id,
-        )
+        content = yield self.handler.on_make_join_request(context, user_id)
         defer.returnValue((200, content))
 
 
@@ -397,9 +371,7 @@ class FederationMakeLeaveServlet(BaseFederationServlet):
 
     @defer.inlineCallbacks
     def on_GET(self, origin, content, query, context, user_id):
-        content = yield self.handler.on_make_leave_request(
-            origin, context, user_id,
-        )
+        content = yield self.handler.on_make_leave_request(context, user_id)
         defer.returnValue((200, content))
 
 

synapse/federation/units.py

@@ -17,9 +17,10 @@
 server protocol.
 """
 
+from synapse.util.jsonobject import JsonEncodedObject
+
 import logging
 
-from synapse.util.jsonobject import JsonEncodedObject
 
 logger = logging.getLogger(__name__)
 

synapse/groups/attestations.py

@@ -23,9 +23,9 @@ If a user leaves (or gets kicked out of) a group, either side can still use
 their attestation to "prove" their membership, until the attestation expires.
 Therefore attestations shouldn't be relied on to prove membership in important
 cases, but can for less important situtations, e.g. showing a users membership
-of groups on their profile, showing flairs, etc.
+of groups on their profile, showing flairs, etc.abs
 
-An attestation is a signed blob of json that looks like:
+An attestsation is a signed blob of json that looks like:
 
     {
         "user_id": "@foo:a.example.com",
@@ -38,14 +38,15 @@ An attestation is a signed blob of json that looks like:
 import logging
 import random
 
-from signedjson.sign import sign_json
-
 from twisted.internet import defer
 
 from synapse.api.errors import SynapseError
 from synapse.types import get_domain_from_id
 from synapse.util.logcontext import run_in_background
 
+from signedjson.sign import sign_json
+
+
 logger = logging.getLogger(__name__)
 
 

synapse/groups/groups_server.py

@@ -16,12 +16,11 @@
 
 import logging
 
-from six import string_types
-
-from twisted.internet import defer
-
 from synapse.api.errors import SynapseError
 from synapse.types import GroupID, RoomID, UserID, get_domain_from_id
+from twisted.internet import defer
+
+from six import string_types
 
 logger = logging.getLogger(__name__)
 

synapse/handlers/__init__.py

@@ -13,13 +13,13 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from .admin import AdminHandler
-from .directory import DirectoryHandler
-from .federation import FederationHandler
-from .identity import IdentityHandler
-from .message import MessageHandler
 from .register import RegistrationHandler
 from .room import RoomContextHandler
+from .message import MessageHandler
+from .federation import FederationHandler
+from .directory import DirectoryHandler
+from .admin import AdminHandler
+from .identity import IdentityHandler
 from .search import SearchHandler
 
 

synapse/handlers/_base.py

@@ -18,10 +18,11 @@ import logging
 from twisted.internet import defer
 
 import synapse.types
-from synapse.api.constants import EventTypes, Membership
+from synapse.api.constants import Membership, EventTypes
 from synapse.api.errors import LimitExceededError
 from synapse.types import UserID
 
+
 logger = logging.getLogger(__name__)
 
 
@@ -113,14 +114,14 @@ class BaseHandler(object):
             if guest_access != "can_join":
                 if context:
                     current_state = yield self.store.get_events(
-                        list(context.current_state_ids.values())
+                        context.current_state_ids.values()
                     )
                 else:
                     current_state = yield self.state_handler.get_current_state(
                         event.room_id
                     )
 
-                current_state = list(current_state.values())
+                current_state = current_state.values()
 
                 logger.info("maybe_kick_guest_users %r", current_state)
                 yield self.kick_guest_users(current_state)

synapse/handlers/admin.py

@@ -13,12 +13,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import logging
-
 from twisted.internet import defer
 
 from ._base import BaseHandler
 
+import logging
+
 logger = logging.getLogger(__name__)
 
 

synapse/handlers/appservice.py

@@ -13,22 +13,22 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import logging
-
-from six import itervalues
-
-from prometheus_client import Counter
-
 from twisted.internet import defer
 
 import synapse
 from synapse.api.constants import EventTypes
-from synapse.util.logcontext import make_deferred_yieldable, run_in_background
 from synapse.util.metrics import Measure
+from synapse.util.logcontext import (
+    make_deferred_yieldable, run_in_background,
+)
+
+import logging
 
 logger = logging.getLogger(__name__)
 
-events_processed_counter = Counter("synapse_handlers_appservice_events_processed", "")
+metrics = synapse.metrics.get_metrics_for(__name__)
+
+events_processed_counter = metrics.register_counter("events_processed")
 
 
 def log_failure(failure):
@@ -120,7 +120,7 @@ class ApplicationServicesHandler(object):
 
                     yield make_deferred_yieldable(defer.gatherResults([
                         run_in_background(handle_room_events, evs)
-                        for evs in itervalues(events_by_room)
+                        for evs in events_by_room.itervalues()
                     ], consumeErrors=True))
 
                     yield self.store.set_appservice_last_pos(upper_bound)
@@ -128,15 +128,18 @@ class ApplicationServicesHandler(object):
                     now = self.clock.time_msec()
                     ts = yield self.store.get_received_ts(events[-1].event_id)
 
-                    synapse.metrics.event_processing_positions.labels(
-                        "appservice_sender").set(upper_bound)
+                    synapse.metrics.event_processing_positions.set(
+                        upper_bound, "appservice_sender",
+                    )
 
-                    events_processed_counter.inc(len(events))
+                    events_processed_counter.inc_by(len(events))
 
-                    synapse.metrics.event_processing_lag.labels(
-                        "appservice_sender").set(now - ts)
-                    synapse.metrics.event_processing_last_ts.labels(
-                        "appservice_sender").set(ts)
+                    synapse.metrics.event_processing_lag.set(
+                        now - ts, "appservice_sender",
+                    )
+                    synapse.metrics.event_processing_last_ts.set(
+                        ts, "appservice_sender",
+                    )
             finally:
                 self.is_processing = False
 

synapse/handlers/auth.py

@@ -13,33 +13,29 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
-import logging
-
-import attr
-import bcrypt
-import pymacaroons
-from canonicaljson import json
-
 from twisted.internet import defer, threads
-from twisted.web.client import PartialDownloadError
 
-import synapse.util.stringutils as stringutils
+from ._base import BaseHandler
 from synapse.api.constants import LoginType
 from synapse.api.errors import (
-    AuthError,
-    Codes,
-    InteractiveAuthIncompleteError,
-    LoginError,
-    StoreError,
+    AuthError, Codes, InteractiveAuthIncompleteError, LoginError, StoreError,
     SynapseError,
 )
 from synapse.module_api import ModuleApi
 from synapse.types import UserID
+from synapse.util.async import run_on_reactor
 from synapse.util.caches.expiringcache import ExpiringCache
 from synapse.util.logcontext import make_deferred_yieldable
 
-from ._base import BaseHandler
+from twisted.web.client import PartialDownloadError
+
+import logging
+import bcrypt
+import pymacaroons
+import simplejson
+
+import synapse.util.stringutils as stringutils
+
 
 logger = logging.getLogger(__name__)
 
@@ -253,7 +249,7 @@ class AuthHandler(BaseHandler):
                 errordict = e.error_dict()
 
         for f in flows:
-            if len(set(f) - set(creds)) == 0:
+            if len(set(f) - set(creds.keys())) == 0:
                 # it's very useful to know what args are stored, but this can
                 # include the password in the case of registering, so only log
                 # the keys (confusingly, clientdict may contain a password
@@ -261,12 +257,12 @@ class AuthHandler(BaseHandler):
                 # and is not sensitive).
                 logger.info(
                     "Auth completed with creds: %r. Client dict has keys: %r",
-                    creds, list(clientdict)
+                    creds, clientdict.keys()
                 )
                 defer.returnValue((creds, clientdict, session['id']))
 
         ret = self._auth_dict_for_flows(flows, session)
-        ret['completed'] = list(creds)
+        ret['completed'] = creds.keys()
         ret.update(errordict)
         raise InteractiveAuthIncompleteError(
             ret,
@@ -406,7 +402,7 @@ class AuthHandler(BaseHandler):
         except PartialDownloadError as pde:
             # Twisted is silly
             data = pde.response
-            resp_body = json.loads(data)
+            resp_body = simplejson.loads(data)
 
         if 'success' in resp_body:
             # Note that we do NOT check the hostname here: we explicitly
@@ -427,11 +423,15 @@ class AuthHandler(BaseHandler):
     def _check_msisdn(self, authdict, _):
         return self._check_threepid('msisdn', authdict)
 
+    @defer.inlineCallbacks
     def _check_dummy_auth(self, authdict, _):
-        return defer.succeed(True)
+        yield run_on_reactor()
+        defer.returnValue(True)
 
     @defer.inlineCallbacks
     def _check_threepid(self, medium, authdict):
+        yield run_on_reactor()
+
         if 'threepid_creds' not in authdict:
             raise LoginError(400, "Missing threepid_creds", Codes.MISSING_PARAM)
 
@@ -825,15 +825,6 @@ class AuthHandler(BaseHandler):
         if medium == 'email':
             address = address.lower()
 
-        identity_handler = self.hs.get_handlers().identity_handler
-        yield identity_handler.unbind_threepid(
-            user_id,
-            {
-                'medium': medium,
-                'address': address,
-            },
-        )
-
         ret = yield self.store.user_delete_threepid(
             user_id, medium, address,
         )
@@ -858,11 +849,7 @@ class AuthHandler(BaseHandler):
             return bcrypt.hashpw(password.encode('utf8') + self.hs.config.password_pepper,
                                  bcrypt.gensalt(self.bcrypt_rounds))
 
-        return make_deferred_yieldable(
-            threads.deferToThreadPool(
-                self.hs.get_reactor(), self.hs.get_reactor().getThreadPool(), _do_hash
-            ),
-        )
+        return make_deferred_yieldable(threads.deferToThread(_do_hash))
 
     def validate_hash(self, password, stored_hash):
         """Validates that self.hash(password) == stored_hash.
@@ -882,21 +869,16 @@ class AuthHandler(BaseHandler):
             )
 
         if stored_hash:
-            return make_deferred_yieldable(
-                threads.deferToThreadPool(
-                    self.hs.get_reactor(),
-                    self.hs.get_reactor().getThreadPool(),
-                    _do_validate_hash,
-                ),
-            )
+            return make_deferred_yieldable(threads.deferToThread(_do_validate_hash))
         else:
             return defer.succeed(False)
 
 
-@attr.s
-class MacaroonGenerator(object):
-
-    hs = attr.ib()
+class MacaroonGeneartor(object):
+    def __init__(self, hs):
+        self.clock = hs.get_clock()
+        self.server_name = hs.config.server_name
+        self.macaroon_secret_key = hs.config.macaroon_secret_key
 
     def generate_access_token(self, user_id, extra_caveats=None):
         extra_caveats = extra_caveats or []
@@ -914,7 +896,7 @@ class MacaroonGenerator(object):
     def generate_short_term_login_token(self, user_id, duration_in_ms=(2 * 60 * 1000)):
         macaroon = self._generate_base_macaroon(user_id)
         macaroon.add_first_party_caveat("type = login")
-        now = self.hs.get_clock().time_msec()
+        now = self.clock.time_msec()
         expiry = now + duration_in_ms
         macaroon.add_first_party_caveat("time < %d" % (expiry,))
         return macaroon.serialize()
@@ -926,9 +908,9 @@ class MacaroonGenerator(object):
 
     def _generate_base_macaroon(self, user_id):
         macaroon = pymacaroons.Macaroon(
-            location=self.hs.config.server_name,
+            location=self.server_name,
             identifier="key",
-            key=self.hs.config.macaroon_secret_key)
+            key=self.macaroon_secret_key)
         macaroon.add_first_party_caveat("gen = 1")
         macaroon.add_first_party_caveat("user_id = %s" % (user_id,))
         return macaroon

synapse/handlers/deactivate_account.py

@@ -12,15 +12,13 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
+from twisted.internet import defer, reactor
 
-from twisted.internet import defer
-
-from synapse.api.errors import SynapseError
+from ._base import BaseHandler
 from synapse.types import UserID, create_requester
 from synapse.util.logcontext import run_in_background
 
-from ._base import BaseHandler
+import logging
 
 logger = logging.getLogger(__name__)
 
@@ -32,7 +30,6 @@ class DeactivateAccountHandler(BaseHandler):
         self._auth_handler = hs.get_auth_handler()
         self._device_handler = hs.get_device_handler()
         self._room_member_handler = hs.get_room_member_handler()
-        self._identity_handler = hs.get_handlers().identity_handler
         self.user_directory_handler = hs.get_user_directory_handler()
 
         # Flag that indicates whether the process to part users from rooms is running
@@ -40,15 +37,14 @@ class DeactivateAccountHandler(BaseHandler):
 
         # Start the user parter loop so it can resume parting users from rooms where
         # it left off (if it has work left to do).
-        hs.get_reactor().callWhenRunning(self._start_user_parting)
+        reactor.callWhenRunning(self._start_user_parting)
 
     @defer.inlineCallbacks
-    def deactivate_account(self, user_id, erase_data):
+    def deactivate_account(self, user_id):
         """Deactivate a user's account
 
         Args:
             user_id (str): ID of user to be deactivated
-            erase_data (bool): whether to GDPR-erase the user's data
 
         Returns:
             Deferred
@@ -56,35 +52,14 @@ class DeactivateAccountHandler(BaseHandler):
         # FIXME: Theoretically there is a race here wherein user resets
         # password using threepid.
 
-        # delete threepids first. We remove these from the IS so if this fails,
-        # leave the user still active so they can try again.
-        # Ideally we would prevent password resets and then do this in the
-        # background thread.
-        threepids = yield self.store.user_get_threepids(user_id)
-        for threepid in threepids:
-            try:
-                yield self._identity_handler.unbind_threepid(
-                    user_id,
-                    {
-                        'medium': threepid['medium'],
-                        'address': threepid['address'],
-                    },
-                )
-            except Exception:
-                # Do we want this to be a fatal error or should we carry on?
-                logger.exception("Failed to remove threepid from ID server")
-                raise SynapseError(400, "Failed to remove threepid from ID server")
-            yield self.store.user_delete_threepid(
-                user_id, threepid['medium'], threepid['address'],
-            )
-
-        # delete any devices belonging to the user, which will also
+        # first delete any devices belonging to the user, which will also
         # delete corresponding access tokens.
         yield self._device_handler.delete_all_devices_for_user(user_id)
         # then delete any remaining access tokens which weren't associated with
         # a device.
         yield self._auth_handler.delete_access_tokens_for_user(user_id)
 
+        yield self.store.user_delete_threepids(user_id)
         yield self.store.user_set_password_hash(user_id, None)
 
         # Add the user to a table of users pending deactivation (ie.
@@ -94,11 +69,6 @@ class DeactivateAccountHandler(BaseHandler):
         # delete from user directory
         yield self.user_directory_handler.handle_user_deactivated(user_id)
 
-        # Mark the user as erased, if they asked for that
-        if erase_data:
-            logger.info("Marking %s as erased", user_id)
-            yield self.store.mark_user_erased(user_id)
-
         # Now start the process that goes through that list and
         # parts users from rooms (if it isn't already running)
         self._start_user_parting()

synapse/handlers/device.py

@@ -12,24 +12,22 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
-
-from six import iteritems, itervalues
-
-from twisted.internet import defer
-
 from synapse.api import errors
 from synapse.api.constants import EventTypes
 from synapse.api.errors import FederationDeniedError
-from synapse.types import RoomStreamToken, get_domain_from_id
 from synapse.util import stringutils
 from synapse.util.async import Linearizer
 from synapse.util.caches.expiringcache import ExpiringCache
-from synapse.util.metrics import measure_func
 from synapse.util.retryutils import NotRetryingDestination
-
+from synapse.util.metrics import measure_func
+from synapse.types import get_domain_from_id, RoomStreamToken
+from twisted.internet import defer
 from ._base import BaseHandler
 
+import logging
+
+from six import itervalues, iteritems
+
 logger = logging.getLogger(__name__)
 
 
@@ -116,7 +114,7 @@ class DeviceHandler(BaseHandler):
             user_id, device_id=None
         )
 
-        devices = list(device_map.values())
+        devices = device_map.values()
         for device in devices:
             _update_device_from_client_ips(device, ips)
 
@@ -189,7 +187,7 @@ class DeviceHandler(BaseHandler):
             defer.Deferred:
         """
         device_map = yield self.store.get_devices_by_user(user_id)
-        device_ids = list(device_map)
+        device_ids = device_map.keys()
         if except_device_id is not None:
             device_ids = [d for d in device_ids if d != except_device_id]
         yield self.delete_devices(user_id, device_ids)
@@ -539,7 +537,7 @@ class DeviceListEduUpdater(object):
                 yield self.device_handler.notify_device_update(user_id, device_ids)
             else:
                 # Simply update the single device, since we know that is the only
-                # change (because of the single prev_id matching the current cache)
+                # change (becuase of the single prev_id matching the current cache)
                 for device_id, stream_id, prev_ids, content in pending_updates:
                     yield self.store.update_remote_device_list_cache_entry(
                         user_id, device_id, content, stream_id,

synapse/handlers/devicemessage.py

@@ -18,9 +18,10 @@ import logging
 from twisted.internet import defer
 
 from synapse.api.errors import SynapseError
-from synapse.types import UserID, get_domain_from_id
+from synapse.types import get_domain_from_id, UserID
 from synapse.util.stringutils import random_string
 
+
 logger = logging.getLogger(__name__)
 
 

synapse/handlers/directory.py

@@ -14,16 +14,15 @@
 # limitations under the License.
 
 
-import logging
-import string
-
 from twisted.internet import defer
+from ._base import BaseHandler
 
+from synapse.api.errors import SynapseError, Codes, CodeMessageException, AuthError
 from synapse.api.constants import EventTypes
-from synapse.api.errors import AuthError, CodeMessageException, Codes, SynapseError
 from synapse.types import RoomAlias, UserID, get_domain_from_id
 
-from ._base import BaseHandler
+import logging
+import string
 
 logger = logging.getLogger(__name__)
 

synapse/handlers/e2e_keys.py

@@ -14,16 +14,17 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import simplejson as json
 import logging
 
+from canonicaljson import encode_canonical_json
+from twisted.internet import defer
 from six import iteritems
 
-from canonicaljson import encode_canonical_json, json
-
-from twisted.internet import defer
-
-from synapse.api.errors import CodeMessageException, FederationDeniedError, SynapseError
-from synapse.types import UserID, get_domain_from_id
+from synapse.api.errors import (
+    SynapseError, CodeMessageException, FederationDeniedError,
+)
+from synapse.types import get_domain_from_id, UserID
 from synapse.util.logcontext import make_deferred_yieldable, run_in_background
 from synapse.util.retryutils import NotRetryingDestination
 
@@ -79,7 +80,7 @@ class E2eKeysHandler(object):
             else:
                 remote_queries[user_id] = device_ids
 
-        # First get local devices.
+        # Firt get local devices.
         failures = {}
         results = {}
         if local_query:
@@ -356,7 +357,7 @@ def _exception_to_failure(e):
     # include ConnectionRefused and other errors
     #
     # Note that some Exceptions (notably twisted's ResponseFailed etc) don't
-    # give a string for e.message, which json then fails to serialize.
+    # give a string for e.message, which simplejson then fails to serialize.
     return {
         "status": 503, "message": str(e.message),
     }

synapse/handlers/events.py

@@ -13,20 +13,19 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from twisted.internet import defer
+
+from synapse.util.logutils import log_function
+from synapse.types import UserID
+from synapse.events.utils import serialize_event
+from synapse.api.constants import Membership, EventTypes
+from synapse.events import EventBase
+
+from ._base import BaseHandler
+
 import logging
 import random
 
-from twisted.internet import defer
-
-from synapse.api.constants import EventTypes, Membership
-from synapse.api.errors import AuthError
-from synapse.events import EventBase
-from synapse.events.utils import serialize_event
-from synapse.types import UserID
-from synapse.util.logutils import log_function
-from synapse.visibility import filter_events_for_client
-
-from ._base import BaseHandler
 
 logger = logging.getLogger(__name__)
 
@@ -131,13 +130,11 @@ class EventStreamHandler(BaseHandler):
 class EventHandler(BaseHandler):
 
     @defer.inlineCallbacks
-    def get_event(self, user, room_id, event_id):
+    def get_event(self, user, event_id):
         """Retrieve a single specified event.
 
         Args:
             user (synapse.types.UserID): The user requesting the event
-            room_id (str|None): The expected room id. We'll return None if the
-                event's room does not match.
             event_id (str): The event ID to obtain.
         Returns:
             dict: An event, or None if there is no event matching this ID.
@@ -146,26 +143,13 @@ class EventHandler(BaseHandler):
             AuthError if the user does not have the rights to inspect this
             event.
         """
-        event = yield self.store.get_event(event_id, check_room_id=room_id)
+        event = yield self.store.get_event(event_id)
 
         if not event:
             defer.returnValue(None)
             return
 
-        users = yield self.store.get_users_in_room(event.room_id)
-        is_peeking = user.to_string() not in users
-
-        filtered = yield filter_events_for_client(
-            self.store,
-            user.to_string(),
-            [event],
-            is_peeking=is_peeking
-        )
-
-        if not filtered:
-            raise AuthError(
-                403,
-                "You don't have permission to access that event."
-            )
+        if hasattr(event, "room_id"):
+            yield self.auth.check_joined_room(event.room_id, user.to_string())
 
         defer.returnValue(event)

synapse/handlers/federation.py

@@ -20,42 +20,39 @@ import itertools
 import logging
 import sys
 
-import six
-from six import iteritems
-from six.moves import http_client
-
 from signedjson.key import decode_verify_key_bytes
 from signedjson.sign import verify_signed_json
+import six
+from six.moves import http_client
+from six import iteritems
+from twisted.internet import defer
 from unpaddedbase64 import decode_base64
 
-from twisted.internet import defer
-
-from synapse.api.constants import EventTypes, Membership, RejectedReason
-from synapse.api.errors import (
-    AuthError,
-    CodeMessageException,
-    FederationDeniedError,
-    FederationError,
-    StoreError,
-    SynapseError,
-)
-from synapse.crypto.event_signing import (
-    add_hashes_and_signatures,
-    compute_event_signature,
-)
-from synapse.events.validator import EventValidator
-from synapse.state import resolve_events_with_factory
-from synapse.types import UserID, get_domain_from_id
-from synapse.util import logcontext, unwrapFirstError
-from synapse.util.async import Linearizer
-from synapse.util.distributor import user_joined_room
-from synapse.util.frozenutils import unfreeze
-from synapse.util.logutils import log_function
-from synapse.util.retryutils import NotRetryingDestination
-from synapse.visibility import filter_events_for_server
-
 from ._base import BaseHandler
 
+from synapse.api.errors import (
+    AuthError, FederationError, StoreError, CodeMessageException, SynapseError,
+    FederationDeniedError,
+)
+from synapse.api.constants import EventTypes, Membership, RejectedReason
+from synapse.events.validator import EventValidator
+from synapse.util import unwrapFirstError, logcontext
+from synapse.util.metrics import measure_func
+from synapse.util.logutils import log_function
+from synapse.util.async import run_on_reactor, Linearizer
+from synapse.util.frozenutils import unfreeze
+from synapse.crypto.event_signing import (
+    compute_event_signature, add_hashes_and_signatures,
+)
+from synapse.types import UserID, get_domain_from_id
+
+from synapse.events.utils import prune_event
+
+from synapse.util.retryutils import NotRetryingDestination
+
+from synapse.util.distributor import user_joined_room
+
+
 logger = logging.getLogger(__name__)
 
 
@@ -93,9 +90,7 @@ class FederationHandler(BaseHandler):
 
     @defer.inlineCallbacks
     @log_function
-    def on_receive_pdu(
-            self, origin, pdu, get_missing=True, sent_to_us_directly=False,
-    ):
+    def on_receive_pdu(self, origin, pdu, get_missing=True):
         """ Process a PDU received via a federation /send/ transaction, or
         via backfill of missing prev_events
 
@@ -109,10 +104,8 @@ class FederationHandler(BaseHandler):
         """
 
         # We reprocess pdus when we have seen them only as outliers
-        existing = yield self.store.get_event(
-            pdu.event_id,
-            allow_none=True,
-            allow_rejected=True,
+        existing = yield self.get_persisted_pdu(
+            origin, pdu.event_id, do_auth=False
         )
 
         # FIXME: Currently we fetch an event again when we already have it
@@ -169,11 +162,14 @@ class FederationHandler(BaseHandler):
                     "Ignoring PDU %s for room %s from %s as we've left the room!",
                     pdu.event_id, pdu.room_id, origin,
                 )
-                defer.returnValue(None)
+                return
 
         state = None
+
         auth_chain = []
 
+        fetch_state = False
+
         # Get missing pdus if necessary.
         if not pdu.internal_metadata.is_outlier():
             # We only backfill backwards to the min depth.
@@ -228,60 +224,26 @@ class FederationHandler(BaseHandler):
                         list(prevs - seen)[:5],
                     )
 
-            if sent_to_us_directly and prevs - seen:
-                # If they have sent it to us directly, and the server
-                # isn't telling us about the auth events that it's
-                # made a message referencing, we explode
-                raise FederationError(
-                    "ERROR",
-                    403,
-                    (
-                        "Your server isn't divulging details about prev_events "
-                        "referenced in this event."
-                    ),
-                    affected=pdu.event_id,
+            if prevs - seen:
+                logger.info(
+                    "Still missing %d events for room %r: %r...",
+                    len(prevs - seen), pdu.room_id, list(prevs - seen)[:5]
                 )
-            elif prevs - seen:
-                # Calculate the state of the previous events, and
-                # de-conflict them to find the current state.
-                state_groups = []
-                auth_chains = set()
-                try:
-                    # Get the state of the events we know about
-                    ours = yield self.store.get_state_groups(pdu.room_id, list(seen))
-                    state_groups.append(ours)
+                fetch_state = True
 
-                    # Ask the remote server for the states we don't
-                    # know about
-                    for p in prevs - seen:
-                        state, got_auth_chain = (
-                            yield self.replication_layer.get_state_for_room(
-                                origin, pdu.room_id, p
-                            )
-                        )
-                        auth_chains.update(got_auth_chain)
-                        state_group = {(x.type, x.state_key): x.event_id for x in state}
-                        state_groups.append(state_group)
-
-                    # Resolve any conflicting state
-                    def fetch(ev_ids):
-                        return self.store.get_events(
-                            ev_ids, get_prev_content=False, check_redacted=False
-                        )
-
-                    state_map = yield resolve_events_with_factory(
-                        state_groups, {pdu.event_id: pdu}, fetch
-                    )
-
-                    state = (yield self.store.get_events(state_map.values())).values()
-                    auth_chain = list(auth_chains)
-                except Exception:
-                    raise FederationError(
-                        "ERROR",
-                        403,
-                        "We can't get valid state history.",
-                        affected=pdu.event_id,
-                    )
+        if fetch_state:
+            # We need to get the state at this event, since we haven't
+            # processed all the prev events.
+            logger.debug(
+                "_handle_new_pdu getting state for %s",
+                pdu.room_id
+            )
+            try:
+                state, auth_chain = yield self.replication_layer.get_state_for_room(
+                    origin, pdu.room_id, pdu.event_id,
+                )
+            except Exception:
+                logger.exception("Failed to get state for event: %s", pdu.event_id)
 
         yield self._process_received_pdu(
             origin,
@@ -359,17 +321,11 @@ class FederationHandler(BaseHandler):
 
         for e in missing_events:
             logger.info("Handling found event %s", e.event_id)
-            try:
-                yield self.on_receive_pdu(
-                    origin,
-                    e,
-                    get_missing=False
-                )
-            except FederationError as e:
-                if e.code == 403:
-                    logger.warn("Event %s failed history check.")
-                else:
-                    raise
+            yield self.on_receive_pdu(
+                origin,
+                e,
+                get_missing=False
+            )
 
     @log_function
     @defer.inlineCallbacks
@@ -500,6 +456,83 @@ class FederationHandler(BaseHandler):
                     user = UserID.from_string(event.state_key)
                     yield user_joined_room(self.distributor, user, event.room_id)
 
+    @measure_func("_filter_events_for_server")
+    @defer.inlineCallbacks
+    def _filter_events_for_server(self, server_name, room_id, events):
+        event_to_state_ids = yield self.store.get_state_ids_for_events(
+            frozenset(e.event_id for e in events),
+            types=(
+                (EventTypes.RoomHistoryVisibility, ""),
+                (EventTypes.Member, None),
+            )
+        )
+
+        # We only want to pull out member events that correspond to the
+        # server's domain.
+
+        def check_match(id):
+            try:
+                return server_name == get_domain_from_id(id)
+            except Exception:
+                return False
+
+        # Parses mapping `event_id -> (type, state_key) -> state event_id`
+        # to get all state ids that we're interested in.
+        event_map = yield self.store.get_events([
+            e_id
+            for key_to_eid in event_to_state_ids.itervalues()
+            for key, e_id in key_to_eid.iteritems()
+            if key[0] != EventTypes.Member or check_match(key[1])
+        ])
+
+        event_to_state = {
+            e_id: {
+                key: event_map[inner_e_id]
+                for key, inner_e_id in key_to_eid.iteritems()
+                if inner_e_id in event_map
+            }
+            for e_id, key_to_eid in event_to_state_ids.iteritems()
+        }
+
+        def redact_disallowed(event, state):
+            if not state:
+                return event
+
+            history = state.get((EventTypes.RoomHistoryVisibility, ''), None)
+            if history:
+                visibility = history.content.get("history_visibility", "shared")
+                if visibility in ["invited", "joined"]:
+                    # We now loop through all state events looking for
+                    # membership states for the requesting server to determine
+                    # if the server is either in the room or has been invited
+                    # into the room.
+                    for ev in state.itervalues():
+                        if ev.type != EventTypes.Member:
+                            continue
+                        try:
+                            domain = get_domain_from_id(ev.state_key)
+                        except Exception:
+                            continue
+
+                        if domain != server_name:
+                            continue
+
+                        memtype = ev.membership
+                        if memtype == Membership.JOIN:
+                            return event
+                        elif memtype == Membership.INVITE:
+                            if visibility == "invited":
+                                return event
+                    else:
+                        return prune_event(event)
+
+            return event
+
+        defer.returnValue([
+            redact_disallowed(e, event_to_state[e.event_id])
+            for e in events
+        ])
+
     @log_function
     @defer.inlineCallbacks
     def backfill(self, dest, room_id, limit, extremities):
@@ -1116,13 +1149,13 @@ class FederationHandler(BaseHandler):
                 user = UserID.from_string(event.state_key)
                 yield user_joined_room(self.distributor, user, event.room_id)
 
-        state_ids = list(context.prev_state_ids.values())
+        state_ids = context.prev_state_ids.values()
         auth_chain = yield self.store.get_auth_chain(state_ids)
 
-        state = yield self.store.get_events(list(context.prev_state_ids.values()))
+        state = yield self.store.get_events(context.prev_state_ids.values())
 
         defer.returnValue({
-            "state": list(state.values()),
+            "state": state.values(),
             "auth_chain": auth_chain,
         })
 
@@ -1349,10 +1382,7 @@ class FederationHandler(BaseHandler):
     def get_state_for_pdu(self, room_id, event_id):
         """Returns the state at the event. i.e. not including said event.
         """
-
-        event = yield self.store.get_event(
-            event_id, allow_none=False, check_room_id=room_id,
-        )
+        yield run_on_reactor()
 
         state_groups = yield self.store.get_state_groups(
             room_id, [event_id]
@@ -1364,7 +1394,8 @@ class FederationHandler(BaseHandler):
                 (e.type, e.state_key): e for e in state
             }
 
-            if event.is_state():
+            event = yield self.store.get_event(event_id)
+            if event and event.is_state():
                 # Get previous state
                 if "replaces_state" in event.unsigned:
                     prev_id = event.unsigned["replaces_state"]
@@ -1374,7 +1405,7 @@ class FederationHandler(BaseHandler):
                 else:
                     del results[(event.type, event.state_key)]
 
-            res = list(results.values())
+            res = results.values()
             for event in res:
                 # We sign these again because there was a bug where we
                 # incorrectly signed things the first time round
@@ -1395,9 +1426,7 @@ class FederationHandler(BaseHandler):
     def get_state_ids_for_pdu(self, room_id, event_id):
         """Returns the state at the event. i.e. not including said event.
         """
-        event = yield self.store.get_event(
-            event_id, allow_none=False, check_room_id=room_id,
-        )
+        yield run_on_reactor()
 
         state_groups = yield self.store.get_state_groups_ids(
             room_id, [event_id]
@@ -1407,7 +1436,8 @@ class FederationHandler(BaseHandler):
             _, state = state_groups.items().pop()
             results = state
 
-            if event.is_state():
+            event = yield self.store.get_event(event_id)
+            if event and event.is_state():
                 # Get previous state
                 if "replaces_state" in event.unsigned:
                     prev_id = event.unsigned["replaces_state"]
@@ -1416,7 +1446,7 @@ class FederationHandler(BaseHandler):
                 else:
                     results.pop((event.type, event.state_key), None)
 
-            defer.returnValue(list(results.values()))
+            defer.returnValue(results.values())
         else:
             defer.returnValue([])
 
@@ -1433,26 +1463,17 @@ class FederationHandler(BaseHandler):
             limit
         )
 
-        events = yield filter_events_for_server(self.store, origin, events)
+        events = yield self._filter_events_for_server(origin, room_id, events)
 
         defer.returnValue(events)
 
     @defer.inlineCallbacks
     @log_function
-    def get_persisted_pdu(self, origin, event_id):
-        """Get an event from the database for the given server.
-
-        Args:
-            origin [str]: hostname of server which is requesting the event; we
-               will check that the server is allowed to see it.
-            event_id [str]: id of the event being requested
+    def get_persisted_pdu(self, origin, event_id, do_auth=True):
+        """ Get a PDU from the database with given origin and id.
 
         Returns:
-            Deferred[EventBase|None]: None if we know nothing about the event;
-                otherwise the (possibly-redacted) event.
-
-        Raises:
-            AuthError if the server is not currently in the room
+            Deferred: Results in a `Pdu`.
         """
         event = yield self.store.get_event(
             event_id,
@@ -1473,17 +1494,20 @@ class FederationHandler(BaseHandler):
                     )
                 )
 
-            in_room = yield self.auth.check_host_in_room(
-                event.room_id,
-                origin
-            )
-            if not in_room:
-                raise AuthError(403, "Host not in room.")
+            if do_auth:
+                in_room = yield self.auth.check_host_in_room(
+                    event.room_id,
+                    origin
+                )
+                if not in_room:
+                    raise AuthError(403, "Host not in room.")
+
+                events = yield self._filter_events_for_server(
+                    origin, event.room_id, [event]
+                )
+
+                event = events[0]
 
-            events = yield filter_events_for_server(
-                self.store, origin, [event],
-            )
-            event = events[0]
             defer.returnValue(event)
         else:
             defer.returnValue(None)
@@ -1713,19 +1737,8 @@ class FederationHandler(BaseHandler):
         defer.returnValue(context)
 
     @defer.inlineCallbacks
-    def on_query_auth(self, origin, event_id, room_id, remote_auth_chain, rejects,
+    def on_query_auth(self, origin, event_id, remote_auth_chain, rejects,
                       missing):
-        in_room = yield self.auth.check_host_in_room(
-            room_id,
-            origin
-        )
-        if not in_room:
-            raise AuthError(403, "Host not in room.")
-
-        event = yield self.store.get_event(
-            event_id, allow_none=False, check_room_id=room_id
-        )
-
         # Just go through and process each event in `remote_auth_chain`. We
         # don't want to fall into the trap of `missing` being wrong.
         for e in remote_auth_chain:
@@ -1735,6 +1748,7 @@ class FederationHandler(BaseHandler):
                 pass
 
         # Now get the current auth_chain for the event.
+        event = yield self.store.get_event(event_id)
         local_auth_chain = yield self.store.get_auth_chain(
             [auth_id for auth_id, _ in event.auth_events],
             include_given=True
@@ -1781,10 +1795,6 @@ class FederationHandler(BaseHandler):
             min_depth=min_depth,
         )
 
-        missing_events = yield filter_events_for_server(
-            self.store, origin, missing_events,
-        )
-
         defer.returnValue(missing_events)
 
     @defer.inlineCallbacks
@@ -1905,7 +1915,7 @@ class FederationHandler(BaseHandler):
                 })
 
                 new_state = self.state_handler.resolve_events(
-                    [list(local_view.values()), list(remote_view.values())],
+                    [local_view.values(), remote_view.values()],
                     event
                 )
 

synapse/handlers/groups_local.py

@@ -14,15 +14,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import logging
-
-from six import iteritems
-
 from twisted.internet import defer
+from six import iteritems
 
 from synapse.api.errors import SynapseError
 from synapse.types import get_domain_from_id
 
+import logging
+
 logger = logging.getLogger(__name__)
 
 

synapse/handlers/identity.py

@@ -1,7 +1,6 @@
 # -*- coding: utf-8 -*-
 # Copyright 2015, 2016 OpenMarket Ltd
 # Copyright 2017 Vector Creations Ltd
-# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -19,18 +18,16 @@
 
 import logging
 
-from canonicaljson import json
+import simplejson as json
 
 from twisted.internet import defer
 
 from synapse.api.errors import (
-    CodeMessageException,
-    Codes,
-    MatrixCodeMessageException,
-    SynapseError,
+    MatrixCodeMessageException, CodeMessageException
 )
-
 from ._base import BaseHandler
+from synapse.util.async import run_on_reactor
+from synapse.api.errors import SynapseError, Codes
 
 logger = logging.getLogger(__name__)
 
@@ -41,7 +38,6 @@ class IdentityHandler(BaseHandler):
         super(IdentityHandler, self).__init__(hs)
 
         self.http_client = hs.get_simple_http_client()
-        self.federation_http_client = hs.get_http_client()
 
         self.trusted_id_servers = set(hs.config.trusted_third_party_id_servers)
         self.trust_any_id_server_just_for_testing_do_not_use = (
@@ -64,6 +60,8 @@ class IdentityHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def threepid_from_creds(self, creds):
+        yield run_on_reactor()
+
         if 'id_server' in creds:
             id_server = creds['id_server']
         elif 'idServer' in creds:
@@ -106,6 +104,7 @@ class IdentityHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def bind_threepid(self, creds, mxid):
+        yield run_on_reactor()
         logger.debug("binding threepid %r to %s", creds, mxid)
         data = None
 
@@ -139,54 +138,10 @@ class IdentityHandler(BaseHandler):
             data = json.loads(e.msg)
         defer.returnValue(data)
 
-    @defer.inlineCallbacks
-    def unbind_threepid(self, mxid, threepid):
-        """
-        Removes a binding from an identity server
-        Args:
-            mxid (str): Matrix user ID of binding to be removed
-            threepid (dict): Dict with medium & address of binding to be removed
-
-        Returns:
-            Deferred[bool]: True on success, otherwise False
-        """
-        logger.debug("unbinding threepid %r from %s", threepid, mxid)
-        if not self.trusted_id_servers:
-            logger.warn("Can't unbind threepid: no trusted ID servers set in config")
-            defer.returnValue(False)
-
-        # We don't track what ID server we added 3pids on (perhaps we ought to)
-        # but we assume that any of the servers in the trusted list are in the
-        # same ID server federation, so we can pick any one of them to send the
-        # deletion request to.
-        id_server = next(iter(self.trusted_id_servers))
-
-        url = "https://%s/_matrix/identity/api/v1/3pid/unbind" % (id_server,)
-        content = {
-            "mxid": mxid,
-            "threepid": threepid,
-        }
-        headers = {}
-        # we abuse the federation http client to sign the request, but we have to send it
-        # using the normal http client since we don't want the SRV lookup and want normal
-        # 'browser-like' HTTPS.
-        self.federation_http_client.sign_request(
-            destination=None,
-            method='POST',
-            url_bytes='/_matrix/identity/api/v1/3pid/unbind'.encode('ascii'),
-            headers_dict=headers,
-            content=content,
-            destination_is=id_server,
-        )
-        yield self.http_client.post_json_get_json(
-            url,
-            content,
-            headers,
-        )
-        defer.returnValue(True)
-
     @defer.inlineCallbacks
     def requestEmailToken(self, id_server, email, client_secret, send_attempt, **kwargs):
+        yield run_on_reactor()
+
         if not self._should_trust_id_server(id_server):
             raise SynapseError(
                 400, "Untrusted ID server '%s'" % id_server,
@@ -221,6 +176,8 @@ class IdentityHandler(BaseHandler):
             self, id_server, country, phone_number,
             client_secret, send_attempt, **kwargs
     ):
+        yield run_on_reactor()
+
         if not self._should_trust_id_server(id_server):
             raise SynapseError(
                 400, "Untrusted ID server '%s'" % id_server,

synapse/handlers/initial_sync.py

@@ -13,8 +13,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import logging
-
 from twisted.internet import defer
 
 from synapse.api.constants import EventTypes, Membership
@@ -23,7 +21,9 @@ from synapse.events.utils import serialize_event
 from synapse.events.validator import EventValidator
 from synapse.handlers.presence import format_user_presence_state
 from synapse.streams.config import PaginationConfig
-from synapse.types import StreamToken, UserID
+from synapse.types import (
+    UserID, StreamToken,
+)
 from synapse.util import unwrapFirstError
 from synapse.util.async import concurrently_execute
 from synapse.util.caches.snapshot_cache import SnapshotCache
@@ -32,6 +32,9 @@ from synapse.visibility import filter_events_for_client
 
 from ._base import BaseHandler
 
+import logging
+
+
 logger = logging.getLogger(__name__)
 
 

synapse/handlers/message.py

@@ -14,31 +14,35 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import logging
+import simplejson
 import sys
 
+from canonicaljson import encode_canonical_json
 import six
-from six import iteritems, itervalues, string_types
-
-from canonicaljson import encode_canonical_json, json
-
-from twisted.internet import defer
+from six import string_types, itervalues, iteritems
+from twisted.internet import defer, reactor
 from twisted.internet.defer import succeed
 from twisted.python.failure import Failure
 
-from synapse.api.constants import MAX_DEPTH, EventTypes, Membership
-from synapse.api.errors import AuthError, Codes, ConsentNotGivenError, SynapseError
+from synapse.api.constants import EventTypes, Membership, MAX_DEPTH
+from synapse.api.errors import (
+    AuthError, Codes, SynapseError,
+    ConsentNotGivenError,
+)
 from synapse.api.urls import ConsentURIBuilder
 from synapse.crypto.event_signing import add_hashes_and_signatures
 from synapse.events.utils import serialize_event
 from synapse.events.validator import EventValidator
-from synapse.replication.http.send_event import send_event_to_master
-from synapse.types import RoomAlias, RoomStreamToken, UserID
-from synapse.util.async import Limiter, ReadWriteLock
-from synapse.util.frozenutils import frozendict_json_encoder
+from synapse.types import (
+    UserID, RoomAlias, RoomStreamToken,
+)
+from synapse.util.async import run_on_reactor, ReadWriteLock, Limiter
 from synapse.util.logcontext import run_in_background
 from synapse.util.metrics import measure_func
+from synapse.util.frozenutils import frozendict_json_encoder
 from synapse.util.stringutils import random_string
 from synapse.visibility import filter_events_for_client
+from synapse.replication.http.send_event import send_event_to_master
 
 from ._base import BaseHandler
 
@@ -153,7 +157,7 @@ class MessageHandler(BaseHandler):
             # remove the purge from the list 24 hours after it completes
             def clear_purge():
                 del self._purges_by_id[purge_id]
-            self.hs.get_reactor().callLater(24 * 3600, clear_purge)
+            reactor.callLater(24 * 3600, clear_purge)
 
     def get_purge_status(self, purge_id):
         """Get the current status of an active purge
@@ -231,7 +235,7 @@ class MessageHandler(BaseHandler):
                     room_id, max_topo
                 )
 
-            events, next_key = yield self.store.paginate_room_events(
+            events, next_key, extremities = yield self.store.paginate_room_events(
                 room_id=room_id,
                 from_key=source_config.from_key,
                 to_key=source_config.to_key,
@@ -384,7 +388,7 @@ class MessageHandler(BaseHandler):
         users_with_profile = yield self.state.get_current_user_in_room(room_id)
 
         # If this is an AS, double check that they are allowed to see the members.
-        # This can either be because the AS user is in the room or because there
+        # This can either be because the AS user is in the room or becuase there
         # is a user in the room that the AS is "interested in"
         if requester.app_service and user_id not in users_with_profile:
             for uid in users_with_profile:
@@ -487,7 +491,7 @@ class EventCreationHandler(object):
                         target, e
                     )
 
-        is_exempt = yield self._is_exempt_from_privacy_policy(builder, requester)
+        is_exempt = yield self._is_exempt_from_privacy_policy(builder)
         if not is_exempt:
             yield self.assert_accepted_privacy_policy(requester)
 
@@ -505,13 +509,12 @@ class EventCreationHandler(object):
 
         defer.returnValue((event, context))
 
-    def _is_exempt_from_privacy_policy(self, builder, requester):
+    def _is_exempt_from_privacy_policy(self, builder):
         """"Determine if an event to be sent is exempt from having to consent
         to the privacy policy
 
         Args:
             builder (synapse.events.builder.EventBuilder): event being created
-            requester (Requster): user requesting this event
 
         Returns:
             Deferred[bool]: true if the event can be sent without the user
@@ -522,9 +525,6 @@ class EventCreationHandler(object):
             membership = builder.content.get("membership", None)
             if membership == Membership.JOIN:
                 return self._is_server_notices_room(builder.room_id)
-            elif membership == Membership.LEAVE:
-                # the user is always allowed to leave (but not kick people)
-                return builder.state_key == requester.user.to_string()
         return succeed(False)
 
     @defer.inlineCallbacks
@@ -572,9 +572,6 @@ class EventCreationHandler(object):
 
         u = yield self.store.get_user_by_id(user_id)
         assert u is not None
-        if u["appservice_id"] is not None:
-            # users registered by an appservice are exempt
-            return
         if u["consent_version"] == self.config.user_consent_version:
             return
 
@@ -793,7 +790,7 @@ class EventCreationHandler(object):
         # Ensure that we can round trip before trying to persist in db
         try:
             dump = frozendict_json_encoder.encode(event.content)
-            json.loads(dump)
+            simplejson.loads(dump)
         except Exception:
             logger.exception("Failed to encode content: %r", event.content)
             raise
@@ -806,7 +803,6 @@ class EventCreationHandler(object):
             # If we're a worker we need to hit out to the master.
             if self.config.worker_app:
                 yield send_event_to_master(
-                    self.hs.get_clock(),
                     self.http_client,
                     host=self.config.worker_replication_host,
                     port=self.config.worker_replication_http_port,
@@ -960,7 +956,9 @@ class EventCreationHandler(object):
             event_stream_id, max_stream_id
         )
 
+        @defer.inlineCallbacks
         def _notify():
+            yield run_on_reactor()
             try:
                 self.notifier.on_new_room_event(
                     event, event_stream_id, max_stream_id,

synapse/handlers/presence.py

@@ -22,44 +22,42 @@ The methods that define policy are:
     - should_notify
 """
 
-import logging
+from twisted.internet import defer, reactor
 from contextlib import contextmanager
 
-from six import iteritems, itervalues
+from six import itervalues, iteritems
 
-from prometheus_client import Counter
-
-from twisted.internet import defer
-
-from synapse.api.constants import PresenceState
 from synapse.api.errors import SynapseError
-from synapse.metrics import LaterGauge
+from synapse.api.constants import PresenceState
 from synapse.storage.presence import UserPresenceState
-from synapse.types import UserID, get_domain_from_id
-from synapse.util.async import Linearizer
+
 from synapse.util.caches.descriptors import cachedInlineCallbacks
+from synapse.util.async import Linearizer
 from synapse.util.logcontext import run_in_background
 from synapse.util.logutils import log_function
 from synapse.util.metrics import Measure
 from synapse.util.wheel_timer import WheelTimer
+from synapse.types import UserID, get_domain_from_id
+import synapse.metrics
+
+import logging
 
 logger = logging.getLogger(__name__)
 
+metrics = synapse.metrics.get_metrics_for(__name__)
 
-notified_presence_counter = Counter("synapse_handler_presence_notified_presence", "")
-federation_presence_out_counter = Counter(
-    "synapse_handler_presence_federation_presence_out", "")
-presence_updates_counter = Counter("synapse_handler_presence_presence_updates", "")
-timers_fired_counter = Counter("synapse_handler_presence_timers_fired", "")
-federation_presence_counter = Counter("synapse_handler_presence_federation_presence", "")
-bump_active_time_counter = Counter("synapse_handler_presence_bump_active_time", "")
+notified_presence_counter = metrics.register_counter("notified_presence")
+federation_presence_out_counter = metrics.register_counter("federation_presence_out")
+presence_updates_counter = metrics.register_counter("presence_updates")
+timers_fired_counter = metrics.register_counter("timers_fired")
+federation_presence_counter = metrics.register_counter("federation_presence")
+bump_active_time_counter = metrics.register_counter("bump_active_time")
 
-get_updates_counter = Counter("synapse_handler_presence_get_updates", "", ["type"])
+get_updates_counter = metrics.register_counter("get_updates", labels=["type"])
 
-notify_reason_counter = Counter(
-    "synapse_handler_presence_notify_reason", "", ["reason"])
-state_transition_counter = Counter(
-    "synapse_handler_presence_state_transition", "", ["from", "to"]
+notify_reason_counter = metrics.register_counter("notify_reason", labels=["reason"])
+state_transition_counter = metrics.register_counter(
+    "state_transition", labels=["from", "to"]
 )
 
 
@@ -144,9 +142,8 @@ class PresenceHandler(object):
             for state in active_presence
         }
 
-        LaterGauge(
-            "synapse_handlers_presence_user_to_current_state_size", "", [],
-            lambda: len(self.user_to_current_state)
+        metrics.register_callback(
+            "user_to_current_state_size", lambda: len(self.user_to_current_state)
         )
 
         now = self.clock.time_msec()
@@ -178,7 +175,7 @@ class PresenceHandler(object):
         # have not yet been persisted
         self.unpersisted_users_changes = set()
 
-        hs.get_reactor().addSystemEventTrigger("before", "shutdown", self._on_shutdown)
+        reactor.addSystemEventTrigger("before", "shutdown", self._on_shutdown)
 
         self.serial_to_user = {}
         self._next_serial = 1
@@ -216,8 +213,7 @@ class PresenceHandler(object):
             60 * 1000,
         )
 
-        LaterGauge("synapse_handlers_presence_wheel_timer_size", "", [],
-                   lambda: len(self.wheel_timer))
+        metrics.register_callback("wheel_timer_size", lambda: len(self.wheel_timer))
 
     @defer.inlineCallbacks
     def _on_shutdown(self):
@@ -320,11 +316,11 @@ class PresenceHandler(object):
 
             # TODO: We should probably ensure there are no races hereafter
 
-            presence_updates_counter.inc(len(new_states))
+            presence_updates_counter.inc_by(len(new_states))
 
             if to_notify:
-                notified_presence_counter.inc(len(to_notify))
-                yield self._persist_and_notify(list(to_notify.values()))
+                notified_presence_counter.inc_by(len(to_notify))
+                yield self._persist_and_notify(to_notify.values())
 
             self.unpersisted_users_changes |= set(s.user_id for s in new_states)
             self.unpersisted_users_changes -= set(to_notify.keys())
@@ -334,7 +330,7 @@ class PresenceHandler(object):
                 if user_id not in to_notify
             }
             if to_federation_ping:
-                federation_presence_out_counter.inc(len(to_federation_ping))
+                federation_presence_out_counter.inc_by(len(to_federation_ping))
 
                 self._push_to_remotes(to_federation_ping.values())
 
@@ -372,7 +368,7 @@ class PresenceHandler(object):
                     for user_id in users_to_check
                 ]
 
-                timers_fired_counter.inc(len(states))
+                timers_fired_counter.inc_by(len(states))
 
                 changes = handle_timeouts(
                     states,
@@ -661,7 +657,7 @@ class PresenceHandler(object):
             updates.append(prev_state.copy_and_replace(**new_fields))
 
         if updates:
-            federation_presence_counter.inc(len(updates))
+            federation_presence_counter.inc_by(len(updates))
             yield self._update_states(updates)
 
     @defer.inlineCallbacks
@@ -686,7 +682,7 @@ class PresenceHandler(object):
         """
 
         updates = yield self.current_state_for_users(target_user_ids)
-        updates = list(updates.values())
+        updates = updates.values()
 
         for user_id in set(target_user_ids) - set(u.user_id for u in updates):
             updates.append(UserPresenceState.default(user_id))
@@ -752,11 +748,11 @@ class PresenceHandler(object):
             self._push_to_remotes([state])
         else:
             user_ids = yield self.store.get_users_in_room(room_id)
-            user_ids = list(filter(self.is_mine_id, user_ids))
+            user_ids = filter(self.is_mine_id, user_ids)
 
             states = yield self.current_state_for_users(user_ids)
 
-            self._push_to_remotes(list(states.values()))
+            self._push_to_remotes(states.values())
 
     @defer.inlineCallbacks
     def get_presence_list(self, observer_user, accepted=None):
@@ -936,28 +932,28 @@ def should_notify(old_state, new_state):
         return False
 
     if old_state.status_msg != new_state.status_msg:
-        notify_reason_counter.labels("status_msg_change").inc()
+        notify_reason_counter.inc("status_msg_change")
         return True
 
     if old_state.state != new_state.state:
-        notify_reason_counter.labels("state_change").inc()
-        state_transition_counter.labels(old_state.state, new_state.state).inc()
+        notify_reason_counter.inc("state_change")
+        state_transition_counter.inc(old_state.state, new_state.state)
         return True
 
     if old_state.state == PresenceState.ONLINE:
         if new_state.currently_active != old_state.currently_active:
-            notify_reason_counter.labels("current_active_change").inc()
+            notify_reason_counter.inc("current_active_change")
             return True
 
         if new_state.last_active_ts - old_state.last_active_ts > LAST_ACTIVE_GRANULARITY:
             # Only notify about last active bumps if we're not currently acive
             if not new_state.currently_active:
-                notify_reason_counter.labels("last_active_change_online").inc()
+                notify_reason_counter.inc("last_active_change_online")
                 return True
 
     elif new_state.last_active_ts - old_state.last_active_ts > LAST_ACTIVE_GRANULARITY:
         # Always notify for a transition where last active gets bumped.
-        notify_reason_counter.labels("last_active_change_not_online").inc()
+        notify_reason_counter.inc("last_active_change_not_online")
         return True
 
     return False
@@ -1031,14 +1027,14 @@ class PresenceEventSource(object):
             if changed is not None and len(changed) < 500:
                 # For small deltas, its quicker to get all changes and then
                 # work out if we share a room or they're in our presence list
-                get_updates_counter.labels("stream").inc()
+                get_updates_counter.inc("stream")
                 for other_user_id in changed:
                     if other_user_id in users_interested_in:
                         user_ids_changed.add(other_user_id)
             else:
                 # Too many possible updates. Find all users we can see and check
                 # if any of them have changed.
-                get_updates_counter.labels("full").inc()
+                get_updates_counter.inc("full")
 
                 if from_key:
                     user_ids_changed = stream_change_cache.get_entities_changed(
@@ -1050,7 +1046,7 @@ class PresenceEventSource(object):
             updates = yield presence.current_state_for_users(user_ids_changed)
 
         if include_offline:
-            defer.returnValue((list(updates.values()), max_token))
+            defer.returnValue((updates.values(), max_token))
         else:
             defer.returnValue(([
                 s for s in itervalues(updates)
@@ -1111,7 +1107,7 @@ def handle_timeouts(user_states, is_mine_fn, syncing_user_ids, now):
         if new_state:
             changes[state.user_id] = new_state
 
-    return list(changes.values())
+    return changes.values()
 
 
 def handle_timeout(state, is_mine, syncing_user_ids, now):

synapse/handlers/profile.py

@@ -17,9 +17,8 @@ import logging
 
 from twisted.internet import defer
 
-from synapse.api.errors import AuthError, CodeMessageException, SynapseError
+from synapse.api.errors import SynapseError, AuthError, CodeMessageException
 from synapse.types import UserID, get_domain_from_id
-
 from ._base import BaseHandler
 
 logger = logging.getLogger(__name__)

synapse/handlers/read_marker.py

@@ -13,14 +13,13 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import logging
+from ._base import BaseHandler
 
 from twisted.internet import defer
 
 from synapse.util.async import Linearizer
 
-from ._base import BaseHandler
-
+import logging
 logger = logging.getLogger(__name__)
 
 

synapse/handlers/receipts.py

@@ -12,15 +12,17 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
+from synapse.util import logcontext
+
+from ._base import BaseHandler
 
 from twisted.internet import defer
 
-from synapse.types import get_domain_from_id
-from synapse.util import logcontext
 from synapse.util.logcontext import PreserveLoggingContext
+from synapse.types import get_domain_from_id
+
+import logging
 
-from ._base import BaseHandler
 
 logger = logging.getLogger(__name__)
 

synapse/handlers/register.py

@@ -18,19 +18,14 @@ import logging
 
 from twisted.internet import defer
 
-from synapse import types
 from synapse.api.errors import (
-    AuthError,
-    Codes,
-    InvalidCaptchaError,
-    RegistrationError,
-    SynapseError,
+    AuthError, Codes, SynapseError, RegistrationError, InvalidCaptchaError
 )
 from synapse.http.client import CaptchaServerHttpClient
-from synapse.types import RoomAlias, RoomID, UserID, create_requester
-from synapse.util.async import Linearizer
+from synapse import types
+from synapse.types import UserID, create_requester, RoomID, RoomAlias
+from synapse.util.async import run_on_reactor, Linearizer
 from synapse.util.threepids import check_3pid_allowed
-
 from ._base import BaseHandler
 
 logger = logging.getLogger(__name__)
@@ -144,6 +139,7 @@ class RegistrationHandler(BaseHandler):
         Raises:
             RegistrationError if there was a problem registering.
         """
+        yield run_on_reactor()
         password_hash = None
         if password:
             password_hash = yield self.auth_handler().hash(password)
@@ -435,6 +431,8 @@ class RegistrationHandler(BaseHandler):
         Raises:
             RegistrationError if there was a problem registering.
         """
+        yield run_on_reactor()
+
         if localpart is None:
             raise SynapseError(400, "Request must include user id")
 

synapse/handlers/room.py

@@ -15,20 +15,23 @@
 # limitations under the License.
 
 """Contains functions for performing events on rooms."""
-import logging
-import math
-import string
-from collections import OrderedDict
-
 from twisted.internet import defer
 
-from synapse.api.constants import EventTypes, JoinRules, RoomCreationPreset
-from synapse.api.errors import AuthError, Codes, StoreError, SynapseError
-from synapse.types import RoomAlias, RoomID, RoomStreamToken, UserID
+from ._base import BaseHandler
+
+from synapse.types import UserID, RoomAlias, RoomID, RoomStreamToken
+from synapse.api.constants import (
+    EventTypes, JoinRules, RoomCreationPreset
+)
+from synapse.api.errors import AuthError, StoreError, SynapseError
 from synapse.util import stringutils
 from synapse.visibility import filter_events_for_client
 
-from ._base import BaseHandler
+from collections import OrderedDict
+
+import logging
+import math
+import string
 
 logger = logging.getLogger(__name__)
 
@@ -112,11 +115,7 @@ class RoomCreationHandler(BaseHandler):
             )
 
             if mapping:
-                raise SynapseError(
-                    400,
-                    "Room alias already taken",
-                    Codes.ROOM_IN_USE
-                )
+                raise SynapseError(400, "Room alias already taken")
         else:
             room_alias = None
 
@@ -456,7 +455,7 @@ class RoomContextHandler(BaseHandler):
         state = yield self.store.get_state_for_events(
             [last_event_id], None
         )
-        results["state"] = list(state[last_event_id].values())
+        results["state"] = state[last_event_id].values()
 
         results["start"] = now_token.copy_and_replace(
             "room_key", results["start"]
@@ -515,7 +514,8 @@ class RoomEventSource(object):
             events = list(room_events)
             events.extend(e for evs, _ in room_to_events.values() for e in evs)
 
-            events.sort(key=lambda e: e.internal_metadata.order)
+            # Order by the stream ordering of the events.
+            events.sort(key=lambda e: e.internal_metadata.stream_ordering)
 
             if limit:
                 events[:] = events[:limit]
@@ -535,7 +535,7 @@ class RoomEventSource(object):
 
     @defer.inlineCallbacks
     def get_pagination_rows(self, user, config, key):
-        events, next_key = yield self.store.paginate_room_events(
+        events, next_key, _ = yield self.store.paginate_room_events(
             room_id=key,
             from_key=config.from_key,
             to_key=config.to_key,

synapse/handlers/room_list.py

@@ -13,24 +13,25 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import logging
-from collections import namedtuple
-
-from six import iteritems
-from six.moves import range
-
-import msgpack
-from unpaddedbase64 import decode_base64, encode_base64
-
 from twisted.internet import defer
 
-from synapse.api.constants import EventTypes, JoinRules
-from synapse.types import ThirdPartyInstanceID
+from six.moves import range
+
+from ._base import BaseHandler
+
+from synapse.api.constants import (
+    EventTypes, JoinRules,
+)
 from synapse.util.async import concurrently_execute
 from synapse.util.caches.descriptors import cachedInlineCallbacks
 from synapse.util.caches.response_cache import ResponseCache
+from synapse.types import ThirdPartyInstanceID
 
-from ._base import BaseHandler
+from collections import namedtuple
+from unpaddedbase64 import encode_base64, decode_base64
+
+import logging
+import msgpack
 
 logger = logging.getLogger(__name__)
 
@@ -38,7 +39,7 @@ REMOTE_ROOM_LIST_POLL_INTERVAL = 60 * 1000
 
 
 # This is used to indicate we should only return rooms published to the main list.
-EMPTY_THIRD_PARTY_ID = ThirdPartyInstanceID(None, None)
+EMTPY_THIRD_PARTY_ID = ThirdPartyInstanceID(None, None)
 
 
 class RoomListHandler(BaseHandler):
@@ -50,7 +51,7 @@ class RoomListHandler(BaseHandler):
 
     def get_local_public_room_list(self, limit=None, since_token=None,
                                    search_filter=None,
-                                   network_tuple=EMPTY_THIRD_PARTY_ID,):
+                                   network_tuple=EMTPY_THIRD_PARTY_ID,):
         """Generate a local public room list.
 
         There are multiple different lists: the main one plus one per third
@@ -87,7 +88,7 @@ class RoomListHandler(BaseHandler):
     @defer.inlineCallbacks
     def _get_public_room_list(self, limit=None, since_token=None,
                               search_filter=None,
-                              network_tuple=EMPTY_THIRD_PARTY_ID,):
+                              network_tuple=EMTPY_THIRD_PARTY_ID,):
         if since_token and since_token != "END":
             since_token = RoomListNextBatch.from_token(since_token)
         else:
@@ -306,7 +307,7 @@ class RoomListHandler(BaseHandler):
         )
 
         event_map = yield self.store.get_events([
-            event_id for key, event_id in iteritems(current_state_ids)
+            event_id for key, event_id in current_state_ids.iteritems()
             if key[0] in (
                 EventTypes.JoinRules,
                 EventTypes.Name,

synapse/handlers/room_member.py

@@ -21,17 +21,19 @@ from six.moves import http_client
 
 from signedjson.key import decode_verify_key_bytes
 from signedjson.sign import verify_signed_json
-from unpaddedbase64 import decode_base64
-
 from twisted.internet import defer
+from unpaddedbase64 import decode_base64
 
 import synapse.server
 import synapse.types
-from synapse.api.constants import EventTypes, Membership
-from synapse.api.errors import AuthError, Codes, SynapseError
-from synapse.types import RoomID, UserID
+from synapse.api.constants import (
+    EventTypes, Membership,
+)
+from synapse.api.errors import AuthError, SynapseError, Codes
+from synapse.types import UserID, RoomID
 from synapse.util.async import Linearizer
-from synapse.util.distributor import user_joined_room, user_left_room
+from synapse.util.distributor import user_left_room, user_joined_room
+
 
 logger = logging.getLogger(__name__)
 
@@ -296,6 +298,16 @@ class RoomMemberHandler(object):
             is_blocked = yield self.store.is_room_blocked(room_id)
             if is_blocked:
                 raise SynapseError(403, "This room has been blocked on this server")
+        else:
+            # we don't allow people to reject invites to, or leave, the
+            # server notice room.
+            is_blocked = yield self._is_server_notice_room(room_id)
+            if is_blocked:
+                raise SynapseError(
+                    http_client.FORBIDDEN,
+                    "You cannot leave this room",
+                    errcode=Codes.CANNOT_LEAVE_SERVER_NOTICE_ROOM,
+                )
 
         if effective_membership_state == Membership.INVITE:
             # block any attempts to invite the server notices mxid
@@ -371,20 +383,6 @@ class RoomMemberHandler(object):
                 if same_sender and same_membership and same_content:
                     defer.returnValue(old_state)
 
-            # we don't allow people to reject invites to the server notice
-            # room, but they can leave it once they are joined.
-            if (
-                old_membership == Membership.INVITE and
-                effective_membership_state == Membership.LEAVE
-            ):
-                is_blocked = yield self._is_server_notice_room(room_id)
-                if is_blocked:
-                    raise SynapseError(
-                        http_client.FORBIDDEN,
-                        "You cannot reject this invite",
-                        errcode=Codes.CANNOT_LEAVE_SERVER_NOTICE_ROOM,
-                    )
-
         is_host_in_room = yield self._is_host_in_room(current_state_ids)
 
         if effective_membership_state == Membership.JOIN:

synapse/handlers/room_member_worker.py

@@ -20,12 +20,11 @@ from twisted.internet import defer
 from synapse.api.errors import SynapseError
 from synapse.handlers.room_member import RoomMemberHandler
 from synapse.replication.http.membership import (
-    get_or_register_3pid_guest,
+    remote_join, remote_reject_invite, get_or_register_3pid_guest,
     notify_user_membership_change,
-    remote_join,
-    remote_reject_invite,
 )
 
+
 logger = logging.getLogger(__name__)
 
 

synapse/handlers/search.py

@@ -13,20 +13,21 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import itertools
-import logging
-
-from unpaddedbase64 import decode_base64, encode_base64
-
 from twisted.internet import defer
 
-from synapse.api.constants import EventTypes, Membership
-from synapse.api.errors import SynapseError
+from ._base import BaseHandler
+
+from synapse.api.constants import Membership, EventTypes
 from synapse.api.filtering import Filter
+from synapse.api.errors import SynapseError
 from synapse.events.utils import serialize_event
 from synapse.visibility import filter_events_for_client
 
-from ._base import BaseHandler
+from unpaddedbase64 import decode_base64, encode_base64
+
+import itertools
+import logging
+
 
 logger = logging.getLogger(__name__)
 
@@ -63,13 +64,6 @@ class SearchHandler(BaseHandler):
             except Exception:
                 raise SynapseError(400, "Invalid batch")
 
-        logger.info(
-            "Search batch properties: %r, %r, %r",
-            batch_group, batch_group_key, batch_token,
-        )
-
-        logger.info("Search content: %s", content)
-
         try:
             room_cat = content["search_categories"]["room_events"]
 
@@ -277,8 +271,6 @@ class SearchHandler(BaseHandler):
             # We should never get here due to the guard earlier.
             raise NotImplementedError()
 
-        logger.info("Found %d events to return", len(allowed_events))
-
         # If client has asked for "context" for each event (i.e. some surrounding
         # events and state), fetch that
         if event_context is not None:
@@ -290,11 +282,6 @@ class SearchHandler(BaseHandler):
                     event.room_id, event.event_id, before_limit, after_limit
                 )
 
-                logger.info(
-                    "Context for search returned %d and %d events",
-                    len(res["events_before"]), len(res["events_after"]),
-                )
-
                 res["events_before"] = yield filter_events_for_client(
                     self.store, user.to_string(), res["events_before"]
                 )
@@ -361,7 +348,7 @@ class SearchHandler(BaseHandler):
             rooms = set(e.room_id for e in allowed_events)
             for room_id in rooms:
                 state = yield self.state_handler.get_current_state(room_id)
-                state_results[room_id] = list(state.values())
+                state_results[room_id] = state.values()
 
             state_results.values()
 

synapse/handlers/set_password.py

@@ -17,7 +17,6 @@ import logging
 from twisted.internet import defer
 
 from synapse.api.errors import Codes, StoreError, SynapseError
-
 from ._base import BaseHandler
 
 logger = logging.getLogger(__name__)

synapse/handlers/sync.py

@@ -13,22 +13,22 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import collections
-import itertools
-import logging
-
-from six import iteritems, itervalues
+from synapse.api.constants import Membership, EventTypes
+from synapse.util.async import concurrently_execute
+from synapse.util.logcontext import LoggingContext
+from synapse.util.metrics import Measure, measure_func
+from synapse.util.caches.response_cache import ResponseCache
+from synapse.push.clientformat import format_push_rules_for_user
+from synapse.visibility import filter_events_for_client
+from synapse.types import RoomStreamToken
 
 from twisted.internet import defer
 
-from synapse.api.constants import EventTypes, Membership
-from synapse.push.clientformat import format_push_rules_for_user
-from synapse.types import RoomStreamToken
-from synapse.util.async import concurrently_execute
-from synapse.util.caches.response_cache import ResponseCache
-from synapse.util.logcontext import LoggingContext
-from synapse.util.metrics import Measure, measure_func
-from synapse.visibility import filter_events_for_client
+import collections
+import logging
+import itertools
+
+from six import itervalues, iteritems
 
 logger = logging.getLogger(__name__)
 
@@ -145,7 +145,7 @@ class SyncResult(collections.namedtuple("SyncResult", [
     "invited",  # InvitedSyncResult for each invited room.
     "archived",  # ArchivedSyncResult for each archived room.
     "to_device",  # List of direct messages for the device.
-    "device_lists",  # List of user_ids whose devices have changed
+    "device_lists",  # List of user_ids whose devices have chanegd
     "device_one_time_keys_count",  # Dict of algorithm to count for one time keys
                                    # for this device
     "groups",
@@ -443,10 +443,6 @@ class SyncHandler(object):
         Returns:
             A Deferred map from ((type, state_key)->Event)
         """
-        # FIXME this claims to get the state at a stream position, but
-        # get_recent_events_for_room operates by topo ordering. This therefore
-        # does not reliably give you the state at the given stream position.
-        # (https://github.com/matrix-org/synapse/issues/3305)
         last_events, _ = yield self.store.get_recent_events_for_room(
             room_id, end_token=stream_position.room_key, limit=1,
         )
@@ -541,11 +537,11 @@ class SyncHandler(object):
 
         state = {}
         if state_ids:
-            state = yield self.store.get_events(list(state_ids.values()))
+            state = yield self.store.get_events(state_ids.values())
 
         defer.returnValue({
             (e.type, e.state_key): e
-            for e in sync_config.filter_collection.filter_room_state(list(state.values()))
+            for e in sync_config.filter_collection.filter_room_state(state.values())
         })
 
     @defer.inlineCallbacks
@@ -894,7 +890,7 @@ class SyncHandler(object):
             presence.extend(states)
 
             # Deduplicate the presence entries so that there's at most one per user
-            presence = list({p.user_id: p for p in presence}.values())
+            presence = {p.user_id: p for p in presence}.values()
 
         presence = sync_config.filter_collection.filter_presence(
             presence
@@ -1046,13 +1042,7 @@ class SyncHandler(object):
 
         Returns:
             Deferred(tuple): Returns a tuple of the form:
-            `(room_entries, invited_rooms, newly_joined_rooms, newly_left_rooms)`
-
-            where:
-                room_entries is a list [RoomSyncResultBuilder]
-                invited_rooms is a list [InvitedSyncResult]
-                newly_joined rooms is a list[str] of room ids
-                newly_left_rooms is a list[str] of room ids
+            `([RoomSyncResultBuilder], [InvitedSyncResult], newly_joined_rooms)`
         """
         user_id = sync_result_builder.sync_config.user.to_string()
         since_token = sync_result_builder.since_token