WIP of --config-check commandline flag

.travis.yml

@@ -4,12 +4,7 @@ language: python
 # tell travis to cache ~/.cache/pip
 cache: pip
 
-before_script:
-  - git remote set-branches --add origin develop
-  - git fetch origin develop
-
 matrix:
-  fast_finish: true
   include:
   - python: 2.7
     env: TOX_ENV=packaging
@@ -19,13 +14,10 @@ matrix:
 
   - python: 2.7
     env: TOX_ENV=py27
-
+    
   - python: 3.6
     env: TOX_ENV=py36
 
-  - python: 3.6
-    env: TOX_ENV=check-newsfragment
-
 install:
   - pip install tox
 

CHANGES.rst

@@ -1,72 +1,3 @@
-Synapse 0.32.2 (2018-07-07)
-===========================
-
-Bugfixes
---------
-
-- Amend the Python dependencies to depend on attrs from PyPI, not attr (`#3492 <https://github.com/matrix-org/synapse/issues/3492>`_)
-
-
-Synapse 0.32.1 (2018-07-06)
-===========================
-
-Bugfixes
---------
-
-- Add explicit dependency on netaddr (`#3488 <https://github.com/matrix-org/synapse/issues/3488>`_)
-
-
-Changes in synapse v0.32.0 (2018-07-06)
-===========================================
-No changes since 0.32.0rc1
-
-Synapse 0.32.0rc1 (2018-07-05)
-==============================
-
-Features
---------
-
-- Add blacklist & whitelist of servers allowed to send events to a room via ``m.room.server_acl`` event.
-- Cache factor override system for specific caches (`#3334 <https://github.com/matrix-org/synapse/issues/3334>`_)
-- Add metrics to track appservice transactions (`#3344 <https://github.com/matrix-org/synapse/issues/3344>`_)
-- Try to log more helpful info when a sig verification fails (`#3372 <https://github.com/matrix-org/synapse/issues/3372>`_)
-- Synapse now uses the best performing JSON encoder/decoder according to your runtime (simplejson on CPython, stdlib json on PyPy). (`#3462 <https://github.com/matrix-org/synapse/issues/3462>`_)
-- Add optional ip_range_whitelist param to AS registration files to lock AS IP access (`#3465 <https://github.com/matrix-org/synapse/issues/3465>`_)
-- Reject invalid server names in federation requests (`#3480 <https://github.com/matrix-org/synapse/issues/3480>`_)
-- Reject invalid server names in homeserver.yaml (`#3483 <https://github.com/matrix-org/synapse/issues/3483>`_)
-
-
-Bugfixes
---------
-
-- Strip access_token from outgoing requests (`#3327 <https://github.com/matrix-org/synapse/issues/3327>`_)
-- Redact AS tokens in logs (`#3349 <https://github.com/matrix-org/synapse/issues/3349>`_)
-- Fix federation backfill from SQLite servers (`#3355 <https://github.com/matrix-org/synapse/issues/3355>`_)
-- Fix event-purge-by-ts admin API (`#3363 <https://github.com/matrix-org/synapse/issues/3363>`_)
-- Fix event filtering in get_missing_events handler (`#3371 <https://github.com/matrix-org/synapse/issues/3371>`_)
-- Synapse is now stricter regarding accepting events which it cannot retrieve the prev_events for. (`#3456 <https://github.com/matrix-org/synapse/issues/3456>`_)
-- Fix bug where synapse would explode when receiving unicode in HTTP User-Agent header (`#3470 <https://github.com/matrix-org/synapse/issues/3470>`_)
-- Invalidate cache on correct thread to avoid race (`#3473 <https://github.com/matrix-org/synapse/issues/3473>`_)
-
-
-Improved Documentation
-----------------------
-
-- ``doc/postgres.rst``: fix display of the last command block. Thanks to @ArchangeGabriel! (`#3340 <https://github.com/matrix-org/synapse/issues/3340>`_)
-
-
-Deprecations and Removals
--------------------------
-
-- Remove was_forgotten_at (`#3324 <https://github.com/matrix-org/synapse/issues/3324>`_)
-
-
-Misc
-----
-
-- `#3332 <https://github.com/matrix-org/synapse/issues/3332>`_, `#3341 <https://github.com/matrix-org/synapse/issues/3341>`_, `#3347 <https://github.com/matrix-org/synapse/issues/3347>`_, `#3348 <https://github.com/matrix-org/synapse/issues/3348>`_, `#3356 <https://github.com/matrix-org/synapse/issues/3356>`_, `#3385 <https://github.com/matrix-org/synapse/issues/3385>`_, `#3446 <https://github.com/matrix-org/synapse/issues/3446>`_, `#3447 <https://github.com/matrix-org/synapse/issues/3447>`_, `#3467 <https://github.com/matrix-org/synapse/issues/3467>`_, `#3474 <https://github.com/matrix-org/synapse/issues/3474>`_
-
-
 Changes in synapse v0.31.2 (2018-06-14)
 =======================================
 

CONTRIBUTING.rst

@@ -48,26 +48,6 @@ Please ensure your changes match the cosmetic style of the existing project,
 and **never** mix cosmetic and functional changes in the same commit, as it
 makes it horribly hard to review otherwise.
 
-Changelog
-~~~~~~~~~
-
-All changes, even minor ones, need a corresponding changelog
-entry. These are managed by Towncrier
-(https://github.com/hawkowl/towncrier).
-
-To create a changelog entry, make a new file in the ``changelog.d``
-file named in the format of ``issuenumberOrPR.type``. The type can be
-one of ``feature``, ``bugfix``, ``removal`` (also used for
-deprecations), or ``misc`` (for internal-only changes). The content of
-the file is your changelog entry, which can contain RestructuredText
-formatting. A note of contributors is welcomed in changelogs for
-non-misc changes (the content of misc changes is not displayed).
-
-For example, a fix for a bug reported in #1234 would have its
-changelog entry in ``changelog.d/1234.bugfix``, and contain content
-like "The security levels of Florbs are now validated when
-recieved over federation. Contributed by Jane Matrix".
-
 Attribution
 ~~~~~~~~~~~
 
@@ -130,15 +110,11 @@ If you agree to this for your contribution, then all that's needed is to
 include the line in your commit or pull request comment::
 
     Signed-off-by: Your Name <your@email.example.org>
-
-We accept contributions under a legally identifiable name, such as
-your name on government documentation or common-law names (names
-claimed by legitimate usage or repute). Unfortunately, we cannot
-accept anonymous contributions at this time.
-
-Git allows you to add this signoff automatically when using the ``-s``
-flag to ``git commit``, which uses the name and email set in your
-``user.name`` and ``user.email`` git configs.
+    
+...using your real name; unfortunately pseudonyms and anonymous contributions
+can't be accepted. Git makes this trivial - just use the -s flag when you do
+``git commit``, having first set ``user.name`` and ``user.email`` git configs
+(which you should have done anyway :)
 
 Conclusion
 ~~~~~~~~~~

MANIFEST.in

@@ -29,8 +29,5 @@ exclude Dockerfile
 exclude .dockerignore
 recursive-exclude jenkins *.sh
 
-include pyproject.toml
-recursive-include changelog.d *
-
 prune .github
 prune demo/etc

changelog.d/.gitignore

@@ -1 +0,0 @@
-!.gitignore

changelog.d/3316.feature

@@ -1 +0,0 @@
-Enforce the specified API for report_event

changelog.d/3496.feature

@@ -1 +0,0 @@
-Include CPU time from database threads in request/block metrics.

changelog.d/3497.feature

@@ -1 +0,0 @@
-Add CPU metrics for _fetch_event_list

changelog.d/3505.feature

@@ -1 +0,0 @@
-Reduce database consumption when processing large numbers of receipts

changelog.d/3521.feature

@@ -1 +0,0 @@
-Cache optimisation for /sync requests

changelog.d/3533.bugfix

@@ -1 +0,0 @@
-Fix queued federation requests being processed in the wrong order

changelog.d/3534.misc

@@ -1 +0,0 @@
-refactor: use parse_{string,integer} and assert's from http.servlet for deduplication

docs/admin_api/user_admin_api.rst

@@ -44,26 +44,13 @@ Deactivate Account
 
 This API deactivates an account. It removes active access tokens, resets the
 password, and deletes third-party IDs (to prevent the user requesting a
-password reset). It can also mark the user as GDPR-erased (stopping their data
-from distributed further, and deleting it entirely if there are no other
-references to it).
+password reset).
 
 The api is::
 
     POST /_matrix/client/r0/admin/deactivate/<user_id>
 
-with a body of:
-
-.. code:: json
-
-    {
-        "erase": true
-    }
-
-including an ``access_token`` of a server admin.
-
-The erase parameter is optional and defaults to 'false'.
-An empty body may be passed for backwards compatibility.
+including an ``access_token`` of a server admin, and an empty request body.
 
 
 Reset password

pyproject.toml

@@ -1,5 +0,0 @@
-[tool.towncrier]
-    package = "synapse"
-    filename = "CHANGES.rst"
-    directory = "changelog.d"
-    issue_format = "`#{issue} <https://github.com/matrix-org/synapse/issues/{issue}>`_"

setup.cfg

@@ -19,15 +19,3 @@ max-line-length = 90
 #  W503 requires that binary operators be at the end, not start, of lines. Erik doesn't like it.
 #  E203 is contrary to PEP8.
 ignore = W503,E203
-
-[isort]
-line_length = 89
-not_skip = __init__.py
-sections=FUTURE,STDLIB,COMPAT,THIRDPARTY,TWISTED,FIRSTPARTY,TESTS,LOCALFOLDER
-default_section=THIRDPARTY
-known_first_party = synapse
-known_tests=tests
-known_compat = mock,six
-known_twisted=twisted,OpenSSL
-multi_line_output=3
-include_trailing_comma=true

synapse/__init__.py

@@ -17,4 +17,4 @@
 """ This is a reference implementation of a Matrix home server.
 """
 
-__version__ = "0.32.2"
+__version__ = "0.31.2"

synapse/api/auth.py

@@ -18,16 +18,14 @@ import logging
 from six import itervalues
 
 import pymacaroons
-from netaddr import IPAddress
-
 from twisted.internet import defer
 
 import synapse.types
 from synapse import event_auth
-from synapse.api.constants import EventTypes, JoinRules, Membership
+from synapse.api.constants import EventTypes, Membership, JoinRules
 from synapse.api.errors import AuthError, Codes
 from synapse.types import UserID
-from synapse.util.caches import CACHE_SIZE_FACTOR, register_cache
+from synapse.util.caches import register_cache, CACHE_SIZE_FACTOR
 from synapse.util.caches.lrucache import LruCache
 from synapse.util.metrics import Measure
 
@@ -193,7 +191,7 @@ class Auth(object):
                     synapse.types.create_requester(user_id, app_service=app_service)
                 )
 
-            access_token = self.get_access_token_from_request(
+            access_token = get_access_token_from_request(
                 request, self.TOKEN_NOT_FOUND_HTTP_STATUS
             )
 
@@ -239,18 +237,13 @@ class Auth(object):
     @defer.inlineCallbacks
     def _get_appservice_user_id(self, request):
         app_service = self.store.get_app_service_by_token(
-            self.get_access_token_from_request(
+            get_access_token_from_request(
                 request, self.TOKEN_NOT_FOUND_HTTP_STATUS
             )
         )
         if app_service is None:
             defer.returnValue((None, None))
 
-        if app_service.ip_range_whitelist:
-            ip_address = IPAddress(self.hs.get_ip_from_request(request))
-            if ip_address not in app_service.ip_range_whitelist:
-                defer.returnValue((None, None))
-
         if "user_id" not in request.args:
             defer.returnValue((app_service.sender, app_service))
 
@@ -495,7 +488,7 @@ class Auth(object):
     def _look_up_user_by_access_token(self, token):
         ret = yield self.store.get_user_by_access_token(token)
         if not ret:
-            logger.warn("Unrecognised access token - not in store.")
+            logger.warn("Unrecognised access token - not in store: %s" % (token,))
             raise AuthError(
                 self.TOKEN_NOT_FOUND_HTTP_STATUS, "Unrecognised access token.",
                 errcode=Codes.UNKNOWN_TOKEN
@@ -513,12 +506,12 @@ class Auth(object):
 
     def get_appservice_by_req(self, request):
         try:
-            token = self.get_access_token_from_request(
+            token = get_access_token_from_request(
                 request, self.TOKEN_NOT_FOUND_HTTP_STATUS
             )
             service = self.store.get_app_service_by_token(token)
             if not service:
-                logger.warn("Unrecognised appservice access token.")
+                logger.warn("Unrecognised appservice access token: %s" % (token,))
                 raise AuthError(
                     self.TOKEN_NOT_FOUND_HTTP_STATUS,
                     "Unrecognised access token.",
@@ -673,67 +666,67 @@ class Auth(object):
                 " edit its room list entry"
             )
 
-    @staticmethod
-    def has_access_token(request):
-        """Checks if the request has an access_token.
 
-        Returns:
-            bool: False if no access_token was given, True otherwise.
-        """
-        query_params = request.args.get("access_token")
-        auth_headers = request.requestHeaders.getRawHeaders(b"Authorization")
-        return bool(query_params) or bool(auth_headers)
+def has_access_token(request):
+    """Checks if the request has an access_token.
 
-    @staticmethod
-    def get_access_token_from_request(request, token_not_found_http_status=401):
-        """Extracts the access_token from the request.
+    Returns:
+        bool: False if no access_token was given, True otherwise.
+    """
+    query_params = request.args.get("access_token")
+    auth_headers = request.requestHeaders.getRawHeaders(b"Authorization")
+    return bool(query_params) or bool(auth_headers)
 
-        Args:
-            request: The http request.
-            token_not_found_http_status(int): The HTTP status code to set in the
-                AuthError if the token isn't found. This is used in some of the
-                legacy APIs to change the status code to 403 from the default of
-                401 since some of the old clients depended on auth errors returning
-                403.
-        Returns:
-            str: The access_token
-        Raises:
-            AuthError: If there isn't an access_token in the request.
-        """
 
-        auth_headers = request.requestHeaders.getRawHeaders(b"Authorization")
-        query_params = request.args.get(b"access_token")
-        if auth_headers:
-            # Try the get the access_token from a "Authorization: Bearer"
-            # header
-            if query_params is not None:
-                raise AuthError(
-                    token_not_found_http_status,
-                    "Mixing Authorization headers and access_token query parameters.",
-                    errcode=Codes.MISSING_TOKEN,
-                )
-            if len(auth_headers) > 1:
-                raise AuthError(
-                    token_not_found_http_status,
-                    "Too many Authorization headers.",
-                    errcode=Codes.MISSING_TOKEN,
-                )
-            parts = auth_headers[0].split(" ")
-            if parts[0] == "Bearer" and len(parts) == 2:
-                return parts[1]
-            else:
-                raise AuthError(
-                    token_not_found_http_status,
-                    "Invalid Authorization header.",
-                    errcode=Codes.MISSING_TOKEN,
-                )
+def get_access_token_from_request(request, token_not_found_http_status=401):
+    """Extracts the access_token from the request.
+
+    Args:
+        request: The http request.
+        token_not_found_http_status(int): The HTTP status code to set in the
+            AuthError if the token isn't found. This is used in some of the
+            legacy APIs to change the status code to 403 from the default of
+            401 since some of the old clients depended on auth errors returning
+            403.
+    Returns:
+        str: The access_token
+    Raises:
+        AuthError: If there isn't an access_token in the request.
+    """
+
+    auth_headers = request.requestHeaders.getRawHeaders(b"Authorization")
+    query_params = request.args.get(b"access_token")
+    if auth_headers:
+        # Try the get the access_token from a "Authorization: Bearer"
+        # header
+        if query_params is not None:
+            raise AuthError(
+                token_not_found_http_status,
+                "Mixing Authorization headers and access_token query parameters.",
+                errcode=Codes.MISSING_TOKEN,
+            )
+        if len(auth_headers) > 1:
+            raise AuthError(
+                token_not_found_http_status,
+                "Too many Authorization headers.",
+                errcode=Codes.MISSING_TOKEN,
+            )
+        parts = auth_headers[0].split(" ")
+        if parts[0] == "Bearer" and len(parts) == 2:
+            return parts[1]
         else:
-            # Try to get the access_token from the query params.
-            if not query_params:
-                raise AuthError(
-                    token_not_found_http_status,
-                    "Missing access token.",
-                    errcode=Codes.MISSING_TOKEN
-                )
+            raise AuthError(
+                token_not_found_http_status,
+                "Invalid Authorization header.",
+                errcode=Codes.MISSING_TOKEN,
+            )
+    else:
+        # Try to get the access_token from the query params.
+        if not query_params:
+            raise AuthError(
+                token_not_found_http_status,
+                "Missing access token.",
+                errcode=Codes.MISSING_TOKEN
+            )
 
-            return query_params[0]
+        return query_params[0]

synapse/api/constants.py

@@ -68,7 +68,6 @@ class EventTypes(object):
 
     RoomHistoryVisibility = "m.room.history_visibility"
     CanonicalAlias = "m.room.canonical_alias"
-    Encryption = "m.room.encryption"
     RoomAvatar = "m.room.avatar"
     GuestAccess = "m.room.guest_access"
 
@@ -77,8 +76,6 @@ class EventTypes(object):
     Topic = "m.room.topic"
     Name = "m.room.name"
 
-    ServerACL = "m.room.server_acl"
-
 
 class RejectedReason(object):
     AUTH_ERROR = "auth_error"

synapse/api/errors.py

@@ -17,11 +17,10 @@
 
 import logging
 
+import simplejson as json
 from six import iteritems
 from six.moves import http_client
 
-from canonicaljson import json
-
 logger = logging.getLogger(__name__)
 
 

synapse/api/filtering.py

@@ -12,15 +12,14 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import jsonschema
-from canonicaljson import json
-from jsonschema import FormatChecker
-
-from twisted.internet import defer
-
 from synapse.api.errors import SynapseError
 from synapse.storage.presence import UserPresenceState
-from synapse.types import RoomID, UserID
+from synapse.types import UserID, RoomID
+from twisted.internet import defer
+
+import simplejson as json
+import jsonschema
+from jsonschema import FormatChecker
 
 FILTER_SCHEMA = {
     "additionalProperties": False,

synapse/api/urls.py

@@ -15,8 +15,8 @@
 # limitations under the License.
 
 """Contains the URL paths to prefix various aspects of the server with. """
-import hmac
 from hashlib import sha256
+import hmac
 
 from six.moves.urllib.parse import urlencode
 

synapse/app/__init__.py

@@ -14,11 +14,9 @@
 # limitations under the License.
 
 import sys
-
-from synapse import python_dependencies  # noqa: E402
-
 sys.dont_write_bytecode = True
 
+from synapse import python_dependencies   # noqa: E402
 
 try:
     python_dependencies.check_requirements()

synapse/app/_base.py

@@ -17,18 +17,15 @@ import gc
 import logging
 import sys
 
-from daemonize import Daemonize
-
-from twisted.internet import error, reactor
-
-from synapse.util import PreserveLoggingContext
-from synapse.util.rlimit import change_resource_limit
-
 try:
     import affinity
 except Exception:
     affinity = None
 
+from daemonize import Daemonize
+from synapse.util import PreserveLoggingContext
+from synapse.util.rlimit import change_resource_limit
+from twisted.internet import error, reactor
 
 logger = logging.getLogger(__name__)
 

synapse/app/appservice.py

@@ -16,9 +16,6 @@
 import logging
 import sys
 
-from twisted.internet import defer, reactor
-from twisted.web.resource import NoResource
-
 import synapse
 from synapse import events
 from synapse.app import _base
@@ -39,6 +36,8 @@ from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext, run_in_background
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
+from twisted.internet import reactor, defer
+from twisted.web.resource import NoResource
 
 logger = logging.getLogger("synapse.app.appservice")
 

synapse/app/client_reader.py

@@ -16,9 +16,6 @@
 import logging
 import sys
 
-from twisted.internet import reactor
-from twisted.web.resource import NoResource
-
 import synapse
 from synapse import events
 from synapse.app import _base
@@ -47,6 +44,8 @@ from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
+from twisted.internet import reactor
+from twisted.web.resource import NoResource
 
 logger = logging.getLogger("synapse.app.client_reader")
 

synapse/app/event_creator.py

@@ -16,9 +16,6 @@
 import logging
 import sys
 
-from twisted.internet import reactor
-from twisted.web.resource import NoResource
-
 import synapse
 from synapse import events
 from synapse.app import _base
@@ -46,10 +43,8 @@ from synapse.replication.slave.storage.room import RoomStore
 from synapse.replication.slave.storage.transactions import TransactionStore
 from synapse.replication.tcp.client import ReplicationClientHandler
 from synapse.rest.client.v1.room import (
+    RoomSendEventRestServlet, RoomMembershipRestServlet, RoomStateEventRestServlet,
     JoinRoomAliasServlet,
-    RoomMembershipRestServlet,
-    RoomSendEventRestServlet,
-    RoomStateEventRestServlet,
 )
 from synapse.server import HomeServer
 from synapse.storage.engines import create_engine
@@ -57,6 +52,8 @@ from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
+from twisted.internet import reactor
+from twisted.web.resource import NoResource
 
 logger = logging.getLogger("synapse.app.event_creator")
 

synapse/app/federation_reader.py

@@ -16,9 +16,6 @@
 import logging
 import sys
 
-from twisted.internet import reactor
-from twisted.web.resource import NoResource
-
 import synapse
 from synapse import events
 from synapse.api.urls import FEDERATION_PREFIX
@@ -44,6 +41,8 @@ from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
+from twisted.internet import reactor
+from twisted.web.resource import NoResource
 
 logger = logging.getLogger("synapse.app.federation_reader")
 

synapse/app/federation_sender.py

@@ -16,9 +16,6 @@
 import logging
 import sys
 
-from twisted.internet import defer, reactor
-from twisted.web.resource import NoResource
-
 import synapse
 from synapse import events
 from synapse.app import _base
@@ -45,6 +42,8 @@ from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext, run_in_background
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
+from twisted.internet import defer, reactor
+from twisted.web.resource import NoResource
 
 logger = logging.getLogger("synapse.app.federation_sender")
 

synapse/app/frontend_proxy.py

@@ -16,9 +16,6 @@
 import logging
 import sys
 
-from twisted.internet import defer, reactor
-from twisted.web.resource import NoResource
-
 import synapse
 from synapse import events
 from synapse.api.errors import SynapseError
@@ -28,7 +25,9 @@ from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
 from synapse.crypto import context_factory
 from synapse.http.server import JsonResource
-from synapse.http.servlet import RestServlet, parse_json_object_from_request
+from synapse.http.servlet import (
+    RestServlet, parse_json_object_from_request,
+)
 from synapse.http.site import SynapseSite
 from synapse.metrics import RegistryProxy
 from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
@@ -45,6 +44,8 @@ from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
+from twisted.internet import defer, reactor
+from twisted.web.resource import NoResource
 
 logger = logging.getLogger("synapse.app.frontend_proxy")
 

synapse/app/homeserver.py

@@ -18,39 +18,27 @@ import logging
 import os
 import sys
 
-from twisted.application import service
-from twisted.internet import defer, reactor
-from twisted.web.resource import EncodingResourceWrapper, NoResource
-from twisted.web.server import GzipEncoderFactory
-from twisted.web.static import File
-
 import synapse
 import synapse.config.logger
 from synapse import events
-from synapse.api.urls import (
-    CONTENT_REPO_PREFIX,
-    FEDERATION_PREFIX,
-    LEGACY_MEDIA_PREFIX,
-    MEDIA_PREFIX,
-    SERVER_KEY_PREFIX,
-    SERVER_KEY_V2_PREFIX,
-    STATIC_PREFIX,
-    WEB_CLIENT_PREFIX,
-)
+from synapse.api.urls import CONTENT_REPO_PREFIX, FEDERATION_PREFIX, \
+    LEGACY_MEDIA_PREFIX, MEDIA_PREFIX, SERVER_KEY_PREFIX, SERVER_KEY_V2_PREFIX, \
+    STATIC_PREFIX, WEB_CLIENT_PREFIX
 from synapse.app import _base
-from synapse.app._base import listen_ssl, listen_tcp, quit_with_error
+from synapse.app._base import quit_with_error, listen_ssl, listen_tcp
 from synapse.config._base import ConfigError
 from synapse.config.homeserver import HomeServerConfig
 from synapse.crypto import context_factory
 from synapse.federation.transport.server import TransportLayerServer
+from synapse.module_api import ModuleApi
 from synapse.http.additional_resource import AdditionalResource
 from synapse.http.server import RootRedirect
 from synapse.http.site import SynapseSite
 from synapse.metrics import RegistryProxy
 from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
-from synapse.module_api import ModuleApi
-from synapse.python_dependencies import CONDITIONAL_REQUIREMENTS, check_requirements
-from synapse.replication.http import REPLICATION_PREFIX, ReplicationRestResource
+from synapse.python_dependencies import CONDITIONAL_REQUIREMENTS, \
+    check_requirements
+from synapse.replication.http import ReplicationRestResource, REPLICATION_PREFIX
 from synapse.replication.tcp.resource import ReplicationStreamProtocolFactory
 from synapse.rest import ClientRestResource
 from synapse.rest.key.v1.server_key_resource import LocalKey
@@ -67,6 +55,11 @@ from synapse.util.manhole import manhole
 from synapse.util.module_loader import load_module
 from synapse.util.rlimit import change_resource_limit
 from synapse.util.versionstring import get_version_string
+from twisted.application import service
+from twisted.internet import defer, reactor
+from twisted.web.resource import EncodingResourceWrapper, NoResource
+from twisted.web.server import GzipEncoderFactory
+from twisted.web.static import File
 
 logger = logging.getLogger("synapse.app.homeserver")
 
@@ -325,6 +318,11 @@ def setup(config_options):
     # check any extra requirements we have now we have a config
     check_requirements(config)
 
+    version_string = "Synapse/" + get_version_string(synapse)
+
+    logger.info("Server hostname: %s", config.server_name)
+    logger.info("Server version: %s", version_string)
+
     events.USE_FROZEN_DICTS = config.use_frozen_dicts
 
     tls_server_context_factory = context_factory.ServerContextFactory(config)
@@ -337,7 +335,7 @@ def setup(config_options):
         db_config=config.database_config,
         tls_server_context_factory=tls_server_context_factory,
         config=config,
-        version_string="Synapse/" + get_version_string(synapse),
+        version_string=version_string,
         database_engine=database_engine,
     )
 

synapse/app/media_repository.py

@@ -16,12 +16,11 @@
 import logging
 import sys
 
-from twisted.internet import reactor
-from twisted.web.resource import NoResource
-
 import synapse
 from synapse import events
-from synapse.api.urls import CONTENT_REPO_PREFIX, LEGACY_MEDIA_PREFIX, MEDIA_PREFIX
+from synapse.api.urls import (
+    CONTENT_REPO_PREFIX, LEGACY_MEDIA_PREFIX, MEDIA_PREFIX
+)
 from synapse.app import _base
 from synapse.config._base import ConfigError
 from synapse.config.homeserver import HomeServerConfig
@@ -44,6 +43,8 @@ from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
+from twisted.internet import reactor
+from twisted.web.resource import NoResource
 
 logger = logging.getLogger("synapse.app.media_repository")
 

synapse/app/pusher.py

@@ -16,9 +16,6 @@
 import logging
 import sys
 
-from twisted.internet import defer, reactor
-from twisted.web.resource import NoResource
-
 import synapse
 from synapse import events
 from synapse.app import _base
@@ -40,6 +37,8 @@ from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext, run_in_background
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
+from twisted.internet import defer, reactor
+from twisted.web.resource import NoResource
 
 logger = logging.getLogger("synapse.app.pusher")
 

synapse/app/synchrotron.py

@@ -17,11 +17,6 @@ import contextlib
 import logging
 import sys
 
-from six import iteritems
-
-from twisted.internet import defer, reactor
-from twisted.web.resource import NoResource
-
 import synapse
 from synapse.api.constants import EventTypes
 from synapse.app import _base
@@ -41,12 +36,12 @@ from synapse.replication.slave.storage.deviceinbox import SlavedDeviceInboxStore
 from synapse.replication.slave.storage.devices import SlavedDeviceStore
 from synapse.replication.slave.storage.events import SlavedEventStore
 from synapse.replication.slave.storage.filtering import SlavedFilteringStore
-from synapse.replication.slave.storage.groups import SlavedGroupServerStore
 from synapse.replication.slave.storage.presence import SlavedPresenceStore
 from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
 from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
 from synapse.replication.slave.storage.registration import SlavedRegistrationStore
 from synapse.replication.slave.storage.room import RoomStore
+from synapse.replication.slave.storage.groups import SlavedGroupServerStore
 from synapse.replication.tcp.client import ReplicationClientHandler
 from synapse.rest.client.v1 import events
 from synapse.rest.client.v1.initial_sync import InitialSyncRestServlet
@@ -61,6 +56,10 @@ from synapse.util.logcontext import LoggingContext, run_in_background
 from synapse.util.manhole import manhole
 from synapse.util.stringutils import random_string
 from synapse.util.versionstring import get_version_string
+from twisted.internet import defer, reactor
+from twisted.web.resource import NoResource
+
+from six import iteritems
 
 logger = logging.getLogger("synapse.app.synchrotron")
 

synapse/app/synctl.py

@@ -16,16 +16,15 @@
 
 import argparse
 import collections
-import errno
 import glob
 import os
 import os.path
 import signal
 import subprocess
 import sys
-import time
-
 import yaml
+import errno
+import time
 
 SYNAPSE = [sys.executable, "-B", "-m", "synapse.app.homeserver"]
 

synapse/app/user_dir.py

@@ -17,9 +17,6 @@
 import logging
 import sys
 
-from twisted.internet import defer, reactor
-from twisted.web.resource import NoResource
-
 import synapse
 from synapse import events
 from synapse.app import _base
@@ -46,6 +43,8 @@ from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext, run_in_background
 from synapse.util.manhole import manhole
 from synapse.util.versionstring import get_version_string
+from twisted.internet import reactor, defer
+from twisted.web.resource import NoResource
 
 logger = logging.getLogger("synapse.app.user_dir")
 

synapse/appservice/__init__.py

@@ -12,17 +12,17 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+from synapse.api.constants import EventTypes
+from synapse.util.caches.descriptors import cachedInlineCallbacks
+from synapse.types import GroupID, get_domain_from_id
+
+from twisted.internet import defer
+
 import logging
 import re
 
 from six import string_types
 
-from twisted.internet import defer
-
-from synapse.api.constants import EventTypes
-from synapse.types import GroupID, get_domain_from_id
-from synapse.util.caches.descriptors import cachedInlineCallbacks
-
 logger = logging.getLogger(__name__)
 
 
@@ -85,8 +85,7 @@ class ApplicationService(object):
     NS_LIST = [NS_USERS, NS_ALIASES, NS_ROOMS]
 
     def __init__(self, token, hostname, url=None, namespaces=None, hs_token=None,
-                 sender=None, id=None, protocols=None, rate_limited=True,
-                 ip_range_whitelist=None):
+                 sender=None, id=None, protocols=None, rate_limited=True):
         self.token = token
         self.url = url
         self.hs_token = hs_token
@@ -94,7 +93,6 @@ class ApplicationService(object):
         self.server_name = hostname
         self.namespaces = self._check_namespaces(namespaces)
         self.id = id
-        self.ip_range_whitelist = ip_range_whitelist
 
         if "|" in self.id:
             raise Exception("application service ID cannot contain '|' character")

synapse/appservice/api.py

@@ -12,19 +12,19 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
-import urllib
-
-from prometheus_client import Counter
-
 from twisted.internet import defer
 
 from synapse.api.constants import ThirdPartyEntityKind
 from synapse.api.errors import CodeMessageException
-from synapse.events.utils import serialize_event
 from synapse.http.client import SimpleHttpClient
-from synapse.types import ThirdPartyInstanceID
+from synapse.events.utils import serialize_event
 from synapse.util.caches.response_cache import ResponseCache
+from synapse.types import ThirdPartyInstanceID
+
+import logging
+import urllib
+
+from prometheus_client import Counter
 
 logger = logging.getLogger(__name__)
 

synapse/appservice/scheduler.py

@@ -48,14 +48,14 @@ UP & quit           +---------- YES                       SUCCESS
 This is all tied together by the AppServiceScheduler which DIs the required
 components.
 """
-import logging
-
 from twisted.internet import defer
 
 from synapse.appservice import ApplicationServiceState
 from synapse.util.logcontext import run_in_background
 from synapse.util.metrics import Measure
 
+import logging
+
 logger = logging.getLogger(__name__)
 
 

synapse/config/_base.py

@@ -16,12 +16,11 @@
 import argparse
 import errno
 import os
+import yaml
 from textwrap import dedent
 
 from six import integer_types
 
-import yaml
-
 
 class ConfigError(Exception):
     pass
@@ -224,6 +223,11 @@ class Config(object):
             action="store_true",
             help="Generate a config file for the server name"
         )
+        config_parser.add_argument(
+            "--check-config",
+            action="store_true",
+            help="Check configuration supplied is valid"
+        )
         config_parser.add_argument(
             "--report-stats",
             action="store",
@@ -251,6 +255,8 @@ class Config(object):
         config_files = find_config_files(search_paths=config_args.config_path)
 
         generate_keys = config_args.generate_keys
+        
+        check_config = config_args.check_config
 
         obj = cls()
 
@@ -334,7 +340,9 @@ class Config(object):
         if generate_keys:
             return None
 
-        obj.invoke_all("read_arguments", args)
+        obj.invoke_all("read_arguments", args)  
+        if check_config:
+            return None
 
         return obj
 

synapse/config/api.py

@@ -12,10 +12,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from synapse.api.constants import EventTypes
-
 from ._base import Config
 
+from synapse.api.constants import EventTypes
+
 
 class ApiConfig(Config):
 

synapse/config/appservice.py

@@ -12,18 +12,16 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import logging
-
-from six import string_types
-from six.moves.urllib import parse as urlparse
-
-import yaml
-from netaddr import IPSet
+from ._base import Config, ConfigError
 
 from synapse.appservice import ApplicationService
 from synapse.types import UserID
 
-from ._base import Config, ConfigError
+import yaml
+import logging
+
+from six import string_types
+from six.moves.urllib import parse as urlparse
 
 logger = logging.getLogger(__name__)
 
@@ -156,13 +154,6 @@ def _load_appservice(hostname, as_info, config_filename):
             " will not receive events or queries.",
             config_filename,
         )
-
-    ip_range_whitelist = None
-    if as_info.get('ip_range_whitelist'):
-        ip_range_whitelist = IPSet(
-            as_info.get('ip_range_whitelist')
-        )
-
     return ApplicationService(
         token=as_info["as_token"],
         hostname=hostname,
@@ -172,6 +163,5 @@ def _load_appservice(hostname, as_info, config_filename):
         sender=user_id,
         id=as_info["id"],
         protocols=protocols,
-        rate_limited=rate_limited,
-        ip_range_whitelist=ip_range_whitelist,
+        rate_limited=rate_limited
     )

synapse/config/homeserver.py

@@ -13,32 +13,32 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+from .tls import TlsConfig
+from .server import ServerConfig
+from .logger import LoggingConfig
+from .database import DatabaseConfig
+from .ratelimiting import RatelimitConfig
+from .repository import ContentRepositoryConfig
+from .captcha import CaptchaConfig
+from .voip import VoipConfig
+from .registration import RegistrationConfig
+from .metrics import MetricsConfig
 from .api import ApiConfig
 from .appservice import AppServiceConfig
-from .captcha import CaptchaConfig
-from .cas import CasConfig
-from .consent_config import ConsentConfig
-from .database import DatabaseConfig
-from .emailconfig import EmailConfig
-from .groups import GroupsConfig
-from .jwt import JWTConfig
 from .key import KeyConfig
-from .logger import LoggingConfig
-from .metrics import MetricsConfig
-from .password import PasswordConfig
-from .password_auth_providers import PasswordAuthProviderConfig
-from .push import PushConfig
-from .ratelimiting import RatelimitConfig
-from .registration import RegistrationConfig
-from .repository import ContentRepositoryConfig
 from .saml2 import SAML2Config
-from .server import ServerConfig
-from .server_notices_config import ServerNoticesConfig
-from .spam_checker import SpamCheckerConfig
-from .tls import TlsConfig
-from .user_directory import UserDirectoryConfig
-from .voip import VoipConfig
+from .cas import CasConfig
+from .password import PasswordConfig
+from .jwt import JWTConfig
+from .password_auth_providers import PasswordAuthProviderConfig
+from .emailconfig import EmailConfig
 from .workers import WorkerConfig
+from .push import PushConfig
+from .spam_checker import SpamCheckerConfig
+from .groups import GroupsConfig
+from .user_directory import UserDirectoryConfig
+from .consent_config import ConsentConfig
+from .server_notices_config import ServerNoticesConfig
 
 
 class HomeServerConfig(TlsConfig, ServerConfig, DatabaseConfig, LoggingConfig,

synapse/config/jwt.py

@@ -15,6 +15,7 @@
 
 from ._base import Config, ConfigError
 
+
 MISSING_JWT = (
     """Missing jwt library. This is required for jwt login.
 

synapse/config/key.py

@@ -13,24 +13,21 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import hashlib
-import logging
-import os
+from ._base import Config, ConfigError
 
+from synapse.util.stringutils import random_string
 from signedjson.key import (
-    NACL_ED25519,
-    decode_signing_key_base64,
-    decode_verify_key_bytes,
-    generate_signing_key,
-    is_signing_algorithm_supported,
-    read_signing_keys,
-    write_signing_keys,
+    generate_signing_key, is_signing_algorithm_supported,
+    decode_signing_key_base64, decode_verify_key_bytes,
+    read_signing_keys, write_signing_keys, NACL_ED25519
 )
 from unpaddedbase64 import decode_base64
+from synapse.util.stringutils import random_string_with_symbols
 
-from synapse.util.stringutils import random_string, random_string_with_symbols
+import os
+import hashlib
+import logging
 
-from ._base import Config, ConfigError
 
 logger = logging.getLogger(__name__)
 

synapse/config/logger.py

@@ -12,22 +12,17 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
-import logging.config
-import os
-import signal
-import sys
-from string import Template
-
-import yaml
-
-from twisted.logger import STDLibLogObserver, globalLogBeginner
-
-import synapse
-from synapse.util.logcontext import LoggingContextFilter
-from synapse.util.versionstring import get_version_string
 
 from ._base import Config
+from synapse.util.logcontext import LoggingContextFilter
+from twisted.logger import globalLogBeginner, STDLibLogObserver
+import logging
+import logging.config
+import yaml
+from string import Template
+import os
+import signal
+
 
 DEFAULT_LOG_CONFIG = Template("""
 version: 1
@@ -207,15 +202,6 @@ def setup_logging(config, use_worker_options=False):
     if getattr(signal, "SIGHUP"):
         signal.signal(signal.SIGHUP, sighup)
 
-    # make sure that the first thing we log is a thing we can grep backwards
-    # for
-    logging.warn("***** STARTING SERVER *****")
-    logging.warn(
-        "Server %s version %s",
-        sys.argv[0], get_version_string(synapse),
-    )
-    logging.info("Server hostname: %s", config.server_name)
-
     # It's critical to point twisted's internal logging somewhere, otherwise it
     # stacks up and leaks kup to 64K object;
     # see: https://twistedmatrix.com/trac/ticket/8164

synapse/config/password_auth_providers.py

@@ -13,10 +13,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from synapse.util.module_loader import load_module
-
 from ._base import Config
 
+from synapse.util.module_loader import load_module
+
 LDAP_PROVIDER = 'ldap_auth_provider.LdapAuthProvider'
 
 

synapse/config/registration.py

@@ -13,11 +13,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from distutils.util import strtobool
+from ._base import Config
 
 from synapse.util.stringutils import random_string_with_symbols
 
-from ._base import Config
+from distutils.util import strtobool
 
 
 class RegistrationConfig(Config):

synapse/config/repository.py

@@ -13,11 +13,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from ._base import Config, ConfigError
 from collections import namedtuple
 
 from synapse.util.module_loader import load_module
 
-from ._base import Config, ConfigError
 
 MISSING_NETADDR = (
     "Missing netaddr library. This is required for URL preview API."

synapse/config/server.py

@@ -16,8 +16,6 @@
 
 import logging
 
-from synapse.http.endpoint import parse_and_validate_server_name
-
 from ._base import Config, ConfigError
 
 logger = logging.Logger(__name__)
@@ -27,12 +25,6 @@ class ServerConfig(Config):
 
     def read_config(self, config):
         self.server_name = config["server_name"]
-
-        try:
-            parse_and_validate_server_name(self.server_name)
-        except ValueError as e:
-            raise ConfigError(str(e))
-
         self.pid_file = self.abspath(config.get("pid_file"))
         self.web_client = config["web_client"]
         self.web_client_location = config.get("web_client_location", None)
@@ -170,8 +162,8 @@ class ServerConfig(Config):
             })
 
     def default_config(self, server_name, **kwargs):
-        _, bind_port = parse_and_validate_server_name(server_name)
-        if bind_port is not None:
+        if ":" in server_name:
+            bind_port = int(server_name.split(":")[1])
             unsecure_port = bind_port - 400
         else:
             bind_port = 8448

synapse/config/server_notices_config.py

@@ -12,9 +12,8 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-from synapse.types import UserID
-
 from ._base import Config
+from synapse.types import UserID
 
 DEFAULT_CONFIG = """\
 # Server Notices room configuration

synapse/config/stats.py

@@ -1,46 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2018 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from ._base import Config
-
-import sys
-
-
-class StatsConfig(Config):
-    """Stats Configuration
-    Configuration for the behaviour of synapse's stats engine
-    """
-
-    def read_config(self, config):
-        self.stats_enable = False
-        self.stats_bucket_size = 86400
-        self.stats_retention = sys.maxint
-        stats_config = config.get("stats", None)
-        if stats_config:
-            self.stats_enable = stats_config.get("enable", self.stats_enable)
-            self.stats_bucket_size = stats_config.get(
-                "bucket_size", self.stats_bucket_size
-            )
-            self.stats_retention = stats_config.get("retention", self.stats_retention)
-
-    def default_config(self, config_dir_path, server_name, **kwargs):
-        return """
-        # Stats configuration
-        #
-        # stats:
-        #    enable: false
-        #    bucket_size: 86400 # 1 day
-        #    retention: 31536000 # 1 year
-        """

synapse/config/tls.py

@@ -13,15 +13,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import os
-import subprocess
-from hashlib import sha256
-
-from unpaddedbase64 import encode_base64
+from ._base import Config
 
 from OpenSSL import crypto
+import subprocess
+import os
 
-from ._base import Config
+from hashlib import sha256
+from unpaddedbase64 import encode_base64
 
 GENERATE_DH_PARAMS = False
 

synapse/crypto/context_factory.py

@@ -12,12 +12,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import logging
-
-from OpenSSL import SSL, crypto
 from twisted.internet import ssl
+from OpenSSL import SSL, crypto
 from twisted.internet._sslverify import _defaultCurveName
 
+import logging
+
 logger = logging.getLogger(__name__)
 
 

synapse/crypto/event_signing.py

@@ -15,15 +15,15 @@
 # limitations under the License.
 
 
-import hashlib
-import logging
+from synapse.api.errors import SynapseError, Codes
+from synapse.events.utils import prune_event
 
 from canonicaljson import encode_canonical_json
+from unpaddedbase64 import encode_base64, decode_base64
 from signedjson.sign import sign_json
-from unpaddedbase64 import decode_base64, encode_base64
 
-from synapse.api.errors import Codes, SynapseError
-from synapse.events.utils import prune_event
+import hashlib
+import logging
 
 logger = logging.getLogger(__name__)
 

synapse/crypto/keyclient.py

@@ -13,16 +13,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from synapse.util import logcontext
+from twisted.web.http import HTTPClient
+from twisted.internet.protocol import Factory
+from twisted.internet import defer, reactor
+from synapse.http.endpoint import matrix_federation_endpoint
+import simplejson as json
 import logging
 
-from canonicaljson import json
-
-from twisted.internet import defer, reactor
-from twisted.internet.protocol import Factory
-from twisted.web.http import HTTPClient
-
-from synapse.http.endpoint import matrix_federation_endpoint
-from synapse.util import logcontext
 
 logger = logging.getLogger(__name__)
 

synapse/crypto/keyring.py

@@ -14,31 +14,9 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import hashlib
-import logging
-import urllib
-from collections import namedtuple
-
-from signedjson.key import (
-    decode_verify_key_bytes,
-    encode_verify_key_base64,
-    is_signing_algorithm_supported,
-)
-from signedjson.sign import (
-    SignatureVerifyException,
-    encode_canonical_json,
-    sign_json,
-    signature_ids,
-    verify_signed_json,
-)
-from unpaddedbase64 import decode_base64, encode_base64
-
-from OpenSSL import crypto
-from twisted.internet import defer
-
-from synapse.api.errors import Codes, SynapseError
 from synapse.crypto.keyclient import fetch_server_key
-from synapse.util import logcontext, unwrapFirstError
+from synapse.api.errors import SynapseError, Codes
+from synapse.util import unwrapFirstError, logcontext
 from synapse.util.logcontext import (
     PreserveLoggingContext,
     preserve_fn,
@@ -46,6 +24,26 @@ from synapse.util.logcontext import (
 )
 from synapse.util.metrics import Measure
 
+from twisted.internet import defer
+
+from signedjson.sign import (
+    verify_signed_json, signature_ids, sign_json, encode_canonical_json,
+    SignatureVerifyException,
+)
+from signedjson.key import (
+    is_signing_algorithm_supported, decode_verify_key_bytes,
+    encode_verify_key_base64,
+)
+from unpaddedbase64 import decode_base64, encode_base64
+
+from OpenSSL import crypto
+
+from collections import namedtuple
+import urllib
+import hashlib
+import logging
+
+
 logger = logging.getLogger(__name__)
 
 

synapse/event_auth.py

@@ -17,11 +17,11 @@ import logging
 
 from canonicaljson import encode_canonical_json
 from signedjson.key import decode_verify_key_bytes
-from signedjson.sign import SignatureVerifyException, verify_signed_json
+from signedjson.sign import verify_signed_json, SignatureVerifyException
 from unpaddedbase64 import decode_base64
 
-from synapse.api.constants import EventTypes, JoinRules, Membership
-from synapse.api.errors import AuthError, EventSizeError, SynapseError
+from synapse.api.constants import EventTypes, Membership, JoinRules
+from synapse.api.errors import AuthError, SynapseError, EventSizeError
 from synapse.types import UserID, get_domain_from_id
 
 logger = logging.getLogger(__name__)
@@ -76,7 +76,6 @@ def check(event, auth_events, do_sig_check=True, do_size_check=True):
         return
 
     if event.type == EventTypes.Create:
-        sender_domain = get_domain_from_id(event.sender)
         room_id_domain = get_domain_from_id(event.room_id)
         if room_id_domain != sender_domain:
             raise AuthError(
@@ -525,11 +524,7 @@ def _check_power_levels(event, auth_events):
                     "to your own"
                 )
 
-        # Check if the old and new levels are greater than the user level
-        # (if defined)
-        old_level_too_big = old_level is not None and old_level > user_level
-        new_level_too_big = new_level is not None and new_level > user_level
-        if old_level_too_big or new_level_too_big:
+        if old_level > user_level or new_level > user_level:
             raise AuthError(
                 403,
                 "You don't have permission to add ops level greater "

synapse/events/__init__.py

@@ -13,8 +13,9 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from synapse.util.caches import intern_dict
 from synapse.util.frozenutils import freeze
+from synapse.util.caches import intern_dict
+
 
 # Whether we should use frozen_dict in FrozenEvent. Using frozen_dicts prevents
 # bugs where we accidentally share e.g. signature dicts. However, converting

synapse/events/builder.py

@@ -13,12 +13,13 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import copy
+from . import EventBase, FrozenEvent, _event_dict_property
 
 from synapse.types import EventID
+
 from synapse.util.stringutils import random_string
 
-from . import EventBase, FrozenEvent, _event_dict_property
+import copy
 
 
 class EventBuilder(EventBase):

synapse/events/snapshot.py

@@ -13,10 +13,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from frozendict import frozendict
-
 from twisted.internet import defer
 
+from frozendict import frozendict
+
 
 class EventContext(object):
     """

synapse/events/utils.py

@@ -13,15 +13,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import re
-
-from six import string_types
+from synapse.api.constants import EventTypes
+from . import EventBase
 
 from frozendict import frozendict
 
-from synapse.api.constants import EventTypes
+import re
 
-from . import EventBase
+from six import string_types
 
 # Split strings on "." but not "\." This uses a negative lookbehind assertion for '\'
 # (?<!stuff) matches if the current position in the string is not preceded

synapse/events/validator.py

@@ -13,11 +13,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from six import string_types
-
-from synapse.api.constants import EventTypes, Membership
-from synapse.api.errors import SynapseError
 from synapse.types import EventID, RoomID, UserID
+from synapse.api.errors import SynapseError
+from synapse.api.constants import EventTypes, Membership
+
+from six import string_types
 
 
 class EventValidator(object):

synapse/federation/federation_base.py

@@ -16,15 +16,14 @@ import logging
 
 import six
 
-from twisted.internet import defer
-
 from synapse.api.constants import MAX_DEPTH
-from synapse.api.errors import Codes, SynapseError
+from synapse.api.errors import SynapseError, Codes
 from synapse.crypto.event_signing import check_event_content_hash
 from synapse.events import FrozenEvent
 from synapse.events.utils import prune_event
-from synapse.http.servlet import assert_params_in_dict
-from synapse.util import logcontext, unwrapFirstError
+from synapse.http.servlet import assert_params_in_request
+from synapse.util import unwrapFirstError, logcontext
+from twisted.internet import defer
 
 logger = logging.getLogger(__name__)
 
@@ -199,7 +198,7 @@ def event_from_pdu_json(pdu_json, outlier=False):
     """
     # we could probably enforce a bunch of other fields here (room_id, sender,
     # origin, etc etc)
-    assert_params_in_dict(pdu_json, ('event_id', 'type', 'depth'))
+    assert_params_in_request(pdu_json, ('event_id', 'type', 'depth'))
 
     depth = pdu_json['depth']
     if not isinstance(depth, six.integer_types):

synapse/federation/federation_client.py

@@ -21,25 +21,25 @@ import random
 
 from six.moves import range
 
-from prometheus_client import Counter
-
 from twisted.internet import defer
 
 from synapse.api.constants import Membership
 from synapse.api.errors import (
-    CodeMessageException,
-    FederationDeniedError,
-    HttpResponseException,
-    SynapseError,
+    CodeMessageException, HttpResponseException, SynapseError, FederationDeniedError
 )
 from synapse.events import builder
-from synapse.federation.federation_base import FederationBase, event_from_pdu_json
+from synapse.federation.federation_base import (
+    FederationBase,
+    event_from_pdu_json,
+)
 from synapse.util import logcontext, unwrapFirstError
 from synapse.util.caches.expiringcache import ExpiringCache
 from synapse.util.logcontext import make_deferred_yieldable, run_in_background
 from synapse.util.logutils import log_function
 from synapse.util.retryutils import NotRetryingDestination
 
+from prometheus_client import Counter
+
 logger = logging.getLogger(__name__)
 
 sent_queries_counter = Counter("synapse_federation_client_sent_queries", "", ["type"])

synapse/federation/federation_server.py

@@ -14,29 +14,28 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import logging
-import re
-
-import six
-from six import iteritems
-
-from canonicaljson import json
-from prometheus_client import Counter
 
+import simplejson as json
 from twisted.internet import defer
-from twisted.internet.abstract import isIPAddress
 
-from synapse.api.constants import EventTypes
-from synapse.api.errors import AuthError, FederationError, NotFoundError, SynapseError
+from synapse.api.errors import AuthError, FederationError, SynapseError, NotFoundError
 from synapse.crypto.event_signing import compute_event_signature
-from synapse.federation.federation_base import FederationBase, event_from_pdu_json
+from synapse.federation.federation_base import (
+    FederationBase,
+    event_from_pdu_json,
+)
+
 from synapse.federation.persistence import TransactionActions
 from synapse.federation.units import Edu, Transaction
-from synapse.http.endpoint import parse_server_name
 from synapse.types import get_domain_from_id
 from synapse.util import async
 from synapse.util.caches.response_cache import ResponseCache
 from synapse.util.logutils import log_function
 
+from prometheus_client import Counter
+
+from six import iteritems
+
 # when processing incoming transactions, we try to handle multiple rooms in
 # parallel, up to this limit.
 TRANSACTION_CONCURRENCY_LIMIT = 10
@@ -75,9 +74,6 @@ class FederationServer(FederationBase):
     @log_function
     def on_backfill_request(self, origin, room_id, versions, limit):
         with (yield self._server_linearizer.queue((origin, room_id))):
-            origin_host, _ = parse_server_name(origin)
-            yield self.check_server_matches_acl(origin_host, room_id)
-
             pdus = yield self.handler.on_backfill_request(
                 origin, room_id, versions, limit
             )
@@ -138,8 +134,6 @@ class FederationServer(FederationBase):
 
         received_pdus_counter.inc(len(transaction.pdus))
 
-        origin_host, _ = parse_server_name(transaction.origin)
-
         pdus_by_room = {}
 
         for p in transaction.pdus:
@@ -160,21 +154,9 @@ class FederationServer(FederationBase):
         # we can process different rooms in parallel (which is useful if they
         # require callouts to other servers to fetch missing events), but
         # impose a limit to avoid going too crazy with ram/cpu.
-
         @defer.inlineCallbacks
         def process_pdus_for_room(room_id):
             logger.debug("Processing PDUs for %s", room_id)
-            try:
-                yield self.check_server_matches_acl(origin_host, room_id)
-            except AuthError as e:
-                logger.warn(
-                    "Ignoring PDUs for room %s from banned server", room_id,
-                )
-                for pdu in pdus_by_room[room_id]:
-                    event_id = pdu.event_id
-                    pdu_results[event_id] = e.error_dict()
-                return
-
             for pdu in pdus_by_room[room_id]:
                 event_id = pdu.event_id
                 try:
@@ -229,9 +211,6 @@ class FederationServer(FederationBase):
         if not event_id:
             raise NotImplementedError("Specify an event")
 
-        origin_host, _ = parse_server_name(origin)
-        yield self.check_server_matches_acl(origin_host, room_id)
-
         in_room = yield self.auth.check_host_in_room(room_id, origin)
         if not in_room:
             raise AuthError(403, "Host not in room.")
@@ -255,9 +234,6 @@ class FederationServer(FederationBase):
         if not event_id:
             raise NotImplementedError("Specify an event")
 
-        origin_host, _ = parse_server_name(origin)
-        yield self.check_server_matches_acl(origin_host, room_id)
-
         in_room = yield self.auth.check_host_in_room(room_id, origin)
         if not in_room:
             raise AuthError(403, "Host not in room.")
@@ -301,7 +277,7 @@ class FederationServer(FederationBase):
     @defer.inlineCallbacks
     @log_function
     def on_pdu_request(self, origin, event_id):
-        pdu = yield self.handler.get_persisted_pdu(origin, event_id)
+        pdu = yield self._get_persisted_pdu(origin, event_id)
 
         if pdu:
             defer.returnValue(
@@ -322,9 +298,7 @@ class FederationServer(FederationBase):
         defer.returnValue((200, resp))
 
     @defer.inlineCallbacks
-    def on_make_join_request(self, origin, room_id, user_id):
-        origin_host, _ = parse_server_name(origin)
-        yield self.check_server_matches_acl(origin_host, room_id)
+    def on_make_join_request(self, room_id, user_id):
         pdu = yield self.handler.on_make_join_request(room_id, user_id)
         time_now = self._clock.time_msec()
         defer.returnValue({"event": pdu.get_pdu_json(time_now)})
@@ -332,8 +306,6 @@ class FederationServer(FederationBase):
     @defer.inlineCallbacks
     def on_invite_request(self, origin, content):
         pdu = event_from_pdu_json(content)
-        origin_host, _ = parse_server_name(origin)
-        yield self.check_server_matches_acl(origin_host, pdu.room_id)
         ret_pdu = yield self.handler.on_invite_request(origin, pdu)
         time_now = self._clock.time_msec()
         defer.returnValue((200, {"event": ret_pdu.get_pdu_json(time_now)}))
@@ -342,10 +314,6 @@ class FederationServer(FederationBase):
     def on_send_join_request(self, origin, content):
         logger.debug("on_send_join_request: content: %s", content)
         pdu = event_from_pdu_json(content)
-
-        origin_host, _ = parse_server_name(origin)
-        yield self.check_server_matches_acl(origin_host, pdu.room_id)
-
         logger.debug("on_send_join_request: pdu sigs: %s", pdu.signatures)
         res_pdus = yield self.handler.on_send_join_request(origin, pdu)
         time_now = self._clock.time_msec()
@@ -357,9 +325,7 @@ class FederationServer(FederationBase):
         }))
 
     @defer.inlineCallbacks
-    def on_make_leave_request(self, origin, room_id, user_id):
-        origin_host, _ = parse_server_name(origin)
-        yield self.check_server_matches_acl(origin_host, room_id)
+    def on_make_leave_request(self, room_id, user_id):
         pdu = yield self.handler.on_make_leave_request(room_id, user_id)
         time_now = self._clock.time_msec()
         defer.returnValue({"event": pdu.get_pdu_json(time_now)})
@@ -368,10 +334,6 @@ class FederationServer(FederationBase):
     def on_send_leave_request(self, origin, content):
         logger.debug("on_send_leave_request: content: %s", content)
         pdu = event_from_pdu_json(content)
-
-        origin_host, _ = parse_server_name(origin)
-        yield self.check_server_matches_acl(origin_host, pdu.room_id)
-
         logger.debug("on_send_leave_request: pdu sigs: %s", pdu.signatures)
         yield self.handler.on_send_leave_request(origin, pdu)
         defer.returnValue((200, {}))
@@ -379,9 +341,6 @@ class FederationServer(FederationBase):
     @defer.inlineCallbacks
     def on_event_auth(self, origin, room_id, event_id):
         with (yield self._server_linearizer.queue((origin, room_id))):
-            origin_host, _ = parse_server_name(origin)
-            yield self.check_server_matches_acl(origin_host, room_id)
-
             time_now = self._clock.time_msec()
             auth_pdus = yield self.handler.on_event_auth(event_id)
             res = {
@@ -410,9 +369,6 @@ class FederationServer(FederationBase):
             Deferred: Results in `dict` with the same format as `content`
         """
         with (yield self._server_linearizer.queue((origin, room_id))):
-            origin_host, _ = parse_server_name(origin)
-            yield self.check_server_matches_acl(origin_host, room_id)
-
             auth_chain = [
                 event_from_pdu_json(e)
                 for e in content["auth_chain"]
@@ -486,9 +442,6 @@ class FederationServer(FederationBase):
     def on_get_missing_events(self, origin, room_id, earliest_events,
                               latest_events, limit, min_depth):
         with (yield self._server_linearizer.queue((origin, room_id))):
-            origin_host, _ = parse_server_name(origin)
-            yield self.check_server_matches_acl(origin_host, room_id)
-
             logger.info(
                 "on_get_missing_events: earliest_events: %r, latest_events: %r,"
                 " limit: %d, min_depth: %d",
@@ -517,6 +470,17 @@ class FederationServer(FederationBase):
         ts_now_ms = self._clock.time_msec()
         return self.store.get_user_id_for_open_id_token(token, ts_now_ms)
 
+    @log_function
+    def _get_persisted_pdu(self, origin, event_id, do_auth=True):
+        """ Get a PDU from the database with given origin and id.
+
+        Returns:
+            Deferred: Results in a `Pdu`.
+        """
+        return self.handler.get_persisted_pdu(
+            origin, event_id, do_auth=do_auth
+        )
+
     def _transaction_from_pdus(self, pdu_list):
         """Returns a new Transaction containing the given PDUs suitable for
         transmission.
@@ -596,9 +560,7 @@ class FederationServer(FederationBase):
                 affected=pdu.event_id,
             )
 
-        yield self.handler.on_receive_pdu(
-            origin, pdu, get_missing=True, sent_to_us_directly=True,
-        )
+        yield self.handler.on_receive_pdu(origin, pdu, get_missing=True)
 
     def __str__(self):
         return "<ReplicationLayer(%s)>" % self.server_name
@@ -626,101 +588,6 @@ class FederationServer(FederationBase):
         )
         defer.returnValue(ret)
 
-    @defer.inlineCallbacks
-    def check_server_matches_acl(self, server_name, room_id):
-        """Check if the given server is allowed by the server ACLs in the room
-
-        Args:
-            server_name (str): name of server, *without any port part*
-            room_id (str): ID of the room to check
-
-        Raises:
-            AuthError if the server does not match the ACL
-        """
-        state_ids = yield self.store.get_current_state_ids(room_id)
-        acl_event_id = state_ids.get((EventTypes.ServerACL, ""))
-
-        if not acl_event_id:
-            return
-
-        acl_event = yield self.store.get_event(acl_event_id)
-        if server_matches_acl_event(server_name, acl_event):
-            return
-
-        raise AuthError(code=403, msg="Server is banned from room")
-
-
-def server_matches_acl_event(server_name, acl_event):
-    """Check if the given server is allowed by the ACL event
-
-    Args:
-        server_name (str): name of server, without any port part
-        acl_event (EventBase): m.room.server_acl event
-
-    Returns:
-        bool: True if this server is allowed by the ACLs
-    """
-    logger.debug("Checking %s against acl %s", server_name, acl_event.content)
-
-    # first of all, check if literal IPs are blocked, and if so, whether the
-    # server name is a literal IP
-    allow_ip_literals = acl_event.content.get("allow_ip_literals", True)
-    if not isinstance(allow_ip_literals, bool):
-        logger.warn("Ignorning non-bool allow_ip_literals flag")
-        allow_ip_literals = True
-    if not allow_ip_literals:
-        # check for ipv6 literals. These start with '['.
-        if server_name[0] == '[':
-            return False
-
-        # check for ipv4 literals. We can just lift the routine from twisted.
-        if isIPAddress(server_name):
-            return False
-
-    # next,  check the deny list
-    deny = acl_event.content.get("deny", [])
-    if not isinstance(deny, (list, tuple)):
-        logger.warn("Ignorning non-list deny ACL %s", deny)
-        deny = []
-    for e in deny:
-        if _acl_entry_matches(server_name, e):
-            # logger.info("%s matched deny rule %s", server_name, e)
-            return False
-
-    # then the allow list.
-    allow = acl_event.content.get("allow", [])
-    if not isinstance(allow, (list, tuple)):
-        logger.warn("Ignorning non-list allow ACL %s", allow)
-        allow = []
-    for e in allow:
-        if _acl_entry_matches(server_name, e):
-            # logger.info("%s matched allow rule %s", server_name, e)
-            return True
-
-    # everything else should be rejected.
-    # logger.info("%s fell through", server_name)
-    return False
-
-
-def _acl_entry_matches(server_name, acl_entry):
-    if not isinstance(acl_entry, six.string_types):
-        logger.warn("Ignoring non-str ACL entry '%s' (is %s)", acl_entry, type(acl_entry))
-        return False
-    regex = _glob_to_regex(acl_entry)
-    return regex.match(server_name)
-
-
-def _glob_to_regex(glob):
-    res = ''
-    for c in glob:
-        if c == '*':
-            res = res + '.*'
-        elif c == '?':
-            res = res + '.'
-        else:
-            res = res + re.escape(c)
-    return re.compile(res + "\\Z", re.IGNORECASE)
-
 
 class FederationHandlerRegistry(object):
     """Allows classes to register themselves as handlers for a given EDU or

synapse/federation/persistence.py

@@ -19,12 +19,13 @@ package.
 These actions are mostly only used by the :py:mod:`.replication` module.
 """
 
-import logging
-
 from twisted.internet import defer
 
 from synapse.util.logutils import log_function
 
+import logging
+
+
 logger = logging.getLogger(__name__)
 
 

synapse/federation/send_queue.py

@@ -29,18 +29,18 @@ dead worker doesn't cause the queues to grow limitlessly.
 Events are replicated via a separate events stream.
 """
 
-import logging
-from collections import namedtuple
+from .units import Edu
 
-from six import iteritems, itervalues
-
-from sortedcontainers import SortedDict
-
-from synapse.metrics import LaterGauge
 from synapse.storage.presence import UserPresenceState
 from synapse.util.metrics import Measure
+from synapse.metrics import LaterGauge
 
-from .units import Edu
+from sortedcontainers import SortedDict
+from collections import namedtuple
+
+import logging
+
+from six import itervalues, iteritems
 
 logger = logging.getLogger(__name__)
 

synapse/federation/transaction_queue.py

@@ -13,37 +13,36 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import datetime
-import logging
-
-from six import itervalues
-
-from prometheus_client import Counter
 
 from twisted.internet import defer
 
-import synapse.metrics
-from synapse.api.errors import FederationDeniedError, HttpResponseException
+from .persistence import TransactionActions
+from .units import Transaction, Edu
+
+from synapse.api.errors import HttpResponseException, FederationDeniedError
+from synapse.util import logcontext, PreserveLoggingContext
+from synapse.util.retryutils import NotRetryingDestination, get_retry_limiter
+from synapse.util.metrics import measure_func
 from synapse.handlers.presence import format_user_presence_state, get_interested_remotes
+import synapse.metrics
+from synapse.metrics import LaterGauge
 from synapse.metrics import (
-    LaterGauge,
-    events_processed_counter,
     sent_edus_counter,
     sent_transactions_counter,
+    events_processed_counter,
 )
-from synapse.util import PreserveLoggingContext, logcontext
-from synapse.util.metrics import measure_func
-from synapse.util.retryutils import NotRetryingDestination, get_retry_limiter
 
-from .persistence import TransactionActions
-from .units import Edu, Transaction
+from prometheus_client import Counter
+
+from six import itervalues
+
+import logging
+
 
 logger = logging.getLogger(__name__)
 
-sent_pdus_destination_dist_count = Counter(
-    "synapse_federation_client_sent_pdu_destinations:count", ""
-)
-sent_pdus_destination_dist_total = Counter(
-    "synapse_federation_client_sent_pdu_destinations:total", ""
+sent_pdus_destination_dist = Counter(
+    "synapse_federation_transaction_queue_sent_pdu_destinations", ""
 )
 
 
@@ -280,8 +279,7 @@ class TransactionQueue(object):
         if not destinations:
             return
 
-        sent_pdus_destination_dist_total.inc(len(destinations))
-        sent_pdus_destination_dist_count.inc()
+        sent_pdus_destination_dist.inc(len(destinations))
 
         for destination in destinations:
             self.pending_pdus_by_dest.setdefault(destination, []).append(

synapse/federation/transport/client.py

@@ -14,14 +14,15 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from twisted.internet import defer
+from synapse.api.constants import Membership
+
+from synapse.api.urls import FEDERATION_PREFIX as PREFIX
+from synapse.util.logutils import log_function
+
 import logging
 import urllib
 
-from twisted.internet import defer
-
-from synapse.api.constants import Membership
-from synapse.api.urls import FEDERATION_PREFIX as PREFIX
-from synapse.util.logutils import log_function
 
 logger = logging.getLogger(__name__)
 

synapse/federation/transport/server.py

@@ -14,27 +14,25 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from twisted.internet import defer
+
+from synapse.api.urls import FEDERATION_PREFIX as PREFIX
+from synapse.api.errors import Codes, SynapseError, FederationDeniedError
+from synapse.http.server import JsonResource
+from synapse.http.servlet import (
+    parse_json_object_from_request, parse_integer_from_args, parse_string_from_args,
+    parse_boolean_from_args,
+)
+from synapse.util.ratelimitutils import FederationRateLimiter
+from synapse.util.versionstring import get_version_string
+from synapse.util.logcontext import run_in_background
+from synapse.types import ThirdPartyInstanceID, get_domain_from_id
+
 import functools
 import logging
 import re
-
-from twisted.internet import defer
-
 import synapse
-from synapse.api.errors import Codes, FederationDeniedError, SynapseError
-from synapse.api.urls import FEDERATION_PREFIX as PREFIX
-from synapse.http.endpoint import parse_and_validate_server_name
-from synapse.http.server import JsonResource
-from synapse.http.servlet import (
-    parse_boolean_from_args,
-    parse_integer_from_args,
-    parse_json_object_from_request,
-    parse_string_from_args,
-)
-from synapse.types import ThirdPartyInstanceID, get_domain_from_id
-from synapse.util.logcontext import run_in_background
-from synapse.util.ratelimitutils import FederationRateLimiter
-from synapse.util.versionstring import get_version_string
+
 
 logger = logging.getLogger(__name__)
 
@@ -101,6 +99,26 @@ class Authenticator(object):
 
         origin = None
 
+        def parse_auth_header(header_str):
+            try:
+                params = auth.split(" ")[1].split(",")
+                param_dict = dict(kv.split("=") for kv in params)
+
+                def strip_quotes(value):
+                    if value.startswith("\""):
+                        return value[1:-1]
+                    else:
+                        return value
+
+                origin = strip_quotes(param_dict["origin"])
+                key = strip_quotes(param_dict["key"])
+                sig = strip_quotes(param_dict["sig"])
+                return (origin, key, sig)
+            except Exception:
+                raise AuthenticationError(
+                    400, "Malformed Authorization header", Codes.UNAUTHORIZED
+                )
+
         auth_headers = request.requestHeaders.getRawHeaders(b"Authorization")
 
         if not auth_headers:
@@ -109,8 +127,8 @@ class Authenticator(object):
             )
 
         for auth in auth_headers:
-            if auth.startswith(b"X-Matrix"):
-                (origin, key, sig) = _parse_auth_header(auth)
+            if auth.startswith("X-Matrix"):
+                (origin, key, sig) = parse_auth_header(auth)
                 json_request["origin"] = origin
                 json_request["signatures"].setdefault(origin, {})[key] = sig
 
@@ -147,48 +165,6 @@ class Authenticator(object):
             logger.exception("Error resetting retry timings on %s", origin)
 
 
-def _parse_auth_header(header_bytes):
-    """Parse an X-Matrix auth header
-
-    Args:
-        header_bytes (bytes): header value
-
-    Returns:
-        Tuple[str, str, str]: origin, key id, signature.
-
-    Raises:
-        AuthenticationError if the header could not be parsed
-    """
-    try:
-        header_str = header_bytes.decode('utf-8')
-        params = header_str.split(" ")[1].split(",")
-        param_dict = dict(kv.split("=") for kv in params)
-
-        def strip_quotes(value):
-            if value.startswith(b"\""):
-                return value[1:-1]
-            else:
-                return value
-
-        origin = strip_quotes(param_dict["origin"])
-
-        # ensure that the origin is a valid server name
-        parse_and_validate_server_name(origin)
-
-        key = strip_quotes(param_dict["key"])
-        sig = strip_quotes(param_dict["sig"])
-        return origin, key, sig
-    except Exception as e:
-        logger.warn(
-            "Error parsing auth header '%s': %s",
-            header_bytes.decode('ascii', 'replace'),
-            e,
-        )
-        raise AuthenticationError(
-            400, "Malformed Authorization header", Codes.UNAUTHORIZED,
-        )
-
-
 class BaseFederationServlet(object):
     REQUIRE_AUTH = True
 
@@ -386,9 +362,7 @@ class FederationMakeJoinServlet(BaseFederationServlet):
 
     @defer.inlineCallbacks
     def on_GET(self, origin, content, query, context, user_id):
-        content = yield self.handler.on_make_join_request(
-            origin, context, user_id,
-        )
+        content = yield self.handler.on_make_join_request(context, user_id)
         defer.returnValue((200, content))
 
 
@@ -397,9 +371,7 @@ class FederationMakeLeaveServlet(BaseFederationServlet):
 
     @defer.inlineCallbacks
     def on_GET(self, origin, content, query, context, user_id):
-        content = yield self.handler.on_make_leave_request(
-            origin, context, user_id,
-        )
+        content = yield self.handler.on_make_leave_request(context, user_id)
         defer.returnValue((200, content))
 
 

synapse/federation/units.py

@@ -17,9 +17,10 @@
 server protocol.
 """
 
+from synapse.util.jsonobject import JsonEncodedObject
+
 import logging
 
-from synapse.util.jsonobject import JsonEncodedObject
 
 logger = logging.getLogger(__name__)
 

synapse/groups/attestations.py

@@ -23,9 +23,9 @@ If a user leaves (or gets kicked out of) a group, either side can still use
 their attestation to "prove" their membership, until the attestation expires.
 Therefore attestations shouldn't be relied on to prove membership in important
 cases, but can for less important situtations, e.g. showing a users membership
-of groups on their profile, showing flairs, etc.
+of groups on their profile, showing flairs, etc.abs
 
-An attestation is a signed blob of json that looks like:
+An attestsation is a signed blob of json that looks like:
 
     {
         "user_id": "@foo:a.example.com",
@@ -38,14 +38,15 @@ An attestation is a signed blob of json that looks like:
 import logging
 import random
 
-from signedjson.sign import sign_json
-
 from twisted.internet import defer
 
 from synapse.api.errors import SynapseError
 from synapse.types import get_domain_from_id
 from synapse.util.logcontext import run_in_background
 
+from signedjson.sign import sign_json
+
+
 logger = logging.getLogger(__name__)
 
 

synapse/groups/groups_server.py

@@ -16,12 +16,11 @@
 
 import logging
 
-from six import string_types
-
-from twisted.internet import defer
-
 from synapse.api.errors import SynapseError
 from synapse.types import GroupID, RoomID, UserID, get_domain_from_id
+from twisted.internet import defer
+
+from six import string_types
 
 logger = logging.getLogger(__name__)
 

synapse/handlers/__init__.py

@@ -13,13 +13,13 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from .admin import AdminHandler
-from .directory import DirectoryHandler
-from .federation import FederationHandler
-from .identity import IdentityHandler
-from .message import MessageHandler
 from .register import RegistrationHandler
 from .room import RoomContextHandler
+from .message import MessageHandler
+from .federation import FederationHandler
+from .directory import DirectoryHandler
+from .admin import AdminHandler
+from .identity import IdentityHandler
 from .search import SearchHandler
 
 

synapse/handlers/_base.py

@@ -18,10 +18,11 @@ import logging
 from twisted.internet import defer
 
 import synapse.types
-from synapse.api.constants import EventTypes, Membership
+from synapse.api.constants import Membership, EventTypes
 from synapse.api.errors import LimitExceededError
 from synapse.types import UserID
 
+
 logger = logging.getLogger(__name__)
 
 

synapse/handlers/admin.py

@@ -13,12 +13,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import logging
-
 from twisted.internet import defer
 
 from ._base import BaseHandler
 
+import logging
+
 logger = logging.getLogger(__name__)
 
 

synapse/handlers/appservice.py

@@ -13,18 +13,19 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import logging
+from twisted.internet import defer
 
 from six import itervalues
 
-from prometheus_client import Counter
-
-from twisted.internet import defer
-
 import synapse
 from synapse.api.constants import EventTypes
-from synapse.util.logcontext import make_deferred_yieldable, run_in_background
 from synapse.util.metrics import Measure
+from synapse.util.logcontext import (
+    make_deferred_yieldable, run_in_background,
+)
+from prometheus_client import Counter
+
+import logging
 
 logger = logging.getLogger(__name__)
 

synapse/handlers/auth.py

@@ -14,24 +14,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import logging
-
-import attr
-import bcrypt
-import pymacaroons
-from canonicaljson import json
-
 from twisted.internet import defer, threads
-from twisted.web.client import PartialDownloadError
 
-import synapse.util.stringutils as stringutils
+from ._base import BaseHandler
 from synapse.api.constants import LoginType
 from synapse.api.errors import (
-    AuthError,
-    Codes,
-    InteractiveAuthIncompleteError,
-    LoginError,
-    StoreError,
+    AuthError, Codes, InteractiveAuthIncompleteError, LoginError, StoreError,
     SynapseError,
 )
 from synapse.module_api import ModuleApi
@@ -39,7 +27,15 @@ from synapse.types import UserID
 from synapse.util.caches.expiringcache import ExpiringCache
 from synapse.util.logcontext import make_deferred_yieldable
 
-from ._base import BaseHandler
+from twisted.web.client import PartialDownloadError
+
+import logging
+import bcrypt
+import pymacaroons
+import simplejson
+
+import synapse.util.stringutils as stringutils
+
 
 logger = logging.getLogger(__name__)
 
@@ -406,7 +402,7 @@ class AuthHandler(BaseHandler):
         except PartialDownloadError as pde:
             # Twisted is silly
             data = pde.response
-            resp_body = json.loads(data)
+            resp_body = simplejson.loads(data)
 
         if 'success' in resp_body:
             # Note that we do NOT check the hostname here: we explicitly
@@ -858,11 +854,7 @@ class AuthHandler(BaseHandler):
             return bcrypt.hashpw(password.encode('utf8') + self.hs.config.password_pepper,
                                  bcrypt.gensalt(self.bcrypt_rounds))
 
-        return make_deferred_yieldable(
-            threads.deferToThreadPool(
-                self.hs.get_reactor(), self.hs.get_reactor().getThreadPool(), _do_hash
-            ),
-        )
+        return make_deferred_yieldable(threads.deferToThread(_do_hash))
 
     def validate_hash(self, password, stored_hash):
         """Validates that self.hash(password) == stored_hash.
@@ -882,21 +874,16 @@ class AuthHandler(BaseHandler):
             )
 
         if stored_hash:
-            return make_deferred_yieldable(
-                threads.deferToThreadPool(
-                    self.hs.get_reactor(),
-                    self.hs.get_reactor().getThreadPool(),
-                    _do_validate_hash,
-                ),
-            )
+            return make_deferred_yieldable(threads.deferToThread(_do_validate_hash))
         else:
             return defer.succeed(False)
 
 
-@attr.s
-class MacaroonGenerator(object):
-
-    hs = attr.ib()
+class MacaroonGeneartor(object):
+    def __init__(self, hs):
+        self.clock = hs.get_clock()
+        self.server_name = hs.config.server_name
+        self.macaroon_secret_key = hs.config.macaroon_secret_key
 
     def generate_access_token(self, user_id, extra_caveats=None):
         extra_caveats = extra_caveats or []
@@ -914,7 +901,7 @@ class MacaroonGenerator(object):
     def generate_short_term_login_token(self, user_id, duration_in_ms=(2 * 60 * 1000)):
         macaroon = self._generate_base_macaroon(user_id)
         macaroon.add_first_party_caveat("type = login")
-        now = self.hs.get_clock().time_msec()
+        now = self.clock.time_msec()
         expiry = now + duration_in_ms
         macaroon.add_first_party_caveat("time < %d" % (expiry,))
         return macaroon.serialize()
@@ -926,9 +913,9 @@ class MacaroonGenerator(object):
 
     def _generate_base_macaroon(self, user_id):
         macaroon = pymacaroons.Macaroon(
-            location=self.hs.config.server_name,
+            location=self.server_name,
             identifier="key",
-            key=self.hs.config.macaroon_secret_key)
+            key=self.macaroon_secret_key)
         macaroon.add_first_party_caveat("gen = 1")
         macaroon.add_first_party_caveat("user_id = %s" % (user_id,))
         return macaroon

synapse/handlers/deactivate_account.py

@@ -12,15 +12,14 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
-
-from twisted.internet import defer
-
-from synapse.api.errors import SynapseError
-from synapse.types import UserID, create_requester
-from synapse.util.logcontext import run_in_background
+from twisted.internet import defer, reactor
 
 from ._base import BaseHandler
+from synapse.types import UserID, create_requester
+from synapse.util.logcontext import run_in_background
+from synapse.api.errors import SynapseError
+
+import logging
 
 logger = logging.getLogger(__name__)
 
@@ -40,15 +39,14 @@ class DeactivateAccountHandler(BaseHandler):
 
         # Start the user parter loop so it can resume parting users from rooms where
         # it left off (if it has work left to do).
-        hs.get_reactor().callWhenRunning(self._start_user_parting)
+        reactor.callWhenRunning(self._start_user_parting)
 
     @defer.inlineCallbacks
-    def deactivate_account(self, user_id, erase_data):
+    def deactivate_account(self, user_id):
         """Deactivate a user's account
 
         Args:
             user_id (str): ID of user to be deactivated
-            erase_data (bool): whether to GDPR-erase the user's data
 
         Returns:
             Deferred
@@ -94,11 +92,6 @@ class DeactivateAccountHandler(BaseHandler):
         # delete from user directory
         yield self.user_directory_handler.handle_user_deactivated(user_id)
 
-        # Mark the user as erased, if they asked for that
-        if erase_data:
-            logger.info("Marking %s as erased", user_id)
-            yield self.store.mark_user_erased(user_id)
-
         # Now start the process that goes through that list and
         # parts users from rooms (if it isn't already running)
         self._start_user_parting()

synapse/handlers/device.py

@@ -12,24 +12,22 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
-
-from six import iteritems, itervalues
-
-from twisted.internet import defer
-
 from synapse.api import errors
 from synapse.api.constants import EventTypes
 from synapse.api.errors import FederationDeniedError
-from synapse.types import RoomStreamToken, get_domain_from_id
 from synapse.util import stringutils
 from synapse.util.async import Linearizer
 from synapse.util.caches.expiringcache import ExpiringCache
-from synapse.util.metrics import measure_func
 from synapse.util.retryutils import NotRetryingDestination
-
+from synapse.util.metrics import measure_func
+from synapse.types import get_domain_from_id, RoomStreamToken
+from twisted.internet import defer
 from ._base import BaseHandler
 
+import logging
+
+from six import itervalues, iteritems
+
 logger = logging.getLogger(__name__)
 
 
@@ -539,7 +537,7 @@ class DeviceListEduUpdater(object):
                 yield self.device_handler.notify_device_update(user_id, device_ids)
             else:
                 # Simply update the single device, since we know that is the only
-                # change (because of the single prev_id matching the current cache)
+                # change (becuase of the single prev_id matching the current cache)
                 for device_id, stream_id, prev_ids, content in pending_updates:
                     yield self.store.update_remote_device_list_cache_entry(
                         user_id, device_id, content, stream_id,

synapse/handlers/devicemessage.py

@@ -18,9 +18,10 @@ import logging
 from twisted.internet import defer
 
 from synapse.api.errors import SynapseError
-from synapse.types import UserID, get_domain_from_id
+from synapse.types import get_domain_from_id, UserID
 from synapse.util.stringutils import random_string
 
+
 logger = logging.getLogger(__name__)
 
 

synapse/handlers/directory.py

@@ -14,16 +14,15 @@
 # limitations under the License.
 
 
-import logging
-import string
-
 from twisted.internet import defer
+from ._base import BaseHandler
 
+from synapse.api.errors import SynapseError, Codes, CodeMessageException, AuthError
 from synapse.api.constants import EventTypes
-from synapse.api.errors import AuthError, CodeMessageException, Codes, SynapseError
 from synapse.types import RoomAlias, UserID, get_domain_from_id
 
-from ._base import BaseHandler
+import logging
+import string
 
 logger = logging.getLogger(__name__)
 

synapse/handlers/e2e_keys.py

@@ -14,16 +14,17 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import simplejson as json
 import logging
 
+from canonicaljson import encode_canonical_json
+from twisted.internet import defer
 from six import iteritems
 
-from canonicaljson import encode_canonical_json, json
-
-from twisted.internet import defer
-
-from synapse.api.errors import CodeMessageException, FederationDeniedError, SynapseError
-from synapse.types import UserID, get_domain_from_id
+from synapse.api.errors import (
+    SynapseError, CodeMessageException, FederationDeniedError,
+)
+from synapse.types import get_domain_from_id, UserID
 from synapse.util.logcontext import make_deferred_yieldable, run_in_background
 from synapse.util.retryutils import NotRetryingDestination
 
@@ -79,7 +80,7 @@ class E2eKeysHandler(object):
             else:
                 remote_queries[user_id] = device_ids
 
-        # First get local devices.
+        # Firt get local devices.
         failures = {}
         results = {}
         if local_query:
@@ -356,7 +357,7 @@ def _exception_to_failure(e):
     # include ConnectionRefused and other errors
     #
     # Note that some Exceptions (notably twisted's ResponseFailed etc) don't
-    # give a string for e.message, which json then fails to serialize.
+    # give a string for e.message, which simplejson then fails to serialize.
     return {
         "status": 503, "message": str(e.message),
     }

synapse/handlers/events.py

@@ -13,18 +13,19 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from twisted.internet import defer
+
+from synapse.util.logutils import log_function
+from synapse.types import UserID
+from synapse.events.utils import serialize_event
+from synapse.api.constants import Membership, EventTypes
+from synapse.events import EventBase
+
+from ._base import BaseHandler
+
 import logging
 import random
 
-from twisted.internet import defer
-
-from synapse.api.constants import EventTypes, Membership
-from synapse.events import EventBase
-from synapse.events.utils import serialize_event
-from synapse.types import UserID
-from synapse.util.logutils import log_function
-
-from ._base import BaseHandler
 
 logger = logging.getLogger(__name__)
 

synapse/handlers/federation.py

@@ -20,42 +20,37 @@ import itertools
 import logging
 import sys
 
-import six
-from six import iteritems
-from six.moves import http_client
-
 from signedjson.key import decode_verify_key_bytes
 from signedjson.sign import verify_signed_json
+import six
+from six.moves import http_client
+from six import iteritems
+from twisted.internet import defer
 from unpaddedbase64 import decode_base64
 
-from twisted.internet import defer
+from ._base import BaseHandler
 
-from synapse.api.constants import EventTypes, Membership, RejectedReason
 from synapse.api.errors import (
-    AuthError,
-    CodeMessageException,
+    AuthError, FederationError, StoreError, CodeMessageException, SynapseError,
     FederationDeniedError,
-    FederationError,
-    StoreError,
-    SynapseError,
 )
-from synapse.crypto.event_signing import (
-    add_hashes_and_signatures,
-    compute_event_signature,
-)
-from synapse.events.utils import prune_event
+from synapse.api.constants import EventTypes, Membership, RejectedReason
 from synapse.events.validator import EventValidator
-from synapse.state import resolve_events_with_factory
-from synapse.types import UserID, get_domain_from_id
-from synapse.util import logcontext, unwrapFirstError
-from synapse.util.async import Linearizer
-from synapse.util.distributor import user_joined_room
-from synapse.util.frozenutils import unfreeze
-from synapse.util.logutils import log_function
+from synapse.util import unwrapFirstError, logcontext
 from synapse.util.metrics import measure_func
+from synapse.util.logutils import log_function
+from synapse.util.async import Linearizer
+from synapse.util.frozenutils import unfreeze
+from synapse.crypto.event_signing import (
+    compute_event_signature, add_hashes_and_signatures,
+)
+from synapse.types import UserID, get_domain_from_id
+
+from synapse.events.utils import prune_event
+
 from synapse.util.retryutils import NotRetryingDestination
 
-from ._base import BaseHandler
+from synapse.util.distributor import user_joined_room
 
 logger = logging.getLogger(__name__)
 
@@ -94,9 +89,7 @@ class FederationHandler(BaseHandler):
 
     @defer.inlineCallbacks
     @log_function
-    def on_receive_pdu(
-            self, origin, pdu, get_missing=True, sent_to_us_directly=False,
-    ):
+    def on_receive_pdu(self, origin, pdu, get_missing=True):
         """ Process a PDU received via a federation /send/ transaction, or
         via backfill of missing prev_events
 
@@ -110,10 +103,8 @@ class FederationHandler(BaseHandler):
         """
 
         # We reprocess pdus when we have seen them only as outliers
-        existing = yield self.store.get_event(
-            pdu.event_id,
-            allow_none=True,
-            allow_rejected=True,
+        existing = yield self.get_persisted_pdu(
+            origin, pdu.event_id, do_auth=False
         )
 
         # FIXME: Currently we fetch an event again when we already have it
@@ -170,11 +161,14 @@ class FederationHandler(BaseHandler):
                     "Ignoring PDU %s for room %s from %s as we've left the room!",
                     pdu.event_id, pdu.room_id, origin,
                 )
-                defer.returnValue(None)
+                return
 
         state = None
+
         auth_chain = []
 
+        fetch_state = False
+
         # Get missing pdus if necessary.
         if not pdu.internal_metadata.is_outlier():
             # We only backfill backwards to the min depth.
@@ -229,60 +223,26 @@ class FederationHandler(BaseHandler):
                         list(prevs - seen)[:5],
                     )
 
-            if sent_to_us_directly and prevs - seen:
-                # If they have sent it to us directly, and the server
-                # isn't telling us about the auth events that it's
-                # made a message referencing, we explode
-                raise FederationError(
-                    "ERROR",
-                    403,
-                    (
-                        "Your server isn't divulging details about prev_events "
-                        "referenced in this event."
-                    ),
-                    affected=pdu.event_id,
+            if prevs - seen:
+                logger.info(
+                    "Still missing %d events for room %r: %r...",
+                    len(prevs - seen), pdu.room_id, list(prevs - seen)[:5]
                 )
-            elif prevs - seen:
-                # Calculate the state of the previous events, and
-                # de-conflict them to find the current state.
-                state_groups = []
-                auth_chains = set()
-                try:
-                    # Get the state of the events we know about
-                    ours = yield self.store.get_state_groups(pdu.room_id, list(seen))
-                    state_groups.append(ours)
+                fetch_state = True
 
-                    # Ask the remote server for the states we don't
-                    # know about
-                    for p in prevs - seen:
-                        state, got_auth_chain = (
-                            yield self.replication_layer.get_state_for_room(
-                                origin, pdu.room_id, p
-                            )
-                        )
-                        auth_chains.update(got_auth_chain)
-                        state_group = {(x.type, x.state_key): x.event_id for x in state}
-                        state_groups.append(state_group)
-
-                    # Resolve any conflicting state
-                    def fetch(ev_ids):
-                        return self.store.get_events(
-                            ev_ids, get_prev_content=False, check_redacted=False
-                        )
-
-                    state_map = yield resolve_events_with_factory(
-                        state_groups, {pdu.event_id: pdu}, fetch
-                    )
-
-                    state = (yield self.store.get_events(state_map.values())).values()
-                    auth_chain = list(auth_chains)
-                except Exception:
-                    raise FederationError(
-                        "ERROR",
-                        403,
-                        "We can't get valid state history.",
-                        affected=pdu.event_id,
-                    )
+        if fetch_state:
+            # We need to get the state at this event, since we haven't
+            # processed all the prev events.
+            logger.debug(
+                "_handle_new_pdu getting state for %s",
+                pdu.room_id
+            )
+            try:
+                state, auth_chain = yield self.replication_layer.get_state_for_room(
+                    origin, pdu.room_id, pdu.event_id,
+                )
+            except Exception:
+                logger.exception("Failed to get state for event: %s", pdu.event_id)
 
         yield self._process_received_pdu(
             origin,
@@ -360,17 +320,11 @@ class FederationHandler(BaseHandler):
 
         for e in missing_events:
             logger.info("Handling found event %s", e.event_id)
-            try:
-                yield self.on_receive_pdu(
-                    origin,
-                    e,
-                    get_missing=False
-                )
-            except FederationError as e:
-                if e.code == 403:
-                    logger.warn("Event %s failed history check.")
-                else:
-                    raise
+            yield self.on_receive_pdu(
+                origin,
+                e,
+                get_missing=False
+            )
 
     @log_function
     @defer.inlineCallbacks
@@ -504,47 +458,6 @@ class FederationHandler(BaseHandler):
     @measure_func("_filter_events_for_server")
     @defer.inlineCallbacks
     def _filter_events_for_server(self, server_name, room_id, events):
-        """Filter the given events for the given server, redacting those the
-        server can't see.
-
-        Assumes the server is currently in the room.
-
-        Returns
-            list[FrozenEvent]
-        """
-        # First lets check to see if all the events have a history visibility
-        # of "shared" or "world_readable". If thats the case then we don't
-        # need to check membership (as we know the server is in the room).
-        event_to_state_ids = yield self.store.get_state_ids_for_events(
-            frozenset(e.event_id for e in events),
-            types=(
-                (EventTypes.RoomHistoryVisibility, ""),
-            )
-        )
-
-        visibility_ids = set()
-        for sids in event_to_state_ids.itervalues():
-            hist = sids.get((EventTypes.RoomHistoryVisibility, ""))
-            if hist:
-                visibility_ids.add(hist)
-
-        # If we failed to find any history visibility events then the default
-        # is "shared" visiblity.
-        if not visibility_ids:
-            defer.returnValue(events)
-
-        event_map = yield self.store.get_events(visibility_ids)
-        all_open = all(
-            e.content.get("history_visibility") in (None, "shared", "world_readable")
-            for e in event_map.itervalues()
-        )
-
-        if all_open:
-            defer.returnValue(events)
-
-        # Ok, so we're dealing with events that have non-trivial visibility
-        # rules, so we need to also get the memberships of the room.
-
         event_to_state_ids = yield self.store.get_state_ids_for_events(
             frozenset(e.event_id for e in events),
             types=(
@@ -580,20 +493,7 @@ class FederationHandler(BaseHandler):
             for e_id, key_to_eid in event_to_state_ids.iteritems()
         }
 
-        erased_senders = yield self.store.are_users_erased(
-            e.sender for e in events,
-        )
-
         def redact_disallowed(event, state):
-            # if the sender has been gdpr17ed, always return a redacted
-            # copy of the event.
-            if erased_senders[event.sender]:
-                logger.info(
-                    "Sender of %s has been erased, redacting",
-                    event.event_id,
-                )
-                return prune_event(event)
-
             if not state:
                 return event
 
@@ -1564,20 +1464,11 @@ class FederationHandler(BaseHandler):
 
     @defer.inlineCallbacks
     @log_function
-    def get_persisted_pdu(self, origin, event_id):
-        """Get an event from the database for the given server.
-
-        Args:
-            origin [str]: hostname of server which is requesting the event; we
-               will check that the server is allowed to see it.
-            event_id [str]: id of the event being requested
+    def get_persisted_pdu(self, origin, event_id, do_auth=True):
+        """ Get a PDU from the database with given origin and id.
 
         Returns:
-            Deferred[EventBase|None]: None if we know nothing about the event;
-                otherwise the (possibly-redacted) event.
-
-        Raises:
-            AuthError if the server is not currently in the room
+            Deferred: Results in a `Pdu`.
         """
         event = yield self.store.get_event(
             event_id,
@@ -1598,17 +1489,20 @@ class FederationHandler(BaseHandler):
                     )
                 )
 
-            in_room = yield self.auth.check_host_in_room(
-                event.room_id,
-                origin
-            )
-            if not in_room:
-                raise AuthError(403, "Host not in room.")
+            if do_auth:
+                in_room = yield self.auth.check_host_in_room(
+                    event.room_id,
+                    origin
+                )
+                if not in_room:
+                    raise AuthError(403, "Host not in room.")
+
+                events = yield self._filter_events_for_server(
+                    origin, event.room_id, [event]
+                )
+
+                event = events[0]
 
-            events = yield self._filter_events_for_server(
-                origin, event.room_id, [event]
-            )
-            event = events[0]
             defer.returnValue(event)
         else:
             defer.returnValue(None)

synapse/handlers/groups_local.py

@@ -14,15 +14,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import logging
-
-from six import iteritems
-
 from twisted.internet import defer
+from six import iteritems
 
 from synapse.api.errors import SynapseError
 from synapse.types import get_domain_from_id
 
+import logging
+
 logger = logging.getLogger(__name__)
 
 

synapse/handlers/identity.py

@@ -19,18 +19,15 @@
 
 import logging
 
-from canonicaljson import json
+import simplejson as json
 
 from twisted.internet import defer
 
 from synapse.api.errors import (
-    CodeMessageException,
-    Codes,
-    MatrixCodeMessageException,
-    SynapseError,
+    MatrixCodeMessageException, CodeMessageException
 )
-
 from ._base import BaseHandler
+from synapse.api.errors import SynapseError, Codes
 
 logger = logging.getLogger(__name__)
 

synapse/handlers/initial_sync.py

@@ -13,8 +13,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import logging
-
 from twisted.internet import defer
 
 from synapse.api.constants import EventTypes, Membership
@@ -23,7 +21,9 @@ from synapse.events.utils import serialize_event
 from synapse.events.validator import EventValidator
 from synapse.handlers.presence import format_user_presence_state
 from synapse.streams.config import PaginationConfig
-from synapse.types import StreamToken, UserID
+from synapse.types import (
+    UserID, StreamToken,
+)
 from synapse.util import unwrapFirstError
 from synapse.util.async import concurrently_execute
 from synapse.util.caches.snapshot_cache import SnapshotCache
@@ -32,6 +32,9 @@ from synapse.visibility import filter_events_for_client
 
 from ._base import BaseHandler
 
+import logging
+
+
 logger = logging.getLogger(__name__)
 
 

synapse/handlers/message.py

@@ -14,31 +14,35 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import logging
+import simplejson
 import sys
 
+from canonicaljson import encode_canonical_json
 import six
-from six import iteritems, itervalues, string_types
-
-from canonicaljson import encode_canonical_json, json
-
-from twisted.internet import defer
+from six import string_types, itervalues, iteritems
+from twisted.internet import defer, reactor
 from twisted.internet.defer import succeed
 from twisted.python.failure import Failure
 
-from synapse.api.constants import MAX_DEPTH, EventTypes, Membership
-from synapse.api.errors import AuthError, Codes, ConsentNotGivenError, SynapseError
+from synapse.api.constants import EventTypes, Membership, MAX_DEPTH
+from synapse.api.errors import (
+    AuthError, Codes, SynapseError,
+    ConsentNotGivenError,
+)
 from synapse.api.urls import ConsentURIBuilder
 from synapse.crypto.event_signing import add_hashes_and_signatures
 from synapse.events.utils import serialize_event
 from synapse.events.validator import EventValidator
-from synapse.replication.http.send_event import send_event_to_master
-from synapse.types import RoomAlias, RoomStreamToken, UserID
-from synapse.util.async import Limiter, ReadWriteLock
-from synapse.util.frozenutils import frozendict_json_encoder
+from synapse.types import (
+    UserID, RoomAlias, RoomStreamToken,
+)
+from synapse.util.async import ReadWriteLock, Limiter
 from synapse.util.logcontext import run_in_background
 from synapse.util.metrics import measure_func
+from synapse.util.frozenutils import frozendict_json_encoder
 from synapse.util.stringutils import random_string
 from synapse.visibility import filter_events_for_client
+from synapse.replication.http.send_event import send_event_to_master
 
 from ._base import BaseHandler
 
@@ -153,7 +157,7 @@ class MessageHandler(BaseHandler):
             # remove the purge from the list 24 hours after it completes
             def clear_purge():
                 del self._purges_by_id[purge_id]
-            self.hs.get_reactor().callLater(24 * 3600, clear_purge)
+            reactor.callLater(24 * 3600, clear_purge)
 
     def get_purge_status(self, purge_id):
         """Get the current status of an active purge
@@ -384,7 +388,7 @@ class MessageHandler(BaseHandler):
         users_with_profile = yield self.state.get_current_user_in_room(room_id)
 
         # If this is an AS, double check that they are allowed to see the members.
-        # This can either be because the AS user is in the room or because there
+        # This can either be because the AS user is in the room or becuase there
         # is a user in the room that the AS is "interested in"
         if requester.app_service and user_id not in users_with_profile:
             for uid in users_with_profile:
@@ -487,7 +491,7 @@ class EventCreationHandler(object):
                         target, e
                     )
 
-        is_exempt = yield self._is_exempt_from_privacy_policy(builder, requester)
+        is_exempt = yield self._is_exempt_from_privacy_policy(builder)
         if not is_exempt:
             yield self.assert_accepted_privacy_policy(requester)
 
@@ -505,13 +509,12 @@ class EventCreationHandler(object):
 
         defer.returnValue((event, context))
 
-    def _is_exempt_from_privacy_policy(self, builder, requester):
+    def _is_exempt_from_privacy_policy(self, builder):
         """"Determine if an event to be sent is exempt from having to consent
         to the privacy policy
 
         Args:
             builder (synapse.events.builder.EventBuilder): event being created
-            requester (Requster): user requesting this event
 
         Returns:
             Deferred[bool]: true if the event can be sent without the user
@@ -522,9 +525,6 @@ class EventCreationHandler(object):
             membership = builder.content.get("membership", None)
             if membership == Membership.JOIN:
                 return self._is_server_notices_room(builder.room_id)
-            elif membership == Membership.LEAVE:
-                # the user is always allowed to leave (but not kick people)
-                return builder.state_key == requester.user.to_string()
         return succeed(False)
 
     @defer.inlineCallbacks
@@ -793,7 +793,7 @@ class EventCreationHandler(object):
         # Ensure that we can round trip before trying to persist in db
         try:
             dump = frozendict_json_encoder.encode(event.content)
-            json.loads(dump)
+            simplejson.loads(dump)
         except Exception:
             logger.exception("Failed to encode content: %r", event.content)
             raise
@@ -806,7 +806,6 @@ class EventCreationHandler(object):
             # If we're a worker we need to hit out to the master.
             if self.config.worker_app:
                 yield send_event_to_master(
-                    self.hs.get_clock(),
                     self.http_client,
                     host=self.config.worker_replication_host,
                     port=self.config.worker_replication_http_port,

synapse/handlers/presence.py

@@ -22,26 +22,27 @@ The methods that define policy are:
     - should_notify
 """
 
-import logging
+from twisted.internet import defer, reactor
 from contextlib import contextmanager
 
-from six import iteritems, itervalues
+from six import itervalues, iteritems
 
-from prometheus_client import Counter
-
-from twisted.internet import defer
-
-from synapse.api.constants import PresenceState
 from synapse.api.errors import SynapseError
-from synapse.metrics import LaterGauge
+from synapse.api.constants import PresenceState
 from synapse.storage.presence import UserPresenceState
-from synapse.types import UserID, get_domain_from_id
-from synapse.util.async import Linearizer
+
 from synapse.util.caches.descriptors import cachedInlineCallbacks
+from synapse.util.async import Linearizer
 from synapse.util.logcontext import run_in_background
 from synapse.util.logutils import log_function
 from synapse.util.metrics import Measure
 from synapse.util.wheel_timer import WheelTimer
+from synapse.types import UserID, get_domain_from_id
+from synapse.metrics import LaterGauge
+
+import logging
+
+from prometheus_client import Counter
 
 logger = logging.getLogger(__name__)
 
@@ -178,7 +179,7 @@ class PresenceHandler(object):
         # have not yet been persisted
         self.unpersisted_users_changes = set()
 
-        hs.get_reactor().addSystemEventTrigger("before", "shutdown", self._on_shutdown)
+        reactor.addSystemEventTrigger("before", "shutdown", self._on_shutdown)
 
         self.serial_to_user = {}
         self._next_serial = 1

synapse/handlers/profile.py

@@ -17,9 +17,8 @@ import logging
 
 from twisted.internet import defer
 
-from synapse.api.errors import AuthError, CodeMessageException, SynapseError
+from synapse.api.errors import SynapseError, AuthError, CodeMessageException
 from synapse.types import UserID, get_domain_from_id
-
 from ._base import BaseHandler
 
 logger = logging.getLogger(__name__)

synapse/handlers/read_marker.py

@@ -13,14 +13,13 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import logging
+from ._base import BaseHandler
 
 from twisted.internet import defer
 
 from synapse.util.async import Linearizer
 
-from ._base import BaseHandler
-
+import logging
 logger = logging.getLogger(__name__)
 
 

synapse/handlers/receipts.py

@@ -12,15 +12,17 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
+from synapse.util import logcontext
+
+from ._base import BaseHandler
 
 from twisted.internet import defer
 
-from synapse.types import get_domain_from_id
-from synapse.util import logcontext
 from synapse.util.logcontext import PreserveLoggingContext
+from synapse.types import get_domain_from_id
+
+import logging
 
-from ._base import BaseHandler
 
 logger = logging.getLogger(__name__)
 

synapse/handlers/register.py

@@ -18,19 +18,14 @@ import logging
 
 from twisted.internet import defer
 
-from synapse import types
 from synapse.api.errors import (
-    AuthError,
-    Codes,
-    InvalidCaptchaError,
-    RegistrationError,
-    SynapseError,
+    AuthError, Codes, SynapseError, RegistrationError, InvalidCaptchaError
 )
 from synapse.http.client import CaptchaServerHttpClient
-from synapse.types import RoomAlias, RoomID, UserID, create_requester
+from synapse import types
+from synapse.types import UserID, create_requester, RoomID, RoomAlias
 from synapse.util.async import Linearizer
 from synapse.util.threepids import check_3pid_allowed
-
 from ._base import BaseHandler
 
 logger = logging.getLogger(__name__)