Merge branch 'develop' of github.com:matrix-org/synapse into matrix-org-hotfixes

v0.18.6-rc3

.github/ISSUE_TEMPLATE.md

@@ -1,47 +0,0 @@
-<!-- 
-
-**IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER**: 
-You will likely get better support more quickly if you ask in ** #matrix:matrix.org ** ;)
-
-
-This is a bug report template. By following the instructions below and
-filling out the sections with your information, you will help the us to get all
-the necessary data to fix your issue.
-
-You can also preview your report before submitting it. You may remove sections
-that aren't relevant to your particular case.
-
-Text between <!-- and --​> marks will be invisible in the report.
-
--->
-
-### Description
-
-Describe here the problem that you are experiencing, or the feature you are requesting.
-
-### Steps to reproduce
-
-- For bugs, list the steps
-- that reproduce the bug
-- using hyphens as bullet points
-
-Describe how what happens differs from what you expected.
-
-If you can identify any relevant log snippets from _homeserver.log_, please include
-those here (please be careful to remove any personal or private data):
-
-### Version information
-
-<!-- IMPORTANT: please answer the following questions, to help us narrow down the problem -->
-
-- **Homeserver**: Was this issue identified on matrix.org or another homeserver?
-
-If not matrix.org:
-- **Version**:        What version of Synapse is running? <!-- 
-You can find the Synapse version by inspecting the server headers (replace matrix.org with
-your own homeserver domain):
-$ curl -v https://matrix.org/_matrix/client/versions 2>&1 | grep "Server:"
--->
-- **Install method**: package manager/git clone/pip      
-- **Platform**:       Tell us about the environment in which your homeserver is operating
-                      - distro, hardware, if it's running in a vm/container, etc.

CHANGES.rst

@@ -1,431 +1,3 @@
-Changes in synapse v0.25.0 (2017-11-15)
-=======================================
-
-Bug fixes:
-
-* Fix port script (PR #2673)
-
-
-Changes in synapse v0.25.0-rc1 (2017-11-14)
-===========================================
-
-Features:
-
-* Add is_public to groups table to allow for private groups (PR #2582)
-* Add a route for determining who you are (PR #2668) Thanks to @turt2live!
-* Add more features to the password providers (PR #2608, #2610, #2620, #2622,
-  #2623, #2624, #2626, #2628, #2629)
-* Add a hook for custom rest endpoints (PR #2627)
-* Add API to update group room visibility (PR #2651)
-
-
-Changes:
-
-* Ignore <noscript> tags when generating URL preview descriptions (PR #2576)
-  Thanks to @maximevaillancourt!
-* Register some /unstable endpoints in /r0 as well (PR #2579) Thanks to
-  @krombel!
-* Support /keys/upload on /r0 as well as /unstable (PR #2585)
-* Front-end proxy: pass through auth header (PR #2586)
-* Allow ASes to deactivate their own users (PR #2589)
-* Remove refresh tokens (PR #2613)
-* Automatically set default displayname on register (PR #2617)
-* Log login requests (PR #2618)
-* Always return `is_public` in the `/groups/:group_id/rooms` API (PR #2630)
-* Avoid no-op media deletes (PR #2637) Thanks to @spantaleev!
-* Fix various embarrassing typos around user_directory and add some doc. (PR
-  #2643)
-* Return whether a user is an admin within a group (PR #2647)
-* Namespace visibility options for groups (PR #2657)
-* Downcase UserIDs on registration (PR #2662)
-* Cache failures when fetching URL previews (PR #2669)
-
-
-Bug fixes:
-
-* Fix port script (PR #2577)
-* Fix error when running synapse with no logfile (PR #2581)
-* Fix UI auth when deleting devices (PR #2591)
-* Fix typo when checking if user is invited to group (PR #2599)
-* Fix the port script to drop NUL values in all tables (PR #2611)
-* Fix appservices being backlogged and not receiving new events due to a bug in
-  notify_interested_services (PR #2631) Thanks to @xyzz!
-* Fix updating rooms avatar/display name when modified by admin (PR #2636)
-  Thanks to @farialima!
-* Fix bug in state group storage (PR #2649)
-* Fix 500 on invalid utf-8 in request (PR #2663)
-
-
-Changes in synapse v0.24.1 (2017-10-24)
-=======================================
-
-Bug fixes:
-
-* Fix updating group profiles over federation (PR #2567)
-
-
-Changes in synapse v0.24.0 (2017-10-23)
-=======================================
-
-No changes since v0.24.0-rc1
-
-
-Changes in synapse v0.24.0-rc1 (2017-10-19)
-===========================================
-
-Features:
-
-* Add Group Server (PR #2352, #2363, #2374, #2377, #2378, #2382, #2410, #2426,
-  #2430, #2454, #2471, #2472, #2544)
-* Add support for channel notifications (PR #2501)
-* Add basic implementation of backup media store (PR #2538)
-* Add config option to auto-join new users to rooms (PR #2545)
-
-
-Changes:
-
-* Make the spam checker a module (PR #2474)
-* Delete expired url cache data (PR #2478)
-* Ignore incoming events for rooms that we have left (PR #2490)
-* Allow spam checker to reject invites too (PR #2492)
-* Add room creation checks to spam checker (PR #2495)
-* Spam checking: add the invitee to user_may_invite (PR #2502)
-* Process events from federation for different rooms in parallel (PR #2520)
-* Allow error strings from spam checker (PR #2531)
-* Improve error handling for missing files in config (PR #2551)
-
-
-Bug fixes:
-
-* Fix handling SERVFAILs when doing AAAA lookups for federation (PR #2477)
-* Fix incompatibility with newer versions of ujson (PR #2483) Thanks to
-  @jeremycline!
-* Fix notification keywords that start/end with non-word chars (PR #2500)
-* Fix stack overflow and logcontexts from linearizer (PR #2532)
-* Fix 500 error when fields missing from power_levels event (PR #2552)
-* Fix 500 error when we get an error handling a PDU (PR #2553)
-
-
-Changes in synapse v0.23.1 (2017-10-02)
-=======================================
-
-Changes:
-
-* Make 'affinity' package optional, as it is not supported on some platforms
-
-
-Changes in synapse v0.23.0 (2017-10-02)
-=======================================
-
-No changes since v0.23.0-rc2
-
-
-Changes in synapse v0.23.0-rc2 (2017-09-26)
-===========================================
-
-Bug fixes:
-
-* Fix regression in performance of syncs (PR #2470)
-
-
-Changes in synapse v0.23.0-rc1 (2017-09-25)
-===========================================
-
-Features:
-
-* Add a frontend proxy worker (PR #2344)
-* Add support for event_id_only push format (PR #2450)
-* Add a PoC for filtering spammy events (PR #2456)
-* Add a config option to block all room invites (PR #2457)
-
-
-Changes:
-
-* Use bcrypt module instead of py-bcrypt (PR #2288) Thanks to @kyrias!
-* Improve performance of generating push notifications (PR #2343, #2357, #2365,
-  #2366, #2371)
-* Improve DB performance for device list handling in sync (PR #2362)
-* Include a sample prometheus config (PR #2416)
-* Document known to work postgres version (PR #2433) Thanks to @ptman!
-
-
-Bug fixes:
-
-* Fix caching error in the push evaluator (PR #2332)
-* Fix bug where pusherpool didn't start and broke some rooms (PR #2342)
-* Fix port script for user directory tables (PR #2375)
-* Fix device lists notifications when user rejoins a room (PR #2443, #2449)
-* Fix sync to always send down current state events in timeline (PR #2451)
-* Fix bug where guest users were incorrectly kicked (PR #2453)
-* Fix bug talking to IPv6 only servers using SRV records (PR #2462)
-
-
-Changes in synapse v0.22.1 (2017-07-06)
-=======================================
-
-Bug fixes:
-
-* Fix bug where pusher pool didn't start and caused issues when
-  interacting with some rooms (PR #2342)
-
-
-Changes in synapse v0.22.0 (2017-07-06)
-=======================================
-
-No changes since v0.22.0-rc2
-
-
-Changes in synapse v0.22.0-rc2 (2017-07-04)
-===========================================
-
-Changes:
-
-* Improve performance of storing user IPs (PR #2307, #2308)
-* Slightly improve performance of verifying access tokens (PR #2320)
-* Slightly improve performance of event persistence (PR #2321)
-* Increase default cache factor size from 0.1 to 0.5 (PR #2330)
-
-Bug fixes:
-
-* Fix bug with storing registration sessions that caused frequent CPU churn
-  (PR #2319)
-
-
-Changes in synapse v0.22.0-rc1 (2017-06-26)
-===========================================
-
-Features:
-
-* Add a user directory API (PR #2252, and many more)
-* Add shutdown room API to remove room from local server (PR #2291)
-* Add API to quarantine media (PR #2292)
-* Add new config option to not send event contents to push servers (PR #2301)
-  Thanks to @cjdelisle!
-
-Changes:
-
-* Various performance fixes (PR #2177, #2233, #2230, #2238, #2248, #2256,
-  #2274)
-* Deduplicate sync filters (PR #2219) Thanks to @krombel!
-* Correct a typo in UPGRADE.rst (PR #2231) Thanks to @aaronraimist!
-* Add count of one time keys to sync stream (PR #2237)
-* Only store event_auth for state events (PR #2247)
-* Store URL cache preview downloads separately (PR #2299)
-
-Bug fixes:
-
-* Fix users not getting notifications when AS listened to that user_id (PR
-  #2216) Thanks to @slipeer!
-* Fix users without push set up not getting notifications after joining rooms
-  (PR #2236)
-* Fix preview url API to trim long descriptions (PR #2243)
-* Fix bug where we used cached but unpersisted state group as prev group,
-  resulting in broken state of restart (PR #2263)
-* Fix removing of pushers when using workers (PR #2267)
-* Fix CORS headers to allow Authorization header (PR #2285) Thanks to @krombel!
-
-
-Changes in synapse v0.21.1 (2017-06-15)
-=======================================
-
-Bug fixes:
-
-* Fix bug in anonymous usage statistic reporting (PR #2281)
-
-
-Changes in synapse v0.21.0 (2017-05-18)
-=======================================
-
-No changes since v0.21.0-rc3
-
-
-Changes in synapse v0.21.0-rc3 (2017-05-17)
-===========================================
-
-Features:
-
-* Add per user rate-limiting overrides (PR #2208)
-* Add config option to limit maximum number of events requested by ``/sync``
-  and ``/messages`` (PR #2221) Thanks to @psaavedra!
-
-
-Changes:
-
-* Various small performance fixes (PR #2201, #2202, #2224, #2226, #2227, #2228,
-  #2229)
-* Update username availability checker API (PR #2209, #2213)
-* When purging, don't de-delta state groups we're about to delete (PR #2214)
-* Documentation to check synapse version (PR #2215) Thanks to @hamber-dick!
-* Add an index to event_search to speed up purge history API (PR #2218)
-
-
-Bug fixes:
-
-* Fix API to allow clients to upload one-time-keys with new sigs (PR #2206)
-
-
-Changes in synapse v0.21.0-rc2 (2017-05-08)
-===========================================
-
-Changes:
-
-* Always mark remotes as up if we receive a signed request from them (PR #2190)
-
-
-Bug fixes:
-
-* Fix bug where users got pushed for rooms they had muted (PR #2200)
-
-
-Changes in synapse v0.21.0-rc1 (2017-05-08)
-===========================================
-
-Features:
-
-* Add username availability checker API (PR #2183)
-* Add read marker API (PR #2120)
-
-
-Changes:
-
-* Enable guest access for the 3pl/3pid APIs (PR #1986)
-* Add setting to support TURN for guests (PR #2011)
-* Various performance improvements (PR #2075, #2076, #2080, #2083, #2108,
-  #2158, #2176, #2185)
-* Make synctl a bit more user friendly (PR #2078, #2127) Thanks @APwhitehat!
-* Replace HTTP replication with TCP replication (PR #2082, #2097, #2098,
-  #2099, #2103, #2014, #2016, #2115, #2116, #2117)
-* Support authenticated SMTP (PR #2102) Thanks @DanielDent!
-* Add a counter metric for successfully-sent transactions (PR #2121)
-* Propagate errors sensibly from proxied IS requests (PR #2147)
-* Add more granular event send metrics (PR #2178)
-
-
-
-Bug fixes:
-
-* Fix nuke-room script to work with current schema (PR #1927) Thanks
-  @zuckschwerdt!
-* Fix db port script to not assume postgres tables are in the public schema
-  (PR #2024) Thanks @jerrykan!
-* Fix getting latest device IP for user with no devices (PR #2118)
-* Fix rejection of invites to unreachable servers (PR #2145)
-* Fix code for reporting old verify keys in synapse (PR #2156)
-* Fix invite state to always include all events (PR #2163)
-* Fix bug where synapse would always fetch state for any missing event (PR #2170)
-* Fix a leak with timed out HTTP connections (PR #2180)
-* Fix bug where we didn't time out HTTP requests to ASes  (PR #2192)
-
-
-Docs:
-
-* Clarify doc for SQLite to PostgreSQL port (PR #1961) Thanks @benhylau!
-* Fix typo in synctl help (PR #2107) Thanks @HarHarLinks!
-* ``web_client_location`` documentation fix (PR #2131) Thanks @matthewjwolff!
-* Update README.rst with FreeBSD changes (PR #2132) Thanks @feld!
-* Clarify setting up metrics (PR #2149) Thanks @encks!
-
-
-Changes in synapse v0.20.0 (2017-04-11)
-=======================================
-
-Bug fixes:
-
-* Fix joining rooms over federation where not all servers in the room saw the
-  new server had joined (PR #2094)
-
-
-Changes in synapse v0.20.0-rc1 (2017-03-30)
-===========================================
-
-Features:
-
-* Add delete_devices API (PR #1993)
-* Add phone number registration/login support (PR #1994, #2055)
-
-
-Changes:
-
-* Use JSONSchema for validation of filters. Thanks @pik! (PR #1783)
-* Reread log config on SIGHUP (PR #1982)
-* Speed up public room list (PR #1989)
-* Add helpful texts to logger config options (PR #1990)
-* Minor ``/sync`` performance improvements. (PR #2002, #2013, #2022)
-* Add some debug to help diagnose weird federation issue (PR #2035)
-* Correctly limit retries for all federation requests (PR #2050, #2061)
-* Don't lock table when persisting new one time keys (PR #2053)
-* Reduce some CPU work on DB threads (PR #2054)
-* Cache hosts in room (PR #2060)
-* Batch sending of device list pokes (PR #2063)
-* Speed up persist event path in certain edge cases (PR #2070)
-
-
-Bug fixes:
-
-* Fix bug where current_state_events renamed to current_state_ids (PR #1849)
-* Fix routing loop when fetching remote media (PR #1992)
-* Fix current_state_events table to not lie (PR #1996)
-* Fix CAS login to handle PartialDownloadError (PR #1997)
-* Fix assertion to stop transaction queue getting wedged (PR #2010)
-* Fix presence to fallback to last_active_ts if it beats the last sync time.
-  Thanks @Half-Shot! (PR #2014)
-* Fix bug when federation received a PDU while a room join is in progress (PR
-  #2016)
-* Fix resetting state on rejected events (PR #2025)
-* Fix installation issues in readme. Thanks @ricco386 (PR #2037)
-* Fix caching of remote servers' signature keys (PR #2042)
-* Fix some leaking log context (PR #2048, #2049, #2057, #2058)
-* Fix rejection of invites not reaching sync (PR #2056)
-
-
-
-Changes in synapse v0.19.3 (2017-03-20)
-=======================================
-
-No changes since v0.19.3-rc2
-
-
-Changes in synapse v0.19.3-rc2 (2017-03-13)
-===========================================
-
-Bug fixes:
-
-* Fix bug in handling of incoming device list updates over federation.
-
-
-
-Changes in synapse v0.19.3-rc1 (2017-03-08)
-===========================================
-
-Features:
-
-* Add some administration functionalities. Thanks to morteza-araby! (PR #1784)
-
-
-Changes:
-
-* Reduce database table sizes (PR #1873, #1916, #1923, #1963)
-* Update contrib/ to not use syutil. Thanks to andrewshadura! (PR #1907)
-* Don't fetch current state when sending an event in common case (PR #1955)
-
-
-Bug fixes:
-
-* Fix synapse_port_db failure. Thanks to Pneumaticat! (PR #1904)
-* Fix caching to not cache error responses (PR #1913)
-* Fix APIs to make kick & ban reasons work (PR #1917)
-* Fix bugs in the /keys/changes api (PR #1921)
-* Fix bug where users couldn't forget rooms they were banned from (PR #1922)
-* Fix issue with long language values in pushers API (PR #1925)
-* Fix a race in transaction queue (PR #1930)
-* Fix dynamic thumbnailing to preserve aspect ratio. Thanks to jkolo! (PR
-  #1945)
-* Fix device list update to not constantly resync (PR #1964)
-* Fix potential for huge memory usage when getting device that have
-  changed (PR #1969)
-
-
-
 Changes in synapse v0.19.2 (2017-02-20)
 =======================================
 

MANIFEST.in

@@ -27,5 +27,4 @@ exclude jenkins*.sh
 exclude jenkins*
 recursive-exclude jenkins *.sh
 
-prune .github
 prune demo/etc

README.rst

@@ -20,7 +20,7 @@ The overall architecture is::
              https://somewhere.org/_matrix      https://elsewhere.net/_matrix
 
 ``#matrix:matrix.org`` is the official support room for Matrix, and can be
-accessed by any client from https://matrix.org/docs/projects/try-matrix-now.html or
+accessed by any client from https://matrix.org/docs/projects/try-matrix-now or
 via IRC bridge at irc://irc.freenode.net/matrix.
 
 Synapse is currently in rapid development, but as of version 0.5 we believe it
@@ -68,7 +68,7 @@ or mandatory service provider in Matrix, unlike WhatsApp, Facebook, Hangouts,
 etc.
 
 We'd like to invite you to join #matrix:matrix.org (via
-https://matrix.org/docs/projects/try-matrix-now.html), run a homeserver, take a look
+https://matrix.org/docs/projects/try-matrix-now), run a homeserver, take a look
 at the `Matrix spec <https://matrix.org/docs/spec>`_, and experiment with the
 `APIs <https://matrix.org/docs/api>`_ and `Client SDKs
 <http://matrix.org/docs/projects/try-matrix-now.html#client-sdks>`_.
@@ -84,7 +84,6 @@ Synapse Installation
 Synapse is the reference python/twisted Matrix homeserver implementation.
 
 System requirements:
-
 - POSIX-compliant system (tested on Linux & OS X)
 - Python 2.7
 - At least 1GB of free RAM if you want to join large public rooms like #matrix:matrix.org
@@ -109,10 +108,10 @@ Installing prerequisites on ArchLinux::
     sudo pacman -S base-devel python2 python-pip \
                    python-setuptools python-virtualenv sqlite3
 
-Installing prerequisites on CentOS 7 or Fedora 25::
+Installing prerequisites on CentOS 7::
 
     sudo yum install libtiff-devel libjpeg-devel libzip-devel freetype-devel \
-                     lcms2-devel libwebp-devel tcl-devel tk-devel redhat-rpm-config \
+                     lcms2-devel libwebp-devel tcl-devel tk-devel \
                      python-virtualenv libffi-devel openssl-devel
     sudo yum groupinstall "Development Tools"
 
@@ -147,7 +146,6 @@ To install the synapse homeserver run::
 
     virtualenv -p python2.7 ~/.synapse
     source ~/.synapse/bin/activate
-    pip install --upgrade pip
     pip install --upgrade setuptools
     pip install https://github.com/matrix-org/synapse/tarball/master
 
@@ -200,11 +198,11 @@ different. See `the spec`__ for more information on key management.)
 .. __: `key_management`_
 
 The default configuration exposes two HTTP ports: 8008 and 8448. Port 8008 is
-configured without TLS; it should be behind a reverse proxy for TLS/SSL
-termination on port 443 which in turn should be used for clients. Port 8448
-is configured to use TLS with a self-signed certificate. If you would like
-to do initial test with a client without having to setup a reverse proxy,
-you can temporarly use another certificate. (Note that a self-signed
+configured without TLS; it is not recommended this be exposed outside your
+local network. Port 8448 is configured to use TLS with a self-signed
+certificate. This is fine for testing with but, to avoid your clients
+complaining about the certificate, you will almost certainly want to use
+another certificate for production purposes. (Note that a self-signed
 certificate is fine for `Federation`_). You can do so by changing
 ``tls_certificate_path``, ``tls_private_key_path`` and ``tls_dh_params_path``
 in ``homeserver.yaml``; alternatively, you can use a reverse-proxy, but be sure
@@ -230,7 +228,6 @@ To get started, it is easiest to use the command line to register new users::
     New user localpart: erikj
     Password:
     Confirm password:
-    Make admin [no]:
     Success!
 
 This process uses a setting ``registration_shared_secret`` in
@@ -246,25 +243,6 @@ Setting up a TURN server
 For reliable VoIP calls to be routed via this homeserver, you MUST configure
 a TURN server.  See `<docs/turn-howto.rst>`_ for details.
 
-IPv6
-----
-
-As of Synapse 0.19 we finally support IPv6, many thanks to @kyrias and @glyph
-for providing PR #1696.
-
-However, for federation to work on hosts with IPv6 DNS servers you **must**
-be running Twisted 17.1.0 or later - see https://github.com/matrix-org/synapse/issues/1002
-for details.  We can't make Synapse depend on Twisted 17.1 by default
-yet as it will break most older distributions (see https://github.com/matrix-org/synapse/pull/1909)
-so if you are using operating system dependencies you'll have to install your
-own Twisted 17.1 package via pip or backports etc.
-
-If you're running in a virtualenv then pip should have installed the newest
-Twisted automatically, but if your virtualenv is old you will need to manually
-upgrade to a newer Twisted dependency via:
-
-    pip install Twisted>=17.1.0
-
 
 Running Synapse
 ===============
@@ -283,16 +261,10 @@ Connecting to Synapse from a client
 The easiest way to try out your new Synapse installation is by connecting to it
 from a web client. The easiest option is probably the one at
 http://riot.im/app. You will need to specify a "Custom server" when you log on
-or register: set this to ``https://domain.tld`` if you setup a reverse proxy
-following the recommended setup, or ``https://localhost:8448`` - remember to specify the
-port (``:8448``) if not ``:443`` unless you changed the configuration. (Leave the identity
+or register: set this to ``https://localhost:8448`` - remember to specify the
+port (``:8448``) unless you changed the configuration. (Leave the identity
 server as the default - see `Identity servers`_.)
 
-If using port 8448 you will run into errors until you accept the self-signed
-certificate. You can easily do this by going to ``https://localhost:8448``
-directly with your browser and accept the presented certificate. You can then
-go back in your web client and proceed further.
-
 If all goes well you should at least be able to log in, create a room, and
 start sending messages.
 
@@ -349,7 +321,7 @@ Debian
 
 Matrix provides official Debian packages via apt from http://matrix.org/packages/debian/.
 Note that these packages do not include a client - choose one from
-https://matrix.org/docs/projects/try-matrix-now.html (or build your own with one of our SDKs :)
+https://matrix.org/docs/projects/try-matrix-now/ (or build your own with one of our SDKs :)
 
 Fedora
 ------
@@ -361,11 +333,8 @@ ArchLinux
 ---------
 
 The quickest way to get up and running with ArchLinux is probably with the community package
-https://www.archlinux.org/packages/community/any/matrix-synapse/, which should pull in most of
-the necessary dependencies. If the default web client is to be served (enabled by default in
-the generated config),
-https://www.archlinux.org/packages/community/any/python2-matrix-angular-sdk/ will also need to
-be installed.
+https://www.archlinux.org/packages/community/any/matrix-synapse/, which should pull in all
+the necessary dependencies.
 
 Alternatively, to install using pip a few changes may be needed as ArchLinux
 defaults to python 3, but synapse currently assumes python 2.7 by default:
@@ -402,7 +371,7 @@ FreeBSD
 
 Synapse can be installed via FreeBSD Ports or Packages contributed by Brendan Molloy from:
 
- - Ports: ``cd /usr/ports/net-im/py-matrix-synapse && make install clean``
+ - Ports: ``cd /usr/ports/net/py-matrix-synapse && make install clean``
  - Packages: ``pkg install py27-matrix-synapse``
 
 
@@ -534,30 +503,6 @@ fix try re-installing from PyPI or directly from
     # Install from github
     pip install --user https://github.com/pyca/pynacl/tarball/master
 
-Running out of File Handles
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-If synapse runs out of filehandles, it typically fails badly - live-locking
-at 100% CPU, and/or failing to accept new TCP connections (blocking the
-connecting client).  Matrix currently can legitimately use a lot of file handles,
-thanks to busy rooms like #matrix:matrix.org containing hundreds of participating
-servers.  The first time a server talks in a room it will try to connect
-simultaneously to all participating servers, which could exhaust the available
-file descriptors between DNS queries & HTTPS sockets, especially if DNS is slow
-to respond.  (We need to improve the routing algorithm used to be better than
-full mesh, but as of June 2017 this hasn't happened yet).
-
-If you hit this failure mode, we recommend increasing the maximum number of
-open file handles to be at least 4096 (assuming a default of 1024 or 256).
-This is typically done by editing ``/etc/security/limits.conf``
-
-Separately, Synapse may leak file handles if inbound HTTP requests get stuck
-during processing - e.g. blocked behind a lock or talking to a remote server etc.
-This is best diagnosed by matching up the 'Received request' and 'Processed request'
-log lines and looking for any 'Processed request' lines which take more than
-a few seconds to execute.  Please let us know at #matrix-dev:matrix.org if
-you see this failure mode so we can help debug it, however.
-
 ArchLinux
 ~~~~~~~~~
 
@@ -599,9 +544,8 @@ you to run your server on a machine that might not have the same name as your
 domain name. For example, you might want to run your server at
 ``synapse.example.com``, but have your Matrix user-ids look like
 ``@user:example.com``. (A SRV record also allows you to change the port from
-the default 8448. However, if you are thinking of using a reverse-proxy on the
-federation port, which is not recommended, be sure to read
-`Reverse-proxying the federation port`_ first.)
+the default 8448. However, if you are thinking of using a reverse-proxy, be
+sure to read `Reverse-proxying the federation port`_ first.)
 
 To use a SRV record, first create your SRV record and publish it in DNS. This
 should have the format ``_matrix._tcp.<yourdomain.com> <ttl> IN SRV 10 0 <port>
@@ -681,7 +625,7 @@ For information on how to install and use PostgreSQL, please see
 Using a reverse proxy with Synapse
 ==================================
 
-It is recommended to put a reverse proxy such as
+It is possible to put a reverse proxy such as
 `nginx <https://nginx.org/en/docs/http/ngx_http_proxy_module.html>`_,
 `Apache <https://httpd.apache.org/docs/current/mod/mod_proxy_http.html>`_ or
 `HAProxy <http://www.haproxy.org/>`_ in front of Synapse. One advantage of
@@ -699,9 +643,9 @@ federation port has a number of pitfalls. It is possible, but be sure to read
 `Reverse-proxying the federation port`_.
 
 The recommended setup is therefore to configure your reverse-proxy on port 443
-to port 8008 of synapse for client connections, but to also directly expose port
-8448 for server-server connections. All the Matrix endpoints begin ``/_matrix``,
-so an example nginx configuration might look like::
+for client connections, but to also expose port 8448 for server-server
+connections. All the Matrix endpoints begin ``/_matrix``, so an example nginx
+configuration might look like::
 
   server {
       listen 443 ssl;
@@ -823,9 +767,7 @@ spidering 'internal' URLs on your network.  At the very least we recommend that
 your loopback and RFC1918 IP addresses are blacklisted.
 
 This also requires the optional lxml and netaddr python dependencies to be
-installed.  This in turn requires the libxml2 library to be available - on
-Debian/Ubuntu this means ``apt-get install libxml2-dev``, or equivalent for
-your OS.
+installed.
 
 
 Password reset
@@ -866,7 +808,7 @@ directory of your choice::
 Synapse has a number of external dependencies, that are easiest
 to install using pip and a virtualenv::
 
-    virtualenv -p python2.7 env
+    virtualenv env
     source env/bin/activate
     python synapse/python_dependencies.py | xargs pip install
     pip install lxml mock
@@ -908,9 +850,12 @@ cache a lot of recent room data and metadata in RAM in order to speed up
 common requests.  We'll improve this in future, but for now the easiest
 way to either reduce the RAM usage (at the risk of slowing things down)
 is to set the almost-undocumented ``SYNAPSE_CACHE_FACTOR`` environment
-variable.  The default is 0.5, which can be decreased to reduce RAM usage
-in memory constrained enviroments, or increased if performance starts to
-degrade.
+variable.  Roughly speaking, a SYNAPSE_CACHE_FACTOR of 1.0 will max out
+at around 3-4GB of resident memory - this is what we currently run the
+matrix.org on.  The default setting is currently 0.1, which is probably
+around a ~700MB footprint.  You can dial it down further to 0.02 if
+desired, which targets roughly ~512MB.  Conversely you can dial it up if
+you need performance for lots of users and have a box with a lot of RAM.
 
 
 .. _`key_management`: https://matrix.org/docs/spec/server_server/unstable.html#retrieving-server-keys

UPGRADE.rst

@@ -5,48 +5,30 @@ Before upgrading check if any special steps are required to upgrade from the
 what you currently have installed to current version of synapse. The extra
 instructions that may be required are listed later in this document.
 
-1. If synapse was installed in a virtualenv then active that virtualenv before
-   upgrading. If synapse is installed in a virtualenv in ``~/.synapse/`` then
-   run:
-
-   .. code:: bash
-
-       source ~/.synapse/bin/activate
-
-2. If synapse was installed using pip then upgrade to the latest version by
-   running:
-
-   .. code:: bash
-
-       pip install --upgrade --process-dependency-links https://github.com/matrix-org/synapse/tarball/master
-
-       # restart synapse
-       synctl restart
-
-
-   If synapse was installed using git then upgrade to the latest version by
-   running:
-
-   .. code:: bash
-
-       # Pull the latest version of the master branch.
-       git pull
-       # Update the versions of synapse's python dependencies.
-       python synapse/python_dependencies.py | xargs pip install --upgrade
-
-       # restart synapse
-       ./synctl restart
-
-
-To check whether your update was sucessful, you can check the Server header
-returned by the Client-Server API:
+If synapse was installed in a virtualenv then active that virtualenv before
+upgrading. If synapse is installed in a virtualenv in ``~/.synapse/`` then run:
 
 .. code:: bash
 
-    # replace <host.name> with the hostname of your synapse homeserver.
-    # You may need to specify a port (eg, :8448) if your server is not
-    # configured on port 443.
-    curl -kv https://<host.name>/_matrix/client/versions 2>&1 | grep "Server:"
+    source ~/.synapse/bin/activate
+
+If synapse was installed using pip then upgrade to the latest version by
+running:
+
+.. code:: bash
+
+    pip install --upgrade --process-dependency-links https://github.com/matrix-org/synapse/tarball/master
+
+If synapse was installed using git then upgrade to the latest version by
+running:
+
+.. code:: bash
+
+    # Pull the latest version of the master branch.
+    git pull
+    # Update the versions of synapse's python dependencies.
+    python synapse/python_dependencies.py | xargs -n1 pip install --upgrade
+
 
 Upgrading to v0.15.0
 ====================
@@ -86,7 +68,7 @@ It has been replaced by specifying a list of application service registrations i
 ``homeserver.yaml``::
 
   app_service_config_files: ["registration-01.yaml", "registration-02.yaml"]
-
+  
 Where ``registration-01.yaml`` looks like::
 
   url: <String>  # e.g. "https://my.application.service.com"
@@ -175,7 +157,7 @@ This release completely changes the database schema and so requires upgrading
 it before starting the new version of the homeserver.
 
 The script "database-prepare-for-0.5.0.sh" should be used to upgrade the
-database. This will save all user information, such as logins and profiles,
+database. This will save all user information, such as logins and profiles, 
 but will otherwise purge the database. This includes messages, which
 rooms the home server was a member of and room alias mappings.
 
@@ -184,18 +166,18 @@ file and ask for help in #matrix:matrix.org. The upgrade process is,
 unfortunately, non trivial and requires human intervention to resolve any
 resulting conflicts during the upgrade process.
 
-Before running the command the homeserver should be first completely
+Before running the command the homeserver should be first completely 
 shutdown. To run it, simply specify the location of the database, e.g.:
 
   ./scripts/database-prepare-for-0.5.0.sh "homeserver.db"
 
-Once this has successfully completed it will be safe to restart the
-homeserver. You may notice that the homeserver takes a few seconds longer to
+Once this has successfully completed it will be safe to restart the 
+homeserver. You may notice that the homeserver takes a few seconds longer to 
 restart than usual as it reinitializes the database.
 
 On startup of the new version, users can either rejoin remote rooms using room
 aliases or by being reinvited. Alternatively, if any other homeserver sends a
-message to a room that the homeserver was previously in the local HS will
+message to a room that the homeserver was previously in the local HS will 
 automatically rejoin the room.
 
 Upgrading to v0.4.0
@@ -254,7 +236,7 @@ automatically generate default config use::
         --config-path homeserver.config \
         --generate-config
 
-This config can be edited if desired, for example to specify a different SSL
+This config can be edited if desired, for example to specify a different SSL 
 certificate to use. Once done you can run the home server using::
 
     $ python synapse/app/homeserver.py --config-path homeserver.config
@@ -275,20 +257,20 @@ This release completely changes the database schema and so requires upgrading
 it before starting the new version of the homeserver.
 
 The script "database-prepare-for-0.0.1.sh" should be used to upgrade the
-database. This will save all user information, such as logins and profiles,
+database. This will save all user information, such as logins and profiles, 
 but will otherwise purge the database. This includes messages, which
 rooms the home server was a member of and room alias mappings.
 
-Before running the command the homeserver should be first completely
+Before running the command the homeserver should be first completely 
 shutdown. To run it, simply specify the location of the database, e.g.:
 
   ./scripts/database-prepare-for-0.0.1.sh "homeserver.db"
 
-Once this has successfully completed it will be safe to restart the
-homeserver. You may notice that the homeserver takes a few seconds longer to
+Once this has successfully completed it will be safe to restart the 
+homeserver. You may notice that the homeserver takes a few seconds longer to 
 restart than usual as it reinitializes the database.
 
 On startup of the new version, users can either rejoin remote rooms using room
 aliases or by being reinvited. Alternatively, if any other homeserver sends a
-message to a room that the homeserver was previously in the local HS will
+message to a room that the homeserver was previously in the local HS will 
 automatically rejoin the room.

contrib/cmdclient/http.py

@@ -36,13 +36,15 @@ class HttpClient(object):
                 the request body. This will be encoded as JSON.
 
         Returns:
-            Deferred: Succeeds when we get a 2xx HTTP response. The result
-            will be the decoded JSON body.
+            Deferred: Succeeds when we get *any* HTTP response.
+
+            The result of the deferred is a tuple of `(code, response)`,
+            where `response` is a dict representing the decoded JSON body.
         """
         pass
 
     def get_json(self, url, args=None):
-        """ Gets some json from the given host homeserver and path
+        """ Get's some json from the given host homeserver and path
 
         Args:
             url (str): The URL to GET data from.
@@ -52,8 +54,10 @@ class HttpClient(object):
                 and *not* a string.
 
         Returns:
-            Deferred: Succeeds when we get a 2xx HTTP response. The result
-            will be the decoded JSON body.
+            Deferred: Succeeds when we get *any* HTTP response.
+
+            The result of the deferred is a tuple of `(code, response)`,
+            where `response` is a dict representing the decoded JSON body.
         """
         pass
 
@@ -210,4 +214,4 @@ class _JsonProducer(object):
         pass
 
     def stopProducing(self):
-        pass
+        pass

contrib/example_log_config.yaml

@@ -39,11 +39,9 @@ loggers:
     synapse:
         level: INFO
 
-    synapse.storage.SQL:
-        # beware: increasing this to DEBUG will make synapse log sensitive
-        # information such as access tokens.
+    synapse.storage:
         level: INFO
-
+    
     # example of enabling debugging for a component:
     #
     # synapse.federation.transport.server:

contrib/prometheus/README

@@ -1,20 +0,0 @@
-This directory contains some sample monitoring config for using the
-'Prometheus' monitoring server against synapse.
-
-To use it, first install prometheus by following the instructions at
-
-  http://prometheus.io/
-
-Then add a new job to the main prometheus.conf file:
-
-  job: {
-    name: "synapse"
-
-    target_group: {
-      target: "http://SERVER.LOCATION.HERE:PORT/_synapse/metrics"
-    }
-  }
-
-Metrics are disabled by default when running synapse; they must be enabled
-with the 'enable-metrics' option, either in the synapse config file or as a
-command-line option.

contrib/prometheus/consoles/synapse.html

@@ -1,395 +0,0 @@
-{{ template "head" . }}
-
-{{ template "prom_content_head" . }}
-<h1>System Resources</h1>
-
-<h3>CPU</h3>
-<div id="process_resource_utime"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#process_resource_utime"),
-  expr: "rate(process_cpu_seconds_total[2m]) * 100",
-  name: "[[job]]",  
-  min: 0,
-  max: 100,
-  renderer: "line",
-  height: 150,
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "%",
-  yTitle: "CPU Usage"
-})
-</script>
-
-<h3>Memory</h3>
-<div id="process_resource_maxrss"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#process_resource_maxrss"),
-  expr: "process_psutil_rss:max",
-  name: "Maxrss",
-  min: 0,
-  renderer: "line",
-  height: 150,
-  yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yUnits: "bytes",
-  yTitle: "Usage"
-})
-</script>
-
-<h3>File descriptors</h3>
-<div id="process_fds"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#process_fds"),
-  expr: "process_open_fds{job='synapse'}",
-  name: "FDs",
-  min: 0,
-  renderer: "line",
-  height: 150,
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "",
-  yTitle: "Descriptors"
-})
-</script>
-
-<h1>Reactor</h1>
-
-<h3>Total reactor time</h3>
-<div id="reactor_total_time"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#reactor_total_time"),
-  expr: "rate(python_twisted_reactor_tick_time:total[2m]) / 1000",
-  name: "time",
-  max: 1,
-  min: 0,
-  renderer: "area",
-  height: 150,
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "s/s",
-  yTitle: "Usage"
-})
-</script>
-
-<h3>Average reactor tick time</h3>
-<div id="reactor_average_time"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#reactor_average_time"),
-  expr: "rate(python_twisted_reactor_tick_time:total[2m]) / rate(python_twisted_reactor_tick_time:count[2m]) / 1000",
-  name: "time",
-  min: 0,
-  renderer: "line",
-  height: 150,
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "s",
-  yTitle: "Time"
-})
-</script>
-
-<h3>Pending calls per tick</h3>
-<div id="reactor_pending_calls"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#reactor_pending_calls"),
-  expr: "rate(python_twisted_reactor_pending_calls:total[30s])/rate(python_twisted_reactor_pending_calls:count[30s])",
-  name: "calls",
-  min: 0,
-  renderer: "line",
-  height: 150,
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yTitle: "Pending Cals"
-})
-</script>
-
-<h1>Storage</h1>
-
-<h3>Queries</h3>
-<div id="synapse_storage_query_time"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_storage_query_time"),
-  expr: "rate(synapse_storage_query_time:count[2m])",
-  name: "[[verb]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yUnits: "queries/s",
-  yTitle: "Queries"
-})
-</script>
-
-<h3>Transactions</h3>
-<div id="synapse_storage_transaction_time"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_storage_transaction_time"),
-  expr: "rate(synapse_storage_transaction_time:count[2m])",
-  name: "[[desc]]",
-  min: 0,
-  yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yUnits: "txn/s",
-  yTitle: "Transactions"
-})
-</script>
-
-<h3>Transaction execution time</h3>
-<div id="synapse_storage_transactions_time_msec"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_storage_transactions_time_msec"),
-  expr: "rate(synapse_storage_transaction_time:total[2m]) / 1000",
-  name: "[[desc]]",
-  min: 0,
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "s/s",
-  yTitle: "Usage"
-})
-</script>
-
-<h3>Database scheduling latency</h3>
-<div id="synapse_storage_schedule_time"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_storage_schedule_time"),
-  expr: "rate(synapse_storage_schedule_time:total[2m]) / 1000",
-  name: "Total latency",
-  min: 0,
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "s/s",
-  yTitle: "Usage"
-})
-</script>
-
-<h3>Cache hit ratio</h3>
-<div id="synapse_cache_ratio"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_cache_ratio"),
-  expr: "rate(synapse_util_caches_cache:total[2m]) * 100",
-  name: "[[name]]",
-  min: 0,
-  max: 100,
-  yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yUnits: "%",
-  yTitle: "Percentage"
-})
-</script>
-
-<h3>Cache size</h3>
-<div id="synapse_cache_size"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_cache_size"),
-  expr: "synapse_util_caches_cache:size",
-  name: "[[name]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yUnits: "",
-  yTitle: "Items"
-})
-</script>
-
-<h1>Requests</h1>
-
-<h3>Requests by Servlet</h3>
-<div id="synapse_http_server_requests_servlet"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_http_server_requests_servlet"),
-  expr: "rate(synapse_http_server_requests:servlet[2m])",
-  name: "[[servlet]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "req/s",
-  yTitle: "Requests"
-})
-</script>
-<h4>&nbsp;(without <tt>EventStreamRestServlet</tt> or <tt>SyncRestServlet</tt>)</h4>
-<div id="synapse_http_server_requests_servlet_minus_events"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_http_server_requests_servlet_minus_events"),
-  expr: "rate(synapse_http_server_requests:servlet{servlet!=\"EventStreamRestServlet\", servlet!=\"SyncRestServlet\"}[2m])",
-  name: "[[servlet]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "req/s",
-  yTitle: "Requests"
-})
-</script>
-
-<h3>Average response times</h3>
-<div id="synapse_http_server_response_time_avg"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_http_server_response_time_avg"),
-  expr: "rate(synapse_http_server_response_time:total[2m]) / rate(synapse_http_server_response_time:count[2m]) / 1000",
-  name: "[[servlet]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "s/req",
-  yTitle: "Response time"
-})
-</script>
-
-<h3>All responses by code</h3>
-<div id="synapse_http_server_responses"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_http_server_responses"),
-  expr: "rate(synapse_http_server_responses[2m])",
-  name: "[[method]] / [[code]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "req/s",
-  yTitle: "Requests"
-})
-</script>
-
-<h3>Error responses by code</h3>
-<div id="synapse_http_server_responses_err"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_http_server_responses_err"),
-  expr: "rate(synapse_http_server_responses{code=~\"[45]..\"}[2m])",
-  name: "[[method]] / [[code]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "req/s",
-  yTitle: "Requests"
-})
-</script>
-
-
-<h3>CPU Usage</h3>
-<div id="synapse_http_server_response_ru_utime"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_http_server_response_ru_utime"),
-  expr: "rate(synapse_http_server_response_ru_utime:total[2m])",
-  name: "[[servlet]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "s/s",
-  yTitle: "CPU Usage"
-})
-</script>
-
-
-<h3>DB Usage</h3>
-<div id="synapse_http_server_response_db_txn_duration"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_http_server_response_db_txn_duration"),
-  expr: "rate(synapse_http_server_response_db_txn_duration:total[2m])",
-  name: "[[servlet]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "s/s",
-  yTitle: "DB Usage"
-})
-</script>
-
-
-<h3>Average event send times</h3>
-<div id="synapse_http_server_send_time_avg"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_http_server_send_time_avg"),
-  expr: "rate(synapse_http_server_response_time:total{servlet='RoomSendEventRestServlet'}[2m]) / rate(synapse_http_server_response_time:count{servlet='RoomSendEventRestServlet'}[2m]) / 1000",
-  name: "[[servlet]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "s/req",
-  yTitle: "Response time"
-})
-</script>
-
-<h1>Federation</h1>
-
-<h3>Sent Messages</h3>
-<div id="synapse_federation_client_sent"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_federation_client_sent"),
-  expr: "rate(synapse_federation_client_sent[2m])",
-  name: "[[type]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "req/s",
-  yTitle: "Requests"
-})
-</script>
-
-<h3>Received Messages</h3>
-<div id="synapse_federation_server_received"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_federation_server_received"),
-  expr: "rate(synapse_federation_server_received[2m])",
-  name: "[[type]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "req/s",
-  yTitle: "Requests"
-})
-</script>
-
-<h3>Pending</h3>
-<div id="synapse_federation_transaction_queue_pending"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_federation_transaction_queue_pending"),
-  expr: "synapse_federation_transaction_queue_pending",
-  name: "[[type]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yUnits: "",
-  yTitle: "Units"
-})
-</script>
-
-<h1>Clients</h1>
-
-<h3>Notifiers</h3>
-<div id="synapse_notifier_listeners"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_notifier_listeners"),
-  expr: "synapse_notifier_listeners",
-  name: "listeners",
-  min: 0,
-  yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yUnits: "",
-  yTitle: "Listeners"
-})
-</script>
-
-<h3>Notified Events</h3>
-<div id="synapse_notifier_notified_events"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_notifier_notified_events"),
-  expr: "rate(synapse_notifier_notified_events[2m])",
-  name: "events",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "events/s",
-  yTitle: "Event rate"
-})
-</script>
-
-{{ template "prom_content_tail" . }}
-
-{{ template "tail" }}

contrib/prometheus/synapse.rules

@@ -1,21 +0,0 @@
-synapse_federation_transaction_queue_pendingEdus:total = sum(synapse_federation_transaction_queue_pendingEdus or absent(synapse_federation_transaction_queue_pendingEdus)*0)
-synapse_federation_transaction_queue_pendingPdus:total = sum(synapse_federation_transaction_queue_pendingPdus or absent(synapse_federation_transaction_queue_pendingPdus)*0)
-
-synapse_http_server_requests:method{servlet=""} = sum(synapse_http_server_requests) by (method)
-synapse_http_server_requests:servlet{method=""} = sum(synapse_http_server_requests) by (servlet)
-
-synapse_http_server_requests:total{servlet=""} = sum(synapse_http_server_requests:by_method) by (servlet)
-
-synapse_cache:hit_ratio_5m = rate(synapse_util_caches_cache:hits[5m]) / rate(synapse_util_caches_cache:total[5m])
-synapse_cache:hit_ratio_30s = rate(synapse_util_caches_cache:hits[30s]) / rate(synapse_util_caches_cache:total[30s])
-
-synapse_federation_client_sent{type="EDU"} = synapse_federation_client_sent_edus + 0
-synapse_federation_client_sent{type="PDU"} = synapse_federation_client_sent_pdu_destinations:count + 0
-synapse_federation_client_sent{type="Query"} = sum(synapse_federation_client_sent_queries) by (job)
-
-synapse_federation_server_received{type="EDU"} = synapse_federation_server_received_edus + 0
-synapse_federation_server_received{type="PDU"} = synapse_federation_server_received_pdus + 0
-synapse_federation_server_received{type="Query"} = sum(synapse_federation_server_received_queries) by (job)
-
-synapse_federation_transaction_queue_pending{type="EDU"} = synapse_federation_transaction_queue_pending_edus + 0
-synapse_federation_transaction_queue_pending{type="PDU"} = synapse_federation_transaction_queue_pending_pdus + 0

contrib/systemd/synapse.service

@@ -1,5 +1,5 @@
 # This assumes that Synapse has been installed as a system package
-# (e.g. https://www.archlinux.org/packages/community/any/matrix-synapse/ for ArchLinux)
+# (e.g. https://aur.archlinux.org/packages/matrix-synapse/ for ArchLinux)
 # rather than in a user home directory or similar under virtualenv.
 
 [Unit]
@@ -9,10 +9,9 @@ Description=Synapse Matrix homeserver
 Type=simple
 User=synapse
 Group=synapse
+EnvironmentFile=-/etc/sysconfig/synapse
 WorkingDirectory=/var/lib/synapse
-ExecStart=/usr/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/synapse/homeserver.yaml
-ExecStop=/usr/bin/synctl stop /etc/synapse/homeserver.yaml
+ExecStart=/usr/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/synapse/homeserver.yaml --log-config=/etc/synapse/log_config.yaml
 
 [Install]
 WantedBy=multi-user.target
-

docs/admin_api/purge_history_api.rst

@@ -4,8 +4,6 @@ Purge History API
 The purge history API allows server admins to purge historic events from their
 database, reclaiming disk space.
 
-**NB!** This will not delete local events (locally sent messages content etc) from the database, but will remove lots of the metadata about them and does dramatically reduce the on disk space usage
-
 Depending on the amount of history being purged a call to the API may take
 several minutes or longer. During this period users will not be able to
 paginate further back in the room from the point being purged from.

docs/admin_api/user_admin_api.rst

@@ -1,73 +0,0 @@
-Query Account
-=============
-
-This API returns information about a specific user account.
-
-The api is::
-
-    GET /_matrix/client/r0/admin/whois/<user_id>
-
-including an ``access_token`` of a server admin.
-
-It returns a JSON body like the following:
-
-.. code:: json
-
-    {
-        "user_id": "<user_id>",
-        "devices": {
-            "": {
-                "sessions": [
-                    {
-                        "connections": [
-                            {
-                                "ip": "1.2.3.4",
-                                "last_seen": 1417222374433,
-                                "user_agent": "Mozilla/5.0 ..."
-                            },
-                            {
-                                "ip": "1.2.3.10",
-                                "last_seen": 1417222374500,
-                                "user_agent": "Dalvik/2.1.0 ..."
-                            }
-                        ]
-                    }
-                ]
-            }
-        }
-    }
-
-``last_seen`` is measured in milliseconds since the Unix epoch.
-
-Deactivate Account
-==================
-
-This API deactivates an account. It removes active access tokens, resets the
-password, and deletes third-party IDs (to prevent the user requesting a
-password reset).
-
-The api is::
-
-    POST /_matrix/client/r0/admin/deactivate/<user_id>
-
-including an ``access_token`` of a server admin, and an empty request body.
-
-
-Reset password
-==============
-
-Changes the password of another user.
-
-The api is::
-
-    POST /_matrix/client/r0/admin/reset_password/<user_id>
-
-with a body of:
-
-.. code:: json
-
-   {
-       "new_password": "<secret>"
-   }
-
-including an ``access_token`` of a server admin.

docs/code_style.rst

@@ -1,119 +1,52 @@
-- Everything should comply with PEP8. Code should pass
-  ``pep8 --max-line-length=100`` without any warnings.
+Basically, PEP8
 
-- **Indenting**:
-
-  - NEVER tabs. 4 spaces to indent.
-
-  - follow PEP8; either hanging indent or multiline-visual indent depending
-    on the size and shape of the arguments and what makes more sense to the
-    author. In other words, both this::
-
-      print("I am a fish %s" % "moo")
-
-    and this::
-
-      print("I am a fish %s" %
-            "moo")
-
-      and this::
-
-        print(
-            "I am a fish %s" %
-            "moo",
-        )
-
-    ...are valid, although given each one takes up 2x more vertical space than
-    the previous, it's up to the author's discretion as to which layout makes
-    most sense for their function invocation.  (e.g. if they want to add
-    comments per-argument, or put expressions in the arguments, or group
-    related arguments together, or want to deliberately extend or preserve
-    vertical/horizontal space)
-
-- **Line length**:
-
-  Max line length is 79 chars (with flexibility to overflow by a "few chars" if
+- NEVER tabs. 4 spaces to indent.
+- Max line width: 79 chars (with flexibility to overflow by a "few chars" if
   the overflowing content is not semantically significant and avoids an
   explosion of vertical whitespace).
-
-  Use parentheses instead of ``\`` for line continuation where ever possible
-  (which is pretty much everywhere).
-
-- **Naming**:
-
-  - Use camel case for class and type names
-  - Use underscores for functions and variables.
-
-- Use double quotes ``"foo"`` rather than single quotes ``'foo'``.
-
-- **Blank lines**:
-
-  - There should be max a single new line between:
-
+- Use camel case for class and type names
+- Use underscores for functions and variables.
+- Use double quotes.
+- Use parentheses instead of '\\' for line continuation where ever possible
+  (which is pretty much everywhere)
+- There should be max a single new line between:
     - statements
     - functions in a class
-
-  - There should be two new lines between:
-
+- There should be two new lines between:
     - definitions in a module (e.g., between different classes)
+- There should be spaces where spaces should be and not where there shouldn't be:
+    - a single space after a comma
+    - a single space before and after for '=' when used as assignment
+    - no spaces before and after for '=' for default values and keyword arguments.
+- Indenting must follow PEP8; either hanging indent or multiline-visual indent
+  depending on the size and shape of the arguments and what makes more sense to
+  the author. In other words, both this::
 
-- **Whitespace**:
+    print("I am a fish %s" % "moo")
 
-  There should be spaces where spaces should be and not where there shouldn't
-  be:
+  and this::
 
-  - a single space after a comma
-  - a single space before and after for '=' when used as assignment
-  - no spaces before and after for '=' for default values and keyword arguments.
+    print("I am a fish %s" %
+          "moo")
 
-- **Comments**: should follow the `google code style
-  <http://google.github.io/styleguide/pyguide.html?showone=Comments#Comments>`_.
-  This is so that we can generate documentation with `sphinx
-  <http://sphinxcontrib-napoleon.readthedocs.org/en/latest/>`_. See the
-  `examples
-  <http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html>`_
-  in the sphinx documentation.
+  and this::
 
-- **Imports**:
+    print(
+        "I am a fish %s" %
+        "moo"
+    )
 
-  - Prefer to import classes and functions than packages or modules.
+  ...are valid, although given each one takes up 2x more vertical space than
+  the previous, it's up to the author's discretion as to which layout makes most
+  sense for their function invocation.  (e.g. if they want to add comments
+  per-argument, or put expressions in the arguments, or group related arguments
+  together, or want to deliberately extend or preserve vertical/horizontal
+  space)
 
-    Example::
+Comments should follow the `google code style <http://google.github.io/styleguide/pyguide.html?showone=Comments#Comments>`_.
+This is so that we can generate documentation with 
+`sphinx <http://sphinxcontrib-napoleon.readthedocs.org/en/latest/>`_. See the
+`examples <http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html>`_
+in the sphinx documentation.
 
-      from synapse.types import UserID
-      ...
-      user_id = UserID(local, server)
-
-    is preferred over::
-
-      from synapse import types
-      ...
-      user_id = types.UserID(local, server)
-
-    (or any other variant).
-
-    This goes against the advice in the Google style guide, but it means that
-    errors in the name are caught early (at import time).
-
-  - Multiple imports from the same package can be combined onto one line::
-
-      from synapse.types import GroupID, RoomID, UserID
-
-    An effort should be made to keep the individual imports in alphabetical
-    order.
-
-    If the list becomes long, wrap it with parentheses and split it over
-    multiple lines.
-
-  - As per `PEP-8 <https://www.python.org/dev/peps/pep-0008/#imports>`_,
-    imports should be grouped in the following order, with a blank line between
-    each group:
-
-    1. standard library imports
-    2. related third party imports
-    3. local application/library specific imports
-
-  - Imports within each group should be sorted alphabetically by module name.
-
-  - Avoid wildcard imports (``from synapse.types import *``) and relative
-    imports (``from .types import UserID``).
+Code should pass pep8 --max-line-length=100 without any warnings.

docs/log_contexts.rst

@@ -1,446 +1,10 @@
-Log contexts
-============
+What do I do about "Unexpected logging context" debug log-lines everywhere?
 
-.. contents::
+<Mjark> The logging context lives in thread local storage
+<Mjark> Sometimes it gets out of sync with what it should actually be, usually because something scheduled something to run on the reactor without preserving the logging context. 
+<Matthew> what is the impact of it getting out of sync? and how and when should we preserve log context?
+<Mjark> The impact is that some of the CPU and database metrics will be under-reported, and some log lines will be mis-attributed.
+<Mjark> It should happen auto-magically in all the APIs that do IO or otherwise defer to the reactor.
+<Erik> Mjark: the other place is if we branch, e.g. using defer.gatherResults
 
-To help track the processing of individual requests, synapse uses a
-'log context' to track which request it is handling at any given moment. This
-is done via a thread-local variable; a ``logging.Filter`` is then used to fish
-the information back out of the thread-local variable and add it to each log
-record.
-
-Logcontexts are also used for CPU and database accounting, so that we can track
-which requests were responsible for high CPU use or database activity.
-
-The ``synapse.util.logcontext`` module provides a facilities for managing the
-current log context (as well as providing the ``LoggingContextFilter`` class).
-
-Deferreds make the whole thing complicated, so this document describes how it
-all works, and how to write code which follows the rules.
-
-Logcontexts without Deferreds
------------------------------
-
-In the absence of any Deferred voodoo, things are simple enough. As with any
-code of this nature, the rule is that our function should leave things as it
-found them:
-
-.. code:: python
-
-    from synapse.util import logcontext         # omitted from future snippets
-
-    def handle_request(request_id):
-        request_context = logcontext.LoggingContext()
-
-        calling_context = logcontext.LoggingContext.current_context()
-        logcontext.LoggingContext.set_current_context(request_context)
-        try:
-            request_context.request = request_id
-            do_request_handling()
-            logger.debug("finished")
-        finally:
-            logcontext.LoggingContext.set_current_context(calling_context)
-
-    def do_request_handling():
-        logger.debug("phew")  # this will be logged against request_id
-
-
-LoggingContext implements the context management methods, so the above can be
-written much more succinctly as:
-
-.. code:: python
-
-    def handle_request(request_id):
-        with logcontext.LoggingContext() as request_context:
-            request_context.request = request_id
-            do_request_handling()
-            logger.debug("finished")
-
-    def do_request_handling():
-        logger.debug("phew")
-
-
-Using logcontexts with Deferreds
---------------------------------
-
-Deferreds — and in particular, ``defer.inlineCallbacks`` — break
-the linear flow of code so that there is no longer a single entry point where
-we should set the logcontext and a single exit point where we should remove it.
-
-Consider the example above, where ``do_request_handling`` needs to do some
-blocking operation, and returns a deferred:
-
-.. code:: python
-
-    @defer.inlineCallbacks
-    def handle_request(request_id):
-        with logcontext.LoggingContext() as request_context:
-            request_context.request = request_id
-            yield do_request_handling()
-            logger.debug("finished")
-
-
-In the above flow:
-
-* The logcontext is set
-* ``do_request_handling`` is called, and returns a deferred
-* ``handle_request`` yields the deferred
-* The ``inlineCallbacks`` wrapper of ``handle_request`` returns a deferred
-
-So we have stopped processing the request (and will probably go on to start
-processing the next), without clearing the logcontext.
-
-To circumvent this problem, synapse code assumes that, wherever you have a
-deferred, you will want to yield on it. To that end, whereever functions return
-a deferred, we adopt the following conventions:
-
-**Rules for functions returning deferreds:**
-
-  * If the deferred is already complete, the function returns with the same
-    logcontext it started with.
-  * If the deferred is incomplete, the function clears the logcontext before
-    returning; when the deferred completes, it restores the logcontext before
-    running any callbacks.
-
-That sounds complicated, but actually it means a lot of code (including the
-example above) "just works". There are two cases:
-
-* If ``do_request_handling`` returns a completed deferred, then the logcontext
-  will still be in place. In this case, execution will continue immediately
-  after the ``yield``; the "finished" line will be logged against the right
-  context, and the ``with`` block restores the original context before we
-  return to the caller.
-
-* If the returned deferred is incomplete, ``do_request_handling`` clears the
-  logcontext before returning. The logcontext is therefore clear when
-  ``handle_request`` yields the deferred. At that point, the ``inlineCallbacks``
-  wrapper adds a callback to the deferred, and returns another (incomplete)
-  deferred to the caller, and it is safe to begin processing the next request.
-
-  Once ``do_request_handling``'s deferred completes, it will reinstate the
-  logcontext, before running the callback added by the ``inlineCallbacks``
-  wrapper. That callback runs the second half of ``handle_request``, so again
-  the "finished" line will be logged against the right
-  context, and the ``with`` block restores the original context.
-
-As an aside, it's worth noting that ``handle_request`` follows our rules -
-though that only matters if the caller has its own logcontext which it cares
-about.
-
-The following sections describe pitfalls and helpful patterns when implementing
-these rules.
-
-Always yield your deferreds
----------------------------
-
-Whenever you get a deferred back from a function, you should ``yield`` on it
-as soon as possible. (Returning it directly to your caller is ok too, if you're
-not doing ``inlineCallbacks``.) Do not pass go; do not do any logging; do not
-call any other functions.
-
-.. code:: python
-
-    @defer.inlineCallbacks
-    def fun():
-        logger.debug("starting")
-        yield do_some_stuff()       # just like this
-
-        d = more_stuff()
-        result = yield d            # also fine, of course
-
-        defer.returnValue(result)
-
-    def nonInlineCallbacksFun():
-        logger.debug("just a wrapper really")
-        return do_some_stuff()      # this is ok too - the caller will yield on
-                                    # it anyway.
-
-Provided this pattern is followed all the way back up to the callchain to where
-the logcontext was set, this will make things work out ok: provided
-``do_some_stuff`` and ``more_stuff`` follow the rules above, then so will
-``fun`` (as wrapped by ``inlineCallbacks``) and ``nonInlineCallbacksFun``.
-
-It's all too easy to forget to ``yield``: for instance if we forgot that
-``do_some_stuff`` returned a deferred, we might plough on regardless. This
-leads to a mess; it will probably work itself out eventually, but not before
-a load of stuff has been logged against the wrong content. (Normally, other
-things will break, more obviously, if you forget to ``yield``, so this tends
-not to be a major problem in practice.)
-
-Of course sometimes you need to do something a bit fancier with your Deferreds
-- not all code follows the linear A-then-B-then-C pattern. Notes on
-implementing more complex patterns are in later sections.
-
-Where you create a new Deferred, make it follow the rules
----------------------------------------------------------
-
-Most of the time, a Deferred comes from another synapse function. Sometimes,
-though, we need to make up a new Deferred, or we get a Deferred back from
-external code. We need to make it follow our rules.
-
-The easy way to do it is with a combination of ``defer.inlineCallbacks``, and
-``logcontext.PreserveLoggingContext``. Suppose we want to implement ``sleep``,
-which returns a deferred which will run its callbacks after a given number of
-seconds. That might look like:
-
-.. code:: python
-
-    # not a logcontext-rules-compliant function
-    def get_sleep_deferred(seconds):
-        d = defer.Deferred()
-        reactor.callLater(seconds, d.callback, None)
-        return d
-
-That doesn't follow the rules, but we can fix it by wrapping it with
-``PreserveLoggingContext`` and ``yield`` ing on it:
-
-.. code:: python
-
-    @defer.inlineCallbacks
-    def sleep(seconds):
-        with PreserveLoggingContext():
-            yield get_sleep_deferred(seconds)
-
-This technique works equally for external functions which return deferreds,
-or deferreds we have made ourselves.
-
-You can also use ``logcontext.make_deferred_yieldable``, which just does the
-boilerplate for you, so the above could be written:
-
-.. code:: python
-
-    def sleep(seconds):
-        return logcontext.make_deferred_yieldable(get_sleep_deferred(seconds))
-
-
-Fire-and-forget
----------------
-
-Sometimes you want to fire off a chain of execution, but not wait for its
-result. That might look a bit like this:
-
-.. code:: python
-
-    @defer.inlineCallbacks
-    def do_request_handling():
-        yield foreground_operation()
-
-        # *don't* do this
-        background_operation()
-
-        logger.debug("Request handling complete")
-
-    @defer.inlineCallbacks
-    def background_operation():
-        yield first_background_step()
-        logger.debug("Completed first step")
-        yield second_background_step()
-        logger.debug("Completed second step")
-
-The above code does a couple of steps in the background after
-``do_request_handling`` has finished. The log lines are still logged against
-the ``request_context`` logcontext, which may or may not be desirable. There
-are two big problems with the above, however. The first problem is that, if
-``background_operation`` returns an incomplete Deferred, it will expect its
-caller to ``yield`` immediately, so will have cleared the logcontext. In this
-example, that means that 'Request handling complete' will be logged without any
-context.
-
-The second problem, which is potentially even worse, is that when the Deferred
-returned by ``background_operation`` completes, it will restore the original
-logcontext. There is nothing waiting on that Deferred, so the logcontext will
-leak into the reactor and possibly get attached to some arbitrary future
-operation.
-
-There are two potential solutions to this.
-
-One option is to surround the call to ``background_operation`` with a
-``PreserveLoggingContext`` call. That will reset the logcontext before
-starting ``background_operation`` (so the context restored when the deferred
-completes will be the empty logcontext), and will restore the current
-logcontext before continuing the foreground process:
-
-.. code:: python
-
-    @defer.inlineCallbacks
-    def do_request_handling():
-        yield foreground_operation()
-
-        # start background_operation off in the empty logcontext, to
-        # avoid leaking the current context into the reactor.
-        with PreserveLoggingContext():
-            background_operation()
-
-        # this will now be logged against the request context
-        logger.debug("Request handling complete")
-
-Obviously that option means that the operations done in
-``background_operation`` would be not be logged against a logcontext (though
-that might be fixed by setting a different logcontext via a ``with
-LoggingContext(...)`` in ``background_operation``).
-
-The second option is to use ``logcontext.preserve_fn``, which wraps a function
-so that it doesn't reset the logcontext even when it returns an incomplete
-deferred, and adds a callback to the returned deferred to reset the
-logcontext. In other words, it turns a function that follows the Synapse rules
-about logcontexts and Deferreds into one which behaves more like an external
-function — the opposite operation to that described in the previous section.
-It can be used like this:
-
-.. code:: python
-
-    @defer.inlineCallbacks
-    def do_request_handling():
-        yield foreground_operation()
-
-        logcontext.preserve_fn(background_operation)()
-
-        # this will now be logged against the request context
-        logger.debug("Request handling complete")
-
-XXX: I think ``preserve_context_over_fn`` is supposed to do the first option,
-but the fact that it does ``preserve_context_over_deferred`` on its results
-means that its use is fraught with difficulty.
-
-Passing synapse deferreds into third-party functions
-----------------------------------------------------
-
-A typical example of this is where we want to collect together two or more
-deferred via ``defer.gatherResults``:
-
-.. code:: python
-
-    d1 = operation1()
-    d2 = operation2()
-    d3 = defer.gatherResults([d1, d2])
-
-This is really a variation of the fire-and-forget problem above, in that we are
-firing off ``d1`` and ``d2`` without yielding on them. The difference
-is that we now have third-party code attached to their callbacks. Anyway either
-technique given in the `Fire-and-forget`_ section will work.
-
-Of course, the new Deferred returned by ``gatherResults`` needs to be wrapped
-in order to make it follow the logcontext rules before we can yield it, as
-described in `Where you create a new Deferred, make it follow the rules`_.
-
-So, option one: reset the logcontext before starting the operations to be
-gathered:
-
-.. code:: python
-
-    @defer.inlineCallbacks
-    def do_request_handling():
-        with PreserveLoggingContext():
-            d1 = operation1()
-            d2 = operation2()
-            result = yield defer.gatherResults([d1, d2])
-
-In this case particularly, though, option two, of using
-``logcontext.preserve_fn`` almost certainly makes more sense, so that
-``operation1`` and ``operation2`` are both logged against the original
-logcontext. This looks like:
-
-.. code:: python
-
-    @defer.inlineCallbacks
-    def do_request_handling():
-        d1 = logcontext.preserve_fn(operation1)()
-        d2 = logcontext.preserve_fn(operation2)()
-
-        with PreserveLoggingContext():
-            result = yield defer.gatherResults([d1, d2])
-
-
-Was all this really necessary?
-------------------------------
-
-The conventions used work fine for a linear flow where everything happens in
-series via ``defer.inlineCallbacks`` and ``yield``, but are certainly tricky to
-follow for any more exotic flows. It's hard not to wonder if we could have done
-something else.
-
-We're not going to rewrite Synapse now, so the following is entirely of
-academic interest, but I'd like to record some thoughts on an alternative
-approach.
-
-I briefly prototyped some code following an alternative set of rules. I think
-it would work, but I certainly didn't get as far as thinking how it would
-interact with concepts as complicated as the cache descriptors.
-
-My alternative rules were:
-
-* functions always preserve the logcontext of their caller, whether or not they
-  are returning a Deferred.
-
-* Deferreds returned by synapse functions run their callbacks in the same
-  context as the function was orignally called in.
-
-The main point of this scheme is that everywhere that sets the logcontext is
-responsible for clearing it before returning control to the reactor.
-
-So, for example, if you were the function which started a ``with
-LoggingContext`` block, you wouldn't ``yield`` within it — instead you'd start
-off the background process, and then leave the ``with`` block to wait for it:
-
-.. code:: python
-
-    def handle_request(request_id):
-        with logcontext.LoggingContext() as request_context:
-            request_context.request = request_id
-            d = do_request_handling()
-
-        def cb(r):
-            logger.debug("finished")
-
-        d.addCallback(cb)
-        return d
-
-(in general, mixing ``with LoggingContext`` blocks and
-``defer.inlineCallbacks`` in the same function leads to slighly
-counter-intuitive code, under this scheme).
-
-Because we leave the original ``with`` block as soon as the Deferred is
-returned (as opposed to waiting for it to be resolved, as we do today), the
-logcontext is cleared before control passes back to the reactor; so if there is
-some code within ``do_request_handling`` which needs to wait for a Deferred to
-complete, there is no need for it to worry about clearing the logcontext before
-doing so:
-
-.. code:: python
-
-    def handle_request():
-        r = do_some_stuff()
-        r.addCallback(do_some_more_stuff)
-        return r
-
-— and provided ``do_some_stuff`` follows the rules of returning a Deferred which
-runs its callbacks in the original logcontext, all is happy.
-
-The business of a Deferred which runs its callbacks in the original logcontext
-isn't hard to achieve — we have it today, in the shape of
-``logcontext._PreservingContextDeferred``:
-
-.. code:: python
-
-    def do_some_stuff():
-        deferred = do_some_io()
-        pcd = _PreservingContextDeferred(LoggingContext.current_context())
-        deferred.chainDeferred(pcd)
-        return pcd
-
-It turns out that, thanks to the way that Deferreds chain together, we
-automatically get the property of a context-preserving deferred with
-``defer.inlineCallbacks``, provided the final Defered the function ``yields``
-on has that property. So we can just write:
-
-.. code:: python
-
-    @defer.inlineCallbacks
-    def handle_request():
-        yield do_some_stuff()
-        yield do_some_more_stuff()
-
-To conclude: I think this scheme would have worked equally well, with less
-danger of messing it up, and probably made some more esoteric code easier to
-write. But again — changing the conventions of the entire Synapse codebase is
-not a sensible option for the marginal improvement offered.
+Unanswered: how and when should we preserve log context?

docs/metrics-howto.rst

@@ -21,12 +21,13 @@ How to monitor Synapse metrics using Prometheus
 
 3. Add a prometheus target for synapse.
 
-   It needs to set the ``metrics_path`` to a non-default value (under ``scrape_configs``)::
+   It needs to set the ``metrics_path`` to a non-default value::
 
     - job_name: "synapse"
       metrics_path: "/_synapse/metrics"
       static_configs:
-        - targets: ["my.server.here:9092"]
+        - targets:
+            "my.server.here:9092"
 
    If your prometheus is older than 1.5.2, you will need to replace 
    ``static_configs`` in the above with ``target_groups``.

docs/password_auth_providers.rst

@@ -1,99 +0,0 @@
-Password auth provider modules
-==============================
-
-Password auth providers offer a way for server administrators to integrate
-their Synapse installation with an existing authentication system.
-
-A password auth provider is a Python class which is dynamically loaded into
-Synapse, and provides a number of methods by which it can integrate with the
-authentication system.
-
-This document serves as a reference for those looking to implement their own
-password auth providers.
-
-Required methods
-----------------
-
-Password auth provider classes must provide the following methods:
-
-*class* ``SomeProvider.parse_config``\(*config*)
-
-    This method is passed the ``config`` object for this module from the
-    homeserver configuration file.
-
-    It should perform any appropriate sanity checks on the provided
-    configuration, and return an object which is then passed into ``__init__``.
-
-*class* ``SomeProvider``\(*config*, *account_handler*)
-
-    The constructor is passed the config object returned by ``parse_config``,
-    and a ``synapse.module_api.ModuleApi`` object which allows the
-    password provider to check if accounts exist and/or create new ones.
-
-Optional methods
-----------------
-
-Password auth provider classes may optionally provide the following methods.
-
-*class* ``SomeProvider.get_db_schema_files``\()
-
-    This method, if implemented, should return an Iterable of ``(name,
-    stream)`` pairs of database schema files. Each file is applied in turn at
-    initialisation, and a record is then made in the database so that it is
-    not re-applied on the next start.
-
-``someprovider.get_supported_login_types``\()
-
-    This method, if implemented, should return a ``dict`` mapping from a login
-    type identifier (such as ``m.login.password``) to an iterable giving the
-    fields which must be provided by the user in the submission to the
-    ``/login`` api. These fields are passed in the ``login_dict`` dictionary
-    to ``check_auth``.
-
-    For example, if a password auth provider wants to implement a custom login
-    type of ``com.example.custom_login``, where the client is expected to pass
-    the fields ``secret1`` and ``secret2``, the provider should implement this
-    method and return the following dict::
-
-      {"com.example.custom_login": ("secret1", "secret2")}
-
-``someprovider.check_auth``\(*username*, *login_type*, *login_dict*)
-
-    This method is the one that does the real work. If implemented, it will be
-    called for each login attempt where the login type matches one of the keys
-    returned by ``get_supported_login_types``.
-
-    It is passed the (possibly UNqualified) ``user`` provided by the client,
-    the login type, and a dictionary of login secrets passed by the client.
-
-    The method should return a Twisted ``Deferred`` object, which resolves to
-    the canonical ``@localpart:domain`` user id if authentication is successful,
-    and ``None`` if not.
-
-    Alternatively, the ``Deferred`` can resolve to a ``(str, func)`` tuple, in
-    which case the second field is a callback which will be called with the
-    result from the ``/login`` call (including ``access_token``, ``device_id``,
-    etc.)
-
-``someprovider.check_password``\(*user_id*, *password*)
-
-    This method provides a simpler interface than ``get_supported_login_types``
-    and ``check_auth`` for password auth providers that just want to provide a
-    mechanism for validating ``m.login.password`` logins.
-
-    Iif implemented, it will be called to check logins with an
-    ``m.login.password`` login type. It is passed a qualified
-    ``@localpart:domain`` user id, and the password provided by the user.
-
-    The method should return a Twisted ``Deferred`` object, which resolves to
-    ``True`` if authentication is successful, and ``False`` if not.
-
-``someprovider.on_logged_out``\(*user_id*, *device_id*, *access_token*)
-
-    This method, if implemented, is called when a user logs out. It is passed
-    the qualified user ID, the ID of the deactivated device (if any: access
-    tokens are occasionally created without an associated device ID), and the
-    (now deactivated) access token.
-
-    It may return a Twisted ``Deferred`` object; the logout request will wait
-    for the deferred to complete but the result is ignored.

docs/postgres.rst

@@ -1,8 +1,6 @@
 Using Postgres
 --------------
 
-Postgres version 9.4 or later is known to work.
-
 Set up database
 ===============
 
@@ -114,9 +112,9 @@ script one last time, e.g. if the SQLite database is at  ``homeserver.db``
 run::
 
     synapse_port_db --sqlite-database homeserver.db \
-        --postgres-config homeserver-postgres.yaml
+        --postgres-config database_config.yaml
 
 Once that has completed, change the synapse config to point at the PostgreSQL
-database configuration file ``homeserver-postgres.yaml`` (i.e. rename it to 
-``homeserver.yaml``) and restart synapse. Synapse should now be running against
+database configuration file using the ``database_config`` parameter (see
+`Synapse Config`_) and restart synapse. Synapse should now be running against
 PostgreSQL.

docs/replication.rst

@@ -26,10 +26,28 @@ expose the append-only log to the readers should be fairly minimal.
 Architecture
 ------------
 
-The Replication Protocol
-~~~~~~~~~~~~~~~~~~~~~~~~
+The Replication API
+~~~~~~~~~~~~~~~~~~~
 
-See ``tcp_replication.rst``
+Synapse will optionally expose a long poll HTTP API for extracting updates. The
+API will have a similar shape to /sync in that clients provide tokens
+indicating where in the log they have reached and a timeout. The synapse server
+then either responds with updates immediately if it already has updates or it
+waits until the timeout for more updates. If the timeout expires and nothing
+happened then the server returns an empty response.
+
+However unlike the /sync API this replication API is returning synapse specific
+data rather than trying to implement a matrix specification. The replication
+results are returned as arrays of rows where the rows are mostly lifted
+directly from the database. This avoids unnecessary JSON parsing on the server
+and hopefully avoids an impedance mismatch between the data returned and the
+required updates to the datastore.
+
+This does not replicate all the database tables as many of the database tables
+are indexes that can be recovered from the contents of other tables.
+
+The format and parameters for the api are documented in
+``synapse/replication/resource.py``.
 
 
 The Slaved DataStore

docs/sphinx/conf.py

@@ -50,7 +50,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = u'Synapse'
-copyright = u'Copyright 2014-2017 OpenMarket Ltd, 2017 Vector Creations Ltd, 2017 New Vector Ltd'
+copyright = u'2014, TNG'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the

docs/tcp_replication.rst

@@ -1,223 +0,0 @@
-TCP Replication
-===============
-
-Motivation
-----------
-
-Previously the workers used an HTTP long poll mechanism to get updates from the
-master, which had the problem of causing a lot of duplicate work on the server.
-This TCP protocol replaces those APIs with the aim of increased efficiency.
-
-
-
-Overview
---------
-
-The protocol is based on fire and forget, line based commands. An example flow
-would be (where '>' indicates master to worker and '<' worker to master flows)::
-
-    > SERVER example.com
-    < REPLICATE events 53
-    > RDATA events 54 ["$foo1:bar.com", ...]
-    > RDATA events 55 ["$foo4:bar.com", ...]
-
-The example shows the server accepting a new connection and sending its identity
-with the ``SERVER`` command, followed by the client asking to subscribe to the
-``events`` stream from the token ``53``. The server then periodically sends ``RDATA``
-commands which have the format ``RDATA <stream_name> <token> <row>``, where the
-format of ``<row>`` is defined by the individual streams.
-
-Error reporting happens by either the client or server sending an `ERROR`
-command, and usually the connection will be closed.
-
-
-Since the protocol is a simple line based, its possible to manually connect to
-the server using a tool like netcat. A few things should be noted when manually
-using the protocol:
-
-* When subscribing to a stream using ``REPLICATE``, the special token ``NOW`` can
-  be used to get all future updates. The special stream name ``ALL`` can be used
-  with ``NOW`` to subscribe to all available streams.
-* The federation stream is only available if federation sending has been
-  disabled on the main process.
-* The server will only time connections out that have sent a ``PING`` command.
-  If a ping is sent then the connection will be closed if no further commands
-  are receieved within 15s. Both the client and server protocol implementations
-  will send an initial PING on connection and ensure at least one command every
-  5s is sent (not necessarily ``PING``).
-* ``RDATA`` commands *usually* include a numeric token, however if the stream
-  has multiple rows to replicate per token the server will send multiple
-  ``RDATA`` commands, with all but the last having a token of ``batch``. See
-  the documentation on ``commands.RdataCommand`` for further details.
-
-
-Architecture
-------------
-
-The basic structure of the protocol is line based, where the initial word of
-each line specifies the command. The rest of the line is parsed based on the
-command. For example, the `RDATA` command is defined as::
-
-    RDATA <stream_name> <token> <row_json>
-
-(Note that `<row_json>` may contains spaces, but cannot contain newlines.)
-
-Blank lines are ignored.
-
-
-Keep alives
-~~~~~~~~~~~
-
-Both sides are expected to send at least one command every 5s or so, and
-should send a ``PING`` command if necessary. If either side do not receive a
-command within e.g. 15s then the connection should be closed.
-
-Because the server may be connected to manually using e.g. netcat, the timeouts
-aren't enabled until an initial ``PING`` command is seen. Both the client and
-server implementations below send a ``PING`` command immediately on connection to
-ensure the timeouts are enabled.
-
-This ensures that both sides can quickly realize if the tcp connection has gone
-and handle the situation appropriately.
-
-
-Start up
-~~~~~~~~
-
-When a new connection is made, the server:
-
-* Sends a ``SERVER`` command, which includes the identity of the server, allowing
-  the client to detect if its connected to the expected server
-* Sends a ``PING`` command as above, to enable the client to time out connections
-  promptly.
-
-The client:
-
-* Sends a ``NAME`` command, allowing the server to associate a human friendly
-  name with the connection. This is optional.
-* Sends a ``PING`` as above
-* For each stream the client wishes to subscribe to it sends a ``REPLICATE``
-  with the stream_name and token it wants to subscribe from.
-* On receipt of a ``SERVER`` command, checks that the server name matches the
-  expected server name.
-
-
-Error handling
-~~~~~~~~~~~~~~
-
-If either side detects an error it can send an ``ERROR`` command and close the
-connection.
-
-If the client side loses the connection to the server it should reconnect,
-following the steps above.
-
-
-Congestion
-~~~~~~~~~~
-
-If the server sends messages faster than the client can consume them the server
-will first buffer a (fairly large) number of commands and then disconnect the
-client. This ensures that we don't queue up an unbounded number of commands in
-memory and gives us a potential oppurtunity to squawk loudly. When/if the client
-recovers it can reconnect to the server and ask for missed messages.
-
-
-Reliability
-~~~~~~~~~~~
-
-In general the replication stream should be considered an unreliable transport
-since e.g. commands are not resent if the connection disappears.
-
-The exception to that are the replication streams, i.e. RDATA commands, since
-these include tokens which can be used to restart the stream on connection
-errors.
-
-The client should keep track of the token in the last RDATA command received
-for each stream so that on reconneciton it can start streaming from the correct
-place. Note: not all RDATA have valid tokens due to batching. See
-``RdataCommand`` for more details.
-
-
-Example
-~~~~~~~
-
-An example iteraction is shown below. Each line is prefixed with '>' or '<' to
-indicate which side is sending, these are *not* included on the wire::
-
-    * connection established *
-    > SERVER localhost:8823
-    > PING 1490197665618
-    < NAME synapse.app.appservice
-    < PING 1490197665618
-    < REPLICATE events 1
-    < REPLICATE backfill 1
-    < REPLICATE caches 1
-    > POSITION events 1
-    > POSITION backfill 1
-    > POSITION caches 1
-    > RDATA caches 2 ["get_user_by_id",["@01register-user:localhost:8823"],1490197670513]
-    > RDATA events 14 ["$149019767112vOHxz:localhost:8823",
-        "!AFDCvgApUmpdfVjIXm:localhost:8823","m.room.guest_access","",null]
-    < PING 1490197675618
-    > ERROR server stopping
-    * connection closed by server *
-
-The ``POSITION`` command sent by the server is used to set the clients position
-without needing to send data with the ``RDATA`` command.
-
-
-An example of a batched set of ``RDATA`` is::
-
-    > RDATA caches batch ["get_user_by_id",["@test:localhost:8823"],1490197670513]
-    > RDATA caches batch ["get_user_by_id",["@test2:localhost:8823"],1490197670513]
-    > RDATA caches batch ["get_user_by_id",["@test3:localhost:8823"],1490197670513]
-    > RDATA caches 54 ["get_user_by_id",["@test4:localhost:8823"],1490197670513]
-
-In this case the client shouldn't advance their caches token until it sees the
-the last ``RDATA``.
-
-
-List of commands
-~~~~~~~~~~~~~~~~
-
-The list of valid commands, with which side can send it: server (S) or client (C):
-
-SERVER (S)
-    Sent at the start to identify which server the client is talking to
-
-RDATA (S)
-    A single update in a stream
-
-POSITION (S)
-    The position of the stream has been updated
-
-ERROR (S, C)
-    There was an error
-
-PING (S, C)
-    Sent periodically to ensure the connection is still alive
-
-NAME (C)
-    Sent at the start by client to inform the server who they are
-
-REPLICATE (C)
-    Asks the server to replicate a given stream
-
-USER_SYNC (C)
-    A user has started or stopped syncing
-
-FEDERATION_ACK (C)
-    Acknowledge receipt of some federation data
-
-REMOVE_PUSHER (C)
-    Inform the server a pusher should be removed
-
-INVALIDATE_CACHE (C)
-    Inform the server a cache should be invalidated
-
-SYNC (S, C)
-    Used exclusively in tests
-
-
-See ``synapse/replication/tcp/commands.py`` for a detailed description and the
-format of each command.

docs/turn-howto.rst

@@ -50,37 +50,14 @@ You may be able to setup coturn via your package manager,  or set it up manually
 
        pwgen -s 64 1
 
- 5. Consider your security settings.  TURN lets users request a relay
-    which will connect to arbitrary IP addresses and ports.  At the least
-    we recommend:
-
-       # VoIP traffic is all UDP. There is no reason to let users connect to arbitrary TCP endpoints via the relay.
-       no-tcp-relay
-
-       # don't let the relay ever try to connect to private IP address ranges within your network (if any)
-       # given the turn server is likely behind your firewall, remember to include any privileged public IPs too.
-       denied-peer-ip=10.0.0.0-10.255.255.255
-       denied-peer-ip=192.168.0.0-192.168.255.255
-       denied-peer-ip=172.16.0.0-172.31.255.255
-
-       # special case the turn server itself so that client->TURN->TURN->client flows work
-       allowed-peer-ip=10.0.0.1
-
-       # consider whether you want to limit the quota of relayed streams per user (or total) to avoid risk of DoS.
-       user-quota=12 # 4 streams per video call, so 12 streams = 3 simultaneous relayed calls per user.
-       total-quota=1200
-
-    Ideally coturn should refuse to relay traffic which isn't SRTP;
-    see https://github.com/matrix-org/synapse/issues/2009
-
- 6. Ensure your firewall allows traffic into the TURN server on
+ 5. Ensure youe firewall allows traffic into the TURN server on
     the ports you've configured it to listen on (remember to allow
-    both TCP and UDP TURN traffic)
+    both TCP and UDP if you've enabled both).
 
- 7. If you've configured coturn to support TLS/DTLS, generate or
+ 6. If you've configured coturn to support TLS/DTLS, generate or
     import your private key and certificate.
 
- 8. Start the turn server::
+ 7. Start the turn server::
  
        bin/turnserver -o
 
@@ -106,19 +83,12 @@ Your home server configuration file needs the following extra keys:
     to refresh credentials. The TURN REST API specification recommends
     one day (86400000).
 
-  4. "turn_allow_guests": Whether to allow guest users to use the TURN
-    server.  This is enabled by default, as otherwise VoIP will not
-    work reliably for guests.  However, it does introduce a security risk
-    as it lets guests connect to arbitrary endpoints without having gone
-    through a CAPTCHA or similar to register a real account.
-
 As an example, here is the relevant section of the config file for
 matrix.org::
 
     turn_uris: [ "turn:turn.matrix.org:3478?transport=udp", "turn:turn.matrix.org:3478?transport=tcp" ]
     turn_shared_secret: n0t4ctuAllymatr1Xd0TorgSshar3d5ecret4obvIousreAsons
     turn_user_lifetime: 86400000
-    turn_allow_guests: True
 
 Now, restart synapse::
 

docs/url_previews.rst

@@ -56,7 +56,6 @@ As a first cut, let's do #2 and have the receiver hit the API to calculate its o
 API
 ---
 
-```
 GET /_matrix/media/r0/preview_url?url=http://wherever.com
 200 OK
 {
@@ -67,7 +66,6 @@ GET /_matrix/media/r0/preview_url?url=http://wherever.com
     "og:description" : "“Synapse 0.12 is out! Lots of polishing, performance & bugfixes: /sync API, /r0 prefix, fulltext search, 3PID invites https://t.co/5alhXLLEGP”"
     "og:site_name"   : "Twitter"
 }
-```
 
 * Downloads the URL
   * If HTML, just stores it in RAM and parses it for OG meta tags

docs/user_directory.md

@@ -1,17 +0,0 @@
-User Directory API Implementation
-=================================
-
-The user directory is currently maintained based on the 'visible' users
-on this particular server - i.e. ones which your account shares a room with, or
-who are present in a publicly viewable room present on the server.
-
-The directory info is stored in various tables, which can (typically after
-DB corruption) get stale or out of sync.  If this happens, for now the
-quickest solution to fix it is:
-
-```
-UPDATE user_directory_stream_pos SET stream_id = NULL;
-```
-
-and restart the synapse, which should then start a background task to
-flush the current tables and regenerate the directory.

docs/workers.rst

@@ -12,7 +12,7 @@ across multiple processes is a recipe for disaster, plus you should be using
 postgres anyway if you care about scalability).
 
 The workers communicate with the master synapse process via a synapse-specific
-TCP protocol called 'replication' - analogous to MySQL or Postgres style
+HTTP protocol called 'replication' - analogous to MySQL or Postgres style
 database replication; feeding a stream of relevant data to the workers so they
 can be kept in sync with the main synapse process and database state.
 
@@ -21,11 +21,16 @@ To enable workers, you need to add a replication listener to the master synapse,
     listeners:
       - port: 9092
         bind_address: '127.0.0.1'
-        type: replication
+        type: http
+        tls: false
+        x_forwarded: false
+        resources:
+          - names: [replication]
+            compress: false
 
 Under **no circumstances** should this replication API listener be exposed to the
 public internet; it currently implements no authentication whatsoever and is
-unencrypted.
+unencrypted HTTP.
 
 You then create a set of configs for the various worker processes.  These should be
 worker configuration files should be stored in a dedicated subdirectory, to allow
@@ -45,16 +50,14 @@ e.g. the HTTP listener that it provides (if any); logging configuration; etc.
 You should minimise the number of overrides though to maintain a usable config.
 
 You must specify the type of worker application (worker_app) and the replication
-endpoint that it's talking to on the main synapse process (worker_replication_host
-and worker_replication_port).
+endpoint that it's talking to on the main synapse process (worker_replication_url).
 
 For instance::
 
     worker_app: synapse.app.synchrotron
 
     # The replication listener on the synapse to talk to.
-    worker_replication_host: 127.0.0.1
-    worker_replication_port: 9092
+    worker_replication_url: http://127.0.0.1:9092/_synapse/replication
 
     worker_listeners:
      - type: http
@@ -92,3 +95,4 @@ To manipulate a specific worker, you pass the -w option to synctl::
 All of the above is highly experimental and subject to change as Synapse evolves,
 but documenting it here to help folks needing highly scalable Synapses similar
 to the one running matrix.org!
+

jenkins-dendron-haproxy-postgres.sh

@@ -1,23 +0,0 @@
-#!/bin/bash
-
-set -eux
-
-: ${WORKSPACE:="$(pwd)"}
-
-export WORKSPACE
-export PYTHONDONTWRITEBYTECODE=yep
-export SYNAPSE_CACHE_FACTOR=1
-
-export HAPROXY_BIN=/home/haproxy/haproxy-1.6.11/haproxy
-
-./jenkins/prepare_synapse.sh
-./jenkins/clone.sh sytest https://github.com/matrix-org/sytest.git
-./jenkins/clone.sh dendron https://github.com/matrix-org/dendron.git
-./dendron/jenkins/build_dendron.sh
-./sytest/jenkins/prep_sytest_for_postgres.sh
-
-./sytest/jenkins/install_and_run.sh \
-    --python $WORKSPACE/.tox/py27/bin/python \
-    --synapse-directory $WORKSPACE \
-    --dendron $WORKSPACE/dendron/bin/dendron \
-    --haproxy \

jenkins-dendron-postgres.sh

@@ -15,6 +15,11 @@ export SYNAPSE_CACHE_FACTOR=1
 ./sytest/jenkins/prep_sytest_for_postgres.sh
 
 ./sytest/jenkins/install_and_run.sh \
-    --python $WORKSPACE/.tox/py27/bin/python \
     --synapse-directory $WORKSPACE \
     --dendron $WORKSPACE/dendron/bin/dendron \
+    --pusher \
+    --synchrotron \
+    --federation-reader \
+    --client-reader \
+    --appservice \
+    --federation-sender \

jenkins-postgres.sh

@@ -14,5 +14,4 @@ export SYNAPSE_CACHE_FACTOR=1
 ./sytest/jenkins/prep_sytest_for_postgres.sh
 
 ./sytest/jenkins/install_and_run.sh \
-    --python $WORKSPACE/.tox/py27/bin/python \
     --synapse-directory $WORKSPACE \

jenkins-sqlite.sh

@@ -12,5 +12,4 @@ export SYNAPSE_CACHE_FACTOR=1
 ./jenkins/clone.sh sytest https://github.com/matrix-org/sytest.git
 
 ./sytest/jenkins/install_and_run.sh \
-    --python $WORKSPACE/.tox/py27/bin/python \
     --synapse-directory $WORKSPACE \

scripts-dev/federation_client.py

@@ -1,30 +1,10 @@
-#!/usr/bin/env python
-#
-# Copyright 2015, 2016 OpenMarket Ltd
-# Copyright 2017 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from __future__ import print_function
-
-import argparse
 import nacl.signing
 import json
 import base64
 import requests
 import sys
 import srvlookup
-import yaml
+
 
 def encode_base64(input_bytes):
     """Encode bytes as a base64 string without any padding."""
@@ -140,13 +120,11 @@ def get_json(origin_name, origin_key, destination, path):
             origin_name, key, sig,
         )
         authorization_headers.append(bytes(header))
-        print ("Authorization: %s" % header, file=sys.stderr)
-
-    dest = lookup(destination, path)
-    print ("Requesting %s" % dest, file=sys.stderr)
+        sys.stderr.write(header)
+        sys.stderr.write("\n")
 
     result = requests.get(
-        dest,
+        lookup(destination, path),
         headers={"Authorization": authorization_headers[0]},
         verify=False,
     )
@@ -155,66 +133,17 @@ def get_json(origin_name, origin_key, destination, path):
 
 
 def main():
-    parser = argparse.ArgumentParser(
-        description=
-            "Signs and sends a federation request to a matrix homeserver",
-    )
+    origin_name, keyfile, destination, path = sys.argv[1:]
 
-    parser.add_argument(
-        "-N", "--server-name",
-        help="Name to give as the local homeserver. If unspecified, will be "
-             "read from the config file.",
-    )
-
-    parser.add_argument(
-        "-k", "--signing-key-path",
-        help="Path to the file containing the private ed25519 key to sign the "
-             "request with.",
-    )
-
-    parser.add_argument(
-        "-c", "--config",
-        default="homeserver.yaml",
-        help="Path to server config file. Ignored if --server-name and "
-             "--signing-key-path are both given.",
-    )
-
-    parser.add_argument(
-        "-d", "--destination",
-        default="matrix.org",
-        help="name of the remote homeserver. We will do SRV lookups and "
-             "connect appropriately.",
-    )
-
-    parser.add_argument(
-        "path",
-        help="request path. We will add '/_matrix/federation/v1/' to this."
-    )
-
-    args = parser.parse_args()
-
-    if not args.server_name or not args.signing_key_path:
-        read_args_from_config(args)
-
-    with open(args.signing_key_path) as f:
+    with open(keyfile) as f:
         key = read_signing_keys(f)[0]
 
     result = get_json(
-        args.server_name, key, args.destination, "/_matrix/federation/v1/" + args.path
+        origin_name, key, destination, "/_matrix/federation/v1/" + path
     )
 
     json.dump(result, sys.stdout)
-    print ("")
-
-
-def read_args_from_config(args):
-    with open(args.config, 'r') as fh:
-        config = yaml.safe_load(fh)
-        if not args.server_name:
-            args.server_name = config['server_name']
-        if not args.signing_key_path:
-            args.signing_key_path = config['signing_key_path']
-
+    print ""
 
 if __name__ == "__main__":
     main()

scripts-dev/nuke-room-from-db.sh

@@ -9,39 +9,16 @@
 ROOMID="$1"
 
 sqlite3 homeserver.db <<EOF
-DELETE FROM event_forward_extremities WHERE room_id = '$ROOMID';
-DELETE FROM event_backward_extremities WHERE room_id = '$ROOMID';
-DELETE FROM event_edges WHERE room_id = '$ROOMID';
-DELETE FROM room_depth WHERE room_id = '$ROOMID';
-DELETE FROM state_forward_extremities WHERE room_id = '$ROOMID';
-DELETE FROM events WHERE room_id = '$ROOMID';
-DELETE FROM event_json WHERE room_id = '$ROOMID';
-DELETE FROM state_events WHERE room_id = '$ROOMID';
-DELETE FROM current_state_events WHERE room_id = '$ROOMID';
-DELETE FROM room_memberships WHERE room_id = '$ROOMID';
+DELETE FROM context_depth WHERE context = '$ROOMID';
+DELETE FROM current_state WHERE context = '$ROOMID';
 DELETE FROM feedback WHERE room_id = '$ROOMID';
-DELETE FROM topics WHERE room_id = '$ROOMID';
-DELETE FROM room_names WHERE room_id = '$ROOMID';
+DELETE FROM messages WHERE room_id = '$ROOMID';
+DELETE FROM pdu_backward_extremities WHERE context = '$ROOMID';
+DELETE FROM pdu_edges WHERE context = '$ROOMID';
+DELETE FROM pdu_forward_extremities WHERE context = '$ROOMID';
+DELETE FROM pdus WHERE context = '$ROOMID';
+DELETE FROM room_data WHERE room_id = '$ROOMID';
+DELETE FROM room_memberships WHERE room_id = '$ROOMID';
 DELETE FROM rooms WHERE room_id = '$ROOMID';
-DELETE FROM room_hosts WHERE room_id = '$ROOMID';
-DELETE FROM room_aliases WHERE room_id = '$ROOMID';
-DELETE FROM state_groups WHERE room_id = '$ROOMID';
-DELETE FROM state_groups_state WHERE room_id = '$ROOMID';
-DELETE FROM receipts_graph WHERE room_id = '$ROOMID';
-DELETE FROM receipts_linearized WHERE room_id = '$ROOMID';
-DELETE FROM event_search_content WHERE c1room_id = '$ROOMID';
-DELETE FROM guest_access WHERE room_id = '$ROOMID';
-DELETE FROM history_visibility WHERE room_id = '$ROOMID';
-DELETE FROM room_tags WHERE room_id = '$ROOMID';
-DELETE FROM room_tags_revisions WHERE room_id = '$ROOMID';
-DELETE FROM room_account_data WHERE room_id = '$ROOMID';
-DELETE FROM event_push_actions WHERE room_id = '$ROOMID';
-DELETE FROM local_invites WHERE room_id = '$ROOMID';
-DELETE FROM pusher_throttle WHERE room_id = '$ROOMID';
-DELETE FROM event_reports WHERE room_id = '$ROOMID';
-DELETE FROM public_room_list_stream WHERE room_id = '$ROOMID';
-DELETE FROM stream_ordering_to_exterm WHERE room_id = '$ROOMID';
-DELETE FROM event_auth WHERE room_id = '$ROOMID';
-DELETE FROM appservice_room_list WHERE room_id = '$ROOMID';
-VACUUM;
+DELETE FROM state_pdus WHERE context = '$ROOMID';
 EOF

scripts/synapse_port_db

@@ -40,16 +40,6 @@ BOOLEAN_COLUMNS = {
     "presence_list": ["accepted"],
     "presence_stream": ["currently_active"],
     "public_room_list_stream": ["visibility"],
-    "device_lists_outbound_pokes": ["sent"],
-    "users_who_share_rooms": ["share_private"],
-    "groups": ["is_public"],
-    "group_rooms": ["is_public"],
-    "group_users": ["is_public", "is_admin"],
-    "group_summary_rooms": ["is_public"],
-    "group_room_categories": ["is_public"],
-    "group_summary_users": ["is_public"],
-    "group_roles": ["is_public"],
-    "local_group_membership": ["is_publicised", "is_admin"],
 }
 
 
@@ -120,7 +110,6 @@ class Store(object):
 
     _simple_update_one = SQLBaseStore.__dict__["_simple_update_one"]
     _simple_update_one_txn = SQLBaseStore.__dict__["_simple_update_one_txn"]
-    _simple_update_txn = SQLBaseStore.__dict__["_simple_update_txn"]
 
     def runInteraction(self, desc, func, *args, **kwargs):
         def r(conn):
@@ -131,7 +120,7 @@ class Store(object):
                     try:
                         txn = conn.cursor()
                         return func(
-                            LoggingTransaction(txn, desc, self.database_engine, [], []),
+                            LoggingTransaction(txn, desc, self.database_engine, []),
                             *args, **kwargs
                         )
                     except self.database_engine.module.DatabaseError as e:
@@ -261,25 +250,6 @@ class Porter(object):
             )
             return
 
-        if table in (
-            "user_directory", "user_directory_search", "users_who_share_rooms",
-            "users_in_pubic_room",
-        ):
-            # We don't port these tables, as they're a faff and we can regenreate
-            # them anyway.
-            self.progress.update(table, table_size)  # Mark table as done
-            return
-
-        if table == "user_directory_stream_pos":
-            # We need to make sure there is a single row, `(X, null), as that is
-            # what synapse expects to be there.
-            yield self.postgres_store._simple_insert(
-                table=table,
-                values={"stream_id": None},
-            )
-            self.progress.update(table, table_size)  # Mark table as done
-            return
-
         forward_select = (
             "SELECT rowid, * FROM %s WHERE rowid >= ? ORDER BY rowid LIMIT ?"
             % (table,)
@@ -327,7 +297,7 @@ class Porter(object):
                     backward_chunk = min(row[0] for row in brows) - 1
 
                 rows = frows + brows
-                rows = self._convert_rows(table, headers, rows)
+                self._convert_rows(table, headers, rows)
 
                 def insert(txn):
                     self.postgres_store.insert_many_txn(
@@ -385,13 +355,10 @@ class Porter(object):
                         " VALUES (?,?,?,?,to_tsvector('english', ?),?,?)"
                     )
 
-                    rows_dict = []
-                    for row in rows:
-                        d = dict(zip(headers, row))
-                        if "\0" in d['value']:
-                            logger.warn('dropping search row %s', d)
-                        else:
-                            rows_dict.append(d)
+                    rows_dict = [
+                        dict(zip(headers, row))
+                        for row in rows
+                    ]
 
                     txn.executemany(sql, [
                         (
@@ -479,7 +446,9 @@ class Porter(object):
 
             postgres_tables = yield self.postgres_store._simple_select_onecol(
                 table="information_schema.tables",
-                keyvalues={},
+                keyvalues={
+                    "table_schema": "public",
+                },
                 retcol="distinct table_name",
             )
 
@@ -563,29 +532,17 @@ class Porter(object):
             i for i, h in enumerate(headers) if h in bool_col_names
         ]
 
-        class BadValueException(Exception):
-            pass
-
         def conv(j, col):
             if j in bool_cols:
                 return bool(col)
-            elif isinstance(col, basestring) and "\0" in col:
-                logger.warn("DROPPING ROW: NUL value in table %s col %s: %r", table, headers[j], col)
-                raise BadValueException();
             return col
 
-        outrows = []
         for i, row in enumerate(rows):
-            try:
-                outrows.append(tuple(
-                    conv(j, col)
-                    for j, col in enumerate(row)
-                    if j > 0
-                ))
-            except BadValueException:
-                pass
-
-        return outrows
+            rows[i] = tuple(
+                conv(j, col)
+                for j, col in enumerate(row)
+                if j > 0
+            )
 
     @defer.inlineCallbacks
     def _setup_sent_transactions(self):
@@ -613,7 +570,7 @@ class Porter(object):
             "select", r,
         )
 
-        rows = self._convert_rows("sent_transactions", headers, rows)
+        self._convert_rows("sent_transactions", headers, rows)
 
         inserted_rows = len(rows)
         if inserted_rows:

synapse/__init__.py

@@ -16,4 +16,4 @@
 """ This is a reference implementation of a Matrix home server.
 """
 
-__version__ = "0.25.0"
+__version__ = "0.19.2"

synapse/api/auth.py

@@ -23,8 +23,7 @@ from synapse import event_auth
 from synapse.api.constants import EventTypes, Membership, JoinRules
 from synapse.api.errors import AuthError, Codes
 from synapse.types import UserID
-from synapse.util.caches import register_cache, CACHE_SIZE_FACTOR
-from synapse.util.caches.lrucache import LruCache
+from synapse.util.logcontext import preserve_context_over_fn
 from synapse.util.metrics import Measure
 
 logger = logging.getLogger(__name__)
@@ -40,10 +39,6 @@ AuthEventTypes = (
 GUEST_DEVICE_ID = "guest_device"
 
 
-class _InvalidMacaroonException(Exception):
-    pass
-
-
 class Auth(object):
     """
     FIXME: This class contains a mix of functions for authenticating users
@@ -56,9 +51,6 @@ class Auth(object):
         self.state = hs.get_state_handler()
         self.TOKEN_NOT_FOUND_HTTP_STATUS = 401
 
-        self.token_cache = LruCache(CACHE_SIZE_FACTOR * 10000)
-        register_cache("token_cache", self.token_cache)
-
     @defer.inlineCallbacks
     def check_from_context(self, event, context, do_sig_check=True):
         auth_events_ids = yield self.compute_auth_events(
@@ -152,8 +144,17 @@ class Auth(object):
     @defer.inlineCallbacks
     def check_host_in_room(self, room_id, host):
         with Measure(self.clock, "check_host_in_room"):
-            latest_event_ids = yield self.store.is_host_joined(room_id, host)
-            defer.returnValue(latest_event_ids)
+            latest_event_ids = yield self.store.get_latest_event_ids_in_room(room_id)
+
+            logger.debug("calling resolve_state_groups from check_host_in_room")
+            entry = yield self.state.resolve_state_groups(
+                room_id, latest_event_ids
+            )
+
+            ret = yield self.store.is_host_joined(
+                room_id, host, entry.state_group, entry.state
+            )
+            defer.returnValue(ret)
 
     def _check_joined_room(self, member, user_id, room_id):
         if not member or member.membership != Membership.JOIN:
@@ -208,8 +209,9 @@ class Auth(object):
                 default=[""]
             )[0]
             if user and access_token and ip_addr:
-                self.store.insert_client_ip(
-                    user_id=user.to_string(),
+                preserve_context_over_fn(
+                    self.store.insert_client_ip,
+                    user=user,
                     access_token=access_token,
                     ip=ip_addr,
                     user_agent=user_agent,
@@ -275,8 +277,8 @@ class Auth(object):
             AuthError if no user by that token exists or the token is invalid.
         """
         try:
-            user_id, guest = self._parse_and_validate_macaroon(token, rights)
-        except _InvalidMacaroonException:
+            macaroon = pymacaroons.Macaroon.deserialize(token)
+        except Exception:  # deserialize can throw more-or-less anything
             # doesn't look like a macaroon: treat it as an opaque token which
             # must be in the database.
             # TODO: it would be nice to get rid of this, but apparently some
@@ -285,8 +287,19 @@ class Auth(object):
             defer.returnValue(r)
 
         try:
+            user_id = self.get_user_id_from_macaroon(macaroon)
             user = UserID.from_string(user_id)
 
+            self.validate_macaroon(
+                macaroon, rights, self.hs.config.expire_access_token,
+                user_id=user_id,
+            )
+
+            guest = False
+            for caveat in macaroon.caveats:
+                if caveat.caveat_id == "guest = true":
+                    guest = True
+
             if guest:
                 # Guest access tokens are not stored in the database (there can
                 # only be one access token per guest, anyway).
@@ -358,55 +371,6 @@ class Auth(object):
                 errcode=Codes.UNKNOWN_TOKEN
             )
 
-    def _parse_and_validate_macaroon(self, token, rights="access"):
-        """Takes a macaroon and tries to parse and validate it. This is cached
-        if and only if rights == access and there isn't an expiry.
-
-        On invalid macaroon raises _InvalidMacaroonException
-
-        Returns:
-            (user_id, is_guest)
-        """
-        if rights == "access":
-            cached = self.token_cache.get(token, None)
-            if cached:
-                return cached
-
-        try:
-            macaroon = pymacaroons.Macaroon.deserialize(token)
-        except Exception:  # deserialize can throw more-or-less anything
-            # doesn't look like a macaroon: treat it as an opaque token which
-            # must be in the database.
-            # TODO: it would be nice to get rid of this, but apparently some
-            # people use access tokens which aren't macaroons
-            raise _InvalidMacaroonException()
-
-        try:
-            user_id = self.get_user_id_from_macaroon(macaroon)
-
-            has_expiry = False
-            guest = False
-            for caveat in macaroon.caveats:
-                if caveat.caveat_id.startswith("time "):
-                    has_expiry = True
-                elif caveat.caveat_id == "guest = true":
-                    guest = True
-
-            self.validate_macaroon(
-                macaroon, rights, self.hs.config.expire_access_token,
-                user_id=user_id,
-            )
-        except (pymacaroons.exceptions.MacaroonException, TypeError, ValueError):
-            raise AuthError(
-                self.TOKEN_NOT_FOUND_HTTP_STATUS, "Invalid macaroon passed.",
-                errcode=Codes.UNKNOWN_TOKEN
-            )
-
-        if not has_expiry and rights == "access":
-            self.token_cache[token] = (user_id, guest)
-
-        return user_id, guest
-
     def get_user_id_from_macaroon(self, macaroon):
         """Retrieve the user_id given by the caveats on the macaroon.
 
@@ -519,14 +483,6 @@ class Auth(object):
             )
 
     def is_server_admin(self, user):
-        """ Check if the given user is a local server admin.
-
-        Args:
-            user (str): mxid of user to check
-
-        Returns:
-            bool: True if the user is an admin
-        """
         return self.store.is_server_admin(user)
 
     @defer.inlineCallbacks

synapse/api/constants.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2017 Vector Creations Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -45,7 +44,6 @@ class JoinRules(object):
 class LoginType(object):
     PASSWORD = u"m.login.password"
     EMAIL_IDENTITY = u"m.login.email.identity"
-    MSISDN = u"m.login.msisdn"
     RECAPTCHA = u"m.login.recaptcha"
     DUMMY = u"m.login.dummy"
 

synapse/api/errors.py

@@ -15,7 +15,6 @@
 
 """Contains exceptions and error codes."""
 
-import json
 import logging
 
 logger = logging.getLogger(__name__)
@@ -51,46 +50,27 @@ class Codes(object):
 
 
 class CodeMessageException(RuntimeError):
-    """An exception with integer code and message string attributes.
+    """An exception with integer code and message string attributes."""
 
-    Attributes:
-        code (int): HTTP error code
-        msg (str): string describing the error
-    """
     def __init__(self, code, msg):
         super(CodeMessageException, self).__init__("%d: %s" % (code, msg))
         self.code = code
         self.msg = msg
+        self.response_code_message = None
 
     def error_dict(self):
         return cs_error(self.msg)
 
 
-class MatrixCodeMessageException(CodeMessageException):
-    """An error from a general matrix endpoint, eg. from a proxied Matrix API call.
-
-    Attributes:
-        errcode (str): Matrix error code e.g 'M_FORBIDDEN'
-    """
-    def __init__(self, code, msg, errcode=Codes.UNKNOWN):
-        super(MatrixCodeMessageException, self).__init__(code, msg)
-        self.errcode = errcode
-
-
 class SynapseError(CodeMessageException):
-    """A base exception type for matrix errors which have an errcode and error
-    message (as well as an HTTP status code).
-
-    Attributes:
-        errcode (str): Matrix error code e.g 'M_FORBIDDEN'
-    """
+    """A base error which can be caught for all synapse events."""
     def __init__(self, code, msg, errcode=Codes.UNKNOWN):
         """Constructs a synapse error.
 
         Args:
             code (int): The integer error code (an HTTP response code)
             msg (str): The human-readable error message.
-            errcode (str): The matrix error code e.g 'M_FORBIDDEN'
+            err (str): The error code e.g 'M_FORBIDDEN'
         """
         super(SynapseError, self).__init__(code, msg)
         self.errcode = errcode
@@ -101,39 +81,6 @@ class SynapseError(CodeMessageException):
             self.errcode,
         )
 
-    @classmethod
-    def from_http_response_exception(cls, err):
-        """Make a SynapseError based on an HTTPResponseException
-
-        This is useful when a proxied request has failed, and we need to
-        decide how to map the failure onto a matrix error to send back to the
-        client.
-
-        An attempt is made to parse the body of the http response as a matrix
-        error. If that succeeds, the errcode and error message from the body
-        are used as the errcode and error message in the new synapse error.
-
-        Otherwise, the errcode is set to M_UNKNOWN, and the error message is
-        set to the reason code from the HTTP response.
-
-        Args:
-            err (HttpResponseException):
-
-        Returns:
-            SynapseError:
-        """
-        # try to parse the body as json, to get better errcode/msg, but
-        # default to M_UNKNOWN with the HTTP status as the error text
-        try:
-            j = json.loads(err.response)
-        except ValueError:
-            j = {}
-        errcode = j.get('errcode', Codes.UNKNOWN)
-        errmsg = j.get('error', err.msg)
-
-        res = SynapseError(err.code, errmsg, errcode)
-        return res
-
 
 class RegistrationError(SynapseError):
     """An error raised when a registration event fails."""
@@ -159,11 +106,13 @@ class UnrecognizedRequestError(SynapseError):
 
 class NotFoundError(SynapseError):
     """An error indicating we can't find the thing you asked for"""
-    def __init__(self, msg="Not found", errcode=Codes.NOT_FOUND):
+    def __init__(self, *args, **kwargs):
+        if "errcode" not in kwargs:
+            kwargs["errcode"] = Codes.NOT_FOUND
         super(NotFoundError, self).__init__(
             404,
-            msg,
-            errcode=errcode
+            "Not found",
+            **kwargs
         )
 
 
@@ -224,6 +173,7 @@ class LimitExceededError(SynapseError):
                  errcode=Codes.LIMIT_EXCEEDED):
         super(LimitExceededError, self).__init__(code, msg, errcode)
         self.retry_after_ms = retry_after_ms
+        self.response_code_message = "Too Many Requests"
 
     def error_dict(self):
         return cs_error(
@@ -293,19 +243,6 @@ class FederationError(RuntimeError):
 
 
 class HttpResponseException(CodeMessageException):
-    """
-    Represents an HTTP-level failure of an outbound request
-
-    Attributes:
-        response (str): body of response
-    """
     def __init__(self, code, msg, response):
-        """
-
-        Args:
-            code (int): HTTP status code
-            msg (str): reason phrase from HTTP response status line
-            response (str): body of response
-        """
-        super(HttpResponseException, self).__init__(code, msg)
         self.response = response
+        super(HttpResponseException, self).__init__(code, msg)

synapse/api/filtering.py

@@ -13,174 +13,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 from synapse.api.errors import SynapseError
-from synapse.storage.presence import UserPresenceState
 from synapse.types import UserID, RoomID
+
 from twisted.internet import defer
 
 import ujson as json
-import jsonschema
-from jsonschema import FormatChecker
-
-FILTER_SCHEMA = {
-    "additionalProperties": False,
-    "type": "object",
-    "properties": {
-        "limit": {
-            "type": "number"
-        },
-        "senders": {
-            "$ref": "#/definitions/user_id_array"
-        },
-        "not_senders": {
-            "$ref": "#/definitions/user_id_array"
-        },
-        # TODO: We don't limit event type values but we probably should...
-        # check types are valid event types
-        "types": {
-            "type": "array",
-            "items": {
-                "type": "string"
-            }
-        },
-        "not_types": {
-            "type": "array",
-            "items": {
-                "type": "string"
-            }
-        }
-    }
-}
-
-ROOM_FILTER_SCHEMA = {
-    "additionalProperties": False,
-    "type": "object",
-    "properties": {
-        "not_rooms": {
-            "$ref": "#/definitions/room_id_array"
-        },
-        "rooms": {
-            "$ref": "#/definitions/room_id_array"
-        },
-        "ephemeral": {
-            "$ref": "#/definitions/room_event_filter"
-        },
-        "include_leave": {
-            "type": "boolean"
-        },
-        "state": {
-            "$ref": "#/definitions/room_event_filter"
-        },
-        "timeline": {
-            "$ref": "#/definitions/room_event_filter"
-        },
-        "account_data": {
-            "$ref": "#/definitions/room_event_filter"
-        },
-    }
-}
-
-ROOM_EVENT_FILTER_SCHEMA = {
-    "additionalProperties": False,
-    "type": "object",
-    "properties": {
-        "limit": {
-            "type": "number"
-        },
-        "senders": {
-            "$ref": "#/definitions/user_id_array"
-        },
-        "not_senders": {
-            "$ref": "#/definitions/user_id_array"
-        },
-        "types": {
-            "type": "array",
-            "items": {
-                "type": "string"
-            }
-        },
-        "not_types": {
-            "type": "array",
-            "items": {
-                "type": "string"
-            }
-        },
-        "rooms": {
-            "$ref": "#/definitions/room_id_array"
-        },
-        "not_rooms": {
-            "$ref": "#/definitions/room_id_array"
-        },
-        "contains_url": {
-            "type": "boolean"
-        }
-    }
-}
-
-USER_ID_ARRAY_SCHEMA = {
-    "type": "array",
-    "items": {
-        "type": "string",
-        "format": "matrix_user_id"
-    }
-}
-
-ROOM_ID_ARRAY_SCHEMA = {
-    "type": "array",
-    "items": {
-        "type": "string",
-        "format": "matrix_room_id"
-    }
-}
-
-USER_FILTER_SCHEMA = {
-    "$schema": "http://json-schema.org/draft-04/schema#",
-    "description": "schema for a Sync filter",
-    "type": "object",
-    "definitions": {
-        "room_id_array": ROOM_ID_ARRAY_SCHEMA,
-        "user_id_array": USER_ID_ARRAY_SCHEMA,
-        "filter": FILTER_SCHEMA,
-        "room_filter": ROOM_FILTER_SCHEMA,
-        "room_event_filter": ROOM_EVENT_FILTER_SCHEMA
-    },
-    "properties": {
-        "presence": {
-            "$ref": "#/definitions/filter"
-        },
-        "account_data": {
-            "$ref": "#/definitions/filter"
-        },
-        "room": {
-            "$ref": "#/definitions/room_filter"
-        },
-        "event_format": {
-            "type": "string",
-            "enum": ["client", "federation"]
-        },
-        "event_fields": {
-            "type": "array",
-            "items": {
-                "type": "string",
-                # Don't allow '\\' in event field filters. This makes matching
-                # events a lot easier as we can then use a negative lookbehind
-                # assertion to split '\.' If we allowed \\ then it would
-                # incorrectly split '\\.' See synapse.events.utils.serialize_event
-                "pattern": "^((?!\\\).)*$"
-            }
-        }
-    },
-    "additionalProperties": False
-}
-
-
-@FormatChecker.cls_checks('matrix_room_id')
-def matrix_room_id_validator(room_id_str):
-    return RoomID.from_string(room_id_str)
-
-
-@FormatChecker.cls_checks('matrix_user_id')
-def matrix_user_id_validator(user_id_str):
-    return UserID.from_string(user_id_str)
 
 
 class Filtering(object):
@@ -215,11 +52,98 @@ class Filtering(object):
         # NB: Filters are the complete json blobs. "Definitions" are an
         # individual top-level key e.g. public_user_data. Filters are made of
         # many definitions.
-        try:
-            jsonschema.validate(user_filter_json, USER_FILTER_SCHEMA,
-                                format_checker=FormatChecker())
-        except jsonschema.ValidationError as e:
-            raise SynapseError(400, e.message)
+
+        top_level_definitions = [
+            "presence", "account_data"
+        ]
+
+        room_level_definitions = [
+            "state", "timeline", "ephemeral", "account_data"
+        ]
+
+        for key in top_level_definitions:
+            if key in user_filter_json:
+                self._check_definition(user_filter_json[key])
+
+        if "room" in user_filter_json:
+            self._check_definition_room_lists(user_filter_json["room"])
+            for key in room_level_definitions:
+                if key in user_filter_json["room"]:
+                    self._check_definition(user_filter_json["room"][key])
+
+        if "event_fields" in user_filter_json:
+            if type(user_filter_json["event_fields"]) != list:
+                raise SynapseError(400, "event_fields must be a list of strings")
+            for field in user_filter_json["event_fields"]:
+                if not isinstance(field, basestring):
+                    raise SynapseError(400, "Event field must be a string")
+                # Don't allow '\\' in event field filters. This makes matching
+                # events a lot easier as we can then use a negative lookbehind
+                # assertion to split '\.' If we allowed \\ then it would
+                # incorrectly split '\\.' See synapse.events.utils.serialize_event
+                if r'\\' in field:
+                    raise SynapseError(
+                        400, r'The escape character \ cannot itself be escaped'
+                    )
+
+    def _check_definition_room_lists(self, definition):
+        """Check that "rooms" and "not_rooms" are lists of room ids if they
+        are present
+
+        Args:
+            definition(dict): The filter definition
+        Raises:
+            SynapseError: If there was a problem with this definition.
+        """
+        # check rooms are valid room IDs
+        room_id_keys = ["rooms", "not_rooms"]
+        for key in room_id_keys:
+            if key in definition:
+                if type(definition[key]) != list:
+                    raise SynapseError(400, "Expected %s to be a list." % key)
+                for room_id in definition[key]:
+                    RoomID.from_string(room_id)
+
+    def _check_definition(self, definition):
+        """Check if the provided definition is valid.
+
+        This inspects not only the types but also the values to make sure they
+        make sense.
+
+        Args:
+            definition(dict): The filter definition
+        Raises:
+            SynapseError: If there was a problem with this definition.
+        """
+        # NB: Filters are the complete json blobs. "Definitions" are an
+        # individual top-level key e.g. public_user_data. Filters are made of
+        # many definitions.
+        if type(definition) != dict:
+            raise SynapseError(
+                400, "Expected JSON object, not %s" % (definition,)
+            )
+
+        self._check_definition_room_lists(definition)
+
+        # check senders are valid user IDs
+        user_id_keys = ["senders", "not_senders"]
+        for key in user_id_keys:
+            if key in definition:
+                if type(definition[key]) != list:
+                    raise SynapseError(400, "Expected %s to be a list." % key)
+                for user_id in definition[key]:
+                    UserID.from_string(user_id)
+
+        # TODO: We don't limit event type values but we probably should...
+        # check types are valid event types
+        event_keys = ["types", "not_types"]
+        for key in event_keys:
+            if key in definition:
+                if type(definition[key]) != list:
+                    raise SynapseError(400, "Expected %s to be a list." % key)
+                for event_type in definition[key]:
+                    if not isinstance(event_type, basestring):
+                        raise SynapseError(400, "Event type should be a string")
 
 
 class FilterCollection(object):
@@ -329,35 +253,19 @@ class Filter(object):
         Returns:
             bool: True if the event matches
         """
-        # We usually get the full "events" as dictionaries coming through,
-        # except for presence which actually gets passed around as its own
-        # namedtuple type.
-        if isinstance(event, UserPresenceState):
-            sender = event.user_id
-            room_id = None
-            ev_type = "m.presence"
-            is_url = False
-        else:
-            sender = event.get("sender", None)
-            if not sender:
-                # Presence events had their 'sender' in content.user_id, but are
-                # now handled above. We don't know if anything else uses this
-                # form. TODO: Check this and probably remove it.
-                content = event.get("content")
-                # account_data has been allowed to have non-dict content, so
-                # check type first
-                if isinstance(content, dict):
-                    sender = content.get("user_id")
-
-            room_id = event.get("room_id", None)
-            ev_type = event.get("type", None)
-            is_url = "url" in event.get("content", {})
+        sender = event.get("sender", None)
+        if not sender:
+            # Presence events have their 'sender' in content.user_id
+            content = event.get("content")
+            # account_data has been allowed to have non-dict content, so check type first
+            if isinstance(content, dict):
+                sender = content.get("user_id")
 
         return self.check_fields(
-            room_id,
+            event.get("room_id", None),
             sender,
-            ev_type,
-            is_url,
+            event.get("type", None),
+            "url" in event.get("content", {})
         )
 
     def check_fields(self, room_id, sender, event_type, contains_url):

synapse/app/_base.py

@@ -1,122 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2017 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import gc
-import logging
-import sys
-
-try:
-    import affinity
-except Exception:
-    affinity = None
-
-from daemonize import Daemonize
-from synapse.util import PreserveLoggingContext
-from synapse.util.rlimit import change_resource_limit
-from twisted.internet import reactor
-
-
-def start_worker_reactor(appname, config):
-    """ Run the reactor in the main process
-
-    Daemonizes if necessary, and then configures some resources, before starting
-    the reactor. Pulls configuration from the 'worker' settings in 'config'.
-
-    Args:
-        appname (str): application name which will be sent to syslog
-        config (synapse.config.Config): config object
-    """
-
-    logger = logging.getLogger(config.worker_app)
-
-    start_reactor(
-        appname,
-        config.soft_file_limit,
-        config.gc_thresholds,
-        config.worker_pid_file,
-        config.worker_daemonize,
-        config.worker_cpu_affinity,
-        logger,
-    )
-
-
-def start_reactor(
-        appname,
-        soft_file_limit,
-        gc_thresholds,
-        pid_file,
-        daemonize,
-        cpu_affinity,
-        logger,
-):
-    """ Run the reactor in the main process
-
-    Daemonizes if necessary, and then configures some resources, before starting
-    the reactor
-
-    Args:
-        appname (str): application name which will be sent to syslog
-        soft_file_limit (int):
-        gc_thresholds:
-        pid_file (str): name of pid file to write to if daemonize is True
-        daemonize (bool): true to run the reactor in a background process
-        cpu_affinity (int|None): cpu affinity mask
-        logger (logging.Logger): logger instance to pass to Daemonize
-    """
-
-    def run():
-        # make sure that we run the reactor with the sentinel log context,
-        # otherwise other PreserveLoggingContext instances will get confused
-        # and complain when they see the logcontext arbitrarily swapping
-        # between the sentinel and `run` logcontexts.
-        with PreserveLoggingContext():
-            logger.info("Running")
-            if cpu_affinity is not None:
-                if not affinity:
-                    quit_with_error(
-                        "Missing package 'affinity' required for cpu_affinity\n"
-                        "option\n\n"
-                        "Install by running:\n\n"
-                        "   pip install affinity\n\n"
-                    )
-                logger.info("Setting CPU affinity to %s" % cpu_affinity)
-                affinity.set_process_affinity_mask(0, cpu_affinity)
-            change_resource_limit(soft_file_limit)
-            if gc_thresholds:
-                gc.set_threshold(*gc_thresholds)
-            reactor.run()
-
-    if daemonize:
-        daemon = Daemonize(
-            app=appname,
-            pid=pid_file,
-            action=run,
-            auto_close_fds=False,
-            verbose=True,
-            logger=logger,
-        )
-        daemon.start()
-    else:
-        run()
-
-
-def quit_with_error(error_string):
-    message_lines = error_string.split("\n")
-    line_length = max([len(l) for l in message_lines if len(l) < 80]) + 2
-    sys.stderr.write("*" * line_length + '\n')
-    for line in message_lines:
-        sys.stderr.write(" %s\n" % (line.rstrip(),))
-    sys.stderr.write("*" * line_length + '\n')
-    sys.exit(1)

synapse/app/appservice.py

@@ -13,31 +13,38 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
-import sys
 
 import synapse
-from synapse import events
-from synapse.app import _base
+
+from synapse.server import HomeServer
 from synapse.config._base import ConfigError
-from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
+from synapse.config.homeserver import HomeServerConfig
 from synapse.http.site import SynapseSite
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
-from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
+from synapse.metrics.resource import MetricsResource, METRICS_PREFIX
 from synapse.replication.slave.storage.directory import DirectoryStore
 from synapse.replication.slave.storage.events import SlavedEventStore
+from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
 from synapse.replication.slave.storage.registration import SlavedRegistrationStore
-from synapse.replication.tcp.client import ReplicationClientHandler
-from synapse.server import HomeServer
 from synapse.storage.engines import create_engine
+from synapse.util.async import sleep
 from synapse.util.httpresourcetree import create_resource_tree
-from synapse.util.logcontext import LoggingContext, preserve_fn
+from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
+from synapse.util.rlimit import change_resource_limit
 from synapse.util.versionstring import get_version_string
-from twisted.internet import reactor
+
+from synapse import events
+
+from twisted.internet import reactor, defer
 from twisted.web.resource import Resource
 
+from daemonize import Daemonize
+
+import sys
+import logging
+import gc
+
 logger = logging.getLogger("synapse.app.appservice")
 
 
@@ -113,25 +120,30 @@ class AppserviceServer(HomeServer):
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 
-        self.get_tcp_replication().start_replication(self)
+    @defer.inlineCallbacks
+    def replicate(self):
+        http_client = self.get_simple_http_client()
+        store = self.get_datastore()
+        replication_url = self.config.worker_replication_url
+        appservice_handler = self.get_application_service_handler()
 
-    def build_tcp_replication(self):
-        return ASReplicationHandler(self)
+        @defer.inlineCallbacks
+        def replicate(results):
+            stream = results.get("events")
+            if stream:
+                max_stream_id = stream["position"]
+                yield appservice_handler.notify_interested_services(max_stream_id)
 
-
-class ASReplicationHandler(ReplicationClientHandler):
-    def __init__(self, hs):
-        super(ASReplicationHandler, self).__init__(hs.get_datastore())
-        self.appservice_handler = hs.get_application_service_handler()
-
-    def on_rdata(self, stream_name, token, rows):
-        super(ASReplicationHandler, self).on_rdata(stream_name, token, rows)
-
-        if stream_name == "events":
-            max_stream_id = self.store.get_room_max_stream_ordering()
-            preserve_fn(
-                self.appservice_handler.notify_interested_services
-            )(max_stream_id)
+        while True:
+            try:
+                args = store.stream_positions()
+                args["timeout"] = 30000
+                result = yield http_client.get_json(replication_url, args=args)
+                yield store.process_replication(result)
+                replicate(result)
+            except:
+                logger.exception("Error replicating from %r", replication_url)
+                yield sleep(30)
 
 
 def start(config_options):
@@ -145,7 +157,7 @@ def start(config_options):
 
     assert config.worker_app == "synapse.app.appservice"
 
-    setup_logging(config, use_worker_options=True)
+    setup_logging(config.worker_log_config, config.worker_log_file)
 
     events.USE_FROZEN_DICTS = config.use_frozen_dicts
 
@@ -174,13 +186,33 @@ def start(config_options):
     ps.setup()
     ps.start_listening(config.worker_listeners)
 
+    def run():
+        with LoggingContext("run"):
+            logger.info("Running")
+            change_resource_limit(config.soft_file_limit)
+            if config.gc_thresholds:
+                gc.set_threshold(*config.gc_thresholds)
+            reactor.run()
+
     def start():
+        ps.replicate()
         ps.get_datastore().start_profiling()
         ps.get_state_handler().start_caching()
 
     reactor.callWhenRunning(start)
 
-    _base.start_worker_reactor("synapse-appservice", config)
+    if config.worker_daemonize:
+        daemon = Daemonize(
+            app="synapse-appservice",
+            pid=config.worker_pid_file,
+            action=run,
+            auto_close_fds=False,
+            verbose=True,
+            logger=logger,
+        )
+        daemon.start()
+    else:
+        run()
 
 
 if __name__ == '__main__':

synapse/app/client_reader.py

@@ -13,39 +13,46 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
-import sys
 
 import synapse
-from synapse import events
-from synapse.app import _base
+
 from synapse.config._base import ConfigError
 from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
-from synapse.crypto import context_factory
-from synapse.http.server import JsonResource
 from synapse.http.site import SynapseSite
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
+from synapse.http.server import JsonResource
+from synapse.metrics.resource import MetricsResource, METRICS_PREFIX
 from synapse.replication.slave.storage._base import BaseSlavedStore
 from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
-from synapse.replication.slave.storage.client_ips import SlavedClientIpStore
-from synapse.replication.slave.storage.directory import DirectoryStore
 from synapse.replication.slave.storage.events import SlavedEventStore
 from synapse.replication.slave.storage.keys import SlavedKeyStore
-from synapse.replication.slave.storage.registration import SlavedRegistrationStore
 from synapse.replication.slave.storage.room import RoomStore
-from synapse.replication.slave.storage.transactions import TransactionStore
-from synapse.replication.tcp.client import ReplicationClientHandler
+from synapse.replication.slave.storage.directory import DirectoryStore
+from synapse.replication.slave.storage.registration import SlavedRegistrationStore
 from synapse.rest.client.v1.room import PublicRoomListRestServlet
 from synapse.server import HomeServer
+from synapse.storage.client_ips import ClientIpStore
 from synapse.storage.engines import create_engine
+from synapse.util.async import sleep
 from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
+from synapse.util.rlimit import change_resource_limit
 from synapse.util.versionstring import get_version_string
-from twisted.internet import reactor
+from synapse.crypto import context_factory
+
+from synapse import events
+
+
+from twisted.internet import reactor, defer
 from twisted.web.resource import Resource
 
+from daemonize import Daemonize
+
+import sys
+import logging
+import gc
+
 logger = logging.getLogger("synapse.app.client_reader")
 
 
@@ -56,9 +63,8 @@ class ClientReaderSlavedStore(
     DirectoryStore,
     SlavedApplicationServiceStore,
     SlavedRegistrationStore,
-    TransactionStore,
-    SlavedClientIpStore,
     BaseSlavedStore,
+    ClientIpStore,  # After BaseSlavedStore because the constructor is different
 ):
     pass
 
@@ -137,10 +143,21 @@ class ClientReaderServer(HomeServer):
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 
-        self.get_tcp_replication().start_replication(self)
+    @defer.inlineCallbacks
+    def replicate(self):
+        http_client = self.get_simple_http_client()
+        store = self.get_datastore()
+        replication_url = self.config.worker_replication_url
 
-    def build_tcp_replication(self):
-        return ReplicationClientHandler(self.get_datastore())
+        while True:
+            try:
+                args = store.stream_positions()
+                args["timeout"] = 30000
+                result = yield http_client.get_json(replication_url, args=args)
+                yield store.process_replication(result)
+            except:
+                logger.exception("Error replicating from %r", replication_url)
+                yield sleep(5)
 
 
 def start(config_options):
@@ -154,7 +171,7 @@ def start(config_options):
 
     assert config.worker_app == "synapse.app.client_reader"
 
-    setup_logging(config, use_worker_options=True)
+    setup_logging(config.worker_log_config, config.worker_log_file)
 
     events.USE_FROZEN_DICTS = config.use_frozen_dicts
 
@@ -175,13 +192,33 @@ def start(config_options):
     ss.get_handlers()
     ss.start_listening(config.worker_listeners)
 
+    def run():
+        with LoggingContext("run"):
+            logger.info("Running")
+            change_resource_limit(config.soft_file_limit)
+            if config.gc_thresholds:
+                gc.set_threshold(*config.gc_thresholds)
+            reactor.run()
+
     def start():
         ss.get_state_handler().start_caching()
         ss.get_datastore().start_profiling()
+        ss.replicate()
 
     reactor.callWhenRunning(start)
 
-    _base.start_worker_reactor("synapse-client-reader", config)
+    if config.worker_daemonize:
+        daemon = Daemonize(
+            app="synapse-client-reader",
+            pid=config.worker_pid_file,
+            action=run,
+            auto_close_fds=False,
+            verbose=True,
+            logger=logger,
+        )
+        daemon.start()
+    else:
+        run()
 
 
 if __name__ == '__main__':

synapse/app/federation_reader.py

@@ -13,36 +13,44 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
-import sys
 
 import synapse
-from synapse import events
-from synapse.api.urls import FEDERATION_PREFIX
-from synapse.app import _base
+
 from synapse.config._base import ConfigError
 from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
-from synapse.crypto import context_factory
-from synapse.federation.transport.server import TransportLayerServer
 from synapse.http.site import SynapseSite
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
+from synapse.metrics.resource import MetricsResource, METRICS_PREFIX
 from synapse.replication.slave.storage._base import BaseSlavedStore
-from synapse.replication.slave.storage.directory import DirectoryStore
 from synapse.replication.slave.storage.events import SlavedEventStore
 from synapse.replication.slave.storage.keys import SlavedKeyStore
 from synapse.replication.slave.storage.room import RoomStore
 from synapse.replication.slave.storage.transactions import TransactionStore
-from synapse.replication.tcp.client import ReplicationClientHandler
+from synapse.replication.slave.storage.directory import DirectoryStore
 from synapse.server import HomeServer
 from synapse.storage.engines import create_engine
+from synapse.util.async import sleep
 from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
+from synapse.util.rlimit import change_resource_limit
 from synapse.util.versionstring import get_version_string
-from twisted.internet import reactor
+from synapse.api.urls import FEDERATION_PREFIX
+from synapse.federation.transport.server import TransportLayerServer
+from synapse.crypto import context_factory
+
+from synapse import events
+
+
+from twisted.internet import reactor, defer
 from twisted.web.resource import Resource
 
+from daemonize import Daemonize
+
+import sys
+import logging
+import gc
+
 logger = logging.getLogger("synapse.app.federation_reader")
 
 
@@ -126,10 +134,21 @@ class FederationReaderServer(HomeServer):
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 
-        self.get_tcp_replication().start_replication(self)
+    @defer.inlineCallbacks
+    def replicate(self):
+        http_client = self.get_simple_http_client()
+        store = self.get_datastore()
+        replication_url = self.config.worker_replication_url
 
-    def build_tcp_replication(self):
-        return ReplicationClientHandler(self.get_datastore())
+        while True:
+            try:
+                args = store.stream_positions()
+                args["timeout"] = 30000
+                result = yield http_client.get_json(replication_url, args=args)
+                yield store.process_replication(result)
+            except:
+                logger.exception("Error replicating from %r", replication_url)
+                yield sleep(5)
 
 
 def start(config_options):
@@ -143,7 +162,7 @@ def start(config_options):
 
     assert config.worker_app == "synapse.app.federation_reader"
 
-    setup_logging(config, use_worker_options=True)
+    setup_logging(config.worker_log_config, config.worker_log_file)
 
     events.USE_FROZEN_DICTS = config.use_frozen_dicts
 
@@ -164,13 +183,33 @@ def start(config_options):
     ss.get_handlers()
     ss.start_listening(config.worker_listeners)
 
+    def run():
+        with LoggingContext("run"):
+            logger.info("Running")
+            change_resource_limit(config.soft_file_limit)
+            if config.gc_thresholds:
+                gc.set_threshold(*config.gc_thresholds)
+            reactor.run()
+
     def start():
         ss.get_state_handler().start_caching()
         ss.get_datastore().start_profiling()
+        ss.replicate()
 
     reactor.callWhenRunning(start)
 
-    _base.start_worker_reactor("synapse-federation-reader", config)
+    if config.worker_daemonize:
+        daemon = Daemonize(
+            app="synapse-federation-reader",
+            pid=config.worker_pid_file,
+            action=run,
+            auto_close_fds=False,
+            verbose=True,
+            logger=logger,
+        )
+        daemon.start()
+    else:
+        run()
 
 
 if __name__ == '__main__':

synapse/app/federation_sender.py

@@ -13,66 +13,53 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
-import sys
 
 import synapse
-from synapse import events
-from synapse.app import _base
+
+from synapse.server import HomeServer
 from synapse.config._base import ConfigError
-from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
+from synapse.config.homeserver import HomeServerConfig
 from synapse.crypto import context_factory
-from synapse.federation import send_queue
 from synapse.http.site import SynapseSite
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
+from synapse.federation import send_queue
+from synapse.federation.units import Edu
+from synapse.metrics.resource import MetricsResource, METRICS_PREFIX
 from synapse.replication.slave.storage.deviceinbox import SlavedDeviceInboxStore
-from synapse.replication.slave.storage.devices import SlavedDeviceStore
 from synapse.replication.slave.storage.events import SlavedEventStore
-from synapse.replication.slave.storage.presence import SlavedPresenceStore
 from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
 from synapse.replication.slave.storage.registration import SlavedRegistrationStore
 from synapse.replication.slave.storage.transactions import TransactionStore
-from synapse.replication.tcp.client import ReplicationClientHandler
-from synapse.server import HomeServer
+from synapse.replication.slave.storage.devices import SlavedDeviceStore
 from synapse.storage.engines import create_engine
-from synapse.util.async import Linearizer
+from synapse.storage.presence import UserPresenceState
+from synapse.util.async import sleep
 from synapse.util.httpresourcetree import create_resource_tree
-from synapse.util.logcontext import LoggingContext, preserve_fn
+from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
+from synapse.util.rlimit import change_resource_limit
 from synapse.util.versionstring import get_version_string
-from twisted.internet import defer, reactor
+
+from synapse import events
+
+from twisted.internet import reactor, defer
 from twisted.web.resource import Resource
 
-logger = logging.getLogger("synapse.app.federation_sender")
+from daemonize import Daemonize
+
+import sys
+import logging
+import gc
+import ujson as json
+
+logger = logging.getLogger("synapse.app.appservice")
 
 
 class FederationSenderSlaveStore(
     SlavedDeviceInboxStore, TransactionStore, SlavedReceiptsStore, SlavedEventStore,
-    SlavedRegistrationStore, SlavedDeviceStore, SlavedPresenceStore,
+    SlavedRegistrationStore, SlavedDeviceStore,
 ):
-    def __init__(self, db_conn, hs):
-        super(FederationSenderSlaveStore, self).__init__(db_conn, hs)
-
-        # We pull out the current federation stream position now so that we
-        # always have a known value for the federation position in memory so
-        # that we don't have to bounce via a deferred once when we start the
-        # replication streams.
-        self.federation_out_pos_startup = self._get_federation_out_pos(db_conn)
-
-    def _get_federation_out_pos(self, db_conn):
-        sql = (
-            "SELECT stream_id FROM federation_stream_position"
-            " WHERE type = ?"
-        )
-        sql = self.database_engine.convert_param_style(sql)
-
-        txn = db_conn.cursor()
-        txn.execute(sql, ("federation",))
-        rows = txn.fetchall()
-        txn.close()
-
-        return rows[0][0] if rows else -1
+    pass
 
 
 class FederationSenderServer(HomeServer):
@@ -140,27 +127,26 @@ class FederationSenderServer(HomeServer):
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 
-        self.get_tcp_replication().start_replication(self)
+    @defer.inlineCallbacks
+    def replicate(self):
+        http_client = self.get_simple_http_client()
+        store = self.get_datastore()
+        replication_url = self.config.worker_replication_url
+        send_handler = FederationSenderHandler(self)
 
-    def build_tcp_replication(self):
-        return FederationSenderReplicationHandler(self)
+        send_handler.on_start()
 
-
-class FederationSenderReplicationHandler(ReplicationClientHandler):
-    def __init__(self, hs):
-        super(FederationSenderReplicationHandler, self).__init__(hs.get_datastore())
-        self.send_handler = FederationSenderHandler(hs, self)
-
-    def on_rdata(self, stream_name, token, rows):
-        super(FederationSenderReplicationHandler, self).on_rdata(
-            stream_name, token, rows
-        )
-        self.send_handler.process_replication_rows(stream_name, token, rows)
-
-    def get_streams_to_replicate(self):
-        args = super(FederationSenderReplicationHandler, self).get_streams_to_replicate()
-        args.update(self.send_handler.stream_positions())
-        return args
+        while True:
+            try:
+                args = store.stream_positions()
+                args.update((yield send_handler.stream_positions()))
+                args["timeout"] = 30000
+                result = yield http_client.get_json(replication_url, args=args)
+                yield store.process_replication(result)
+                yield send_handler.process_replication(result)
+            except:
+                logger.exception("Error replicating from %r", replication_url)
+                yield sleep(30)
 
 
 def start(config_options):
@@ -174,7 +160,7 @@ def start(config_options):
 
     assert config.worker_app == "synapse.app.federation_sender"
 
-    setup_logging(config, use_worker_options=True)
+    setup_logging(config.worker_log_config, config.worker_log_file)
 
     events.USE_FROZEN_DICTS = config.use_frozen_dicts
 
@@ -206,27 +192,42 @@ def start(config_options):
     ps.setup()
     ps.start_listening(config.worker_listeners)
 
+    def run():
+        with LoggingContext("run"):
+            logger.info("Running")
+            change_resource_limit(config.soft_file_limit)
+            if config.gc_thresholds:
+                gc.set_threshold(*config.gc_thresholds)
+            reactor.run()
+
     def start():
+        ps.replicate()
         ps.get_datastore().start_profiling()
         ps.get_state_handler().start_caching()
 
     reactor.callWhenRunning(start)
-    _base.start_worker_reactor("synapse-federation-sender", config)
+
+    if config.worker_daemonize:
+        daemon = Daemonize(
+            app="synapse-federation-sender",
+            pid=config.worker_pid_file,
+            action=run,
+            auto_close_fds=False,
+            verbose=True,
+            logger=logger,
+        )
+        daemon.start()
+    else:
+        run()
 
 
 class FederationSenderHandler(object):
     """Processes the replication stream and forwards the appropriate entries
     to the federation sender.
     """
-    def __init__(self, hs, replication_client):
+    def __init__(self, hs):
         self.store = hs.get_datastore()
         self.federation_sender = hs.get_federation_sender()
-        self.replication_client = replication_client
-
-        self.federation_position = self.store.federation_out_pos_startup
-        self._fed_position_linearizer = Linearizer(name="_fed_position_linearizer")
-
-        self._last_ack = self.federation_position
 
         self._room_serials = {}
         self._room_typing = {}
@@ -238,35 +239,98 @@ class FederationSenderHandler(object):
             self.store.get_room_max_stream_ordering()
         )
 
+    @defer.inlineCallbacks
     def stream_positions(self):
-        return {"federation": self.federation_position}
+        stream_id = yield self.store.get_federation_out_pos("federation")
+        defer.returnValue({
+            "federation": stream_id,
 
-    def process_replication_rows(self, stream_name, token, rows):
-        # The federation stream contains things that we want to send out, e.g.
-        # presence, typing, etc.
-        if stream_name == "federation":
-            send_queue.process_rows_for_federation(self.federation_sender, rows)
-            preserve_fn(self.update_token)(token)
-
-        # We also need to poke the federation sender when new events happen
-        elif stream_name == "events":
-            self.federation_sender.notify_new_events(token)
+            # Ack stuff we've "processed", this should only be called from
+            # one process.
+            "federation_ack": stream_id,
+        })
 
     @defer.inlineCallbacks
-    def update_token(self, token):
-        self.federation_position = token
+    def process_replication(self, result):
+        # The federation stream contains things that we want to send out, e.g.
+        # presence, typing, etc.
+        fed_stream = result.get("federation")
+        if fed_stream:
+            latest_id = int(fed_stream["position"])
 
-        # We linearize here to ensure we don't have races updating the token
-        with (yield self._fed_position_linearizer.queue(None)):
-            if self._last_ack < self.federation_position:
-                yield self.store.update_federation_out_pos(
-                    "federation", self.federation_position
-                )
+            # The federation stream containis a bunch of different types of
+            # rows that need to be handled differently. We parse the rows, put
+            # them into the appropriate collection and then send them off.
+            presence_to_send = {}
+            keyed_edus = {}
+            edus = {}
+            failures = {}
+            device_destinations = set()
 
-                # We ACK this token over replication so that the master can drop
-                # its in memory queues
-                self.replication_client.send_federation_ack(self.federation_position)
-                self._last_ack = self.federation_position
+            # Parse the rows in the stream
+            for row in fed_stream["rows"]:
+                position, typ, content_js = row
+                content = json.loads(content_js)
+
+                if typ == send_queue.PRESENCE_TYPE:
+                    destination = content["destination"]
+                    state = UserPresenceState.from_dict(content["state"])
+
+                    presence_to_send.setdefault(destination, []).append(state)
+                elif typ == send_queue.KEYED_EDU_TYPE:
+                    key = content["key"]
+                    edu = Edu(**content["edu"])
+
+                    keyed_edus.setdefault(
+                        edu.destination, {}
+                    )[(edu.destination, tuple(key))] = edu
+                elif typ == send_queue.EDU_TYPE:
+                    edu = Edu(**content)
+
+                    edus.setdefault(edu.destination, []).append(edu)
+                elif typ == send_queue.FAILURE_TYPE:
+                    destination = content["destination"]
+                    failure = content["failure"]
+
+                    failures.setdefault(destination, []).append(failure)
+                elif typ == send_queue.DEVICE_MESSAGE_TYPE:
+                    device_destinations.add(content["destination"])
+                else:
+                    raise Exception("Unrecognised federation type: %r", typ)
+
+            # We've finished collecting, send everything off
+            for destination, states in presence_to_send.items():
+                self.federation_sender.send_presence(destination, states)
+
+            for destination, edu_map in keyed_edus.items():
+                for key, edu in edu_map.items():
+                    self.federation_sender.send_edu(
+                        edu.destination, edu.edu_type, edu.content, key=key,
+                    )
+
+            for destination, edu_list in edus.items():
+                for edu in edu_list:
+                    self.federation_sender.send_edu(
+                        edu.destination, edu.edu_type, edu.content, key=None,
+                    )
+
+            for destination, failure_list in failures.items():
+                for failure in failure_list:
+                    self.federation_sender.send_failure(destination, failure)
+
+            for destination in device_destinations:
+                self.federation_sender.send_device_messages(destination)
+
+            # Record where we are in the stream.
+            yield self.store.update_federation_out_pos(
+                "federation", latest_id
+            )
+
+        # We also need to poke the federation sender when new events happen
+        event_stream = result.get("events")
+        if event_stream:
+            latest_pos = event_stream["position"]
+            self.federation_sender.notify_new_events(latest_pos)
 
 
 if __name__ == '__main__':

synapse/app/frontend_proxy.py

@@ -1,245 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-# Copyright 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-import logging
-import sys
-
-import synapse
-from synapse import events
-from synapse.api.errors import SynapseError
-from synapse.app import _base
-from synapse.config._base import ConfigError
-from synapse.config.homeserver import HomeServerConfig
-from synapse.config.logger import setup_logging
-from synapse.crypto import context_factory
-from synapse.http.server import JsonResource
-from synapse.http.servlet import (
-    RestServlet, parse_json_object_from_request,
-)
-from synapse.http.site import SynapseSite
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
-from synapse.replication.slave.storage._base import BaseSlavedStore
-from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
-from synapse.replication.slave.storage.client_ips import SlavedClientIpStore
-from synapse.replication.slave.storage.devices import SlavedDeviceStore
-from synapse.replication.slave.storage.registration import SlavedRegistrationStore
-from synapse.replication.tcp.client import ReplicationClientHandler
-from synapse.rest.client.v2_alpha._base import client_v2_patterns
-from synapse.server import HomeServer
-from synapse.storage.engines import create_engine
-from synapse.util.httpresourcetree import create_resource_tree
-from synapse.util.logcontext import LoggingContext
-from synapse.util.manhole import manhole
-from synapse.util.versionstring import get_version_string
-from twisted.internet import defer, reactor
-from twisted.web.resource import Resource
-
-logger = logging.getLogger("synapse.app.frontend_proxy")
-
-
-class KeyUploadServlet(RestServlet):
-    PATTERNS = client_v2_patterns("/keys/upload(/(?P<device_id>[^/]+))?$")
-
-    def __init__(self, hs):
-        """
-        Args:
-            hs (synapse.server.HomeServer): server
-        """
-        super(KeyUploadServlet, self).__init__()
-        self.auth = hs.get_auth()
-        self.store = hs.get_datastore()
-        self.http_client = hs.get_simple_http_client()
-        self.main_uri = hs.config.worker_main_http_uri
-
-    @defer.inlineCallbacks
-    def on_POST(self, request, device_id):
-        requester = yield self.auth.get_user_by_req(request, allow_guest=True)
-        user_id = requester.user.to_string()
-        body = parse_json_object_from_request(request)
-
-        if device_id is not None:
-            # passing the device_id here is deprecated; however, we allow it
-            # for now for compatibility with older clients.
-            if (requester.device_id is not None and
-                    device_id != requester.device_id):
-                logger.warning("Client uploading keys for a different device "
-                               "(logged in as %s, uploading for %s)",
-                               requester.device_id, device_id)
-        else:
-            device_id = requester.device_id
-
-        if device_id is None:
-            raise SynapseError(
-                400,
-                "To upload keys, you must pass device_id when authenticating"
-            )
-
-        if body:
-            # They're actually trying to upload something, proxy to main synapse.
-            # Pass through the auth headers, if any, in case the access token
-            # is there.
-            auth_headers = request.requestHeaders.getRawHeaders("Authorization", [])
-            headers = {
-                "Authorization": auth_headers,
-            }
-            result = yield self.http_client.post_json_get_json(
-                self.main_uri + request.uri,
-                body,
-                headers=headers,
-            )
-
-            defer.returnValue((200, result))
-        else:
-            # Just interested in counts.
-            result = yield self.store.count_e2e_one_time_keys(user_id, device_id)
-            defer.returnValue((200, {"one_time_key_counts": result}))
-
-
-class FrontendProxySlavedStore(
-    SlavedDeviceStore,
-    SlavedClientIpStore,
-    SlavedApplicationServiceStore,
-    SlavedRegistrationStore,
-    BaseSlavedStore,
-):
-    pass
-
-
-class FrontendProxyServer(HomeServer):
-    def get_db_conn(self, run_new_connection=True):
-        # Any param beginning with cp_ is a parameter for adbapi, and should
-        # not be passed to the database engine.
-        db_params = {
-            k: v for k, v in self.db_config.get("args", {}).items()
-            if not k.startswith("cp_")
-        }
-        db_conn = self.database_engine.module.connect(**db_params)
-
-        if run_new_connection:
-            self.database_engine.on_new_connection(db_conn)
-        return db_conn
-
-    def setup(self):
-        logger.info("Setting up.")
-        self.datastore = FrontendProxySlavedStore(self.get_db_conn(), self)
-        logger.info("Finished setting up.")
-
-    def _listen_http(self, listener_config):
-        port = listener_config["port"]
-        bind_addresses = listener_config["bind_addresses"]
-        site_tag = listener_config.get("tag", port)
-        resources = {}
-        for res in listener_config["resources"]:
-            for name in res["names"]:
-                if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(self)
-                elif name == "client":
-                    resource = JsonResource(self, canonical_json=False)
-                    KeyUploadServlet(self).register(resource)
-                    resources.update({
-                        "/_matrix/client/r0": resource,
-                        "/_matrix/client/unstable": resource,
-                        "/_matrix/client/v2_alpha": resource,
-                        "/_matrix/client/api/v1": resource,
-                    })
-
-        root_resource = create_resource_tree(resources, Resource())
-
-        for address in bind_addresses:
-            reactor.listenTCP(
-                port,
-                SynapseSite(
-                    "synapse.access.http.%s" % (site_tag,),
-                    site_tag,
-                    listener_config,
-                    root_resource,
-                ),
-                interface=address
-            )
-
-        logger.info("Synapse client reader now listening on port %d", port)
-
-    def start_listening(self, listeners):
-        for listener in listeners:
-            if listener["type"] == "http":
-                self._listen_http(listener)
-            elif listener["type"] == "manhole":
-                bind_addresses = listener["bind_addresses"]
-
-                for address in bind_addresses:
-                    reactor.listenTCP(
-                        listener["port"],
-                        manhole(
-                            username="matrix",
-                            password="rabbithole",
-                            globals={"hs": self},
-                        ),
-                        interface=address
-                    )
-            else:
-                logger.warn("Unrecognized listener type: %s", listener["type"])
-
-        self.get_tcp_replication().start_replication(self)
-
-    def build_tcp_replication(self):
-        return ReplicationClientHandler(self.get_datastore())
-
-
-def start(config_options):
-    try:
-        config = HomeServerConfig.load_config(
-            "Synapse frontend proxy", config_options
-        )
-    except ConfigError as e:
-        sys.stderr.write("\n" + e.message + "\n")
-        sys.exit(1)
-
-    assert config.worker_app == "synapse.app.frontend_proxy"
-
-    assert config.worker_main_http_uri is not None
-
-    setup_logging(config, use_worker_options=True)
-
-    events.USE_FROZEN_DICTS = config.use_frozen_dicts
-
-    database_engine = create_engine(config.database_config)
-
-    tls_server_context_factory = context_factory.ServerContextFactory(config)
-
-    ss = FrontendProxyServer(
-        config.server_name,
-        db_config=config.database_config,
-        tls_server_context_factory=tls_server_context_factory,
-        config=config,
-        version_string="Synapse/" + get_version_string(synapse),
-        database_engine=database_engine,
-    )
-
-    ss.setup()
-    ss.get_handlers()
-    ss.start_listening(config.worker_listeners)
-
-    def start():
-        ss.get_state_handler().start_caching()
-        ss.get_datastore().start_profiling()
-
-    reactor.callWhenRunning(start)
-
-    _base.start_worker_reactor("synapse-frontend-proxy", config)
-
-
-if __name__ == '__main__':
-    with LoggingContext("main"):
-        start(sys.argv[1:])

synapse/app/homeserver.py

@@ -13,52 +13,59 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
+import synapse
+
 import gc
 import logging
 import os
 import sys
-
-import synapse
-import synapse.config.logger
-from synapse import events
-from synapse.api.urls import CONTENT_REPO_PREFIX, FEDERATION_PREFIX, \
-    LEGACY_MEDIA_PREFIX, MEDIA_PREFIX, SERVER_KEY_PREFIX, SERVER_KEY_V2_PREFIX, \
-    STATIC_PREFIX, WEB_CLIENT_PREFIX
-from synapse.app import _base
-from synapse.app._base import quit_with_error
 from synapse.config._base import ConfigError
-from synapse.config.homeserver import HomeServerConfig
-from synapse.crypto import context_factory
-from synapse.federation.transport.server import TransportLayerServer
-from synapse.module_api import ModuleApi
-from synapse.http.additional_resource import AdditionalResource
-from synapse.http.server import RootRedirect
-from synapse.http.site import SynapseSite
-from synapse.metrics import register_memory_metrics
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
-from synapse.python_dependencies import CONDITIONAL_REQUIREMENTS, \
-    check_requirements
-from synapse.replication.tcp.resource import ReplicationStreamProtocolFactory
+
+from synapse.python_dependencies import (
+    check_requirements, DEPENDENCY_LINKS
+)
+
 from synapse.rest import ClientRestResource
-from synapse.rest.key.v1.server_key_resource import LocalKey
-from synapse.rest.key.v2 import KeyApiV2Resource
+from synapse.storage.engines import create_engine, IncorrectDatabaseSetup
+from synapse.storage import are_all_users_on_domain
+from synapse.storage.prepare_database import UpgradeDatabaseException, prepare_database
+
+from synapse.server import HomeServer
+
+from twisted.internet import reactor, task, defer
+from twisted.application import service
+from twisted.web.resource import Resource, EncodingResourceWrapper
+from twisted.web.static import File
+from twisted.web.server import GzipEncoderFactory
+from synapse.http.server import RootRedirect
 from synapse.rest.media.v0.content_repository import ContentRepoResource
 from synapse.rest.media.v1.media_repository import MediaRepositoryResource
-from synapse.server import HomeServer
-from synapse.storage import are_all_users_on_domain
-from synapse.storage.engines import IncorrectDatabaseSetup, create_engine
-from synapse.storage.prepare_database import UpgradeDatabaseException, prepare_database
-from synapse.util.httpresourcetree import create_resource_tree
+from synapse.rest.key.v1.server_key_resource import LocalKey
+from synapse.rest.key.v2 import KeyApiV2Resource
+from synapse.api.urls import (
+    FEDERATION_PREFIX, WEB_CLIENT_PREFIX, CONTENT_REPO_PREFIX,
+    SERVER_KEY_PREFIX, LEGACY_MEDIA_PREFIX, MEDIA_PREFIX, STATIC_PREFIX,
+    SERVER_KEY_V2_PREFIX,
+)
+from synapse.config.homeserver import HomeServerConfig
+from synapse.crypto import context_factory
 from synapse.util.logcontext import LoggingContext
-from synapse.util.manhole import manhole
-from synapse.util.module_loader import load_module
+from synapse.metrics import register_memory_metrics, get_metrics_for
+from synapse.metrics.resource import MetricsResource, METRICS_PREFIX
+from synapse.replication.resource import ReplicationResource, REPLICATION_PREFIX
+from synapse.federation.transport.server import TransportLayerServer
+
 from synapse.util.rlimit import change_resource_limit
 from synapse.util.versionstring import get_version_string
-from twisted.application import service
-from twisted.internet import defer, reactor
-from twisted.web.resource import EncodingResourceWrapper, Resource
-from twisted.web.server import GzipEncoderFactory
-from twisted.web.static import File
+from synapse.util.httpresourcetree import create_resource_tree
+from synapse.util.manhole import manhole
+
+from synapse.http.site import SynapseSite
+
+from synapse import events
+
+from daemonize import Daemonize
 
 logger = logging.getLogger("synapse.app.homeserver")
 
@@ -83,7 +90,7 @@ def build_resource_for_web_client(hs):
                 "\n"
                 "You can also disable hosting of the webclient via the\n"
                 "configuration option `web_client`\n"
-                % {"dep": CONDITIONAL_REQUIREMENTS["web_client"].keys()[0]}
+                % {"dep": DEPENDENCY_LINKS["matrix-angular-sdk"]}
             )
         syweb_path = os.path.dirname(syweb.__file__)
         webclient_path = os.path.join(syweb_path, "webclient")
@@ -110,18 +117,55 @@ class SynapseHomeServer(HomeServer):
         resources = {}
         for res in listener_config["resources"]:
             for name in res["names"]:
-                resources.update(self._configure_named_resource(
-                    name, res.get("compress", False),
-                ))
+                if name == "client":
+                    client_resource = ClientRestResource(self)
+                    if res["compress"]:
+                        client_resource = gz_wrap(client_resource)
 
-        additional_resources = listener_config.get("additional_resources", {})
-        logger.debug("Configuring additional resources: %r",
-                     additional_resources)
-        module_api = ModuleApi(self, self.get_auth_handler())
-        for path, resmodule in additional_resources.items():
-            handler_cls, config = load_module(resmodule)
-            handler = handler_cls(config, module_api)
-            resources[path] = AdditionalResource(self, handler.handle_request)
+                    resources.update({
+                        "/_matrix/client/api/v1": client_resource,
+                        "/_matrix/client/r0": client_resource,
+                        "/_matrix/client/unstable": client_resource,
+                        "/_matrix/client/v2_alpha": client_resource,
+                        "/_matrix/client/versions": client_resource,
+                    })
+
+                if name == "federation":
+                    resources.update({
+                        FEDERATION_PREFIX: TransportLayerServer(self),
+                    })
+
+                if name in ["static", "client"]:
+                    resources.update({
+                        STATIC_PREFIX: File(
+                            os.path.join(os.path.dirname(synapse.__file__), "static")
+                        ),
+                    })
+
+                if name in ["media", "federation", "client"]:
+                    media_repo = MediaRepositoryResource(self)
+                    resources.update({
+                        MEDIA_PREFIX: media_repo,
+                        LEGACY_MEDIA_PREFIX: media_repo,
+                        CONTENT_REPO_PREFIX: ContentRepoResource(
+                            self, self.config.uploads_path
+                        ),
+                    })
+
+                if name in ["keys", "federation"]:
+                    resources.update({
+                        SERVER_KEY_PREFIX: LocalKey(self),
+                        SERVER_KEY_V2_PREFIX: KeyApiV2Resource(self),
+                    })
+
+                if name == "webclient":
+                    resources[WEB_CLIENT_PREFIX] = build_resource_for_web_client(self)
+
+                if name == "metrics" and self.get_config().enable_metrics:
+                    resources[METRICS_PREFIX] = MetricsResource(self)
+
+                if name == "replication":
+                    resources[REPLICATION_PREFIX] = ReplicationResource(self)
 
         if WEB_CLIENT_PREFIX in resources:
             root_resource = RootRedirect(WEB_CLIENT_PREFIX)
@@ -157,67 +201,6 @@ class SynapseHomeServer(HomeServer):
                 )
         logger.info("Synapse now listening on port %d", port)
 
-    def _configure_named_resource(self, name, compress=False):
-        """Build a resource map for a named resource
-
-        Args:
-            name (str): named resource: one of "client", "federation", etc
-            compress (bool): whether to enable gzip compression for this
-                resource
-
-        Returns:
-            dict[str, Resource]: map from path to HTTP resource
-        """
-        resources = {}
-        if name == "client":
-            client_resource = ClientRestResource(self)
-            if compress:
-                client_resource = gz_wrap(client_resource)
-
-            resources.update({
-                "/_matrix/client/api/v1": client_resource,
-                "/_matrix/client/r0": client_resource,
-                "/_matrix/client/unstable": client_resource,
-                "/_matrix/client/v2_alpha": client_resource,
-                "/_matrix/client/versions": client_resource,
-            })
-
-        if name == "federation":
-            resources.update({
-                FEDERATION_PREFIX: TransportLayerServer(self),
-            })
-
-        if name in ["static", "client"]:
-            resources.update({
-                STATIC_PREFIX: File(
-                    os.path.join(os.path.dirname(synapse.__file__), "static")
-                ),
-            })
-
-        if name in ["media", "federation", "client"]:
-            media_repo = MediaRepositoryResource(self)
-            resources.update({
-                MEDIA_PREFIX: media_repo,
-                LEGACY_MEDIA_PREFIX: media_repo,
-                CONTENT_REPO_PREFIX: ContentRepoResource(
-                    self, self.config.uploads_path
-                ),
-            })
-
-        if name in ["keys", "federation"]:
-            resources.update({
-                SERVER_KEY_PREFIX: LocalKey(self),
-                SERVER_KEY_V2_PREFIX: KeyApiV2Resource(self),
-            })
-
-        if name == "webclient":
-            resources[WEB_CLIENT_PREFIX] = build_resource_for_web_client(self)
-
-        if name == "metrics" and self.get_config().enable_metrics:
-            resources[METRICS_PREFIX] = MetricsResource(self)
-
-        return resources
-
     def start_listening(self):
         config = self.get_config()
 
@@ -237,16 +220,6 @@ class SynapseHomeServer(HomeServer):
                         ),
                         interface=address
                     )
-            elif listener["type"] == "replication":
-                bind_addresses = listener["bind_addresses"]
-                for address in bind_addresses:
-                    factory = ReplicationStreamProtocolFactory(self)
-                    server_listener = reactor.listenTCP(
-                        listener["port"], factory, interface=address
-                    )
-                    reactor.addSystemEventTrigger(
-                        "before", "shutdown", server_listener.stopListening,
-                    )
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 
@@ -280,6 +253,16 @@ class SynapseHomeServer(HomeServer):
         return db_conn
 
 
+def quit_with_error(error_string):
+    message_lines = error_string.split("\n")
+    line_length = max([len(l) for l in message_lines if len(l) < 80]) + 2
+    sys.stderr.write("*" * line_length + '\n')
+    for line in message_lines:
+        sys.stderr.write(" %s\n" % (line.rstrip(),))
+    sys.stderr.write("*" * line_length + '\n')
+    sys.exit(1)
+
+
 def setup(config_options):
     """
     Args:
@@ -303,7 +286,7 @@ def setup(config_options):
         # generating config files and shouldn't try to continue.
         sys.exit(0)
 
-    synapse.config.logger.setup_logging(config, use_worker_options=False)
+    config.setup_logging()
 
     # check any extra requirements we have now we have a config
     check_requirements(config)
@@ -406,8 +389,7 @@ def run(hs):
         ThreadPool._worker = profile(ThreadPool._worker)
         reactor.run = profile(reactor.run)
 
-    clock = hs.get_clock()
-    start_time = clock.time()
+    start_time = hs.get_clock().time()
 
     stats = {}
 
@@ -419,23 +401,41 @@ def run(hs):
         if uptime < 0:
             uptime = 0
 
+        # If the stats directory is empty then this is the first time we've
+        # reported stats.
+        first_time = not stats
+
         stats["homeserver"] = hs.config.server_name
         stats["timestamp"] = now
         stats["uptime_seconds"] = uptime
         stats["total_users"] = yield hs.get_datastore().count_all_users()
 
-        total_nonbridged_users = yield hs.get_datastore().count_nonbridged_users()
-        stats["total_nonbridged_users"] = total_nonbridged_users
-
         room_count = yield hs.get_datastore().get_room_count()
         stats["total_room_count"] = room_count
 
         stats["daily_active_users"] = yield hs.get_datastore().count_daily_users()
-        stats["daily_active_rooms"] = yield hs.get_datastore().count_daily_active_rooms()
-        stats["daily_messages"] = yield hs.get_datastore().count_daily_messages()
+        daily_messages = yield hs.get_datastore().count_daily_messages()
+        if daily_messages is not None:
+            stats["daily_messages"] = daily_messages
+        else:
+            stats.pop("daily_messages", None)
 
-        daily_sent_messages = yield hs.get_datastore().count_daily_sent_messages()
-        stats["daily_sent_messages"] = daily_sent_messages
+        if first_time:
+            # Add callbacks to report the synapse stats as metrics whenever
+            # prometheus requests them, typically every 30s.
+            # As some of the stats are expensive to calculate we only update
+            # them when synapse phones home to matrix.org every 24 hours.
+            metrics = get_metrics_for("synapse.usage")
+            metrics.add_callback("timestamp", lambda: stats["timestamp"])
+            metrics.add_callback("uptime_seconds", lambda: stats["uptime_seconds"])
+            metrics.add_callback("total_users", lambda: stats["total_users"])
+            metrics.add_callback("total_room_count", lambda: stats["total_room_count"])
+            metrics.add_callback(
+                "daily_active_users", lambda: stats["daily_active_users"]
+            )
+            metrics.add_callback(
+                "daily_messages", lambda: stats.get("daily_messages", 0)
+            )
 
         logger.info("Reporting stats to matrix.org: %s" % (stats,))
         try:
@@ -447,25 +447,36 @@ def run(hs):
             logger.warn("Error reporting stats: %s", e)
 
     if hs.config.report_stats:
-        logger.info("Scheduling stats reporting for 3 hour intervals")
-        clock.looping_call(phone_stats_home, 3 * 60 * 60 * 1000)
+        phone_home_task = task.LoopingCall(phone_stats_home)
+        logger.info("Scheduling stats reporting for 24 hour intervals")
+        phone_home_task.start(60 * 60 * 24, now=False)
 
-        # We wait 5 minutes to send the first set of stats as the server can
-        # be quite busy the first few minutes
-        clock.call_later(5 * 60, phone_stats_home)
+    def in_thread():
+        # Uncomment to enable tracing of log context changes.
+        # sys.settrace(logcontext_tracer)
+        with LoggingContext("run"):
+            change_resource_limit(hs.config.soft_file_limit)
+            if hs.config.gc_thresholds:
+                gc.set_threshold(*hs.config.gc_thresholds)
+            reactor.run()
 
-    if hs.config.daemonize and hs.config.print_pidfile:
-        print (hs.config.pid_file)
+    if hs.config.daemonize:
 
-    _base.start_reactor(
-        "synapse-homeserver",
-        hs.config.soft_file_limit,
-        hs.config.gc_thresholds,
-        hs.config.pid_file,
-        hs.config.daemonize,
-        hs.config.cpu_affinity,
-        logger,
-    )
+        if hs.config.print_pidfile:
+            print (hs.config.pid_file)
+
+        daemon = Daemonize(
+            app="synapse-homeserver",
+            pid=hs.config.pid_file,
+            action=lambda: in_thread(),
+            auto_close_fds=False,
+            verbose=True,
+            logger=logger,
+        )
+
+        daemon.start()
+    else:
+        in_thread()
 
 
 def main():

synapse/app/media_repository.py

@@ -13,49 +13,55 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
-import sys
 
 import synapse
-from synapse import events
-from synapse.api.urls import (
-    CONTENT_REPO_PREFIX, LEGACY_MEDIA_PREFIX, MEDIA_PREFIX
-)
-from synapse.app import _base
+
 from synapse.config._base import ConfigError
 from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
-from synapse.crypto import context_factory
 from synapse.http.site import SynapseSite
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
+from synapse.metrics.resource import MetricsResource, METRICS_PREFIX
 from synapse.replication.slave.storage._base import BaseSlavedStore
 from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
-from synapse.replication.slave.storage.client_ips import SlavedClientIpStore
 from synapse.replication.slave.storage.registration import SlavedRegistrationStore
-from synapse.replication.slave.storage.transactions import TransactionStore
-from synapse.replication.tcp.client import ReplicationClientHandler
 from synapse.rest.media.v0.content_repository import ContentRepoResource
 from synapse.rest.media.v1.media_repository import MediaRepositoryResource
 from synapse.server import HomeServer
+from synapse.storage.client_ips import ClientIpStore
 from synapse.storage.engines import create_engine
 from synapse.storage.media_repository import MediaRepositoryStore
+from synapse.util.async import sleep
 from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext
 from synapse.util.manhole import manhole
+from synapse.util.rlimit import change_resource_limit
 from synapse.util.versionstring import get_version_string
-from twisted.internet import reactor
+from synapse.api.urls import (
+    CONTENT_REPO_PREFIX, LEGACY_MEDIA_PREFIX, MEDIA_PREFIX
+)
+from synapse.crypto import context_factory
+
+from synapse import events
+
+
+from twisted.internet import reactor, defer
 from twisted.web.resource import Resource
 
+from daemonize import Daemonize
+
+import sys
+import logging
+import gc
+
 logger = logging.getLogger("synapse.app.media_repository")
 
 
 class MediaRepositorySlavedStore(
     SlavedApplicationServiceStore,
     SlavedRegistrationStore,
-    SlavedClientIpStore,
-    TransactionStore,
     BaseSlavedStore,
     MediaRepositoryStore,
+    ClientIpStore,
 ):
     pass
 
@@ -134,10 +140,21 @@ class MediaRepositoryServer(HomeServer):
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 
-        self.get_tcp_replication().start_replication(self)
+    @defer.inlineCallbacks
+    def replicate(self):
+        http_client = self.get_simple_http_client()
+        store = self.get_datastore()
+        replication_url = self.config.worker_replication_url
 
-    def build_tcp_replication(self):
-        return ReplicationClientHandler(self.get_datastore())
+        while True:
+            try:
+                args = store.stream_positions()
+                args["timeout"] = 30000
+                result = yield http_client.get_json(replication_url, args=args)
+                yield store.process_replication(result)
+            except:
+                logger.exception("Error replicating from %r", replication_url)
+                yield sleep(5)
 
 
 def start(config_options):
@@ -151,7 +168,7 @@ def start(config_options):
 
     assert config.worker_app == "synapse.app.media_repository"
 
-    setup_logging(config, use_worker_options=True)
+    setup_logging(config.worker_log_config, config.worker_log_file)
 
     events.USE_FROZEN_DICTS = config.use_frozen_dicts
 
@@ -172,13 +189,33 @@ def start(config_options):
     ss.get_handlers()
     ss.start_listening(config.worker_listeners)
 
+    def run():
+        with LoggingContext("run"):
+            logger.info("Running")
+            change_resource_limit(config.soft_file_limit)
+            if config.gc_thresholds:
+                gc.set_threshold(*config.gc_thresholds)
+            reactor.run()
+
     def start():
         ss.get_state_handler().start_caching()
         ss.get_datastore().start_profiling()
+        ss.replicate()
 
     reactor.callWhenRunning(start)
 
-    _base.start_worker_reactor("synapse-media-repository", config)
+    if config.worker_daemonize:
+        daemon = Daemonize(
+            app="synapse-media-repository",
+            pid=config.worker_pid_file,
+            action=run,
+            auto_close_fds=False,
+            verbose=True,
+            logger=logger,
+        )
+        daemon.start()
+    else:
+        run()
 
 
 if __name__ == '__main__':

synapse/app/pusher.py

@@ -13,33 +13,40 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
-import sys
 
 import synapse
-from synapse import events
-from synapse.app import _base
+
+from synapse.server import HomeServer
 from synapse.config._base import ConfigError
-from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
+from synapse.config.homeserver import HomeServerConfig
 from synapse.http.site import SynapseSite
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
-from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
+from synapse.metrics.resource import MetricsResource, METRICS_PREFIX
+from synapse.storage.roommember import RoomMemberStore
 from synapse.replication.slave.storage.events import SlavedEventStore
 from synapse.replication.slave.storage.pushers import SlavedPusherStore
 from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
-from synapse.replication.tcp.client import ReplicationClientHandler
-from synapse.server import HomeServer
-from synapse.storage import DataStore
+from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
 from synapse.storage.engines import create_engine
-from synapse.storage.roommember import RoomMemberStore
+from synapse.storage import DataStore
+from synapse.util.async import sleep
 from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext, preserve_fn
 from synapse.util.manhole import manhole
+from synapse.util.rlimit import change_resource_limit
 from synapse.util.versionstring import get_version_string
-from twisted.internet import defer, reactor
+
+from synapse import events
+
+from twisted.internet import reactor, defer
 from twisted.web.resource import Resource
 
+from daemonize import Daemonize
+
+import sys
+import logging
+import gc
+
 logger = logging.getLogger("synapse.app.pusher")
 
 
@@ -81,6 +88,7 @@ class PusherSlaveStore(
 
 
 class PusherServer(HomeServer):
+
     def get_db_conn(self, run_new_connection=True):
         # Any param beginning with cp_ is a parameter for adbapi, and should
         # not be passed to the database engine.
@@ -100,7 +108,16 @@ class PusherServer(HomeServer):
         logger.info("Finished setting up.")
 
     def remove_pusher(self, app_id, push_key, user_id):
-        self.get_tcp_replication().send_remove_pusher(app_id, push_key, user_id)
+        http_client = self.get_simple_http_client()
+        replication_url = self.config.worker_replication_url
+        url = replication_url + "/remove_pushers"
+        return http_client.post_json_get_json(url, {
+            "remove": [{
+                "app_id": app_id,
+                "push_key": push_key,
+                "user_id": user_id,
+            }]
+        })
 
     def _listen_http(self, listener_config):
         port = listener_config["port"]
@@ -148,52 +165,73 @@ class PusherServer(HomeServer):
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 
-        self.get_tcp_replication().start_replication(self)
-
-    def build_tcp_replication(self):
-        return PusherReplicationHandler(self)
-
-
-class PusherReplicationHandler(ReplicationClientHandler):
-    def __init__(self, hs):
-        super(PusherReplicationHandler, self).__init__(hs.get_datastore())
-
-        self.pusher_pool = hs.get_pusherpool()
-
-    def on_rdata(self, stream_name, token, rows):
-        super(PusherReplicationHandler, self).on_rdata(stream_name, token, rows)
-        preserve_fn(self.poke_pushers)(stream_name, token, rows)
-
     @defer.inlineCallbacks
-    def poke_pushers(self, stream_name, token, rows):
-        if stream_name == "pushers":
-            for row in rows:
-                if row.deleted:
-                    yield self.stop_pusher(row.user_id, row.app_id, row.pushkey)
-                else:
-                    yield self.start_pusher(row.user_id, row.app_id, row.pushkey)
-        elif stream_name == "events":
-            yield self.pusher_pool.on_new_notifications(
-                token, token,
-            )
-        elif stream_name == "receipts":
-            yield self.pusher_pool.on_new_receipts(
-                token, token, set(row.room_id for row in rows)
-            )
+    def replicate(self):
+        http_client = self.get_simple_http_client()
+        store = self.get_datastore()
+        replication_url = self.config.worker_replication_url
+        pusher_pool = self.get_pusherpool()
 
-    def stop_pusher(self, user_id, app_id, pushkey):
-        key = "%s:%s" % (app_id, pushkey)
-        pushers_for_user = self.pusher_pool.pushers.get(user_id, {})
-        pusher = pushers_for_user.pop(key, None)
-        if pusher is None:
-            return
-        logger.info("Stopping pusher %r / %r", user_id, key)
-        pusher.on_stop()
+        def stop_pusher(user_id, app_id, pushkey):
+            key = "%s:%s" % (app_id, pushkey)
+            pushers_for_user = pusher_pool.pushers.get(user_id, {})
+            pusher = pushers_for_user.pop(key, None)
+            if pusher is None:
+                return
+            logger.info("Stopping pusher %r / %r", user_id, key)
+            pusher.on_stop()
 
-    def start_pusher(self, user_id, app_id, pushkey):
-        key = "%s:%s" % (app_id, pushkey)
-        logger.info("Starting pusher %r / %r", user_id, key)
-        return self.pusher_pool._refresh_pusher(app_id, pushkey, user_id)
+        def start_pusher(user_id, app_id, pushkey):
+            key = "%s:%s" % (app_id, pushkey)
+            logger.info("Starting pusher %r / %r", user_id, key)
+            return pusher_pool._refresh_pusher(app_id, pushkey, user_id)
+
+        @defer.inlineCallbacks
+        def poke_pushers(results):
+            pushers_rows = set(
+                map(tuple, results.get("pushers", {}).get("rows", []))
+            )
+            deleted_pushers_rows = set(
+                map(tuple, results.get("deleted_pushers", {}).get("rows", []))
+            )
+            for row in sorted(pushers_rows | deleted_pushers_rows):
+                if row in deleted_pushers_rows:
+                    user_id, app_id, pushkey = row[1:4]
+                    stop_pusher(user_id, app_id, pushkey)
+                elif row in pushers_rows:
+                    user_id = row[1]
+                    app_id = row[5]
+                    pushkey = row[8]
+                    yield start_pusher(user_id, app_id, pushkey)
+
+            stream = results.get("events")
+            if stream and stream["rows"]:
+                min_stream_id = stream["rows"][0][0]
+                max_stream_id = stream["position"]
+                preserve_fn(pusher_pool.on_new_notifications)(
+                    min_stream_id, max_stream_id
+                )
+
+            stream = results.get("receipts")
+            if stream and stream["rows"]:
+                rows = stream["rows"]
+                affected_room_ids = set(row[1] for row in rows)
+                min_stream_id = rows[0][0]
+                max_stream_id = stream["position"]
+                preserve_fn(pusher_pool.on_new_receipts)(
+                    min_stream_id, max_stream_id, affected_room_ids
+                )
+
+        while True:
+            try:
+                args = store.stream_positions()
+                args["timeout"] = 30000
+                result = yield http_client.get_json(replication_url, args=args)
+                yield store.process_replication(result)
+                poke_pushers(result)
+            except:
+                logger.exception("Error replicating from %r", replication_url)
+                yield sleep(30)
 
 
 def start(config_options):
@@ -207,7 +245,7 @@ def start(config_options):
 
     assert config.worker_app == "synapse.app.pusher"
 
-    setup_logging(config, use_worker_options=True)
+    setup_logging(config.worker_log_config, config.worker_log_file)
 
     events.USE_FROZEN_DICTS = config.use_frozen_dicts
 
@@ -236,14 +274,34 @@ def start(config_options):
     ps.setup()
     ps.start_listening(config.worker_listeners)
 
+    def run():
+        with LoggingContext("run"):
+            logger.info("Running")
+            change_resource_limit(config.soft_file_limit)
+            if config.gc_thresholds:
+                gc.set_threshold(*config.gc_thresholds)
+            reactor.run()
+
     def start():
+        ps.replicate()
         ps.get_pusherpool().start()
         ps.get_datastore().start_profiling()
         ps.get_state_handler().start_caching()
 
     reactor.callWhenRunning(start)
 
-    _base.start_worker_reactor("synapse-pusher", config)
+    if config.worker_daemonize:
+        daemon = Daemonize(
+            app="synapse-pusher",
+            pid=config.worker_pid_file,
+            action=run,
+            auto_close_fds=False,
+            verbose=True,
+            logger=logger,
+        )
+        daemon.start()
+    else:
+        run()
 
 
 if __name__ == '__main__':

synapse/app/synchrotron.py

@@ -13,51 +13,58 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import contextlib
-import logging
-import sys
 
 import synapse
-from synapse.api.constants import EventTypes
-from synapse.app import _base
+
+from synapse.api.constants import EventTypes, PresenceState
 from synapse.config._base import ConfigError
 from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
-from synapse.handlers.presence import PresenceHandler, get_interested_parties
-from synapse.http.server import JsonResource
+from synapse.events import FrozenEvent
+from synapse.handlers.presence import PresenceHandler
 from synapse.http.site import SynapseSite
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
+from synapse.http.server import JsonResource
+from synapse.metrics.resource import MetricsResource, METRICS_PREFIX
+from synapse.rest.client.v2_alpha import sync
+from synapse.rest.client.v1 import events
+from synapse.rest.client.v1.room import RoomInitialSyncRestServlet
+from synapse.rest.client.v1.initial_sync import InitialSyncRestServlet
 from synapse.replication.slave.storage._base import BaseSlavedStore
+from synapse.replication.slave.storage.events import SlavedEventStore
+from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
 from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
 from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
-from synapse.replication.slave.storage.client_ips import SlavedClientIpStore
+from synapse.replication.slave.storage.registration import SlavedRegistrationStore
+from synapse.replication.slave.storage.filtering import SlavedFilteringStore
+from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
+from synapse.replication.slave.storage.presence import SlavedPresenceStore
 from synapse.replication.slave.storage.deviceinbox import SlavedDeviceInboxStore
 from synapse.replication.slave.storage.devices import SlavedDeviceStore
-from synapse.replication.slave.storage.events import SlavedEventStore
-from synapse.replication.slave.storage.filtering import SlavedFilteringStore
-from synapse.replication.slave.storage.presence import SlavedPresenceStore
-from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
-from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
-from synapse.replication.slave.storage.registration import SlavedRegistrationStore
 from synapse.replication.slave.storage.room import RoomStore
-from synapse.replication.slave.storage.groups import SlavedGroupServerStore
-from synapse.replication.tcp.client import ReplicationClientHandler
-from synapse.rest.client.v1 import events
-from synapse.rest.client.v1.initial_sync import InitialSyncRestServlet
-from synapse.rest.client.v1.room import RoomInitialSyncRestServlet
-from synapse.rest.client.v2_alpha import sync
 from synapse.server import HomeServer
+from synapse.storage.client_ips import ClientIpStore
 from synapse.storage.engines import create_engine
-from synapse.storage.presence import UserPresenceState
+from synapse.storage.presence import PresenceStore, UserPresenceState
 from synapse.storage.roommember import RoomMemberStore
+from synapse.util.async import sleep
 from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext, preserve_fn
 from synapse.util.manhole import manhole
+from synapse.util.rlimit import change_resource_limit
 from synapse.util.stringutils import random_string
 from synapse.util.versionstring import get_version_string
-from twisted.internet import defer, reactor
+
+from twisted.internet import reactor, defer
 from twisted.web.resource import Resource
 
+from daemonize import Daemonize
+
+import sys
+import logging
+import contextlib
+import gc
+import ujson as json
+
 logger = logging.getLogger("synapse.app.synchrotron")
 
 
@@ -70,12 +77,11 @@ class SynchrotronSlavedStore(
     SlavedRegistrationStore,
     SlavedFilteringStore,
     SlavedPresenceStore,
-    SlavedGroupServerStore,
     SlavedDeviceInboxStore,
     SlavedDeviceStore,
-    SlavedClientIpStore,
     RoomStore,
     BaseSlavedStore,
+    ClientIpStore,  # After BaseSlavedStore because the constructor is different
 ):
     who_forgot_in_room = (
         RoomMemberStore.__dict__["who_forgot_in_room"]
@@ -85,17 +91,27 @@ class SynchrotronSlavedStore(
         RoomMemberStore.__dict__["did_forget"]
     )
 
+    # XXX: This is a bit broken because we don't persist the accepted list in a
+    # way that can be replicated. This means that we don't have a way to
+    # invalidate the cache correctly.
+    get_presence_list_accepted = PresenceStore.__dict__[
+        "get_presence_list_accepted"
+    ]
+    get_presence_list_observers_accepted = PresenceStore.__dict__[
+        "get_presence_list_observers_accepted"
+    ]
+
 
 UPDATE_SYNCING_USERS_MS = 10 * 1000
 
 
 class SynchrotronPresence(object):
     def __init__(self, hs):
-        self.hs = hs
         self.is_mine_id = hs.is_mine_id
         self.http_client = hs.get_simple_http_client()
         self.store = hs.get_datastore()
         self.user_to_num_current_syncs = {}
+        self.syncing_users_url = hs.config.worker_replication_url + "/syncing_users"
         self.clock = hs.get_clock()
         self.notifier = hs.get_notifier()
 
@@ -105,52 +121,17 @@ class SynchrotronPresence(object):
             for state in active_presence
         }
 
-        # user_id -> last_sync_ms. Lists the users that have stopped syncing
-        # but we haven't notified the master of that yet
-        self.users_going_offline = {}
-
-        self._send_stop_syncing_loop = self.clock.looping_call(
-            self.send_stop_syncing, 10 * 1000
-        )
-
         self.process_id = random_string(16)
         logger.info("Presence process_id is %r", self.process_id)
 
-    def send_user_sync(self, user_id, is_syncing, last_sync_ms):
-        self.hs.get_tcp_replication().send_user_sync(user_id, is_syncing, last_sync_ms)
+        self._sending_sync = False
+        self._need_to_send_sync = False
+        self.clock.looping_call(
+            self._send_syncing_users_regularly,
+            UPDATE_SYNCING_USERS_MS,
+        )
 
-    def mark_as_coming_online(self, user_id):
-        """A user has started syncing. Send a UserSync to the master, unless they
-        had recently stopped syncing.
-
-        Args:
-            user_id (str)
-        """
-        going_offline = self.users_going_offline.pop(user_id, None)
-        if not going_offline:
-            # Safe to skip because we haven't yet told the master they were offline
-            self.send_user_sync(user_id, True, self.clock.time_msec())
-
-    def mark_as_going_offline(self, user_id):
-        """A user has stopped syncing. We wait before notifying the master as
-        its likely they'll come back soon. This allows us to avoid sending
-        a stopped syncing immediately followed by a started syncing notification
-        to the master
-
-        Args:
-            user_id (str)
-        """
-        self.users_going_offline[user_id] = self.clock.time_msec()
-
-    def send_stop_syncing(self):
-        """Check if there are any users who have stopped syncing a while ago
-        and haven't come back yet. If there are poke the master about them.
-        """
-        now = self.clock.time_msec()
-        for user_id, last_sync_ms in self.users_going_offline.items():
-            if now - last_sync_ms > 10 * 1000:
-                self.users_going_offline.pop(user_id, None)
-                self.send_user_sync(user_id, False, last_sync_ms)
+        reactor.addSystemEventTrigger("before", "shutdown", self._on_shutdown)
 
     def set_state(self, user, state, ignore_status_msg=False):
         # TODO Hows this supposed to work?
@@ -158,16 +139,18 @@ class SynchrotronPresence(object):
 
     get_states = PresenceHandler.get_states.__func__
     get_state = PresenceHandler.get_state.__func__
+    _get_interested_parties = PresenceHandler._get_interested_parties.__func__
     current_state_for_users = PresenceHandler.current_state_for_users.__func__
 
+    @defer.inlineCallbacks
     def user_syncing(self, user_id, affect_presence):
         if affect_presence:
             curr_sync = self.user_to_num_current_syncs.get(user_id, 0)
             self.user_to_num_current_syncs[user_id] = curr_sync + 1
-
-            # If we went from no in flight sync to some, notify replication
-            if self.user_to_num_current_syncs[user_id] == 1:
-                self.mark_as_coming_online(user_id)
+            prev_states = yield self.current_state_for_users([user_id])
+            if prev_states[user_id].state == PresenceState.OFFLINE:
+                # TODO: Don't block the sync request on this HTTP hit.
+                yield self._send_syncing_users_now()
 
         def _end():
             # We check that the user_id is in user_to_num_current_syncs because
@@ -176,10 +159,6 @@ class SynchrotronPresence(object):
             if affect_presence and user_id in self.user_to_num_current_syncs:
                 self.user_to_num_current_syncs[user_id] -= 1
 
-                # If we went from one in flight sync to non, notify replication
-                if self.user_to_num_current_syncs[user_id] == 0:
-                    self.mark_as_going_offline(user_id)
-
         @contextlib.contextmanager
         def _user_syncing():
             try:
@@ -187,12 +166,56 @@ class SynchrotronPresence(object):
             finally:
                 _end()
 
-        return defer.succeed(_user_syncing())
+        defer.returnValue(_user_syncing())
+
+    @defer.inlineCallbacks
+    def _on_shutdown(self):
+        # When the synchrotron is shutdown tell the master to clear the in
+        # progress syncs for this process
+        self.user_to_num_current_syncs.clear()
+        yield self._send_syncing_users_now()
+
+    def _send_syncing_users_regularly(self):
+        # Only send an update if we aren't in the middle of sending one.
+        if not self._sending_sync:
+            preserve_fn(self._send_syncing_users_now)()
+
+    @defer.inlineCallbacks
+    def _send_syncing_users_now(self):
+        if self._sending_sync:
+            # We don't want to race with sending another update.
+            # Instead we wait for that update to finish and send another
+            # update afterwards.
+            self._need_to_send_sync = True
+            return
+
+        # Flag that we are sending an update.
+        self._sending_sync = True
+
+        yield self.http_client.post_json_get_json(self.syncing_users_url, {
+            "process_id": self.process_id,
+            "syncing_users": [
+                user_id for user_id, count in self.user_to_num_current_syncs.items()
+                if count > 0
+            ],
+        })
+
+        # Unset the flag as we are no longer sending an update.
+        self._sending_sync = False
+        if self._need_to_send_sync:
+            # If something happened while we were sending the update then
+            # we might need to send another update.
+            # TODO: Check if the update that was sent matches the current state
+            # as we only need to send an update if they are different.
+            self._need_to_send_sync = False
+            yield self._send_syncing_users_now()
 
     @defer.inlineCallbacks
     def notify_from_replication(self, states, stream_id):
-        parties = yield get_interested_parties(self.store, states)
-        room_ids_to_states, users_to_states = parties
+        parties = yield self._get_interested_parties(
+            states, calculate_remote_hosts=False
+        )
+        room_ids_to_states, users_to_states, _ = parties
 
         self.notifier.on_new_event(
             "presence_key", stream_id, rooms=room_ids_to_states.keys(),
@@ -200,24 +223,26 @@ class SynchrotronPresence(object):
         )
 
     @defer.inlineCallbacks
-    def process_replication_rows(self, token, rows):
-        states = [UserPresenceState(
-            row.user_id, row.state, row.last_active_ts,
-            row.last_federation_update_ts, row.last_user_sync_ts, row.status_msg,
-            row.currently_active
-        ) for row in rows]
+    def process_replication(self, result):
+        stream = result.get("presence", {"rows": []})
+        states = []
+        for row in stream["rows"]:
+            (
+                position, user_id, state, last_active_ts,
+                last_federation_update_ts, last_user_sync_ts, status_msg,
+                currently_active
+            ) = row
+            state = UserPresenceState(
+                user_id, state, last_active_ts,
+                last_federation_update_ts, last_user_sync_ts, status_msg,
+                currently_active
+            )
+            self.user_to_current_state[user_id] = state
+            states.append(state)
 
-        for state in states:
-            self.user_to_current_state[row.user_id] = state
-
-        stream_id = token
-        yield self.notify_from_replication(states, stream_id)
-
-    def get_currently_syncing_users(self):
-        return [
-            user_id for user_id, count in self.user_to_num_current_syncs.iteritems()
-            if count > 0
-        ]
+        if states and "position" in stream:
+            stream_id = int(stream["position"])
+            yield self.notify_from_replication(states, stream_id)
 
 
 class SynchrotronTyping(object):
@@ -232,12 +257,16 @@ class SynchrotronTyping(object):
         # value which we *must* use for the next replication request.
         return {"typing": self._latest_room_serial}
 
-    def process_replication_rows(self, token, rows):
-        self._latest_room_serial = token
+    def process_replication(self, result):
+        stream = result.get("typing")
+        if stream:
+            self._latest_room_serial = int(stream["position"])
 
-        for row in rows:
-            self._room_serials[row.room_id] = token
-            self._room_typing[row.room_id] = row.user_ids
+            for row in stream["rows"]:
+                position, room_id, typing_json = row
+                typing = json.loads(typing_json)
+                self._room_serials[room_id] = position
+                self._room_typing[room_id] = typing
 
 
 class SynchrotronApplicationService(object):
@@ -322,10 +351,114 @@ class SynchrotronServer(HomeServer):
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 
-        self.get_tcp_replication().start_replication(self)
+    @defer.inlineCallbacks
+    def replicate(self):
+        http_client = self.get_simple_http_client()
+        store = self.get_datastore()
+        replication_url = self.config.worker_replication_url
+        notifier = self.get_notifier()
+        presence_handler = self.get_presence_handler()
+        typing_handler = self.get_typing_handler()
 
-    def build_tcp_replication(self):
-        return SyncReplicationHandler(self)
+        def notify_from_stream(
+            result, stream_name, stream_key, room=None, user=None
+        ):
+            stream = result.get(stream_name)
+            if stream:
+                position_index = stream["field_names"].index("position")
+                if room:
+                    room_index = stream["field_names"].index(room)
+                if user:
+                    user_index = stream["field_names"].index(user)
+
+                users = ()
+                rooms = ()
+                for row in stream["rows"]:
+                    position = row[position_index]
+
+                    if user:
+                        users = (row[user_index],)
+
+                    if room:
+                        rooms = (row[room_index],)
+
+                    notifier.on_new_event(
+                        stream_key, position, users=users, rooms=rooms
+                    )
+
+        @defer.inlineCallbacks
+        def notify_device_list_update(result):
+            stream = result.get("device_lists")
+            if not stream:
+                return
+
+            position_index = stream["field_names"].index("position")
+            user_index = stream["field_names"].index("user_id")
+
+            for row in stream["rows"]:
+                position = row[position_index]
+                user_id = row[user_index]
+
+                rooms = yield store.get_rooms_for_user(user_id)
+                room_ids = [r.room_id for r in rooms]
+
+                notifier.on_new_event(
+                    "device_list_key", position, rooms=room_ids,
+                )
+
+        @defer.inlineCallbacks
+        def notify(result):
+            stream = result.get("events")
+            if stream:
+                max_position = stream["position"]
+                for row in stream["rows"]:
+                    position = row[0]
+                    internal = json.loads(row[1])
+                    event_json = json.loads(row[2])
+                    event = FrozenEvent(event_json, internal_metadata_dict=internal)
+                    extra_users = ()
+                    if event.type == EventTypes.Member:
+                        extra_users = (event.state_key,)
+                    notifier.on_new_room_event(
+                        event, position, max_position, extra_users
+                    )
+
+            notify_from_stream(
+                result, "push_rules", "push_rules_key", user="user_id"
+            )
+            notify_from_stream(
+                result, "user_account_data", "account_data_key", user="user_id"
+            )
+            notify_from_stream(
+                result, "room_account_data", "account_data_key", user="user_id"
+            )
+            notify_from_stream(
+                result, "tag_account_data", "account_data_key", user="user_id"
+            )
+            notify_from_stream(
+                result, "receipts", "receipt_key", room="room_id"
+            )
+            notify_from_stream(
+                result, "typing", "typing_key", room="room_id"
+            )
+            notify_from_stream(
+                result, "to_device", "to_device_key", user="user_id"
+            )
+            yield notify_device_list_update(result)
+
+        while True:
+            try:
+                args = store.stream_positions()
+                args.update(typing_handler.stream_positions())
+                args["timeout"] = 30000
+                result = yield http_client.get_json(replication_url, args=args)
+                yield store.process_replication(result)
+                typing_handler.process_replication(result)
+                yield presence_handler.process_replication(result)
+                yield notify(result)
+            except:
+                logger.exception("Error replicating from %r", replication_url)
+                yield sleep(5)
 
     def build_presence_handler(self):
         return SynchrotronPresence(self)
@@ -334,83 +467,6 @@ class SynchrotronServer(HomeServer):
         return SynchrotronTyping(self)
 
 
-class SyncReplicationHandler(ReplicationClientHandler):
-    def __init__(self, hs):
-        super(SyncReplicationHandler, self).__init__(hs.get_datastore())
-
-        self.store = hs.get_datastore()
-        self.typing_handler = hs.get_typing_handler()
-        self.presence_handler = hs.get_presence_handler()
-        self.notifier = hs.get_notifier()
-
-        self.presence_handler.sync_callback = self.send_user_sync
-
-    def on_rdata(self, stream_name, token, rows):
-        super(SyncReplicationHandler, self).on_rdata(stream_name, token, rows)
-
-        preserve_fn(self.process_and_notify)(stream_name, token, rows)
-
-    def get_streams_to_replicate(self):
-        args = super(SyncReplicationHandler, self).get_streams_to_replicate()
-        args.update(self.typing_handler.stream_positions())
-        return args
-
-    def get_currently_syncing_users(self):
-        return self.presence_handler.get_currently_syncing_users()
-
-    @defer.inlineCallbacks
-    def process_and_notify(self, stream_name, token, rows):
-        if stream_name == "events":
-            # We shouldn't get multiple rows per token for events stream, so
-            # we don't need to optimise this for multiple rows.
-            for row in rows:
-                event = yield self.store.get_event(row.event_id)
-                extra_users = ()
-                if event.type == EventTypes.Member:
-                    extra_users = (event.state_key,)
-                max_token = self.store.get_room_max_stream_ordering()
-                self.notifier.on_new_room_event(
-                    event, token, max_token, extra_users
-                )
-        elif stream_name == "push_rules":
-            self.notifier.on_new_event(
-                "push_rules_key", token, users=[row.user_id for row in rows],
-            )
-        elif stream_name in ("account_data", "tag_account_data",):
-            self.notifier.on_new_event(
-                "account_data_key", token, users=[row.user_id for row in rows],
-            )
-        elif stream_name == "receipts":
-            self.notifier.on_new_event(
-                "receipt_key", token, rooms=[row.room_id for row in rows],
-            )
-        elif stream_name == "typing":
-            self.typing_handler.process_replication_rows(token, rows)
-            self.notifier.on_new_event(
-                "typing_key", token, rooms=[row.room_id for row in rows],
-            )
-        elif stream_name == "to_device":
-            entities = [row.entity for row in rows if row.entity.startswith("@")]
-            if entities:
-                self.notifier.on_new_event(
-                    "to_device_key", token, users=entities,
-                )
-        elif stream_name == "device_lists":
-            all_room_ids = set()
-            for row in rows:
-                room_ids = yield self.store.get_rooms_for_user(row.user_id)
-                all_room_ids.update(room_ids)
-            self.notifier.on_new_event(
-                "device_list_key", token, rooms=all_room_ids,
-            )
-        elif stream_name == "presence":
-            yield self.presence_handler.process_replication_rows(token, rows)
-        elif stream_name == "receipts":
-            self.notifier.on_new_event(
-                "groups_key", token, users=[row.user_id for row in rows],
-            )
-
-
 def start(config_options):
     try:
         config = HomeServerConfig.load_config(
@@ -422,7 +478,7 @@ def start(config_options):
 
     assert config.worker_app == "synapse.app.synchrotron"
 
-    setup_logging(config, use_worker_options=True)
+    setup_logging(config.worker_log_config, config.worker_log_file)
 
     synapse.events.USE_FROZEN_DICTS = config.use_frozen_dicts
 
@@ -440,13 +496,33 @@ def start(config_options):
     ss.setup()
     ss.start_listening(config.worker_listeners)
 
+    def run():
+        with LoggingContext("run"):
+            logger.info("Running")
+            change_resource_limit(config.soft_file_limit)
+            if config.gc_thresholds:
+                gc.set_threshold(*config.gc_thresholds)
+            reactor.run()
+
     def start():
         ss.get_datastore().start_profiling()
+        ss.replicate()
         ss.get_state_handler().start_caching()
 
     reactor.callWhenRunning(start)
 
-    _base.start_worker_reactor("synapse-synchrotron", config)
+    if config.worker_daemonize:
+        daemon = Daemonize(
+            app="synapse-synchrotron",
+            pid=config.worker_pid_file,
+            action=run,
+            auto_close_fds=False,
+            verbose=True,
+            logger=logger,
+        )
+        daemon.start()
+    else:
+        run()
 
 
 if __name__ == '__main__':

synapse/app/synctl.py

@@ -23,27 +23,14 @@ import signal
 import subprocess
 import sys
 import yaml
-import errno
-import time
 
 SYNAPSE = [sys.executable, "-B", "-m", "synapse.app.homeserver"]
 
 GREEN = "\x1b[1;32m"
-YELLOW = "\x1b[1;33m"
 RED = "\x1b[1;31m"
 NORMAL = "\x1b[m"
 
 
-def pid_running(pid):
-    try:
-        os.kill(pid, 0)
-        return True
-    except OSError, err:
-        if err.errno == errno.EPERM:
-            return True
-        return False
-
-
 def write(message, colour=NORMAL, stream=sys.stdout):
     if colour == NORMAL:
         stream.write(message + "\n")
@@ -51,11 +38,6 @@ def write(message, colour=NORMAL, stream=sys.stdout):
         stream.write(colour + message + NORMAL + "\n")
 
 
-def abort(message, colour=RED, stream=sys.stderr):
-    write(message, colour, stream)
-    sys.exit(1)
-
-
 def start(configfile):
     write("Starting ...")
     args = SYNAPSE
@@ -63,8 +45,7 @@ def start(configfile):
 
     try:
         subprocess.check_call(args)
-        write("started synapse.app.homeserver(%r)" %
-              (configfile,), colour=GREEN)
+        write("started synapse.app.homeserver(%r)" % (configfile,), colour=GREEN)
     except subprocess.CalledProcessError as e:
         write(
             "error starting (exit code: %d); see above for logs" % e.returncode,
@@ -95,16 +76,8 @@ def start_worker(app, configfile, worker_configfile):
 def stop(pidfile, app):
     if os.path.exists(pidfile):
         pid = int(open(pidfile).read())
-        try:
-            os.kill(pid, signal.SIGTERM)
-            write("stopped %s" % (app,), colour=GREEN)
-        except OSError, err:
-            if err.errno == errno.ESRCH:
-                write("%s not running" % (app,), colour=YELLOW)
-            elif err.errno == errno.EPERM:
-                abort("Cannot stop %s: Operation not permitted" % (app,))
-            else:
-                abort("Cannot stop %s: Unknown error" % (app,))
+        os.kill(pid, signal.SIGTERM)
+        write("stopped %s" % (app,), colour=GREEN)
 
 
 Worker = collections.namedtuple("Worker", [
@@ -125,7 +98,7 @@ def main():
         "configfile",
         nargs="?",
         default="homeserver.yaml",
-        help="the homeserver config file, defaults to homeserver.yaml",
+        help="the homeserver config file, defaults to homserver.yaml",
     )
     parser.add_argument(
         "-w", "--worker",
@@ -202,8 +175,7 @@ def main():
         worker_app = worker_config["worker_app"]
         worker_pidfile = worker_config["worker_pid_file"]
         worker_daemonize = worker_config["worker_daemonize"]
-        assert worker_daemonize, "In config %r: expected '%s' to be True" % (
-            worker_configfile, "worker_daemonize")
+        assert worker_daemonize  # TODO print something more user friendly
         worker_cache_factor = worker_config.get("synctl_cache_factor")
         workers.append(Worker(
             worker_app, worker_configfile, worker_pidfile, worker_cache_factor,
@@ -218,25 +190,10 @@ def main():
         if start_stop_synapse:
             stop(pidfile, "synapse.app.homeserver")
 
-    # Wait for synapse to actually shutdown before starting it again
-    if action == "restart":
-        running_pids = []
-        if start_stop_synapse and os.path.exists(pidfile):
-            running_pids.append(int(open(pidfile).read()))
-        for worker in workers:
-            if os.path.exists(worker.pidfile):
-                running_pids.append(int(open(worker.pidfile).read()))
-        if len(running_pids) > 0:
-            write("Waiting for process to exit before restarting...")
-            for running_pid in running_pids:
-                while pid_running(running_pid):
-                    time.sleep(0.2)
+        # TODO: Wait for synapse to actually shutdown before starting it again
 
     if action == "start" or action == "restart":
         if start_stop_synapse:
-            # Check if synapse is already running
-            if os.path.exists(pidfile) and pid_running(int(open(pidfile).read())):
-                abort("synapse.app.homeserver already running")
             start(configfile)
 
         for worker in workers:

synapse/app/user_dir.py

@@ -1,241 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-# Copyright 2017 Vector Creations Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import logging
-import sys
-
-import synapse
-from synapse import events
-from synapse.app import _base
-from synapse.config._base import ConfigError
-from synapse.config.homeserver import HomeServerConfig
-from synapse.config.logger import setup_logging
-from synapse.crypto import context_factory
-from synapse.http.server import JsonResource
-from synapse.http.site import SynapseSite
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
-from synapse.replication.slave.storage._base import BaseSlavedStore
-from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
-from synapse.replication.slave.storage.client_ips import SlavedClientIpStore
-from synapse.replication.slave.storage.events import SlavedEventStore
-from synapse.replication.slave.storage.registration import SlavedRegistrationStore
-from synapse.replication.tcp.client import ReplicationClientHandler
-from synapse.rest.client.v2_alpha import user_directory
-from synapse.server import HomeServer
-from synapse.storage.engines import create_engine
-from synapse.storage.user_directory import UserDirectoryStore
-from synapse.util.caches.stream_change_cache import StreamChangeCache
-from synapse.util.httpresourcetree import create_resource_tree
-from synapse.util.logcontext import LoggingContext, preserve_fn
-from synapse.util.manhole import manhole
-from synapse.util.versionstring import get_version_string
-from twisted.internet import reactor
-from twisted.web.resource import Resource
-
-logger = logging.getLogger("synapse.app.user_dir")
-
-
-class UserDirectorySlaveStore(
-    SlavedEventStore,
-    SlavedApplicationServiceStore,
-    SlavedRegistrationStore,
-    SlavedClientIpStore,
-    UserDirectoryStore,
-    BaseSlavedStore,
-):
-    def __init__(self, db_conn, hs):
-        super(UserDirectorySlaveStore, self).__init__(db_conn, hs)
-
-        events_max = self._stream_id_gen.get_current_token()
-        curr_state_delta_prefill, min_curr_state_delta_id = self._get_cache_dict(
-            db_conn, "current_state_delta_stream",
-            entity_column="room_id",
-            stream_column="stream_id",
-            max_value=events_max,  # As we share the stream id with events token
-            limit=1000,
-        )
-        self._curr_state_delta_stream_cache = StreamChangeCache(
-            "_curr_state_delta_stream_cache", min_curr_state_delta_id,
-            prefilled_cache=curr_state_delta_prefill,
-        )
-
-        self._current_state_delta_pos = events_max
-
-    def stream_positions(self):
-        result = super(UserDirectorySlaveStore, self).stream_positions()
-        result["current_state_deltas"] = self._current_state_delta_pos
-        return result
-
-    def process_replication_rows(self, stream_name, token, rows):
-        if stream_name == "current_state_deltas":
-            self._current_state_delta_pos = token
-            for row in rows:
-                self._curr_state_delta_stream_cache.entity_has_changed(
-                    row.room_id, token
-                )
-        return super(UserDirectorySlaveStore, self).process_replication_rows(
-            stream_name, token, rows
-        )
-
-
-class UserDirectoryServer(HomeServer):
-    def get_db_conn(self, run_new_connection=True):
-        # Any param beginning with cp_ is a parameter for adbapi, and should
-        # not be passed to the database engine.
-        db_params = {
-            k: v for k, v in self.db_config.get("args", {}).items()
-            if not k.startswith("cp_")
-        }
-        db_conn = self.database_engine.module.connect(**db_params)
-
-        if run_new_connection:
-            self.database_engine.on_new_connection(db_conn)
-        return db_conn
-
-    def setup(self):
-        logger.info("Setting up.")
-        self.datastore = UserDirectorySlaveStore(self.get_db_conn(), self)
-        logger.info("Finished setting up.")
-
-    def _listen_http(self, listener_config):
-        port = listener_config["port"]
-        bind_addresses = listener_config["bind_addresses"]
-        site_tag = listener_config.get("tag", port)
-        resources = {}
-        for res in listener_config["resources"]:
-            for name in res["names"]:
-                if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(self)
-                elif name == "client":
-                    resource = JsonResource(self, canonical_json=False)
-                    user_directory.register_servlets(self, resource)
-                    resources.update({
-                        "/_matrix/client/r0": resource,
-                        "/_matrix/client/unstable": resource,
-                        "/_matrix/client/v2_alpha": resource,
-                        "/_matrix/client/api/v1": resource,
-                    })
-
-        root_resource = create_resource_tree(resources, Resource())
-
-        for address in bind_addresses:
-            reactor.listenTCP(
-                port,
-                SynapseSite(
-                    "synapse.access.http.%s" % (site_tag,),
-                    site_tag,
-                    listener_config,
-                    root_resource,
-                ),
-                interface=address
-            )
-
-        logger.info("Synapse user_dir now listening on port %d", port)
-
-    def start_listening(self, listeners):
-        for listener in listeners:
-            if listener["type"] == "http":
-                self._listen_http(listener)
-            elif listener["type"] == "manhole":
-                bind_addresses = listener["bind_addresses"]
-
-                for address in bind_addresses:
-                    reactor.listenTCP(
-                        listener["port"],
-                        manhole(
-                            username="matrix",
-                            password="rabbithole",
-                            globals={"hs": self},
-                        ),
-                        interface=address
-                    )
-            else:
-                logger.warn("Unrecognized listener type: %s", listener["type"])
-
-        self.get_tcp_replication().start_replication(self)
-
-    def build_tcp_replication(self):
-        return UserDirectoryReplicationHandler(self)
-
-
-class UserDirectoryReplicationHandler(ReplicationClientHandler):
-    def __init__(self, hs):
-        super(UserDirectoryReplicationHandler, self).__init__(hs.get_datastore())
-        self.user_directory = hs.get_user_directory_handler()
-
-    def on_rdata(self, stream_name, token, rows):
-        super(UserDirectoryReplicationHandler, self).on_rdata(
-            stream_name, token, rows
-        )
-        if stream_name == "current_state_deltas":
-            preserve_fn(self.user_directory.notify_new_event)()
-
-
-def start(config_options):
-    try:
-        config = HomeServerConfig.load_config(
-            "Synapse user directory", config_options
-        )
-    except ConfigError as e:
-        sys.stderr.write("\n" + e.message + "\n")
-        sys.exit(1)
-
-    assert config.worker_app == "synapse.app.user_dir"
-
-    setup_logging(config, use_worker_options=True)
-
-    events.USE_FROZEN_DICTS = config.use_frozen_dicts
-
-    database_engine = create_engine(config.database_config)
-
-    if config.update_user_directory:
-        sys.stderr.write(
-            "\nThe update_user_directory must be disabled in the main synapse process"
-            "\nbefore they can be run in a separate worker."
-            "\nPlease add ``update_user_directory: false`` to the main config"
-            "\n"
-        )
-        sys.exit(1)
-
-    # Force the pushers to start since they will be disabled in the main config
-    config.update_user_directory = True
-
-    tls_server_context_factory = context_factory.ServerContextFactory(config)
-
-    ps = UserDirectoryServer(
-        config.server_name,
-        db_config=config.database_config,
-        tls_server_context_factory=tls_server_context_factory,
-        config=config,
-        version_string="Synapse/" + get_version_string(synapse),
-        database_engine=database_engine,
-    )
-
-    ps.setup()
-    ps.start_listening(config.worker_listeners)
-
-    def start():
-        ps.get_datastore().start_profiling()
-        ps.get_state_handler().start_caching()
-
-    reactor.callWhenRunning(start)
-
-    _base.start_worker_reactor("synapse-user-dir", config)
-
-
-if __name__ == '__main__':
-    with LoggingContext("main"):
-        start(sys.argv[1:])

synapse/appservice/__init__.py

@@ -13,7 +13,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 from synapse.api.constants import EventTypes
-from synapse.util.caches.descriptors import cachedInlineCallbacks
 
 from twisted.internet import defer
 
@@ -125,23 +124,29 @@ class ApplicationService(object):
                     raise ValueError(
                         "Expected bool for 'exclusive' in ns '%s'" % ns
                     )
-                regex = regex_obj.get("regex")
-                if isinstance(regex, basestring):
-                    regex_obj["regex"] = re.compile(regex)  # Pre-compile regex
-                else:
+                if not isinstance(regex_obj.get("regex"), basestring):
                     raise ValueError(
                         "Expected string for 'regex' in ns '%s'" % ns
                     )
         return namespaces
 
-    def _matches_regex(self, test_string, namespace_key):
+    def _matches_regex(self, test_string, namespace_key, return_obj=False):
+        if not isinstance(test_string, basestring):
+            logger.error(
+                "Expected a string to test regex against, but got %s",
+                test_string
+            )
+            return False
+
         for regex_obj in self.namespaces[namespace_key]:
-            if regex_obj["regex"].match(test_string):
-                return regex_obj
-        return None
+            if re.match(regex_obj["regex"], test_string):
+                if return_obj:
+                    return regex_obj
+                return True
+        return False
 
     def _is_exclusive(self, ns_key, test_string):
-        regex_obj = self._matches_regex(test_string, ns_key)
+        regex_obj = self._matches_regex(test_string, ns_key, return_obj=True)
         if regex_obj:
             return regex_obj["exclusive"]
         return False
@@ -161,14 +166,7 @@ class ApplicationService(object):
         if not store:
             defer.returnValue(False)
 
-        does_match = yield self._matches_user_in_member_list(event.room_id, store)
-        defer.returnValue(does_match)
-
-    @cachedInlineCallbacks(num_args=1, cache_context=True)
-    def _matches_user_in_member_list(self, room_id, store, cache_context):
-        member_list = yield store.get_users_in_room(
-            room_id, on_invalidate=cache_context.invalidate
-        )
+        member_list = yield store.get_users_in_room(event.room_id)
 
         # check joined member events
         for user_id in member_list:
@@ -221,10 +219,10 @@ class ApplicationService(object):
         )
 
     def is_interested_in_alias(self, alias):
-        return bool(self._matches_regex(alias, ApplicationService.NS_ALIASES))
+        return self._matches_regex(alias, ApplicationService.NS_ALIASES)
 
     def is_interested_in_room(self, room_id):
-        return bool(self._matches_regex(room_id, ApplicationService.NS_ROOMS))
+        return self._matches_regex(room_id, ApplicationService.NS_ROOMS)
 
     def is_exclusive_user(self, user_id):
         return (
@@ -241,16 +239,6 @@ class ApplicationService(object):
     def is_exclusive_room(self, room_id):
         return self._is_exclusive(ApplicationService.NS_ROOMS, room_id)
 
-    def get_exlusive_user_regexes(self):
-        """Get the list of regexes used to determine if a user is exclusively
-        registered by the AS
-        """
-        return [
-            regex_obj["regex"]
-            for regex_obj in self.namespaces[ApplicationService.NS_USERS]
-            if regex_obj["exclusive"]
-        ]
-
     def is_rate_limited(self):
         return self.rate_limited
 

synapse/appservice/api.py

@@ -18,7 +18,6 @@ from synapse.api.constants import ThirdPartyEntityKind
 from synapse.api.errors import CodeMessageException
 from synapse.http.client import SimpleHttpClient
 from synapse.events.utils import serialize_event
-from synapse.util.logcontext import preserve_fn, make_deferred_yieldable
 from synapse.util.caches.response_cache import ResponseCache
 from synapse.types import ThirdPartyInstanceID
 
@@ -193,12 +192,9 @@ class ApplicationServiceApi(SimpleHttpClient):
                 defer.returnValue(None)
 
         key = (service.id, protocol)
-        result = self.protocol_meta_cache.get(key)
-        if not result:
-            result = self.protocol_meta_cache.set(
-                key, preserve_fn(_get)()
-            )
-        return make_deferred_yieldable(result)
+        return self.protocol_meta_cache.get(key) or (
+            self.protocol_meta_cache.set(key, _get())
+        )
 
     @defer.inlineCallbacks
     def push_bulk(self, service, events, txn_id=None):

synapse/appservice/scheduler.py

@@ -123,7 +123,7 @@ class _ServiceQueuer(object):
                 with Measure(self.clock, "servicequeuer.send"):
                     try:
                         yield self.txn_ctrl.send(service, events)
-                    except Exception:
+                    except:
                         logger.exception("AS request failed")
         finally:
             self.requests_in_flight.discard(service.id)

synapse/config/_base.py

@@ -81,38 +81,22 @@ class Config(object):
     def abspath(file_path):
         return os.path.abspath(file_path) if file_path else file_path
 
-    @classmethod
-    def path_exists(cls, file_path):
-        """Check if a file exists
-
-        Unlike os.path.exists, this throws an exception if there is an error
-        checking if the file exists (for example, if there is a perms error on
-        the parent dir).
-
-        Returns:
-            bool: True if the file exists; False if not.
-        """
-        try:
-            os.stat(file_path)
-            return True
-        except OSError as e:
-            if e.errno != errno.ENOENT:
-                raise e
-            return False
-
     @classmethod
     def check_file(cls, file_path, config_name):
         if file_path is None:
             raise ConfigError(
                 "Missing config for %s."
+                " You must specify a path for the config file. You can "
+                "do this with the -c or --config-path option. "
+                "Adding --generate-config along with --server-name "
+                "<server name> will generate a config file at the given path."
                 % (config_name,)
             )
-        try:
-            os.stat(file_path)
-        except OSError as e:
+        if not os.path.exists(file_path):
             raise ConfigError(
-                "Error accessing file '%s' (config for %s): %s"
-                % (file_path, config_name, e.strerror)
+                "File %s config for %s doesn't exist."
+                " Try running again with --generate-config"
+                % (file_path, config_name,)
             )
         return cls.abspath(file_path)
 
@@ -264,7 +248,7 @@ class Config(object):
                     " -c CONFIG-FILE\""
                 )
             (config_path,) = config_files
-            if not cls.path_exists(config_path):
+            if not os.path.exists(config_path):
                 if config_args.keys_directory:
                     config_dir_path = config_args.keys_directory
                 else:
@@ -277,7 +261,7 @@ class Config(object):
                         "Must specify a server_name to a generate config for."
                         " Pass -H server.name."
                     )
-                if not cls.path_exists(config_dir_path):
+                if not os.path.exists(config_dir_path):
                     os.makedirs(config_dir_path)
                 with open(config_path, "wb") as config_file:
                     config_bytes, config = obj.generate_config(

synapse/config/cas.py

@@ -41,7 +41,7 @@ class CasConfig(Config):
         #cas_config:
         #   enabled: true
         #   server_url: "https://cas-server.com"
-        #   service_url: "https://homeserver.domain.com:8448"
+        #   service_url: "https://homesever.domain.com:8448"
         #   #required_attributes:
         #   #    name: value
         """

synapse/config/emailconfig.py

@@ -71,15 +71,6 @@ class EmailConfig(Config):
             self.email_riot_base_url = email_config.get(
                 "riot_base_url", None
             )
-            self.email_smtp_user = email_config.get(
-                "smtp_user", None
-            )
-            self.email_smtp_pass = email_config.get(
-                "smtp_pass", None
-            )
-            self.require_transport_security = email_config.get(
-                "require_transport_security", False
-            )
             if "app_name" in email_config:
                 self.email_app_name = email_config["app_name"]
             else:
@@ -100,17 +91,10 @@ class EmailConfig(Config):
         # Defining a custom URL for Riot is only needed if email notifications
         # should contain links to a self-hosted installation of Riot; when set
         # the "app_name" setting is ignored.
-        #
-        # If your SMTP server requires authentication, the optional smtp_user &
-        # smtp_pass variables should be used
-        #
         #email:
         #   enable_notifs: false
         #   smtp_host: "localhost"
         #   smtp_port: 25
-        #   smtp_user: "exampleusername"
-        #   smtp_pass: "examplepassword"
-        #   require_transport_security: False
         #   notif_from: "Your Friendly %(app)s Home Server <noreply@example.com>"
         #   app_name: Matrix
         #   template_dir: res/templates

synapse/config/groups.py

@@ -1,32 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2017 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from ._base import Config
-
-
-class GroupsConfig(Config):
-    def read_config(self, config):
-        self.enable_group_creation = config.get("enable_group_creation", False)
-        self.group_creation_prefix = config.get("group_creation_prefix", "")
-
-    def default_config(self, **kwargs):
-        return """\
-        # Whether to allow non server admins to create groups on this server
-        enable_group_creation: false
-
-        # If enabled, non server admins can only create groups with local parts
-        # starting with this prefix
-        # group_creation_prefix: "unofficial/"
-        """

synapse/config/homeserver.py

@@ -33,9 +33,6 @@ from .jwt import JWTConfig
 from .password_auth_providers import PasswordAuthProviderConfig
 from .emailconfig import EmailConfig
 from .workers import WorkerConfig
-from .push import PushConfig
-from .spam_checker import SpamCheckerConfig
-from .groups import GroupsConfig
 
 
 class HomeServerConfig(TlsConfig, ServerConfig, DatabaseConfig, LoggingConfig,
@@ -43,8 +40,7 @@ class HomeServerConfig(TlsConfig, ServerConfig, DatabaseConfig, LoggingConfig,
                        VoipConfig, RegistrationConfig, MetricsConfig, ApiConfig,
                        AppServiceConfig, KeyConfig, SAML2Config, CasConfig,
                        JWTConfig, PasswordConfig, EmailConfig,
-                       WorkerConfig, PasswordAuthProviderConfig, PushConfig,
-                       SpamCheckerConfig, GroupsConfig,):
+                       WorkerConfig, PasswordAuthProviderConfig,):
     pass
 
 

synapse/config/key.py

@@ -118,9 +118,10 @@ class KeyConfig(Config):
         signing_keys = self.read_file(signing_key_path, "signing_key")
         try:
             return read_signing_keys(signing_keys.splitlines(True))
-        except Exception as e:
+        except Exception:
             raise ConfigError(
-                "Error reading signing_key: %s" % (str(e))
+                "Error reading signing_key."
+                " Try running again with --generate-config"
             )
 
     def read_old_signing_keys(self, old_signing_keys):
@@ -140,8 +141,7 @@ class KeyConfig(Config):
 
     def generate_files(self, config):
         signing_key_path = config["signing_key_path"]
-
-        if not self.path_exists(signing_key_path):
+        if not os.path.exists(signing_key_path):
             with open(signing_key_path, "w") as signing_key_file:
                 key_id = "a_" + random_string(4)
                 write_signing_keys(

synapse/config/logger.py

@@ -45,6 +45,7 @@ handlers:
     maxBytes: 104857600
     backupCount: 10
     filters: [context]
+    level: INFO
   console:
     class: logging.StreamHandler
     formatter: precise
@@ -55,8 +56,6 @@ loggers:
         level: INFO
 
     synapse.storage.SQL:
-        # beware: increasing this to DEBUG will make synapse log sensitive
-        # information such as access tokens.
         level: INFO
 
 root:
@@ -69,7 +68,6 @@ class LoggingConfig(Config):
 
     def read_config(self, config):
         self.verbosity = config.get("verbose", 0)
-        self.no_redirect_stdio = config.get("no_redirect_stdio", False)
         self.log_config = self.abspath(config.get("log_config"))
         self.log_file = self.abspath(config.get("log_file"))
 
@@ -79,10 +77,10 @@ class LoggingConfig(Config):
             os.path.join(config_dir_path, server_name + ".log.config")
         )
         return """
-        # Logging verbosity level. Ignored if log_config is specified.
+        # Logging verbosity level.
         verbose: 0
 
-        # File to write logging to. Ignored if log_config is specified.
+        # File to write logging to
         log_file: "%(log_file)s"
 
         # A yaml python logging config file
@@ -92,8 +90,6 @@ class LoggingConfig(Config):
     def read_arguments(self, args):
         if args.verbose is not None:
             self.verbosity = args.verbose
-        if args.no_redirect_stdio is not None:
-            self.no_redirect_stdio = args.no_redirect_stdio
         if args.log_config is not None:
             self.log_config = args.log_config
         if args.log_file is not None:
@@ -103,22 +99,16 @@ class LoggingConfig(Config):
         logging_group = parser.add_argument_group("logging")
         logging_group.add_argument(
             '-v', '--verbose', dest="verbose", action='count',
-            help="The verbosity level. Specify multiple times to increase "
-            "verbosity. (Ignored if --log-config is specified.)"
+            help="The verbosity level."
         )
         logging_group.add_argument(
             '-f', '--log-file', dest="log_file",
-            help="File to log to. (Ignored if --log-config is specified.)"
+            help="File to log to."
         )
         logging_group.add_argument(
             '--log-config', dest="log_config", default=None,
             help="Python logging config file"
         )
-        logging_group.add_argument(
-            '-n', '--no-redirect-stdio',
-            action='store_true', default=None,
-            help="Do not redirect stdout/stderr to the log"
-        )
 
     def generate_files(self, config):
         log_config = config.get("log_config")
@@ -128,33 +118,22 @@ class LoggingConfig(Config):
                     DEFAULT_LOG_CONFIG.substitute(log_file=config["log_file"])
                 )
 
+    def setup_logging(self):
+        setup_logging(self.log_config, self.log_file, self.verbosity)
 
-def setup_logging(config, use_worker_options=False):
-    """ Set up python logging
-
-    Args:
-        config (LoggingConfig | synapse.config.workers.WorkerConfig):
-            configuration data
-
-        use_worker_options (bool): True to use 'worker_log_config' and
-            'worker_log_file' options instead of 'log_config' and 'log_file'.
-    """
-    log_config = (config.worker_log_config if use_worker_options
-                  else config.log_config)
-    log_file = (config.worker_log_file if use_worker_options
-                else config.log_file)
 
+def setup_logging(log_config=None, log_file=None, verbosity=None):
     log_format = (
         "%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s"
         " - %(message)s"
     )
-
     if log_config is None:
+
         level = logging.INFO
         level_for_storage = logging.INFO
-        if config.verbosity:
+        if verbosity:
             level = logging.DEBUG
-            if config.verbosity > 1:
+            if verbosity > 1:
                 level_for_storage = logging.DEBUG
 
         # FIXME: we need a logging.WARN for a -q quiet option
@@ -174,37 +153,24 @@ def setup_logging(config, use_worker_options=False):
                 logger.info("Closing log file due to SIGHUP")
                 handler.doRollover()
                 logger.info("Opened new log file due to SIGHUP")
+
+            # TODO(paul): obviously this is a terrible mechanism for
+            #   stealing SIGHUP, because it means no other part of synapse
+            #   can use it instead. If we want to catch SIGHUP anywhere
+            #   else as well, I'd suggest we find a nicer way to broadcast
+            #   it around.
+            if getattr(signal, "SIGHUP"):
+                signal.signal(signal.SIGHUP, sighup)
         else:
             handler = logging.StreamHandler()
-
-            def sighup(signum, stack):
-                pass
-
         handler.setFormatter(formatter)
 
         handler.addFilter(LoggingContextFilter(request=""))
 
         logger.addHandler(handler)
     else:
-        def load_log_config():
-            with open(log_config, 'r') as f:
-                logging.config.dictConfig(yaml.load(f))
-
-        def sighup(signum, stack):
-            # it might be better to use a file watcher or something for this.
-            logging.info("Reloading log config from %s due to SIGHUP",
-                         log_config)
-            load_log_config()
-
-        load_log_config()
-
-    # TODO(paul): obviously this is a terrible mechanism for
-    #   stealing SIGHUP, because it means no other part of synapse
-    #   can use it instead. If we want to catch SIGHUP anywhere
-    #   else as well, I'd suggest we find a nicer way to broadcast
-    #   it around.
-    if getattr(signal, "SIGHUP"):
-        signal.signal(signal.SIGHUP, sighup)
+        with open(log_config, 'r') as f:
+            logging.config.dictConfig(yaml.load(f))
 
     # It's critical to point twisted's internal logging somewhere, otherwise it
     # stacks up and leaks kup to 64K object;
@@ -217,7 +183,4 @@ def setup_logging(config, use_worker_options=False):
     #
     # However this may not be too much of a problem if we are just writing to a file.
     observer = STDLibLogObserver()
-    globalLogBeginner.beginLoggingTo(
-        [observer],
-        redirectStandardIO=not config.no_redirect_stdio,
-    )
+    globalLogBeginner.beginLoggingTo([observer])

synapse/config/password_auth_providers.py

@@ -13,41 +13,44 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from ._base import Config
+from ._base import Config, ConfigError
 
-from synapse.util.module_loader import load_module
-
-LDAP_PROVIDER = 'ldap_auth_provider.LdapAuthProvider'
+import importlib
 
 
 class PasswordAuthProviderConfig(Config):
     def read_config(self, config):
         self.password_providers = []
-        providers = []
 
         # We want to be backwards compatible with the old `ldap_config`
         # param.
         ldap_config = config.get("ldap_config", {})
-        if ldap_config.get("enabled", False):
-            providers.append[{
-                'module': LDAP_PROVIDER,
-                'config': ldap_config,
-            }]
+        self.ldap_enabled = ldap_config.get("enabled", False)
+        if self.ldap_enabled:
+            from ldap_auth_provider import LdapAuthProvider
+            parsed_config = LdapAuthProvider.parse_config(ldap_config)
+            self.password_providers.append((LdapAuthProvider, parsed_config))
 
-        providers.extend(config.get("password_providers", []))
+        providers = config.get("password_providers", [])
         for provider in providers:
-            mod_name = provider['module']
-
             # This is for backwards compat when the ldap auth provider resided
             # in this package.
-            if mod_name == "synapse.util.ldap_auth_provider.LdapAuthProvider":
-                mod_name = LDAP_PROVIDER
-
-            (provider_class, provider_config) = load_module({
-                "module": mod_name,
-                "config": provider['config'],
-            })
+            if provider['module'] == "synapse.util.ldap_auth_provider.LdapAuthProvider":
+                from ldap_auth_provider import LdapAuthProvider
+                provider_class = LdapAuthProvider
+            else:
+                # We need to import the module, and then pick the class out of
+                # that, so we split based on the last dot.
+                module, clz = provider['module'].rsplit(".", 1)
+                module = importlib.import_module(module)
+                provider_class = getattr(module, clz)
 
+            try:
+                provider_config = provider_class.parse_config(provider["config"])
+            except Exception as e:
+                raise ConfigError(
+                    "Failed to parse config for %r: %r" % (provider['module'], e)
+                )
             self.password_providers.append((provider_class, provider_config))
 
     def default_config(self, **kwargs):

synapse/config/push.py

@@ -1,45 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2015, 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from ._base import Config
-
-
-class PushConfig(Config):
-    def read_config(self, config):
-        self.push_redact_content = False
-
-        push_config = config.get("email", {})
-        self.push_redact_content = push_config.get("redact_content", False)
-
-    def default_config(self, config_dir_path, server_name, **kwargs):
-        return """
-        # Control how push messages are sent to google/apple to notifications.
-        # Normally every message said in a room with one or more people using
-        # mobile devices will be posted to a push server hosted by matrix.org
-        # which is registered with google and apple in order to allow push
-        # notifications to be sent to these mobile devices.
-        #
-        # Setting redact_content to true will make the push messages contain no
-        # message content which will provide increased privacy. This is a
-        # temporary solution pending improvements to Android and iPhone apps
-        # to get content from the app rather than the notification.
-        #
-        # For modern android devices the notification content will still appear
-        # because it is loaded by the app. iPhone, however will send a
-        # notification saying only that a message arrived and who it came from.
-        #
-        #push:
-        #   redact_content: false
-        """

synapse/config/registration.py

@@ -41,8 +41,6 @@ class RegistrationConfig(Config):
             self.allow_guest_access and config.get("invite_3pid_guest", False)
         )
 
-        self.auto_join_rooms = config.get("auto_join_rooms", [])
-
     def default_config(self, **kwargs):
         registration_shared_secret = random_string_with_symbols(50)
 
@@ -71,12 +69,6 @@ class RegistrationConfig(Config):
         trusted_third_party_id_servers:
             - matrix.org
             - vector.im
-            - riot.im
-
-        # Users who register on this homeserver will automatically be joined
-        # to these rooms
-        #auto_join_rooms:
-        #    - "#example:example.com"
         """ % locals()
 
     def add_arguments(self, parser):

synapse/config/repository.py

@@ -70,19 +70,7 @@ class ContentRepositoryConfig(Config):
         self.max_upload_size = self.parse_size(config["max_upload_size"])
         self.max_image_pixels = self.parse_size(config["max_image_pixels"])
         self.max_spider_size = self.parse_size(config["max_spider_size"])
-
         self.media_store_path = self.ensure_directory(config["media_store_path"])
-
-        self.backup_media_store_path = config.get("backup_media_store_path")
-        if self.backup_media_store_path:
-            self.backup_media_store_path = self.ensure_directory(
-                self.backup_media_store_path
-            )
-
-        self.synchronous_backup_media_store = config.get(
-            "synchronous_backup_media_store", False
-        )
-
         self.uploads_path = self.ensure_directory(config["uploads_path"])
         self.dynamic_thumbnails = config["dynamic_thumbnails"]
         self.thumbnail_requirements = parse_thumbnail_requirements(
@@ -127,14 +115,6 @@ class ContentRepositoryConfig(Config):
         # Directory where uploaded images and attachments are stored.
         media_store_path: "%(media_store)s"
 
-        # A secondary directory where uploaded images and attachments are
-        # stored as a backup.
-        # backup_media_store_path: "%(media_store)s"
-
-        # Whether to wait for successful write to backup media store before
-        # returning successfully.
-        # synchronous_backup_media_store: false
-
         # Directory where in-progress uploads are stored.
         uploads_path: "%(uploads_path)s"
 

synapse/config/server.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2017 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -30,25 +29,12 @@ class ServerConfig(Config):
         self.user_agent_suffix = config.get("user_agent_suffix")
         self.use_frozen_dicts = config.get("use_frozen_dicts", False)
         self.public_baseurl = config.get("public_baseurl")
-        self.cpu_affinity = config.get("cpu_affinity")
 
         # Whether to send federation traffic out in this process. This only
         # applies to some federation traffic, and so shouldn't be used to
         # "disable" federation
         self.send_federation = config.get("send_federation", True)
 
-        # Whether to update the user directory or not. This should be set to
-        # false only if we are updating the user directory in a worker
-        self.update_user_directory = config.get("update_user_directory", True)
-
-        self.filter_timeline_limit = config.get("filter_timeline_limit", -1)
-
-        # Whether we should block invites sent to users on this server
-        # (other than those sent by local server admins)
-        self.block_non_admin_invites = config.get(
-            "block_non_admin_invites", False,
-        )
-
         if self.public_baseurl is not None:
             if self.public_baseurl[-1] != '/':
                 self.public_baseurl += '/'
@@ -155,36 +141,9 @@ class ServerConfig(Config):
         # When running as a daemon, the file to store the pid in
         pid_file: %(pid_file)s
 
-        # CPU affinity mask. Setting this restricts the CPUs on which the
-        # process will be scheduled. It is represented as a bitmask, with the
-        # lowest order bit corresponding to the first logical CPU and the
-        # highest order bit corresponding to the last logical CPU. Not all CPUs
-        # may exist on a given system but a mask may specify more CPUs than are
-        # present.
-        #
-        # For example:
-        #    0x00000001  is processor #0,
-        #    0x00000003  is processors #0 and #1,
-        #    0xFFFFFFFF  is all processors (#0 through #31).
-        #
-        # Pinning a Python process to a single CPU is desirable, because Python
-        # is inherently single-threaded due to the GIL, and can suffer a
-        # 30-40%% slowdown due to cache blow-out and thread context switching
-        # if the scheduler happens to schedule the underlying threads across
-        # different cores. See
-        # https://www.mirantis.com/blog/improve-performance-python-programs-restricting-single-cpu/.
-        #
-        # cpu_affinity: 0xFFFFFFFF
-
         # Whether to serve a web client from the HTTP/HTTPS root resource.
         web_client: True
 
-        # The root directory to server for the above web client.
-        # If left undefined, synapse will serve the matrix-angular-sdk web client.
-        # Make sure matrix-angular-sdk is installed with pip if web_client is True
-        # and web_client_location is undefined
-        # web_client_location: "/path/to/web/root"
-
         # The public-facing base URL for the client API (not including _matrix/...)
         # public_baseurl: https://example.com:8448/
 
@@ -196,14 +155,6 @@ class ServerConfig(Config):
         # The GC threshold parameters to pass to `gc.set_threshold`, if defined
         # gc_thresholds: [700, 10, 10]
 
-        # Set the limit on the returned events in the timeline in the get
-        # and sync operations. The default value is -1, means no upper limit.
-        # filter_timeline_limit: 5000
-
-        # Whether room invites to users on this server should be blocked
-        # (except those sent by local server admins). The default is False.
-        # block_non_admin_invites: True
-
         # List of ports that Synapse should listen on, their purpose and their
         # configuration.
         listeners:
@@ -247,13 +198,6 @@ class ServerConfig(Config):
               - names: [federation]  # Federation APIs
                 compress: false
 
-            # optional list of additional endpoints which can be loaded via
-            # dynamic modules
-            # additional_resources:
-            #   "/_matrix/my/custom/endpoint":
-            #     module: my_module.CustomRequestHandler
-            #     config: {}
-
           # Unsecure HTTP listener,
           # For when matrix traffic passes through loadbalancer that unwraps TLS.
           - port: %(unsecure_port)s
@@ -310,7 +254,7 @@ def read_gc_thresholds(thresholds):
         return (
             int(thresholds[0]), int(thresholds[1]), int(thresholds[2]),
         )
-    except Exception:
+    except:
         raise ConfigError(
             "Value of `gc_threshold` must be a list of three integers if set"
         )

synapse/config/spam_checker.py

@@ -1,35 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2017 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from synapse.util.module_loader import load_module
-
-from ._base import Config
-
-
-class SpamCheckerConfig(Config):
-    def read_config(self, config):
-        self.spam_checker = None
-
-        provider = config.get("spam_checker", None)
-        if provider is not None:
-            self.spam_checker = load_module(provider)
-
-    def default_config(self, **kwargs):
-        return """\
-        # spam_checker:
-        #     module: "my_custom_project.SuperSpamChecker"
-        #     config:
-        #         example_option: 'things'
-        """

synapse/config/tls.py

@@ -109,12 +109,6 @@ class TlsConfig(Config):
         # key. It may be necessary to publish the fingerprints of a new
         # certificate and wait until the "valid_until_ts" of the previous key
         # responses have passed before deploying it.
-        #
-        # You can calculate a fingerprint from a given TLS listener via:
-        # openssl s_client -connect $host:$port < /dev/null 2> /dev/null |
-        #   openssl x509 -outform DER | openssl sha256 -binary | base64 | tr -d '='
-        # or by checking matrix.org/federationtester/api/report?server_name=$host
-        #
         tls_fingerprints: []
         # tls_fingerprints: [{"sha256": "<base64_encoded_sha256_fingerprint>"}]
         """ % locals()
@@ -132,7 +126,7 @@ class TlsConfig(Config):
         tls_private_key_path = config["tls_private_key_path"]
         tls_dh_params_path = config["tls_dh_params_path"]
 
-        if not self.path_exists(tls_private_key_path):
+        if not os.path.exists(tls_private_key_path):
             with open(tls_private_key_path, "w") as private_key_file:
                 tls_private_key = crypto.PKey()
                 tls_private_key.generate_key(crypto.TYPE_RSA, 2048)
@@ -147,7 +141,7 @@ class TlsConfig(Config):
                     crypto.FILETYPE_PEM, private_key_pem
                 )
 
-        if not self.path_exists(tls_certificate_path):
+        if not os.path.exists(tls_certificate_path):
             with open(tls_certificate_path, "w") as certificate_file:
                 cert = crypto.X509()
                 subject = cert.get_subject()
@@ -165,7 +159,7 @@ class TlsConfig(Config):
 
                 certificate_file.write(cert_pem)
 
-        if not self.path_exists(tls_dh_params_path):
+        if not os.path.exists(tls_dh_params_path):
             if GENERATE_DH_PARAMS:
                 subprocess.check_call([
                     "openssl", "dhparam",

synapse/config/voip.py

@@ -23,7 +23,6 @@ class VoipConfig(Config):
         self.turn_username = config.get("turn_username")
         self.turn_password = config.get("turn_password")
         self.turn_user_lifetime = self.parse_duration(config["turn_user_lifetime"])
-        self.turn_allow_guests = config.get("turn_allow_guests", True)
 
     def default_config(self, **kwargs):
         return """\
@@ -42,11 +41,4 @@ class VoipConfig(Config):
 
         # How long generated TURN credentials last
         turn_user_lifetime: "1h"
-
-        # Whether guests should be allowed to use the TURN server.
-        # This defaults to True, otherwise VoIP will be unreliable for guests.
-        # However, it does introduce a slight security risk as it allows users to
-        # connect to arbitrary endpoints without having first signed up for a
-        # valid account (e.g. by passing a CAPTCHA).
-        turn_allow_guests: True
         """

synapse/config/workers.py

@@ -28,12 +28,7 @@ class WorkerConfig(Config):
         self.worker_pid_file = config.get("worker_pid_file")
         self.worker_log_file = config.get("worker_log_file")
         self.worker_log_config = config.get("worker_log_config")
-        self.worker_replication_host = config.get("worker_replication_host", None)
-        self.worker_replication_port = config.get("worker_replication_port", None)
-        self.worker_name = config.get("worker_name", self.worker_app)
-
-        self.worker_main_http_uri = config.get("worker_main_http_uri", None)
-        self.worker_cpu_affinity = config.get("worker_cpu_affinity")
+        self.worker_replication_url = config.get("worker_replication_url")
 
         if self.worker_listeners:
             for listener in self.worker_listeners:

synapse/crypto/context_factory.py

@@ -34,7 +34,7 @@ class ServerContextFactory(ssl.ContextFactory):
         try:
             _ecCurve = _OpenSSLECCurve(_defaultCurveName)
             _ecCurve.addECKeyToContext(context)
-        except Exception:
+        except:
             logger.exception("Failed to enable elliptic curve for TLS")
         context.set_options(SSL.OP_NO_SSLv2 | SSL.OP_NO_SSLv3)
         context.use_certificate_chain_file(config.tls_certificate_file)

synapse/crypto/event_signing.py

@@ -43,7 +43,7 @@ def check_event_content_hash(event, hash_algorithm=hashlib.sha256):
     message_hash_base64 = event.hashes[name]
     try:
         message_hash_bytes = decode_base64(message_hash_base64)
-    except Exception:
+    except:
         raise SynapseError(
             400,
             "Invalid base64: %s" % (message_hash_base64,),

synapse/crypto/keyclient.py

@@ -13,11 +13,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from synapse.util import logcontext
+
 from twisted.web.http import HTTPClient
 from twisted.internet.protocol import Factory
 from twisted.internet import defer, reactor
 from synapse.http.endpoint import matrix_federation_endpoint
+from synapse.util.logcontext import (
+    preserve_context_over_fn, preserve_context_over_deferred
+)
 import simplejson as json
 import logging
 
@@ -40,10 +43,14 @@ def fetch_server_key(server_name, ssl_context_factory, path=KEY_API_V1):
 
     for i in range(5):
         try:
-            with logcontext.PreserveLoggingContext():
-                protocol = yield endpoint.connect(factory)
-                server_response, server_certificate = yield protocol.remote_key
-                defer.returnValue((server_response, server_certificate))
+            protocol = yield preserve_context_over_fn(
+                endpoint.connect, factory
+            )
+            server_response, server_certificate = yield preserve_context_over_deferred(
+                protocol.remote_key
+            )
+            defer.returnValue((server_response, server_certificate))
+            return
         except SynapseKeyClientError as e:
             logger.exception("Error getting key for %r" % (server_name,))
             if e.status.startswith("4"):

synapse/crypto/keyring.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2017 New Vector Ltd.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -16,9 +15,11 @@
 
 from synapse.crypto.keyclient import fetch_server_key
 from synapse.api.errors import SynapseError, Codes
-from synapse.util import unwrapFirstError, logcontext
+from synapse.util.retryutils import get_retry_limiter
+from synapse.util import unwrapFirstError
+from synapse.util.async import ObservableDeferred
 from synapse.util.logcontext import (
-    PreserveLoggingContext,
+    preserve_context_over_deferred, preserve_context_over_fn, PreserveLoggingContext,
     preserve_fn
 )
 from synapse.util.metrics import Measure
@@ -57,8 +58,7 @@ Attributes:
     json_object(dict): The JSON object to verify.
     deferred(twisted.internet.defer.Deferred):
         A deferred (server_name, key_id, verify_key) tuple that resolves when
-        a verify key has been fetched. The deferreds' callbacks are run with no
-        logcontext.
+        a verify key has been fetched
 """
 
 
@@ -75,41 +75,31 @@ class Keyring(object):
         self.perspective_servers = self.config.perspectives
         self.hs = hs
 
-        # map from server name to Deferred. Has an entry for each server with
-        # an ongoing key download; the Deferred completes once the download
-        # completes.
-        #
-        # These are regular, logcontext-agnostic Deferreds.
         self.key_downloads = {}
 
     def verify_json_for_server(self, server_name, json_object):
-        return logcontext.make_deferred_yieldable(
-            self.verify_json_objects_for_server(
-                [(server_name, json_object)]
-            )[0]
-        )
+        return self.verify_json_objects_for_server(
+            [(server_name, json_object)]
+        )[0]
 
     def verify_json_objects_for_server(self, server_and_json):
-        """Bulk verifies signatures of json objects, bulk fetching keys as
+        """Bulk verfies signatures of json objects, bulk fetching keys as
         necessary.
 
         Args:
             server_and_json (list): List of pairs of (server_name, json_object)
 
         Returns:
-            List<Deferred>: for each input pair, a deferred indicating success
-                or failure to verify each json object's signature for the given
-                server_name. The deferreds run their callbacks in the sentinel
-                logcontext.
+            list of deferreds indicating success or failure to verify each
+            json object's signature for the given server_name.
         """
         verify_requests = []
 
         for server_name, json_object in server_and_json:
+            logger.debug("Verifying for %s", server_name)
 
             key_ids = signature_ids(json_object, server_name)
             if not key_ids:
-                logger.warn("Request from %s: no supported signature keys",
-                            server_name)
                 deferred = defer.fail(SynapseError(
                     400,
                     "Not signed with a supported algorithm",
@@ -118,81 +108,97 @@ class Keyring(object):
             else:
                 deferred = defer.Deferred()
 
-            logger.debug("Verifying for %s with key_ids %s",
-                         server_name, key_ids)
-
             verify_request = VerifyKeyRequest(
                 server_name, key_ids, json_object, deferred
             )
 
             verify_requests.append(verify_request)
 
-        preserve_fn(self._start_key_lookups)(verify_requests)
+        @defer.inlineCallbacks
+        def handle_key_deferred(verify_request):
+            server_name = verify_request.server_name
+            try:
+                _, key_id, verify_key = yield verify_request.deferred
+            except IOError as e:
+                logger.warn(
+                    "Got IOError when downloading keys for %s: %s %s",
+                    server_name, type(e).__name__, str(e.message),
+                )
+                raise SynapseError(
+                    502,
+                    "Error downloading keys for %s" % (server_name,),
+                    Codes.UNAUTHORIZED,
+                )
+            except Exception as e:
+                logger.exception(
+                    "Got Exception when downloading keys for %s: %s %s",
+                    server_name, type(e).__name__, str(e.message),
+                )
+                raise SynapseError(
+                    401,
+                    "No key for %s with id %s" % (server_name, key_ids),
+                    Codes.UNAUTHORIZED,
+                )
+
+            json_object = verify_request.json_object
+
+            try:
+                verify_signed_json(json_object, server_name, verify_key)
+            except:
+                raise SynapseError(
+                    401,
+                    "Invalid signature for server %s with key %s:%s" % (
+                        server_name, verify_key.alg, verify_key.version
+                    ),
+                    Codes.UNAUTHORIZED,
+                )
+
+        server_to_deferred = {
+            server_name: defer.Deferred()
+            for server_name, _ in server_and_json
+        }
+
+        with PreserveLoggingContext():
+
+            # We want to wait for any previous lookups to complete before
+            # proceeding.
+            wait_on_deferred = self.wait_for_previous_lookups(
+                [server_name for server_name, _ in server_and_json],
+                server_to_deferred,
+            )
+
+            # Actually start fetching keys.
+            wait_on_deferred.addBoth(
+                lambda _: self.get_server_verify_keys(verify_requests)
+            )
+
+            # When we've finished fetching all the keys for a given server_name,
+            # resolve the deferred passed to `wait_for_previous_lookups` so that
+            # any lookups waiting will proceed.
+            server_to_request_ids = {}
+
+            def remove_deferreds(res, server_name, verify_request):
+                request_id = id(verify_request)
+                server_to_request_ids[server_name].discard(request_id)
+                if not server_to_request_ids[server_name]:
+                    d = server_to_deferred.pop(server_name, None)
+                    if d:
+                        d.callback(None)
+                return res
+
+            for verify_request in verify_requests:
+                server_name = verify_request.server_name
+                request_id = id(verify_request)
+                server_to_request_ids.setdefault(server_name, set()).add(request_id)
+                deferred.addBoth(remove_deferreds, server_name, verify_request)
 
         # Pass those keys to handle_key_deferred so that the json object
         # signatures can be verified
-        handle = preserve_fn(_handle_key_deferred)
         return [
-            handle(rq) for rq in verify_requests
+            preserve_context_over_fn(handle_key_deferred, verify_request)
+            for verify_request in verify_requests
         ]
 
-    @defer.inlineCallbacks
-    def _start_key_lookups(self, verify_requests):
-        """Sets off the key fetches for each verify request
-
-        Once each fetch completes, verify_request.deferred will be resolved.
-
-        Args:
-            verify_requests (List[VerifyKeyRequest]):
-        """
-
-        # create a deferred for each server we're going to look up the keys
-        # for; we'll resolve them once we have completed our lookups.
-        # These will be passed into wait_for_previous_lookups to block
-        # any other lookups until we have finished.
-        # The deferreds are called with no logcontext.
-        server_to_deferred = {
-            rq.server_name: defer.Deferred()
-            for rq in verify_requests
-        }
-
-        # We want to wait for any previous lookups to complete before
-        # proceeding.
-        yield self.wait_for_previous_lookups(
-            [rq.server_name for rq in verify_requests],
-            server_to_deferred,
-        )
-
-        # Actually start fetching keys.
-        self._get_server_verify_keys(verify_requests)
-
-        # When we've finished fetching all the keys for a given server_name,
-        # resolve the deferred passed to `wait_for_previous_lookups` so that
-        # any lookups waiting will proceed.
-        #
-        # map from server name to a set of request ids
-        server_to_request_ids = {}
-
-        for verify_request in verify_requests:
-            server_name = verify_request.server_name
-            request_id = id(verify_request)
-            server_to_request_ids.setdefault(server_name, set()).add(request_id)
-
-        def remove_deferreds(res, verify_request):
-            server_name = verify_request.server_name
-            request_id = id(verify_request)
-            server_to_request_ids[server_name].discard(request_id)
-            if not server_to_request_ids[server_name]:
-                d = server_to_deferred.pop(server_name, None)
-                if d:
-                    d.callback(None)
-            return res
-
-        for verify_request in verify_requests:
-            verify_request.deferred.addBoth(
-                remove_deferreds, verify_request,
-            )
-
     @defer.inlineCallbacks
     def wait_for_previous_lookups(self, server_names, server_to_deferred):
         """Waits for any previous key lookups for the given servers to finish.
@@ -200,13 +206,7 @@ class Keyring(object):
         Args:
             server_names (list): list of server_names we want to lookup
             server_to_deferred (dict): server_name to deferred which gets
-                resolved once we've finished looking up keys for that server.
-                The Deferreds should be regular twisted ones which call their
-                callbacks with no logcontext.
-
-        Returns: a Deferred which resolves once all key lookups for the given
-            servers have completed. Follows the synapse rules of logcontext
-            preservation.
+                resolved once we've finished looking up keys for that server
         """
         while True:
             wait_on = [
@@ -220,23 +220,19 @@ class Keyring(object):
             else:
                 break
 
-        def rm(r, server_name_):
-            self.key_downloads.pop(server_name_, None)
-            return r
-
         for server_name, deferred in server_to_deferred.items():
-            self.key_downloads[server_name] = deferred
-            deferred.addBoth(rm, server_name)
+            d = ObservableDeferred(preserve_context_over_deferred(deferred))
+            self.key_downloads[server_name] = d
 
-    def _get_server_verify_keys(self, verify_requests):
-        """Tries to find at least one key for each verify request
+            def rm(r, server_name):
+                self.key_downloads.pop(server_name, None)
+                return r
 
-        For each verify_request, verify_request.deferred is called back with
-        params (server_name, key_id, VerifyKey) if a key is found, or errbacked
-        with a SynapseError if none of the keys are found.
+            d.addBoth(rm, server_name)
 
-        Args:
-            verify_requests (list[VerifyKeyRequest]): list of verify requests
+    def get_server_verify_keys(self, verify_requests):
+        """Takes a dict of KeyGroups and tries to find at least one key for
+        each group.
         """
 
         # These are functions that produce keys given a list of key ids
@@ -249,11 +245,8 @@ class Keyring(object):
         @defer.inlineCallbacks
         def do_iterations():
             with Measure(self.clock, "get_server_verify_keys"):
-                # dict[str, dict[str, VerifyKey]]: results so far.
-                # map server_name -> key_id -> VerifyKey
                 merged_results = {}
 
-                # dict[str, set(str)]: keys to fetch for each server
                 missing_keys = {}
                 for verify_request in verify_requests:
                     missing_keys.setdefault(verify_request.server_name, set()).update(
@@ -297,37 +290,25 @@ class Keyring(object):
                     if not missing_keys:
                         break
 
-                with PreserveLoggingContext():
-                    for verify_request in requests_missing_keys:
-                        verify_request.deferred.errback(SynapseError(
-                            401,
-                            "No key for %s with id %s" % (
-                                verify_request.server_name, verify_request.key_ids,
-                            ),
-                            Codes.UNAUTHORIZED,
-                        ))
+                for verify_request in requests_missing_keys.values():
+                    verify_request.deferred.errback(SynapseError(
+                        401,
+                        "No key for %s with id %s" % (
+                            verify_request.server_name, verify_request.key_ids,
+                        ),
+                        Codes.UNAUTHORIZED,
+                    ))
 
         def on_err(err):
-            with PreserveLoggingContext():
-                for verify_request in verify_requests:
-                    if not verify_request.deferred.called:
-                        verify_request.deferred.errback(err)
+            for verify_request in verify_requests:
+                if not verify_request.deferred.called:
+                    verify_request.deferred.errback(err)
 
-        preserve_fn(do_iterations)().addErrback(on_err)
+        do_iterations().addErrback(on_err)
 
     @defer.inlineCallbacks
     def get_keys_from_store(self, server_name_and_key_ids):
-        """
-
-        Args:
-            server_name_and_key_ids (list[(str, iterable[str])]):
-                list of (server_name, iterable[key_id]) tuples to fetch keys for
-
-        Returns:
-            Deferred: resolves to dict[str, dict[str, VerifyKey]]: map from
-                server_name -> key_id -> VerifyKey
-        """
-        res = yield logcontext.make_deferred_yieldable(defer.gatherResults(
+        res = yield preserve_context_over_deferred(defer.gatherResults(
             [
                 preserve_fn(self.store.get_server_verify_keys)(
                     server_name, key_ids
@@ -335,7 +316,7 @@ class Keyring(object):
                 for server_name, key_ids in server_name_and_key_ids
             ],
             consumeErrors=True,
-        ).addErrback(unwrapFirstError))
+        )).addErrback(unwrapFirstError)
 
         defer.returnValue(dict(res))
 
@@ -356,13 +337,13 @@ class Keyring(object):
                 )
                 defer.returnValue({})
 
-        results = yield logcontext.make_deferred_yieldable(defer.gatherResults(
+        results = yield preserve_context_over_deferred(defer.gatherResults(
             [
                 preserve_fn(get_key)(p_name, p_keys)
                 for p_name, p_keys in self.perspective_servers.items()
             ],
             consumeErrors=True,
-        ).addErrback(unwrapFirstError))
+        )).addErrback(unwrapFirstError)
 
         union_of_keys = {}
         for result in results:
@@ -375,34 +356,40 @@ class Keyring(object):
     def get_keys_from_server(self, server_name_and_key_ids):
         @defer.inlineCallbacks
         def get_key(server_name, key_ids):
-            keys = None
-            try:
-                keys = yield self.get_server_verify_key_v2_direct(
-                    server_name, key_ids
-                )
-            except Exception as e:
-                logger.info(
-                    "Unable to get key %r for %r directly: %s %s",
-                    key_ids, server_name,
-                    type(e).__name__, str(e.message),
-                )
+            limiter = yield get_retry_limiter(
+                server_name,
+                self.clock,
+                self.store,
+            )
+            with limiter:
+                keys = None
+                try:
+                    keys = yield self.get_server_verify_key_v2_direct(
+                        server_name, key_ids
+                    )
+                except Exception as e:
+                    logger.info(
+                        "Unable to get key %r for %r directly: %s %s",
+                        key_ids, server_name,
+                        type(e).__name__, str(e.message),
+                    )
 
-            if not keys:
-                keys = yield self.get_server_verify_key_v1_direct(
-                    server_name, key_ids
-                )
+                if not keys:
+                    keys = yield self.get_server_verify_key_v1_direct(
+                        server_name, key_ids
+                    )
 
-                keys = {server_name: keys}
+                    keys = {server_name: keys}
 
             defer.returnValue(keys)
 
-        results = yield logcontext.make_deferred_yieldable(defer.gatherResults(
+        results = yield preserve_context_over_deferred(defer.gatherResults(
             [
                 preserve_fn(get_key)(server_name, key_ids)
                 for server_name, key_ids in server_name_and_key_ids
             ],
             consumeErrors=True,
-        ).addErrback(unwrapFirstError))
+        )).addErrback(unwrapFirstError)
 
         merged = {}
         for result in results:
@@ -479,7 +466,7 @@ class Keyring(object):
             for server_name, response_keys in processed_response.items():
                 keys.setdefault(server_name, {}).update(response_keys)
 
-        yield logcontext.make_deferred_yieldable(defer.gatherResults(
+        yield preserve_context_over_deferred(defer.gatherResults(
             [
                 preserve_fn(self.store_keys)(
                     server_name=server_name,
@@ -489,7 +476,7 @@ class Keyring(object):
                 for server_name, response_keys in keys.items()
             ],
             consumeErrors=True
-        ).addErrback(unwrapFirstError))
+        )).addErrback(unwrapFirstError)
 
         defer.returnValue(keys)
 
@@ -537,7 +524,7 @@ class Keyring(object):
 
             keys.update(response_keys)
 
-        yield logcontext.make_deferred_yieldable(defer.gatherResults(
+        yield preserve_context_over_deferred(defer.gatherResults(
             [
                 preserve_fn(self.store_keys)(
                     server_name=key_server_name,
@@ -547,7 +534,7 @@ class Keyring(object):
                 for key_server_name, verify_keys in keys.items()
             ],
             consumeErrors=True
-        ).addErrback(unwrapFirstError))
+        )).addErrback(unwrapFirstError)
 
         defer.returnValue(keys)
 
@@ -613,7 +600,7 @@ class Keyring(object):
         response_keys.update(verify_keys)
         response_keys.update(old_verify_keys)
 
-        yield logcontext.make_deferred_yieldable(defer.gatherResults(
+        yield preserve_context_over_deferred(defer.gatherResults(
             [
                 preserve_fn(self.store.store_server_keys_json)(
                     server_name=server_name,
@@ -626,7 +613,7 @@ class Keyring(object):
                 for key_id in updated_key_ids
             ],
             consumeErrors=True,
-        ).addErrback(unwrapFirstError))
+        )).addErrback(unwrapFirstError)
 
         results[server_name] = response_keys
 
@@ -704,6 +691,7 @@ class Keyring(object):
 
         defer.returnValue(verify_keys)
 
+    @defer.inlineCallbacks
     def store_keys(self, server_name, from_server, verify_keys):
         """Store a collection of verify keys for a given server
         Args:
@@ -714,7 +702,7 @@ class Keyring(object):
             A deferred that completes when the keys are stored.
         """
         # TODO(markjh): Store whether the keys have expired.
-        return logcontext.make_deferred_yieldable(defer.gatherResults(
+        yield preserve_context_over_deferred(defer.gatherResults(
             [
                 preserve_fn(self.store.store_server_verify_key)(
                     server_name, server_name, key.time_added, key
@@ -722,48 +710,4 @@ class Keyring(object):
                 for key_id, key in verify_keys.items()
             ],
             consumeErrors=True,
-        ).addErrback(unwrapFirstError))
-
-
-@defer.inlineCallbacks
-def _handle_key_deferred(verify_request):
-    server_name = verify_request.server_name
-    try:
-        with PreserveLoggingContext():
-            _, key_id, verify_key = yield verify_request.deferred
-    except IOError as e:
-        logger.warn(
-            "Got IOError when downloading keys for %s: %s %s",
-            server_name, type(e).__name__, str(e.message),
-        )
-        raise SynapseError(
-            502,
-            "Error downloading keys for %s" % (server_name,),
-            Codes.UNAUTHORIZED,
-        )
-    except Exception as e:
-        logger.exception(
-            "Got Exception when downloading keys for %s: %s %s",
-            server_name, type(e).__name__, str(e.message),
-        )
-        raise SynapseError(
-            401,
-            "No key for %s with id %s" % (server_name, verify_request.key_ids),
-            Codes.UNAUTHORIZED,
-        )
-
-    json_object = verify_request.json_object
-
-    logger.debug("Got key %s %s:%s for server %s, verifying" % (
-        key_id, verify_key.alg, verify_key.version, server_name,
-    ))
-    try:
-        verify_signed_json(json_object, server_name, verify_key)
-    except Exception:
-        raise SynapseError(
-            401,
-            "Invalid signature for server %s with key %s:%s" % (
-                server_name, verify_key.alg, verify_key.version
-            ),
-            Codes.UNAUTHORIZED,
-        )
+        )).addErrback(unwrapFirstError)

synapse/event_auth.py

@@ -443,12 +443,12 @@ def _check_power_levels(event, auth_events):
     for k, v in user_list.items():
         try:
             UserID.from_string(k)
-        except Exception:
+        except:
             raise SynapseError(400, "Not a valid user_id: %s" % (k,))
 
         try:
             int(v)
-        except Exception:
+        except:
             raise SynapseError(400, "Not a valid power level: %s" % (v,))
 
     key = (event.type, event.state_key, )
@@ -470,14 +470,14 @@ def _check_power_levels(event, auth_events):
         ("invite", None),
     ]
 
-    old_list = current_state.content.get("users", {})
+    old_list = current_state.content.get("users")
     for user in set(old_list.keys() + user_list.keys()):
         levels_to_check.append(
             (user, "users")
         )
 
-    old_list = current_state.content.get("events", {})
-    new_list = event.content.get("events", {})
+    old_list = current_state.content.get("events")
+    new_list = event.content.get("events")
     for ev_id in set(old_list.keys() + new_list.keys()):
         levels_to_check.append(
             (ev_id, "events")

synapse/events/builder.py

@@ -55,7 +55,7 @@ class EventBuilderFactory(object):
 
         local_part = str(int(self.clock.time())) + i + random_string(5)
 
-        e_id = EventID(local_part, self.hostname)
+        e_id = EventID.create(local_part, self.hostname)
 
         return e_id.to_string()
 

synapse/events/snapshot.py

@@ -15,32 +15,6 @@
 
 
 class EventContext(object):
-    """
-    Attributes:
-        current_state_ids (dict[(str, str), str]):
-            The current state map including the current event.
-            (type, state_key) -> event_id
-
-        prev_state_ids (dict[(str, str), str]):
-            The current state map excluding the current event.
-            (type, state_key) -> event_id
-
-        state_group (int): state group id
-        rejected (bool|str): A rejection reason if the event was rejected, else
-            False
-
-        push_actions (list[(str, list[object])]): list of (user_id, actions)
-            tuples
-
-        prev_group (int): Previously persisted state group. ``None`` for an
-            outlier.
-        delta_ids (dict[(str, str), str]): Delta from ``prev_group``.
-            (type, state_key) -> event_id. ``None`` for an outlier.
-
-        prev_state_events (?): XXX: is this ever set to anything other than
-            the empty list?
-    """
-
     __slots__ = [
         "current_state_ids",
         "prev_state_ids",
@@ -50,7 +24,6 @@ class EventContext(object):
         "prev_group",
         "delta_ids",
         "prev_state_events",
-        "app_service",
     ]
 
     def __init__(self):
@@ -69,5 +42,3 @@ class EventContext(object):
         self.delta_ids = None
 
         self.prev_state_events = None
-
-        self.app_service = None

synapse/events/spamcheck.py

@@ -1,113 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2017 New Vector Ltd.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-
-class SpamChecker(object):
-    def __init__(self, hs):
-        self.spam_checker = None
-
-        module = None
-        config = None
-        try:
-            module, config = hs.config.spam_checker
-        except Exception:
-            pass
-
-        if module is not None:
-            self.spam_checker = module(config=config)
-
-    def check_event_for_spam(self, event):
-        """Checks if a given event is considered "spammy" by this server.
-
-        If the server considers an event spammy, then it will be rejected if
-        sent by a local user. If it is sent by a user on another server, then
-        users receive a blank event.
-
-        Args:
-            event (synapse.events.EventBase): the event to be checked
-
-        Returns:
-            bool: True if the event is spammy.
-        """
-        if self.spam_checker is None:
-            return False
-
-        return self.spam_checker.check_event_for_spam(event)
-
-    def user_may_invite(self, inviter_userid, invitee_userid, room_id):
-        """Checks if a given user may send an invite
-
-        If this method returns false, the invite will be rejected.
-
-        Args:
-            userid (string): The sender's user ID
-
-        Returns:
-            bool: True if the user may send an invite, otherwise False
-        """
-        if self.spam_checker is None:
-            return True
-
-        return self.spam_checker.user_may_invite(inviter_userid, invitee_userid, room_id)
-
-    def user_may_create_room(self, userid):
-        """Checks if a given user may create a room
-
-        If this method returns false, the creation request will be rejected.
-
-        Args:
-            userid (string): The sender's user ID
-
-        Returns:
-            bool: True if the user may create a room, otherwise False
-        """
-        if self.spam_checker is None:
-            return True
-
-        return self.spam_checker.user_may_create_room(userid)
-
-    def user_may_create_room_alias(self, userid, room_alias):
-        """Checks if a given user may create a room alias
-
-        If this method returns false, the association request will be rejected.
-
-        Args:
-            userid (string): The sender's user ID
-            room_alias (string): The alias to be created
-
-        Returns:
-            bool: True if the user may create a room alias, otherwise False
-        """
-        if self.spam_checker is None:
-            return True
-
-        return self.spam_checker.user_may_create_room_alias(userid, room_alias)
-
-    def user_may_publish_room(self, userid, room_id):
-        """Checks if a given user may publish a room to the directory
-
-        If this method returns false, the publish request will be rejected.
-
-        Args:
-            userid (string): The sender's user ID
-            room_id (string): The ID of the room that would be published
-
-        Returns:
-            bool: True if the user may publish the room, otherwise False
-        """
-        if self.spam_checker is None:
-            return True
-
-        return self.spam_checker.user_may_publish_room(userid, room_id)

synapse/events/utils.py

@@ -225,22 +225,7 @@ def format_event_for_client_v2_without_room_id(d):
 
 def serialize_event(e, time_now_ms, as_client_event=True,
                     event_format=format_event_for_client_v1,
-                    token_id=None, only_event_fields=None, is_invite=False):
-    """Serialize event for clients
-
-    Args:
-        e (EventBase)
-        time_now_ms (int)
-        as_client_event (bool)
-        event_format
-        token_id
-        only_event_fields
-        is_invite (bool): Whether this is an invite that is being sent to the
-            invitee
-
-    Returns:
-        dict
-    """
+                    token_id=None, only_event_fields=None):
     # FIXME(erikj): To handle the case of presence events and the like
     if not isinstance(e, EventBase):
         return e
@@ -266,12 +251,6 @@ def serialize_event(e, time_now_ms, as_client_event=True,
             if txn_id is not None:
                 d["unsigned"]["transaction_id"] = txn_id
 
-    # If this is an invite for somebody else, then we don't care about the
-    # invite_room_state as that's meant solely for the invitee. Other clients
-    # will already have the state since they're in the room.
-    if not is_invite:
-        d["unsigned"].pop("invite_room_state", None)
-
     if as_client_event:
         d = event_format(d)
 

synapse/federation/federation_base.py

@@ -12,20 +12,28 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
+
+
+from twisted.internet import defer
+
+from synapse.events.utils import prune_event
+
+from synapse.crypto.event_signing import check_event_content_hash
 
 from synapse.api.errors import SynapseError
-from synapse.crypto.event_signing import check_event_content_hash
-from synapse.events.utils import prune_event
-from synapse.util import unwrapFirstError, logcontext
-from twisted.internet import defer
+
+from synapse.util import unwrapFirstError
+from synapse.util.logcontext import preserve_fn, preserve_context_over_deferred
+
+import logging
+
 
 logger = logging.getLogger(__name__)
 
 
 class FederationBase(object):
     def __init__(self, hs):
-        self.spam_checker = hs.get_spam_checker()
+        pass
 
     @defer.inlineCallbacks
     def _check_sigs_and_hash_and_fetch(self, origin, pdus, outlier=False,
@@ -49,52 +57,56 @@ class FederationBase(object):
         """
         deferreds = self._check_sigs_and_hashes(pdus)
 
-        @defer.inlineCallbacks
-        def handle_check_result(pdu, deferred):
-            try:
-                res = yield logcontext.make_deferred_yieldable(deferred)
-            except SynapseError:
-                res = None
+        def callback(pdu):
+            return pdu
 
+        def errback(failure, pdu):
+            failure.trap(SynapseError)
+            return None
+
+        def try_local_db(res, pdu):
             if not res:
                 # Check local db.
-                res = yield self.store.get_event(
+                return self.store.get_event(
                     pdu.event_id,
                     allow_rejected=True,
                     allow_none=True,
                 )
+            return res
 
+        def try_remote(res, pdu):
             if not res and pdu.origin != origin:
-                try:
-                    res = yield self.get_pdu(
-                        destinations=[pdu.origin],
-                        event_id=pdu.event_id,
-                        outlier=outlier,
-                        timeout=10000,
-                    )
-                except SynapseError:
-                    pass
+                return self.get_pdu(
+                    destinations=[pdu.origin],
+                    event_id=pdu.event_id,
+                    outlier=outlier,
+                    timeout=10000,
+                ).addErrback(lambda e: None)
+            return res
 
+        def warn(res, pdu):
             if not res:
                 logger.warn(
                     "Failed to find copy of %s with valid signature",
                     pdu.event_id,
                 )
+            return res
 
-            defer.returnValue(res)
-
-        handle = logcontext.preserve_fn(handle_check_result)
-        deferreds2 = [
-            handle(pdu, deferred)
-            for pdu, deferred in zip(pdus, deferreds)
-        ]
-
-        valid_pdus = yield logcontext.make_deferred_yieldable(
-            defer.gatherResults(
-                deferreds2,
-                consumeErrors=True,
+        for pdu, deferred in zip(pdus, deferreds):
+            deferred.addCallbacks(
+                callback, errback, errbackArgs=[pdu]
+            ).addCallback(
+                try_local_db, pdu
+            ).addCallback(
+                try_remote, pdu
+            ).addCallback(
+                warn, pdu
             )
-        ).addErrback(unwrapFirstError)
+
+        valid_pdus = yield preserve_context_over_deferred(defer.gatherResults(
+            deferreds,
+            consumeErrors=True
+        )).addErrback(unwrapFirstError)
 
         if include_none:
             defer.returnValue(valid_pdus)
@@ -102,24 +114,15 @@ class FederationBase(object):
             defer.returnValue([p for p in valid_pdus if p])
 
     def _check_sigs_and_hash(self, pdu):
-        return logcontext.make_deferred_yieldable(
-            self._check_sigs_and_hashes([pdu])[0],
-        )
+        return self._check_sigs_and_hashes([pdu])[0]
 
     def _check_sigs_and_hashes(self, pdus):
-        """Checks that each of the received events is correctly signed by the
-        sending server.
-
-        Args:
-            pdus (list[FrozenEvent]): the events to be checked
+        """Throws a SynapseError if a PDU does not have the correct
+        signatures.
 
         Returns:
-            list[Deferred]: for each input event, a deferred which:
-              * returns the original event if the checks pass
-              * returns a redacted version of the event (if the signature
-                matched but the hash did not)
-              * throws a SynapseError if the signature check failed.
-            The deferreds run their callbacks in the sentinel logcontext.
+            FrozenEvent: Either the given event or it redacted if it failed the
+            content hash check.
         """
 
         redacted_pdus = [
@@ -127,38 +130,26 @@ class FederationBase(object):
             for pdu in pdus
         ]
 
-        deferreds = self.keyring.verify_json_objects_for_server([
+        deferreds = preserve_fn(self.keyring.verify_json_objects_for_server)([
             (p.origin, p.get_pdu_json())
             for p in redacted_pdus
         ])
 
-        ctx = logcontext.LoggingContext.current_context()
-
         def callback(_, pdu, redacted):
-            with logcontext.PreserveLoggingContext(ctx):
-                if not check_event_content_hash(pdu):
-                    logger.warn(
-                        "Event content has been tampered, redacting %s: %s",
-                        pdu.event_id, pdu.get_pdu_json()
-                    )
-                    return redacted
-
-                if self.spam_checker.check_event_for_spam(pdu):
-                    logger.warn(
-                        "Event contains spam, redacting %s: %s",
-                        pdu.event_id, pdu.get_pdu_json()
-                    )
-                    return redacted
-
-                return pdu
+            if not check_event_content_hash(pdu):
+                logger.warn(
+                    "Event content has been tampered, redacting %s: %s",
+                    pdu.event_id, pdu.get_pdu_json()
+                )
+                return redacted
+            return pdu
 
         def errback(failure, pdu):
             failure.trap(SynapseError)
-            with logcontext.PreserveLoggingContext(ctx):
-                logger.warn(
-                    "Signature check failed for %s",
-                    pdu.event_id,
-                )
+            logger.warn(
+                "Signature check failed for %s",
+                pdu.event_id,
+            )
             return failure
 
         for deferred, pdu, redacted in zip(deferreds, pdus, redacted_pdus):

synapse/federation/federation_client.py

@@ -22,14 +22,14 @@ from synapse.api.constants import Membership
 from synapse.api.errors import (
     CodeMessageException, HttpResponseException, SynapseError,
 )
-from synapse.util import unwrapFirstError, logcontext
+from synapse.util import unwrapFirstError
 from synapse.util.caches.expiringcache import ExpiringCache
 from synapse.util.logutils import log_function
 from synapse.util.logcontext import preserve_fn, preserve_context_over_deferred
 from synapse.events import FrozenEvent, builder
 import synapse.metrics
 
-from synapse.util.retryutils import NotRetryingDestination
+from synapse.util.retryutils import get_retry_limiter, NotRetryingDestination
 
 import copy
 import itertools
@@ -88,7 +88,7 @@ class FederationClient(FederationBase):
 
     @log_function
     def make_query(self, destination, query_type, args,
-                   retry_on_dns_fail=False, ignore_backoff=False):
+                   retry_on_dns_fail=False):
         """Sends a federation Query to a remote homeserver of the given type
         and arguments.
 
@@ -98,8 +98,6 @@ class FederationClient(FederationBase):
                 handler name used in register_query_handler().
             args (dict): Mapping of strings to strings containing the details
                 of the query request.
-            ignore_backoff (bool): true to ignore the historical backoff data
-                and try the request anyway.
 
         Returns:
             a Deferred which will eventually yield a JSON object from the
@@ -108,8 +106,7 @@ class FederationClient(FederationBase):
         sent_queries_counter.inc(query_type)
 
         return self.transport_layer.make_query(
-            destination, query_type, args, retry_on_dns_fail=retry_on_dns_fail,
-            ignore_backoff=ignore_backoff,
+            destination, query_type, args, retry_on_dns_fail=retry_on_dns_fail
         )
 
     @log_function
@@ -189,10 +186,10 @@ class FederationClient(FederationBase):
         ]
 
         # FIXME: We should handle signature failures more gracefully.
-        pdus[:] = yield logcontext.make_deferred_yieldable(defer.gatherResults(
+        pdus[:] = yield preserve_context_over_deferred(defer.gatherResults(
             self._check_sigs_and_hashes(pdus),
             consumeErrors=True,
-        ).addErrback(unwrapFirstError))
+        )).addErrback(unwrapFirstError)
 
         defer.returnValue(pdus)
 
@@ -209,7 +206,8 @@ class FederationClient(FederationBase):
 
         Args:
             destinations (list): Which home servers to query
-            event_id (str): event to fetch
+            pdu_origin (str): The home server that originally sent the pdu.
+            event_id (str)
             outlier (bool): Indicates whether the PDU is an `outlier`, i.e. if
                 it's from an arbitary point in the context as opposed to part
                 of the current block of PDUs. Defaults to `False`
@@ -237,24 +235,31 @@ class FederationClient(FederationBase):
                 continue
 
             try:
-                transaction_data = yield self.transport_layer.get_event(
-                    destination, event_id, timeout=timeout,
+                limiter = yield get_retry_limiter(
+                    destination,
+                    self._clock,
+                    self.store,
                 )
 
-                logger.debug("transaction_data %r", transaction_data)
+                with limiter:
+                    transaction_data = yield self.transport_layer.get_event(
+                        destination, event_id, timeout=timeout,
+                    )
 
-                pdu_list = [
-                    self.event_from_pdu_json(p, outlier=outlier)
-                    for p in transaction_data["pdus"]
-                ]
+                    logger.debug("transaction_data %r", transaction_data)
 
-                if pdu_list and pdu_list[0]:
-                    pdu = pdu_list[0]
+                    pdu_list = [
+                        self.event_from_pdu_json(p, outlier=outlier)
+                        for p in transaction_data["pdus"]
+                    ]
 
-                    # Check signatures are correct.
-                    signed_pdu = yield self._check_sigs_and_hash(pdu)
+                    if pdu_list and pdu_list[0]:
+                        pdu = pdu_list[0]
 
-                    break
+                        # Check signatures are correct.
+                        signed_pdu = yield self._check_sigs_and_hashes([pdu])[0]
+
+                        break
 
                 pdu_attempts[destination] = now
 
@@ -474,13 +479,8 @@ class FederationClient(FederationBase):
             content (object): Any additional data to put into the content field
                 of the event.
         Return:
-            Deferred: resolves to a tuple of (origin (str), event (object))
-            where origin is the remote homeserver which generated the event.
-
-            Fails with a ``CodeMessageException`` if the chosen remote server
-            returns a 300/400 code.
-
-            Fails with a ``RuntimeError`` if no servers were reachable.
+            A tuple of (origin (str), event (object)) where origin is the remote
+            homeserver which generated the event.
         """
         valid_memberships = {Membership.JOIN, Membership.LEAVE}
         if membership not in valid_memberships:
@@ -533,27 +533,6 @@ class FederationClient(FederationBase):
 
     @defer.inlineCallbacks
     def send_join(self, destinations, pdu):
-        """Sends a join event to one of a list of homeservers.
-
-        Doing so will cause the remote server to add the event to the graph,
-        and send the event out to the rest of the federation.
-
-        Args:
-            destinations (str): Candidate homeservers which are probably
-                participating in the room.
-            pdu (BaseEvent): event to be sent
-
-        Return:
-            Deferred: resolves to a dict with members ``origin`` (a string
-            giving the serer the event was sent to, ``state`` (?) and
-            ``auth_chain``.
-
-            Fails with a ``CodeMessageException`` if the chosen remote server
-            returns a 300/400 code.
-
-            Fails with a ``RuntimeError`` if no servers were reachable.
-        """
-
         for destination in destinations:
             if destination == self.server_name:
                 continue
@@ -661,26 +640,6 @@ class FederationClient(FederationBase):
 
     @defer.inlineCallbacks
     def send_leave(self, destinations, pdu):
-        """Sends a leave event to one of a list of homeservers.
-
-        Doing so will cause the remote server to add the event to the graph,
-        and send the event out to the rest of the federation.
-
-        This is mostly useful to reject received invites.
-
-        Args:
-            destinations (str): Candidate homeservers which are probably
-                participating in the room.
-            pdu (BaseEvent): event to be sent
-
-        Return:
-            Deferred: resolves to None.
-
-            Fails with a ``CodeMessageException`` if the chosen remote server
-            returns a non-200 code.
-
-            Fails with a ``RuntimeError`` if no servers were reachable.
-        """
         for destination in destinations:
             if destination == self.server_name:
                 continue

synapse/federation/federation_server.py

@@ -12,13 +12,14 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
+
 from twisted.internet import defer
 
 from .federation_base import FederationBase
 from .units import Transaction, Edu
 
-from synapse.util import async
-from synapse.util.logcontext import make_deferred_yieldable, preserve_fn
+from synapse.util.async import Linearizer
 from synapse.util.logutils import log_function
 from synapse.util.caches.response_cache import ResponseCache
 from synapse.events import FrozenEvent
@@ -32,9 +33,6 @@ from synapse.crypto.event_signing import compute_event_signature
 import simplejson as json
 import logging
 
-# when processing incoming transactions, we try to handle multiple rooms in
-# parallel, up to this limit.
-TRANSACTION_CONCURRENCY_LIMIT = 10
 
 logger = logging.getLogger(__name__)
 
@@ -54,8 +52,8 @@ class FederationServer(FederationBase):
 
         self.auth = hs.get_auth()
 
-        self._server_linearizer = async.Linearizer("fed_server")
-        self._transaction_linearizer = async.Linearizer("fed_txn_handler")
+        self._room_pdu_linearizer = Linearizer("fed_room_pdu")
+        self._server_linearizer = Linearizer("fed_server")
 
         # We cache responses to state queries, as they take a while and often
         # come in waves.
@@ -112,41 +110,25 @@ class FederationServer(FederationBase):
     @defer.inlineCallbacks
     @log_function
     def on_incoming_transaction(self, transaction_data):
-        # keep this as early as possible to make the calculated origin ts as
-        # accurate as possible.
-        request_time = self._clock.time_msec()
-
         transaction = Transaction(**transaction_data)
 
-        if not transaction.transaction_id:
-            raise Exception("Transaction missing transaction_id")
-        if not transaction.origin:
-            raise Exception("Transaction missing origin")
+        received_pdus_counter.inc_by(len(transaction.pdus))
+
+        for p in transaction.pdus:
+            if "unsigned" in p:
+                unsigned = p["unsigned"]
+                if "age" in unsigned:
+                    p["age"] = unsigned["age"]
+            if "age" in p:
+                p["age_ts"] = int(self._clock.time_msec()) - int(p["age"])
+                del p["age"]
+
+        pdu_list = [
+            self.event_from_pdu_json(p) for p in transaction.pdus
+        ]
 
         logger.debug("[%s] Got transaction", transaction.transaction_id)
 
-        # use a linearizer to ensure that we don't process the same transaction
-        # multiple times in parallel.
-        with (yield self._transaction_linearizer.queue(
-                (transaction.origin, transaction.transaction_id),
-        )):
-            result = yield self._handle_incoming_transaction(
-                transaction, request_time,
-            )
-
-        defer.returnValue(result)
-
-    @defer.inlineCallbacks
-    def _handle_incoming_transaction(self, transaction, request_time):
-        """ Process an incoming transaction and return the HTTP response
-
-        Args:
-            transaction (Transaction): incoming transaction
-            request_time (int): timestamp that the HTTP request arrived at
-
-        Returns:
-            Deferred[(int, object)]: http response code and body
-        """
         response = yield self.transaction_actions.have_responded(transaction)
 
         if response:
@@ -159,49 +141,38 @@ class FederationServer(FederationBase):
 
         logger.debug("[%s] Transaction is new", transaction.transaction_id)
 
-        received_pdus_counter.inc_by(len(transaction.pdus))
+        results = []
 
-        pdus_by_room = {}
-
-        for p in transaction.pdus:
-            if "unsigned" in p:
-                unsigned = p["unsigned"]
-                if "age" in unsigned:
-                    p["age"] = unsigned["age"]
-            if "age" in p:
-                p["age_ts"] = request_time - int(p["age"])
-                del p["age"]
-
-            event = self.event_from_pdu_json(p)
-            room_id = event.room_id
-            pdus_by_room.setdefault(room_id, []).append(event)
-
-        pdu_results = {}
-
-        # we can process different rooms in parallel (which is useful if they
-        # require callouts to other servers to fetch missing events), but
-        # impose a limit to avoid going too crazy with ram/cpu.
-        @defer.inlineCallbacks
-        def process_pdus_for_room(room_id):
-            logger.debug("Processing PDUs for %s", room_id)
-            for pdu in pdus_by_room[room_id]:
-                event_id = pdu.event_id
-                try:
-                    yield self._handle_received_pdu(
-                        transaction.origin, pdu
+        for pdu in pdu_list:
+            # check that it's actually being sent from a valid destination to
+            # workaround bug #1753 in 0.18.5 and 0.18.6
+            if transaction.origin != get_domain_from_id(pdu.event_id):
+                if not (
+                    pdu.type == 'm.room.member' and
+                    pdu.content and
+                    pdu.content.get("membership", None) == 'join' and
+                    self.hs.is_mine_id(pdu.state_key)
+                ):
+                    logger.info(
+                        "Discarding PDU %s from invalid origin %s",
+                        pdu.event_id, transaction.origin
+                    )
+                    continue
+                else:
+                    logger.info(
+                        "Accepting join PDU %s from %s",
+                        pdu.event_id, transaction.origin
                     )
-                    pdu_results[event_id] = {}
-                except FederationError as e:
-                    logger.warn("Error handling PDU %s: %s", event_id, e)
-                    pdu_results[event_id] = {"error": str(e)}
-                except Exception as e:
-                    pdu_results[event_id] = {"error": str(e)}
-                    logger.exception("Failed to handle PDU %s", event_id)
 
-        yield async.concurrently_execute(
-            process_pdus_for_room, pdus_by_room.keys(),
-            TRANSACTION_CONCURRENCY_LIMIT,
-        )
+            try:
+                yield self._handle_new_pdu(transaction.origin, pdu)
+                results.append({})
+            except FederationError as e:
+                self.send_failure(e, transaction.origin)
+                results.append({"error": str(e)})
+            except Exception as e:
+                results.append({"error": str(e)})
+                logger.exception("Failed to handle PDU")
 
         if hasattr(transaction, "edus"):
             for edu in (Edu(**x) for x in transaction.edus):
@@ -211,16 +182,17 @@ class FederationServer(FederationBase):
                     edu.content
                 )
 
-        pdu_failures = getattr(transaction, "pdu_failures", [])
-        for failure in pdu_failures:
-            logger.info("Got failure %r", failure)
+            for failure in getattr(transaction, "pdu_failures", []):
+                logger.info("Got failure %r", failure)
+
+        logger.debug("Returning: %s", str(results))
 
         response = {
-            "pdus": pdu_results,
+            "pdus": dict(zip(
+                (p.event_id for p in pdu_list), results
+            )),
         }
 
-        logger.debug("Returning: %s", str(response))
-
         yield self.transaction_actions.set_response(
             transaction,
             200, response
@@ -254,13 +226,12 @@ class FederationServer(FederationBase):
         result = self._state_resp_cache.get((room_id, event_id))
         if not result:
             with (yield self._server_linearizer.queue((origin, room_id))):
-                d = self._state_resp_cache.set(
+                resp = yield self._state_resp_cache.set(
                     (room_id, event_id),
-                    preserve_fn(self._on_context_state_request_compute)(room_id, event_id)
+                    self._on_context_state_request_compute(room_id, event_id)
                 )
-                resp = yield make_deferred_yieldable(d)
         else:
-            resp = yield make_deferred_yieldable(result)
+            resp = yield result
 
         defer.returnValue((200, resp))
 
@@ -466,16 +437,6 @@ class FederationServer(FederationBase):
                         key_id: json.loads(json_bytes)
                     }
 
-        logger.info(
-            "Claimed one-time-keys: %s",
-            ",".join((
-                "%s for %s:%s" % (key_id, user_id, device_id)
-                for user_id, user_keys in json_result.iteritems()
-                for device_id, device_keys in user_keys.iteritems()
-                for key_id, _ in device_keys.iteritems()
-            )),
-        )
-
         defer.returnValue({"one_time_keys": json_result})
 
     @defer.inlineCallbacks
@@ -536,39 +497,26 @@ class FederationServer(FederationBase):
         )
 
     @defer.inlineCallbacks
-    def _handle_received_pdu(self, origin, pdu):
-        """ Process a PDU received in a federation /send/ transaction.
+    @log_function
+    def _handle_new_pdu(self, origin, pdu, get_missing=True):
 
-        Args:
-            origin (str): server which sent the pdu
-            pdu (FrozenEvent): received pdu
+        # We reprocess pdus when we have seen them only as outliers
+        existing = yield self._get_persisted_pdu(
+            origin, pdu.event_id, do_auth=False
+        )
 
-        Returns (Deferred): completes with None
-        Raises: FederationError if the signatures / hash do not match
-    """
-        # check that it's actually being sent from a valid destination to
-        # workaround bug #1753 in 0.18.5 and 0.18.6
-        if origin != get_domain_from_id(pdu.event_id):
-            # We continue to accept join events from any server; this is
-            # necessary for the federation join dance to work correctly.
-            # (When we join over federation, the "helper" server is
-            # responsible for sending out the join event, rather than the
-            # origin. See bug #1893).
-            if not (
-                pdu.type == 'm.room.member' and
-                pdu.content and
-                pdu.content.get("membership", None) == 'join'
-            ):
-                logger.info(
-                    "Discarding PDU %s from invalid origin %s",
-                    pdu.event_id, origin
-                )
-                return
-            else:
-                logger.info(
-                    "Accepting join PDU %s from %s",
-                    pdu.event_id, origin
-                )
+        # FIXME: Currently we fetch an event again when we already have it
+        # if it has been marked as an outlier.
+
+        already_seen = (
+            existing and (
+                not existing.internal_metadata.is_outlier()
+                or pdu.internal_metadata.is_outlier()
+            )
+        )
+        if already_seen:
+            logger.debug("Already seen pdu %s", pdu.event_id)
+            return
 
         # Check signature.
         try:
@@ -581,7 +529,143 @@ class FederationServer(FederationBase):
                 affected=pdu.event_id,
             )
 
-        yield self.handler.on_receive_pdu(origin, pdu, get_missing=True)
+        state = None
+
+        auth_chain = []
+
+        have_seen = yield self.store.have_events(
+            [ev for ev, _ in pdu.prev_events]
+        )
+
+        fetch_state = False
+
+        # Get missing pdus if necessary.
+        if not pdu.internal_metadata.is_outlier():
+            # We only backfill backwards to the min depth.
+            min_depth = yield self.handler.get_min_depth_for_context(
+                pdu.room_id
+            )
+
+            logger.debug(
+                "_handle_new_pdu min_depth for %s: %d",
+                pdu.room_id, min_depth
+            )
+
+            prevs = {e_id for e_id, _ in pdu.prev_events}
+            seen = set(have_seen.keys())
+
+            if min_depth and pdu.depth < min_depth:
+                # This is so that we don't notify the user about this
+                # message, to work around the fact that some events will
+                # reference really really old events we really don't want to
+                # send to the clients.
+                pdu.internal_metadata.outlier = True
+            elif min_depth and pdu.depth > min_depth:
+                if get_missing and prevs - seen:
+                    # If we're missing stuff, ensure we only fetch stuff one
+                    # at a time.
+                    logger.info(
+                        "Acquiring lock for room %r to fetch %d missing events: %r...",
+                        pdu.room_id, len(prevs - seen), list(prevs - seen)[:5],
+                    )
+                    with (yield self._room_pdu_linearizer.queue(pdu.room_id)):
+                        logger.info(
+                            "Acquired lock for room %r to fetch %d missing events",
+                            pdu.room_id, len(prevs - seen),
+                        )
+
+                        # We recalculate seen, since it may have changed.
+                        have_seen = yield self.store.have_events(prevs)
+                        seen = set(have_seen.keys())
+
+                        if prevs - seen:
+                            latest = yield self.store.get_latest_event_ids_in_room(
+                                pdu.room_id
+                            )
+
+                            # We add the prev events that we have seen to the latest
+                            # list to ensure the remote server doesn't give them to us
+                            latest = set(latest)
+                            latest |= seen
+
+                            logger.info(
+                                "Missing %d events for room %r: %r...",
+                                len(prevs - seen), pdu.room_id, list(prevs - seen)[:5]
+                            )
+
+                            # XXX: we set timeout to 10s to help workaround
+                            # https://github.com/matrix-org/synapse/issues/1733.
+                            # The reason is to avoid holding the linearizer lock
+                            # whilst processing inbound /send transactions, causing
+                            # FDs to stack up and block other inbound transactions
+                            # which empirically can currently take up to 30 minutes.
+                            #
+                            # N.B. this explicitly disables retry attempts.
+                            #
+                            # N.B. this also increases our chances of falling back to
+                            # fetching fresh state for the room if the missing event
+                            # can't be found, which slightly reduces our security.
+                            # it may also increase our DAG extremity count for the room,
+                            # causing additional state resolution?  See #1760.
+                            # However, fetching state doesn't hold the linearizer lock
+                            # apparently.
+                            #
+                            # see https://github.com/matrix-org/synapse/pull/1744
+
+                            missing_events = yield self.get_missing_events(
+                                origin,
+                                pdu.room_id,
+                                earliest_events_ids=list(latest),
+                                latest_events=[pdu],
+                                limit=10,
+                                min_depth=min_depth,
+                                timeout=10000,
+                            )
+
+                            # We want to sort these by depth so we process them and
+                            # tell clients about them in order.
+                            missing_events.sort(key=lambda x: x.depth)
+
+                            for e in missing_events:
+                                yield self._handle_new_pdu(
+                                    origin,
+                                    e,
+                                    get_missing=False
+                                )
+
+                            have_seen = yield self.store.have_events(
+                                [ev for ev, _ in pdu.prev_events]
+                            )
+
+            prevs = {e_id for e_id, _ in pdu.prev_events}
+            seen = set(have_seen.keys())
+            if prevs - seen:
+                logger.info(
+                    "Still missing %d events for room %r: %r...",
+                    len(prevs - seen), pdu.room_id, list(prevs - seen)[:5]
+                )
+                fetch_state = True
+
+        if fetch_state:
+            # We need to get the state at this event, since we haven't
+            # processed all the prev events.
+            logger.debug(
+                "_handle_new_pdu getting state for %s",
+                pdu.room_id
+            )
+            try:
+                state, auth_chain = yield self.get_state_for_room(
+                    origin, pdu.room_id, pdu.event_id,
+                )
+            except:
+                logger.exception("Failed to get state for event: %s", pdu.event_id)
+
+        yield self.handler.on_receive_pdu(
+            origin,
+            pdu,
+            state=state,
+            auth_chain=auth_chain,
+        )
 
     def __str__(self):
         return "<ReplicationLayer(%s)>" % self.server_name

synapse/federation/send_queue.py

@@ -31,41 +31,41 @@ Events are replicated via a separate events stream.
 
 from .units import Edu
 
-from synapse.storage.presence import UserPresenceState
 from synapse.util.metrics import Measure
 import synapse.metrics
 
 from blist import sorteddict
-from collections import namedtuple
-
-import logging
-
-logger = logging.getLogger(__name__)
+import ujson
 
 
 metrics = synapse.metrics.get_metrics_for(__name__)
 
 
+PRESENCE_TYPE = "p"
+KEYED_EDU_TYPE = "k"
+EDU_TYPE = "e"
+FAILURE_TYPE = "f"
+DEVICE_MESSAGE_TYPE = "d"
+
+
 class FederationRemoteSendQueue(object):
     """A drop in replacement for TransactionQueue"""
 
     def __init__(self, hs):
         self.server_name = hs.hostname
         self.clock = hs.get_clock()
-        self.notifier = hs.get_notifier()
-        self.is_mine_id = hs.is_mine_id
 
-        self.presence_map = {}  # Pending presence map user_id -> UserPresenceState
-        self.presence_changed = sorteddict()  # Stream position -> user_id
+        self.presence_map = {}
+        self.presence_changed = sorteddict()
 
-        self.keyed_edu = {}  # (destination, key) -> EDU
-        self.keyed_edu_changed = sorteddict()  # stream position -> (destination, key)
+        self.keyed_edu = {}
+        self.keyed_edu_changed = sorteddict()
 
-        self.edus = sorteddict()  # stream position -> Edu
+        self.edus = sorteddict()
 
-        self.failures = sorteddict()  # stream position -> (destination, Failure)
+        self.failures = sorteddict()
 
-        self.device_messages = sorteddict()  # stream position -> destination
+        self.device_messages = sorteddict()
 
         self.pos = 1
         self.pos_time = sorteddict()
@@ -121,9 +121,7 @@ class FederationRemoteSendQueue(object):
                 del self.presence_changed[key]
 
             user_ids = set(
-                user_id
-                for uids in self.presence_changed.itervalues()
-                for user_id in uids
+                user_id for uids in self.presence_changed.values() for _, user_id in uids
             )
 
             to_del = [
@@ -188,50 +186,37 @@ class FederationRemoteSendQueue(object):
         else:
             self.edus[pos] = edu
 
-        self.notifier.on_new_replication_data()
-
-    def send_presence(self, states):
-        """As per TransactionQueue
-
-        Args:
-            states (list(UserPresenceState))
-        """
+    def send_presence(self, destination, states):
+        """As per TransactionQueue"""
         pos = self._next_pos()
 
-        # We only want to send presence for our own users, so lets always just
-        # filter here just in case.
-        local_states = filter(lambda s: self.is_mine_id(s.user_id), states)
+        self.presence_map.update({
+            state.user_id: state
+            for state in states
+        })
 
-        self.presence_map.update({state.user_id: state for state in local_states})
-        self.presence_changed[pos] = [state.user_id for state in local_states]
-
-        self.notifier.on_new_replication_data()
+        self.presence_changed[pos] = [
+            (destination, state.user_id) for state in states
+        ]
 
     def send_failure(self, failure, destination):
         """As per TransactionQueue"""
         pos = self._next_pos()
 
         self.failures[pos] = (destination, str(failure))
-        self.notifier.on_new_replication_data()
 
     def send_device_messages(self, destination):
         """As per TransactionQueue"""
         pos = self._next_pos()
         self.device_messages[pos] = destination
-        self.notifier.on_new_replication_data()
 
     def get_current_token(self):
         return self.pos - 1
 
-    def federation_ack(self, token):
-        self._clear_queue_before_pos(token)
-
-    def get_replication_rows(self, from_token, to_token, limit, federation_ack=None):
-        """Get rows to be sent over federation between the two tokens
-
+    def get_replication_rows(self, token, limit, federation_ack=None):
+        """
         Args:
-            from_token (int)
-            to_token(int)
+            token (int)
             limit (int)
             federation_ack (int): Optional. The position where the worker is
                 explicitly acknowledged it has handled. Allows us to drop
@@ -240,11 +225,9 @@ class FederationRemoteSendQueue(object):
         # TODO: Handle limit.
 
         # To handle restarts where we wrap around
-        if from_token > self.pos:
-            from_token = -1
+        if token > self.pos:
+            token = -1
 
-        # list of tuple(int, BaseFederationRow), where the first is the position
-        # of the federation stream.
         rows = []
 
         # There should be only one reader, so lets delete everything its
@@ -254,295 +237,62 @@ class FederationRemoteSendQueue(object):
 
         # Fetch changed presence
         keys = self.presence_changed.keys()
-        i = keys.bisect_right(from_token)
-        j = keys.bisect_right(to_token) + 1
-        dest_user_ids = [
-            (pos, user_id)
-            for pos in keys[i:j]
-            for user_id in self.presence_changed[pos]
-        ]
+        i = keys.bisect_right(token)
+        dest_user_ids = set(
+            (pos, dest_user_id)
+            for pos in keys[i:]
+            for dest_user_id in self.presence_changed[pos]
+        )
 
-        for (key, user_id) in dest_user_ids:
-            rows.append((key, PresenceRow(
-                state=self.presence_map[user_id],
-            )))
+        for (key, (dest, user_id)) in dest_user_ids:
+            rows.append((key, PRESENCE_TYPE, ujson.dumps({
+                "destination": dest,
+                "state": self.presence_map[user_id].as_dict(),
+            })))
 
         # Fetch changes keyed edus
         keys = self.keyed_edu_changed.keys()
-        i = keys.bisect_right(from_token)
-        j = keys.bisect_right(to_token) + 1
-        # We purposefully clobber based on the key here, python dict comprehensions
-        # always use the last value, so this will correctly point to the last
-        # stream position.
-        keyed_edus = {self.keyed_edu_changed[k]: k for k in keys[i:j]}
+        i = keys.bisect_right(token)
+        keyed_edus = set((k, self.keyed_edu_changed[k]) for k in keys[i:])
 
-        for ((destination, edu_key), pos) in keyed_edus.iteritems():
-            rows.append((pos, KeyedEduRow(
-                key=edu_key,
-                edu=self.keyed_edu[(destination, edu_key)],
-            )))
+        for (pos, (destination, edu_key)) in keyed_edus:
+            rows.append(
+                (pos, KEYED_EDU_TYPE, ujson.dumps({
+                    "key": edu_key,
+                    "edu": self.keyed_edu[(destination, edu_key)].get_internal_dict(),
+                }))
+            )
 
         # Fetch changed edus
         keys = self.edus.keys()
-        i = keys.bisect_right(from_token)
-        j = keys.bisect_right(to_token) + 1
-        edus = ((k, self.edus[k]) for k in keys[i:j])
+        i = keys.bisect_right(token)
+        edus = set((k, self.edus[k]) for k in keys[i:])
 
         for (pos, edu) in edus:
-            rows.append((pos, EduRow(edu)))
+            rows.append((pos, EDU_TYPE, ujson.dumps(edu.get_internal_dict())))
 
         # Fetch changed failures
         keys = self.failures.keys()
-        i = keys.bisect_right(from_token)
-        j = keys.bisect_right(to_token) + 1
-        failures = ((k, self.failures[k]) for k in keys[i:j])
+        i = keys.bisect_right(token)
+        failures = set((k, self.failures[k]) for k in keys[i:])
 
         for (pos, (destination, failure)) in failures:
-            rows.append((pos, FailureRow(
-                destination=destination,
-                failure=failure,
-            )))
+            rows.append((pos, FAILURE_TYPE, ujson.dumps({
+                "destination": destination,
+                "failure": failure,
+            })))
 
         # Fetch changed device messages
         keys = self.device_messages.keys()
-        i = keys.bisect_right(from_token)
-        j = keys.bisect_right(to_token) + 1
-        device_messages = {self.device_messages[k]: k for k in keys[i:j]}
+        i = keys.bisect_right(token)
+        device_messages = set((k, self.device_messages[k]) for k in keys[i:])
 
-        for (destination, pos) in device_messages.iteritems():
-            rows.append((pos, DeviceRow(
-                destination=destination,
-            )))
+        for (pos, destination) in device_messages:
+            rows.append((pos, DEVICE_MESSAGE_TYPE, ujson.dumps({
+                "destination": destination,
+            })))
 
         # Sort rows based on pos
         rows.sort()
 
-        return [(pos, row.TypeId, row.to_data()) for pos, row in rows]
-
-
-class BaseFederationRow(object):
-    """Base class for rows to be sent in the federation stream.
-
-    Specifies how to identify, serialize and deserialize the different types.
-    """
-
-    TypeId = None  # Unique string that ids the type. Must be overriden in sub classes.
-
-    @staticmethod
-    def from_data(data):
-        """Parse the data from the federation stream into a row.
-
-        Args:
-            data: The value of ``data`` from FederationStreamRow.data, type
-                depends on the type of stream
-        """
-        raise NotImplementedError()
-
-    def to_data(self):
-        """Serialize this row to be sent over the federation stream.
-
-        Returns:
-            The value to be sent in FederationStreamRow.data. The type depends
-            on the type of stream.
-        """
-        raise NotImplementedError()
-
-    def add_to_buffer(self, buff):
-        """Add this row to the appropriate field in the buffer ready for this
-        to be sent over federation.
-
-        We use a buffer so that we can batch up events that have come in at
-        the same time and send them all at once.
-
-        Args:
-            buff (BufferedToSend)
-        """
-        raise NotImplementedError()
-
-
-class PresenceRow(BaseFederationRow, namedtuple("PresenceRow", (
-    "state",  # UserPresenceState
-))):
-    TypeId = "p"
-
-    @staticmethod
-    def from_data(data):
-        return PresenceRow(
-            state=UserPresenceState.from_dict(data)
-        )
-
-    def to_data(self):
-        return self.state.as_dict()
-
-    def add_to_buffer(self, buff):
-        buff.presence.append(self.state)
-
-
-class KeyedEduRow(BaseFederationRow, namedtuple("KeyedEduRow", (
-    "key",  # tuple(str) - the edu key passed to send_edu
-    "edu",  # Edu
-))):
-    """Streams EDUs that have an associated key that is ued to clobber. For example,
-    typing EDUs clobber based on room_id.
-    """
-
-    TypeId = "k"
-
-    @staticmethod
-    def from_data(data):
-        return KeyedEduRow(
-            key=tuple(data["key"]),
-            edu=Edu(**data["edu"]),
-        )
-
-    def to_data(self):
-        return {
-            "key": self.key,
-            "edu": self.edu.get_internal_dict(),
-        }
-
-    def add_to_buffer(self, buff):
-        buff.keyed_edus.setdefault(
-            self.edu.destination, {}
-        )[self.key] = self.edu
-
-
-class EduRow(BaseFederationRow, namedtuple("EduRow", (
-    "edu",  # Edu
-))):
-    """Streams EDUs that don't have keys. See KeyedEduRow
-    """
-    TypeId = "e"
-
-    @staticmethod
-    def from_data(data):
-        return EduRow(Edu(**data))
-
-    def to_data(self):
-        return self.edu.get_internal_dict()
-
-    def add_to_buffer(self, buff):
-        buff.edus.setdefault(self.edu.destination, []).append(self.edu)
-
-
-class FailureRow(BaseFederationRow, namedtuple("FailureRow", (
-    "destination",  # str
-    "failure",
-))):
-    """Streams failures to a remote server. Failures are issued when there was
-    something wrong with a transaction the remote sent us, e.g. it included
-    an event that was invalid.
-    """
-
-    TypeId = "f"
-
-    @staticmethod
-    def from_data(data):
-        return FailureRow(
-            destination=data["destination"],
-            failure=data["failure"],
-        )
-
-    def to_data(self):
-        return {
-            "destination": self.destination,
-            "failure": self.failure,
-        }
-
-    def add_to_buffer(self, buff):
-        buff.failures.setdefault(self.destination, []).append(self.failure)
-
-
-class DeviceRow(BaseFederationRow, namedtuple("DeviceRow", (
-    "destination",  # str
-))):
-    """Streams the fact that either a) there is pending to device messages for
-    users on the remote, or b) a local users device has changed and needs to
-    be sent to the remote.
-    """
-    TypeId = "d"
-
-    @staticmethod
-    def from_data(data):
-        return DeviceRow(destination=data["destination"])
-
-    def to_data(self):
-        return {"destination": self.destination}
-
-    def add_to_buffer(self, buff):
-        buff.device_destinations.add(self.destination)
-
-
-TypeToRow = {
-    Row.TypeId: Row
-    for Row in (
-        PresenceRow,
-        KeyedEduRow,
-        EduRow,
-        FailureRow,
-        DeviceRow,
-    )
-}
-
-
-ParsedFederationStreamData = namedtuple("ParsedFederationStreamData", (
-    "presence",  # list(UserPresenceState)
-    "keyed_edus",  # dict of destination -> { key -> Edu }
-    "edus",  # dict of destination -> [Edu]
-    "failures",  # dict of destination -> [failures]
-    "device_destinations",  # set of destinations
-))
-
-
-def process_rows_for_federation(transaction_queue, rows):
-    """Parse a list of rows from the federation stream and put them in the
-    transaction queue ready for sending to the relevant homeservers.
-
-    Args:
-        transaction_queue (TransactionQueue)
-        rows (list(synapse.replication.tcp.streams.FederationStreamRow))
-    """
-
-    # The federation stream contains a bunch of different types of
-    # rows that need to be handled differently. We parse the rows, put
-    # them into the appropriate collection and then send them off.
-
-    buff = ParsedFederationStreamData(
-        presence=[],
-        keyed_edus={},
-        edus={},
-        failures={},
-        device_destinations=set(),
-    )
-
-    # Parse the rows in the stream and add to the buffer
-    for row in rows:
-        if row.type not in TypeToRow:
-            logger.error("Unrecognized federation row type %r", row.type)
-            continue
-
-        RowType = TypeToRow[row.type]
-        parsed_row = RowType.from_data(row.data)
-        parsed_row.add_to_buffer(buff)
-
-    if buff.presence:
-        transaction_queue.send_presence(buff.presence)
-
-    for destination, edu_map in buff.keyed_edus.iteritems():
-        for key, edu in edu_map.items():
-            transaction_queue.send_edu(
-                edu.destination, edu.edu_type, edu.content, key=key,
-            )
-
-    for destination, edu_list in buff.edus.iteritems():
-        for edu in edu_list:
-            transaction_queue.send_edu(
-                edu.destination, edu.edu_type, edu.content, key=None,
-            )
-
-    for destination, failure_list in buff.failures.iteritems():
-        for failure in failure_list:
-            transaction_queue.send_failure(destination, failure)
-
-    for destination in buff.device_destinations:
-        transaction_queue.send_device_messages(destination)
+        return rows

synapse/federation/transaction_queue.py

@@ -12,7 +12,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import datetime
+
 
 from twisted.internet import defer
 
@@ -20,11 +20,14 @@ from .persistence import TransactionActions
 from .units import Transaction, Edu
 
 from synapse.api.errors import HttpResponseException
-from synapse.util import logcontext
 from synapse.util.async import run_on_reactor
-from synapse.util.retryutils import NotRetryingDestination, get_retry_limiter
+from synapse.util.logcontext import preserve_context_over_fn
+from synapse.util.retryutils import (
+    get_retry_limiter, NotRetryingDestination,
+)
 from synapse.util.metrics import measure_func
-from synapse.handlers.presence import format_user_presence_state, get_interested_remotes
+from synapse.types import get_domain_from_id
+from synapse.handlers.presence import format_user_presence_state
 import synapse.metrics
 
 import logging
@@ -40,8 +43,6 @@ sent_pdus_destination_dist = client_metrics.register_distribution(
 )
 sent_edus_counter = client_metrics.register_counter("sent_edus")
 
-sent_transactions_counter = client_metrics.register_counter("sent_transactions")
-
 
 class TransactionQueue(object):
     """This class makes sure we only have one transaction in flight at
@@ -78,18 +79,8 @@ class TransactionQueue(object):
         # destination -> list of tuple(edu, deferred)
         self.pending_edus_by_dest = edus = {}
 
-        # Map of user_id -> UserPresenceState for all the pending presence
-        # to be sent out by user_id. Entries here get processed and put in
-        # pending_presence_by_dest
-        self.pending_presence = {}
-
-        # Map of destination -> user_id -> UserPresenceState of pending presence
-        # to be sent to each destinations
+        # Presence needs to be separate as we send single aggragate EDUs
         self.pending_presence_by_dest = presence = {}
-
-        # Pending EDUs by their "key". Keyed EDUs are EDUs that get clobbered
-        # based on their key (e.g. typing events by room_id)
-        # Map of destination -> (edu_type, key) -> Edu
         self.pending_edus_keyed_by_dest = edus_keyed = {}
 
         metrics.register_callback(
@@ -108,12 +99,7 @@ class TransactionQueue(object):
         # destination -> list of tuple(failure, deferred)
         self.pending_failures_by_dest = {}
 
-        # destination -> stream_id of last successfully sent to-device message.
-        # NB: may be a long or an int.
         self.last_device_stream_id_by_dest = {}
-
-        # destination -> stream_id of last successfully sent device list
-        # update.
         self.last_device_list_stream_id_by_dest = {}
 
         # HACK to get unique tx id
@@ -124,8 +110,6 @@ class TransactionQueue(object):
         self._is_processing = False
         self._last_poked_id = -1
 
-        self._processing_pending_presence = False
-
     def can_send_to(self, destination):
         """Can we send messages to the given server?
 
@@ -182,13 +166,15 @@ class TransactionQueue(object):
                     # Otherwise if the last member on a server in a room is
                     # banned then it won't receive the event because it won't
                     # be in the room after the ban.
-                    destinations = yield self.state.get_current_hosts_in_room(
+                    users_in_room = yield self.state.get_current_user_in_room(
                         event.room_id, latest_event_ids=[
                             prev_id for prev_id, _ in event.prev_events
                         ],
                     )
-                    destinations = set(destinations)
 
+                    destinations = set(
+                        get_domain_from_id(user_id) for user_id in users_in_room
+                    )
                     if send_on_behalf_of is not None:
                         # If we are sending the event on behalf of another server
                         # then it already has the event and there is no reason to
@@ -231,73 +217,21 @@ class TransactionQueue(object):
                 (pdu, order)
             )
 
-            self._attempt_new_transaction(destination)
+            preserve_context_over_fn(
+                self._attempt_new_transaction, destination
+            )
 
-    @logcontext.preserve_fn  # the caller should not yield on this
-    @defer.inlineCallbacks
-    def send_presence(self, states):
-        """Send the new presence states to the appropriate destinations.
-
-        This actually queues up the presence states ready for sending and
-        triggers a background task to process them and send out the transactions.
-
-        Args:
-            states (list(UserPresenceState))
-        """
-
-        # First we queue up the new presence by user ID, so multiple presence
-        # updates in quick successtion are correctly handled
-        # We only want to send presence for our own users, so lets always just
-        # filter here just in case.
-        self.pending_presence.update({
-            state.user_id: state for state in states
-            if self.is_mine_id(state.user_id)
-        })
-
-        # We then handle the new pending presence in batches, first figuring
-        # out the destinations we need to send each state to and then poking it
-        # to attempt a new transaction. We linearize this so that we don't
-        # accidentally mess up the ordering and send multiple presence updates
-        # in the wrong order
-        if self._processing_pending_presence:
+    def send_presence(self, destination, states):
+        if not self.can_send_to(destination):
             return
 
-        self._processing_pending_presence = True
-        try:
-            while True:
-                states_map = self.pending_presence
-                self.pending_presence = {}
+        self.pending_presence_by_dest.setdefault(destination, {}).update({
+            state.user_id: state for state in states
+        })
 
-                if not states_map:
-                    break
-
-                yield self._process_presence_inner(states_map.values())
-        finally:
-            self._processing_pending_presence = False
-
-    @measure_func("txnqueue._process_presence")
-    @defer.inlineCallbacks
-    def _process_presence_inner(self, states):
-        """Given a list of states populate self.pending_presence_by_dest and
-        poke to send a new transaction to each destination
-
-        Args:
-            states (list(UserPresenceState))
-        """
-        hosts_and_states = yield get_interested_remotes(self.store, states, self.state)
-
-        for destinations, states in hosts_and_states:
-            for destination in destinations:
-                if not self.can_send_to(destination):
-                    continue
-
-                self.pending_presence_by_dest.setdefault(
-                    destination, {}
-                ).update({
-                    state.user_id: state for state in states
-                })
-
-                self._attempt_new_transaction(destination)
+        preserve_context_over_fn(
+            self._attempt_new_transaction, destination
+        )
 
     def send_edu(self, destination, edu_type, content, key=None):
         edu = Edu(
@@ -319,7 +253,9 @@ class TransactionQueue(object):
         else:
             self.pending_edus_by_dest.setdefault(destination, []).append(edu)
 
-        self._attempt_new_transaction(destination)
+        preserve_context_over_fn(
+            self._attempt_new_transaction, destination
+        )
 
     def send_failure(self, failure, destination):
         if destination == self.server_name or destination == "localhost":
@@ -332,7 +268,9 @@ class TransactionQueue(object):
             destination, []
         ).append(failure)
 
-        self._attempt_new_transaction(destination)
+        preserve_context_over_fn(
+            self._attempt_new_transaction, destination
+        )
 
     def send_device_messages(self, destination):
         if destination == self.server_name or destination == "localhost":
@@ -341,24 +279,15 @@ class TransactionQueue(object):
         if not self.can_send_to(destination):
             return
 
-        self._attempt_new_transaction(destination)
+        preserve_context_over_fn(
+            self._attempt_new_transaction, destination
+        )
 
     def get_current_token(self):
         return 0
 
+    @defer.inlineCallbacks
     def _attempt_new_transaction(self, destination):
-        """Try to start a new transaction to this destination
-
-        If there is already a transaction in progress to this destination,
-        returns immediately. Otherwise kicks off the process of sending a
-        transaction in the background.
-
-        Args:
-            destination (str):
-
-        Returns:
-            None
-        """
         # list of (pending_pdu, deferred, order)
         if destination in self.pending_transactions:
             # XXX: pending_transactions can get stuck on by a never-ending
@@ -371,33 +300,20 @@ class TransactionQueue(object):
             )
             return
 
-        logger.debug("TX [%s] Starting transaction loop", destination)
-
-        # Drop the logcontext before starting the transaction. It doesn't
-        # really make sense to log all the outbound transactions against
-        # whatever path led us to this point: that's pretty arbitrary really.
-        #
-        # (this also means we can fire off _perform_transaction without
-        # yielding)
-        with logcontext.PreserveLoggingContext():
-            self._transaction_transmission_loop(destination)
-
-    @defer.inlineCallbacks
-    def _transaction_transmission_loop(self, destination):
-        pending_pdus = []
         try:
             self.pending_transactions[destination] = 1
 
-            # This will throw if we wouldn't retry. We do this here so we fail
-            # quickly, but we will later check this again in the http client,
-            # hence why we throw the result away.
-            yield get_retry_limiter(destination, self.clock, self.store)
-
             # XXX: what's this for?
             yield run_on_reactor()
 
-            pending_pdus = []
             while True:
+                limiter = yield get_retry_limiter(
+                    destination,
+                    self.clock,
+                    self.store,
+                    backoff_on_404=True,  # If we get a 404 the other side has gone
+                )
+
                 device_message_edus, device_stream_id, dev_list_id = (
                     yield self._get_new_device_messages(destination)
                 )
@@ -453,9 +369,9 @@ class TransactionQueue(object):
 
                 success = yield self._send_new_transaction(
                     destination, pending_pdus, pending_edus, pending_failures,
+                    limiter=limiter,
                 )
                 if success:
-                    sent_transactions_counter.inc()
                     # Remove the acknowledged device messages from the database
                     # Only bother if we actually sent some device messages
                     if device_message_edus:
@@ -471,24 +387,12 @@ class TransactionQueue(object):
                     self.last_device_list_stream_id_by_dest[destination] = dev_list_id
                 else:
                     break
-        except NotRetryingDestination as e:
+        except NotRetryingDestination:
             logger.debug(
-                "TX [%s] not ready for retry yet (next retry at %s) - "
+                "TX [%s] not ready for retry yet - "
                 "dropping transaction for now",
                 destination,
-                datetime.datetime.fromtimestamp(
-                    (e.retry_last_ts + e.retry_interval) / 1000.0
-                ),
             )
-        except Exception as e:
-            logger.warn(
-                "TX [%s] Failed to send transaction: %s",
-                destination,
-                e,
-            )
-            for p, _ in pending_pdus:
-                logger.info("Failed to send event %s to %s", p.event_id,
-                            destination)
         finally:
             # We want to be *very* sure we delete this after we stop processing
             self.pending_transactions.pop(destination, None)
@@ -528,7 +432,7 @@ class TransactionQueue(object):
     @measure_func("_send_new_transaction")
     @defer.inlineCallbacks
     def _send_new_transaction(self, destination, pending_pdus, pending_edus,
-                              pending_failures):
+                              pending_failures, limiter):
 
         # Sort based on the order field
         pending_pdus.sort(key=lambda t: t[1])
@@ -538,104 +442,132 @@ class TransactionQueue(object):
 
         success = True
 
-        logger.debug("TX [%s] _attempt_new_transaction", destination)
-
-        txn_id = str(self._next_txn_id)
-
-        logger.debug(
-            "TX [%s] {%s} Attempting new transaction"
-            " (pdus: %d, edus: %d, failures: %d)",
-            destination, txn_id,
-            len(pdus),
-            len(edus),
-            len(failures)
-        )
-
-        logger.debug("TX [%s] Persisting transaction...", destination)
-
-        transaction = Transaction.create_new(
-            origin_server_ts=int(self.clock.time_msec()),
-            transaction_id=txn_id,
-            origin=self.server_name,
-            destination=destination,
-            pdus=pdus,
-            edus=edus,
-            pdu_failures=failures,
-        )
-
-        self._next_txn_id += 1
-
-        yield self.transaction_actions.prepare_to_send(transaction)
-
-        logger.debug("TX [%s] Persisted transaction", destination)
-        logger.info(
-            "TX [%s] {%s} Sending transaction [%s],"
-            " (PDUs: %d, EDUs: %d, failures: %d)",
-            destination, txn_id,
-            transaction.transaction_id,
-            len(pdus),
-            len(edus),
-            len(failures),
-        )
-
-        # Actually send the transaction
-
-        # FIXME (erikj): This is a bit of a hack to make the Pdu age
-        # keys work
-        def json_data_cb():
-            data = transaction.get_dict()
-            now = int(self.clock.time_msec())
-            if "pdus" in data:
-                for p in data["pdus"]:
-                    if "age_ts" in p:
-                        unsigned = p.setdefault("unsigned", {})
-                        unsigned["age"] = now - int(p["age_ts"])
-                        del p["age_ts"]
-            return data
-
         try:
-            response = yield self.transport_layer.send_transaction(
-                transaction, json_data_cb
+            logger.debug("TX [%s] _attempt_new_transaction", destination)
+
+            txn_id = str(self._next_txn_id)
+
+            logger.debug(
+                "TX [%s] {%s} Attempting new transaction"
+                " (pdus: %d, edus: %d, failures: %d)",
+                destination, txn_id,
+                len(pdus),
+                len(edus),
+                len(failures)
             )
-            code = 200
 
-            if response:
-                for e_id, r in response.get("pdus", {}).items():
-                    if "error" in r:
-                        logger.warn(
-                            "Transaction returned error for %s: %s",
-                            e_id, r,
+            logger.debug("TX [%s] Persisting transaction...", destination)
+
+            transaction = Transaction.create_new(
+                origin_server_ts=int(self.clock.time_msec()),
+                transaction_id=txn_id,
+                origin=self.server_name,
+                destination=destination,
+                pdus=pdus,
+                edus=edus,
+                pdu_failures=failures,
+            )
+
+            self._next_txn_id += 1
+
+            yield self.transaction_actions.prepare_to_send(transaction)
+
+            logger.debug("TX [%s] Persisted transaction", destination)
+            logger.info(
+                "TX [%s] {%s} Sending transaction [%s],"
+                " (PDUs: %d, EDUs: %d, failures: %d)",
+                destination, txn_id,
+                transaction.transaction_id,
+                len(pdus),
+                len(edus),
+                len(failures),
+            )
+
+            with limiter:
+                # Actually send the transaction
+
+                # FIXME (erikj): This is a bit of a hack to make the Pdu age
+                # keys work
+                def json_data_cb():
+                    data = transaction.get_dict()
+                    now = int(self.clock.time_msec())
+                    if "pdus" in data:
+                        for p in data["pdus"]:
+                            if "age_ts" in p:
+                                unsigned = p.setdefault("unsigned", {})
+                                unsigned["age"] = now - int(p["age_ts"])
+                                del p["age_ts"]
+                    return data
+
+                try:
+                    response = yield self.transport_layer.send_transaction(
+                        transaction, json_data_cb
+                    )
+                    code = 200
+
+                    if response:
+                        for e_id, r in response.get("pdus", {}).items():
+                            if "error" in r:
+                                logger.warn(
+                                    "Transaction returned error for %s: %s",
+                                    e_id, r,
+                                )
+                except HttpResponseException as e:
+                    code = e.code
+                    response = e.response
+
+                    if e.code in (401, 404, 429) or 500 <= e.code:
+                        logger.info(
+                            "TX [%s] {%s} got %d response",
+                            destination, txn_id, code
                         )
-        except HttpResponseException as e:
-            code = e.code
-            response = e.response
+                        raise e
 
-            if e.code in (401, 404, 429) or 500 <= e.code:
                 logger.info(
                     "TX [%s] {%s} got %d response",
                     destination, txn_id, code
                 )
-                raise e
 
-        logger.info(
-            "TX [%s] {%s} got %d response",
-            destination, txn_id, code
-        )
+                logger.debug("TX [%s] Sent transaction", destination)
+                logger.debug("TX [%s] Marking as delivered...", destination)
 
-        logger.debug("TX [%s] Sent transaction", destination)
-        logger.debug("TX [%s] Marking as delivered...", destination)
+            yield self.transaction_actions.delivered(
+                transaction, code, response
+            )
 
-        yield self.transaction_actions.delivered(
-            transaction, code, response
-        )
+            logger.debug("TX [%s] Marked as delivered", destination)
 
-        logger.debug("TX [%s] Marked as delivered", destination)
+            if code != 200:
+                for p in pdus:
+                    logger.info(
+                        "Failed to send event %s to %s", p.event_id, destination
+                    )
+                success = False
+        except RuntimeError as e:
+            # We capture this here as there as nothing actually listens
+            # for this finishing functions deferred.
+            logger.warn(
+                "TX [%s] Problem in _attempt_transaction: %s",
+                destination,
+                e,
+            )
 
-        if code != 200:
-            for p in pdus:
-                logger.info(
-                    "Failed to send event %s to %s", p.event_id, destination
-                )
             success = False
 
+            for p in pdus:
+                logger.info("Failed to send event %s to %s", p.event_id, destination)
+        except Exception as e:
+            # We capture this here as there as nothing actually listens
+            # for this finishing functions deferred.
+            logger.warn(
+                "TX [%s] Problem in _attempt_transaction: %s",
+                destination,
+                e,
+            )
+
+            success = False
+
+            for p in pdus:
+                logger.info("Failed to send event %s to %s", p.event_id, destination)
+
         defer.returnValue(success)

synapse/federation/transport/client.py

@@ -163,7 +163,6 @@ class TransportLayerClient(object):
             data=json_data,
             json_data_callback=json_data_callback,
             long_retries=True,
-            backoff_on_404=True,  # If we get a 404 the other side has gone
         )
 
         logger.debug(
@@ -175,8 +174,7 @@ class TransportLayerClient(object):
 
     @defer.inlineCallbacks
     @log_function
-    def make_query(self, destination, query_type, args, retry_on_dns_fail,
-                   ignore_backoff=False):
+    def make_query(self, destination, query_type, args, retry_on_dns_fail):
         path = PREFIX + "/query/%s" % query_type
 
         content = yield self.client.get_json(
@@ -185,7 +183,6 @@ class TransportLayerClient(object):
             args=args,
             retry_on_dns_fail=retry_on_dns_fail,
             timeout=10000,
-            ignore_backoff=ignore_backoff,
         )
 
         defer.returnValue(content)
@@ -193,26 +190,6 @@ class TransportLayerClient(object):
     @defer.inlineCallbacks
     @log_function
     def make_membership_event(self, destination, room_id, user_id, membership):
-        """Asks a remote server to build and sign us a membership event
-
-        Note that this does not append any events to any graphs.
-
-        Args:
-            destination (str): address of remote homeserver
-            room_id (str): room to join/leave
-            user_id (str): user to be joined/left
-            membership (str): one of join/leave
-
-        Returns:
-            Deferred: Succeeds when we get a 2xx HTTP response. The result
-            will be the decoded JSON body (ie, the new event).
-
-            Fails with ``HTTPRequestException`` if we get an HTTP response
-            code >= 300.
-
-            Fails with ``NotRetryingDestination`` if we are not yet ready
-            to retry this server.
-        """
         valid_memberships = {Membership.JOIN, Membership.LEAVE}
         if membership not in valid_memberships:
             raise RuntimeError(
@@ -221,23 +198,11 @@ class TransportLayerClient(object):
             )
         path = PREFIX + "/make_%s/%s/%s" % (membership, room_id, user_id)
 
-        ignore_backoff = False
-        retry_on_dns_fail = False
-
-        if membership == Membership.LEAVE:
-            # we particularly want to do our best to send leave events. The
-            # problem is that if it fails, we won't retry it later, so if the
-            # remote server was just having a momentary blip, the room will be
-            # out of sync.
-            ignore_backoff = True
-            retry_on_dns_fail = True
-
         content = yield self.client.get_json(
             destination=destination,
             path=path,
-            retry_on_dns_fail=retry_on_dns_fail,
+            retry_on_dns_fail=False,
             timeout=20000,
-            ignore_backoff=ignore_backoff,
         )
 
         defer.returnValue(content)
@@ -264,12 +229,6 @@ class TransportLayerClient(object):
             destination=destination,
             path=path,
             data=content,
-
-            # we want to do our best to send this through. The problem is
-            # that if it fails, we won't retry it later, so if the remote
-            # server was just having a momentary blip, the room will be out of
-            # sync.
-            ignore_backoff=True,
         )
 
         defer.returnValue(response)
@@ -283,7 +242,6 @@ class TransportLayerClient(object):
             destination=destination,
             path=path,
             data=content,
-            ignore_backoff=True,
         )
 
         defer.returnValue(response)
@@ -311,7 +269,6 @@ class TransportLayerClient(object):
             destination=remote_server,
             path=path,
             args=args,
-            ignore_backoff=True,
         )
 
         defer.returnValue(response)
@@ -471,418 +428,3 @@ class TransportLayerClient(object):
         )
 
         defer.returnValue(content)
-
-    @log_function
-    def get_group_profile(self, destination, group_id, requester_user_id):
-        """Get a group profile
-        """
-        path = PREFIX + "/groups/%s/profile" % (group_id,)
-
-        return self.client.get_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def update_group_profile(self, destination, group_id, requester_user_id, content):
-        """Update a remote group profile
-
-        Args:
-            destination (str)
-            group_id (str)
-            requester_user_id (str)
-            content (dict): The new profile of the group
-        """
-        path = PREFIX + "/groups/%s/profile" % (group_id,)
-
-        return self.client.post_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            data=content,
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def get_group_summary(self, destination, group_id, requester_user_id):
-        """Get a group summary
-        """
-        path = PREFIX + "/groups/%s/summary" % (group_id,)
-
-        return self.client.get_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def get_rooms_in_group(self, destination, group_id, requester_user_id):
-        """Get all rooms in a group
-        """
-        path = PREFIX + "/groups/%s/rooms" % (group_id,)
-
-        return self.client.get_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            ignore_backoff=True,
-        )
-
-    def add_room_to_group(self, destination, group_id, requester_user_id, room_id,
-                          content):
-        """Add a room to a group
-        """
-        path = PREFIX + "/groups/%s/room/%s" % (group_id, room_id,)
-
-        return self.client.post_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            data=content,
-            ignore_backoff=True,
-        )
-
-    def update_room_in_group(self, destination, group_id, requester_user_id, room_id,
-                             config_key, content):
-        """Update room in group
-        """
-        path = PREFIX + "/groups/%s/room/%s/config/%s" % (group_id, room_id, config_key,)
-
-        return self.client.post_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            data=content,
-            ignore_backoff=True,
-        )
-
-    def remove_room_from_group(self, destination, group_id, requester_user_id, room_id):
-        """Remove a room from a group
-        """
-        path = PREFIX + "/groups/%s/room/%s" % (group_id, room_id,)
-
-        return self.client.delete_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def get_users_in_group(self, destination, group_id, requester_user_id):
-        """Get users in a group
-        """
-        path = PREFIX + "/groups/%s/users" % (group_id,)
-
-        return self.client.get_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def get_invited_users_in_group(self, destination, group_id, requester_user_id):
-        """Get users that have been invited to a group
-        """
-        path = PREFIX + "/groups/%s/invited_users" % (group_id,)
-
-        return self.client.get_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def accept_group_invite(self, destination, group_id, user_id, content):
-        """Accept a group invite
-        """
-        path = PREFIX + "/groups/%s/users/%s/accept_invite" % (group_id, user_id)
-
-        return self.client.post_json(
-            destination=destination,
-            path=path,
-            data=content,
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def invite_to_group(self, destination, group_id, user_id, requester_user_id, content):
-        """Invite a user to a group
-        """
-        path = PREFIX + "/groups/%s/users/%s/invite" % (group_id, user_id)
-
-        return self.client.post_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            data=content,
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def invite_to_group_notification(self, destination, group_id, user_id, content):
-        """Sent by group server to inform a user's server that they have been
-        invited.
-        """
-
-        path = PREFIX + "/groups/local/%s/users/%s/invite" % (group_id, user_id)
-
-        return self.client.post_json(
-            destination=destination,
-            path=path,
-            data=content,
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def remove_user_from_group(self, destination, group_id, requester_user_id,
-                               user_id, content):
-        """Remove a user fron a group
-        """
-        path = PREFIX + "/groups/%s/users/%s/remove" % (group_id, user_id)
-
-        return self.client.post_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            data=content,
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def remove_user_from_group_notification(self, destination, group_id, user_id,
-                                            content):
-        """Sent by group server to inform a user's server that they have been
-        kicked from the group.
-        """
-
-        path = PREFIX + "/groups/local/%s/users/%s/remove" % (group_id, user_id)
-
-        return self.client.post_json(
-            destination=destination,
-            path=path,
-            data=content,
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def renew_group_attestation(self, destination, group_id, user_id, content):
-        """Sent by either a group server or a user's server to periodically update
-        the attestations
-        """
-
-        path = PREFIX + "/groups/%s/renew_attestation/%s" % (group_id, user_id)
-
-        return self.client.post_json(
-            destination=destination,
-            path=path,
-            data=content,
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def update_group_summary_room(self, destination, group_id, user_id, room_id,
-                                  category_id, content):
-        """Update a room entry in a group summary
-        """
-        if category_id:
-            path = PREFIX + "/groups/%s/summary/categories/%s/rooms/%s" % (
-                group_id, category_id, room_id,
-            )
-        else:
-            path = PREFIX + "/groups/%s/summary/rooms/%s" % (group_id, room_id,)
-
-        return self.client.post_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": user_id},
-            data=content,
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def delete_group_summary_room(self, destination, group_id, user_id, room_id,
-                                  category_id):
-        """Delete a room entry in a group summary
-        """
-        if category_id:
-            path = PREFIX + "/groups/%s/summary/categories/%s/rooms/%s" % (
-                group_id, category_id, room_id,
-            )
-        else:
-            path = PREFIX + "/groups/%s/summary/rooms/%s" % (group_id, room_id,)
-
-        return self.client.delete_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": user_id},
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def get_group_categories(self, destination, group_id, requester_user_id):
-        """Get all categories in a group
-        """
-        path = PREFIX + "/groups/%s/categories" % (group_id,)
-
-        return self.client.get_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def get_group_category(self, destination, group_id, requester_user_id, category_id):
-        """Get category info in a group
-        """
-        path = PREFIX + "/groups/%s/categories/%s" % (group_id, category_id,)
-
-        return self.client.get_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def update_group_category(self, destination, group_id, requester_user_id, category_id,
-                              content):
-        """Update a category in a group
-        """
-        path = PREFIX + "/groups/%s/categories/%s" % (group_id, category_id,)
-
-        return self.client.post_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            data=content,
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def delete_group_category(self, destination, group_id, requester_user_id,
-                              category_id):
-        """Delete a category in a group
-        """
-        path = PREFIX + "/groups/%s/categories/%s" % (group_id, category_id,)
-
-        return self.client.delete_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def get_group_roles(self, destination, group_id, requester_user_id):
-        """Get all roles in a group
-        """
-        path = PREFIX + "/groups/%s/roles" % (group_id,)
-
-        return self.client.get_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def get_group_role(self, destination, group_id, requester_user_id, role_id):
-        """Get a roles info
-        """
-        path = PREFIX + "/groups/%s/roles/%s" % (group_id, role_id,)
-
-        return self.client.get_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def update_group_role(self, destination, group_id, requester_user_id, role_id,
-                          content):
-        """Update a role in a group
-        """
-        path = PREFIX + "/groups/%s/roles/%s" % (group_id, role_id,)
-
-        return self.client.post_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            data=content,
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def delete_group_role(self, destination, group_id, requester_user_id, role_id):
-        """Delete a role in a group
-        """
-        path = PREFIX + "/groups/%s/roles/%s" % (group_id, role_id,)
-
-        return self.client.delete_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def update_group_summary_user(self, destination, group_id, requester_user_id,
-                                  user_id, role_id, content):
-        """Update a users entry in a group
-        """
-        if role_id:
-            path = PREFIX + "/groups/%s/summary/roles/%s/users/%s" % (
-                group_id, role_id, user_id,
-            )
-        else:
-            path = PREFIX + "/groups/%s/summary/users/%s" % (group_id, user_id,)
-
-        return self.client.post_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            data=content,
-            ignore_backoff=True,
-        )
-
-    @log_function
-    def delete_group_summary_user(self, destination, group_id, requester_user_id,
-                                  user_id, role_id):
-        """Delete a users entry in a group
-        """
-        if role_id:
-            path = PREFIX + "/groups/%s/summary/roles/%s/users/%s" % (
-                group_id, role_id, user_id,
-            )
-        else:
-            path = PREFIX + "/groups/%s/summary/users/%s" % (group_id, user_id,)
-
-        return self.client.delete_json(
-            destination=destination,
-            path=path,
-            args={"requester_user_id": requester_user_id},
-            ignore_backoff=True,
-        )
-
-    def bulk_get_publicised_groups(self, destination, user_ids):
-        """Get the groups a list of users are publicising
-        """
-
-        path = PREFIX + "/get_groups_publicised"
-
-        content = {"user_ids": user_ids}
-
-        return self.client.post_json(
-            destination=destination,
-            path=path,
-            data=content,
-            ignore_backoff=True,
-        )

synapse/federation/transport/server.py

@@ -24,8 +24,7 @@ from synapse.http.servlet import (
 )
 from synapse.util.ratelimitutils import FederationRateLimiter
 from synapse.util.versionstring import get_version_string
-from synapse.util.logcontext import preserve_fn
-from synapse.types import ThirdPartyInstanceID, get_domain_from_id
+from synapse.types import ThirdPartyInstanceID
 
 import functools
 import logging
@@ -80,7 +79,6 @@ class Authenticator(object):
     def __init__(self, hs):
         self.keyring = hs.get_keyring()
         self.server_name = hs.hostname
-        self.store = hs.get_datastore()
 
     # A method just so we can pass 'self' as the authenticator to the Servlets
     @defer.inlineCallbacks
@@ -112,7 +110,7 @@ class Authenticator(object):
                 key = strip_quotes(param_dict["key"])
                 sig = strip_quotes(param_dict["sig"])
                 return (origin, key, sig)
-            except Exception:
+            except:
                 raise AuthenticationError(
                     400, "Malformed Authorization header", Codes.UNAUTHORIZED
                 )
@@ -140,23 +138,18 @@ class Authenticator(object):
         logger.info("Request from %s", origin)
         request.authenticated_entity = origin
 
-        # If we get a valid signed request from the other side, its probably
-        # alive
-        retry_timings = yield self.store.get_destination_retry_timings(origin)
-        if retry_timings and retry_timings["retry_last_ts"]:
-            logger.info("Marking origin %r as up", origin)
-            preserve_fn(self.store.set_destination_retry_timings)(origin, 0, 0)
-
         defer.returnValue(origin)
 
 
 class BaseFederationServlet(object):
     REQUIRE_AUTH = True
 
-    def __init__(self, handler, authenticator, ratelimiter, server_name):
+    def __init__(self, handler, authenticator, ratelimiter, server_name,
+                 room_list_handler):
         self.handler = handler
         self.authenticator = authenticator
         self.ratelimiter = ratelimiter
+        self.room_list_handler = room_list_handler
 
     def _wrap(self, func):
         authenticator = self.authenticator
@@ -177,7 +170,7 @@ class BaseFederationServlet(object):
                 if self.REQUIRE_AUTH:
                     logger.exception("authenticate_request failed")
                     raise
-            except Exception:
+            except:
                 logger.exception("authenticate_request failed")
                 raise
 
@@ -270,7 +263,7 @@ class FederationSendServlet(BaseFederationServlet):
             code, response = yield self.handler.on_incoming_transaction(
                 transaction_data
             )
-        except Exception:
+        except:
             logger.exception("on_incoming_transaction failed")
             raise
 
@@ -588,7 +581,7 @@ class PublicRoomList(BaseFederationServlet):
         else:
             network_tuple = ThirdPartyInstanceID(None, None)
 
-        data = yield self.handler.get_local_public_room_list(
+        data = yield self.room_list_handler.get_local_public_room_list(
             limit, since_token,
             network_tuple=network_tuple
         )
@@ -609,515 +602,7 @@ class FederationVersionServlet(BaseFederationServlet):
         }))
 
 
-class FederationGroupsProfileServlet(BaseFederationServlet):
-    """Get/set the basic profile of a group on behalf of a user
-    """
-    PATH = "/groups/(?P<group_id>[^/]*)/profile$"
-
-    @defer.inlineCallbacks
-    def on_GET(self, origin, content, query, group_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        new_content = yield self.handler.get_group_profile(
-            group_id, requester_user_id
-        )
-
-        defer.returnValue((200, new_content))
-
-    @defer.inlineCallbacks
-    def on_POST(self, origin, content, query, group_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        new_content = yield self.handler.update_group_profile(
-            group_id, requester_user_id, content
-        )
-
-        defer.returnValue((200, new_content))
-
-
-class FederationGroupsSummaryServlet(BaseFederationServlet):
-    PATH = "/groups/(?P<group_id>[^/]*)/summary$"
-
-    @defer.inlineCallbacks
-    def on_GET(self, origin, content, query, group_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        new_content = yield self.handler.get_group_summary(
-            group_id, requester_user_id
-        )
-
-        defer.returnValue((200, new_content))
-
-
-class FederationGroupsRoomsServlet(BaseFederationServlet):
-    """Get the rooms in a group on behalf of a user
-    """
-    PATH = "/groups/(?P<group_id>[^/]*)/rooms$"
-
-    @defer.inlineCallbacks
-    def on_GET(self, origin, content, query, group_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        new_content = yield self.handler.get_rooms_in_group(
-            group_id, requester_user_id
-        )
-
-        defer.returnValue((200, new_content))
-
-
-class FederationGroupsAddRoomsServlet(BaseFederationServlet):
-    """Add/remove room from group
-    """
-    PATH = "/groups/(?P<group_id>[^/]*)/room/(?P<room_id>[^/]*)$"
-
-    @defer.inlineCallbacks
-    def on_POST(self, origin, content, query, group_id, room_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        new_content = yield self.handler.add_room_to_group(
-            group_id, requester_user_id, room_id, content
-        )
-
-        defer.returnValue((200, new_content))
-
-    @defer.inlineCallbacks
-    def on_DELETE(self, origin, content, query, group_id, room_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        new_content = yield self.handler.remove_room_from_group(
-            group_id, requester_user_id, room_id,
-        )
-
-        defer.returnValue((200, new_content))
-
-
-class FederationGroupsAddRoomsConfigServlet(BaseFederationServlet):
-    """Update room config in group
-    """
-    PATH = (
-        "/groups/(?P<group_id>[^/]*)/room/(?P<room_id>[^/]*)"
-        "/config/(?P<config_key>[^/]*)$"
-    )
-
-    @defer.inlineCallbacks
-    def on_POST(self, origin, content, query, group_id, room_id, config_key):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        result = yield self.groups_handler.update_room_in_group(
-            group_id, requester_user_id, room_id, config_key, content,
-        )
-
-        defer.returnValue((200, result))
-
-
-class FederationGroupsUsersServlet(BaseFederationServlet):
-    """Get the users in a group on behalf of a user
-    """
-    PATH = "/groups/(?P<group_id>[^/]*)/users$"
-
-    @defer.inlineCallbacks
-    def on_GET(self, origin, content, query, group_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        new_content = yield self.handler.get_users_in_group(
-            group_id, requester_user_id
-        )
-
-        defer.returnValue((200, new_content))
-
-
-class FederationGroupsInvitedUsersServlet(BaseFederationServlet):
-    """Get the users that have been invited to a group
-    """
-    PATH = "/groups/(?P<group_id>[^/]*)/invited_users$"
-
-    @defer.inlineCallbacks
-    def on_GET(self, origin, content, query, group_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        new_content = yield self.handler.get_invited_users_in_group(
-            group_id, requester_user_id
-        )
-
-        defer.returnValue((200, new_content))
-
-
-class FederationGroupsInviteServlet(BaseFederationServlet):
-    """Ask a group server to invite someone to the group
-    """
-    PATH = "/groups/(?P<group_id>[^/]*)/users/(?P<user_id>[^/]*)/invite$"
-
-    @defer.inlineCallbacks
-    def on_POST(self, origin, content, query, group_id, user_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        new_content = yield self.handler.invite_to_group(
-            group_id, user_id, requester_user_id, content,
-        )
-
-        defer.returnValue((200, new_content))
-
-
-class FederationGroupsAcceptInviteServlet(BaseFederationServlet):
-    """Accept an invitation from the group server
-    """
-    PATH = "/groups/(?P<group_id>[^/]*)/users/(?P<user_id>[^/]*)/accept_invite$"
-
-    @defer.inlineCallbacks
-    def on_POST(self, origin, content, query, group_id, user_id):
-        if get_domain_from_id(user_id) != origin:
-            raise SynapseError(403, "user_id doesn't match origin")
-
-        new_content = yield self.handler.accept_invite(
-            group_id, user_id, content,
-        )
-
-        defer.returnValue((200, new_content))
-
-
-class FederationGroupsRemoveUserServlet(BaseFederationServlet):
-    """Leave or kick a user from the group
-    """
-    PATH = "/groups/(?P<group_id>[^/]*)/users/(?P<user_id>[^/]*)/remove$"
-
-    @defer.inlineCallbacks
-    def on_POST(self, origin, content, query, group_id, user_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        new_content = yield self.handler.remove_user_from_group(
-            group_id, user_id, requester_user_id, content,
-        )
-
-        defer.returnValue((200, new_content))
-
-
-class FederationGroupsLocalInviteServlet(BaseFederationServlet):
-    """A group server has invited a local user
-    """
-    PATH = "/groups/local/(?P<group_id>[^/]*)/users/(?P<user_id>[^/]*)/invite$"
-
-    @defer.inlineCallbacks
-    def on_POST(self, origin, content, query, group_id, user_id):
-        if get_domain_from_id(group_id) != origin:
-            raise SynapseError(403, "group_id doesn't match origin")
-
-        new_content = yield self.handler.on_invite(
-            group_id, user_id, content,
-        )
-
-        defer.returnValue((200, new_content))
-
-
-class FederationGroupsRemoveLocalUserServlet(BaseFederationServlet):
-    """A group server has removed a local user
-    """
-    PATH = "/groups/local/(?P<group_id>[^/]*)/users/(?P<user_id>[^/]*)/remove$"
-
-    @defer.inlineCallbacks
-    def on_POST(self, origin, content, query, group_id, user_id):
-        if get_domain_from_id(group_id) != origin:
-            raise SynapseError(403, "user_id doesn't match origin")
-
-        new_content = yield self.handler.user_removed_from_group(
-            group_id, user_id, content,
-        )
-
-        defer.returnValue((200, new_content))
-
-
-class FederationGroupsRenewAttestaionServlet(BaseFederationServlet):
-    """A group or user's server renews their attestation
-    """
-    PATH = "/groups/(?P<group_id>[^/]*)/renew_attestation/(?P<user_id>[^/]*)$"
-
-    @defer.inlineCallbacks
-    def on_POST(self, origin, content, query, group_id, user_id):
-        # We don't need to check auth here as we check the attestation signatures
-
-        new_content = yield self.handler.on_renew_attestation(
-            group_id, user_id, content
-        )
-
-        defer.returnValue((200, new_content))
-
-
-class FederationGroupsSummaryRoomsServlet(BaseFederationServlet):
-    """Add/remove a room from the group summary, with optional category.
-
-    Matches both:
-        - /groups/:group/summary/rooms/:room_id
-        - /groups/:group/summary/categories/:category/rooms/:room_id
-    """
-    PATH = (
-        "/groups/(?P<group_id>[^/]*)/summary"
-        "(/categories/(?P<category_id>[^/]+))?"
-        "/rooms/(?P<room_id>[^/]*)$"
-    )
-
-    @defer.inlineCallbacks
-    def on_POST(self, origin, content, query, group_id, category_id, room_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        if category_id == "":
-            raise SynapseError(400, "category_id cannot be empty string")
-
-        resp = yield self.handler.update_group_summary_room(
-            group_id, requester_user_id,
-            room_id=room_id,
-            category_id=category_id,
-            content=content,
-        )
-
-        defer.returnValue((200, resp))
-
-    @defer.inlineCallbacks
-    def on_DELETE(self, origin, content, query, group_id, category_id, room_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        if category_id == "":
-            raise SynapseError(400, "category_id cannot be empty string")
-
-        resp = yield self.handler.delete_group_summary_room(
-            group_id, requester_user_id,
-            room_id=room_id,
-            category_id=category_id,
-        )
-
-        defer.returnValue((200, resp))
-
-
-class FederationGroupsCategoriesServlet(BaseFederationServlet):
-    """Get all categories for a group
-    """
-    PATH = (
-        "/groups/(?P<group_id>[^/]*)/categories/$"
-    )
-
-    @defer.inlineCallbacks
-    def on_GET(self, origin, content, query, group_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        resp = yield self.handler.get_group_categories(
-            group_id, requester_user_id,
-        )
-
-        defer.returnValue((200, resp))
-
-
-class FederationGroupsCategoryServlet(BaseFederationServlet):
-    """Add/remove/get a category in a group
-    """
-    PATH = (
-        "/groups/(?P<group_id>[^/]*)/categories/(?P<category_id>[^/]+)$"
-    )
-
-    @defer.inlineCallbacks
-    def on_GET(self, origin, content, query, group_id, category_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        resp = yield self.handler.get_group_category(
-            group_id, requester_user_id, category_id
-        )
-
-        defer.returnValue((200, resp))
-
-    @defer.inlineCallbacks
-    def on_POST(self, origin, content, query, group_id, category_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        if category_id == "":
-            raise SynapseError(400, "category_id cannot be empty string")
-
-        resp = yield self.handler.upsert_group_category(
-            group_id, requester_user_id, category_id, content,
-        )
-
-        defer.returnValue((200, resp))
-
-    @defer.inlineCallbacks
-    def on_DELETE(self, origin, content, query, group_id, category_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        if category_id == "":
-            raise SynapseError(400, "category_id cannot be empty string")
-
-        resp = yield self.handler.delete_group_category(
-            group_id, requester_user_id, category_id,
-        )
-
-        defer.returnValue((200, resp))
-
-
-class FederationGroupsRolesServlet(BaseFederationServlet):
-    """Get roles in a group
-    """
-    PATH = (
-        "/groups/(?P<group_id>[^/]*)/roles/$"
-    )
-
-    @defer.inlineCallbacks
-    def on_GET(self, origin, content, query, group_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        resp = yield self.handler.get_group_roles(
-            group_id, requester_user_id,
-        )
-
-        defer.returnValue((200, resp))
-
-
-class FederationGroupsRoleServlet(BaseFederationServlet):
-    """Add/remove/get a role in a group
-    """
-    PATH = (
-        "/groups/(?P<group_id>[^/]*)/roles/(?P<role_id>[^/]+)$"
-    )
-
-    @defer.inlineCallbacks
-    def on_GET(self, origin, content, query, group_id, role_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        resp = yield self.handler.get_group_role(
-            group_id, requester_user_id, role_id
-        )
-
-        defer.returnValue((200, resp))
-
-    @defer.inlineCallbacks
-    def on_POST(self, origin, content, query, group_id, role_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        if role_id == "":
-            raise SynapseError(400, "role_id cannot be empty string")
-
-        resp = yield self.handler.update_group_role(
-            group_id, requester_user_id, role_id, content,
-        )
-
-        defer.returnValue((200, resp))
-
-    @defer.inlineCallbacks
-    def on_DELETE(self, origin, content, query, group_id, role_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        if role_id == "":
-            raise SynapseError(400, "role_id cannot be empty string")
-
-        resp = yield self.handler.delete_group_role(
-            group_id, requester_user_id, role_id,
-        )
-
-        defer.returnValue((200, resp))
-
-
-class FederationGroupsSummaryUsersServlet(BaseFederationServlet):
-    """Add/remove a user from the group summary, with optional role.
-
-    Matches both:
-        - /groups/:group/summary/users/:user_id
-        - /groups/:group/summary/roles/:role/users/:user_id
-    """
-    PATH = (
-        "/groups/(?P<group_id>[^/]*)/summary"
-        "(/roles/(?P<role_id>[^/]+))?"
-        "/users/(?P<user_id>[^/]*)$"
-    )
-
-    @defer.inlineCallbacks
-    def on_POST(self, origin, content, query, group_id, role_id, user_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        if role_id == "":
-            raise SynapseError(400, "role_id cannot be empty string")
-
-        resp = yield self.handler.update_group_summary_user(
-            group_id, requester_user_id,
-            user_id=user_id,
-            role_id=role_id,
-            content=content,
-        )
-
-        defer.returnValue((200, resp))
-
-    @defer.inlineCallbacks
-    def on_DELETE(self, origin, content, query, group_id, role_id, user_id):
-        requester_user_id = parse_string_from_args(query, "requester_user_id")
-        if get_domain_from_id(requester_user_id) != origin:
-            raise SynapseError(403, "requester_user_id doesn't match origin")
-
-        if role_id == "":
-            raise SynapseError(400, "role_id cannot be empty string")
-
-        resp = yield self.handler.delete_group_summary_user(
-            group_id, requester_user_id,
-            user_id=user_id,
-            role_id=role_id,
-        )
-
-        defer.returnValue((200, resp))
-
-
-class FederationGroupsBulkPublicisedServlet(BaseFederationServlet):
-    """Get roles in a group
-    """
-    PATH = (
-        "/get_groups_publicised$"
-    )
-
-    @defer.inlineCallbacks
-    def on_POST(self, origin, content, query):
-        resp = yield self.handler.bulk_get_publicised_groups(
-            content["user_ids"], proxy=False,
-        )
-
-        defer.returnValue((200, resp))
-
-
-FEDERATION_SERVLET_CLASSES = (
+SERVLET_CLASSES = (
     FederationSendServlet,
     FederationPullServlet,
     FederationEventServlet,
@@ -1140,83 +625,17 @@ FEDERATION_SERVLET_CLASSES = (
     FederationThirdPartyInviteExchangeServlet,
     On3pidBindServlet,
     OpenIdUserInfo,
+    PublicRoomList,
     FederationVersionServlet,
 )
 
 
-ROOM_LIST_CLASSES = (
-    PublicRoomList,
-)
-
-GROUP_SERVER_SERVLET_CLASSES = (
-    FederationGroupsProfileServlet,
-    FederationGroupsSummaryServlet,
-    FederationGroupsRoomsServlet,
-    FederationGroupsUsersServlet,
-    FederationGroupsInvitedUsersServlet,
-    FederationGroupsInviteServlet,
-    FederationGroupsAcceptInviteServlet,
-    FederationGroupsRemoveUserServlet,
-    FederationGroupsSummaryRoomsServlet,
-    FederationGroupsCategoriesServlet,
-    FederationGroupsCategoryServlet,
-    FederationGroupsRolesServlet,
-    FederationGroupsRoleServlet,
-    FederationGroupsSummaryUsersServlet,
-    FederationGroupsAddRoomsServlet,
-    FederationGroupsAddRoomsConfigServlet,
-)
-
-
-GROUP_LOCAL_SERVLET_CLASSES = (
-    FederationGroupsLocalInviteServlet,
-    FederationGroupsRemoveLocalUserServlet,
-    FederationGroupsBulkPublicisedServlet,
-)
-
-
-GROUP_ATTESTATION_SERVLET_CLASSES = (
-    FederationGroupsRenewAttestaionServlet,
-)
-
-
 def register_servlets(hs, resource, authenticator, ratelimiter):
-    for servletclass in FEDERATION_SERVLET_CLASSES:
+    for servletclass in SERVLET_CLASSES:
         servletclass(
             handler=hs.get_replication_layer(),
             authenticator=authenticator,
             ratelimiter=ratelimiter,
             server_name=hs.hostname,
-        ).register(resource)
-
-    for servletclass in ROOM_LIST_CLASSES:
-        servletclass(
-            handler=hs.get_room_list_handler(),
-            authenticator=authenticator,
-            ratelimiter=ratelimiter,
-            server_name=hs.hostname,
-        ).register(resource)
-
-    for servletclass in GROUP_SERVER_SERVLET_CLASSES:
-        servletclass(
-            handler=hs.get_groups_server_handler(),
-            authenticator=authenticator,
-            ratelimiter=ratelimiter,
-            server_name=hs.hostname,
-        ).register(resource)
-
-    for servletclass in GROUP_LOCAL_SERVLET_CLASSES:
-        servletclass(
-            handler=hs.get_groups_local_handler(),
-            authenticator=authenticator,
-            ratelimiter=ratelimiter,
-            server_name=hs.hostname,
-        ).register(resource)
-
-    for servletclass in GROUP_ATTESTATION_SERVLET_CLASSES:
-        servletclass(
-            handler=hs.get_groups_attestation_renewer(),
-            authenticator=authenticator,
-            ratelimiter=ratelimiter,
-            server_name=hs.hostname,
+            room_list_handler=hs.get_room_list_handler(),
         ).register(resource)

synapse/groups/attestations.py

@@ -1,195 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2017 Vector Creations Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""Attestations ensure that users and groups can't lie about their memberships.
-
-When a user joins a group the HS and GS swap attestations, which allow them
-both to independently prove to third parties their membership.These
-attestations have a validity period so need to be periodically renewed.
-
-If a user leaves (or gets kicked out of) a group, either side can still use
-their attestation to "prove" their membership, until the attestation expires.
-Therefore attestations shouldn't be relied on to prove membership in important
-cases, but can for less important situtations, e.g. showing a users membership
-of groups on their profile, showing flairs, etc.abs
-
-An attestsation is a signed blob of json that looks like:
-
-    {
-        "user_id": "@foo:a.example.com",
-        "group_id": "+bar:b.example.com",
-        "valid_until_ms": 1507994728530,
-        "signatures":{"matrix.org":{"ed25519:auto":"..."}}
-    }
-"""
-
-import logging
-import random
-
-from twisted.internet import defer
-
-from synapse.api.errors import SynapseError
-from synapse.types import get_domain_from_id
-from synapse.util.logcontext import preserve_fn
-
-from signedjson.sign import sign_json
-
-
-logger = logging.getLogger(__name__)
-
-
-# Default validity duration for new attestations we create
-DEFAULT_ATTESTATION_LENGTH_MS = 3 * 24 * 60 * 60 * 1000
-
-# We add some jitter to the validity duration of attestations so that if we
-# add lots of users at once we don't need to renew them all at once.
-# The jitter is a multiplier picked randomly between the first and second number
-DEFAULT_ATTESTATION_JITTER = (0.9, 1.3)
-
-# Start trying to update our attestations when they come this close to expiring
-UPDATE_ATTESTATION_TIME_MS = 1 * 24 * 60 * 60 * 1000
-
-
-class GroupAttestationSigning(object):
-    """Creates and verifies group attestations.
-    """
-    def __init__(self, hs):
-        self.keyring = hs.get_keyring()
-        self.clock = hs.get_clock()
-        self.server_name = hs.hostname
-        self.signing_key = hs.config.signing_key[0]
-
-    @defer.inlineCallbacks
-    def verify_attestation(self, attestation, group_id, user_id, server_name=None):
-        """Verifies that the given attestation matches the given parameters.
-
-        An optional server_name can be supplied to explicitly set which server's
-        signature is expected. Otherwise assumes that either the group_id or user_id
-        is local and uses the other's server as the one to check.
-        """
-
-        if not server_name:
-            if get_domain_from_id(group_id) == self.server_name:
-                server_name = get_domain_from_id(user_id)
-            elif get_domain_from_id(user_id) == self.server_name:
-                server_name = get_domain_from_id(group_id)
-            else:
-                raise Exception("Expected either group_id or user_id to be local")
-
-        if user_id != attestation["user_id"]:
-            raise SynapseError(400, "Attestation has incorrect user_id")
-
-        if group_id != attestation["group_id"]:
-            raise SynapseError(400, "Attestation has incorrect group_id")
-        valid_until_ms = attestation["valid_until_ms"]
-
-        # TODO: We also want to check that *new* attestations that people give
-        # us to store are valid for at least a little while.
-        if valid_until_ms < self.clock.time_msec():
-            raise SynapseError(400, "Attestation expired")
-
-        yield self.keyring.verify_json_for_server(server_name, attestation)
-
-    def create_attestation(self, group_id, user_id):
-        """Create an attestation for the group_id and user_id with default
-        validity length.
-        """
-        validity_period = DEFAULT_ATTESTATION_LENGTH_MS
-        validity_period *= random.uniform(*DEFAULT_ATTESTATION_JITTER)
-        valid_until_ms = int(self.clock.time_msec() + validity_period)
-
-        return sign_json({
-            "group_id": group_id,
-            "user_id": user_id,
-            "valid_until_ms": valid_until_ms,
-        }, self.server_name, self.signing_key)
-
-
-class GroupAttestionRenewer(object):
-    """Responsible for sending and receiving attestation updates.
-    """
-
-    def __init__(self, hs):
-        self.clock = hs.get_clock()
-        self.store = hs.get_datastore()
-        self.assestations = hs.get_groups_attestation_signing()
-        self.transport_client = hs.get_federation_transport_client()
-        self.is_mine_id = hs.is_mine_id
-        self.attestations = hs.get_groups_attestation_signing()
-
-        self._renew_attestations_loop = self.clock.looping_call(
-            self._renew_attestations, 30 * 60 * 1000,
-        )
-
-    @defer.inlineCallbacks
-    def on_renew_attestation(self, group_id, user_id, content):
-        """When a remote updates an attestation
-        """
-        attestation = content["attestation"]
-
-        if not self.is_mine_id(group_id) and not self.is_mine_id(user_id):
-            raise SynapseError(400, "Neither user not group are on this server")
-
-        yield self.attestations.verify_attestation(
-            attestation,
-            user_id=user_id,
-            group_id=group_id,
-        )
-
-        yield self.store.update_remote_attestion(group_id, user_id, attestation)
-
-        defer.returnValue({})
-
-    @defer.inlineCallbacks
-    def _renew_attestations(self):
-        """Called periodically to check if we need to update any of our attestations
-        """
-
-        now = self.clock.time_msec()
-
-        rows = yield self.store.get_attestations_need_renewals(
-            now + UPDATE_ATTESTATION_TIME_MS
-        )
-
-        @defer.inlineCallbacks
-        def _renew_attestation(group_id, user_id):
-            if not self.is_mine_id(group_id):
-                destination = get_domain_from_id(group_id)
-            elif not self.is_mine_id(user_id):
-                destination = get_domain_from_id(user_id)
-            else:
-                logger.warn(
-                    "Incorrectly trying to do attestations for user: %r in %r",
-                    user_id, group_id,
-                )
-                yield self.store.remove_attestation_renewal(group_id, user_id)
-                return
-
-            attestation = self.attestations.create_attestation(group_id, user_id)
-
-            yield self.transport_client.renew_group_attestation(
-                destination, group_id, user_id,
-                content={"attestation": attestation},
-            )
-
-            yield self.store.update_attestation_renewal(
-                group_id, user_id, attestation
-            )
-
-        for row in rows:
-            group_id = row["group_id"]
-            user_id = row["user_id"]
-
-            preserve_fn(_renew_attestation)(group_id, user_id)

synapse/groups/groups_server.py

@@ -1,864 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2017 Vector Creations Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import logging
-
-from synapse.api.errors import SynapseError
-from synapse.types import GroupID, RoomID, UserID, get_domain_from_id
-from twisted.internet import defer
-
-logger = logging.getLogger(__name__)
-
-
-# TODO: Allow users to "knock" or simpkly join depending on rules
-# TODO: Federation admin APIs
-# TODO: is_priveged flag to users and is_public to users and rooms
-# TODO: Audit log for admins (profile updates, membership changes, users who tried
-#       to join but were rejected, etc)
-# TODO: Flairs
-
-
-class GroupsServerHandler(object):
-    def __init__(self, hs):
-        self.hs = hs
-        self.store = hs.get_datastore()
-        self.room_list_handler = hs.get_room_list_handler()
-        self.auth = hs.get_auth()
-        self.clock = hs.get_clock()
-        self.keyring = hs.get_keyring()
-        self.is_mine_id = hs.is_mine_id
-        self.signing_key = hs.config.signing_key[0]
-        self.server_name = hs.hostname
-        self.attestations = hs.get_groups_attestation_signing()
-        self.transport_client = hs.get_federation_transport_client()
-        self.profile_handler = hs.get_profile_handler()
-
-        # Ensure attestations get renewed
-        hs.get_groups_attestation_renewer()
-
-    @defer.inlineCallbacks
-    def check_group_is_ours(self, group_id, requester_user_id,
-                            and_exists=False, and_is_admin=None):
-        """Check that the group is ours, and optionally if it exists.
-
-        If group does exist then return group.
-
-        Args:
-            group_id (str)
-            and_exists (bool): whether to also check if group exists
-            and_is_admin (str): whether to also check if given str is a user_id
-                that is an admin
-        """
-        if not self.is_mine_id(group_id):
-            raise SynapseError(400, "Group not on this server")
-
-        group = yield self.store.get_group(group_id)
-        if and_exists and not group:
-            raise SynapseError(404, "Unknown group")
-
-        is_user_in_group = yield self.store.is_user_in_group(requester_user_id, group_id)
-        if group and not is_user_in_group and not group["is_public"]:
-            raise SynapseError(404, "Unknown group")
-
-        if and_is_admin:
-            is_admin = yield self.store.is_user_admin_in_group(group_id, and_is_admin)
-            if not is_admin:
-                raise SynapseError(403, "User is not admin in group")
-
-        defer.returnValue(group)
-
-    @defer.inlineCallbacks
-    def get_group_summary(self, group_id, requester_user_id):
-        """Get the summary for a group as seen by requester_user_id.
-
-        The group summary consists of the profile of the room, and a curated
-        list of users and rooms. These list *may* be organised by role/category.
-        The roles/categories are ordered, and so are the users/rooms within them.
-
-        A user/room may appear in multiple roles/categories.
-        """
-        yield self.check_group_is_ours(group_id, requester_user_id, and_exists=True)
-
-        is_user_in_group = yield self.store.is_user_in_group(requester_user_id, group_id)
-
-        profile = yield self.get_group_profile(group_id, requester_user_id)
-
-        users, roles = yield self.store.get_users_for_summary_by_role(
-            group_id, include_private=is_user_in_group,
-        )
-
-        # TODO: Add profiles to users
-
-        rooms, categories = yield self.store.get_rooms_for_summary_by_category(
-            group_id, include_private=is_user_in_group,
-        )
-
-        for room_entry in rooms:
-            room_id = room_entry["room_id"]
-            joined_users = yield self.store.get_users_in_room(room_id)
-            entry = yield self.room_list_handler.generate_room_entry(
-                room_id, len(joined_users),
-                with_alias=False, allow_private=True,
-            )
-            entry = dict(entry)  # so we don't change whats cached
-            entry.pop("room_id", None)
-
-            room_entry["profile"] = entry
-
-        rooms.sort(key=lambda e: e.get("order", 0))
-
-        for entry in users:
-            user_id = entry["user_id"]
-
-            if not self.is_mine_id(requester_user_id):
-                attestation = yield self.store.get_remote_attestation(group_id, user_id)
-                if not attestation:
-                    continue
-
-                entry["attestation"] = attestation
-            else:
-                entry["attestation"] = self.attestations.create_attestation(
-                    group_id, user_id,
-                )
-
-            user_profile = yield self.profile_handler.get_profile_from_cache(user_id)
-            entry.update(user_profile)
-
-        users.sort(key=lambda e: e.get("order", 0))
-
-        membership_info = yield self.store.get_users_membership_info_in_group(
-            group_id, requester_user_id,
-        )
-
-        defer.returnValue({
-            "profile": profile,
-            "users_section": {
-                "users": users,
-                "roles": roles,
-                "total_user_count_estimate": 0,  # TODO
-            },
-            "rooms_section": {
-                "rooms": rooms,
-                "categories": categories,
-                "total_room_count_estimate": 0,  # TODO
-            },
-            "user": membership_info,
-        })
-
-    @defer.inlineCallbacks
-    def update_group_summary_room(self, group_id, requester_user_id,
-                                  room_id, category_id, content):
-        """Add/update a room to the group summary
-        """
-        yield self.check_group_is_ours(
-            group_id,
-            requester_user_id,
-            and_exists=True,
-            and_is_admin=requester_user_id,
-        )
-
-        RoomID.from_string(room_id)  # Ensure valid room id
-
-        order = content.get("order", None)
-
-        is_public = _parse_visibility_from_contents(content)
-
-        yield self.store.add_room_to_summary(
-            group_id=group_id,
-            room_id=room_id,
-            category_id=category_id,
-            order=order,
-            is_public=is_public,
-        )
-
-        defer.returnValue({})
-
-    @defer.inlineCallbacks
-    def delete_group_summary_room(self, group_id, requester_user_id,
-                                  room_id, category_id):
-        """Remove a room from the summary
-        """
-        yield self.check_group_is_ours(
-            group_id,
-            requester_user_id,
-            and_exists=True,
-            and_is_admin=requester_user_id,
-        )
-
-        yield self.store.remove_room_from_summary(
-            group_id=group_id,
-            room_id=room_id,
-            category_id=category_id,
-        )
-
-        defer.returnValue({})
-
-    @defer.inlineCallbacks
-    def get_group_categories(self, group_id, requester_user_id):
-        """Get all categories in a group (as seen by user)
-        """
-        yield self.check_group_is_ours(group_id, requester_user_id, and_exists=True)
-
-        categories = yield self.store.get_group_categories(
-            group_id=group_id,
-        )
-        defer.returnValue({"categories": categories})
-
-    @defer.inlineCallbacks
-    def get_group_category(self, group_id, requester_user_id, category_id):
-        """Get a specific category in a group (as seen by user)
-        """
-        yield self.check_group_is_ours(group_id, requester_user_id, and_exists=True)
-
-        res = yield self.store.get_group_category(
-            group_id=group_id,
-            category_id=category_id,
-        )
-
-        defer.returnValue(res)
-
-    @defer.inlineCallbacks
-    def update_group_category(self, group_id, requester_user_id, category_id, content):
-        """Add/Update a group category
-        """
-        yield self.check_group_is_ours(
-            group_id,
-            requester_user_id,
-            and_exists=True,
-            and_is_admin=requester_user_id,
-        )
-
-        is_public = _parse_visibility_from_contents(content)
-        profile = content.get("profile")
-
-        yield self.store.upsert_group_category(
-            group_id=group_id,
-            category_id=category_id,
-            is_public=is_public,
-            profile=profile,
-        )
-
-        defer.returnValue({})
-
-    @defer.inlineCallbacks
-    def delete_group_category(self, group_id, requester_user_id, category_id):
-        """Delete a group category
-        """
-        yield self.check_group_is_ours(
-            group_id,
-            requester_user_id,
-            and_exists=True,
-            and_is_admin=requester_user_id
-        )
-
-        yield self.store.remove_group_category(
-            group_id=group_id,
-            category_id=category_id,
-        )
-
-        defer.returnValue({})
-
-    @defer.inlineCallbacks
-    def get_group_roles(self, group_id, requester_user_id):
-        """Get all roles in a group (as seen by user)
-        """
-        yield self.check_group_is_ours(group_id, requester_user_id, and_exists=True)
-
-        roles = yield self.store.get_group_roles(
-            group_id=group_id,
-        )
-        defer.returnValue({"roles": roles})
-
-    @defer.inlineCallbacks
-    def get_group_role(self, group_id, requester_user_id, role_id):
-        """Get a specific role in a group (as seen by user)
-        """
-        yield self.check_group_is_ours(group_id, requester_user_id, and_exists=True)
-
-        res = yield self.store.get_group_role(
-            group_id=group_id,
-            role_id=role_id,
-        )
-        defer.returnValue(res)
-
-    @defer.inlineCallbacks
-    def update_group_role(self, group_id, requester_user_id, role_id, content):
-        """Add/update a role in a group
-        """
-        yield self.check_group_is_ours(
-            group_id,
-            requester_user_id,
-            and_exists=True,
-            and_is_admin=requester_user_id,
-        )
-
-        is_public = _parse_visibility_from_contents(content)
-
-        profile = content.get("profile")
-
-        yield self.store.upsert_group_role(
-            group_id=group_id,
-            role_id=role_id,
-            is_public=is_public,
-            profile=profile,
-        )
-
-        defer.returnValue({})
-
-    @defer.inlineCallbacks
-    def delete_group_role(self, group_id, requester_user_id, role_id):
-        """Remove role from group
-        """
-        yield self.check_group_is_ours(
-            group_id,
-            requester_user_id,
-            and_exists=True,
-            and_is_admin=requester_user_id,
-        )
-
-        yield self.store.remove_group_role(
-            group_id=group_id,
-            role_id=role_id,
-        )
-
-        defer.returnValue({})
-
-    @defer.inlineCallbacks
-    def update_group_summary_user(self, group_id, requester_user_id, user_id, role_id,
-                                  content):
-        """Add/update a users entry in the group summary
-        """
-        yield self.check_group_is_ours(
-            group_id, requester_user_id, and_exists=True, and_is_admin=requester_user_id,
-        )
-
-        order = content.get("order", None)
-
-        is_public = _parse_visibility_from_contents(content)
-
-        yield self.store.add_user_to_summary(
-            group_id=group_id,
-            user_id=user_id,
-            role_id=role_id,
-            order=order,
-            is_public=is_public,
-        )
-
-        defer.returnValue({})
-
-    @defer.inlineCallbacks
-    def delete_group_summary_user(self, group_id, requester_user_id, user_id, role_id):
-        """Remove a user from the group summary
-        """
-        yield self.check_group_is_ours(
-            group_id, requester_user_id, and_exists=True, and_is_admin=requester_user_id,
-        )
-
-        yield self.store.remove_user_from_summary(
-            group_id=group_id,
-            user_id=user_id,
-            role_id=role_id,
-        )
-
-        defer.returnValue({})
-
-    @defer.inlineCallbacks
-    def get_group_profile(self, group_id, requester_user_id):
-        """Get the group profile as seen by requester_user_id
-        """
-
-        yield self.check_group_is_ours(group_id, requester_user_id)
-
-        group_description = yield self.store.get_group(group_id)
-
-        if group_description:
-            defer.returnValue(group_description)
-        else:
-            raise SynapseError(404, "Unknown group")
-
-    @defer.inlineCallbacks
-    def update_group_profile(self, group_id, requester_user_id, content):
-        """Update the group profile
-        """
-        yield self.check_group_is_ours(
-            group_id, requester_user_id, and_exists=True, and_is_admin=requester_user_id,
-        )
-
-        profile = {}
-        for keyname in ("name", "avatar_url", "short_description",
-                        "long_description"):
-            if keyname in content:
-                value = content[keyname]
-                if not isinstance(value, basestring):
-                    raise SynapseError(400, "%r value is not a string" % (keyname,))
-                profile[keyname] = value
-
-        yield self.store.update_group_profile(group_id, profile)
-
-    @defer.inlineCallbacks
-    def get_users_in_group(self, group_id, requester_user_id):
-        """Get the users in group as seen by requester_user_id.
-
-        The ordering is arbitrary at the moment
-        """
-
-        yield self.check_group_is_ours(group_id, requester_user_id, and_exists=True)
-
-        is_user_in_group = yield self.store.is_user_in_group(requester_user_id, group_id)
-
-        user_results = yield self.store.get_users_in_group(
-            group_id, include_private=is_user_in_group,
-        )
-
-        chunk = []
-        for user_result in user_results:
-            g_user_id = user_result["user_id"]
-            is_public = user_result["is_public"]
-            is_privileged = user_result["is_admin"]
-
-            entry = {"user_id": g_user_id}
-
-            profile = yield self.profile_handler.get_profile_from_cache(g_user_id)
-            entry.update(profile)
-
-            entry["is_public"] = bool(is_public)
-            entry["is_privileged"] = bool(is_privileged)
-
-            if not self.is_mine_id(g_user_id):
-                attestation = yield self.store.get_remote_attestation(group_id, g_user_id)
-                if not attestation:
-                    continue
-
-                entry["attestation"] = attestation
-            else:
-                entry["attestation"] = self.attestations.create_attestation(
-                    group_id, g_user_id,
-                )
-
-            chunk.append(entry)
-
-        # TODO: If admin add lists of users whose attestations have timed out
-
-        defer.returnValue({
-            "chunk": chunk,
-            "total_user_count_estimate": len(user_results),
-        })
-
-    @defer.inlineCallbacks
-    def get_invited_users_in_group(self, group_id, requester_user_id):
-        """Get the users that have been invited to a group as seen by requester_user_id.
-
-        The ordering is arbitrary at the moment
-        """
-
-        yield self.check_group_is_ours(group_id, requester_user_id, and_exists=True)
-
-        is_user_in_group = yield self.store.is_user_in_group(requester_user_id, group_id)
-
-        if not is_user_in_group:
-            raise SynapseError(403, "User not in group")
-
-        invited_users = yield self.store.get_invited_users_in_group(group_id)
-
-        user_profiles = []
-
-        for user_id in invited_users:
-            user_profile = {
-                "user_id": user_id
-            }
-            try:
-                profile = yield self.profile_handler.get_profile_from_cache(user_id)
-                user_profile.update(profile)
-            except Exception as e:
-                logger.warn("Error getting profile for %s: %s", user_id, e)
-            user_profiles.append(user_profile)
-
-        defer.returnValue({
-            "chunk": user_profiles,
-            "total_user_count_estimate": len(invited_users),
-        })
-
-    @defer.inlineCallbacks
-    def get_rooms_in_group(self, group_id, requester_user_id):
-        """Get the rooms in group as seen by requester_user_id
-
-        This returns rooms in order of decreasing number of joined users
-        """
-
-        yield self.check_group_is_ours(group_id, requester_user_id, and_exists=True)
-
-        is_user_in_group = yield self.store.is_user_in_group(requester_user_id, group_id)
-
-        room_results = yield self.store.get_rooms_in_group(
-            group_id, include_private=is_user_in_group,
-        )
-
-        chunk = []
-        for room_result in room_results:
-            room_id = room_result["room_id"]
-
-            joined_users = yield self.store.get_users_in_room(room_id)
-            entry = yield self.room_list_handler.generate_room_entry(
-                room_id, len(joined_users),
-                with_alias=False, allow_private=True,
-            )
-
-            if not entry:
-                continue
-
-            entry["is_public"] = bool(room_result["is_public"])
-
-            chunk.append(entry)
-
-        chunk.sort(key=lambda e: -e["num_joined_members"])
-
-        defer.returnValue({
-            "chunk": chunk,
-            "total_room_count_estimate": len(room_results),
-        })
-
-    @defer.inlineCallbacks
-    def add_room_to_group(self, group_id, requester_user_id, room_id, content):
-        """Add room to group
-        """
-        RoomID.from_string(room_id)  # Ensure valid room id
-
-        yield self.check_group_is_ours(
-            group_id, requester_user_id, and_exists=True, and_is_admin=requester_user_id
-        )
-
-        is_public = _parse_visibility_from_contents(content)
-
-        yield self.store.add_room_to_group(group_id, room_id, is_public=is_public)
-
-        defer.returnValue({})
-
-    @defer.inlineCallbacks
-    def update_room_in_group(self, group_id, requester_user_id, room_id, config_key,
-                             content):
-        """Update room in group
-        """
-        RoomID.from_string(room_id)  # Ensure valid room id
-
-        yield self.check_group_is_ours(
-            group_id, requester_user_id, and_exists=True, and_is_admin=requester_user_id
-        )
-
-        if config_key == "m.visibility":
-            is_public = _parse_visibility_dict(content)
-
-            yield self.store.update_room_in_group_visibility(
-                group_id, room_id,
-                is_public=is_public,
-            )
-        else:
-            raise SynapseError(400, "Uknown config option")
-
-        defer.returnValue({})
-
-    @defer.inlineCallbacks
-    def remove_room_from_group(self, group_id, requester_user_id, room_id):
-        """Remove room from group
-        """
-        yield self.check_group_is_ours(
-            group_id, requester_user_id, and_exists=True, and_is_admin=requester_user_id
-        )
-
-        yield self.store.remove_room_from_group(group_id, room_id)
-
-        defer.returnValue({})
-
-    @defer.inlineCallbacks
-    def invite_to_group(self, group_id, user_id, requester_user_id, content):
-        """Invite user to group
-        """
-
-        group = yield self.check_group_is_ours(
-            group_id, requester_user_id, and_exists=True, and_is_admin=requester_user_id
-        )
-
-        # TODO: Check if user knocked
-        # TODO: Check if user is already invited
-
-        content = {
-            "profile": {
-                "name": group["name"],
-                "avatar_url": group["avatar_url"],
-            },
-            "inviter": requester_user_id,
-        }
-
-        if self.hs.is_mine_id(user_id):
-            groups_local = self.hs.get_groups_local_handler()
-            res = yield groups_local.on_invite(group_id, user_id, content)
-            local_attestation = None
-        else:
-            local_attestation = self.attestations.create_attestation(group_id, user_id)
-            content.update({
-                "attestation": local_attestation,
-            })
-
-            res = yield self.transport_client.invite_to_group_notification(
-                get_domain_from_id(user_id), group_id, user_id, content
-            )
-
-            user_profile = res.get("user_profile", {})
-            yield self.store.add_remote_profile_cache(
-                user_id,
-                displayname=user_profile.get("displayname"),
-                avatar_url=user_profile.get("avatar_url"),
-            )
-
-        if res["state"] == "join":
-            if not self.hs.is_mine_id(user_id):
-                remote_attestation = res["attestation"]
-
-                yield self.attestations.verify_attestation(
-                    remote_attestation,
-                    user_id=user_id,
-                    group_id=group_id,
-                )
-            else:
-                remote_attestation = None
-
-            yield self.store.add_user_to_group(
-                group_id, user_id,
-                is_admin=False,
-                is_public=False,  # TODO
-                local_attestation=local_attestation,
-                remote_attestation=remote_attestation,
-            )
-        elif res["state"] == "invite":
-            yield self.store.add_group_invite(
-                group_id, user_id,
-            )
-            defer.returnValue({
-                "state": "invite"
-            })
-        elif res["state"] == "reject":
-            defer.returnValue({
-                "state": "reject"
-            })
-        else:
-            raise SynapseError(502, "Unknown state returned by HS")
-
-    @defer.inlineCallbacks
-    def accept_invite(self, group_id, requester_user_id, content):
-        """User tries to accept an invite to the group.
-
-        This is different from them asking to join, and so should error if no
-        invite exists (and they're not a member of the group)
-        """
-
-        yield self.check_group_is_ours(group_id, requester_user_id, and_exists=True)
-
-        is_invited = yield self.store.is_user_invited_to_local_group(
-            group_id, requester_user_id,
-        )
-        if not is_invited:
-            raise SynapseError(403, "User not invited to group")
-
-        if not self.hs.is_mine_id(requester_user_id):
-            local_attestation = self.attestations.create_attestation(
-                group_id, requester_user_id,
-            )
-            remote_attestation = content["attestation"]
-
-            yield self.attestations.verify_attestation(
-                remote_attestation,
-                user_id=requester_user_id,
-                group_id=group_id,
-            )
-        else:
-            local_attestation = None
-            remote_attestation = None
-
-        is_public = _parse_visibility_from_contents(content)
-
-        yield self.store.add_user_to_group(
-            group_id, requester_user_id,
-            is_admin=False,
-            is_public=is_public,
-            local_attestation=local_attestation,
-            remote_attestation=remote_attestation,
-        )
-
-        defer.returnValue({
-            "state": "join",
-            "attestation": local_attestation,
-        })
-
-    @defer.inlineCallbacks
-    def knock(self, group_id, requester_user_id, content):
-        """A user requests becoming a member of the group
-        """
-        yield self.check_group_is_ours(group_id, requester_user_id, and_exists=True)
-
-        raise NotImplementedError()
-
-    @defer.inlineCallbacks
-    def accept_knock(self, group_id, requester_user_id, content):
-        """Accept a users knock to the room.
-
-        Errors if the user hasn't knocked, rather than inviting them.
-        """
-
-        yield self.check_group_is_ours(group_id, requester_user_id, and_exists=True)
-
-        raise NotImplementedError()
-
-    @defer.inlineCallbacks
-    def remove_user_from_group(self, group_id, user_id, requester_user_id, content):
-        """Remove a user from the group; either a user is leaving or an admin
-        kicked them.
-        """
-
-        yield self.check_group_is_ours(group_id, requester_user_id, and_exists=True)
-
-        is_kick = False
-        if requester_user_id != user_id:
-            is_admin = yield self.store.is_user_admin_in_group(
-                group_id, requester_user_id
-            )
-            if not is_admin:
-                raise SynapseError(403, "User is not admin in group")
-
-            is_kick = True
-
-        yield self.store.remove_user_from_group(
-            group_id, user_id,
-        )
-
-        if is_kick:
-            if self.hs.is_mine_id(user_id):
-                groups_local = self.hs.get_groups_local_handler()
-                yield groups_local.user_removed_from_group(group_id, user_id, {})
-            else:
-                yield self.transport_client.remove_user_from_group_notification(
-                    get_domain_from_id(user_id), group_id, user_id, {}
-                )
-
-        if not self.hs.is_mine_id(user_id):
-            yield self.store.maybe_delete_remote_profile_cache(user_id)
-
-        defer.returnValue({})
-
-    @defer.inlineCallbacks
-    def create_group(self, group_id, requester_user_id, content):
-        group = yield self.check_group_is_ours(group_id, requester_user_id)
-
-        logger.info("Attempting to create group with ID: %r", group_id)
-
-        # parsing the id into a GroupID validates it.
-        group_id_obj = GroupID.from_string(group_id)
-
-        if group:
-            raise SynapseError(400, "Group already exists")
-
-        is_admin = yield self.auth.is_server_admin(UserID.from_string(requester_user_id))
-        if not is_admin:
-            if not self.hs.config.enable_group_creation:
-                raise SynapseError(
-                    403, "Only a server admin can create groups on this server",
-                )
-            localpart = group_id_obj.localpart
-            if not localpart.startswith(self.hs.config.group_creation_prefix):
-                raise SynapseError(
-                    400,
-                    "Can only create groups with prefix %r on this server" % (
-                        self.hs.config.group_creation_prefix,
-                    ),
-                )
-
-        profile = content.get("profile", {})
-        name = profile.get("name")
-        avatar_url = profile.get("avatar_url")
-        short_description = profile.get("short_description")
-        long_description = profile.get("long_description")
-        user_profile = content.get("user_profile", {})
-
-        yield self.store.create_group(
-            group_id,
-            requester_user_id,
-            name=name,
-            avatar_url=avatar_url,
-            short_description=short_description,
-            long_description=long_description,
-        )
-
-        if not self.hs.is_mine_id(requester_user_id):
-            remote_attestation = content["attestation"]
-
-            yield self.attestations.verify_attestation(
-                remote_attestation,
-                user_id=requester_user_id,
-                group_id=group_id,
-            )
-
-            local_attestation = self.attestations.create_attestation(
-                group_id,
-                requester_user_id,
-            )
-        else:
-            local_attestation = None
-            remote_attestation = None
-
-        yield self.store.add_user_to_group(
-            group_id, requester_user_id,
-            is_admin=True,
-            is_public=True,  # TODO
-            local_attestation=local_attestation,
-            remote_attestation=remote_attestation,
-        )
-
-        if not self.hs.is_mine_id(requester_user_id):
-            yield self.store.add_remote_profile_cache(
-                requester_user_id,
-                displayname=user_profile.get("displayname"),
-                avatar_url=user_profile.get("avatar_url"),
-            )
-
-        defer.returnValue({
-            "group_id": group_id,
-        })
-
-
-def _parse_visibility_from_contents(content):
-    """Given a content for a request parse out whether the entity should be
-    public or not
-    """
-
-    visibility = content.get("m.visibility")
-    if visibility:
-        return _parse_visibility_dict(visibility)
-    else:
-        is_public = True
-
-    return is_public
-
-
-def _parse_visibility_dict(visibility):
-    """Given a dict for the "m.visibility" config return if the entity should
-    be public or not
-    """
-    vis_type = visibility.get("type")
-    if not vis_type:
-        return True
-
-    if vis_type not in ("public", "private"):
-        raise SynapseError(
-            400, "Synapse only supports 'public'/'private' visibility"
-        )
-    return vis_type == "public"

synapse/handlers/__init__.py

@@ -20,6 +20,7 @@ from .room import (
 from .room_member import RoomMemberHandler
 from .message import MessageHandler
 from .federation import FederationHandler
+from .profile import ProfileHandler
 from .directory import DirectoryHandler
 from .admin import AdminHandler
 from .identity import IdentityHandler
@@ -51,6 +52,7 @@ class Handlers(object):
         self.room_creation_handler = RoomCreationHandler(hs)
         self.room_member_handler = RoomMemberHandler(hs)
         self.federation_handler = FederationHandler(hs)
+        self.profile_handler = ProfileHandler(hs)
         self.directory_handler = DirectoryHandler(hs)
         self.admin_handler = AdminHandler(hs)
         self.identity_handler = IdentityHandler(hs)

synapse/handlers/_base.py

@@ -53,20 +53,7 @@ class BaseHandler(object):
 
         self.event_builder_factory = hs.get_event_builder_factory()
 
-    @defer.inlineCallbacks
-    def ratelimit(self, requester, update=True):
-        """Ratelimits requests.
-
-        Args:
-            requester (Requester)
-            update (bool): Whether to record that a request is being processed.
-                Set to False when doing multiple checks for one request (e.g.
-                to check up front if we would reject the request), and set to
-                True for the last call for a given request.
-
-        Raises:
-            LimitExceededError if the request should be ratelimited
-        """
+    def ratelimit(self, requester):
         time_now = self.clock.time()
         user_id = requester.user.to_string()
 
@@ -80,25 +67,10 @@ class BaseHandler(object):
         if requester.app_service and not requester.app_service.is_rate_limited():
             return
 
-        # Check if there is a per user override in the DB.
-        override = yield self.store.get_ratelimit_for_user(user_id)
-        if override:
-            # If overriden with a null Hz then ratelimiting has been entirely
-            # disabled for the user
-            if not override.messages_per_second:
-                return
-
-            messages_per_second = override.messages_per_second
-            burst_count = override.burst_count
-        else:
-            messages_per_second = self.hs.config.rc_messages_per_second
-            burst_count = self.hs.config.rc_message_burst_count
-
         allowed, time_allowed = self.ratelimiter.send_message(
             user_id, time_now,
-            msg_rate_hz=messages_per_second,
-            burst_count=burst_count,
-            update=update,
+            msg_rate_hz=self.hs.config.rc_messages_per_second,
+            burst_count=self.hs.config.rc_message_burst_count,
         )
         if not allowed:
             raise LimitExceededError(

synapse/handlers/appservice.py

@@ -70,10 +70,11 @@ class ApplicationServicesHandler(object):
         with Measure(self.clock, "notify_interested_services"):
             self.is_processing = True
             try:
+                upper_bound = self.current_max
                 limit = 100
                 while True:
                     upper_bound, events = yield self.store.get_new_events_for_appservice(
-                        self.current_max, limit
+                        upper_bound, limit
                     )
 
                     if not events:
@@ -104,6 +105,9 @@ class ApplicationServicesHandler(object):
                             )
 
                     yield self.store.set_appservice_last_pos(upper_bound)
+
+                    if len(events) < limit:
+                        break
             finally:
                 self.is_processing = False
 

synapse/handlers/auth.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014 - 2016 OpenMarket Ltd
-# Copyright 2017 Vector Creations Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -13,15 +12,14 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
 from twisted.internet import defer
 
 from ._base import BaseHandler
 from synapse.api.constants import LoginType
-from synapse.api.errors import AuthError, LoginError, Codes, StoreError, SynapseError
-from synapse.module_api import ModuleApi
 from synapse.types import UserID
+from synapse.api.errors import AuthError, LoginError, Codes, StoreError, SynapseError
 from synapse.util.async import run_on_reactor
-from synapse.util.caches.expiringcache import ExpiringCache
 
 from twisted.web.client import PartialDownloadError
 
@@ -49,21 +47,15 @@ class AuthHandler(BaseHandler):
             LoginType.PASSWORD: self._check_password_auth,
             LoginType.RECAPTCHA: self._check_recaptcha,
             LoginType.EMAIL_IDENTITY: self._check_email_identity,
-            LoginType.MSISDN: self._check_msisdn,
             LoginType.DUMMY: self._check_dummy_auth,
         }
         self.bcrypt_rounds = hs.config.bcrypt_rounds
+        self.sessions = {}
 
-        # This is not a cache per se, but a store of all current sessions that
-        # expire after N hours
-        self.sessions = ExpiringCache(
-            cache_name="register_sessions",
-            clock=hs.get_clock(),
-            expiry_ms=self.SESSION_EXPIRE_MS,
-            reset_expiry_on_get=True,
+        account_handler = _AccountHandler(
+            hs, check_user_exists=self.check_user_exists
         )
 
-        account_handler = ModuleApi(hs, self)
         self.password_providers = [
             module(config=config, account_handler=account_handler)
             for module, config in hs.config.password_providers
@@ -72,24 +64,14 @@ class AuthHandler(BaseHandler):
         logger.info("Extra password_providers: %r", self.password_providers)
 
         self.hs = hs  # FIXME better possibility to access registrationHandler later?
+        self.device_handler = hs.get_device_handler()
         self.macaroon_gen = hs.get_macaroon_generator()
-        self._password_enabled = hs.config.password_enabled
-
-        login_types = set()
-        if self._password_enabled:
-            login_types.add(LoginType.PASSWORD)
-        for provider in self.password_providers:
-            if hasattr(provider, "get_supported_login_types"):
-                login_types.update(
-                    provider.get_supported_login_types().keys()
-                )
-        self._supported_login_types = frozenset(login_types)
 
     @defer.inlineCallbacks
     def check_auth(self, flows, clientdict, clientip):
         """
         Takes a dictionary sent by the client in the login / registration
-        protocol and handles the User-Interactive Auth flow.
+        protocol and handles the login flow.
 
         As a side effect, this function fills in the 'creds' key on the user's
         session with a map, which maps each auth-type (str) to the relevant
@@ -267,19 +249,16 @@ class AuthHandler(BaseHandler):
         sess = self._get_session_info(session_id)
         return sess.setdefault('serverdict', {}).get(key, default)
 
-    @defer.inlineCallbacks
     def _check_password_auth(self, authdict, _):
         if "user" not in authdict or "password" not in authdict:
             raise LoginError(400, "", Codes.MISSING_PARAM)
 
         user_id = authdict["user"]
         password = authdict["password"]
+        if not user_id.startswith('@'):
+            user_id = UserID.create(user_id, self.hs.hostname).to_string()
 
-        (canonical_id, callback) = yield self.validate_login(user_id, {
-            "type": LoginType.PASSWORD,
-            "password": password,
-        })
-        defer.returnValue(canonical_id)
+        return self._check_password(user_id, password)
 
     @defer.inlineCallbacks
     def _check_recaptcha(self, authdict, clientip):
@@ -328,47 +307,31 @@ class AuthHandler(BaseHandler):
                 defer.returnValue(True)
         raise LoginError(401, "", errcode=Codes.UNAUTHORIZED)
 
+    @defer.inlineCallbacks
     def _check_email_identity(self, authdict, _):
-        return self._check_threepid('email', authdict)
-
-    def _check_msisdn(self, authdict, _):
-        return self._check_threepid('msisdn', authdict)
-
-    @defer.inlineCallbacks
-    def _check_dummy_auth(self, authdict, _):
-        yield run_on_reactor()
-        defer.returnValue(True)
-
-    @defer.inlineCallbacks
-    def _check_threepid(self, medium, authdict):
         yield run_on_reactor()
 
         if 'threepid_creds' not in authdict:
             raise LoginError(400, "Missing threepid_creds", Codes.MISSING_PARAM)
 
         threepid_creds = authdict['threepid_creds']
-
         identity_handler = self.hs.get_handlers().identity_handler
 
-        logger.info("Getting validated threepid. threepidcreds: %r", (threepid_creds,))
+        logger.info("Getting validated threepid. threepidcreds: %r" % (threepid_creds,))
         threepid = yield identity_handler.threepid_from_creds(threepid_creds)
 
         if not threepid:
             raise LoginError(401, "", errcode=Codes.UNAUTHORIZED)
 
-        if threepid['medium'] != medium:
-            raise LoginError(
-                401,
-                "Expecting threepid of type '%s', got '%s'" % (
-                    medium, threepid['medium'],
-                ),
-                errcode=Codes.UNAUTHORIZED
-            )
-
         threepid['threepid_creds'] = authdict['threepid_creds']
 
         defer.returnValue(threepid)
 
+    @defer.inlineCallbacks
+    def _check_dummy_auth(self, authdict, _):
+        yield run_on_reactor()
+        defer.returnValue(True)
+
     def _get_params_recaptcha(self):
         return {"public_key": self.hs.config.recaptcha_public_key}
 
@@ -408,8 +371,26 @@ class AuthHandler(BaseHandler):
 
         return self.sessions[session_id]
 
+    def validate_password_login(self, user_id, password):
+        """
+        Authenticates the user with their username and password.
+
+        Used only by the v1 login API.
+
+        Args:
+            user_id (str): complete @user:id
+            password (str): Password
+        Returns:
+            defer.Deferred: (str) canonical user id
+        Raises:
+            StoreError if there was a problem accessing the database
+            LoginError if there was an authentication problem.
+        """
+        return self._check_password(user_id, password)
+
     @defer.inlineCallbacks
-    def get_access_token_for_user_id(self, user_id, device_id=None):
+    def get_access_token_for_user_id(self, user_id, device_id=None,
+                                     initial_display_name=None):
         """
         Creates a new access token for the user with the given user ID.
 
@@ -423,10 +404,13 @@ class AuthHandler(BaseHandler):
             device_id (str|None): the device ID to associate with the tokens.
                None to leave the tokens unassociated with a device (deprecated:
                we should always have a device ID)
+            initial_display_name (str): display name to associate with the
+               device if it needs re-registering
         Returns:
               The access token for the user's session.
         Raises:
             StoreError if there was a problem storing the token.
+            LoginError if there was an authentication problem.
         """
         logger.info("Logging in user %s on device %s", user_id, device_id)
         access_token = yield self.issue_access_token(user_id, device_id)
@@ -436,11 +420,9 @@ class AuthHandler(BaseHandler):
         # really don't want is active access_tokens without a record of the
         # device, so we double-check it here.
         if device_id is not None:
-            try:
-                yield self.store.get_device(user_id, device_id)
-            except StoreError:
-                yield self.store.delete_access_token(access_token)
-                raise StoreError(400, "Login raced against device deletion")
+            yield self.device_handler.check_device_registered(
+                user_id, device_id, initial_display_name
+            )
 
         defer.returnValue(access_token)
 
@@ -492,115 +474,29 @@ class AuthHandler(BaseHandler):
             )
         defer.returnValue(result)
 
-    def get_supported_login_types(self):
-        """Get a the login types supported for the /login API
-
-        By default this is just 'm.login.password' (unless password_enabled is
-        False in the config file), but password auth providers can provide
-        other login types.
-
-        Returns:
-            Iterable[str]: login types
-        """
-        return self._supported_login_types
-
     @defer.inlineCallbacks
-    def validate_login(self, username, login_submission):
-        """Authenticates the user for the /login API
+    def _check_password(self, user_id, password):
+        """Authenticate a user against the LDAP and local databases.
 
-        Also used by the user-interactive auth flow to validate
-        m.login.password auth types.
+        user_id is checked case insensitively against the local database, but
+        will throw if there are multiple inexact matches.
 
         Args:
-            username (str): username supplied by the user
-            login_submission (dict): the whole of the login submission
-                (including 'type' and other relevant fields)
+            user_id (str): complete @user:id
         Returns:
-            Deferred[str, func]: canonical user id, and optional callback
-                to be called once the access token and device id are issued
+            (str) the canonical_user_id
         Raises:
-            StoreError if there was a problem accessing the database
-            SynapseError if there was a problem with the request
-            LoginError if there was an authentication problem.
+            LoginError if login fails
         """
-
-        if username.startswith('@'):
-            qualified_user_id = username
-        else:
-            qualified_user_id = UserID(
-                username, self.hs.hostname
-            ).to_string()
-
-        login_type = login_submission.get("type")
-        known_login_type = False
-
-        # special case to check for "password" for the check_password interface
-        # for the auth providers
-        password = login_submission.get("password")
-        if login_type == LoginType.PASSWORD:
-            if not self._password_enabled:
-                raise SynapseError(400, "Password login has been disabled.")
-            if not password:
-                raise SynapseError(400, "Missing parameter: password")
-
         for provider in self.password_providers:
-            if (hasattr(provider, "check_password")
-                    and login_type == LoginType.PASSWORD):
-                known_login_type = True
-                is_valid = yield provider.check_password(
-                    qualified_user_id, password,
-                )
-                if is_valid:
-                    defer.returnValue(qualified_user_id)
+            is_valid = yield provider.check_password(user_id, password)
+            if is_valid:
+                defer.returnValue(user_id)
 
-            if (not hasattr(provider, "get_supported_login_types")
-                    or not hasattr(provider, "check_auth")):
-                # this password provider doesn't understand custom login types
-                continue
+        canonical_user_id = yield self._check_local_password(user_id, password)
 
-            supported_login_types = provider.get_supported_login_types()
-            if login_type not in supported_login_types:
-                # this password provider doesn't understand this login type
-                continue
-
-            known_login_type = True
-            login_fields = supported_login_types[login_type]
-
-            missing_fields = []
-            login_dict = {}
-            for f in login_fields:
-                if f not in login_submission:
-                    missing_fields.append(f)
-                else:
-                    login_dict[f] = login_submission[f]
-            if missing_fields:
-                raise SynapseError(
-                    400, "Missing parameters for login type %s: %s" % (
-                        login_type,
-                        missing_fields,
-                    ),
-                )
-
-            result = yield provider.check_auth(
-                username, login_type, login_dict,
-            )
-            if result:
-                if isinstance(result, str):
-                    result = (result, None)
-                defer.returnValue(result)
-
-        if login_type == LoginType.PASSWORD:
-            known_login_type = True
-
-            canonical_user_id = yield self._check_local_password(
-                qualified_user_id, password,
-            )
-
-            if canonical_user_id:
-                defer.returnValue((canonical_user_id, None))
-
-        if not known_login_type:
-            raise SynapseError(400, "Unknown login type %s" % login_type)
+        if canonical_user_id:
+            defer.returnValue(canonical_user_id)
 
         # unknown username or invalid password. We raise a 403 here, but note
         # that if we're doing user-interactive login, it turns all LoginErrors
@@ -661,80 +557,13 @@ class AuthHandler(BaseHandler):
             if e.code == 404:
                 raise SynapseError(404, "Unknown user", Codes.NOT_FOUND)
             raise e
-        yield self.delete_access_tokens_for_user(
-            user_id, except_token_id=except_access_token_id,
+        yield self.store.user_delete_access_tokens(
+            user_id, except_access_token_id
         )
         yield self.hs.get_pusherpool().remove_pushers_by_user(
             user_id, except_access_token_id
         )
 
-    @defer.inlineCallbacks
-    def deactivate_account(self, user_id):
-        """Deactivate a user's account
-
-        Args:
-            user_id (str): ID of user to be deactivated
-
-        Returns:
-            Deferred
-        """
-        # FIXME: Theoretically there is a race here wherein user resets
-        # password using threepid.
-        yield self.delete_access_tokens_for_user(user_id)
-        yield self.store.user_delete_threepids(user_id)
-        yield self.store.user_set_password_hash(user_id, None)
-
-    @defer.inlineCallbacks
-    def delete_access_token(self, access_token):
-        """Invalidate a single access token
-
-        Args:
-            access_token (str): access token to be deleted
-
-        Returns:
-            Deferred
-        """
-        user_info = yield self.auth.get_user_by_access_token(access_token)
-        yield self.store.delete_access_token(access_token)
-
-        # see if any of our auth providers want to know about this
-        for provider in self.password_providers:
-            if hasattr(provider, "on_logged_out"):
-                yield provider.on_logged_out(
-                    user_id=str(user_info["user"]),
-                    device_id=user_info["device_id"],
-                    access_token=access_token,
-                )
-
-    @defer.inlineCallbacks
-    def delete_access_tokens_for_user(self, user_id, except_token_id=None,
-                                      device_id=None):
-        """Invalidate access tokens belonging to a user
-
-        Args:
-            user_id (str):  ID of user the tokens belong to
-            except_token_id (str|None): access_token ID which should *not* be
-                deleted
-            device_id (str|None):  ID of device the tokens are associated with.
-                If None, tokens associated with any device (or no device) will
-                be deleted
-        Returns:
-            Deferred
-        """
-        tokens_and_devices = yield self.store.user_delete_access_tokens(
-            user_id, except_token_id=except_token_id, device_id=device_id,
-        )
-
-        # see if any of our auth providers want to know about this
-        for provider in self.password_providers:
-            if hasattr(provider, "on_logged_out"):
-                for token, device_id in tokens_and_devices:
-                    yield provider.on_logged_out(
-                        user_id=user_id,
-                        device_id=device_id,
-                        access_token=token,
-                    )
-
     @defer.inlineCallbacks
     def add_threepid(self, user_id, medium, address, validated_at):
         # 'Canonicalise' email addresses down to lower case.
@@ -770,6 +599,16 @@ class AuthHandler(BaseHandler):
         logger.debug("Saving session %s", session)
         session["last_used"] = self.hs.get_clock().time_msec()
         self.sessions[session["id"]] = session
+        self._prune_sessions()
+
+    def _prune_sessions(self):
+        for sid, sess in self.sessions.items():
+            last_used = 0
+            if 'last_used' in sess:
+                last_used = sess['last_used']
+            now = self.hs.get_clock().time_msec()
+            if last_used < now - AuthHandler.SESSION_EXPIRE_MS:
+                del self.sessions[sid]
 
     def hash(self, password):
         """Computes a secure hash of password.
@@ -840,3 +679,30 @@ class MacaroonGeneartor(object):
         macaroon.add_first_party_caveat("gen = 1")
         macaroon.add_first_party_caveat("user_id = %s" % (user_id,))
         return macaroon
+
+
+class _AccountHandler(object):
+    """A proxy object that gets passed to password auth providers so they
+    can register new users etc if necessary.
+    """
+    def __init__(self, hs, check_user_exists):
+        self.hs = hs
+
+        self._check_user_exists = check_user_exists
+
+    def check_user_exists(self, user_id):
+        """Check if user exissts.
+
+        Returns:
+            Deferred(bool)
+        """
+        return self._check_user_exists(user_id)
+
+    def register(self, localpart):
+        """Registers a new user with given localpart
+
+        Returns:
+            Deferred: a 2-tuple of (user_id, access_token)
+        """
+        reg = self.hs.get_handlers().registration_handler
+        return reg.register(localpart=localpart)

synapse/handlers/device.py

@@ -16,8 +16,6 @@ from synapse.api import errors
 from synapse.api.constants import EventTypes
 from synapse.util import stringutils
 from synapse.util.async import Linearizer
-from synapse.util.caches.expiringcache import ExpiringCache
-from synapse.util.retryutils import NotRetryingDestination
 from synapse.util.metrics import measure_func
 from synapse.types import get_domain_from_id, RoomStreamToken
 from twisted.internet import defer
@@ -34,14 +32,12 @@ class DeviceHandler(BaseHandler):
 
         self.hs = hs
         self.state = hs.get_state_handler()
-        self._auth_handler = hs.get_auth_handler()
         self.federation_sender = hs.get_federation_sender()
         self.federation = hs.get_replication_layer()
-
-        self._edu_updater = DeviceListEduUpdater(hs, self)
+        self._remote_edue_linearizer = Linearizer(name="remote_device_list")
 
         self.federation.register_edu_handler(
-            "m.device_list_update", self._edu_updater.incoming_device_list_update,
+            "m.device_list_update", self._incoming_device_list_update,
         )
         self.federation.register_query_handler(
             "user_devices", self.on_federation_query_user_devices,
@@ -107,7 +103,7 @@ class DeviceHandler(BaseHandler):
         device_map = yield self.store.get_devices_by_user(user_id)
 
         ips = yield self.store.get_last_client_ip_by_device(
-            user_id, device_id=None
+            devices=((user_id, device_id) for device_id in device_map.keys())
         )
 
         devices = device_map.values()
@@ -134,7 +130,7 @@ class DeviceHandler(BaseHandler):
         except errors.StoreError:
             raise errors.NotFoundError
         ips = yield self.store.get_last_client_ip_by_device(
-            user_id, device_id,
+            devices=((user_id, device_id),)
         )
         _update_device_from_client_ips(device, ips)
         defer.returnValue(device)
@@ -160,8 +156,9 @@ class DeviceHandler(BaseHandler):
             else:
                 raise
 
-        yield self._auth_handler.delete_access_tokens_for_user(
+        yield self.store.user_delete_access_tokens(
             user_id, device_id=device_id,
+            delete_refresh_tokens=True,
         )
 
         yield self.store.delete_e2e_keys_by_device(
@@ -170,39 +167,6 @@ class DeviceHandler(BaseHandler):
 
         yield self.notify_device_update(user_id, [device_id])
 
-    @defer.inlineCallbacks
-    def delete_devices(self, user_id, device_ids):
-        """ Delete several devices
-
-        Args:
-            user_id (str):
-            device_ids (str): The list of device IDs to delete
-
-        Returns:
-            defer.Deferred:
-        """
-
-        try:
-            yield self.store.delete_devices(user_id, device_ids)
-        except errors.StoreError, e:
-            if e.code == 404:
-                # no match
-                pass
-            else:
-                raise
-
-        # Delete access tokens and e2e keys for each device. Not optimised as it is not
-        # considered as part of a critical path.
-        for device_id in device_ids:
-            yield self._auth_handler.delete_access_tokens_for_user(
-                user_id, device_id=device_id,
-            )
-            yield self.store.delete_e2e_keys_by_device(
-                user_id=user_id, device_id=device_id
-            )
-
-        yield self.notify_device_update(user_id, device_ids)
-
     @defer.inlineCallbacks
     def update_device(self, user_id, device_id, content):
         """ Update the given device
@@ -248,7 +212,8 @@ class DeviceHandler(BaseHandler):
             user_id, device_ids, list(hosts)
         )
 
-        room_ids = yield self.store.get_rooms_for_user(user_id)
+        rooms = yield self.store.get_rooms_for_user(user_id)
+        room_ids = [r.room_id for r in rooms]
 
         yield self.notifier.on_new_event(
             "device_list_key", position, rooms=room_ids,
@@ -269,9 +234,8 @@ class DeviceHandler(BaseHandler):
             user_id (str)
             from_token (StreamToken)
         """
-        now_token = yield self.hs.get_event_sources().get_current_token()
-
-        room_ids = yield self.store.get_rooms_for_user(user_id)
+        rooms = yield self.store.get_rooms_for_user(user_id)
+        room_ids = set(r.room_id for r in rooms)
 
         # First we check if any devices have changed
         changed = yield self.store.get_user_whose_devices_changed(
@@ -281,30 +245,11 @@ class DeviceHandler(BaseHandler):
         # Then work out if any users have since joined
         rooms_changed = self.store.get_rooms_that_changed(room_ids, from_token.room_key)
 
-        member_events = yield self.store.get_membership_changes_for_user(
-            user_id, from_token.room_key, now_token.room_key
-        )
-        rooms_changed.update(event.room_id for event in member_events)
-
         stream_ordering = RoomStreamToken.parse_stream_token(
-            from_token.room_key
-        ).stream
+            from_token.room_key).stream
 
         possibly_changed = set(changed)
-        possibly_left = set()
         for room_id in rooms_changed:
-            current_state_ids = yield self.store.get_current_state_ids(room_id)
-
-            # The user may have left the room
-            # TODO: Check if they actually did or if we were just invited.
-            if room_id not in room_ids:
-                for key, event_id in current_state_ids.iteritems():
-                    etype, state_key = key
-                    if etype != EventTypes.Member:
-                        continue
-                    possibly_left.add(state_key)
-                continue
-
             # Fetch the current state at the time.
             try:
                 event_ids = yield self.store.get_forward_extremeties_for_room(
@@ -315,6 +260,8 @@ class DeviceHandler(BaseHandler):
                 # ordering: treat it the same as a new room
                 event_ids = []
 
+            current_state_ids = yield self.state.get_current_state_ids(room_id)
+
             # special-case for an empty prev state: include all members
             # in the changed list
             if not event_ids:
@@ -325,25 +272,9 @@ class DeviceHandler(BaseHandler):
                     possibly_changed.add(state_key)
                 continue
 
-            current_member_id = current_state_ids.get((EventTypes.Member, user_id))
-            if not current_member_id:
-                continue
-
             # mapping from event_id -> state_dict
             prev_state_ids = yield self.store.get_state_ids_for_events(event_ids)
 
-            # Check if we've joined the room? If so we just blindly add all the users to
-            # the "possibly changed" users.
-            for state_dict in prev_state_ids.itervalues():
-                member_event = state_dict.get((EventTypes.Member, user_id), None)
-                if not member_event or member_event != current_member_id:
-                    for key, event_id in current_state_ids.iteritems():
-                        etype, state_key = key
-                        if etype != EventTypes.Member:
-                            continue
-                        possibly_changed.add(state_key)
-                    break
-
             # If there has been any change in membership, include them in the
             # possibly changed list. We'll check if they are joined below,
             # and we're not toooo worried about spuriously adding users.
@@ -354,30 +285,71 @@ class DeviceHandler(BaseHandler):
 
                 # check if this member has changed since any of the extremities
                 # at the stream_ordering, and add them to the list if so.
-                for state_dict in prev_state_ids.itervalues():
+                for state_dict in prev_state_ids.values():
                     prev_event_id = state_dict.get(key, None)
                     if not prev_event_id or prev_event_id != event_id:
-                        if state_key != user_id:
-                            possibly_changed.add(state_key)
+                        possibly_changed.add(state_key)
                         break
 
-        if possibly_changed or possibly_left:
-            users_who_share_room = yield self.store.get_users_who_share_room_with_user(
-                user_id
-            )
+        users_who_share_room = yield self.store.get_users_who_share_room_with_user(
+            user_id
+        )
 
-            # Take the intersection of the users whose devices may have changed
-            # and those that actually still share a room with the user
-            possibly_joined = possibly_changed & users_who_share_room
-            possibly_left = (possibly_changed | possibly_left) - users_who_share_room
-        else:
-            possibly_joined = []
-            possibly_left = []
+        # Take the intersection of the users whose devices may have changed
+        # and those that actually still share a room with the user
+        defer.returnValue(users_who_share_room & possibly_changed)
 
-        defer.returnValue({
-            "changed": list(possibly_joined),
-            "left": list(possibly_left),
-        })
+    @measure_func("_incoming_device_list_update")
+    @defer.inlineCallbacks
+    def _incoming_device_list_update(self, origin, edu_content):
+        user_id = edu_content["user_id"]
+        device_id = edu_content["device_id"]
+        stream_id = edu_content["stream_id"]
+        prev_ids = edu_content.get("prev_id", [])
+
+        if get_domain_from_id(user_id) != origin:
+            # TODO: Raise?
+            logger.warning("Got device list update edu for %r from %r", user_id, origin)
+            return
+
+        rooms = yield self.store.get_rooms_for_user(user_id)
+        if not rooms:
+            # We don't share any rooms with this user. Ignore update, as we
+            # probably won't get any further updates.
+            return
+
+        with (yield self._remote_edue_linearizer.queue(user_id)):
+            # If the prev id matches whats in our cache table, then we don't need
+            # to resync the users device list, otherwise we do.
+            resync = True
+            if len(prev_ids) == 1:
+                extremity = yield self.store.get_device_list_last_stream_id_for_remote(
+                    user_id
+                )
+                logger.info("Extrem: %r, prev_ids: %r", extremity, prev_ids)
+                if extremity and prev_ids[0] and int(extremity) >= int(prev_ids[0]):
+                    resync = False
+
+            if resync:
+                # Fetch all devices for the user.
+                result = yield self.federation.query_user_devices(origin, user_id)
+                stream_id = result["stream_id"]
+                devices = result["devices"]
+                yield self.store.update_remote_device_list_cache(
+                    user_id, devices, stream_id,
+                )
+                device_ids = [device["device_id"] for device in devices]
+                yield self.notify_device_update(user_id, device_ids)
+            else:
+                # Simply update the single device, since we know that is the only
+                # change (becuase of the single prev_id matching the current cache)
+                content = dict(edu_content)
+                for key in ("user_id", "device_id", "stream_id", "prev_ids"):
+                    content.pop(key, None)
+                yield self.store.update_remote_device_list_cache_entry(
+                    user_id, device_id, content, stream_id,
+                )
+                yield self.notify_device_update(user_id, [device_id])
 
     @defer.inlineCallbacks
     def on_federation_query_user_devices(self, user_id):
@@ -391,8 +363,8 @@ class DeviceHandler(BaseHandler):
     @defer.inlineCallbacks
     def user_left_room(self, user, room_id):
         user_id = user.to_string()
-        room_ids = yield self.store.get_rooms_for_user(user_id)
-        if not room_ids:
+        rooms = yield self.store.get_rooms_for_user(user_id)
+        if not rooms:
             # We no longer share rooms with this user, so we'll no longer
             # receive device updates. Mark this in DB.
             yield self.store.mark_remote_user_device_list_as_unsubscribed(user_id)
@@ -404,155 +376,3 @@ def _update_device_from_client_ips(device, client_ips):
         "last_seen_ts": ip.get("last_seen"),
         "last_seen_ip": ip.get("ip"),
     })
-
-
-class DeviceListEduUpdater(object):
-    "Handles incoming device list updates from federation and updates the DB"
-
-    def __init__(self, hs, device_handler):
-        self.store = hs.get_datastore()
-        self.federation = hs.get_replication_layer()
-        self.clock = hs.get_clock()
-        self.device_handler = device_handler
-
-        self._remote_edu_linearizer = Linearizer(name="remote_device_list")
-
-        # user_id -> list of updates waiting to be handled.
-        self._pending_updates = {}
-
-        # Recently seen stream ids. We don't bother keeping these in the DB,
-        # but they're useful to have them about to reduce the number of spurious
-        # resyncs.
-        self._seen_updates = ExpiringCache(
-            cache_name="device_update_edu",
-            clock=self.clock,
-            max_len=10000,
-            expiry_ms=30 * 60 * 1000,
-            iterable=True,
-        )
-
-    @defer.inlineCallbacks
-    def incoming_device_list_update(self, origin, edu_content):
-        """Called on incoming device list update from federation. Responsible
-        for parsing the EDU and adding to pending updates list.
-        """
-
-        user_id = edu_content.pop("user_id")
-        device_id = edu_content.pop("device_id")
-        stream_id = str(edu_content.pop("stream_id"))  # They may come as ints
-        prev_ids = edu_content.pop("prev_id", [])
-        prev_ids = [str(p) for p in prev_ids]   # They may come as ints
-
-        if get_domain_from_id(user_id) != origin:
-            # TODO: Raise?
-            logger.warning("Got device list update edu for %r from %r", user_id, origin)
-            return
-
-        room_ids = yield self.store.get_rooms_for_user(user_id)
-        if not room_ids:
-            # We don't share any rooms with this user. Ignore update, as we
-            # probably won't get any further updates.
-            return
-
-        self._pending_updates.setdefault(user_id, []).append(
-            (device_id, stream_id, prev_ids, edu_content)
-        )
-
-        yield self._handle_device_updates(user_id)
-
-    @measure_func("_incoming_device_list_update")
-    @defer.inlineCallbacks
-    def _handle_device_updates(self, user_id):
-        "Actually handle pending updates."
-
-        with (yield self._remote_edu_linearizer.queue(user_id)):
-            pending_updates = self._pending_updates.pop(user_id, [])
-            if not pending_updates:
-                # This can happen since we batch updates
-                return
-
-            # Given a list of updates we check if we need to resync. This
-            # happens if we've missed updates.
-            resync = yield self._need_to_do_resync(user_id, pending_updates)
-
-            if resync:
-                # Fetch all devices for the user.
-                origin = get_domain_from_id(user_id)
-                try:
-                    result = yield self.federation.query_user_devices(origin, user_id)
-                except NotRetryingDestination:
-                    # TODO: Remember that we are now out of sync and try again
-                    # later
-                    logger.warn(
-                        "Failed to handle device list update for %s,"
-                        " we're not retrying the remote",
-                        user_id,
-                    )
-                    # We abort on exceptions rather than accepting the update
-                    # as otherwise synapse will 'forget' that its device list
-                    # is out of date. If we bail then we will retry the resync
-                    # next time we get a device list update for this user_id.
-                    # This makes it more likely that the device lists will
-                    # eventually become consistent.
-                    return
-                except Exception:
-                    # TODO: Remember that we are now out of sync and try again
-                    # later
-                    logger.exception(
-                        "Failed to handle device list update for %s", user_id
-                    )
-                    return
-
-                stream_id = result["stream_id"]
-                devices = result["devices"]
-                yield self.store.update_remote_device_list_cache(
-                    user_id, devices, stream_id,
-                )
-                device_ids = [device["device_id"] for device in devices]
-                yield self.device_handler.notify_device_update(user_id, device_ids)
-            else:
-                # Simply update the single device, since we know that is the only
-                # change (becuase of the single prev_id matching the current cache)
-                for device_id, stream_id, prev_ids, content in pending_updates:
-                    yield self.store.update_remote_device_list_cache_entry(
-                        user_id, device_id, content, stream_id,
-                    )
-
-                yield self.device_handler.notify_device_update(
-                    user_id, [device_id for device_id, _, _, _ in pending_updates]
-                )
-
-            self._seen_updates.setdefault(user_id, set()).update(
-                stream_id for _, stream_id, _, _ in pending_updates
-            )
-
-    @defer.inlineCallbacks
-    def _need_to_do_resync(self, user_id, updates):
-        """Given a list of updates for a user figure out if we need to do a full
-        resync, or whether we have enough data that we can just apply the delta.
-        """
-        seen_updates = self._seen_updates.get(user_id, set())
-
-        extremity = yield self.store.get_device_list_last_stream_id_for_remote(
-            user_id
-        )
-
-        stream_id_in_updates = set()  # stream_ids in updates list
-        for _, stream_id, prev_ids, _ in updates:
-            if not prev_ids:
-                # We always do a resync if there are no previous IDs
-                defer.returnValue(True)
-
-            for prev_id in prev_ids:
-                if prev_id == extremity:
-                    continue
-                elif prev_id in seen_updates:
-                    continue
-                elif prev_id in stream_id_in_updates:
-                    continue
-                else:
-                    defer.returnValue(True)
-
-            stream_id_in_updates.add(stream_id)
-
-        defer.returnValue(False)

synapse/handlers/directory.py

@@ -40,8 +40,6 @@ class DirectoryHandler(BaseHandler):
             "directory", self.on_directory_query
         )
 
-        self.spam_checker = hs.get_spam_checker()
-
     @defer.inlineCallbacks
     def _create_association(self, room_alias, room_id, servers=None, creator=None):
         # general association creation for both human users and app services
@@ -75,11 +73,6 @@ class DirectoryHandler(BaseHandler):
         # association creation for human users
         # TODO(erikj): Do user auth.
 
-        if not self.spam_checker.user_may_create_room_alias(user_id, room_alias):
-            raise SynapseError(
-                403, "This user is not permitted to create this alias",
-            )
-
         can_create = yield self.can_modify_alias(
             room_alias,
             user_id=user_id
@@ -182,7 +175,6 @@ class DirectoryHandler(BaseHandler):
                         "room_alias": room_alias.to_string(),
                     },
                     retry_on_dns_fail=False,
-                    ignore_backoff=True,
                 )
             except CodeMessageException as e:
                 logging.warn("Error retrieving alias")
@@ -334,14 +326,6 @@ class DirectoryHandler(BaseHandler):
         room_id (str)
         visibility (str): "public" or "private"
         """
-        if not self.spam_checker.user_may_publish_room(
-            requester.user.to_string(), room_id
-        ):
-            raise AuthError(
-                403,
-                "This user is not permitted to publish rooms to the room list"
-            )
-
         if requester.is_guest:
             raise AuthError(403, "Guests cannot edit the published room list")
 

synapse/handlers/e2e_keys.py

@@ -21,8 +21,8 @@ from twisted.internet import defer
 
 from synapse.api.errors import SynapseError, CodeMessageException
 from synapse.types import get_domain_from_id
-from synapse.util.logcontext import preserve_fn, make_deferred_yieldable
-from synapse.util.retryutils import NotRetryingDestination
+from synapse.util.logcontext import preserve_fn, preserve_context_over_deferred
+from synapse.util.retryutils import get_retry_limiter, NotRetryingDestination
 
 logger = logging.getLogger(__name__)
 
@@ -121,11 +121,15 @@ class E2eKeysHandler(object):
         def do_remote_query(destination):
             destination_query = remote_queries_not_in_cache[destination]
             try:
-                remote_result = yield self.federation.query_client_keys(
-                    destination,
-                    {"device_keys": destination_query},
-                    timeout=timeout
+                limiter = yield get_retry_limiter(
+                    destination, self.clock, self.store
                 )
+                with limiter:
+                    remote_result = yield self.federation.query_client_keys(
+                        destination,
+                        {"device_keys": destination_query},
+                        timeout=timeout
+                    )
 
                 for user_id, keys in remote_result["device_keys"].items():
                     if user_id in destination_query:
@@ -145,7 +149,7 @@ class E2eKeysHandler(object):
                     "status": 503, "message": e.message
                 }
 
-        yield make_deferred_yieldable(defer.gatherResults([
+        yield preserve_context_over_deferred(defer.gatherResults([
             preserve_fn(do_remote_query)(destination)
             for destination in remote_queries_not_in_cache
         ]))
@@ -235,14 +239,18 @@ class E2eKeysHandler(object):
         def claim_client_keys(destination):
             device_keys = remote_queries[destination]
             try:
-                remote_result = yield self.federation.claim_client_keys(
-                    destination,
-                    {"one_time_keys": device_keys},
-                    timeout=timeout
+                limiter = yield get_retry_limiter(
+                    destination, self.clock, self.store
                 )
-                for user_id, keys in remote_result["one_time_keys"].items():
-                    if user_id in device_keys:
-                        json_result[user_id] = keys
+                with limiter:
+                    remote_result = yield self.federation.claim_client_keys(
+                        destination,
+                        {"one_time_keys": device_keys},
+                        timeout=timeout
+                    )
+                    for user_id, keys in remote_result["one_time_keys"].items():
+                        if user_id in device_keys:
+                            json_result[user_id] = keys
             except CodeMessageException as e:
                 failures[destination] = {
                     "status": e.code, "message": e.message
@@ -257,21 +265,11 @@ class E2eKeysHandler(object):
                     "status": 503, "message": e.message
                 }
 
-        yield make_deferred_yieldable(defer.gatherResults([
+        yield preserve_context_over_deferred(defer.gatherResults([
             preserve_fn(claim_client_keys)(destination)
             for destination in remote_queries
         ]))
 
-        logger.info(
-            "Claimed one-time-keys: %s",
-            ",".join((
-                "%s for %s:%s" % (key_id, user_id, device_id)
-                for user_id, user_keys in json_result.iteritems()
-                for device_id, device_keys in user_keys.iteritems()
-                for key_id, _ in device_keys.iteritems()
-            )),
-        )
-
         defer.returnValue({
             "one_time_keys": json_result,
             "failures": failures
@@ -298,8 +296,19 @@ class E2eKeysHandler(object):
 
         one_time_keys = keys.get("one_time_keys", None)
         if one_time_keys:
-            yield self._upload_one_time_keys_for_user(
-                user_id, device_id, time_now, one_time_keys,
+            logger.info(
+                "Adding %d one_time_keys for device %r for user %r at %d",
+                len(one_time_keys), device_id, user_id, time_now
+            )
+            key_list = []
+            for key_id, key_json in one_time_keys.items():
+                algorithm, key_id = key_id.split(":")
+                key_list.append((
+                    algorithm, key_id, encode_canonical_json(key_json)
+                ))
+
+            yield self.store.add_e2e_one_time_keys(
+                user_id, device_id, time_now, key_list
             )
 
         # the device should have been registered already, but it may have been
@@ -307,63 +316,8 @@ class E2eKeysHandler(object):
         # old access_token without an associated device_id. Either way, we
         # need to double-check the device is registered to avoid ending up with
         # keys without a corresponding device.
-        yield self.device_handler.check_device_registered(user_id, device_id)
+        self.device_handler.check_device_registered(user_id, device_id)
 
         result = yield self.store.count_e2e_one_time_keys(user_id, device_id)
 
         defer.returnValue({"one_time_key_counts": result})
-
-    @defer.inlineCallbacks
-    def _upload_one_time_keys_for_user(self, user_id, device_id, time_now,
-                                       one_time_keys):
-        logger.info(
-            "Adding one_time_keys %r for device %r for user %r at %d",
-            one_time_keys.keys(), device_id, user_id, time_now,
-        )
-
-        # make a list of (alg, id, key) tuples
-        key_list = []
-        for key_id, key_obj in one_time_keys.items():
-            algorithm, key_id = key_id.split(":")
-            key_list.append((
-                algorithm, key_id, key_obj
-            ))
-
-        # First we check if we have already persisted any of the keys.
-        existing_key_map = yield self.store.get_e2e_one_time_keys(
-            user_id, device_id, [k_id for _, k_id, _ in key_list]
-        )
-
-        new_keys = []  # Keys that we need to insert. (alg, id, json) tuples.
-        for algorithm, key_id, key in key_list:
-            ex_json = existing_key_map.get((algorithm, key_id), None)
-            if ex_json:
-                if not _one_time_keys_match(ex_json, key):
-                    raise SynapseError(
-                        400,
-                        ("One time key %s:%s already exists. "
-                         "Old key: %s; new key: %r") %
-                        (algorithm, key_id, ex_json, key)
-                    )
-            else:
-                new_keys.append((algorithm, key_id, encode_canonical_json(key)))
-
-        yield self.store.add_e2e_one_time_keys(
-            user_id, device_id, time_now, new_keys
-        )
-
-
-def _one_time_keys_match(old_key_json, new_key):
-    old_key = json.loads(old_key_json)
-
-    # if either is a string rather than an object, they must match exactly
-    if not isinstance(old_key, dict) or not isinstance(new_key, dict):
-        return old_key == new_key
-
-    # otherwise, we strip off the 'signatures' if any, because it's legitimate
-    # for different upload attempts to have different signatures.
-    old_key.pop("signatures", None)
-    new_key_copy = dict(new_key)
-    new_key_copy.pop("signatures", None)
-
-    return old_key == new_key_copy

synapse/handlers/federation.py

@@ -25,10 +25,13 @@ from synapse.api.errors import (
 )
 from synapse.api.constants import EventTypes, Membership, RejectedReason
 from synapse.events.validator import EventValidator
-from synapse.util import unwrapFirstError, logcontext
+from synapse.util import unwrapFirstError
+from synapse.util.logcontext import (
+    PreserveLoggingContext, preserve_fn, preserve_context_over_deferred
+)
 from synapse.util.metrics import measure_func
 from synapse.util.logutils import log_function
-from synapse.util.async import run_on_reactor, Linearizer
+from synapse.util.async import run_on_reactor
 from synapse.util.frozenutils import unfreeze
 from synapse.crypto.event_signing import (
     compute_event_signature, add_hashes_and_signatures,
@@ -39,6 +42,7 @@ from synapse.events.utils import prune_event
 
 from synapse.util.retryutils import NotRetryingDestination
 
+from synapse.push.action_generator import ActionGenerator
 from synapse.util.distributor import user_joined_room
 
 from twisted.internet import defer
@@ -70,258 +74,34 @@ class FederationHandler(BaseHandler):
         self.state_handler = hs.get_state_handler()
         self.server_name = hs.hostname
         self.keyring = hs.get_keyring()
-        self.action_generator = hs.get_action_generator()
-        self.is_mine_id = hs.is_mine_id
-        self.pusher_pool = hs.get_pusherpool()
-        self.spam_checker = hs.get_spam_checker()
 
         self.replication_layer.set_handler(self)
 
         # When joining a room we need to queue any events for that room up
         self.room_queues = {}
-        self._room_pdu_linearizer = Linearizer("fed_room_pdu")
-
-    @defer.inlineCallbacks
-    @log_function
-    def on_receive_pdu(self, origin, pdu, get_missing=True):
-        """ Process a PDU received via a federation /send/ transaction, or
-        via backfill of missing prev_events
-
-        Args:
-            origin (str): server which initiated the /send/ transaction. Will
-                be used to fetch missing events or state.
-            pdu (FrozenEvent): received PDU
-            get_missing (bool): True if we should fetch missing prev_events
-
-        Returns (Deferred): completes with None
-        """
-
-        # We reprocess pdus when we have seen them only as outliers
-        existing = yield self.get_persisted_pdu(
-            origin, pdu.event_id, do_auth=False
-        )
-
-        # FIXME: Currently we fetch an event again when we already have it
-        # if it has been marked as an outlier.
-
-        already_seen = (
-            existing and (
-                not existing.internal_metadata.is_outlier()
-                or pdu.internal_metadata.is_outlier()
-            )
-        )
-        if already_seen:
-            logger.debug("Already seen pdu %s", pdu.event_id)
-            return
-
-        # If we are currently in the process of joining this room, then we
-        # queue up events for later processing.
-        if pdu.room_id in self.room_queues:
-            logger.info("Ignoring PDU %s for room %s from %s for now; join "
-                        "in progress", pdu.event_id, pdu.room_id, origin)
-            self.room_queues[pdu.room_id].append((pdu, origin))
-            return
-
-        # If we're no longer in the room just ditch the event entirely. This
-        # is probably an old server that has come back and thinks we're still
-        # in the room (or we've been rejoined to the room by a state reset).
-        #
-        # If we were never in the room then maybe our database got vaped and
-        # we should check if we *are* in fact in the room. If we are then we
-        # can magically rejoin the room.
-        is_in_room = yield self.auth.check_host_in_room(
-            pdu.room_id,
-            self.server_name
-        )
-        if not is_in_room:
-            was_in_room = yield self.store.was_host_joined(
-                pdu.room_id, self.server_name,
-            )
-            if was_in_room:
-                logger.info(
-                    "Ignoring PDU %s for room %s from %s as we've left the room!",
-                    pdu.event_id, pdu.room_id, origin,
-                )
-                return
-
-        state = None
-
-        auth_chain = []
-
-        have_seen = yield self.store.have_events(
-            [ev for ev, _ in pdu.prev_events]
-        )
-
-        fetch_state = False
-
-        # Get missing pdus if necessary.
-        if not pdu.internal_metadata.is_outlier():
-            # We only backfill backwards to the min depth.
-            min_depth = yield self.get_min_depth_for_context(
-                pdu.room_id
-            )
-
-            logger.debug(
-                "_handle_new_pdu min_depth for %s: %d",
-                pdu.room_id, min_depth
-            )
-
-            prevs = {e_id for e_id, _ in pdu.prev_events}
-            seen = set(have_seen.keys())
-
-            if min_depth and pdu.depth < min_depth:
-                # This is so that we don't notify the user about this
-                # message, to work around the fact that some events will
-                # reference really really old events we really don't want to
-                # send to the clients.
-                pdu.internal_metadata.outlier = True
-            elif min_depth and pdu.depth > min_depth:
-                if get_missing and prevs - seen:
-                    # If we're missing stuff, ensure we only fetch stuff one
-                    # at a time.
-                    logger.info(
-                        "Acquiring lock for room %r to fetch %d missing events: %r...",
-                        pdu.room_id, len(prevs - seen), list(prevs - seen)[:5],
-                    )
-                    with (yield self._room_pdu_linearizer.queue(pdu.room_id)):
-                        logger.info(
-                            "Acquired lock for room %r to fetch %d missing events",
-                            pdu.room_id, len(prevs - seen),
-                        )
-
-                        yield self._get_missing_events_for_pdu(
-                            origin, pdu, prevs, min_depth
-                        )
-
-                        # Update the set of things we've seen after trying to
-                        # fetch the missing stuff
-                        have_seen = yield self.store.have_events(prevs)
-                        seen = set(have_seen.iterkeys())
-
-                        if not prevs - seen:
-                            logger.info(
-                                "Found all missing prev events for %s", pdu.event_id
-                            )
-                elif prevs - seen:
-                    logger.info(
-                        "Not fetching %d missing events for room %r,event %s: %r...",
-                        len(prevs - seen), pdu.room_id, pdu.event_id,
-                        list(prevs - seen)[:5],
-                    )
-
-            if prevs - seen:
-                logger.info(
-                    "Still missing %d events for room %r: %r...",
-                    len(prevs - seen), pdu.room_id, list(prevs - seen)[:5]
-                )
-                fetch_state = True
-
-        if fetch_state:
-            # We need to get the state at this event, since we haven't
-            # processed all the prev events.
-            logger.debug(
-                "_handle_new_pdu getting state for %s",
-                pdu.room_id
-            )
-            try:
-                state, auth_chain = yield self.replication_layer.get_state_for_room(
-                    origin, pdu.room_id, pdu.event_id,
-                )
-            except Exception:
-                logger.exception("Failed to get state for event: %s", pdu.event_id)
-
-        yield self._process_received_pdu(
-            origin,
-            pdu,
-            state=state,
-            auth_chain=auth_chain,
-        )
-
-    @defer.inlineCallbacks
-    def _get_missing_events_for_pdu(self, origin, pdu, prevs, min_depth):
-        """
-        Args:
-            origin (str): Origin of the pdu. Will be called to get the missing events
-            pdu: received pdu
-            prevs (set(str)): List of event ids which we are missing
-            min_depth (int): Minimum depth of events to return.
-        """
-        # We recalculate seen, since it may have changed.
-        have_seen = yield self.store.have_events(prevs)
-        seen = set(have_seen.keys())
-
-        if not prevs - seen:
-            return
-
-        latest = yield self.store.get_latest_event_ids_in_room(
-            pdu.room_id
-        )
-
-        # We add the prev events that we have seen to the latest
-        # list to ensure the remote server doesn't give them to us
-        latest = set(latest)
-        latest |= seen
-
-        logger.info(
-            "Missing %d events for room %r pdu %s: %r...",
-            len(prevs - seen), pdu.room_id, pdu.event_id, list(prevs - seen)[:5]
-        )
-
-        # XXX: we set timeout to 10s to help workaround
-        # https://github.com/matrix-org/synapse/issues/1733.
-        # The reason is to avoid holding the linearizer lock
-        # whilst processing inbound /send transactions, causing
-        # FDs to stack up and block other inbound transactions
-        # which empirically can currently take up to 30 minutes.
-        #
-        # N.B. this explicitly disables retry attempts.
-        #
-        # N.B. this also increases our chances of falling back to
-        # fetching fresh state for the room if the missing event
-        # can't be found, which slightly reduces our security.
-        # it may also increase our DAG extremity count for the room,
-        # causing additional state resolution?  See #1760.
-        # However, fetching state doesn't hold the linearizer lock
-        # apparently.
-        #
-        # see https://github.com/matrix-org/synapse/pull/1744
-
-        missing_events = yield self.replication_layer.get_missing_events(
-            origin,
-            pdu.room_id,
-            earliest_events_ids=list(latest),
-            latest_events=[pdu],
-            limit=10,
-            min_depth=min_depth,
-            timeout=10000,
-        )
-
-        logger.info(
-            "Got %d events: %r...",
-            len(missing_events), [e.event_id for e in missing_events[:5]]
-        )
-
-        # We want to sort these by depth so we process them and
-        # tell clients about them in order.
-        missing_events.sort(key=lambda x: x.depth)
-
-        for e in missing_events:
-            logger.info("Handling found event %s", e.event_id)
-            yield self.on_receive_pdu(
-                origin,
-                e,
-                get_missing=False
-            )
 
     @log_function
     @defer.inlineCallbacks
-    def _process_received_pdu(self, origin, pdu, state, auth_chain):
-        """ Called when we have a new pdu. We need to do auth checks and put it
-        through the StateHandler.
+    def on_receive_pdu(self, origin, pdu, state=None, auth_chain=None):
+        """ Called by the ReplicationLayer when we have a new pdu. We need to
+        do auth checks and put it through the StateHandler.
+
+        auth_chain and state are None if we already have the necessary state
+        and prev_events in the db
         """
         event = pdu
 
-        logger.debug("Processing event: %s", event)
+        logger.debug("Got event: %s", event.event_id)
+
+        # If we are currently in the process of joining this room, then we
+        # queue up events for later processing.
+        if event.room_id in self.room_queues:
+            self.room_queues[event.room_id].append((pdu, origin))
+            return
+
+        logger.debug("Processing event: %s", event.event_id)
+
+        logger.debug("Event: %s", event)
 
         # FIXME (erikj): Awful hack to make the case where we are not currently
         # in the room work
@@ -401,6 +181,13 @@ class FederationHandler(BaseHandler):
                     affected=event.event_id,
                 )
 
+        # if we're receiving valid events from an origin,
+        # it's probably a good idea to mark it as not in retry-state
+        # for sending (although this is a bit of a leap)
+        retry_timings = yield self.store.get_destination_retry_timings(origin)
+        if retry_timings and retry_timings["retry_last_ts"]:
+            self.store.set_destination_retry_timings(origin, 0, 0)
+
         room = yield self.store.get_room(event.room_id)
 
         if not room:
@@ -419,10 +206,11 @@ class FederationHandler(BaseHandler):
             target_user = UserID.from_string(target_user_id)
             extra_users.append(target_user)
 
-        self.notifier.on_new_room_event(
-            event, event_stream_id, max_stream_id,
-            extra_users=extra_users
-        )
+        with PreserveLoggingContext():
+            self.notifier.on_new_room_event(
+                event, event_stream_id, max_stream_id,
+                extra_users=extra_users
+            )
 
         if event.type == EventTypes.Member:
             if event.membership == Membership.JOIN:
@@ -461,7 +249,7 @@ class FederationHandler(BaseHandler):
         def check_match(id):
             try:
                 return server_name == get_domain_from_id(id)
-            except Exception:
+            except:
                 return False
 
         # Parses mapping `event_id -> (type, state_key) -> state event_id`
@@ -499,7 +287,7 @@ class FederationHandler(BaseHandler):
                             continue
                         try:
                             domain = get_domain_from_id(ev.state_key)
-                        except Exception:
+                        except:
                             continue
 
                         if domain != server_name:
@@ -610,9 +398,9 @@ class FederationHandler(BaseHandler):
                     missing_auth - failed_to_fetch
                 )
 
-                results = yield logcontext.make_deferred_yieldable(defer.gatherResults(
+                results = yield preserve_context_over_deferred(defer.gatherResults(
                     [
-                        logcontext.preserve_fn(self.replication_layer.get_pdu)(
+                        preserve_fn(self.replication_layer.get_pdu)(
                             [dest],
                             event_id,
                             outlier=True,
@@ -738,7 +526,7 @@ class FederationHandler(BaseHandler):
                         joined_domains[dom] = min(d, old_d)
                     else:
                         joined_domains[dom] = d
-                except Exception:
+                except:
                     pass
 
             return sorted(joined_domains.items(), key=lambda d: d[1])
@@ -804,14 +592,10 @@ class FederationHandler(BaseHandler):
         event_ids = list(extremities.keys())
 
         logger.debug("calling resolve_state_groups in _maybe_backfill")
-        states = yield logcontext.make_deferred_yieldable(defer.gatherResults(
-            [
-                logcontext.preserve_fn(self.state_handler.resolve_state_groups)(
-                    room_id, [e]
-                )
-                for e in event_ids
-            ], consumeErrors=True,
-        ))
+        states = yield preserve_context_over_deferred(defer.gatherResults([
+            preserve_fn(self.state_handler.resolve_state_groups)(room_id, [e])
+            for e in event_ids
+        ]))
         states = dict(zip(event_ids, [s.state for s in states]))
 
         state_map = yield self.store.get_events(
@@ -857,11 +641,7 @@ class FederationHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def on_event_auth(self, event_id):
-        event = yield self.store.get_event(event_id)
-        auth = yield self.store.get_auth_chain(
-            [auth_id for auth_id, _ in event.auth_events],
-            include_given=True
-        )
+        auth = yield self.store.get_auth_chain([event_id])
 
         for event in auth:
             event.signatures.update(
@@ -890,6 +670,8 @@ class FederationHandler(BaseHandler):
         """
         logger.debug("Joining %s to %s", joinee, room_id)
 
+        yield self.store.clean_room_for_join(room_id)
+
         origin, event = yield self._make_and_verify_event(
             target_hosts,
             room_id,
@@ -898,15 +680,7 @@ class FederationHandler(BaseHandler):
             content,
         )
 
-        # This shouldn't happen, because the RoomMemberHandler has a
-        # linearizer lock which only allows one operation per user per room
-        # at a time - so this is just paranoia.
-        assert (room_id not in self.room_queues)
-
         self.room_queues[room_id] = []
-
-        yield self.store.clean_room_for_join(room_id)
-
         handled_events = set()
 
         try:
@@ -940,7 +714,7 @@ class FederationHandler(BaseHandler):
                     room_creator_user_id="",
                     is_public=False
                 )
-            except Exception:
+            except:
                 # FIXME
                 pass
 
@@ -948,44 +722,28 @@ class FederationHandler(BaseHandler):
                 origin, auth_chain, state, event
             )
 
-            self.notifier.on_new_room_event(
-                event, event_stream_id, max_stream_id,
-                extra_users=[joinee]
-            )
+            with PreserveLoggingContext():
+                self.notifier.on_new_room_event(
+                    event, event_stream_id, max_stream_id,
+                    extra_users=[joinee]
+                )
 
             logger.debug("Finished joining %s to %s", joinee, room_id)
         finally:
             room_queue = self.room_queues[room_id]
             del self.room_queues[room_id]
 
-            # we don't need to wait for the queued events to be processed -
-            # it's just a best-effort thing at this point. We do want to do
-            # them roughly in order, though, otherwise we'll end up making
-            # lots of requests for missing prev_events which we do actually
-            # have. Hence we fire off the deferred, but don't wait for it.
+            for p, origin in room_queue:
+                if p.event_id in handled_events:
+                    continue
 
-            logcontext.preserve_fn(self._handle_queued_pdus)(room_queue)
+                try:
+                    self.on_receive_pdu(origin, p)
+                except:
+                    logger.exception("Couldn't handle pdu")
 
         defer.returnValue(True)
 
-    @defer.inlineCallbacks
-    def _handle_queued_pdus(self, room_queue):
-        """Process PDUs which got queued up while we were busy send_joining.
-
-        Args:
-            room_queue (list[FrozenEvent, str]): list of PDUs to be processed
-                and the servers that sent them
-        """
-        for p, origin in room_queue:
-            try:
-                logger.info("Processing queued PDU %s which was received "
-                            "while we were joining %s", p.event_id, p.room_id)
-                yield self.on_receive_pdu(origin, p)
-            except Exception as e:
-                logger.warn(
-                    "Error handling queued PDU %s from %s: %s",
-                    p.event_id, origin, e)
-
     @defer.inlineCallbacks
     @log_function
     def on_make_join_request(self, room_id, user_id):
@@ -1033,19 +791,9 @@ class FederationHandler(BaseHandler):
         )
 
         event.internal_metadata.outlier = False
-        # Send this event on behalf of the origin server.
-        #
-        # The reasons we have the destination server rather than the origin
-        # server send it are slightly mysterious: the origin server should have
-        # all the neccessary state once it gets the response to the send_join,
-        # so it could send the event itself if it wanted to. It may be that
-        # doing it this way reduces failure modes, or avoids certain attacks
-        # where a new server selectively tells a subset of the federation that
-        # it has joined.
-        #
-        # The fact is that, as of the current writing, Synapse doesn't send out
-        # the join event over federation after joining, and changing it now
-        # would introduce the danger of backwards-compatibility problems.
+        # Send this event on behalf of the origin server since they may not
+        # have an up to data view of the state of the room at this event so
+        # will not know which servers to send the event to.
         event.internal_metadata.send_on_behalf_of = origin
 
         context, event_stream_id, max_stream_id = yield self._handle_new_event(
@@ -1064,9 +812,10 @@ class FederationHandler(BaseHandler):
             target_user = UserID.from_string(target_user_id)
             extra_users.append(target_user)
 
-        self.notifier.on_new_room_event(
-            event, event_stream_id, max_stream_id, extra_users=extra_users
-        )
+        with PreserveLoggingContext():
+            self.notifier.on_new_room_event(
+                event, event_stream_id, max_stream_id, extra_users=extra_users
+            )
 
         if event.type == EventTypes.Member:
             if event.content["membership"] == Membership.JOIN:
@@ -1074,7 +823,9 @@ class FederationHandler(BaseHandler):
                 yield user_joined_room(self.distributor, user, event.room_id)
 
         state_ids = context.prev_state_ids.values()
-        auth_chain = yield self.store.get_auth_chain(state_ids)
+        auth_chain = yield self.store.get_auth_chain(set(
+            [event.event_id] + state_ids
+        ))
 
         state = yield self.store.get_events(context.prev_state_ids.values())
 
@@ -1091,34 +842,6 @@ class FederationHandler(BaseHandler):
         """
         event = pdu
 
-        if event.state_key is None:
-            raise SynapseError(400, "The invite event did not have a state key")
-
-        is_blocked = yield self.store.is_room_blocked(event.room_id)
-        if is_blocked:
-            raise SynapseError(403, "This room has been blocked on this server")
-
-        if self.hs.config.block_non_admin_invites:
-            raise SynapseError(403, "This server does not accept room invites")
-
-        if not self.spam_checker.user_may_invite(
-            event.sender, event.state_key, event.room_id,
-        ):
-            raise SynapseError(
-                403, "This user is not permitted to send invites to this server/user"
-            )
-
-        membership = event.content.get("membership")
-        if event.type != EventTypes.Member or membership != Membership.INVITE:
-            raise SynapseError(400, "The event was not an m.room.member invite event")
-
-        sender_domain = get_domain_from_id(event.sender)
-        if sender_domain != origin:
-            raise SynapseError(400, "The invite event was not from the server sending it")
-
-        if not self.is_mine_id(event.state_key):
-            raise SynapseError(400, "The invite event must be for this server")
-
         event.internal_metadata.outlier = True
         event.internal_metadata.invite_from_remote = True
 
@@ -1138,38 +861,48 @@ class FederationHandler(BaseHandler):
         )
 
         target_user = UserID.from_string(event.state_key)
-        self.notifier.on_new_room_event(
-            event, event_stream_id, max_stream_id,
-            extra_users=[target_user],
-        )
+        with PreserveLoggingContext():
+            self.notifier.on_new_room_event(
+                event, event_stream_id, max_stream_id,
+                extra_users=[target_user],
+            )
 
         defer.returnValue(event)
 
     @defer.inlineCallbacks
     def do_remotely_reject_invite(self, target_hosts, room_id, user_id):
-        origin, event = yield self._make_and_verify_event(
-            target_hosts,
-            room_id,
-            user_id,
-            "leave"
-        )
-        # Mark as outlier as we don't have any state for this event; we're not
-        # even in the room.
-        event.internal_metadata.outlier = True
-        event = self._sign_event(event)
+        try:
+            origin, event = yield self._make_and_verify_event(
+                target_hosts,
+                room_id,
+                user_id,
+                "leave"
+            )
+            signed_event = self._sign_event(event)
+        except SynapseError:
+            raise
+        except CodeMessageException as e:
+            logger.warn("Failed to reject invite: %s", e)
+            raise SynapseError(500, "Failed to reject invite")
 
-        # Try the host that we succesfully called /make_leave/ on first for
-        # the /send_leave/ request.
+        # Try the host we successfully got a response to /make_join/
+        # request first.
         try:
             target_hosts.remove(origin)
             target_hosts.insert(0, origin)
         except ValueError:
             pass
 
-        yield self.replication_layer.send_leave(
-            target_hosts,
-            event
-        )
+        try:
+            yield self.replication_layer.send_leave(
+                target_hosts,
+                signed_event
+            )
+        except SynapseError:
+            raise
+        except CodeMessageException as e:
+            logger.warn("Failed to reject invite: %s", e)
+            raise SynapseError(500, "Failed to reject invite")
 
         context = yield self.state_handler.compute_event_context(event)
 
@@ -1290,9 +1023,10 @@ class FederationHandler(BaseHandler):
             target_user = UserID.from_string(target_user_id)
             extra_users.append(target_user)
 
-        self.notifier.on_new_room_event(
-            event, event_stream_id, max_stream_id, extra_users=extra_users
-        )
+        with PreserveLoggingContext():
+            self.notifier.on_new_room_event(
+                event, event_stream_id, max_stream_id, extra_users=extra_users
+            )
 
         defer.returnValue(None)
 
@@ -1327,7 +1061,7 @@ class FederationHandler(BaseHandler):
             for event in res:
                 # We sign these again because there was a bug where we
                 # incorrectly signed things the first time round
-                if self.is_mine_id(event.event_id):
+                if self.hs.is_mine_id(event.event_id):
                     event.signatures.update(
                         compute_event_signature(
                             event,
@@ -1362,7 +1096,7 @@ class FederationHandler(BaseHandler):
                     if prev_id != event.event_id:
                         results[(event.type, event.state_key)] = prev_id
                 else:
-                    results.pop((event.type, event.state_key), None)
+                    del results[(event.type, event.state_key)]
 
             defer.returnValue(results.values())
         else:
@@ -1400,7 +1134,7 @@ class FederationHandler(BaseHandler):
         )
 
         if event:
-            if self.is_mine_id(event.event_id):
+            if self.hs.is_mine_id(event.event_id):
                 # FIXME: This is a temporary work around where we occasionally
                 # return events slightly differently than when they were
                 # originally signed
@@ -1444,8 +1178,9 @@ class FederationHandler(BaseHandler):
             auth_events=auth_events,
         )
 
-        if not event.internal_metadata.is_outlier() and not backfilled:
-            yield self.action_generator.handle_push_actions_for_event(
+        if not event.internal_metadata.is_outlier():
+            action_generator = ActionGenerator(self.hs)
+            yield action_generator.handle_push_actions_for_event(
                 event, context
             )
 
@@ -1458,7 +1193,7 @@ class FederationHandler(BaseHandler):
         if not backfilled:
             # this intentionally does not yield: we don't care about the result
             # and don't need to wait for it.
-            logcontext.preserve_fn(self.pusher_pool.on_new_notifications)(
+            preserve_fn(self.hs.get_pusherpool().on_new_notifications)(
                 event_stream_id, max_stream_id
             )
 
@@ -1471,16 +1206,16 @@ class FederationHandler(BaseHandler):
         a bunch of outliers, but not a chunk of individual events that depend
         on each other for state calculations.
         """
-        contexts = yield logcontext.make_deferred_yieldable(defer.gatherResults(
+        contexts = yield preserve_context_over_deferred(defer.gatherResults(
             [
-                logcontext.preserve_fn(self._prep_event)(
+                preserve_fn(self._prep_event)(
                     origin,
                     ev_info["event"],
                     state=ev_info.get("state"),
                     auth_events=ev_info.get("auth_events"),
                 )
                 for ev_info in event_infos
-            ], consumeErrors=True,
+            ]
         ))
 
         yield self.store.persist_events(
@@ -1590,17 +1325,7 @@ class FederationHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def _prep_event(self, origin, event, state=None, auth_events=None):
-        """
 
-        Args:
-            origin:
-            event:
-            state:
-            auth_events:
-
-        Returns:
-            Deferred, which resolves to synapse.events.snapshot.EventContext
-        """
         context = yield self.state_handler.compute_event_context(
             event, old_state=state,
         )
@@ -1637,7 +1362,7 @@ class FederationHandler(BaseHandler):
 
             context.rejected = RejectedReason.AUTH_ERROR
 
-        if event.type == EventTypes.GuestAccess and not context.rejected:
+        if event.type == EventTypes.GuestAccess:
             yield self.maybe_kick_guest_users(event)
 
         defer.returnValue(context)
@@ -1654,11 +1379,7 @@ class FederationHandler(BaseHandler):
                 pass
 
         # Now get the current auth_chain for the event.
-        event = yield self.store.get_event(event_id)
-        local_auth_chain = yield self.store.get_auth_chain(
-            [auth_id for auth_id, _ in event.auth_events],
-            include_given=True
-        )
+        local_auth_chain = yield self.store.get_auth_chain([event_id])
 
         # TODO: Check if we would now reject event_id. If so we need to tell
         # everyone.
@@ -1706,17 +1427,6 @@ class FederationHandler(BaseHandler):
     @defer.inlineCallbacks
     @log_function
     def do_auth(self, origin, event, context, auth_events):
-        """
-
-        Args:
-            origin (str):
-            event (synapse.events.FrozenEvent):
-            context (synapse.events.snapshot.EventContext):
-            auth_events (dict[(str, str)->str]):
-
-        Returns:
-            defer.Deferred[None]
-        """
         # Check if we have all the auth events.
         current_state = set(e.event_id for e in auth_events.values())
         event_auth_events = set(e_id for e_id, _ in event.auth_events)
@@ -1786,7 +1496,7 @@ class FederationHandler(BaseHandler):
                     [e_id for e_id, _ in event.auth_events]
                 )
                 seen_events = set(have_events.keys())
-            except Exception:
+            except:
                 # FIXME:
                 logger.exception("Failed to get auth chain")
 
@@ -1799,17 +1509,18 @@ class FederationHandler(BaseHandler):
             # Do auth conflict res.
             logger.info("Different auth: %s", different_auth)
 
-            different_events = yield logcontext.make_deferred_yieldable(
-                defer.gatherResults([
-                    logcontext.preserve_fn(self.store.get_event)(
+            different_events = yield preserve_context_over_deferred(defer.gatherResults(
+                [
+                    preserve_fn(self.store.get_event)(
                         d,
                         allow_none=True,
                         allow_rejected=False,
                     )
                     for d in different_auth
                     if d in have_events and not have_events[d]
-                ], consumeErrors=True)
-            ).addErrback(unwrapFirstError)
+                ],
+                consumeErrors=True
+            )).addErrback(unwrapFirstError)
 
             if different_events:
                 local_view = dict(auth_events)
@@ -1828,9 +1539,16 @@ class FederationHandler(BaseHandler):
                 current_state = set(e.event_id for e in auth_events.values())
                 different_auth = event_auth_events - current_state
 
-                self._update_context_for_auth_events(
-                    context, auth_events, event_key,
-                )
+                context.current_state_ids = dict(context.current_state_ids)
+                context.current_state_ids.update({
+                    k: a.event_id for k, a in auth_events.items()
+                    if k != event_key
+                })
+                context.prev_state_ids = dict(context.prev_state_ids)
+                context.prev_state_ids.update({
+                    k: a.event_id for k, a in auth_events.items()
+                })
+                context.state_group = self.store.get_next_state_group()
 
         if different_auth and not event.internal_metadata.is_outlier():
             logger.info("Different auth after resolution: %s", different_auth)
@@ -1854,9 +1572,7 @@ class FederationHandler(BaseHandler):
                 auth_ids = yield self.auth.compute_auth_events(
                     event, context.prev_state_ids
                 )
-                local_auth_chain = yield self.store.get_auth_chain(
-                    auth_ids, include_given=True
-                )
+                local_auth_chain = yield self.store.get_auth_chain(auth_ids)
 
                 try:
                     # 2. Get remote difference.
@@ -1903,16 +1619,23 @@ class FederationHandler(BaseHandler):
                         except AuthError:
                             pass
 
-                except Exception:
+                except:
                     # FIXME:
                     logger.exception("Failed to query auth chain")
 
                 # 4. Look at rejects and their proofs.
                 # TODO.
 
-                self._update_context_for_auth_events(
-                    context, auth_events, event_key,
-                )
+                context.current_state_ids = dict(context.current_state_ids)
+                context.current_state_ids.update({
+                    k: a.event_id for k, a in auth_events.items()
+                    if k != event_key
+                })
+                context.prev_state_ids = dict(context.prev_state_ids)
+                context.prev_state_ids.update({
+                    k: a.event_id for k, a in auth_events.items()
+                })
+                context.state_group = self.store.get_next_state_group()
 
         try:
             self.auth.check(event, auth_events=auth_events)
@@ -1920,35 +1643,6 @@ class FederationHandler(BaseHandler):
             logger.warn("Failed auth resolution for %r because %s", event, e)
             raise e
 
-    def _update_context_for_auth_events(self, context, auth_events,
-                                        event_key):
-        """Update the state_ids in an event context after auth event resolution
-
-        Args:
-            context (synapse.events.snapshot.EventContext): event context
-                to be updated
-
-            auth_events (dict[(str, str)->str]): Events to update in the event
-                context.
-
-            event_key ((str, str)): (type, state_key) for the current event.
-                this will not be included in the current_state in the context.
-        """
-        state_updates = {
-            k: a.event_id for k, a in auth_events.iteritems()
-            if k != event_key
-        }
-        context.current_state_ids = dict(context.current_state_ids)
-        context.current_state_ids.update(state_updates)
-        if context.delta_ids is not None:
-            context.delta_ids = dict(context.delta_ids)
-            context.delta_ids.update(state_updates)
-        context.prev_state_ids = dict(context.prev_state_ids)
-        context.prev_state_ids.update({
-            k: a.event_id for k, a in auth_events.iteritems()
-        })
-        context.state_group = self.store.get_next_state_group()
-
     @defer.inlineCallbacks
     def construct_auth_difference(self, local_auth, remote_auth):
         """ Given a local and remote auth chain, find the differences. This
@@ -1992,7 +1686,7 @@ class FederationHandler(BaseHandler):
         def get_next(it, opt=None):
             try:
                 return it.next()
-            except Exception:
+            except:
                 return opt
 
         current_local = get_next(local_iter)
@@ -2146,14 +1840,6 @@ class FederationHandler(BaseHandler):
     @defer.inlineCallbacks
     @log_function
     def on_exchange_third_party_invite_request(self, origin, room_id, event_dict):
-        """Handle an exchange_third_party_invite request from a remote server
-
-        The remote server will call this when it wants to turn a 3pid invite
-        into a normal m.room.member invite.
-
-        Returns:
-            Deferred: resolves (to None)
-        """
         builder = self.event_builder_factory.new(event_dict)
 
         message_handler = self.hs.get_handlers().message_handler
@@ -2172,12 +1858,9 @@ class FederationHandler(BaseHandler):
             raise e
         yield self._check_signature(event, context)
 
-        # XXX we send the invite here, but send_membership_event also sends it,
-        # so we end up making two requests. I think this is redundant.
         returned_invite = yield self.send_invite(origin, event)
         # TODO: Make sure the signatures actually are correct.
         event.signatures.update(returned_invite.signatures)
-
         member_handler = self.hs.get_handlers().room_member_handler
         yield member_handler.send_membership_event(None, event, context)
 

synapse/handlers/groups_local.py

@@ -1,418 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2017 Vector Creations Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from twisted.internet import defer
-
-from synapse.api.errors import SynapseError
-from synapse.types import get_domain_from_id
-
-import logging
-
-logger = logging.getLogger(__name__)
-
-
-def _create_rerouter(func_name):
-    """Returns a function that looks at the group id and calls the function
-    on federation or the local group server if the group is local
-    """
-    def f(self, group_id, *args, **kwargs):
-        if self.is_mine_id(group_id):
-            return getattr(self.groups_server_handler, func_name)(
-                group_id, *args, **kwargs
-            )
-        else:
-            destination = get_domain_from_id(group_id)
-            return getattr(self.transport_client, func_name)(
-                destination, group_id, *args, **kwargs
-            )
-    return f
-
-
-class GroupsLocalHandler(object):
-    def __init__(self, hs):
-        self.hs = hs
-        self.store = hs.get_datastore()
-        self.room_list_handler = hs.get_room_list_handler()
-        self.groups_server_handler = hs.get_groups_server_handler()
-        self.transport_client = hs.get_federation_transport_client()
-        self.auth = hs.get_auth()
-        self.clock = hs.get_clock()
-        self.keyring = hs.get_keyring()
-        self.is_mine_id = hs.is_mine_id
-        self.signing_key = hs.config.signing_key[0]
-        self.server_name = hs.hostname
-        self.notifier = hs.get_notifier()
-        self.attestations = hs.get_groups_attestation_signing()
-
-        self.profile_handler = hs.get_profile_handler()
-
-        # Ensure attestations get renewed
-        hs.get_groups_attestation_renewer()
-
-    # The following functions merely route the query to the local groups server
-    # or federation depending on if the group is local or remote
-
-    get_group_profile = _create_rerouter("get_group_profile")
-    update_group_profile = _create_rerouter("update_group_profile")
-    get_rooms_in_group = _create_rerouter("get_rooms_in_group")
-
-    get_invited_users_in_group = _create_rerouter("get_invited_users_in_group")
-
-    add_room_to_group = _create_rerouter("add_room_to_group")
-    update_room_in_group = _create_rerouter("update_room_in_group")
-    remove_room_from_group = _create_rerouter("remove_room_from_group")
-
-    update_group_summary_room = _create_rerouter("update_group_summary_room")
-    delete_group_summary_room = _create_rerouter("delete_group_summary_room")
-
-    update_group_category = _create_rerouter("update_group_category")
-    delete_group_category = _create_rerouter("delete_group_category")
-    get_group_category = _create_rerouter("get_group_category")
-    get_group_categories = _create_rerouter("get_group_categories")
-
-    update_group_summary_user = _create_rerouter("update_group_summary_user")
-    delete_group_summary_user = _create_rerouter("delete_group_summary_user")
-
-    update_group_role = _create_rerouter("update_group_role")
-    delete_group_role = _create_rerouter("delete_group_role")
-    get_group_role = _create_rerouter("get_group_role")
-    get_group_roles = _create_rerouter("get_group_roles")
-
-    @defer.inlineCallbacks
-    def get_group_summary(self, group_id, requester_user_id):
-        """Get the group summary for a group.
-
-        If the group is remote we check that the users have valid attestations.
-        """
-        if self.is_mine_id(group_id):
-            res = yield self.groups_server_handler.get_group_summary(
-                group_id, requester_user_id
-            )
-        else:
-            res = yield self.transport_client.get_group_summary(
-                get_domain_from_id(group_id), group_id, requester_user_id,
-            )
-
-            group_server_name = get_domain_from_id(group_id)
-
-            # Loop through the users and validate the attestations.
-            chunk = res["users_section"]["users"]
-            valid_users = []
-            for entry in chunk:
-                g_user_id = entry["user_id"]
-                attestation = entry.pop("attestation", {})
-                try:
-                    if get_domain_from_id(g_user_id) != group_server_name:
-                        yield self.attestations.verify_attestation(
-                            attestation,
-                            group_id=group_id,
-                            user_id=g_user_id,
-                            server_name=get_domain_from_id(g_user_id),
-                        )
-                    valid_users.append(entry)
-                except Exception as e:
-                    logger.info("Failed to verify user is in group: %s", e)
-
-            res["users_section"]["users"] = valid_users
-
-            res["users_section"]["users"].sort(key=lambda e: e.get("order", 0))
-            res["rooms_section"]["rooms"].sort(key=lambda e: e.get("order", 0))
-
-        # Add `is_publicised` flag to indicate whether the user has publicised their
-        # membership of the group on their profile
-        result = yield self.store.get_publicised_groups_for_user(requester_user_id)
-        is_publicised = group_id in result
-
-        res.setdefault("user", {})["is_publicised"] = is_publicised
-
-        defer.returnValue(res)
-
-    @defer.inlineCallbacks
-    def create_group(self, group_id, user_id, content):
-        """Create a group
-        """
-
-        logger.info("Asking to create group with ID: %r", group_id)
-
-        if self.is_mine_id(group_id):
-            res = yield self.groups_server_handler.create_group(
-                group_id, user_id, content
-            )
-            local_attestation = None
-            remote_attestation = None
-        else:
-            local_attestation = self.attestations.create_attestation(group_id, user_id)
-            content["attestation"] = local_attestation
-
-            content["user_profile"] = yield self.profile_handler.get_profile(user_id)
-
-            res = yield self.transport_client.create_group(
-                get_domain_from_id(group_id), group_id, user_id, content,
-            )
-
-            remote_attestation = res["attestation"]
-            yield self.attestations.verify_attestation(
-                remote_attestation,
-                group_id=group_id,
-                user_id=user_id,
-                server_name=get_domain_from_id(group_id),
-            )
-
-        is_publicised = content.get("publicise", False)
-        token = yield self.store.register_user_group_membership(
-            group_id, user_id,
-            membership="join",
-            is_admin=True,
-            local_attestation=local_attestation,
-            remote_attestation=remote_attestation,
-            is_publicised=is_publicised,
-        )
-        self.notifier.on_new_event(
-            "groups_key", token, users=[user_id],
-        )
-
-        defer.returnValue(res)
-
-    @defer.inlineCallbacks
-    def get_users_in_group(self, group_id, requester_user_id):
-        """Get users in a group
-        """
-        if self.is_mine_id(group_id):
-            res = yield self.groups_server_handler.get_users_in_group(
-                group_id, requester_user_id
-            )
-            defer.returnValue(res)
-
-        group_server_name = get_domain_from_id(group_id)
-
-        res = yield self.transport_client.get_users_in_group(
-            get_domain_from_id(group_id), group_id, requester_user_id,
-        )
-
-        chunk = res["chunk"]
-        valid_entries = []
-        for entry in chunk:
-            g_user_id = entry["user_id"]
-            attestation = entry.pop("attestation", {})
-            try:
-                if get_domain_from_id(g_user_id) != group_server_name:
-                    yield self.attestations.verify_attestation(
-                        attestation,
-                        group_id=group_id,
-                        user_id=g_user_id,
-                        server_name=get_domain_from_id(g_user_id),
-                    )
-                valid_entries.append(entry)
-            except Exception as e:
-                logger.info("Failed to verify user is in group: %s", e)
-
-        res["chunk"] = valid_entries
-
-        defer.returnValue(res)
-
-    @defer.inlineCallbacks
-    def join_group(self, group_id, user_id, content):
-        """Request to join a group
-        """
-        raise NotImplementedError()  # TODO
-
-    @defer.inlineCallbacks
-    def accept_invite(self, group_id, user_id, content):
-        """Accept an invite to a group
-        """
-        if self.is_mine_id(group_id):
-            yield self.groups_server_handler.accept_invite(
-                group_id, user_id, content
-            )
-            local_attestation = None
-            remote_attestation = None
-        else:
-            local_attestation = self.attestations.create_attestation(group_id, user_id)
-            content["attestation"] = local_attestation
-
-            res = yield self.transport_client.accept_group_invite(
-                get_domain_from_id(group_id), group_id, user_id, content,
-            )
-
-            remote_attestation = res["attestation"]
-
-            yield self.attestations.verify_attestation(
-                remote_attestation,
-                group_id=group_id,
-                user_id=user_id,
-                server_name=get_domain_from_id(group_id),
-            )
-
-        # TODO: Check that the group is public and we're being added publically
-        is_publicised = content.get("publicise", False)
-
-        token = yield self.store.register_user_group_membership(
-            group_id, user_id,
-            membership="join",
-            is_admin=False,
-            local_attestation=local_attestation,
-            remote_attestation=remote_attestation,
-            is_publicised=is_publicised,
-        )
-        self.notifier.on_new_event(
-            "groups_key", token, users=[user_id],
-        )
-
-        defer.returnValue({})
-
-    @defer.inlineCallbacks
-    def invite(self, group_id, user_id, requester_user_id, config):
-        """Invite a user to a group
-        """
-        content = {
-            "requester_user_id": requester_user_id,
-            "config": config,
-        }
-        if self.is_mine_id(group_id):
-            res = yield self.groups_server_handler.invite_to_group(
-                group_id, user_id, requester_user_id, content,
-            )
-        else:
-            res = yield self.transport_client.invite_to_group(
-                get_domain_from_id(group_id), group_id, user_id, requester_user_id,
-                content,
-            )
-
-        defer.returnValue(res)
-
-    @defer.inlineCallbacks
-    def on_invite(self, group_id, user_id, content):
-        """One of our users were invited to a group
-        """
-        # TODO: Support auto join and rejection
-
-        if not self.is_mine_id(user_id):
-            raise SynapseError(400, "User not on this server")
-
-        local_profile = {}
-        if "profile" in content:
-            if "name" in content["profile"]:
-                local_profile["name"] = content["profile"]["name"]
-            if "avatar_url" in content["profile"]:
-                local_profile["avatar_url"] = content["profile"]["avatar_url"]
-
-        token = yield self.store.register_user_group_membership(
-            group_id, user_id,
-            membership="invite",
-            content={"profile": local_profile, "inviter": content["inviter"]},
-        )
-        self.notifier.on_new_event(
-            "groups_key", token, users=[user_id],
-        )
-        try:
-            user_profile = yield self.profile_handler.get_profile(user_id)
-        except Exception as e:
-            logger.warn("No profile for user %s: %s", user_id, e)
-            user_profile = {}
-
-        defer.returnValue({"state": "invite", "user_profile": user_profile})
-
-    @defer.inlineCallbacks
-    def remove_user_from_group(self, group_id, user_id, requester_user_id, content):
-        """Remove a user from a group
-        """
-        if user_id == requester_user_id:
-            token = yield self.store.register_user_group_membership(
-                group_id, user_id,
-                membership="leave",
-            )
-            self.notifier.on_new_event(
-                "groups_key", token, users=[user_id],
-            )
-
-            # TODO: Should probably remember that we tried to leave so that we can
-            # retry if the group server is currently down.
-
-        if self.is_mine_id(group_id):
-            res = yield self.groups_server_handler.remove_user_from_group(
-                group_id, user_id, requester_user_id, content,
-            )
-        else:
-            content["requester_user_id"] = requester_user_id
-            res = yield self.transport_client.remove_user_from_group(
-                get_domain_from_id(group_id), group_id, requester_user_id,
-                user_id, content,
-            )
-
-        defer.returnValue(res)
-
-    @defer.inlineCallbacks
-    def user_removed_from_group(self, group_id, user_id, content):
-        """One of our users was removed/kicked from a group
-        """
-        # TODO: Check if user in group
-        token = yield self.store.register_user_group_membership(
-            group_id, user_id,
-            membership="leave",
-        )
-        self.notifier.on_new_event(
-            "groups_key", token, users=[user_id],
-        )
-
-    @defer.inlineCallbacks
-    def get_joined_groups(self, user_id):
-        group_ids = yield self.store.get_joined_groups(user_id)
-        defer.returnValue({"groups": group_ids})
-
-    @defer.inlineCallbacks
-    def get_publicised_groups_for_user(self, user_id):
-        if self.hs.is_mine_id(user_id):
-            result = yield self.store.get_publicised_groups_for_user(user_id)
-            defer.returnValue({"groups": result})
-        else:
-            result = yield self.transport_client.get_publicised_groups_for_user(
-                get_domain_from_id(user_id), user_id
-            )
-            # TODO: Verify attestations
-            defer.returnValue(result)
-
-    @defer.inlineCallbacks
-    def bulk_get_publicised_groups(self, user_ids, proxy=True):
-        destinations = {}
-        local_users = set()
-
-        for user_id in user_ids:
-            if self.hs.is_mine_id(user_id):
-                local_users.add(user_id)
-            else:
-                destinations.setdefault(
-                    get_domain_from_id(user_id), set()
-                ).add(user_id)
-
-        if not proxy and destinations:
-            raise SynapseError(400, "Some user_ids are not local")
-
-        results = {}
-        failed_results = []
-        for destination, dest_user_ids in destinations.iteritems():
-            try:
-                r = yield self.transport_client.bulk_get_publicised_groups(
-                    destination, list(dest_user_ids),
-                )
-                results.update(r["users"])
-            except Exception:
-                failed_results.extend(dest_user_ids)
-
-        for uid in local_users:
-            results[uid] = yield self.store.get_publicised_groups_for_user(
-                uid
-            )
-
-        defer.returnValue({"users": results})

synapse/handlers/identity.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2015, 2016 OpenMarket Ltd
-# Copyright 2017 Vector Creations Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -18,7 +17,7 @@
 from twisted.internet import defer
 
 from synapse.api.errors import (
-    MatrixCodeMessageException, CodeMessageException
+    CodeMessageException
 )
 from ._base import BaseHandler
 from synapse.util.async import run_on_reactor
@@ -90,9 +89,6 @@ class IdentityHandler(BaseHandler):
                 ),
                 {'sid': creds['sid'], 'client_secret': client_secret}
             )
-        except MatrixCodeMessageException as e:
-            logger.info("getValidated3pid failed with Matrix error: %r", e)
-            raise SynapseError(e.code, e.msg, e.errcode)
         except CodeMessageException as e:
             data = json.loads(e.msg)
 
@@ -154,7 +150,7 @@ class IdentityHandler(BaseHandler):
         params.update(kwargs)
 
         try:
-            data = yield self.http_client.post_json_get_json(
+            data = yield self.http_client.post_urlencoded_get_json(
                 "https://%s%s" % (
                     id_server,
                     "/_matrix/identity/api/v1/validate/email/requestToken"
@@ -162,46 +158,6 @@ class IdentityHandler(BaseHandler):
                 params
             )
             defer.returnValue(data)
-        except MatrixCodeMessageException as e:
-            logger.info("Proxied requestToken failed with Matrix error: %r", e)
-            raise SynapseError(e.code, e.msg, e.errcode)
-        except CodeMessageException as e:
-            logger.info("Proxied requestToken failed: %r", e)
-            raise e
-
-    @defer.inlineCallbacks
-    def requestMsisdnToken(
-            self, id_server, country, phone_number,
-            client_secret, send_attempt, **kwargs
-    ):
-        yield run_on_reactor()
-
-        if not self._should_trust_id_server(id_server):
-            raise SynapseError(
-                400, "Untrusted ID server '%s'" % id_server,
-                Codes.SERVER_NOT_TRUSTED
-            )
-
-        params = {
-            'country': country,
-            'phone_number': phone_number,
-            'client_secret': client_secret,
-            'send_attempt': send_attempt,
-        }
-        params.update(kwargs)
-
-        try:
-            data = yield self.http_client.post_json_get_json(
-                "https://%s%s" % (
-                    id_server,
-                    "/_matrix/identity/api/v1/validate/msisdn/requestToken"
-                ),
-                params
-            )
-            defer.returnValue(data)
-        except MatrixCodeMessageException as e:
-            logger.info("Proxied requestToken failed with Matrix error: %r", e)
-            raise SynapseError(e.code, e.msg, e.errcode)
         except CodeMessageException as e:
             logger.info("Proxied requestToken failed: %r", e)
             raise e

synapse/handlers/initial_sync.py

@@ -19,7 +19,6 @@ from synapse.api.constants import EventTypes, Membership
 from synapse.api.errors import AuthError, Codes
 from synapse.events.utils import serialize_event
 from synapse.events.validator import EventValidator
-from synapse.handlers.presence import format_user_presence_state
 from synapse.streams.config import PaginationConfig
 from synapse.types import (
     UserID, StreamToken,
@@ -214,7 +213,7 @@ class InitialSyncHandler(BaseHandler):
                     })
 
                 d["account_data"] = account_data_events
-            except Exception:
+            except:
                 logger.exception("Failed to get snapshot")
 
         yield concurrently_execute(handle_room, room_list, 10)
@@ -226,17 +225,9 @@ class InitialSyncHandler(BaseHandler):
                 "content": content,
             })
 
-        now = self.clock.time_msec()
-
         ret = {
             "rooms": rooms_ret,
-            "presence": [
-                {
-                    "type": "m.presence",
-                    "content": format_user_presence_state(event, now),
-                }
-                for event in presence
-            ],
+            "presence": presence,
             "account_data": account_data_events,
             "receipts": receipt,
             "end": now_token.to_string(),
@@ -372,6 +363,7 @@ class InitialSyncHandler(BaseHandler):
 
         @defer.inlineCallbacks
         def get_presence():
+            defer.returnValue([])
             states = yield presence_handler.get_states(
                 [m.user_id for m in room_members],
                 as_event=True,

synapse/handlers/message.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014 - 2016 OpenMarket Ltd
-# Copyright 2017 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -13,20 +12,21 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
 from twisted.internet import defer
 
 from synapse.api.constants import EventTypes, Membership
-from synapse.api.errors import AuthError, Codes, SynapseError
+from synapse.api.errors import AuthError, Codes, SynapseError, LimitExceededError
 from synapse.crypto.event_signing import add_hashes_and_signatures
 from synapse.events.utils import serialize_event
 from synapse.events.validator import EventValidator
+from synapse.push.action_generator import ActionGenerator
 from synapse.types import (
     UserID, RoomAlias, RoomStreamToken,
 )
 from synapse.util.async import run_on_reactor, ReadWriteLock, Limiter
 from synapse.util.logcontext import preserve_fn
 from synapse.util.metrics import measure_func
-from synapse.util.frozenutils import unfreeze
 from synapse.visibility import filter_events_for_client
 
 from ._base import BaseHandler
@@ -35,7 +35,6 @@ from canonicaljson import encode_canonical_json
 
 import logging
 import random
-import ujson
 
 logger = logging.getLogger(__name__)
 
@@ -48,20 +47,13 @@ class MessageHandler(BaseHandler):
         self.state = hs.get_state_handler()
         self.clock = hs.get_clock()
         self.validator = EventValidator()
-        self.profile_handler = hs.get_profile_handler()
 
         self.pagination_lock = ReadWriteLock()
 
-        self.pusher_pool = hs.get_pusherpool()
-
         # We arbitrarily limit concurrent event creation for a room to 5.
         # This is to stop us from diverging history *too* much.
         self.limiter = Limiter(max_count=5)
 
-        self.action_generator = hs.get_action_generator()
-
-        self.spam_checker = hs.get_spam_checker()
-
     @defer.inlineCallbacks
     def purge_history(self, room_id, event_id):
         event = yield self.store.get_event(event_id)
@@ -183,8 +175,7 @@ class MessageHandler(BaseHandler):
         defer.returnValue(chunk)
 
     @defer.inlineCallbacks
-    def create_event(self, requester, event_dict, token_id=None, txn_id=None,
-                     prev_event_ids=None):
+    def create_event(self, event_dict, token_id=None, txn_id=None, prev_event_ids=None):
         """
         Given a dict from a client, create a new event.
 
@@ -194,7 +185,6 @@ class MessageHandler(BaseHandler):
         Adds display names to Join membership events.
 
         Args:
-            requester
             event_dict (dict): An entire event
             token_id (str)
             txn_id (str)
@@ -214,7 +204,7 @@ class MessageHandler(BaseHandler):
 
                 if membership in {Membership.JOIN, Membership.INVITE}:
                     # If event doesn't include a display name, add one.
-                    profile = self.profile_handler
+                    profile = self.hs.get_handlers().profile_handler
                     content = builder.content
 
                     try:
@@ -236,7 +226,6 @@ class MessageHandler(BaseHandler):
 
             event, context = yield self._create_new_client_event(
                 builder=builder,
-                requester=requester,
                 prev_event_ids=prev_event_ids,
             )
 
@@ -262,7 +251,17 @@ class MessageHandler(BaseHandler):
         # We check here if we are currently being rate limited, so that we
         # don't do unnecessary work. We check again just before we actually
         # send the event.
-        yield self.ratelimit(requester, update=False)
+        time_now = self.clock.time()
+        allowed, time_allowed = self.ratelimiter.send_message(
+            event.sender, time_now,
+            msg_rate_hz=self.hs.config.rc_messages_per_second,
+            burst_count=self.hs.config.rc_message_burst_count,
+            update=False,
+        )
+        if not allowed:
+            raise LimitExceededError(
+                retry_after_ms=int(1000 * (time_allowed - time_now)),
+            )
 
         user = UserID.from_string(event.sender)
 
@@ -320,20 +319,10 @@ class MessageHandler(BaseHandler):
         See self.create_event and self.send_nonmember_event.
         """
         event, context = yield self.create_event(
-            requester,
             event_dict,
             token_id=requester.access_token_id,
             txn_id=txn_id
         )
-
-        spam_error = self.spam_checker.check_event_for_spam(event)
-        if spam_error:
-            if not isinstance(spam_error, basestring):
-                spam_error = "Spam is not permitted here"
-            raise SynapseError(
-                403, spam_error, Codes.FORBIDDEN
-            )
-
         yield self.send_nonmember_event(
             requester,
             event,
@@ -425,54 +414,9 @@ class MessageHandler(BaseHandler):
             [serialize_event(c, now) for c in room_state.values()]
         )
 
-    @defer.inlineCallbacks
-    def get_joined_members(self, requester, room_id):
-        """Get all the joined members in the room and their profile information.
-
-        If the user has left the room return the state events from when they left.
-
-        Args:
-            requester(Requester): The user requesting state events.
-            room_id(str): The room ID to get all state events from.
-        Returns:
-            A dict of user_id to profile info
-        """
-        user_id = requester.user.to_string()
-        if not requester.app_service:
-            # We check AS auth after fetching the room membership, as it
-            # requires us to pull out all joined members anyway.
-            membership, _ = yield self._check_in_room_or_world_readable(
-                room_id, user_id
-            )
-            if membership != Membership.JOIN:
-                raise NotImplementedError(
-                    "Getting joined members after leaving is not implemented"
-                )
-
-        users_with_profile = yield self.state.get_current_user_in_room(room_id)
-
-        # If this is an AS, double check that they are allowed to see the members.
-        # This can either be because the AS user is in the room or becuase there
-        # is a user in the room that the AS is "interested in"
-        if requester.app_service and user_id not in users_with_profile:
-            for uid in users_with_profile:
-                if requester.app_service.is_interested_in_user(uid):
-                    break
-            else:
-                # Loop fell through, AS has no interested users in room
-                raise AuthError(403, "Appservice not in room")
-
-        defer.returnValue({
-            user_id: {
-                "avatar_url": profile.avatar_url,
-                "display_name": profile.display_name,
-            }
-            for user_id, profile in users_with_profile.iteritems()
-        })
-
     @measure_func("_create_new_client_event")
     @defer.inlineCallbacks
-    def _create_new_client_event(self, builder, requester=None, prev_event_ids=None):
+    def _create_new_client_event(self, builder, prev_event_ids=None):
         if prev_event_ids:
             prev_events = yield self.store.add_event_hashes(prev_event_ids)
             prev_max_depth = yield self.store.get_max_depth_of_events(prev_event_ids)
@@ -512,8 +456,6 @@ class MessageHandler(BaseHandler):
         state_handler = self.state_handler
 
         context = yield state_handler.compute_event_context(builder)
-        if requester:
-            context.app_service = requester.app_service
 
         if builder.is_state():
             builder.prev_state = yield self.store.add_event_hashes(
@@ -551,7 +493,7 @@ class MessageHandler(BaseHandler):
         # We now need to go and hit out to wherever we need to hit out to.
 
         if ratelimit:
-            yield self.ratelimit(requester)
+            self.ratelimit(requester)
 
         try:
             yield self.auth.check_from_context(event, context)
@@ -559,14 +501,6 @@ class MessageHandler(BaseHandler):
             logger.warn("Denying new event %r because %s", event, err)
             raise err
 
-        # Ensure that we can round trip before trying to persist in db
-        try:
-            dump = ujson.dumps(unfreeze(event.content))
-            ujson.loads(dump)
-        except Exception:
-            logger.exception("Failed to encode content: %r", event.content)
-            raise
-
         yield self.maybe_kick_guest_users(event, context)
 
         if event.type == EventTypes.CanonicalAlias:
@@ -597,9 +531,9 @@ class MessageHandler(BaseHandler):
 
                 state_to_include_ids = [
                     e_id
-                    for k, e_id in context.current_state_ids.iteritems()
+                    for k, e_id in context.current_state_ids.items()
                     if k[0] in self.hs.config.room_invite_state_types
-                    or k == (EventTypes.Member, event.sender)
+                    or k[0] == EventTypes.Member and k[1] == event.sender
                 ]
 
                 state_to_include = yield self.store.get_events(state_to_include_ids)
@@ -611,7 +545,7 @@ class MessageHandler(BaseHandler):
                         "content": e.content,
                         "sender": e.sender,
                     }
-                    for e in state_to_include.itervalues()
+                    for e in state_to_include.values()
                 ]
 
                 invitee = UserID.from_string(event.state_key)
@@ -660,7 +594,8 @@ class MessageHandler(BaseHandler):
                 "Changing the room create event is forbidden",
             )
 
-        yield self.action_generator.handle_push_actions_for_event(
+        action_generator = ActionGenerator(self.hs)
+        yield action_generator.handle_push_actions_for_event(
             event, context
         )
 
@@ -670,16 +605,19 @@ class MessageHandler(BaseHandler):
 
         # this intentionally does not yield: we don't care about the result
         # and don't need to wait for it.
-        preserve_fn(self.pusher_pool.on_new_notifications)(
+        preserve_fn(self.hs.get_pusherpool().on_new_notifications)(
             event_stream_id, max_stream_id
         )
 
         @defer.inlineCallbacks
         def _notify():
             yield run_on_reactor()
-            self.notifier.on_new_room_event(
+            yield self.notifier.on_new_room_event(
                 event, event_stream_id, max_stream_id,
                 extra_users=extra_users
             )
 
         preserve_fn(_notify)()
+
+        # If invite, remove room_state from unsigned before sending.
+        event.unsigned.pop("invite_room_state", None)

synapse/handlers/presence.py

@@ -29,8 +29,6 @@ from synapse.api.errors import SynapseError
 from synapse.api.constants import PresenceState
 from synapse.storage.presence import UserPresenceState
 
-from synapse.util.caches.descriptors import cachedInlineCallbacks
-from synapse.util.async import Linearizer
 from synapse.util.logcontext import preserve_fn
 from synapse.util.logutils import log_function
 from synapse.util.metrics import Measure
@@ -188,7 +186,6 @@ class PresenceHandler(object):
         # process_id to millisecond timestamp last updated.
         self.external_process_to_current_syncs = {}
         self.external_process_last_updated_ms = {}
-        self.external_sync_linearizer = Linearizer(name="external_sync_linearizer")
 
         # Start a LoopingCall in 30s that fires every 5s.
         # The initial delay is to allow disconnected clients a chance to
@@ -318,7 +315,11 @@ class PresenceHandler(object):
             if to_federation_ping:
                 federation_presence_out_counter.inc_by(len(to_federation_ping))
 
-                self._push_to_remotes(to_federation_ping.values())
+                _, _, hosts_to_states = yield self._get_interested_parties(
+                    to_federation_ping.values()
+                )
+
+                self._push_to_remotes(hosts_to_states)
 
     def _handle_timeouts(self):
         """Checks the presence of users that have timed out and updates as
@@ -364,7 +365,7 @@ class PresenceHandler(object):
                 )
 
             preserve_fn(self._update_states)(changes)
-        except Exception:
+        except:
             logger.exception("Exception in _handle_timeouts loop")
 
     @defer.inlineCallbacks
@@ -372,6 +373,7 @@ class PresenceHandler(object):
         """We've seen the user do something that indicates they're interacting
         with the app.
         """
+        return
         user_id = user.to_string()
 
         bump_active_time_counter.inc()
@@ -401,6 +403,7 @@ class PresenceHandler(object):
                 Useful for streams that are not associated with an actual
                 client that is being used by a user.
         """
+        affect_presence = False
         if affect_presence:
             curr_sync = self.user_to_num_current_syncs.get(user_id, 0)
             self.user_to_num_current_syncs[user_id] = curr_sync + 1
@@ -462,6 +465,7 @@ class PresenceHandler(object):
             syncing_user_ids(set(str)): The set of user_ids that are
                 currently syncing on that server.
         """
+        return
 
         # Grab the previous list of user_ids that were syncing on that process
         prev_syncing_user_ids = (
@@ -506,73 +510,6 @@ class PresenceHandler(object):
         self.external_process_last_updated_ms[process_id] = self.clock.time_msec()
         self.external_process_to_current_syncs[process_id] = syncing_user_ids
 
-    @defer.inlineCallbacks
-    def update_external_syncs_row(self, process_id, user_id, is_syncing, sync_time_msec):
-        """Update the syncing users for an external process as a delta.
-
-        Args:
-            process_id (str): An identifier for the process the users are
-                syncing against. This allows synapse to process updates
-                as user start and stop syncing against a given process.
-            user_id (str): The user who has started or stopped syncing
-            is_syncing (bool): Whether or not the user is now syncing
-            sync_time_msec(int): Time in ms when the user was last syncing
-        """
-        with (yield self.external_sync_linearizer.queue(process_id)):
-            prev_state = yield self.current_state_for_user(user_id)
-
-            process_presence = self.external_process_to_current_syncs.setdefault(
-                process_id, set()
-            )
-
-            updates = []
-            if is_syncing and user_id not in process_presence:
-                if prev_state.state == PresenceState.OFFLINE:
-                    updates.append(prev_state.copy_and_replace(
-                        state=PresenceState.ONLINE,
-                        last_active_ts=sync_time_msec,
-                        last_user_sync_ts=sync_time_msec,
-                    ))
-                else:
-                    updates.append(prev_state.copy_and_replace(
-                        last_user_sync_ts=sync_time_msec,
-                    ))
-                process_presence.add(user_id)
-            elif user_id in process_presence:
-                updates.append(prev_state.copy_and_replace(
-                    last_user_sync_ts=sync_time_msec,
-                ))
-
-            if not is_syncing:
-                process_presence.discard(user_id)
-
-            if updates:
-                yield self._update_states(updates)
-
-            self.external_process_last_updated_ms[process_id] = self.clock.time_msec()
-
-    @defer.inlineCallbacks
-    def update_external_syncs_clear(self, process_id):
-        """Marks all users that had been marked as syncing by a given process
-        as offline.
-
-        Used when the process has stopped/disappeared.
-        """
-        with (yield self.external_sync_linearizer.queue(process_id)):
-            process_presence = self.external_process_to_current_syncs.pop(
-                process_id, set()
-            )
-            prev_states = yield self.current_state_for_users(process_presence)
-            time_now_ms = self.clock.time_msec()
-
-            yield self._update_states([
-                prev_state.copy_and_replace(
-                    last_user_sync_ts=time_now_ms,
-                )
-                for prev_state in prev_states.itervalues()
-            ])
-            self.external_process_last_updated_ms.pop(process_id, None)
-
     @defer.inlineCallbacks
     def current_state_for_user(self, user_id):
         """Get the current presence state for a user.
@@ -592,14 +529,14 @@ class PresenceHandler(object):
             for user_id in user_ids
         }
 
-        missing = [user_id for user_id, state in states.iteritems() if not state]
+        missing = [user_id for user_id, state in states.items() if not state]
         if missing:
             # There are things not in our in memory cache. Lets pull them out of
             # the database.
             res = yield self.store.get_presence_for_users(missing)
             states.update(res)
 
-            missing = [user_id for user_id, state in states.iteritems() if not state]
+            missing = [user_id for user_id, state in states.items() if not state]
             if missing:
                 new = {
                     user_id: UserPresenceState.default(user_id)
@@ -610,6 +547,55 @@ class PresenceHandler(object):
 
         defer.returnValue(states)
 
+    @defer.inlineCallbacks
+    def _get_interested_parties(self, states, calculate_remote_hosts=True):
+        """Given a list of states return which entities (rooms, users, servers)
+        are interested in the given states.
+
+        Returns:
+            3-tuple: `(room_ids_to_states, users_to_states, hosts_to_states)`,
+            with each item being a dict of `entity_name` -> `[UserPresenceState]`
+        """
+        room_ids_to_states = {}
+        users_to_states = {}
+        for state in states:
+            events = yield self.store.get_rooms_for_user(state.user_id)
+            for e in events:
+                room_ids_to_states.setdefault(e.room_id, []).append(state)
+
+            plist = yield self.store.get_presence_list_observers_accepted(state.user_id)
+            for u in plist:
+                users_to_states.setdefault(u, []).append(state)
+
+            # Always notify self
+            users_to_states.setdefault(state.user_id, []).append(state)
+
+        hosts_to_states = {}
+        if calculate_remote_hosts:
+            for room_id, states in room_ids_to_states.items():
+                local_states = filter(lambda s: self.is_mine_id(s.user_id), states)
+                if not local_states:
+                    continue
+
+                users = yield self.store.get_users_in_room(room_id)
+                hosts = set(get_domain_from_id(u) for u in users)
+
+                for host in hosts:
+                    hosts_to_states.setdefault(host, []).extend(local_states)
+
+        for user_id, states in users_to_states.items():
+            local_states = filter(lambda s: self.is_mine_id(s.user_id), states)
+            if not local_states:
+                continue
+
+            host = get_domain_from_id(user_id)
+            hosts_to_states.setdefault(host, []).extend(local_states)
+
+        # TODO: de-dup hosts_to_states, as a single host might have multiple
+        # of same presence
+
+        defer.returnValue((room_ids_to_states, users_to_states, hosts_to_states))
+
     @defer.inlineCallbacks
     def _persist_and_notify(self, states):
         """Persist states in the database, poke the notifier and send to
@@ -617,33 +603,34 @@ class PresenceHandler(object):
         """
         stream_id, max_token = yield self.store.update_presence(states)
 
-        parties = yield get_interested_parties(self.store, states)
-        room_ids_to_states, users_to_states = parties
+        parties = yield self._get_interested_parties(states)
+        room_ids_to_states, users_to_states, hosts_to_states = parties
 
         self.notifier.on_new_event(
             "presence_key", stream_id, rooms=room_ids_to_states.keys(),
-            users=[UserID.from_string(u) for u in users_to_states]
+            users=[UserID.from_string(u) for u in users_to_states.keys()]
         )
 
-        self._push_to_remotes(states)
+        self._push_to_remotes(hosts_to_states)
 
     @defer.inlineCallbacks
     def notify_for_states(self, state, stream_id):
-        parties = yield get_interested_parties(self.store, [state])
-        room_ids_to_states, users_to_states = parties
+        parties = yield self._get_interested_parties([state])
+        room_ids_to_states, users_to_states, hosts_to_states = parties
 
         self.notifier.on_new_event(
             "presence_key", stream_id, rooms=room_ids_to_states.keys(),
-            users=[UserID.from_string(u) for u in users_to_states]
+            users=[UserID.from_string(u) for u in users_to_states.keys()]
         )
 
-    def _push_to_remotes(self, states):
+    def _push_to_remotes(self, hosts_to_states):
         """Sends state updates to remote servers.
 
         Args:
-            states (list(UserPresenceState))
+            hosts_to_states (dict): Mapping `server_name` -> `[UserPresenceState]`
         """
-        self.federation.send_presence(states)
+        for host, states in hosts_to_states.items():
+            self.federation.send_presence(host, states)
 
     @defer.inlineCallbacks
     def incoming_presence(self, origin, content):
@@ -735,7 +722,9 @@ class PresenceHandler(object):
                 for state in updates
             ])
         else:
-            defer.returnValue(updates)
+            defer.returnValue([
+                format_user_presence_state(state, now) for state in updates
+            ])
 
     @defer.inlineCallbacks
     def set_state(self, target_user, state, ignore_status_msg=False):
@@ -780,17 +769,18 @@ class PresenceHandler(object):
         # don't need to send to local clients here, as that is done as part
         # of the event stream/sync.
         # TODO: Only send to servers not already in the room.
+        user_ids = yield self.store.get_users_in_room(room_id)
         if self.is_mine(user):
             state = yield self.current_state_for_user(user.to_string())
 
-            self._push_to_remotes([state])
+            hosts = set(get_domain_from_id(u) for u in user_ids)
+            self._push_to_remotes({host: (state,) for host in hosts})
         else:
-            user_ids = yield self.store.get_users_in_room(room_id)
             user_ids = filter(self.is_mine_id, user_ids)
 
             states = yield self.current_state_for_users(user_ids)
 
-            self._push_to_remotes(states.values())
+            self._push_to_remotes({user.domain: states.values()})
 
     @defer.inlineCallbacks
     def get_presence_list(self, observer_user, accepted=None):
@@ -808,9 +798,6 @@ class PresenceHandler(object):
             as_event=False,
         )
 
-        now = self.clock.time_msec()
-        results[:] = [format_user_presence_state(r, now) for r in results]
-
         is_accepted = {
             row["observed_user_id"]: row["accepted"] for row in presence_list
         }
@@ -863,7 +850,6 @@ class PresenceHandler(object):
             )
 
             state_dict = yield self.get_state(observed_user, as_event=False)
-            state_dict = format_user_presence_state(state_dict, self.clock.time_msec())
 
             self.federation.send_edu(
                 destination=observer_user.domain,
@@ -927,12 +913,11 @@ class PresenceHandler(object):
     def is_visible(self, observed_user, observer_user):
         """Returns whether a user can see another user's presence.
         """
-        observer_room_ids = yield self.store.get_rooms_for_user(
-            observer_user.to_string()
-        )
-        observed_room_ids = yield self.store.get_rooms_for_user(
-            observed_user.to_string()
-        )
+        observer_rooms = yield self.store.get_rooms_for_user(observer_user.to_string())
+        observed_rooms = yield self.store.get_rooms_for_user(observed_user.to_string())
+
+        observer_room_ids = set(r.room_id for r in observer_rooms)
+        observed_room_ids = set(r.room_id for r in observed_rooms)
 
         if observer_room_ids & observed_room_ids:
             defer.returnValue(True)
@@ -997,18 +982,14 @@ def should_notify(old_state, new_state):
     return False
 
 
-def format_user_presence_state(state, now, include_user_id=True):
+def format_user_presence_state(state, now):
     """Convert UserPresenceState to a format that can be sent down to clients
     and to other servers.
-
-    The "user_id" is optional so that this function can be used to format presence
-    updates for client /sync responses and for federation /send requests.
     """
     content = {
         "presence": state.state,
+        "user_id": state.user_id,
     }
-    if include_user_id:
-        content["user_id"] = state.user_id
     if state.last_active_ts:
         content["last_active_ago"] = now - state.last_active_ts
     if state.status_msg and state.state != PresenceState.OFFLINE:
@@ -1047,6 +1028,7 @@ class PresenceEventSource(object):
         # sending down the rare duplicate is not a concern.
 
         with Measure(self.clock, "presence.get_new_events"):
+            user_id = user.to_string()
             if from_key is not None:
                 from_key = int(from_key)
 
@@ -1055,7 +1037,18 @@ class PresenceEventSource(object):
 
             max_token = self.store.get_current_presence_token()
 
-            users_interested_in = yield self._get_interested_in(user, explicit_room_id)
+            plist = yield self.store.get_presence_list_accepted(user.localpart)
+            users_interested_in = set(row["observed_user_id"] for row in plist)
+            users_interested_in.add(user_id)  # So that we receive our own presence
+
+            users_who_share_room = yield self.store.get_users_who_share_room_with_user(
+                user_id
+            )
+            users_interested_in.update(users_who_share_room)
+
+            if explicit_room_id:
+                user_ids = yield self.store.get_users_in_room(explicit_room_id)
+                users_interested_in.update(user_ids)
 
             user_ids_changed = set()
             changed = None
@@ -1083,13 +1076,16 @@ class PresenceEventSource(object):
 
             updates = yield presence.current_state_for_users(user_ids_changed)
 
-        if include_offline:
-            defer.returnValue((updates.values(), max_token))
-        else:
-            defer.returnValue(([
-                s for s in updates.itervalues()
-                if s.state != PresenceState.OFFLINE
-            ], max_token))
+        now = self.clock.time_msec()
+
+        defer.returnValue(([
+            {
+                "type": "m.presence",
+                "content": format_user_presence_state(s, now),
+            }
+            for s in updates.values()
+            if include_offline or s.state != PresenceState.OFFLINE
+        ], max_token))
 
     def get_current_key(self):
         return self.store.get_current_presence_token()
@@ -1097,31 +1093,6 @@ class PresenceEventSource(object):
     def get_pagination_rows(self, user, pagination_config, key):
         return self.get_new_events(user, from_key=None, include_offline=False)
 
-    @cachedInlineCallbacks(num_args=2, cache_context=True)
-    def _get_interested_in(self, user, explicit_room_id, cache_context):
-        """Returns the set of users that the given user should see presence
-        updates for
-        """
-        user_id = user.to_string()
-        plist = yield self.store.get_presence_list_accepted(
-            user.localpart, on_invalidate=cache_context.invalidate,
-        )
-        users_interested_in = set(row["observed_user_id"] for row in plist)
-        users_interested_in.add(user_id)  # So that we receive our own presence
-
-        users_who_share_room = yield self.store.get_users_who_share_room_with_user(
-            user_id, on_invalidate=cache_context.invalidate,
-        )
-        users_interested_in.update(users_who_share_room)
-
-        if explicit_room_id:
-            user_ids = yield self.store.get_users_in_room(
-                explicit_room_id, on_invalidate=cache_context.invalidate,
-            )
-            users_interested_in.update(user_ids)
-
-        defer.returnValue(users_interested_in)
-
 
 def handle_timeouts(user_states, is_mine_fn, syncing_user_ids, now):
     """Checks the presence of users that have timed out and updates as
@@ -1189,10 +1160,7 @@ def handle_timeout(state, is_mine, syncing_user_ids, now):
         # If there are have been no sync for a while (and none ongoing),
         # set presence to offline
         if user_id not in syncing_user_ids:
-            # If the user has done something recently but hasn't synced,
-            # don't set them as offline.
-            sync_or_active = max(state.last_user_sync_ts, state.last_active_ts)
-            if now - sync_or_active > SYNC_ONLINE_TIMEOUT:
+            if now - state.last_user_sync_ts > SYNC_ONLINE_TIMEOUT:
                 state = state.copy_and_replace(
                     state=PresenceState.OFFLINE,
                     status_msg=None,
@@ -1290,66 +1258,3 @@ def handle_update(prev_state, new_state, is_mine, wheel_timer, now):
         persist_and_notify = True
 
     return new_state, persist_and_notify, federation_ping
-
-
-@defer.inlineCallbacks
-def get_interested_parties(store, states):
-    """Given a list of states return which entities (rooms, users)
-    are interested in the given states.
-
-    Args:
-        states (list(UserPresenceState))
-
-    Returns:
-        2-tuple: `(room_ids_to_states, users_to_states)`,
-        with each item being a dict of `entity_name` -> `[UserPresenceState]`
-    """
-    room_ids_to_states = {}
-    users_to_states = {}
-    for state in states:
-        room_ids = yield store.get_rooms_for_user(state.user_id)
-        for room_id in room_ids:
-            room_ids_to_states.setdefault(room_id, []).append(state)
-
-        plist = yield store.get_presence_list_observers_accepted(state.user_id)
-        for u in plist:
-            users_to_states.setdefault(u, []).append(state)
-
-        # Always notify self
-        users_to_states.setdefault(state.user_id, []).append(state)
-
-    defer.returnValue((room_ids_to_states, users_to_states))
-
-
-@defer.inlineCallbacks
-def get_interested_remotes(store, states, state_handler):
-    """Given a list of presence states figure out which remote servers
-    should be sent which.
-
-    All the presence states should be for local users only.
-
-    Args:
-        store (DataStore)
-        states (list(UserPresenceState))
-
-    Returns:
-        Deferred list of ([destinations], [UserPresenceState]), where for
-        each row the list of UserPresenceState should be sent to each
-        destination
-    """
-    hosts_and_states = []
-
-    # First we look up the rooms each user is in (as well as any explicit
-    # subscriptions), then for each distinct room we look up the remote
-    # hosts in those rooms.
-    room_ids_to_states, users_to_states = yield get_interested_parties(store, states)
-
-    for room_id, states in room_ids_to_states.iteritems():
-        hosts = yield state_handler.get_current_hosts_in_room(room_id)
-        hosts_and_states.append((hosts, states))
-
-    for user_id, states in users_to_states.iteritems():
-        host = get_domain_from_id(user_id)
-        hosts_and_states.append(([host], states))
-
-    defer.returnValue(hosts_and_states)