More merging

When a push is rejected by the push gateway then synapse needs to
remove the pusher from the database. However we probably don't want
to do that directly from the slave, so we add an HTTP API to synapse
to remove the pusher from the database.

CHANGES.rst

@@ -1,175 +1,3 @@
-Changes in synapse v0.16.1-r1 (2016-07-08)
-==========================================
-
-THIS IS A CRITICAL SECURITY UPDATE.
-
-This fixes a bug which allowed users' accounts to be accessed by unauthorised
-users.
-
-Changes in synapse v0.16.1 (2016-06-20)
-=======================================
-
-Bug fixes:
-
-* Fix assorted bugs in ``/preview_url`` (PR #872)
-* Fix TypeError when setting unicode passwords (PR #873)
-
-
-Performance improvements:
-
-* Turn ``use_frozen_events`` off by default (PR #877)
-* Disable responding with canonical json for federation (PR #878)
-
-
-Changes in synapse v0.16.1-rc1 (2016-06-15)
-===========================================
-
-Features: None
-
-Changes:
-
-* Log requester for ``/publicRoom`` endpoints when possible (PR #856)
-* 502 on ``/thumbnail`` when can't connect to remote server (PR #862)
-* Linearize fetching of gaps on incoming events (PR #871)
-
-
-Bugs fixes:
-
-* Fix bug where rooms where marked as published by default (PR #857)
-* Fix bug where joining room with an event with invalid sender (PR #868)
-* Fix bug where backfilled events were sent down sync streams (PR #869)
-* Fix bug where outgoing connections could wedge indefinitely, causing push
-  notifications to be unreliable (PR #870)
-
-
-Performance improvements:
-
-* Improve ``/publicRooms`` performance(PR #859)
-
-
-Changes in synapse v0.16.0 (2016-06-09)
-=======================================
-
-NB: As of v0.14 all AS config files must have an ID field.
-
-
-Bug fixes:
-
-* Don't make rooms published by default (PR #857)
-
-Changes in synapse v0.16.0-rc2 (2016-06-08)
-===========================================
-
-Features:
-
-* Add configuration option for tuning GC via ``gc.set_threshold`` (PR #849)
-
-Changes:
-
-* Record metrics about GC (PR #771, #847, #852)
-* Add metric counter for number of persisted events (PR #841)
-
-Bug fixes:
-
-* Fix 'From' header in email notifications (PR #843)
-* Fix presence where timeouts were not being fired for the first 8h after
-  restarts (PR #842)
-* Fix bug where synapse sent malformed transactions to AS's when retrying
-  transactions (Commits 310197b, 8437906)
-
-Performance improvements:
-
-* Remove event fetching from DB threads (PR #835)
-* Change the way we cache events (PR #836)
-* Add events to cache when we persist them (PR #840)
-
-
-Changes in synapse v0.16.0-rc1 (2016-06-03)
-===========================================
-
-Version 0.15 was not released. See v0.15.0-rc1 below for additional changes.
-
-Features:
-
-* Add email notifications for missed messages (PR #759, #786, #799, #810, #815,
-  #821)
-* Add a ``url_preview_ip_range_whitelist`` config param (PR #760)
-* Add /report endpoint (PR #762)
-* Add basic ignore user API (PR #763)
-* Add an openidish mechanism for proving that you own a given user_id (PR #765)
-* Allow clients to specify a server_name to avoid 'No known servers' (PR #794)
-* Add secondary_directory_servers option to fetch room list from other servers
-  (PR #808, #813)
-
-Changes:
-
-* Report per request metrics for all of the things using request_handler (PR
-  #756)
-* Correctly handle ``NULL`` password hashes from the database (PR #775)
-* Allow receipts for events we haven't seen in the db (PR #784)
-* Make synctl read a cache factor from config file (PR #785)
-* Increment badge count per missed convo, not per msg (PR #793)
-* Special case m.room.third_party_invite event auth to match invites (PR #814)
-
-
-Bug fixes:
-
-* Fix typo in event_auth servlet path (PR #757)
-* Fix password reset (PR #758)
-
-
-Performance improvements:
-
-* Reduce database inserts when sending transactions (PR #767)
-* Queue events by room for persistence (PR #768)
-* Add cache to ``get_user_by_id`` (PR #772)
-* Add and use ``get_domain_from_id`` (PR #773)
-* Use tree cache for ``get_linearized_receipts_for_room`` (PR #779)
-* Remove unused indices (PR #782)
-* Add caches to ``bulk_get_push_rules*`` (PR #804)
-* Cache ``get_event_reference_hashes`` (PR #806)
-* Add ``get_users_with_read_receipts_in_room`` cache (PR #809)
-* Use state to calculate ``get_users_in_room`` (PR #811)
-* Load push rules in storage layer so that they get cached (PR #825)
-* Make ``get_joined_hosts_for_room`` use get_users_in_room (PR #828)
-* Poke notifier on next reactor tick (PR #829)
-* Change CacheMetrics to be quicker (PR #830)
-
-
-Changes in synapse v0.15.0-rc1 (2016-04-26)
-===========================================
-
-Features:
-
-* Add login support for Javascript Web Tokens, thanks to Niklas Riekenbrauck
-  (PR #671,#687)
-* Add URL previewing support (PR #688)
-* Add login support for LDAP, thanks to Christoph Witzany (PR #701)
-* Add GET endpoint for pushers (PR #716)
-
-Changes:
-
-* Never notify for member events (PR #667)
-* Deduplicate identical ``/sync`` requests (PR #668)
-* Require user to have left room to forget room (PR #673)
-* Use DNS cache if within TTL (PR #677)
-* Let users see their own leave events (PR #699)
-* Deduplicate membership changes (PR #700)
-* Increase performance of pusher code (PR #705)
-* Respond with error status 504 if failed to talk to remote server (PR #731)
-* Increase search performance on postgres (PR #745)
-
-Bug fixes:
-
-* Fix bug where disabling all notifications still resulted in push (PR #678)
-* Fix bug where users couldn't reject remote invites if remote refused (PR #691)
-* Fix bug where synapse attempted to backfill from itself (PR #693)
-* Fix bug where profile information was not correctly added when joining remote
-  rooms (PR #703)
-* Fix bug where register API required incorrect key name for AS registration
-  (PR #727)
-
-
 Changes in synapse v0.14.0 (2016-03-30)
 =======================================
 
@@ -683,7 +511,7 @@ Configuration:
 
 * Add support for changing the bind host of the metrics listener via the
   ``metrics_bind_host`` option.
-
+ 
 
 Changes in synapse v0.9.0-r5 (2015-05-21)
 =========================================
@@ -1025,7 +853,7 @@ See UPGRADE for information about changes to the client server API, including
 breaking backwards compatibility with VoIP calls and registration API.
 
 Homeserver:
- * When a user changes their displayname or avatar the server will now update
+ * When a user changes their displayname or avatar the server will now update 
    all their join states to reflect this.
  * The server now adds "age" key to events to indicate how old they are. This
    is clock independent, so at no point does any server or webclient have to
@@ -1083,7 +911,7 @@ Changes in synapse 0.2.2 (2014-09-06)
 =====================================
 
 Homeserver:
- * When the server returns state events it now also includes the previous
+ * When the server returns state events it now also includes the previous 
    content.
  * Add support for inviting people when creating a new room.
  * Make the homeserver inform the room via `m.room.aliases` when a new alias
@@ -1095,7 +923,7 @@ Webclient:
  * Handle `m.room.aliases` events.
  * Asynchronously send messages and show a local echo.
  * Inform the UI when a message failed to send.
- * Only autoscroll on receiving a new message if the user was already at the
+ * Only autoscroll on receiving a new message if the user was already at the 
    bottom of the screen.
  * Add support for ban/kick reasons.
 

MANIFEST.in

@@ -11,7 +11,6 @@ recursive-include synapse/storage/schema *.sql
 recursive-include synapse/storage/schema *.py
 
 recursive-include docs *
-recursive-include res *
 recursive-include scripts *
 recursive-include scripts-dev *
 recursive-include tests *.py

README.rst

@@ -58,13 +58,12 @@ the spec in the context of a codebase and let you run your own homeserver and
 generally help bootstrap the ecosystem.
 
 In Matrix, every user runs one or more Matrix clients, which connect through to
-a Matrix homeserver. The homeserver stores all their personal chat history and
-user account information - much as a mail client connects through to an
-IMAP/SMTP server. Just like email, you can either run your own Matrix
-homeserver and control and own your own communications and history or use one
-hosted by someone else (e.g. matrix.org) - there is no single point of control
-or mandatory service provider in Matrix, unlike WhatsApp, Facebook, Hangouts,
-etc.
+a Matrix homeserver which stores all their personal chat history and user
+account information - much as a mail client connects through to an IMAP/SMTP
+server. Just like email, you can either run your own Matrix homeserver and
+control and own your own communications and history or use one hosted by
+someone else (e.g. matrix.org) - there is no single point of control or
+mandatory service provider in Matrix, unlike WhatsApp, Facebook, Hangouts, etc.
 
 Synapse ships with two basic demo Matrix clients: webclient (a basic group chat
 web client demo implemented in AngularJS) and cmdclient (a basic Python
@@ -618,7 +617,7 @@ Building internal API documentation::
 
 
 
-Help!! Synapse eats all my RAM!
+Halp!! Synapse eats all my RAM!
 ===============================
 
 Synapse's architecture is quite RAM hungry currently - we deliberately

contrib/systemd/synapse.service

@@ -9,7 +9,6 @@ Description=Synapse Matrix homeserver
 Type=simple
 User=synapse
 Group=synapse
-EnvironmentFile=-/etc/sysconfig/synapse
 WorkingDirectory=/var/lib/synapse
 ExecStart=/usr/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/synapse/homeserver.yaml --log-config=/etc/synapse/log_config.yaml
 

docs/application_services.rst

@@ -32,4 +32,5 @@ The format of the AS configuration file is as follows:
 
 See the spec_ for further details on how application services work.
 
-.. _spec: https://matrix.org/docs/spec/application_service/unstable.html
+.. _spec: https://github.com/matrix-org/matrix-doc/blob/master/specification/25_application_service_api.rst#application-service-api
+

docs/code_style.rst

@@ -43,10 +43,7 @@ Basically, PEP8
   together, or want to deliberately extend or preserve vertical/horizontal
   space)
 
-Comments should follow the `google code style <http://google.github.io/styleguide/pyguide.html?showone=Comments#Comments>`_.
-This is so that we can generate documentation with 
-`sphinx <http://sphinxcontrib-napoleon.readthedocs.org/en/latest/>`_. See the
-`examples <http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html>`_
-in the sphinx documentation.
+Comments should follow the google code style. This is so that we can generate
+documentation with sphinx (http://sphinxcontrib-napoleon.readthedocs.org/en/latest/) 
 
 Code should pass pep8 --max-line-length=100 without any warnings.

docs/log_contexts.rst

@@ -1,10 +0,0 @@
-What do I do about "Unexpected logging context" debug log-lines everywhere?
-
-<Mjark> The logging context lives in thread local storage
-<Mjark> Sometimes it gets out of sync with what it should actually be, usually because something scheduled something to run on the reactor without preserving the logging context. 
-<Matthew> what is the impact of it getting out of sync? and how and when should we preserve log context?
-<Mjark> The impact is that some of the CPU and database metrics will be under-reported, and some log lines will be mis-attributed.
-<Mjark> It should happen auto-magically in all the APIs that do IO or otherwise defer to the reactor.
-<Erik> Mjark: the other place is if we branch, e.g. using defer.gatherResults
-
-Unanswered: how and when should we preserve log context?

docs/turn-howto.rst

@@ -9,35 +9,31 @@ the Home Server to generate credentials that are valid for use on the TURN
 server through the use of a secret shared between the Home Server and the
 TURN server.
 
-This document describes how to install coturn
-(https://github.com/coturn/coturn) which also supports the TURN REST API,
+This document described how to install coturn
+(https://code.google.com/p/coturn/) which also supports the TURN REST API,
 and integrate it with synapse.
 
 coturn Setup
 ============
 
-You may be able to setup coturn via your package manager,  or set it up manually using the usual ``configure, make, make install`` process.  
-
  1. Check out coturn::
- 
-      git clone https://github.com/coturn/coturn.git coturn
+      svn checkout http://coturn.googlecode.com/svn/trunk/ coturn
       cd coturn
 
  2. Configure it::
- 
       ./configure
 
-    You may need to install ``libevent2``: if so, you should do so
+    You may need to install libevent2: if so, you should do so
     in the way recommended by your operating system.
     You can ignore warnings about lack of database support: a
     database is unnecessary for this purpose.
 
  3. Build and install it::
- 
       make
       make install
 
- 4. Create or edit the config file in ``/etc/turnserver.conf``. The relevant
+ 4. Make a config file in /etc/turnserver.conf. You can customise
+    a config file from turnserver.conf.default. The relevant
     lines, with example values, are::
 
       lt-cred-mech
@@ -45,7 +41,7 @@ You may be able to setup coturn via your package manager,  or set it up manually
       static-auth-secret=[your secret key here]
       realm=turn.myserver.org
 
-    See turnserver.conf for explanations of the options.
+    See turnserver.conf.default for explanations of the options.
     One way to generate the static-auth-secret is with pwgen::
 
        pwgen -s 64 1
@@ -58,7 +54,6 @@ You may be able to setup coturn via your package manager,  or set it up manually
     import your private key and certificate.
 
  7. Start the turn server::
- 
        bin/turnserver -o
 
 

jenkins-dendron-postgres.sh

@@ -1,87 +0,0 @@
-#!/bin/bash
-
-set -eux
-
-: ${WORKSPACE:="$(pwd)"}
-
-export PYTHONDONTWRITEBYTECODE=yep
-export SYNAPSE_CACHE_FACTOR=1
-
-# Output test results as junit xml
-export TRIAL_FLAGS="--reporter=subunit"
-export TOXSUFFIX="| subunit-1to2 | subunit2junitxml --no-passthrough --output-to=results.xml"
-# Write coverage reports to a separate file for each process
-export COVERAGE_OPTS="-p"
-export DUMP_COVERAGE_COMMAND="coverage help"
-
-# Output flake8 violations to violations.flake8.log
-# Don't exit with non-0 status code on Jenkins,
-# so that the build steps continue and a later step can decided whether to
-# UNSTABLE or FAILURE this build.
-export PEP8SUFFIX="--output-file=violations.flake8.log || echo flake8 finished with status code \$?"
-
-rm .coverage* || echo "No coverage files to remove"
-
-tox --notest -e py27
-
-TOX_BIN=$WORKSPACE/.tox/py27/bin
-python synapse/python_dependencies.py | xargs -n1 $TOX_BIN/pip install
-$TOX_BIN/pip install psycopg2
-$TOX_BIN/pip install lxml
-
-: ${GIT_BRANCH:="origin/$(git rev-parse --abbrev-ref HEAD)"}
-
-if [[ ! -e .dendron-base ]]; then
-    git clone https://github.com/matrix-org/dendron.git .dendron-base --mirror
-else
-    (cd .dendron-base; git fetch -p)
-fi
-
-rm -rf dendron
-git clone .dendron-base dendron --shared
-cd dendron
-
-: ${GOPATH:=${WORKSPACE}/.gopath}
-if [[ "${GOPATH}" != *:* ]]; then
-  mkdir -p "${GOPATH}"
-  export PATH="${GOPATH}/bin:${PATH}"
-fi
-export GOPATH
-
-git checkout "${GIT_BRANCH}" || (echo >&2 "No ref ${GIT_BRANCH} found, falling back to develop" ; git checkout develop)
-
-go get github.com/constabulary/gb/...
-gb generate
-gb build
-
-cd ..
-
-
-if [[ ! -e .sytest-base ]]; then
-  git clone https://github.com/matrix-org/sytest.git .sytest-base --mirror
-else
-  (cd .sytest-base; git fetch -p)
-fi
-
-rm -rf sytest
-git clone .sytest-base sytest --shared
-cd sytest
-
-git checkout "${GIT_BRANCH}" || (echo >&2 "No ref ${GIT_BRANCH} found, falling back to develop" ; git checkout develop)
-
-: ${PORT_BASE:=8000}
-: ${PORT_COUNT=20}
-
-./jenkins/prep_sytest_for_postgres.sh
-
-mkdir -p var
-
-echo >&2 "Running sytest with PostgreSQL";
-./jenkins/install_and_run.sh --python $TOX_BIN/python \
-                             --synapse-directory $WORKSPACE \
-                             --dendron $WORKSPACE/dendron/bin/dendron \
-                             --pusher \
-                             --synchrotron \
-                             --port-range ${PORT_BASE}:$((PORT_BASE+PORT_COUNT-1))
-
-cd ..

jenkins-postgres.sh

@@ -44,7 +44,6 @@ cd sytest
 git checkout "${GIT_BRANCH}" || (echo >&2 "No ref ${GIT_BRANCH} found, falling back to develop" ; git checkout develop)
 
 : ${PORT_BASE:=8000}
-: ${PORT_COUNT=20}
 
 ./jenkins/prep_sytest_for_postgres.sh
 
@@ -52,7 +51,7 @@ echo >&2 "Running sytest with PostgreSQL";
 ./jenkins/install_and_run.sh --coverage \
                              --python $TOX_BIN/python \
                              --synapse-directory $WORKSPACE \
-                             --port-range ${PORT_BASE}:$((PORT_BASE+PORT_COUNT-1)) \
+                             --port-base $PORT_BASE
 
 cd ..
 cp sytest/.coverage.* .

jenkins-sqlite.sh

@@ -41,12 +41,11 @@ cd sytest
 
 git checkout "${GIT_BRANCH}" || (echo >&2 "No ref ${GIT_BRANCH} found, falling back to develop" ; git checkout develop)
 
-: ${PORT_COUNT=20}
-: ${PORT_BASE:=8000}
+: ${PORT_BASE:=8500}
 ./jenkins/install_and_run.sh --coverage \
                              --python $TOX_BIN/python \
                              --synapse-directory $WORKSPACE \
-                             --port-range ${PORT_BASE}:$((PORT_BASE+PORT_COUNT-1)) \
+                             --port-base $PORT_BASE
 
 cd ..
 cp sytest/.coverage.* .

res/templates/mail-Vector.css

@@ -1,7 +0,0 @@
-.header {
-    border-bottom: 4px solid #e4f7ed ! important;
-}
-
-.notif_link a, .footer a {
-    color: #76CFA6 ! important;
-}

res/templates/mail.css

@@ -1,156 +0,0 @@
-body {
-    margin: 0px;
-}
-
-pre, code {
-    word-break: break-word;
-    white-space: pre-wrap;
-}
-
-#page {
-    font-family: 'Open Sans', Helvetica, Arial, Sans-Serif;
-    font-color: #454545;
-    font-size: 12pt;
-    width: 100%;
-    padding: 20px;
-}
-
-#inner {
-    width: 640px;
-}
-
-.header {
-    width: 100%;
-    height: 87px;
-    color: #454545;
-    border-bottom: 4px solid #e5e5e5;
-}
-
-.logo {
-    text-align: right;
-    margin-left: 20px;
-}
-
-.salutation {
-    padding-top: 10px;
-    font-weight: bold;
-}
-
-.summarytext {
-}
-
-.room {
-    width: 100%;
-    color: #454545;
-    border-bottom: 1px solid #e5e5e5;
-}
-
-.room_header td {
-    padding-top: 38px;
-    padding-bottom: 10px;
-    border-bottom: 1px solid #e5e5e5;
-}
-
-.room_name {
-    vertical-align: middle;
-    font-size: 18px;
-    font-weight: bold;
-}
-
-.room_header h2 {
-    margin-top: 0px;
-    margin-left: 75px;
-    font-size: 20px;
-}
-
-.room_avatar {
-    width: 56px;
-    line-height: 0px;
-    text-align: center;
-    vertical-align: middle;
-}
-
-.room_avatar img {
-    width: 48px;
-    height: 48px;
-    object-fit: cover;
-    border-radius: 24px;
-}
-
-.notif {
-    border-bottom: 1px solid #e5e5e5;
-    margin-top: 16px;
-    padding-bottom: 16px;
-}
-
-.historical_message .sender_avatar {
-    opacity: 0.3;
-}
-
-/* spell out opacity and historical_message class names for Outlook aka Word */
-.historical_message .sender_name {
-    color: #e3e3e3;
-}
-
-.historical_message .message_time {
-    color: #e3e3e3;
-}
-
-.historical_message .message_body {
-    color: #c7c7c7;
-}
-
-.historical_message td,
-.message td {
-    padding-top: 10px;
-}
-
-.sender_avatar {
-    width: 56px;
-    text-align: center;
-    vertical-align: top;
-}
-
-.sender_avatar img {
-    margin-top: -2px;
-    width: 32px;
-    height: 32px;
-    border-radius: 16px;
-}
-
-.sender_name  {
-    display: inline;
-    font-size: 13px;
-    color: #a2a2a2;
-}
-
-.message_time  {
-    text-align: right;
-    width: 100px;
-    font-size: 11px;
-    color: #a2a2a2;
-}
-
-.message_body {
-}
-
-.notif_link td {
-    padding-top: 10px;
-    padding-bottom: 10px;
-    font-weight: bold;
-}
-
-.notif_link a, .footer a {
-    color: #454545;
-    text-decoration: none;
-}
-
-.debug {
-    font-size: 10px;
-    color: #888;
-}
-
-.footer {
-    margin-top: 20px;
-    text-align: center;
-}

res/templates/notif.html

@@ -1,45 +0,0 @@
-{% for message in notif.messages %}
-    <tr class="{{ "historical_message" if message.is_historical else "message" }}">
-        <td class="sender_avatar">
-            {% if loop.index0 == 0 or notif.messages[loop.index0 - 1].sender_name != notif.messages[loop.index0].sender_name %}
-                {% if message.sender_avatar_url %}
-                    <img alt="" class="sender_avatar" src="{{ message.sender_avatar_url|mxc_to_http(32,32) }}"  />
-                {% else %}
-                    {% if message.sender_hash % 3 == 0 %}
-                        <img class="sender_avatar" src="https://vector.im/beta/img/76cfa6.png"  />
-                    {% elif message.sender_hash % 3 == 1 %}
-                        <img class="sender_avatar" src="https://vector.im/beta/img/50e2c2.png"  />
-                    {% else %}
-                        <img class="sender_avatar" src="https://vector.im/beta/img/f4c371.png"  />
-                    {% endif %}
-                {% endif %}
-            {% endif %}
-        </td>
-        <td class="message_contents">
-            {% if loop.index0 == 0 or notif.messages[loop.index0 - 1].sender_name != notif.messages[loop.index0].sender_name %}
-                <div class="sender_name">{% if message.msgtype == "m.emote" %}*{% endif %} {{ message.sender_name }}</div>
-            {% endif %}
-            <div class="message_body">
-                {% if message.msgtype == "m.text" %}
-                    {{ message.body_text_html }}
-                {% elif message.msgtype == "m.emote" %}
-                    {{ message.body_text_html }}
-                {% elif message.msgtype == "m.notice" %}
-                    {{ message.body_text_html }}
-                {% elif message.msgtype == "m.image" %}
-                    <img src="{{ message.image_url|mxc_to_http(640, 480, scale) }}" />
-                {% elif message.msgtype == "m.file" %}
-                    <span class="filename">{{ message.body_text_plain }}</span>
-                {% endif %}
-            </div>
-        </td>
-        <td class="message_time">{{ message.ts|format_ts("%H:%M") }}</td>
-    </tr>
-{% endfor %}
-<tr class="notif_link">
-    <td></td>
-    <td>
-        <a href="{{ notif.link }}">View {{ room.title }}</a>
-    </td>
-    <td></td>
-</tr>

res/templates/notif.txt

@@ -1,16 +0,0 @@
-{% for message in notif.messages %}
-{% if message.msgtype == "m.emote" %}* {% endif %}{{ message.sender_name }} ({{ message.ts|format_ts("%H:%M") }})
-{% if message.msgtype == "m.text" %}
-{{ message.body_text_plain }}
-{% elif message.msgtype == "m.emote" %}
-{{ message.body_text_plain }}
-{% elif message.msgtype == "m.notice" %}
-{{ message.body_text_plain }}
-{% elif message.msgtype == "m.image" %}
-{{ message.body_text_plain }}
-{% elif message.msgtype == "m.file" %}
-{{ message.body_text_plain }}
-{% endif %}
-{% endfor %}
-
-View {{ room.title }} at {{ notif.link }}

res/templates/notif_mail.html

@@ -1,53 +0,0 @@
-<!doctype html>
-<html lang="en">
-    <head>
-        <style type="text/css">
-            {% include 'mail.css' without context %}
-            {% include "mail-%s.css" % app_name ignore missing without context %}
-        </style>
-    </head>
-    <body>
-        <table id="page">
-            <tr>
-                <td> </td>
-                <td id="inner">
-                    <table class="header">
-                        <tr>
-                            <td>
-                                <div class="salutation">Hi {{ user_display_name }},</div>
-                                <div class="summarytext">{{ summary_text }}</div>
-                            </td>
-                            <td class="logo">
-                                {% if app_name == "Vector" %}
-                                    <img src="http://matrix.org/img/vector-logo-email.png" width="64" height="83" alt="[Vector]"/>
-                                {% else %}
-                                    <img src="http://matrix.org/img/matrix-120x51.png" width="120" height="51" alt="[matrix]"/>
-                                {% endif %}
-                            </td>
-                        </tr>
-                    </table>
-                    {% for room in rooms %}
-                        {% include 'room.html' with context %}
-                    {% endfor %}
-                    <div class="footer">
-                        <a href="{{ unsubscribe_link }}">Unsubscribe</a>
-                        <br/>
-                        <br/>
-                        <div class="debug">
-                            Sending email at {{ reason.now|format_ts("%c") }} due to activity in room {{ reason.room_name }} because
-                            an event was received at {{ reason.received_at|format_ts("%c") }}
-                            which is more than {{ "%.1f"|format(reason.delay_before_mail_ms / (60*1000)) }} ({{ reason.delay_before_mail_ms }}) mins ago,
-                            {% if reason.last_sent_ts %}
-                                and the last time we sent a mail for this room was {{ reason.last_sent_ts|format_ts("%c") }},
-                                which is more than {{ "%.1f"|format(reason.throttle_ms / (60*1000)) }} (current throttle_ms) mins ago.
-                            {% else %}
-                                and we don't have a last time we sent a mail for this room.
-                            {% endif %}
-                        </div>
-                    </div>
-                </td>
-                <td> </td>
-            </tr>
-        </table>
-    </body>
-</html>

res/templates/notif_mail.txt

@@ -1,10 +0,0 @@
-Hi {{ user_display_name }},
-
-{{ summary_text }}
-
-{% for room in rooms %}
-{% include 'room.txt' with context %}
-{% endfor %}
-
-You can disable these notifications at {{ unsubscribe_link }}
-

res/templates/room.html

@@ -1,33 +0,0 @@
-<table class="room">
-    <tr class="room_header">
-        <td class="room_avatar">
-            {% if room.avatar_url %}
-                <img alt="" src="{{ room.avatar_url|mxc_to_http(48,48) }}" />
-            {% else %}
-                {% if room.hash % 3 == 0 %}
-                    <img alt="" src="https://vector.im/beta/img/76cfa6.png"  />
-                {% elif room.hash % 3 == 1 %}
-                    <img alt="" src="https://vector.im/beta/img/50e2c2.png"  />
-                {% else %}
-                    <img alt="" src="https://vector.im/beta/img/f4c371.png"  />
-                {% endif %}
-            {% endif %}
-        </td>
-        <td class="room_name" colspan="2">
-            {{ room.title }}
-        </td>
-    </tr>
-    {% if room.invite %}
-        <tr>
-            <td></td>
-            <td>
-                <a href="{{ room.link }}">Join the conversation.</a>
-            </td>
-            <td></td>
-        </tr>
-    {% else %}
-        {% for notif in room.notifs %}
-            {% include 'notif.html' with context %}
-        {% endfor %}
-    {% endif %}
-</table>

res/templates/room.txt

@@ -1,9 +0,0 @@
-{{ room.title }}
-
-{% if room.invite %}
-    You've been invited, join at {{ room.link }}
-{% else %}
-    {% for notif in room.notifs %}
-        {% include 'notif.txt' with context %}
-    {% endfor %}
-{% endif %}

scripts/hash_password

@@ -1,16 +1,10 @@
 #!/usr/bin/env python
 
 import argparse
-
-import sys
-
 import bcrypt
 import getpass
 
-import yaml
-
 bcrypt_rounds=12
-password_pepper = ""
 
 def prompt_for_pass():
     password = getpass.getpass("Password: ")
@@ -34,22 +28,12 @@ if __name__ == "__main__":
         default=None,
         help="New password for user. Will prompt if omitted.",
     )
-    parser.add_argument(
-        "-c", "--config",
-        type=argparse.FileType('r'),
-        help="Path to server config file. Used to read in bcrypt_rounds and password_pepper.",
-    )
 
     args = parser.parse_args()
-    if "config" in args and args.config:
-        config = yaml.safe_load(args.config)
-        bcrypt_rounds = config.get("bcrypt_rounds", bcrypt_rounds)
-        password_config = config.get("password_config", {})
-        password_pepper = password_config.get("pepper", password_pepper)
     password = args.password
 
     if not password:
         password = prompt_for_pass()
 
-    print bcrypt.hashpw(password + password_pepper, bcrypt.gensalt(bcrypt_rounds))
+    print bcrypt.hashpw(password, bcrypt.gensalt(bcrypt_rounds))
 

scripts/register_new_matrix_user

@@ -25,26 +25,18 @@ import urllib2
 import yaml
 
 
-def request_registration(user, password, server_location, shared_secret, admin=False):
+def request_registration(user, password, server_location, shared_secret):
     mac = hmac.new(
         key=shared_secret,
+        msg=user,
         digestmod=hashlib.sha1,
-    )
-
-    mac.update(user)
-    mac.update("\x00")
-    mac.update(password)
-    mac.update("\x00")
-    mac.update("admin" if admin else "notadmin")
-
-    mac = mac.hexdigest()
+    ).hexdigest()
 
     data = {
         "user": user,
         "password": password,
         "mac": mac,
         "type": "org.matrix.login.shared_secret",
-        "admin": admin,
     }
 
     server_location = server_location.rstrip("/")
@@ -76,7 +68,7 @@ def request_registration(user, password, server_location, shared_secret, admin=F
         sys.exit(1)
 
 
-def register_new_user(user, password, server_location, shared_secret, admin):
+def register_new_user(user, password, server_location, shared_secret):
     if not user:
         try:
             default_user = getpass.getuser()
@@ -107,14 +99,7 @@ def register_new_user(user, password, server_location, shared_secret, admin):
             print "Passwords do not match"
             sys.exit(1)
 
-    if not admin:
-        admin = raw_input("Make admin [no]: ")
-        if admin in ("y", "yes", "true"):
-            admin = True
-        else:
-            admin = False
-
-    request_registration(user, password, server_location, shared_secret, bool(admin))
+    request_registration(user, password, server_location, shared_secret)
 
 
 if __name__ == "__main__":
@@ -134,11 +119,6 @@ if __name__ == "__main__":
         default=None,
         help="New password for user. Will prompt if omitted.",
     )
-    parser.add_argument(
-        "-a", "--admin",
-        action="store_true",
-        help="Register new user as an admin. Will prompt if omitted.",
-    )
 
     group = parser.add_mutually_exclusive_group(required=True)
     group.add_argument(
@@ -171,4 +151,4 @@ if __name__ == "__main__":
     else:
         secret = args.shared_secret
 
-    register_new_user(args.user, args.password, args.server_url, secret, args.admin)
+    register_new_user(args.user, args.password, args.server_url, secret)

scripts/synapse_port_db

@@ -214,10 +214,6 @@ class Porter(object):
 
         self.progress.add_table(table, postgres_size, table_size)
 
-        if table == "event_search":
-            yield self.handle_search_table(postgres_size, table_size, next_chunk)
-            return
-
         select = (
             "SELECT rowid, * FROM %s WHERE rowid >= ? ORDER BY rowid LIMIT ?"
             % (table,)
@@ -236,19 +232,51 @@ class Porter(object):
             if rows:
                 next_chunk = rows[-1][0] + 1
 
-                self._convert_rows(table, headers, rows)
+                if table == "event_search":
+                    # We have to treat event_search differently since it has a
+                    # different structure in the two different databases.
+                    def insert(txn):
+                        sql = (
+                            "INSERT INTO event_search (event_id, room_id, key, sender, vector)"
+                            " VALUES (?,?,?,?,to_tsvector('english', ?))"
+                        )
 
-                def insert(txn):
-                    self.postgres_store.insert_many_txn(
-                        txn, table, headers[1:], rows
-                    )
+                        rows_dict = [
+                            dict(zip(headers, row))
+                            for row in rows
+                        ]
 
-                    self.postgres_store._simple_update_one_txn(
-                        txn,
-                        table="port_from_sqlite3",
-                        keyvalues={"table_name": table},
-                        updatevalues={"rowid": next_chunk},
-                    )
+                        txn.executemany(sql, [
+                            (
+                                row["event_id"],
+                                row["room_id"],
+                                row["key"],
+                                row["sender"],
+                                row["value"],
+                            )
+                            for row in rows_dict
+                        ])
+
+                        self.postgres_store._simple_update_one_txn(
+                            txn,
+                            table="port_from_sqlite3",
+                            keyvalues={"table_name": table},
+                            updatevalues={"rowid": next_chunk},
+                        )
+                else:
+                    self._convert_rows(table, headers, rows)
+
+                    def insert(txn):
+                        self.postgres_store.insert_many_txn(
+                            txn, table, headers[1:], rows
+                        )
+
+                        self.postgres_store._simple_update_one_txn(
+                            txn,
+                            table="port_from_sqlite3",
+                            keyvalues={"table_name": table},
+                            updatevalues={"rowid": next_chunk},
+                        )
 
                 yield self.postgres_store.execute(insert)
 
@@ -258,73 +286,6 @@ class Porter(object):
             else:
                 return
 
-    @defer.inlineCallbacks
-    def handle_search_table(self, postgres_size, table_size, next_chunk):
-        select = (
-            "SELECT es.rowid, es.*, e.origin_server_ts, e.stream_ordering"
-            " FROM event_search as es"
-            " INNER JOIN events AS e USING (event_id, room_id)"
-            " WHERE es.rowid >= ?"
-            " ORDER BY es.rowid LIMIT ?"
-        )
-
-        while True:
-            def r(txn):
-                txn.execute(select, (next_chunk, self.batch_size,))
-                rows = txn.fetchall()
-                headers = [column[0] for column in txn.description]
-
-                return headers, rows
-
-            headers, rows = yield self.sqlite_store.runInteraction("select", r)
-
-            if rows:
-                next_chunk = rows[-1][0] + 1
-
-                # We have to treat event_search differently since it has a
-                # different structure in the two different databases.
-                def insert(txn):
-                    sql = (
-                        "INSERT INTO event_search (event_id, room_id, key,"
-                        " sender, vector, origin_server_ts, stream_ordering)"
-                        " VALUES (?,?,?,?,to_tsvector('english', ?),?,?)"
-                    )
-
-                    rows_dict = [
-                        dict(zip(headers, row))
-                        for row in rows
-                    ]
-
-                    txn.executemany(sql, [
-                        (
-                            row["event_id"],
-                            row["room_id"],
-                            row["key"],
-                            row["sender"],
-                            row["value"],
-                            row["origin_server_ts"],
-                            row["stream_ordering"],
-                        )
-                        for row in rows_dict
-                    ])
-
-                    self.postgres_store._simple_update_one_txn(
-                        txn,
-                        table="port_from_sqlite3",
-                        keyvalues={"table_name": "event_search"},
-                        updatevalues={"rowid": next_chunk},
-                    )
-
-                yield self.postgres_store.execute(insert)
-
-                postgres_size += len(rows)
-
-                self.progress.update("event_search", postgres_size)
-
-            else:
-                return
-
-
     def setup_db(self, db_config, database_engine):
         db_conn = database_engine.module.connect(
             **{

synapse/__init__.py

@@ -16,4 +16,4 @@
 """ This is a reference implementation of a Matrix home server.
 """
 
-__version__ = "0.16.1-r1"
+__version__ = "0.14.0"

synapse/api/auth.py

@@ -13,6 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+"""This module contains classes for authenticating the user."""
 from canonicaljson import encode_canonical_json
 from signedjson.key import decode_verify_key_bytes
 from signedjson.sign import verify_signed_json, SignatureVerifyException
@@ -21,7 +22,7 @@ from twisted.internet import defer
 
 from synapse.api.constants import EventTypes, Membership, JoinRules
 from synapse.api.errors import AuthError, Codes, SynapseError, EventSizeError
-from synapse.types import Requester, UserID, get_domain_from_id
+from synapse.types import Requester, RoomID, UserID, EventID
 from synapse.util.logutils import log_function
 from synapse.util.logcontext import preserve_context_over_fn
 from synapse.util.metrics import Measure
@@ -41,20 +42,13 @@ AuthEventTypes = (
 
 
 class Auth(object):
-    """
-    FIXME: This class contains a mix of functions for authenticating users
-    of our client-server API and authenticating events added to room graphs.
-    """
+
     def __init__(self, hs):
         self.hs = hs
         self.clock = hs.get_clock()
         self.store = hs.get_datastore()
         self.state = hs.get_state_handler()
         self.TOKEN_NOT_FOUND_HTTP_STATUS = 401
-        # Docs for these currently lives at
-        # https://github.com/matrix-org/matrix-doc/blob/master/drafts/macaroons_caveats.rst
-        # In addition, we have type == delete_pusher which grants access only to
-        # delete pushers.
         self._KNOWN_CAVEAT_PREFIXES = set([
             "gen = ",
             "guest = ",
@@ -97,8 +91,8 @@ class Auth(object):
                     "Room %r does not exist" % (event.room_id,)
                 )
 
-            creating_domain = get_domain_from_id(event.room_id)
-            originating_domain = get_domain_from_id(event.sender)
+            creating_domain = RoomID.from_string(event.room_id).domain
+            originating_domain = UserID.from_string(event.sender).domain
             if creating_domain != originating_domain:
                 if not self.can_federate(event, auth_events):
                     raise AuthError(
@@ -126,24 +120,6 @@ class Auth(object):
                 return allowed
 
             self.check_event_sender_in_room(event, auth_events)
-
-            # Special case to allow m.room.third_party_invite events wherever
-            # a user is allowed to issue invites.  Fixes
-            # https://github.com/vector-im/vector-web/issues/1208 hopefully
-            if event.type == EventTypes.ThirdPartyInvite:
-                user_level = self._get_user_power_level(event.user_id, auth_events)
-                invite_level = self._get_named_level(auth_events, "invite", 0)
-
-                if user_level < invite_level:
-                    raise AuthError(
-                        403, (
-                            "You cannot issue a third party invite for %s." %
-                            (event.content.display_name,)
-                        )
-                    )
-                else:
-                    return True
-
             self._can_send_event(event, auth_events)
 
             if event.type == EventTypes.PowerLevels:
@@ -243,7 +219,7 @@ class Auth(object):
         for event in curr_state.values():
             if event.type == EventTypes.Member:
                 try:
-                    if get_domain_from_id(event.state_key) != host:
+                    if UserID.from_string(event.state_key).domain != host:
                         continue
                 except:
                     logger.warn("state_key not user_id: %s", event.state_key)
@@ -290,8 +266,8 @@ class Auth(object):
 
         target_user_id = event.state_key
 
-        creating_domain = get_domain_from_id(event.room_id)
-        target_domain = get_domain_from_id(target_user_id)
+        creating_domain = RoomID.from_string(event.room_id).domain
+        target_domain = UserID.from_string(target_user_id).domain
         if creating_domain != target_domain:
             if not self.can_federate(event, auth_events):
                 raise AuthError(
@@ -531,7 +507,7 @@ class Auth(object):
             return default
 
     @defer.inlineCallbacks
-    def get_user_by_req(self, request, allow_guest=False, rights="access"):
+    def get_user_by_req(self, request, allow_guest=False):
         """ Get a registered user's ID.
 
         Args:
@@ -553,7 +529,7 @@ class Auth(object):
                 )
 
             access_token = request.args["access_token"][0]
-            user_info = yield self.get_user_by_access_token(access_token, rights)
+            user_info = yield self.get_user_by_access_token(access_token)
             user = user_info["user"]
             token_id = user_info["token_id"]
             is_guest = user_info["is_guest"]
@@ -614,7 +590,7 @@ class Auth(object):
         defer.returnValue(user_id)
 
     @defer.inlineCallbacks
-    def get_user_by_access_token(self, token, rights="access"):
+    def get_user_by_access_token(self, token):
         """ Get a registered user's ID.
 
         Args:
@@ -625,7 +601,7 @@ class Auth(object):
             AuthError if no user by that token exists or the token is invalid.
         """
         try:
-            ret = yield self.get_user_from_macaroon(token, rights)
+            ret = yield self.get_user_from_macaroon(token)
         except AuthError:
             # TODO(daniel): Remove this fallback when all existing access tokens
             # have been re-issued as macaroons.
@@ -633,26 +609,20 @@ class Auth(object):
         defer.returnValue(ret)
 
     @defer.inlineCallbacks
-    def get_user_from_macaroon(self, macaroon_str, rights="access"):
+    def get_user_from_macaroon(self, macaroon_str):
         try:
             macaroon = pymacaroons.Macaroon.deserialize(macaroon_str)
+            self.validate_macaroon(macaroon, "access", False)
 
             user_prefix = "user_id = "
             user = None
-            user_id = None
             guest = False
             for caveat in macaroon.caveats:
                 if caveat.caveat_id.startswith(user_prefix):
-                    user_id = caveat.caveat_id[len(user_prefix):]
-                    user = UserID.from_string(user_id)
+                    user = UserID.from_string(caveat.caveat_id[len(user_prefix):])
                 elif caveat.caveat_id == "guest = true":
                     guest = True
 
-            self.validate_macaroon(
-                macaroon, rights, self.hs.config.expire_access_token,
-                user_id=user_id,
-            )
-
             if user is None:
                 raise AuthError(
                     self.TOKEN_NOT_FOUND_HTTP_STATUS, "No user caveat in macaroon",
@@ -665,13 +635,6 @@ class Auth(object):
                     "is_guest": True,
                     "token_id": None,
                 }
-            elif rights == "delete_pusher":
-                # We don't store these tokens in the database
-                ret = {
-                    "user": user,
-                    "is_guest": False,
-                    "token_id": None,
-                }
             else:
                 # This codepath exists so that we can actually return a
                 # token ID, because we use token IDs in place of device
@@ -697,14 +660,13 @@ class Auth(object):
                 errcode=Codes.UNKNOWN_TOKEN
             )
 
-    def validate_macaroon(self, macaroon, type_string, verify_expiry, user_id):
+    def validate_macaroon(self, macaroon, type_string, verify_expiry):
         """
         validate that a Macaroon is understood by and was signed by this server.
 
         Args:
             macaroon(pymacaroons.Macaroon): The macaroon to validate
-            type_string(str): The kind of token required (e.g. "access", "refresh",
-                              "delete_pusher")
+            type_string(str): The kind of token this is (e.g. "access", "refresh")
             verify_expiry(bool): Whether to verify whether the macaroon has expired.
                 This should really always be True, but no clients currently implement
                 token refresh, so we can't enforce expiry yet.
@@ -712,7 +674,7 @@ class Auth(object):
         v = pymacaroons.Verifier()
         v.satisfy_exact("gen = 1")
         v.satisfy_exact("type = " + type_string)
-        v.satisfy_exact("user_id = %s" % user_id)
+        v.satisfy_general(lambda c: c.startswith("user_id = "))
         v.satisfy_exact("guest = true")
         if verify_expiry:
             v.satisfy_general(self._verify_expiry)
@@ -927,8 +889,8 @@ class Auth(object):
         if user_level >= redact_level:
             return False
 
-        redacter_domain = get_domain_from_id(event.event_id)
-        redactee_domain = get_domain_from_id(event.redacts)
+        redacter_domain = EventID.from_string(event.event_id).domain
+        redactee_domain = EventID.from_string(event.redacts).domain
         if redacter_domain == redactee_domain:
             return True
 

synapse/api/errors.py

@@ -42,9 +42,8 @@ class Codes(object):
     TOO_LARGE = "M_TOO_LARGE"
     EXCLUSIVE = "M_EXCLUSIVE"
     THREEPID_AUTH_FAILED = "M_THREEPID_AUTH_FAILED"
-    THREEPID_IN_USE = "M_THREEPID_IN_USE"
+    THREEPID_IN_USE = "THREEPID_IN_USE"
     INVALID_USERNAME = "M_INVALID_USERNAME"
-    SERVER_NOT_TRUSTED = "M_SERVER_NOT_TRUSTED"
 
 
 class CodeMessageException(RuntimeError):

synapse/api/filtering.py

@@ -15,8 +15,6 @@
 from synapse.api.errors import SynapseError
 from synapse.types import UserID, RoomID
 
-from twisted.internet import defer
-
 import ujson as json
 
 
@@ -26,10 +24,10 @@ class Filtering(object):
         super(Filtering, self).__init__()
         self.store = hs.get_datastore()
 
-    @defer.inlineCallbacks
     def get_user_filter(self, user_localpart, filter_id):
-        result = yield self.store.get_user_filter(user_localpart, filter_id)
-        defer.returnValue(FilterCollection(result))
+        result = self.store.get_user_filter(user_localpart, filter_id)
+        result.addCallback(FilterCollection)
+        return result
 
     def add_user_filter(self, user_localpart, user_filter):
         self.check_valid_filter(user_filter)

synapse/app/homeserver.py

@@ -16,10 +16,12 @@
 
 import synapse
 
-import gc
+import contextlib
 import logging
 import os
+import re
 import sys
+import time
 from synapse.config._base import ConfigError
 
 from synapse.python_dependencies import (
@@ -33,11 +35,18 @@ from synapse.storage.prepare_database import UpgradeDatabaseException, prepare_d
 
 from synapse.server import HomeServer
 
+
+from twisted.conch.manhole import ColoredManhole
+from twisted.conch.insults import insults
+from twisted.conch import manhole_ssh
+from twisted.cred import checkers, portal
+
+
 from twisted.internet import reactor, task, defer
 from twisted.application import service
 from twisted.web.resource import Resource, EncodingResourceWrapper
 from twisted.web.static import File
-from twisted.web.server import GzipEncoderFactory
+from twisted.web.server import Site, GzipEncoderFactory, Request
 from synapse.http.server import RootRedirect
 from synapse.rest.media.v0.content_repository import ContentRepoResource
 from synapse.rest.media.v1.media_repository import MediaRepositoryResource
@@ -57,10 +66,6 @@ from synapse.federation.transport.server import TransportLayerServer
 
 from synapse.util.rlimit import change_resource_limit
 from synapse.util.versionstring import get_version_string
-from synapse.util.httpresourcetree import create_resource_tree
-from synapse.util.manhole import manhole
-
-from synapse.http.site import SynapseSite
 
 from synapse import events
 
@@ -69,6 +74,9 @@ from daemonize import Daemonize
 logger = logging.getLogger("synapse.app.homeserver")
 
 
+ACCESS_TOKEN_RE = re.compile(r'(\?.*access(_|%5[Ff])token=)[^&]*(.*)$')
+
+
 def gz_wrap(r):
     return EncodingResourceWrapper(r, [GzipEncoderFactory()])
 
@@ -147,7 +155,7 @@ class SynapseHomeServer(HomeServer):
                         MEDIA_PREFIX: media_repo,
                         LEGACY_MEDIA_PREFIX: media_repo,
                         CONTENT_REPO_PREFIX: ContentRepoResource(
-                            self, self.config.uploads_path
+                            self, self.config.uploads_path, self.auth, self.content_addr
                         ),
                     })
 
@@ -166,12 +174,7 @@ class SynapseHomeServer(HomeServer):
                 if name == "replication":
                     resources[REPLICATION_PREFIX] = ReplicationResource(self)
 
-        if WEB_CLIENT_PREFIX in resources:
-            root_resource = RootRedirect(WEB_CLIENT_PREFIX)
-        else:
-            root_resource = Resource()
-
-        root_resource = create_resource_tree(resources, root_resource)
+        root_resource = create_resource_tree(resources)
         if tls:
             reactor.listenSSL(
                 port,
@@ -204,13 +207,24 @@ class SynapseHomeServer(HomeServer):
             if listener["type"] == "http":
                 self._listener_http(config, listener)
             elif listener["type"] == "manhole":
+                checker = checkers.InMemoryUsernamePasswordDatabaseDontUse(
+                    matrix="rabbithole"
+                )
+
+                rlm = manhole_ssh.TerminalRealm()
+                rlm.chainedProtocolFactory = lambda: insults.ServerProtocol(
+                    ColoredManhole,
+                    {
+                        "__name__": "__console__",
+                        "hs": self,
+                    }
+                )
+
+                f = manhole_ssh.ConchFactory(portal.Portal(rlm, [checker]))
+
                 reactor.listenTCP(
                     listener["port"],
-                    manhole(
-                        username="matrix",
-                        password="rabbithole",
-                        globals={"hs": self},
-                    ),
+                    f,
                     interface=listener.get("bind_address", '127.0.0.1')
                 )
             else:
@@ -266,9 +280,10 @@ def setup(config_options):
         HomeServer
     """
     try:
-        config = HomeServerConfig.load_or_generate_config(
+        config = HomeServerConfig.load_config(
             "Synapse Homeserver",
             config_options,
+            generate_section="Homeserver"
         )
     except ConfigError as e:
         sys.stderr.write("\n" + e.message + "\n")
@@ -301,6 +316,7 @@ def setup(config_options):
         db_config=config.database_config,
         tls_server_context_factory=tls_server_context_factory,
         config=config,
+        content_addr=config.content_addr,
         version_string=version_string,
         database_engine=database_engine,
     )
@@ -350,13 +366,215 @@ class SynapseService(service.Service):
     def startService(self):
         hs = setup(self.config)
         change_resource_limit(hs.config.soft_file_limit)
-        if hs.config.gc_thresholds:
-            gc.set_threshold(*hs.config.gc_thresholds)
 
     def stopService(self):
         return self._port.stopListening()
 
 
+class SynapseRequest(Request):
+    def __init__(self, site, *args, **kw):
+        Request.__init__(self, *args, **kw)
+        self.site = site
+        self.authenticated_entity = None
+        self.start_time = 0
+
+    def __repr__(self):
+        # We overwrite this so that we don't log ``access_token``
+        return '<%s at 0x%x method=%s uri=%s clientproto=%s site=%s>' % (
+            self.__class__.__name__,
+            id(self),
+            self.method,
+            self.get_redacted_uri(),
+            self.clientproto,
+            self.site.site_tag,
+        )
+
+    def get_redacted_uri(self):
+        return ACCESS_TOKEN_RE.sub(
+            r'\1<redacted>\3',
+            self.uri
+        )
+
+    def get_user_agent(self):
+        return self.requestHeaders.getRawHeaders("User-Agent", [None])[-1]
+
+    def started_processing(self):
+        self.site.access_logger.info(
+            "%s - %s - Received request: %s %s",
+            self.getClientIP(),
+            self.site.site_tag,
+            self.method,
+            self.get_redacted_uri()
+        )
+        self.start_time = int(time.time() * 1000)
+
+    def finished_processing(self):
+
+        try:
+            context = LoggingContext.current_context()
+            ru_utime, ru_stime = context.get_resource_usage()
+            db_txn_count = context.db_txn_count
+            db_txn_duration = context.db_txn_duration
+        except:
+            ru_utime, ru_stime = (0, 0)
+            db_txn_count, db_txn_duration = (0, 0)
+
+        self.site.access_logger.info(
+            "%s - %s - {%s}"
+            " Processed request: %dms (%dms, %dms) (%dms/%d)"
+            " %sB %s \"%s %s %s\" \"%s\"",
+            self.getClientIP(),
+            self.site.site_tag,
+            self.authenticated_entity,
+            int(time.time() * 1000) - self.start_time,
+            int(ru_utime * 1000),
+            int(ru_stime * 1000),
+            int(db_txn_duration * 1000),
+            int(db_txn_count),
+            self.sentLength,
+            self.code,
+            self.method,
+            self.get_redacted_uri(),
+            self.clientproto,
+            self.get_user_agent(),
+        )
+
+    @contextlib.contextmanager
+    def processing(self):
+        self.started_processing()
+        yield
+        self.finished_processing()
+
+
+class XForwardedForRequest(SynapseRequest):
+    def __init__(self, *args, **kw):
+        SynapseRequest.__init__(self, *args, **kw)
+
+    """
+    Add a layer on top of another request that only uses the value of an
+    X-Forwarded-For header as the result of C{getClientIP}.
+    """
+    def getClientIP(self):
+        """
+        @return: The client address (the first address) in the value of the
+            I{X-Forwarded-For header}.  If the header is not present, return
+            C{b"-"}.
+        """
+        return self.requestHeaders.getRawHeaders(
+            b"x-forwarded-for", [b"-"])[0].split(b",")[0].strip()
+
+
+class SynapseRequestFactory(object):
+    def __init__(self, site, x_forwarded_for):
+        self.site = site
+        self.x_forwarded_for = x_forwarded_for
+
+    def __call__(self, *args, **kwargs):
+        if self.x_forwarded_for:
+            return XForwardedForRequest(self.site, *args, **kwargs)
+        else:
+            return SynapseRequest(self.site, *args, **kwargs)
+
+
+class SynapseSite(Site):
+    """
+    Subclass of a twisted http Site that does access logging with python's
+    standard logging
+    """
+    def __init__(self, logger_name, site_tag, config, resource, *args, **kwargs):
+        Site.__init__(self, resource, *args, **kwargs)
+
+        self.site_tag = site_tag
+
+        proxied = config.get("x_forwarded", False)
+        self.requestFactory = SynapseRequestFactory(self, proxied)
+        self.access_logger = logging.getLogger(logger_name)
+
+    def log(self, request):
+        pass
+
+
+def create_resource_tree(desired_tree, redirect_root_to_web_client=True):
+    """Create the resource tree for this Home Server.
+
+    This in unduly complicated because Twisted does not support putting
+    child resources more than 1 level deep at a time.
+
+    Args:
+        web_client (bool): True to enable the web client.
+        redirect_root_to_web_client (bool): True to redirect '/' to the
+        location of the web client. This does nothing if web_client is not
+        True.
+    """
+    if redirect_root_to_web_client and WEB_CLIENT_PREFIX in desired_tree:
+        root_resource = RootRedirect(WEB_CLIENT_PREFIX)
+    else:
+        root_resource = Resource()
+
+    # ideally we'd just use getChild and putChild but getChild doesn't work
+    # unless you give it a Request object IN ADDITION to the name :/ So
+    # instead, we'll store a copy of this mapping so we can actually add
+    # extra resources to existing nodes. See self._resource_id for the key.
+    resource_mappings = {}
+    for full_path, res in desired_tree.items():
+        logger.info("Attaching %s to path %s", res, full_path)
+        last_resource = root_resource
+        for path_seg in full_path.split('/')[1:-1]:
+            if path_seg not in last_resource.listNames():
+                # resource doesn't exist, so make a "dummy resource"
+                child_resource = Resource()
+                last_resource.putChild(path_seg, child_resource)
+                res_id = _resource_id(last_resource, path_seg)
+                resource_mappings[res_id] = child_resource
+                last_resource = child_resource
+            else:
+                # we have an existing Resource, use that instead.
+                res_id = _resource_id(last_resource, path_seg)
+                last_resource = resource_mappings[res_id]
+
+        # ===========================
+        # now attach the actual desired resource
+        last_path_seg = full_path.split('/')[-1]
+
+        # if there is already a resource here, thieve its children and
+        # replace it
+        res_id = _resource_id(last_resource, last_path_seg)
+        if res_id in resource_mappings:
+            # there is a dummy resource at this path already, which needs
+            # to be replaced with the desired resource.
+            existing_dummy_resource = resource_mappings[res_id]
+            for child_name in existing_dummy_resource.listNames():
+                child_res_id = _resource_id(
+                    existing_dummy_resource, child_name
+                )
+                child_resource = resource_mappings[child_res_id]
+                # steal the children
+                res.putChild(child_name, child_resource)
+
+        # finally, insert the desired resource in the right place
+        last_resource.putChild(last_path_seg, res)
+        res_id = _resource_id(last_resource, last_path_seg)
+        resource_mappings[res_id] = res
+
+    return root_resource
+
+
+def _resource_id(resource, path_seg):
+    """Construct an arbitrary resource ID so you can retrieve the mapping
+    later.
+
+    If you want to represent resource A putChild resource B with path C,
+    the mapping should looks like _resource_id(A,C) = B.
+
+    Args:
+        resource (Resource): The *parent* Resourceb
+        path_seg (str): The name of the child Resource to be attached.
+    Returns:
+        str: A unique string which can be a key to the child Resource.
+    """
+    return "%s-%s" % (resource, path_seg)
+
+
 def run(hs):
     PROFILE_SYNAPSE = False
     if PROFILE_SYNAPSE:
@@ -423,8 +641,6 @@ def run(hs):
         # sys.settrace(logcontext_tracer)
         with LoggingContext("run"):
             change_resource_limit(hs.config.soft_file_limit)
-            if hs.config.gc_thresholds:
-                gc.set_threshold(*hs.config.gc_thresholds)
             reactor.run()
 
     if hs.config.daemonize:

synapse/app/pusher.py

@@ -17,78 +17,54 @@
 import synapse
 
 from synapse.server import HomeServer
+from synapse.util.versionstring import get_version_string
 from synapse.config._base import ConfigError
-from synapse.config.logger import setup_logging
-from synapse.config.homeserver import HomeServerConfig
-from synapse.http.site import SynapseSite
-from synapse.metrics.resource import MetricsResource, METRICS_PREFIX
-from synapse.storage.roommember import RoomMemberStore
+from synapse.config.database import DatabaseConfig
+from synapse.config.logger import LoggingConfig
 from synapse.replication.slave.storage.events import SlavedEventStore
 from synapse.replication.slave.storage.pushers import SlavedPusherStore
 from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
-from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
 from synapse.storage.engines import create_engine
 from synapse.storage import DataStore
 from synapse.util.async import sleep
-from synapse.util.httpresourcetree import create_resource_tree
-from synapse.util.logcontext import LoggingContext, preserve_fn
-from synapse.util.manhole import manhole
-from synapse.util.rlimit import change_resource_limit
-from synapse.util.versionstring import get_version_string
+from synapse.util.logcontext import (LoggingContext, preserve_fn)
 
 from twisted.internet import reactor, defer
-from twisted.web.resource import Resource
-
-from daemonize import Daemonize
 
 import sys
 import logging
-import gc
 
 logger = logging.getLogger("synapse.app.pusher")
 
 
+class SlaveConfig(DatabaseConfig):
+    def read_config(self, config):
+        self.replication_url = config["replication_url"]
+        self.server_name = config["server_name"]
+        self.use_insecure_ssl_client_just_for_testing_do_not_use = True
+        self.user_agent_suffix = None
+        self.start_pushers = True
+
+    def default_config(self, **kwargs):
+        return """\
+        ## Slave ##
+        #replication_url: https://localhost:{replication_port}/_synapse/replication
+
+        report_stats: False
+        """
+
+
+class PusherSlaveConfig(SlaveConfig, LoggingConfig):
+    pass
+
+
 class PusherSlaveStore(
-    SlavedEventStore, SlavedPusherStore, SlavedReceiptsStore,
-    SlavedAccountDataStore
+    SlavedEventStore, SlavedPusherStore, SlavedReceiptsStore
 ):
     update_pusher_last_stream_ordering_and_success = (
         DataStore.update_pusher_last_stream_ordering_and_success.__func__
     )
 
-    update_pusher_failing_since = (
-        DataStore.update_pusher_failing_since.__func__
-    )
-
-    update_pusher_last_stream_ordering = (
-        DataStore.update_pusher_last_stream_ordering.__func__
-    )
-
-    get_throttle_params_by_room = (
-        DataStore.get_throttle_params_by_room.__func__
-    )
-
-    set_throttle_params = (
-        DataStore.set_throttle_params.__func__
-    )
-
-    get_time_of_last_push_action_before = (
-        DataStore.get_time_of_last_push_action_before.__func__
-    )
-
-    get_profile_displayname = (
-        DataStore.get_profile_displayname.__func__
-    )
-
-    # XXX: This is a bit broken because we don't persist forgotten rooms
-    # in a way that they can be streamed. This means that we don't have a
-    # way to invalidate the forgotten rooms cache correctly.
-    # For now we expire the cache every 10 minutes.
-    BROKEN_CACHE_EXPIRY_MS = 60 * 60 * 1000
-    who_forgot_in_room = (
-        RoomMemberStore.__dict__["who_forgot_in_room"]
-    )
-
 
 class PusherServer(HomeServer):
 
@@ -112,7 +88,7 @@ class PusherServer(HomeServer):
 
     def remove_pusher(self, app_id, push_key, user_id):
         http_client = self.get_simple_http_client()
-        replication_url = self.config.worker_replication_url
+        replication_url = self.config.replication_url
         url = replication_url + "/remove_pushers"
         return http_client.post_json_get_json(url, {
             "remove": [{
@@ -122,53 +98,12 @@ class PusherServer(HomeServer):
             }]
         })
 
-    def _listen_http(self, listener_config):
-        port = listener_config["port"]
-        bind_address = listener_config.get("bind_address", "")
-        site_tag = listener_config.get("tag", port)
-        resources = {}
-        for res in listener_config["resources"]:
-            for name in res["names"]:
-                if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(self)
-
-        root_resource = create_resource_tree(resources, Resource())
-        reactor.listenTCP(
-            port,
-            SynapseSite(
-                "synapse.access.http.%s" % (site_tag,),
-                site_tag,
-                listener_config,
-                root_resource,
-            ),
-            interface=bind_address
-        )
-        logger.info("Synapse pusher now listening on port %d", port)
-
-    def start_listening(self, listeners):
-        for listener in listeners:
-            if listener["type"] == "http":
-                self._listen_http(listener)
-            elif listener["type"] == "manhole":
-                reactor.listenTCP(
-                    listener["port"],
-                    manhole(
-                        username="matrix",
-                        password="rabbithole",
-                        globals={"hs": self},
-                    ),
-                    interface=listener.get("bind_address", '127.0.0.1')
-                )
-            else:
-                logger.warn("Unrecognized listener type: %s", listener["type"])
-
     @defer.inlineCallbacks
     def replicate(self):
         http_client = self.get_simple_http_client()
         store = self.get_datastore()
-        replication_url = self.config.worker_replication_url
+        replication_url = self.config.replication_url
         pusher_pool = self.get_pusherpool()
-        clock = self.get_clock()
 
         def stop_pusher(user_id, app_id, pushkey):
             key = "%s:%s" % (app_id, pushkey)
@@ -220,52 +155,28 @@ class PusherServer(HomeServer):
                     min_stream_id, max_stream_id, affected_room_ids
                 )
 
-        def expire_broken_caches():
-            store.who_forgot_in_room.invalidate_all()
-
-        next_expire_broken_caches_ms = 0
         while True:
             try:
                 args = store.stream_positions()
                 args["timeout"] = 30000
                 result = yield http_client.get_json(replication_url, args=args)
-                now_ms = clock.time_msec()
-                if now_ms > next_expire_broken_caches_ms:
-                    expire_broken_caches()
-                    next_expire_broken_caches_ms = (
-                        now_ms + store.BROKEN_CACHE_EXPIRY_MS
-                    )
                 yield store.process_replication(result)
                 poke_pushers(result)
             except:
                 logger.exception("Error replicating from %r", replication_url)
-                yield sleep(30)
+                sleep(30)
 
 
-def start(config_options):
+def setup(config_options):
     try:
-        config = HomeServerConfig.load_config(
+        config = PusherSlaveConfig.load_config(
             "Synapse pusher", config_options
         )
     except ConfigError as e:
         sys.stderr.write("\n" + e.message + "\n")
         sys.exit(1)
 
-    assert config.worker_app == "synapse.app.pusher"
-
-    setup_logging(config.worker_log_config, config.worker_log_file)
-
-    if config.start_pushers:
-        sys.stderr.write(
-            "\nThe pushers must be disabled in the main synapse process"
-            "\nbefore they can be run in a separate worker."
-            "\nPlease add ``start_pushers: false`` to the main config"
-            "\n"
-        )
-        sys.exit(1)
-
-    # Force the pushers to start since they will be disabled in the main config
-    config.start_pushers = True
+    config.setup_logging()
 
     database_engine = create_engine(config.database_config)
 
@@ -278,15 +189,6 @@ def start(config_options):
     )
 
     ps.setup()
-    ps.start_listening(config.worker_listeners)
-
-    def run():
-        with LoggingContext("run"):
-            logger.info("Running")
-            change_resource_limit(config.soft_file_limit)
-            if config.gc_thresholds:
-                gc.set_threshold(*config.gc_thresholds)
-            reactor.run()
 
     def start():
         ps.replicate()
@@ -295,20 +197,10 @@ def start(config_options):
 
     reactor.callWhenRunning(start)
 
-    if config.worker_daemonize:
-        daemon = Daemonize(
-            app="synapse-pusher",
-            pid=config.worker_pid_file,
-            action=run,
-            auto_close_fds=False,
-            verbose=True,
-            logger=logger,
-        )
-        daemon.start()
-    else:
-        run()
+    return ps
 
 
 if __name__ == '__main__':
     with LoggingContext("main"):
-        ps = start(sys.argv[1:])
+        ps = setup(sys.argv[1:])
+        reactor.run()

synapse/app/synchrotron.py

@@ -1,465 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-# Copyright 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import synapse
-
-from synapse.api.constants import EventTypes, PresenceState
-from synapse.config._base import ConfigError
-from synapse.config.homeserver import HomeServerConfig
-from synapse.config.logger import setup_logging
-from synapse.events import FrozenEvent
-from synapse.handlers.presence import PresenceHandler
-from synapse.http.site import SynapseSite
-from synapse.http.server import JsonResource
-from synapse.metrics.resource import MetricsResource, METRICS_PREFIX
-from synapse.rest.client.v2_alpha import sync
-from synapse.replication.slave.storage._base import BaseSlavedStore
-from synapse.replication.slave.storage.events import SlavedEventStore
-from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
-from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
-from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
-from synapse.replication.slave.storage.registration import SlavedRegistrationStore
-from synapse.replication.slave.storage.filtering import SlavedFilteringStore
-from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
-from synapse.replication.slave.storage.presence import SlavedPresenceStore
-from synapse.server import HomeServer
-from synapse.storage.client_ips import ClientIpStore
-from synapse.storage.engines import create_engine
-from synapse.storage.presence import PresenceStore, UserPresenceState
-from synapse.storage.roommember import RoomMemberStore
-from synapse.util.async import sleep
-from synapse.util.httpresourcetree import create_resource_tree
-from synapse.util.logcontext import LoggingContext, preserve_fn
-from synapse.util.manhole import manhole
-from synapse.util.rlimit import change_resource_limit
-from synapse.util.stringutils import random_string
-from synapse.util.versionstring import get_version_string
-
-from twisted.internet import reactor, defer
-from twisted.web.resource import Resource
-
-from daemonize import Daemonize
-
-import sys
-import logging
-import contextlib
-import gc
-import ujson as json
-
-logger = logging.getLogger("synapse.app.synchrotron")
-
-
-class SynchrotronSlavedStore(
-    SlavedPushRuleStore,
-    SlavedEventStore,
-    SlavedReceiptsStore,
-    SlavedAccountDataStore,
-    SlavedApplicationServiceStore,
-    SlavedRegistrationStore,
-    SlavedFilteringStore,
-    SlavedPresenceStore,
-    BaseSlavedStore,
-    ClientIpStore,  # After BaseSlavedStore because the constructor is different
-):
-    # XXX: This is a bit broken because we don't persist forgotten rooms
-    # in a way that they can be streamed. This means that we don't have a
-    # way to invalidate the forgotten rooms cache correctly.
-    # For now we expire the cache every 10 minutes.
-    BROKEN_CACHE_EXPIRY_MS = 60 * 60 * 1000
-    who_forgot_in_room = (
-        RoomMemberStore.__dict__["who_forgot_in_room"]
-    )
-
-    # XXX: This is a bit broken because we don't persist the accepted list in a
-    # way that can be replicated. This means that we don't have a way to
-    # invalidate the cache correctly.
-    get_presence_list_accepted = PresenceStore.__dict__[
-        "get_presence_list_accepted"
-    ]
-
-UPDATE_SYNCING_USERS_MS = 10 * 1000
-
-
-class SynchrotronPresence(object):
-    def __init__(self, hs):
-        self.http_client = hs.get_simple_http_client()
-        self.store = hs.get_datastore()
-        self.user_to_num_current_syncs = {}
-        self.syncing_users_url = hs.config.worker_replication_url + "/syncing_users"
-        self.clock = hs.get_clock()
-
-        active_presence = self.store.take_presence_startup_info()
-        self.user_to_current_state = {
-            state.user_id: state
-            for state in active_presence
-        }
-
-        self.process_id = random_string(16)
-        logger.info("Presence process_id is %r", self.process_id)
-
-        self._sending_sync = False
-        self._need_to_send_sync = False
-        self.clock.looping_call(
-            self._send_syncing_users_regularly,
-            UPDATE_SYNCING_USERS_MS,
-        )
-
-        reactor.addSystemEventTrigger("before", "shutdown", self._on_shutdown)
-
-    def set_state(self, user, state):
-        # TODO Hows this supposed to work?
-        pass
-
-    get_states = PresenceHandler.get_states.__func__
-    current_state_for_users = PresenceHandler.current_state_for_users.__func__
-
-    @defer.inlineCallbacks
-    def user_syncing(self, user_id, affect_presence):
-        if affect_presence:
-            curr_sync = self.user_to_num_current_syncs.get(user_id, 0)
-            self.user_to_num_current_syncs[user_id] = curr_sync + 1
-            prev_states = yield self.current_state_for_users([user_id])
-            if prev_states[user_id].state == PresenceState.OFFLINE:
-                # TODO: Don't block the sync request on this HTTP hit.
-                yield self._send_syncing_users_now()
-
-        def _end():
-            # We check that the user_id is in user_to_num_current_syncs because
-            # user_to_num_current_syncs may have been cleared if we are
-            # shutting down.
-            if affect_presence and user_id in self.user_to_num_current_syncs:
-                self.user_to_num_current_syncs[user_id] -= 1
-
-        @contextlib.contextmanager
-        def _user_syncing():
-            try:
-                yield
-            finally:
-                _end()
-
-        defer.returnValue(_user_syncing())
-
-    @defer.inlineCallbacks
-    def _on_shutdown(self):
-        # When the synchrotron is shutdown tell the master to clear the in
-        # progress syncs for this process
-        self.user_to_num_current_syncs.clear()
-        yield self._send_syncing_users_now()
-
-    def _send_syncing_users_regularly(self):
-        # Only send an update if we aren't in the middle of sending one.
-        if not self._sending_sync:
-            preserve_fn(self._send_syncing_users_now)()
-
-    @defer.inlineCallbacks
-    def _send_syncing_users_now(self):
-        if self._sending_sync:
-            # We don't want to race with sending another update.
-            # Instead we wait for that update to finish and send another
-            # update afterwards.
-            self._need_to_send_sync = True
-            return
-
-        # Flag that we are sending an update.
-        self._sending_sync = True
-
-        yield self.http_client.post_json_get_json(self.syncing_users_url, {
-            "process_id": self.process_id,
-            "syncing_users": [
-                user_id for user_id, count in self.user_to_num_current_syncs.items()
-                if count > 0
-            ],
-        })
-
-        # Unset the flag as we are no longer sending an update.
-        self._sending_sync = False
-        if self._need_to_send_sync:
-            # If something happened while we were sending the update then
-            # we might need to send another update.
-            # TODO: Check if the update that was sent matches the current state
-            # as we only need to send an update if they are different.
-            self._need_to_send_sync = False
-            yield self._send_syncing_users_now()
-
-    def process_replication(self, result):
-        stream = result.get("presence", {"rows": []})
-        for row in stream["rows"]:
-            (
-                position, user_id, state, last_active_ts,
-                last_federation_update_ts, last_user_sync_ts, status_msg,
-                currently_active
-            ) = row
-            self.user_to_current_state[user_id] = UserPresenceState(
-                user_id, state, last_active_ts,
-                last_federation_update_ts, last_user_sync_ts, status_msg,
-                currently_active
-            )
-
-
-class SynchrotronTyping(object):
-    def __init__(self, hs):
-        self._latest_room_serial = 0
-        self._room_serials = {}
-        self._room_typing = {}
-
-    def stream_positions(self):
-        return {"typing": self._latest_room_serial}
-
-    def process_replication(self, result):
-        stream = result.get("typing")
-        if stream:
-            self._latest_room_serial = int(stream["position"])
-
-            for row in stream["rows"]:
-                position, room_id, typing_json = row
-                typing = json.loads(typing_json)
-                self._room_serials[room_id] = position
-                self._room_typing[room_id] = typing
-
-
-class SynchrotronApplicationService(object):
-    def notify_interested_services(self, event):
-        pass
-
-
-class SynchrotronServer(HomeServer):
-    def get_db_conn(self, run_new_connection=True):
-        # Any param beginning with cp_ is a parameter for adbapi, and should
-        # not be passed to the database engine.
-        db_params = {
-            k: v for k, v in self.db_config.get("args", {}).items()
-            if not k.startswith("cp_")
-        }
-        db_conn = self.database_engine.module.connect(**db_params)
-
-        if run_new_connection:
-            self.database_engine.on_new_connection(db_conn)
-        return db_conn
-
-    def setup(self):
-        logger.info("Setting up.")
-        self.datastore = SynchrotronSlavedStore(self.get_db_conn(), self)
-        logger.info("Finished setting up.")
-
-    def _listen_http(self, listener_config):
-        port = listener_config["port"]
-        bind_address = listener_config.get("bind_address", "")
-        site_tag = listener_config.get("tag", port)
-        resources = {}
-        for res in listener_config["resources"]:
-            for name in res["names"]:
-                if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(self)
-                elif name == "client":
-                    resource = JsonResource(self, canonical_json=False)
-                    sync.register_servlets(self, resource)
-                    resources.update({
-                        "/_matrix/client/r0": resource,
-                        "/_matrix/client/unstable": resource,
-                        "/_matrix/client/v2_alpha": resource,
-                    })
-
-        root_resource = create_resource_tree(resources, Resource())
-        reactor.listenTCP(
-            port,
-            SynapseSite(
-                "synapse.access.http.%s" % (site_tag,),
-                site_tag,
-                listener_config,
-                root_resource,
-            ),
-            interface=bind_address
-        )
-        logger.info("Synapse synchrotron now listening on port %d", port)
-
-    def start_listening(self, listeners):
-        for listener in listeners:
-            if listener["type"] == "http":
-                self._listen_http(listener)
-            elif listener["type"] == "manhole":
-                reactor.listenTCP(
-                    listener["port"],
-                    manhole(
-                        username="matrix",
-                        password="rabbithole",
-                        globals={"hs": self},
-                    ),
-                    interface=listener.get("bind_address", '127.0.0.1')
-                )
-            else:
-                logger.warn("Unrecognized listener type: %s", listener["type"])
-
-    @defer.inlineCallbacks
-    def replicate(self):
-        http_client = self.get_simple_http_client()
-        store = self.get_datastore()
-        replication_url = self.config.worker_replication_url
-        clock = self.get_clock()
-        notifier = self.get_notifier()
-        presence_handler = self.get_presence_handler()
-        typing_handler = self.get_typing_handler()
-
-        def expire_broken_caches():
-            store.who_forgot_in_room.invalidate_all()
-            store.get_presence_list_accepted.invalidate_all()
-
-        def notify_from_stream(
-            result, stream_name, stream_key, room=None, user=None
-        ):
-            stream = result.get(stream_name)
-            if stream:
-                position_index = stream["field_names"].index("position")
-                if room:
-                    room_index = stream["field_names"].index(room)
-                if user:
-                    user_index = stream["field_names"].index(user)
-
-                users = ()
-                rooms = ()
-                for row in stream["rows"]:
-                    position = row[position_index]
-
-                    if user:
-                        users = (row[user_index],)
-
-                    if room:
-                        rooms = (row[room_index],)
-
-                    notifier.on_new_event(
-                        stream_key, position, users=users, rooms=rooms
-                    )
-
-        def notify(result):
-            stream = result.get("events")
-            if stream:
-                max_position = stream["position"]
-                for row in stream["rows"]:
-                    position = row[0]
-                    internal = json.loads(row[1])
-                    event_json = json.loads(row[2])
-                    event = FrozenEvent(event_json, internal_metadata_dict=internal)
-                    extra_users = ()
-                    if event.type == EventTypes.Member:
-                        extra_users = (event.state_key,)
-                    notifier.on_new_room_event(
-                        event, position, max_position, extra_users
-                    )
-
-            notify_from_stream(
-                result, "push_rules", "push_rules_key", user="user_id"
-            )
-            notify_from_stream(
-                result, "user_account_data", "account_data_key", user="user_id"
-            )
-            notify_from_stream(
-                result, "room_account_data", "account_data_key", user="user_id"
-            )
-            notify_from_stream(
-                result, "tag_account_data", "account_data_key", user="user_id"
-            )
-            notify_from_stream(
-                result, "receipts", "receipt_key", room="room_id"
-            )
-            notify_from_stream(
-                result, "typing", "typing_key", room="room_id"
-            )
-
-        next_expire_broken_caches_ms = 0
-        while True:
-            try:
-                args = store.stream_positions()
-                args.update(typing_handler.stream_positions())
-                args["timeout"] = 30000
-                result = yield http_client.get_json(replication_url, args=args)
-                now_ms = clock.time_msec()
-                if now_ms > next_expire_broken_caches_ms:
-                    expire_broken_caches()
-                    next_expire_broken_caches_ms = (
-                        now_ms + store.BROKEN_CACHE_EXPIRY_MS
-                    )
-                yield store.process_replication(result)
-                typing_handler.process_replication(result)
-                presence_handler.process_replication(result)
-                notify(result)
-            except:
-                logger.exception("Error replicating from %r", replication_url)
-                yield sleep(5)
-
-    def build_presence_handler(self):
-        return SynchrotronPresence(self)
-
-    def build_typing_handler(self):
-        return SynchrotronTyping(self)
-
-
-def start(config_options):
-    try:
-        config = HomeServerConfig.load_config(
-            "Synapse synchrotron", config_options
-        )
-    except ConfigError as e:
-        sys.stderr.write("\n" + e.message + "\n")
-        sys.exit(1)
-
-    assert config.worker_app == "synapse.app.synchrotron"
-
-    setup_logging(config.worker_log_config, config.worker_log_file)
-
-    database_engine = create_engine(config.database_config)
-
-    ss = SynchrotronServer(
-        config.server_name,
-        db_config=config.database_config,
-        config=config,
-        version_string=get_version_string("Synapse", synapse),
-        database_engine=database_engine,
-        application_service_handler=SynchrotronApplicationService(),
-    )
-
-    ss.setup()
-    ss.start_listening(config.worker_listeners)
-
-    def run():
-        with LoggingContext("run"):
-            logger.info("Running")
-            change_resource_limit(config.soft_file_limit)
-            if config.gc_thresholds:
-                gc.set_threshold(*config.gc_thresholds)
-            reactor.run()
-
-    def start():
-        ss.get_datastore().start_profiling()
-        ss.replicate()
-
-    reactor.callWhenRunning(start)
-
-    if config.worker_daemonize:
-        daemon = Daemonize(
-            app="synapse-synchrotron",
-            pid=config.worker_pid_file,
-            action=run,
-            auto_close_fds=False,
-            verbose=True,
-            logger=logger,
-        )
-        daemon.start()
-    else:
-        run()
-
-
-if __name__ == '__main__':
-    with LoggingContext("main"):
-        start(sys.argv[1:])

synapse/app/synctl.py

@@ -14,14 +14,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import argparse
-import collections
-import glob
+import sys
 import os
 import os.path
-import signal
 import subprocess
-import sys
+import signal
 import yaml
 
 SYNAPSE = ["python", "-B", "-m", "synapse.app.homeserver"]
@@ -31,181 +28,56 @@ RED = "\x1b[1;31m"
 NORMAL = "\x1b[m"
 
 
-def write(message, colour=NORMAL, stream=sys.stdout):
-    if colour == NORMAL:
-        stream.write(message + "\n")
-    else:
-        stream.write(colour + message + NORMAL + "\n")
-
-
 def start(configfile):
-    write("Starting ...")
+    print ("Starting ...")
     args = SYNAPSE
     args.extend(["--daemonize", "-c", configfile])
 
     try:
         subprocess.check_call(args)
-        write("started synapse.app.homeserver(%r)" % (configfile,), colour=GREEN)
+        print (GREEN + "started" + NORMAL)
     except subprocess.CalledProcessError as e:
-        write(
-            "error starting (exit code: %d); see above for logs" % e.returncode,
-            colour=RED,
+        print (
+            RED +
+            "error starting (exit code: %d); see above for logs" % e.returncode +
+            NORMAL
         )
 
 
-def start_worker(app, configfile, worker_configfile):
-    args = [
-        "python", "-B",
-        "-m", app,
-        "-c", configfile,
-        "-c", worker_configfile
-    ]
-
-    try:
-        subprocess.check_call(args)
-        write("started %s(%r)" % (app, worker_configfile), colour=GREEN)
-    except subprocess.CalledProcessError as e:
-        write(
-            "error starting %s(%r) (exit code: %d); see above for logs" % (
-                app, worker_configfile, e.returncode,
-            ),
-            colour=RED,
-        )
-
-
-def stop(pidfile, app):
+def stop(pidfile):
     if os.path.exists(pidfile):
         pid = int(open(pidfile).read())
         os.kill(pid, signal.SIGTERM)
-        write("stopped %s" % (app,), colour=GREEN)
-
-
-Worker = collections.namedtuple("Worker", [
-    "app", "configfile", "pidfile", "cache_factor"
-])
+        print (GREEN + "stopped" + NORMAL)
 
 
 def main():
-
-    parser = argparse.ArgumentParser()
-
-    parser.add_argument(
-        "action",
-        choices=["start", "stop", "restart"],
-        help="whether to start, stop or restart the synapse",
-    )
-    parser.add_argument(
-        "configfile",
-        nargs="?",
-        default="homeserver.yaml",
-        help="the homeserver config file, defaults to homserver.yaml",
-    )
-    parser.add_argument(
-        "-w", "--worker",
-        metavar="WORKERCONFIG",
-        help="start or stop a single worker",
-    )
-    parser.add_argument(
-        "-a", "--all-processes",
-        metavar="WORKERCONFIGDIR",
-        help="start or stop all the workers in the given directory"
-             " and the main synapse process",
-    )
-
-    options = parser.parse_args()
-
-    if options.worker and options.all_processes:
-        write(
-            'Cannot use "--worker" with "--all-processes"',
-            stream=sys.stderr
-        )
-        sys.exit(1)
-
-    configfile = options.configfile
+    configfile = sys.argv[2] if len(sys.argv) == 3 else "homeserver.yaml"
 
     if not os.path.exists(configfile):
-        write(
+        sys.stderr.write(
             "No config file found\n"
             "To generate a config file, run '%s -c %s --generate-config"
             " --server-name=<server name>'\n" % (
-                " ".join(SYNAPSE), options.configfile
-            ),
-            stream=sys.stderr,
+                " ".join(SYNAPSE), configfile
+            )
         )
         sys.exit(1)
 
-    with open(configfile) as stream:
-        config = yaml.load(stream)
-
+    config = yaml.load(open(configfile))
     pidfile = config["pid_file"]
-    cache_factor = config.get("synctl_cache_factor")
-    start_stop_synapse = True
 
-    if cache_factor:
-        os.environ["SYNAPSE_CACHE_FACTOR"] = str(cache_factor)
-
-    worker_configfiles = []
-    if options.worker:
-        start_stop_synapse = False
-        worker_configfile = options.worker
-        if not os.path.exists(worker_configfile):
-            write(
-                "No worker config found at %r" % (worker_configfile,),
-                stream=sys.stderr,
-            )
-            sys.exit(1)
-        worker_configfiles.append(worker_configfile)
-
-    if options.all_processes:
-        worker_configdir = options.all_processes
-        if not os.path.isdir(worker_configdir):
-            write(
-                "No worker config directory found at %r" % (worker_configdir,),
-                stream=sys.stderr,
-            )
-            sys.exit(1)
-        worker_configfiles.extend(sorted(glob.glob(
-            os.path.join(worker_configdir, "*.yaml")
-        )))
-
-    workers = []
-    for worker_configfile in worker_configfiles:
-        with open(worker_configfile) as stream:
-            worker_config = yaml.load(stream)
-        worker_app = worker_config["worker_app"]
-        worker_pidfile = worker_config["worker_pid_file"]
-        worker_daemonize = worker_config["worker_daemonize"]
-        assert worker_daemonize  # TODO print something more user friendly
-        worker_cache_factor = worker_config.get("synctl_cache_factor")
-        workers.append(Worker(
-            worker_app, worker_configfile, worker_pidfile, worker_cache_factor,
-        ))
-
-    action = options.action
-
-    if action == "stop" or action == "restart":
-        for worker in workers:
-            stop(worker.pidfile, worker.app)
-
-        if start_stop_synapse:
-            stop(pidfile, "synapse.app.homeserver")
-
-        # TODO: Wait for synapse to actually shutdown before starting it again
-
-    if action == "start" or action == "restart":
-        if start_stop_synapse:
-            start(configfile)
-
-        for worker in workers:
-            if worker.cache_factor:
-                os.environ["SYNAPSE_CACHE_FACTOR"] = str(worker.cache_factor)
-
-            start_worker(worker.app, configfile, worker.configfile)
-
-            if cache_factor:
-                os.environ["SYNAPSE_CACHE_FACTOR"] = str(cache_factor)
-            else:
-                os.environ.pop("SYNAPSE_CACHE_FACTOR", None)
+    action = sys.argv[1] if sys.argv[1:] else "usage"
+    if action == "start":
+        start(configfile)
+    elif action == "stop":
+        stop(pidfile)
+    elif action == "restart":
+        stop(pidfile)
+        start(configfile)
+    else:
+        sys.stderr.write("Usage: %s [start|stop|restart] [configfile]\n" % (sys.argv[0],))
+        sys.exit(1)
 
 
 if __name__ == "__main__":

synapse/appservice/scheduler.py

@@ -56,22 +56,22 @@ import logging
 logger = logging.getLogger(__name__)
 
 
-class ApplicationServiceScheduler(object):
+class AppServiceScheduler(object):
     """ Public facing API for this module. Does the required DI to tie the
     components together. This also serves as the "event_pool", which in this
     case is a simple array.
     """
 
-    def __init__(self, hs):
-        self.clock = hs.get_clock()
-        self.store = hs.get_datastore()
-        self.as_api = hs.get_application_service_api()
+    def __init__(self, clock, store, as_api):
+        self.clock = clock
+        self.store = store
+        self.as_api = as_api
 
         def create_recoverer(service, callback):
-            return _Recoverer(self.clock, self.store, self.as_api, service, callback)
+            return _Recoverer(clock, store, as_api, service, callback)
 
         self.txn_ctrl = _TransactionController(
-            self.clock, self.store, self.as_api, create_recoverer
+            clock, store, as_api, create_recoverer
         )
         self.queuer = _ServiceQueuer(self.txn_ctrl)
 

synapse/config/_base.py

@@ -157,40 +157,9 @@ class Config(object):
         return default_config, config
 
     @classmethod
-    def load_config(cls, description, argv):
-        config_parser = argparse.ArgumentParser(
-            description=description,
-        )
-        config_parser.add_argument(
-            "-c", "--config-path",
-            action="append",
-            metavar="CONFIG_FILE",
-            help="Specify config file. Can be given multiple times and"
-                 " may specify directories containing *.yaml files."
-        )
-
-        config_parser.add_argument(
-            "--keys-directory",
-            metavar="DIRECTORY",
-            help="Where files such as certs and signing keys are stored when"
-                 " their location is given explicitly in the config."
-                 " Defaults to the directory containing the last config file",
-        )
-
-        config_args = config_parser.parse_args(argv)
-
-        config_files = find_config_files(search_paths=config_args.config_path)
-
+    def load_config(cls, description, argv, generate_section=None):
         obj = cls()
-        obj.read_config_files(
-            config_files,
-            keys_directory=config_args.keys_directory,
-            generate_keys=False,
-        )
-        return obj
 
-    @classmethod
-    def load_or_generate_config(cls, description, argv):
         config_parser = argparse.ArgumentParser(add_help=False)
         config_parser.add_argument(
             "-c", "--config-path",
@@ -207,7 +176,7 @@ class Config(object):
         config_parser.add_argument(
             "--report-stats",
             action="store",
-            help="Whether the generated config reports anonymized usage statistics",
+            help="Stuff",
             choices=["yes", "no"]
         )
         config_parser.add_argument(
@@ -228,11 +197,36 @@ class Config(object):
         )
         config_args, remaining_args = config_parser.parse_known_args(argv)
 
-        config_files = find_config_files(search_paths=config_args.config_path)
-
         generate_keys = config_args.generate_keys
 
-        obj = cls()
+        config_files = []
+        if config_args.config_path:
+            for config_path in config_args.config_path:
+                if os.path.isdir(config_path):
+                    # We accept specifying directories as config paths, we search
+                    # inside that directory for all files matching *.yaml, and then
+                    # we apply them in *sorted* order.
+                    files = []
+                    for entry in os.listdir(config_path):
+                        entry_path = os.path.join(config_path, entry)
+                        if not os.path.isfile(entry_path):
+                            print (
+                                "Found subdirectory in config directory: %r. IGNORING."
+                            ) % (entry_path, )
+                            continue
+
+                        if not entry.endswith(".yaml"):
+                            print (
+                                "Found file in config directory that does not"
+                                " end in '.yaml': %r. IGNORING."
+                            ) % (entry_path, )
+                            continue
+
+                        files.append(entry_path)
+
+                    config_files.extend(sorted(files))
+                else:
+                    config_files.append(config_path)
 
         if config_args.generate_config:
             if config_args.report_stats is None:
@@ -305,43 +299,28 @@ class Config(object):
                 " -c CONFIG-FILE\""
             )
 
-        obj.read_config_files(
-            config_files,
-            keys_directory=config_args.keys_directory,
-            generate_keys=generate_keys,
-        )
-
-        if generate_keys:
-            return None
-
-        obj.invoke_all("read_arguments", args)
-
-        return obj
-
-    def read_config_files(self, config_files, keys_directory=None,
-                          generate_keys=False):
-        if not keys_directory:
-            keys_directory = os.path.dirname(config_files[-1])
-
-        config_dir_path = os.path.abspath(keys_directory)
+        if config_args.keys_directory:
+            config_dir_path = config_args.keys_directory
+        else:
+            config_dir_path = os.path.dirname(config_args.config_path[-1])
+        config_dir_path = os.path.abspath(config_dir_path)
 
         specified_config = {}
         for config_file in config_files:
-            yaml_config = self.read_config_file(config_file)
+            yaml_config = cls.read_config_file(config_file)
             specified_config.update(yaml_config)
 
         if "server_name" not in specified_config:
             raise ConfigError(MISSING_SERVER_NAME)
 
         server_name = specified_config["server_name"]
-        _, config = self.generate_config(
+        _, config = obj.generate_config(
             config_dir_path=config_dir_path,
             server_name=server_name,
             is_generating_file=False,
         )
         config.pop("log_config")
         config.update(specified_config)
-
         if "report_stats" not in config:
             raise ConfigError(
                 MISSING_REPORT_STATS_CONFIG_INSTRUCTIONS + "\n" +
@@ -349,51 +328,11 @@ class Config(object):
             )
 
         if generate_keys:
-            self.invoke_all("generate_files", config)
+            obj.invoke_all("generate_files", config)
             return
 
-        self.invoke_all("read_config", config)
+        obj.invoke_all("read_config", config)
 
+        obj.invoke_all("read_arguments", args)
 
-def find_config_files(search_paths):
-    """Finds config files using a list of search paths. If a path is a file
-    then that file path is added to the list. If a search path is a directory
-    then all the "*.yaml" files in that directory are added to the list in
-    sorted order.
-
-    Args:
-        search_paths(list(str)): A list of paths to search.
-
-    Returns:
-        list(str): A list of file paths.
-    """
-
-    config_files = []
-    if search_paths:
-        for config_path in search_paths:
-            if os.path.isdir(config_path):
-                # We accept specifying directories as config paths, we search
-                # inside that directory for all files matching *.yaml, and then
-                # we apply them in *sorted* order.
-                files = []
-                for entry in os.listdir(config_path):
-                    entry_path = os.path.join(config_path, entry)
-                    if not os.path.isfile(entry_path):
-                        print (
-                            "Found subdirectory in config directory: %r. IGNORING."
-                        ) % (entry_path, )
-                        continue
-
-                    if not entry.endswith(".yaml"):
-                        print (
-                            "Found file in config directory that does not"
-                            " end in '.yaml': %r. IGNORING."
-                        ) % (entry_path, )
-                        continue
-
-                    files.append(entry_path)
-
-                config_files.extend(sorted(files))
-            else:
-                config_files.append(config_path)
-    return config_files
+        return obj

synapse/config/appservice.py

@@ -12,16 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from ._base import Config, ConfigError
-
-from synapse.appservice import ApplicationService
-from synapse.types import UserID
-
-import urllib
-import yaml
-import logging
-
-logger = logging.getLogger(__name__)
+from ._base import Config
 
 
 class AppServiceConfig(Config):
@@ -34,99 +25,3 @@ class AppServiceConfig(Config):
         # A list of application service config file to use
         app_service_config_files: []
         """
-
-
-def load_appservices(hostname, config_files):
-    """Returns a list of Application Services from the config files."""
-    if not isinstance(config_files, list):
-        logger.warning(
-            "Expected %s to be a list of AS config files.", config_files
-        )
-        return []
-
-    # Dicts of value -> filename
-    seen_as_tokens = {}
-    seen_ids = {}
-
-    appservices = []
-
-    for config_file in config_files:
-        try:
-            with open(config_file, 'r') as f:
-                appservice = _load_appservice(
-                    hostname, yaml.load(f), config_file
-                )
-                if appservice.id in seen_ids:
-                    raise ConfigError(
-                        "Cannot reuse ID across application services: "
-                        "%s (files: %s, %s)" % (
-                            appservice.id, config_file, seen_ids[appservice.id],
-                        )
-                    )
-                seen_ids[appservice.id] = config_file
-                if appservice.token in seen_as_tokens:
-                    raise ConfigError(
-                        "Cannot reuse as_token across application services: "
-                        "%s (files: %s, %s)" % (
-                            appservice.token,
-                            config_file,
-                            seen_as_tokens[appservice.token],
-                        )
-                    )
-                seen_as_tokens[appservice.token] = config_file
-                logger.info("Loaded application service: %s", appservice)
-                appservices.append(appservice)
-        except Exception as e:
-            logger.error("Failed to load appservice from '%s'", config_file)
-            logger.exception(e)
-            raise
-    return appservices
-
-
-def _load_appservice(hostname, as_info, config_filename):
-    required_string_fields = [
-        "id", "url", "as_token", "hs_token", "sender_localpart"
-    ]
-    for field in required_string_fields:
-        if not isinstance(as_info.get(field), basestring):
-            raise KeyError("Required string field: '%s' (%s)" % (
-                field, config_filename,
-            ))
-
-    localpart = as_info["sender_localpart"]
-    if urllib.quote(localpart) != localpart:
-        raise ValueError(
-            "sender_localpart needs characters which are not URL encoded."
-        )
-    user = UserID(localpart, hostname)
-    user_id = user.to_string()
-
-    # namespace checks
-    if not isinstance(as_info.get("namespaces"), dict):
-        raise KeyError("Requires 'namespaces' object.")
-    for ns in ApplicationService.NS_LIST:
-        # specific namespaces are optional
-        if ns in as_info["namespaces"]:
-            # expect a list of dicts with exclusive and regex keys
-            for regex_obj in as_info["namespaces"][ns]:
-                if not isinstance(regex_obj, dict):
-                    raise ValueError(
-                        "Expected namespace entry in %s to be an object,"
-                        " but got %s", ns, regex_obj
-                    )
-                if not isinstance(regex_obj.get("regex"), basestring):
-                    raise ValueError(
-                        "Missing/bad type 'regex' key in %s", regex_obj
-                    )
-                if not isinstance(regex_obj.get("exclusive"), bool):
-                    raise ValueError(
-                        "Missing/bad type 'exclusive' key in %s", regex_obj
-                    )
-    return ApplicationService(
-        token=as_info["as_token"],
-        url=as_info["url"],
-        namespaces=as_info["namespaces"],
-        hs_token=as_info["hs_token"],
-        sender=user_id,
-        id=as_info["id"],
-    )

synapse/config/captcha.py

@@ -27,7 +27,6 @@ class CaptchaConfig(Config):
     def default_config(self, **kwargs):
         return """\
         ## Captcha ##
-        # See docs/CAPTCHA_SETUP for full details of configuring this.
 
         # This Home Server's ReCAPTCHA public key.
         recaptcha_public_key: "YOUR_PUBLIC_KEY"

synapse/config/emailconfig.py

@@ -1,98 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2015, 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# This file can't be called email.py because if it is, we cannot:
-import email.utils
-
-from ._base import Config
-
-
-class EmailConfig(Config):
-    def read_config(self, config):
-        self.email_enable_notifs = False
-
-        email_config = config.get("email", {})
-        self.email_enable_notifs = email_config.get("enable_notifs", False)
-
-        if self.email_enable_notifs:
-            # make sure we can import the required deps
-            import jinja2
-            import bleach
-            # prevent unused warnings
-            jinja2
-            bleach
-
-            required = [
-                "smtp_host",
-                "smtp_port",
-                "notif_from",
-                "template_dir",
-                "notif_template_html",
-                "notif_template_text",
-            ]
-
-            missing = []
-            for k in required:
-                if k not in email_config:
-                    missing.append(k)
-
-            if (len(missing) > 0):
-                raise RuntimeError(
-                    "email.enable_notifs is True but required keys are missing: %s" %
-                    (", ".join(["email." + k for k in missing]),)
-                )
-
-            if config.get("public_baseurl") is None:
-                raise RuntimeError(
-                    "email.enable_notifs is True but no public_baseurl is set"
-                )
-
-            self.email_smtp_host = email_config["smtp_host"]
-            self.email_smtp_port = email_config["smtp_port"]
-            self.email_notif_from = email_config["notif_from"]
-            self.email_template_dir = email_config["template_dir"]
-            self.email_notif_template_html = email_config["notif_template_html"]
-            self.email_notif_template_text = email_config["notif_template_text"]
-            self.email_notif_for_new_users = email_config.get(
-                "notif_for_new_users", True
-            )
-            if "app_name" in email_config:
-                self.email_app_name = email_config["app_name"]
-            else:
-                self.email_app_name = "Matrix"
-
-            # make sure it's valid
-            parsed = email.utils.parseaddr(self.email_notif_from)
-            if parsed[1] == '':
-                raise RuntimeError("Invalid notif_from address")
-        else:
-            self.email_enable_notifs = False
-            # Not much point setting defaults for the rest: it would be an
-            # error for them to be used.
-
-    def default_config(self, config_dir_path, server_name, **kwargs):
-        return """
-        # Enable sending emails for notification events
-        #email:
-        #   enable_notifs: false
-        #   smtp_host: "localhost"
-        #   smtp_port: 25
-        #   notif_from: "Your Friendly %(app)s Home Server <noreply@example.com>"
-        #   app_name: Matrix
-        #   template_dir: res/templates
-        #   notif_template_html: notif_mail.html
-        #   notif_template_text: notif_mail.txt
-        #   notif_for_new_users: True
-        """

synapse/config/homeserver.py

@@ -31,16 +31,13 @@ from .cas import CasConfig
 from .password import PasswordConfig
 from .jwt import JWTConfig
 from .ldap import LDAPConfig
-from .emailconfig import EmailConfig
-from .workers import WorkerConfig
 
 
 class HomeServerConfig(TlsConfig, ServerConfig, DatabaseConfig, LoggingConfig,
                        RatelimitConfig, ContentRepositoryConfig, CaptchaConfig,
                        VoipConfig, RegistrationConfig, MetricsConfig, ApiConfig,
                        AppServiceConfig, KeyConfig, SAML2Config, CasConfig,
-                       JWTConfig, LDAPConfig, PasswordConfig, EmailConfig,
-                       WorkerConfig,):
+                       JWTConfig, LDAPConfig, PasswordConfig,):
     pass
 
 

synapse/config/jwt.py

@@ -13,16 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from ._base import Config, ConfigError
-
-
-MISSING_JWT = (
-    """Missing jwt library. This is required for jwt login.
-
-    Install by running:
-        pip install pyjwt
-    """
-)
+from ._base import Config
 
 
 class JWTConfig(Config):
@@ -32,12 +23,6 @@ class JWTConfig(Config):
             self.jwt_enabled = jwt_config.get("enabled", False)
             self.jwt_secret = jwt_config["secret"]
             self.jwt_algorithm = jwt_config["algorithm"]
-
-            try:
-                import jwt
-                jwt  # To stop unused lint.
-            except ImportError:
-                raise ConfigError(MISSING_JWT)
         else:
             self.jwt_enabled = False
             self.jwt_secret = None
@@ -45,8 +30,6 @@ class JWTConfig(Config):
 
     def default_config(self, **kwargs):
         return """\
-        # The JWT needs to contain a globally unique "sub" (subject) claim.
-        #
         # jwt_config:
         #    enabled: true
         #    secret: "a secret"

synapse/config/key.py

@@ -57,8 +57,6 @@ class KeyConfig(Config):
             seed = self.signing_key[0].seed
             self.macaroon_secret_key = hashlib.sha256(seed)
 
-        self.expire_access_token = config.get("expire_access_token", False)
-
     def default_config(self, config_dir_path, server_name, is_generating_file=False,
                        **kwargs):
         base_key_name = os.path.join(config_dir_path, server_name)
@@ -71,9 +69,6 @@ class KeyConfig(Config):
         return """\
         macaroon_secret_key: "%(macaroon_secret_key)s"
 
-        # Used to enable access token expiration.
-        expire_access_token: False
-
         ## Signing Keys ##
 
         # Path to the signing key to sign messages with

synapse/config/ldap.py

@@ -13,88 +13,40 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from ._base import Config, ConfigError
-
-
-MISSING_LDAP3 = (
-    "Missing ldap3 library. This is required for LDAP Authentication."
-)
-
-
-class LDAPMode(object):
-    SIMPLE = "simple",
-    SEARCH = "search",
-
-    LIST = (SIMPLE, SEARCH)
+from ._base import Config
 
 
 class LDAPConfig(Config):
     def read_config(self, config):
-        ldap_config = config.get("ldap_config", {})
-
-        self.ldap_enabled = ldap_config.get("enabled", False)
-
-        if self.ldap_enabled:
-            # verify dependencies are available
-            try:
-                import ldap3
-                ldap3  # to stop unused lint
-            except ImportError:
-                raise ConfigError(MISSING_LDAP3)
-
-            self.ldap_mode = LDAPMode.SIMPLE
-
-            # verify config sanity
-            self.require_keys(ldap_config, [
-                "uri",
-                "base",
-                "attributes",
-            ])
-
-            self.ldap_uri = ldap_config["uri"]
-            self.ldap_start_tls = ldap_config.get("start_tls", False)
-            self.ldap_base = ldap_config["base"]
-            self.ldap_attributes = ldap_config["attributes"]
-
-            if "bind_dn" in ldap_config:
-                self.ldap_mode = LDAPMode.SEARCH
-                self.require_keys(ldap_config, [
-                    "bind_dn",
-                    "bind_password",
-                ])
-
-                self.ldap_bind_dn = ldap_config["bind_dn"]
-                self.ldap_bind_password = ldap_config["bind_password"]
-                self.ldap_filter = ldap_config.get("filter", None)
-
-            # verify attribute lookup
-            self.require_keys(ldap_config['attributes'], [
-                "uid",
-                "name",
-                "mail",
-            ])
-
-    def require_keys(self, config, required):
-        missing = [key for key in required if key not in config]
-        if missing:
-            raise ConfigError(
-                "LDAP enabled but missing required config values: {}".format(
-                    ", ".join(missing)
-                )
-            )
+        ldap_config = config.get("ldap_config", None)
+        if ldap_config:
+            self.ldap_enabled = ldap_config.get("enabled", False)
+            self.ldap_server = ldap_config["server"]
+            self.ldap_port = ldap_config["port"]
+            self.ldap_tls = ldap_config.get("tls", False)
+            self.ldap_search_base = ldap_config["search_base"]
+            self.ldap_search_property = ldap_config["search_property"]
+            self.ldap_email_property = ldap_config["email_property"]
+            self.ldap_full_name_property = ldap_config["full_name_property"]
+        else:
+            self.ldap_enabled = False
+            self.ldap_server = None
+            self.ldap_port = None
+            self.ldap_tls = False
+            self.ldap_search_base = None
+            self.ldap_search_property = None
+            self.ldap_email_property = None
+            self.ldap_full_name_property = None
 
     def default_config(self, **kwargs):
         return """\
         # ldap_config:
         #   enabled: true
-        #   uri: "ldap://ldap.example.com:389"
-        #   start_tls: true
-        #   base: "ou=users,dc=example,dc=com"
-        #   attributes:
-        #      uid: "cn"
-        #      mail: "email"
-        #      name: "givenName"
-        #   #bind_dn:
-        #   #bind_password:
-        #   #filter: "(objectClass=posixAccount)"
+        #   server: "ldap://localhost"
+        #   port: 389
+        #   tls: false
+        #   search_base: "ou=Users,dc=example,dc=com"
+        #   search_property: "cn"
+        #   email_property: "email"
+        #   full_name_property: "givenName"
         """

synapse/config/logger.py

@@ -126,58 +126,54 @@ class LoggingConfig(Config):
                 )
 
     def setup_logging(self):
-        setup_logging(self.log_config, self.log_file, self.verbosity)
+        log_format = (
+            "%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s"
+            " - %(message)s"
+        )
+        if self.log_config is None:
 
+            level = logging.INFO
+            level_for_storage = logging.INFO
+            if self.verbosity:
+                level = logging.DEBUG
+                if self.verbosity > 1:
+                    level_for_storage = logging.DEBUG
 
-def setup_logging(log_config=None, log_file=None, verbosity=None):
-    log_format = (
-        "%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s"
-        " - %(message)s"
-    )
-    if log_config is None:
+            # FIXME: we need a logging.WARN for a -q quiet option
+            logger = logging.getLogger('')
+            logger.setLevel(level)
 
-        level = logging.INFO
-        level_for_storage = logging.INFO
-        if verbosity:
-            level = logging.DEBUG
-            if verbosity > 1:
-                level_for_storage = logging.DEBUG
+            logging.getLogger('synapse.storage').setLevel(level_for_storage)
 
-        # FIXME: we need a logging.WARN for a -q quiet option
-        logger = logging.getLogger('')
-        logger.setLevel(level)
+            formatter = logging.Formatter(log_format)
+            if self.log_file:
+                # TODO: Customisable file size / backup count
+                handler = logging.handlers.RotatingFileHandler(
+                    self.log_file, maxBytes=(1000 * 1000 * 100), backupCount=3
+                )
 
-        logging.getLogger('synapse.storage').setLevel(level_for_storage)
+                def sighup(signum, stack):
+                    logger.info("Closing log file due to SIGHUP")
+                    handler.doRollover()
+                    logger.info("Opened new log file due to SIGHUP")
 
-        formatter = logging.Formatter(log_format)
-        if log_file:
-            # TODO: Customisable file size / backup count
-            handler = logging.handlers.RotatingFileHandler(
-                log_file, maxBytes=(1000 * 1000 * 100), backupCount=3
-            )
+                # TODO(paul): obviously this is a terrible mechanism for
+                #   stealing SIGHUP, because it means no other part of synapse
+                #   can use it instead. If we want to catch SIGHUP anywhere
+                #   else as well, I'd suggest we find a nicer way to broadcast
+                #   it around.
+                if getattr(signal, "SIGHUP"):
+                    signal.signal(signal.SIGHUP, sighup)
+            else:
+                handler = logging.StreamHandler()
+            handler.setFormatter(formatter)
 
-            def sighup(signum, stack):
-                logger.info("Closing log file due to SIGHUP")
-                handler.doRollover()
-                logger.info("Opened new log file due to SIGHUP")
+            handler.addFilter(LoggingContextFilter(request=""))
 
-            # TODO(paul): obviously this is a terrible mechanism for
-            #   stealing SIGHUP, because it means no other part of synapse
-            #   can use it instead. If we want to catch SIGHUP anywhere
-            #   else as well, I'd suggest we find a nicer way to broadcast
-            #   it around.
-            if getattr(signal, "SIGHUP"):
-                signal.signal(signal.SIGHUP, sighup)
+            logger.addHandler(handler)
         else:
-            handler = logging.StreamHandler()
-        handler.setFormatter(formatter)
+            with open(self.log_config, 'r') as f:
+                logging.config.dictConfig(yaml.load(f))
 
-        handler.addFilter(LoggingContextFilter(request=""))
-
-        logger.addHandler(handler)
-    else:
-        with open(log_config, 'r') as f:
-            logging.config.dictConfig(yaml.load(f))
-
-    observer = PythonLoggingObserver()
-    observer.start()
+        observer = PythonLoggingObserver()
+        observer.start()

synapse/config/password.py

@@ -23,14 +23,10 @@ class PasswordConfig(Config):
     def read_config(self, config):
         password_config = config.get("password_config", {})
         self.password_enabled = password_config.get("enabled", True)
-        self.password_pepper = password_config.get("pepper", "")
 
     def default_config(self, config_dir_path, server_name, **kwargs):
         return """
         # Enable password for login.
         password_config:
            enabled: true
-           # Uncomment and change to a secret random string for extra security.
-           # DO NOT CHANGE THIS AFTER INITIAL SETUP!
-           #pepper: ""
         """

synapse/config/registration.py

@@ -32,7 +32,6 @@ class RegistrationConfig(Config):
             )
 
         self.registration_shared_secret = config.get("registration_shared_secret")
-        self.user_creation_max_duration = int(config["user_creation_max_duration"])
 
         self.bcrypt_rounds = config.get("bcrypt_rounds", 12)
         self.trusted_third_party_id_servers = config["trusted_third_party_id_servers"]
@@ -55,11 +54,6 @@ class RegistrationConfig(Config):
         # secret, even if registration is otherwise disabled.
         registration_shared_secret: "%(registration_shared_secret)s"
 
-        # Sets the expiry for the short term user creation in
-        # milliseconds. For instance the bellow duration is two weeks
-        # in milliseconds.
-        user_creation_max_duration: 1209600000
-
         # Set the number of bcrypt rounds used to generate password hash.
         # Larger numbers increase the work factor needed to generate the hash.
         # The default number of rounds is 12.

synapse/config/repository.py

@@ -100,13 +100,8 @@ class ContentRepositoryConfig(Config):
                     "to work"
                 )
 
-            self.url_preview_ip_range_whitelist = IPSet(
-                config.get("url_preview_ip_range_whitelist", ())
-            )
-
-            self.url_preview_url_blacklist = config.get(
-                "url_preview_url_blacklist", ()
-            )
+            if "url_preview_url_blacklist" in config:
+                self.url_preview_url_blacklist = config["url_preview_url_blacklist"]
 
     def default_config(self, **kwargs):
         media_store = self.default_path("media_store")
@@ -167,15 +162,6 @@ class ContentRepositoryConfig(Config):
         # - '10.0.0.0/8'
         # - '172.16.0.0/12'
         # - '192.168.0.0/16'
-        #
-        # List of IP address CIDR ranges that the URL preview spider is allowed
-        # to access even if they are specified in url_preview_ip_range_blacklist.
-        # This is useful for specifying exceptions to wide-ranging blacklisted
-        # target IP ranges - e.g. for enabling URL previews for a specific private
-        # website only visible in your network.
-        #
-        # url_preview_ip_range_whitelist:
-        # - '192.168.1.1'
 
         # Optional list of URL matches that the URL preview spider is
         # denied from accessing.  You should use url_preview_ip_range_blacklist

synapse/config/server.py

@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from ._base import Config, ConfigError
+from ._base import Config
 
 
 class ServerConfig(Config):
@@ -27,19 +27,11 @@ class ServerConfig(Config):
         self.daemonize = config.get("daemonize")
         self.print_pidfile = config.get("print_pidfile")
         self.user_agent_suffix = config.get("user_agent_suffix")
-        self.use_frozen_dicts = config.get("use_frozen_dicts", False)
-        self.public_baseurl = config.get("public_baseurl")
-        self.secondary_directory_servers = config.get("secondary_directory_servers", [])
-
-        if self.public_baseurl is not None:
-            if self.public_baseurl[-1] != '/':
-                self.public_baseurl += '/'
+        self.use_frozen_dicts = config.get("use_frozen_dicts", True)
         self.start_pushers = config.get("start_pushers", True)
 
         self.listeners = config.get("listeners", [])
 
-        self.gc_thresholds = read_gc_thresholds(config.get("gc_thresholds", None))
-
         bind_port = config.get("bind_port")
         if bind_port:
             self.listeners = []
@@ -107,6 +99,26 @@ class ServerConfig(Config):
                 ]
             })
 
+        # Attempt to guess the content_addr for the v0 content repostitory
+        content_addr = config.get("content_addr")
+        if not content_addr:
+            for listener in self.listeners:
+                if listener["type"] == "http" and not listener.get("tls", False):
+                    unsecure_port = listener["port"]
+                    break
+            else:
+                raise RuntimeError("Could not determine 'content_addr'")
+
+            host = self.server_name
+            if ':' not in host:
+                host = "%s:%d" % (host, unsecure_port)
+            else:
+                host = host.split(':')[0]
+                host = "%s:%d" % (host, unsecure_port)
+            content_addr = "http://%s" % (host,)
+
+        self.content_addr = content_addr
+
     def default_config(self, server_name, **kwargs):
         if ":" in server_name:
             bind_port = int(server_name.split(":")[1])
@@ -131,25 +143,11 @@ class ServerConfig(Config):
         # Whether to serve a web client from the HTTP/HTTPS root resource.
         web_client: True
 
-        # The public-facing base URL for the client API (not including _matrix/...)
-        # public_baseurl: https://example.com:8448/
-
         # Set the soft limit on the number of file descriptors synapse can use
         # Zero is used to indicate synapse should set the soft limit to the
         # hard limit.
         soft_file_limit: 0
 
-        # The GC threshold parameters to pass to `gc.set_threshold`, if defined
-        # gc_thresholds: [700, 10, 10]
-
-        # A list of other Home Servers to fetch the public room directory from
-        # and include in the public room directory of this home server
-        # This is a temporary stopgap solution to populate new server with a
-        # list of rooms until there exists a good solution of a decentralized
-        # room directory.
-        # secondary_directory_servers:
-        #     - matrix.org
-
         # List of ports that Synapse should listen on, their purpose and their
         # configuration.
         listeners:
@@ -231,20 +229,3 @@ class ServerConfig(Config):
                                   type=int,
                                   help="Turn on the twisted telnet manhole"
                                   " service on the given port.")
-
-
-def read_gc_thresholds(thresholds):
-    """Reads the three integer thresholds for garbage collection. Ensures that
-    the thresholds are integers if thresholds are supplied.
-    """
-    if thresholds is None:
-        return None
-    try:
-        assert len(thresholds) == 3
-        return (
-            int(thresholds[0]), int(thresholds[1]), int(thresholds[2]),
-        )
-    except:
-        raise ConfigError(
-            "Value of `gc_threshold` must be a list of three integers if set"
-        )

synapse/config/workers.py

@@ -1,31 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2016 matrix.org
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from ._base import Config
-
-
-class WorkerConfig(Config):
-    """The workers are processes run separately to the main synapse process.
-    They have their own pid_file and listener configuration. They use the
-    replication_url to talk to the main synapse process."""
-
-    def read_config(self, config):
-        self.worker_app = config.get("worker_app")
-        self.worker_listeners = config.get("worker_listeners")
-        self.worker_daemonize = config.get("worker_daemonize")
-        self.worker_pid_file = config.get("worker_pid_file")
-        self.worker_log_file = config.get("worker_log_file")
-        self.worker_log_config = config.get("worker_log_config")
-        self.worker_replication_url = config.get("worker_replication_url")

synapse/federation/federation_base.py

@@ -31,9 +31,6 @@ logger = logging.getLogger(__name__)
 
 
 class FederationBase(object):
-    def __init__(self, hs):
-        pass
-
     @defer.inlineCallbacks
     def _check_sigs_and_hash_and_fetch(self, origin, pdus, outlier=False,
                                        include_none=False):

synapse/federation/federation_client.py

@@ -24,7 +24,6 @@ from synapse.api.errors import (
     CodeMessageException, HttpResponseException, SynapseError,
 )
 from synapse.util import unwrapFirstError
-from synapse.util.async import concurrently_execute
 from synapse.util.caches.expiringcache import ExpiringCache
 from synapse.util.logutils import log_function
 from synapse.events import FrozenEvent
@@ -52,8 +51,6 @@ sent_queries_counter = metrics.register_counter("sent_queries", labels=["type"])
 
 
 class FederationClient(FederationBase):
-    def __init__(self, hs):
-        super(FederationClient, self).__init__(hs)
 
     def start_get_pdu_cache(self):
         self._get_pdu_cache = ExpiringCache(
@@ -553,25 +550,6 @@ class FederationClient(FederationBase):
 
         raise RuntimeError("Failed to send to any server.")
 
-    @defer.inlineCallbacks
-    def get_public_rooms(self, destinations):
-        results_by_server = {}
-
-        @defer.inlineCallbacks
-        def _get_result(s):
-            if s == self.server_name:
-                defer.returnValue()
-
-            try:
-                result = yield self.transport_layer.get_public_rooms(s)
-                results_by_server[s] = result
-            except:
-                logger.exception("Error getting room list from server %r", s)
-
-        yield concurrently_execute(_get_result, destinations, 3)
-
-        defer.returnValue(results_by_server)
-
     @defer.inlineCallbacks
     def query_auth(self, destination, room_id, event_id, local_auth):
         """

synapse/federation/federation_server.py

@@ -19,7 +19,6 @@ from twisted.internet import defer
 from .federation_base import FederationBase
 from .units import Transaction, Edu
 
-from synapse.util.async import Linearizer
 from synapse.util.logutils import log_function
 from synapse.events import FrozenEvent
 import synapse.metrics
@@ -45,12 +44,6 @@ received_queries_counter = metrics.register_counter("received_queries", labels=[
 
 
 class FederationServer(FederationBase):
-    def __init__(self, hs):
-        super(FederationServer, self).__init__(hs)
-
-        self._room_pdu_linearizer = Linearizer()
-        self._server_linearizer = Linearizer()
-
     def set_handler(self, handler):
         """Sets the handler that the replication layer will use to communicate
         receipt of new PDUs from other home servers. The required methods are
@@ -90,14 +83,11 @@ class FederationServer(FederationBase):
     @defer.inlineCallbacks
     @log_function
     def on_backfill_request(self, origin, room_id, versions, limit):
-        with (yield self._server_linearizer.queue((origin, room_id))):
-            pdus = yield self.handler.on_backfill_request(
-                origin, room_id, versions, limit
-            )
+        pdus = yield self.handler.on_backfill_request(
+            origin, room_id, versions, limit
+        )
 
-            res = self._transaction_from_pdus(pdus).get_dict()
-
-        defer.returnValue((200, res))
+        defer.returnValue((200, self._transaction_from_pdus(pdus).get_dict()))
 
     @defer.inlineCallbacks
     @log_function
@@ -188,28 +178,24 @@ class FederationServer(FederationBase):
     @defer.inlineCallbacks
     @log_function
     def on_context_state_request(self, origin, room_id, event_id):
-        with (yield self._server_linearizer.queue((origin, room_id))):
-            if event_id:
-                pdus = yield self.handler.get_state_for_pdu(
-                    origin, room_id, event_id,
-                )
-                auth_chain = yield self.store.get_auth_chain(
-                    [pdu.event_id for pdu in pdus]
-                )
+        if event_id:
+            pdus = yield self.handler.get_state_for_pdu(
+                origin, room_id, event_id,
+            )
+            auth_chain = yield self.store.get_auth_chain(
+                [pdu.event_id for pdu in pdus]
+            )
 
-                for event in auth_chain:
-                    # We sign these again because there was a bug where we
-                    # incorrectly signed things the first time round
-                    if self.hs.is_mine_id(event.event_id):
-                        event.signatures.update(
-                            compute_event_signature(
-                                event,
-                                self.hs.hostname,
-                                self.hs.config.signing_key[0]
-                            )
-                        )
-            else:
-                raise NotImplementedError("Specify an event")
+            for event in auth_chain:
+                event.signatures.update(
+                    compute_event_signature(
+                        event,
+                        self.hs.hostname,
+                        self.hs.config.signing_key[0]
+                    )
+                )
+        else:
+            raise NotImplementedError("Specify an event")
 
         defer.returnValue((200, {
             "pdus": [pdu.get_pdu_json() for pdu in pdus],
@@ -288,16 +274,14 @@ class FederationServer(FederationBase):
 
     @defer.inlineCallbacks
     def on_event_auth(self, origin, room_id, event_id):
-        with (yield self._server_linearizer.queue((origin, room_id))):
-            time_now = self._clock.time_msec()
-            auth_pdus = yield self.handler.on_event_auth(event_id)
-            res = {
-                "auth_chain": [a.get_pdu_json(time_now) for a in auth_pdus],
-            }
-        defer.returnValue((200, res))
+        time_now = self._clock.time_msec()
+        auth_pdus = yield self.handler.on_event_auth(event_id)
+        defer.returnValue((200, {
+            "auth_chain": [a.get_pdu_json(time_now) for a in auth_pdus],
+        }))
 
     @defer.inlineCallbacks
-    def on_query_auth_request(self, origin, content, room_id, event_id):
+    def on_query_auth_request(self, origin, content, event_id):
         """
         Content is a dict with keys::
             auth_chain (list): A list of events that give the auth chain.
@@ -316,33 +300,32 @@ class FederationServer(FederationBase):
         Returns:
             Deferred: Results in `dict` with the same format as `content`
         """
-        with (yield self._server_linearizer.queue((origin, room_id))):
-            auth_chain = [
-                self.event_from_pdu_json(e)
-                for e in content["auth_chain"]
-            ]
+        auth_chain = [
+            self.event_from_pdu_json(e)
+            for e in content["auth_chain"]
+        ]
 
-            signed_auth = yield self._check_sigs_and_hash_and_fetch(
-                origin, auth_chain, outlier=True
-            )
+        signed_auth = yield self._check_sigs_and_hash_and_fetch(
+            origin, auth_chain, outlier=True
+        )
 
-            ret = yield self.handler.on_query_auth(
-                origin,
-                event_id,
-                signed_auth,
-                content.get("rejects", []),
-                content.get("missing", []),
-            )
+        ret = yield self.handler.on_query_auth(
+            origin,
+            event_id,
+            signed_auth,
+            content.get("rejects", []),
+            content.get("missing", []),
+        )
 
-            time_now = self._clock.time_msec()
-            send_content = {
-                "auth_chain": [
-                    e.get_pdu_json(time_now)
-                    for e in ret["auth_chain"]
-                ],
-                "rejects": ret.get("rejects", []),
-                "missing": ret.get("missing", []),
-            }
+        time_now = self._clock.time_msec()
+        send_content = {
+            "auth_chain": [
+                e.get_pdu_json(time_now)
+                for e in ret["auth_chain"]
+            ],
+            "rejects": ret.get("rejects", []),
+            "missing": ret.get("missing", []),
+        }
 
         defer.returnValue(
             (200, send_content)
@@ -394,34 +377,16 @@ class FederationServer(FederationBase):
     @log_function
     def on_get_missing_events(self, origin, room_id, earliest_events,
                               latest_events, limit, min_depth):
-        with (yield self._server_linearizer.queue((origin, room_id))):
-            logger.info(
-                "on_get_missing_events: earliest_events: %r, latest_events: %r,"
-                " limit: %d, min_depth: %d",
-                earliest_events, latest_events, limit, min_depth
-            )
-            missing_events = yield self.handler.on_get_missing_events(
-                origin, room_id, earliest_events, latest_events, limit, min_depth
-            )
+        missing_events = yield self.handler.on_get_missing_events(
+            origin, room_id, earliest_events, latest_events, limit, min_depth
+        )
 
-            if len(missing_events) < 5:
-                logger.info(
-                    "Returning %d events: %r", len(missing_events), missing_events
-                )
-            else:
-                logger.info("Returning %d events", len(missing_events))
-
-            time_now = self._clock.time_msec()
+        time_now = self._clock.time_msec()
 
         defer.returnValue({
             "events": [ev.get_pdu_json(time_now) for ev in missing_events],
         })
 
-    @log_function
-    def on_openid_userinfo(self, token):
-        ts_now_ms = self._clock.time_msec()
-        return self.store.get_user_id_for_open_id_token(token, ts_now_ms)
-
     @log_function
     def _get_persisted_pdu(self, origin, event_id, do_auth=True):
         """ Get a PDU from the database with given origin and id.
@@ -511,59 +476,42 @@ class FederationServer(FederationBase):
                 pdu.internal_metadata.outlier = True
             elif min_depth and pdu.depth > min_depth:
                 if get_missing and prevs - seen:
-                    # If we're missing stuff, ensure we only fetch stuff one
-                    # at a time.
-                    with (yield self._room_pdu_linearizer.queue(pdu.room_id)):
-                        # We recalculate seen, since it may have changed.
-                        have_seen = yield self.store.have_events(prevs)
-                        seen = set(have_seen.keys())
+                    latest = yield self.store.get_latest_event_ids_in_room(
+                        pdu.room_id
+                    )
 
-                        if prevs - seen:
-                            latest = yield self.store.get_latest_event_ids_in_room(
-                                pdu.room_id
-                            )
+                    # We add the prev events that we have seen to the latest
+                    # list to ensure the remote server doesn't give them to us
+                    latest = set(latest)
+                    latest |= seen
 
-                            # We add the prev events that we have seen to the latest
-                            # list to ensure the remote server doesn't give them to us
-                            latest = set(latest)
-                            latest |= seen
+                    missing_events = yield self.get_missing_events(
+                        origin,
+                        pdu.room_id,
+                        earliest_events_ids=list(latest),
+                        latest_events=[pdu],
+                        limit=10,
+                        min_depth=min_depth,
+                    )
 
-                            logger.info(
-                                "Missing %d events for room %r: %r...",
-                                len(prevs - seen), pdu.room_id, list(prevs - seen)[:5]
-                            )
+                    # We want to sort these by depth so we process them and
+                    # tell clients about them in order.
+                    missing_events.sort(key=lambda x: x.depth)
 
-                            missing_events = yield self.get_missing_events(
-                                origin,
-                                pdu.room_id,
-                                earliest_events_ids=list(latest),
-                                latest_events=[pdu],
-                                limit=10,
-                                min_depth=min_depth,
-                            )
+                    for e in missing_events:
+                        yield self._handle_new_pdu(
+                            origin,
+                            e,
+                            get_missing=False
+                        )
 
-                            # We want to sort these by depth so we process them and
-                            # tell clients about them in order.
-                            missing_events.sort(key=lambda x: x.depth)
-
-                            for e in missing_events:
-                                yield self._handle_new_pdu(
-                                    origin,
-                                    e,
-                                    get_missing=False
-                                )
-
-                            have_seen = yield self.store.have_events(
-                                [ev for ev, _ in pdu.prev_events]
-                            )
+                    have_seen = yield self.store.have_events(
+                        [ev for ev, _ in pdu.prev_events]
+                    )
 
             prevs = {e_id for e_id, _ in pdu.prev_events}
             seen = set(have_seen.keys())
             if prevs - seen:
-                logger.info(
-                    "Still missing %d events for room %r: %r...",
-                    len(prevs - seen), pdu.room_id, list(prevs - seen)[:5]
-                )
                 fetch_state = True
 
         if fetch_state:

synapse/federation/replication.py

@@ -72,7 +72,5 @@ class ReplicationLayer(FederationClient, FederationServer):
 
         self.hs = hs
 
-        super(ReplicationLayer, self).__init__(hs)
-
     def __str__(self):
         return "<ReplicationLayer(%s)>" % self.server_name

synapse/federation/transaction_queue.py

@@ -20,7 +20,6 @@ from .persistence import TransactionActions
 from .units import Transaction
 
 from synapse.api.errors import HttpResponseException
-from synapse.util.async import run_on_reactor
 from synapse.util.logutils import log_function
 from synapse.util.logcontext import PreserveLoggingContext
 from synapse.util.retryutils import (
@@ -200,8 +199,6 @@ class TransactionQueue(object):
     @defer.inlineCallbacks
     @log_function
     def _attempt_new_transaction(self, destination):
-        yield run_on_reactor()
-
         # list of (pending_pdu, deferred, order)
         if destination in self.pending_transactions:
             # XXX: pending_transactions can get stuck on by a never-ending

synapse/federation/transport/client.py

@@ -224,18 +224,6 @@ class TransportLayerClient(object):
 
         defer.returnValue(response)
 
-    @defer.inlineCallbacks
-    @log_function
-    def get_public_rooms(self, remote_server):
-        path = PREFIX + "/publicRooms"
-
-        response = yield self.client.get_json(
-            destination=remote_server,
-            path=path,
-        )
-
-        defer.returnValue(response)
-
     @defer.inlineCallbacks
     @log_function
     def exchange_third_party_invite(self, destination, room_id, event_dict):

synapse/federation/transport/server.py

@@ -18,7 +18,7 @@ from twisted.internet import defer
 from synapse.api.urls import FEDERATION_PREFIX as PREFIX
 from synapse.api.errors import Codes, SynapseError
 from synapse.http.server import JsonResource
-from synapse.http.servlet import parse_json_object_from_request, parse_string
+from synapse.http.servlet import parse_json_object_from_request
 from synapse.util.ratelimitutils import FederationRateLimiter
 
 import functools
@@ -37,7 +37,7 @@ class TransportLayerServer(JsonResource):
         self.hs = hs
         self.clock = hs.get_clock()
 
-        super(TransportLayerServer, self).__init__(hs, canonical_json=False)
+        super(TransportLayerServer, self).__init__(hs)
 
         self.authenticator = Authenticator(hs)
         self.ratelimiter = FederationRateLimiter(
@@ -134,12 +134,10 @@ class Authenticator(object):
 
 
 class BaseFederationServlet(object):
-    def __init__(self, handler, authenticator, ratelimiter, server_name,
-                 room_list_handler):
+    def __init__(self, handler, authenticator, ratelimiter, server_name):
         self.handler = handler
         self.authenticator = authenticator
         self.ratelimiter = ratelimiter
-        self.room_list_handler = room_list_handler
 
     def _wrap(self, code):
         authenticator = self.authenticator
@@ -325,7 +323,7 @@ class FederationSendLeaveServlet(BaseFederationServlet):
 
 
 class FederationEventAuthServlet(BaseFederationServlet):
-    PATH = "/event_auth/(?P<context>[^/]*)/(?P<event_id>[^/]*)"
+    PATH = "/event_auth(?P<context>[^/]*)/(?P<event_id>[^/]*)"
 
     def on_GET(self, origin, content, query, context, event_id):
         return self.handler.on_event_auth(origin, context, event_id)
@@ -388,7 +386,7 @@ class FederationQueryAuthServlet(BaseFederationServlet):
     @defer.inlineCallbacks
     def on_POST(self, origin, content, query, context, event_id):
         new_content = yield self.handler.on_query_auth_request(
-            origin, content, context, event_id
+            origin, content, event_id
         )
 
         defer.returnValue((200, new_content))
@@ -450,89 +448,6 @@ class On3pidBindServlet(BaseFederationServlet):
         return code
 
 
-class OpenIdUserInfo(BaseFederationServlet):
-    """
-    Exchange a bearer token for information about a user.
-
-    The response format should be compatible with:
-        http://openid.net/specs/openid-connect-core-1_0.html#UserInfoResponse
-
-    GET /openid/userinfo?access_token=ABDEFGH HTTP/1.1
-
-    HTTP/1.1 200 OK
-    Content-Type: application/json
-
-    {
-        "sub": "@userpart:example.org",
-    }
-    """
-
-    PATH = "/openid/userinfo"
-
-    @defer.inlineCallbacks
-    def on_GET(self, request):
-        token = parse_string(request, "access_token")
-        if token is None:
-            defer.returnValue((401, {
-                "errcode": "M_MISSING_TOKEN", "error": "Access Token required"
-            }))
-            return
-
-        user_id = yield self.handler.on_openid_userinfo(token)
-
-        if user_id is None:
-            defer.returnValue((401, {
-                "errcode": "M_UNKNOWN_TOKEN",
-                "error": "Access Token unknown or expired"
-            }))
-
-        defer.returnValue((200, {"sub": user_id}))
-
-    # Avoid doing remote HS authorization checks which are done by default by
-    # BaseFederationServlet.
-    def _wrap(self, code):
-        return code
-
-
-class PublicRoomList(BaseFederationServlet):
-    """
-    Fetch the public room list for this server.
-
-    This API returns information in the same format as /publicRooms on the
-    client API, but will only ever include local public rooms and hence is
-    intended for consumption by other home servers.
-
-    GET /publicRooms HTTP/1.1
-
-    HTTP/1.1 200 OK
-    Content-Type: application/json
-
-    {
-        "chunk": [
-            {
-                "aliases": [
-                    "#test:localhost"
-                ],
-                "guest_can_join": false,
-                "name": "test room",
-                "num_joined_members": 3,
-                "room_id": "!whkydVegtvatLfXmPN:localhost",
-                "world_readable": false
-            }
-        ],
-        "end": "END",
-        "start": "START"
-    }
-    """
-
-    PATH = "/publicRooms"
-
-    @defer.inlineCallbacks
-    def on_GET(self, origin, content, query):
-        data = yield self.room_list_handler.get_local_public_room_list()
-        defer.returnValue((200, data))
-
-
 SERVLET_CLASSES = (
     FederationSendServlet,
     FederationPullServlet,
@@ -553,8 +468,6 @@ SERVLET_CLASSES = (
     FederationClientKeysClaimServlet,
     FederationThirdPartyInviteExchangeServlet,
     On3pidBindServlet,
-    OpenIdUserInfo,
-    PublicRoomList,
 )
 
 
@@ -565,5 +478,4 @@ def register_servlets(hs, resource, authenticator, ratelimiter):
             authenticator=authenticator,
             ratelimiter=ratelimiter,
             server_name=hs.hostname,
-            room_list_handler=hs.get_room_list_handler(),
         ).register(resource)

synapse/handlers/__init__.py

@@ -13,17 +13,24 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from synapse.appservice.scheduler import AppServiceScheduler
+from synapse.appservice.api import ApplicationServiceApi
 from .register import RegistrationHandler
 from .room import (
-    RoomCreationHandler, RoomContextHandler,
+    RoomCreationHandler, RoomListHandler, RoomContextHandler,
 )
 from .room_member import RoomMemberHandler
 from .message import MessageHandler
 from .events import EventStreamHandler, EventHandler
 from .federation import FederationHandler
 from .profile import ProfileHandler
+from .presence import PresenceHandler
 from .directory import DirectoryHandler
+from .typing import TypingNotificationHandler
 from .admin import AdminHandler
+from .appservice import ApplicationServicesHandler
+from .sync import SyncHandler
+from .auth import AuthHandler
 from .identity import IdentityHandler
 from .receipts import ReceiptsHandler
 from .search import SearchHandler
@@ -46,9 +53,22 @@ class Handlers(object):
         self.event_handler = EventHandler(hs)
         self.federation_handler = FederationHandler(hs)
         self.profile_handler = ProfileHandler(hs)
+        self.presence_handler = PresenceHandler(hs)
+        self.room_list_handler = RoomListHandler(hs)
         self.directory_handler = DirectoryHandler(hs)
+        self.typing_notification_handler = TypingNotificationHandler(hs)
         self.admin_handler = AdminHandler(hs)
         self.receipts_handler = ReceiptsHandler(hs)
+        asapi = ApplicationServiceApi(hs)
+        self.appservice_handler = ApplicationServicesHandler(
+            hs, asapi, AppServiceScheduler(
+                clock=hs.get_clock(),
+                store=hs.get_datastore(),
+                as_api=asapi
+            )
+        )
+        self.sync_handler = SyncHandler(hs)
+        self.auth_handler = AuthHandler(hs)
         self.identity_handler = IdentityHandler(hs)
         self.search_handler = SearchHandler(hs)
         self.room_context_handler = RoomContextHandler(hs)

synapse/handlers/_base.py

@@ -15,10 +15,13 @@
 
 from twisted.internet import defer
 
-from synapse.api.errors import LimitExceededError
+from synapse.api.errors import LimitExceededError, SynapseError, AuthError
+from synapse.crypto.event_signing import add_hashes_and_signatures
 from synapse.api.constants import Membership, EventTypes
-from synapse.types import UserID, Requester
+from synapse.types import UserID, RoomAlias, Requester
+from synapse.push.action_generator import ActionGenerator
 
+from synapse.util.logcontext import PreserveLoggingContext, preserve_fn
 
 import logging
 
@@ -26,6 +29,23 @@ import logging
 logger = logging.getLogger(__name__)
 
 
+VISIBILITY_PRIORITY = (
+    "world_readable",
+    "shared",
+    "invited",
+    "joined",
+)
+
+
+MEMBERSHIP_PRIORITY = (
+    Membership.JOIN,
+    Membership.INVITE,
+    Membership.KNOCK,
+    Membership.LEAVE,
+    Membership.BAN,
+)
+
+
 class BaseHandler(object):
     """
     Common base class for the event handlers.
@@ -45,10 +65,161 @@ class BaseHandler(object):
         self.clock = hs.get_clock()
         self.hs = hs
 
+        self.signing_key = hs.config.signing_key[0]
         self.server_name = hs.hostname
 
         self.event_builder_factory = hs.get_event_builder_factory()
 
+    @defer.inlineCallbacks
+    def filter_events_for_clients(self, user_tuples, events, event_id_to_state):
+        """ Returns dict of user_id -> list of events that user is allowed to
+        see.
+
+        Args:
+            user_tuples (str, bool): (user id, is_peeking) for each user to be
+                checked. is_peeking should be true if:
+                * the user is not currently a member of the room, and:
+                * the user has not been a member of the room since the
+                given events
+            events ([synapse.events.EventBase]): list of events to filter
+        """
+        forgotten = yield defer.gatherResults([
+            self.store.who_forgot_in_room(
+                room_id,
+            )
+            for room_id in frozenset(e.room_id for e in events)
+        ], consumeErrors=True)
+
+        # Set of membership event_ids that have been forgotten
+        event_id_forgotten = frozenset(
+            row["event_id"] for rows in forgotten for row in rows
+        )
+
+        def allowed(event, user_id, is_peeking):
+            """
+            Args:
+                event (synapse.events.EventBase): event to check
+                user_id (str)
+                is_peeking (bool)
+            """
+            state = event_id_to_state[event.event_id]
+
+            # get the room_visibility at the time of the event.
+            visibility_event = state.get((EventTypes.RoomHistoryVisibility, ""), None)
+            if visibility_event:
+                visibility = visibility_event.content.get("history_visibility", "shared")
+            else:
+                visibility = "shared"
+
+            if visibility not in VISIBILITY_PRIORITY:
+                visibility = "shared"
+
+            # if it was world_readable, it's easy: everyone can read it
+            if visibility == "world_readable":
+                return True
+
+            # Always allow history visibility events on boundaries. This is done
+            # by setting the effective visibility to the least restrictive
+            # of the old vs new.
+            if event.type == EventTypes.RoomHistoryVisibility:
+                prev_content = event.unsigned.get("prev_content", {})
+                prev_visibility = prev_content.get("history_visibility", None)
+
+                if prev_visibility not in VISIBILITY_PRIORITY:
+                    prev_visibility = "shared"
+
+                new_priority = VISIBILITY_PRIORITY.index(visibility)
+                old_priority = VISIBILITY_PRIORITY.index(prev_visibility)
+                if old_priority < new_priority:
+                    visibility = prev_visibility
+
+            # likewise, if the event is the user's own membership event, use
+            # the 'most joined' membership
+            membership = None
+            if event.type == EventTypes.Member and event.state_key == user_id:
+                membership = event.content.get("membership", None)
+                if membership not in MEMBERSHIP_PRIORITY:
+                    membership = "leave"
+
+                prev_content = event.unsigned.get("prev_content", {})
+                prev_membership = prev_content.get("membership", None)
+                if prev_membership not in MEMBERSHIP_PRIORITY:
+                    prev_membership = "leave"
+
+                new_priority = MEMBERSHIP_PRIORITY.index(membership)
+                old_priority = MEMBERSHIP_PRIORITY.index(prev_membership)
+                if old_priority < new_priority:
+                    membership = prev_membership
+
+            # otherwise, get the user's membership at the time of the event.
+            if membership is None:
+                membership_event = state.get((EventTypes.Member, user_id), None)
+                if membership_event:
+                    if membership_event.event_id not in event_id_forgotten:
+                        membership = membership_event.membership
+
+            # if the user was a member of the room at the time of the event,
+            # they can see it.
+            if membership == Membership.JOIN:
+                return True
+
+            if visibility == "joined":
+                # we weren't a member at the time of the event, so we can't
+                # see this event.
+                return False
+
+            elif visibility == "invited":
+                # user can also see the event if they were *invited* at the time
+                # of the event.
+                return membership == Membership.INVITE
+
+            else:
+                # visibility is shared: user can also see the event if they have
+                # become a member since the event
+                #
+                # XXX: if the user has subsequently joined and then left again,
+                # ideally we would share history up to the point they left. But
+                # we don't know when they left.
+                return not is_peeking
+
+        defer.returnValue({
+            user_id: [
+                event
+                for event in events
+                if allowed(event, user_id, is_peeking)
+            ]
+            for user_id, is_peeking in user_tuples
+        })
+
+    @defer.inlineCallbacks
+    def _filter_events_for_client(self, user_id, events, is_peeking=False):
+        """
+        Check which events a user is allowed to see
+
+        Args:
+            user_id(str): user id to be checked
+            events([synapse.events.EventBase]): list of events to be checked
+            is_peeking(bool): should be True if:
+              * the user is not currently a member of the room, and:
+              * the user has not been a member of the room since the given
+                events
+
+        Returns:
+            [synapse.events.EventBase]
+        """
+        types = (
+            (EventTypes.RoomHistoryVisibility, ""),
+            (EventTypes.Member, user_id),
+        )
+        event_id_to_state = yield self.store.get_state_for_events(
+            frozenset(e.event_id for e in events),
+            types=types
+        )
+        res = yield self.filter_events_for_clients(
+            [(user_id, is_peeking)], events, event_id_to_state
+        )
+        defer.returnValue(res.get(user_id, []))
+
     def ratelimit(self, requester):
         time_now = self.clock.time()
         allowed, time_allowed = self.ratelimiter.send_message(
@@ -61,6 +232,56 @@ class BaseHandler(object):
                 retry_after_ms=int(1000 * (time_allowed - time_now)),
             )
 
+    @defer.inlineCallbacks
+    def _create_new_client_event(self, builder, prev_event_ids=None):
+        if prev_event_ids:
+            prev_events = yield self.store.add_event_hashes(prev_event_ids)
+            prev_max_depth = yield self.store.get_max_depth_of_events(prev_event_ids)
+            depth = prev_max_depth + 1
+        else:
+            latest_ret = yield self.store.get_latest_event_ids_and_hashes_in_room(
+                builder.room_id,
+            )
+
+            if latest_ret:
+                depth = max([d for _, _, d in latest_ret]) + 1
+            else:
+                depth = 1
+
+            prev_events = [
+                (event_id, prev_hashes)
+                for event_id, prev_hashes, _ in latest_ret
+            ]
+
+        builder.prev_events = prev_events
+        builder.depth = depth
+
+        state_handler = self.state_handler
+
+        context = yield state_handler.compute_event_context(builder)
+
+        if builder.is_state():
+            builder.prev_state = yield self.store.add_event_hashes(
+                context.prev_state_events
+            )
+
+        yield self.auth.add_auth_events(builder, context)
+
+        add_hashes_and_signatures(
+            builder, self.server_name, self.signing_key
+        )
+
+        event = builder.build()
+
+        logger.debug(
+            "Created event %s with current state: %s",
+            event.event_id, context.current_state,
+        )
+
+        defer.returnValue(
+            (event, context,)
+        )
+
     def is_host_in_room(self, current_state):
         room_members = [
             (state_key, event.membership)
@@ -75,12 +296,153 @@ class BaseHandler(object):
                 return True
         for (state_key, membership) in room_members:
             if (
-                self.hs.is_mine_id(state_key)
+                UserID.from_string(state_key).domain == self.hs.hostname
                 and membership == Membership.JOIN
             ):
                 return True
         return False
 
+    @defer.inlineCallbacks
+    def handle_new_client_event(
+        self,
+        requester,
+        event,
+        context,
+        ratelimit=True,
+        extra_users=[]
+    ):
+        # We now need to go and hit out to wherever we need to hit out to.
+
+        if ratelimit:
+            self.ratelimit(requester)
+
+        try:
+            self.auth.check(event, auth_events=context.current_state)
+        except AuthError as err:
+            logger.warn("Denying new event %r because %s", event, err)
+            raise err
+
+        yield self.maybe_kick_guest_users(event, context.current_state.values())
+
+        if event.type == EventTypes.CanonicalAlias:
+            # Check the alias is acually valid (at this time at least)
+            room_alias_str = event.content.get("alias", None)
+            if room_alias_str:
+                room_alias = RoomAlias.from_string(room_alias_str)
+                directory_handler = self.hs.get_handlers().directory_handler
+                mapping = yield directory_handler.get_association(room_alias)
+
+                if mapping["room_id"] != event.room_id:
+                    raise SynapseError(
+                        400,
+                        "Room alias %s does not point to the room" % (
+                            room_alias_str,
+                        )
+                    )
+
+        federation_handler = self.hs.get_handlers().federation_handler
+
+        if event.type == EventTypes.Member:
+            if event.content["membership"] == Membership.INVITE:
+                def is_inviter_member_event(e):
+                    return (
+                        e.type == EventTypes.Member and
+                        e.sender == event.sender
+                    )
+
+                event.unsigned["invite_room_state"] = [
+                    {
+                        "type": e.type,
+                        "state_key": e.state_key,
+                        "content": e.content,
+                        "sender": e.sender,
+                    }
+                    for k, e in context.current_state.items()
+                    if e.type in self.hs.config.room_invite_state_types
+                    or is_inviter_member_event(e)
+                ]
+
+                invitee = UserID.from_string(event.state_key)
+                if not self.hs.is_mine(invitee):
+                    # TODO: Can we add signature from remote server in a nicer
+                    # way? If we have been invited by a remote server, we need
+                    # to get them to sign the event.
+
+                    returned_invite = yield federation_handler.send_invite(
+                        invitee.domain,
+                        event,
+                    )
+
+                    event.unsigned.pop("room_state", None)
+
+                    # TODO: Make sure the signatures actually are correct.
+                    event.signatures.update(
+                        returned_invite.signatures
+                    )
+
+        if event.type == EventTypes.Redaction:
+            if self.auth.check_redaction(event, auth_events=context.current_state):
+                original_event = yield self.store.get_event(
+                    event.redacts,
+                    check_redacted=False,
+                    get_prev_content=False,
+                    allow_rejected=False,
+                    allow_none=False
+                )
+                if event.user_id != original_event.user_id:
+                    raise AuthError(
+                        403,
+                        "You don't have permission to redact events"
+                    )
+
+        if event.type == EventTypes.Create and context.current_state:
+            raise AuthError(
+                403,
+                "Changing the room create event is forbidden",
+            )
+
+        action_generator = ActionGenerator(self.hs)
+        yield action_generator.handle_push_actions_for_event(
+            event, context, self
+        )
+
+        (event_stream_id, max_stream_id) = yield self.store.persist_event(
+            event, context=context
+        )
+
+        # this intentionally does not yield: we don't care about the result
+        # and don't need to wait for it.
+        preserve_fn(self.hs.get_pusherpool().on_new_notifications)(
+            event_stream_id, max_stream_id
+        )
+
+        destinations = set()
+        for k, s in context.current_state.items():
+            try:
+                if k[0] == EventTypes.Member:
+                    if s.content["membership"] == Membership.JOIN:
+                        destinations.add(
+                            UserID.from_string(s.state_key).domain
+                        )
+            except SynapseError:
+                logger.warn(
+                    "Failed to get destination from event %s", s.event_id
+                )
+
+        with PreserveLoggingContext():
+            # Don't block waiting on waking up all the listeners.
+            self.notifier.on_new_room_event(
+                event, event_stream_id, max_stream_id,
+                extra_users=extra_users
+            )
+
+        # If invite, remove room_state from unsigned before sending.
+        event.unsigned.pop("invite_room_state", None)
+
+        federation_handler.handle_new_event(
+            event, destinations=destinations,
+        )
+
     @defer.inlineCallbacks
     def maybe_kick_guest_users(self, event, current_state):
         # Technically this function invalidates current_state by changing it.

synapse/handlers/appservice.py

@@ -17,6 +17,7 @@ from twisted.internet import defer
 
 from synapse.api.constants import EventTypes
 from synapse.appservice import ApplicationService
+from synapse.types import UserID
 
 import logging
 
@@ -34,13 +35,16 @@ def log_failure(failure):
     )
 
 
+# NB: Purposefully not inheriting BaseHandler since that contains way too much
+# setup code which this handler does not need or use. This makes testing a lot
+# easier.
 class ApplicationServicesHandler(object):
 
-    def __init__(self, hs):
+    def __init__(self, hs, appservice_api, appservice_scheduler):
         self.store = hs.get_datastore()
-        self.is_mine_id = hs.is_mine_id
-        self.appservice_api = hs.get_application_service_api()
-        self.scheduler = hs.get_application_service_scheduler()
+        self.hs = hs
+        self.appservice_api = appservice_api
+        self.scheduler = appservice_scheduler
         self.started_scheduler = False
 
     @defer.inlineCallbacks
@@ -165,7 +169,8 @@ class ApplicationServicesHandler(object):
 
     @defer.inlineCallbacks
     def _is_unknown_user(self, user_id):
-        if not self.is_mine_id(user_id):
+        user = UserID.from_string(user_id)
+        if not self.hs.is_mine(user):
             # we don't know if they are unknown or not since it isn't one of our
             # users. We can't poke ASes.
             defer.returnValue(False)

synapse/handlers/auth.py

@@ -18,9 +18,8 @@ from twisted.internet import defer
 from ._base import BaseHandler
 from synapse.api.constants import LoginType
 from synapse.types import UserID
-from synapse.api.errors import AuthError, LoginError, Codes, StoreError, SynapseError
+from synapse.api.errors import AuthError, LoginError, Codes
 from synapse.util.async import run_on_reactor
-from synapse.config.ldap import LDAPMode
 
 from twisted.web.client import PartialDownloadError
 
@@ -29,12 +28,6 @@ import bcrypt
 import pymacaroons
 import simplejson
 
-try:
-    import ldap3
-except ImportError:
-    ldap3 = None
-    pass
-
 import synapse.util.stringutils as stringutils
 
 
@@ -57,20 +50,17 @@ class AuthHandler(BaseHandler):
         self.INVALID_TOKEN_HTTP_STATUS = 401
 
         self.ldap_enabled = hs.config.ldap_enabled
-        if self.ldap_enabled:
-            if not ldap3:
-                raise RuntimeError(
-                    'Missing ldap3 library. This is required for LDAP Authentication.'
-                )
-            self.ldap_mode = hs.config.ldap_mode
-            self.ldap_uri = hs.config.ldap_uri
-            self.ldap_start_tls = hs.config.ldap_start_tls
-            self.ldap_base = hs.config.ldap_base
-            self.ldap_filter = hs.config.ldap_filter
-            self.ldap_attributes = hs.config.ldap_attributes
-            if self.ldap_mode == LDAPMode.SEARCH:
-                self.ldap_bind_dn = hs.config.ldap_bind_dn
-                self.ldap_bind_password = hs.config.ldap_bind_password
+        self.ldap_server = hs.config.ldap_server
+        self.ldap_port = hs.config.ldap_port
+        self.ldap_tls = hs.config.ldap_tls
+        self.ldap_search_base = hs.config.ldap_search_base
+        self.ldap_search_property = hs.config.ldap_search_property
+        self.ldap_email_property = hs.config.ldap_email_property
+        self.ldap_full_name_property = hs.config.ldap_full_name_property
+
+        if self.ldap_enabled is True:
+            import ldap
+            logger.info("Import ldap version: %s", ldap.__version__)
 
         self.hs = hs  # FIXME better possibility to access registrationHandler later?
 
@@ -462,167 +452,40 @@ class AuthHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def _check_ldap_password(self, user_id, password):
-        """ Attempt to authenticate a user against an LDAP Server
-            and register an account if none exists.
-
-            Returns:
-                True if authentication against LDAP was successful
-        """
-
-        if not ldap3 or not self.ldap_enabled:
+        if not self.ldap_enabled:
+            logger.debug("LDAP not configured")
             defer.returnValue(False)
 
-        if self.ldap_mode not in LDAPMode.LIST:
-            raise RuntimeError(
-                'Invalid ldap mode specified: {mode}'.format(
-                    mode=self.ldap_mode
-                )
-            )
+        import ldap
 
+        logger.info("Authenticating %s with LDAP" % user_id)
         try:
-            server = ldap3.Server(self.ldap_uri)
-            logger.debug(
-                "Attempting ldap connection with %s",
-                self.ldap_uri
-            )
+            ldap_url = "%s:%s" % (self.ldap_server, self.ldap_port)
+            logger.debug("Connecting LDAP server at %s" % ldap_url)
+            l = ldap.initialize(ldap_url)
+            if self.ldap_tls:
+                logger.debug("Initiating TLS")
+                self._connection.start_tls_s()
 
-            localpart = UserID.from_string(user_id).localpart
-            if self.ldap_mode == LDAPMode.SIMPLE:
-                # bind with the the local users ldap credentials
-                bind_dn = "{prop}={value},{base}".format(
-                    prop=self.ldap_attributes['uid'],
-                    value=localpart,
-                    base=self.ldap_base
-                )
-                conn = ldap3.Connection(server, bind_dn, password)
-                logger.debug(
-                    "Established ldap connection in simple mode: %s",
-                    conn
-                )
+            local_name = UserID.from_string(user_id).localpart
 
-                if self.ldap_start_tls:
-                    conn.start_tls()
-                    logger.debug(
-                        "Upgraded ldap connection in simple mode through StartTLS: %s",
-                        conn
-                    )
+            dn = "%s=%s, %s" % (
+                self.ldap_search_property,
+                local_name,
+                self.ldap_search_base)
+            logger.debug("DN for LDAP authentication: %s" % dn)
 
-                conn.bind()
+            l.simple_bind_s(dn.encode('utf-8'), password.encode('utf-8'))
 
-            elif self.ldap_mode == LDAPMode.SEARCH:
-                # connect with preconfigured credentials and search for local user
-                conn = ldap3.Connection(
-                    server,
-                    self.ldap_bind_dn,
-                    self.ldap_bind_password
-                )
-                logger.debug(
-                    "Established ldap connection in search mode: %s",
-                    conn
-                )
-
-                if self.ldap_start_tls:
-                    conn.start_tls()
-                    logger.debug(
-                        "Upgraded ldap connection in search mode through StartTLS: %s",
-                        conn
-                    )
-
-                conn.bind()
-
-                # find matching dn
-                query = "({prop}={value})".format(
-                    prop=self.ldap_attributes['uid'],
-                    value=localpart
-                )
-                if self.ldap_filter:
-                    query = "(&{query}{filter})".format(
-                        query=query,
-                        filter=self.ldap_filter
-                    )
-                logger.debug("ldap search filter: %s", query)
-                result = conn.search(self.ldap_base, query)
-
-                if result and len(conn.response) == 1:
-                    # found exactly one result
-                    user_dn = conn.response[0]['dn']
-                    logger.debug('ldap search found dn: %s', user_dn)
-
-                    # unbind and reconnect, rebind with found dn
-                    conn.unbind()
-                    conn = ldap3.Connection(
-                        server,
-                        user_dn,
-                        password,
-                        auto_bind=True
-                    )
-                else:
-                    # found 0 or > 1 results, abort!
-                    logger.warn(
-                        "ldap search returned unexpected (%d!=1) amount of results",
-                        len(conn.response)
-                    )
-                    defer.returnValue(False)
-
-            logger.info(
-                "User authenticated against ldap server: %s",
-                conn
-            )
-
-            # check for existing account, if none exists, create one
             if not (yield self.does_user_exist(user_id)):
-                # query user metadata for account creation
-                query = "({prop}={value})".format(
-                    prop=self.ldap_attributes['uid'],
-                    value=localpart
+                handler = self.hs.get_handlers().registration_handler
+                user_id, access_token = (
+                    yield handler.register(localpart=local_name)
                 )
 
-                if self.ldap_mode == LDAPMode.SEARCH and self.ldap_filter:
-                    query = "(&{filter}{user_filter})".format(
-                        filter=query,
-                        user_filter=self.ldap_filter
-                    )
-                logger.debug("ldap registration filter: %s", query)
-
-                result = conn.search(
-                    search_base=self.ldap_base,
-                    search_filter=query,
-                    attributes=[
-                        self.ldap_attributes['name'],
-                        self.ldap_attributes['mail']
-                    ]
-                )
-
-                if len(conn.response) == 1:
-                    attrs = conn.response[0]['attributes']
-                    mail = attrs[self.ldap_attributes['mail']][0]
-                    name = attrs[self.ldap_attributes['name']][0]
-
-                    # create account
-                    registration_handler = self.hs.get_handlers().registration_handler
-                    user_id, access_token = (
-                        yield registration_handler.register(localpart=localpart)
-                    )
-
-                    # TODO: bind email, set displayname with data from ldap directory
-
-                    logger.info(
-                        "ldap registration successful: %d: %s (%s, %)",
-                        user_id,
-                        localpart,
-                        name,
-                        mail
-                    )
-                else:
-                    logger.warn(
-                        "ldap registration failed: unexpected (%d!=1) amount of results",
-                        len(result)
-                    )
-                    defer.returnValue(False)
-
             defer.returnValue(True)
-        except ldap3.core.exceptions.LDAPException as e:
-            logger.warn("Error during ldap authentication: %s", e)
+        except ldap.LDAPError, e:
+            logger.warn("LDAP error: %s", e)
             defer.returnValue(False)
 
     @defer.inlineCallbacks
@@ -658,19 +521,14 @@ class AuthHandler(BaseHandler):
         ))
         return m.serialize()
 
-    def generate_short_term_login_token(self, user_id, duration_in_ms=(2 * 60 * 1000)):
+    def generate_short_term_login_token(self, user_id):
         macaroon = self._generate_base_macaroon(user_id)
         macaroon.add_first_party_caveat("type = login")
         now = self.hs.get_clock().time_msec()
-        expiry = now + duration_in_ms
+        expiry = now + (2 * 60 * 1000)
         macaroon.add_first_party_caveat("time < %d" % (expiry,))
         return macaroon.serialize()
 
-    def generate_delete_pusher_token(self, user_id):
-        macaroon = self._generate_base_macaroon(user_id)
-        macaroon.add_first_party_caveat("type = delete_pusher")
-        return macaroon.serialize()
-
     def validate_short_term_login_token_and_get_user_id(self, login_token):
         try:
             macaroon = pymacaroons.Macaroon.deserialize(login_token)
@@ -705,12 +563,7 @@ class AuthHandler(BaseHandler):
 
         except_access_token_ids = [requester.access_token_id] if requester else []
 
-        try:
-            yield self.store.user_set_password_hash(user_id, password_hash)
-        except StoreError as e:
-            if e.code == 404:
-                raise SynapseError(404, "Unknown user", Codes.NOT_FOUND)
-            raise e
+        yield self.store.user_set_password_hash(user_id, password_hash)
         yield self.store.user_delete_access_tokens(
             user_id, except_access_token_ids
         )
@@ -750,8 +603,7 @@ class AuthHandler(BaseHandler):
         Returns:
             Hashed password (str).
         """
-        return bcrypt.hashpw(password + self.hs.config.password_pepper,
-                             bcrypt.gensalt(self.bcrypt_rounds))
+        return bcrypt.hashpw(password, bcrypt.gensalt(self.bcrypt_rounds))
 
     def validate_hash(self, password, stored_hash):
         """Validates that self.hash(password) == stored_hash.
@@ -763,8 +615,4 @@ class AuthHandler(BaseHandler):
         Returns:
             Whether self.hash(password) == stored_hash (bool).
         """
-        if stored_hash:
-            return bcrypt.hashpw(password + self.hs.config.password_pepper,
-                                 stored_hash.encode('utf-8')) == stored_hash
-        else:
-            return False
+        return bcrypt.hashpw(password, stored_hash) == stored_hash

synapse/handlers/directory.py

@@ -33,7 +33,6 @@ class DirectoryHandler(BaseHandler):
         super(DirectoryHandler, self).__init__(hs)
 
         self.state = hs.get_state_handler()
-        self.appservice_handler = hs.get_application_service_handler()
 
         self.federation = hs.get_replication_layer()
         self.federation.register_query_handler(
@@ -282,7 +281,7 @@ class DirectoryHandler(BaseHandler):
         )
         if not result:
             # Query AS to see if it exists
-            as_handler = self.appservice_handler
+            as_handler = self.hs.get_handlers().appservice_handler
             result = yield as_handler.query_room_alias_exists(room_alias)
         defer.returnValue(result)
 

synapse/handlers/events.py

@@ -58,7 +58,7 @@ class EventStreamHandler(BaseHandler):
         If `only_keys` is not None, events from keys will be sent down.
         """
         auth_user = UserID.from_string(auth_user_id)
-        presence_handler = self.hs.get_presence_handler()
+        presence_handler = self.hs.get_handlers().presence_handler
 
         context = yield presence_handler.user_syncing(
             auth_user_id, affect_presence=affect_presence,

synapse/handlers/federation.py

@@ -33,7 +33,7 @@ from synapse.util.frozenutils import unfreeze
 from synapse.crypto.event_signing import (
     compute_event_signature, add_hashes_and_signatures,
 )
-from synapse.types import UserID, get_domain_from_id
+from synapse.types import UserID
 
 from synapse.events.utils import prune_event
 
@@ -66,6 +66,10 @@ class FederationHandler(BaseHandler):
 
         self.hs = hs
 
+        self.distributor.observe("user_joined_room", self.user_joined_room)
+
+        self.waiting_for_join_list = {}
+
         self.store = hs.get_datastore()
         self.replication_layer = hs.get_replication_layer()
         self.state_handler = hs.get_state_handler()
@@ -345,21 +349,19 @@ class FederationHandler(BaseHandler):
         )
 
         missing_auth = required_auth - set(auth_events)
-        if missing_auth:
-            logger.info("Missing auth for backfill: %r", missing_auth)
-            results = yield defer.gatherResults(
-                [
-                    self.replication_layer.get_pdu(
-                        [dest],
-                        event_id,
-                        outlier=True,
-                        timeout=10000,
-                    )
-                    for event_id in missing_auth
-                ],
-                consumeErrors=True
-            ).addErrback(unwrapFirstError)
-            auth_events.update({a.event_id: a for a in results})
+        results = yield defer.gatherResults(
+            [
+                self.replication_layer.get_pdu(
+                    [dest],
+                    event_id,
+                    outlier=True,
+                    timeout=10000,
+                )
+                for event_id in missing_auth
+            ],
+            consumeErrors=True
+        ).addErrback(unwrapFirstError)
+        auth_events.update({a.event_id: a for a in results})
 
         ev_infos = []
         for a in auth_events.values():
@@ -401,7 +403,7 @@ class FederationHandler(BaseHandler):
             # previous to work out the state.
             # TODO: We can probably do something more clever here.
             yield self._handle_new_event(
-                dest, event, backfilled=True,
+                dest, event
             )
 
         defer.returnValue(events)
@@ -451,7 +453,7 @@ class FederationHandler(BaseHandler):
             joined_domains = {}
             for u, d in joined_users:
                 try:
-                    dom = get_domain_from_id(u)
+                    dom = UserID.from_string(u).domain
                     old_d = joined_domains.get(dom)
                     if old_d:
                         joined_domains[dom] = min(d, old_d)
@@ -680,8 +682,7 @@ class FederationHandler(BaseHandler):
         })
 
         try:
-            message_handler = self.hs.get_handlers().message_handler
-            event, context = yield message_handler._create_new_client_event(
+            event, context = yield self._create_new_client_event(
                 builder=builder,
             )
         except AuthError as e:
@@ -742,7 +743,9 @@ class FederationHandler(BaseHandler):
             try:
                 if k[0] == EventTypes.Member:
                     if s.content["membership"] == Membership.JOIN:
-                        destinations.add(get_domain_from_id(s.state_key))
+                        destinations.add(
+                            UserID.from_string(s.state_key).domain
+                        )
             except:
                 logger.warn(
                     "Failed to get destination from event %s", s.event_id
@@ -912,8 +915,7 @@ class FederationHandler(BaseHandler):
             "state_key": user_id,
         })
 
-        message_handler = self.hs.get_handlers().message_handler
-        event, context = yield message_handler._create_new_client_event(
+        event, context = yield self._create_new_client_event(
             builder=builder,
         )
 
@@ -968,7 +970,9 @@ class FederationHandler(BaseHandler):
             try:
                 if k[0] == EventTypes.Member:
                     if s.content["membership"] == Membership.LEAVE:
-                        destinations.add(get_domain_from_id(s.state_key))
+                        destinations.add(
+                            UserID.from_string(s.state_key).domain
+                        )
             except:
                 logger.warn(
                     "Failed to get destination from event %s", s.event_id
@@ -1018,16 +1022,13 @@ class FederationHandler(BaseHandler):
 
             res = results.values()
             for event in res:
-                # We sign these again because there was a bug where we
-                # incorrectly signed things the first time round
-                if self.hs.is_mine_id(event.event_id):
-                    event.signatures.update(
-                        compute_event_signature(
-                            event,
-                            self.hs.hostname,
-                            self.hs.config.signing_key[0]
-                        )
+                event.signatures.update(
+                    compute_event_signature(
+                        event,
+                        self.hs.hostname,
+                        self.hs.config.signing_key[0]
                     )
+                )
 
             defer.returnValue(res)
         else:
@@ -1092,6 +1093,15 @@ class FederationHandler(BaseHandler):
     def get_min_depth_for_context(self, context):
         return self.store.get_min_depth(context)
 
+    @log_function
+    def user_joined_room(self, user, room_id):
+        waiters = self.waiting_for_join_list.get(
+            (user.to_string(), room_id),
+            []
+        )
+        while waiters:
+            waiters.pop().callback(None)
+
     @defer.inlineCallbacks
     @log_function
     def _handle_new_event(self, origin, event, state=None, auth_events=None,
@@ -1105,7 +1115,7 @@ class FederationHandler(BaseHandler):
         if not event.internal_metadata.is_outlier():
             action_generator = ActionGenerator(self.hs)
             yield action_generator.handle_push_actions_for_event(
-                event, context
+                event, context, self
             )
 
         event_stream_id, max_stream_id = yield self.store.persist_event(
@@ -1413,7 +1423,7 @@ class FederationHandler(BaseHandler):
                 local_view = dict(auth_events)
                 remote_view = dict(auth_events)
                 remote_view.update({
-                    (d.type, d.state_key): d for d in different_events if d
+                    (d.type, d.state_key): d for d in different_events
                 })
 
                 new_state, prev_state = self.state_handler.resolve_events(
@@ -1682,10 +1692,7 @@ class FederationHandler(BaseHandler):
         if (yield self.auth.check_host_in_room(room_id, self.hs.hostname)):
             builder = self.event_builder_factory.new(event_dict)
             EventValidator().validate_new(builder)
-            message_handler = self.hs.get_handlers().message_handler
-            event, context = yield message_handler._create_new_client_event(
-                builder=builder
-            )
+            event, context = yield self._create_new_client_event(builder=builder)
 
             event, context = yield self.add_display_name_to_third_party_invite(
                 event_dict, event, context
@@ -1713,8 +1720,7 @@ class FederationHandler(BaseHandler):
     def on_exchange_third_party_invite_request(self, origin, room_id, event_dict):
         builder = self.event_builder_factory.new(event_dict)
 
-        message_handler = self.hs.get_handlers().message_handler
-        event, context = yield message_handler._create_new_client_event(
+        event, context = yield self._create_new_client_event(
             builder=builder,
         )
 
@@ -1753,8 +1759,7 @@ class FederationHandler(BaseHandler):
         event_dict["content"]["third_party_invite"]["display_name"] = display_name
         builder = self.event_builder_factory.new(event_dict)
         EventValidator().validate_new(builder)
-        message_handler = self.hs.get_handlers().message_handler
-        event, context = yield message_handler._create_new_client_event(builder=builder)
+        event, context = yield self._create_new_client_event(builder=builder)
         defer.returnValue((event, context))
 
     @defer.inlineCallbacks

synapse/handlers/identity.py

@@ -21,7 +21,7 @@ from synapse.api.errors import (
 )
 from ._base import BaseHandler
 from synapse.util.async import run_on_reactor
-from synapse.api.errors import SynapseError, Codes
+from synapse.api.errors import SynapseError
 
 import json
 import logging
@@ -41,20 +41,6 @@ class IdentityHandler(BaseHandler):
             hs.config.use_insecure_ssl_client_just_for_testing_do_not_use
         )
 
-    def _should_trust_id_server(self, id_server):
-        if id_server not in self.trusted_id_servers:
-            if self.trust_any_id_server_just_for_testing_do_not_use:
-                logger.warn(
-                    "Trusting untrustworthy ID server %r even though it isn't"
-                    " in the trusted id list for testing because"
-                    " 'use_insecure_ssl_client_just_for_testing_do_not_use'"
-                    " is set in the config",
-                    id_server,
-                )
-            else:
-                return False
-        return True
-
     @defer.inlineCallbacks
     def threepid_from_creds(self, creds):
         yield run_on_reactor()
@@ -73,12 +59,19 @@ class IdentityHandler(BaseHandler):
         else:
             raise SynapseError(400, "No client_secret in creds")
 
-        if not self._should_trust_id_server(id_server):
-            logger.warn(
-                '%s is not a trusted ID server: rejecting 3pid ' +
-                'credentials', id_server
-            )
-            defer.returnValue(None)
+        if id_server not in self.trusted_id_servers:
+            if self.trust_any_id_server_just_for_testing_do_not_use:
+                logger.warn(
+                    "Trusting untrustworthy ID server %r even though it isn't"
+                    " in the trusted id list for testing because"
+                    " 'use_insecure_ssl_client_just_for_testing_do_not_use'"
+                    " is set in the config",
+                    id_server,
+                )
+            else:
+                logger.warn('%s is not a trusted ID server: rejecting 3pid ' +
+                            'credentials', id_server)
+                defer.returnValue(None)
 
         data = {}
         try:
@@ -136,12 +129,6 @@ class IdentityHandler(BaseHandler):
     def requestEmailToken(self, id_server, email, client_secret, send_attempt, **kwargs):
         yield run_on_reactor()
 
-        if not self._should_trust_id_server(id_server):
-            raise SynapseError(
-                400, "Untrusted ID server '%s'" % id_server,
-                Codes.SERVER_NOT_TRUSTED
-            )
-
         params = {
             'email': email,
             'client_secret': client_secret,

synapse/handlers/message.py

@@ -17,19 +17,13 @@ from twisted.internet import defer
 
 from synapse.api.constants import EventTypes, Membership
 from synapse.api.errors import AuthError, Codes, SynapseError
-from synapse.crypto.event_signing import add_hashes_and_signatures
+from synapse.streams.config import PaginationConfig
 from synapse.events.utils import serialize_event
 from synapse.events.validator import EventValidator
-from synapse.push.action_generator import ActionGenerator
-from synapse.streams.config import PaginationConfig
-from synapse.types import (
-    UserID, RoomAlias, RoomStreamToken, StreamToken, get_domain_from_id
-)
 from synapse.util import unwrapFirstError
-from synapse.util.async import concurrently_execute, run_on_reactor, ReadWriteLock
+from synapse.util.async import concurrently_execute
 from synapse.util.caches.snapshot_cache import SnapshotCache
-from synapse.util.logcontext import preserve_fn
-from synapse.visibility import filter_events_for_client
+from synapse.types import UserID, RoomStreamToken, StreamToken
 
 from ._base import BaseHandler
 
@@ -50,20 +44,6 @@ class MessageHandler(BaseHandler):
         self.validator = EventValidator()
         self.snapshot_cache = SnapshotCache()
 
-        self.pagination_lock = ReadWriteLock()
-
-    @defer.inlineCallbacks
-    def purge_history(self, room_id, event_id):
-        event = yield self.store.get_event(event_id)
-
-        if event.room_id != room_id:
-            raise SynapseError(400, "Event is for wrong room.")
-
-        depth = event.depth
-
-        with (yield self.pagination_lock.write(room_id)):
-            yield self.store.delete_old_state(room_id, depth)
-
     @defer.inlineCallbacks
     def get_messages(self, requester, room_id=None, pagin_config=None,
                      as_client_event=True):
@@ -99,113 +79,17 @@ class MessageHandler(BaseHandler):
 
         source_config = pagin_config.get_source_config("room")
 
-        with (yield self.pagination_lock.read(room_id)):
-            membership, member_event_id = yield self._check_in_room_or_world_readable(
-                room_id, user_id
-            )
-
-            if source_config.direction == 'b':
-                # if we're going backwards, we might need to backfill. This
-                # requires that we have a topo token.
-                if room_token.topological:
-                    max_topo = room_token.topological
-                else:
-                    max_topo = yield self.store.get_max_topological_token(
-                        room_id, room_token.stream
-                    )
-
-                if membership == Membership.LEAVE:
-                    # If they have left the room then clamp the token to be before
-                    # they left the room, to save the effort of loading from the
-                    # database.
-                    leave_token = yield self.store.get_topological_token_for_event(
-                        member_event_id
-                    )
-                    leave_token = RoomStreamToken.parse(leave_token)
-                    if leave_token.topological < max_topo:
-                        source_config.from_key = str(leave_token)
-
-                yield self.hs.get_handlers().federation_handler.maybe_backfill(
-                    room_id, max_topo
-                )
-
-            events, next_key = yield data_source.get_pagination_rows(
-                requester.user, source_config, room_id
-            )
-
-            next_token = pagin_config.from_token.copy_and_replace(
-                "room_key", next_key
-            )
-
-        if not events:
-            defer.returnValue({
-                "chunk": [],
-                "start": pagin_config.from_token.to_string(),
-                "end": next_token.to_string(),
-            })
-
-        events = yield filter_events_for_client(
-            self.store,
-            user_id,
-            events,
-            is_peeking=(member_event_id is None),
-        )
-
-        time_now = self.clock.time_msec()
-
-        chunk = {
-            "chunk": [
-                serialize_event(e, time_now, as_client_event)
-                for e in events
-            ],
-            "start": pagin_config.from_token.to_string(),
-            "end": next_token.to_string(),
-        }
-
-        defer.returnValue(chunk)
-
-    @defer.inlineCallbacks
-    def get_files(self, requester, room_id, pagin_config):
-        """Get files in a room.
-
-        Args:
-            requester (Requester): The user requesting files.
-            room_id (str): The room they want files from.
-            pagin_config (synapse.api.streams.PaginationConfig): The pagination
-                config rules to apply, if any.
-            as_client_event (bool): True to get events in client-server format.
-        Returns:
-            dict: Pagination API results
-        """
-        user_id = requester.user.to_string()
-
-        if pagin_config.from_token:
-            room_token = pagin_config.from_token.room_key
-        else:
-            pagin_config.from_token = (
-                yield self.hs.get_event_sources().get_current_token(
-                    direction='b'
-                )
-            )
-            room_token = pagin_config.from_token.room_key
-
-        room_token = RoomStreamToken.parse(room_token)
-
-        pagin_config.from_token = pagin_config.from_token.copy_and_replace(
-            "room_key", str(room_token)
-        )
-
-        source_config = pagin_config.get_source_config("room")
-
         membership, member_event_id = yield self._check_in_room_or_world_readable(
             room_id, user_id
         )
 
         if source_config.direction == 'b':
+            # if we're going backwards, we might need to backfill. This
+            # requires that we have a topo token.
             if room_token.topological:
                 max_topo = room_token.topological
             else:
-                max_topo = yield self.store.get_max_topological_token(
+                max_topo = yield self.store.get_max_topological_token_for_stream_and_room(
                     room_id, room_token.stream
                 )
 
@@ -220,12 +104,12 @@ class MessageHandler(BaseHandler):
                 if leave_token.topological < max_topo:
                     source_config.from_key = str(leave_token)
 
-        events, next_key = yield self.store.paginate_room_file_events(
-            room_id,
-            from_key=source_config.from_key,
-            to_key=source_config.to_key,
-            direction=source_config.direction,
-            limit=source_config.limit,
+            yield self.hs.get_handlers().federation_handler.maybe_backfill(
+                room_id, max_topo
+            )
+
+        events, next_key = yield data_source.get_pagination_rows(
+            requester.user, source_config, room_id
         )
 
         next_token = pagin_config.from_token.copy_and_replace(
@@ -239,8 +123,7 @@ class MessageHandler(BaseHandler):
                 "end": next_token.to_string(),
             })
 
-        events = yield filter_events_for_client(
-            self.store,
+        events = yield self._filter_events_for_client(
             user_id,
             events,
             is_peeking=(member_event_id is None),
@@ -250,7 +133,7 @@ class MessageHandler(BaseHandler):
 
         chunk = {
             "chunk": [
-                serialize_event(e, time_now)
+                serialize_event(e, time_now, as_client_event)
                 for e in events
             ],
             "start": pagin_config.from_token.to_string(),
@@ -346,7 +229,7 @@ class MessageHandler(BaseHandler):
         )
 
         if event.type == EventTypes.Message:
-            presence = self.hs.get_presence_handler()
+            presence = self.hs.get_handlers().presence_handler
             yield presence.bump_presence_active_time(user)
 
     def deduplicate_state_event(self, event, context):
@@ -600,8 +483,8 @@ class MessageHandler(BaseHandler):
                     ]
                 ).addErrback(unwrapFirstError)
 
-                messages = yield filter_events_for_client(
-                    self.store, user_id, messages
+                messages = yield self._filter_events_for_client(
+                    user_id, messages
                 )
 
                 start_token = now_token.copy_and_replace("room_key", token[0])
@@ -736,8 +619,8 @@ class MessageHandler(BaseHandler):
             end_token=stream_token
         )
 
-        messages = yield filter_events_for_client(
-            self.store, user_id, messages, is_peeking=is_peeking
+        messages = yield self._filter_events_for_client(
+            user_id, messages, is_peeking=is_peeking
         )
 
         start_token = StreamToken.START.copy_and_replace("room_key", token[0])
@@ -784,7 +667,7 @@ class MessageHandler(BaseHandler):
             and m.content["membership"] == Membership.JOIN
         ]
 
-        presence_handler = self.hs.get_presence_handler()
+        presence_handler = self.hs.get_handlers().presence_handler
 
         @defer.inlineCallbacks
         def get_presence():
@@ -817,8 +700,8 @@ class MessageHandler(BaseHandler):
             consumeErrors=True,
         ).addErrback(unwrapFirstError)
 
-        messages = yield filter_events_for_client(
-            self.store, user_id, messages, is_peeking=is_peeking,
+        messages = yield self._filter_events_for_client(
+            user_id, messages, is_peeking=is_peeking,
         )
 
         start_token = now_token.copy_and_replace("room_key", token[0])
@@ -841,196 +724,3 @@ class MessageHandler(BaseHandler):
             ret["membership"] = membership
 
         defer.returnValue(ret)
-
-    @defer.inlineCallbacks
-    def _create_new_client_event(self, builder, prev_event_ids=None):
-        if prev_event_ids:
-            prev_events = yield self.store.add_event_hashes(prev_event_ids)
-            prev_max_depth = yield self.store.get_max_depth_of_events(prev_event_ids)
-            depth = prev_max_depth + 1
-        else:
-            latest_ret = yield self.store.get_latest_event_ids_and_hashes_in_room(
-                builder.room_id,
-            )
-
-            if latest_ret:
-                depth = max([d for _, _, d in latest_ret]) + 1
-            else:
-                depth = 1
-
-            prev_events = [
-                (event_id, prev_hashes)
-                for event_id, prev_hashes, _ in latest_ret
-            ]
-
-        builder.prev_events = prev_events
-        builder.depth = depth
-
-        state_handler = self.state_handler
-
-        context = yield state_handler.compute_event_context(builder)
-
-        if builder.is_state():
-            builder.prev_state = yield self.store.add_event_hashes(
-                context.prev_state_events
-            )
-
-        yield self.auth.add_auth_events(builder, context)
-
-        signing_key = self.hs.config.signing_key[0]
-        add_hashes_and_signatures(
-            builder, self.server_name, signing_key
-        )
-
-        event = builder.build()
-
-        logger.debug(
-            "Created event %s with current state: %s",
-            event.event_id, context.current_state,
-        )
-
-        defer.returnValue(
-            (event, context,)
-        )
-
-    @defer.inlineCallbacks
-    def handle_new_client_event(
-        self,
-        requester,
-        event,
-        context,
-        ratelimit=True,
-        extra_users=[]
-    ):
-        # We now need to go and hit out to wherever we need to hit out to.
-
-        if ratelimit:
-            self.ratelimit(requester)
-
-        try:
-            self.auth.check(event, auth_events=context.current_state)
-        except AuthError as err:
-            logger.warn("Denying new event %r because %s", event, err)
-            raise err
-
-        yield self.maybe_kick_guest_users(event, context.current_state.values())
-
-        if event.type == EventTypes.CanonicalAlias:
-            # Check the alias is acually valid (at this time at least)
-            room_alias_str = event.content.get("alias", None)
-            if room_alias_str:
-                room_alias = RoomAlias.from_string(room_alias_str)
-                directory_handler = self.hs.get_handlers().directory_handler
-                mapping = yield directory_handler.get_association(room_alias)
-
-                if mapping["room_id"] != event.room_id:
-                    raise SynapseError(
-                        400,
-                        "Room alias %s does not point to the room" % (
-                            room_alias_str,
-                        )
-                    )
-
-        federation_handler = self.hs.get_handlers().federation_handler
-
-        if event.type == EventTypes.Member:
-            if event.content["membership"] == Membership.INVITE:
-                def is_inviter_member_event(e):
-                    return (
-                        e.type == EventTypes.Member and
-                        e.sender == event.sender
-                    )
-
-                event.unsigned["invite_room_state"] = [
-                    {
-                        "type": e.type,
-                        "state_key": e.state_key,
-                        "content": e.content,
-                        "sender": e.sender,
-                    }
-                    for k, e in context.current_state.items()
-                    if e.type in self.hs.config.room_invite_state_types
-                    or is_inviter_member_event(e)
-                ]
-
-                invitee = UserID.from_string(event.state_key)
-                if not self.hs.is_mine(invitee):
-                    # TODO: Can we add signature from remote server in a nicer
-                    # way? If we have been invited by a remote server, we need
-                    # to get them to sign the event.
-
-                    returned_invite = yield federation_handler.send_invite(
-                        invitee.domain,
-                        event,
-                    )
-
-                    event.unsigned.pop("room_state", None)
-
-                    # TODO: Make sure the signatures actually are correct.
-                    event.signatures.update(
-                        returned_invite.signatures
-                    )
-
-        if event.type == EventTypes.Redaction:
-            if self.auth.check_redaction(event, auth_events=context.current_state):
-                original_event = yield self.store.get_event(
-                    event.redacts,
-                    check_redacted=False,
-                    get_prev_content=False,
-                    allow_rejected=False,
-                    allow_none=False
-                )
-                if event.user_id != original_event.user_id:
-                    raise AuthError(
-                        403,
-                        "You don't have permission to redact events"
-                    )
-
-        if event.type == EventTypes.Create and context.current_state:
-            raise AuthError(
-                403,
-                "Changing the room create event is forbidden",
-            )
-
-        action_generator = ActionGenerator(self.hs)
-        yield action_generator.handle_push_actions_for_event(
-            event, context
-        )
-
-        (event_stream_id, max_stream_id) = yield self.store.persist_event(
-            event, context=context
-        )
-
-        # this intentionally does not yield: we don't care about the result
-        # and don't need to wait for it.
-        preserve_fn(self.hs.get_pusherpool().on_new_notifications)(
-            event_stream_id, max_stream_id
-        )
-
-        destinations = set()
-        for k, s in context.current_state.items():
-            try:
-                if k[0] == EventTypes.Member:
-                    if s.content["membership"] == Membership.JOIN:
-                        destinations.add(get_domain_from_id(s.state_key))
-            except SynapseError:
-                logger.warn(
-                    "Failed to get destination from event %s", s.event_id
-                )
-
-        @defer.inlineCallbacks
-        def _notify():
-            yield run_on_reactor()
-            self.notifier.on_new_room_event(
-                event, event_stream_id, max_stream_id,
-                extra_users=extra_users
-            )
-
-        preserve_fn(_notify)()
-
-        # If invite, remove room_state from unsigned before sending.
-        event.unsigned.pop("invite_room_state", None)
-
-        federation_handler.handle_new_event(
-            event, destinations=destinations,
-        )

synapse/handlers/presence.py

@@ -33,9 +33,11 @@ from synapse.util.logcontext import preserve_fn
 from synapse.util.logutils import log_function
 from synapse.util.metrics import Measure
 from synapse.util.wheel_timer import WheelTimer
-from synapse.types import UserID, get_domain_from_id
+from synapse.types import UserID
 import synapse.metrics
 
+from ._base import BaseHandler
+
 import logging
 
 
@@ -50,8 +52,6 @@ timers_fired_counter = metrics.register_counter("timers_fired")
 federation_presence_counter = metrics.register_counter("federation_presence")
 bump_active_time_counter = metrics.register_counter("bump_active_time")
 
-get_updates_counter = metrics.register_counter("get_updates", labels=["type"])
-
 
 # If a user was last active in the last LAST_ACTIVE_GRANULARITY, consider them
 # "currently_active"
@@ -70,18 +70,14 @@ FEDERATION_TIMEOUT = 30 * 60 * 1000
 # How often to resend presence to remote servers
 FEDERATION_PING_INTERVAL = 25 * 60 * 1000
 
-# How long we will wait before assuming that the syncs from an external process
-# are dead.
-EXTERNAL_PROCESS_EXPIRY = 5 * 60 * 1000
-
 assert LAST_ACTIVE_GRANULARITY < IDLE_TIMER
 
 
-class PresenceHandler(object):
+class PresenceHandler(BaseHandler):
 
     def __init__(self, hs):
-        self.is_mine = hs.is_mine
-        self.is_mine_id = hs.is_mine_id
+        super(PresenceHandler, self).__init__(hs)
+        self.hs = hs
         self.clock = hs.get_clock()
         self.store = hs.get_datastore()
         self.wheel_timer = WheelTimer()
@@ -142,7 +138,7 @@ class PresenceHandler(object):
                 obj=state.user_id,
                 then=state.last_user_sync_ts + SYNC_ONLINE_TIMEOUT,
             )
-            if self.is_mine_id(state.user_id):
+            if self.hs.is_mine_id(state.user_id):
                 self.wheel_timer.insert(
                     now=now,
                     obj=state.user_id,
@@ -164,26 +160,15 @@ class PresenceHandler(object):
         self.serial_to_user = {}
         self._next_serial = 1
 
-        # Keeps track of the number of *ongoing* syncs on this process. While
-        # this is non zero a user will never go offline.
+        # Keeps track of the number of *ongoing* syncs. While this is non zero
+        # a user will never go offline.
         self.user_to_num_current_syncs = {}
 
-        # Keeps track of the number of *ongoing* syncs on other processes.
-        # While any sync is ongoing on another process the user will never
-        # go offline.
-        # Each process has a unique identifier and an update frequency. If
-        # no update is received from that process within the update period then
-        # we assume that all the sync requests on that process have stopped.
-        # Stored as a dict from process_id to set of user_id, and a dict of
-        # process_id to millisecond timestamp last updated.
-        self.external_process_to_current_syncs = {}
-        self.external_process_last_updated_ms = {}
-
         # Start a LoopingCall in 30s that fires every 5s.
         # The initial delay is to allow disconnected clients a chance to
         # reconnect before we treat them as offline.
         self.clock.call_later(
-            30,
+            0 * 1000,
             self.clock.looping_call,
             self._handle_timeouts,
             5000,
@@ -243,7 +228,7 @@ class PresenceHandler(object):
 
                 new_state, should_notify, should_ping = handle_update(
                     prev_state, new_state,
-                    is_mine=self.is_mine_id(user_id),
+                    is_mine=self.hs.is_mine_id(user_id),
                     wheel_timer=self.wheel_timer,
                     now=now
                 )
@@ -283,48 +268,31 @@ class PresenceHandler(object):
         """Checks the presence of users that have timed out and updates as
         appropriate.
         """
-        logger.info("Handling presence timeouts")
         now = self.clock.time_msec()
 
-        try:
-            with Measure(self.clock, "presence_handle_timeouts"):
-                # Fetch the list of users that *may* have timed out. Things may have
-                # changed since the timeout was set, so we won't necessarily have to
-                # take any action.
-                users_to_check = set(self.wheel_timer.fetch(now))
+        with Measure(self.clock, "presence_handle_timeouts"):
+            # Fetch the list of users that *may* have timed out. Things may have
+            # changed since the timeout was set, so we won't necessarily have to
+            # take any action.
+            users_to_check = self.wheel_timer.fetch(now)
 
-                # Check whether the lists of syncing processes from an external
-                # process have expired.
-                expired_process_ids = [
-                    process_id for process_id, last_update
-                    in self.external_process_last_updated_ms.items()
-                    if now - last_update > EXTERNAL_PROCESS_EXPIRY
-                ]
-                for process_id in expired_process_ids:
-                    users_to_check.update(
-                        self.external_process_last_updated_ms.pop(process_id, ())
-                    )
-                    self.external_process_last_update.pop(process_id)
-
-                states = [
-                    self.user_to_current_state.get(
-                        user_id, UserPresenceState.default(user_id)
-                    )
-                    for user_id in users_to_check
-                ]
-
-                timers_fired_counter.inc_by(len(states))
-
-                changes = handle_timeouts(
-                    states,
-                    is_mine_fn=self.is_mine_id,
-                    syncing_user_ids=self.get_currently_syncing_users(),
-                    now=now,
+            states = [
+                self.user_to_current_state.get(
+                    user_id, UserPresenceState.default(user_id)
                 )
+                for user_id in set(users_to_check)
+            ]
 
-            preserve_fn(self._update_states)(changes)
-        except:
-            logger.exception("Exception in _handle_timeouts loop")
+            timers_fired_counter.inc_by(len(states))
+
+            changes = handle_timeouts(
+                states,
+                is_mine_fn=self.hs.is_mine_id,
+                user_to_num_current_syncs=self.user_to_num_current_syncs,
+                now=now,
+            )
+
+        preserve_fn(self._update_states)(changes)
 
     @defer.inlineCallbacks
     def bump_presence_active_time(self, user):
@@ -397,74 +365,6 @@ class PresenceHandler(object):
 
         defer.returnValue(_user_syncing())
 
-    def get_currently_syncing_users(self):
-        """Get the set of user ids that are currently syncing on this HS.
-        Returns:
-            set(str): A set of user_id strings.
-        """
-        syncing_user_ids = {
-            user_id for user_id, count in self.user_to_num_current_syncs.items()
-            if count
-        }
-        for user_ids in self.external_process_to_current_syncs.values():
-            syncing_user_ids.update(user_ids)
-        return syncing_user_ids
-
-    @defer.inlineCallbacks
-    def update_external_syncs(self, process_id, syncing_user_ids):
-        """Update the syncing users for an external process
-
-        Args:
-            process_id(str): An identifier for the process the users are
-                syncing against. This allows synapse to process updates
-                as user start and stop syncing against a given process.
-            syncing_user_ids(set(str)): The set of user_ids that are
-                currently syncing on that server.
-        """
-
-        # Grab the previous list of user_ids that were syncing on that process
-        prev_syncing_user_ids = (
-            self.external_process_to_current_syncs.get(process_id, set())
-        )
-        # Grab the current presence state for both the users that are syncing
-        # now and the users that were syncing before this update.
-        prev_states = yield self.current_state_for_users(
-            syncing_user_ids | prev_syncing_user_ids
-        )
-        updates = []
-        time_now_ms = self.clock.time_msec()
-
-        # For each new user that is syncing check if we need to mark them as
-        # being online.
-        for new_user_id in syncing_user_ids - prev_syncing_user_ids:
-            prev_state = prev_states[new_user_id]
-            if prev_state.state == PresenceState.OFFLINE:
-                updates.append(prev_state.copy_and_replace(
-                    state=PresenceState.ONLINE,
-                    last_active_ts=time_now_ms,
-                    last_user_sync_ts=time_now_ms,
-                ))
-            else:
-                updates.append(prev_state.copy_and_replace(
-                    last_user_sync_ts=time_now_ms,
-                ))
-
-        # For each user that is still syncing or stopped syncing update the
-        # last sync time so that we will correctly apply the grace period when
-        # they stop syncing.
-        for old_user_id in prev_syncing_user_ids:
-            prev_state = prev_states[old_user_id]
-            updates.append(prev_state.copy_and_replace(
-                last_user_sync_ts=time_now_ms,
-            ))
-
-        yield self._update_states(updates)
-
-        # Update the last updated time for the process. We expire the entries
-        # if we don't receive an update in the given timeframe.
-        self.external_process_last_updated_ms[process_id] = self.clock.time_msec()
-        self.external_process_to_current_syncs[process_id] = syncing_user_ids
-
     @defer.inlineCallbacks
     def current_state_for_user(self, user_id):
         """Get the current presence state for a user.
@@ -527,7 +427,7 @@ class PresenceHandler(object):
 
         hosts_to_states = {}
         for room_id, states in room_ids_to_states.items():
-            local_states = filter(lambda s: self.is_mine_id(s.user_id), states)
+            local_states = filter(lambda s: self.hs.is_mine_id(s.user_id), states)
             if not local_states:
                 continue
 
@@ -536,11 +436,11 @@ class PresenceHandler(object):
                 hosts_to_states.setdefault(host, []).extend(local_states)
 
         for user_id, states in users_to_states.items():
-            local_states = filter(lambda s: self.is_mine_id(s.user_id), states)
+            local_states = filter(lambda s: self.hs.is_mine_id(s.user_id), states)
             if not local_states:
                 continue
 
-            host = get_domain_from_id(user_id)
+            host = UserID.from_string(user_id).domain
             hosts_to_states.setdefault(host, []).extend(local_states)
 
         # TODO: de-dup hosts_to_states, as a single host might have multiple
@@ -711,14 +611,14 @@ class PresenceHandler(object):
         # don't need to send to local clients here, as that is done as part
         # of the event stream/sync.
         # TODO: Only send to servers not already in the room.
-        if self.is_mine(user):
+        if self.hs.is_mine(user):
             state = yield self.current_state_for_user(user.to_string())
 
             hosts = yield self.store.get_joined_hosts_for_room(room_id)
             self._push_to_remotes({host: (state,) for host in hosts})
         else:
             user_ids = yield self.store.get_users_in_room(room_id)
-            user_ids = filter(self.is_mine_id, user_ids)
+            user_ids = filter(self.hs.is_mine_id, user_ids)
 
             states = yield self.current_state_for_users(user_ids)
 
@@ -728,7 +628,7 @@ class PresenceHandler(object):
     def get_presence_list(self, observer_user, accepted=None):
         """Returns the presence for all users in their presence list.
         """
-        if not self.is_mine(observer_user):
+        if not self.hs.is_mine(observer_user):
             raise SynapseError(400, "User is not hosted on this Home Server")
 
         presence_list = yield self.store.get_presence_list(
@@ -759,7 +659,7 @@ class PresenceHandler(object):
             observer_user.localpart, observed_user.to_string()
         )
 
-        if self.is_mine(observed_user):
+        if self.hs.is_mine(observed_user):
             yield self.invite_presence(observed_user, observer_user)
         else:
             yield self.federation.send_edu(
@@ -775,11 +675,11 @@ class PresenceHandler(object):
     def invite_presence(self, observed_user, observer_user):
         """Handles new presence invites.
         """
-        if not self.is_mine(observed_user):
+        if not self.hs.is_mine(observed_user):
             raise SynapseError(400, "User is not hosted on this Home Server")
 
         # TODO: Don't auto accept
-        if self.is_mine(observer_user):
+        if self.hs.is_mine(observer_user):
             yield self.accept_presence(observed_user, observer_user)
         else:
             self.federation.send_edu(
@@ -842,7 +742,7 @@ class PresenceHandler(object):
         Returns:
             A Deferred.
         """
-        if not self.is_mine(observer_user):
+        if not self.hs.is_mine(observer_user):
             raise SynapseError(400, "User is not hosted on this Home Server")
 
         yield self.store.del_presence_list(
@@ -934,11 +834,7 @@ def _format_user_presence_state(state, now):
 
 class PresenceEventSource(object):
     def __init__(self, hs):
-        # We can't call get_presence_handler here because there's a cycle:
-        #
-        #   Presence -> Notifier -> PresenceEventSource -> Presence
-        #
-        self.get_presence_handler = hs.get_presence_handler
+        self.hs = hs
         self.clock = hs.get_clock()
         self.store = hs.get_datastore()
 
@@ -964,7 +860,7 @@ class PresenceEventSource(object):
                 from_key = int(from_key)
             room_ids = room_ids or []
 
-            presence = self.get_presence_handler()
+            presence = self.hs.get_handlers().presence_handler
             stream_change_cache = self.store.presence_stream_cache
 
             if not room_ids:
@@ -981,13 +877,13 @@ class PresenceEventSource(object):
 
             user_ids_changed = set()
             changed = None
-            if from_key:
-                changed = stream_change_cache.get_all_entities_changed(from_key)
-
-            if changed is not None and len(changed) < 500:
+            if from_key and max_token - from_key < 100:
                 # For small deltas, its quicker to get all changes and then
                 # work out if we share a room or they're in our presence list
-                get_updates_counter.inc("stream")
+                changed = stream_change_cache.get_all_entities_changed(from_key)
+
+            # get_all_entities_changed can return None
+            if changed is not None:
                 for other_user_id in changed:
                     if other_user_id in friends:
                         user_ids_changed.add(other_user_id)
@@ -999,8 +895,6 @@ class PresenceEventSource(object):
             else:
                 # Too many possible updates. Find all users we can see and check
                 # if any of them have changed.
-                get_updates_counter.inc("full")
-
                 user_ids_to_check = set()
                 for room_id in room_ids:
                     users = yield self.store.get_users_in_room(room_id)
@@ -1039,14 +933,15 @@ class PresenceEventSource(object):
         return self.get_new_events(user, from_key=None, include_offline=False)
 
 
-def handle_timeouts(user_states, is_mine_fn, syncing_user_ids, now):
+def handle_timeouts(user_states, is_mine_fn, user_to_num_current_syncs, now):
     """Checks the presence of users that have timed out and updates as
     appropriate.
 
     Args:
         user_states(list): List of UserPresenceState's to check.
         is_mine_fn (fn): Function that returns if a user_id is ours
-        syncing_user_ids (set): Set of user_ids with active syncs.
+        user_to_num_current_syncs (dict): Mapping of user_id to number of currently
+            active syncs.
         now (int): Current time in ms.
 
     Returns:
@@ -1057,20 +952,21 @@ def handle_timeouts(user_states, is_mine_fn, syncing_user_ids, now):
     for state in user_states:
         is_mine = is_mine_fn(state.user_id)
 
-        new_state = handle_timeout(state, is_mine, syncing_user_ids, now)
+        new_state = handle_timeout(state, is_mine, user_to_num_current_syncs, now)
         if new_state:
             changes[state.user_id] = new_state
 
     return changes.values()
 
 
-def handle_timeout(state, is_mine, syncing_user_ids, now):
+def handle_timeout(state, is_mine, user_to_num_current_syncs, now):
     """Checks the presence of the user to see if any of the timers have elapsed
 
     Args:
         state (UserPresenceState)
         is_mine (bool): Whether the user is ours
-        syncing_user_ids (set): Set of user_ids with active syncs.
+        user_to_num_current_syncs (dict): Mapping of user_id to number of currently
+            active syncs.
         now (int): Current time in ms.
 
     Returns:
@@ -1104,7 +1000,7 @@ def handle_timeout(state, is_mine, syncing_user_ids, now):
 
         # If there are have been no sync for a while (and none ongoing),
         # set presence to offline
-        if user_id not in syncing_user_ids:
+        if not user_to_num_current_syncs.get(user_id, 0):
             if now - state.last_user_sync_ts > SYNC_ONLINE_TIMEOUT:
                 state = state.copy_and_replace(
                     state=PresenceState.OFFLINE,

synapse/handlers/profile.py

@@ -36,6 +36,13 @@ class ProfileHandler(BaseHandler):
             "profile", self.on_profile_query
         )
 
+        distributor = hs.get_distributor()
+
+        distributor.observe("registered_user", self.registered_user)
+
+    def registered_user(self, user):
+        return self.store.create_profile(user.localpart)
+
     @defer.inlineCallbacks
     def get_displayname(self, target_user):
         if self.hs.is_mine(target_user):

synapse/handlers/receipts.py

@@ -29,8 +29,6 @@ class ReceiptsHandler(BaseHandler):
     def __init__(self, hs):
         super(ReceiptsHandler, self).__init__(hs)
 
-        self.server_name = hs.config.server_name
-        self.store = hs.get_datastore()
         self.hs = hs
         self.federation = hs.get_replication_layer()
         self.federation.register_edu_handler(
@@ -133,9 +131,12 @@ class ReceiptsHandler(BaseHandler):
             event_ids = receipt["event_ids"]
             data = receipt["data"]
 
-            remotedomains = yield self.store.get_joined_hosts_for_room(room_id)
-            remotedomains = remotedomains.copy()
-            remotedomains.discard(self.server_name)
+            remotedomains = set()
+
+            rm_handler = self.hs.get_handlers().room_member_handler
+            yield rm_handler.fetch_room_distributions_into(
+                room_id, localusers=None, remotedomains=remotedomains
+            )
 
             logger.debug("Sending receipt to: %r", remotedomains)
 

synapse/handlers/register.py

@@ -16,13 +16,14 @@
 """Contains functions for registering clients."""
 from twisted.internet import defer
 
-from synapse.types import UserID, Requester
+from synapse.types import UserID
 from synapse.api.errors import (
     AuthError, Codes, SynapseError, RegistrationError, InvalidCaptchaError
 )
 from ._base import BaseHandler
 from synapse.util.async import run_on_reactor
 from synapse.http.client import CaptchaServerHttpClient
+from synapse.util.distributor import registered_user
 
 import logging
 import urllib
@@ -36,6 +37,8 @@ class RegistrationHandler(BaseHandler):
         super(RegistrationHandler, self).__init__(hs)
 
         self.auth = hs.get_auth()
+        self.distributor = hs.get_distributor()
+        self.distributor.declare("registered_user")
         self.captcha_client = CaptchaServerHttpClient(hs)
 
         self._next_generated_user_id = None
@@ -90,8 +93,7 @@ class RegistrationHandler(BaseHandler):
         password=None,
         generate_token=True,
         guest_access_token=None,
-        make_guest=False,
-        admin=False,
+        make_guest=False
     ):
         """Registers a new client on the server.
 
@@ -138,12 +140,9 @@ class RegistrationHandler(BaseHandler):
                 password_hash=password_hash,
                 was_guest=was_guest,
                 make_guest=make_guest,
-                create_profile_with_localpart=(
-                    # If the user was a guest then they already have a profile
-                    None if was_guest else user.localpart
-                ),
-                admin=admin,
             )
+
+            yield registered_user(self.distributor, user)
         else:
             # autogen a sequential user ID
             attempts = 0
@@ -161,8 +160,7 @@ class RegistrationHandler(BaseHandler):
                         user_id=user_id,
                         token=token,
                         password_hash=password_hash,
-                        make_guest=make_guest,
-                        create_profile_with_localpart=user.localpart,
+                        make_guest=make_guest
                     )
                 except SynapseError:
                     # if user id is taken, just generate another
@@ -170,6 +168,7 @@ class RegistrationHandler(BaseHandler):
                     user_id = None
                     token = None
                     attempts += 1
+            yield registered_user(self.distributor, user)
 
         # We used to generate default identicons here, but nowadays
         # we want clients to generate their own as part of their branding
@@ -202,8 +201,8 @@ class RegistrationHandler(BaseHandler):
             token=token,
             password_hash="",
             appservice_id=service_id,
-            create_profile_with_localpart=user.localpart,
         )
+        yield registered_user(self.distributor, user)
         defer.returnValue((user_id, token))
 
     @defer.inlineCallbacks
@@ -249,9 +248,9 @@ class RegistrationHandler(BaseHandler):
             yield self.store.register(
                 user_id=user_id,
                 token=token,
-                password_hash=None,
-                create_profile_with_localpart=user.localpart,
+                password_hash=None
             )
+            yield registered_user(self.distributor, user)
         except Exception as e:
             yield self.store.add_access_token_to_user(user_id, token)
             # Ignore Registration errors
@@ -359,62 +358,8 @@ class RegistrationHandler(BaseHandler):
         )
         defer.returnValue(data)
 
-    @defer.inlineCallbacks
-    def get_or_create_user(self, localpart, displayname, duration_seconds,
-                           password_hash=None):
-        """Creates a new user if the user does not exist,
-        else revokes all previous access tokens and generates a new one.
-
-        Args:
-            localpart : The local part of the user ID to register. If None,
-              one will be randomly generated.
-        Returns:
-            A tuple of (user_id, access_token).
-        Raises:
-            RegistrationError if there was a problem registering.
-        """
-        yield run_on_reactor()
-
-        if localpart is None:
-            raise SynapseError(400, "Request must include user id")
-
-        need_register = True
-
-        try:
-            yield self.check_username(localpart)
-        except SynapseError as e:
-            if e.errcode == Codes.USER_IN_USE:
-                need_register = False
-            else:
-                raise
-
-        user = UserID(localpart, self.hs.hostname)
-        user_id = user.to_string()
-        token = self.auth_handler().generate_short_term_login_token(
-            user_id, duration_seconds)
-
-        if need_register:
-            yield self.store.register(
-                user_id=user_id,
-                token=token,
-                password_hash=password_hash,
-                create_profile_with_localpart=user.localpart,
-            )
-        else:
-            yield self.store.user_delete_access_tokens(user_id=user_id)
-            yield self.store.add_access_token_to_user(user_id=user_id, token=token)
-
-        if displayname is not None:
-            logger.info("setting user display name: %s -> %s", user_id, displayname)
-            profile_handler = self.hs.get_handlers().profile_handler
-            yield profile_handler.set_displayname(
-                user, Requester(user, token, False), displayname
-            )
-
-        defer.returnValue((user_id, token))
-
     def auth_handler(self):
-        return self.hs.get_auth_handler()
+        return self.hs.get_handlers().auth_handler
 
     @defer.inlineCallbacks
     def guest_access_token_for(self, medium, address, inviter_user_id):

synapse/handlers/room.py

@@ -20,13 +20,12 @@ from ._base import BaseHandler
 
 from synapse.types import UserID, RoomAlias, RoomID, RoomStreamToken
 from synapse.api.constants import (
-    EventTypes, JoinRules, RoomCreationPreset, Membership,
+    EventTypes, JoinRules, RoomCreationPreset,
 )
 from synapse.api.errors import AuthError, StoreError, SynapseError
 from synapse.util import stringutils
 from synapse.util.async import concurrently_execute
 from synapse.util.caches.response_cache import ResponseCache
-from synapse.visibility import filter_events_for_client
 
 from collections import OrderedDict
 
@@ -36,8 +35,6 @@ import string
 
 logger = logging.getLogger(__name__)
 
-REMOTE_ROOM_LIST_POLL_INTERVAL = 60 * 1000
-
 id_server_scheme = "https://"
 
 
@@ -346,14 +343,8 @@ class RoomListHandler(BaseHandler):
     def __init__(self, hs):
         super(RoomListHandler, self).__init__(hs)
         self.response_cache = ResponseCache()
-        self.remote_list_request_cache = ResponseCache()
-        self.remote_list_cache = {}
-        self.fetch_looping_call = hs.get_clock().looping_call(
-            self.fetch_all_remote_lists, REMOTE_ROOM_LIST_POLL_INTERVAL
-        )
-        self.fetch_all_remote_lists()
 
-    def get_local_public_room_list(self):
+    def get_public_room_list(self):
         result = self.response_cache.get(())
         if not result:
             result = self.response_cache.set((), self._get_public_room_list())
@@ -367,10 +358,14 @@ class RoomListHandler(BaseHandler):
 
         @defer.inlineCallbacks
         def handle_room(room_id):
-            current_state = yield self.state_handler.get_current_state(room_id)
+            # We pull each bit of state out indvidually to avoid pulling the
+            # full state into memory. Due to how the caching works this should
+            # be fairly quick, even if not originally in the cache.
+            def get_state(etype, state_key):
+                return self.state_handler.get_current_state(room_id, etype, state_key)
 
             # Double check that this is actually a public room.
-            join_rules_event = current_state.get((EventTypes.JoinRules, ""))
+            join_rules_event = yield get_state(EventTypes.JoinRules, "")
             if join_rules_event:
                 join_rule = join_rules_event.content.get("join_rule", None)
                 if join_rule and join_rule != JoinRules.PUBLIC:
@@ -378,51 +373,47 @@ class RoomListHandler(BaseHandler):
 
             result = {"room_id": room_id}
 
-            num_joined_users = len([
-                1 for _, event in current_state.items()
-                if event.type == EventTypes.Member
-                and event.membership == Membership.JOIN
-            ])
-            if num_joined_users == 0:
+            joined_users = yield self.store.get_users_in_room(room_id)
+            if len(joined_users) == 0:
                 return
 
-            result["num_joined_members"] = num_joined_users
+            result["num_joined_members"] = len(joined_users)
 
             aliases = yield self.store.get_aliases_for_room(room_id)
             if aliases:
                 result["aliases"] = aliases
 
-            name_event = yield current_state.get((EventTypes.Name, ""))
+            name_event = yield get_state(EventTypes.Name, "")
             if name_event:
                 name = name_event.content.get("name", None)
                 if name:
                     result["name"] = name
 
-            topic_event = current_state.get((EventTypes.Topic, ""))
+            topic_event = yield get_state(EventTypes.Topic, "")
             if topic_event:
                 topic = topic_event.content.get("topic", None)
                 if topic:
                     result["topic"] = topic
 
-            canonical_event = current_state.get((EventTypes.CanonicalAlias, ""))
+            canonical_event = yield get_state(EventTypes.CanonicalAlias, "")
             if canonical_event:
                 canonical_alias = canonical_event.content.get("alias", None)
                 if canonical_alias:
                     result["canonical_alias"] = canonical_alias
 
-            visibility_event = current_state.get((EventTypes.RoomHistoryVisibility, ""))
+            visibility_event = yield get_state(EventTypes.RoomHistoryVisibility, "")
             visibility = None
             if visibility_event:
                 visibility = visibility_event.content.get("history_visibility", None)
             result["world_readable"] = visibility == "world_readable"
 
-            guest_event = current_state.get((EventTypes.GuestAccess, ""))
+            guest_event = yield get_state(EventTypes.GuestAccess, "")
             guest = None
             if guest_event:
                 guest = guest_event.content.get("guest_access", None)
             result["guest_can_join"] = guest == "can_join"
 
-            avatar_event = current_state.get(("m.room.avatar", ""))
+            avatar_event = yield get_state("m.room.avatar", "")
             if avatar_event:
                 avatar_url = avatar_event.content.get("url", None)
                 if avatar_url:
@@ -435,55 +426,6 @@ class RoomListHandler(BaseHandler):
         # FIXME (erikj): START is no longer a valid value
         defer.returnValue({"start": "START", "end": "END", "chunk": results})
 
-    @defer.inlineCallbacks
-    def fetch_all_remote_lists(self):
-        deferred = self.hs.get_replication_layer().get_public_rooms(
-            self.hs.config.secondary_directory_servers
-        )
-        self.remote_list_request_cache.set((), deferred)
-        self.remote_list_cache = yield deferred
-
-    @defer.inlineCallbacks
-    def get_aggregated_public_room_list(self):
-        """
-        Get the public room list from this server and the servers
-        specified in the secondary_directory_servers config option.
-        XXX: Pagination...
-        """
-        # We return the results from out cache which is updated by a looping call,
-        # unless we're missing a cache entry, in which case wait for the result
-        # of the fetch if there's one in progress. If not, omit that server.
-        wait = False
-        for s in self.hs.config.secondary_directory_servers:
-            if s not in self.remote_list_cache:
-                logger.warn("No cached room list from %s: waiting for fetch", s)
-                wait = True
-                break
-
-        if wait and self.remote_list_request_cache.get(()):
-            yield self.remote_list_request_cache.get(())
-
-        public_rooms = yield self.get_local_public_room_list()
-
-        # keep track of which room IDs we've seen so we can de-dup
-        room_ids = set()
-
-        # tag all the ones in our list with our server name.
-        # Also add the them to the de-deping set
-        for room in public_rooms['chunk']:
-            room["server_name"] = self.hs.hostname
-            room_ids.add(room["room_id"])
-
-        # Now add the results from federation
-        for server_name, server_result in self.remote_list_cache.items():
-            for room in server_result["chunk"]:
-                if room["room_id"] not in room_ids:
-                    room["server_name"] = server_name
-                    public_rooms["chunk"].append(room)
-                    room_ids.add(room["room_id"])
-
-        defer.returnValue(public_rooms)
-
 
 class RoomContextHandler(BaseHandler):
     @defer.inlineCallbacks
@@ -507,12 +449,10 @@ class RoomContextHandler(BaseHandler):
         now_token = yield self.hs.get_event_sources().get_current_token()
 
         def filter_evts(events):
-            return filter_events_for_client(
-                self.store,
+            return self._filter_events_for_client(
                 user.to_string(),
                 events,
-                is_peeking=is_guest
-            )
+                is_peeking=is_guest)
 
         event = yield self.store.get_event(event_id, get_prev_content=True,
                                            allow_none=True)

synapse/handlers/room_member.py

@@ -55,6 +55,35 @@ class RoomMemberHandler(BaseHandler):
         self.distributor.declare("user_joined_room")
         self.distributor.declare("user_left_room")
 
+    @defer.inlineCallbacks
+    def get_room_members(self, room_id):
+        users = yield self.store.get_users_in_room(room_id)
+
+        defer.returnValue([UserID.from_string(u) for u in users])
+
+    @defer.inlineCallbacks
+    def fetch_room_distributions_into(self, room_id, localusers=None,
+                                      remotedomains=None, ignore_user=None):
+        """Fetch the distribution of a room, adding elements to either
+        'localusers' or 'remotedomains', which should be a set() if supplied.
+        If ignore_user is set, ignore that user.
+
+        This function returns nothing; its result is performed by the
+        side-effect on the two passed sets. This allows easy accumulation of
+        member lists of multiple rooms at once if required.
+        """
+        members = yield self.get_room_members(room_id)
+        for member in members:
+            if ignore_user is not None and member == ignore_user:
+                continue
+
+            if self.hs.is_mine(member):
+                if localusers is not None:
+                    localusers.add(member)
+            else:
+                if remotedomains is not None:
+                    remotedomains.add(member.domain)
+
     @defer.inlineCallbacks
     def _local_membership_update(
         self, requester, target, room_id, membership,
@@ -84,7 +113,7 @@ class RoomMemberHandler(BaseHandler):
             prev_event_ids=prev_event_ids,
         )
 
-        yield msg_handler.handle_new_client_event(
+        yield self.handle_new_client_event(
             requester,
             event,
             context,
@@ -203,7 +232,7 @@ class RoomMemberHandler(BaseHandler):
         if old_membership == "ban" and action != "unban":
             raise SynapseError(
                 403,
-                "Cannot %s user who was banned" % (action,),
+                "Cannot %s user who was is banned" % (action,),
                 errcode=Codes.BAD_STATE
             )
 
@@ -328,7 +357,7 @@ class RoomMemberHandler(BaseHandler):
                 # so don't really fit into the general auth process.
                 raise AuthError(403, "Guest access not allowed")
 
-        yield message_handler.handle_new_client_event(
+        yield self.handle_new_client_event(
             requester,
             event,
             context,
@@ -397,6 +426,21 @@ class RoomMemberHandler(BaseHandler):
         if invite:
             defer.returnValue(UserID.from_string(invite.sender))
 
+    @defer.inlineCallbacks
+    def get_joined_rooms_for_user(self, user):
+        """Returns a list of roomids that the user has any of the given
+        membership states in."""
+
+        rooms = yield self.store.get_rooms_for_user(
+            user.to_string(),
+        )
+
+        # For some reason the list of events contains duplicates
+        # TODO(paul): work out why because I really don't think it should
+        room_ids = set(r.room_id for r in rooms)
+
+        defer.returnValue(room_ids)
+
     @defer.inlineCallbacks
     def do_3pid_invite(
             self,
@@ -413,7 +457,8 @@ class RoomMemberHandler(BaseHandler):
         )
 
         if invitee:
-            yield self.update_membership(
+            handler = self.hs.get_handlers().room_member_handler
+            yield handler.update_membership(
                 requester,
                 UserID.from_string(invitee),
                 room_id,

synapse/handlers/search.py

@@ -21,7 +21,6 @@ from synapse.api.constants import Membership, EventTypes
 from synapse.api.filtering import Filter
 from synapse.api.errors import SynapseError
 from synapse.events.utils import serialize_event
-from synapse.visibility import filter_events_for_client
 
 from unpaddedbase64 import decode_base64, encode_base64
 
@@ -173,8 +172,8 @@ class SearchHandler(BaseHandler):
 
             filtered_events = search_filter.filter([r["event"] for r in results])
 
-            events = yield filter_events_for_client(
-                self.store, user.to_string(), filtered_events
+            events = yield self._filter_events_for_client(
+                user.to_string(), filtered_events
             )
 
             events.sort(key=lambda e: -rank_map[e.event_id])
@@ -224,8 +223,8 @@ class SearchHandler(BaseHandler):
                     r["event"] for r in results
                 ])
 
-                events = yield filter_events_for_client(
-                    self.store, user.to_string(), filtered_events
+                events = yield self._filter_events_for_client(
+                    user.to_string(), filtered_events
                 )
 
                 room_events.extend(events)
@@ -282,12 +281,12 @@ class SearchHandler(BaseHandler):
                     event.room_id, event.event_id, before_limit, after_limit
                 )
 
-                res["events_before"] = yield filter_events_for_client(
-                    self.store, user.to_string(), res["events_before"]
+                res["events_before"] = yield self._filter_events_for_client(
+                    user.to_string(), res["events_before"]
                 )
 
-                res["events_after"] = yield filter_events_for_client(
-                    self.store, user.to_string(), res["events_after"]
+                res["events_after"] = yield self._filter_events_for_client(
+                    user.to_string(), res["events_after"]
                 )
 
                 res["start"] = now_token.copy_and_replace(

synapse/handlers/typing.py

@@ -15,6 +15,8 @@
 
 from twisted.internet import defer
 
+from ._base import BaseHandler
+
 from synapse.api.errors import SynapseError, AuthError
 from synapse.util.logcontext import PreserveLoggingContext
 from synapse.util.metrics import Measure
@@ -30,16 +32,14 @@ logger = logging.getLogger(__name__)
 
 # A tiny object useful for storing a user's membership in a room, as a mapping
 # key
-RoomMember = namedtuple("RoomMember", ("room_id", "user_id"))
+RoomMember = namedtuple("RoomMember", ("room_id", "user"))
 
 
-class TypingHandler(object):
+class TypingNotificationHandler(BaseHandler):
     def __init__(self, hs):
-        self.store = hs.get_datastore()
-        self.server_name = hs.config.server_name
-        self.auth = hs.get_auth()
-        self.is_mine_id = hs.is_mine_id
-        self.notifier = hs.get_notifier()
+        super(TypingNotificationHandler, self).__init__(hs)
+
+        self.homeserver = hs
 
         self.clock = hs.get_clock()
 
@@ -67,23 +67,20 @@ class TypingHandler(object):
 
     @defer.inlineCallbacks
     def started_typing(self, target_user, auth_user, room_id, timeout):
-        target_user_id = target_user.to_string()
-        auth_user_id = auth_user.to_string()
-
-        if not self.is_mine_id(target_user_id):
+        if not self.hs.is_mine(target_user):
             raise SynapseError(400, "User is not hosted on this Home Server")
 
-        if target_user_id != auth_user_id:
+        if target_user != auth_user:
             raise AuthError(400, "Cannot set another user's typing state")
 
-        yield self.auth.check_joined_room(room_id, target_user_id)
+        yield self.auth.check_joined_room(room_id, target_user.to_string())
 
         logger.debug(
-            "%s has started typing in %s", target_user_id, room_id
+            "%s has started typing in %s", target_user.to_string(), room_id
         )
 
         until = self.clock.time_msec() + timeout
-        member = RoomMember(room_id=room_id, user_id=target_user_id)
+        member = RoomMember(room_id=room_id, user=target_user)
 
         was_present = member in self._member_typing_until
 
@@ -107,28 +104,25 @@ class TypingHandler(object):
 
         yield self._push_update(
             room_id=room_id,
-            user_id=target_user_id,
+            user=target_user,
             typing=True,
         )
 
     @defer.inlineCallbacks
     def stopped_typing(self, target_user, auth_user, room_id):
-        target_user_id = target_user.to_string()
-        auth_user_id = auth_user.to_string()
-
-        if not self.is_mine_id(target_user_id):
+        if not self.hs.is_mine(target_user):
             raise SynapseError(400, "User is not hosted on this Home Server")
 
-        if target_user_id != auth_user_id:
+        if target_user != auth_user:
             raise AuthError(400, "Cannot set another user's typing state")
 
-        yield self.auth.check_joined_room(room_id, target_user_id)
+        yield self.auth.check_joined_room(room_id, target_user.to_string())
 
         logger.debug(
-            "%s has stopped typing in %s", target_user_id, room_id
+            "%s has stopped typing in %s", target_user.to_string(), room_id
         )
 
-        member = RoomMember(room_id=room_id, user_id=target_user_id)
+        member = RoomMember(room_id=room_id, user=target_user)
 
         if member in self._member_typing_timer:
             self.clock.cancel_call_later(self._member_typing_timer[member])
@@ -138,9 +132,8 @@ class TypingHandler(object):
 
     @defer.inlineCallbacks
     def user_left_room(self, user, room_id):
-        user_id = user.to_string()
-        if self.is_mine_id(user_id):
-            member = RoomMember(room_id=room_id, user_id=user_id)
+        if self.hs.is_mine(user):
+            member = RoomMember(room_id=room_id, user=user)
             yield self._stopped_typing(member)
 
     @defer.inlineCallbacks
@@ -151,7 +144,7 @@ class TypingHandler(object):
 
         yield self._push_update(
             room_id=member.room_id,
-            user_id=member.user_id,
+            user=member.user,
             typing=False,
         )
 
@@ -163,53 +156,61 @@ class TypingHandler(object):
             del self._member_typing_timer[member]
 
     @defer.inlineCallbacks
-    def _push_update(self, room_id, user_id, typing):
-        domains = yield self.store.get_joined_hosts_for_room(room_id)
+    def _push_update(self, room_id, user, typing):
+        localusers = set()
+        remotedomains = set()
+
+        rm_handler = self.homeserver.get_handlers().room_member_handler
+        yield rm_handler.fetch_room_distributions_into(
+            room_id, localusers=localusers, remotedomains=remotedomains
+        )
+
+        if localusers:
+            self._push_update_local(
+                room_id=room_id,
+                user=user,
+                typing=typing
+            )
 
         deferreds = []
-        for domain in domains:
-            if domain == self.server_name:
-                self._push_update_local(
-                    room_id=room_id,
-                    user_id=user_id,
-                    typing=typing
-                )
-            else:
-                deferreds.append(self.federation.send_edu(
-                    destination=domain,
-                    edu_type="m.typing",
-                    content={
-                        "room_id": room_id,
-                        "user_id": user_id,
-                        "typing": typing,
-                    },
-                ))
+        for domain in remotedomains:
+            deferreds.append(self.federation.send_edu(
+                destination=domain,
+                edu_type="m.typing",
+                content={
+                    "room_id": room_id,
+                    "user_id": user.to_string(),
+                    "typing": typing,
+                },
+            ))
 
         yield defer.DeferredList(deferreds, consumeErrors=True)
 
     @defer.inlineCallbacks
     def _recv_edu(self, origin, content):
         room_id = content["room_id"]
-        user_id = content["user_id"]
+        user = UserID.from_string(content["user_id"])
 
-        # Check that the string is a valid user id
-        UserID.from_string(user_id)
+        localusers = set()
 
-        domains = yield self.store.get_joined_hosts_for_room(room_id)
+        rm_handler = self.homeserver.get_handlers().room_member_handler
+        yield rm_handler.fetch_room_distributions_into(
+            room_id, localusers=localusers
+        )
 
-        if self.server_name in domains:
+        if localusers:
             self._push_update_local(
                 room_id=room_id,
-                user_id=user_id,
+                user=user,
                 typing=content["typing"]
             )
 
-    def _push_update_local(self, room_id, user_id, typing):
+    def _push_update_local(self, room_id, user, typing):
         room_set = self._room_typing.setdefault(room_id, set())
         if typing:
-            room_set.add(user_id)
+            room_set.add(user)
         else:
-            room_set.discard(user_id)
+            room_set.discard(user)
 
         self._latest_room_serial += 1
         self._room_serials[room_id] = self._latest_room_serial
@@ -221,14 +222,13 @@ class TypingHandler(object):
 
     def get_all_typing_updates(self, last_id, current_id):
         # TODO: Work out a way to do this without scanning the entire state.
-        if last_id == current_id:
-            return []
-
         rows = []
         for room_id, serial in self._room_serials.items():
             if last_id < serial and serial <= current_id:
                 typing = self._room_typing[room_id]
-                typing_bytes = json.dumps(list(typing), ensure_ascii=False)
+                typing_bytes = json.dumps([
+                    u.to_string() for u in typing
+                ], ensure_ascii=False)
                 rows.append((serial, room_id, typing_bytes))
         rows.sort()
         return rows
@@ -238,26 +238,34 @@ class TypingNotificationEventSource(object):
     def __init__(self, hs):
         self.hs = hs
         self.clock = hs.get_clock()
-        # We can't call get_typing_handler here because there's a cycle:
-        #
-        #   Typing -> Notifier -> TypingNotificationEventSource -> Typing
-        #
-        self.get_typing_handler = hs.get_typing_handler
+        self._handler = None
+        self._room_member_handler = None
+
+    def handler(self):
+        # Avoid cyclic dependency in handler setup
+        if not self._handler:
+            self._handler = self.hs.get_handlers().typing_notification_handler
+        return self._handler
+
+    def room_member_handler(self):
+        if not self._room_member_handler:
+            self._room_member_handler = self.hs.get_handlers().room_member_handler
+        return self._room_member_handler
 
     def _make_event_for(self, room_id):
-        typing = self.get_typing_handler()._room_typing[room_id]
+        typing = self.handler()._room_typing[room_id]
         return {
             "type": "m.typing",
             "room_id": room_id,
             "content": {
-                "user_ids": list(typing),
+                "user_ids": [u.to_string() for u in typing],
             },
         }
 
     def get_new_events(self, from_key, room_ids, **kwargs):
         with Measure(self.clock, "typing.get_new_events"):
             from_key = int(from_key)
-            handler = self.get_typing_handler()
+            handler = self.handler()
 
             events = []
             for room_id in room_ids:
@@ -271,7 +279,7 @@ class TypingNotificationEventSource(object):
             return events, handler._latest_room_serial
 
     def get_current_key(self):
-        return self.get_typing_handler()._latest_room_serial
+        return self.handler()._latest_room_serial
 
     def get_pagination_rows(self, user, pagination_config, key):
         return ([], pagination_config.from_key)

synapse/http/client.py

@@ -24,13 +24,12 @@ from synapse.http.endpoint import SpiderEndpoint
 
 from canonicaljson import encode_canonical_json
 
-from twisted.internet import defer, reactor, ssl, protocol, task
+from twisted.internet import defer, reactor, ssl, protocol
 from twisted.internet.endpoints import SSL4ClientEndpoint, TCP4ClientEndpoint
 from twisted.web.client import (
     BrowserLikeRedirectAgent, ContentDecoderAgent, GzipDecoder, Agent,
-    readBody, PartialDownloadError,
+    readBody, FileBodyProducer, PartialDownloadError,
 )
-from twisted.web.client import FileBodyProducer as TwistedFileBodyProducer
 from twisted.web.http import PotentialDataLoss
 from twisted.web.http_headers import Headers
 from twisted.web._newclient import ResponseDone
@@ -381,14 +380,13 @@ class CaptchaServerHttpClient(SimpleHttpClient):
 class SpiderEndpointFactory(object):
     def __init__(self, hs):
         self.blacklist = hs.config.url_preview_ip_range_blacklist
-        self.whitelist = hs.config.url_preview_ip_range_whitelist
         self.policyForHTTPS = hs.get_http_client_context_factory()
 
     def endpointForURI(self, uri):
         logger.info("Getting endpoint for %s", uri.toBytes())
         if uri.scheme == "http":
             return SpiderEndpoint(
-                reactor, uri.host, uri.port, self.blacklist, self.whitelist,
+                reactor, uri.host, uri.port, self.blacklist,
                 endpoint=TCP4ClientEndpoint,
                 endpoint_kw_args={
                     'timeout': 15
@@ -397,7 +395,7 @@ class SpiderEndpointFactory(object):
         elif uri.scheme == "https":
             tlsPolicy = self.policyForHTTPS.creatorForNetloc(uri.host, uri.port)
             return SpiderEndpoint(
-                reactor, uri.host, uri.port, self.blacklist, self.whitelist,
+                reactor, uri.host, uri.port, self.blacklist,
                 endpoint=SSL4ClientEndpoint,
                 endpoint_kw_args={
                     'sslContextFactory': tlsPolicy,
@@ -469,26 +467,3 @@ class InsecureInterceptableContextFactory(ssl.ContextFactory):
 
     def creatorForNetloc(self, hostname, port):
         return self
-
-
-class FileBodyProducer(TwistedFileBodyProducer):
-    """Workaround for https://twistedmatrix.com/trac/ticket/8473
-
-    We override the pauseProducing and resumeProducing methods in twisted's
-    FileBodyProducer so that they do not raise exceptions if the task has
-    already completed.
-    """
-
-    def pauseProducing(self):
-        try:
-            super(FileBodyProducer, self).pauseProducing()
-        except task.TaskDone:
-            # task has already completed
-            pass
-
-    def resumeProducing(self):
-        try:
-            super(FileBodyProducer, self).resumeProducing()
-        except task.NotPaused:
-            # task was not paused (probably because it had already completed)
-            pass

synapse/http/endpoint.py

@@ -79,13 +79,12 @@ class SpiderEndpoint(object):
     """An endpoint which refuses to connect to blacklisted IP addresses
     Implements twisted.internet.interfaces.IStreamClientEndpoint.
     """
-    def __init__(self, reactor, host, port, blacklist, whitelist,
+    def __init__(self, reactor, host, port, blacklist,
                  endpoint=TCP4ClientEndpoint, endpoint_kw_args={}):
         self.reactor = reactor
         self.host = host
         self.port = port
         self.blacklist = blacklist
-        self.whitelist = whitelist
         self.endpoint = endpoint
         self.endpoint_kw_args = endpoint_kw_args
 
@@ -94,13 +93,10 @@ class SpiderEndpoint(object):
         address = yield self.reactor.resolve(self.host)
 
         from netaddr import IPAddress
-        ip_address = IPAddress(address)
-
-        if ip_address in self.blacklist:
-            if self.whitelist is None or ip_address not in self.whitelist:
-                raise ConnectError(
-                    "Refusing to spider blacklisted IP address %s" % address
-                )
+        if IPAddress(address) in self.blacklist:
+            raise ConnectError(
+                "Refusing to spider blacklisted IP address %s" % address
+            )
 
         logger.info("Connecting to %s:%s", address, self.port)
         endpoint = self.endpoint(

synapse/http/server.py

@@ -74,12 +74,7 @@ response_db_txn_duration = metrics.register_distribution(
 _next_request_id = 0
 
 
-def request_handler(report_metrics=True):
-    """Decorator for ``wrap_request_handler``"""
-    return lambda request_handler: wrap_request_handler(request_handler, report_metrics)
-
-
-def wrap_request_handler(request_handler, report_metrics):
+def request_handler(request_handler):
     """Wraps a method that acts as a request handler with the necessary logging
     and exception handling.
 
@@ -101,12 +96,7 @@ def wrap_request_handler(request_handler, report_metrics):
         global _next_request_id
         request_id = "%s-%s" % (request.method, _next_request_id)
         _next_request_id += 1
-
         with LoggingContext(request_id) as request_context:
-            if report_metrics:
-                request_metrics = RequestMetrics()
-                request_metrics.start(self.clock)
-
             request_context.request = request_id
             with request.processing():
                 try:
@@ -143,14 +133,6 @@ def wrap_request_handler(request_handler, report_metrics):
                         },
                         send_cors=True
                     )
-                finally:
-                    try:
-                        if report_metrics:
-                            request_metrics.stop(
-                                self.clock, request, self.__class__.__name__
-                            )
-                    except:
-                        pass
     return wrapped_request_handler
 
 
@@ -215,23 +197,19 @@ class JsonResource(HttpServer, resource.Resource):
         self._async_render(request)
         return server.NOT_DONE_YET
 
-    # Disable metric reporting because _async_render does its own metrics.
-    # It does its own metric reporting because _async_render dispatches to
-    # a callback and it's the class name of that callback we want to report
-    # against rather than the JsonResource itself.
-    @request_handler(report_metrics=False)
+    @request_handler
     @defer.inlineCallbacks
     def _async_render(self, request):
         """ This gets called from render() every time someone sends us a request.
             This checks if anyone has registered a callback for that method and
             path.
         """
+        start = self.clock.time_msec()
         if request.method == "OPTIONS":
             self._send_response(request, 200, {})
             return
 
-        request_metrics = RequestMetrics()
-        request_metrics.start(self.clock)
+        start_context = LoggingContext.current_context()
 
         # Loop through all the registered callbacks to check if the method
         # and path regex match
@@ -263,7 +241,40 @@ class JsonResource(HttpServer, resource.Resource):
                 self._send_response(request, code, response)
 
             try:
-                request_metrics.stop(self.clock, request, servlet_classname)
+                context = LoggingContext.current_context()
+
+                tag = ""
+                if context:
+                    tag = context.tag
+
+                    if context != start_context:
+                        logger.warn(
+                            "Context have unexpectedly changed %r, %r",
+                            context, self.start_context
+                        )
+                        return
+
+                incoming_requests_counter.inc(request.method, servlet_classname, tag)
+
+                response_timer.inc_by(
+                    self.clock.time_msec() - start, request.method,
+                    servlet_classname, tag
+                )
+
+                ru_utime, ru_stime = context.get_resource_usage()
+
+                response_ru_utime.inc_by(
+                    ru_utime, request.method, servlet_classname, tag
+                )
+                response_ru_stime.inc_by(
+                    ru_stime, request.method, servlet_classname, tag
+                )
+                response_db_txn_count.inc_by(
+                    context.db_txn_count, request.method, servlet_classname, tag
+                )
+                response_db_txn_duration.inc_by(
+                    context.db_txn_duration, request.method, servlet_classname, tag
+                )
             except:
                 pass
 
@@ -296,48 +307,6 @@ class JsonResource(HttpServer, resource.Resource):
         )
 
 
-class RequestMetrics(object):
-    def start(self, clock):
-        self.start = clock.time_msec()
-        self.start_context = LoggingContext.current_context()
-
-    def stop(self, clock, request, servlet_classname):
-        context = LoggingContext.current_context()
-
-        tag = ""
-        if context:
-            tag = context.tag
-
-            if context != self.start_context:
-                logger.warn(
-                    "Context have unexpectedly changed %r, %r",
-                    context, self.start_context
-                )
-                return
-
-        incoming_requests_counter.inc(request.method, servlet_classname, tag)
-
-        response_timer.inc_by(
-            clock.time_msec() - self.start, request.method,
-            servlet_classname, tag
-        )
-
-        ru_utime, ru_stime = context.get_resource_usage()
-
-        response_ru_utime.inc_by(
-            ru_utime, request.method, servlet_classname, tag
-        )
-        response_ru_stime.inc_by(
-            ru_stime, request.method, servlet_classname, tag
-        )
-        response_db_txn_count.inc_by(
-            context.db_txn_count, request.method, servlet_classname, tag
-        )
-        response_db_txn_duration.inc_by(
-            context.db_txn_duration, request.method, servlet_classname, tag
-        )
-
-
 class RootRedirect(resource.Resource):
     """Redirects the root '/' path to another path."""
 

synapse/http/site.py

@@ -1,146 +0,0 @@
-# Copyright 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from synapse.util.logcontext import LoggingContext
-from twisted.web.server import Site, Request
-
-import contextlib
-import logging
-import re
-import time
-
-ACCESS_TOKEN_RE = re.compile(r'(\?.*access(_|%5[Ff])token=)[^&]*(.*)$')
-
-
-class SynapseRequest(Request):
-    def __init__(self, site, *args, **kw):
-        Request.__init__(self, *args, **kw)
-        self.site = site
-        self.authenticated_entity = None
-        self.start_time = 0
-
-    def __repr__(self):
-        # We overwrite this so that we don't log ``access_token``
-        return '<%s at 0x%x method=%s uri=%s clientproto=%s site=%s>' % (
-            self.__class__.__name__,
-            id(self),
-            self.method,
-            self.get_redacted_uri(),
-            self.clientproto,
-            self.site.site_tag,
-        )
-
-    def get_redacted_uri(self):
-        return ACCESS_TOKEN_RE.sub(
-            r'\1<redacted>\3',
-            self.uri
-        )
-
-    def get_user_agent(self):
-        return self.requestHeaders.getRawHeaders("User-Agent", [None])[-1]
-
-    def started_processing(self):
-        self.site.access_logger.info(
-            "%s - %s - Received request: %s %s",
-            self.getClientIP(),
-            self.site.site_tag,
-            self.method,
-            self.get_redacted_uri()
-        )
-        self.start_time = int(time.time() * 1000)
-
-    def finished_processing(self):
-
-        try:
-            context = LoggingContext.current_context()
-            ru_utime, ru_stime = context.get_resource_usage()
-            db_txn_count = context.db_txn_count
-            db_txn_duration = context.db_txn_duration
-        except:
-            ru_utime, ru_stime = (0, 0)
-            db_txn_count, db_txn_duration = (0, 0)
-
-        self.site.access_logger.info(
-            "%s - %s - {%s}"
-            " Processed request: %dms (%dms, %dms) (%dms/%d)"
-            " %sB %s \"%s %s %s\" \"%s\"",
-            self.getClientIP(),
-            self.site.site_tag,
-            self.authenticated_entity,
-            int(time.time() * 1000) - self.start_time,
-            int(ru_utime * 1000),
-            int(ru_stime * 1000),
-            int(db_txn_duration * 1000),
-            int(db_txn_count),
-            self.sentLength,
-            self.code,
-            self.method,
-            self.get_redacted_uri(),
-            self.clientproto,
-            self.get_user_agent(),
-        )
-
-    @contextlib.contextmanager
-    def processing(self):
-        self.started_processing()
-        yield
-        self.finished_processing()
-
-
-class XForwardedForRequest(SynapseRequest):
-    def __init__(self, *args, **kw):
-        SynapseRequest.__init__(self, *args, **kw)
-
-    """
-    Add a layer on top of another request that only uses the value of an
-    X-Forwarded-For header as the result of C{getClientIP}.
-    """
-    def getClientIP(self):
-        """
-        @return: The client address (the first address) in the value of the
-            I{X-Forwarded-For header}.  If the header is not present, return
-            C{b"-"}.
-        """
-        return self.requestHeaders.getRawHeaders(
-            b"x-forwarded-for", [b"-"])[0].split(b",")[0].strip()
-
-
-class SynapseRequestFactory(object):
-    def __init__(self, site, x_forwarded_for):
-        self.site = site
-        self.x_forwarded_for = x_forwarded_for
-
-    def __call__(self, *args, **kwargs):
-        if self.x_forwarded_for:
-            return XForwardedForRequest(self.site, *args, **kwargs)
-        else:
-            return SynapseRequest(self.site, *args, **kwargs)
-
-
-class SynapseSite(Site):
-    """
-    Subclass of a twisted http Site that does access logging with python's
-    standard logging
-    """
-    def __init__(self, logger_name, site_tag, config, resource, *args, **kwargs):
-        Site.__init__(self, resource, *args, **kwargs)
-
-        self.site_tag = site_tag
-
-        proxied = config.get("x_forwarded", False)
-        self.requestFactory = SynapseRequestFactory(self, proxied)
-        self.access_logger = logging.getLogger(logger_name)
-
-    def log(self, request):
-        pass

synapse/metrics/__init__.py

@@ -22,7 +22,6 @@ import functools
 import os
 import stat
 import time
-import gc
 
 from twisted.internet import reactor
 
@@ -34,7 +33,11 @@ from .metric import (
 logger = logging.getLogger(__name__)
 
 
-all_metrics = []
+# We'll keep all the available metrics in a single toplevel dict, one shared
+# for the entire process. We don't currently support per-HomeServer instances
+# of metrics, because in practice any one python VM will host only one
+# HomeServer anyway. This makes a lot of implementation neater
+all_metrics = {}
 
 
 class Metrics(object):
@@ -50,7 +53,7 @@ class Metrics(object):
 
         metric = metric_class(full_name, *args, **kwargs)
 
-        all_metrics.append(metric)
+        all_metrics[full_name] = metric
         return metric
 
     def register_counter(self, *args, **kwargs):
@@ -81,12 +84,12 @@ def render_all():
     # TODO(paul): Internal hack
     update_resource_metrics()
 
-    for metric in all_metrics:
+    for name in sorted(all_metrics.keys()):
         try:
-            strs += metric.render()
+            strs += all_metrics[name].render()
         except Exception:
-            strs += ["# FAILED to render"]
-            logger.exception("Failed to render metric")
+            strs += ["# FAILED to render %s" % name]
+            logger.exception("Failed to render %s metric", name)
 
     strs.append("")  # to generate a final CRLF
 
@@ -153,13 +156,6 @@ reactor_metrics = get_metrics_for("reactor")
 tick_time = reactor_metrics.register_distribution("tick_time")
 pending_calls_metric = reactor_metrics.register_distribution("pending_calls")
 
-gc_time = reactor_metrics.register_distribution("gc_time", labels=["gen"])
-gc_unreachable = reactor_metrics.register_counter("gc_unreachable", labels=["gen"])
-
-reactor_metrics.register_callback(
-    "gc_counts", lambda: {(i,): v for i, v in enumerate(gc.get_count())}, labels=["gen"]
-)
-
 
 def runUntilCurrentTimer(func):
 
@@ -186,22 +182,6 @@ def runUntilCurrentTimer(func):
         end = time.time() * 1000
         tick_time.inc_by(end - start)
         pending_calls_metric.inc_by(num_pending)
-
-        # Check if we need to do a manual GC (since its been disabled), and do
-        # one if necessary.
-        threshold = gc.get_threshold()
-        counts = gc.get_count()
-        for i in (2, 1, 0):
-            if threshold[i] < counts[i]:
-                logger.info("Collecting gc %d", i)
-
-                start = time.time() * 1000
-                unreachable = gc.collect(i)
-                end = time.time() * 1000
-
-                gc_time.inc_by(end - start, i)
-                gc_unreachable.inc_by(unreachable, i)
-
         return ret
 
     return f
@@ -216,9 +196,5 @@ try:
     # runUntilCurrent is called when we have pending calls. It is called once
     # per iteratation after fd polling.
     reactor.runUntilCurrent = runUntilCurrentTimer(reactor.runUntilCurrent)
-
-    # We manually run the GC each reactor tick so that we can get some metrics
-    # about time spent doing GC,
-    gc.disable()
 except AttributeError:
     pass

synapse/metrics/metric.py

@@ -47,6 +47,9 @@ class BaseMetric(object):
                       for k, v in zip(self.labels, values)])
         )
 
+    def render(self):
+        return map_concat(self.render_item, sorted(self.counts.keys()))
+
 
 class CounterMetric(BaseMetric):
     """The simplest kind of metric; one that stores a monotonically-increasing
@@ -80,9 +83,6 @@ class CounterMetric(BaseMetric):
     def render_item(self, k):
         return ["%s%s %d" % (self.name, self._render_key(k), self.counts[k])]
 
-    def render(self):
-        return map_concat(self.render_item, sorted(self.counts.keys()))
-
 
 class CallbackMetric(BaseMetric):
     """A metric that returns the numeric value returned by a callback whenever
@@ -126,30 +126,30 @@ class DistributionMetric(object):
 
 
 class CacheMetric(object):
-    __slots__ = ("name", "cache_name", "hits", "misses", "size_callback")
+    """A combination of two CounterMetrics, one to count cache hits and one to
+    count a total, and a callback metric to yield the current size.
 
-    def __init__(self, name, size_callback, cache_name):
+    This metric generates standard metric name pairs, so that monitoring rules
+    can easily be applied to measure hit ratio."""
+
+    def __init__(self, name, size_callback, labels=[]):
         self.name = name
-        self.cache_name = cache_name
 
-        self.hits = 0
-        self.misses = 0
+        self.hits = CounterMetric(name + ":hits", labels=labels)
+        self.total = CounterMetric(name + ":total", labels=labels)
 
-        self.size_callback = size_callback
+        self.size = CallbackMetric(
+            name + ":size",
+            callback=size_callback,
+            labels=labels,
+        )
 
-    def inc_hits(self):
-        self.hits += 1
+    def inc_hits(self, *values):
+        self.hits.inc(*values)
+        self.total.inc(*values)
 
-    def inc_misses(self):
-        self.misses += 1
+    def inc_misses(self, *values):
+        self.total.inc(*values)
 
     def render(self):
-        size = self.size_callback()
-        hits = self.hits
-        total = self.misses + self.hits
-
-        return [
-            """%s:hits{name="%s"} %d""" % (self.name, self.cache_name, hits),
-            """%s:total{name="%s"} %d""" % (self.name, self.cache_name, total),
-            """%s:size{name="%s"} %d""" % (self.name, self.cache_name, size),
-        ]
+        return self.hits.render() + self.total.render() + self.size.render()

synapse/notifier.py

@@ -14,14 +14,13 @@
 # limitations under the License.
 
 from twisted.internet import defer
-from synapse.api.constants import EventTypes, Membership
+from synapse.api.constants import EventTypes
 from synapse.api.errors import AuthError
 
 from synapse.util.logutils import log_function
 from synapse.util.async import ObservableDeferred
 from synapse.util.logcontext import PreserveLoggingContext
 from synapse.types import StreamToken
-from synapse.visibility import filter_events_for_client
 import synapse.metrics
 
 from collections import namedtuple
@@ -140,6 +139,8 @@ class Notifier(object):
     UNUSED_STREAM_EXPIRY_MS = 10 * 60 * 1000
 
     def __init__(self, hs):
+        self.hs = hs
+
         self.user_to_user_stream = {}
         self.room_to_user_streams = {}
         self.appservice_to_user_streams = {}
@@ -149,8 +150,10 @@ class Notifier(object):
         self.pending_new_room_events = []
 
         self.clock = hs.get_clock()
-        self.appservice_handler = hs.get_application_service_handler()
-        self.state_handler = hs.get_state_handler()
+
+        hs.get_distributor().observe(
+            "user_joined_room", self._user_joined_room
+        )
 
         self.clock.looping_call(
             self.remove_expired_streams, self.UNUSED_STREAM_EXPIRY_MS
@@ -228,7 +231,9 @@ class Notifier(object):
     def _on_new_room_event(self, event, room_stream_id, extra_users=[]):
         """Notify any user streams that are interested in this room event"""
         # poke any interested application service.
-        self.appservice_handler.notify_interested_services(event)
+        self.hs.get_handlers().appservice_handler.notify_interested_services(
+            event
+        )
 
         app_streams = set()
 
@@ -244,9 +249,6 @@ class Notifier(object):
                 )
                 app_streams |= app_user_streams
 
-        if event.type == EventTypes.Member and event.membership == Membership.JOIN:
-            self._user_joined_room(event.state_key, event.room_id)
-
         self.on_new_event(
             "room_key", room_stream_id,
             users=extra_users,
@@ -396,8 +398,8 @@ class Notifier(object):
                 )
 
                 if name == "room":
-                    new_events = yield filter_events_for_client(
-                        self.store,
+                    room_member_handler = self.hs.get_handlers().room_member_handler
+                    new_events = yield room_member_handler._filter_events_for_client(
                         user.to_string(),
                         new_events,
                         is_peeking=is_peeking,
@@ -446,7 +448,7 @@ class Notifier(object):
 
     @defer.inlineCallbacks
     def _is_world_readable(self, room_id):
-        state = yield self.state_handler.get_current_state(
+        state = yield self.hs.get_state_handler().get_current_state(
             room_id,
             EventTypes.RoomHistoryVisibility
         )
@@ -482,8 +484,9 @@ class Notifier(object):
                 user_stream.appservice, set()
             ).add(user_stream)
 
-    def _user_joined_room(self, user_id, room_id):
-        new_user_stream = self.user_to_user_stream.get(user_id)
+    def _user_joined_room(self, user, room_id):
+        user = str(user)
+        new_user_stream = self.user_to_user_stream.get(user)
         if new_user_stream is not None:
             room_streams = self.room_to_user_streams.setdefault(room_id, set())
             room_streams.add(new_user_stream)

synapse/push/action_generator.py

@@ -37,14 +37,14 @@ class ActionGenerator:
         # tag (ie. we just need all the users).
 
     @defer.inlineCallbacks
-    def handle_push_actions_for_event(self, event, context):
+    def handle_push_actions_for_event(self, event, context, handler):
         with Measure(self.clock, "handle_push_actions_for_event"):
             bulk_evaluator = yield evaluator_for_event(
-                event, self.hs, self.store, context.current_state
+                event, self.hs, self.store
             )
 
             actions_by_user = yield bulk_evaluator.action_for_event_by_user(
-                event, context.current_state
+                event, handler, context.current_state
             )
 
             context.push_actions = [

synapse/push/bulk_push_rule_evaluator.py

@@ -14,56 +14,83 @@
 # limitations under the License.
 
 import logging
+import ujson as json
 
 from twisted.internet import defer
 
+from .baserules import list_with_base_rules
 from .push_rule_evaluator import PushRuleEvaluatorForEvent
 
-from synapse.api.constants import EventTypes, Membership
-from synapse.visibility import filter_events_for_clients
+from synapse.api.constants import EventTypes
 
 
 logger = logging.getLogger(__name__)
 
 
+def decode_rule_json(rule):
+    rule['conditions'] = json.loads(rule['conditions'])
+    rule['actions'] = json.loads(rule['actions'])
+    return rule
+
+
 @defer.inlineCallbacks
 def _get_rules(room_id, user_ids, store):
     rules_by_user = yield store.bulk_get_push_rules(user_ids)
+    rules_enabled_by_user = yield store.bulk_get_push_rules_enabled(user_ids)
 
-    rules_by_user = {k: v for k, v in rules_by_user.items() if v is not None}
+    rules_by_user = {
+        uid: list_with_base_rules([
+            decode_rule_json(rule_list)
+            for rule_list in rules_by_user.get(uid, [])
+        ])
+        for uid in user_ids
+    }
+
+    # We apply the rules-enabled map here: bulk_get_push_rules doesn't
+    # fetch disabled rules, but this won't account for any server default
+    # rules the user has disabled, so we need to do this too.
+    for uid in user_ids:
+        if uid not in rules_enabled_by_user:
+            continue
+
+        user_enabled_map = rules_enabled_by_user[uid]
+
+        for i, rule in enumerate(rules_by_user[uid]):
+            rule_id = rule['rule_id']
+
+            if rule_id in user_enabled_map:
+                if rule.get('enabled', True) != bool(user_enabled_map[rule_id]):
+                    # Rules are cached across users.
+                    rule = dict(rule)
+                    rule['enabled'] = bool(user_enabled_map[rule_id])
+                    rules_by_user[uid][i] = rule
 
     defer.returnValue(rules_by_user)
 
 
 @defer.inlineCallbacks
-def evaluator_for_event(event, hs, store, current_state):
+def evaluator_for_event(event, hs, store):
     room_id = event.room_id
+
+    # users in the room who have pushers need to get push rules run because
+    # that's how their pushers work
+    users_with_pushers = yield store.get_users_with_pushers_in_room(room_id)
+
     # We also will want to generate notifs for other people in the room so
     # their unread countss are correct in the event stream, but to avoid
     # generating them for bot / AS users etc, we only do so for people who've
     # sent a read receipt into the room.
 
-    local_users_in_room = set(
-        e.state_key for e in current_state.values()
-        if e.type == EventTypes.Member and e.membership == Membership.JOIN
-        and hs.is_mine_id(e.state_key)
-    )
+    all_in_room = yield store.get_users_in_room(room_id)
+    all_in_room = set(all_in_room)
 
-    # users in the room who have pushers need to get push rules run because
-    # that's how their pushers work
-    if_users_with_pushers = yield store.get_if_users_have_pushers(
-        local_users_in_room
-    )
-    user_ids = set(
-        uid for uid, have_pusher in if_users_with_pushers.items() if have_pusher
-    )
-
-    users_with_receipts = yield store.get_users_with_read_receipts_in_room(room_id)
+    receipts = yield store.get_receipts_for_room(room_id, "m.read")
 
     # any users with pushers must be ours: they have pushers
-    for uid in users_with_receipts:
-        if uid in local_users_in_room:
-            user_ids.add(uid)
+    user_ids = set(users_with_pushers)
+    for r in receipts:
+        if hs.is_mine_id(r['user_id']) and r['user_id'] in all_in_room:
+            user_ids.add(r['user_id'])
 
     # if this event is an invite event, we may need to run rules for the user
     # who's been invited, otherwise they won't get told they've been invited
@@ -74,6 +101,8 @@ def evaluator_for_event(event, hs, store, current_state):
             if has_pusher:
                 user_ids.add(invited_user)
 
+    user_ids = list(user_ids)
+
     rules_by_user = yield _get_rules(room_id, user_ids, store)
 
     defer.returnValue(BulkPushRuleEvaluator(
@@ -97,7 +126,7 @@ class BulkPushRuleEvaluator:
         self.store = store
 
     @defer.inlineCallbacks
-    def action_for_event_by_user(self, event, current_state):
+    def action_for_event_by_user(self, event, handler, current_state):
         actions_by_user = {}
 
         # None of these users can be peeking since this list of users comes
@@ -107,14 +136,11 @@ class BulkPushRuleEvaluator:
             (u, False) for u in self.rules_by_user.keys()
         ]
 
-        filtered_by_user = yield filter_events_for_clients(
-            self.store, user_tuples, [event], {event.event_id: current_state}
+        filtered_by_user = yield handler.filter_events_for_clients(
+            user_tuples, [event], {event.event_id: current_state}
         )
 
-        room_members = set(
-            e.state_key for e in current_state.values()
-            if e.type == EventTypes.Member and e.membership == Membership.JOIN
-        )
+        room_members = yield self.store.get_users_in_room(self.room_id)
 
         evaluator = PushRuleEvaluatorForEvent(event, len(room_members))
 

synapse/push/clientformat.py

@@ -13,19 +13,29 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from synapse.push.baserules import list_with_base_rules
+
 from synapse.push.rulekinds import (
     PRIORITY_CLASS_MAP, PRIORITY_CLASS_INVERSE_MAP
 )
 
 import copy
+import simplejson as json
 
 
-def format_push_rules_for_user(user, ruleslist):
+def format_push_rules_for_user(user, rawrules, enabled_map):
     """Converts a list of rawrules and a enabled map into nested dictionaries
     to match the Matrix client-server format for push rules"""
 
+    ruleslist = []
+    for rawrule in rawrules:
+        rule = dict(rawrule)
+        rule["conditions"] = json.loads(rawrule["conditions"])
+        rule["actions"] = json.loads(rawrule["actions"])
+        ruleslist.append(rule)
+
     # We're going to be mutating this a lot, so do a deep copy
-    ruleslist = copy.deepcopy(ruleslist)
+    ruleslist = copy.deepcopy(list_with_base_rules(ruleslist))
 
     rules = {'global': {}, 'device': {}}
 
@@ -50,7 +60,9 @@ def format_push_rules_for_user(user, ruleslist):
 
         template_rule = _rule_to_template(r)
         if template_rule:
-            if 'enabled' in r:
+            if r['rule_id'] in enabled_map:
+                template_rule['enabled'] = enabled_map[r['rule_id']]
+            elif 'enabled' in r:
                 template_rule['enabled'] = r['enabled']
             else:
                 template_rule['enabled'] = True

synapse/push/emailpusher.py

@@ -1,283 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from twisted.internet import defer, reactor
-
-import logging
-
-from synapse.util.metrics import Measure
-from synapse.util.logcontext import LoggingContext
-
-from mailer import Mailer
-
-logger = logging.getLogger(__name__)
-
-# The amount of time we always wait before ever emailing about a notification
-# (to give the user a chance to respond to other push or notice the window)
-DELAY_BEFORE_MAIL_MS = 10 * 60 * 1000
-
-# THROTTLE is the minimum time between mail notifications sent for a given room.
-# Each room maintains its own throttle counter, but each new mail notification
-# sends the pending notifications for all rooms.
-THROTTLE_START_MS = 10 * 60 * 1000
-THROTTLE_MAX_MS = 24 * 60 * 60 * 1000  # 24h
-# THROTTLE_MULTIPLIER = 6              # 10 mins, 1 hour, 6 hours, 24 hours
-THROTTLE_MULTIPLIER = 144              # 10 mins, 24 hours - i.e. jump straight to 1 day
-
-# If no event triggers a notification for this long after the previous,
-# the throttle is released.
-# 12 hours - a gap of 12 hours in conversation is surely enough to merit a new
-# notification when things get going again...
-THROTTLE_RESET_AFTER_MS = (12 * 60 * 60 * 1000)
-
-# does each email include all unread notifs, or just the ones which have happened
-# since the last mail?
-# XXX: this is currently broken as it includes ones from parted rooms(!)
-INCLUDE_ALL_UNREAD_NOTIFS = False
-
-
-class EmailPusher(object):
-    """
-    A pusher that sends email notifications about events (approximately)
-    when they happen.
-    This shares quite a bit of code with httpusher: it would be good to
-    factor out the common parts
-    """
-    def __init__(self, hs, pusherdict):
-        self.hs = hs
-        self.store = self.hs.get_datastore()
-        self.clock = self.hs.get_clock()
-        self.pusher_id = pusherdict['id']
-        self.user_id = pusherdict['user_name']
-        self.app_id = pusherdict['app_id']
-        self.email = pusherdict['pushkey']
-        self.last_stream_ordering = pusherdict['last_stream_ordering']
-        self.timed_call = None
-        self.throttle_params = None
-
-        # See httppusher
-        self.max_stream_ordering = None
-
-        self.processing = False
-
-        if self.hs.config.email_enable_notifs:
-            if 'data' in pusherdict and 'brand' in pusherdict['data']:
-                app_name = pusherdict['data']['brand']
-            else:
-                app_name = self.hs.config.email_app_name
-
-            self.mailer = Mailer(self.hs, app_name)
-        else:
-            self.mailer = None
-
-    @defer.inlineCallbacks
-    def on_started(self):
-        if self.mailer is not None:
-            self.throttle_params = yield self.store.get_throttle_params_by_room(
-                self.pusher_id
-            )
-            yield self._process()
-
-    def on_stop(self):
-        if self.timed_call:
-            self.timed_call.cancel()
-
-    @defer.inlineCallbacks
-    def on_new_notifications(self, min_stream_ordering, max_stream_ordering):
-        self.max_stream_ordering = max(max_stream_ordering, self.max_stream_ordering)
-        yield self._process()
-
-    def on_new_receipts(self, min_stream_id, max_stream_id):
-        # We could wake up and cancel the timer but there tend to be quite a
-        # lot of read receipts so it's probably less work to just let the
-        # timer fire
-        return defer.succeed(None)
-
-    @defer.inlineCallbacks
-    def on_timer(self):
-        self.timed_call = None
-        yield self._process()
-
-    @defer.inlineCallbacks
-    def _process(self):
-        if self.processing:
-            return
-
-        with LoggingContext("emailpush._process"):
-            with Measure(self.clock, "emailpush._process"):
-                try:
-                    self.processing = True
-                    # if the max ordering changes while we're running _unsafe_process,
-                    # call it again, and so on until we've caught up.
-                    while True:
-                        starting_max_ordering = self.max_stream_ordering
-                        try:
-                            yield self._unsafe_process()
-                        except:
-                            logger.exception("Exception processing notifs")
-                        if self.max_stream_ordering == starting_max_ordering:
-                            break
-                finally:
-                    self.processing = False
-
-    @defer.inlineCallbacks
-    def _unsafe_process(self):
-        """
-        Main logic of the push loop without the wrapper function that sets
-        up logging, measures and guards against multiple instances of it
-        being run.
-        """
-        start = 0 if INCLUDE_ALL_UNREAD_NOTIFS else self.last_stream_ordering
-        unprocessed = yield self.store.get_unread_push_actions_for_user_in_range(
-            self.user_id, start, self.max_stream_ordering
-        )
-
-        soonest_due_at = None
-
-        for push_action in unprocessed:
-            received_at = push_action['received_ts']
-            if received_at is None:
-                received_at = 0
-            notif_ready_at = received_at + DELAY_BEFORE_MAIL_MS
-
-            room_ready_at = self.room_ready_to_notify_at(
-                push_action['room_id']
-            )
-
-            should_notify_at = max(notif_ready_at, room_ready_at)
-
-            if should_notify_at < self.clock.time_msec():
-                # one of our notifications is ready for sending, so we send
-                # *one* email updating the user on their notifications,
-                # we then consider all previously outstanding notifications
-                # to be delivered.
-
-                reason = {
-                    'room_id': push_action['room_id'],
-                    'now': self.clock.time_msec(),
-                    'received_at': received_at,
-                    'delay_before_mail_ms': DELAY_BEFORE_MAIL_MS,
-                    'last_sent_ts': self.get_room_last_sent_ts(push_action['room_id']),
-                    'throttle_ms': self.get_room_throttle_ms(push_action['room_id']),
-                }
-
-                yield self.send_notification(unprocessed, reason)
-
-                yield self.save_last_stream_ordering_and_success(max([
-                    ea['stream_ordering'] for ea in unprocessed
-                ]))
-
-                # we update the throttle on all the possible unprocessed push actions
-                for ea in unprocessed:
-                    yield self.sent_notif_update_throttle(
-                        ea['room_id'], ea
-                    )
-                break
-            else:
-                if soonest_due_at is None or should_notify_at < soonest_due_at:
-                    soonest_due_at = should_notify_at
-
-                if self.timed_call is not None:
-                    self.timed_call.cancel()
-                    self.timed_call = None
-
-        if soonest_due_at is not None:
-            self.timed_call = reactor.callLater(
-                self.seconds_until(soonest_due_at), self.on_timer
-            )
-
-    @defer.inlineCallbacks
-    def save_last_stream_ordering_and_success(self, last_stream_ordering):
-        self.last_stream_ordering = last_stream_ordering
-        yield self.store.update_pusher_last_stream_ordering_and_success(
-            self.app_id, self.email, self.user_id,
-            last_stream_ordering, self.clock.time_msec()
-        )
-
-    def seconds_until(self, ts_msec):
-        return (ts_msec - self.clock.time_msec()) / 1000
-
-    def get_room_throttle_ms(self, room_id):
-        if room_id in self.throttle_params:
-            return self.throttle_params[room_id]["throttle_ms"]
-        else:
-            return 0
-
-    def get_room_last_sent_ts(self, room_id):
-        if room_id in self.throttle_params:
-            return self.throttle_params[room_id]["last_sent_ts"]
-        else:
-            return 0
-
-    def room_ready_to_notify_at(self, room_id):
-        """
-        Determines whether throttling should prevent us from sending an email
-        for the given room
-        Returns: The timestamp when we are next allowed to send an email notif
-        for this room
-        """
-        last_sent_ts = self.get_room_last_sent_ts(room_id)
-        throttle_ms = self.get_room_throttle_ms(room_id)
-
-        may_send_at = last_sent_ts + throttle_ms
-        return may_send_at
-
-    @defer.inlineCallbacks
-    def sent_notif_update_throttle(self, room_id, notified_push_action):
-        # We have sent a notification, so update the throttle accordingly.
-        # If the event that triggered the notif happened more than
-        # THROTTLE_RESET_AFTER_MS after the previous one that triggered a
-        # notif, we release the throttle. Otherwise, the throttle is increased.
-        time_of_previous_notifs = yield self.store.get_time_of_last_push_action_before(
-            notified_push_action['stream_ordering']
-        )
-
-        time_of_this_notifs = notified_push_action['received_ts']
-
-        if time_of_previous_notifs is not None and time_of_this_notifs is not None:
-            gap = time_of_this_notifs - time_of_previous_notifs
-        else:
-            # if we don't know the arrival time of one of the notifs (it was not
-            # stored prior to email notification code) then assume a gap of
-            # zero which will just not reset the throttle
-            gap = 0
-
-        current_throttle_ms = self.get_room_throttle_ms(room_id)
-
-        if gap > THROTTLE_RESET_AFTER_MS:
-            new_throttle_ms = THROTTLE_START_MS
-        else:
-            if current_throttle_ms == 0:
-                new_throttle_ms = THROTTLE_START_MS
-            else:
-                new_throttle_ms = min(
-                    current_throttle_ms * THROTTLE_MULTIPLIER,
-                    THROTTLE_MAX_MS
-                )
-        self.throttle_params[room_id] = {
-            "last_sent_ts": self.clock.time_msec(),
-            "throttle_ms": new_throttle_ms
-        }
-        yield self.store.set_throttle_params(
-            self.pusher_id, room_id, self.throttle_params[room_id]
-        )
-
-    @defer.inlineCallbacks
-    def send_notification(self, push_actions, reason):
-        logger.info("Sending notif email for user %r", self.user_id)
-
-        yield self.mailer.send_notification_mail(
-            self.app_id, self.user_id, self.email, push_actions, reason
-        )

synapse/push/httppusher.py

@@ -38,7 +38,6 @@ class HttpPusher(object):
         self.hs = hs
         self.store = self.hs.get_datastore()
         self.clock = self.hs.get_clock()
-        self.state_handler = self.hs.get_state_handler()
         self.user_id = pusherdict['user_name']
         self.app_id = pusherdict['app_id']
         self.app_display_name = pusherdict['app_display_name']
@@ -238,9 +237,7 @@ class HttpPusher(object):
 
     @defer.inlineCallbacks
     def _build_notification_dict(self, event, tweaks, badge):
-        ctx = yield push_tools.get_context_for_event(
-            self.state_handler, event, self.user_id
-        )
+        ctx = yield push_tools.get_context_for_event(self.hs.get_datastore(), event)
 
         d = {
             'notification': {
@@ -272,8 +269,8 @@ class HttpPusher(object):
         if 'content' in event:
             d['notification']['content'] = event.content
 
-        # We no longer send aliases separately, instead, we send the human
-        # readable name of the room, which may be an alias.
+        if len(ctx['aliases']):
+            d['notification']['room_alias'] = ctx['aliases'][0]
         if 'sender_display_name' in ctx and len(ctx['sender_display_name']) > 0:
             d['notification']['sender_display_name'] = ctx['sender_display_name']
         if 'name' in ctx and len(ctx['name']) > 0:

synapse/push/mailer.py

@@ -1,515 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from twisted.internet import defer
-from twisted.mail.smtp import sendmail
-
-import email.utils
-import email.mime.multipart
-from email.mime.text import MIMEText
-from email.mime.multipart import MIMEMultipart
-
-from synapse.util.async import concurrently_execute
-from synapse.util.presentable_names import (
-    calculate_room_name, name_from_member_event, descriptor_from_member_events
-)
-from synapse.types import UserID
-from synapse.api.errors import StoreError
-from synapse.api.constants import EventTypes
-from synapse.visibility import filter_events_for_client
-
-import jinja2
-import bleach
-
-import time
-import urllib
-
-import logging
-logger = logging.getLogger(__name__)
-
-
-MESSAGE_FROM_PERSON_IN_ROOM = "You have a message on %(app)s from %(person)s " \
-                              "in the %(room)s room..."
-MESSAGE_FROM_PERSON = "You have a message on %(app)s from %(person)s..."
-MESSAGES_FROM_PERSON = "You have messages on %(app)s from %(person)s..."
-MESSAGES_IN_ROOM = "You have messages on %(app)s in the %(room)s room..."
-MESSAGES_IN_ROOM_AND_OTHERS = \
-    "You have messages on %(app)s in the %(room)s room and others..."
-MESSAGES_FROM_PERSON_AND_OTHERS = \
-    "You have messages on %(app)s from %(person)s and others..."
-INVITE_FROM_PERSON_TO_ROOM = "%(person)s has invited you to join the " \
-                             "%(room)s room on %(app)s..."
-INVITE_FROM_PERSON = "%(person)s has invited you to chat on %(app)s..."
-
-CONTEXT_BEFORE = 1
-CONTEXT_AFTER = 1
-
-# From https://github.com/matrix-org/matrix-react-sdk/blob/master/src/HtmlUtils.js
-ALLOWED_TAGS = [
-    'font',  # custom to matrix for IRC-style font coloring
-    'del',  # for markdown
-    # deliberately no h1/h2 to stop people shouting.
-    'h3', 'h4', 'h5', 'h6', 'blockquote', 'p', 'a', 'ul', 'ol',
-    'nl', 'li', 'b', 'i', 'u', 'strong', 'em', 'strike', 'code', 'hr', 'br', 'div',
-    'table', 'thead', 'caption', 'tbody', 'tr', 'th', 'td', 'pre'
-]
-ALLOWED_ATTRS = {
-    # custom ones first:
-    "font": ["color"],  # custom to matrix
-    "a": ["href", "name", "target"],  # remote target: custom to matrix
-    # We don't currently allow img itself by default, but this
-    # would make sense if we did
-    "img": ["src"],
-}
-# When bleach release a version with this option, we can specify schemes
-# ALLOWED_SCHEMES = ["http", "https", "ftp", "mailto"]
-
-
-class Mailer(object):
-    def __init__(self, hs, app_name):
-        self.hs = hs
-        self.store = self.hs.get_datastore()
-        self.auth_handler = self.hs.get_auth_handler()
-        self.state_handler = self.hs.get_state_handler()
-        loader = jinja2.FileSystemLoader(self.hs.config.email_template_dir)
-        self.app_name = app_name
-        logger.info("Created Mailer for app_name %s" % app_name)
-        env = jinja2.Environment(loader=loader)
-        env.filters["format_ts"] = format_ts_filter
-        env.filters["mxc_to_http"] = self.mxc_to_http_filter
-        self.notif_template_html = env.get_template(
-            self.hs.config.email_notif_template_html
-        )
-        self.notif_template_text = env.get_template(
-            self.hs.config.email_notif_template_text
-        )
-
-    @defer.inlineCallbacks
-    def send_notification_mail(self, app_id, user_id, email_address,
-                               push_actions, reason):
-        try:
-            from_string = self.hs.config.email_notif_from % {
-                "app": self.app_name
-            }
-        except TypeError:
-            from_string = self.hs.config.email_notif_from
-
-        raw_from = email.utils.parseaddr(from_string)[1]
-        raw_to = email.utils.parseaddr(email_address)[1]
-
-        if raw_to == '':
-            raise RuntimeError("Invalid 'to' address")
-
-        rooms_in_order = deduped_ordered_list(
-            [pa['room_id'] for pa in push_actions]
-        )
-
-        notif_events = yield self.store.get_events(
-            [pa['event_id'] for pa in push_actions]
-        )
-
-        notifs_by_room = {}
-        for pa in push_actions:
-            notifs_by_room.setdefault(pa["room_id"], []).append(pa)
-
-        # collect the current state for all the rooms in which we have
-        # notifications
-        state_by_room = {}
-
-        try:
-            user_display_name = yield self.store.get_profile_displayname(
-                UserID.from_string(user_id).localpart
-            )
-            if user_display_name is None:
-                user_display_name = user_id
-        except StoreError:
-            user_display_name = user_id
-
-        @defer.inlineCallbacks
-        def _fetch_room_state(room_id):
-            room_state = yield self.state_handler.get_current_state(room_id)
-            state_by_room[room_id] = room_state
-
-        # Run at most 3 of these at once: sync does 10 at a time but email
-        # notifs are much less realtime than sync so we can afford to wait a bit.
-        yield concurrently_execute(_fetch_room_state, rooms_in_order, 3)
-
-        # actually sort our so-called rooms_in_order list, most recent room first
-        rooms_in_order.sort(
-            key=lambda r: -(notifs_by_room[r][-1]['received_ts'] or 0)
-        )
-
-        rooms = []
-
-        for r in rooms_in_order:
-            roomvars = yield self.get_room_vars(
-                r, user_id, notifs_by_room[r], notif_events, state_by_room[r]
-            )
-            rooms.append(roomvars)
-
-        reason['room_name'] = calculate_room_name(
-            state_by_room[reason['room_id']], user_id, fallback_to_members=True
-        )
-
-        summary_text = self.make_summary_text(
-            notifs_by_room, state_by_room, notif_events, user_id, reason
-        )
-
-        template_vars = {
-            "user_display_name": user_display_name,
-            "unsubscribe_link": self.make_unsubscribe_link(
-                user_id, app_id, email_address
-            ),
-            "summary_text": summary_text,
-            "app_name": self.app_name,
-            "rooms": rooms,
-            "reason": reason,
-        }
-
-        html_text = self.notif_template_html.render(**template_vars)
-        html_part = MIMEText(html_text, "html", "utf8")
-
-        plain_text = self.notif_template_text.render(**template_vars)
-        text_part = MIMEText(plain_text, "plain", "utf8")
-
-        multipart_msg = MIMEMultipart('alternative')
-        multipart_msg['Subject'] = "[%s] %s" % (self.app_name, summary_text)
-        multipart_msg['From'] = from_string
-        multipart_msg['To'] = email_address
-        multipart_msg['Date'] = email.utils.formatdate()
-        multipart_msg['Message-ID'] = email.utils.make_msgid()
-        multipart_msg.attach(text_part)
-        multipart_msg.attach(html_part)
-
-        logger.info("Sending email push notification to %s" % email_address)
-        # logger.debug(html_text)
-
-        yield sendmail(
-            self.hs.config.email_smtp_host,
-            raw_from, raw_to, multipart_msg.as_string(),
-            port=self.hs.config.email_smtp_port
-        )
-
-    @defer.inlineCallbacks
-    def get_room_vars(self, room_id, user_id, notifs, notif_events, room_state):
-        my_member_event = room_state[("m.room.member", user_id)]
-        is_invite = my_member_event.content["membership"] == "invite"
-
-        room_vars = {
-            "title": calculate_room_name(room_state, user_id),
-            "hash": string_ordinal_total(room_id),  # See sender avatar hash
-            "notifs": [],
-            "invite": is_invite,
-            "link": self.make_room_link(room_id),
-        }
-
-        if not is_invite:
-            for n in notifs:
-                notifvars = yield self.get_notif_vars(
-                    n, user_id, notif_events[n['event_id']], room_state
-                )
-
-                # merge overlapping notifs together.
-                # relies on the notifs being in chronological order.
-                merge = False
-                if room_vars['notifs'] and 'messages' in room_vars['notifs'][-1]:
-                    prev_messages = room_vars['notifs'][-1]['messages']
-                    for message in notifvars['messages']:
-                        pm = filter(lambda pm: pm['id'] == message['id'], prev_messages)
-                        if pm:
-                            if not message["is_historical"]:
-                                pm[0]["is_historical"] = False
-                            merge = True
-                        elif merge:
-                            # we're merging, so append any remaining messages
-                            # in this notif to the previous one
-                            prev_messages.append(message)
-
-                if not merge:
-                    room_vars['notifs'].append(notifvars)
-
-        defer.returnValue(room_vars)
-
-    @defer.inlineCallbacks
-    def get_notif_vars(self, notif, user_id, notif_event, room_state):
-        results = yield self.store.get_events_around(
-            notif['room_id'], notif['event_id'],
-            before_limit=CONTEXT_BEFORE, after_limit=CONTEXT_AFTER
-        )
-
-        ret = {
-            "link": self.make_notif_link(notif),
-            "ts": notif['received_ts'],
-            "messages": [],
-        }
-
-        the_events = yield filter_events_for_client(
-            self.store, user_id, results["events_before"]
-        )
-        the_events.append(notif_event)
-
-        for event in the_events:
-            messagevars = self.get_message_vars(notif, event, room_state)
-            if messagevars is not None:
-                ret['messages'].append(messagevars)
-
-        defer.returnValue(ret)
-
-    def get_message_vars(self, notif, event, room_state):
-        if event.type != EventTypes.Message:
-            return None
-
-        sender_state_event = room_state[("m.room.member", event.sender)]
-        sender_name = name_from_member_event(sender_state_event)
-        sender_avatar_url = sender_state_event.content.get("avatar_url")
-
-        # 'hash' for deterministically picking default images: use
-        # sender_hash % the number of default images to choose from
-        sender_hash = string_ordinal_total(event.sender)
-
-        msgtype = event.content.get("msgtype")
-
-        ret = {
-            "msgtype": msgtype,
-            "is_historical": event.event_id != notif['event_id'],
-            "id": event.event_id,
-            "ts": event.origin_server_ts,
-            "sender_name": sender_name,
-            "sender_avatar_url": sender_avatar_url,
-            "sender_hash": sender_hash,
-        }
-
-        if msgtype == "m.text":
-            self.add_text_message_vars(ret, event)
-        elif msgtype == "m.image":
-            self.add_image_message_vars(ret, event)
-
-        if "body" in event.content:
-            ret["body_text_plain"] = event.content["body"]
-
-        return ret
-
-    def add_text_message_vars(self, messagevars, event):
-        msgformat = event.content.get("format")
-
-        messagevars["format"] = msgformat
-
-        formatted_body = event.content.get("formatted_body")
-        body = event.content.get("body")
-
-        if msgformat == "org.matrix.custom.html" and formatted_body:
-            messagevars["body_text_html"] = safe_markup(formatted_body)
-        elif body:
-            messagevars["body_text_html"] = safe_text(body)
-
-        return messagevars
-
-    def add_image_message_vars(self, messagevars, event):
-        messagevars["image_url"] = event.content["url"]
-
-        return messagevars
-
-    def make_summary_text(self, notifs_by_room, state_by_room,
-                          notif_events, user_id, reason):
-        if len(notifs_by_room) == 1:
-            # Only one room has new stuff
-            room_id = notifs_by_room.keys()[0]
-
-            # If the room has some kind of name, use it, but we don't
-            # want the generated-from-names one here otherwise we'll
-            # end up with, "new message from Bob in the Bob room"
-            room_name = calculate_room_name(
-                state_by_room[room_id], user_id, fallback_to_members=False
-            )
-
-            my_member_event = state_by_room[room_id][("m.room.member", user_id)]
-            if my_member_event.content["membership"] == "invite":
-                inviter_member_event = state_by_room[room_id][
-                    ("m.room.member", my_member_event.sender)
-                ]
-                inviter_name = name_from_member_event(inviter_member_event)
-
-                if room_name is None:
-                    return INVITE_FROM_PERSON % {
-                        "person": inviter_name,
-                        "app": self.app_name
-                    }
-                else:
-                    return INVITE_FROM_PERSON_TO_ROOM % {
-                        "person": inviter_name,
-                        "room": room_name,
-                        "app": self.app_name,
-                    }
-
-            sender_name = None
-            if len(notifs_by_room[room_id]) == 1:
-                # There is just the one notification, so give some detail
-                event = notif_events[notifs_by_room[room_id][0]["event_id"]]
-                if ("m.room.member", event.sender) in state_by_room[room_id]:
-                    state_event = state_by_room[room_id][("m.room.member", event.sender)]
-                    sender_name = name_from_member_event(state_event)
-
-                if sender_name is not None and room_name is not None:
-                    return MESSAGE_FROM_PERSON_IN_ROOM % {
-                        "person": sender_name,
-                        "room": room_name,
-                        "app": self.app_name,
-                    }
-                elif sender_name is not None:
-                    return MESSAGE_FROM_PERSON % {
-                        "person": sender_name,
-                        "app": self.app_name,
-                    }
-            else:
-                # There's more than one notification for this room, so just
-                # say there are several
-                if room_name is not None:
-                    return MESSAGES_IN_ROOM % {
-                        "room": room_name,
-                        "app": self.app_name,
-                    }
-                else:
-                    # If the room doesn't have a name, say who the messages
-                    # are from explicitly to avoid, "messages in the Bob room"
-                    sender_ids = list(set([
-                        notif_events[n['event_id']].sender
-                        for n in notifs_by_room[room_id]
-                    ]))
-
-                    return MESSAGES_FROM_PERSON % {
-                        "person": descriptor_from_member_events([
-                            state_by_room[room_id][("m.room.member", s)]
-                            for s in sender_ids
-                        ]),
-                        "app": self.app_name,
-                    }
-        else:
-            # Stuff's happened in multiple different rooms
-
-            # ...but we still refer to the 'reason' room which triggered the mail
-            if reason['room_name'] is not None:
-                return MESSAGES_IN_ROOM_AND_OTHERS % {
-                    "room": reason['room_name'],
-                    "app": self.app_name,
-                }
-            else:
-                # If the reason room doesn't have a name, say who the messages
-                # are from explicitly to avoid, "messages in the Bob room"
-                sender_ids = list(set([
-                    notif_events[n['event_id']].sender
-                    for n in notifs_by_room[reason['room_id']]
-                ]))
-
-                return MESSAGES_FROM_PERSON_AND_OTHERS % {
-                    "person": descriptor_from_member_events([
-                        state_by_room[reason['room_id']][("m.room.member", s)]
-                        for s in sender_ids
-                    ]),
-                    "app": self.app_name,
-                }
-
-    def make_room_link(self, room_id):
-        # need /beta for Universal Links to work on iOS
-        if self.app_name == "Vector":
-            return "https://vector.im/beta/#/room/%s" % (room_id,)
-        else:
-            return "https://matrix.to/#/%s" % (room_id,)
-
-    def make_notif_link(self, notif):
-        # need /beta for Universal Links to work on iOS
-        if self.app_name == "Vector":
-            return "https://vector.im/beta/#/room/%s/%s" % (
-                notif['room_id'], notif['event_id']
-            )
-        else:
-            return "https://matrix.to/#/%s/%s" % (
-                notif['room_id'], notif['event_id']
-            )
-
-    def make_unsubscribe_link(self, user_id, app_id, email_address):
-        params = {
-            "access_token": self.auth_handler.generate_delete_pusher_token(user_id),
-            "app_id": app_id,
-            "pushkey": email_address,
-        }
-
-        # XXX: make r0 once API is stable
-        return "%s_matrix/client/unstable/pushers/remove?%s" % (
-            self.hs.config.public_baseurl,
-            urllib.urlencode(params),
-        )
-
-    def mxc_to_http_filter(self, value, width, height, resize_method="crop"):
-        if value[0:6] != "mxc://":
-            return ""
-
-        serverAndMediaId = value[6:]
-        fragment = None
-        if '#' in serverAndMediaId:
-            (serverAndMediaId, fragment) = serverAndMediaId.split('#', 1)
-            fragment = "#" + fragment
-
-        params = {
-            "width": width,
-            "height": height,
-            "method": resize_method,
-        }
-        return "%s_matrix/media/v1/thumbnail/%s?%s%s" % (
-            self.hs.config.public_baseurl,
-            serverAndMediaId,
-            urllib.urlencode(params),
-            fragment or "",
-        )
-
-
-def safe_markup(raw_html):
-    return jinja2.Markup(bleach.linkify(bleach.clean(
-        raw_html, tags=ALLOWED_TAGS, attributes=ALLOWED_ATTRS,
-        # bleach master has this, but it isn't released yet
-        # protocols=ALLOWED_SCHEMES,
-        strip=True
-    )))
-
-
-def safe_text(raw_text):
-    """
-    Process text: treat it as HTML but escape any tags (ie. just escape the
-    HTML) then linkify it.
-    """
-    return jinja2.Markup(bleach.linkify(bleach.clean(
-        raw_text, tags=[], attributes={},
-        strip=False
-    )))
-
-
-def deduped_ordered_list(l):
-    seen = set()
-    ret = []
-    for item in l:
-        if item not in seen:
-            seen.add(item)
-            ret.append(item)
-    return ret
-
-
-def string_ordinal_total(s):
-    tot = 0
-    for c in s:
-        tot += ord(c)
-    return tot
-
-
-def format_ts_filter(value, format):
-    return time.strftime(format, time.localtime(value / 1000))

synapse/push/push_tools.py

@@ -14,9 +14,6 @@
 # limitations under the License.
 
 from twisted.internet import defer
-from synapse.util.presentable_names import (
-    calculate_room_name, name_from_member_event
-)
 
 
 @defer.inlineCallbacks
@@ -41,28 +38,29 @@ def get_badge_count(store, user_id):
                     r.room_id, user_id, last_unread_event_id
                 )
             )
-            # return one badge count per conversation, as count per
-            # message is so noisy as to be almost useless
-            badge += 1 if notifs["notify_count"] else 0
+            badge += notifs["notify_count"]
     defer.returnValue(badge)
 
 
 @defer.inlineCallbacks
-def get_context_for_event(state_handler, ev, user_id):
-    ctx = {}
-
-    room_state = yield state_handler.get_current_state(ev.room_id)
-
-    # we no longer bother setting room_alias, and make room_name the
-    # human-readable name instead, be that m.room.namer, an alias or
-    # a list of people in the room
-    name = calculate_room_name(
-        room_state, user_id, fallback_to_single_member=False
+def get_context_for_event(store, ev):
+    name_aliases = yield store.get_room_name_and_aliases(
+        ev.room_id
     )
-    if name:
-        ctx['name'] = name
 
-    sender_state_event = room_state[("m.room.member", ev.sender)]
-    ctx['sender_display_name'] = name_from_member_event(sender_state_event)
+    ctx = {'aliases': name_aliases[1]}
+    if name_aliases[0] is not None:
+        ctx['name'] = name_aliases[0]
+
+    their_member_events_for_room = yield store.get_current_state(
+        room_id=ev.room_id,
+        event_type='m.room.member',
+        state_key=ev.user_id
+    )
+    for mev in their_member_events_for_room:
+        if mev.content['membership'] == 'join' and 'displayname' in mev.content:
+            dn = mev.content['displayname']
+            if dn is not None:
+                ctx['sender_display_name'] = dn
 
     defer.returnValue(ctx)

synapse/push/pusher.py

@@ -1,47 +1,10 @@
-# -*- coding: utf-8 -*-
-# Copyright 2014-2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
 from httppusher import HttpPusher
 
-import logging
-logger = logging.getLogger(__name__)
-
-# We try importing this if we can (it will fail if we don't
-# have the optional email dependencies installed). We don't
-# yet have the config to know if we need the email pusher,
-# but importing this after daemonizing seems to fail
-# (even though a simple test of importing from a daemonized
-# process works fine)
-try:
-    from synapse.push.emailpusher import EmailPusher
-except:
-    pass
+PUSHER_TYPES = {
+    'http': HttpPusher
+}
 
 
 def create_pusher(hs, pusherdict):
-    logger.info("trying to create_pusher for %r", pusherdict)
-
-    PUSHER_TYPES = {
-        "http": HttpPusher,
-    }
-
-    logger.info("email enable notifs: %r", hs.config.email_enable_notifs)
-    if hs.config.email_enable_notifs:
-        PUSHER_TYPES["email"] = EmailPusher
-        logger.info("defined email pusher type")
-
     if pusherdict['kind'] in PUSHER_TYPES:
-        logger.info("found pusher")
         return PUSHER_TYPES[pusherdict['kind']](hs, pusherdict)

synapse/push/pusherpool.py

@@ -17,6 +17,7 @@
 from twisted.internet import defer
 
 import pusher
+from synapse.push import PusherConfigException
 from synapse.util.logcontext import preserve_fn
 from synapse.util.async import run_on_reactor
 
@@ -49,7 +50,6 @@ class PusherPool:
         # recreated, added and started: this means we have only one
         # code path adding pushers.
         pusher.create_pusher(self.hs, {
-            "id": None,
             "user_name": user_id,
             "kind": kind,
             "app_id": app_id,
@@ -185,8 +185,8 @@ class PusherPool:
         for pusherdict in pushers:
             try:
                 p = pusher.create_pusher(self.hs, pusherdict)
-            except:
-                logger.exception("Couldn't start a pusher: caught Exception")
+            except PusherConfigException:
+                logger.exception("Couldn't start a pusher: caught PusherConfigException")
                 continue
             if p:
                 appid_pushkey = "%s:%s" % (

synapse/python_dependencies.py

@@ -36,6 +36,7 @@ REQUIREMENTS = {
     "blist": ["blist"],
     "pysaml2>=3.0.0,<4.0.0": ["saml2>=3.0.0,<4.0.0"],
     "pymacaroons-pynacl": ["pymacaroons"],
+    "pyjwt": ["jwt"],
 }
 CONDITIONAL_REQUIREMENTS = {
     "web_client": {
@@ -44,13 +45,6 @@ CONDITIONAL_REQUIREMENTS = {
     "preview_url": {
         "netaddr>=0.7.18": ["netaddr"],
     },
-    "email.enable_notifs": {
-        "Jinja2>=2.8": ["Jinja2>=2.8"],
-        "bleach>=1.4.2": ["bleach>=1.4.2"],
-    },
-    "ldap": {
-        "ldap3>=1.0": ["ldap3>=1.0"],
-    },
 }
 
 

synapse/replication/presence_resource.py

@@ -1,59 +0,0 @@
-# Copyright 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from synapse.http.server import respond_with_json_bytes, request_handler
-from synapse.http.servlet import parse_json_object_from_request
-
-from twisted.web.resource import Resource
-from twisted.web.server import NOT_DONE_YET
-from twisted.internet import defer
-
-
-class PresenceResource(Resource):
-    """
-    HTTP endpoint for marking users as syncing.
-
-    POST /_synapse/replication/presence HTTP/1.1
-    Content-Type: application/json
-
-    {
-        "process_id": "<process_id>",
-        "syncing_users": ["<user_id>"]
-    }
-    """
-
-    def __init__(self, hs):
-        Resource.__init__(self)  # Resource is old-style, so no super()
-
-        self.version_string = hs.version_string
-        self.clock = hs.get_clock()
-        self.presence_handler = hs.get_presence_handler()
-
-    def render_POST(self, request):
-        self._async_render_POST(request)
-        return NOT_DONE_YET
-
-    @request_handler()
-    @defer.inlineCallbacks
-    def _async_render_POST(self, request):
-        content = parse_json_object_from_request(request)
-
-        process_id = content["process_id"]
-        syncing_user_ids = content["syncing_users"]
-
-        yield self.presence_handler.update_external_syncs(
-            process_id, set(syncing_user_ids)
-        )
-
-        respond_with_json_bytes(request, 200, "{}")

synapse/replication/pusher_resource.py

@@ -31,13 +31,12 @@ class PusherResource(Resource):
         self.version_string = hs.version_string
         self.store = hs.get_datastore()
         self.notifier = hs.get_notifier()
-        self.clock = hs.get_clock()
 
     def render_POST(self, request):
         self._async_render_POST(request)
         return NOT_DONE_YET
 
-    @request_handler()
+    @request_handler
     @defer.inlineCallbacks
     def _async_render_POST(self, request):
         content = parse_json_object_from_request(request)

synapse/replication/resource.py

@@ -16,7 +16,6 @@
 from synapse.http.servlet import parse_integer, parse_string
 from synapse.http.server import request_handler, finish_request
 from synapse.replication.pusher_resource import PusherResource
-from synapse.replication.presence_resource import PresenceResource
 
 from twisted.web.resource import Resource
 from twisted.web.server import NOT_DONE_YET
@@ -110,13 +109,11 @@ class ReplicationResource(Resource):
         self.version_string = hs.version_string
         self.store = hs.get_datastore()
         self.sources = hs.get_event_sources()
-        self.presence_handler = hs.get_presence_handler()
-        self.typing_handler = hs.get_typing_handler()
+        self.presence_handler = hs.get_handlers().presence_handler
+        self.typing_handler = hs.get_handlers().typing_notification_handler
         self.notifier = hs.notifier
-        self.clock = hs.get_clock()
 
         self.putChild("remove_pushers", PusherResource(hs))
-        self.putChild("syncing_users", PresenceResource(hs))
 
     def render_GET(self, request):
         self._async_render_GET(request)
@@ -142,7 +139,7 @@ class ReplicationResource(Resource):
             state_token,
         ))
 
-    @request_handler()
+    @request_handler
     @defer.inlineCallbacks
     def _async_render_GET(self, request):
         limit = parse_integer(request, "limit", 100)
@@ -161,15 +158,6 @@ class ReplicationResource(Resource):
 
         result = yield self.notifier.wait_for_replication(replicate, timeout)
 
-        for stream_name, stream_content in result.items():
-            logger.info(
-                "Replicating %d rows of %s from %s -> %s",
-                len(stream_content["rows"]),
-                stream_name,
-                request_streams.get(stream_name),
-                stream_content["position"],
-            )
-
         request.write(json.dumps(result, ensure_ascii=False))
         finish_request(request)
 
@@ -394,7 +382,7 @@ class _Writer(object):
             position = rows[-1][0]
 
         self.streams[name] = {
-            "position": position if type(position) is int else str(position),
+            "position": str(position),
             "field_names": fields,
             "rows": rows,
         }

synapse/replication/slave/storage/account_data.py

@@ -1,100 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from ._base import BaseSlavedStore
-from ._slaved_id_tracker import SlavedIdTracker
-from synapse.storage import DataStore
-from synapse.storage.account_data import AccountDataStore
-from synapse.storage.tags import TagsStore
-from synapse.util.caches.stream_change_cache import StreamChangeCache
-
-
-class SlavedAccountDataStore(BaseSlavedStore):
-
-    def __init__(self, db_conn, hs):
-        super(SlavedAccountDataStore, self).__init__(db_conn, hs)
-        self._account_data_id_gen = SlavedIdTracker(
-            db_conn, "account_data_max_stream_id", "stream_id",
-        )
-        self._account_data_stream_cache = StreamChangeCache(
-            "AccountDataAndTagsChangeCache",
-            self._account_data_id_gen.get_current_token(),
-        )
-
-    get_account_data_for_user = (
-        AccountDataStore.__dict__["get_account_data_for_user"]
-    )
-
-    get_global_account_data_by_type_for_users = (
-        AccountDataStore.__dict__["get_global_account_data_by_type_for_users"]
-    )
-
-    get_global_account_data_by_type_for_user = (
-        AccountDataStore.__dict__["get_global_account_data_by_type_for_user"]
-    )
-
-    get_tags_for_user = TagsStore.__dict__["get_tags_for_user"]
-
-    get_updated_tags = DataStore.get_updated_tags.__func__
-    get_updated_account_data_for_user = (
-        DataStore.get_updated_account_data_for_user.__func__
-    )
-
-    def get_max_account_data_stream_id(self):
-        return self._account_data_id_gen.get_current_token()
-
-    def stream_positions(self):
-        result = super(SlavedAccountDataStore, self).stream_positions()
-        position = self._account_data_id_gen.get_current_token()
-        result["user_account_data"] = position
-        result["room_account_data"] = position
-        result["tag_account_data"] = position
-        return result
-
-    def process_replication(self, result):
-        stream = result.get("user_account_data")
-        if stream:
-            self._account_data_id_gen.advance(int(stream["position"]))
-            for row in stream["rows"]:
-                position, user_id, data_type = row[:3]
-                self.get_global_account_data_by_type_for_user.invalidate(
-                    (data_type, user_id,)
-                )
-                self.get_account_data_for_user.invalidate((user_id,))
-                self._account_data_stream_cache.entity_has_changed(
-                    user_id, position
-                )
-
-        stream = result.get("room_account_data")
-        if stream:
-            self._account_data_id_gen.advance(int(stream["position"]))
-            for row in stream["rows"]:
-                position, user_id = row[:2]
-                self.get_account_data_for_user.invalidate((user_id,))
-                self._account_data_stream_cache.entity_has_changed(
-                    user_id, position
-                )
-
-        stream = result.get("tag_account_data")
-        if stream:
-            self._account_data_id_gen.advance(int(stream["position"]))
-            for row in stream["rows"]:
-                position, user_id = row[:2]
-                self.get_tags_for_user.invalidate((user_id,))
-                self._account_data_stream_cache.entity_has_changed(
-                    user_id, position
-                )
-
-        return super(SlavedAccountDataStore, self).process_replication(result)

synapse/replication/slave/storage/appservice.py

@@ -1,30 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2015, 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from ._base import BaseSlavedStore
-from synapse.storage import DataStore
-from synapse.config.appservice import load_appservices
-
-
-class SlavedApplicationServiceStore(BaseSlavedStore):
-    def __init__(self, db_conn, hs):
-        super(SlavedApplicationServiceStore, self).__init__(db_conn, hs)
-        self.services_cache = load_appservices(
-            hs.config.server_name,
-            hs.config.app_service_config_files
-        )
-
-    get_app_service_by_token = DataStore.get_app_service_by_token.__func__
-    get_app_service_by_user_id = DataStore.get_app_service_by_user_id.__func__

synapse/replication/slave/storage/events.py

@@ -18,11 +18,11 @@ from ._slaved_id_tracker import SlavedIdTracker
 from synapse.api.constants import EventTypes
 from synapse.events import FrozenEvent
 from synapse.storage import DataStore
+from synapse.storage.room import RoomStore
 from synapse.storage.roommember import RoomMemberStore
 from synapse.storage.event_federation import EventFederationStore
 from synapse.storage.event_push_actions import EventPushActionsStore
 from synapse.storage.state import StateStore
-from synapse.storage.stream import StreamStore
 from synapse.util.caches.stream_change_cache import StreamChangeCache
 
 import ujson as json
@@ -57,12 +57,10 @@ class SlavedEventStore(BaseSlavedStore):
             "EventsRoomStreamChangeCache", min_event_val,
             prefilled_cache=event_cache_prefill,
         )
-        self._membership_stream_cache = StreamChangeCache(
-            "MembershipStreamChangeCache", events_max,
-        )
 
     # Cached functions can't be accessed through a class instance so we need
     # to reach inside the __dict__ to extract them.
+    get_room_name_and_aliases = RoomStore.__dict__["get_room_name_and_aliases"]
     get_rooms_for_user = RoomMemberStore.__dict__["get_rooms_for_user"]
     get_users_in_room = RoomMemberStore.__dict__["get_users_in_room"]
     get_latest_event_ids_in_room = EventFederationStore.__dict__[
@@ -77,21 +75,6 @@ class SlavedEventStore(BaseSlavedStore):
     get_unread_event_push_actions_by_room_for_user = (
         EventPushActionsStore.__dict__["get_unread_event_push_actions_by_room_for_user"]
     )
-    _get_state_group_for_events = (
-        StateStore.__dict__["_get_state_group_for_events"]
-    )
-    _get_state_group_for_event = (
-        StateStore.__dict__["_get_state_group_for_event"]
-    )
-    _get_state_groups_from_groups = (
-        StateStore.__dict__["_get_state_groups_from_groups"]
-    )
-    _get_state_group_from_group = (
-        StateStore.__dict__["_get_state_group_from_group"]
-    )
-    get_recent_event_ids_for_room = (
-        StreamStore.__dict__["get_recent_event_ids_for_room"]
-    )
 
     get_unread_push_actions_for_user_in_range = (
         DataStore.get_unread_push_actions_for_user_in_range.__func__
@@ -100,7 +83,6 @@ class SlavedEventStore(BaseSlavedStore):
         DataStore.get_push_action_users_in_range.__func__
     )
     get_event = DataStore.get_event.__func__
-    get_events = DataStore.get_events.__func__
     get_current_state = DataStore.get_current_state.__func__
     get_current_state_for_key = DataStore.get_current_state_for_key.__func__
     get_rooms_for_user_where_membership_is = (
@@ -113,39 +95,30 @@ class SlavedEventStore(BaseSlavedStore):
     get_room_events_stream_for_room = (
         DataStore.get_room_events_stream_for_room.__func__
     )
-    get_events_around = DataStore.get_events_around.__func__
-    get_state_for_event = DataStore.get_state_for_event.__func__
-    get_state_for_events = DataStore.get_state_for_events.__func__
-    get_state_groups = DataStore.get_state_groups.__func__
-    get_recent_events_for_room = DataStore.get_recent_events_for_room.__func__
-    get_room_events_stream_for_rooms = (
-        DataStore.get_room_events_stream_for_rooms.__func__
-    )
-    get_stream_token_for_event = DataStore.get_stream_token_for_event.__func__
 
-    _set_before_and_after = staticmethod(DataStore._set_before_and_after)
+    _set_before_and_after = DataStore._set_before_and_after
 
     _get_events = DataStore._get_events.__func__
     _get_events_from_cache = DataStore._get_events_from_cache.__func__
 
     _invalidate_get_event_cache = DataStore._invalidate_get_event_cache.__func__
+    _parse_events_txn = DataStore._parse_events_txn.__func__
+    _get_events_txn = DataStore._get_events_txn.__func__
     _enqueue_events = DataStore._enqueue_events.__func__
     _do_fetch = DataStore._do_fetch.__func__
+    _fetch_events_txn = DataStore._fetch_events_txn.__func__
     _fetch_event_rows = DataStore._fetch_event_rows.__func__
     _get_event_from_row = DataStore._get_event_from_row.__func__
+    _get_event_from_row_txn = DataStore._get_event_from_row_txn.__func__
     _get_rooms_for_user_where_membership_is_txn = (
         DataStore._get_rooms_for_user_where_membership_is_txn.__func__
     )
     _get_members_rows_txn = DataStore._get_members_rows_txn.__func__
-    _get_state_for_groups = DataStore._get_state_for_groups.__func__
-    _get_all_state_from_cache = DataStore._get_all_state_from_cache.__func__
-    _get_events_around_txn = DataStore._get_events_around_txn.__func__
-    _get_some_state_from_cache = DataStore._get_some_state_from_cache.__func__
 
     def stream_positions(self):
         result = super(SlavedEventStore, self).stream_positions()
         result["events"] = self._stream_id_gen.get_current_token()
-        result["backfill"] = -self._backfill_id_gen.get_current_token()
+        result["backfill"] = self._backfill_id_gen.get_current_token()
         return result
 
     def process_replication(self, result):
@@ -155,7 +128,7 @@ class SlavedEventStore(BaseSlavedStore):
 
         stream = result.get("events")
         if stream:
-            self._stream_id_gen.advance(int(stream["position"]))
+            self._stream_id_gen.advance(stream["position"])
             for row in stream["rows"]:
                 self._process_replication_row(
                     row, backfilled=False, state_resets=state_resets
@@ -163,7 +136,7 @@ class SlavedEventStore(BaseSlavedStore):
 
         stream = result.get("backfill")
         if stream:
-            self._backfill_id_gen.advance(-int(stream["position"]))
+            self._backfill_id_gen.advance(stream["position"])
             for row in stream["rows"]:
                 self._process_replication_row(
                     row, backfilled=True, state_resets=state_resets
@@ -171,14 +144,12 @@ class SlavedEventStore(BaseSlavedStore):
 
         stream = result.get("forward_ex_outliers")
         if stream:
-            self._stream_id_gen.advance(int(stream["position"]))
             for row in stream["rows"]:
                 event_id = row[1]
                 self._invalidate_get_event_cache(event_id)
 
         stream = result.get("backward_ex_outliers")
         if stream:
-            self._backfill_id_gen.advance(-int(stream["position"]))
             for row in stream["rows"]:
                 event_id = row[1]
                 self._invalidate_get_event_cache(event_id)
@@ -200,6 +171,7 @@ class SlavedEventStore(BaseSlavedStore):
             self.get_rooms_for_user.invalidate_all()
             self.get_users_in_room.invalidate((event.room_id,))
             # self.get_joined_hosts_for_room.invalidate((event.room_id,))
+            self.get_room_name_and_aliases.invalidate((event.room_id,))
 
         self._invalidate_get_event_cache(event.event_id)
 
@@ -225,9 +197,9 @@ class SlavedEventStore(BaseSlavedStore):
             self.get_rooms_for_user.invalidate((event.state_key,))
             # self.get_joined_hosts_for_room.invalidate((event.room_id,))
             self.get_users_in_room.invalidate((event.room_id,))
-            self._membership_stream_cache.entity_has_changed(
-                event.state_key, event.internal_metadata.stream_ordering
-            )
+            # self._membership_stream_cache.entity_has_changed(
+            #    event.state_key, event.internal_metadata.stream_ordering
+            # )
             self.get_invited_rooms_for_user.invalidate((event.state_key,))
 
         if not event.is_state():
@@ -243,3 +215,9 @@ class SlavedEventStore(BaseSlavedStore):
         self._get_current_state_for_key.invalidate((
             event.room_id, event.type, event.state_key
         ))
+
+        if event.type in [EventTypes.Name, EventTypes.Aliases]:
+            self.get_room_name_and_aliases.invalidate(
+                (event.room_id,)
+            )
+            pass

synapse/replication/slave/storage/filtering.py

@@ -1,25 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2015, 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from ._base import BaseSlavedStore
-from synapse.storage.filtering import FilteringStore
-
-
-class SlavedFilteringStore(BaseSlavedStore):
-    def __init__(self, db_conn, hs):
-        super(SlavedFilteringStore, self).__init__(db_conn, hs)
-
-    # Filters are immutable so this cache doesn't need to be expired
-    get_user_filter = FilteringStore.__dict__["get_user_filter"]

synapse/replication/slave/storage/presence.py

@@ -1,59 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from ._base import BaseSlavedStore
-from ._slaved_id_tracker import SlavedIdTracker
-
-from synapse.util.caches.stream_change_cache import StreamChangeCache
-from synapse.storage import DataStore
-
-
-class SlavedPresenceStore(BaseSlavedStore):
-    def __init__(self, db_conn, hs):
-        super(SlavedPresenceStore, self).__init__(db_conn, hs)
-        self._presence_id_gen = SlavedIdTracker(
-            db_conn, "presence_stream", "stream_id",
-        )
-
-        self._presence_on_startup = self._get_active_presence(db_conn)
-
-        self.presence_stream_cache = self.presence_stream_cache = StreamChangeCache(
-            "PresenceStreamChangeCache", self._presence_id_gen.get_current_token()
-        )
-
-    _get_active_presence = DataStore._get_active_presence.__func__
-    take_presence_startup_info = DataStore.take_presence_startup_info.__func__
-    get_presence_for_users = DataStore.get_presence_for_users.__func__
-
-    def get_current_presence_token(self):
-        return self._presence_id_gen.get_current_token()
-
-    def stream_positions(self):
-        result = super(SlavedPresenceStore, self).stream_positions()
-        position = self._presence_id_gen.get_current_token()
-        result["presence"] = position
-        return result
-
-    def process_replication(self, result):
-        stream = result.get("presence")
-        if stream:
-            self._presence_id_gen.advance(int(stream["position"]))
-            for row in stream["rows"]:
-                position, user_id = row[:2]
-                self.presence_stream_cache.entity_has_changed(
-                    user_id, position
-                )
-
-        return super(SlavedPresenceStore, self).process_replication(result)

synapse/replication/slave/storage/push_rule.py

@@ -1,67 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2015, 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from .events import SlavedEventStore
-from ._slaved_id_tracker import SlavedIdTracker
-from synapse.storage import DataStore
-from synapse.storage.push_rule import PushRuleStore
-from synapse.util.caches.stream_change_cache import StreamChangeCache
-
-
-class SlavedPushRuleStore(SlavedEventStore):
-    def __init__(self, db_conn, hs):
-        super(SlavedPushRuleStore, self).__init__(db_conn, hs)
-        self._push_rules_stream_id_gen = SlavedIdTracker(
-            db_conn, "push_rules_stream", "stream_id",
-        )
-        self.push_rules_stream_cache = StreamChangeCache(
-            "PushRulesStreamChangeCache",
-            self._push_rules_stream_id_gen.get_current_token(),
-        )
-
-    get_push_rules_for_user = PushRuleStore.__dict__["get_push_rules_for_user"]
-    get_push_rules_enabled_for_user = (
-        PushRuleStore.__dict__["get_push_rules_enabled_for_user"]
-    )
-    have_push_rules_changed_for_user = (
-        DataStore.have_push_rules_changed_for_user.__func__
-    )
-
-    def get_push_rules_stream_token(self):
-        return (
-            self._push_rules_stream_id_gen.get_current_token(),
-            self._stream_id_gen.get_current_token(),
-        )
-
-    def stream_positions(self):
-        result = super(SlavedPushRuleStore, self).stream_positions()
-        result["push_rules"] = self._push_rules_stream_id_gen.get_current_token()
-        return result
-
-    def process_replication(self, result):
-        stream = result.get("push_rules")
-        if stream:
-            for row in stream["rows"]:
-                position = row[0]
-                user_id = row[2]
-                self.get_push_rules_for_user.invalidate((user_id,))
-                self.get_push_rules_enabled_for_user.invalidate((user_id,))
-                self.push_rules_stream_cache.entity_has_changed(
-                    user_id, position
-                )
-
-            self._push_rules_stream_id_gen.advance(int(stream["position"]))
-
-        return super(SlavedPushRuleStore, self).process_replication(result)

synapse/replication/slave/storage/pushers.py

@@ -43,10 +43,10 @@ class SlavedPusherStore(BaseSlavedStore):
     def process_replication(self, result):
         stream = result.get("pushers")
         if stream:
-            self._pushers_id_gen.advance(int(stream["position"]))
+            self._pushers_id_gen.advance(stream["position"])
 
         stream = result.get("deleted_pushers")
         if stream:
-            self._pushers_id_gen.advance(int(stream["position"]))
+            self._pushers_id_gen.advance(stream["position"])
 
         return super(SlavedPusherStore, self).process_replication(result)

synapse/replication/slave/storage/receipts.py

@@ -18,7 +18,6 @@ from ._slaved_id_tracker import SlavedIdTracker
 
 from synapse.storage import DataStore
 from synapse.storage.receipts import ReceiptsStore
-from synapse.util.caches.stream_change_cache import StreamChangeCache
 
 # So, um, we want to borrow a load of functions intended for reading from
 # a DataStore, but we don't want to take functions that either write to the
@@ -38,28 +37,11 @@ class SlavedReceiptsStore(BaseSlavedStore):
             db_conn, "receipts_linearized", "stream_id"
         )
 
-        self._receipts_stream_cache = StreamChangeCache(
-            "ReceiptsRoomChangeCache", self._receipts_id_gen.get_current_token()
-        )
-
     get_receipts_for_user = ReceiptsStore.__dict__["get_receipts_for_user"]
-    get_linearized_receipts_for_room = (
-        ReceiptsStore.__dict__["get_linearized_receipts_for_room"]
-    )
-    _get_linearized_receipts_for_rooms = (
-        ReceiptsStore.__dict__["_get_linearized_receipts_for_rooms"]
-    )
-    get_last_receipt_event_id_for_user = (
-        ReceiptsStore.__dict__["get_last_receipt_event_id_for_user"]
-    )
 
     get_max_receipt_stream_id = DataStore.get_max_receipt_stream_id.__func__
     get_all_updated_receipts = DataStore.get_all_updated_receipts.__func__
 
-    get_linearized_receipts_for_rooms = (
-        DataStore.get_linearized_receipts_for_rooms.__func__
-    )
-
     def stream_positions(self):
         result = super(SlavedReceiptsStore, self).stream_positions()
         result["receipts"] = self._receipts_id_gen.get_current_token()
@@ -68,17 +50,12 @@ class SlavedReceiptsStore(BaseSlavedStore):
     def process_replication(self, result):
         stream = result.get("receipts")
         if stream:
-            self._receipts_id_gen.advance(int(stream["position"]))
+            self._receipts_id_gen.advance(stream["position"])
             for row in stream["rows"]:
-                position, room_id, receipt_type, user_id = row[:4]
+                room_id, receipt_type, user_id = row[1:4]
                 self.invalidate_caches_for_receipt(room_id, receipt_type, user_id)
-                self._receipts_stream_cache.entity_has_changed(room_id, position)
 
         return super(SlavedReceiptsStore, self).process_replication(result)
 
     def invalidate_caches_for_receipt(self, room_id, receipt_type, user_id):
         self.get_receipts_for_user.invalidate((user_id, receipt_type))
-        self.get_linearized_receipts_for_room.invalidate_many((room_id,))
-        self.get_last_receipt_event_id_for_user.invalidate(
-            (user_id, room_id, receipt_type)
-        )

synapse/replication/slave/storage/registration.py

@@ -1,30 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2015, 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from ._base import BaseSlavedStore
-from synapse.storage import DataStore
-from synapse.storage.registration import RegistrationStore
-
-
-class SlavedRegistrationStore(BaseSlavedStore):
-    def __init__(self, db_conn, hs):
-        super(SlavedRegistrationStore, self).__init__(db_conn, hs)
-
-    # TODO: use the cached version and invalidate deleted tokens
-    get_user_by_access_token = RegistrationStore.__dict__[
-        "get_user_by_access_token"
-    ].orig
-
-    _query_for_auth = DataStore._query_for_auth.__func__

synapse/rest/__init__.py

@@ -44,8 +44,6 @@ from synapse.rest.client.v2_alpha import (
     tokenrefresh,
     tags,
     account_data,
-    report_event,
-    openid,
 )
 
 from synapse.http.server import JsonResource
@@ -88,5 +86,3 @@ class ClientRestResource(JsonResource):
         tokenrefresh.register_servlets(hs, client_resource)
         tags.register_servlets(hs, client_resource)
         account_data.register_servlets(hs, client_resource)
-        report_event.register_servlets(hs, client_resource)
-        openid.register_servlets(hs, client_resource)