Add basic file API

.github/ISSUE_TEMPLATE.md

@@ -1,47 +0,0 @@
-<!-- 
-
-**IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER**: 
-You will likely get better support more quickly if you ask in ** #matrix:matrix.org ** ;)
-
-
-This is a bug report template. By following the instructions below and
-filling out the sections with your information, you will help the us to get all
-the necessary data to fix your issue.
-
-You can also preview your report before submitting it. You may remove sections
-that aren't relevant to your particular case.
-
-Text between <!-- and --​> marks will be invisible in the report.
-
--->
-
-### Description
-
-Describe here the problem that you are experiencing, or the feature you are requesting.
-
-### Steps to reproduce
-
-- For bugs, list the steps
-- that reproduce the bug
-- using hyphens as bullet points
-
-Describe how what happens differs from what you expected.
-
-If you can identify any relevant log snippets from _homeserver.log_, please include
-those here (please be careful to remove any personal or private data):
-
-### Version information
-
-<!-- IMPORTANT: please answer the following questions, to help us narrow down the problem -->
-
-- **Homeserver**: Was this issue identified on matrix.org or another homeserver?
-
-If not matrix.org:
-- **Version**:        What version of Synapse is running? <!-- 
-You can find the Synapse version by inspecting the server headers (replace matrix.org with
-your own homeserver domain):
-$ curl -v https://matrix.org/_matrix/client/versions 2>&1 | grep "Server:"
--->
-- **Install method**: package manager/git clone/pip      
-- **Platform**:       Tell us about the environment in which your homeserver is operating
-                      - distro, hardware, if it's running in a vm/container, etc.

.gitignore

@@ -24,10 +24,10 @@ homeserver*.yaml
 .coverage
 htmlcov
 
-demo/*/*.db
-demo/*/*.log
-demo/*/*.log.*
-demo/*/*.pid
+demo/*.db
+demo/*.log
+demo/*.log.*
+demo/*.pid
 demo/media_store.*
 demo/etc
 
@@ -46,5 +46,3 @@ static/client/register/register_config.js
 
 env/
 *.config
-
-.vscode/

.travis.yml

@@ -1,17 +0,0 @@
-sudo: false
-language: python
-python: 2.7
-
-# tell travis to cache ~/.cache/pip
-cache: pip
-
-env:
-  - TOX_ENV=packaging
-  - TOX_ENV=pep8
-  - TOX_ENV=py27
-
-install:
-  - pip install tox
-
-script:
-  - tox -e $TOX_ENV

CONTRIBUTING.rst

@@ -30,12 +30,8 @@ use github's pull request workflow to review the contribution, and either ask
 you to make any refinements needed or merge it and make them ourselves. The
 changes will then land on master when we next do a release.
 
-We use `Jenkins <http://matrix.org/jenkins>`_ and 
-`Travis <https://travis-ci.org/matrix-org/synapse>`_ for continuous
-integration. All pull requests to synapse get automatically tested by Travis; 
-the Jenkins builds require an adminstrator to start them. If your change 
-breaks the build, this will be shown in github, so please keep an eye on the 
-pull request for feedback.
+We use Jenkins for continuous integration (http://matrix.org/jenkins), and
+typically all pull requests get automatically tested Jenkins: if your change breaks the build, Jenkins will yell about it in #matrix-dev:matrix.org so please lurk there and keep an eye open.
 
 Code style
 ~~~~~~~~~~

MANIFEST.in

@@ -14,7 +14,6 @@ recursive-include docs *
 recursive-include res *
 recursive-include scripts *
 recursive-include scripts-dev *
-recursive-include synapse *.pyi
 recursive-include tests *.py
 
 recursive-include synapse/static *.css
@@ -24,8 +23,5 @@ recursive-include synapse/static *.js
 
 exclude jenkins.sh
 exclude jenkins*.sh
-exclude jenkins*
-recursive-exclude jenkins *.sh
 
-prune .github
 prune demo/etc

README.rst

@@ -11,8 +11,8 @@ VoIP.  The basics you need to know to get up and running are:
   like ``#matrix:matrix.org`` or ``#test:localhost:8448``.
 
 - Matrix user IDs look like ``@matthew:matrix.org`` (although in the future
-  you will normally refer to yourself and others using a third party identifier
-  (3PID): email address, phone number, etc rather than manipulating Matrix user IDs)
+  you will normally refer to yourself and others using a 3PID: email
+  address, phone number, etc rather than manipulating Matrix user IDs)
 
 The overall architecture is::
 
@@ -20,13 +20,12 @@ The overall architecture is::
              https://somewhere.org/_matrix      https://elsewhere.net/_matrix
 
 ``#matrix:matrix.org`` is the official support room for Matrix, and can be
-accessed by any client from https://matrix.org/docs/projects/try-matrix-now.html or
-via IRC bridge at irc://irc.freenode.net/matrix.
+accessed by any client from https://matrix.org/blog/try-matrix-now or via IRC
+bridge at irc://irc.freenode.net/matrix.
 
 Synapse is currently in rapid development, but as of version 0.5 we believe it
 is sufficiently stable to be run as an internet-facing service for real usage!
 
-
 About Matrix
 ============
 
@@ -53,10 +52,10 @@ generation of fully open and interoperable messaging and VoIP apps for the
 internet.
 
 Synapse is a reference "homeserver" implementation of Matrix from the core
-development team at matrix.org, written in Python/Twisted.  It is intended to
-showcase the concept of Matrix and let folks see the spec in the context of a
-codebase and let you run your own homeserver and generally help bootstrap the
-ecosystem.
+development team at matrix.org, written in Python/Twisted for clarity and
+simplicity.  It is intended to showcase the concept of Matrix and let folks see
+the spec in the context of a codebase and let you run your own homeserver and
+generally help bootstrap the ecosystem.
 
 In Matrix, every user runs one or more Matrix clients, which connect through to
 a Matrix homeserver. The homeserver stores all their personal chat history and
@@ -67,16 +66,26 @@ hosted by someone else (e.g. matrix.org) - there is no single point of control
 or mandatory service provider in Matrix, unlike WhatsApp, Facebook, Hangouts,
 etc.
 
+Synapse ships with two basic demo Matrix clients: webclient (a basic group chat
+web client demo implemented in AngularJS) and cmdclient (a basic Python
+command line utility which lets you easily see what the JSON APIs are up to).
+
+Meanwhile, iOS and Android SDKs and clients are available from:
+
+- https://github.com/matrix-org/matrix-ios-sdk
+- https://github.com/matrix-org/matrix-ios-kit
+- https://github.com/matrix-org/matrix-ios-console
+- https://github.com/matrix-org/matrix-android-sdk
+
 We'd like to invite you to join #matrix:matrix.org (via
-https://matrix.org/docs/projects/try-matrix-now.html), run a homeserver, take a look
-at the `Matrix spec <https://matrix.org/docs/spec>`_, and experiment with the
-`APIs <https://matrix.org/docs/api>`_ and `Client SDKs
-<http://matrix.org/docs/projects/try-matrix-now.html#client-sdks>`_.
+https://matrix.org/blog/try-matrix-now), run a homeserver, take a look at the
+Matrix spec at https://matrix.org/docs/spec and API docs at
+https://matrix.org/docs/api, experiment with the APIs and the demo clients, and
+report any bugs via https://matrix.org/jira.
 
 Thanks for using Matrix!
 
-[1] End-to-end encryption is currently in beta: `blog post <https://matrix.org/blog/2016/11/21/matrixs-olm-end-to-end-encryption-security-assessment-released-and-implemented-cross-platform-on-riot-at-last>`_.
-
+[1] End-to-end encryption is currently in development - see https://matrix.org/git/olm
 
 Synapse Installation
 ====================
@@ -84,17 +93,11 @@ Synapse Installation
 Synapse is the reference python/twisted Matrix homeserver implementation.
 
 System requirements:
-
 - POSIX-compliant system (tested on Linux & OS X)
 - Python 2.7
-- At least 1GB of free RAM if you want to join large public rooms like #matrix:matrix.org
+- At least 512 MB RAM.
 
-Installing from source
-----------------------
-(Prebuilt packages are available for some platforms - see `Platform-Specific
-Instructions`_.)
-
-Synapse is written in python but some of the libraries it uses are written in
+Synapse is written in python but some of the libraries is uses are written in
 C. So before we can install synapse itself we need a working C compiler and the
 header files for python C extensions.
 
@@ -109,10 +112,10 @@ Installing prerequisites on ArchLinux::
     sudo pacman -S base-devel python2 python-pip \
                    python-setuptools python-virtualenv sqlite3
 
-Installing prerequisites on CentOS 7 or Fedora 25::
+Installing prerequisites on CentOS 7::
 
     sudo yum install libtiff-devel libjpeg-devel libzip-devel freetype-devel \
-                     lcms2-devel libwebp-devel tcl-devel tk-devel redhat-rpm-config \
+                     lcms2-devel libwebp-devel tcl-devel tk-devel \
                      python-virtualenv libffi-devel openssl-devel
     sudo yum groupinstall "Development Tools"
 
@@ -121,7 +124,6 @@ Installing prerequisites on Mac OS X::
     xcode-select --install
     sudo easy_install pip
     sudo pip install virtualenv
-    brew install pkg-config libffi
 
 Installing prerequisites on Raspbian::
 
@@ -132,22 +134,10 @@ Installing prerequisites on Raspbian::
     sudo pip install --upgrade ndg-httpsclient
     sudo pip install --upgrade virtualenv
 
-Installing prerequisites on openSUSE::
-
-    sudo zypper in -t pattern devel_basis
-    sudo zypper in python-pip python-setuptools sqlite3 python-virtualenv \
-                   python-devel libffi-devel libopenssl-devel libjpeg62-devel
-
-Installing prerequisites on OpenBSD::
-
-    doas pkg_add python libffi py-pip py-setuptools sqlite3 py-virtualenv \
-                 libxslt
-
 To install the synapse homeserver run::
 
     virtualenv -p python2.7 ~/.synapse
     source ~/.synapse/bin/activate
-    pip install --upgrade pip
     pip install --upgrade setuptools
     pip install https://github.com/matrix-org/synapse/tarball/master
 
@@ -155,7 +145,7 @@ This installs synapse, along with the libraries it uses, into a virtual
 environment under ``~/.synapse``.  Feel free to pick a different directory
 if you prefer.
 
-In case of problems, please see the _`Troubleshooting` section below.
+In case of problems, please see the _Troubleshooting section below.
 
 Alternatively, Silvio Fricke has contributed a Dockerfile to automate the
 above in Docker at https://registry.hub.docker.com/u/silviof/docker-matrix/.
@@ -164,65 +154,29 @@ Also, Martin Giess has created an auto-deployment process with vagrant/ansible,
 tested with VirtualBox/AWS/DigitalOcean - see https://github.com/EMnify/matrix-synapse-auto-deploy 
 for details.
 
-Configuring synapse
--------------------
-
-Before you can start Synapse, you will need to generate a configuration
-file. To do this, run (in your virtualenv, as before)::
+To set up your homeserver, run (in your virtualenv, as before)::
 
     cd ~/.synapse
     python -m synapse.app.homeserver \
-        --server-name my.domain.name \
+        --server-name machine.my.domain.name \
         --config-path homeserver.yaml \
         --generate-config \
         --report-stats=[yes|no]
 
-... substituting an appropriate value for ``--server-name``. The server name
-determines the "domain" part of user-ids for users on your server: these will
-all be of the format ``@user:my.domain.name``. It also determines how other
-matrix servers will reach yours for `Federation`_. For a test configuration,
-set this to the hostname of your server. For a more production-ready setup, you
-will probably want to specify your domain (``example.com``) rather than a
-matrix-specific hostname here (in the same way that your email address is
-probably ``user@example.com`` rather than ``user@email.example.com``) - but
-doing so may require more advanced setup - see `Setting up
-Federation`_. Beware that the server name cannot be changed later.
+...substituting your host and domain name as appropriate.
 
-This command will generate you a config file that you can then customise, but it will
+This will generate you a config file that you can then customise, but it will
 also generate a set of keys for you. These keys will allow your Home Server to
 identify itself to other Home Servers, so don't lose or delete them. It would be
-wise to back them up somewhere safe. (If, for whatever reason, you do need to
+wise to back them up somewhere safe. If, for whatever reason, you do need to
 change your Home Server's keys, you may find that other Home Servers have the
 old key cached. If you update the signing key, you should change the name of the
-key in the ``<server name>.signing.key`` file (the second word) to something
-different. See `the spec`__ for more information on key management.)
+key in the <server name>.signing.key file (the second word) to something different.
 
-.. __: `key_management`_
-
-The default configuration exposes two HTTP ports: 8008 and 8448. Port 8008 is
-configured without TLS; it should be behind a reverse proxy for TLS/SSL
-termination on port 443 which in turn should be used for clients. Port 8448
-is configured to use TLS with a self-signed certificate. If you would like
-to do initial test with a client without having to setup a reverse proxy,
-you can temporarly use another certificate. (Note that a self-signed
-certificate is fine for `Federation`_). You can do so by changing
-``tls_certificate_path``, ``tls_private_key_path`` and ``tls_dh_params_path``
-in ``homeserver.yaml``; alternatively, you can use a reverse-proxy, but be sure
-to read `Using a reverse proxy with Synapse`_ when doing so.
-
-Apart from port 8448 using TLS, both ports are the same in the default
-configuration.
-
-Registering a user
-------------------
-
-You will need at least one user on your server in order to use a Matrix
-client. Users can be registered either `via a Matrix client`__, or via a
-commandline script.
-
-.. __: `client-user-reg`_
-
-To get started, it is easiest to use the command line to register new users::
+By default, registration of new users is disabled. You can either enable
+registration in the config by specifying ``enable_registration: true``
+(it is then recommended to also set up CAPTCHA - see docs/CAPTCHA_SETUP), or
+you can use the command line to register new users::
 
     $ source ~/.synapse/bin/activate
     $ synctl start # if not already running
@@ -230,41 +184,10 @@ To get started, it is easiest to use the command line to register new users::
     New user localpart: erikj
     Password:
     Confirm password:
-    Make admin [no]:
     Success!
 
-This process uses a setting ``registration_shared_secret`` in
-``homeserver.yaml``, which is shared between Synapse itself and the
-``register_new_matrix_user`` script. It doesn't matter what it is (a random
-value is generated by ``--generate-config``), but it should be kept secret, as
-anyone with knowledge of it can register users on your server even if
-``enable_registration`` is ``false``.
-
-Setting up a TURN server
-------------------------
-
 For reliable VoIP calls to be routed via this homeserver, you MUST configure
-a TURN server.  See `<docs/turn-howto.rst>`_ for details.
-
-IPv6
-----
-
-As of Synapse 0.19 we finally support IPv6, many thanks to @kyrias and @glyph
-for providing PR #1696.
-
-However, for federation to work on hosts with IPv6 DNS servers you **must**
-be running Twisted 17.1.0 or later - see https://github.com/matrix-org/synapse/issues/1002
-for details.  We can't make Synapse depend on Twisted 17.1 by default
-yet as it will break most older distributions (see https://github.com/matrix-org/synapse/pull/1909)
-so if you are using operating system dependencies you'll have to install your
-own Twisted 17.1 package via pip or backports etc.
-
-If you're running in a virtualenv then pip should have installed the newest
-Twisted automatically, but if your virtualenv is old you will need to manually
-upgrade to a newer Twisted dependency via:
-
-    pip install Twisted>=17.1.0
-
+a TURN server.  See docs/turn-howto.rst for details.
 
 Running Synapse
 ===============
@@ -276,72 +199,29 @@ run (e.g. ``~/.synapse``), and::
     source ./bin/activate
     synctl start
 
+Using PostgreSQL
+================
 
-Connecting to Synapse from a client
-===================================
+As of Synapse 0.9, `PostgreSQL <http://www.postgresql.org>`_ is supported as an
+alternative to the `SQLite <http://sqlite.org/>`_ database that Synapse has
+traditionally used for convenience and simplicity.
 
-The easiest way to try out your new Synapse installation is by connecting to it
-from a web client. The easiest option is probably the one at
-http://riot.im/app. You will need to specify a "Custom server" when you log on
-or register: set this to ``https://domain.tld`` if you setup a reverse proxy
-following the recommended setup, or ``https://localhost:8448`` - remember to specify the
-port (``:8448``) if not ``:443`` unless you changed the configuration. (Leave the identity
-server as the default - see `Identity servers`_.)
+The advantages of Postgres include:
 
-If using port 8448 you will run into errors until you accept the self-signed
-certificate. You can easily do this by going to ``https://localhost:8448``
-directly with your browser and accept the presented certificate. You can then
-go back in your web client and proceed further.
+* significant performance improvements due to the superior threading and
+  caching model, smarter query optimiser
+* allowing the DB to be run on separate hardware
+* allowing basic active/backup high-availability with a "hot spare" synapse
+  pointing at the same DB master, as well as enabling DB replication in
+  synapse itself.
 
-If all goes well you should at least be able to log in, create a room, and
-start sending messages.
+The only disadvantage is that the code is relatively new as of April 2015 and
+may have a few regressions relative to SQLite.
 
-(The homeserver runs a web client by default at https://localhost:8448/, though
-as of the time of writing it is somewhat outdated and not really recommended -
-https://github.com/matrix-org/synapse/issues/1527).
+For information on how to install and use PostgreSQL, please see
+`docs/postgres.rst <docs/postgres.rst>`_.
 
-.. _`client-user-reg`:
-
-Registering a new user from a client
-------------------------------------
-
-By default, registration of new users via Matrix clients is disabled. To enable
-it, specify ``enable_registration: true`` in ``homeserver.yaml``. (It is then
-recommended to also set up CAPTCHA - see `<docs/CAPTCHA_SETUP.rst>`_.)
-
-Once ``enable_registration`` is set to ``true``, it is possible to register a
-user via `riot.im <https://riot.im/app/#/register>`_ or other Matrix clients.
-
-Your new user name will be formed partly from the ``server_name`` (see
-`Configuring synapse`_), and partly from a localpart you specify when you
-create the account. Your name will take the form of::
-
-    @localpart:my.domain.name
-
-(pronounced "at localpart on my dot domain dot name").
-
-As when logging in, you will need to specify a "Custom server".  Specify your
-desired ``localpart`` in the 'User name' box.
-
-
-Security Note
-=============
-
-Matrix serves raw user generated data in some APIs - specifically the `content
-repository endpoints <http://matrix.org/docs/spec/client_server/latest.html#get-matrix-media-r0-download-servername-mediaid>`_.
-
-Whilst we have tried to mitigate against possible XSS attacks (e.g.
-https://github.com/matrix-org/synapse/pull/1021) we recommend running
-matrix homeservers on a dedicated domain name, to limit any malicious user generated
-content served to web browsers a matrix API from being able to attack webapps hosted
-on the same domain.  This is particularly true of sharing a matrix webclient and
-server on the same domain.
-
-See https://github.com/vector-im/vector-web/issues/1977 and
-https://developer.github.com/changes/2014-04-25-user-content-security for more details.
-
-
-Platform-Specific Instructions
+Platform Specific Instructions
 ==============================
 
 Debian
@@ -349,27 +229,21 @@ Debian
 
 Matrix provides official Debian packages via apt from http://matrix.org/packages/debian/.
 Note that these packages do not include a client - choose one from
-https://matrix.org/docs/projects/try-matrix-now.html (or build your own with one of our SDKs :)
+https://matrix.org/blog/try-matrix-now/ (or build your own with one of our SDKs :)
 
 Fedora
 ------
 
-Synapse is in the Fedora repositories as ``matrix-synapse``::
-
-    sudo dnf install matrix-synapse
-
 Oleg Girko provides Fedora RPMs at
 https://obs.infoserver.lv/project/monitor/matrix-synapse
 
 ArchLinux
 ---------
 
-The quickest way to get up and running with ArchLinux is probably with the community package
-https://www.archlinux.org/packages/community/any/matrix-synapse/, which should pull in most of
-the necessary dependencies. If the default web client is to be served (enabled by default in
-the generated config),
-https://www.archlinux.org/packages/community/any/python2-matrix-angular-sdk/ will also need to
-be installed.
+The quickest way to get up and running with ArchLinux is probably with Ivan
+Shapovalov's AUR package from
+https://aur.archlinux.org/packages/matrix-synapse/, which should pull in all
+the necessary dependencies.
 
 Alternatively, to install using pip a few changes may be needed as ArchLinux
 defaults to python 3, but synapse currently assumes python 2.7 by default:
@@ -406,35 +280,9 @@ FreeBSD
 
 Synapse can be installed via FreeBSD Ports or Packages contributed by Brendan Molloy from:
 
- - Ports: ``cd /usr/ports/net-im/py-matrix-synapse && make install clean``
+ - Ports: ``cd /usr/ports/net/py-matrix-synapse && make install clean``
  - Packages: ``pkg install py27-matrix-synapse``
 
-
-OpenBSD
--------
-
-There is currently no port for OpenBSD. Additionally, OpenBSD's security
-settings require a slightly more difficult installation process.
-
-1) Create a new directory in ``/usr/local`` called ``_synapse``. Also, create a
-   new user called ``_synapse`` and set that directory as the new user's home.
-   This is required because, by default, OpenBSD only allows binaries which need
-   write and execute permissions on the same memory space to be run from
-   ``/usr/local``.
-2) ``su`` to the new ``_synapse`` user and change to their home directory.
-3) Create a new virtualenv: ``virtualenv -p python2.7 ~/.synapse``
-4) Source the virtualenv configuration located at
-   ``/usr/local/_synapse/.synapse/bin/activate``. This is done in ``ksh`` by
-   using the ``.`` command, rather than ``bash``'s ``source``.
-5) Optionally, use ``pip`` to install ``lxml``, which Synapse needs to parse
-   webpages for their titles.
-6) Use ``pip`` to install this repository: ``pip install
-   https://github.com/matrix-org/synapse/tarball/master``
-7) Optionally, change ``_synapse``'s shell to ``/bin/false`` to reduce the
-   chance of a compromised Synapse server being used to take over your box.
-
-After this, you may proceed with the rest of the install directions.
-
 NixOS
 -----
 
@@ -474,7 +322,6 @@ Troubleshooting:
   you do, you may need to create a symlink to ``libsodium.a`` so ``ld`` can find
   it: ``ln -s /usr/local/lib/libsodium.a /usr/lib/libsodium.a``
 
-
 Troubleshooting
 ===============
 
@@ -538,30 +385,6 @@ fix try re-installing from PyPI or directly from
     # Install from github
     pip install --user https://github.com/pyca/pynacl/tarball/master
 
-Running out of File Handles
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-If synapse runs out of filehandles, it typically fails badly - live-locking
-at 100% CPU, and/or failing to accept new TCP connections (blocking the
-connecting client).  Matrix currently can legitimately use a lot of file handles,
-thanks to busy rooms like #matrix:matrix.org containing hundreds of participating
-servers.  The first time a server talks in a room it will try to connect
-simultaneously to all participating servers, which could exhaust the available
-file descriptors between DNS queries & HTTPS sockets, especially if DNS is slow
-to respond.  (We need to improve the routing algorithm used to be better than
-full mesh, but as of June 2017 this hasn't happened yet).
-
-If you hit this failure mode, we recommend increasing the maximum number of
-open file handles to be at least 4096 (assuming a default of 1024 or 256).
-This is typically done by editing ``/etc/security/limits.conf``
-
-Separately, Synapse may leak file handles if inbound HTTP requests get stuck
-during processing - e.g. blocked behind a lock or talking to a remote server etc.
-This is best diagnosed by matching up the 'Received request' and 'Processed request'
-log lines and looking for any 'Processed request' lines which take more than
-a few seconds to execute.  Please let us know at #matrix-dev:matrix.org if
-you see this failure mode so we can help debug it, however.
-
 ArchLinux
 ~~~~~~~~~
 
@@ -572,6 +395,37 @@ you will need to explicitly call Python2.7 - either running as::
 
 ...or by editing synctl with the correct python executable.
 
+Synapse Development
+===================
+
+To check out a synapse for development, clone the git repo into a working
+directory of your choice::
+
+    git clone https://github.com/matrix-org/synapse.git
+    cd synapse
+
+Synapse has a number of external dependencies, that are easiest
+to install using pip and a virtualenv::
+
+    virtualenv env
+    source env/bin/activate
+    python synapse/python_dependencies.py | xargs -n1 pip install
+    pip install setuptools_trial mock
+
+This will run a process of downloading and installing all the needed
+dependencies into a virtual env.
+
+Once this is done, you may wish to run Synapse's unit tests, to
+check that everything is installed as it should be::
+
+    python setup.py test
+
+This should end with a 'PASSED' result::
+
+    Ran 143 tests in 0.601s
+
+    PASSED (successes=143)
+
 
 Upgrading an existing Synapse
 =============================
@@ -582,259 +436,143 @@ versions of synapse.
 
 .. _UPGRADE.rst: UPGRADE.rst
 
-.. _federation:
-
 Setting up Federation
 =====================
 
-Federation is the process by which users on different servers can participate
-in the same room. For this to work, those other servers must be able to contact
-yours to send messages.
+In order for other homeservers to send messages to your server, it will need to
+be publicly visible on the internet, and they will need to know its host name.
+You have two choices here, which will influence the form of your Matrix user
+IDs:
 
-As explained in `Configuring synapse`_, the ``server_name`` in your
-``homeserver.yaml`` file determines the way that other servers will reach
-yours. By default, they will treat it as a hostname and try to connect to
-port 8448. This is easy to set up and will work with the default configuration,
-provided you set the ``server_name`` to match your machine's public DNS
-hostname.
+1) Use the machine's own hostname as available on public DNS in the form of
+   its A or AAAA records. This is easier to set up initially, perhaps for
+   testing, but lacks the flexibility of SRV.
 
-For a more flexible configuration, you can set up a DNS SRV record. This allows
-you to run your server on a machine that might not have the same name as your
-domain name. For example, you might want to run your server at
-``synapse.example.com``, but have your Matrix user-ids look like
-``@user:example.com``. (A SRV record also allows you to change the port from
-the default 8448. However, if you are thinking of using a reverse-proxy on the
-federation port, which is not recommended, be sure to read
-`Reverse-proxying the federation port`_ first.)
+2) Set up a SRV record for your domain name. This requires you create a SRV
+   record in DNS, but gives the flexibility to run the server on your own
+   choice of TCP port, on a machine that might not be the same name as the
+   domain name.
 
-To use a SRV record, first create your SRV record and publish it in DNS. This
-should have the format ``_matrix._tcp.<yourdomain.com> <ttl> IN SRV 10 0 <port>
-<synapse.server.name>``. The DNS record should then look something like::
-
-    $ dig -t srv _matrix._tcp.example.com
-    _matrix._tcp.example.com. 3600    IN      SRV     10 0 8448 synapse.example.com.
-
-You can then configure your homeserver to use ``<yourdomain.com>`` as the domain in
-its user-ids, by setting ``server_name``::
+For the first form, simply pass the required hostname (of the machine) as the
+--server-name parameter::
 
     python -m synapse.app.homeserver \
-        --server-name <yourdomain.com> \
+        --server-name machine.my.domain.name \
         --config-path homeserver.yaml \
         --generate-config
     python -m synapse.app.homeserver --config-path homeserver.yaml
 
-If you've already generated the config file, you need to edit the ``server_name``
-in your ``homeserver.yaml`` file. If you've already started Synapse and a
+Alternatively, you can run ``synctl start`` to guide you through the process.
+
+For the second form, first create your SRV record and publish it in DNS. This
+needs to be named _matrix._tcp.YOURDOMAIN, and point at at least one hostname
+and port where the server is running.  (At the current time synapse does not
+support clustering multiple servers into a single logical homeserver).  The DNS
+record would then look something like::
+
+    $ dig -t srv _matrix._tcp.machine.my.domain.name
+    _matrix._tcp    IN      SRV     10 0 8448 machine.my.domain.name.
+
+
+At this point, you should then run the homeserver with the hostname of this
+SRV record, as that is the name other machines will expect it to have::
+
+    python -m synapse.app.homeserver \
+        --server-name YOURDOMAIN \
+        --config-path homeserver.yaml \
+        --generate-config
+    python -m synapse.app.homeserver --config-path homeserver.yaml
+
+
+If you've already generated the config file, you need to edit the "server_name"
+in you  ```homeserver.yaml``` file. If you've already started Synapse and a
 database has been created, you will have to recreate the database.
 
-If all goes well, you should be able to `connect to your server with a client`__,
-and then join a room via federation. (Try ``#matrix-dev:matrix.org`` as a first
-step. "Matrix HQ"'s sheer size and activity level tends to make even the
-largest boxes pause for thought.)
-
-.. __: `Connecting to Synapse from a client`_
-
-Troubleshooting
----------------
-
-You can use the federation tester to check if your homeserver is all set:
-``https://matrix.org/federationtester/api/report?server_name=<your_server_name>``
-If any of the attributes under "checks" is false, federation won't work.
-
-The typical failure mode with federation is that when you try to join a room,
-it is rejected with "401: Unauthorized". Generally this means that other
-servers in the room couldn't access yours. (Joining a room over federation is a
-complicated dance which requires connections in both directions).
-
-So, things to check are:
-
-* If you are trying to use a reverse-proxy, read `Reverse-proxying the
-  federation port`_.
-* If you are not using a SRV record, check that your ``server_name`` (the part
-  of your user-id after the ``:``) matches your hostname, and that port 8448 on
-  that hostname is reachable from outside your network.
-* If you *are* using a SRV record, check that it matches your ``server_name``
-  (it should be ``_matrix._tcp.<server_name>``), and that the port and hostname
-  it specifies are reachable from outside your network.
+You may additionally want to pass one or more "-v" options, in order to
+increase the verbosity of logging output; at least for initial testing.
 
 Running a Demo Federation of Synapses
 -------------------------------------
 
 If you want to get up and running quickly with a trio of homeservers in a
-private federation, there is a script in the ``demo`` directory. This is mainly
-useful just for development purposes. See `<demo/README>`_.
+private federation (``localhost:8080``, ``localhost:8081`` and
+``localhost:8082``) which you can then access through the webclient running at
+http://localhost:8080. Simply run::
+
+    demo/start.sh
+
+This is mainly useful just for development purposes.
+
+Running The Demo Web Client
+===========================
+
+The homeserver runs a web client by default at https://localhost:8448/.
+
+If this is the first time you have used the client from that browser (it uses
+HTML5 local storage to remember its config), you will need to log in to your
+account. If you don't yet have an account, because you've just started the
+homeserver for the first time, then you'll need to register one.
 
 
-Using PostgreSQL
-================
+Registering A New Account
+-------------------------
 
-As of Synapse 0.9, `PostgreSQL <http://www.postgresql.org>`_ is supported as an
-alternative to the `SQLite <http://sqlite.org/>`_ database that Synapse has
-traditionally used for convenience and simplicity.
+Your new user name will be formed partly from the hostname your server is
+running as, and partly from a localpart you specify when you create the
+account. Your name will take the form of::
 
-The advantages of Postgres include:
+    @localpart:my.domain.here
+         (pronounced "at localpart on my dot domain dot here")
 
-* significant performance improvements due to the superior threading and
-  caching model, smarter query optimiser
-* allowing the DB to be run on separate hardware
-* allowing basic active/backup high-availability with a "hot spare" synapse
-  pointing at the same DB master, as well as enabling DB replication in
-  synapse itself.
+Specify your desired localpart in the topmost box of the "Register for an
+account" form, and click the "Register" button. Hostnames can contain ports if
+required due to lack of SRV records (e.g. @matthew:localhost:8448 on an
+internal synapse sandbox running on localhost).
 
-For information on how to install and use PostgreSQL, please see
-`docs/postgres.rst <docs/postgres.rst>`_.
+If registration fails, you may need to enable it in the homeserver (see
+`Synapse Installation`_ above)
 
 
-.. _reverse-proxy:
-
-Using a reverse proxy with Synapse
-==================================
-
-It is recommended to put a reverse proxy such as
-`nginx <https://nginx.org/en/docs/http/ngx_http_proxy_module.html>`_,
-`Apache <https://httpd.apache.org/docs/current/mod/mod_proxy_http.html>`_ or
-`HAProxy <http://www.haproxy.org/>`_ in front of Synapse. One advantage of
-doing so is that it means that you can expose the default https port (443) to
-Matrix clients without needing to run Synapse with root privileges.
-
-The most important thing to know here is that Matrix clients and other Matrix
-servers do not necessarily need to connect to your server via the same
-port. Indeed, clients will use port 443 by default, whereas servers default to
-port 8448. Where these are different, we refer to the 'client port' and the
-'federation port'.
-
-The next most important thing to know is that using a reverse-proxy on the
-federation port has a number of pitfalls. It is possible, but be sure to read
-`Reverse-proxying the federation port`_.
-
-The recommended setup is therefore to configure your reverse-proxy on port 443
-to port 8008 of synapse for client connections, but to also directly expose port
-8448 for server-server connections. All the Matrix endpoints begin ``/_matrix``,
-so an example nginx configuration might look like::
-
-  server {
-      listen 443 ssl;
-      listen [::]:443 ssl;
-      server_name matrix.example.com;
-
-      location /_matrix {
-          proxy_pass http://localhost:8008;
-          proxy_set_header X-Forwarded-For $remote_addr;
-      }
-  }
-
-You will also want to set ``bind_addresses: ['127.0.0.1']`` and ``x_forwarded: true``
-for port 8008 in ``homeserver.yaml`` to ensure that client IP addresses are
-recorded correctly.
-
-Having done so, you can then use ``https://matrix.example.com`` (instead of
-``https://matrix.example.com:8448``) as the "Custom server" when `Connecting to
-Synapse from a client`_.
-
-Reverse-proxying the federation port
-------------------------------------
-
-There are two issues to consider before using a reverse-proxy on the federation
-port:
-
-* Due to the way SSL certificates are managed in the Matrix federation protocol
-  (see `spec`__), Synapse needs to be configured with the path to the SSL
-  certificate, *even if you do not terminate SSL at Synapse*.
-
-  .. __: `key_management`_
-
-* Synapse does not currently support SNI on the federation protocol
-  (`bug #1491 <https://github.com/matrix-org/synapse/issues/1491>`_), which
-  means that using name-based virtual hosting is unreliable.
-
-Furthermore, a number of the normal reasons for using a reverse-proxy do not
-apply:
-
-* Other servers will connect on port 8448 by default, so there is no need to
-  listen on port 443 (for federation, at least), which avoids the need for root
-  privileges and virtual hosting.
-
-* A self-signed SSL certificate is fine for federation, so there is no need to
-  automate renewals. (The certificate generated by ``--generate-config`` is
-  valid for 10 years.)
-
-If you want to set up a reverse-proxy on the federation port despite these
-caveats, you will need to do the following:
-
-* In ``homeserver.yaml``, set ``tls_certificate_path`` to the path to the SSL
-  certificate file used by your reverse-proxy, and set ``no_tls`` to ``True``.
-  (``tls_private_key_path`` will be ignored if ``no_tls`` is ``True``.)
-
-* In your reverse-proxy configuration:
-
-  * If there are other virtual hosts on the same port, make sure that the
-    *default* one uses the certificate configured above.
-
-  * Forward ``/_matrix`` to Synapse.
-
-* If your reverse-proxy is not listening on port 8448, publish a SRV record to
-  tell other servers how to find you. See `Setting up Federation`_.
-
-When updating the SSL certificate, just update the file pointed to by
-``tls_certificate_path``: there is no need to restart synapse. (You may like to
-use a symbolic link to help make this process atomic.)
-
-The most common mistake when setting up federation is not to tell Synapse about
-your SSL certificate. To check it, you can visit
-``https://matrix.org/federationtester/api/report?server_name=<your_server_name>``.
-Unfortunately, there is no UI for this yet, but, you should see
-``"MatchingTLSFingerprint": true``. If not, check that
-``Certificates[0].SHA256Fingerprint`` (the fingerprint of the certificate
-presented by your reverse-proxy) matches ``Keys.tls_fingerprints[0].sha256``
-(the fingerprint of the certificate Synapse is using).
+Logging In To An Existing Account
+---------------------------------
 
+Just enter the ``@localpart:my.domain.here`` Matrix user ID and password into
+the form and click the Login button.
 
 Identity Servers
 ================
 
-Identity servers have the job of mapping email addresses and other 3rd Party
-IDs (3PIDs) to Matrix user IDs, as well as verifying the ownership of 3PIDs
-before creating that mapping.
+The job of authenticating 3PIDs and tracking which 3PIDs are associated with a
+given Matrix user is very security-sensitive, as there is obvious risk of spam
+if it is too easy to sign up for Matrix accounts or harvest 3PID data.
+Meanwhile the job of publishing the end-to-end encryption public keys for
+Matrix users is also very security-sensitive for similar reasons.
 
-**They are not where accounts or credentials are stored - these live on home
-servers. Identity Servers are just for mapping 3rd party IDs to matrix IDs.**
+Therefore the role of managing trusted identity in the Matrix ecosystem is
+farmed out to a cluster of known trusted ecosystem partners, who run 'Matrix
+Identity Servers' such as ``sydent``, whose role is purely to authenticate and
+track 3PID logins and publish end-user public keys.
 
-This process is very security-sensitive, as there is obvious risk of spam if it
-is too easy to sign up for Matrix accounts or harvest 3PID data. In the longer
-term, we hope to create a decentralised system to manage it (`matrix-doc #712
-<https://github.com/matrix-org/matrix-doc/issues/712>`_), but in the meantime,
-the role of managing trusted identity in the Matrix ecosystem is farmed out to
-a cluster of known trusted ecosystem partners, who run 'Matrix Identity
-Servers' such as `Sydent <https://github.com/matrix-org/sydent>`_, whose role
-is purely to authenticate and track 3PID logins and publish end-user public
-keys.
-
-You can host your own copy of Sydent, but this will prevent you reaching other
-users in the Matrix ecosystem via their email address, and prevent them finding
-you. We therefore recommend that you use one of the centralised identity servers
-at ``https://matrix.org`` or ``https://vector.im`` for now.
-
-To reiterate: the Identity server will only be used if you choose to associate
-an email address with your account, or send an invite to another user via their
-email address.
+It's currently early days for identity servers as Matrix is not yet using 3PIDs
+as the primary means of identity and E2E encryption is not complete. As such,
+we are running a single identity server (https://matrix.org) at the current
+time.
 
 
 URL Previews
 ============
 
-Synapse 0.15.0 introduces a new API for previewing URLs at
-``/_matrix/media/r0/preview_url``.  This is disabled by default.  To turn it on
-you must enable the ``url_preview_enabled: True`` config parameter and
-explicitly specify the IP ranges that Synapse is not allowed to spider for
-previewing in the ``url_preview_ip_range_blacklist`` configuration parameter.
-This is critical from a security perspective to stop arbitrary Matrix users
-spidering 'internal' URLs on your network.  At the very least we recommend that
-your loopback and RFC1918 IP addresses are blacklisted.
+Synapse 0.15.0 introduces an experimental new API for previewing URLs at
+/_matrix/media/r0/preview_url.  This is disabled by default.  To turn it on
+you must enable the `url_preview_enabled: True` config parameter and explicitly
+specify the IP ranges that Synapse is not allowed to spider for previewing in
+the `url_preview_ip_range_blacklist` configuration parameter.  This is critical
+from a security perspective to stop arbitrary Matrix users spidering 'internal'
+URLs on your network.  At the very least we recommend that your loopback and
+RFC1918 IP addresses are blacklisted.
 
 This also requires the optional lxml and netaddr python dependencies to be
-installed.  This in turn requires the libxml2 library to be available - on
-Debian/Ubuntu this means ``apt-get install libxml2-dev``, or equivalent for
-your OS.
+installed.
 
 
 Password reset
@@ -845,7 +583,7 @@ server, they can request a password-reset token via clients such as Vector.
 
 A manual password reset can be done via direct database access as follows.
 
-First calculate the hash of the new password::
+First calculate the hash of the new password:
 
     $ source ~/.synapse/bin/activate
     $ ./scripts/hash_password
@@ -853,58 +591,17 @@ First calculate the hash of the new password::
     Confirm password: 
     $2a$12$xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 
-Then update the `users` table in the database::
+Then update the `users` table in the database:
 
     UPDATE users SET password_hash='$2a$12$xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
         WHERE name='@test:test.com';
 
+Where's the spec?!
+==================
 
-Synapse Development
-===================
+The source of the matrix spec lives at https://github.com/matrix-org/matrix-doc.
+A recent HTML snapshot of this lives at http://matrix.org/docs/spec
 
-Before setting up a development environment for synapse, make sure you have the
-system dependencies (such as the python header files) installed - see
-`Installing from source`_.
-
-To check out a synapse for development, clone the git repo into a working
-directory of your choice::
-
-    git clone https://github.com/matrix-org/synapse.git
-    cd synapse
-
-Synapse has a number of external dependencies, that are easiest
-to install using pip and a virtualenv::
-
-    virtualenv -p python2.7 env
-    source env/bin/activate
-    python synapse/python_dependencies.py | xargs pip install
-    pip install lxml mock
-
-This will run a process of downloading and installing all the needed
-dependencies into a virtual env.
-
-Once this is done, you may wish to run Synapse's unit tests, to
-check that everything is installed as it should be::
-
-    PYTHONPATH="." trial tests
-
-This should end with a 'PASSED' result::
-
-    Ran 143 tests in 0.601s
-
-    PASSED (successes=143)
-
-Running the Integration Tests
-=============================
-
-Synapse is accompanied by `SyTest <https://github.com/matrix-org/sytest>`_,
-a Matrix homeserver integration testing suite, which uses HTTP requests to
-access the API as a Matrix client would. It is able to run Synapse directly from
-the source tree, so installation of the server is not required.
-
-Testing with SyTest is recommended for verifying that changes related to the
-Client-Server API are functioning correctly. See the `installation instructions
-<https://github.com/matrix-org/sytest#installing>`_ for details.
 
 Building Internal API Documentation
 ===================================
@@ -920,6 +617,7 @@ Building internal API documentation::
     python setup.py build_sphinx
 
 
+
 Help!! Synapse eats all my RAM!
 ===============================
 
@@ -928,9 +626,10 @@ cache a lot of recent room data and metadata in RAM in order to speed up
 common requests.  We'll improve this in future, but for now the easiest
 way to either reduce the RAM usage (at the risk of slowing things down)
 is to set the almost-undocumented ``SYNAPSE_CACHE_FACTOR`` environment
-variable.  The default is 0.5, which can be decreased to reduce RAM usage
-in memory constrained enviroments, or increased if performance starts to
-degrade.
+variable.  Roughly speaking, a SYNAPSE_CACHE_FACTOR of 1.0 will max out
+at around 3-4GB of resident memory - this is what we currently run the
+matrix.org on.  The default setting is currently 0.1, which is probably
+around a ~700MB footprint.  You can dial it down further to 0.02 if
+desired, which targets roughly ~512MB.  Conversely you can dial it up if
+you need performance for lots of users and have a box with a lot of RAM.
 
-
-.. _`key_management`: https://matrix.org/docs/spec/server_server/unstable.html#retrieving-server-keys

UPGRADE.rst

@@ -5,25 +5,20 @@ Before upgrading check if any special steps are required to upgrade from the
 what you currently have installed to current version of synapse. The extra
 instructions that may be required are listed later in this document.
 
-1. If synapse was installed in a virtualenv then active that virtualenv before
-   upgrading. If synapse is installed in a virtualenv in ``~/.synapse/`` then
-   run:
+If synapse was installed in a virtualenv then active that virtualenv before
+upgrading. If synapse is installed in a virtualenv in ``~/.synapse/`` then run:
 
 .. code:: bash
 
     source ~/.synapse/bin/activate
 
-2. If synapse was installed using pip then upgrade to the latest version by
+If synapse was installed using pip then upgrade to the latest version by
 running:
 
 .. code:: bash
 
     pip install --upgrade --process-dependency-links https://github.com/matrix-org/synapse/tarball/master
 
-       # restart synapse
-       synctl restart
-
-
 If synapse was installed using git then upgrade to the latest version by
 running:
 
@@ -32,33 +27,8 @@ instructions that may be required are listed later in this document.
     # Pull the latest version of the master branch.
     git pull
     # Update the versions of synapse's python dependencies.
-       python synapse/python_dependencies.py | xargs pip install --upgrade
+    python synapse/python_dependencies.py | xargs -n1 pip install
 
-       # restart synapse
-       ./synctl restart
-
-
-To check whether your update was sucessful, you can check the Server header
-returned by the Client-Server API:
-
-.. code:: bash
-
-    # replace <host.name> with the hostname of your synapse homeserver.
-    # You may need to specify a port (eg, :8448) if your server is not
-    # configured on port 443.
-    curl -kv https://<host.name>/_matrix/client/versions 2>&1 | grep "Server:"
-
-Upgrading to $NEXT_VERSION
-====================
-
-This release expands the anonymous usage stats sent if the opt-in
-``report_stats`` configuration is set to ``true``. We now capture RSS memory 
-and cpu use at a very coarse level. This requires administrators to install
-the optional ``psutil`` python module.
-
-We would appreciate it if you could assist by ensuring this module is available
-and ``report_stats`` is enabled. This will let us see if performance changes to
-synapse are having an impact to the general community.
 
 Upgrading to v0.15.0
 ====================

contrib/cmdclient/console.py

@@ -32,7 +32,7 @@ import urlparse
 import nacl.signing
 import nacl.encoding
 
-from signedjson.sign import verify_signed_json, SignatureVerifyException
+from syutil.crypto.jsonsign import verify_signed_json, SignatureVerifyException
 
 CONFIG_JSON = "cmdclient_config.json"
 

contrib/cmdclient/http.py

@@ -36,13 +36,15 @@ class HttpClient(object):
                 the request body. This will be encoded as JSON.
 
         Returns:
-            Deferred: Succeeds when we get a 2xx HTTP response. The result
-            will be the decoded JSON body.
+            Deferred: Succeeds when we get *any* HTTP response.
+
+            The result of the deferred is a tuple of `(code, response)`,
+            where `response` is a dict representing the decoded JSON body.
         """
         pass
 
     def get_json(self, url, args=None):
-        """ Gets some json from the given host homeserver and path
+        """ Get's some json from the given host homeserver and path
 
         Args:
             url (str): The URL to GET data from.
@@ -52,8 +54,10 @@ class HttpClient(object):
                 and *not* a string.
 
         Returns:
-            Deferred: Succeeds when we get a 2xx HTTP response. The result
-            will be the decoded JSON body.
+            Deferred: Succeeds when we get *any* HTTP response.
+
+            The result of the deferred is a tuple of `(code, response)`,
+            where `response` is a dict representing the decoded JSON body.
         """
         pass
 

contrib/example_log_config.yaml

@@ -1,50 +0,0 @@
-# Example log_config file for synapse. To enable, point `log_config` to it in 
-# `homeserver.yaml`, and restart synapse.
-#
-# This configuration will produce similar results to the defaults within 
-# synapse, but can be edited to give more flexibility.
-
-version: 1
-
-formatters:
-  fmt:
-    format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s- %(message)s'
-
-filters:
-  context:
-    (): synapse.util.logcontext.LoggingContextFilter
-    request: ""
-
-handlers:
-  # example output to console
-  console:
-    class: logging.StreamHandler
-    filters: [context]
-
-  # example output to file - to enable, edit 'root' config below.
-  file:
-    class: logging.handlers.RotatingFileHandler
-    formatter: fmt
-    filename: /var/log/synapse/homeserver.log
-    maxBytes: 100000000
-    backupCount: 3
-    filters: [context]
-
-
-root:
-    level: INFO
-    handlers: [console] # to use file handler instead, switch to [file]
-    
-loggers:
-    synapse:
-        level: INFO
-
-    synapse.storage.SQL:
-        # beware: increasing this to DEBUG will make synapse log sensitive
-        # information such as access tokens.
-        level: INFO
-
-    # example of enabling debugging for a component:
-    #
-    # synapse.federation.transport.server:
-    #    level: DEBUG

contrib/graph/graph3.py

@@ -22,8 +22,6 @@ import argparse
 from synapse.events import FrozenEvent
 from synapse.util.frozenutils import unfreeze
 
-from six import string_types
-
 
 def make_graph(file_name, room_id, file_prefix, limit):
     print "Reading lines"
@@ -60,7 +58,7 @@ def make_graph(file_name, room_id, file_prefix, limit):
         for key, value in unfreeze(event.get_dict()["content"]).items():
             if value is None:
                 value = "<null>"
-            elif isinstance(value, string_types):
+            elif isinstance(value, basestring):
                 pass
             else:
                 value = json.dumps(value)

contrib/prometheus/README

@@ -1,37 +0,0 @@
-This directory contains some sample monitoring config for using the
-'Prometheus' monitoring server against synapse.
-
-To use it, first install prometheus by following the instructions at
-
-  http://prometheus.io/
-
-### for Prometheus v1
-Add a new job to the main prometheus.conf file:
-
-  job: {
-    name: "synapse"
-
-    target_group: {
-      target: "http://SERVER.LOCATION.HERE:PORT/_synapse/metrics"
-    }
-  }
-
-### for Prometheus v2
-Add a new job to the main prometheus.yml file:
-
-  - job_name: "synapse"
-    metrics_path: "/_synapse/metrics"
-    # when endpoint uses https:
-    scheme: "https"
-
-    static_configs:
-    - targets: ['SERVER.LOCATION:PORT']
-
-To use `synapse.rules` add
-
-    rule_files:
-      - "/PATH/TO/synapse-v2.rules"
-
-Metrics are disabled by default when running synapse; they must be enabled
-with the 'enable-metrics' option, either in the synapse config file or as a
-command-line option.

contrib/prometheus/consoles/synapse.html

@@ -1,395 +0,0 @@
-{{ template "head" . }}
-
-{{ template "prom_content_head" . }}
-<h1>System Resources</h1>
-
-<h3>CPU</h3>
-<div id="process_resource_utime"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#process_resource_utime"),
-  expr: "rate(process_cpu_seconds_total[2m]) * 100",
-  name: "[[job]]",  
-  min: 0,
-  max: 100,
-  renderer: "line",
-  height: 150,
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "%",
-  yTitle: "CPU Usage"
-})
-</script>
-
-<h3>Memory</h3>
-<div id="process_resource_maxrss"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#process_resource_maxrss"),
-  expr: "process_psutil_rss:max",
-  name: "Maxrss",
-  min: 0,
-  renderer: "line",
-  height: 150,
-  yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yUnits: "bytes",
-  yTitle: "Usage"
-})
-</script>
-
-<h3>File descriptors</h3>
-<div id="process_fds"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#process_fds"),
-  expr: "process_open_fds{job='synapse'}",
-  name: "FDs",
-  min: 0,
-  renderer: "line",
-  height: 150,
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "",
-  yTitle: "Descriptors"
-})
-</script>
-
-<h1>Reactor</h1>
-
-<h3>Total reactor time</h3>
-<div id="reactor_total_time"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#reactor_total_time"),
-  expr: "rate(python_twisted_reactor_tick_time:total[2m]) / 1000",
-  name: "time",
-  max: 1,
-  min: 0,
-  renderer: "area",
-  height: 150,
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "s/s",
-  yTitle: "Usage"
-})
-</script>
-
-<h3>Average reactor tick time</h3>
-<div id="reactor_average_time"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#reactor_average_time"),
-  expr: "rate(python_twisted_reactor_tick_time:total[2m]) / rate(python_twisted_reactor_tick_time:count[2m]) / 1000",
-  name: "time",
-  min: 0,
-  renderer: "line",
-  height: 150,
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "s",
-  yTitle: "Time"
-})
-</script>
-
-<h3>Pending calls per tick</h3>
-<div id="reactor_pending_calls"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#reactor_pending_calls"),
-  expr: "rate(python_twisted_reactor_pending_calls:total[30s])/rate(python_twisted_reactor_pending_calls:count[30s])",
-  name: "calls",
-  min: 0,
-  renderer: "line",
-  height: 150,
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yTitle: "Pending Cals"
-})
-</script>
-
-<h1>Storage</h1>
-
-<h3>Queries</h3>
-<div id="synapse_storage_query_time"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_storage_query_time"),
-  expr: "rate(synapse_storage_query_time:count[2m])",
-  name: "[[verb]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yUnits: "queries/s",
-  yTitle: "Queries"
-})
-</script>
-
-<h3>Transactions</h3>
-<div id="synapse_storage_transaction_time"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_storage_transaction_time"),
-  expr: "rate(synapse_storage_transaction_time:count[2m])",
-  name: "[[desc]]",
-  min: 0,
-  yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yUnits: "txn/s",
-  yTitle: "Transactions"
-})
-</script>
-
-<h3>Transaction execution time</h3>
-<div id="synapse_storage_transactions_time_msec"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_storage_transactions_time_msec"),
-  expr: "rate(synapse_storage_transaction_time:total[2m]) / 1000",
-  name: "[[desc]]",
-  min: 0,
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "s/s",
-  yTitle: "Usage"
-})
-</script>
-
-<h3>Database scheduling latency</h3>
-<div id="synapse_storage_schedule_time"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_storage_schedule_time"),
-  expr: "rate(synapse_storage_schedule_time:total[2m]) / 1000",
-  name: "Total latency",
-  min: 0,
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "s/s",
-  yTitle: "Usage"
-})
-</script>
-
-<h3>Cache hit ratio</h3>
-<div id="synapse_cache_ratio"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_cache_ratio"),
-  expr: "rate(synapse_util_caches_cache:total[2m]) * 100",
-  name: "[[name]]",
-  min: 0,
-  max: 100,
-  yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yUnits: "%",
-  yTitle: "Percentage"
-})
-</script>
-
-<h3>Cache size</h3>
-<div id="synapse_cache_size"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_cache_size"),
-  expr: "synapse_util_caches_cache:size",
-  name: "[[name]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yUnits: "",
-  yTitle: "Items"
-})
-</script>
-
-<h1>Requests</h1>
-
-<h3>Requests by Servlet</h3>
-<div id="synapse_http_server_request_count_servlet"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_http_server_request_count_servlet"),
-  expr: "rate(synapse_http_server_request_count:servlet[2m])",
-  name: "[[servlet]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "req/s",
-  yTitle: "Requests"
-})
-</script>
-<h4>&nbsp;(without <tt>EventStreamRestServlet</tt> or <tt>SyncRestServlet</tt>)</h4>
-<div id="synapse_http_server_request_count_servlet_minus_events"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_http_server_request_count_servlet_minus_events"),
-  expr: "rate(synapse_http_server_request_count:servlet{servlet!=\"EventStreamRestServlet\", servlet!=\"SyncRestServlet\"}[2m])",
-  name: "[[servlet]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "req/s",
-  yTitle: "Requests"
-})
-</script>
-
-<h3>Average response times</h3>
-<div id="synapse_http_server_response_time_avg"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_http_server_response_time_avg"),
-  expr: "rate(synapse_http_server_response_time_seconds[2m]) / rate(synapse_http_server_response_count[2m]) / 1000",
-  name: "[[servlet]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "s/req",
-  yTitle: "Response time"
-})
-</script>
-
-<h3>All responses by code</h3>
-<div id="synapse_http_server_responses"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_http_server_responses"),
-  expr: "rate(synapse_http_server_responses[2m])",
-  name: "[[method]] / [[code]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "req/s",
-  yTitle: "Requests"
-})
-</script>
-
-<h3>Error responses by code</h3>
-<div id="synapse_http_server_responses_err"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_http_server_responses_err"),
-  expr: "rate(synapse_http_server_responses{code=~\"[45]..\"}[2m])",
-  name: "[[method]] / [[code]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "req/s",
-  yTitle: "Requests"
-})
-</script>
-
-
-<h3>CPU Usage</h3>
-<div id="synapse_http_server_response_ru_utime"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_http_server_response_ru_utime"),
-  expr: "rate(synapse_http_server_response_ru_utime_seconds[2m])",
-  name: "[[servlet]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "s/s",
-  yTitle: "CPU Usage"
-})
-</script>
-
-
-<h3>DB Usage</h3>
-<div id="synapse_http_server_response_db_txn_duration"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_http_server_response_db_txn_duration"),
-  expr: "rate(synapse_http_server_response_db_txn_duration_seconds[2m])",
-  name: "[[servlet]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "s/s",
-  yTitle: "DB Usage"
-})
-</script>
-
-
-<h3>Average event send times</h3>
-<div id="synapse_http_server_send_time_avg"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_http_server_send_time_avg"),
-  expr: "rate(synapse_http_server_response_time_second{servlet='RoomSendEventRestServlet'}[2m]) / rate(synapse_http_server_response_count{servlet='RoomSendEventRestServlet'}[2m]) / 1000",
-  name: "[[servlet]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "s/req",
-  yTitle: "Response time"
-})
-</script>
-
-<h1>Federation</h1>
-
-<h3>Sent Messages</h3>
-<div id="synapse_federation_client_sent"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_federation_client_sent"),
-  expr: "rate(synapse_federation_client_sent[2m])",
-  name: "[[type]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "req/s",
-  yTitle: "Requests"
-})
-</script>
-
-<h3>Received Messages</h3>
-<div id="synapse_federation_server_received"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_federation_server_received"),
-  expr: "rate(synapse_federation_server_received[2m])",
-  name: "[[type]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "req/s",
-  yTitle: "Requests"
-})
-</script>
-
-<h3>Pending</h3>
-<div id="synapse_federation_transaction_queue_pending"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_federation_transaction_queue_pending"),
-  expr: "synapse_federation_transaction_queue_pending",
-  name: "[[type]]",
-  yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yUnits: "",
-  yTitle: "Units"
-})
-</script>
-
-<h1>Clients</h1>
-
-<h3>Notifiers</h3>
-<div id="synapse_notifier_listeners"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_notifier_listeners"),
-  expr: "synapse_notifier_listeners",
-  name: "listeners",
-  min: 0,
-  yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
-  yUnits: "",
-  yTitle: "Listeners"
-})
-</script>
-
-<h3>Notified Events</h3>
-<div id="synapse_notifier_notified_events"></div>
-<script>
-new PromConsole.Graph({
-  node: document.querySelector("#synapse_notifier_notified_events"),
-  expr: "rate(synapse_notifier_notified_events[2m])",
-  name: "events",
-  yAxisFormatter: PromConsole.NumberFormatter.humanize,
-  yHoverFormatter: PromConsole.NumberFormatter.humanize,
-  yUnits: "events/s",
-  yTitle: "Event rate"
-})
-</script>
-
-{{ template "prom_content_tail" . }}
-
-{{ template "tail" }}

contrib/prometheus/synapse-v1.rules

@@ -1,21 +0,0 @@
-synapse_federation_transaction_queue_pendingEdus:total = sum(synapse_federation_transaction_queue_pendingEdus or absent(synapse_federation_transaction_queue_pendingEdus)*0)
-synapse_federation_transaction_queue_pendingPdus:total = sum(synapse_federation_transaction_queue_pendingPdus or absent(synapse_federation_transaction_queue_pendingPdus)*0)
-
-synapse_http_server_request_count:method{servlet=""} = sum(synapse_http_server_request_count) by (method)
-synapse_http_server_request_count:servlet{method=""} = sum(synapse_http_server_request_count) by (servlet)
-
-synapse_http_server_request_count:total{servlet=""} = sum(synapse_http_server_request_count:by_method) by (servlet)
-
-synapse_cache:hit_ratio_5m = rate(synapse_util_caches_cache:hits[5m]) / rate(synapse_util_caches_cache:total[5m])
-synapse_cache:hit_ratio_30s = rate(synapse_util_caches_cache:hits[30s]) / rate(synapse_util_caches_cache:total[30s])
-
-synapse_federation_client_sent{type="EDU"} = synapse_federation_client_sent_edus + 0
-synapse_federation_client_sent{type="PDU"} = synapse_federation_client_sent_pdu_destinations:count + 0
-synapse_federation_client_sent{type="Query"} = sum(synapse_federation_client_sent_queries) by (job)
-
-synapse_federation_server_received{type="EDU"} = synapse_federation_server_received_edus + 0
-synapse_federation_server_received{type="PDU"} = synapse_federation_server_received_pdus + 0
-synapse_federation_server_received{type="Query"} = sum(synapse_federation_server_received_queries) by (job)
-
-synapse_federation_transaction_queue_pending{type="EDU"} = synapse_federation_transaction_queue_pending_edus + 0
-synapse_federation_transaction_queue_pending{type="PDU"} = synapse_federation_transaction_queue_pending_pdus + 0

contrib/prometheus/synapse-v2.rules

@@ -1,60 +0,0 @@
-groups:
-- name: synapse
-  rules:
-  - record: "synapse_federation_transaction_queue_pendingEdus:total"
-    expr: "sum(synapse_federation_transaction_queue_pendingEdus or absent(synapse_federation_transaction_queue_pendingEdus)*0)"
-  - record: "synapse_federation_transaction_queue_pendingPdus:total"
-    expr:   "sum(synapse_federation_transaction_queue_pendingPdus or absent(synapse_federation_transaction_queue_pendingPdus)*0)"
-  - record: 'synapse_http_server_request_count:method'
-    labels:
-      servlet: ""
-    expr: "sum(synapse_http_server_request_count) by (method)"
-  - record: 'synapse_http_server_request_count:servlet'
-    labels:
-      method: ""
-    expr: 'sum(synapse_http_server_request_count) by (servlet)'
-
-  - record: 'synapse_http_server_request_count:total'
-    labels:
-      servlet: ""
-    expr: 'sum(synapse_http_server_request_count:by_method) by (servlet)'
-
-  - record: 'synapse_cache:hit_ratio_5m'
-    expr: 'rate(synapse_util_caches_cache:hits[5m]) / rate(synapse_util_caches_cache:total[5m])'
-  - record: 'synapse_cache:hit_ratio_30s'
-    expr: 'rate(synapse_util_caches_cache:hits[30s]) / rate(synapse_util_caches_cache:total[30s])'
-
-  - record: 'synapse_federation_client_sent'
-    labels:
-      type: "EDU"
-    expr: 'synapse_federation_client_sent_edus + 0'
-  - record: 'synapse_federation_client_sent'
-    labels:
-      type: "PDU"
-    expr: 'synapse_federation_client_sent_pdu_destinations:count + 0'
-  - record: 'synapse_federation_client_sent'
-    labels:
-      type: "Query"
-    expr: 'sum(synapse_federation_client_sent_queries) by (job)'
-
-  - record: 'synapse_federation_server_received'
-    labels:
-      type: "EDU"
-    expr: 'synapse_federation_server_received_edus + 0'
-  - record: 'synapse_federation_server_received'
-    labels:
-      type: "PDU"
-    expr: 'synapse_federation_server_received_pdus + 0'
-  - record: 'synapse_federation_server_received'
-    labels:
-      type: "Query"
-    expr: 'sum(synapse_federation_server_received_queries) by (job)'
-
-  - record: 'synapse_federation_transaction_queue_pending'
-    labels:
-      type: "EDU"
-    expr: 'synapse_federation_transaction_queue_pending_edus + 0'
-  - record: 'synapse_federation_transaction_queue_pending'
-    labels:
-      type: "PDU"
-    expr: 'synapse_federation_transaction_queue_pending_pdus + 0'

contrib/systemd/synapse.service

@@ -1,5 +1,5 @@
 # This assumes that Synapse has been installed as a system package
-# (e.g. https://www.archlinux.org/packages/community/any/matrix-synapse/ for ArchLinux)
+# (e.g. https://aur.archlinux.org/packages/matrix-synapse/ for ArchLinux)
 # rather than in a user home directory or similar under virtualenv.
 
 [Unit]
@@ -9,10 +9,9 @@ Description=Synapse Matrix homeserver
 Type=simple
 User=synapse
 Group=synapse
+EnvironmentFile=-/etc/sysconfig/synapse
 WorkingDirectory=/var/lib/synapse
-ExecStart=/usr/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/synapse/homeserver.yaml
-ExecStop=/usr/bin/synctl stop /etc/synapse/homeserver.yaml
+ExecStart=/usr/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/synapse/homeserver.yaml --log-config=/etc/synapse/log_config.yaml
 
 [Install]
 WantedBy=multi-user.target
-

docs/CAPTCHA_SETUP

@@ -11,12 +11,12 @@ https://developers.google.com/recaptcha/
 Setting ReCaptcha Keys
 ----------------------
 The keys are a config option on the home server config. If they are not 
-visible, you can generate them via --generate-config. Set the following value::
+visible, you can generate them via --generate-config. Set the following value:
 
   recaptcha_public_key: YOUR_PUBLIC_KEY
   recaptcha_private_key: YOUR_PRIVATE_KEY
   
-In addition, you MUST enable captchas via::
+In addition, you MUST enable captchas via:
 
   enable_registration_captcha: true
 
@@ -25,5 +25,7 @@ Configuring IP used for auth
 The ReCaptcha API requires that the IP address of the user who solved the
 captcha is sent. If the client is connecting through a proxy or load balancer,
 it may be required to use the X-Forwarded-For (XFF) header instead of the origin
-IP address. This can be configured using the x_forwarded directive in the
-listeners section of the homeserver.yaml configuration file.
+IP address. This can be configured as an option on the home server like so:
+
+  captcha_ip_origin_is_x_forwarded: true
+

docs/admin_api/README.rst

@@ -1,12 +0,0 @@
-Admin APIs
-==========
-
-This directory includes documentation for the various synapse specific admin
-APIs available.
-
-Only users that are server admins can use these APIs. A user can be marked as a
-server admin by updating the database directly, e.g.:
-
-``UPDATE users SET admin = 1 WHERE name = '@foo:bar.com'``
-
-Restarting may be required for the changes to register.

docs/admin_api/media_admin_api.md

@@ -1,23 +0,0 @@
-# List all media in a room
-
-This API gets a list of known media in a room.
-
-The API is:
-```
-GET /_matrix/client/r0/admin/room/<room_id>/media
-```
-including an `access_token` of a server admin.
-
-It returns a JSON body like the following:
-```
-{
-    "local": [
-        "mxc://localhost/xwvutsrqponmlkjihgfedcba",
-        "mxc://localhost/abcdefghijklmnopqrstuvwx"
-    ],
-    "remote": [
-        "mxc://matrix.org/xwvutsrqponmlkjihgfedcba",
-        "mxc://matrix.org/abcdefghijklmnopqrstuvwx"
-    ]
-}
-```

docs/admin_api/purge_history_api.rst

@@ -1,63 +0,0 @@
-Purge History API
-=================
-
-The purge history API allows server admins to purge historic events from their
-database, reclaiming disk space.
-
-Depending on the amount of history being purged a call to the API may take
-several minutes or longer. During this period users will not be able to
-paginate further back in the room from the point being purged from.
-
-The API is:
-
-``POST /_matrix/client/r0/admin/purge_history/<room_id>[/<event_id>]``
-
-including an ``access_token`` of a server admin.
-
-By default, events sent by local users are not deleted, as they may represent
-the only copies of this content in existence. (Events sent by remote users are
-deleted.)
-
-Room state data (such as joins, leaves, topic) is always preserved.
-
-To delete local message events as well, set ``delete_local_events`` in the body:
-
-.. code:: json
-
-   {
-       "delete_local_events": true
-   }
-
-The caller must specify the point in the room to purge up to. This can be
-specified by including an event_id in the URI, or by setting a
-``purge_up_to_event_id`` or ``purge_up_to_ts`` in the request body. If an event
-id is given, that event (and others at the same graph depth) will be retained.
-If ``purge_up_to_ts`` is given, it should be a timestamp since the unix epoch,
-in milliseconds.
-
-The API starts the purge running, and returns immediately with a JSON body with
-a purge id:
-
-.. code:: json
-
-    {
-        "purge_id": "<opaque id>"
-    }
-
-Purge status query
-------------------
-
-It is possible to poll for updates on recent purges with a second API;
-
-``GET /_matrix/client/r0/admin/purge_history_status/<purge_id>``
-
-(again, with a suitable ``access_token``). This API returns a JSON body like
-the following:
-
-.. code:: json
-
-    {
-        "status": "active"
-    }
-
-The status will be one of ``active``, ``complete``, or ``failed``.

docs/admin_api/purge_remote_media.rst

@@ -1,17 +0,0 @@
-Purge Remote Media API
-======================
-
-The purge remote media API allows server admins to purge old cached remote
-media.
-
-The API is::
-
-    POST /_matrix/client/r0/admin/purge_media_cache?before_ts=<unix_timestamp_in_ms>&access_token=<access_token>
-
-    {}
-
-Which will remove all cached media that was last accessed before
-``<unix_timestamp_in_ms>``.
-
-If the user re-requests purged remote media, synapse will re-request the media
-from the originating server.

docs/admin_api/user_admin_api.rst

@@ -1,73 +0,0 @@
-Query Account
-=============
-
-This API returns information about a specific user account.
-
-The api is::
-
-    GET /_matrix/client/r0/admin/whois/<user_id>
-
-including an ``access_token`` of a server admin.
-
-It returns a JSON body like the following:
-
-.. code:: json
-
-    {
-        "user_id": "<user_id>",
-        "devices": {
-            "": {
-                "sessions": [
-                    {
-                        "connections": [
-                            {
-                                "ip": "1.2.3.4",
-                                "last_seen": 1417222374433,
-                                "user_agent": "Mozilla/5.0 ..."
-                            },
-                            {
-                                "ip": "1.2.3.10",
-                                "last_seen": 1417222374500,
-                                "user_agent": "Dalvik/2.1.0 ..."
-                            }
-                        ]
-                    }
-                ]
-            }
-        }
-    }
-
-``last_seen`` is measured in milliseconds since the Unix epoch.
-
-Deactivate Account
-==================
-
-This API deactivates an account. It removes active access tokens, resets the
-password, and deletes third-party IDs (to prevent the user requesting a
-password reset).
-
-The api is::
-
-    POST /_matrix/client/r0/admin/deactivate/<user_id>
-
-including an ``access_token`` of a server admin, and an empty request body.
-
-
-Reset password
-==============
-
-Changes the password of another user.
-
-The api is::
-
-    POST /_matrix/client/r0/admin/reset_password/<user_id>
-
-with a body of:
-
-.. code:: json
-
-   {
-       "new_password": "<secret>"
-   }
-
-including an ``access_token`` of a server admin.

docs/code_style.rst

@@ -1,13 +1,26 @@
-- Everything should comply with PEP8. Code should pass
-  ``pep8 --max-line-length=100`` without any warnings.
-
-- **Indenting**:
+Basically, PEP8
 
 - NEVER tabs. 4 spaces to indent.
-
-  - follow PEP8; either hanging indent or multiline-visual indent depending
-    on the size and shape of the arguments and what makes more sense to the
-    author. In other words, both this::
+- Max line width: 79 chars (with flexibility to overflow by a "few chars" if
+  the overflowing content is not semantically significant and avoids an
+  explosion of vertical whitespace).
+- Use camel case for class and type names
+- Use underscores for functions and variables.
+- Use double quotes.
+- Use parentheses instead of '\\' for line continuation where ever possible
+  (which is pretty much everywhere)
+- There should be max a single new line between:
+    - statements
+    - functions in a class
+- There should be two new lines between:
+    - definitions in a module (e.g., between different classes)
+- There should be spaces where spaces should be and not where there shouldn't be:
+    - a single space after a comma
+    - a single space before and after for '=' when used as assignment
+    - no spaces before and after for '=' for default values and keyword arguments.
+- Indenting must follow PEP8; either hanging indent or multiline-visual indent
+  depending on the size and shape of the arguments and what makes more sense to
+  the author. In other words, both this::
 
     print("I am a fish %s" % "moo")
 
@@ -20,100 +33,20 @@
 
     print(
         "I am a fish %s" %
-            "moo",
+        "moo"
     )
 
   ...are valid, although given each one takes up 2x more vertical space than
-    the previous, it's up to the author's discretion as to which layout makes
-    most sense for their function invocation.  (e.g. if they want to add
-    comments per-argument, or put expressions in the arguments, or group
-    related arguments together, or want to deliberately extend or preserve
-    vertical/horizontal space)
+  the previous, it's up to the author's discretion as to which layout makes most
+  sense for their function invocation.  (e.g. if they want to add comments
+  per-argument, or put expressions in the arguments, or group related arguments
+  together, or want to deliberately extend or preserve vertical/horizontal
+  space)
 
-- **Line length**:
-
-  Max line length is 79 chars (with flexibility to overflow by a "few chars" if
-  the overflowing content is not semantically significant and avoids an
-  explosion of vertical whitespace).
-
-  Use parentheses instead of ``\`` for line continuation where ever possible
-  (which is pretty much everywhere).
-
-- **Naming**:
-
-  - Use camel case for class and type names
-  - Use underscores for functions and variables.
-
-- Use double quotes ``"foo"`` rather than single quotes ``'foo'``.
-
-- **Blank lines**:
-
-  - There should be max a single new line between:
-
-    - statements
-    - functions in a class
-
-  - There should be two new lines between:
-
-    - definitions in a module (e.g., between different classes)
-
-- **Whitespace**:
-
-  There should be spaces where spaces should be and not where there shouldn't
-  be:
-
-  - a single space after a comma
-  - a single space before and after for '=' when used as assignment
-  - no spaces before and after for '=' for default values and keyword arguments.
-
-- **Comments**: should follow the `google code style
-  <http://google.github.io/styleguide/pyguide.html?showone=Comments#Comments>`_.
-  This is so that we can generate documentation with `sphinx
-  <http://sphinxcontrib-napoleon.readthedocs.org/en/latest/>`_. See the
-  `examples
-  <http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html>`_
+Comments should follow the `google code style <http://google.github.io/styleguide/pyguide.html?showone=Comments#Comments>`_.
+This is so that we can generate documentation with 
+`sphinx <http://sphinxcontrib-napoleon.readthedocs.org/en/latest/>`_. See the
+`examples <http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html>`_
 in the sphinx documentation.
 
-- **Imports**:
-
-  - Prefer to import classes and functions than packages or modules.
-
-    Example::
-
-      from synapse.types import UserID
-      ...
-      user_id = UserID(local, server)
-
-    is preferred over::
-
-      from synapse import types
-      ...
-      user_id = types.UserID(local, server)
-
-    (or any other variant).
-
-    This goes against the advice in the Google style guide, but it means that
-    errors in the name are caught early (at import time).
-
-  - Multiple imports from the same package can be combined onto one line::
-
-      from synapse.types import GroupID, RoomID, UserID
-
-    An effort should be made to keep the individual imports in alphabetical
-    order.
-
-    If the list becomes long, wrap it with parentheses and split it over
-    multiple lines.
-
-  - As per `PEP-8 <https://www.python.org/dev/peps/pep-0008/#imports>`_,
-    imports should be grouped in the following order, with a blank line between
-    each group:
-
-    1. standard library imports
-    2. related third party imports
-    3. local application/library specific imports
-
-  - Imports within each group should be sorted alphabetically by module name.
-
-  - Avoid wildcard imports (``from synapse.types import *``) and relative
-    imports (``from .types import UserID``).
+Code should pass pep8 --max-line-length=100 without any warnings.

docs/log_contexts.rst

@@ -1,442 +1,10 @@
-Log contexts
-============
+What do I do about "Unexpected logging context" debug log-lines everywhere?
 
-.. contents::
+<Mjark> The logging context lives in thread local storage
+<Mjark> Sometimes it gets out of sync with what it should actually be, usually because something scheduled something to run on the reactor without preserving the logging context. 
+<Matthew> what is the impact of it getting out of sync? and how and when should we preserve log context?
+<Mjark> The impact is that some of the CPU and database metrics will be under-reported, and some log lines will be mis-attributed.
+<Mjark> It should happen auto-magically in all the APIs that do IO or otherwise defer to the reactor.
+<Erik> Mjark: the other place is if we branch, e.g. using defer.gatherResults
 
-To help track the processing of individual requests, synapse uses a
-'log context' to track which request it is handling at any given moment. This
-is done via a thread-local variable; a ``logging.Filter`` is then used to fish
-the information back out of the thread-local variable and add it to each log
-record.
-
-Logcontexts are also used for CPU and database accounting, so that we can track
-which requests were responsible for high CPU use or database activity.
-
-The ``synapse.util.logcontext`` module provides a facilities for managing the
-current log context (as well as providing the ``LoggingContextFilter`` class).
-
-Deferreds make the whole thing complicated, so this document describes how it
-all works, and how to write code which follows the rules.
-
-Logcontexts without Deferreds
------------------------------
-
-In the absence of any Deferred voodoo, things are simple enough. As with any
-code of this nature, the rule is that our function should leave things as it
-found them:
-
-.. code:: python
-
-    from synapse.util import logcontext         # omitted from future snippets
-
-    def handle_request(request_id):
-        request_context = logcontext.LoggingContext()
-
-        calling_context = logcontext.LoggingContext.current_context()
-        logcontext.LoggingContext.set_current_context(request_context)
-        try:
-            request_context.request = request_id
-            do_request_handling()
-            logger.debug("finished")
-        finally:
-            logcontext.LoggingContext.set_current_context(calling_context)
-
-    def do_request_handling():
-        logger.debug("phew")  # this will be logged against request_id
-
-
-LoggingContext implements the context management methods, so the above can be
-written much more succinctly as:
-
-.. code:: python
-
-    def handle_request(request_id):
-        with logcontext.LoggingContext() as request_context:
-            request_context.request = request_id
-            do_request_handling()
-            logger.debug("finished")
-
-    def do_request_handling():
-        logger.debug("phew")
-
-
-Using logcontexts with Deferreds
---------------------------------
-
-Deferreds — and in particular, ``defer.inlineCallbacks`` — break
-the linear flow of code so that there is no longer a single entry point where
-we should set the logcontext and a single exit point where we should remove it.
-
-Consider the example above, where ``do_request_handling`` needs to do some
-blocking operation, and returns a deferred:
-
-.. code:: python
-
-    @defer.inlineCallbacks
-    def handle_request(request_id):
-        with logcontext.LoggingContext() as request_context:
-            request_context.request = request_id
-            yield do_request_handling()
-            logger.debug("finished")
-
-
-In the above flow:
-
-* The logcontext is set
-* ``do_request_handling`` is called, and returns a deferred
-* ``handle_request`` yields the deferred
-* The ``inlineCallbacks`` wrapper of ``handle_request`` returns a deferred
-
-So we have stopped processing the request (and will probably go on to start
-processing the next), without clearing the logcontext.
-
-To circumvent this problem, synapse code assumes that, wherever you have a
-deferred, you will want to yield on it. To that end, whereever functions return
-a deferred, we adopt the following conventions:
-
-**Rules for functions returning deferreds:**
-
-  * If the deferred is already complete, the function returns with the same
-    logcontext it started with.
-  * If the deferred is incomplete, the function clears the logcontext before
-    returning; when the deferred completes, it restores the logcontext before
-    running any callbacks.
-
-That sounds complicated, but actually it means a lot of code (including the
-example above) "just works". There are two cases:
-
-* If ``do_request_handling`` returns a completed deferred, then the logcontext
-  will still be in place. In this case, execution will continue immediately
-  after the ``yield``; the "finished" line will be logged against the right
-  context, and the ``with`` block restores the original context before we
-  return to the caller.
-
-* If the returned deferred is incomplete, ``do_request_handling`` clears the
-  logcontext before returning. The logcontext is therefore clear when
-  ``handle_request`` yields the deferred. At that point, the ``inlineCallbacks``
-  wrapper adds a callback to the deferred, and returns another (incomplete)
-  deferred to the caller, and it is safe to begin processing the next request.
-
-  Once ``do_request_handling``'s deferred completes, it will reinstate the
-  logcontext, before running the callback added by the ``inlineCallbacks``
-  wrapper. That callback runs the second half of ``handle_request``, so again
-  the "finished" line will be logged against the right
-  context, and the ``with`` block restores the original context.
-
-As an aside, it's worth noting that ``handle_request`` follows our rules -
-though that only matters if the caller has its own logcontext which it cares
-about.
-
-The following sections describe pitfalls and helpful patterns when implementing
-these rules.
-
-Always yield your deferreds
----------------------------
-
-Whenever you get a deferred back from a function, you should ``yield`` on it
-as soon as possible. (Returning it directly to your caller is ok too, if you're
-not doing ``inlineCallbacks``.) Do not pass go; do not do any logging; do not
-call any other functions.
-
-.. code:: python
-
-    @defer.inlineCallbacks
-    def fun():
-        logger.debug("starting")
-        yield do_some_stuff()       # just like this
-
-        d = more_stuff()
-        result = yield d            # also fine, of course
-
-        defer.returnValue(result)
-
-    def nonInlineCallbacksFun():
-        logger.debug("just a wrapper really")
-        return do_some_stuff()      # this is ok too - the caller will yield on
-                                    # it anyway.
-
-Provided this pattern is followed all the way back up to the callchain to where
-the logcontext was set, this will make things work out ok: provided
-``do_some_stuff`` and ``more_stuff`` follow the rules above, then so will
-``fun`` (as wrapped by ``inlineCallbacks``) and ``nonInlineCallbacksFun``.
-
-It's all too easy to forget to ``yield``: for instance if we forgot that
-``do_some_stuff`` returned a deferred, we might plough on regardless. This
-leads to a mess; it will probably work itself out eventually, but not before
-a load of stuff has been logged against the wrong content. (Normally, other
-things will break, more obviously, if you forget to ``yield``, so this tends
-not to be a major problem in practice.)
-
-Of course sometimes you need to do something a bit fancier with your Deferreds
-- not all code follows the linear A-then-B-then-C pattern. Notes on
-implementing more complex patterns are in later sections.
-
-Where you create a new Deferred, make it follow the rules
----------------------------------------------------------
-
-Most of the time, a Deferred comes from another synapse function. Sometimes,
-though, we need to make up a new Deferred, or we get a Deferred back from
-external code. We need to make it follow our rules.
-
-The easy way to do it is with a combination of ``defer.inlineCallbacks``, and
-``logcontext.PreserveLoggingContext``. Suppose we want to implement ``sleep``,
-which returns a deferred which will run its callbacks after a given number of
-seconds. That might look like:
-
-.. code:: python
-
-    # not a logcontext-rules-compliant function
-    def get_sleep_deferred(seconds):
-        d = defer.Deferred()
-        reactor.callLater(seconds, d.callback, None)
-        return d
-
-That doesn't follow the rules, but we can fix it by wrapping it with
-``PreserveLoggingContext`` and ``yield`` ing on it:
-
-.. code:: python
-
-    @defer.inlineCallbacks
-    def sleep(seconds):
-        with PreserveLoggingContext():
-            yield get_sleep_deferred(seconds)
-
-This technique works equally for external functions which return deferreds,
-or deferreds we have made ourselves.
-
-You can also use ``logcontext.make_deferred_yieldable``, which just does the
-boilerplate for you, so the above could be written:
-
-.. code:: python
-
-    def sleep(seconds):
-        return logcontext.make_deferred_yieldable(get_sleep_deferred(seconds))
-
-
-Fire-and-forget
----------------
-
-Sometimes you want to fire off a chain of execution, but not wait for its
-result. That might look a bit like this:
-
-.. code:: python
-
-    @defer.inlineCallbacks
-    def do_request_handling():
-        yield foreground_operation()
-
-        # *don't* do this
-        background_operation()
-
-        logger.debug("Request handling complete")
-
-    @defer.inlineCallbacks
-    def background_operation():
-        yield first_background_step()
-        logger.debug("Completed first step")
-        yield second_background_step()
-        logger.debug("Completed second step")
-
-The above code does a couple of steps in the background after
-``do_request_handling`` has finished. The log lines are still logged against
-the ``request_context`` logcontext, which may or may not be desirable. There
-are two big problems with the above, however. The first problem is that, if
-``background_operation`` returns an incomplete Deferred, it will expect its
-caller to ``yield`` immediately, so will have cleared the logcontext. In this
-example, that means that 'Request handling complete' will be logged without any
-context.
-
-The second problem, which is potentially even worse, is that when the Deferred
-returned by ``background_operation`` completes, it will restore the original
-logcontext. There is nothing waiting on that Deferred, so the logcontext will
-leak into the reactor and possibly get attached to some arbitrary future
-operation.
-
-There are two potential solutions to this.
-
-One option is to surround the call to ``background_operation`` with a
-``PreserveLoggingContext`` call. That will reset the logcontext before
-starting ``background_operation`` (so the context restored when the deferred
-completes will be the empty logcontext), and will restore the current
-logcontext before continuing the foreground process:
-
-.. code:: python
-
-    @defer.inlineCallbacks
-    def do_request_handling():
-        yield foreground_operation()
-
-        # start background_operation off in the empty logcontext, to
-        # avoid leaking the current context into the reactor.
-        with PreserveLoggingContext():
-            background_operation()
-
-        # this will now be logged against the request context
-        logger.debug("Request handling complete")
-
-Obviously that option means that the operations done in
-``background_operation`` would be not be logged against a logcontext (though
-that might be fixed by setting a different logcontext via a ``with
-LoggingContext(...)`` in ``background_operation``).
-
-The second option is to use ``logcontext.run_in_background``, which wraps a
-function so that it doesn't reset the logcontext even when it returns an
-incomplete deferred, and adds a callback to the returned deferred to reset the
-logcontext. In other words, it turns a function that follows the Synapse rules
-about logcontexts and Deferreds into one which behaves more like an external
-function — the opposite operation to that described in the previous section.
-It can be used like this:
-
-.. code:: python
-
-    @defer.inlineCallbacks
-    def do_request_handling():
-        yield foreground_operation()
-
-        logcontext.run_in_background(background_operation)
-
-        # this will now be logged against the request context
-        logger.debug("Request handling complete")
-
-Passing synapse deferreds into third-party functions
-----------------------------------------------------
-
-A typical example of this is where we want to collect together two or more
-deferred via ``defer.gatherResults``:
-
-.. code:: python
-
-    d1 = operation1()
-    d2 = operation2()
-    d3 = defer.gatherResults([d1, d2])
-
-This is really a variation of the fire-and-forget problem above, in that we are
-firing off ``d1`` and ``d2`` without yielding on them. The difference
-is that we now have third-party code attached to their callbacks. Anyway either
-technique given in the `Fire-and-forget`_ section will work.
-
-Of course, the new Deferred returned by ``gatherResults`` needs to be wrapped
-in order to make it follow the logcontext rules before we can yield it, as
-described in `Where you create a new Deferred, make it follow the rules`_.
-
-So, option one: reset the logcontext before starting the operations to be
-gathered:
-
-.. code:: python
-
-    @defer.inlineCallbacks
-    def do_request_handling():
-        with PreserveLoggingContext():
-            d1 = operation1()
-            d2 = operation2()
-            result = yield defer.gatherResults([d1, d2])
-
-In this case particularly, though, option two, of using
-``logcontext.preserve_fn`` almost certainly makes more sense, so that
-``operation1`` and ``operation2`` are both logged against the original
-logcontext. This looks like:
-
-.. code:: python
-
-    @defer.inlineCallbacks
-    def do_request_handling():
-        d1 = logcontext.preserve_fn(operation1)()
-        d2 = logcontext.preserve_fn(operation2)()
-
-        with PreserveLoggingContext():
-            result = yield defer.gatherResults([d1, d2])
-
-
-Was all this really necessary?
-------------------------------
-
-The conventions used work fine for a linear flow where everything happens in
-series via ``defer.inlineCallbacks`` and ``yield``, but are certainly tricky to
-follow for any more exotic flows. It's hard not to wonder if we could have done
-something else.
-
-We're not going to rewrite Synapse now, so the following is entirely of
-academic interest, but I'd like to record some thoughts on an alternative
-approach.
-
-I briefly prototyped some code following an alternative set of rules. I think
-it would work, but I certainly didn't get as far as thinking how it would
-interact with concepts as complicated as the cache descriptors.
-
-My alternative rules were:
-
-* functions always preserve the logcontext of their caller, whether or not they
-  are returning a Deferred.
-
-* Deferreds returned by synapse functions run their callbacks in the same
-  context as the function was orignally called in.
-
-The main point of this scheme is that everywhere that sets the logcontext is
-responsible for clearing it before returning control to the reactor.
-
-So, for example, if you were the function which started a ``with
-LoggingContext`` block, you wouldn't ``yield`` within it — instead you'd start
-off the background process, and then leave the ``with`` block to wait for it:
-
-.. code:: python
-
-    def handle_request(request_id):
-        with logcontext.LoggingContext() as request_context:
-            request_context.request = request_id
-            d = do_request_handling()
-
-        def cb(r):
-            logger.debug("finished")
-
-        d.addCallback(cb)
-        return d
-
-(in general, mixing ``with LoggingContext`` blocks and
-``defer.inlineCallbacks`` in the same function leads to slighly
-counter-intuitive code, under this scheme).
-
-Because we leave the original ``with`` block as soon as the Deferred is
-returned (as opposed to waiting for it to be resolved, as we do today), the
-logcontext is cleared before control passes back to the reactor; so if there is
-some code within ``do_request_handling`` which needs to wait for a Deferred to
-complete, there is no need for it to worry about clearing the logcontext before
-doing so:
-
-.. code:: python
-
-    def handle_request():
-        r = do_some_stuff()
-        r.addCallback(do_some_more_stuff)
-        return r
-
-— and provided ``do_some_stuff`` follows the rules of returning a Deferred which
-runs its callbacks in the original logcontext, all is happy.
-
-The business of a Deferred which runs its callbacks in the original logcontext
-isn't hard to achieve — we have it today, in the shape of
-``logcontext._PreservingContextDeferred``:
-
-.. code:: python
-
-    def do_some_stuff():
-        deferred = do_some_io()
-        pcd = _PreservingContextDeferred(LoggingContext.current_context())
-        deferred.chainDeferred(pcd)
-        return pcd
-
-It turns out that, thanks to the way that Deferreds chain together, we
-automatically get the property of a context-preserving deferred with
-``defer.inlineCallbacks``, provided the final Defered the function ``yields``
-on has that property. So we can just write:
-
-.. code:: python
-
-    @defer.inlineCallbacks
-    def handle_request():
-        yield do_some_stuff()
-        yield do_some_more_stuff()
-
-To conclude: I think this scheme would have worked equally well, with less
-danger of messing it up, and probably made some more esoteric code easier to
-write. But again — changing the conventions of the entire Synapse codebase is
-not a sensible option for the marginal improvement offered.
+Unanswered: how and when should we preserve log context?

docs/metrics-howto.rst

@@ -1,115 +1,50 @@
 How to monitor Synapse metrics using Prometheus
 ===============================================
 
-1. Install prometheus:
-
+1: Install prometheus:
   Follow instructions at http://prometheus.io/docs/introduction/install/
 
-2. Enable synapse metrics:
-
+2: Enable synapse metrics:
   Simply setting a (local) port number will enable it. Pick a port.
   prometheus itself defaults to 9090, so starting just above that for
   locally monitored services seems reasonable. E.g. 9092:
 
-   Add to homeserver.yaml::
+  Add to homeserver.yaml
 
     metrics_port: 9092
 
-   Also ensure that ``enable_metrics`` is set to ``True``.
+  Restart synapse
 
-   Restart synapse.
+3: Check out synapse-prometheus-config
+  https://github.com/matrix-org/synapse-prometheus-config
 
-3. Add a prometheus target for synapse.
+4: Add ``synapse.html`` and ``synapse.rules``
+  The ``.html`` file needs to appear in prometheus's ``consoles`` directory,
+  and the ``.rules`` file needs to be invoked somewhere in the main config
+  file. A symlink to each from the git checkout into the prometheus directory
+  might be easiest to ensure ``git pull`` keeps it updated.
 
-   It needs to set the ``metrics_path`` to a non-default value (under ``scrape_configs``)::
+5: Add a prometheus target for synapse
+  This is easiest if prometheus runs on the same machine as synapse, as it can
+  then just use localhost::
 
-    - job_name: "synapse"
-      metrics_path: "/_synapse/metrics"
-      static_configs:
-        - targets: ["my.server.here:9092"]
+    global: {
+      rule_file: "synapse.rules"
+    }
 
-   If your prometheus is older than 1.5.2, you will need to replace
-   ``static_configs`` in the above with ``target_groups``.
+    job: {
+      name: "synapse"
 
-   Restart prometheus.
+      target_group: {
+        target: "http://localhost:9092/"
+      }
+    }
 
+6: Start prometheus::
 
-Block and response metrics renamed for 0.27.0
----------------------------------------------
+   ./prometheus -config.file=prometheus.conf
 
-Synapse 0.27.0 begins the process of rationalising the duplicate ``*:count``
-metrics reported for the resource tracking for code blocks and HTTP requests.
+7: Wait a few seconds for it to start and perform the first scrape,
+   then visit the console:
 
-At the same time, the corresponding ``*:total`` metrics are being renamed, as
-the ``:total`` suffix no longer makes sense in the absence of a corresponding
-``:count`` metric.
-
-To enable a graceful migration path, this release just adds new names for the
-metrics being renamed. A future release will remove the old ones.
-
-The following table shows the new metrics, and the old metrics which they are
-replacing.
-
-==================================================== ===================================================
-New name                                             Old name
-==================================================== ===================================================
-synapse_util_metrics_block_count                     synapse_util_metrics_block_timer:count
-synapse_util_metrics_block_count                     synapse_util_metrics_block_ru_utime:count
-synapse_util_metrics_block_count                     synapse_util_metrics_block_ru_stime:count
-synapse_util_metrics_block_count                     synapse_util_metrics_block_db_txn_count:count
-synapse_util_metrics_block_count                     synapse_util_metrics_block_db_txn_duration:count
-
-synapse_util_metrics_block_time_seconds              synapse_util_metrics_block_timer:total
-synapse_util_metrics_block_ru_utime_seconds          synapse_util_metrics_block_ru_utime:total
-synapse_util_metrics_block_ru_stime_seconds          synapse_util_metrics_block_ru_stime:total
-synapse_util_metrics_block_db_txn_count              synapse_util_metrics_block_db_txn_count:total
-synapse_util_metrics_block_db_txn_duration_seconds   synapse_util_metrics_block_db_txn_duration:total
-
-synapse_http_server_response_count                   synapse_http_server_requests
-synapse_http_server_response_count                   synapse_http_server_response_time:count
-synapse_http_server_response_count                   synapse_http_server_response_ru_utime:count
-synapse_http_server_response_count                   synapse_http_server_response_ru_stime:count
-synapse_http_server_response_count                   synapse_http_server_response_db_txn_count:count
-synapse_http_server_response_count                   synapse_http_server_response_db_txn_duration:count
-
-synapse_http_server_response_time_seconds            synapse_http_server_response_time:total
-synapse_http_server_response_ru_utime_seconds        synapse_http_server_response_ru_utime:total
-synapse_http_server_response_ru_stime_seconds        synapse_http_server_response_ru_stime:total
-synapse_http_server_response_db_txn_count            synapse_http_server_response_db_txn_count:total
-synapse_http_server_response_db_txn_duration_seconds synapse_http_server_response_db_txn_duration:total
-==================================================== ===================================================
-
-
-Standard Metric Names
----------------------
-
-As of synapse version 0.18.2, the format of the process-wide metrics has been
-changed to fit prometheus standard naming conventions. Additionally the units
-have been changed to seconds, from miliseconds.
-
-================================== =============================
-New name                           Old name
-================================== =============================
-process_cpu_user_seconds_total     process_resource_utime / 1000
-process_cpu_system_seconds_total   process_resource_stime / 1000
-process_open_fds (no 'type' label) process_fds
-================================== =============================
-
-The python-specific counts of garbage collector performance have been renamed.
-
-=========================== ======================
-New name                    Old name
-=========================== ======================
-python_gc_time              reactor_gc_time
-python_gc_unreachable_total reactor_gc_unreachable
-python_gc_counts            reactor_gc_counts
-=========================== ======================
-
-The twisted-specific reactor metrics have been renamed.
-
-==================================== =====================
-New name                             Old name
-==================================== =====================
-python_twisted_reactor_pending_calls reactor_pending_calls
-python_twisted_reactor_tick_time     reactor_tick_time
-==================================== =====================
+    http://server-where-prometheus-runs:9090/consoles/synapse.html

docs/password_auth_providers.rst

@@ -1,99 +0,0 @@
-Password auth provider modules
-==============================
-
-Password auth providers offer a way for server administrators to integrate
-their Synapse installation with an existing authentication system.
-
-A password auth provider is a Python class which is dynamically loaded into
-Synapse, and provides a number of methods by which it can integrate with the
-authentication system.
-
-This document serves as a reference for those looking to implement their own
-password auth providers.
-
-Required methods
-----------------
-
-Password auth provider classes must provide the following methods:
-
-*class* ``SomeProvider.parse_config``\(*config*)
-
-    This method is passed the ``config`` object for this module from the
-    homeserver configuration file.
-
-    It should perform any appropriate sanity checks on the provided
-    configuration, and return an object which is then passed into ``__init__``.
-
-*class* ``SomeProvider``\(*config*, *account_handler*)
-
-    The constructor is passed the config object returned by ``parse_config``,
-    and a ``synapse.module_api.ModuleApi`` object which allows the
-    password provider to check if accounts exist and/or create new ones.
-
-Optional methods
-----------------
-
-Password auth provider classes may optionally provide the following methods.
-
-*class* ``SomeProvider.get_db_schema_files``\()
-
-    This method, if implemented, should return an Iterable of ``(name,
-    stream)`` pairs of database schema files. Each file is applied in turn at
-    initialisation, and a record is then made in the database so that it is
-    not re-applied on the next start.
-
-``someprovider.get_supported_login_types``\()
-
-    This method, if implemented, should return a ``dict`` mapping from a login
-    type identifier (such as ``m.login.password``) to an iterable giving the
-    fields which must be provided by the user in the submission to the
-    ``/login`` api. These fields are passed in the ``login_dict`` dictionary
-    to ``check_auth``.
-
-    For example, if a password auth provider wants to implement a custom login
-    type of ``com.example.custom_login``, where the client is expected to pass
-    the fields ``secret1`` and ``secret2``, the provider should implement this
-    method and return the following dict::
-
-      {"com.example.custom_login": ("secret1", "secret2")}
-
-``someprovider.check_auth``\(*username*, *login_type*, *login_dict*)
-
-    This method is the one that does the real work. If implemented, it will be
-    called for each login attempt where the login type matches one of the keys
-    returned by ``get_supported_login_types``.
-
-    It is passed the (possibly UNqualified) ``user`` provided by the client,
-    the login type, and a dictionary of login secrets passed by the client.
-
-    The method should return a Twisted ``Deferred`` object, which resolves to
-    the canonical ``@localpart:domain`` user id if authentication is successful,
-    and ``None`` if not.
-
-    Alternatively, the ``Deferred`` can resolve to a ``(str, func)`` tuple, in
-    which case the second field is a callback which will be called with the
-    result from the ``/login`` call (including ``access_token``, ``device_id``,
-    etc.)
-
-``someprovider.check_password``\(*user_id*, *password*)
-
-    This method provides a simpler interface than ``get_supported_login_types``
-    and ``check_auth`` for password auth providers that just want to provide a
-    mechanism for validating ``m.login.password`` logins.
-
-    Iif implemented, it will be called to check logins with an
-    ``m.login.password`` login type. It is passed a qualified
-    ``@localpart:domain`` user id, and the password provided by the user.
-
-    The method should return a Twisted ``Deferred`` object, which resolves to
-    ``True`` if authentication is successful, and ``False`` if not.
-
-``someprovider.on_logged_out``\(*user_id*, *device_id*, *access_token*)
-
-    This method, if implemented, is called when a user logs out. It is passed
-    the qualified user ID, the ID of the deactivated device (if any: access
-    tokens are occasionally created without an associated device ID), and the
-    (now deactivated) access token.
-
-    It may return a Twisted ``Deferred`` object; the logout request will wait
-    for the deferred to complete but the result is ignored.

docs/postgres.rst

@@ -1,8 +1,6 @@
 Using Postgres
 --------------
 
-Postgres version 9.4 or later is known to work.
-
 Set up database
 ===============
 
@@ -114,9 +112,9 @@ script one last time, e.g. if the SQLite database is at  ``homeserver.db``
 run::
 
     synapse_port_db --sqlite-database homeserver.db \
-        --postgres-config homeserver-postgres.yaml
+        --postgres-config database_config.yaml
 
 Once that has completed, change the synapse config to point at the PostgreSQL
-database configuration file ``homeserver-postgres.yaml`` (i.e. rename it to 
-``homeserver.yaml``) and restart synapse. Synapse should now be running against
+database configuration file using the ``database_config`` parameter (see
+`Synapse Config`_) and restart synapse. Synapse should now be running against
 PostgreSQL.

docs/replication.rst

@@ -26,10 +26,28 @@ expose the append-only log to the readers should be fairly minimal.
 Architecture
 ------------
 
-The Replication Protocol
-~~~~~~~~~~~~~~~~~~~~~~~~
+The Replication API
+~~~~~~~~~~~~~~~~~~~
 
-See ``tcp_replication.rst``
+Synapse will optionally expose a long poll HTTP API for extracting updates. The
+API will have a similar shape to /sync in that clients provide tokens
+indicating where in the log they have reached and a timeout. The synapse server
+then either responds with updates immediately if it already has updates or it
+waits until the timeout for more updates. If the timeout expires and nothing
+happened then the server returns an empty response.
+
+However unlike the /sync API this replication API is returning synapse specific
+data rather than trying to implement a matrix specification. The replication
+results are returned as arrays of rows where the rows are mostly lifted
+directly from the database. This avoids unnecessary JSON parsing on the server
+and hopefully avoids an impedance mismatch between the data returned and the
+required updates to the datastore.
+
+This does not replicate all the database tables as many of the database tables
+are indexes that can be recovered from the contents of other tables.
+
+The format and parameters for the api are documented in
+``synapse/replication/resource.py``.
 
 
 The Slaved DataStore

docs/sphinx/conf.py

@@ -50,7 +50,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = u'Synapse'
-copyright = u'Copyright 2014-2017 OpenMarket Ltd, 2017 Vector Creations Ltd, 2017 New Vector Ltd'
+copyright = u'2014, TNG'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the

docs/tcp_replication.rst

@@ -1,223 +0,0 @@
-TCP Replication
-===============
-
-Motivation
-----------
-
-Previously the workers used an HTTP long poll mechanism to get updates from the
-master, which had the problem of causing a lot of duplicate work on the server.
-This TCP protocol replaces those APIs with the aim of increased efficiency.
-
-
-
-Overview
---------
-
-The protocol is based on fire and forget, line based commands. An example flow
-would be (where '>' indicates master to worker and '<' worker to master flows)::
-
-    > SERVER example.com
-    < REPLICATE events 53
-    > RDATA events 54 ["$foo1:bar.com", ...]
-    > RDATA events 55 ["$foo4:bar.com", ...]
-
-The example shows the server accepting a new connection and sending its identity
-with the ``SERVER`` command, followed by the client asking to subscribe to the
-``events`` stream from the token ``53``. The server then periodically sends ``RDATA``
-commands which have the format ``RDATA <stream_name> <token> <row>``, where the
-format of ``<row>`` is defined by the individual streams.
-
-Error reporting happens by either the client or server sending an `ERROR`
-command, and usually the connection will be closed.
-
-
-Since the protocol is a simple line based, its possible to manually connect to
-the server using a tool like netcat. A few things should be noted when manually
-using the protocol:
-
-* When subscribing to a stream using ``REPLICATE``, the special token ``NOW`` can
-  be used to get all future updates. The special stream name ``ALL`` can be used
-  with ``NOW`` to subscribe to all available streams.
-* The federation stream is only available if federation sending has been
-  disabled on the main process.
-* The server will only time connections out that have sent a ``PING`` command.
-  If a ping is sent then the connection will be closed if no further commands
-  are receieved within 15s. Both the client and server protocol implementations
-  will send an initial PING on connection and ensure at least one command every
-  5s is sent (not necessarily ``PING``).
-* ``RDATA`` commands *usually* include a numeric token, however if the stream
-  has multiple rows to replicate per token the server will send multiple
-  ``RDATA`` commands, with all but the last having a token of ``batch``. See
-  the documentation on ``commands.RdataCommand`` for further details.
-
-
-Architecture
-------------
-
-The basic structure of the protocol is line based, where the initial word of
-each line specifies the command. The rest of the line is parsed based on the
-command. For example, the `RDATA` command is defined as::
-
-    RDATA <stream_name> <token> <row_json>
-
-(Note that `<row_json>` may contains spaces, but cannot contain newlines.)
-
-Blank lines are ignored.
-
-
-Keep alives
-~~~~~~~~~~~
-
-Both sides are expected to send at least one command every 5s or so, and
-should send a ``PING`` command if necessary. If either side do not receive a
-command within e.g. 15s then the connection should be closed.
-
-Because the server may be connected to manually using e.g. netcat, the timeouts
-aren't enabled until an initial ``PING`` command is seen. Both the client and
-server implementations below send a ``PING`` command immediately on connection to
-ensure the timeouts are enabled.
-
-This ensures that both sides can quickly realize if the tcp connection has gone
-and handle the situation appropriately.
-
-
-Start up
-~~~~~~~~
-
-When a new connection is made, the server:
-
-* Sends a ``SERVER`` command, which includes the identity of the server, allowing
-  the client to detect if its connected to the expected server
-* Sends a ``PING`` command as above, to enable the client to time out connections
-  promptly.
-
-The client:
-
-* Sends a ``NAME`` command, allowing the server to associate a human friendly
-  name with the connection. This is optional.
-* Sends a ``PING`` as above
-* For each stream the client wishes to subscribe to it sends a ``REPLICATE``
-  with the stream_name and token it wants to subscribe from.
-* On receipt of a ``SERVER`` command, checks that the server name matches the
-  expected server name.
-
-
-Error handling
-~~~~~~~~~~~~~~
-
-If either side detects an error it can send an ``ERROR`` command and close the
-connection.
-
-If the client side loses the connection to the server it should reconnect,
-following the steps above.
-
-
-Congestion
-~~~~~~~~~~
-
-If the server sends messages faster than the client can consume them the server
-will first buffer a (fairly large) number of commands and then disconnect the
-client. This ensures that we don't queue up an unbounded number of commands in
-memory and gives us a potential oppurtunity to squawk loudly. When/if the client
-recovers it can reconnect to the server and ask for missed messages.
-
-
-Reliability
-~~~~~~~~~~~
-
-In general the replication stream should be considered an unreliable transport
-since e.g. commands are not resent if the connection disappears.
-
-The exception to that are the replication streams, i.e. RDATA commands, since
-these include tokens which can be used to restart the stream on connection
-errors.
-
-The client should keep track of the token in the last RDATA command received
-for each stream so that on reconneciton it can start streaming from the correct
-place. Note: not all RDATA have valid tokens due to batching. See
-``RdataCommand`` for more details.
-
-
-Example
-~~~~~~~
-
-An example iteraction is shown below. Each line is prefixed with '>' or '<' to
-indicate which side is sending, these are *not* included on the wire::
-
-    * connection established *
-    > SERVER localhost:8823
-    > PING 1490197665618
-    < NAME synapse.app.appservice
-    < PING 1490197665618
-    < REPLICATE events 1
-    < REPLICATE backfill 1
-    < REPLICATE caches 1
-    > POSITION events 1
-    > POSITION backfill 1
-    > POSITION caches 1
-    > RDATA caches 2 ["get_user_by_id",["@01register-user:localhost:8823"],1490197670513]
-    > RDATA events 14 ["$149019767112vOHxz:localhost:8823",
-        "!AFDCvgApUmpdfVjIXm:localhost:8823","m.room.guest_access","",null]
-    < PING 1490197675618
-    > ERROR server stopping
-    * connection closed by server *
-
-The ``POSITION`` command sent by the server is used to set the clients position
-without needing to send data with the ``RDATA`` command.
-
-
-An example of a batched set of ``RDATA`` is::
-
-    > RDATA caches batch ["get_user_by_id",["@test:localhost:8823"],1490197670513]
-    > RDATA caches batch ["get_user_by_id",["@test2:localhost:8823"],1490197670513]
-    > RDATA caches batch ["get_user_by_id",["@test3:localhost:8823"],1490197670513]
-    > RDATA caches 54 ["get_user_by_id",["@test4:localhost:8823"],1490197670513]
-
-In this case the client shouldn't advance their caches token until it sees the
-the last ``RDATA``.
-
-
-List of commands
-~~~~~~~~~~~~~~~~
-
-The list of valid commands, with which side can send it: server (S) or client (C):
-
-SERVER (S)
-    Sent at the start to identify which server the client is talking to
-
-RDATA (S)
-    A single update in a stream
-
-POSITION (S)
-    The position of the stream has been updated
-
-ERROR (S, C)
-    There was an error
-
-PING (S, C)
-    Sent periodically to ensure the connection is still alive
-
-NAME (C)
-    Sent at the start by client to inform the server who they are
-
-REPLICATE (C)
-    Asks the server to replicate a given stream
-
-USER_SYNC (C)
-    A user has started or stopped syncing
-
-FEDERATION_ACK (C)
-    Acknowledge receipt of some federation data
-
-REMOVE_PUSHER (C)
-    Inform the server a pusher should be removed
-
-INVALIDATE_CACHE (C)
-    Inform the server a cache should be invalidated
-
-SYNC (S, C)
-    Used exclusively in tests
-
-
-See ``synapse/replication/tcp/commands.py`` for a detailed description and the
-format of each command.

docs/turn-howto.rst

@@ -50,37 +50,14 @@ You may be able to setup coturn via your package manager,  or set it up manually
 
        pwgen -s 64 1
 
- 5. Consider your security settings.  TURN lets users request a relay
-    which will connect to arbitrary IP addresses and ports.  At the least
-    we recommend:
-
-       # VoIP traffic is all UDP. There is no reason to let users connect to arbitrary TCP endpoints via the relay.
-       no-tcp-relay
-
-       # don't let the relay ever try to connect to private IP address ranges within your network (if any)
-       # given the turn server is likely behind your firewall, remember to include any privileged public IPs too.
-       denied-peer-ip=10.0.0.0-10.255.255.255
-       denied-peer-ip=192.168.0.0-192.168.255.255
-       denied-peer-ip=172.16.0.0-172.31.255.255
-
-       # special case the turn server itself so that client->TURN->TURN->client flows work
-       allowed-peer-ip=10.0.0.1
-
-       # consider whether you want to limit the quota of relayed streams per user (or total) to avoid risk of DoS.
-       user-quota=12 # 4 streams per video call, so 12 streams = 3 simultaneous relayed calls per user.
-       total-quota=1200
-
-    Ideally coturn should refuse to relay traffic which isn't SRTP;
-    see https://github.com/matrix-org/synapse/issues/2009
-
- 6. Ensure your firewall allows traffic into the TURN server on
+ 5. Ensure youe firewall allows traffic into the TURN server on
     the ports you've configured it to listen on (remember to allow
-    both TCP and UDP TURN traffic)
+    both TCP and UDP if you've enabled both).
 
- 7. If you've configured coturn to support TLS/DTLS, generate or
+ 6. If you've configured coturn to support TLS/DTLS, generate or
     import your private key and certificate.
 
- 8. Start the turn server::
+ 7. Start the turn server::
  
        bin/turnserver -o
 
@@ -106,19 +83,12 @@ Your home server configuration file needs the following extra keys:
     to refresh credentials. The TURN REST API specification recommends
     one day (86400000).
 
-  4. "turn_allow_guests": Whether to allow guest users to use the TURN
-    server.  This is enabled by default, as otherwise VoIP will not
-    work reliably for guests.  However, it does introduce a security risk
-    as it lets guests connect to arbitrary endpoints without having gone
-    through a CAPTCHA or similar to register a real account.
-
 As an example, here is the relevant section of the config file for
 matrix.org::
 
     turn_uris: [ "turn:turn.matrix.org:3478?transport=udp", "turn:turn.matrix.org:3478?transport=tcp" ]
     turn_shared_secret: n0t4ctuAllymatr1Xd0TorgSshar3d5ecret4obvIousreAsons
     turn_user_lifetime: 86400000
-    turn_allow_guests: True
 
 Now, restart synapse::
 

docs/url_previews.rst

@@ -56,7 +56,6 @@ As a first cut, let's do #2 and have the receiver hit the API to calculate its o
 API
 ---
 
-```
 GET /_matrix/media/r0/preview_url?url=http://wherever.com
 200 OK
 {
@@ -67,7 +66,6 @@ GET /_matrix/media/r0/preview_url?url=http://wherever.com
     "og:description" : "“Synapse 0.12 is out! Lots of polishing, performance & bugfixes: /sync API, /r0 prefix, fulltext search, 3PID invites https://t.co/5alhXLLEGP”"
     "og:site_name"   : "Twitter"
 }
-```
 
 * Downloads the URL
   * If HTML, just stores it in RAM and parses it for OG meta tags

docs/user_directory.md

@@ -1,17 +0,0 @@
-User Directory API Implementation
-=================================
-
-The user directory is currently maintained based on the 'visible' users
-on this particular server - i.e. ones which your account shares a room with, or
-who are present in a publicly viewable room present on the server.
-
-The directory info is stored in various tables, which can (typically after
-DB corruption) get stale or out of sync.  If this happens, for now the
-quickest solution to fix it is:
-
-```
-UPDATE user_directory_stream_pos SET stream_id = NULL;
-```
-
-and restart the synapse, which should then start a background task to
-flush the current tables and regenerate the directory.

docs/workers.rst

@@ -1,245 +0,0 @@
-Scaling synapse via workers
-===========================
-
-Synapse has experimental support for splitting out functionality into
-multiple separate python processes, helping greatly with scalability.  These
-processes are called 'workers', and are (eventually) intended to scale
-horizontally independently.
-
-All of the below is highly experimental and subject to change as Synapse evolves,
-but documenting it here to help folks needing highly scalable Synapses similar
-to the one running matrix.org!
-
-All processes continue to share the same database instance, and as such, workers
-only work with postgres based synapse deployments (sharing a single sqlite
-across multiple processes is a recipe for disaster, plus you should be using
-postgres anyway if you care about scalability).
-
-The workers communicate with the master synapse process via a synapse-specific
-TCP protocol called 'replication' - analogous to MySQL or Postgres style
-database replication; feeding a stream of relevant data to the workers so they
-can be kept in sync with the main synapse process and database state.
-
-Configuration
--------------
-
-To make effective use of the workers, you will need to configure an HTTP
-reverse-proxy such as nginx or haproxy, which will direct incoming requests to
-the correct worker, or to the main synapse instance. Note that this includes
-requests made to the federation port. The caveats regarding running a
-reverse-proxy on the federation port still apply (see
-https://github.com/matrix-org/synapse/blob/master/README.rst#reverse-proxying-the-federation-port).
-
-To enable workers, you need to add two replication listeners to the master
-synapse, e.g.::
-
-    listeners:
-      # The TCP replication port
-      - port: 9092
-        bind_address: '127.0.0.1'
-        type: replication
-      # The HTTP replication port
-      - port: 9093
-        bind_address: '127.0.0.1'
-        type: http
-        resources:
-         - names: [replication]
-
-Under **no circumstances** should these replication API listeners be exposed to
-the public internet; it currently implements no authentication whatsoever and is
-unencrypted.
-
-(Roughly, the TCP port is used for streaming data from the master to the
-workers, and the HTTP port for the workers to send data to the main
-synapse process.)
-
-You then create a set of configs for the various worker processes.  These
-should be worker configuration files, and should be stored in a dedicated
-subdirectory, to allow synctl to manipulate them. An additional configuration
-for the master synapse process will need to be created because the process will
-not be started automatically. That configuration should look like this::
-
-    worker_app: synapse.app.homeserver
-    daemonize: true
-
-Each worker configuration file inherits the configuration of the main homeserver
-configuration file.  You can then override configuration specific to that worker,
-e.g. the HTTP listener that it provides (if any); logging configuration; etc.
-You should minimise the number of overrides though to maintain a usable config.
-
-You must specify the type of worker application (``worker_app``). The currently
-available worker applications are listed below. You must also specify the
-replication endpoints that it's talking to on the main synapse process.
-``worker_replication_host`` should specify the host of the main synapse,
-``worker_replication_port`` should point to the TCP replication listener port and
-``worker_replication_http_port`` should point to the HTTP replication port.
-
-Currently, only the ``event_creator`` worker requires specifying
-``worker_replication_http_port``.
-
-For instance::
-
-    worker_app: synapse.app.synchrotron
-
-    # The replication listener on the synapse to talk to.
-    worker_replication_host: 127.0.0.1
-    worker_replication_port: 9092
-    worker_replication_http_port: 9093
-
-    worker_listeners:
-     - type: http
-       port: 8083
-       resources:
-         - names:
-           - client
-
-    worker_daemonize: True
-    worker_pid_file: /home/matrix/synapse/synchrotron.pid
-    worker_log_config: /home/matrix/synapse/config/synchrotron_log_config.yaml
-
-...is a full configuration for a synchrotron worker instance, which will expose a
-plain HTTP ``/sync`` endpoint on port 8083 separately from the ``/sync`` endpoint provided
-by the main synapse.
-
-Obviously you should configure your reverse-proxy to route the relevant
-endpoints to the worker (``localhost:8083`` in the above example).
-
-Finally, to actually run your worker-based synapse, you must pass synctl the -a
-commandline option to tell it to operate on all the worker configurations found
-in the given directory, e.g.::
-
-    synctl -a $CONFIG/workers start
-
-Currently one should always restart all workers when restarting or upgrading
-synapse, unless you explicitly know it's safe not to.  For instance, restarting
-synapse without restarting all the synchrotrons may result in broken typing
-notifications.
-
-To manipulate a specific worker, you pass the -w option to synctl::
-
-    synctl -w $CONFIG/workers/synchrotron.yaml restart
-
-
-Available worker applications
------------------------------
-
-``synapse.app.pusher``
-~~~~~~~~~~~~~~~~~~~~~~
-
-Handles sending push notifications to sygnal and email. Doesn't handle any
-REST endpoints itself, but you should set ``start_pushers: False`` in the
-shared configuration file to stop the main synapse sending these notifications.
-
-Note this worker cannot be load-balanced: only one instance should be active.
-
-``synapse.app.synchrotron``
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The synchrotron handles ``sync`` requests from clients. In particular, it can
-handle REST endpoints matching the following regular expressions::
-
-    ^/_matrix/client/(v2_alpha|r0)/sync$
-    ^/_matrix/client/(api/v1|v2_alpha|r0)/events$
-    ^/_matrix/client/(api/v1|r0)/initialSync$
-    ^/_matrix/client/(api/v1|r0)/rooms/[^/]+/initialSync$
-
-The above endpoints should all be routed to the synchrotron worker by the
-reverse-proxy configuration.
-
-It is possible to run multiple instances of the synchrotron to scale
-horizontally. In this case the reverse-proxy should be configured to
-load-balance across the instances, though it will be more efficient if all
-requests from a particular user are routed to a single instance. Extracting
-a userid from the access token is currently left as an exercise for the reader.
-
-``synapse.app.appservice``
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Handles sending output traffic to Application Services. Doesn't handle any
-REST endpoints itself, but you should set ``notify_appservices: False`` in the
-shared configuration file to stop the main synapse sending these notifications.
-
-Note this worker cannot be load-balanced: only one instance should be active.
-
-``synapse.app.federation_reader``
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Handles a subset of federation endpoints. In particular, it can handle REST
-endpoints matching the following regular expressions::
-
-    ^/_matrix/federation/v1/event/
-    ^/_matrix/federation/v1/state/
-    ^/_matrix/federation/v1/state_ids/
-    ^/_matrix/federation/v1/backfill/
-    ^/_matrix/federation/v1/get_missing_events/
-    ^/_matrix/federation/v1/publicRooms
-
-The above endpoints should all be routed to the federation_reader worker by the
-reverse-proxy configuration.
-
-``synapse.app.federation_sender``
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Handles sending federation traffic to other servers. Doesn't handle any
-REST endpoints itself, but you should set ``send_federation: False`` in the
-shared configuration file to stop the main synapse sending this traffic.
-
-Note this worker cannot be load-balanced: only one instance should be active.
-
-``synapse.app.media_repository``
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Handles the media repository. It can handle all endpoints starting with::
-
-    /_matrix/media/
-
-You should also set ``enable_media_repo: False`` in the shared configuration
-file to stop the main synapse running background jobs related to managing the
-media repository.
-
-Note this worker cannot be load-balanced: only one instance should be active.
-
-``synapse.app.client_reader``
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Handles client API endpoints. It can handle REST endpoints matching the
-following regular expressions::
-
-    ^/_matrix/client/(api/v1|r0|unstable)/publicRooms$
-
-``synapse.app.user_dir``
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-Handles searches in the user directory. It can handle REST endpoints matching
-the following regular expressions::
-
-    ^/_matrix/client/(api/v1|r0|unstable)/user_directory/search$
-
-``synapse.app.frontend_proxy``
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Proxies some frequently-requested client endpoints to add caching and remove
-load from the main synapse. It can handle REST endpoints matching the following
-regular expressions::
-
-    ^/_matrix/client/(api/v1|r0|unstable)/keys/upload
-
-It will proxy any requests it cannot handle to the main synapse instance. It
-must therefore be configured with the location of the main instance, via
-the ``worker_main_http_uri`` setting in the frontend_proxy worker configuration
-file. For example::
-
-    worker_main_http_uri: http://127.0.0.1:8008
-
-
-``synapse.app.event_creator``
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Handles some event creation. It can handle REST endpoints matching::
-
-    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/send
-    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$
-    ^/_matrix/client/(api/v1|r0|unstable)/join/
-
-It will create events locally and then send them on to the main synapse
-instance to be persisted and handled.

jenkins-dendron-haproxy-postgres.sh

@@ -1,23 +0,0 @@
-#!/bin/bash
-
-set -eux
-
-: ${WORKSPACE:="$(pwd)"}
-
-export WORKSPACE
-export PYTHONDONTWRITEBYTECODE=yep
-export SYNAPSE_CACHE_FACTOR=1
-
-export HAPROXY_BIN=/home/haproxy/haproxy-1.6.11/haproxy
-
-./jenkins/prepare_synapse.sh
-./jenkins/clone.sh sytest https://github.com/matrix-org/sytest.git
-./jenkins/clone.sh dendron https://github.com/matrix-org/dendron.git
-./dendron/jenkins/build_dendron.sh
-./sytest/jenkins/prep_sytest_for_postgres.sh
-
-./sytest/jenkins/install_and_run.sh \
-    --python $WORKSPACE/.tox/py27/bin/python \
-    --synapse-directory $WORKSPACE \
-    --dendron $WORKSPACE/dendron/bin/dendron \
-    --haproxy \

jenkins-dendron-postgres.sh

@@ -4,17 +4,84 @@ set -eux
 
 : ${WORKSPACE:="$(pwd)"}
 
-export WORKSPACE
 export PYTHONDONTWRITEBYTECODE=yep
 export SYNAPSE_CACHE_FACTOR=1
 
-./jenkins/prepare_synapse.sh
-./jenkins/clone.sh sytest https://github.com/matrix-org/sytest.git
-./jenkins/clone.sh dendron https://github.com/matrix-org/dendron.git
-./dendron/jenkins/build_dendron.sh
-./sytest/jenkins/prep_sytest_for_postgres.sh
+# Output test results as junit xml
+export TRIAL_FLAGS="--reporter=subunit"
+export TOXSUFFIX="| subunit-1to2 | subunit2junitxml --no-passthrough --output-to=results.xml"
+# Write coverage reports to a separate file for each process
+export COVERAGE_OPTS="-p"
+export DUMP_COVERAGE_COMMAND="coverage help"
 
-./sytest/jenkins/install_and_run.sh \
-    --python $WORKSPACE/.tox/py27/bin/python \
+# Output flake8 violations to violations.flake8.log
+# Don't exit with non-0 status code on Jenkins,
+# so that the build steps continue and a later step can decided whether to
+# UNSTABLE or FAILURE this build.
+export PEP8SUFFIX="--output-file=violations.flake8.log || echo flake8 finished with status code \$?"
+
+rm .coverage* || echo "No coverage files to remove"
+
+tox --notest -e py27
+
+TOX_BIN=$WORKSPACE/.tox/py27/bin
+python synapse/python_dependencies.py | xargs -n1 $TOX_BIN/pip install
+$TOX_BIN/pip install psycopg2
+$TOX_BIN/pip install lxml
+
+: ${GIT_BRANCH:="origin/$(git rev-parse --abbrev-ref HEAD)"}
+
+if [[ ! -e .dendron-base ]]; then
+    git clone https://github.com/matrix-org/dendron.git .dendron-base --mirror
+else
+    (cd .dendron-base; git fetch -p)
+fi
+
+rm -rf dendron
+git clone .dendron-base dendron --shared
+cd dendron
+
+: ${GOPATH:=${WORKSPACE}/.gopath}
+if [[ "${GOPATH}" != *:* ]]; then
+  mkdir -p "${GOPATH}"
+  export PATH="${GOPATH}/bin:${PATH}"
+fi
+export GOPATH
+
+git checkout "${GIT_BRANCH}" || (echo >&2 "No ref ${GIT_BRANCH} found, falling back to develop" ; git checkout develop)
+
+go get github.com/constabulary/gb/...
+gb generate
+gb build
+
+cd ..
+
+
+if [[ ! -e .sytest-base ]]; then
+  git clone https://github.com/matrix-org/sytest.git .sytest-base --mirror
+else
+  (cd .sytest-base; git fetch -p)
+fi
+
+rm -rf sytest
+git clone .sytest-base sytest --shared
+cd sytest
+
+git checkout "${GIT_BRANCH}" || (echo >&2 "No ref ${GIT_BRANCH} found, falling back to develop" ; git checkout develop)
+
+: ${PORT_BASE:=8000}
+: ${PORT_COUNT=20}
+
+./jenkins/prep_sytest_for_postgres.sh
+
+mkdir -p var
+
+echo >&2 "Running sytest with PostgreSQL";
+./jenkins/install_and_run.sh --python $TOX_BIN/python \
                              --synapse-directory $WORKSPACE \
                              --dendron $WORKSPACE/dendron/bin/dendron \
+                             --pusher \
+                             --synchrotron \
+                             --port-range ${PORT_BASE}:$((PORT_BASE+PORT_COUNT-1))
+
+cd ..

jenkins-postgres.sh

@@ -4,15 +4,61 @@ set -eux
 
 : ${WORKSPACE:="$(pwd)"}
 
-export WORKSPACE
 export PYTHONDONTWRITEBYTECODE=yep
 export SYNAPSE_CACHE_FACTOR=1
 
-./jenkins/prepare_synapse.sh
-./jenkins/clone.sh sytest https://github.com/matrix-org/sytest.git
+# Output test results as junit xml
+export TRIAL_FLAGS="--reporter=subunit"
+export TOXSUFFIX="| subunit-1to2 | subunit2junitxml --no-passthrough --output-to=results.xml"
+# Write coverage reports to a separate file for each process
+export COVERAGE_OPTS="-p"
+export DUMP_COVERAGE_COMMAND="coverage help"
 
-./sytest/jenkins/prep_sytest_for_postgres.sh
+# Output flake8 violations to violations.flake8.log
+# Don't exit with non-0 status code on Jenkins,
+# so that the build steps continue and a later step can decided whether to
+# UNSTABLE or FAILURE this build.
+export PEP8SUFFIX="--output-file=violations.flake8.log || echo flake8 finished with status code \$?"
 
-./sytest/jenkins/install_and_run.sh \
-    --python $WORKSPACE/.tox/py27/bin/python \
+rm .coverage* || echo "No coverage files to remove"
+
+tox --notest -e py27
+
+TOX_BIN=$WORKSPACE/.tox/py27/bin
+python synapse/python_dependencies.py | xargs -n1 $TOX_BIN/pip install
+$TOX_BIN/pip install psycopg2
+$TOX_BIN/pip install lxml
+
+: ${GIT_BRANCH:="origin/$(git rev-parse --abbrev-ref HEAD)"}
+
+if [[ ! -e .sytest-base ]]; then
+  git clone https://github.com/matrix-org/sytest.git .sytest-base --mirror
+else
+  (cd .sytest-base; git fetch -p)
+fi
+
+rm -rf sytest
+git clone .sytest-base sytest --shared
+cd sytest
+
+git checkout "${GIT_BRANCH}" || (echo >&2 "No ref ${GIT_BRANCH} found, falling back to develop" ; git checkout develop)
+
+: ${PORT_BASE:=8000}
+: ${PORT_COUNT=20}
+
+./jenkins/prep_sytest_for_postgres.sh
+
+echo >&2 "Running sytest with PostgreSQL";
+./jenkins/install_and_run.sh --coverage \
+                             --python $TOX_BIN/python \
                              --synapse-directory $WORKSPACE \
+                             --port-range ${PORT_BASE}:$((PORT_BASE+PORT_COUNT-1)) \
+
+cd ..
+cp sytest/.coverage.* .
+
+# Combine the coverage reports
+echo "Combining:" .coverage.*
+$TOX_BIN/python -m coverage combine
+# Output coverage to coverage.xml
+$TOX_BIN/coverage xml -o coverage.xml

jenkins-sqlite.sh

@@ -4,13 +4,55 @@ set -eux
 
 : ${WORKSPACE:="$(pwd)"}
 
-export WORKSPACE
 export PYTHONDONTWRITEBYTECODE=yep
 export SYNAPSE_CACHE_FACTOR=1
 
-./jenkins/prepare_synapse.sh
-./jenkins/clone.sh sytest https://github.com/matrix-org/sytest.git
+# Output test results as junit xml
+export TRIAL_FLAGS="--reporter=subunit"
+export TOXSUFFIX="| subunit-1to2 | subunit2junitxml --no-passthrough --output-to=results.xml"
+# Write coverage reports to a separate file for each process
+export COVERAGE_OPTS="-p"
+export DUMP_COVERAGE_COMMAND="coverage help"
 
-./sytest/jenkins/install_and_run.sh \
-    --python $WORKSPACE/.tox/py27/bin/python \
+# Output flake8 violations to violations.flake8.log
+# Don't exit with non-0 status code on Jenkins,
+# so that the build steps continue and a later step can decided whether to
+# UNSTABLE or FAILURE this build.
+export PEP8SUFFIX="--output-file=violations.flake8.log || echo flake8 finished with status code \$?"
+
+rm .coverage* || echo "No coverage files to remove"
+
+tox --notest -e py27
+TOX_BIN=$WORKSPACE/.tox/py27/bin
+python synapse/python_dependencies.py | xargs -n1 $TOX_BIN/pip install
+$TOX_BIN/pip install lxml
+
+: ${GIT_BRANCH:="origin/$(git rev-parse --abbrev-ref HEAD)"}
+
+if [[ ! -e .sytest-base ]]; then
+  git clone https://github.com/matrix-org/sytest.git .sytest-base --mirror
+else
+  (cd .sytest-base; git fetch -p)
+fi
+
+rm -rf sytest
+git clone .sytest-base sytest --shared
+cd sytest
+
+git checkout "${GIT_BRANCH}" || (echo >&2 "No ref ${GIT_BRANCH} found, falling back to develop" ; git checkout develop)
+
+: ${PORT_COUNT=20}
+: ${PORT_BASE:=8000}
+./jenkins/install_and_run.sh --coverage \
+                             --python $TOX_BIN/python \
                              --synapse-directory $WORKSPACE \
+                             --port-range ${PORT_BASE}:$((PORT_BASE+PORT_COUNT-1)) \
+
+cd ..
+cp sytest/.coverage.* .
+
+# Combine the coverage reports
+echo "Combining:" .coverage.*
+$TOX_BIN/python -m coverage combine
+# Output coverage to coverage.xml
+$TOX_BIN/coverage xml -o coverage.xml

jenkins-unittests.sh

@@ -22,9 +22,4 @@ export PEP8SUFFIX="--output-file=violations.flake8.log || echo flake8 finished w
 
 rm .coverage* || echo "No coverage files to remove"
 
-tox --notest -e py27
-TOX_BIN=$WORKSPACE/.tox/py27/bin
-python synapse/python_dependencies.py | xargs -n1 $TOX_BIN/pip install
-$TOX_BIN/pip install lxml
-
 tox -e py27

jenkins/clone.sh

@@ -1,44 +0,0 @@
-#! /bin/bash
-
-# This clones a project from github into a named subdirectory
-# If the project has a branch with the same name as this branch
-# then it will checkout that branch after cloning.
-# Otherwise it will checkout "origin/develop."
-# The first argument is the name of the directory to checkout
-# the branch into.
-# The second argument is the URL of the remote repository to checkout.
-# Usually something like https://github.com/matrix-org/sytest.git
-
-set -eux
-
-NAME=$1
-PROJECT=$2
-BASE=".$NAME-base"
-
-# Update our mirror.
-if [ ! -d ".$NAME-base" ]; then
-  # Create a local mirror of the source repository.
-  # This saves us from having to download the entire repository
-  # when this script is next run.
-  git clone "$PROJECT" "$BASE" --mirror
-else
-  # Fetch any updates from the source repository.
-  (cd "$BASE"; git fetch -p)
-fi
-
-# Remove the existing repository so that we have a clean copy
-rm -rf "$NAME"
-# Cloning with --shared means that we will share portions of the
-# .git directory with our local mirror.
-git clone "$BASE" "$NAME" --shared
-
-# Jenkins may have supplied us with the name of the branch in the
-# environment. Otherwise we will have to guess based on the current
-# commit.
-: ${GIT_BRANCH:="origin/$(git rev-parse --abbrev-ref HEAD)"}
-cd "$NAME"
-# check out the relevant branch
-git checkout "${GIT_BRANCH}" || (
-    echo >&2 "No ref ${GIT_BRANCH} found, falling back to develop"
-    git checkout "origin/develop"
-)

jenkins/prepare_synapse.sh

@@ -1,20 +0,0 @@
-#! /bin/bash
-
-cd "`dirname $0`/.."
-
-TOX_DIR=$WORKSPACE/.tox
-
-mkdir -p $TOX_DIR
-
-if ! [ $TOX_DIR -ef .tox ]; then
-    ln -s "$TOX_DIR" .tox
-fi
-
-# set up the virtualenv
-tox -e py27 --notest -v
-
-TOX_BIN=$TOX_DIR/py27/bin
-$TOX_BIN/pip install setuptools
-{ python synapse/python_dependencies.py
-  echo lxml psycopg2
-} | xargs $TOX_BIN/pip install

res/templates/notif_mail.html

@@ -18,9 +18,7 @@
                                 <div class="summarytext">{{ summary_text }}</div>
                             </td>
                             <td class="logo">
-                                {% if app_name == "Riot" %}
-                                    <img src="http://matrix.org/img/riot-logo-email.png" width="83" height="83" alt="[Riot]"/>
-                                {% elif app_name == "Vector" %}
+                                {% if app_name == "Vector" %}
                                     <img src="http://matrix.org/img/vector-logo-email.png" width="64" height="83" alt="[Vector]"/>
                                 {% else %}
                                     <img src="http://matrix.org/img/matrix-120x51.png" width="120" height="51" alt="[matrix]"/>

scripts-dev/federation_client.py

@@ -1,30 +1,10 @@
-#!/usr/bin/env python
-#
-# Copyright 2015, 2016 OpenMarket Ltd
-# Copyright 2017 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from __future__ import print_function
-
-import argparse
 import nacl.signing
 import json
 import base64
 import requests
 import sys
 import srvlookup
-import yaml
+
 
 def encode_base64(input_bytes):
     """Encode bytes as a base64 string without any padding."""
@@ -123,124 +103,44 @@ def lookup(destination, path):
         except:
             return "https://%s:%d%s" % (destination, 8448, path)
 
-
-def request_json(method, origin_name, origin_key, destination, path, content):
-    if method is None:
-        if content is None:
-            method = "GET"
-        else:
-            method = "POST"
-
-    json_to_sign = {
-        "method": method,
+def get_json(origin_name, origin_key, destination, path):
+    request_json = {
+        "method": "GET",
         "uri": path,
         "origin": origin_name,
         "destination": destination,
     }
 
-    if content is not None:
-        json_to_sign["content"] = json.loads(content)
-
-    signed_json = sign_json(json_to_sign, origin_key, origin_name)
+    signed_json = sign_json(request_json, origin_key, origin_name)
 
     authorization_headers = []
 
     for key, sig in signed_json["signatures"][origin_name].items():
-        header = "X-Matrix origin=%s,key=\"%s\",sig=\"%s\"" % (
+        authorization_headers.append(bytes(
+            "X-Matrix origin=%s,key=\"%s\",sig=\"%s\"" % (
                 origin_name, key, sig,
             )
-        authorization_headers.append(bytes(header))
-        print ("Authorization: %s" % header, file=sys.stderr)
+        ))
 
-    dest = lookup(destination, path)
-    print ("Requesting %s" % dest, file=sys.stderr)
-
-    result = requests.request(
-        method=method,
-        url=dest,
+    result = requests.get(
+        lookup(destination, path),
         headers={"Authorization": authorization_headers[0]},
         verify=False,
-        data=content,
     )
-    sys.stderr.write("Status Code: %d\n" % (result.status_code,))
     return result.json()
 
 
 def main():
-    parser = argparse.ArgumentParser(
-        description=
-            "Signs and sends a federation request to a matrix homeserver",
-    )
+    origin_name, keyfile, destination, path = sys.argv[1:]
 
-    parser.add_argument(
-        "-N", "--server-name",
-        help="Name to give as the local homeserver. If unspecified, will be "
-             "read from the config file.",
-    )
-
-    parser.add_argument(
-        "-k", "--signing-key-path",
-        help="Path to the file containing the private ed25519 key to sign the "
-             "request with.",
-    )
-
-    parser.add_argument(
-        "-c", "--config",
-        default="homeserver.yaml",
-        help="Path to server config file. Ignored if --server-name and "
-             "--signing-key-path are both given.",
-    )
-
-    parser.add_argument(
-        "-d", "--destination",
-        default="matrix.org",
-        help="name of the remote homeserver. We will do SRV lookups and "
-             "connect appropriately.",
-    )
-
-    parser.add_argument(
-        "-X", "--method",
-        help="HTTP method to use for the request. Defaults to GET if --data is"
-             "unspecified, POST if it is."
-    )
-
-    parser.add_argument(
-        "--body",
-        help="Data to send as the body of the HTTP request"
-    )
-
-    parser.add_argument(
-        "path",
-        help="request path. We will add '/_matrix/federation/v1/' to this."
-    )
-
-    args = parser.parse_args()
-
-    if not args.server_name or not args.signing_key_path:
-        read_args_from_config(args)
-
-    with open(args.signing_key_path) as f:
+    with open(keyfile) as f:
         key = read_signing_keys(f)[0]
 
-    result = request_json(
-        args.method,
-        args.server_name, key, args.destination,
-        "/_matrix/federation/v1/" + args.path,
-        content=args.body,
+    result = get_json(
+        origin_name, key, destination, "/_matrix/federation/v1/" + path
     )
 
     json.dump(result, sys.stdout)
-    print ("")
-
-
-def read_args_from_config(args):
-    with open(args.config, 'r') as fh:
-        config = yaml.safe_load(fh)
-        if not args.server_name:
-            args.server_name = config['server_name']
-        if not args.signing_key_path:
-            args.signing_key_path = config['signing_key_path']
-
 
 if __name__ == "__main__":
     main()

scripts-dev/nuke-room-from-db.sh

@@ -9,39 +9,16 @@
 ROOMID="$1"
 
 sqlite3 homeserver.db <<EOF
-DELETE FROM event_forward_extremities WHERE room_id = '$ROOMID';
-DELETE FROM event_backward_extremities WHERE room_id = '$ROOMID';
-DELETE FROM event_edges WHERE room_id = '$ROOMID';
-DELETE FROM room_depth WHERE room_id = '$ROOMID';
-DELETE FROM state_forward_extremities WHERE room_id = '$ROOMID';
-DELETE FROM events WHERE room_id = '$ROOMID';
-DELETE FROM event_json WHERE room_id = '$ROOMID';
-DELETE FROM state_events WHERE room_id = '$ROOMID';
-DELETE FROM current_state_events WHERE room_id = '$ROOMID';
-DELETE FROM room_memberships WHERE room_id = '$ROOMID';
+DELETE FROM context_depth WHERE context = '$ROOMID';
+DELETE FROM current_state WHERE context = '$ROOMID';
 DELETE FROM feedback WHERE room_id = '$ROOMID';
-DELETE FROM topics WHERE room_id = '$ROOMID';
-DELETE FROM room_names WHERE room_id = '$ROOMID';
+DELETE FROM messages WHERE room_id = '$ROOMID';
+DELETE FROM pdu_backward_extremities WHERE context = '$ROOMID';
+DELETE FROM pdu_edges WHERE context = '$ROOMID';
+DELETE FROM pdu_forward_extremities WHERE context = '$ROOMID';
+DELETE FROM pdus WHERE context = '$ROOMID';
+DELETE FROM room_data WHERE room_id = '$ROOMID';
+DELETE FROM room_memberships WHERE room_id = '$ROOMID';
 DELETE FROM rooms WHERE room_id = '$ROOMID';
-DELETE FROM room_hosts WHERE room_id = '$ROOMID';
-DELETE FROM room_aliases WHERE room_id = '$ROOMID';
-DELETE FROM state_groups WHERE room_id = '$ROOMID';
-DELETE FROM state_groups_state WHERE room_id = '$ROOMID';
-DELETE FROM receipts_graph WHERE room_id = '$ROOMID';
-DELETE FROM receipts_linearized WHERE room_id = '$ROOMID';
-DELETE FROM event_search_content WHERE c1room_id = '$ROOMID';
-DELETE FROM guest_access WHERE room_id = '$ROOMID';
-DELETE FROM history_visibility WHERE room_id = '$ROOMID';
-DELETE FROM room_tags WHERE room_id = '$ROOMID';
-DELETE FROM room_tags_revisions WHERE room_id = '$ROOMID';
-DELETE FROM room_account_data WHERE room_id = '$ROOMID';
-DELETE FROM event_push_actions WHERE room_id = '$ROOMID';
-DELETE FROM local_invites WHERE room_id = '$ROOMID';
-DELETE FROM pusher_throttle WHERE room_id = '$ROOMID';
-DELETE FROM event_reports WHERE room_id = '$ROOMID';
-DELETE FROM public_room_list_stream WHERE room_id = '$ROOMID';
-DELETE FROM stream_ordering_to_exterm WHERE room_id = '$ROOMID';
-DELETE FROM event_auth WHERE room_id = '$ROOMID';
-DELETE FROM appservice_room_list WHERE room_id = '$ROOMID';
-VACUUM;
+DELETE FROM state_pdus WHERE context = '$ROOMID';
 EOF

scripts/move_remote_media_to_new_store.py

@@ -1,133 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-# Copyright 2017 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""
-Moves a list of remote media from one media store to another.
-
-The input should be a list of media files to be moved, one per line. Each line
-should be formatted::
-
-    <origin server>|<file id>
-
-This can be extracted from postgres with::
-
-    psql --tuples-only -A -c "select media_origin, filesystem_id from
-        matrix.remote_media_cache where ..."
-
-To use, pipe the above into::
-
-    PYTHON_PATH=. ./scripts/move_remote_media_to_new_store.py <source repo> <dest repo>
-"""
-
-from __future__ import print_function
-
-import argparse
-import logging
-
-import sys
-
-import os
-
-import shutil
-
-from synapse.rest.media.v1.filepath import MediaFilePaths
-
-logger = logging.getLogger()
-
-
-def main(src_repo, dest_repo):
-    src_paths = MediaFilePaths(src_repo)
-    dest_paths = MediaFilePaths(dest_repo)
-    for line in sys.stdin:
-        line = line.strip()
-        parts = line.split('|')
-        if len(parts) != 2:
-            print("Unable to parse input line %s" % line, file=sys.stderr)
-            exit(1)
-
-        move_media(parts[0], parts[1], src_paths, dest_paths)
-
-
-def move_media(origin_server, file_id, src_paths, dest_paths):
-    """Move the given file, and any thumbnails, to the dest repo
-
-    Args:
-        origin_server (str):
-        file_id (str):
-        src_paths (MediaFilePaths):
-        dest_paths (MediaFilePaths):
-    """
-    logger.info("%s/%s", origin_server, file_id)
-
-    # check that the original exists
-    original_file = src_paths.remote_media_filepath(origin_server, file_id)
-    if not os.path.exists(original_file):
-        logger.warn(
-            "Original for %s/%s (%s) does not exist",
-            origin_server, file_id, original_file,
-        )
-    else:
-        mkdir_and_move(
-            original_file,
-            dest_paths.remote_media_filepath(origin_server, file_id),
-        )
-
-    # now look for thumbnails
-    original_thumb_dir = src_paths.remote_media_thumbnail_dir(
-        origin_server, file_id,
-    )
-    if not os.path.exists(original_thumb_dir):
-        return
-
-    mkdir_and_move(
-        original_thumb_dir,
-        dest_paths.remote_media_thumbnail_dir(origin_server, file_id)
-    )
-
-
-def mkdir_and_move(original_file, dest_file):
-    dirname = os.path.dirname(dest_file)
-    if not os.path.exists(dirname):
-        logger.debug("mkdir %s", dirname)
-        os.makedirs(dirname)
-    logger.debug("mv %s %s", original_file, dest_file)
-    shutil.move(original_file, dest_file)
-
-
-if __name__ == "__main__":
-    parser = argparse.ArgumentParser(
-        description=__doc__,
-        formatter_class = argparse.RawDescriptionHelpFormatter,
-    )
-    parser.add_argument(
-        "-v", action='store_true', help='enable debug logging')
-    parser.add_argument(
-        "src_repo",
-        help="Path to source content repo",
-    )
-    parser.add_argument(
-        "dest_repo",
-        help="Path to source content repo",
-    )
-    args = parser.parse_args()
-
-    logging_config = {
-        "level": logging.DEBUG if args.v else logging.INFO,
-        "format": "%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(message)s"
-    }
-    logging.basicConfig(**logging_config)
-
-    main(args.src_repo, args.dest_repo)

scripts/synapse_port_db

@@ -1,7 +1,6 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 # Copyright 2015, 2016 OpenMarket Ltd
-# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -30,29 +29,16 @@ import time
 import traceback
 import yaml
 
-from six import string_types
-
 
 logger = logging.getLogger("synapse_port_db")
 
 
 BOOLEAN_COLUMNS = {
-    "events": ["processed", "outlier", "contains_url"],
+    "events": ["processed", "outlier"],
     "rooms": ["is_public"],
     "event_edges": ["is_state"],
     "presence_list": ["accepted"],
     "presence_stream": ["currently_active"],
-    "public_room_list_stream": ["visibility"],
-    "device_lists_outbound_pokes": ["sent"],
-    "users_who_share_rooms": ["share_private"],
-    "groups": ["is_public"],
-    "group_rooms": ["is_public"],
-    "group_users": ["is_public", "is_admin"],
-    "group_summary_rooms": ["is_public"],
-    "group_room_categories": ["is_public"],
-    "group_summary_users": ["is_public"],
-    "group_roles": ["is_public"],
-    "local_group_membership": ["is_publicised", "is_admin"],
 }
 
 
@@ -85,14 +71,6 @@ APPEND_ONLY_TABLES = [
     "event_to_state_groups",
     "rejections",
     "event_search",
-    "presence_stream",
-    "push_rules_stream",
-    "current_state_resets",
-    "ex_outlier_stream",
-    "cache_invalidation_stream",
-    "public_room_list_stream",
-    "state_group_edges",
-    "stream_ordering_to_exterm",
 ]
 
 
@@ -114,16 +92,11 @@ class Store(object):
 
     _simple_select_onecol_txn = SQLBaseStore.__dict__["_simple_select_onecol_txn"]
     _simple_select_onecol = SQLBaseStore.__dict__["_simple_select_onecol"]
-    _simple_select_one = SQLBaseStore.__dict__["_simple_select_one"]
-    _simple_select_one_txn = SQLBaseStore.__dict__["_simple_select_one_txn"]
     _simple_select_one_onecol = SQLBaseStore.__dict__["_simple_select_one_onecol"]
-    _simple_select_one_onecol_txn = SQLBaseStore.__dict__[
-        "_simple_select_one_onecol_txn"
-    ]
+    _simple_select_one_onecol_txn = SQLBaseStore.__dict__["_simple_select_one_onecol_txn"]
 
     _simple_update_one = SQLBaseStore.__dict__["_simple_update_one"]
     _simple_update_one_txn = SQLBaseStore.__dict__["_simple_update_one_txn"]
-    _simple_update_txn = SQLBaseStore.__dict__["_simple_update_txn"]
 
     def runInteraction(self, desc, func, *args, **kwargs):
         def r(conn):
@@ -134,7 +107,7 @@ class Store(object):
                     try:
                         txn = conn.cursor()
                         return func(
-                            LoggingTransaction(txn, desc, self.database_engine, [], []),
+                            LoggingTransaction(txn, desc, self.database_engine, []),
                             *args, **kwargs
                         )
                     except self.database_engine.module.DatabaseError as e:
@@ -185,40 +158,31 @@ class Porter(object):
     def setup_table(self, table):
         if table in APPEND_ONLY_TABLES:
             # It's safe to just carry on inserting.
-            row = yield self.postgres_store._simple_select_one(
+            next_chunk = yield self.postgres_store._simple_select_one_onecol(
                 table="port_from_sqlite3",
                 keyvalues={"table_name": table},
-                retcols=("forward_rowid", "backward_rowid"),
+                retcol="rowid",
                 allow_none=True,
             )
 
             total_to_port = None
-            if row is None:
+            if next_chunk is None:
                 if table == "sent_transactions":
-                    forward_chunk, already_ported, total_to_port = (
+                    next_chunk, already_ported, total_to_port = (
                         yield self._setup_sent_transactions()
                     )
-                    backward_chunk = 0
                 else:
                     yield self.postgres_store._simple_insert(
                         table="port_from_sqlite3",
-                        values={
-                            "table_name": table,
-                            "forward_rowid": 1,
-                            "backward_rowid": 0,
-                        }
+                        values={"table_name": table, "rowid": 1}
                     )
 
-                    forward_chunk = 1
-                    backward_chunk = 0
+                    next_chunk = 1
                     already_ported = 0
-            else:
-                forward_chunk = row["forward_rowid"]
-                backward_chunk = row["backward_rowid"]
 
             if total_to_port is None:
                 already_ported, total_to_port = yield self._get_total_count_to_port(
-                    table, forward_chunk, backward_chunk
+                    table, next_chunk
                 )
         else:
             def delete_all(txn):
@@ -232,111 +196,47 @@ class Porter(object):
 
             yield self.postgres_store._simple_insert(
                 table="port_from_sqlite3",
-                values={
-                    "table_name": table,
-                    "forward_rowid": 1,
-                    "backward_rowid": 0,
-                }
+                values={"table_name": table, "rowid": 0}
             )
 
-            forward_chunk = 1
-            backward_chunk = 0
+            next_chunk = 1
 
             already_ported, total_to_port = yield self._get_total_count_to_port(
-                table, forward_chunk, backward_chunk
+                table, next_chunk
             )
 
-        defer.returnValue(
-            (table, already_ported, total_to_port, forward_chunk, backward_chunk)
-        )
+        defer.returnValue((table, already_ported, total_to_port, next_chunk))
 
     @defer.inlineCallbacks
-    def handle_table(self, table, postgres_size, table_size, forward_chunk,
-                     backward_chunk):
-        logger.info(
-            "Table %s: %i/%i (rows %i-%i) already ported",
-            table, postgres_size, table_size,
-            backward_chunk+1, forward_chunk-1,
-        )
-
+    def handle_table(self, table, postgres_size, table_size, next_chunk):
         if not table_size:
             return
 
         self.progress.add_table(table, postgres_size, table_size)
 
         if table == "event_search":
-            yield self.handle_search_table(
-                postgres_size, table_size, forward_chunk, backward_chunk
-            )
+            yield self.handle_search_table(postgres_size, table_size, next_chunk)
             return
 
-        if table in (
-            "user_directory", "user_directory_search", "users_who_share_rooms",
-            "users_in_pubic_room",
-        ):
-            # We don't port these tables, as they're a faff and we can regenreate
-            # them anyway.
-            self.progress.update(table, table_size)  # Mark table as done
-            return
-
-        if table == "user_directory_stream_pos":
-            # We need to make sure there is a single row, `(X, null), as that is
-            # what synapse expects to be there.
-            yield self.postgres_store._simple_insert(
-                table=table,
-                values={"stream_id": None},
-            )
-            self.progress.update(table, table_size)  # Mark table as done
-            return
-
-        forward_select = (
+        select = (
             "SELECT rowid, * FROM %s WHERE rowid >= ? ORDER BY rowid LIMIT ?"
             % (table,)
         )
 
-        backward_select = (
-            "SELECT rowid, * FROM %s WHERE rowid <= ? ORDER BY rowid LIMIT ?"
-            % (table,)
-        )
-
-        do_forward = [True]
-        do_backward = [True]
-
         while True:
             def r(txn):
-                forward_rows = []
-                backward_rows = []
-                if do_forward[0]:
-                    txn.execute(forward_select, (forward_chunk, self.batch_size,))
-                    forward_rows = txn.fetchall()
-                    if not forward_rows:
-                        do_forward[0] = False
-
-                if do_backward[0]:
-                    txn.execute(backward_select, (backward_chunk, self.batch_size,))
-                    backward_rows = txn.fetchall()
-                    if not backward_rows:
-                        do_backward[0] = False
-
-                if forward_rows or backward_rows:
+                txn.execute(select, (next_chunk, self.batch_size,))
+                rows = txn.fetchall()
                 headers = [column[0] for column in txn.description]
-                else:
-                    headers = None
 
-                return headers, forward_rows, backward_rows
+                return headers, rows
 
-            headers, frows, brows = yield self.sqlite_store.runInteraction(
-                "select", r
-            )
+            headers, rows = yield self.sqlite_store.runInteraction("select", r)
 
-            if frows or brows:
-                if frows:
-                    forward_chunk = max(row[0] for row in frows) + 1
-                if brows:
-                    backward_chunk = min(row[0] for row in brows) - 1
+            if rows:
+                next_chunk = rows[-1][0] + 1
 
-                rows = frows + brows
-                rows = self._convert_rows(table, headers, rows)
+                self._convert_rows(table, headers, rows)
 
                 def insert(txn):
                     self.postgres_store.insert_many_txn(
@@ -347,10 +247,7 @@ class Porter(object):
                         txn,
                         table="port_from_sqlite3",
                         keyvalues={"table_name": table},
-                        updatevalues={
-                            "forward_rowid": forward_chunk,
-                            "backward_rowid": backward_chunk,
-                        },
+                        updatevalues={"rowid": next_chunk},
                     )
 
                 yield self.postgres_store.execute(insert)
@@ -362,8 +259,7 @@ class Porter(object):
                 return
 
     @defer.inlineCallbacks
-    def handle_search_table(self, postgres_size, table_size, forward_chunk,
-                            backward_chunk):
+    def handle_search_table(self, postgres_size, table_size, next_chunk):
         select = (
             "SELECT es.rowid, es.*, e.origin_server_ts, e.stream_ordering"
             " FROM event_search as es"
@@ -374,7 +270,7 @@ class Porter(object):
 
         while True:
             def r(txn):
-                txn.execute(select, (forward_chunk, self.batch_size,))
+                txn.execute(select, (next_chunk, self.batch_size,))
                 rows = txn.fetchall()
                 headers = [column[0] for column in txn.description]
 
@@ -383,7 +279,7 @@ class Porter(object):
             headers, rows = yield self.sqlite_store.runInteraction("select", r)
 
             if rows:
-                forward_chunk = rows[-1][0] + 1
+                next_chunk = rows[-1][0] + 1
 
                 # We have to treat event_search differently since it has a
                 # different structure in the two different databases.
@@ -394,13 +290,10 @@ class Porter(object):
                         " VALUES (?,?,?,?,to_tsvector('english', ?),?,?)"
                     )
 
-                    rows_dict = []
-                    for row in rows:
-                        d = dict(zip(headers, row))
-                        if "\0" in d['value']:
-                            logger.warn('dropping search row %s', d)
-                        else:
-                            rows_dict.append(d)
+                    rows_dict = [
+                        dict(zip(headers, row))
+                        for row in rows
+                    ]
 
                     txn.executemany(sql, [
                         (
@@ -419,10 +312,7 @@ class Porter(object):
                         txn,
                         table="port_from_sqlite3",
                         keyvalues={"table_name": "event_search"},
-                        updatevalues={
-                            "forward_rowid": forward_chunk,
-                            "backward_rowid": backward_chunk,
-                        },
+                        updatevalues={"rowid": next_chunk},
                     )
 
                 yield self.postgres_store.execute(insert)
@@ -434,6 +324,7 @@ class Porter(object):
             else:
                 return
 
+
     def setup_db(self, db_config, database_engine):
         db_conn = database_engine.module.connect(
             **{
@@ -476,41 +367,6 @@ class Porter(object):
             self.progress.set_state("Preparing PostgreSQL")
             self.setup_db(postgres_config, postgres_engine)
 
-            self.progress.set_state("Creating port tables")
-            def create_port_table(txn):
-                txn.execute(
-                    "CREATE TABLE IF NOT EXISTS port_from_sqlite3 ("
-                    " table_name varchar(100) NOT NULL UNIQUE,"
-                    " forward_rowid bigint NOT NULL,"
-                    " backward_rowid bigint NOT NULL"
-                    ")"
-                )
-
-            # The old port script created a table with just a "rowid" column.
-            # We want people to be able to rerun this script from an old port
-            # so that they can pick up any missing events that were not
-            # ported across.
-            def alter_table(txn):
-                txn.execute(
-                    "ALTER TABLE IF EXISTS port_from_sqlite3"
-                    " RENAME rowid TO forward_rowid"
-                )
-                txn.execute(
-                    "ALTER TABLE IF EXISTS port_from_sqlite3"
-                    " ADD backward_rowid bigint NOT NULL DEFAULT 0"
-                )
-
-            try:
-                yield self.postgres_store.runInteraction(
-                    "alter_table", alter_table
-                )
-            except Exception as e:
-                pass
-
-            yield self.postgres_store.runInteraction(
-                "create_port_table", create_port_table
-            )
-
             # Step 2. Get tables.
             self.progress.set_state("Fetching tables")
             sqlite_tables = yield self.sqlite_store._simple_select_onecol(
@@ -523,15 +379,36 @@ class Porter(object):
 
             postgres_tables = yield self.postgres_store._simple_select_onecol(
                 table="information_schema.tables",
-                keyvalues={},
+                keyvalues={
+                    "table_schema": "public",
+                },
                 retcol="distinct table_name",
             )
 
             tables = set(sqlite_tables) & set(postgres_tables)
+
+            self.progress.set_state("Creating tables")
+
             logger.info("Found %d tables", len(tables))
 
-            # Step 3. Figure out what still needs copying
-            self.progress.set_state("Checking on port progress")
+            def create_port_table(txn):
+                txn.execute(
+                    "CREATE TABLE port_from_sqlite3 ("
+                    " table_name varchar(100) NOT NULL UNIQUE,"
+                    " rowid bigint NOT NULL"
+                    ")"
+                )
+
+            try:
+                yield self.postgres_store.runInteraction(
+                    "create_port_table", create_port_table
+                )
+            except Exception as e:
+                logger.info("Failed to create port table: %s", e)
+
+            self.progress.set_state("Setting up")
+
+            # Set up tables.
             setup_res = yield defer.gatherResults(
                 [
                     self.setup_table(table)
@@ -542,8 +419,7 @@ class Porter(object):
                 consumeErrors=True,
             )
 
-            # Step 4. Do the copying.
-            self.progress.set_state("Copying to postgres")
+            # Process tables.
             yield defer.gatherResults(
                 [
                     self.handle_table(*res)
@@ -552,9 +428,6 @@ class Porter(object):
                 consumeErrors=True,
             )
 
-            # Step 5. Do final post-processing
-            yield self._setup_state_group_id_seq()
-
             self.progress.done()
         except:
             global end_error_exec_info
@@ -570,29 +443,17 @@ class Porter(object):
             i for i, h in enumerate(headers) if h in bool_col_names
         ]
 
-        class BadValueException(Exception):
-            pass
-
         def conv(j, col):
             if j in bool_cols:
                 return bool(col)
-            elif isinstance(col, string_types) and "\0" in col:
-                logger.warn("DROPPING ROW: NUL value in table %s col %s: %r", table, headers[j], col)
-                raise BadValueException();
             return col
 
-        outrows = []
         for i, row in enumerate(rows):
-            try:
-                outrows.append(tuple(
+            rows[i] = tuple(
                 conv(j, col)
                 for j, col in enumerate(row)
                 if j > 0
-                ))
-            except BadValueException:
-                pass
-
-        return outrows
+            )
 
     @defer.inlineCallbacks
     def _setup_sent_transactions(self):
@@ -620,7 +481,7 @@ class Porter(object):
             "select", r,
         )
 
-        rows = self._convert_rows("sent_transactions", headers, rows)
+        self._convert_rows("sent_transactions", headers, rows)
 
         inserted_rows = len(rows)
         if inserted_rows:
@@ -653,11 +514,7 @@ class Porter(object):
 
         yield self.postgres_store._simple_insert(
             table="port_from_sqlite3",
-            values={
-                "table_name": "sent_transactions",
-                "forward_rowid": next_chunk,
-                "backward_rowid": 0,
-            }
+            values={"table_name": "sent_transactions", "rowid": next_chunk}
         )
 
         def get_sent_table_size(txn):
@@ -678,18 +535,13 @@ class Porter(object):
         defer.returnValue((next_chunk, inserted_rows, total_count))
 
     @defer.inlineCallbacks
-    def _get_remaining_count_to_port(self, table, forward_chunk, backward_chunk):
-        frows = yield self.sqlite_store.execute_sql(
+    def _get_remaining_count_to_port(self, table, next_chunk):
+        rows = yield self.sqlite_store.execute_sql(
             "SELECT count(*) FROM %s WHERE rowid >= ?" % (table,),
-            forward_chunk,
+            next_chunk,
         )
 
-        brows = yield self.sqlite_store.execute_sql(
-            "SELECT count(*) FROM %s WHERE rowid <= ?" % (table,),
-            backward_chunk,
-        )
-
-        defer.returnValue(frows[0][0] + brows[0][0])
+        defer.returnValue(rows[0][0])
 
     @defer.inlineCallbacks
     def _get_already_ported_count(self, table):
@@ -700,10 +552,10 @@ class Porter(object):
         defer.returnValue(rows[0][0])
 
     @defer.inlineCallbacks
-    def _get_total_count_to_port(self, table, forward_chunk, backward_chunk):
+    def _get_total_count_to_port(self, table, next_chunk):
         remaining, done = yield defer.gatherResults(
             [
-                self._get_remaining_count_to_port(table, forward_chunk, backward_chunk),
+                self._get_remaining_count_to_port(table, next_chunk),
                 self._get_already_ported_count(table),
             ],
             consumeErrors=True,
@@ -714,16 +566,6 @@ class Porter(object):
 
         defer.returnValue((done, remaining + done))
 
-    def _setup_state_group_id_seq(self):
-        def r(txn):
-            txn.execute("SELECT MAX(id) FROM state_groups")
-            next_id = txn.fetchone()[0]+1
-            txn.execute(
-                "ALTER SEQUENCE state_group_id_seq RESTART WITH %s",
-                (next_id,),
-            )
-        return self.postgres_store.runInteraction("setup_state_group_id_seq", r)
-
 
 ##############################################
 ###### The following is simply UI stuff ######

scripts/sync_room_to_group.pl

@@ -1,45 +0,0 @@
-#!/usr/bin/env perl
-
-use strict;
-use warnings;
-
-use JSON::XS;
-use LWP::UserAgent;
-use URI::Escape;
-
-if (@ARGV < 4) {
-    die "usage: $0 <homeserver url> <access_token> <room_id|room_alias> <group_id>\n";
-}
-
-my ($hs, $access_token, $room_id, $group_id) = @ARGV;
-my $ua = LWP::UserAgent->new();
-$ua->timeout(10);
-
-if ($room_id =~ /^#/) {
-    $room_id = uri_escape($room_id);
-    $room_id = decode_json($ua->get("${hs}/_matrix/client/r0/directory/room/${room_id}?access_token=${access_token}")->decoded_content)->{room_id};
-}
-
-my $room_users  = [ keys %{decode_json($ua->get("${hs}/_matrix/client/r0/rooms/${room_id}/joined_members?access_token=${access_token}")->decoded_content)->{joined}} ];
-my $group_users = [
-    (map { $_->{user_id} } @{decode_json($ua->get("${hs}/_matrix/client/unstable/groups/${group_id}/users?access_token=${access_token}" )->decoded_content)->{chunk}}),
-    (map { $_->{user_id} } @{decode_json($ua->get("${hs}/_matrix/client/unstable/groups/${group_id}/invited_users?access_token=${access_token}" )->decoded_content)->{chunk}}),
-];
-
-die "refusing to sync from empty room" unless (@$room_users);
-die "refusing to sync to empty group" unless (@$group_users);
-
-my $diff = {};
-foreach my $user (@$room_users) { $diff->{$user}++ }
-foreach my $user (@$group_users) { $diff->{$user}-- }
-
-foreach my $user (keys %$diff) {
-    if ($diff->{$user} == 1) {
-        warn "inviting $user";
-        print STDERR $ua->put("${hs}/_matrix/client/unstable/groups/${group_id}/admin/users/invite/${user}?access_token=${access_token}", Content=>'{}')->status_line."\n";
-    }
-    elsif ($diff->{$user} == -1) {
-        warn "removing $user";
-        print STDERR $ua->put("${hs}/_matrix/client/unstable/groups/${group_id}/admin/users/remove/${user}?access_token=${access_token}", Content=>'{}')->status_line."\n";
-    }
-}

setup.cfg

@@ -16,5 +16,7 @@ ignore =
 
 [flake8]
 max-line-length = 90
-#  W503 requires that binary operators be at the end, not start, of lines. Erik doesn't like it.
-ignore = W503
+ignore = W503 ; W503 requires that binary operators be at the end, not start, of lines. Erik doesn't like it.
+
+[pep8]
+max-line-length = 90

setup.py

@@ -23,45 +23,6 @@ import sys
 here = os.path.abspath(os.path.dirname(__file__))
 
 
-# Some notes on `setup.py test`:
-#
-# Once upon a time we used to try to make `setup.py test` run `tox` to run the
-# tests. That's a bad idea for three reasons:
-#
-# 1: `setup.py test` is supposed to find out whether the tests work in the
-#    *current* environmentt, not whatever tox sets up.
-# 2: Empirically, trying to install tox during the test run wasn't working ("No
-#    module named virtualenv").
-# 3: The tox documentation advises against it[1].
-#
-# Even further back in time, we used to use setuptools_trial [2]. That has its
-# own set of issues: for instance, it requires installation of Twisted to build
-# an sdist (because the recommended mode of usage is to add it to
-# `setup_requires`). That in turn means that in order to successfully run tox
-# you have to have the python header files installed for whichever version of
-# python tox uses (which is python3 on recent ubuntus, for example).
-#
-# So, for now at least, we stick with what appears to be the convention among
-# Twisted projects, and don't attempt to do anything when someone runs
-# `setup.py test`; instead we direct people to run `trial` directly if they
-# care.
-#
-# [1]: http://tox.readthedocs.io/en/2.5.0/example/basic.html#integration-with-setup-py-test-command
-# [2]: https://pypi.python.org/pypi/setuptools_trial
-class TestCommand(Command):
-    user_options = []
-
-    def initialize_options(self):
-        pass
-
-    def finalize_options(self):
-        pass
-
-    def run(self):
-        print ("""Synapse's tests cannot be run via setup.py. To run them, try:
-     PYTHONPATH="." trial tests
-""")
-
 def read_file(path_segments):
     """Read a file from the package. Takes a list of strings to join to
     make the path"""
@@ -78,6 +39,38 @@ def exec_file(path_segments):
     return result
 
 
+class Tox(Command):
+    user_options = [('tox-args=', 'a', "Arguments to pass to tox")]
+
+    def initialize_options(self):
+        self.tox_args = None
+
+    def finalize_options(self):
+        self.test_args = []
+        self.test_suite = True
+
+    def run(self):
+        #import here, cause outside the eggs aren't loaded
+        try:
+            import tox
+        except ImportError:
+            try:
+                self.distribution.fetch_build_eggs("tox")
+                import tox
+            except:
+                raise RuntimeError(
+                    "The tests need 'tox' to run. Please install 'tox'."
+                )
+        import shlex
+        args = self.tox_args
+        if args:
+            args = shlex.split(self.tox_args)
+        else:
+            args = []
+        errno = tox.cmdline(args=args)
+        sys.exit(errno)
+
+
 version = exec_file(("synapse", "__init__.py"))["__version__"]
 dependencies = exec_file(("synapse", "python_dependencies.py"))
 long_description = read_file(("README.rst",))
@@ -93,5 +86,5 @@ setup(
     zip_safe=False,
     long_description=long_description,
     scripts=["synctl"] + glob.glob("scripts/*"),
-    cmdclass={'test': TestCommand},
+    cmdclass={'test': Tox},
 )

synapse/__init__.py

@@ -16,4 +16,4 @@
 """ This is a reference implementation of a Matrix home server.
 """
 
-__version__ = "0.27.3-rc2"
+__version__ = "0.16.1-r1"

synapse/api/constants.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2017 Vector Creations Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -44,8 +43,10 @@ class JoinRules(object):
 
 class LoginType(object):
     PASSWORD = u"m.login.password"
+    OAUTH = u"m.login.oauth2"
+    EMAIL_CODE = u"m.login.email.code"
+    EMAIL_URL = u"m.login.email.url"
     EMAIL_IDENTITY = u"m.login.email.identity"
-    MSISDN = u"m.login.msisdn"
     RECAPTCHA = u"m.login.recaptcha"
     DUMMY = u"m.login.dummy"
 
@@ -84,8 +85,3 @@ class RoomCreationPreset(object):
     PRIVATE_CHAT = "private_chat"
     PUBLIC_CHAT = "public_chat"
     TRUSTED_PRIVATE_CHAT = "trusted_private_chat"
-
-
-class ThirdPartyEntityKind(object):
-    USER = "user"
-    LOCATION = "location"

synapse/api/errors.py

@@ -17,8 +17,6 @@
 
 import logging
 
-import simplejson as json
-
 logger = logging.getLogger(__name__)
 
 
@@ -41,58 +39,36 @@ class Codes(object):
     CAPTCHA_NEEDED = "M_CAPTCHA_NEEDED"
     CAPTCHA_INVALID = "M_CAPTCHA_INVALID"
     MISSING_PARAM = "M_MISSING_PARAM"
-    INVALID_PARAM = "M_INVALID_PARAM"
     TOO_LARGE = "M_TOO_LARGE"
     EXCLUSIVE = "M_EXCLUSIVE"
     THREEPID_AUTH_FAILED = "M_THREEPID_AUTH_FAILED"
     THREEPID_IN_USE = "M_THREEPID_IN_USE"
-    THREEPID_NOT_FOUND = "M_THREEPID_NOT_FOUND"
-    THREEPID_DENIED = "M_THREEPID_DENIED"
     INVALID_USERNAME = "M_INVALID_USERNAME"
     SERVER_NOT_TRUSTED = "M_SERVER_NOT_TRUSTED"
 
 
 class CodeMessageException(RuntimeError):
-    """An exception with integer code and message string attributes.
+    """An exception with integer code and message string attributes."""
 
-    Attributes:
-        code (int): HTTP error code
-        msg (str): string describing the error
-    """
     def __init__(self, code, msg):
         super(CodeMessageException, self).__init__("%d: %s" % (code, msg))
         self.code = code
         self.msg = msg
+        self.response_code_message = None
 
     def error_dict(self):
         return cs_error(self.msg)
 
 
-class MatrixCodeMessageException(CodeMessageException):
-    """An error from a general matrix endpoint, eg. from a proxied Matrix API call.
-
-    Attributes:
-        errcode (str): Matrix error code e.g 'M_FORBIDDEN'
-    """
-    def __init__(self, code, msg, errcode=Codes.UNKNOWN):
-        super(MatrixCodeMessageException, self).__init__(code, msg)
-        self.errcode = errcode
-
-
 class SynapseError(CodeMessageException):
-    """A base exception type for matrix errors which have an errcode and error
-    message (as well as an HTTP status code).
-
-    Attributes:
-        errcode (str): Matrix error code e.g 'M_FORBIDDEN'
-    """
+    """A base error which can be caught for all synapse events."""
     def __init__(self, code, msg, errcode=Codes.UNKNOWN):
         """Constructs a synapse error.
 
         Args:
             code (int): The integer error code (an HTTP response code)
             msg (str): The human-readable error message.
-            errcode (str): The matrix error code e.g 'M_FORBIDDEN'
+            err (str): The error code e.g 'M_FORBIDDEN'
         """
         super(SynapseError, self).__init__(code, msg)
         self.errcode = errcode
@@ -103,87 +79,12 @@ class SynapseError(CodeMessageException):
             self.errcode,
         )
 
-    @classmethod
-    def from_http_response_exception(cls, err):
-        """Make a SynapseError based on an HTTPResponseException
-
-        This is useful when a proxied request has failed, and we need to
-        decide how to map the failure onto a matrix error to send back to the
-        client.
-
-        An attempt is made to parse the body of the http response as a matrix
-        error. If that succeeds, the errcode and error message from the body
-        are used as the errcode and error message in the new synapse error.
-
-        Otherwise, the errcode is set to M_UNKNOWN, and the error message is
-        set to the reason code from the HTTP response.
-
-        Args:
-            err (HttpResponseException):
-
-        Returns:
-            SynapseError:
-        """
-        # try to parse the body as json, to get better errcode/msg, but
-        # default to M_UNKNOWN with the HTTP status as the error text
-        try:
-            j = json.loads(err.response)
-        except ValueError:
-            j = {}
-        errcode = j.get('errcode', Codes.UNKNOWN)
-        errmsg = j.get('error', err.msg)
-
-        res = SynapseError(err.code, errmsg, errcode)
-        return res
-
 
 class RegistrationError(SynapseError):
     """An error raised when a registration event fails."""
     pass
 
 
-class FederationDeniedError(SynapseError):
-    """An error raised when the server tries to federate with a server which
-    is not on its federation whitelist.
-
-    Attributes:
-        destination (str): The destination which has been denied
-    """
-
-    def __init__(self, destination):
-        """Raised by federation client or server to indicate that we are
-        are deliberately not attempting to contact a given server because it is
-        not on our federation whitelist.
-
-        Args:
-            destination (str): the domain in question
-        """
-
-        self.destination = destination
-
-        super(FederationDeniedError, self).__init__(
-            code=403,
-            msg="Federation denied with %s." % (self.destination,),
-            errcode=Codes.FORBIDDEN,
-        )
-
-
-class InteractiveAuthIncompleteError(Exception):
-    """An error raised when UI auth is not yet complete
-
-    (This indicates we should return a 401 with 'result' as the body)
-
-    Attributes:
-        result (dict): the server response to the request, which should be
-            passed back to the client
-    """
-    def __init__(self, result):
-        super(InteractiveAuthIncompleteError, self).__init__(
-            "Interactive auth not yet complete",
-        )
-        self.result = result
-
-
 class UnrecognizedRequestError(SynapseError):
     """An error indicating we don't understand the request you're trying to make"""
     def __init__(self, *args, **kwargs):
@@ -203,11 +104,13 @@ class UnrecognizedRequestError(SynapseError):
 
 class NotFoundError(SynapseError):
     """An error indicating we can't find the thing you asked for"""
-    def __init__(self, msg="Not found", errcode=Codes.NOT_FOUND):
+    def __init__(self, *args, **kwargs):
+        if "errcode" not in kwargs:
+            kwargs["errcode"] = Codes.NOT_FOUND
         super(NotFoundError, self).__init__(
             404,
-            msg,
-            errcode=errcode
+            "Not found",
+            **kwargs
         )
 
 
@@ -268,6 +171,7 @@ class LimitExceededError(SynapseError):
                  errcode=Codes.LIMIT_EXCEEDED):
         super(LimitExceededError, self).__init__(code, msg, errcode)
         self.retry_after_ms = retry_after_ms
+        self.response_code_message = "Too Many Requests"
 
     def error_dict(self):
         return cs_error(
@@ -337,19 +241,6 @@ class FederationError(RuntimeError):
 
 
 class HttpResponseException(CodeMessageException):
-    """
-    Represents an HTTP-level failure of an outbound request
-
-    Attributes:
-        response (str): body of response
-    """
     def __init__(self, code, msg, response):
-        """
-
-        Args:
-            code (int): HTTP status code
-            msg (str): reason phrase from HTTP response status line
-            response (str): body of response
-        """
-        super(HttpResponseException, self).__init__(code, msg)
         self.response = response
+        super(HttpResponseException, self).__init__(code, msg)

synapse/api/filtering.py

@@ -13,174 +13,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 from synapse.api.errors import SynapseError
-from synapse.storage.presence import UserPresenceState
 from synapse.types import UserID, RoomID
+
 from twisted.internet import defer
 
-import simplejson as json
-import jsonschema
-from jsonschema import FormatChecker
-
-FILTER_SCHEMA = {
-    "additionalProperties": False,
-    "type": "object",
-    "properties": {
-        "limit": {
-            "type": "number"
-        },
-        "senders": {
-            "$ref": "#/definitions/user_id_array"
-        },
-        "not_senders": {
-            "$ref": "#/definitions/user_id_array"
-        },
-        # TODO: We don't limit event type values but we probably should...
-        # check types are valid event types
-        "types": {
-            "type": "array",
-            "items": {
-                "type": "string"
-            }
-        },
-        "not_types": {
-            "type": "array",
-            "items": {
-                "type": "string"
-            }
-        }
-    }
-}
-
-ROOM_FILTER_SCHEMA = {
-    "additionalProperties": False,
-    "type": "object",
-    "properties": {
-        "not_rooms": {
-            "$ref": "#/definitions/room_id_array"
-        },
-        "rooms": {
-            "$ref": "#/definitions/room_id_array"
-        },
-        "ephemeral": {
-            "$ref": "#/definitions/room_event_filter"
-        },
-        "include_leave": {
-            "type": "boolean"
-        },
-        "state": {
-            "$ref": "#/definitions/room_event_filter"
-        },
-        "timeline": {
-            "$ref": "#/definitions/room_event_filter"
-        },
-        "account_data": {
-            "$ref": "#/definitions/room_event_filter"
-        },
-    }
-}
-
-ROOM_EVENT_FILTER_SCHEMA = {
-    "additionalProperties": False,
-    "type": "object",
-    "properties": {
-        "limit": {
-            "type": "number"
-        },
-        "senders": {
-            "$ref": "#/definitions/user_id_array"
-        },
-        "not_senders": {
-            "$ref": "#/definitions/user_id_array"
-        },
-        "types": {
-            "type": "array",
-            "items": {
-                "type": "string"
-            }
-        },
-        "not_types": {
-            "type": "array",
-            "items": {
-                "type": "string"
-            }
-        },
-        "rooms": {
-            "$ref": "#/definitions/room_id_array"
-        },
-        "not_rooms": {
-            "$ref": "#/definitions/room_id_array"
-        },
-        "contains_url": {
-            "type": "boolean"
-        }
-    }
-}
-
-USER_ID_ARRAY_SCHEMA = {
-    "type": "array",
-    "items": {
-        "type": "string",
-        "format": "matrix_user_id"
-    }
-}
-
-ROOM_ID_ARRAY_SCHEMA = {
-    "type": "array",
-    "items": {
-        "type": "string",
-        "format": "matrix_room_id"
-    }
-}
-
-USER_FILTER_SCHEMA = {
-    "$schema": "http://json-schema.org/draft-04/schema#",
-    "description": "schema for a Sync filter",
-    "type": "object",
-    "definitions": {
-        "room_id_array": ROOM_ID_ARRAY_SCHEMA,
-        "user_id_array": USER_ID_ARRAY_SCHEMA,
-        "filter": FILTER_SCHEMA,
-        "room_filter": ROOM_FILTER_SCHEMA,
-        "room_event_filter": ROOM_EVENT_FILTER_SCHEMA
-    },
-    "properties": {
-        "presence": {
-            "$ref": "#/definitions/filter"
-        },
-        "account_data": {
-            "$ref": "#/definitions/filter"
-        },
-        "room": {
-            "$ref": "#/definitions/room_filter"
-        },
-        "event_format": {
-            "type": "string",
-            "enum": ["client", "federation"]
-        },
-        "event_fields": {
-            "type": "array",
-            "items": {
-                "type": "string",
-                # Don't allow '\\' in event field filters. This makes matching
-                # events a lot easier as we can then use a negative lookbehind
-                # assertion to split '\.' If we allowed \\ then it would
-                # incorrectly split '\\.' See synapse.events.utils.serialize_event
-                "pattern": "^((?!\\\).)*$"
-            }
-        }
-    },
-    "additionalProperties": False
-}
-
-
-@FormatChecker.cls_checks('matrix_room_id')
-def matrix_room_id_validator(room_id_str):
-    return RoomID.from_string(room_id_str)
-
-
-@FormatChecker.cls_checks('matrix_user_id')
-def matrix_user_id_validator(user_id_str):
-    return UserID.from_string(user_id_str)
+import ujson as json
 
 
 class Filtering(object):
@@ -215,11 +52,83 @@ class Filtering(object):
         # NB: Filters are the complete json blobs. "Definitions" are an
         # individual top-level key e.g. public_user_data. Filters are made of
         # many definitions.
-        try:
-            jsonschema.validate(user_filter_json, USER_FILTER_SCHEMA,
-                                format_checker=FormatChecker())
-        except jsonschema.ValidationError as e:
-            raise SynapseError(400, e.message)
+
+        top_level_definitions = [
+            "presence", "account_data"
+        ]
+
+        room_level_definitions = [
+            "state", "timeline", "ephemeral", "account_data"
+        ]
+
+        for key in top_level_definitions:
+            if key in user_filter_json:
+                self._check_definition(user_filter_json[key])
+
+        if "room" in user_filter_json:
+            self._check_definition_room_lists(user_filter_json["room"])
+            for key in room_level_definitions:
+                if key in user_filter_json["room"]:
+                    self._check_definition(user_filter_json["room"][key])
+
+    def _check_definition_room_lists(self, definition):
+        """Check that "rooms" and "not_rooms" are lists of room ids if they
+        are present
+
+        Args:
+            definition(dict): The filter definition
+        Raises:
+            SynapseError: If there was a problem with this definition.
+        """
+        # check rooms are valid room IDs
+        room_id_keys = ["rooms", "not_rooms"]
+        for key in room_id_keys:
+            if key in definition:
+                if type(definition[key]) != list:
+                    raise SynapseError(400, "Expected %s to be a list." % key)
+                for room_id in definition[key]:
+                    RoomID.from_string(room_id)
+
+    def _check_definition(self, definition):
+        """Check if the provided definition is valid.
+
+        This inspects not only the types but also the values to make sure they
+        make sense.
+
+        Args:
+            definition(dict): The filter definition
+        Raises:
+            SynapseError: If there was a problem with this definition.
+        """
+        # NB: Filters are the complete json blobs. "Definitions" are an
+        # individual top-level key e.g. public_user_data. Filters are made of
+        # many definitions.
+        if type(definition) != dict:
+            raise SynapseError(
+                400, "Expected JSON object, not %s" % (definition,)
+            )
+
+        self._check_definition_room_lists(definition)
+
+        # check senders are valid user IDs
+        user_id_keys = ["senders", "not_senders"]
+        for key in user_id_keys:
+            if key in definition:
+                if type(definition[key]) != list:
+                    raise SynapseError(400, "Expected %s to be a list." % key)
+                for user_id in definition[key]:
+                    UserID.from_string(user_id)
+
+        # TODO: We don't limit event type values but we probably should...
+        # check types are valid event types
+        event_keys = ["types", "not_types"]
+        for key in event_keys:
+            if key in definition:
+                if type(definition[key]) != list:
+                    raise SynapseError(400, "Expected %s to be a list." % key)
+                for event_type in definition[key]:
+                    if not isinstance(event_type, basestring):
+                        raise SynapseError(400, "Event type should be a string")
 
 
 class FilterCollection(object):
@@ -243,7 +152,6 @@ class FilterCollection(object):
         self.include_leave = filter_json.get("room", {}).get(
             "include_leave", False
         )
-        self.event_fields = filter_json.get("event_fields", [])
 
     def __repr__(self):
         return "<FilterCollection %s>" % (json.dumps(self._filter_json),)
@@ -278,89 +186,32 @@ class FilterCollection(object):
     def filter_room_account_data(self, events):
         return self._room_account_data.filter(self._room_filter.filter(events))
 
-    def blocks_all_presence(self):
-        return (
-            self._presence_filter.filters_all_types() or
-            self._presence_filter.filters_all_senders()
-        )
-
-    def blocks_all_room_ephemeral(self):
-        return (
-            self._room_ephemeral_filter.filters_all_types() or
-            self._room_ephemeral_filter.filters_all_senders() or
-            self._room_ephemeral_filter.filters_all_rooms()
-        )
-
-    def blocks_all_room_timeline(self):
-        return (
-            self._room_timeline_filter.filters_all_types() or
-            self._room_timeline_filter.filters_all_senders() or
-            self._room_timeline_filter.filters_all_rooms()
-        )
-
 
 class Filter(object):
     def __init__(self, filter_json):
         self.filter_json = filter_json
 
-        self.types = self.filter_json.get("types", None)
-        self.not_types = self.filter_json.get("not_types", [])
-
-        self.rooms = self.filter_json.get("rooms", None)
-        self.not_rooms = self.filter_json.get("not_rooms", [])
-
-        self.senders = self.filter_json.get("senders", None)
-        self.not_senders = self.filter_json.get("not_senders", [])
-
-        self.contains_url = self.filter_json.get("contains_url", None)
-
-    def filters_all_types(self):
-        return "*" in self.not_types
-
-    def filters_all_senders(self):
-        return "*" in self.not_senders
-
-    def filters_all_rooms(self):
-        return "*" in self.not_rooms
-
     def check(self, event):
         """Checks whether the filter matches the given event.
 
         Returns:
             bool: True if the event matches
         """
-        # We usually get the full "events" as dictionaries coming through,
-        # except for presence which actually gets passed around as its own
-        # namedtuple type.
-        if isinstance(event, UserPresenceState):
-            sender = event.user_id
-            room_id = None
-            ev_type = "m.presence"
-            is_url = False
-        else:
         sender = event.get("sender", None)
         if not sender:
-                # Presence events had their 'sender' in content.user_id, but are
-                # now handled above. We don't know if anything else uses this
-                # form. TODO: Check this and probably remove it.
+            # Presence events have their 'sender' in content.user_id
             content = event.get("content")
-                # account_data has been allowed to have non-dict content, so
-                # check type first
+            # account_data has been allowed to have non-dict content, so check type first
             if isinstance(content, dict):
                 sender = content.get("user_id")
 
-            room_id = event.get("room_id", None)
-            ev_type = event.get("type", None)
-            is_url = "url" in event.get("content", {})
-
         return self.check_fields(
-            room_id,
+            event.get("room_id", None),
             sender,
-            ev_type,
-            is_url,
+            event.get("type", None),
         )
 
-    def check_fields(self, room_id, sender, event_type, contains_url):
+    def check_fields(self, room_id, sender, event_type):
         """Checks whether the filter matches the given event fields.
 
         Returns:
@@ -374,20 +225,15 @@ class Filter(object):
 
         for name, match_func in literal_keys.items():
             not_name = "not_%s" % (name,)
-            disallowed_values = getattr(self, not_name)
+            disallowed_values = self.filter_json.get(not_name, [])
             if any(map(match_func, disallowed_values)):
                 return False
 
-            allowed_values = getattr(self, name)
+            allowed_values = self.filter_json.get(name, None)
             if allowed_values is not None:
                 if not any(map(match_func, allowed_values)):
                     return False
 
-        contains_url_filter = self.filter_json.get("contains_url")
-        if contains_url_filter is not None:
-            if contains_url_filter != contains_url:
-                return False
-
         return True
 
     def filter_rooms(self, room_ids):

synapse/api/ratelimiting.py

@@ -23,7 +23,7 @@ class Ratelimiter(object):
     def __init__(self):
         self.message_counts = collections.OrderedDict()
 
-    def send_message(self, user_id, time_now_s, msg_rate_hz, burst_count, update=True):
+    def send_message(self, user_id, time_now_s, msg_rate_hz, burst_count):
         """Can the user send a message?
         Args:
             user_id: The user sending a message.
@@ -32,15 +32,12 @@ class Ratelimiter(object):
                 second.
             burst_count: How many messages the user can send before being
                 limited.
-            update (bool): Whether to update the message rates or not. This is
-                useful to check if a message would be allowed to be sent before
-                its ready to be actually sent.
         Returns:
             A pair of a bool indicating if they can send a message now and a
                 time in seconds of when they can next send a message.
         """
         self.prune_message_counts(time_now_s)
-        message_count, time_start, _ignored = self.message_counts.get(
+        message_count, time_start, _ignored = self.message_counts.pop(
             user_id, (0., time_now_s, None),
         )
         time_delta = time_now_s - time_start
@@ -55,7 +52,6 @@ class Ratelimiter(object):
             allowed = True
             message_count += 1
 
-        if update:
         self.message_counts[user_id] = (
             message_count, time_start, msg_rate_hz
         )

synapse/api/urls.py

@@ -25,3 +25,4 @@ SERVER_KEY_PREFIX = "/_matrix/key/v1"
 SERVER_KEY_V2_PREFIX = "/_matrix/key/v2"
 MEDIA_PREFIX = "/_matrix/media/r0"
 LEGACY_MEDIA_PREFIX = "/_matrix/media/v1"
+APP_SERVICE_PREFIX = "/_matrix/appservice/v1"

synapse/app/__init__.py

@@ -16,11 +16,13 @@
 import sys
 sys.dont_write_bytecode = True
 
-from synapse import python_dependencies   # noqa: E402
+from synapse.python_dependencies import (
+    check_requirements, MissingRequirementError
+)  # NOQA
 
 try:
-    python_dependencies.check_requirements()
-except python_dependencies.MissingRequirementError as e:
+    check_requirements()
+except MissingRequirementError as e:
     message = "\n".join([
         "Missing Requirement: %s" % (e.message,),
         "To install run:",

synapse/app/_base.py

@@ -1,178 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2017 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import gc
-import logging
-import sys
-
-try:
-    import affinity
-except Exception:
-    affinity = None
-
-from daemonize import Daemonize
-from synapse.util import PreserveLoggingContext
-from synapse.util.rlimit import change_resource_limit
-from twisted.internet import error, reactor
-
-logger = logging.getLogger(__name__)
-
-
-def start_worker_reactor(appname, config):
-    """ Run the reactor in the main process
-
-    Daemonizes if necessary, and then configures some resources, before starting
-    the reactor. Pulls configuration from the 'worker' settings in 'config'.
-
-    Args:
-        appname (str): application name which will be sent to syslog
-        config (synapse.config.Config): config object
-    """
-
-    logger = logging.getLogger(config.worker_app)
-
-    start_reactor(
-        appname,
-        config.soft_file_limit,
-        config.gc_thresholds,
-        config.worker_pid_file,
-        config.worker_daemonize,
-        config.worker_cpu_affinity,
-        logger,
-    )
-
-
-def start_reactor(
-        appname,
-        soft_file_limit,
-        gc_thresholds,
-        pid_file,
-        daemonize,
-        cpu_affinity,
-        logger,
-):
-    """ Run the reactor in the main process
-
-    Daemonizes if necessary, and then configures some resources, before starting
-    the reactor
-
-    Args:
-        appname (str): application name which will be sent to syslog
-        soft_file_limit (int):
-        gc_thresholds:
-        pid_file (str): name of pid file to write to if daemonize is True
-        daemonize (bool): true to run the reactor in a background process
-        cpu_affinity (int|None): cpu affinity mask
-        logger (logging.Logger): logger instance to pass to Daemonize
-    """
-
-    def run():
-        # make sure that we run the reactor with the sentinel log context,
-        # otherwise other PreserveLoggingContext instances will get confused
-        # and complain when they see the logcontext arbitrarily swapping
-        # between the sentinel and `run` logcontexts.
-        with PreserveLoggingContext():
-            logger.info("Running")
-            if cpu_affinity is not None:
-                if not affinity:
-                    quit_with_error(
-                        "Missing package 'affinity' required for cpu_affinity\n"
-                        "option\n\n"
-                        "Install by running:\n\n"
-                        "   pip install affinity\n\n"
-                    )
-                logger.info("Setting CPU affinity to %s" % cpu_affinity)
-                affinity.set_process_affinity_mask(0, cpu_affinity)
-            change_resource_limit(soft_file_limit)
-            if gc_thresholds:
-                gc.set_threshold(*gc_thresholds)
-            reactor.run()
-
-    if daemonize:
-        daemon = Daemonize(
-            app=appname,
-            pid=pid_file,
-            action=run,
-            auto_close_fds=False,
-            verbose=True,
-            logger=logger,
-        )
-        daemon.start()
-    else:
-        run()
-
-
-def quit_with_error(error_string):
-    message_lines = error_string.split("\n")
-    line_length = max([len(l) for l in message_lines if len(l) < 80]) + 2
-    sys.stderr.write("*" * line_length + '\n')
-    for line in message_lines:
-        sys.stderr.write(" %s\n" % (line.rstrip(),))
-    sys.stderr.write("*" * line_length + '\n')
-    sys.exit(1)
-
-
-def listen_tcp(bind_addresses, port, factory, backlog=50):
-    """
-    Create a TCP socket for a port and several addresses
-    """
-    for address in bind_addresses:
-        try:
-            reactor.listenTCP(
-                port,
-                factory,
-                backlog,
-                address
-            )
-        except error.CannotListenError as e:
-            check_bind_error(e, address, bind_addresses)
-
-
-def listen_ssl(bind_addresses, port, factory, context_factory, backlog=50):
-    """
-    Create an SSL socket for a port and several addresses
-    """
-    for address in bind_addresses:
-        try:
-            reactor.listenSSL(
-                port,
-                factory,
-                context_factory,
-                backlog,
-                address
-            )
-        except error.CannotListenError as e:
-            check_bind_error(e, address, bind_addresses)
-
-
-def check_bind_error(e, address, bind_addresses):
-    """
-    This method checks an exception occurred while binding on 0.0.0.0.
-    If :: is specified in the bind addresses a warning is shown.
-    The exception is still raised otherwise.
-
-    Binding on both 0.0.0.0 and :: causes an exception on Linux and macOS
-    because :: binds on both IPv4 and IPv6 (as per RFC 3493).
-    When binding on 0.0.0.0 after :: this can safely be ignored.
-
-    Args:
-        e (Exception): Exception that was caught.
-        address (str): Address on which binding was attempted.
-        bind_addresses (list): Addresses on which the service listens.
-    """
-    if address == '0.0.0.0' and '::' in bind_addresses:
-        logger.warn('Failed to listen on 0.0.0.0, continuing because listening on [::]')
-    else:
-        raise e

synapse/app/appservice.py

@@ -1,171 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-# Copyright 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-import logging
-import sys
-
-import synapse
-from synapse import events
-from synapse.app import _base
-from synapse.config._base import ConfigError
-from synapse.config.homeserver import HomeServerConfig
-from synapse.config.logger import setup_logging
-from synapse.http.site import SynapseSite
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
-from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
-from synapse.replication.slave.storage.directory import DirectoryStore
-from synapse.replication.slave.storage.events import SlavedEventStore
-from synapse.replication.slave.storage.registration import SlavedRegistrationStore
-from synapse.replication.tcp.client import ReplicationClientHandler
-from synapse.server import HomeServer
-from synapse.storage.engines import create_engine
-from synapse.util.httpresourcetree import create_resource_tree
-from synapse.util.logcontext import LoggingContext, preserve_fn
-from synapse.util.manhole import manhole
-from synapse.util.versionstring import get_version_string
-from twisted.internet import reactor
-from twisted.web.resource import NoResource
-
-logger = logging.getLogger("synapse.app.appservice")
-
-
-class AppserviceSlaveStore(
-    DirectoryStore, SlavedEventStore, SlavedApplicationServiceStore,
-    SlavedRegistrationStore,
-):
-    pass
-
-
-class AppserviceServer(HomeServer):
-    def setup(self):
-        logger.info("Setting up.")
-        self.datastore = AppserviceSlaveStore(self.get_db_conn(), self)
-        logger.info("Finished setting up.")
-
-    def _listen_http(self, listener_config):
-        port = listener_config["port"]
-        bind_addresses = listener_config["bind_addresses"]
-        site_tag = listener_config.get("tag", port)
-        resources = {}
-        for res in listener_config["resources"]:
-            for name in res["names"]:
-                if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(self)
-
-        root_resource = create_resource_tree(resources, NoResource())
-
-        _base.listen_tcp(
-            bind_addresses,
-            port,
-            SynapseSite(
-                "synapse.access.http.%s" % (site_tag,),
-                site_tag,
-                listener_config,
-                root_resource,
-            )
-        )
-
-        logger.info("Synapse appservice now listening on port %d", port)
-
-    def start_listening(self, listeners):
-        for listener in listeners:
-            if listener["type"] == "http":
-                self._listen_http(listener)
-            elif listener["type"] == "manhole":
-                _base.listen_tcp(
-                    listener["bind_addresses"],
-                    listener["port"],
-                    manhole(
-                        username="matrix",
-                        password="rabbithole",
-                        globals={"hs": self},
-                    )
-                )
-            else:
-                logger.warn("Unrecognized listener type: %s", listener["type"])
-
-        self.get_tcp_replication().start_replication(self)
-
-    def build_tcp_replication(self):
-        return ASReplicationHandler(self)
-
-
-class ASReplicationHandler(ReplicationClientHandler):
-    def __init__(self, hs):
-        super(ASReplicationHandler, self).__init__(hs.get_datastore())
-        self.appservice_handler = hs.get_application_service_handler()
-
-    def on_rdata(self, stream_name, token, rows):
-        super(ASReplicationHandler, self).on_rdata(stream_name, token, rows)
-
-        if stream_name == "events":
-            max_stream_id = self.store.get_room_max_stream_ordering()
-            preserve_fn(
-                self.appservice_handler.notify_interested_services
-            )(max_stream_id)
-
-
-def start(config_options):
-    try:
-        config = HomeServerConfig.load_config(
-            "Synapse appservice", config_options
-        )
-    except ConfigError as e:
-        sys.stderr.write("\n" + e.message + "\n")
-        sys.exit(1)
-
-    assert config.worker_app == "synapse.app.appservice"
-
-    setup_logging(config, use_worker_options=True)
-
-    events.USE_FROZEN_DICTS = config.use_frozen_dicts
-
-    database_engine = create_engine(config.database_config)
-
-    if config.notify_appservices:
-        sys.stderr.write(
-            "\nThe appservices must be disabled in the main synapse process"
-            "\nbefore they can be run in a separate worker."
-            "\nPlease add ``notify_appservices: false`` to the main config"
-            "\n"
-        )
-        sys.exit(1)
-
-    # Force the pushers to start since they will be disabled in the main config
-    config.notify_appservices = True
-
-    ps = AppserviceServer(
-        config.server_name,
-        db_config=config.database_config,
-        config=config,
-        version_string="Synapse/" + get_version_string(synapse),
-        database_engine=database_engine,
-    )
-
-    ps.setup()
-    ps.start_listening(config.worker_listeners)
-
-    def start():
-        ps.get_datastore().start_profiling()
-        ps.get_state_handler().start_caching()
-
-    reactor.callWhenRunning(start)
-
-    _base.start_worker_reactor("synapse-appservice", config)
-
-
-if __name__ == '__main__':
-    with LoggingContext("main"):
-        start(sys.argv[1:])

synapse/app/client_reader.py

@@ -1,172 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-# Copyright 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-import logging
-import sys
-
-import synapse
-from synapse import events
-from synapse.app import _base
-from synapse.config._base import ConfigError
-from synapse.config.homeserver import HomeServerConfig
-from synapse.config.logger import setup_logging
-from synapse.crypto import context_factory
-from synapse.http.server import JsonResource
-from synapse.http.site import SynapseSite
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
-from synapse.replication.slave.storage._base import BaseSlavedStore
-from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
-from synapse.replication.slave.storage.client_ips import SlavedClientIpStore
-from synapse.replication.slave.storage.directory import DirectoryStore
-from synapse.replication.slave.storage.events import SlavedEventStore
-from synapse.replication.slave.storage.keys import SlavedKeyStore
-from synapse.replication.slave.storage.registration import SlavedRegistrationStore
-from synapse.replication.slave.storage.room import RoomStore
-from synapse.replication.slave.storage.transactions import TransactionStore
-from synapse.replication.tcp.client import ReplicationClientHandler
-from synapse.rest.client.v1.room import PublicRoomListRestServlet
-from synapse.server import HomeServer
-from synapse.storage.engines import create_engine
-from synapse.util.httpresourcetree import create_resource_tree
-from synapse.util.logcontext import LoggingContext
-from synapse.util.manhole import manhole
-from synapse.util.versionstring import get_version_string
-from twisted.internet import reactor
-from twisted.web.resource import NoResource
-
-logger = logging.getLogger("synapse.app.client_reader")
-
-
-class ClientReaderSlavedStore(
-    SlavedEventStore,
-    SlavedKeyStore,
-    RoomStore,
-    DirectoryStore,
-    SlavedApplicationServiceStore,
-    SlavedRegistrationStore,
-    TransactionStore,
-    SlavedClientIpStore,
-    BaseSlavedStore,
-):
-    pass
-
-
-class ClientReaderServer(HomeServer):
-    def setup(self):
-        logger.info("Setting up.")
-        self.datastore = ClientReaderSlavedStore(self.get_db_conn(), self)
-        logger.info("Finished setting up.")
-
-    def _listen_http(self, listener_config):
-        port = listener_config["port"]
-        bind_addresses = listener_config["bind_addresses"]
-        site_tag = listener_config.get("tag", port)
-        resources = {}
-        for res in listener_config["resources"]:
-            for name in res["names"]:
-                if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(self)
-                elif name == "client":
-                    resource = JsonResource(self, canonical_json=False)
-                    PublicRoomListRestServlet(self).register(resource)
-                    resources.update({
-                        "/_matrix/client/r0": resource,
-                        "/_matrix/client/unstable": resource,
-                        "/_matrix/client/v2_alpha": resource,
-                        "/_matrix/client/api/v1": resource,
-                    })
-
-        root_resource = create_resource_tree(resources, NoResource())
-
-        _base.listen_tcp(
-            bind_addresses,
-            port,
-            SynapseSite(
-                "synapse.access.http.%s" % (site_tag,),
-                site_tag,
-                listener_config,
-                root_resource,
-            )
-        )
-
-        logger.info("Synapse client reader now listening on port %d", port)
-
-    def start_listening(self, listeners):
-        for listener in listeners:
-            if listener["type"] == "http":
-                self._listen_http(listener)
-            elif listener["type"] == "manhole":
-                _base.listen_tcp(
-                    listener["bind_addresses"],
-                    listener["port"],
-                    manhole(
-                        username="matrix",
-                        password="rabbithole",
-                        globals={"hs": self},
-                    )
-                )
-
-            else:
-                logger.warn("Unrecognized listener type: %s", listener["type"])
-
-        self.get_tcp_replication().start_replication(self)
-
-    def build_tcp_replication(self):
-        return ReplicationClientHandler(self.get_datastore())
-
-
-def start(config_options):
-    try:
-        config = HomeServerConfig.load_config(
-            "Synapse client reader", config_options
-        )
-    except ConfigError as e:
-        sys.stderr.write("\n" + e.message + "\n")
-        sys.exit(1)
-
-    assert config.worker_app == "synapse.app.client_reader"
-
-    setup_logging(config, use_worker_options=True)
-
-    events.USE_FROZEN_DICTS = config.use_frozen_dicts
-
-    database_engine = create_engine(config.database_config)
-
-    tls_server_context_factory = context_factory.ServerContextFactory(config)
-
-    ss = ClientReaderServer(
-        config.server_name,
-        db_config=config.database_config,
-        tls_server_context_factory=tls_server_context_factory,
-        config=config,
-        version_string="Synapse/" + get_version_string(synapse),
-        database_engine=database_engine,
-    )
-
-    ss.setup()
-    ss.start_listening(config.worker_listeners)
-
-    def start():
-        ss.get_state_handler().start_caching()
-        ss.get_datastore().start_profiling()
-
-    reactor.callWhenRunning(start)
-
-    _base.start_worker_reactor("synapse-client-reader", config)
-
-
-if __name__ == '__main__':
-    with LoggingContext("main"):
-        start(sys.argv[1:])

synapse/app/event_creator.py

@@ -1,189 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-# Copyright 2018 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-import logging
-import sys
-
-import synapse
-from synapse import events
-from synapse.app import _base
-from synapse.config._base import ConfigError
-from synapse.config.homeserver import HomeServerConfig
-from synapse.config.logger import setup_logging
-from synapse.crypto import context_factory
-from synapse.http.server import JsonResource
-from synapse.http.site import SynapseSite
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
-from synapse.replication.slave.storage._base import BaseSlavedStore
-from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
-from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
-from synapse.replication.slave.storage.client_ips import SlavedClientIpStore
-from synapse.replication.slave.storage.devices import SlavedDeviceStore
-from synapse.replication.slave.storage.directory import DirectoryStore
-from synapse.replication.slave.storage.events import SlavedEventStore
-from synapse.replication.slave.storage.profile import SlavedProfileStore
-from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
-from synapse.replication.slave.storage.pushers import SlavedPusherStore
-from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
-from synapse.replication.slave.storage.registration import SlavedRegistrationStore
-from synapse.replication.slave.storage.room import RoomStore
-from synapse.replication.slave.storage.transactions import TransactionStore
-from synapse.replication.tcp.client import ReplicationClientHandler
-from synapse.rest.client.v1.room import (
-    RoomSendEventRestServlet, RoomMembershipRestServlet, RoomStateEventRestServlet,
-    JoinRoomAliasServlet,
-)
-from synapse.server import HomeServer
-from synapse.storage.engines import create_engine
-from synapse.util.httpresourcetree import create_resource_tree
-from synapse.util.logcontext import LoggingContext
-from synapse.util.manhole import manhole
-from synapse.util.versionstring import get_version_string
-from twisted.internet import reactor
-from twisted.web.resource import NoResource
-
-logger = logging.getLogger("synapse.app.event_creator")
-
-
-class EventCreatorSlavedStore(
-    DirectoryStore,
-    TransactionStore,
-    SlavedProfileStore,
-    SlavedAccountDataStore,
-    SlavedPusherStore,
-    SlavedReceiptsStore,
-    SlavedPushRuleStore,
-    SlavedDeviceStore,
-    SlavedClientIpStore,
-    SlavedApplicationServiceStore,
-    SlavedEventStore,
-    SlavedRegistrationStore,
-    RoomStore,
-    BaseSlavedStore,
-):
-    pass
-
-
-class EventCreatorServer(HomeServer):
-    def setup(self):
-        logger.info("Setting up.")
-        self.datastore = EventCreatorSlavedStore(self.get_db_conn(), self)
-        logger.info("Finished setting up.")
-
-    def _listen_http(self, listener_config):
-        port = listener_config["port"]
-        bind_addresses = listener_config["bind_addresses"]
-        site_tag = listener_config.get("tag", port)
-        resources = {}
-        for res in listener_config["resources"]:
-            for name in res["names"]:
-                if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(self)
-                elif name == "client":
-                    resource = JsonResource(self, canonical_json=False)
-                    RoomSendEventRestServlet(self).register(resource)
-                    RoomMembershipRestServlet(self).register(resource)
-                    RoomStateEventRestServlet(self).register(resource)
-                    JoinRoomAliasServlet(self).register(resource)
-                    resources.update({
-                        "/_matrix/client/r0": resource,
-                        "/_matrix/client/unstable": resource,
-                        "/_matrix/client/v2_alpha": resource,
-                        "/_matrix/client/api/v1": resource,
-                    })
-
-        root_resource = create_resource_tree(resources, NoResource())
-
-        _base.listen_tcp(
-            bind_addresses,
-            port,
-            SynapseSite(
-                "synapse.access.http.%s" % (site_tag,),
-                site_tag,
-                listener_config,
-                root_resource,
-            )
-        )
-
-        logger.info("Synapse event creator now listening on port %d", port)
-
-    def start_listening(self, listeners):
-        for listener in listeners:
-            if listener["type"] == "http":
-                self._listen_http(listener)
-            elif listener["type"] == "manhole":
-                _base.listen_tcp(
-                    listener["bind_addresses"],
-                    listener["port"],
-                    manhole(
-                        username="matrix",
-                        password="rabbithole",
-                        globals={"hs": self},
-                    )
-                )
-            else:
-                logger.warn("Unrecognized listener type: %s", listener["type"])
-
-        self.get_tcp_replication().start_replication(self)
-
-    def build_tcp_replication(self):
-        return ReplicationClientHandler(self.get_datastore())
-
-
-def start(config_options):
-    try:
-        config = HomeServerConfig.load_config(
-            "Synapse event creator", config_options
-        )
-    except ConfigError as e:
-        sys.stderr.write("\n" + e.message + "\n")
-        sys.exit(1)
-
-    assert config.worker_app == "synapse.app.event_creator"
-
-    assert config.worker_replication_http_port is not None
-
-    setup_logging(config, use_worker_options=True)
-
-    events.USE_FROZEN_DICTS = config.use_frozen_dicts
-
-    database_engine = create_engine(config.database_config)
-
-    tls_server_context_factory = context_factory.ServerContextFactory(config)
-
-    ss = EventCreatorServer(
-        config.server_name,
-        db_config=config.database_config,
-        tls_server_context_factory=tls_server_context_factory,
-        config=config,
-        version_string="Synapse/" + get_version_string(synapse),
-        database_engine=database_engine,
-    )
-
-    ss.setup()
-    ss.start_listening(config.worker_listeners)
-
-    def start():
-        ss.get_state_handler().start_caching()
-        ss.get_datastore().start_profiling()
-
-    reactor.callWhenRunning(start)
-
-    _base.start_worker_reactor("synapse-event-creator", config)
-
-
-if __name__ == '__main__':
-    with LoggingContext("main"):
-        start(sys.argv[1:])

synapse/app/federation_reader.py

@@ -1,160 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-# Copyright 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-import logging
-import sys
-
-import synapse
-from synapse import events
-from synapse.api.urls import FEDERATION_PREFIX
-from synapse.app import _base
-from synapse.config._base import ConfigError
-from synapse.config.homeserver import HomeServerConfig
-from synapse.config.logger import setup_logging
-from synapse.crypto import context_factory
-from synapse.federation.transport.server import TransportLayerServer
-from synapse.http.site import SynapseSite
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
-from synapse.replication.slave.storage._base import BaseSlavedStore
-from synapse.replication.slave.storage.directory import DirectoryStore
-from synapse.replication.slave.storage.events import SlavedEventStore
-from synapse.replication.slave.storage.keys import SlavedKeyStore
-from synapse.replication.slave.storage.room import RoomStore
-from synapse.replication.slave.storage.transactions import TransactionStore
-from synapse.replication.tcp.client import ReplicationClientHandler
-from synapse.server import HomeServer
-from synapse.storage.engines import create_engine
-from synapse.util.httpresourcetree import create_resource_tree
-from synapse.util.logcontext import LoggingContext
-from synapse.util.manhole import manhole
-from synapse.util.versionstring import get_version_string
-from twisted.internet import reactor
-from twisted.web.resource import NoResource
-
-logger = logging.getLogger("synapse.app.federation_reader")
-
-
-class FederationReaderSlavedStore(
-    SlavedEventStore,
-    SlavedKeyStore,
-    RoomStore,
-    DirectoryStore,
-    TransactionStore,
-    BaseSlavedStore,
-):
-    pass
-
-
-class FederationReaderServer(HomeServer):
-    def setup(self):
-        logger.info("Setting up.")
-        self.datastore = FederationReaderSlavedStore(self.get_db_conn(), self)
-        logger.info("Finished setting up.")
-
-    def _listen_http(self, listener_config):
-        port = listener_config["port"]
-        bind_addresses = listener_config["bind_addresses"]
-        site_tag = listener_config.get("tag", port)
-        resources = {}
-        for res in listener_config["resources"]:
-            for name in res["names"]:
-                if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(self)
-                elif name == "federation":
-                    resources.update({
-                        FEDERATION_PREFIX: TransportLayerServer(self),
-                    })
-
-        root_resource = create_resource_tree(resources, NoResource())
-
-        _base.listen_tcp(
-            bind_addresses,
-            port,
-            SynapseSite(
-                "synapse.access.http.%s" % (site_tag,),
-                site_tag,
-                listener_config,
-                root_resource,
-            )
-        )
-
-        logger.info("Synapse federation reader now listening on port %d", port)
-
-    def start_listening(self, listeners):
-        for listener in listeners:
-            if listener["type"] == "http":
-                self._listen_http(listener)
-            elif listener["type"] == "manhole":
-                _base.listen_tcp(
-                    listener["bind_addresses"],
-                    listener["port"],
-                    manhole(
-                        username="matrix",
-                        password="rabbithole",
-                        globals={"hs": self},
-                    )
-                )
-            else:
-                logger.warn("Unrecognized listener type: %s", listener["type"])
-
-        self.get_tcp_replication().start_replication(self)
-
-    def build_tcp_replication(self):
-        return ReplicationClientHandler(self.get_datastore())
-
-
-def start(config_options):
-    try:
-        config = HomeServerConfig.load_config(
-            "Synapse federation reader", config_options
-        )
-    except ConfigError as e:
-        sys.stderr.write("\n" + e.message + "\n")
-        sys.exit(1)
-
-    assert config.worker_app == "synapse.app.federation_reader"
-
-    setup_logging(config, use_worker_options=True)
-
-    events.USE_FROZEN_DICTS = config.use_frozen_dicts
-
-    database_engine = create_engine(config.database_config)
-
-    tls_server_context_factory = context_factory.ServerContextFactory(config)
-
-    ss = FederationReaderServer(
-        config.server_name,
-        db_config=config.database_config,
-        tls_server_context_factory=tls_server_context_factory,
-        config=config,
-        version_string="Synapse/" + get_version_string(synapse),
-        database_engine=database_engine,
-    )
-
-    ss.setup()
-    ss.start_listening(config.worker_listeners)
-
-    def start():
-        ss.get_state_handler().start_caching()
-        ss.get_datastore().start_profiling()
-
-    reactor.callWhenRunning(start)
-
-    _base.start_worker_reactor("synapse-federation-reader", config)
-
-
-if __name__ == '__main__':
-    with LoggingContext("main"):
-        start(sys.argv[1:])

synapse/app/federation_sender.py

@@ -1,257 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-# Copyright 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-import logging
-import sys
-
-import synapse
-from synapse import events
-from synapse.app import _base
-from synapse.config._base import ConfigError
-from synapse.config.homeserver import HomeServerConfig
-from synapse.config.logger import setup_logging
-from synapse.crypto import context_factory
-from synapse.federation import send_queue
-from synapse.http.site import SynapseSite
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
-from synapse.replication.slave.storage.deviceinbox import SlavedDeviceInboxStore
-from synapse.replication.slave.storage.devices import SlavedDeviceStore
-from synapse.replication.slave.storage.events import SlavedEventStore
-from synapse.replication.slave.storage.presence import SlavedPresenceStore
-from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
-from synapse.replication.slave.storage.registration import SlavedRegistrationStore
-from synapse.replication.slave.storage.transactions import TransactionStore
-from synapse.replication.tcp.client import ReplicationClientHandler
-from synapse.server import HomeServer
-from synapse.storage.engines import create_engine
-from synapse.util.async import Linearizer
-from synapse.util.httpresourcetree import create_resource_tree
-from synapse.util.logcontext import LoggingContext, preserve_fn
-from synapse.util.manhole import manhole
-from synapse.util.versionstring import get_version_string
-from twisted.internet import defer, reactor
-from twisted.web.resource import NoResource
-
-logger = logging.getLogger("synapse.app.federation_sender")
-
-
-class FederationSenderSlaveStore(
-    SlavedDeviceInboxStore, TransactionStore, SlavedReceiptsStore, SlavedEventStore,
-    SlavedRegistrationStore, SlavedDeviceStore, SlavedPresenceStore,
-):
-    def __init__(self, db_conn, hs):
-        super(FederationSenderSlaveStore, self).__init__(db_conn, hs)
-
-        # We pull out the current federation stream position now so that we
-        # always have a known value for the federation position in memory so
-        # that we don't have to bounce via a deferred once when we start the
-        # replication streams.
-        self.federation_out_pos_startup = self._get_federation_out_pos(db_conn)
-
-    def _get_federation_out_pos(self, db_conn):
-        sql = (
-            "SELECT stream_id FROM federation_stream_position"
-            " WHERE type = ?"
-        )
-        sql = self.database_engine.convert_param_style(sql)
-
-        txn = db_conn.cursor()
-        txn.execute(sql, ("federation",))
-        rows = txn.fetchall()
-        txn.close()
-
-        return rows[0][0] if rows else -1
-
-
-class FederationSenderServer(HomeServer):
-    def setup(self):
-        logger.info("Setting up.")
-        self.datastore = FederationSenderSlaveStore(self.get_db_conn(), self)
-        logger.info("Finished setting up.")
-
-    def _listen_http(self, listener_config):
-        port = listener_config["port"]
-        bind_addresses = listener_config["bind_addresses"]
-        site_tag = listener_config.get("tag", port)
-        resources = {}
-        for res in listener_config["resources"]:
-            for name in res["names"]:
-                if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(self)
-
-        root_resource = create_resource_tree(resources, NoResource())
-
-        _base.listen_tcp(
-            bind_addresses,
-            port,
-            SynapseSite(
-                "synapse.access.http.%s" % (site_tag,),
-                site_tag,
-                listener_config,
-                root_resource,
-            )
-        )
-
-        logger.info("Synapse federation_sender now listening on port %d", port)
-
-    def start_listening(self, listeners):
-        for listener in listeners:
-            if listener["type"] == "http":
-                self._listen_http(listener)
-            elif listener["type"] == "manhole":
-                _base.listen_tcp(
-                    listener["bind_addresses"],
-                    listener["port"],
-                    manhole(
-                        username="matrix",
-                        password="rabbithole",
-                        globals={"hs": self},
-                    )
-                )
-            else:
-                logger.warn("Unrecognized listener type: %s", listener["type"])
-
-        self.get_tcp_replication().start_replication(self)
-
-    def build_tcp_replication(self):
-        return FederationSenderReplicationHandler(self)
-
-
-class FederationSenderReplicationHandler(ReplicationClientHandler):
-    def __init__(self, hs):
-        super(FederationSenderReplicationHandler, self).__init__(hs.get_datastore())
-        self.send_handler = FederationSenderHandler(hs, self)
-
-    def on_rdata(self, stream_name, token, rows):
-        super(FederationSenderReplicationHandler, self).on_rdata(
-            stream_name, token, rows
-        )
-        self.send_handler.process_replication_rows(stream_name, token, rows)
-
-    def get_streams_to_replicate(self):
-        args = super(FederationSenderReplicationHandler, self).get_streams_to_replicate()
-        args.update(self.send_handler.stream_positions())
-        return args
-
-
-def start(config_options):
-    try:
-        config = HomeServerConfig.load_config(
-            "Synapse federation sender", config_options
-        )
-    except ConfigError as e:
-        sys.stderr.write("\n" + e.message + "\n")
-        sys.exit(1)
-
-    assert config.worker_app == "synapse.app.federation_sender"
-
-    setup_logging(config, use_worker_options=True)
-
-    events.USE_FROZEN_DICTS = config.use_frozen_dicts
-
-    database_engine = create_engine(config.database_config)
-
-    if config.send_federation:
-        sys.stderr.write(
-            "\nThe send_federation must be disabled in the main synapse process"
-            "\nbefore they can be run in a separate worker."
-            "\nPlease add ``send_federation: false`` to the main config"
-            "\n"
-        )
-        sys.exit(1)
-
-    # Force the pushers to start since they will be disabled in the main config
-    config.send_federation = True
-
-    tls_server_context_factory = context_factory.ServerContextFactory(config)
-
-    ps = FederationSenderServer(
-        config.server_name,
-        db_config=config.database_config,
-        tls_server_context_factory=tls_server_context_factory,
-        config=config,
-        version_string="Synapse/" + get_version_string(synapse),
-        database_engine=database_engine,
-    )
-
-    ps.setup()
-    ps.start_listening(config.worker_listeners)
-
-    def start():
-        ps.get_datastore().start_profiling()
-        ps.get_state_handler().start_caching()
-
-    reactor.callWhenRunning(start)
-    _base.start_worker_reactor("synapse-federation-sender", config)
-
-
-class FederationSenderHandler(object):
-    """Processes the replication stream and forwards the appropriate entries
-    to the federation sender.
-    """
-    def __init__(self, hs, replication_client):
-        self.store = hs.get_datastore()
-        self.federation_sender = hs.get_federation_sender()
-        self.replication_client = replication_client
-
-        self.federation_position = self.store.federation_out_pos_startup
-        self._fed_position_linearizer = Linearizer(name="_fed_position_linearizer")
-
-        self._last_ack = self.federation_position
-
-        self._room_serials = {}
-        self._room_typing = {}
-
-    def on_start(self):
-        # There may be some events that are persisted but haven't been sent,
-        # so send them now.
-        self.federation_sender.notify_new_events(
-            self.store.get_room_max_stream_ordering()
-        )
-
-    def stream_positions(self):
-        return {"federation": self.federation_position}
-
-    def process_replication_rows(self, stream_name, token, rows):
-        # The federation stream contains things that we want to send out, e.g.
-        # presence, typing, etc.
-        if stream_name == "federation":
-            send_queue.process_rows_for_federation(self.federation_sender, rows)
-            preserve_fn(self.update_token)(token)
-
-        # We also need to poke the federation sender when new events happen
-        elif stream_name == "events":
-            self.federation_sender.notify_new_events(token)
-
-    @defer.inlineCallbacks
-    def update_token(self, token):
-        self.federation_position = token
-
-        # We linearize here to ensure we don't have races updating the token
-        with (yield self._fed_position_linearizer.queue(None)):
-            if self._last_ack < self.federation_position:
-                yield self.store.update_federation_out_pos(
-                    "federation", self.federation_position
-                )
-
-                # We ACK this token over replication so that the master can drop
-                # its in memory queues
-                self.replication_client.send_federation_ack(self.federation_position)
-                self._last_ack = self.federation_position
-
-
-if __name__ == '__main__':
-    with LoggingContext("main"):
-        start(sys.argv[1:])

synapse/app/frontend_proxy.py

@@ -1,258 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-# Copyright 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-import logging
-import sys
-
-import synapse
-from synapse import events
-from synapse.api.errors import SynapseError
-from synapse.app import _base
-from synapse.config._base import ConfigError
-from synapse.config.homeserver import HomeServerConfig
-from synapse.config.logger import setup_logging
-from synapse.crypto import context_factory
-from synapse.http.server import JsonResource
-from synapse.http.servlet import (
-    RestServlet, parse_json_object_from_request,
-)
-from synapse.http.site import SynapseSite
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
-from synapse.replication.slave.storage._base import BaseSlavedStore
-from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
-from synapse.replication.slave.storage.client_ips import SlavedClientIpStore
-from synapse.replication.slave.storage.devices import SlavedDeviceStore
-from synapse.replication.slave.storage.registration import SlavedRegistrationStore
-from synapse.replication.tcp.client import ReplicationClientHandler
-from synapse.rest.client.v1.base import ClientV1RestServlet, client_path_patterns
-from synapse.rest.client.v2_alpha._base import client_v2_patterns
-from synapse.server import HomeServer
-from synapse.storage.engines import create_engine
-from synapse.util.httpresourcetree import create_resource_tree
-from synapse.util.logcontext import LoggingContext
-from synapse.util.manhole import manhole
-from synapse.util.versionstring import get_version_string
-from twisted.internet import defer, reactor
-from twisted.web.resource import NoResource
-
-logger = logging.getLogger("synapse.app.frontend_proxy")
-
-
-class PresenceStatusStubServlet(ClientV1RestServlet):
-    PATTERNS = client_path_patterns("/presence/(?P<user_id>[^/]*)/status")
-
-    def __init__(self, hs):
-        super(PresenceStatusStubServlet, self).__init__(hs)
-        self.http_client = hs.get_simple_http_client()
-        self.auth = hs.get_auth()
-        self.main_uri = hs.config.worker_main_http_uri
-
-    @defer.inlineCallbacks
-    def on_GET(self, request, user_id):
-        # Pass through the auth headers, if any, in case the access token
-        # is there.
-        auth_headers = request.requestHeaders.getRawHeaders("Authorization", [])
-        headers = {
-            "Authorization": auth_headers,
-        }
-        result = yield self.http_client.get_json(
-            self.main_uri + request.uri,
-            headers=headers,
-        )
-        defer.returnValue((200, result))
-
-    @defer.inlineCallbacks
-    def on_PUT(self, request, user_id):
-        yield self.auth.get_user_by_req(request)
-        defer.returnValue((200, {}))
-
-
-class KeyUploadServlet(RestServlet):
-    PATTERNS = client_v2_patterns("/keys/upload(/(?P<device_id>[^/]+))?$")
-
-    def __init__(self, hs):
-        """
-        Args:
-            hs (synapse.server.HomeServer): server
-        """
-        super(KeyUploadServlet, self).__init__()
-        self.auth = hs.get_auth()
-        self.store = hs.get_datastore()
-        self.http_client = hs.get_simple_http_client()
-        self.main_uri = hs.config.worker_main_http_uri
-
-    @defer.inlineCallbacks
-    def on_POST(self, request, device_id):
-        requester = yield self.auth.get_user_by_req(request, allow_guest=True)
-        user_id = requester.user.to_string()
-        body = parse_json_object_from_request(request)
-
-        if device_id is not None:
-            # passing the device_id here is deprecated; however, we allow it
-            # for now for compatibility with older clients.
-            if (requester.device_id is not None and
-                    device_id != requester.device_id):
-                logger.warning("Client uploading keys for a different device "
-                               "(logged in as %s, uploading for %s)",
-                               requester.device_id, device_id)
-        else:
-            device_id = requester.device_id
-
-        if device_id is None:
-            raise SynapseError(
-                400,
-                "To upload keys, you must pass device_id when authenticating"
-            )
-
-        if body:
-            # They're actually trying to upload something, proxy to main synapse.
-            # Pass through the auth headers, if any, in case the access token
-            # is there.
-            auth_headers = request.requestHeaders.getRawHeaders(b"Authorization", [])
-            headers = {
-                "Authorization": auth_headers,
-            }
-            result = yield self.http_client.post_json_get_json(
-                self.main_uri + request.uri,
-                body,
-                headers=headers,
-            )
-
-            defer.returnValue((200, result))
-        else:
-            # Just interested in counts.
-            result = yield self.store.count_e2e_one_time_keys(user_id, device_id)
-            defer.returnValue((200, {"one_time_key_counts": result}))
-
-
-class FrontendProxySlavedStore(
-    SlavedDeviceStore,
-    SlavedClientIpStore,
-    SlavedApplicationServiceStore,
-    SlavedRegistrationStore,
-    BaseSlavedStore,
-):
-    pass
-
-
-class FrontendProxyServer(HomeServer):
-    def setup(self):
-        logger.info("Setting up.")
-        self.datastore = FrontendProxySlavedStore(self.get_db_conn(), self)
-        logger.info("Finished setting up.")
-
-    def _listen_http(self, listener_config):
-        port = listener_config["port"]
-        bind_addresses = listener_config["bind_addresses"]
-        site_tag = listener_config.get("tag", port)
-        resources = {}
-        for res in listener_config["resources"]:
-            for name in res["names"]:
-                if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(self)
-                elif name == "client":
-                    resource = JsonResource(self, canonical_json=False)
-                    KeyUploadServlet(self).register(resource)
-                    PresenceStatusStubServlet(self).register(resource)
-                    resources.update({
-                        "/_matrix/client/r0": resource,
-                        "/_matrix/client/unstable": resource,
-                        "/_matrix/client/v2_alpha": resource,
-                        "/_matrix/client/api/v1": resource,
-                    })
-
-        root_resource = create_resource_tree(resources, NoResource())
-
-        _base.listen_tcp(
-            bind_addresses,
-            port,
-            SynapseSite(
-                "synapse.access.http.%s" % (site_tag,),
-                site_tag,
-                listener_config,
-                root_resource,
-            )
-        )
-
-        logger.info("Synapse client reader now listening on port %d", port)
-
-    def start_listening(self, listeners):
-        for listener in listeners:
-            if listener["type"] == "http":
-                self._listen_http(listener)
-            elif listener["type"] == "manhole":
-                _base.listen_tcp(
-                    listener["bind_addresses"],
-                    listener["port"],
-                    manhole(
-                        username="matrix",
-                        password="rabbithole",
-                        globals={"hs": self},
-                    )
-                )
-            else:
-                logger.warn("Unrecognized listener type: %s", listener["type"])
-
-        self.get_tcp_replication().start_replication(self)
-
-    def build_tcp_replication(self):
-        return ReplicationClientHandler(self.get_datastore())
-
-
-def start(config_options):
-    try:
-        config = HomeServerConfig.load_config(
-            "Synapse frontend proxy", config_options
-        )
-    except ConfigError as e:
-        sys.stderr.write("\n" + e.message + "\n")
-        sys.exit(1)
-
-    assert config.worker_app == "synapse.app.frontend_proxy"
-
-    assert config.worker_main_http_uri is not None
-
-    setup_logging(config, use_worker_options=True)
-
-    events.USE_FROZEN_DICTS = config.use_frozen_dicts
-
-    database_engine = create_engine(config.database_config)
-
-    tls_server_context_factory = context_factory.ServerContextFactory(config)
-
-    ss = FrontendProxyServer(
-        config.server_name,
-        db_config=config.database_config,
-        tls_server_context_factory=tls_server_context_factory,
-        config=config,
-        version_string="Synapse/" + get_version_string(synapse),
-        database_engine=database_engine,
-    )
-
-    ss.setup()
-    ss.start_listening(config.worker_listeners)
-
-    def start():
-        ss.get_state_handler().start_caching()
-        ss.get_datastore().start_profiling()
-
-    reactor.callWhenRunning(start)
-
-    _base.start_worker_reactor("synapse-frontend-proxy", config)
-
-
-if __name__ == '__main__':
-    with LoggingContext("main"):
-        start(sys.argv[1:])

synapse/app/homeserver.py

@@ -13,53 +13,58 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
+import synapse
+
 import gc
 import logging
 import os
 import sys
-
-import synapse
-import synapse.config.logger
-from synapse import events
-from synapse.api.urls import CONTENT_REPO_PREFIX, FEDERATION_PREFIX, \
-    LEGACY_MEDIA_PREFIX, MEDIA_PREFIX, SERVER_KEY_PREFIX, SERVER_KEY_V2_PREFIX, \
-    STATIC_PREFIX, WEB_CLIENT_PREFIX
-from synapse.app import _base
-from synapse.app._base import quit_with_error, listen_ssl, listen_tcp
 from synapse.config._base import ConfigError
-from synapse.config.homeserver import HomeServerConfig
-from synapse.crypto import context_factory
-from synapse.federation.transport.server import TransportLayerServer
-from synapse.module_api import ModuleApi
-from synapse.http.additional_resource import AdditionalResource
-from synapse.http.server import RootRedirect
-from synapse.http.site import SynapseSite
-from synapse.metrics import register_memory_metrics
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
-from synapse.python_dependencies import CONDITIONAL_REQUIREMENTS, \
-    check_requirements
-from synapse.replication.http import ReplicationRestResource, REPLICATION_PREFIX
-from synapse.replication.tcp.resource import ReplicationStreamProtocolFactory
+
+from synapse.python_dependencies import (
+    check_requirements, DEPENDENCY_LINKS
+)
+
 from synapse.rest import ClientRestResource
+from synapse.storage.engines import create_engine, IncorrectDatabaseSetup
+from synapse.storage import are_all_users_on_domain
+from synapse.storage.prepare_database import UpgradeDatabaseException, prepare_database
+
+from synapse.server import HomeServer
+
+from twisted.internet import reactor, task, defer
+from twisted.application import service
+from twisted.web.resource import Resource, EncodingResourceWrapper
+from twisted.web.static import File
+from twisted.web.server import GzipEncoderFactory
+from synapse.http.server import RootRedirect
+from synapse.rest.media.v0.content_repository import ContentRepoResource
+from synapse.rest.media.v1.media_repository import MediaRepositoryResource
 from synapse.rest.key.v1.server_key_resource import LocalKey
 from synapse.rest.key.v2 import KeyApiV2Resource
-from synapse.rest.media.v0.content_repository import ContentRepoResource
-from synapse.server import HomeServer
-from synapse.storage import are_all_users_on_domain
-from synapse.storage.engines import IncorrectDatabaseSetup, create_engine
-from synapse.storage.prepare_database import UpgradeDatabaseException, prepare_database
-from synapse.util.caches import CACHE_SIZE_FACTOR
-from synapse.util.httpresourcetree import create_resource_tree
+from synapse.api.urls import (
+    FEDERATION_PREFIX, WEB_CLIENT_PREFIX, CONTENT_REPO_PREFIX,
+    SERVER_KEY_PREFIX, LEGACY_MEDIA_PREFIX, MEDIA_PREFIX, STATIC_PREFIX,
+    SERVER_KEY_V2_PREFIX,
+)
+from synapse.config.homeserver import HomeServerConfig
+from synapse.crypto import context_factory
 from synapse.util.logcontext import LoggingContext
-from synapse.util.manhole import manhole
-from synapse.util.module_loader import load_module
+from synapse.metrics.resource import MetricsResource, METRICS_PREFIX
+from synapse.replication.resource import ReplicationResource, REPLICATION_PREFIX
+from synapse.federation.transport.server import TransportLayerServer
+
 from synapse.util.rlimit import change_resource_limit
 from synapse.util.versionstring import get_version_string
-from twisted.application import service
-from twisted.internet import defer, reactor
-from twisted.web.resource import EncodingResourceWrapper, NoResource
-from twisted.web.server import GzipEncoderFactory
-from twisted.web.static import File
+from synapse.util.httpresourcetree import create_resource_tree
+from synapse.util.manhole import manhole
+
+from synapse.http.site import SynapseSite
+
+from synapse import events
+
+from daemonize import Daemonize
 
 logger = logging.getLogger("synapse.app.homeserver")
 
@@ -84,7 +89,7 @@ def build_resource_for_web_client(hs):
                 "\n"
                 "You can also disable hosting of the webclient via the\n"
                 "configuration option `web_client`\n"
-                % {"dep": CONDITIONAL_REQUIREMENTS["web_client"].keys()[0]}
+                % {"dep": DEPENDENCY_LINKS["matrix-angular-sdk"]}
             )
         syweb_path = os.path.dirname(syweb.__file__)
         webclient_path = os.path.join(syweb_path, "webclient")
@@ -101,7 +106,7 @@ def build_resource_for_web_client(hs):
 class SynapseHomeServer(HomeServer):
     def _listener_http(self, config, listener_config):
         port = listener_config["port"]
-        bind_addresses = listener_config["bind_addresses"]
+        bind_address = listener_config.get("bind_address", "")
         tls = listener_config.get("tls", False)
         site_tag = listener_config.get("tag", port)
 
@@ -111,67 +116,9 @@ class SynapseHomeServer(HomeServer):
         resources = {}
         for res in listener_config["resources"]:
             for name in res["names"]:
-                resources.update(self._configure_named_resource(
-                    name, res.get("compress", False),
-                ))
-
-        additional_resources = listener_config.get("additional_resources", {})
-        logger.debug("Configuring additional resources: %r",
-                     additional_resources)
-        module_api = ModuleApi(self, self.get_auth_handler())
-        for path, resmodule in additional_resources.items():
-            handler_cls, config = load_module(resmodule)
-            handler = handler_cls(config, module_api)
-            resources[path] = AdditionalResource(self, handler.handle_request)
-
-        if WEB_CLIENT_PREFIX in resources:
-            root_resource = RootRedirect(WEB_CLIENT_PREFIX)
-        else:
-            root_resource = NoResource()
-
-        root_resource = create_resource_tree(resources, root_resource)
-
-        if tls:
-            listen_ssl(
-                bind_addresses,
-                port,
-                SynapseSite(
-                    "synapse.access.https.%s" % (site_tag,),
-                    site_tag,
-                    listener_config,
-                    root_resource,
-                ),
-                self.tls_server_context_factory,
-            )
-
-        else:
-            listen_tcp(
-                bind_addresses,
-                port,
-                SynapseSite(
-                    "synapse.access.http.%s" % (site_tag,),
-                    site_tag,
-                    listener_config,
-                    root_resource,
-                )
-            )
-        logger.info("Synapse now listening on port %d", port)
-
-    def _configure_named_resource(self, name, compress=False):
-        """Build a resource map for a named resource
-
-        Args:
-            name (str): named resource: one of "client", "federation", etc
-            compress (bool): whether to enable gzip compression for this
-                resource
-
-        Returns:
-            dict[str, Resource]: map from path to HTTP resource
-        """
-        resources = {}
                 if name == "client":
                     client_resource = ClientRestResource(self)
-            if compress:
+                    if res["compress"]:
                         client_resource = gz_wrap(client_resource)
 
                     resources.update({
@@ -195,8 +142,7 @@ class SynapseHomeServer(HomeServer):
                     })
 
                 if name in ["media", "federation", "client"]:
-            if self.get_config().enable_media_repo:
-                media_repo = self.get_media_repository_resource()
+                    media_repo = MediaRepositoryResource(self)
                     resources.update({
                         MEDIA_PREFIX: media_repo,
                         LEGACY_MEDIA_PREFIX: media_repo,
@@ -204,10 +150,6 @@ class SynapseHomeServer(HomeServer):
                             self, self.config.uploads_path
                         ),
                     })
-            elif name == "media":
-                raise ConfigError(
-                    "'media' resource conflicts with enable_media_repo=False",
-                )
 
                 if name in ["keys", "federation"]:
                     resources.update({
@@ -222,9 +164,38 @@ class SynapseHomeServer(HomeServer):
                     resources[METRICS_PREFIX] = MetricsResource(self)
 
                 if name == "replication":
-            resources[REPLICATION_PREFIX] = ReplicationRestResource(self)
+                    resources[REPLICATION_PREFIX] = ReplicationResource(self)
 
-        return resources
+        if WEB_CLIENT_PREFIX in resources:
+            root_resource = RootRedirect(WEB_CLIENT_PREFIX)
+        else:
+            root_resource = Resource()
+
+        root_resource = create_resource_tree(resources, root_resource)
+        if tls:
+            reactor.listenSSL(
+                port,
+                SynapseSite(
+                    "synapse.access.https.%s" % (site_tag,),
+                    site_tag,
+                    listener_config,
+                    root_resource,
+                ),
+                self.tls_server_context_factory,
+                interface=bind_address
+            )
+        else:
+            reactor.listenTCP(
+                port,
+                SynapseSite(
+                    "synapse.access.http.%s" % (site_tag,),
+                    site_tag,
+                    listener_config,
+                    root_resource,
+                ),
+                interface=bind_address
+            )
+        logger.info("Synapse now listening on port %d", port)
 
     def start_listening(self):
         config = self.get_config()
@@ -233,24 +204,14 @@ class SynapseHomeServer(HomeServer):
             if listener["type"] == "http":
                 self._listener_http(config, listener)
             elif listener["type"] == "manhole":
-                listen_tcp(
-                    listener["bind_addresses"],
+                reactor.listenTCP(
                     listener["port"],
                     manhole(
                         username="matrix",
                         password="rabbithole",
                         globals={"hs": self},
-                    )
-                )
-            elif listener["type"] == "replication":
-                bind_addresses = listener["bind_addresses"]
-                for address in bind_addresses:
-                    factory = ReplicationStreamProtocolFactory(self)
-                    server_listener = reactor.listenTCP(
-                        listener["port"], factory, interface=address
-                    )
-                    reactor.addSystemEventTrigger(
-                        "before", "shutdown", server_listener.stopListening,
+                    ),
+                    interface=listener.get("bind_address", '127.0.0.1')
                 )
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
@@ -271,6 +232,29 @@ class SynapseHomeServer(HomeServer):
         except IncorrectDatabaseSetup as e:
             quit_with_error(e.message)
 
+    def get_db_conn(self, run_new_connection=True):
+        # Any param beginning with cp_ is a parameter for adbapi, and should
+        # not be passed to the database engine.
+        db_params = {
+            k: v for k, v in self.db_config.get("args", {}).items()
+            if not k.startswith("cp_")
+        }
+        db_conn = self.database_engine.module.connect(**db_params)
+
+        if run_new_connection:
+            self.database_engine.on_new_connection(db_conn)
+        return db_conn
+
+
+def quit_with_error(error_string):
+    message_lines = error_string.split("\n")
+    line_length = max([len(l) for l in message_lines if len(l) < 80]) + 2
+    sys.stderr.write("*" * line_length + '\n')
+    for line in message_lines:
+        sys.stderr.write(" %s\n" % (line.rstrip(),))
+    sys.stderr.write("*" * line_length + '\n')
+    sys.exit(1)
+
 
 def setup(config_options):
     """
@@ -295,12 +279,12 @@ def setup(config_options):
         # generating config files and shouldn't try to continue.
         sys.exit(0)
 
-    synapse.config.logger.setup_logging(config, use_worker_options=False)
+    config.setup_logging()
 
     # check any extra requirements we have now we have a config
     check_requirements(config)
 
-    version_string = "Synapse/" + get_version_string(synapse)
+    version_string = get_version_string("Synapse", synapse)
 
     logger.info("Server hostname: %s", config.server_name)
     logger.info("Server version: %s", version_string)
@@ -349,9 +333,7 @@ def setup(config_options):
         hs.get_state_handler().start_caching()
         hs.get_datastore().start_profiling()
         hs.get_datastore().start_doing_background_updates()
-        hs.get_federation_client().start_get_pdu_cache()
-
-        register_memory_metrics(hs)
+        hs.get_replication_layer().start_get_pdu_cache()
 
     reactor.callWhenRunning(start)
 
@@ -398,14 +380,7 @@ def run(hs):
         ThreadPool._worker = profile(ThreadPool._worker)
         reactor.run = profile(reactor.run)
 
-    clock = hs.get_clock()
-    start_time = clock.time()
-
-    stats = {}
-
-    # Contains the list of processes we will be monitoring
-    # currently either 0 or 1
-    stats_process = []
+    start_time = hs.get_clock().time()
 
     @defer.inlineCallbacks
     def phone_stats_home():
@@ -415,36 +390,19 @@ def run(hs):
         if uptime < 0:
             uptime = 0
 
+        stats = {}
         stats["homeserver"] = hs.config.server_name
         stats["timestamp"] = now
         stats["uptime_seconds"] = uptime
         stats["total_users"] = yield hs.get_datastore().count_all_users()
 
-        total_nonbridged_users = yield hs.get_datastore().count_nonbridged_users()
-        stats["total_nonbridged_users"] = total_nonbridged_users
-
         room_count = yield hs.get_datastore().get_room_count()
         stats["total_room_count"] = room_count
 
         stats["daily_active_users"] = yield hs.get_datastore().count_daily_users()
-        stats["daily_active_rooms"] = yield hs.get_datastore().count_daily_active_rooms()
-        stats["daily_messages"] = yield hs.get_datastore().count_daily_messages()
-
-        r30_results = yield hs.get_datastore().count_r30_users()
-        for name, count in r30_results.iteritems():
-            stats["r30_users_" + name] = count
-
-        daily_sent_messages = yield hs.get_datastore().count_daily_sent_messages()
-        stats["daily_sent_messages"] = daily_sent_messages
-        stats["cache_factor"] = CACHE_SIZE_FACTOR
-        stats["event_cache_size"] = hs.config.event_cache_size
-
-        if len(stats_process) > 0:
-            stats["memory_rss"] = 0
-            stats["cpu_average"] = 0
-            for process in stats_process:
-                stats["memory_rss"] += process.memory_info().rss
-                stats["cpu_average"] += int(process.cpu_percent(interval=None))
+        daily_messages = yield hs.get_datastore().count_daily_messages()
+        if daily_messages is not None:
+            stats["daily_messages"] = daily_messages
 
         logger.info("Reporting stats to matrix.org: %s" % (stats,))
         try:
@@ -455,49 +413,38 @@ def run(hs):
         except Exception as e:
             logger.warn("Error reporting stats: %s", e)
 
-    def performance_stats_init():
-        try:
-            import psutil
-            process = psutil.Process()
-            # Ensure we can fetch both, and make the initial request for cpu_percent
-            # so the next request will use this as the initial point.
-            process.memory_info().rss
-            process.cpu_percent(interval=None)
-            logger.info("report_stats can use psutil")
-            stats_process.append(process)
-        except (ImportError, AttributeError):
-            logger.warn(
-                "report_stats enabled but psutil is not installed or incorrect version."
-                " Disabling reporting of memory/cpu stats."
-                " Ensuring psutil is available will help matrix.org track performance"
-                " changes across releases."
-            )
-
     if hs.config.report_stats:
-        logger.info("Scheduling stats reporting for 3 hour intervals")
-        clock.looping_call(phone_stats_home, 3 * 60 * 60 * 1000)
+        phone_home_task = task.LoopingCall(phone_stats_home)
+        logger.info("Scheduling stats reporting for 24 hour intervals")
+        phone_home_task.start(60 * 60 * 24, now=False)
 
-        # We need to defer this init for the cases that we daemonize
-        # otherwise the process ID we get is that of the non-daemon process
-        clock.call_later(0, performance_stats_init)
+    def in_thread():
+        # Uncomment to enable tracing of log context changes.
+        # sys.settrace(logcontext_tracer)
+        with LoggingContext("run"):
+            change_resource_limit(hs.config.soft_file_limit)
+            if hs.config.gc_thresholds:
+                gc.set_threshold(*hs.config.gc_thresholds)
+            reactor.run()
 
-        # We wait 5 minutes to send the first set of stats as the server can
-        # be quite busy the first few minutes
-        clock.call_later(5 * 60, phone_stats_home)
+    if hs.config.daemonize:
 
-    if hs.config.daemonize and hs.config.print_pidfile:
+        if hs.config.print_pidfile:
             print (hs.config.pid_file)
 
-    _base.start_reactor(
-        "synapse-homeserver",
-        hs.config.soft_file_limit,
-        hs.config.gc_thresholds,
-        hs.config.pid_file,
-        hs.config.daemonize,
-        hs.config.cpu_affinity,
-        logger,
+        daemon = Daemonize(
+            app="synapse-homeserver",
+            pid=hs.config.pid_file,
+            action=lambda: in_thread(),
+            auto_close_fds=False,
+            verbose=True,
+            logger=logger,
         )
 
+        daemon.start()
+    else:
+        in_thread()
+
 
 def main():
     with LoggingContext("main"):

synapse/app/media_repository.py

@@ -1,174 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-# Copyright 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-import logging
-import sys
-
-import synapse
-from synapse import events
-from synapse.api.urls import (
-    CONTENT_REPO_PREFIX, LEGACY_MEDIA_PREFIX, MEDIA_PREFIX
-)
-from synapse.app import _base
-from synapse.config._base import ConfigError
-from synapse.config.homeserver import HomeServerConfig
-from synapse.config.logger import setup_logging
-from synapse.crypto import context_factory
-from synapse.http.site import SynapseSite
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
-from synapse.replication.slave.storage._base import BaseSlavedStore
-from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
-from synapse.replication.slave.storage.client_ips import SlavedClientIpStore
-from synapse.replication.slave.storage.registration import SlavedRegistrationStore
-from synapse.replication.slave.storage.transactions import TransactionStore
-from synapse.replication.tcp.client import ReplicationClientHandler
-from synapse.rest.media.v0.content_repository import ContentRepoResource
-from synapse.server import HomeServer
-from synapse.storage.engines import create_engine
-from synapse.storage.media_repository import MediaRepositoryStore
-from synapse.util.httpresourcetree import create_resource_tree
-from synapse.util.logcontext import LoggingContext
-from synapse.util.manhole import manhole
-from synapse.util.versionstring import get_version_string
-from twisted.internet import reactor
-from twisted.web.resource import NoResource
-
-logger = logging.getLogger("synapse.app.media_repository")
-
-
-class MediaRepositorySlavedStore(
-    SlavedApplicationServiceStore,
-    SlavedRegistrationStore,
-    SlavedClientIpStore,
-    TransactionStore,
-    BaseSlavedStore,
-    MediaRepositoryStore,
-):
-    pass
-
-
-class MediaRepositoryServer(HomeServer):
-    def setup(self):
-        logger.info("Setting up.")
-        self.datastore = MediaRepositorySlavedStore(self.get_db_conn(), self)
-        logger.info("Finished setting up.")
-
-    def _listen_http(self, listener_config):
-        port = listener_config["port"]
-        bind_addresses = listener_config["bind_addresses"]
-        site_tag = listener_config.get("tag", port)
-        resources = {}
-        for res in listener_config["resources"]:
-            for name in res["names"]:
-                if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(self)
-                elif name == "media":
-                    media_repo = self.get_media_repository_resource()
-                    resources.update({
-                        MEDIA_PREFIX: media_repo,
-                        LEGACY_MEDIA_PREFIX: media_repo,
-                        CONTENT_REPO_PREFIX: ContentRepoResource(
-                            self, self.config.uploads_path
-                        ),
-                    })
-
-        root_resource = create_resource_tree(resources, NoResource())
-
-        _base.listen_tcp(
-            bind_addresses,
-            port,
-            SynapseSite(
-                "synapse.access.http.%s" % (site_tag,),
-                site_tag,
-                listener_config,
-                root_resource,
-            )
-        )
-
-        logger.info("Synapse media repository now listening on port %d", port)
-
-    def start_listening(self, listeners):
-        for listener in listeners:
-            if listener["type"] == "http":
-                self._listen_http(listener)
-            elif listener["type"] == "manhole":
-                _base.listen_tcp(
-                    listener["bind_addresses"],
-                    listener["port"],
-                    manhole(
-                        username="matrix",
-                        password="rabbithole",
-                        globals={"hs": self},
-                    )
-                )
-            else:
-                logger.warn("Unrecognized listener type: %s", listener["type"])
-
-        self.get_tcp_replication().start_replication(self)
-
-    def build_tcp_replication(self):
-        return ReplicationClientHandler(self.get_datastore())
-
-
-def start(config_options):
-    try:
-        config = HomeServerConfig.load_config(
-            "Synapse media repository", config_options
-        )
-    except ConfigError as e:
-        sys.stderr.write("\n" + e.message + "\n")
-        sys.exit(1)
-
-    assert config.worker_app == "synapse.app.media_repository"
-
-    if config.enable_media_repo:
-        _base.quit_with_error(
-            "enable_media_repo must be disabled in the main synapse process\n"
-            "before the media repo can be run in a separate worker.\n"
-            "Please add ``enable_media_repo: false`` to the main config\n"
-        )
-
-    setup_logging(config, use_worker_options=True)
-
-    events.USE_FROZEN_DICTS = config.use_frozen_dicts
-
-    database_engine = create_engine(config.database_config)
-
-    tls_server_context_factory = context_factory.ServerContextFactory(config)
-
-    ss = MediaRepositoryServer(
-        config.server_name,
-        db_config=config.database_config,
-        tls_server_context_factory=tls_server_context_factory,
-        config=config,
-        version_string="Synapse/" + get_version_string(synapse),
-        database_engine=database_engine,
-    )
-
-    ss.setup()
-    ss.start_listening(config.worker_listeners)
-
-    def start():
-        ss.get_state_handler().start_caching()
-        ss.get_datastore().start_profiling()
-
-    reactor.callWhenRunning(start)
-
-    _base.start_worker_reactor("synapse-media-repository", config)
-
-
-if __name__ == '__main__':
-    with LoggingContext("main"):
-        start(sys.argv[1:])

synapse/app/pusher.py

@@ -13,31 +13,37 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
-import sys
 
 import synapse
-from synapse import events
-from synapse.app import _base
+
+from synapse.server import HomeServer
 from synapse.config._base import ConfigError
-from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
+from synapse.config.homeserver import HomeServerConfig
 from synapse.http.site import SynapseSite
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
-from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
+from synapse.metrics.resource import MetricsResource, METRICS_PREFIX
+from synapse.storage.roommember import RoomMemberStore
 from synapse.replication.slave.storage.events import SlavedEventStore
 from synapse.replication.slave.storage.pushers import SlavedPusherStore
 from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
-from synapse.replication.tcp.client import ReplicationClientHandler
-from synapse.server import HomeServer
-from synapse.storage import DataStore
+from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
 from synapse.storage.engines import create_engine
+from synapse.storage import DataStore
+from synapse.util.async import sleep
 from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext, preserve_fn
 from synapse.util.manhole import manhole
+from synapse.util.rlimit import change_resource_limit
 from synapse.util.versionstring import get_version_string
-from twisted.internet import defer, reactor
-from twisted.web.resource import NoResource
+
+from twisted.internet import reactor, defer
+from twisted.web.resource import Resource
+
+from daemonize import Daemonize
+
+import sys
+import logging
+import gc
 
 logger = logging.getLogger("synapse.app.pusher")
 
@@ -74,19 +80,51 @@ class PusherSlaveStore(
         DataStore.get_profile_displayname.__func__
     )
 
+    # XXX: This is a bit broken because we don't persist forgotten rooms
+    # in a way that they can be streamed. This means that we don't have a
+    # way to invalidate the forgotten rooms cache correctly.
+    # For now we expire the cache every 10 minutes.
+    BROKEN_CACHE_EXPIRY_MS = 60 * 60 * 1000
+    who_forgot_in_room = (
+        RoomMemberStore.__dict__["who_forgot_in_room"]
+    )
+
 
 class PusherServer(HomeServer):
+
+    def get_db_conn(self, run_new_connection=True):
+        # Any param beginning with cp_ is a parameter for adbapi, and should
+        # not be passed to the database engine.
+        db_params = {
+            k: v for k, v in self.db_config.get("args", {}).items()
+            if not k.startswith("cp_")
+        }
+        db_conn = self.database_engine.module.connect(**db_params)
+
+        if run_new_connection:
+            self.database_engine.on_new_connection(db_conn)
+        return db_conn
+
     def setup(self):
         logger.info("Setting up.")
         self.datastore = PusherSlaveStore(self.get_db_conn(), self)
         logger.info("Finished setting up.")
 
     def remove_pusher(self, app_id, push_key, user_id):
-        self.get_tcp_replication().send_remove_pusher(app_id, push_key, user_id)
+        http_client = self.get_simple_http_client()
+        replication_url = self.config.worker_replication_url
+        url = replication_url + "/remove_pushers"
+        return http_client.post_json_get_json(url, {
+            "remove": [{
+                "app_id": app_id,
+                "push_key": push_key,
+                "user_id": user_id,
+            }]
+        })
 
     def _listen_http(self, listener_config):
         port = listener_config["port"]
-        bind_addresses = listener_config["bind_addresses"]
+        bind_address = listener_config.get("bind_address", "")
         site_tag = listener_config.get("tag", port)
         resources = {}
         for res in listener_config["resources"]:
@@ -94,19 +132,17 @@ class PusherServer(HomeServer):
                 if name == "metrics":
                     resources[METRICS_PREFIX] = MetricsResource(self)
 
-        root_resource = create_resource_tree(resources, NoResource())
-
-        _base.listen_tcp(
-            bind_addresses,
+        root_resource = create_resource_tree(resources, Resource())
+        reactor.listenTCP(
             port,
             SynapseSite(
                 "synapse.access.http.%s" % (site_tag,),
                 site_tag,
                 listener_config,
                 root_resource,
+            ),
+            interface=bind_address
         )
-        )
-
         logger.info("Synapse pusher now listening on port %d", port)
 
     def start_listening(self, listeners):
@@ -114,64 +150,96 @@ class PusherServer(HomeServer):
             if listener["type"] == "http":
                 self._listen_http(listener)
             elif listener["type"] == "manhole":
-                _base.listen_tcp(
-                    listener["bind_addresses"],
+                reactor.listenTCP(
                     listener["port"],
                     manhole(
                         username="matrix",
                         password="rabbithole",
                         globals={"hs": self},
-                    )
+                    ),
+                    interface=listener.get("bind_address", '127.0.0.1')
                 )
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 
-        self.get_tcp_replication().start_replication(self)
-
-    def build_tcp_replication(self):
-        return PusherReplicationHandler(self)
-
-
-class PusherReplicationHandler(ReplicationClientHandler):
-    def __init__(self, hs):
-        super(PusherReplicationHandler, self).__init__(hs.get_datastore())
-
-        self.pusher_pool = hs.get_pusherpool()
-
-    def on_rdata(self, stream_name, token, rows):
-        super(PusherReplicationHandler, self).on_rdata(stream_name, token, rows)
-        preserve_fn(self.poke_pushers)(stream_name, token, rows)
-
     @defer.inlineCallbacks
-    def poke_pushers(self, stream_name, token, rows):
-        if stream_name == "pushers":
-            for row in rows:
-                if row.deleted:
-                    yield self.stop_pusher(row.user_id, row.app_id, row.pushkey)
-                else:
-                    yield self.start_pusher(row.user_id, row.app_id, row.pushkey)
-        elif stream_name == "events":
-            yield self.pusher_pool.on_new_notifications(
-                token, token,
-            )
-        elif stream_name == "receipts":
-            yield self.pusher_pool.on_new_receipts(
-                token, token, set(row.room_id for row in rows)
-            )
+    def replicate(self):
+        http_client = self.get_simple_http_client()
+        store = self.get_datastore()
+        replication_url = self.config.worker_replication_url
+        pusher_pool = self.get_pusherpool()
+        clock = self.get_clock()
 
-    def stop_pusher(self, user_id, app_id, pushkey):
+        def stop_pusher(user_id, app_id, pushkey):
             key = "%s:%s" % (app_id, pushkey)
-        pushers_for_user = self.pusher_pool.pushers.get(user_id, {})
+            pushers_for_user = pusher_pool.pushers.get(user_id, {})
             pusher = pushers_for_user.pop(key, None)
             if pusher is None:
                 return
             logger.info("Stopping pusher %r / %r", user_id, key)
             pusher.on_stop()
 
-    def start_pusher(self, user_id, app_id, pushkey):
+        def start_pusher(user_id, app_id, pushkey):
             key = "%s:%s" % (app_id, pushkey)
             logger.info("Starting pusher %r / %r", user_id, key)
-        return self.pusher_pool._refresh_pusher(app_id, pushkey, user_id)
+            return pusher_pool._refresh_pusher(app_id, pushkey, user_id)
+
+        @defer.inlineCallbacks
+        def poke_pushers(results):
+            pushers_rows = set(
+                map(tuple, results.get("pushers", {}).get("rows", []))
+            )
+            deleted_pushers_rows = set(
+                map(tuple, results.get("deleted_pushers", {}).get("rows", []))
+            )
+            for row in sorted(pushers_rows | deleted_pushers_rows):
+                if row in deleted_pushers_rows:
+                    user_id, app_id, pushkey = row[1:4]
+                    stop_pusher(user_id, app_id, pushkey)
+                elif row in pushers_rows:
+                    user_id = row[1]
+                    app_id = row[5]
+                    pushkey = row[8]
+                    yield start_pusher(user_id, app_id, pushkey)
+
+            stream = results.get("events")
+            if stream:
+                min_stream_id = stream["rows"][0][0]
+                max_stream_id = stream["position"]
+                preserve_fn(pusher_pool.on_new_notifications)(
+                    min_stream_id, max_stream_id
+                )
+
+            stream = results.get("receipts")
+            if stream:
+                rows = stream["rows"]
+                affected_room_ids = set(row[1] for row in rows)
+                min_stream_id = rows[0][0]
+                max_stream_id = stream["position"]
+                preserve_fn(pusher_pool.on_new_receipts)(
+                    min_stream_id, max_stream_id, affected_room_ids
+                )
+
+        def expire_broken_caches():
+            store.who_forgot_in_room.invalidate_all()
+
+        next_expire_broken_caches_ms = 0
+        while True:
+            try:
+                args = store.stream_positions()
+                args["timeout"] = 30000
+                result = yield http_client.get_json(replication_url, args=args)
+                now_ms = clock.time_msec()
+                if now_ms > next_expire_broken_caches_ms:
+                    expire_broken_caches()
+                    next_expire_broken_caches_ms = (
+                        now_ms + store.BROKEN_CACHE_EXPIRY_MS
+                    )
+                yield store.process_replication(result)
+                poke_pushers(result)
+            except:
+                logger.exception("Error replicating from %r", replication_url)
+                yield sleep(30)
 
 
 def start(config_options):
@@ -185,9 +253,7 @@ def start(config_options):
 
     assert config.worker_app == "synapse.app.pusher"
 
-    setup_logging(config, use_worker_options=True)
-
-    events.USE_FROZEN_DICTS = config.use_frozen_dicts
+    setup_logging(config.worker_log_config, config.worker_log_file)
 
     if config.start_pushers:
         sys.stderr.write(
@@ -207,21 +273,40 @@ def start(config_options):
         config.server_name,
         db_config=config.database_config,
         config=config,
-        version_string="Synapse/" + get_version_string(synapse),
+        version_string=get_version_string("Synapse", synapse),
         database_engine=database_engine,
     )
 
     ps.setup()
     ps.start_listening(config.worker_listeners)
 
+    def run():
+        with LoggingContext("run"):
+            logger.info("Running")
+            change_resource_limit(config.soft_file_limit)
+            if config.gc_thresholds:
+                gc.set_threshold(*config.gc_thresholds)
+            reactor.run()
+
     def start():
+        ps.replicate()
         ps.get_pusherpool().start()
         ps.get_datastore().start_profiling()
-        ps.get_state_handler().start_caching()
 
     reactor.callWhenRunning(start)
 
-    _base.start_worker_reactor("synapse-pusher", config)
+    if config.worker_daemonize:
+        daemon = Daemonize(
+            app="synapse-pusher",
+            pid=config.worker_pid_file,
+            action=run,
+            auto_close_fds=False,
+            verbose=True,
+            logger=logger,
+        )
+        daemon.start()
+    else:
+        run()
 
 
 if __name__ == '__main__':

synapse/app/synchrotron.py

@@ -13,87 +13,93 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import contextlib
-import logging
-import sys
 
 import synapse
-from synapse.api.constants import EventTypes
-from synapse.app import _base
+
+from synapse.api.constants import EventTypes, PresenceState
 from synapse.config._base import ConfigError
 from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
-from synapse.handlers.presence import PresenceHandler, get_interested_parties
-from synapse.http.server import JsonResource
+from synapse.events import FrozenEvent
+from synapse.handlers.presence import PresenceHandler
 from synapse.http.site import SynapseSite
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
+from synapse.http.server import JsonResource
+from synapse.metrics.resource import MetricsResource, METRICS_PREFIX
+from synapse.rest.client.v2_alpha import sync
 from synapse.replication.slave.storage._base import BaseSlavedStore
+from synapse.replication.slave.storage.events import SlavedEventStore
+from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
 from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
 from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
-from synapse.replication.slave.storage.client_ips import SlavedClientIpStore
-from synapse.replication.slave.storage.deviceinbox import SlavedDeviceInboxStore
-from synapse.replication.slave.storage.devices import SlavedDeviceStore
-from synapse.replication.slave.storage.events import SlavedEventStore
-from synapse.replication.slave.storage.filtering import SlavedFilteringStore
-from synapse.replication.slave.storage.presence import SlavedPresenceStore
-from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
-from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
 from synapse.replication.slave.storage.registration import SlavedRegistrationStore
-from synapse.replication.slave.storage.room import RoomStore
-from synapse.replication.slave.storage.groups import SlavedGroupServerStore
-from synapse.replication.tcp.client import ReplicationClientHandler
-from synapse.rest.client.v1 import events
-from synapse.rest.client.v1.initial_sync import InitialSyncRestServlet
-from synapse.rest.client.v1.room import RoomInitialSyncRestServlet
-from synapse.rest.client.v2_alpha import sync
+from synapse.replication.slave.storage.filtering import SlavedFilteringStore
+from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
+from synapse.replication.slave.storage.presence import SlavedPresenceStore
 from synapse.server import HomeServer
+from synapse.storage.client_ips import ClientIpStore
 from synapse.storage.engines import create_engine
-from synapse.storage.presence import UserPresenceState
+from synapse.storage.presence import PresenceStore, UserPresenceState
 from synapse.storage.roommember import RoomMemberStore
+from synapse.util.async import sleep
 from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.logcontext import LoggingContext, preserve_fn
 from synapse.util.manhole import manhole
+from synapse.util.rlimit import change_resource_limit
 from synapse.util.stringutils import random_string
 from synapse.util.versionstring import get_version_string
-from twisted.internet import defer, reactor
-from twisted.web.resource import NoResource
+
+from twisted.internet import reactor, defer
+from twisted.web.resource import Resource
+
+from daemonize import Daemonize
+
+import sys
+import logging
+import contextlib
+import gc
+import ujson as json
 
 logger = logging.getLogger("synapse.app.synchrotron")
 
 
 class SynchrotronSlavedStore(
+    SlavedPushRuleStore,
+    SlavedEventStore,
     SlavedReceiptsStore,
     SlavedAccountDataStore,
     SlavedApplicationServiceStore,
     SlavedRegistrationStore,
     SlavedFilteringStore,
     SlavedPresenceStore,
-    SlavedGroupServerStore,
-    SlavedDeviceInboxStore,
-    SlavedDeviceStore,
-    SlavedPushRuleStore,
-    SlavedEventStore,
-    SlavedClientIpStore,
-    RoomStore,
     BaseSlavedStore,
+    ClientIpStore,  # After BaseSlavedStore because the constructor is different
 ):
-    did_forget = (
-        RoomMemberStore.__dict__["did_forget"]
+    # XXX: This is a bit broken because we don't persist forgotten rooms
+    # in a way that they can be streamed. This means that we don't have a
+    # way to invalidate the forgotten rooms cache correctly.
+    # For now we expire the cache every 10 minutes.
+    BROKEN_CACHE_EXPIRY_MS = 60 * 60 * 1000
+    who_forgot_in_room = (
+        RoomMemberStore.__dict__["who_forgot_in_room"]
     )
 
+    # XXX: This is a bit broken because we don't persist the accepted list in a
+    # way that can be replicated. This means that we don't have a way to
+    # invalidate the cache correctly.
+    get_presence_list_accepted = PresenceStore.__dict__[
+        "get_presence_list_accepted"
+    ]
 
 UPDATE_SYNCING_USERS_MS = 10 * 1000
 
 
 class SynchrotronPresence(object):
     def __init__(self, hs):
-        self.hs = hs
-        self.is_mine_id = hs.is_mine_id
         self.http_client = hs.get_simple_http_client()
         self.store = hs.get_datastore()
         self.user_to_num_current_syncs = {}
+        self.syncing_users_url = hs.config.worker_replication_url + "/syncing_users"
         self.clock = hs.get_clock()
-        self.notifier = hs.get_notifier()
 
         active_presence = self.store.take_presence_startup_info()
         self.user_to_current_state = {
@@ -101,70 +107,34 @@ class SynchrotronPresence(object):
             for state in active_presence
         }
 
-        # user_id -> last_sync_ms. Lists the users that have stopped syncing
-        # but we haven't notified the master of that yet
-        self.users_going_offline = {}
-
-        self._send_stop_syncing_loop = self.clock.looping_call(
-            self.send_stop_syncing, 10 * 1000
-        )
-
         self.process_id = random_string(16)
         logger.info("Presence process_id is %r", self.process_id)
 
-    def send_user_sync(self, user_id, is_syncing, last_sync_ms):
-        return
-        self.hs.get_tcp_replication().send_user_sync(user_id, is_syncing, last_sync_ms)
+        self._sending_sync = False
+        self._need_to_send_sync = False
+        self.clock.looping_call(
+            self._send_syncing_users_regularly,
+            UPDATE_SYNCING_USERS_MS,
+        )
 
-    def mark_as_coming_online(self, user_id):
-        """A user has started syncing. Send a UserSync to the master, unless they
-        had recently stopped syncing.
+        reactor.addSystemEventTrigger("before", "shutdown", self._on_shutdown)
 
-        Args:
-            user_id (str)
-        """
-        going_offline = self.users_going_offline.pop(user_id, None)
-        if not going_offline:
-            # Safe to skip because we haven't yet told the master they were offline
-            self.send_user_sync(user_id, True, self.clock.time_msec())
-
-    def mark_as_going_offline(self, user_id):
-        """A user has stopped syncing. We wait before notifying the master as
-        its likely they'll come back soon. This allows us to avoid sending
-        a stopped syncing immediately followed by a started syncing notification
-        to the master
-
-        Args:
-            user_id (str)
-        """
-        self.users_going_offline[user_id] = self.clock.time_msec()
-
-    def send_stop_syncing(self):
-        """Check if there are any users who have stopped syncing a while ago
-        and haven't come back yet. If there are poke the master about them.
-        """
-        now = self.clock.time_msec()
-        for user_id, last_sync_ms in self.users_going_offline.items():
-            if now - last_sync_ms > 10 * 1000:
-                self.users_going_offline.pop(user_id, None)
-                self.send_user_sync(user_id, False, last_sync_ms)
-
-    def set_state(self, user, state, ignore_status_msg=False):
+    def set_state(self, user, state):
         # TODO Hows this supposed to work?
         pass
 
     get_states = PresenceHandler.get_states.__func__
-    get_state = PresenceHandler.get_state.__func__
     current_state_for_users = PresenceHandler.current_state_for_users.__func__
 
+    @defer.inlineCallbacks
     def user_syncing(self, user_id, affect_presence):
         if affect_presence:
             curr_sync = self.user_to_num_current_syncs.get(user_id, 0)
             self.user_to_num_current_syncs[user_id] = curr_sync + 1
-
-            # If we went from no in flight sync to some, notify replication
-            if self.user_to_num_current_syncs[user_id] == 1:
-                self.mark_as_coming_online(user_id)
+            prev_states = yield self.current_state_for_users([user_id])
+            if prev_states[user_id].state == PresenceState.OFFLINE:
+                # TODO: Don't block the sync request on this HTTP hit.
+                yield self._send_syncing_users_now()
 
         def _end():
             # We check that the user_id is in user_to_num_current_syncs because
@@ -173,10 +143,6 @@ class SynchrotronPresence(object):
             if affect_presence and user_id in self.user_to_num_current_syncs:
                 self.user_to_num_current_syncs[user_id] -= 1
 
-                # If we went from one in flight sync to non, notify replication
-                if self.user_to_num_current_syncs[user_id] == 0:
-                    self.mark_as_going_offline(user_id)
-
         @contextlib.contextmanager
         def _user_syncing():
             try:
@@ -184,39 +150,63 @@ class SynchrotronPresence(object):
             finally:
                 _end()
 
-        return defer.succeed(_user_syncing())
+        defer.returnValue(_user_syncing())
 
     @defer.inlineCallbacks
-    def notify_from_replication(self, states, stream_id):
-        parties = yield get_interested_parties(self.store, states)
-        room_ids_to_states, users_to_states = parties
+    def _on_shutdown(self):
+        # When the synchrotron is shutdown tell the master to clear the in
+        # progress syncs for this process
+        self.user_to_num_current_syncs.clear()
+        yield self._send_syncing_users_now()
 
-        self.notifier.on_new_event(
-            "presence_key", stream_id, rooms=room_ids_to_states.keys(),
-            users=users_to_states.keys()
-        )
+    def _send_syncing_users_regularly(self):
+        # Only send an update if we aren't in the middle of sending one.
+        if not self._sending_sync:
+            preserve_fn(self._send_syncing_users_now)()
 
     @defer.inlineCallbacks
-    def process_replication_rows(self, token, rows):
-        states = [UserPresenceState(
-            row.user_id, row.state, row.last_active_ts,
-            row.last_federation_update_ts, row.last_user_sync_ts, row.status_msg,
-            row.currently_active
-        ) for row in rows]
+    def _send_syncing_users_now(self):
+        if self._sending_sync:
+            # We don't want to race with sending another update.
+            # Instead we wait for that update to finish and send another
+            # update afterwards.
+            self._need_to_send_sync = True
+            return
 
-        for state in states:
-            self.user_to_current_state[row.user_id] = state
+        # Flag that we are sending an update.
+        self._sending_sync = True
 
-        stream_id = token
-        yield self.notify_from_replication(states, stream_id)
-
-    def get_currently_syncing_users(self):
-        # presence is disabled on matrix.org, so we return the empty set
-        return set()
-        return [
-            user_id for user_id, count in self.user_to_num_current_syncs.iteritems()
+        yield self.http_client.post_json_get_json(self.syncing_users_url, {
+            "process_id": self.process_id,
+            "syncing_users": [
+                user_id for user_id, count in self.user_to_num_current_syncs.items()
                 if count > 0
-        ]
+            ],
+        })
+
+        # Unset the flag as we are no longer sending an update.
+        self._sending_sync = False
+        if self._need_to_send_sync:
+            # If something happened while we were sending the update then
+            # we might need to send another update.
+            # TODO: Check if the update that was sent matches the current state
+            # as we only need to send an update if they are different.
+            self._need_to_send_sync = False
+            yield self._send_syncing_users_now()
+
+    def process_replication(self, result):
+        stream = result.get("presence", {"rows": []})
+        for row in stream["rows"]:
+            (
+                position, user_id, state, last_active_ts,
+                last_federation_update_ts, last_user_sync_ts, status_msg,
+                currently_active
+            ) = row
+            self.user_to_current_state[user_id] = UserPresenceState(
+                user_id, state, last_active_ts,
+                last_federation_update_ts, last_user_sync_ts, status_msg,
+                currently_active
+            )
 
 
 class SynchrotronTyping(object):
@@ -226,17 +216,18 @@ class SynchrotronTyping(object):
         self._room_typing = {}
 
     def stream_positions(self):
-        # We must update this typing token from the response of the previous
-        # sync. In particular, the stream id may "reset" back to zero/a low
-        # value which we *must* use for the next replication request.
         return {"typing": self._latest_room_serial}
 
-    def process_replication_rows(self, token, rows):
-        self._latest_room_serial = token
+    def process_replication(self, result):
+        stream = result.get("typing")
+        if stream:
+            self._latest_room_serial = int(stream["position"])
 
-        for row in rows:
-            self._room_serials[row.room_id] = token
-            self._room_typing[row.room_id] = row.user_ids
+            for row in stream["rows"]:
+                position, room_id, typing_json = row
+                typing = json.loads(typing_json)
+                self._room_serials[room_id] = position
+                self._room_typing[room_id] = typing
 
 
 class SynchrotronApplicationService(object):
@@ -245,6 +236,19 @@ class SynchrotronApplicationService(object):
 
 
 class SynchrotronServer(HomeServer):
+    def get_db_conn(self, run_new_connection=True):
+        # Any param beginning with cp_ is a parameter for adbapi, and should
+        # not be passed to the database engine.
+        db_params = {
+            k: v for k, v in self.db_config.get("args", {}).items()
+            if not k.startswith("cp_")
+        }
+        db_conn = self.database_engine.module.connect(**db_params)
+
+        if run_new_connection:
+            self.database_engine.on_new_connection(db_conn)
+        return db_conn
+
     def setup(self):
         logger.info("Setting up.")
         self.datastore = SynchrotronSlavedStore(self.get_db_conn(), self)
@@ -252,7 +256,7 @@ class SynchrotronServer(HomeServer):
 
     def _listen_http(self, listener_config):
         port = listener_config["port"]
-        bind_addresses = listener_config["bind_addresses"]
+        bind_address = listener_config.get("bind_address", "")
         site_tag = listener_config.get("tag", port)
         resources = {}
         for res in listener_config["resources"]:
@@ -262,29 +266,23 @@ class SynchrotronServer(HomeServer):
                 elif name == "client":
                     resource = JsonResource(self, canonical_json=False)
                     sync.register_servlets(self, resource)
-                    events.register_servlets(self, resource)
-                    InitialSyncRestServlet(self).register(resource)
-                    RoomInitialSyncRestServlet(self).register(resource)
                     resources.update({
                         "/_matrix/client/r0": resource,
                         "/_matrix/client/unstable": resource,
                         "/_matrix/client/v2_alpha": resource,
-                        "/_matrix/client/api/v1": resource,
                     })
 
-        root_resource = create_resource_tree(resources, NoResource())
-
-        _base.listen_tcp(
-            bind_addresses,
+        root_resource = create_resource_tree(resources, Resource())
+        reactor.listenTCP(
             port,
             SynapseSite(
                 "synapse.access.http.%s" % (site_tag,),
                 site_tag,
                 listener_config,
                 root_resource,
+            ),
+            interface=bind_address
         )
-        )
-
         logger.info("Synapse synchrotron now listening on port %d", port)
 
     def start_listening(self, listeners):
@@ -292,22 +290,113 @@ class SynchrotronServer(HomeServer):
             if listener["type"] == "http":
                 self._listen_http(listener)
             elif listener["type"] == "manhole":
-                _base.listen_tcp(
-                    listener["bind_addresses"],
+                reactor.listenTCP(
                     listener["port"],
                     manhole(
                         username="matrix",
                         password="rabbithole",
                         globals={"hs": self},
-                    )
+                    ),
+                    interface=listener.get("bind_address", '127.0.0.1')
                 )
             else:
                 logger.warn("Unrecognized listener type: %s", listener["type"])
 
-        self.get_tcp_replication().start_replication(self)
+    @defer.inlineCallbacks
+    def replicate(self):
+        http_client = self.get_simple_http_client()
+        store = self.get_datastore()
+        replication_url = self.config.worker_replication_url
+        clock = self.get_clock()
+        notifier = self.get_notifier()
+        presence_handler = self.get_presence_handler()
+        typing_handler = self.get_typing_handler()
 
-    def build_tcp_replication(self):
-        return SyncReplicationHandler(self)
+        def expire_broken_caches():
+            store.who_forgot_in_room.invalidate_all()
+            store.get_presence_list_accepted.invalidate_all()
+
+        def notify_from_stream(
+            result, stream_name, stream_key, room=None, user=None
+        ):
+            stream = result.get(stream_name)
+            if stream:
+                position_index = stream["field_names"].index("position")
+                if room:
+                    room_index = stream["field_names"].index(room)
+                if user:
+                    user_index = stream["field_names"].index(user)
+
+                users = ()
+                rooms = ()
+                for row in stream["rows"]:
+                    position = row[position_index]
+
+                    if user:
+                        users = (row[user_index],)
+
+                    if room:
+                        rooms = (row[room_index],)
+
+                    notifier.on_new_event(
+                        stream_key, position, users=users, rooms=rooms
+                    )
+
+        def notify(result):
+            stream = result.get("events")
+            if stream:
+                max_position = stream["position"]
+                for row in stream["rows"]:
+                    position = row[0]
+                    internal = json.loads(row[1])
+                    event_json = json.loads(row[2])
+                    event = FrozenEvent(event_json, internal_metadata_dict=internal)
+                    extra_users = ()
+                    if event.type == EventTypes.Member:
+                        extra_users = (event.state_key,)
+                    notifier.on_new_room_event(
+                        event, position, max_position, extra_users
+                    )
+
+            notify_from_stream(
+                result, "push_rules", "push_rules_key", user="user_id"
+            )
+            notify_from_stream(
+                result, "user_account_data", "account_data_key", user="user_id"
+            )
+            notify_from_stream(
+                result, "room_account_data", "account_data_key", user="user_id"
+            )
+            notify_from_stream(
+                result, "tag_account_data", "account_data_key", user="user_id"
+            )
+            notify_from_stream(
+                result, "receipts", "receipt_key", room="room_id"
+            )
+            notify_from_stream(
+                result, "typing", "typing_key", room="room_id"
+            )
+
+        next_expire_broken_caches_ms = 0
+        while True:
+            try:
+                args = store.stream_positions()
+                args.update(typing_handler.stream_positions())
+                args["timeout"] = 30000
+                result = yield http_client.get_json(replication_url, args=args)
+                now_ms = clock.time_msec()
+                if now_ms > next_expire_broken_caches_ms:
+                    expire_broken_caches()
+                    next_expire_broken_caches_ms = (
+                        now_ms + store.BROKEN_CACHE_EXPIRY_MS
+                    )
+                yield store.process_replication(result)
+                typing_handler.process_replication(result)
+                presence_handler.process_replication(result)
+                notify(result)
+            except:
+                logger.exception("Error replicating from %r", replication_url)
+                yield sleep(5)
 
     def build_presence_handler(self):
         return SynchrotronPresence(self)
@@ -316,82 +405,6 @@ class SynchrotronServer(HomeServer):
         return SynchrotronTyping(self)
 
 
-class SyncReplicationHandler(ReplicationClientHandler):
-    def __init__(self, hs):
-        super(SyncReplicationHandler, self).__init__(hs.get_datastore())
-
-        self.store = hs.get_datastore()
-        self.typing_handler = hs.get_typing_handler()
-        # NB this is a SynchrotronPresence, not a normal PresenceHandler
-        self.presence_handler = hs.get_presence_handler()
-        self.notifier = hs.get_notifier()
-
-    def on_rdata(self, stream_name, token, rows):
-        super(SyncReplicationHandler, self).on_rdata(stream_name, token, rows)
-
-        preserve_fn(self.process_and_notify)(stream_name, token, rows)
-
-    def get_streams_to_replicate(self):
-        args = super(SyncReplicationHandler, self).get_streams_to_replicate()
-        args.update(self.typing_handler.stream_positions())
-        return args
-
-    def get_currently_syncing_users(self):
-        return self.presence_handler.get_currently_syncing_users()
-
-    @defer.inlineCallbacks
-    def process_and_notify(self, stream_name, token, rows):
-        if stream_name == "events":
-            # We shouldn't get multiple rows per token for events stream, so
-            # we don't need to optimise this for multiple rows.
-            for row in rows:
-                event = yield self.store.get_event(row.event_id)
-                extra_users = ()
-                if event.type == EventTypes.Member:
-                    extra_users = (event.state_key,)
-                max_token = self.store.get_room_max_stream_ordering()
-                self.notifier.on_new_room_event(
-                    event, token, max_token, extra_users
-                )
-        elif stream_name == "push_rules":
-            self.notifier.on_new_event(
-                "push_rules_key", token, users=[row.user_id for row in rows],
-            )
-        elif stream_name in ("account_data", "tag_account_data",):
-            self.notifier.on_new_event(
-                "account_data_key", token, users=[row.user_id for row in rows],
-            )
-        elif stream_name == "receipts":
-            self.notifier.on_new_event(
-                "receipt_key", token, rooms=[row.room_id for row in rows],
-            )
-        elif stream_name == "typing":
-            self.typing_handler.process_replication_rows(token, rows)
-            self.notifier.on_new_event(
-                "typing_key", token, rooms=[row.room_id for row in rows],
-            )
-        elif stream_name == "to_device":
-            entities = [row.entity for row in rows if row.entity.startswith("@")]
-            if entities:
-                self.notifier.on_new_event(
-                    "to_device_key", token, users=entities,
-                )
-        elif stream_name == "device_lists":
-            all_room_ids = set()
-            for row in rows:
-                room_ids = yield self.store.get_rooms_for_user(row.user_id)
-                all_room_ids.update(room_ids)
-            self.notifier.on_new_event(
-                "device_list_key", token, rooms=all_room_ids,
-            )
-        elif stream_name == "presence":
-            yield self.presence_handler.process_replication_rows(token, rows)
-        elif stream_name == "receipts":
-            self.notifier.on_new_event(
-                "groups_key", token, users=[row.user_id for row in rows],
-            )
-
-
 def start(config_options):
     try:
         config = HomeServerConfig.load_config(
@@ -403,9 +416,7 @@ def start(config_options):
 
     assert config.worker_app == "synapse.app.synchrotron"
 
-    setup_logging(config, use_worker_options=True)
-
-    synapse.events.USE_FROZEN_DICTS = config.use_frozen_dicts
+    setup_logging(config.worker_log_config, config.worker_log_file)
 
     database_engine = create_engine(config.database_config)
 
@@ -413,7 +424,7 @@ def start(config_options):
         config.server_name,
         db_config=config.database_config,
         config=config,
-        version_string="Synapse/" + get_version_string(synapse),
+        version_string=get_version_string("Synapse", synapse),
         database_engine=database_engine,
         application_service_handler=SynchrotronApplicationService(),
     )
@@ -421,13 +432,32 @@ def start(config_options):
     ss.setup()
     ss.start_listening(config.worker_listeners)
 
+    def run():
+        with LoggingContext("run"):
+            logger.info("Running")
+            change_resource_limit(config.soft_file_limit)
+            if config.gc_thresholds:
+                gc.set_threshold(*config.gc_thresholds)
+            reactor.run()
+
     def start():
         ss.get_datastore().start_profiling()
-        ss.get_state_handler().start_caching()
+        ss.replicate()
 
     reactor.callWhenRunning(start)
 
-    _base.start_worker_reactor("synapse-synchrotron", config)
+    if config.worker_daemonize:
+        daemon = Daemonize(
+            app="synapse-synchrotron",
+            pid=config.worker_pid_file,
+            action=run,
+            auto_close_fds=False,
+            verbose=True,
+            logger=logger,
+        )
+        daemon.start()
+    else:
+        run()
 
 
 if __name__ == '__main__':

synapse/app/synctl.py

@@ -23,27 +23,14 @@ import signal
 import subprocess
 import sys
 import yaml
-import errno
-import time
 
-SYNAPSE = [sys.executable, "-B", "-m", "synapse.app.homeserver"]
+SYNAPSE = ["python", "-B", "-m", "synapse.app.homeserver"]
 
 GREEN = "\x1b[1;32m"
-YELLOW = "\x1b[1;33m"
 RED = "\x1b[1;31m"
 NORMAL = "\x1b[m"
 
 
-def pid_running(pid):
-    try:
-        os.kill(pid, 0)
-        return True
-    except OSError as err:
-        if err.errno == errno.EPERM:
-            return True
-        return False
-
-
 def write(message, colour=NORMAL, stream=sys.stdout):
     if colour == NORMAL:
         stream.write(message + "\n")
@@ -51,11 +38,6 @@ def write(message, colour=NORMAL, stream=sys.stdout):
         stream.write(colour + message + NORMAL + "\n")
 
 
-def abort(message, colour=RED, stream=sys.stderr):
-    write(message, colour, stream)
-    sys.exit(1)
-
-
 def start(configfile):
     write("Starting ...")
     args = SYNAPSE
@@ -63,8 +45,7 @@ def start(configfile):
 
     try:
         subprocess.check_call(args)
-        write("started synapse.app.homeserver(%r)" %
-              (configfile,), colour=GREEN)
+        write("started synapse.app.homeserver(%r)" % (configfile,), colour=GREEN)
     except subprocess.CalledProcessError as e:
         write(
             "error starting (exit code: %d); see above for logs" % e.returncode,
@@ -95,20 +76,12 @@ def start_worker(app, configfile, worker_configfile):
 def stop(pidfile, app):
     if os.path.exists(pidfile):
         pid = int(open(pidfile).read())
-        try:
         os.kill(pid, signal.SIGTERM)
         write("stopped %s" % (app,), colour=GREEN)
-        except OSError as err:
-            if err.errno == errno.ESRCH:
-                write("%s not running" % (app,), colour=YELLOW)
-            elif err.errno == errno.EPERM:
-                abort("Cannot stop %s: Operation not permitted" % (app,))
-            else:
-                abort("Cannot stop %s: Unknown error" % (app,))
 
 
 Worker = collections.namedtuple("Worker", [
-    "app", "configfile", "pidfile", "cache_factor", "cache_factors",
+    "app", "configfile", "pidfile", "cache_factor"
 ])
 
 
@@ -125,7 +98,7 @@ def main():
         "configfile",
         nargs="?",
         default="homeserver.yaml",
-        help="the homeserver config file, defaults to homeserver.yaml",
+        help="the homeserver config file, defaults to homserver.yaml",
     )
     parser.add_argument(
         "-w", "--worker",
@@ -171,10 +144,6 @@ def main():
     if cache_factor:
         os.environ["SYNAPSE_CACHE_FACTOR"] = str(cache_factor)
 
-    cache_factors = config.get("synctl_cache_factors", {})
-    for cache_name, factor in cache_factors.iteritems():
-        os.environ["SYNAPSE_CACHE_FACTOR_" + cache_name.upper()] = str(factor)
-
     worker_configfiles = []
     if options.worker:
         start_stop_synapse = False
@@ -188,9 +157,6 @@ def main():
         worker_configfiles.append(worker_configfile)
 
     if options.all_processes:
-        # To start the main synapse with -a you need to add a worker file
-        # with worker_app == "synapse.app.homeserver"
-        start_stop_synapse = False
         worker_configdir = options.all_processes
         if not os.path.isdir(worker_configdir):
             write(
@@ -207,37 +173,12 @@ def main():
         with open(worker_configfile) as stream:
             worker_config = yaml.load(stream)
         worker_app = worker_config["worker_app"]
-        if worker_app == "synapse.app.homeserver":
-            # We need to special case all of this to pick up options that may
-            # be set in the main config file or in this worker config file.
-            worker_pidfile = (
-                worker_config.get("pid_file")
-                or pidfile
-            )
-            worker_cache_factor = worker_config.get("synctl_cache_factor") or cache_factor
-            worker_cache_factors = (
-                worker_config.get("synctl_cache_factors")
-                or cache_factors
-            )
-            daemonize = worker_config.get("daemonize") or config.get("daemonize")
-            assert daemonize, "Main process must have daemonize set to true"
-
-            # The master process doesn't support using worker_* config.
-            for key in worker_config:
-                if key == "worker_app":  # But we allow worker_app
-                    continue
-                assert not key.startswith("worker_"), \
-                    "Main process cannot use worker_* config"
-        else:
         worker_pidfile = worker_config["worker_pid_file"]
         worker_daemonize = worker_config["worker_daemonize"]
-            assert worker_daemonize, "In config %r: expected '%s' to be True" % (
-                worker_configfile, "worker_daemonize")
+        assert worker_daemonize  # TODO print something more user friendly
         worker_cache_factor = worker_config.get("synctl_cache_factor")
-            worker_cache_factors = worker_config.get("synctl_cache_factors", {})
         workers.append(Worker(
             worker_app, worker_configfile, worker_pidfile, worker_cache_factor,
-            worker_cache_factors,
         ))
 
     action = options.action
@@ -249,42 +190,22 @@ def main():
         if start_stop_synapse:
             stop(pidfile, "synapse.app.homeserver")
 
-    # Wait for synapse to actually shutdown before starting it again
-    if action == "restart":
-        running_pids = []
-        if start_stop_synapse and os.path.exists(pidfile):
-            running_pids.append(int(open(pidfile).read()))
-        for worker in workers:
-            if os.path.exists(worker.pidfile):
-                running_pids.append(int(open(worker.pidfile).read()))
-        if len(running_pids) > 0:
-            write("Waiting for process to exit before restarting...")
-            for running_pid in running_pids:
-                while pid_running(running_pid):
-                    time.sleep(0.2)
-            write("All processes exited; now restarting...")
+        # TODO: Wait for synapse to actually shutdown before starting it again
 
     if action == "start" or action == "restart":
         if start_stop_synapse:
-            # Check if synapse is already running
-            if os.path.exists(pidfile) and pid_running(int(open(pidfile).read())):
-                abort("synapse.app.homeserver already running")
             start(configfile)
 
         for worker in workers:
-            env = os.environ.copy()
-
             if worker.cache_factor:
                 os.environ["SYNAPSE_CACHE_FACTOR"] = str(worker.cache_factor)
 
-            for cache_name, factor in worker.cache_factors.iteritems():
-                os.environ["SYNAPSE_CACHE_FACTOR_" + cache_name.upper()] = str(factor)
-
             start_worker(worker.app, configfile, worker.configfile)
 
-            # Reset env back to the original
-            os.environ.clear()
-            os.environ.update(env)
+            if cache_factor:
+                os.environ["SYNAPSE_CACHE_FACTOR"] = str(cache_factor)
+            else:
+                os.environ.pop("SYNAPSE_CACHE_FACTOR", None)
 
 
 if __name__ == "__main__":

synapse/app/user_dir.py

@@ -1,224 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-# Copyright 2017 Vector Creations Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import logging
-import sys
-
-import synapse
-from synapse import events
-from synapse.app import _base
-from synapse.config._base import ConfigError
-from synapse.config.homeserver import HomeServerConfig
-from synapse.config.logger import setup_logging
-from synapse.crypto import context_factory
-from synapse.http.server import JsonResource
-from synapse.http.site import SynapseSite
-from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
-from synapse.replication.slave.storage._base import BaseSlavedStore
-from synapse.replication.slave.storage.appservice import SlavedApplicationServiceStore
-from synapse.replication.slave.storage.client_ips import SlavedClientIpStore
-from synapse.replication.slave.storage.events import SlavedEventStore
-from synapse.replication.slave.storage.registration import SlavedRegistrationStore
-from synapse.replication.tcp.client import ReplicationClientHandler
-from synapse.rest.client.v2_alpha import user_directory
-from synapse.server import HomeServer
-from synapse.storage.engines import create_engine
-from synapse.storage.user_directory import UserDirectoryStore
-from synapse.util.caches.stream_change_cache import StreamChangeCache
-from synapse.util.httpresourcetree import create_resource_tree
-from synapse.util.logcontext import LoggingContext, preserve_fn
-from synapse.util.manhole import manhole
-from synapse.util.versionstring import get_version_string
-from twisted.internet import reactor
-from twisted.web.resource import NoResource
-
-logger = logging.getLogger("synapse.app.user_dir")
-
-
-class UserDirectorySlaveStore(
-    SlavedEventStore,
-    SlavedApplicationServiceStore,
-    SlavedRegistrationStore,
-    SlavedClientIpStore,
-    UserDirectoryStore,
-    BaseSlavedStore,
-):
-    def __init__(self, db_conn, hs):
-        super(UserDirectorySlaveStore, self).__init__(db_conn, hs)
-
-        events_max = self._stream_id_gen.get_current_token()
-        curr_state_delta_prefill, min_curr_state_delta_id = self._get_cache_dict(
-            db_conn, "current_state_delta_stream",
-            entity_column="room_id",
-            stream_column="stream_id",
-            max_value=events_max,  # As we share the stream id with events token
-            limit=1000,
-        )
-        self._curr_state_delta_stream_cache = StreamChangeCache(
-            "_curr_state_delta_stream_cache", min_curr_state_delta_id,
-            prefilled_cache=curr_state_delta_prefill,
-        )
-
-        self._current_state_delta_pos = events_max
-
-    def stream_positions(self):
-        result = super(UserDirectorySlaveStore, self).stream_positions()
-        result["current_state_deltas"] = self._current_state_delta_pos
-        return result
-
-    def process_replication_rows(self, stream_name, token, rows):
-        if stream_name == "current_state_deltas":
-            self._current_state_delta_pos = token
-            for row in rows:
-                self._curr_state_delta_stream_cache.entity_has_changed(
-                    row.room_id, token
-                )
-        return super(UserDirectorySlaveStore, self).process_replication_rows(
-            stream_name, token, rows
-        )
-
-
-class UserDirectoryServer(HomeServer):
-    def setup(self):
-        logger.info("Setting up.")
-        self.datastore = UserDirectorySlaveStore(self.get_db_conn(), self)
-        logger.info("Finished setting up.")
-
-    def _listen_http(self, listener_config):
-        port = listener_config["port"]
-        bind_addresses = listener_config["bind_addresses"]
-        site_tag = listener_config.get("tag", port)
-        resources = {}
-        for res in listener_config["resources"]:
-            for name in res["names"]:
-                if name == "metrics":
-                    resources[METRICS_PREFIX] = MetricsResource(self)
-                elif name == "client":
-                    resource = JsonResource(self, canonical_json=False)
-                    user_directory.register_servlets(self, resource)
-                    resources.update({
-                        "/_matrix/client/r0": resource,
-                        "/_matrix/client/unstable": resource,
-                        "/_matrix/client/v2_alpha": resource,
-                        "/_matrix/client/api/v1": resource,
-                    })
-
-        root_resource = create_resource_tree(resources, NoResource())
-
-        _base.listen_tcp(
-            bind_addresses,
-            port,
-            SynapseSite(
-                "synapse.access.http.%s" % (site_tag,),
-                site_tag,
-                listener_config,
-                root_resource,
-            )
-        )
-
-        logger.info("Synapse user_dir now listening on port %d", port)
-
-    def start_listening(self, listeners):
-        for listener in listeners:
-            if listener["type"] == "http":
-                self._listen_http(listener)
-            elif listener["type"] == "manhole":
-                _base.listen_tcp(
-                    listener["bind_addresses"],
-                    listener["port"],
-                    manhole(
-                        username="matrix",
-                        password="rabbithole",
-                        globals={"hs": self},
-                    )
-                )
-            else:
-                logger.warn("Unrecognized listener type: %s", listener["type"])
-
-        self.get_tcp_replication().start_replication(self)
-
-    def build_tcp_replication(self):
-        return UserDirectoryReplicationHandler(self)
-
-
-class UserDirectoryReplicationHandler(ReplicationClientHandler):
-    def __init__(self, hs):
-        super(UserDirectoryReplicationHandler, self).__init__(hs.get_datastore())
-        self.user_directory = hs.get_user_directory_handler()
-
-    def on_rdata(self, stream_name, token, rows):
-        super(UserDirectoryReplicationHandler, self).on_rdata(
-            stream_name, token, rows
-        )
-        if stream_name == "current_state_deltas":
-            preserve_fn(self.user_directory.notify_new_event)()
-
-
-def start(config_options):
-    try:
-        config = HomeServerConfig.load_config(
-            "Synapse user directory", config_options
-        )
-    except ConfigError as e:
-        sys.stderr.write("\n" + e.message + "\n")
-        sys.exit(1)
-
-    assert config.worker_app == "synapse.app.user_dir"
-
-    setup_logging(config, use_worker_options=True)
-
-    events.USE_FROZEN_DICTS = config.use_frozen_dicts
-
-    database_engine = create_engine(config.database_config)
-
-    if config.update_user_directory:
-        sys.stderr.write(
-            "\nThe update_user_directory must be disabled in the main synapse process"
-            "\nbefore they can be run in a separate worker."
-            "\nPlease add ``update_user_directory: false`` to the main config"
-            "\n"
-        )
-        sys.exit(1)
-
-    # Force the pushers to start since they will be disabled in the main config
-    config.update_user_directory = True
-
-    tls_server_context_factory = context_factory.ServerContextFactory(config)
-
-    ps = UserDirectoryServer(
-        config.server_name,
-        db_config=config.database_config,
-        tls_server_context_factory=tls_server_context_factory,
-        config=config,
-        version_string="Synapse/" + get_version_string(synapse),
-        database_engine=database_engine,
-    )
-
-    ps.setup()
-    ps.start_listening(config.worker_listeners)
-
-    def start():
-        ps.get_datastore().start_profiling()
-        ps.get_state_handler().start_caching()
-
-    reactor.callWhenRunning(start)
-
-    _base.start_worker_reactor("synapse-user-dir", config)
-
-
-if __name__ == '__main__':
-    with LoggingContext("main"):
-        start(sys.argv[1:])

synapse/appservice/__init__.py

@@ -13,16 +13,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 from synapse.api.constants import EventTypes
-from synapse.util.caches.descriptors import cachedInlineCallbacks
-from synapse.types import GroupID, get_domain_from_id
-
-from twisted.internet import defer
 
 import logging
 import re
 
-from six import string_types
-
 logger = logging.getLogger(__name__)
 
 
@@ -84,27 +78,15 @@ class ApplicationService(object):
     # values.
     NS_LIST = [NS_USERS, NS_ALIASES, NS_ROOMS]
 
-    def __init__(self, token, hostname, url=None, namespaces=None, hs_token=None,
-                 sender=None, id=None, protocols=None, rate_limited=True):
+    def __init__(self, token, url=None, namespaces=None, hs_token=None,
+                 sender=None, id=None):
         self.token = token
         self.url = url
         self.hs_token = hs_token
         self.sender = sender
-        self.server_name = hostname
         self.namespaces = self._check_namespaces(namespaces)
         self.id = id
 
-        if "|" in self.id:
-            raise Exception("application service ID cannot contain '|' character")
-
-        # .protocols is a publicly visible field
-        if protocols:
-            self.protocols = set(protocols)
-        else:
-            self.protocols = set()
-
-        self.rate_limited = rate_limited
-
     def _check_namespaces(self, namespaces):
         # Sanity check that it is of the form:
         # {
@@ -129,112 +111,92 @@ class ApplicationService(object):
                     raise ValueError(
                         "Expected bool for 'exclusive' in ns '%s'" % ns
                     )
-                group_id = regex_obj.get("group_id")
-                if group_id:
-                    if not isinstance(group_id, str):
-                        raise ValueError(
-                            "Expected string for 'group_id' in ns '%s'" % ns
-                        )
-                    try:
-                        GroupID.from_string(group_id)
-                    except Exception:
-                        raise ValueError(
-                            "Expected valid group ID for 'group_id' in ns '%s'" % ns
-                        )
-
-                    if get_domain_from_id(group_id) != self.server_name:
-                        raise ValueError(
-                            "Expected 'group_id' to be this host in ns '%s'" % ns
-                        )
-
-                regex = regex_obj.get("regex")
-                if isinstance(regex, string_types):
-                    regex_obj["regex"] = re.compile(regex)  # Pre-compile regex
-                else:
+                if not isinstance(regex_obj.get("regex"), basestring):
                     raise ValueError(
                         "Expected string for 'regex' in ns '%s'" % ns
                     )
         return namespaces
 
-    def _matches_regex(self, test_string, namespace_key):
+    def _matches_regex(self, test_string, namespace_key, return_obj=False):
+        if not isinstance(test_string, basestring):
+            logger.error(
+                "Expected a string to test regex against, but got %s",
+                test_string
+            )
+            return False
+
         for regex_obj in self.namespaces[namespace_key]:
-            if regex_obj["regex"].match(test_string):
+            if re.match(regex_obj["regex"], test_string):
+                if return_obj:
                     return regex_obj
-        return None
+                return True
+        return False
 
     def _is_exclusive(self, ns_key, test_string):
-        regex_obj = self._matches_regex(test_string, ns_key)
+        regex_obj = self._matches_regex(test_string, ns_key, return_obj=True)
         if regex_obj:
             return regex_obj["exclusive"]
         return False
 
-    @defer.inlineCallbacks
-    def _matches_user(self, event, store):
-        if not event:
-            defer.returnValue(False)
-
-        if self.is_interested_in_user(event.sender):
-            defer.returnValue(True)
+    def _matches_user(self, event, member_list):
+        if (hasattr(event, "sender") and
+                self.is_interested_in_user(event.sender)):
+            return True
         # also check m.room.member state key
-        if (event.type == EventTypes.Member and
-                self.is_interested_in_user(event.state_key)):
-            defer.returnValue(True)
-
-        if not store:
-            defer.returnValue(False)
-
-        does_match = yield self._matches_user_in_member_list(event.room_id, store)
-        defer.returnValue(does_match)
-
-    @cachedInlineCallbacks(num_args=1, cache_context=True)
-    def _matches_user_in_member_list(self, room_id, store, cache_context):
-        member_list = yield store.get_users_in_room(
-            room_id, on_invalidate=cache_context.invalidate
-        )
-
+        if (hasattr(event, "type") and event.type == EventTypes.Member
+                and hasattr(event, "state_key")
+                and self.is_interested_in_user(event.state_key)):
+            return True
         # check joined member events
         for user_id in member_list:
             if self.is_interested_in_user(user_id):
-                defer.returnValue(True)
-        defer.returnValue(False)
+                return True
+        return False
 
     def _matches_room_id(self, event):
         if hasattr(event, "room_id"):
             return self.is_interested_in_room(event.room_id)
         return False
 
-    @defer.inlineCallbacks
-    def _matches_aliases(self, event, store):
-        if not store or not event:
-            defer.returnValue(False)
-
-        alias_list = yield store.get_aliases_for_room(event.room_id)
+    def _matches_aliases(self, event, alias_list):
         for alias in alias_list:
             if self.is_interested_in_alias(alias):
-                defer.returnValue(True)
-        defer.returnValue(False)
+                return True
+        return False
 
-    @defer.inlineCallbacks
-    def is_interested(self, event, store=None):
+    def is_interested(self, event, restrict_to=None, aliases_for_event=None,
+                      member_list=None):
         """Check if this service is interested in this event.
 
         Args:
             event(Event): The event to check.
-            store(DataStore)
+            restrict_to(str): The namespace to restrict regex tests to.
+            aliases_for_event(list): A list of all the known room aliases for
+            this event.
+            member_list(list): A list of all joined user_ids in this room.
         Returns:
             bool: True if this service would like to know about this event.
         """
-        # Do cheap checks first
-        if self._matches_room_id(event):
-            defer.returnValue(True)
+        if aliases_for_event is None:
+            aliases_for_event = []
+        if member_list is None:
+            member_list = []
 
-        if (yield self._matches_aliases(event, store)):
-            defer.returnValue(True)
+        if restrict_to and restrict_to not in ApplicationService.NS_LIST:
+            # this is a programming error, so fail early and raise a general
+            # exception
+            raise Exception("Unexpected restrict_to value: %s". restrict_to)
 
-        if (yield self._matches_user(event, store)):
-            defer.returnValue(True)
-
-        defer.returnValue(False)
+        if not restrict_to:
+            return (self._matches_user(event, member_list)
+                    or self._matches_aliases(event, aliases_for_event)
+                    or self._matches_room_id(event))
+        elif restrict_to == ApplicationService.NS_ALIASES:
+            return self._matches_aliases(event, aliases_for_event)
+        elif restrict_to == ApplicationService.NS_ROOMS:
+            return self._matches_room_id(event)
+        elif restrict_to == ApplicationService.NS_USERS:
+            return self._matches_user(event, member_list)
 
     def is_interested_in_user(self, user_id):
         return (
@@ -243,10 +205,10 @@ class ApplicationService(object):
         )
 
     def is_interested_in_alias(self, alias):
-        return bool(self._matches_regex(alias, ApplicationService.NS_ALIASES))
+        return self._matches_regex(alias, ApplicationService.NS_ALIASES)
 
     def is_interested_in_room(self, room_id):
-        return bool(self._matches_regex(room_id, ApplicationService.NS_ROOMS))
+        return self._matches_regex(room_id, ApplicationService.NS_ROOMS)
 
     def is_exclusive_user(self, user_id):
         return (
@@ -254,42 +216,11 @@ class ApplicationService(object):
             or user_id == self.sender
         )
 
-    def is_interested_in_protocol(self, protocol):
-        return protocol in self.protocols
-
     def is_exclusive_alias(self, alias):
         return self._is_exclusive(ApplicationService.NS_ALIASES, alias)
 
     def is_exclusive_room(self, room_id):
         return self._is_exclusive(ApplicationService.NS_ROOMS, room_id)
 
-    def get_exlusive_user_regexes(self):
-        """Get the list of regexes used to determine if a user is exclusively
-        registered by the AS
-        """
-        return [
-            regex_obj["regex"]
-            for regex_obj in self.namespaces[ApplicationService.NS_USERS]
-            if regex_obj["exclusive"]
-        ]
-
-    def get_groups_for_user(self, user_id):
-        """Get the groups that this user is associated with by this AS
-
-        Args:
-            user_id (str): The ID of the user.
-
-        Returns:
-            iterable[str]: an iterable that yields group_id strings.
-        """
-        return (
-            regex_obj["group_id"]
-            for regex_obj in self.namespaces[ApplicationService.NS_USERS]
-            if "group_id" in regex_obj and regex_obj["regex"].match(user_id)
-        )
-
-    def is_rate_limited(self):
-        return self.rate_limited
-
     def __str__(self):
         return "ApplicationService: %s" % (self.__dict__,)

synapse/appservice/api.py

@@ -14,13 +14,9 @@
 # limitations under the License.
 from twisted.internet import defer
 
-from synapse.api.constants import ThirdPartyEntityKind
 from synapse.api.errors import CodeMessageException
 from synapse.http.client import SimpleHttpClient
 from synapse.events.utils import serialize_event
-from synapse.util.logcontext import preserve_fn, make_deferred_yieldable
-from synapse.util.caches.response_cache import ResponseCache
-from synapse.types import ThirdPartyInstanceID
 
 import logging
 import urllib
@@ -28,42 +24,6 @@ import urllib
 logger = logging.getLogger(__name__)
 
 
-HOUR_IN_MS = 60 * 60 * 1000
-
-
-APP_SERVICE_PREFIX = "/_matrix/app/unstable"
-
-
-def _is_valid_3pe_metadata(info):
-    if "instances" not in info:
-        return False
-    if not isinstance(info["instances"], list):
-        return False
-    return True
-
-
-def _is_valid_3pe_result(r, field):
-    if not isinstance(r, dict):
-        return False
-
-    for k in (field, "protocol"):
-        if k not in r:
-            return False
-        if not isinstance(r[k], str):
-            return False
-
-    if "fields" not in r:
-        return False
-    fields = r["fields"]
-    if not isinstance(fields, dict):
-        return False
-    for k in fields.keys():
-        if not isinstance(fields[k], str):
-            return False
-
-    return True
-
-
 class ApplicationServiceApi(SimpleHttpClient):
     """This class manages HS -> AS communications, including querying and
     pushing.
@@ -73,13 +33,8 @@ class ApplicationServiceApi(SimpleHttpClient):
         super(ApplicationServiceApi, self).__init__(hs)
         self.clock = hs.get_clock()
 
-        self.protocol_meta_cache = ResponseCache(hs, "as_protocol_meta",
-                                                 timeout_ms=HOUR_IN_MS)
-
     @defer.inlineCallbacks
     def query_user(self, service, user_id):
-        if service.url is None:
-            defer.returnValue(False)
         uri = service.url + ("/users/%s" % urllib.quote(user_id))
         response = None
         try:
@@ -99,8 +54,6 @@ class ApplicationServiceApi(SimpleHttpClient):
 
     @defer.inlineCallbacks
     def query_alias(self, service, alias):
-        if service.url is None:
-            defer.returnValue(False)
         uri = service.url + ("/rooms/%s" % urllib.quote(alias))
         response = None
         try:
@@ -118,94 +71,8 @@ class ApplicationServiceApi(SimpleHttpClient):
             logger.warning("query_alias to %s threw exception %s", uri, ex)
         defer.returnValue(False)
 
-    @defer.inlineCallbacks
-    def query_3pe(self, service, kind, protocol, fields):
-        if kind == ThirdPartyEntityKind.USER:
-            required_field = "userid"
-        elif kind == ThirdPartyEntityKind.LOCATION:
-            required_field = "alias"
-        else:
-            raise ValueError(
-                "Unrecognised 'kind' argument %r to query_3pe()", kind
-            )
-        if service.url is None:
-            defer.returnValue([])
-
-        uri = "%s%s/thirdparty/%s/%s" % (
-            service.url,
-            APP_SERVICE_PREFIX,
-            kind,
-            urllib.quote(protocol)
-        )
-        try:
-            response = yield self.get_json(uri, fields)
-            if not isinstance(response, list):
-                logger.warning(
-                    "query_3pe to %s returned an invalid response %r",
-                    uri, response
-                )
-                defer.returnValue([])
-
-            ret = []
-            for r in response:
-                if _is_valid_3pe_result(r, field=required_field):
-                    ret.append(r)
-                else:
-                    logger.warning(
-                        "query_3pe to %s returned an invalid result %r",
-                        uri, r
-                    )
-
-            defer.returnValue(ret)
-        except Exception as ex:
-            logger.warning("query_3pe to %s threw exception %s", uri, ex)
-            defer.returnValue([])
-
-    def get_3pe_protocol(self, service, protocol):
-        if service.url is None:
-            defer.returnValue({})
-
-        @defer.inlineCallbacks
-        def _get():
-            uri = "%s%s/thirdparty/protocol/%s" % (
-                service.url,
-                APP_SERVICE_PREFIX,
-                urllib.quote(protocol)
-            )
-            try:
-                info = yield self.get_json(uri, {})
-
-                if not _is_valid_3pe_metadata(info):
-                    logger.warning("query_3pe_protocol to %s did not return a"
-                                   " valid result", uri)
-                    defer.returnValue(None)
-
-                for instance in info.get("instances", []):
-                    network_id = instance.get("network_id", None)
-                    if network_id is not None:
-                        instance["instance_id"] = ThirdPartyInstanceID(
-                            service.id, network_id,
-                        ).to_string()
-
-                defer.returnValue(info)
-            except Exception as ex:
-                logger.warning("query_3pe_protocol to %s threw exception %s",
-                               uri, ex)
-                defer.returnValue(None)
-
-        key = (service.id, protocol)
-        result = self.protocol_meta_cache.get(key)
-        if not result:
-            result = self.protocol_meta_cache.set(
-                key, preserve_fn(_get)()
-            )
-        return make_deferred_yieldable(result)
-
     @defer.inlineCallbacks
     def push_bulk(self, service, events, txn_id=None):
-        if service.url is None:
-            defer.returnValue(True)
-
         events = self._serialize(events)
 
         if txn_id is None:

synapse/appservice/scheduler.py

@@ -48,12 +48,9 @@ UP & quit           +---------- YES                       SUCCESS
 This is all tied together by the AppServiceScheduler which DIs the required
 components.
 """
-from twisted.internet import defer
 
 from synapse.appservice import ApplicationServiceState
-from synapse.util.logcontext import preserve_fn
-from synapse.util.metrics import Measure
-
+from twisted.internet import defer
 import logging
 
 logger = logging.getLogger(__name__)
@@ -76,7 +73,7 @@ class ApplicationServiceScheduler(object):
         self.txn_ctrl = _TransactionController(
             self.clock, self.store, self.as_api, create_recoverer
         )
-        self.queuer = _ServiceQueuer(self.txn_ctrl, self.clock)
+        self.queuer = _ServiceQueuer(self.txn_ctrl)
 
     @defer.inlineCallbacks
     def start(self):
@@ -97,36 +94,38 @@ class _ServiceQueuer(object):
     this schedules any other events in the queue to run.
     """
 
-    def __init__(self, txn_ctrl, clock):
+    def __init__(self, txn_ctrl):
         self.queued_events = {}  # dict of {service_id: [events]}
-        self.requests_in_flight = set()
+        self.pending_requests = {}  # dict of {service_id: Deferred}
         self.txn_ctrl = txn_ctrl
-        self.clock = clock
 
     def enqueue(self, service, event):
         # if this service isn't being sent something
-        self.queued_events.setdefault(service.id, []).append(event)
-        preserve_fn(self._send_request)(service)
+        if not self.pending_requests.get(service.id):
+            self._send_request(service, [event])
+        else:
+            # add to queue for this service
+            if service.id not in self.queued_events:
+                self.queued_events[service.id] = []
+            self.queued_events[service.id].append(event)
 
-    @defer.inlineCallbacks
-    def _send_request(self, service):
-        if service.id in self.requests_in_flight:
-            return
+    def _send_request(self, service, events):
+        # send request and add callbacks
+        d = self.txn_ctrl.send(service, events)
+        d.addBoth(self._on_request_finish)
+        d.addErrback(self._on_request_fail)
+        self.pending_requests[service.id] = d
 
-        self.requests_in_flight.add(service.id)
-        try:
-            while True:
-                events = self.queued_events.pop(service.id, [])
-                if not events:
-                    return
+    def _on_request_finish(self, service):
+        self.pending_requests[service.id] = None
+        # if there are queued events, then send them.
+        if (service.id in self.queued_events
+                and len(self.queued_events[service.id]) > 0):
+            self._send_request(service, self.queued_events[service.id])
+            self.queued_events[service.id] = []
 
-                with Measure(self.clock, "servicequeuer.send"):
-                    try:
-                        yield self.txn_ctrl.send(service, events)
-                    except Exception:
-                        logger.exception("AS request failed")
-        finally:
-            self.requests_in_flight.discard(service.id)
+    def _on_request_fail(self, err):
+        logger.error("AS request failed: %s", err)
 
 
 class _TransactionController(object):
@@ -150,12 +149,14 @@ class _TransactionController(object):
             if service_is_up:
                 sent = yield txn.send(self.as_api)
                 if sent:
-                    yield txn.complete(self.store)
+                    txn.complete(self.store)
                 else:
-                    preserve_fn(self._start_recoverer)(service)
+                    self._start_recoverer(service)
         except Exception as e:
             logger.exception(e)
-            preserve_fn(self._start_recoverer)(service)
+            self._start_recoverer(service)
+        # request has finished
+        defer.returnValue(service)
 
     @defer.inlineCallbacks
     def on_recovered(self, recoverer):

synapse/config/_base.py

@@ -19,8 +19,6 @@ import os
 import yaml
 from textwrap import dedent
 
-from six import integer_types
-
 
 class ConfigError(Exception):
     pass
@@ -51,7 +49,7 @@ Missing mandatory `server_name` config option.
 class Config(object):
     @staticmethod
     def parse_size(value):
-        if isinstance(value, integer_types):
+        if isinstance(value, int) or isinstance(value, long):
             return value
         sizes = {"K": 1024, "M": 1024 * 1024}
         size = 1
@@ -63,15 +61,14 @@ class Config(object):
 
     @staticmethod
     def parse_duration(value):
-        if isinstance(value, integer_types):
+        if isinstance(value, int) or isinstance(value, long):
             return value
         second = 1000
-        minute = 60 * second
-        hour = 60 * minute
+        hour = 60 * 60 * second
         day = 24 * hour
         week = 7 * day
         year = 365 * day
-        sizes = {"s": second, "m": minute, "h": hour, "d": day, "w": week, "y": year}
+        sizes = {"s": second, "h": hour, "d": day, "w": week, "y": year}
         size = 1
         suffix = value[-1]
         if suffix in sizes:
@@ -83,38 +80,22 @@ class Config(object):
     def abspath(file_path):
         return os.path.abspath(file_path) if file_path else file_path
 
-    @classmethod
-    def path_exists(cls, file_path):
-        """Check if a file exists
-
-        Unlike os.path.exists, this throws an exception if there is an error
-        checking if the file exists (for example, if there is a perms error on
-        the parent dir).
-
-        Returns:
-            bool: True if the file exists; False if not.
-        """
-        try:
-            os.stat(file_path)
-            return True
-        except OSError as e:
-            if e.errno != errno.ENOENT:
-                raise e
-            return False
-
     @classmethod
     def check_file(cls, file_path, config_name):
         if file_path is None:
             raise ConfigError(
                 "Missing config for %s."
+                " You must specify a path for the config file. You can "
+                "do this with the -c or --config-path option. "
+                "Adding --generate-config along with --server-name "
+                "<server name> will generate a config file at the given path."
                 % (config_name,)
             )
-        try:
-            os.stat(file_path)
-        except OSError as e:
+        if not os.path.exists(file_path):
             raise ConfigError(
-                "Error accessing file '%s' (config for %s): %s"
-                % (file_path, config_name, e.strerror)
+                "File %s config for %s doesn't exist."
+                " Try running again with --generate-config"
+                % (file_path, config_name,)
             )
         return cls.abspath(file_path)
 
@@ -266,7 +247,7 @@ class Config(object):
                     " -c CONFIG-FILE\""
                 )
             (config_path,) = config_files
-            if not cls.path_exists(config_path):
+            if not os.path.exists(config_path):
                 if config_args.keys_directory:
                     config_dir_path = config_args.keys_directory
                 else:
@@ -279,7 +260,7 @@ class Config(object):
                         "Must specify a server_name to a generate config for."
                         " Pass -H server.name."
                     )
-                if not cls.path_exists(config_dir_path):
+                if not os.path.exists(config_dir_path):
                     os.makedirs(config_dir_path)
                 with open(config_path, "wb") as config_file:
                     config_bytes, config = obj.generate_config(
@@ -290,22 +271,22 @@ class Config(object):
                     )
                     obj.invoke_all("generate_files", config)
                     config_file.write(config_bytes)
-                print((
+                print (
                     "A config file has been generated in %r for server name"
                     " %r with corresponding SSL keys and self-signed"
                     " certificates. Please review this file and customise it"
                     " to your needs."
-                ) % (config_path, server_name))
+                ) % (config_path, server_name)
                 print (
                     "If this server name is incorrect, you will need to"
                     " regenerate the SSL certificates"
                 )
                 return
             else:
-                print((
+                print (
                     "Config file %r already exists. Generating any missing key"
                     " files."
-                ) % (config_path,))
+                ) % (config_path,)
                 generate_keys = True
 
         parser = argparse.ArgumentParser(

synapse/config/appservice.py

@@ -21,8 +21,6 @@ import urllib
 import yaml
 import logging
 
-from six import string_types
-
 logger = logging.getLogger(__name__)
 
 
@@ -30,7 +28,6 @@ class AppServiceConfig(Config):
 
     def read_config(self, config):
         self.app_service_config_files = config.get("app_service_config_files", [])
-        self.notify_appservices = config.get("notify_appservices", True)
 
     def default_config(cls, **kwargs):
         return """\
@@ -88,22 +85,14 @@ def load_appservices(hostname, config_files):
 
 def _load_appservice(hostname, as_info, config_filename):
     required_string_fields = [
-        "id", "as_token", "hs_token", "sender_localpart"
+        "id", "url", "as_token", "hs_token", "sender_localpart"
     ]
     for field in required_string_fields:
-        if not isinstance(as_info.get(field), string_types):
+        if not isinstance(as_info.get(field), basestring):
             raise KeyError("Required string field: '%s' (%s)" % (
                 field, config_filename,
             ))
 
-    # 'url' must either be a string or explicitly null, not missing
-    # to avoid accidentally turning off push for ASes.
-    if (not isinstance(as_info.get("url"), string_types) and
-            as_info.get("url", "") is not None):
-        raise KeyError(
-            "Required string field or explicit null: 'url' (%s)" % (config_filename,)
-        )
-
     localpart = as_info["sender_localpart"]
     if urllib.quote(localpart) != localpart:
         raise ValueError(
@@ -112,11 +101,6 @@ def _load_appservice(hostname, as_info, config_filename):
     user = UserID(localpart, hostname)
     user_id = user.to_string()
 
-    # Rate limiting for users of this AS is on by default (excludes sender)
-    rate_limited = True
-    if isinstance(as_info.get("rate_limited"), bool):
-        rate_limited = as_info.get("rate_limited")
-
     # namespace checks
     if not isinstance(as_info.get("namespaces"), dict):
         raise KeyError("Requires 'namespaces' object.")
@@ -130,7 +114,7 @@ def _load_appservice(hostname, as_info, config_filename):
                         "Expected namespace entry in %s to be an object,"
                         " but got %s", ns, regex_obj
                     )
-                if not isinstance(regex_obj.get("regex"), string_types):
+                if not isinstance(regex_obj.get("regex"), basestring):
                     raise ValueError(
                         "Missing/bad type 'regex' key in %s", regex_obj
                     )
@@ -138,30 +122,11 @@ def _load_appservice(hostname, as_info, config_filename):
                     raise ValueError(
                         "Missing/bad type 'exclusive' key in %s", regex_obj
                     )
-    # protocols check
-    protocols = as_info.get("protocols")
-    if protocols:
-        # Because strings are lists in python
-        if isinstance(protocols, str) or not isinstance(protocols, list):
-            raise KeyError("Optional 'protocols' must be a list if present.")
-        for p in protocols:
-            if not isinstance(p, str):
-                raise KeyError("Bad value for 'protocols' item")
-
-    if as_info["url"] is None:
-        logger.info(
-            "(%s) Explicitly empty 'url' provided. This application service"
-            " will not receive events or queries.",
-            config_filename,
-        )
     return ApplicationService(
         token=as_info["as_token"],
-        hostname=hostname,
         url=as_info["url"],
         namespaces=as_info["namespaces"],
         hs_token=as_info["hs_token"],
         sender=user_id,
         id=as_info["id"],
-        protocols=protocols,
-        rate_limited=rate_limited
     )

synapse/config/cas.py

@@ -41,7 +41,7 @@ class CasConfig(Config):
         #cas_config:
         #   enabled: true
         #   server_url: "https://cas-server.com"
-        #   service_url: "https://homeserver.domain.com:8448"
+        #   service_url: "https://homesever.domain.com:8448"
         #   #required_attributes:
         #   #    name: value
         """

synapse/config/emailconfig.py

@@ -68,18 +68,6 @@ class EmailConfig(Config):
             self.email_notif_for_new_users = email_config.get(
                 "notif_for_new_users", True
             )
-            self.email_riot_base_url = email_config.get(
-                "riot_base_url", None
-            )
-            self.email_smtp_user = email_config.get(
-                "smtp_user", None
-            )
-            self.email_smtp_pass = email_config.get(
-                "smtp_pass", None
-            )
-            self.require_transport_security = email_config.get(
-                "require_transport_security", False
-            )
             if "app_name" in email_config:
                 self.email_app_name = email_config["app_name"]
             else:
@@ -97,25 +85,14 @@ class EmailConfig(Config):
     def default_config(self, config_dir_path, server_name, **kwargs):
         return """
         # Enable sending emails for notification events
-        # Defining a custom URL for Riot is only needed if email notifications
-        # should contain links to a self-hosted installation of Riot; when set
-        # the "app_name" setting is ignored.
-        #
-        # If your SMTP server requires authentication, the optional smtp_user &
-        # smtp_pass variables should be used
-        #
         #email:
         #   enable_notifs: false
         #   smtp_host: "localhost"
         #   smtp_port: 25
-        #   smtp_user: "exampleusername"
-        #   smtp_pass: "examplepassword"
-        #   require_transport_security: False
         #   notif_from: "Your Friendly %(app)s Home Server <noreply@example.com>"
         #   app_name: Matrix
         #   template_dir: res/templates
         #   notif_template_html: notif_mail.html
         #   notif_template_text: notif_mail.txt
         #   notif_for_new_users: True
-        #   riot_base_url: "http://localhost/riot"
         """

synapse/config/groups.py

@@ -1,32 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2017 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from ._base import Config
-
-
-class GroupsConfig(Config):
-    def read_config(self, config):
-        self.enable_group_creation = config.get("enable_group_creation", False)
-        self.group_creation_prefix = config.get("group_creation_prefix", "")
-
-    def default_config(self, **kwargs):
-        return """\
-        # Whether to allow non server admins to create groups on this server
-        enable_group_creation: false
-
-        # If enabled, non server admins can only create groups with local parts
-        # starting with this prefix
-        # group_creation_prefix: "unofficial/"
-        """

synapse/config/homeserver.py

@@ -30,22 +30,17 @@ from .saml2 import SAML2Config
 from .cas import CasConfig
 from .password import PasswordConfig
 from .jwt import JWTConfig
-from .password_auth_providers import PasswordAuthProviderConfig
+from .ldap import LDAPConfig
 from .emailconfig import EmailConfig
 from .workers import WorkerConfig
-from .push import PushConfig
-from .spam_checker import SpamCheckerConfig
-from .groups import GroupsConfig
-from .user_directory import UserDirectoryConfig
 
 
 class HomeServerConfig(TlsConfig, ServerConfig, DatabaseConfig, LoggingConfig,
                        RatelimitConfig, ContentRepositoryConfig, CaptchaConfig,
                        VoipConfig, RegistrationConfig, MetricsConfig, ApiConfig,
                        AppServiceConfig, KeyConfig, SAML2Config, CasConfig,
-                       JWTConfig, PasswordConfig, EmailConfig,
-                       WorkerConfig, PasswordAuthProviderConfig, PushConfig,
-                       SpamCheckerConfig, GroupsConfig, UserDirectoryConfig,):
+                       JWTConfig, LDAPConfig, PasswordConfig, EmailConfig,
+                       WorkerConfig,):
     pass
 
 

synapse/config/key.py

@@ -118,9 +118,10 @@ class KeyConfig(Config):
         signing_keys = self.read_file(signing_key_path, "signing_key")
         try:
             return read_signing_keys(signing_keys.splitlines(True))
-        except Exception as e:
+        except Exception:
             raise ConfigError(
-                "Error reading signing_key: %s" % (str(e))
+                "Error reading signing_key."
+                " Try running again with --generate-config"
             )
 
     def read_old_signing_keys(self, old_signing_keys):
@@ -140,8 +141,7 @@ class KeyConfig(Config):
 
     def generate_files(self, config):
         signing_key_path = config["signing_key_path"]
-
-        if not self.path_exists(signing_key_path):
+        if not os.path.exists(signing_key_path):
             with open(signing_key_path, "w") as signing_key_file:
                 key_id = "a_" + random_string(4)
                 write_signing_keys(

synapse/config/ldap.py

@@ -0,0 +1,100 @@
+# -*- coding: utf-8 -*-
+# Copyright 2015 Niklas Riekenbrauck
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from ._base import Config, ConfigError
+
+
+MISSING_LDAP3 = (
+    "Missing ldap3 library. This is required for LDAP Authentication."
+)
+
+
+class LDAPMode(object):
+    SIMPLE = "simple",
+    SEARCH = "search",
+
+    LIST = (SIMPLE, SEARCH)
+
+
+class LDAPConfig(Config):
+    def read_config(self, config):
+        ldap_config = config.get("ldap_config", {})
+
+        self.ldap_enabled = ldap_config.get("enabled", False)
+
+        if self.ldap_enabled:
+            # verify dependencies are available
+            try:
+                import ldap3
+                ldap3  # to stop unused lint
+            except ImportError:
+                raise ConfigError(MISSING_LDAP3)
+
+            self.ldap_mode = LDAPMode.SIMPLE
+
+            # verify config sanity
+            self.require_keys(ldap_config, [
+                "uri",
+                "base",
+                "attributes",
+            ])
+
+            self.ldap_uri = ldap_config["uri"]
+            self.ldap_start_tls = ldap_config.get("start_tls", False)
+            self.ldap_base = ldap_config["base"]
+            self.ldap_attributes = ldap_config["attributes"]
+
+            if "bind_dn" in ldap_config:
+                self.ldap_mode = LDAPMode.SEARCH
+                self.require_keys(ldap_config, [
+                    "bind_dn",
+                    "bind_password",
+                ])
+
+                self.ldap_bind_dn = ldap_config["bind_dn"]
+                self.ldap_bind_password = ldap_config["bind_password"]
+                self.ldap_filter = ldap_config.get("filter", None)
+
+            # verify attribute lookup
+            self.require_keys(ldap_config['attributes'], [
+                "uid",
+                "name",
+                "mail",
+            ])
+
+    def require_keys(self, config, required):
+        missing = [key for key in required if key not in config]
+        if missing:
+            raise ConfigError(
+                "LDAP enabled but missing required config values: {}".format(
+                    ", ".join(missing)
+                )
+            )
+
+    def default_config(self, **kwargs):
+        return """\
+        # ldap_config:
+        #   enabled: true
+        #   uri: "ldap://ldap.example.com:389"
+        #   start_tls: true
+        #   base: "ou=users,dc=example,dc=com"
+        #   attributes:
+        #      uid: "cn"
+        #      mail: "email"
+        #      name: "givenName"
+        #   #bind_dn:
+        #   #bind_password:
+        #   #filter: "(objectClass=posixAccount)"
+        """

synapse/config/logger.py

@@ -15,13 +15,14 @@
 
 from ._base import Config
 from synapse.util.logcontext import LoggingContextFilter
-from twisted.logger import globalLogBeginner, STDLibLogObserver
+from twisted.python.log import PythonLoggingObserver
 import logging
 import logging.config
 import yaml
 from string import Template
 import os
 import signal
+from synapse.util.debug import debug_deferreds
 
 
 DEFAULT_LOG_CONFIG = Template("""
@@ -29,8 +30,8 @@ version: 1
 
 formatters:
   precise:
-        format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - \
-%(request)s - %(message)s'
+   format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s\
+- %(message)s'
 
 filters:
   context:
@@ -45,18 +46,16 @@ handlers:
     maxBytes: 104857600
     backupCount: 10
     filters: [context]
+    level: INFO
   console:
     class: logging.StreamHandler
     formatter: precise
-        filters: [context]
 
 loggers:
     synapse:
         level: INFO
 
     synapse.storage.SQL:
-        # beware: increasing this to DEBUG will make synapse log sensitive
-        # information such as access tokens.
         level: INFO
 
 root:
@@ -69,24 +68,35 @@ class LoggingConfig(Config):
 
     def read_config(self, config):
         self.verbosity = config.get("verbose", 0)
-        self.no_redirect_stdio = config.get("no_redirect_stdio", False)
         self.log_config = self.abspath(config.get("log_config"))
         self.log_file = self.abspath(config.get("log_file"))
+        if config.get("full_twisted_stacktraces"):
+            debug_deferreds()
 
     def default_config(self, config_dir_path, server_name, **kwargs):
+        log_file = self.abspath("homeserver.log")
         log_config = self.abspath(
             os.path.join(config_dir_path, server_name + ".log.config")
         )
         return """
+        # Logging verbosity level.
+        verbose: 0
+
+        # File to write logging to
+        log_file: "%(log_file)s"
+
         # A yaml python logging config file
         log_config: "%(log_config)s"
+
+        # Stop twisted from discarding the stack traces of exceptions in
+        # deferreds by waiting a reactor tick before running a deferred's
+        # callbacks.
+        # full_twisted_stacktraces: true
         """ % locals()
 
     def read_arguments(self, args):
         if args.verbose is not None:
             self.verbosity = args.verbose
-        if args.no_redirect_stdio is not None:
-            self.no_redirect_stdio = args.no_redirect_stdio
         if args.log_config is not None:
             self.log_config = args.log_config
         if args.log_file is not None:
@@ -96,68 +106,48 @@ class LoggingConfig(Config):
         logging_group = parser.add_argument_group("logging")
         logging_group.add_argument(
             '-v', '--verbose', dest="verbose", action='count',
-            help="The verbosity level. Specify multiple times to increase "
-            "verbosity. (Ignored if --log-config is specified.)"
+            help="The verbosity level."
         )
         logging_group.add_argument(
             '-f', '--log-file', dest="log_file",
-            help="File to log to. (Ignored if --log-config is specified.)"
+            help="File to log to."
         )
         logging_group.add_argument(
             '--log-config', dest="log_config", default=None,
             help="Python logging config file"
         )
-        logging_group.add_argument(
-            '-n', '--no-redirect-stdio',
-            action='store_true', default=None,
-            help="Do not redirect stdout/stderr to the log"
-        )
 
     def generate_files(self, config):
         log_config = config.get("log_config")
         if log_config and not os.path.exists(log_config):
-            log_file = self.abspath("homeserver.log")
             with open(log_config, "wb") as log_config_file:
                 log_config_file.write(
-                    DEFAULT_LOG_CONFIG.substitute(log_file=log_file)
+                    DEFAULT_LOG_CONFIG.substitute(log_file=config["log_file"])
                 )
 
+    def setup_logging(self):
+        setup_logging(self.log_config, self.log_file, self.verbosity)
 
-def setup_logging(config, use_worker_options=False):
-    """ Set up python logging
-
-    Args:
-        config (LoggingConfig | synapse.config.workers.WorkerConfig):
-            configuration data
-
-        use_worker_options (bool): True to use 'worker_log_config' and
-            'worker_log_file' options instead of 'log_config' and 'log_file'.
-    """
-    log_config = (config.worker_log_config if use_worker_options
-                  else config.log_config)
-    log_file = (config.worker_log_file if use_worker_options
-                else config.log_file)
 
+def setup_logging(log_config=None, log_file=None, verbosity=None):
     log_format = (
         "%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s"
         " - %(message)s"
     )
-
     if log_config is None:
-        # We don't have a logfile, so fall back to the 'verbosity' param from
-        # the config or cmdline. (Note that we generate a log config for new
-        # installs, so this will be an unusual case)
+
         level = logging.INFO
         level_for_storage = logging.INFO
-        if config.verbosity:
+        if verbosity:
             level = logging.DEBUG
-            if config.verbosity > 1:
+            if verbosity > 1:
                 level_for_storage = logging.DEBUG
 
+        # FIXME: we need a logging.WARN for a -q quiet option
         logger = logging.getLogger('')
         logger.setLevel(level)
 
-        logging.getLogger('synapse.storage.SQL').setLevel(level_for_storage)
+        logging.getLogger('synapse.storage').setLevel(level_for_storage)
 
         formatter = logging.Formatter(log_format)
         if log_file:
@@ -170,29 +160,6 @@ def setup_logging(config, use_worker_options=False):
                 logger.info("Closing log file due to SIGHUP")
                 handler.doRollover()
                 logger.info("Opened new log file due to SIGHUP")
-        else:
-            handler = logging.StreamHandler()
-
-            def sighup(signum, stack):
-                pass
-
-        handler.setFormatter(formatter)
-
-        handler.addFilter(LoggingContextFilter(request=""))
-
-        logger.addHandler(handler)
-    else:
-        def load_log_config():
-            with open(log_config, 'r') as f:
-                logging.config.dictConfig(yaml.load(f))
-
-        def sighup(signum, stack):
-            # it might be better to use a file watcher or something for this.
-            logging.info("Reloading log config from %s due to SIGHUP",
-                         log_config)
-            load_log_config()
-
-        load_log_config()
 
             # TODO(paul): obviously this is a terrible mechanism for
             #   stealing SIGHUP, because it means no other part of synapse
@@ -201,19 +168,16 @@ def setup_logging(config, use_worker_options=False):
             #   it around.
             if getattr(signal, "SIGHUP"):
                 signal.signal(signal.SIGHUP, sighup)
+        else:
+            handler = logging.StreamHandler()
+        handler.setFormatter(formatter)
 
-    # It's critical to point twisted's internal logging somewhere, otherwise it
-    # stacks up and leaks kup to 64K object;
-    # see: https://twistedmatrix.com/trac/ticket/8164
-    #
-    # Routing to the python logging framework could be a performance problem if
-    # the handlers blocked for a long time as python.logging is a blocking API
-    # see https://twistedmatrix.com/documents/current/core/howto/logger.html
-    # filed as https://github.com/matrix-org/synapse/issues/1727
-    #
-    # However this may not be too much of a problem if we are just writing to a file.
-    observer = STDLibLogObserver()
-    globalLogBeginner.beginLoggingTo(
-        [observer],
-        redirectStandardIO=not config.no_redirect_stdio,
-    )
+        handler.addFilter(LoggingContextFilter(request=""))
+
+        logger.addHandler(handler)
+    else:
+        with open(log_config, 'r') as f:
+            logging.config.dictConfig(yaml.load(f))
+
+    observer = PythonLoggingObserver()
+    observer.start()

synapse/config/password_auth_providers.py

@@ -1,69 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2016 Openmarket
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from ._base import Config
-
-from synapse.util.module_loader import load_module
-
-LDAP_PROVIDER = 'ldap_auth_provider.LdapAuthProvider'
-
-
-class PasswordAuthProviderConfig(Config):
-    def read_config(self, config):
-        self.password_providers = []
-        providers = []
-
-        # We want to be backwards compatible with the old `ldap_config`
-        # param.
-        ldap_config = config.get("ldap_config", {})
-        if ldap_config.get("enabled", False):
-            providers.append({
-                'module': LDAP_PROVIDER,
-                'config': ldap_config,
-            })
-
-        providers.extend(config.get("password_providers", []))
-        for provider in providers:
-            mod_name = provider['module']
-
-            # This is for backwards compat when the ldap auth provider resided
-            # in this package.
-            if mod_name == "synapse.util.ldap_auth_provider.LdapAuthProvider":
-                mod_name = LDAP_PROVIDER
-
-            (provider_class, provider_config) = load_module({
-                "module": mod_name,
-                "config": provider['config'],
-            })
-
-            self.password_providers.append((provider_class, provider_config))
-
-    def default_config(self, **kwargs):
-        return """\
-        # password_providers:
-        #     - module: "ldap_auth_provider.LdapAuthProvider"
-        #       config:
-        #         enabled: true
-        #         uri: "ldap://ldap.example.com:389"
-        #         start_tls: true
-        #         base: "ou=users,dc=example,dc=com"
-        #         attributes:
-        #            uid: "cn"
-        #            mail: "email"
-        #            name: "givenName"
-        #         #bind_dn:
-        #         #bind_password:
-        #         #filter: "(objectClass=posixAccount)"
-        """

synapse/config/push.py

@@ -1,61 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2015, 2016 OpenMarket Ltd
-# Copyright 2017 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from ._base import Config
-
-
-class PushConfig(Config):
-    def read_config(self, config):
-        push_config = config.get("push", {})
-        self.push_include_content = push_config.get("include_content", True)
-
-        # There was a a 'redact_content' setting but mistakenly read from the
-        # 'email'section'. Check for the flag in the 'push' section, and log,
-        # but do not honour it to avoid nasty surprises when people upgrade.
-        if push_config.get("redact_content") is not None:
-            print(
-                "The push.redact_content content option has never worked. "
-                "Please set push.include_content if you want this behaviour"
-            )
-
-        # Now check for the one in the 'email' section and honour it,
-        # with a warning.
-        push_config = config.get("email", {})
-        redact_content = push_config.get("redact_content")
-        if redact_content is not None:
-            print(
-                "The 'email.redact_content' option is deprecated: "
-                "please set push.include_content instead"
-            )
-            self.push_include_content = not redact_content
-
-    def default_config(self, config_dir_path, server_name, **kwargs):
-        return """
-        # Clients requesting push notifications can either have the body of
-        # the message sent in the notification poke along with other details
-        # like the sender, or just the event ID and room ID (`event_id_only`).
-        # If clients choose the former, this option controls whether the
-        # notification request includes the content of the event (other details
-        # like the sender are still included). For `event_id_only` push, it
-        # has no effect.
-
-        # For modern android devices the notification content will still appear
-        # because it is loaded by the app. iPhone, however will send a
-        # notification saying only that a message arrived and who it came from.
-        #
-        #push:
-        #   include_content: true
-        """

synapse/config/registration.py

@@ -31,9 +31,8 @@ class RegistrationConfig(Config):
                 strtobool(str(config["disable_registration"]))
             )
 
-        self.registrations_require_3pid = config.get("registrations_require_3pid", [])
-        self.allowed_local_3pids = config.get("allowed_local_3pids", [])
         self.registration_shared_secret = config.get("registration_shared_secret")
+        self.user_creation_max_duration = int(config["user_creation_max_duration"])
 
         self.bcrypt_rounds = config.get("bcrypt_rounds", 12)
         self.trusted_third_party_id_servers = config["trusted_third_party_id_servers"]
@@ -43,8 +42,6 @@ class RegistrationConfig(Config):
             self.allow_guest_access and config.get("invite_3pid_guest", False)
         )
 
-        self.auto_join_rooms = config.get("auto_join_rooms", [])
-
     def default_config(self, **kwargs):
         registration_shared_secret = random_string_with_symbols(50)
 
@@ -54,32 +51,18 @@ class RegistrationConfig(Config):
         # Enable registration for new users.
         enable_registration: False
 
-        # The user must provide all of the below types of 3PID when registering.
-        #
-        # registrations_require_3pid:
-        #     - email
-        #     - msisdn
-
-        # Mandate that users are only allowed to associate certain formats of
-        # 3PIDs with accounts on this server.
-        #
-        # allowed_local_3pids:
-        #     - medium: email
-        #       pattern: ".*@matrix\\.org"
-        #     - medium: email
-        #       pattern: ".*@vector\\.im"
-        #     - medium: msisdn
-        #       pattern: "\\+44"
-
         # If set, allows registration by anyone who also has the shared
         # secret, even if registration is otherwise disabled.
         registration_shared_secret: "%(registration_shared_secret)s"
 
+        # Sets the expiry for the short term user creation in
+        # milliseconds. For instance the bellow duration is two weeks
+        # in milliseconds.
+        user_creation_max_duration: 1209600000
+
         # Set the number of bcrypt rounds used to generate password hash.
         # Larger numbers increase the work factor needed to generate the hash.
-        # The default number is 12 (which equates to 2^12 rounds).
-        # N.B. that increasing this will exponentially increase the time required
-        # to register or login - e.g. 24 => 2^24 rounds which will take >20 mins.
+        # The default number of rounds is 12.
         bcrypt_rounds: 12
 
         # Allows users to register as guests without a password/email/etc, and
@@ -92,12 +75,6 @@ class RegistrationConfig(Config):
         trusted_third_party_id_servers:
             - matrix.org
             - vector.im
-            - riot.im
-
-        # Users who register on this homeserver will automatically be joined
-        # to these rooms
-        #auto_join_rooms:
-        #    - "#example:example.com"
         """ % locals()
 
     def add_arguments(self, parser):

synapse/config/repository.py

@@ -16,8 +16,6 @@
 from ._base import Config, ConfigError
 from collections import namedtuple
 
-from synapse.util.module_loader import load_module
-
 
 MISSING_NETADDR = (
     "Missing netaddr library. This is required for URL preview API."
@@ -38,14 +36,6 @@ ThumbnailRequirement = namedtuple(
     "ThumbnailRequirement", ["width", "height", "method", "media_type"]
 )
 
-MediaStorageProviderConfig = namedtuple(
-    "MediaStorageProviderConfig", (
-        "store_local",  # Whether to store newly uploaded local files
-        "store_remote",  # Whether to store newly downloaded remote files
-        "store_synchronous",  # Whether to wait for successful storage for local uploads
-    ),
-)
-
 
 def parse_thumbnail_requirements(thumbnail_sizes):
     """ Takes a list of dictionaries with "width", "height", and "method" keys
@@ -80,64 +70,7 @@ class ContentRepositoryConfig(Config):
         self.max_upload_size = self.parse_size(config["max_upload_size"])
         self.max_image_pixels = self.parse_size(config["max_image_pixels"])
         self.max_spider_size = self.parse_size(config["max_spider_size"])
-
         self.media_store_path = self.ensure_directory(config["media_store_path"])
-
-        backup_media_store_path = config.get("backup_media_store_path")
-
-        synchronous_backup_media_store = config.get(
-            "synchronous_backup_media_store", False
-        )
-
-        storage_providers = config.get("media_storage_providers", [])
-
-        if backup_media_store_path:
-            if storage_providers:
-                raise ConfigError(
-                    "Cannot use both 'backup_media_store_path' and 'storage_providers'"
-                )
-
-            storage_providers = [{
-                "module": "file_system",
-                "store_local": True,
-                "store_synchronous": synchronous_backup_media_store,
-                "store_remote": True,
-                "config": {
-                    "directory": backup_media_store_path,
-                }
-            }]
-
-        # This is a list of config that can be used to create the storage
-        # providers. The entries are tuples of (Class, class_config,
-        # MediaStorageProviderConfig), where Class is the class of the provider,
-        # the class_config the config to pass to it, and
-        # MediaStorageProviderConfig are options for StorageProviderWrapper.
-        #
-        # We don't create the storage providers here as not all workers need
-        # them to be started.
-        self.media_storage_providers = []
-
-        for provider_config in storage_providers:
-            # We special case the module "file_system" so as not to need to
-            # expose FileStorageProviderBackend
-            if provider_config["module"] == "file_system":
-                provider_config["module"] = (
-                    "synapse.rest.media.v1.storage_provider"
-                    ".FileStorageProviderBackend"
-                )
-
-            provider_class, parsed_config = load_module(provider_config)
-
-            wrapper_config = MediaStorageProviderConfig(
-                provider_config.get("store_local", False),
-                provider_config.get("store_remote", False),
-                provider_config.get("store_synchronous", False),
-            )
-
-            self.media_storage_providers.append(
-                (provider_class, parsed_config, wrapper_config,)
-            )
-
         self.uploads_path = self.ensure_directory(config["uploads_path"])
         self.dynamic_thumbnails = config["dynamic_thumbnails"]
         self.thumbnail_requirements = parse_thumbnail_requirements(
@@ -182,20 +115,6 @@ class ContentRepositoryConfig(Config):
         # Directory where uploaded images and attachments are stored.
         media_store_path: "%(media_store)s"
 
-        # Media storage providers allow media to be stored in different
-        # locations.
-        # media_storage_providers:
-        # - module: file_system
-        #   # Whether to write new local files.
-        #   store_local: false
-        #   # Whether to write new remote media
-        #   store_remote: false
-        #   # Whether to block upload requests waiting for write to this
-        #   # provider to complete
-        #   store_synchronous: false
-        #   config:
-        #     directory: /mnt/some/other/directory
-
         # Directory where in-progress uploads are stored.
         uploads_path: "%(uploads_path)s"
 
@@ -248,8 +167,6 @@ class ContentRepositoryConfig(Config):
         # - '10.0.0.0/8'
         # - '172.16.0.0/12'
         # - '192.168.0.0/16'
-        # - '100.64.0.0/10'
-        # - '169.254.0.0/16'
         #
         # List of IP address CIDR ranges that the URL preview spider is allowed
         # to access even if they are specified in url_preview_ip_range_blacklist.

synapse/config/server.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2017 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -30,41 +29,7 @@ class ServerConfig(Config):
         self.user_agent_suffix = config.get("user_agent_suffix")
         self.use_frozen_dicts = config.get("use_frozen_dicts", False)
         self.public_baseurl = config.get("public_baseurl")
-        self.cpu_affinity = config.get("cpu_affinity")
-
-        # Whether to send federation traffic out in this process. This only
-        # applies to some federation traffic, and so shouldn't be used to
-        # "disable" federation
-        self.send_federation = config.get("send_federation", True)
-
-        # Whether to update the user directory or not. This should be set to
-        # false only if we are updating the user directory in a worker
-        self.update_user_directory = config.get("update_user_directory", True)
-
-        # whether to enable the media repository endpoints. This should be set
-        # to false if the media repository is running as a separate endpoint;
-        # doing so ensures that we will not run cache cleanup jobs on the
-        # master, potentially causing inconsistency.
-        self.enable_media_repo = config.get("enable_media_repo", True)
-
-        self.filter_timeline_limit = config.get("filter_timeline_limit", -1)
-
-        # Whether we should block invites sent to users on this server
-        # (other than those sent by local server admins)
-        self.block_non_admin_invites = config.get(
-            "block_non_admin_invites", False,
-        )
-
-        # FIXME: federation_domain_whitelist needs sytests
-        self.federation_domain_whitelist = None
-        federation_domain_whitelist = config.get(
-            "federation_domain_whitelist", None
-        )
-        # turn the whitelist into a hash for speed of lookup
-        if federation_domain_whitelist is not None:
-            self.federation_domain_whitelist = {}
-            for domain in federation_domain_whitelist:
-                self.federation_domain_whitelist[domain] = True
+        self.secondary_directory_servers = config.get("secondary_directory_servers", [])
 
         if self.public_baseurl is not None:
             if self.public_baseurl[-1] != '/':
@@ -73,15 +38,6 @@ class ServerConfig(Config):
 
         self.listeners = config.get("listeners", [])
 
-        for listener in self.listeners:
-            bind_address = listener.pop("bind_address", None)
-            bind_addresses = listener.setdefault("bind_addresses", [])
-
-            if bind_address:
-                bind_addresses.append(bind_address)
-            elif not bind_addresses:
-                bind_addresses.append('')
-
         self.gc_thresholds = read_gc_thresholds(config.get("gc_thresholds", None))
 
         bind_port = config.get("bind_port")
@@ -94,7 +50,7 @@ class ServerConfig(Config):
 
             self.listeners.append({
                 "port": bind_port,
-                "bind_addresses": [bind_host],
+                "bind_address": bind_host,
                 "tls": True,
                 "type": "http",
                 "resources": [
@@ -113,7 +69,7 @@ class ServerConfig(Config):
             if unsecure_port:
                 self.listeners.append({
                     "port": unsecure_port,
-                    "bind_addresses": [bind_host],
+                    "bind_address": bind_host,
                     "tls": False,
                     "type": "http",
                     "resources": [
@@ -132,7 +88,7 @@ class ServerConfig(Config):
         if manhole:
             self.listeners.append({
                 "port": manhole,
-                "bind_addresses": ["127.0.0.1"],
+                "bind_address": "127.0.0.1",
                 "type": "manhole",
             })
 
@@ -140,7 +96,7 @@ class ServerConfig(Config):
         if metrics_port:
             self.listeners.append({
                 "port": metrics_port,
-                "bind_addresses": [config.get("metrics_bind_host", "127.0.0.1")],
+                "bind_address": config.get("metrics_bind_host", "127.0.0.1"),
                 "tls": False,
                 "type": "http",
                 "resources": [
@@ -172,36 +128,9 @@ class ServerConfig(Config):
         # When running as a daemon, the file to store the pid in
         pid_file: %(pid_file)s
 
-        # CPU affinity mask. Setting this restricts the CPUs on which the
-        # process will be scheduled. It is represented as a bitmask, with the
-        # lowest order bit corresponding to the first logical CPU and the
-        # highest order bit corresponding to the last logical CPU. Not all CPUs
-        # may exist on a given system but a mask may specify more CPUs than are
-        # present.
-        #
-        # For example:
-        #    0x00000001  is processor #0,
-        #    0x00000003  is processors #0 and #1,
-        #    0xFFFFFFFF  is all processors (#0 through #31).
-        #
-        # Pinning a Python process to a single CPU is desirable, because Python
-        # is inherently single-threaded due to the GIL, and can suffer a
-        # 30-40%% slowdown due to cache blow-out and thread context switching
-        # if the scheduler happens to schedule the underlying threads across
-        # different cores. See
-        # https://www.mirantis.com/blog/improve-performance-python-programs-restricting-single-cpu/.
-        #
-        # cpu_affinity: 0xFFFFFFFF
-
         # Whether to serve a web client from the HTTP/HTTPS root resource.
         web_client: True
 
-        # The root directory to server for the above web client.
-        # If left undefined, synapse will serve the matrix-angular-sdk web client.
-        # Make sure matrix-angular-sdk is installed with pip if web_client is True
-        # and web_client_location is undefined
-        # web_client_location: "/path/to/web/root"
-
         # The public-facing base URL for the client API (not including _matrix/...)
         # public_baseurl: https://example.com:8448/
 
@@ -213,24 +142,13 @@ class ServerConfig(Config):
         # The GC threshold parameters to pass to `gc.set_threshold`, if defined
         # gc_thresholds: [700, 10, 10]
 
-        # Set the limit on the returned events in the timeline in the get
-        # and sync operations. The default value is -1, means no upper limit.
-        # filter_timeline_limit: 5000
-
-        # Whether room invites to users on this server should be blocked
-        # (except those sent by local server admins). The default is False.
-        # block_non_admin_invites: True
-
-        # Restrict federation to the following whitelist of domains.
-        # N.B. we recommend also firewalling your federation listener to limit
-        # inbound federation traffic as early as possible, rather than relying
-        # purely on this application-layer restriction.  If not specified, the
-        # default is to whitelist everything.
-        #
-        # federation_domain_whitelist:
-        #  - lon.example.com
-        #  - nyc.example.com
-        #  - syd.example.com
+        # A list of other Home Servers to fetch the public room directory from
+        # and include in the public room directory of this home server
+        # This is a temporary stopgap solution to populate new server with a
+        # list of rooms until there exists a good solution of a decentralized
+        # room directory.
+        # secondary_directory_servers:
+        #     - matrix.org
 
         # List of ports that Synapse should listen on, their purpose and their
         # configuration.
@@ -241,13 +159,9 @@ class ServerConfig(Config):
             # The port to listen for HTTPS requests on.
             port: %(bind_port)s
 
-            # Local addresses to listen on.
-            # On Linux and Mac OS, `::` will listen on all IPv4 and IPv6
-            # addresses by default. For most other OSes, this will only listen
-            # on IPv6.
-            bind_addresses:
-              - '::'
-              - '0.0.0.0'
+            # Local interface to listen on.
+            # The empty string will cause synapse to listen on all interfaces.
+            bind_address: ''
 
             # This is a 'http' listener, allows us to specify 'resources'.
             type: http
@@ -274,18 +188,11 @@ class ServerConfig(Config):
               - names: [federation]  # Federation APIs
                 compress: false
 
-            # optional list of additional endpoints which can be loaded via
-            # dynamic modules
-            # additional_resources:
-            #   "/_matrix/my/custom/endpoint":
-            #     module: my_module.CustomRequestHandler
-            #     config: {}
-
           # Unsecure HTTP listener,
           # For when matrix traffic passes through loadbalancer that unwraps TLS.
           - port: %(unsecure_port)s
             tls: false
-            bind_addresses: ['::', '0.0.0.0']
+            bind_address: ''
             type: http
 
             x_forwarded: false
@@ -299,7 +206,7 @@ class ServerConfig(Config):
           # Turn on the twisted ssh manhole service on localhost on the given
           # port.
           # - port: 9000
-          #   bind_addresses: ['::1', '127.0.0.1']
+          #   bind_address: 127.0.0.1
           #   type: manhole
         """ % locals()
 
@@ -337,7 +244,7 @@ def read_gc_thresholds(thresholds):
         return (
             int(thresholds[0]), int(thresholds[1]), int(thresholds[2]),
         )
-    except Exception:
+    except:
         raise ConfigError(
             "Value of `gc_threshold` must be a list of three integers if set"
         )

synapse/config/spam_checker.py

@@ -1,35 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2017 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from synapse.util.module_loader import load_module
-
-from ._base import Config
-
-
-class SpamCheckerConfig(Config):
-    def read_config(self, config):
-        self.spam_checker = None
-
-        provider = config.get("spam_checker", None)
-        if provider is not None:
-            self.spam_checker = load_module(provider)
-
-    def default_config(self, **kwargs):
-        return """\
-        # spam_checker:
-        #     module: "my_custom_project.SuperSpamChecker"
-        #     config:
-        #         example_option: 'things'
-        """

synapse/config/tls.py

@@ -19,9 +19,6 @@ from OpenSSL import crypto
 import subprocess
 import os
 
-from hashlib import sha256
-from unpaddedbase64 import encode_base64
-
 GENERATE_DH_PARAMS = False
 
 
@@ -45,19 +42,6 @@ class TlsConfig(Config):
             config.get("tls_dh_params_path"), "tls_dh_params"
         )
 
-        self.tls_fingerprints = config["tls_fingerprints"]
-
-        # Check that our own certificate is included in the list of fingerprints
-        # and include it if it is not.
-        x509_certificate_bytes = crypto.dump_certificate(
-            crypto.FILETYPE_ASN1,
-            self.tls_certificate
-        )
-        sha256_fingerprint = encode_base64(sha256(x509_certificate_bytes).digest())
-        sha256_fingerprints = set(f["sha256"] for f in self.tls_fingerprints)
-        if sha256_fingerprint not in sha256_fingerprints:
-            self.tls_fingerprints.append({u"sha256": sha256_fingerprint})
-
         # This config option applies to non-federation HTTP clients
         # (e.g. for talking to recaptcha, identity servers, and such)
         # It should never be used in production, and is intended for
@@ -89,34 +73,6 @@ class TlsConfig(Config):
 
         # Don't bind to the https port
         no_tls: False
-
-        # List of allowed TLS fingerprints for this server to publish along
-        # with the signing keys for this server. Other matrix servers that
-        # make HTTPS requests to this server will check that the TLS
-        # certificates returned by this server match one of the fingerprints.
-        #
-        # Synapse automatically adds the fingerprint of its own certificate
-        # to the list. So if federation traffic is handled directly by synapse
-        # then no modification to the list is required.
-        #
-        # If synapse is run behind a load balancer that handles the TLS then it
-        # will be necessary to add the fingerprints of the certificates used by
-        # the loadbalancers to this list if they are different to the one
-        # synapse is using.
-        #
-        # Homeservers are permitted to cache the list of TLS fingerprints
-        # returned in the key responses up to the "valid_until_ts" returned in
-        # key. It may be necessary to publish the fingerprints of a new
-        # certificate and wait until the "valid_until_ts" of the previous key
-        # responses have passed before deploying it.
-        #
-        # You can calculate a fingerprint from a given TLS listener via:
-        # openssl s_client -connect $host:$port < /dev/null 2> /dev/null |
-        #   openssl x509 -outform DER | openssl sha256 -binary | base64 | tr -d '='
-        # or by checking matrix.org/federationtester/api/report?server_name=$host
-        #
-        tls_fingerprints: []
-        # tls_fingerprints: [{"sha256": "<base64_encoded_sha256_fingerprint>"}]
         """ % locals()
 
     def read_tls_certificate(self, cert_path):
@@ -132,7 +88,7 @@ class TlsConfig(Config):
         tls_private_key_path = config["tls_private_key_path"]
         tls_dh_params_path = config["tls_dh_params_path"]
 
-        if not self.path_exists(tls_private_key_path):
+        if not os.path.exists(tls_private_key_path):
             with open(tls_private_key_path, "w") as private_key_file:
                 tls_private_key = crypto.PKey()
                 tls_private_key.generate_key(crypto.TYPE_RSA, 2048)
@@ -147,7 +103,7 @@ class TlsConfig(Config):
                     crypto.FILETYPE_PEM, private_key_pem
                 )
 
-        if not self.path_exists(tls_certificate_path):
+        if not os.path.exists(tls_certificate_path):
             with open(tls_certificate_path, "w") as certificate_file:
                 cert = crypto.X509()
                 subject = cert.get_subject()
@@ -165,7 +121,7 @@ class TlsConfig(Config):
 
                 certificate_file.write(cert_pem)
 
-        if not self.path_exists(tls_dh_params_path):
+        if not os.path.exists(tls_dh_params_path):
             if GENERATE_DH_PARAMS:
                 subprocess.check_call([
                     "openssl", "dhparam",

synapse/config/user_directory.py

@@ -1,44 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2017 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from ._base import Config
-
-
-class UserDirectoryConfig(Config):
-    """User Directory Configuration
-    Configuration for the behaviour of the /user_directory API
-    """
-
-    def read_config(self, config):
-        self.user_directory_search_all_users = False
-        user_directory_config = config.get("user_directory", None)
-        if user_directory_config:
-            self.user_directory_search_all_users = (
-                user_directory_config.get("search_all_users", False)
-            )
-
-    def default_config(self, config_dir_path, server_name, **kwargs):
-        return """
-        # User Directory configuration
-        #
-        # 'search_all_users' defines whether to search all users visible to your HS
-        # when searching the user directory, rather than limiting to users visible
-        # in public rooms.  Defaults to false.  If you set it True, you'll have to run
-        # UPDATE user_directory_stream_pos SET stream_id = NULL;
-        # on your database to tell it to rebuild the user_directory search indexes.
-        #
-        #user_directory:
-        #   search_all_users: false
-        """

synapse/config/voip.py

@@ -19,11 +19,8 @@ class VoipConfig(Config):
 
     def read_config(self, config):
         self.turn_uris = config.get("turn_uris", [])
-        self.turn_shared_secret = config.get("turn_shared_secret")
-        self.turn_username = config.get("turn_username")
-        self.turn_password = config.get("turn_password")
+        self.turn_shared_secret = config["turn_shared_secret"]
         self.turn_user_lifetime = self.parse_duration(config["turn_user_lifetime"])
-        self.turn_allow_guests = config.get("turn_allow_guests", True)
 
     def default_config(self, **kwargs):
         return """\
@@ -35,18 +32,6 @@ class VoipConfig(Config):
         # The shared secret used to compute passwords for the TURN server
         turn_shared_secret: "YOUR_SHARED_SECRET"
 
-        # The Username and password if the TURN server needs them and
-        # does not use a token
-        #turn_username: "TURNSERVER_USERNAME"
-        #turn_password: "TURNSERVER_PASSWORD"
-
         # How long generated TURN credentials last
         turn_user_lifetime: "1h"
-
-        # Whether guests should be allowed to use the TURN server.
-        # This defaults to True, otherwise VoIP will be unreliable for guests.
-        # However, it does introduce a slight security risk as it allows users to
-        # connect to arbitrary endpoints without having first signed up for a
-        # valid account (e.g. by passing a CAPTCHA).
-        turn_allow_guests: True
         """

synapse/config/workers.py

@@ -23,37 +23,9 @@ class WorkerConfig(Config):
 
     def read_config(self, config):
         self.worker_app = config.get("worker_app")
-
-        # Canonicalise worker_app so that master always has None
-        if self.worker_app == "synapse.app.homeserver":
-            self.worker_app = None
-
         self.worker_listeners = config.get("worker_listeners")
         self.worker_daemonize = config.get("worker_daemonize")
         self.worker_pid_file = config.get("worker_pid_file")
         self.worker_log_file = config.get("worker_log_file")
         self.worker_log_config = config.get("worker_log_config")
-
-        # The host used to connect to the main synapse
-        self.worker_replication_host = config.get("worker_replication_host", None)
-
-        # The port on the main synapse for TCP replication
-        self.worker_replication_port = config.get("worker_replication_port", None)
-
-        # The port on the main synapse for HTTP replication endpoint
-        self.worker_replication_http_port = config.get("worker_replication_http_port")
-
-        self.worker_name = config.get("worker_name", self.worker_app)
-
-        self.worker_main_http_uri = config.get("worker_main_http_uri", None)
-        self.worker_cpu_affinity = config.get("worker_cpu_affinity")
-
-        if self.worker_listeners:
-            for listener in self.worker_listeners:
-                bind_address = listener.pop("bind_address", None)
-                bind_addresses = listener.setdefault("bind_addresses", [])
-
-                if bind_address:
-                    bind_addresses.append(bind_address)
-                elif not bind_addresses:
-                    bind_addresses.append('')
+        self.worker_replication_url = config.get("worker_replication_url")

synapse/crypto/context_factory.py

@@ -34,7 +34,7 @@ class ServerContextFactory(ssl.ContextFactory):
         try:
             _ecCurve = _OpenSSLECCurve(_defaultCurveName)
             _ecCurve.addECKeyToContext(context)
-        except Exception:
+        except:
             logger.exception("Failed to enable elliptic curve for TLS")
         context.set_options(SSL.OP_NO_SSLv2 | SSL.OP_NO_SSLv3)
         context.use_certificate_chain_file(config.tls_certificate_file)

synapse/crypto/event_signing.py

@@ -32,25 +32,18 @@ def check_event_content_hash(event, hash_algorithm=hashlib.sha256):
     """Check whether the hash for this PDU matches the contents"""
     name, expected_hash = compute_content_hash(event, hash_algorithm)
     logger.debug("Expecting hash: %s", encode_base64(expected_hash))
-
-    # some malformed events lack a 'hashes'. Protect against it being missing
-    # or a weird type by basically treating it the same as an unhashed event.
-    hashes = event.get("hashes")
-    if not isinstance(hashes, dict):
-        raise SynapseError(400, "Malformed 'hashes'", Codes.UNAUTHORIZED)
-
-    if name not in hashes:
+    if name not in event.hashes:
         raise SynapseError(
             400,
             "Algorithm %s not in hashes %s" % (
-                name, list(hashes),
+                name, list(event.hashes),
             ),
             Codes.UNAUTHORIZED,
         )
-    message_hash_base64 = hashes[name]
+    message_hash_base64 = event.hashes[name]
     try:
         message_hash_bytes = decode_base64(message_hash_base64)
-    except Exception:
+    except:
         raise SynapseError(
             400,
             "Invalid base64: %s" % (message_hash_base64,),

synapse/crypto/keyclient.py

@@ -13,11 +13,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from synapse.util import logcontext
+
 from twisted.web.http import HTTPClient
 from twisted.internet.protocol import Factory
 from twisted.internet import defer, reactor
 from synapse.http.endpoint import matrix_federation_endpoint
+from synapse.util.logcontext import (
+    preserve_context_over_fn, preserve_context_over_deferred
+)
 import simplejson as json
 import logging
 
@@ -40,10 +43,14 @@ def fetch_server_key(server_name, ssl_context_factory, path=KEY_API_V1):
 
     for i in range(5):
         try:
-            with logcontext.PreserveLoggingContext():
-                protocol = yield endpoint.connect(factory)
-                server_response, server_certificate = yield protocol.remote_key
+            protocol = yield preserve_context_over_fn(
+                endpoint.connect, factory
+            )
+            server_response, server_certificate = yield preserve_context_over_deferred(
+                protocol.remote_key
+            )
             defer.returnValue((server_response, server_certificate))
+            return
         except SynapseKeyClientError as e:
             logger.exception("Error getting key for %r" % (server_name,))
             if e.status.startswith("4"):
@@ -70,12 +77,10 @@ class SynapseKeyClientProtocol(HTTPClient):
     def __init__(self):
         self.remote_key = defer.Deferred()
         self.host = None
-        self._peer = None
 
     def connectionMade(self):
-        self._peer = self.transport.getPeer()
-        logger.debug("Connected to %s", self._peer)
-
+        self.host = self.transport.getHost()
+        logger.debug("Connected to %s", self.host)
         self.sendCommand(b"GET", self.path)
         if self.host:
             self.sendHeader(b"Host", self.host)
@@ -119,10 +124,7 @@ class SynapseKeyClientProtocol(HTTPClient):
         self.timer.cancel()
 
     def on_timeout(self):
-        logger.debug(
-            "Timeout waiting for response from %s: %s",
-            self.host, self._peer,
-        )
+        logger.debug("Timeout waiting for response from %s", self.host)
         self.errback(IOError("Timeout waiting for response"))
         self.transport.abortConnection()
 
@@ -131,5 +133,4 @@ class SynapseKeyClientFactory(Factory):
     def protocol(self):
         protocol = SynapseKeyClientProtocol()
         protocol.path = self.path
-        protocol.host = self.host
         return protocol

synapse/crypto/keyring.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2017 New Vector Ltd.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -16,12 +15,13 @@
 
 from synapse.crypto.keyclient import fetch_server_key
 from synapse.api.errors import SynapseError, Codes
-from synapse.util import unwrapFirstError, logcontext
+from synapse.util.retryutils import get_retry_limiter
+from synapse.util import unwrapFirstError
+from synapse.util.async import ObservableDeferred
 from synapse.util.logcontext import (
-    PreserveLoggingContext,
+    preserve_context_over_deferred, preserve_context_over_fn, PreserveLoggingContext,
     preserve_fn
 )
-from synapse.util.metrics import Measure
 
 from twisted.internet import defer
 
@@ -44,26 +44,7 @@ import logging
 logger = logging.getLogger(__name__)
 
 
-VerifyKeyRequest = namedtuple("VerifyRequest", (
-    "server_name", "key_ids", "json_object", "deferred"
-))
-"""
-A request for a verify key to verify a JSON object.
-
-Attributes:
-    server_name(str): The name of the server to verify against.
-    key_ids(set(str)): The set of key_ids to that could be used to verify the
-        JSON object
-    json_object(dict): The JSON object to verify.
-    deferred(twisted.internet.defer.Deferred):
-        A deferred (server_name, key_id, verify_key) tuple that resolves when
-        a verify key has been fetched. The deferreds' callbacks are run with no
-        logcontext.
-"""
-
-
-class KeyLookupError(ValueError):
-    pass
+KeyGroup = namedtuple("KeyGroup", ("server_name", "group_id", "key_ids"))
 
 
 class Keyring(object):
@@ -75,123 +56,138 @@ class Keyring(object):
         self.perspective_servers = self.config.perspectives
         self.hs = hs
 
-        # map from server name to Deferred. Has an entry for each server with
-        # an ongoing key download; the Deferred completes once the download
-        # completes.
-        #
-        # These are regular, logcontext-agnostic Deferreds.
         self.key_downloads = {}
 
     def verify_json_for_server(self, server_name, json_object):
-        return logcontext.make_deferred_yieldable(
-            self.verify_json_objects_for_server(
+        return self.verify_json_objects_for_server(
             [(server_name, json_object)]
         )[0]
-        )
 
     def verify_json_objects_for_server(self, server_and_json):
-        """Bulk verifies signatures of json objects, bulk fetching keys as
+        """Bulk verfies signatures of json objects, bulk fetching keys as
         necessary.
 
         Args:
             server_and_json (list): List of pairs of (server_name, json_object)
 
         Returns:
-            List<Deferred>: for each input pair, a deferred indicating success
-                or failure to verify each json object's signature for the given
-                server_name. The deferreds run their callbacks in the sentinel
-                logcontext.
+            list of deferreds indicating success or failure to verify each
+            json object's signature for the given server_name.
         """
-        verify_requests = []
+        group_id_to_json = {}
+        group_id_to_group = {}
+        group_ids = []
+
+        next_group_id = 0
+        deferreds = {}
 
         for server_name, json_object in server_and_json:
+            logger.debug("Verifying for %s", server_name)
+            group_id = next_group_id
+            next_group_id += 1
+            group_ids.append(group_id)
 
             key_ids = signature_ids(json_object, server_name)
             if not key_ids:
-                logger.warn("Request from %s: no supported signature keys",
-                            server_name)
-                deferred = defer.fail(SynapseError(
+                deferreds[group_id] = defer.fail(SynapseError(
                     400,
                     "Not signed with a supported algorithm",
                     Codes.UNAUTHORIZED,
                 ))
             else:
-                deferred = defer.Deferred()
+                deferreds[group_id] = defer.Deferred()
 
-            logger.debug("Verifying for %s with key_ids %s",
-                         server_name, key_ids)
+            group = KeyGroup(server_name, group_id, key_ids)
 
-            verify_request = VerifyKeyRequest(
-                server_name, key_ids, json_object, deferred
-            )
-
-            verify_requests.append(verify_request)
-
-        preserve_fn(self._start_key_lookups)(verify_requests)
-
-        # Pass those keys to handle_key_deferred so that the json object
-        # signatures can be verified
-        handle = preserve_fn(_handle_key_deferred)
-        return [
-            handle(rq) for rq in verify_requests
-        ]
+            group_id_to_group[group_id] = group
+            group_id_to_json[group_id] = json_object
 
         @defer.inlineCallbacks
-    def _start_key_lookups(self, verify_requests):
-        """Sets off the key fetches for each verify request
+        def handle_key_deferred(group, deferred):
+            server_name = group.server_name
+            try:
+                _, _, key_id, verify_key = yield deferred
+            except IOError as e:
+                logger.warn(
+                    "Got IOError when downloading keys for %s: %s %s",
+                    server_name, type(e).__name__, str(e.message),
+                )
+                raise SynapseError(
+                    502,
+                    "Error downloading keys for %s" % (server_name,),
+                    Codes.UNAUTHORIZED,
+                )
+            except Exception as e:
+                logger.exception(
+                    "Got Exception when downloading keys for %s: %s %s",
+                    server_name, type(e).__name__, str(e.message),
+                )
+                raise SynapseError(
+                    401,
+                    "No key for %s with id %s" % (server_name, key_ids),
+                    Codes.UNAUTHORIZED,
+                )
 
-        Once each fetch completes, verify_request.deferred will be resolved.
+            json_object = group_id_to_json[group.group_id]
 
-        Args:
-            verify_requests (List[VerifyKeyRequest]):
-        """
+            try:
+                verify_signed_json(json_object, server_name, verify_key)
+            except:
+                raise SynapseError(
+                    401,
+                    "Invalid signature for server %s with key %s:%s" % (
+                        server_name, verify_key.alg, verify_key.version
+                    ),
+                    Codes.UNAUTHORIZED,
+                )
 
-        # create a deferred for each server we're going to look up the keys
-        # for; we'll resolve them once we have completed our lookups.
-        # These will be passed into wait_for_previous_lookups to block
-        # any other lookups until we have finished.
-        # The deferreds are called with no logcontext.
         server_to_deferred = {
-            rq.server_name: defer.Deferred()
-            for rq in verify_requests
+            server_name: defer.Deferred()
+            for server_name, _ in server_and_json
         }
 
+        with PreserveLoggingContext():
+
             # We want to wait for any previous lookups to complete before
             # proceeding.
-        yield self.wait_for_previous_lookups(
-            [rq.server_name for rq in verify_requests],
+            wait_on_deferred = self.wait_for_previous_lookups(
+                [server_name for server_name, _ in server_and_json],
                 server_to_deferred,
             )
 
             # Actually start fetching keys.
-        self._get_server_verify_keys(verify_requests)
+            wait_on_deferred.addBoth(
+                lambda _: self.get_server_verify_keys(group_id_to_group, deferreds)
+            )
 
             # When we've finished fetching all the keys for a given server_name,
             # resolve the deferred passed to `wait_for_previous_lookups` so that
             # any lookups waiting will proceed.
-        #
-        # map from server name to a set of request ids
-        server_to_request_ids = {}
+            server_to_gids = {}
 
-        for verify_request in verify_requests:
-            server_name = verify_request.server_name
-            request_id = id(verify_request)
-            server_to_request_ids.setdefault(server_name, set()).add(request_id)
-
-        def remove_deferreds(res, verify_request):
-            server_name = verify_request.server_name
-            request_id = id(verify_request)
-            server_to_request_ids[server_name].discard(request_id)
-            if not server_to_request_ids[server_name]:
+            def remove_deferreds(res, server_name, group_id):
+                server_to_gids[server_name].discard(group_id)
+                if not server_to_gids[server_name]:
                     d = server_to_deferred.pop(server_name, None)
                     if d:
                         d.callback(None)
                 return res
 
-        for verify_request in verify_requests:
-            verify_request.deferred.addBoth(
-                remove_deferreds, verify_request,
+            for g_id, deferred in deferreds.items():
+                server_name = group_id_to_group[g_id].server_name
+                server_to_gids.setdefault(server_name, set()).add(g_id)
+                deferred.addBoth(remove_deferreds, server_name, g_id)
+
+        # Pass those keys to handle_key_deferred so that the json object
+        # signatures can be verified
+        return [
+            preserve_context_over_fn(
+                handle_key_deferred,
+                group_id_to_group[g_id],
+                deferreds[g_id],
             )
+            for g_id in group_ids
+        ]
 
     @defer.inlineCallbacks
     def wait_for_previous_lookups(self, server_names, server_to_deferred):
@@ -200,13 +196,7 @@ class Keyring(object):
         Args:
             server_names (list): list of server_names we want to lookup
             server_to_deferred (dict): server_name to deferred which gets
-                resolved once we've finished looking up keys for that server.
-                The Deferreds should be regular twisted ones which call their
-                callbacks with no logcontext.
-
-        Returns: a Deferred which resolves once all key lookups for the given
-            servers have completed. Follows the synapse rules of logcontext
-            preservation.
+                resolved once we've finished looking up keys for that server
         """
         while True:
             wait_on = [
@@ -220,23 +210,19 @@ class Keyring(object):
             else:
                 break
 
-        def rm(r, server_name_):
-            self.key_downloads.pop(server_name_, None)
+        for server_name, deferred in server_to_deferred.items():
+            d = ObservableDeferred(preserve_context_over_deferred(deferred))
+            self.key_downloads[server_name] = d
+
+            def rm(r, server_name):
+                self.key_downloads.pop(server_name, None)
                 return r
 
-        for server_name, deferred in server_to_deferred.items():
-            self.key_downloads[server_name] = deferred
-            deferred.addBoth(rm, server_name)
+            d.addBoth(rm, server_name)
 
-    def _get_server_verify_keys(self, verify_requests):
-        """Tries to find at least one key for each verify request
-
-        For each verify_request, verify_request.deferred is called back with
-        params (server_name, key_id, VerifyKey) if a key is found, or errbacked
-        with a SynapseError if none of the keys are found.
-
-        Args:
-            verify_requests (list[VerifyKeyRequest]): list of verify requests
+    def get_server_verify_keys(self, group_id_to_group, group_id_to_deferred):
+        """Takes a dict of KeyGroups and tries to find at least one key for
+        each group.
         """
 
         # These are functions that produce keys given a list of key ids
@@ -248,94 +234,76 @@ class Keyring(object):
 
         @defer.inlineCallbacks
         def do_iterations():
-            with Measure(self.clock, "get_server_verify_keys"):
-                # dict[str, dict[str, VerifyKey]]: results so far.
-                # map server_name -> key_id -> VerifyKey
             merged_results = {}
 
-                # dict[str, set(str)]: keys to fetch for each server
             missing_keys = {}
-                for verify_request in verify_requests:
-                    missing_keys.setdefault(verify_request.server_name, set()).update(
-                        verify_request.key_ids
+            for group in group_id_to_group.values():
+                missing_keys.setdefault(group.server_name, set()).update(
+                    group.key_ids
                 )
 
             for fn in key_fetch_fns:
                 results = yield fn(missing_keys.items())
                 merged_results.update(results)
 
-                    # We now need to figure out which verify requests we have keys
-                    # for and which we don't
-                    missing_keys = {}
-                    requests_missing_keys = []
-                    for verify_request in verify_requests:
-                        server_name = verify_request.server_name
-                        result_keys = merged_results[server_name]
-
-                        if verify_request.deferred.called:
-                            # We've already called this deferred, which probably
-                            # means that we've already found a key for it.
-                            continue
-
-                        for key_id in verify_request.key_ids:
-                            if key_id in result_keys:
+                # We now need to figure out which groups we have keys for
+                # and which we don't
+                missing_groups = {}
+                for group in group_id_to_group.values():
+                    for key_id in group.key_ids:
+                        if key_id in merged_results[group.server_name]:
                             with PreserveLoggingContext():
-                                    verify_request.deferred.callback((
-                                        server_name,
+                                group_id_to_deferred[group.group_id].callback((
+                                    group.group_id,
+                                    group.server_name,
                                     key_id,
-                                        result_keys[key_id],
+                                    merged_results[group.server_name][key_id],
                                 ))
                             break
                     else:
-                            # The else block is only reached if the loop above
-                            # doesn't break.
-                            missing_keys.setdefault(server_name, set()).update(
-                                verify_request.key_ids
-                            )
-                            requests_missing_keys.append(verify_request)
+                        missing_groups.setdefault(
+                            group.server_name, []
+                        ).append(group)
 
-                    if not missing_keys:
+                if not missing_groups:
                     break
 
-                with PreserveLoggingContext():
-                    for verify_request in requests_missing_keys:
-                        verify_request.deferred.errback(SynapseError(
+                missing_keys = {
+                    server_name: set(
+                        key_id for group in groups for key_id in group.key_ids
+                    )
+                    for server_name, groups in missing_groups.items()
+                }
+
+            for group in missing_groups.values():
+                group_id_to_deferred[group.group_id].errback(SynapseError(
                     401,
                     "No key for %s with id %s" % (
-                                verify_request.server_name, verify_request.key_ids,
+                        group.server_name, group.key_ids,
                     ),
                     Codes.UNAUTHORIZED,
                 ))
 
         def on_err(err):
-            with PreserveLoggingContext():
-                for verify_request in verify_requests:
-                    if not verify_request.deferred.called:
-                        verify_request.deferred.errback(err)
+            for deferred in group_id_to_deferred.values():
+                if not deferred.called:
+                    deferred.errback(err)
 
-        preserve_fn(do_iterations)().addErrback(on_err)
+        do_iterations().addErrback(on_err)
+
+        return group_id_to_deferred
 
     @defer.inlineCallbacks
     def get_keys_from_store(self, server_name_and_key_ids):
-        """
-
-        Args:
-            server_name_and_key_ids (list[(str, iterable[str])]):
-                list of (server_name, iterable[key_id]) tuples to fetch keys for
-
-        Returns:
-            Deferred: resolves to dict[str, dict[str, VerifyKey]]: map from
-                server_name -> key_id -> VerifyKey
-        """
-        res = yield logcontext.make_deferred_yieldable(defer.gatherResults(
+        res = yield defer.gatherResults(
             [
-                preserve_fn(self.store.get_server_verify_keys)(
+                self.store.get_server_verify_keys(
                     server_name, key_ids
                 ).addCallback(lambda ks, server: (server, ks), server_name)
                 for server_name, key_ids in server_name_and_key_ids
             ],
             consumeErrors=True,
-        ).addErrback(unwrapFirstError))
+        ).addErrback(unwrapFirstError)
 
         defer.returnValue(dict(res))
 
@@ -356,13 +324,13 @@ class Keyring(object):
                 )
                 defer.returnValue({})
 
-        results = yield logcontext.make_deferred_yieldable(defer.gatherResults(
+        results = yield defer.gatherResults(
             [
-                preserve_fn(get_key)(p_name, p_keys)
+                get_key(p_name, p_keys)
                 for p_name, p_keys in self.perspective_servers.items()
             ],
             consumeErrors=True,
-        ).addErrback(unwrapFirstError))
+        ).addErrback(unwrapFirstError)
 
         union_of_keys = {}
         for result in results:
@@ -375,6 +343,12 @@ class Keyring(object):
     def get_keys_from_server(self, server_name_and_key_ids):
         @defer.inlineCallbacks
         def get_key(server_name, key_ids):
+            limiter = yield get_retry_limiter(
+                server_name,
+                self.clock,
+                self.store,
+            )
+            with limiter:
                 keys = None
                 try:
                     keys = yield self.get_server_verify_key_v2_direct(
@@ -382,7 +356,7 @@ class Keyring(object):
                     )
                 except Exception as e:
                     logger.info(
-                    "Unable to get key %r for %r directly: %s %s",
+                        "Unable to getting key %r for %r directly: %s %s",
                         key_ids, server_name,
                         type(e).__name__, str(e.message),
                     )
@@ -396,13 +370,13 @@ class Keyring(object):
 
             defer.returnValue(keys)
 
-        results = yield logcontext.make_deferred_yieldable(defer.gatherResults(
+        results = yield defer.gatherResults(
             [
-                preserve_fn(get_key)(server_name, key_ids)
+                get_key(server_name, key_ids)
                 for server_name, key_ids in server_name_and_key_ids
             ],
             consumeErrors=True,
-        ).addErrback(unwrapFirstError))
+        ).addErrback(unwrapFirstError)
 
         merged = {}
         for result in results:
@@ -444,7 +418,7 @@ class Keyring(object):
         for response in responses:
             if (u"signatures" not in response
                     or perspective_name not in response[u"signatures"]):
-                raise KeyLookupError(
+                raise ValueError(
                     "Key response not signed by perspective server"
                     " %r" % (perspective_name,)
                 )
@@ -467,21 +441,21 @@ class Keyring(object):
                     list(response[u"signatures"][perspective_name]),
                     list(perspective_keys)
                 )
-                raise KeyLookupError(
+                raise ValueError(
                     "Response not signed with a known key for perspective"
                     " server %r" % (perspective_name,)
                 )
 
             processed_response = yield self.process_v2_response(
-                perspective_name, response, only_from_server=False
+                perspective_name, response
             )
 
             for server_name, response_keys in processed_response.items():
                 keys.setdefault(server_name, {}).update(response_keys)
 
-        yield logcontext.make_deferred_yieldable(defer.gatherResults(
+        yield defer.gatherResults(
             [
-                preserve_fn(self.store_keys)(
+                self.store_keys(
                     server_name=server_name,
                     from_server=perspective_name,
                     verify_keys=response_keys,
@@ -489,7 +463,7 @@ class Keyring(object):
                 for server_name, response_keys in keys.items()
             ],
             consumeErrors=True
-        ).addErrback(unwrapFirstError))
+        ).addErrback(unwrapFirstError)
 
         defer.returnValue(keys)
 
@@ -510,10 +484,10 @@ class Keyring(object):
 
             if (u"signatures" not in response
                     or server_name not in response[u"signatures"]):
-                raise KeyLookupError("Key response not signed by remote server")
+                raise ValueError("Key response not signed by remote server")
 
             if "tls_fingerprints" not in response:
-                raise KeyLookupError("Key response missing TLS fingerprints")
+                raise ValueError("Key response missing TLS fingerprints")
 
             certificate_bytes = crypto.dump_certificate(
                 crypto.FILETYPE_ASN1, tls_certificate
@@ -527,7 +501,7 @@ class Keyring(object):
                     response_sha256_fingerprints.add(fingerprint[u"sha256"])
 
             if sha256_fingerprint_b64 not in response_sha256_fingerprints:
-                raise KeyLookupError("TLS certificate not allowed by fingerprints")
+                raise ValueError("TLS certificate not allowed by fingerprints")
 
             response_keys = yield self.process_v2_response(
                 from_server=server_name,
@@ -537,7 +511,7 @@ class Keyring(object):
 
             keys.update(response_keys)
 
-        yield logcontext.make_deferred_yieldable(defer.gatherResults(
+        yield defer.gatherResults(
             [
                 preserve_fn(self.store_keys)(
                     server_name=key_server_name,
@@ -547,13 +521,13 @@ class Keyring(object):
                 for key_server_name, verify_keys in keys.items()
             ],
             consumeErrors=True
-        ).addErrback(unwrapFirstError))
+        ).addErrback(unwrapFirstError)
 
         defer.returnValue(keys)
 
     @defer.inlineCallbacks
     def process_v2_response(self, from_server, response_json,
-                            requested_ids=[], only_from_server=True):
+                            requested_ids=[]):
         time_now_ms = self.clock.time_msec()
         response_keys = {}
         verify_keys = {}
@@ -577,16 +551,9 @@ class Keyring(object):
 
         results = {}
         server_name = response_json["server_name"]
-        if only_from_server:
-            if server_name != from_server:
-                raise KeyLookupError(
-                    "Expected a response for server %r not %r" % (
-                        from_server, server_name
-                    )
-                )
         for key_id in response_json["signatures"].get(server_name, {}):
             if key_id not in response_json["verify_keys"]:
-                raise KeyLookupError(
+                raise ValueError(
                     "Key response must include verification keys for all"
                     " signatures"
                 )
@@ -613,7 +580,7 @@ class Keyring(object):
         response_keys.update(verify_keys)
         response_keys.update(old_verify_keys)
 
-        yield logcontext.make_deferred_yieldable(defer.gatherResults(
+        yield defer.gatherResults(
             [
                 preserve_fn(self.store.store_server_keys_json)(
                     server_name=server_name,
@@ -626,7 +593,7 @@ class Keyring(object):
                 for key_id in updated_key_ids
             ],
             consumeErrors=True,
-        ).addErrback(unwrapFirstError))
+        ).addErrback(unwrapFirstError)
 
         results[server_name] = response_keys
 
@@ -654,15 +621,15 @@ class Keyring(object):
 
         if ("signatures" not in response
                 or server_name not in response["signatures"]):
-            raise KeyLookupError("Key response not signed by remote server")
+            raise ValueError("Key response not signed by remote server")
 
         if "tls_certificate" not in response:
-            raise KeyLookupError("Key response missing TLS certificate")
+            raise ValueError("Key response missing TLS certificate")
 
         tls_certificate_b64 = response["tls_certificate"]
 
         if encode_base64(x509_certificate_bytes) != tls_certificate_b64:
-            raise KeyLookupError("TLS certificate doesn't match")
+            raise ValueError("TLS certificate doesn't match")
 
         # Cache the result in the datastore.
 
@@ -678,7 +645,7 @@ class Keyring(object):
 
         for key_id in response["signatures"][server_name]:
             if key_id not in response["verify_keys"]:
-                raise KeyLookupError(
+                raise ValueError(
                     "Key response must include verification keys for all"
                     " signatures"
                 )
@@ -704,6 +671,7 @@ class Keyring(object):
 
         defer.returnValue(verify_keys)
 
+    @defer.inlineCallbacks
     def store_keys(self, server_name, from_server, verify_keys):
         """Store a collection of verify keys for a given server
         Args:
@@ -714,7 +682,7 @@ class Keyring(object):
             A deferred that completes when the keys are stored.
         """
         # TODO(markjh): Store whether the keys have expired.
-        return logcontext.make_deferred_yieldable(defer.gatherResults(
+        yield defer.gatherResults(
             [
                 preserve_fn(self.store.store_server_verify_key)(
                     server_name, server_name, key.time_added, key
@@ -722,48 +690,4 @@ class Keyring(object):
                 for key_id, key in verify_keys.items()
             ],
             consumeErrors=True,
-        ).addErrback(unwrapFirstError))
-
-
-@defer.inlineCallbacks
-def _handle_key_deferred(verify_request):
-    server_name = verify_request.server_name
-    try:
-        with PreserveLoggingContext():
-            _, key_id, verify_key = yield verify_request.deferred
-    except IOError as e:
-        logger.warn(
-            "Got IOError when downloading keys for %s: %s %s",
-            server_name, type(e).__name__, str(e.message),
-        )
-        raise SynapseError(
-            502,
-            "Error downloading keys for %s" % (server_name,),
-            Codes.UNAUTHORIZED,
-        )
-    except Exception as e:
-        logger.exception(
-            "Got Exception when downloading keys for %s: %s %s",
-            server_name, type(e).__name__, str(e.message),
-        )
-        raise SynapseError(
-            401,
-            "No key for %s with id %s" % (server_name, verify_request.key_ids),
-            Codes.UNAUTHORIZED,
-        )
-
-    json_object = verify_request.json_object
-
-    logger.debug("Got key %s %s:%s for server %s, verifying" % (
-        key_id, verify_key.alg, verify_key.version, server_name,
-    ))
-    try:
-        verify_signed_json(json_object, server_name, verify_key)
-    except Exception:
-        raise SynapseError(
-            401,
-            "Invalid signature for server %s with key %s:%s" % (
-                server_name, verify_key.alg, verify_key.version
-            ),
-            Codes.UNAUTHORIZED,
-        )
+        ).addErrback(unwrapFirstError)

synapse/event_auth.py

@@ -1,678 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2014 - 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import logging
-
-from canonicaljson import encode_canonical_json
-from signedjson.key import decode_verify_key_bytes
-from signedjson.sign import verify_signed_json, SignatureVerifyException
-from unpaddedbase64 import decode_base64
-
-from synapse.api.constants import EventTypes, Membership, JoinRules
-from synapse.api.errors import AuthError, SynapseError, EventSizeError
-from synapse.types import UserID, get_domain_from_id
-
-logger = logging.getLogger(__name__)
-
-
-def check(event, auth_events, do_sig_check=True, do_size_check=True):
-    """ Checks if this event is correctly authed.
-
-    Args:
-        event: the event being checked.
-        auth_events (dict: event-key -> event): the existing room state.
-
-
-    Returns:
-        True if the auth checks pass.
-    """
-    if do_size_check:
-        _check_size_limits(event)
-
-    if not hasattr(event, "room_id"):
-        raise AuthError(500, "Event has no room_id: %s" % event)
-
-    if do_sig_check:
-        sender_domain = get_domain_from_id(event.sender)
-        event_id_domain = get_domain_from_id(event.event_id)
-
-        is_invite_via_3pid = (
-            event.type == EventTypes.Member
-            and event.membership == Membership.INVITE
-            and "third_party_invite" in event.content
-        )
-
-        # Check the sender's domain has signed the event
-        if not event.signatures.get(sender_domain):
-            # We allow invites via 3pid to have a sender from a different
-            # HS, as the sender must match the sender of the original
-            # 3pid invite. This is checked further down with the
-            # other dedicated membership checks.
-            if not is_invite_via_3pid:
-                raise AuthError(403, "Event not signed by sender's server")
-
-        # Check the event_id's domain has signed the event
-        if not event.signatures.get(event_id_domain):
-            raise AuthError(403, "Event not signed by sending server")
-
-    if auth_events is None:
-        # Oh, we don't know what the state of the room was, so we
-        # are trusting that this is allowed (at least for now)
-        logger.warn("Trusting event: %s", event.event_id)
-        return True
-
-    if event.type == EventTypes.Create:
-        room_id_domain = get_domain_from_id(event.room_id)
-        if room_id_domain != sender_domain:
-            raise AuthError(
-                403,
-                "Creation event's room_id domain does not match sender's"
-            )
-        # FIXME
-        return True
-
-    creation_event = auth_events.get((EventTypes.Create, ""), None)
-
-    if not creation_event:
-        raise SynapseError(
-            403,
-            "Room %r does not exist" % (event.room_id,)
-        )
-
-    creating_domain = get_domain_from_id(event.room_id)
-    originating_domain = get_domain_from_id(event.sender)
-    if creating_domain != originating_domain:
-        if not _can_federate(event, auth_events):
-            raise AuthError(
-                403,
-                "This room has been marked as unfederatable."
-            )
-
-    # FIXME: Temp hack
-    if event.type == EventTypes.Aliases:
-        if not event.is_state():
-            raise AuthError(
-                403,
-                "Alias event must be a state event",
-            )
-        if not event.state_key:
-            raise AuthError(
-                403,
-                "Alias event must have non-empty state_key"
-            )
-        sender_domain = get_domain_from_id(event.sender)
-        if event.state_key != sender_domain:
-            raise AuthError(
-                403,
-                "Alias event's state_key does not match sender's domain"
-            )
-        return True
-
-    if logger.isEnabledFor(logging.DEBUG):
-        logger.debug(
-            "Auth events: %s",
-            [a.event_id for a in auth_events.values()]
-        )
-
-    if event.type == EventTypes.Member:
-        allowed = _is_membership_change_allowed(
-            event, auth_events
-        )
-        if allowed:
-            logger.debug("Allowing! %s", event)
-        else:
-            logger.debug("Denying! %s", event)
-        return allowed
-
-    _check_event_sender_in_room(event, auth_events)
-
-    # Special case to allow m.room.third_party_invite events wherever
-    # a user is allowed to issue invites.  Fixes
-    # https://github.com/vector-im/vector-web/issues/1208 hopefully
-    if event.type == EventTypes.ThirdPartyInvite:
-        user_level = get_user_power_level(event.user_id, auth_events)
-        invite_level = _get_named_level(auth_events, "invite", 0)
-
-        if user_level < invite_level:
-            raise AuthError(
-                403, (
-                    "You cannot issue a third party invite for %s." %
-                    (event.content.display_name,)
-                )
-            )
-        else:
-            return True
-
-    _can_send_event(event, auth_events)
-
-    if event.type == EventTypes.PowerLevels:
-        _check_power_levels(event, auth_events)
-
-    if event.type == EventTypes.Redaction:
-        check_redaction(event, auth_events)
-
-    logger.debug("Allowing! %s", event)
-
-
-def _check_size_limits(event):
-    def too_big(field):
-        raise EventSizeError("%s too large" % (field,))
-
-    if len(event.user_id) > 255:
-        too_big("user_id")
-    if len(event.room_id) > 255:
-        too_big("room_id")
-    if event.is_state() and len(event.state_key) > 255:
-        too_big("state_key")
-    if len(event.type) > 255:
-        too_big("type")
-    if len(event.event_id) > 255:
-        too_big("event_id")
-    if len(encode_canonical_json(event.get_pdu_json())) > 65536:
-        too_big("event")
-
-
-def _can_federate(event, auth_events):
-    creation_event = auth_events.get((EventTypes.Create, ""))
-
-    return creation_event.content.get("m.federate", True) is True
-
-
-def _is_membership_change_allowed(event, auth_events):
-    membership = event.content["membership"]
-
-    # Check if this is the room creator joining:
-    if len(event.prev_events) == 1 and Membership.JOIN == membership:
-        # Get room creation event:
-        key = (EventTypes.Create, "", )
-        create = auth_events.get(key)
-        if create and event.prev_events[0][0] == create.event_id:
-            if create.content["creator"] == event.state_key:
-                return True
-
-    target_user_id = event.state_key
-
-    creating_domain = get_domain_from_id(event.room_id)
-    target_domain = get_domain_from_id(target_user_id)
-    if creating_domain != target_domain:
-        if not _can_federate(event, auth_events):
-            raise AuthError(
-                403,
-                "This room has been marked as unfederatable."
-            )
-
-    # get info about the caller
-    key = (EventTypes.Member, event.user_id, )
-    caller = auth_events.get(key)
-
-    caller_in_room = caller and caller.membership == Membership.JOIN
-    caller_invited = caller and caller.membership == Membership.INVITE
-
-    # get info about the target
-    key = (EventTypes.Member, target_user_id, )
-    target = auth_events.get(key)
-
-    target_in_room = target and target.membership == Membership.JOIN
-    target_banned = target and target.membership == Membership.BAN
-
-    key = (EventTypes.JoinRules, "", )
-    join_rule_event = auth_events.get(key)
-    if join_rule_event:
-        join_rule = join_rule_event.content.get(
-            "join_rule", JoinRules.INVITE
-        )
-    else:
-        join_rule = JoinRules.INVITE
-
-    user_level = get_user_power_level(event.user_id, auth_events)
-    target_level = get_user_power_level(
-        target_user_id, auth_events
-    )
-
-    # FIXME (erikj): What should we do here as the default?
-    ban_level = _get_named_level(auth_events, "ban", 50)
-
-    logger.debug(
-        "_is_membership_change_allowed: %s",
-        {
-            "caller_in_room": caller_in_room,
-            "caller_invited": caller_invited,
-            "target_banned": target_banned,
-            "target_in_room": target_in_room,
-            "membership": membership,
-            "join_rule": join_rule,
-            "target_user_id": target_user_id,
-            "event.user_id": event.user_id,
-        }
-    )
-
-    if Membership.INVITE == membership and "third_party_invite" in event.content:
-        if not _verify_third_party_invite(event, auth_events):
-            raise AuthError(403, "You are not invited to this room.")
-        if target_banned:
-            raise AuthError(
-                403, "%s is banned from the room" % (target_user_id,)
-            )
-        return True
-
-    if Membership.JOIN != membership:
-        if (caller_invited
-                and Membership.LEAVE == membership
-                and target_user_id == event.user_id):
-            return True
-
-        if not caller_in_room:  # caller isn't joined
-            raise AuthError(
-                403,
-                "%s not in room %s." % (event.user_id, event.room_id,)
-            )
-
-    if Membership.INVITE == membership:
-        # TODO (erikj): We should probably handle this more intelligently
-        # PRIVATE join rules.
-
-        # Invites are valid iff caller is in the room and target isn't.
-        if target_banned:
-            raise AuthError(
-                403, "%s is banned from the room" % (target_user_id,)
-            )
-        elif target_in_room:  # the target is already in the room.
-            raise AuthError(403, "%s is already in the room." %
-                                 target_user_id)
-        else:
-            invite_level = _get_named_level(auth_events, "invite", 0)
-
-            if user_level < invite_level:
-                raise AuthError(
-                    403, "You cannot invite user %s." % target_user_id
-                )
-    elif Membership.JOIN == membership:
-        # Joins are valid iff caller == target and they were:
-        # invited: They are accepting the invitation
-        # joined: It's a NOOP
-        if event.user_id != target_user_id:
-            raise AuthError(403, "Cannot force another user to join.")
-        elif target_banned:
-            raise AuthError(403, "You are banned from this room")
-        elif join_rule == JoinRules.PUBLIC:
-            pass
-        elif join_rule == JoinRules.INVITE:
-            if not caller_in_room and not caller_invited:
-                raise AuthError(403, "You are not invited to this room.")
-        else:
-            # TODO (erikj): may_join list
-            # TODO (erikj): private rooms
-            raise AuthError(403, "You are not allowed to join this room")
-    elif Membership.LEAVE == membership:
-        # TODO (erikj): Implement kicks.
-        if target_banned and user_level < ban_level:
-            raise AuthError(
-                403, "You cannot unban user %s." % (target_user_id,)
-            )
-        elif target_user_id != event.user_id:
-            kick_level = _get_named_level(auth_events, "kick", 50)
-
-            if user_level < kick_level or user_level <= target_level:
-                raise AuthError(
-                    403, "You cannot kick user %s." % target_user_id
-                )
-    elif Membership.BAN == membership:
-        if user_level < ban_level or user_level <= target_level:
-            raise AuthError(403, "You don't have permission to ban")
-    else:
-        raise AuthError(500, "Unknown membership %s" % membership)
-
-    return True
-
-
-def _check_event_sender_in_room(event, auth_events):
-    key = (EventTypes.Member, event.user_id, )
-    member_event = auth_events.get(key)
-
-    return _check_joined_room(
-        member_event,
-        event.user_id,
-        event.room_id
-    )
-
-
-def _check_joined_room(member, user_id, room_id):
-    if not member or member.membership != Membership.JOIN:
-        raise AuthError(403, "User %s not in room %s (%s)" % (
-            user_id, room_id, repr(member)
-        ))
-
-
-def get_send_level(etype, state_key, auth_events):
-    key = (EventTypes.PowerLevels, "", )
-    send_level_event = auth_events.get(key)
-    send_level = None
-    if send_level_event:
-        send_level = send_level_event.content.get("events", {}).get(
-            etype
-        )
-        if send_level is None:
-            if state_key is not None:
-                send_level = send_level_event.content.get(
-                    "state_default", 50
-                )
-            else:
-                send_level = send_level_event.content.get(
-                    "events_default", 0
-                )
-
-    if send_level:
-        send_level = int(send_level)
-    else:
-        send_level = 0
-
-    return send_level
-
-
-def _can_send_event(event, auth_events):
-    send_level = get_send_level(
-        event.type, event.get("state_key", None), auth_events
-    )
-    user_level = get_user_power_level(event.user_id, auth_events)
-
-    if user_level < send_level:
-        raise AuthError(
-            403,
-            "You don't have permission to post that to the room. " +
-            "user_level (%d) < send_level (%d)" % (user_level, send_level)
-        )
-
-    # Check state_key
-    if hasattr(event, "state_key"):
-        if event.state_key.startswith("@"):
-            if event.state_key != event.user_id:
-                raise AuthError(
-                    403,
-                    "You are not allowed to set others state"
-                )
-
-    return True
-
-
-def check_redaction(event, auth_events):
-    """Check whether the event sender is allowed to redact the target event.
-
-    Returns:
-        True if the the sender is allowed to redact the target event if the
-        target event was created by them.
-        False if the sender is allowed to redact the target event with no
-        further checks.
-
-    Raises:
-        AuthError if the event sender is definitely not allowed to redact
-        the target event.
-    """
-    user_level = get_user_power_level(event.user_id, auth_events)
-
-    redact_level = _get_named_level(auth_events, "redact", 50)
-
-    if user_level >= redact_level:
-        return False
-
-    redacter_domain = get_domain_from_id(event.event_id)
-    redactee_domain = get_domain_from_id(event.redacts)
-    if redacter_domain == redactee_domain:
-        return True
-
-    raise AuthError(
-        403,
-        "You don't have permission to redact events"
-    )
-
-
-def _check_power_levels(event, auth_events):
-    user_list = event.content.get("users", {})
-    # Validate users
-    for k, v in user_list.items():
-        try:
-            UserID.from_string(k)
-        except Exception:
-            raise SynapseError(400, "Not a valid user_id: %s" % (k,))
-
-        try:
-            int(v)
-        except Exception:
-            raise SynapseError(400, "Not a valid power level: %s" % (v,))
-
-    key = (event.type, event.state_key, )
-    current_state = auth_events.get(key)
-
-    if not current_state:
-        return
-
-    user_level = get_user_power_level(event.user_id, auth_events)
-
-    # Check other levels:
-    levels_to_check = [
-        ("users_default", None),
-        ("events_default", None),
-        ("state_default", None),
-        ("ban", None),
-        ("redact", None),
-        ("kick", None),
-        ("invite", None),
-    ]
-
-    old_list = current_state.content.get("users", {})
-    for user in set(old_list.keys() + user_list.keys()):
-        levels_to_check.append(
-            (user, "users")
-        )
-
-    old_list = current_state.content.get("events", {})
-    new_list = event.content.get("events", {})
-    for ev_id in set(old_list.keys() + new_list.keys()):
-        levels_to_check.append(
-            (ev_id, "events")
-        )
-
-    old_state = current_state.content
-    new_state = event.content
-
-    for level_to_check, dir in levels_to_check:
-        old_loc = old_state
-        new_loc = new_state
-        if dir:
-            old_loc = old_loc.get(dir, {})
-            new_loc = new_loc.get(dir, {})
-
-        if level_to_check in old_loc:
-            old_level = int(old_loc[level_to_check])
-        else:
-            old_level = None
-
-        if level_to_check in new_loc:
-            new_level = int(new_loc[level_to_check])
-        else:
-            new_level = None
-
-        if new_level is not None and old_level is not None:
-            if new_level == old_level:
-                continue
-
-        if dir == "users" and level_to_check != event.user_id:
-            if old_level == user_level:
-                raise AuthError(
-                    403,
-                    "You don't have permission to remove ops level equal "
-                    "to your own"
-                )
-
-        if old_level > user_level or new_level > user_level:
-            raise AuthError(
-                403,
-                "You don't have permission to add ops level greater "
-                "than your own"
-            )
-
-
-def _get_power_level_event(auth_events):
-    key = (EventTypes.PowerLevels, "", )
-    return auth_events.get(key)
-
-
-def get_user_power_level(user_id, auth_events):
-    power_level_event = _get_power_level_event(auth_events)
-
-    if power_level_event:
-        level = power_level_event.content.get("users", {}).get(user_id)
-        if not level:
-            level = power_level_event.content.get("users_default", 0)
-
-        if level is None:
-            return 0
-        else:
-            return int(level)
-    else:
-        key = (EventTypes.Create, "", )
-        create_event = auth_events.get(key)
-        if (create_event is not None and
-                create_event.content["creator"] == user_id):
-            return 100
-        else:
-            return 0
-
-
-def _get_named_level(auth_events, name, default):
-    power_level_event = _get_power_level_event(auth_events)
-
-    if not power_level_event:
-        return default
-
-    level = power_level_event.content.get(name, None)
-    if level is not None:
-        return int(level)
-    else:
-        return default
-
-
-def _verify_third_party_invite(event, auth_events):
-    """
-    Validates that the invite event is authorized by a previous third-party invite.
-
-    Checks that the public key, and keyserver, match those in the third party invite,
-    and that the invite event has a signature issued using that public key.
-
-    Args:
-        event: The m.room.member join event being validated.
-        auth_events: All relevant previous context events which may be used
-            for authorization decisions.
-
-    Return:
-        True if the event fulfills the expectations of a previous third party
-        invite event.
-    """
-    if "third_party_invite" not in event.content:
-        return False
-    if "signed" not in event.content["third_party_invite"]:
-        return False
-    signed = event.content["third_party_invite"]["signed"]
-    for key in {"mxid", "token"}:
-        if key not in signed:
-            return False
-
-    token = signed["token"]
-
-    invite_event = auth_events.get(
-        (EventTypes.ThirdPartyInvite, token,)
-    )
-    if not invite_event:
-        return False
-
-    if invite_event.sender != event.sender:
-        return False
-
-    if event.user_id != invite_event.user_id:
-        return False
-
-    if signed["mxid"] != event.state_key:
-        return False
-    if signed["token"] != token:
-        return False
-
-    for public_key_object in get_public_keys(invite_event):
-        public_key = public_key_object["public_key"]
-        try:
-            for server, signature_block in signed["signatures"].items():
-                for key_name, encoded_signature in signature_block.items():
-                    if not key_name.startswith("ed25519:"):
-                        continue
-                    verify_key = decode_verify_key_bytes(
-                        key_name,
-                        decode_base64(public_key)
-                    )
-                    verify_signed_json(signed, server, verify_key)
-
-                    # We got the public key from the invite, so we know that the
-                    # correct server signed the signed bundle.
-                    # The caller is responsible for checking that the signing
-                    # server has not revoked that public key.
-                    return True
-        except (KeyError, SignatureVerifyException,):
-            continue
-    return False
-
-
-def get_public_keys(invite_event):
-    public_keys = []
-    if "public_key" in invite_event.content:
-        o = {
-            "public_key": invite_event.content["public_key"],
-        }
-        if "key_validity_url" in invite_event.content:
-            o["key_validity_url"] = invite_event.content["key_validity_url"]
-        public_keys.append(o)
-    public_keys.extend(invite_event.content.get("public_keys", []))
-    return public_keys
-
-
-def auth_types_for_event(event):
-    """Given an event, return a list of (EventType, StateKey) that may be
-    needed to auth the event. The returned list may be a superset of what
-    would actually be required depending on the full state of the room.
-
-    Used to limit the number of events to fetch from the database to
-    actually auth the event.
-    """
-    if event.type == EventTypes.Create:
-        return []
-
-    auth_types = []
-
-    auth_types.append((EventTypes.PowerLevels, "", ))
-    auth_types.append((EventTypes.Member, event.user_id, ))
-    auth_types.append((EventTypes.Create, "", ))
-
-    if event.type == EventTypes.Member:
-        membership = event.content["membership"]
-        if membership in [Membership.JOIN, Membership.INVITE]:
-            auth_types.append((EventTypes.JoinRules, "", ))
-
-        auth_types.append((EventTypes.Member, event.state_key, ))
-
-        if membership == Membership.INVITE:
-            if "third_party_invite" in event.content:
-                key = (
-                    EventTypes.ThirdPartyInvite,
-                    event.content["third_party_invite"]["signed"]["token"]
-                )
-                auth_types.append(key)
-
-    return auth_types

synapse/events/__init__.py

@@ -36,15 +36,6 @@ class _EventInternalMetadata(object):
     def is_invite_from_remote(self):
         return getattr(self, "invite_from_remote", False)
 
-    def get_send_on_behalf_of(self):
-        """Whether this server should send the event on behalf of another server.
-        This is used by the federation "send_join" API to forward the initial join
-        event for a server in the room.
-
-        returns a str with the name of the server this event is sent on behalf of.
-        """
-        return getattr(self, "send_on_behalf_of", None)
-
 
 def _event_dict_property(key):
     def getter(self):
@@ -79,6 +70,7 @@ class EventBase(object):
     auth_events = _event_dict_property("auth_events")
     depth = _event_dict_property("depth")
     content = _event_dict_property("content")
+    event_id = _event_dict_property("event_id")
     hashes = _event_dict_property("hashes")
     origin = _event_dict_property("origin")
     origin_server_ts = _event_dict_property("origin_server_ts")
@@ -87,6 +79,8 @@ class EventBase(object):
     redacts = _event_dict_property("redacts")
     room_id = _event_dict_property("room_id")
     sender = _event_dict_property("sender")
+    state_key = _event_dict_property("state_key")
+    type = _event_dict_property("type")
     user_id = _event_dict_property("sender")
 
     @property
@@ -105,7 +99,7 @@ class EventBase(object):
 
         return d
 
-    def get(self, key, default=None):
+    def get(self, key, default):
         return self._event_dict.get(key, default)
 
     def get_internal_metadata_dict(self):
@@ -159,11 +153,6 @@ class FrozenEvent(EventBase):
         else:
             frozen_dict = event_dict
 
-        self.event_id = event_dict["event_id"]
-        self.type = event_dict["type"]
-        if "state_key" in event_dict:
-            self.state_key = event_dict["state_key"]
-
         super(FrozenEvent, self).__init__(
             frozen_dict,
             signatures=signatures,

synapse/events/builder.py

@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from . import EventBase, FrozenEvent, _event_dict_property
+from . import EventBase, FrozenEvent
 
 from synapse.types import EventID
 
@@ -34,10 +34,6 @@ class EventBuilder(EventBase):
             internal_metadata_dict=internal_metadata_dict,
         )
 
-    event_id = _event_dict_property("event_id")
-    state_key = _event_dict_property("state_key")
-    type = _event_dict_property("type")
-
     def build(self):
         return FrozenEvent.from_event(self)
 
@@ -55,7 +51,7 @@ class EventBuilderFactory(object):
 
         local_part = str(int(self.clock.time())) + i + random_string(5)
 
-        e_id = EventID(local_part, self.hostname)
+        e_id = EventID.create(local_part, self.hostname)
 
         return e_id.to_string()