Actually fix exceptions

Don't log ERROR when no profile exists
Merge pull request #4235 from matrix-org/travis/fix-auto-invite-errors
2018-11-29 11:46:28 +00:00 · 2018-11-29 11:32:03 +00:00 · 2018-11-28 18:53:22 -08:00 · 2018-11-28 08:57:56 -07:00 · 2018-11-22 09:00:25 -07:00 · 2018-11-22 08:50:05 -07:00
94 changed files with 674 additions and 1404 deletions
@@ -4,8 +4,8 @@ jobs:
    machine: true
    steps:
      - checkout
-      - run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:${CIRCLE_TAG} .
-      - run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:${CIRCLE_TAG}-py3 --build-arg PYTHON_VERSION=3.6 .
+      - run: docker build -f docker/Dockerfile -t matrixdotorg/synapse:${CIRCLE_TAG} .
+      - run: docker build -f docker/Dockerfile -t matrixdotorg/synapse:${CIRCLE_TAG}-py3 --build-arg PYTHON_VERSION=3.6 .
      - run: docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
      - run: docker push matrixdotorg/synapse:${CIRCLE_TAG}
      - run: docker push matrixdotorg/synapse:${CIRCLE_TAG}-py3
@@ -13,9 +13,13 @@ jobs:
    machine: true
    steps:
      - checkout
-      - run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:${CIRCLE_SHA1} .
-      - run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:${CIRCLE_SHA1}-py3 --build-arg PYTHON_VERSION=3.6 .
+      - run: docker build -f docker/Dockerfile -t matrixdotorg/synapse:${CIRCLE_SHA1} .
+      - run: docker build -f docker/Dockerfile -t matrixdotorg/synapse:${CIRCLE_SHA1}-py3 --build-arg PYTHON_VERSION=3.6 .
      - run: docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
+      - run: docker tag matrixdotorg/synapse:${CIRCLE_SHA1} matrixdotorg/synapse:latest
+      - run: docker tag matrixdotorg/synapse:${CIRCLE_SHA1}-py3 matrixdotorg/synapse:latest-py3
+      - run: docker push matrixdotorg/synapse:${CIRCLE_SHA1}
+      - run: docker push matrixdotorg/synapse:${CIRCLE_SHA1}-py3
      - run: docker push matrixdotorg/synapse:latest
      - run: docker push matrixdotorg/synapse:latest-py3
  sytestpy2:
@@ -20,7 +20,7 @@ else
 fi

 # Show what we are before
-git --no-pager show -s
+git show -s

 # Set up username so it can do a merge
 git config --global user.email bot@matrix.org
@@ -31,4 +31,4 @@ git fetch -u origin $GITBASE
 git merge --no-edit origin/$GITBASE

 # Show what we are after.
-git --no-pager show -s
+git show -s
@@ -1,9 +0,0 @@
-# EditorConfig https://EditorConfig.org
-
-# top-most EditorConfig file
-root = true
-
-# 4 space indentation
-[*.py]
-indent_style = space
-indent_size = 4
@@ -36,24 +36,24 @@ matrix:
    env: TOX_ENV="pep8,check_isort"

  - python: 2.7
-    env: TOX_ENV=py27,codecov TRIAL_FLAGS="-j 2"
+    env: TOX_ENV=py27 TRIAL_FLAGS="-j 2"

  - python: 2.7
    env: TOX_ENV=py27-old TRIAL_FLAGS="-j 2"

  - python: 2.7
-    env: TOX_ENV=py27-postgres,codecov TRIAL_FLAGS="-j 4"
+    env: TOX_ENV=py27-postgres TRIAL_FLAGS="-j 4"
    services:
      - postgresql

  - python: 3.5
-    env: TOX_ENV=py35,codecov TRIAL_FLAGS="-j 2"
+    env: TOX_ENV=py35 TRIAL_FLAGS="-j 2"

  - python: 3.6
-    env: TOX_ENV=py36,codecov TRIAL_FLAGS="-j 2"
+    env: TOX_ENV=py36 TRIAL_FLAGS="-j 2"

  - python: 3.6
-    env: TOX_ENV=py36-postgres,codecov TRIAL_FLAGS="-j 4"
+    env: TOX_ENV=py36-postgres TRIAL_FLAGS="-j 4"
    services:
      - postgresql

@@ -1,73 +1,3 @@
-Synapse 0.34.0rc2 (2018-12-11)
-==============================
-
-Features
--------
-
- Add a welcome page for the client API port. Credit to @krombel! ([\#4289](https://github.com/matrix-org/synapse/issues/4289))
- Remove Matrix console from the default distribution ([\#4290](https://github.com/matrix-org/synapse/issues/4290))
-
-
-Internal Changes
----------------
-
- Disable pager when running git-show in CI ([\#4291](https://github.com/matrix-org/synapse/issues/4291))
-
-
-Synapse 0.34.0rc1 (2018-12-04)
-==============================
-
-Synapse 0.34 is the first release to fully support Python 3. We recommend
-upgrading to Python 3, but make sure to read the
-[upgrade notes](UPGRADE.rst#upgrading-to-v0340) when doing so.
-
-Features
--------
-
- Add option to track MAU stats (but not limit people) ([\#3830](https://github.com/matrix-org/synapse/issues/3830))
- Add an option to enable recording IPs for appservice users ([\#3831](https://github.com/matrix-org/synapse/issues/3831))
- Rename login type m.login.cas to m.login.sso ([\#4220](https://github.com/matrix-org/synapse/issues/4220))
- Add an option to disable search for homeservers that may not be interested in it. ([\#4230](https://github.com/matrix-org/synapse/issues/4230))
-
-
-Bugfixes
--------
-
- Pushrules can now again be made with non-ASCII rule IDs. ([\#4165](https://github.com/matrix-org/synapse/issues/4165))
- The media repository now no longer fails to decode UTF-8 filenames when downloading remote media. ([\#4176](https://github.com/matrix-org/synapse/issues/4176))
- URL previews now correctly decode non-UTF-8 text if the header contains a `<meta http-equiv="Content-Type"` header. ([\#4183](https://github.com/matrix-org/synapse/issues/4183))
- Fix an issue where public consent URLs had two slashes. ([\#4192](https://github.com/matrix-org/synapse/issues/4192))
- Fallback auth now accepts the session parameter on Python 3. ([\#4197](https://github.com/matrix-org/synapse/issues/4197))
- Remove riot.im from the list of trusted Identity Servers in the default configuration ([\#4207](https://github.com/matrix-org/synapse/issues/4207))
- fix start up failure when mau_limit_reserved_threepids set and db is postgres ([\#4211](https://github.com/matrix-org/synapse/issues/4211))
- Fix auto join failures for servers that require user consent ([\#4223](https://github.com/matrix-org/synapse/issues/4223))
- Fix exception caused by non-ascii event IDs ([\#4241](https://github.com/matrix-org/synapse/issues/4241))
- Pushers can now be unsubscribed from on Python 3. ([\#4250](https://github.com/matrix-org/synapse/issues/4250))
- Fix UnicodeDecodeError when postgres is configured to give non-English errors ([\#4253](https://github.com/matrix-org/synapse/issues/4253))
-
-
-Internal Changes
----------------
-
- A coveragerc file, as well as the py36-coverage tox target, have been added. ([\#4180](https://github.com/matrix-org/synapse/issues/4180))
- Add a GitHub pull request template and add multiple issue templates ([\#4182](https://github.com/matrix-org/synapse/issues/4182))
- Update README to reflect the fact that #1491 is fixed ([\#4188](https://github.com/matrix-org/synapse/issues/4188))
- Run the AS senders as background processes to fix warnings ([\#4189](https://github.com/matrix-org/synapse/issues/4189))
- Add some diagnostics to the tests to detect logcontext problems ([\#4190](https://github.com/matrix-org/synapse/issues/4190))
- Add missing `jpeg` package prerequisite for OpenBSD in README. ([\#4193](https://github.com/matrix-org/synapse/issues/4193))
- Add a note saying you need to manually reclaim disk space after using the Purge History API ([\#4200](https://github.com/matrix-org/synapse/issues/4200))
- More logcontext checking in unittests ([\#4205](https://github.com/matrix-org/synapse/issues/4205))
- Ignore __pycache__ directories in the database schema folder ([\#4214](https://github.com/matrix-org/synapse/issues/4214))
- Add note to UPGRADE.rst about removing riot.im from list of trusted identity servers ([\#4224](https://github.com/matrix-org/synapse/issues/4224))
- Added automated coverage reporting to CI. ([\#4225](https://github.com/matrix-org/synapse/issues/4225))
- Garbage-collect after each unit test to fix logcontext leaks ([\#4227](https://github.com/matrix-org/synapse/issues/4227))
- add more detail to logging regarding "More than one row matched" error ([\#4234](https://github.com/matrix-org/synapse/issues/4234))
- Drop sent_transactions table ([\#4244](https://github.com/matrix-org/synapse/issues/4244))
- Add a basic .editorconfig ([\#4257](https://github.com/matrix-org/synapse/issues/4257))
- Update README.rst and UPGRADE.rst for Python 3. ([\#4260](https://github.com/matrix-org/synapse/issues/4260))
- Remove obsolete `verbose` and `log_file` settings from `homeserver.yaml` for Docker image. ([\#4261](https://github.com/matrix-org/synapse/issues/4261))
-
-
 Synapse 0.33.9 (2018-11-19)
 ===========================

@@ -141,7 +71,7 @@ Synapse 0.33.8rc2 (2018-10-31)
 Bugfixes
 --------

- Searches that request profile info now no longer fail with a 500. Fixes
+- Searches that request profile info now no longer fail with a 500. Fixes 
  a regression in 0.33.8rc1. ([\#4122](https://github.com/matrix-org/synapse/issues/4122))


@@ -102,7 +102,7 @@ Sign off
 In order to have a concrete record that your contribution is intentional
 and you agree to license it under the same terms as the project's license, we've adopted the
 same lightweight approach that the Linux Kernel
-`submitting patches process <https://www.kernel.org/doc/html/latest/process/submitting-patches.html#sign-your-work-the-developer-s-certificate-of-origin>`_, Docker
+(https://www.kernel.org/doc/Documentation/SubmittingPatches), Docker
 (https://github.com/docker/docker/blob/master/CONTRIBUTING.md), and many other
 projects use: the DCO (Developer Certificate of Origin:
 http://developercertificate.org/). This is a simple declaration that you wrote
@@ -26,7 +26,6 @@ recursive-include synapse/static *.js
 exclude Dockerfile
 exclude .dockerignore
 exclude test_postgresql.sh
-exclude .editorconfig

 include pyproject.toml
 recursive-include changelog.d *
@@ -86,7 +86,7 @@ Synapse is the reference Python/Twisted Matrix homeserver implementation.
 System requirements:

 - POSIX-compliant system (tested on Linux & OS X)
- Python 3.5, 3.6, or 2.7
+- Python 2.7
 - At least 1GB of free RAM if you want to join large public rooms like #matrix:matrix.org

 Installing from source
@@ -101,13 +101,13 @@ header files for Python C extensions.

 Installing prerequisites on Ubuntu or Debian::

-    sudo apt-get install build-essential python3-dev libffi-dev \
+    sudo apt-get install build-essential python2.7-dev libffi-dev \
                         python-pip python-setuptools sqlite3 \
                         libssl-dev python-virtualenv libjpeg-dev libxslt1-dev

 Installing prerequisites on ArchLinux::

-    sudo pacman -S base-devel python python-pip \
+    sudo pacman -S base-devel python2 python-pip \
                   python-setuptools python-virtualenv sqlite3

 Installing prerequisites on CentOS 7 or Fedora 25::
@@ -126,9 +126,12 @@ Installing prerequisites on Mac OS X::

 Installing prerequisites on Raspbian::

-    sudo apt-get install build-essential python3-dev libffi-dev \
+    sudo apt-get install build-essential python2.7-dev libffi-dev \
                         python-pip python-setuptools sqlite3 \
                         libssl-dev python-virtualenv libjpeg-dev
+    sudo pip install --upgrade pip
+    sudo pip install --upgrade ndg-httpsclient
+    sudo pip install --upgrade virtualenv

 Installing prerequisites on openSUSE::

@@ -143,21 +146,20 @@ Installing prerequisites on OpenBSD::

 To install the Synapse homeserver run::

-    mkdir -p ~/synapse
-    virtualenv -p python3 ~/synapse/env
-    source ~/synapse/env/bin/activate
+    virtualenv -p python2.7 ~/.synapse
+    source ~/.synapse/bin/activate
    pip install --upgrade pip
    pip install --upgrade setuptools
    pip install matrix-synapse

 This installs Synapse, along with the libraries it uses, into a virtual
-environment under ``~/synapse/env``.  Feel free to pick a different directory
+environment under ``~/.synapse``.  Feel free to pick a different directory
 if you prefer.

 This Synapse installation can then be later upgraded by using pip again with the
 update flag::

-    source ~/synapse/env/bin/activate
+    source ~/.synapse/bin/activate
    pip install -U matrix-synapse

 In case of problems, please see the _`Troubleshooting` section below.
@@ -238,7 +240,7 @@ commandline script.

 To get started, it is easiest to use the command line to register new users::

-    $ source ~/synapse/env/bin/activate
+    $ source ~/.synapse/bin/activate
    $ synctl start # if not already running
    $ register_new_matrix_user -c homeserver.yaml https://localhost:8448
    New user localpart: erikj
@@ -264,12 +266,13 @@ Running Synapse
 ===============

 To actually run your new homeserver, pick a working directory for Synapse to
-run (e.g. ``~/synapse``), and::
+run (e.g. ``~/.synapse``), and::

-    cd ~/synapse
-    source env/bin/activate
+    cd ~/.synapse
+    source ./bin/activate
    synctl start

+
 Connecting to Synapse from a client
 ===================================

@@ -289,6 +292,10 @@ go back in your web client and proceed further.
 If all goes well you should at least be able to log in, create a room, and
 start sending messages.

+(The homeserver runs a web client by default at https://localhost:8448/, though
+as of the time of writing it is somewhat outdated and not really recommended -
+https://github.com/matrix-org/synapse/issues/1527).
+
 .. _`client-user-reg`:

 Registering a new user from a client
@@ -326,7 +333,7 @@ content served to web browsers a matrix API from being able to attack webapps ho
 on the same domain.  This is particularly true of sharing a matrix webclient and
 server on the same domain.

-See https://github.com/vector-im/riot-web/issues/1977 and
+See https://github.com/vector-im/vector-web/issues/1977 and
 https://developer.github.com/changes/2014-04-25-user-content-security for more details.


@@ -368,19 +375,40 @@ ArchLinux

 The quickest way to get up and running with ArchLinux is probably with the community package
 https://www.archlinux.org/packages/community/any/matrix-synapse/, which should pull in most of
-the necessary dependencies.
+the necessary dependencies. If the default web client is to be served (enabled by default in
+the generated config),
+https://www.archlinux.org/packages/community/any/python2-matrix-angular-sdk/ will also need to
+be installed.
+
+Alternatively, to install using pip a few changes may be needed as ArchLinux
+defaults to python 3, but synapse currently assumes python 2.7 by default:

 pip may be outdated (6.0.7-1 and needs to be upgraded to 6.0.8-1 )::

-    sudo pip install --upgrade pip
+    sudo pip2.7 install --upgrade pip
+
+You also may need to explicitly specify python 2.7 again during the install
+request::
+
+    pip2.7 install https://github.com/matrix-org/synapse/tarball/master

 If you encounter an error with lib bcrypt causing an Wrong ELF Class:
 ELFCLASS32 (x64 Systems), you may need to reinstall py-bcrypt to correctly
 compile it under the right architecture. (This should not be needed if
 installing under virtualenv)::

-    sudo pip uninstall py-bcrypt
-    sudo pip install py-bcrypt
+    sudo pip2.7 uninstall py-bcrypt
+    sudo pip2.7 install py-bcrypt
+
+During setup of Synapse you need to call python2.7 directly again::
+
+    cd ~/.synapse
+    python2.7 -m synapse.app.homeserver \
+      --server-name machine.my.domain.name \
+      --config-path homeserver.yaml \
+      --generate-config
+
+...substituting your host and domain name as appropriate.

 FreeBSD
 -------
@@ -447,7 +475,7 @@ You can fix this by manually upgrading pip and virtualenv::

    sudo pip install --upgrade virtualenv

-You can next rerun ``virtualenv -p python3 synapse`` to update the virtual env.
+You can next rerun ``virtualenv -p python2.7 synapse`` to update the virtual env.

 Installing may fail during installing virtualenv with ``InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.``
 You can fix this  by manually installing ndg-httpsclient::
@@ -496,6 +524,16 @@ log lines and looking for any 'Processed request' lines which take more than
 a few seconds to execute.  Please let us know at #matrix-dev:matrix.org if
 you see this failure mode so we can help debug it, however.

+ArchLinux
+~~~~~~~~~
+
+If running `$ synctl start` fails with 'returned non-zero exit status 1',
+you will need to explicitly call Python2.7 - either running as::
+
+    python2.7 -m synapse.app.homeserver --daemonize -c homeserver.yaml
+
+...or by editing synctl with the correct python executable.
+

 Upgrading an existing Synapse
 =============================
@@ -693,7 +731,7 @@ port:

 * Until v0.33.3, Synapse did not support SNI on the federation port
  (`bug #1491 <https://github.com/matrix-org/synapse/issues/1491>`_). This bug
-  is now fixed, but means that federating with older servers can be unreliable
+  is now fixed, but means that federating with older servers can be unreliable 
  when using name-based virtual hosting.

 Furthermore, a number of the normal reasons for using a reverse-proxy do not
@@ -790,7 +828,7 @@ Password reset
 ==============

 If a user has registered an email address to their account using an identity
-server, they can request a password-reset token via clients such as Riot.
+server, they can request a password-reset token via clients such as Vector.

 A manual password reset can be done via direct database access as follows.

@@ -48,78 +48,6 @@ returned by the Client-Server API:
    # configured on port 443.
    curl -kv https://<host.name>/_matrix/client/versions 2>&1 | grep "Server:"

-Upgrading to v0.34.0
-====================
-
-1. This release is the first to fully support Python 3. We recommend switching
-   to Python 3, as it has been shown to give performance improvements.
-
-   For users who have installed Synapse into a virtualenv, we recommend doing
-   this by creating a new virtualenv. For example::
-
-       virtualenv -p python3 ~/synapse/env3
-       source ~/synapse/env3/bin/activate
-       pip install matrix-synapse
-
-   You can then start synapse as normal, having activated the new virtualenv::
-
-       cd ~/synapse
-       source env3/bin/activate
-       synctl start
-
-   Users who have installed from distribution packages should see the relevant
-   package documentation.
-
-   * When upgrading to Python 3, you **must** make sure that your log files are
-     configured as UTF-8, by adding ``encoding: utf8`` to the
-     ``RotatingFileHandler`` configuration (if you have one) in your
-     ``<server>.log.config`` file. For example, if your ``log.config`` file
-     contains::
-
-       handlers:
-         file:
-           class: logging.handlers.RotatingFileHandler
-           formatter: precise
-           filename: homeserver.log
-           maxBytes: 104857600
-           backupCount: 10
-           filters: [context]
-         console:
-           class: logging.StreamHandler
-           formatter: precise
-           filters: [context]
-
-     Then you should update this to be::
-
-       handlers:
-         file:
-           class: logging.handlers.RotatingFileHandler
-           formatter: precise
-           filename: homeserver.log
-           maxBytes: 104857600
-           backupCount: 10
-           filters: [context]
-           encoding: utf8
-         console:
-           class: logging.StreamHandler
-           formatter: precise
-           filters: [context]
-
-     There is no need to revert this change if downgrading to Python 2.
-
-2. This release removes the ``riot.im`` from the default list of trusted
-   identity servers.
-
-   If ``riot.im`` is in your homeserver's list of
-   ``trusted_third_party_id_servers``, you should remove it. It was added in
-   case a hypothetical future identity server was put there. If you don't
-   remove it, users may be unable to deactivate their accounts.
-
-3. This release no longer installs the (unmaintained) Matrix Console web client
-   as part of the default installation. It is possible to re-enable it by
-   installing it separately and setting the ``web_client_location`` config
-   option, but please consider switching to another client.
-
 Upgrading to v0.33.7
 ====================

@@ -0,0 +1 @@
+Add option to track MAU stats (but not limit people)
@@ -0,0 +1 @@
+The media repository now no longer fails to decode UTF-8 filenames when downloading remote media.
@@ -0,0 +1 @@
+A coveragerc file, as well as the py36-coverage tox target, have been added.
@@ -0,0 +1 @@
+Add a GitHub pull request template and add multiple issue templates
@@ -0,0 +1 @@
+URL previews now correctly decode non-UTF-8 text if the header contains a `<meta http-equiv="Content-Type"` header.
@@ -0,0 +1 @@
+Update README to reflect the fact that #1491 is fixed
@@ -0,0 +1 @@
+Fix an issue where public consent URLs had two slashes.
@@ -0,0 +1 @@
+Add missing `jpeg` package prerequisite for OpenBSD in README.
@@ -0,0 +1 @@
+Fallback auth now accepts the session parameter on Python 3.
@@ -0,0 +1 @@
+Add a note saying you need to manually reclaim disk space after using the Purge History API
@@ -0,0 +1 @@
+Fix logcontext leaks in EmailPusher and in tests
@@ -0,0 +1 @@
+Remove riot.im from the list of trusted Identity Servers in the default configuration
@@ -1 +0,0 @@
-Support for serving .well-known files
@@ -1 +0,0 @@
-Fix CAS login when username is not valid in an MXID
@@ -1 +0,0 @@
-Rework SAML2 authentication
@@ -1 +0,0 @@
-drop undocumented dependency on dateutil
@@ -1 +0,0 @@
-Rework SAML2 authentication
@@ -1 +0,0 @@
-SAML2 authentication: Initialise user display name from SAML2 data
@@ -1 +0,0 @@
-Update the example systemd config to use a virtualenv
@@ -1 +0,0 @@
-Update link to kernel DCO guide
@@ -1 +0,0 @@
-Send CORS headers for /media/config
@@ -1 +0,0 @@
-Make isort tox check print diff when it fails
@@ -1 +0,0 @@
-Add 'sandbox' to CSP for media reprository
@@ -1 +0,0 @@
-Make the new landing page prettier.
@@ -1 +0,0 @@
-Log room_id in Unknown room errors
@@ -1,31 +0,0 @@
-# Example systemd configuration file for synapse. Copy into
-#    /etc/systemd/system/, update the paths if necessary, then:
-#
-#    systemctl enable matrix-synapse
-#    systemctl start matrix-synapse
-#
-# This assumes that Synapse has been installed in a virtualenv in
-# /opt/synapse/env.
-#
-# **NOTE:** This is an example service file that may change in the future. If you
-# wish to use this please copy rather than symlink it.
-
-[Unit]
-Description=Synapse Matrix homeserver
-
-[Service]
-Type=simple
-Restart=on-abort
-
-User=synapse
-Group=nogroup
-
-WorkingDirectory=/opt/synapse
-ExecStart=/opt/synapse/env/bin/python -m synapse.app.homeserver --config-path=/opt/synapse/homeserver.yaml
-
-# adjust the cache factor if necessary
-# Environment=SYNAPSE_CACHE_FACTOR=2.0
-
-[Install]
-WantedBy=multi-user.target
-
@@ -0,0 +1,22 @@
+# This assumes that Synapse has been installed as a system package
+# (e.g. https://www.archlinux.org/packages/community/any/matrix-synapse/ for ArchLinux)
+# rather than in a user home directory or similar under virtualenv.
+
+# **NOTE:** This is an example service file that may change in the future. If you
+# wish to use this please copy rather than symlink it.
+
+[Unit]
+Description=Synapse Matrix homeserver
+
+[Service]
+Type=simple
+User=synapse
+Group=synapse
+WorkingDirectory=/var/lib/synapse
+ExecStart=/usr/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/synapse/homeserver.yaml
+ExecStop=/usr/bin/synctl stop /etc/synapse/homeserver.yaml
+# EnvironmentFile=-/etc/sysconfig/synapse  # Can be used to e.g. set SYNAPSE_CACHE_FACTOR 
+
+[Install]
+WantedBy=multi-user.target
+
@@ -14,7 +14,6 @@ server_name: "{{ SYNAPSE_SERVER_NAME }}"
 pid_file: /homeserver.pid
 web_client: False
 soft_file_limit: 0
-log_config: "/compiled/log.config"

 ## Ports ##

@@ -68,6 +67,9 @@ database:
 ## Performance ##

 event_cache_size: "{{ SYNAPSE_EVENT_CACHE_SIZE or "10K" }}"
+verbose: 0
+log_file: "/data/homeserver.log"
+log_config: "/compiled/log.config"

 ## Ratelimiting ##

@@ -163,7 +163,7 @@ the logcontext was set, this will make things work out ok: provided
 It's all too easy to forget to ``yield``: for instance if we forgot that
 ``do_some_stuff`` returned a deferred, we might plough on regardless. This
 leads to a mess; it will probably work itself out eventually, but not before
-a load of stuff has been logged against the wrong context. (Normally, other
+a load of stuff has been logged against the wrong content. (Normally, other
 things will break, more obviously, if you forget to ``yield``, so this tends
 not to be a major problem in practice.)

@@ -440,59 +440,3 @@ To conclude: I think this scheme would have worked equally well, with less
 danger of messing it up, and probably made some more esoteric code easier to
 write. But again — changing the conventions of the entire Synapse codebase is
 not a sensible option for the marginal improvement offered.
-
-
-A note on garbage-collection of Deferred chains
-----------------------------------------------
-
-It turns out that our logcontext rules do not play nicely with Deferred
-chains which get orphaned and garbage-collected.
-
-Imagine we have some code that looks like this:
-
-.. code:: python
-
-    listener_queue = []
-
-    def on_something_interesting():
-        for d in listener_queue:
-            d.callback("foo")
-
-    @defer.inlineCallbacks
-    def await_something_interesting():
-        new_deferred = defer.Deferred()
-        listener_queue.append(new_deferred)
-
-        with PreserveLoggingContext():
-            yield new_deferred
-
-Obviously, the idea here is that we have a bunch of things which are waiting
-for an event. (It's just an example of the problem here, but a relatively
-common one.)
-
-Now let's imagine two further things happen. First of all, whatever was
-waiting for the interesting thing goes away. (Perhaps the request times out,
-or something *even more* interesting happens.)
-
-Secondly, let's suppose that we decide that the interesting thing is never
-going to happen, and we reset the listener queue:
-
-.. code:: python
-
-    def reset_listener_queue():
-        listener_queue.clear()
-
-So, both ends of the deferred chain have now dropped their references, and the
-deferred chain is now orphaned, and will be garbage-collected at some point.
-Note that ``await_something_interesting`` is a generator function, and when
-Python garbage-collects generator functions, it gives them a chance to clean
-up by making the ``yield`` raise a ``GeneratorExit`` exception. In our case,
-that means that the ``__exit__`` handler of ``PreserveLoggingContext`` will
-carefully restore the request context, but there is now nothing waiting for
-its return, so the request context is never cleared.
-
-To reiterate, this problem only arises when *both* ends of a deferred chain
-are dropped. Dropping the the reference to a deferred you're supposed to be
-calling is probably bad practice, so this doesn't actually happen too much.
-Unfortunately, when it does happen, it will lead to leaked logcontexts which
-are incredibly hard to track down.
@@ -27,4 +27,4 @@ try:
 except ImportError:
    pass

-__version__ = "0.34.0rc2"
+__version__ = "0.33.9"
@@ -188,33 +188,17 @@ class Auth(object):
        """
        # Can optionally look elsewhere in the request (e.g. headers)
        try:
-            ip_addr = self.hs.get_ip_from_request(request)
-            user_agent = request.requestHeaders.getRawHeaders(
-                b"User-Agent",
-                default=[b""]
-            )[0].decode('ascii', 'surrogateescape')
+            user_id, app_service = yield self._get_appservice_user_id(request)
+            if user_id:
+                request.authenticated_entity = user_id
+                defer.returnValue(
+                    synapse.types.create_requester(user_id, app_service=app_service)
+                )

            access_token = self.get_access_token_from_request(
                request, self.TOKEN_NOT_FOUND_HTTP_STATUS
            )

-            user_id, app_service = yield self._get_appservice_user_id(request)
-            if user_id:
-                request.authenticated_entity = user_id
-
-                if ip_addr and self.hs.config.track_appservice_user_ips:
-                    yield self.store.insert_client_ip(
-                        user_id=user_id,
-                        access_token=access_token,
-                        ip=ip_addr,
-                        user_agent=user_agent,
-                        device_id="dummy-device",  # stubbed
-                    )
-
-                defer.returnValue(
-                    synapse.types.create_requester(user_id, app_service=app_service)
-                )
-
            user_info = yield self.get_user_by_access_token(access_token, rights)
            user = user_info["user"]
            token_id = user_info["token_id"]
@@ -224,6 +208,11 @@ class Auth(object):
            # stubbed out.
            device_id = user_info.get("device_id")

+            ip_addr = self.hs.get_ip_from_request(request)
+            user_agent = request.requestHeaders.getRawHeaders(
+                b"User-Agent",
+                default=[b""]
+            )[0].decode('ascii', 'surrogateescape')
            if user and access_token and ip_addr:
                yield self.store.insert_client_ip(
                    user_id=user.to_string(),
@@ -54,13 +54,12 @@ from synapse.metrics import RegistryProxy
 from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
 from synapse.module_api import ModuleApi
-from synapse.python_dependencies import check_requirements
+from synapse.python_dependencies import CONDITIONAL_REQUIREMENTS, check_requirements
 from synapse.replication.http import REPLICATION_PREFIX, ReplicationRestResource
 from synapse.replication.tcp.resource import ReplicationStreamProtocolFactory
 from synapse.rest import ClientRestResource
 from synapse.rest.key.v2 import KeyApiV2Resource
 from synapse.rest.media.v0.content_repository import ContentRepoResource
-from synapse.rest.well_known import WellKnownResource
 from synapse.server import HomeServer
 from synapse.storage import DataStore, are_all_users_on_domain
 from synapse.storage.engines import IncorrectDatabaseSetup, create_engine
@@ -80,6 +79,36 @@ def gz_wrap(r):
    return EncodingResourceWrapper(r, [GzipEncoderFactory()])


+def build_resource_for_web_client(hs):
+    webclient_path = hs.get_config().web_client_location
+    if not webclient_path:
+        try:
+            import syweb
+        except ImportError:
+            quit_with_error(
+                "Could not find a webclient.\n\n"
+                "Please either install the matrix-angular-sdk or configure\n"
+                "the location of the source to serve via the configuration\n"
+                "option `web_client_location`\n\n"
+                "To install the `matrix-angular-sdk` via pip, run:\n\n"
+                "    pip install '%(dep)s'\n"
+                "\n"
+                "You can also disable hosting of the webclient via the\n"
+                "configuration option `web_client`\n"
+                % {"dep": CONDITIONAL_REQUIREMENTS["web_client"].keys()[0]}
+            )
+        syweb_path = os.path.dirname(syweb.__file__)
+        webclient_path = os.path.join(syweb_path, "webclient")
+    # GZip is disabled here due to
+    # https://twistedmatrix.com/trac/ticket/7678
+    # (It can stay enabled for the API resources: they call
+    # write() with the whole body and then finish() straight
+    # after and so do not trigger the bug.
+    # GzipFile was removed in commit 184ba09
+    # return GzipFile(webclient_path)  # TODO configurable?
+    return File(webclient_path)  # TODO configurable?
+
+
 class SynapseHomeServer(HomeServer):
    DATASTORE_CLASS = DataStore

@@ -108,11 +137,8 @@ class SynapseHomeServer(HomeServer):
            handler = handler_cls(config, module_api)
            resources[path] = AdditionalResource(self, handler.handle_request)

-        # try to find something useful to redirect '/' to
        if WEB_CLIENT_PREFIX in resources:
            root_resource = RootRedirect(WEB_CLIENT_PREFIX)
-        elif STATIC_PREFIX in resources:
-            root_resource = RootRedirect(STATIC_PREFIX)
        else:
            root_resource = NoResource()

@@ -169,13 +195,8 @@ class SynapseHomeServer(HomeServer):
                "/_matrix/client/unstable": client_resource,
                "/_matrix/client/v2_alpha": client_resource,
                "/_matrix/client/versions": client_resource,
-                "/.well-known/matrix/client": WellKnownResource(self),
            })

-            if self.get_config().saml2_enabled:
-                from synapse.rest.saml2 import SAML2Resource
-                resources["/_matrix/saml2"] = SAML2Resource(self)
-
        if name == "consent":
            from synapse.rest.consent.consent_resource import ConsentResource
            consent_resource = ConsentResource(self)
@@ -216,16 +237,7 @@ class SynapseHomeServer(HomeServer):
            resources[SERVER_KEY_V2_PREFIX] = KeyApiV2Resource(self)

        if name == "webclient":
-            webclient_path = self.get_config().web_client_location
-
-            if webclient_path is None:
-                logger.warning(
-                    "Not enabling webclient resource, as web_client_location is unset."
-                )
-            else:
-                # GZip is disabled here due to
-                # https://twistedmatrix.com/trac/ticket/7678
-                resources[WEB_CLIENT_PREFIX] = File(webclient_path)
+            resources[WEB_CLIENT_PREFIX] = build_resource_for_web_client(self)

        if name == "metrics" and self.get_config().enable_metrics:
            resources[METRICS_PREFIX] = MetricsResource(RegistryProxy)
@@ -53,8 +53,8 @@ import logging
 from twisted.internet import defer

 from synapse.appservice import ApplicationServiceState
-from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.util.logcontext import run_in_background
+from synapse.util.metrics import Measure

 logger = logging.getLogger(__name__)

@@ -104,23 +104,14 @@ class _ServiceQueuer(object):
        self.clock = clock

    def enqueue(self, service, event):
+        # if this service isn't being sent something
        self.queued_events.setdefault(service.id, []).append(event)
-
-        # start a sender for this appservice if we don't already have one
-
-        if service.id in self.requests_in_flight:
-            return
-
-        run_as_background_process(
-            "as-sender-%s" % (service.id, ),
-            self._send_request, service,
-        )
+        run_in_background(self._send_request, service)

    @defer.inlineCallbacks
    def _send_request(self, service):
-        # sanity-check: we shouldn't get here if this service already has a sender
-        # running.
-        assert(service.id not in self.requests_in_flight)
+        if service.id in self.requests_in_flight:
+            return

        self.requests_in_flight.add(service.id)
        try:
@@ -128,10 +119,12 @@ class _ServiceQueuer(object):
                events = self.queued_events.pop(service.id, [])
                if not events:
                    return
-                try:
-                    yield self.txn_ctrl.send(service, events)
-                except Exception:
-                    logger.exception("AS request failed")
+
+                with Measure(self.clock, "servicequeuer.send"):
+                    try:
+                        yield self.txn_ctrl.send(service, events)
+                    except Exception:
+                        logger.exception("AS request failed")
        finally:
            self.requests_in_flight.discard(service.id)

@@ -230,12 +223,7 @@ class _Recoverer(object):
        self.backoff_counter = 1

    def recover(self):
-        def _retry():
-            run_as_background_process(
-                "as-recoverer-%s" % (self.service.id,),
-                self.retry,
-            )
-        self.clock.call_later((2 ** self.backoff_counter), _retry)
+        self.clock.call_later((2 ** self.backoff_counter), self.retry)

    def _backoff(self):
        # cap the backoff to be around 8.5min => (2^9) = 512 secs
@@ -33,16 +33,11 @@ class AppServiceConfig(Config):
    def read_config(self, config):
        self.app_service_config_files = config.get("app_service_config_files", [])
        self.notify_appservices = config.get("notify_appservices", True)
-        self.track_appservice_user_ips = config.get("track_appservice_user_ips", False)

    def default_config(cls, **kwargs):
        return """\
        # A list of application service config file to use
        app_service_config_files: []
-
-        # Whether or not to track application service IP addresses. Implicitly
-        # enables MAU tracking for application service users.
-        track_appservice_user_ips: False
        """


@@ -32,7 +32,7 @@ from .ratelimiting import RatelimitConfig
 from .registration import RegistrationConfig
 from .repository import ContentRepositoryConfig
 from .room_directory import RoomDirectoryConfig
-from .saml2_config import SAML2Config
+from .saml2 import SAML2Config
 from .server import ServerConfig
 from .server_notices_config import ServerNoticesConfig
 from .spam_checker import SpamCheckerConfig
@@ -37,7 +37,6 @@ class RegistrationConfig(Config):

        self.bcrypt_rounds = config.get("bcrypt_rounds", 12)
        self.trusted_third_party_id_servers = config["trusted_third_party_id_servers"]
-        self.default_identity_server = config.get("default_identity_server")
        self.allow_guest_access = config.get("allow_guest_access", False)

        self.invite_3pid_guest = (
@@ -92,14 +91,6 @@ class RegistrationConfig(Config):
        # accessible to anonymous users.
        allow_guest_access: False

-        # The identity server which we suggest that clients should use when users log
-        # in on this server.
-        #
-        # (By default, no suggestion is made, so it is left up to the client.
-        # This setting is ignored unless public_baseurl is also set.)
-        #
-        # default_identity_server: https://matrix.org
-
        # The list of identity servers trusted to verify third party
        # identifiers by this server.
        #
@@ -0,0 +1,55 @@
+# -*- coding: utf-8 -*-
+# Copyright 2015 Ericsson
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from ._base import Config
+
+
+class SAML2Config(Config):
+    """SAML2 Configuration
+    Synapse uses pysaml2 libraries for providing SAML2 support
+
+    config_path:      Path to the sp_conf.py configuration file
+    idp_redirect_url: Identity provider URL which will redirect
+                      the user back to /login/saml2 with proper info.
+
+    sp_conf.py file is something like:
+    https://github.com/rohe/pysaml2/blob/master/example/sp-repoze/sp_conf.py.example
+
+    More information: https://pythonhosted.org/pysaml2/howto/config.html
+    """
+
+    def read_config(self, config):
+        saml2_config = config.get("saml2_config", None)
+        if saml2_config:
+            self.saml2_enabled = saml2_config.get("enabled", True)
+            self.saml2_config_path = saml2_config["config_path"]
+            self.saml2_idp_redirect_url = saml2_config["idp_redirect_url"]
+        else:
+            self.saml2_enabled = False
+            self.saml2_config_path = None
+            self.saml2_idp_redirect_url = None
+
+    def default_config(self, config_dir_path, server_name, **kwargs):
+        return """
+        # Enable SAML2 for registration and login. Uses pysaml2
+        # config_path:      Path to the sp_conf.py configuration file
+        # idp_redirect_url: Identity provider URL which will redirect
+        #                   the user back to /login/saml2 with proper info.
+        # See pysaml2 docs for format of config.
+        #saml2_config:
+        #   enabled: true
+        #   config_path: "%s/sp_conf.py"
+        #   idp_redirect_url: "http://%s/idp"
+        """ % (config_dir_path, server_name)
@@ -1,110 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2018 New Vector Ltd.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from ._base import Config, ConfigError
-
-
-class SAML2Config(Config):
-    def read_config(self, config):
-        self.saml2_enabled = False
-
-        saml2_config = config.get("saml2_config")
-
-        if not saml2_config or not saml2_config.get("enabled", True):
-            return
-
-        self.saml2_enabled = True
-
-        import saml2.config
-        self.saml2_sp_config = saml2.config.SPConfig()
-        self.saml2_sp_config.load(self._default_saml_config_dict())
-        self.saml2_sp_config.load(saml2_config.get("sp_config", {}))
-
-        config_path = saml2_config.get("config_path", None)
-        if config_path is not None:
-            self.saml2_sp_config.load_file(config_path)
-
-    def _default_saml_config_dict(self):
-        import saml2
-
-        public_baseurl = self.public_baseurl
-        if public_baseurl is None:
-            raise ConfigError(
-                "saml2_config requires a public_baseurl to be set"
-            )
-
-        metadata_url = public_baseurl + "_matrix/saml2/metadata.xml"
-        response_url = public_baseurl + "_matrix/saml2/authn_response"
-        return {
-            "entityid": metadata_url,
-
-            "service": {
-                "sp": {
-                    "endpoints": {
-                        "assertion_consumer_service": [
-                            (response_url, saml2.BINDING_HTTP_POST),
-                        ],
-                    },
-                    "required_attributes": ["uid"],
-                    "optional_attributes": ["mail", "surname", "givenname"],
-                },
-            }
-        }
-
-    def default_config(self, config_dir_path, server_name, **kwargs):
-        return """
-        # Enable SAML2 for registration and login. Uses pysaml2.
-        #
-        # saml2_config:
-        #
-        #   # The following is the configuration for the pysaml2 Service Provider.
-        #   # See pysaml2 docs for format of config.
-        #   #
-        #   # Default values will be used for the 'entityid' and 'service' settings,
-        #   # so it is not normally necessary to specify them unless you need to
-        #   # override them.
-        #
-        #   sp_config:
-        #     # point this to the IdP's metadata. You can use either a local file or
-        #     # (preferably) a URL.
-        #     metadata:
-        #       # local: ["saml2/idp.xml"]
-        #       remote:
-        #         - url: https://our_idp/metadata.xml
-        #
-        #     # The following is just used to generate our metadata xml, and you
-        #     # may well not need it, depending on your setup. Alternatively you
-        #     # may need a whole lot more detail - see the pysaml2 docs!
-        #
-        #     description: ["My awesome SP", "en"]
-        #     name: ["Test SP", "en"]
-        #
-        #     organization:
-        #       name: Example com
-        #       display_name:
-        #         - ["Example co", "en"]
-        #       url: "http://example.com"
-        #
-        #     contact_person:
-        #       - given_name: Bob
-        #         sur_name: "the Sysadmin"
-        #         email_address": ["admin@example.com"]
-        #         contact_type": technical
-        #
-        #   # Instead of putting the config inline as above, you can specify a
-        #   # separate pysaml2 configuration file:
-        #   #
-        #   # config_path: "%(config_dir_path)s/sp_conf.py"
-        """ % {"config_dir_path": config_dir_path}
@@ -34,6 +34,7 @@ class ServerConfig(Config):
            raise ConfigError(str(e))

        self.pid_file = self.abspath(config.get("pid_file"))
+        self.web_client = config["web_client"]
        self.web_client_location = config.get("web_client_location", None)
        self.soft_file_limit = config["soft_file_limit"]
        self.daemonize = config.get("daemonize")
@@ -61,11 +62,6 @@ class ServerConfig(Config):
        # master, potentially causing inconsistency.
        self.enable_media_repo = config.get("enable_media_repo", True)

-        # whether to enable search. If disabled, new entries will not be inserted
-        # into the search tables and they will not be indexed. Users will receive
-        # errors when attempting to search for messages.
-        self.enable_search = config.get("enable_search", True)
-
        self.filter_timeline_limit = config.get("filter_timeline_limit", -1)

        # Whether we should block invites sent to users on this server
@@ -127,9 +123,6 @@ class ServerConfig(Config):
            elif not bind_addresses:
                bind_addresses.append('')

-        if not self.web_client_location:
-            _warn_if_webclient_configured(self.listeners)
-
        self.gc_thresholds = read_gc_thresholds(config.get("gc_thresholds", None))

        bind_port = config.get("bind_port")
@@ -138,6 +131,8 @@ class ServerConfig(Config):
            bind_host = config.get("bind_host", "")
            gzip_responses = config.get("gzip_responses", True)

+            names = ["client", "webclient"] if self.web_client else ["client"]
+
            self.listeners.append({
                "port": bind_port,
                "bind_addresses": [bind_host],
@@ -145,7 +140,7 @@ class ServerConfig(Config):
                "type": "http",
                "resources": [
                    {
-                        "names": ["client"],
+                        "names": names,
                        "compress": gzip_responses,
                    },
                    {
@@ -164,7 +159,7 @@ class ServerConfig(Config):
                    "type": "http",
                    "resources": [
                        {
-                            "names": ["client"],
+                            "names": names,
                            "compress": gzip_responses,
                        },
                        {
@@ -247,9 +242,13 @@ class ServerConfig(Config):
        #
        # cpu_affinity: 0xFFFFFFFF

-        # The path to the web client which will be served at /_matrix/client/
-        # if 'webclient' is configured under the 'listeners' configuration.
-        #
+        # Whether to serve a web client from the HTTP/HTTPS root resource.
+        web_client: True
+
+        # The root directory to server for the above web client.
+        # If left undefined, synapse will serve the matrix-angular-sdk web client.
+        # Make sure matrix-angular-sdk is installed with pip if web_client is True
+        # and web_client_location is undefined
        # web_client_location: "/path/to/web/root"

        # The public-facing base URL for the client API (not including _matrix/...)
@@ -316,8 +315,8 @@ class ServerConfig(Config):
              -
                # List of resources to host on this listener.
                names:
-                  - client       # The client-server APIs, both v1 and v2
-                  # - webclient  # A web client. Requires web_client_location to be set.
+                  - client     # The client-server APIs, both v1 and v2
+                  - webclient  # The bundled webclient.

                # Should synapse compress HTTP responses to clients that support it?
                # This should be disabled if running synapse behind a load balancer
@@ -344,7 +343,7 @@ class ServerConfig(Config):
            x_forwarded: false

            resources:
-              - names: [client]
+              - names: [client, webclient]
                compress: true
              - names: [federation]
                compress: false
@@ -385,12 +384,7 @@ class ServerConfig(Config):
          # mau_limit_reserved_threepids:
          # - medium: 'email'
          #   address: 'reserved_user@example.com'
-          #
-          # Room searching
-          #
-          # If disabled, new messages will not be indexed for searching and users
-          # will receive errors when searching for messages. Defaults to enabled.
-          # enable_search: true
+
        """ % locals()

    def read_arguments(self, args):
@@ -448,19 +442,3 @@ def read_gc_thresholds(thresholds):
        raise ConfigError(
            "Value of `gc_threshold` must be a list of three integers if set"
        )
-
-
-NO_MORE_WEB_CLIENT_WARNING = """
-Synapse no longer includes a web client. To enable a web client, configure
-web_client_location. To remove this warning, remove 'webclient' from the 'listeners'
-configuration.
-"""
-
-
-def _warn_if_webclient_configured(listeners):
-    for listener in listeners:
-        for res in listener.get("resources", []):
-            for name in res.get("names", []):
-                if name == 'webclient':
-                    logger.warning(NO_MORE_WEB_CLIENT_WARNING)
-                    return
@@ -563,10 +563,10 @@ class AuthHandler(BaseHandler):
        insensitively, but return None if there are multiple inexact matches.

        Args:
-            (unicode|bytes) user_id: complete @user:id
+            (str) user_id: complete @user:id

        Returns:
-            defer.Deferred: (unicode) canonical_user_id, or None if zero or
+            defer.Deferred: (str) canonical_user_id, or None if zero or
            multiple matches
        """
        res = yield self._find_user_id_and_pwd_hash(user_id)
@@ -954,15 +954,6 @@ class MacaroonGenerator(object):
        return macaroon.serialize()

    def generate_short_term_login_token(self, user_id, duration_in_ms=(2 * 60 * 1000)):
-        """
-
-        Args:
-            user_id (unicode):
-            duration_in_ms (int):
-
-        Returns:
-            unicode
-        """
        macaroon = self._generate_base_macaroon(user_id)
        macaroon.add_first_party_caveat("type = login")
        now = self.hs.get_clock().time_msec()
@@ -259,10 +259,8 @@ class DirectoryHandler(BaseHandler):
                servers = result["servers"]

        if not room_id:
-            raise SynapseError(
-                404,
+            raise NotFoundError(
                "Room alias %s not found" % (room_alias.to_string(),),
-                Codes.NOT_FOUND
            )

        users = yield self.state.get_current_user_in_room(room_id)
@@ -302,10 +300,8 @@ class DirectoryHandler(BaseHandler):
                "servers": result.servers,
            })
        else:
-            raise SynapseError(
-                404,
+            raise NotFoundError(
                "Room alias %r not found" % (room_alias.to_string(),),
-                Codes.NOT_FOUND
            )

    @defer.inlineCallbacks
@@ -48,13 +48,14 @@ from synapse.crypto.event_signing import (
    compute_event_signature,
 )
 from synapse.events.validator import EventValidator
+from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.replication.http.federation import (
    ReplicationCleanRoomRestServlet,
    ReplicationFederationSendEventsRestServlet,
 )
 from synapse.replication.http.membership import ReplicationUserJoinedLeftRoomRestServlet
 from synapse.state import StateResolutionStore, resolve_events_with_store
-from synapse.types import UserID, get_domain_from_id
+from synapse.types import UserID, create_requester, get_domain_from_id
 from synapse.util import logcontext, unwrapFirstError
 from synapse.util.async_helpers import Linearizer
 from synapse.util.distributor import user_joined_room
@@ -105,6 +106,7 @@ class FederationHandler(BaseHandler):

        self.hs = hs

+        self.clock = hs.get_clock()
        self.store = hs.get_datastore()  # type: synapse.storage.DataStore
        self.federation_client = hs.get_federation_client()
        self.state_handler = hs.get_state_handler()
@@ -1300,8 +1302,37 @@ class FederationHandler(BaseHandler):
        context = yield self.state_handler.compute_event_context(event)
        yield self.persist_events_and_notify([(event, context)])

+        sender = UserID.from_string(event.sender)
+        target = UserID.from_string(event.state_key)
+        if (sender.localpart == target.localpart):
+            run_as_background_process(
+                "_auto_accept_invite",
+                self._auto_accept_invite,
+                sender, target, event.room_id,
+            )
+
        defer.returnValue(event)

+    @defer.inlineCallbacks
+    def _auto_accept_invite(self, sender, target, room_id):
+        joined = False
+        for attempt in range(0, 10):
+            try:
+                yield self.hs.get_room_member_handler().update_membership(
+                    requester=create_requester(target.to_string()),
+                    target=target,
+                    room_id=room_id,
+                    action="join",
+                )
+                joined = True
+                break
+            except Exception:
+                # We're going to retry, but we should log the error
+                logger.exception("Error auto-accepting invite on attempt %d" % attempt)
+                yield self.clock.sleep(1)
+        if not joined:
+            logger.error("Giving up on trying to auto-accept invite: too many attempts")
+
    @defer.inlineCallbacks
    def do_remotely_reject_invite(self, target_hosts, room_id, user_id):
        origin, event = yield self._make_and_verify_event(
@@ -126,7 +126,6 @@ class RegistrationHandler(BaseHandler):
        make_guest=False,
        admin=False,
        threepid=None,
-        default_display_name=None,
    ):
        """Registers a new client on the server.

@@ -141,8 +140,6 @@ class RegistrationHandler(BaseHandler):
              since it offers no means of associating a device_id with the
              access_token. Instead you should call auth_handler.issue_access_token
              after registration.
-            default_display_name (unicode|None): if set, the new user's displayname
-              will be set to this. Defaults to 'localpart'.
        Returns:
            A tuple of (user_id, access_token).
        Raises:
@@ -172,13 +169,6 @@ class RegistrationHandler(BaseHandler):
            user = UserID(localpart, self.hs.hostname)
            user_id = user.to_string()

-            if was_guest:
-                # If the user was a guest then they already have a profile
-                default_display_name = None
-
-            elif default_display_name is None:
-                default_display_name = localpart
-
            token = None
            if generate_token:
                token = self.macaroon_gen.generate_access_token(user_id)
@@ -188,7 +178,10 @@ class RegistrationHandler(BaseHandler):
                password_hash=password_hash,
                was_guest=was_guest,
                make_guest=make_guest,
-                create_profile_with_displayname=default_display_name,
+                create_profile_with_localpart=(
+                    # If the user was a guest then they already have a profile
+                    None if was_guest else user.localpart
+                ),
                admin=admin,
            )

@@ -210,15 +203,13 @@ class RegistrationHandler(BaseHandler):
                yield self.check_user_id_not_appservice_exclusive(user_id)
                if generate_token:
                    token = self.macaroon_gen.generate_access_token(user_id)
-                if default_display_name is None:
-                    default_display_name = localpart
                try:
                    yield self.store.register(
                        user_id=user_id,
                        token=token,
                        password_hash=password_hash,
                        make_guest=make_guest,
-                        create_profile_with_displayname=default_display_name,
+                        create_profile_with_localpart=user.localpart,
                    )
                except SynapseError:
                    # if user id is taken, just generate another
@@ -226,19 +217,7 @@ class RegistrationHandler(BaseHandler):
                    user_id = None
                    token = None
                    attempts += 1
-        if not self.hs.config.user_consent_at_registration:
-            yield self._auto_join_rooms(user_id)

-        defer.returnValue((user_id, token))
-
-    @defer.inlineCallbacks
-    def _auto_join_rooms(self, user_id):
-        """Automatically joins users to auto join rooms - creating the room in the first place
-        if the user is the first to be created.
-
-        Args:
-            user_id(str): The user to join
-        """
        # auto-join the user to any rooms we're supposed to dump them into
        fake_requester = create_requester(user_id)

@@ -247,6 +226,7 @@ class RegistrationHandler(BaseHandler):
        if self.hs.config.autocreate_auto_join_rooms:
            count = yield self.store.count_all_users()
            should_auto_create_rooms = count == 1
+
        for r in self.hs.config.auto_join_rooms:
            try:
                if should_auto_create_rooms:
@@ -276,15 +256,7 @@ class RegistrationHandler(BaseHandler):
            except Exception as e:
                logger.error("Failed to join new user to %r: %r", r, e)

-    @defer.inlineCallbacks
-    def post_consent_actions(self, user_id):
-        """A series of registration actions that can only be carried out once consent
-        has been granted
-
-        Args:
-            user_id (str): The user to join
-        """
-        yield self._auto_join_rooms(user_id)
+        defer.returnValue((user_id, token))

    @defer.inlineCallbacks
    def appservice_register(self, user_localpart, as_token):
@@ -309,7 +281,7 @@ class RegistrationHandler(BaseHandler):
            user_id=user_id,
            password_hash="",
            appservice_id=service_id,
-            create_profile_with_displayname=user.localpart,
+            create_profile_with_localpart=user.localpart,
        )
        defer.returnValue(user_id)

@@ -336,6 +308,35 @@ class RegistrationHandler(BaseHandler):
        else:
            logger.info("Valid captcha entered from %s", ip)

+    @defer.inlineCallbacks
+    def register_saml2(self, localpart):
+        """
+        Registers email_id as SAML2 Based Auth.
+        """
+        if types.contains_invalid_mxid_characters(localpart):
+            raise SynapseError(
+                400,
+                "User ID can only contain characters a-z, 0-9, or '=_-./'",
+            )
+        yield self.auth.check_auth_blocking()
+        user = UserID(localpart, self.hs.hostname)
+        user_id = user.to_string()
+
+        yield self.check_user_id_not_appservice_exclusive(user_id)
+        token = self.macaroon_gen.generate_access_token(user_id)
+        try:
+            yield self.store.register(
+                user_id=user_id,
+                token=token,
+                password_hash=None,
+                create_profile_with_localpart=user.localpart,
+            )
+        except Exception as e:
+            yield self.store.add_access_token_to_user(user_id, token)
+            # Ignore Registration errors
+            logger.exception(e)
+        defer.returnValue((user_id, token))
+
    @defer.inlineCallbacks
    def register_email(self, threepidCreds):
        """
@@ -487,7 +488,7 @@ class RegistrationHandler(BaseHandler):
                user_id=user_id,
                token=token,
                password_hash=password_hash,
-                create_profile_with_displayname=user.localpart,
+                create_profile_with_localpart=user.localpart,
            )
        else:
            yield self._auth_handler.delete_access_tokens_for_user(user_id)
@@ -28,8 +28,9 @@ from twisted.internet import defer
 import synapse.server
 import synapse.types
 from synapse.api.constants import EventTypes, Membership
-from synapse.api.errors import AuthError, Codes, SynapseError
-from synapse.types import RoomID, UserID
+from synapse.api.errors import AuthError, Codes, NotFoundError, SynapseError
+from synapse.types import RoomAlias, RoomID, UserID
+from synapse.util import logcontext
 from synapse.util.async_helpers import Linearizer
 from synapse.util.distributor import user_joined_room, user_left_room

@@ -416,6 +417,10 @@ class RoomMemberHandler(object):
                ret = yield self._remote_join(
                    requester, remote_room_hosts, room_id, target, content
                )
+                logcontext.run_in_background(
+                    self._send_merged_user_invites,
+                    requester, room_id,
+                )
                defer.returnValue(ret)

        elif effective_membership_state == Membership.LEAVE:
@@ -450,8 +455,58 @@ class RoomMemberHandler(object):
            prev_events_and_hashes=prev_events_and_hashes,
            content=content,
        )
+        if effective_membership_state == Membership.JOIN:
+            logcontext.run_in_background(
+                self._send_merged_user_invites,
+                requester, room_id,
+            )
        defer.returnValue(res)

+    @defer.inlineCallbacks
+    def _send_merged_user_invites(self, requester, room_id):
+        try:
+            profile_alias = "#_profile_%s:%s" % (
+                requester.user.localpart, self.hs.hostname,
+            )
+            profile_alias = RoomAlias.from_string(profile_alias)
+            try:
+                profile_room_id, remote_room_hosts = yield self.lookup_room_alias(
+                    profile_alias,
+                )
+            except NotFoundError:
+                logger.info(
+                    "Not sending merged invites as %s does not exists",
+                    profile_alias
+                )
+                return
+
+            linked_accounts = yield self.state_handler.get_current_state(
+                room_id=profile_room_id.to_string(),
+                event_type="m.linked_accounts",
+                state_key="",
+            )
+            if not linked_accounts or not linked_accounts.content['all_children']:
+                return
+            for child_id in linked_accounts.content['all_children']:
+                child = UserID.from_string(child_id)
+                if self.hs.is_mine(child) or child_id == requester.user.to_string():
+                    # TODO: Handle auto-invite for local users (not a priority)
+                    continue
+                try:
+                    yield self.update_membership(
+                        requester=requester,
+                        target=child,
+                        room_id=room_id,
+                        action="invite",
+                    )
+                except Exception:
+                    logger.exception("Failed to invite %s to %s", child_id, room_id)
+        except Exception:
+            logger.exception(
+                "Failed to send invites to children of %s in %s",
+                requester.user.to_string(), room_id,
+            )
+
    @defer.inlineCallbacks
    def send_membership_event(
            self,
@@ -578,7 +633,7 @@ class RoomMemberHandler(object):
        mapping = yield directory_handler.get_association(room_alias)

        if not mapping:
-            raise SynapseError(404, "No such room alias")
+            raise NotFoundError("No such room alias")

        room_id = mapping["room_id"]
        servers = mapping["servers"]
@@ -50,9 +50,6 @@ class SearchHandler(BaseHandler):
            dict to be returned to the client with results of search
        """

-        if not self.hs.config.enable_search:
-            raise SynapseError(400, "Search is disabled on this homeserver")
-
        batch_group = None
        batch_group_key = None
        batch_token = None
@@ -53,6 +53,7 @@ REQUIREMENTS = {
    "pillow>=3.1.2": ["PIL"],
    "sortedcontainers>=1.4.4": ["sortedcontainers"],
    "psutil>=2.0.0": ["psutil>=2.0.0"],
+    "pysaml2>=3.0.0": ["saml2"],
    "pymacaroons-pynacl>=0.9.3": ["pymacaroons"],
    "msgpack-python>=0.4.2": ["msgpack"],
    "phonenumbers>=8.2.0": ["phonenumbers"],
@@ -68,6 +69,9 @@ REQUIREMENTS = {
 }

 CONDITIONAL_REQUIREMENTS = {
+    "web_client": {
+        "matrix_angular_sdk>=0.6.8": ["syweb>=0.6.8"],
+    },
    "email.enable_notifs": {
        "Jinja2>=2.8": ["Jinja2>=2.8"],
        "bleach>=1.4.2": ["bleach>=1.4.2"],
@@ -77,10 +81,7 @@ CONDITIONAL_REQUIREMENTS = {
    },
    "postgres": {
        "psycopg2>=2.6": ["psycopg2"]
-    },
-    "saml2": {
-        "pysaml2>=4.5.0": ["saml2"],
-    },
+    }
 }


@@ -18,17 +18,17 @@ import xml.etree.ElementTree as ET

 from six.moves import urllib

+from canonicaljson import json
+from saml2 import BINDING_HTTP_POST, config
+from saml2.client import Saml2Client
+
 from twisted.internet import defer
 from twisted.web.client import PartialDownloadError

 from synapse.api.errors import Codes, LoginError, SynapseError
 from synapse.http.server import finish_request
-from synapse.http.servlet import (
-    RestServlet,
-    parse_json_object_from_request,
-    parse_string,
-)
-from synapse.types import UserID, map_username_to_mxid_localpart
+from synapse.http.servlet import parse_json_object_from_request
+from synapse.types import UserID
 from synapse.util.msisdn import phone_number_to_msisdn

 from .base import ClientV1RestServlet, client_path_patterns
@@ -81,13 +81,15 @@ def login_id_thirdparty_from_phone(identifier):

 class LoginRestServlet(ClientV1RestServlet):
    PATTERNS = client_path_patterns("/login$")
+    SAML2_TYPE = "m.login.saml2"
    CAS_TYPE = "m.login.cas"
-    SSO_TYPE = "m.login.sso"
    TOKEN_TYPE = "m.login.token"
    JWT_TYPE = "m.login.jwt"

    def __init__(self, hs):
        super(LoginRestServlet, self).__init__(hs)
+        self.idp_redirect_url = hs.config.saml2_idp_redirect_url
+        self.saml2_enabled = hs.config.saml2_enabled
        self.jwt_enabled = hs.config.jwt_enabled
        self.jwt_secret = hs.config.jwt_secret
        self.jwt_algorithm = hs.config.jwt_algorithm
@@ -100,11 +102,9 @@ class LoginRestServlet(ClientV1RestServlet):
        flows = []
        if self.jwt_enabled:
            flows.append({"type": LoginRestServlet.JWT_TYPE})
+        if self.saml2_enabled:
+            flows.append({"type": LoginRestServlet.SAML2_TYPE})
        if self.cas_enabled:
-            flows.append({"type": LoginRestServlet.SSO_TYPE})
-
-            # we advertise CAS for backwards compat, though MSC1721 renamed it
-            # to SSO.
            flows.append({"type": LoginRestServlet.CAS_TYPE})

            # While its valid for us to advertise this login type generally,
@@ -129,8 +129,18 @@ class LoginRestServlet(ClientV1RestServlet):
    def on_POST(self, request):
        login_submission = parse_json_object_from_request(request)
        try:
-            if self.jwt_enabled and (login_submission["type"] ==
-                                     LoginRestServlet.JWT_TYPE):
+            if self.saml2_enabled and (login_submission["type"] ==
+                                       LoginRestServlet.SAML2_TYPE):
+                relay_state = ""
+                if "relay_state" in login_submission:
+                    relay_state = "&RelayState=" + urllib.parse.quote(
+                                  login_submission["relay_state"])
+                result = {
+                    "uri": "%s%s" % (self.idp_redirect_url, relay_state)
+                }
+                defer.returnValue((200, result))
+            elif self.jwt_enabled and (login_submission["type"] ==
+                                       LoginRestServlet.JWT_TYPE):
                result = yield self.do_jwt_login(login_submission)
                defer.returnValue(result)
            elif login_submission["type"] == LoginRestServlet.TOKEN_TYPE:
@@ -330,11 +340,55 @@ class LoginRestServlet(ClientV1RestServlet):
        )


-class CasRedirectServlet(RestServlet):
-    PATTERNS = client_path_patterns("/login/(cas|sso)/redirect")
+class SAML2RestServlet(ClientV1RestServlet):
+    PATTERNS = client_path_patterns("/login/saml2", releases=())

    def __init__(self, hs):
-        super(CasRedirectServlet, self).__init__()
+        super(SAML2RestServlet, self).__init__(hs)
+        self.sp_config = hs.config.saml2_config_path
+        self.handlers = hs.get_handlers()
+
+    @defer.inlineCallbacks
+    def on_POST(self, request):
+        saml2_auth = None
+        try:
+            conf = config.SPConfig()
+            conf.load_file(self.sp_config)
+            SP = Saml2Client(conf)
+            saml2_auth = SP.parse_authn_request_response(
+                request.args['SAMLResponse'][0], BINDING_HTTP_POST)
+        except Exception as e:        # Not authenticated
+            logger.exception(e)
+        if saml2_auth and saml2_auth.status_ok() and not saml2_auth.not_signed:
+            username = saml2_auth.name_id.text
+            handler = self.handlers.registration_handler
+            (user_id, token) = yield handler.register_saml2(username)
+            # Forward to the RelayState callback along with ava
+            if 'RelayState' in request.args:
+                request.redirect(urllib.parse.unquote(
+                                 request.args['RelayState'][0]) +
+                                 '?status=authenticated&access_token=' +
+                                 token + '&user_id=' + user_id + '&ava=' +
+                                 urllib.quote(json.dumps(saml2_auth.ava)))
+                finish_request(request)
+                defer.returnValue(None)
+            defer.returnValue((200, {"status": "authenticated",
+                                     "user_id": user_id, "token": token,
+                                     "ava": saml2_auth.ava}))
+        elif 'RelayState' in request.args:
+            request.redirect(urllib.parse.unquote(
+                             request.args['RelayState'][0]) +
+                             '?status=not_authenticated')
+            finish_request(request)
+            defer.returnValue(None)
+        defer.returnValue((200, {"status": "not_authenticated"}))
+
+
+class CasRedirectServlet(ClientV1RestServlet):
+    PATTERNS = client_path_patterns("/login/cas/redirect", releases=())
+
+    def __init__(self, hs):
+        super(CasRedirectServlet, self).__init__(hs)
        self.cas_server_url = hs.config.cas_server_url.encode('ascii')
        self.cas_service_url = hs.config.cas_service_url.encode('ascii')

@@ -362,15 +416,17 @@ class CasTicketServlet(ClientV1RestServlet):
        self.cas_server_url = hs.config.cas_server_url
        self.cas_service_url = hs.config.cas_service_url
        self.cas_required_attributes = hs.config.cas_required_attributes
-        self._sso_auth_handler = SSOAuthHandler(hs)
+        self.auth_handler = hs.get_auth_handler()
+        self.handlers = hs.get_handlers()
+        self.macaroon_gen = hs.get_macaroon_generator()

    @defer.inlineCallbacks
    def on_GET(self, request):
-        client_redirect_url = parse_string(request, "redirectUrl", required=True)
+        client_redirect_url = request.args[b"redirectUrl"][0]
        http_client = self.hs.get_simple_http_client()
        uri = self.cas_server_url + "/proxyValidate"
        args = {
-            "ticket": parse_string(request, "ticket", required=True),
+            "ticket": request.args[b"ticket"][0].decode('ascii'),
            "service": self.cas_service_url
        }
        try:
@@ -382,6 +438,7 @@ class CasTicketServlet(ClientV1RestServlet):
        result = yield self.handle_cas_response(request, body, client_redirect_url)
        defer.returnValue(result)

+    @defer.inlineCallbacks
    def handle_cas_response(self, request, cas_response_body, client_redirect_url):
        user, attributes = self.parse_cas_response(cas_response_body)

@@ -397,9 +454,28 @@ class CasTicketServlet(ClientV1RestServlet):
                if required_value != actual_value:
                    raise LoginError(401, "Unauthorized", errcode=Codes.UNAUTHORIZED)

-        return self._sso_auth_handler.on_successful_auth(
-            user, request, client_redirect_url,
+        user_id = UserID(user, self.hs.hostname).to_string()
+        auth_handler = self.auth_handler
+        registered_user_id = yield auth_handler.check_user_exists(user_id)
+        if not registered_user_id:
+            registered_user_id, _ = (
+                yield self.handlers.registration_handler.register(localpart=user)
+            )
+
+        login_token = self.macaroon_gen.generate_short_term_login_token(
+            registered_user_id
        )
+        redirect_url = self.add_login_token_to_redirect_url(client_redirect_url,
+                                                            login_token)
+        request.redirect(redirect_url)
+        finish_request(request)
+
+    def add_login_token_to_redirect_url(self, url, token):
+        url_parts = list(urllib.parse.urlparse(url))
+        query = dict(urllib.parse.parse_qsl(url_parts[4]))
+        query.update({"loginToken": token})
+        url_parts[4] = urllib.parse.urlencode(query).encode('ascii')
+        return urllib.parse.urlunparse(url_parts)

    def parse_cas_response(self, cas_response_body):
        user = None
@@ -434,78 +510,10 @@ class CasTicketServlet(ClientV1RestServlet):
        return user, attributes


-class SSOAuthHandler(object):
-    """
-    Utility class for Resources and Servlets which handle the response from a SSO
-    service
-
-    Args:
-        hs (synapse.server.HomeServer)
-    """
-    def __init__(self, hs):
-        self._hostname = hs.hostname
-        self._auth_handler = hs.get_auth_handler()
-        self._registration_handler = hs.get_handlers().registration_handler
-        self._macaroon_gen = hs.get_macaroon_generator()
-
-    @defer.inlineCallbacks
-    def on_successful_auth(
-        self, username, request, client_redirect_url,
-        user_display_name=None,
-    ):
-        """Called once the user has successfully authenticated with the SSO.
-
-        Registers the user if necessary, and then returns a redirect (with
-        a login token) to the client.
-
-        Args:
-            username (unicode|bytes): the remote user id. We'll map this onto
-                something sane for a MXID localpath.
-
-            request (SynapseRequest): the incoming request from the browser. We'll
-                respond to it with a redirect.
-
-            client_redirect_url (unicode): the redirect_url the client gave us when
-                it first started the process.
-
-            user_display_name (unicode|None): if set, and we have to register a new user,
-                we will set their displayname to this.
-
-        Returns:
-            Deferred[none]: Completes once we have handled the request.
-        """
-        localpart = map_username_to_mxid_localpart(username)
-        user_id = UserID(localpart, self._hostname).to_string()
-        registered_user_id = yield self._auth_handler.check_user_exists(user_id)
-        if not registered_user_id:
-            registered_user_id, _ = (
-                yield self._registration_handler.register(
-                    localpart=localpart,
-                    generate_token=False,
-                    default_display_name=user_display_name,
-                )
-            )
-
-        login_token = self._macaroon_gen.generate_short_term_login_token(
-            registered_user_id
-        )
-        redirect_url = self._add_login_token_to_redirect_url(
-            client_redirect_url, login_token
-        )
-        request.redirect(redirect_url)
-        finish_request(request)
-
-    @staticmethod
-    def _add_login_token_to_redirect_url(url, token):
-        url_parts = list(urllib.parse.urlparse(url))
-        query = dict(urllib.parse.parse_qsl(url_parts[4]))
-        query.update({"loginToken": token})
-        url_parts[4] = urllib.parse.urlencode(query)
-        return urllib.parse.urlunparse(url_parts)
-
-
 def register_servlets(hs, http_server):
    LoginRestServlet(hs).register(http_server)
+    if hs.config.saml2_enabled:
+        SAML2RestServlet(hs).register(http_server)
    if hs.config.cas_enabled:
        CasRedirectServlet(hs).register(http_server)
        CasTicketServlet(hs).register(http_server)
@@ -42,7 +42,7 @@ class PushRuleRestServlet(ClientV1RestServlet):

    @defer.inlineCallbacks
    def on_PUT(self, request):
-        spec = _rule_spec_from_path([x.decode('utf8') for x in request.postpath])
+        spec = _rule_spec_from_path(request.postpath)
        try:
            priority_class = _priority_class_from_spec(spec)
        except InvalidRuleException as e:
@@ -103,7 +103,7 @@ class PushRuleRestServlet(ClientV1RestServlet):

    @defer.inlineCallbacks
    def on_DELETE(self, request):
-        spec = _rule_spec_from_path([x.decode('utf8') for x in request.postpath])
+        spec = _rule_spec_from_path(request.postpath)

        requester = yield self.auth.get_user_by_req(request)
        user_id = requester.user.to_string()
@@ -134,7 +134,7 @@ class PushRuleRestServlet(ClientV1RestServlet):

        rules = format_push_rules_for_user(requester.user, rules)

-        path = [x.decode('utf8') for x in request.postpath][1:]
+        path = request.postpath[1:]

        if path == []:
            # we're a reference impl: pedantry is our job.
@@ -142,10 +142,11 @@ class PushRuleRestServlet(ClientV1RestServlet):
                PushRuleRestServlet.SLIGHTLY_PEDANTIC_TRAILING_SLASH_ERROR
            )

-        if path[0] == '':
+        if path[0] == b'':
            defer.returnValue((200, rules))
-        elif path[0] == 'global':
-            result = _filter_ruleset_with_path(rules['global'], path[1:])
+        elif path[0] == b'global':
+            path = [x.decode('ascii') for x in path[1:]]
+            result = _filter_ruleset_with_path(rules['global'], path)
            defer.returnValue((200, result))
        else:
            raise UnrecognizedRequestError()
@@ -189,24 +190,12 @@ class PushRuleRestServlet(ClientV1RestServlet):


 def _rule_spec_from_path(path):
-    """Turn a sequence of path components into a rule spec
-
-    Args:
-        path (sequence[unicode]): the URL path components.
-
-    Returns:
-        dict: rule spec dict, containing scope/template/rule_id entries,
-            and possibly attr.
-
-    Raises:
-        UnrecognizedRequestError if the path components cannot be parsed.
-    """
    if len(path) < 2:
        raise UnrecognizedRequestError()
-    if path[0] != 'pushrules':
+    if path[0] != b'pushrules':
        raise UnrecognizedRequestError()

-    scope = path[1]
+    scope = path[1].decode('ascii')
    path = path[2:]
    if scope != 'global':
        raise UnrecognizedRequestError()
@@ -214,13 +203,13 @@ def _rule_spec_from_path(path):
    if len(path) == 0:
        raise UnrecognizedRequestError()

-    template = path[0]
+    template = path[0].decode('ascii')
    path = path[1:]

    if len(path) == 0 or len(path[0]) == 0:
        raise UnrecognizedRequestError()

-    rule_id = path[0]
+    rule_id = path[0].decode('ascii')

    spec = {
        'scope': scope,
@@ -231,7 +220,7 @@ def _rule_spec_from_path(path):
    path = path[1:]

    if len(path) > 0 and len(path[0]) > 0:
-        spec['attr'] = path[0]
+        spec['attr'] = path[0].decode('ascii')

    return spec

@@ -142,7 +142,7 @@ class PushersRemoveRestServlet(RestServlet):
    To allow pusher to be delete by clicking a link (ie. GET request)
    """
    PATTERNS = client_path_patterns("/pushers/remove$")
-    SUCCESS_HTML = b"<html><body>You have been unsubscribed</body><html>"
+    SUCCESS_HTML = "<html><body>You have been unsubscribed</body><html>"

    def __init__(self, hs):
        super(PushersRemoveRestServlet, self).__init__()
@@ -457,7 +457,6 @@ class RegisterRestServlet(RestServlet):
            yield self.store.user_set_consent_version(
                registered_user_id, self.hs.config.user_consent_version,
            )
-            yield self.registration_handler.post_consent_actions(registered_user_id)

        defer.returnValue((200, return_dict))

@@ -89,7 +89,6 @@ class ConsentResource(Resource):

        self.hs = hs
        self.store = hs.get_datastore()
-        self.registration_handler = hs.get_handlers().registration_handler

        # this is required by the request_handler wrapper
        self.clock = hs.get_clock()
@@ -200,7 +199,6 @@ class ConsentResource(Resource):
            if e.code != 404:
                raise
            raise NotFoundError("Unknown user")
-        yield self.registration_handler.post_consent_actions(qualified_user_id)

        try:
            self._render_template(request, "success.html")
@@ -41,7 +41,7 @@ class MediaConfigResource(Resource):
    @defer.inlineCallbacks
    def _async_render_GET(self, request):
        yield self.auth.get_user_by_req(request)
-        respond_with_json(request, 200, self.limits_dict, send_cors=True)
+        respond_with_json(request, 200, self.limits_dict)

    def render_OPTIONS(self, request):
        respond_with_json(request, 200, {}, send_cors=True)
@@ -48,8 +48,7 @@ class DownloadResource(Resource):
        set_cors_headers(request)
        request.setHeader(
            b"Content-Security-Policy",
-            b"sandbox;"
-            b" default-src 'none';"
+            b"default-src 'none';"
            b" script-src 'none';"
            b" plugin-types application/pdf;"
            b" style-src 'unsafe-inline';"
@@ -1,36 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2018 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-
-import saml2.metadata
-
-from twisted.web.resource import Resource
-
-
-class SAML2MetadataResource(Resource):
-    """A Twisted web resource which renders the SAML metadata"""
-
-    isLeaf = 1
-
-    def __init__(self, hs):
-        Resource.__init__(self)
-        self.sp_config = hs.config.saml2_sp_config
-
-    def render_GET(self, request):
-        metadata_xml = saml2.metadata.create_metadata_string(
-            configfile=None, config=self.sp_config,
-        )
-        request.setHeader(b"Content-Type", b"text/xml; charset=utf-8")
-        return metadata_xml
@@ -1,74 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Copyright 2018 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-import logging
-
-import saml2
-from saml2.client import Saml2Client
-
-from twisted.web.resource import Resource
-from twisted.web.server import NOT_DONE_YET
-
-from synapse.api.errors import CodeMessageException
-from synapse.http.server import wrap_html_request_handler
-from synapse.http.servlet import parse_string
-from synapse.rest.client.v1.login import SSOAuthHandler
-
-logger = logging.getLogger(__name__)
-
-
-class SAML2ResponseResource(Resource):
-    """A Twisted web resource which handles the SAML response"""
-
-    isLeaf = 1
-
-    def __init__(self, hs):
-        Resource.__init__(self)
-
-        self._saml_client = Saml2Client(hs.config.saml2_sp_config)
-        self._sso_auth_handler = SSOAuthHandler(hs)
-
-    def render_POST(self, request):
-        self._async_render_POST(request)
-        return NOT_DONE_YET
-
-    @wrap_html_request_handler
-    def _async_render_POST(self, request):
-        resp_bytes = parse_string(request, 'SAMLResponse', required=True)
-        relay_state = parse_string(request, 'RelayState', required=True)
-
-        try:
-            saml2_auth = self._saml_client.parse_authn_request_response(
-                resp_bytes, saml2.BINDING_HTTP_POST,
-            )
-        except Exception as e:
-            logger.warning("Exception parsing SAML2 response", exc_info=1)
-            raise CodeMessageException(
-                400, "Unable to parse SAML2 response: %s" % (e,),
-            )
-
-        if saml2_auth.not_signed:
-            raise CodeMessageException(400, "SAML2 response was not signed")
-
-        if "uid" not in saml2_auth.ava:
-            raise CodeMessageException(400, "uid not in SAML2 response")
-
-        username = saml2_auth.ava["uid"][0]
-
-        displayName = saml2_auth.ava.get("displayName", [None])[0]
-        return self._sso_auth_handler.on_successful_auth(
-            username, request, relay_state,
-            user_display_name=displayName,
-        )
@@ -1,70 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2018 New Vector Ltd.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import json
-import logging
-
-from twisted.web.resource import Resource
-
-logger = logging.getLogger(__name__)
-
-
-class WellKnownBuilder(object):
-    """Utility to construct the well-known response
-
-    Args:
-        hs (synapse.server.HomeServer):
-    """
-    def __init__(self, hs):
-        self._config = hs.config
-
-    def get_well_known(self):
-        # if we don't have a public_base_url, we can't help much here.
-        if self._config.public_baseurl is None:
-            return None
-
-        result = {
-            "m.homeserver": {
-                "base_url": self._config.public_baseurl,
-            },
-        }
-
-        if self._config.default_identity_server:
-            result["m.identity_server"] = {
-                "base_url": self._config.default_identity_server,
-            }
-
-        return result
-
-
-class WellKnownResource(Resource):
-    """A Twisted web resource which renders the .well-known file"""
-
-    isLeaf = 1
-
-    def __init__(self, hs):
-        Resource.__init__(self)
-        self._well_known_builder = WellKnownBuilder(hs)
-
-    def render_GET(self, request):
-        r = self._well_known_builder.get_well_known()
-        if not r:
-            request.setResponseCode(404)
-            request.setHeader(b"Content-Type", b"text/plain")
-            return b'.well-known not available'
-
-        logger.error("returning: %s", r)
-        request.setHeader(b"Content-Type", b"application/json")
-        return json.dumps(r).encode("utf-8")
@@ -298,8 +298,6 @@ def _resolve_normal_events(events, auth_events):

 def _ordered_events(events):
    def key_func(e):
-        # we have to use utf-8 rather than ascii here because it turns out we allow
-        # people to send us events with non-ascii event IDs :/
-        return -int(e.depth), hashlib.sha1(e.event_id.encode('utf-8')).hexdigest()
+        return -int(e.depth), hashlib.sha1(e.event_id.encode('ascii')).hexdigest()

    return sorted(events, key=key_func)
@@ -12,30 +12,35 @@
        <h1>Log in with one of the following methods</h1>

        <span id="feedback" style="color: #f00"></span>
+        <br/>
+        <br/>

        <div id="loading">
            <img src="spinner.gif" />
        </div>

-        <div id="sso_flow" class="login_flow" style="display:none">
-            Single-sign on:
-            <form id="sso_form" action="/_matrix/client/r0/login/sso/redirect" method="get">
-                <input id="sso_redirect_url" type="hidden" name="redirectUrl" value=""/>
-                <input type="submit" value="Log in"/>
-            </form>
+        <div id="cas_flow" class="login_flow" style="display:none"
+                onclick="gotoCas(); return false;">
+            CAS Authentication: <button id="cas_button" style="margin: 10px">Log in</button>
        </div>

-        <div id="password_flow" class="login_flow" style="display:none">
-            Password Authentication:
-            <form onsubmit="matrixLogin.password_login(); return false;">
-                <input id="user_id" size="32" type="text" placeholder="Matrix ID (e.g. bob)" autocapitalize="off" autocorrect="off" />
-                <br/>
-                <input id="password" size="32" type="password" placeholder="Password"/>
-                <br/>
+        <br/>

-                <input type="submit" value="Log in"/>
-            </form>
-        </div>
+        <form id="password_form" class="login_flow" style="display:none"
+                onsubmit="matrixLogin.password_login(); return false;">
+            <div>
+                Password Authentication:<br/>
+
+                <div style="text-align: center">
+                    <input id="user_id" size="32" type="text" placeholder="Matrix ID (e.g. bob)" autocapitalize="off" autocorrect="off" />
+                    <br/>
+                    <input id="password" size="32" type="password" placeholder="Password"/>
+                    <br/>
+
+                    <button type="submit" style="margin: 10px">Log in</button>
+                </div>
+            </div>
+        </form>

        <div id="no_login_types" type="button" class="login_flow" style="display:none">
            Log in currently unavailable.
@@ -1,8 +1,7 @@
 window.matrixLogin = {
-    endpoint: location.origin + "/_matrix/client/r0/login",
+    endpoint: location.origin + "/_matrix/client/api/v1/login",
    serverAcceptsPassword: false,
-    serverAcceptsCas: false,
-    serverAcceptsSso: false,
+    serverAcceptsCas: false
 };

 var submitPassword = function(user, pwd) {
@@ -41,6 +40,12 @@ var errorFunc = function(err) {
    }
 };

+var gotoCas = function() {
+    var this_page = window.location.origin + window.location.pathname;
+    var redirect_url = matrixLogin.endpoint + "/cas/redirect?redirectUrl=" + encodeURIComponent(this_page);
+    window.location.replace(redirect_url);
+}
+
 var setFeedbackString = function(text) {
    $("#feedback").text(text);
 };
@@ -48,18 +53,12 @@ var setFeedbackString = function(text) {
 var show_login = function() {
    $("#loading").hide();

-    var this_page = window.location.origin + window.location.pathname;
-    $("#sso_redirect_url").val(encodeURIComponent(this_page));
-
    if (matrixLogin.serverAcceptsPassword) {
-        $("#password_flow").show();
+        $("#password_form").show();
    }

-    if (matrixLogin.serverAcceptsSso) {
-        $("#sso_flow").show();
-    } else if (matrixLogin.serverAcceptsCas) {
-        $("#sso_form").attr("action", "/_matrix/client/r0/login/cas/redirect");
-        $("#sso_flow").show();
+    if (matrixLogin.serverAcceptsCas) {
+        $("#cas_flow").show();
    }

    if (!matrixLogin.serverAcceptsPassword && !matrixLogin.serverAcceptsCas) {
@@ -68,8 +67,8 @@ var show_login = function() {
 };

 var show_spinner = function() {
-    $("#password_flow").hide();
-    $("#sso_flow").hide();
+    $("#password_form").hide();
+    $("#cas_flow").hide();
    $("#no_login_types").hide();
    $("#loading").show();
 };
@@ -85,10 +84,7 @@ var fetch_info = function(cb) {
                matrixLogin.serverAcceptsCas = true;
                console.log("Server accepts CAS");
            }
-            if ("m.login.sso" === flow.type) {
-                matrixLogin.serverAcceptsSso = true;
-                console.log("Server accepts SSO");
-            }
+
            if ("m.login.password" === flow.type) {
                matrixLogin.serverAcceptsPassword = true;
                console.log("Server accepts password");
@@ -19,23 +19,30 @@ a:hover   { color: #000; }
 a:active  { color: #000; }

 input {
+   width: 90%
+}
+
+textarea, input {
+   font-family: inherit;
+   font-size: inherit;
   margin: 5px;
 }

-textbox, input[type="text"], input[type="password"] {
-   width: 90%;
+.smallPrint {
+    color: #888;
+    font-size: 9pt ! important;
+    font-style: italic ! important;
 }

-form {
-    text-align: center;
-    margin: 10px 0 0 0;
+.g-recaptcha div {
+    margin: auto;
 }

 .login_flow {
-    width: 300px;
    text-align: left;
    padding: 10px;
    margin-bottom: 40px;
+    display: inline-block;

    -webkit-border-radius: 10px;
    -moz-border-radius: 10px;
@@ -1,63 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-  <head>
-   <title>Synapse is running</title>
-   <style>
-       body {
-         font-family: -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;
-         max-width: 40em;
-         margin: auto;
-         text-align: center;
-       }
-       h1, p {
-         margin: 1.5em;
-       }
-       hr {
-        border: none;
-        background-color: #ccc;
-        color: #ccc;
-        height: 1px;
-        width: 7em;
-        margin-top: 4em;
-      }
-       .logo {
-         display: block;
-         width: 12em;
-         margin: 4em auto;
-       }
-   </style>
-  </head>
-  <body>
-    <div class="logo">
-       <svg role="img" aria-label="[Matrix logo]" viewBox="0 0 200 85" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
-          <g id="parent" stroke="none" stroke-width="1" fill="none" fill-rule="evenodd">
-              <g id="child" transform="translate(-122.000000, -6.000000)" fill="#000000" fill-rule="nonzero">
-                  <g id="matrix-logo" transform="translate(122.000000, 6.000000)">
-                      <polygon id="left-bracket" points="2.24708861 1.93811009 2.24708861 82.7268844 8.10278481 82.7268844 8.10278481 84.6652459 0 84.6652459 0 0 8.10278481 0 8.10278481 1.93811009"></polygon>
-                      <path d="M24.8073418,27.5493174 L24.8073418,31.6376991 L24.924557,31.6376991 C26.0227848,30.0814294 27.3455696,28.8730642 28.8951899,28.0163743 C30.4437975,27.1611927 32.2189873,26.7318422 34.218481,26.7318422 C36.1394937,26.7318422 37.8946835,27.102622 39.4825316,27.8416679 C41.0708861,28.5819706 42.276962,29.8856073 43.1005063,31.7548404 C44.0017722,30.431345 45.2270886,29.2629486 46.7767089,28.2506569 C48.3253165,27.2388679 50.158481,26.7318422 52.2764557,26.7318422 C53.8843038,26.7318422 55.3736709,26.9269101 56.7473418,27.3162917 C58.1189873,27.7056734 59.295443,28.3285835 60.2759494,29.185022 C61.255443,30.0422147 62.02,31.1615927 62.5701266,32.5426532 C63.1187342,33.9262275 63.3936709,35.5898349 63.3936709,37.5372459 L63.3936709,57.7443688 L55.0410127,57.7441174 L55.0410127,40.6319376 C55.0410127,39.6201486 55.0020253,38.6661761 54.9232911,37.7700202 C54.8440506,36.8751211 54.6293671,36.0968606 54.2764557,35.4339817 C53.9232911,34.772611 53.403038,34.2464807 52.7177215,33.8568477 C52.0313924,33.4689743 51.0997468,33.2731523 49.9235443,33.2731523 C48.7473418,33.2731523 47.7962025,33.4983853 47.0706329,33.944578 C46.344557,34.393033 45.7764557,34.9774826 45.3650633,35.6969211 C44.9534177,36.4181193 44.6787342,37.2353431 44.5417722,38.150855 C44.4037975,39.0653615 44.3356962,39.9904257 44.3356962,40.9247908 L44.3356962,57.7443688 L35.9835443,57.7443688 L35.9835443,40.8079009 C35.9835443,39.9124991 35.963038,39.0263982 35.9253165,38.150855 C35.8853165,37.2743064 35.7192405,36.4666349 35.424557,35.7263321 C35.1303797,34.9872862 34.64,34.393033 33.9539241,33.944578 C33.2675949,33.4983853 32.2579747,33.2731523 30.9248101,33.2731523 C30.5321519,33.2731523 30.0126582,33.3608826 29.3663291,33.5365945 C28.7192405,33.7118037 28.0913924,34.0433688 27.4840506,34.5292789 C26.875443,35.0164459 26.3564557,35.7172826 25.9250633,36.6315376 C25.4934177,37.5470495 25.2779747,38.7436 25.2779747,40.2229486 L25.2779747,57.7441174 L16.9260759,57.7443688 L16.9260759,27.5493174 L24.8073418,27.5493174 Z" id="m"></path>
-                      <path d="M68.7455696,31.9886202 C69.6075949,30.7033339 70.7060759,29.672189 72.0397468,28.8926716 C73.3724051,28.1141596 74.8716456,27.5596239 76.5387342,27.2283101 C78.2050633,26.8977505 79.8817722,26.7315908 81.5678481,26.7315908 C83.0974684,26.7315908 84.6458228,26.8391798 86.2144304,27.0525982 C87.7827848,27.2675248 89.2144304,27.6865688 90.5086076,28.3087248 C91.8025316,28.9313835 92.8610127,29.7983798 93.6848101,30.9074514 C94.5083544,32.0170257 94.92,33.4870734 94.92,35.3173431 L94.92,51.026844 C94.92,52.3913138 94.998481,53.6941963 95.1556962,54.9400165 C95.3113924,56.1865908 95.5863291,57.120956 95.9787342,57.7436147 L87.5091139,57.7436147 C87.3518987,57.276055 87.2240506,56.7996972 87.1265823,56.3125303 C87.0278481,55.8266202 86.9592405,55.3301523 86.9207595,54.8236294 C85.5873418,56.1865908 84.0182278,57.1405633 82.2156962,57.6857982 C80.4113924,58.2295248 78.5683544,58.503022 76.6860759,58.503022 C75.2346835,58.503022 73.8817722,58.3275615 72.6270886,57.9776459 C71.3718987,57.6269761 70.2744304,57.082244 69.3334177,56.3411872 C68.3921519,55.602644 67.656962,54.6680275 67.1275949,53.5390972 C66.5982278,52.410167 66.3331646,51.065556 66.3331646,49.5087835 C66.3331646,47.7961578 66.6367089,46.384178 67.2455696,45.2756092 C67.8529114,44.1652807 68.6367089,43.2799339 69.5987342,42.6173064 C70.5589873,41.9556844 71.6567089,41.4592165 72.8924051,41.1284055 C74.1273418,40.7978459 75.3721519,40.5356606 76.6270886,40.3398385 C77.8820253,40.1457761 79.116962,39.9896716 80.3329114,39.873033 C81.5483544,39.7558917 82.6270886,39.5804312 83.5681013,39.3469028 C84.5093671,39.1133743 85.2536709,38.7732624 85.8032911,38.3250587 C86.3513924,37.8773578 86.6063291,37.2252881 86.5678481,36.3680954 C86.5678481,35.4731963 86.4210127,34.7620532 86.1268354,34.2366771 C85.8329114,33.7113009 85.4405063,33.3018092 84.9506329,33.0099615 C84.4602532,32.7181138 83.8916456,32.5232972 83.2450633,32.4255119 C82.5977215,32.3294862 81.9010127,32.2797138 81.156962,32.2797138 C79.5098734,32.2797138 78.2159494,32.6303835 77.2746835,33.3312202 C76.3339241,34.0320569 75.7837975,35.2007046 75.6275949,36.8354037 L67.275443,36.8354037 C67.3924051,34.8892495 67.8817722,33.2726495 68.7455696,31.9886202 Z M85.2440506,43.6984752 C84.7149367,43.873433 84.1460759,44.0189798 83.5387342,44.1361211 C82.9306329,44.253011 82.2936709,44.350545 81.6270886,44.4279688 C80.96,44.5066495 80.2934177,44.6034294 79.6273418,44.7203193 C78.9994937,44.8362037 78.3820253,44.9933138 77.7749367,45.1871248 C77.1663291,45.3829468 76.636962,45.6451321 76.1865823,45.9759431 C75.7349367,46.3070055 75.3724051,46.7263009 75.0979747,47.2313156 C74.8232911,47.7375872 74.6863291,48.380356 74.6863291,49.1588679 C74.6863291,49.8979138 74.8232911,50.5218294 75.0979747,51.026844 C75.3724051,51.5338697 75.7455696,51.9328037 76.2159494,52.2246514 C76.6863291,52.5164991 77.2349367,52.7213706 77.8632911,52.8375064 C78.4898734,52.9546477 79.136962,53.012967 79.8037975,53.012967 C81.4506329,53.012967 82.724557,52.740978 83.6273418,52.1952404 C84.5288608,51.6507596 85.1949367,50.9981872 85.6270886,50.2382771 C86.0579747,49.4793725 86.323038,48.7119211 86.4212658,47.9321523 C86.518481,47.1536404 86.5681013,46.5304789 86.5681013,46.063422 L86.5681013,42.9677248 C86.2146835,43.2799339 85.7736709,43.5230147 85.2440506,43.6984752 Z" id="a"></path>
-                      <path d="M116.917975,27.5493174 L116.917975,33.0976917 L110.801266,33.0976917 L110.801266,48.0492936 C110.801266,49.4502128 111.036203,50.3850807 111.507089,50.8518862 C111.976962,51.3191945 112.918734,51.5527229 114.33038,51.5527229 C114.801013,51.5527229 115.251392,51.5336183 115.683038,51.4944037 C116.114177,51.4561945 116.526076,51.3968697 116.917975,51.3194459 L116.917975,57.7438661 C116.212152,57.860756 115.427595,57.9381798 114.565316,57.9778972 C113.702785,58.0153523 112.859747,58.0357138 112.036203,58.0357138 C110.742278,58.0357138 109.516456,57.9477321 108.36,57.7722716 C107.202785,57.5975651 106.183544,57.2577046 105.301519,56.7509303 C104.418987,56.2454128 103.722785,55.5242147 103.213418,54.5898495 C102.703038,53.6562385 102.448608,52.4292716 102.448608,50.9099541 L102.448608,33.0976917 L97.3903797,33.0976917 L97.3903797,27.5493174 L102.448608,27.5493174 L102.448608,18.4967596 L110.801013,18.4967596 L110.801013,27.5493174 L116.917975,27.5493174 Z" id="t"></path>
-                      <path d="M128.857975,27.5493174 L128.857975,33.1565138 L128.975696,33.1565138 C129.367089,32.2213945 129.896203,31.3559064 130.563544,30.557033 C131.23038,29.7596679 131.99443,29.0776844 132.857215,28.5130936 C133.719241,27.9495083 134.641266,27.5113596 135.622532,27.1988991 C136.601772,26.8879468 137.622025,26.7315908 138.681013,26.7315908 C139.229873,26.7315908 139.836962,26.8296275 140.504304,27.0239413 L140.504304,34.7336477 C140.111646,34.6552183 139.641013,34.586844 139.092658,34.5290275 C138.543291,34.4704569 138.014177,34.4410459 137.504304,34.4410459 C135.974937,34.4410459 134.681013,34.6949358 133.622785,35.2004532 C132.564051,35.7067248 131.711392,36.397255 131.064051,37.2735523 C130.417215,38.1501009 129.955443,39.1714422 129.681266,40.3398385 C129.407089,41.5074807 129.269873,42.7736624 129.269873,44.1361211 L129.269873,57.7438661 L120.917722,57.7438661 L120.917722,27.5493174 L128.857975,27.5493174 Z" id="r"></path>
-                      <path d="M144.033165,22.8767376 L144.033165,16.0435798 L152.386076,16.0435798 L152.386076,22.8767376 L144.033165,22.8767376 Z M152.386076,27.5493174 L152.386076,57.7438661 L144.033165,57.7438661 L144.033165,27.5493174 L152.386076,27.5493174 Z" id="i"></path>
-                      <polygon id="x" points="156.738228 27.5493174 166.266582 27.5493174 171.619494 35.4337303 176.913418 27.5493174 186.147848 27.5493174 176.148861 41.6831927 187.383544 57.7441174 177.85443 57.7441174 171.501772 48.2245028 165.148861 57.7441174 155.797468 57.7441174 166.737468 41.8589046"></polygon>
-                      <polygon id="right-bracket" points="197.580759 82.7268844 197.580759 1.93811009 191.725063 1.93811009 191.725063 0 199.828354 0 199.828354 84.6652459 191.725063 84.6652459 191.725063 82.7268844"></polygon>
-                  </g>
-              </g>
-          </g>
-      </svg>
-    </div>
-    <h1>It works! Synapse is running</h1>
-    <p>Your Synapse server is listening on this port and is ready for messages.</p>
-    <p>To use this server you'll need <a href="https://matrix.org/docs/projects/try-matrix-now.html#clients" target="_blank">a Matrix client</a>.
-    </p>
-    <p>Welcome to the Matrix universe :)</p>
-    <hr>
-    <p>
-      <small>
-        <a href="https://matrix.org" target="_blank">
-          matrix.org
-        </a>
-      </small>
-    </p>
-  </body>
-</html>
@@ -14,10 +14,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-import calendar
+import datetime
 import logging
 import time

+from dateutil import tz
+
 from synapse.api.constants import PresenceState
 from synapse.storage.devices import DeviceStore
 from synapse.storage.user_erasure_store import UserErasureStore
@@ -117,6 +119,7 @@ class DataStore(RoomMemberStore, RoomStore,
            db_conn, "device_lists_stream", "stream_id",
        )

+        self._transaction_id_gen = IdGenerator(db_conn, "sent_transactions", "id")
        self._access_tokens_id_gen = IdGenerator(db_conn, "access_tokens", "id")
        self._event_reports_id_gen = IdGenerator(db_conn, "event_reports", "id")
        self._push_rule_id_gen = IdGenerator(db_conn, "push_rules", "id")
@@ -355,11 +358,10 @@ class DataStore(RoomMemberStore, RoomStore,
        """
        Returns millisecond unixtime for start of UTC day.
        """
-        now = time.gmtime()
-        today_start = calendar.timegm((
-            now.tm_year, now.tm_mon, now.tm_mday, 0, 0, 0,
-        ))
-        return today_start * 1000
+        now = datetime.datetime.utcnow()
+        today_start = datetime.datetime(now.year, now.month,
+                                        now.day, tzinfo=tz.tzutc())
+        return int(time.mktime(today_start.timetuple())) * 1000

    def generate_user_daily_visits(self):
        """
@@ -29,7 +29,6 @@ from synapse.api.errors import StoreError
 from synapse.storage.engines import PostgresEngine
 from synapse.util.caches.descriptors import Cache
 from synapse.util.logcontext import LoggingContext, PreserveLoggingContext
-from synapse.util.stringutils import exception_to_unicode

 logger = logging.getLogger(__name__)

@@ -250,32 +249,32 @@ class SQLBaseStore(object):
                except self.database_engine.module.OperationalError as e:
                    # This can happen if the database disappears mid
                    # transaction.
-                    logger.warning(
+                    logger.warn(
                        "[TXN OPERROR] {%s} %s %d/%d",
-                        name, exception_to_unicode(e), i, N
+                        name, e, i, N
                    )
                    if i < N:
                        i += 1
                        try:
                            conn.rollback()
                        except self.database_engine.module.Error as e1:
-                            logger.warning(
+                            logger.warn(
                                "[TXN EROLL] {%s} %s",
-                                name, exception_to_unicode(e1),
+                                name, e1,
                            )
                        continue
                    raise
                except self.database_engine.module.DatabaseError as e:
                    if self.database_engine.is_deadlock(e):
-                        logger.warning("[TXN DEADLOCK] {%s} %d/%d", name, i, N)
+                        logger.warn("[TXN DEADLOCK] {%s} %d/%d", name, i, N)
                        if i < N:
                            i += 1
                            try:
                                conn.rollback()
                            except self.database_engine.module.Error as e1:
-                                logger.warning(
+                                logger.warn(
                                    "[TXN EROLL] {%s} %s",
-                                    name, exception_to_unicode(e1),
+                                    name, e1,
                                )
                            continue
                    raise
@@ -850,9 +849,9 @@ class SQLBaseStore(object):
        rowcount = cls._simple_update_txn(txn, table, keyvalues, updatevalues)

        if rowcount == 0:
-            raise StoreError(404, "No row found (%s)" % (table,))
+            raise StoreError(404, "No row found")
        if rowcount > 1:
-            raise StoreError(500, "More than one row matched (%s)" % (table,))
+            raise StoreError(500, "More than one row matched")

    @staticmethod
    def _simple_select_one_txn(txn, table, keyvalues, retcols,
@@ -869,9 +868,9 @@ class SQLBaseStore(object):
        if not row:
            if allow_none:
                return None
-            raise StoreError(404, "No row found (%s)" % (table,))
+            raise StoreError(404, "No row found")
        if txn.rowcount > 1:
-            raise StoreError(500, "More than one row matched (%s)" % (table,))
+            raise StoreError(500, "More than one row matched")

        return dict(zip(retcols, row))

@@ -903,9 +902,9 @@ class SQLBaseStore(object):

        txn.execute(sql, list(keyvalues.values()))
        if txn.rowcount == 0:
-            raise StoreError(404, "No row found (%s)" % (table,))
+            raise StoreError(404, "No row found")
        if txn.rowcount > 1:
-            raise StoreError(500, "More than one row matched (%s)" % (table,))
+            raise StoreError(500, "more than one row matched")

    def _simple_delete(self, table, keyvalues, desc):
        return self.runInteraction(
@@ -34,9 +34,8 @@ class MonthlyActiveUsersStore(SQLBaseStore):
        self.hs = hs
        self.reserved_users = ()
        # Do not add more reserved users than the total allowable number
-        self._new_transaction(
-            dbconn, "initialise_mau_threepids", [], [],
-            self._initialise_reserved_users,
+        self._initialise_reserved_users(
+            dbconn.cursor(),
            hs.config.mau_limits_reserved_threepids[:self.hs.config.max_mau_value],
        )

@@ -25,7 +25,7 @@ logger = logging.getLogger(__name__)

 # Remember to update this number every time a change is made to database
 # schema files, so the users will be informed on server restarts.
-SCHEMA_VERSION = 53
+SCHEMA_VERSION = 52

 dir_path = os.path.abspath(os.path.dirname(__file__))

@@ -257,7 +257,7 @@ def _upgrade_existing_database(cur, current_version, applied_delta_files,
                module.run_create(cur, database_engine)
                if not is_empty:
                    module.run_upgrade(cur, database_engine, config=config)
-            elif ext == ".pyc" or file_name == "__pycache__":
+            elif ext == ".pyc":
                # Sometimes .pyc files turn up anyway even though we've
                # disabled their generation; e.g. from distribution package
                # installers. Silently skip it
@@ -22,7 +22,6 @@ from twisted.internet import defer
 from synapse.api.errors import Codes, StoreError
 from synapse.storage import background_updates
 from synapse.storage._base import SQLBaseStore
-from synapse.types import UserID
 from synapse.util.caches.descriptors import cached, cachedInlineCallbacks


@@ -168,7 +167,7 @@ class RegistrationStore(RegistrationWorkerStore,

    def register(self, user_id, token=None, password_hash=None,
                 was_guest=False, make_guest=False, appservice_id=None,
-                 create_profile_with_displayname=None, admin=False):
+                 create_profile_with_localpart=None, admin=False):
        """Attempts to register an account.

        Args:
@@ -182,8 +181,8 @@ class RegistrationStore(RegistrationWorkerStore,
            make_guest (boolean): True if the the new user should be guest,
                false to add a regular user account.
            appservice_id (str): The ID of the appservice registering the user.
-            create_profile_with_displayname (unicode): Optionally create a profile for
-                the user, setting their displayname to the given value
+            create_profile_with_localpart (str): Optionally create a profile for
+                the given localpart.
        Raises:
            StoreError if the user_id could not be registered.
        """
@@ -196,7 +195,7 @@ class RegistrationStore(RegistrationWorkerStore,
            was_guest,
            make_guest,
            appservice_id,
-            create_profile_with_displayname,
+            create_profile_with_localpart,
            admin
        )

@@ -209,11 +208,9 @@ class RegistrationStore(RegistrationWorkerStore,
        was_guest,
        make_guest,
        appservice_id,
-        create_profile_with_displayname,
+        create_profile_with_localpart,
        admin,
    ):
-        user_id_obj = UserID.from_string(user_id)
-
        now = int(self.clock.time())

        next_id = self._access_tokens_id_gen.get_next()
@@ -276,15 +273,12 @@ class RegistrationStore(RegistrationWorkerStore,
                (next_id, user_id, token,)
            )

-        if create_profile_with_displayname:
+        if create_profile_with_localpart:
            # set a default displayname serverside to avoid ugly race
            # between auto-joins and clients trying to set displaynames
-            #
-            # *obviously* the 'profiles' table uses localpart for user_id
-            # while everything else uses the full mxid.
            txn.execute(
                "INSERT INTO profiles(user_id, displayname) VALUES (?,?)",
-                (user_id_obj.localpart, create_profile_with_displayname)
+                (create_profile_with_localpart, create_profile_with_localpart)
            )

        self._invalidate_cache_and_stream(
@@ -1,4 +1,4 @@
-/* Copyright 2018 New Vector Ltd
+/* Copyright 2015, 2016 OpenMarket Ltd
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
@@ -13,4 +13,4 @@
 * limitations under the License.
 */

-DROP TABLE IF EXISTS sent_transactions;
+CREATE INDEX IF NOT EXISTS sent_transaction_txn_id ON sent_transactions(transaction_id);
@@ -1,5 +1,4 @@
-# -*- coding: utf-8 -*-
-# Copyright 2018 New Vector Ltd
+# Copyright 2016 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -12,18 +11,22 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
 import logging

-from twisted.web.resource import Resource
-
-from synapse.rest.saml2.metadata_resource import SAML2MetadataResource
-from synapse.rest.saml2.response_resource import SAML2ResponseResource
+from synapse.storage.engines import PostgresEngine

 logger = logging.getLogger(__name__)


-class SAML2Resource(Resource):
-    def __init__(self, hs):
-        Resource.__init__(self)
-        self.putChild(b"metadata.xml", SAML2MetadataResource(hs))
-        self.putChild(b"authn_response", SAML2ResponseResource(hs))
+def run_create(cur, database_engine, *args, **kwargs):
+    if isinstance(database_engine, PostgresEngine):
+        cur.execute("TRUNCATE sent_transactions")
+    else:
+        cur.execute("DELETE FROM sent_transactions")
+
+    cur.execute("CREATE INDEX sent_transactions_ts ON sent_transactions(ts)")
+
+
+def run_upgrade(cur, database_engine, *args, **kwargs):
+    pass
@@ -25,6 +25,25 @@ CREATE TABLE IF NOT EXISTS received_transactions(

 CREATE INDEX transactions_have_ref ON received_transactions(origin, has_been_referenced);-- WHERE has_been_referenced = 0;

+
+-- Stores what transactions we've sent, what their response was (if we got one) and whether we have
+-- since referenced the transaction in another outgoing transaction
+CREATE TABLE IF NOT EXISTS sent_transactions(
+    id INTEGER PRIMARY KEY AUTOINCREMENT, -- This is used to apply insertion ordering
+    transaction_id TEXT,
+    destination TEXT,
+    response_code INTEGER DEFAULT 0,
+    response_json TEXT,
+    ts BIGINT
+);
+
+CREATE INDEX sent_transaction_dest ON sent_transactions(destination);
+CREATE INDEX sent_transaction_txn_id ON sent_transactions(transaction_id);
+-- So that we can do an efficient look up of all transactions that have yet to be successfully
+-- sent.
+CREATE INDEX sent_transaction_sent ON sent_transactions(response_code);
+
+
 -- For sent transactions only.
 CREATE TABLE IF NOT EXISTS transaction_id_to_pdu(
    transaction_id INTEGER,
@@ -25,6 +25,25 @@ CREATE TABLE IF NOT EXISTS received_transactions(

 CREATE INDEX transactions_have_ref ON received_transactions(origin, has_been_referenced);-- WHERE has_been_referenced = 0;

+
+-- Stores what transactions we've sent, what their response was (if we got one) and whether we have
+-- since referenced the transaction in another outgoing transaction
+CREATE TABLE IF NOT EXISTS sent_transactions(
+    id BIGINT PRIMARY KEY, -- This is used to apply insertion ordering
+    transaction_id TEXT,
+    destination TEXT,
+    response_code INTEGER DEFAULT 0,
+    response_json TEXT,
+    ts BIGINT
+);
+
+CREATE INDEX sent_transaction_dest ON sent_transactions(destination);
+CREATE INDEX sent_transaction_txn_id ON sent_transactions(transaction_id);
+-- So that we can do an efficient look up of all transactions that have yet to be successfully
+-- sent.
+CREATE INDEX sent_transaction_sent ON sent_transactions(response_code);
+
+
 -- For sent transactions only.
 CREATE TABLE IF NOT EXISTS transaction_id_to_pdu(
    transaction_id INTEGER,
@@ -45,10 +45,6 @@ class SearchStore(BackgroundUpdateStore):

    def __init__(self, db_conn, hs):
        super(SearchStore, self).__init__(db_conn, hs)
-
-        if not hs.config.enable_search:
-            return
-
        self.register_background_update_handler(
            self.EVENT_SEARCH_UPDATE_NAME, self._background_reindex_search
        )
@@ -320,8 +316,6 @@ class SearchStore(BackgroundUpdateStore):
            entries (iterable[SearchEntry]):
                entries to be added to the table
        """
-        if not self.hs.config.enable_search:
-            return
        if isinstance(self.database_engine, PostgresEngine):
            sql = (
                "INSERT INTO event_search"
@@ -432,7 +432,7 @@ class StateGroupWorkerStore(EventsWorkerStore, SQLBaseStore):
        create_id = state_ids.get((EventTypes.Create, ""))

        if not create_id:
-            raise NotFoundError("Unknown room %s" % (room_id))
+            raise NotFoundError("Unknown room")

        create_event = yield self.get_event(create_id)
        defer.returnValue(create_event.content.get("room_version", "1"))
@@ -12,7 +12,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import re
 import string
 from collections import namedtuple

@@ -229,71 +228,6 @@ def contains_invalid_mxid_characters(localpart):
    return any(c not in mxid_localpart_allowed_characters for c in localpart)


-UPPER_CASE_PATTERN = re.compile(b"[A-Z_]")
-
-# the following is a pattern which matches '=', and bytes which are not allowed in a mxid
-# localpart.
-#
-# It works by:
-#  * building a string containing the allowed characters (excluding '=')
-#  * escaping every special character with a backslash (to stop '-' being interpreted as a
-#    range operator)
-#  * wrapping it in a '[^...]' regex
-#  * converting the whole lot to a 'bytes' sequence, so that we can use it to match
-#    bytes rather than strings
-#
-NON_MXID_CHARACTER_PATTERN = re.compile(
-    ("[^%s]" % (
-        re.escape("".join(mxid_localpart_allowed_characters - {"="}),),
-    )).encode("ascii"),
-)
-
-
-def map_username_to_mxid_localpart(username, case_sensitive=False):
-    """Map a username onto a string suitable for a MXID
-
-    This follows the algorithm laid out at
-    https://matrix.org/docs/spec/appendices.html#mapping-from-other-character-sets.
-
-    Args:
-        username (unicode|bytes): username to be mapped
-        case_sensitive (bool): true if TEST and test should be mapped
-            onto different mxids
-
-    Returns:
-        unicode: string suitable for a mxid localpart
-    """
-    if not isinstance(username, bytes):
-        username = username.encode('utf-8')
-
-    # first we sort out upper-case characters
-    if case_sensitive:
-        def f1(m):
-            return b"_" + m.group().lower()
-
-        username = UPPER_CASE_PATTERN.sub(f1, username)
-    else:
-        username = username.lower()
-
-    # then we sort out non-ascii characters
-    def f2(m):
-        g = m.group()[0]
-        if isinstance(g, str):
-            # on python 2, we need to do a ord(). On python 3, the
-            # byte itself will do.
-            g = ord(g)
-        return b"=%02x" % (g,)
-
-    username = NON_MXID_CHARACTER_PATTERN.sub(f2, username)
-
-    # we also do the =-escaping to mxids starting with an underscore.
-    username = re.sub(b'^_', b'=5f', username)
-
-    # we should now only have ascii bytes left, so can decode back to a
-    # unicode.
-    return username.decode('ascii')
-
-
 class StreamToken(
    namedtuple("Token", (
        "room_key",
@@ -16,8 +16,7 @@
 import random
 import string

-import six
-from six import PY2, PY3
+from six import PY3
 from six.moves import range

 _string_with_symbols = (
@@ -72,39 +71,3 @@ def to_ascii(s):
        return s.encode("ascii")
    except UnicodeEncodeError:
        return s
-
-
-def exception_to_unicode(e):
-    """Helper function to extract the text of an exception as a unicode string
-
-    Args:
-        e (Exception): exception to be stringified
-
-    Returns:
-        unicode
-    """
-    # urgh, this is a mess. The basic problem here is that psycopg2 constructs its
-    # exceptions with PyErr_SetString, with a (possibly non-ascii) argument. str() will
-    # then produce the raw byte sequence. Under Python 2, this will then cause another
-    # error if it gets mixed with a `unicode` object, as per
-    # https://github.com/matrix-org/synapse/issues/4252
-
-    # First of all, if we're under python3, everything is fine because it will sort this
-    # nonsense out for us.
-    if not PY2:
-        return str(e)
-
-    # otherwise let's have a stab at decoding the exception message. We'll circumvent
-    # Exception.__str__(), which would explode if someone raised Exception(u'non-ascii')
-    # and instead look at what is in the args member.
-
-    if len(e.args) == 0:
-        return u""
-    elif len(e.args) > 1:
-        return six.text_type(repr(e.args))
-
-    msg = e.args[0]
-    if isinstance(msg, bytes):
-        return msg.decode('utf-8', errors='replace')
-    else:
-        return msg
@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -16,9 +15,7 @@

 from twisted.trial import util

-import tests.patch_inline_callbacks
-
-# attempt to do the patch before we load any synapse code
-tests.patch_inline_callbacks.do_patch()
+from tests import utils

 util.DEFAULT_TIMEOUT_DURATION = 10
+utils.setupdb()
@@ -63,14 +63,6 @@ class KeyringTestCase(unittest.TestCase):
        keys = self.mock_perspective_server.get_verify_keys()
        self.hs.config.perspectives = {self.mock_perspective_server.server_name: keys}

-    def assert_sentinel_context(self):
-        if LoggingContext.current_context() != LoggingContext.sentinel:
-            self.fail(
-                "Expected sentinel context but got %s" % (
-                    LoggingContext.current_context(),
-                )
-            )
-
    def check_context(self, _, expected):
        self.assertEquals(
            getattr(LoggingContext.current_context(), "request", None), expected
@@ -78,6 +70,8 @@ class KeyringTestCase(unittest.TestCase):

    @defer.inlineCallbacks
    def test_wait_for_previous_lookups(self):
+        sentinel_context = LoggingContext.current_context()
+
        kr = keyring.Keyring(self.hs)

        lookup_1_deferred = defer.Deferred()
@@ -105,10 +99,8 @@ class KeyringTestCase(unittest.TestCase):
                ["server1"], {"server1": lookup_2_deferred}
            )
            self.assertFalse(wait_2_deferred.called)
-
            # ... so we should have reset the LoggingContext.
-            self.assert_sentinel_context()
-
+            self.assertIs(LoggingContext.current_context(), sentinel_context)
            wait_2_deferred.addBoth(self.check_context, "two")

            # let the first lookup complete (in the sentinel context)
@@ -206,6 +198,8 @@ class KeyringTestCase(unittest.TestCase):
        json1 = {}
        signedjson.sign.sign_json(json1, "server9", key1)

+        sentinel_context = LoggingContext.current_context()
+
        with LoggingContext("one") as context_one:
            context_one.request = "one"

@@ -219,7 +213,7 @@ class KeyringTestCase(unittest.TestCase):

            defer = kr.verify_json_for_server("server9", json1)
            self.assertFalse(defer.called)
-            self.assert_sentinel_context()
+            self.assertIs(LoggingContext.current_context(), sentinel_context)
            yield defer

            self.assertIs(LoggingContext.current_context(), context_one)
@@ -129,12 +129,28 @@ class RegistrationTestCase(unittest.TestCase):
        with self.assertRaises(ResourceLimitError):
            yield self.handler.register(localpart="local_part")

+    @defer.inlineCallbacks
+    def test_register_saml2_mau_blocked(self):
+        self.hs.config.limit_usage_by_mau = True
+        self.store.get_monthly_active_count = Mock(
+            return_value=defer.succeed(self.lots_of_users)
+        )
+        with self.assertRaises(ResourceLimitError):
+            yield self.handler.register_saml2(localpart="local_part")
+
+        self.store.get_monthly_active_count = Mock(
+            return_value=defer.succeed(self.hs.config.max_mau_value)
+        )
+        with self.assertRaises(ResourceLimitError):
+            yield self.handler.register_saml2(localpart="local_part")
+
    @defer.inlineCallbacks
    def test_auto_create_auto_join_rooms(self):
        room_alias_str = "#room:test"
        self.hs.config.auto_join_rooms = [room_alias_str]
        res = yield self.handler.register(localpart='jeff')
        rooms = yield self.store.get_rooms_for_user(res[0])
+
        directory_handler = self.hs.get_handlers().directory_handler
        room_alias = RoomAlias.from_string(room_alias_str)
        room_id = yield directory_handler.get_association(room_alias)
@@ -168,14 +184,3 @@ class RegistrationTestCase(unittest.TestCase):
        res = yield self.handler.register(localpart='jeff')
        rooms = yield self.store.get_rooms_for_user(res[0])
        self.assertEqual(len(rooms), 0)
-
-    @defer.inlineCallbacks
-    def test_auto_create_auto_join_where_no_consent(self):
-        self.hs.config.user_consent_at_registration = True
-        self.hs.config.block_events_without_consent_error = "Error"
-        room_alias_str = "#room:test"
-        self.hs.config.auto_join_rooms = [room_alias_str]
-        res = yield self.handler.register(localpart='jeff')
-        yield self.handler.post_consent_actions(res[0])
-        rooms = yield self.store.get_rooms_for_user(res[0])
-        self.assertEqual(len(rooms), 0)
@@ -1,90 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2018 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from __future__ import print_function
-
-import functools
-import sys
-
-from twisted.internet import defer
-from twisted.internet.defer import Deferred
-from twisted.python.failure import Failure
-
-
-def do_patch():
-    """
-    Patch defer.inlineCallbacks so that it checks the state of the logcontext on exit
-    """
-
-    from synapse.util.logcontext import LoggingContext
-
-    orig_inline_callbacks = defer.inlineCallbacks
-
-    def new_inline_callbacks(f):
-
-        orig = orig_inline_callbacks(f)
-
-        @functools.wraps(f)
-        def wrapped(*args, **kwargs):
-            start_context = LoggingContext.current_context()
-
-            try:
-                res = orig(*args, **kwargs)
-            except Exception:
-                if LoggingContext.current_context() != start_context:
-                    err = "%s changed context from %s to %s on exception" % (
-                        f, start_context, LoggingContext.current_context()
-                    )
-                    print(err, file=sys.stderr)
-                    raise Exception(err)
-                raise
-
-            if not isinstance(res, Deferred) or res.called:
-                if LoggingContext.current_context() != start_context:
-                    err = "%s changed context from %s to %s" % (
-                        f, start_context, LoggingContext.current_context()
-                    )
-                    # print the error to stderr because otherwise all we
-                    # see in travis-ci is the 500 error
-                    print(err, file=sys.stderr)
-                    raise Exception(err)
-                return res
-
-            if LoggingContext.current_context() != LoggingContext.sentinel:
-                err = (
-                    "%s returned incomplete deferred in non-sentinel context "
-                    "%s (start was %s)"
-                ) % (
-                    f, LoggingContext.current_context(), start_context,
-                )
-                print(err, file=sys.stderr)
-                raise Exception(err)
-
-            def check_ctx(r):
-                if LoggingContext.current_context() != start_context:
-                    err = "%s completion of %s changed context from %s to %s" % (
-                        "Failure" if isinstance(r, Failure) else "Success",
-                        f, start_context, LoggingContext.current_context(),
-                    )
-                    print(err, file=sys.stderr)
-                    raise Exception(err)
-                return r
-
-            res.addBoth(check_ctx)
-            return res
-
-        return wrapped
-
-    defer.inlineCallbacks = new_inline_callbacks
@@ -30,7 +30,6 @@ from synapse.rest.media.v1._base import FileInfo
 from synapse.rest.media.v1.filepath import MediaFilePaths
 from synapse.rest.media.v1.media_storage import MediaStorage
 from synapse.rest.media.v1.storage_provider import FileStorageProviderBackend
-from synapse.util.logcontext import make_deferred_yieldable
 from synapse.util.module_loader import load_module

 from tests import unittest
@@ -114,7 +113,7 @@ class MediaRepoTests(unittest.HomeserverTestCase):
            d = Deferred()
            d.addCallback(write_to)
            self.fetches.append((d, destination, path, args))
-            return make_deferred_yieldable(d)
+            return d

        client = Mock()
        client.get_file = get_file
@@ -1,58 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2018 New Vector
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-
-from synapse.rest.well_known import WellKnownResource
-
-from tests import unittest
-
-
-class WellKnownTests(unittest.HomeserverTestCase):
-    def setUp(self):
-        super(WellKnownTests, self).setUp()
-
-        # replace the JsonResource with a WellKnownResource
-        self.resource = WellKnownResource(self.hs)
-
-    def test_well_known(self):
-        self.hs.config.public_baseurl = "https://tesths"
-        self.hs.config.default_identity_server = "https://testis"
-
-        request, channel = self.make_request(
-            "GET",
-            "/.well-known/matrix/client",
-            shorthand=False,
-        )
-        self.render(request)
-
-        self.assertEqual(request.code, 200)
-        self.assertEqual(
-            channel.json_body, {
-                "m.homeserver": {"base_url": "https://tesths"},
-                "m.identity_server": {"base_url": "https://testis"},
-            }
-        )
-
-    def test_well_known_no_public_baseurl(self):
-        self.hs.config.public_baseurl = None
-
-        request, channel = self.make_request(
-            "GET",
-            "/.well-known/matrix/client",
-            shorthand=False,
-        )
-        self.render(request)
-
-        self.assertEqual(request.code, 404)
@@ -149,7 +149,7 @@ class MonthlyActiveUsersTestCase(HomeserverTestCase):

    def test_populate_monthly_users_is_guest(self):
        # Test that guest users are not added to mau list
-        user_id = "@user_id:host"
+        user_id = "user_id"
        self.store.register(
            user_id=user_id, token="123", password_hash=None, make_guest=True
        )
@@ -14,7 +14,7 @@
 # limitations under the License.

 from synapse.api.errors import SynapseError
-from synapse.types import GroupID, RoomAlias, UserID, map_username_to_mxid_localpart
+from synapse.types import GroupID, RoomAlias, UserID

 from tests import unittest
 from tests.utils import TestHomeServer
@@ -79,32 +79,3 @@ class GroupIDTestCase(unittest.TestCase):
            except SynapseError as exc:
                self.assertEqual(400, exc.code)
                self.assertEqual("M_UNKNOWN", exc.errcode)
-
-
-class MapUsernameTestCase(unittest.TestCase):
-    def testPassThrough(self):
-        self.assertEqual(map_username_to_mxid_localpart("test1234"), "test1234")
-
-    def testUpperCase(self):
-        self.assertEqual(map_username_to_mxid_localpart("tEST_1234"), "test_1234")
-        self.assertEqual(
-            map_username_to_mxid_localpart("tEST_1234", case_sensitive=True),
-            "t_e_s_t__1234",
-        )
-
-    def testSymbols(self):
-        self.assertEqual(
-            map_username_to_mxid_localpart("test=$?_1234"),
-            "test=3d=24=3f_1234",
-        )
-
-    def testLeadingUnderscore(self):
-        self.assertEqual(map_username_to_mxid_localpart("_test_1234"), "=5ftest_1234")
-
-    def testNonAscii(self):
-        # this should work with either a unicode or a bytes
-        self.assertEqual(map_username_to_mxid_localpart(u'têst'), "t=c3=aast")
-        self.assertEqual(
-            map_username_to_mxid_localpart(u'têst'.encode('utf-8')),
-            "t=c3=aast",
-        )
@@ -13,7 +13,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import gc
+
 import hashlib
 import hmac
 import logging
@@ -31,12 +31,10 @@ from synapse.http.server import JsonResource
 from synapse.http.site import SynapseRequest
 from synapse.server import HomeServer
 from synapse.types import UserID, create_requester
-from synapse.util.logcontext import LoggingContext, LoggingContextFilter
+from synapse.util.logcontext import LoggingContextFilter

 from tests.server import get_clock, make_request, render, setup_test_homeserver
-from tests.utils import default_config, setupdb
-
-setupdb()
+from tests.utils import default_config

 # Set up putting Synapse's logs into Trial's.
 rootLogger = logging.getLogger()
@@ -104,16 +102,8 @@ class TestCase(unittest.TestCase):
            # traceback when a unit test exits leaving things on the reactor.
            twisted.internet.base.DelayedCall.debug = True

-            # if we're not starting in the sentinel logcontext, then to be honest
-            # all future bets are off.
-            if LoggingContext.current_context() is not LoggingContext.sentinel:
-                self.fail(
-                    "Test starting with non-sentinel logging context %s" % (
-                        LoggingContext.current_context(),
-                    )
-                )
-
            old_level = logging.getLogger().level
+
            if old_level != level:

                @around(self)
@@ -125,16 +115,6 @@ class TestCase(unittest.TestCase):
            logging.getLogger().setLevel(level)
            return orig()

-        @around(self)
-        def tearDown(orig):
-            ret = orig()
-            # force a GC to workaround problems with deferreds leaking logcontexts when
-            # they are GCed (see the logcontext docs)
-            gc.collect()
-            LoggingContext.set_current_context(LoggingContext.sentinel)
-
-            return ret
-
    def assertObjectHasAttributes(self, attrs, obj):
        """Asserts that the given object has each of the attributes given, and
        that the value of each matches according to assertEquals."""
@@ -139,7 +139,6 @@ def default_config(name):
    config.admin_contact = None
    config.rc_messages_per_second = 10000
    config.rc_message_burst_count = 10000
-    config.saml2_enabled = False

    config.use_frozen_dicts = False

@@ -7,7 +7,6 @@ deps =
    mock
    python-subunit
    junitxml
-    coverage

    # needed by some of the tests
    lxml
@@ -28,15 +27,11 @@ deps =

 setenv =
    PYTHONDONTWRITEBYTECODE = no_byte_code
-    COVERAGE_PROCESS_START = {toxinidir}/.coveragerc

 [testenv]
 deps =
    {[base]deps}

-whitelist_externals =
-    sh
-
 setenv =
    {[base]setenv}

@@ -44,9 +39,7 @@ passenv = *

 commands =
    /usr/bin/find "{toxinidir}" -name '*.pyc' -delete
-    # Add this so that coverage will run on subprocesses
-    sh -c 'echo "import coverage; coverage.process_startup()" > {envsitepackagesdir}/../sitecustomize.py'
-    {envbindir}/coverage run "{envbindir}/trial" {env:TRIAL_FLAGS:} {posargs:tests} {env:TOXSUFFIX:}
+    "{envbindir}/trial" {env:TRIAL_FLAGS:} {posargs:tests} {env:TOXSUFFIX:}

 [testenv:py27]

@@ -108,6 +101,17 @@ usedevelop=true
 [testenv:py36]
 usedevelop=true

+
+[testenv:py36-coverage]
+usedevelop=true
+deps =
+    {[base]deps}
+    coverage
+commands =
+    /usr/bin/find "{toxinidir}" -name '*.pyc' -delete
+    python -m coverage run -m twisted.trial {env:TRIAL_FLAGS:} {posargs:tests} {env:TOXSUFFIX:}
+
+
 [testenv:py36-postgres]
 usedevelop=true
 deps =
@@ -134,7 +138,7 @@ commands = /bin/sh -c "flake8 synapse tests scripts scripts-dev scripts/hash_pas
 [testenv:check_isort]
 skip_install = True
 deps = isort
-commands = /bin/sh -c "isort -c -df -sp setup.cfg -rc synapse tests"
+commands = /bin/sh -c "isort -c -sp setup.cfg -rc synapse tests"

 [testenv:check-newsfragment]
 skip_install = True
@@ -142,12 +146,3 @@ deps = towncrier>=18.6.0rc1
 commands =
   python -m towncrier.check --compare-with=origin/develop
 basepython = python3.6
-
-[testenv:codecov]
-skip_install = True
-deps =
-    coverage
-    codecov
-commands =
-    coverage combine
-    codecov -X gcov
Author	SHA1	Message	Date
Erik Johnston	230474b620	Actually fix exceptions	2018-11-29 11:46:28 +00:00
Erik Johnston	cf09912280	Don't log ERROR when no profile exists	2018-11-29 11:32:03 +00:00
Matthew Hodgson	cd317a1910	Merge pull request #4235 from matrix-org/travis/fix-auto-invite-errors Catch room profile errors and anything else that can go wrong	2018-11-28 18:53:22 -08:00
Travis Ralston	11a168442d	Catch room profile errors and anything else that can go wrong Fixes an issue where things become unhappy when the room profile for a user is missing.	2018-11-28 08:57:56 -07:00
Travis Ralston	e8d99369bc	Merge pull request #4218 from matrix-org/travis/account-merging Proof of concept for auto-accepting invites on merged accounts	2018-11-22 09:00:25 -07:00
Travis Ralston	921469383e	Use run_as_background_process	2018-11-22 08:50:05 -07:00
Travis Ralston	ccbf6bb222	Safer execution	2018-11-22 08:47:35 -07:00
Travis Ralston	c68d510564	Preserve log contexts in the room_member_handler	2018-11-21 13:21:21 -07:00
Travis Ralston	ce1b393682	Proof of concept for auto-accepting invites This is for demonstration purposes only. In practice this would actually look up the right profile and use the right thing, not to mention be in a more reasonable location.	2018-11-21 13:03:35 -07:00
				`@@ -0,0 +1 @@`
				`Add option to track MAU stats (but not limit people)`
				`@@ -0,0 +1 @@`
				`The media repository now no longer fails to decode UTF-8 filenames when downloading remote media.`
				`@@ -0,0 +1 @@`
				`A coveragerc file, as well as the py36-coverage tox target, have been added.`
				`@@ -0,0 +1 @@`
				`Add a GitHub pull request template and add multiple issue templates`
				`@@ -0,0 +1 @@`
				URL previews now correctly decode non-UTF-8 text if the header contains a `<meta http-equiv="Content-Type"` header.
				`@@ -0,0 +1 @@`
				`Update README to reflect the fact that #1491 is fixed`
				`@@ -0,0 +1 @@`
				`Fix an issue where public consent URLs had two slashes.`
				`@@ -0,0 +1 @@`
				Add missing `jpeg` package prerequisite for OpenBSD in README.
				`@@ -0,0 +1 @@`
				`Fallback auth now accepts the session parameter on Python 3.`