WIP

* by default include m.room.encryption on invites

* fix constant

* changelog

CHANGES.md

@@ -1,35 +1,3 @@
-Synapse 0.99.0rc4 (2019-02-01)
-==============================
-
-Internal Changes
-----------------
-
-- Update federation routing logic to check .well-known before SRV ([\#4539](https://github.com/matrix-org/synapse/issues/4539))
-- Improve performance of handling servers with invalid .well-known ([\#4542](https://github.com/matrix-org/synapse/issues/4542))
-- Treat an invalid .well-known file the same as an absent one ([\#4544](https://github.com/matrix-org/synapse/issues/4544))
-
-
-Synapse 0.99.0rc3 (2019-01-31)
-==============================
-
-Bugfixes
---------
-
-- Fix infinite loop when an event is redacted in a v3 room ([\#4535](https://github.com/matrix-org/synapse/issues/4535))
-
-
-Improved Documentation
-----------------------
-
-- Update debian installation instructions ([\#4526](https://github.com/matrix-org/synapse/issues/4526))
-
-
-Internal Changes
-----------------
-
-- Add some debug for membership syncing issues ([\#4538](https://github.com/matrix-org/synapse/issues/4538))
-
-
 Synapse 0.99.0rc2 (2019-01-30)
 ==============================
 

README.rst

@@ -4,15 +4,15 @@ Introduction
 ============
 
 Matrix is an ambitious new ecosystem for open federated Instant Messaging and
-VoIP.  The basics you need to know to get up and running are:
+VoIP. The basics you need to know to get up and running are:
 
-- Everything in Matrix happens in a room.  Rooms are distributed and do not
-  exist on any single server.  Rooms can be located using convenience aliases
+- Everything in Matrix happens in a room. Rooms are distributed and do not
+  exist on any single server. Rooms can be located using convenience aliases
   like ``#matrix:matrix.org`` or ``#test:localhost:8448``.
 
 - Matrix user IDs look like ``@matthew:matrix.org`` (although in the future
   you will normally refer to yourself and others using a third party identifier
-  (3PID): email address, phone number, etc rather than manipulating Matrix user IDs)
+  (3PID): email address, phone number, etc rather than manipulating Matrix user IDs).
 
 The overall architecture is::
 
@@ -89,155 +89,160 @@ System requirements:
 - Python 3.5, 3.6, 3.7, or 2.7
 - At least 1GB of free RAM if you want to join large public rooms like #matrix:matrix.org
 
-Installing from source
-----------------------
+The currently supported environment is [Ubuntu 18.04
+LTS](http://releases.ubuntu.com/18.04/).
 
-(Prebuilt packages are available for some platforms - see `Platform-Specific
-Instructions`_.)
+Recommended installation procedure
+----------------------------------
 
-Synapse is written in Python but some of the libraries it uses are written in
-C. So before we can install Synapse itself we need a working C compiler and the
-header files for Python C extensions.
+Building and running Synapse from source in a python3 environment is the
+recommended path for installation, as it is the most well-tested route.
+Binary packages are available for various platforms, but not officially
+supported by the Synapse team. See `Platform Specific Instructions`_ for
+details.
+
+Install prerequisites
+*********************
 
 Installing prerequisites on Ubuntu or Debian::
 
-    sudo apt-get install build-essential python3-dev libffi-dev \
-                         python-pip python-setuptools sqlite3 \
-                         libssl-dev python-virtualenv libjpeg-dev libxslt1-dev
+    sudo apt-get update && sudo apt-get dist-upgrade
+    sudo apt-get install build-essential python3-dev python3-venv \
+                         python3-pip python-setuptools libssl-dev \
+                         libjpeg-dev libffi-dev zlib1g-dev \
+                         libxslt1-dev postgresql libwebp-dev libpq-dev
+
+**TODO: Update and check non-debian distro pre-req's for new process**
 
 Installing prerequisites on ArchLinux::
 
     sudo pacman -S base-devel python python-pip \
-                   python-setuptools python-virtualenv sqlite3
+                   python-setuptools python-virtualenv
 
-Installing prerequisites on CentOS 7 or Fedora 25::
+Installing prerequisites on CentOS 7 or Fedora::
 
     sudo yum install libtiff-devel libjpeg-devel libzip-devel freetype-devel \
                      lcms2-devel libwebp-devel tcl-devel tk-devel redhat-rpm-config \
                      python-virtualenv libffi-devel openssl-devel
     sudo yum groupinstall "Development Tools"
 
-Installing prerequisites on Mac OS X::
-
-    xcode-select --install
-    sudo easy_install pip
-    sudo pip install virtualenv
-    brew install pkg-config libffi
-
 Installing prerequisites on Raspbian::
 
-    sudo apt-get install build-essential python3-dev libffi-dev \
-                         python-pip python-setuptools sqlite3 \
-                         libssl-dev python-virtualenv libjpeg-dev
+    sudo apt-get update && sudo apt-get dist-upgrade
+    sudo apt-get install build-essential python3-dev python3-venv \
+                         python3-pip python-setuptools libssl-dev \
+                         libjpeg-dev libffi-dev zlib1g-dev \
+                         libxslt1-dev postgresql libwebp-dev libpq-dev
 
-Installing prerequisites on openSUSE::
 
-    sudo zypper in -t pattern devel_basis
-    sudo zypper in python-pip python-setuptools sqlite3 python-virtualenv \
-                   python-devel libffi-devel libopenssl-devel libjpeg62-devel
+Set up python environment
+*************************
 
-Installing prerequisites on OpenBSD::
+Add a new user for Synapse and log in as them::
 
-    doas pkg_add python libffi py-pip py-setuptools sqlite3 py-virtualenv \
-                 libxslt jpeg
+    useradd matrix
+    su -l matrix
 
-To install the Synapse homeserver run::
+Create a python3 virtualenv and install dependencies::
 
-    mkdir -p ~/synapse
-    virtualenv -p python3 ~/synapse/env
-    source ~/synapse/env/bin/activate
-    pip install --upgrade pip
-    pip install --upgrade setuptools
-    pip install matrix-synapse[all]
+    python3 -m venv matrix-synapse
+    ./matrix-synapse/bin/python -m pip install -U pip setuptools wheel
+    ./matrix-synapse/bin/python -m pip install -U matrix-synapse[all]
 
-This installs Synapse, along with the libraries it uses, into a virtual
-environment under ``~/synapse/env``.  Feel free to pick a different directory
-if you prefer.
+Create a Synapse configuration directory. **Make sure you change
+``matrix.mydomain.com`` to your own domain**::
 
-This Synapse installation can then be later upgraded by using pip again with the
-update flag::
+    mkdir cfg
+    ./matrix-synapse/bin/python -m synapse.app.homeserver --generate-config \
+                                -H matrix.mydomain.com \ # Change
+                                -c cfg/homeserver.yaml \
+                                --report-stats=yes
 
-    source ~/synapse/env/bin/activate
-    pip install -U matrix-synapse[all]
+Installing postgres
+*******************
 
-In case of problems, please see the _`Troubleshooting` section below.
+`PostgreSQL <https://www.postgresql.org/>`_ is the recommended database backend
+supported by Synapse. If you are upgrading from SQLite, please consult the
+`documentation on how to switch
+<https://github.com/matrix-org/synapse/blob/master/docs/postgres.rst#porting-from-sqlite>`_
+for improved performance.
 
-There is an offical synapse image available at
-https://hub.docker.com/r/matrixdotorg/synapse/tags/ which can be used with
-the docker-compose file available at `contrib/docker <contrib/docker>`_. Further information on
-this including configuration options is available in the README on
-hub.docker.com.
+Enable and start postgresql::
 
-Alternatively, Andreas Peters (previously Silvio Fricke) has contributed a
-Dockerfile to automate a synapse server in a single Docker image, at
-https://hub.docker.com/r/avhost/docker-matrix/tags/
+    systemctl enable postgresql && systemctl start postgresql
 
-Slavi Pantaleev has created an Ansible playbook,
-which installs the offical Docker image of Matrix Synapse
-along with many other Matrix-related services (Postgres database, riot-web, coturn, mxisd, SSL support, etc.).
-For more details, see
-https://github.com/spantaleev/matrix-docker-ansible-deploy
+Assuming your postgres user is called ``postgres``, login and create a user.
+This will prompt for a password, make sure you set a strong passphrase::
 
-Configuring Synapse
--------------------
+    su - postgres
+    createuser --pwprompt synapse_user
 
-Before you can start Synapse, you will need to generate a configuration
-file. To do this, run (in your virtualenv, as before)::
+Create a Synapse database::
 
-    cd ~/synapse
-    python -m synapse.app.homeserver \
-        --server-name my.domain.name \
-        --config-path homeserver.yaml \
-        --generate-config \
-        --report-stats=[yes|no]
+    CREATE DATABASE synapse
+     ENCODING 'UTF8'
+     LC_COLLATE='C'
+     LC_CTYPE='C'
+     template=template0
+     OWNER synapse_user;
 
-... substituting an appropriate value for ``--server-name``. The server name
-determines the "domain" part of user-ids for users on your server: these will
-all be of the format ``@user:my.domain.name``. It also determines how other
-matrix servers will reach yours for `Federation`_. For a test configuration,
-set this to the hostname of your server. For a more production-ready setup, you
-will probably want to specify your domain (``example.com``) rather than a
-matrix-specific hostname here (in the same way that your email address is
-probably ``user@example.com`` rather than ``user@email.example.com``) - but
-doing so may require more advanced setup - see `Setting up
-Federation`_. Beware that the server name cannot be changed later.
+Finally, edit the ``database`` section in your ``cfg/homeserver.yaml`` file
+to point to the new database::
 
-This command will generate you a config file that you can then customise, but it will
-also generate a set of keys for you. These keys will allow your Home Server to
-identify itself to other Home Servers, so don't lose or delete them. It would be
-wise to back them up somewhere safe. (If, for whatever reason, you do need to
-change your Home Server's keys, you may find that other Home Servers have the
-old key cached. If you update the signing key, you should change the name of the
-key in the ``<server name>.signing.key`` file (the second word) to something
-different. See `the spec`__ for more information on key management.)
+    database:
+        name: psycopg2
+        args:
+            user: synapse_user
+            password: <password defined in the createuser step>
+            database: synapse
+            host: localhost
+            cp_min: 5
+            cp_max: 10
 
-.. __: `key_management`_
+More information can be found at `Using Postgres with Synapse
+<docs/postgres.rst>`_.
 
-The default configuration exposes two HTTP ports: 8008 and 8448. Port 8008 is
-configured without TLS; it should be behind a reverse proxy for TLS/SSL
-termination on port 443 which in turn should be used for clients. Port 8448
-is configured to use TLS for `Federation`_ with a self-signed or verified
-certificate, but please be aware that a valid certificate will be required in
-Synapse v1.0.
+Systemd
+*******
 
-If you would like to use your own certificates, you can do so by changing
-``tls_certificate_path`` and ``tls_private_key_path`` in ``homeserver.yaml``;
-alternatively, you can use a reverse-proxy. Apart from port 8448 using TLS,
-both ports are the same in the default configuration.
+Running Synapse under `systemd <https://en.wikipedia.org/wiki/Systemd>`_ is
+recommended, as it allows for simple management and automatic restarts in case
+of a server error. To integrate Synapse with systemd, create a file at
+`/etc/systemd/system/synapse.service` with the following contents::
+
+    [Unit]
+    Description="Synapse homeserver"
+
+    [Service]
+    ExecStart=/home/matrix/matrix-synapse/bin/python -m synapse.app.homeserver
+    PIDFile=/home/matrix/matrix-synapse/homeserver.pid
+    Type=forking
+    WorkingDirectory=/home/matrix/matrix-synapse/
+    Restart=always
+
+Then tell systemd to update service file information::
+
+    sudo systemctl daemon-reload
+
+Synapse should now be enabled to run under Systemd, but **don't start Synapse
+yet!**
 
 
 ACME setup
-----------
+**********
 
-Synapse v1.0 will require valid TLS certificates for communication between servers
+Synapse requires valid TLS certificates for communication between servers
 (port ``8448`` by default) in addition to those that are client-facing (port
-``443``). In the case that your `server_name` config variable is the same as
-the hostname that the client connects to, then the same certificate can be
-used between client and federation ports without issue. Synapse v0.99.0+
-**will provision server-to-server certificates automatically for you for
-free** through `Let's Encrypt
+``443``). Synapse **will provision server-to-server certificates
+automatically for you for free** through `Let's Encrypt
 <https://letsencrypt.org/>`_ if you tell it to.
 
+    Note: Synapse does not currently hot-renew Let's Encrypt certificates for
+    you, it only checks for certificates that need renewing on restart. This
+    functionality will be implemented promptly, but if in the meantime your
+    federation certificates expire, simply restarting Synapse should renew
+    them automatically.
+
 In order for Synapse to complete the ACME challenge to provision a
 certificate, it needs access to port 80. Typically listening on port 80 is
 only granted to applications running as root. There are thus two solutions to
@@ -245,54 +250,81 @@ this problem.
 
 **Using a reverse proxy**
 
-A reverse proxy such as Apache or nginx allows a single process (the web
-server) to listen on port 80 and proxy traffic to the appropriate program
-running on your server. It is the recommended method for setting up ACME as
-it allows you to use your existing webserver while also allowing Synapse to
-provision certificates as needed.
+A reverse proxy such as Apache or Nginx allows a single process (the web
+server) to listen on port 80 and redirect traffic to the appropriate program
+running on your server.
 
-For nginx users, add the following line to your existing ``server`` block::
-
-    location /.well-known/acme-challenge {
-        proxy_pass http://localhost:8009/;
-    }
-
-For Apache, add the following to your existing webserver config::
-
-    ProxyPass /.well-known/acme-challenge http://localhost:8009/.well-known/acme-challenge
-
-Make sure to restart/reload your webserver after making changes.
 
 
 **Authbind**
 
-``authbind`` allows a program which does not run as root to bind to
-low-numbered ports in a controlled way. The setup is simpler, but requires a
-webserver not to already be running on port 80. **This includes every time
-Synapse renews a certificate**, which may be cumbersome if you usually run a
-web server on port 80. Nevertheless, if you're sure port 80 is not being used
-for any other purpose then all that is necessary is the following:
+``authbind`` allows a program which does not or should not run as root to
+bind to low-numbered ports in a controlled way. The setup is simpler, but
+requires a webserver not to already be running on port 80. **This includes
+every time Synapse renews a certificate**, which may be cumbersome if you
+usually run a web server on port 80. Nevertheless, if that isn't a concern,
+follow the instructions below.
 
-Install ``authbind``. For example, on Debian/Ubuntu::
+Install ``authbind``. This can be done on Ubuntu/Debian with::
 
     sudo apt-get install authbind
 
-Allow ``authbind`` to bind port 80::
+**Add authbind to the systemd script**
 
-    sudo touch /etc/authbind/byport/80
-    sudo chmod 777 /etc/authbind/byport/80
 
-When Synapse is started, use the following syntax::
+**TODO: This right?** If you would like to use your own
+certificates, specifying them in Synapse's config file is sufficient.
 
-    authbind --deep <synapse start command>
 
-Finally, once Synapse's is able to listen on port 80 for ACME challenge
-requests, it must be told to perform ACME provisioning by setting ``enabled``
-to true under the ``acme`` section in ``homeserver.yaml``::
+**TODO: Fit this in**
+These keys will allow your Home Server to identify itself to other Home
+Servers, so don't lose or delete them. It would be wise to back them up
+somewhere safe. (If, for whatever reason, you do need to change your Home
+Server's keys, you may find that other Home Servers have the old key cached.
+If you update the signing key, you should change the name of the key in the
+``<server name>.signing.key`` file (the second word) to something different.
+See `the spec`__ for more information on key management.)
 
-    acme:
-        enabled: true
+**TODO: Does this still work?** This Synapse installation can then be later
+upgraded by using pip again with the update flag::
 
+    source ~/synapse/env/bin/activate
+    pip install -U matrix-synapse[all]
+
+In case of problems, please see the _`Troubleshooting` section below.
+
+We have now created a "matrix" user with its own home directory that stores
+Synapse's data and configuration files, backed by a postgres database, all
+packaged into a isolated python virtual environment.
+
+Configuring Synapse
+-------------------
+
+Before starting Synapse, inspect the ``cfg/homeserver.yaml`` file. ``server_name``
+determines the "domain" part of user-ids for users on your server, which will
+all be of the format ``@user:my.domain.name``. It also determines how other
+matrix servers will reach yours for `Federation`_. For a test configuration,
+set this to the hostname of your server. For a more production-ready setup, you
+will probably want to specify your domain (``example.com``) rather than a
+matrix-specific hostname here (in the same way that your email address is
+probably ``user@example.com`` rather than ``user@email.example.com``) - but
+doing so may require more advanced setup - see `Setting up
+Federation`_. **Be aware that the server name cannot be changed later.**
+
+.. __: `key_management`_
+
+The default configuration exposes two HTTP ports: 8008 and 8448. Port 8008 is
+configured without TLS; it should be behind a reverse proxy for TLS/SSL
+termination on port 443 which in turn should be used for clients. Port 8448
+is configured to use TLS with a self-signed certificate. If you would like
+to do an initial test with a client without having to setup a reverse proxy,
+you can temporarly use another certificate. You can do so by changing
+``tls_certificate_path`` and ``tls_private_key_path``
+in ``homeserver.yaml``; alternatively, you can use a reverse-proxy, but be sure
+to read `Using a reverse proxy with Synapse`_ when doing so.
+
+Apart from port 8448 using TLS, both ports are the same in the default
+configuration.
 
 Registering a user
 ------------------
@@ -330,6 +362,8 @@ a TURN server.  See `<docs/turn-howto.rst>`_ for details.
 Running Synapse
 ===============
 
+**TODO: Needs update**
+
 To actually run your new homeserver, pick a working directory for Synapse to
 run (e.g. ``~/synapse``), and::
 
@@ -337,6 +371,16 @@ run (e.g. ``~/synapse``), and::
     source env/bin/activate
     synctl start
 
+Upgrading an existing Synapse
+=============================
+
+The instructions for upgrading synapse are in `UPGRADE.rst`_.
+Please check these instructions as upgrading may require extra steps for some
+versions of synapse.
+
+.. _UPGRADE.rst: UPGRADE.rst
+
+
 Connecting to Synapse from a client
 ===================================
 
@@ -348,11 +392,10 @@ following the recommended setup, or ``https://localhost:8448`` - remember to spe
 port (``:8448``) if not ``:443`` unless you changed the configuration. (Leave the identity
 server as the default - see `Identity servers`_.)
 
-If using port 8448 you will run into errors if you are using a self-signed
-certificate. To overcome this, simply go to ``https://localhost:8448``
+If using port 8448 you will run into errors until you accept the self-signed
+certificate. You can easily do this by going to ``https://localhost:8448``
 directly with your browser and accept the presented certificate. You can then
-go back in your web client and proceed further. Valid federation certificates
-should not have this problem.
+go back in your web client and proceed further.
 
 If all goes well you should at least be able to log in, create a room, and
 start sending messages.
@@ -398,41 +441,20 @@ See https://github.com/vector-im/riot-web/issues/1977 and
 https://developer.github.com/changes/2014-04-25-user-content-security for more details.
 
 
-Platform-Specific Instructions
-==============================
+Platform-Specific Packages
+==========================
 
-Debian/Ubuntu
--------------
+Note that the only officially supported installation method is what is listed
+in `Synapse installation`_. Instructions and packages for other platforms are
+listed below, but beware that they may be outdated.
 
-Matrix.org packages
-~~~~~~~~~~~~~~~~~~~
+Debian
+------
 
-Matrix.org provides Debian/Ubuntu packages of the latest stable version of
-Synapse via https://matrix.org/packages/debian/. To use them::
+Matrix provides official Debian packages via apt from https://matrix.org/packages/debian/.
 
-    sudo apt install -y lsb-release curl apt-transport-https
-    echo "deb https://matrix.org/packages/debian `lsb_release -cs` main" |
-        sudo tee /etc/apt/sources.list.d/matrix-org.list
-    curl "https://matrix.org/packages/debian/repo-key.asc" |
-        sudo apt-key add -
-    sudo apt update
-    sudo apt install matrix-synapse-py3
-
-Downstream Debian/Ubuntu packages
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-For ``buster`` and ``sid``, Synapse is available in the Debian repositories and
-it should be possible to install it with simply::
-
-    sudo apt install matrix-synapse
-
-There is also a version of ``matrix-synapse`` in ``stretch-backports``. Please
-see the `Debian documentation on backports
-<https://backports.debian.org/Instructions/>`_ for information on how to use
-them.
-
-We do not recommend using the packages in downstream Ubuntu at this time, as
-they are old and suffer from known security vulnerabilities.
+Note that these packages do not include a client - choose one from
+https://matrix.org/docs/projects/try-matrix-now.html (or build your own with one of our SDKs :).
 
 Fedora
 ------
@@ -484,7 +506,6 @@ Synapse can be installed via FreeBSD Ports or Packages contributed by Brendan Mo
  - Ports: ``cd /usr/ports/net-im/py-matrix-synapse && make install clean``
  - Packages: ``pkg install py27-matrix-synapse``
 
-
 OpenBSD
 -------
 
@@ -518,12 +539,33 @@ https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/misc/matrix-
 Windows Install
 ---------------
 
-If you wish to run or develop Synapse on Windows, the Windows Subsystem For
-Linux provides a Linux environment on Windows 10 which is capable of using the
-Debian, Fedora, or source installation methods. More information about WSL can
-be found at https://docs.microsoft.com/en-us/windows/wsl/install-win10 for
-Windows 10 and https://docs.microsoft.com/en-us/windows/wsl/install-on-server
-for Windows Server.
+Running Synapse on Windows is not recommended or supported. However, if you
+wish to run Synapse on Windows, the Windows Subsystem For Linux provides a
+Linux environment on Windows 10 which is capable of using the Debian, Fedora,
+or source installation methods. More information about WSL can be found at
+https://docs.microsoft.com/en-us/windows/wsl/install-win10 for Windows 10 and
+https://docs.microsoft.com/en-us/windows/wsl/install-on-server for Windows
+Server.
+
+
+Alternative installation methods
+================================
+
+There is an offical synapse image available at
+https://hub.docker.com/r/matrixdotorg/synapse/tags/ which can be used with
+the docker-compose file available at `contrib/docker <contrib/docker>`_.
+Further information on this including configuration options is available in
+the README on hub.docker.com.
+
+Alternatively, Andreas Peters (previously Silvio Fricke) has contributed a
+Dockerfile to automate a synapse server in a single Docker image, at
+https://hub.docker.com/r/avhost/docker-matrix/tags/
+
+Slavi Pantaleev has created an Ansible playbook, which installs the offical
+Docker image of Matrix Synapse along with many other Matrix-related services
+(Postgres database, riot-web, coturn, mxisd, SSL support, etc.). For more
+details, see https://github.com/spantaleev/matrix-docker-ansible-deploy
+
 
 Troubleshooting
 ===============
@@ -567,7 +609,7 @@ failing, e.g.::
     pip install twisted
 
 Running out of File Handles
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+***************************
 
 If synapse runs out of filehandles, it typically fails badly - live-locking
 at 100% CPU, and/or failing to accept new TCP connections (blocking the
@@ -590,16 +632,6 @@ log lines and looking for any 'Processed request' lines which take more than
 a few seconds to execute.  Please let us know at #matrix-dev:matrix.org if
 you see this failure mode so we can help debug it, however.
 
-
-Upgrading an existing Synapse
-=============================
-
-The instructions for upgrading synapse are in `UPGRADE.rst`_.
-Please check these instructions as upgrading may require extra steps for some
-versions of synapse.
-
-.. _UPGRADE.rst: UPGRADE.rst
-
 .. _federation:
 
 Setting up Federation
@@ -621,7 +653,9 @@ you to run your server on a machine that might not have the same name as your
 domain name. For example, you might want to run your server at
 ``synapse.example.com``, but have your Matrix user-ids look like
 ``@user:example.com``. (A SRV record also allows you to change the port from
-the default 8448).
+the default 8448. However, if you are thinking of using a reverse-proxy on the
+federation port, which is not recommended, be sure to read
+`Reverse-proxying the federation port`_ first.)
 
 To use a SRV record, first create your SRV record and publish it in DNS. This
 should have the format ``_matrix._tcp.<yourdomain.com> <ttl> IN SRV 10 0 <port>
@@ -659,8 +693,6 @@ Troubleshooting
 You can use the federation tester to check if your homeserver is all set:
 ``https://matrix.org/federationtester/api/report?server_name=<your_server_name>``
 If any of the attributes under "checks" is false, federation won't work.
-There is also a nicer interface available from a community member at
-`<https://neo.lain.haus/fed-tester>`_.
 
 The typical failure mode with federation is that when you try to join a room,
 it is rejected with "401: Unauthorized". Generally this means that other
@@ -669,6 +701,8 @@ complicated dance which requires connections in both directions).
 
 So, things to check are:
 
+* If you are trying to use a reverse-proxy, read `Reverse-proxying the
+  federation port`_.
 * If you are not using a SRV record, check that your ``server_name`` (the part
   of your user-id after the ``:``) matches your hostname, and that port 8448 on
   that hostname is reachable from outside your network.
@@ -723,8 +757,14 @@ port. Indeed, clients will use port 443 by default, whereas servers default to
 port 8448. Where these are different, we refer to the 'client port' and the
 'federation port'.
 
-All Matrix endpoints begin with ``/_matrix``, so an example nginx
-configuration for forwarding client connections to Synapse might look like::
+The next most important thing to know is that using a reverse-proxy on the
+federation port has a number of pitfalls. It is possible, but be sure to read
+`Reverse-proxying the federation port`_.
+
+The recommended setup is therefore to configure your reverse-proxy on port 443
+to port 8008 of synapse for client connections, but to also directly expose port
+8448 for server-server connections. All the Matrix endpoints begin ``/_matrix``,
+so an example nginx configuration might look like::
 
   server {
       listen 443 ssl;
@@ -765,6 +805,64 @@ Having done so, you can then use ``https://matrix.example.com`` (instead of
 ``https://matrix.example.com:8448``) as the "Custom server" when `Connecting to
 Synapse from a client`_.
 
+Reverse-proxying the federation port
+------------------------------------
+
+There are two issues to consider before using a reverse-proxy on the federation
+port:
+
+* Due to the way SSL certificates are managed in the Matrix federation protocol
+  (see `spec`__), Synapse needs to be configured with the path to the SSL
+  certificate, *even if you do not terminate SSL at Synapse*.
+
+  .. __: `key_management`_
+
+* Until v0.33.3, Synapse did not support SNI on the federation port
+  (`bug #1491 <https://github.com/matrix-org/synapse/issues/1491>`_). This bug
+  is now fixed, but means that federating with older servers can be unreliable
+  when using name-based virtual hosting.
+
+Furthermore, a number of the normal reasons for using a reverse-proxy do not
+apply:
+
+* Other servers will connect on port 8448 by default, so there is no need to
+  listen on port 443 (for federation, at least), which avoids the need for root
+  privileges and virtual hosting.
+
+* A self-signed SSL certificate is fine for federation, so there is no need to
+  automate renewals. (The certificate generated by ``--generate-config`` is
+  valid for 10 years.)
+
+If you want to set up a reverse-proxy on the federation port despite these
+caveats, you will need to do the following:
+
+* In ``homeserver.yaml``, set ``tls_certificate_path`` to the path to the SSL
+  certificate file used by your reverse-proxy, and set ``no_tls`` to ``True``.
+  (``tls_private_key_path`` will be ignored if ``no_tls`` is ``True``.)
+
+* In your reverse-proxy configuration:
+
+  * If there are other virtual hosts on the same port, make sure that the
+    *default* one uses the certificate configured above.
+
+  * Forward ``/_matrix`` to Synapse.
+
+* If your reverse-proxy is not listening on port 8448, publish a SRV record to
+  tell other servers how to find you. See `Setting up Federation`_.
+
+When updating the SSL certificate, just update the file pointed to by
+``tls_certificate_path`` and then restart Synapse. (You may like to use a symbolic link
+to help make this process atomic.)
+
+The most common mistake when setting up federation is not to tell Synapse about
+your SSL certificate. To check it, you can visit
+``https://matrix.org/federationtester/api/report?server_name=<your_server_name>``.
+Unfortunately, there is no UI for this yet, but, you should see
+``"MatchingTLSFingerprint": true``. If not, check that
+``Certificates[0].SHA256Fingerprint`` (the fingerprint of the certificate
+presented by your reverse-proxy) matches ``Keys.tls_fingerprints[0].sha256``
+(the fingerprint of the certificate Synapse is using).
+
 
 Identity Servers
 ================

changelog.d/3902.feature

@@ -0,0 +1 @@
+Include m.room.encryption on invites by default

changelog.d/4547.misc

@@ -1 +0,0 @@
-Add docs for ACME setup to README.

contrib/prometheus/README

@@ -6,10 +6,8 @@ To use it, first install prometheus by following the instructions at
   http://prometheus.io/
 
 ### for Prometheus v1
-
 Add a new job to the main prometheus.conf file:
 
-```yaml
   job: {
     name: "synapse"
 
@@ -17,12 +15,10 @@ Add a new job to the main prometheus.conf file:
       target: "http://SERVER.LOCATION.HERE:PORT/_synapse/metrics"
     }
   }
-```
 
 ### for Prometheus v2
 Add a new job to the main prometheus.yml file:
 
-```yaml
   - job_name: "synapse"
     metrics_path: "/_synapse/metrics"
     # when endpoint uses https:
@@ -30,14 +26,11 @@ Add a new job to the main prometheus.yml file:
 
     static_configs:
     - targets: ['SERVER.LOCATION:PORT']
-```
 
 To use `synapse.rules` add
 
-```yaml
     rule_files:
       - "/PATH/TO/synapse-v2.rules"
-```
 
 Metrics are disabled by default when running synapse; they must be enabled
 with the 'enable-metrics' option, either in the synapse config file or as a

synapse/__init__.py

@@ -27,4 +27,4 @@ try:
 except ImportError:
     pass
 
-__version__ = "0.99.0rc4"
+__version__ = "0.99.0rc2"

synapse/api/constants.py

@@ -73,6 +73,7 @@ class EventTypes(object):
     RoomHistoryVisibility = "m.room.history_visibility"
     CanonicalAlias = "m.room.canonical_alias"
     RoomAvatar = "m.room.avatar"
+    RoomEncryption = "m.room.encryption"
     GuestAccess = "m.room.guest_access"
 
     # These are used for validation

synapse/config/api.py

@@ -24,6 +24,7 @@ class ApiConfig(Config):
             EventTypes.JoinRules,
             EventTypes.CanonicalAlias,
             EventTypes.RoomAvatar,
+            EventTypes.RoomEncryption,
             EventTypes.Name,
         ])
 
@@ -36,5 +37,6 @@ class ApiConfig(Config):
             - "{JoinRules}"
             - "{CanonicalAlias}"
             - "{RoomAvatar}"
+            - "{RoomEncryption}"
             - "{Name}"
         """.format(**vars(EventTypes))

synapse/federation/transport/client.py

@@ -167,7 +167,7 @@ class TransportLayerClient(object):
         # generated by the json_data_callback.
         json_data = transaction.get_dict()
 
-        path = _create_v1_path("/send/%s", transaction.transaction_id)
+        path = _create_v1_path("/send/%s/", transaction.transaction_id)
 
         response = yield self.client.put_json(
             transaction.destination,

synapse/federation/transport/server.py

@@ -366,7 +366,7 @@ class FederationSendServlet(BaseFederationServlet):
 
 
 class FederationEventServlet(BaseFederationServlet):
-    PATH = "/event/(?P<event_id>[^/]*)"
+    PATH = "/event/(?P<event_id>[^/]*)/"
 
     # This is when someone asks for a data item for a given server data_id pair.
     def on_GET(self, origin, content, query, event_id):
@@ -374,7 +374,7 @@ class FederationEventServlet(BaseFederationServlet):
 
 
 class FederationStateServlet(BaseFederationServlet):
-    PATH = "/state/(?P<context>[^/]*)"
+    PATH = "/state/(?P<context>[^/]*)/"
 
     # This is when someone asks for all data for a given context.
     def on_GET(self, origin, content, query, context):
@@ -386,7 +386,7 @@ class FederationStateServlet(BaseFederationServlet):
 
 
 class FederationStateIdsServlet(BaseFederationServlet):
-    PATH = "/state_ids/(?P<room_id>[^/]*)"
+    PATH = "/state_ids/(?P<room_id>[^/]*)/"
 
     def on_GET(self, origin, content, query, room_id):
         return self.handler.on_state_ids_request(
@@ -397,7 +397,7 @@ class FederationStateIdsServlet(BaseFederationServlet):
 
 
 class FederationBackfillServlet(BaseFederationServlet):
-    PATH = "/backfill/(?P<context>[^/]*)"
+    PATH = "/backfill/(?P<context>[^/]*)/"
 
     def on_GET(self, origin, content, query, context):
         versions = [x.decode('ascii') for x in query[b"v"]]

synapse/handlers/sync.py

@@ -895,17 +895,14 @@ class SyncHandler(object):
         Returns:
             Deferred(SyncResult)
         """
+        logger.info("Calculating sync response for %r", sync_config.user)
+
         # NB: The now_token gets changed by some of the generate_sync_* methods,
         # this is due to some of the underlying streams not supporting the ability
         # to query up to a given point.
         # Always use the `now_token` in `SyncResultBuilder`
         now_token = yield self.event_sources.get_current_token()
 
-        logger.info(
-            "Calculating sync response for %r between %s and %s",
-            sync_config.user, since_token, now_token,
-        )
-
         user_id = sync_config.user.to_string()
         app_service = self.store.get_app_service_by_user_id(user_id)
         if app_service:
@@ -1393,12 +1390,6 @@ class SyncHandler(object):
         room_entries = []
         invited = []
         for room_id, events in iteritems(mem_change_events_by_room_id):
-            logger.info(
-                "Membership changes in %s: [%s]",
-                room_id,
-                ", ".join(("%s (%s)" % (e.event_id, e.membership) for e in events)),
-            )
-
             non_joins = [e for e in events if e.membership != Membership.JOIN]
             has_join = len(non_joins) != len(events)
 

synapse/http/federation/matrix_federation_agent.py

@@ -23,17 +23,14 @@ from zope.interface import implementer
 
 from twisted.internet import defer
 from twisted.internet.endpoints import HostnameEndpoint, wrapClientTLS
-from twisted.internet.interfaces import IStreamClientEndpoint
 from twisted.web.client import URI, Agent, HTTPConnectionPool, RedirectAgent, readBody
 from twisted.web.http import stringToDatetime
 from twisted.web.http_headers import Headers
 from twisted.web.iweb import IAgent
 
 from synapse.http.federation.srv_resolver import SrvResolver, pick_server_from_list
-from synapse.util import Clock
 from synapse.util.caches.ttlcache import TTLCache
 from synapse.util.logcontext import make_deferred_yieldable
-from synapse.util.metrics import Measure
 
 # period to cache .well-known results for by default
 WELL_KNOWN_DEFAULT_CACHE_PERIOD = 24 * 3600
@@ -47,6 +44,7 @@ WELL_KNOWN_INVALID_CACHE_PERIOD = 1 * 3600
 # cap for .well-known cache period
 WELL_KNOWN_MAX_CACHE_PERIOD = 48 * 3600
 
+
 logger = logging.getLogger(__name__)
 well_known_cache = TTLCache('well-known')
 
@@ -80,8 +78,6 @@ class MatrixFederationAgent(object):
         _well_known_cache=well_known_cache,
     ):
         self._reactor = reactor
-        self._clock = Clock(reactor)
-
         self._tls_client_options_factory = tls_client_options_factory
         if _srv_resolver is None:
             _srv_resolver = SrvResolver()
@@ -102,10 +98,6 @@ class MatrixFederationAgent(object):
         )
         self._well_known_agent = _well_known_agent
 
-        # our cache of .well-known lookup results, mapping from server name
-        # to delegated name. The values can be:
-        #   `bytes`:     a valid server-name
-        #   `None`:      there is no (valid) .well-known here
         self._well_known_cache = _well_known_cache
 
     @defer.inlineCallbacks
@@ -160,9 +152,12 @@ class MatrixFederationAgent(object):
         class EndpointFactory(object):
             @staticmethod
             def endpointForURI(_uri):
-                ep = LoggingHostnameEndpoint(
-                    self._reactor, res.target_host, res.target_port,
+                logger.info(
+                    "Connecting to %s:%i",
+                    res.target_host.decode("ascii"),
+                    res.target_port,
                 )
+                ep = HostnameEndpoint(self._reactor, res.target_host, res.target_port)
                 if tls_options is not None:
                     ep = wrapClientTLS(tls_options, ep)
                 return ep
@@ -215,7 +210,11 @@ class MatrixFederationAgent(object):
                 target_port=parsed_uri.port,
             ))
 
-        if lookup_well_known:
+        # try a SRV lookup
+        service_name = b"_matrix._tcp.%s" % (parsed_uri.host,)
+        server_list = yield self._srv_resolver.resolve_service(service_name)
+
+        if not server_list and lookup_well_known:
             # try a .well-known lookup
             well_known_server = yield self._get_well_known(parsed_uri.host)
 
@@ -251,10 +250,6 @@ class MatrixFederationAgent(object):
                 res = yield self._route_matrix_uri(new_uri, lookup_well_known=False)
                 defer.returnValue(res)
 
-        # try a SRV lookup
-        service_name = b"_matrix._tcp.%s" % (parsed_uri.host,)
-        server_list = yield self._srv_resolver.resolve_service(service_name)
-
         if not server_list:
             target_host = parsed_uri.host
             port = 8448
@@ -288,32 +283,14 @@ class MatrixFederationAgent(object):
                 None if there was no .well-known file.
         """
         try:
-            result = self._well_known_cache[server_name]
+            cached = self._well_known_cache[server_name]
+            defer.returnValue(cached)
         except KeyError:
-            # TODO: should we linearise so that we don't end up doing two .well-known
-            # requests for the same server in parallel?
-            with Measure(self._clock, "get_well_known"):
-                result, cache_period = yield self._do_get_well_known(server_name)
+            pass
 
-            if cache_period > 0:
-                self._well_known_cache.set(server_name, result, cache_period)
+        # TODO: should we linearise so that we don't end up doing two .well-known requests
+        # for the same server in parallel?
 
-        defer.returnValue(result)
-
-    @defer.inlineCallbacks
-    def _do_get_well_known(self, server_name):
-        """Actually fetch and parse a .well-known, without checking the cache
-
-        Args:
-            server_name (bytes): name of the server, from the requested url
-
-        Returns:
-            Deferred[Tuple[bytes|None|object],int]:
-                result, cache period, where result is one of:
-                 - the new server name from the .well-known (as a `bytes`)
-                 - None if there was no .well-known file.
-                 - INVALID_WELL_KNOWN if the .well-known was invalid
-        """
         uri = b"https://%s/.well-known/matrix/server" % (server_name, )
         uri_str = uri.decode("ascii")
         logger.info("Fetching %s", uri_str)
@@ -324,13 +301,6 @@ class MatrixFederationAgent(object):
             body = yield make_deferred_yieldable(readBody(response))
             if response.code != 200:
                 raise Exception("Non-200 response %s" % (response.code, ))
-
-            parsed_body = json.loads(body.decode('utf-8'))
-            logger.info("Response from .well-known: %s", parsed_body)
-            if not isinstance(parsed_body, dict):
-                raise Exception("not a dict")
-            if "m.server" not in parsed_body:
-                raise Exception("Missing key 'm.server'")
         except Exception as e:
             logger.info("Error fetching %s: %s", uri_str, e)
 
@@ -338,7 +308,19 @@ class MatrixFederationAgent(object):
             # after startup
             cache_period = WELL_KNOWN_INVALID_CACHE_PERIOD
             cache_period += random.uniform(0, WELL_KNOWN_DEFAULT_CACHE_PERIOD_JITTER)
-            defer.returnValue((None, cache_period))
+
+            self._well_known_cache.set(server_name, None, cache_period)
+            defer.returnValue(None)
+
+        try:
+            parsed_body = json.loads(body.decode('utf-8'))
+            logger.info("Response from .well-known: %s", parsed_body)
+            if not isinstance(parsed_body, dict):
+                raise Exception("not a dict")
+            if "m.server" not in parsed_body:
+                raise Exception("Missing key 'm.server'")
+        except Exception as e:
+            raise Exception("invalid .well-known response from %s: %s" % (uri_str, e,))
 
         result = parsed_body["m.server"].encode("ascii")
 
@@ -354,20 +336,10 @@ class MatrixFederationAgent(object):
         else:
             cache_period = min(cache_period, WELL_KNOWN_MAX_CACHE_PERIOD)
 
-        defer.returnValue((result, cache_period))
+        if cache_period > 0:
+            self._well_known_cache.set(server_name, result, cache_period)
 
-
-@implementer(IStreamClientEndpoint)
-class LoggingHostnameEndpoint(object):
-    """A wrapper for HostnameEndpint which logs when it connects"""
-    def __init__(self, reactor, host, port, *args, **kwargs):
-        self.host = host
-        self.port = port
-        self.ep = HostnameEndpoint(reactor, host, port, *args, **kwargs)
-
-    def connect(self, protocol_factory):
-        logger.info("Connecting to %s:%i", self.host.decode("ascii"), self.port)
-        return self.ep.connect(protocol_factory)
+        defer.returnValue(result)
 
 
 def _cache_period_from_headers(headers, time_now=time.time):

synapse/rest/client/v2_alpha/groups.py

@@ -190,7 +190,7 @@ class GroupCategoriesServlet(RestServlet):
     """Get all group categories
     """
     PATTERNS = client_v2_patterns(
-        "/groups/(?P<group_id>[^/]*)/categories$"
+        "/groups/(?P<group_id>[^/]*)/categories/$"
     )
 
     def __init__(self, hs):

synapse/storage/events_worker.py

@@ -161,12 +161,6 @@ class EventsWorkerStore(SQLBaseStore):
             log_ctx = LoggingContext.current_context()
             log_ctx.record_event_fetch(len(missing_events_ids))
 
-            # Note that _enqueue_events is also responsible for turning db rows
-            # into FrozenEvents (via _get_event_from_row), which involves seeing if
-            # the events have been redacted, and if so pulling the redaction event out
-            # of the database to check it.
-            #
-            # _enqueue_events is a bit of a rubbish name but naming is hard.
             missing_events = yield self._enqueue_events(
                 missing_events_ids,
                 allow_rejected=allow_rejected,
@@ -185,35 +179,14 @@ class EventsWorkerStore(SQLBaseStore):
             # instead.
             if not allow_rejected and entry.event.type == EventTypes.Redaction:
                 if entry.event.internal_metadata.need_to_check_redaction():
-                    # XXX: we need to avoid calling get_event here.
-                    #
-                    # The problem is that we end up at this point when an event
-                    # which has been redacted is pulled out of the database by
-                    # _enqueue_events, because _enqueue_events needs to check the
-                    # redaction before it can cache the redacted event. So obviously,
-                    # calling get_event to get the redacted event out of the database
-                    # gives us an infinite loop.
-                    #
-                    # For now (quick hack to fix during 0.99 release cycle), we just
-                    # go and fetch the relevant row from the db, but it would be nice
-                    # to think about how we can cache this rather than hit the db
-                    # every time we access a redaction event.
-                    #
-                    # One thought on how to do this:
-                    #  1. split _get_events up so that it is divided into (a) get the
-                    #     rawish event from the db/cache, (b) do the redaction/rejection
-                    #     filtering
-                    #  2. have _get_event_from_row just call the first half of that
-
-                    orig_sender = yield self._simple_select_one_onecol(
-                        table="events",
-                        keyvalues={"event_id": entry.event.redacts},
-                        retcol="sender",
+                    orig = yield self.get_event(
+                        entry.event.redacts,
                         allow_none=True,
+                        allow_rejected=True,
+                        get_prev_content=False,
                     )
-
                     expected_domain = get_domain_from_id(entry.event.sender)
-                    if orig_sender and get_domain_from_id(orig_sender) == expected_domain:
+                    if orig and get_domain_from_id(orig.sender) == expected_domain:
                         # This redaction event is allowed. Mark as not needing a
                         # recheck.
                         entry.event.internal_metadata.recheck_redaction = False

tests/http/federation/test_matrix_federation_agent.py

@@ -124,7 +124,7 @@ class MatrixFederationAgentTests(TestCase):
                 _check_logcontext(context)
 
     def _handle_well_known_connection(
-        self, client_factory, expected_sni, content, response_headers={},
+        self, client_factory, expected_sni, target_server, response_headers={},
     ):
         """Handle an outgoing HTTPs connection: wire it up to a server, check that the
         request is for a .well-known, and send the response.
@@ -132,7 +132,8 @@ class MatrixFederationAgentTests(TestCase):
         Args:
             client_factory (IProtocolFactory): outgoing connection
             expected_sni (bytes): SNI that we expect the outgoing connection to send
-            content (bytes): content to send back as the .well-known
+            target_server (bytes): target server that we should redirect to in the
+                .well-known response.
         Returns:
             HTTPChannel: server impl
         """
@@ -144,10 +145,10 @@ class MatrixFederationAgentTests(TestCase):
         # check the .well-known request and send a response
         self.assertEqual(len(well_known_server.requests), 1)
         request = well_known_server.requests[0]
-        self._send_well_known_response(request, content, headers=response_headers)
+        self._send_well_known_response(request, target_server, headers=response_headers)
         return well_known_server
 
-    def _send_well_known_response(self, request, content, headers={}):
+    def _send_well_known_response(self, request, target_server, headers={}):
         """Check that an incoming request looks like a valid .well-known request, and
         send back the response.
         """
@@ -160,7 +161,7 @@ class MatrixFederationAgentTests(TestCase):
         # send back a response
         for k, v in headers.items():
             request.setHeader(k, v)
-        request.write(content)
+        request.write(b'{ "m.server": "%s" }' % (target_server,))
         request.finish()
 
         self.reactor.pump((0.1, ))
@@ -357,8 +358,9 @@ class MatrixFederationAgentTests(TestCase):
         # Nothing happened yet
         self.assertNoResult(test_d)
 
-        # No SRV record lookup yet
-        self.mock_resolver.resolve_service.assert_not_called()
+        self.mock_resolver.resolve_service.assert_called_once_with(
+            b"_matrix._tcp.testserv",
+        )
 
         # there should be an attempt to connect on port 443 for the .well-known
         clients = self.reactor.tcpClients
@@ -374,11 +376,6 @@ class MatrixFederationAgentTests(TestCase):
         # .well-known request fails.
         self.reactor.pump((0.4,))
 
-        # now there should be a SRV lookup
-        self.mock_resolver.resolve_service.assert_called_once_with(
-            b"_matrix._tcp.testserv",
-        )
-
         # we should fall back to a direct connection
         self.assertEqual(len(clients), 2)
         (host, port, client_factory, _timeout, _bindAddress) = clients[1]
@@ -406,7 +403,8 @@ class MatrixFederationAgentTests(TestCase):
         self.successResultOf(test_d)
 
     def test_get_well_known(self):
-        """Test the behaviour when the .well-known delegates elsewhere
+        """Test the behaviour when the server name has no port and no SRV record, but
+        the .well-known redirects elsewhere
         """
 
         self.mock_resolver.resolve_service.side_effect = lambda _: []
@@ -418,6 +416,11 @@ class MatrixFederationAgentTests(TestCase):
         # Nothing happened yet
         self.assertNoResult(test_d)
 
+        self.mock_resolver.resolve_service.assert_called_once_with(
+            b"_matrix._tcp.testserv",
+        )
+        self.mock_resolver.resolve_service.reset_mock()
+
         # there should be an attempt to connect on port 443 for the .well-known
         clients = self.reactor.tcpClients
         self.assertEqual(len(clients), 1)
@@ -426,11 +429,10 @@ class MatrixFederationAgentTests(TestCase):
         self.assertEqual(port, 443)
 
         self._handle_well_known_connection(
-            client_factory, expected_sni=b"testserv",
-            content=b'{ "m.server": "target-server" }',
+            client_factory, expected_sni=b"testserv", target_server=b"target-server",
         )
 
-        # there should be a SRV lookup
+        # there should be another SRV lookup
         self.mock_resolver.resolve_service.assert_called_once_with(
             b"_matrix._tcp.target-server",
         )
@@ -481,6 +483,11 @@ class MatrixFederationAgentTests(TestCase):
         # Nothing happened yet
         self.assertNoResult(test_d)
 
+        self.mock_resolver.resolve_service.assert_called_once_with(
+            b"_matrix._tcp.testserv",
+        )
+        self.mock_resolver.resolve_service.reset_mock()
+
         # there should be an attempt to connect on port 443 for the .well-known
         clients = self.reactor.tcpClients
         self.assertEqual(len(clients), 1)
@@ -522,7 +529,7 @@ class MatrixFederationAgentTests(TestCase):
 
         self.reactor.pump((0.1, ))
 
-        # there should be a SRV lookup
+        # there should be another SRV lookup
         self.mock_resolver.resolve_service.assert_called_once_with(
             b"_matrix._tcp.target-server",
         )
@@ -560,64 +567,6 @@ class MatrixFederationAgentTests(TestCase):
         self.well_known_cache.expire()
         self.assertNotIn(b"testserv", self.well_known_cache)
 
-    def test_get_invalid_well_known(self):
-        """
-        Test the behaviour when the server name has an *invalid* well-known (and no SRV)
-        """
-
-        self.mock_resolver.resolve_service.side_effect = lambda _: []
-        self.reactor.lookups["testserv"] = "1.2.3.4"
-
-        test_d = self._make_get_request(b"matrix://testserv/foo/bar")
-
-        # Nothing happened yet
-        self.assertNoResult(test_d)
-
-        # No SRV record lookup yet
-        self.mock_resolver.resolve_service.assert_not_called()
-
-        # there should be an attempt to connect on port 443 for the .well-known
-        clients = self.reactor.tcpClients
-        self.assertEqual(len(clients), 1)
-        (host, port, client_factory, _timeout, _bindAddress) = clients.pop()
-        self.assertEqual(host, '1.2.3.4')
-        self.assertEqual(port, 443)
-
-        self._handle_well_known_connection(
-            client_factory, expected_sni=b"testserv", content=b'NOT JSON',
-        )
-
-        # now there should be a SRV lookup
-        self.mock_resolver.resolve_service.assert_called_once_with(
-            b"_matrix._tcp.testserv",
-        )
-
-        # we should fall back to a direct connection
-        self.assertEqual(len(clients), 1)
-        (host, port, client_factory, _timeout, _bindAddress) = clients.pop()
-        self.assertEqual(host, '1.2.3.4')
-        self.assertEqual(port, 8448)
-
-        # make a test server, and wire up the client
-        http_server = self._make_connection(
-            client_factory,
-            expected_sni=b'testserv',
-        )
-
-        self.assertEqual(len(http_server.requests), 1)
-        request = http_server.requests[0]
-        self.assertEqual(request.method, b'GET')
-        self.assertEqual(request.path, b'/foo/bar')
-        self.assertEqual(
-            request.requestHeaders.getRawHeaders(b'host'),
-            [b'testserv'],
-        )
-
-        # finish the request
-        request.finish()
-        self.reactor.pump((0.1,))
-        self.successResultOf(test_d)
-
     def test_get_hostname_srv(self):
         """
         Test the behaviour when there is a single SRV record
@@ -632,7 +581,6 @@ class MatrixFederationAgentTests(TestCase):
         # Nothing happened yet
         self.assertNoResult(test_d)
 
-        # the request for a .well-known will have failed with a DNS lookup error.
         self.mock_resolver.resolve_service.assert_called_once_with(
             b"_matrix._tcp.testserv",
         )
@@ -665,9 +613,11 @@ class MatrixFederationAgentTests(TestCase):
         self.successResultOf(test_d)
 
     def test_get_well_known_srv(self):
-        """Test the behaviour when the .well-known redirects to a place where there
-        is a SRV.
+        """Test the behaviour when the server name has no port and no SRV record, but
+        the .well-known redirects to a place where there is a SRV.
         """
+
+        self.mock_resolver.resolve_service.side_effect = lambda _: []
         self.reactor.lookups["testserv"] = "1.2.3.4"
         self.reactor.lookups["srvtarget"] = "5.6.7.8"
 
@@ -676,6 +626,11 @@ class MatrixFederationAgentTests(TestCase):
         # Nothing happened yet
         self.assertNoResult(test_d)
 
+        self.mock_resolver.resolve_service.assert_called_once_with(
+            b"_matrix._tcp.testserv",
+        )
+        self.mock_resolver.resolve_service.reset_mock()
+
         # there should be an attempt to connect on port 443 for the .well-known
         clients = self.reactor.tcpClients
         self.assertEqual(len(clients), 1)
@@ -688,11 +643,10 @@ class MatrixFederationAgentTests(TestCase):
         ]
 
         self._handle_well_known_connection(
-            client_factory, expected_sni=b"testserv",
-            content=b'{ "m.server": "target-server" }',
+            client_factory, expected_sni=b"testserv", target_server=b"target-server",
         )
 
-        # there should be a SRV lookup
+        # there should be another SRV lookup
         self.mock_resolver.resolve_service.assert_called_once_with(
             b"_matrix._tcp.target-server",
         )
@@ -737,8 +691,9 @@ class MatrixFederationAgentTests(TestCase):
         # Nothing happened yet
         self.assertNoResult(test_d)
 
-        # No SRV record lookup yet
-        self.mock_resolver.resolve_service.assert_not_called()
+        self.mock_resolver.resolve_service.assert_called_once_with(
+            b"_matrix._tcp.xn--bcher-kva.com",
+        )
 
         # there should be an attempt to connect on port 443 for the .well-known
         clients = self.reactor.tcpClients
@@ -754,11 +709,6 @@ class MatrixFederationAgentTests(TestCase):
         # .well-known request fails.
         self.reactor.pump((0.4,))
 
-        # now there should have been a SRV lookup
-        self.mock_resolver.resolve_service.assert_called_once_with(
-            b"_matrix._tcp.xn--bcher-kva.com",
-        )
-
         # We should fall back to port 8448
         clients = self.reactor.tcpClients
         self.assertEqual(len(clients), 2)
@@ -856,7 +806,7 @@ class MatrixFederationAgentTests(TestCase):
             client_factory,
             expected_sni=b"testserv",
             response_headers={b'Cache-Control': b'max-age=10'},
-            content=b'{ "m.server": "target-server" }',
+            target_server=b"target-server",
         )
 
         r = self.successResultOf(fetch_d)
@@ -884,7 +834,7 @@ class MatrixFederationAgentTests(TestCase):
         self._handle_well_known_connection(
             client_factory,
             expected_sni=b"testserv",
-            content=b'{ "m.server": "other-server" }',
+            target_server=b"other-server",
         )
 
         r = self.successResultOf(fetch_d)