Unsecure -> insecure

typos
Default PL100 to enable encryption in a room (#7230 )
2020-04-13 14:02:34 +01:00 · 2020-04-11 20:55:18 +01:00 · 2020-04-09 18:45:38 +01:00 · 2020-04-09 13:28:13 -04:00 · 2020-04-09 13:53:35 +01:00 · 2020-04-09 12:49:05 +01:00
258 changed files with 6577 additions and 2831 deletions
--- a/.buildkite/worker-blacklist
+++ b/.buildkite/worker-blacklist
@@ -5,8 +5,6 @@ Message history can be paginated

 Can re-join room if re-invited

-/upgrade creates a new room
-
 The only membership state included in an initial sync is for all the senders in the timeline

 Local device key changes get to remote servers
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,3 +1,40 @@
+Next version
+============
+
+* A new template (`sso_auth_confirm.html`) was added to Synapse. If your Synapse
+  is configured to use SSO and a custom `sso_redirect_confirm_template_dir`
+  configuration then this template will need to be duplicated into that
+  directory.
+
+Synapse 1.12.3 (2020-04-03)
+===========================
+
+- Remove the the pin to Pillow 7.0 which was introduced in Synapse 1.12.2, and
+correctly fix the issue with building the Debian packages. ([\#7212](https://github.com/matrix-org/synapse/issues/7212))
+
+Synapse 1.12.2 (2020-04-02)
+===========================
+
+This release works around [an issue](https://github.com/matrix-org/synapse/issues/7208) with building the debian packages.
+
+No other significant changes since 1.12.1.
+
+Synapse 1.12.1 (2020-04-02)
+===========================
+
+No significant changes since 1.12.1rc1.
+
+
+Synapse 1.12.1rc1 (2020-03-31)
+==============================
+
+Bugfixes
+--------
+
+- Fix starting workers when federation sending not split out. ([\#7133](https://github.com/matrix-org/synapse/issues/7133)). Introduced in v1.12.0.
+- Avoid importing `sqlite3` when using the postgres backend. Contributed by David Vo. ([\#7155](https://github.com/matrix-org/synapse/issues/7155)). Introduced in v1.12.0rc1.
+- Fix a bug which could cause outbound federation traffic to stop working if a client uploaded an incorrect e2e device signature. ([\#7177](https://github.com/matrix-org/synapse/issues/7177)). Introduced in v1.11.0.
+
 Synapse 1.12.0 (2020-03-23)
 ===========================

--- a/INSTALL.md
+++ b/INSTALL.md
@@ -2,7 +2,6 @@
 - [Installing Synapse](#installing-synapse)
  - [Installing from source](#installing-from-source)
    - [Platform-Specific Instructions](#platform-specific-instructions)
-    - [Troubleshooting Installation](#troubleshooting-installation)
  - [Prebuilt packages](#prebuilt-packages)
 - [Setting up Synapse](#setting-up-synapse)
  - [TLS certificates](#tls-certificates)
@@ -10,6 +9,7 @@
  - [Registering a user](#registering-a-user)
  - [Setting up a TURN server](#setting-up-a-turn-server)
  - [URL previews](#url-previews)
+- [Troubleshooting Installation](#troubleshooting-installation)

 # Choosing your server name

@@ -36,7 +36,7 @@ that your email address is probably `user@example.com` rather than
 System requirements:

 - POSIX-compliant system (tested on Linux & OS X)
- Python 3.5, 3.6, 3.7 or 3.8.
+- Python 3.5.2 or later, up to Python 3.8.
 - At least 1GB of free RAM if you want to join large public rooms like #matrix:matrix.org

 Synapse is written in Python but some of the libraries it uses are written in
@@ -70,7 +70,7 @@ pip install -U matrix-synapse
 ```

 Before you can start Synapse, you will need to generate a configuration
-file. To do this, run (in your virtualenv, as before)::
+file. To do this, run (in your virtualenv, as before):

 ```
 cd ~/synapse
@@ -84,22 +84,24 @@ python -m synapse.app.homeserver \
 ... substituting an appropriate value for `--server-name`.

 This command will generate you a config file that you can then customise, but it will
-also generate a set of keys for you. These keys will allow your Home Server to
-identify itself to other Home Servers, so don't lose or delete them. It would be
+also generate a set of keys for you. These keys will allow your homeserver to
+identify itself to other homeserver, so don't lose or delete them. It would be
 wise to back them up somewhere safe. (If, for whatever reason, you do need to
-change your Home Server's keys, you may find that other Home Servers have the
+change your homeserver's keys, you may find that other homeserver have the
 old key cached. If you update the signing key, you should change the name of the
 key in the `<server name>.signing.key` file (the second word) to something
 different. See the
 [spec](https://matrix.org/docs/spec/server_server/latest.html#retrieving-server-keys)
-for more information on key management.)
+for more information on key management).

 To actually run your new homeserver, pick a working directory for Synapse to
-run (e.g. `~/synapse`), and::
+run (e.g. `~/synapse`), and:

-    cd ~/synapse
-    source env/bin/activate
-    synctl start
+```
+cd ~/synapse
+source env/bin/activate
+synctl start
+```

 ### Platform-Specific Instructions

@@ -110,7 +112,7 @@ Installing prerequisites on Ubuntu or Debian:
 ```
 sudo apt-get install build-essential python3-dev libffi-dev \
                     python3-pip python3-setuptools sqlite3 \
-                     libssl-dev python3-virtualenv libjpeg-dev libxslt1-dev
+                     libssl-dev virtualenv libjpeg-dev libxslt1-dev
 ```

 #### ArchLinux
@@ -188,7 +190,7 @@ doas pkg_add python libffi py-pip py-setuptools sqlite3 py-virtualenv \
 There is currently no port for OpenBSD. Additionally, OpenBSD's security
 settings require a slightly more difficult installation process.

-XXX: I suspect this is out of date.
+(XXX: I suspect this is out of date)

 1. Create a new directory in `/usr/local` called `_synapse`. Also, create a
   new user called `_synapse` and set that directory as the new user's home.
@@ -196,7 +198,7 @@ XXX: I suspect this is out of date.
   write and execute permissions on the same memory space to be run from
   `/usr/local`.
 2. `su` to the new `_synapse` user and change to their home directory.
-3. Create a new virtualenv: `virtualenv -p python2.7 ~/.synapse`
+3. Create a new virtualenv: `virtualenv -p python3 ~/.synapse`
 4. Source the virtualenv configuration located at
   `/usr/local/_synapse/.synapse/bin/activate`. This is done in `ksh` by
   using the `.` command, rather than `bash`'s `source`.
@@ -217,45 +219,6 @@ be found at https://docs.microsoft.com/en-us/windows/wsl/install-win10 for
 Windows 10 and https://docs.microsoft.com/en-us/windows/wsl/install-on-server
 for Windows Server.

-### Troubleshooting Installation
-
-XXX a bunch of this is no longer relevant.
-
-Synapse requires pip 8 or later, so if your OS provides too old a version you
-may need to manually upgrade it::
-
-    sudo pip install --upgrade pip
-
-Installing may fail with `Could not find any downloads that satisfy the requirement pymacaroons-pynacl (from matrix-synapse==0.12.0)`.
-You can fix this by manually upgrading pip and virtualenv::
-
-    sudo pip install --upgrade virtualenv
-
-You can next rerun `virtualenv -p python3 synapse` to update the virtual env.
-
-Installing may fail during installing virtualenv with `InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.`
-You can fix this  by manually installing ndg-httpsclient::
-
-    pip install --upgrade ndg-httpsclient
-
-Installing may fail with `mock requires setuptools>=17.1. Aborting installation`.
-You can fix this by upgrading setuptools::
-
-    pip install --upgrade setuptools
-
-If pip crashes mid-installation for reason (e.g. lost terminal), pip may
-refuse to run until you remove the temporary installation directory it
-created. To reset the installation::
-
-    rm -rf /tmp/pip_install_matrix
-
-pip seems to leak *lots* of memory during installation.  For instance, a Linux
-host with 512MB of RAM may run out of memory whilst installing Twisted.  If this
-happens, you will have to individually install the dependencies which are
-failing, e.g.::
-
-    pip install twisted
-
 ## Prebuilt packages

 As an alternative to installing from source, prebuilt packages are available
@@ -314,7 +277,7 @@ For `buster` and `sid`, Synapse is available in the Debian repositories and
 it should be possible to install it with simply:

 ```
-    sudo apt install matrix-synapse
+sudo apt install matrix-synapse
 ```

 There is also a version of `matrix-synapse` in `stretch-backports`. Please see
@@ -375,15 +338,17 @@ sudo pip install py-bcrypt

 Synapse can be found in the void repositories as 'synapse':

-    xbps-install -Su
-    xbps-install -S synapse
+```
+xbps-install -Su
+xbps-install -S synapse
+```

 ### FreeBSD

 Synapse can be installed via FreeBSD Ports or Packages contributed by Brendan Molloy from:

 - Ports: `cd /usr/ports/net-im/py-matrix-synapse && make install clean`
- - Packages: `pkg install py27-matrix-synapse`
+ - Packages: `pkg install py37-matrix-synapse`


 ### NixOS
@@ -420,6 +385,7 @@ so, you will need to edit `homeserver.yaml`, as follows:
      resources:
        - names: [client, federation]
  ```
+
 * You will also need to uncomment the `tls_certificate_path` and
  `tls_private_key_path` lines under the `TLS` section. You can either
  point these settings at an existing certificate and key, or you can
@@ -427,15 +393,15 @@ so, you will need to edit `homeserver.yaml`, as follows:
  for having Synapse automatically provision and renew federation
  certificates through ACME can be found at [ACME.md](docs/ACME.md).
  Note that, as pointed out in that document, this feature will not
-  work with installs set up after November 2019. 
-  
+  work with installs set up after November 2019.
+
  If you are using your own certificate, be sure to use a `.pem` file that
  includes the full certificate chain including any intermediate certificates
  (for instance, if using certbot, use `fullchain.pem` as your certificate, not
  `cert.pem`).

 For a more detailed guide to configuring your server for federation, see
-[federate.md](docs/federate.md)
+[federate.md](docs/federate.md).


 ## Email
@@ -482,7 +448,7 @@ on your server even if `enable_registration` is `false`.
 ## Setting up a TURN server

 For reliable VoIP calls to be routed via this homeserver, you MUST configure
-a TURN server.  See [docs/turn-howto.md](docs/turn-howto.md) for details.
+a TURN server. See [docs/turn-howto.md](docs/turn-howto.md) for details.

 ## URL previews

@@ -491,10 +457,24 @@ turn it on you must enable the `url_preview_enabled: True` config parameter
 and explicitly specify the IP ranges that Synapse is not allowed to spider for
 previewing in the `url_preview_ip_range_blacklist` configuration parameter.
 This is critical from a security perspective to stop arbitrary Matrix users
-spidering 'internal' URLs on your network.  At the very least we recommend that
+spidering 'internal' URLs on your network. At the very least we recommend that
 your loopback and RFC1918 IP addresses are blacklisted.

-This also requires the optional lxml and netaddr python dependencies to be
-installed.  This in turn requires the libxml2 library to be available - on
+This also requires the optional `lxml` and `netaddr` python dependencies to be
+installed. This in turn requires the `libxml2` library to be available - on
 Debian/Ubuntu this means `apt-get install libxml2-dev`, or equivalent for
 your OS.
+
+# Troubleshooting Installation
+
+`pip` seems to leak *lots* of memory during installation. For instance, a Linux
+host with 512MB of RAM may run out of memory whilst installing Twisted. If this
+happens, you will have to individually install the dependencies which are
+failing, e.g.:
+
+```
+pip install twisted
+```
+
+If you have any other problems, feel free to ask in
+[#synapse:matrix.org](https://matrix.to/#/#synapse:matrix.org).
--- a/changelog.d/6446.misc
+++ b/changelog.d/6446.misc
@@ -0,0 +1 @@
+Add benchmarks for LruCache.
--- a/changelog.d/6573.bugfix
+++ b/changelog.d/6573.bugfix
@@ -0,0 +1 @@
+Don't attempt to use an invalid sqlite config if no database configuration is provided. Contributed by @nekatak.
--- a/changelog.d/6634.bugfix
+++ b/changelog.d/6634.bugfix
@@ -0,0 +1 @@
+Fix single-sign on with CAS systems: pass the same service URL when requesting the CAS ticket and when calling the `proxyValidate` URL. Contributed by @Naugrimm.
--- a/changelog.d/6639.bugfix
+++ b/changelog.d/6639.bugfix
@@ -0,0 +1 @@
+Fix missing field `default` when fetching user-defined push rules.
--- a/changelog.d/6892.doc
+++ b/changelog.d/6892.doc
@@ -0,0 +1 @@
+Update Debian installation instructions to recommend installing the `virtualenv` package instead of `python3-virtualenv`.
--- a/changelog.d/6899.bugfix
+++ b/changelog.d/6899.bugfix
@@ -0,0 +1 @@
+Improve error responses when accessing remote public room lists.
--- a/changelog.d/6946.bugfix
+++ b/changelog.d/6946.bugfix
@@ -0,0 +1 @@
+Transfer alias mappings on room upgrade.
--- a/changelog.d/6988.doc
+++ b/changelog.d/6988.doc
@@ -0,0 +1 @@
+Improve the documentation for database configuration.
--- a/changelog.d/7006.feature
+++ b/changelog.d/7006.feature
@@ -0,0 +1 @@
+Extend the `web_client_location` option to accept an absolute URL to use as a redirect. Adds a warning when running the web client on the same hostname as homeserver. Contributed by Martin Milata.
--- a/changelog.d/7009.feature
+++ b/changelog.d/7009.feature
@@ -0,0 +1 @@
+Set `Referrer-Policy` header to `no-referrer` on media downloads.
--- a/changelog.d/7010.misc
+++ b/changelog.d/7010.misc
@@ -0,0 +1 @@
+Change device list streams to have one row per ID.
--- a/changelog.d/7011.misc
+++ b/changelog.d/7011.misc
@@ -0,0 +1 @@
+Remove concept of a non-limited stream.
--- a/changelog.d/7024.misc
+++ b/changelog.d/7024.misc
@@ -0,0 +1 @@
+Move catchup of replication streams logic to worker.
--- a/changelog.d/7051.feature
+++ b/changelog.d/7051.feature
@@ -0,0 +1 @@
+Admin API `POST /_synapse/admin/v1/join/<roomIdOrAlias>` to join users to a room like `auto_join_rooms` for creation of users.
--- a/changelog.d/7068.bugfix
+++ b/changelog.d/7068.bugfix
@@ -0,0 +1 @@
+Ensure that a user inteactive authentication session is tied to a single request.
--- a/changelog.d/7089.bugfix
+++ b/changelog.d/7089.bugfix
@@ -0,0 +1 @@
+Fix a bug in the federation API which could cause occasional "Failed to get PDU" errors.
--- a/changelog.d/7096.feature
+++ b/changelog.d/7096.feature
@@ -0,0 +1 @@
+Add options to prevent users from changing their profile or associated 3PIDs.
--- a/changelog.d/7102.feature
+++ b/changelog.d/7102.feature
@@ -0,0 +1 @@
+Support SSO in the user interactive authentication workflow.
--- a/changelog.d/7107.doc
+++ b/changelog.d/7107.doc
@@ -0,0 +1 @@
+Update pre-built package name for FreeBSD.
--- a/changelog.d/7109.bugfix
+++ b/changelog.d/7109.bugfix
@@ -0,0 +1 @@
+Return the proper error (M_BAD_ALIAS) when a non-existant canonical alias is provided.
--- a/changelog.d/7110.misc
+++ b/changelog.d/7110.misc
@@ -0,0 +1 @@
+Convert some of synapse.rest.media to async/await.
--- a/changelog.d/7115.misc
+++ b/changelog.d/7115.misc
@@ -0,0 +1 @@
+De-duplicate / remove unused REST code for login and auth.
--- a/changelog.d/7116.misc
+++ b/changelog.d/7116.misc
@@ -0,0 +1 @@
+Convert `*StreamRow` classes to inner classes.
--- a/changelog.d/7117.bugfix
+++ b/changelog.d/7117.bugfix
@@ -0,0 +1 @@
+Fix a bug which meant that groups updates were not correctly replicated between workers.
--- a/changelog.d/7118.feature
+++ b/changelog.d/7118.feature
@@ -0,0 +1 @@
+Allow server admins to define and enforce a password policy (MSC2000).
--- a/changelog.d/7119.doc
+++ b/changelog.d/7119.doc
@@ -0,0 +1 @@
+Update postgres docs with login troubleshooting information.
--- a/changelog.d/7120.misc
+++ b/changelog.d/7120.misc
@@ -0,0 +1 @@
+Clean up some LoggingContext code.
--- a/changelog.d/7128.misc
+++ b/changelog.d/7128.misc
@@ -0,0 +1 @@
+Add explicit `instance_id` for USER_SYNC commands and remove implicit `conn_id` usage.
--- a/changelog.d/7133.bugfix
+++ b/changelog.d/7133.bugfix
@@ -0,0 +1 @@
+Fix starting workers when federation sending not split out.
--- a/changelog.d/7136.misc
+++ b/changelog.d/7136.misc
@@ -0,0 +1 @@
+Refactored the CAS authentication logic to a separate class.
--- a/changelog.d/7137.removal
+++ b/changelog.d/7137.removal
@@ -0,0 +1 @@
+Remove nonfunctional `captcha_bypass_secret` option from `homeserver.yaml`.
--- a/changelog.d/7141.doc
+++ b/changelog.d/7141.doc
@@ -0,0 +1 @@
+Clean up INSTALL.md a bit.
--- a/changelog.d/7147.doc
+++ b/changelog.d/7147.doc
@@ -0,0 +1 @@
+Add documentation for running a local CAS server for testing.
--- a/changelog.d/7150.bugfix
+++ b/changelog.d/7150.bugfix
@@ -0,0 +1 @@
+Ensure `is_verified` is a boolean in responses to `GET /_matrix/client/r0/room_keys/keys`. Also warn the user if they forgot the `version` query param.
--- a/changelog.d/7151.bugfix
+++ b/changelog.d/7151.bugfix
@@ -0,0 +1 @@
+Fix error page being shown when a custom SAML handler attempted to redirect when processing an auth response.
--- a/changelog.d/7152.feature
+++ b/changelog.d/7152.feature
@@ -0,0 +1 @@
+Improve the support for SSO authentication on the login fallback page.
--- a/changelog.d/7153.feature
+++ b/changelog.d/7153.feature
@@ -0,0 +1 @@
+Always whitelist the login fallback in the SSO configuration if `public_baseurl` is set.
--- a/changelog.d/7155.bugfix
+++ b/changelog.d/7155.bugfix
@@ -0,0 +1 @@
+Avoid importing `sqlite3` when using the postgres backend. Contributed by David Vo.
--- a/changelog.d/7157.misc
+++ b/changelog.d/7157.misc
@@ -0,0 +1 @@
+Add tests for outbound device pokes.
--- a/changelog.d/7158.misc
+++ b/changelog.d/7158.misc
@@ -0,0 +1 @@
+Fix device list update stream ids going backward.
--- a/changelog.d/7159.bugfix
+++ b/changelog.d/7159.bugfix
@@ -0,0 +1 @@
+Fix excessive CPU usage by `prune_old_outbound_device_pokes` job.
--- a/changelog.d/7160.feature
+++ b/changelog.d/7160.feature
@@ -0,0 +1 @@
+Always send users their own device updates.
--- a/changelog.d/7167.doc
+++ b/changelog.d/7167.doc
@@ -0,0 +1 @@
+Improve README.md by being explicit about public IP recommendation for TURN relaying.
--- a/changelog.d/7171.doc
+++ b/changelog.d/7171.doc
@@ -0,0 +1 @@
+Fix a small typo in the `metrics_flags` config option.
--- a/changelog.d/7177.bugfix
+++ b/changelog.d/7177.bugfix
@@ -0,0 +1 @@
+Fix a bug which could cause outbound federation traffic to stop working if a client uploaded an incorrect e2e device signature.
--- a/changelog.d/7178.bugfix
+++ b/changelog.d/7178.bugfix
@@ -0,0 +1 @@
+Fix a bug which could cause incorrect 'cyclic dependency' error.
--- a/changelog.d/7181.misc
+++ b/changelog.d/7181.misc
@@ -0,0 +1 @@
+Clean up some LoggingContext code.
--- a/changelog.d/7183.misc
+++ b/changelog.d/7183.misc
@@ -0,0 +1 @@
+Clean up some LoggingContext code.
--- a/changelog.d/7184.misc
+++ b/changelog.d/7184.misc
@@ -0,0 +1 @@
+Convert some of synapse.rest.media to async/await.
--- a/changelog.d/7185.misc
+++ b/changelog.d/7185.misc
@@ -0,0 +1 @@
+Move client command handling out of TCP protocol.
--- a/changelog.d/7186.feature
+++ b/changelog.d/7186.feature
@@ -0,0 +1 @@
+Support SSO in the user interactive authentication workflow.
--- a/changelog.d/7187.misc
+++ b/changelog.d/7187.misc
@@ -0,0 +1 @@
+Move server command handling out of TCP protocol.
--- a/changelog.d/7188.misc
+++ b/changelog.d/7188.misc
@@ -0,0 +1 @@
+Fix consistency of HTTP status codes reported in log lines.
--- a/changelog.d/7190.misc
+++ b/changelog.d/7190.misc
@@ -0,0 +1 @@
+Only run one background database update at a time.
--- a/changelog.d/7191.feature
+++ b/changelog.d/7191.feature
@@ -0,0 +1 @@
+Admin users are no longer required to be in a room to create an alias for it.
--- a/changelog.d/7192.misc
+++ b/changelog.d/7192.misc
@@ -0,0 +1 @@
+Remove sent outbound device list pokes from the database.
--- a/changelog.d/7193.misc
+++ b/changelog.d/7193.misc
@@ -0,0 +1 @@
+Add a background database update job to clear out duplicate `device_lists_outbound_pokes`.
--- a/changelog.d/7195.misc
+++ b/changelog.d/7195.misc
@@ -0,0 +1 @@
+Move catchup of replication streams logic to worker.
--- a/changelog.d/7199.bugfix
+++ b/changelog.d/7199.bugfix
@@ -0,0 +1 @@
+Fix a bug that could cause a user to be invited to a server notices (aka System Alerts) room without any notice being sent.
--- a/changelog.d/7203.bugfix
+++ b/changelog.d/7203.bugfix
@@ -0,0 +1 @@
+Fix some worker-mode replication handling not being correctly recorded in CPU usage stats.
--- a/changelog.d/7207.misc
+++ b/changelog.d/7207.misc
@@ -0,0 +1 @@
+Remove some extraneous debugging log lines.
--- a/changelog.d/7219.misc
+++ b/changelog.d/7219.misc
@@ -0,0 +1 @@
+Add typing information to federation server code.
--- a/changelog.d/7226.misc
+++ b/changelog.d/7226.misc
@@ -0,0 +1 @@
+Move catchup of replication streams logic to worker.
--- a/changelog.d/7228.misc
+++ b/changelog.d/7228.misc
@@ -0,0 +1 @@
+Unblacklist '/upgrade creates a new room' sytest for workers.
--- a/changelog.d/7230.feature
+++ b/changelog.d/7230.feature
@@ -0,0 +1 @@
+Require admin privileges to enable room encryption by default. This does not affect existing rooms.
--- a/changelog.d/7233.misc
+++ b/changelog.d/7233.misc
@@ -0,0 +1 @@
+Remove redundant checks on `daemonize` from synctl.
--- a/changelog.d/7234.doc
+++ b/changelog.d/7234.doc
@@ -0,0 +1 @@
+Update the contributed documentation on managing synapse workers with systemd, and bring it into the core distribution.
--- a/changelog.d/7235.feature
+++ b/changelog.d/7235.feature
@@ -0,0 +1 @@
+Improve the support for SSO authentication on the login fallback page.
--- a/changelog.d/7236.misc
+++ b/changelog.d/7236.misc
@@ -0,0 +1 @@
+Upgrade jQuery to v3.4.1 on fallback login/registration pages.
--- a/changelog.d/7237.misc
+++ b/changelog.d/7237.misc
@@ -0,0 +1 @@
+Change log line that told user to implement onLogin/onRegister fallback js functions to a warning, instead of an info, so it's more visible.
--- a/changelog.d/7238.doc
+++ b/changelog.d/7238.doc
@@ -0,0 +1 @@
+Add documentation to the `password_providers` config option. Add known password provider implementations to docs.
--- a/changelog.d/7240.bugfix
+++ b/changelog.d/7240.bugfix
@@ -0,0 +1 @@
+Do not allow a deactivated user to login via SSO.
--- a/changelog.d/7241.misc
+++ b/changelog.d/7241.misc
@@ -0,0 +1 @@
+Convert some of synapse.rest.media to async/await.
--- a/changelog.d/7248.doc
+++ b/changelog.d/7248.doc
@@ -0,0 +1 @@
+Add documentation to the `password_providers` config option. Add known password provider implementations to docs.
--- a/changelog.d/7249.bugfix
+++ b/changelog.d/7249.bugfix
@@ -0,0 +1 @@
+Fix --help command-line argument.
--- a/changelog.d/7329.misc
+++ b/changelog.d/7329.misc
@@ -0,0 +1 @@
+Move catchup of replication streams logic to worker.
--- a/contrib/systemd-with-workers/README.md
+++ b/contrib/systemd-with-workers/README.md
@@ -1,150 +1,2 @@
-# Setup Synapse with Workers and Systemd
-
-This is a setup for managing synapse with systemd including support for
-managing workers. It provides a `matrix-synapse`, as well as a
-`matrix-synapse-worker@` service for any workers you require. Additionally to
-group the required services it sets up a `matrix.target`. You can use this to
-automatically start any bot- or bridge-services. More on this in
-[Bots and Bridges](#bots-and-bridges).
-
-See the folder [system](system) for any service and target files.
-
-The folder [workers](workers) contains an example configuration for the
-`federation_reader` worker. Pay special attention to the name of the
-configuration file. In order to work with the `matrix-synapse-worker@.service`
-service, it needs to have the exact same name as the worker app.
-
-This setup expects neither the homeserver nor any workers to fork. Forking is
-handled by systemd.
-
-## Setup
-
-1. Adjust your matrix configs. Make sure that the worker config files have the
-exact same name as the worker app. Compare `matrix-synapse-worker@.service` for
-why. You can find an example worker config in the [workers](workers) folder. See
-below for relevant settings in the `homeserver.yaml`.
-2. Copy the `*.service` and `*.target` files in [system](system) to
-`/etc/systemd/system`.
-3. `systemctl enable matrix-synapse.service` this adds the homeserver
-app to the `matrix.target`
-4. *Optional.* `systemctl enable
-matrix-synapse-worker@federation_reader.service` this adds the federation_reader
-app to the `matrix-synapse.service`
-5. *Optional.* Repeat step 4 for any additional workers you require.
-6. *Optional.* Add any bots or bridges by enabling them.
-7. Start all matrix related services via `systemctl start matrix.target`
-8. *Optional.* Enable autostart of all matrix related services on system boot
-via `systemctl enable matrix.target`
-
-## Usage
-
-After you have setup you can use the following commands to manage your synapse
-installation:
-
-```
-# Start matrix-synapse, all workers and any enabled bots or bridges.
-systemctl start matrix.target
-
-# Restart matrix-synapse and all workers (not necessarily restarting bots
-# or bridges, see "Bots and Bridges")
-systemctl restart matrix-synapse.service
-
-# Stop matrix-synapse and all workers (not necessarily restarting bots
-# or bridges, see "Bots and Bridges")
-systemctl stop matrix-synapse.service
-
-# Restart a specific worker (i. e. federation_reader), the homeserver is
-# unaffected by this.
-systemctl restart matrix-synapse-worker@federation_reader.service
-
-# Add a new worker (assuming all configs are setup already)
-systemctl enable matrix-synapse-worker@federation_writer.service
-systemctl restart matrix-synapse.service
-```
-
-## The Configs
-
-Make sure the `worker_app` is set in the `homeserver.yaml` and it does not fork.
-
-```
-worker_app: synapse.app.homeserver 
-daemonize: false
-```
-
-None of the workers should fork, as forking is handled by systemd. Hence make
-sure this is present in all worker config files.
-
-```
-worker_daemonize: false
-```
-
-The config files of all workers are expected to be located in
-`/etc/matrix-synapse/workers`. If you want to use a different location you have
-to edit the provided `*.service` files accordingly.  
-
-## Bots and Bridges
-
-Most bots and bridges do not care if the homeserver goes down or is restarted.
-Depending on the implementation this may crash them though. So look up the docs
-or ask the community of the specific bridge or bot you want to run to make sure
-you choose the correct setup.
-
-Whichever configuration you choose, after the setup the following will enable
-automatically starting (and potentially restarting) your bot/bridge with the
-`matrix.target`.
-
-```
-systemctl enable <yourBotOrBridgeName>.service
-```
-
-**Note** that from an inactive synapse the bots/bridges will only be started with
-synapse if you start the `matrix.target`, not if you start the
-`matrix-synapse.service`. This is on purpose. Think of `matrix-synapse.service`
-as *just* synapse, but `matrix.target` being anything matrix related, including
-synapse and any and all enabled bots and bridges.
-
-### Start with synapse but ignore synapse going down
-
-If the bridge can handle shutdowns of the homeserver you'll want to install the
-service in the `matrix.target` and optionally add a
-`After=matrix-synapse.service` dependency to have the bot/bridge start after
-synapse on starting everything.
-
-In this case the service file should look like this.
-
-```
-[Unit]
-# ...
-# Optional, this will only ensure that if you start everything, synapse will
-# be started before the bot/bridge will be started.
-After=matrix-synapse.service
-
-[Service]
-# ...
-
-[Install]
-WantedBy=matrix.target
-```
-
-### Stop/restart when synapse stops/restarts
-
-If the bridge can't handle shutdowns of the homeserver you'll still want to
-install the service in the `matrix.target` but also have to specify the
-`After=matrix-synapse.service` *and* `BindsTo=matrix-synapse.service`
-dependencies to have the bot/bridge stop/restart with synapse.
-
-In this case the service file should look like this.
-
-```
-[Unit]
-# ...
-# Mandatory
-After=matrix-synapse.service
-BindsTo=matrix-synapse.service
-
-[Service]
-# ...
-
-[Install]
-WantedBy=matrix.target
-```
+The documentation for using systemd to manage synapse workers is now part of
+the main synapse distribution. See [docs/systemd-with-workers](../../docs/systemd-with-workers).
--- a/contrib/systemd-with-workers/system/matrix-synapse-worker@.service
+++ b/contrib/systemd-with-workers/system/matrix-synapse-worker@.service
@@ -1,19 +0,0 @@
-[Unit]
-Description=Synapse Matrix Worker
-After=matrix-synapse.service
-BindsTo=matrix-synapse.service
-
-[Service]
-Type=notify
-NotifyAccess=main
-User=matrix-synapse
-WorkingDirectory=/var/lib/matrix-synapse
-EnvironmentFile=/etc/default/matrix-synapse
-ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.%i --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --config-path=/etc/matrix-synapse/workers/%i.yaml
-ExecReload=/bin/kill -HUP $MAINPID
-Restart=always
-RestartSec=3
-SyslogIdentifier=matrix-synapse-%i
-
-[Install]
-WantedBy=matrix-synapse.service
--- a/contrib/systemd-with-workers/system/matrix.target
+++ b/contrib/systemd-with-workers/system/matrix.target
@@ -1,7 +0,0 @@
-[Unit]
-Description=Contains matrix services like synapse, bridges and bots
-After=network.target
-AllowIsolate=no
-
-[Install]
-WantedBy=multi-user.target
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,32 @@
+matrix-synapse-py3 (1.12.3ubuntu1) UNRELEASED; urgency=medium
+
+  * Add information about .well-known files to Debian installation scripts.
+
+ -- Patrick Cloke <patrickc@matrix.org>  Mon, 06 Apr 2020 10:10:38 -0400
+
+matrix-synapse-py3 (1.12.3) stable; urgency=medium
+
+  [ Richard van der Hoff ]
+  * Update the Debian build scripts to handle the new installation paths
+   for the support libraries introduced by Pillow 7.1.1.
+
+  [ Synapse Packaging team ]
+  * New synapse release 1.12.3.
+
+ -- Synapse Packaging team <packages@matrix.org>  Fri, 03 Apr 2020 10:55:03 +0100
+
+matrix-synapse-py3 (1.12.2) stable; urgency=medium
+
+  * New synapse release 1.12.2.
+
+ -- Synapse Packaging team <packages@matrix.org>  Mon, 02 Apr 2020 19:02:17 +0000
+
+matrix-synapse-py3 (1.12.1) stable; urgency=medium
+
+  * New synapse release 1.12.1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Mon, 02 Apr 2020 11:30:47 +0000
+
 matrix-synapse-py3 (1.12.0) stable; urgency=medium

  * New synapse release 1.12.0.
--- a/debian/po/templates.pot
+++ b/debian/po/templates.pot
@@ -1,14 +1,14 @@
 # SOME DESCRIPTIVE TITLE.
 # Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER
-# This file is distributed under the same license as the matrix-synapse package.
+# This file is distributed under the same license as the matrix-synapse-py3 package.
 # FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
 #
 #, fuzzy
 msgid ""
 msgstr ""
-"Project-Id-Version: matrix-synapse\n"
-"Report-Msgid-Bugs-To: matrix-synapse@packages.debian.org\n"
-"POT-Creation-Date: 2017-02-21 07:51+0000\n"
+"Project-Id-Version: matrix-synapse-py3\n"
+"Report-Msgid-Bugs-To: matrix-synapse-py3@packages.debian.org\n"
+"POT-Creation-Date: 2020-04-06 16:39-0400\n"
 "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
 "Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
 "Language-Team: LANGUAGE <LL@li.org>\n"
@@ -28,7 +28,10 @@ msgstr ""
 #: ../templates:1001
 msgid ""
 "The name that this homeserver will appear as, to clients and other servers "
-"via federation. This name should match the SRV record published in DNS."
+"via federation. This is normally the public hostname of the server running "
+"synapse, but can be different if you set up delegation. Please refer to the "
+"delegation documentation in this case: https://github.com/matrix-org/synapse/"
+"blob/master/docs/delegate.md."
 msgstr ""

 #. Type: boolean
--- a/debian/rules
+++ b/debian/rules
@@ -15,17 +15,38 @@ override_dh_installinit:
 # we don't really want to strip the symbols from our object files.
 override_dh_strip:

+# dh_shlibdeps calls dpkg-shlibdeps, which finds all the binary files
+# (executables and shared libs) in the package, and looks for the shared
+# libraries that they depend on. It then adds a dependency on the package that
+# contains that library to the package.
+#
+# We make two modifications to that process...
+#
 override_dh_shlibdeps:
-        # make the postgres package's dependencies a recommendation
-        # rather than a hard dependency.
+        # Firstly, postgres is not a hard dependency for us, so we want to make
+        # the things that psycopg2 depends on (such as libpq) be
+        # recommendations rather than hard dependencies. We do so by
+        # running dpkg-shlibdeps manually on psycopg2's libs.
+        #
 	find debian/$(PACKAGE_NAME)/ -path '*/site-packages/psycopg2/*.so' | \
 	    xargs dpkg-shlibdeps -Tdebian/$(PACKAGE_NAME).substvars \
 	        -pshlibs1 -dRecommends

-        # all the other dependencies can be normal 'Depends' requirements,
-        # except for PIL's, which is self-contained and which confuses
-        # dpkg-shlibdeps.
-	dh_shlibdeps -X site-packages/PIL/.libs -X site-packages/psycopg2
+        # secondly, we exclude PIL's libraries from the process. They are known
+        # to be self-contained, but they have interdependencies and
+        # dpkg-shlibdeps doesn't know how to resolve them.
+        #
+        # As of Pillow 7.1.0, these libraries are in
+        # site-packages/Pillow.libs. Previously, they were in
+        # site-packages/PIL/.libs.
+        #
+        # (we also need to exclude psycopg2, of course, since we've already
+        # dealt with that.)
+        #
+	dh_shlibdeps \
+	    -X site-packages/PIL/.libs \
+	    -X site-packages/Pillow.libs \
+	    -X site-packages/psycopg2

 override_dh_virtualenv:
 	./debian/build_virtualenv
--- a/debian/templates
+++ b/debian/templates
@@ -2,8 +2,10 @@ Template: matrix-synapse/server-name
 Type: string
 _Description: Name of the server:
 The name that this homeserver will appear as, to clients and other
- servers via federation. This name should match the SRV record
- published in DNS.
+ servers via federation. This is normally the public hostname of the
+ server running synapse, but can be different if you set up delegation.
+ Please refer to the delegation documentation in this case:
+ https://github.com/matrix-org/synapse/blob/master/docs/delegate.md.

 Template: matrix-synapse/report-stats
 Type: boolean
--- a/docs/admin_api/room_membership.md
+++ b/docs/admin_api/room_membership.md
@@ -0,0 +1,34 @@
+# Edit Room Membership API
+
+This API allows an administrator to join an user account with a given `user_id`
+to a room with a given `room_id_or_alias`. You can only modify the membership of
+local users. The server administrator must be in the room and have permission to
+invite users.
+
+## Parameters
+
+The following parameters are available:
+
+* `user_id` - Fully qualified user: for example, `@user:server.com`.
+* `room_id_or_alias` - The room identifier or alias to join: for example,
+  `!636q39766251:server.com`.
+
+## Usage
+
+```
+POST /_synapse/admin/v1/join/<room_id_or_alias>
+
+{
+  "user_id": "@user:server.com"
+}
+```
+
+Including an `access_token` of a server admin.
+
+Response:
+
+```
+{
+  "room_id": "!636q39766251:server.com"
+}
+```
--- a/docs/dev/cas.md
+++ b/docs/dev/cas.md
@@ -0,0 +1,64 @@
+# How to test CAS as a developer without a server
+
+The [django-mama-cas](https://github.com/jbittel/django-mama-cas) project is an
+easy to run CAS implementation built on top of Django.
+
+## Prerequisites
+
+1. Create a new virtualenv: `python3 -m venv <your virtualenv>`
+2. Activate your virtualenv: `source /path/to/your/virtualenv/bin/activate`
+3. Install Django and django-mama-cas:
+   ```
+   python -m pip install "django<3" "django-mama-cas==2.4.0"
+   ```
+4. Create a Django project in the current directory:
+   ```
+   django-admin startproject cas_test .
+   ```
+5. Follow the [install directions](https://django-mama-cas.readthedocs.io/en/latest/installation.html#configuring) for django-mama-cas
+6. Setup the SQLite database: `python manage.py migrate`
+7. Create a user:
+   ```
+   python manage.py createsuperuser
+   ```
+   1. Use whatever you want as the username and password.
+   2. Leave the other fields blank.
+8. Use the built-in Django test server to serve the CAS endpoints on port 8000:
+   ```
+   python manage.py runserver
+   ```
+
+You should now have a Django project configured to serve CAS authentication with
+a single user created.
+
+## Configure Synapse (and Riot) to use CAS
+
+1. Modify your `homeserver.yaml` to enable CAS and point it to your locally
+   running Django test server:
+   ```yaml
+   cas_config:
+     enabled: true
+     server_url: "http://localhost:8000"
+     service_url: "http://localhost:8081"
+     #displayname_attribute: name
+     #required_attributes:
+     #    name: value
+   ```
+2. Restart Synapse.
+
+Note that the above configuration assumes the homeserver is running on port 8081
+and that the CAS server is on port 8000, both on localhost.
+
+## Testing the configuration
+
+Then in Riot:
+
+1. Visit the login page with a Riot pointing at your homeserver.
+2. Click the Single Sign-On button.
+3. Login using the credentials created with `createsuperuser`.
+4. You should be logged in.
+
+If you want to repeat this process you'll need to manually logout first:
+
+1. http://localhost:8000/admin/
+2. Click "logout" in the top right.
--- a/docs/dev/saml.md
+++ b/docs/dev/saml.md
@@ -18,9 +18,13 @@ To make Synapse (and therefore Riot) use it:
       metadata:
         local: ["samling.xml"]   
   ```
-5. Run `apt-get install xmlsec1` and `pip install --upgrade --force 'pysaml2>=4.5.0'` to ensure
+5. Ensure that your `homeserver.yaml` has a setting for `public_baseurl`:
+   ```yaml
+   public_baseurl: http://localhost:8080/
+   ```
+6. Run `apt-get install xmlsec1` and `pip install --upgrade --force 'pysaml2>=4.5.0'` to ensure
   the dependencies are installed and ready to go.
-6. Restart Synapse.
+7. Restart Synapse.

 Then in Riot:

--- a/docs/log_contexts.md
+++ b/docs/log_contexts.md
@@ -29,14 +29,13 @@ from synapse.logging import context         # omitted from future snippets
 def handle_request(request_id):
    request_context = context.LoggingContext()

-    calling_context = context.LoggingContext.current_context()
-    context.LoggingContext.set_current_context(request_context)
+    calling_context = context.set_current_context(request_context)
    try:
        request_context.request = request_id
        do_request_handling()
        logger.debug("finished")
    finally:
-        context.LoggingContext.set_current_context(calling_context)
+        context.set_current_context(calling_context)

 def do_request_handling():
    logger.debug("phew")  # this will be logged against request_id
--- a/docs/password_auth_providers.md
+++ b/docs/password_auth_providers.md
@@ -9,7 +9,11 @@ into Synapse, and provides a number of methods by which it can integrate
 with the authentication system.

 This document serves as a reference for those looking to implement their
-own password auth providers.
+own password auth providers. Additionally, here is a list of known
+password auth provider module implementations:
+
+* [matrix-synapse-ldap3](https://github.com/matrix-org/matrix-synapse-ldap3/)
+* [matrix-synapse-shared-secret-auth](https://github.com/devture/matrix-synapse-shared-secret-auth)

 ## Required methods

--- a/docs/postgres.md
+++ b/docs/postgres.md
@@ -61,7 +61,33 @@ Note that the PostgreSQL database *must* have the correct encoding set

 You may need to enable password authentication so `synapse_user` can
 connect to the database. See
-<https://www.postgresql.org/docs/11/auth-pg-hba-conf.html>.
+<https://www.postgresql.org/docs/current/auth-pg-hba-conf.html>.
+
+If you get an error along the lines of `FATAL:  Ident authentication failed for
+user "synapse_user"`, you may need to use an authentication method other than
+`ident`:
+
+* If the `synapse_user` user has a password, add the password to the `database:`
+  section of `homeserver.yaml`. Then add the following to `pg_hba.conf`:
+
+  ```
+  host    synapse     synapse_user    ::1/128     md5  # or `scram-sha-256` instead of `md5` if you use that
+  ```
+
+* If the `synapse_user` user does not have a password, then a password doesn't
+  have to be added to `homeserver.yaml`. But the following does need to be added
+  to `pg_hba.conf`:
+
+  ```
+  host    synapse     synapse_user    ::1/128     trust
+  ```
+
+Note that line order matters in `pg_hba.conf`, so make sure that if you do add a
+new line, it is inserted before:
+
+```
+host    all         all             ::1/128     ident
+```

 ### Fixing incorrect `COLLATE` or `CTYPE`

@@ -72,8 +98,7 @@ underneath the database, or if a different version of the locale is used on any
 replicas.

 The safest way to fix the issue is to take a dump and recreate the database with
-the correct `COLLATE` and `CTYPE` parameters (as per
-[docs/postgres.md](docs/postgres.md)). It is also possible to change the
+the correct `COLLATE` and `CTYPE` parameters (as shown above). It is also possible to change the
 parameters on a live database and run a `REINDEX` on the entire database,
 however extreme care must be taken to avoid database corruption.

@@ -105,19 +130,41 @@ of free memory the database host has available.
 When you are ready to start using PostgreSQL, edit the `database`
 section in your config file to match the following lines:

-    database:
-        name: psycopg2
-        args:
-            user: <user>
-            password: <pass>
-            database: <db>
-            host: <host>
-            cp_min: 5
-            cp_max: 10
+```yaml
+database:
+  name: psycopg2
+  args:
+    user: <user>
+    password: <pass>
+    database: <db>
+    host: <host>
+    cp_min: 5
+    cp_max: 10
+```

 All key, values in `args` are passed to the `psycopg2.connect(..)`
 function, except keys beginning with `cp_`, which are consumed by the
-twisted adbapi connection pool.
+twisted adbapi connection pool. See the [libpq
+documentation](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS)
+for a list of options which can be passed.
+
+You should consider tuning the `args.keepalives_*` options if there is any danger of
+the connection between your homeserver and database dropping, otherwise Synapse
+may block for an extended period while it waits for a response from the
+database server. Example values might be:
+
+```yaml
+# seconds of inactivity after which TCP should send a keepalive message to the server
+keepalives_idle: 10
+
+# the number of seconds after which a TCP keepalive message that is not
+# acknowledged by the server should be retransmitted
+keepalives_interval: 10
+
+# the number of TCP keepalives that can be lost before the client's connection
+# to the server is considered dead
+keepalives_count: 3
+```

 ## Porting from SQLite

--- a/docs/sample_config.yaml
+++ b/docs/sample_config.yaml
@@ -33,10 +33,15 @@ server_name: "SERVERNAME"
 #
 pid_file: DATADIR/homeserver.pid

-# The path to the web client which will be served at /_matrix/client/
-# if 'webclient' is configured under the 'listeners' configuration.
+# The absolute URL to the web client which /_matrix/client will redirect
+# to if 'webclient' is configured under the 'listeners' configuration.
 #
-#web_client_location: "/path/to/web/root"
+# This option can be also set to the filesystem path to the web client
+# which will be served at /_matrix/client/ if 'webclient' is configured
+# under the 'listeners' configuration, however this is a security risk:
+# https://github.com/matrix-org/synapse#security-note
+#
+#web_client_location: https://riot.example.com/

 # The public-facing base URL that clients use to access this HS
 # (not including _matrix/...). This is the same URL a user would
@@ -578,13 +583,46 @@ acme:

 ## Database ##

+# The 'database' setting defines the database that synapse uses to store all of
+# its data.
+#
+# 'name' gives the database engine to use: either 'sqlite3' (for SQLite) or
+# 'psycopg2' (for PostgreSQL).
+#
+# 'args' gives options which are passed through to the database engine,
+# except for options starting 'cp_', which are used to configure the Twisted
+# connection pool. For a reference to valid arguments, see:
+#   * for sqlite: https://docs.python.org/3/library/sqlite3.html#sqlite3.connect
+#   * for postgres: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS
+#   * for the connection pool: https://twistedmatrix.com/documents/current/api/twisted.enterprise.adbapi.ConnectionPool.html#__init__
+#
+#
+# Example SQLite configuration:
+#
+#database:
+#  name: sqlite3
+#  args:
+#    database: /path/to/homeserver.db
+#
+#
+# Example Postgres configuration:
+#
+#database:
+#  name: psycopg2
+#  args:
+#    user: synapse
+#    password: secretpassword
+#    database: synapse
+#    host: localhost
+#    cp_min: 5
+#    cp_max: 10
+#
+# For more information on using Synapse with Postgres, see `docs/postgres.md`.
+#
 database:
-  # The database engine name
-  name: "sqlite3"
-  # Arguments to pass to the engine
+  name: sqlite3
  args:
-    # Path to the database
-    database: "DATADIR/homeserver.db"
+    database: DATADIR/homeserver.db

 # Number of events to cache in memory.
 #
@@ -839,10 +877,6 @@ media_store_path: "DATADIR/media_store"
 #
 #enable_registration_captcha: false

-# A secret key used to bypass the captcha test entirely.
-#
-#captcha_bypass_secret: "YOUR_SECRET_HERE"
-
 # The API endpoint to use for verifying m.login.recaptcha responses.
 #
 #recaptcha_siteverify_api: "https://www.recaptcha.net/recaptcha/api/siteverify"
@@ -1057,6 +1091,29 @@ account_threepid_delegates:
    #email: https://example.com     # Delegate email sending to example.com
    #msisdn: http://localhost:8090  # Delegate SMS sending to this local process

+# Whether users are allowed to change their displayname after it has
+# been initially set. Useful when provisioning users based on the
+# contents of a third-party directory.
+#
+# Does not apply to server administrators. Defaults to 'true'
+#
+#enable_set_displayname: false
+
+# Whether users are allowed to change their avatar after it has been
+# initially set. Useful when provisioning users based on the contents
+# of a third-party directory.
+#
+# Does not apply to server administrators. Defaults to 'true'
+#
+#enable_set_avatar_url: false
+
+# Whether users can change the 3PIDs associated with their accounts
+# (email address and msisdn).
+#
+# Defaults to 'true'
+#
+#enable_3pid_changes: false
+
 # Users who register on this homeserver will automatically be joined
 # to these rooms
 #
@@ -1092,7 +1149,7 @@ account_threepid_delegates:
 # enabled by default, either for performance reasons or limited use.
 #
 metrics_flags:
-    # Publish synapse_federation_known_servers, a g auge of the number of
+    # Publish synapse_federation_known_servers, a gauge of the number of
    # servers this homeserver knows about, including itself. May cause
    # performance problems on large homeservers.
    #
@@ -1392,6 +1449,10 @@ sso:
    # phishing attacks from evil.site. To avoid this, include a slash after the
    # hostname: "https://my.client/".
    #
+    # If public_baseurl is set, then the login fallback page (used by clients
+    # that don't natively support the required login flows) is whitelisted in
+    # addition to any URLs in this list.
+    #
    # By default, this list is empty.
    #
    #client_whitelist:
@@ -1453,6 +1514,41 @@ password_config:
   #
   #pepper: "EVEN_MORE_SECRET"

+   # Define and enforce a password policy. Each parameter is optional.
+   # This is an implementation of MSC2000.
+   #
+   policy:
+      # Whether to enforce the password policy.
+      # Defaults to 'false'.
+      #
+      #enabled: true
+
+      # Minimum accepted length for a password.
+      # Defaults to 0.
+      #
+      #minimum_length: 15
+
+      # Whether a password must contain at least one digit.
+      # Defaults to 'false'.
+      #
+      #require_digit: true
+
+      # Whether a password must contain at least one symbol.
+      # A symbol is any character that's not a number or a letter.
+      # Defaults to 'false'.
+      #
+      #require_symbol: true
+
+      # Whether a password must contain at least one lowercase letter.
+      # Defaults to 'false'.
+      #
+      #require_lowercase: true
+
+      # Whether a password must contain at least one lowercase letter.
+      # Defaults to 'false'.
+      #
+      #require_uppercase: true
+

 # Configuration for sending emails from Synapse.
 #
@@ -1561,7 +1657,19 @@ email:
  #template_dir: "res/templates"


-#password_providers:
+# Password providers allow homeserver administrators to integrate
+# their Synapse installation with existing authentication methods
+# ex. LDAP, external tokens, etc.
+#
+# For more information and known implementations, please see
+# https://github.com/matrix-org/synapse/blob/master/docs/password_auth_providers.md
+#
+# Note: instances wishing to use SAML or CAS authentication should
+# instead use the `saml2_config` or `cas_config` options,
+# respectively.
+#
+password_providers:
+#    # Example config for an LDAP auth provider
 #    - module: "ldap_auth_provider.LdapAuthProvider"
 #      config:
 #        enabled: true
--- a/docs/systemd-with-workers/README.md
+++ b/docs/systemd-with-workers/README.md
@@ -0,0 +1,67 @@
+# Setting up Synapse with Workers and Systemd
+
+This is a setup for managing synapse with systemd, including support for
+managing workers. It provides a `matrix-synapse` service for the master, as
+well as a `matrix-synapse-worker@` service template for any workers you
+require. Additionally, to group the required services, it sets up a
+`matrix-synapse.target`.
+
+See the folder [system](system) for the systemd unit files.
+
+The folder [workers](workers) contains an example configuration for the
+`federation_reader` worker.
+
+## Synapse configuration files
+
+See [workers.md](../workers.md) for information on how to set up the
+configuration files and reverse-proxy correctly. You can find an example worker
+config in the [workers](workers) folder.
+
+Systemd manages daemonization itself, so ensure that none of the configuration
+files set either `daemonize` or `worker_daemonize`.
+
+The config files of all workers are expected to be located in
+`/etc/matrix-synapse/workers`. If you want to use a different location, edit
+the provided `*.service` files accordingly.
+
+There is no need for a separate configuration file for the master process.
+
+## Set up
+
+1. Adjust synapse configuration files as above.
+1. Copy the `*.service` and `*.target` files in [system](system) to
+`/etc/systemd/system`.
+1. Run `systemctl deamon-reload` to tell systemd to load the new unit files.
+1. Run `systemctl enable matrix-synapse.service`. This will configure the
+synapse master process to be started as part of the `matrix-synapse.target`
+target.
+1. For each worker process to be enabled, run `systemctl enable
+matrix-synapse-worker@<worker_name>.service`. For each `<worker_name>`, there
+should be a corresponding configuration file
+`/etc/matrix-synapse/workers/<worker_name>.yaml`.
+1. Start all the synapse processes with `systemctl start matrix-synapse.target`.
+1. Tell systemd to start synapse on boot with `systemctl enable matrix-synapse.target`/
+
+## Usage
+
+Once the services are correctly set up, you can use the following commands
+to manage your synapse installation:
+
+```sh
+# Restart Synapse master and all workers
+systemctl restart matrix-synapse.target
+
+# Stop Synapse and all workers
+systemctl stop matrix-synapse.target
+
+# Restart the master alone
+systemctl start matrix-synapse.service
+
+# Restart a specific worker (eg. federation_reader); the master is
+# unaffected by this.
+systemctl restart matrix-synapse-worker@federation_reader.service
+
+# Add a new worker (assuming all configs are set up already)
+systemctl enable matrix-synapse-worker@federation_writer.service
+systemctl restart matrix-synapse.target
+```
--- a/docs/systemd-with-workers/system/matrix-synapse-worker@.service
+++ b/docs/systemd-with-workers/system/matrix-synapse-worker@.service
@@ -0,0 +1,20 @@
+[Unit]
+Description=Synapse %i
+
+# This service should be restarted when the synapse target is restarted.
+PartOf=matrix-synapse.target
+
+[Service]
+Type=notify
+NotifyAccess=main
+User=matrix-synapse
+WorkingDirectory=/var/lib/matrix-synapse
+EnvironmentFile=/etc/default/matrix-synapse
+ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.generic_worker --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --config-path=/etc/matrix-synapse/workers/%i.yaml
+ExecReload=/bin/kill -HUP $MAINPID
+Restart=always
+RestartSec=3
+SyslogIdentifier=matrix-synapse-%i
+
+[Install]
+WantedBy=matrix-synapse.target
--- a/contrib/systemd-with-workers/system/matrix-synapse.service
+++ b/contrib/systemd-with-workers/system/matrix-synapse.service
@@ -1,5 +1,8 @@
 [Unit]
-Description=Synapse Matrix Homeserver
+Description=Synapse master
+
+# This service should be restarted when the synapse target is restarted.
+PartOf=matrix-synapse.target

 [Service]
 Type=notify
@@ -15,4 +18,4 @@ RestartSec=3
 SyslogIdentifier=matrix-synapse

 [Install]
-WantedBy=matrix.target
+WantedBy=matrix-synapse.target
--- a/docs/systemd-with-workers/system/matrix-synapse.target
+++ b/docs/systemd-with-workers/system/matrix-synapse.target
@@ -0,0 +1,6 @@
+[Unit]
+Description=Synapse parent target
+After=network.target
+
+[Install]
+WantedBy=multi-user.target
--- a/contrib/systemd-with-workers/workers/federation_reader.yaml
+++ b/contrib/systemd-with-workers/workers/federation_reader.yaml
@@ -10,5 +10,4 @@ worker_listeners:
      resources:
          - names: [federation]

-worker_daemonize: false
 worker_log_config: /etc/matrix-synapse/federation-reader-log.yaml
--- a/docs/tcp_replication.md
+++ b/docs/tcp_replication.md
@@ -14,16 +14,16 @@ example flow would be (where '>' indicates master to worker and
 '<' worker to master flows):

    > SERVER example.com
-    < REPLICATE events 53
+    < REPLICATE
+    > POSITION events 53
    > RDATA events 54 ["$foo1:bar.com", ...]
    > RDATA events 55 ["$foo4:bar.com", ...]

-The example shows the server accepting a new connection and sending its
-identity with the `SERVER` command, followed by the client asking to
-subscribe to the `events` stream from the token `53`. The server then
-periodically sends `RDATA` commands which have the format
-`RDATA <stream_name> <token> <row>`, where the format of `<row>` is
-defined by the individual streams.
+The example shows the server accepting a new connection and sending its identity
+with the `SERVER` command, followed by the client server to respond with the
+position of all streams. The server then periodically sends `RDATA` commands
+which have the format `RDATA <stream_name> <token> <row>`, where the format of
+`<row>` is defined by the individual streams.

 Error reporting happens by either the client or server sending an ERROR
 command, and usually the connection will be closed.
@@ -32,9 +32,6 @@ Since the protocol is a simple line based, its possible to manually
 connect to the server using a tool like netcat. A few things should be
 noted when manually using the protocol:

-   When subscribing to a stream using `REPLICATE`, the special token
-    `NOW` can be used to get all future updates. The special stream name
-    `ALL` can be used with `NOW` to subscribe to all available streams.
 -   The federation stream is only available if federation sending has
    been disabled on the main process.
 -   The server will only time connections out that have sent a `PING`
@@ -91,9 +88,7 @@ The client:
 -   Sends a `NAME` command, allowing the server to associate a human
    friendly name with the connection. This is optional.
 -   Sends a `PING` as above
-   For each stream the client wishes to subscribe to it sends a
-    `REPLICATE` with the `stream_name` and token it wants to subscribe
-    from.
+-   Sends a `REPLICATE` to get the current position of all streams.
 -   On receipt of a `SERVER` command, checks that the server name
    matches the expected server name.

@@ -140,9 +135,7 @@ the wire:
    > PING 1490197665618
    < NAME synapse.app.appservice
    < PING 1490197665618
-    < REPLICATE events 1
-    < REPLICATE backfill 1
-    < REPLICATE caches 1
+    < REPLICATE
    > POSITION events 1
    > POSITION backfill 1
    > POSITION caches 1
@@ -181,9 +174,9 @@ client (C):

 #### POSITION (S)

-   The position of the stream has been updated. Sent to the client
-    after all missing updates for a stream have been sent to the client
-    and they're now up to date.
+   On receipt of a POSITION command clients should check if they have missed any
+   updates, and if so then fetch them out of band. Sent in response to a
+   REPLICATE command (but can happen at any time).

 #### ERROR (S, C)

@@ -199,25 +192,18 @@ client (C):

 #### REPLICATE (C)

-Asks the server to replicate a given stream. The syntax is:
-
-```
-    REPLICATE <stream_name> <token>
-```
-
-Where `<token>` may be either:
- * a numeric stream_id to stream updates since (exclusive)
- * `NOW` to stream all subsequent updates.
-
-The `<stream_name>` is the name of a replication stream to subscribe
-to (see [here](../synapse/replication/tcp/streams/_base.py) for a list
-of streams). It can also be `ALL` to subscribe to all known streams,
-in which case the `<token>` must be set to `NOW`.
+Asks the server for the current position of all streams.

 #### USER_SYNC (C)

   A user has started or stopped syncing

+#### CLEAR_USER_SYNC (C)
+
+   The server should clear all associated user sync data from the worker.
+
+   This is used when a worker is shutting down.
+
 #### FEDERATION_ACK (C)

   Acknowledge receipt of some federation data
--- a/docs/turn-howto.md
+++ b/docs/turn-howto.md
@@ -11,6 +11,13 @@ TURN server.

 The following sections describe how to install [coturn](<https://github.com/coturn/coturn>) (which implements the TURN REST API) and integrate it with synapse.

+## Requirements
+
+For TURN relaying with `coturn` to work, it must be hosted on a server/endpoint with a public IP.
+
+Hosting TURN behind a NAT (even with appropriate port forwarding) is known to cause issues
+and to often not work.
+
 ## `coturn` Setup

 ### Initial installation
--- a/Show More
+++ b/Show More
				`@@ -0,0 +1 @@`
				`Don't attempt to use an invalid sqlite config if no database configuration is provided. Contributed by @nekatak.`
				`@@ -0,0 +1 @@`
				Fix single-sign on with CAS systems: pass the same service URL when requesting the CAS ticket and when calling the `proxyValidate` URL. Contributed by @Naugrimm.
				`@@ -0,0 +1 @@`
				Fix missing field `default` when fetching user-defined push rules.
				`@@ -0,0 +1 @@`
				Update Debian installation instructions to recommend installing the `virtualenv` package instead of `python3-virtualenv`.
				`@@ -0,0 +1 @@`
				`Improve error responses when accessing remote public room lists.`
				`@@ -0,0 +1 @@`
				`Improve the documentation for database configuration.`
				`@@ -0,0 +1 @@`
				Extend the `web_client_location` option to accept an absolute URL to use as a redirect. Adds a warning when running the web client on the same hostname as homeserver. Contributed by Martin Milata.
				`@@ -0,0 +1 @@`
				Set `Referrer-Policy` header to `no-referrer` on media downloads.
				`@@ -0,0 +1 @@`
				`Change device list streams to have one row per ID.`
				`@@ -0,0 +1 @@`
				`Move catchup of replication streams logic to worker.`