update sample config

.buildkite/pipeline.yml

@@ -26,6 +26,16 @@ steps:
       - docker#v3.0.1:
           image: "python:3.6"
 
+  - command:
+      - "python -m pip install tox"
+      - "scripts-dev/check-newsfragment"
+    label: ":newspaper: Newsfile"
+    branches: "!master !develop !release-*"
+    plugins:
+      - docker#v3.0.1:
+          image: "python:3.6"
+          propagate-environment: true
+
   - wait
 
   - command:

CHANGES.md

@@ -1,110 +1,3 @@
-Synapse 0.99.4 (2019-05-15)
-===========================
-
-No significant changes.
-
-
-Synapse 0.99.4rc1 (2019-05-13)
-==============================
-
-Features
---------
-
-- Add systemd-python to the optional dependencies to enable logging to the systemd journal. Install with `pip install matrix-synapse[systemd]`. ([\#4339](https://github.com/matrix-org/synapse/issues/4339))
-- Add a default .m.rule.tombstone push rule. ([\#4867](https://github.com/matrix-org/synapse/issues/4867))
-- Add ability for password provider modules to bind email addresses to users upon registration. ([\#4947](https://github.com/matrix-org/synapse/issues/4947))
-- Implementation of [MSC1711](https://github.com/matrix-org/matrix-doc/pull/1711) including config options for requiring valid TLS certificates for federation traffic, the ability to disable TLS validation for specific domains, and the ability to specify your own list of CA certificates. ([\#4967](https://github.com/matrix-org/synapse/issues/4967))
-- Remove presence list support as per MSC 1819. ([\#4989](https://github.com/matrix-org/synapse/issues/4989))
-- Reduce CPU usage starting pushers during start up. ([\#4991](https://github.com/matrix-org/synapse/issues/4991))
-- Add a delete group admin API. ([\#5002](https://github.com/matrix-org/synapse/issues/5002))
-- Add config option to block users from looking up 3PIDs. ([\#5010](https://github.com/matrix-org/synapse/issues/5010))
-- Add context to phonehome stats. ([\#5020](https://github.com/matrix-org/synapse/issues/5020))
-- Configure the example systemd units to have a log identifier of `matrix-synapse`
-  instead of the executable name, `python`.
-  Contributed by Christoph Müller. ([\#5023](https://github.com/matrix-org/synapse/issues/5023))
-- Add time-based account expiration. ([\#5027](https://github.com/matrix-org/synapse/issues/5027), [\#5047](https://github.com/matrix-org/synapse/issues/5047), [\#5073](https://github.com/matrix-org/synapse/issues/5073), [\#5116](https://github.com/matrix-org/synapse/issues/5116))
-- Add support for handling `/versions`, `/voip` and `/push_rules` client endpoints to client_reader worker. ([\#5063](https://github.com/matrix-org/synapse/issues/5063), [\#5065](https://github.com/matrix-org/synapse/issues/5065), [\#5070](https://github.com/matrix-org/synapse/issues/5070))
-- Add a configuration option to require authentication on /publicRooms and /profile endpoints. ([\#5083](https://github.com/matrix-org/synapse/issues/5083))
-- Move admin APIs to `/_synapse/admin/v1`. (The old paths are retained for backwards-compatibility, for now). ([\#5119](https://github.com/matrix-org/synapse/issues/5119))
-- Implement an admin API for sending server notices. Many thanks to @krombel who provided a foundation for this work. ([\#5121](https://github.com/matrix-org/synapse/issues/5121), [\#5142](https://github.com/matrix-org/synapse/issues/5142))
-
-
-Bugfixes
---------
-
-- Avoid redundant URL encoding of redirect URL for SSO login in the fallback login page. Fixes a regression introduced in [#4220](https://github.com/matrix-org/synapse/pull/4220). Contributed by Marcel Fabian Krüger ("[zaugin](https://github.com/zauguin)"). ([\#4555](https://github.com/matrix-org/synapse/issues/4555))
-- Fix bug where presence updates were sent to all servers in a room when a new server joined, rather than to just the new server. ([\#4942](https://github.com/matrix-org/synapse/issues/4942), [\#5103](https://github.com/matrix-org/synapse/issues/5103))
-- Fix sync bug which made accepting invites unreliable in worker-mode synapses. ([\#4955](https://github.com/matrix-org/synapse/issues/4955), [\#4956](https://github.com/matrix-org/synapse/issues/4956))
-- start.sh: Fix the --no-rate-limit option for messages and make it bypass rate limit on registration and login too. ([\#4981](https://github.com/matrix-org/synapse/issues/4981))
-- Transfer related groups on room upgrade. ([\#4990](https://github.com/matrix-org/synapse/issues/4990))
-- Prevent the ability to kick users from a room they aren't in. ([\#4999](https://github.com/matrix-org/synapse/issues/4999))
-- Fix issue #4596 so synapse_port_db script works with --curses option on Python 3. Contributed by Anders Jensen-Waud <anders@jensenwaud.com>. ([\#5003](https://github.com/matrix-org/synapse/issues/5003))
-- Clients timing out/disappearing while downloading from the media repository will now no longer log a spurious "Producer was not unregistered" message. ([\#5009](https://github.com/matrix-org/synapse/issues/5009))
-- Fix "cannot import name execute_batch" error with postgres. ([\#5032](https://github.com/matrix-org/synapse/issues/5032))
-- Fix disappearing exceptions in manhole. ([\#5035](https://github.com/matrix-org/synapse/issues/5035))
-- Workaround bug in twisted where attempting too many concurrent DNS requests could cause it to hang due to running out of file descriptors. ([\#5037](https://github.com/matrix-org/synapse/issues/5037))
-- Make sure we're not registering the same 3pid twice on registration. ([\#5071](https://github.com/matrix-org/synapse/issues/5071))
-- Don't crash on lack of expiry templates. ([\#5077](https://github.com/matrix-org/synapse/issues/5077))
-- Fix the ratelimiting on third party invites. ([\#5104](https://github.com/matrix-org/synapse/issues/5104))
-- Add some missing limitations to room alias creation. ([\#5124](https://github.com/matrix-org/synapse/issues/5124), [\#5128](https://github.com/matrix-org/synapse/issues/5128))
-- Limit the number of EDUs in transactions to 100 as expected by synapse. Thanks to @superboum for this work! ([\#5138](https://github.com/matrix-org/synapse/issues/5138))
-
-Internal Changes
-----------------
-
-- Add test to verify threepid auth check added in #4435. ([\#4474](https://github.com/matrix-org/synapse/issues/4474))
-- Fix/improve some docstrings in the replication code. ([\#4949](https://github.com/matrix-org/synapse/issues/4949))
-- Split synapse.replication.tcp.streams into smaller files. ([\#4953](https://github.com/matrix-org/synapse/issues/4953))
-- Refactor replication row generation/parsing. ([\#4954](https://github.com/matrix-org/synapse/issues/4954))
-- Run `black` to clean up formatting on `synapse/storage/roommember.py` and `synapse/storage/events.py`. ([\#4959](https://github.com/matrix-org/synapse/issues/4959))
-- Remove log line for password via the admin API. ([\#4965](https://github.com/matrix-org/synapse/issues/4965))
-- Fix typo in TLS filenames in docker/README.md. Also add the '-p' commandline option to the 'docker run' example. Contributed by Jurrie Overgoor. ([\#4968](https://github.com/matrix-org/synapse/issues/4968))
-- Refactor room version definitions. ([\#4969](https://github.com/matrix-org/synapse/issues/4969))
-- Reduce log level of .well-known/matrix/client responses. ([\#4972](https://github.com/matrix-org/synapse/issues/4972))
-- Add `config.signing_key_path` that can be read by `synapse.config` utility. ([\#4974](https://github.com/matrix-org/synapse/issues/4974))
-- Track which identity server is used when binding a threepid and use that for unbinding, as per MSC1915. ([\#4982](https://github.com/matrix-org/synapse/issues/4982))
-- Rewrite KeyringTestCase as a HomeserverTestCase. ([\#4985](https://github.com/matrix-org/synapse/issues/4985))
-- README updates: Corrected the default POSTGRES_USER. Added port forwarding hint in TLS section. ([\#4987](https://github.com/matrix-org/synapse/issues/4987))
-- Remove a number of unused tables from the database schema. ([\#4992](https://github.com/matrix-org/synapse/issues/4992), [\#5028](https://github.com/matrix-org/synapse/issues/5028), [\#5033](https://github.com/matrix-org/synapse/issues/5033))
-- Run `black` on the remainder of `synapse/storage/`. ([\#4996](https://github.com/matrix-org/synapse/issues/4996))
-- Fix grammar in get_current_users_in_room and give it a docstring. ([\#4998](https://github.com/matrix-org/synapse/issues/4998))
-- Clean up some code in the server-key Keyring. ([\#5001](https://github.com/matrix-org/synapse/issues/5001))
-- Convert SYNAPSE_NO_TLS Docker variable to boolean for user friendliness. Contributed by Gabriel Eckerson. ([\#5005](https://github.com/matrix-org/synapse/issues/5005))
-- Refactor synapse.storage._base._simple_select_list_paginate. ([\#5007](https://github.com/matrix-org/synapse/issues/5007))
-- Store the notary server name correctly in server_keys_json. ([\#5024](https://github.com/matrix-org/synapse/issues/5024))
-- Rewrite Datastore.get_server_verify_keys to reduce the number of database transactions. ([\#5030](https://github.com/matrix-org/synapse/issues/5030))
-- Remove extraneous period from copyright headers. ([\#5046](https://github.com/matrix-org/synapse/issues/5046))
-- Update documentation for where to get Synapse packages. ([\#5067](https://github.com/matrix-org/synapse/issues/5067))
-- Add workarounds for pep-517 install errors. ([\#5098](https://github.com/matrix-org/synapse/issues/5098))
-- Improve logging when event-signature checks fail. ([\#5100](https://github.com/matrix-org/synapse/issues/5100))
-- Factor out an "assert_requester_is_admin" function. ([\#5120](https://github.com/matrix-org/synapse/issues/5120))
-- Remove the requirement to authenticate for /admin/server_version. ([\#5122](https://github.com/matrix-org/synapse/issues/5122))
-- Prevent an exception from being raised in a IResolutionReceiver and use a more generic error message for blacklisted URL previews. ([\#5155](https://github.com/matrix-org/synapse/issues/5155))
-- Run `black` on the tests directory. ([\#5170](https://github.com/matrix-org/synapse/issues/5170))
-- Fix CI after new release of isort. ([\#5179](https://github.com/matrix-org/synapse/issues/5179))
-- Fix bogus imports in unit tests. ([\#5154](https://github.com/matrix-org/synapse/issues/5154))
-
-
-Synapse 0.99.3.2 (2019-05-03)
-=============================
-
-Internal Changes
-----------------
-
-- Ensure that we have `urllib3` <1.25, to resolve incompatibility with `requests`. ([\#5135](https://github.com/matrix-org/synapse/issues/5135))
-
-
-Synapse 0.99.3.1 (2019-05-03)
-=============================
-
-Security update
----------------
-
-This release includes two security fixes:
-
-- Switch to using a cryptographically-secure random number generator for token strings, ensuring they cannot be predicted by an attacker. Thanks to @opnsec for identifying and responsibly disclosing this issue! ([\#5133](https://github.com/matrix-org/synapse/issues/5133))
-- Blacklist 0.0.0.0 and :: by default for URL previews. Thanks to @opnsec for identifying and responsibly disclosing this issue too! ([\#5134](https://github.com/matrix-org/synapse/issues/5134))
-
 Synapse 0.99.3 (2019-04-01)
 ===========================
 

INSTALL.md

@@ -257,29 +257,18 @@ https://github.com/spantaleev/matrix-docker-ansible-deploy
 #### Matrix.org packages
 
 Matrix.org provides Debian/Ubuntu packages of the latest stable version of
-Synapse via https://packages.matrix.org/debian/. They are available for Debian
-9 (Stretch), Ubuntu 16.04 (Xenial), and later. To use them:
+Synapse via https://matrix.org/packages/debian/. To use them:
 
 ```
-sudo apt install -y lsb-release wget apt-transport-https
-sudo wget -O /usr/share/keyrings/matrix-org-archive-keyring.gpg https://packages.matrix.org/debian/matrix-org-archive-keyring.gpg
-echo "deb [signed-by=/usr/share/keyrings/matrix-org-archive-keyring.gpg] https://packages.matrix.org/debian/ $(lsb_release -cs) main" |
+sudo apt install -y lsb-release curl apt-transport-https
+echo "deb https://matrix.org/packages/debian `lsb_release -cs` main" |
     sudo tee /etc/apt/sources.list.d/matrix-org.list
+curl "https://matrix.org/packages/debian/repo-key.asc" |
+    sudo apt-key add -
 sudo apt update
 sudo apt install matrix-synapse-py3
 ```
 
-**Note**: if you followed a previous version of these instructions which
-recommended using `apt-key add` to add an old key from
-`https://matrix.org/packages/debian/`, you should note that this key has been
-revoked. You should remove the old key with `sudo apt-key remove
-C35EB17E1EAE708E6603A9B3AD0592FE47F0DF61`, and follow the above instructions to
-update your configuration.
-
-The fingerprint of the repository signing key (as shown by `gpg
-/usr/share/keyrings/matrix-org-archive-keyring.gpg`) is
-`AAF9AE843A7584B5A3E4CD2BCF45A512DE2DA058`.
-
 #### Downstream Debian/Ubuntu packages
 
 For `buster` and `sid`, Synapse is available in the Debian repositories and

MANIFEST.in

@@ -43,8 +43,3 @@ prune .buildkite
 
 exclude jenkins*
 recursive-exclude jenkins *.sh
-
-# FIXME: we shouldn't have these templates here
-recursive-include res/templates-dinsic *.css
-recursive-include res/templates-dinsic *.html
-recursive-include res/templates-dinsic *.txt

README.rst

@@ -173,7 +173,7 @@ Synapse offers two database engines:
  * `PostgreSQL <https://www.postgresql.org>`_
 
 By default Synapse uses SQLite in and doing so trades performance for convenience.
-SQLite is only recommended in Synapse for testing purposes or for servers with
+SQLite is only recommended in Synapse for testing purposes or for servers with 
 light workloads.
 
 Almost all installations should opt to use PostreSQL. Advantages include:
@@ -272,7 +272,7 @@ to install using pip and a virtualenv::
 
     virtualenv -p python3 env
     source env/bin/activate
-    python -m pip install --no-pep-517 -e .[all]
+    python -m pip install -e .[all]
 
 This will run a process of downloading and installing all the needed
 dependencies into a virtual env.

changelog.d/3484.misc

@@ -1 +0,0 @@
-Make /sync attempt to return device updates for both joined and invited users. Note that this doesn't currently work correctly due to other bugs.

changelog.d/4474.misc

@@ -0,0 +1 @@
+Add test to verify threepid auth check added in #4435.

changelog.d/4555.bugfix

@@ -0,0 +1 @@
+Avoid redundant URL encoding of redirect URL for SSO login in the fallback login page. Fixes a regression introduced in [#4220](https://github.com/matrix-org/synapse/pull/4220). Contributed by Marcel Fabian Krüger ("[zaugin](https://github.com/zauguin)").

changelog.d/4942.bugfix

@@ -0,0 +1 @@
+Fix bug where presence updates were sent to all servers in a room when a new server joined, rather than to just the new server.

changelog.d/4947.feature

@@ -0,0 +1 @@
+Add ability for password provider modules to bind email addresses to users upon registration.

changelog.d/4949.misc

@@ -0,0 +1 @@
+Fix/improve some docstrings in the replication code.

changelog.d/4953.misc

@@ -0,0 +1,2 @@
+Split synapse.replication.tcp.streams into smaller files.
+

changelog.d/4954.misc

@@ -0,0 +1 @@
+Refactor replication row generation/parsing.

changelog.d/4955.bugfix

@@ -0,0 +1 @@
+Fix sync bug which made accepting invites unreliable in worker-mode synapses.

changelog.d/4956.bugfix

@@ -0,0 +1 @@
+Fix sync bug which made accepting invites unreliable in worker-mode synapses.

changelog.d/4959.misc

@@ -0,0 +1 @@
+Run `black` to clean up formatting on `synapse/storage/roommember.py` and `synapse/storage/events.py`.

changelog.d/4965.misc

@@ -0,0 +1 @@
+Remove log line for password via the admin API.

changelog.d/4968.misc

@@ -0,0 +1 @@
+Fix typo in TLS filenames in docker/README.md. Also add the '-p' commandline option to the 'docker run' example. Contributed by Jurrie Overgoor.

changelog.d/4969.misc

@@ -0,0 +1,2 @@
+Refactor room version definitions.
+

changelog.d/4974.misc

@@ -0,0 +1 @@
+Add `config.signing_key_path` that can be read by `synapse.config` utility.

changelog.d/4981.bugfix

@@ -0,0 +1 @@
+start.sh: Fix the --no-rate-limit option for messages and make it bypass rate limit on registration and login too.

changelog.d/4982.misc

@@ -0,0 +1 @@
+Track which identity server is used when binding a threepid and use that for unbinding, as per MSC1915.

changelog.d/4985.misc

@@ -0,0 +1 @@
+Rewrite KeyringTestCase as a HomeserverTestCase.

changelog.d/4987.misc

@@ -0,0 +1 @@
+README updates: Corrected the default POSTGRES_USER. Added port forwarding hint in TLS section.

changelog.d/4989.feature

@@ -0,0 +1 @@
+Remove presence list support as per MSC 1819.

changelog.d/4990.bugfix

@@ -0,0 +1 @@
+Transfer related groups on room upgrade.

changelog.d/4991.feature

@@ -0,0 +1 @@
+Reduce CPU usage starting pushers during start up.

changelog.d/4996.misc

@@ -0,0 +1 @@
+Run `black` on the remainder of `synapse/storage/`.

changelog.d/4998.misc

@@ -0,0 +1 @@
+Fix grammar in get_current_users_in_room and give it a docstring.

changelog.d/4999.bugfix

@@ -0,0 +1 @@
+Prevent the ability to kick users from a room they aren't in.

changelog.d/5002.feature

@@ -0,0 +1 @@
+Add a delete group admin API.

changelog.d/5003.bugfix

@@ -0,0 +1 @@
+Fix issue #4596 so synapse_port_db script works with --curses option on Python 3. Contributed by Anders Jensen-Waud <anders@jensenwaud.com>.

changelog.d/5007.misc

@@ -0,0 +1 @@
+Refactor synapse.storage._base._simple_select_list_paginate.

changelog.d/5010.feature

@@ -0,0 +1 @@
+Add config option to block users from looking up 3PIDs.

changelog.d/5020.feature

@@ -0,0 +1 @@
+Add context to phonehome stats.

changelog.d/5043.feature

@@ -1 +0,0 @@
-Add ability to blacklist IP ranges for the federation client.

changelog.d/5083.feature

@@ -1 +0,0 @@
-Adds auth_profile_reqs option to require access_token to GET /profile endpoints on CS API

changelog.d/5098.misc

@@ -1 +0,0 @@
-Add workarounds for pep-517 install errors.

changelog.d/5171.misc

@@ -1 +0,0 @@
-Update tests to consistently be configured via the same code that is used when loading from configuration files.

changelog.d/5181.feature

@@ -1 +0,0 @@
-Ratelimiting configuration for clients sending messages and the federation server has been altered to match login ratelimiting. The old configuration names will continue working. Check the sample config for details of the new names.

changelog.d/5183.misc

@@ -1 +0,0 @@
-Allow client event serialization to be async.

changelog.d/5184.misc

@@ -1 +0,0 @@
-Expose DataStore._get_events as get_events_as_list.

changelog.d/5185.misc

@@ -1 +0,0 @@
-Update tests to consistently be configured via the same code that is used when loading from configuration files.

changelog.d/5187.bugfix

@@ -1 +0,0 @@
-Fix a bug where the register endpoint would fail with M_THREEPID_IN_USE instead of returning an account previously registered in the same session.

changelog.d/5190.feature

@@ -1 +0,0 @@
-Drop support for the undocumented /_matrix/client/v2_alpha API prefix.

changelog.d/5196.feature

@@ -1 +0,0 @@
-Add an option to disable per-room profiles.

contrib/systemd-with-workers/system/matrix-synapse-worker@.service

@@ -12,7 +12,6 @@ ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.%i --config-path=/
 ExecReload=/bin/kill -HUP $MAINPID
 Restart=always
 RestartSec=3
-SyslogIdentifier=matrix-synapse-%i
 
 [Install]
 WantedBy=matrix-synapse.service

contrib/systemd-with-workers/system/matrix-synapse.service

@@ -11,7 +11,6 @@ ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --confi
 ExecReload=/bin/kill -HUP $MAINPID
 Restart=always
 RestartSec=3
-SyslogIdentifier=matrix-synapse
 
 [Install]
 WantedBy=matrix.target

contrib/systemd/matrix-synapse.service

@@ -22,10 +22,10 @@ Group=nogroup
 
 WorkingDirectory=/opt/synapse
 ExecStart=/opt/synapse/env/bin/python -m synapse.app.homeserver --config-path=/opt/synapse/homeserver.yaml
-SyslogIdentifier=matrix-synapse
 
 # adjust the cache factor if necessary
 # Environment=SYNAPSE_CACHE_FACTOR=2.0
 
 [Install]
 WantedBy=multi-user.target
+

debian/changelog

@@ -1,25 +1,3 @@
-matrix-synapse-py3 (0.99.4) stable; urgency=medium
-
-  [ Christoph Müller ]
-  * Configure the systemd units to have a log identifier of `matrix-synapse`
-
-  [ Synapse Packaging team ]
-  * New synapse release 0.99.4.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 15 May 2019 13:58:08 +0100
-
-matrix-synapse-py3 (0.99.3.2) stable; urgency=medium
-
-  * New synapse release 0.99.3.2.
-
- -- Synapse Packaging team <packages@matrix.org>  Fri, 03 May 2019 18:56:20 +0100
-
-matrix-synapse-py3 (0.99.3.1) stable; urgency=medium
-
-  * New synapse release 0.99.3.1.
-
- -- Synapse Packaging team <packages@matrix.org>  Fri, 03 May 2019 16:02:43 +0100
-
 matrix-synapse-py3 (0.99.3) stable; urgency=medium
 
   [ Richard van der Hoff ]

debian/matrix-synapse.service

@@ -11,7 +11,6 @@ ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --confi
 ExecReload=/bin/kill -HUP $MAINPID
 Restart=always
 RestartSec=3
-SyslogIdentifier=matrix-synapse
 
 [Install]
 WantedBy=multi-user.target

docker/Dockerfile-dhvirtualenv

@@ -50,15 +50,12 @@ RUN apt-get update -qq -o Acquire::Languages=none \
         debhelper \
         devscripts \
         dh-systemd \
-        libsystemd-dev \
         lsb-release \
-        pkg-config \
         python3-dev \
         python3-pip \
         python3-setuptools \
         python3-venv \
-        sqlite3 \
-        libpq-dev
+        sqlite3
 
 COPY --from=builder /dh-virtualenv_1.1-1_all.deb /
 

docker/README.md

@@ -102,9 +102,8 @@ when ``SYNAPSE_CONFIG_PATH`` is not set.
 * ``SYNAPSE_SERVER_NAME`` (mandatory), the server public hostname.
 * ``SYNAPSE_REPORT_STATS``, (mandatory, ``yes`` or ``no``), enable anonymous
   statistics reporting back to the Matrix project which helps us to get funding.
-* `SYNAPSE_NO_TLS`, (accepts `true`, `false`, `on`, `off`, `1`, `0`, `yes`, `no`]): disable
-  TLS in Synapse (use this if you run your own TLS-capable reverse proxy). Defaults
-  to `false` (ie, TLS is enabled by default).
+* ``SYNAPSE_NO_TLS``, set this variable to disable TLS in Synapse (use this if
+  you run your own TLS-capable reverse proxy).
 * ``SYNAPSE_ENABLE_REGISTRATION``, set this variable to enable registration on
   the Synapse instance.
 * ``SYNAPSE_ALLOW_GUEST``, set this variable to allow guest joining this server.

docker/start.py

@@ -59,18 +59,6 @@ else:
         if not os.path.exists("/compiled"): os.mkdir("/compiled")
 
         config_path = "/compiled/homeserver.yaml"
-        
-        # Convert SYNAPSE_NO_TLS to boolean if exists
-        if "SYNAPSE_NO_TLS" in environ:
-            tlsanswerstring = str.lower(environ["SYNAPSE_NO_TLS"])
-            if tlsanswerstring in ("true", "on", "1", "yes"):
-                environ["SYNAPSE_NO_TLS"] = True
-            else:
-                if tlsanswerstring in ("false", "off", "0", "no"):
-                    environ["SYNAPSE_NO_TLS"] = False
-                else:
-                    print("Environment variable \"SYNAPSE_NO_TLS\" found but value \"" + tlsanswerstring + "\" unrecognized; exiting.")
-                    sys.exit(2)
 
         convert("/conf/homeserver.yaml", config_path, environ)
         convert("/conf/log.config", "/compiled/log.config", environ)

docs/MSC1711_certificates_FAQ.md

@@ -177,6 +177,7 @@ You can do this with a `.well-known` file as follows:
        on `customer.example.net:8000` it correctly handles HTTP requests with
        Host header set to `customer.example.net:8000`.
 
+
 ## FAQ
 
 ### Synapse 0.99.0 has just been released, what do I need to do right now?

docs/admin_api/account_validity.rst

@@ -1,42 +0,0 @@
-Account validity API
-====================
-
-This API allows a server administrator to manage the validity of an account. To
-use it, you must enable the account validity feature (under
-``account_validity``) in Synapse's configuration.
-
-Renew account
--------------
-
-This API extends the validity of an account by as much time as configured in the
-``period`` parameter from the ``account_validity`` configuration.
-
-The API is::
-
-    POST /_synapse/admin/v1/account_validity/validity
-
-with the following body:
-
-.. code:: json
-
-    {
-        "user_id": "<user ID for the account to renew>",
-        "expiration_ts": 0,
-        "enable_renewal_emails": true
-    }
-
-
-``expiration_ts`` is an optional parameter and overrides the expiration date,
-which otherwise defaults to now + validity period.
-
-``enable_renewal_emails`` is also an optional parameter and enables/disables
-sending renewal emails to the user. Defaults to true.
-
-The API returns with the new expiration date for this account, as a timestamp in
-milliseconds since epoch:
-
-.. code:: json
-
-    {
-        "expiration_ts": 0
-    }

docs/admin_api/delete_group.md

@@ -8,7 +8,7 @@ being deleted.
 The API is:
 
 ```
-POST /_synapse/admin/v1/delete_group/<group_id>
+POST /_matrix/client/r0/admin/delete_group/<group_id>
 ```
 
 including an `access_token` of a server admin.

docs/admin_api/media_admin_api.md

@@ -4,7 +4,7 @@ This API gets a list of known media in a room.
 
 The API is:
 ```
-GET /_synapse/admin/v1/room/<room_id>/media
+GET /_matrix/client/r0/admin/room/<room_id>/media
 ```
 including an `access_token` of a server admin.
 

docs/admin_api/purge_history_api.rst

@@ -10,7 +10,7 @@ paginate further back in the room from the point being purged from.
 
 The API is:
 
-``POST /_synapse/admin/v1/purge_history/<room_id>[/<event_id>]``
+``POST /_matrix/client/r0/admin/purge_history/<room_id>[/<event_id>]``
 
 including an ``access_token`` of a server admin.
 
@@ -49,7 +49,7 @@ Purge status query
 
 It is possible to poll for updates on recent purges with a second API;
 
-``GET /_synapse/admin/v1/purge_history_status/<purge_id>``
+``GET /_matrix/client/r0/admin/purge_history_status/<purge_id>``
 
 (again, with a suitable ``access_token``). This API returns a JSON body like
 the following:

docs/admin_api/purge_remote_media.rst

@@ -6,7 +6,7 @@ media.
 
 The API is::
 
-    POST /_synapse/admin/v1/purge_media_cache?before_ts=<unix_timestamp_in_ms>&access_token=<access_token>
+    POST /_matrix/client/r0/admin/purge_media_cache?before_ts=<unix_timestamp_in_ms>&access_token=<access_token>
 
     {}
 

docs/admin_api/register_api.rst

@@ -12,7 +12,7 @@ is not enabled.
 
 To fetch the nonce, you need to request one from the API::
 
-  > GET /_synapse/admin/v1/register
+  > GET /_matrix/client/r0/admin/register
 
   < {"nonce": "thisisanonce"}
 
@@ -22,7 +22,7 @@ body containing the nonce, username, password, whether they are an admin
 
 As an example::
 
-  > POST /_synapse/admin/v1/register
+  > POST /_matrix/client/r0/admin/register
   > {
      "nonce": "thisisanonce",
      "username": "pepper_roni",

docs/admin_api/server_notices.md

@@ -1,48 +0,0 @@
-# Server Notices
-
-The API to send notices is as follows:
-
-```
-POST /_synapse/admin/v1/send_server_notice
-```
-
-or:
-
-```
-PUT /_synapse/admin/v1/send_server_notice/{txnId}
-```
-
-You will need to authenticate with an access token for an admin user.
-
-When using the `PUT` form, retransmissions with the same transaction ID will be
-ignored in the same way as with `PUT
-/_matrix/client/r0/rooms/{roomId}/send/{eventType}/{txnId}`.
-
-The request body should look something like the following:
-
-```json
-{
-    "user_id": "@target_user:server_name",
-    "content": {
-        "msgtype": "m.text",
-        "body": "This is my message"
-    }
-}
-```
-
-You can optionally include the following additional parameters:
-
-* `type`: the type of event. Defaults to `m.room.message`.
-* `state_key`: Setting this will result in a state event being sent.
-
-
-Once the notice has been sent, the API will return the following response:
-
-```json
-{
-    "event_id": "<event_id>"
-}
-```
-
-Note that server notices must be enabled in `homeserver.yaml` before this API
-can be used. See [server_notices.md](../server_notices.md) for more information.

docs/admin_api/user_admin_api.rst

@@ -5,7 +5,7 @@ This API returns information about a specific user account.
 
 The api is::
 
-    GET /_synapse/admin/v1/whois/<user_id>
+    GET /_matrix/client/r0/admin/whois/<user_id>
 
 including an ``access_token`` of a server admin.
 
@@ -50,7 +50,7 @@ references to it).
 
 The api is::
 
-    POST /_synapse/admin/v1/deactivate/<user_id>
+    POST /_matrix/client/r0/admin/deactivate/<user_id>
 
 with a body of:
 
@@ -73,7 +73,7 @@ Changes the password of another user.
 
 The api is::
 
-    POST /_synapse/admin/v1/reset_password/<user_id>
+    POST /_matrix/client/r0/admin/reset_password/<user_id>
 
 with a body of:
 

docs/admin_api/version_api.rst

@@ -8,7 +8,9 @@ contains Synapse version information).
 
 The api is::
 
-    GET /_synapse/admin/v1/server_version
+    GET /_matrix/client/r0/admin/server_version
+
+including an ``access_token`` of a server admin.
 
 It returns a JSON body like the following:
 

docs/metrics-howto.rst

@@ -48,10 +48,7 @@ How to monitor Synapse metrics using Prometheus
     - job_name: "synapse"
       metrics_path: "/_synapse/metrics"
       static_configs:
-        - targets: ["my.server.here:port"]
-
-   where ``my.server.here`` is the IP address of Synapse, and ``port`` is the listener port
-   configured with the ``metrics`` resource.
+        - targets: ["my.server.here:9092"]
 
    If your prometheus is older than 1.5.2, you will need to replace
    ``static_configs`` in the above with ``target_groups``.

docs/reverse_proxy.rst

@@ -69,7 +69,6 @@ Let's assume that we expect clients to connect to our server at
           SSLEngine on
           ServerName matrix.example.com;
 
-          AllowEncodedSlashes NoDecode
           ProxyPass /_matrix http://127.0.0.1:8008/_matrix nocanon
           ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix
       </VirtualHost>
@@ -78,7 +77,6 @@ Let's assume that we expect clients to connect to our server at
           SSLEngine on
           ServerName example.com;
           
-          AllowEncodedSlashes NoDecode
           ProxyPass /_matrix http://127.0.0.1:8008/_matrix nocanon
           ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix
       </VirtualHost>

docs/sample_config.yaml

@@ -69,20 +69,6 @@ pid_file: DATADIR/homeserver.pid
 #
 #use_presence: false
 
-# Whether to require authentication to retrieve profile data (avatars,
-# display names) of other users through the client API. Defaults to
-# 'false'. Note that profile data is also available via the federation
-# API, so this setting is of limited value if federation is enabled on
-# the server.
-#
-#require_auth_for_profile_requests: true
-
-# If set to 'true', requires authentication to access the server's
-# public rooms directory through the client API, and forbids any other
-# homeserver to fetch it via federation. Defaults to 'false'.
-#
-#restrict_public_rooms_to_local_users: true
-
 # The GC threshold parameters to pass to `gc.set_threshold`, if defined
 #
 #gc_thresholds: [700, 10, 10]
@@ -115,24 +101,6 @@ pid_file: DATADIR/homeserver.pid
 #  - nyc.example.com
 #  - syd.example.com
 
-# Prevent federation requests from being sent to the following
-# blacklist IP address CIDR ranges. If this option is not specified, or
-# specified with an empty list, no ip range blacklist will be enforced.
-#
-# (0.0.0.0 and :: are always blacklisted, whether or not they are explicitly
-# listed here, since they correspond to unroutable addresses.)
-#
-federation_ip_range_blacklist:
-  - '127.0.0.0/8'
-  - '10.0.0.0/8'
-  - '172.16.0.0/12'
-  - '192.168.0.0/16'
-  - '100.64.0.0/10'
-  - '169.254.0.0/16'
-  - '::1/128'
-  - 'fe80::/64'
-  - 'fc00::/7'
-
 # List of ports that Synapse should listen on, their purpose and their
 # configuration.
 #
@@ -168,8 +136,8 @@ federation_ip_range_blacklist:
 #
 # Valid resource names are:
 #
-#   client: the client-server API (/_matrix/client), and the synapse admin
-#       API (/_synapse/admin). Also implies 'media' and 'static'.
+#   client: the client-server API (/_matrix/client). Also implies 'media' and
+#       'static'.
 #
 #   consent: user consent forms (/_matrix/consent). See
 #       docs/consent_tracking.md.
@@ -271,17 +239,6 @@ listeners:
 # Used by phonehome stats to group together related servers.
 #server_context: context
 
-# Whether to require a user to be in the room to add an alias to it.
-# Defaults to 'true'.
-#
-#require_membership_for_aliases: false
-
-# Whether to allow per-room membership profiles through the send of membership
-# events with profile information that differ from the target's global profile.
-# Defaults to 'true'.
-#
-#allow_per_room_profiles: false
-
 
 ## TLS ##
 
@@ -303,40 +260,6 @@ listeners:
 #
 #tls_private_key_path: "CONFDIR/SERVERNAME.tls.key"
 
-# Whether to verify TLS certificates when sending federation traffic.
-#
-# This currently defaults to `false`, however this will change in
-# Synapse 1.0 when valid federation certificates will be required.
-#
-#federation_verify_certificates: true
-
-# Skip federation certificate verification on the following whitelist
-# of domains.
-#
-# This setting should only be used in very specific cases, such as
-# federation over Tor hidden services and similar. For private networks
-# of homeservers, you likely want to use a private CA instead.
-#
-# Only effective if federation_verify_certicates is `true`.
-#
-#federation_certificate_verification_whitelist:
-#  - lon.example.com
-#  - *.domain.com
-#  - *.onion
-
-# List of custom certificate authorities for federation traffic.
-#
-# This setting should only normally be used within a private network of
-# homeservers.
-#
-# Note that this list will replace those that are provided by your
-# operating environment. Certificates must be in PEM format.
-#
-#federation_custom_ca_list:
-#  - myCA1.pem
-#  - myCA2.pem
-#  - myCA3.pem
-
 # ACME support: This will configure Synapse to request a valid TLS certificate
 # for your configured `server_name` via Let's Encrypt.
 #
@@ -452,15 +375,21 @@ log_config: "CONFDIR/SERVERNAME.log.config"
 
 ## Ratelimiting ##
 
-# Ratelimiting settings for client actions (registration, login, messaging).
+# Number of messages a client can send per second
+#
+#rc_messages_per_second: 0.2
+
+# Number of message a client can send before being throttled
+#
+#rc_message_burst_count: 10.0
+
+# Ratelimiting settings for registration and login.
 #
 # Each ratelimiting configuration is made of two parameters:
 #   - per_second: number of requests a client can send per second.
 #   - burst_count: number of requests a client can send before being throttled.
 #
 # Synapse currently uses the following configurations:
-#   - one for messages that ratelimits sending based on the account the client
-#     is using
 #   - one for registration that ratelimits registration requests based on the
 #     client's IP address.
 #   - one for login that ratelimits login requests based on the client's IP
@@ -473,10 +402,6 @@ log_config: "CONFDIR/SERVERNAME.log.config"
 #
 # The defaults are as shown below.
 #
-#rc_message:
-#  per_second: 0.2
-#  burst_count: 10
-#
 #rc_registration:
 #  per_second: 0.17
 #  burst_count: 3
@@ -492,28 +417,29 @@ log_config: "CONFDIR/SERVERNAME.log.config"
 #    per_second: 0.17
 #    burst_count: 3
 
+# The federation window size in milliseconds
+#
+#federation_rc_window_size: 1000
 
-# Ratelimiting settings for incoming federation
+# The number of federation requests from a single server in a window
+# before the server will delay processing the request.
 #
-# The rc_federation configuration is made up of the following settings:
-#   - window_size: window size in milliseconds
-#   - sleep_limit: number of federation requests from a single server in
-#     a window before the server will delay processing the request.
-#   - sleep_delay: duration in milliseconds to delay processing events
-#     from remote servers by if they go over the sleep limit.
-#   - reject_limit: maximum number of concurrent federation requests
-#     allowed from a single server
-#   - concurrent: number of federation requests to concurrently process
-#     from a single server
+#federation_rc_sleep_limit: 10
+
+# The duration in milliseconds to delay processing events from
+# remote servers by if they go over the sleep limit.
 #
-# The defaults are as shown below.
+#federation_rc_sleep_delay: 500
+
+# The maximum number of concurrent federation requests allowed
+# from a single server
 #
-#rc_federation:
-#  window_size: 1000
-#  sleep_limit: 10
-#  sleep_delay: 500
-#  reject_limit: 50
-#  concurrent: 3
+#federation_rc_reject_limit: 50
+
+# The number of federation requests to concurrently process from a
+# single server
+#
+#federation_rc_concurrent: 3
 
 # Target outgoing federation transaction frequency for sending read-receipts,
 # per-room.
@@ -583,12 +509,11 @@ uploads_path: "DATADIR/uploads"
 #    height: 600
 #    method: scale
 
-# Is the preview URL API enabled?
+# Is the preview URL API enabled?  If enabled, you *must* specify
+# an explicit url_preview_ip_range_blacklist of IPs that the spider is
+# denied from accessing.
 #
-# 'false' by default: uncomment the following to enable it (and specify a
-# url_preview_ip_range_blacklist blacklist).
-#
-#url_preview_enabled: true
+#url_preview_enabled: false
 
 # List of IP address CIDR ranges that the URL preview spider is denied
 # from accessing.  There are no defaults: you must explicitly
@@ -598,12 +523,6 @@ uploads_path: "DATADIR/uploads"
 # synapse to issue arbitrary GET requests to your internal services,
 # causing serious security issues.
 #
-# (0.0.0.0 and :: are always blacklisted, whether or not they are explicitly
-# listed here, since they correspond to unroutable addresses.)
-#
-# This must be specified if url_preview_enabled is set. It is recommended that
-# you uncomment the following list as a starting point.
-#
 #url_preview_ip_range_blacklist:
 #  - '127.0.0.0/8'
 #  - '10.0.0.0/8'
@@ -614,7 +533,7 @@ uploads_path: "DATADIR/uploads"
 #  - '::1/128'
 #  - 'fe80::/64'
 #  - 'fc00::/7'
-
+#
 # List of IP address CIDR ranges that the URL preview spider is allowed
 # to access even if they are specified in url_preview_ip_range_blacklist.
 # This is useful for specifying exceptions to wide-ranging blacklisted
@@ -727,32 +646,6 @@ uploads_path: "DATADIR/uploads"
 #
 #enable_registration: false
 
-# Optional account validity configuration. This allows for accounts to be denied
-# any request after a given period.
-#
-# ``enabled`` defines whether the account validity feature is enabled. Defaults
-# to False.
-#
-# ``period`` allows setting the period after which an account is valid
-# after its registration. When renewing the account, its validity period
-# will be extended by this amount of time. This parameter is required when using
-# the account validity feature.
-#
-# ``renew_at`` is the amount of time before an account's expiry date at which
-# Synapse will send an email to the account's email address with a renewal link.
-# This needs the ``email`` and ``public_baseurl`` configuration sections to be
-# filled.
-#
-# ``renew_email_subject`` is the subject of the email sent out with the renewal
-# link. ``%(app)s`` can be used as a placeholder for the ``app_name`` parameter
-# from the ``email`` section.
-#
-#account_validity:
-#  enabled: True
-#  period: 6w
-#  renew_at: 1w
-#  renew_email_subject: "Renew your %(app)s account"
-
 # The user must provide all of the below types of 3PID when registering.
 #
 #registrations_require_3pid:
@@ -764,32 +657,9 @@ uploads_path: "DATADIR/uploads"
 #
 #disable_msisdn_registration: true
 
-# Derive the user's matrix ID from a type of 3PID used when registering.
-# This overrides any matrix ID the user proposes when calling /register
-# The 3PID type should be present in registrations_require_3pid to avoid
-# users failing to register if they don't specify the right kind of 3pid.
-#
-#register_mxid_from_3pid: email
-
-# Uncomment to set the display name of new users to their email address,
-# rather than using the default heuristic.
-#
-#register_just_use_email_for_display_name: true
-
 # Mandate that users are only allowed to associate certain formats of
 # 3PIDs with accounts on this server.
 #
-# Use an Identity Server to establish which 3PIDs are allowed to register?
-# Overrides allowed_local_3pids below.
-#
-#check_is_for_allowed_local_3pids: matrix.org
-#
-# If you are using an IS you can also check whether that IS registers
-# pending invites for the given 3PID (and then allow it to sign up on
-# the platform):
-#
-#allow_invited_3pids: False
-#
 #allowed_local_3pids:
 #  - medium: email
 #    pattern: '.*@matrix\.org'
@@ -798,11 +668,6 @@ uploads_path: "DATADIR/uploads"
 #  - medium: msisdn
 #    pattern: '\+44'
 
-# If true, stop users from trying to change the 3PIDs associated with
-# their accounts.
-#
-#disable_3pid_changes: False
-
 # Enable 3PIDs lookup requests to identity servers from this server.
 #
 #enable_3pid_lookup: true
@@ -844,30 +709,6 @@ uploads_path: "DATADIR/uploads"
 #  - matrix.org
 #  - vector.im
 
-# If enabled, user IDs, display names and avatar URLs will be replicated
-# to this server whenever they change.
-# This is an experimental API currently implemented by sydent to support
-# cross-homeserver user directories.
-#
-#replicate_user_profiles_to: example.com
-
-# If specified, attempt to replay registrations, profile changes & 3pid
-# bindings on the given target homeserver via the AS API. The HS is authed
-# via a given AS token.
-#
-#shadow_server:
-#  hs_url: https://shadow.example.com
-#  hs: shadow.example.com
-#  as_token: 12u394refgbdhivsia
-
-# If enabled, don't let users set their own display names/avatars
-# other than for the very first time (unless they are a server admin).
-# Useful when provisioning users based on the contents of a 3rd party
-# directory and to avoid ambiguities.
-#
-#disable_set_displayname: False
-#disable_set_avatar_url: False
-
 # Users who register on this homeserver will automatically be joined
 # to these rooms
 #
@@ -1050,7 +891,7 @@ password_config:
 
 
 
-# Enable sending emails for notification events or expiry notices
+# Enable sending emails for notification events
 # Defining a custom URL for Riot is only needed if email notifications
 # should contain links to a self-hosted installation of Riot; when set
 # the "app_name" setting is ignored.
@@ -1072,9 +913,6 @@ password_config:
 #   #template_dir: res/templates
 #   notif_template_html: notif_mail.html
 #   notif_template_text: notif_mail.txt
-#   # Templates for account expiry notices.
-#   expiry_template_html: notice_expiry.html
-#   expiry_template_text: notice_expiry.txt
 #   notif_for_new_users: True
 #   riot_base_url: "http://localhost/riot"
 
@@ -1144,11 +982,6 @@ password_config:
 #user_directory:
 #  enabled: true
 #  search_all_users: false
-#
-#  # If this is set, user search will be delegated to this ID server instead
-#  # of synapse performing the search itself.
-#  # This is an experimental API.
-#  defer_to_id_server: https://id.example.com
 
 
 # User Consent configuration

docs/server_notices.md

@@ -1,4 +1,5 @@
-# Server Notices
+Server Notices
+==============
 
 'Server Notices' are a new feature introduced in Synapse 0.30. They provide a
 channel whereby server administrators can send messages to users on the server.
@@ -10,7 +11,8 @@ they may also find a use for features such as "Message of the day".
 This is a feature specific to Synapse, but it uses standard Matrix
 communication mechanisms, so should work with any Matrix client.
 
-## User experience
+User experience
+---------------
 
 When the user is first sent a server notice, they will get an invitation to a
 room (typically called 'Server Notices', though this is configurable in
@@ -27,7 +29,8 @@ levels.
 Having joined the room, the user can leave the room if they want. Subsequent
 server notices will then cause a new room to be created.
 
-## Synapse configuration
+Synapse configuration
+---------------------
 
 Server notices come from a specific user id on the server. Server
 administrators are free to choose the user id - something like `server` is
@@ -55,7 +58,17 @@ room which will be created.
 `system_mxid_display_name` and `system_mxid_avatar_url` can be used to set the
 displayname and avatar of the Server Notices user.
 
-## Sending notices
+Sending notices
+---------------
 
-To send server notices to users you can use the
-[admin_api](admin_api/server_notices.md).
+As of the current version of synapse, there is no convenient interface for
+sending notices (other than the automated ones sent as part of consent
+tracking).
+
+In the meantime, it is possible to test this feature using the manhole. Having
+gone into the manhole as described in [manhole.md](manhole.md), a notice can be
+sent with something like:
+
+```
+>>> hs.get_server_notices_manager().send_notice('@user:server.com', {'msgtype':'m.text', 'body':'foo'})
+```

docs/workers.rst

@@ -227,12 +227,6 @@ following regular expressions::
     ^/_matrix/client/(api/v1|r0|unstable)/account/3pid$
     ^/_matrix/client/(api/v1|r0|unstable)/keys/query$
     ^/_matrix/client/(api/v1|r0|unstable)/keys/changes$
-    ^/_matrix/client/versions$
-    ^/_matrix/client/(api/v1|r0|unstable)/voip/turnServer$
-
-Additionally, the following REST endpoints can be handled for GET requests::
-
-    ^/_matrix/client/(api/v1|r0|unstable)/pushrules/.*$
 
 Additionally, the following REST endpoints can be handled, but all requests must
 be routed to the same instance::

res/templates-dinsic/mail-Vector.css

@@ -1,7 +0,0 @@
-.header {
-    border-bottom: 4px solid #e4f7ed ! important;
-}
-
-.notif_link a, .footer a {
-    color: #76CFA6 ! important;
-}

res/templates-dinsic/mail.css

@@ -1,156 +0,0 @@
-body {
-    margin: 0px;
-}
-
-pre, code {
-    word-break: break-word;
-    white-space: pre-wrap;
-}
-
-#page {
-    font-family: 'Open Sans', Helvetica, Arial, Sans-Serif;
-    font-color: #454545;
-    font-size: 12pt;
-    width: 100%;
-    padding: 20px;
-}
-
-#inner {
-    width: 640px;
-}
-
-.header {
-    width: 100%;
-    height: 87px;
-    color: #454545;
-    border-bottom: 4px solid #e5e5e5;
-}
-
-.logo {
-    text-align: right;
-    margin-left: 20px;
-}
-
-.salutation {
-    padding-top: 10px;
-    font-weight: bold;
-}
-
-.summarytext {
-}
-
-.room {
-    width: 100%;
-    color: #454545;
-    border-bottom: 1px solid #e5e5e5;
-}
-
-.room_header td {
-    padding-top: 38px;
-    padding-bottom: 10px;
-    border-bottom: 1px solid #e5e5e5;
-}
-
-.room_name {
-    vertical-align: middle;
-    font-size: 18px;
-    font-weight: bold;
-}
-
-.room_header h2 {
-    margin-top: 0px;
-    margin-left: 75px;
-    font-size: 20px;
-}
-
-.room_avatar {
-    width: 56px;
-    line-height: 0px;
-    text-align: center;
-    vertical-align: middle;
-}
-
-.room_avatar img {
-    width: 48px;
-    height: 48px;
-    object-fit: cover;
-    border-radius: 24px;
-}
-
-.notif {
-    border-bottom: 1px solid #e5e5e5;
-    margin-top: 16px;
-    padding-bottom: 16px;
-}
-
-.historical_message .sender_avatar {
-    opacity: 0.3;
-}
-
-/* spell out opacity and historical_message class names for Outlook aka Word */
-.historical_message .sender_name {
-    color: #e3e3e3;
-}
-
-.historical_message .message_time {
-    color: #e3e3e3;
-}
-
-.historical_message .message_body {
-    color: #c7c7c7;
-}
-
-.historical_message td,
-.message td {
-    padding-top: 10px;
-}
-
-.sender_avatar {
-    width: 56px;
-    text-align: center;
-    vertical-align: top;
-}
-
-.sender_avatar img {
-    margin-top: -2px;
-    width: 32px;
-    height: 32px;
-    border-radius: 16px;
-}
-
-.sender_name  {
-    display: inline;
-    font-size: 13px;
-    color: #a2a2a2;
-}
-
-.message_time  {
-    text-align: right;
-    width: 100px;
-    font-size: 11px;
-    color: #a2a2a2;
-}
-
-.message_body {
-}
-
-.notif_link td {
-    padding-top: 10px;
-    padding-bottom: 10px;
-    font-weight: bold;
-}
-
-.notif_link a, .footer a {
-    color: #454545;
-    text-decoration: none;
-}
-
-.debug {
-    font-size: 10px;
-    color: #888;
-}
-
-.footer {
-    margin-top: 20px;
-    text-align: center;
-}

res/templates-dinsic/notif.html

@@ -1,45 +0,0 @@
-{% for message in notif.messages %}
-    <tr class="{{ "historical_message" if message.is_historical else "message" }}">
-        <td class="sender_avatar">
-            {% if loop.index0 == 0 or notif.messages[loop.index0 - 1].sender_name != notif.messages[loop.index0].sender_name %}
-                {% if message.sender_avatar_url %}
-                    <img alt="" class="sender_avatar" src="{{ message.sender_avatar_url|mxc_to_http(32,32) }}"  />
-                {% else %}
-                    {% if message.sender_hash % 3 == 0 %}
-                        <img class="sender_avatar" src="https://vector.im/beta/img/76cfa6.png"  />
-                    {% elif message.sender_hash % 3 == 1 %}
-                        <img class="sender_avatar" src="https://vector.im/beta/img/50e2c2.png"  />
-                    {% else %}
-                        <img class="sender_avatar" src="https://vector.im/beta/img/f4c371.png"  />
-                    {% endif %}
-                {% endif %}
-            {% endif %}
-        </td>
-        <td class="message_contents">
-            {% if loop.index0 == 0 or notif.messages[loop.index0 - 1].sender_name != notif.messages[loop.index0].sender_name %}
-                <div class="sender_name">{% if message.msgtype == "m.emote" %}*{% endif %} {{ message.sender_name }}</div>
-            {% endif %}
-            <div class="message_body">
-                {% if message.msgtype == "m.text" %}
-                    {{ message.body_text_html }}
-                {% elif message.msgtype == "m.emote" %}
-                    {{ message.body_text_html }}
-                {% elif message.msgtype == "m.notice" %}
-                    {{ message.body_text_html }}
-                {% elif message.msgtype == "m.image" %}
-                    <img src="{{ message.image_url|mxc_to_http(640, 480, scale) }}" />
-                {% elif message.msgtype == "m.file" %}
-                    <span class="filename">{{ message.body_text_plain }}</span>
-                {% endif %}
-            </div>
-        </td>
-        <td class="message_time">{{ message.ts|format_ts("%H:%M") }}</td>
-    </tr>
-{% endfor %}
-<tr class="notif_link">
-    <td></td>
-    <td>
-        <a href="{{ notif.link }}">Voir {{ room.title }}</a>
-    </td>
-    <td></td>
-</tr>

res/templates-dinsic/notif.txt

@@ -1,16 +0,0 @@
-{% for message in notif.messages %}
-{% if message.msgtype == "m.emote" %}* {% endif %}{{ message.sender_name }} ({{ message.ts|format_ts("%H:%M") }})
-{% if message.msgtype == "m.text" %}
-{{ message.body_text_plain }}
-{% elif message.msgtype == "m.emote" %}
-{{ message.body_text_plain }}
-{% elif message.msgtype == "m.notice" %}
-{{ message.body_text_plain }}
-{% elif message.msgtype == "m.image" %}
-{{ message.body_text_plain }}
-{% elif message.msgtype == "m.file" %}
-{{ message.body_text_plain }}
-{% endif %}
-{% endfor %}
-
-Voir {{ room.title }} à {{ notif.link }}

res/templates-dinsic/notif_mail.html

@@ -1,55 +0,0 @@
-<!doctype html>
-<html lang="en">
-    <head>
-        <style type="text/css">
-            {% include 'mail.css' without context %}
-            {% include "mail-%s.css" % app_name ignore missing without context %}
-        </style>
-    </head>
-    <body>
-        <table id="page">
-            <tr>
-                <td> </td>
-                <td id="inner">
-                    <table class="header">
-                        <tr>
-                            <td>
-                                <div class="salutation">Bonjour {{ user_display_name }},</div>
-                                <div class="summarytext">{{ summary_text }}</div>
-                            </td>
-                            <td class="logo">
-                                {% if app_name == "Riot" %}
-                                    <img src="http://matrix.org/img/riot-logo-email.png" width="83" height="83" alt="[Riot]"/>
-                                {% elif app_name == "Vector" %}
-                                    <img src="http://matrix.org/img/vector-logo-email.png" width="64" height="83" alt="[Vector]"/>
-                                {% else %}
-                                    <img src="http://matrix.org/img/matrix-120x51.png" width="120" height="51" alt="[matrix]"/>
-                                {% endif %}
-                            </td>
-                        </tr>
-                    </table>
-                    {% for room in rooms %}
-                        {% include 'room.html' with context %}
-                    {% endfor %}
-                    <div class="footer">
-                        <a href="{{ unsubscribe_link }}">Se désinscrire</a>
-                        <br/>
-                        <br/>
-                        <div class="debug">
-                            Sending email at {{ reason.now|format_ts("%c") }} due to activity in room {{ reason.room_name }} because
-                            an event was received at {{ reason.received_at|format_ts("%c") }}
-                            which is more than {{ "%.1f"|format(reason.delay_before_mail_ms / (60*1000)) }} ({{ reason.delay_before_mail_ms }}) mins ago,
-                            {% if reason.last_sent_ts %}
-                                and the last time we sent a mail for this room was {{ reason.last_sent_ts|format_ts("%c") }},
-                                which is more than {{ "%.1f"|format(reason.throttle_ms / (60*1000)) }} (current throttle_ms) mins ago.
-                            {% else %}
-                                and we don't have a last time we sent a mail for this room.
-                            {% endif %}
-                        </div>
-                    </div>
-                </td>
-                <td> </td>
-            </tr>
-        </table>
-    </body>
-</html>

res/templates-dinsic/notif_mail.txt

@@ -1,10 +0,0 @@
-Bonjour {{ user_display_name }},
-
-{{ summary_text }}
-
-{% for room in rooms %}
-{% include 'room.txt' with context %}
-{% endfor %}
-
-Vous pouvez désactiver ces notifications en cliquant ici {{ unsubscribe_link }}
-

res/templates-dinsic/room.html

@@ -1,33 +0,0 @@
-<table class="room">
-    <tr class="room_header">
-        <td class="room_avatar">
-            {% if room.avatar_url %}
-                <img alt="" src="{{ room.avatar_url|mxc_to_http(48,48) }}" />
-            {% else %}
-                {% if room.hash % 3 == 0 %}
-                    <img alt="" src="https://vector.im/beta/img/76cfa6.png"  />
-                {% elif room.hash % 3 == 1 %}
-                    <img alt="" src="https://vector.im/beta/img/50e2c2.png"  />
-                {% else %}
-                    <img alt="" src="https://vector.im/beta/img/f4c371.png"  />
-                {% endif %}
-            {% endif %}
-        </td>
-        <td class="room_name" colspan="2">
-            {{ room.title }}
-        </td>
-    </tr>
-    {% if room.invite %}
-        <tr>
-            <td></td>
-            <td>
-                <a href="{{ room.link }}">Rejoindre la conversation.</a>
-            </td>
-            <td></td>
-        </tr>
-    {% else %}
-        {% for notif in room.notifs %}
-            {% include 'notif.html' with context %}
-        {% endfor %}
-    {% endif %}
-</table>

res/templates-dinsic/room.txt

@@ -1,9 +0,0 @@
-{{ room.title }}
-
-{% if room.invite %}
-    Vous avez été invité, rejoignez la conversation en cliquant sur le lien suivant {{ room.link }}
-{% else %}
-    {% for notif in room.notifs %}
-        {% include 'notif.txt' with context %}
-    {% endfor %}
-{% endif %}

scripts-dev/build_debian_packages

@@ -24,7 +24,6 @@ DISTS = (
     "ubuntu:xenial",
     "ubuntu:bionic",
     "ubuntu:cosmic",
-    "ubuntu:disco",
 )
 
 DESC = '''\

scripts/synapse_port_db

@@ -58,11 +58,15 @@ BOOLEAN_COLUMNS = {
 
 
 APPEND_ONLY_TABLES = [
+    "event_content_hashes",
     "event_reference_hashes",
+    "event_signatures",
+    "event_edge_hashes",
     "events",
     "event_json",
     "state_events",
     "room_memberships",
+    "feedback",
     "topics",
     "room_names",
     "rooms",
@@ -84,6 +88,7 @@ APPEND_ONLY_TABLES = [
     "event_search",
     "presence_stream",
     "push_rules_stream",
+    "current_state_resets",
     "ex_outlier_stream",
     "cache_invalidation_stream",
     "public_room_list_stream",

setup.py

@@ -86,9 +86,13 @@ long_description = read_file(("README.rst",))
 
 REQUIREMENTS = dependencies['REQUIREMENTS']
 CONDITIONAL_REQUIREMENTS = dependencies['CONDITIONAL_REQUIREMENTS']
-ALL_OPTIONAL_REQUIREMENTS = dependencies['ALL_OPTIONAL_REQUIREMENTS']
 
 # Make `pip install matrix-synapse[all]` install all the optional dependencies.
+ALL_OPTIONAL_REQUIREMENTS = set()
+
+for optional_deps in CONDITIONAL_REQUIREMENTS.values():
+    ALL_OPTIONAL_REQUIREMENTS = set(optional_deps) | ALL_OPTIONAL_REQUIREMENTS
+
 CONDITIONAL_REQUIREMENTS["all"] = list(ALL_OPTIONAL_REQUIREMENTS)
 
 

synapse/__init__.py

@@ -27,4 +27,4 @@ try:
 except ImportError:
     pass
 
-__version__ = "0.99.4"
+__version__ = "0.99.3"

synapse/api/auth.py

@@ -64,8 +64,6 @@ class Auth(object):
         self.token_cache = LruCache(CACHE_SIZE_FACTOR * 10000)
         register_cache("cache", "token_cache", self.token_cache)
 
-        self._account_validity = hs.config.account_validity
-
     @defer.inlineCallbacks
     def check_from_context(self, room_version, event, context, do_sig_check=True):
         prev_state_ids = yield context.get_prev_state_ids(self.store)
@@ -207,7 +205,6 @@ class Auth(object):
             )
 
             user_id, app_service = yield self._get_appservice_user_id(request)
-
             if user_id:
                 request.authenticated_entity = user_id
 
@@ -229,17 +226,6 @@ class Auth(object):
             token_id = user_info["token_id"]
             is_guest = user_info["is_guest"]
 
-            # Deny the request if the user account has expired.
-            if self._account_validity.enabled:
-                user_id = user.to_string()
-                expiration_ts = yield self.store.get_expiration_ts_for_user(user_id)
-                if expiration_ts is not None and self.clock.time_msec() >= expiration_ts:
-                    raise AuthError(
-                        403,
-                        "User account has expired",
-                        errcode=Codes.EXPIRED_ACCOUNT,
-                    )
-
             # device_id may not be present if get_user_by_access_token has been
             # stubbed out.
             device_id = user_info.get("device_id")
@@ -269,40 +255,39 @@ class Auth(object):
                 errcode=Codes.MISSING_TOKEN
             )
 
+    @defer.inlineCallbacks
     def _get_appservice_user_id(self, request):
         app_service = self.store.get_app_service_by_token(
             self.get_access_token_from_request(
                 request, self.TOKEN_NOT_FOUND_HTTP_STATUS
             )
         )
-
         if app_service is None:
-            return(None, None)
+            defer.returnValue((None, None))
 
         if app_service.ip_range_whitelist:
             ip_address = IPAddress(self.hs.get_ip_from_request(request))
             if ip_address not in app_service.ip_range_whitelist:
-                return(None, None)
+                defer.returnValue((None, None))
 
         if b"user_id" not in request.args:
-            return(app_service.sender, app_service)
+            defer.returnValue((app_service.sender, app_service))
 
         user_id = request.args[b"user_id"][0].decode('utf8')
         if app_service.sender == user_id:
-            return(app_service.sender, app_service)
+            defer.returnValue((app_service.sender, app_service))
 
         if not app_service.is_interested_in_user(user_id):
             raise AuthError(
                 403,
                 "Application service cannot masquerade as this user."
             )
-        # Let ASes manipulate nonexistent users (e.g. to shadow-register them)
-        # if not (yield self.store.get_user_by_id(user_id)):
-        #     raise AuthError(
-        #         403,
-        #         "Application service has not registered this user"
-        #     )
-        return(user_id, app_service)
+        if not (yield self.store.get_user_by_id(user_id)):
+            raise AuthError(
+                403,
+                "Application service has not registered this user"
+            )
+        defer.returnValue((user_id, app_service))
 
     @defer.inlineCallbacks
     def get_user_by_access_token(self, token, rights="access"):
@@ -535,21 +520,30 @@ class Auth(object):
         defer.returnValue(user_info)
 
     def get_appservice_by_req(self, request):
-        (user_id, app_service) = self._get_appservice_user_id(request)
-        if not app_service:
-            raise AuthError(
-                self.TOKEN_NOT_FOUND_HTTP_STATUS,
-                "Unrecognised access token.",
-                errcode=Codes.UNKNOWN_TOKEN,
+        try:
+            token = self.get_access_token_from_request(
+                request, self.TOKEN_NOT_FOUND_HTTP_STATUS
+            )
+            service = self.store.get_app_service_by_token(token)
+            if not service:
+                logger.warn("Unrecognised appservice access token.")
+                raise AuthError(
+                    self.TOKEN_NOT_FOUND_HTTP_STATUS,
+                    "Unrecognised access token.",
+                    errcode=Codes.UNKNOWN_TOKEN
+                )
+            request.authenticated_entity = service.sender
+            return defer.succeed(service)
+        except KeyError:
+            raise AuthError(
+                self.TOKEN_NOT_FOUND_HTTP_STATUS, "Missing access token."
             )
-        request.authenticated_entity = app_service.sender
-        return app_service
 
     def is_server_admin(self, user):
         """ Check if the given user is a local server admin.
 
         Args:
-            user (UserID): user to check
+            user (str): mxid of user to check
 
         Returns:
             bool: True if the user is an admin

synapse/api/constants.py

@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
 # Copyright 2017 Vector Creations Ltd
-# Copyright 2018 New Vector Ltd
+# Copyright 2018 New Vector Ltd.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -20,9 +20,6 @@
 # the "depth" field on events is limited to 2**63 - 1
 MAX_DEPTH = 2**63 - 1
 
-# the maximum length for a room alias is 255 characters
-MAX_ALIAS_LENGTH = 255
-
 
 class Membership(object):
 
@@ -79,7 +76,6 @@ class EventTypes(object):
     RoomAvatar = "m.room.avatar"
     RoomEncryption = "m.room.encryption"
     GuestAccess = "m.room.guest_access"
-    Encryption = "m.room.encryption"
 
     # These are used for validation
     Message = "m.room.message"

synapse/api/errors.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2018 New Vector Ltd
+# Copyright 2018 New Vector Ltd.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -60,7 +60,6 @@ class Codes(object):
     UNSUPPORTED_ROOM_VERSION = "M_UNSUPPORTED_ROOM_VERSION"
     INCOMPATIBLE_ROOM_VERSION = "M_INCOMPATIBLE_ROOM_VERSION"
     WRONG_ROOM_KEYS_VERSION = "M_WRONG_ROOM_KEYS_VERSION"
-    EXPIRED_ACCOUNT = "ORG_MATRIX_EXPIRED_ACCOUNT"
 
 
 class CodeMessageException(RuntimeError):

synapse/api/urls.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2018 New Vector Ltd
+# Copyright 2018 New Vector Ltd.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -22,7 +22,8 @@ from six.moves.urllib.parse import urlencode
 
 from synapse.config import ConfigError
 
-CLIENT_API_PREFIX = "/_matrix/client"
+CLIENT_PREFIX = "/_matrix/client/api/v1"
+CLIENT_V2_ALPHA_PREFIX = "/_matrix/client/v2_alpha"
 FEDERATION_PREFIX = "/_matrix/federation"
 FEDERATION_V1_PREFIX = FEDERATION_PREFIX + "/v1"
 FEDERATION_V2_PREFIX = FEDERATION_PREFIX + "/v2"

synapse/app/_base.py

@@ -22,14 +22,13 @@ import traceback
 import psutil
 from daemonize import Daemonize
 
-from twisted.internet import defer, error, reactor
+from twisted.internet import error, reactor
 from twisted.protocols.tls import TLSMemoryBIOFactory
 
 import synapse
 from synapse.app import check_bind_error
 from synapse.crypto import context_factory
 from synapse.util import PreserveLoggingContext
-from synapse.util.async_helpers import Linearizer
 from synapse.util.rlimit import change_resource_limit
 from synapse.util.versionstring import get_version_string
 
@@ -100,8 +99,6 @@ def start_reactor(
         logger (logging.Logger): logger instance to pass to Daemonize
     """
 
-    install_dns_limiter(reactor)
-
     def run():
         # make sure that we run the reactor with the sentinel log context,
         # otherwise other PreserveLoggingContext instances will get confused
@@ -315,81 +312,3 @@ def setup_sentry(hs):
         name = hs.config.worker_name if hs.config.worker_name else "master"
         scope.set_tag("worker_app", app)
         scope.set_tag("worker_name", name)
-
-
-def install_dns_limiter(reactor, max_dns_requests_in_flight=100):
-    """Replaces the resolver with one that limits the number of in flight DNS
-    requests.
-
-    This is to workaround https://twistedmatrix.com/trac/ticket/9620, where we
-    can run out of file descriptors and infinite loop if we attempt to do too
-    many DNS queries at once
-    """
-    new_resolver = _LimitedHostnameResolver(
-        reactor.nameResolver, max_dns_requests_in_flight,
-    )
-
-    reactor.installNameResolver(new_resolver)
-
-
-class _LimitedHostnameResolver(object):
-    """Wraps a IHostnameResolver, limiting the number of in-flight DNS lookups.
-    """
-
-    def __init__(self, resolver, max_dns_requests_in_flight):
-        self._resolver = resolver
-        self._limiter = Linearizer(
-            name="dns_client_limiter", max_count=max_dns_requests_in_flight,
-        )
-
-    def resolveHostName(self, resolutionReceiver, hostName, portNumber=0,
-                        addressTypes=None, transportSemantics='TCP'):
-        # Note this is happening deep within the reactor, so we don't need to
-        # worry about log contexts.
-
-        # We need this function to return `resolutionReceiver` so we do all the
-        # actual logic involving deferreds in a separate function.
-        self._resolve(
-            resolutionReceiver, hostName, portNumber,
-            addressTypes, transportSemantics,
-        )
-
-        return resolutionReceiver
-
-    @defer.inlineCallbacks
-    def _resolve(self, resolutionReceiver, hostName, portNumber=0,
-                 addressTypes=None, transportSemantics='TCP'):
-
-        with (yield self._limiter.queue(())):
-            # resolveHostName doesn't return a Deferred, so we need to hook into
-            # the receiver interface to get told when resolution has finished.
-
-            deferred = defer.Deferred()
-            receiver = _DeferredResolutionReceiver(resolutionReceiver, deferred)
-
-            self._resolver.resolveHostName(
-                receiver, hostName, portNumber,
-                addressTypes, transportSemantics,
-            )
-
-            yield deferred
-
-
-class _DeferredResolutionReceiver(object):
-    """Wraps a IResolutionReceiver and simply resolves the given deferred when
-    resolution is complete
-    """
-
-    def __init__(self, receiver, deferred):
-        self._receiver = receiver
-        self._deferred = deferred
-
-    def resolutionBegan(self, resolutionInProgress):
-        self._receiver.resolutionBegan(resolutionInProgress)
-
-    def addressResolved(self, address):
-        self._receiver.addressResolved(address)
-
-    def resolutionComplete(self):
-        self._deferred.callback(())
-        self._receiver.resolutionComplete()

synapse/app/client_reader.py

@@ -45,7 +45,6 @@ from synapse.replication.slave.storage.room import RoomStore
 from synapse.replication.slave.storage.transactions import SlavedTransactionStore
 from synapse.replication.tcp.client import ReplicationClientHandler
 from synapse.rest.client.v1.login import LoginRestServlet
-from synapse.rest.client.v1.push_rule import PushRuleRestServlet
 from synapse.rest.client.v1.room import (
     JoinedRoomMemberListRestServlet,
     PublicRoomListRestServlet,
@@ -53,11 +52,9 @@ from synapse.rest.client.v1.room import (
     RoomMemberListRestServlet,
     RoomStateRestServlet,
 )
-from synapse.rest.client.v1.voip import VoipRestServlet
 from synapse.rest.client.v2_alpha.account import ThreepidRestServlet
 from synapse.rest.client.v2_alpha.keys import KeyChangesServlet, KeyQueryServlet
 from synapse.rest.client.v2_alpha.register import RegisterRestServlet
-from synapse.rest.client.versions import VersionsRestServlet
 from synapse.server import HomeServer
 from synapse.storage.engines import create_engine
 from synapse.util.httpresourcetree import create_resource_tree
@@ -112,12 +109,12 @@ class ClientReaderServer(HomeServer):
                     ThreepidRestServlet(self).register(resource)
                     KeyQueryServlet(self).register(resource)
                     KeyChangesServlet(self).register(resource)
-                    VoipRestServlet(self).register(resource)
-                    PushRuleRestServlet(self).register(resource)
-                    VersionsRestServlet().register(resource)
 
                     resources.update({
-                        "/_matrix/client": resource,
+                        "/_matrix/client/r0": resource,
+                        "/_matrix/client/unstable": resource,
+                        "/_matrix/client/v2_alpha": resource,
+                        "/_matrix/client/api/v1": resource,
                     })
 
         root_resource = create_resource_tree(resources, NoResource())

synapse/app/homeserver.py

@@ -62,7 +62,6 @@ from synapse.python_dependencies import check_requirements
 from synapse.replication.http import REPLICATION_PREFIX, ReplicationRestResource
 from synapse.replication.tcp.resource import ReplicationStreamProtocolFactory
 from synapse.rest import ClientRestResource
-from synapse.rest.admin import AdminRestResource
 from synapse.rest.key.v2 import KeyApiV2Resource
 from synapse.rest.media.v0.content_repository import ContentRepoResource
 from synapse.rest.well_known import WellKnownResource
@@ -181,7 +180,6 @@ class SynapseHomeServer(HomeServer):
                 "/_matrix/client/v2_alpha": client_resource,
                 "/_matrix/client/versions": client_resource,
                 "/.well-known/matrix/client": WellKnownResource(self),
-                "/_synapse/admin": AdminRestResource(self),
             })
 
             if self.get_config().saml2_enabled:

synapse/appservice/__init__.py

@@ -265,7 +265,7 @@ class ApplicationService(object):
     def is_exclusive_room(self, room_id):
         return self._is_exclusive(ApplicationService.NS_ROOMS, room_id)
 
-    def get_exclusive_user_regexes(self):
+    def get_exlusive_user_regexes(self):
         """Get the list of regexes used to determine if a user is exclusively
         registered by the AS
         """

synapse/config/emailconfig.py

@@ -71,12 +71,6 @@ class EmailConfig(Config):
             self.email_notif_from = email_config["notif_from"]
             self.email_notif_template_html = email_config["notif_template_html"]
             self.email_notif_template_text = email_config["notif_template_text"]
-            self.email_expiry_template_html = email_config.get(
-                "expiry_template_html", "notice_expiry.html",
-            )
-            self.email_expiry_template_text = email_config.get(
-                "expiry_template_text", "notice_expiry.txt",
-            )
 
             template_dir = email_config.get("template_dir")
             # we need an absolute path, because we change directory after starting (and
@@ -126,7 +120,7 @@ class EmailConfig(Config):
 
     def default_config(self, config_dir_path, server_name, **kwargs):
         return """
-        # Enable sending emails for notification events or expiry notices
+        # Enable sending emails for notification events
         # Defining a custom URL for Riot is only needed if email notifications
         # should contain links to a self-hosted installation of Riot; when set
         # the "app_name" setting is ignored.
@@ -148,9 +142,6 @@ class EmailConfig(Config):
         #   #template_dir: res/templates
         #   notif_template_html: notif_mail.html
         #   notif_template_text: notif_mail.txt
-        #   # Templates for account expiry notices.
-        #   expiry_template_html: notice_expiry.html
-        #   expiry_template_text: notice_expiry.txt
         #   notif_for_new_users: True
         #   riot_base_url: "http://localhost/riot"
         """

synapse/config/ratelimiting.py

@@ -16,56 +16,16 @@ from ._base import Config
 
 
 class RateLimitConfig(object):
-    def __init__(self, config, defaults={"per_second": 0.17, "burst_count": 3.0}):
-        self.per_second = config.get("per_second", defaults["per_second"])
-        self.burst_count = config.get("burst_count", defaults["burst_count"])
-
-
-class FederationRateLimitConfig(object):
-    _items_and_default = {
-        "window_size": 10000,
-        "sleep_limit": 10,
-        "sleep_delay": 500,
-        "reject_limit": 50,
-        "concurrent": 3,
-    }
-
-    def __init__(self, **kwargs):
-        for i in self._items_and_default.keys():
-            setattr(self, i, kwargs.get(i) or self._items_and_default[i])
+    def __init__(self, config):
+        self.per_second = config.get("per_second", 0.17)
+        self.burst_count = config.get("burst_count", 3.0)
 
 
 class RatelimitConfig(Config):
+
     def read_config(self, config):
-
-        # Load the new-style messages config if it exists. Otherwise fall back
-        # to the old method.
-        if "rc_message" in config:
-            self.rc_message = RateLimitConfig(
-                config["rc_message"], defaults={"per_second": 0.2, "burst_count": 10.0}
-            )
-        else:
-            self.rc_message = RateLimitConfig(
-                {
-                    "per_second": config.get("rc_messages_per_second", 0.2),
-                    "burst_count": config.get("rc_message_burst_count", 10.0),
-                }
-            )
-
-        # Load the new-style federation config, if it exists. Otherwise, fall
-        # back to the old method.
-        if "federation_rc" in config:
-            self.rc_federation = FederationRateLimitConfig(**config["rc_federation"])
-        else:
-            self.rc_federation = FederationRateLimitConfig(
-                **{
-                    "window_size": config.get("federation_rc_window_size"),
-                    "sleep_limit": config.get("federation_rc_sleep_limit"),
-                    "sleep_delay": config.get("federation_rc_sleep_delay"),
-                    "reject_limit": config.get("federation_rc_reject_limit"),
-                    "concurrent": config.get("federation_rc_concurrent"),
-                }
-            )
+        self.rc_messages_per_second = config.get("rc_messages_per_second", 0.2)
+        self.rc_message_burst_count = config.get("rc_message_burst_count", 10.0)
 
         self.rc_registration = RateLimitConfig(config.get("rc_registration", {}))
 
@@ -73,26 +33,38 @@ class RatelimitConfig(Config):
         self.rc_login_address = RateLimitConfig(rc_login_config.get("address", {}))
         self.rc_login_account = RateLimitConfig(rc_login_config.get("account", {}))
         self.rc_login_failed_attempts = RateLimitConfig(
-            rc_login_config.get("failed_attempts", {})
+            rc_login_config.get("failed_attempts", {}),
         )
 
+        self.federation_rc_window_size = config.get("federation_rc_window_size", 1000)
+        self.federation_rc_sleep_limit = config.get("federation_rc_sleep_limit", 10)
+        self.federation_rc_sleep_delay = config.get("federation_rc_sleep_delay", 500)
+        self.federation_rc_reject_limit = config.get("federation_rc_reject_limit", 50)
+        self.federation_rc_concurrent = config.get("federation_rc_concurrent", 3)
+
         self.federation_rr_transactions_per_room_per_second = config.get(
-            "federation_rr_transactions_per_room_per_second", 50
+            "federation_rr_transactions_per_room_per_second", 50,
         )
 
     def default_config(self, **kwargs):
         return """\
         ## Ratelimiting ##
 
-        # Ratelimiting settings for client actions (registration, login, messaging).
+        # Number of messages a client can send per second
+        #
+        #rc_messages_per_second: 0.2
+
+        # Number of message a client can send before being throttled
+        #
+        #rc_message_burst_count: 10.0
+
+        # Ratelimiting settings for registration and login.
         #
         # Each ratelimiting configuration is made of two parameters:
         #   - per_second: number of requests a client can send per second.
         #   - burst_count: number of requests a client can send before being throttled.
         #
         # Synapse currently uses the following configurations:
-        #   - one for messages that ratelimits sending based on the account the client
-        #     is using
         #   - one for registration that ratelimits registration requests based on the
         #     client's IP address.
         #   - one for login that ratelimits login requests based on the client's IP
@@ -105,10 +77,6 @@ class RatelimitConfig(Config):
         #
         # The defaults are as shown below.
         #
-        #rc_message:
-        #  per_second: 0.2
-        #  burst_count: 10
-        #
         #rc_registration:
         #  per_second: 0.17
         #  burst_count: 3
@@ -124,28 +92,29 @@ class RatelimitConfig(Config):
         #    per_second: 0.17
         #    burst_count: 3
 
+        # The federation window size in milliseconds
+        #
+        #federation_rc_window_size: 1000
 
-        # Ratelimiting settings for incoming federation
+        # The number of federation requests from a single server in a window
+        # before the server will delay processing the request.
         #
-        # The rc_federation configuration is made up of the following settings:
-        #   - window_size: window size in milliseconds
-        #   - sleep_limit: number of federation requests from a single server in
-        #     a window before the server will delay processing the request.
-        #   - sleep_delay: duration in milliseconds to delay processing events
-        #     from remote servers by if they go over the sleep limit.
-        #   - reject_limit: maximum number of concurrent federation requests
-        #     allowed from a single server
-        #   - concurrent: number of federation requests to concurrently process
-        #     from a single server
+        #federation_rc_sleep_limit: 10
+
+        # The duration in milliseconds to delay processing events from
+        # remote servers by if they go over the sleep limit.
         #
-        # The defaults are as shown below.
+        #federation_rc_sleep_delay: 500
+
+        # The maximum number of concurrent federation requests allowed
+        # from a single server
         #
-        #rc_federation:
-        #  window_size: 1000
-        #  sleep_limit: 10
-        #  sleep_delay: 500
-        #  reject_limit: 50
-        #  concurrent: 3
+        #federation_rc_reject_limit: 50
+
+        # The number of federation requests to concurrently process from a
+        # single server
+        #
+        #federation_rc_concurrent: 3
 
         # Target outgoing federation transaction frequency for sending read-receipts,
         # per-room.

synapse/config/registration.py

@@ -20,29 +20,6 @@ from synapse.types import RoomAlias
 from synapse.util.stringutils import random_string_with_symbols
 
 
-class AccountValidityConfig(Config):
-    def __init__(self, config, synapse_config):
-        self.enabled = config.get("enabled", False)
-        self.renew_by_email_enabled = ("renew_at" in config)
-
-        if self.enabled:
-            if "period" in config:
-                self.period = self.parse_duration(config["period"])
-            else:
-                raise ConfigError("'period' is required when using account validity")
-
-            if "renew_at" in config:
-                self.renew_at = self.parse_duration(config["renew_at"])
-
-            if "renew_email_subject" in config:
-                self.renew_email_subject = config["renew_email_subject"]
-            else:
-                self.renew_email_subject = "Renew your %(app)s account"
-
-        if self.renew_by_email_enabled and "public_baseurl" not in synapse_config:
-            raise ConfigError("Can't send renewal emails without 'public_baseurl'")
-
-
 class RegistrationConfig(Config):
 
     def read_config(self, config):
@@ -54,25 +31,10 @@ class RegistrationConfig(Config):
                 strtobool(str(config["disable_registration"]))
             )
 
-        self.account_validity = AccountValidityConfig(
-            config.get("account_validity", {}), config,
-        )
-
         self.registrations_require_3pid = config.get("registrations_require_3pid", [])
         self.allowed_local_3pids = config.get("allowed_local_3pids", [])
-        self.check_is_for_allowed_local_3pids = config.get(
-            "check_is_for_allowed_local_3pids", None
-        )
-        self.allow_invited_3pids = config.get("allow_invited_3pids", False)
-
-        self.disable_3pid_changes = config.get("disable_3pid_changes", False)
-
         self.enable_3pid_lookup = config.get("enable_3pid_lookup", True)
         self.registration_shared_secret = config.get("registration_shared_secret")
-        self.register_mxid_from_3pid = config.get("register_mxid_from_3pid")
-        self.register_just_use_email_for_display_name = config.get(
-            "register_just_use_email_for_display_name", False,
-        )
 
         self.bcrypt_rounds = config.get("bcrypt_rounds", 12)
         self.trusted_third_party_id_servers = config.get(
@@ -92,16 +54,6 @@ class RegistrationConfig(Config):
                 raise ConfigError('Invalid auto_join_rooms entry %s' % (room_alias,))
         self.autocreate_auto_join_rooms = config.get("autocreate_auto_join_rooms", True)
 
-        self.disable_set_displayname = config.get("disable_set_displayname", False)
-        self.disable_set_avatar_url = config.get("disable_set_avatar_url", False)
-
-        self.replicate_user_profiles_to = config.get("replicate_user_profiles_to", [])
-        if not isinstance(self.replicate_user_profiles_to, list):
-            self.replicate_user_profiles_to = [self.replicate_user_profiles_to, ]
-
-        self.shadow_server = config.get("shadow_server", None)
-        self.rewrite_identity_server_urls = config.get("rewrite_identity_server_urls", {})
-
         self.disable_msisdn_registration = (
             config.get("disable_msisdn_registration", False)
         )
@@ -124,32 +76,6 @@ class RegistrationConfig(Config):
         #
         #enable_registration: false
 
-        # Optional account validity configuration. This allows for accounts to be denied
-        # any request after a given period.
-        #
-        # ``enabled`` defines whether the account validity feature is enabled. Defaults
-        # to False.
-        #
-        # ``period`` allows setting the period after which an account is valid
-        # after its registration. When renewing the account, its validity period
-        # will be extended by this amount of time. This parameter is required when using
-        # the account validity feature.
-        #
-        # ``renew_at`` is the amount of time before an account's expiry date at which
-        # Synapse will send an email to the account's email address with a renewal link.
-        # This needs the ``email`` and ``public_baseurl`` configuration sections to be
-        # filled.
-        #
-        # ``renew_email_subject`` is the subject of the email sent out with the renewal
-        # link. ``%%(app)s`` can be used as a placeholder for the ``app_name`` parameter
-        # from the ``email`` section.
-        #
-        #account_validity:
-        #  enabled: True
-        #  period: 6w
-        #  renew_at: 1w
-        #  renew_email_subject: "Renew your %%(app)s account"
-
         # The user must provide all of the below types of 3PID when registering.
         #
         #registrations_require_3pid:
@@ -161,32 +87,9 @@ class RegistrationConfig(Config):
         #
         #disable_msisdn_registration: true
 
-        # Derive the user's matrix ID from a type of 3PID used when registering.
-        # This overrides any matrix ID the user proposes when calling /register
-        # The 3PID type should be present in registrations_require_3pid to avoid
-        # users failing to register if they don't specify the right kind of 3pid.
-        #
-        #register_mxid_from_3pid: email
-
-        # Uncomment to set the display name of new users to their email address,
-        # rather than using the default heuristic.
-        #
-        #register_just_use_email_for_display_name: true
-
         # Mandate that users are only allowed to associate certain formats of
         # 3PIDs with accounts on this server.
         #
-        # Use an Identity Server to establish which 3PIDs are allowed to register?
-        # Overrides allowed_local_3pids below.
-        #
-        #check_is_for_allowed_local_3pids: matrix.org
-        #
-        # If you are using an IS you can also check whether that IS registers
-        # pending invites for the given 3PID (and then allow it to sign up on
-        # the platform):
-        #
-        #allow_invited_3pids: False
-        #
         #allowed_local_3pids:
         #  - medium: email
         #    pattern: '.*@matrix\\.org'
@@ -195,11 +98,6 @@ class RegistrationConfig(Config):
         #  - medium: msisdn
         #    pattern: '\\+44'
 
-        # If true, stop users from trying to change the 3PIDs associated with
-        # their accounts.
-        #
-        #disable_3pid_changes: False
-
         # Enable 3PIDs lookup requests to identity servers from this server.
         #
         #enable_3pid_lookup: true
@@ -241,30 +139,6 @@ class RegistrationConfig(Config):
         #  - matrix.org
         #  - vector.im
 
-        # If enabled, user IDs, display names and avatar URLs will be replicated
-        # to this server whenever they change.
-        # This is an experimental API currently implemented by sydent to support
-        # cross-homeserver user directories.
-        #
-        #replicate_user_profiles_to: example.com
-
-        # If specified, attempt to replay registrations, profile changes & 3pid
-        # bindings on the given target homeserver via the AS API. The HS is authed
-        # via a given AS token.
-        #
-        #shadow_server:
-        #  hs_url: https://shadow.example.com
-        #  hs: shadow.example.com
-        #  as_token: 12u394refgbdhivsia
-
-        # If enabled, don't let users set their own display names/avatars
-        # other than for the very first time (unless they are a server admin).
-        # Useful when provisioning users based on the contents of a 3rd party
-        # directory and to avoid ambiguities.
-        #
-        #disable_set_displayname: False
-        #disable_set_avatar_url: False
-
         # Users who register on this homeserver will automatically be joined
         # to these rooms
         #

synapse/config/repository.py

@@ -186,21 +186,17 @@ class ContentRepositoryConfig(Config):
             except ImportError:
                 raise ConfigError(MISSING_NETADDR)
 
-            if "url_preview_ip_range_blacklist" not in config:
+            if "url_preview_ip_range_blacklist" in config:
+                self.url_preview_ip_range_blacklist = IPSet(
+                    config["url_preview_ip_range_blacklist"]
+                )
+            else:
                 raise ConfigError(
                     "For security, you must specify an explicit target IP address "
                     "blacklist in url_preview_ip_range_blacklist for url previewing "
                     "to work"
                 )
 
-            self.url_preview_ip_range_blacklist = IPSet(
-                config["url_preview_ip_range_blacklist"]
-            )
-
-            # we always blacklist '0.0.0.0' and '::', which are supposed to be
-            # unroutable addresses.
-            self.url_preview_ip_range_blacklist.update(['0.0.0.0', '::'])
-
             self.url_preview_ip_range_whitelist = IPSet(
                 config.get("url_preview_ip_range_whitelist", ())
             )
@@ -264,12 +260,11 @@ class ContentRepositoryConfig(Config):
         #thumbnail_sizes:
 %(formatted_thumbnail_sizes)s
 
-        # Is the preview URL API enabled?
+        # Is the preview URL API enabled?  If enabled, you *must* specify
+        # an explicit url_preview_ip_range_blacklist of IPs that the spider is
+        # denied from accessing.
         #
-        # 'false' by default: uncomment the following to enable it (and specify a
-        # url_preview_ip_range_blacklist blacklist).
-        #
-        #url_preview_enabled: true
+        #url_preview_enabled: false
 
         # List of IP address CIDR ranges that the URL preview spider is denied
         # from accessing.  There are no defaults: you must explicitly
@@ -279,12 +274,6 @@ class ContentRepositoryConfig(Config):
         # synapse to issue arbitrary GET requests to your internal services,
         # causing serious security issues.
         #
-        # (0.0.0.0 and :: are always blacklisted, whether or not they are explicitly
-        # listed here, since they correspond to unroutable addresses.)
-        #
-        # This must be specified if url_preview_enabled is set. It is recommended that
-        # you uncomment the following list as a starting point.
-        #
         #url_preview_ip_range_blacklist:
         #  - '127.0.0.0/8'
         #  - '10.0.0.0/8'
@@ -295,7 +284,7 @@ class ContentRepositoryConfig(Config):
         #  - '::1/128'
         #  - 'fe80::/64'
         #  - 'fc00::/7'
-
+        #
         # List of IP address CIDR ranges that the URL preview spider is allowed
         # to access even if they are specified in url_preview_ip_range_blacklist.
         # This is useful for specifying exceptions to wide-ranging blacklisted

synapse/config/saml2_config.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2018 New Vector Ltd
+# Copyright 2018 New Vector Ltd.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/server.py

@@ -1,7 +1,6 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
 # Copyright 2017-2018 New Vector Ltd
-# Copyright 2019 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -18,8 +17,6 @@
 import logging
 import os.path
 
-from netaddr import IPSet
-
 from synapse.http.endpoint import parse_and_validate_server_name
 from synapse.python_dependencies import DependencyException, check_requirements
 
@@ -75,19 +72,6 @@ class ServerConfig(Config):
         # master, potentially causing inconsistency.
         self.enable_media_repo = config.get("enable_media_repo", True)
 
-        # Whether to require authentication to retrieve profile data (avatars,
-        # display names) of other users through the client API.
-        self.require_auth_for_profile_requests = config.get(
-            "require_auth_for_profile_requests", False,
-        )
-
-        # If set to 'True', requires authentication to access the server's
-        # public rooms directory through the client API, and forbids any other
-        # homeserver to fetch it via federation.
-        self.restrict_public_rooms_to_local_users = config.get(
-            "restrict_public_rooms_to_local_users", False,
-        )
-
         # whether to enable search. If disabled, new entries will not be inserted
         # into the search tables and they will not be indexed. Users will receive
         # errors when attempting to search for messages.
@@ -130,34 +114,14 @@ class ServerConfig(Config):
         # FIXME: federation_domain_whitelist needs sytests
         self.federation_domain_whitelist = None
         federation_domain_whitelist = config.get(
-            "federation_domain_whitelist", None,
+            "federation_domain_whitelist", None
         )
-
+        # turn the whitelist into a hash for speed of lookup
         if federation_domain_whitelist is not None:
-            # turn the whitelist into a hash for speed of lookup
             self.federation_domain_whitelist = {}
-
             for domain in federation_domain_whitelist:
                 self.federation_domain_whitelist[domain] = True
 
-        self.federation_ip_range_blacklist = config.get(
-            "federation_ip_range_blacklist", [],
-        )
-
-        # Attempt to create an IPSet from the given ranges
-        try:
-            self.federation_ip_range_blacklist = IPSet(
-                self.federation_ip_range_blacklist
-            )
-
-            # Always blacklist 0.0.0.0, ::
-            self.federation_ip_range_blacklist.update(["0.0.0.0", "::"])
-        except Exception as e:
-            raise ConfigError(
-                "Invalid range(s) provided in "
-                "federation_ip_range_blacklist: %s" % e
-            )
-
         if self.public_baseurl is not None:
             if self.public_baseurl[-1] != '/':
                 self.public_baseurl += '/'
@@ -168,16 +132,6 @@ class ServerConfig(Config):
         # sending out any replication updates.
         self.replication_torture_level = config.get("replication_torture_level")
 
-        # Whether to require a user to be in the room to add an alias to it.
-        # Defaults to True.
-        self.require_membership_for_aliases = config.get(
-            "require_membership_for_aliases", True,
-        )
-
-        # Whether to allow per-room membership profiles through the send of membership
-        # events with profile information that differ from the target's global profile.
-        self.allow_per_room_profiles = config.get("allow_per_room_profiles", True)
-
         self.listeners = []
         for listener in config.get("listeners", []):
             if not isinstance(listener.get("port", None), int):
@@ -365,20 +319,6 @@ class ServerConfig(Config):
         #
         #use_presence: false
 
-        # Whether to require authentication to retrieve profile data (avatars,
-        # display names) of other users through the client API. Defaults to
-        # 'false'. Note that profile data is also available via the federation
-        # API, so this setting is of limited value if federation is enabled on
-        # the server.
-        #
-        #require_auth_for_profile_requests: true
-
-        # If set to 'true', requires authentication to access the server's
-        # public rooms directory through the client API, and forbids any other
-        # homeserver to fetch it via federation. Defaults to 'false'.
-        #
-        #restrict_public_rooms_to_local_users: true
-
         # The GC threshold parameters to pass to `gc.set_threshold`, if defined
         #
         #gc_thresholds: [700, 10, 10]
@@ -411,24 +351,6 @@ class ServerConfig(Config):
         #  - nyc.example.com
         #  - syd.example.com
 
-        # Prevent federation requests from being sent to the following
-        # blacklist IP address CIDR ranges. If this option is not specified, or
-        # specified with an empty list, no ip range blacklist will be enforced.
-        #
-        # (0.0.0.0 and :: are always blacklisted, whether or not they are explicitly
-        # listed here, since they correspond to unroutable addresses.)
-        #
-        federation_ip_range_blacklist:
-          - '127.0.0.0/8'
-          - '10.0.0.0/8'
-          - '172.16.0.0/12'
-          - '192.168.0.0/16'
-          - '100.64.0.0/10'
-          - '169.254.0.0/16'
-          - '::1/128'
-          - 'fe80::/64'
-          - 'fc00::/7'
-
         # List of ports that Synapse should listen on, their purpose and their
         # configuration.
         #
@@ -464,8 +386,8 @@ class ServerConfig(Config):
         #
         # Valid resource names are:
         #
-        #   client: the client-server API (/_matrix/client), and the synapse admin
-        #       API (/_synapse/admin). Also implies 'media' and 'static'.
+        #   client: the client-server API (/_matrix/client). Also implies 'media' and
+        #       'static'.
         #
         #   consent: user consent forms (/_matrix/consent). See
         #       docs/consent_tracking.md.
@@ -566,17 +488,6 @@ class ServerConfig(Config):
 
         # Used by phonehome stats to group together related servers.
         #server_context: context
-
-        # Whether to require a user to be in the room to add an alias to it.
-        # Defaults to 'true'.
-        #
-        #require_membership_for_aliases: false
-
-        # Whether to allow per-room membership profiles through the send of membership
-        # events with profile information that differ from the target's global profile.
-        # Defaults to 'true'.
-        #
-        #allow_per_room_profiles: false
         """ % locals()
 
     def read_arguments(self, args):

synapse/config/tls.py

@@ -24,10 +24,8 @@ import six
 from unpaddedbase64 import encode_base64
 
 from OpenSSL import crypto
-from twisted.internet._sslverify import Certificate, trustRootFromCertificates
 
 from synapse.config._base import Config, ConfigError
-from synapse.util import glob_to_regex
 
 logger = logging.getLogger(__name__)
 
@@ -72,53 +70,6 @@ class TlsConfig(Config):
 
         self.tls_fingerprints = list(self._original_tls_fingerprints)
 
-        # Whether to verify certificates on outbound federation traffic
-        self.federation_verify_certificates = config.get(
-            "federation_verify_certificates", False,
-        )
-
-        # Whitelist of domains to not verify certificates for
-        fed_whitelist_entries = config.get(
-            "federation_certificate_verification_whitelist", [],
-        )
-
-        # Support globs (*) in whitelist values
-        self.federation_certificate_verification_whitelist = []
-        for entry in fed_whitelist_entries:
-            # Convert globs to regex
-            entry_regex = glob_to_regex(entry)
-            self.federation_certificate_verification_whitelist.append(entry_regex)
-
-        # List of custom certificate authorities for federation traffic validation
-        custom_ca_list = config.get(
-            "federation_custom_ca_list", None,
-        )
-
-        # Read in and parse custom CA certificates
-        self.federation_ca_trust_root = None
-        if custom_ca_list is not None:
-            if len(custom_ca_list) == 0:
-                # A trustroot cannot be generated without any CA certificates.
-                # Raise an error if this option has been specified without any
-                # corresponding certificates.
-                raise ConfigError("federation_custom_ca_list specified without "
-                                  "any certificate files")
-
-            certs = []
-            for ca_file in custom_ca_list:
-                logger.debug("Reading custom CA certificate file: %s", ca_file)
-                content = self.read_file(ca_file)
-
-                # Parse the CA certificates
-                try:
-                    cert_base = Certificate.loadPEM(content)
-                    certs.append(cert_base)
-                except Exception as e:
-                    raise ConfigError("Error parsing custom CA certificate file %s: %s"
-                                      % (ca_file, e))
-
-            self.federation_ca_trust_root = trustRootFromCertificates(certs)
-
         # This config option applies to non-federation HTTP clients
         # (e.g. for talking to recaptcha, identity servers, and such)
         # It should never be used in production, and is intended for
@@ -148,15 +99,15 @@ class TlsConfig(Config):
         try:
             with open(self.tls_certificate_file, 'rb') as f:
                 cert_pem = f.read()
-        except Exception as e:
-            raise ConfigError("Failed to read existing certificate file %s: %s"
-                              % (self.tls_certificate_file, e))
+        except Exception:
+            logger.exception("Failed to read existing certificate off disk!")
+            raise
 
         try:
             tls_certificate = crypto.load_certificate(crypto.FILETYPE_PEM, cert_pem)
-        except Exception as e:
-            raise ConfigError("Failed to parse existing certificate file %s: %s"
-                              % (self.tls_certificate_file, e))
+        except Exception:
+            logger.exception("Failed to parse existing certificate off disk!")
+            raise
 
         if not allow_self_signed:
             if tls_certificate.get_subject() == tls_certificate.get_issuer():
@@ -241,40 +192,6 @@ class TlsConfig(Config):
         #
         #tls_private_key_path: "%(tls_private_key_path)s"
 
-        # Whether to verify TLS certificates when sending federation traffic.
-        #
-        # This currently defaults to `false`, however this will change in
-        # Synapse 1.0 when valid federation certificates will be required.
-        #
-        #federation_verify_certificates: true
-
-        # Skip federation certificate verification on the following whitelist
-        # of domains.
-        #
-        # This setting should only be used in very specific cases, such as
-        # federation over Tor hidden services and similar. For private networks
-        # of homeservers, you likely want to use a private CA instead.
-        #
-        # Only effective if federation_verify_certicates is `true`.
-        #
-        #federation_certificate_verification_whitelist:
-        #  - lon.example.com
-        #  - *.domain.com
-        #  - *.onion
-
-        # List of custom certificate authorities for federation traffic.
-        #
-        # This setting should only normally be used within a private network of
-        # homeservers.
-        #
-        # Note that this list will replace those that are provided by your
-        # operating environment. Certificates must be in PEM format.
-        #
-        #federation_custom_ca_list:
-        #  - myCA1.pem
-        #  - myCA2.pem
-        #  - myCA3.pem
-
         # ACME support: This will configure Synapse to request a valid TLS certificate
         # for your configured `server_name` via Let's Encrypt.
         #

synapse/config/user_directory.py

@@ -24,7 +24,6 @@ class UserDirectoryConfig(Config):
     def read_config(self, config):
         self.user_directory_search_enabled = True
         self.user_directory_search_all_users = False
-        self.user_directory_defer_to_id_server = None
         user_directory_config = config.get("user_directory", None)
         if user_directory_config:
             self.user_directory_search_enabled = (
@@ -33,9 +32,6 @@ class UserDirectoryConfig(Config):
             self.user_directory_search_all_users = (
                 user_directory_config.get("search_all_users", False)
             )
-            self.user_directory_defer_to_id_server = (
-                user_directory_config.get("defer_to_id_server", None)
-            )
 
     def default_config(self, config_dir_path, server_name, **kwargs):
         return """
@@ -54,9 +50,4 @@ class UserDirectoryConfig(Config):
         #user_directory:
         #  enabled: true
         #  search_all_users: false
-        #
-        #  # If this is set, user search will be delegated to this ID server instead
-        #  # of synapse performing the search itself.
-        #  # This is an experimental API.
-        #  defer_to_id_server: https://id.example.com
         """

synapse/crypto/context_factory.py

@@ -18,10 +18,10 @@ import logging
 from zope.interface import implementer
 
 from OpenSSL import SSL, crypto
-from twisted.internet._sslverify import ClientTLSOptions, _defaultCurveName
+from twisted.internet._sslverify import _defaultCurveName
 from twisted.internet.abstract import isIPAddress, isIPv6Address
 from twisted.internet.interfaces import IOpenSSLClientConnectionCreator
-from twisted.internet.ssl import CertificateOptions, ContextFactory, platformTrust
+from twisted.internet.ssl import CertificateOptions, ContextFactory
 from twisted.python.failure import Failure
 
 logger = logging.getLogger(__name__)
@@ -90,7 +90,7 @@ def _tolerateErrors(wrapped):
 
 
 @implementer(IOpenSSLClientConnectionCreator)
-class ClientTLSOptionsNoVerify(object):
+class ClientTLSOptions(object):
     """
     Client creator for TLS without certificate identity verification. This is a
     copy of twisted.internet._sslverify.ClientTLSOptions with the identity
@@ -127,30 +127,9 @@ class ClientTLSOptionsFactory(object):
     to remote servers for federation."""
 
     def __init__(self, config):
-        self._config = config
-        self._options_noverify = CertificateOptions()
-
-        # Check if we're using a custom list of a CA certificates
-        trust_root = config.federation_ca_trust_root
-        if trust_root is None:
-            # Use CA root certs provided by OpenSSL
-            trust_root = platformTrust()
-
-        self._options_verify = CertificateOptions(trustRoot=trust_root)
+        # We don't use config options yet
+        self._options = CertificateOptions(verify=False)
 
     def get_options(self, host):
         # Use _makeContext so that we get a fresh OpenSSL CTX each time.
-
-        # Check if certificate verification has been enabled
-        should_verify = self._config.federation_verify_certificates
-
-        # Check if we've disabled certificate verification for this host
-        if should_verify:
-            for regex in self._config.federation_certificate_verification_whitelist:
-                if regex.match(host):
-                    should_verify = False
-                    break
-
-        if should_verify:
-            return ClientTLSOptions(host, self._options_verify._makeContext())
-        return ClientTLSOptionsNoVerify(host, self._options_noverify._makeContext())
+        return ClientTLSOptions(host, self._options._makeContext())

synapse/crypto/keyring.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2017, 2018 New Vector Ltd
+# Copyright 2017, 2018 New Vector Ltd.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -20,7 +20,6 @@ from collections import namedtuple
 from six import raise_from
 from six.moves import urllib
 
-import nacl.signing
 from signedjson.key import (
     decode_verify_key_bytes,
     encode_verify_key_base64,
@@ -114,54 +113,40 @@ class Keyring(object):
                 server_name. The deferreds run their callbacks in the sentinel
                 logcontext.
         """
-        # a list of VerifyKeyRequests
         verify_requests = []
-        handle = preserve_fn(_handle_key_deferred)
 
-        def process(server_name, json_object):
-            """Process an entry in the request list
+        for server_name, json_object in server_and_json:
 
-            Given a (server_name, json_object) pair from the request list,
-            adds a key request to verify_requests, and returns a deferred which will
-            complete or fail (in the sentinel context) when verification completes.
-            """
             key_ids = signature_ids(json_object, server_name)
-
             if not key_ids:
-                return defer.fail(
-                    SynapseError(
-                        400,
-                        "Not signed by %s" % (server_name,),
-                        Codes.UNAUTHORIZED,
-                    )
-                )
+                logger.warn("Request from %s: no supported signature keys",
+                            server_name)
+                deferred = defer.fail(SynapseError(
+                    400,
+                    "Not signed with a supported algorithm",
+                    Codes.UNAUTHORIZED,
+                ))
+            else:
+                deferred = defer.Deferred()
 
             logger.debug("Verifying for %s with key_ids %s",
                          server_name, key_ids)
 
-            # add the key request to the queue, but don't start it off yet.
             verify_request = VerifyKeyRequest(
-                server_name, key_ids, json_object, defer.Deferred(),
+                server_name, key_ids, json_object, deferred
             )
+
             verify_requests.append(verify_request)
 
-            # now run _handle_key_deferred, which will wait for the key request
-            # to complete and then do the verification.
-            #
-            # We want _handle_key_request to log to the right context, so we
-            # wrap it with preserve_fn (aka run_in_background)
-            return handle(verify_request)
+        run_in_background(self._start_key_lookups, verify_requests)
 
-        results = [
-            process(server_name, json_object)
-            for server_name, json_object in server_and_json
+        # Pass those keys to handle_key_deferred so that the json object
+        # signatures can be verified
+        handle = preserve_fn(_handle_key_deferred)
+        return [
+            handle(rq) for rq in verify_requests
         ]
 
-        if verify_requests:
-            run_in_background(self._start_key_lookups, verify_requests)
-
-        return results
-
     @defer.inlineCallbacks
     def _start_key_lookups(self, verify_requests):
         """Sets off the key fetches for each verify request
@@ -289,6 +274,10 @@ class Keyring(object):
         @defer.inlineCallbacks
         def do_iterations():
             with Measure(self.clock, "get_server_verify_keys"):
+                # dict[str, dict[str, VerifyKey]]: results so far.
+                # map server_name -> key_id -> VerifyKey
+                merged_results = {}
+
                 # dict[str, set(str)]: keys to fetch for each server
                 missing_keys = {}
                 for verify_request in verify_requests:
@@ -298,29 +287,29 @@ class Keyring(object):
 
                 for fn in key_fetch_fns:
                     results = yield fn(missing_keys.items())
+                    merged_results.update(results)
 
                     # We now need to figure out which verify requests we have keys
                     # for and which we don't
                     missing_keys = {}
                     requests_missing_keys = []
                     for verify_request in verify_requests:
+                        server_name = verify_request.server_name
+                        result_keys = merged_results[server_name]
+
                         if verify_request.deferred.called:
                             # We've already called this deferred, which probably
                             # means that we've already found a key for it.
                             continue
 
-                        server_name = verify_request.server_name
-
-                        # see if any of the keys we got this time are sufficient to
-                        # complete this VerifyKeyRequest.
-                        result_keys = results.get(server_name, {})
                         for key_id in verify_request.key_ids:
-                            key = result_keys.get(key_id)
-                            if key:
+                            if key_id in result_keys:
                                 with PreserveLoggingContext():
-                                    verify_request.deferred.callback(
-                                        (server_name, key_id, key)
-                                    )
+                                    verify_request.deferred.callback((
+                                        server_name,
+                                        key_id,
+                                        result_keys[key_id],
+                                    ))
                                 break
                         else:
                             # The else block is only reached if the loop above
@@ -354,24 +343,27 @@ class Keyring(object):
     @defer.inlineCallbacks
     def get_keys_from_store(self, server_name_and_key_ids):
         """
+
         Args:
-            server_name_and_key_ids (iterable(Tuple[str, iterable[str]]):
+            server_name_and_key_ids (list[(str, iterable[str])]):
                 list of (server_name, iterable[key_id]) tuples to fetch keys for
 
         Returns:
-            Deferred: resolves to dict[str, dict[str, VerifyKey|None]]: map from
+            Deferred: resolves to dict[str, dict[str, VerifyKey]]: map from
                 server_name -> key_id -> VerifyKey
         """
-        keys_to_fetch = (
-            (server_name, key_id)
-            for server_name, key_ids in server_name_and_key_ids
-            for key_id in key_ids
-        )
-        res = yield self.store.get_server_verify_keys(keys_to_fetch)
-        keys = {}
-        for (server_name, key_id), key in res.items():
-            keys.setdefault(server_name, {})[key_id] = key
-        defer.returnValue(keys)
+        res = yield logcontext.make_deferred_yieldable(defer.gatherResults(
+            [
+                run_in_background(
+                    self.store.get_server_verify_keys,
+                    server_name, key_ids,
+                ).addCallback(lambda ks, server: (server, ks), server_name)
+                for server_name, key_ids in server_name_and_key_ids
+            ],
+            consumeErrors=True,
+        ).addErrback(unwrapFirstError))
+
+        defer.returnValue(dict(res))
 
     @defer.inlineCallbacks
     def get_keys_from_perspectives(self, server_name_and_key_ids):
@@ -502,11 +494,11 @@ class Keyring(object):
                 )
 
             processed_response = yield self.process_v2_response(
-                perspective_name, response
+                perspective_name, response, only_from_server=False
             )
-            server_name = response["server_name"]
 
-            keys.setdefault(server_name, {}).update(processed_response)
+            for server_name, response_keys in processed_response.items():
+                keys.setdefault(server_name, {}).update(response_keys)
 
         yield logcontext.make_deferred_yieldable(defer.gatherResults(
             [
@@ -525,7 +517,7 @@ class Keyring(object):
 
     @defer.inlineCallbacks
     def get_server_verify_key_v2_direct(self, server_name, key_ids):
-        keys = {}  # type: dict[str, nacl.signing.VerifyKey]
+        keys = {}
 
         for requested_key_id in key_ids:
             if requested_key_id in keys:
@@ -550,11 +542,6 @@ class Keyring(object):
                     or server_name not in response[u"signatures"]):
                 raise KeyLookupError("Key response not signed by remote server")
 
-            if response["server_name"] != server_name:
-                raise KeyLookupError("Expected a response for server %r not %r" % (
-                    server_name, response["server_name"]
-                ))
-
             response_keys = yield self.process_v2_response(
                 from_server=server_name,
                 requested_ids=[requested_key_id],
@@ -563,45 +550,24 @@ class Keyring(object):
 
             keys.update(response_keys)
 
-        yield self.store_keys(
-            server_name=server_name,
-            from_server=server_name,
-            verify_keys=keys,
-        )
-        defer.returnValue({server_name: keys})
+        yield logcontext.make_deferred_yieldable(defer.gatherResults(
+            [
+                run_in_background(
+                    self.store_keys,
+                    server_name=key_server_name,
+                    from_server=server_name,
+                    verify_keys=verify_keys,
+                )
+                for key_server_name, verify_keys in keys.items()
+            ],
+            consumeErrors=True
+        ).addErrback(unwrapFirstError))
+
+        defer.returnValue(keys)
 
     @defer.inlineCallbacks
-    def process_v2_response(
-        self, from_server, response_json, requested_ids=[],
-    ):
-        """Parse a 'Server Keys' structure from the result of a /key request
-
-        This is used to parse either the entirety of the response from
-        GET /_matrix/key/v2/server, or a single entry from the list returned by
-        POST /_matrix/key/v2/query.
-
-        Checks that each signature in the response that claims to come from the origin
-        server is valid. (Does not check that there actually is such a signature, for
-        some reason.)
-
-        Stores the json in server_keys_json so that it can be used for future responses
-        to /_matrix/key/v2/query.
-
-        Args:
-            from_server (str): the name of the server producing this result: either
-                the origin server for a /_matrix/key/v2/server request, or the notary
-                for a /_matrix/key/v2/query.
-
-            response_json (dict): the json-decoded Server Keys response object
-
-            requested_ids (iterable[str]): a list of the key IDs that were requested.
-                We will store the json for these key ids as well as any that are
-                actually in the response
-
-        Returns:
-            Deferred[dict[str, nacl.signing.VerifyKey]]:
-                map from key_id to key object
-        """
+    def process_v2_response(self, from_server, response_json,
+                            requested_ids=[], only_from_server=True):
         time_now_ms = self.clock.time_msec()
         response_keys = {}
         verify_keys = {}
@@ -623,7 +589,15 @@ class Keyring(object):
                 verify_key.time_added = time_now_ms
                 old_verify_keys[key_id] = verify_key
 
+        results = {}
         server_name = response_json["server_name"]
+        if only_from_server:
+            if server_name != from_server:
+                raise KeyLookupError(
+                    "Expected a response for server %r not %r" % (
+                        from_server, server_name
+                    )
+                )
         for key_id in response_json["signatures"].get(server_name, {}):
             if key_id not in response_json["verify_keys"]:
                 raise KeyLookupError(
@@ -659,7 +633,7 @@ class Keyring(object):
                     self.store.store_server_keys_json,
                     server_name=server_name,
                     key_id=key_id,
-                    from_server=from_server,
+                    from_server=server_name,
                     ts_now_ms=time_now_ms,
                     ts_expires_ms=ts_valid_until_ms,
                     key_json_bytes=signed_key_json_bytes,
@@ -669,7 +643,9 @@ class Keyring(object):
             consumeErrors=True,
         ).addErrback(unwrapFirstError))
 
-        defer.returnValue(response_keys)
+        results[server_name] = response_keys
+
+        defer.returnValue(results)
 
     def store_keys(self, server_name, from_server, verify_keys):
         """Store a collection of verify keys for a given server