comments

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

MANIFEST.in

@@ -38,16 +38,14 @@ exclude sytest-blacklist
 include pyproject.toml
 recursive-include changelog.d *
 
-prune .buildkite
-prune .circleci
-prune .codecov.yml
-prune .coveragerc
 prune .github
-prune debian
 prune demo/etc
 prune docker
-prune mypy.ini
-prune stubs
+prune .circleci
+prune .coveragerc
+prune debian
+prune .codecov.yml
+prune .buildkite
 
 exclude jenkins*
 recursive-exclude jenkins *.sh

UPGRADE.rst

@@ -49,56 +49,6 @@ returned by the Client-Server API:
     # configured on port 443.
     curl -kv https://<host.name>/_matrix/client/versions 2>&1 | grep "Server:"
 
-Upgrading to v1.4.0
-===================
-
-Config options
---------------
-
-**Note: Registration by email address or phone number will not work in this release unless
-some config options are changed from their defaults.**
-
-This is due to Synapse v1.4.0 now defaulting to sending registration and password reset tokens
-itself. This is for security reasons as well as putting less reliance on identity servers.
-However, currently Synapse only supports sending emails, and does not have support for
-phone-based password reset or account registration. If Synapse is configured to handle these on
-its own, phone-based password resets and registration will be disabled. For Synapse to send
-emails, the ``email`` block of the config must be filled out. If not, then password resets and
-registration via email will be disabled entirely.
-
-This release also deprecates the ``email.trust_identity_server_for_password_resets`` option and
-replaces it with the ``account_threepid_delegates`` dictionary. This option defines whether the
-homeserver should delegate an external server (typically an `identity server
-<https://matrix.org/docs/spec/identity_service/r0.2.1>`_) to handle sending password reset or
-registration messages via email and SMS.
-
-If ``email.trust_identity_server_for_password_resets`` is set to ``true``, and
-``account_threepid_delegates.email`` is not set, then the first entry in
-``trusted_third_party_id_servers`` will be used as the account threepid delegate for email.
-This is to ensure compatibility with existing Synapse installs that set up external server
-handling for these tasks before v1.4.0. If ``email.trust_identity_server_for_password_resets``
-is ``true`` and no trusted identity server domains are configured, Synapse will throw an error.
-
-If ``email.trust_identity_server_for_password_resets`` is ``false`` or absent and a threepid
-type in ``account_threepid_delegates`` is not set to a domain, then Synapse will attempt to
-send password reset and registration messages for that type.
-
-Email templates
----------------
-
-If you have configured a custom template directory with the ``email.template_dir`` option, be
-aware that there are new templates regarding registration. ``registration.html`` and
-``registration.txt`` have been added and contain the content that is sent to a client upon
-registering via an email address.
-
-``registration_success.html`` and ``registration_failure.html`` are also new HTML templates
-that will be shown to the user when they click the link in their registration emai , either
-showing them a success or failure page (assuming a redirect URL is not configured).
-
-Synapse will expect these files to exist inside the configured template directory. To view the
-default templates, see `synapse/res/templates
-<https://github.com/matrix-org/synapse/tree/master/synapse/res/templates>`_.
-
 Upgrading to v1.2.0
 ===================
 
@@ -182,19 +132,6 @@ server for password resets, set ``trust_identity_server_for_password_resets`` to
 See the `sample configuration file <docs/sample_config.yaml>`_
 for more details on these settings.
 
-New email templates
----------------
-Some new templates have been added to the default template directory for the purpose of the
-homeserver sending its own password reset emails. If you have configured a custom
-``template_dir`` in your Synapse config, these files will need to be added.
-
-``password_reset.html`` and ``password_reset.txt`` are HTML and plain text templates
-respectively that contain the contents of what will be emailed to the user upon attempting to
-reset their password via email. ``password_reset_success.html`` and
-``password_reset_failure.html`` are HTML files that the content of which (assuming no redirect
-URL is set) will be shown to the user after they attempt to click the link in the email sent
-to them.
-
 Upgrading to v0.99.0
 ====================
 

changelog.d/5835.feature

@@ -1 +0,0 @@
-Add the ability to send registration emails from the homeserver rather than delegating to an identity server.

changelog.d/5868.feature

@@ -1 +0,0 @@
-Add `m.require_identity_server` key to `/versions`'s `unstable_features` section.

changelog.d/5875.misc

@@ -1 +0,0 @@
-Deprecate the `trusted_third_party_id_servers` option.

changelog.d/5876.feature

@@ -1 +0,0 @@
-Replace `trust_identity_server_for_password_resets` config option with `account_threepid_delegates`.

changelog.d/5897.feature

@@ -1 +0,0 @@
-Switch to using the v2 Identity Service `/lookup` API where available, with fallback to v1. (Implements [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134) plus id_access_token authentication for v2 Identity Service APIs from [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140)).

changelog.d/5934.feature

@@ -1 +0,0 @@
-Redact events in the database that have been redacted for a month.

changelog.d/5940.feature

@@ -1 +0,0 @@
-Add the ability to send registration emails from the homeserver rather than delegating to an identity server.

changelog.d/5969.feature

@@ -1 +0,0 @@
-Replace `trust_identity_server_for_password_resets` config option with `account_threepid_delegates`.

changelog.d/5981.feature

@@ -1 +0,0 @@
-Setting metrics_flags.known_servers to True in the configuration will publish the synapse_federation_known_servers metric over Prometheus. This represents the total number of servers your server knows about (i.e. is in rooms with), including itself.

changelog.d/5985.feature

@@ -1 +0,0 @@
-Check at setup that opentracing is installed if it's enabled in the config.

changelog.d/5989.misc

@@ -1 +0,0 @@
-Clean up dependency checking at setup.

changelog.d/5991.bugfix

@@ -1 +0,0 @@
-Fix invalid references to None while opentracing if the log context slips.

changelog.d/5992.feature

@@ -0,0 +1 @@
+Give appropriate exit codes when synctl fails.

changelog.d/5993.feature

@@ -1 +0,0 @@
-Add the ability to send registration emails from the homeserver rather than delegating to an identity server.

changelog.d/5994.feature

@@ -1 +0,0 @@
-Add the ability to send registration emails from the homeserver rather than delegating to an identity server.

changelog.d/5995.bugfix

@@ -1 +0,0 @@
-Return a M_MISSING_PARAM if `sid` is not provided to `/account/3pid`.

changelog.d/5998.bugfix

@@ -1 +0,0 @@
-Fix room and user stats tracking.

changelog.d/6003.misc

@@ -1 +0,0 @@
-Add opentracing span over HTTP push processing.

changelog.d/6004.bugfix

@@ -1 +0,0 @@
-Only count real users when checking for auto-creation of auto-join room.

changelog.d/6005.feature

@@ -1 +0,0 @@
-The new Prometheus metric `synapse_build_info` exposes the Python version, OS version, and Synapse version of the running server.

changelog.d/6009.misc

@@ -1 +0,0 @@
-Small refactor of function arguments and docstrings in RoomMemberHandler.

changelog.d/6010.misc

@@ -1 +0,0 @@
-Remove unused `origin` argument on FederationHandler.add_display_name_to_third_party_invite.

changelog.d/6011.feature

@@ -1 +0,0 @@
-Use account_threepid_delegate.email and account_threepid_delegate.msisdn for validating threepid sessions.

changelog.d/6012.feature

@@ -1 +0,0 @@
-Add report_stats_endpoint option to configure where stats are reported to, if enabled. Contributed by @Sorunome.

changelog.d/6013.misc

@@ -1 +0,0 @@
-Compatibility with v2 Identity Service APIs other than /lookup.

changelog.d/6015.feature

@@ -1 +0,0 @@
-Add config option to increase ratelimits for room admins redacting messages.

changelog.d/6017.misc

@@ -1 +0,0 @@
-Clean up some code in the retry logic.

changelog.d/6020.bugfix

@@ -1 +0,0 @@
-Ensure support users can be registered even if MAU limit is reached.

changelog.d/6023.misc

@@ -1 +0,0 @@
-Fix the structured logging tests stomping on the global log configuration for subsequent tests.

changelog.d/6024.bugfix

@@ -1 +0,0 @@
-Fix bug where login error was shown incorrectly on SSO fallback login.

changelog.d/6025.bugfix

@@ -1 +0,0 @@
-Fix bug in calculating the federation retry backoff period.

changelog.d/6026.feature

@@ -1 +0,0 @@
-Stop sending federation transactions to servers which have been down for a long time.

changelog.d/6032.misc

@@ -1 +0,0 @@
-Add developer documentation for using SAML2.

contrib/cmdclient/console.py

@@ -37,8 +37,6 @@ from signedjson.sign import verify_signed_json, SignatureVerifyException
 
 CONFIG_JSON = "cmdclient_config.json"
 
-# TODO: The concept of trusted identity servers has been deprecated. This option and checks
-#  should be removed
 TRUSTED_ID_SERVERS = ["localhost:8001"]
 
 

docs/dev/saml.md

@@ -1,37 +0,0 @@
-# How to test SAML as a developer without a server
-
-https://capriza.github.io/samling/samling.html (https://github.com/capriza/samling) is a great
-resource for being able to tinker with the SAML options within Synapse without needing to
-deploy and configure a complicated software stack.
-
-To make Synapse (and therefore Riot) use it:
-
-1. Use the samling.html URL above or deploy your own and visit the IdP Metadata tab.
-2. Copy the XML to your clipboard.
-3. On your Synapse server, create a new file `samling.xml` next to your `homeserver.yaml` with
-   the XML from step 2 as the contents.
-4. Edit your `homeserver.yaml` to include:
-   ```yaml
-   saml2_config:
-     sp_config:
-       allow_unknown_attributes: true  # Works around a bug with AVA Hashes: https://github.com/IdentityPython/pysaml2/issues/388
-       metadata:
-         local: ["samling.xml"]   
-   ```
-5. Run `apt-get install xmlsec1` and `pip install --upgrade --force 'pysaml2>=4.5.0'` to ensure
-   the dependencies are installed and ready to go.
-6. Restart Synapse.
-
-Then in Riot:
-
-1. Visit the login page with a Riot pointing at your homeserver.
-2. Click the Single Sign-On button.
-3. On the samling page, enter a Name Identifier and add a SAML Attribute for `uid=your_localpart`.
-   The response must also be signed.
-4. Click "Next".
-5. Click "Post Response" (change nothing).
-6. You should be logged in.
-
-If you try and repeat this process, you may be automatically logged in using the information you
-gave previously. To fix this, open your developer console (`F12` or `Ctrl+Shift+I`) while on the
-samling page and clear the site data. In Chrome, this will be a button on the Application tab.

docs/sample_config.yaml

@@ -306,13 +306,6 @@ listeners:
 #
 #allow_per_room_profiles: false
 
-# How long to keep redacted events in unredacted form in the database. After
-# this period redacted events get replaced with their redacted form in the DB.
-#
-# Defaults to `7d`. Set to `null` to disable.
-#
-redaction_retention_period: 7d
-
 
 ## TLS ##
 
@@ -518,9 +511,6 @@ log_config: "CONFDIR/SERVERNAME.log.config"
 #   - one for login that ratelimits login requests based on the account the
 #     client is attempting to log into, based on the amount of failed login
 #     attempts for this account.
-#   - one for ratelimiting redactions by room admins. If this is not explicitly
-#     set then it uses the same ratelimiting as per rc_message. This is useful
-#     to allow room admins to deal with abuse quickly.
 #
 # The defaults are as shown below.
 #
@@ -542,10 +532,6 @@ log_config: "CONFDIR/SERVERNAME.log.config"
 #  failed_attempts:
 #    per_second: 0.17
 #    burst_count: 3
-#
-#rc_admin_redaction:
-#  per_second: 1
-#  burst_count: 50
 
 
 # Ratelimiting settings for incoming federation
@@ -905,42 +891,10 @@ uploads_path: "DATADIR/uploads"
 # Also defines the ID server which will be called when an account is
 # deactivated (one will be picked arbitrarily).
 #
-# Note: This option is deprecated. Since v0.99.4, Synapse has tracked which identity
-# server a 3PID has been bound to. For 3PIDs bound before then, Synapse runs a
-# background migration script, informing itself that the identity server all of its
-# 3PIDs have been bound to is likely one of the below.
-#
-# As of Synapse v1.4.0, all other functionality of this option has been deprecated, and
-# it is now solely used for the purposes of the background migration script, and can be
-# removed once it has run.
 #trusted_third_party_id_servers:
 #  - matrix.org
 #  - vector.im
 
-# Handle threepid (email/phone etc) registration and password resets through a set of
-# *trusted* identity servers. Note that this allows the configured identity server to
-# reset passwords for accounts!
-#
-# Be aware that if `email` is not set, and SMTP options have not been
-# configured in the email config block, registration and user password resets via
-# email will be globally disabled.
-#
-# Additionally, if `msisdn` is not set, registration and password resets via msisdn
-# will be disabled regardless. This is due to Synapse currently not supporting any
-# method of sending SMS messages on its own.
-#
-# To enable using an identity server for operations regarding a particular third-party
-# identifier type, set the value to the URL of that identity server as shown in the
-# examples below.
-#
-# Servers handling the these requests must answer the `/requestToken` endpoints defined
-# by the Matrix Identity Service API specification:
-# https://matrix.org/docs/spec/identity_service/latest
-#
-account_threepid_delegates:
-    #email: https://example.com     # Delegate email sending to matrix.org
-    #msisdn: http://localhost:8090  # Delegate SMS sending to this local process
-
 # Users who register on this homeserver will automatically be joined
 # to these rooms
 #
@@ -972,24 +926,9 @@ account_threepid_delegates:
 #sentry:
 #    dsn: "..."
 
-# Flags to enable Prometheus metrics which are not suitable to be
-# enabled by default, either for performance reasons or limited use.
-#
-metrics_flags:
-    # Publish synapse_federation_known_servers, a g auge of the number of
-    # servers this homeserver knows about, including itself. May cause
-    # performance problems on large homeservers.
-    #
-    #known_servers: true
-
 # Whether or not to report anonymized homeserver usage statistics.
 # report_stats: true|false
 
-# The endpoint to report the anonymized homeserver usage statistics to.
-# Defaults to https://matrix.org/report-usage-stats/push
-#
-#report_stats_endpoint: https://example.com/report-usage-stats/push
-
 
 ## API Configuration ##
 
@@ -1225,6 +1164,19 @@ password_config:
 #   #
 #   riot_base_url: "http://localhost/riot"
 #
+#   # Enable sending password reset emails via the configured, trusted
+#   # identity servers
+#   #
+#   # IMPORTANT! This will give a malicious or overtaken identity server
+#   # the ability to reset passwords for your users! Make absolutely sure
+#   # that you want to do this! It is strongly recommended that password
+#   # reset emails be sent by the homeserver instead
+#   #
+#   # If this option is set to false and SMTP options have not been
+#   # configured, resetting user passwords via email will be disabled
+#   #
+#   #trust_identity_server_for_password_resets: false
+#
 #   # Configure the time that a validation email or text message code
 #   # will expire after sending
 #   #
@@ -1256,22 +1208,11 @@ password_config:
 #   #password_reset_template_html: password_reset.html
 #   #password_reset_template_text: password_reset.txt
 #
-#   # Templates for registration emails sent by the homeserver
-#   #
-#   #registration_template_html: registration.html
-#   #registration_template_text: registration.txt
-#
 #   # Templates for password reset success and failure pages that a user
 #   # will see after attempting to reset their password
 #   #
 #   #password_reset_template_success_html: password_reset_success.html
 #   #password_reset_template_failure_html: password_reset_failure.html
-#
-#   # Templates for registration success and failure pages that a user
-#   # will see after attempting to register using an email or phone
-#   #
-#   #registration_template_success_html: registration_success.html
-#   #registration_template_failure_html: registration_failure.html
 
 
 #password_providers:

mypy.ini

@@ -1,54 +0,0 @@
-[mypy]
-namespace_packages=True
-plugins=mypy_zope:plugin
-follow_imports=skip
-mypy_path=stubs
-
-[mypy-synapse.config.homeserver]
-# this is a mess because of the metaclass shenanigans
-ignore_errors = True
-
-[mypy-zope]
-ignore_missing_imports = True
-
-[mypy-constantly]
-ignore_missing_imports = True
-
-[mypy-twisted.*]
-ignore_missing_imports = True
-
-[mypy-treq.*]
-ignore_missing_imports = True
-
-[mypy-hyperlink]
-ignore_missing_imports = True
-
-[mypy-h11]
-ignore_missing_imports = True
-
-[mypy-opentracing]
-ignore_missing_imports = True
-
-[mypy-OpenSSL]
-ignore_missing_imports = True
-
-[mypy-netaddr]
-ignore_missing_imports = True
-
-[mypy-saml2.*]
-ignore_missing_imports = True
-
-[mypy-unpaddedbase64]
-ignore_missing_imports = True
-
-[mypy-canonicaljson]
-ignore_missing_imports = True
-
-[mypy-jaeger_client]
-ignore_missing_imports = True
-
-[mypy-jsonschema]
-ignore_missing_imports = True
-
-[mypy-signedjson.*]
-ignore_missing_imports = True

synapse/api/auth.py

@@ -25,7 +25,7 @@ from twisted.internet import defer
 import synapse.logging.opentracing as opentracing
 import synapse.types
 from synapse import event_auth
-from synapse.api.constants import EventTypes, JoinRules, Membership, UserTypes
+from synapse.api.constants import EventTypes, JoinRules, Membership
 from synapse.api.errors import (
     AuthError,
     Codes,
@@ -709,7 +709,7 @@ class Auth(object):
             )
 
     @defer.inlineCallbacks
-    def check_auth_blocking(self, user_id=None, threepid=None, user_type=None):
+    def check_auth_blocking(self, user_id=None, threepid=None):
         """Checks if the user should be rejected for some external reason,
         such as monthly active user limiting or global disable flag
 
@@ -722,9 +722,6 @@ class Auth(object):
                 with a MAU blocked server, normally they would be rejected but their
                 threepid is on the reserved list. user_id and
                 threepid should never be set at the same time.
-
-            user_type(str|None): If present, is used to decide whether to check against
-                certain blocking reasons like MAU.
         """
 
         # Never fail an auth check for the server notices users or support user
@@ -762,10 +759,6 @@ class Auth(object):
                     self.hs.config.mau_limits_reserved_threepids, threepid
                 ):
                     return
-            elif user_type == UserTypes.SUPPORT:
-                # If the user does not exist yet and is of type "support",
-                # allow registration. Support users are excluded from MAU checks.
-                return
             # Else if there is no room in the MAU bucket, bail
             current_mau = yield self.store.get_monthly_active_count()
             if current_mau >= self.hs.config.max_mau_value:

synapse/app/client_reader.py

@@ -119,7 +119,7 @@ class ClientReaderServer(HomeServer):
                     KeyChangesServlet(self).register(resource)
                     VoipRestServlet(self).register(resource)
                     PushRuleRestServlet(self).register(resource)
-                    VersionsRestServlet(self).register(resource)
+                    VersionsRestServlet().register(resource)
 
                     resources.update({"/_matrix/client": resource})
 

synapse/app/homeserver.py

@@ -561,12 +561,10 @@ def run(hs):
 
         stats["database_engine"] = hs.get_datastore().database_engine_name
         stats["database_server_version"] = hs.get_datastore().get_server_version()
-        logger.info(
-            "Reporting stats to %s: %s" % (hs.config.report_stats_endpoint, stats)
-        )
+        logger.info("Reporting stats to matrix.org: %s" % (stats,))
         try:
             yield hs.get_simple_http_client().put_json(
-                hs.config.report_stats_endpoint, stats
+                "https://matrix.org/report-usage-stats/push", stats
             )
         except Exception as e:
             logger.warn("Error reporting stats: %s", e)

synapse/config/emailconfig.py

@@ -20,7 +20,6 @@ from __future__ import print_function
 # This file can't be called email.py because if it is, we cannot:
 import email.utils
 import os
-from enum import Enum
 
 import pkg_resources
 
@@ -75,48 +74,19 @@ class EmailConfig(Config):
             "renew_at"
         )
 
-        self.threepid_behaviour_email = (
-            # Have Synapse handle the email sending if account_threepid_delegates.email
-            # is not defined
-            # msisdn is currently always remote while Synapse does not support any method of
-            # sending SMS messages
-            ThreepidBehaviour.REMOTE
-            if self.account_threepid_delegate_email
-            else ThreepidBehaviour.LOCAL
+        email_trust_identity_server_for_password_resets = email_config.get(
+            "trust_identity_server_for_password_resets", False
         )
-        # Prior to Synapse v1.4.0, there was another option that defined whether Synapse would
-        # use an identity server to password reset tokens on its behalf. We now warn the user
-        # if they have this set and tell them to use the updated option, while using a default
-        # identity server in the process.
-        self.using_identity_server_from_trusted_list = False
-        if (
-            not self.account_threepid_delegate_email
-            and config.get("trust_identity_server_for_password_resets", False) is True
-        ):
-            # Use the first entry in self.trusted_third_party_id_servers instead
-            if self.trusted_third_party_id_servers:
-                # XXX: It's a little confusing that account_threepid_delegate_email is modified
-                # both in RegistrationConfig and here. We should factor this bit out
-                self.account_threepid_delegate_email = self.trusted_third_party_id_servers[
-                    0
-                ]
-                self.using_identity_server_from_trusted_list = True
-            else:
-                raise ConfigError(
-                    "Attempted to use an identity server from"
-                    '"trusted_third_party_id_servers" but it is empty.'
-                )
-
-        self.local_threepid_handling_disabled_due_to_email_config = False
-        if (
-            self.threepid_behaviour_email == ThreepidBehaviour.LOCAL
-            and email_config == {}
-        ):
+        self.email_password_reset_behaviour = (
+            "remote" if email_trust_identity_server_for_password_resets else "local"
+        )
+        self.password_resets_were_disabled_due_to_email_config = False
+        if self.email_password_reset_behaviour == "local" and email_config == {}:
             # We cannot warn the user this has happened here
             # Instead do so when a user attempts to reset their password
-            self.local_threepid_handling_disabled_due_to_email_config = True
+            self.password_resets_were_disabled_due_to_email_config = True
 
-            self.threepid_behaviour_email = ThreepidBehaviour.OFF
+            self.email_password_reset_behaviour = "off"
 
         # Get lifetime of a validation token in milliseconds
         self.email_validation_token_lifetime = self.parse_duration(
@@ -126,7 +96,7 @@ class EmailConfig(Config):
         if (
             self.email_enable_notifs
             or account_validity_renewal_enabled
-            or self.threepid_behaviour_email == ThreepidBehaviour.LOCAL
+            or self.email_password_reset_behaviour == "local"
         ):
             # make sure we can import the required deps
             import jinja2
@@ -136,7 +106,7 @@ class EmailConfig(Config):
             jinja2
             bleach
 
-        if self.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
+        if self.email_password_reset_behaviour == "local":
             required = ["smtp_host", "smtp_port", "notif_from"]
 
             missing = []
@@ -155,45 +125,28 @@ class EmailConfig(Config):
                     % (", ".join(missing),)
                 )
 
-            # These email templates have placeholders in them, and thus must be
-            # parsed using a templating engine during a request
+            # Templates for password reset emails
             self.email_password_reset_template_html = email_config.get(
                 "password_reset_template_html", "password_reset.html"
             )
             self.email_password_reset_template_text = email_config.get(
                 "password_reset_template_text", "password_reset.txt"
             )
-            self.email_registration_template_html = email_config.get(
-                "registration_template_html", "registration.html"
-            )
-            self.email_registration_template_text = email_config.get(
-                "registration_template_text", "registration.txt"
-            )
             self.email_password_reset_template_failure_html = email_config.get(
                 "password_reset_template_failure_html", "password_reset_failure.html"
             )
-            self.email_registration_template_failure_html = email_config.get(
-                "registration_template_failure_html", "registration_failure.html"
-            )
-
-            # These templates do not support any placeholder variables, so we
-            # will read them from disk once during setup
+            # This template does not support any replaceable variables, so we will
+            # read it from the disk once during setup
             email_password_reset_template_success_html = email_config.get(
                 "password_reset_template_success_html", "password_reset_success.html"
             )
-            email_registration_template_success_html = email_config.get(
-                "registration_template_success_html", "registration_success.html"
-            )
 
             # Check templates exist
             for f in [
                 self.email_password_reset_template_html,
                 self.email_password_reset_template_text,
-                self.email_registration_template_html,
-                self.email_registration_template_text,
                 self.email_password_reset_template_failure_html,
                 email_password_reset_template_success_html,
-                email_registration_template_success_html,
             ]:
                 p = os.path.join(self.email_template_dir, f)
                 if not os.path.isfile(p):
@@ -203,15 +156,9 @@ class EmailConfig(Config):
             filepath = os.path.join(
                 self.email_template_dir, email_password_reset_template_success_html
             )
-            self.email_password_reset_template_success_html = self.read_file(
+            self.email_password_reset_template_success_html_content = self.read_file(
                 filepath, "email.password_reset_template_success_html"
             )
-            filepath = os.path.join(
-                self.email_template_dir, email_registration_template_success_html
-            )
-            self.email_registration_template_success_html_content = self.read_file(
-                filepath, "email.registration_template_success_html"
-            )
 
         if self.email_enable_notifs:
             required = [
@@ -292,6 +239,19 @@ class EmailConfig(Config):
         #   #
         #   riot_base_url: "http://localhost/riot"
         #
+        #   # Enable sending password reset emails via the configured, trusted
+        #   # identity servers
+        #   #
+        #   # IMPORTANT! This will give a malicious or overtaken identity server
+        #   # the ability to reset passwords for your users! Make absolutely sure
+        #   # that you want to do this! It is strongly recommended that password
+        #   # reset emails be sent by the homeserver instead
+        #   #
+        #   # If this option is set to false and SMTP options have not been
+        #   # configured, resetting user passwords via email will be disabled
+        #   #
+        #   #trust_identity_server_for_password_resets: false
+        #
         #   # Configure the time that a validation email or text message code
         #   # will expire after sending
         #   #
@@ -323,35 +283,9 @@ class EmailConfig(Config):
         #   #password_reset_template_html: password_reset.html
         #   #password_reset_template_text: password_reset.txt
         #
-        #   # Templates for registration emails sent by the homeserver
-        #   #
-        #   #registration_template_html: registration.html
-        #   #registration_template_text: registration.txt
-        #
         #   # Templates for password reset success and failure pages that a user
         #   # will see after attempting to reset their password
         #   #
         #   #password_reset_template_success_html: password_reset_success.html
         #   #password_reset_template_failure_html: password_reset_failure.html
-        #
-        #   # Templates for registration success and failure pages that a user
-        #   # will see after attempting to register using an email or phone
-        #   #
-        #   #registration_template_success_html: registration_success.html
-        #   #registration_template_failure_html: registration_failure.html
         """
-
-
-class ThreepidBehaviour(Enum):
-    """
-    Enum to define the behaviour of Synapse with regards to when it contacts an identity
-    server for 3pid registration and password resets
-
-    REMOTE = use an external server to send tokens
-    LOCAL = send tokens ourselves
-    OFF = disable registration via 3pid and password resets
-    """
-
-    REMOTE = "remote"
-    LOCAL = "local"
-    OFF = "off"

synapse/config/logger.py

@@ -21,12 +21,7 @@ from string import Template
 
 import yaml
 
-from twisted.logger import (
-    ILogObserver,
-    LogBeginner,
-    STDLibLogObserver,
-    globalLogBeginner,
-)
+from twisted.logger import STDLibLogObserver, globalLogBeginner
 
 import synapse
 from synapse.app import _base as appbase
@@ -129,7 +124,7 @@ class LoggingConfig(Config):
                 log_config_file.write(DEFAULT_LOG_CONFIG.substitute(log_file=log_file))
 
 
-def _setup_stdlib_logging(config, log_config, logBeginner: LogBeginner):
+def _setup_stdlib_logging(config, log_config):
     """
     Set up Python stdlib logging.
     """
@@ -170,12 +165,12 @@ def _setup_stdlib_logging(config, log_config, logBeginner: LogBeginner):
 
         return observer(event)
 
-    logBeginner.beginLoggingTo([_log], redirectStandardIO=not config.no_redirect_stdio)
+    globalLogBeginner.beginLoggingTo(
+        [_log], redirectStandardIO=not config.no_redirect_stdio
+    )
     if not config.no_redirect_stdio:
         print("Redirected stdout/stderr to logs")
 
-    return observer
-
 
 def _reload_stdlib_logging(*args, log_config=None):
     logger = logging.getLogger("")
@@ -186,9 +181,7 @@ def _reload_stdlib_logging(*args, log_config=None):
     logging.config.dictConfig(log_config)
 
 
-def setup_logging(
-    hs, config, use_worker_options=False, logBeginner: LogBeginner = globalLogBeginner
-) -> ILogObserver:
+def setup_logging(hs, config, use_worker_options=False):
     """
     Set up the logging subsystem.
 
@@ -198,12 +191,6 @@ def setup_logging(
 
         use_worker_options (bool): True to use the 'worker_log_config' option
             instead of 'log_config'.
-
-        logBeginner: The Twisted logBeginner to use.
-
-    Returns:
-        The "root" Twisted Logger observer, suitable for sending logs to from a
-        Logger instance.
     """
     log_config = config.worker_log_config if use_worker_options else config.log_config
 
@@ -223,12 +210,10 @@ def setup_logging(
     log_config_body = read_config()
 
     if log_config_body and log_config_body.get("structured") is True:
-        logger = setup_structured_logging(
-            hs, config, log_config_body, logBeginner=logBeginner
-        )
+        setup_structured_logging(hs, config, log_config_body)
         appbase.register_sighup(read_config, callback=reload_structured_logging)
     else:
-        logger = _setup_stdlib_logging(config, log_config_body, logBeginner=logBeginner)
+        _setup_stdlib_logging(config, log_config_body)
         appbase.register_sighup(read_config, callback=_reload_stdlib_logging)
 
     # make sure that the first thing we log is a thing we can grep backwards
@@ -236,5 +221,3 @@ def setup_logging(
     logging.warn("***** STARTING SERVER *****")
     logging.warn("Server %s version %s", sys.argv[0], get_version_string(synapse))
     logging.info("Server hostname: %s", config.server_name)
-
-    return logger

synapse/config/metrics.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2015, 2016 OpenMarket Ltd
-# Copyright 2019 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,47 +13,26 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import attr
-
-from synapse.python_dependencies import DependencyException, check_requirements
-
 from ._base import Config, ConfigError
 
-
-@attr.s
-class MetricsFlags(object):
-    known_servers = attr.ib(default=False, validator=attr.validators.instance_of(bool))
-
-    @classmethod
-    def all_off(cls):
-        """
-        Instantiate the flags with all options set to off.
-        """
-        return cls(**{x.name: False for x in attr.fields(cls)})
+MISSING_SENTRY = """Missing sentry-sdk library. This is required to enable sentry
+    integration.
+    """
 
 
 class MetricsConfig(Config):
     def read_config(self, config, **kwargs):
         self.enable_metrics = config.get("enable_metrics", False)
         self.report_stats = config.get("report_stats", None)
-        self.report_stats_endpoint = config.get(
-            "report_stats_endpoint", "https://matrix.org/report-usage-stats/push"
-        )
         self.metrics_port = config.get("metrics_port")
         self.metrics_bind_host = config.get("metrics_bind_host", "127.0.0.1")
 
-        if self.enable_metrics:
-            _metrics_config = config.get("metrics_flags") or {}
-            self.metrics_flags = MetricsFlags(**_metrics_config)
-        else:
-            self.metrics_flags = MetricsFlags.all_off()
-
         self.sentry_enabled = "sentry" in config
         if self.sentry_enabled:
             try:
-                check_requirements("sentry")
-            except DependencyException as e:
-                raise ConfigError(e.message)
+                import sentry_sdk  # noqa F401
+            except ImportError:
+                raise ConfigError(MISSING_SENTRY)
 
             self.sentry_dsn = config["sentry"].get("dsn")
             if not self.sentry_dsn:
@@ -80,16 +58,6 @@ class MetricsConfig(Config):
         #sentry:
         #    dsn: "..."
 
-        # Flags to enable Prometheus metrics which are not suitable to be
-        # enabled by default, either for performance reasons or limited use.
-        #
-        metrics_flags:
-            # Publish synapse_federation_known_servers, a g auge of the number of
-            # servers this homeserver knows about, including itself. May cause
-            # performance problems on large homeservers.
-            #
-            #known_servers: true
-
         # Whether or not to report anonymized homeserver usage statistics.
         """
 
@@ -98,10 +66,4 @@ class MetricsConfig(Config):
         else:
             res += "report_stats: %s\n" % ("true" if report_stats else "false")
 
-        res += """
-        # The endpoint to report the anonymized homeserver usage statistics to.
-        # Defaults to https://matrix.org/report-usage-stats/push
-        #
-        #report_stats_endpoint: https://example.com/report-usage-stats/push
-        """
         return res

synapse/config/ratelimiting.py

@@ -80,12 +80,6 @@ class RatelimitConfig(Config):
             "federation_rr_transactions_per_room_per_second", 50
         )
 
-        rc_admin_redaction = config.get("rc_admin_redaction")
-        if rc_admin_redaction:
-            self.rc_admin_redaction = RateLimitConfig(rc_admin_redaction)
-        else:
-            self.rc_admin_redaction = None
-
     def generate_config_section(self, **kwargs):
         return """\
         ## Ratelimiting ##
@@ -108,9 +102,6 @@ class RatelimitConfig(Config):
         #   - one for login that ratelimits login requests based on the account the
         #     client is attempting to log into, based on the amount of failed login
         #     attempts for this account.
-        #   - one for ratelimiting redactions by room admins. If this is not explicitly
-        #     set then it uses the same ratelimiting as per rc_message. This is useful
-        #     to allow room admins to deal with abuse quickly.
         #
         # The defaults are as shown below.
         #
@@ -132,10 +123,6 @@ class RatelimitConfig(Config):
         #  failed_attempts:
         #    per_second: 0.17
         #    burst_count: 3
-        #
-        #rc_admin_redaction:
-        #  per_second: 1
-        #  burst_count: 50
 
 
         # Ratelimiting settings for incoming federation

synapse/config/registration.py

@@ -99,10 +99,6 @@ class RegistrationConfig(Config):
         self.trusted_third_party_id_servers = config.get(
             "trusted_third_party_id_servers", ["matrix.org", "vector.im"]
         )
-        account_threepid_delegates = config.get("account_threepid_delegates") or {}
-        self.account_threepid_delegate_email = account_threepid_delegates.get("email")
-        self.account_threepid_delegate_msisdn = account_threepid_delegates.get("msisdn")
-
         self.default_identity_server = config.get("default_identity_server")
         self.allow_guest_access = config.get("allow_guest_access", False)
 
@@ -261,42 +257,10 @@ class RegistrationConfig(Config):
         # Also defines the ID server which will be called when an account is
         # deactivated (one will be picked arbitrarily).
         #
-        # Note: This option is deprecated. Since v0.99.4, Synapse has tracked which identity
-        # server a 3PID has been bound to. For 3PIDs bound before then, Synapse runs a
-        # background migration script, informing itself that the identity server all of its
-        # 3PIDs have been bound to is likely one of the below.
-        #
-        # As of Synapse v1.4.0, all other functionality of this option has been deprecated, and
-        # it is now solely used for the purposes of the background migration script, and can be
-        # removed once it has run.
         #trusted_third_party_id_servers:
         #  - matrix.org
         #  - vector.im
 
-        # Handle threepid (email/phone etc) registration and password resets through a set of
-        # *trusted* identity servers. Note that this allows the configured identity server to
-        # reset passwords for accounts!
-        #
-        # Be aware that if `email` is not set, and SMTP options have not been
-        # configured in the email config block, registration and user password resets via
-        # email will be globally disabled.
-        #
-        # Additionally, if `msisdn` is not set, registration and password resets via msisdn
-        # will be disabled regardless. This is due to Synapse currently not supporting any
-        # method of sending SMS messages on its own.
-        #
-        # To enable using an identity server for operations regarding a particular third-party
-        # identifier type, set the value to the URL of that identity server as shown in the
-        # examples below.
-        #
-        # Servers handling the these requests must answer the `/requestToken` endpoints defined
-        # by the Matrix Identity Service API specification:
-        # https://matrix.org/docs/spec/identity_service/latest
-        #
-        account_threepid_delegates:
-            #email: https://example.com     # Delegate email sending to matrix.org
-            #msisdn: http://localhost:8090  # Delegate SMS sending to this local process
-
         # Users who register on this homeserver will automatically be joined
         # to these rooms
         #

synapse/config/repository.py

@@ -16,7 +16,6 @@
 import os
 from collections import namedtuple
 
-from synapse.python_dependencies import DependencyException, check_requirements
 from synapse.util.module_loader import load_module
 
 from ._base import Config, ConfigError
@@ -35,6 +34,17 @@ THUMBNAIL_SIZE_YAML = """\
         #    method: %(method)s
 """
 
+MISSING_NETADDR = "Missing netaddr library. This is required for URL preview API."
+
+MISSING_LXML = """Missing lxml library. This is required for URL preview API.
+
+    Install by running:
+        pip install lxml
+
+    Requires libxslt1-dev system package.
+    """
+
+
 ThumbnailRequirement = namedtuple(
     "ThumbnailRequirement", ["width", "height", "method", "media_type"]
 )
@@ -161,10 +171,16 @@ class ContentRepositoryConfig(Config):
         self.url_preview_enabled = config.get("url_preview_enabled", False)
         if self.url_preview_enabled:
             try:
-                check_requirements("url_preview")
+                import lxml
 
-            except DependencyException as e:
-                raise ConfigError(e.message)
+                lxml  # To stop unused lint.
+            except ImportError:
+                raise ConfigError(MISSING_LXML)
+
+            try:
+                from netaddr import IPSet
+            except ImportError:
+                raise ConfigError(MISSING_NETADDR)
 
             if "url_preview_ip_range_blacklist" not in config:
                 raise ConfigError(
@@ -173,9 +189,6 @@ class ContentRepositoryConfig(Config):
                     "to work"
                 )
 
-            # netaddr is a dependency for url_preview
-            from netaddr import IPSet
-
             self.url_preview_ip_range_blacklist = IPSet(
                 config["url_preview_ip_range_blacklist"]
             )

synapse/config/server.py

@@ -162,16 +162,6 @@ class ServerConfig(Config):
 
         self.mau_trial_days = config.get("mau_trial_days", 0)
 
-        # How long to keep redacted events in the database in unredacted form
-        # before redacting them.
-        redaction_retention_period = config.get("redaction_retention_period", "7d")
-        if redaction_retention_period is not None:
-            self.redaction_retention_period = self.parse_duration(
-                redaction_retention_period
-            )
-        else:
-            self.redaction_retention_period = None
-
         # Options to disable HS
         self.hs_disabled = config.get("hs_disabled", False)
         self.hs_disabled_message = config.get("hs_disabled_message", "")
@@ -728,13 +718,6 @@ class ServerConfig(Config):
         # Defaults to 'true'.
         #
         #allow_per_room_profiles: false
-
-        # How long to keep redacted events in unredacted form in the database. After
-        # this period redacted events get replaced with their redacted form in the DB.
-        #
-        # Defaults to `7d`. Set to `null` to disable.
-        #
-        redaction_retention_period: 7d
         """
             % locals()
         )

synapse/config/tracer.py

@@ -13,8 +13,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from synapse.python_dependencies import DependencyException, check_requirements
-
 from ._base import Config, ConfigError
 
 
@@ -34,11 +32,6 @@ class TracerConfig(Config):
         if not self.opentracer_enabled:
             return
 
-        try:
-            check_requirements("opentracing")
-        except DependencyException as e:
-            raise ConfigError(e.message)
-
         # The tracer is enabled so sanitize the config
 
         self.opentracer_whitelist = opentracing_config.get("homeserver_whitelist", [])

synapse/federation/federation_server.py

@@ -669,9 +669,9 @@ class FederationServer(FederationBase):
         return ret
 
     @defer.inlineCallbacks
-    def on_exchange_third_party_invite_request(self, room_id, event_dict):
+    def on_exchange_third_party_invite_request(self, origin, room_id, event_dict):
         ret = yield self.handler.on_exchange_third_party_invite_request(
-            room_id, event_dict
+            origin, room_id, event_dict
         )
         return ret
 

synapse/federation/transport/server.py

@@ -575,7 +575,7 @@ class FederationThirdPartyInviteExchangeServlet(BaseFederationServlet):
 
     async def on_PUT(self, origin, content, query, room_id):
         content = await self.handler.on_exchange_third_party_invite_request(
-            room_id, content
+            origin, room_id, content
         )
         return 200, content
 

synapse/handlers/_base.py

@@ -45,7 +45,6 @@ class BaseHandler(object):
         self.state_handler = hs.get_state_handler()
         self.distributor = hs.get_distributor()
         self.ratelimiter = hs.get_ratelimiter()
-        self.admin_redaction_ratelimiter = hs.get_admin_redaction_ratelimiter()
         self.clock = hs.get_clock()
         self.hs = hs
 
@@ -54,7 +53,7 @@ class BaseHandler(object):
         self.event_builder_factory = hs.get_event_builder_factory()
 
     @defer.inlineCallbacks
-    def ratelimit(self, requester, update=True, is_admin_redaction=False):
+    def ratelimit(self, requester, update=True):
         """Ratelimits requests.
 
         Args:
@@ -63,9 +62,6 @@ class BaseHandler(object):
                 Set to False when doing multiple checks for one request (e.g.
                 to check up front if we would reject the request), and set to
                 True for the last call for a given request.
-            is_admin_redaction (bool): Whether this is a room admin/moderator
-                redacting an event. If so then we may apply different
-                ratelimits depending on config.
 
         Raises:
             LimitExceededError if the request should be ratelimited
@@ -94,33 +90,16 @@ class BaseHandler(object):
             messages_per_second = override.messages_per_second
             burst_count = override.burst_count
         else:
-            # We default to different values if this is an admin redaction and
-            # the config is set
-            if is_admin_redaction and self.hs.config.rc_admin_redaction:
-                messages_per_second = self.hs.config.rc_admin_redaction.per_second
-                burst_count = self.hs.config.rc_admin_redaction.burst_count
-            else:
-                messages_per_second = self.hs.config.rc_message.per_second
-                burst_count = self.hs.config.rc_message.burst_count
+            messages_per_second = self.hs.config.rc_message.per_second
+            burst_count = self.hs.config.rc_message.burst_count
 
-        if is_admin_redaction and self.hs.config.rc_admin_redaction:
-            # If we have separate config for admin redactions we use a separate
-            # ratelimiter
-            allowed, time_allowed = self.admin_redaction_ratelimiter.can_do_action(
-                user_id,
-                time_now,
-                rate_hz=messages_per_second,
-                burst_count=burst_count,
-                update=update,
-            )
-        else:
-            allowed, time_allowed = self.ratelimiter.can_do_action(
-                user_id,
-                time_now,
-                rate_hz=messages_per_second,
-                burst_count=burst_count,
-                update=update,
-            )
+        allowed, time_allowed = self.ratelimiter.can_do_action(
+            user_id,
+            time_now,
+            rate_hz=messages_per_second,
+            burst_count=burst_count,
+            update=update,
+        )
         if not allowed:
             raise LimitExceededError(
                 retry_after_ms=int(1000 * (time_allowed - time_now))

synapse/handlers/account_validity.py

@@ -38,7 +38,6 @@ logger = logging.getLogger(__name__)
 class AccountValidityHandler(object):
     def __init__(self, hs):
         self.hs = hs
-        self.config = hs.config
         self.store = self.hs.get_datastore()
         self.sendmail = self.hs.get_sendmail()
         self.clock = self.hs.get_clock()
@@ -63,14 +62,9 @@ class AccountValidityHandler(object):
             self._raw_from = email.utils.parseaddr(self._from_string)[1]
 
             self._template_html, self._template_text = load_jinja2_templates(
-                self.config.email_template_dir,
-                [
-                    self.config.email_expiry_template_html,
-                    self.config.email_expiry_template_text,
-                ],
-                apply_format_ts_filter=True,
-                apply_mxc_to_http_filter=True,
-                public_baseurl=self.config.public_baseurl,
+                config=self.hs.config,
+                template_html_name=self.hs.config.email_expiry_template_html,
+                template_text_name=self.hs.config.email_expiry_template_text,
             )
 
             # Check the renewal emails to send and send them every 30min.

synapse/handlers/auth.py

@@ -38,7 +38,6 @@ from synapse.api.errors import (
     UserDeactivatedError,
 )
 from synapse.api.ratelimiting import Ratelimiter
-from synapse.config.emailconfig import ThreepidBehaviour
 from synapse.logging.context import defer_to_thread
 from synapse.module_api import ModuleApi
 from synapse.types import UserID
@@ -159,7 +158,7 @@ class AuthHandler(BaseHandler):
         return params
 
     @defer.inlineCallbacks
-    def check_auth(self, flows, clientdict, clientip):
+    def check_auth(self, flows, clientdict, clientip, password_servlet=False):
         """
         Takes a dictionary sent by the client in the login / registration
         protocol and handles the User-Interactive Auth flow.
@@ -183,6 +182,16 @@ class AuthHandler(BaseHandler):
 
             clientip (str): The IP address of the client.
 
+            password_servlet (bool): Whether the request originated from
+                PasswordRestServlet.
+                XXX: This is a temporary hack to distinguish between checking
+                for threepid validations locally (in the case of password
+                resets) and using the identity server (in the case of binding
+                a 3PID during registration). Once we start using the
+                homeserver for both tasks, this distinction will no longer be
+                necessary.
+
+
         Returns:
             defer.Deferred[dict, dict, str]: a deferred tuple of
                 (creds, params, session_id).
@@ -238,7 +247,9 @@ class AuthHandler(BaseHandler):
         if "type" in authdict:
             login_type = authdict["type"]
             try:
-                result = yield self._check_auth_dict(authdict, clientip)
+                result = yield self._check_auth_dict(
+                    authdict, clientip, password_servlet=password_servlet
+                )
                 if result:
                     creds[login_type] = result
                     self._save_session(session)
@@ -345,7 +356,7 @@ class AuthHandler(BaseHandler):
         return sess.setdefault("serverdict", {}).get(key, default)
 
     @defer.inlineCallbacks
-    def _check_auth_dict(self, authdict, clientip):
+    def _check_auth_dict(self, authdict, clientip, password_servlet=False):
         """Attempt to validate the auth dict provided by a client
 
         Args:
@@ -363,7 +374,11 @@ class AuthHandler(BaseHandler):
         login_type = authdict["type"]
         checker = self.checkers.get(login_type)
         if checker is not None:
-            res = yield checker(authdict, clientip=clientip)
+            # XXX: Temporary workaround for having Synapse handle password resets
+            # See AuthHandler.check_auth for further details
+            res = yield checker(
+                authdict, clientip=clientip, password_servlet=password_servlet
+            )
             return res
 
         # build a v1-login-style dict out of the authdict and fall back to the
@@ -434,7 +449,7 @@ class AuthHandler(BaseHandler):
         return defer.succeed(True)
 
     @defer.inlineCallbacks
-    def _check_threepid(self, medium, authdict, **kwargs):
+    def _check_threepid(self, medium, authdict, password_servlet=False, **kwargs):
         if "threepid_creds" not in authdict:
             raise LoginError(400, "Missing threepid_creds", Codes.MISSING_PARAM)
 
@@ -443,18 +458,12 @@ class AuthHandler(BaseHandler):
         identity_handler = self.hs.get_handlers().identity_handler
 
         logger.info("Getting validated threepid. threepidcreds: %r", (threepid_creds,))
-        if self.hs.config.threepid_behaviour_email == ThreepidBehaviour.REMOTE:
-            if medium == "email":
-                threepid = yield identity_handler.threepid_from_creds(
-                    self.hs.config.account_threepid_delegate_email, threepid_creds
-                )
-            elif medium == "msisdn":
-                threepid = yield identity_handler.threepid_from_creds(
-                    self.hs.config.account_threepid_delegate_msisdn, threepid_creds
-                )
-            else:
-                raise SynapseError(400, "Unrecognized threepid medium: %s" % (medium,))
-        elif self.hs.config.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
+        if (
+            not password_servlet
+            or self.hs.config.email_password_reset_behaviour == "remote"
+        ):
+            threepid = yield identity_handler.threepid_from_creds(threepid_creds)
+        elif self.hs.config.email_password_reset_behaviour == "local":
             row = yield self.store.get_threepid_validation_session(
                 medium,
                 threepid_creds["client_secret"],

synapse/handlers/federation.py

@@ -2530,17 +2530,12 @@ class FederationHandler(BaseHandler):
 
     @defer.inlineCallbacks
     @log_function
-    def on_exchange_third_party_invite_request(self, room_id, event_dict):
+    def on_exchange_third_party_invite_request(self, origin, room_id, event_dict):
         """Handle an exchange_third_party_invite request from a remote server
 
         The remote server will call this when it wants to turn a 3pid invite
         into a normal m.room.member invite.
 
-        Args:
-            room_id (str): The ID of the room.
-
-            event_dict (dict[str, Any]): Dictionary containing the event body.
-
         Returns:
             Deferred: resolves (to None)
         """

synapse/handlers/identity.py

@@ -29,7 +29,6 @@ from synapse.api.errors import (
     HttpResponseException,
     SynapseError,
 )
-from synapse.util.stringutils import random_string
 
 from ._base import BaseHandler
 
@@ -42,7 +41,25 @@ class IdentityHandler(BaseHandler):
 
         self.http_client = hs.get_simple_http_client()
         self.federation_http_client = hs.get_http_client()
-        self.hs = hs
+
+        self.trusted_id_servers = set(hs.config.trusted_third_party_id_servers)
+        self.trust_any_id_server_just_for_testing_do_not_use = (
+            hs.config.use_insecure_ssl_client_just_for_testing_do_not_use
+        )
+
+    def _should_trust_id_server(self, id_server):
+        if id_server not in self.trusted_id_servers:
+            if self.trust_any_id_server_just_for_testing_do_not_use:
+                logger.warn(
+                    "Trusting untrustworthy ID server %r even though it isn't"
+                    " in the trusted id list for testing because"
+                    " 'use_insecure_ssl_client_just_for_testing_do_not_use'"
+                    " is set in the config",
+                    id_server,
+                )
+            else:
+                return False
+        return True
 
     def _extract_items_from_creds_dict(self, creds):
         """
@@ -75,52 +92,66 @@ class IdentityHandler(BaseHandler):
         return client_secret, id_server, id_access_token
 
     @defer.inlineCallbacks
-    def threepid_from_creds(self, id_server, creds):
+    def threepid_from_creds(self, creds, use_v2=True):
         """
-        Retrieve and validate a threepid identifier from a "credentials" dictionary against a
-        given identity server
+        Retrieve and validate a threepid identitier from a "credentials" dictionary
 
         Args:
-            id_server (str|None): The identity server to validate 3PIDs against. If None,
-                we will attempt to extract id_server creds
-
-            creds (dict[str, str]): Dictionary containing the following keys:
-                * id_server|idServer: An optional domain name of an identity server
+            creds (dict[str, str]): Dictionary of credentials that contain the following keys:
                 * client_secret|clientSecret: A unique secret str provided by the client
-                * sid: The ID of the validation session
+                * id_server|idServer: the domain of the identity server to query
+                * id_access_token: The access token to authenticate to the identity
+                    server with. Required if use_v2 is true
+            use_v2 (bool): Whether to use v2 Identity Service API endpoints
 
         Returns:
             Deferred[dict[str,str|int]|None]: A dictionary consisting of response params to
                 the /getValidated3pid endpoint of the Identity Service API, or None if the
                 threepid was not found
         """
-        client_secret = creds.get("client_secret") or creds.get("clientSecret")
-        if not client_secret:
-            raise SynapseError(
-                400, "Missing param client_secret in creds", errcode=Codes.MISSING_PARAM
-            )
-        session_id = creds.get("sid")
-        if not session_id:
-            raise SynapseError(
-                400, "Missing param session_id in creds", errcode=Codes.MISSING_PARAM
-            )
-        if not id_server:
-            # Attempt to get the id_server from the creds dict
-            id_server = creds.get("id_server") or creds.get("idServer")
-            if not id_server:
-                raise SynapseError(
-                    400, "Missing param id_server in creds", errcode=Codes.MISSING_PARAM
-                )
-
-        query_params = {"sid": session_id, "client_secret": client_secret}
-
-        url = "https://%s%s" % (
-            id_server,
-            "/_matrix/identity/api/v1/3pid/getValidated3pid",
+        client_secret, id_server, id_access_token = self._extract_items_from_creds_dict(
+            creds
         )
 
-        data = yield self.http_client.get_json(url, query_params)
-        return data if "medium" in data else None
+        # If an id_access_token is not supplied, force usage of v1
+        if id_access_token is None:
+            use_v2 = False
+
+        query_params = {"sid": creds["sid"], "client_secret": client_secret}
+
+        # Decide which API endpoint URLs and query parameters to use
+        if use_v2:
+            url = "https://%s%s" % (
+                id_server,
+                "/_matrix/identity/v2/3pid/getValidated3pid",
+            )
+            query_params["id_access_token"] = id_access_token
+        else:
+            url = "https://%s%s" % (
+                id_server,
+                "/_matrix/identity/api/v1/3pid/getValidated3pid",
+            )
+
+        if not self._should_trust_id_server(id_server):
+            logger.warn(
+                "%s is not a trusted ID server: rejecting 3pid " + "credentials",
+                id_server,
+            )
+            return None
+
+        try:
+            data = yield self.http_client.get_json(url, query_params)
+            return data if "medium" in data else None
+        except HttpResponseException as e:
+            if e.code != 404 or not use_v2:
+                # Generic failure
+                logger.info("getValidated3pid failed with Matrix error: %r", e)
+                raise e.to_synapse_error()
+
+        # This identity server is too old to understand Identity Service API v2
+        # Attempt v1 endpoint
+        logger.info("Got 404 when POSTing JSON %s, falling back to v1 URL", url)
+        return (yield self.threepid_from_creds(creds, use_v2=False))
 
     @defer.inlineCallbacks
     def bind_threepid(self, creds, mxid, use_v2=True):
@@ -144,29 +175,20 @@ class IdentityHandler(BaseHandler):
             creds
         )
 
-        sid = creds.get("sid")
-        if not sid:
-            raise SynapseError(
-                400, "No sid in three_pid_creds", errcode=Codes.MISSING_PARAM
-            )
-
         # If an id_access_token is not supplied, force usage of v1
         if id_access_token is None:
             use_v2 = False
 
         # Decide which API endpoint URLs to use
-        headers = {}
-        bind_data = {"sid": sid, "client_secret": client_secret, "mxid": mxid}
+        bind_data = {"sid": creds["sid"], "client_secret": client_secret, "mxid": mxid}
         if use_v2:
             bind_url = "https://%s/_matrix/identity/v2/3pid/bind" % (id_server,)
-            headers["Authorization"] = create_id_access_token_header(id_access_token)
+            bind_data["id_access_token"] = id_access_token
         else:
             bind_url = "https://%s/_matrix/identity/api/v1/3pid/bind" % (id_server,)
 
         try:
-            data = yield self.http_client.post_json_get_json(
-                bind_url, bind_data, headers=headers
-            )
+            data = yield self.http_client.post_json_get_json(bind_url, bind_data)
             logger.debug("bound threepid %r to %s", creds, mxid)
 
             # Remember where we bound the threepid
@@ -283,122 +305,28 @@ class IdentityHandler(BaseHandler):
 
         return changed
 
-    @defer.inlineCallbacks
-    def send_threepid_validation(
-        self,
-        email_address,
-        client_secret,
-        send_attempt,
-        send_email_func,
-        next_link=None,
-    ):
-        """Send a threepid validation email for password reset or
-        registration purposes
-
-        Args:
-            email_address (str): The user's email address
-            client_secret (str): The provided client secret
-            send_attempt (int): Which send attempt this is
-            send_email_func (func): A function that takes an email address, token,
-                                    client_secret and session_id, sends an email
-                                    and returns a Deferred.
-            next_link (str|None): The URL to redirect the user to after validation
-
-        Returns:
-            The new session_id upon success
-
-        Raises:
-            SynapseError is an error occurred when sending the email
-        """
-        # Check that this email/client_secret/send_attempt combo is new or
-        # greater than what we've seen previously
-        session = yield self.store.get_threepid_validation_session(
-            "email", client_secret, address=email_address, validated=False
-        )
-
-        # Check to see if a session already exists and that it is not yet
-        # marked as validated
-        if session and session.get("validated_at") is None:
-            session_id = session["session_id"]
-            last_send_attempt = session["last_send_attempt"]
-
-            # Check that the send_attempt is higher than previous attempts
-            if send_attempt <= last_send_attempt:
-                # If not, just return a success without sending an email
-                return session_id
-        else:
-            # An non-validated session does not exist yet.
-            # Generate a session id
-            session_id = random_string(16)
-
-        # Generate a new validation token
-        token = random_string(32)
-
-        # Send the mail with the link containing the token, client_secret
-        # and session_id
-        try:
-            yield send_email_func(email_address, token, client_secret, session_id)
-        except Exception:
-            logger.exception(
-                "Error sending threepid validation email to %s", email_address
-            )
-            raise SynapseError(500, "An error was encountered when sending the email")
-
-        token_expires = (
-            self.hs.clock.time_msec() + self.hs.config.email_validation_token_lifetime
-        )
-
-        yield self.store.start_or_continue_validation_session(
-            "email",
-            email_address,
-            session_id,
-            client_secret,
-            send_attempt,
-            next_link,
-            token,
-            token_expires,
-        )
-
-        return session_id
-
     @defer.inlineCallbacks
     def requestEmailToken(
         self, id_server, email, client_secret, send_attempt, next_link=None
     ):
-        """
-        Request an external server send an email on our behalf for the purposes of threepid
-        validation.
+        if not self._should_trust_id_server(id_server):
+            raise SynapseError(
+                400, "Untrusted ID server '%s'" % id_server, Codes.SERVER_NOT_TRUSTED
+            )
 
-        Args:
-            id_server (str): The identity server to proxy to
-            email (str): The email to send the message to
-            client_secret (str): The unique client_secret sends by the user
-            send_attempt (int): Which attempt this is
-            next_link: A link to redirect the user to once they submit the token
-
-        Returns:
-            The json response body from the server
-        """
         params = {
             "email": email,
             "client_secret": client_secret,
             "send_attempt": send_attempt,
         }
-        if next_link:
-            params["next_link"] = next_link
 
-        if self.hs.config.using_identity_server_from_trusted_list:
-            # Warn that a deprecated config option is in use
-            logger.warn(
-                'The config option "trust_identity_server_for_password_resets" '
-                'has been replaced by "account_threepid_delegate". '
-                "Please consult the sample config at docs/sample_config.yaml for "
-                "details and update your config file."
-            )
+        if next_link:
+            params.update({"next_link": next_link})
 
         try:
             data = yield self.http_client.post_json_get_json(
-                id_server + "/_matrix/identity/api/v1/validate/email/requestToken",
+                "https://%s%s"
+                % (id_server, "/_matrix/identity/api/v1/validate/email/requestToken"),
                 params,
             )
             return data
@@ -408,85 +336,28 @@ class IdentityHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def requestMsisdnToken(
-        self,
-        id_server,
-        country,
-        phone_number,
-        client_secret,
-        send_attempt,
-        next_link=None,
+        self, id_server, country, phone_number, client_secret, send_attempt, **kwargs
     ):
-        """
-        Request an external server send an SMS message on our behalf for the purposes of
-        threepid validation.
-        Args:
-            id_server (str): The identity server to proxy to
-            country (str): The country code of the phone number
-            phone_number (str): The number to send the message to
-            client_secret (str): The unique client_secret sends by the user
-            send_attempt (int): Which attempt this is
-            next_link: A link to redirect the user to once they submit the token
+        if not self._should_trust_id_server(id_server):
+            raise SynapseError(
+                400, "Untrusted ID server '%s'" % id_server, Codes.SERVER_NOT_TRUSTED
+            )
 
-        Returns:
-            The json response body from the server
-        """
         params = {
             "country": country,
             "phone_number": phone_number,
             "client_secret": client_secret,
             "send_attempt": send_attempt,
         }
-        if next_link:
-            params["next_link"] = next_link
-
-        if self.hs.config.using_identity_server_from_trusted_list:
-            # Warn that a deprecated config option is in use
-            logger.warn(
-                'The config option "trust_identity_server_for_password_resets" '
-                'has been replaced by "account_threepid_delegate". '
-                "Please consult the sample config at docs/sample_config.yaml for "
-                "details and update your config file."
-            )
+        params.update(kwargs)
 
         try:
             data = yield self.http_client.post_json_get_json(
-                id_server + "/_matrix/identity/api/v1/validate/msisdn/requestToken",
+                "https://%s%s"
+                % (id_server, "/_matrix/identity/api/v1/validate/msisdn/requestToken"),
                 params,
             )
             return data
         except HttpResponseException as e:
             logger.info("Proxied requestToken failed: %r", e)
             raise e.to_synapse_error()
-
-
-def create_id_access_token_header(id_access_token):
-    """Create an Authorization header for passing to SimpleHttpClient as the header value
-    of an HTTP request.
-
-    Args:
-        id_access_token (str): An identity server access token.
-
-    Returns:
-        list[str]: The ascii-encoded bearer token encased in a list.
-    """
-    # Prefix with Bearer
-    bearer_token = "Bearer %s" % id_access_token
-
-    # Encode headers to standard ascii
-    bearer_token.encode("ascii")
-
-    # Return as a list as that's how SimpleHttpClient takes header values
-    return [bearer_token]
-
-
-class LookupAlgorithm:
-    """
-    Supported hashing algorithms when performing a 3PID lookup.
-
-    SHA256 - Hashing an (address, medium, pepper) combo with sha256, then url-safe base64
-        encoding
-    NONE - Not performing any hashing. Simply sending an (address, medium) combo in plaintext
-    """
-
-    SHA256 = "sha256"
-    NONE = "none"

synapse/handlers/message.py

@@ -729,27 +729,7 @@ class EventCreationHandler(object):
         assert not self.config.worker_app
 
         if ratelimit:
-            # We check if this is a room admin redacting an event so that we
-            # can apply different ratelimiting. We do this by simply checking
-            # it's not a self-redaction (to avoid having to look up whether the
-            # user is actually admin or not).
-            is_admin_redaction = False
-            if event.type == EventTypes.Redaction:
-                original_event = yield self.store.get_event(
-                    event.redacts,
-                    check_redacted=False,
-                    get_prev_content=False,
-                    allow_rejected=False,
-                    allow_none=True,
-                )
-
-                is_admin_redaction = (
-                    original_event and event.sender != original_event.sender
-                )
-
-            yield self.base_handler.ratelimit(
-                requester, is_admin_redaction=is_admin_redaction
-            )
+            yield self.base_handler.ratelimit(requester)
 
         yield self.base_handler.maybe_kick_guest_users(event, context)
 

synapse/handlers/register.py

@@ -275,12 +275,16 @@ class RegistrationHandler(BaseHandler):
         fake_requester = create_requester(user_id)
 
         # try to create the room if we're the first real user on the server. Note
-        # that an auto-generated support or bot user is not a real user and will never be
+        # that an auto-generated support user is not a real user and will never be
         # the user to create the room
         should_auto_create_rooms = False
-        is_real_user = yield self.store.is_real_user(user_id)
-        if self.hs.config.autocreate_auto_join_rooms and is_real_user:
-            count = yield self.store.count_real_users()
+        is_support = yield self.store.is_support_user(user_id)
+        # There is an edge case where the first user is the support user, then
+        # the room is never created, though this seems unlikely and
+        # recoverable from given the support user being involved in the first
+        # place.
+        if self.hs.config.autocreate_auto_join_rooms and not is_support:
+            count = yield self.store.count_all_users()
             should_auto_create_rooms = count == 1
         for r in self.hs.config.auto_join_rooms:
             logger.info("Auto-joining %s to %s", user_id, r)

synapse/handlers/room.py

@@ -579,8 +579,8 @@ class RoomCreationHandler(BaseHandler):
 
         room_id = yield self._generate_room_id(creator_id=user_id, is_public=is_public)
 
-        directory_handler = self.hs.get_handlers().directory_handler
         if room_alias:
+            directory_handler = self.hs.get_handlers().directory_handler
             yield directory_handler.create_association(
                 requester=requester,
                 room_id=room_id,
@@ -665,7 +665,6 @@ class RoomCreationHandler(BaseHandler):
 
         for invite_3pid in invite_3pid_list:
             id_server = invite_3pid["id_server"]
-            id_access_token = invite_3pid.get("id_access_token")  # optional
             address = invite_3pid["address"]
             medium = invite_3pid["medium"]
             yield self.hs.get_room_member_handler().do_3pid_invite(
@@ -676,7 +675,6 @@ class RoomCreationHandler(BaseHandler):
                 id_server,
                 requester,
                 txn_id=None,
-                id_access_token=id_access_token,
             )
 
         result = {"room_id": room_id}

synapse/handlers/room_member.py

@@ -29,11 +29,9 @@ from twisted.internet import defer
 from synapse import types
 from synapse.api.constants import EventTypes, Membership
 from synapse.api.errors import AuthError, Codes, HttpResponseException, SynapseError
-from synapse.handlers.identity import LookupAlgorithm, create_id_access_token_header
 from synapse.types import RoomID, UserID
 from synapse.util.async_helpers import Linearizer
 from synapse.util.distributor import user_joined_room, user_left_room
-from synapse.util.hash import sha256_and_url_safe_base64
 
 from ._base import BaseHandler
 
@@ -102,7 +100,7 @@ class RoomMemberHandler(object):
         raise NotImplementedError()
 
     @abc.abstractmethod
-    def _remote_reject_invite(self, requester, remote_room_hosts, room_id, target):
+    def _remote_reject_invite(self, remote_room_hosts, room_id, target):
         """Attempt to reject an invite for a room this server is not in. If we
         fail to do so we locally mark the invite as rejected.
 
@@ -512,7 +510,9 @@ class RoomMemberHandler(object):
         return res
 
     @defer.inlineCallbacks
-    def send_membership_event(self, requester, event, context, ratelimit=True):
+    def send_membership_event(
+        self, requester, event, context, remote_room_hosts=None, ratelimit=True
+    ):
         """
         Change the membership status of a user in a room.
 
@@ -522,10 +522,16 @@ class RoomMemberHandler(object):
                 act as the sender, will be skipped.
             event (SynapseEvent): The membership event.
             context: The context of the event.
+            is_guest (bool): Whether the sender is a guest.
+            room_hosts ([str]): Homeservers which are likely to already be in
+                the room, and could be danced with in order to join this
+                homeserver for the first time.
             ratelimit (bool): Whether to rate limit this request.
         Raises:
             SynapseError if there was a problem changing the membership.
         """
+        remote_room_hosts = remote_room_hosts or []
+
         target_user = UserID.from_string(event.state_key)
         room_id = event.room_id
 
@@ -628,7 +634,7 @@ class RoomMemberHandler(object):
             servers.remove(room_alias.domain)
         servers.insert(0, room_alias.domain)
 
-        return RoomID.from_string(room_id), servers
+        return (RoomID.from_string(room_id), servers)
 
     @defer.inlineCallbacks
     def _get_inviter(self, user_id, room_id):
@@ -640,15 +646,7 @@ class RoomMemberHandler(object):
 
     @defer.inlineCallbacks
     def do_3pid_invite(
-        self,
-        room_id,
-        inviter,
-        medium,
-        address,
-        id_server,
-        requester,
-        txn_id,
-        id_access_token=None,
+        self, room_id, inviter, medium, address, id_server, requester, txn_id
     ):
         if self.config.block_non_admin_invites:
             is_requester_admin = yield self.auth.is_server_admin(requester.user)
@@ -671,12 +669,7 @@ class RoomMemberHandler(object):
                 Codes.FORBIDDEN,
             )
 
-        if not self._enable_lookup:
-            raise SynapseError(
-                403, "Looking up third-party identifiers is denied from this server"
-            )
-
-        invitee = yield self._lookup_3pid(id_server, medium, address, id_access_token)
+        invitee = yield self._lookup_3pid(id_server, medium, address)
 
         if invitee:
             yield self.update_membership(
@@ -688,47 +681,9 @@ class RoomMemberHandler(object):
             )
 
     @defer.inlineCallbacks
-    def _lookup_3pid(self, id_server, medium, address, id_access_token=None):
+    def _lookup_3pid(self, id_server, medium, address):
         """Looks up a 3pid in the passed identity server.
 
-        Args:
-            id_server (str): The server name (including port, if required)
-                of the identity server to use.
-            medium (str): The type of the third party identifier (e.g. "email").
-            address (str): The third party identifier (e.g. "foo@example.com").
-            id_access_token (str|None): The access token to authenticate to the identity
-                server with
-
-        Returns:
-            str|None: the matrix ID of the 3pid, or None if it is not recognized.
-        """
-        if id_access_token is not None:
-            try:
-                results = yield self._lookup_3pid_v2(
-                    id_server, id_access_token, medium, address
-                )
-                return results
-
-            except Exception as e:
-                # Catch HttpResponseExcept for a non-200 response code
-                # Check if this identity server does not know about v2 lookups
-                if isinstance(e, HttpResponseException) and e.code == 404:
-                    # This is an old identity server that does not yet support v2 lookups
-                    logger.warning(
-                        "Attempted v2 lookup on v1 identity server %s. Falling "
-                        "back to v1",
-                        id_server,
-                    )
-                else:
-                    logger.warning("Error when looking up hashing details: %s", e)
-                    return None
-
-        return (yield self._lookup_3pid_v1(id_server, medium, address))
-
-    @defer.inlineCallbacks
-    def _lookup_3pid_v1(self, id_server, medium, address):
-        """Looks up a 3pid in the passed identity server using v1 lookup.
-
         Args:
             id_server (str): The server name (including port, if required)
                 of the identity server to use.
@@ -738,6 +693,10 @@ class RoomMemberHandler(object):
         Returns:
             str: the matrix ID of the 3pid, or None if it is not recognized.
         """
+        if not self._enable_lookup:
+            raise SynapseError(
+                403, "Looking up third-party identifiers is denied from this server"
+            )
         try:
             data = yield self.simple_http_client.get_json(
                 "%s%s/_matrix/identity/api/v1/lookup" % (id_server_scheme, id_server),
@@ -751,116 +710,9 @@ class RoomMemberHandler(object):
                 return data["mxid"]
 
         except IOError as e:
-            logger.warning("Error from v1 identity server lookup: %s" % (e,))
-
-        return None
-
-    @defer.inlineCallbacks
-    def _lookup_3pid_v2(self, id_server, id_access_token, medium, address):
-        """Looks up a 3pid in the passed identity server using v2 lookup.
-
-        Args:
-            id_server (str): The server name (including port, if required)
-                of the identity server to use.
-            id_access_token (str): The access token to authenticate to the identity server with
-            medium (str): The type of the third party identifier (e.g. "email").
-            address (str): The third party identifier (e.g. "foo@example.com").
-
-        Returns:
-            Deferred[str|None]: the matrix ID of the 3pid, or None if it is not recognised.
-        """
-        # Check what hashing details are supported by this identity server
-        hash_details = yield self.simple_http_client.get_json(
-            "%s%s/_matrix/identity/v2/hash_details" % (id_server_scheme, id_server),
-            {"access_token": id_access_token},
-        )
-
-        if not isinstance(hash_details, dict):
-            logger.warning(
-                "Got non-dict object when checking hash details of %s%s: %s",
-                id_server_scheme,
-                id_server,
-                hash_details,
-            )
-            raise SynapseError(
-                400,
-                "Non-dict object from %s%s during v2 hash_details request: %s"
-                % (id_server_scheme, id_server, hash_details),
-            )
-
-        # Extract information from hash_details
-        supported_lookup_algorithms = hash_details.get("algorithms")
-        lookup_pepper = hash_details.get("lookup_pepper")
-        if (
-            not supported_lookup_algorithms
-            or not isinstance(supported_lookup_algorithms, list)
-            or not lookup_pepper
-            or not isinstance(lookup_pepper, str)
-        ):
-            raise SynapseError(
-                400,
-                "Invalid hash details received from identity server %s%s: %s"
-                % (id_server_scheme, id_server, hash_details),
-            )
-
-        # Check if any of the supported lookup algorithms are present
-        if LookupAlgorithm.SHA256 in supported_lookup_algorithms:
-            # Perform a hashed lookup
-            lookup_algorithm = LookupAlgorithm.SHA256
-
-            # Hash address, medium and the pepper with sha256
-            to_hash = "%s %s %s" % (address, medium, lookup_pepper)
-            lookup_value = sha256_and_url_safe_base64(to_hash)
-
-        elif LookupAlgorithm.NONE in supported_lookup_algorithms:
-            # Perform a non-hashed lookup
-            lookup_algorithm = LookupAlgorithm.NONE
-
-            # Combine together plaintext address and medium
-            lookup_value = "%s %s" % (address, medium)
-
-        else:
-            logger.warning(
-                "None of the provided lookup algorithms of %s are supported: %s",
-                id_server,
-                supported_lookup_algorithms,
-            )
-            raise SynapseError(
-                400,
-                "Provided identity server does not support any v2 lookup "
-                "algorithms that this homeserver supports.",
-            )
-
-        # Authenticate with identity server given the access token from the client
-        headers = {"Authorization": create_id_access_token_header(id_access_token)}
-
-        try:
-            lookup_results = yield self.simple_http_client.post_json_get_json(
-                "%s%s/_matrix/identity/v2/lookup" % (id_server_scheme, id_server),
-                {
-                    "addresses": [lookup_value],
-                    "algorithm": lookup_algorithm,
-                    "pepper": lookup_pepper,
-                },
-                headers=headers,
-            )
-        except Exception as e:
-            logger.warning("Error when performing a v2 3pid lookup: %s", e)
-            raise SynapseError(
-                500, "Unknown error occurred during identity server lookup"
-            )
-
-        # Check for a mapping from what we looked up to an MXID
-        if "mappings" not in lookup_results or not isinstance(
-            lookup_results["mappings"], dict
-        ):
-            logger.warning("No results from 3pid lookup")
+            logger.warn("Error from identity server lookup: %s" % (e,))
             return None
 
-        # Return the MXID if it's available, or None otherwise
-        mxid = lookup_results["mappings"].get(lookup_value)
-        return mxid
-
     @defer.inlineCallbacks
     def _verify_any_signature(self, data, server_hostname):
         if server_hostname not in data["signatures"]:
@@ -1000,6 +852,7 @@ class RoomMemberHandler(object):
                 display_name (str): A user-friendly name to represent the invited
                     user.
         """
+
         is_url = "%s%s/_matrix/identity/api/v1/store-invite" % (
             id_server_scheme,
             id_server,
@@ -1017,6 +870,7 @@ class RoomMemberHandler(object):
             "sender_display_name": inviter_display_name,
             "sender_avatar_url": inviter_avatar_url,
         }
+
         try:
             data = yield self.simple_http_client.post_json_get_json(
                 is_url, invite_config
@@ -1203,7 +1057,7 @@ class RoomMemberMasterHandler(RoomMemberHandler):
             # The 'except' clause is very broad, but we need to
             # capture everything from DNS failures upwards
             #
-            logger.warning("Failed to reject invite: %s", e)
+            logger.warn("Failed to reject invite: %s", e)
 
             yield self.store.locally_reject_invite(target.to_string(), room_id)
             return {}

synapse/handlers/stats.py

@@ -260,9 +260,7 @@ class StatsHandler(StateDeltasHandler):
                         room_stats_delta["local_users_in_room"] += delta
 
             elif typ == EventTypes.Create:
-                room_state["is_federatable"] = (
-                    event_content.get("m.federate", True) is True
-                )
+                room_state["is_federatable"] = event_content.get("m.federate", True)
                 if sender and self.is_mine_id(sender):
                     user_to_stats_deltas.setdefault(sender, Counter())[
                         "rooms_created"

synapse/logging/_structured.py

@@ -18,7 +18,6 @@ import os.path
 import sys
 import typing
 import warnings
-from typing import List
 
 import attr
 from constantly import NamedConstant, Names, ValueConstant, Values
@@ -34,6 +33,7 @@ from twisted.logger import (
     LogLevelFilterPredicate,
     LogPublisher,
     eventAsText,
+    globalLogBeginner,
     jsonFileLogObserver,
 )
 
@@ -134,7 +134,7 @@ class PythonStdlibToTwistedLogger(logging.Handler):
         )
 
 
-def SynapseFileLogObserver(outFile: typing.IO[str]) -> FileLogObserver:
+def SynapseFileLogObserver(outFile: typing.io.TextIO) -> FileLogObserver:
     """
     A log observer that formats events like the traditional log formatter and
     sends them to `outFile`.
@@ -265,7 +265,7 @@ def setup_structured_logging(
     hs,
     config,
     log_config: dict,
-    logBeginner: LogBeginner,
+    logBeginner: LogBeginner = globalLogBeginner,
     redirect_stdlib_logging: bool = True,
 ) -> LogPublisher:
     """
@@ -286,7 +286,7 @@ def setup_structured_logging(
     if "drains" not in log_config:
         raise ConfigError("The logging configuration requires a list of drains.")
 
-    observers = []  # type: List[ILogObserver]
+    observers = []
 
     for observer in parse_drain_configs(log_config["drains"]):
         # Pipe drains

synapse/logging/_terse_json.py

@@ -21,11 +21,10 @@ import sys
 from collections import deque
 from ipaddress import IPv4Address, IPv6Address, ip_address
 from math import floor
-from typing import IO
+from typing.io import TextIO
 
 import attr
 from simplejson import dumps
-from zope.interface import implementer
 
 from twisted.application.internet import ClientService
 from twisted.internet.endpoints import (
@@ -34,7 +33,7 @@ from twisted.internet.endpoints import (
     TCP6ClientEndpoint,
 )
 from twisted.internet.protocol import Factory, Protocol
-from twisted.logger import FileLogObserver, ILogObserver, Logger
+from twisted.logger import FileLogObserver, Logger
 from twisted.python.failure import Failure
 
 
@@ -130,7 +129,7 @@ def flatten_event(event: dict, metadata: dict, include_time: bool = False):
     return new_event
 
 
-def TerseJSONToConsoleLogObserver(outFile: IO[str], metadata: dict) -> FileLogObserver:
+def TerseJSONToConsoleLogObserver(outFile: TextIO, metadata: dict) -> FileLogObserver:
     """
     A log observer that formats events to a flattened JSON representation.
 
@@ -147,7 +146,6 @@ def TerseJSONToConsoleLogObserver(outFile: IO[str], metadata: dict) -> FileLogOb
 
 
 @attr.s
-@implementer(ILogObserver)
 class TerseJSONToTCPLogObserver(object):
     """
     An IObserver that writes JSON logs to a TCP target.

synapse/logging/opentracing.py

@@ -223,8 +223,8 @@ try:
     from jaeger_client import Config as JaegerConfig
     from synapse.logging.scopecontextmanager import LogContextScopeManager
 except ImportError:
-    JaegerConfig = None  # type: ignore
-    LogContextScopeManager = None  # type: ignore
+    JaegerConfig = None
+    LogContextScopeManager = None
 
 
 logger = logging.getLogger(__name__)
@@ -702,15 +702,15 @@ def trace(func=None, opname=None):
         _opname = opname if opname else func.__name__
 
         @wraps(func)
-        def _trace_inner(*args, **kwargs):
+        def _trace_inner(self, *args, **kwargs):
             if opentracing is None:
-                return func(*args, **kwargs)
+                return func(self, *args, **kwargs)
 
             scope = start_active_span(_opname)
             scope.__enter__()
 
             try:
-                result = func(*args, **kwargs)
+                result = func(self, *args, **kwargs)
                 if isinstance(result, defer.Deferred):
 
                     def call_back(result):
@@ -750,13 +750,13 @@ def tag_args(func):
         return func
 
     @wraps(func)
-    def _tag_args_inner(*args, **kwargs):
+    def _tag_args_inner(self, *args, **kwargs):
         argspec = inspect.getargspec(func)
         for i, arg in enumerate(argspec.args[1:]):
             set_tag("ARG_" + arg, args[i])
         set_tag("args", args[len(argspec.args) :])
         set_tag("kwargs", kwargs)
-        return func(*args, **kwargs)
+        return func(self, *args, **kwargs)
 
     return _tag_args_inner
 

synapse/metrics/__init__.py

@@ -20,7 +20,6 @@ import os
 import platform
 import threading
 import time
-from typing import Dict, Union
 
 import six
 
@@ -30,20 +29,20 @@ from prometheus_client.core import REGISTRY, GaugeMetricFamily, HistogramMetricF
 
 from twisted.internet import reactor
 
-import synapse
 from synapse.metrics._exposition import (
     MetricsResource,
     generate_latest,
     start_http_server,
 )
-from synapse.util.versionstring import get_version_string
 
 logger = logging.getLogger(__name__)
 
 METRICS_PREFIX = "/_synapse/metrics"
 
 running_on_pypy = platform.python_implementation() == "PyPy"
-all_gauges = {}  # type: Dict[str, Union[LaterGauge, InFlightGauge, BucketCollector]]
+all_metrics = []
+all_collectors = []
+all_gauges = {}
 
 HAVE_PROC_SELF_STAT = os.path.exists("/proc/self/stat")
 
@@ -386,16 +385,6 @@ event_processing_last_ts = Gauge("synapse_event_processing_last_ts", "", ["name"
 # finished being processed.
 event_processing_lag = Gauge("synapse_event_processing_lag", "", ["name"])
 
-# Build info of the running server.
-build_info = Gauge(
-    "synapse_build_info", "Build information", ["pythonversion", "version", "osversion"]
-)
-build_info.labels(
-    " ".join([platform.python_implementation(), platform.python_version()]),
-    get_version_string(synapse),
-    " ".join([platform.system(), platform.release()]),
-).set(1)
-
 last_ticked = time.time()
 
 

synapse/metrics/_exposition.py

@@ -36,9 +36,7 @@ from twisted.web.resource import Resource
 try:
     from prometheus_client.samples import Sample
 except ImportError:
-    Sample = namedtuple(
-        "Sample", ["name", "labels", "value", "timestamp", "exemplar"]
-    )  # type: ignore
+    Sample = namedtuple("Sample", ["name", "labels", "value", "timestamp", "exemplar"])
 
 
 CONTENT_TYPE_LATEST = str("text/plain; version=0.0.4; charset=utf-8")

synapse/push/httppusher.py

@@ -22,7 +22,6 @@ from prometheus_client import Counter
 from twisted.internet import defer
 from twisted.internet.error import AlreadyCalled, AlreadyCancelled
 
-from synapse.logging import opentracing
 from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.push import PusherConfigException
 
@@ -195,17 +194,7 @@ class HttpPusher(object):
         )
 
         for push_action in unprocessed:
-            with opentracing.start_active_span(
-                "http-push",
-                tags={
-                    "authenticated_entity": self.user_id,
-                    "event_id": push_action["event_id"],
-                    "app_id": self.app_id,
-                    "app_display_name": self.app_display_name,
-                },
-            ):
-                processed = yield self._process_one(push_action)
-
+            processed = yield self._process_one(push_action)
             if processed:
                 http_push_processed_counter.inc()
                 self.backoff_delay = HttpPusher.INITIAL_BACKOFF_SEC

synapse/push/mailer.py

@@ -131,11 +131,14 @@ class Mailer(object):
             email_address (str): Email address we're sending the password
                 reset to
             token (str): Unique token generated by the server to verify
-                the email was received
+                password reset email was received
             client_secret (str): Unique token generated by the client to
                 group together multiple email sending attempts
             sid (str): The generated session ID
         """
+        if email.utils.parseaddr(email_address)[1] == "":
+            raise RuntimeError("Invalid 'to' email address")
+
         link = (
             self.hs.config.public_baseurl
             + "_matrix/client/unstable/password_reset/email/submit_token"
@@ -146,34 +149,7 @@ class Mailer(object):
 
         yield self.send_email(
             email_address,
-            "[%s] Password Reset" % self.hs.config.server_name,
-            template_vars,
-        )
-
-    @defer.inlineCallbacks
-    def send_registration_mail(self, email_address, token, client_secret, sid):
-        """Send an email with a registration confirmation link to a user
-
-        Args:
-            email_address (str): Email address we're sending the registration
-                link to
-            token (str): Unique token generated by the server to verify
-                the email was received
-            client_secret (str): Unique token generated by the client to
-                group together multiple email sending attempts
-            sid (str): The generated session ID
-        """
-        link = (
-            self.hs.config.public_baseurl
-            + "_matrix/client/unstable/registration/email/submit_token"
-            "?token=%s&client_secret=%s&sid=%s" % (token, client_secret, sid)
-        )
-
-        template_vars = {"link": link}
-
-        yield self.send_email(
-            email_address,
-            "[%s] Register your Email Address" % self.hs.config.server_name,
+            "[%s] Password Reset Email" % self.hs.config.server_name,
             template_vars,
         )
 
@@ -629,50 +605,25 @@ def format_ts_filter(value, format):
     return time.strftime(format, time.localtime(value / 1000))
 
 
-def load_jinja2_templates(
-    template_dir,
-    template_filenames,
-    apply_format_ts_filter=False,
-    apply_mxc_to_http_filter=False,
-    public_baseurl=None,
-):
-    """Loads and returns one or more jinja2 templates and applies optional filters
-
-    Args:
-        template_dir (str): The directory where templates are stored
-        template_filenames (list[str]): A list of template filenames
-        apply_format_ts_filter (bool): Whether to apply a template filter that formats
-            timestamps
-        apply_mxc_to_http_filter (bool): Whether to apply a template filter that converts
-            mxc urls to http urls
-        public_baseurl (str|None): The public baseurl of the server. Required for
-            apply_mxc_to_http_filter to be enabled
+def load_jinja2_templates(config, template_html_name, template_text_name):
+    """Load the jinja2 email templates from disk
 
     Returns:
-        A list of jinja2 templates corresponding to the given list of filenames,
-        with order preserved
+        (template_html, template_text)
     """
-    logger.info(
-        "loading email templates %s from '%s'", template_filenames, template_dir
-    )
-    loader = jinja2.FileSystemLoader(template_dir)
+    logger.info("loading email templates from '%s'", config.email_template_dir)
+    loader = jinja2.FileSystemLoader(config.email_template_dir)
     env = jinja2.Environment(loader=loader)
+    env.filters["format_ts"] = format_ts_filter
+    env.filters["mxc_to_http"] = _create_mxc_to_http_filter(config)
 
-    if apply_format_ts_filter:
-        env.filters["format_ts"] = format_ts_filter
+    template_html = env.get_template(template_html_name)
+    template_text = env.get_template(template_text_name)
 
-    if apply_mxc_to_http_filter and public_baseurl:
-        env.filters["mxc_to_http"] = _create_mxc_to_http_filter(public_baseurl)
-
-    templates = []
-    for template_filename in template_filenames:
-        template = env.get_template(template_filename)
-        templates.append(template)
-
-    return templates
+    return template_html, template_text
 
 
-def _create_mxc_to_http_filter(public_baseurl):
+def _create_mxc_to_http_filter(config):
     def mxc_to_http_filter(value, width, height, resize_method="crop"):
         if value[0:6] != "mxc://":
             return ""
@@ -685,7 +636,7 @@ def _create_mxc_to_http_filter(public_baseurl):
 
         params = {"width": width, "height": height, "method": resize_method}
         return "%s_matrix/media/v1/thumbnail/%s?%s%s" % (
-            public_baseurl,
+            config.public_baseurl,
             serverAndMediaId,
             urllib.parse.urlencode(params),
             fragment or "",

synapse/push/pusher.py

@@ -35,7 +35,6 @@ except Exception:
 class PusherFactory(object):
     def __init__(self, hs):
         self.hs = hs
-        self.config = hs.config
 
         self.pusher_types = {"http": HttpPusher}
 
@@ -43,16 +42,12 @@ class PusherFactory(object):
         if hs.config.email_enable_notifs:
             self.mailers = {}  # app_name -> Mailer
 
-            self.notif_template_html, self.notif_template_text = load_jinja2_templates(
-                self.config.email_template_dir,
-                [
-                    self.config.email_notif_template_html,
-                    self.config.email_notif_template_text,
-                ],
-                apply_format_ts_filter=True,
-                apply_mxc_to_http_filter=True,
-                public_baseurl=self.config.public_baseurl,
+            templates = load_jinja2_templates(
+                config=hs.config,
+                template_html_name=hs.config.email_notif_template_html,
+                template_text_name=hs.config.email_notif_template_text,
             )
+            self.notif_template_html, self.notif_template_text = templates
 
             self.pusher_types["email"] = self._create_email_pusher
 
@@ -83,6 +78,6 @@ class PusherFactory(object):
         if "data" in pusherdict and "brand" in pusherdict["data"]:
             app_name = pusherdict["data"]["brand"]
         else:
-            app_name = self.config.email_app_name
+            app_name = self.hs.config.email_app_name
 
         return app_name

synapse/python_dependencies.py

@@ -15,7 +15,6 @@
 # limitations under the License.
 
 import logging
-from typing import Set
 
 from pkg_resources import (
     DistributionNotFound,
@@ -98,7 +97,7 @@ CONDITIONAL_REQUIREMENTS = {
     "jwt": ["pyjwt>=1.6.4"],
 }
 
-ALL_OPTIONAL_REQUIREMENTS = set()  # type: Set[str]
+ALL_OPTIONAL_REQUIREMENTS = set()
 
 for name, optional_deps in CONDITIONAL_REQUIREMENTS.items():
     # Exclude systemd as it's a system-based requirement.
@@ -148,13 +147,7 @@ def check_requirements(for_feature=None):
             )
         except DistributionNotFound:
             deps_needed.append(dependency)
-            if for_feature:
-                errors.append(
-                    "Needed %s for the '%s' feature but it was not installed"
-                    % (dependency, for_feature)
-                )
-            else:
-                errors.append("Needed %s but it was not installed" % (dependency,))
+            errors.append("Needed %s but it was not installed" % (dependency,))
 
     if not for_feature:
         # Check the optional dependencies are up to date. We allow them to not be
@@ -175,8 +168,8 @@ def check_requirements(for_feature=None):
                 pass
 
     if deps_needed:
-        for err in errors:
-            logging.error(err)
+        for e in errors:
+            logging.error(e)
 
         raise DependencyException(deps_needed)
 

synapse/res/templates/password_reset.html

@@ -4,6 +4,6 @@
 
     <a href="{{ link }}">{{ link }}</a>
 
-    <p>If this was not you, <strong>do not</strong> click the link above and instead contact your server administrator. Thank you.</p>
+    <p>If this was not you, please disregard this email and contact your server administrator. Thank you.</p>
 </body>
 </html>

synapse/res/templates/password_reset.txt

@@ -3,5 +3,5 @@ was you, please click the link below to confirm resetting your password:
 
 {{ link }}
 
-If this was not you, DO NOT click the link above and instead contact your
-server administrator. Thank you.
+If this was not you, please disregard this email and contact your server
+administrator. Thank you.

synapse/res/templates/password_reset_failure.html

@@ -1,8 +1,6 @@
 <html>
 <head></head>
 <body>
-<p>The request failed for the following reason: {{ failure_reason }}.</p>
-
-<p>Your password has not been reset.</p>
+<p>{{ failure_reason }}. Your password has not been reset.</p>
 </body>
 </html>

synapse/res/templates/registration.html

@@ -1,11 +0,0 @@
-<html>
-<body>
-    <p>You have asked us to register this email with a new Matrix account. If this was you, please click the link below to confirm your email address:</p>
-
-    <a href="{{ link }}">Verify Your Email Address</a>
-
-    <p>If this was not you, you can safely disregard this email.</p>
-
-    <p>Thank you.</p>
-</body>
-</html>

synapse/res/templates/registration.txt

@@ -1,10 +0,0 @@
-Hello there,
-
-You have asked us to register this email with a new Matrix account. If this
-was you, please click the link below to confirm your email address:
-
-{{ link }}
-
-If this was not you, you can safely disregard this email.
-
-Thank you.

synapse/res/templates/registration_failure.html

@@ -1,6 +0,0 @@
-<html>
-<head></head>
-<body>
-<p>Validation failed for the following reason: {{ failure_reason }}.</p>
-</body>
-</html>

synapse/res/templates/registration_success.html

@@ -1,6 +0,0 @@
-<html>
-<head></head>
-<body>
-<p>Your email has now been validated, please return to your client. You may now close this window.</p>
-</body>
-</html>

synapse/rest/__init__.py

@@ -73,7 +73,7 @@ class ClientRestResource(JsonResource):
 
     @staticmethod
     def register_servlets(client_resource, hs):
-        versions.register_servlets(hs, client_resource)
+        versions.register_servlets(client_resource)
 
         # Deprecated in r0
         initial_sync.register_servlets(hs, client_resource)

synapse/rest/client/v1/room.py

@@ -701,7 +701,6 @@ class RoomMembershipRestServlet(TransactionRestServlet):
                 content["id_server"],
                 requester,
                 txn_id,
-                content.get("id_access_token"),
             )
             return 200, {}
 

synapse/rest/client/v2_alpha/_base.py

@@ -37,7 +37,6 @@ def client_patterns(path_regex, releases=(0,), unstable=True, v1=False):
         SRE_Pattern
     """
     patterns = []
-
     if unstable:
         unstable_prefix = CLIENT_API_PREFIX + "/unstable"
         patterns.append(re.compile("^" + unstable_prefix + path_regex))
@@ -47,7 +46,6 @@ def client_patterns(path_regex, releases=(0,), unstable=True, v1=False):
     for release in releases:
         new_prefix = CLIENT_API_PREFIX + "/r%d" % (release,)
         patterns.append(re.compile("^" + new_prefix + path_regex))
-
     return patterns
 
 

synapse/rest/client/v2_alpha/account.py

@@ -18,11 +18,12 @@ import logging
 
 from six.moves import http_client
 
+import jinja2
+
 from twisted.internet import defer
 
 from synapse.api.constants import LoginType
 from synapse.api.errors import Codes, SynapseError, ThreepidValidationError
-from synapse.config.emailconfig import ThreepidBehaviour
 from synapse.http.server import finish_request
 from synapse.http.servlet import (
     RestServlet,
@@ -30,8 +31,8 @@ from synapse.http.servlet import (
     parse_json_object_from_request,
     parse_string,
 )
-from synapse.push.mailer import Mailer, load_jinja2_templates
 from synapse.util.msisdn import phone_number_to_msisdn
+from synapse.util.stringutils import random_string
 from synapse.util.threepids import check_3pid_allowed
 
 from ._base import client_patterns, interactive_auth_handler
@@ -49,28 +50,25 @@ class EmailPasswordRequestTokenRestServlet(RestServlet):
         self.config = hs.config
         self.identity_handler = hs.get_handlers().identity_handler
 
-        if self.config.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
-            template_html, template_text = load_jinja2_templates(
-                self.config.email_template_dir,
-                [
-                    self.config.email_password_reset_template_html,
-                    self.config.email_password_reset_template_text,
-                ],
-                apply_format_ts_filter=True,
-                apply_mxc_to_http_filter=True,
-                public_baseurl=self.config.public_baseurl,
+        if self.config.email_password_reset_behaviour == "local":
+            from synapse.push.mailer import Mailer, load_jinja2_templates
+
+            templates = load_jinja2_templates(
+                config=hs.config,
+                template_html_name=hs.config.email_password_reset_template_html,
+                template_text_name=hs.config.email_password_reset_template_text,
             )
             self.mailer = Mailer(
                 hs=self.hs,
                 app_name=self.config.email_app_name,
-                template_html=template_html,
-                template_text=template_text,
+                template_html=templates[0],
+                template_text=templates[1],
             )
 
     @defer.inlineCallbacks
     def on_POST(self, request):
-        if self.config.threepid_behaviour_email == ThreepidBehaviour.OFF:
-            if self.config.local_threepid_handling_disabled_due_to_email_config:
+        if self.config.email_password_reset_behaviour == "off":
+            if self.config.password_resets_were_disabled_due_to_email_config:
                 logger.warn(
                     "User password resets have been disabled due to lack of email config"
                 )
@@ -95,39 +93,25 @@ class EmailPasswordRequestTokenRestServlet(RestServlet):
                 Codes.THREEPID_DENIED,
             )
 
-        existing_user_id = yield self.hs.get_datastore().get_user_id_by_threepid(
+        existingUid = yield self.hs.get_datastore().get_user_id_by_threepid(
             "email", email
         )
 
-        if existing_user_id is None:
+        if existingUid is None:
             raise SynapseError(400, "Email not found", Codes.THREEPID_NOT_FOUND)
 
-        if self.config.threepid_behaviour_email == ThreepidBehaviour.REMOTE:
-            # Have the configured identity server handle the request
-            if not self.hs.config.account_threepid_delegate_email:
-                logger.warn(
-                    "No upstream email account_threepid_delegate configured on the server to "
-                    "handle this request"
-                )
-                raise SynapseError(
-                    400, "Password reset by email is not supported on this homeserver"
-                )
+        if self.config.email_password_reset_behaviour == "remote":
+            if "id_server" not in body:
+                raise SynapseError(400, "Missing 'id_server' param in body")
 
+            # Have the identity server handle the password reset flow
             ret = yield self.identity_handler.requestEmailToken(
-                self.hs.config.account_threepid_delegate_email,
-                email,
-                client_secret,
-                send_attempt,
-                next_link,
+                body["id_server"], email, client_secret, send_attempt, next_link
             )
         else:
             # Send password reset emails from Synapse
-            sid = yield self.identity_handler.send_threepid_validation(
-                email,
-                client_secret,
-                send_attempt,
-                self.mailer.send_password_reset_mail,
-                next_link,
+            sid = yield self.send_password_reset(
+                email, client_secret, send_attempt, next_link
             )
 
             # Wrap the session id in a JSON object
@@ -135,6 +119,74 @@ class EmailPasswordRequestTokenRestServlet(RestServlet):
 
         return 200, ret
 
+    @defer.inlineCallbacks
+    def send_password_reset(self, email, client_secret, send_attempt, next_link=None):
+        """Send a password reset email
+
+        Args:
+            email (str): The user's email address
+            client_secret (str): The provided client secret
+            send_attempt (int): Which send attempt this is
+
+        Returns:
+            The new session_id upon success
+
+        Raises:
+            SynapseError is an error occurred when sending the email
+        """
+        # Check that this email/client_secret/send_attempt combo is new or
+        # greater than what we've seen previously
+        session = yield self.datastore.get_threepid_validation_session(
+            "email", client_secret, address=email, validated=False
+        )
+
+        # Check to see if a session already exists and that it is not yet
+        # marked as validated
+        if session and session.get("validated_at") is None:
+            session_id = session["session_id"]
+            last_send_attempt = session["last_send_attempt"]
+
+            # Check that the send_attempt is higher than previous attempts
+            if send_attempt <= last_send_attempt:
+                # If not, just return a success without sending an email
+                return session_id
+        else:
+            # An non-validated session does not exist yet.
+            # Generate a session id
+            session_id = random_string(16)
+
+        # Generate a new validation token
+        token = random_string(32)
+
+        # Send the mail with the link containing the token, client_secret
+        # and session_id
+        try:
+            yield self.mailer.send_password_reset_mail(
+                email, token, client_secret, session_id
+            )
+        except Exception:
+            logger.exception("Error sending a password reset email to %s", email)
+            raise SynapseError(
+                500, "An error was encountered when sending the password reset email"
+            )
+
+        token_expires = (
+            self.hs.clock.time_msec() + self.config.email_validation_token_lifetime
+        )
+
+        yield self.datastore.start_or_continue_validation_session(
+            "email",
+            email,
+            session_id,
+            client_secret,
+            send_attempt,
+            next_link,
+            token,
+            token_expires,
+        )
+
+        return session_id
+
 
 class MsisdnPasswordRequestTokenRestServlet(RestServlet):
     PATTERNS = client_patterns("/account/password/msisdn/requestToken$")
@@ -150,15 +202,11 @@ class MsisdnPasswordRequestTokenRestServlet(RestServlet):
         body = parse_json_object_from_request(request)
 
         assert_params_in_dict(
-            body, ["client_secret", "country", "phone_number", "send_attempt"]
+            body,
+            ["id_server", "client_secret", "country", "phone_number", "send_attempt"],
         )
-        client_secret = body["client_secret"]
-        country = body["country"]
-        phone_number = body["phone_number"]
-        send_attempt = body["send_attempt"]
-        next_link = body.get("next_link")  # Optional param
 
-        msisdn = phone_number_to_msisdn(country, phone_number)
+        msisdn = phone_number_to_msisdn(body["country"], body["phone_number"])
 
         if not check_3pid_allowed(self.hs, "msisdn", msisdn):
             raise SynapseError(
@@ -167,32 +215,12 @@ class MsisdnPasswordRequestTokenRestServlet(RestServlet):
                 Codes.THREEPID_DENIED,
             )
 
-        existing_user_id = yield self.datastore.get_user_id_by_threepid(
-            "msisdn", msisdn
-        )
+        existingUid = yield self.datastore.get_user_id_by_threepid("msisdn", msisdn)
 
-        if existing_user_id is None:
+        if existingUid is None:
             raise SynapseError(400, "MSISDN not found", Codes.THREEPID_NOT_FOUND)
 
-        if not self.hs.config.account_threepid_delegate_msisdn:
-            logger.warn(
-                "No upstream msisdn account_threepid_delegate configured on the server to "
-                "handle this request"
-            )
-            raise SynapseError(
-                400,
-                "Password reset by phone number is not supported on this homeserver",
-            )
-
-        ret = yield self.identity_handler.requestMsisdnToken(
-            self.hs.config.account_threepid_delegate_msisdn,
-            country,
-            phone_number,
-            client_secret,
-            send_attempt,
-            next_link,
-        )
-
+        ret = yield self.identity_handler.requestMsisdnToken(**body)
         return 200, ret
 
 
@@ -213,32 +241,31 @@ class PasswordResetSubmitTokenServlet(RestServlet):
         self.auth = hs.get_auth()
         self.config = hs.config
         self.clock = hs.get_clock()
-        self.store = hs.get_datastore()
+        self.datastore = hs.get_datastore()
 
     @defer.inlineCallbacks
     def on_GET(self, request, medium):
-        # We currently only handle threepid token submissions for email
         if medium != "email":
             raise SynapseError(
                 400, "This medium is currently not supported for password resets"
             )
-        if self.config.threepid_behaviour_email == ThreepidBehaviour.OFF:
-            if self.config.local_threepid_handling_disabled_due_to_email_config:
+        if self.config.email_password_reset_behaviour == "off":
+            if self.config.password_resets_were_disabled_due_to_email_config:
                 logger.warn(
-                    "Password reset emails have been disabled due to lack of an email config"
+                    "User password resets have been disabled due to lack of email config"
                 )
             raise SynapseError(
-                400, "Email-based password resets are disabled on this server"
+                400, "Email-based password resets have been disabled on this server"
             )
 
-        sid = parse_string(request, "sid", required=True)
-        client_secret = parse_string(request, "client_secret", required=True)
-        token = parse_string(request, "token", required=True)
+        sid = parse_string(request, "sid")
+        client_secret = parse_string(request, "client_secret")
+        token = parse_string(request, "token")
 
-        # Attempt to validate a 3PID session
+        # Attempt to validate a 3PID sesssion
         try:
             # Mark the session as valid
-            next_link = yield self.store.validate_threepid_session(
+            next_link = yield self.datastore.validate_threepid_session(
                 sid, client_secret, token, self.clock.time_msec()
             )
 
@@ -255,22 +282,38 @@ class PasswordResetSubmitTokenServlet(RestServlet):
                     return None
 
             # Otherwise show the success template
-            html = self.config.email_password_reset_template_success_html
+            html = self.config.email_password_reset_template_success_html_content
             request.setResponseCode(200)
         except ThreepidValidationError as e:
-            request.setResponseCode(e.code)
-
             # Show a failure page with a reason
-            html_template, = load_jinja2_templates(
+            html = self.load_jinja2_template(
                 self.config.email_template_dir,
-                [self.config.email_password_reset_template_failure_html],
+                self.config.email_password_reset_template_failure_html,
+                template_vars={"failure_reason": e.msg},
             )
-
-            template_vars = {"failure_reason": e.msg}
-            html = html_template.render(**template_vars)
+            request.setResponseCode(e.code)
 
         request.write(html.encode("utf-8"))
         finish_request(request)
+        return None
+
+    def load_jinja2_template(self, template_dir, template_filename, template_vars):
+        """Loads a jinja2 template with variables to insert
+
+        Args:
+            template_dir (str): The directory where templates are stored
+            template_filename (str): The name of the template in the template_dir
+            template_vars (Dict): Dictionary of keys in the template
+                alongside their values to insert
+
+        Returns:
+            str containing the contents of the rendered template
+        """
+        loader = jinja2.FileSystemLoader(template_dir)
+        env = jinja2.Environment(loader=loader)
+
+        template = env.get_template(template_filename)
+        return template.render(**template_vars)
 
     @defer.inlineCallbacks
     def on_POST(self, request, medium):
@@ -282,7 +325,7 @@ class PasswordResetSubmitTokenServlet(RestServlet):
         body = parse_json_object_from_request(request)
         assert_params_in_dict(body, ["sid", "client_secret", "token"])
 
-        valid, _ = yield self.store.validate_threepid_session(
+        valid, _ = yield self.datastore.validate_threepid_validation_token(
             body["sid"], body["client_secret"], body["token"], self.clock.time_msec()
         )
         response_code = 200 if valid else 400
@@ -328,6 +371,7 @@ class PasswordRestServlet(RestServlet):
                 [[LoginType.EMAIL_IDENTITY], [LoginType.MSISDN]],
                 body,
                 self.hs.get_ip_from_request(request),
+                password_servlet=True,
             )
 
             if LoginType.EMAIL_IDENTITY in result:
@@ -410,11 +454,10 @@ class EmailThreepidRequestTokenRestServlet(RestServlet):
     PATTERNS = client_patterns("/account/3pid/email/requestToken$")
 
     def __init__(self, hs):
-        super(EmailThreepidRequestTokenRestServlet, self).__init__()
         self.hs = hs
-        self.config = hs.config
+        super(EmailThreepidRequestTokenRestServlet, self).__init__()
         self.identity_handler = hs.get_handlers().identity_handler
-        self.store = self.hs.get_datastore()
+        self.datastore = self.hs.get_datastore()
 
     @defer.inlineCallbacks
     def on_POST(self, request):
@@ -422,29 +465,22 @@ class EmailThreepidRequestTokenRestServlet(RestServlet):
         assert_params_in_dict(
             body, ["id_server", "client_secret", "email", "send_attempt"]
         )
-        id_server = "https://" + body["id_server"]  # Assume https
-        client_secret = body["client_secret"]
-        email = body["email"]
-        send_attempt = body["send_attempt"]
-        next_link = body.get("next_link")  # Optional param
 
-        if not check_3pid_allowed(self.hs, "email", email):
+        if not check_3pid_allowed(self.hs, "email", body["email"]):
             raise SynapseError(
                 403,
                 "Your email domain is not authorized on this server",
                 Codes.THREEPID_DENIED,
             )
 
-        existing_user_id = yield self.store.get_user_id_by_threepid(
+        existingUid = yield self.datastore.get_user_id_by_threepid(
             "email", body["email"]
         )
 
-        if existing_user_id is not None:
+        if existingUid is not None:
             raise SynapseError(400, "Email is already in use", Codes.THREEPID_IN_USE)
 
-        ret = yield self.identity_handler.requestEmailToken(
-            id_server, email, client_secret, send_attempt, next_link
-        )
+        ret = yield self.identity_handler.requestEmailToken(**body)
         return 200, ret
 
 
@@ -454,8 +490,8 @@ class MsisdnThreepidRequestTokenRestServlet(RestServlet):
     def __init__(self, hs):
         self.hs = hs
         super(MsisdnThreepidRequestTokenRestServlet, self).__init__()
-        self.store = self.hs.get_datastore()
         self.identity_handler = hs.get_handlers().identity_handler
+        self.datastore = self.hs.get_datastore()
 
     @defer.inlineCallbacks
     def on_POST(self, request):
@@ -464,14 +500,8 @@ class MsisdnThreepidRequestTokenRestServlet(RestServlet):
             body,
             ["id_server", "client_secret", "country", "phone_number", "send_attempt"],
         )
-        id_server = "https://" + body["id_server"]  # Assume https
-        client_secret = body["client_secret"]
-        country = body["country"]
-        phone_number = body["phone_number"]
-        send_attempt = body["send_attempt"]
-        next_link = body.get("next_link")  # Optional param
 
-        msisdn = phone_number_to_msisdn(country, phone_number)
+        msisdn = phone_number_to_msisdn(body["country"], body["phone_number"])
 
         if not check_3pid_allowed(self.hs, "msisdn", msisdn):
             raise SynapseError(
@@ -480,14 +510,12 @@ class MsisdnThreepidRequestTokenRestServlet(RestServlet):
                 Codes.THREEPID_DENIED,
             )
 
-        existing_user_id = yield self.store.get_user_id_by_threepid("msisdn", msisdn)
+        existingUid = yield self.datastore.get_user_id_by_threepid("msisdn", msisdn)
 
-        if existing_user_id is not None:
+        if existingUid is not None:
             raise SynapseError(400, "MSISDN is already in use", Codes.THREEPID_IN_USE)
 
-        ret = yield self.identity_handler.requestMsisdnToken(
-            id_server, country, phone_number, client_secret, send_attempt, next_link
-        )
+        ret = yield self.identity_handler.requestMsisdnToken(**body)
         return 200, ret
 
 
@@ -523,8 +551,7 @@ class ThreepidRestServlet(RestServlet):
         requester = yield self.auth.get_user_by_req(request)
         user_id = requester.user.to_string()
 
-        # Specify None as the identity server to retrieve it from the request body instead
-        threepid = yield self.identity_handler.threepid_from_creds(None, threepid_creds)
+        threepid = yield self.identity_handler.threepid_from_creds(threepid_creds)
 
         if not threepid:
             raise SynapseError(400, "Failed to auth 3pid", Codes.THREEPID_AUTH_FAILED)

synapse/rest/client/v2_alpha/register.py

@@ -28,20 +28,16 @@ from synapse.api.errors import (
     Codes,
     LimitExceededError,
     SynapseError,
-    ThreepidValidationError,
     UnrecognizedRequestError,
 )
-from synapse.config.emailconfig import ThreepidBehaviour
 from synapse.config.ratelimiting import FederationRateLimitConfig
 from synapse.config.server import is_threepid_reserved
-from synapse.http.server import finish_request
 from synapse.http.servlet import (
     RestServlet,
     assert_params_in_dict,
     parse_json_object_from_request,
     parse_string,
 )
-from synapse.push.mailer import load_jinja2_templates
 from synapse.util.msisdn import phone_number_to_msisdn
 from synapse.util.ratelimitutils import FederationRateLimiter
 from synapse.util.threepids import check_3pid_allowed
@@ -74,92 +70,30 @@ class EmailRegisterRequestTokenRestServlet(RestServlet):
         super(EmailRegisterRequestTokenRestServlet, self).__init__()
         self.hs = hs
         self.identity_handler = hs.get_handlers().identity_handler
-        self.config = hs.config
-
-        if self.hs.config.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
-            from synapse.push.mailer import Mailer, load_jinja2_templates
-
-            template_html, template_text = load_jinja2_templates(
-                self.config.email_template_dir,
-                [
-                    self.config.email_registration_template_html,
-                    self.config.email_registration_template_text,
-                ],
-                apply_format_ts_filter=True,
-                apply_mxc_to_http_filter=True,
-                public_baseurl=self.config.public_baseurl,
-            )
-            self.mailer = Mailer(
-                hs=self.hs,
-                app_name=self.config.email_app_name,
-                template_html=template_html,
-                template_text=template_text,
-            )
 
     @defer.inlineCallbacks
     def on_POST(self, request):
-        if self.hs.config.threepid_behaviour_email == ThreepidBehaviour.OFF:
-            if self.hs.config.local_threepid_handling_disabled_due_to_email_config:
-                logger.warn(
-                    "Email registration has been disabled due to lack of email config"
-                )
-            raise SynapseError(
-                400, "Email-based registration has been disabled on this server"
-            )
         body = parse_json_object_from_request(request)
 
-        assert_params_in_dict(body, ["client_secret", "email", "send_attempt"])
+        assert_params_in_dict(
+            body, ["id_server", "client_secret", "email", "send_attempt"]
+        )
 
-        # Extract params from body
-        client_secret = body["client_secret"]
-        email = body["email"]
-        send_attempt = body["send_attempt"]
-        next_link = body.get("next_link")  # Optional param
-
-        if not check_3pid_allowed(self.hs, "email", email):
+        if not check_3pid_allowed(self.hs, "email", body["email"]):
             raise SynapseError(
                 403,
                 "Your email domain is not authorized to register on this server",
                 Codes.THREEPID_DENIED,
             )
 
-        existing_user_id = yield self.hs.get_datastore().get_user_id_by_threepid(
+        existingUid = yield self.hs.get_datastore().get_user_id_by_threepid(
             "email", body["email"]
         )
 
-        if existing_user_id is not None:
+        if existingUid is not None:
             raise SynapseError(400, "Email is already in use", Codes.THREEPID_IN_USE)
 
-        if self.config.threepid_behaviour_email == ThreepidBehaviour.REMOTE:
-            if not self.hs.config.account_threepid_delegate_email:
-                logger.warn(
-                    "No upstream email account_threepid_delegate configured on the server to "
-                    "handle this request"
-                )
-                raise SynapseError(
-                    400, "Registration by email is not supported on this homeserver"
-                )
-
-            ret = yield self.identity_handler.requestEmailToken(
-                self.hs.config.account_threepid_delegate_email,
-                email,
-                client_secret,
-                send_attempt,
-                next_link,
-            )
-        else:
-            # Send registration emails from Synapse
-            sid = yield self.identity_handler.send_threepid_validation(
-                email,
-                client_secret,
-                send_attempt,
-                self.mailer.send_registration_mail,
-                next_link,
-            )
-
-            # Wrap the session id in a JSON object
-            ret = {"sid": sid}
-
+        ret = yield self.identity_handler.requestEmailToken(**body)
         return 200, ret
 
 
@@ -180,15 +114,11 @@ class MsisdnRegisterRequestTokenRestServlet(RestServlet):
         body = parse_json_object_from_request(request)
 
         assert_params_in_dict(
-            body, ["client_secret", "country", "phone_number", "send_attempt"]
+            body,
+            ["id_server", "client_secret", "country", "phone_number", "send_attempt"],
         )
-        client_secret = body["client_secret"]
-        country = body["country"]
-        phone_number = body["phone_number"]
-        send_attempt = body["send_attempt"]
-        next_link = body.get("next_link")  # Optional param
 
-        msisdn = phone_number_to_msisdn(country, phone_number)
+        msisdn = phone_number_to_msisdn(body["country"], body["phone_number"])
 
         if not check_3pid_allowed(self.hs, "msisdn", msisdn):
             raise SynapseError(
@@ -197,114 +127,19 @@ class MsisdnRegisterRequestTokenRestServlet(RestServlet):
                 Codes.THREEPID_DENIED,
             )
 
-        existing_user_id = yield self.hs.get_datastore().get_user_id_by_threepid(
+        existingUid = yield self.hs.get_datastore().get_user_id_by_threepid(
             "msisdn", msisdn
         )
 
-        if existing_user_id is not None:
+        if existingUid is not None:
             raise SynapseError(
                 400, "Phone number is already in use", Codes.THREEPID_IN_USE
             )
 
-        if not self.hs.config.account_threepid_delegate_msisdn:
-            logger.warn(
-                "No upstream msisdn account_threepid_delegate configured on the server to "
-                "handle this request"
-            )
-            raise SynapseError(
-                400, "Registration by phone number is not supported on this homeserver"
-            )
-
-        ret = yield self.identity_handler.requestMsisdnToken(
-            self.hs.config.account_threepid_delegate_msisdn,
-            country,
-            phone_number,
-            client_secret,
-            send_attempt,
-            next_link,
-        )
-
+        ret = yield self.identity_handler.requestMsisdnToken(**body)
         return 200, ret
 
 
-class RegistrationSubmitTokenServlet(RestServlet):
-    """Handles registration 3PID validation token submission"""
-
-    PATTERNS = client_patterns(
-        "/registration/(?P<medium>[^/]*)/submit_token$", releases=(), unstable=True
-    )
-
-    def __init__(self, hs):
-        """
-        Args:
-            hs (synapse.server.HomeServer): server
-        """
-        super(RegistrationSubmitTokenServlet, self).__init__()
-        self.hs = hs
-        self.auth = hs.get_auth()
-        self.config = hs.config
-        self.clock = hs.get_clock()
-        self.store = hs.get_datastore()
-
-    @defer.inlineCallbacks
-    def on_GET(self, request, medium):
-        if medium != "email":
-            raise SynapseError(
-                400, "This medium is currently not supported for registration"
-            )
-        if self.config.threepid_behaviour_email == ThreepidBehaviour.OFF:
-            if self.config.local_threepid_handling_disabled_due_to_email_config:
-                logger.warn(
-                    "User registration via email has been disabled due to lack of email config"
-                )
-            raise SynapseError(
-                400, "Email-based registration is disabled on this server"
-            )
-
-        sid = parse_string(request, "sid", required=True)
-        client_secret = parse_string(request, "client_secret", required=True)
-        token = parse_string(request, "token", required=True)
-
-        # Attempt to validate a 3PID session
-        try:
-            # Mark the session as valid
-            next_link = yield self.store.validate_threepid_session(
-                sid, client_secret, token, self.clock.time_msec()
-            )
-
-            # Perform a 302 redirect if next_link is set
-            if next_link:
-                if next_link.startswith("file:///"):
-                    logger.warn(
-                        "Not redirecting to next_link as it is a local file: address"
-                    )
-                else:
-                    request.setResponseCode(302)
-                    request.setHeader("Location", next_link)
-                    finish_request(request)
-                    return None
-
-            # Otherwise show the success template
-            html = self.config.email_registration_template_success_html_content
-
-            request.setResponseCode(200)
-        except ThreepidValidationError as e:
-            # Show a failure page with a reason
-            request.setResponseCode(e.code)
-
-            # Show a failure page with a reason
-            html_template, = load_jinja2_templates(
-                self.config.email_template_dir,
-                [self.config.email_registration_template_failure_html],
-            )
-
-            template_vars = {"failure_reason": e.msg}
-            html = html_template.render(**template_vars)
-
-        request.write(html.encode("utf-8"))
-        finish_request(request)
-
-
 class UsernameAvailabilityRestServlet(RestServlet):
     PATTERNS = client_patterns("/register/available")
 
@@ -603,11 +438,11 @@ class RegisterRestServlet(RestServlet):
                         medium = auth_result[login_type]["medium"]
                         address = auth_result[login_type]["address"]
 
-                        existing_user_id = yield self.store.get_user_id_by_threepid(
+                        existingUid = yield self.store.get_user_id_by_threepid(
                             medium, address
                         )
 
-                        if existing_user_id is not None:
+                        if existingUid is not None:
                             raise SynapseError(
                                 400,
                                 "%s is already in use" % medium,
@@ -715,5 +550,4 @@ def register_servlets(hs, http_server):
     EmailRegisterRequestTokenRestServlet(hs).register(http_server)
     MsisdnRegisterRequestTokenRestServlet(hs).register(http_server)
     UsernameAvailabilityRestServlet(hs).register(http_server)
-    RegistrationSubmitTokenServlet(hs).register(http_server)
     RegisterRestServlet(hs).register(http_server)

synapse/rest/client/versions.py

@@ -24,10 +24,6 @@ logger = logging.getLogger(__name__)
 class VersionsRestServlet(RestServlet):
     PATTERNS = [re.compile("^/_matrix/client/versions$")]
 
-    def __init__(self, hs):
-        super(VersionsRestServlet, self).__init__()
-        self.config = hs.config
-
     def on_GET(self, request):
         return (
             200,
@@ -53,5 +49,5 @@ class VersionsRestServlet(RestServlet):
         )
 
 
-def register_servlets(hs, http_server):
-    VersionsRestServlet(hs).register(http_server)
+def register_servlets(http_server):
+    VersionsRestServlet().register(http_server)

synapse/server.py

@@ -221,7 +221,6 @@ class HomeServer(object):
         self.clock = Clock(reactor)
         self.distributor = Distributor()
         self.ratelimiter = Ratelimiter()
-        self.admin_redaction_ratelimiter = Ratelimiter()
         self.registration_ratelimiter = Ratelimiter()
 
         self.datastore = None
@@ -280,9 +279,6 @@ class HomeServer(object):
     def get_registration_ratelimiter(self):
         return self.registration_ratelimiter
 
-    def get_admin_redaction_ratelimiter(self):
-        return self.admin_redaction_ratelimiter
-
     def build_federation_client(self):
         return FederationClient(self)
 

synapse/static/client/login/js/login.js

@@ -62,7 +62,7 @@ var show_login = function() {
         $("#sso_flow").show();
     }
 
-    if (!matrixLogin.serverAcceptsPassword && !matrixLogin.serverAcceptsCas && !matrixLogin.serverAcceptsSso) {
+    if (!matrixLogin.serverAcceptsPassword && !matrixLogin.serverAcceptsCas) {
         $("#no_login_types").show();
     }
 };

synapse/storage/events.py

@@ -23,7 +23,7 @@ from functools import wraps
 from six import iteritems, text_type
 from six.moves import range
 
-from canonicaljson import encode_canonical_json, json
+from canonicaljson import json
 from prometheus_client import Counter, Histogram
 
 from twisted.internet import defer
@@ -33,7 +33,6 @@ from synapse.api.constants import EventTypes
 from synapse.api.errors import SynapseError
 from synapse.events import EventBase  # noqa: F401
 from synapse.events.snapshot import EventContext  # noqa: F401
-from synapse.events.utils import prune_event_dict
 from synapse.logging.context import PreserveLoggingContext, make_deferred_yieldable
 from synapse.logging.utils import log_function
 from synapse.metrics import BucketCollector
@@ -263,14 +262,6 @@ class EventsStore(
 
         hs.get_clock().looping_call(read_forward_extremities, 60 * 60 * 1000)
 
-        def _censor_redactions():
-            return run_as_background_process(
-                "_censor_redactions", self._censor_redactions
-            )
-
-        if self.hs.config.redaction_retention_period is not None:
-            hs.get_clock().looping_call(_censor_redactions, 5 * 60 * 1000)
-
     @defer.inlineCallbacks
     def _read_forward_extremities(self):
         def fetch(txn):
@@ -1557,98 +1548,6 @@ class EventsStore(
             (event.event_id, event.redacts),
         )
 
-    @defer.inlineCallbacks
-    def _censor_redactions(self):
-        """Censors all redactions older than the configured period that haven't
-        been censored yet.
-
-        By censor we mean update the event_json table with the redacted event.
-
-        Returns:
-            Deferred
-        """
-
-        if self.hs.config.redaction_retention_period is None:
-            return
-
-        max_pos = yield self.find_first_stream_ordering_after_ts(
-            self._clock.time_msec() - self.hs.config.redaction_retention_period
-        )
-
-        # We fetch all redactions that:
-        #   1. point to an event we have,
-        #   2. has a stream ordering from before the cut off, and
-        #   3. we haven't yet censored.
-        #
-        # This is limited to 100 events to ensure that we don't try and do too
-        # much at once. We'll get called again so this should eventually catch
-        # up.
-        #
-        # We use the range [-max_pos, max_pos] to handle backfilled events,
-        # which are given negative stream ordering.
-        sql = """
-            SELECT redact_event.event_id, redacts FROM redactions
-            INNER JOIN events AS redact_event USING (event_id)
-            INNER JOIN events AS original_event ON (
-                redact_event.room_id = original_event.room_id
-                AND redacts = original_event.event_id
-            )
-            WHERE NOT have_censored
-            AND ? <= redact_event.stream_ordering AND redact_event.stream_ordering <= ?
-            ORDER BY redact_event.stream_ordering ASC
-            LIMIT ?
-        """
-
-        rows = yield self._execute(
-            "_censor_redactions_fetch", None, sql, -max_pos, max_pos, 100
-        )
-
-        updates = []
-
-        for redaction_id, event_id in rows:
-            redaction_event = yield self.get_event(redaction_id, allow_none=True)
-            original_event = yield self.get_event(
-                event_id, allow_rejected=True, allow_none=True
-            )
-
-            # The SQL above ensures that we have both the redaction and
-            # original event, so if the `get_event` calls return None it
-            # means that the redaction wasn't allowed. Either way we know that
-            # the result won't change so we mark the fact that we've checked.
-            if (
-                redaction_event
-                and original_event
-                and original_event.internal_metadata.is_redacted()
-            ):
-                # Redaction was allowed
-                pruned_json = encode_canonical_json(
-                    prune_event_dict(original_event.get_dict())
-                )
-            else:
-                # Redaction wasn't allowed
-                pruned_json = None
-
-            updates.append((redaction_id, event_id, pruned_json))
-
-        def _update_censor_txn(txn):
-            for redaction_id, event_id, pruned_json in updates:
-                if pruned_json:
-                    self._simple_update_one_txn(
-                        txn,
-                        table="event_json",
-                        keyvalues={"event_id": event_id},
-                        updatevalues={"json": pruned_json},
-                    )
-
-                self._simple_update_one_txn(
-                    txn,
-                    table="redactions",
-                    keyvalues={"event_id": redaction_id},
-                    updatevalues={"have_censored": True},
-                )
-
-        yield self.runInteraction("_update_censor_txn", _update_censor_txn)
-
     @defer.inlineCallbacks
     def count_daily_messages(self):
         """

synapse/storage/registration.py

@@ -322,19 +322,6 @@ class RegistrationWorkerStore(SQLBaseStore):
 
         return None
 
-    @cachedInlineCallbacks()
-    def is_real_user(self, user_id):
-        """Determines if the user is a real user, ie does not have a 'user_type'.
-
-        Args:
-            user_id (str): user id to test
-
-        Returns:
-            Deferred[bool]: True if user 'user_type' is null or empty string
-        """
-        res = yield self.runInteraction("is_real_user", self.is_real_user_txn, user_id)
-        return res
-
     @cachedInlineCallbacks()
     def is_support_user(self, user_id):
         """Determines if the user is of type UserTypes.SUPPORT
@@ -350,16 +337,6 @@ class RegistrationWorkerStore(SQLBaseStore):
         )
         return res
 
-    def is_real_user_txn(self, txn, user_id):
-        res = self._simple_select_one_onecol_txn(
-            txn=txn,
-            table="users",
-            keyvalues={"name": user_id},
-            retcol="user_type",
-            allow_none=True,
-        )
-        return res is None
-
     def is_support_user_txn(self, txn, user_id):
         res = self._simple_select_one_onecol_txn(
             txn=txn,
@@ -444,20 +421,6 @@ class RegistrationWorkerStore(SQLBaseStore):
         ret = yield self.runInteraction("count_users", _count_users)
         return ret
 
-    @defer.inlineCallbacks
-    def count_real_users(self):
-        """Counts all users without a special user_type registered on the homeserver."""
-
-        def _count_users(txn):
-            txn.execute("SELECT COUNT(*) AS users FROM users where user_type is null")
-            rows = self.cursor_to_dict(txn)
-            if rows:
-                return rows[0]["users"]
-            return 0
-
-        ret = yield self.runInteraction("count_real_users", _count_users)
-        return ret
-
     @defer.inlineCallbacks
     def find_next_generated_user_id_localpart(self):
         """
@@ -651,85 +614,6 @@ class RegistrationWorkerStore(SQLBaseStore):
         # Convert the integer into a boolean.
         return res == 1
 
-    def get_threepid_validation_session(
-        self, medium, client_secret, address=None, sid=None, validated=True
-    ):
-        """Gets a session_id and last_send_attempt (if available) for a
-        client_secret/medium/(address|session_id) combo
-
-        Args:
-            medium (str|None): The medium of the 3PID
-            address (str|None): The address of the 3PID
-            sid (str|None): The ID of the validation session
-            client_secret (str|None): A unique string provided by the client to
-                help identify this validation attempt
-            validated (bool|None): Whether sessions should be filtered by
-                whether they have been validated already or not. None to
-                perform no filtering
-
-        Returns:
-            deferred {str, int}|None: A dict containing the
-                latest session_id and send_attempt count for this 3PID.
-                Otherwise None if there hasn't been a previous attempt
-        """
-        keyvalues = {"medium": medium, "client_secret": client_secret}
-        if address:
-            keyvalues["address"] = address
-        if sid:
-            keyvalues["session_id"] = sid
-
-        assert address or sid
-
-        def get_threepid_validation_session_txn(txn):
-            sql = """
-                SELECT address, session_id, medium, client_secret,
-                last_send_attempt, validated_at
-                FROM threepid_validation_session WHERE %s
-                """ % (
-                " AND ".join("%s = ?" % k for k in iterkeys(keyvalues)),
-            )
-
-            if validated is not None:
-                sql += " AND validated_at IS " + ("NOT NULL" if validated else "NULL")
-
-            sql += " LIMIT 1"
-
-            txn.execute(sql, list(keyvalues.values()))
-            rows = self.cursor_to_dict(txn)
-            if not rows:
-                return None
-
-            return rows[0]
-
-        return self.runInteraction(
-            "get_threepid_validation_session", get_threepid_validation_session_txn
-        )
-
-    def delete_threepid_session(self, session_id):
-        """Removes a threepid validation session from the database. This can
-        be done after validation has been performed and whatever action was
-        waiting on it has been carried out
-
-        Args:
-            session_id (str): The ID of the session to delete
-        """
-
-        def delete_threepid_session_txn(txn):
-            self._simple_delete_txn(
-                txn,
-                table="threepid_validation_token",
-                keyvalues={"session_id": session_id},
-            )
-            self._simple_delete_txn(
-                txn,
-                table="threepid_validation_session",
-                keyvalues={"session_id": session_id},
-            )
-
-        return self.runInteraction(
-            "delete_threepid_session", delete_threepid_session_txn
-        )
-
 
 class RegistrationStore(
     RegistrationWorkerStore, background_updates.BackgroundUpdateStore
@@ -1198,6 +1082,60 @@ class RegistrationStore(
 
         return 1
 
+    def get_threepid_validation_session(
+        self, medium, client_secret, address=None, sid=None, validated=True
+    ):
+        """Gets a session_id and last_send_attempt (if available) for a
+        client_secret/medium/(address|session_id) combo
+
+        Args:
+            medium (str|None): The medium of the 3PID
+            address (str|None): The address of the 3PID
+            sid (str|None): The ID of the validation session
+            client_secret (str|None): A unique string provided by the client to
+                help identify this validation attempt
+            validated (bool|None): Whether sessions should be filtered by
+                whether they have been validated already or not. None to
+                perform no filtering
+
+        Returns:
+            deferred {str, int}|None: A dict containing the
+                latest session_id and send_attempt count for this 3PID.
+                Otherwise None if there hasn't been a previous attempt
+        """
+        keyvalues = {"medium": medium, "client_secret": client_secret}
+        if address:
+            keyvalues["address"] = address
+        if sid:
+            keyvalues["session_id"] = sid
+
+        assert address or sid
+
+        def get_threepid_validation_session_txn(txn):
+            sql = """
+                SELECT address, session_id, medium, client_secret,
+                last_send_attempt, validated_at
+                FROM threepid_validation_session WHERE %s
+                """ % (
+                " AND ".join("%s = ?" % k for k in iterkeys(keyvalues)),
+            )
+
+            if validated is not None:
+                sql += " AND validated_at IS " + ("NOT NULL" if validated else "NULL")
+
+            sql += " LIMIT 1"
+
+            txn.execute(sql, list(keyvalues.values()))
+            rows = self.cursor_to_dict(txn)
+            if not rows:
+                return None
+
+            return rows[0]
+
+        return self.runInteraction(
+            "get_threepid_validation_session", get_threepid_validation_session_txn
+        )
+
     def validate_threepid_session(self, session_id, client_secret, token, current_ts):
         """Attempt to validate a threepid session using a token
 
@@ -1385,6 +1323,31 @@ class RegistrationStore(
             self.clock.time_msec(),
         )
 
+    def delete_threepid_session(self, session_id):
+        """Removes a threepid validation session from the database. This can
+        be done after validation has been performed and whatever action was
+        waiting on it has been carried out
+
+        Args:
+            session_id (str): The ID of the session to delete
+        """
+
+        def delete_threepid_session_txn(txn):
+            self._simple_delete_txn(
+                txn,
+                table="threepid_validation_token",
+                keyvalues={"session_id": session_id},
+            )
+            self._simple_delete_txn(
+                txn,
+                table="threepid_validation_session",
+                keyvalues={"session_id": session_id},
+            )
+
+        return self.runInteraction(
+            "delete_threepid_session", delete_threepid_session_txn
+        )
+
     def set_user_deactivated_status_txn(self, txn, user_id, deactivated):
         self._simple_update_one_txn(
             txn=txn,

synapse/storage/roommember.py

@@ -24,10 +24,8 @@ from canonicaljson import json
 from twisted.internet import defer
 
 from synapse.api.constants import EventTypes, Membership
-from synapse.metrics import LaterGauge
 from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.storage._base import LoggingTransaction
-from synapse.storage.engines import Sqlite3Engine
 from synapse.storage.events_worker import EventsWorkerStore
 from synapse.types import get_domain_from_id
 from synapse.util.async_helpers import Linearizer
@@ -76,63 +74,6 @@ class RoomMemberWorkerStore(EventsWorkerStore):
         self._check_safe_current_state_events_membership_updated_txn(txn)
         txn.close()
 
-        if self.hs.config.metrics_flags.known_servers:
-            self._known_servers_count = 1
-            self.hs.get_clock().looping_call(
-                run_as_background_process,
-                60 * 1000,
-                "_count_known_servers",
-                self._count_known_servers,
-            )
-            self.hs.get_clock().call_later(
-                1000,
-                run_as_background_process,
-                "_count_known_servers",
-                self._count_known_servers,
-            )
-            LaterGauge(
-                "synapse_federation_known_servers",
-                "",
-                [],
-                lambda: self._known_servers_count,
-            )
-
-    @defer.inlineCallbacks
-    def _count_known_servers(self):
-        """
-        Count the servers that this server knows about.
-
-        The statistic is stored on the class for the
-        `synapse_federation_known_servers` LaterGauge to collect.
-        """
-
-        def _transact(txn):
-            if isinstance(self.database_engine, Sqlite3Engine):
-                query = """
-                    SELECT COUNT(DISTINCT substr(out.user_id, pos+1))
-                    FROM (
-                        SELECT rm.user_id as user_id, instr(rm.user_id, ':')
-                            AS pos FROM room_memberships as rm
-                        INNER JOIN current_state_events as c ON rm.event_id = c.event_id
-                        WHERE c.type = 'm.room.member'
-                    ) as out
-                """
-            else:
-                query = """
-                    SELECT COUNT(DISTINCT split_part(state_key, ':', 2))
-                    FROM current_state_events
-                    WHERE type = 'm.room.member' AND membership = 'join';
-                """
-            txn.execute(query)
-            return list(txn)[0][0]
-
-        count = yield self.runInteraction("get_known_servers", _transact)
-
-        # We always know about ourselves, even if we have nothing in
-        # room_memberships (for example, the server is new).
-        self._known_servers_count = max([count, 1])
-        return self._known_servers_count
-
     def _check_safe_current_state_events_membership_updated_txn(self, txn):
         """Checks if it is safe to assume the new current_state_events
         membership column is up to date

synapse/storage/schema/delta/56/redaction_censor.sql

@@ -1,17 +0,0 @@
-/* Copyright 2019 The Matrix.org Foundation C.I.C.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-ALTER TABLE redactions ADD COLUMN have_censored BOOL NOT NULL DEFAULT false;
-CREATE INDEX redactions_have_censored ON redactions(event_id) WHERE not have_censored;

synapse/storage/stats.py

@@ -823,9 +823,7 @@ class StatsStore(StateDeltasStore):
             elif event.type == EventTypes.CanonicalAlias:
                 room_state["canonical_alias"] = event.content.get("alias")
             elif event.type == EventTypes.Create:
-                room_state["is_federatable"] = (
-                    event.content.get("m.federate", True) is True
-                )
+                room_state["is_federatable"] = event.content.get("m.federate", True)
 
         yield self.update_room_state(room_id, room_state)
 

synapse/storage/transactions.py

@@ -250,6 +250,26 @@ class TransactionStore(SQLBaseStore):
                 },
             )
 
+    def get_destinations_needing_retry(self):
+        """Get all destinations which are due a retry for sending a transaction.
+
+        Returns:
+            list: A list of dicts
+        """
+
+        return self.runInteraction(
+            "get_destinations_needing_retry", self._get_destinations_needing_retry
+        )
+
+    def _get_destinations_needing_retry(self, txn):
+        query = (
+            "SELECT * FROM destinations"
+            " WHERE retry_last_ts > 0 and retry_next_ts < ?"
+        )
+
+        txn.execute(query, (self._clock.time_msec(),))
+        return self.cursor_to_dict(txn)
+
     def _start_cleanup_transactions(self):
         return run_as_background_process(
             "cleanup_transactions", self._cleanup_transactions

synapse/util/hash.py

@@ -1,33 +0,0 @@
-# -*- coding: utf-8 -*-
-
-# Copyright 2019 The Matrix.org Foundation C.I.C.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import hashlib
-
-import unpaddedbase64
-
-
-def sha256_and_url_safe_base64(input_text):
-    """SHA256 hash an input string, encode the digest as url-safe base64, and
-    return
-
-    :param input_text: string to hash
-    :type input_text: str
-
-    :returns a sha256 hashed and url-safe base64 encoded digest
-    :rtype: str
-    """
-    digest = hashlib.sha256(input_text.encode()).digest()
-    return unpaddedbase64.encode_base64(digest, urlsafe=True)

synapse/util/retryutils.py

@@ -22,15 +22,6 @@ from synapse.api.errors import CodeMessageException
 
 logger = logging.getLogger(__name__)
 
-# the intial backoff, after the first transaction fails
-MIN_RETRY_INTERVAL = 10 * 60 * 1000
-
-# how much we multiply the backoff by after each subsequent fail
-RETRY_MULTIPLIER = 5
-
-# a cap on the backoff. (Essentially none)
-MAX_RETRY_INTERVAL = 2 ** 63
-
 
 class NotRetryingDestination(Exception):
     def __init__(self, retry_last_ts, retry_interval, destination):
@@ -121,6 +112,9 @@ class RetryDestinationLimiter(object):
         clock,
         store,
         retry_interval,
+        min_retry_interval=10 * 60 * 1000,
+        max_retry_interval=24 * 60 * 60 * 1000,
+        multiplier_retry_interval=5,
         backoff_on_404=False,
         backoff_on_failure=True,
     ):
@@ -136,6 +130,12 @@ class RetryDestinationLimiter(object):
             retry_interval (int): The next retry interval taken from the
                 database in milliseconds, or zero if the last request was
                 successful.
+            min_retry_interval (int): The minimum retry interval to use after
+                a failed request, in milliseconds.
+            max_retry_interval (int): The maximum retry interval to use after
+                a failed request, in milliseconds.
+            multiplier_retry_interval (int): The multiplier to use to increase
+                the retry interval after a failed request.
             backoff_on_404 (bool): Back off if we get a 404
 
             backoff_on_failure (bool): set to False if we should not increase the
@@ -146,6 +146,9 @@ class RetryDestinationLimiter(object):
         self.destination = destination
 
         self.retry_interval = retry_interval
+        self.min_retry_interval = min_retry_interval
+        self.max_retry_interval = max_retry_interval
+        self.multiplier_retry_interval = multiplier_retry_interval
         self.backoff_on_404 = backoff_on_404
         self.backoff_on_failure = backoff_on_failure
 
@@ -193,14 +196,13 @@ class RetryDestinationLimiter(object):
         else:
             # We couldn't connect.
             if self.retry_interval:
-                self.retry_interval = int(
-                    self.retry_interval * RETRY_MULTIPLIER * random.uniform(0.8, 1.4)
-                )
+                self.retry_interval *= self.multiplier_retry_interval
+                self.retry_interval *= int(random.uniform(0.8, 1.4))
 
-                if self.retry_interval >= MAX_RETRY_INTERVAL:
-                    self.retry_interval = MAX_RETRY_INTERVAL
+                if self.retry_interval >= self.max_retry_interval:
+                    self.retry_interval = self.max_retry_interval
             else:
-                self.retry_interval = MIN_RETRY_INTERVAL
+                self.retry_interval = self.min_retry_interval
 
             logger.info(
                 "Connection to %s was unsuccessful (%s(%s)); backoff now %i",

synctl

@@ -71,7 +71,20 @@ def abort(message, colour=RED, stream=sys.stderr):
     sys.exit(1)
 
 
-def start(configfile, daemonize=True):
+def start(configfile: str, daemonize: bool = True) -> bool:
+    """Attempts to start synapse.
+    Args:
+        configfile: path to a yaml synapse config file
+        daemonize: whether to daemonize synapse or keep it attached to the current
+            session
+
+    Returns:
+        True if the process started successfully
+        False if there was an error starting the process
+
+        If deamonize is False it will only return once synapse exits.
+    """
+
     write("Starting ...")
     args = SYNAPSE
 
@@ -83,25 +96,40 @@ def start(configfile, daemonize=True):
     try:
         subprocess.check_call(args)
         write("started synapse.app.homeserver(%r)" % (configfile,), colour=GREEN)
+        return True
     except subprocess.CalledProcessError as e:
         write(
             "error starting (exit code: %d); see above for logs" % e.returncode,
             colour=RED,
         )
+        return False
 
 
-def start_worker(app, configfile, worker_configfile):
+def start_worker(app: str, configfile: str, worker_configfile: str) -> bool:
+    """Attempts to start a synapse worker.
+    Args:
+        app: name of the worker's appservice
+        configfile: path to a yaml synapse config file
+        worker_configfile: path to worker specific yaml synapse file
+
+    Returns:
+        True if the process started successfully
+        False if there was an error starting the process
+    """
+
     args = [sys.executable, "-B", "-m", app, "-c", configfile, "-c", worker_configfile]
 
     try:
         subprocess.check_call(args)
         write("started %s(%r)" % (app, worker_configfile), colour=GREEN)
+        return True
     except subprocess.CalledProcessError as e:
         write(
             "error starting %s(%r) (exit code: %d); see above for logs"
             % (app, worker_configfile, e.returncode),
             colour=RED,
         )
+        return False
 
 
 def stop(pidfile, app):
@@ -292,11 +320,14 @@ def main():
             write("All processes exited; now restarting...")
 
     if action == "start" or action == "restart":
+        error = False
         if start_stop_synapse:
             # Check if synapse is already running
             if os.path.exists(pidfile) and pid_running(int(open(pidfile).read())):
                 abort("synapse.app.homeserver already running")
-            start(configfile, bool(options.daemonize))
+
+            if not start(configfile, bool(options.daemonize)):
+                error = True
 
         for worker in workers:
             env = os.environ.copy()
@@ -307,12 +338,16 @@ def main():
             for cache_name, factor in iteritems(worker.cache_factors):
                 os.environ["SYNAPSE_CACHE_FACTOR_" + cache_name.upper()] = str(factor)
 
-            start_worker(worker.app, configfile, worker.configfile)
+            if not start_worker(worker.app, configfile, worker.configfile):
+                error = True
 
             # Reset env back to the original
             os.environ.clear()
             os.environ.update(env)
 
+        if error:
+            exit(1)
+
 
 if __name__ == "__main__":
     main()

tests/api/test_auth.py

@@ -21,7 +21,6 @@ from twisted.internet import defer
 
 import synapse.handlers.auth
 from synapse.api.auth import Auth
-from synapse.api.constants import UserTypes
 from synapse.api.errors import (
     AuthError,
     Codes,
@@ -336,23 +335,6 @@ class AuthTestCase(unittest.TestCase):
         )
         yield self.auth.check_auth_blocking()
 
-    @defer.inlineCallbacks
-    def test_blocking_mau__depending_on_user_type(self):
-        self.hs.config.max_mau_value = 50
-        self.hs.config.limit_usage_by_mau = True
-
-        self.store.get_monthly_active_count = Mock(return_value=defer.succeed(100))
-        # Support users allowed
-        yield self.auth.check_auth_blocking(user_type=UserTypes.SUPPORT)
-        self.store.get_monthly_active_count = Mock(return_value=defer.succeed(100))
-        # Bots not allowed
-        with self.assertRaises(ResourceLimitError):
-            yield self.auth.check_auth_blocking(user_type=UserTypes.BOT)
-        self.store.get_monthly_active_count = Mock(return_value=defer.succeed(100))
-        # Real users not allowed
-        with self.assertRaises(ResourceLimitError):
-            yield self.auth.check_auth_blocking()
-
     @defer.inlineCallbacks
     def test_reserved_threepid(self):
         self.hs.config.limit_usage_by_mau = True

tests/config/test_generate.py

@@ -17,8 +17,6 @@ import os.path
 import re
 import shutil
 import tempfile
-from contextlib import redirect_stdout
-from io import StringIO
 
 from synapse.config.homeserver import HomeServerConfig
 
@@ -34,18 +32,17 @@ class ConfigGenerationTestCase(unittest.TestCase):
         shutil.rmtree(self.dir)
 
     def test_generate_config_generates_files(self):
-        with redirect_stdout(StringIO()):
-            HomeServerConfig.load_or_generate_config(
-                "",
-                [
-                    "--generate-config",
-                    "-c",
-                    self.file,
-                    "--report-stats=yes",
-                    "-H",
-                    "lemurs.win",
-                ],
-            )
+        HomeServerConfig.load_or_generate_config(
+            "",
+            [
+                "--generate-config",
+                "-c",
+                self.file,
+                "--report-stats=yes",
+                "-H",
+                "lemurs.win",
+            ],
+        )
 
         self.assertSetEqual(
             set(["homeserver.yaml", "lemurs.win.log.config", "lemurs.win.signing.key"]),

tests/config/test_load.py

@@ -15,8 +15,6 @@
 import os.path
 import shutil
 import tempfile
-from contextlib import redirect_stdout
-from io import StringIO
 
 import yaml
 
@@ -28,6 +26,7 @@ from tests import unittest
 class ConfigLoadingTestCase(unittest.TestCase):
     def setUp(self):
         self.dir = tempfile.mkdtemp()
+        print(self.dir)
         self.file = os.path.join(self.dir, "homeserver.yaml")
 
     def tearDown(self):
@@ -95,27 +94,18 @@ class ConfigLoadingTestCase(unittest.TestCase):
         )
         self.assertTrue(config.enable_registration)
 
-    def test_stats_enabled(self):
-        self.generate_config_and_remove_lines_containing("enable_metrics")
-        self.add_lines_to_config(["enable_metrics: true"])
-
-        # The default Metrics Flags are off by default.
-        config = HomeServerConfig.load_config("", ["-c", self.file])
-        self.assertFalse(config.metrics_flags.known_servers)
-
     def generate_config(self):
-        with redirect_stdout(StringIO()):
-            HomeServerConfig.load_or_generate_config(
-                "",
-                [
-                    "--generate-config",
-                    "-c",
-                    self.file,
-                    "--report-stats=yes",
-                    "-H",
-                    "lemurs.win",
-                ],
-            )
+        HomeServerConfig.load_or_generate_config(
+            "",
+            [
+                "--generate-config",
+                "-c",
+                self.file,
+                "--report-stats=yes",
+                "-H",
+                "lemurs.win",
+            ],
+        )
 
     def generate_config_and_remove_lines_containing(self, needle):
         self.generate_config()

tests/handlers/test_register.py

@@ -171,11 +171,11 @@ class RegistrationTestCase(unittest.HomeserverTestCase):
         rooms = self.get_success(self.store.get_rooms_for_user(user_id))
         self.assertEqual(len(rooms), 0)
 
-    def test_auto_create_auto_join_rooms_when_user_is_not_a_real_user(self):
+    def test_auto_create_auto_join_rooms_when_support_user_exists(self):
         room_alias_str = "#room:test"
         self.hs.config.auto_join_rooms = [room_alias_str]
 
-        self.store.is_real_user = Mock(return_value=False)
+        self.store.is_support_user = Mock(return_value=True)
         user_id = self.get_success(self.handler.register_user(localpart="support"))
         rooms = self.get_success(self.store.get_rooms_for_user(user_id))
         self.assertEqual(len(rooms), 0)
@@ -183,31 +183,6 @@ class RegistrationTestCase(unittest.HomeserverTestCase):
         room_alias = RoomAlias.from_string(room_alias_str)
         self.get_failure(directory_handler.get_association(room_alias), SynapseError)
 
-    def test_auto_create_auto_join_rooms_when_user_is_the_first_real_user(self):
-        room_alias_str = "#room:test"
-        self.hs.config.auto_join_rooms = [room_alias_str]
-
-        self.store.count_real_users = Mock(return_value=1)
-        self.store.is_real_user = Mock(return_value=True)
-        user_id = self.get_success(self.handler.register_user(localpart="real"))
-        rooms = self.get_success(self.store.get_rooms_for_user(user_id))
-        directory_handler = self.hs.get_handlers().directory_handler
-        room_alias = RoomAlias.from_string(room_alias_str)
-        room_id = self.get_success(directory_handler.get_association(room_alias))
-
-        self.assertTrue(room_id["room_id"] in rooms)
-        self.assertEqual(len(rooms), 1)
-
-    def test_auto_create_auto_join_rooms_when_user_is_not_the_first_real_user(self):
-        room_alias_str = "#room:test"
-        self.hs.config.auto_join_rooms = [room_alias_str]
-
-        self.store.count_real_users = Mock(return_value=2)
-        self.store.is_real_user = Mock(return_value=True)
-        user_id = self.get_success(self.handler.register_user(localpart="real"))
-        rooms = self.get_success(self.store.get_rooms_for_user(user_id))
-        self.assertEqual(len(rooms), 0)
-
     def test_auto_create_auto_join_where_no_consent(self):
         """Test to ensure that the first user is not auto-joined to a room if
         they have not given general consent.

tests/logging/test_structured.py

@@ -13,7 +13,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import logging
 import os
 import os.path
 import shutil
@@ -34,20 +33,7 @@ class FakeBeginner(object):
         self.observers = observers
 
 
-class StructuredLoggingTestBase(object):
-    """
-    Test base that registers a cleanup handler to reset the stdlib log handler
-    to 'unset'.
-    """
-
-    def prepare(self, reactor, clock, hs):
-        def _cleanup():
-            logging.getLogger("synapse").setLevel(logging.NOTSET)
-
-        self.addCleanup(_cleanup)
-
-
-class StructuredLoggingTestCase(StructuredLoggingTestBase, HomeserverTestCase):
+class StructuredLoggingTestCase(HomeserverTestCase):
     """
     Tests for Synapse's structured logging support.
     """
@@ -153,9 +139,7 @@ class StructuredLoggingTestCase(StructuredLoggingTestBase, HomeserverTestCase):
         self.assertEqual(logs[0]["request"], "somereq")
 
 
-class StructuredLoggingConfigurationFileTestCase(
-    StructuredLoggingTestBase, HomeserverTestCase
-):
+class StructuredLoggingConfigurationFileTestCase(HomeserverTestCase):
     def make_homeserver(self, reactor, clock):
 
         tempdir = self.mktemp()
@@ -195,11 +179,10 @@ class StructuredLoggingConfigurationFileTestCase(
         """
         When a structured logging config is given, Synapse will use it.
         """
-        beginner = FakeBeginner()
-        publisher = setup_logging(self.hs, self.hs.config, logBeginner=beginner)
+        setup_logging(self.hs, self.hs.config)
 
         # Make a logger and send an event
-        logger = Logger(namespace="tests.logging.test_structured", observer=publisher)
+        logger = Logger(namespace="tests.logging.test_structured")
 
         with LoggingContext("testcontext", request="somereq"):
             logger.info("Hello there, {name}!", name="steve")