Merge pull request #5946 from matrix-org/rei/rss_inc8

Separated Statistics [8/7ish EOF]

INSTALL.md

@@ -36,7 +36,7 @@ that your email address is probably `user@example.com` rather than
 System requirements:
 
 - POSIX-compliant system (tested on Linux & OS X)
-- Python 3.5, 3.6, or 3.7
+- Python 3.5, 3.6, 3.7, or 2.7
 - At least 1GB of free RAM if you want to join large public rooms like #matrix:matrix.org
 
 Synapse is written in Python but some of the libraries it uses are written in
@@ -421,7 +421,7 @@ If Synapse is not configured with an SMTP server, password reset via email will
 
 The easiest way to create a new user is to do so from a client like [Riot](https://riot.im).
 
-Alternatively you can do so from the command line if you have installed via pip.
+Alternatively you can do so from the command line if you have installed via pip. 
 
 This can be done as follows:
 

MANIFEST.in

@@ -38,16 +38,14 @@ exclude sytest-blacklist
 include pyproject.toml
 recursive-include changelog.d *
 
-prune .buildkite
-prune .circleci
-prune .codecov.yml
-prune .coveragerc
 prune .github
-prune debian
 prune demo/etc
 prune docker
-prune mypy.ini
-prune stubs
+prune .circleci
+prune .coveragerc
+prune debian
+prune .codecov.yml
+prune .buildkite
 
 exclude jenkins*
 recursive-exclude jenkins *.sh

UPGRADE.rst

@@ -49,56 +49,6 @@ returned by the Client-Server API:
     # configured on port 443.
     curl -kv https://<host.name>/_matrix/client/versions 2>&1 | grep "Server:"
 
-Upgrading to v1.4.0
-===================
-
-Config options
---------------
-
-**Note: Registration by email address or phone number will not work in this release unless
-some config options are changed from their defaults.**
-
-This is due to Synapse v1.4.0 now defaulting to sending registration and password reset tokens
-itself. This is for security reasons as well as putting less reliance on identity servers.
-However, currently Synapse only supports sending emails, and does not have support for
-phone-based password reset or account registration. If Synapse is configured to handle these on
-its own, phone-based password resets and registration will be disabled. For Synapse to send
-emails, the ``email`` block of the config must be filled out. If not, then password resets and
-registration via email will be disabled entirely.
-
-This release also deprecates the ``email.trust_identity_server_for_password_resets`` option and
-replaces it with the ``account_threepid_delegates`` dictionary. This option defines whether the
-homeserver should delegate an external server (typically an `identity server
-<https://matrix.org/docs/spec/identity_service/r0.2.1>`_) to handle sending password reset or
-registration messages via email and SMS.
-
-If ``email.trust_identity_server_for_password_resets`` is set to ``true``, and
-``account_threepid_delegates.email`` is not set, then the first entry in
-``trusted_third_party_id_servers`` will be used as the account threepid delegate for email.
-This is to ensure compatibility with existing Synapse installs that set up external server
-handling for these tasks before v1.4.0. If ``email.trust_identity_server_for_password_resets``
-is ``true`` and no trusted identity server domains are configured, Synapse will throw an error.
-
-If ``email.trust_identity_server_for_password_resets`` is ``false`` or absent and a threepid
-type in ``account_threepid_delegates`` is not set to a domain, then Synapse will attempt to
-send password reset and registration messages for that type.
-
-Email templates
----------------
-
-If you have configured a custom template directory with the ``email.template_dir`` option, be
-aware that there are new templates regarding registration. ``registration.html`` and
-``registration.txt`` have been added and contain the content that is sent to a client upon
-registering via an email address.
-
-``registration_success.html`` and ``registration_failure.html`` are also new HTML templates
-that will be shown to the user when they click the link in their registration emai , either
-showing them a success or failure page (assuming a redirect URL is not configured).
-
-Synapse will expect these files to exist inside the configured template directory. To view the
-default templates, see `synapse/res/templates
-<https://github.com/matrix-org/synapse/tree/master/synapse/res/templates>`_.
-
 Upgrading to v1.2.0
 ===================
 
@@ -182,19 +132,6 @@ server for password resets, set ``trust_identity_server_for_password_resets`` to
 See the `sample configuration file <docs/sample_config.yaml>`_
 for more details on these settings.
 
-New email templates
----------------
-Some new templates have been added to the default template directory for the purpose of the
-homeserver sending its own password reset emails. If you have configured a custom
-``template_dir`` in your Synapse config, these files will need to be added.
-
-``password_reset.html`` and ``password_reset.txt`` are HTML and plain text templates
-respectively that contain the contents of what will be emailed to the user upon attempting to
-reset their password via email. ``password_reset_success.html`` and
-``password_reset_failure.html`` are HTML files that the content of which (assuming no redirect
-URL is set) will be shown to the user after they attempt to click the link in the email sent
-to them.
-
 Upgrading to v0.99.0
 ====================
 

changelog.d/5835.feature

@@ -1 +0,0 @@
-Add the ability to send registration emails from the homeserver rather than delegating to an identity server.

changelog.d/5853.feature

@@ -1 +0,0 @@
-Opentracing for device list updates.

changelog.d/5868.feature

@@ -1 +0,0 @@
-Add `m.require_identity_server` key to `/versions`'s `unstable_features` section.

changelog.d/5875.misc

@@ -1 +0,0 @@
-Deprecate the `trusted_third_party_id_servers` option.

changelog.d/5876.feature

@@ -1 +0,0 @@
-Replace `trust_identity_server_for_password_resets` config option with `account_threepid_delegates`.

changelog.d/5879.misc

@@ -0,0 +1 @@
+Rework room and user statistics to separate current & historical rows, as well as track stats correctly.

changelog.d/5892.misc

@@ -1 +0,0 @@
-Compatibility with v2 Identity Service APIs other than /lookup.

changelog.d/5897.feature

@@ -1 +0,0 @@
-Switch to using the v2 Identity Service `/lookup` API where available, with fallback to v1. (Implements [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134) plus id_access_token authentication for v2 Identity Service APIs from [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140)).

changelog.d/5915.bugfix

@@ -1 +0,0 @@
-Fix 404 for thumbnail download when `dynamic_thumbnails` is `false` and the thumbnail was dynamically generated. Fix reported by rkfg.

changelog.d/5934.feature

@@ -1 +0,0 @@
-Redact events in the database that have been redacted for a month.

changelog.d/5940.feature

@@ -1 +0,0 @@
-Add the ability to send registration emails from the homeserver rather than delegating to an identity server.

changelog.d/5953.misc

@@ -1 +0,0 @@
-Update INSTALL.md to say that Python 2 is no longer supported.

changelog.d/5962.misc

@@ -1 +0,0 @@
-Remove unnecessary return statements in the codebase which were the result of a regex run.

changelog.d/5963.misc

@@ -1 +0,0 @@
-Remove left-over methods from C/S registration API.

changelog.d/5964.feature

@@ -1 +0,0 @@
-Remove `bind_email` and `bind_msisdn` parameters from /register ala MSC2140.

changelog.d/5966.bugfix

@@ -1 +0,0 @@
-Fix admin API for listing media in a room not being available with an external media repo.

changelog.d/5967.bugfix

@@ -1 +0,0 @@
-Fix list media admin API always returning an error.

changelog.d/5969.feature

@@ -1 +0,0 @@
-Replace `trust_identity_server_for_password_resets` config option with `account_threepid_delegates`.

changelog.d/5970.docker

@@ -1 +0,0 @@
-Avoid changing UID/GID if they are already correct.

changelog.d/5971.bugfix

@@ -1 +0,0 @@
-Fix room and user stats tracking.

changelog.d/5975.misc

@@ -1 +0,0 @@
-Cleanup event auth type initialisation.

changelog.d/5980.feature

@@ -1 +0,0 @@
-Add POST /_matrix/client/r0/account/3pid/unbind endpoint from MSC2140 for unbinding a 3PID from an identity server without removing it from the homeserver user account.

changelog.d/5981.feature

@@ -1 +0,0 @@
-Setting metrics_flags.known_servers to True in the configuration will publish the synapse_federation_known_servers metric over Prometheus. This represents the total number of servers your server knows about (i.e. is in rooms with), including itself.

changelog.d/5982.bugfix

@@ -1 +0,0 @@
-Include missing opentracing contexts in outbout replication requests.

changelog.d/5983.feature

@@ -1 +0,0 @@
-Add minimum opentracing for client servlets.

changelog.d/5984.bugfix

@@ -1 +0,0 @@
-Fix sending of EDUs when opentracing is enabled with an empty whitelist.

changelog.d/5985.feature

@@ -1 +0,0 @@
-Check at setup that opentracing is installed if it's enabled in the config.

changelog.d/5986.feature

@@ -1 +0,0 @@
-Trace replication send times.

changelog.d/5988.bugfix

@@ -1 +0,0 @@
-Fix invalid references to None while opentracing if the log context slips.

changelog.d/5989.misc

@@ -1 +0,0 @@
-Clean up dependency checking at setup.

changelog.d/5991.bugfix

@@ -1 +0,0 @@
-Fix invalid references to None while opentracing if the log context slips.

changelog.d/5993.feature

@@ -1 +0,0 @@
-Add the ability to send registration emails from the homeserver rather than delegating to an identity server.

changelog.d/5994.feature

@@ -1 +0,0 @@
-Add the ability to send registration emails from the homeserver rather than delegating to an identity server.

changelog.d/5995.bugfix

@@ -1 +0,0 @@
-Return a M_MISSING_PARAM if `sid` is not provided to `/account/3pid`.

changelog.d/5998.bugfix

@@ -1 +0,0 @@
-Fix room and user stats tracking.

changelog.d/6003.misc

@@ -1 +0,0 @@
-Add opentracing span over HTTP push processing.

changelog.d/6004.bugfix

@@ -1 +0,0 @@
-Only count real users when checking for auto-creation of auto-join room.

changelog.d/6005.feature

@@ -1 +0,0 @@
-The new Prometheus metric `synapse_build_info` exposes the Python version, OS version, and Synapse version of the running server.

changelog.d/6009.misc

@@ -1 +0,0 @@
-Small refactor of function arguments and docstrings in RoomMemberHandler.

changelog.d/6010.misc

@@ -1 +0,0 @@
-Remove unused `origin` argument on FederationHandler.add_display_name_to_third_party_invite.

changelog.d/6011.feature

@@ -1 +0,0 @@
-Use account_threepid_delegate.email and account_threepid_delegate.msisdn for validating threepid sessions.

changelog.d/6012.feature

@@ -1 +0,0 @@
-Add report_stats_endpoint option to configure where stats are reported to, if enabled. Contributed by @Sorunome.

changelog.d/6013.misc

@@ -1 +0,0 @@
-Compatibility with v2 Identity Service APIs other than /lookup.

changelog.d/6015.feature

@@ -1 +0,0 @@
-Add config option to increase ratelimits for room admins redacting messages.

changelog.d/6017.misc

@@ -1 +0,0 @@
-Clean up some code in the retry logic.

changelog.d/6020.bugfix

@@ -1 +0,0 @@
-Ensure support users can be registered even if MAU limit is reached.

changelog.d/6023.misc

@@ -1 +0,0 @@
-Fix the structured logging tests stomping on the global log configuration for subsequent tests.

changelog.d/6024.bugfix

@@ -1 +0,0 @@
-Fix bug where login error was shown incorrectly on SSO fallback login.

changelog.d/6025.bugfix

@@ -1 +0,0 @@
-Fix bug in calculating the federation retry backoff period.

changelog.d/6026.feature

@@ -1 +0,0 @@
-Stop sending federation transactions to servers which have been down for a long time.

changelog.d/6032.misc

@@ -1 +0,0 @@
-Add developer documentation for using SAML2.

contrib/cmdclient/console.py

@@ -37,8 +37,6 @@ from signedjson.sign import verify_signed_json, SignatureVerifyException
 
 CONFIG_JSON = "cmdclient_config.json"
 
-# TODO: The concept of trusted identity servers has been deprecated. This option and checks
-#  should be removed
 TRUSTED_ID_SERVERS = ["localhost:8001"]
 
 
@@ -270,7 +268,6 @@ class SynapseCmd(cmd.Cmd):
 
     @defer.inlineCallbacks
     def _do_emailrequest(self, args):
-        # TODO: Update to use v2 Identity Service API endpoint
         url = (
             self._identityServerUrl()
             + "/_matrix/identity/api/v1/validate/email/requestToken"
@@ -305,7 +302,6 @@ class SynapseCmd(cmd.Cmd):
 
     @defer.inlineCallbacks
     def _do_emailvalidate(self, args):
-        # TODO: Update to use v2 Identity Service API endpoint
         url = (
             self._identityServerUrl()
             + "/_matrix/identity/api/v1/validate/email/submitToken"
@@ -334,7 +330,6 @@ class SynapseCmd(cmd.Cmd):
 
     @defer.inlineCallbacks
     def _do_3pidbind(self, args):
-        # TODO: Update to use v2 Identity Service API endpoint
         url = self._identityServerUrl() + "/_matrix/identity/api/v1/3pid/bind"
 
         json_res = yield self.http_client.do_request(
@@ -403,7 +398,6 @@ class SynapseCmd(cmd.Cmd):
     @defer.inlineCallbacks
     def _do_invite(self, roomid, userstring):
         if not userstring.startswith("@") and self._is_on("complete_usernames"):
-            # TODO: Update to use v2 Identity Service API endpoint
             url = self._identityServerUrl() + "/_matrix/identity/api/v1/lookup"
 
             json_res = yield self.http_client.do_request(
@@ -413,7 +407,6 @@ class SynapseCmd(cmd.Cmd):
             mxid = None
 
             if "mxid" in json_res and "signatures" in json_res:
-                # TODO: Update to use v2 Identity Service API endpoint
                 url = (
                     self._identityServerUrl()
                     + "/_matrix/identity/api/v1/pubkey/ed25519"

docker/start.py

@@ -41,8 +41,8 @@ def generate_config_from_template(config_dir, config_path, environ, ownership):
         config_dir (str): where to put generated config files
         config_path (str): where to put the main config file
         environ (dict): environment dictionary
-        ownership (str|None): "<user>:<group>" string which will be used to set
-            ownership of the generated configs. If None, ownership will not change.
+        ownership (str): "<user>:<group>" string which will be used to set
+            ownership of the generated configs
     """
     for v in ("SYNAPSE_SERVER_NAME", "SYNAPSE_REPORT_STATS"):
         if v not in environ:
@@ -105,24 +105,24 @@ def generate_config_from_template(config_dir, config_path, environ, ownership):
     log("Generating log config file " + log_config_file)
     convert("/conf/log.config", log_config_file, environ)
 
+    subprocess.check_output(["chown", "-R", ownership, "/data"])
+
     # Hopefully we already have a signing key, but generate one if not.
-    args = [
-        "python",
-        "-m",
-        "synapse.app.homeserver",
-        "--config-path",
-        config_path,
-        # tell synapse to put generated keys in /data rather than /compiled
-        "--keys-directory",
-        config_dir,
-        "--generate-keys",
-    ]
-
-    if ownership is not None:
-        subprocess.check_output(["chown", "-R", ownership, "/data"])
-        args = ["su-exec", ownership] + args
-
-    subprocess.check_output(args)
+    subprocess.check_output(
+        [
+            "su-exec",
+            ownership,
+            "python",
+            "-m",
+            "synapse.app.homeserver",
+            "--config-path",
+            config_path,
+            # tell synapse to put generated keys in /data rather than /compiled
+            "--keys-directory",
+            config_dir,
+            "--generate-keys",
+        ]
+    )
 
 
 def run_generate_config(environ, ownership):
@@ -130,7 +130,7 @@ def run_generate_config(environ, ownership):
 
     Args:
         environ (dict): env var dict
-        ownership (str|None): "userid:groupid" arg for chmod. If None, ownership will not change.
+        ownership (str): "userid:groupid" arg for chmod
 
     Never returns.
     """
@@ -149,6 +149,9 @@ def run_generate_config(environ, ownership):
         log("Creating log config %s" % (log_config_file,))
         convert("/conf/log.config", log_config_file, environ)
 
+    # make sure that synapse has perms to write to the data dir.
+    subprocess.check_output(["chown", ownership, data_dir])
+
     args = [
         "python",
         "-m",
@@ -167,33 +170,12 @@ def run_generate_config(environ, ownership):
         "--open-private-ports",
     ]
     # log("running %s" % (args, ))
-
-    if ownership is not None:
-        args = ["su-exec", ownership] + args
-        os.execv("/sbin/su-exec", args)
-
-        # make sure that synapse has perms to write to the data dir.
-        subprocess.check_output(["chown", ownership, data_dir])
-    else:
-        os.execv("/usr/local/bin/python", args)
+    os.execv("/usr/local/bin/python", args)
 
 
 def main(args, environ):
     mode = args[1] if len(args) > 1 else None
-    desired_uid = int(environ.get("UID", "991"))
-    desired_gid = int(environ.get("GID", "991"))
-    if (desired_uid == os.getuid()) and (desired_gid == os.getgid()):
-        ownership = None
-    else:
-        ownership = "{}:{}".format(desired_uid, desired_gid)
-
-    log(
-        "Container running as UserID %s:%s, ENV (or defaults) requests %s:%s"
-        % (os.getuid(), os.getgid(), desired_uid, desired_gid)
-    )
-
-    if ownership is None:
-        log("Will not perform chmod/su-exec as UserID already matches request")
+    ownership = "{}:{}".format(environ.get("UID", 991), environ.get("GID", 991))
 
     # In generate mode, generate a configuration and missing keys, then exit
     if mode == "generate":
@@ -245,12 +227,16 @@ def main(args, environ):
 
     log("Starting synapse with config file " + config_path)
 
-    args = ["python", "-m", "synapse.app.homeserver", "--config-path", config_path]
-    if ownership is not None:
-        args = ["su-exec", ownership] + args
-        os.execv("/sbin/su-exec", args)
-    else:
-        os.execv("/usr/local/bin/python", args)
+    args = [
+        "su-exec",
+        ownership,
+        "python",
+        "-m",
+        "synapse.app.homeserver",
+        "--config-path",
+        config_path,
+    ]
+    os.execv("/sbin/su-exec", args)
 
 
 if __name__ == "__main__":

docs/dev/saml.md

@@ -1,37 +0,0 @@
-# How to test SAML as a developer without a server
-
-https://capriza.github.io/samling/samling.html (https://github.com/capriza/samling) is a great
-resource for being able to tinker with the SAML options within Synapse without needing to
-deploy and configure a complicated software stack.
-
-To make Synapse (and therefore Riot) use it:
-
-1. Use the samling.html URL above or deploy your own and visit the IdP Metadata tab.
-2. Copy the XML to your clipboard.
-3. On your Synapse server, create a new file `samling.xml` next to your `homeserver.yaml` with
-   the XML from step 2 as the contents.
-4. Edit your `homeserver.yaml` to include:
-   ```yaml
-   saml2_config:
-     sp_config:
-       allow_unknown_attributes: true  # Works around a bug with AVA Hashes: https://github.com/IdentityPython/pysaml2/issues/388
-       metadata:
-         local: ["samling.xml"]   
-   ```
-5. Run `apt-get install xmlsec1` and `pip install --upgrade --force 'pysaml2>=4.5.0'` to ensure
-   the dependencies are installed and ready to go.
-6. Restart Synapse.
-
-Then in Riot:
-
-1. Visit the login page with a Riot pointing at your homeserver.
-2. Click the Single Sign-On button.
-3. On the samling page, enter a Name Identifier and add a SAML Attribute for `uid=your_localpart`.
-   The response must also be signed.
-4. Click "Next".
-5. Click "Post Response" (change nothing).
-6. You should be logged in.
-
-If you try and repeat this process, you may be automatically logged in using the information you
-gave previously. To fix this, open your developer console (`F12` or `Ctrl+Shift+I`) while on the
-samling page and clear the site data. In Chrome, this will be a button on the Application tab.

docs/room_and_user_statistics.md

@@ -2,8 +2,68 @@ Room and User Statistics
 ========================
 
 Synapse maintains room and user statistics (as well as a cache of room state),
-in various tables. These can be used for administrative purposes but are also
-used when generating the public room directory.
+in various tables.
+
+These can be used for administrative purposes but are also used when generating
+the public room directory. If these tables get stale or out of sync (possibly
+after database corruption), you may wish to regenerate them.
+
+
+# Synapse Administrator Documentation
+
+## Various SQL scripts that you may find useful
+
+### Delete stats, including historical stats
+
+```sql
+DELETE FROM room_stats_current;
+DELETE FROM room_stats_historical;
+DELETE FROM user_stats_current;
+DELETE FROM user_stats_historical;
+```
+
+### Regenerate stats (all subjects)
+
+```sql
+BEGIN;
+    DELETE FROM stats_incremental_position;
+    INSERT INTO stats_incremental_position (
+        state_delta_stream_id,
+        total_events_min_stream_ordering,
+        total_events_max_stream_ordering,
+        is_background_contract
+    ) VALUES (NULL, NULL, NULL, FALSE), (NULL, NULL, NULL, TRUE);
+COMMIT;
+
+DELETE FROM room_stats_current;
+DELETE FROM user_stats_current;
+```
+
+then follow the steps below for **'Regenerate stats (missing subjects only)'**
+
+### Regenerate stats (missing subjects only)
+
+```sql
+-- Set up staging tables
+-- we depend on current_state_events_membership because this is used
+-- in our counting.
+INSERT INTO background_updates (update_name, progress_json) VALUES
+    ('populate_stats_prepare', '{}', 'current_state_events_membership');
+
+-- Run through each room and update stats
+INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
+    ('populate_stats_process_rooms', '{}', 'populate_stats_prepare');
+
+-- Run through each user and update stats.
+INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
+    ('populate_stats_process_users', '{}', 'populate_stats_process_rooms');
+
+-- Clean up staging tables
+INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
+    ('populate_stats_cleanup', '{}', 'populate_stats_process_users');
+```
+
+then **restart Synapse**.
 
 
 # Synapse Developer Documentation
@@ -29,6 +89,8 @@ Stats are maintained as time series. There are two kinds of column:
     occurred within the time slice given by `(end_ts − bucket_size)…end_ts`
     or `start_ts…end_ts`. (Imagine a histogram for these values)
 
+Currently, only absolute columns are in use.
+
 Stats are maintained in two tables (for each type): current and historical.
 
 Current stats correspond to the present values. Each subject can only have one
@@ -39,15 +101,26 @@ entries.
 
 ## Concepts around the management of stats
 
-### Current rows
+### current rows
 
 Current rows contain the most up-to-date statistics for a room.
 They only contain absolute columns
 
-### Historical rows
+#### incomplete current rows
+
+There are also **incomplete** current rows, which are current rows that do not
+contain a full count yet – this is because they are waiting for the regeneration
+process to give them an initial count. Incomplete current rows DO NOT contain
+correct and up-to-date values. As such, *incomplete rows are not old-collected*.
+Instead, old incomplete rows will be extended so they are no longer old.
+
+### historical rows
 
 Historical rows can always be considered to be valid for the time slice and
-end time specified.
+end time specified. (This, of course, assumes a lack of defects in the code
+to track the statistics, and assumes integrity of the database).
+
+Even still, there are two considerations that we may need to bear in mind:
 
 * historical rows will not exist for every time slice – they will be omitted
     if there were no changes. In this case, the following assumptions can be
@@ -57,6 +130,7 @@ end time specified.
 * historical rows will not be retained forever – rows older than a configurable
     time will be purged.
 
-#### Purge
+#### purge
 
 The purging of historical rows is not yet implemented.
+

docs/sample_config.yaml

@@ -306,13 +306,6 @@ listeners:
 #
 #allow_per_room_profiles: false
 
-# How long to keep redacted events in unredacted form in the database. After
-# this period redacted events get replaced with their redacted form in the DB.
-#
-# Defaults to `7d`. Set to `null` to disable.
-#
-redaction_retention_period: 7d
-
 
 ## TLS ##
 
@@ -518,9 +511,6 @@ log_config: "CONFDIR/SERVERNAME.log.config"
 #   - one for login that ratelimits login requests based on the account the
 #     client is attempting to log into, based on the amount of failed login
 #     attempts for this account.
-#   - one for ratelimiting redactions by room admins. If this is not explicitly
-#     set then it uses the same ratelimiting as per rc_message. This is useful
-#     to allow room admins to deal with abuse quickly.
 #
 # The defaults are as shown below.
 #
@@ -542,10 +532,6 @@ log_config: "CONFDIR/SERVERNAME.log.config"
 #  failed_attempts:
 #    per_second: 0.17
 #    burst_count: 3
-#
-#rc_admin_redaction:
-#  per_second: 1
-#  burst_count: 50
 
 
 # Ratelimiting settings for incoming federation
@@ -905,42 +891,10 @@ uploads_path: "DATADIR/uploads"
 # Also defines the ID server which will be called when an account is
 # deactivated (one will be picked arbitrarily).
 #
-# Note: This option is deprecated. Since v0.99.4, Synapse has tracked which identity
-# server a 3PID has been bound to. For 3PIDs bound before then, Synapse runs a
-# background migration script, informing itself that the identity server all of its
-# 3PIDs have been bound to is likely one of the below.
-#
-# As of Synapse v1.4.0, all other functionality of this option has been deprecated, and
-# it is now solely used for the purposes of the background migration script, and can be
-# removed once it has run.
 #trusted_third_party_id_servers:
 #  - matrix.org
 #  - vector.im
 
-# Handle threepid (email/phone etc) registration and password resets through a set of
-# *trusted* identity servers. Note that this allows the configured identity server to
-# reset passwords for accounts!
-#
-# Be aware that if `email` is not set, and SMTP options have not been
-# configured in the email config block, registration and user password resets via
-# email will be globally disabled.
-#
-# Additionally, if `msisdn` is not set, registration and password resets via msisdn
-# will be disabled regardless. This is due to Synapse currently not supporting any
-# method of sending SMS messages on its own.
-#
-# To enable using an identity server for operations regarding a particular third-party
-# identifier type, set the value to the URL of that identity server as shown in the
-# examples below.
-#
-# Servers handling the these requests must answer the `/requestToken` endpoints defined
-# by the Matrix Identity Service API specification:
-# https://matrix.org/docs/spec/identity_service/latest
-#
-account_threepid_delegates:
-    #email: https://example.com     # Delegate email sending to matrix.org
-    #msisdn: http://localhost:8090  # Delegate SMS sending to this local process
-
 # Users who register on this homeserver will automatically be joined
 # to these rooms
 #
@@ -972,24 +926,9 @@ account_threepid_delegates:
 #sentry:
 #    dsn: "..."
 
-# Flags to enable Prometheus metrics which are not suitable to be
-# enabled by default, either for performance reasons or limited use.
-#
-metrics_flags:
-    # Publish synapse_federation_known_servers, a g auge of the number of
-    # servers this homeserver knows about, including itself. May cause
-    # performance problems on large homeservers.
-    #
-    #known_servers: true
-
 # Whether or not to report anonymized homeserver usage statistics.
 # report_stats: true|false
 
-# The endpoint to report the anonymized homeserver usage statistics to.
-# Defaults to https://matrix.org/report-usage-stats/push
-#
-#report_stats_endpoint: https://example.com/report-usage-stats/push
-
 
 ## API Configuration ##
 
@@ -1225,6 +1164,19 @@ password_config:
 #   #
 #   riot_base_url: "http://localhost/riot"
 #
+#   # Enable sending password reset emails via the configured, trusted
+#   # identity servers
+#   #
+#   # IMPORTANT! This will give a malicious or overtaken identity server
+#   # the ability to reset passwords for your users! Make absolutely sure
+#   # that you want to do this! It is strongly recommended that password
+#   # reset emails be sent by the homeserver instead
+#   #
+#   # If this option is set to false and SMTP options have not been
+#   # configured, resetting user passwords via email will be disabled
+#   #
+#   #trust_identity_server_for_password_resets: false
+#
 #   # Configure the time that a validation email or text message code
 #   # will expire after sending
 #   #
@@ -1256,22 +1208,11 @@ password_config:
 #   #password_reset_template_html: password_reset.html
 #   #password_reset_template_text: password_reset.txt
 #
-#   # Templates for registration emails sent by the homeserver
-#   #
-#   #registration_template_html: registration.html
-#   #registration_template_text: registration.txt
-#
 #   # Templates for password reset success and failure pages that a user
 #   # will see after attempting to reset their password
 #   #
 #   #password_reset_template_success_html: password_reset_success.html
 #   #password_reset_template_failure_html: password_reset_failure.html
-#
-#   # Templates for registration success and failure pages that a user
-#   # will see after attempting to register using an email or phone
-#   #
-#   #registration_template_success_html: registration_success.html
-#   #registration_template_failure_html: registration_failure.html
 
 
 #password_providers:

mypy.ini

@@ -1,54 +0,0 @@
-[mypy]
-namespace_packages=True
-plugins=mypy_zope:plugin
-follow_imports=skip
-mypy_path=stubs
-
-[mypy-synapse.config.homeserver]
-# this is a mess because of the metaclass shenanigans
-ignore_errors = True
-
-[mypy-zope]
-ignore_missing_imports = True
-
-[mypy-constantly]
-ignore_missing_imports = True
-
-[mypy-twisted.*]
-ignore_missing_imports = True
-
-[mypy-treq.*]
-ignore_missing_imports = True
-
-[mypy-hyperlink]
-ignore_missing_imports = True
-
-[mypy-h11]
-ignore_missing_imports = True
-
-[mypy-opentracing]
-ignore_missing_imports = True
-
-[mypy-OpenSSL]
-ignore_missing_imports = True
-
-[mypy-netaddr]
-ignore_missing_imports = True
-
-[mypy-saml2.*]
-ignore_missing_imports = True
-
-[mypy-unpaddedbase64]
-ignore_missing_imports = True
-
-[mypy-canonicaljson]
-ignore_missing_imports = True
-
-[mypy-jaeger_client]
-ignore_missing_imports = True
-
-[mypy-jsonschema]
-ignore_missing_imports = True
-
-[mypy-signedjson.*]
-ignore_missing_imports = True

synapse/api/auth.py

@@ -25,7 +25,7 @@ from twisted.internet import defer
 import synapse.logging.opentracing as opentracing
 import synapse.types
 from synapse import event_auth
-from synapse.api.constants import EventTypes, JoinRules, Membership, UserTypes
+from synapse.api.constants import EventTypes, JoinRules, Membership
 from synapse.api.errors import (
     AuthError,
     Codes,
@@ -704,12 +704,13 @@ class Auth(object):
                 and visibility.content["history_visibility"] == "world_readable"
             ):
                 return Membership.JOIN, None
+                return
             raise AuthError(
                 403, "Guest access not allowed", errcode=Codes.GUEST_ACCESS_FORBIDDEN
             )
 
     @defer.inlineCallbacks
-    def check_auth_blocking(self, user_id=None, threepid=None, user_type=None):
+    def check_auth_blocking(self, user_id=None, threepid=None):
         """Checks if the user should be rejected for some external reason,
         such as monthly active user limiting or global disable flag
 
@@ -722,9 +723,6 @@ class Auth(object):
                 with a MAU blocked server, normally they would be rejected but their
                 threepid is on the reserved list. user_id and
                 threepid should never be set at the same time.
-
-            user_type(str|None): If present, is used to decide whether to check against
-                certain blocking reasons like MAU.
         """
 
         # Never fail an auth check for the server notices users or support user
@@ -762,10 +760,6 @@ class Auth(object):
                     self.hs.config.mau_limits_reserved_threepids, threepid
                 ):
                     return
-            elif user_type == UserTypes.SUPPORT:
-                # If the user does not exist yet and is of type "support",
-                # allow registration. Support users are excluded from MAU checks.
-                return
             # Else if there is no room in the MAU bucket, bail
             current_mau = yield self.store.get_monthly_active_count()
             if current_mau >= self.hs.config.max_mau_value:

synapse/app/client_reader.py

@@ -119,7 +119,7 @@ class ClientReaderServer(HomeServer):
                     KeyChangesServlet(self).register(resource)
                     VoipRestServlet(self).register(resource)
                     PushRuleRestServlet(self).register(resource)
-                    VersionsRestServlet(self).register(resource)
+                    VersionsRestServlet().register(resource)
 
                     resources.update({"/_matrix/client": resource})
 

synapse/app/homeserver.py

@@ -561,12 +561,10 @@ def run(hs):
 
         stats["database_engine"] = hs.get_datastore().database_engine_name
         stats["database_server_version"] = hs.get_datastore().get_server_version()
-        logger.info(
-            "Reporting stats to %s: %s" % (hs.config.report_stats_endpoint, stats)
-        )
+        logger.info("Reporting stats to matrix.org: %s" % (stats,))
         try:
             yield hs.get_simple_http_client().put_json(
-                hs.config.report_stats_endpoint, stats
+                "https://matrix.org/report-usage-stats/push", stats
             )
         except Exception as e:
             logger.warn("Error reporting stats: %s", e)

synapse/appservice/api.py

@@ -107,6 +107,7 @@ class ApplicationServiceApi(SimpleHttpClient):
         except CodeMessageException as e:
             if e.code == 404:
                 return False
+                return
             logger.warning("query_user to %s received %s", uri, e.code)
         except Exception as ex:
             logger.warning("query_user to %s threw exception %s", uri, ex)
@@ -126,6 +127,7 @@ class ApplicationServiceApi(SimpleHttpClient):
             logger.warning("query_alias to %s received %s", uri, e.code)
             if e.code == 404:
                 return False
+                return
         except Exception as ex:
             logger.warning("query_alias to %s threw exception %s", uri, ex)
         return False
@@ -228,6 +230,7 @@ class ApplicationServiceApi(SimpleHttpClient):
             sent_transactions_counter.labels(service.id).inc()
             sent_events_counter.labels(service.id).inc(len(events))
             return True
+            return
         except CodeMessageException as e:
             logger.warning("push_bulk to %s received %s", uri, e.code)
         except Exception as ex:

synapse/config/emailconfig.py

@@ -20,7 +20,6 @@ from __future__ import print_function
 # This file can't be called email.py because if it is, we cannot:
 import email.utils
 import os
-from enum import Enum
 
 import pkg_resources
 
@@ -75,48 +74,19 @@ class EmailConfig(Config):
             "renew_at"
         )
 
-        self.threepid_behaviour_email = (
-            # Have Synapse handle the email sending if account_threepid_delegates.email
-            # is not defined
-            # msisdn is currently always remote while Synapse does not support any method of
-            # sending SMS messages
-            ThreepidBehaviour.REMOTE
-            if self.account_threepid_delegate_email
-            else ThreepidBehaviour.LOCAL
+        email_trust_identity_server_for_password_resets = email_config.get(
+            "trust_identity_server_for_password_resets", False
         )
-        # Prior to Synapse v1.4.0, there was another option that defined whether Synapse would
-        # use an identity server to password reset tokens on its behalf. We now warn the user
-        # if they have this set and tell them to use the updated option, while using a default
-        # identity server in the process.
-        self.using_identity_server_from_trusted_list = False
-        if (
-            not self.account_threepid_delegate_email
-            and config.get("trust_identity_server_for_password_resets", False) is True
-        ):
-            # Use the first entry in self.trusted_third_party_id_servers instead
-            if self.trusted_third_party_id_servers:
-                # XXX: It's a little confusing that account_threepid_delegate_email is modified
-                # both in RegistrationConfig and here. We should factor this bit out
-                self.account_threepid_delegate_email = self.trusted_third_party_id_servers[
-                    0
-                ]
-                self.using_identity_server_from_trusted_list = True
-            else:
-                raise ConfigError(
-                    "Attempted to use an identity server from"
-                    '"trusted_third_party_id_servers" but it is empty.'
-                )
-
-        self.local_threepid_handling_disabled_due_to_email_config = False
-        if (
-            self.threepid_behaviour_email == ThreepidBehaviour.LOCAL
-            and email_config == {}
-        ):
+        self.email_password_reset_behaviour = (
+            "remote" if email_trust_identity_server_for_password_resets else "local"
+        )
+        self.password_resets_were_disabled_due_to_email_config = False
+        if self.email_password_reset_behaviour == "local" and email_config == {}:
             # We cannot warn the user this has happened here
             # Instead do so when a user attempts to reset their password
-            self.local_threepid_handling_disabled_due_to_email_config = True
+            self.password_resets_were_disabled_due_to_email_config = True
 
-            self.threepid_behaviour_email = ThreepidBehaviour.OFF
+            self.email_password_reset_behaviour = "off"
 
         # Get lifetime of a validation token in milliseconds
         self.email_validation_token_lifetime = self.parse_duration(
@@ -126,7 +96,7 @@ class EmailConfig(Config):
         if (
             self.email_enable_notifs
             or account_validity_renewal_enabled
-            or self.threepid_behaviour_email == ThreepidBehaviour.LOCAL
+            or self.email_password_reset_behaviour == "local"
         ):
             # make sure we can import the required deps
             import jinja2
@@ -136,7 +106,7 @@ class EmailConfig(Config):
             jinja2
             bleach
 
-        if self.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
+        if self.email_password_reset_behaviour == "local":
             required = ["smtp_host", "smtp_port", "notif_from"]
 
             missing = []
@@ -155,45 +125,28 @@ class EmailConfig(Config):
                     % (", ".join(missing),)
                 )
 
-            # These email templates have placeholders in them, and thus must be
-            # parsed using a templating engine during a request
+            # Templates for password reset emails
             self.email_password_reset_template_html = email_config.get(
                 "password_reset_template_html", "password_reset.html"
             )
             self.email_password_reset_template_text = email_config.get(
                 "password_reset_template_text", "password_reset.txt"
             )
-            self.email_registration_template_html = email_config.get(
-                "registration_template_html", "registration.html"
-            )
-            self.email_registration_template_text = email_config.get(
-                "registration_template_text", "registration.txt"
-            )
             self.email_password_reset_template_failure_html = email_config.get(
                 "password_reset_template_failure_html", "password_reset_failure.html"
             )
-            self.email_registration_template_failure_html = email_config.get(
-                "registration_template_failure_html", "registration_failure.html"
-            )
-
-            # These templates do not support any placeholder variables, so we
-            # will read them from disk once during setup
+            # This template does not support any replaceable variables, so we will
+            # read it from the disk once during setup
             email_password_reset_template_success_html = email_config.get(
                 "password_reset_template_success_html", "password_reset_success.html"
             )
-            email_registration_template_success_html = email_config.get(
-                "registration_template_success_html", "registration_success.html"
-            )
 
             # Check templates exist
             for f in [
                 self.email_password_reset_template_html,
                 self.email_password_reset_template_text,
-                self.email_registration_template_html,
-                self.email_registration_template_text,
                 self.email_password_reset_template_failure_html,
                 email_password_reset_template_success_html,
-                email_registration_template_success_html,
             ]:
                 p = os.path.join(self.email_template_dir, f)
                 if not os.path.isfile(p):
@@ -203,15 +156,9 @@ class EmailConfig(Config):
             filepath = os.path.join(
                 self.email_template_dir, email_password_reset_template_success_html
             )
-            self.email_password_reset_template_success_html = self.read_file(
+            self.email_password_reset_template_success_html_content = self.read_file(
                 filepath, "email.password_reset_template_success_html"
             )
-            filepath = os.path.join(
-                self.email_template_dir, email_registration_template_success_html
-            )
-            self.email_registration_template_success_html_content = self.read_file(
-                filepath, "email.registration_template_success_html"
-            )
 
         if self.email_enable_notifs:
             required = [
@@ -292,6 +239,19 @@ class EmailConfig(Config):
         #   #
         #   riot_base_url: "http://localhost/riot"
         #
+        #   # Enable sending password reset emails via the configured, trusted
+        #   # identity servers
+        #   #
+        #   # IMPORTANT! This will give a malicious or overtaken identity server
+        #   # the ability to reset passwords for your users! Make absolutely sure
+        #   # that you want to do this! It is strongly recommended that password
+        #   # reset emails be sent by the homeserver instead
+        #   #
+        #   # If this option is set to false and SMTP options have not been
+        #   # configured, resetting user passwords via email will be disabled
+        #   #
+        #   #trust_identity_server_for_password_resets: false
+        #
         #   # Configure the time that a validation email or text message code
         #   # will expire after sending
         #   #
@@ -323,35 +283,9 @@ class EmailConfig(Config):
         #   #password_reset_template_html: password_reset.html
         #   #password_reset_template_text: password_reset.txt
         #
-        #   # Templates for registration emails sent by the homeserver
-        #   #
-        #   #registration_template_html: registration.html
-        #   #registration_template_text: registration.txt
-        #
         #   # Templates for password reset success and failure pages that a user
         #   # will see after attempting to reset their password
         #   #
         #   #password_reset_template_success_html: password_reset_success.html
         #   #password_reset_template_failure_html: password_reset_failure.html
-        #
-        #   # Templates for registration success and failure pages that a user
-        #   # will see after attempting to register using an email or phone
-        #   #
-        #   #registration_template_success_html: registration_success.html
-        #   #registration_template_failure_html: registration_failure.html
         """
-
-
-class ThreepidBehaviour(Enum):
-    """
-    Enum to define the behaviour of Synapse with regards to when it contacts an identity
-    server for 3pid registration and password resets
-
-    REMOTE = use an external server to send tokens
-    LOCAL = send tokens ourselves
-    OFF = disable registration via 3pid and password resets
-    """
-
-    REMOTE = "remote"
-    LOCAL = "local"
-    OFF = "off"

synapse/config/logger.py

@@ -21,12 +21,7 @@ from string import Template
 
 import yaml
 
-from twisted.logger import (
-    ILogObserver,
-    LogBeginner,
-    STDLibLogObserver,
-    globalLogBeginner,
-)
+from twisted.logger import STDLibLogObserver, globalLogBeginner
 
 import synapse
 from synapse.app import _base as appbase
@@ -129,7 +124,7 @@ class LoggingConfig(Config):
                 log_config_file.write(DEFAULT_LOG_CONFIG.substitute(log_file=log_file))
 
 
-def _setup_stdlib_logging(config, log_config, logBeginner: LogBeginner):
+def _setup_stdlib_logging(config, log_config):
     """
     Set up Python stdlib logging.
     """
@@ -170,12 +165,12 @@ def _setup_stdlib_logging(config, log_config, logBeginner: LogBeginner):
 
         return observer(event)
 
-    logBeginner.beginLoggingTo([_log], redirectStandardIO=not config.no_redirect_stdio)
+    globalLogBeginner.beginLoggingTo(
+        [_log], redirectStandardIO=not config.no_redirect_stdio
+    )
     if not config.no_redirect_stdio:
         print("Redirected stdout/stderr to logs")
 
-    return observer
-
 
 def _reload_stdlib_logging(*args, log_config=None):
     logger = logging.getLogger("")
@@ -186,9 +181,7 @@ def _reload_stdlib_logging(*args, log_config=None):
     logging.config.dictConfig(log_config)
 
 
-def setup_logging(
-    hs, config, use_worker_options=False, logBeginner: LogBeginner = globalLogBeginner
-) -> ILogObserver:
+def setup_logging(hs, config, use_worker_options=False):
     """
     Set up the logging subsystem.
 
@@ -198,12 +191,6 @@ def setup_logging(
 
         use_worker_options (bool): True to use the 'worker_log_config' option
             instead of 'log_config'.
-
-        logBeginner: The Twisted logBeginner to use.
-
-    Returns:
-        The "root" Twisted Logger observer, suitable for sending logs to from a
-        Logger instance.
     """
     log_config = config.worker_log_config if use_worker_options else config.log_config
 
@@ -223,12 +210,10 @@ def setup_logging(
     log_config_body = read_config()
 
     if log_config_body and log_config_body.get("structured") is True:
-        logger = setup_structured_logging(
-            hs, config, log_config_body, logBeginner=logBeginner
-        )
+        setup_structured_logging(hs, config, log_config_body)
         appbase.register_sighup(read_config, callback=reload_structured_logging)
     else:
-        logger = _setup_stdlib_logging(config, log_config_body, logBeginner=logBeginner)
+        _setup_stdlib_logging(config, log_config_body)
         appbase.register_sighup(read_config, callback=_reload_stdlib_logging)
 
     # make sure that the first thing we log is a thing we can grep backwards
@@ -236,5 +221,3 @@ def setup_logging(
     logging.warn("***** STARTING SERVER *****")
     logging.warn("Server %s version %s", sys.argv[0], get_version_string(synapse))
     logging.info("Server hostname: %s", config.server_name)
-
-    return logger

synapse/config/metrics.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2015, 2016 OpenMarket Ltd
-# Copyright 2019 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,47 +13,26 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import attr
-
-from synapse.python_dependencies import DependencyException, check_requirements
-
 from ._base import Config, ConfigError
 
-
-@attr.s
-class MetricsFlags(object):
-    known_servers = attr.ib(default=False, validator=attr.validators.instance_of(bool))
-
-    @classmethod
-    def all_off(cls):
-        """
-        Instantiate the flags with all options set to off.
-        """
-        return cls(**{x.name: False for x in attr.fields(cls)})
+MISSING_SENTRY = """Missing sentry-sdk library. This is required to enable sentry
+    integration.
+    """
 
 
 class MetricsConfig(Config):
     def read_config(self, config, **kwargs):
         self.enable_metrics = config.get("enable_metrics", False)
         self.report_stats = config.get("report_stats", None)
-        self.report_stats_endpoint = config.get(
-            "report_stats_endpoint", "https://matrix.org/report-usage-stats/push"
-        )
         self.metrics_port = config.get("metrics_port")
         self.metrics_bind_host = config.get("metrics_bind_host", "127.0.0.1")
 
-        if self.enable_metrics:
-            _metrics_config = config.get("metrics_flags") or {}
-            self.metrics_flags = MetricsFlags(**_metrics_config)
-        else:
-            self.metrics_flags = MetricsFlags.all_off()
-
         self.sentry_enabled = "sentry" in config
         if self.sentry_enabled:
             try:
-                check_requirements("sentry")
-            except DependencyException as e:
-                raise ConfigError(e.message)
+                import sentry_sdk  # noqa F401
+            except ImportError:
+                raise ConfigError(MISSING_SENTRY)
 
             self.sentry_dsn = config["sentry"].get("dsn")
             if not self.sentry_dsn:
@@ -80,16 +58,6 @@ class MetricsConfig(Config):
         #sentry:
         #    dsn: "..."
 
-        # Flags to enable Prometheus metrics which are not suitable to be
-        # enabled by default, either for performance reasons or limited use.
-        #
-        metrics_flags:
-            # Publish synapse_federation_known_servers, a g auge of the number of
-            # servers this homeserver knows about, including itself. May cause
-            # performance problems on large homeservers.
-            #
-            #known_servers: true
-
         # Whether or not to report anonymized homeserver usage statistics.
         """
 
@@ -98,10 +66,4 @@ class MetricsConfig(Config):
         else:
             res += "report_stats: %s\n" % ("true" if report_stats else "false")
 
-        res += """
-        # The endpoint to report the anonymized homeserver usage statistics to.
-        # Defaults to https://matrix.org/report-usage-stats/push
-        #
-        #report_stats_endpoint: https://example.com/report-usage-stats/push
-        """
         return res

synapse/config/ratelimiting.py

@@ -80,12 +80,6 @@ class RatelimitConfig(Config):
             "federation_rr_transactions_per_room_per_second", 50
         )
 
-        rc_admin_redaction = config.get("rc_admin_redaction")
-        if rc_admin_redaction:
-            self.rc_admin_redaction = RateLimitConfig(rc_admin_redaction)
-        else:
-            self.rc_admin_redaction = None
-
     def generate_config_section(self, **kwargs):
         return """\
         ## Ratelimiting ##
@@ -108,9 +102,6 @@ class RatelimitConfig(Config):
         #   - one for login that ratelimits login requests based on the account the
         #     client is attempting to log into, based on the amount of failed login
         #     attempts for this account.
-        #   - one for ratelimiting redactions by room admins. If this is not explicitly
-        #     set then it uses the same ratelimiting as per rc_message. This is useful
-        #     to allow room admins to deal with abuse quickly.
         #
         # The defaults are as shown below.
         #
@@ -132,10 +123,6 @@ class RatelimitConfig(Config):
         #  failed_attempts:
         #    per_second: 0.17
         #    burst_count: 3
-        #
-        #rc_admin_redaction:
-        #  per_second: 1
-        #  burst_count: 50
 
 
         # Ratelimiting settings for incoming federation

synapse/config/registration.py

@@ -99,10 +99,6 @@ class RegistrationConfig(Config):
         self.trusted_third_party_id_servers = config.get(
             "trusted_third_party_id_servers", ["matrix.org", "vector.im"]
         )
-        account_threepid_delegates = config.get("account_threepid_delegates") or {}
-        self.account_threepid_delegate_email = account_threepid_delegates.get("email")
-        self.account_threepid_delegate_msisdn = account_threepid_delegates.get("msisdn")
-
         self.default_identity_server = config.get("default_identity_server")
         self.allow_guest_access = config.get("allow_guest_access", False)
 
@@ -261,42 +257,10 @@ class RegistrationConfig(Config):
         # Also defines the ID server which will be called when an account is
         # deactivated (one will be picked arbitrarily).
         #
-        # Note: This option is deprecated. Since v0.99.4, Synapse has tracked which identity
-        # server a 3PID has been bound to. For 3PIDs bound before then, Synapse runs a
-        # background migration script, informing itself that the identity server all of its
-        # 3PIDs have been bound to is likely one of the below.
-        #
-        # As of Synapse v1.4.0, all other functionality of this option has been deprecated, and
-        # it is now solely used for the purposes of the background migration script, and can be
-        # removed once it has run.
         #trusted_third_party_id_servers:
         #  - matrix.org
         #  - vector.im
 
-        # Handle threepid (email/phone etc) registration and password resets through a set of
-        # *trusted* identity servers. Note that this allows the configured identity server to
-        # reset passwords for accounts!
-        #
-        # Be aware that if `email` is not set, and SMTP options have not been
-        # configured in the email config block, registration and user password resets via
-        # email will be globally disabled.
-        #
-        # Additionally, if `msisdn` is not set, registration and password resets via msisdn
-        # will be disabled regardless. This is due to Synapse currently not supporting any
-        # method of sending SMS messages on its own.
-        #
-        # To enable using an identity server for operations regarding a particular third-party
-        # identifier type, set the value to the URL of that identity server as shown in the
-        # examples below.
-        #
-        # Servers handling the these requests must answer the `/requestToken` endpoints defined
-        # by the Matrix Identity Service API specification:
-        # https://matrix.org/docs/spec/identity_service/latest
-        #
-        account_threepid_delegates:
-            #email: https://example.com     # Delegate email sending to matrix.org
-            #msisdn: http://localhost:8090  # Delegate SMS sending to this local process
-
         # Users who register on this homeserver will automatically be joined
         # to these rooms
         #

synapse/config/repository.py

@@ -16,7 +16,6 @@
 import os
 from collections import namedtuple
 
-from synapse.python_dependencies import DependencyException, check_requirements
 from synapse.util.module_loader import load_module
 
 from ._base import Config, ConfigError
@@ -35,6 +34,17 @@ THUMBNAIL_SIZE_YAML = """\
         #    method: %(method)s
 """
 
+MISSING_NETADDR = "Missing netaddr library. This is required for URL preview API."
+
+MISSING_LXML = """Missing lxml library. This is required for URL preview API.
+
+    Install by running:
+        pip install lxml
+
+    Requires libxslt1-dev system package.
+    """
+
+
 ThumbnailRequirement = namedtuple(
     "ThumbnailRequirement", ["width", "height", "method", "media_type"]
 )
@@ -161,10 +171,16 @@ class ContentRepositoryConfig(Config):
         self.url_preview_enabled = config.get("url_preview_enabled", False)
         if self.url_preview_enabled:
             try:
-                check_requirements("url_preview")
+                import lxml
 
-            except DependencyException as e:
-                raise ConfigError(e.message)
+                lxml  # To stop unused lint.
+            except ImportError:
+                raise ConfigError(MISSING_LXML)
+
+            try:
+                from netaddr import IPSet
+            except ImportError:
+                raise ConfigError(MISSING_NETADDR)
 
             if "url_preview_ip_range_blacklist" not in config:
                 raise ConfigError(
@@ -173,9 +189,6 @@ class ContentRepositoryConfig(Config):
                     "to work"
                 )
 
-            # netaddr is a dependency for url_preview
-            from netaddr import IPSet
-
             self.url_preview_ip_range_blacklist = IPSet(
                 config["url_preview_ip_range_blacklist"]
             )

synapse/config/server.py

@@ -162,16 +162,6 @@ class ServerConfig(Config):
 
         self.mau_trial_days = config.get("mau_trial_days", 0)
 
-        # How long to keep redacted events in the database in unredacted form
-        # before redacting them.
-        redaction_retention_period = config.get("redaction_retention_period", "7d")
-        if redaction_retention_period is not None:
-            self.redaction_retention_period = self.parse_duration(
-                redaction_retention_period
-            )
-        else:
-            self.redaction_retention_period = None
-
         # Options to disable HS
         self.hs_disabled = config.get("hs_disabled", False)
         self.hs_disabled_message = config.get("hs_disabled_message", "")
@@ -728,13 +718,6 @@ class ServerConfig(Config):
         # Defaults to 'true'.
         #
         #allow_per_room_profiles: false
-
-        # How long to keep redacted events in unredacted form in the database. After
-        # this period redacted events get replaced with their redacted form in the DB.
-        #
-        # Defaults to `7d`. Set to `null` to disable.
-        #
-        redaction_retention_period: 7d
         """
             % locals()
         )

synapse/config/tracer.py

@@ -13,8 +13,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from synapse.python_dependencies import DependencyException, check_requirements
-
 from ._base import Config, ConfigError
 
 
@@ -34,11 +32,6 @@ class TracerConfig(Config):
         if not self.opentracer_enabled:
             return
 
-        try:
-            check_requirements("opentracing")
-        except DependencyException as e:
-            raise ConfigError(e.message)
-
         # The tracer is enabled so sanitize the config
 
         self.opentracer_whitelist = opentracing_config.get("homeserver_whitelist", [])

synapse/event_auth.py

@@ -637,11 +637,11 @@ def auth_types_for_event(event):
     if event.type == EventTypes.Create:
         return []
 
-    auth_types = [
-        (EventTypes.PowerLevels, ""),
-        (EventTypes.Member, event.sender),
-        (EventTypes.Create, ""),
-    ]
+    auth_types = []
+
+    auth_types.append((EventTypes.PowerLevels, ""))
+    auth_types.append((EventTypes.Member, event.sender))
+    auth_types.append((EventTypes.Create, ""))
 
     if event.type == EventTypes.Member:
         membership = event.content["membership"]

synapse/federation/federation_server.py

@@ -669,9 +669,9 @@ class FederationServer(FederationBase):
         return ret
 
     @defer.inlineCallbacks
-    def on_exchange_third_party_invite_request(self, room_id, event_dict):
+    def on_exchange_third_party_invite_request(self, origin, room_id, event_dict):
         ret = yield self.handler.on_exchange_third_party_invite_request(
-            room_id, event_dict
+            origin, room_id, event_dict
         )
         return ret
 

synapse/federation/sender/transaction_manager.py

@@ -26,7 +26,6 @@ from synapse.logging.opentracing import (
     set_tag,
     start_active_span_follows_from,
     tags,
-    whitelisted_homeserver,
 )
 from synapse.util.metrics import measure_func
 
@@ -60,15 +59,9 @@ class TransactionManager(object):
         # The span_contexts is a generator so that it won't be evaluated if
         # opentracing is disabled. (Yay speed!)
 
-        span_contexts = []
-        keep_destination = whitelisted_homeserver(destination)
-
-        for edu in pending_edus:
-            context = edu.get_context()
-            if context:
-                span_contexts.append(extract_text_map(json.loads(context)))
-            if keep_destination:
-                edu.strip_context()
+        span_contexts = (
+            extract_text_map(json.loads(edu.get_context())) for edu in pending_edus
+        )
 
         with start_active_span_follows_from("send_transaction", span_contexts):
 

synapse/federation/transport/server.py

@@ -342,11 +342,7 @@ class BaseFederationServlet(object):
                 continue
 
             server.register_paths(
-                method,
-                (pattern,),
-                self._wrap(code),
-                self.__class__.__name__,
-                trace=False,
+                method, (pattern,), self._wrap(code), self.__class__.__name__
             )
 
 
@@ -575,7 +571,7 @@ class FederationThirdPartyInviteExchangeServlet(BaseFederationServlet):
 
     async def on_PUT(self, origin, content, query, room_id):
         content = await self.handler.on_exchange_third_party_invite_request(
-            room_id, content
+            origin, room_id, content
         )
         return 200, content
 

synapse/federation/units.py

@@ -41,9 +41,6 @@ class Edu(JsonEncodedObject):
     def get_context(self):
         return getattr(self, "content", {}).get("org.matrix.opentracing_context", "{}")
 
-    def strip_context(self):
-        getattr(self, "content", {})["org.matrix.opentracing_context"] = "{}"
-
 
 class Transaction(JsonEncodedObject):
     """ A transaction is a list of Pdus and Edus to be sent to a remote home

synapse/handlers/_base.py

@@ -45,7 +45,6 @@ class BaseHandler(object):
         self.state_handler = hs.get_state_handler()
         self.distributor = hs.get_distributor()
         self.ratelimiter = hs.get_ratelimiter()
-        self.admin_redaction_ratelimiter = hs.get_admin_redaction_ratelimiter()
         self.clock = hs.get_clock()
         self.hs = hs
 
@@ -54,7 +53,7 @@ class BaseHandler(object):
         self.event_builder_factory = hs.get_event_builder_factory()
 
     @defer.inlineCallbacks
-    def ratelimit(self, requester, update=True, is_admin_redaction=False):
+    def ratelimit(self, requester, update=True):
         """Ratelimits requests.
 
         Args:
@@ -63,9 +62,6 @@ class BaseHandler(object):
                 Set to False when doing multiple checks for one request (e.g.
                 to check up front if we would reject the request), and set to
                 True for the last call for a given request.
-            is_admin_redaction (bool): Whether this is a room admin/moderator
-                redacting an event. If so then we may apply different
-                ratelimits depending on config.
 
         Raises:
             LimitExceededError if the request should be ratelimited
@@ -94,33 +90,16 @@ class BaseHandler(object):
             messages_per_second = override.messages_per_second
             burst_count = override.burst_count
         else:
-            # We default to different values if this is an admin redaction and
-            # the config is set
-            if is_admin_redaction and self.hs.config.rc_admin_redaction:
-                messages_per_second = self.hs.config.rc_admin_redaction.per_second
-                burst_count = self.hs.config.rc_admin_redaction.burst_count
-            else:
-                messages_per_second = self.hs.config.rc_message.per_second
-                burst_count = self.hs.config.rc_message.burst_count
+            messages_per_second = self.hs.config.rc_message.per_second
+            burst_count = self.hs.config.rc_message.burst_count
 
-        if is_admin_redaction and self.hs.config.rc_admin_redaction:
-            # If we have separate config for admin redactions we use a separate
-            # ratelimiter
-            allowed, time_allowed = self.admin_redaction_ratelimiter.can_do_action(
-                user_id,
-                time_now,
-                rate_hz=messages_per_second,
-                burst_count=burst_count,
-                update=update,
-            )
-        else:
-            allowed, time_allowed = self.ratelimiter.can_do_action(
-                user_id,
-                time_now,
-                rate_hz=messages_per_second,
-                burst_count=burst_count,
-                update=update,
-            )
+        allowed, time_allowed = self.ratelimiter.can_do_action(
+            user_id,
+            time_now,
+            rate_hz=messages_per_second,
+            burst_count=burst_count,
+            update=update,
+        )
         if not allowed:
             raise LimitExceededError(
                 retry_after_ms=int(1000 * (time_allowed - time_now))

synapse/handlers/account_validity.py

@@ -38,7 +38,6 @@ logger = logging.getLogger(__name__)
 class AccountValidityHandler(object):
     def __init__(self, hs):
         self.hs = hs
-        self.config = hs.config
         self.store = self.hs.get_datastore()
         self.sendmail = self.hs.get_sendmail()
         self.clock = self.hs.get_clock()
@@ -63,14 +62,9 @@ class AccountValidityHandler(object):
             self._raw_from = email.utils.parseaddr(self._from_string)[1]
 
             self._template_html, self._template_text = load_jinja2_templates(
-                self.config.email_template_dir,
-                [
-                    self.config.email_expiry_template_html,
-                    self.config.email_expiry_template_text,
-                ],
-                apply_format_ts_filter=True,
-                apply_mxc_to_http_filter=True,
-                public_baseurl=self.config.public_baseurl,
+                config=self.hs.config,
+                template_html_name=self.hs.config.email_expiry_template_html,
+                template_text_name=self.hs.config.email_expiry_template_text,
             )
 
             # Check the renewal emails to send and send them every 30min.

synapse/handlers/appservice.py

@@ -294,10 +294,12 @@ class ApplicationServicesHandler(object):
             # we don't know if they are unknown or not since it isn't one of our
             # users. We can't poke ASes.
             return False
+            return
 
         user_info = yield self.store.get_user_by_id(user_id)
         if user_info:
             return False
+            return
 
         # user not found; could be the AS though, so check.
         services = self.store.get_app_services()

synapse/handlers/auth.py

@@ -38,7 +38,6 @@ from synapse.api.errors import (
     UserDeactivatedError,
 )
 from synapse.api.ratelimiting import Ratelimiter
-from synapse.config.emailconfig import ThreepidBehaviour
 from synapse.logging.context import defer_to_thread
 from synapse.module_api import ModuleApi
 from synapse.types import UserID
@@ -159,7 +158,7 @@ class AuthHandler(BaseHandler):
         return params
 
     @defer.inlineCallbacks
-    def check_auth(self, flows, clientdict, clientip):
+    def check_auth(self, flows, clientdict, clientip, password_servlet=False):
         """
         Takes a dictionary sent by the client in the login / registration
         protocol and handles the User-Interactive Auth flow.
@@ -183,6 +182,16 @@ class AuthHandler(BaseHandler):
 
             clientip (str): The IP address of the client.
 
+            password_servlet (bool): Whether the request originated from
+                PasswordRestServlet.
+                XXX: This is a temporary hack to distinguish between checking
+                for threepid validations locally (in the case of password
+                resets) and using the identity server (in the case of binding
+                a 3PID during registration). Once we start using the
+                homeserver for both tasks, this distinction will no longer be
+                necessary.
+
+
         Returns:
             defer.Deferred[dict, dict, str]: a deferred tuple of
                 (creds, params, session_id).
@@ -238,7 +247,9 @@ class AuthHandler(BaseHandler):
         if "type" in authdict:
             login_type = authdict["type"]
             try:
-                result = yield self._check_auth_dict(authdict, clientip)
+                result = yield self._check_auth_dict(
+                    authdict, clientip, password_servlet=password_servlet
+                )
                 if result:
                     creds[login_type] = result
                     self._save_session(session)
@@ -345,7 +356,7 @@ class AuthHandler(BaseHandler):
         return sess.setdefault("serverdict", {}).get(key, default)
 
     @defer.inlineCallbacks
-    def _check_auth_dict(self, authdict, clientip):
+    def _check_auth_dict(self, authdict, clientip, password_servlet=False):
         """Attempt to validate the auth dict provided by a client
 
         Args:
@@ -363,7 +374,11 @@ class AuthHandler(BaseHandler):
         login_type = authdict["type"]
         checker = self.checkers.get(login_type)
         if checker is not None:
-            res = yield checker(authdict, clientip=clientip)
+            # XXX: Temporary workaround for having Synapse handle password resets
+            # See AuthHandler.check_auth for further details
+            res = yield checker(
+                authdict, clientip=clientip, password_servlet=password_servlet
+            )
             return res
 
         # build a v1-login-style dict out of the authdict and fall back to the
@@ -434,7 +449,7 @@ class AuthHandler(BaseHandler):
         return defer.succeed(True)
 
     @defer.inlineCallbacks
-    def _check_threepid(self, medium, authdict, **kwargs):
+    def _check_threepid(self, medium, authdict, password_servlet=False, **kwargs):
         if "threepid_creds" not in authdict:
             raise LoginError(400, "Missing threepid_creds", Codes.MISSING_PARAM)
 
@@ -443,18 +458,12 @@ class AuthHandler(BaseHandler):
         identity_handler = self.hs.get_handlers().identity_handler
 
         logger.info("Getting validated threepid. threepidcreds: %r", (threepid_creds,))
-        if self.hs.config.threepid_behaviour_email == ThreepidBehaviour.REMOTE:
-            if medium == "email":
-                threepid = yield identity_handler.threepid_from_creds(
-                    self.hs.config.account_threepid_delegate_email, threepid_creds
-                )
-            elif medium == "msisdn":
-                threepid = yield identity_handler.threepid_from_creds(
-                    self.hs.config.account_threepid_delegate_msisdn, threepid_creds
-                )
-            else:
-                raise SynapseError(400, "Unrecognized threepid medium: %s" % (medium,))
-        elif self.hs.config.threepid_behaviour_email == ThreepidBehaviour.LOCAL:
+        if (
+            not password_servlet
+            or self.hs.config.email_password_reset_behaviour == "remote"
+        ):
+            threepid = yield identity_handler.threepid_from_creds(threepid_creds)
+        elif self.hs.config.email_password_reset_behaviour == "local":
             row = yield self.store.get_threepid_validation_session(
                 medium,
                 threepid_creds["client_secret"],

synapse/handlers/device.py

@@ -25,7 +25,6 @@ from synapse.api.errors import (
     HttpResponseException,
     RequestSendFailed,
 )
-from synapse.logging.opentracing import log_kv, set_tag, trace
 from synapse.types import RoomStreamToken, get_domain_from_id
 from synapse.util import stringutils
 from synapse.util.async_helpers import Linearizer
@@ -46,7 +45,6 @@ class DeviceWorkerHandler(BaseHandler):
         self.state = hs.get_state_handler()
         self._auth_handler = hs.get_auth_handler()
 
-    @trace
     @defer.inlineCallbacks
     def get_devices_by_user(self, user_id):
         """
@@ -58,7 +56,6 @@ class DeviceWorkerHandler(BaseHandler):
             defer.Deferred: list[dict[str, X]]: info on each device
         """
 
-        set_tag("user_id", user_id)
         device_map = yield self.store.get_devices_by_user(user_id)
 
         ips = yield self.store.get_last_client_ip_by_device(user_id, device_id=None)
@@ -67,10 +64,8 @@ class DeviceWorkerHandler(BaseHandler):
         for device in devices:
             _update_device_from_client_ips(device, ips)
 
-        log_kv(device_map)
         return devices
 
-    @trace
     @defer.inlineCallbacks
     def get_device(self, user_id, device_id):
         """ Retrieve the given device
@@ -90,14 +85,9 @@ class DeviceWorkerHandler(BaseHandler):
             raise errors.NotFoundError
         ips = yield self.store.get_last_client_ip_by_device(user_id, device_id)
         _update_device_from_client_ips(device, ips)
-
-        set_tag("device", device)
-        set_tag("ips", ips)
-
         return device
 
     @measure_func("device.get_user_ids_changed")
-    @trace
     @defer.inlineCallbacks
     def get_user_ids_changed(self, user_id, from_token):
         """Get list of users that have had the devices updated, or have newly
@@ -107,9 +97,6 @@ class DeviceWorkerHandler(BaseHandler):
             user_id (str)
             from_token (StreamToken)
         """
-
-        set_tag("user_id", user_id)
-        set_tag("from_token", from_token)
         now_room_key = yield self.store.get_room_events_max_id()
 
         room_ids = yield self.store.get_rooms_for_user(user_id)
@@ -161,9 +148,6 @@ class DeviceWorkerHandler(BaseHandler):
             # special-case for an empty prev state: include all members
             # in the changed list
             if not event_ids:
-                log_kv(
-                    {"event": "encountered empty previous state", "room_id": room_id}
-                )
                 for key, event_id in iteritems(current_state_ids):
                     etype, state_key = key
                     if etype != EventTypes.Member:
@@ -216,11 +200,7 @@ class DeviceWorkerHandler(BaseHandler):
             possibly_joined = []
             possibly_left = []
 
-        result = {"changed": list(possibly_joined), "left": list(possibly_left)}
-
-        log_kv(result)
-
-        return result
+        return {"changed": list(possibly_joined), "left": list(possibly_left)}
 
 
 class DeviceHandler(DeviceWorkerHandler):
@@ -287,7 +267,6 @@ class DeviceHandler(DeviceWorkerHandler):
 
         raise errors.StoreError(500, "Couldn't generate a device ID.")
 
-    @trace
     @defer.inlineCallbacks
     def delete_device(self, user_id, device_id):
         """ Delete the given device
@@ -305,10 +284,6 @@ class DeviceHandler(DeviceWorkerHandler):
         except errors.StoreError as e:
             if e.code == 404:
                 # no match
-                set_tag("error", True)
-                log_kv(
-                    {"reason": "User doesn't have device id.", "device_id": device_id}
-                )
                 pass
             else:
                 raise
@@ -321,7 +296,6 @@ class DeviceHandler(DeviceWorkerHandler):
 
         yield self.notify_device_update(user_id, [device_id])
 
-    @trace
     @defer.inlineCallbacks
     def delete_all_devices_for_user(self, user_id, except_device_id=None):
         """Delete all of the user's devices
@@ -357,8 +331,6 @@ class DeviceHandler(DeviceWorkerHandler):
         except errors.StoreError as e:
             if e.code == 404:
                 # no match
-                set_tag("error", True)
-                set_tag("reason", "User doesn't have that device id.")
                 pass
             else:
                 raise
@@ -399,7 +371,6 @@ class DeviceHandler(DeviceWorkerHandler):
             else:
                 raise
 
-    @trace
     @measure_func("notify_device_update")
     @defer.inlineCallbacks
     def notify_device_update(self, user_id, device_ids):
@@ -415,8 +386,6 @@ class DeviceHandler(DeviceWorkerHandler):
             hosts.update(get_domain_from_id(u) for u in users_who_share_room)
             hosts.discard(self.server_name)
 
-        set_tag("target_hosts", hosts)
-
         position = yield self.store.add_device_change_to_streams(
             user_id, device_ids, list(hosts)
         )
@@ -436,7 +405,6 @@ class DeviceHandler(DeviceWorkerHandler):
             )
             for host in hosts:
                 self.federation_sender.send_device_messages(host)
-                log_kv({"message": "sent device update to host", "host": host})
 
     @defer.inlineCallbacks
     def on_federation_query_user_devices(self, user_id):
@@ -483,15 +451,12 @@ class DeviceListUpdater(object):
             iterable=True,
         )
 
-    @trace
     @defer.inlineCallbacks
     def incoming_device_list_update(self, origin, edu_content):
         """Called on incoming device list update from federation. Responsible
         for parsing the EDU and adding to pending updates list.
         """
 
-        set_tag("origin", origin)
-        set_tag("edu_content", edu_content)
         user_id = edu_content.pop("user_id")
         device_id = edu_content.pop("device_id")
         stream_id = str(edu_content.pop("stream_id"))  # They may come as ints
@@ -506,30 +471,12 @@ class DeviceListUpdater(object):
                 device_id,
                 origin,
             )
-
-            set_tag("error", True)
-            log_kv(
-                {
-                    "message": "Got a device list update edu from a user and "
-                    "device which does not match the origin of the request.",
-                    "user_id": user_id,
-                    "device_id": device_id,
-                }
-            )
             return
 
         room_ids = yield self.store.get_rooms_for_user(user_id)
         if not room_ids:
             # We don't share any rooms with this user. Ignore update, as we
             # probably won't get any further updates.
-            set_tag("error", True)
-            log_kv(
-                {
-                    "message": "Got an update from a user for which "
-                    "we don't share any rooms",
-                    "other user_id": user_id,
-                }
-            )
             logger.warning(
                 "Got device list update edu for %r/%r, but don't share a room",
                 user_id,
@@ -631,7 +578,6 @@ class DeviceListUpdater(object):
             request:
             https://matrix.org/docs/spec/server_server/r0.1.2#get-matrix-federation-v1-user-devices-userid
         """
-        log_kv({"message": "Doing resync to update device list."})
         # Fetch all devices for the user.
         origin = get_domain_from_id(user_id)
         try:
@@ -648,20 +594,13 @@ class DeviceListUpdater(object):
             # eventually become consistent.
             return
         except FederationDeniedError as e:
-            set_tag("error", True)
-            log_kv({"reason": "FederationDeniedError"})
             logger.info(e)
             return
-        except Exception as e:
+        except Exception:
             # TODO: Remember that we are now out of sync and try again
             # later
-            set_tag("error", True)
-            log_kv(
-                {"message": "Exception raised by federation request", "exception": e}
-            )
             logger.exception("Failed to handle device list update for %s", user_id)
             return
-        log_kv({"result": result})
         stream_id = result["stream_id"]
         devices = result["devices"]
 

synapse/handlers/devicemessage.py

@@ -22,9 +22,9 @@ from twisted.internet import defer
 from synapse.api.errors import SynapseError
 from synapse.logging.opentracing import (
     get_active_span_text_map,
-    log_kv,
     set_tag,
     start_active_span,
+    whitelisted_homeserver,
 )
 from synapse.types import UserID, get_domain_from_id
 from synapse.util.stringutils import random_string
@@ -86,8 +86,7 @@ class DeviceMessageHandler(object):
 
     @defer.inlineCallbacks
     def send_device_message(self, sender_user_id, message_type, messages):
-        set_tag("number_of_messages", len(messages))
-        set_tag("sender", sender_user_id)
+
         local_messages = {}
         remote_messages = {}
         for user_id, by_device in messages.items():
@@ -120,10 +119,11 @@ class DeviceMessageHandler(object):
                     "sender": sender_user_id,
                     "type": message_type,
                     "message_id": message_id,
-                    "org.matrix.opentracing_context": json.dumps(context),
+                    "org.matrix.opentracing_context": json.dumps(context)
+                    if whitelisted_homeserver(destination)
+                    else None,
                 }
 
-        log_kv({"local_messages": local_messages})
         stream_id = yield self.store.add_messages_to_device_inbox(
             local_messages, remote_edu_contents
         )
@@ -132,7 +132,6 @@ class DeviceMessageHandler(object):
             "to_device_key", stream_id, users=local_messages.keys()
         )
 
-        log_kv({"remote_messages": remote_messages})
         for destination in remote_messages.keys():
             # Enqueue a new federation transaction to send the new
             # device messages to each remote destination.

synapse/handlers/events.py

@@ -167,6 +167,7 @@ class EventHandler(BaseHandler):
 
         if not event:
             return None
+            return
 
         users = yield self.store.get_users_in_room(event.room_id)
         is_peeking = user.to_string() not in users

synapse/handlers/federation.py

@@ -2530,17 +2530,12 @@ class FederationHandler(BaseHandler):
 
     @defer.inlineCallbacks
     @log_function
-    def on_exchange_third_party_invite_request(self, room_id, event_dict):
+    def on_exchange_third_party_invite_request(self, origin, room_id, event_dict):
         """Handle an exchange_third_party_invite request from a remote server
 
         The remote server will call this when it wants to turn a 3pid invite
         into a normal m.room.member invite.
 
-        Args:
-            room_id (str): The ID of the room.
-
-            event_dict (dict[str, Any]): Dictionary containing the event body.
-
         Returns:
             Deferred: resolves (to None)
         """

synapse/handlers/identity.py

@@ -29,7 +29,6 @@ from synapse.api.errors import (
     HttpResponseException,
     SynapseError,
 )
-from synapse.util.stringutils import random_string
 
 from ._base import BaseHandler
 
@@ -42,130 +41,86 @@ class IdentityHandler(BaseHandler):
 
         self.http_client = hs.get_simple_http_client()
         self.federation_http_client = hs.get_http_client()
-        self.hs = hs
 
-    def _extract_items_from_creds_dict(self, creds):
-        """
-        Retrieve entries from a "credentials" dictionary
+        self.trusted_id_servers = set(hs.config.trusted_third_party_id_servers)
+        self.trust_any_id_server_just_for_testing_do_not_use = (
+            hs.config.use_insecure_ssl_client_just_for_testing_do_not_use
+        )
 
-        Args:
-            creds (dict[str, str]): Dictionary of credentials that contain the following keys:
-                * client_secret|clientSecret: A unique secret str provided by the client
-                * id_server|idServer: the domain of the identity server to query
-                * id_access_token: The access token to authenticate to the identity
-                    server with.
-
-        Returns:
-            tuple(str, str, str|None): A tuple containing the client_secret, the id_server,
-                and the id_access_token value if available.
-        """
-        client_secret = creds.get("client_secret") or creds.get("clientSecret")
-        if not client_secret:
-            raise SynapseError(
-                400, "No client_secret in creds", errcode=Codes.MISSING_PARAM
-            )
-
-        id_server = creds.get("id_server") or creds.get("idServer")
-        if not id_server:
-            raise SynapseError(
-                400, "No id_server in creds", errcode=Codes.MISSING_PARAM
-            )
-
-        id_access_token = creds.get("id_access_token")
-        return client_secret, id_server, id_access_token
-
-    @defer.inlineCallbacks
-    def threepid_from_creds(self, id_server, creds):
-        """
-        Retrieve and validate a threepid identifier from a "credentials" dictionary against a
-        given identity server
-
-        Args:
-            id_server (str|None): The identity server to validate 3PIDs against. If None,
-                we will attempt to extract id_server creds
-
-            creds (dict[str, str]): Dictionary containing the following keys:
-                * id_server|idServer: An optional domain name of an identity server
-                * client_secret|clientSecret: A unique secret str provided by the client
-                * sid: The ID of the validation session
-
-        Returns:
-            Deferred[dict[str,str|int]|None]: A dictionary consisting of response params to
-                the /getValidated3pid endpoint of the Identity Service API, or None if the
-                threepid was not found
-        """
-        client_secret = creds.get("client_secret") or creds.get("clientSecret")
-        if not client_secret:
-            raise SynapseError(
-                400, "Missing param client_secret in creds", errcode=Codes.MISSING_PARAM
-            )
-        session_id = creds.get("sid")
-        if not session_id:
-            raise SynapseError(
-                400, "Missing param session_id in creds", errcode=Codes.MISSING_PARAM
-            )
-        if not id_server:
-            # Attempt to get the id_server from the creds dict
-            id_server = creds.get("id_server") or creds.get("idServer")
-            if not id_server:
-                raise SynapseError(
-                    400, "Missing param id_server in creds", errcode=Codes.MISSING_PARAM
+    def _should_trust_id_server(self, id_server):
+        if id_server not in self.trusted_id_servers:
+            if self.trust_any_id_server_just_for_testing_do_not_use:
+                logger.warn(
+                    "Trusting untrustworthy ID server %r even though it isn't"
+                    " in the trusted id list for testing because"
+                    " 'use_insecure_ssl_client_just_for_testing_do_not_use'"
+                    " is set in the config",
+                    id_server,
                 )
-
-        query_params = {"sid": session_id, "client_secret": client_secret}
-
-        url = "https://%s%s" % (
-            id_server,
-            "/_matrix/identity/api/v1/3pid/getValidated3pid",
-        )
-
-        data = yield self.http_client.get_json(url, query_params)
-        return data if "medium" in data else None
+            else:
+                return False
+        return True
 
     @defer.inlineCallbacks
-    def bind_threepid(self, creds, mxid, use_v2=True):
-        """Bind a 3PID to an identity server
-
-        Args:
-            creds (dict[str, str]): Dictionary of credentials that contain the following keys:
-                * client_secret|clientSecret: A unique secret str provided by the client
-                * id_server|idServer: the domain of the identity server to query
-                * id_access_token: The access token to authenticate to the identity
-                    server with. Required if use_v2 is true
-            mxid (str): The MXID to bind the 3PID to
-            use_v2 (bool): Whether to use v2 Identity Service API endpoints
-
-        Returns:
-            Deferred[dict]: The response from the identity server
-        """
-        logger.debug("binding threepid %r to %s", creds, mxid)
-
-        client_secret, id_server, id_access_token = self._extract_items_from_creds_dict(
-            creds
-        )
-
-        sid = creds.get("sid")
-        if not sid:
-            raise SynapseError(
-                400, "No sid in three_pid_creds", errcode=Codes.MISSING_PARAM
-            )
-
-        # If an id_access_token is not supplied, force usage of v1
-        if id_access_token is None:
-            use_v2 = False
-
-        # Decide which API endpoint URLs to use
-        headers = {}
-        bind_data = {"sid": sid, "client_secret": client_secret, "mxid": mxid}
-        if use_v2:
-            bind_url = "https://%s/_matrix/identity/v2/3pid/bind" % (id_server,)
-            headers["Authorization"] = create_id_access_token_header(id_access_token)
+    def threepid_from_creds(self, creds):
+        if "id_server" in creds:
+            id_server = creds["id_server"]
+        elif "idServer" in creds:
+            id_server = creds["idServer"]
         else:
-            bind_url = "https://%s/_matrix/identity/api/v1/3pid/bind" % (id_server,)
+            raise SynapseError(400, "No id_server in creds")
+
+        if "client_secret" in creds:
+            client_secret = creds["client_secret"]
+        elif "clientSecret" in creds:
+            client_secret = creds["clientSecret"]
+        else:
+            raise SynapseError(400, "No client_secret in creds")
+
+        if not self._should_trust_id_server(id_server):
+            logger.warn(
+                "%s is not a trusted ID server: rejecting 3pid " + "credentials",
+                id_server,
+            )
+            return None
+
+        try:
+            data = yield self.http_client.get_json(
+                "https://%s%s"
+                % (id_server, "/_matrix/identity/api/v1/3pid/getValidated3pid"),
+                {"sid": creds["sid"], "client_secret": client_secret},
+            )
+        except HttpResponseException as e:
+            logger.info("getValidated3pid failed with Matrix error: %r", e)
+            raise e.to_synapse_error()
+
+        if "medium" in data:
+            return data
+        return None
+
+    @defer.inlineCallbacks
+    def bind_threepid(self, creds, mxid):
+        logger.debug("binding threepid %r to %s", creds, mxid)
+        data = None
+
+        if "id_server" in creds:
+            id_server = creds["id_server"]
+        elif "idServer" in creds:
+            id_server = creds["idServer"]
+        else:
+            raise SynapseError(400, "No id_server in creds")
+
+        if "client_secret" in creds:
+            client_secret = creds["client_secret"]
+        elif "clientSecret" in creds:
+            client_secret = creds["clientSecret"]
+        else:
+            raise SynapseError(400, "No client_secret in creds")
 
         try:
             data = yield self.http_client.post_json_get_json(
-                bind_url, bind_data, headers=headers
+                "https://%s%s" % (id_server, "/_matrix/identity/api/v1/3pid/bind"),
+                {"sid": creds["sid"], "client_secret": client_secret, "mxid": mxid},
             )
             logger.debug("bound threepid %r to %s", creds, mxid)
 
@@ -176,23 +131,13 @@ class IdentityHandler(BaseHandler):
                 address=data["address"],
                 id_server=id_server,
             )
-
-            return data
-        except HttpResponseException as e:
-            if e.code != 404 or not use_v2:
-                logger.error("3PID bind failed with Matrix error: %r", e)
-                raise e.to_synapse_error()
         except CodeMessageException as e:
             data = json.loads(e.msg)  # XXX WAT?
-            return data
-
-        logger.info("Got 404 when POSTing JSON %s, falling back to v1 URL", bind_url)
-        return (yield self.bind_threepid(creds, mxid, use_v2=False))
+        return data
 
     @defer.inlineCallbacks
     def try_unbind_threepid(self, mxid, threepid):
-        """Attempt to remove a 3PID from an identity server, or if one is not provided, all
-        identity servers we're aware the binding is present on
+        """Removes a binding from an identity server
 
         Args:
             mxid (str): Matrix user ID of binding to be removed
@@ -243,8 +188,6 @@ class IdentityHandler(BaseHandler):
             server doesn't support unbinding
         """
         url = "https://%s/_matrix/identity/api/v1/3pid/unbind" % (id_server,)
-        url_bytes = "/_matrix/identity/api/v1/3pid/unbind".encode("ascii")
-
         content = {
             "mxid": mxid,
             "threepid": {"medium": threepid["medium"], "address": threepid["address"]},
@@ -256,7 +199,7 @@ class IdentityHandler(BaseHandler):
         auth_headers = self.federation_http_client.build_auth_headers(
             destination=None,
             method="POST",
-            url_bytes=url_bytes,
+            url_bytes="/_matrix/identity/api/v1/3pid/unbind".encode("ascii"),
             content=content,
             destination_is=id_server,
         )
@@ -283,122 +226,28 @@ class IdentityHandler(BaseHandler):
 
         return changed
 
-    @defer.inlineCallbacks
-    def send_threepid_validation(
-        self,
-        email_address,
-        client_secret,
-        send_attempt,
-        send_email_func,
-        next_link=None,
-    ):
-        """Send a threepid validation email for password reset or
-        registration purposes
-
-        Args:
-            email_address (str): The user's email address
-            client_secret (str): The provided client secret
-            send_attempt (int): Which send attempt this is
-            send_email_func (func): A function that takes an email address, token,
-                                    client_secret and session_id, sends an email
-                                    and returns a Deferred.
-            next_link (str|None): The URL to redirect the user to after validation
-
-        Returns:
-            The new session_id upon success
-
-        Raises:
-            SynapseError is an error occurred when sending the email
-        """
-        # Check that this email/client_secret/send_attempt combo is new or
-        # greater than what we've seen previously
-        session = yield self.store.get_threepid_validation_session(
-            "email", client_secret, address=email_address, validated=False
-        )
-
-        # Check to see if a session already exists and that it is not yet
-        # marked as validated
-        if session and session.get("validated_at") is None:
-            session_id = session["session_id"]
-            last_send_attempt = session["last_send_attempt"]
-
-            # Check that the send_attempt is higher than previous attempts
-            if send_attempt <= last_send_attempt:
-                # If not, just return a success without sending an email
-                return session_id
-        else:
-            # An non-validated session does not exist yet.
-            # Generate a session id
-            session_id = random_string(16)
-
-        # Generate a new validation token
-        token = random_string(32)
-
-        # Send the mail with the link containing the token, client_secret
-        # and session_id
-        try:
-            yield send_email_func(email_address, token, client_secret, session_id)
-        except Exception:
-            logger.exception(
-                "Error sending threepid validation email to %s", email_address
-            )
-            raise SynapseError(500, "An error was encountered when sending the email")
-
-        token_expires = (
-            self.hs.clock.time_msec() + self.hs.config.email_validation_token_lifetime
-        )
-
-        yield self.store.start_or_continue_validation_session(
-            "email",
-            email_address,
-            session_id,
-            client_secret,
-            send_attempt,
-            next_link,
-            token,
-            token_expires,
-        )
-
-        return session_id
-
     @defer.inlineCallbacks
     def requestEmailToken(
         self, id_server, email, client_secret, send_attempt, next_link=None
     ):
-        """
-        Request an external server send an email on our behalf for the purposes of threepid
-        validation.
+        if not self._should_trust_id_server(id_server):
+            raise SynapseError(
+                400, "Untrusted ID server '%s'" % id_server, Codes.SERVER_NOT_TRUSTED
+            )
 
-        Args:
-            id_server (str): The identity server to proxy to
-            email (str): The email to send the message to
-            client_secret (str): The unique client_secret sends by the user
-            send_attempt (int): Which attempt this is
-            next_link: A link to redirect the user to once they submit the token
-
-        Returns:
-            The json response body from the server
-        """
         params = {
             "email": email,
             "client_secret": client_secret,
             "send_attempt": send_attempt,
         }
-        if next_link:
-            params["next_link"] = next_link
 
-        if self.hs.config.using_identity_server_from_trusted_list:
-            # Warn that a deprecated config option is in use
-            logger.warn(
-                'The config option "trust_identity_server_for_password_resets" '
-                'has been replaced by "account_threepid_delegate". '
-                "Please consult the sample config at docs/sample_config.yaml for "
-                "details and update your config file."
-            )
+        if next_link:
+            params.update({"next_link": next_link})
 
         try:
             data = yield self.http_client.post_json_get_json(
-                id_server + "/_matrix/identity/api/v1/validate/email/requestToken",
+                "https://%s%s"
+                % (id_server, "/_matrix/identity/api/v1/validate/email/requestToken"),
                 params,
             )
             return data
@@ -408,85 +257,28 @@ class IdentityHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def requestMsisdnToken(
-        self,
-        id_server,
-        country,
-        phone_number,
-        client_secret,
-        send_attempt,
-        next_link=None,
+        self, id_server, country, phone_number, client_secret, send_attempt, **kwargs
     ):
-        """
-        Request an external server send an SMS message on our behalf for the purposes of
-        threepid validation.
-        Args:
-            id_server (str): The identity server to proxy to
-            country (str): The country code of the phone number
-            phone_number (str): The number to send the message to
-            client_secret (str): The unique client_secret sends by the user
-            send_attempt (int): Which attempt this is
-            next_link: A link to redirect the user to once they submit the token
+        if not self._should_trust_id_server(id_server):
+            raise SynapseError(
+                400, "Untrusted ID server '%s'" % id_server, Codes.SERVER_NOT_TRUSTED
+            )
 
-        Returns:
-            The json response body from the server
-        """
         params = {
             "country": country,
             "phone_number": phone_number,
             "client_secret": client_secret,
             "send_attempt": send_attempt,
         }
-        if next_link:
-            params["next_link"] = next_link
-
-        if self.hs.config.using_identity_server_from_trusted_list:
-            # Warn that a deprecated config option is in use
-            logger.warn(
-                'The config option "trust_identity_server_for_password_resets" '
-                'has been replaced by "account_threepid_delegate". '
-                "Please consult the sample config at docs/sample_config.yaml for "
-                "details and update your config file."
-            )
+        params.update(kwargs)
 
         try:
             data = yield self.http_client.post_json_get_json(
-                id_server + "/_matrix/identity/api/v1/validate/msisdn/requestToken",
+                "https://%s%s"
+                % (id_server, "/_matrix/identity/api/v1/validate/msisdn/requestToken"),
                 params,
             )
             return data
         except HttpResponseException as e:
             logger.info("Proxied requestToken failed: %r", e)
             raise e.to_synapse_error()
-
-
-def create_id_access_token_header(id_access_token):
-    """Create an Authorization header for passing to SimpleHttpClient as the header value
-    of an HTTP request.
-
-    Args:
-        id_access_token (str): An identity server access token.
-
-    Returns:
-        list[str]: The ascii-encoded bearer token encased in a list.
-    """
-    # Prefix with Bearer
-    bearer_token = "Bearer %s" % id_access_token
-
-    # Encode headers to standard ascii
-    bearer_token.encode("ascii")
-
-    # Return as a list as that's how SimpleHttpClient takes header values
-    return [bearer_token]
-
-
-class LookupAlgorithm:
-    """
-    Supported hashing algorithms when performing a 3PID lookup.
-
-    SHA256 - Hashing an (address, medium, pepper) combo with sha256, then url-safe base64
-        encoding
-    NONE - Not performing any hashing. Simply sending an (address, medium) combo in plaintext
-    """
-
-    SHA256 = "sha256"
-    NONE = "none"

synapse/handlers/initial_sync.py

@@ -450,6 +450,7 @@ class InitialSyncHandler(BaseHandler):
             # else it will throw.
             member_event = yield self.auth.check_user_was_in_room(room_id, user_id)
             return member_event.membership, member_event.event_id
+            return
         except AuthError:
             visibility = yield self.state_handler.get_current_state(
                 room_id, EventTypes.RoomHistoryVisibility, ""
@@ -459,6 +460,7 @@ class InitialSyncHandler(BaseHandler):
                 and visibility.content["history_visibility"] == "world_readable"
             ):
                 return Membership.JOIN, None
+                return
             raise AuthError(
                 403, "Guest access not allowed", errcode=Codes.GUEST_ACCESS_FORBIDDEN
             )

synapse/handlers/message.py

@@ -729,27 +729,7 @@ class EventCreationHandler(object):
         assert not self.config.worker_app
 
         if ratelimit:
-            # We check if this is a room admin redacting an event so that we
-            # can apply different ratelimiting. We do this by simply checking
-            # it's not a self-redaction (to avoid having to look up whether the
-            # user is actually admin or not).
-            is_admin_redaction = False
-            if event.type == EventTypes.Redaction:
-                original_event = yield self.store.get_event(
-                    event.redacts,
-                    check_redacted=False,
-                    get_prev_content=False,
-                    allow_rejected=False,
-                    allow_none=True,
-                )
-
-                is_admin_redaction = (
-                    original_event and event.sender != original_event.sender
-                )
-
-            yield self.base_handler.ratelimit(
-                requester, is_admin_redaction=is_admin_redaction
-            )
+            yield self.base_handler.ratelimit(requester)
 
         yield self.base_handler.maybe_kick_guest_users(event, context)
 

synapse/handlers/presence.py

@@ -255,7 +255,7 @@ class PresenceHandler(object):
         self.unpersisted_users_changes = set()
 
         if unpersisted:
-            logger.info("Persisting %d unpersisted presence updates", len(unpersisted))
+            logger.info("Persisting %d upersisted presence updates", len(unpersisted))
             yield self.store.update_presence(
                 [self.user_to_current_state[user_id] for user_id in unpersisted]
             )

synapse/handlers/register.py

@@ -24,11 +24,13 @@ from synapse.api.errors import (
     AuthError,
     Codes,
     ConsentNotGivenError,
+    InvalidCaptchaError,
     LimitExceededError,
     RegistrationError,
     SynapseError,
 )
 from synapse.config.server import is_threepid_reserved
+from synapse.http.client import CaptchaServerHttpClient
 from synapse.http.servlet import assert_params_in_dict
 from synapse.replication.http.login import RegisterDeviceReplicationServlet
 from synapse.replication.http.register import (
@@ -37,6 +39,7 @@ from synapse.replication.http.register import (
 )
 from synapse.types import RoomAlias, RoomID, UserID, create_requester
 from synapse.util.async_helpers import Linearizer
+from synapse.util.threepids import check_3pid_allowed
 
 from ._base import BaseHandler
 
@@ -56,6 +59,7 @@ class RegistrationHandler(BaseHandler):
         self._auth_handler = hs.get_auth_handler()
         self.profile_handler = hs.get_profile_handler()
         self.user_directory_handler = hs.get_user_directory_handler()
+        self.captcha_client = CaptchaServerHttpClient(hs)
         self.identity_handler = self.hs.get_handlers().identity_handler
         self.ratelimiter = hs.get_registration_ratelimiter()
 
@@ -275,12 +279,16 @@ class RegistrationHandler(BaseHandler):
         fake_requester = create_requester(user_id)
 
         # try to create the room if we're the first real user on the server. Note
-        # that an auto-generated support or bot user is not a real user and will never be
+        # that an auto-generated support user is not a real user and will never be
         # the user to create the room
         should_auto_create_rooms = False
-        is_real_user = yield self.store.is_real_user(user_id)
-        if self.hs.config.autocreate_auto_join_rooms and is_real_user:
-            count = yield self.store.count_real_users()
+        is_support = yield self.store.is_support_user(user_id)
+        # There is an edge case where the first user is the support user, then
+        # the room is never created, though this seems unlikely and
+        # recoverable from given the support user being involved in the first
+        # place.
+        if self.hs.config.autocreate_auto_join_rooms and not is_support:
+            count = yield self.store.count_all_users()
             should_auto_create_rooms = count == 1
         for r in self.hs.config.auto_join_rooms:
             logger.info("Auto-joining %s to %s", user_id, r)
@@ -354,6 +362,70 @@ class RegistrationHandler(BaseHandler):
         )
         return user_id
 
+    @defer.inlineCallbacks
+    def check_recaptcha(self, ip, private_key, challenge, response):
+        """
+        Checks a recaptcha is correct.
+
+        Used only by c/s api v1
+        """
+
+        captcha_response = yield self._validate_captcha(
+            ip, private_key, challenge, response
+        )
+        if not captcha_response["valid"]:
+            logger.info(
+                "Invalid captcha entered from %s. Error: %s",
+                ip,
+                captcha_response["error_url"],
+            )
+            raise InvalidCaptchaError(error_url=captcha_response["error_url"])
+        else:
+            logger.info("Valid captcha entered from %s", ip)
+
+    @defer.inlineCallbacks
+    def register_email(self, threepidCreds):
+        """
+        Registers emails with an identity server.
+
+        Used only by c/s api v1
+        """
+
+        for c in threepidCreds:
+            logger.info(
+                "validating threepidcred sid %s on id server %s",
+                c["sid"],
+                c["idServer"],
+            )
+            try:
+                threepid = yield self.identity_handler.threepid_from_creds(c)
+            except Exception:
+                logger.exception("Couldn't validate 3pid")
+                raise RegistrationError(400, "Couldn't validate 3pid")
+
+            if not threepid:
+                raise RegistrationError(400, "Couldn't validate 3pid")
+            logger.info(
+                "got threepid with medium '%s' and address '%s'",
+                threepid["medium"],
+                threepid["address"],
+            )
+
+            if not check_3pid_allowed(self.hs, threepid["medium"], threepid["address"]):
+                raise RegistrationError(403, "Third party identifier is not allowed")
+
+    @defer.inlineCallbacks
+    def bind_emails(self, user_id, threepidCreds):
+        """Links emails with a user ID and informs an identity server.
+
+        Used only by c/s api v1
+        """
+
+        # Now we have a matrix ID, bind it to the threepids we were given
+        for c in threepidCreds:
+            # XXX: This should be a deferred list, shouldn't it?
+            yield self.identity_handler.bind_threepid(c, user_id)
+
     def check_user_id_not_appservice_exclusive(self, user_id, allowed_appservice=None):
         # don't allow people to register the server notices mxid
         if self._server_notices_mxid is not None:
@@ -391,8 +463,45 @@ class RegistrationHandler(BaseHandler):
         self._next_generated_user_id += 1
         return str(id)
 
+    @defer.inlineCallbacks
+    def _validate_captcha(self, ip_addr, private_key, challenge, response):
+        """Validates the captcha provided.
+
+        Used only by c/s api v1
+
+        Returns:
+            dict: Containing 'valid'(bool) and 'error_url'(str) if invalid.
+
+        """
+        response = yield self._submit_captcha(ip_addr, private_key, challenge, response)
+        # parse Google's response. Lovely format..
+        lines = response.split("\n")
+        json = {
+            "valid": lines[0] == "true",
+            "error_url": "http://www.recaptcha.net/recaptcha/api/challenge?"
+            + "error=%s" % lines[1],
+        }
+        return json
+
+    @defer.inlineCallbacks
+    def _submit_captcha(self, ip_addr, private_key, challenge, response):
+        """
+        Used only by c/s api v1
+        """
+        data = yield self.captcha_client.post_urlencoded_get_raw(
+            "http://www.recaptcha.net:80/recaptcha/api/verify",
+            args={
+                "privatekey": private_key,
+                "remoteip": ip_addr,
+                "challenge": challenge,
+                "response": response,
+            },
+        )
+        return data
+
     @defer.inlineCallbacks
     def _join_user_to_room(self, requester, room_identifier):
+        room_id = None
         room_member_handler = self.hs.get_room_member_handler()
         if RoomID.is_valid(room_identifier):
             room_id = room_identifier
@@ -539,7 +648,9 @@ class RegistrationHandler(BaseHandler):
         return (device_id, access_token)
 
     @defer.inlineCallbacks
-    def post_registration_actions(self, user_id, auth_result, access_token):
+    def post_registration_actions(
+        self, user_id, auth_result, access_token, bind_email, bind_msisdn
+    ):
         """A user has completed registration
 
         Args:
@@ -548,10 +659,18 @@ class RegistrationHandler(BaseHandler):
                 registered user.
             access_token (str|None): The access token of the newly logged in
                 device, or None if `inhibit_login` enabled.
+            bind_email (bool): Whether to bind the email with the identity
+                server.
+            bind_msisdn (bool): Whether to bind the msisdn with the identity
+                server.
         """
         if self.hs.config.worker_app:
             yield self._post_registration_client(
-                user_id=user_id, auth_result=auth_result, access_token=access_token
+                user_id=user_id,
+                auth_result=auth_result,
+                access_token=access_token,
+                bind_email=bind_email,
+                bind_msisdn=bind_msisdn,
             )
             return
 
@@ -564,11 +683,13 @@ class RegistrationHandler(BaseHandler):
             ):
                 yield self.store.upsert_monthly_active_user(user_id)
 
-            yield self._register_email_threepid(user_id, threepid, access_token)
+            yield self._register_email_threepid(
+                user_id, threepid, access_token, bind_email
+            )
 
         if auth_result and LoginType.MSISDN in auth_result:
             threepid = auth_result[LoginType.MSISDN]
-            yield self._register_msisdn_threepid(user_id, threepid)
+            yield self._register_msisdn_threepid(user_id, threepid, bind_msisdn)
 
         if auth_result and LoginType.TERMS in auth_result:
             yield self._on_user_consented(user_id, self.hs.config.user_consent_version)
@@ -587,12 +708,14 @@ class RegistrationHandler(BaseHandler):
         yield self.post_consent_actions(user_id)
 
     @defer.inlineCallbacks
-    def _register_email_threepid(self, user_id, threepid, token):
+    def _register_email_threepid(self, user_id, threepid, token, bind_email):
         """Add an email address as a 3pid identifier
 
         Also adds an email pusher for the email address, if configured in the
         HS config
 
+        Also optionally binds emails to the given user_id on the identity server
+
         Must be called on master.
 
         Args:
@@ -600,6 +723,8 @@ class RegistrationHandler(BaseHandler):
             threepid (object): m.login.email.identity auth response
             token (str|None): access_token for the user, or None if not logged
                 in.
+            bind_email (bool): true if the client requested the email to be
+                bound at the identity server
         Returns:
             defer.Deferred:
         """
@@ -641,15 +766,29 @@ class RegistrationHandler(BaseHandler):
                 data={},
             )
 
+        if bind_email:
+            logger.info("bind_email specified: binding")
+            logger.debug("Binding emails %s to %s" % (threepid, user_id))
+            yield self.identity_handler.bind_threepid(
+                threepid["threepid_creds"], user_id
+            )
+        else:
+            logger.info("bind_email not specified: not binding email")
+
     @defer.inlineCallbacks
-    def _register_msisdn_threepid(self, user_id, threepid):
+    def _register_msisdn_threepid(self, user_id, threepid, bind_msisdn):
         """Add a phone number as a 3pid identifier
 
+        Also optionally binds msisdn to the given user_id on the identity server
+
         Must be called on master.
 
         Args:
             user_id (str): id of user
             threepid (object): m.login.msisdn auth response
+            token (str): access_token for the user
+            bind_email (bool): true if the client requested the email to be
+                bound at the identity server
         Returns:
             defer.Deferred:
         """
@@ -665,3 +804,12 @@ class RegistrationHandler(BaseHandler):
         yield self._auth_handler.add_threepid(
             user_id, threepid["medium"], threepid["address"], threepid["validated_at"]
         )
+
+        if bind_msisdn:
+            logger.info("bind_msisdn specified: binding")
+            logger.debug("Binding msisdn %s to %s", threepid, user_id)
+            yield self.identity_handler.bind_threepid(
+                threepid["threepid_creds"], user_id
+            )
+        else:
+            logger.info("bind_msisdn not specified: not binding msisdn")

synapse/handlers/room.py

@@ -579,8 +579,8 @@ class RoomCreationHandler(BaseHandler):
 
         room_id = yield self._generate_room_id(creator_id=user_id, is_public=is_public)
 
-        directory_handler = self.hs.get_handlers().directory_handler
         if room_alias:
+            directory_handler = self.hs.get_handlers().directory_handler
             yield directory_handler.create_association(
                 requester=requester,
                 room_id=room_id,
@@ -665,7 +665,6 @@ class RoomCreationHandler(BaseHandler):
 
         for invite_3pid in invite_3pid_list:
             id_server = invite_3pid["id_server"]
-            id_access_token = invite_3pid.get("id_access_token")  # optional
             address = invite_3pid["address"]
             medium = invite_3pid["medium"]
             yield self.hs.get_room_member_handler().do_3pid_invite(
@@ -676,7 +675,6 @@ class RoomCreationHandler(BaseHandler):
                 id_server,
                 requester,
                 txn_id=None,
-                id_access_token=id_access_token,
             )
 
         result = {"room_id": room_id}
@@ -854,6 +852,7 @@ class RoomContextHandler(object):
         )
         if not event:
             return None
+            return
 
         filtered = yield (filter_evts([event]))
         if not filtered:

synapse/handlers/room_member.py

@@ -29,11 +29,9 @@ from twisted.internet import defer
 from synapse import types
 from synapse.api.constants import EventTypes, Membership
 from synapse.api.errors import AuthError, Codes, HttpResponseException, SynapseError
-from synapse.handlers.identity import LookupAlgorithm, create_id_access_token_header
 from synapse.types import RoomID, UserID
 from synapse.util.async_helpers import Linearizer
 from synapse.util.distributor import user_joined_room, user_left_room
-from synapse.util.hash import sha256_and_url_safe_base64
 
 from ._base import BaseHandler
 
@@ -102,7 +100,7 @@ class RoomMemberHandler(object):
         raise NotImplementedError()
 
     @abc.abstractmethod
-    def _remote_reject_invite(self, requester, remote_room_hosts, room_id, target):
+    def _remote_reject_invite(self, remote_room_hosts, room_id, target):
         """Attempt to reject an invite for a room this server is not in. If we
         fail to do so we locally mark the invite as rejected.
 
@@ -512,7 +510,9 @@ class RoomMemberHandler(object):
         return res
 
     @defer.inlineCallbacks
-    def send_membership_event(self, requester, event, context, ratelimit=True):
+    def send_membership_event(
+        self, requester, event, context, remote_room_hosts=None, ratelimit=True
+    ):
         """
         Change the membership status of a user in a room.
 
@@ -522,10 +522,16 @@ class RoomMemberHandler(object):
                 act as the sender, will be skipped.
             event (SynapseEvent): The membership event.
             context: The context of the event.
+            is_guest (bool): Whether the sender is a guest.
+            room_hosts ([str]): Homeservers which are likely to already be in
+                the room, and could be danced with in order to join this
+                homeserver for the first time.
             ratelimit (bool): Whether to rate limit this request.
         Raises:
             SynapseError if there was a problem changing the membership.
         """
+        remote_room_hosts = remote_room_hosts or []
+
         target_user = UserID.from_string(event.state_key)
         room_id = event.room_id
 
@@ -628,7 +634,7 @@ class RoomMemberHandler(object):
             servers.remove(room_alias.domain)
         servers.insert(0, room_alias.domain)
 
-        return RoomID.from_string(room_id), servers
+        return (RoomID.from_string(room_id), servers)
 
     @defer.inlineCallbacks
     def _get_inviter(self, user_id, room_id):
@@ -640,15 +646,7 @@ class RoomMemberHandler(object):
 
     @defer.inlineCallbacks
     def do_3pid_invite(
-        self,
-        room_id,
-        inviter,
-        medium,
-        address,
-        id_server,
-        requester,
-        txn_id,
-        id_access_token=None,
+        self, room_id, inviter, medium, address, id_server, requester, txn_id
     ):
         if self.config.block_non_admin_invites:
             is_requester_admin = yield self.auth.is_server_admin(requester.user)
@@ -671,12 +669,7 @@ class RoomMemberHandler(object):
                 Codes.FORBIDDEN,
             )
 
-        if not self._enable_lookup:
-            raise SynapseError(
-                403, "Looking up third-party identifiers is denied from this server"
-            )
-
-        invitee = yield self._lookup_3pid(id_server, medium, address, id_access_token)
+        invitee = yield self._lookup_3pid(id_server, medium, address)
 
         if invitee:
             yield self.update_membership(
@@ -688,47 +681,9 @@ class RoomMemberHandler(object):
             )
 
     @defer.inlineCallbacks
-    def _lookup_3pid(self, id_server, medium, address, id_access_token=None):
+    def _lookup_3pid(self, id_server, medium, address):
         """Looks up a 3pid in the passed identity server.
 
-        Args:
-            id_server (str): The server name (including port, if required)
-                of the identity server to use.
-            medium (str): The type of the third party identifier (e.g. "email").
-            address (str): The third party identifier (e.g. "foo@example.com").
-            id_access_token (str|None): The access token to authenticate to the identity
-                server with
-
-        Returns:
-            str|None: the matrix ID of the 3pid, or None if it is not recognized.
-        """
-        if id_access_token is not None:
-            try:
-                results = yield self._lookup_3pid_v2(
-                    id_server, id_access_token, medium, address
-                )
-                return results
-
-            except Exception as e:
-                # Catch HttpResponseExcept for a non-200 response code
-                # Check if this identity server does not know about v2 lookups
-                if isinstance(e, HttpResponseException) and e.code == 404:
-                    # This is an old identity server that does not yet support v2 lookups
-                    logger.warning(
-                        "Attempted v2 lookup on v1 identity server %s. Falling "
-                        "back to v1",
-                        id_server,
-                    )
-                else:
-                    logger.warning("Error when looking up hashing details: %s", e)
-                    return None
-
-        return (yield self._lookup_3pid_v1(id_server, medium, address))
-
-    @defer.inlineCallbacks
-    def _lookup_3pid_v1(self, id_server, medium, address):
-        """Looks up a 3pid in the passed identity server using v1 lookup.
-
         Args:
             id_server (str): The server name (including port, if required)
                 of the identity server to use.
@@ -738,6 +693,10 @@ class RoomMemberHandler(object):
         Returns:
             str: the matrix ID of the 3pid, or None if it is not recognized.
         """
+        if not self._enable_lookup:
+            raise SynapseError(
+                403, "Looking up third-party identifiers is denied from this server"
+            )
         try:
             data = yield self.simple_http_client.get_json(
                 "%s%s/_matrix/identity/api/v1/lookup" % (id_server_scheme, id_server),
@@ -751,116 +710,9 @@ class RoomMemberHandler(object):
                 return data["mxid"]
 
         except IOError as e:
-            logger.warning("Error from v1 identity server lookup: %s" % (e,))
-
-        return None
-
-    @defer.inlineCallbacks
-    def _lookup_3pid_v2(self, id_server, id_access_token, medium, address):
-        """Looks up a 3pid in the passed identity server using v2 lookup.
-
-        Args:
-            id_server (str): The server name (including port, if required)
-                of the identity server to use.
-            id_access_token (str): The access token to authenticate to the identity server with
-            medium (str): The type of the third party identifier (e.g. "email").
-            address (str): The third party identifier (e.g. "foo@example.com").
-
-        Returns:
-            Deferred[str|None]: the matrix ID of the 3pid, or None if it is not recognised.
-        """
-        # Check what hashing details are supported by this identity server
-        hash_details = yield self.simple_http_client.get_json(
-            "%s%s/_matrix/identity/v2/hash_details" % (id_server_scheme, id_server),
-            {"access_token": id_access_token},
-        )
-
-        if not isinstance(hash_details, dict):
-            logger.warning(
-                "Got non-dict object when checking hash details of %s%s: %s",
-                id_server_scheme,
-                id_server,
-                hash_details,
-            )
-            raise SynapseError(
-                400,
-                "Non-dict object from %s%s during v2 hash_details request: %s"
-                % (id_server_scheme, id_server, hash_details),
-            )
-
-        # Extract information from hash_details
-        supported_lookup_algorithms = hash_details.get("algorithms")
-        lookup_pepper = hash_details.get("lookup_pepper")
-        if (
-            not supported_lookup_algorithms
-            or not isinstance(supported_lookup_algorithms, list)
-            or not lookup_pepper
-            or not isinstance(lookup_pepper, str)
-        ):
-            raise SynapseError(
-                400,
-                "Invalid hash details received from identity server %s%s: %s"
-                % (id_server_scheme, id_server, hash_details),
-            )
-
-        # Check if any of the supported lookup algorithms are present
-        if LookupAlgorithm.SHA256 in supported_lookup_algorithms:
-            # Perform a hashed lookup
-            lookup_algorithm = LookupAlgorithm.SHA256
-
-            # Hash address, medium and the pepper with sha256
-            to_hash = "%s %s %s" % (address, medium, lookup_pepper)
-            lookup_value = sha256_and_url_safe_base64(to_hash)
-
-        elif LookupAlgorithm.NONE in supported_lookup_algorithms:
-            # Perform a non-hashed lookup
-            lookup_algorithm = LookupAlgorithm.NONE
-
-            # Combine together plaintext address and medium
-            lookup_value = "%s %s" % (address, medium)
-
-        else:
-            logger.warning(
-                "None of the provided lookup algorithms of %s are supported: %s",
-                id_server,
-                supported_lookup_algorithms,
-            )
-            raise SynapseError(
-                400,
-                "Provided identity server does not support any v2 lookup "
-                "algorithms that this homeserver supports.",
-            )
-
-        # Authenticate with identity server given the access token from the client
-        headers = {"Authorization": create_id_access_token_header(id_access_token)}
-
-        try:
-            lookup_results = yield self.simple_http_client.post_json_get_json(
-                "%s%s/_matrix/identity/v2/lookup" % (id_server_scheme, id_server),
-                {
-                    "addresses": [lookup_value],
-                    "algorithm": lookup_algorithm,
-                    "pepper": lookup_pepper,
-                },
-                headers=headers,
-            )
-        except Exception as e:
-            logger.warning("Error when performing a v2 3pid lookup: %s", e)
-            raise SynapseError(
-                500, "Unknown error occurred during identity server lookup"
-            )
-
-        # Check for a mapping from what we looked up to an MXID
-        if "mappings" not in lookup_results or not isinstance(
-            lookup_results["mappings"], dict
-        ):
-            logger.warning("No results from 3pid lookup")
+            logger.warn("Error from identity server lookup: %s" % (e,))
             return None
 
-        # Return the MXID if it's available, or None otherwise
-        mxid = lookup_results["mappings"].get(lookup_value)
-        return mxid
-
     @defer.inlineCallbacks
     def _verify_any_signature(self, data, server_hostname):
         if server_hostname not in data["signatures"]:
@@ -1000,6 +852,7 @@ class RoomMemberHandler(object):
                 display_name (str): A user-friendly name to represent the invited
                     user.
         """
+
         is_url = "%s%s/_matrix/identity/api/v1/store-invite" % (
             id_server_scheme,
             id_server,
@@ -1017,6 +870,7 @@ class RoomMemberHandler(object):
             "sender_display_name": inviter_display_name,
             "sender_avatar_url": inviter_avatar_url,
         }
+
         try:
             data = yield self.simple_http_client.post_json_get_json(
                 is_url, invite_config
@@ -1108,7 +962,9 @@ class RoomMemberMasterHandler(RoomMemberHandler):
         )
 
         if complexity:
-            return complexity["v1"] > max_complexity
+            if complexity["v1"] > max_complexity:
+                return True
+            return False
         return None
 
     @defer.inlineCallbacks
@@ -1124,7 +980,10 @@ class RoomMemberMasterHandler(RoomMemberHandler):
         max_complexity = self.hs.config.limit_remote_rooms.complexity
         complexity = yield self.store.get_room_complexity(room_id)
 
-        return complexity["v1"] > max_complexity
+        if complexity["v1"] > max_complexity:
+            return True
+
+        return False
 
     @defer.inlineCallbacks
     def _remote_join(self, requester, remote_room_hosts, room_id, user, content):
@@ -1203,7 +1062,7 @@ class RoomMemberMasterHandler(RoomMemberHandler):
             # The 'except' clause is very broad, but we need to
             # capture everything from DNS failures upwards
             #
-            logger.warning("Failed to reject invite: %s", e)
+            logger.warn("Failed to reject invite: %s", e)
 
             yield self.store.locally_reject_invite(target.to_string(), room_id)
             return {}

synapse/handlers/stats.py

@@ -14,14 +14,15 @@
 # limitations under the License.
 
 import logging
-from collections import Counter
 
 from twisted.internet import defer
 
-from synapse.api.constants import EventTypes, Membership
+from synapse.api.constants import EventTypes, JoinRules, Membership
 from synapse.handlers.state_deltas import StateDeltasHandler
 from synapse.metrics import event_processing_positions
 from synapse.metrics.background_process_metrics import run_as_background_process
+from synapse.types import UserID
+from synapse.util.metrics import Measure
 
 logger = logging.getLogger(__name__)
 
@@ -48,9 +49,6 @@ class StatsHandler(StateDeltasHandler):
         # The current position in the current_state_delta stream
         self.pos = None
 
-        # Guard to ensure we only process deltas one at a time
-        self._is_processing = False
-
         if hs.config.stats_enabled:
             self.notifier.add_replication_callback(self.notify_new_event)
 
@@ -61,84 +59,70 @@ class StatsHandler(StateDeltasHandler):
     def notify_new_event(self):
         """Called when there may be more deltas to process
         """
-        if not self.hs.config.stats_enabled or self._is_processing:
+        if not self.hs.config.stats_enabled:
             return
 
-        self._is_processing = True
+        lock = self.store.stats_delta_processing_lock
 
         @defer.inlineCallbacks
         def process():
+            yield lock.acquire()
             try:
                 yield self._unsafe_process()
             finally:
-                self._is_processing = False
+                yield lock.release()
 
-        run_as_background_process("stats.notify_new_event", process)
+        if not lock.locked:
+            # we only want to run this process one-at-a-time,
+            # and also, if the initial background updater wants us to keep out,
+            # we should respect that.
+            run_as_background_process("stats.notify_new_event", process)
 
     @defer.inlineCallbacks
     def _unsafe_process(self):
         # If self.pos is None then means we haven't fetched it from DB
-        if self.pos is None:
+        # If None is one of the values, then means that the stats regenerator has not (or had not) yet unwedged us
+        #   but note that this might be outdated, so we retrieve the positions again.
+        if self.pos is None or None in self.pos.values():
             self.pos = yield self.store.get_stats_positions()
 
+        # If still contains a None position, then the stats regenerator hasn't started yet
+        if None in self.pos.values():
+            return None
+
         # Loop round handling deltas until we're up to date
 
         while True:
-            deltas = yield self.store.get_current_state_deltas(self.pos)
+            with Measure(self.clock, "stats_delta"):
+                deltas = yield self.store.get_current_state_deltas(
+                    self.pos["state_delta_stream_id"]
+                )
 
-            if deltas:
-                logger.debug("Handling %d state deltas", len(deltas))
-                room_deltas, user_deltas = yield self._handle_deltas(deltas)
+                if deltas:
+                    logger.debug("Handling %d state deltas", len(deltas))
+                    yield self._handle_deltas(deltas)
 
-                max_pos = deltas[-1]["stream_id"]
-            else:
-                room_deltas = {}
-                user_deltas = {}
-                max_pos = yield self.store.get_room_max_stream_ordering()
+                    self.pos["state_delta_stream_id"] = deltas[-1]["stream_id"]
+                    yield self.store.update_stats_positions(self.pos)
+
+                    event_processing_positions.labels("stats").set(
+                        self.pos["state_delta_stream_id"]
+                    )
 
             # Then count deltas for total_events and total_event_bytes.
-            room_count, user_count = yield self.store.get_changes_room_total_events_and_bytes(
-                self.pos, max_pos
-            )
+            with Measure(self.clock, "stats_total_events_and_bytes"):
+                self.pos, had_counts = yield self.store.incremental_update_room_total_events_and_bytes(
+                    self.pos
+                )
 
-            for room_id, fields in room_count.items():
-                room_deltas.setdefault(room_id, {}).update(fields)
-
-            for user_id, fields in user_count.items():
-                user_deltas.setdefault(user_id, {}).update(fields)
-
-            logger.debug("room_deltas: %s", room_deltas)
-            logger.debug("user_deltas: %s", user_deltas)
-
-            # Always call this so that we update the stats position.
-            yield self.store.bulk_update_stats_delta(
-                self.clock.time_msec(),
-                updates={"room": room_deltas, "user": user_deltas},
-                stream_id=max_pos,
-            )
-
-            event_processing_positions.labels("stats").set(max_pos)
-
-            if self.pos == max_pos:
+            if not deltas and not had_counts:
                 break
 
-            self.pos = max_pos
-
     @defer.inlineCallbacks
     def _handle_deltas(self, deltas):
-        """Called with the state deltas to process
-
-        Returns:
-            Deferred[tuple[dict[str, Counter], dict[str, counter]]]
-            Resovles to two dicts, the room deltas and the user deltas,
-            mapping from room/user ID to changes in the various fields.
         """
-
-        room_to_stats_deltas = {}
-        user_to_stats_deltas = {}
-
-        room_to_state_updates = {}
-
+        Called with the state deltas to process
+        """
         for delta in deltas:
             typ = delta["type"]
             state_key = delta["state_key"]
@@ -147,7 +131,7 @@ class StatsHandler(StateDeltasHandler):
             stream_id = delta["stream_id"]
             prev_event_id = delta["prev_event_id"]
 
-            logger.debug("Handling: %r, %r %r, %s", room_id, typ, state_key, event_id)
+            logger.debug("Handling: %r %r, %s", typ, state_key, event_id)
 
             token = yield self.store.get_earliest_token_for_stats("room", room_id)
 
@@ -169,22 +153,23 @@ class StatsHandler(StateDeltasHandler):
 
             event_content = {}
 
-            sender = None
             if event_id is not None:
                 event = yield self.store.get_event(event_id, allow_none=True)
                 if event:
                     event_content = event.content or {}
-                    sender = event.sender
+
+            # We can't afford for this time to stray into the past, so we count
+            # it as now.
+            stream_timestamp = int(self.clock.time_msec())
 
             # All the values in this dict are deltas (RELATIVE changes)
-            room_stats_delta = room_to_stats_deltas.setdefault(room_id, Counter())
-
-            room_state = room_to_state_updates.setdefault(room_id, {})
+            room_stats_delta = {}
+            is_newly_created = False
 
             if prev_event_id is None:
                 # this state event doesn't overwrite another,
                 # so it is a new effective/current state event
-                room_stats_delta["current_state_events"] += 1
+                room_stats_delta["current_state_events"] = 1
 
             if typ == EventTypes.Member:
                 # we could use _get_key_change here but it's a bit inefficient
@@ -212,13 +197,13 @@ class StatsHandler(StateDeltasHandler):
                 elif membership == prev_membership:
                     pass  # noop
                 elif prev_membership == Membership.JOIN:
-                    room_stats_delta["joined_members"] -= 1
+                    room_stats_delta["joined_members"] = -1
                 elif prev_membership == Membership.INVITE:
-                    room_stats_delta["invited_members"] -= 1
+                    room_stats_delta["invited_members"] = -1
                 elif prev_membership == Membership.LEAVE:
-                    room_stats_delta["left_members"] -= 1
+                    room_stats_delta["left_members"] = -1
                 elif prev_membership == Membership.BAN:
-                    room_stats_delta["banned_members"] -= 1
+                    room_stats_delta["banned_members"] = -1
                 else:
                     raise ValueError(
                         "%r is not a valid prev_membership" % (prev_membership,)
@@ -227,19 +212,13 @@ class StatsHandler(StateDeltasHandler):
                 if membership == prev_membership:
                     pass  # noop
                 if membership == Membership.JOIN:
-                    room_stats_delta["joined_members"] += 1
+                    room_stats_delta["joined_members"] = +1
                 elif membership == Membership.INVITE:
-                    room_stats_delta["invited_members"] += 1
-
-                    if sender and self.is_mine_id(sender):
-                        user_to_stats_deltas.setdefault(sender, Counter())[
-                            "invites_sent"
-                        ] += 1
-
+                    room_stats_delta["invited_members"] = +1
                 elif membership == Membership.LEAVE:
-                    room_stats_delta["left_members"] += 1
+                    room_stats_delta["left_members"] = +1
                 elif membership == Membership.BAN:
-                    room_stats_delta["banned_members"] += 1
+                    room_stats_delta["banned_members"] = +1
                 else:
                     raise ValueError("%r is not a valid membership" % (membership,))
 
@@ -251,42 +230,152 @@ class StatsHandler(StateDeltasHandler):
                     )
 
                     if has_changed_joinedness:
+                        # update user_stats as it's one of our users
+                        public = yield self._is_public_room(room_id)
+
+                        field = "public_rooms" if public else "private_rooms"
                         delta = +1 if membership == Membership.JOIN else -1
 
-                        user_to_stats_deltas.setdefault(user_id, Counter())[
-                            "joined_rooms"
-                        ] += delta
-
-                        room_stats_delta["local_users_in_room"] += delta
+                        yield self.store.update_stats_delta(
+                            stream_timestamp, "user", user_id, {field: delta}
+                        )
 
             elif typ == EventTypes.Create:
-                room_state["is_federatable"] = (
-                    event_content.get("m.federate", True) is True
+                # Newly created room. Add it with all blank portions.
+                yield self.store.update_room_state(
+                    room_id,
+                    {
+                        "join_rules": None,
+                        "history_visibility": None,
+                        "encryption": None,
+                        "name": None,
+                        "topic": None,
+                        "avatar": None,
+                        "canonical_alias": None,
+                    },
                 )
-                if sender and self.is_mine_id(sender):
-                    user_to_stats_deltas.setdefault(sender, Counter())[
-                        "rooms_created"
-                    ] += 1
+
+                is_newly_created = True
+
             elif typ == EventTypes.JoinRules:
-                room_state["join_rules"] = event_content.get("join_rule")
-            elif typ == EventTypes.RoomHistoryVisibility:
-                room_state["history_visibility"] = event_content.get(
-                    "history_visibility"
+                old_room_state = yield self.store.get_room_stats_state(room_id)
+                yield self.store.update_room_state(
+                    room_id, {"join_rules": event_content.get("join_rule")}
                 )
+
+                # whether the room would be public anyway,
+                # because of history_visibility
+                other_field_gives_publicity = (
+                    old_room_state["history_visibility"] == "world_readable"
+                )
+
+                if not other_field_gives_publicity:
+                    is_public = yield self._get_key_change(
+                        prev_event_id, event_id, "join_rule", JoinRules.PUBLIC
+                    )
+                    if is_public is not None:
+                        yield self.update_public_room_stats(
+                            stream_timestamp, room_id, is_public
+                        )
+
+            elif typ == EventTypes.RoomHistoryVisibility:
+                old_room_state = yield self.store.get_room_stats_state(room_id)
+                yield self.store.update_room_state(
+                    room_id,
+                    {"history_visibility": event_content.get("history_visibility")},
+                )
+
+                # whether the room would be public anyway,
+                # because of join_rule
+                other_field_gives_publicity = (
+                    old_room_state["join_rules"] == JoinRules.PUBLIC
+                )
+
+                if not other_field_gives_publicity:
+                    is_public = yield self._get_key_change(
+                        prev_event_id, event_id, "history_visibility", "world_readable"
+                    )
+                    if is_public is not None:
+                        yield self.update_public_room_stats(
+                            stream_timestamp, room_id, is_public
+                        )
+
             elif typ == EventTypes.Encryption:
-                room_state["encryption"] = event_content.get("algorithm")
+                yield self.store.update_room_state(
+                    room_id, {"encryption": event_content.get("algorithm")}
+                )
             elif typ == EventTypes.Name:
-                room_state["name"] = event_content.get("name")
+                yield self.store.update_room_state(
+                    room_id, {"name": event_content.get("name")}
+                )
             elif typ == EventTypes.Topic:
-                room_state["topic"] = event_content.get("topic")
+                yield self.store.update_room_state(
+                    room_id, {"topic": event_content.get("topic")}
+                )
             elif typ == EventTypes.RoomAvatar:
-                room_state["avatar"] = event_content.get("url")
+                yield self.store.update_room_state(
+                    room_id, {"avatar": event_content.get("url")}
+                )
             elif typ == EventTypes.CanonicalAlias:
-                room_state["canonical_alias"] = event_content.get("alias")
-            elif typ == EventTypes.GuestAccess:
-                room_state["guest_access"] = event_content.get("guest_access")
+                yield self.store.update_room_state(
+                    room_id, {"canonical_alias": event_content.get("alias")}
+                )
 
-        for room_id, state in room_to_state_updates.items():
-            yield self.store.update_room_state(room_id, state)
+            if is_newly_created:
+                yield self.store.update_stats_delta(
+                    stream_timestamp,
+                    "room",
+                    room_id,
+                    room_stats_delta,
+                    complete_with_stream_id=stream_id,
+                )
 
-        return room_to_stats_deltas, user_to_stats_deltas
+            elif len(room_stats_delta) > 0:
+                yield self.store.update_stats_delta(
+                    stream_timestamp, "room", room_id, room_stats_delta
+                )
+
+    @defer.inlineCallbacks
+    def update_public_room_stats(self, ts, room_id, is_public):
+        """
+        Increment/decrement a user's number of public rooms when a room they are
+        in changes to/from public visibility.
+
+        Args:
+            ts (int): Timestamp in seconds
+            room_id (str)
+            is_public (bool)
+        """
+        # For now, blindly iterate over all local users in the room so that
+        # we can handle the whole problem of copying buckets over as needed
+        user_ids = yield self.store.get_users_in_room(room_id)
+
+        for user_id in user_ids:
+            if self.hs.is_mine(UserID.from_string(user_id)):
+                yield self.store.update_stats_delta(
+                    ts,
+                    "user",
+                    user_id,
+                    {
+                        "public_rooms": +1 if is_public else -1,
+                        "private_rooms": -1 if is_public else +1,
+                    },
+                )
+
+    @defer.inlineCallbacks
+    def _is_public_room(self, room_id):
+        join_rules = yield self.state.get_current_state(room_id, EventTypes.JoinRules)
+        history_visibility = yield self.state.get_current_state(
+            room_id, EventTypes.RoomHistoryVisibility
+        )
+
+        if (join_rules and join_rules.content.get("join_rule") == JoinRules.PUBLIC) or (
+            (
+                history_visibility
+                and history_visibility.content.get("history_visibility")
+                == "world_readable"
+            )
+        ):
+            return True
+        else:
+            return False

synapse/handlers/sync.py

@@ -578,6 +578,7 @@ class SyncHandler(object):
 
         if not last_events:
             return None
+            return
 
         last_event = last_events[-1]
         state_ids = yield self.store.get_state_ids_for_event(

synapse/http/client.py

@@ -35,7 +35,7 @@ from twisted.internet.interfaces import (
 )
 from twisted.python.failure import Failure
 from twisted.web._newclient import ResponseDone
-from twisted.web.client import Agent, HTTPConnectionPool, readBody
+from twisted.web.client import Agent, HTTPConnectionPool, PartialDownloadError, readBody
 from twisted.web.http import PotentialDataLoss
 from twisted.web.http_headers import Headers
 
@@ -46,7 +46,6 @@ from synapse.http import (
     redact_uri,
 )
 from synapse.logging.context import make_deferred_yieldable
-from synapse.logging.opentracing import set_tag, start_active_span, tags
 from synapse.util.async_helpers import timeout_deferred
 from synapse.util.caches import CACHE_SIZE_FACTOR
 
@@ -270,56 +269,42 @@ class SimpleHttpClient(object):
         # log request but strip `access_token` (AS requests for example include this)
         logger.info("Sending request %s %s", method, redact_uri(uri))
 
-        with start_active_span(
-            "outgoing-client-request",
-            tags={
-                tags.SPAN_KIND: tags.SPAN_KIND_RPC_CLIENT,
-                tags.HTTP_METHOD: method,
-                tags.HTTP_URL: uri,
-            },
-            finish_on_close=True,
-        ):
-            try:
-                body_producer = None
-                if data is not None:
-                    body_producer = QuieterFileBodyProducer(BytesIO(data))
+        try:
+            body_producer = None
+            if data is not None:
+                body_producer = QuieterFileBodyProducer(BytesIO(data))
 
-                request_deferred = treq.request(
-                    method,
-                    uri,
-                    agent=self.agent,
-                    data=body_producer,
-                    headers=headers,
-                    **self._extra_treq_args
-                )
-                request_deferred = timeout_deferred(
-                    request_deferred,
-                    60,
-                    self.hs.get_reactor(),
-                    cancelled_to_request_timed_out_error,
-                )
-                response = yield make_deferred_yieldable(request_deferred)
+            request_deferred = treq.request(
+                method,
+                uri,
+                agent=self.agent,
+                data=body_producer,
+                headers=headers,
+                **self._extra_treq_args
+            )
+            request_deferred = timeout_deferred(
+                request_deferred,
+                60,
+                self.hs.get_reactor(),
+                cancelled_to_request_timed_out_error,
+            )
+            response = yield make_deferred_yieldable(request_deferred)
 
-                incoming_responses_counter.labels(method, response.code).inc()
-                logger.info(
-                    "Received response to %s %s: %s",
-                    method,
-                    redact_uri(uri),
-                    response.code,
-                )
-                return response
-            except Exception as e:
-                incoming_responses_counter.labels(method, "ERR").inc()
-                logger.info(
-                    "Error sending request to  %s %s: %s %s",
-                    method,
-                    redact_uri(uri),
-                    type(e).__name__,
-                    e.args[0],
-                )
-                set_tag(tags.ERROR, True)
-                set_tag("error_reason", e.args[0])
-                raise
+            incoming_responses_counter.labels(method, response.code).inc()
+            logger.info(
+                "Received response to %s %s: %s", method, redact_uri(uri), response.code
+            )
+            return response
+        except Exception as e:
+            incoming_responses_counter.labels(method, "ERR").inc()
+            logger.info(
+                "Error sending request to  %s %s: %s %s",
+                method,
+                redact_uri(uri),
+                type(e).__name__,
+                e.args[0],
+            )
+            raise
 
     @defer.inlineCallbacks
     def post_urlencoded_get_json(self, uri, args={}, headers=None):
@@ -614,6 +599,38 @@ def _readBodyToFile(response, stream, max_size):
     return d
 
 
+class CaptchaServerHttpClient(SimpleHttpClient):
+    """
+    Separate HTTP client for talking to google's captcha servers
+    Only slightly special because accepts partial download responses
+
+    used only by c/s api v1
+    """
+
+    @defer.inlineCallbacks
+    def post_urlencoded_get_raw(self, url, args={}):
+        query_bytes = urllib.parse.urlencode(encode_urlencode_args(args), True)
+
+        response = yield self.request(
+            "POST",
+            url,
+            data=query_bytes,
+            headers=Headers(
+                {
+                    b"Content-Type": [b"application/x-www-form-urlencoded"],
+                    b"User-Agent": [self.user_agent],
+                }
+            ),
+        )
+
+        try:
+            body = yield make_deferred_yieldable(readBody(response))
+            return body
+        except PartialDownloadError as e:
+            # twisted dislikes google's response, no content length.
+            return e.response
+
+
 def encode_urlencode_args(args):
     return {k: encode_urlencode_arg(v) for k, v in args.items()}
 

synapse/http/matrixfederationclient.py

@@ -345,6 +345,7 @@ class MatrixFederationHttpClient(object):
         else:
             query_bytes = b""
 
+        # Retreive current span
         scope = start_active_span(
             "outgoing-federation-request",
             tags={

synapse/http/server.py

@@ -40,7 +40,6 @@ from synapse.api.errors import (
     UnrecognizedRequestError,
 )
 from synapse.logging.context import preserve_fn
-from synapse.logging.opentracing import trace_servlet
 from synapse.util.caches import intern_dict
 
 logger = logging.getLogger(__name__)
@@ -258,9 +257,7 @@ class JsonResource(HttpServer, resource.Resource):
         self.path_regexs = {}
         self.hs = hs
 
-    def register_paths(
-        self, method, path_patterns, callback, servlet_classname, trace=True
-    ):
+    def register_paths(self, method, path_patterns, callback, servlet_classname):
         """
         Registers a request handler against a regular expression. Later request URLs are
         checked against these regular expressions in order to identify an appropriate
@@ -276,16 +273,8 @@ class JsonResource(HttpServer, resource.Resource):
 
             servlet_classname (str): The name of the handler to be used in prometheus
                 and opentracing logs.
-
-            trace (bool): Whether we should start a span to trace the servlet.
         """
         method = method.encode("utf-8")  # method is bytes on py3
-
-        if trace:
-            # We don't extract the context from the servlet because we can't
-            # trust the sender
-            callback = trace_servlet(servlet_classname)(callback)
-
         for path_pattern in path_patterns:
             logger.debug("Registering for %s %s", method, path_pattern.pattern)
             self.path_regexs.setdefault(method, []).append(

synapse/http/servlet.py

@@ -20,6 +20,7 @@ import logging
 from canonicaljson import json
 
 from synapse.api.errors import Codes, SynapseError
+from synapse.logging.opentracing import trace_servlet
 
 logger = logging.getLogger(__name__)
 
@@ -297,7 +298,10 @@ class RestServlet(object):
                     servlet_classname = self.__class__.__name__
                     method_handler = getattr(self, "on_%s" % (method,))
                     http_server.register_paths(
-                        method, patterns, method_handler, servlet_classname
+                        method,
+                        patterns,
+                        trace_servlet(servlet_classname)(method_handler),
+                        servlet_classname,
                     )
 
         else:

synapse/logging/_structured.py

@@ -18,7 +18,6 @@ import os.path
 import sys
 import typing
 import warnings
-from typing import List
 
 import attr
 from constantly import NamedConstant, Names, ValueConstant, Values
@@ -34,6 +33,7 @@ from twisted.logger import (
     LogLevelFilterPredicate,
     LogPublisher,
     eventAsText,
+    globalLogBeginner,
     jsonFileLogObserver,
 )
 
@@ -134,7 +134,7 @@ class PythonStdlibToTwistedLogger(logging.Handler):
         )
 
 
-def SynapseFileLogObserver(outFile: typing.IO[str]) -> FileLogObserver:
+def SynapseFileLogObserver(outFile: typing.io.TextIO) -> FileLogObserver:
     """
     A log observer that formats events like the traditional log formatter and
     sends them to `outFile`.
@@ -265,7 +265,7 @@ def setup_structured_logging(
     hs,
     config,
     log_config: dict,
-    logBeginner: LogBeginner,
+    logBeginner: LogBeginner = globalLogBeginner,
     redirect_stdlib_logging: bool = True,
 ) -> LogPublisher:
     """
@@ -286,7 +286,7 @@ def setup_structured_logging(
     if "drains" not in log_config:
         raise ConfigError("The logging configuration requires a list of drains.")
 
-    observers = []  # type: List[ILogObserver]
+    observers = []
 
     for observer in parse_drain_configs(log_config["drains"]):
         # Pipe drains

synapse/logging/_terse_json.py

@@ -21,11 +21,10 @@ import sys
 from collections import deque
 from ipaddress import IPv4Address, IPv6Address, ip_address
 from math import floor
-from typing import IO
+from typing.io import TextIO
 
 import attr
 from simplejson import dumps
-from zope.interface import implementer
 
 from twisted.application.internet import ClientService
 from twisted.internet.endpoints import (
@@ -34,7 +33,7 @@ from twisted.internet.endpoints import (
     TCP6ClientEndpoint,
 )
 from twisted.internet.protocol import Factory, Protocol
-from twisted.logger import FileLogObserver, ILogObserver, Logger
+from twisted.logger import FileLogObserver, Logger
 from twisted.python.failure import Failure
 
 
@@ -130,7 +129,7 @@ def flatten_event(event: dict, metadata: dict, include_time: bool = False):
     return new_event
 
 
-def TerseJSONToConsoleLogObserver(outFile: IO[str], metadata: dict) -> FileLogObserver:
+def TerseJSONToConsoleLogObserver(outFile: TextIO, metadata: dict) -> FileLogObserver:
     """
     A log observer that formats events to a flattened JSON representation.
 
@@ -147,7 +146,6 @@ def TerseJSONToConsoleLogObserver(outFile: IO[str], metadata: dict) -> FileLogOb
 
 
 @attr.s
-@implementer(ILogObserver)
 class TerseJSONToTCPLogObserver(object):
     """
     An IObserver that writes JSON logs to a TCP target.