changelog

It appears as though Python only checks to see if the key exists in a dictionary, not necessarily for a useful value. This means that when clients submit (valid) requests with `m.relates_to: null` and Synapse later reads it, it gets a None reference error on access.

This is the easier route than guarding all the places where it could be None.

.buildkite/pipeline.yml

@@ -26,6 +26,16 @@ steps:
       - docker#v3.0.1:
           image: "python:3.6"
 
+  - command:
+      - "python -m pip install tox"
+      - "scripts-dev/check-newsfragment"
+    label: ":newspaper: Newsfile"
+    branches: "!master !develop !release-*"
+    plugins:
+      - docker#v3.0.1:
+          image: "python:3.6"
+          propagate-environment: true
+
   - wait
 
   - command:

CHANGES.md

@@ -1,3 +1,54 @@
+Synapse 0.99.5.1 (2019-05-22)
+=============================
+
+No significant changes.
+
+
+Synapse 0.99.5 (2019-05-22)
+===========================
+
+No significant changes.
+
+
+Synapse 0.99.5rc1 (2019-05-21)
+==============================
+
+Features
+--------
+
+- Add ability to blacklist IP ranges for the federation client. ([\#5043](https://github.com/matrix-org/synapse/issues/5043))
+- Ratelimiting configuration for clients sending messages and the federation server has been altered to match login ratelimiting. The old configuration names will continue working. Check the sample config for details of the new names. ([\#5181](https://github.com/matrix-org/synapse/issues/5181))
+- Drop support for the undocumented /_matrix/client/v2_alpha API prefix. ([\#5190](https://github.com/matrix-org/synapse/issues/5190))
+- Add an option to disable per-room profiles. ([\#5196](https://github.com/matrix-org/synapse/issues/5196))
+- Stick an expiration date to any registered user missing one at startup if account validity is enabled. ([\#5204](https://github.com/matrix-org/synapse/issues/5204))
+- Add experimental support for relations (aka reactions and edits). ([\#5209](https://github.com/matrix-org/synapse/issues/5209), [\#5211](https://github.com/matrix-org/synapse/issues/5211), [\#5203](https://github.com/matrix-org/synapse/issues/5203), [\#5212](https://github.com/matrix-org/synapse/issues/5212))
+- Add a room version 4 which uses a new event ID format, as per [MSC2002](https://github.com/matrix-org/matrix-doc/pull/2002). ([\#5210](https://github.com/matrix-org/synapse/issues/5210), [\#5217](https://github.com/matrix-org/synapse/issues/5217))
+
+
+Bugfixes
+--------
+
+- Fix image orientation when generating thumbnails (needs pillow>=4.3.0). Contributed by Pau Rodriguez-Estivill. ([\#5039](https://github.com/matrix-org/synapse/issues/5039))
+- Exclude soft-failed events from forward-extremity candidates: fixes "No forward extremities left!" error. ([\#5146](https://github.com/matrix-org/synapse/issues/5146))
+- Re-order stages in registration flows such that msisdn and email verification are done last. ([\#5174](https://github.com/matrix-org/synapse/issues/5174))
+- Fix 3pid guest invites. ([\#5177](https://github.com/matrix-org/synapse/issues/5177))
+- Fix a bug where the register endpoint would fail with M_THREEPID_IN_USE instead of returning an account previously registered in the same session. ([\#5187](https://github.com/matrix-org/synapse/issues/5187))
+- Prevent registration for user ids that are too long to fit into a state key. Contributed by Reid Anderson. ([\#5198](https://github.com/matrix-org/synapse/issues/5198))
+- Fix incompatibility between ACME support and Python 3.5.2. ([\#5218](https://github.com/matrix-org/synapse/issues/5218))
+- Fix error handling for rooms whose versions are unknown. ([\#5219](https://github.com/matrix-org/synapse/issues/5219))
+
+
+Internal Changes
+----------------
+
+- Make /sync attempt to return device updates for both joined and invited users. Note that this doesn't currently work correctly due to other bugs. ([\#3484](https://github.com/matrix-org/synapse/issues/3484))
+- Update tests to consistently be configured via the same code that is used when loading from configuration files. ([\#5171](https://github.com/matrix-org/synapse/issues/5171), [\#5185](https://github.com/matrix-org/synapse/issues/5185))
+- Allow client event serialization to be async. ([\#5183](https://github.com/matrix-org/synapse/issues/5183))
+- Expose DataStore._get_events as get_events_as_list. ([\#5184](https://github.com/matrix-org/synapse/issues/5184))
+- Make generating SQL bounds for pagination generic. ([\#5191](https://github.com/matrix-org/synapse/issues/5191))
+- Stop telling people to install the optional dependencies by default. ([\#5197](https://github.com/matrix-org/synapse/issues/5197))
+
+
 Synapse 0.99.4 (2019-05-15)
 ===========================
 

INSTALL.md

@@ -35,7 +35,7 @@ virtualenv -p python3 ~/synapse/env
 source ~/synapse/env/bin/activate
 pip install --upgrade pip
 pip install --upgrade setuptools
-pip install matrix-synapse[all]
+pip install matrix-synapse
 ```
 
 This will download Synapse from [PyPI](https://pypi.org/project/matrix-synapse)
@@ -48,7 +48,7 @@ update flag:
 
 ```
 source ~/synapse/env/bin/activate
-pip install -U matrix-synapse[all]
+pip install -U matrix-synapse
 ```
 
 Before you can start Synapse, you will need to generate a configuration

MANIFEST.in

@@ -43,8 +43,3 @@ prune .buildkite
 
 exclude jenkins*
 recursive-exclude jenkins *.sh
-
-# FIXME: we shouldn't have these templates here
-recursive-include res/templates-dinsic *.css
-recursive-include res/templates-dinsic *.html
-recursive-include res/templates-dinsic *.txt

changelog.d/3484.misc

@@ -1 +0,0 @@
-Make /sync attempt to return device updates for both joined and invited users. Note that this doesn't currently work correctly due to other bugs.

changelog.d/4338.feature

@@ -0,0 +1 @@
+Synapse now more efficiently collates room statistics.

changelog.d/5043.feature

@@ -1 +0,0 @@
-Add ability to blacklist IP ranges for the federation client.

changelog.d/5083.feature

@@ -1 +0,0 @@
-Adds auth_profile_reqs option to require access_token to GET /profile endpoints on CS API

changelog.d/5098.misc

@@ -1 +0,0 @@
-Add workarounds for pep-517 install errors.

changelog.d/5171.misc

@@ -1 +0,0 @@
-Update tests to consistently be configured via the same code that is used when loading from configuration files.

changelog.d/5181.feature

@@ -1 +0,0 @@
-Ratelimiting configuration for clients sending messages and the federation server has been altered to match login ratelimiting. The old configuration names will continue working. Check the sample config for details of the new names.

changelog.d/5183.misc

@@ -1 +0,0 @@
-Allow client event serialization to be async.

changelog.d/5184.misc

@@ -1 +0,0 @@
-Expose DataStore._get_events as get_events_as_list.

changelog.d/5185.misc

@@ -1 +0,0 @@
-Update tests to consistently be configured via the same code that is used when loading from configuration files.

changelog.d/5187.bugfix

@@ -1 +0,0 @@
-Fix a bug where the register endpoint would fail with M_THREEPID_IN_USE instead of returning an account previously registered in the same session.

changelog.d/5190.feature

@@ -1 +0,0 @@
-Drop support for the undocumented /_matrix/client/v2_alpha API prefix.

changelog.d/5196.feature

@@ -1 +0,0 @@
-Add an option to disable per-room profiles.

changelog.d/5200.bugfix

@@ -0,0 +1 @@
+Fix worker registration bug caused by ClientReaderSlavedStore being unable to see get_profileinfo.

changelog.d/5204.feature

@@ -1 +0,0 @@
-Stick an expiration date to any registered user missing one at startup if account validity is enabled.

changelog.d/5214.feature

@@ -1 +0,0 @@
-Allow server admins to define and enforce a password policy (MSC2000).

changelog.d/5230.misc

@@ -0,0 +1 @@
+Remove urllib3 pin as requests 2.22.0 has been released supporting urllib3 1.25.2.

changelog.d/5232.misc

@@ -0,0 +1 @@
+Run black on synapse.crypto.keyring.

changelog.d/5239.bugfix

@@ -0,0 +1 @@
+Fix 500 Internal Server Error when sending an event with `m.relates_to: null`.

changelog.d/5268.bugfix

@@ -1 +0,0 @@
-Fix schema update for account validity.

changelog.d/5276.feature

@@ -1 +0,0 @@
-Allow configuring a range for the account validity startup job.

debian/changelog

@@ -1,3 +1,9 @@
+matrix-synapse-py3 (0.99.5.1) stable; urgency=medium
+
+  * New synapse release 0.99.5.1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Wed, 22 May 2019 16:22:24 +0000
+
 matrix-synapse-py3 (0.99.4) stable; urgency=medium
 
   [ Christoph Müller ]

debian/test/.gitignore

@@ -0,0 +1,2 @@
+.vagrant
+*.log

debian/test/provision.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+#
+# provisioning script for vagrant boxes for testing the matrix-synapse debs.
+#
+# Will install the most recent matrix-synapse-py3 deb for this platform from
+# the /debs directory.
+
+set -e
+
+apt-get update
+apt-get install -y lsb-release
+
+deb=`ls /debs/matrix-synapse-py3_*+$(lsb_release -cs)*.deb | sort | tail -n1`
+
+debconf-set-selections <<EOF
+matrix-synapse matrix-synapse/report-stats boolean false
+matrix-synapse matrix-synapse/server-name string localhost:18448
+EOF
+
+dpkg -i "$deb"
+
+sed -i -e '/port: 8...$/{s/8448/18448/; s/8008/18008/}' -e '$aregistration_shared_secret: secret' /etc/matrix-synapse/homeserver.yaml
+systemctl restart matrix-synapse

debian/test/stretch/Vagrantfile

@@ -0,0 +1,13 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+ver = `cd ../../..; dpkg-parsechangelog -S Version`.strip()
+
+Vagrant.configure("2") do |config|
+  config.vm.box = "debian/stretch64"
+
+  config.vm.synced_folder ".", "/vagrant", disabled: true
+  config.vm.synced_folder "../../../../debs", "/debs", type: "nfs"
+
+  config.vm.provision "shell", path: "../provision.sh"
+end

debian/test/xenial/Vagrantfile

@@ -0,0 +1,10 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+Vagrant.configure("2") do |config|
+  config.vm.box = "ubuntu/xenial64"
+
+  config.vm.synced_folder ".", "/vagrant", disabled: true
+  config.vm.synced_folder "../../../../debs", "/debs"
+  config.vm.provision "shell", path: "../provision.sh"
+end

docker/README.md

@@ -161,7 +161,7 @@ specify values for `SYNAPSE_CONFIG_PATH`, `SYNAPSE_SERVER_NAME` and
 example:
 
 ```
-docker run -it --rm
+docker run -it --rm \
     --mount type=volume,src=synapse-data,dst=/data \
     -e SYNAPSE_CONFIG_PATH=/data/homeserver.yaml \
     -e SYNAPSE_SERVER_NAME=my.matrix.host \

docs/postgres.rst

@@ -3,6 +3,28 @@ Using Postgres
 
 Postgres version 9.4 or later is known to work.
 
+Install postgres client libraries
+=================================
+
+Synapse will require the python postgres client library in order to connect to
+a postgres database.
+
+* If you are using the `matrix.org debian/ubuntu
+  packages <../INSTALL.md#matrixorg-packages>`_,
+  the necessary libraries will already be installed.
+
+* For other pre-built packages, please consult the documentation from the
+  relevant package.
+
+* If you installed synapse `in a virtualenv 
+  <../INSTALL.md#installing-from-source>`_, you can install the library with::
+
+      ~/synapse/env/bin/pip install matrix-synapse[postgres]
+
+  (substituting the path to your virtualenv for ``~/synapse/env``, if you used a
+  different path). You will require the postgres development files. These are in
+  the ``libpq-dev`` package on Debian-derived distributions.
+
 Set up database
 ===============
 
@@ -26,29 +48,6 @@ encoding use, e.g.::
 This would create an appropriate database named ``synapse`` owned by the
 ``synapse_user`` user (which must already exist).
 
-Set up client in Debian/Ubuntu
-===========================
-
-Postgres support depends on the postgres python connector ``psycopg2``. In the
-virtual env::
-
-    sudo apt-get install libpq-dev
-    pip install psycopg2
-
-Set up client in RHEL/CentOs 7
-==============================
-
-Make sure you have the appropriate version of postgres-devel installed. For a
-postgres 9.4, use the postgres 9.4 packages from
-[here](https://wiki.postgresql.org/wiki/YUM_Installation).
-
-As with Debian/Ubuntu, postgres support depends on the postgres python connector
-``psycopg2``. In the virtual env::
-
-    sudo yum install postgresql-devel libpqxx-devel.x86_64
-    export PATH=/usr/pgsql-9.4/bin/:$PATH
-    pip install psycopg2
-
 Tuning Postgres
 ===============
 

docs/sample_config.yaml

@@ -753,9 +753,7 @@ uploads_path: "DATADIR/uploads"
 # This means that, if a validity period is set, and Synapse is restarted (it will
 # then derive an expiration date from the current validity period), and some time
 # after that the validity period changes and Synapse is restarted, the users'
-# expiration dates won't be updated unless their account is manually renewed. This
-# date will be randomly selected within a range [now + period - d ; now + period],
-# where d is equal to 10% of the validity period.
+# expiration dates won't be updated unless their account is manually renewed.
 #
 #account_validity:
 #  enabled: True
@@ -774,32 +772,9 @@ uploads_path: "DATADIR/uploads"
 #
 #disable_msisdn_registration: true
 
-# Derive the user's matrix ID from a type of 3PID used when registering.
-# This overrides any matrix ID the user proposes when calling /register
-# The 3PID type should be present in registrations_require_3pid to avoid
-# users failing to register if they don't specify the right kind of 3pid.
-#
-#register_mxid_from_3pid: email
-
-# Uncomment to set the display name of new users to their email address,
-# rather than using the default heuristic.
-#
-#register_just_use_email_for_display_name: true
-
 # Mandate that users are only allowed to associate certain formats of
 # 3PIDs with accounts on this server.
 #
-# Use an Identity Server to establish which 3PIDs are allowed to register?
-# Overrides allowed_local_3pids below.
-#
-#check_is_for_allowed_local_3pids: matrix.org
-#
-# If you are using an IS you can also check whether that IS registers
-# pending invites for the given 3PID (and then allow it to sign up on
-# the platform):
-#
-#allow_invited_3pids: False
-#
 #allowed_local_3pids:
 #  - medium: email
 #    pattern: '.*@matrix\.org'
@@ -808,11 +783,6 @@ uploads_path: "DATADIR/uploads"
 #  - medium: msisdn
 #    pattern: '\+44'
 
-# If true, stop users from trying to change the 3PIDs associated with
-# their accounts.
-#
-#disable_3pid_changes: False
-
 # Enable 3PIDs lookup requests to identity servers from this server.
 #
 #enable_3pid_lookup: true
@@ -854,30 +824,6 @@ uploads_path: "DATADIR/uploads"
 #  - matrix.org
 #  - vector.im
 
-# If enabled, user IDs, display names and avatar URLs will be replicated
-# to this server whenever they change.
-# This is an experimental API currently implemented by sydent to support
-# cross-homeserver user directories.
-#
-#replicate_user_profiles_to: example.com
-
-# If specified, attempt to replay registrations, profile changes & 3pid
-# bindings on the given target homeserver via the AS API. The HS is authed
-# via a given AS token.
-#
-#shadow_server:
-#  hs_url: https://shadow.example.com
-#  hs: shadow.example.com
-#  as_token: 12u394refgbdhivsia
-
-# If enabled, don't let users set their own display names/avatars
-# other than for the very first time (unless they are a server admin).
-# Useful when provisioning users based on the contents of a 3rd party
-# directory and to avoid ambiguities.
-#
-#disable_set_displayname: False
-#disable_set_avatar_url: False
-
 # Users who register on this homeserver will automatically be joined
 # to these rooms
 #
@@ -1058,36 +1004,6 @@ password_config:
    #
    #pepper: "EVEN_MORE_SECRET"
 
-   # Define and enforce a password policy. Each parameter is optional, boolean
-   # parameters default to 'false' and integer parameters default to 0.
-   # This is an early implementation of MSC2000.
-   #
-   #policy:
-      # Whether to enforce the password policy.
-      #
-      #enabled: true
-
-      # Minimum accepted length for a password.
-      #
-      #minimum_length: 15
-
-      # Whether a password must contain at least one digit.
-      #
-      #require_digit: true
-
-      # Whether a password must contain at least one symbol.
-      # A symbol is any character that's not a number or a letter.
-      #
-      #require_symbol: true
-
-      # Whether a password must contain at least one lowercase letter.
-      #
-      #require_lowercase: true
-
-      # Whether a password must contain at least one lowercase letter.
-      #
-      #require_uppercase: true
-
 
 
 # Enable sending emails for notification events or expiry notices
@@ -1184,11 +1100,6 @@ password_config:
 #user_directory:
 #  enabled: true
 #  search_all_users: false
-#
-#  # If this is set, user search will be delegated to this ID server instead
-#  # of synapse performing the search itself.
-#  # This is an experimental API.
-#  defer_to_id_server: https://id.example.com
 
 
 # User Consent configuration
@@ -1242,6 +1153,22 @@ password_config:
 #
 
 
+
+# Local statistics collection. Used in populating the room directory.
+#
+# 'bucket_size' controls how large each statistics timeslice is. It can
+# be defined in a human readable short form -- e.g. "1d", "1y".
+#
+# 'retention' controls how long historical statistics will be kept for.
+# It can be defined in a human readable short form -- e.g. "1d", "1y".
+#
+#
+#stats:
+#   enabled: true
+#   bucket_size: 1d
+#   retention: 1y
+
+
 # Server Notices room configuration
 #
 # Uncomment this section to enable a room which can be used to send notices

res/templates-dinsic/mail-Vector.css

@@ -1,7 +0,0 @@
-.header {
-    border-bottom: 4px solid #e4f7ed ! important;
-}
-
-.notif_link a, .footer a {
-    color: #76CFA6 ! important;
-}

res/templates-dinsic/mail.css

@@ -1,156 +0,0 @@
-body {
-    margin: 0px;
-}
-
-pre, code {
-    word-break: break-word;
-    white-space: pre-wrap;
-}
-
-#page {
-    font-family: 'Open Sans', Helvetica, Arial, Sans-Serif;
-    font-color: #454545;
-    font-size: 12pt;
-    width: 100%;
-    padding: 20px;
-}
-
-#inner {
-    width: 640px;
-}
-
-.header {
-    width: 100%;
-    height: 87px;
-    color: #454545;
-    border-bottom: 4px solid #e5e5e5;
-}
-
-.logo {
-    text-align: right;
-    margin-left: 20px;
-}
-
-.salutation {
-    padding-top: 10px;
-    font-weight: bold;
-}
-
-.summarytext {
-}
-
-.room {
-    width: 100%;
-    color: #454545;
-    border-bottom: 1px solid #e5e5e5;
-}
-
-.room_header td {
-    padding-top: 38px;
-    padding-bottom: 10px;
-    border-bottom: 1px solid #e5e5e5;
-}
-
-.room_name {
-    vertical-align: middle;
-    font-size: 18px;
-    font-weight: bold;
-}
-
-.room_header h2 {
-    margin-top: 0px;
-    margin-left: 75px;
-    font-size: 20px;
-}
-
-.room_avatar {
-    width: 56px;
-    line-height: 0px;
-    text-align: center;
-    vertical-align: middle;
-}
-
-.room_avatar img {
-    width: 48px;
-    height: 48px;
-    object-fit: cover;
-    border-radius: 24px;
-}
-
-.notif {
-    border-bottom: 1px solid #e5e5e5;
-    margin-top: 16px;
-    padding-bottom: 16px;
-}
-
-.historical_message .sender_avatar {
-    opacity: 0.3;
-}
-
-/* spell out opacity and historical_message class names for Outlook aka Word */
-.historical_message .sender_name {
-    color: #e3e3e3;
-}
-
-.historical_message .message_time {
-    color: #e3e3e3;
-}
-
-.historical_message .message_body {
-    color: #c7c7c7;
-}
-
-.historical_message td,
-.message td {
-    padding-top: 10px;
-}
-
-.sender_avatar {
-    width: 56px;
-    text-align: center;
-    vertical-align: top;
-}
-
-.sender_avatar img {
-    margin-top: -2px;
-    width: 32px;
-    height: 32px;
-    border-radius: 16px;
-}
-
-.sender_name  {
-    display: inline;
-    font-size: 13px;
-    color: #a2a2a2;
-}
-
-.message_time  {
-    text-align: right;
-    width: 100px;
-    font-size: 11px;
-    color: #a2a2a2;
-}
-
-.message_body {
-}
-
-.notif_link td {
-    padding-top: 10px;
-    padding-bottom: 10px;
-    font-weight: bold;
-}
-
-.notif_link a, .footer a {
-    color: #454545;
-    text-decoration: none;
-}
-
-.debug {
-    font-size: 10px;
-    color: #888;
-}
-
-.footer {
-    margin-top: 20px;
-    text-align: center;
-}

res/templates-dinsic/notif.html

@@ -1,45 +0,0 @@
-{% for message in notif.messages %}
-    <tr class="{{ "historical_message" if message.is_historical else "message" }}">
-        <td class="sender_avatar">
-            {% if loop.index0 == 0 or notif.messages[loop.index0 - 1].sender_name != notif.messages[loop.index0].sender_name %}
-                {% if message.sender_avatar_url %}
-                    <img alt="" class="sender_avatar" src="{{ message.sender_avatar_url|mxc_to_http(32,32) }}"  />
-                {% else %}
-                    {% if message.sender_hash % 3 == 0 %}
-                        <img class="sender_avatar" src="https://vector.im/beta/img/76cfa6.png"  />
-                    {% elif message.sender_hash % 3 == 1 %}
-                        <img class="sender_avatar" src="https://vector.im/beta/img/50e2c2.png"  />
-                    {% else %}
-                        <img class="sender_avatar" src="https://vector.im/beta/img/f4c371.png"  />
-                    {% endif %}
-                {% endif %}
-            {% endif %}
-        </td>
-        <td class="message_contents">
-            {% if loop.index0 == 0 or notif.messages[loop.index0 - 1].sender_name != notif.messages[loop.index0].sender_name %}
-                <div class="sender_name">{% if message.msgtype == "m.emote" %}*{% endif %} {{ message.sender_name }}</div>
-            {% endif %}
-            <div class="message_body">
-                {% if message.msgtype == "m.text" %}
-                    {{ message.body_text_html }}
-                {% elif message.msgtype == "m.emote" %}
-                    {{ message.body_text_html }}
-                {% elif message.msgtype == "m.notice" %}
-                    {{ message.body_text_html }}
-                {% elif message.msgtype == "m.image" %}
-                    <img src="{{ message.image_url|mxc_to_http(640, 480, scale) }}" />
-                {% elif message.msgtype == "m.file" %}
-                    <span class="filename">{{ message.body_text_plain }}</span>
-                {% endif %}
-            </div>
-        </td>
-        <td class="message_time">{{ message.ts|format_ts("%H:%M") }}</td>
-    </tr>
-{% endfor %}
-<tr class="notif_link">
-    <td></td>
-    <td>
-        <a href="{{ notif.link }}">Voir {{ room.title }}</a>
-    </td>
-    <td></td>
-</tr>

res/templates-dinsic/notif.txt

@@ -1,16 +0,0 @@
-{% for message in notif.messages %}
-{% if message.msgtype == "m.emote" %}* {% endif %}{{ message.sender_name }} ({{ message.ts|format_ts("%H:%M") }})
-{% if message.msgtype == "m.text" %}
-{{ message.body_text_plain }}
-{% elif message.msgtype == "m.emote" %}
-{{ message.body_text_plain }}
-{% elif message.msgtype == "m.notice" %}
-{{ message.body_text_plain }}
-{% elif message.msgtype == "m.image" %}
-{{ message.body_text_plain }}
-{% elif message.msgtype == "m.file" %}
-{{ message.body_text_plain }}
-{% endif %}
-{% endfor %}
-
-Voir {{ room.title }} à {{ notif.link }}

res/templates-dinsic/notif_mail.html

@@ -1,55 +0,0 @@
-<!doctype html>
-<html lang="en">
-    <head>
-        <style type="text/css">
-            {% include 'mail.css' without context %}
-            {% include "mail-%s.css" % app_name ignore missing without context %}
-        </style>
-    </head>
-    <body>
-        <table id="page">
-            <tr>
-                <td> </td>
-                <td id="inner">
-                    <table class="header">
-                        <tr>
-                            <td>
-                                <div class="salutation">Bonjour {{ user_display_name }},</div>
-                                <div class="summarytext">{{ summary_text }}</div>
-                            </td>
-                            <td class="logo">
-                                {% if app_name == "Riot" %}
-                                    <img src="http://matrix.org/img/riot-logo-email.png" width="83" height="83" alt="[Riot]"/>
-                                {% elif app_name == "Vector" %}
-                                    <img src="http://matrix.org/img/vector-logo-email.png" width="64" height="83" alt="[Vector]"/>
-                                {% else %}
-                                    <img src="http://matrix.org/img/matrix-120x51.png" width="120" height="51" alt="[matrix]"/>
-                                {% endif %}
-                            </td>
-                        </tr>
-                    </table>
-                    {% for room in rooms %}
-                        {% include 'room.html' with context %}
-                    {% endfor %}
-                    <div class="footer">
-                        <a href="{{ unsubscribe_link }}">Se désinscrire</a>
-                        <br/>
-                        <br/>
-                        <div class="debug">
-                            Sending email at {{ reason.now|format_ts("%c") }} due to activity in room {{ reason.room_name }} because
-                            an event was received at {{ reason.received_at|format_ts("%c") }}
-                            which is more than {{ "%.1f"|format(reason.delay_before_mail_ms / (60*1000)) }} ({{ reason.delay_before_mail_ms }}) mins ago,
-                            {% if reason.last_sent_ts %}
-                                and the last time we sent a mail for this room was {{ reason.last_sent_ts|format_ts("%c") }},
-                                which is more than {{ "%.1f"|format(reason.throttle_ms / (60*1000)) }} (current throttle_ms) mins ago.
-                            {% else %}
-                                and we don't have a last time we sent a mail for this room.
-                            {% endif %}
-                        </div>
-                    </div>
-                </td>
-                <td> </td>
-            </tr>
-        </table>
-    </body>
-</html>

res/templates-dinsic/notif_mail.txt

@@ -1,10 +0,0 @@
-Bonjour {{ user_display_name }},
-
-{{ summary_text }}
-
-{% for room in rooms %}
-{% include 'room.txt' with context %}
-{% endfor %}
-
-Vous pouvez désactiver ces notifications en cliquant ici {{ unsubscribe_link }}
-

res/templates-dinsic/room.html

@@ -1,33 +0,0 @@
-<table class="room">
-    <tr class="room_header">
-        <td class="room_avatar">
-            {% if room.avatar_url %}
-                <img alt="" src="{{ room.avatar_url|mxc_to_http(48,48) }}" />
-            {% else %}
-                {% if room.hash % 3 == 0 %}
-                    <img alt="" src="https://vector.im/beta/img/76cfa6.png"  />
-                {% elif room.hash % 3 == 1 %}
-                    <img alt="" src="https://vector.im/beta/img/50e2c2.png"  />
-                {% else %}
-                    <img alt="" src="https://vector.im/beta/img/f4c371.png"  />
-                {% endif %}
-            {% endif %}
-        </td>
-        <td class="room_name" colspan="2">
-            {{ room.title }}
-        </td>
-    </tr>
-    {% if room.invite %}
-        <tr>
-            <td></td>
-            <td>
-                <a href="{{ room.link }}">Rejoindre la conversation.</a>
-            </td>
-            <td></td>
-        </tr>
-    {% else %}
-        {% for notif in room.notifs %}
-            {% include 'notif.html' with context %}
-        {% endfor %}
-    {% endif %}
-</table>

res/templates-dinsic/room.txt

@@ -1,9 +0,0 @@
-{{ room.title }}
-
-{% if room.invite %}
-    Vous avez été invité, rejoignez la conversation en cliquant sur le lien suivant {{ room.link }}
-{% else %}
-    {% for notif in room.notifs %}
-        {% include 'notif.txt' with context %}
-    {% endfor %}
-{% endif %}

synapse/__init__.py

@@ -27,4 +27,4 @@ try:
 except ImportError:
     pass
 
-__version__ = "0.99.4"
+__version__ = "0.99.5.1"

synapse/api/auth.py

@@ -207,7 +207,6 @@ class Auth(object):
             )
 
             user_id, app_service = yield self._get_appservice_user_id(request)
-
             if user_id:
                 request.authenticated_entity = user_id
 
@@ -269,40 +268,39 @@ class Auth(object):
                 errcode=Codes.MISSING_TOKEN
             )
 
+    @defer.inlineCallbacks
     def _get_appservice_user_id(self, request):
         app_service = self.store.get_app_service_by_token(
             self.get_access_token_from_request(
                 request, self.TOKEN_NOT_FOUND_HTTP_STATUS
             )
         )
-
         if app_service is None:
-            return(None, None)
+            defer.returnValue((None, None))
 
         if app_service.ip_range_whitelist:
             ip_address = IPAddress(self.hs.get_ip_from_request(request))
             if ip_address not in app_service.ip_range_whitelist:
-                return(None, None)
+                defer.returnValue((None, None))
 
         if b"user_id" not in request.args:
-            return(app_service.sender, app_service)
+            defer.returnValue((app_service.sender, app_service))
 
         user_id = request.args[b"user_id"][0].decode('utf8')
         if app_service.sender == user_id:
-            return(app_service.sender, app_service)
+            defer.returnValue((app_service.sender, app_service))
 
         if not app_service.is_interested_in_user(user_id):
             raise AuthError(
                 403,
                 "Application service cannot masquerade as this user."
             )
-        # Let ASes manipulate nonexistent users (e.g. to shadow-register them)
-        # if not (yield self.store.get_user_by_id(user_id)):
-        #     raise AuthError(
-        #         403,
-        #         "Application service has not registered this user"
-        #     )
-        return(user_id, app_service)
+        if not (yield self.store.get_user_by_id(user_id)):
+            raise AuthError(
+                403,
+                "Application service has not registered this user"
+            )
+        defer.returnValue((user_id, app_service))
 
     @defer.inlineCallbacks
     def get_user_by_access_token(self, token, rights="access"):
@@ -535,15 +533,24 @@ class Auth(object):
         defer.returnValue(user_info)
 
     def get_appservice_by_req(self, request):
-        (user_id, app_service) = self._get_appservice_user_id(request)
-        if not app_service:
-            raise AuthError(
-                self.TOKEN_NOT_FOUND_HTTP_STATUS,
-                "Unrecognised access token.",
-                errcode=Codes.UNKNOWN_TOKEN,
+        try:
+            token = self.get_access_token_from_request(
+                request, self.TOKEN_NOT_FOUND_HTTP_STATUS
+            )
+            service = self.store.get_app_service_by_token(token)
+            if not service:
+                logger.warn("Unrecognised appservice access token.")
+                raise AuthError(
+                    self.TOKEN_NOT_FOUND_HTTP_STATUS,
+                    "Unrecognised access token.",
+                    errcode=Codes.UNKNOWN_TOKEN
+                )
+            request.authenticated_entity = service.sender
+            return defer.succeed(service)
+        except KeyError:
+            raise AuthError(
+                self.TOKEN_NOT_FOUND_HTTP_STATUS, "Missing access token."
             )
-        request.authenticated_entity = app_service.sender
-        return app_service
 
     def is_server_admin(self, user):
         """ Check if the given user is a local server admin.

synapse/api/constants.py

@@ -23,6 +23,9 @@ MAX_DEPTH = 2**63 - 1
 # the maximum length for a room alias is 255 characters
 MAX_ALIAS_LENGTH = 255
 
+# the maximum length for a user id is 255 characters
+MAX_USERID_LENGTH = 255
+
 
 class Membership(object):
 
@@ -76,10 +79,10 @@ class EventTypes(object):
 
     RoomHistoryVisibility = "m.room.history_visibility"
     CanonicalAlias = "m.room.canonical_alias"
+    Encryption = "m.room.encryption"
     RoomAvatar = "m.room.avatar"
     RoomEncryption = "m.room.encryption"
     GuestAccess = "m.room.guest_access"
-    Encryption = "m.room.encryption"
 
     # These are used for validation
     Message = "m.room.message"
@@ -117,3 +120,11 @@ class UserTypes(object):
     """
     SUPPORT = "support"
     ALL_USER_TYPES = (SUPPORT,)
+
+
+class RelationTypes(object):
+    """The types of relations known to this server.
+    """
+    ANNOTATION = "m.annotation"
+    REPLACE = "m.replace"
+    REFERENCE = "m.reference"

synapse/api/errors.py

@@ -1,7 +1,6 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2017-2018 New Vector Ltd
-# Copyright 2019 The Matrix.org Foundation C.I.C.
+# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -62,13 +61,6 @@ class Codes(object):
     INCOMPATIBLE_ROOM_VERSION = "M_INCOMPATIBLE_ROOM_VERSION"
     WRONG_ROOM_KEYS_VERSION = "M_WRONG_ROOM_KEYS_VERSION"
     EXPIRED_ACCOUNT = "ORG_MATRIX_EXPIRED_ACCOUNT"
-    PASSWORD_TOO_SHORT = "M_PASSWORD_TOO_SHORT"
-    PASSWORD_NO_DIGIT = "M_PASSWORD_NO_DIGIT"
-    PASSWORD_NO_UPPERCASE = "M_PASSWORD_NO_UPPERCASE"
-    PASSWORD_NO_LOWERCASE = "M_PASSWORD_NO_LOWERCASE"
-    PASSWORD_NO_SYMBOL = "M_PASSWORD_NO_SYMBOL"
-    PASSWORD_IN_DICTIONARY = "M_PASSWORD_IN_DICTIONARY"
-    WEAK_PASSWORD = "M_WEAK_PASSWORD"
 
 
 class CodeMessageException(RuntimeError):
@@ -336,9 +328,23 @@ class RoomKeysVersionError(SynapseError):
         self.current_version = current_version
 
 
-class IncompatibleRoomVersionError(SynapseError):
-    """A server is trying to join a room whose version it does not support."""
+class UnsupportedRoomVersionError(SynapseError):
+    """The client's request to create a room used a room version that the server does
+    not support."""
+    def __init__(self):
+        super(UnsupportedRoomVersionError, self).__init__(
+            code=400,
+            msg="Homeserver does not support this room version",
+            errcode=Codes.UNSUPPORTED_ROOM_VERSION,
+        )
 
+
+class IncompatibleRoomVersionError(SynapseError):
+    """A server is trying to join a room whose version it does not support.
+
+    Unlike UnsupportedRoomVersionError, it is specific to the case of the make_join
+    failing.
+    """
     def __init__(self, room_version):
         super(IncompatibleRoomVersionError, self).__init__(
             code=400,
@@ -357,22 +363,6 @@ class IncompatibleRoomVersionError(SynapseError):
         )
 
 
-class PasswordRefusedError(SynapseError):
-    """A password has been refused, either during password reset/change or registration.
-    """
-
-    def __init__(
-        self,
-        msg="This password doesn't comply with the server's policy",
-        errcode=Codes.WEAK_PASSWORD,
-    ):
-        super(PasswordRefusedError, self).__init__(
-            code=400,
-            msg=msg,
-            errcode=errcode,
-        )
-
-
 class RequestSendFailed(RuntimeError):
     """Sending a HTTP request over federation failed due to not being able to
     talk to the remote server for some reason.

synapse/api/room_versions.py

@@ -19,13 +19,15 @@ class EventFormatVersions(object):
     """This is an internal enum for tracking the version of the event format,
     independently from the room version.
     """
-    V1 = 1   # $id:server format
-    V2 = 2   # MSC1659-style $hash format: introduced for room v3
+    V1 = 1   # $id:server event id format
+    V2 = 2   # MSC1659-style $hash event id format: introduced for room v3
+    V3 = 3   # MSC1884-style $hash format: introduced for room v4
 
 
 KNOWN_EVENT_FORMAT_VERSIONS = {
     EventFormatVersions.V1,
     EventFormatVersions.V2,
+    EventFormatVersions.V3,
 }
 
 
@@ -75,6 +77,12 @@ class RoomVersions(object):
         EventFormatVersions.V2,
         StateResolutionVersions.V2,
     )
+    V4 = RoomVersion(
+        "4",
+        RoomDisposition.STABLE,
+        EventFormatVersions.V3,
+        StateResolutionVersions.V2,
+    )
 
 
 # the version we will give rooms which are created on this server
@@ -87,5 +95,6 @@ KNOWN_ROOM_VERSIONS = {
         RoomVersions.V2,
         RoomVersions.V3,
         RoomVersions.STATE_V2_TEST,
+        RoomVersions.V4,
     )
 }   # type: dict[str, RoomVersion]

synapse/app/client_reader.py

@@ -38,6 +38,7 @@ from synapse.replication.slave.storage.devices import SlavedDeviceStore
 from synapse.replication.slave.storage.directory import DirectoryStore
 from synapse.replication.slave.storage.events import SlavedEventStore
 from synapse.replication.slave.storage.keys import SlavedKeyStore
+from synapse.replication.slave.storage.profile import SlavedProfileStore
 from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
 from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
 from synapse.replication.slave.storage.registration import SlavedRegistrationStore
@@ -81,6 +82,7 @@ class ClientReaderSlavedStore(
     SlavedApplicationServiceStore,
     SlavedRegistrationStore,
     SlavedTransactionStore,
+    SlavedProfileStore,
     SlavedClientIpStore,
     BaseSlavedStore,
 ):

synapse/appservice/__init__.py

@@ -265,7 +265,7 @@ class ApplicationService(object):
     def is_exclusive_room(self, room_id):
         return self._is_exclusive(ApplicationService.NS_ROOMS, room_id)
 
-    def get_exclusive_user_regexes(self):
+    def get_exlusive_user_regexes(self):
         """Get the list of regexes used to determine if a user is exclusively
         registered by the AS
         """

synapse/config/homeserver.py

@@ -13,6 +13,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
 from .api import ApiConfig
 from .appservice import AppServiceConfig
 from .captcha import CaptchaConfig
@@ -36,20 +37,41 @@ from .saml2_config import SAML2Config
 from .server import ServerConfig
 from .server_notices_config import ServerNoticesConfig
 from .spam_checker import SpamCheckerConfig
+from .stats import StatsConfig
 from .tls import TlsConfig
 from .user_directory import UserDirectoryConfig
 from .voip import VoipConfig
 from .workers import WorkerConfig
 
 
-class HomeServerConfig(ServerConfig, TlsConfig, DatabaseConfig, LoggingConfig,
-                       RatelimitConfig, ContentRepositoryConfig, CaptchaConfig,
-                       VoipConfig, RegistrationConfig, MetricsConfig, ApiConfig,
-                       AppServiceConfig, KeyConfig, SAML2Config, CasConfig,
-                       JWTConfig, PasswordConfig, EmailConfig,
-                       WorkerConfig, PasswordAuthProviderConfig, PushConfig,
-                       SpamCheckerConfig, GroupsConfig, UserDirectoryConfig,
-                       ConsentConfig,
-                       ServerNoticesConfig, RoomDirectoryConfig,
-                       ):
+class HomeServerConfig(
+    ServerConfig,
+    TlsConfig,
+    DatabaseConfig,
+    LoggingConfig,
+    RatelimitConfig,
+    ContentRepositoryConfig,
+    CaptchaConfig,
+    VoipConfig,
+    RegistrationConfig,
+    MetricsConfig,
+    ApiConfig,
+    AppServiceConfig,
+    KeyConfig,
+    SAML2Config,
+    CasConfig,
+    JWTConfig,
+    PasswordConfig,
+    EmailConfig,
+    WorkerConfig,
+    PasswordAuthProviderConfig,
+    PushConfig,
+    SpamCheckerConfig,
+    GroupsConfig,
+    UserDirectoryConfig,
+    ConsentConfig,
+    StatsConfig,
+    ServerNoticesConfig,
+    RoomDirectoryConfig,
+):
     pass

synapse/config/password.py

@@ -1,7 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2015-2016 OpenMarket Ltd
-# Copyright 2017-2018 New Vector Ltd
-# Copyright 2019 The Matrix.org Foundation C.I.C.
+# Copyright 2015, 2016 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -30,10 +28,6 @@ class PasswordConfig(Config):
         self.password_enabled = password_config.get("enabled", True)
         self.password_pepper = password_config.get("pepper", "")
 
-        # Password policy
-        self.password_policy = password_config.get("policy", {})
-        self.password_policy_enabled = self.password_policy.pop("enabled", False)
-
     def default_config(self, config_dir_path, server_name, **kwargs):
         return """\
         password_config:
@@ -45,34 +39,4 @@ class PasswordConfig(Config):
            # DO NOT CHANGE THIS AFTER INITIAL SETUP!
            #
            #pepper: "EVEN_MORE_SECRET"
-
-           # Define and enforce a password policy. Each parameter is optional, boolean
-           # parameters default to 'false' and integer parameters default to 0.
-           # This is an early implementation of MSC2000.
-           #
-           #policy:
-              # Whether to enforce the password policy.
-              #
-              #enabled: true
-
-              # Minimum accepted length for a password.
-              #
-              #minimum_length: 15
-
-              # Whether a password must contain at least one digit.
-              #
-              #require_digit: true
-
-              # Whether a password must contain at least one symbol.
-              # A symbol is any character that's not a number or a letter.
-              #
-              #require_symbol: true
-
-              # Whether a password must contain at least one lowercase letter.
-              #
-              #require_lowercase: true
-
-              # Whether a password must contain at least one lowercase letter.
-              #
-              #require_uppercase: true
         """

synapse/config/registration.py

@@ -39,8 +39,6 @@ class AccountValidityConfig(Config):
             else:
                 self.renew_email_subject = "Renew your %(app)s account"
 
-            self.startup_job_max_delta = self.period * 10. / 100.
-
         if self.renew_by_email_enabled and "public_baseurl" not in synapse_config:
             raise ConfigError("Can't send renewal emails without 'public_baseurl'")
 
@@ -62,19 +60,8 @@ class RegistrationConfig(Config):
 
         self.registrations_require_3pid = config.get("registrations_require_3pid", [])
         self.allowed_local_3pids = config.get("allowed_local_3pids", [])
-        self.check_is_for_allowed_local_3pids = config.get(
-            "check_is_for_allowed_local_3pids", None
-        )
-        self.allow_invited_3pids = config.get("allow_invited_3pids", False)
-
-        self.disable_3pid_changes = config.get("disable_3pid_changes", False)
-
         self.enable_3pid_lookup = config.get("enable_3pid_lookup", True)
         self.registration_shared_secret = config.get("registration_shared_secret")
-        self.register_mxid_from_3pid = config.get("register_mxid_from_3pid")
-        self.register_just_use_email_for_display_name = config.get(
-            "register_just_use_email_for_display_name", False,
-        )
 
         self.bcrypt_rounds = config.get("bcrypt_rounds", 12)
         self.trusted_third_party_id_servers = config.get(
@@ -94,16 +81,6 @@ class RegistrationConfig(Config):
                 raise ConfigError('Invalid auto_join_rooms entry %s' % (room_alias,))
         self.autocreate_auto_join_rooms = config.get("autocreate_auto_join_rooms", True)
 
-        self.disable_set_displayname = config.get("disable_set_displayname", False)
-        self.disable_set_avatar_url = config.get("disable_set_avatar_url", False)
-
-        self.replicate_user_profiles_to = config.get("replicate_user_profiles_to", [])
-        if not isinstance(self.replicate_user_profiles_to, list):
-            self.replicate_user_profiles_to = [self.replicate_user_profiles_to, ]
-
-        self.shadow_server = config.get("shadow_server", None)
-        self.rewrite_identity_server_urls = config.get("rewrite_identity_server_urls", {})
-
         self.disable_msisdn_registration = (
             config.get("disable_msisdn_registration", False)
         )
@@ -152,9 +129,7 @@ class RegistrationConfig(Config):
         # This means that, if a validity period is set, and Synapse is restarted (it will
         # then derive an expiration date from the current validity period), and some time
         # after that the validity period changes and Synapse is restarted, the users'
-        # expiration dates won't be updated unless their account is manually renewed. This
-        # date will be randomly selected within a range [now + period - d ; now + period],
-        # where d is equal to 10%% of the validity period.
+        # expiration dates won't be updated unless their account is manually renewed.
         #
         #account_validity:
         #  enabled: True
@@ -173,32 +148,9 @@ class RegistrationConfig(Config):
         #
         #disable_msisdn_registration: true
 
-        # Derive the user's matrix ID from a type of 3PID used when registering.
-        # This overrides any matrix ID the user proposes when calling /register
-        # The 3PID type should be present in registrations_require_3pid to avoid
-        # users failing to register if they don't specify the right kind of 3pid.
-        #
-        #register_mxid_from_3pid: email
-
-        # Uncomment to set the display name of new users to their email address,
-        # rather than using the default heuristic.
-        #
-        #register_just_use_email_for_display_name: true
-
         # Mandate that users are only allowed to associate certain formats of
         # 3PIDs with accounts on this server.
         #
-        # Use an Identity Server to establish which 3PIDs are allowed to register?
-        # Overrides allowed_local_3pids below.
-        #
-        #check_is_for_allowed_local_3pids: matrix.org
-        #
-        # If you are using an IS you can also check whether that IS registers
-        # pending invites for the given 3PID (and then allow it to sign up on
-        # the platform):
-        #
-        #allow_invited_3pids: False
-        #
         #allowed_local_3pids:
         #  - medium: email
         #    pattern: '.*@matrix\\.org'
@@ -207,11 +159,6 @@ class RegistrationConfig(Config):
         #  - medium: msisdn
         #    pattern: '\\+44'
 
-        # If true, stop users from trying to change the 3PIDs associated with
-        # their accounts.
-        #
-        #disable_3pid_changes: False
-
         # Enable 3PIDs lookup requests to identity servers from this server.
         #
         #enable_3pid_lookup: true
@@ -253,30 +200,6 @@ class RegistrationConfig(Config):
         #  - matrix.org
         #  - vector.im
 
-        # If enabled, user IDs, display names and avatar URLs will be replicated
-        # to this server whenever they change.
-        # This is an experimental API currently implemented by sydent to support
-        # cross-homeserver user directories.
-        #
-        #replicate_user_profiles_to: example.com
-
-        # If specified, attempt to replay registrations, profile changes & 3pid
-        # bindings on the given target homeserver via the AS API. The HS is authed
-        # via a given AS token.
-        #
-        #shadow_server:
-        #  hs_url: https://shadow.example.com
-        #  hs: shadow.example.com
-        #  as_token: 12u394refgbdhivsia
-
-        # If enabled, don't let users set their own display names/avatars
-        # other than for the very first time (unless they are a server admin).
-        # Useful when provisioning users based on the contents of a 3rd party
-        # directory and to avoid ambiguities.
-        #
-        #disable_set_displayname: False
-        #disable_set_avatar_url: False
-
         # Users who register on this homeserver will automatically be joined
         # to these rooms
         #

synapse/config/server.py

@@ -101,6 +101,11 @@ class ServerConfig(Config):
             "block_non_admin_invites", False,
         )
 
+        # Whether to enable experimental MSC1849 (aka relations) support
+        self.experimental_msc1849_support_enabled = config.get(
+            "experimental_msc1849_support_enabled", False,
+        )
+
         # Options to control access by tracking MAU
         self.limit_usage_by_mau = config.get("limit_usage_by_mau", False)
         self.max_mau_value = 0

synapse/config/stats.py

@@ -0,0 +1,60 @@
+# -*- coding: utf-8 -*-
+# Copyright 2018 New Vector Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import division
+
+import sys
+
+from ._base import Config
+
+
+class StatsConfig(Config):
+    """Stats Configuration
+    Configuration for the behaviour of synapse's stats engine
+    """
+
+    def read_config(self, config):
+        self.stats_enabled = True
+        self.stats_bucket_size = 86400
+        self.stats_retention = sys.maxsize
+        stats_config = config.get("stats", None)
+        if stats_config:
+            self.stats_enabled = stats_config.get("enabled", self.stats_enabled)
+            self.stats_bucket_size = (
+                self.parse_duration(stats_config.get("bucket_size", "1d")) / 1000
+            )
+            self.stats_retention = (
+                self.parse_duration(
+                    stats_config.get("retention", "%ds" % (sys.maxsize,))
+                )
+                / 1000
+            )
+
+    def default_config(self, config_dir_path, server_name, **kwargs):
+        return """
+        # Local statistics collection. Used in populating the room directory.
+        #
+        # 'bucket_size' controls how large each statistics timeslice is. It can
+        # be defined in a human readable short form -- e.g. "1d", "1y".
+        #
+        # 'retention' controls how long historical statistics will be kept for.
+        # It can be defined in a human readable short form -- e.g. "1d", "1y".
+        #
+        #
+        #stats:
+        #   enabled: true
+        #   bucket_size: 1d
+        #   retention: 1y
+        """

synapse/config/user_directory.py

@@ -24,7 +24,6 @@ class UserDirectoryConfig(Config):
     def read_config(self, config):
         self.user_directory_search_enabled = True
         self.user_directory_search_all_users = False
-        self.user_directory_defer_to_id_server = None
         user_directory_config = config.get("user_directory", None)
         if user_directory_config:
             self.user_directory_search_enabled = (
@@ -33,9 +32,6 @@ class UserDirectoryConfig(Config):
             self.user_directory_search_all_users = (
                 user_directory_config.get("search_all_users", False)
             )
-            self.user_directory_defer_to_id_server = (
-                user_directory_config.get("defer_to_id_server", None)
-            )
 
     def default_config(self, config_dir_path, server_name, **kwargs):
         return """
@@ -54,9 +50,4 @@ class UserDirectoryConfig(Config):
         #user_directory:
         #  enabled: true
         #  search_all_users: false
-        #
-        #  # If this is set, user search will be delegated to this ID server instead
-        #  # of synapse performing the search itself.
-        #  # This is an experimental API.
-        #  defer_to_id_server: https://id.example.com
         """

synapse/crypto/keyring.py

@@ -56,9 +56,9 @@ from synapse.util.retryutils import NotRetryingDestination
 logger = logging.getLogger(__name__)
 
 
-VerifyKeyRequest = namedtuple("VerifyRequest", (
-    "server_name", "key_ids", "json_object", "deferred"
-))
+VerifyKeyRequest = namedtuple(
+    "VerifyRequest", ("server_name", "key_ids", "json_object", "deferred")
+)
 """
 A request for a verify key to verify a JSON object.
 
@@ -96,9 +96,7 @@ class Keyring(object):
 
     def verify_json_for_server(self, server_name, json_object):
         return logcontext.make_deferred_yieldable(
-            self.verify_json_objects_for_server(
-                [(server_name, json_object)]
-            )[0]
+            self.verify_json_objects_for_server([(server_name, json_object)])[0]
         )
 
     def verify_json_objects_for_server(self, server_and_json):
@@ -130,18 +128,15 @@ class Keyring(object):
             if not key_ids:
                 return defer.fail(
                     SynapseError(
-                        400,
-                        "Not signed by %s" % (server_name,),
-                        Codes.UNAUTHORIZED,
+                        400, "Not signed by %s" % (server_name,), Codes.UNAUTHORIZED
                     )
                 )
 
-            logger.debug("Verifying for %s with key_ids %s",
-                         server_name, key_ids)
+            logger.debug("Verifying for %s with key_ids %s", server_name, key_ids)
 
             # add the key request to the queue, but don't start it off yet.
             verify_request = VerifyKeyRequest(
-                server_name, key_ids, json_object, defer.Deferred(),
+                server_name, key_ids, json_object, defer.Deferred()
             )
             verify_requests.append(verify_request)
 
@@ -179,15 +174,13 @@ class Keyring(object):
             # any other lookups until we have finished.
             # The deferreds are called with no logcontext.
             server_to_deferred = {
-                rq.server_name: defer.Deferred()
-                for rq in verify_requests
+                rq.server_name: defer.Deferred() for rq in verify_requests
             }
 
             # We want to wait for any previous lookups to complete before
             # proceeding.
             yield self.wait_for_previous_lookups(
-                [rq.server_name for rq in verify_requests],
-                server_to_deferred,
+                [rq.server_name for rq in verify_requests], server_to_deferred
             )
 
             # Actually start fetching keys.
@@ -216,9 +209,7 @@ class Keyring(object):
                 return res
 
             for verify_request in verify_requests:
-                verify_request.deferred.addBoth(
-                    remove_deferreds, verify_request,
-                )
+                verify_request.deferred.addBoth(remove_deferreds, verify_request)
         except Exception:
             logger.exception("Error starting key lookups")
 
@@ -248,7 +239,8 @@ class Keyring(object):
                 break
             logger.info(
                 "Waiting for existing lookups for %s to complete [loop %i]",
-                [w[0] for w in wait_on], loop_count,
+                [w[0] for w in wait_on],
+                loop_count,
             )
             with PreserveLoggingContext():
                 yield defer.DeferredList((w[1] for w in wait_on))
@@ -335,13 +327,14 @@ class Keyring(object):
 
                 with PreserveLoggingContext():
                     for verify_request in requests_missing_keys:
-                        verify_request.deferred.errback(SynapseError(
-                            401,
-                            "No key for %s with id %s" % (
-                                verify_request.server_name, verify_request.key_ids,
-                            ),
-                            Codes.UNAUTHORIZED,
-                        ))
+                        verify_request.deferred.errback(
+                            SynapseError(
+                                401,
+                                "No key for %s with id %s"
+                                % (verify_request.server_name, verify_request.key_ids),
+                                Codes.UNAUTHORIZED,
+                            )
+                        )
 
         def on_err(err):
             with PreserveLoggingContext():
@@ -383,25 +376,26 @@ class Keyring(object):
                 )
                 defer.returnValue(result)
             except KeyLookupError as e:
-                logger.warning(
-                    "Key lookup failed from %r: %s", perspective_name, e,
-                )
+                logger.warning("Key lookup failed from %r: %s", perspective_name, e)
             except Exception as e:
                 logger.exception(
                     "Unable to get key from %r: %s %s",
                     perspective_name,
-                    type(e).__name__, str(e),
+                    type(e).__name__,
+                    str(e),
                 )
 
             defer.returnValue({})
 
-        results = yield logcontext.make_deferred_yieldable(defer.gatherResults(
-            [
-                run_in_background(get_key, p_name, p_keys)
-                for p_name, p_keys in self.perspective_servers.items()
-            ],
-            consumeErrors=True,
-        ).addErrback(unwrapFirstError))
+        results = yield logcontext.make_deferred_yieldable(
+            defer.gatherResults(
+                [
+                    run_in_background(get_key, p_name, p_keys)
+                    for p_name, p_keys in self.perspective_servers.items()
+                ],
+                consumeErrors=True,
+            ).addErrback(unwrapFirstError)
+        )
 
         union_of_keys = {}
         for result in results:
@@ -412,32 +406,30 @@ class Keyring(object):
 
     @defer.inlineCallbacks
     def get_keys_from_server(self, server_name_and_key_ids):
-        results = yield logcontext.make_deferred_yieldable(defer.gatherResults(
-            [
-                run_in_background(
-                    self.get_server_verify_key_v2_direct,
-                    server_name,
-                    key_ids,
-                )
-                for server_name, key_ids in server_name_and_key_ids
-            ],
-            consumeErrors=True,
-        ).addErrback(unwrapFirstError))
+        results = yield logcontext.make_deferred_yieldable(
+            defer.gatherResults(
+                [
+                    run_in_background(
+                        self.get_server_verify_key_v2_direct, server_name, key_ids
+                    )
+                    for server_name, key_ids in server_name_and_key_ids
+                ],
+                consumeErrors=True,
+            ).addErrback(unwrapFirstError)
+        )
 
         merged = {}
         for result in results:
             merged.update(result)
 
-        defer.returnValue({
-            server_name: keys
-            for server_name, keys in merged.items()
-            if keys
-        })
+        defer.returnValue(
+            {server_name: keys for server_name, keys in merged.items() if keys}
+        )
 
     @defer.inlineCallbacks
-    def get_server_verify_key_v2_indirect(self, server_names_and_key_ids,
-                                          perspective_name,
-                                          perspective_keys):
+    def get_server_verify_key_v2_indirect(
+        self, server_names_and_key_ids, perspective_name, perspective_keys
+    ):
         # TODO(mark): Set the minimum_valid_until_ts to that needed by
         # the events being validated or the current time if validating
         # an incoming request.
@@ -448,9 +440,7 @@ class Keyring(object):
                 data={
                     u"server_keys": {
                         server_name: {
-                            key_id: {
-                                u"minimum_valid_until_ts": 0
-                            } for key_id in key_ids
+                            key_id: {u"minimum_valid_until_ts": 0} for key_id in key_ids
                         }
                         for server_name, key_ids in server_names_and_key_ids
                     }
@@ -458,21 +448,19 @@ class Keyring(object):
                 long_retries=True,
             )
         except (NotRetryingDestination, RequestSendFailed) as e:
-            raise_from(
-                KeyLookupError("Failed to connect to remote server"), e,
-            )
+            raise_from(KeyLookupError("Failed to connect to remote server"), e)
         except HttpResponseException as e:
-            raise_from(
-                KeyLookupError("Remote server returned an error"), e,
-            )
+            raise_from(KeyLookupError("Remote server returned an error"), e)
 
         keys = {}
 
         responses = query_response["server_keys"]
 
         for response in responses:
-            if (u"signatures" not in response
-                    or perspective_name not in response[u"signatures"]):
+            if (
+                u"signatures" not in response
+                or perspective_name not in response[u"signatures"]
+            ):
                 raise KeyLookupError(
                     "Key response not signed by perspective server"
                     " %r" % (perspective_name,)
@@ -482,9 +470,7 @@ class Keyring(object):
             for key_id in response[u"signatures"][perspective_name]:
                 if key_id in perspective_keys:
                     verify_signed_json(
-                        response,
-                        perspective_name,
-                        perspective_keys[key_id]
+                        response, perspective_name, perspective_keys[key_id]
                     )
                     verified = True
 
@@ -494,7 +480,7 @@ class Keyring(object):
                     " known key, signed with: %r, known keys: %r",
                     perspective_name,
                     list(response[u"signatures"][perspective_name]),
-                    list(perspective_keys)
+                    list(perspective_keys),
                 )
                 raise KeyLookupError(
                     "Response not signed with a known key for perspective"
@@ -508,18 +494,20 @@ class Keyring(object):
 
             keys.setdefault(server_name, {}).update(processed_response)
 
-        yield logcontext.make_deferred_yieldable(defer.gatherResults(
-            [
-                run_in_background(
-                    self.store_keys,
-                    server_name=server_name,
-                    from_server=perspective_name,
-                    verify_keys=response_keys,
-                )
-                for server_name, response_keys in keys.items()
-            ],
-            consumeErrors=True
-        ).addErrback(unwrapFirstError))
+        yield logcontext.make_deferred_yieldable(
+            defer.gatherResults(
+                [
+                    run_in_background(
+                        self.store_keys,
+                        server_name=server_name,
+                        from_server=perspective_name,
+                        verify_keys=response_keys,
+                    )
+                    for server_name, response_keys in keys.items()
+                ],
+                consumeErrors=True,
+            ).addErrback(unwrapFirstError)
+        )
 
         defer.returnValue(keys)
 
@@ -534,26 +522,26 @@ class Keyring(object):
             try:
                 response = yield self.client.get_json(
                     destination=server_name,
-                    path="/_matrix/key/v2/server/" + urllib.parse.quote(requested_key_id),
+                    path="/_matrix/key/v2/server/"
+                    + urllib.parse.quote(requested_key_id),
                     ignore_backoff=True,
                 )
             except (NotRetryingDestination, RequestSendFailed) as e:
-                raise_from(
-                    KeyLookupError("Failed to connect to remote server"), e,
-                )
+                raise_from(KeyLookupError("Failed to connect to remote server"), e)
             except HttpResponseException as e:
-                raise_from(
-                    KeyLookupError("Remote server returned an error"), e,
-                )
+                raise_from(KeyLookupError("Remote server returned an error"), e)
 
-            if (u"signatures" not in response
-                    or server_name not in response[u"signatures"]):
+            if (
+                u"signatures" not in response
+                or server_name not in response[u"signatures"]
+            ):
                 raise KeyLookupError("Key response not signed by remote server")
 
             if response["server_name"] != server_name:
-                raise KeyLookupError("Expected a response for server %r not %r" % (
-                    server_name, response["server_name"]
-                ))
+                raise KeyLookupError(
+                    "Expected a response for server %r not %r"
+                    % (server_name, response["server_name"])
+                )
 
             response_keys = yield self.process_v2_response(
                 from_server=server_name,
@@ -564,16 +552,12 @@ class Keyring(object):
             keys.update(response_keys)
 
         yield self.store_keys(
-            server_name=server_name,
-            from_server=server_name,
-            verify_keys=keys,
+            server_name=server_name, from_server=server_name, verify_keys=keys
         )
         defer.returnValue({server_name: keys})
 
     @defer.inlineCallbacks
-    def process_v2_response(
-        self, from_server, response_json, requested_ids=[],
-    ):
+    def process_v2_response(self, from_server, response_json, requested_ids=[]):
         """Parse a 'Server Keys' structure from the result of a /key request
 
         This is used to parse either the entirety of the response from
@@ -627,20 +611,13 @@ class Keyring(object):
         for key_id in response_json["signatures"].get(server_name, {}):
             if key_id not in response_json["verify_keys"]:
                 raise KeyLookupError(
-                    "Key response must include verification keys for all"
-                    " signatures"
+                    "Key response must include verification keys for all" " signatures"
                 )
             if key_id in verify_keys:
-                verify_signed_json(
-                    response_json,
-                    server_name,
-                    verify_keys[key_id]
-                )
+                verify_signed_json(response_json, server_name, verify_keys[key_id])
 
         signed_key_json = sign_json(
-            response_json,
-            self.config.server_name,
-            self.config.signing_key[0],
+            response_json, self.config.server_name, self.config.signing_key[0]
         )
 
         signed_key_json_bytes = encode_canonical_json(signed_key_json)
@@ -653,21 +630,23 @@ class Keyring(object):
         response_keys.update(verify_keys)
         response_keys.update(old_verify_keys)
 
-        yield logcontext.make_deferred_yieldable(defer.gatherResults(
-            [
-                run_in_background(
-                    self.store.store_server_keys_json,
-                    server_name=server_name,
-                    key_id=key_id,
-                    from_server=from_server,
-                    ts_now_ms=time_now_ms,
-                    ts_expires_ms=ts_valid_until_ms,
-                    key_json_bytes=signed_key_json_bytes,
-                )
-                for key_id in updated_key_ids
-            ],
-            consumeErrors=True,
-        ).addErrback(unwrapFirstError))
+        yield logcontext.make_deferred_yieldable(
+            defer.gatherResults(
+                [
+                    run_in_background(
+                        self.store.store_server_keys_json,
+                        server_name=server_name,
+                        key_id=key_id,
+                        from_server=from_server,
+                        ts_now_ms=time_now_ms,
+                        ts_expires_ms=ts_valid_until_ms,
+                        key_json_bytes=signed_key_json_bytes,
+                    )
+                    for key_id in updated_key_ids
+                ],
+                consumeErrors=True,
+            ).addErrback(unwrapFirstError)
+        )
 
         defer.returnValue(response_keys)
 
@@ -681,16 +660,21 @@ class Keyring(object):
             A deferred that completes when the keys are stored.
         """
         # TODO(markjh): Store whether the keys have expired.
-        return logcontext.make_deferred_yieldable(defer.gatherResults(
-            [
-                run_in_background(
-                    self.store.store_server_verify_key,
-                    server_name, server_name, key.time_added, key
-                )
-                for key_id, key in verify_keys.items()
-            ],
-            consumeErrors=True,
-        ).addErrback(unwrapFirstError))
+        return logcontext.make_deferred_yieldable(
+            defer.gatherResults(
+                [
+                    run_in_background(
+                        self.store.store_server_verify_key,
+                        server_name,
+                        server_name,
+                        key.time_added,
+                        key,
+                    )
+                    for key_id, key in verify_keys.items()
+                ],
+                consumeErrors=True,
+            ).addErrback(unwrapFirstError)
+        )
 
 
 @defer.inlineCallbacks
@@ -713,17 +697,19 @@ def _handle_key_deferred(verify_request):
     except KeyLookupError as e:
         logger.warn(
             "Failed to download keys for %s: %s %s",
-            server_name, type(e).__name__, str(e),
+            server_name,
+            type(e).__name__,
+            str(e),
         )
         raise SynapseError(
-            502,
-            "Error downloading keys for %s" % (server_name,),
-            Codes.UNAUTHORIZED,
+            502, "Error downloading keys for %s" % (server_name,), Codes.UNAUTHORIZED
         )
     except Exception as e:
         logger.exception(
             "Got Exception when downloading keys for %s: %s %s",
-            server_name, type(e).__name__, str(e),
+            server_name,
+            type(e).__name__,
+            str(e),
         )
         raise SynapseError(
             401,
@@ -733,22 +719,24 @@ def _handle_key_deferred(verify_request):
 
     json_object = verify_request.json_object
 
-    logger.debug("Got key %s %s:%s for server %s, verifying" % (
-        key_id, verify_key.alg, verify_key.version, server_name,
-    ))
+    logger.debug(
+        "Got key %s %s:%s for server %s, verifying"
+        % (key_id, verify_key.alg, verify_key.version, server_name)
+    )
     try:
         verify_signed_json(json_object, server_name, verify_key)
     except SignatureVerifyException as e:
         logger.debug(
             "Error verifying signature for %s:%s:%s with key %s: %s",
-            server_name, verify_key.alg, verify_key.version,
+            server_name,
+            verify_key.alg,
+            verify_key.version,
             encode_verify_key_base64(verify_key),
             str(e),
         )
         raise SynapseError(
             401,
-            "Invalid signature for server %s with key %s:%s: %s" % (
-                server_name, verify_key.alg, verify_key.version, str(e),
-            ),
+            "Invalid signature for server %s with key %s:%s: %s"
+            % (server_name, verify_key.alg, verify_key.version, str(e)),
             Codes.UNAUTHORIZED,
         )

synapse/events/__init__.py

@@ -21,6 +21,7 @@ import six
 
 from unpaddedbase64 import encode_base64
 
+from synapse.api.errors import UnsupportedRoomVersionError
 from synapse.api.room_versions import KNOWN_ROOM_VERSIONS, EventFormatVersions
 from synapse.util.caches import intern_dict
 from synapse.util.frozenutils import freeze
@@ -335,13 +336,32 @@ class FrozenEventV2(EventBase):
         return self.__repr__()
 
     def __repr__(self):
-        return "<FrozenEventV2 event_id='%s', type='%s', state_key='%s'>" % (
+        return "<%s event_id='%s', type='%s', state_key='%s'>" % (
+            self.__class__.__name__,
             self.event_id,
             self.get("type", None),
             self.get("state_key", None),
         )
 
 
+class FrozenEventV3(FrozenEventV2):
+    """FrozenEventV3, which differs from FrozenEventV2 only in the event_id format"""
+    format_version = EventFormatVersions.V3  # All events of this type are V3
+
+    @property
+    def event_id(self):
+        # We have to import this here as otherwise we get an import loop which
+        # is hard to break.
+        from synapse.crypto.event_signing import compute_event_reference_hash
+
+        if self._event_id:
+            return self._event_id
+        self._event_id = "$" + encode_base64(
+            compute_event_reference_hash(self)[1], urlsafe=True
+        )
+        return self._event_id
+
+
 def room_version_to_event_format(room_version):
     """Converts a room version string to the event format
 
@@ -350,12 +370,15 @@ def room_version_to_event_format(room_version):
 
     Returns:
         int
+
+    Raises:
+        UnsupportedRoomVersionError if the room version is unknown
     """
     v = KNOWN_ROOM_VERSIONS.get(room_version)
 
     if not v:
-        # We should have already checked version, so this should not happen
-        raise RuntimeError("Unrecognized room version %s" % (room_version,))
+        # this can happen if support is withdrawn for a room version
+        raise UnsupportedRoomVersionError()
 
     return v.event_format
 
@@ -376,6 +399,8 @@ def event_type_from_format_version(format_version):
         return FrozenEvent
     elif format_version == EventFormatVersions.V2:
         return FrozenEventV2
+    elif format_version == EventFormatVersions.V3:
+        return FrozenEventV3
     else:
         raise Exception(
             "No event format %r" % (format_version,)

synapse/events/builder.py

@@ -18,6 +18,7 @@ import attr
 from twisted.internet import defer
 
 from synapse.api.constants import MAX_DEPTH
+from synapse.api.errors import UnsupportedRoomVersionError
 from synapse.api.room_versions import (
     KNOWN_EVENT_FORMAT_VERSIONS,
     KNOWN_ROOM_VERSIONS,
@@ -178,9 +179,8 @@ class EventBuilderFactory(object):
         """
         v = KNOWN_ROOM_VERSIONS.get(room_version)
         if not v:
-            raise Exception(
-                "No event format defined for version %r" % (room_version,)
-            )
+            # this can happen if support is withdrawn for a room version
+            raise UnsupportedRoomVersionError()
         return self.for_room_version(v, key_values)
 
     def for_room_version(self, room_version, key_values):

synapse/events/spamcheck.py

@@ -46,26 +46,13 @@ class SpamChecker(object):
 
         return self.spam_checker.check_event_for_spam(event)
 
-    def user_may_invite(self, inviter_userid, invitee_userid, third_party_invite,
-                        room_id, new_room, published_room):
+    def user_may_invite(self, inviter_userid, invitee_userid, room_id):
         """Checks if a given user may send an invite
 
         If this method returns false, the invite will be rejected.
 
         Args:
-            inviter_userid (str)
-            invitee_userid (str|None): The user ID of the invitee. Is None
-                if this is a third party invite and the 3PID is not bound to a
-                user ID.
-            third_party_invite (dict|None): If a third party invite then is a
-                dict containing the medium and address of the invitee.
-            room_id (str)
-            new_room (bool): Whether the user is being invited to the room as
-                part of a room creation, if so the invitee would have been
-                included in the call to `user_may_create_room`.
-            published_room (bool): Whether the room the user is being invited
-                to has been published in the local homeserver's public room
-                directory.
+            userid (string): The sender's user ID
 
         Returns:
             bool: True if the user may send an invite, otherwise False
@@ -73,25 +60,15 @@ class SpamChecker(object):
         if self.spam_checker is None:
             return True
 
-        return self.spam_checker.user_may_invite(
-            inviter_userid, invitee_userid, third_party_invite, room_id, new_room,
-            published_room,
-        )
+        return self.spam_checker.user_may_invite(inviter_userid, invitee_userid, room_id)
 
-    def user_may_create_room(self, userid, invite_list, third_party_invite_list,
-                             cloning):
+    def user_may_create_room(self, userid):
         """Checks if a given user may create a room
 
         If this method returns false, the creation request will be rejected.
 
         Args:
             userid (string): The sender's user ID
-            invite_list (list[str]): List of user IDs that would be invited to
-                the new room.
-            third_party_invite_list (list[dict]): List of third party invites
-                for the new room.
-            cloning (bool): Whether the user is cloning an existing room, e.g.
-                upgrading a room.
 
         Returns:
             bool: True if the user may create a room, otherwise False
@@ -99,9 +76,7 @@ class SpamChecker(object):
         if self.spam_checker is None:
             return True
 
-        return self.spam_checker.user_may_create_room(
-            userid, invite_list, third_party_invite_list, cloning,
-        )
+        return self.spam_checker.user_may_create_room(userid)
 
     def user_may_create_room_alias(self, userid, room_alias):
         """Checks if a given user may create a room alias
@@ -136,21 +111,3 @@ class SpamChecker(object):
             return True
 
         return self.spam_checker.user_may_publish_room(userid, room_id)
-
-    def user_may_join_room(self, userid, room_id, is_invited):
-        """Checks if a given users is allowed to join a room.
-
-        Is not called when the user creates a room.
-
-        Args:
-            userid (str)
-            room_id (str)
-            is_invited (bool): Whether the user is invited into the room
-
-        Returns:
-            bool: Whether the user may join the room
-        """
-        if self.spam_checker is None:
-            return True
-
-        return self.spam_checker.user_may_join_room(userid, room_id, is_invited)

synapse/events/utils.py

@@ -21,7 +21,7 @@ from frozendict import frozendict
 
 from twisted.internet import defer
 
-from synapse.api.constants import EventTypes
+from synapse.api.constants import EventTypes, RelationTypes
 from synapse.util.async_helpers import yieldable_gather_results
 
 from . import EventBase
@@ -324,8 +324,12 @@ class EventClientSerializer(object):
     """
 
     def __init__(self, hs):
-        pass
+        self.store = hs.get_datastore()
+        self.experimental_msc1849_support_enabled = (
+            hs.config.experimental_msc1849_support_enabled
+        )
 
+    @defer.inlineCallbacks
     def serialize_event(self, event, time_now, **kwargs):
         """Serializes a single event.
 
@@ -337,8 +341,52 @@ class EventClientSerializer(object):
         Returns:
             Deferred[dict]: The serialized event
         """
-        event = serialize_event(event, time_now, **kwargs)
-        return defer.succeed(event)
+        # To handle the case of presence events and the like
+        if not isinstance(event, EventBase):
+            defer.returnValue(event)
+
+        event_id = event.event_id
+        serialized_event = serialize_event(event, time_now, **kwargs)
+
+        # If MSC1849 is enabled then we need to look if thre are any relations
+        # we need to bundle in with the event
+        if self.experimental_msc1849_support_enabled:
+            annotations = yield self.store.get_aggregation_groups_for_event(
+                event_id,
+            )
+            references = yield self.store.get_relations_for_event(
+                event_id, RelationTypes.REFERENCE, direction="f",
+            )
+
+            if annotations.chunk:
+                r = serialized_event["unsigned"].setdefault("m.relations", {})
+                r[RelationTypes.ANNOTATION] = annotations.to_dict()
+
+            if references.chunk:
+                r = serialized_event["unsigned"].setdefault("m.relations", {})
+                r[RelationTypes.REFERENCE] = references.to_dict()
+
+            edit = None
+            if event.type == EventTypes.Message:
+                edit = yield self.store.get_applicable_edit(event_id)
+
+            if edit:
+                # If there is an edit replace the content, preserving existing
+                # relations.
+
+                relations = event.content.get("m.relates_to")
+                serialized_event["content"] = edit.content.get("m.new_content", {})
+                if relations:
+                    serialized_event["content"]["m.relates_to"] = relations
+                else:
+                    serialized_event["content"].pop("m.relates_to", None)
+
+                r = serialized_event["unsigned"].setdefault("m.relations", {})
+                r[RelationTypes.REPLACE] = {
+                    "event_id": edit.event_id,
+                }
+
+        defer.returnValue(serialized_event)
 
     def serialize_events(self, events, time_now, **kwargs):
         """Serializes multiple events.

synapse/federation/federation_server.py

@@ -33,6 +33,7 @@ from synapse.api.errors import (
     IncompatibleRoomVersionError,
     NotFoundError,
     SynapseError,
+    UnsupportedRoomVersionError,
 )
 from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
 from synapse.crypto.event_signing import compute_event_signature
@@ -198,11 +199,22 @@ class FederationServer(FederationBase):
 
             try:
                 room_version = yield self.store.get_room_version(room_id)
-                format_ver = room_version_to_event_format(room_version)
             except NotFoundError:
                 logger.info("Ignoring PDU for unknown room_id: %s", room_id)
                 continue
 
+            try:
+                format_ver = room_version_to_event_format(room_version)
+            except UnsupportedRoomVersionError:
+                # this can happen if support for a given room version is withdrawn,
+                # so that we still get events for said room.
+                logger.info(
+                    "Ignoring PDU for room %s with unknown version %s",
+                    room_id,
+                    room_version,
+                )
+                continue
+
             event = event_from_pdu_json(p, format_ver)
             pdus_by_room.setdefault(room_id, []).append(event)
 

synapse/handlers/deactivate_account.py

@@ -33,7 +33,6 @@ class DeactivateAccountHandler(BaseHandler):
         self._device_handler = hs.get_device_handler()
         self._room_member_handler = hs.get_room_member_handler()
         self._identity_handler = hs.get_handlers().identity_handler
-        self._profile_handler = hs.get_profile_handler()
         self.user_directory_handler = hs.get_user_directory_handler()
 
         # Flag that indicates whether the process to part users from rooms is running
@@ -99,9 +98,6 @@ class DeactivateAccountHandler(BaseHandler):
 
         yield self.store.user_set_password_hash(user_id, None)
 
-        user = UserID.from_string(user_id)
-        yield self._profile_handler.set_active(user, False, False)
-
         # Add the user to a table of users pending deactivation (ie.
         # removal from all the rooms they're a member of)
         yield self.store.add_user_pending_deactivation(user_id)

synapse/handlers/device.py

@@ -568,12 +568,6 @@ class DeviceListEduUpdater(object):
                 stream_id = result["stream_id"]
                 devices = result["devices"]
 
-                for device in devices:
-                    logger.debug(
-                        "Handling resync update %r/%r, ID: %r",
-                        user_id, device["device_id"], stream_id,
-                    )
-
                 # If the remote server has more than ~1000 devices for this user
                 # we assume that something is going horribly wrong (e.g. a bot
                 # that logs in and creates a new device every time it tries to

synapse/handlers/federation.py

@@ -1340,12 +1340,8 @@ class FederationHandler(BaseHandler):
         if self.hs.config.block_non_admin_invites:
             raise SynapseError(403, "This server does not accept room invites")
 
-        is_published = yield self.store.is_room_published(event.room_id)
-
         if not self.spam_checker.user_may_invite(
-            event.sender, event.state_key, None,
-            room_id=event.room_id, new_room=False,
-            published_room=is_published,
+            event.sender, event.state_key, event.room_id,
         ):
             raise SynapseError(
                 403, "This user is not permitted to send invites to this server/user"
@@ -1920,6 +1916,11 @@ class FederationHandler(BaseHandler):
                     event.room_id, latest_event_ids=extrem_ids,
                 )
 
+            logger.debug(
+                "Doing soft-fail check for %s: state %s",
+                event.event_id, current_state_ids,
+            )
+
             # Now check if event pass auth against said current state
             auth_types = auth_types_for_event(event)
             current_state_ids = [
@@ -1936,7 +1937,7 @@ class FederationHandler(BaseHandler):
                 self.auth.check(room_version, event, auth_events=current_auth_events)
             except AuthError as e:
                 logger.warn(
-                    "Failed current state auth resolution for %r because %s",
+                    "Soft-failing %r because %s",
                     event, e,
                 )
                 event.internal_metadata.soft_failed = True

synapse/handlers/identity.py

@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 # Copyright 2015, 2016 OpenMarket Ltd
 # Copyright 2017 Vector Creations Ltd
-# Copyright 2018, 2019 New Vector Ltd
+# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -20,18 +20,13 @@
 import logging
 
 from canonicaljson import json
-from signedjson.key import decode_verify_key_bytes
-from signedjson.sign import verify_signed_json
-from unpaddedbase64 import decode_base64
 
 from twisted.internet import defer
 
 from synapse.api.errors import (
-    AuthError,
     CodeMessageException,
     Codes,
     HttpResponseException,
-    ProxiedRequestError,
     SynapseError,
 )
 
@@ -52,8 +47,6 @@ class IdentityHandler(BaseHandler):
         self.trust_any_id_server_just_for_testing_do_not_use = (
             hs.config.use_insecure_ssl_client_just_for_testing_do_not_use
         )
-        self.rewrite_identity_server_urls = hs.config.rewrite_identity_server_urls
-        self._enable_lookup = hs.config.enable_3pid_lookup
 
     def _should_trust_id_server(self, id_server):
         if id_server not in self.trusted_id_servers:
@@ -91,10 +84,7 @@ class IdentityHandler(BaseHandler):
                 'credentials', id_server
             )
             defer.returnValue(None)
-        # if we have a rewrite rule set for the identity server,
-        # apply it now.
-        if id_server in self.rewrite_identity_server_urls:
-            id_server = self.rewrite_identity_server_urls[id_server]
+
         try:
             data = yield self.http_client.get_json(
                 "https://%s%s" % (
@@ -129,10 +119,7 @@ class IdentityHandler(BaseHandler):
             client_secret = creds['clientSecret']
         else:
             raise SynapseError(400, "No client_secret in creds")
-        # if we have a rewrite rule set for the identity server,
-        # apply it now.
-        if id_server in self.rewrite_identity_server_urls:
-            id_server = self.rewrite_identity_server_urls[id_server]
+
         try:
             data = yield self.http_client.post_urlencoded_get_json(
                 "https://%s%s" % (
@@ -234,16 +221,6 @@ class IdentityHandler(BaseHandler):
             b"Authorization": auth_headers,
         }
 
-        # if we have a rewrite rule set for the identity server,
-        # apply it now.
-        #
-        # Note that destination_is has to be the real id_server, not
-        # the server we connect to.
-        if id_server in self.rewrite_identity_server_urls:
-            id_server = self.rewrite_identity_server_urls[id_server]
-
-        url = "https://%s/_matrix/identity/api/v1/3pid/unbind" % (id_server,)
-
         try:
             yield self.http_client.post_json_get_json(
                 url,
@@ -283,10 +260,7 @@ class IdentityHandler(BaseHandler):
             'send_attempt': send_attempt,
         }
         params.update(kwargs)
-        # if we have a rewrite rule set for the identity server,
-        # apply it now.
-        if id_server in self.rewrite_identity_server_urls:
-            id_server = self.rewrite_identity_server_urls[id_server]
+
         try:
             data = yield self.http_client.post_json_get_json(
                 "https://%s%s" % (
@@ -318,10 +292,7 @@ class IdentityHandler(BaseHandler):
             'send_attempt': send_attempt,
         }
         params.update(kwargs)
-        # if we have a rewrite rule set for the identity server,
-        # apply it now.
-        if id_server in self.rewrite_identity_server_urls:
-            id_server = self.rewrite_identity_server_urls[id_server]
+
         try:
             data = yield self.http_client.post_json_get_json(
                 "https://%s%s" % (
@@ -334,125 +305,3 @@ class IdentityHandler(BaseHandler):
         except HttpResponseException as e:
             logger.info("Proxied requestToken failed: %r", e)
             raise e.to_synapse_error()
-
-    @defer.inlineCallbacks
-    def lookup_3pid(self, id_server, medium, address):
-        """Looks up a 3pid in the passed identity server.
-
-        Args:
-            id_server (str): The server name (including port, if required)
-                of the identity server to use.
-            medium (str): The type of the third party identifier (e.g. "email").
-            address (str): The third party identifier (e.g. "foo@example.com").
-
-        Returns:
-            Deferred[dict]: The result of the lookup. See
-            https://matrix.org/docs/spec/identity_service/r0.1.0.html#association-lookup
-            for details
-        """
-        if not self._should_trust_id_server(id_server):
-            raise SynapseError(
-                400, "Untrusted ID server '%s'" % id_server,
-                Codes.SERVER_NOT_TRUSTED
-            )
-
-        if not self._enable_lookup:
-            raise AuthError(
-                403, "Looking up third-party identifiers is denied from this server",
-            )
-
-        target = self.rewrite_identity_server_urls.get(id_server, id_server)
-
-        try:
-            data = yield self.http_client.get_json(
-                "https://%s/_matrix/identity/api/v1/lookup" % (target,),
-                {
-                    "medium": medium,
-                    "address": address,
-                }
-            )
-
-            if "mxid" in data:
-                if "signatures" not in data:
-                    raise AuthError(401, "No signatures on 3pid binding")
-                yield self._verify_any_signature(data, id_server)
-
-        except HttpResponseException as e:
-            logger.info("Proxied lookup failed: %r", e)
-            raise e.to_synapse_error()
-        except IOError as e:
-            logger.info("Failed to contact %r: %s", id_server, e)
-            raise ProxiedRequestError(503, "Failed to contact identity server")
-
-        defer.returnValue(data)
-
-    @defer.inlineCallbacks
-    def bulk_lookup_3pid(self, id_server, threepids):
-        """Looks up given 3pids in the passed identity server.
-
-        Args:
-            id_server (str): The server name (including port, if required)
-                of the identity server to use.
-            threepids ([[str, str]]): The third party identifiers to lookup, as
-                a list of 2-string sized lists ([medium, address]).
-
-        Returns:
-            Deferred[dict]: The result of the lookup. See
-            https://matrix.org/docs/spec/identity_service/r0.1.0.html#association-lookup
-            for details
-        """
-        if not self._should_trust_id_server(id_server):
-            raise SynapseError(
-                400, "Untrusted ID server '%s'" % id_server,
-                Codes.SERVER_NOT_TRUSTED
-            )
-
-        if not self._enable_lookup:
-            raise AuthError(
-                403, "Looking up third-party identifiers is denied from this server",
-            )
-
-        target = self.rewrite_identity_server_urls.get(id_server, id_server)
-
-        try:
-            data = yield self.http_client.post_json_get_json(
-                "https://%s/_matrix/identity/api/v1/bulk_lookup" % (target,),
-                {
-                    "threepids": threepids,
-                }
-            )
-
-        except HttpResponseException as e:
-            logger.info("Proxied lookup failed: %r", e)
-            raise e.to_synapse_error()
-        except IOError as e:
-            logger.info("Failed to contact %r: %s", id_server, e)
-            raise ProxiedRequestError(503, "Failed to contact identity server")
-
-        defer.returnValue(data)
-
-    @defer.inlineCallbacks
-    def _verify_any_signature(self, data, server_hostname):
-        if server_hostname not in data["signatures"]:
-            raise AuthError(401, "No signature from server %s" % (server_hostname,))
-
-        for key_name, signature in data["signatures"][server_hostname].items():
-            target = self.rewrite_identity_server_urls.get(
-                server_hostname, server_hostname,
-            )
-
-            key_data = yield self.http_client.get_json(
-                "https://%s/_matrix/identity/api/v1/pubkey/%s" %
-                (target, key_name,),
-            )
-            if "public_key" not in key_data:
-                raise AuthError(401, "No public key named %s from %s" %
-                                (key_name, server_hostname,))
-            verify_signed_json(
-                data,
-                server_hostname,
-                decode_verify_key_bytes(key_name, decode_base64(key_data["public_key"]))
-            )
-            return
-
-        raise AuthError(401, "No signature from server %s" % (server_hostname,))

synapse/handlers/message.py

@@ -22,7 +22,7 @@ from canonicaljson import encode_canonical_json, json
 from twisted.internet import defer
 from twisted.internet.defer import succeed
 
-from synapse.api.constants import EventTypes, Membership
+from synapse.api.constants import EventTypes, Membership, RelationTypes
 from synapse.api.errors import (
     AuthError,
     Codes,
@@ -601,6 +601,20 @@ class EventCreationHandler(object):
 
         self.validator.validate_new(event)
 
+        # If this event is an annotation then we check that that the sender
+        # can't annotate the same way twice (e.g. stops users from liking an
+        # event multiple times).
+        relation = event.content.get("m.relates_to", {})
+        if relation.get("rel_type") == RelationTypes.ANNOTATION:
+            relates_to = relation["event_id"]
+            aggregation_key = relation["key"]
+
+            already_exists = yield self.store.has_user_annotated_event(
+                relates_to, event.type, aggregation_key, event.sender,
+            )
+            if already_exists:
+                raise SynapseError(400, "Can't send same reaction twice")
+
         logger.debug(
             "Created event %s",
             event.event_id,

synapse/handlers/password_policy.py

@@ -1,93 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2019 New Vector Ltd
-# Copyright 2019 The Matrix.org Foundation C.I.C.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import logging
-import re
-
-from synapse.api.errors import Codes, PasswordRefusedError
-
-logger = logging.getLogger(__name__)
-
-
-class PasswordPolicyHandler(object):
-    def __init__(self, hs):
-        self.policy = hs.config.password_policy
-        self.enabled = hs.config.password_policy_enabled
-
-        # Regexps for the spec'd policy parameters.
-        self.regexp_digit = re.compile("[0-9]")
-        self.regexp_symbol = re.compile("[^a-zA-Z0-9]")
-        self.regexp_uppercase = re.compile("[A-Z]")
-        self.regexp_lowercase = re.compile("[a-z]")
-
-    def validate_password(self, password):
-        """Checks whether a given password complies with the server's policy.
-
-        Args:
-            password (str): The password to check against the server's policy.
-
-        Raises:
-            PasswordRefusedError: The password doesn't comply with the server's policy.
-        """
-
-        if not self.enabled:
-            return
-
-        minimum_accepted_length = self.policy.get("minimum_length", 0)
-        if len(password) < minimum_accepted_length:
-            raise PasswordRefusedError(
-                msg=(
-                    "The password must be at least %d characters long"
-                    % minimum_accepted_length
-                ),
-                errcode=Codes.PASSWORD_TOO_SHORT,
-            )
-
-        if (
-            self.policy.get("require_digit", False) and
-            self.regexp_digit.search(password) is None
-        ):
-            raise PasswordRefusedError(
-                msg="The password must include at least one digit",
-                errcode=Codes.PASSWORD_NO_DIGIT,
-            )
-
-        if (
-            self.policy.get("require_symbol", False) and
-            self.regexp_symbol.search(password) is None
-        ):
-            raise PasswordRefusedError(
-                msg="The password must include at least one symbol",
-                errcode=Codes.PASSWORD_NO_SYMBOL,
-            )
-
-        if (
-            self.policy.get("require_uppercase", False) and
-            self.regexp_uppercase.search(password) is None
-        ):
-            raise PasswordRefusedError(
-                msg="The password must include at least one uppercase letter",
-                errcode=Codes.PASSWORD_NO_UPPERCASE,
-            )
-
-        if (
-            self.policy.get("require_lowercase", False) and
-            self.regexp_lowercase.search(password) is None
-        ):
-            raise PasswordRefusedError(
-                msg="The password must include at least one lowercase letter",
-                errcode=Codes.PASSWORD_NO_LOWERCASE,
-            )

synapse/handlers/profile.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -16,11 +15,7 @@
 
 import logging
 
-from six.moves import range
-
-from signedjson.sign import sign_json
-
-from twisted.internet import defer, reactor
+from twisted.internet import defer
 
 from synapse.api.errors import (
     AuthError,
@@ -31,7 +26,6 @@ from synapse.api.errors import (
 )
 from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.types import UserID, get_domain_from_id
-from synapse.util.logcontext import run_in_background
 
 from ._base import BaseHandler
 
@@ -46,8 +40,6 @@ class BaseProfileHandler(BaseHandler):
     subclass MasterProfileHandler
     """
 
-    PROFILE_REPLICATE_INTERVAL = 2 * 60 * 1000
-
     def __init__(self, hs):
         super(BaseProfileHandler, self).__init__(hs)
 
@@ -58,84 +50,6 @@ class BaseProfileHandler(BaseHandler):
 
         self.user_directory_handler = hs.get_user_directory_handler()
 
-        self.http_client = hs.get_simple_http_client()
-
-        if hs.config.worker_app is None:
-            self.clock.looping_call(
-                self._start_update_remote_profile_cache, self.PROFILE_UPDATE_MS,
-            )
-
-            if len(self.hs.config.replicate_user_profiles_to) > 0:
-                reactor.callWhenRunning(self._assign_profile_replication_batches)
-                reactor.callWhenRunning(self._replicate_profiles)
-                # Add a looping call to replicate_profiles: this handles retries
-                # if the replication is unsuccessful when the user updated their
-                # profile.
-                self.clock.looping_call(
-                    self._replicate_profiles, self.PROFILE_REPLICATE_INTERVAL
-                )
-
-    @defer.inlineCallbacks
-    def _assign_profile_replication_batches(self):
-        """If no profile replication has been done yet, allocate replication batch
-        numbers to each profile to start the replication process.
-        """
-        logger.info("Assigning profile batch numbers...")
-        total = 0
-        while True:
-            assigned = yield self.store.assign_profile_batch()
-            total += assigned
-            if assigned == 0:
-                break
-        logger.info("Assigned %d profile batch numbers", total)
-
-    @defer.inlineCallbacks
-    def _replicate_profiles(self):
-        """If any profile data has been updated and not pushed to the replication targets,
-        replicate it.
-        """
-        host_batches = yield self.store.get_replication_hosts()
-        latest_batch = yield self.store.get_latest_profile_replication_batch_number()
-        if latest_batch is None:
-            latest_batch = -1
-        for repl_host in self.hs.config.replicate_user_profiles_to:
-            if repl_host not in host_batches:
-                host_batches[repl_host] = -1
-            try:
-                for i in range(host_batches[repl_host] + 1, latest_batch + 1):
-                    yield self._replicate_host_profile_batch(repl_host, i)
-            except Exception:
-                logger.exception(
-                    "Exception while replicating to %s: aborting for now", repl_host,
-                )
-
-    @defer.inlineCallbacks
-    def _replicate_host_profile_batch(self, host, batchnum):
-        logger.info("Replicating profile batch %d to %s", batchnum, host)
-        batch_rows = yield self.store.get_profile_batch(batchnum)
-        batch = {
-            UserID(r["user_id"], self.hs.hostname).to_string(): ({
-                "display_name": r["displayname"],
-                "avatar_url": r["avatar_url"],
-            } if r["active"] else None) for r in batch_rows
-        }
-
-        url = "https://%s/_matrix/identity/api/v1/replicate_profiles" % (host,)
-        body = {
-            "batchnum": batchnum,
-            "batch": batch,
-            "origin_server": self.hs.hostname,
-        }
-        signed_body = sign_json(body, self.hs.hostname, self.hs.config.signing_key[0])
-        try:
-            yield self.http_client.post_json_get_json(url, signed_body)
-            yield self.store.update_replication_batch_for_host(host, batchnum)
-            logger.info("Sucessfully replicated profile batch %d to %s", batchnum, host)
-        except Exception:
-            # This will get retried when the looping call next comes around
-            logger.exception("Failed to replicate profile batch %d to %s", batchnum, host)
-            raise
-
     @defer.inlineCallbacks
     def get_profile(self, user_id):
         target_user = UserID.from_string(user_id)
@@ -245,25 +159,14 @@ class BaseProfileHandler(BaseHandler):
         if not self.hs.is_mine(target_user):
             raise SynapseError(400, "User is not hosted on this Home Server")
 
-        if not by_admin and requester and target_user != requester.user:
+        if not by_admin and target_user != requester.user:
             raise AuthError(400, "Cannot set another user's displayname")
 
-        if not by_admin and self.hs.config.disable_set_displayname:
-            profile = yield self.store.get_profileinfo(target_user.localpart)
-            if profile.display_name:
-                raise SynapseError(400, "Changing displayname is disabled on this server")
-
         if new_displayname == '':
             new_displayname = None
 
-        if len(self.hs.config.replicate_user_profiles_to) > 0:
-            cur_batchnum = yield self.store.get_latest_profile_replication_batch_number()
-            new_batchnum = 0 if cur_batchnum is None else cur_batchnum + 1
-        else:
-            new_batchnum = None
-
         yield self.store.set_profile_displayname(
-            target_user.localpart, new_displayname, new_batchnum
+            target_user.localpart, new_displayname
         )
 
         if self.hs.config.user_directory_search_all_users:
@@ -272,37 +175,7 @@ class BaseProfileHandler(BaseHandler):
                 target_user.to_string(), profile
             )
 
-        if requester:
-            yield self._update_join_states(requester, target_user)
-
-        # start a profile replication push
-        run_in_background(self._replicate_profiles)
-
-    @defer.inlineCallbacks
-    def set_active(self, target_user, active, hide):
-        """
-        Sets the 'active' flag on a user profile. If set to false, the user
-        account is considered deactivated or hidden.
-
-        If 'hide' is true, then we interpret active=False as a request to try to
-        hide the user rather than deactivating it.  This means withholding the
-        profile from replication (and mark it as inactive) rather than clearing
-        the profile from the HS DB. Note that unlike set_displayname and
-        set_avatar_url, this does *not* perform authorization checks! This is
-        because the only place it's used currently is in account deactivation
-        where we've already done these checks anyway.
-        """
-        if len(self.hs.config.replicate_user_profiles_to) > 0:
-            cur_batchnum = yield self.store.get_latest_profile_replication_batch_number()
-            new_batchnum = 0 if cur_batchnum is None else cur_batchnum + 1
-        else:
-            new_batchnum = None
-        yield self.store.set_profile_active(
-            target_user.localpart, active, hide, new_batchnum
-        )
-
-        # start a profile replication push
-        run_in_background(self._replicate_profiles)
+        yield self._update_join_states(requester, target_user)
 
     @defer.inlineCallbacks
     def get_avatar_url(self, target_user):
@@ -344,19 +217,8 @@ class BaseProfileHandler(BaseHandler):
         if not by_admin and target_user != requester.user:
             raise AuthError(400, "Cannot set another user's avatar_url")
 
-        if not by_admin and self.hs.config.disable_set_avatar_url:
-            profile = yield self.store.get_profileinfo(target_user.localpart)
-            if profile.avatar_url:
-                raise SynapseError(400, "Changing avatar url is disabled on this server")
-
-        if len(self.hs.config.replicate_user_profiles_to) > 0:
-            cur_batchnum = yield self.store.get_latest_profile_replication_batch_number()
-            new_batchnum = 0 if cur_batchnum is None else cur_batchnum + 1
-        else:
-            new_batchnum = None
-
         yield self.store.set_profile_avatar_url(
-            target_user.localpart, new_avatar_url, new_batchnum,
+            target_user.localpart, new_avatar_url
         )
 
         if self.hs.config.user_directory_search_all_users:
@@ -367,9 +229,6 @@ class BaseProfileHandler(BaseHandler):
 
         yield self._update_join_states(requester, target_user)
 
-        # start a profile replication push
-        run_in_background(self._replicate_profiles)
-
     @defer.inlineCallbacks
     def on_profile_query(self, args):
         user = UserID.from_string(args["user_id"])

synapse/handlers/register.py

@@ -19,7 +19,7 @@ import logging
 from twisted.internet import defer
 
 from synapse import types
-from synapse.api.constants import LoginType
+from synapse.api.constants import MAX_USERID_LENGTH, LoginType
 from synapse.api.errors import (
     AuthError,
     Codes,
@@ -61,7 +61,6 @@ class RegistrationHandler(BaseHandler):
         self.profile_handler = hs.get_profile_handler()
         self.user_directory_handler = hs.get_user_directory_handler()
         self.captcha_client = CaptchaServerHttpClient(hs)
-        self.http_client = hs.get_simple_http_client()
         self.identity_handler = self.hs.get_handlers().identity_handler
         self.ratelimiter = hs.get_registration_ratelimiter()
 
@@ -124,6 +123,15 @@ class RegistrationHandler(BaseHandler):
 
         self.check_user_id_not_appservice_exclusive(user_id)
 
+        if len(user_id) > MAX_USERID_LENGTH:
+            raise SynapseError(
+                400,
+                "User ID may not be longer than %s characters" % (
+                    MAX_USERID_LENGTH,
+                ),
+                Codes.INVALID_USERNAME
+            )
+
         users = yield self.store.get_users_by_id_case_insensitive(user_id)
         if users:
             if not guest_access_token:
@@ -226,11 +234,6 @@ class RegistrationHandler(BaseHandler):
                 address=address,
             )
 
-            if default_display_name:
-                yield self.profile_handler.set_displayname(
-                    user, None, default_display_name, by_admin=True,
-                )
-
             if self.hs.config.user_directory_search_all_users:
                 profile = yield self.store.get_profileinfo(localpart)
                 yield self.user_directory_handler.handle_local_profile_change(
@@ -260,11 +263,6 @@ class RegistrationHandler(BaseHandler):
                         create_profile_with_displayname=default_display_name,
                         address=address,
                     )
-
-                    yield self.profile_handler.set_displayname(
-                        user, None, default_display_name, by_admin=True,
-                    )
-
                 except SynapseError:
                     # if user id is taken, just generate another
                     user = None
@@ -358,9 +356,7 @@ class RegistrationHandler(BaseHandler):
         yield self._auto_join_rooms(user_id)
 
     @defer.inlineCallbacks
-    def appservice_register(self, user_localpart, as_token, password, display_name):
-        # FIXME: this should be factored out and merged with normal register()
-
+    def appservice_register(self, user_localpart, as_token):
         user = UserID(user_localpart, self.hs.hostname)
         user_id = user.to_string()
         service = self.store.get_app_service_by_token(as_token)
@@ -378,29 +374,12 @@ class RegistrationHandler(BaseHandler):
             user_id, allowed_appservice=service
         )
 
-        password_hash = ""
-        if password:
-            password_hash = yield self.auth_handler().hash(password)
-
-        display_name = display_name or user.localpart
-
         yield self.register_with_store(
             user_id=user_id,
-            password_hash=password_hash,
+            password_hash="",
             appservice_id=service_id,
-            create_profile_with_displayname=display_name,
+            create_profile_with_displayname=user.localpart,
         )
-
-        yield self.profile_handler.set_displayname(
-            user, None, display_name, by_admin=True,
-        )
-
-        if self.hs.config.user_directory_search_all_users:
-            profile = yield self.store.get_profileinfo(user_localpart)
-            yield self.user_directory_handler.handle_local_profile_change(
-                user_id, profile
-            )
-
         defer.returnValue(user_id)
 
     @defer.inlineCallbacks
@@ -426,39 +405,6 @@ class RegistrationHandler(BaseHandler):
         else:
             logger.info("Valid captcha entered from %s", ip)
 
-    @defer.inlineCallbacks
-    def register_saml2(self, localpart):
-        """
-        Registers email_id as SAML2 Based Auth.
-        """
-        if types.contains_invalid_mxid_characters(localpart):
-            raise SynapseError(
-                400,
-                "User ID can only contain characters a-z, 0-9, or '=_-./'",
-            )
-        yield self.auth.check_auth_blocking()
-        user = UserID(localpart, self.hs.hostname)
-        user_id = user.to_string()
-
-        yield self.check_user_id_not_appservice_exclusive(user_id)
-        token = self.macaroon_gen.generate_access_token(user_id)
-        try:
-            yield self.register_with_store(
-                user_id=user_id,
-                token=token,
-                password_hash=None,
-                create_profile_with_displayname=user.localpart,
-            )
-
-            yield self.profile_handler.set_displayname(
-                user, None, user.localpart, by_admin=True,
-            )
-        except Exception as e:
-            yield self.store.add_access_token_to_user(user_id, token)
-            # Ignore Registration errors
-            logger.exception(e)
-        defer.returnValue((user_id, token))
-
     @defer.inlineCallbacks
     def register_email(self, threepidCreds):
         """
@@ -481,9 +427,7 @@ class RegistrationHandler(BaseHandler):
             logger.info("got threepid with medium '%s' and address '%s'",
                         threepid['medium'], threepid['address'])
 
-            if not (
-                yield check_3pid_allowed(self.hs, threepid['medium'], threepid['address'])
-            ):
+            if not check_3pid_allowed(self.hs, threepid['medium'], threepid['address']):
                 raise RegistrationError(
                     403, "Third party identifier is not allowed"
                 )
@@ -524,39 +468,6 @@ class RegistrationHandler(BaseHandler):
                     errcode=Codes.EXCLUSIVE
                 )
 
-    @defer.inlineCallbacks
-    def shadow_register(self, localpart, display_name, auth_result, params):
-        """Invokes the current registration on another server, using
-        shared secret registration, passing in any auth_results from
-        other registration UI auth flows (e.g. validated 3pids)
-        Useful for setting up shadow/backup accounts on a parallel deployment.
-        """
-
-        # TODO: retries
-        shadow_hs_url = self.hs.config.shadow_server.get("hs_url")
-        as_token = self.hs.config.shadow_server.get("as_token")
-
-        yield self.http_client.post_json_get_json(
-            "%s/_matrix/client/r0/register?access_token=%s" % (
-                shadow_hs_url, as_token,
-            ),
-            {
-                # XXX: auth_result is an unspecified extension for shadow registration
-                'auth_result': auth_result,
-                # XXX: another unspecified extension for shadow registration to ensure
-                # that the displayname is correctly set by the masters erver
-                'display_name': display_name,
-                'username': localpart,
-                'password': params.get("password"),
-                'bind_email': params.get("bind_email"),
-                'bind_msisdn': params.get("bind_msisdn"),
-                'device_id': params.get("device_id"),
-                'initial_device_display_name': params.get("initial_device_display_name"),
-                'inhibit_login': False,
-                'access_token': as_token,
-            }
-        )
-
     @defer.inlineCallbacks
     def _generate_user_id(self, reseed=False):
         if reseed or self._next_generated_user_id is None:
@@ -643,16 +554,18 @@ class RegistrationHandler(BaseHandler):
                 user_id=user_id,
                 token=token,
                 password_hash=password_hash,
-                create_profile_with_displayname=displayname or user.localpart,
+                create_profile_with_displayname=user.localpart,
             )
-            if displayname is not None:
-                yield self.profile_handler.set_displayname(
-                    user, None, displayname or user.localpart, by_admin=True,
-                )
         else:
             yield self._auth_handler.delete_access_tokens_for_user(user_id)
             yield self.store.add_access_token_to_user(user_id=user_id, token=token)
 
+        if displayname is not None:
+            logger.info("setting user display name: %s -> %s", user_id, displayname)
+            yield self.profile_handler.set_displayname(
+                user, requester, displayname, by_admin=True,
+            )
+
         defer.returnValue((user_id, token))
 
     @defer.inlineCallbacks

synapse/handlers/room.py

@@ -49,14 +49,12 @@ class RoomCreationHandler(BaseHandler):
             "history_visibility": "shared",
             "original_invitees_have_ops": False,
             "guest_can_join": True,
-            "encryption_alg": "m.megolm.v1.aes-sha2",
         },
         RoomCreationPreset.TRUSTED_PRIVATE_CHAT: {
             "join_rules": JoinRules.INVITE,
             "history_visibility": "shared",
             "original_invitees_have_ops": True,
             "guest_can_join": True,
-            "encryption_alg": "m.megolm.v1.aes-sha2",
         },
         RoomCreationPreset.PUBLIC_CHAT: {
             "join_rules": JoinRules.PUBLIC,
@@ -76,8 +74,6 @@ class RoomCreationHandler(BaseHandler):
         # linearizer to stop two upgrades happening at once
         self._upgrade_linearizer = Linearizer("room_upgrade_linearizer")
 
-        self._server_notices_mxid = hs.config.server_notices_mxid
-
     @defer.inlineCallbacks
     def upgrade_room(self, requester, old_room_id, new_version):
         """Replace a room with a new room with a different version
@@ -251,22 +247,7 @@ class RoomCreationHandler(BaseHandler):
         """
         user_id = requester.user.to_string()
 
-        if (self._server_notices_mxid is not None and
-                requester.user.to_string() == self._server_notices_mxid):
-            # allow the server notices mxid to create rooms
-            is_requester_admin = True
-
-        else:
-            is_requester_admin = yield self.auth.is_server_admin(
-                requester.user,
-            )
-
-        if not is_requester_admin and not self.spam_checker.user_may_create_room(
-            user_id,
-            invite_list=[],
-            third_party_invite_list=[],
-            cloning=True,
-        ):
+        if not self.spam_checker.user_may_create_room(user_id):
             raise SynapseError(403, "You are not permitted to create rooms")
 
         creation_content = {
@@ -488,24 +469,7 @@ class RoomCreationHandler(BaseHandler):
 
         yield self.auth.check_auth_blocking(user_id)
 
-        invite_list = config.get("invite", [])
-        invite_3pid_list = config.get("invite_3pid", [])
-
-        if (self._server_notices_mxid is not None and
-                requester.user.to_string() == self._server_notices_mxid):
-            # allow the server notices mxid to create rooms
-            is_requester_admin = True
-        else:
-            is_requester_admin = yield self.auth.is_server_admin(
-                requester.user,
-            )
-
-        if not is_requester_admin and not self.spam_checker.user_may_create_room(
-            user_id,
-            invite_list=invite_list,
-            third_party_invite_list=invite_3pid_list,
-            cloning=False,
-        ):
+        if not self.spam_checker.user_may_create_room(user_id):
             raise SynapseError(403, "You are not permitted to create rooms")
 
         if ratelimit:
@@ -548,6 +512,7 @@ class RoomCreationHandler(BaseHandler):
         else:
             room_alias = None
 
+        invite_list = config.get("invite", [])
         for i in invite_list:
             try:
                 UserID.from_string(i)
@@ -558,6 +523,8 @@ class RoomCreationHandler(BaseHandler):
             requester,
         )
 
+        invite_3pid_list = config.get("invite_3pid", [])
+
         visibility = config.get("visibility", None)
         is_public = visibility == "public"
 
@@ -643,7 +610,6 @@ class RoomCreationHandler(BaseHandler):
                 "invite",
                 ratelimit=False,
                 content=content,
-                new_room=True,
             )
 
         for invite_3pid in invite_3pid_list:
@@ -658,7 +624,6 @@ class RoomCreationHandler(BaseHandler):
                 id_server,
                 requester,
                 txn_id=None,
-                new_room=True,
             )
 
         result = {"room_id": room_id}
@@ -729,7 +694,6 @@ class RoomCreationHandler(BaseHandler):
             "join",
             ratelimit=False,
             content=creator_join_profile,
-            new_room=True,
         )
 
         # We treat the power levels override specially as this needs to be one
@@ -805,15 +769,6 @@ class RoomCreationHandler(BaseHandler):
                 content=content,
             )
 
-        if "encryption_alg" in config:
-            yield send(
-                etype=EventTypes.Encryption,
-                state_key="",
-                content={
-                    'algorithm': config["encryption_alg"],
-                }
-            )
-
     @defer.inlineCallbacks
     def _generate_room_id(self, creator_id, is_public):
         # autogen room IDs and try to create it. We may clash, so just

synapse/handlers/room_member.py

@@ -20,12 +20,16 @@ import logging
 
 from six.moves import http_client
 
+from signedjson.key import decode_verify_key_bytes
+from signedjson.sign import verify_signed_json
+from unpaddedbase64 import decode_base64
+
 from twisted.internet import defer
 
 import synapse.server
 import synapse.types
 from synapse.api.constants import EventTypes, Membership
-from synapse.api.errors import AuthError, Codes, ProxiedRequestError, SynapseError
+from synapse.api.errors import AuthError, Codes, SynapseError
 from synapse.types import RoomID, UserID
 from synapse.util.async_helpers import Linearizer
 from synapse.util.distributor import user_joined_room, user_left_room
@@ -63,14 +67,12 @@ class RoomMemberHandler(object):
         self.registration_handler = hs.get_registration_handler()
         self.profile_handler = hs.get_profile_handler()
         self.event_creation_handler = hs.get_event_creation_handler()
-        self.identity_handler = hs.get_handlers().identity_handler
 
         self.member_linearizer = Linearizer(name="member")
 
         self.clock = hs.get_clock()
         self.spam_checker = hs.get_spam_checker()
         self._server_notices_mxid = self.config.server_notices_mxid
-        self.rewrite_identity_server_urls = self.config.rewrite_identity_server_urls
         self._enable_lookup = hs.config.enable_3pid_lookup
         self.allow_per_room_profiles = self.config.allow_per_room_profiles
 
@@ -315,31 +317,8 @@ class RoomMemberHandler(object):
             third_party_signed=None,
             ratelimit=True,
             content=None,
-            new_room=False,
             require_consent=True,
     ):
-        """Update a users membership in a room
-
-        Args:
-            requester (Requester)
-            target (UserID)
-            room_id (str)
-            action (str): The "action" the requester is performing against the
-                target. One of join/leave/kick/ban/invite/unban.
-            txn_id (str|None): The transaction ID associated with the request,
-                or None not provided.
-            remote_room_hosts (list[str]|None): List of remote servers to try
-                and join via if server isn't already in the room.
-            third_party_signed (dict|None): The signed object for third party
-                invites.
-            ratelimit (bool): Whether to apply ratelimiting to this request.
-            content (dict|None): Fields to include in the new events content.
-            new_room (bool): Whether these membership changes are happening
-                as part of a room creation (e.g. initial joins and invites)
-
-        Returns:
-            Deferred[FrozenEvent]
-        """
         key = (room_id,)
 
         with (yield self.member_linearizer.queue(key)):
@@ -353,7 +332,6 @@ class RoomMemberHandler(object):
                 third_party_signed=third_party_signed,
                 ratelimit=ratelimit,
                 content=content,
-                new_room=new_room,
                 require_consent=require_consent,
             )
 
@@ -371,7 +349,6 @@ class RoomMemberHandler(object):
             third_party_signed=None,
             ratelimit=True,
             content=None,
-            new_room=False,
             require_consent=True,
     ):
         content_specified = bool(content)
@@ -439,14 +416,8 @@ class RoomMemberHandler(object):
                     )
                     block_invite = True
 
-                is_published = yield self.store.is_room_published(room_id)
-
                 if not self.spam_checker.user_may_invite(
-                    requester.user.to_string(), target.to_string(),
-                    third_party_invite=None,
-                    room_id=room_id,
-                    new_room=new_room,
-                    published_room=is_published,
+                    requester.user.to_string(), target.to_string(), room_id,
                 ):
                     logger.info("Blocking invite due to spam checker")
                     block_invite = True
@@ -525,29 +496,8 @@ class RoomMemberHandler(object):
                     # so don't really fit into the general auth process.
                     raise AuthError(403, "Guest access not allowed")
 
-            if (self._server_notices_mxid is not None and
-                    requester.user.to_string() == self._server_notices_mxid):
-                # allow the server notices mxid to join rooms
-                is_requester_admin = True
-
-            else:
-                is_requester_admin = yield self.auth.is_server_admin(
-                    requester.user,
-                )
-
-            inviter = yield self._get_inviter(target.to_string(), room_id)
-            if not is_requester_admin:
-                # We assume that if the spam checker allowed the user to create
-                # a room then they're allowed to join it.
-                if not new_room and not self.spam_checker.user_may_join_room(
-                    target.to_string(), room_id,
-                    is_invited=inviter is not None,
-                ):
-                    raise SynapseError(
-                        403, "Not allowed to join this room",
-                    )
-
             if not is_host_in_room:
+                inviter = yield self._get_inviter(target.to_string(), room_id)
                 if inviter and not self.hs.is_mine(inviter):
                     remote_room_hosts.append(inviter.domain)
 
@@ -757,8 +707,7 @@ class RoomMemberHandler(object):
             address,
             id_server,
             requester,
-            txn_id,
-            new_room=False,
+            txn_id
     ):
         if self.config.block_non_admin_invites:
             is_requester_admin = yield self.auth.is_server_admin(
@@ -778,23 +727,6 @@ class RoomMemberHandler(object):
             id_server, medium, address
         )
 
-        is_published = yield self.store.is_room_published(room_id)
-
-        if not self.spam_checker.user_may_invite(
-            requester.user.to_string(), invitee,
-            third_party_invite={
-                "medium": medium,
-                "address": address,
-            },
-            room_id=room_id,
-            new_room=new_room,
-            published_room=is_published,
-        ):
-            logger.info("Blocking invite due to spam checker")
-            raise SynapseError(
-                403, "Invites have been disabled on this server",
-            )
-
         if invitee:
             yield self.update_membership(
                 requester,
@@ -814,20 +746,6 @@ class RoomMemberHandler(object):
                 txn_id=txn_id
             )
 
-    def _get_id_server_target(self, id_server):
-        """Looks up an id_server's actual http endpoint
-
-        Args:
-            id_server (str): the server name to lookup.
-
-        Returns:
-            the http endpoint to connect to.
-        """
-        if id_server in self.rewrite_identity_server_urls:
-            return self.rewrite_identity_server_urls[id_server]
-
-        return id_server
-
     @defer.inlineCallbacks
     def _lookup_3pid(self, id_server, medium, address):
         """Looks up a 3pid in the passed identity server.
@@ -841,13 +759,48 @@ class RoomMemberHandler(object):
         Returns:
             str: the matrix ID of the 3pid, or None if it is not recognized.
         """
+        if not self._enable_lookup:
+            raise SynapseError(
+                403, "Looking up third-party identifiers is denied from this server",
+            )
         try:
-            data = yield self.identity_handler.lookup_3pid(id_server, medium, address)
-            defer.returnValue(data.get("mxid"))
-        except ProxiedRequestError as e:
+            data = yield self.simple_http_client.get_json(
+                "%s%s/_matrix/identity/api/v1/lookup" % (id_server_scheme, id_server,),
+                {
+                    "medium": medium,
+                    "address": address,
+                }
+            )
+
+            if "mxid" in data:
+                if "signatures" not in data:
+                    raise AuthError(401, "No signatures on 3pid binding")
+                yield self._verify_any_signature(data, id_server)
+                defer.returnValue(data["mxid"])
+
+        except IOError as e:
             logger.warn("Error from identity server lookup: %s" % (e,))
             defer.returnValue(None)
 
+    @defer.inlineCallbacks
+    def _verify_any_signature(self, data, server_hostname):
+        if server_hostname not in data["signatures"]:
+            raise AuthError(401, "No signature from server %s" % (server_hostname,))
+        for key_name, signature in data["signatures"][server_hostname].items():
+            key_data = yield self.simple_http_client.get_json(
+                "%s%s/_matrix/identity/api/v1/pubkey/%s" %
+                (id_server_scheme, server_hostname, key_name,),
+            )
+            if "public_key" not in key_data:
+                raise AuthError(401, "No public key named %s from %s" %
+                                (key_name, server_hostname,))
+            verify_signed_json(
+                data,
+                server_hostname,
+                decode_verify_key_bytes(key_name, decode_base64(key_data["public_key"]))
+            )
+            return
+
     @defer.inlineCallbacks
     def _make_and_store_3pid_invite(
             self,
@@ -973,9 +926,8 @@ class RoomMemberHandler(object):
                     user.
         """
 
-        target = self._get_id_server_target(id_server)
         is_url = "%s%s/_matrix/identity/api/v1/store-invite" % (
-            id_server_scheme, target,
+            id_server_scheme, id_server,
         )
 
         invite_config = {
@@ -992,7 +944,7 @@ class RoomMemberHandler(object):
         }
 
         if self.config.invite_3pid_guest:
-            guest_access_token, guest_user_id = yield self.get_or_register_3pid_guest(
+            guest_user_id, guest_access_token = yield self.get_or_register_3pid_guest(
                 requester=requester,
                 medium=medium,
                 address=address,
@@ -1015,7 +967,7 @@ class RoomMemberHandler(object):
             fallback_public_key = {
                 "public_key": data["public_key"],
                 "key_validity_url": "%s%s/_matrix/identity/api/v1/pubkey/isvalid" % (
-                    id_server_scheme, target,
+                    id_server_scheme, id_server,
                 ),
             }
         else:

synapse/handlers/set_password.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2017-2018 New Vector Ltd
-# Copyright 2019 The Matrix.org Foundation C.I.C.
+# Copyright 2017 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -30,12 +29,9 @@ class SetPasswordHandler(BaseHandler):
         super(SetPasswordHandler, self).__init__(hs)
         self._auth_handler = hs.get_auth_handler()
         self._device_handler = hs.get_device_handler()
-        self._password_policy_handler = hs.get_password_policy_handler()
 
     @defer.inlineCallbacks
     def set_password(self, user_id, newpassword, requester=None):
-        self._password_policy_handler.validate_password(newpassword)
-
         password_hash = yield self._auth_handler.hash(newpassword)
 
         except_device_id = requester.device_id if requester else None

synapse/handlers/stats.py

@@ -0,0 +1,325 @@
+# -*- coding: utf-8 -*-
+# Copyright 2018 New Vector Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+
+from twisted.internet import defer
+
+from synapse.api.constants import EventTypes, JoinRules, Membership
+from synapse.handlers.state_deltas import StateDeltasHandler
+from synapse.metrics import event_processing_positions
+from synapse.metrics.background_process_metrics import run_as_background_process
+from synapse.types import UserID
+from synapse.util.metrics import Measure
+
+logger = logging.getLogger(__name__)
+
+
+class StatsHandler(StateDeltasHandler):
+    """Handles keeping the *_stats tables updated with a simple time-series of
+    information about the users, rooms and media on the server, such that admins
+    have some idea of who is consuming their resources.
+
+    Heavily derived from UserDirectoryHandler
+    """
+
+    def __init__(self, hs):
+        super(StatsHandler, self).__init__(hs)
+        self.hs = hs
+        self.store = hs.get_datastore()
+        self.state = hs.get_state_handler()
+        self.server_name = hs.hostname
+        self.clock = hs.get_clock()
+        self.notifier = hs.get_notifier()
+        self.is_mine_id = hs.is_mine_id
+        self.stats_bucket_size = hs.config.stats_bucket_size
+
+        # The current position in the current_state_delta stream
+        self.pos = None
+
+        # Guard to ensure we only process deltas one at a time
+        self._is_processing = False
+
+        if hs.config.stats_enabled:
+            self.notifier.add_replication_callback(self.notify_new_event)
+
+            # We kick this off so that we don't have to wait for a change before
+            # we start populating stats
+            self.clock.call_later(0, self.notify_new_event)
+
+    def notify_new_event(self):
+        """Called when there may be more deltas to process
+        """
+        if not self.hs.config.stats_enabled:
+            return
+
+        if self._is_processing:
+            return
+
+        @defer.inlineCallbacks
+        def process():
+            try:
+                yield self._unsafe_process()
+            finally:
+                self._is_processing = False
+
+        self._is_processing = True
+        run_as_background_process("stats.notify_new_event", process)
+
+    @defer.inlineCallbacks
+    def _unsafe_process(self):
+        # If self.pos is None then means we haven't fetched it from DB
+        if self.pos is None:
+            self.pos = yield self.store.get_stats_stream_pos()
+
+        # If still None then the initial background update hasn't happened yet
+        if self.pos is None:
+            defer.returnValue(None)
+
+        # Loop round handling deltas until we're up to date
+        while True:
+            with Measure(self.clock, "stats_delta"):
+                deltas = yield self.store.get_current_state_deltas(self.pos)
+                if not deltas:
+                    return
+
+                logger.info("Handling %d state deltas", len(deltas))
+                yield self._handle_deltas(deltas)
+
+                self.pos = deltas[-1]["stream_id"]
+                yield self.store.update_stats_stream_pos(self.pos)
+
+                event_processing_positions.labels("stats").set(self.pos)
+
+    @defer.inlineCallbacks
+    def _handle_deltas(self, deltas):
+        """
+        Called with the state deltas to process
+        """
+        for delta in deltas:
+            typ = delta["type"]
+            state_key = delta["state_key"]
+            room_id = delta["room_id"]
+            event_id = delta["event_id"]
+            stream_id = delta["stream_id"]
+            prev_event_id = delta["prev_event_id"]
+
+            logger.debug("Handling: %r %r, %s", typ, state_key, event_id)
+
+            token = yield self.store.get_earliest_token_for_room_stats(room_id)
+
+            # If the earliest token to begin from is larger than our current
+            # stream ID, skip processing this delta.
+            if token is not None and token >= stream_id:
+                logger.debug(
+                    "Ignoring: %s as earlier than this room's initial ingestion event",
+                    event_id,
+                )
+                continue
+
+            if event_id is None and prev_event_id is None:
+                # Errr...
+                continue
+
+            event_content = {}
+
+            if event_id is not None:
+                event_content = (yield self.store.get_event(event_id)).content or {}
+
+            # quantise time to the nearest bucket
+            now = yield self.store.get_received_ts(event_id)
+            now = (now // 1000 // self.stats_bucket_size) * self.stats_bucket_size
+
+            if typ == EventTypes.Member:
+                # we could use _get_key_change here but it's a bit inefficient
+                # given we're not testing for a specific result; might as well
+                # just grab the prev_membership and membership strings and
+                # compare them.
+                prev_event_content = {}
+                if prev_event_id is not None:
+                    prev_event_content = (
+                        yield self.store.get_event(prev_event_id)
+                    ).content
+
+                membership = event_content.get("membership", Membership.LEAVE)
+                prev_membership = prev_event_content.get("membership", Membership.LEAVE)
+
+                if prev_membership == membership:
+                    continue
+
+                if prev_membership == Membership.JOIN:
+                    yield self.store.update_stats_delta(
+                        now, "room", room_id, "joined_members", -1
+                    )
+                elif prev_membership == Membership.INVITE:
+                    yield self.store.update_stats_delta(
+                        now, "room", room_id, "invited_members", -1
+                    )
+                elif prev_membership == Membership.LEAVE:
+                    yield self.store.update_stats_delta(
+                        now, "room", room_id, "left_members", -1
+                    )
+                elif prev_membership == Membership.BAN:
+                    yield self.store.update_stats_delta(
+                        now, "room", room_id, "banned_members", -1
+                    )
+                else:
+                    err = "%s is not a valid prev_membership" % (repr(prev_membership),)
+                    logger.error(err)
+                    raise ValueError(err)
+
+                if membership == Membership.JOIN:
+                    yield self.store.update_stats_delta(
+                        now, "room", room_id, "joined_members", +1
+                    )
+                elif membership == Membership.INVITE:
+                    yield self.store.update_stats_delta(
+                        now, "room", room_id, "invited_members", +1
+                    )
+                elif membership == Membership.LEAVE:
+                    yield self.store.update_stats_delta(
+                        now, "room", room_id, "left_members", +1
+                    )
+                elif membership == Membership.BAN:
+                    yield self.store.update_stats_delta(
+                        now, "room", room_id, "banned_members", +1
+                    )
+                else:
+                    err = "%s is not a valid membership" % (repr(membership),)
+                    logger.error(err)
+                    raise ValueError(err)
+
+                user_id = state_key
+                if self.is_mine_id(user_id):
+                    # update user_stats as it's one of our users
+                    public = yield self._is_public_room(room_id)
+
+                    if membership == Membership.LEAVE:
+                        yield self.store.update_stats_delta(
+                            now,
+                            "user",
+                            user_id,
+                            "public_rooms" if public else "private_rooms",
+                            -1,
+                        )
+                    elif membership == Membership.JOIN:
+                        yield self.store.update_stats_delta(
+                            now,
+                            "user",
+                            user_id,
+                            "public_rooms" if public else "private_rooms",
+                            +1,
+                        )
+
+            elif typ == EventTypes.Create:
+                # Newly created room. Add it with all blank portions.
+                yield self.store.update_room_state(
+                    room_id,
+                    {
+                        "join_rules": None,
+                        "history_visibility": None,
+                        "encryption": None,
+                        "name": None,
+                        "topic": None,
+                        "avatar": None,
+                        "canonical_alias": None,
+                    },
+                )
+
+            elif typ == EventTypes.JoinRules:
+                yield self.store.update_room_state(
+                    room_id, {"join_rules": event_content.get("join_rule")}
+                )
+
+                is_public = yield self._get_key_change(
+                    prev_event_id, event_id, "join_rule", JoinRules.PUBLIC
+                )
+                if is_public is not None:
+                    yield self.update_public_room_stats(now, room_id, is_public)
+
+            elif typ == EventTypes.RoomHistoryVisibility:
+                yield self.store.update_room_state(
+                    room_id,
+                    {"history_visibility": event_content.get("history_visibility")},
+                )
+
+                is_public = yield self._get_key_change(
+                    prev_event_id, event_id, "history_visibility", "world_readable"
+                )
+                if is_public is not None:
+                    yield self.update_public_room_stats(now, room_id, is_public)
+
+            elif typ == EventTypes.Encryption:
+                yield self.store.update_room_state(
+                    room_id, {"encryption": event_content.get("algorithm")}
+                )
+            elif typ == EventTypes.Name:
+                yield self.store.update_room_state(
+                    room_id, {"name": event_content.get("name")}
+                )
+            elif typ == EventTypes.Topic:
+                yield self.store.update_room_state(
+                    room_id, {"topic": event_content.get("topic")}
+                )
+            elif typ == EventTypes.RoomAvatar:
+                yield self.store.update_room_state(
+                    room_id, {"avatar": event_content.get("url")}
+                )
+            elif typ == EventTypes.CanonicalAlias:
+                yield self.store.update_room_state(
+                    room_id, {"canonical_alias": event_content.get("alias")}
+                )
+
+    @defer.inlineCallbacks
+    def update_public_room_stats(self, ts, room_id, is_public):
+        """
+        Increment/decrement a user's number of public rooms when a room they are
+        in changes to/from public visibility.
+
+        Args:
+            ts (int): Timestamp in seconds
+            room_id (str)
+            is_public (bool)
+        """
+        # For now, blindly iterate over all local users in the room so that
+        # we can handle the whole problem of copying buckets over as needed
+        user_ids = yield self.store.get_users_in_room(room_id)
+
+        for user_id in user_ids:
+            if self.hs.is_mine(UserID.from_string(user_id)):
+                yield self.store.update_stats_delta(
+                    ts, "user", user_id, "public_rooms", +1 if is_public else -1
+                )
+                yield self.store.update_stats_delta(
+                    ts, "user", user_id, "private_rooms", -1 if is_public else +1
+                )
+
+    @defer.inlineCallbacks
+    def _is_public_room(self, room_id):
+        join_rules = yield self.state.get_current_state(room_id, EventTypes.JoinRules)
+        history_visibility = yield self.state.get_current_state(
+            room_id, EventTypes.RoomHistoryVisibility
+        )
+
+        if (join_rules and join_rules.content.get("join_rule") == JoinRules.PUBLIC) or (
+            (
+                history_visibility
+                and history_visibility.content.get("history_visibility")
+                == "world_readable"
+            )
+        ):
+            defer.returnValue(True)
+        else:
+            defer.returnValue(False)

synapse/python_dependencies.py

@@ -16,7 +16,12 @@
 
 import logging
 
-from pkg_resources import DistributionNotFound, VersionConflict, get_distribution
+from pkg_resources import (
+    DistributionNotFound,
+    Requirement,
+    VersionConflict,
+    get_provider,
+)
 
 logger = logging.getLogger(__name__)
 
@@ -53,7 +58,7 @@ REQUIREMENTS = [
     "pyasn1-modules>=0.0.7",
     "daemonize>=2.3.1",
     "bcrypt>=3.1.0",
-    "pillow>=3.1.2",
+    "pillow>=4.3.0",
     "sortedcontainers>=1.4.4",
     "psutil>=2.0.0",
     "pymacaroons>=0.13.0",
@@ -69,14 +74,6 @@ REQUIREMENTS = [
     "attrs>=17.4.0",
 
     "netaddr>=0.7.18",
-
-    # requests is a transitive dep of treq, and urlib3 is a transitive dep
-    # of requests, as well as of sentry-sdk.
-    #
-    # As of requests 2.21, requests does not yet support urllib3 1.25.
-    # (If we do not pin it here, pip will give us the latest urllib3
-    # due to the dep via sentry-sdk.)
-    "urllib3<1.25",
 ]
 
 CONDITIONAL_REQUIREMENTS = {
@@ -91,7 +88,13 @@ CONDITIONAL_REQUIREMENTS = {
 
     # ACME support is required to provision TLS certificates from authorities
     # that use the protocol, such as Let's Encrypt.
-    "acme": ["txacme>=0.9.2"],
+    "acme": [
+        "txacme>=0.9.2",
+
+        # txacme depends on eliot. Eliot 1.8.0 is incompatible with
+        # python 3.5.2, as per https://github.com/itamarst/eliot/issues/418
+        'eliot<1.8.0;python_version<"3.5.3"',
+    ],
 
     "saml2": ["pysaml2>=4.5.0"],
     "systemd": ["systemd-python>=231"],
@@ -125,10 +128,10 @@ class DependencyException(Exception):
     @property
     def dependencies(self):
         for i in self.args[0]:
-            yield '"' + i + '"'
+            yield "'" + i + "'"
 
 
-def check_requirements(for_feature=None, _get_distribution=get_distribution):
+def check_requirements(for_feature=None):
     deps_needed = []
     errors = []
 
@@ -139,7 +142,7 @@ def check_requirements(for_feature=None, _get_distribution=get_distribution):
 
     for dependency in reqs:
         try:
-            _get_distribution(dependency)
+            _check_requirement(dependency)
         except VersionConflict as e:
             deps_needed.append(dependency)
             errors.append(
@@ -157,7 +160,7 @@ def check_requirements(for_feature=None, _get_distribution=get_distribution):
 
         for dependency in OPTS:
             try:
-                _get_distribution(dependency)
+                _check_requirement(dependency)
             except VersionConflict as e:
                 deps_needed.append(dependency)
                 errors.append(
@@ -175,6 +178,23 @@ def check_requirements(for_feature=None, _get_distribution=get_distribution):
         raise DependencyException(deps_needed)
 
 
+def _check_requirement(dependency_string):
+    """Parses a dependency string, and checks if the specified requirement is installed
+
+    Raises:
+        VersionConflict if the requirement is installed, but with the the wrong version
+        DistributionNotFound if nothing is found to provide the requirement
+    """
+    req = Requirement.parse(dependency_string)
+
+    # first check if the markers specify that this requirement needs installing
+    if req.marker is not None and not req.marker.evaluate():
+        # not required for this environment
+        return
+
+    get_provider(req)
+
+
 if __name__ == "__main__":
     import sys
 

synapse/replication/slave/storage/events.py

@@ -23,6 +23,7 @@ from synapse.replication.tcp.streams.events import (
 from synapse.storage.event_federation import EventFederationWorkerStore
 from synapse.storage.event_push_actions import EventPushActionsWorkerStore
 from synapse.storage.events_worker import EventsWorkerStore
+from synapse.storage.relations import RelationsWorkerStore
 from synapse.storage.roommember import RoomMemberWorkerStore
 from synapse.storage.signatures import SignatureWorkerStore
 from synapse.storage.state import StateGroupWorkerStore
@@ -52,6 +53,7 @@ class SlavedEventStore(EventFederationWorkerStore,
                        EventsWorkerStore,
                        SignatureWorkerStore,
                        UserErasureWorkerStore,
+                       RelationsWorkerStore,
                        BaseSlavedStore):
 
     def __init__(self, db_conn, hs):
@@ -89,7 +91,7 @@ class SlavedEventStore(EventFederationWorkerStore,
             for row in rows:
                 self.invalidate_caches_for_event(
                     -token, row.event_id, row.room_id, row.type, row.state_key,
-                    row.redacts,
+                    row.redacts, row.relates_to,
                     backfilled=True,
                 )
         return super(SlavedEventStore, self).process_replication_rows(
@@ -102,7 +104,7 @@ class SlavedEventStore(EventFederationWorkerStore,
         if row.type == EventsStreamEventRow.TypeId:
             self.invalidate_caches_for_event(
                 token, data.event_id, data.room_id, data.type, data.state_key,
-                data.redacts,
+                data.redacts, data.relates_to,
                 backfilled=False,
             )
         elif row.type == EventsStreamCurrentStateRow.TypeId:
@@ -114,7 +116,8 @@ class SlavedEventStore(EventFederationWorkerStore,
             raise Exception("Unknown events stream row type %s" % (row.type, ))
 
     def invalidate_caches_for_event(self, stream_ordering, event_id, room_id,
-                                    etype, state_key, redacts, backfilled):
+                                    etype, state_key, redacts, relates_to,
+                                    backfilled):
         self._invalidate_get_event_cache(event_id)
 
         self.get_latest_event_ids_in_room.invalidate((room_id,))
@@ -136,3 +139,8 @@ class SlavedEventStore(EventFederationWorkerStore,
                 state_key, stream_ordering
             )
             self.get_invited_rooms_for_user.invalidate((state_key,))
+
+        if relates_to:
+            self.get_relations_for_event.invalidate_many((relates_to,))
+            self.get_aggregation_groups_for_event.invalidate_many((relates_to,))
+            self.get_applicable_edit.invalidate((relates_to,))

synapse/replication/tcp/streams/_base.py

@@ -32,6 +32,7 @@ BackfillStreamRow = namedtuple("BackfillStreamRow", (
     "type",  # str
     "state_key",  # str, optional
     "redacts",  # str, optional
+    "relates_to",  # str, optional
 ))
 PresenceStreamRow = namedtuple("PresenceStreamRow", (
     "user_id",  # str

synapse/replication/tcp/streams/events.py

@@ -80,11 +80,12 @@ class BaseEventsStreamRow(object):
 class EventsStreamEventRow(BaseEventsStreamRow):
     TypeId = "ev"
 
-    event_id = attr.ib()   # str
-    room_id = attr.ib()    # str
-    type = attr.ib()       # str
-    state_key = attr.ib()  # str, optional
-    redacts = attr.ib()    # str, optional
+    event_id = attr.ib()    # str
+    room_id = attr.ib()     # str
+    type = attr.ib()        # str
+    state_key = attr.ib()   # str, optional
+    redacts = attr.ib()     # str, optional
+    relates_to = attr.ib()  # str, optional
 
 
 @attr.s(slots=True, frozen=True)

synapse/rest/__init__.py

@@ -41,10 +41,10 @@ from synapse.rest.client.v2_alpha import (
     keys,
     notifications,
     openid,
-    password_policy,
     read_marker,
     receipts,
     register,
+    relations,
     report_event,
     room_keys,
     room_upgrade_rest_servlet,
@@ -116,7 +116,7 @@ class ClientRestResource(JsonResource):
         room_upgrade_rest_servlet.register_servlets(hs, client_resource)
         capabilities.register_servlets(hs, client_resource)
         account_validity.register_servlets(hs, client_resource)
-        password_policy.register_servlets(hs, client_resource)
+        relations.register_servlets(hs, client_resource)
 
         # moving to /_synapse/admin
         synapse.rest.admin.register_servlets_for_client_rest_resource(

synapse/rest/client/v1/profile.py

@@ -14,8 +14,6 @@
 # limitations under the License.
 
 """ This module contains REST servlets to do with profile: /profile/<paths> """
-import logging
-
 from twisted.internet import defer
 
 from synapse.http.servlet import parse_json_object_from_request
@@ -23,8 +21,6 @@ from synapse.types import UserID
 
 from .base import ClientV1RestServlet, client_path_patterns
 
-logger = logging.getLogger(__name__)
-
 
 class ProfileDisplaynameRestServlet(ClientV1RestServlet):
     PATTERNS = client_path_patterns("/profile/(?P<user_id>[^/]*)/displayname")
@@ -32,7 +28,6 @@ class ProfileDisplaynameRestServlet(ClientV1RestServlet):
     def __init__(self, hs):
         super(ProfileDisplaynameRestServlet, self).__init__(hs)
         self.profile_handler = hs.get_profile_handler()
-        self.http_client = hs.get_simple_http_client()
 
     @defer.inlineCallbacks
     def on_GET(self, request, user_id):
@@ -70,30 +65,11 @@ class ProfileDisplaynameRestServlet(ClientV1RestServlet):
         yield self.profile_handler.set_displayname(
             user, requester, new_name, is_admin)
 
-        if self.hs.config.shadow_server:
-            shadow_user = UserID(
-                user.localpart, self.hs.config.shadow_server.get("hs")
-            )
-            self.shadow_displayname(shadow_user.to_string(), content)
-
         defer.returnValue((200, {}))
 
     def on_OPTIONS(self, request, user_id):
         return (200, {})
 
-    @defer.inlineCallbacks
-    def shadow_displayname(self, user_id, body):
-        # TODO: retries
-        shadow_hs_url = self.hs.config.shadow_server.get("hs_url")
-        as_token = self.hs.config.shadow_server.get("as_token")
-
-        yield self.http_client.put_json(
-            "%s/_matrix/client/r0/profile/%s/displayname?access_token=%s&user_id=%s" % (
-                shadow_hs_url, user_id, as_token, user_id
-            ),
-            body
-        )
-
 
 class ProfileAvatarURLRestServlet(ClientV1RestServlet):
     PATTERNS = client_path_patterns("/profile/(?P<user_id>[^/]*)/avatar_url")
@@ -101,7 +77,6 @@ class ProfileAvatarURLRestServlet(ClientV1RestServlet):
     def __init__(self, hs):
         super(ProfileAvatarURLRestServlet, self).__init__(hs)
         self.profile_handler = hs.get_profile_handler()
-        self.http_client = hs.get_simple_http_client()
 
     @defer.inlineCallbacks
     def on_GET(self, request, user_id):
@@ -138,30 +113,11 @@ class ProfileAvatarURLRestServlet(ClientV1RestServlet):
         yield self.profile_handler.set_avatar_url(
             user, requester, new_name, is_admin)
 
-        if self.hs.config.shadow_server:
-            shadow_user = UserID(
-                user.localpart, self.hs.config.shadow_server.get("hs")
-            )
-            self.shadow_avatar_url(shadow_user.to_string(), content)
-
         defer.returnValue((200, {}))
 
     def on_OPTIONS(self, request, user_id):
         return (200, {})
 
-    @defer.inlineCallbacks
-    def shadow_avatar_url(self, user_id, body):
-        # TODO: retries
-        shadow_hs_url = self.hs.config.shadow_server.get("hs_url")
-        as_token = self.hs.config.shadow_server.get("as_token")
-
-        yield self.http_client.put_json(
-            "%s/_matrix/client/r0/profile/%s/avatar_url?access_token=%s&user_id=%s" % (
-                shadow_hs_url, user_id, as_token, user_id
-            ),
-            body
-        )
-
 
 class ProfileRestServlet(ClientV1RestServlet):
     PATTERNS = client_path_patterns("/profile/(?P<user_id>[^/]*)")

synapse/rest/client/v1/room.py

@@ -201,6 +201,11 @@ class RoomSendEventRestServlet(ClientV1RestServlet):
         requester = yield self.auth.get_user_by_req(request, allow_guest=True)
         content = parse_json_object_from_request(request)
 
+        # Pull out the relationship early if the client sent us something
+        # which cannot possibly be processed by us.
+        if content.get("m.relates_to", "not None") is None:
+            del content["m.relates_to"]
+
         event_dict = {
             "type": event_type,
             "content": content,
@@ -677,8 +682,7 @@ class RoomMembershipRestServlet(ClientV1RestServlet):
                 content["address"],
                 content["id_server"],
                 requester,
-                txn_id,
-                new_room=False,
+                txn_id
             )
             defer.returnValue((200, {}))
             return

synapse/rest/client/v2_alpha/account.py

@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 # Copyright 2015, 2016 OpenMarket Ltd
 # Copyright 2017 Vector Creations Ltd
-# Copyright 2018, 2019 New Vector Ltd
+# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,7 +15,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import logging
-import re
 
 from six.moves import http_client
 
@@ -27,9 +26,7 @@ from synapse.http.servlet import (
     RestServlet,
     assert_params_in_dict,
     parse_json_object_from_request,
-    parse_string,
 )
-from synapse.types import UserID
 from synapse.util.msisdn import phone_number_to_msisdn
 from synapse.util.threepids import check_3pid_allowed
 
@@ -54,10 +51,10 @@ class EmailPasswordRequestTokenRestServlet(RestServlet):
             'id_server', 'client_secret', 'email', 'send_attempt'
         ])
 
-        if not (yield check_3pid_allowed(self.hs, "email", body['email'])):
+        if not check_3pid_allowed(self.hs, "email", body['email']):
             raise SynapseError(
                 403,
-                "Your email is not authorized on this server",
+                "Your email domain is not authorized on this server",
                 Codes.THREEPID_DENIED,
             )
 
@@ -92,7 +89,7 @@ class MsisdnPasswordRequestTokenRestServlet(RestServlet):
 
         msisdn = phone_number_to_msisdn(body['country'], body['phone_number'])
 
-        if not (yield check_3pid_allowed(self.hs, "msisdn", msisdn)):
+        if not check_3pid_allowed(self.hs, "msisdn", msisdn):
             raise SynapseError(
                 403,
                 "Account phone numbers are not authorized on this server",
@@ -120,7 +117,6 @@ class PasswordRestServlet(RestServlet):
         self.auth_handler = hs.get_auth_handler()
         self.datastore = self.hs.get_datastore()
         self._set_password_handler = hs.get_set_password_handler()
-        self.http_client = hs.get_simple_http_client()
 
     @interactive_auth_handler
     @defer.inlineCallbacks
@@ -139,13 +135,9 @@ class PasswordRestServlet(RestServlet):
 
         if self.auth.has_access_token(request):
             requester = yield self.auth.get_user_by_req(request)
-            # blindly trust ASes without UI-authing them
-            if requester.app_service:
-                params = body
-            else:
-                params = yield self.auth_handler.validate_user_via_ui_auth(
-                    requester, body, self.hs.get_ip_from_request(request),
-                )
+            params = yield self.auth_handler.validate_user_via_ui_auth(
+                requester, body, self.hs.get_ip_from_request(request),
+            )
             user_id = requester.user.to_string()
         else:
             requester = None
@@ -181,30 +173,11 @@ class PasswordRestServlet(RestServlet):
             user_id, new_password, requester
         )
 
-        if self.hs.config.shadow_server:
-            shadow_user = UserID(
-                requester.user.localpart, self.hs.config.shadow_server.get("hs")
-            )
-            self.shadow_password(params, shadow_user.to_string())
-
         defer.returnValue((200, {}))
 
     def on_OPTIONS(self, _):
         return 200, {}
 
-    @defer.inlineCallbacks
-    def shadow_password(self, body, user_id):
-        # TODO: retries
-        shadow_hs_url = self.hs.config.shadow_server.get("hs_url")
-        as_token = self.hs.config.shadow_server.get("as_token")
-
-        yield self.http_client.post_json_get_json(
-            "%s/_matrix/client/r0/account/password?access_token=%s&user_id=%s" % (
-                shadow_hs_url, as_token, user_id,
-            ),
-            body
-        )
-
 
 class DeactivateAccountRestServlet(RestServlet):
     PATTERNS = client_v2_patterns("/account/deactivate$")
@@ -271,10 +244,10 @@ class EmailThreepidRequestTokenRestServlet(RestServlet):
             ['id_server', 'client_secret', 'email', 'send_attempt'],
         )
 
-        if not (yield check_3pid_allowed(self.hs, "email", body['email'])):
+        if not check_3pid_allowed(self.hs, "email", body['email']):
             raise SynapseError(
                 403,
-                "Your email is not authorized on this server",
+                "Your email domain is not authorized on this server",
                 Codes.THREEPID_DENIED,
             )
 
@@ -308,7 +281,7 @@ class MsisdnThreepidRequestTokenRestServlet(RestServlet):
 
         msisdn = phone_number_to_msisdn(body['country'], body['phone_number'])
 
-        if not (yield check_3pid_allowed(self.hs, "msisdn", msisdn)):
+        if not check_3pid_allowed(self.hs, "msisdn", msisdn):
             raise SynapseError(
                 403,
                 "Account phone numbers are not authorized on this server",
@@ -335,8 +308,7 @@ class ThreepidRestServlet(RestServlet):
         self.identity_handler = hs.get_handlers().identity_handler
         self.auth = hs.get_auth()
         self.auth_handler = hs.get_auth_handler()
-        self.datastore = hs.get_datastore()
-        self.http_client = hs.get_simple_http_client()
+        self.datastore = self.hs.get_datastore()
 
     @defer.inlineCallbacks
     def on_GET(self, request):
@@ -350,38 +322,27 @@ class ThreepidRestServlet(RestServlet):
 
     @defer.inlineCallbacks
     def on_POST(self, request):
-        if self.hs.config.disable_3pid_changes:
-            raise SynapseError(400, "3PID changes disabled on this server")
-
         body = parse_json_object_from_request(request)
 
+        threePidCreds = body.get('threePidCreds')
+        threePidCreds = body.get('three_pid_creds', threePidCreds)
+        if threePidCreds is None:
+            raise SynapseError(400, "Missing param", Codes.MISSING_PARAM)
+
         requester = yield self.auth.get_user_by_req(request)
         user_id = requester.user.to_string()
 
-        # skip validation if this is a shadow 3PID from an AS
-        if not requester.app_service:
-            threePidCreds = body.get('threePidCreds')
-            threePidCreds = body.get('three_pid_creds', threePidCreds)
-            if threePidCreds is None:
-                raise SynapseError(400, "Missing param", Codes.MISSING_PARAM)
+        threepid = yield self.identity_handler.threepid_from_creds(threePidCreds)
 
-            threepid = yield self.identity_handler.threepid_from_creds(threePidCreds)
+        if not threepid:
+            raise SynapseError(
+                400, "Failed to auth 3pid", Codes.THREEPID_AUTH_FAILED
+            )
 
-            if not threepid:
-                raise SynapseError(
-                    400, "Failed to auth 3pid", Codes.THREEPID_AUTH_FAILED
-                )
-
-            for reqd in ['medium', 'address', 'validated_at']:
-                if reqd not in threepid:
-                    logger.warn("Couldn't add 3pid: invalid response from ID server")
-                    raise SynapseError(500, "Invalid response from ID Server")
-        else:
-            # XXX: ASes pass in a validated threepid directly to bypass the IS.
-            # This makes the API entirely change shape when we have an AS token;
-            # it really should be an entirely separate API - perhaps
-            # /account/3pid/replicate or something.
-            threepid = body.get('threepid')
+        for reqd in ['medium', 'address', 'validated_at']:
+            if reqd not in threepid:
+                logger.warn("Couldn't add 3pid: invalid response from ID server")
+                raise SynapseError(500, "Invalid response from ID Server")
 
         yield self.auth_handler.add_threepid(
             user_id,
@@ -390,7 +351,7 @@ class ThreepidRestServlet(RestServlet):
             threepid['validated_at'],
         )
 
-        if not requester.app_service and ('bind' in body and body['bind']):
+        if 'bind' in body and body['bind']:
             logger.debug(
                 "Binding threepid %s to %s",
                 threepid, user_id
@@ -399,43 +360,19 @@ class ThreepidRestServlet(RestServlet):
                 threePidCreds, user_id
             )
 
-        if self.hs.config.shadow_server:
-            shadow_user = UserID(
-                requester.user.localpart, self.hs.config.shadow_server.get("hs")
-            )
-            self.shadow_3pid({'threepid': threepid}, shadow_user.to_string())
-
         defer.returnValue((200, {}))
 
-    @defer.inlineCallbacks
-    def shadow_3pid(self, body, user_id):
-        # TODO: retries
-        shadow_hs_url = self.hs.config.shadow_server.get("hs_url")
-        as_token = self.hs.config.shadow_server.get("as_token")
-
-        yield self.http_client.post_json_get_json(
-            "%s/_matrix/client/r0/account/3pid?access_token=%s&user_id=%s" % (
-                shadow_hs_url, as_token, user_id,
-            ),
-            body
-        )
-
 
 class ThreepidDeleteRestServlet(RestServlet):
     PATTERNS = client_v2_patterns("/account/3pid/delete$")
 
     def __init__(self, hs):
         super(ThreepidDeleteRestServlet, self).__init__()
-        self.hs = hs
         self.auth = hs.get_auth()
         self.auth_handler = hs.get_auth_handler()
-        self.http_client = hs.get_simple_http_client()
 
     @defer.inlineCallbacks
     def on_POST(self, request):
-        if self.hs.config.disable_3pid_changes:
-            raise SynapseError(400, "3PID changes disabled on this server")
-
         body = parse_json_object_from_request(request)
         assert_params_in_dict(body, ['medium', 'address'])
 
@@ -453,12 +390,6 @@ class ThreepidDeleteRestServlet(RestServlet):
             logger.exception("Failed to remove threepid")
             raise SynapseError(500, "Failed to remove threepid")
 
-        if self.hs.config.shadow_server:
-            shadow_user = UserID(
-                requester.user.localpart, self.hs.config.shadow_server.get("hs")
-            )
-            self.shadow_3pid_delete(body, shadow_user.to_string())
-
         if ret:
             id_server_unbind_result = "success"
         else:
@@ -468,78 +399,6 @@ class ThreepidDeleteRestServlet(RestServlet):
             "id_server_unbind_result": id_server_unbind_result,
         }))
 
-    @defer.inlineCallbacks
-    def shadow_3pid_delete(self, body, user_id):
-        # TODO: retries
-        shadow_hs_url = self.hs.config.shadow_server.get("hs_url")
-        as_token = self.hs.config.shadow_server.get("as_token")
-
-        yield self.http_client.post_json_get_json(
-            "%s/_matrix/client/r0/account/3pid/delete?access_token=%s&user_id=%s" % (
-                shadow_hs_url, as_token, user_id
-            ),
-            body
-        )
-
-
-class ThreepidLookupRestServlet(RestServlet):
-    PATTERNS = [re.compile("^/_matrix/client/unstable/account/3pid/lookup$")]
-
-    def __init__(self, hs):
-        super(ThreepidLookupRestServlet, self).__init__()
-        self.auth = hs.get_auth()
-        self.identity_handler = hs.get_handlers().identity_handler
-
-    @defer.inlineCallbacks
-    def on_GET(self, request):
-        """Proxy a /_matrix/identity/api/v1/lookup request to an identity
-        server
-        """
-        yield self.auth.get_user_by_req(request)
-
-        # Verify query parameters
-        query_params = request.args
-        assert_params_in_dict(query_params, [b"medium", b"address", b"id_server"])
-
-        # Retrieve needed information from query parameters
-        medium = parse_string(request, "medium")
-        address = parse_string(request, "address")
-        id_server = parse_string(request, "id_server")
-
-        # Proxy the request to the identity server. lookup_3pid handles checking
-        # if the lookup is allowed so we don't need to do it here.
-        ret = yield self.identity_handler.lookup_3pid(id_server, medium, address)
-
-        defer.returnValue((200, ret))
-
-
-class ThreepidBulkLookupRestServlet(RestServlet):
-    PATTERNS = [re.compile("^/_matrix/client/unstable/account/3pid/bulk_lookup$")]
-
-    def __init__(self, hs):
-        super(ThreepidBulkLookupRestServlet, self).__init__()
-        self.auth = hs.get_auth()
-        self.identity_handler = hs.get_handlers().identity_handler
-
-    @defer.inlineCallbacks
-    def on_POST(self, request):
-        """Proxy a /_matrix/identity/api/v1/bulk_lookup request to an identity
-        server
-        """
-        yield self.auth.get_user_by_req(request)
-
-        body = parse_json_object_from_request(request)
-
-        assert_params_in_dict(body, ["threepids", "id_server"])
-
-        # Proxy the request to the identity server. lookup_3pid handles checking
-        # if the lookup is allowed so we don't need to do it here.
-        ret = yield self.identity_handler.bulk_lookup_3pid(
-            body["id_server"], body["threepids"],
-        )
-
-        defer.returnValue((200, ret))
-
 
 class WhoamiRestServlet(RestServlet):
     PATTERNS = client_v2_patterns("/account/whoami$")
@@ -564,6 +423,4 @@ def register_servlets(hs, http_server):
     MsisdnThreepidRequestTokenRestServlet(hs).register(http_server)
     ThreepidRestServlet(hs).register(http_server)
     ThreepidDeleteRestServlet(hs).register(http_server)
-    ThreepidLookupRestServlet(hs).register(http_server)
-    ThreepidBulkLookupRestServlet(hs).register(http_server)
     WhoamiRestServlet(hs).register(http_server)

synapse/rest/client/v2_alpha/account_data.py

@@ -19,7 +19,6 @@ from twisted.internet import defer
 
 from synapse.api.errors import AuthError, NotFoundError, SynapseError
 from synapse.http.servlet import RestServlet, parse_json_object_from_request
-from synapse.types import UserID
 
 from ._base import client_v2_patterns
 
@@ -40,7 +39,6 @@ class AccountDataServlet(RestServlet):
         self.auth = hs.get_auth()
         self.store = hs.get_datastore()
         self.notifier = hs.get_notifier()
-        self._profile_handler = hs.get_profile_handler()
 
     @defer.inlineCallbacks
     def on_PUT(self, request, user_id, account_data_type):
@@ -50,11 +48,6 @@ class AccountDataServlet(RestServlet):
 
         body = parse_json_object_from_request(request)
 
-        if account_data_type == "im.vector.hide_profile":
-            user = UserID.from_string(user_id)
-            hide_profile = body.get('hide_profile')
-            yield self._profile_handler.set_active(user, not hide_profile, True)
-
         max_id = yield self.store.add_account_data_for_user(
             user_id, account_data_type, body
         )

synapse/rest/client/v2_alpha/password_policy.py

@@ -1,58 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2019 The Matrix.org Foundation C.I.C.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import logging
-
-from synapse.http.servlet import RestServlet
-
-from ._base import client_v2_patterns
-
-logger = logging.getLogger(__name__)
-
-
-class PasswordPolicyServlet(RestServlet):
-    PATTERNS = client_v2_patterns("/password_policy$")
-
-    def __init__(self, hs):
-        """
-        Args:
-            hs (synapse.server.HomeServer): server
-        """
-        super(PasswordPolicyServlet, self).__init__()
-
-        self.policy = hs.config.password_policy
-        self.enabled = hs.config.password_policy_enabled
-
-    def on_GET(self, request):
-        if not self.enabled or not self.policy:
-            return (200, {})
-
-        policy = {}
-
-        for param in [
-            "minimum_length",
-            "require_digit",
-            "require_symbol",
-            "require_lowercase",
-            "require_uppercase",
-        ]:
-            if param in self.policy:
-                policy["m.%s" % param] = self.policy[param]
-
-        return (200, policy)
-
-
-def register_servlets(hs, http_server):
-    PasswordPolicyServlet(hs).register(http_server)

synapse/rest/client/v2_alpha/register.py

@@ -1,7 +1,6 @@
 # -*- coding: utf-8 -*-
-# Copyright 2015-2016 OpenMarket Ltd
-# Copyright 2017-2018 New Vector Ltd
-# Copyright 2019 The Matrix.org Foundation C.I.C.
+# Copyright 2015 - 2016 OpenMarket Ltd
+# Copyright 2017 Vector Creations Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -17,9 +16,7 @@
 
 import hmac
 import logging
-import re
 from hashlib import sha1
-from string import capwords
 
 from six import string_types
 
@@ -82,10 +79,10 @@ class EmailRegisterRequestTokenRestServlet(RestServlet):
             'id_server', 'client_secret', 'email', 'send_attempt'
         ])
 
-        if not (yield check_3pid_allowed(self.hs, "email", body['email'])):
+        if not check_3pid_allowed(self.hs, "email", body['email']):
             raise SynapseError(
                 403,
-                "Your email is not authorized to register on this server",
+                "Your email domain is not authorized to register on this server",
                 Codes.THREEPID_DENIED,
             )
 
@@ -124,7 +121,7 @@ class MsisdnRegisterRequestTokenRestServlet(RestServlet):
 
         msisdn = phone_number_to_msisdn(body['country'], body['phone_number'])
 
-        if not (yield check_3pid_allowed(self.hs, "msisdn", msisdn)):
+        if not check_3pid_allowed(self.hs, "msisdn", msisdn):
             raise SynapseError(
                 403,
                 "Phone numbers are not authorized to register on this server",
@@ -203,7 +200,6 @@ class RegisterRestServlet(RestServlet):
         self.room_member_handler = hs.get_room_member_handler()
         self.macaroon_gen = hs.get_macaroon_generator()
         self.ratelimiter = hs.get_registration_ratelimiter()
-        self.password_policy_handler = hs.get_password_policy_handler()
         self.clock = hs.get_clock()
 
     @interactive_auth_handler
@@ -247,7 +243,6 @@ class RegisterRestServlet(RestServlet):
             if (not isinstance(body['password'], string_types) or
                     len(body['password']) > 512):
                 raise SynapseError(400, "Invalid password")
-            self.password_policy_handler.validate_password(body['password'])
             desired_password = body["password"]
 
         desired_username = None
@@ -257,8 +252,6 @@ class RegisterRestServlet(RestServlet):
                 raise SynapseError(400, "Invalid username")
             desired_username = body['username']
 
-        desired_display_name = body.get('display_name')
-
         appservice = None
         if self.auth.has_access_token(request):
             appservice = yield self.auth.get_appservice_by_req(request)
@@ -282,8 +275,7 @@ class RegisterRestServlet(RestServlet):
 
             if isinstance(desired_username, string_types):
                 result = yield self._do_appservice_registration(
-                    desired_username, desired_password, desired_display_name,
-                    access_token, body
+                    desired_username, access_token, body
                 )
             defer.returnValue((200, result))  # we throw for non 200 responses
             return
@@ -356,18 +348,22 @@ class RegisterRestServlet(RestServlet):
         if self.hs.config.enable_registration_captcha:
             # only support 3PIDless registration if no 3PIDs are required
             if not require_email and not require_msisdn:
-                flows.extend([[LoginType.RECAPTCHA]])
+                # Also add a dummy flow here, otherwise if a client completes
+                # recaptcha first we'll assume they were going for this flow
+                # and complete the request, when they could have been trying to
+                # complete one of the flows with email/msisdn auth.
+                flows.extend([[LoginType.RECAPTCHA, LoginType.DUMMY]])
             # only support the email-only flow if we don't require MSISDN 3PIDs
             if not require_msisdn:
-                flows.extend([[LoginType.EMAIL_IDENTITY, LoginType.RECAPTCHA]])
+                flows.extend([[LoginType.RECAPTCHA, LoginType.EMAIL_IDENTITY]])
 
             if show_msisdn:
                 # only support the MSISDN-only flow if we don't require email 3PIDs
                 if not require_email:
-                    flows.extend([[LoginType.MSISDN, LoginType.RECAPTCHA]])
+                    flows.extend([[LoginType.RECAPTCHA, LoginType.MSISDN]])
                 # always let users provide both MSISDN & email
                 flows.extend([
-                    [LoginType.MSISDN, LoginType.EMAIL_IDENTITY, LoginType.RECAPTCHA],
+                    [LoginType.RECAPTCHA, LoginType.MSISDN, LoginType.EMAIL_IDENTITY],
                 ])
         else:
             # only support 3PIDless registration if no 3PIDs are required
@@ -390,7 +386,15 @@ class RegisterRestServlet(RestServlet):
         if self.hs.config.user_consent_at_registration:
             new_flows = []
             for flow in flows:
-                flow.append(LoginType.TERMS)
+                inserted = False
+                # m.login.terms should go near the end but before msisdn or email auth
+                for i, stage in enumerate(flow):
+                    if stage == LoginType.EMAIL_IDENTITY or stage == LoginType.MSISDN:
+                        flow.insert(i, LoginType.TERMS)
+                        inserted = True
+                        break
+                if not inserted:
+                    flow.append(LoginType.TERMS)
             flows.extend(new_flows)
 
         auth_result, params, session_id = yield self.auth_handler.check_auth(
@@ -409,7 +413,7 @@ class RegisterRestServlet(RestServlet):
                     medium = auth_result[login_type]['medium']
                     address = auth_result[login_type]['address']
 
-                    if not (yield check_3pid_allowed(self.hs, medium, address)):
+                    if not check_3pid_allowed(self.hs, medium, address):
                         raise SynapseError(
                             403,
                             "Third party identifiers (email/phone numbers)" +
@@ -417,95 +421,6 @@ class RegisterRestServlet(RestServlet):
                             Codes.THREEPID_DENIED,
                         )
 
-                    existingUid = yield self.store.get_user_id_by_threepid(
-                        medium, address,
-                    )
-
-                    if existingUid is not None:
-                        raise SynapseError(
-                            400,
-                            "%s is already in use" % medium,
-                            Codes.THREEPID_IN_USE,
-                        )
-
-        if self.hs.config.register_mxid_from_3pid:
-            # override the desired_username based on the 3PID if any.
-            # reset it first to avoid folks picking their own username.
-            desired_username = None
-
-            # we should have an auth_result at this point if we're going to progress
-            # to register the user (i.e. we haven't picked up a registered_user_id
-            # from our session store), in which case get ready and gen the
-            # desired_username
-            if auth_result:
-                if (
-                    self.hs.config.register_mxid_from_3pid == 'email' and
-                    LoginType.EMAIL_IDENTITY in auth_result
-                ):
-                    address = auth_result[LoginType.EMAIL_IDENTITY]['address']
-                    desired_username = synapse.types.strip_invalid_mxid_characters(
-                        address.replace('@', '-').lower()
-                    )
-
-                    # find a unique mxid for the account, suffixing numbers
-                    # if needed
-                    while True:
-                        try:
-                            yield self.registration_handler.check_username(
-                                desired_username,
-                                guest_access_token=guest_access_token,
-                                assigned_user_id=registered_user_id,
-                            )
-                            # if we got this far we passed the check.
-                            break
-                        except SynapseError as e:
-                            if e.errcode == Codes.USER_IN_USE:
-                                m = re.match(r'^(.*?)(\d+)$', desired_username)
-                                if m:
-                                    desired_username = m.group(1) + str(
-                                        int(m.group(2)) + 1
-                                    )
-                                else:
-                                    desired_username += "1"
-                            else:
-                                # something else went wrong.
-                                break
-
-                    if self.hs.config.register_just_use_email_for_display_name:
-                        desired_display_name = address
-                    else:
-                        # XXX: a nasty heuristic to turn an email address into
-                        # a displayname, as part of register_mxid_from_3pid
-                        parts = address.replace('.', ' ').split('@')
-                        org_parts = parts[1].split(' ')
-
-                        if org_parts[-2] == "matrix" and org_parts[-1] == "org":
-                            org = "Tchap Admin"
-                        elif org_parts[-2] == "gouv" and org_parts[-1] == "fr":
-                            org = org_parts[-3] if len(org_parts) > 2 else org_parts[-2]
-                        else:
-                            org = org_parts[-2]
-
-                        desired_display_name = (
-                            capwords(parts[0]) + " [" + capwords(org) + "]"
-                        )
-                elif (
-                    self.hs.config.register_mxid_from_3pid == 'msisdn' and
-                    LoginType.MSISDN in auth_result
-                ):
-                    desired_username = auth_result[LoginType.MSISDN]['address']
-                else:
-                    raise SynapseError(
-                        400, "Cannot derive mxid from 3pid; no recognised 3pid"
-                    )
-
-        if desired_username is not None:
-            yield self.registration_handler.check_username(
-                desired_username,
-                guest_access_token=guest_access_token,
-                assigned_user_id=registered_user_id,
-            )
-
         if registered_user_id is not None:
             logger.info(
                 "Already registered user ID %r for this session",
@@ -517,16 +432,9 @@ class RegisterRestServlet(RestServlet):
             # NB: This may be from the auth handler and NOT from the POST
             assert_params_in_dict(params, ["password"])
 
-            if not self.hs.config.register_mxid_from_3pid:
-                desired_username = params.get("username", None)
-            else:
-                # we keep the original desired_username derived from the 3pid above
-                pass
-
+            desired_username = params.get("username", None)
             guest_access_token = params.get("guest_access_token", None)
-
-            # XXX: don't we need to validate these for length etc like we did on
-            # the ones from the JSON body earlier on in the method?
+            new_password = params.get("password", None)
 
             if desired_username is not None:
                 desired_username = desired_username.lower()
@@ -559,10 +467,9 @@ class RegisterRestServlet(RestServlet):
 
             (registered_user_id, _) = yield self.registration_handler.register(
                 localpart=desired_username,
-                password=params.get("password", None),
+                password=new_password,
                 guest_access_token=guest_access_token,
                 generate_token=False,
-                default_display_name=desired_display_name,
                 threepid=threepid,
                 address=client_addr,
             )
@@ -574,14 +481,6 @@ class RegisterRestServlet(RestServlet):
                 ):
                     yield self.store.upsert_monthly_active_user(registered_user_id)
 
-            if self.hs.config.shadow_server:
-                yield self.registration_handler.shadow_register(
-                    localpart=desired_username,
-                    display_name=desired_display_name,
-                    auth_result=auth_result,
-                    params=params,
-                )
-
             # remember that we've now registered that user account, and with
             #  what user ID (since the user may not have specified)
             self.auth_handler.set_session_data(
@@ -609,33 +508,11 @@ class RegisterRestServlet(RestServlet):
         return 200, {}
 
     @defer.inlineCallbacks
-    def _do_appservice_registration(
-        self, username, password, display_name, as_token, body
-    ):
-
-        # FIXME: appservice_register() is horribly duplicated with register()
-        # and they should probably just be combined together with a config flag.
+    def _do_appservice_registration(self, username, as_token, body):
         user_id = yield self.registration_handler.appservice_register(
-            username, as_token, password, display_name
+            username, as_token
         )
-        result = yield self._create_registration_details(user_id, body)
-
-        auth_result = body.get('auth_result')
-        if auth_result and LoginType.EMAIL_IDENTITY in auth_result:
-            threepid = auth_result[LoginType.EMAIL_IDENTITY]
-            yield self._register_email_threepid(
-                user_id, threepid, result["access_token"],
-                body.get("bind_email")
-            )
-
-        if auth_result and LoginType.MSISDN in auth_result:
-            threepid = auth_result[LoginType.MSISDN]
-            yield self._register_msisdn_threepid(
-                user_id, threepid, result["access_token"],
-                body.get("bind_msisdn")
-            )
-
-        defer.returnValue(result)
+        defer.returnValue((yield self._create_registration_details(user_id, body)))
 
     @defer.inlineCallbacks
     def _do_shared_secret_registration(self, username, password, body):

synapse/rest/client/v2_alpha/relations.py

@@ -0,0 +1,338 @@
+# -*- coding: utf-8 -*-
+# Copyright 2019 New Vector Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""This class implements the proposed relation APIs from MSC 1849.
+
+Since the MSC has not been approved all APIs here are unstable and may change at
+any time to reflect changes in the MSC.
+"""
+
+import logging
+
+from twisted.internet import defer
+
+from synapse.api.constants import EventTypes, RelationTypes
+from synapse.api.errors import SynapseError
+from synapse.http.servlet import (
+    RestServlet,
+    parse_integer,
+    parse_json_object_from_request,
+    parse_string,
+)
+from synapse.rest.client.transactions import HttpTransactionCache
+from synapse.storage.relations import AggregationPaginationToken, RelationPaginationToken
+
+from ._base import client_v2_patterns
+
+logger = logging.getLogger(__name__)
+
+
+class RelationSendServlet(RestServlet):
+    """Helper API for sending events that have relation data.
+
+    Example API shape to send a 👍 reaction to a room:
+
+        POST /rooms/!foo/send_relation/$bar/m.annotation/m.reaction?key=%F0%9F%91%8D
+        {}
+
+        {
+            "event_id": "$foobar"
+        }
+    """
+
+    PATTERN = (
+        "/rooms/(?P<room_id>[^/]*)/send_relation"
+        "/(?P<parent_id>[^/]*)/(?P<relation_type>[^/]*)/(?P<event_type>[^/]*)"
+    )
+
+    def __init__(self, hs):
+        super(RelationSendServlet, self).__init__()
+        self.auth = hs.get_auth()
+        self.event_creation_handler = hs.get_event_creation_handler()
+        self.txns = HttpTransactionCache(hs)
+
+    def register(self, http_server):
+        http_server.register_paths(
+            "POST",
+            client_v2_patterns(self.PATTERN + "$", releases=()),
+            self.on_PUT_or_POST,
+        )
+        http_server.register_paths(
+            "PUT",
+            client_v2_patterns(self.PATTERN + "/(?P<txn_id>[^/]*)$", releases=()),
+            self.on_PUT,
+        )
+
+    def on_PUT(self, request, *args, **kwargs):
+        return self.txns.fetch_or_execute_request(
+            request, self.on_PUT_or_POST, request, *args, **kwargs
+        )
+
+    @defer.inlineCallbacks
+    def on_PUT_or_POST(
+        self, request, room_id, parent_id, relation_type, event_type, txn_id=None
+    ):
+        requester = yield self.auth.get_user_by_req(request, allow_guest=True)
+
+        if event_type == EventTypes.Member:
+            # Add relations to a membership is meaningless, so we just deny it
+            # at the CS API rather than trying to handle it correctly.
+            raise SynapseError(400, "Cannot send member events with relations")
+
+        content = parse_json_object_from_request(request)
+
+        aggregation_key = parse_string(request, "key", encoding="utf-8")
+
+        content["m.relates_to"] = {
+            "event_id": parent_id,
+            "key": aggregation_key,
+            "rel_type": relation_type,
+        }
+
+        event_dict = {
+            "type": event_type,
+            "content": content,
+            "room_id": room_id,
+            "sender": requester.user.to_string(),
+        }
+
+        event = yield self.event_creation_handler.create_and_send_nonmember_event(
+            requester, event_dict=event_dict, txn_id=txn_id
+        )
+
+        defer.returnValue((200, {"event_id": event.event_id}))
+
+
+class RelationPaginationServlet(RestServlet):
+    """API to paginate relations on an event by topological ordering, optionally
+    filtered by relation type and event type.
+    """
+
+    PATTERNS = client_v2_patterns(
+        "/rooms/(?P<room_id>[^/]*)/relations/(?P<parent_id>[^/]*)"
+        "(/(?P<relation_type>[^/]*)(/(?P<event_type>[^/]*))?)?$",
+        releases=(),
+    )
+
+    def __init__(self, hs):
+        super(RelationPaginationServlet, self).__init__()
+        self.auth = hs.get_auth()
+        self.store = hs.get_datastore()
+        self.clock = hs.get_clock()
+        self._event_serializer = hs.get_event_client_serializer()
+        self.event_handler = hs.get_event_handler()
+
+    @defer.inlineCallbacks
+    def on_GET(self, request, room_id, parent_id, relation_type=None, event_type=None):
+        requester = yield self.auth.get_user_by_req(request, allow_guest=True)
+
+        yield self.auth.check_in_room_or_world_readable(
+            room_id, requester.user.to_string()
+        )
+
+        # This checks that a) the event exists and b) the user is allowed to
+        # view it.
+        yield self.event_handler.get_event(requester.user, room_id, parent_id)
+
+        limit = parse_integer(request, "limit", default=5)
+        from_token = parse_string(request, "from")
+        to_token = parse_string(request, "to")
+
+        if from_token:
+            from_token = RelationPaginationToken.from_string(from_token)
+
+        if to_token:
+            to_token = RelationPaginationToken.from_string(to_token)
+
+        result = yield self.store.get_relations_for_event(
+            event_id=parent_id,
+            relation_type=relation_type,
+            event_type=event_type,
+            limit=limit,
+            from_token=from_token,
+            to_token=to_token,
+        )
+
+        events = yield self.store.get_events_as_list(
+            [c["event_id"] for c in result.chunk]
+        )
+
+        now = self.clock.time_msec()
+        events = yield self._event_serializer.serialize_events(events, now)
+
+        return_value = result.to_dict()
+        return_value["chunk"] = events
+
+        defer.returnValue((200, return_value))
+
+
+class RelationAggregationPaginationServlet(RestServlet):
+    """API to paginate aggregation groups of relations, e.g. paginate the
+    types and counts of the reactions on the events.
+
+    Example request and response:
+
+        GET /rooms/{room_id}/aggregations/{parent_id}
+
+        {
+            chunk: [
+                {
+                    "type": "m.reaction",
+                    "key": "👍",
+                    "count": 3
+                }
+            ]
+        }
+    """
+
+    PATTERNS = client_v2_patterns(
+        "/rooms/(?P<room_id>[^/]*)/aggregations/(?P<parent_id>[^/]*)"
+        "(/(?P<relation_type>[^/]*)(/(?P<event_type>[^/]*))?)?$",
+        releases=(),
+    )
+
+    def __init__(self, hs):
+        super(RelationAggregationPaginationServlet, self).__init__()
+        self.auth = hs.get_auth()
+        self.store = hs.get_datastore()
+        self.event_handler = hs.get_event_handler()
+
+    @defer.inlineCallbacks
+    def on_GET(self, request, room_id, parent_id, relation_type=None, event_type=None):
+        requester = yield self.auth.get_user_by_req(request, allow_guest=True)
+
+        yield self.auth.check_in_room_or_world_readable(
+            room_id, requester.user.to_string()
+        )
+
+        # This checks that a) the event exists and b) the user is allowed to
+        # view it.
+        yield self.event_handler.get_event(requester.user, room_id, parent_id)
+
+        if relation_type not in (RelationTypes.ANNOTATION, None):
+            raise SynapseError(400, "Relation type must be 'annotation'")
+
+        limit = parse_integer(request, "limit", default=5)
+        from_token = parse_string(request, "from")
+        to_token = parse_string(request, "to")
+
+        if from_token:
+            from_token = AggregationPaginationToken.from_string(from_token)
+
+        if to_token:
+            to_token = AggregationPaginationToken.from_string(to_token)
+
+        res = yield self.store.get_aggregation_groups_for_event(
+            event_id=parent_id,
+            event_type=event_type,
+            limit=limit,
+            from_token=from_token,
+            to_token=to_token,
+        )
+
+        defer.returnValue((200, res.to_dict()))
+
+
+class RelationAggregationGroupPaginationServlet(RestServlet):
+    """API to paginate within an aggregation group of relations, e.g. paginate
+    all the 👍 reactions on an event.
+
+    Example request and response:
+
+        GET /rooms/{room_id}/aggregations/{parent_id}/m.annotation/m.reaction/👍
+
+        {
+            chunk: [
+                {
+                    "type": "m.reaction",
+                    "content": {
+                        "m.relates_to": {
+                            "rel_type": "m.annotation",
+                            "key": "👍"
+                        }
+                    }
+                },
+                ...
+            ]
+        }
+    """
+
+    PATTERNS = client_v2_patterns(
+        "/rooms/(?P<room_id>[^/]*)/aggregations/(?P<parent_id>[^/]*)"
+        "/(?P<relation_type>[^/]*)/(?P<event_type>[^/]*)/(?P<key>[^/]*)$",
+        releases=(),
+    )
+
+    def __init__(self, hs):
+        super(RelationAggregationGroupPaginationServlet, self).__init__()
+        self.auth = hs.get_auth()
+        self.store = hs.get_datastore()
+        self.clock = hs.get_clock()
+        self._event_serializer = hs.get_event_client_serializer()
+        self.event_handler = hs.get_event_handler()
+
+    @defer.inlineCallbacks
+    def on_GET(self, request, room_id, parent_id, relation_type, event_type, key):
+        requester = yield self.auth.get_user_by_req(request, allow_guest=True)
+
+        yield self.auth.check_in_room_or_world_readable(
+            room_id, requester.user.to_string()
+        )
+
+        # This checks that a) the event exists and b) the user is allowed to
+        # view it.
+        yield self.event_handler.get_event(requester.user, room_id, parent_id)
+
+        if relation_type != RelationTypes.ANNOTATION:
+            raise SynapseError(400, "Relation type must be 'annotation'")
+
+        limit = parse_integer(request, "limit", default=5)
+        from_token = parse_string(request, "from")
+        to_token = parse_string(request, "to")
+
+        if from_token:
+            from_token = RelationPaginationToken.from_string(from_token)
+
+        if to_token:
+            to_token = RelationPaginationToken.from_string(to_token)
+
+        result = yield self.store.get_relations_for_event(
+            event_id=parent_id,
+            relation_type=relation_type,
+            event_type=event_type,
+            aggregation_key=key,
+            limit=limit,
+            from_token=from_token,
+            to_token=to_token,
+        )
+
+        events = yield self.store.get_events_as_list(
+            [c["event_id"] for c in result.chunk]
+        )
+
+        now = self.clock.time_msec()
+        events = yield self._event_serializer.serialize_events(events, now)
+
+        return_value = result.to_dict()
+        return_value["chunk"] = events
+
+        defer.returnValue((200, return_value))
+
+
+def register_servlets(hs, http_server):
+    RelationSendServlet(hs).register(http_server)
+    RelationPaginationServlet(hs).register(http_server)
+    RelationAggregationPaginationServlet(hs).register(http_server)
+    RelationAggregationGroupPaginationServlet(hs).register(http_server)

synapse/rest/client/v2_alpha/user_directory.py

@@ -15,8 +15,6 @@
 
 import logging
 
-from signedjson.sign import sign_json
-
 from twisted.internet import defer
 
 from synapse.api.errors import SynapseError
@@ -39,7 +37,6 @@ class UserDirectorySearchRestServlet(RestServlet):
         self.hs = hs
         self.auth = hs.get_auth()
         self.user_directory_handler = hs.get_user_directory_handler()
-        self.http_client = hs.get_simple_http_client()
 
     @defer.inlineCallbacks
     def on_POST(self, request):
@@ -70,14 +67,6 @@ class UserDirectorySearchRestServlet(RestServlet):
 
         body = parse_json_object_from_request(request)
 
-        if self.hs.config.user_directory_defer_to_id_server:
-            signed_body = sign_json(body, self.hs.hostname, self.hs.config.signing_key[0])
-            url = "%s/_matrix/identity/api/v1/user_directory/search" % (
-                self.hs.config.user_directory_defer_to_id_server,
-            )
-            resp = yield self.http_client.post_json_get_json(url, signed_body)
-            defer.returnValue((200, resp))
-
         limit = body.get("limit", 10)
         limit = min(limit, 50)
 

synapse/rest/media/v1/media_repository.py

@@ -444,6 +444,9 @@ class MediaRepository(object):
             )
             return
 
+        if thumbnailer.transpose_method is not None:
+            m_width, m_height = thumbnailer.transpose()
+
         if t_method == "crop":
             t_byte_source = thumbnailer.crop(t_width, t_height, t_type)
         elif t_method == "scale":
@@ -578,6 +581,12 @@ class MediaRepository(object):
             )
             return
 
+        if thumbnailer.transpose_method is not None:
+            m_width, m_height = yield logcontext.defer_to_thread(
+                self.hs.get_reactor(),
+                thumbnailer.transpose
+            )
+
         # We deduplicate the thumbnail sizes by ignoring the cropped versions if
         # they have the same dimensions of a scaled one.
         thumbnails = {}

synapse/rest/media/v1/thumbnailer.py

@@ -20,6 +20,17 @@ import PIL.Image as Image
 
 logger = logging.getLogger(__name__)
 
+EXIF_ORIENTATION_TAG = 0x0112
+EXIF_TRANSPOSE_MAPPINGS = {
+    2: Image.FLIP_LEFT_RIGHT,
+    3: Image.ROTATE_180,
+    4: Image.FLIP_TOP_BOTTOM,
+    5: Image.TRANSPOSE,
+    6: Image.ROTATE_270,
+    7: Image.TRANSVERSE,
+    8: Image.ROTATE_90
+}
+
 
 class Thumbnailer(object):
 
@@ -31,6 +42,30 @@ class Thumbnailer(object):
     def __init__(self, input_path):
         self.image = Image.open(input_path)
         self.width, self.height = self.image.size
+        self.transpose_method = None
+        try:
+            # We don't use ImageOps.exif_transpose since it crashes with big EXIF
+            image_exif = self.image._getexif()
+            if image_exif is not None:
+                image_orientation = image_exif.get(EXIF_ORIENTATION_TAG)
+                self.transpose_method = EXIF_TRANSPOSE_MAPPINGS.get(image_orientation)
+        except Exception as e:
+            # A lot of parsing errors can happen when parsing EXIF
+            logger.info("Error parsing image EXIF information: %s", e)
+
+    def transpose(self):
+        """Transpose the image using its EXIF Orientation tag
+
+        Returns:
+            Tuple[int, int]: (width, height) containing the new image size in pixels.
+        """
+        if self.transpose_method is not None:
+            self.image = self.image.transpose(self.transpose_method)
+            self.width, self.height = self.image.size
+            self.transpose_method = None
+            # We don't need EXIF any more
+            self.image.info["exif"] = None
+        return self.image.size
 
     def aspect(self, max_width, max_height):
         """Calculate the largest size that preserves aspect ratio which

synapse/rulecheck/domain_rule_checker.py

@@ -1,172 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2018 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import logging
-
-from synapse.config._base import ConfigError
-
-logger = logging.getLogger(__name__)
-
-
-class DomainRuleChecker(object):
-    """
-    A re-implementation of the SpamChecker that prevents users in one domain from
-    inviting users in other domains to rooms, based on a configuration.
-
-    Takes a config in the format:
-
-    spam_checker:
-        module: "rulecheck.DomainRuleChecker"
-        config:
-          domain_mapping:
-            "inviter_domain": [ "invitee_domain_permitted", "other_domain_permitted" ]
-            "other_inviter_domain": [ "invitee_domain_permitted" ]
-          default: False
-
-          # Only let local users join rooms if they were explicitly invited.
-          can_only_join_rooms_with_invite: false
-
-          # Only let local users create rooms if they are inviting only one
-          # other user, and that user matches the rules above.
-          can_only_create_one_to_one_rooms: false
-
-          # Only let local users invite during room creation, regardless of the
-          # domain mapping rules above.
-          can_only_invite_during_room_creation: false
-
-          # Prevent local users from inviting users from certain domains to
-          # rooms published in the room directory.
-          domains_prevented_from_being_invited_to_published_rooms: []
-
-          # Allow third party invites
-          can_invite_by_third_party_id: true
-
-    Don't forget to consider if you can invite users from your own domain.
-    """
-
-    def __init__(self, config):
-        self.domain_mapping = config["domain_mapping"] or {}
-        self.default = config["default"]
-
-        self.can_only_join_rooms_with_invite = config.get(
-            "can_only_join_rooms_with_invite", False,
-        )
-        self.can_only_create_one_to_one_rooms = config.get(
-            "can_only_create_one_to_one_rooms", False,
-        )
-        self.can_only_invite_during_room_creation = config.get(
-            "can_only_invite_during_room_creation", False,
-        )
-        self.can_invite_by_third_party_id = config.get(
-            "can_invite_by_third_party_id", True,
-        )
-        self.domains_prevented_from_being_invited_to_published_rooms = config.get(
-            "domains_prevented_from_being_invited_to_published_rooms", [],
-        )
-
-    def check_event_for_spam(self, event):
-        """Implements synapse.events.SpamChecker.check_event_for_spam
-        """
-        return False
-
-    def user_may_invite(self, inviter_userid, invitee_userid, third_party_invite,
-                        room_id, new_room, published_room=False):
-        """Implements synapse.events.SpamChecker.user_may_invite
-        """
-        if self.can_only_invite_during_room_creation and not new_room:
-            return False
-
-        if not self.can_invite_by_third_party_id and third_party_invite:
-            return False
-
-        # This is a third party invite (without a bound mxid), so unless we have
-        # banned all third party invites (above) we allow it.
-        if not invitee_userid:
-            return True
-
-        inviter_domain = self._get_domain_from_id(inviter_userid)
-        invitee_domain = self._get_domain_from_id(invitee_userid)
-
-        if inviter_domain not in self.domain_mapping:
-            return self.default
-
-        if (
-            published_room and
-            invitee_domain in self.domains_prevented_from_being_invited_to_published_rooms
-        ):
-            return False
-
-        return invitee_domain in self.domain_mapping[inviter_domain]
-
-    def user_may_create_room(self, userid, invite_list, third_party_invite_list,
-                             cloning):
-        """Implements synapse.events.SpamChecker.user_may_create_room
-        """
-
-        if cloning:
-            return True
-
-        if not self.can_invite_by_third_party_id and third_party_invite_list:
-            return False
-
-        number_of_invites = len(invite_list) + len(third_party_invite_list)
-
-        if self.can_only_create_one_to_one_rooms and number_of_invites != 1:
-            return False
-
-        return True
-
-    def user_may_create_room_alias(self, userid, room_alias):
-        """Implements synapse.events.SpamChecker.user_may_create_room_alias
-        """
-        return True
-
-    def user_may_publish_room(self, userid, room_id):
-        """Implements synapse.events.SpamChecker.user_may_publish_room
-        """
-        return True
-
-    def user_may_join_room(self, userid, room_id, is_invited):
-        """Implements synapse.events.SpamChecker.user_may_join_room
-        """
-        if self.can_only_join_rooms_with_invite and not is_invited:
-            return False
-
-        return True
-
-    @staticmethod
-    def parse_config(config):
-        """Implements synapse.events.SpamChecker.parse_config
-        """
-        if "default" in config:
-            return config
-        else:
-            raise ConfigError("No default set for spam_config DomainRuleChecker")
-
-    @staticmethod
-    def _get_domain_from_id(mxid):
-        """Parses a string and returns the domain part of the mxid.
-
-        Args:
-           mxid (str): a valid mxid
-
-        Returns:
-           str: the domain part of the mxid
-
-        """
-        idx = mxid.find(":")
-        if idx == -1:
-            raise Exception("Invalid ID: %r" % (mxid,))
-        return mxid[idx + 1:]

synapse/server.py

@@ -1,7 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2017-2018 New Vector Ltd
-# Copyright 2019 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -64,7 +62,6 @@ from synapse.handlers.groups_local import GroupsLocalHandler
 from synapse.handlers.initial_sync import InitialSyncHandler
 from synapse.handlers.message import EventCreationHandler, MessageHandler
 from synapse.handlers.pagination import PaginationHandler
-from synapse.handlers.password_policy import PasswordPolicyHandler
 from synapse.handlers.presence import PresenceHandler
 from synapse.handlers.profile import BaseProfileHandler, MasterProfileHandler
 from synapse.handlers.read_marker import ReadMarkerHandler
@@ -75,6 +72,7 @@ from synapse.handlers.room_list import RoomListHandler
 from synapse.handlers.room_member import RoomMemberMasterHandler
 from synapse.handlers.room_member_worker import RoomMemberWorkerHandler
 from synapse.handlers.set_password import SetPasswordHandler
+from synapse.handlers.stats import StatsHandler
 from synapse.handlers.sync import SyncHandler
 from synapse.handlers.typing import TypingHandler
 from synapse.handlers.user_directory import UserDirectoryHandler
@@ -142,6 +140,7 @@ class HomeServer(object):
         'acme_handler',
         'auth_handler',
         'device_handler',
+        'stats_handler',
         'e2e_keys_handler',
         'e2e_room_keys_handler',
         'event_handler',
@@ -190,11 +189,11 @@ class HomeServer(object):
         'registration_handler',
         'account_validity_handler',
         'event_client_serializer',
-        'password_policy_handler',
     ]
 
     REQUIRED_ON_MASTER_STARTUP = [
         "user_directory_handler",
+        "stats_handler"
     ]
 
     # This is overridden in derived application classes
@@ -478,6 +477,9 @@ class HomeServer(object):
     def build_secrets(self):
         return Secrets()
 
+    def build_stats_handler(self):
+        return StatsHandler(self)
+
     def build_spam_checker(self):
         return SpamChecker(self)
 
@@ -520,9 +522,6 @@ class HomeServer(object):
     def build_event_client_serializer(self):
         return EventClientSerializer(self)
 
-    def build_password_policy_handler(self):
-        return PasswordPolicyHandler(self)
-
     def remove_pusher(self, app_id, push_key, user_id):
         return self.get_pusherpool().remove_pusher(app_id, push_key, user_id)
 

synapse/storage/__init__.py

@@ -49,11 +49,13 @@ from .pusher import PusherStore
 from .receipts import ReceiptsStore
 from .registration import RegistrationStore
 from .rejections import RejectionsStore
+from .relations import RelationsStore
 from .room import RoomStore
 from .roommember import RoomMemberStore
 from .search import SearchStore
 from .signatures import SignatureStore
 from .state import StateStore
+from .stats import StatsStore
 from .stream import StreamStore
 from .tags import TagsStore
 from .transactions import TransactionStore
@@ -99,6 +101,8 @@ class DataStore(
     GroupServerStore,
     UserErasureStore,
     MonthlyActiveUsersStore,
+    StatsStore,
+    RelationsStore,
 ):
     def __init__(self, db_conn, hs):
         self.hs = hs

synapse/storage/_base.py

@@ -16,7 +16,6 @@
 # limitations under the License.
 import itertools
 import logging
-import random
 import sys
 import threading
 import time
@@ -248,8 +247,6 @@ class SQLBaseStore(object):
                 self._check_safe_to_upsert,
             )
 
-        self.rand = random.SystemRandom()
-
         if self._account_validity.enabled:
             self._clock.call_later(
                 0.0,
@@ -311,36 +308,21 @@ class SQLBaseStore(object):
             res = self.cursor_to_dict(txn)
             if res:
                 for user in res:
-                    self.set_expiration_date_for_user_txn(
-                        txn,
-                        user["name"],
-                        use_delta=True,
-                    )
+                    self.set_expiration_date_for_user_txn(txn, user["name"])
 
         yield self.runInteraction(
             "get_users_with_no_expiration_date",
             select_users_with_no_expiration_date_txn,
         )
 
-    def set_expiration_date_for_user_txn(self, txn, user_id, use_delta=False):
+    def set_expiration_date_for_user_txn(self, txn, user_id):
         """Sets an expiration date to the account with the given user ID.
 
         Args:
              user_id (str): User ID to set an expiration date for.
-             use_delta (bool): If set to False, the expiration date for the user will be
-                now + validity period. If set to True, this expiration date will be a
-                random value in the [now + period - d ; now + period] range, d being a
-                delta equal to 10% of the validity period.
         """
         now_ms = self._clock.time_msec()
         expiration_ts = now_ms + self._account_validity.period
-
-        if use_delta:
-            expiration_ts = self.rand.randrange(
-                expiration_ts - self._account_validity.startup_job_max_delta,
-                expiration_ts,
-            )
-
         self._simple_insert_txn(
             txn,
             "account_validity",

synapse/storage/appservice.py

@@ -35,7 +35,7 @@ def _make_exclusive_regex(services_cache):
     exclusive_user_regexes = [
         regex.pattern
         for service in services_cache
-        for regex in service.get_exclusive_user_regexes()
+        for regex in service.get_exlusive_user_regexes()
     ]
     if exclusive_user_regexes:
         exclusive_user_regex = "|".join("(" + r + ")" for r in exclusive_user_regexes)

synapse/storage/events.py

@@ -575,10 +575,11 @@ class EventsStore(
 
         def _get_events(txn, batch):
             sql = """
-            SELECT prev_event_id
+            SELECT prev_event_id, internal_metadata
             FROM event_edges
                 INNER JOIN events USING (event_id)
                 LEFT JOIN rejections USING (event_id)
+                LEFT JOIN event_json USING (event_id)
             WHERE
                 prev_event_id IN (%s)
                 AND NOT events.outlier
@@ -588,7 +589,11 @@ class EventsStore(
             )
 
             txn.execute(sql, batch)
-            results.extend(r[0] for r in txn)
+            results.extend(
+                r[0]
+                for r in txn
+                if not json.loads(r[1]).get("soft_failed")
+            )
 
         for chunk in batch_iter(event_ids, 100):
             yield self.runInteraction("_get_events_which_are_prevs", _get_events, chunk)
@@ -1325,6 +1330,9 @@ class EventsStore(
                     txn, event.room_id, event.redacts
                 )
 
+                # Remove from relations table.
+                self._handle_redaction(txn, event.redacts)
+
         # Update the event_forward_extremities, event_backward_extremities and
         # event_edges tables.
         self._handle_mult_prev_events(
@@ -1351,6 +1359,8 @@ class EventsStore(
                 # Insert into the event_search table.
                 self._store_guest_access_txn(txn, event)
 
+            self._handle_event_relations(txn, event)
+
         # Insert into the room_memberships table.
         self._store_room_members_txn(
             txn,
@@ -1655,10 +1665,11 @@ class EventsStore(
         def get_all_new_forward_event_rows(txn):
             sql = (
                 "SELECT e.stream_ordering, e.event_id, e.room_id, e.type,"
-                " state_key, redacts"
+                " state_key, redacts, relates_to_id"
                 " FROM events AS e"
                 " LEFT JOIN redactions USING (event_id)"
                 " LEFT JOIN state_events USING (event_id)"
+                " LEFT JOIN event_relations USING (event_id)"
                 " WHERE ? < stream_ordering AND stream_ordering <= ?"
                 " ORDER BY stream_ordering ASC"
                 " LIMIT ?"
@@ -1673,11 +1684,12 @@ class EventsStore(
 
             sql = (
                 "SELECT event_stream_ordering, e.event_id, e.room_id, e.type,"
-                " state_key, redacts"
+                " state_key, redacts, relates_to_id"
                 " FROM events AS e"
                 " INNER JOIN ex_outlier_stream USING (event_id)"
                 " LEFT JOIN redactions USING (event_id)"
                 " LEFT JOIN state_events USING (event_id)"
+                " LEFT JOIN event_relations USING (event_id)"
                 " WHERE ? < event_stream_ordering"
                 " AND event_stream_ordering <= ?"
                 " ORDER BY event_stream_ordering DESC"
@@ -1698,10 +1710,11 @@ class EventsStore(
         def get_all_new_backfill_event_rows(txn):
             sql = (
                 "SELECT -e.stream_ordering, e.event_id, e.room_id, e.type,"
-                " state_key, redacts"
+                " state_key, redacts, relates_to_id"
                 " FROM events AS e"
                 " LEFT JOIN redactions USING (event_id)"
                 " LEFT JOIN state_events USING (event_id)"
+                " LEFT JOIN event_relations USING (event_id)"
                 " WHERE ? > stream_ordering AND stream_ordering >= ?"
                 " ORDER BY stream_ordering ASC"
                 " LIMIT ?"
@@ -1716,11 +1729,12 @@ class EventsStore(
 
             sql = (
                 "SELECT -event_stream_ordering, e.event_id, e.room_id, e.type,"
-                " state_key, redacts"
+                " state_key, redacts, relates_to_id"
                 " FROM events AS e"
                 " INNER JOIN ex_outlier_stream USING (event_id)"
                 " LEFT JOIN redactions USING (event_id)"
                 " LEFT JOIN state_events USING (event_id)"
+                " LEFT JOIN event_relations USING (event_id)"
                 " WHERE ? > event_stream_ordering"
                 " AND event_stream_ordering >= ?"
                 " ORDER BY event_stream_ordering DESC"

synapse/storage/events_worker.py

@@ -611,3 +611,27 @@ class EventsWorkerStore(SQLBaseStore):
             return res
 
         return self.runInteraction("get_rejection_reasons", f)
+
+    def _get_total_state_event_counts_txn(self, txn, room_id):
+        """
+        See get_state_event_counts.
+        """
+        sql = "SELECT COUNT(*) FROM state_events WHERE room_id=?"
+        txn.execute(sql, (room_id,))
+        row = txn.fetchone()
+        return row[0] if row else 0
+
+    def get_total_state_event_counts(self, room_id):
+        """
+        Gets the total number of state events in a room.
+
+        Args:
+            room_id (str)
+
+        Returns:
+            Deferred[int]
+        """
+        return self.runInteraction(
+            "get_total_state_event_counts",
+            self._get_total_state_event_counts_txn, room_id
+        )

synapse/storage/profile.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2014-2016 OpenMarket Ltd
-# Copyright 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -21,8 +20,6 @@ from synapse.storage.roommember import ProfileInfo
 
 from ._base import SQLBaseStore
 
-BATCH_SIZE = 100
-
 
 class ProfileWorkerStore(SQLBaseStore):
     @defer.inlineCallbacks
@@ -64,55 +61,6 @@ class ProfileWorkerStore(SQLBaseStore):
             desc="get_profile_avatar_url",
         )
 
-    def get_latest_profile_replication_batch_number(self):
-        def f(txn):
-            txn.execute("SELECT MAX(batch) as maxbatch FROM profiles")
-            rows = self.cursor_to_dict(txn)
-            return rows[0]['maxbatch']
-        return self.runInteraction(
-            "get_latest_profile_replication_batch_number", f,
-        )
-
-    def get_profile_batch(self, batchnum):
-        return self._simple_select_list(
-            table="profiles",
-            keyvalues={
-                "batch": batchnum,
-            },
-            retcols=("user_id", "displayname", "avatar_url", "active"),
-            desc="get_profile_batch",
-        )
-
-    def assign_profile_batch(self):
-        def f(txn):
-            sql = (
-                "UPDATE profiles SET batch = "
-                "(SELECT COALESCE(MAX(batch), -1) + 1 FROM profiles) "
-                "WHERE user_id in ("
-                "    SELECT user_id FROM profiles WHERE batch is NULL limit ?"
-                ")"
-            )
-            txn.execute(sql, (BATCH_SIZE,))
-            return txn.rowcount
-        return self.runInteraction("assign_profile_batch", f)
-
-    def get_replication_hosts(self):
-        def f(txn):
-            txn.execute("SELECT host, last_synced_batch FROM profile_replication_status")
-            rows = self.cursor_to_dict(txn)
-            return {r['host']: r['last_synced_batch'] for r in rows}
-        return self.runInteraction("get_replication_hosts", f)
-
-    def update_replication_batch_for_host(self, host, last_synced_batch):
-        return self._simple_upsert(
-            table="profile_replication_status",
-            keyvalues={"host": host},
-            values={
-                "last_synced_batch": last_synced_batch,
-            },
-            desc="update_replication_batch_for_host",
-        )
-
     def get_from_remote_profile_cache(self, user_id):
         return self._simple_select_one(
             table="remote_profile_cache",
@@ -122,46 +70,25 @@ class ProfileWorkerStore(SQLBaseStore):
             desc="get_from_remote_profile_cache",
         )
 
-    def set_profile_displayname(self, user_localpart, new_displayname, batchnum):
-        return self._simple_upsert(
+    def create_profile(self, user_localpart):
+        return self._simple_insert(
+            table="profiles", values={"user_id": user_localpart}, desc="create_profile"
+        )
+
+    def set_profile_displayname(self, user_localpart, new_displayname):
+        return self._simple_update_one(
             table="profiles",
             keyvalues={"user_id": user_localpart},
-            values={
-                "displayname": new_displayname,
-                "batch": batchnum,
-            },
+            updatevalues={"displayname": new_displayname},
             desc="set_profile_displayname",
-            lock=False  # we can do this because user_id has a unique index
         )
 
-    def set_profile_avatar_url(self, user_localpart, new_avatar_url, batchnum):
-        return self._simple_upsert(
+    def set_profile_avatar_url(self, user_localpart, new_avatar_url):
+        return self._simple_update_one(
             table="profiles",
             keyvalues={"user_id": user_localpart},
-            values={
-                "avatar_url": new_avatar_url,
-                "batch": batchnum,
-            },
+            updatevalues={"avatar_url": new_avatar_url},
             desc="set_profile_avatar_url",
-            lock=False  # we can do this because user_id has a unique index
-        )
-
-    def set_profile_active(self, user_localpart, active, hide, batchnum):
-        values = {
-            "active": int(active),
-            "batch": batchnum,
-        }
-        if not active and not hide:
-            # we are deactivating for real (not in hide mode)
-            # so clear the profile.
-            values["avatar_url"] = None
-            values["displayname"] = None
-        return self._simple_upsert(
-            table="profiles",
-            keyvalues={"user_id": user_localpart},
-            values=values,
-            desc="set_profile_active",
-            lock=False  # we can do this because user_id has a unique index
         )
 
 

synapse/storage/relations.py

@@ -0,0 +1,476 @@
+# -*- coding: utf-8 -*-
+# Copyright 2019 New Vector Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+
+import attr
+
+from twisted.internet import defer
+
+from synapse.api.constants import RelationTypes
+from synapse.api.errors import SynapseError
+from synapse.storage._base import SQLBaseStore
+from synapse.storage.stream import generate_pagination_where_clause
+from synapse.util.caches.descriptors import cached, cachedInlineCallbacks
+
+logger = logging.getLogger(__name__)
+
+
+@attr.s
+class PaginationChunk(object):
+    """Returned by relation pagination APIs.
+
+    Attributes:
+        chunk (list): The rows returned by pagination
+        next_batch (Any|None): Token to fetch next set of results with, if
+            None then there are no more results.
+        prev_batch (Any|None): Token to fetch previous set of results with, if
+            None then there are no previous results.
+    """
+
+    chunk = attr.ib()
+    next_batch = attr.ib(default=None)
+    prev_batch = attr.ib(default=None)
+
+    def to_dict(self):
+        d = {"chunk": self.chunk}
+
+        if self.next_batch:
+            d["next_batch"] = self.next_batch.to_string()
+
+        if self.prev_batch:
+            d["prev_batch"] = self.prev_batch.to_string()
+
+        return d
+
+
+@attr.s(frozen=True, slots=True)
+class RelationPaginationToken(object):
+    """Pagination token for relation pagination API.
+
+    As the results are order by topological ordering, we can use the
+    `topological_ordering` and `stream_ordering` fields of the events at the
+    boundaries of the chunk as pagination tokens.
+
+    Attributes:
+        topological (int): The topological ordering of the boundary event
+        stream (int): The stream ordering of the boundary event.
+    """
+
+    topological = attr.ib()
+    stream = attr.ib()
+
+    @staticmethod
+    def from_string(string):
+        try:
+            t, s = string.split("-")
+            return RelationPaginationToken(int(t), int(s))
+        except ValueError:
+            raise SynapseError(400, "Invalid token")
+
+    def to_string(self):
+        return "%d-%d" % (self.topological, self.stream)
+
+    def as_tuple(self):
+        return attr.astuple(self)
+
+
+@attr.s(frozen=True, slots=True)
+class AggregationPaginationToken(object):
+    """Pagination token for relation aggregation pagination API.
+
+    As the results are order by count and then MAX(stream_ordering) of the
+    aggregation groups, we can just use them as our pagination token.
+
+    Attributes:
+        count (int): The count of relations in the boundar group.
+        stream (int): The MAX stream ordering in the boundary group.
+    """
+
+    count = attr.ib()
+    stream = attr.ib()
+
+    @staticmethod
+    def from_string(string):
+        try:
+            c, s = string.split("-")
+            return AggregationPaginationToken(int(c), int(s))
+        except ValueError:
+            raise SynapseError(400, "Invalid token")
+
+    def to_string(self):
+        return "%d-%d" % (self.count, self.stream)
+
+    def as_tuple(self):
+        return attr.astuple(self)
+
+
+class RelationsWorkerStore(SQLBaseStore):
+    @cached(tree=True)
+    def get_relations_for_event(
+        self,
+        event_id,
+        relation_type=None,
+        event_type=None,
+        aggregation_key=None,
+        limit=5,
+        direction="b",
+        from_token=None,
+        to_token=None,
+    ):
+        """Get a list of relations for an event, ordered by topological ordering.
+
+        Args:
+            event_id (str): Fetch events that relate to this event ID.
+            relation_type (str|None): Only fetch events with this relation
+                type, if given.
+            event_type (str|None): Only fetch events with this event type, if
+                given.
+            aggregation_key (str|None): Only fetch events with this aggregation
+                key, if given.
+            limit (int): Only fetch the most recent `limit` events.
+            direction (str): Whether to fetch the most recent first (`"b"`) or
+                the oldest first (`"f"`).
+            from_token (RelationPaginationToken|None): Fetch rows from the given
+                token, or from the start if None.
+            to_token (RelationPaginationToken|None): Fetch rows up to the given
+                token, or up to the end if None.
+
+        Returns:
+            Deferred[PaginationChunk]: List of event IDs that match relations
+            requested. The rows are of the form `{"event_id": "..."}`.
+        """
+
+        where_clause = ["relates_to_id = ?"]
+        where_args = [event_id]
+
+        if relation_type is not None:
+            where_clause.append("relation_type = ?")
+            where_args.append(relation_type)
+
+        if event_type is not None:
+            where_clause.append("type = ?")
+            where_args.append(event_type)
+
+        if aggregation_key:
+            where_clause.append("aggregation_key = ?")
+            where_args.append(aggregation_key)
+
+        pagination_clause = generate_pagination_where_clause(
+            direction=direction,
+            column_names=("topological_ordering", "stream_ordering"),
+            from_token=attr.astuple(from_token) if from_token else None,
+            to_token=attr.astuple(to_token) if to_token else None,
+            engine=self.database_engine,
+        )
+
+        if pagination_clause:
+            where_clause.append(pagination_clause)
+
+        if direction == "b":
+            order = "DESC"
+        else:
+            order = "ASC"
+
+        sql = """
+            SELECT event_id, topological_ordering, stream_ordering
+            FROM event_relations
+            INNER JOIN events USING (event_id)
+            WHERE %s
+            ORDER BY topological_ordering %s, stream_ordering %s
+            LIMIT ?
+        """ % (
+            " AND ".join(where_clause),
+            order,
+            order,
+        )
+
+        def _get_recent_references_for_event_txn(txn):
+            txn.execute(sql, where_args + [limit + 1])
+
+            last_topo_id = None
+            last_stream_id = None
+            events = []
+            for row in txn:
+                events.append({"event_id": row[0]})
+                last_topo_id = row[1]
+                last_stream_id = row[2]
+
+            next_batch = None
+            if len(events) > limit and last_topo_id and last_stream_id:
+                next_batch = RelationPaginationToken(last_topo_id, last_stream_id)
+
+            return PaginationChunk(
+                chunk=list(events[:limit]), next_batch=next_batch, prev_batch=from_token
+            )
+
+        return self.runInteraction(
+            "get_recent_references_for_event", _get_recent_references_for_event_txn
+        )
+
+    @cached(tree=True)
+    def get_aggregation_groups_for_event(
+        self,
+        event_id,
+        event_type=None,
+        limit=5,
+        direction="b",
+        from_token=None,
+        to_token=None,
+    ):
+        """Get a list of annotations on the event, grouped by event type and
+        aggregation key, sorted by count.
+
+        This is used e.g. to get the what and how many reactions have happend
+        on an event.
+
+        Args:
+            event_id (str): Fetch events that relate to this event ID.
+            event_type (str|None): Only fetch events with this event type, if
+                given.
+            limit (int): Only fetch the `limit` groups.
+            direction (str): Whether to fetch the highest count first (`"b"`) or
+                the lowest count first (`"f"`).
+            from_token (AggregationPaginationToken|None): Fetch rows from the
+                given token, or from the start if None.
+            to_token (AggregationPaginationToken|None): Fetch rows up to the
+                given token, or up to the end if None.
+
+
+        Returns:
+            Deferred[PaginationChunk]: List of groups of annotations that
+            match. Each row is a dict with `type`, `key` and `count` fields.
+        """
+
+        where_clause = ["relates_to_id = ?", "relation_type = ?"]
+        where_args = [event_id, RelationTypes.ANNOTATION]
+
+        if event_type:
+            where_clause.append("type = ?")
+            where_args.append(event_type)
+
+        having_clause = generate_pagination_where_clause(
+            direction=direction,
+            column_names=("COUNT(*)", "MAX(stream_ordering)"),
+            from_token=attr.astuple(from_token) if from_token else None,
+            to_token=attr.astuple(to_token) if to_token else None,
+            engine=self.database_engine,
+        )
+
+        if direction == "b":
+            order = "DESC"
+        else:
+            order = "ASC"
+
+        if having_clause:
+            having_clause = "HAVING " + having_clause
+        else:
+            having_clause = ""
+
+        sql = """
+            SELECT type, aggregation_key, COUNT(DISTINCT sender), MAX(stream_ordering)
+            FROM event_relations
+            INNER JOIN events USING (event_id)
+            WHERE {where_clause}
+            GROUP BY relation_type, type, aggregation_key
+            {having_clause}
+            ORDER BY COUNT(*) {order}, MAX(stream_ordering) {order}
+            LIMIT ?
+        """.format(
+            where_clause=" AND ".join(where_clause),
+            order=order,
+            having_clause=having_clause,
+        )
+
+        def _get_aggregation_groups_for_event_txn(txn):
+            txn.execute(sql, where_args + [limit + 1])
+
+            next_batch = None
+            events = []
+            for row in txn:
+                events.append({"type": row[0], "key": row[1], "count": row[2]})
+                next_batch = AggregationPaginationToken(row[2], row[3])
+
+            if len(events) <= limit:
+                next_batch = None
+
+            return PaginationChunk(
+                chunk=list(events[:limit]), next_batch=next_batch, prev_batch=from_token
+            )
+
+        return self.runInteraction(
+            "get_aggregation_groups_for_event", _get_aggregation_groups_for_event_txn
+        )
+
+    @cachedInlineCallbacks()
+    def get_applicable_edit(self, event_id):
+        """Get the most recent edit (if any) that has happened for the given
+        event.
+
+        Correctly handles checking whether edits were allowed to happen.
+
+        Args:
+            event_id (str): The original event ID
+
+        Returns:
+            Deferred[EventBase|None]: Returns the most recent edit, if any.
+        """
+
+        # We only allow edits for `m.room.message` events that have the same sender
+        # and event type. We can't assert these things during regular event auth so
+        # we have to do the checks post hoc.
+
+        # Fetches latest edit that has the same type and sender as the
+        # original, and is an `m.room.message`.
+        sql = """
+            SELECT edit.event_id FROM events AS edit
+            INNER JOIN event_relations USING (event_id)
+            INNER JOIN events AS original ON
+                original.event_id = relates_to_id
+                AND edit.type = original.type
+                AND edit.sender = original.sender
+            WHERE
+                relates_to_id = ?
+                AND relation_type = ?
+                AND edit.type = 'm.room.message'
+            ORDER by edit.origin_server_ts DESC, edit.event_id DESC
+            LIMIT 1
+        """
+
+        def _get_applicable_edit_txn(txn):
+            txn.execute(sql, (event_id, RelationTypes.REPLACE))
+            row = txn.fetchone()
+            if row:
+                return row[0]
+
+        edit_id = yield self.runInteraction(
+            "get_applicable_edit", _get_applicable_edit_txn
+        )
+
+        if not edit_id:
+            return
+
+        edit_event = yield self.get_event(edit_id, allow_none=True)
+        defer.returnValue(edit_event)
+
+    def has_user_annotated_event(self, parent_id, event_type, aggregation_key, sender):
+        """Check if a user has already annotated an event with the same key
+        (e.g. already liked an event).
+
+        Args:
+            parent_id (str): The event being annotated
+            event_type (str): The event type of the annotation
+            aggregation_key (str): The aggregation key of the annotation
+            sender (str): The sender of the annotation
+
+        Returns:
+            Deferred[bool]
+        """
+
+        sql = """
+            SELECT 1 FROM event_relations
+            INNER JOIN events USING (event_id)
+            WHERE
+                relates_to_id = ?
+                AND relation_type = ?
+                AND type = ?
+                AND sender = ?
+                AND aggregation_key = ?
+            LIMIT 1;
+        """
+
+        def _get_if_user_has_annotated_event(txn):
+            txn.execute(
+                sql,
+                (
+                    parent_id,
+                    RelationTypes.ANNOTATION,
+                    event_type,
+                    sender,
+                    aggregation_key,
+                ),
+            )
+
+            return bool(txn.fetchone())
+
+        return self.runInteraction(
+            "get_if_user_has_annotated_event", _get_if_user_has_annotated_event
+        )
+
+
+class RelationsStore(RelationsWorkerStore):
+    def _handle_event_relations(self, txn, event):
+        """Handles inserting relation data during peristence of events
+
+        Args:
+            txn
+            event (EventBase)
+        """
+        relation = event.content.get("m.relates_to")
+        if not relation:
+            # No relations
+            return
+
+        rel_type = relation.get("rel_type")
+        if rel_type not in (
+            RelationTypes.ANNOTATION,
+            RelationTypes.REFERENCE,
+            RelationTypes.REPLACE,
+        ):
+            # Unknown relation type
+            return
+
+        parent_id = relation.get("event_id")
+        if not parent_id:
+            # Invalid relation
+            return
+
+        aggregation_key = relation.get("key")
+
+        self._simple_insert_txn(
+            txn,
+            table="event_relations",
+            values={
+                "event_id": event.event_id,
+                "relates_to_id": parent_id,
+                "relation_type": rel_type,
+                "aggregation_key": aggregation_key,
+            },
+        )
+
+        txn.call_after(self.get_relations_for_event.invalidate_many, (parent_id,))
+        txn.call_after(
+            self.get_aggregation_groups_for_event.invalidate_many, (parent_id,)
+        )
+
+        if rel_type == RelationTypes.REPLACE:
+            txn.call_after(self.get_applicable_edit.invalidate, (parent_id,))
+
+    def _handle_redaction(self, txn, redacted_event_id):
+        """Handles receiving a redaction and checking whether we need to remove
+        any redacted relations from the database.
+
+        Args:
+            txn
+            redacted_event_id (str): The event that was redacted.
+        """
+
+        self._simple_delete_txn(
+            txn,
+            table="event_relations",
+            keyvalues={
+                "event_id": redacted_event_id,
+            }
+        )

synapse/storage/room.py

@@ -171,24 +171,6 @@ class RoomWorkerStore(SQLBaseStore):
             desc="is_room_blocked",
         )
 
-    @defer.inlineCallbacks
-    def is_room_published(self, room_id):
-        """Check whether a room has been published in the local public room
-        directory.
-
-        Args:
-            room_id (str)
-        Returns:
-            bool: Whether the room is currently published in the room directory
-        """
-        # Get room information
-        room_info = yield self.get_room(room_id)
-        if not room_info:
-            defer.returnValue(False)
-
-        # Check the is_public value
-        defer.returnValue(room_info.get("is_public", False))
-
     @cachedInlineCallbacks(max_entries=10000)
     def get_ratelimit_for_user(self, user_id):
         """Check if there are any overrides for ratelimiting for the given

synapse/storage/roommember.py

@@ -142,6 +142,38 @@ class RoomMemberWorkerStore(EventsWorkerStore):
 
         return self.runInteraction("get_room_summary", _get_room_summary_txn)
 
+    def _get_user_count_in_room_txn(self, txn, room_id, membership):
+        """
+        See get_user_count_in_room.
+        """
+        sql = (
+            "SELECT count(*) FROM room_memberships as m"
+            " INNER JOIN current_state_events as c"
+            " ON m.event_id = c.event_id "
+            " AND m.room_id = c.room_id "
+            " AND m.user_id = c.state_key"
+            " WHERE c.type = 'm.room.member' AND c.room_id = ? AND m.membership = ?"
+        )
+
+        txn.execute(sql, (room_id, membership))
+        row = txn.fetchone()
+        return row[0]
+
+    def get_user_count_in_room(self, room_id, membership):
+        """
+        Get the user count in a room with a particular membership.
+
+        Args:
+            room_id (str)
+            membership (Membership)
+
+        Returns:
+            Deferred[int]
+        """
+        return self.runInteraction(
+            "get_users_in_room", self._get_user_count_in_room_txn, room_id, membership
+        )
+
     @cached()
     def get_invited_rooms_for_user(self, user_id):
         """ Get all the rooms the user is invited to

synapse/storage/schema/delta/48/profiles_batch.sql

@@ -1,36 +0,0 @@
-/* Copyright 2018 New Vector Ltd
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *    http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-/*
- * Add a batch number to track changes to profiles and the
- * order they're made in so we can replicate user profiles
- * to other hosts as they change
- */
-ALTER TABLE profiles ADD COLUMN batch BIGINT DEFAULT NULL;
-
-/*
- * Index on the batch number so we can get profiles
- * by their batch
- */
-CREATE INDEX profiles_batch_idx ON profiles(batch);
-
-/*
- * A table to track what batch of user profiles has been
- * synced to what profile replication target.
- */
-CREATE TABLE profile_replication_status (
-    host TEXT NOT NULL,
-    last_synced_batch BIGINT NOT NULL
-);