Switch to using a delta file

.buildkite/merge_base_branch.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 
-set -e
+set -ex
 
 if [[ "$BUILDKITE_BRANCH" =~ ^(develop|master|dinsic|shhs|release-.*)$ ]]; then
     echo "Not merging forward, as this is a release branch"
@@ -18,8 +18,6 @@ else
     GITBASE=$BUILDKITE_PULL_REQUEST_BASE_BRANCH
 fi
 
-echo "--- merge_base_branch $GITBASE"
-
 # Show what we are before
 git --no-pager show -s
 

.buildkite/worker-blacklist

@@ -28,32 +28,3 @@ User sees updates to presence from other users in the incremental sync.
 Gapped incremental syncs include all state changes
 
 Old members are included in gappy incr LL sync if they start speaking
-
-# new failures as of https://github.com/matrix-org/sytest/pull/732
-Device list doesn't change if remote server is down
-Remote servers cannot set power levels in rooms without existing powerlevels
-Remote servers should reject attempts by non-creators to set the power levels
-
-# new failures as of https://github.com/matrix-org/sytest/pull/753
-GET /rooms/:room_id/messages returns a message
-GET /rooms/:room_id/messages lazy loads members correctly
-Read receipts are sent as events
-Only original members of the room can see messages from erased users
-Device deletion propagates over federation
-If user leaves room, remote user changes device and rejoins we see update in /sync and /keys/changes
-Changing user-signing key notifies local users
-Newly updated tags appear in an incremental v2 /sync
-Server correctly handles incoming m.device_list_update
-Local device key changes get to remote servers with correct prev_id
-AS-ghosted users can use rooms via AS
-Ghost user must register before joining room
-Test that a message is pushed
-Invites are pushed
-Rooms with aliases are correctly named in pushed
-Rooms with names are correctly named in pushed
-Rooms with canonical alias are correctly named in pushed
-Rooms with many users are correctly pushed
-Don't get pushed for rooms you've muted
-Rejected events are not pushed
-Test that rejected pushers are removed.
-Events come down the correct room

CHANGES.md

@@ -1,95 +1,3 @@
-Synapse 1.6.0 (2019-11-26)
-==========================
-
-Bugfixes
---------
-
-- Fix phone home stats reporting. ([\#6418](https://github.com/matrix-org/synapse/issues/6418))
-
-
-Synapse 1.6.0rc2 (2019-11-25)
-=============================
-
-Bugfixes
---------
-
-- Fix a bug which could cause the background database update hander for event labels to get stuck in a loop raising exceptions. ([\#6407](https://github.com/matrix-org/synapse/issues/6407))
-
-
-Synapse 1.6.0rc1 (2019-11-20)
-=============================
-
-Features
---------
-
-- Add federation support for cross-signing. ([\#5727](https://github.com/matrix-org/synapse/issues/5727))
-- Increase default room version from 4 to 5, thereby enforcing server key validity period checks. ([\#6220](https://github.com/matrix-org/synapse/issues/6220))
-- Add support for outbound http proxying via http_proxy/HTTPS_PROXY env vars. ([\#6238](https://github.com/matrix-org/synapse/issues/6238))
-- Implement label-based filtering on `/sync` and `/messages` ([MSC2326](https://github.com/matrix-org/matrix-doc/pull/2326)). ([\#6301](https://github.com/matrix-org/synapse/issues/6301), [\#6310](https://github.com/matrix-org/synapse/issues/6310), [\#6340](https://github.com/matrix-org/synapse/issues/6340))
-
-
-Bugfixes
---------
-
-- Fix LruCache callback deduplication for Python 3.8. Contributed by @V02460. ([\#6213](https://github.com/matrix-org/synapse/issues/6213))
-- Remove a room from a server's public rooms list on room upgrade. ([\#6232](https://github.com/matrix-org/synapse/issues/6232), [\#6235](https://github.com/matrix-org/synapse/issues/6235))
-- Delete keys from key backup when deleting backup versions. ([\#6253](https://github.com/matrix-org/synapse/issues/6253))
-- Make notification of cross-signing signatures work with workers. ([\#6254](https://github.com/matrix-org/synapse/issues/6254))
-- Fix exception when remote servers attempt to join a room that they're not allowed to join. ([\#6278](https://github.com/matrix-org/synapse/issues/6278))
-- Prevent errors from appearing on Synapse startup if `git` is not installed. ([\#6284](https://github.com/matrix-org/synapse/issues/6284))
-- Appservice requests will no longer contain a double slash prefix when the appservice url provided ends in a slash. ([\#6306](https://github.com/matrix-org/synapse/issues/6306))
-- Fix `/purge_room` admin API. ([\#6307](https://github.com/matrix-org/synapse/issues/6307))
-- Fix the `hidden` field in the `devices` table for SQLite versions prior to 3.23.0. ([\#6313](https://github.com/matrix-org/synapse/issues/6313))
-- Fix bug which casued rejected events to be persisted with the wrong room state. ([\#6320](https://github.com/matrix-org/synapse/issues/6320))
-- Fix bug where `rc_login` ratelimiting would prematurely kick in. ([\#6335](https://github.com/matrix-org/synapse/issues/6335))
-- Prevent the server taking a long time to start up when guest registration is enabled. ([\#6338](https://github.com/matrix-org/synapse/issues/6338))
-- Fix bug where upgrading a guest account to a full user would fail when account validity is enabled. ([\#6359](https://github.com/matrix-org/synapse/issues/6359))
-- Fix `to_device` stream ID getting reset every time Synapse restarts, which had the potential to cause unable to decrypt errors. ([\#6363](https://github.com/matrix-org/synapse/issues/6363))
-- Fix permission denied error when trying to generate a config file with the docker image. ([\#6389](https://github.com/matrix-org/synapse/issues/6389))
-
-
-Improved Documentation
-----------------------
-
-- Contributor documentation now mentions script to run linters. ([\#6164](https://github.com/matrix-org/synapse/issues/6164))
-- Modify CAPTCHA_SETUP.md to update the terms `private key` and `public key` to `secret key` and `site key` respectively. Contributed by Yash Jipkate. ([\#6257](https://github.com/matrix-org/synapse/issues/6257))
-- Update `INSTALL.md` Email section to talk about `account_threepid_delegates`. ([\#6272](https://github.com/matrix-org/synapse/issues/6272))
-- Fix a small typo in `account_threepid_delegates` configuration option. ([\#6273](https://github.com/matrix-org/synapse/issues/6273))
-
-
-Internal Changes
-----------------
-
-- Add a CI job to test the `synapse_port_db` script. ([\#6140](https://github.com/matrix-org/synapse/issues/6140), [\#6276](https://github.com/matrix-org/synapse/issues/6276))
-- Convert EventContext to an attrs. ([\#6218](https://github.com/matrix-org/synapse/issues/6218))
-- Move `persist_events` out from main data store. ([\#6240](https://github.com/matrix-org/synapse/issues/6240), [\#6300](https://github.com/matrix-org/synapse/issues/6300))
-- Reduce verbosity of user/room stats. ([\#6250](https://github.com/matrix-org/synapse/issues/6250))
-- Reduce impact of debug logging. ([\#6251](https://github.com/matrix-org/synapse/issues/6251))
-- Expose some homeserver functionality to spam checkers. ([\#6259](https://github.com/matrix-org/synapse/issues/6259))
-- Change cache descriptors to always return deferreds. ([\#6263](https://github.com/matrix-org/synapse/issues/6263), [\#6291](https://github.com/matrix-org/synapse/issues/6291))
-- Fix incorrect comment regarding the functionality of an `if` statement. ([\#6269](https://github.com/matrix-org/synapse/issues/6269))
-- Update CI to run `isort` over the `scripts` and `scripts-dev` directories. ([\#6270](https://github.com/matrix-org/synapse/issues/6270))
-- Replace every instance of `logger.warn` method with `logger.warning` as the former is deprecated. ([\#6271](https://github.com/matrix-org/synapse/issues/6271), [\#6314](https://github.com/matrix-org/synapse/issues/6314))
-- Port replication http server endpoints to async/await. ([\#6274](https://github.com/matrix-org/synapse/issues/6274))
-- Port room rest handlers to async/await. ([\#6275](https://github.com/matrix-org/synapse/issues/6275))
-- Remove redundant CLI parameters on CI's `flake8` step. ([\#6277](https://github.com/matrix-org/synapse/issues/6277))
-- Port `federation_server.py` to async/await. ([\#6279](https://github.com/matrix-org/synapse/issues/6279))
-- Port receipt and read markers to async/wait. ([\#6280](https://github.com/matrix-org/synapse/issues/6280))
-- Split out state storage into separate data store. ([\#6294](https://github.com/matrix-org/synapse/issues/6294), [\#6295](https://github.com/matrix-org/synapse/issues/6295))
-- Refactor EventContext for clarity. ([\#6298](https://github.com/matrix-org/synapse/issues/6298))
-- Update the version of black used to 19.10b0. ([\#6304](https://github.com/matrix-org/synapse/issues/6304))
-- Add some documentation about worker replication. ([\#6305](https://github.com/matrix-org/synapse/issues/6305))
-- Move admin endpoints into separate files. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#6308](https://github.com/matrix-org/synapse/issues/6308))
-- Document the use of `lint.sh` for code style enforcement & extend it to run on specified paths only. ([\#6312](https://github.com/matrix-org/synapse/issues/6312))
-- Add optional python dependencies and dependant binary libraries to snapcraft packaging. ([\#6317](https://github.com/matrix-org/synapse/issues/6317))
-- Remove the dependency on psutil and replace functionality with the stdlib `resource` module. ([\#6318](https://github.com/matrix-org/synapse/issues/6318), [\#6336](https://github.com/matrix-org/synapse/issues/6336))
-- Improve documentation for EventContext fields. ([\#6319](https://github.com/matrix-org/synapse/issues/6319))
-- Add some checks that we aren't using state from rejected events. ([\#6330](https://github.com/matrix-org/synapse/issues/6330))
-- Add continuous integration for python 3.8. ([\#6341](https://github.com/matrix-org/synapse/issues/6341))
-- Correct spacing/case of various instances of the word "homeserver". ([\#6357](https://github.com/matrix-org/synapse/issues/6357))
-- Temporarily blacklist the failing unit test PurgeRoomTestCase.test_purge_room. ([\#6361](https://github.com/matrix-org/synapse/issues/6361))
-
-
 Synapse 1.5.1 (2019-11-06)
 ==========================
 

INSTALL.md

@@ -133,9 +133,9 @@ sudo yum install libtiff-devel libjpeg-devel libzip-devel freetype-devel \
 sudo yum groupinstall "Development Tools"
 ```
 
-#### macOS
+#### Mac OS X
 
-Installing prerequisites on macOS:
+Installing prerequisites on Mac OS X:
 
 ```
 xcode-select --install
@@ -144,14 +144,6 @@ sudo pip install virtualenv
 brew install pkg-config libffi
 ```
 
-On macOS Catalina (10.15) you may need to explicitly install OpenSSL
-via brew and inform `pip` about it so that `psycopg2` builds:
-
-```
-brew install openssl@1.1
-export LDFLAGS=-L/usr/local/Cellar/openssl\@1.1/1.1.1d/lib/
-```
-
 #### OpenSUSE
 
 Installing prerequisites on openSUSE:

changelog.d/5727.feature

@@ -0,0 +1 @@
+Add federation support for cross-signing.

changelog.d/5815.feature

@@ -1 +0,0 @@
-Implement per-room message retention policies.

changelog.d/5858.feature

@@ -1 +0,0 @@
-Add etag and count fields to key backup endpoints to help clients guess if there are new keys.

changelog.d/6140.misc

@@ -0,0 +1 @@
+Add a CI job to test the `synapse_port_db` script.

changelog.d/6164.doc

@@ -0,0 +1 @@
+Contributor documentation now mentions script to run linters.

changelog.d/6213.bugfix

@@ -0,0 +1 @@
+Fix LruCache callback deduplication for Python 3.8. Contributed by @V02460.

changelog.d/6218.misc

@@ -0,0 +1 @@
+Convert EventContext to an attrs.

changelog.d/6220.feature

@@ -0,0 +1 @@
+Increase default room version from 4 to 5, thereby enforcing server key validity period checks.

changelog.d/6232.bugfix

@@ -0,0 +1 @@
+Remove a room from a server's public rooms list on room upgrade.

changelog.d/6235.bugfix

@@ -0,0 +1 @@
+Remove a room from a server's public rooms list on room upgrade.

changelog.d/6238.feature

@@ -0,0 +1 @@
+Add support for outbound http proxying via http_proxy/HTTPS_PROXY env vars.

changelog.d/6240.misc

@@ -0,0 +1 @@
+Move `persist_events` out from main data store.

changelog.d/6250.misc

@@ -0,0 +1 @@
+Reduce verbosity of user/room stats.

changelog.d/6251.misc

@@ -0,0 +1 @@
+Reduce impact of debug logging.

changelog.d/6253.bugfix

@@ -0,0 +1 @@
+Delete keys from key backup when deleting backup versions.

changelog.d/6254.bugfix

@@ -0,0 +1 @@
+Make notification of cross-signing signatures work with workers.

changelog.d/6257.doc

@@ -0,0 +1 @@
+Modify CAPTCHA_SETUP.md to update the terms `private key` and `public key` to `secret key` and `site key` respectively. Contributed by Yash Jipkate.

changelog.d/6259.misc

@@ -0,0 +1 @@
+Expose some homeserver functionality to spam checkers.

changelog.d/6263.misc

@@ -0,0 +1 @@
+Change cache descriptors to always return deferreds.

changelog.d/6269.misc

@@ -0,0 +1 @@
+Fix incorrect comment regarding the functionality of an `if` statement.

changelog.d/6270.misc

@@ -0,0 +1 @@
+Update CI to run `isort` over the `scripts` and `scripts-dev` directories.

changelog.d/6271.misc

@@ -0,0 +1 @@
+Replace every instance of `logger.warn` method with `logger.warning` as the former is deprecated.

changelog.d/6272.doc

@@ -0,0 +1 @@
+Update `INSTALL.md` Email section to talk about `account_threepid_delegates`.

changelog.d/6273.doc

@@ -0,0 +1 @@
+Fix a small typo in `account_threepid_delegates` configuration option.

changelog.d/6274.misc

@@ -0,0 +1 @@
+Port replication http server endpoints to async/await.

changelog.d/6275.misc

@@ -0,0 +1 @@
+Port room rest handlers to async/await.

changelog.d/6276.misc

@@ -0,0 +1 @@
+Add a CI job to test the `synapse_port_db` script.

changelog.d/6277.misc

@@ -0,0 +1 @@
+Remove redundant CLI parameters on CI's `flake8` step.

changelog.d/6278.bugfix

@@ -0,0 +1 @@
+Fix exception when remote servers attempt to join a room that they're not allowed to join.

changelog.d/6279.misc

@@ -0,0 +1 @@
+Port `federation_server.py` to async/await.

changelog.d/6280.misc

@@ -0,0 +1 @@
+Port receipt and read markers to async/wait.

changelog.d/6284.bugfix

@@ -0,0 +1 @@
+Prevent errors from appearing on Synapse startup if `git` is not installed.

changelog.d/6291.misc

@@ -0,0 +1 @@
+Change cache descriptors to always return deferreds.

changelog.d/6294.misc

@@ -0,0 +1 @@
+Split out state storage into separate data store.

changelog.d/6295.misc

@@ -0,0 +1 @@
+Split out state storage into separate data store.

changelog.d/6298.misc

@@ -0,0 +1 @@
+Refactor EventContext for clarity.

changelog.d/6300.misc

@@ -0,0 +1 @@
+Move `persist_events` out from main data store.

changelog.d/6301.feature

@@ -0,0 +1 @@
+Implement label-based filtering on `/sync` and `/messages` ([MSC2326](https://github.com/matrix-org/matrix-doc/pull/2326)).

changelog.d/6304.misc

@@ -0,0 +1 @@
+Update the version of black used to 19.10b0.

changelog.d/6305.misc

@@ -0,0 +1 @@
+Add some documentation about worker replication.

changelog.d/6306.bugfix

@@ -0,0 +1 @@
+Appservice requests will no longer contain a double slash prefix when the appservice url provided ends in a slash.

changelog.d/6307.bugfix

@@ -0,0 +1 @@
+Fix `/purge_room` admin API.

changelog.d/6310.feature

@@ -0,0 +1 @@
+Implement label-based filtering on `/sync` and `/messages` ([MSC2326](https://github.com/matrix-org/matrix-doc/pull/2326)).

changelog.d/6312.misc

@@ -0,0 +1 @@
+Document the use of `lint.sh` for code style enforcement & extend it to run on specified paths only.

changelog.d/6313.bugfix

@@ -0,0 +1 @@
+Fix the `hidden` field in the `devices` table for SQLite versions prior to 3.23.0.

changelog.d/6314.misc

@@ -0,0 +1 @@
+Replace every instance of `logger.warn` method with `logger.warning` as the former is deprecated.

changelog.d/6317.misc

@@ -0,0 +1 @@
+Add optional python dependencies and dependant binary libraries to snapcraft packaging.

changelog.d/6318.misc

@@ -0,0 +1 @@
+Remove the dependency on psutil and replace functionality with the stdlib `resource` module.

changelog.d/6319.misc

@@ -0,0 +1 @@
+Improve documentation for EventContext fields.

changelog.d/6320.bugfix

@@ -0,0 +1 @@
+Fix bug which casued rejected events to be persisted with the wrong room state.

changelog.d/6322.misc

@@ -1 +0,0 @@
-Improve the performance of outputting structured logging.

changelog.d/6330.misc

@@ -0,0 +1 @@
+Add some checks that we aren't using state from rejected events.

changelog.d/6332.bugfix

@@ -1 +0,0 @@
-Fix caching devices for remote users when using workers, so that we don't attempt to refetch (and potentially fail) each time a user requests devices.

changelog.d/6333.bugfix

@@ -1 +0,0 @@
-Prevent account data syncs getting lost across TCP replication.

changelog.d/6336.misc

@@ -0,0 +1 @@
+Remove the dependency on psutil and replace functionality with the stdlib `resource` module.

changelog.d/6338.bugfix

@@ -0,0 +1 @@
+Prevent the server taking a long time to start up when guest registration is enabled.

changelog.d/6340.feature

@@ -0,0 +1 @@
+Implement label-based filtering on `/sync` and `/messages` ([MSC2326](https://github.com/matrix-org/matrix-doc/pull/2326)).

changelog.d/6341.misc

@@ -0,0 +1 @@
+Add continuous integration for python 3.8.

changelog.d/6343.misc

@@ -1 +0,0 @@
-Refactor some code in the event authentication path for clarity.

changelog.d/6357.misc

@@ -0,0 +1 @@
+Correct spacing/case of various instances of the word "homeserver".

changelog.d/6359.bugfix

@@ -0,0 +1 @@
+Fix bug where upgrading a guest account to a full user would fail when account validity is enabled.

changelog.d/6361.misc

@@ -0,0 +1 @@
+Temporarily blacklist the failing unit test PurgeRoomTestCase.test_purge_room.

changelog.d/6362.misc

@@ -1 +0,0 @@
-Clean up some unnecessary quotation marks around the codebase.

changelog.d/6363.bugfix

@@ -0,0 +1 @@
+Fix `to_device` stream ID getting reset every time Synapse restarts, which had the potential to cause unable to decrypt errors.

changelog.d/6375.bugfix

@@ -0,0 +1 @@
+Fix `to_device` stream ID getting reset every time Synapse restarts, which had the potential to cause unable to decrypt errors.

changelog.d/6379.misc

@@ -1 +0,0 @@
-Complain on startup instead of 500'ing during runtime when `public_baseurl` isn't set when necessary.

changelog.d/6388.doc

@@ -1 +0,0 @@
-Fix link in the user directory documentation.

changelog.d/6390.doc

@@ -1 +0,0 @@
-Add build instructions to the docker readme.

changelog.d/6392.misc

@@ -1 +0,0 @@
-Add a test scenario to make sure room history purges don't break `/messages` in the future.

changelog.d/6408.bugfix

@@ -1 +0,0 @@
-Fix an intermittent exception when handling read-receipts.

changelog.d/6420.bugfix

@@ -1 +0,0 @@
-Fix broken guest registration when there are existing blocks of numeric user IDs.

changelog.d/6421.bugfix

@@ -1 +0,0 @@
-Fix startup error when http proxy is defined.

changelog.d/6429.misc

@@ -1 +0,0 @@
-Add more tests to the blacklist when running in worker mode.

debian/changelog

@@ -1,9 +1,3 @@
-matrix-synapse-py3 (1.6.0) stable; urgency=medium
-
-  * New synapse release 1.6.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 26 Nov 2019 12:15:40 +0000
-
 matrix-synapse-py3 (1.5.1) stable; urgency=medium
 
   * New synapse release 1.5.1.

docker/README.md

@@ -130,15 +130,3 @@ docker run -it --rm \
 This will generate the same configuration file as the legacy mode used, but
 will store it in `/data/homeserver.yaml` instead of a temporary location. You
 can then use it as shown above at [Running synapse](#running-synapse).
-
-## Building the image
-
-If you need to build the image from a Synapse checkout, use the following `docker
- build` command from the repo's root:
- 
-```
-docker build -t matrixdotorg/synapse -f docker/Dockerfile .
-```
-
-You can choose to build a different docker image by changing the value of the `-f` flag to
-point to another Dockerfile.

docker/start.py

@@ -169,11 +169,11 @@ def run_generate_config(environ, ownership):
     # log("running %s" % (args, ))
 
     if ownership is not None:
-        # make sure that synapse has perms to write to the data dir.
-        subprocess.check_output(["chown", ownership, data_dir])
-
         args = ["su-exec", ownership] + args
         os.execv("/sbin/su-exec", args)
+
+        # make sure that synapse has perms to write to the data dir.
+        subprocess.check_output(["chown", ownership, data_dir])
     else:
         os.execv("/usr/local/bin/python", args)
 

docs/sample_config.yaml

@@ -328,69 +328,6 @@ listeners:
 #
 #user_ips_max_age: 14d
 
-# Message retention policy at the server level.
-#
-# Room admins and mods can define a retention period for their rooms using the
-# 'm.room.retention' state event, and server admins can cap this period by setting
-# the 'allowed_lifetime_min' and 'allowed_lifetime_max' config options.
-#
-# If this feature is enabled, Synapse will regularly look for and purge events
-# which are older than the room's maximum retention period. Synapse will also
-# filter events received over federation so that events that should have been
-# purged are ignored and not stored again.
-#
-retention:
-  # The message retention policies feature is disabled by default. Uncomment the
-  # following line to enable it.
-  #
-  #enabled: true
-
-  # Default retention policy. If set, Synapse will apply it to rooms that lack the
-  # 'm.room.retention' state event. Currently, the value of 'min_lifetime' doesn't
-  # matter much because Synapse doesn't take it into account yet.
-  #
-  #default_policy:
-  #  min_lifetime: 1d
-  #  max_lifetime: 1y
-
-  # Retention policy limits. If set, a user won't be able to send a
-  # 'm.room.retention' event which features a 'min_lifetime' or a 'max_lifetime'
-  # that's not within this range. This is especially useful in closed federations,
-  # in which server admins can make sure every federating server applies the same
-  # rules.
-  #
-  #allowed_lifetime_min: 1d
-  #allowed_lifetime_max: 1y
-
-  # Server admins can define the settings of the background jobs purging the
-  # events which lifetime has expired under the 'purge_jobs' section.
-  #
-  # If no configuration is provided, a single job will be set up to delete expired
-  # events in every room daily.
-  #
-  # Each job's configuration defines which range of message lifetimes the job
-  # takes care of. For example, if 'shortest_max_lifetime' is '2d' and
-  # 'longest_max_lifetime' is '3d', the job will handle purging expired events in
-  # rooms whose state defines a 'max_lifetime' that's both higher than 2 days, and
-  # lower than or equal to 3 days. Both the minimum and the maximum value of a
-  # range are optional, e.g. a job with no 'shortest_max_lifetime' and a
-  # 'longest_max_lifetime' of '3d' will handle every room with a retention policy
-  # which 'max_lifetime' is lower than or equal to three days.
-  #
-  # The rationale for this per-job configuration is that some rooms might have a
-  # retention policy with a low 'max_lifetime', where history needs to be purged
-  # of outdated messages on a very frequent basis (e.g. every 5min), but not want
-  # that purge to be performed by a job that's iterating over every room it knows,
-  # which would be quite heavy on the server.
-  #
-  #purge_jobs:
-  #  - shortest_max_lifetime: 1d
-  #    longest_max_lifetime: 3d
-  #    interval: 5m:
-  #  - shortest_max_lifetime: 3d
-  #    longest_max_lifetime: 1y
-  #    interval: 24h
-
 
 ## TLS ##
 

docs/user_directory.md

@@ -7,6 +7,7 @@ who are present in a publicly viewable room present on the server.
 
 The directory info is stored in various tables, which can (typically after
 DB corruption) get stale or out of sync.  If this happens, for now the
-solution to fix it is to execute the SQL [here](../synapse/storage/data_stores/main/schema/delta/53/user_dir_populate.sql)
+solution to fix it is to execute the SQL here
+https://github.com/matrix-org/synapse/blob/master/synapse/storage/schema/delta/53/user_dir_populate.sql
 and then restart synapse. This should then start a background task to
 flush the current tables and regenerate the directory.

synapse/__init__.py

@@ -36,7 +36,7 @@ try:
 except ImportError:
     pass
 
-__version__ = "1.6.0"
+__version__ = "1.5.1"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when

synapse/api/constants.py

@@ -94,8 +94,6 @@ class EventTypes(object):
     ServerACL = "m.room.server_acl"
     Pinned = "m.room.pinned_events"
 
-    Retention = "m.room.retention"
-
 
 class RejectedReason(object):
     AUTH_ERROR = "auth_error"

synapse/app/federation_sender.py

@@ -69,7 +69,7 @@ class FederationSenderSlaveStore(
         self.federation_out_pos_startup = self._get_federation_out_pos(db_conn)
 
     def _get_federation_out_pos(self, db_conn):
-        sql = "SELECT stream_id FROM federation_stream_position WHERE type = ?"
+        sql = "SELECT stream_id FROM federation_stream_position" " WHERE type = ?"
         sql = self.database_engine.convert_param_style(sql)
 
         txn = db_conn.cursor()

synapse/app/homeserver.py

@@ -585,7 +585,7 @@ def run(hs):
     def performance_stats_init():
         _stats_process.clear()
         _stats_process.append(
-            (int(hs.get_clock().time()), resource.getrusage(resource.RUSAGE_SELF))
+            (int(hs.get_clock().time(), resource.getrusage(resource.RUSAGE_SELF)))
         )
 
     def start_phone_stats_home():

synapse/appservice/api.py

@@ -185,7 +185,7 @@ class ApplicationServiceApi(SimpleHttpClient):
 
                 if not _is_valid_3pe_metadata(info):
                     logger.warning(
-                        "query_3pe_protocol to %s did not return a valid result", uri
+                        "query_3pe_protocol to %s did not return a" " valid result", uri
                     )
                     return None
 

synapse/config/appservice.py

@@ -134,7 +134,7 @@ def _load_appservice(hostname, as_info, config_filename):
             for regex_obj in as_info["namespaces"][ns]:
                 if not isinstance(regex_obj, dict):
                     raise ValueError(
-                        "Expected namespace entry in %s to be an object, but got %s",
+                        "Expected namespace entry in %s to be an object," " but got %s",
                         ns,
                         regex_obj,
                     )

synapse/config/emailconfig.py

@@ -146,8 +146,6 @@ class EmailConfig(Config):
                 if k not in email_config:
                     missing.append("email." + k)
 
-            # public_baseurl is required to build password reset and validation links that
-            # will be emailed to users
             if config.get("public_baseurl") is None:
                 missing.append("public_baseurl")
 

synapse/config/registration.py

@@ -106,13 +106,6 @@ class RegistrationConfig(Config):
         account_threepid_delegates = config.get("account_threepid_delegates") or {}
         self.account_threepid_delegate_email = account_threepid_delegates.get("email")
         self.account_threepid_delegate_msisdn = account_threepid_delegates.get("msisdn")
-        if self.account_threepid_delegate_msisdn and not self.public_baseurl:
-            raise ConfigError(
-                "The configuration option `public_baseurl` is required if "
-                "`account_threepid_delegate.msisdn` is set, such that "
-                "clients know where to submit validation tokens to. Please "
-                "configure `public_baseurl`."
-            )
 
         self.default_identity_server = config.get("default_identity_server")
         self.allow_guest_access = config.get("allow_guest_access", False)

synapse/config/room_directory.py

@@ -170,7 +170,7 @@ class _RoomDirectoryRule(object):
             self.action = action
         else:
             raise ConfigError(
-                "%s rules can only have action of 'allow' or 'deny'" % (option_name,)
+                "%s rules can only have action of 'allow'" " or 'deny'" % (option_name,)
             )
 
         self._alias_matches_all = alias == "*"

synapse/config/server.py

@@ -19,7 +19,7 @@ import logging
 import os.path
 import re
 from textwrap import indent
-from typing import Dict, List, Optional
+from typing import List
 
 import attr
 import yaml
@@ -223,7 +223,7 @@ class ServerConfig(Config):
             self.federation_ip_range_blacklist.update(["0.0.0.0", "::"])
         except Exception as e:
             raise ConfigError(
-                "Invalid range(s) provided in federation_ip_range_blacklist: %s" % e
+                "Invalid range(s) provided in " "federation_ip_range_blacklist: %s" % e
             )
 
         if self.public_baseurl is not None:
@@ -246,124 +246,6 @@ class ServerConfig(Config):
         # events with profile information that differ from the target's global profile.
         self.allow_per_room_profiles = config.get("allow_per_room_profiles", True)
 
-        retention_config = config.get("retention")
-        if retention_config is None:
-            retention_config = {}
-
-        self.retention_enabled = retention_config.get("enabled", False)
-
-        retention_default_policy = retention_config.get("default_policy")
-
-        if retention_default_policy is not None:
-            self.retention_default_min_lifetime = retention_default_policy.get(
-                "min_lifetime"
-            )
-            if self.retention_default_min_lifetime is not None:
-                self.retention_default_min_lifetime = self.parse_duration(
-                    self.retention_default_min_lifetime
-                )
-
-            self.retention_default_max_lifetime = retention_default_policy.get(
-                "max_lifetime"
-            )
-            if self.retention_default_max_lifetime is not None:
-                self.retention_default_max_lifetime = self.parse_duration(
-                    self.retention_default_max_lifetime
-                )
-
-            if (
-                self.retention_default_min_lifetime is not None
-                and self.retention_default_max_lifetime is not None
-                and (
-                    self.retention_default_min_lifetime
-                    > self.retention_default_max_lifetime
-                )
-            ):
-                raise ConfigError(
-                    "The default retention policy's 'min_lifetime' can not be greater"
-                    " than its 'max_lifetime'"
-                )
-        else:
-            self.retention_default_min_lifetime = None
-            self.retention_default_max_lifetime = None
-
-        self.retention_allowed_lifetime_min = retention_config.get(
-            "allowed_lifetime_min"
-        )
-        if self.retention_allowed_lifetime_min is not None:
-            self.retention_allowed_lifetime_min = self.parse_duration(
-                self.retention_allowed_lifetime_min
-            )
-
-        self.retention_allowed_lifetime_max = retention_config.get(
-            "allowed_lifetime_max"
-        )
-        if self.retention_allowed_lifetime_max is not None:
-            self.retention_allowed_lifetime_max = self.parse_duration(
-                self.retention_allowed_lifetime_max
-            )
-
-        if (
-            self.retention_allowed_lifetime_min is not None
-            and self.retention_allowed_lifetime_max is not None
-            and self.retention_allowed_lifetime_min
-            > self.retention_allowed_lifetime_max
-        ):
-            raise ConfigError(
-                "Invalid retention policy limits: 'allowed_lifetime_min' can not be"
-                " greater than 'allowed_lifetime_max'"
-            )
-
-        self.retention_purge_jobs = []  # type: List[Dict[str, Optional[int]]]
-        for purge_job_config in retention_config.get("purge_jobs", []):
-            interval_config = purge_job_config.get("interval")
-
-            if interval_config is None:
-                raise ConfigError(
-                    "A retention policy's purge jobs configuration must have the"
-                    " 'interval' key set."
-                )
-
-            interval = self.parse_duration(interval_config)
-
-            shortest_max_lifetime = purge_job_config.get("shortest_max_lifetime")
-
-            if shortest_max_lifetime is not None:
-                shortest_max_lifetime = self.parse_duration(shortest_max_lifetime)
-
-            longest_max_lifetime = purge_job_config.get("longest_max_lifetime")
-
-            if longest_max_lifetime is not None:
-                longest_max_lifetime = self.parse_duration(longest_max_lifetime)
-
-            if (
-                shortest_max_lifetime is not None
-                and longest_max_lifetime is not None
-                and shortest_max_lifetime > longest_max_lifetime
-            ):
-                raise ConfigError(
-                    "A retention policy's purge jobs configuration's"
-                    " 'shortest_max_lifetime' value can not be greater than its"
-                    " 'longest_max_lifetime' value."
-                )
-
-            self.retention_purge_jobs.append(
-                {
-                    "interval": interval,
-                    "shortest_max_lifetime": shortest_max_lifetime,
-                    "longest_max_lifetime": longest_max_lifetime,
-                }
-            )
-
-        if not self.retention_purge_jobs:
-            self.retention_purge_jobs = [
-                {
-                    "interval": self.parse_duration("1d"),
-                    "shortest_max_lifetime": None,
-                    "longest_max_lifetime": None,
-                }
-            ]
-
         self.listeners = []  # type: List[dict]
         for listener in config.get("listeners", []):
             if not isinstance(listener.get("port", None), int):
@@ -879,69 +761,6 @@ class ServerConfig(Config):
         # Defaults to `28d`. Set to `null` to disable clearing out of old rows.
         #
         #user_ips_max_age: 14d
-
-        # Message retention policy at the server level.
-        #
-        # Room admins and mods can define a retention period for their rooms using the
-        # 'm.room.retention' state event, and server admins can cap this period by setting
-        # the 'allowed_lifetime_min' and 'allowed_lifetime_max' config options.
-        #
-        # If this feature is enabled, Synapse will regularly look for and purge events
-        # which are older than the room's maximum retention period. Synapse will also
-        # filter events received over federation so that events that should have been
-        # purged are ignored and not stored again.
-        #
-        retention:
-          # The message retention policies feature is disabled by default. Uncomment the
-          # following line to enable it.
-          #
-          #enabled: true
-
-          # Default retention policy. If set, Synapse will apply it to rooms that lack the
-          # 'm.room.retention' state event. Currently, the value of 'min_lifetime' doesn't
-          # matter much because Synapse doesn't take it into account yet.
-          #
-          #default_policy:
-          #  min_lifetime: 1d
-          #  max_lifetime: 1y
-
-          # Retention policy limits. If set, a user won't be able to send a
-          # 'm.room.retention' event which features a 'min_lifetime' or a 'max_lifetime'
-          # that's not within this range. This is especially useful in closed federations,
-          # in which server admins can make sure every federating server applies the same
-          # rules.
-          #
-          #allowed_lifetime_min: 1d
-          #allowed_lifetime_max: 1y
-
-          # Server admins can define the settings of the background jobs purging the
-          # events which lifetime has expired under the 'purge_jobs' section.
-          #
-          # If no configuration is provided, a single job will be set up to delete expired
-          # events in every room daily.
-          #
-          # Each job's configuration defines which range of message lifetimes the job
-          # takes care of. For example, if 'shortest_max_lifetime' is '2d' and
-          # 'longest_max_lifetime' is '3d', the job will handle purging expired events in
-          # rooms whose state defines a 'max_lifetime' that's both higher than 2 days, and
-          # lower than or equal to 3 days. Both the minimum and the maximum value of a
-          # range are optional, e.g. a job with no 'shortest_max_lifetime' and a
-          # 'longest_max_lifetime' of '3d' will handle every room with a retention policy
-          # which 'max_lifetime' is lower than or equal to three days.
-          #
-          # The rationale for this per-job configuration is that some rooms might have a
-          # retention policy with a low 'max_lifetime', where history needs to be purged
-          # of outdated messages on a very frequent basis (e.g. every 5min), but not want
-          # that purge to be performed by a job that's iterating over every room it knows,
-          # which would be quite heavy on the server.
-          #
-          #purge_jobs:
-          #  - shortest_max_lifetime: 1d
-          #    longest_max_lifetime: 3d
-          #    interval: 5m:
-          #  - shortest_max_lifetime: 3d
-          #    longest_max_lifetime: 1y
-          #    interval: 24h
         """
             % locals()
         )
@@ -968,14 +787,14 @@ class ServerConfig(Config):
             "--print-pidfile",
             action="store_true",
             default=None,
-            help="Print the path to the pidfile just before daemonizing",
+            help="Print the path to the pidfile just" " before daemonizing",
         )
         server_group.add_argument(
             "--manhole",
             metavar="PORT",
             dest="manhole",
             type=int,
-            help="Turn on the twisted telnet manhole service on the given port.",
+            help="Turn on the twisted telnet manhole" " service on the given port.",
         )
 
 

synapse/events/validator.py

@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from six import integer_types, string_types
+from six import string_types
 
 from synapse.api.constants import MAX_ALIAS_LENGTH, EventTypes, Membership
 from synapse.api.errors import Codes, SynapseError
@@ -22,12 +22,11 @@ from synapse.types import EventID, RoomID, UserID
 
 
 class EventValidator(object):
-    def validate_new(self, event, config):
+    def validate_new(self, event):
         """Validates the event has roughly the right format
 
         Args:
-            event (FrozenEvent): The event to validate.
-            config (Config): The homeserver's configuration.
+            event (FrozenEvent)
         """
         self.validate_builder(event)
 
@@ -68,99 +67,6 @@ class EventValidator(object):
                             Codes.INVALID_PARAM,
                         )
 
-        if event.type == EventTypes.Retention:
-            self._validate_retention(event, config)
-
-    def _validate_retention(self, event, config):
-        """Checks that an event that defines the retention policy for a room respects the
-        boundaries imposed by the server's administrator.
-
-        Args:
-            event (FrozenEvent): The event to validate.
-            config (Config): The homeserver's configuration.
-        """
-        min_lifetime = event.content.get("min_lifetime")
-        max_lifetime = event.content.get("max_lifetime")
-
-        if min_lifetime is not None:
-            if not isinstance(min_lifetime, integer_types):
-                raise SynapseError(
-                    code=400,
-                    msg="'min_lifetime' must be an integer",
-                    errcode=Codes.BAD_JSON,
-                )
-
-            if (
-                config.retention_allowed_lifetime_min is not None
-                and min_lifetime < config.retention_allowed_lifetime_min
-            ):
-                raise SynapseError(
-                    code=400,
-                    msg=(
-                        "'min_lifetime' can't be lower than the minimum allowed"
-                        " value enforced by the server's administrator"
-                    ),
-                    errcode=Codes.BAD_JSON,
-                )
-
-            if (
-                config.retention_allowed_lifetime_max is not None
-                and min_lifetime > config.retention_allowed_lifetime_max
-            ):
-                raise SynapseError(
-                    code=400,
-                    msg=(
-                        "'min_lifetime' can't be greater than the maximum allowed"
-                        " value enforced by the server's administrator"
-                    ),
-                    errcode=Codes.BAD_JSON,
-                )
-
-        if max_lifetime is not None:
-            if not isinstance(max_lifetime, integer_types):
-                raise SynapseError(
-                    code=400,
-                    msg="'max_lifetime' must be an integer",
-                    errcode=Codes.BAD_JSON,
-                )
-
-            if (
-                config.retention_allowed_lifetime_min is not None
-                and max_lifetime < config.retention_allowed_lifetime_min
-            ):
-                raise SynapseError(
-                    code=400,
-                    msg=(
-                        "'max_lifetime' can't be lower than the minimum allowed value"
-                        " enforced by the server's administrator"
-                    ),
-                    errcode=Codes.BAD_JSON,
-                )
-
-            if (
-                config.retention_allowed_lifetime_max is not None
-                and max_lifetime > config.retention_allowed_lifetime_max
-            ):
-                raise SynapseError(
-                    code=400,
-                    msg=(
-                        "'max_lifetime' can't be greater than the maximum allowed"
-                        " value enforced by the server's administrator"
-                    ),
-                    errcode=Codes.BAD_JSON,
-                )
-
-        if (
-            min_lifetime is not None
-            and max_lifetime is not None
-            and min_lifetime > max_lifetime
-        ):
-            raise SynapseError(
-                code=400,
-                msg="'min_lifetime' can't be greater than 'max_lifetime",
-                errcode=Codes.BAD_JSON,
-            )
-
     def validate_builder(self, event):
         """Validates that the builder/event has roughly the right format. Only
         checks values that we expect a proto event to have, rather than all the

synapse/federation/persistence.py

@@ -44,7 +44,7 @@ class TransactionActions(object):
             response code and response body.
         """
         if not transaction.transaction_id:
-            raise RuntimeError("Cannot persist a transaction with no transaction_id")
+            raise RuntimeError("Cannot persist a transaction with no " "transaction_id")
 
         return self.store.get_received_txn_response(transaction.transaction_id, origin)
 
@@ -56,7 +56,7 @@ class TransactionActions(object):
             Deferred
         """
         if not transaction.transaction_id:
-            raise RuntimeError("Cannot persist a transaction with no transaction_id")
+            raise RuntimeError("Cannot persist a transaction with no " "transaction_id")
 
         return self.store.set_received_txn_response(
             transaction.transaction_id, origin, code, response

synapse/federation/sender/__init__.py

@@ -49,7 +49,7 @@ sent_pdus_destination_dist_count = Counter(
 
 sent_pdus_destination_dist_total = Counter(
     "synapse_federation_client_sent_pdu_destinations:total",
-    "Total number of PDUs queued for sending across all destinations",
+    "" "Total number of PDUs queued for sending across all destinations",
 )
 
 

synapse/federation/sender/transaction_manager.py

@@ -84,7 +84,7 @@ class TransactionManager(object):
             txn_id = str(self._next_txn_id)
 
             logger.debug(
-                "TX [%s] {%s} Attempting new transaction (pdus: %d, edus: %d)",
+                "TX [%s] {%s} Attempting new transaction" " (pdus: %d, edus: %d)",
                 destination,
                 txn_id,
                 len(pdus),
@@ -103,7 +103,7 @@ class TransactionManager(object):
             self._next_txn_id += 1
 
             logger.info(
-                "TX [%s] {%s} Sending transaction [%s], (PDUs: %d, EDUs: %d)",
+                "TX [%s] {%s} Sending transaction [%s]," " (PDUs: %d, EDUs: %d)",
                 destination,
                 txn_id,
                 transaction.transaction_id,

synapse/handlers/auth.py

@@ -102,9 +102,8 @@ class AuthHandler(BaseHandler):
                         login_types.append(t)
         self._supported_login_types = login_types
 
-        # Ratelimiter for failed auth during UIA. Uses same ratelimit config
-        # as per `rc_login.failed_attempts`.
-        self._failed_uia_attempts_ratelimiter = Ratelimiter()
+        self._account_ratelimiter = Ratelimiter()
+        self._failed_attempts_ratelimiter = Ratelimiter()
 
         self._clock = self.hs.get_clock()
 
@@ -134,38 +133,12 @@ class AuthHandler(BaseHandler):
 
             AuthError if the client has completed a login flow, and it gives
                 a different user to `requester`
-
-            LimitExceededError if the ratelimiter's failed request count for this
-                user is too high to proceed
-
         """
 
-        user_id = requester.user.to_string()
-
-        # Check if we should be ratelimited due to too many previous failed attempts
-        self._failed_uia_attempts_ratelimiter.ratelimit(
-            user_id,
-            time_now_s=self._clock.time(),
-            rate_hz=self.hs.config.rc_login_failed_attempts.per_second,
-            burst_count=self.hs.config.rc_login_failed_attempts.burst_count,
-            update=False,
-        )
-
         # build a list of supported flows
         flows = [[login_type] for login_type in self._supported_login_types]
 
-        try:
-            result, params, _ = yield self.check_auth(flows, request_body, clientip)
-        except LoginError:
-            # Update the ratelimite to say we failed (`can_do_action` doesn't raise).
-            self._failed_uia_attempts_ratelimiter.can_do_action(
-                user_id,
-                time_now_s=self._clock.time(),
-                rate_hz=self.hs.config.rc_login_failed_attempts.per_second,
-                burst_count=self.hs.config.rc_login_failed_attempts.burst_count,
-                update=True,
-            )
-            raise
+        result, params, _ = yield self.check_auth(flows, request_body, clientip)
 
         # find the completed login type
         for login_type in self._supported_login_types:
@@ -528,8 +501,11 @@ class AuthHandler(BaseHandler):
             multiple matches
 
         Raises:
+            LimitExceededError if the ratelimiter's login requests count for this
+                user is too high too proceed.
             UserDeactivatedError if a user is found but is deactivated.
         """
+        self.ratelimit_login_per_account(user_id)
         res = yield self._find_user_id_and_pwd_hash(user_id)
         if res is not None:
             return res[0]
@@ -596,6 +572,8 @@ class AuthHandler(BaseHandler):
             StoreError if there was a problem accessing the database
             SynapseError if there was a problem with the request
             LoginError if there was an authentication problem.
+            LimitExceededError if the ratelimiter's login requests count for this
+                user is too high too proceed.
         """
 
         if username.startswith("@"):
@@ -603,6 +581,8 @@ class AuthHandler(BaseHandler):
         else:
             qualified_user_id = UserID(username, self.hs.hostname).to_string()
 
+        self.ratelimit_login_per_account(qualified_user_id)
+
         login_type = login_submission.get("type")
         known_login_type = False
 
@@ -670,6 +650,15 @@ class AuthHandler(BaseHandler):
         if not known_login_type:
             raise SynapseError(400, "Unknown login type %s" % login_type)
 
+        # unknown username or invalid password.
+        self._failed_attempts_ratelimiter.ratelimit(
+            qualified_user_id.lower(),
+            time_now_s=self._clock.time(),
+            rate_hz=self.hs.config.rc_login_failed_attempts.per_second,
+            burst_count=self.hs.config.rc_login_failed_attempts.burst_count,
+            update=True,
+        )
+
         # We raise a 403 here, but note that if we're doing user-interactive
         # login, it turns all LoginErrors into a 401 anyway.
         raise LoginError(403, "Invalid password", errcode=Codes.FORBIDDEN)
@@ -721,6 +710,10 @@ class AuthHandler(BaseHandler):
         Returns:
             Deferred[unicode] the canonical_user_id, or Deferred[None] if
                 unknown user/bad password
+
+        Raises:
+            LimitExceededError if the ratelimiter's login requests count for this
+                user is too high too proceed.
         """
         lookupres = yield self._find_user_id_and_pwd_hash(user_id)
         if not lookupres:
@@ -749,7 +742,7 @@ class AuthHandler(BaseHandler):
             auth_api.validate_macaroon(macaroon, "login", user_id)
         except Exception:
             raise AuthError(403, "Invalid token", errcode=Codes.FORBIDDEN)
-
+        self.ratelimit_login_per_account(user_id)
         yield self.auth.check_auth_blocking(user_id)
         return user_id
 
@@ -919,6 +912,35 @@ class AuthHandler(BaseHandler):
         else:
             return defer.succeed(False)
 
+    def ratelimit_login_per_account(self, user_id):
+        """Checks whether the process must be stopped because of ratelimiting.
+
+        Checks against two ratelimiters: the generic one for login attempts per
+        account and the one specific to failed attempts.
+
+        Args:
+            user_id (unicode): complete @user:id
+
+        Raises:
+            LimitExceededError if one of the ratelimiters' login requests count
+                for this user is too high too proceed.
+        """
+        self._failed_attempts_ratelimiter.ratelimit(
+            user_id.lower(),
+            time_now_s=self._clock.time(),
+            rate_hz=self.hs.config.rc_login_failed_attempts.per_second,
+            burst_count=self.hs.config.rc_login_failed_attempts.burst_count,
+            update=False,
+        )
+
+        self._account_ratelimiter.ratelimit(
+            user_id.lower(),
+            time_now_s=self._clock.time(),
+            rate_hz=self.hs.config.rc_login_account.per_second,
+            burst_count=self.hs.config.rc_login_account.burst_count,
+            update=True,
+        )
+
 
 @attr.s
 class MacaroonGenerator(object):

synapse/handlers/directory.py

@@ -119,7 +119,7 @@ class DirectoryHandler(BaseHandler):
             if not service.is_interested_in_alias(room_alias.to_string()):
                 raise SynapseError(
                     400,
-                    "This application service has not reserved this kind of alias.",
+                    "This application service has not reserved" " this kind of alias.",
                     errcode=Codes.EXCLUSIVE,
                 )
         else:

synapse/handlers/e2e_keys.py

@@ -30,7 +30,6 @@ from twisted.internet import defer
 from synapse.api.errors import CodeMessageException, Codes, NotFoundError, SynapseError
 from synapse.logging.context import make_deferred_yieldable, run_in_background
 from synapse.logging.opentracing import log_kv, set_tag, tag_args, trace
-from synapse.replication.http.devices import ReplicationUserDevicesResyncRestServlet
 from synapse.types import (
     UserID,
     get_domain_from_id,
@@ -54,12 +53,6 @@ class E2eKeysHandler(object):
 
         self._edu_updater = SigningKeyEduUpdater(hs, self)
 
-        self._is_master = hs.config.worker_app is None
-        if not self._is_master:
-            self._user_device_resync_client = ReplicationUserDevicesResyncRestServlet.make_client(
-                hs
-            )
-
         federation_registry = hs.get_federation_registry()
 
         # FIXME: switch to m.signing_key_update when MSC1756 is merged into the spec
@@ -198,15 +191,9 @@ class E2eKeysHandler(object):
                 # probably be tracking their device lists. However, we haven't
                 # done an initial sync on the device list so we do it now.
                 try:
-                    if self._is_master:
-                        user_devices = yield self.device_handler.device_list_updater.user_device_resync(
-                            user_id
-                        )
-                    else:
-                        user_devices = yield self._user_device_resync_client(
-                            user_id=user_id
-                        )
-
+                    user_devices = yield self.device_handler.device_list_updater.user_device_resync(
+                        user_id
+                    )
                     user_devices = user_devices["devices"]
                     for device in user_devices:
                         results[user_id] = {device["device_id"]: device["keys"]}

synapse/handlers/e2e_room_keys.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 # Copyright 2017, 2018 New Vector Ltd
-# Copyright 2019 Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -104,35 +103,14 @@ class E2eRoomKeysHandler(object):
                 rooms
             session_id(string): session ID to delete keys for, for None to delete keys
                 for all sessions
-        Raises:
-            NotFoundError: if the backup version does not exist
         Returns:
-            A dict containing the count and etag for the backup version
+            A deferred of the deletion transaction
         """
 
         # lock for consistency with uploading
         with (yield self._upload_linearizer.queue(user_id)):
-            # make sure the backup version exists
-            try:
-                version_info = yield self.store.get_e2e_room_keys_version_info(
-                    user_id, version
-                )
-            except StoreError as e:
-                if e.code == 404:
-                    raise NotFoundError("Unknown backup version")
-                else:
-                    raise
-
             yield self.store.delete_e2e_room_keys(user_id, version, room_id, session_id)
 
-            version_etag = version_info["etag"] + 1
-            yield self.store.update_e2e_room_keys_version(
-                user_id, version, None, version_etag
-            )
-
-            count = yield self.store.count_e2e_room_keys(user_id, version)
-            return {"etag": str(version_etag), "count": count}
-
     @trace
     @defer.inlineCallbacks
     def upload_room_keys(self, user_id, version, room_keys):
@@ -160,9 +138,6 @@ class E2eRoomKeysHandler(object):
             }
         }
 
-        Returns:
-            A dict containing the count and etag for the backup version
-
         Raises:
             NotFoundError: if there are no versions defined
             RoomKeysVersionError: if the uploaded version is not the current version
@@ -196,62 +171,59 @@ class E2eRoomKeysHandler(object):
                     else:
                         raise
 
-            # Fetch any existing room keys for the sessions that have been
-            # submitted.  Then compare them with the submitted keys.  If the
-            # key is new, insert it; if the key should be updated, then update
-            # it; otherwise, drop it.
-            existing_keys = yield self.store.get_e2e_room_keys_multi(
-                user_id, version, room_keys["rooms"]
-            )
-            to_insert = []  # batch the inserts together
-            changed = False  # if anything has changed, we need to update the etag
+            # go through the room_keys.
+            # XXX: this should/could be done concurrently, given we're in a lock.
             for room_id, room in iteritems(room_keys["rooms"]):
-                for session_id, room_key in iteritems(room["sessions"]):
-                    log_kv(
-                        {
-                            "message": "Trying to upload room key",
-                            "room_id": room_id,
-                            "session_id": session_id,
-                            "user_id": user_id,
-                        }
+                for session_id, session in iteritems(room["sessions"]):
+                    yield self._upload_room_key(
+                        user_id, version, room_id, session_id, session
                     )
-                    current_room_key = existing_keys.get(room_id, {}).get(session_id)
-                    if current_room_key:
-                        if self._should_replace_room_key(current_room_key, room_key):
-                            log_kv({"message": "Replacing room key."})
-                            # updates are done one at a time in the DB, so send
-                            # updates right away rather than batching them up,
-                            # like we do with the inserts
-                            yield self.store.update_e2e_room_key(
-                                user_id, version, room_id, session_id, room_key
-                            )
-                            changed = True
-                        else:
-                            log_kv({"message": "Not replacing room_key."})
-                    else:
-                        log_kv(
-                            {
-                                "message": "Room key not found.",
-                                "room_id": room_id,
-                                "user_id": user_id,
-                            }
-                        )
-                        log_kv({"message": "Replacing room key."})
-                        to_insert.append((room_id, session_id, room_key))
-                        changed = True
 
-            if len(to_insert):
-                yield self.store.add_e2e_room_keys(user_id, version, to_insert)
+    @defer.inlineCallbacks
+    def _upload_room_key(self, user_id, version, room_id, session_id, room_key):
+        """Upload a given room_key for a given room and session into a given
+        version of the backup.  Merges the key with any which might already exist.
 
-            version_etag = version_info["etag"]
-            if changed:
-                version_etag = version_etag + 1
-                yield self.store.update_e2e_room_keys_version(
-                    user_id, version, None, version_etag
+        Args:
+            user_id(str): the user whose backup we're setting
+            version(str): the version ID of the backup we're updating
+            room_id(str): the ID of the room whose keys we're setting
+            session_id(str): the session whose room_key we're setting
+            room_key(dict): the room_key being set
+        """
+        log_kv(
+            {
+                "message": "Trying to upload room key",
+                "room_id": room_id,
+                "session_id": session_id,
+                "user_id": user_id,
+            }
+        )
+        # get the room_key for this particular row
+        current_room_key = None
+        try:
+            current_room_key = yield self.store.get_e2e_room_key(
+                user_id, version, room_id, session_id
+            )
+        except StoreError as e:
+            if e.code == 404:
+                log_kv(
+                    {
+                        "message": "Room key not found.",
+                        "room_id": room_id,
+                        "user_id": user_id,
+                    }
                 )
+            else:
+                raise
 
-            count = yield self.store.count_e2e_room_keys(user_id, version)
-            return {"etag": str(version_etag), "count": count}
+        if self._should_replace_room_key(current_room_key, room_key):
+            log_kv({"message": "Replacing room key."})
+            yield self.store.set_e2e_room_key(
+                user_id, version, room_id, session_id, room_key
+            )
+        else:
+            log_kv({"message": "Not replacing room_key."})
 
     @staticmethod
     def _should_replace_room_key(current_room_key, room_key):
@@ -342,8 +314,6 @@ class E2eRoomKeysHandler(object):
                     raise NotFoundError("Unknown backup version")
                 else:
                     raise
-
-            res["count"] = yield self.store.count_e2e_room_keys(user_id, res["version"])
             return res
 
     @trace