Return empty object on success rather than null.

Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
Add admin API docs for setting admin bits on users
2019-08-20 10:29:36 +01:00 · 2019-08-20 10:24:38 +01:00 · 2019-08-19 14:44:53 +01:00 · 2019-08-19 14:42:55 +01:00 · 2019-08-19 14:41:07 +01:00 · 2019-08-19 10:20:59 +01:00
97 changed files with 563 additions and 509 deletions
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,3 +1,102 @@
+Synapse 1.3.1 (2019-08-17)
+==========================
+
+Features
+--------
+
+- Drop hard dependency on `sdnotify` python package. ([\#5871](https://github.com/matrix-org/synapse/issues/5871))
+
+
+Bugfixes
+--------
+
+- Fix startup issue (hang on ACME provisioning) due to ordering of Twisted reactor startup. Thanks to @chrismoos for supplying the fix. ([\#5867](https://github.com/matrix-org/synapse/issues/5867))
+
+
+Synapse 1.3.0 (2019-08-15)
+==========================
+
+Bugfixes
+--------
+
+- Fix 500 Internal Server Error on `publicRooms` when the public room list was
+  cached. ([\#5851](https://github.com/matrix-org/synapse/issues/5851))
+
+
+Synapse 1.3.0rc1 (2019-08-13)
+==========================
+
+Features
+--------
+
+- Use `M_USER_DEACTIVATED` instead of `M_UNKNOWN` for errcode when a deactivated user attempts to login. ([\#5686](https://github.com/matrix-org/synapse/issues/5686))
+- Add sd_notify hooks to ease systemd integration and allows usage of Type=Notify. ([\#5732](https://github.com/matrix-org/synapse/issues/5732))
+- Synapse will no longer serve any media repo admin endpoints when `enable_media_repo` is set to False in the configuration. If a media repo worker is used, the admin APIs relating to the media repo will be served from it instead. ([\#5754](https://github.com/matrix-org/synapse/issues/5754), [\#5848](https://github.com/matrix-org/synapse/issues/5848))
+- Synapse can now be configured to not join remote rooms of a given "complexity" (currently, state events) over federation. This option can be used to prevent adverse performance on resource-constrained homeservers. ([\#5783](https://github.com/matrix-org/synapse/issues/5783))
+- Allow defining HTML templates to serve the user on account renewal attempt when using the account validity feature. ([\#5807](https://github.com/matrix-org/synapse/issues/5807))
+
+
+Bugfixes
+--------
+
+- Fix UISIs during homeserver outage. ([\#5693](https://github.com/matrix-org/synapse/issues/5693), [\#5789](https://github.com/matrix-org/synapse/issues/5789))
+- Fix stack overflow in server key lookup code. ([\#5724](https://github.com/matrix-org/synapse/issues/5724))
+- start.sh no longer uses deprecated cli option. ([\#5725](https://github.com/matrix-org/synapse/issues/5725))
+- Log when we receive an event receipt from an unexpected origin. ([\#5743](https://github.com/matrix-org/synapse/issues/5743))
+- Fix debian packaging scripts to correctly build sid packages. ([\#5775](https://github.com/matrix-org/synapse/issues/5775))
+- Correctly handle redactions of redactions. ([\#5788](https://github.com/matrix-org/synapse/issues/5788))
+- Return 404 instead of 403 when accessing /rooms/{roomId}/event/{eventId} for an event without the appropriate permissions. ([\#5798](https://github.com/matrix-org/synapse/issues/5798))
+- Fix check that tombstone is a state event in push rules. ([\#5804](https://github.com/matrix-org/synapse/issues/5804))
+- Fix error when trying to login as a deactivated user when using a worker to handle login. ([\#5806](https://github.com/matrix-org/synapse/issues/5806))
+- Fix bug where user `/sync` stream could get wedged in rare circumstances. ([\#5825](https://github.com/matrix-org/synapse/issues/5825))
+- The purge_remote_media.sh script was fixed. ([\#5839](https://github.com/matrix-org/synapse/issues/5839))
+
+
+Deprecations and Removals
+-------------------------
+
+- Synapse now no longer accepts the `-v`/`--verbose`, `-f`/`--log-file`, or `--log-config` command line flags, and removes the deprecated `verbose` and `log_file` configuration file options. Users of these options should migrate their options into the dedicated log configuration. ([\#5678](https://github.com/matrix-org/synapse/issues/5678), [\#5729](https://github.com/matrix-org/synapse/issues/5729))
+- Remove non-functional 'expire_access_token' setting. ([\#5782](https://github.com/matrix-org/synapse/issues/5782))
+
+
+Internal Changes
+----------------
+
+- Make Jaeger fully configurable. ([\#5694](https://github.com/matrix-org/synapse/issues/5694))
+- Add precautionary measures to prevent future abuse of `window.opener` in default welcome page. ([\#5695](https://github.com/matrix-org/synapse/issues/5695))
+- Reduce database IO usage by optimising queries for current membership. ([\#5706](https://github.com/matrix-org/synapse/issues/5706), [\#5738](https://github.com/matrix-org/synapse/issues/5738), [\#5746](https://github.com/matrix-org/synapse/issues/5746), [\#5752](https://github.com/matrix-org/synapse/issues/5752), [\#5770](https://github.com/matrix-org/synapse/issues/5770), [\#5774](https://github.com/matrix-org/synapse/issues/5774), [\#5792](https://github.com/matrix-org/synapse/issues/5792), [\#5793](https://github.com/matrix-org/synapse/issues/5793))
+- Improve caching when fetching `get_filtered_current_state_ids`. ([\#5713](https://github.com/matrix-org/synapse/issues/5713))
+- Don't accept opentracing data from clients. ([\#5715](https://github.com/matrix-org/synapse/issues/5715))
+- Speed up PostgreSQL unit tests in CI. ([\#5717](https://github.com/matrix-org/synapse/issues/5717))
+- Update the coding style document. ([\#5719](https://github.com/matrix-org/synapse/issues/5719))
+- Improve database query performance when recording retry intervals for remote hosts. ([\#5720](https://github.com/matrix-org/synapse/issues/5720))
+- Add a set of opentracing utils. ([\#5722](https://github.com/matrix-org/synapse/issues/5722))
+- Cache result of get_version_string to reduce overhead of `/version` federation requests. ([\#5730](https://github.com/matrix-org/synapse/issues/5730))
+- Return 'user_type' in admin API user endpoints results. ([\#5731](https://github.com/matrix-org/synapse/issues/5731))
+- Don't package the sytest test blacklist file. ([\#5733](https://github.com/matrix-org/synapse/issues/5733))
+- Replace uses of returnValue with plain return, as returnValue is not needed on Python 3. ([\#5736](https://github.com/matrix-org/synapse/issues/5736))
+- Blacklist some flakey tests in worker mode. ([\#5740](https://github.com/matrix-org/synapse/issues/5740))
+- Fix some error cases in the caching layer. ([\#5749](https://github.com/matrix-org/synapse/issues/5749))
+- Add a prometheus metric for pending cache lookups. ([\#5750](https://github.com/matrix-org/synapse/issues/5750))
+- Stop trying to fetch events with event_id=None. ([\#5753](https://github.com/matrix-org/synapse/issues/5753))
+- Convert RedactionTestCase to modern test style. ([\#5768](https://github.com/matrix-org/synapse/issues/5768))
+- Allow looping calls to be given arguments. ([\#5780](https://github.com/matrix-org/synapse/issues/5780))
+- Set the logs emitted when checking typing and presence timeouts to DEBUG level, not INFO. ([\#5785](https://github.com/matrix-org/synapse/issues/5785))
+- Remove DelayedCall debugging from the test suite, as it is no longer required in the vast majority of Synapse's tests. ([\#5787](https://github.com/matrix-org/synapse/issues/5787))
+- Remove some spurious exceptions from the logs where we failed to talk to a remote server. ([\#5790](https://github.com/matrix-org/synapse/issues/5790))
+- Improve performance when making `.well-known` requests by sharing the SSL options between requests. ([\#5794](https://github.com/matrix-org/synapse/issues/5794))
+- Disable codecov GitHub comments on PRs. ([\#5796](https://github.com/matrix-org/synapse/issues/5796))
+- Don't allow clients to send tombstone events that reference the room it's sent in. ([\#5801](https://github.com/matrix-org/synapse/issues/5801))
+- Deny redactions of events sent in a different room. ([\#5802](https://github.com/matrix-org/synapse/issues/5802))
+- Deny sending well known state types as non-state events. ([\#5805](https://github.com/matrix-org/synapse/issues/5805))
+- Handle incorrectly encoded query params correctly by returning a 400. ([\#5808](https://github.com/matrix-org/synapse/issues/5808))
+- Handle pusher being deleted during processing rather than logging an exception. ([\#5809](https://github.com/matrix-org/synapse/issues/5809))
+- Return 502 not 500 when failing to reach any remote server. ([\#5810](https://github.com/matrix-org/synapse/issues/5810))
+- Reduce global pauses in the events stream caused by expensive state resolution during persistence. ([\#5826](https://github.com/matrix-org/synapse/issues/5826))
+- Add a lower bound to well-known lookup cache time to avoid repeated lookups. ([\#5836](https://github.com/matrix-org/synapse/issues/5836))
+- Whitelist history visbility sytests in worker mode tests. ([\#5843](https://github.com/matrix-org/synapse/issues/5843))
+
+
 Synapse 1.2.1 (2019-07-26)
 ==========================

--- a/INSTALL.md
+++ b/INSTALL.md
@@ -419,12 +419,11 @@ If Synapse is not configured with an SMTP server, password reset via email will

 ## Registering a user

-You will need at least one user on your server in order to use a Matrix
-client. Users can be registered either via a Matrix client, or via a
-commandline script.
+The easiest way to create a new user is to do so from a client like [Riot](https://riot.im).

-To get started, it is easiest to use the command line to register new
-users. This can be done as follows:
+Alternatively you can do so from the command line if you have installed via pip. 
+
+This can be done as follows:

 ```
 $ source ~/synapse/env/bin/activate
--- a/changelog.d/5633.bugfix
+++ b/changelog.d/5633.bugfix
@@ -0,0 +1 @@
+Don't create broken room when power_level_content_override.users does not contain creator_id.
--- a/changelog.d/5678.removal
+++ b/changelog.d/5678.removal
@@ -1 +0,0 @@
-Synapse now no longer accepts the `-v`/`--verbose`, `-f`/`--log-file`, or `--log-config` command line flags, and removes the deprecated `verbose` and `log_file` configuration file options. Users of these options should migrate their options into the dedicated log configuration.
--- a/changelog.d/5686.feature
+++ b/changelog.d/5686.feature
@@ -1 +0,0 @@
-Use `M_USER_DEACTIVATED` instead of `M_UNKNOWN` for errcode when a deactivated user attempts to login.
--- a/changelog.d/5693.bugfix
+++ b/changelog.d/5693.bugfix
@@ -1 +0,0 @@
-Fix UISIs during homeserver outage.
--- a/changelog.d/5694.misc
+++ b/changelog.d/5694.misc
@@ -1 +0,0 @@
-Make Jaeger fully configurable.
--- a/changelog.d/5695.misc
+++ b/changelog.d/5695.misc
@@ -1 +0,0 @@
-Add precautionary measures to prevent future abuse of `window.opener` in default welcome page.
--- a/changelog.d/5706.misc
+++ b/changelog.d/5706.misc
@@ -1 +0,0 @@
-Reduce database IO usage by optimising queries for current membership.
--- a/changelog.d/5713.misc
+++ b/changelog.d/5713.misc
@@ -1 +0,0 @@
-Improve caching when fetching `get_filtered_current_state_ids`.
--- a/changelog.d/5715.misc
+++ b/changelog.d/5715.misc
@@ -1 +0,0 @@
-Don't accept opentracing data from clients.
--- a/changelog.d/5717.misc
+++ b/changelog.d/5717.misc
@@ -1 +0,0 @@
-Speed up PostgreSQL unit tests in CI.
--- a/changelog.d/5719.misc
+++ b/changelog.d/5719.misc
@@ -1 +0,0 @@
-Update the coding style document.
--- a/changelog.d/5720.misc
+++ b/changelog.d/5720.misc
@@ -1 +0,0 @@
-Improve database query performance when recording retry intervals for remote hosts.
--- a/changelog.d/5722.misc
+++ b/changelog.d/5722.misc
@@ -1 +0,0 @@
-Add a set of opentracing utils.
--- a/changelog.d/5724.bugfix
+++ b/changelog.d/5724.bugfix
@@ -1 +0,0 @@
-Fix stack overflow in server key lookup code.
--- a/changelog.d/5725.bugfix
+++ b/changelog.d/5725.bugfix
@@ -1 +0,0 @@
-start.sh no longer uses deprecated cli option.
--- a/changelog.d/5729.removal
+++ b/changelog.d/5729.removal
@@ -1 +0,0 @@
- Synapse now no longer accepts the `-v`/`--verbose`, `-f`/`--log-file`, or `--log-config` command line flags, and removes the deprecated `verbose` and `log_file` configuration file options. Users of these options should migrate their options into the dedicated log configuration.
--- a/changelog.d/5730.misc
+++ b/changelog.d/5730.misc
@@ -1 +0,0 @@
-Cache result of get_version_string to reduce overhead of `/version` federation requests.
--- a/changelog.d/5731.misc
+++ b/changelog.d/5731.misc
@@ -1 +0,0 @@
-Return 'user_type' in admin API user endpoints results.
--- a/changelog.d/5732.feature
+++ b/changelog.d/5732.feature
@@ -1 +0,0 @@
-Add sd_notify hooks to ease systemd integration and allows usage of Type=Notify.
--- a/changelog.d/5733.misc
+++ b/changelog.d/5733.misc
@@ -1 +0,0 @@
-Don't package the sytest test blacklist file.
--- a/changelog.d/5736.misc
+++ b/changelog.d/5736.misc
@@ -1 +0,0 @@
-Replace uses of returnValue with plain return, as returnValue is not needed on Python 3.
--- a/changelog.d/5738.misc
+++ b/changelog.d/5738.misc
@@ -1 +0,0 @@
-Reduce database IO usage by optimising queries for current membership.
--- a/changelog.d/5740.misc
+++ b/changelog.d/5740.misc
@@ -1 +0,0 @@
-Blacklist some flakey tests in worker mode.
--- a/changelog.d/5743.bugfix
+++ b/changelog.d/5743.bugfix
@@ -1 +0,0 @@
-Log when we receive an event receipt from an unexpected origin.
--- a/changelog.d/5746.misc
+++ b/changelog.d/5746.misc
@@ -1 +0,0 @@
-Reduce database IO usage by optimising queries for current membership.
--- a/changelog.d/5749.misc
+++ b/changelog.d/5749.misc
@@ -1 +0,0 @@
-Fix some error cases in the caching layer.
--- a/changelog.d/5750.misc
+++ b/changelog.d/5750.misc
@@ -1 +0,0 @@
-Add a prometheus metric for pending cache lookups.
--- a/changelog.d/5752.misc
+++ b/changelog.d/5752.misc
@@ -1 +0,0 @@
-Reduce database IO usage by optimising queries for current membership.
--- a/changelog.d/5753.misc
+++ b/changelog.d/5753.misc
@@ -1 +0,0 @@
-Stop trying to fetch events with event_id=None.
--- a/changelog.d/5754.feature
+++ b/changelog.d/5754.feature
@@ -1 +0,0 @@
-Synapse will no longer serve any media repo admin endpoints when `enable_media_repo` is set to False in the configuration. If a media repo worker is used, the admin APIs relating to the media repo will be served from it instead.
--- a/changelog.d/5768.misc
+++ b/changelog.d/5768.misc
@@ -1 +0,0 @@
-Convert RedactionTestCase to modern test style.
--- a/changelog.d/5770.misc
+++ b/changelog.d/5770.misc
@@ -1 +0,0 @@
-Reduce database IO usage by optimising queries for current membership.
--- a/changelog.d/5774.misc
+++ b/changelog.d/5774.misc
@@ -1 +0,0 @@
-Reduce database IO usage by optimising queries for current membership.
--- a/changelog.d/5775.bugfix
+++ b/changelog.d/5775.bugfix
@@ -1 +0,0 @@
-Fix debian packaging scripts to correctly build sid packages.
--- a/changelog.d/5780.misc
+++ b/changelog.d/5780.misc
@@ -1 +0,0 @@
-Allow looping calls to be given arguments.
--- a/changelog.d/5782.removal
+++ b/changelog.d/5782.removal
@@ -1 +0,0 @@
-Remove non-functional 'expire_access_token' setting.
--- a/changelog.d/5783.feature
+++ b/changelog.d/5783.feature
@@ -1 +0,0 @@
-Synapse can now be configured to not join remote rooms of a given "complexity" (currently, state events) over federation. This option can be used to prevent adverse performance on resource-constrained homeservers.
--- a/changelog.d/5785.misc
+++ b/changelog.d/5785.misc
@@ -1 +0,0 @@
-Set the logs emitted when checking typing and presence timeouts to DEBUG level, not INFO.
--- a/changelog.d/5787.misc
+++ b/changelog.d/5787.misc
@@ -1 +0,0 @@
-Remove DelayedCall debugging from the test suite, as it is no longer required in the vast majority of Synapse's tests.
--- a/changelog.d/5788.bugfix
+++ b/changelog.d/5788.bugfix
@@ -1 +0,0 @@
-Correctly handle redactions of redactions.
--- a/changelog.d/5789.bugfix
+++ b/changelog.d/5789.bugfix
@@ -1 +0,0 @@
-Fix UISIs during homeserver outage.
--- a/changelog.d/5790.misc
+++ b/changelog.d/5790.misc
@@ -1 +0,0 @@
-Remove some spurious exceptions from the logs where we failed to talk to a remote server.
--- a/changelog.d/5792.misc
+++ b/changelog.d/5792.misc
@@ -1 +0,0 @@
-Reduce database IO usage by optimising queries for current membership.
--- a/changelog.d/5793.misc
+++ b/changelog.d/5793.misc
@@ -1 +0,0 @@
-Reduce database IO usage by optimising queries for current membership.
--- a/changelog.d/5794.misc
+++ b/changelog.d/5794.misc
@@ -1 +0,0 @@
-Improve performance when making `.well-known` requests by sharing the SSL options between requests.
--- a/changelog.d/5796.misc
+++ b/changelog.d/5796.misc
@@ -1 +0,0 @@
-Disable codecov GitHub comments on PRs.
--- a/changelog.d/5798.bugfix
+++ b/changelog.d/5798.bugfix
@@ -1 +0,0 @@
-Return 404 instead of 403 when accessing /rooms/{roomId}/event/{eventId} for an event without the appropriate permissions.
--- a/changelog.d/5801.misc
+++ b/changelog.d/5801.misc
@@ -1 +0,0 @@
-Don't allow clients to send tombstone events that reference the room it's sent in.
--- a/changelog.d/5802.misc
+++ b/changelog.d/5802.misc
@@ -1 +0,0 @@
-Deny redactions of events sent in a different room. 
--- a/changelog.d/5804.bugfix
+++ b/changelog.d/5804.bugfix
@@ -1 +0,0 @@
-Fix check that tombstone is a state event in push rules.
--- a/changelog.d/5805.misc
+++ b/changelog.d/5805.misc
@@ -1 +0,0 @@
-Deny sending well known state types as non-state events.
--- a/changelog.d/5806.bugfix
+++ b/changelog.d/5806.bugfix
@@ -1 +0,0 @@
-Fix error when trying to login as a deactivated user when using a worker to handle login.
--- a/changelog.d/5807.feature
+++ b/changelog.d/5807.feature
@@ -1 +0,0 @@
-Allow defining HTML templates to serve the user on account renewal attempt when using the account validity feature.
--- a/changelog.d/5808.misc
+++ b/changelog.d/5808.misc
@@ -1 +0,0 @@
-Handle incorrectly encoded query params correctly by returning a 400.
--- a/changelog.d/5810.misc
+++ b/changelog.d/5810.misc
@@ -1 +0,0 @@
-Return 502 not 500 when failing to reach any remote server.
--- a/changelog.d/5825.bugfix
+++ b/changelog.d/5825.bugfix
@@ -1 +0,0 @@
-Fix bug where user `/sync` stream could get wedged in rare circumstances.
--- a/changelog.d/5826.misc
+++ b/changelog.d/5826.misc
@@ -1 +0,0 @@
-Reduce global pauses in the events stream caused by expensive state resolution during persistence.
--- a/changelog.d/5836.misc
+++ b/changelog.d/5836.misc
@@ -1 +0,0 @@
-Add a lower bound to well-known lookup cache time to avoid repeated lookups.
--- a/changelog.d/5839.bugfix
+++ b/changelog.d/5839.bugfix
@@ -1 +0,0 @@
-The purge_remote_media.sh script was fixed.
--- a/changelog.d/5843.misc
+++ b/changelog.d/5843.misc
@@ -1 +0,0 @@
-Whitelist history visbility sytests in worker mode tests.
--- a/changelog.d/5844.misc
+++ b/changelog.d/5844.misc
@@ -0,0 +1 @@
+Retry well-known lookup before the cache expires, giving a grace period where the remote well-known can be down but we still use the old result.
--- a/changelog.d/5856.feature
+++ b/changelog.d/5856.feature
@@ -0,0 +1 @@
+Add a tag recording a request's authenticated entity and corresponding servlet in opentracing.
--- a/changelog.d/5857.bugfix
+++ b/changelog.d/5857.bugfix
@@ -0,0 +1 @@
+Fix database index so that different backup versions can have the same sessions.
--- a/changelog.d/5860.misc
+++ b/changelog.d/5860.misc
@@ -0,0 +1 @@
+Remove log line for debugging issue #5407.
--- a/changelog.d/5863.bugfix
+++ b/changelog.d/5863.bugfix
@@ -0,0 +1 @@
+Fix Synapse looking for config options `password_reset_failure_template` and `password_reset_success_template`, when they are actually `password_reset_template_failure_html`, `password_reset_template_success_html`.
--- a/changelog.d/5878.feature
+++ b/changelog.d/5878.feature
@@ -0,0 +1 @@
+Add admin API endpoint for setting whether or not a user is a server administrator.
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,8 +1,18 @@
-matrix-synapse-py3 (1.2.1) stable; urgency=medium
+matrix-synapse-py3 (1.3.1) stable; urgency=medium

-  * New synapse release 1.2.1.
+  * New synapse release 1.3.1.

- -- Synapse Packaging team <packages@matrix.org>  Fri, 26 Jul 2019 11:32:47 +0100
+ -- Synapse Packaging team <packages@matrix.org>  Sat, 17 Aug 2019 09:15:49 +0100
+
+matrix-synapse-py3 (1.3.0) stable; urgency=medium
+
+  [ Andrew Morgan ]
+  * Remove libsqlite3-dev from required build dependencies.
+
+  [ Synapse Packaging team ]
+  * New synapse release 1.3.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Thu, 15 Aug 2019 12:04:23 +0100

 matrix-synapse-py3 (1.2.0) stable; urgency=medium

--- a/debian/control
+++ b/debian/control
@@ -15,7 +15,6 @@ Build-Depends:
 python3-setuptools,
 python3-pip,
 python3-venv,
- libsqlite3-dev,
 tar,
 Standards-Version: 3.9.8
 Homepage: https://github.com/matrix-org/synapse
--- a/docs/admin_api/user_admin_api.rst
+++ b/docs/admin_api/user_admin_api.rst
@@ -84,3 +84,23 @@ with a body of:
   }

 including an ``access_token`` of a server admin.
+
+
+Change whether a user is a server administrator or not
+======================================================
+
+Note that you cannot demote yourself.
+
+The api is::
+
+    PUT /_synapse/admin/v1/users/<user_id>/admin
+
+with a body of:
+
+.. code:: json
+
+    {
+        "admin": true
+    }
+
+including an ``access_token`` of a server admin.
--- a/docs/federation_side_bus.md
+++ b/docs/federation_side_bus.md
@@ -1,332 +0,0 @@
-%%%
-title = "The Federation Side Bus"
-abbrev = "federation-side-bus"
-docName = "federation-side-bus"
-ipr = "none"
-workgroup = "Synapse"
-
-[seriesInfo]
-name = "RFC"
-stream = "IETF"
-status = "informational"
-value = "federation-side-bus"
-
-[pi]
-toc = "yes"
-topblock = "yes"
-
-[[author]]
-initials = "A."
-surname = "Brown"
-fullname = "Amber Brown"
-organization = "New Vector"
-    [author.address]
-    email = "amberb@matrix.org"
-%%%
-
-.# Abstract
-
-Proposal for the "Federation Side Bus" project. Proposed refactoring of federation transport code as well as externally communicating code. Proposed implementation of a message-bus style system for external communication. Proposed implementation of a prioritisation system covering different remote hosts based on liveliness as well as prioritisation of outgoing requests when experiencing backpressure.
-
-{mainmatter}
-
-# Introduction
-
-On smaller machines, Synapse has problems when interacting with the federation in large rooms. Existing experience had pointed at state resolution being the performance killer, but further research with small homeservers has revealed the performance problems when communicating with many servers. The linear characteristics of having more servers in federation turns into a significant cliff in the realm of 200 or more servers on low-powered hardware, causing a "meltdown" and causing cascading failures as the server's non-responsiveness causes timeouts to clients and other servers.
-
-## Terminology
-
-The keywords **MUST**, **MUST NOT**, **REQUIRED**, **SHALL**, **SHALL NOT**, **SHOULD**, **SHOULD NOT**, **RECOMMENDED**, **MAY**, and **OPTIONAL**, when they appear in this document, are to be interpreted as described in [@!RFC2119].
-
-Additionally, the key words "**MIGHT**", "**COULD**", "**MAY WISH TO**", "**WOULD PROBABLY**", "**SHOULD CONSIDER**", and "**MUST (BUT WE KNOW YOU WON'T)**" in this document are to interpreted as described in [@!RFC6919].
-
-The keywords **PDU**, **EDU**, and **QUERY** in this document are to be interpreted as described in the Matrix Server to Server Specification [@!s2sapi].
-
-**EXTERNAL COMMUNICATION** is defined as outgoing communication with another logical service, such as a web server or chat bot. Communication with the configured database, the filesystem, or with workers is not included in this definition.
-
-**FEDERATION REQUESTS** are defined as any HTTP API call in the Matrix Server-to-Server specification, including PDUs, EDUs, or queries.
-
-**DEFERRED** can mean either the literal Twisted Deferred, or a native coroutine that can await Deferreds. Which is used in the code depends on its use of native coroutines. APIs SHOULD try and implement native coroutines where possible, but they are described as "Deferreds" for brevity.
-
-# The Status Quo
-
-Synapse currently performs poorly under the following situations:
-
- Joining a room with many servers, where the presence storm can cause Synapse to lock up and time out the room join to the client, making it seem as if joining the room "failed" when it did not
- Sending a message in a room with many servers (sending PDUs) causes CPU and RAM spikes
- Presence and typing in a room with many servers (sending EDUs) causes CPU and RAM spikes
- Viewing the user list of a large room, causing many concurrent profile fetches
- Many users doing queries about remote users
-
-This can be attributed to the following fundamental issues:
-
- Synapse's use of the network is unintelligent and is not aware of resource constraints (e.g. connection limits),
- Synapse does not leverage persistent network connections and pipelining/HTTP 2.0,
- Synapse does not gracefully degrade under pressure situations, but shows total system failure.
-
-In addition, the following issues make it more difficult to fix the above without a comprehensive approach:
-
- Synapse does not assign a priority to hosts, meaning that any naive rate limiting (such as the transaction queue on `atleastfornow.net`) can cause a poor user experience as misbehaving/timing out hosts can take up a slot that a well-behaved server or servers with users that are being actively communicated with could use
- Synapse does not assign a priority to requests, making a rudimentary rate limiting system difficult
- Large parts of the codebase can make external requests independently
- Synapse uses conventional HTTP clients that are poorly suited to the "message bus" style of usage that is required.
-
-For large servers with workers, this can be mitigated somewhat by just throwing more hardware at the problem. For smaller ones, especially on constrained hardware (think ARM or shared hosting), this lack of rate limiting can cause hard spinning, swamping of resources, and total system failure.
-
-## The Status Quo
-
-Currently, Synapse talks over to other servers in the following places:
-
- Keyring (perspectives and origin), for fetching server keys
- TransportLayerClient:
-    - s.federation.federation_client
-        - general federation queries
-        - client key query
-        - user device query
-        - backfill
-        - individual PDU collection
-        - fetching remote room state
-        - sending joins
-        - sending invites
-        - sending leaves
-        - getting public rooms
-        - querying auth chain
-        - getting missing events
-        - room complexity
-    - s.federation.sender.transaction_manager
-        - sending transactions
-    - s.groups.attestations
-        - fetching attestations
-    - s.groups.groups_server
-        - inviting and removing from group
-    - s.groups.groups_client
-        - fetching groups
-        - fetching users from groups
- Media Repo
-    - download_remote_file (linearised)
- Appservices
- Identity services
-
-TODO: More detail?
-
-Furthermore, profiles and room directory use the general query API.
-
-# Proposing The Federation Side Bus
-
-The Federation Side Bus project remodels how Synapse approaches external communication. It draws naming parallels with the system bus design of personal computers and servers from the 1990s and 2000s, where the Front Side Bus was used to describe the communication interface between the CPU and its I/O systems (where the "back side bus" was instead between CPUs).
-
-The core of the proposal is the definition of the "southbridge" (named for the I/O controller hub on a computer's FSB). The Southbridge is the only place where external communication is allowed to occur, and has a small but versatile interface for invoking said communication. This abstraction allows the Southbridge to be more intelligent about the use of network resources, as it can control all outbound data.
-
-There are also additional abstractions and reworking of existing ones to make the internal logic more consistent. This is mostly focused on the reorganisation of the Federation code and the shifting of the Media Repo logic from being in REST servlets to handlers of their own. A reworking of ".well-known" resolution as well as hostname resolution in general is also proposed, with the end goal of increasing reliability and reducing the amount of code that needs to consider SRV/.well-known solving.
-
-The Federation Side Bus will not alter Synapse's interaction with any of the Matrix standards, but will present the foundation for the future implementation of transports other than HTTP. HTTP/1.1 over TLS is targeted as the primary transport for Federation for this proposal, although HTTP/2.0 can be considered a "stretch goal" and desirable for its multiplexing and long-concurrent-connection qualities that would further reduce resource usage.
-
-# Architecture
-
-## The Southbridge
-
-The Southbridge fully encapsulates all external communication (apart from DNS resolution). It consists of a number of queues, connection pools, and associated prioritisation and batching systems.
-
-### Initial Federation Queue
-
-Zero-length queue that routes Federation requests through to the host ranker.
-
-### Host Ranker
-
-Tracks the performance of outbound requests and routes new requests through the different queues based on Matrix host.
-
-### Priority-Aware Federation Queue
-
-A queue that enqueues events based on the Matrix host and requests a connection from the pool. When it has acquired a connection, it sends the events it has. If there is network pressure, the queue is responsible for giving up the connection based on a deadline. It is aware of federation semantics, and can intelligently collapse or discard EDUs or queries.
-
-### The Request Queue
-
-Holds HTTP requests and requests a connection from the connection pool to send them on. Used for general purpose queries (for example, .well-known lookups or URL previews).
-
-### The Connection Pool
-
-Holds open HTTP connections and is responsible for establishing new ones. Operates on a callback basis with the queues. Hands over a connection to the queue requesting it, and is told when the queue is done with it. Assigns deadlines for the queues to follow (e.g. time spent processing) to ensure fairness.
-
-## Federation Subsystem
-
-The Federation Subsystem sees a number of changes, mostly revolving around refactoring the existing code and formalising interfaces.
-
-### Federation Resolver
-
-Translates a Matrix homeserver hostname into "real" addresses that it can be contacted on. It is considered authoritive to the rest of the system.
-
-### Externaliser
-
-Queues a Federation request in the Message Queue after attaching the "real address" information.
-
-### Controllers
-
-Shifting of Federation logic into more logically separated modules, such as separating by purpose (messages, queries, presence, etc) for clarity.
-
-## Media Subsystem
-
-The functionality of the Media Repository REST APIs refactored into a handler.
-
-## DNS Resolver
-
-Resolves domain names to DNS records. Although informally implemented in Synapse, this new subsystem would centralise a lot of the functionality of the various DNS resolvers used.
-
-# Implementation Plan
-
-The implementation plan has three phases -- cleanup, plumbing, and optimising.
-
-Cleanup focuses on shifting about existing code to fit the new model better. This involves implementing the Federation Resolver and cleaning up the media APIs.
-
-Plumbing involves laying the groundwork for the changes. This involves writing a more controllable HTTP client, implementing the queueing and connection pool, and hooking it up to the existing Federation abstraction. The development of other queues and pools (like for URL previews, well-known lookups, etc) will also be done here, although can be done concurrently.
-
-Optimising involves using these abstractions to allow Synapse to operate with network activity restrictions. This includes adding rate limiting, EDU collapsing,
-
-## Decouple the Media APIs from the REST APIs
-
-This should all be moved out into a handler of its own, instead of existing in the REST APIs.
-
-## Implement the Federation Resolver
-
-The base of the Federation Resolver can be implemented and placed in Synapse without much disruption.
-
-Requirements:
-
- A ResolvedFederationAddress object which can encapsulate the results.
- Simple, one-function-call API to fetch the information about the "real host".
-    - Inputs:
-        - Matrix server name.
-    - Outputs:
-        - Hostname to verify the TLS certificate against (which might not be the Matrix server name if .well-known is in use)
-        - A list of IP addresses to contact the Matrix service by. This SHOULD contain priority and weight data to allow the connection pool to connect to preferred hosts, but MAY just be ordered in rank of preference without any priority or weight information.
-        - MUST be encapsulated in a ResolvedFederationAddress object.
-
-## Implement the HTTP/1.1 Transport
-
-Implement in the current MatrixFederationAgent and SimpleHTTPClient, with a basic connection pool.
-
-The justification for this is that the current HTTP client libraries rely on controlling the connection itself, while we want to operate on a lower level and control the connection ourselves, and give it to the client instead. It represents an inversion of the concerns, which is why we have to provide this part ourselves.
-
-This is not a large asking, as the h11 library implements all the logic (and is a much more solid HTTP state machine than Twisted's current HTTP Agent implementation). If it implements IAgent, we may wish to contribute this up to Twisted.
-
-Requirements:
-
- A HTTP/1.1 compliant transport.
-    - SHOULD utilise the h11 library.
-    - MUST support HTTP/1.1 keep-alive, but MUST NOT send multiple requests at once (pipelining).
-    - SHOULD implement Twisted's IAgent/IResponse interface.
-    - MUST take a TCP connection as an argument. The client MUST NOT instantiate the connection itself.
- A basic connection pool.
-    - MUST implement a method to request a connection from the ResolvedFederationAddress object that returns a Deferred resolving to the TCP connection.
-    - MAY use the first IP listed in the ResolvedFederationAddress (matching current behaviour).
-    - MUST verify the TLS matches the hostname in the ResolvedFederationAddress when the connection is made.
-    - MUST return an error to the connection requests if the TLS connection fails.
-    - SHOULD keep connections around until they time out, and serve them to subsequent requests if they are alive.
-    - SHOULD NOT implement any form of rate limiting, as that will be implemented later.
- MatrixFederationClient MUST use this connection pool and transport in place of treq.
- MatrixFederationClient MUST query the Federation Resolver for the ResolvedFederationAddress to use.
- Users of the MatrixFederationClient MUST NOT call the FederationResolver before making the request.
-
-Questions:
-
- Do we need to support HTTP/1.0?
-    - I don't think it's realistically required, and is expensive. The specification lists "HTTP/1.1" specifically in the examples, but does not call out HTTP/1.1 as the minimum supported version.
-
-## Implement the Federation Queue
-
-Implement the Federation Queue API. This Queue is not used at this stage.
-
-Requirements:
-
- FederationResponse object
-    - MUST be the root interface for the purposes of typing.
-    - MUST have a common "status code" attribute with the numerical code and description.
- FederationErrorResponse object
-    - MUST implement FederationResponse
-    - MUST have errcode and error from the JSON body as attributes, and all other keys in an 'other' mapping.
- FederationQueryResponse object
-    - MUST implement FederationResponse
-    - MAY have further subclasses that implement particular responses to queries.
-    - MUST have the JSON response as an attribute.
- FederationTransactionResponse object
-    - MUST implement FederationResponse
-    - MUST have the PDU processing results as an attribute.
-
- OutgoingEDU object
-    - MUST have edu_type and content as attributes.
-    - MUST have the time that it was created.
- OutgoingPDU object
-    - MUST have a content attribute which contains the PDU data.
- OutgoingQuery object
-    - MUST have a template of the path.
-    - MUST NOT add query or body parameters to the path.
-    - MUST store the path, query, and JSON body arguments.
-    - MUST implement a method that returns the fully resolved path with query arguments and the body as a dictionary, for consumption by the Queue.
-    - MAY have subclasses that create more usable instantiators based on the particular query.
-
- The base FederationQueue
-    - MUST request a connection from the ConnectionPool to send requests.
-    - MUST return the connection to the Connection Pool when it has sent its requests.
-    - MUST NOT send more requests than were initially in its queue when the connection was granted from the Pool.
-    - MUST create a HTTP Transport for its uses. It MUST destroy it after the connection is returned.
-    - MUST be able to encode JSON bodies and create requests.
-    - MUST be able to create a transaction from the EDUs/PDUs in the queue when it has a connection.
-    - MAY collapse EDUs based on their time of creation or "cancelling out".
-    - MAY remove EDUs from the queue when under queue pressure.
-    - SHOULD send PDUs and EDUs in the order they were given. Future implementations MAY prioritise certain PDUs over others (e.g. direct messages).
-    - MUST remove EDUs/PDUs that have been sent in a transaction from the queue.
-    - MUST remove queries that have been given a response from the queue.
-    - MAY retry queries that fail with transient errors instead of delivering the real error to the querier.
-    - MUST remove queries from the queue that have passed their wall-clock timeout and return a FederationErrorResponse, even if they have not been sent.
- An API to add a EDU/PDU onto the Queue.
-    - MUST require a ResolvedFederationAddress.
-    - MUST take a OutgoingPDU or OutgoingEDU object.
- An API to make a Federation query.
-    - MUST require a ResolvedFederationAddress.
-    - MUST list an acceptable timeout. This MAY be 0 to mean that the query should be retried forever.
-    - MUST return a Deferred that fires with a FederationResponse.
-
-Questions:
-
- Typing on interfaces -- there's a mypy zope.interface plugin?
- What to do with backpressure on down hosts? Do we discard the queue?
-
-## Handle Transactions and Queries via the Federation Queue
-
-Move the FederationSender code to use the Federation Queue.
-
-Requirements:
-
- Externaliser
-    - Takes Queries/EDUs/PDUs and queries the Federation Resolver for the real host information, and then forwards it to the queue.
- Synapse MUST instantiate the Externaliser, Federation Queue, and the Federation Connection Pool on startup.
- synapse.federation.sender.FederationSender MUST send events to the Externaliser.
- ... more words here...
-
-## Implement the General Purpose Queue
-
-A Queue that takes general HTTP requests and forwards them to a pool.
-
-## Handle General Purpose External Communication Via the General Purpose Queue
-
-Move the URL previewer, well-known lookup to use the General Purpose Queue
-
-## Implement Queuing and Pooling for Pushers, Appservices, and Identity Servers
-
-Questions:
-
- Is this really needed? Fitting with the existing abstraction is useful, even if it will never rate limit the pool, and we'll get the benefits of the smarter connection pooling
-
-## TODO: Lay out the optimising section
-
-{backmatter}
-
-
-<reference anchor='s2sapi' target='https://matrix.org/docs/spec/server_server/latest'>
-    <front>
-        <title>Federation API</title>
-        <author>
-            <organization>Matrix.org Foundation C.I.C.</organization>
-        </author>
-        <date year='2019'/>
-    </front>
-</reference>
--- a/synapse/init.py
+++ b/synapse/init.py
@@ -35,4 +35,4 @@ try:
 except ImportError:
    pass

-__version__ = "1.2.1"
+__version__ = "1.3.1"
--- a/synapse/api/auth.py
+++ b/synapse/api/auth.py
@@ -22,6 +22,7 @@ from netaddr import IPAddress

 from twisted.internet import defer

+import synapse.logging.opentracing as opentracing
 import synapse.types
 from synapse import event_auth
 from synapse.api.constants import EventTypes, JoinRules, Membership
@@ -178,6 +179,7 @@ class Auth(object):
    def get_public_keys(self, invite_event):
        return event_auth.get_public_keys(invite_event)

+    @opentracing.trace
    @defer.inlineCallbacks
    def get_user_by_req(
        self, request, allow_guest=False, rights="access", allow_expired=False
@@ -209,6 +211,7 @@ class Auth(object):
            user_id, app_service = yield self._get_appservice_user_id(request)
            if user_id:
                request.authenticated_entity = user_id
+                opentracing.set_tag("authenticated_entity", user_id)

                if ip_addr and self.hs.config.track_appservice_user_ips:
                    yield self.store.insert_client_ip(
@@ -259,6 +262,7 @@ class Auth(object):
                )

            request.authenticated_entity = user.to_string()
+            opentracing.set_tag("authenticated_entity", user.to_string())

            return synapse.types.create_requester(
                user, token_id, is_guest, device_id, app_service=app_service
--- a/synapse/app/_base.py
+++ b/synapse/app/_base.py
@@ -17,10 +17,10 @@ import gc
 import logging
 import os
 import signal
+import socket
 import sys
 import traceback

-import sdnotify
 from daemonize import Daemonize

 from twisted.internet import defer, error, reactor
@@ -246,13 +246,12 @@ def start(hs, listeners=None):
            def handle_sighup(*args, **kwargs):
                # Tell systemd our state, if we're using it. This will silently fail if
                # we're not using systemd.
-                sd_channel = sdnotify.SystemdNotifier()
-                sd_channel.notify("RELOADING=1")
+                sdnotify(b"RELOADING=1")

                for i in _sighup_callbacks:
                    i(hs)

-                sd_channel.notify("READY=1")
+                sdnotify(b"READY=1")

            signal.signal(signal.SIGHUP, handle_sighup)

@@ -308,16 +307,12 @@ def setup_sdnotify(hs):

    # Tell systemd our state, if we're using it. This will silently fail if
    # we're not using systemd.
-    sd_channel = sdnotify.SystemdNotifier()
-
    hs.get_reactor().addSystemEventTrigger(
-        "after",
-        "startup",
-        lambda: sd_channel.notify("READY=1\nMAINPID=%s" % (os.getpid())),
+        "after", "startup", sdnotify, b"READY=1\nMAINPID=%i" % (os.getpid(),)
    )

    hs.get_reactor().addSystemEventTrigger(
-        "before", "shutdown", lambda: sd_channel.notify("STOPPING=1")
+        "before", "shutdown", sdnotify, b"STOPPING=1"
    )


@@ -414,3 +409,35 @@ class _DeferredResolutionReceiver(object):
    def resolutionComplete(self):
        self._deferred.callback(())
        self._receiver.resolutionComplete()
+
+
+sdnotify_sockaddr = os.getenv("NOTIFY_SOCKET")
+
+
+def sdnotify(state):
+    """
+    Send a notification to systemd, if the NOTIFY_SOCKET env var is set.
+
+    This function is based on the sdnotify python package, but since it's only a few
+    lines of code, it's easier to duplicate it here than to add a dependency on a
+    package which many OSes don't include as a matter of principle.
+
+    Args:
+        state (bytes): notification to send
+    """
+    if not isinstance(state, bytes):
+        raise TypeError("sdnotify should be called with a bytes")
+    if not sdnotify_sockaddr:
+        return
+    addr = sdnotify_sockaddr
+    if addr[0] == "@":
+        addr = "\0" + addr[1:]
+
+    try:
+        with socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) as sock:
+            sock.connect(addr)
+            sock.sendall(state)
+    except Exception as e:
+        # this is a bit surprising, since we don't expect to have a NOTIFY_SOCKET
+        # unless systemd is expecting us to notify it.
+        logger.warning("Unable to send notification to systemd: %s", e)
--- a/synapse/app/homeserver.py
+++ b/synapse/app/homeserver.py
@@ -447,7 +447,7 @@ def setup(config_options):
                reactor.stop()
            sys.exit(1)

-    reactor.addSystemEventTrigger("before", "startup", start)
+    reactor.callWhenRunning(start)

    return hs

--- a/synapse/config/emailconfig.py
+++ b/synapse/config/emailconfig.py
@@ -132,21 +132,21 @@ class EmailConfig(Config):
            self.email_password_reset_template_text = email_config.get(
                "password_reset_template_text", "password_reset.txt"
            )
-            self.email_password_reset_failure_template = email_config.get(
-                "password_reset_failure_template", "password_reset_failure.html"
+            self.email_password_reset_template_failure_html = email_config.get(
+                "password_reset_template_failure_html", "password_reset_failure.html"
            )
            # This template does not support any replaceable variables, so we will
            # read it from the disk once during setup
-            email_password_reset_success_template = email_config.get(
-                "password_reset_success_template", "password_reset_success.html"
+            email_password_reset_template_success_html = email_config.get(
+                "password_reset_template_success_html", "password_reset_success.html"
            )

            # Check templates exist
            for f in [
                self.email_password_reset_template_html,
                self.email_password_reset_template_text,
-                self.email_password_reset_failure_template,
-                email_password_reset_success_template,
+                self.email_password_reset_template_failure_html,
+                email_password_reset_template_success_html,
            ]:
                p = os.path.join(self.email_template_dir, f)
                if not os.path.isfile(p):
@@ -154,9 +154,9 @@ class EmailConfig(Config):

            # Retrieve content of web templates
            filepath = os.path.join(
-                self.email_template_dir, email_password_reset_success_template
+                self.email_template_dir, email_password_reset_template_success_html
            )
-            self.email_password_reset_success_html_content = self.read_file(
+            self.email_password_reset_template_success_html_content = self.read_file(
                filepath, "email.password_reset_template_success_html"
            )

--- a/synapse/config/repository.py
+++ b/synapse/config/repository.py
@@ -93,7 +93,7 @@ class ContentRepositoryConfig(Config):
        # current worker app is the media repo.
        if (
            self.enable_media_repo is False
-            and config.worker_app != "synapse.app.media_repository"
+            and config.get("worker_app") != "synapse.app.media_repository"
        ):
            self.can_load_media_repo = False
            return
--- a/synapse/federation/transport/server.py
+++ b/synapse/federation/transport/server.py
@@ -19,8 +19,9 @@ import functools
 import logging
 import re

+from twisted.internet.defer import maybeDeferred
+
 import synapse
-import synapse.logging.opentracing as opentracing
 from synapse.api.errors import Codes, FederationDeniedError, SynapseError
 from synapse.api.room_versions import RoomVersions
 from synapse.api.urls import (
@@ -37,6 +38,7 @@ from synapse.http.servlet import (
    parse_string_from_args,
 )
 from synapse.logging.context import run_in_background
+from synapse.logging.opentracing import start_active_span_from_context, tags
 from synapse.types import ThirdPartyInstanceID, get_domain_from_id
 from synapse.util.ratelimitutils import FederationRateLimiter
 from synapse.util.versionstring import get_version_string
@@ -287,16 +289,17 @@ class BaseFederationServlet(object):
                raise

            # Start an opentracing span
-            with opentracing.start_active_span_from_context(
+            with start_active_span_from_context(
                request.requestHeaders,
                "incoming-federation-request",
                tags={
                    "request_id": request.get_request_id(),
-                    opentracing.tags.SPAN_KIND: opentracing.tags.SPAN_KIND_RPC_SERVER,
-                    opentracing.tags.HTTP_METHOD: request.get_method(),
-                    opentracing.tags.HTTP_URL: request.get_redacted_uri(),
-                    opentracing.tags.PEER_HOST_IPV6: request.getClientIP(),
+                    tags.SPAN_KIND: tags.SPAN_KIND_RPC_SERVER,
+                    tags.HTTP_METHOD: request.get_method(),
+                    tags.HTTP_URL: request.get_redacted_uri(),
+                    tags.PEER_HOST_IPV6: request.getClientIP(),
                    "authenticated_entity": origin,
+                    "servlet_name": request.request_metrics.name,
                },
            ):
                if origin:
@@ -745,8 +748,12 @@ class PublicRoomList(BaseFederationServlet):
        else:
            network_tuple = ThirdPartyInstanceID(None, None)

-        data = await self.handler.get_local_public_room_list(
-            limit, since_token, network_tuple=network_tuple, from_federation=True
+        data = await maybeDeferred(
+            self.handler.get_local_public_room_list,
+            limit,
+            since_token,
+            network_tuple=network_tuple,
+            from_federation=True,
        )
        return 200, data

--- a/synapse/handlers/admin.py
+++ b/synapse/handlers/admin.py
@@ -94,6 +94,16 @@ class AdminHandler(BaseHandler):

        return ret

+    def set_user_server_admin(self, user, admin):
+        """
+        Set the admin bit on a user.
+
+        Args:
+            user_id (UserID): the (necessarily local) user to manipulate
+            admin (bool): whether or not the user should be an admin of this server
+        """
+        return self.store.set_server_admin(user, admin)
+
    @defer.inlineCallbacks
    def export_user_data(self, user_id, writer):
        """Write all data we have on the user to the given writer.
--- a/synapse/handlers/room.py
+++ b/synapse/handlers/room.py
@@ -560,6 +560,18 @@ class RoomCreationHandler(BaseHandler):

        yield self.event_creation_handler.assert_accepted_privacy_policy(requester)

+        power_level_content_override = config.get("power_level_content_override")
+        if (
+            power_level_content_override
+            and "users" in power_level_content_override
+            and user_id not in power_level_content_override["users"]
+        ):
+            raise SynapseError(
+                400,
+                "Not a valid power_level_content_override: 'users' did not contain %s"
+                % (user_id,),
+            )
+
        invite_3pid_list = config.get("invite_3pid", [])

        visibility = config.get("visibility", None)
@@ -604,7 +616,7 @@ class RoomCreationHandler(BaseHandler):
            initial_state=initial_state,
            creation_content=creation_content,
            room_alias=room_alias,
-            power_level_content_override=config.get("power_level_content_override"),
+            power_level_content_override=power_level_content_override,
            creator_join_profile=creator_join_profile,
        )

--- a/synapse/handlers/sync.py
+++ b/synapse/handlers/sync.py
@@ -786,9 +786,8 @@ class SyncHandler(object):
                        batch.events[0].event_id, state_filter=state_filter
                    )
                else:
-                    # Its not clear how we get here, but empirically we do
-                    # (#5407). Logging has been added elsewhere to try and
-                    # figure out where this state comes from.
+                    # We can get here if the user has ignored the senders of all
+                    # the recent events.
                    state_at_timeline_start = yield self.get_state_at(
                        room_id, stream_position=now_token, state_filter=state_filter
                    )
@@ -1771,20 +1770,9 @@ class SyncHandler(object):
            newly_joined_room=newly_joined,
        )

-        if not batch and batch.limited:
-            # This resulted in #5407, which is weird, so lets log! We do it
-            # here as we have the maximum amount of information.
-            user_id = sync_result_builder.sync_config.user.to_string()
-            logger.info(
-                "Issue #5407: Found limited batch with no events. user %s, room %s,"
-                " sync_config %s, newly_joined %s, events %s, batch %s.",
-                user_id,
-                room_id,
-                sync_config,
-                newly_joined,
-                events,
-                batch,
-            )
+        # Note: `batch` can be both empty and limited here in the case where
+        # `_load_filtered_recents` can't find any events the user should see
+        # (e.g. due to having ignored the sender of the last 50 events).

        if newly_joined:
            # debug for https://github.com/matrix-org/synapse/issues/4422
--- a/synapse/http/federation/well_known_resolver.py
+++ b/synapse/http/federation/well_known_resolver.py
@@ -44,6 +44,12 @@ WELL_KNOWN_MAX_CACHE_PERIOD = 48 * 3600
 # lower bound for .well-known cache period
 WELL_KNOWN_MIN_CACHE_PERIOD = 5 * 60

+# Attempt to refetch a cached well-known N% of the TTL before it expires.
+# e.g. if set to 0.2 and we have a cached entry with a TTL of 5mins, then
+# we'll start trying to refetch 1 minute before it expires.
+WELL_KNOWN_GRACE_PERIOD_FACTOR = 0.2
+
+
 logger = logging.getLogger(__name__)


@@ -80,15 +86,38 @@ class WellKnownResolver(object):
            Deferred[WellKnownLookupResult]: The result of the lookup
        """
        try:
-            result = self._well_known_cache[server_name]
+            prev_result, expiry, ttl = self._well_known_cache.get_with_expiry(
+                server_name
+            )
+
+            now = self._clock.time()
+            if now < expiry - WELL_KNOWN_GRACE_PERIOD_FACTOR * ttl:
+                return WellKnownLookupResult(delegated_server=prev_result)
        except KeyError:
-            # TODO: should we linearise so that we don't end up doing two .well-known
-            # requests for the same server in parallel?
+            prev_result = None
+
+        # TODO: should we linearise so that we don't end up doing two .well-known
+        # requests for the same server in parallel?
+        try:
            with Measure(self._clock, "get_well_known"):
                result, cache_period = yield self._do_get_well_known(server_name)

-            if cache_period > 0:
-                self._well_known_cache.set(server_name, result, cache_period)
+        except _FetchWellKnownFailure as e:
+            if prev_result and e.temporary:
+                # This is a temporary failure and we have a still valid cached
+                # result, so lets return that. Hopefully the next time we ask
+                # the remote will be back up again.
+                return WellKnownLookupResult(delegated_server=prev_result)
+
+            result = None
+
+            # add some randomness to the TTL to avoid a stampeding herd every hour
+            # after startup
+            cache_period = WELL_KNOWN_INVALID_CACHE_PERIOD
+            cache_period += random.uniform(0, WELL_KNOWN_DEFAULT_CACHE_PERIOD_JITTER)
+
+        if cache_period > 0:
+            self._well_known_cache.set(server_name, result, cache_period)

        return WellKnownLookupResult(delegated_server=result)

@@ -99,40 +128,42 @@ class WellKnownResolver(object):
        Args:
            server_name (bytes): name of the server, from the requested url

+        Raises:
+            _FetchWellKnownFailure if we fail to lookup a result
+
        Returns:
-            Deferred[Tuple[bytes|None|object],int]:
-                result, cache period, where result is one of:
-                 - the new server name from the .well-known (as a `bytes`)
-                 - None if there was no .well-known file.
-                 - INVALID_WELL_KNOWN if the .well-known was invalid
+            Deferred[Tuple[bytes,int]]: The lookup result and cache period.
        """
        uri = b"https://%s/.well-known/matrix/server" % (server_name,)
        uri_str = uri.decode("ascii")
        logger.info("Fetching %s", uri_str)
+
+        # We do this in two steps to differentiate between possibly transient
+        # errors (e.g. can't connect to host, 503 response) and more permenant
+        # errors (such as getting a 404 response).
        try:
            response = yield make_deferred_yieldable(
                self._well_known_agent.request(b"GET", uri)
            )
            body = yield make_deferred_yieldable(readBody(response))
+
+            if 500 <= response.code < 600:
+                raise Exception("Non-200 response %s" % (response.code,))
+        except Exception as e:
+            logger.info("Error fetching %s: %s", uri_str, e)
+            raise _FetchWellKnownFailure(temporary=True)
+
+        try:
            if response.code != 200:
                raise Exception("Non-200 response %s" % (response.code,))

            parsed_body = json.loads(body.decode("utf-8"))
            logger.info("Response from .well-known: %s", parsed_body)
-            if not isinstance(parsed_body, dict):
-                raise Exception("not a dict")
-            if "m.server" not in parsed_body:
-                raise Exception("Missing key 'm.server'")
+
+            result = parsed_body["m.server"].encode("ascii")
        except Exception as e:
            logger.info("Error fetching %s: %s", uri_str, e)
-
-            # add some randomness to the TTL to avoid a stampeding herd every hour
-            # after startup
-            cache_period = WELL_KNOWN_INVALID_CACHE_PERIOD
-            cache_period += random.uniform(0, WELL_KNOWN_DEFAULT_CACHE_PERIOD_JITTER)
-            return (None, cache_period)
-
-        result = parsed_body["m.server"].encode("ascii")
+            raise _FetchWellKnownFailure(temporary=False)

        cache_period = _cache_period_from_headers(
            response.headers, time_now=self._reactor.seconds
@@ -185,3 +216,10 @@ def _parse_cache_control(headers):
            v = splits[1] if len(splits) > 1 else None
            cache_controls[k] = v
    return cache_controls
+
+
+@attr.s()
+class _FetchWellKnownFailure(Exception):
+    # True if we didn't get a non-5xx HTTP response, i.e. this may or may not be
+    # a temporary failure.
+    temporary = attr.ib()
--- a/synapse/http/matrixfederationclient.py
+++ b/synapse/http/matrixfederationclient.py
@@ -36,7 +36,6 @@ from twisted.internet.task import _EPSILON, Cooperator
 from twisted.web._newclient import ResponseDone
 from twisted.web.http_headers import Headers

-import synapse.logging.opentracing as opentracing
 import synapse.metrics
 import synapse.util.retryutils
 from synapse.api.errors import (
@@ -50,6 +49,12 @@ from synapse.http import QuieterFileBodyProducer
 from synapse.http.client import BlacklistingAgentWrapper, IPBlacklistingResolver
 from synapse.http.federation.matrix_federation_agent import MatrixFederationAgent
 from synapse.logging.context import make_deferred_yieldable
+from synapse.logging.opentracing import (
+    inject_active_span_byte_dict,
+    set_tag,
+    start_active_span,
+    tags,
+)
 from synapse.util.async_helpers import timeout_deferred
 from synapse.util.metrics import Measure

@@ -341,20 +346,20 @@ class MatrixFederationHttpClient(object):
            query_bytes = b""

        # Retreive current span
-        scope = opentracing.start_active_span(
+        scope = start_active_span(
            "outgoing-federation-request",
            tags={
-                opentracing.tags.SPAN_KIND: opentracing.tags.SPAN_KIND_RPC_CLIENT,
-                opentracing.tags.PEER_ADDRESS: request.destination,
-                opentracing.tags.HTTP_METHOD: request.method,
-                opentracing.tags.HTTP_URL: request.path,
+                tags.SPAN_KIND: tags.SPAN_KIND_RPC_CLIENT,
+                tags.PEER_ADDRESS: request.destination,
+                tags.HTTP_METHOD: request.method,
+                tags.HTTP_URL: request.path,
            },
            finish_on_close=True,
        )

        # Inject the span into the headers
        headers_dict = {}
-        opentracing.inject_active_span_byte_dict(headers_dict, request.destination)
+        inject_active_span_byte_dict(headers_dict, request.destination)

        headers_dict[b"User-Agent"] = [self.version_string_bytes]

@@ -436,9 +441,7 @@ class MatrixFederationHttpClient(object):
                        response.phrase.decode("ascii", errors="replace"),
                    )

-                    opentracing.set_tag(
-                        opentracing.tags.HTTP_STATUS_CODE, response.code
-                    )
+                    set_tag(tags.HTTP_STATUS_CODE, response.code)

                    if 200 <= response.code < 300:
                        pass
--- a/synapse/push/emailpusher.py
+++ b/synapse/push/emailpusher.py
@@ -234,13 +234,19 @@ class EmailPusher(object):
            return

        self.last_stream_ordering = last_stream_ordering
-        yield self.store.update_pusher_last_stream_ordering_and_success(
-            self.app_id,
-            self.email,
-            self.user_id,
-            last_stream_ordering,
-            self.clock.time_msec(),
+        pusher_still_exists = (
+            yield self.store.update_pusher_last_stream_ordering_and_success(
+                self.app_id,
+                self.email,
+                self.user_id,
+                last_stream_ordering,
+                self.clock.time_msec(),
+            )
        )
+        if not pusher_still_exists:
+            # The pusher has been deleted while we were processing, so
+            # lets just stop and return.
+            self.on_stop()

    def seconds_until(self, ts_msec):
        secs = (ts_msec - self.clock.time_msec()) / 1000
--- a/synapse/push/httppusher.py
+++ b/synapse/push/httppusher.py
@@ -199,13 +199,21 @@ class HttpPusher(object):
                http_push_processed_counter.inc()
                self.backoff_delay = HttpPusher.INITIAL_BACKOFF_SEC
                self.last_stream_ordering = push_action["stream_ordering"]
-                yield self.store.update_pusher_last_stream_ordering_and_success(
-                    self.app_id,
-                    self.pushkey,
-                    self.user_id,
-                    self.last_stream_ordering,
-                    self.clock.time_msec(),
+                pusher_still_exists = (
+                    yield self.store.update_pusher_last_stream_ordering_and_success(
+                        self.app_id,
+                        self.pushkey,
+                        self.user_id,
+                        self.last_stream_ordering,
+                        self.clock.time_msec(),
+                    )
                )
+                if not pusher_still_exists:
+                    # The pusher has been deleted while we were processing, so
+                    # lets just stop and return.
+                    self.on_stop()
+                    return
+
                if self.failing_since:
                    self.failing_since = None
                    yield self.store.update_pusher_failing_since(
@@ -234,12 +242,17 @@ class HttpPusher(object):
                    )
                    self.backoff_delay = HttpPusher.INITIAL_BACKOFF_SEC
                    self.last_stream_ordering = push_action["stream_ordering"]
-                    yield self.store.update_pusher_last_stream_ordering(
+                    pusher_still_exists = yield self.store.update_pusher_last_stream_ordering(
                        self.app_id,
                        self.pushkey,
                        self.user_id,
                        self.last_stream_ordering,
                    )
+                    if not pusher_still_exists:
+                        # The pusher has been deleted while we were processing, so
+                        # lets just stop and return.
+                        self.on_stop()
+                        return

                    self.failing_since = None
                    yield self.store.update_pusher_failing_since(
--- a/synapse/python_dependencies.py
+++ b/synapse/python_dependencies.py
@@ -72,7 +72,6 @@ REQUIREMENTS = [
    "netaddr>=0.7.18",
    "Jinja2>=2.9",
    "bleach>=1.4.3",
-    "sdnotify>=0.3",
 ]

 CONDITIONAL_REQUIREMENTS = {
--- a/synapse/rest/admin/init.py
+++ b/synapse/rest/admin/init.py
@@ -43,6 +43,7 @@ from synapse.rest.admin._base import (
 )
 from synapse.rest.admin.media import register_servlets_for_media_repo
 from synapse.rest.admin.server_notice_servlet import SendServerNoticeServlet
+from synapse.rest.admin.users import UserAdminServlet
 from synapse.types import UserID, create_requester
 from synapse.util.versionstring import get_version_string

@@ -740,6 +741,7 @@ def register_servlets(hs, http_server):
    register_servlets_for_client_rest_resource(hs, http_server)
    SendServerNoticeServlet(hs).register(http_server)
    VersionServlet(hs).register(http_server)
+    UserAdminServlet(hs).register(http_server)


 def register_servlets_for_client_rest_resource(hs, http_server):
--- a/synapse/rest/admin/users.py
+++ b/synapse/rest/admin/users.py
@@ -0,0 +1,76 @@
+# -*- coding: utf-8 -*-
+# Copyright 2019 The Matrix.org Foundation C.I.C.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import re
+
+from twisted.internet import defer
+
+from synapse.api.errors import SynapseError
+from synapse.http.servlet import (
+    RestServlet,
+    assert_params_in_dict,
+    parse_json_object_from_request,
+)
+from synapse.rest.admin import assert_requester_is_admin
+from synapse.types import UserID
+
+
+class UserAdminServlet(RestServlet):
+    """
+    Set whether or not a user is a server administrator.
+
+    Note that only local users can be server administrators, and that an
+    administrator may not demote themselves.
+
+    Only server administrators can use this API.
+
+    Example:
+        PUT /_synapse/admin/v1/users/@reivilibre:librepush.net/admin
+        {
+            "admin": true
+        }
+    """
+
+    PATTERNS = (re.compile("^/_synapse/admin/v1/users/(?P<user_id>@[^/]*)/admin$"),)
+
+    def __init__(self, hs):
+        self.hs = hs
+        self.auth = hs.get_auth()
+        self.handlers = hs.get_handlers()
+
+    @defer.inlineCallbacks
+    def on_PUT(self, request, user_id):
+        yield assert_requester_is_admin(self.auth, request)
+        requester = yield self.auth.get_user_by_req(request)
+        auth_user = requester.user
+
+        target_user = UserID.from_string(user_id)
+
+        body = parse_json_object_from_request(request)
+
+        assert_params_in_dict(body, ["admin"])
+
+        if not self.hs.is_mine(target_user):
+            raise SynapseError(400, "Only local users can be admins of this homeserver")
+
+        set_admin_to = bool(body["admin"])
+
+        if target_user == auth_user and not set_admin_to:
+            raise SynapseError(400, "You may not demote yourself.")
+
+        yield self.handlers.admin_handler.set_user_server_admin(
+            target_user, set_admin_to
+        )
+
+        return (200, {})
--- a/synapse/rest/client/v2_alpha/account.py
+++ b/synapse/rest/client/v2_alpha/account.py
@@ -282,13 +282,13 @@ class PasswordResetSubmitTokenServlet(RestServlet):
                    return None

            # Otherwise show the success template
-            html = self.config.email_password_reset_success_html_content
+            html = self.config.email_password_reset_template_success_html_content
            request.setResponseCode(200)
        except ThreepidValidationError as e:
            # Show a failure page with a reason
            html = self.load_jinja2_template(
                self.config.email_template_dir,
-                self.config.email_password_reset_failure_template,
+                self.config.email_password_reset_template_failure_html,
                template_vars={"failure_reason": e.msg},
            )
            request.setResponseCode(e.code)
--- a/synapse/storage/e2e_room_keys.py
+++ b/synapse/storage/e2e_room_keys.py
@@ -82,11 +82,11 @@ class EndToEndRoomKeyStore(SQLBaseStore):
            table="e2e_room_keys",
            keyvalues={
                "user_id": user_id,
+                "version": version,
                "room_id": room_id,
                "session_id": session_id,
            },
            values={
-                "version": version,
                "first_message_index": room_key["first_message_index"],
                "forwarded_count": room_key["forwarded_count"],
                "is_verified": room_key["is_verified"],
--- a/synapse/storage/pusher.py
+++ b/synapse/storage/pusher.py
@@ -308,22 +308,36 @@ class PusherStore(PusherWorkerStore):
    def update_pusher_last_stream_ordering_and_success(
        self, app_id, pushkey, user_id, last_stream_ordering, last_success
    ):
-        yield self._simple_update_one(
-            "pushers",
-            {"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
-            {
+        """Update the last stream ordering position we've processed up to for
+        the given pusher.
+
+        Args:
+            app_id (str)
+            pushkey (str)
+            last_stream_ordering (int)
+            last_success (int)
+
+        Returns:
+            Deferred[bool]: True if the pusher still exists; False if it has been deleted.
+        """
+        updated = yield self._simple_update(
+            table="pushers",
+            keyvalues={"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
+            updatevalues={
                "last_stream_ordering": last_stream_ordering,
                "last_success": last_success,
            },
            desc="update_pusher_last_stream_ordering_and_success",
        )

+        return bool(updated)
+
    @defer.inlineCallbacks
    def update_pusher_failing_since(self, app_id, pushkey, user_id, failing_since):
-        yield self._simple_update_one(
-            "pushers",
-            {"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
-            {"failing_since": failing_since},
+        yield self._simple_update(
+            table="pushers",
+            keyvalues={"app_id": app_id, "pushkey": pushkey, "user_name": user_id},
+            updatevalues={"failing_since": failing_since},
            desc="update_pusher_failing_since",
        )

--- a/synapse/storage/registration.py
+++ b/synapse/storage/registration.py
@@ -272,6 +272,14 @@ class RegistrationWorkerStore(SQLBaseStore):

    @defer.inlineCallbacks
    def is_server_admin(self, user):
+        """Determines if a user is an admin of this homeserver.
+
+        Args:
+            user (UserID): user ID of the user to test
+
+        Returns (bool):
+            true iff the user is a server admin, false otherwise.
+        """
        res = yield self._simple_select_one_onecol(
            table="users",
            keyvalues={"name": user.to_string()},
@@ -282,6 +290,21 @@ class RegistrationWorkerStore(SQLBaseStore):

        return res if res else False

+    def set_server_admin(self, user, admin):
+        """Sets whether a user is an admin of this homeserver.
+
+        Args:
+            user (UserID): user ID of the user to test
+            admin (bool): true iff the user is to be a server admin,
+                false otherwise.
+        """
+        return self._simple_update_one(
+            table="users",
+            keyvalues={"name": user.to_string()},
+            updatevalues={"admin": 1 if admin else 0},
+            desc="set_server_admin",
+        )
+
    def _query_for_auth(self, txn, token):
        sql = (
            "SELECT users.name, users.is_guest, access_tokens.id as token_id,"
--- a/synapse/storage/schema/delta/56/fix_room_keys_index.sql
+++ b/synapse/storage/schema/delta/56/fix_room_keys_index.sql
@@ -0,0 +1,18 @@
+/* Copyright 2019 Matrix.org Foundation CIC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+-- version is supposed to be part of the room keys index
+CREATE UNIQUE INDEX e2e_room_keys_with_version_idx ON e2e_room_keys(user_id, version, room_id, session_id);
+DROP INDEX IF EXISTS e2e_room_keys_idx;
--- a/synapse/util/caches/ttlcache.py
+++ b/synapse/util/caches/ttlcache.py
@@ -55,7 +55,7 @@ class TTLCache(object):
        if e != SENTINEL:
            self._expiry_list.remove(e)

-        entry = _CacheEntry(expiry_time=expiry, key=key, value=value)
+        entry = _CacheEntry(expiry_time=expiry, ttl=ttl, key=key, value=value)
        self._data[key] = entry
        self._expiry_list.add(entry)

@@ -87,7 +87,8 @@ class TTLCache(object):
            key: key to look up

        Returns:
-            Tuple[Any, float]: the value from the cache, and the expiry time
+            Tuple[Any, float, float]: the value from the cache, the expiry time
+            and the TTL

        Raises:
            KeyError if the entry is not found
@@ -99,7 +100,7 @@ class TTLCache(object):
            self._metrics.inc_misses()
            raise
        self._metrics.inc_hits()
-        return e.value, e.expiry_time
+        return e.value, e.expiry_time, e.ttl

    def pop(self, key, default=SENTINEL):
        """Remove a value from the cache
@@ -158,5 +159,6 @@ class _CacheEntry(object):

    # expiry_time is the first attribute, so that entries are sorted by expiry.
    expiry_time = attr.ib()
+    ttl = attr.ib()
    key = attr.ib()
    value = attr.ib()
--- a/tests/http/federation/test_matrix_federation_agent.py
+++ b/tests/http/federation/test_matrix_federation_agent.py
@@ -987,6 +987,75 @@ class MatrixFederationAgentTests(TestCase):
        r = self.successResultOf(fetch_d)
        self.assertEqual(r.delegated_server, b"other-server")

+    def test_well_known_cache_with_temp_failure(self):
+        """Test that we refetch well-known before the cache expires, and that
+        it ignores transient errors.
+        """
+
+        well_known_resolver = WellKnownResolver(
+            self.reactor,
+            Agent(self.reactor, contextFactory=self.tls_factory),
+            well_known_cache=self.well_known_cache,
+        )
+
+        self.reactor.lookups["testserv"] = "1.2.3.4"
+
+        fetch_d = well_known_resolver.get_well_known(b"testserv")
+
+        # there should be an attempt to connect on port 443 for the .well-known
+        clients = self.reactor.tcpClients
+        self.assertEqual(len(clients), 1)
+        (host, port, client_factory, _timeout, _bindAddress) = clients.pop(0)
+        self.assertEqual(host, "1.2.3.4")
+        self.assertEqual(port, 443)
+
+        well_known_server = self._handle_well_known_connection(
+            client_factory,
+            expected_sni=b"testserv",
+            response_headers={b"Cache-Control": b"max-age=1000"},
+            content=b'{ "m.server": "target-server" }',
+        )
+
+        r = self.successResultOf(fetch_d)
+        self.assertEqual(r.delegated_server, b"target-server")
+
+        # close the tcp connection
+        well_known_server.loseConnection()
+
+        # Get close to the cache expiry, this will cause the resolver to do
+        # another lookup.
+        self.reactor.pump((900.0,))
+
+        fetch_d = well_known_resolver.get_well_known(b"testserv")
+        clients = self.reactor.tcpClients
+        (host, port, client_factory, _timeout, _bindAddress) = clients.pop(0)
+
+        # fonx the connection attempt, this will be treated as a temporary
+        # failure.
+        client_factory.clientConnectionFailed(None, Exception("nope"))
+
+        # attemptdelay on the hostnameendpoint is 0.3, so takes that long before the
+        # .well-known request fails.
+        self.reactor.pump((0.4,))
+
+        # Resolver should return cached value, despite the lookup failing.
+        r = self.successResultOf(fetch_d)
+        self.assertEqual(r.delegated_server, b"target-server")
+
+        # Expire the cache and repeat the request
+        self.reactor.pump((100.0,))
+
+        # Repated the request, this time it should fail if the lookup fails.
+        fetch_d = well_known_resolver.get_well_known(b"testserv")
+
+        clients = self.reactor.tcpClients
+        (host, port, client_factory, _timeout, _bindAddress) = clients.pop(0)
+        client_factory.clientConnectionFailed(None, Exception("nope"))
+        self.reactor.pump((0.4,))
+
+        r = self.successResultOf(fetch_d)
+        self.assertEqual(r.delegated_server, None)
+

 class TestCachePeriodFromHeaders(TestCase):
    def test_cache_control(self):
--- a/tests/util/caches/test_ttlcache.py
+++ b/tests/util/caches/test_ttlcache.py
@@ -36,7 +36,7 @@ class CacheTestCase(unittest.TestCase):
        self.assertTrue("one" in self.cache)
        self.assertEqual(self.cache.get("one"), "1")
        self.assertEqual(self.cache["one"], "1")
-        self.assertEqual(self.cache.get_with_expiry("one"), ("1", 110))
+        self.assertEqual(self.cache.get_with_expiry("one"), ("1", 110, 10))
        self.assertEqual(self.cache._metrics.hits, 3)
        self.assertEqual(self.cache._metrics.misses, 0)

@@ -77,7 +77,7 @@ class CacheTestCase(unittest.TestCase):
        self.assertEqual(self.cache["two"], "2")
        self.assertEqual(self.cache["three"], "3")

-        self.assertEqual(self.cache.get_with_expiry("two"), ("2", 120))
+        self.assertEqual(self.cache.get_with_expiry("two"), ("2", 120, 20))

        self.assertEqual(self.cache._metrics.hits, 5)
        self.assertEqual(self.cache._metrics.misses, 0)
Author	SHA1	Message	Date
Olivier Wilkinson (reivilibre)	6e827507f7	Return empty object on success rather than null. Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>	2019-08-20 10:29:36 +01:00
Olivier Wilkinson (reivilibre)	0e99412f4c	Add admin API docs for setting admin bits on users Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>	2019-08-20 10:24:38 +01:00
Olivier Wilkinson (reivilibre)	7fd0c90234	Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>	2019-08-19 14:44:53 +01:00
Olivier Wilkinson (reivilibre)	ebd2cd84d5	Add admin API for setting the admin bit of a user.	2019-08-19 14:42:55 +01:00
Olivier Wilkinson (reivilibre)	c497e13734	Introduce `set_server_admin` as dual to `is_server_admin`.	2019-08-19 14:41:07 +01:00
Erik Johnston	d514dac0b2	Merge pull request #5860 from matrix-org/erikj/update_5704_comments Remove logging for #5407 and update comments	2019-08-19 10:20:59 +01:00
Brendan Abolivier	bdd201ea7f	Merge branch 'master' into develop	2019-08-17 10:50:42 +01:00
Richard van der Hoff	74fb729213	1.3.1	2019-08-17 09:16:17 +01:00
Richard van der Hoff	412c6e21a8	Drop dependency on sdnotify (#5871 ) ... to save OSes which don't use it from having to maintain a port. Fixes #5865.	2019-08-17 09:09:52 +01:00
Hubert Chathi	8a5f6ed130	Merge pull request #5857 from matrix-org/uhoreg/fix_e2e_room_keys_index add the version field to the index for e2e_room_keys	2019-08-16 17:45:50 -07:00
Richard van der Hoff	c188bd2c12	add attribution	2019-08-16 23:19:23 +01:00
Chris Moos	20402aa128	Add changelog entry.	2019-08-16 22:16:21 +01:00
Chris Moos	6d86df73f1	Fix issue with Synapse not starting up. Fixes #5866 . Signed-off-by: Chris Moos <chris@chrismoos.com>	2019-08-16 22:16:13 +01:00
Jorik Schellekens	87fa26006b	Opentracing misc (#5856 ) Add authenticated_entity and servlet_names tags. Functionally: - Add a tag for authenticated_entity - Add a tag for servlet_names Stylistically: Moved to importing methods directly from opentracing.	2019-08-16 16:13:25 +01:00
Erik Johnston	ebba15ee7f	Newsfile	2019-08-16 13:29:41 +01:00
Hubert Chathi	e132ba79ae	fix changelog	2019-08-15 21:02:40 -07:00
Andrew Morgan	b13cac896d	Fix up password reset template config names (#5863 ) Fixes #5833 The emailconfig code was attempting to pull incorrect config file names. This corrects that, while also marking a difference between a config file variable that's a filepath versus a str containing HTML.	2019-08-15 16:27:11 +01:00
Brendan Abolivier	ce5f1cb98c	Merge branch 'master' into develop	2019-08-15 12:38:21 +01:00
Brendan Abolivier	6382914587	Merge tag 'v1.3.0' Synapse 1.3.0 (2019-08-15) ========================== Bugfixes -------- - Fix 500 Internal Server Error on `publicRooms` when the public room list was cached. ([\#5851](https://github.com/matrix-org/synapse/issues/5851)) Synapse 1.3.0rc1 (2019-08-13) ========================== Features -------- - Use `M_USER_DEACTIVATED` instead of `M_UNKNOWN` for errcode when a deactivated user attempts to login. ([\#5686](https://github.com/matrix-org/synapse/issues/5686)) - Add sd_notify hooks to ease systemd integration and allows usage of Type=Notify. ([\#5732](https://github.com/matrix-org/synapse/issues/5732)) - Synapse will no longer serve any media repo admin endpoints when `enable_media_repo` is set to False in the configuration. If a media repo worker is used, the admin APIs relating to the media repo will be served from it instead. ([\#5754](https://github.com/matrix-org/synapse/issues/5754), [\#5848](https://github.com/matrix-org/synapse/issues/5848)) - Synapse can now be configured to not join remote rooms of a given "complexity" (currently, state events) over federation. This option can be used to prevent adverse performance on resource-constrained homeservers. ([\#5783](https://github.com/matrix-org/synapse/issues/5783)) - Allow defining HTML templates to serve the user on account renewal attempt when using the account validity feature. ([\#5807](https://github.com/matrix-org/synapse/issues/5807)) Bugfixes -------- - Fix UISIs during homeserver outage. ([\#5693](https://github.com/matrix-org/synapse/issues/5693), [\#5789](https://github.com/matrix-org/synapse/issues/5789)) - Fix stack overflow in server key lookup code. ([\#5724](https://github.com/matrix-org/synapse/issues/5724)) - start.sh no longer uses deprecated cli option. ([\#5725](https://github.com/matrix-org/synapse/issues/5725)) - Log when we receive an event receipt from an unexpected origin. ([\#5743](https://github.com/matrix-org/synapse/issues/5743)) - Fix debian packaging scripts to correctly build sid packages. ([\#5775](https://github.com/matrix-org/synapse/issues/5775)) - Correctly handle redactions of redactions. ([\#5788](https://github.com/matrix-org/synapse/issues/5788)) - Return 404 instead of 403 when accessing /rooms/{roomId}/event/{eventId} for an event without the appropriate permissions. ([\#5798](https://github.com/matrix-org/synapse/issues/5798)) - Fix check that tombstone is a state event in push rules. ([\#5804](https://github.com/matrix-org/synapse/issues/5804)) - Fix error when trying to login as a deactivated user when using a worker to handle login. ([\#5806](https://github.com/matrix-org/synapse/issues/5806)) - Fix bug where user `/sync` stream could get wedged in rare circumstances. ([\#5825](https://github.com/matrix-org/synapse/issues/5825)) - The purge_remote_media.sh script was fixed. ([\#5839](https://github.com/matrix-org/synapse/issues/5839)) Deprecations and Removals ------------------------- - Synapse now no longer accepts the `-v`/`--verbose`, `-f`/`--log-file`, or `--log-config` command line flags, and removes the deprecated `verbose` and `log_file` configuration file options. Users of these options should migrate their options into the dedicated log configuration. ([\#5678](https://github.com/matrix-org/synapse/issues/5678), [\#5729](https://github.com/matrix-org/synapse/issues/5729)) - Remove non-functional 'expire_access_token' setting. ([\#5782](https://github.com/matrix-org/synapse/issues/5782)) Internal Changes ---------------- - Make Jaeger fully configurable. ([\#5694](https://github.com/matrix-org/synapse/issues/5694)) - Add precautionary measures to prevent future abuse of `window.opener` in default welcome page. ([\#5695](https://github.com/matrix-org/synapse/issues/5695)) - Reduce database IO usage by optimising queries for current membership. ([\#5706](https://github.com/matrix-org/synapse/issues/5706), [\#5738](https://github.com/matrix-org/synapse/issues/5738), [\#5746](https://github.com/matrix-org/synapse/issues/5746), [\#5752](https://github.com/matrix-org/synapse/issues/5752), [\#5770](https://github.com/matrix-org/synapse/issues/5770), [\#5774](https://github.com/matrix-org/synapse/issues/5774), [\#5792](https://github.com/matrix-org/synapse/issues/5792), [\#5793](https://github.com/matrix-org/synapse/issues/5793)) - Improve caching when fetching `get_filtered_current_state_ids`. ([\#5713](https://github.com/matrix-org/synapse/issues/5713)) - Don't accept opentracing data from clients. ([\#5715](https://github.com/matrix-org/synapse/issues/5715)) - Speed up PostgreSQL unit tests in CI. ([\#5717](https://github.com/matrix-org/synapse/issues/5717)) - Update the coding style document. ([\#5719](https://github.com/matrix-org/synapse/issues/5719)) - Improve database query performance when recording retry intervals for remote hosts. ([\#5720](https://github.com/matrix-org/synapse/issues/5720)) - Add a set of opentracing utils. ([\#5722](https://github.com/matrix-org/synapse/issues/5722)) - Cache result of get_version_string to reduce overhead of `/version` federation requests. ([\#5730](https://github.com/matrix-org/synapse/issues/5730)) - Return 'user_type' in admin API user endpoints results. ([\#5731](https://github.com/matrix-org/synapse/issues/5731)) - Don't package the sytest test blacklist file. ([\#5733](https://github.com/matrix-org/synapse/issues/5733)) - Replace uses of returnValue with plain return, as returnValue is not needed on Python 3. ([\#5736](https://github.com/matrix-org/synapse/issues/5736)) - Blacklist some flakey tests in worker mode. ([\#5740](https://github.com/matrix-org/synapse/issues/5740)) - Fix some error cases in the caching layer. ([\#5749](https://github.com/matrix-org/synapse/issues/5749)) - Add a prometheus metric for pending cache lookups. ([\#5750](https://github.com/matrix-org/synapse/issues/5750)) - Stop trying to fetch events with event_id=None. ([\#5753](https://github.com/matrix-org/synapse/issues/5753)) - Convert RedactionTestCase to modern test style. ([\#5768](https://github.com/matrix-org/synapse/issues/5768)) - Allow looping calls to be given arguments. ([\#5780](https://github.com/matrix-org/synapse/issues/5780)) - Set the logs emitted when checking typing and presence timeouts to DEBUG level, not INFO. ([\#5785](https://github.com/matrix-org/synapse/issues/5785)) - Remove DelayedCall debugging from the test suite, as it is no longer required in the vast majority of Synapse's tests. ([\#5787](https://github.com/matrix-org/synapse/issues/5787)) - Remove some spurious exceptions from the logs where we failed to talk to a remote server. ([\#5790](https://github.com/matrix-org/synapse/issues/5790)) - Improve performance when making `.well-known` requests by sharing the SSL options between requests. ([\#5794](https://github.com/matrix-org/synapse/issues/5794)) - Disable codecov GitHub comments on PRs. ([\#5796](https://github.com/matrix-org/synapse/issues/5796)) - Don't allow clients to send tombstone events that reference the room it's sent in. ([\#5801](https://github.com/matrix-org/synapse/issues/5801)) - Deny redactions of events sent in a different room. ([\#5802](https://github.com/matrix-org/synapse/issues/5802)) - Deny sending well known state types as non-state events. ([\#5805](https://github.com/matrix-org/synapse/issues/5805)) - Handle incorrectly encoded query params correctly by returning a 400. ([\#5808](https://github.com/matrix-org/synapse/issues/5808)) - Handle pusher being deleted during processing rather than logging an exception. ([\#5809](https://github.com/matrix-org/synapse/issues/5809)) - Return 502 not 500 when failing to reach any remote server. ([\#5810](https://github.com/matrix-org/synapse/issues/5810)) - Reduce global pauses in the events stream caused by expensive state resolution during persistence. ([\#5826](https://github.com/matrix-org/synapse/issues/5826)) - Add a lower bound to well-known lookup cache time to avoid repeated lookups. ([\#5836](https://github.com/matrix-org/synapse/issues/5836)) - Whitelist history visbility sytests in worker mode tests. ([\#5843](https://github.com/matrix-org/synapse/issues/5843))	2019-08-15 12:37:45 +01:00
Brendan Abolivier	fb5acd7039	1.3.0	2019-08-15 12:05:24 +01:00
Erik Johnston	748aa38378	Remove logging for #5407 and update comments	2019-08-15 12:02:18 +01:00
Andrew Morgan	8cf7fbbce0	Remove libsqlite3-dev from required build dependencies. (#5766 )	2019-08-15 11:32:23 +01:00
reivilibre	7809f0c022	Merge pull request #5851 from matrix-org/rei/roomdir_maybedeferred Room Directory: Wrap `get_local_public_room_list` call in `maybeDeferred`	2019-08-15 11:02:33 +01:00
Michael Telatynski	baee288fb4	Don't create broken room when power_level_content_override.users does not contain creator_id. (#5633 )	2019-08-15 09:45:57 +01:00
Hubert Chathi	c058aeb88d	update set_e2e_room_key to agree with fixed index	2019-08-14 18:02:58 -07:00
Hubert Chathi	81b8080acd	add changelog	2019-08-14 17:53:33 -07:00
Hubert Chathi	b7f7cc7ace	add the version field to the index for e2e_room_keys	2019-08-14 17:14:40 -07:00
reivilibre	d6de55bce9	Update changelog.d/5851.bugfix Use imperative Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2019-08-14 14:53:49 +01:00
Olivier Wilkinson (reivilibre)	3ad24ab386	Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>	2019-08-14 14:53:49 +01:00
Olivier Wilkinson (reivilibre)	1b63ccd848	Wrap `get_local_public_room_list` call in `maybeDeferred` because it is cached and so does not always return a `Deferred`. `await` does not silently pass-through non-Deferreds like `yield` used to. Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>	2019-08-14 14:53:49 +01:00
Erik Johnston	09f6152a11	Merge pull request #5844 from matrix-org/erikj/retry_well_known_lookup Retry well-known lookup before expiry.	2019-08-14 09:53:33 +01:00
Brendan Abolivier	f70d0a1dd9	1.3.0rc1	2019-08-13 18:20:09 +01:00
Brendan Abolivier	3039be82ce	Merge pull request #5848 from matrix-org/hawkowl/fix-mediarepo-worker-startup Fix mediarepo worker startup	2019-08-13 17:38:11 +01:00
Amber H. Brown	28bce1ac7c	changelog	2019-08-14 02:08:24 +10:00
Amber H. Brown	18bdac8ee4	fix config being a dict, actually	2019-08-14 02:06:42 +10:00
Erik Johnston	aedfec3ad7	Newsfile	2019-08-13 16:20:38 +01:00
Erik Johnston	17e1e80726	Retry well-known lookup before expiry. This gives a bit of a grace period where we can attempt to refetch a remote `well-known`, while still using the cached result if that fails. Hopefully this will make the well-known resolution a bit more torelant of failures, rather than it immediately treating failures as "no result" and caching that for an hour.	2019-08-13 16:20:38 +01:00
Erik Johnston	af187805b3	Merge pull request #5809 from matrix-org/erikj/handle_pusher_stop Handle pusher being deleted during processing.	2019-08-13 14:08:29 +01:00
Erik Johnston	96bdd661b8	Remove redundant return	2019-08-13 12:50:36 +01:00
Erik Johnston	fb3469f53a	Clarify docstring Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2019-08-13 10:17:23 +01:00
Neil Johnson	1016f303e5	make user creation steps clearer	2019-08-08 14:58:21 +01:00
Erik Johnston	312cc48e2b	Newsfile	2019-08-01 13:45:09 +01:00
Erik Johnston	d02e41dcb2	Handle pusher being deleted during processing. Instead of throwing a StoreError lets break out of processing loop and mark the pusher as stopped.	2019-08-01 13:44:12 +01:00
				`@@ -0,0 +1 @@`
				`Don't create broken room when power_level_content_override.users does not contain creator_id.`
				`@@ -1 +0,0 @@`
				Synapse now no longer accepts the `-v`/`--verbose`, `-f`/`--log-file`, or `--log-config` command line flags, and removes the deprecated `verbose` and `log_file` configuration file options. Users of these options should migrate their options into the dedicated log configuration.
				`@@ -1 +0,0 @@`
				Use `M_USER_DEACTIVATED` instead of `M_UNKNOWN` for errcode when a deactivated user attempts to login.
				`@@ -1 +0,0 @@`
				Add precautionary measures to prevent future abuse of `window.opener` in default welcome page.
				`@@ -1 +0,0 @@`
				`Reduce database IO usage by optimising queries for current membership.`
				`@@ -1 +0,0 @@`
				Improve caching when fetching `get_filtered_current_state_ids`.
				`@@ -1 +0,0 @@`
				`Don't accept opentracing data from clients.`
				`@@ -1 +0,0 @@`
				`Improve database query performance when recording retry intervals for remote hosts.`
				`@@ -1 +0,0 @@`
				`Fix stack overflow in server key lookup code.`