Newsfile

The logic is "designed" to "handle" the case where the servers view of
the state at an event doesn't match what the remote server set as the
auth events. With some hand waving the server would try and come to some
sort of conclusion of which side was correct, involving state
resolution, but this could come up with interesting results.

The entire process is unspecced and buggy, so let's just remove it.

.circleci/config.yml

@@ -4,16 +4,18 @@ jobs:
     machine: true
     steps:
       - checkout
-      - run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:${CIRCLE_TAG} .
+      - run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:${CIRCLE_TAG} -t matrixdotorg/synapse:${CIRCLE_TAG}-py3 .
       - run: docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
       - run: docker push matrixdotorg/synapse:${CIRCLE_TAG}
+      - run: docker push matrixdotorg/synapse:${CIRCLE_TAG}-py3
   dockerhubuploadlatest:
     machine: true
     steps:
       - checkout
-      - run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:latest .
+      - run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:latest -t matrixdotorg/synapse:latest-py3 .
       - run: docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
       - run: docker push matrixdotorg/synapse:latest
+      - run: docker push matrixdotorg/synapse:latest-py3
 
 workflows:
   version: 2

.github/ISSUE_TEMPLATE/BUG_REPORT.md

@@ -4,12 +4,12 @@ about: Create a report to help us improve
 
 ---
 
-<!--
-
 **THIS IS NOT A SUPPORT CHANNEL!**
 **IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER**,
 please ask in **#synapse:matrix.org** (using a matrix.org account if necessary)
 
+<!--
+
 If you want to report a security issue, please see https://matrix.org/security-disclosure-policy/
 
 This is a bug report template. By following the instructions below and

CHANGES.md

@@ -1,213 +1,3 @@
-Synapse 1.20.0rc4 (2020-09-16)
-==============================
-
-Synapse 1.20.0rc4 is identical to 1.20.0rc3, with the addition of the security fix that was included in 1.19.2.
-
-
-Synapse 1.19.2 (2020-09-16)
-===========================
-
-Due to the issue below server admins are encouraged to upgrade as soon as possible.
-
-Bugfixes
---------
-
-- Fix joining rooms over federation that include malformed events. ([\#8324](https://github.com/matrix-org/synapse/issues/8324))
-
-
-Synapse 1.20.0rc3 (2020-09-11)
-==============================
-
-Bugfixes
---------
-
-- Fix a bug introduced in v1.20.0rc1 where the wrong exception was raised when invalid JSON data is encountered. ([\#8291](https://github.com/matrix-org/synapse/issues/8291))
-
-
-Synapse 1.20.0rc2 (2020-09-09)
-==============================
-
-Bugfixes
---------
-
-- Fix a bug introduced in v1.20.0rc1 causing some features related to notifications to misbehave following the implementation of unread counts. ([\#8280](https://github.com/matrix-org/synapse/issues/8280))
-
-
-Synapse 1.20.0rc1 (2020-09-08)
-==============================
-
-Removal warning
----------------
-
-Some older clients used a [disallowed character](https://matrix.org/docs/spec/client_server/r0.6.1#post-matrix-client-r0-register-email-requesttoken) (`:`) in the `client_secret` parameter of various endpoints. The incorrect behaviour was allowed for backwards compatibility, but is now being removed from Synapse as most users have updated their client. Further context can be found at [\#6766](https://github.com/matrix-org/synapse/issues/6766).
-
-Features
---------
-
-- Add an endpoint to query your shared rooms with another user as an implementation of [MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666). ([\#7785](https://github.com/matrix-org/synapse/issues/7785))
-- Iteratively encode JSON to avoid blocking the reactor. ([\#8013](https://github.com/matrix-org/synapse/issues/8013), [\#8116](https://github.com/matrix-org/synapse/issues/8116))
-- Add support for shadow-banning users (ignoring any message send requests). ([\#8034](https://github.com/matrix-org/synapse/issues/8034), [\#8092](https://github.com/matrix-org/synapse/issues/8092), [\#8095](https://github.com/matrix-org/synapse/issues/8095), [\#8142](https://github.com/matrix-org/synapse/issues/8142), [\#8152](https://github.com/matrix-org/synapse/issues/8152), [\#8157](https://github.com/matrix-org/synapse/issues/8157), [\#8158](https://github.com/matrix-org/synapse/issues/8158), [\#8176](https://github.com/matrix-org/synapse/issues/8176))
-- Use the default template file when its equivalent is not found in a custom template directory. ([\#8037](https://github.com/matrix-org/synapse/issues/8037), [\#8107](https://github.com/matrix-org/synapse/issues/8107), [\#8252](https://github.com/matrix-org/synapse/issues/8252))
-- Add unread messages count to sync responses, as specified in [MSC2654](https://github.com/matrix-org/matrix-doc/pull/2654). ([\#8059](https://github.com/matrix-org/synapse/issues/8059), [\#8254](https://github.com/matrix-org/synapse/issues/8254), [\#8270](https://github.com/matrix-org/synapse/issues/8270), [\#8274](https://github.com/matrix-org/synapse/issues/8274))
-- Optimise `/federation/v1/user/devices/` API by only returning devices with encryption keys. ([\#8198](https://github.com/matrix-org/synapse/issues/8198))
-
-
-Bugfixes
---------
-
-- Fix a memory leak by limiting the length of time that messages will be queued for a remote server that has been unreachable. ([\#7864](https://github.com/matrix-org/synapse/issues/7864))
-- Fix `Re-starting finished log context PUT-nnnn` warning when event persistence failed. ([\#8081](https://github.com/matrix-org/synapse/issues/8081))
-- Synapse now correctly enforces the valid characters in the `client_secret` parameter used in various endpoints. ([\#8101](https://github.com/matrix-org/synapse/issues/8101))
-- Fix a bug introduced in v1.7.2 impacting message retention policies that would allow federated homeservers to dictate a retention period that's lower than the configured minimum allowed duration in the configuration file. ([\#8104](https://github.com/matrix-org/synapse/issues/8104))
-- Fix a long-standing bug where invalid JSON would be accepted by Synapse. ([\#8106](https://github.com/matrix-org/synapse/issues/8106))
-- Fix a bug introduced in Synapse v1.12.0 which could cause `/sync` requests to fail with a 404 if you had a very old outstanding room invite. ([\#8110](https://github.com/matrix-org/synapse/issues/8110))
-- Return a proper error code when the rooms of an invalid group are requested. ([\#8129](https://github.com/matrix-org/synapse/issues/8129))
-- Fix a bug which could cause a leaked postgres connection if synapse was set to daemonize. ([\#8131](https://github.com/matrix-org/synapse/issues/8131))
-- Clarify the error code if a user tries to register with a numeric ID. This bug was introduced in v1.15.0. ([\#8135](https://github.com/matrix-org/synapse/issues/8135))
-- Fix a bug where appservices with ratelimiting disabled would still be ratelimited when joining rooms. This bug was introduced in v1.19.0. ([\#8139](https://github.com/matrix-org/synapse/issues/8139))
-- Fix logging in via OpenID Connect with a provider that uses integer user IDs. ([\#8190](https://github.com/matrix-org/synapse/issues/8190))
-- Fix a longstanding bug where user directory updates could break when unexpected profile data was included in events. ([\#8223](https://github.com/matrix-org/synapse/issues/8223))
-- Fix a longstanding bug where stats updates could break when unexpected profile data was included in events. ([\#8226](https://github.com/matrix-org/synapse/issues/8226))
-- Fix slow start times for large servers by removing a table scan of the `users` table from startup code. ([\#8271](https://github.com/matrix-org/synapse/issues/8271))
-
-
-Updates to the Docker image
----------------------------
-
-- Fix builds of the Docker image on non-x86 platforms. ([\#8144](https://github.com/matrix-org/synapse/issues/8144))
-- Added curl for healthcheck support and readme updates for the change. Contributed by @maquis196. ([\#8147](https://github.com/matrix-org/synapse/issues/8147))
-
-
-Improved Documentation
-----------------------
-
-- Link to matrix-synapse-rest-password-provider in the password provider documentation. ([\#8111](https://github.com/matrix-org/synapse/issues/8111))
-- Updated documentation to note that Synapse does not follow `HTTP 308` redirects due to an upstream library not supporting them. Contributed by Ryan Cole. ([\#8120](https://github.com/matrix-org/synapse/issues/8120))
-- Explain better what GDPR-erased means when deactivating a user. ([\#8189](https://github.com/matrix-org/synapse/issues/8189))
-
-
-Internal Changes
-----------------
-
-- Add filter `name` to the `/users` admin API, which filters by user ID or displayname. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#7377](https://github.com/matrix-org/synapse/issues/7377), [\#8163](https://github.com/matrix-org/synapse/issues/8163))
-- Reduce run times of some unit tests by advancing the reactor a fewer number of times. ([\#7757](https://github.com/matrix-org/synapse/issues/7757))
-- Don't fail `/submit_token` requests on incorrect session ID if `request_token_inhibit_3pid_errors` is turned on. ([\#7991](https://github.com/matrix-org/synapse/issues/7991))
-- Convert various parts of the codebase to async/await. ([\#8071](https://github.com/matrix-org/synapse/issues/8071), [\#8072](https://github.com/matrix-org/synapse/issues/8072), [\#8074](https://github.com/matrix-org/synapse/issues/8074), [\#8075](https://github.com/matrix-org/synapse/issues/8075), [\#8076](https://github.com/matrix-org/synapse/issues/8076), [\#8087](https://github.com/matrix-org/synapse/issues/8087), [\#8100](https://github.com/matrix-org/synapse/issues/8100), [\#8119](https://github.com/matrix-org/synapse/issues/8119), [\#8121](https://github.com/matrix-org/synapse/issues/8121), [\#8133](https://github.com/matrix-org/synapse/issues/8133), [\#8156](https://github.com/matrix-org/synapse/issues/8156), [\#8162](https://github.com/matrix-org/synapse/issues/8162), [\#8166](https://github.com/matrix-org/synapse/issues/8166), [\#8168](https://github.com/matrix-org/synapse/issues/8168), [\#8173](https://github.com/matrix-org/synapse/issues/8173), [\#8191](https://github.com/matrix-org/synapse/issues/8191), [\#8192](https://github.com/matrix-org/synapse/issues/8192), [\#8193](https://github.com/matrix-org/synapse/issues/8193), [\#8194](https://github.com/matrix-org/synapse/issues/8194), [\#8195](https://github.com/matrix-org/synapse/issues/8195), [\#8197](https://github.com/matrix-org/synapse/issues/8197), [\#8199](https://github.com/matrix-org/synapse/issues/8199), [\#8200](https://github.com/matrix-org/synapse/issues/8200), [\#8201](https://github.com/matrix-org/synapse/issues/8201), [\#8202](https://github.com/matrix-org/synapse/issues/8202), [\#8207](https://github.com/matrix-org/synapse/issues/8207), [\#8213](https://github.com/matrix-org/synapse/issues/8213), [\#8214](https://github.com/matrix-org/synapse/issues/8214))
-- Remove some unused database functions. ([\#8085](https://github.com/matrix-org/synapse/issues/8085))
-- Add type hints to various parts of the codebase. ([\#8090](https://github.com/matrix-org/synapse/issues/8090), [\#8127](https://github.com/matrix-org/synapse/issues/8127), [\#8187](https://github.com/matrix-org/synapse/issues/8187), [\#8241](https://github.com/matrix-org/synapse/issues/8241), [\#8140](https://github.com/matrix-org/synapse/issues/8140), [\#8183](https://github.com/matrix-org/synapse/issues/8183), [\#8232](https://github.com/matrix-org/synapse/issues/8232), [\#8235](https://github.com/matrix-org/synapse/issues/8235), [\#8237](https://github.com/matrix-org/synapse/issues/8237), [\#8244](https://github.com/matrix-org/synapse/issues/8244))
-- Return the previous stream token if a non-member event is a duplicate. ([\#8093](https://github.com/matrix-org/synapse/issues/8093), [\#8112](https://github.com/matrix-org/synapse/issues/8112))
-- Separate `get_current_token` into two since there are two different use cases for it. ([\#8113](https://github.com/matrix-org/synapse/issues/8113))
-- Remove `ChainedIdGenerator`. ([\#8123](https://github.com/matrix-org/synapse/issues/8123))
-- Reduce the amount of whitespace in JSON stored and sent in responses. ([\#8124](https://github.com/matrix-org/synapse/issues/8124))
-- Update the test federation client to handle streaming responses. ([\#8130](https://github.com/matrix-org/synapse/issues/8130))
-- Micro-optimisations to `get_auth_chain_ids`. ([\#8132](https://github.com/matrix-org/synapse/issues/8132))
-- Refactor `StreamIdGenerator` and `MultiWriterIdGenerator` to have the same interface. ([\#8161](https://github.com/matrix-org/synapse/issues/8161))
-- Add functions to `MultiWriterIdGen` used by events stream. ([\#8164](https://github.com/matrix-org/synapse/issues/8164), [\#8179](https://github.com/matrix-org/synapse/issues/8179))
-- Fix tests that were broken due to the merge of 1.19.1. ([\#8167](https://github.com/matrix-org/synapse/issues/8167))
-- Make `SlavedIdTracker.advance` have the same interface as `MultiWriterIDGenerator`. ([\#8171](https://github.com/matrix-org/synapse/issues/8171))
-- Remove unused `is_guest` parameter from, and add safeguard to, `MessageHandler.get_room_data`. ([\#8174](https://github.com/matrix-org/synapse/issues/8174), [\#8181](https://github.com/matrix-org/synapse/issues/8181))
-- Standardize the mypy configuration. ([\#8175](https://github.com/matrix-org/synapse/issues/8175))
-- Refactor some of `LoginRestServlet`'s helper methods, and move them to `AuthHandler` for easier reuse. ([\#8182](https://github.com/matrix-org/synapse/issues/8182))
-- Fix `wait_for_stream_position` to allow multiple waiters on same stream ID. ([\#8196](https://github.com/matrix-org/synapse/issues/8196))
-- Make `MultiWriterIDGenerator` work for streams that use negative values. ([\#8203](https://github.com/matrix-org/synapse/issues/8203))
-- Refactor queries for device keys and cross-signatures. ([\#8204](https://github.com/matrix-org/synapse/issues/8204), [\#8205](https://github.com/matrix-org/synapse/issues/8205), [\#8222](https://github.com/matrix-org/synapse/issues/8222), [\#8224](https://github.com/matrix-org/synapse/issues/8224), [\#8225](https://github.com/matrix-org/synapse/issues/8225), [\#8231](https://github.com/matrix-org/synapse/issues/8231), [\#8233](https://github.com/matrix-org/synapse/issues/8233), [\#8234](https://github.com/matrix-org/synapse/issues/8234))
-- Fix type hints for functions decorated with `@cached`. ([\#8240](https://github.com/matrix-org/synapse/issues/8240))
-- Remove obsolete `order` field from federation send queues. ([\#8245](https://github.com/matrix-org/synapse/issues/8245))
-- Stop sub-classing from object. ([\#8249](https://github.com/matrix-org/synapse/issues/8249))
-- Add more logging to debug slow startup. ([\#8264](https://github.com/matrix-org/synapse/issues/8264))
-- Do not attempt to upgrade database schema on worker processes. ([\#8266](https://github.com/matrix-org/synapse/issues/8266), [\#8276](https://github.com/matrix-org/synapse/issues/8276))
-
-
-Synapse 1.19.1 (2020-08-27)
-===========================
-
-No significant changes.
-
-
-Synapse 1.19.1rc1 (2020-08-25)
-==============================
-
-Bugfixes
---------
-
-- Fix a bug introduced in v1.19.0 where appservices with ratelimiting disabled would still be ratelimited when joining rooms. ([\#8139](https://github.com/matrix-org/synapse/issues/8139))
-- Fix a bug introduced in v1.19.0 that would cause e.g. profile updates to fail due to incorrect application of rate limits on join requests. ([\#8153](https://github.com/matrix-org/synapse/issues/8153))
-
-
-Synapse 1.19.0 (2020-08-17)
-===========================
-
-No significant changes since 1.19.0rc1.
-
-Removal warning
----------------
-
-As outlined in the [previous release](https://github.com/matrix-org/synapse/releases/tag/v1.18.0), we are no longer publishing Docker images with the `-py3` tag suffix. On top of that, we have also removed the `latest-py3` tag. Please see [the announcement in the upgrade notes for 1.18.0](https://github.com/matrix-org/synapse/blob/develop/UPGRADE.rst#upgrading-to-v1180).
-
-
-Synapse 1.19.0rc1 (2020-08-13)
-==============================
-
-Features
---------
-
-- Add option to allow server admins to join rooms which fail complexity checks. Contributed by @lugino-emeritus. ([\#7902](https://github.com/matrix-org/synapse/issues/7902))
-- Add an option to purge room or not with delete room admin endpoint (`POST /_synapse/admin/v1/rooms/<room_id>/delete`). Contributed by @dklimpel. ([\#7964](https://github.com/matrix-org/synapse/issues/7964))
-- Add rate limiting to users joining rooms. ([\#8008](https://github.com/matrix-org/synapse/issues/8008))
-- Add a `/health` endpoint to every configured HTTP listener that can be used as a health check endpoint by load balancers. ([\#8048](https://github.com/matrix-org/synapse/issues/8048))
-- Allow login to be blocked based on the values of SAML attributes. ([\#8052](https://github.com/matrix-org/synapse/issues/8052))
-- Allow guest access to the `GET /_matrix/client/r0/rooms/{room_id}/members` endpoint, according to MSC2689. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#7314](https://github.com/matrix-org/synapse/issues/7314))
-
-
-Bugfixes
---------
-
-- Fix a bug introduced in Synapse v1.7.2 which caused inaccurate membership counts in the room directory. ([\#7977](https://github.com/matrix-org/synapse/issues/7977))
-- Fix a long standing bug: 'Duplicate key value violates unique constraint "event_relations_id"' when message retention is configured. ([\#7978](https://github.com/matrix-org/synapse/issues/7978))
-- Fix "no create event in auth events" when trying to reject invitation after inviter leaves. Bug introduced in Synapse v1.10.0. ([\#7980](https://github.com/matrix-org/synapse/issues/7980))
-- Fix various comments and minor discrepencies in server notices code. ([\#7996](https://github.com/matrix-org/synapse/issues/7996))
-- Fix a long standing bug where HTTP HEAD requests resulted in a 400 error. ([\#7999](https://github.com/matrix-org/synapse/issues/7999))
-- Fix a long-standing bug which caused two copies of some log lines to be written when synctl was used along with a MemoryHandler logger. ([\#8011](https://github.com/matrix-org/synapse/issues/8011), [\#8012](https://github.com/matrix-org/synapse/issues/8012))
-
-
-Updates to the Docker image
----------------------------
-
-- We no longer publish Docker images with the `-py3` tag suffix, as [announced in the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/UPGRADE.rst#upgrading-to-v1180). ([\#8056](https://github.com/matrix-org/synapse/issues/8056))
-
-
-Improved Documentation
-----------------------
-
-- Document how to set up a client .well-known file and fix several pieces of outdated documentation. ([\#7899](https://github.com/matrix-org/synapse/issues/7899))
-- Improve workers docs. ([\#7990](https://github.com/matrix-org/synapse/issues/7990), [\#8000](https://github.com/matrix-org/synapse/issues/8000))
-- Fix typo in `docs/workers.md`. ([\#7992](https://github.com/matrix-org/synapse/issues/7992))
-- Add documentation for how to undo a room shutdown. ([\#7998](https://github.com/matrix-org/synapse/issues/7998), [\#8010](https://github.com/matrix-org/synapse/issues/8010))
-
-
-Internal Changes
-----------------
-
-- Reduce the amount of whitespace in JSON stored and sent in responses. Contributed by David Vo. ([\#7372](https://github.com/matrix-org/synapse/issues/7372))
-- Switch to the JSON implementation from the standard library and bump the minimum version of the canonicaljson library to 1.2.0. ([\#7936](https://github.com/matrix-org/synapse/issues/7936), [\#7979](https://github.com/matrix-org/synapse/issues/7979))
-- Convert various parts of the codebase to async/await. ([\#7947](https://github.com/matrix-org/synapse/issues/7947), [\#7948](https://github.com/matrix-org/synapse/issues/7948), [\#7949](https://github.com/matrix-org/synapse/issues/7949), [\#7951](https://github.com/matrix-org/synapse/issues/7951), [\#7963](https://github.com/matrix-org/synapse/issues/7963), [\#7973](https://github.com/matrix-org/synapse/issues/7973), [\#7975](https://github.com/matrix-org/synapse/issues/7975), [\#7976](https://github.com/matrix-org/synapse/issues/7976), [\#7981](https://github.com/matrix-org/synapse/issues/7981), [\#7987](https://github.com/matrix-org/synapse/issues/7987), [\#7989](https://github.com/matrix-org/synapse/issues/7989), [\#8003](https://github.com/matrix-org/synapse/issues/8003), [\#8014](https://github.com/matrix-org/synapse/issues/8014), [\#8016](https://github.com/matrix-org/synapse/issues/8016), [\#8027](https://github.com/matrix-org/synapse/issues/8027), [\#8031](https://github.com/matrix-org/synapse/issues/8031), [\#8032](https://github.com/matrix-org/synapse/issues/8032), [\#8035](https://github.com/matrix-org/synapse/issues/8035), [\#8042](https://github.com/matrix-org/synapse/issues/8042), [\#8044](https://github.com/matrix-org/synapse/issues/8044), [\#8045](https://github.com/matrix-org/synapse/issues/8045), [\#8061](https://github.com/matrix-org/synapse/issues/8061), [\#8062](https://github.com/matrix-org/synapse/issues/8062), [\#8063](https://github.com/matrix-org/synapse/issues/8063), [\#8066](https://github.com/matrix-org/synapse/issues/8066), [\#8069](https://github.com/matrix-org/synapse/issues/8069), [\#8070](https://github.com/matrix-org/synapse/issues/8070))
-- Move some database-related log lines from the default logger to the database/transaction loggers. ([\#7952](https://github.com/matrix-org/synapse/issues/7952))
-- Add a script to detect source code files using non-unix line terminators. ([\#7965](https://github.com/matrix-org/synapse/issues/7965), [\#7970](https://github.com/matrix-org/synapse/issues/7970))
-- Log the SAML session ID during creation. ([\#7971](https://github.com/matrix-org/synapse/issues/7971))
-- Implement new experimental push rules for some users. ([\#7997](https://github.com/matrix-org/synapse/issues/7997))
-- Remove redundant and unreliable signature check for v1 Identity Service lookup responses. ([\#8001](https://github.com/matrix-org/synapse/issues/8001))
-- Improve the performance of the register endpoint. ([\#8009](https://github.com/matrix-org/synapse/issues/8009))
-- Reduce less useful output in the newsfragment CI step. Add a link to the changelog section of the contributing guide on error. ([\#8024](https://github.com/matrix-org/synapse/issues/8024))
-- Rename storage layer objects to be more sensible. ([\#8033](https://github.com/matrix-org/synapse/issues/8033))
-- Change the default log config to reduce disk I/O and storage for new servers. ([\#8040](https://github.com/matrix-org/synapse/issues/8040))
-- Add an assertion on `prev_events` in `create_new_client_event`. ([\#8041](https://github.com/matrix-org/synapse/issues/8041))
-- Add a comment to `ServerContextFactory` about the use of `SSLv23_METHOD`. ([\#8043](https://github.com/matrix-org/synapse/issues/8043))
-- Log `OPTIONS` requests at `DEBUG` rather than `INFO` level to reduce amount logged at `INFO`. ([\#8049](https://github.com/matrix-org/synapse/issues/8049))
-- Reduce amount of outbound request logging at `INFO` level. ([\#8050](https://github.com/matrix-org/synapse/issues/8050))
-- It is no longer necessary to explicitly define `filters` in the logging configuration. (Continuing to do so is redundant but harmless.) ([\#8051](https://github.com/matrix-org/synapse/issues/8051))
-- Add and improve type hints. ([\#8058](https://github.com/matrix-org/synapse/issues/8058), [\#8064](https://github.com/matrix-org/synapse/issues/8064), [\#8060](https://github.com/matrix-org/synapse/issues/8060), [\#8067](https://github.com/matrix-org/synapse/issues/8067))
-
-
 Synapse 1.18.0 (2020-07-30)
 ===========================
 

UPGRADE.rst

@@ -75,59 +75,6 @@ for example:
      wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb
      dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
 
-Upgrading to v1.21.0
-====================
-
-Forwarding ``/_synapse/client`` through your reverse proxy
-----------------------------------------------------------
-
-The `reverse proxy documentation
-<https://github.com/matrix-org/synapse/blob/develop/docs/reverse_proxy.md>`_ has been updated
-to include reverse proxy directives for ``/_synapse/client/*`` endpoints. As the user password
-reset flow now uses endpoints under this prefix, **you must update your reverse proxy
-configurations for user password reset to work**.
-
-Additionally, note that the `Synapse worker documentation
-<https://github.com/matrix-org/synapse/blob/develop/docs/workers.md>`_ has been updated to
- state that the ``/_synapse/client/password_reset/email/submit_token`` endpoint can be handled
-by all workers. If you make use of Synapse's worker feature, please update your reverse proxy
-configuration to reflect this change.
-
-New HTML templates
-------------------
-
-A new HTML template,
-`password_reset_confirmation.html <https://github.com/matrix-org/synapse/blob/develop/synapse/res/templates/password_reset_confirmation.html>`_,
-has been added to the ``synapse/res/templates`` directory. If you are using a
-custom template directory, you may want to copy the template over and modify it.
-
-Note that as of v1.20.0, templates do not need to be included in custom template
-directories for Synapse to start. The default templates will be used if a custom
-template cannot be found.
-
-This page will appear to the user after clicking a password reset link that has
-been emailed to them.
-
-To complete password reset, the page must include a way to make a `POST`
-request to
-``/_synapse/client/password_reset/{medium}/submit_token``
-with the query parameters from the original link, presented as a URL-encoded form. See the file
-itself for more details.
-
-Updated Single Sign-on HTML Templates
--------------------------------------
-
-The ``saml_error.html`` template was removed from Synapse and replaced with the
-``sso_error.html`` template. If your Synapse is configured to use SAML and a
-custom ``sso_redirect_confirm_template_dir`` configuration then any customisations
-of the ``saml_error.html`` template will need to be merged into the ``sso_error.html``
-template. These templates are similar, but the parameters are slightly different:
-
-* The ``msg`` parameter should be renamed to ``error_description``.
-* There is no longer a ``code`` parameter for the response code.
-* A string ``error`` parameter is available that includes a short hint of why a
-  user is seeing the error page.
-
 Upgrading to v1.18.0
 ====================
 

changelog.d/7124.bugfix

@@ -1 +0,0 @@
-Fix a bug in the media repository where remote thumbnails with the same size but different crop methods would overwrite each other. Contributed by @deepbluev7.

changelog.d/7314.misc

@@ -0,0 +1 @@
+Allow guest access to the `GET /_matrix/client/r0/rooms/{room_id}/members` endpoint, according to MSC2689. Contributed by Awesome Technologies Innovationslabor GmbH.

changelog.d/7736.feature

@@ -0,0 +1 @@
+Add unread messages count to sync responses, as specified in [MSC2654](https://github.com/matrix-org/matrix-doc/pull/2654).

changelog.d/7796.bugfix

@@ -1 +0,0 @@
-Fix inconsistent handling of non-existent push rules, and stop tracking the `enabled` state of removed push rules.

changelog.d/7899.doc

@@ -0,0 +1 @@
+Document how to set up a Client Well-Known file and fix several pieces of outdated documentation.

changelog.d/7902.feature

@@ -0,0 +1 @@
+Add option to allow server admins to join rooms which fail complexity checks. Contributed by @lugino-emeritus.

changelog.d/7936.misc

@@ -0,0 +1 @@
+Switch to the JSON implementation from the standard library and bump the minimum version of the canonicaljson library to 1.2.0.

changelog.d/7947.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/7948.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/7949.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/7951.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/7952.misc

@@ -0,0 +1 @@
+Move some database-related log lines from the default logger to the database/transaction loggers.

changelog.d/7963.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/7964.feature

@@ -0,0 +1 @@
+Add an option to purge room or not with delete room admin endpoint (`POST /_synapse/admin/v1/rooms/<room_id>/delete`). Contributed by @dklimpel.

changelog.d/7965.misc

@@ -0,0 +1 @@
+Add a script to detect source code files using non-unix line terminators.

changelog.d/7970.misc

@@ -0,0 +1 @@
+Add a script to detect source code files using non-unix line terminators.

changelog.d/7971.misc

@@ -0,0 +1 @@
+Log the SAML session ID during creation.

changelog.d/7973.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/7975.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/7976.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/7977.bugfix

@@ -0,0 +1 @@
+Fix a bug introduced in Synapse v1.7.2 which caused inaccurate membership counts in the room directory.

changelog.d/7978.bugfix

@@ -0,0 +1 @@
+Fix a long standing bug: 'Duplicate key value violates unique constraint "event_relations_id"' when message retention is configured.

changelog.d/7979.misc

@@ -0,0 +1 @@
+Switch to the JSON implementation from the standard library and bump the minimum version of the canonicaljson library to 1.2.0.

changelog.d/7980.bugfix

@@ -0,0 +1 @@
+Fix "no create event in auth events" when trying to reject invitation after inviter leaves. Bug introduced in Synapse v1.10.0.

changelog.d/7981.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/7987.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/7989.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/7990.doc

@@ -0,0 +1 @@
+Improve workers docs.

changelog.d/7992.doc

@@ -0,0 +1 @@
+Fix typo in `docs/workers.md`.

changelog.d/7996.bugfix

@@ -0,0 +1 @@
+Fix various comments and minor discrepencies in server notices code.

changelog.d/7998.doc

@@ -0,0 +1 @@
+Add documentation for how to undo a room shutdown.

changelog.d/7999.bugfix

@@ -0,0 +1 @@
+Fix a long standing bug where HTTP HEAD requests resulted in a 400 error.

changelog.d/8001.misc

@@ -0,0 +1 @@
+Remove redundant and unreliable signature check for v1 Identity Service lookup responses.

changelog.d/8003.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8004.feature

@@ -1 +0,0 @@
-Require the user to confirm that their password should be reset after clicking the email confirmation link.

changelog.d/8008.feature

@@ -0,0 +1 @@
+Add rate limiting to users joining rooms.

changelog.d/8025.bugfix

@@ -0,0 +1 @@
+Fix bug where state (e.g. power levels) would reset incorrectly when receiving an event from a remote server.

changelog.d/8208.misc

@@ -1 +0,0 @@
-Fix tests on distros which disable TLSv1.0. Contributed by @danc86.

changelog.d/8216.misc

@@ -1 +0,0 @@
-Simplify the distributor code to avoid unnecessary work.

changelog.d/8227.doc

@@ -1 +0,0 @@
-Add `/_synapse/client` to the reverse proxy documentation.

changelog.d/8230.misc

@@ -1 +0,0 @@
-Track the latest event for every destination and room for catch-up after federation outage.

changelog.d/8236.bugfix

@@ -1 +0,0 @@
-Fix a longstanding bug where files that could not be thumbnailed would result in an Internal Server Error.

changelog.d/8243.misc

@@ -1 +0,0 @@
-Remove the 'populate_stats_process_rooms_2' background job and restore functionality to 'populate_stats_process_rooms'.

changelog.d/8247.misc

@@ -1 +0,0 @@
-Track the `stream_ordering` of the last successfully-sent event to every destination, so we can use this information to 'catch up' a remote server after an outage.

changelog.d/8248.feature

@@ -1 +0,0 @@
-Consolidate the SSO error template across all configuration.

changelog.d/8250.misc

@@ -1 +0,0 @@
-Clean up type hints for `PaginationConfig`.

changelog.d/8256.misc

@@ -1 +0,0 @@
-Track the latest event for every destination and room for catch-up after federation outage.

changelog.d/8257.misc

@@ -1 +0,0 @@
-Fix non-user visible bug in implementation of `MultiWriterIdGenerator.get_current_token_for_writer`.

changelog.d/8258.misc

@@ -1 +0,0 @@
-Track the `stream_ordering` of the last successfully-sent event to every destination, so we can use this information to 'catch up' a remote server after an outage.

changelog.d/8259.misc

@@ -1 +0,0 @@
-Switch to the JSON implementation from the standard library.

changelog.d/8260.misc

@@ -1 +0,0 @@
-Add type hints to `synapse.util.async_helpers`.

changelog.d/8261.misc

@@ -1 +0,0 @@
-Simplify tests that mock asynchronous functions.

changelog.d/8262.bugfix

@@ -1 +0,0 @@
-Upgrade canonicaljson to version 1.4.0 to fix an unicode encoding issue.

changelog.d/8265.bugfix

@@ -1 +0,0 @@
-Fix logstanding bug which could lead to incomplete database upgrades on SQLite.

changelog.d/8268.bugfix

@@ -1 +0,0 @@
-Fix stack overflow when stderr is redirected to the logging system, and the logging system encounters an error.

changelog.d/8272.bugfix

@@ -1 +0,0 @@
-Fix messages over federation being lost until an event is sent into the same room.

changelog.d/8275.feature

@@ -1 +0,0 @@
-Add a config option to specify a whitelist of domains that a user can be redirected to after validating their email or phone number.

changelog.d/8278.bugfix

@@ -1 +0,0 @@
-Fix a bug which cause the logging system to report errors, if `DEBUG` was enabled and no `context` filter was applied.

changelog.d/8279.misc

@@ -1 +0,0 @@
-Add type hints to `StreamToken` and `RoomStreamToken` classes.

changelog.d/8281.misc

@@ -1 +0,0 @@
-Change `StreamToken.room_key` to be a `RoomStreamToken` instance.

changelog.d/8282.misc

@@ -1 +0,0 @@
-Clean up type hints for `PaginationConfig`.

changelog.d/8285.misc

@@ -1 +0,0 @@
-Blacklist [MSC2753](https://github.com/matrix-org/matrix-doc/pull/2753) SyTests until it is implemented.

changelog.d/8287.bugfix

@@ -1 +0,0 @@
-Fix edge case where push could get delayed for a user until a later event was pushed.

changelog.d/8288.misc

@@ -1 +0,0 @@
-Refactor notifier code to correctly use the max event stream position.

changelog.d/8294.feature

@@ -1 +0,0 @@
-Add experimental support for sharding event persister.

changelog.d/8296.misc

@@ -1 +0,0 @@
-Use slotted classes where possible.

changelog.d/8305.feature

@@ -1 +0,0 @@
-Add the room topic and avatar to the room details admin API.

changelog.d/8324.bugfix

@@ -1 +0,0 @@
-Fix fetching events from remote servers that are malformed.

changelog.d/8326.misc

@@ -1 +0,0 @@
-Update outdated usages of `metaclass` to python 3 syntax.

changelog.d/8329.bugfix

@@ -1 +0,0 @@
-Fix UnboundLocalError from occuring when appservices send malformed register request.

changelog.d/8331.misc

@@ -1 +0,0 @@
-Use the `admin_patterns` helper in additional locations.

changelog.d/8337.misc

@@ -1 +0,0 @@
-Remove `__future__` imports related to Python 2 compatibility.

contrib/cmdclient/console.py

@@ -15,6 +15,8 @@
 # limitations under the License.
 
 """ Starts a synapse client console. """
+from __future__ import print_function
+
 import argparse
 import cmd
 import getpass

contrib/cmdclient/http.py

@@ -13,6 +13,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from __future__ import print_function
+
 import json
 import urllib
 from pprint import pformat
@@ -22,7 +24,7 @@ from twisted.web.client import Agent, readBody
 from twisted.web.http_headers import Headers
 
 
-class HttpClient:
+class HttpClient(object):
     """ Interface for talking json over http
     """
 
@@ -167,7 +169,7 @@ class TwistedHttpClient(HttpClient):
         return d
 
 
-class _RawProducer:
+class _RawProducer(object):
     def __init__(self, data):
         self.data = data
         self.body = data
@@ -184,7 +186,7 @@ class _RawProducer:
         pass
 
 
-class _JsonProducer:
+class _JsonProducer(object):
     """ Used by the twisted http client to create the HTTP body from json
     """
 

contrib/experiments/cursesio.py

@@ -141,7 +141,7 @@ class CursesStdIO:
         curses.endwin()
 
 
-class Callback:
+class Callback(object):
     def __init__(self, stdio):
         self.stdio = stdio
 

contrib/experiments/test_messaging.py

@@ -55,7 +55,7 @@ def excpetion_errback(failure):
     logging.exception(failure)
 
 
-class InputOutput:
+class InputOutput(object):
     """ This is responsible for basic I/O so that a user can interact with
     the example app.
     """
@@ -132,7 +132,7 @@ class IOLoggerHandler(logging.Handler):
         self.io.print_log(msg)
 
 
-class Room:
+class Room(object):
     """ Used to store (in memory) the current membership state of a room, and
     which home servers we should send PDUs associated with the room to.
     """

contrib/graph/graph.py

@@ -1,3 +1,5 @@
+from __future__ import print_function
+
 import argparse
 import cgi
 import datetime

contrib/graph/graph3.py

@@ -1,3 +1,5 @@
+from __future__ import print_function
+
 import argparse
 import cgi
 import datetime

contrib/jitsimeetbridge/jitsimeetbridge.py

@@ -10,6 +10,8 @@ the bridge.
 Requires:
 npm install jquery jsdom
 """
+from __future__ import print_function
+
 import json
 import subprocess
 import time

contrib/scripts/kick_users.py

@@ -1,4 +1,5 @@
 #!/usr/bin/env python
+from __future__ import print_function
 
 import json
 import sys
@@ -7,6 +8,11 @@ from argparse import ArgumentParser
 
 import requests
 
+try:
+    raw_input
+except NameError:  # Python 3
+    raw_input = input
+
 
 def _mkurl(template, kws):
     for key in kws:
@@ -52,7 +58,7 @@ def main(hs, room_id, access_token, user_id_prefix, why):
     print("The following user IDs will be kicked from %s" % room_name)
     for uid in kick_list:
         print(uid)
-    doit = input("Continue? [Y]es\n")
+    doit = raw_input("Continue? [Y]es\n")
     if len(doit) > 0 and doit.lower() == "y":
         print("Kicking members...")
         # encode them all

debian/changelog

@@ -1,30 +1,12 @@
-matrix-synapse-py3 (1.20.0ubuntu1) UNRELEASED; urgency=medium
-
-  * Use Type=notify in systemd service
-
- -- Dexter Chua <dec41@srcf.net>  Wed, 26 Aug 2020 12:41:36 +0000
-
-matrix-synapse-py3 (1.19.2) stable; urgency=medium
-
-  * New synapse release 1.19.2.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 16 Sep 2020 12:50:30 +0100
-
-matrix-synapse-py3 (1.19.1) stable; urgency=medium
-
-  * New synapse release 1.19.1.
-
- -- Synapse Packaging team <packages@matrix.org>  Thu, 27 Aug 2020 10:50:19 +0100
-
-matrix-synapse-py3 (1.19.0) stable; urgency=medium
+matrix-synapse-py3 (1.xx.0) stable; urgency=medium
 
   [ Synapse Packaging team ]
-  * New synapse release 1.19.0.
+  * New synapse release 1.xx.0.
 
   [ Aaron Raimist ]
   * Fix outdated documentation for SYNAPSE_CACHE_FACTOR
 
- -- Synapse Packaging team <packages@matrix.org>  Mon, 17 Aug 2020 14:06:42 +0100
+ -- Synapse Packaging team <packages@matrix.org>  XXXXX
 
 matrix-synapse-py3 (1.18.0) stable; urgency=medium
 

debian/matrix-synapse.service

@@ -2,7 +2,7 @@
 Description=Synapse Matrix homeserver
 
 [Service]
-Type=notify
+Type=simple
 User=matrix-synapse
 WorkingDirectory=/var/lib/matrix-synapse
 EnvironmentFile=/etc/default/matrix-synapse

docker/Dockerfile

@@ -19,16 +19,11 @@ ARG PYTHON_VERSION=3.7
 FROM docker.io/python:${PYTHON_VERSION}-slim as builder
 
 # install the OS build deps
+
+
 RUN apt-get update && apt-get install -y \
     build-essential \
-    libffi-dev \
-    libjpeg-dev \
     libpq-dev \
-    libssl-dev \
-    libwebp-dev \
-    libxml++2.6-dev \
-    libxslt1-dev \
-    zlib1g-dev \
  && rm -rf /var/lib/apt/lists/*
 
 # Build dependencies that are not available as wheels, to speed up rebuilds
@@ -60,12 +55,9 @@ RUN pip install --prefix="/install" --no-warn-script-location \
 FROM docker.io/python:${PYTHON_VERSION}-slim
 
 RUN apt-get update && apt-get install -y \
-    curl \
-    gosu \
-    libjpeg62-turbo \
     libpq5 \
-    libwebp6 \
     xmlsec1 \
+    gosu \
  && rm -rf /var/lib/apt/lists/*
 
 COPY --from=builder /install /usr/local
@@ -77,6 +69,3 @@ VOLUME ["/data"]
 EXPOSE 8008/tcp 8009/tcp 8448/tcp
 
 ENTRYPOINT ["/start.py"]
-
-HEALTHCHECK --interval=1m --timeout=5s \
-  CMD curl -fSs http://localhost:8008/health || exit 1

docker/README.md

@@ -162,32 +162,3 @@ docker build -t matrixdotorg/synapse -f docker/Dockerfile .
 
 You can choose to build a different docker image by changing the value of the `-f` flag to
 point to another Dockerfile.
-
-## Disabling the healthcheck
-
-If you are using a non-standard port or tls inside docker you can disable the healthcheck
-whilst running the above `docker run` commands. 
-
-```
-   --no-healthcheck
-```
-## Setting custom healthcheck on docker run
-
-If you wish to point the healthcheck at a different port with docker command, add the following
-
-```
-  --health-cmd 'curl -fSs http://localhost:1234/health'
-```
-
-## Setting the healthcheck in docker-compose file
-
-You can add the following to set a custom healthcheck in a docker compose file.
-You will need version >2.1 for this to work. 
-
-```
-healthcheck:
-  test: ["CMD", "curl", "-fSs", "http://localhost:8008/health"]
-  interval: 1m
-  timeout: 10s
-  retries: 3
-```

docker/conf/log.config

@@ -4,10 +4,16 @@ formatters:
   precise:
    format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s - %(message)s'
 
+filters:
+  context:
+    (): synapse.logging.context.LoggingContextFilter
+    request: ""
+
 handlers:
   console:
     class: logging.StreamHandler
     formatter: precise
+    filters: [context]
 
 loggers:
     synapse.storage.SQL:

docs/admin_api/rooms.md

@@ -275,8 +275,6 @@ The following fields are possible in the JSON response body:
 
 * `room_id` - The ID of the room.
 * `name` - The name of the room.
-* `topic` - The topic of the room.
-* `avatar` - The `mxc` URI to the avatar of the room.
 * `canonical_alias` - The canonical (main) alias address of the room.
 * `joined_members` - How many users are currently in the room.
 * `joined_local_members` - How many local users are currently in the room.
@@ -306,8 +304,6 @@ Response:
 {
   "room_id": "!mscvqgqpHYjBGDxNym:matrix.org",
   "name": "Music Theory",
-  "avatar": "mxc://matrix.org/AQDaVFlbkQoErdOgqWRgiGSV",
-  "topic": "Theory, Composition, Notation, Analysis",
   "canonical_alias": "#musictheory:matrix.org",
   "joined_members": 127
   "joined_local_members": 2,

docs/admin_api/shutdown_room.md

@@ -79,20 +79,13 @@ Response:
 the structure can and does change without notice.
 
 First, it's important to understand that a room shutdown is very destructive. Undoing a shutdown is not as simple as pretending it
-never happened - work has to be done to move forward instead of resetting the past. In fact, in some cases it might not be possible
-to recover at all:
+never happened - work has to be done to move forward instead of resetting the past.
 
-* If the room was invite-only, your users will need to be re-invited.
-* If the room no longer has any members at all, it'll be impossible to rejoin.
-* The first user to rejoin will have to do so via an alias on a different server.
-
-With all that being said, if you still want to try and recover the room:
-
-1. For safety reasons, shut down Synapse.
+1. For safety reasons, it is recommended to shut down Synapse prior to continuing.
 2. In the database, run `DELETE FROM blocked_rooms WHERE room_id = '!example:example.org';`
    * For caution: it's recommended to run this in a transaction: `BEGIN; DELETE ...;`, verify you got 1 result, then `COMMIT;`.
    * The room ID is the same one supplied to the shutdown room API, not the Content Violation room.
-3. Restart Synapse.
+3. Restart Synapse (required).
 
 You will have to manually handle, if you so choose, the following:
 

docs/admin_api/user_admin_api.rst

@@ -108,7 +108,7 @@ The api is::
 
     GET /_synapse/admin/v2/users?from=0&limit=10&guests=false
 
-To use it, you will need to authenticate by providing an ``access_token`` for a
+To use it, you will need to authenticate by providing an `access_token` for a
 server admin: see `README.rst <README.rst>`_.
 
 The parameter ``from`` is optional but used for pagination, denoting the
@@ -119,11 +119,8 @@ from a previous call.
 The parameter ``limit`` is optional but is used for pagination, denoting the
 maximum number of items to return in this call. Defaults to ``100``.
 
-The parameter ``user_id`` is optional and filters to only return users with user IDs
-that contain this value. This parameter is ignored when using the ``name`` parameter.
-
-The parameter ``name`` is optional and filters to only return users with user ID localparts
-**or** displaynames that contain this value.
+The parameter ``user_id`` is optional and filters to only users with user IDs
+that contain this value.
 
 The parameter ``guests`` is optional and if ``false`` will **exclude** guest users.
 Defaults to ``true`` to include guest users.
@@ -214,11 +211,9 @@ Deactivate Account
 
 This API deactivates an account. It removes active access tokens, resets the
 password, and deletes third-party IDs (to prevent the user requesting a
-password reset).
-
-It can also mark the user as GDPR-erased. This means messages sent by the
-user will still be visible by anyone that was in the room when these messages
-were sent, but hidden from users joining the room afterwards.
+password reset). It can also mark the user as GDPR-erased (stopping their data
+from distributed further, and deleting it entirely if there are no other
+references to it).
 
 The api is::
 

docs/federate.md

@@ -47,18 +47,6 @@ you invite them to. This can be caused by an incorrectly-configured reverse
 proxy: see [reverse_proxy.md](<reverse_proxy.md>) for instructions on how to correctly
 configure a reverse proxy.
 
-### Known issues
-
-**HTTP `308 Permanent Redirect` redirects are not followed**: Due to missing features
-in the HTTP library used by Synapse, 308 redirects are currently not followed by
-federating servers, which can cause `M_UNKNOWN` or `401 Unauthorized` errors. This
-may affect users who are redirecting apex-to-www (e.g. `example.com` -> `www.example.com`),
-and especially users of the Kubernetes *Nginx Ingress* module, which uses 308 redirect
-codes by default. For those Kubernetes users, [this Stackoverflow post](https://stackoverflow.com/a/52617528/5096871) 
-might be helpful. For other users, switching to a `301 Moved Permanently` code may be
-an option. 308 redirect codes will be supported properly in a future
-release of Synapse.
-
 ## Running a demo federation of Synapses
 
 If you want to get up and running quickly with a trio of homeservers in a

docs/password_auth_providers.md

@@ -14,7 +14,6 @@ password auth provider module implementations:
 
 * [matrix-synapse-ldap3](https://github.com/matrix-org/matrix-synapse-ldap3/)
 * [matrix-synapse-shared-secret-auth](https://github.com/devture/matrix-synapse-shared-secret-auth)
-* [matrix-synapse-rest-password-provider](https://github.com/ma1uta/matrix-synapse-rest-password-provider)
 
 ## Required methods
 

docs/reverse_proxy.md

@@ -11,7 +11,7 @@ privileges.
 
 **NOTE**: Your reverse proxy must not `canonicalise` or `normalise`
 the requested URI in any way (for example, by decoding `%xx` escapes).
-Beware that Apache *will* canonicalise URIs unless you specify
+Beware that Apache *will* canonicalise URIs unless you specifify
 `nocanon`.
 
 When setting up a reverse proxy, remember that Matrix clients and other
@@ -23,10 +23,6 @@ specification](https://matrix.org/docs/spec/server_server/latest#resolving-serve
 for more details of the algorithm used for federation connections, and
 [delegate.md](<delegate.md>) for instructions on setting up delegation.
 
-Endpoints that are part of the standardised Matrix specification are
-located under `/_matrix`, whereas endpoints specific to Synapse are
-located under `/_synapse/client`.
-
 Let's assume that we expect clients to connect to our server at
 `https://matrix.example.com`, and other servers to connect at
 `https://example.com:8448`.  The following sections detail the configuration of
@@ -49,7 +45,7 @@ server {
 
     server_name matrix.example.com;
 
-    location ~* ^(\/_matrix|\/_synapse\/client) {
+    location /_matrix {
         proxy_pass http://localhost:8008;
         proxy_set_header X-Forwarded-For $remote_addr;
         # Nginx by default only allows file uploads up to 1M in size
@@ -69,10 +65,6 @@ matrix.example.com {
   proxy /_matrix http://localhost:8008 {
     transparent
   }
-
-  proxy /_synapse/client http://localhost:8008 {
-    transparent
-  }
 }
 
 example.com:8448 {
@@ -87,7 +79,6 @@ example.com:8448 {
 ```
 matrix.example.com {
   reverse_proxy /_matrix/* http://localhost:8008
-  reverse_proxy /_synapse/client/* http://localhost:8008
 }
 
 example.com:8448 {
@@ -105,8 +96,6 @@ example.com:8448 {
     AllowEncodedSlashes NoDecode
     ProxyPass /_matrix http://127.0.0.1:8008/_matrix nocanon
     ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix
-    ProxyPass /_synapse/client http://127.0.0.1:8008/_synapse/client nocanon
-    ProxyPassReverse /_synapse/client http://127.0.0.1:8008/_synapse/client
 </VirtualHost>
 
 <VirtualHost *:8448>
@@ -130,7 +119,6 @@ frontend https
   # Matrix client traffic
   acl matrix-host hdr(host) -i matrix.example.com
   acl matrix-path path_beg /_matrix
-  acl matrix-path path_beg /_synapse/client
 
   use_backend matrix if matrix-host matrix-path
 
@@ -151,17 +139,3 @@ client IP addresses are recorded correctly.
 Having done so, you can then use `https://matrix.example.com` (instead
 of `https://matrix.example.com:8448`) as the "Custom server" when
 connecting to Synapse from a client.
-
-
-## Health check endpoint
-
-Synapse exposes a health check endpoint for use by reverse proxies.
-Each configured HTTP listener has a `/health` endpoint which always returns
-200 OK (and doesn't get logged).
-
-## Synapse administration endpoints
-
-Endpoints for administering your Synapse instance are placed under
-`/_synapse/admin`. These require authentication through an access token of an
-admin user. However as access to these endpoints grants the caller a lot of power,
-we do not recommend exposing them to the public internet without good reason.

docs/sample_config.yaml

@@ -378,10 +378,11 @@ retention:
   #  min_lifetime: 1d
   #  max_lifetime: 1y
 
-  # Retention policy limits. If set, and the state of a room contains a
-  # 'm.room.retention' event in its state which contains a 'min_lifetime' or a
-  # 'max_lifetime' that's out of these bounds, Synapse will cap the room's policy
-  # to these limits when running purge jobs.
+  # Retention policy limits. If set, a user won't be able to send a
+  # 'm.room.retention' event which features a 'min_lifetime' or a 'max_lifetime'
+  # that's not within this range. This is especially useful in closed federations,
+  # in which server admins can make sure every federating server applies the same
+  # rules.
   #
   #allowed_lifetime_min: 1d
   #allowed_lifetime_max: 1y
@@ -407,19 +408,12 @@ retention:
   # (e.g. every 12h), but not want that purge to be performed by a job that's
   # iterating over every room it knows, which could be heavy on the server.
   #
-  # If any purge job is configured, it is strongly recommended to have at least
-  # a single job with neither 'shortest_max_lifetime' nor 'longest_max_lifetime'
-  # set, or one job without 'shortest_max_lifetime' and one job without
-  # 'longest_max_lifetime' set. Otherwise some rooms might be ignored, even if
-  # 'allowed_lifetime_min' and 'allowed_lifetime_max' are set, because capping a
-  # room's policy to these values is done after the policies are retrieved from
-  # Synapse's database (which is done using the range specified in a purge job's
-  # configuration).
-  #
   #purge_jobs:
-  #  - longest_max_lifetime: 3d
+  #  - shortest_max_lifetime: 1d
+  #    longest_max_lifetime: 3d
   #    interval: 12h
   #  - shortest_max_lifetime: 3d
+  #    longest_max_lifetime: 1y
   #    interval: 1d
 
 # Inhibits the /requestToken endpoints from returning an error that might leak
@@ -432,24 +426,6 @@ retention:
 #
 #request_token_inhibit_3pid_errors: true
 
-# A list of domains that the domain portion of 'next_link' parameters
-# must match.
-#
-# This parameter is optionally provided by clients while requesting
-# validation of an email or phone number, and maps to a link that
-# users will be automatically redirected to after validation
-# succeeds. Clients can make use this parameter to aid the validation
-# process.
-#
-# The whitelist is applied whether the homeserver or an
-# identity server is handling validation.
-#
-# The default value is no whitelist functionality; all domains are
-# allowed. Setting this value to an empty list will instead disallow
-# all domains.
-#
-#next_link_domain_whitelist: ["matrix.org"]
-
 
 ## TLS ##
 
@@ -1485,14 +1461,11 @@ trusted_key_servers:
 # At least one of `sp_config` or `config_path` must be set in this section to
 # enable SAML login.
 #
-# You will probably also want to set the following options to `false` to
+# (You will probably also want to set the following options to `false` to
 # disable the regular login/registration flows:
 #   * enable_registration
 #   * password_config.enabled
 #
-# You will also want to investigate the settings under the "sso" configuration
-# section below.
-#
 # Once SAML support is enabled, a metadata file will be exposed at
 # https://<server>:<port>/_matrix/saml2/metadata.xml, which you may be able to
 # use to configure your SAML IdP with. Alternatively, you can manually configure
@@ -1604,16 +1577,30 @@ saml2_config:
   #
   #grandfathered_mxid_source_attribute: upn
 
-  # It is possible to configure Synapse to only allow logins if SAML attributes
-  # match particular values. The requirements can be listed under
-  # `attribute_requirements` as shown below. All of the listed attributes must
-  # match for the login to be permitted.
+  # Directory in which Synapse will try to find the template files below.
+  # If not set, default templates from within the Synapse package will be used.
   #
-  #attribute_requirements:
-  #  - attribute: userGroup
-  #    value: "staff"
-  #  - attribute: department
-  #    value: "sales"
+  # DO NOT UNCOMMENT THIS SETTING unless you want to customise the templates.
+  # If you *do* uncomment it, you will need to make sure that all the templates
+  # below are in the directory.
+  #
+  # Synapse will look for the following templates in this directory:
+  #
+  # * HTML page to display to users if something goes wrong during the
+  #   authentication process: 'saml_error.html'.
+  #
+  #   When rendering, this template is given the following variables:
+  #     * code: an HTML error code corresponding to the error that is being
+  #       returned (typically 400 or 500)
+  #
+  #     * msg: a textual message describing the error.
+  #
+  #   The variables will automatically be HTML-escaped.
+  #
+  # You can see the default templates at:
+  # https://github.com/matrix-org/synapse/tree/master/synapse/res/templates
+  #
+  #template_dir: "res/templates"
 
 
 # OpenID Connect integration. The following settings can be used to make Synapse
@@ -2004,7 +1991,9 @@ email:
   # Directory in which Synapse will try to find the template files below.
   # If not set, default templates from within the Synapse package will be used.
   #
-  # Do not uncomment this setting unless you want to customise the templates.
+  # DO NOT UNCOMMENT THIS SETTING unless you want to customise the templates.
+  # If you *do* uncomment it, you will need to make sure that all the templates
+  # below are in the directory.
   #
   # Synapse will look for the following templates in this directory:
   #
@@ -2017,13 +2006,9 @@ email:
   # * The contents of password reset emails sent by the homeserver:
   #   'password_reset.html' and 'password_reset.txt'
   #
-  # * An HTML page that a user will see when they follow the link in the password
-  #   reset email. The user will be asked to confirm the action before their
-  #   password is reset: 'password_reset_confirmation.html'
-  #
-  # * HTML pages for success and failure that a user will see when they confirm
-  #   the password reset flow using the page above: 'password_reset_success.html'
-  #   and 'password_reset_failure.html'
+  # * HTML pages for success and failure that a user will see when they follow
+  #   the link in the password reset email: 'password_reset_success.html' and
+  #   'password_reset_failure.html'
   #
   # * The contents of address verification emails sent during registration:
   #   'registration.html' and 'registration.txt'

docs/sample_log_config.yaml

@@ -11,33 +11,24 @@ formatters:
     precise:
         format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s - %(message)s'
 
+filters:
+    context:
+        (): synapse.logging.context.LoggingContextFilter
+        request: ""
+
 handlers:
     file:
-        class: logging.handlers.TimedRotatingFileHandler
+        class: logging.handlers.RotatingFileHandler
         formatter: precise
         filename: /var/log/matrix-synapse/homeserver.log
-        when: midnight
-        backupCount: 3  # Does not include the current log file.
+        maxBytes: 104857600
+        backupCount: 10
+        filters: [context]
         encoding: utf8
-
-    # Default to buffering writes to log file for efficiency. This means that
-    # will be a delay for INFO/DEBUG logs to get written, but WARNING/ERROR
-    # logs will still be flushed immediately.
-    buffer:
-        class: logging.handlers.MemoryHandler
-        target: file
-        # The capacity is the number of log lines that are buffered before
-        # being written to disk. Increasing this will lead to better
-        # performance, at the expensive of it taking longer for log lines to
-        # be written to disk.
-        capacity: 10
-        flushLevel: 30  # Flush for WARNING logs as well
-
-    # A handler that writes logs to stderr. Unused by default, but can be used
-    # instead of "buffer" and "file" in the logger handlers.
     console:
         class: logging.StreamHandler
         formatter: precise
+        filters: [context]
 
 loggers:
     synapse.storage.SQL:
@@ -45,23 +36,8 @@ loggers:
         # information such as access tokens.
         level: INFO
 
-    twisted:
-        # We send the twisted logging directly to the file handler,
-        # to work around https://github.com/matrix-org/synapse/issues/3471
-        # when using "buffer" logger. Use "console" to log to stderr instead.
-        handlers: [file]
-        propagate: false
-
 root:
     level: INFO
-
-    # Write logs to the `buffer` handler, which will buffer them together in memory,
-    # then write them to a file.
-    #
-    # Replace "buffer" with "console" to log to stderr instead. (Note that you'll
-    # also need to update the configuation for the `twisted` logger above, in
-    # this case.)
-    #
-    handlers: [buffer]
+    handlers: [file, console]
 
 disable_existing_loggers: false

docs/systemd-with-workers/system/matrix-synapse-worker@.service

@@ -1,14 +1,9 @@
 [Unit]
 Description=Synapse %i
 AssertPathExists=/etc/matrix-synapse/workers/%i.yaml
-
 # This service should be restarted when the synapse target is restarted.
 PartOf=matrix-synapse.target
 
-# if this is started at the same time as the main, let the main process start
-# first, to initialise the database schema.
-After=matrix-synapse.service
-
 [Service]
 Type=notify
 NotifyAccess=main

docs/systemd-with-workers/workers/federation_reader.yaml

@@ -1,7 +1,7 @@
 worker_app: synapse.app.federation_reader
-worker_name: federation_reader1
 
 worker_replication_host: 127.0.0.1
+worker_replication_port: 9092
 worker_replication_http_port: 9093
 
 worker_listeners:

docs/user_directory.md

@@ -7,6 +7,6 @@ who are present in a publicly viewable room present on the server.
 
 The directory info is stored in various tables, which can (typically after
 DB corruption) get stale or out of sync.  If this happens, for now the
-solution to fix it is to execute the SQL [here](../synapse/storage/databases/main/schema/delta/53/user_dir_populate.sql)
+solution to fix it is to execute the SQL [here](../synapse/storage/data_stores/main/schema/delta/53/user_dir_populate.sql)
 and then restart synapse. This should then start a background task to
 flush the current tables and regenerate the directory.