Proof-of-concept for using jsonschema on an endpoint.

.buildkite/worker-blacklist

@@ -1,10 +1,41 @@
 # This file serves as a blacklist for SyTest tests that we expect will fail in
 # Synapse when run under worker mode. For more details, see sytest-blacklist.
 
+Message history can be paginated
+
 Can re-join room if re-invited
 
+The only membership state included in an initial sync is for all the senders in the timeline
+
+Local device key changes get to remote servers
+
+If remote user leaves room we no longer receive device updates
+
+Forgotten room messages cannot be paginated
+
+Inbound federation can get public room list
+
+Members from the gap are included in gappy incr LL sync
+
+Leaves are present in non-gapped incremental syncs
+
+Old leaves are present in gapped incremental syncs
+
+User sees updates to presence from other users in the incremental sync.
+
+Gapped incremental syncs include all state changes
+
+Old members are included in gappy incr LL sync if they start speaking
+
 # new failures as of https://github.com/matrix-org/sytest/pull/732
 Device list doesn't change if remote server is down
+Remote servers cannot set power levels in rooms without existing powerlevels
+Remote servers should reject attempts by non-creators to set the power levels
 
 # https://buildkite.com/matrix-dot-org/synapse/builds/6134#6f67bf47-e234-474d-80e8-c6e1868b15c5
 Server correctly handles incoming m.device_list_update
+
+# this fails reliably with a torture level of 100 due to https://github.com/matrix-org/synapse/issues/6536
+Outbound federation requests missing prev_events and then asks for /state_ids and resolves the state
+
+Can get rooms/{roomId}/members at a given point

.circleci/config.yml

@@ -1,36 +1,22 @@
-version: 2.1
+version: 2
 jobs:
   dockerhubuploadrelease:
-    docker:
-      - image: docker:git
+    machine: true
     steps:
       - checkout
-      - setup_remote_docker
-      - docker_prepare
+      - run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:${CIRCLE_TAG} .
       - run: docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
-      - docker_build:
-          tag: -t matrixdotorg/synapse:${CIRCLE_TAG}
-          platforms: linux/amd64
-      - docker_build:
-          tag: -t matrixdotorg/synapse:${CIRCLE_TAG}
-          platforms: linux/amd64,linux/arm/v7,linux/arm64
-
+      - run: docker push matrixdotorg/synapse:${CIRCLE_TAG}
   dockerhubuploadlatest:
-    docker:
-      - image: docker:git
+    machine: true
     steps:
       - checkout
-      - setup_remote_docker
-      - docker_prepare
+      - run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:latest .
       - run: docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
-      - docker_build:
-          tag: -t matrixdotorg/synapse:latest
-          platforms: linux/amd64
-      - docker_build:
-          tag: -t matrixdotorg/synapse:latest
-          platforms: linux/amd64,linux/arm/v7,linux/arm64
+      - run: docker push matrixdotorg/synapse:latest
 
 workflows:
+  version: 2
   build:
     jobs:
       - dockerhubuploadrelease:
@@ -43,33 +29,3 @@ workflows:
           filters:
             branches:
               only: master
-
-commands:
-  docker_prepare:
-    description: Downloads the buildx cli plugin and enables multiarch images
-    parameters:
-      buildx_version:
-        type: string
-        default: "v0.4.1"
-    steps:
-      - run: apk add --no-cache curl
-      - run: mkdir -vp ~/.docker/cli-plugins/ ~/dockercache
-      - run: curl --silent -L "https://github.com/docker/buildx/releases/download/<< parameters.buildx_version >>/buildx-<< parameters.buildx_version >>.linux-amd64" > ~/.docker/cli-plugins/docker-buildx
-      - run: chmod a+x ~/.docker/cli-plugins/docker-buildx
-      # install qemu links in /proc/sys/fs/binfmt_misc on the docker instance running the circleci job
-      - run: docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
-      # create a context named `builder` for the builds
-      - run: docker context create builder
-      # create a buildx builder using the new context, and set it as the default
-      - run: docker buildx create builder --use
-
-  docker_build:
-    description: Builds and pushed images to dockerhub using buildx
-    parameters:
-      platforms:
-        type: string
-        default: linux/amd64
-      tag:
-        type: string
-    steps:
-      - run: docker buildx build -f docker/Dockerfile --push --platform << parameters.platforms >> --label gitsha1=${CIRCLE_SHA1} << parameters.tag >> --progress=plain .

.gitignore

@@ -21,7 +21,6 @@ _trial_temp*/
 /.python-version
 /*.signing.key
 /env/
-/.venv*/
 /homeserver*.yaml
 /logs
 /media_store/

CHANGES.md

@@ -1,366 +1,3 @@
-Synapse 1.22.1 (2020-10-30)
-===========================
-
-Bugfixes
---------
-
-- Fix a bug where an appservice may not be forwarded events for a room it was recently invited to. Broke in v1.22.0. ([\#8676](https://github.com/matrix-org/synapse/issues/8676))
-- Fix `Object of type frozendict is not JSON serializable` exceptions when using third-party event rules. Broke in v1.22.0. ([\#8678](https://github.com/matrix-org/synapse/issues/8678))
-
-
-Synapse 1.22.0 (2020-10-27)
-===========================
-
-No significant changes.
-
-
-Synapse 1.22.0rc2 (2020-10-26)
-==============================
-
-Bugfixes
---------
-
-- Fix bugs where ephemeral events were not sent to appservices. Broke in v1.22.0rc1. ([\#8648](https://github.com/matrix-org/synapse/issues/8648), [\#8656](https://github.com/matrix-org/synapse/issues/8656))
-- Fix `user_daily_visits` table to not have duplicate rows per user/device due to multiple user agents. Broke in v1.22.0rc1. ([\#8654](https://github.com/matrix-org/synapse/issues/8654))
-
-Synapse 1.22.0rc1 (2020-10-22)
-==============================
-
-Features
---------
-
-- Add a configuration option for always using the "userinfo endpoint" for OpenID Connect. This fixes support for some identity providers, e.g. GitLab. Contributed by Benjamin Koch. ([\#7658](https://github.com/matrix-org/synapse/issues/7658))
-- Add ability for `ThirdPartyEventRules` modules to query and manipulate whether a room is in the public rooms directory. ([\#8292](https://github.com/matrix-org/synapse/issues/8292), [\#8467](https://github.com/matrix-org/synapse/issues/8467))
-- Add support for olm fallback keys ([MSC2732](https://github.com/matrix-org/matrix-doc/pull/2732)). ([\#8312](https://github.com/matrix-org/synapse/issues/8312), [\#8501](https://github.com/matrix-org/synapse/issues/8501))
-- Add support for running background tasks in a separate worker process. ([\#8369](https://github.com/matrix-org/synapse/issues/8369), [\#8458](https://github.com/matrix-org/synapse/issues/8458), [\#8489](https://github.com/matrix-org/synapse/issues/8489), [\#8513](https://github.com/matrix-org/synapse/issues/8513), [\#8544](https://github.com/matrix-org/synapse/issues/8544), [\#8599](https://github.com/matrix-org/synapse/issues/8599))
-- Add support for device dehydration ([MSC2697](https://github.com/matrix-org/matrix-doc/pull/2697)). ([\#8380](https://github.com/matrix-org/synapse/issues/8380))
-- Add support for [MSC2409](https://github.com/matrix-org/matrix-doc/pull/2409), which allows sending typing, read receipts, and presence events to appservices. ([\#8437](https://github.com/matrix-org/synapse/issues/8437), [\#8590](https://github.com/matrix-org/synapse/issues/8590))
-- Change default room version to "6", per [MSC2788](https://github.com/matrix-org/matrix-doc/pull/2788). ([\#8461](https://github.com/matrix-org/synapse/issues/8461))
-- Add the ability to send non-membership events into a room via the `ModuleApi`. ([\#8479](https://github.com/matrix-org/synapse/issues/8479))
-- Increase default upload size limit from 10M to 50M. Contributed by @Akkowicz. ([\#8502](https://github.com/matrix-org/synapse/issues/8502))
-- Add support for modifying event content in `ThirdPartyRules` modules. ([\#8535](https://github.com/matrix-org/synapse/issues/8535), [\#8564](https://github.com/matrix-org/synapse/issues/8564))
-
-
-Bugfixes
---------
-
-- Fix a longstanding bug where invalid ignored users in account data could break clients. ([\#8454](https://github.com/matrix-org/synapse/issues/8454))
-- Fix a bug where backfilling a room with an event that was missing the `redacts` field would break. ([\#8457](https://github.com/matrix-org/synapse/issues/8457))
-- Don't attempt to respond to some requests if the client has already disconnected. ([\#8465](https://github.com/matrix-org/synapse/issues/8465))
-- Fix message duplication if something goes wrong after persisting the event. ([\#8476](https://github.com/matrix-org/synapse/issues/8476))
-- Fix incremental sync returning an incorrect `prev_batch` token in timeline section, which when used to paginate returned events that were included in the incremental sync. Broken since v0.16.0. ([\#8486](https://github.com/matrix-org/synapse/issues/8486))
-- Expose the `uk.half-shot.msc2778.login.application_service` to clients from the login API. This feature was added in v1.21.0, but was not exposed as a potential login flow. ([\#8504](https://github.com/matrix-org/synapse/issues/8504))
-- Fix error code for `/profile/{userId}/displayname` to be `M_BAD_JSON`. ([\#8517](https://github.com/matrix-org/synapse/issues/8517))
-- Fix a bug introduced in v1.7.0 that could cause Synapse to insert values from non-state `m.room.retention` events into the `room_retention` database table. ([\#8527](https://github.com/matrix-org/synapse/issues/8527))
-- Fix not sending events over federation when using sharded event writers. ([\#8536](https://github.com/matrix-org/synapse/issues/8536))
-- Fix a long standing bug where email notifications for encrypted messages were blank. ([\#8545](https://github.com/matrix-org/synapse/issues/8545))
-- Fix increase in the number of `There was no active span...` errors logged when using OpenTracing. ([\#8567](https://github.com/matrix-org/synapse/issues/8567))
-- Fix a bug that prevented errors encountered during execution of the `synapse_port_db` from being correctly printed. ([\#8585](https://github.com/matrix-org/synapse/issues/8585))
-- Fix appservice transactions to only include a maximum of 100 persistent and 100 ephemeral events. ([\#8606](https://github.com/matrix-org/synapse/issues/8606))
-
-
-Updates to the Docker image
----------------------------
-
-- Added multi-arch support (arm64,arm/v7) for the docker images. Contributed by @maquis196. ([\#7921](https://github.com/matrix-org/synapse/issues/7921))
-- Add support for passing commandline args to the synapse process. Contributed by @samuel-p. ([\#8390](https://github.com/matrix-org/synapse/issues/8390))
-
-
-Improved Documentation
-----------------------
-
-- Update the directions for using the manhole with coroutines. ([\#8462](https://github.com/matrix-org/synapse/issues/8462))
-- Improve readme by adding new shield.io badges. ([\#8493](https://github.com/matrix-org/synapse/issues/8493))
-- Added note about docker in manhole.md regarding which ip address to bind to. Contributed by @Maquis196. ([\#8526](https://github.com/matrix-org/synapse/issues/8526))
-- Document the new behaviour of the `allowed_lifetime_min` and `allowed_lifetime_max` settings in the room retention configuration. ([\#8529](https://github.com/matrix-org/synapse/issues/8529))
-
-
-Deprecations and Removals
--------------------------
-
-- Drop unused `device_max_stream_id` table. ([\#8589](https://github.com/matrix-org/synapse/issues/8589))
-
-
-Internal Changes
-----------------
-
-- Check for unreachable code with mypy. ([\#8432](https://github.com/matrix-org/synapse/issues/8432))
-- Add unit test for event persister sharding. ([\#8433](https://github.com/matrix-org/synapse/issues/8433))
-- Allow events to be sent to clients sooner when using sharded event persisters. ([\#8439](https://github.com/matrix-org/synapse/issues/8439), [\#8488](https://github.com/matrix-org/synapse/issues/8488), [\#8496](https://github.com/matrix-org/synapse/issues/8496), [\#8499](https://github.com/matrix-org/synapse/issues/8499))
-- Configure `public_baseurl` when using demo scripts. ([\#8443](https://github.com/matrix-org/synapse/issues/8443))
-- Add SQL logging on queries that happen during startup. ([\#8448](https://github.com/matrix-org/synapse/issues/8448))
-- Speed up unit tests when using PostgreSQL. ([\#8450](https://github.com/matrix-org/synapse/issues/8450))
-- Remove redundant database loads of stream_ordering for events we already have. ([\#8452](https://github.com/matrix-org/synapse/issues/8452))
-- Reduce inconsistencies between codepaths for membership and non-membership events. ([\#8463](https://github.com/matrix-org/synapse/issues/8463))
-- Combine `SpamCheckerApi` with the more generic `ModuleApi`. ([\#8464](https://github.com/matrix-org/synapse/issues/8464))
-- Additional testing for `ThirdPartyEventRules`. ([\#8468](https://github.com/matrix-org/synapse/issues/8468))
-- Add `-d` option to `./scripts-dev/lint.sh` to lint files that have changed since the last git commit. ([\#8472](https://github.com/matrix-org/synapse/issues/8472))
-- Unblacklist some sytests. ([\#8474](https://github.com/matrix-org/synapse/issues/8474))
-- Include the log level in the phone home stats. ([\#8477](https://github.com/matrix-org/synapse/issues/8477))
-- Remove outdated sphinx documentation, scripts and configuration. ([\#8480](https://github.com/matrix-org/synapse/issues/8480))
-- Clarify error message when plugin config parsers raise an error. ([\#8492](https://github.com/matrix-org/synapse/issues/8492))
-- Remove the deprecated `Handlers` object. ([\#8494](https://github.com/matrix-org/synapse/issues/8494))
-- Fix a threadsafety bug in unit tests. ([\#8497](https://github.com/matrix-org/synapse/issues/8497))
-- Add user agent to user_daily_visits table. ([\#8503](https://github.com/matrix-org/synapse/issues/8503))
-- Add type hints to various parts of the code base. ([\#8407](https://github.com/matrix-org/synapse/issues/8407), [\#8505](https://github.com/matrix-org/synapse/issues/8505), [\#8507](https://github.com/matrix-org/synapse/issues/8507), [\#8547](https://github.com/matrix-org/synapse/issues/8547), [\#8562](https://github.com/matrix-org/synapse/issues/8562), [\#8609](https://github.com/matrix-org/synapse/issues/8609))
-- Remove unused code from the test framework. ([\#8514](https://github.com/matrix-org/synapse/issues/8514))
-- Apply some internal fixes to the `HomeServer` class to make its code more idiomatic and statically-verifiable. ([\#8515](https://github.com/matrix-org/synapse/issues/8515))
-- Factor out common code between `RoomMemberHandler._locally_reject_invite` and `EventCreationHandler.create_event`. ([\#8537](https://github.com/matrix-org/synapse/issues/8537))
-- Improve database performance by executing more queries without starting transactions. ([\#8542](https://github.com/matrix-org/synapse/issues/8542))
-- Rename `Cache` to `DeferredCache`, to better reflect its purpose. ([\#8548](https://github.com/matrix-org/synapse/issues/8548))
-- Move metric registration code down into `LruCache`. ([\#8561](https://github.com/matrix-org/synapse/issues/8561), [\#8591](https://github.com/matrix-org/synapse/issues/8591))
-- Replace `DeferredCache` with the lighter-weight `LruCache` where possible. ([\#8563](https://github.com/matrix-org/synapse/issues/8563))
-- Add virtualenv-generated folders to `.gitignore`. ([\#8566](https://github.com/matrix-org/synapse/issues/8566))
-- Add `get_immediate` method to `DeferredCache`. ([\#8568](https://github.com/matrix-org/synapse/issues/8568))
-- Fix mypy not properly checking across the codebase, additionally, fix a typing assertion error in `handlers/auth.py`. ([\#8569](https://github.com/matrix-org/synapse/issues/8569))
-- Fix `synmark` benchmark runner. ([\#8571](https://github.com/matrix-org/synapse/issues/8571))
-- Modify `DeferredCache.get()` to return `Deferred`s instead of `ObservableDeferred`s. ([\#8572](https://github.com/matrix-org/synapse/issues/8572))
-- Adjust a protocol-type definition to fit `sqlite3` assertions. ([\#8577](https://github.com/matrix-org/synapse/issues/8577))
-- Support macOS on the `synmark` benchmark runner. ([\#8578](https://github.com/matrix-org/synapse/issues/8578))
-- Update `mypy` static type checker to 0.790. ([\#8583](https://github.com/matrix-org/synapse/issues/8583), [\#8600](https://github.com/matrix-org/synapse/issues/8600))
-- Re-organize the structured logging code to separate the TCP transport handling from the JSON formatting. ([\#8587](https://github.com/matrix-org/synapse/issues/8587))
-- Remove extraneous unittest logging decorators from unit tests. ([\#8592](https://github.com/matrix-org/synapse/issues/8592))
-- Minor optimisations in caching code. ([\#8593](https://github.com/matrix-org/synapse/issues/8593), [\#8594](https://github.com/matrix-org/synapse/issues/8594))
-
-
-Synapse 1.21.2 (2020-10-15)
-===========================
-
-Debian packages and Docker images have been rebuilt using the latest versions of dependency libraries, including authlib 0.15.1. Please see bugfixes below.
-
-Security advisory
------------------
-
-* HTML pages served via Synapse were vulnerable to cross-site scripting (XSS)
-  attacks. All server administrators are encouraged to upgrade.
-  ([\#8444](https://github.com/matrix-org/synapse/pull/8444))
-  ([CVE-2020-26891](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-26891))
-
-  This fix was originally included in v1.21.0 but was missing a security advisory.
-
-  This was reported by [Denis Kasak](https://github.com/dkasak).
-
-Bugfixes
---------
-
-- Fix rare bug where sending an event would fail due to a racey assertion. ([\#8530](https://github.com/matrix-org/synapse/issues/8530))
-- An updated version of the authlib dependency is included in the Docker and Debian images to fix an issue using OpenID Connect. See [\#8534](https://github.com/matrix-org/synapse/issues/8534) for details.
-
-
-Synapse 1.21.1 (2020-10-13)
-===========================
-
-This release fixes a regression in v1.21.0 that prevented debian packages from being built.
-It is otherwise identical to v1.21.0.
-
-Synapse 1.21.0 (2020-10-12)
-===========================
-
-No significant changes since v1.21.0rc3.
-
-As [noted in
-v1.20.0](https://github.com/matrix-org/synapse/blob/release-v1.21.0/CHANGES.md#synapse-1200-2020-09-22),
-a future release will drop support for accessing Synapse's
-[Admin API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api) under the
-`/_matrix/client/*` endpoint prefixes. At that point, the Admin API will only
-be accessible under `/_synapse/admin`.
-
-
-Synapse 1.21.0rc3 (2020-10-08)
-==============================
-
-Bugfixes
---------
-
-- Fix duplication of events on high traffic servers, caused by PostgreSQL `could not serialize access due to concurrent update` errors. ([\#8456](https://github.com/matrix-org/synapse/issues/8456))
-
-
-Internal Changes
-----------------
-
-- Add Groovy Gorilla to the list of distributions we build `.deb`s for. ([\#8475](https://github.com/matrix-org/synapse/issues/8475))
-
-
-Synapse 1.21.0rc2 (2020-10-02)
-==============================
-
-Features
---------
-
-- Convert additional templates from inline HTML to Jinja2 templates. ([\#8444](https://github.com/matrix-org/synapse/issues/8444))
-
-Bugfixes
---------
-
-- Fix a regression in v1.21.0rc1 which broke thumbnails of remote media. ([\#8438](https://github.com/matrix-org/synapse/issues/8438))
-- Do not expose the experimental `uk.half-shot.msc2778.login.application_service` flow in the login API, which caused a compatibility problem with Element iOS. ([\#8440](https://github.com/matrix-org/synapse/issues/8440))
-- Fix malformed log line in new federation "catch up" logic. ([\#8442](https://github.com/matrix-org/synapse/issues/8442))
-- Fix DB query on startup for negative streams which caused long start up times. Introduced in [\#8374](https://github.com/matrix-org/synapse/issues/8374). ([\#8447](https://github.com/matrix-org/synapse/issues/8447))
-
-
-Synapse 1.21.0rc1 (2020-10-01)
-==============================
-
-Features
---------
-
-- Require the user to confirm that their password should be reset after clicking the email confirmation link. ([\#8004](https://github.com/matrix-org/synapse/issues/8004))
-- Add an admin API `GET /_synapse/admin/v1/event_reports` to read entries of table `event_reports`. Contributed by @dklimpel. ([\#8217](https://github.com/matrix-org/synapse/issues/8217))
-- Consolidate the SSO error template across all configuration. ([\#8248](https://github.com/matrix-org/synapse/issues/8248), [\#8405](https://github.com/matrix-org/synapse/issues/8405))
-- Add a configuration option to specify a whitelist of domains that a user can be redirected to after validating their email or phone number. ([\#8275](https://github.com/matrix-org/synapse/issues/8275), [\#8417](https://github.com/matrix-org/synapse/issues/8417))
-- Add experimental support for sharding event persister. ([\#8294](https://github.com/matrix-org/synapse/issues/8294), [\#8387](https://github.com/matrix-org/synapse/issues/8387), [\#8396](https://github.com/matrix-org/synapse/issues/8396), [\#8419](https://github.com/matrix-org/synapse/issues/8419))
-- Add the room topic and avatar to the room details admin API. ([\#8305](https://github.com/matrix-org/synapse/issues/8305))
-- Add an admin API for querying rooms where a user is a member. Contributed by @dklimpel. ([\#8306](https://github.com/matrix-org/synapse/issues/8306))
-- Add `uk.half-shot.msc2778.login.application_service` login type to allow appservices to login. ([\#8320](https://github.com/matrix-org/synapse/issues/8320))
-- Add a configuration option that allows existing users to log in with OpenID Connect. Contributed by @BBBSnowball and @OmmyZhang. ([\#8345](https://github.com/matrix-org/synapse/issues/8345))
-- Add prometheus metrics for replication requests. ([\#8406](https://github.com/matrix-org/synapse/issues/8406))
-- Support passing additional single sign-on parameters to the client. ([\#8413](https://github.com/matrix-org/synapse/issues/8413))
-- Add experimental reporting of metrics on expensive rooms for state-resolution. ([\#8420](https://github.com/matrix-org/synapse/issues/8420))
-- Add experimental prometheus metric to track numbers of "large" rooms for state resolutiom. ([\#8425](https://github.com/matrix-org/synapse/issues/8425))
-- Add prometheus metrics to track federation delays. ([\#8430](https://github.com/matrix-org/synapse/issues/8430))
-
-
-Bugfixes
---------
-
-- Fix a bug in the media repository where remote thumbnails with the same size but different crop methods would overwrite each other. Contributed by @deepbluev7. ([\#7124](https://github.com/matrix-org/synapse/issues/7124))
-- Fix inconsistent handling of non-existent push rules, and stop tracking the `enabled` state of removed push rules. ([\#7796](https://github.com/matrix-org/synapse/issues/7796))
-- Fix a longstanding bug when storing a media file with an empty `upload_name`. ([\#7905](https://github.com/matrix-org/synapse/issues/7905))
-- Fix messages not being sent over federation until an event is sent into the same room. ([\#8230](https://github.com/matrix-org/synapse/issues/8230), [\#8247](https://github.com/matrix-org/synapse/issues/8247), [\#8258](https://github.com/matrix-org/synapse/issues/8258), [\#8272](https://github.com/matrix-org/synapse/issues/8272), [\#8322](https://github.com/matrix-org/synapse/issues/8322))
-- Fix a longstanding bug where files that could not be thumbnailed would result in an Internal Server Error. ([\#8236](https://github.com/matrix-org/synapse/issues/8236), [\#8435](https://github.com/matrix-org/synapse/issues/8435))
-- Upgrade minimum version of `canonicaljson` to version 1.4.0, to fix an unicode encoding issue. ([\#8262](https://github.com/matrix-org/synapse/issues/8262))
-- Fix longstanding bug which could lead to incomplete database upgrades on SQLite. ([\#8265](https://github.com/matrix-org/synapse/issues/8265))
-- Fix stack overflow when stderr is redirected to the logging system, and the logging system encounters an error. ([\#8268](https://github.com/matrix-org/synapse/issues/8268))
-- Fix a bug which cause the logging system to report errors, if `DEBUG` was enabled and no `context` filter was applied. ([\#8278](https://github.com/matrix-org/synapse/issues/8278))
-- Fix edge case where push could get delayed for a user until a later event was pushed. ([\#8287](https://github.com/matrix-org/synapse/issues/8287))
-- Fix fetching malformed events from remote servers. ([\#8324](https://github.com/matrix-org/synapse/issues/8324))
-- Fix `UnboundLocalError` from occuring when appservices send a malformed register request. ([\#8329](https://github.com/matrix-org/synapse/issues/8329))
-- Don't send push notifications to expired user accounts. ([\#8353](https://github.com/matrix-org/synapse/issues/8353))
-- Fix a regression in v1.19.0 with reactivating users through the admin API. ([\#8362](https://github.com/matrix-org/synapse/issues/8362))
-- Fix a bug where during device registration the length of the device name wasn't limited. ([\#8364](https://github.com/matrix-org/synapse/issues/8364))
-- Include `guest_access` in the fields that are checked for null bytes when updating `room_stats_state`. Broke in v1.7.2. ([\#8373](https://github.com/matrix-org/synapse/issues/8373))
-- Fix theoretical race condition where events are not sent down `/sync` if the synchrotron worker is restarted without restarting other workers. ([\#8374](https://github.com/matrix-org/synapse/issues/8374))
-- Fix a bug which could cause errors in rooms with malformed membership events, on servers using sqlite. ([\#8385](https://github.com/matrix-org/synapse/issues/8385))
-- Fix "Re-starting finished log context" warning when receiving an event we already had over federation. ([\#8398](https://github.com/matrix-org/synapse/issues/8398))
-- Fix incorrect handling of timeouts on outgoing HTTP requests. ([\#8400](https://github.com/matrix-org/synapse/issues/8400))
-- Fix a regression in v1.20.0 in the `synapse_port_db` script regarding the `ui_auth_sessions_ips` table. ([\#8410](https://github.com/matrix-org/synapse/issues/8410))
-- Remove unnecessary 3PID registration check when resetting password via an email address. Bug introduced in v0.34.0rc2. ([\#8414](https://github.com/matrix-org/synapse/issues/8414))
-
-
-Improved Documentation
-----------------------
-
-- Add `/_synapse/client` to the reverse proxy documentation. ([\#8227](https://github.com/matrix-org/synapse/issues/8227))
-- Add note to the reverse proxy settings documentation about disabling Apache's mod_security2. Contributed by Julian Fietkau (@jfietkau). ([\#8375](https://github.com/matrix-org/synapse/issues/8375))
-- Improve description of `server_name` config option in `homserver.yaml`. ([\#8415](https://github.com/matrix-org/synapse/issues/8415))
-
-
-Deprecations and Removals
--------------------------
-
-- Drop support for `prometheus_client` older than 0.4.0. ([\#8426](https://github.com/matrix-org/synapse/issues/8426))
-
-
-Internal Changes
-----------------
-
-- Fix tests on distros which disable TLSv1.0. Contributed by @danc86. ([\#8208](https://github.com/matrix-org/synapse/issues/8208))
-- Simplify the distributor code to avoid unnecessary work. ([\#8216](https://github.com/matrix-org/synapse/issues/8216))
-- Remove the `populate_stats_process_rooms_2` background job and restore functionality to `populate_stats_process_rooms`. ([\#8243](https://github.com/matrix-org/synapse/issues/8243))
-- Clean up type hints for `PaginationConfig`. ([\#8250](https://github.com/matrix-org/synapse/issues/8250), [\#8282](https://github.com/matrix-org/synapse/issues/8282))
-- Track the latest event for every destination and room for catch-up after federation outage. ([\#8256](https://github.com/matrix-org/synapse/issues/8256))
-- Fix non-user visible bug in implementation of `MultiWriterIdGenerator.get_current_token_for_writer`. ([\#8257](https://github.com/matrix-org/synapse/issues/8257))
-- Switch to the JSON implementation from the standard library. ([\#8259](https://github.com/matrix-org/synapse/issues/8259))
-- Add type hints to `synapse.util.async_helpers`. ([\#8260](https://github.com/matrix-org/synapse/issues/8260))
-- Simplify tests that mock asynchronous functions. ([\#8261](https://github.com/matrix-org/synapse/issues/8261))
-- Add type hints to `StreamToken` and `RoomStreamToken` classes. ([\#8279](https://github.com/matrix-org/synapse/issues/8279))
-- Change `StreamToken.room_key` to be a `RoomStreamToken` instance. ([\#8281](https://github.com/matrix-org/synapse/issues/8281))
-- Refactor notifier code to correctly use the max event stream position. ([\#8288](https://github.com/matrix-org/synapse/issues/8288))
-- Use slotted classes where possible. ([\#8296](https://github.com/matrix-org/synapse/issues/8296))
-- Support testing the local Synapse checkout against the [Complement homeserver test suite](https://github.com/matrix-org/complement/). ([\#8317](https://github.com/matrix-org/synapse/issues/8317))
-- Update outdated usages of `metaclass` to python 3 syntax. ([\#8326](https://github.com/matrix-org/synapse/issues/8326))
-- Move lint-related dependencies to package-extra field, update CONTRIBUTING.md to utilise this. ([\#8330](https://github.com/matrix-org/synapse/issues/8330), [\#8377](https://github.com/matrix-org/synapse/issues/8377))
-- Use the `admin_patterns` helper in additional locations. ([\#8331](https://github.com/matrix-org/synapse/issues/8331))
-- Fix test logging to allow braces in log output. ([\#8335](https://github.com/matrix-org/synapse/issues/8335))
-- Remove `__future__` imports related to Python 2 compatibility. ([\#8337](https://github.com/matrix-org/synapse/issues/8337))
-- Simplify `super()` calls to Python 3 syntax. ([\#8344](https://github.com/matrix-org/synapse/issues/8344))
-- Fix bad merge from `release-v1.20.0` branch to `develop`. ([\#8354](https://github.com/matrix-org/synapse/issues/8354))
-- Factor out a `_send_dummy_event_for_room` method. ([\#8370](https://github.com/matrix-org/synapse/issues/8370))
-- Improve logging of state resolution. ([\#8371](https://github.com/matrix-org/synapse/issues/8371))
-- Add type annotations to `SimpleHttpClient`. ([\#8372](https://github.com/matrix-org/synapse/issues/8372))
-- Refactor ID generators to use `async with` syntax. ([\#8383](https://github.com/matrix-org/synapse/issues/8383))
-- Add `EventStreamPosition` type. ([\#8388](https://github.com/matrix-org/synapse/issues/8388))
-- Create a mechanism for marking tests "logcontext clean". ([\#8399](https://github.com/matrix-org/synapse/issues/8399))
-- A pair of tiny cleanups in the federation request code. ([\#8401](https://github.com/matrix-org/synapse/issues/8401))
-- Add checks on startup that PostgreSQL sequences are consistent with their associated tables. ([\#8402](https://github.com/matrix-org/synapse/issues/8402))
-- Do not include appservice users when calculating the total MAU for a server. ([\#8404](https://github.com/matrix-org/synapse/issues/8404))
-- Typing fixes for `synapse.handlers.federation`. ([\#8422](https://github.com/matrix-org/synapse/issues/8422))
-- Various refactors to simplify stream token handling. ([\#8423](https://github.com/matrix-org/synapse/issues/8423))
-- Make stream token serializing/deserializing async. ([\#8427](https://github.com/matrix-org/synapse/issues/8427))
-
-
-Synapse 1.20.1 (2020-09-24)
-===========================
-
-Bugfixes
---------
-
-- Fix a bug introduced in v1.20.0 which caused the `synapse_port_db` script to fail. ([\#8386](https://github.com/matrix-org/synapse/issues/8386))
-- Fix a bug introduced in v1.20.0 which caused variables to be incorrectly escaped in Jinja2 templates. ([\#8394](https://github.com/matrix-org/synapse/issues/8394))
-
-
-Synapse 1.20.0 (2020-09-22)
-===========================
-
-No significant changes since v1.20.0rc5.
-
-Removal warning
----------------
-
-Historically, the [Synapse Admin
-API](https://github.com/matrix-org/synapse/tree/master/docs) has been
-accessible under the `/_matrix/client/api/v1/admin`,
-`/_matrix/client/unstable/admin`, `/_matrix/client/r0/admin` and
-`/_synapse/admin` prefixes. In a future release, we will be dropping support
-for accessing Synapse's Admin API using the `/_matrix/client/*` prefixes.
-
-From that point, the Admin API will only be accessible under `/_synapse/admin`.
-This makes it easier for homeserver admins to lock down external access to the
-Admin API endpoints.
-
-Synapse 1.20.0rc5 (2020-09-18)
-==============================
-
-In addition to the below, Synapse 1.20.0rc5 also includes the bug fix that was included in 1.19.3.
-
-Features
---------
-
-- Add flags to the `/versions` endpoint for whether new rooms default to using E2EE. ([\#8343](https://github.com/matrix-org/synapse/issues/8343))
-
-
-Bugfixes
---------
-
-- Fix rate limiting of federation `/send` requests. ([\#8342](https://github.com/matrix-org/synapse/issues/8342))
-- Fix a longstanding bug where back pagination over federation could get stuck if it failed to handle a received event. ([\#8349](https://github.com/matrix-org/synapse/issues/8349))
-
-
-Internal Changes
-----------------
-
-- Blacklist [MSC2753](https://github.com/matrix-org/matrix-doc/pull/2753) SyTests until it is implemented. ([\#8285](https://github.com/matrix-org/synapse/issues/8285))
-
-
-Synapse 1.19.3 (2020-09-18)
-===========================
-
-Bugfixes
---------
-
-- Partially mitigate bug where newly joined servers couldn't get past events in a room when there is a malformed event. ([\#8350](https://github.com/matrix-org/synapse/issues/8350))
-
-
 Synapse 1.20.0rc4 (2020-09-16)
 ==============================
 

CONTRIBUTING.md

@@ -17,9 +17,9 @@ https://help.github.com/articles/using-pull-requests/) to ask us to pull your
 changes into our repo.
 
 Some other points to follow:
-
+ 
  * Please base your changes on the `develop` branch.
-
+  
  * Please follow the [code style requirements](#code-style).
 
  * Please include a [changelog entry](#changelog) with each PR.
@@ -46,7 +46,7 @@ locally. You'll need python 3.6 or later, and to install a number of tools:
 
 ```
 # Install the dependencies
-pip install -e ".[lint,mypy]"
+pip install -U black flake8 flake8-comprehensions isort
 
 # Run the linter script
 ./scripts-dev/lint.sh
@@ -63,10 +63,6 @@ run-time:
 ./scripts-dev/lint.sh path/to/file1.py path/to/file2.py path/to/folder
 ```
 
-You can also provide the `-d` option, which will lint the files that have been
-changed since the last git commit. This will often be significantly faster than
-linting the whole codebase.
-
 Before pushing new changes, ensure they don't produce linting errors. Commit any
 files that were corrected.
 

INSTALL.md

@@ -57,7 +57,7 @@ light workloads.
 System requirements:
 
 - POSIX-compliant system (tested on Linux & OS X)
-- Python 3.5.2 or later, up to Python 3.9.
+- Python 3.5.2 or later, up to Python 3.8.
 - At least 1GB of free RAM if you want to join large public rooms like #matrix:matrix.org
 
 Synapse is written in Python but some of the libraries it uses are written in

README.rst

@@ -1,6 +1,10 @@
-=========================================================
-Synapse |support| |development| |license| |pypi| |python|
-=========================================================
+================
+Synapse |shield|
+================
+
+.. |shield| image:: https://img.shields.io/matrix/synapse:matrix.org?label=support&logo=matrix
+  :alt: (get support on #synapse:matrix.org)
+  :target: https://matrix.to/#/#synapse:matrix.org
 
 .. contents::
 
@@ -256,9 +260,9 @@ directory of your choice::
 Synapse has a number of external dependencies, that are easiest
 to install using pip and a virtualenv::
 
-    python3 -m venv ./env
-    source ./env/bin/activate
-    pip install -e ".[all,test]"
+    virtualenv -p python3 env
+    source env/bin/activate
+    python -m pip install --no-use-pep517 -e ".[all]"
 
 This will run a process of downloading and installing all the needed
 dependencies into a virtual env.
@@ -270,9 +274,9 @@ check that everything is installed as it should be::
 
 This should end with a 'PASSED' result::
 
-    Ran 1266 tests in 643.930s
+    Ran 143 tests in 0.601s
 
-    PASSED (skips=15, successes=1251)
+    PASSED (successes=143)
 
 Running the Integration Tests
 =============================
@@ -286,6 +290,19 @@ Testing with SyTest is recommended for verifying that changes related to the
 Client-Server API are functioning correctly. See the `installation instructions
 <https://github.com/matrix-org/sytest#installing>`_ for details.
 
+Building Internal API Documentation
+===================================
+
+Before building internal API documentation install sphinx and
+sphinxcontrib-napoleon::
+
+    pip install sphinx
+    pip install sphinxcontrib-napoleon
+
+Building internal API documentation::
+
+    python setup.py build_sphinx
+
 Troubleshooting
 ===============
 
@@ -370,23 +387,3 @@ something like the following in their logs::
 
 This is normally caused by a misconfiguration in your reverse-proxy. See
 `<docs/reverse_proxy.md>`_ and double-check that your settings are correct.
-
-.. |support| image:: https://img.shields.io/matrix/synapse:matrix.org?label=support&logo=matrix
-  :alt: (get support on #synapse:matrix.org)
-  :target: https://matrix.to/#/#synapse:matrix.org
-
-.. |development| image:: https://img.shields.io/matrix/synapse-dev:matrix.org?label=development&logo=matrix
-  :alt: (discuss development on #synapse-dev:matrix.org)
-  :target: https://matrix.to/#/#synapse-dev:matrix.org
-
-.. |license| image:: https://img.shields.io/github/license/matrix-org/synapse
-  :alt: (check license in LICENSE file)
-  :target: LICENSE
-
-.. |pypi| image:: https://img.shields.io/pypi/v/matrix-synapse
-  :alt: (latest version released on PyPi)
-  :target: https://pypi.org/project/matrix-synapse
-
-.. |python| image:: https://img.shields.io/pypi/pyversions/matrix-synapse
-  :alt: (supported python versions)
-  :target: https://pypi.org/project/matrix-synapse

UPGRADE.rst

@@ -75,39 +75,6 @@ for example:
      wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb
      dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
 
-Upgrading to v1.23.0
-====================
-
-Structured logging configuration breaking changes
--------------------------------------------------
-
-This release deprecates use of the ``structured: true`` logging configuration for
-structured logging. If your logging configuration contains ``structured: true``
-then it should be modified based on the `structured logging documentation
-<https://github.com/matrix-org/synapse/blob/master/docs/structured_logging.md>`_.
-
-The ``structured`` and ``drains`` logging options are now deprecated and should
-be replaced by standard logging configuration of ``handlers`` and ``formatters`.
-
-A future will release of Synapse will make using ``structured: true`` an error.
-
-Upgrading to v1.22.0
-====================
-
-ThirdPartyEventRules breaking changes
--------------------------------------
-
-This release introduces a backwards-incompatible change to modules making use of
-``ThirdPartyEventRules`` in Synapse. If you make use of a module defined under the
-``third_party_event_rules`` config option, please make sure it is updated to handle
-the below change:
-
-The ``http_client`` argument is no longer passed to modules as they are initialised. Instead,
-modules are expected to make use of the ``http_client`` property on the ``ModuleApi`` class.
-Modules are now passed a ``module_api`` argument during initialisation, which is an instance of
-``ModuleApi``. ``ModuleApi`` instances have a ``http_client`` property which acts the same as
-the ``http_client`` argument previously passed to ``ThirdPartyEventRules`` modules.
-
 Upgrading to v1.21.0
 ====================
 

changelog.d/7124.bugfix

@@ -0,0 +1 @@
+Fix a bug in the media repository where remote thumbnails with the same size but different crop methods would overwrite each other. Contributed by @deepbluev7.

changelog.d/7796.bugfix

@@ -0,0 +1 @@
+Fix inconsistent handling of non-existent push rules, and stop tracking the `enabled` state of removed push rules.

changelog.d/8004.feature

@@ -0,0 +1 @@
+Require the user to confirm that their password should be reset after clicking the email confirmation link.

changelog.d/8208.misc

@@ -0,0 +1 @@
+Fix tests on distros which disable TLSv1.0. Contributed by @danc86.

changelog.d/8216.misc

@@ -0,0 +1 @@
+Simplify the distributor code to avoid unnecessary work.

changelog.d/8227.doc

@@ -0,0 +1 @@
+Add `/_synapse/client` to the reverse proxy documentation.

changelog.d/8230.misc

@@ -0,0 +1 @@
+Track the latest event for every destination and room for catch-up after federation outage.

changelog.d/8236.bugfix

@@ -0,0 +1 @@
+Fix a longstanding bug where files that could not be thumbnailed would result in an Internal Server Error.

changelog.d/8243.misc

@@ -0,0 +1 @@
+Remove the 'populate_stats_process_rooms_2' background job and restore functionality to 'populate_stats_process_rooms'.

changelog.d/8247.misc

@@ -0,0 +1 @@
+Track the `stream_ordering` of the last successfully-sent event to every destination, so we can use this information to 'catch up' a remote server after an outage.

changelog.d/8248.feature

@@ -0,0 +1 @@
+Consolidate the SSO error template across all configuration.

changelog.d/8250.misc

@@ -0,0 +1 @@
+Clean up type hints for `PaginationConfig`.

changelog.d/8256.misc

@@ -0,0 +1 @@
+Track the latest event for every destination and room for catch-up after federation outage.

changelog.d/8257.misc

@@ -0,0 +1 @@
+Fix non-user visible bug in implementation of `MultiWriterIdGenerator.get_current_token_for_writer`.

changelog.d/8258.misc

@@ -0,0 +1 @@
+Track the `stream_ordering` of the last successfully-sent event to every destination, so we can use this information to 'catch up' a remote server after an outage.

changelog.d/8259.misc

@@ -0,0 +1 @@
+Switch to the JSON implementation from the standard library.

changelog.d/8260.misc

@@ -0,0 +1 @@
+Add type hints to `synapse.util.async_helpers`.

changelog.d/8261.misc

@@ -0,0 +1 @@
+Simplify tests that mock asynchronous functions.

changelog.d/8262.bugfix

@@ -0,0 +1 @@
+Upgrade canonicaljson to version 1.4.0 to fix an unicode encoding issue.

changelog.d/8265.bugfix

@@ -0,0 +1 @@
+Fix logstanding bug which could lead to incomplete database upgrades on SQLite.

changelog.d/8268.bugfix

@@ -0,0 +1 @@
+Fix stack overflow when stderr is redirected to the logging system, and the logging system encounters an error.

changelog.d/8272.bugfix

@@ -0,0 +1 @@
+Fix messages over federation being lost until an event is sent into the same room.

changelog.d/8275.feature

@@ -0,0 +1 @@
+Add a config option to specify a whitelist of domains that a user can be redirected to after validating their email or phone number.

changelog.d/8278.bugfix

@@ -0,0 +1 @@
+Fix a bug which cause the logging system to report errors, if `DEBUG` was enabled and no `context` filter was applied.

changelog.d/8279.misc

@@ -0,0 +1 @@
+Add type hints to `StreamToken` and `RoomStreamToken` classes.

changelog.d/8281.misc

@@ -0,0 +1 @@
+Change `StreamToken.room_key` to be a `RoomStreamToken` instance.

changelog.d/8282.misc

@@ -0,0 +1 @@
+Clean up type hints for `PaginationConfig`.

changelog.d/8285.misc

@@ -0,0 +1 @@
+Blacklist [MSC2753](https://github.com/matrix-org/matrix-doc/pull/2753) SyTests until it is implemented.

changelog.d/8287.bugfix

@@ -0,0 +1 @@
+Fix edge case where push could get delayed for a user until a later event was pushed.

changelog.d/8288.misc

@@ -0,0 +1 @@
+Refactor notifier code to correctly use the max event stream position.

changelog.d/8294.feature

@@ -0,0 +1 @@
+Add experimental support for sharding event persister.

changelog.d/8296.misc

@@ -0,0 +1 @@
+Use slotted classes where possible.

changelog.d/8305.feature

@@ -0,0 +1 @@
+Add the room topic and avatar to the room details admin API.

changelog.d/8324.bugfix

@@ -0,0 +1 @@
+Fix fetching events from remote servers that are malformed.

changelog.d/8326.misc

@@ -0,0 +1 @@
+Update outdated usages of `metaclass` to python 3 syntax.

changelog.d/8329.bugfix

@@ -0,0 +1 @@
+Fix UnboundLocalError from occuring when appservices send malformed register request.

changelog.d/8331.misc

@@ -0,0 +1 @@
+Use the `admin_patterns` helper in additional locations.

changelog.d/8337.misc

@@ -0,0 +1 @@
+Remove `__future__` imports related to Python 2 compatibility.

changelog.d/8455.bugfix

@@ -1 +0,0 @@
-Fix fetching of E2E cross signing keys over federation when only one of the master key and device signing key is cached already.

changelog.d/8519.feature

@@ -1 +0,0 @@
-Add an admin api to delete a single file or files were not used for a defined time from server. Contributed by @dklimpel.

changelog.d/8539.feature

@@ -1 +0,0 @@
-Split admin API for reported events (`GET /_synapse/admin/v1/event_reports`) into detail and list endpoints. This is a breaking change to #8217 which was introduced in Synapse v1.21.0. Those who already use this API should check their scripts. Contributed by @dklimpel.

changelog.d/8559.misc

@@ -1 +0,0 @@
-Optimise `/createRoom` with multiple invited users.

changelog.d/8580.bugfix

@@ -1 +0,0 @@
-Fix a bug where Synapse would blindly forward bad responses from federation to clients when retrieving profile information.

changelog.d/8582.doc

@@ -1 +0,0 @@
-Instructions for Azure AD in the OpenID Connect documentation. Contributed by peterk.

changelog.d/8595.misc

@@ -1 +0,0 @@
-Implement and use an @lru_cache decorator.

changelog.d/8607.feature

@@ -1 +0,0 @@
-Support generating structured logs via the standard logging configuration.

changelog.d/8610.feature

@@ -1 +0,0 @@
-Add an admin APIs to allow server admins to list users' pushers. Contributed by @dklimpel.

changelog.d/8614.misc

@@ -1 +0,0 @@
-Don't instansiate Requester directly.

changelog.d/8615.misc

@@ -1 +0,0 @@
-Type hints for `RegistrationStore`.

changelog.d/8616.misc

@@ -1 +0,0 @@
-Change schema to support access tokens belonging to one user but granting access to another.

changelog.d/8620.bugfix

@@ -1 +0,0 @@
-Fix a bug where the account validity endpoint would silently fail if the user ID did not have an expiration time. It now returns a 400 error.

changelog.d/8621.misc

@@ -1 +0,0 @@
-Remove unused OPTIONS handlers.

changelog.d/8627.bugfix

@@ -1 +0,0 @@
-Fix email notifications for invites without local state.

changelog.d/8628.bugfix

@@ -1 +0,0 @@
-Fix handling of invalid group IDs to return a 400 rather than log an exception and return a 500.

changelog.d/8632.bugfix

@@ -1 +0,0 @@
-Fix handling of User-Agent headers that are invalid UTF-8, which caused user agents of users to not get correctly recorded.

changelog.d/8633.misc

@@ -1 +0,0 @@
-Run `mypy` as part of the lint.sh script.

changelog.d/8634.misc

@@ -1 +0,0 @@
-Correct Synapse's PyPI package name in the OpenID Connect installation instructions.

changelog.d/8635.doc

@@ -1 +0,0 @@
-Improve the sample configuration for single sign-on providers.

changelog.d/8639.misc

@@ -1 +0,0 @@
-Fix typos and spelling errors in the code.

changelog.d/8640.misc

@@ -1 +0,0 @@
-Reduce number of OpenTracing spans started.

changelog.d/8643.bugfix

@@ -1 +0,0 @@
-Fix a bug in the `joined_rooms` admin API if the user has never joined any rooms. The bug was introduced, along with the API, in v1.21.0.

changelog.d/8644.misc

@@ -1 +0,0 @@
-Add field `total` to device list in admin API.

changelog.d/8647.feature

@@ -1 +0,0 @@
-Add an admin API `GET /_synapse/admin/v1/users/<user_id>/media` to get information about uploaded media. Contributed by @dklimpel.

changelog.d/8655.misc

@@ -1 +0,0 @@
-Add more type hints to the application services code.

changelog.d/8657.doc

@@ -1 +0,0 @@
-Fix the filepath of Dex's example config and the link to Dex's Getting Started guide in the OpenID Connect docs.

changelog.d/8664.misc

@@ -1 +0,0 @@
-Tell Black to format code for Python 3.5.

changelog.d/8665.doc

@@ -1 +0,0 @@
-Note support for Python 3.9.

changelog.d/8666.doc

@@ -1 +0,0 @@
-Minor updates to docs on running tests.

changelog.d/8667.doc

@@ -1 +0,0 @@
-Interlink prometheus/grafana documentation.

changelog.d/8668.misc

@@ -1 +0,0 @@
-Reduce number of OpenTracing spans started.

changelog.d/8669.misc

@@ -1 +0,0 @@
-Don't pull event from DB when handling replication traffic.

changelog.d/8670.misc

@@ -1 +0,0 @@
-Reduce number of OpenTracing spans started.

changelog.d/8671.misc

@@ -1 +0,0 @@
-Abstract some invite-related code in preparation for landing knocking.

changelog.d/8679.misc

@@ -1 +0,0 @@
-Clarify representation of events in logfiles.

changelog.d/8680.misc

@@ -1 +0,0 @@
-Don't require `hiredis` package to be installed to run unit tests.

changelog.d/8682.bugfix

@@ -1 +0,0 @@
-Fix exception during handling multiple concurrent requests for remote media when using multiple media repositories.

changelog.d/8684.misc

@@ -1 +0,0 @@
-Fix typing info on cache call signature to accept `on_invalidate`.

changelog.d/8685.feature

@@ -1 +0,0 @@
-Support generating structured logs via the standard logging configuration.

changelog.d/8688.misc

@@ -1 +0,0 @@
-Abstract some invite-related code in preparation for landing knocking.

changelog.d/8689.feature

@@ -1 +0,0 @@
-Add an admin APIs to allow server admins to list users' pushers. Contributed by @dklimpel.

changelog.d/8690.misc

@@ -1 +0,0 @@
-Fail tests if they do not await coroutines.

changelog.d/8693.misc

@@ -1 +0,0 @@
-Add more type hints to the application services code.

contrib/grafana/README.md

@@ -3,4 +3,4 @@
 0. Set up Prometheus and Grafana. Out of scope for this readme. Useful documentation about using Grafana with Prometheus: http://docs.grafana.org/features/datasources/prometheus/
 1. Have your Prometheus scrape your Synapse. https://github.com/matrix-org/synapse/blob/master/docs/metrics-howto.md
 2. Import dashboard into Grafana. Download `synapse.json`. Import it to Grafana and select the correct Prometheus datasource. http://docs.grafana.org/reference/export_import/
-3. Set up required recording rules. https://github.com/matrix-org/synapse/tree/master/contrib/prometheus
+3. Set up additional recording rules

debian/build_virtualenv

@@ -42,7 +42,7 @@ dh_virtualenv \
     --preinstall="mock" \
     --extra-pip-arg="--no-cache-dir" \
     --extra-pip-arg="--compile" \
-    --extras="all,systemd,test"
+    --extras="all,systemd"
 
 PACKAGE_BUILD_DIR="debian/matrix-synapse-py3"
 VIRTUALENV_DIR="${PACKAGE_BUILD_DIR}${DH_VIRTUALENV_INSTALL_ROOT}/matrix-synapse"

debian/changelog

@@ -1,59 +1,8 @@
-matrix-synapse-py3 (1.22.1) stable; urgency=medium
+matrix-synapse-py3 (1.20.0ubuntu1) UNRELEASED; urgency=medium
 
-  * New synapse release 1.22.1.
-
- -- Synapse Packaging team <packages@matrix.org>  Fri, 30 Oct 2020 15:25:37 +0000
-
-matrix-synapse-py3 (1.22.0) stable; urgency=medium
-
-  * New synapse release 1.22.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 27 Oct 2020 12:07:12 +0000
-
-matrix-synapse-py3 (1.21.2) stable; urgency=medium
-
-  [ Synapse Packaging team ]
-  * New synapse release 1.21.2.
-
- -- Synapse Packaging team <packages@matrix.org>  Thu, 15 Oct 2020 09:23:27 -0400
-
-matrix-synapse-py3 (1.21.1) stable; urgency=medium
-
-  [ Synapse Packaging team ]
-  * New synapse release 1.21.1.
-
-  [ Andrew Morgan ]
-  * Explicitly install "test" python dependencies.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 13 Oct 2020 10:24:13 +0100
-
-matrix-synapse-py3 (1.21.0) stable; urgency=medium
-
-  * New synapse release 1.21.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Mon, 12 Oct 2020 15:47:44 +0100
-
-matrix-synapse-py3 (1.20.1) stable; urgency=medium
-
-  * New synapse release 1.20.1.
-
- -- Synapse Packaging team <packages@matrix.org>  Thu, 24 Sep 2020 16:25:22 +0100
-
-matrix-synapse-py3 (1.20.0) stable; urgency=medium
-
-  [ Synapse Packaging team ]
-  * New synapse release 1.20.0.
-
-  [ Dexter Chua ]
   * Use Type=notify in systemd service
 
- -- Synapse Packaging team <packages@matrix.org>  Tue, 22 Sep 2020 15:19:32 +0100
-
-matrix-synapse-py3 (1.19.3) stable; urgency=medium
-
-  * New synapse release 1.19.3.
-
- -- Synapse Packaging team <packages@matrix.org>  Fri, 18 Sep 2020 14:59:30 +0100
+ -- Dexter Chua <dec41@srcf.net>  Wed, 26 Aug 2020 12:41:36 +0000
 
 matrix-synapse-py3 (1.19.2) stable; urgency=medium
 

demo/start.sh

@@ -30,8 +30,6 @@ for port in 8080 8081 8082; do
     if ! grep -F "Customisation made by demo/start.sh" -q  $DIR/etc/$port.config; then
         printf '\n\n# Customisation made by demo/start.sh\n' >> $DIR/etc/$port.config
 
-        echo "public_baseurl: http://localhost:$port/" >> $DIR/etc/$port.config
-
         echo 'enable_registration: true' >> $DIR/etc/$port.config
 
         # Warning, this heredoc depends on the interaction of tabs and spaces. Please don't

docker/README.md

@@ -83,7 +83,7 @@ docker logs synapse
 If all is well, you should now be able to connect to http://localhost:8008 and
 see a confirmation message.
 
-The following environment variables are supported in `run` mode:
+The following environment variables are supported in run mode:
 
 * `SYNAPSE_CONFIG_DIR`: where additional config files are stored. Defaults to
   `/data`.
@@ -94,20 +94,6 @@ The following environment variables are supported in `run` mode:
 * `UID`, `GID`: the user and group id to run Synapse as. Defaults to `991`, `991`.
 * `TZ`: the [timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) the container will run with. Defaults to `UTC`.
 
-For more complex setups (e.g. for workers) you can also pass your args directly to synapse using `run` mode. For example like this:
-
-```
-docker run -d --name synapse \
-    --mount type=volume,src=synapse-data,dst=/data \
-    -p 8008:8008 \
-    matrixdotorg/synapse:latest run \
-    -m synapse.app.generic_worker \
-    --config-path=/data/homeserver.yaml \
-    --config-path=/data/generic_worker.yaml
-```
-
-If you do not provide `-m`, the value of the `SYNAPSE_WORKER` environment variable is used. If you do not provide at least one `--config-path` or `-c`, the value of the `SYNAPSE_CONFIG_PATH` environment variable is used instead.
-
 ## Generating an (admin) user
 
 After synapse is running, you may wish to create a user via `register_new_matrix_user`.

docker/conf/homeserver.yaml

@@ -90,7 +90,7 @@ federation_rc_concurrent: 3
 
 media_store_path: "/data/media"
 uploads_path: "/data/uploads"
-max_upload_size: "{{ SYNAPSE_MAX_UPLOAD_SIZE or "50M" }}"
+max_upload_size: "{{ SYNAPSE_MAX_UPLOAD_SIZE or "10M" }}"
 max_image_pixels: "32M"
 dynamic_thumbnails: false
 

docker/start.py

@@ -179,7 +179,7 @@ def run_generate_config(environ, ownership):
 
 
 def main(args, environ):
-    mode = args[1] if len(args) > 1 else "run"
+    mode = args[1] if len(args) > 1 else None
     desired_uid = int(environ.get("UID", "991"))
     desired_gid = int(environ.get("GID", "991"))
     synapse_worker = environ.get("SYNAPSE_WORKER", "synapse.app.homeserver")
@@ -205,47 +205,36 @@ def main(args, environ):
             config_dir, config_path, environ, ownership
         )
 
-    if mode != "run":
+    if mode is not None:
         error("Unknown execution mode '%s'" % (mode,))
 
-    args = args[2:]
+    config_dir = environ.get("SYNAPSE_CONFIG_DIR", "/data")
+    config_path = environ.get("SYNAPSE_CONFIG_PATH", config_dir + "/homeserver.yaml")
 
-    if "-m" not in args:
-        args = ["-m", synapse_worker] + args
-
-    # if there are no config files passed to synapse, try adding the default file
-    if not any(p.startswith("--config-path") or p.startswith("-c") for p in args):
-        config_dir = environ.get("SYNAPSE_CONFIG_DIR", "/data")
-        config_path = environ.get(
-            "SYNAPSE_CONFIG_PATH", config_dir + "/homeserver.yaml"
-        )
-
-        if not os.path.exists(config_path):
-            if "SYNAPSE_SERVER_NAME" in environ:
-                error(
-                    """\
+    if not os.path.exists(config_path):
+        if "SYNAPSE_SERVER_NAME" in environ:
+            error(
+                """\
 Config file '%s' does not exist.
 
 The synapse docker image no longer supports generating a config file on-the-fly
 based on environment variables. You can migrate to a static config file by
 running with 'migrate_config'. See the README for more details.
 """
-                    % (config_path,)
-                )
-
-            error(
-                "Config file '%s' does not exist. You should either create a new "
-                "config file by running with the `generate` argument (and then edit "
-                "the resulting file before restarting) or specify the path to an "
-                "existing config file with the SYNAPSE_CONFIG_PATH variable."
                 % (config_path,)
             )
 
-        args += ["--config-path", config_path]
+        error(
+            "Config file '%s' does not exist. You should either create a new "
+            "config file by running with the `generate` argument (and then edit "
+            "the resulting file before restarting) or specify the path to an "
+            "existing config file with the SYNAPSE_CONFIG_PATH variable."
+            % (config_path,)
+        )
 
-    log("Starting synapse with args " + " ".join(args))
+    log("Starting synapse with config file " + config_path)
 
-    args = ["python"] + args
+    args = ["python", "-m", synapse_worker, "--config-path", config_path]
     if ownership is not None:
         args = ["gosu", ownership] + args
         os.execv("/usr/sbin/gosu", args)

docs/admin_api/event_reports.rst

@@ -1,165 +0,0 @@
-Show reported events
-====================
-
-This API returns information about reported events.
-
-The api is::
-
-    GET /_synapse/admin/v1/event_reports?from=0&limit=10
-
-To use it, you will need to authenticate by providing an ``access_token`` for a
-server admin: see `README.rst <README.rst>`_.
-
-It returns a JSON body like the following:
-
-.. code:: jsonc
-
-    {
-        "event_reports": [
-            {
-                "event_id": "$bNUFCwGzWca1meCGkjp-zwslF-GfVcXukvRLI1_FaVY",
-                "id": 2,
-                "reason": "foo",
-                "score": -100,
-                "received_ts": 1570897107409,
-                "canonical_alias": "#alias1:matrix.org",
-                "room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
-                "name": "Matrix HQ",
-                "sender": "@foobar:matrix.org",
-                "user_id": "@foo:matrix.org"
-            },
-            {
-                "event_id": "$3IcdZsDaN_En-S1DF4EMCy3v4gNRKeOJs8W5qTOKj4I",
-                "id": 3,
-                "reason": "bar",
-                "score": -100,
-                "received_ts": 1598889612059,
-                "canonical_alias": "#alias2:matrix.org",
-                "room_id": "!eGvUQuTCkHGVwNMOjv:matrix.org",
-                "name": "Your room name here",
-                "sender": "@foobar:matrix.org",
-                "user_id": "@bar:matrix.org"
-            }
-        ],
-        "next_token": 2,
-        "total": 4
-    }
-
-To paginate, check for ``next_token`` and if present, call the endpoint again
-with ``from`` set to the value of ``next_token``. This will return a new page.
-
-If the endpoint does not return a ``next_token`` then there are no more
-reports to paginate through.
-
-**URL parameters:**
-
-- ``limit``: integer - Is optional but is used for pagination,
-  denoting the maximum number of items to return in this call. Defaults to ``100``.
-- ``from``: integer - Is optional but used for pagination,
-  denoting the offset in the returned results. This should be treated as an opaque value and
-  not explicitly set to anything other than the return value of ``next_token`` from a previous call.
-  Defaults to ``0``.
-- ``dir``: string - Direction of event report order. Whether to fetch the most recent first (``b``) or the
-  oldest first (``f``). Defaults to ``b``.
-- ``user_id``: string - Is optional and filters to only return users with user IDs that contain this value.
-  This is the user who reported the event and wrote the reason.
-- ``room_id``: string - Is optional and filters to only return rooms with room IDs that contain this value.
-
-**Response**
-
-The following fields are returned in the JSON response body:
-
-- ``id``: integer - ID of event report.
-- ``received_ts``: integer - The timestamp (in milliseconds since the unix epoch) when this report was sent.
-- ``room_id``: string - The ID of the room in which the event being reported is located.
-- ``name``: string - The name of the room.
-- ``event_id``: string - The ID of the reported event.
-- ``user_id``: string - This is the user who reported the event and wrote the reason.
-- ``reason``: string - Comment made by the ``user_id`` in this report. May be blank.
-- ``score``: integer - Content is reported based upon a negative score, where -100 is "most offensive" and 0 is "inoffensive".
-- ``sender``: string - This is the ID of the user who sent the original message/event that was reported.
-- ``canonical_alias``: string - The canonical alias of the room. ``null`` if the room does not have a canonical alias set.
-- ``next_token``: integer - Indication for pagination. See above.
-- ``total``: integer - Total number of event reports related to the query (``user_id`` and ``room_id``).
-
-Show details of a specific event report
-=======================================
-
-This API returns information about a specific event report.
-
-The api is::
-
-    GET /_synapse/admin/v1/event_reports/<report_id>
-
-To use it, you will need to authenticate by providing an ``access_token`` for a
-server admin: see `README.rst <README.rst>`_.
-
-It returns a JSON body like the following:
-
-.. code:: jsonc
-
-    {
-        "event_id": "$bNUFCwGzWca1meCGkjp-zwslF-GfVcXukvRLI1_FaVY",
-        "event_json": {
-            "auth_events": [
-                "$YK4arsKKcc0LRoe700pS8DSjOvUT4NDv0HfInlMFw2M",
-                "$oggsNXxzPFRE3y53SUNd7nsj69-QzKv03a1RucHu-ws"
-            ],
-            "content": {
-                "body": "matrix.org: This Week in Matrix",
-                "format": "org.matrix.custom.html",
-                "formatted_body": "<strong>matrix.org</strong>:<br><a href=\"https://matrix.org/blog/\"><strong>This Week in Matrix</strong></a>",
-                "msgtype": "m.notice"
-            },
-            "depth": 546,
-            "hashes": {
-                "sha256": "xK1//xnmvHJIOvbgXlkI8eEqdvoMmihVDJ9J4SNlsAw"
-            },
-            "origin": "matrix.org",
-            "origin_server_ts": 1592291711430,
-            "prev_events": [
-                "$YK4arsKKcc0LRoe700pS8DSjOvUT4NDv0HfInlMFw2M"
-            ],
-            "prev_state": [],
-            "room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
-            "sender": "@foobar:matrix.org",
-            "signatures": {
-                "matrix.org": {
-                    "ed25519:a_JaEG": "cs+OUKW/iHx5pEidbWxh0UiNNHwe46Ai9LwNz+Ah16aWDNszVIe2gaAcVZfvNsBhakQTew51tlKmL2kspXk/Dg"
-                }
-            },
-            "type": "m.room.message",
-            "unsigned": {
-                "age_ts": 1592291711430,
-            }
-        },
-        "id": <report_id>,
-        "reason": "foo",
-        "score": -100,
-        "received_ts": 1570897107409,
-        "canonical_alias": "#alias1:matrix.org",
-        "room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
-        "name": "Matrix HQ",
-        "sender": "@foobar:matrix.org",
-        "user_id": "@foo:matrix.org"
-    }
-
-**URL parameters:**
-
-- ``report_id``: string - The ID of the event report.
-
-**Response**
-
-The following fields are returned in the JSON response body:
-
-- ``id``: integer - ID of event report.
-- ``received_ts``: integer - The timestamp (in milliseconds since the unix epoch) when this report was sent.
-- ``room_id``: string - The ID of the room in which the event being reported is located.
-- ``name``: string - The name of the room.
-- ``event_id``: string - The ID of the reported event.
-- ``user_id``: string - This is the user who reported the event and wrote the reason.
-- ``reason``: string - Comment made by the ``user_id`` in this report. May be blank.
-- ``score``: integer - Content is reported based upon a negative score, where -100 is "most offensive" and 0 is "inoffensive".
-- ``sender``: string - This is the ID of the user who sent the original message/event that was reported.
-- ``canonical_alias``: string - The canonical alias of the room. ``null`` if the room does not have a canonical alias set.
-- ``event_json``: object - Details of the original event that was reported.

docs/admin_api/media_admin_api.md

@@ -100,82 +100,3 @@ Response:
   "num_quarantined": 10  # The number of media items successfully quarantined
 }
 ```
-
-# Delete local media
-This API deletes the *local* media from the disk of your own server.
-This includes any local thumbnails and copies of media downloaded from
-remote homeservers.
-This API will not affect media that has been uploaded to external
-media repositories (e.g https://github.com/turt2live/matrix-media-repo/).
-See also [purge_remote_media.rst](purge_remote_media.rst).
-
-## Delete a specific local media
-Delete a specific `media_id`.
-
-Request:
-
-```
-DELETE /_synapse/admin/v1/media/<server_name>/<media_id>
-
-{}
-```
-
-URL Parameters
-
-* `server_name`: string - The name of your local server (e.g `matrix.org`)
-* `media_id`: string - The ID of the media (e.g `abcdefghijklmnopqrstuvwx`)
-
-Response:
-
-```json
-    {
-       "deleted_media": [
-          "abcdefghijklmnopqrstuvwx"
-       ],
-       "total": 1
-    }
-```
-
-The following fields are returned in the JSON response body:
-
-* `deleted_media`: an array of strings - List of deleted `media_id`
-* `total`: integer - Total number of deleted `media_id`
-
-## Delete local media by date or size
-
-Request:
-
-```
-POST /_synapse/admin/v1/media/<server_name>/delete?before_ts=<before_ts>
-
-{}
-```
-
-URL Parameters
-
-* `server_name`: string - The name of your local server (e.g `matrix.org`).
-* `before_ts`: string representing a positive integer - Unix timestamp in ms.
-Files that were last used before this timestamp will be deleted. It is the timestamp of
-last access and not the timestamp creation. 
-* `size_gt`: Optional - string representing a positive integer - Size of the media in bytes.
-Files that are larger will be deleted. Defaults to `0`.
-* `keep_profiles`: Optional - string representing a boolean - Switch to also delete files
-that are still used in image data (e.g user profile, room avatar).
-If `false` these files will be deleted. Defaults to `true`.
-
-Response:
-
-```json
-    {
-       "deleted_media": [
-          "abcdefghijklmnopqrstuvwx",
-          "abcdefghijklmnopqrstuvwz"
-       ],
-       "total": 2
-    }
-```
-
-The following fields are returned in the JSON response body:
-
-* `deleted_media`: an array of strings - List of deleted `media_id`
-* `total`: integer - Total number of deleted `media_id`