Install gcc, lower poetry verbosity

Broke in https://github.com/matrix-org/synapse/pull/16019. This would have been caught
if someone ran 'devenv up' during manual testing of that PR, or better yet, if we had
CI for the nix developer environment.

Switch back to the latest release version, which doesn't have the upstream issue:
https://github.com/cachix/devenv/issues/756

.github/workflows/twisted_trunk.yml

@@ -5,9 +5,6 @@ on:
     - cron: 0 8 * * *
 
   workflow_dispatch:
-    # NB: inputs are only present when this workflow is dispatched manually.
-    # (The default below is the default field value in the form to trigger
-    # a manual dispatch). Otherwise the inputs will evaluate to null.
     inputs:
       twisted_ref:
         description: Commit, branch or tag to checkout from upstream Twisted.
@@ -52,7 +49,7 @@ jobs:
           extras: "all"
       - run: |
           poetry remove twisted
-          poetry add --extras tls git+https://github.com/twisted/twisted.git#${{ inputs.twisted_ref || 'trunk' }}
+          poetry add --extras tls git+https://github.com/twisted/twisted.git#${{ inputs.twisted_ref }}
           poetry install --no-interaction --extras "all test"
       - name: Remove warn_unused_ignores from mypy config
         run: sed '/warn_unused_ignores = True/d' -i mypy.ini

CHANGES.md

@@ -1,213 +1,3 @@
-# Synapse 1.91.1 (2023-09-04)
-
-### Bugfixes
-
-- Fix a performance regression introduced in Synapse 1.91.0 where event persistence would cause an excessive linear growth in CPU usage. ([\#16220](https://github.com/matrix-org/synapse/issues/16220))
-
-
-# Synapse 1.91.0 (2023-08-30)
-
-No significant changes since 1.91.0rc1.
-
-
-# Synapse 1.91.0rc1 (2023-08-23)
-
-### Features
-
-- Implements an admin API to lock an user without deactivating them. Based on [MSC3939](https://github.com/matrix-org/matrix-spec-proposals/pull/3939). ([\#15870](https://github.com/matrix-org/synapse/issues/15870))
-- Implements a task scheduler for	resumable potentially long running tasks. ([\#15891](https://github.com/matrix-org/synapse/issues/15891))
-- Allow specifying `client_secret_path` as alternative to `client_secret` for OIDC providers. This avoids leaking the client secret in the homeserver config. Contributed by @Ma27. ([\#16030](https://github.com/matrix-org/synapse/issues/16030))
-- Allow customising the IdP display name, icon, and brand for SAML and CAS providers (in addition to OIDC provider). ([\#16094](https://github.com/matrix-org/synapse/issues/16094))
-- Add an `admins` query parameter to the [List Accounts](https://matrix-org.github.io/synapse/v1.91/admin_api/user_admin_api.html#list-accounts) [admin API](https://matrix-org.github.io/synapse/v1.91/usage/administration/admin_api/index.html), to include only admins or to exclude admins in user queries. ([\#16114](https://github.com/matrix-org/synapse/issues/16114))
-
-### Bugfixes
-
-- Fix long-standing bug where concurrent requests to change a user's push rules could cause a deadlock. Contributed by Nick @ Beeper (@fizzadar). ([\#16052](https://github.com/matrix-org/synapse/issues/16052))
-- Fix a long-standing bu in `/sync` where timeout=0 does not skip caching, resulting in slow calls in cases where there are no new changes. Contributed by @PlasmaIntec. ([\#16080](https://github.com/matrix-org/synapse/issues/16080))
-- Fix performance of state resolutions for large, old rooms that did not have the full auth chain persisted. ([\#16116](https://github.com/matrix-org/synapse/issues/16116))
-- Filter out user agent references to the sliding sync proxy and rust-sdk from the user_daily_visits table to ensure that Element X can be represented fully. ([\#16124](https://github.com/matrix-org/synapse/issues/16124))
-- User constent and 3-PID changes capability cannot be enabled when using experimental [MSC3861](https://github.com/matrix-org/matrix-spec-proposals/pull/3861) support. ([\#16127](https://github.com/matrix-org/synapse/issues/16127), [\#16134](https://github.com/matrix-org/synapse/issues/16134))
-- Fix a rare race that could block new events from being sent for up to two minutes. Introduced in v1.90.0. ([\#16133](https://github.com/matrix-org/synapse/issues/16133), [\#16169](https://github.com/matrix-org/synapse/issues/16169))
-- Fix performance degredation when there are a lot of in-flight replication requests. ([\#16148](https://github.com/matrix-org/synapse/issues/16148))
-- Fix a bug introduced in 1.87 where synapse would send an excessive amount of federation requests to servers which have been offline for a long time. Contributed by Nico. ([\#16156](https://github.com/matrix-org/synapse/issues/16156), [\#16164](https://github.com/matrix-org/synapse/issues/16164))
-
-### Improved Documentation
-
-- Structured logging docs: add a link to explain the ELK stack ([\#16091](https://github.com/matrix-org/synapse/issues/16091))
-
-### Internal Changes
-
-- Update dehydrated devices implementation. ([\#16010](https://github.com/matrix-org/synapse/issues/16010))
-- Fix database performance of read/write worker locks. ([\#16061](https://github.com/matrix-org/synapse/issues/16061))
-- Fix building the nix development environment on MacOS systems. ([\#16063](https://github.com/matrix-org/synapse/issues/16063))
-- Override global statement timeout when creating indexes in Postgres. ([\#16085](https://github.com/matrix-org/synapse/issues/16085))
-- Fix the type annotation on `run_db_interaction` in the Module API. ([\#16089](https://github.com/matrix-org/synapse/issues/16089))
-- Clean-up the presence code. ([\#16092](https://github.com/matrix-org/synapse/issues/16092))
-- Run `pyupgrade` for Python 3.8+. ([\#16110](https://github.com/matrix-org/synapse/issues/16110))
-- Rename pagination and purge locks and add comments to explain why they exist and how they work. ([\#16112](https://github.com/matrix-org/synapse/issues/16112))
-- Attempt to fix the twisted trunk job. ([\#16115](https://github.com/matrix-org/synapse/issues/16115))
-- Cache token introspection response from OIDC provider. ([\#16117](https://github.com/matrix-org/synapse/issues/16117))
-- Add cache to `get_server_keys_json_for_remote`. ([\#16123](https://github.com/matrix-org/synapse/issues/16123))
-- Add an admin endpoint to allow authorizing server to signal token revocations. ([\#16125](https://github.com/matrix-org/synapse/issues/16125))
-- Add response time metrics for introspection requests for delegated auth. ([\#16131](https://github.com/matrix-org/synapse/issues/16131))
-- MSC3861: allow impersonation by an admin user using `_oidc_admin_impersonate_user_id` query parameter. ([\#16132](https://github.com/matrix-org/synapse/issues/16132))
-- Increase performance of read/write locks. ([\#16149](https://github.com/matrix-org/synapse/issues/16149))
-- Improve presence tests. ([\#16150](https://github.com/matrix-org/synapse/issues/16150), [\#16151](https://github.com/matrix-org/synapse/issues/16151), [\#16158](https://github.com/matrix-org/synapse/issues/16158))
-- Raised the poetry-core version cap to 1.7.0. ([\#16152](https://github.com/matrix-org/synapse/issues/16152))
-- Fix assertion in user directory unit tests. ([\#16157](https://github.com/matrix-org/synapse/issues/16157))
-- Reduce scope of locks when paginating to alleviate DB contention. ([\#16159](https://github.com/matrix-org/synapse/issues/16159))
-- Reduce DB contention on worker locks. ([\#16160](https://github.com/matrix-org/synapse/issues/16160))
-- Task scheduler: mark task as active if we are scheduling as soon as possible. ([\#16165](https://github.com/matrix-org/synapse/issues/16165))
-
-### Updates to locked dependencies
-
-* Bump click from 8.1.6 to 8.1.7. ([\#16145](https://github.com/matrix-org/synapse/issues/16145))
-* Bump gitpython from 3.1.31 to 3.1.32. ([\#16103](https://github.com/matrix-org/synapse/issues/16103))
-* Bump ijson from 3.2.1 to 3.2.3. ([\#16143](https://github.com/matrix-org/synapse/issues/16143))
-* Bump isort from 5.11.5 to 5.12.0. ([\#16108](https://github.com/matrix-org/synapse/issues/16108))
-* Bump log from 0.4.19 to 0.4.20. ([\#16109](https://github.com/matrix-org/synapse/issues/16109))
-* Bump pygithub from 1.59.0 to 1.59.1. ([\#16144](https://github.com/matrix-org/synapse/issues/16144))
-* Bump sentry-sdk from 1.28.1 to 1.29.2. ([\#16142](https://github.com/matrix-org/synapse/issues/16142))
-* Bump serde from 1.0.183 to 1.0.184. ([\#16139](https://github.com/matrix-org/synapse/issues/16139))
-* Bump txredisapi from 1.4.9 to 1.4.10. ([\#16107](https://github.com/matrix-org/synapse/issues/16107))
-* Bump types-bleach from 6.0.0.3 to 6.0.0.4. ([\#16106](https://github.com/matrix-org/synapse/issues/16106))
-* Bump types-pillow from 10.0.0.1 to 10.0.0.2. ([\#16105](https://github.com/matrix-org/synapse/issues/16105))
-* Bump types-pyopenssl from 23.2.0.1 to 23.2.0.2. ([\#16146](https://github.com/matrix-org/synapse/issues/16146))
-
-# Synapse 1.91.0rc1 (2023-08-23)
-
-### Features
-
-- Implements an admin API to lock an user without deactivating them. Based on [MSC3939](https://github.com/matrix-org/matrix-spec-proposals/pull/3939). ([\#15870](https://github.com/matrix-org/synapse/issues/15870))
-- Allow specifying `client_secret_path` as alternative to `client_secret` for OIDC providers. This avoids leaking the client secret in the homeserver config. Contributed by @Ma27. ([\#16030](https://github.com/matrix-org/synapse/issues/16030))
-- Allow customising the IdP display name, icon, and brand for SAML and CAS providers (in addition to OIDC provider). ([\#16094](https://github.com/matrix-org/synapse/issues/16094))
-- Add an `admins` query parameter to the [List Accounts](https://matrix-org.github.io/synapse/v1.91/admin_api/user_admin_api.html#list-accounts) [admin API](https://matrix-org.github.io/synapse/v1.91/usage/administration/admin_api/index.html), to include only admins or to exclude admins in user queries. ([\#16114](https://github.com/matrix-org/synapse/issues/16114))
-
-### Bugfixes
-
-- Fix long-standing bug where concurrent requests to change a user's push rules could cause a deadlock. Contributed by Nick @ Beeper (@fizzadar). ([\#16052](https://github.com/matrix-org/synapse/issues/16052))
-- Fix a long-standing bug in `/sync` where timeout=0 does not skip caching, resulting in slow calls in cases where there are no new changes. Contributed by @PlasmaIntec. ([\#16080](https://github.com/matrix-org/synapse/issues/16080))
-- Fix performance of state resolutions for large, old rooms that did not have the full auth chain persisted. ([\#16116](https://github.com/matrix-org/synapse/issues/16116))
-- Filter out user agent references to the sliding sync proxy and rust-sdk from the `user_daily_visits` table to ensure that Element X can be represented fully. ([\#16124](https://github.com/matrix-org/synapse/issues/16124))
-- User constent and third-party ID changes capability cannot be enabled when using experimental [MSC3861](https://github.com/matrix-org/matrix-spec-proposals/pull/3861) support. ([\#16127](https://github.com/matrix-org/synapse/issues/16127), [\#16134](https://github.com/matrix-org/synapse/issues/16134))
-- Fix a rare race that could block new events from being sent for up to two minutes. Introduced in v1.90.0. ([\#16133](https://github.com/matrix-org/synapse/issues/16133), [\#16169](https://github.com/matrix-org/synapse/issues/16169))
-- Fix performance degredation when there are a lot of in-flight replication requests. ([\#16148](https://github.com/matrix-org/synapse/issues/16148))
-- Fix a bug introduced in 1.87 where synapse would send an excessive amount of federation requests to servers which have been offline for a long time. Contributed by Nico. ([\#16156](https://github.com/matrix-org/synapse/issues/16156), [\#16164](https://github.com/matrix-org/synapse/issues/16164))
-
-### Improved Documentation
-
-- Structured logging docs: add a link to explain the ELK stack ([\#16091](https://github.com/matrix-org/synapse/issues/16091))
-
-### Internal Changes
-
-- Update dehydrated devices implementation. ([\#16010](https://github.com/matrix-org/synapse/issues/16010))
-- Fix database performance of read/write worker locks. ([\#16061](https://github.com/matrix-org/synapse/issues/16061))
-- Fix building the nix development environment on MacOS systems. ([\#16063](https://github.com/matrix-org/synapse/issues/16063))
-- Override global statement timeout when creating indexes in Postgres. ([\#16085](https://github.com/matrix-org/synapse/issues/16085))
-- Fix the type annotation on `run_db_interaction` in the Module API. ([\#16089](https://github.com/matrix-org/synapse/issues/16089))
-- Clean-up the presence code. ([\#16092](https://github.com/matrix-org/synapse/issues/16092))
-- Run `pyupgrade` for Python 3.8+. ([\#16110](https://github.com/matrix-org/synapse/issues/16110))
-- Rename pagination and purge locks and add comments to explain why they exist and how they work. ([\#16112](https://github.com/matrix-org/synapse/issues/16112))
-- Attempt to fix the twisted trunk job. ([\#16115](https://github.com/matrix-org/synapse/issues/16115))
-- Cache token introspection response from OIDC provider. ([\#16117](https://github.com/matrix-org/synapse/issues/16117))
-- Add cache to `get_server_keys_json_for_remote`. ([\#16123](https://github.com/matrix-org/synapse/issues/16123))
-- Add an admin endpoint to allow authorizing server to signal token revocations. ([\#16125](https://github.com/matrix-org/synapse/issues/16125))
-- Add response time metrics for introspection requests for delegated auth. ([\#16131](https://github.com/matrix-org/synapse/issues/16131))
-- [MSC3861](https://github.com/matrix-org/matrix-spec-proposals/pull/3861): allow impersonation by an admin user using `_oidc_admin_impersonate_user_id` query parameter. ([\#16132](https://github.com/matrix-org/synapse/issues/16132))
-- Increase performance of read/write locks. ([\#16149](https://github.com/matrix-org/synapse/issues/16149))
-- Improve presence tests. ([\#16150](https://github.com/matrix-org/synapse/issues/16150), [\#16151](https://github.com/matrix-org/synapse/issues/16151), [\#16158](https://github.com/matrix-org/synapse/issues/16158))
-- Raised the poetry-core version cap to 1.7.0. ([\#16152](https://github.com/matrix-org/synapse/issues/16152))
-- Fix assertion in user directory unit tests. ([\#16157](https://github.com/matrix-org/synapse/issues/16157))
-- Reduce scope of locks when paginating to alleviate DB contention. ([\#16159](https://github.com/matrix-org/synapse/issues/16159))
-- Reduce DB contention on worker locks. ([\#16160](https://github.com/matrix-org/synapse/issues/16160))
-- Task scheduler: mark task as active if we are scheduling as soon as possible. ([\#16165](https://github.com/matrix-org/synapse/issues/16165))
-- Implements a task scheduler for	resumable potentially long running tasks. ([\#15891](https://github.com/matrix-org/synapse/issues/15891))
-
-### Updates to locked dependencies
-
-* Bump click from 8.1.6 to 8.1.7. ([\#16145](https://github.com/matrix-org/synapse/issues/16145))
-* Bump gitpython from 3.1.31 to 3.1.32. ([\#16103](https://github.com/matrix-org/synapse/issues/16103))
-* Bump ijson from 3.2.1 to 3.2.3. ([\#16143](https://github.com/matrix-org/synapse/issues/16143))
-* Bump isort from 5.11.5 to 5.12.0. ([\#16108](https://github.com/matrix-org/synapse/issues/16108))
-* Bump log from 0.4.19 to 0.4.20. ([\#16109](https://github.com/matrix-org/synapse/issues/16109))
-* Bump pygithub from 1.59.0 to 1.59.1. ([\#16144](https://github.com/matrix-org/synapse/issues/16144))
-* Bump sentry-sdk from 1.28.1 to 1.29.2. ([\#16142](https://github.com/matrix-org/synapse/issues/16142))
-* Bump serde from 1.0.183 to 1.0.184. ([\#16139](https://github.com/matrix-org/synapse/issues/16139))
-* Bump txredisapi from 1.4.9 to 1.4.10. ([\#16107](https://github.com/matrix-org/synapse/issues/16107))
-* Bump types-bleach from 6.0.0.3 to 6.0.0.4. ([\#16106](https://github.com/matrix-org/synapse/issues/16106))
-* Bump types-pillow from 10.0.0.1 to 10.0.0.2. ([\#16105](https://github.com/matrix-org/synapse/issues/16105))
-* Bump types-pyopenssl from 23.2.0.1 to 23.2.0.2. ([\#16146](https://github.com/matrix-org/synapse/issues/16146))
-
-# Synapse 1.90.0 (2023-08-15)
-
-No significant changes since 1.90.0rc1.
-
-
-# Synapse 1.90.0rc1 (2023-08-08)
-
-### Features
-
-- Scope transaction IDs to devices (implement [MSC3970](https://github.com/matrix-org/matrix-spec-proposals/pull/3970)). ([\#15629](https://github.com/matrix-org/synapse/issues/15629))
-- Remove old rows from the `cache_invalidation_stream_by_instance` table automatically (this table is unused in SQLite). ([\#15868](https://github.com/matrix-org/synapse/issues/15868))
-
-### Bugfixes
-
-- Fix a long-standing bug where purging history and paginating simultaneously could lead to database corruption when using workers. ([\#15791](https://github.com/matrix-org/synapse/issues/15791))
-- Fix a long-standing bug where profile endpoint returned a 404 when the user's display name was empty. ([\#16012](https://github.com/matrix-org/synapse/issues/16012))
-- Fix a long-standing bug where the `synapse_port_db` failed to configure sequences for application services and partial stated rooms. ([\#16043](https://github.com/matrix-org/synapse/issues/16043))
-- Fix long-standing bug with deletion in dehydrated devices v2. ([\#16046](https://github.com/matrix-org/synapse/issues/16046))
-
-### Updates to the Docker image
-
-- Add `org.opencontainers.image.version` labels to Docker containers [published by Matrix.org](https://hub.docker.com/r/matrixdotorg/synapse). Contributed by Mo Balaa. ([\#15972](https://github.com/matrix-org/synapse/issues/15972), [\#16009](https://github.com/matrix-org/synapse/issues/16009))
-
-### Improved Documentation
-
-- Add a internal documentation page describing the ["streams" used within Synapse](https://matrix-org.github.io/synapse/v1.90/development/synapse_architecture/streams.html). ([\#16015](https://github.com/matrix-org/synapse/issues/16015))
-- Clarify comment on the keys/upload over replication enpoint. ([\#16016](https://github.com/matrix-org/synapse/issues/16016))
-- Do not expose Admin API in caddy reverse proxy example. Contributed by @NilsIrl. ([\#16027](https://github.com/matrix-org/synapse/issues/16027))
-
-### Deprecations and Removals
-
-- Remove support for legacy application service paths. ([\#15964](https://github.com/matrix-org/synapse/issues/15964))
-- Move support for application service query parameter authorization behind a configuration option. ([\#16017](https://github.com/matrix-org/synapse/issues/16017))
-
-### Internal Changes
-
-- Update SQL queries to inline boolean parameters as supported in SQLite 3.27. ([\#15525](https://github.com/matrix-org/synapse/issues/15525))
-- Allow for the configuration of the backoff algorithm for federation destinations. ([\#15754](https://github.com/matrix-org/synapse/issues/15754))
-- Allow modules to check whether the current worker is configured to run background tasks. ([\#15991](https://github.com/matrix-org/synapse/issues/15991))
-- Update support for [MSC3958](https://github.com/matrix-org/matrix-spec-proposals/pull/3958) to match the latest revision of the MSC. ([\#15992](https://github.com/matrix-org/synapse/issues/15992))
-- Allow modules to schedule delayed background calls. ([\#15993](https://github.com/matrix-org/synapse/issues/15993))
-- Properly overwrite the `redacts` content-property for forwards-compatibility with room versions 1 through 10. ([\#16013](https://github.com/matrix-org/synapse/issues/16013))
-- Fix building the nix development environment on MacOS systems. ([\#16019](https://github.com/matrix-org/synapse/issues/16019))
-- Remove leading and trailing spaces when setting a display name. ([\#16031](https://github.com/matrix-org/synapse/issues/16031))
-- Combine duplicated code. ([\#16023](https://github.com/matrix-org/synapse/issues/16023))
-- Collect additional metrics from `ResponseCache` for eviction. ([\#16028](https://github.com/matrix-org/synapse/issues/16028))
-- Fix endpoint improperly declaring support for MSC3814. ([\#16068](https://github.com/matrix-org/synapse/issues/16068))
-- Drop backwards compat hack for event serialization. ([\#16069](https://github.com/matrix-org/synapse/issues/16069))
-
-### Updates to locked dependencies
-
-* Update PyYAML to 6.0.1. ([\#16011](https://github.com/matrix-org/synapse/issues/16011))
-* Bump cryptography from 41.0.2 to 41.0.3. ([\#16048](https://github.com/matrix-org/synapse/issues/16048))
-* Bump furo from 2023.5.20 to 2023.7.26. ([\#16077](https://github.com/matrix-org/synapse/issues/16077))
-* Bump immutabledict from 2.2.4 to 3.0.0. ([\#16034](https://github.com/matrix-org/synapse/issues/16034))
-* Update certifi to 2023.7.22 and pygments to 2.15.1. ([\#16044](https://github.com/matrix-org/synapse/issues/16044))
-* Bump jsonschema from 4.18.3 to 4.19.0. ([\#16081](https://github.com/matrix-org/synapse/issues/16081))
-* Bump phonenumbers from 8.13.14 to 8.13.18. ([\#16076](https://github.com/matrix-org/synapse/issues/16076))
-* Bump regex from 1.9.1 to 1.9.3. ([\#16073](https://github.com/matrix-org/synapse/issues/16073))
-* Bump serde from 1.0.171 to 1.0.175. ([\#15982](https://github.com/matrix-org/synapse/issues/15982))
-* Bump serde from 1.0.175 to 1.0.179. ([\#16033](https://github.com/matrix-org/synapse/issues/16033))
-* Bump serde from 1.0.179 to 1.0.183. ([\#16074](https://github.com/matrix-org/synapse/issues/16074))
-* Bump serde_json from 1.0.103 to 1.0.104. ([\#16032](https://github.com/matrix-org/synapse/issues/16032))
-* Bump service-identity from 21.1.0 to 23.1.0. ([\#16038](https://github.com/matrix-org/synapse/issues/16038))
-* Bump types-commonmark from 0.9.2.3 to 0.9.2.4. ([\#16037](https://github.com/matrix-org/synapse/issues/16037))
-* Bump types-jsonschema from 4.17.0.8 to 4.17.0.10. ([\#16036](https://github.com/matrix-org/synapse/issues/16036))
-* Bump types-netaddr from 0.8.0.8 to 0.8.0.9. ([\#16035](https://github.com/matrix-org/synapse/issues/16035))
-* Bump types-opentracing from 2.4.10.5 to 2.4.10.6. ([\#16078](https://github.com/matrix-org/synapse/issues/16078))
-* Bump types-setuptools from 68.0.0.0 to 68.0.0.3. ([\#16079](https://github.com/matrix-org/synapse/issues/16079))
-
 # Synapse 1.89.0 (2023-08-01)
 
 No significant changes since 1.89.0rc1.

Cargo.lock

@@ -132,9 +132,9 @@ dependencies = [
 
 [[package]]
 name = "log"
-version = "0.4.20"
+version = "0.4.19"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b5e6163cb8c49088c2c36f57875e58ccd8c87c7427f7fbd50ea6710b2f3f2e8f"
+checksum = "b06a4cde4c0f271a446782e3eff8de789548ce57dbc8eca9292c27f4a42004b4"
 
 [[package]]
 name = "memchr"
@@ -291,9 +291,9 @@ dependencies = [
 
 [[package]]
 name = "regex"
-version = "1.9.3"
+version = "1.9.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "81bc1d4caf89fac26a70747fe603c130093b53c773888797a6329091246d651a"
+checksum = "b2eae68fc220f7cf2532e4494aded17545fce192d59cd996e0fe7887f4ceb575"
 dependencies = [
  "aho-corasick",
  "memchr",
@@ -303,9 +303,9 @@ dependencies = [
 
 [[package]]
 name = "regex-automata"
-version = "0.3.6"
+version = "0.3.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fed1ceff11a1dddaee50c9dc8e4938bd106e9d89ae372f192311e7da498e3b69"
+checksum = "83d3daa6976cffb758ec878f108ba0e062a45b2d6ca3a2cca965338855476caf"
 dependencies = [
  "aho-corasick",
  "memchr",
@@ -314,9 +314,9 @@ dependencies = [
 
 [[package]]
 name = "regex-syntax"
-version = "0.7.4"
+version = "0.7.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e5ea92a5b6195c6ef2a0295ea818b312502c6fc94dde986c5553242e18fd4ce2"
+checksum = "2ab07dc67230e4a4718e70fd5c20055a4334b121f1f9db8fe63ef39ce9b8c846"
 
 [[package]]
 name = "ryu"
@@ -332,22 +332,22 @@ checksum = "d29ab0c6d3fc0ee92fe66e2d99f700eab17a8d57d1c1d3b748380fb20baa78cd"
 
 [[package]]
 name = "serde"
-version = "1.0.184"
+version = "1.0.179"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2c911f4b04d7385c9035407a4eff5903bf4fe270fa046fda448b69e797f4fff0"
+checksum = "0a5bf42b8d227d4abf38a1ddb08602e229108a517cd4e5bb28f9c7eaafdce5c0"
 dependencies = [
  "serde_derive",
 ]
 
 [[package]]
 name = "serde_derive"
-version = "1.0.184"
+version = "1.0.179"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c1df27f5b29406ada06609b2e2f77fb34f6dbb104a457a671cc31dbed237e09e"
+checksum = "741e124f5485c7e60c03b043f79f320bff3527f4bbf12cf3831750dc46a0ec2c"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.28",
+ "syn 2.0.25",
 ]
 
 [[package]]
@@ -386,9 +386,9 @@ dependencies = [
 
 [[package]]
 name = "syn"
-version = "2.0.28"
+version = "2.0.25"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "04361975b3f5e348b2189d8dc55bc942f278b2d482a6a0365de5bdd62d351567"
+checksum = "15e3fc8c0c74267e2df136e5e5fb656a464158aa57624053375eb9c8c6e25ae2"
 dependencies = [
  "proc-macro2",
  "quote",

changelog.d/15525.misc

@@ -0,0 +1 @@
+Update SQL queries to inline boolean parameters as supported in SQLite 3.27.

changelog.d/15629.feature

@@ -0,0 +1 @@
+Scope transaction IDs to devices (implement [MSC3970](https://github.com/matrix-org/matrix-spec-proposals/pull/3970)).

changelog.d/15754.misc

@@ -0,0 +1 @@
+Allow for the configuration of the backoff algorithm for federation destinations.

changelog.d/15791.bugfix

@@ -0,0 +1 @@
+Fix bug where purging history and paginating simultaneously could lead to database corruption when using workers.

changelog.d/15964.removal

@@ -0,0 +1 @@
+Remove support for legacy application service paths.

changelog.d/15972.docker

@@ -0,0 +1 @@
+Add `org.opencontainers.image.version` labels to Docker containers [published by Matrix.org](https://hub.docker.com/r/matrixdotorg/synapse). Contributed by Mo Balaa.

changelog.d/15991.misc

@@ -0,0 +1 @@
+Allow modules to check whether the current worker is configured to run background tasks.

changelog.d/15992.misc

@@ -0,0 +1 @@
+Update support for [MSC3958](https://github.com/matrix-org/matrix-spec-proposals/pull/3958) to match the latest revision of the MSC.

changelog.d/16009.docker

@@ -0,0 +1 @@
+Add `org.opencontainers.image.version` labels to Docker containers [published by Matrix.org](https://hub.docker.com/r/matrixdotorg/synapse). Contributed by Mo Balaa.

changelog.d/16011.misc

@@ -0,0 +1 @@
+Update PyYAML to 6.0.1.

changelog.d/16012.bugfix

@@ -0,0 +1 @@
+Fix 404 not found code returned on profile endpoint when the display name is empty but not the avatar URL.

changelog.d/16013.misc

@@ -0,0 +1 @@
+Properly overwrite the `redacts` content-property for forwards-compatibility with room versions 1 through 10.

changelog.d/16016.doc

@@ -0,0 +1,2 @@
+Clarify comment on the keys/upload over replication enpoint.
+

changelog.d/16017.removal

@@ -0,0 +1 @@
+Move support for application service query parameter authorization behind a configuration option.

changelog.d/16019.misc

@@ -0,0 +1 @@
+Fix building the nix development environment on MacOS systems.

changelog.d/16023.misc

@@ -0,0 +1 @@
+Combine duplicated code.

changelog.d/16027.doc

@@ -0,0 +1 @@
+Do not expose Admin API in caddy reverse proxy example. Contributed by @NilsIrl.

changelog.d/16028.misc

@@ -0,0 +1 @@
+Collect additional metrics from `ResponseCache` for eviction.

changelog.d/16031.bugfix

@@ -0,0 +1 @@
+Remove leading and trailing spaces when setting a display name.

changelog.d/16043.bugfix

@@ -0,0 +1 @@
+Fix a long-standing bug where the `synapse_port_db` failed to configure sequences for application services and partial stated rooms.

changelog.d/16044.misc

@@ -0,0 +1 @@
+Update certifi to 2023.7.22 and pygments to 2.15.1.

changelog.d/16063.misc

@@ -0,0 +1 @@
+Fix building the nix development environment on MacOS systems.

contrib/cmdclient/console.py

@@ -769,7 +769,7 @@ def main(server_url, identity_server_url, username, token, config_path):
     global CONFIG_JSON
     CONFIG_JSON = config_path  # bit cheeky, but just overwrite the global
     try:
-        with open(config_path) as config:
+        with open(config_path, "r") as config:
             syn_cmd.config = json.load(config)
             try:
                 http_client.verbose = "on" == syn_cmd.config["verbose"]

debian/changelog

@@ -1,33 +1,3 @@
-matrix-synapse-py3 (1.91.1) stable; urgency=medium
-
-  * New Synapse release 1.91.1.
-
- -- Synapse Packaging team <packages@matrix.org>  Mon, 04 Sep 2023 14:03:18 +0100
-
-matrix-synapse-py3 (1.91.0) stable; urgency=medium
-
-  * New Synapse release 1.91.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 30 Aug 2023 11:18:10 +0100
-
-matrix-synapse-py3 (1.91.0~rc1) stable; urgency=medium
-
-  * New Synapse release 1.91.0rc1.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 23 Aug 2023 09:47:18 -0700
-
-matrix-synapse-py3 (1.90.0) stable; urgency=medium
-
-  * New Synapse release 1.90.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 15 Aug 2023 11:17:34 +0100
-
-matrix-synapse-py3 (1.90.0~rc1) stable; urgency=medium
-
-  * New Synapse release 1.90.0rc1.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 08 Aug 2023 15:29:34 +0100
-
 matrix-synapse-py3 (1.89.0) stable; urgency=medium
 
   * New Synapse release 1.89.0.

docker/configure_workers_and_start.py

@@ -861,7 +861,7 @@ def generate_worker_files(
         # Then a worker config file
         convert(
             "/conf/worker.yaml.j2",
-            f"/conf/workers/{worker_name}.yaml",
+            "/conf/workers/{name}.yaml".format(name=worker_name),
             **worker_config,
             worker_log_config_filepath=log_config_filepath,
             using_unix_sockets=using_unix_sockets,

docker/start.py

@@ -82,7 +82,7 @@ def generate_config_from_template(
                 with open(filename) as handle:
                     value = handle.read()
             else:
-                log(f"Generating a random secret for {secret}")
+                log("Generating a random secret for {}".format(secret))
                 value = codecs.encode(os.urandom(32), "hex").decode()
                 with open(filename, "w") as handle:
                     handle.write(value)

docs/SUMMARY.md

@@ -97,7 +97,6 @@
     - [Cancellation](development/synapse_architecture/cancellation.md)
     - [Log Contexts](log_contexts.md)
     - [Replication](replication.md)
-    - [Streams](development/synapse_architecture/streams.md)
     - [TCP Replication](tcp_replication.md)
     - [Faster remote joins](development/synapse_architecture/faster_joins.md)
   - [Internal Documentation](development/internal_documentation/README.md)

docs/admin_api/user_admin_api.md

@@ -146,7 +146,6 @@ Body parameters:
 - `admin` - **bool**, optional, defaults to `false`. Whether the user is a homeserver administrator,
   granting them access to the Admin API, among other things.
 - `deactivated` - **bool**, optional. If unspecified, deactivation state will be left unchanged.
-- `locked` - **bool**, optional. If unspecified, locked state will be left unchanged.
 
   Note: the `password` field must also be set if both of the following are true:
   - `deactivated` is set to `false` and the user was previously deactivated (you are reactivating this user)
@@ -219,8 +218,6 @@ The following parameters should be set in the URL:
   **or** displaynames that contain this value.
 - `guests` - string representing a bool - Is optional and if `false` will **exclude** guest users.
   Defaults to `true` to include guest users.
-- `admins` - Optional flag to filter admins. If `true`, only admins are queried. If `false`, admins are excluded from 
-  the query. When the flag is absent (the default), **both** admins and non-admins are included in the search results.
 - `deactivated` - string representing a bool - Is optional and if `true` will **include** deactivated users.
   Defaults to `false` to exclude deactivated users.
 - `limit` - string representing a positive integer - Is optional but is used for pagination,

docs/development/synapse_architecture/streams.md

@@ -1,157 +0,0 @@
-## Streams
-
-Synapse has a concept of "streams", which are roughly described in [`id_generators.py`](
-    https://github.com/matrix-org/synapse/blob/develop/synapse/storage/util/id_generators.py
-).
-Generally speaking, streams are a series of notifications that something in Synapse's database has changed that the application might need to respond to.
-For example:
-
-- The events stream reports new events (PDUs) that Synapse creates, or that Synapse accepts from another homeserver.
-- The account data stream reports changes to users' [account data](https://spec.matrix.org/v1.7/client-server-api/#client-config).
-- The to-device stream reports when a device has a new [to-device message](https://spec.matrix.org/v1.7/client-server-api/#send-to-device-messaging).
-
-See [`synapse.replication.tcp.streams`](
-    https://github.com/matrix-org/synapse/blob/develop/synapse/replication/tcp/streams/__init__.py
-) for the full list of streams.
-
-It is very helpful to understand the streams mechanism when working on any part of Synapse that needs to respond to changes—especially if those changes are made by different workers.
-To that end, let's describe streams formally, paraphrasing from the docstring of [`AbstractStreamIdGenerator`](
-    https://github.com/matrix-org/synapse/blob/a719b703d9bd0dade2565ddcad0e2f3a7a9d4c37/synapse/storage/util/id_generators.py#L96
-).
-
-### Definition
-
-A stream is an append-only log `T1, T2, ..., Tn, ...` of facts[^1] which grows over time.
-Only "writers" can add facts to a stream, and there may be multiple writers.
-
-Each fact has an ID, called its "stream ID".
-Readers should only process facts in ascending stream ID order.
-
-Roughly speaking, each stream is backed by a database table.
-It should have a `stream_id` (or similar) bigint column holding stream IDs, plus additional columns as necessary to describe the fact.
-Typically, a fact is expressed with a single row in its backing table.[^2]
-Within a stream, no two facts may have the same stream_id.
-
-> _Aside_. Some additional notes on streams' backing tables.
->
-> 1. Rich would like to [ditch the backing tables](https://github.com/matrix-org/synapse/issues/13456).
-> 2. The backing tables may have other uses.
-     >    For example, the events table serves backs the events stream, and is read when processing new events.
-     >    But old rows are read from the table all the time, whenever Synapse needs to lookup some facts about an event.
-> 3. Rich suspects that sometimes the stream is backed by multiple tables, so the stream proper is the union of those tables.
-
-Stream writers can "reserve" a stream ID, and then later mark it as having being completed.
-Stream writers need to track the completion of each stream fact.
-In the happy case, completion means a fact has been written to the stream table.
-But unhappy cases (e.g. transaction rollback due to an error) also count as completion.
-Once completed, the rows written with that stream ID are fixed, and no new rows
-will be inserted with that ID.
-
-### Current stream ID
-
-For any given stream reader (including writers themselves), we may define a per-writer current stream ID:
-
-> The current stream ID _for a writer W_ is the largest stream ID such that
-> all transactions added by W with equal or smaller ID have completed.
-
-Similarly, there is a "linear" notion of current stream ID:
-
-> The "linear" current stream ID is the largest stream ID such that
-> all facts (added by any writer) with equal or smaller ID have completed.
-
-Because different stream readers A and B learn about new facts at different times, A and B may disagree about current stream IDs.
-Put differently: we should think of stream readers as being independent of each other, proceeding through a stream of facts at different rates.
-
-**NB.** For both senses of "current", that if a writer opens a transaction that never completes, the current stream ID will never advance beyond that writer's last written stream ID.
-
-For single-writer streams, the per-writer current ID and the linear current ID are the same.
-Both senses of current ID are monotonic, but they may "skip" or jump over IDs because facts complete out of order.
-
-
-_Example_.
-Consider a single-writer stream which is initially at ID 1.
-
-| Action     | Current stream ID | Notes                                           |
-|------------|-------------------|-------------------------------------------------|
-|            | 1                 |                                                 |
-| Reserve 2  | 1                 |                                                 |
-| Reserve 3  | 1                 |                                                 |
-| Complete 3 | 1                 | current ID unchanged, waiting for 2 to complete |
-| Complete 2 | 3                 | current ID jumps from 1 -> 3                    |
-| Reserve 4  | 3                 |                                                 |
-| Reserve 5  | 3                 |                                                 |
-| Reserve 6  | 3                 |                                                 |
-| Complete 5 | 3                 |                                                 |
-| Complete 4 | 5                 | current ID jumps 3->5, even though 6 is pending |
-| Complete 6 | 6                 |                                                 |
-
-
-### Multi-writer streams
-
-There are two ways to view a multi-writer stream.
-
-1. Treat it as a collection of distinct single-writer streams, one
-   for each writer.
-2. Treat it as a single stream.
-
-The single stream (option 2) is conceptually simpler, and easier to represent (a single stream id).
-However, it requires each reader to know about the entire set of writers, to ensures that readers don't erroneously advance their current stream position too early and miss a fact from an unknown writer.
-In contrast, multiple parallel streams (option 1) are more complex, requiring more state to represent (map from writer to stream id).
-The payoff for doing so is that readers can "peek" ahead to facts that completed on one writer no matter the state of the others, reducing latency.
-
-Note that a multi-writer stream can be viewed in both ways.
-For example, the events stream is treated as multiple single-writer streams (option 1) by the sync handler, so that events are sent to clients as soon as possible.
-But the background process that works through events treats them as a single linear stream.
-
-Another useful example is the cache invalidation stream.
-The facts this stream holds are instructions to "you should now invalidate these cache entries".
-We only ever treat this as a multiple single-writer streams as there is no important ordering between cache invalidations.
-(Invalidations are self-contained facts; and the invalidations commute/are idempotent).
-
-### Writing to streams
-
-Writers need to track:
- - track their current position (i.e. its own per-writer stream ID).
- - their facts currently awaiting completion.
-
-At startup, 
- - the current position of that writer can be found by querying the database (which suggests that facts need to be written to the database atomically, in a transaction); and
- - there are no facts awaiting completion.
-
-To reserve a stream ID, call [`nextval`](https://www.postgresql.org/docs/current/functions-sequence.html) on the appropriate postgres sequence.
-
-To write a fact to the stream: insert the appropriate rows to the appropriate backing table.
-
-To complete a fact, first remove it from your map of facts currently awaiting completion.
-Then, if no earlier fact is awaiting completion, the writer can advance its current position in that stream.
-Upon doing so it should emit an `RDATA` message[^3], once for every fact between the old and the new stream ID.
-
-### Subscribing to streams
-
-Readers need to track the current position of every writer.
-
-At startup, they can find this by contacting each writer with a `REPLICATE` message,
-requesting that all writers reply describing their current position in their streams.
-Writers reply with a `POSITION` message.
-
-To learn about new facts, readers should listen for `RDATA` messages and process them to respond to the new fact.
-The `RDATA` itself is not a self-contained representation of the fact;
-readers will have to query the stream tables for the full details.
-Readers must also advance their record of the writer's current position for that stream.
-
-# Summary
-
-In a nutshell: we have an append-only log with a "buffer/scratchpad" at the end where we have to wait for the sequence to be linear and contiguous.
-
-
----
-
-[^1]: we use the word _fact_ here for two reasons.
-Firstly, the word "event" is already heavily overloaded (PDUs, EDUs, account data, ...) and we don't need to make that worse.
-Secondly, "fact" emphasises that the things we append to a stream cannot change after the fact.
-
-[^2]: A fact might be expressed with 0 rows, e.g. if we opened a transaction to persist an event, but failed and rolled the transaction back before marking the fact as completed.
-In principle a fact might be expressed with 2 or more rows; if so, each of those rows should share the fact's stream ID.
-
-[^3]: This communication used to happen directly with the writers [over TCP](../../tcp_replication.md);
-nowadays it's done via Redis's Pubsub.

docs/structured_logging.md

@@ -3,7 +3,7 @@
 A structured logging system can be useful when your logs are destined for a
 machine to parse and process. By maintaining its machine-readable characteristics,
 it enables more efficient searching and aggregations when consumed by software
-such as the [ELK stack](https://opensource.com/article/18/9/open-source-log-aggregation-tools).
+such as the "ELK stack".
 
 Synapse's structured logging system is configured via the file that Synapse's
 `log_config` config option points to. The file should include a formatter which

docs/usage/configuration/config_documentation.md

@@ -3025,16 +3025,6 @@ enable SAML login. You can either put your entire pysaml config inline using the
 option, or you can specify a path to a psyaml config file with the sub-option `config_path`.
 This setting has the following sub-options:
 
-* `idp_name`: A user-facing name for this identity provider, which is used to
-   offer the user a choice of login mechanisms.
-* `idp_icon`: An optional icon for this identity provider, which is presented
-   by clients and Synapse's own IdP picker page. If given, must be an
-   MXC URI of the format `mxc://<server-name>/<media-id>`. (An easy way to
-   obtain such an MXC URI is to upload an image to an (unencrypted) room
-   and then copy the "url" from the source of the event.)
-* `idp_brand`: An optional brand for this identity provider, allowing clients
-   to style the login flow according to the identity provider in question.
-   See the [spec](https://spec.matrix.org/latest/) for possible options here.
 * `sp_config`: the configuration for the pysaml2 Service Provider. See pysaml2 docs for format of config.
    Default values will be used for the `entityid` and `service` settings,
    so it is not normally necessary to specify them unless you need to
@@ -3186,7 +3176,7 @@ Options for each entry include:
 
 * `idp_icon`: An optional icon for this identity provider, which is presented
    by clients and Synapse's own IdP picker page. If given, must be an
-   MXC URI of the format `mxc://<server-name>/<media-id>`. (An easy way to
+   MXC URI of the format mxc://<server-name>/<media-id>. (An easy way to
    obtain such an MXC URI is to upload an image to an (unencrypted) room
    and then copy the "url" from the source of the event.)
 
@@ -3204,14 +3194,6 @@ Options for each entry include:
 
 * `client_secret`: oauth2 client secret to use. May be omitted if
   `client_secret_jwt_key` is given, or if `client_auth_method` is 'none'.
-  Must be omitted if `client_secret_path` is specified.
-
-* `client_secret_path`: path to the oauth2 client secret to use. With that
-   it's not necessary to leak secrets into the config file itself.
-   Mutually exclusive with `client_secret`. Can be omitted if
-   `client_secret_jwt_key` is specified.
-
-   *Added in Synapse 1.91.0.*
 
 * `client_secret_jwt_key`: Alternative to client_secret: details of a key used
    to create a JSON Web Token to be used as an OAuth2 client secret. If
@@ -3409,16 +3391,6 @@ Enable Central Authentication Service (CAS) for registration and login.
 Has the following sub-options:
 * `enabled`: Set this to true to enable authorization against a CAS server.
    Defaults to false.
-* `idp_name`: A user-facing name for this identity provider, which is used to
-   offer the user a choice of login mechanisms.
-* `idp_icon`: An optional icon for this identity provider, which is presented
-   by clients and Synapse's own IdP picker page. If given, must be an
-   MXC URI of the format `mxc://<server-name>/<media-id>`. (An easy way to
-   obtain such an MXC URI is to upload an image to an (unencrypted) room
-   and then copy the "url" from the source of the event.)
-* `idp_brand`: An optional brand for this identity provider, allowing clients
-   to style the login flow according to the identity provider in question.
-   See the [spec](https://spec.matrix.org/latest/) for possible options here.
 * `server_url`: The URL of the CAS authorization endpoint.
 * `displayname_attribute`: The attribute of the CAS response to use as the display name.
    If no name is given here, no displayname will be set.
@@ -3659,7 +3631,6 @@ This option has the following sub-options:
 * `prefer_local_users`: Defines whether to prefer local users in search query results.
    If set to true, local users are more likely to appear above remote users when searching the
    user directory. Defaults to false.
-* `show_locked_users`: Defines whether to show locked users in search query results. Defaults to false.
 
 Example configuration:
 ```yaml
@@ -3667,7 +3638,6 @@ user_directory:
     enabled: false
     search_all_users: true
     prefer_local_users: true
-    show_locked_users: true
 ```
 ---
 ### `user_consent`

flake.nix

@@ -90,6 +90,9 @@
                   # The rust-analyzer language server implementation.
                   rust-analyzer
 
+		  # For building any Python bindings to C or Rust code.
+		  gcc
+
                   # Native dependencies for running Synapse.
                   icu
                   libffi
@@ -122,7 +125,7 @@
                 languages.python.poetry.activate.enable = true;
                 # Install all extra Python dependencies; this is needed to run the unit
                 # tests and utilitise all Synapse features.
-                languages.python.poetry.install.arguments = ["--extras all"];
+                languages.python.poetry.install.arguments = ["-v" "--extras all"];
                 # Install the 'matrix-synapse' package from the local checkout.
                 languages.python.poetry.install.installRootPackage = true;
 

mypy.ini

@@ -45,13 +45,6 @@ warn_unused_ignores = False
 disallow_untyped_defs = False
 disallow_incomplete_defs = False
 
-[mypy-synapse.util.manhole]
-# This module imports something from Twisted which has a bad annotation in Twisted trunk,
-# but is unannotated in Twisted's latest release. We want to type-ignore the problem 
-# in the twisted trunk job, even though it has no effect on normal mypy runs.
-warn_unused_ignores = False
-
-
 ;; Dependencies without annotations
 ;; Before ignoring a module, check to see if type stubs are available.
 ;; The `typeshed` project maintains stubs here:

poetry.lock

@@ -397,13 +397,13 @@ files = [
 
 [[package]]
 name = "click"
-version = "8.1.7"
+version = "8.1.6"
 description = "Composable command line interface toolkit"
 optional = false
 python-versions = ">=3.7"
 files = [
-    {file = "click-8.1.7-py3-none-any.whl", hash = "sha256:ae74fb96c20a0277a1d615f1e4d73c8414f5a98db8b799a7931d1582f3390c28"},
-    {file = "click-8.1.7.tar.gz", hash = "sha256:ca9853ad459e787e2192211578cc907e7594e294c7ccc834310722b41b9ca6de"},
+    {file = "click-8.1.6-py3-none-any.whl", hash = "sha256:fa244bb30b3b5ee2cae3da8f55c9e5e0c0e86093306301fb418eb9dc40fbded5"},
+    {file = "click-8.1.6.tar.gz", hash = "sha256:48ee849951919527a045bfe3bf7baa8a959c423134e1a5b98c05c20ba75a1cbd"},
 ]
 
 [package.dependencies]
@@ -558,13 +558,13 @@ dev = ["Sphinx", "coverage", "flake8", "lxml", "lxml-stubs", "memory-profiler",
 
 [[package]]
 name = "furo"
-version = "2023.7.26"
+version = "2023.5.20"
 description = "A clean customisable Sphinx documentation theme."
 optional = false
 python-versions = ">=3.7"
 files = [
-    {file = "furo-2023.7.26-py3-none-any.whl", hash = "sha256:1c7936929ec57c5ddecc7c85f07fa8b2ce536b5c89137764cca508be90e11efd"},
-    {file = "furo-2023.7.26.tar.gz", hash = "sha256:257f63bab97aa85213a1fa24303837a3c3f30be92901ec732fea74290800f59e"},
+    {file = "furo-2023.5.20-py3-none-any.whl", hash = "sha256:594a8436ddfe0c071f3a9e9a209c314a219d8341f3f1af33fdf7c69544fab9e6"},
+    {file = "furo-2023.5.20.tar.gz", hash = "sha256:40e09fa17c6f4b22419d122e933089226dcdb59747b5b6c79363089827dea16f"},
 ]
 
 [package.dependencies]
@@ -589,13 +589,13 @@ smmap = ">=3.0.1,<6"
 
 [[package]]
 name = "gitpython"
-version = "3.1.32"
+version = "3.1.31"
 description = "GitPython is a Python library used to interact with Git repositories"
 optional = false
 python-versions = ">=3.7"
 files = [
-    {file = "GitPython-3.1.32-py3-none-any.whl", hash = "sha256:e3d59b1c2c6ebb9dfa7a184daf3b6dd4914237e7488a1730a6d8f6f5d0b4187f"},
-    {file = "GitPython-3.1.32.tar.gz", hash = "sha256:8d9b8cb1e80b9735e8717c9362079d3ce4c6e5ddeebedd0361b228c3a67a62f6"},
+    {file = "GitPython-3.1.31-py3-none-any.whl", hash = "sha256:f04893614f6aa713a60cbbe1e6a97403ef633103cdd0ef5eb6efe0deb98dbe8d"},
+    {file = "GitPython-3.1.31.tar.gz", hash = "sha256:8ce3bcf69adfdf7c7d503e78fd3b1c492af782d58893b650adb2ac8912ddd573"},
 ]
 
 [package.dependencies]
@@ -726,89 +726,89 @@ files = [
 
 [[package]]
 name = "ijson"
-version = "3.2.3"
+version = "3.2.1"
 description = "Iterative JSON parser with standard Python iterator interfaces"
 optional = false
 python-versions = "*"
 files = [
-    {file = "ijson-3.2.3-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:0a4ae076bf97b0430e4e16c9cb635a6b773904aec45ed8dcbc9b17211b8569ba"},
-    {file = "ijson-3.2.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:cfced0a6ec85916eb8c8e22415b7267ae118eaff2a860c42d2cc1261711d0d31"},
-    {file = "ijson-3.2.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:0b9d1141cfd1e6d6643aa0b4876730d0d28371815ce846d2e4e84a2d4f471cf3"},
-    {file = "ijson-3.2.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9e0a27db6454edd6013d40a956d008361aac5bff375a9c04ab11fc8c214250b5"},
-    {file = "ijson-3.2.3-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:3c0d526ccb335c3c13063c273637d8611f32970603dfb182177b232d01f14c23"},
-    {file = "ijson-3.2.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:545a30b3659df2a3481593d30d60491d1594bc8005f99600e1bba647bb44cbb5"},
-    {file = "ijson-3.2.3-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:9680e37a10fedb3eab24a4a7e749d8a73f26f1a4c901430e7aa81b5da15f7307"},
-    {file = "ijson-3.2.3-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:2a80c0bb1053055d1599e44dc1396f713e8b3407000e6390add72d49633ff3bb"},
-    {file = "ijson-3.2.3-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:f05ed49f434ce396ddcf99e9fd98245328e99f991283850c309f5e3182211a79"},
-    {file = "ijson-3.2.3-cp310-cp310-win32.whl", hash = "sha256:b4eb2304573c9fdf448d3fa4a4fdcb727b93002b5c5c56c14a5ffbbc39f64ae4"},
-    {file = "ijson-3.2.3-cp310-cp310-win_amd64.whl", hash = "sha256:923131f5153c70936e8bd2dd9dcfcff43c67a3d1c789e9c96724747423c173eb"},
-    {file = "ijson-3.2.3-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:904f77dd3d87736ff668884fe5197a184748eb0c3e302ded61706501d0327465"},
-    {file = "ijson-3.2.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:0974444c1f416e19de1e9f567a4560890095e71e81623c509feff642114c1e53"},
-    {file = "ijson-3.2.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:c1a4b8eb69b6d7b4e94170aa991efad75ba156b05f0de2a6cd84f991def12ff9"},
-    {file = "ijson-3.2.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d052417fd7ce2221114f8d3b58f05a83c1a2b6b99cafe0b86ac9ed5e2fc889df"},
-    {file = "ijson-3.2.3-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:7b8064a85ec1b0beda7dd028e887f7112670d574db606f68006c72dd0bb0e0e2"},
-    {file = "ijson-3.2.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:eaac293853f1342a8d2a45ac1f723c860f700860e7743fb97f7b76356df883a8"},
-    {file = "ijson-3.2.3-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:6c32c18a934c1dc8917455b0ce478fd7a26c50c364bd52c5a4fb0fc6bb516af7"},
-    {file = "ijson-3.2.3-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:713a919e0220ac44dab12b5fed74f9130f3480e55e90f9d80f58de129ea24f83"},
-    {file = "ijson-3.2.3-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:4a3a6a2fbbe7550ffe52d151cf76065e6b89cfb3e9d0463e49a7e322a25d0426"},
-    {file = "ijson-3.2.3-cp311-cp311-win32.whl", hash = "sha256:6a4db2f7fb9acfb855c9ae1aae602e4648dd1f88804a0d5cfb78c3639bcf156c"},
-    {file = "ijson-3.2.3-cp311-cp311-win_amd64.whl", hash = "sha256:ccd6be56335cbb845f3d3021b1766299c056c70c4c9165fb2fbe2d62258bae3f"},
-    {file = "ijson-3.2.3-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:eeb286639649fb6bed37997a5e30eefcacddac79476d24128348ec890b2a0ccb"},
-    {file = "ijson-3.2.3-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:396338a655fb9af4ac59dd09c189885b51fa0eefc84d35408662031023c110d1"},
-    {file = "ijson-3.2.3-cp36-cp36m-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0e0243d166d11a2a47c17c7e885debf3b19ed136be2af1f5d1c34212850236ac"},
-    {file = "ijson-3.2.3-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:85afdb3f3a5d0011584d4fa8e6dccc5936be51c27e84cd2882fe904ca3bd04c5"},
-    {file = "ijson-3.2.3-cp36-cp36m-musllinux_1_1_aarch64.whl", hash = "sha256:4fc35d569eff3afa76bfecf533f818ecb9390105be257f3f83c03204661ace70"},
-    {file = "ijson-3.2.3-cp36-cp36m-musllinux_1_1_i686.whl", hash = "sha256:455d7d3b7a6aacfb8ab1ebcaf697eedf5be66e044eac32508fccdc633d995f0e"},
-    {file = "ijson-3.2.3-cp36-cp36m-musllinux_1_1_x86_64.whl", hash = "sha256:c63f3d57dbbac56cead05b12b81e8e1e259f14ce7f233a8cbe7fa0996733b628"},
-    {file = "ijson-3.2.3-cp36-cp36m-win32.whl", hash = "sha256:a4d7fe3629de3ecb088bff6dfe25f77be3e8261ed53d5e244717e266f8544305"},
-    {file = "ijson-3.2.3-cp36-cp36m-win_amd64.whl", hash = "sha256:96190d59f015b5a2af388a98446e411f58ecc6a93934e036daa75f75d02386a0"},
-    {file = "ijson-3.2.3-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:35194e0b8a2bda12b4096e2e792efa5d4801a0abb950c48ade351d479cd22ba5"},
-    {file = "ijson-3.2.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d1053fb5f0b010ee76ca515e6af36b50d26c1728ad46be12f1f147a835341083"},
-    {file = "ijson-3.2.3-cp37-cp37m-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:211124cff9d9d139dd0dfced356f1472860352c055d2481459038b8205d7d742"},
-    {file = "ijson-3.2.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:92dc4d48e9f6a271292d6079e9fcdce33c83d1acf11e6e12696fb05c5889fe74"},
-    {file = "ijson-3.2.3-cp37-cp37m-musllinux_1_1_aarch64.whl", hash = "sha256:3dcc33ee56f92a77f48776014ddb47af67c33dda361e84371153c4f1ed4434e1"},
-    {file = "ijson-3.2.3-cp37-cp37m-musllinux_1_1_i686.whl", hash = "sha256:98c6799925a5d1988da4cd68879b8eeab52c6e029acc45e03abb7921a4715c4b"},
-    {file = "ijson-3.2.3-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:4252e48c95cd8ceefc2caade310559ab61c37d82dfa045928ed05328eb5b5f65"},
-    {file = "ijson-3.2.3-cp37-cp37m-win32.whl", hash = "sha256:644f4f03349ff2731fd515afd1c91b9e439e90c9f8c28292251834154edbffca"},
-    {file = "ijson-3.2.3-cp37-cp37m-win_amd64.whl", hash = "sha256:ba33c764afa9ecef62801ba7ac0319268a7526f50f7601370d9f8f04e77fc02b"},
-    {file = "ijson-3.2.3-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:4b2ec8c2a3f1742cbd5f36b65e192028e541b5fd8c7fd97c1fc0ca6c427c704a"},
-    {file = "ijson-3.2.3-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:7dc357da4b4ebd8903e77dbcc3ce0555ee29ebe0747c3c7f56adda423df8ec89"},
-    {file = "ijson-3.2.3-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:bcc51c84bb220ac330122468fe526a7777faa6464e3b04c15b476761beea424f"},
-    {file = "ijson-3.2.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f8d54b624629f9903005c58d9321a036c72f5c212701bbb93d1a520ecd15e370"},
-    {file = "ijson-3.2.3-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:d6ea7c7e3ec44742e867c72fd750c6a1e35b112f88a917615332c4476e718d40"},
-    {file = "ijson-3.2.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:916acdc5e504f8b66c3e287ada5d4b39a3275fc1f2013c4b05d1ab9933671a6c"},
-    {file = "ijson-3.2.3-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:81815b4184b85ce124bfc4c446d5f5e5e643fc119771c5916f035220ada29974"},
-    {file = "ijson-3.2.3-cp38-cp38-musllinux_1_1_i686.whl", hash = "sha256:b49fd5fe1cd9c1c8caf6c59f82b08117dd6bea2ec45b641594e25948f48f4169"},
-    {file = "ijson-3.2.3-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:86b3c91fdcb8ffb30556c9669930f02b7642de58ca2987845b04f0d7fe46d9a8"},
-    {file = "ijson-3.2.3-cp38-cp38-win32.whl", hash = "sha256:a729b0c8fb935481afe3cf7e0dadd0da3a69cc7f145dbab8502e2f1e01d85a7c"},
-    {file = "ijson-3.2.3-cp38-cp38-win_amd64.whl", hash = "sha256:d34e049992d8a46922f96483e96b32ac4c9cffd01a5c33a928e70a283710cd58"},
-    {file = "ijson-3.2.3-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:9c2a12dcdb6fa28f333bf10b3a0f80ec70bc45280d8435be7e19696fab2bc706"},
-    {file = "ijson-3.2.3-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:1844c5b57da21466f255a0aeddf89049e730d7f3dfc4d750f0e65c36e6a61a7c"},
-    {file = "ijson-3.2.3-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:2ec3e5ff2515f1c40ef6a94983158e172f004cd643b9e4b5302017139b6c96e4"},
-    {file = "ijson-3.2.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:46bafb1b9959872a1f946f8dd9c6f1a30a970fc05b7bfae8579da3f1f988e598"},
-    {file = "ijson-3.2.3-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ab4db9fee0138b60e31b3c02fff8a4c28d7b152040553b6a91b60354aebd4b02"},
-    {file = "ijson-3.2.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f4bc87e69d1997c6a55fff5ee2af878720801ff6ab1fb3b7f94adda050651e37"},
-    {file = "ijson-3.2.3-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:e9fd906f0c38e9f0bfd5365e1bed98d649f506721f76bb1a9baa5d7374f26f19"},
-    {file = "ijson-3.2.3-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:e84d27d1acb60d9102728d06b9650e5b7e5cb0631bd6e3dfadba8fb6a80d6c2f"},
-    {file = "ijson-3.2.3-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:2cc04fc0a22bb945cd179f614845c8b5106c0b3939ee0d84ce67c7a61ac1a936"},
-    {file = "ijson-3.2.3-cp39-cp39-win32.whl", hash = "sha256:e641814793a037175f7ec1b717ebb68f26d89d82cfd66f36e588f32d7e488d5f"},
-    {file = "ijson-3.2.3-cp39-cp39-win_amd64.whl", hash = "sha256:6bd3e7e91d031f1e8cea7ce53f704ab74e61e505e8072467e092172422728b22"},
-    {file = "ijson-3.2.3-pp37-pypy37_pp73-macosx_10_9_x86_64.whl", hash = "sha256:06f9707da06a19b01013f8c65bf67db523662a9b4a4ff027e946e66c261f17f0"},
-    {file = "ijson-3.2.3-pp37-pypy37_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:be8495f7c13fa1f622a2c6b64e79ac63965b89caf664cc4e701c335c652d15f2"},
-    {file = "ijson-3.2.3-pp37-pypy37_pp73-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:7596b42f38c3dcf9d434dddd50f46aeb28e96f891444c2b4b1266304a19a2c09"},
-    {file = "ijson-3.2.3-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fbac4e9609a1086bbad075beb2ceec486a3b138604e12d2059a33ce2cba93051"},
-    {file = "ijson-3.2.3-pp37-pypy37_pp73-win_amd64.whl", hash = "sha256:db2d6341f9cb538253e7fe23311d59252f124f47165221d3c06a7ed667ecd595"},
-    {file = "ijson-3.2.3-pp38-pypy38_pp73-macosx_10_9_x86_64.whl", hash = "sha256:fa8b98be298efbb2588f883f9953113d8a0023ab39abe77fe734b71b46b1220a"},
-    {file = "ijson-3.2.3-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:674e585361c702fad050ab4c153fd168dc30f5980ef42b64400bc84d194e662d"},
-    {file = "ijson-3.2.3-pp38-pypy38_pp73-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:fd12e42b9cb9c0166559a3ffa276b4f9fc9d5b4c304e5a13668642d34b48b634"},
-    {file = "ijson-3.2.3-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d31e0d771d82def80cd4663a66de277c3b44ba82cd48f630526b52f74663c639"},
-    {file = "ijson-3.2.3-pp38-pypy38_pp73-win_amd64.whl", hash = "sha256:7ce4c70c23521179d6da842bb9bc2e36bb9fad1e0187e35423ff0f282890c9ca"},
-    {file = "ijson-3.2.3-pp39-pypy39_pp73-macosx_10_9_x86_64.whl", hash = "sha256:39f551a6fbeed4433c85269c7c8778e2aaea2501d7ebcb65b38f556030642c17"},
-    {file = "ijson-3.2.3-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3b14d322fec0de7af16f3ef920bf282f0dd747200b69e0b9628117f381b7775b"},
-    {file = "ijson-3.2.3-pp39-pypy39_pp73-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:7851a341429b12d4527ca507097c959659baf5106c7074d15c17c387719ffbcd"},
-    {file = "ijson-3.2.3-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:db3bf1b42191b5cc9b6441552fdcb3b583594cb6b19e90d1578b7cbcf80d0fae"},
-    {file = "ijson-3.2.3-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:6f662dc44362a53af3084d3765bb01cd7b4734d1f484a6095cad4cb0cbfe5374"},
-    {file = "ijson-3.2.3.tar.gz", hash = "sha256:10294e9bf89cb713da05bc4790bdff616610432db561964827074898e174f917"},
+    {file = "ijson-3.2.1-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:6f827f6961f093e1055a2be0c3137f0e7d667979da455ac9648f72d4a2bb8970"},
+    {file = "ijson-3.2.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:b6e51f4497065cd0d09f5e906cd538a8d22609eab716e3c883769acf147ab1b6"},
+    {file = "ijson-3.2.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:f022686c40bff3e340627a5a0c9212718d529e787ada3b76ba546d47a9ecdbbd"},
+    {file = "ijson-3.2.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e4105c15a13fa1dc24ebd3bf2e679fa14dcbfcc48bc39138a0fa3f4ddf6cc09b"},
+    {file = "ijson-3.2.1-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:404423e666f185dfb753ddc92705c84dffdc4cc872aaf825bbe0607893cb5b02"},
+    {file = "ijson-3.2.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:39e71f32830827cf21d0233a814092e5a23668e18f52eca5cac4f670d9df1240"},
+    {file = "ijson-3.2.1-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:43af7ed5292caa1452747e2b62485b6c0ece4bcbc5bf6f2758abd547e4124a14"},
+    {file = "ijson-3.2.1-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:e805aa6897a11b0f73f1f6bca078981df8960aeeccf527a214f240409c742bab"},
+    {file = "ijson-3.2.1-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:5b2df0bd84889e9017a670328fe3e82ec509fd6744c7ac2c99c7ee2300d76afa"},
+    {file = "ijson-3.2.1-cp310-cp310-win32.whl", hash = "sha256:675259c7ea7f51ffaf8cb9e79bf875e28bb09622892943f4f415588fd7ab7bec"},
+    {file = "ijson-3.2.1-cp310-cp310-win_amd64.whl", hash = "sha256:90d4b2eb771a3585c8186820fe50e3282ef62477b865e765a50a8295674abeac"},
+    {file = "ijson-3.2.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:fc581a61e210bf6013c1fa6536566e51127be1cfbd69539b63d8b813206d2fe0"},
+    {file = "ijson-3.2.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:75cdf7ad4c00a8f5ac94ff27e3b7c1bf5ac463f125bca2be1744c5bc9600db5c"},
+    {file = "ijson-3.2.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:85a2bf4636ace4d92e7c5d857a1c5694f42407c868953cf2927f18127bcd0d58"},
+    {file = "ijson-3.2.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9fe0cb66e7dd4aa11da5fff60bdf5ee04819a5e6a57acf7ca12c65f7fc009afc"},
+    {file = "ijson-3.2.1-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:c6f7957ad38cb714378944032f2c2ee9c6531b5b0b38c5ccd08cedbb0ceddd02"},
+    {file = "ijson-3.2.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:13283d264cca8a63e5bad91e82eec39711e95893e7e8d4a419799a8c5f85203a"},
+    {file = "ijson-3.2.1-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:12c24cde850fe79bc806be0e9fc38b47dd5ac0a223070ccb12e9b695425e2936"},
+    {file = "ijson-3.2.1-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:2ce8eed838e5a0791cb5948117b5453f2b3b3c28d93d06ee2bbf2c198c47881c"},
+    {file = "ijson-3.2.1-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:b81c2589f191b0dc741f532be00b4bea617297dd9698431c8053e2d28272d4db"},
+    {file = "ijson-3.2.1-cp311-cp311-win32.whl", hash = "sha256:ba2beac56ac96f728d0f2430e4c667c66819a423d321bb9db9ebdebd803e1b5b"},
+    {file = "ijson-3.2.1-cp311-cp311-win_amd64.whl", hash = "sha256:c71614ed4bbc6a32ff1e42d7ce92a176fb67d658913343792d2c4567aa130817"},
+    {file = "ijson-3.2.1-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:683fc8b0ea085e75ea34044fdc70649b37367d494f132a2bd1e59d7135054d89"},
+    {file = "ijson-3.2.1-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:deeaecec2f4e20e8bec20b0a5cdc34daebe7903f2e700f7dcaef68b5925d35ea"},
+    {file = "ijson-3.2.1-cp36-cp36m-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:11923ac3188877f19dbb7051f7345202701cc39bf8e5ac44f8ae536c9eca8c82"},
+    {file = "ijson-3.2.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:400deefcdae21e90fc39c1dcfc6ba2df24537e8c65bd57b763ed5256b73ba64d"},
+    {file = "ijson-3.2.1-cp36-cp36m-musllinux_1_1_aarch64.whl", hash = "sha256:56bc4bad53770710a3a91944fe640fdeb269987a14352b74ebbad2aa55801c00"},
+    {file = "ijson-3.2.1-cp36-cp36m-musllinux_1_1_i686.whl", hash = "sha256:f5a179523e085126844c6161aabcd193dbb5747bd01fadb68e92abf048f32ec9"},
+    {file = "ijson-3.2.1-cp36-cp36m-musllinux_1_1_x86_64.whl", hash = "sha256:ee24655986e4415fbb7a0cf51445fff3072ceac0e219f4bbbd5c53535a3c5680"},
+    {file = "ijson-3.2.1-cp36-cp36m-win32.whl", hash = "sha256:4a5c672b0540005c1bb0bba97aa559a87a2e4ee409fc68e2f5ba5b30f009ac99"},
+    {file = "ijson-3.2.1-cp36-cp36m-win_amd64.whl", hash = "sha256:cfaf1d89b0e122e69c87a15db6d6f44feb9db96d2af7fe88cdc464177a257b5d"},
+    {file = "ijson-3.2.1-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:1cbd052eb67c1b3611f25974ba967886e89391faaf55afec93808c19f06ca612"},
+    {file = "ijson-3.2.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f13ffc491886e5d7bde7d68712d168bce0141b2a918db1164bc8599c0123e293"},
+    {file = "ijson-3.2.1-cp37-cp37m-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:bc4c4fc6bafc777f8422fe36edb1cbd72a13cb29695893a064c9c95776a4bdf9"},
+    {file = "ijson-3.2.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a42fcb2bf9748c26f004690b2feb6e13e4875bb7c9d83535f887c21e0a982a7c"},
+    {file = "ijson-3.2.1-cp37-cp37m-musllinux_1_1_aarch64.whl", hash = "sha256:0c92f7bc2f3a947c2ba7f7aa48382c36079f8259c930e81d9164341f9b853c45"},
+    {file = "ijson-3.2.1-cp37-cp37m-musllinux_1_1_i686.whl", hash = "sha256:fd497042db562379339660e787bc8679ed3abaa740768d39bc3746e769e7c7a5"},
+    {file = "ijson-3.2.1-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:7d61c7cd8ddd75dcef818ff5a111a31b902a6a0e410ee0c2b2ecaa6dac92658a"},
+    {file = "ijson-3.2.1-cp37-cp37m-win32.whl", hash = "sha256:36caf624d263fc40e7e805d759d09ea368d8cf497aecb3241ac2f0a286ad8eca"},
+    {file = "ijson-3.2.1-cp37-cp37m-win_amd64.whl", hash = "sha256:32f9ed25ff80942e433119600bca13b86a8f9b8b0966edbc1d91a48ccbdd4d54"},
+    {file = "ijson-3.2.1-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:e89bbd747140eac3a3c9e7e5835b90d85c4a02763fc5134861bfc1ea03b66ae7"},
+    {file = "ijson-3.2.1-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:d69b4b1d509de36ec42a0e4af30ede39fb754e4039b2928ef7282ebc2125ffdd"},
+    {file = "ijson-3.2.1-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:e7feb0771f50deabe6ce85b210fa9e005843d3d3c60fb3315d69e1f9d0d75e0c"},
+    {file = "ijson-3.2.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5fd8148a363888054ff06eaaa1103f2f98720ab39666084a214e4fedfc13cf64"},
+    {file = "ijson-3.2.1-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:598638dcc5141e9ae269903901877103f5362e0db4443e34721df8f8d34577b4"},
+    {file = "ijson-3.2.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e979190b7d0fabca20d6b7224ab1c1aa461ad1ab72ba94f1bb1e5894cd59f342"},
+    {file = "ijson-3.2.1-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:bc810eb80b4f486c7957201ba2a53f53ddc9b3233af67e4359e29371bf04883b"},
+    {file = "ijson-3.2.1-cp38-cp38-musllinux_1_1_i686.whl", hash = "sha256:26e758584611dfe826dd18ffd94dc0d8a062ce56e41674ad3bfa371c7b78c4b5"},
+    {file = "ijson-3.2.1-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:24e9ae5b35b85ea094b6c36495bc856089254aed6a48bada8d7eec5a04f74439"},
+    {file = "ijson-3.2.1-cp38-cp38-win32.whl", hash = "sha256:4b5dc7b5b4b8cb3087d188f37911cd67e26672d33d3571e73440de3f0a86f7e6"},
+    {file = "ijson-3.2.1-cp38-cp38-win_amd64.whl", hash = "sha256:1af94ff40609270bbb3eac47e072582bb578f5023fac8408cccd80fe5892d221"},
+    {file = "ijson-3.2.1-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:2dda67affceebc52c8bc5fe72c3a4a1e338e4d4b0497dbac5089c2d3862df214"},
+    {file = "ijson-3.2.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:bd780303ddfedc8d57cdb9f2d53a8cea2f2f4a6fb857bf8fe5a0c3ab1d4ca901"},
+    {file = "ijson-3.2.1-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:4fbab6af1bab88a8e46beda08cf44610eed0adb8d157a1a60b4bb6c3a121c6de"},
+    {file = "ijson-3.2.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c97a07988a1e0ce2bc8e8a62eb5f25195a3bd58a939ac353cbc6018a548cc08d"},
+    {file = "ijson-3.2.1-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:a65671a6826ae723837143914c823ad7bcc0d1a3e38d87c71df897a2556fb48f"},
+    {file = "ijson-3.2.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a1806372008bbed9ee92db5747e38c047fa1c4ee89cb2dd5daaa57feb46ce50a"},
+    {file = "ijson-3.2.1-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:91e5a8e96f78a59e2520078c227a4fec5bf91c13adeded9e33fb13981cb823c3"},
+    {file = "ijson-3.2.1-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:1f820fce8ef093718f2319ff6f1322390664659b783775919dadccb1b470153d"},
+    {file = "ijson-3.2.1-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:bca3e8c91a1076a20620dbaa6a2848772b0e8a4055e86d42d3fa39221b53ed1a"},
+    {file = "ijson-3.2.1-cp39-cp39-win32.whl", hash = "sha256:de87f137b7438d43840f4339a37d4e6a58c987f4bb2a70609969f854f8ae20f3"},
+    {file = "ijson-3.2.1-cp39-cp39-win_amd64.whl", hash = "sha256:0caebb8350b47266a58b766ec08e1de441d6d160702c428b5cf7504d93c832c4"},
+    {file = "ijson-3.2.1-pp37-pypy37_pp73-macosx_10_9_x86_64.whl", hash = "sha256:37389785c1abd27fcc24800fcfa9a6b1022743413e4056507fd32356b623ff33"},
+    {file = "ijson-3.2.1-pp37-pypy37_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4b364b82231d51cbeae52468c3b27e8a042e544ab764c8f3975e912cf010603f"},
+    {file = "ijson-3.2.1-pp37-pypy37_pp73-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0a5999d0ec28a8ec47cf20c736fd4f895dc077bf6441bf237b00b074315a295d"},
+    {file = "ijson-3.2.1-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8bd481857a39305517fb6f1313d558c2dc4e78c9e9384cc5bc1c3e28f1afbedf"},
+    {file = "ijson-3.2.1-pp37-pypy37_pp73-win_amd64.whl", hash = "sha256:545f62f12f89350d4d73f2a779cb269198ae578fac080085a1927148b803e602"},
+    {file = "ijson-3.2.1-pp38-pypy38_pp73-macosx_10_9_x86_64.whl", hash = "sha256:4d5622505d01c2f3d7b9638c1eb8c747eb550936b505225893704289ff28576f"},
+    {file = "ijson-3.2.1-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:20293bb36423b129fad3753858ccf7b2ccb5b2c0d3759efe810d0b9d79633a7e"},
+    {file = "ijson-3.2.1-pp38-pypy38_pp73-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:7cd8a4921b852fd2cb5b0c985540c97ff6893139a57fe7121d510ec5d1c0ca44"},
+    {file = "ijson-3.2.1-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:dc902ff1ae1efed7d526294d7a9dd3df66d29b2cdc05fb5479838fef1327a534"},
+    {file = "ijson-3.2.1-pp38-pypy38_pp73-win_amd64.whl", hash = "sha256:2925a7978d8170146a9cb49a15a982b71fbbf21980bf2e16cd90c528545b7c02"},
+    {file = "ijson-3.2.1-pp39-pypy39_pp73-macosx_10_9_x86_64.whl", hash = "sha256:c21c6509f6944939399f3630c5dc424d30d71d375f6cd58f9af56158fdf7251c"},
+    {file = "ijson-3.2.1-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f5729fc7648bc972d70922d7dad15459cca3a9e5ed0328eb9ae3ffa004066194"},
+    {file = "ijson-3.2.1-pp39-pypy39_pp73-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:805a2d5ed5a15d60327bc9347f2d4125ab621fb18071db98b1c598f1ee99e8f1"},
+    {file = "ijson-3.2.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3d0220a4b6c63f44589e429157174e3f4b8d1e534d5fb82bdb43a7f8dd77ae4b"},
+    {file = "ijson-3.2.1-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:271d9b7c083f65c58ff0afd9dbb5d2f3d445f734632aebfef4a261b0a337abdb"},
+    {file = "ijson-3.2.1.tar.gz", hash = "sha256:8574bf19f31fab870488769ad919a80f130825236ac8bde9a733f69c2961d7a7"},
 ]
 
 [[package]]
@@ -887,17 +887,17 @@ scripts = ["click (>=6.0)", "twisted (>=16.4.0)"]
 
 [[package]]
 name = "isort"
-version = "5.12.0"
+version = "5.11.5"
 description = "A Python utility / library to sort Python imports."
 optional = false
-python-versions = ">=3.8.0"
+python-versions = ">=3.7.0"
 files = [
-    {file = "isort-5.12.0-py3-none-any.whl", hash = "sha256:f84c2818376e66cf843d497486ea8fed8700b340f308f076c6fb1229dff318b6"},
-    {file = "isort-5.12.0.tar.gz", hash = "sha256:8bef7dde241278824a6d83f44a544709b065191b95b6e50894bdc722fcba0504"},
+    {file = "isort-5.11.5-py3-none-any.whl", hash = "sha256:ba1d72fb2595a01c7895a5128f9585a5cc4b6d395f1c8d514989b9a7eb2a8746"},
+    {file = "isort-5.11.5.tar.gz", hash = "sha256:6be1f76a507cb2ecf16c7cf14a37e41609ca082330be4e3436a18ef74add55db"},
 ]
 
 [package.extras]
-colors = ["colorama (>=0.4.3)"]
+colors = ["colorama (>=0.4.3,<0.5.0)"]
 pipfile-deprecated-finder = ["pip-shims (>=0.5.2)", "pipreqs", "requirementslib"]
 plugins = ["setuptools"]
 requirements-deprecated-finder = ["pip-api", "pipreqs"]
@@ -973,13 +973,13 @@ i18n = ["Babel (>=2.7)"]
 
 [[package]]
 name = "jsonschema"
-version = "4.19.0"
+version = "4.18.3"
 description = "An implementation of JSON Schema validation for Python"
 optional = false
 python-versions = ">=3.8"
 files = [
-    {file = "jsonschema-4.19.0-py3-none-any.whl", hash = "sha256:043dc26a3845ff09d20e4420d6012a9c91c9aa8999fa184e7efcfeccb41e32cb"},
-    {file = "jsonschema-4.19.0.tar.gz", hash = "sha256:6e1e7569ac13be8139b2dd2c21a55d350066ee3f80df06c608b398cdc6f30e8f"},
+    {file = "jsonschema-4.18.3-py3-none-any.whl", hash = "sha256:aab78b34c2de001c6b692232f08c21a97b436fe18e0b817bf0511046924fceef"},
+    {file = "jsonschema-4.18.3.tar.gz", hash = "sha256:64b7104d72efe856bea49ca4af37a14a9eba31b40bb7238179f3803130fd34d9"},
 ]
 
 [package.dependencies]
@@ -1610,13 +1610,13 @@ files = [
 
 [[package]]
 name = "phonenumbers"
-version = "8.13.18"
+version = "8.13.14"
 description = "Python version of Google's common library for parsing, formatting, storing and validating international phone numbers."
 optional = false
 python-versions = "*"
 files = [
-    {file = "phonenumbers-8.13.18-py2.py3-none-any.whl", hash = "sha256:3d802739a22592e4127139349937753dee9b6a20bdd5d56847cd885bdc766b1f"},
-    {file = "phonenumbers-8.13.18.tar.gz", hash = "sha256:b360c756252805d44b447b5bca6d250cf6bd6c69b6f0f4258f3bfe5ab81bef69"},
+    {file = "phonenumbers-8.13.14-py2.py3-none-any.whl", hash = "sha256:a4b20b6ba7dd402728f5cc8e86e1f29b1a873af45f5381dbee7e3083af497ff6"},
+    {file = "phonenumbers-8.13.14.tar.gz", hash = "sha256:5fa952b4abf9fccdaf1f130d96114a520c48890d4091b50a064e22c0fdc12dec"},
 ]
 
 [[package]]
@@ -1881,13 +1881,13 @@ email = ["email-validator (>=1.0.3)"]
 
 [[package]]
 name = "pygithub"
-version = "1.59.1"
+version = "1.59.0"
 description = "Use the full Github API v3"
 optional = false
 python-versions = ">=3.7"
 files = [
-    {file = "PyGithub-1.59.1-py3-none-any.whl", hash = "sha256:3d87a822e6c868142f0c2c4bf16cce4696b5a7a4d142a7bd160e1bdf75bc54a9"},
-    {file = "PyGithub-1.59.1.tar.gz", hash = "sha256:c44e3a121c15bf9d3a5cc98d94c9a047a5132a9b01d22264627f58ade9ddc217"},
+    {file = "PyGithub-1.59.0-py3-none-any.whl", hash = "sha256:126bdbae72087d8d038b113aab6b059b4553cb59348e3024bb1a1cae406ace9e"},
+    {file = "PyGithub-1.59.0.tar.gz", hash = "sha256:6e05ff49bac3caa7d1d6177a10c6e55a3e20c85b92424cc198571fd0cf786690"},
 ]
 
 [package.dependencies]
@@ -2385,13 +2385,13 @@ doc = ["Sphinx", "sphinx-rtd-theme"]
 
 [[package]]
 name = "sentry-sdk"
-version = "1.29.2"
+version = "1.28.1"
 description = "Python client for Sentry (https://sentry.io)"
 optional = true
 python-versions = "*"
 files = [
-    {file = "sentry-sdk-1.29.2.tar.gz", hash = "sha256:a99ee105384788c3f228726a88baf515fe7b5f1d2d0f215a03d194369f158df7"},
-    {file = "sentry_sdk-1.29.2-py2.py3-none-any.whl", hash = "sha256:3e17215d8006612e2df02b0e73115eb8376c37e3f586d8436fa41644e605074d"},
+    {file = "sentry-sdk-1.28.1.tar.gz", hash = "sha256:dcd88c68aa64dae715311b5ede6502fd684f70d00a7cd4858118f0ba3153a3ae"},
+    {file = "sentry_sdk-1.28.1-py2.py3-none-any.whl", hash = "sha256:6bdb25bd9092478d3a817cb0d01fa99e296aea34d404eac3ca0037faa5c2aa0a"},
 ]
 
 [package.dependencies]
@@ -2921,13 +2921,13 @@ files = [
 
 [[package]]
 name = "txredisapi"
-version = "1.4.10"
+version = "1.4.9"
 description = "non-blocking redis client for python"
 optional = true
 python-versions = "*"
 files = [
-    {file = "txredisapi-1.4.10-py3-none-any.whl", hash = "sha256:0a6ea77f27f8cf092f907654f08302a97b48fa35f24e0ad99dfb74115f018161"},
-    {file = "txredisapi-1.4.10.tar.gz", hash = "sha256:7609a6af6ff4619a3189c0adfb86aeda789afba69eb59fc1e19ac0199e725395"},
+    {file = "txredisapi-1.4.9-py3-none-any.whl", hash = "sha256:72e6ad09cc5fffe3bec2e55e5bfb74407bd357565fc212e6003f7e26ef7d8f78"},
+    {file = "txredisapi-1.4.9.tar.gz", hash = "sha256:c9607062d05e4d0b8ef84719eb76a3fe7d5ccd606a2acf024429da51d6e84559"},
 ]
 
 [package.dependencies]
@@ -2936,13 +2936,13 @@ twisted = "*"
 
 [[package]]
 name = "types-bleach"
-version = "6.0.0.4"
+version = "6.0.0.3"
 description = "Typing stubs for bleach"
 optional = false
 python-versions = "*"
 files = [
-    {file = "types-bleach-6.0.0.4.tar.gz", hash = "sha256:357b0226f65c4f20ab3b13ca8d78a6b91c78aad256d8ec168d4e90fc3303ebd4"},
-    {file = "types_bleach-6.0.0.4-py3-none-any.whl", hash = "sha256:2b8767eb407c286b7f02803678732e522e04db8d56cbc9f1270bee49627eae92"},
+    {file = "types-bleach-6.0.0.3.tar.gz", hash = "sha256:8ce7896d4f658c562768674ffcf07492c7730e128018f03edd163ff912bfadee"},
+    {file = "types_bleach-6.0.0.3-py3-none-any.whl", hash = "sha256:d43eaf30a643ca824e16e2dcdb0c87ef9226237e2fa3ac4732a50cb3f32e145f"},
 ]
 
 [[package]]
@@ -2980,24 +2980,24 @@ files = [
 
 [[package]]
 name = "types-opentracing"
-version = "2.4.10.6"
+version = "2.4.10.5"
 description = "Typing stubs for opentracing"
 optional = false
 python-versions = "*"
 files = [
-    {file = "types-opentracing-2.4.10.6.tar.gz", hash = "sha256:87a1bdfce9de5e555e30497663583b9b9c3bb494d029ef9806aa1f137c19e744"},
-    {file = "types_opentracing-2.4.10.6-py3-none-any.whl", hash = "sha256:25914c834db033a4a38fc322df0b5e5e14503b0ac97f78304ae180d721555e97"},
+    {file = "types-opentracing-2.4.10.5.tar.gz", hash = "sha256:852d13ab1324832835d50c00cfd58b9267f0e79ec3189e5664c2a90c26880fd4"},
+    {file = "types_opentracing-2.4.10.5-py3-none-any.whl", hash = "sha256:8f12ab4dce3e298a8e6655da9a6d52171e7a275357eae4cec22a1663d94023a7"},
 ]
 
 [[package]]
 name = "types-pillow"
-version = "10.0.0.2"
+version = "10.0.0.1"
 description = "Typing stubs for Pillow"
 optional = false
 python-versions = "*"
 files = [
-    {file = "types-Pillow-10.0.0.2.tar.gz", hash = "sha256:fe09380ab22d412ced989a067e9ee4af719fa3a47ba1b53b232b46514a871042"},
-    {file = "types_Pillow-10.0.0.2-py3-none-any.whl", hash = "sha256:29d51a3ce6ef51fabf728a504d33b4836187ff14256b2e86996d55c91ab214b1"},
+    {file = "types-Pillow-10.0.0.1.tar.gz", hash = "sha256:834a07a04504f8bf37936679bc6a5802945e7644d0727460c0c4d4307967e2a3"},
+    {file = "types_Pillow-10.0.0.1-py3-none-any.whl", hash = "sha256:be576b67418f1cb3b93794cf7946581be1009a33a10085b3c132eb0875a819b4"},
 ]
 
 [[package]]
@@ -3013,13 +3013,13 @@ files = [
 
 [[package]]
 name = "types-pyopenssl"
-version = "23.2.0.2"
+version = "23.2.0.1"
 description = "Typing stubs for pyOpenSSL"
 optional = false
 python-versions = "*"
 files = [
-    {file = "types-pyOpenSSL-23.2.0.2.tar.gz", hash = "sha256:6a010dac9ecd42b582d7dd2cc3e9e40486b79b3b64bb2fffba1474ff96af906d"},
-    {file = "types_pyOpenSSL-23.2.0.2-py3-none-any.whl", hash = "sha256:19536aa3debfbe25a918cf0d898e9f5fbbe6f3594a429da7914bf331deb1b342"},
+    {file = "types-pyOpenSSL-23.2.0.1.tar.gz", hash = "sha256:beeb5d22704c625a1e4b6dc756355c5b4af0b980138b702a9d9f932acf020903"},
+    {file = "types_pyOpenSSL-23.2.0.1-py3-none-any.whl", hash = "sha256:0568553f104466f1b8e0db3360fbe6770137d02e21a1a45c209bf2b1b03d90d4"},
 ]
 
 [package.dependencies]
@@ -3052,13 +3052,13 @@ types-urllib3 = "*"
 
 [[package]]
 name = "types-setuptools"
-version = "68.0.0.3"
+version = "68.0.0.0"
 description = "Typing stubs for setuptools"
 optional = false
 python-versions = "*"
 files = [
-    {file = "types-setuptools-68.0.0.3.tar.gz", hash = "sha256:d57ae6076100b5704b3cc869fdefc671e1baf4c2cd6643f84265dfc0b955bf05"},
-    {file = "types_setuptools-68.0.0.3-py3-none-any.whl", hash = "sha256:fec09e5c18264c5c09351c00be01a34456fb7a88e457abe97401325f84ad9d36"},
+    {file = "types-setuptools-68.0.0.0.tar.gz", hash = "sha256:fc958b4123b155ffc069a66d3af5fe6c1f9d0600c35c0c8444b2ab4147112641"},
+    {file = "types_setuptools-68.0.0.0-py3-none-any.whl", hash = "sha256:cc00e09ba8f535362cbe1ea8b8407d15d14b59c57f4190cceaf61a9e57616446"},
 ]
 
 [[package]]

pyproject.toml

@@ -89,7 +89,7 @@ manifest-path = "rust/Cargo.toml"
 
 [tool.poetry]
 name = "matrix-synapse"
-version = "1.91.1"
+version = "1.89.0"
 description = "Homeserver for the Matrix decentralised comms protocol"
 authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
 license = "Apache-2.0"
@@ -367,7 +367,7 @@ furo = ">=2022.12.7,<2024.0.0"
 # system changes.
 # We are happy to raise these upper bounds upon request,
 # provided we check that it's safe to do so (i.e. that CI passes).
-requires = ["poetry-core>=1.1.0,<=1.7.0", "setuptools_rust>=1.3,<=1.6.0"]
+requires = ["poetry-core>=1.1.0,<=1.6.0", "setuptools_rust>=1.3,<=1.6.0"]
 build-backend = "poetry.core.masonry.api"
 
 

scripts-dev/build_debian_packages.py

@@ -47,7 +47,7 @@ can be passed on the commandline for debugging.
 projdir = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
 
 
-class Builder:
+class Builder(object):
     def __init__(
         self,
         redirect_stdout: bool = False,

scripts-dev/check_schema_delta.py

@@ -43,7 +43,7 @@ def main(force_colors: bool) -> None:
     diffs: List[git.Diff] = repo.remote().refs.develop.commit.diff(None)
 
     # Get the schema version of the local file to check against current schema on develop
-    with open("synapse/storage/schema/__init__.py") as file:
+    with open("synapse/storage/schema/__init__.py", "r") as file:
         local_schema = file.read()
     new_locals: Dict[str, Any] = {}
     exec(local_schema, new_locals)

scripts-dev/federation_client.py

@@ -247,7 +247,7 @@ def main() -> None:
 
 
 def read_args_from_config(args: argparse.Namespace) -> None:
-    with open(args.config) as fh:
+    with open(args.config, "r") as fh:
         config = yaml.safe_load(fh)
 
         if not args.server_name:

scripts-dev/release.py

@@ -1,4 +1,5 @@
 #!/usr/bin/env python
+# -*- coding: utf-8 -*-
 # Copyright 2020 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");

scripts-dev/sign_json.py

@@ -145,7 +145,7 @@ Example usage:
 
 
 def read_args_from_config(args: argparse.Namespace) -> None:
-    with open(args.config) as fh:
+    with open(args.config, "r") as fh:
         config = yaml.safe_load(fh)
         if not args.server_name:
             args.server_name = config["server_name"]

synapse/__init__.py

@@ -25,11 +25,7 @@ from synapse.util.rust import check_rust_lib_up_to_date
 from synapse.util.stringutils import strtobool
 
 # Check that we're not running on an unsupported Python version.
-#
-# Note that we use an (unneeded) variable here so that pyupgrade doesn't nuke the
-# if-statement completely.
-py_version = sys.version_info
-if py_version < (3, 8):
+if sys.version_info < (3, 8):
     print("Synapse requires Python 3.8 or above.")
     sys.exit(1)
 
@@ -82,7 +78,7 @@ try:
 except ImportError:
     pass
 
-import synapse.util  # noqa: E402
+import synapse.util
 
 __version__ = synapse.util.SYNAPSE_VERSION
 

synapse/_scripts/synapse_port_db.py

@@ -123,7 +123,7 @@ BOOLEAN_COLUMNS = {
     "redactions": ["have_censored"],
     "room_stats_state": ["is_federatable"],
     "rooms": ["is_public", "has_auth_chain_index"],
-    "users": ["shadow_banned", "approved", "locked"],
+    "users": ["shadow_banned", "approved"],
     "un_partial_stated_event_stream": ["rejection_status_changed"],
     "users_who_share_rooms": ["share_private"],
     "per_user_experimental_features": ["enabled"],
@@ -1205,10 +1205,10 @@ class CursesProgress(Progress):
         self.total_processed = 0
         self.total_remaining = 0
 
-        super().__init__()
+        super(CursesProgress, self).__init__()
 
     def update(self, table: str, num_done: int) -> None:
-        super().update(table, num_done)
+        super(CursesProgress, self).update(table, num_done)
 
         self.total_processed = 0
         self.total_remaining = 0
@@ -1304,7 +1304,7 @@ class TerminalProgress(Progress):
     """Just prints progress to the terminal"""
 
     def update(self, table: str, num_done: int) -> None:
-        super().update(table, num_done)
+        super(TerminalProgress, self).update(table, num_done)
 
         data = self.tables[table]
 

synapse/_scripts/update_synapse_database.py

@@ -38,7 +38,7 @@ class MockHomeserver(HomeServer):
     DATASTORE_CLASS = DataStore  # type: ignore [assignment]
 
     def __init__(self, config: HomeServerConfig):
-        super().__init__(
+        super(MockHomeserver, self).__init__(
             hostname=config.server.server_name,
             config=config,
             reactor=reactor,

synapse/api/auth/__init__.py

@@ -60,7 +60,6 @@ class Auth(Protocol):
         request: SynapseRequest,
         allow_guest: bool = False,
         allow_expired: bool = False,
-        allow_locked: bool = False,
     ) -> Requester:
         """Get a registered user's ID.
 

synapse/api/auth/internal.py

@@ -58,7 +58,6 @@ class InternalAuth(BaseAuth):
         request: SynapseRequest,
         allow_guest: bool = False,
         allow_expired: bool = False,
-        allow_locked: bool = False,
     ) -> Requester:
         """Get a registered user's ID.
 
@@ -80,7 +79,7 @@ class InternalAuth(BaseAuth):
         parent_span = active_span()
         with start_active_span("get_user_by_req"):
             requester = await self._wrapped_get_user_by_req(
-                request, allow_guest, allow_expired, allow_locked
+                request, allow_guest, allow_expired
             )
 
             if parent_span:
@@ -108,7 +107,6 @@ class InternalAuth(BaseAuth):
         request: SynapseRequest,
         allow_guest: bool,
         allow_expired: bool,
-        allow_locked: bool,
     ) -> Requester:
         """Helper for get_user_by_req
 
@@ -128,17 +126,6 @@ class InternalAuth(BaseAuth):
                     access_token, allow_expired=allow_expired
                 )
 
-                # Deny the request if the user account is locked.
-                if not allow_locked and await self.store.get_user_locked_status(
-                    requester.user.to_string()
-                ):
-                    raise AuthError(
-                        401,
-                        "User account has been locked",
-                        errcode=Codes.USER_LOCKED,
-                        additional_fields={"soft_logout": True},
-                    )
-
                 # Deny the request if the user account has expired.
                 # This check is only done for regular users, not appservice ones.
                 if not allow_expired:

synapse/api/auth/msc3861_delegated.py

@@ -20,7 +20,6 @@ from authlib.oauth2.auth import encode_client_secret_basic, encode_client_secret
 from authlib.oauth2.rfc7523 import ClientSecretJWT, PrivateKeyJWT, private_key_jwt_sign
 from authlib.oauth2.rfc7662 import IntrospectionToken
 from authlib.oidc.discovery import OpenIDProviderMetadata, get_well_known_url
-from prometheus_client import Histogram
 
 from twisted.web.client import readBody
 from twisted.web.http_headers import Headers
@@ -28,7 +27,6 @@ from twisted.web.http_headers import Headers
 from synapse.api.auth.base import BaseAuth
 from synapse.api.errors import (
     AuthError,
-    Codes,
     HttpResponseException,
     InvalidClientTokenError,
     OAuthInsufficientScopeError,
@@ -40,20 +38,12 @@ from synapse.logging.context import make_deferred_yieldable
 from synapse.types import Requester, UserID, create_requester
 from synapse.util import json_decoder
 from synapse.util.caches.cached_call import RetryOnExceptionCachedCall
-from synapse.util.caches.expiringcache import ExpiringCache
 
 if TYPE_CHECKING:
     from synapse.server import HomeServer
 
 logger = logging.getLogger(__name__)
 
-introspection_response_timer = Histogram(
-    "synapse_api_auth_delegated_introspection_response",
-    "Time taken to get a response for an introspection request",
-    ["code"],
-)
-
-
 # Scope as defined by MSC2967
 # https://github.com/matrix-org/matrix-spec-proposals/pull/2967
 SCOPE_MATRIX_API = "urn:matrix:org.matrix.msc2967.client:api:*"
@@ -115,14 +105,6 @@ class MSC3861DelegatedAuth(BaseAuth):
 
         self._issuer_metadata = RetryOnExceptionCachedCall(self._load_metadata)
 
-        self._clock = hs.get_clock()
-        self._token_cache: ExpiringCache[str, IntrospectionToken] = ExpiringCache(
-            cache_name="introspection_token_cache",
-            clock=self._clock,
-            max_len=10000,
-            expiry_ms=5 * 60 * 1000,
-        )
-
         if isinstance(auth_method, PrivateKeyJWTWithKid):
             # Use the JWK as the client secret when using the private_key_jwt method
             assert self._config.jwk, "No JWK provided"
@@ -161,20 +143,6 @@ class MSC3861DelegatedAuth(BaseAuth):
         Returns:
             The introspection response
         """
-        # check the cache before doing a request
-        introspection_token = self._token_cache.get(token, None)
-
-        if introspection_token:
-            # check the expiration field of the token (if it exists)
-            exp = introspection_token.get("exp", None)
-            if exp:
-                time_now = self._clock.time()
-                expired = time_now > exp
-                if not expired:
-                    return introspection_token
-            else:
-                return introspection_token
-
         metadata = await self._issuer_metadata.get()
         introspection_endpoint = metadata.get("introspection_endpoint")
         raw_headers: Dict[str, str] = {
@@ -188,37 +156,22 @@ class MSC3861DelegatedAuth(BaseAuth):
 
         # Fill the body/headers with credentials
         uri, raw_headers, body = self._client_auth.prepare(
-            method="POST",
-            uri=introspection_endpoint,
-            headers=raw_headers,
-            body=body,
+            method="POST", uri=introspection_endpoint, headers=raw_headers, body=body
         )
         headers = Headers({k: [v] for (k, v) in raw_headers.items()})
 
         # Do the actual request
         # We're not using the SimpleHttpClient util methods as we don't want to
         # check the HTTP status code, and we do the body encoding ourselves.
-
-        start_time = self._clock.time()
-        try:
-            response = await self._http_client.request(
-                method="POST",
-                uri=uri,
-                data=body.encode("utf-8"),
-                headers=headers,
-            )
-
-            resp_body = await make_deferred_yieldable(readBody(response))
-        except Exception:
-            end_time = self._clock.time()
-            introspection_response_timer.labels("ERR").observe(end_time - start_time)
-            raise
-
-        end_time = self._clock.time()
-        introspection_response_timer.labels(response.code).observe(
-            end_time - start_time
+        response = await self._http_client.request(
+            method="POST",
+            uri=uri,
+            data=body.encode("utf-8"),
+            headers=headers,
         )
 
+        resp_body = await make_deferred_yieldable(readBody(response))
+
         if response.code < 200 or response.code >= 300:
             raise HttpResponseException(
                 response.code,
@@ -233,27 +186,16 @@ class MSC3861DelegatedAuth(BaseAuth):
                 "The introspection endpoint returned an invalid JSON response."
             )
 
-        expiration = resp.get("exp", None)
-        if expiration:
-            if self._clock.time() > expiration:
-                raise InvalidClientTokenError("Token is expired.")
-
-        introspection_token = IntrospectionToken(**resp)
-
-        # add token to cache
-        self._token_cache[token] = introspection_token
-
-        return introspection_token
+        return IntrospectionToken(**resp)
 
     async def is_server_admin(self, requester: Requester) -> bool:
-        return SCOPE_SYNAPSE_ADMIN in requester.scope
+        return "urn:synapse:admin:*" in requester.scope
 
     async def get_user_by_req(
         self,
         request: SynapseRequest,
         allow_guest: bool = False,
         allow_expired: bool = False,
-        allow_locked: bool = False,
     ) -> Requester:
         access_token = self.get_access_token_from_request(request)
 
@@ -263,36 +205,6 @@ class MSC3861DelegatedAuth(BaseAuth):
             # so that we don't provision the user if they don't have enough permission:
             requester = await self.get_user_by_access_token(access_token, allow_expired)
 
-            # Allow impersonation by an admin user using `_oidc_admin_impersonate_user_id` query parameter
-            if request.args is not None:
-                user_id_params = request.args.get(b"_oidc_admin_impersonate_user_id")
-                if user_id_params:
-                    if await self.is_server_admin(requester):
-                        user_id_str = user_id_params[0].decode("ascii")
-                        impersonated_user_id = UserID.from_string(user_id_str)
-                        logging.info(f"Admin impersonation of user {user_id_str}")
-                        requester = create_requester(
-                            user_id=impersonated_user_id,
-                            scope=[SCOPE_MATRIX_API],
-                            authenticated_entity=requester.user.to_string(),
-                        )
-                    else:
-                        raise AuthError(
-                            401,
-                            "Impersonation not possible by a non admin user",
-                        )
-
-            # Deny the request if the user account is locked.
-            if not allow_locked and await self.store.get_user_locked_status(
-                requester.user.to_string()
-            ):
-                raise AuthError(
-                    401,
-                    "User account has been locked",
-                    errcode=Codes.USER_LOCKED,
-                    additional_fields={"soft_logout": True},
-                )
-
         if not allow_guest and requester.is_guest:
             raise OAuthInsufficientScopeError([SCOPE_MATRIX_API])
 
@@ -309,14 +221,14 @@ class MSC3861DelegatedAuth(BaseAuth):
             # XXX: This is a temporary solution so that the admin API can be called by
             # the OIDC provider. This will be removed once we have OIDC client
             # credentials grant support in matrix-authentication-service.
-            logging.info("Admin token used")
+            logging.info("Admin toked used")
             # XXX: that user doesn't exist and won't be provisioned.
             # This is mostly fine for admin calls, but we should also think about doing
             # requesters without a user_id.
             admin_user = UserID("__oidc_admin", self._hostname)
             return create_requester(
                 user_id=admin_user,
-                scope=[SCOPE_SYNAPSE_ADMIN],
+                scope=["urn:synapse:admin:*"],
             )
 
         try:
@@ -438,16 +350,3 @@ class MSC3861DelegatedAuth(BaseAuth):
             scope=scope,
             is_guest=(has_guest_scope and not has_user_scope),
         )
-
-    def invalidate_cached_tokens(self, keys: List[str]) -> None:
-        """
-        Invalidate the entry(s) in the introspection token cache corresponding to the given key
-        """
-        for key in keys:
-            self._token_cache.invalidate(key)
-
-    def invalidate_token_cache(self) -> None:
-        """
-        Invalidate the entire token cache.
-        """
-        self._token_cache.invalidate_all()

synapse/api/constants.py

@@ -18,7 +18,8 @@
 """Contains constants from the specification."""
 
 import enum
-from typing import Final
+
+from typing_extensions import Final
 
 # the max size of a (canonical-json-encoded) event
 MAX_PDU_SIZE = 65536

synapse/api/errors.py

@@ -80,8 +80,6 @@ class Codes(str, Enum):
     WEAK_PASSWORD = "M_WEAK_PASSWORD"
     INVALID_SIGNATURE = "M_INVALID_SIGNATURE"
     USER_DEACTIVATED = "M_USER_DEACTIVATED"
-    # USER_LOCKED = "M_USER_LOCKED"
-    USER_LOCKED = "ORG_MATRIX_MSC3939_USER_LOCKED"
 
     # Part of MSC3848
     # https://github.com/matrix-org/matrix-spec-proposals/pull/3848

synapse/app/generic_worker.py

@@ -91,7 +91,6 @@ from synapse.storage.databases.main.state import StateGroupWorkerStore
 from synapse.storage.databases.main.stats import StatsStore
 from synapse.storage.databases.main.stream import StreamWorkerStore
 from synapse.storage.databases.main.tags import TagsWorkerStore
-from synapse.storage.databases.main.task_scheduler import TaskSchedulerWorkerStore
 from synapse.storage.databases.main.transactions import TransactionWorkerStore
 from synapse.storage.databases.main.ui_auth import UIAuthWorkerStore
 from synapse.storage.databases.main.user_directory import UserDirectoryStore
@@ -145,7 +144,6 @@ class GenericWorkerStore(
     TransactionWorkerStore,
     LockStore,
     SessionStore,
-    TaskSchedulerWorkerStore,
 ):
     # Properties that multiple storage classes define. Tell mypy what the
     # expected type is.

synapse/config/cas.py

@@ -47,10 +47,6 @@ class CasConfig(Config):
                 required_attributes
             )
 
-            self.idp_name = cas_config.get("idp_name", "CAS")
-            self.idp_icon = cas_config.get("idp_icon")
-            self.idp_brand = cas_config.get("idp_brand")
-
         else:
             self.cas_server_url = None
             self.cas_service_url = None

synapse/config/experimental.py

@@ -173,13 +173,6 @@ class MSC3861:
                 ("enable_registration",),
             )
 
-        # We only need to test the user consent version, as if it must be set if the user_consent section was present in the config
-        if root.consent.user_consent_version is not None:
-            raise ConfigError(
-                "User consent cannot be enabled when OAuth delegation is enabled",
-                ("user_consent",),
-            )
-
         if (
             root.oidc.oidc_enabled
             or root.saml2.saml2_enabled
@@ -223,12 +216,6 @@ class MSC3861:
                 ("session_lifetime",),
             )
 
-        if root.registration.enable_3pid_changes:
-            raise ConfigError(
-                "enable_3pid_changes cannot be enabled when OAuth delegation is enabled",
-                ("enable_3pid_changes",),
-            )
-
 
 @attr.s(auto_attribs=True, frozen=True, slots=True)
 class MSC3866Config:

synapse/config/oidc.py

@@ -280,20 +280,6 @@ def _parse_oidc_config_dict(
         for x in oidc_config.get("attribute_requirements", [])
     ]
 
-    # Read from either `client_secret_path` or `client_secret`. If both exist, error.
-    client_secret = oidc_config.get("client_secret")
-    client_secret_path = oidc_config.get("client_secret_path")
-    if client_secret_path is not None:
-        if client_secret is None:
-            client_secret = read_file(
-                client_secret_path, config_path + ("client_secret_path",)
-            ).rstrip("\n")
-        else:
-            raise ConfigError(
-                "Cannot specify both client_secret and client_secret_path",
-                config_path + ("client_secret",),
-            )
-
     return OidcProviderConfig(
         idp_id=idp_id,
         idp_name=oidc_config.get("idp_name", "OIDC"),
@@ -302,7 +288,7 @@ def _parse_oidc_config_dict(
         discover=oidc_config.get("discover", True),
         issuer=oidc_config["issuer"],
         client_id=oidc_config["client_id"],
-        client_secret=client_secret,
+        client_secret=oidc_config.get("client_secret"),
         client_secret_jwt_key=client_secret_jwt_key,
         client_auth_method=oidc_config.get("client_auth_method", "client_secret_basic"),
         pkce_method=oidc_config.get("pkce_method", "auto"),

synapse/config/registration.py

@@ -133,16 +133,7 @@ class RegistrationConfig(Config):
 
         self.enable_set_displayname = config.get("enable_set_displayname", True)
         self.enable_set_avatar_url = config.get("enable_set_avatar_url", True)
-
-        # The default value of enable_3pid_changes is True, unless msc3861 is enabled.
-        msc3861_enabled = (
-            (config.get("experimental_features") or {})
-            .get("msc3861", {})
-            .get("enabled", False)
-        )
-        self.enable_3pid_changes = config.get(
-            "enable_3pid_changes", not msc3861_enabled
-        )
+        self.enable_3pid_changes = config.get("enable_3pid_changes", True)
 
         self.disable_msisdn_registration = config.get(
             "disable_msisdn_registration", False

synapse/config/saml2.py

@@ -89,14 +89,8 @@ class SAML2Config(Config):
             "grandfathered_mxid_source_attribute", "uid"
         )
 
-        # refers to a SAML IdP entity ID
         self.saml2_idp_entityid = saml2_config.get("idp_entityid", None)
 
-        # IdP properties for Matrix clients
-        self.idp_name = saml2_config.get("idp_name", "SAML")
-        self.idp_icon = saml2_config.get("idp_icon")
-        self.idp_brand = saml2_config.get("idp_brand")
-
         # user_mapping_provider may be None if the key is present but has no value
         ump_dict = saml2_config.get("user_mapping_provider") or {}
 

synapse/config/user_directory.py

@@ -35,4 +35,3 @@ class UserDirectoryConfig(Config):
         self.user_directory_search_prefer_local_users = user_directory_config.get(
             "prefer_local_users", False
         )
-        self.show_locked_users = user_directory_config.get("show_locked_users", False)

synapse/events/snapshot.py

@@ -186,6 +186,9 @@ class EventContext(UnpersistedEventContextBase):
             ),
             "app_service_id": self.app_service.id if self.app_service else None,
             "partial_state": self.partial_state,
+            # add dummy delta_ids and prev_group for backwards compatibility
+            "delta_ids": None,
+            "prev_group": None,
         }
 
     @staticmethod
@@ -200,6 +203,13 @@ class EventContext(UnpersistedEventContextBase):
         Returns:
             The event context.
         """
+        # workaround for backwards/forwards compatibility: if the input doesn't have a value
+        # for "state_group_deltas" just assign an empty dict
+        state_group_deltas = input.get("state_group_deltas", None)
+        if state_group_deltas:
+            state_group_deltas = _decode_state_group_delta(state_group_deltas)
+        else:
+            state_group_deltas = {}
 
         context = EventContext(
             # We use the state_group and prev_state_id stuff to pull the
@@ -207,7 +217,7 @@ class EventContext(UnpersistedEventContextBase):
             storage=storage,
             state_group=input["state_group"],
             state_group_before_event=input["state_group_before_event"],
-            state_group_deltas=_decode_state_group_delta(input["state_group_deltas"]),
+            state_group_deltas=state_group_deltas,
             state_delta_due_to_event=_decode_state_dict(
                 input["state_delta_due_to_event"]
             ),

synapse/federation/federation_server.py

@@ -63,7 +63,7 @@ from synapse.federation.federation_base import (
 )
 from synapse.federation.persistence import TransactionActions
 from synapse.federation.units import Edu, Transaction
-from synapse.handlers.worker_lock import NEW_EVENT_DURING_PURGE_LOCK_NAME
+from synapse.handlers.worker_lock import DELETE_ROOM_LOCK_NAME
 from synapse.http.servlet import assert_params_in_dict
 from synapse.logging.context import (
     make_deferred_yieldable,
@@ -1245,7 +1245,7 @@ class FederationServer(FederationBase):
                         # while holding the `_INBOUND_EVENT_HANDLING_LOCK_NAME`
                         # lock.
                         async with self._worker_lock_handler.acquire_read_write_lock(
-                            NEW_EVENT_DURING_PURGE_LOCK_NAME, room_id, write=False
+                            DELETE_ROOM_LOCK_NAME, room_id, write=False
                         ):
                             await self._federation_event_handler.on_receive_pdu(
                                 origin, event

synapse/handlers/admin.py

@@ -67,7 +67,6 @@ class AdminHandler:
             "name",
             "admin",
             "deactivated",
-            "locked",
             "shadow_banned",
             "creation_ts",
             "appservice_id",

synapse/handlers/cas.py

@@ -76,13 +76,12 @@ class CasHandler:
         self.idp_id = "cas"
 
         # user-facing name of this auth provider
-        self.idp_name = hs.config.cas.idp_name
+        self.idp_name = "CAS"
 
-        # MXC URI for icon for this auth provider
-        self.idp_icon = hs.config.cas.idp_icon
-
-        # optional brand identifier for this auth provider
-        self.idp_brand = hs.config.cas.idp_brand
+        # we do not currently support brands/icons for CAS auth, but this is required by
+        # the SsoIdentityProvider protocol type.
+        self.idp_icon = None
+        self.idp_brand = None
 
         self._sso_handler = hs.get_sso_handler()
 

synapse/handlers/device.py

@@ -385,7 +385,6 @@ class DeviceHandler(DeviceWorkerHandler):
         self.federation_sender = hs.get_federation_sender()
         self._account_data_handler = hs.get_account_data_handler()
         self._storage_controllers = hs.get_storage_controllers()
-        self.db_pool = hs.get_datastores().main.db_pool
 
         self.device_list_updater = DeviceListUpdater(hs, self)
 
@@ -657,17 +656,15 @@ class DeviceHandler(DeviceWorkerHandler):
         device_id: Optional[str],
         device_data: JsonDict,
         initial_device_display_name: Optional[str] = None,
-        keys_for_device: Optional[JsonDict] = None,
     ) -> str:
-        """Store a dehydrated device for a user, optionally storing the keys associated with
-        it as well.  If the user had a previous dehydrated device, it is removed.
+        """Store a dehydrated device for a user.  If the user had a previous
+        dehydrated device, it is removed.
 
         Args:
             user_id: the user that we are storing the device for
             device_id: device id supplied by client
             device_data: the dehydrated device information
             initial_device_display_name: The display name to use for the device
-            keys_for_device: keys for the dehydrated device
         Returns:
             device id of the dehydrated device
         """
@@ -676,16 +673,11 @@ class DeviceHandler(DeviceWorkerHandler):
             device_id,
             initial_device_display_name,
         )
-
-        time_now = self.clock.time_msec()
-
         old_device_id = await self.store.store_dehydrated_device(
-            user_id, device_id, device_data, time_now, keys_for_device
+            user_id, device_id, device_data
         )
-
         if old_device_id is not None:
             await self.delete_devices(user_id, [old_device_id])
-
         return device_id
 
     async def rehydrate_device(
@@ -730,22 +722,6 @@ class DeviceHandler(DeviceWorkerHandler):
 
         return {"success": True}
 
-    async def delete_dehydrated_device(self, user_id: str, device_id: str) -> None:
-        """
-        Delete a stored dehydrated device.
-
-        Args:
-            user_id: the user_id to delete the device from
-            device_id: id of the dehydrated device to delete
-        """
-        success = await self.store.remove_dehydrated_device(user_id, device_id)
-
-        if not success:
-            raise errors.NotFoundError()
-
-        await self.delete_devices(user_id, [device_id])
-        await self.store.delete_e2e_keys_by_device(user_id=user_id, device_id=device_id)
-
     @wrap_as_background_process("_handle_new_device_update_async")
     async def _handle_new_device_update_async(self) -> None:
         """Called when we have a new local device list update that we need to

synapse/handlers/devicemessage.py

@@ -367,6 +367,19 @@ class DeviceMessageHandler:
                     errcode=Codes.INVALID_PARAM,
                 )
 
+            # if we have a since token, delete any to-device messages before that token
+            # (since we now know that the device has received them)
+            deleted = await self.store.delete_messages_for_device(
+                user_id, device_id, since_stream_id
+            )
+            logger.debug(
+                "Deleted %d to-device messages up to %d for user_id %s device_id %s",
+                deleted,
+                since_stream_id,
+                user_id,
+                device_id,
+            )
+
         to_token = self.event_sources.get_current_token().to_device_key
 
         messages, stream_id = await self.store.get_messages_for_device(

synapse/handlers/federation.py

@@ -60,7 +60,6 @@ from synapse.events import EventBase
 from synapse.events.snapshot import EventContext, UnpersistedEventContextBase
 from synapse.events.validator import EventValidator
 from synapse.federation.federation_client import InvalidResponseError
-from synapse.handlers.pagination import PURGE_PAGINATION_LOCK_NAME
 from synapse.http.servlet import assert_params_in_dict
 from synapse.logging.context import nested_logging_context
 from synapse.logging.opentracing import SynapseTags, set_tag, tag_args, trace
@@ -153,7 +152,6 @@ class FederationHandler:
         self._device_handler = hs.get_device_handler()
         self._bulk_push_rule_evaluator = hs.get_bulk_push_rule_evaluator()
         self._notifier = hs.get_notifier()
-        self._worker_locks = hs.get_worker_locks_handler()
 
         self._clean_room_for_join_client = ReplicationCleanRoomRestServlet.make_client(
             hs
@@ -202,7 +200,7 @@ class FederationHandler:
     @trace
     @tag_args
     async def maybe_backfill(
-        self, room_id: str, current_depth: int, limit: int, record_time: bool = True
+        self, room_id: str, current_depth: int, limit: int
     ) -> bool:
         """Checks the database to see if we should backfill before paginating,
         and if so do.
@@ -215,25 +213,21 @@ class FederationHandler:
             limit: The number of events that the pagination request will
                 return. This is used as part of the heuristic to decide if we
                 should back paginate.
-            record_time: Whether to record the time it takes to backfill.
 
         Returns:
             True if we actually tried to backfill something, otherwise False.
         """
         # Starting the processing time here so we can include the room backfill
         # linearizer lock queue in the timing
-        processing_start_time = self.clock.time_msec() if record_time else 0
+        processing_start_time = self.clock.time_msec()
 
         async with self._room_backfill.queue(room_id):
-            async with self._worker_locks.acquire_read_write_lock(
-                PURGE_PAGINATION_LOCK_NAME, room_id, write=False
-            ):
-                return await self._maybe_backfill_inner(
-                    room_id,
-                    current_depth,
-                    limit,
-                    processing_start_time=processing_start_time,
-                )
+            return await self._maybe_backfill_inner(
+                room_id,
+                current_depth,
+                limit,
+                processing_start_time=processing_start_time,
+            )
 
     @trace
     @tag_args
@@ -311,21 +305,12 @@ class FederationHandler:
         # of history that extends all the way back to where we are currently paginating
         # and it's within the 100 events that are returned from `/backfill`.
         if not sorted_backfill_points and current_depth != MAX_DEPTH:
-            # Check that we actually have later backfill points, if not just return.
-            have_later_backfill_points = await self.store.get_backfill_points_in_room(
-                room_id=room_id,
-                current_depth=MAX_DEPTH,
-                limit=1,
-            )
-            if not have_later_backfill_points:
-                return False
-
             logger.debug(
                 "_maybe_backfill_inner: all backfill points are *after* current depth. Trying again with later backfill points."
             )
             run_as_background_process(
                 "_maybe_backfill_inner_anyway_with_max_depth",
-                self.maybe_backfill,
+                self._maybe_backfill_inner,
                 room_id=room_id,
                 # We use `MAX_DEPTH` so that we find all backfill points next
                 # time (all events are below the `MAX_DEPTH`)
@@ -334,7 +319,7 @@ class FederationHandler:
                 # We don't want to start another timing observation from this
                 # nested recursive call. The top-most call can record the time
                 # overall otherwise the smaller one will throw off the results.
-                record_time=False,
+                processing_start_time=None,
             )
             # We return `False` because we're backfilling in the background and there is
             # no new events immediately for the caller to know about yet.

synapse/handlers/message.py

@@ -53,7 +53,7 @@ from synapse.events.snapshot import EventContext, UnpersistedEventContextBase
 from synapse.events.utils import SerializeEventConfig, maybe_upsert_event_field
 from synapse.events.validator import EventValidator
 from synapse.handlers.directory import DirectoryHandler
-from synapse.handlers.worker_lock import NEW_EVENT_DURING_PURGE_LOCK_NAME
+from synapse.handlers.worker_lock import DELETE_ROOM_LOCK_NAME
 from synapse.logging import opentracing
 from synapse.logging.context import make_deferred_yieldable, run_in_background
 from synapse.metrics.background_process_metrics import run_as_background_process
@@ -1034,7 +1034,7 @@ class EventCreationHandler:
                     )
 
         async with self._worker_lock_handler.acquire_read_write_lock(
-            NEW_EVENT_DURING_PURGE_LOCK_NAME, room_id, write=False
+            DELETE_ROOM_LOCK_NAME, room_id, write=False
         ):
             return await self._create_and_send_nonmember_event_locked(
                 requester=requester,
@@ -1978,7 +1978,7 @@ class EventCreationHandler:
 
         for room_id in room_ids:
             async with self._worker_lock_handler.acquire_read_write_lock(
-                NEW_EVENT_DURING_PURGE_LOCK_NAME, room_id, write=False
+                DELETE_ROOM_LOCK_NAME, room_id, write=False
             ):
                 dummy_event_sent = await self._send_dummy_event_for_room(room_id)
 

synapse/handlers/pagination.py

@@ -24,7 +24,6 @@ from synapse.api.errors import SynapseError
 from synapse.api.filtering import Filter
 from synapse.events.utils import SerializeEventConfig
 from synapse.handlers.room import ShutdownRoomResponse
-from synapse.handlers.worker_lock import NEW_EVENT_DURING_PURGE_LOCK_NAME
 from synapse.logging.opentracing import trace
 from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.rest.admin._base import assert_user_is_admin
@@ -47,10 +46,9 @@ logger = logging.getLogger(__name__)
 BACKFILL_BECAUSE_TOO_MANY_GAPS_THRESHOLD = 3
 
 
-# This is used to avoid purging a room several time at the same moment,
-# and also paginating during a purge. Pagination can trigger backfill,
-# which would create old events locally, and would potentially clash with the room delete.
-PURGE_PAGINATION_LOCK_NAME = "purge_pagination_lock"
+PURGE_HISTORY_LOCK_NAME = "purge_history_lock"
+
+DELETE_ROOM_LOCK_NAME = "delete_room_lock"
 
 
 @attr.s(slots=True, auto_attribs=True)
@@ -365,7 +363,7 @@ class PaginationHandler:
         self._purges_in_progress_by_room.add(room_id)
         try:
             async with self._worker_locks.acquire_read_write_lock(
-                PURGE_PAGINATION_LOCK_NAME, room_id, write=True
+                PURGE_HISTORY_LOCK_NAME, room_id, write=True
             ):
                 await self._storage_controllers.purge_events.purge_history(
                     room_id, token, delete_local_events
@@ -423,10 +421,7 @@ class PaginationHandler:
             force: set true to skip checking for joined users.
         """
         async with self._worker_locks.acquire_multi_read_write_lock(
-            [
-                (PURGE_PAGINATION_LOCK_NAME, room_id),
-                (NEW_EVENT_DURING_PURGE_LOCK_NAME, room_id),
-            ],
+            [(PURGE_HISTORY_LOCK_NAME, room_id), (DELETE_ROOM_LOCK_NAME, room_id)],
             write=True,
         ):
             # first check that we have no users in this room
@@ -487,150 +482,155 @@ class PaginationHandler:
 
         room_token = from_token.room_key
 
-        (membership, member_event_id) = (None, None)
-        if not use_admin_priviledge:
-            (
-                membership,
-                member_event_id,
-            ) = await self.auth.check_user_in_room_or_world_readable(
-                room_id, requester, allow_departed_users=True
-            )
-
-        if pagin_config.direction == Direction.BACKWARDS:
-            # if we're going backwards, we might need to backfill. This
-            # requires that we have a topo token.
-            if room_token.topological:
-                curr_topo = room_token.topological
-            else:
-                curr_topo = await self.store.get_current_topological_token(
-                    room_id, room_token.stream
-                )
-
-        # If they have left the room then clamp the token to be before
-        # they left the room, to save the effort of loading from the
-        # database.
-        if (
-            pagin_config.direction == Direction.BACKWARDS
-            and not use_admin_priviledge
-            and membership == Membership.LEAVE
+        async with self._worker_locks.acquire_read_write_lock(
+            PURGE_HISTORY_LOCK_NAME, room_id, write=False
         ):
-            # This is only None if the room is world_readable, in which case
-            # "Membership.JOIN" would have been returned and we should never hit
-            # this branch.
-            assert member_event_id
-
-            leave_token = await self.store.get_topological_token_for_event(
-                member_event_id
-            )
-            assert leave_token.topological is not None
-
-            if leave_token.topological < curr_topo:
-                from_token = from_token.copy_and_replace(
-                    StreamKeyType.ROOM, leave_token
+            (membership, member_event_id) = (None, None)
+            if not use_admin_priviledge:
+                (
+                    membership,
+                    member_event_id,
+                ) = await self.auth.check_user_in_room_or_world_readable(
+                    room_id, requester, allow_departed_users=True
                 )
 
-        to_room_key = None
-        if pagin_config.to_token:
-            to_room_key = pagin_config.to_token.room_key
-
-        # Initially fetch the events from the database. With any luck, we can return
-        # these without blocking on backfill (handled below).
-        events, next_key = await self.store.paginate_room_events(
-            room_id=room_id,
-            from_key=from_token.room_key,
-            to_key=to_room_key,
-            direction=pagin_config.direction,
-            limit=pagin_config.limit,
-            event_filter=event_filter,
-        )
-
-        if pagin_config.direction == Direction.BACKWARDS:
-            # We use a `Set` because there can be multiple events at a given depth
-            # and we only care about looking at the unique continum of depths to
-            # find gaps.
-            event_depths: Set[int] = {event.depth for event in events}
-            sorted_event_depths = sorted(event_depths)
-
-            # Inspect the depths of the returned events to see if there are any gaps
-            found_big_gap = False
-            number_of_gaps = 0
-            previous_event_depth = (
-                sorted_event_depths[0] if len(sorted_event_depths) > 0 else 0
-            )
-            for event_depth in sorted_event_depths:
-                # We don't expect a negative depth but we'll just deal with it in
-                # any case by taking the absolute value to get the true gap between
-                # any two integers.
-                depth_gap = abs(event_depth - previous_event_depth)
-                # A `depth_gap` of 1 is a normal continuous chain to the next event
-                # (1 <-- 2 <-- 3) so anything larger indicates a missing event (it's
-                # also possible there is no event at a given depth but we can't ever
-                # know that for sure)
-                if depth_gap > 1:
-                    number_of_gaps += 1
-
-                # We only tolerate a small number single-event long gaps in the
-                # returned events because those are most likely just events we've
-                # failed to pull in the past. Anything longer than that is probably
-                # a sign that we're missing a decent chunk of history and we should
-                # try to backfill it.
-                #
-                # XXX: It's possible we could tolerate longer gaps if we checked
-                # that a given events `prev_events` is one that has failed pull
-                # attempts and we could just treat it like a dead branch of history
-                # for now or at least something that we don't need the block the
-                # client on to try pulling.
-                #
-                # XXX: If we had something like MSC3871 to indicate gaps in the
-                # timeline to the client, we could also get away with any sized gap
-                # and just have the client refetch the holes as they see fit.
-                if depth_gap > 2:
-                    found_big_gap = True
-                    break
-                previous_event_depth = event_depth
-
-            # Backfill in the foreground if we found a big gap, have too many holes,
-            # or we don't have enough events to fill the limit that the client asked
-            # for.
-            missing_too_many_events = (
-                number_of_gaps > BACKFILL_BECAUSE_TOO_MANY_GAPS_THRESHOLD
-            )
-            not_enough_events_to_fill_response = len(events) < pagin_config.limit
-            if (
-                found_big_gap
-                or missing_too_many_events
-                or not_enough_events_to_fill_response
-            ):
-                did_backfill = await self.hs.get_federation_handler().maybe_backfill(
-                    room_id,
-                    curr_topo,
-                    limit=pagin_config.limit,
-                )
-
-                # If we did backfill something, refetch the events from the database to
-                # catch anything new that might have been added since we last fetched.
-                if did_backfill:
-                    events, next_key = await self.store.paginate_room_events(
-                        room_id=room_id,
-                        from_key=from_token.room_key,
-                        to_key=to_room_key,
-                        direction=pagin_config.direction,
-                        limit=pagin_config.limit,
-                        event_filter=event_filter,
+            if pagin_config.direction == Direction.BACKWARDS:
+                # if we're going backwards, we might need to backfill. This
+                # requires that we have a topo token.
+                if room_token.topological:
+                    curr_topo = room_token.topological
+                else:
+                    curr_topo = await self.store.get_current_topological_token(
+                        room_id, room_token.stream
                     )
-            else:
-                # Otherwise, we can backfill in the background for eventual
-                # consistency's sake but we don't need to block the client waiting
-                # for a costly federation call and processing.
-                run_as_background_process(
-                    "maybe_backfill_in_the_background",
-                    self.hs.get_federation_handler().maybe_backfill,
-                    room_id,
-                    curr_topo,
-                    limit=pagin_config.limit,
-                )
 
-        next_token = from_token.copy_and_replace(StreamKeyType.ROOM, next_key)
+            # If they have left the room then clamp the token to be before
+            # they left the room, to save the effort of loading from the
+            # database.
+            if (
+                pagin_config.direction == Direction.BACKWARDS
+                and not use_admin_priviledge
+                and membership == Membership.LEAVE
+            ):
+                # This is only None if the room is world_readable, in which case
+                # "Membership.JOIN" would have been returned and we should never hit
+                # this branch.
+                assert member_event_id
+
+                leave_token = await self.store.get_topological_token_for_event(
+                    member_event_id
+                )
+                assert leave_token.topological is not None
+
+                if leave_token.topological < curr_topo:
+                    from_token = from_token.copy_and_replace(
+                        StreamKeyType.ROOM, leave_token
+                    )
+
+            to_room_key = None
+            if pagin_config.to_token:
+                to_room_key = pagin_config.to_token.room_key
+
+            # Initially fetch the events from the database. With any luck, we can return
+            # these without blocking on backfill (handled below).
+            events, next_key = await self.store.paginate_room_events(
+                room_id=room_id,
+                from_key=from_token.room_key,
+                to_key=to_room_key,
+                direction=pagin_config.direction,
+                limit=pagin_config.limit,
+                event_filter=event_filter,
+            )
+
+            if pagin_config.direction == Direction.BACKWARDS:
+                # We use a `Set` because there can be multiple events at a given depth
+                # and we only care about looking at the unique continum of depths to
+                # find gaps.
+                event_depths: Set[int] = {event.depth for event in events}
+                sorted_event_depths = sorted(event_depths)
+
+                # Inspect the depths of the returned events to see if there are any gaps
+                found_big_gap = False
+                number_of_gaps = 0
+                previous_event_depth = (
+                    sorted_event_depths[0] if len(sorted_event_depths) > 0 else 0
+                )
+                for event_depth in sorted_event_depths:
+                    # We don't expect a negative depth but we'll just deal with it in
+                    # any case by taking the absolute value to get the true gap between
+                    # any two integers.
+                    depth_gap = abs(event_depth - previous_event_depth)
+                    # A `depth_gap` of 1 is a normal continuous chain to the next event
+                    # (1 <-- 2 <-- 3) so anything larger indicates a missing event (it's
+                    # also possible there is no event at a given depth but we can't ever
+                    # know that for sure)
+                    if depth_gap > 1:
+                        number_of_gaps += 1
+
+                    # We only tolerate a small number single-event long gaps in the
+                    # returned events because those are most likely just events we've
+                    # failed to pull in the past. Anything longer than that is probably
+                    # a sign that we're missing a decent chunk of history and we should
+                    # try to backfill it.
+                    #
+                    # XXX: It's possible we could tolerate longer gaps if we checked
+                    # that a given events `prev_events` is one that has failed pull
+                    # attempts and we could just treat it like a dead branch of history
+                    # for now or at least something that we don't need the block the
+                    # client on to try pulling.
+                    #
+                    # XXX: If we had something like MSC3871 to indicate gaps in the
+                    # timeline to the client, we could also get away with any sized gap
+                    # and just have the client refetch the holes as they see fit.
+                    if depth_gap > 2:
+                        found_big_gap = True
+                        break
+                    previous_event_depth = event_depth
+
+                # Backfill in the foreground if we found a big gap, have too many holes,
+                # or we don't have enough events to fill the limit that the client asked
+                # for.
+                missing_too_many_events = (
+                    number_of_gaps > BACKFILL_BECAUSE_TOO_MANY_GAPS_THRESHOLD
+                )
+                not_enough_events_to_fill_response = len(events) < pagin_config.limit
+                if (
+                    found_big_gap
+                    or missing_too_many_events
+                    or not_enough_events_to_fill_response
+                ):
+                    did_backfill = (
+                        await self.hs.get_federation_handler().maybe_backfill(
+                            room_id,
+                            curr_topo,
+                            limit=pagin_config.limit,
+                        )
+                    )
+
+                    # If we did backfill something, refetch the events from the database to
+                    # catch anything new that might have been added since we last fetched.
+                    if did_backfill:
+                        events, next_key = await self.store.paginate_room_events(
+                            room_id=room_id,
+                            from_key=from_token.room_key,
+                            to_key=to_room_key,
+                            direction=pagin_config.direction,
+                            limit=pagin_config.limit,
+                            event_filter=event_filter,
+                        )
+                else:
+                    # Otherwise, we can backfill in the background for eventual
+                    # consistency's sake but we don't need to block the client waiting
+                    # for a costly federation call and processing.
+                    run_as_background_process(
+                        "maybe_backfill_in_the_background",
+                        self.hs.get_federation_handler().maybe_backfill,
+                        room_id,
+                        curr_topo,
+                        limit=pagin_config.limit,
+                    )
+
+            next_token = from_token.copy_and_replace(StreamKeyType.ROOM, next_key)
 
         # if no events are returned from pagination, that implies
         # we have reached the end of the available events.
@@ -761,7 +761,7 @@ class PaginationHandler:
         self._purges_in_progress_by_room.add(room_id)
         try:
             async with self._worker_locks.acquire_read_write_lock(
-                PURGE_PAGINATION_LOCK_NAME, room_id, write=True
+                PURGE_HISTORY_LOCK_NAME, room_id, write=True
             ):
                 self._delete_by_id[delete_id].status = DeleteStatus.STATUS_SHUTTING_DOWN
                 self._delete_by_id[

synapse/handlers/presence.py

@@ -30,9 +30,9 @@ from types import TracebackType
 from typing import (
     TYPE_CHECKING,
     Any,
+    Awaitable,
     Callable,
     Collection,
-    ContextManager,
     Dict,
     Generator,
     Iterable,
@@ -44,6 +44,7 @@ from typing import (
 )
 
 from prometheus_client import Counter
+from typing_extensions import ContextManager
 
 import synapse.metrics
 from synapse.api.constants import EduTypes, EventTypes, Membership, PresenceState
@@ -53,10 +54,7 @@ from synapse.appservice import ApplicationService
 from synapse.events.presence_router import PresenceRouter
 from synapse.logging.context import run_in_background
 from synapse.metrics import LaterGauge
-from synapse.metrics.background_process_metrics import (
-    run_as_background_process,
-    wrap_as_background_process,
-)
+from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.replication.http.presence import (
     ReplicationBumpPresenceActiveTime,
     ReplicationPresenceSetState,
@@ -143,8 +141,6 @@ class BasePresenceHandler(abc.ABC):
         self.state = hs.get_state_handler()
         self.is_mine_id = hs.is_mine_id
 
-        self._presence_enabled = hs.config.server.use_presence
-
         self._federation = None
         if hs.should_send_federation():
             self._federation = hs.get_federation_sender()
@@ -153,15 +149,6 @@ class BasePresenceHandler(abc.ABC):
 
         self._busy_presence_enabled = hs.config.experimental.msc3026_enabled
 
-        self.VALID_PRESENCE: Tuple[str, ...] = (
-            PresenceState.ONLINE,
-            PresenceState.UNAVAILABLE,
-            PresenceState.OFFLINE,
-        )
-
-        if self._busy_presence_enabled:
-            self.VALID_PRESENCE += (PresenceState.BUSY,)
-
         active_presence = self.store.take_presence_startup_info()
         self.user_to_current_state = {state.user_id: state for state in active_presence}
 
@@ -408,6 +395,8 @@ class WorkerPresenceHandler(BasePresenceHandler):
 
         self._presence_writer_instance = hs.config.worker.writers.presence[0]
 
+        self._presence_enabled = hs.config.server.use_presence
+
         # Route presence EDUs to the right worker
         hs.get_federation_registry().register_instances_for_edu(
             EduTypes.PRESENCE,
@@ -432,6 +421,8 @@ class WorkerPresenceHandler(BasePresenceHandler):
             self.send_stop_syncing, UPDATE_SYNCING_USERS_MS
         )
 
+        self._busy_presence_enabled = hs.config.experimental.msc3026_enabled
+
         hs.get_reactor().addSystemEventTrigger(
             "before",
             "shutdown",
@@ -499,9 +490,7 @@ class WorkerPresenceHandler(BasePresenceHandler):
             # what the spec wants: see comment in the BasePresenceHandler version
             # of this function.
             await self.set_state(
-                UserID.from_string(user_id),
-                {"presence": presence_state},
-                ignore_status_msg=True,
+                UserID.from_string(user_id), {"presence": presence_state}, True
             )
 
         curr_sync = self._user_to_num_current_syncs.get(user_id, 0)
@@ -612,13 +601,22 @@ class WorkerPresenceHandler(BasePresenceHandler):
         """
         presence = state["presence"]
 
-        if presence not in self.VALID_PRESENCE:
+        valid_presence = (
+            PresenceState.ONLINE,
+            PresenceState.UNAVAILABLE,
+            PresenceState.OFFLINE,
+            PresenceState.BUSY,
+        )
+
+        if presence not in valid_presence or (
+            presence == PresenceState.BUSY and not self._busy_presence_enabled
+        ):
             raise SynapseError(400, "Invalid presence state")
 
         user_id = target_user.to_string()
 
         # If presence is disabled, no-op
-        if not self._presence_enabled:
+        if not self.hs.config.server.use_presence:
             return
 
         # Proxy request to instance that writes presence
@@ -635,7 +633,7 @@ class WorkerPresenceHandler(BasePresenceHandler):
         with the app.
         """
         # If presence is disabled, no-op
-        if not self._presence_enabled:
+        if not self.hs.config.server.use_presence:
             return
 
         # Proxy request to instance that writes presence
@@ -651,6 +649,7 @@ class PresenceHandler(BasePresenceHandler):
         self.hs = hs
         self.wheel_timer: WheelTimer[str] = WheelTimer()
         self.notifier = hs.get_notifier()
+        self._presence_enabled = hs.config.server.use_presence
 
         federation_registry = hs.get_federation_registry()
 
@@ -701,6 +700,8 @@ class PresenceHandler(BasePresenceHandler):
             self._on_shutdown,
         )
 
+        self._next_serial = 1
+
         # Keeps track of the number of *ongoing* syncs on this process. While
         # this is non zero a user will never go offline.
         self.user_to_num_current_syncs: Dict[str, int] = {}
@@ -722,17 +723,22 @@ class PresenceHandler(BasePresenceHandler):
             # Start a LoopingCall in 30s that fires every 5s.
             # The initial delay is to allow disconnected clients a chance to
             # reconnect before we treat them as offline.
-            self.clock.call_later(
-                30, self.clock.looping_call, self._handle_timeouts, 5000
-            )
+            def run_timeout_handler() -> Awaitable[None]:
+                return run_as_background_process(
+                    "handle_presence_timeouts", self._handle_timeouts
+                )
 
             self.clock.call_later(
-                60,
-                self.clock.looping_call,
-                self._persist_unpersisted_changes,
-                60 * 1000,
+                30, self.clock.looping_call, run_timeout_handler, 5000
             )
 
+            def run_persister() -> Awaitable[None]:
+                return run_as_background_process(
+                    "persist_presence_changes", self._persist_unpersisted_changes
+                )
+
+            self.clock.call_later(60, self.clock.looping_call, run_persister, 60 * 1000)
+
         LaterGauge(
             "synapse_handlers_presence_wheel_timer_size",
             "",
@@ -777,7 +783,6 @@ class PresenceHandler(BasePresenceHandler):
             )
         logger.info("Finished _on_shutdown")
 
-    @wrap_as_background_process("persist_presence_changes")
     async def _persist_unpersisted_changes(self) -> None:
         """We periodically persist the unpersisted changes, as otherwise they
         may stack up and slow down shutdown times.
@@ -893,7 +898,6 @@ class PresenceHandler(BasePresenceHandler):
                         states, [destination]
                     )
 
-    @wrap_as_background_process("handle_presence_timeouts")
     async def _handle_timeouts(self) -> None:
         """Checks the presence of users that have timed out and updates as
         appropriate.
@@ -951,7 +955,7 @@ class PresenceHandler(BasePresenceHandler):
         with the app.
         """
         # If presence is disabled, no-op
-        if not self._presence_enabled:
+        if not self.hs.config.server.use_presence:
             return
 
         user_id = user.to_string()
@@ -986,51 +990,56 @@ class PresenceHandler(BasePresenceHandler):
                 client that is being used by a user.
             presence_state: The presence state indicated in the sync request
         """
-        if not affect_presence or not self._presence_enabled:
-            return _NullContextManager()
+        # Override if it should affect the user's presence, if presence is
+        # disabled.
+        if not self.hs.config.server.use_presence:
+            affect_presence = False
 
-        curr_sync = self.user_to_num_current_syncs.get(user_id, 0)
-        self.user_to_num_current_syncs[user_id] = curr_sync + 1
+        if affect_presence:
+            curr_sync = self.user_to_num_current_syncs.get(user_id, 0)
+            self.user_to_num_current_syncs[user_id] = curr_sync + 1
 
-        prev_state = await self.current_state_for_user(user_id)
-
-        # If they're busy then they don't stop being busy just by syncing,
-        # so just update the last sync time.
-        if prev_state.state != PresenceState.BUSY:
-            # XXX: We set_state separately here and just update the last_active_ts above
-            # This keeps the logic as similar as possible between the worker and single
-            # process modes. Using set_state will actually cause last_active_ts to be
-            # updated always, which is not what the spec calls for, but synapse has done
-            # this for... forever, I think.
-            await self.set_state(
-                UserID.from_string(user_id),
-                {"presence": presence_state},
-                ignore_status_msg=True,
-            )
-            # Retrieve the new state for the logic below. This should come from the
-            # in-memory cache.
             prev_state = await self.current_state_for_user(user_id)
 
-        # To keep the single process behaviour consistent with worker mode, run the
-        # same logic as `update_external_syncs_row`, even though it looks weird.
-        if prev_state.state == PresenceState.OFFLINE:
-            await self._update_states(
-                [
-                    prev_state.copy_and_replace(
-                        state=PresenceState.ONLINE,
-                        last_active_ts=self.clock.time_msec(),
-                        last_user_sync_ts=self.clock.time_msec(),
-                    )
-                ]
-            )
-        # otherwise, set the new presence state & update the last sync time,
-        # but don't update last_active_ts as this isn't an indication that
-        # they've been active (even though it's probably been updated by
-        # set_state above)
-        else:
-            await self._update_states(
-                [prev_state.copy_and_replace(last_user_sync_ts=self.clock.time_msec())]
-            )
+            # If they're busy then they don't stop being busy just by syncing,
+            # so just update the last sync time.
+            if prev_state.state != PresenceState.BUSY:
+                # XXX: We set_state separately here and just update the last_active_ts above
+                # This keeps the logic as similar as possible between the worker and single
+                # process modes. Using set_state will actually cause last_active_ts to be
+                # updated always, which is not what the spec calls for, but synapse has done
+                # this for... forever, I think.
+                await self.set_state(
+                    UserID.from_string(user_id), {"presence": presence_state}, True
+                )
+                # Retrieve the new state for the logic below. This should come from the
+                # in-memory cache.
+                prev_state = await self.current_state_for_user(user_id)
+
+            # To keep the single process behaviour consistent with worker mode, run the
+            # same logic as `update_external_syncs_row`, even though it looks weird.
+            if prev_state.state == PresenceState.OFFLINE:
+                await self._update_states(
+                    [
+                        prev_state.copy_and_replace(
+                            state=PresenceState.ONLINE,
+                            last_active_ts=self.clock.time_msec(),
+                            last_user_sync_ts=self.clock.time_msec(),
+                        )
+                    ]
+                )
+            # otherwise, set the new presence state & update the last sync time,
+            # but don't update last_active_ts as this isn't an indication that
+            # they've been active (even though it's probably been updated by
+            # set_state above)
+            else:
+                await self._update_states(
+                    [
+                        prev_state.copy_and_replace(
+                            last_user_sync_ts=self.clock.time_msec()
+                        )
+                    ]
+                )
 
         async def _end() -> None:
             try:
@@ -1052,7 +1061,8 @@ class PresenceHandler(BasePresenceHandler):
             try:
                 yield
             finally:
-                run_in_background(_end)
+                if affect_presence:
+                    run_in_background(_end)
 
         return _user_syncing()
 
@@ -1219,11 +1229,20 @@ class PresenceHandler(BasePresenceHandler):
         status_msg = state.get("status_msg", None)
         presence = state["presence"]
 
-        if presence not in self.VALID_PRESENCE:
+        valid_presence = (
+            PresenceState.ONLINE,
+            PresenceState.UNAVAILABLE,
+            PresenceState.OFFLINE,
+            PresenceState.BUSY,
+        )
+
+        if presence not in valid_presence or (
+            presence == PresenceState.BUSY and not self._busy_presence_enabled
+        ):
             raise SynapseError(400, "Invalid presence state")
 
         # If presence is disabled, no-op
-        if not self._presence_enabled:
+        if not self.hs.config.server.use_presence:
             return
 
         user_id = target_user.to_string()

synapse/handlers/room_member.py

@@ -39,7 +39,7 @@ from synapse.events import EventBase
 from synapse.events.snapshot import EventContext
 from synapse.handlers.profile import MAX_AVATAR_URL_LEN, MAX_DISPLAYNAME_LEN
 from synapse.handlers.state_deltas import MatchChange, StateDeltasHandler
-from synapse.handlers.worker_lock import NEW_EVENT_DURING_PURGE_LOCK_NAME
+from synapse.handlers.worker_lock import DELETE_ROOM_LOCK_NAME
 from synapse.logging import opentracing
 from synapse.metrics import event_processing_positions
 from synapse.metrics.background_process_metrics import run_as_background_process
@@ -621,7 +621,7 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):
         async with self.member_as_limiter.queue(as_id):
             async with self.member_linearizer.queue(key):
                 async with self._worker_lock_handler.acquire_read_write_lock(
-                    NEW_EVENT_DURING_PURGE_LOCK_NAME, room_id, write=False
+                    DELETE_ROOM_LOCK_NAME, room_id, write=False
                 ):
                     with opentracing.start_active_span("update_membership_locked"):
                         result = await self.update_membership_locked(

synapse/handlers/saml.py

@@ -74,13 +74,12 @@ class SamlHandler:
         self.idp_id = "saml"
 
         # user-facing name of this auth provider
-        self.idp_name = hs.config.saml2.idp_name
+        self.idp_name = "SAML"
 
-        # MXC URI for icon for this auth provider
-        self.idp_icon = hs.config.saml2.idp_icon
-
-        # optional brand identifier for this auth provider
-        self.idp_brand = hs.config.saml2.idp_brand
+        # we do not currently support icons/brands for SAML auth, but this is required by
+        # the SsoIdentityProvider protocol type.
+        self.idp_icon = None
+        self.idp_brand = None
 
         # a map from saml session id to Saml2SessionData object
         self._outstanding_requests_dict: Dict[str, Saml2SessionData] = {}

synapse/handlers/sso.py

@@ -24,14 +24,13 @@ from typing import (
     Iterable,
     List,
     Mapping,
-    NoReturn,
     Optional,
     Set,
 )
 from urllib.parse import urlencode
 
 import attr
-from typing_extensions import Protocol
+from typing_extensions import NoReturn, Protocol
 
 from twisted.web.iweb import IRequest
 from twisted.web.server import Request
@@ -792,7 +791,7 @@ class SsoHandler:
 
             if code != 200:
                 raise Exception(
-                    f"GET request to download sso avatar image returned {code}"
+                    "GET request to download sso avatar image returned {}".format(code)
                 )
 
             # upload name includes hash of the image file's content so that we can

synapse/handlers/stats.py

@@ -14,15 +14,9 @@
 # limitations under the License.
 import logging
 from collections import Counter
-from typing import (
-    TYPE_CHECKING,
-    Any,
-    Counter as CounterType,
-    Dict,
-    Iterable,
-    Optional,
-    Tuple,
-)
+from typing import TYPE_CHECKING, Any, Dict, Iterable, Optional, Tuple
+
+from typing_extensions import Counter as CounterType
 
 from synapse.api.constants import EventContentFields, EventTypes, Membership
 from synapse.metrics import event_processing_positions

synapse/handlers/sync.py

@@ -387,16 +387,16 @@ class SyncHandler:
                 from_token=since_token,
             )
 
-        # if nothing has happened in any of the users' rooms since /sync was called,
-        # the resultant next_batch will be the same as since_token (since the result
-        # is generated when wait_for_events is first called, and not regenerated
-        # when wait_for_events times out).
-        #
-        # If that happens, we mustn't cache it, so that when the client comes back
-        # with the same cache token, we don't immediately return the same empty
-        # result, causing a tightloop. (#8518)
-        if result.next_batch == since_token:
-            cache_context.should_cache = False
+            # if nothing has happened in any of the users' rooms since /sync was called,
+            # the resultant next_batch will be the same as since_token (since the result
+            # is generated when wait_for_events is first called, and not regenerated
+            # when wait_for_events times out).
+            #
+            # If that happens, we mustn't cache it, so that when the client comes back
+            # with the same cache token, we don't immediately return the same empty
+            # result, causing a tightloop. (#8518)
+            if result.next_batch == since_token:
+                cache_context.should_cache = False
 
         if result:
             if sync_config.filter_collection.lazy_load_members():
@@ -1442,9 +1442,11 @@ class SyncHandler:
 
         # Now we have our list of joined room IDs, exclude as configured and freeze
         joined_room_ids = frozenset(
-            room_id
-            for room_id in mutable_joined_room_ids
-            if room_id not in mutable_rooms_to_exclude
+            (
+                room_id
+                for room_id in mutable_joined_room_ids
+                if room_id not in mutable_rooms_to_exclude
+            )
         )
 
         logger.debug(

synapse/handlers/user_directory.py

@@ -94,7 +94,6 @@ class UserDirectoryHandler(StateDeltasHandler):
         self.is_mine_id = hs.is_mine_id
         self.update_user_directory = hs.config.worker.should_update_user_directory
         self.search_all_users = hs.config.userdirectory.user_directory_search_all_users
-        self.show_locked_users = hs.config.userdirectory.show_locked_users
         self._spam_checker_module_callbacks = hs.get_module_api_callbacks().spam_checker
         self._hs = hs
 
@@ -145,9 +144,7 @@ class UserDirectoryHandler(StateDeltasHandler):
                     ]
                 }
         """
-        results = await self.store.search_user_dir(
-            user_id, search_term, limit, self.show_locked_users
-        )
+        results = await self.store.search_user_dir(user_id, search_term, limit)
 
         # Remove any spammy users from the results.
         non_spammy_users = []

synapse/handlers/worker_lock.py

@@ -42,11 +42,7 @@ if TYPE_CHECKING:
     from synapse.server import HomeServer
 
 
-# This lock is used to avoid creating an event while we are purging the room.
-# We take a read lock when creating an event, and a write one when purging a room.
-# This is because it is fine to create several events concurrently, since referenced events
-# will not disappear under our feet as long as we don't delete the room.
-NEW_EVENT_DURING_PURGE_LOCK_NAME = "new_event_during_purge_lock"
+DELETE_ROOM_LOCK_NAME = "delete_room_lock"
 
 
 class WorkerLocksHandler:

synapse/logging/_remote.py

@@ -18,9 +18,10 @@ import traceback
 from collections import deque
 from ipaddress import IPv4Address, IPv6Address, ip_address
 from math import floor
-from typing import Callable, Deque, Optional
+from typing import Callable, Optional
 
 import attr
+from typing_extensions import Deque
 from zope.interface import implementer
 
 from twisted.application.internet import ClientService

synapse/module_api/__init__.py

@@ -31,10 +31,9 @@ from typing import (
 
 import attr
 import jinja2
-from typing_extensions import Concatenate, ParamSpec
+from typing_extensions import ParamSpec
 
 from twisted.internet import defer
-from twisted.internet.interfaces import IDelayedCall
 from twisted.web.resource import Resource
 
 from synapse.api import errors
@@ -885,7 +884,7 @@ class ModuleApi:
     def run_db_interaction(
         self,
         desc: str,
-        func: Callable[Concatenate[LoggingTransaction, P], T],
+        func: Callable[P, T],
         *args: P.args,
         **kwargs: P.kwargs,
     ) -> "defer.Deferred[T]":
@@ -1243,46 +1242,6 @@ class ModuleApi:
         """
         return self._hs.config.worker.run_background_tasks
 
-    def delayed_background_call(
-        self,
-        msec: float,
-        f: Callable,
-        *args: object,
-        desc: Optional[str] = None,
-        **kwargs: object,
-    ) -> IDelayedCall:
-        """Wraps a function as a background process and calls it in a given number of milliseconds.
-
-        The scheduled call is not persistent: if the current Synapse instance is
-        restarted before the call is made, the call will not be made.
-
-        Added in Synapse v1.90.0.
-
-        Args:
-            msec: How long to wait before calling, in milliseconds.
-            f: The function to call once. f can be either synchronous or
-                asynchronous, and must follow Synapse's logcontext rules.
-                More info about logcontexts is available at
-                https://matrix-org.github.io/synapse/latest/log_contexts.html
-            *args: Positional arguments to pass to function.
-            desc: The background task's description. Default to the function's name.
-            **kwargs: Keyword arguments to pass to function.
-
-        Returns:
-            IDelayedCall handle from twisted, which allows to cancel the delayed call if desired.
-        """
-
-        if desc is None:
-            desc = f.__name__
-
-        return self._clock.call_later(
-            # convert ms to seconds as needed by call_later.
-            msec * 0.001,
-            run_as_background_process,
-            desc,
-            lambda: maybe_awaitable(f(*args, **kwargs)),
-        )
-
     async def sleep(self, seconds: float) -> None:
         """Sleeps for the given number of seconds.
 

synapse/module_api/callbacks/spamchecker_callbacks.py

@@ -426,7 +426,9 @@ class SpamCheckerModuleApiCallbacks:
                 generally discouraged as it doesn't support internationalization.
         """
         for callback in self._check_event_for_spam_callbacks:
-            with Measure(self.clock, f"{callback.__module__}.{callback.__qualname__}"):
+            with Measure(
+                self.clock, "{}.{}".format(callback.__module__, callback.__qualname__)
+            ):
                 res = await delay_cancellation(callback(event))
                 if res is False or res == self.NOT_SPAM:
                     # This spam-checker accepts the event.
@@ -479,7 +481,9 @@ class SpamCheckerModuleApiCallbacks:
             True if the event should be silently dropped
         """
         for callback in self._should_drop_federated_event_callbacks:
-            with Measure(self.clock, f"{callback.__module__}.{callback.__qualname__}"):
+            with Measure(
+                self.clock, "{}.{}".format(callback.__module__, callback.__qualname__)
+            ):
                 res: Union[bool, str] = await delay_cancellation(callback(event))
             if res:
                 return res
@@ -501,7 +505,9 @@ class SpamCheckerModuleApiCallbacks:
             NOT_SPAM if the operation is permitted, [Codes, Dict] otherwise.
         """
         for callback in self._user_may_join_room_callbacks:
-            with Measure(self.clock, f"{callback.__module__}.{callback.__qualname__}"):
+            with Measure(
+                self.clock, "{}.{}".format(callback.__module__, callback.__qualname__)
+            ):
                 res = await delay_cancellation(callback(user_id, room_id, is_invited))
                 # Normalize return values to `Codes` or `"NOT_SPAM"`.
                 if res is True or res is self.NOT_SPAM:
@@ -540,7 +546,9 @@ class SpamCheckerModuleApiCallbacks:
             NOT_SPAM if the operation is permitted, Codes otherwise.
         """
         for callback in self._user_may_invite_callbacks:
-            with Measure(self.clock, f"{callback.__module__}.{callback.__qualname__}"):
+            with Measure(
+                self.clock, "{}.{}".format(callback.__module__, callback.__qualname__)
+            ):
                 res = await delay_cancellation(
                     callback(inviter_userid, invitee_userid, room_id)
                 )
@@ -585,7 +593,9 @@ class SpamCheckerModuleApiCallbacks:
             NOT_SPAM if the operation is permitted, Codes otherwise.
         """
         for callback in self._user_may_send_3pid_invite_callbacks:
-            with Measure(self.clock, f"{callback.__module__}.{callback.__qualname__}"):
+            with Measure(
+                self.clock, "{}.{}".format(callback.__module__, callback.__qualname__)
+            ):
                 res = await delay_cancellation(
                     callback(inviter_userid, medium, address, room_id)
                 )
@@ -620,7 +630,9 @@ class SpamCheckerModuleApiCallbacks:
             userid: The ID of the user attempting to create a room
         """
         for callback in self._user_may_create_room_callbacks:
-            with Measure(self.clock, f"{callback.__module__}.{callback.__qualname__}"):
+            with Measure(
+                self.clock, "{}.{}".format(callback.__module__, callback.__qualname__)
+            ):
                 res = await delay_cancellation(callback(userid))
                 if res is True or res is self.NOT_SPAM:
                     continue
@@ -654,7 +666,9 @@ class SpamCheckerModuleApiCallbacks:
 
         """
         for callback in self._user_may_create_room_alias_callbacks:
-            with Measure(self.clock, f"{callback.__module__}.{callback.__qualname__}"):
+            with Measure(
+                self.clock, "{}.{}".format(callback.__module__, callback.__qualname__)
+            ):
                 res = await delay_cancellation(callback(userid, room_alias))
                 if res is True or res is self.NOT_SPAM:
                     continue
@@ -687,7 +701,9 @@ class SpamCheckerModuleApiCallbacks:
             room_id: The ID of the room that would be published
         """
         for callback in self._user_may_publish_room_callbacks:
-            with Measure(self.clock, f"{callback.__module__}.{callback.__qualname__}"):
+            with Measure(
+                self.clock, "{}.{}".format(callback.__module__, callback.__qualname__)
+            ):
                 res = await delay_cancellation(callback(userid, room_id))
                 if res is True or res is self.NOT_SPAM:
                     continue
@@ -726,7 +742,9 @@ class SpamCheckerModuleApiCallbacks:
             True if the user is spammy.
         """
         for callback in self._check_username_for_spam_callbacks:
-            with Measure(self.clock, f"{callback.__module__}.{callback.__qualname__}"):
+            with Measure(
+                self.clock, "{}.{}".format(callback.__module__, callback.__qualname__)
+            ):
                 # Make a copy of the user profile object to ensure the spam checker cannot
                 # modify it.
                 res = await delay_cancellation(callback(user_profile.copy()))
@@ -758,7 +776,9 @@ class SpamCheckerModuleApiCallbacks:
         """
 
         for callback in self._check_registration_for_spam_callbacks:
-            with Measure(self.clock, f"{callback.__module__}.{callback.__qualname__}"):
+            with Measure(
+                self.clock, "{}.{}".format(callback.__module__, callback.__qualname__)
+            ):
                 behaviour = await delay_cancellation(
                     callback(email_threepid, username, request_info, auth_provider_id)
                 )
@@ -800,7 +820,9 @@ class SpamCheckerModuleApiCallbacks:
         """
 
         for callback in self._check_media_file_for_spam_callbacks:
-            with Measure(self.clock, f"{callback.__module__}.{callback.__qualname__}"):
+            with Measure(
+                self.clock, "{}.{}".format(callback.__module__, callback.__qualname__)
+            ):
                 res = await delay_cancellation(callback(file_wrapper, file_info))
                 # Normalize return values to `Codes` or `"NOT_SPAM"`.
                 if res is False or res is self.NOT_SPAM:
@@ -847,7 +869,9 @@ class SpamCheckerModuleApiCallbacks:
         """
 
         for callback in self._check_login_for_spam_callbacks:
-            with Measure(self.clock, f"{callback.__module__}.{callback.__qualname__}"):
+            with Measure(
+                self.clock, "{}.{}".format(callback.__module__, callback.__qualname__)
+            ):
                 res = await delay_cancellation(
                     callback(
                         user_id,

synapse/replication/tcp/client.py

@@ -14,9 +14,7 @@
 """A replication client for use by synapse workers.
 """
 import logging
-from typing import TYPE_CHECKING, Dict, Iterable, Optional, Set, Tuple
-
-from sortedcontainers import SortedList
+from typing import TYPE_CHECKING, Dict, Iterable, List, Optional, Set, Tuple
 
 from twisted.internet import defer
 from twisted.internet.defer import Deferred
@@ -28,7 +26,6 @@ from synapse.logging.context import PreserveLoggingContext, make_deferred_yielda
 from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.replication.tcp.streams import (
     AccountDataStream,
-    CachesStream,
     DeviceListsStream,
     PushersStream,
     PushRulesStream,
@@ -76,7 +73,6 @@ class ReplicationDataHandler:
         self._instance_name = hs.get_instance_name()
         self._typing_handler = hs.get_typing_handler()
         self._state_storage_controller = hs.get_storage_controllers().state
-        self.auth = hs.get_auth()
 
         self._notify_pushers = hs.config.worker.start_pushers
         self._pusher_pool = hs.get_pusherpool()
@@ -88,9 +84,7 @@ class ReplicationDataHandler:
 
         # Map from stream and instance to list of deferreds waiting for the stream to
         # arrive at a particular position. The lists are sorted by stream position.
-        self._streams_to_waiters: Dict[
-            Tuple[str, str], SortedList[Tuple[int, Deferred]]
-        ] = {}
+        self._streams_to_waiters: Dict[Tuple[str, str], List[Tuple[int, Deferred]]] = {}
 
     async def on_rdata(
         self, stream_name: str, instance_name: str, token: int, rows: list
@@ -224,16 +218,6 @@ class ReplicationDataHandler:
                 self._state_storage_controller.notify_event_un_partial_stated(
                     row.event_id
                 )
-        # invalidate the introspection token cache
-        elif stream_name == CachesStream.NAME:
-            for row in rows:
-                if row.cache_func == "introspection_token_invalidation":
-                    if row.keys[0] is None:
-                        # invalidate the whole cache
-                        # mypy ignore - the token cache is defined on MSC3861DelegatedAuth
-                        self.auth.invalidate_token_cache()  # type: ignore[attr-defined]
-                    else:
-                        self.auth.invalidate_cached_tokens(row.keys)  # type: ignore[attr-defined]
 
         await self._presence_handler.process_replication_rows(
             stream_name, instance_name, token, rows
@@ -242,9 +226,7 @@ class ReplicationDataHandler:
         # Notify any waiting deferreds. The list is ordered by position so we
         # just iterate through the list until we reach a position that is
         # greater than the received row position.
-        waiting_list = self._streams_to_waiters.get((stream_name, instance_name))
-        if not waiting_list:
-            return
+        waiting_list = self._streams_to_waiters.get((stream_name, instance_name), [])
 
         # Index of first item with a position after the current token, i.e we
         # have called all deferreds before this index. If not overwritten by
@@ -268,7 +250,7 @@ class ReplicationDataHandler:
 
         # Drop all entries in the waiting list that were called in the above
         # loop. (This maintains the order so no need to resort)
-        del waiting_list[:index_of_first_deferred_not_called]
+        waiting_list[:] = waiting_list[index_of_first_deferred_not_called:]
 
         for deferred in deferreds_to_callback:
             try:
@@ -328,10 +310,11 @@ class ReplicationDataHandler:
         )
 
         waiting_list = self._streams_to_waiters.setdefault(
-            (stream_name, instance_name), SortedList(key=lambda t: t[0])
+            (stream_name, instance_name), []
         )
 
-        waiting_list.add((position, deferred))
+        waiting_list.append((position, deferred))
+        waiting_list.sort(key=lambda t: t[0])
 
         # We measure here to get in flight counts and average waiting time.
         with Measure(self._clock, "repl.wait_for_stream_position"):

synapse/replication/tcp/handler.py

@@ -17,7 +17,6 @@ from typing import (
     TYPE_CHECKING,
     Any,
     Awaitable,
-    Deque,
     Dict,
     Iterable,
     Iterator,
@@ -30,6 +29,7 @@ from typing import (
 )
 
 from prometheus_client import Counter
+from typing_extensions import Deque
 
 from twisted.internet.protocol import ReconnectingClientFactory
 

synapse/rest/admin/__init__.py

@@ -47,7 +47,6 @@ from synapse.rest.admin.federation import (
     ListDestinationsRestServlet,
 )
 from synapse.rest.admin.media import ListMediaInRoom, register_servlets_for_media_repo
-from synapse.rest.admin.oidc import OIDCTokenRevocationRestServlet
 from synapse.rest.admin.registration_tokens import (
     ListRegistrationTokensRestServlet,
     NewRegistrationTokenRestServlet,
@@ -298,8 +297,6 @@ def register_servlets(hs: "HomeServer", http_server: HttpServer) -> None:
     BackgroundUpdateRestServlet(hs).register(http_server)
     BackgroundUpdateStartJobRestServlet(hs).register(http_server)
     ExperimentalFeaturesRestServlet(hs).register(http_server)
-    if hs.config.experimental.msc3861.enabled:
-        OIDCTokenRevocationRestServlet(hs).register(http_server)
 
 
 def register_servlets_for_client_rest_resource(

synapse/rest/admin/oidc.py

@@ -1,55 +0,0 @@
-# Copyright 2023 The Matrix.org Foundation C.I.C
-#
-#  Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-#  You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-#  Unless required by applicable law or agreed to in writing, software
-#  distributed under the License is distributed on an "AS IS" BASIS,
-#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-#  See the License for the specific language governing permissions and
-#  limitations under the License.
-from http import HTTPStatus
-from typing import TYPE_CHECKING, Dict, Tuple
-
-from synapse.http.servlet import RestServlet
-from synapse.http.site import SynapseRequest
-from synapse.rest.admin._base import admin_patterns, assert_requester_is_admin
-
-if TYPE_CHECKING:
-    from synapse.server import HomeServer
-
-
-class OIDCTokenRevocationRestServlet(RestServlet):
-    """
-    Delete a given token introspection response - identified by the `jti` field - from the
-    introspection token cache when a token is revoked at the authorizing server
-    """
-
-    PATTERNS = admin_patterns("/OIDC_token_revocation/(?P<token_id>[^/]*)")
-
-    def __init__(self, hs: "HomeServer"):
-        super().__init__()
-        auth = hs.get_auth()
-
-        # If this endpoint is loaded then we must have enabled delegated auth.
-        from synapse.api.auth.msc3861_delegated import MSC3861DelegatedAuth
-
-        assert isinstance(auth, MSC3861DelegatedAuth)
-
-        self.auth = auth
-        self.store = hs.get_datastores().main
-
-    async def on_DELETE(
-        self, request: SynapseRequest, token_id: str
-    ) -> Tuple[HTTPStatus, Dict]:
-        await assert_requester_is_admin(self.auth, request)
-
-        self.auth._token_cache.invalidate(token_id)
-
-        # make sure we invalidate the cache on any workers
-        await self.store.stream_introspection_token_invalidation((token_id,))
-
-        return HTTPStatus.OK, {}

synapse/rest/admin/users.py

@@ -109,8 +109,6 @@ class UsersRestServletV2(RestServlet):
             )
         deactivated = parse_boolean(request, "deactivated", default=False)
 
-        admins = parse_boolean(request, "admins")
-
         # If support for MSC3866 is not enabled, apply no filtering based on the
         # `approved` column.
         if self._msc3866_enabled:
@@ -148,7 +146,6 @@ class UsersRestServletV2(RestServlet):
             name,
             guests,
             deactivated,
-            admins,
             order_by,
             direction,
             approved,
@@ -283,17 +280,6 @@ class UserRestServletV2(RestServlet):
                 HTTPStatus.BAD_REQUEST, "'deactivated' parameter is not of type boolean"
             )
 
-        lock = body.get("locked", False)
-        if not isinstance(lock, bool):
-            raise SynapseError(
-                HTTPStatus.BAD_REQUEST, "'locked' parameter is not of type boolean"
-            )
-
-        if deactivate and lock:
-            raise SynapseError(
-                HTTPStatus.BAD_REQUEST, "An user can't be deactivated and locked"
-            )
-
         approved: Optional[bool] = None
         if "approved" in body and self._msc3866_enabled:
             approved = body["approved"]
@@ -411,12 +397,6 @@ class UserRestServletV2(RestServlet):
                         target_user.to_string()
                     )
 
-            if "locked" in body:
-                if lock and not user["locked"]:
-                    await self.store.set_user_locked_status(user_id, True)
-                elif not lock and user["locked"]:
-                    await self.store.set_user_locked_status(user_id, False)
-
             if "user_type" in body:
                 await self.store.set_user_type(target_user, user_type)
 

synapse/rest/client/devices.py

@@ -29,6 +29,7 @@ from synapse.http.servlet import (
     parse_integer,
 )
 from synapse.http.site import SynapseRequest
+from synapse.replication.http.devices import ReplicationUploadKeysForUserRestServlet
 from synapse.rest.client._base import client_patterns, interactive_auth_handler
 from synapse.rest.client.models import AuthenticationData
 from synapse.rest.models import RequestBodyModel
@@ -231,7 +232,7 @@ class DehydratedDeviceDataModel(RequestBodyModel):
 class DehydratedDeviceServlet(RestServlet):
     """Retrieve or store a dehydrated device.
 
-    Implements MSC2697.
+    Implements either MSC2697 or MSC3814.
 
     GET /org.matrix.msc2697.v2/dehydrated_device
 
@@ -265,12 +266,7 @@ class DehydratedDeviceServlet(RestServlet):
 
     """
 
-    PATTERNS = client_patterns(
-        "/org.matrix.msc2697.v2/dehydrated_device$",
-        releases=(),
-    )
-
-    def __init__(self, hs: "HomeServer"):
+    def __init__(self, hs: "HomeServer", msc2697: bool = True):
         super().__init__()
         self.hs = hs
         self.auth = hs.get_auth()
@@ -278,6 +274,13 @@ class DehydratedDeviceServlet(RestServlet):
         assert isinstance(handler, DeviceHandler)
         self.device_handler = handler
 
+        self.PATTERNS = client_patterns(
+            "/org.matrix.msc2697.v2/dehydrated_device$"
+            if msc2697
+            else "/org.matrix.msc3814.v1/dehydrated_device$",
+            releases=(),
+        )
+
     async def on_GET(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
         requester = await self.auth.get_user_by_req(request)
         dehydrated_device = await self.device_handler.get_dehydrated_device(
@@ -479,6 +482,13 @@ class DehydratedDeviceV2Servlet(RestServlet):
         self.e2e_keys_handler = hs.get_e2e_keys_handler()
         self.device_handler = handler
 
+        if hs.config.worker.worker_app is None:
+            # if main process
+            self.key_uploader = self.e2e_keys_handler.upload_keys_for_user
+        else:
+            # then a worker
+            self.key_uploader = ReplicationUploadKeysForUserRestServlet.make_client(hs)
+
     async def on_GET(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
         requester = await self.auth.get_user_by_req(request)
 
@@ -503,8 +513,10 @@ class DehydratedDeviceV2Servlet(RestServlet):
         if dehydrated_device is not None:
             (device_id, device_data) = dehydrated_device
 
-            await self.device_handler.delete_dehydrated_device(
-                requester.user.to_string(), device_id
+            result = await self.device_handler.rehydrate_device(
+                requester.user.to_string(),
+                self.auth.get_access_token_from_request(request),
+                device_id,
             )
 
             result = {"device_id": device_id}
@@ -526,14 +538,6 @@ class DehydratedDeviceV2Servlet(RestServlet):
         requester = await self.auth.get_user_by_req(request)
         user_id = requester.user.to_string()
 
-        old_dehydrated_device = await self.device_handler.get_dehydrated_device(user_id)
-
-        # if an old device exists, delete it before creating a new one
-        if old_dehydrated_device:
-            await self.device_handler.delete_dehydrated_device(
-                user_id, old_dehydrated_device[0]
-            )
-
         device_info = submission.dict()
         if "device_keys" not in device_info.keys():
             raise SynapseError(
@@ -541,12 +545,18 @@ class DehydratedDeviceV2Servlet(RestServlet):
                 "Device key(s) not found, these must be provided.",
             )
 
+        # TODO: Those two operations, creating a device and storing the
+        # device's keys should be atomic.
         device_id = await self.device_handler.store_dehydrated_device(
             requester.user.to_string(),
             submission.device_id,
             submission.device_data.dict(),
             submission.initial_device_display_name,
-            device_info,
+        )
+
+        # TODO: Do we need to do something with the result here?
+        await self.key_uploader(
+            user_id=user_id, device_id=submission.device_id, keys=submission.dict()
         )
 
         return 200, {"device_id": device_id}
@@ -563,7 +573,7 @@ def register_servlets(hs: "HomeServer", http_server: HttpServer) -> None:
     if hs.config.worker.worker_app is None:
         DeviceRestServlet(hs).register(http_server)
         if hs.config.experimental.msc2697_enabled:
-            DehydratedDeviceServlet(hs).register(http_server)
+            DehydratedDeviceServlet(hs, msc2697=True).register(http_server)
             ClaimDehydratedDeviceServlet(hs).register(http_server)
         if hs.config.experimental.msc3814_enabled:
             DehydratedDeviceV2Servlet(hs).register(http_server)

synapse/rest/client/logout.py

@@ -40,9 +40,7 @@ class LogoutRestServlet(RestServlet):
         self._device_handler = handler
 
     async def on_POST(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
-        requester = await self.auth.get_user_by_req(
-            request, allow_expired=True, allow_locked=True
-        )
+        requester = await self.auth.get_user_by_req(request, allow_expired=True)
 
         if requester.device_id is None:
             # The access token wasn't associated with a device.
@@ -69,9 +67,7 @@ class LogoutAllRestServlet(RestServlet):
         self._device_handler = handler
 
     async def on_POST(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
-        requester = await self.auth.get_user_by_req(
-            request, allow_expired=True, allow_locked=True
-        )
+        requester = await self.auth.get_user_by_req(request, allow_expired=True)
         user_id = requester.user.to_string()
 
         # first delete all of the user's devices

synapse/rest/client/push_rule.py

@@ -32,7 +32,6 @@ from synapse.push.rulekinds import PRIORITY_CLASS_MAP
 from synapse.rest.client._base import client_patterns
 from synapse.storage.push_rule import InconsistentRuleException, RuleNotFoundException
 from synapse.types import JsonDict
-from synapse.util.async_helpers import Linearizer
 
 if TYPE_CHECKING:
     from synapse.server import HomeServer
@@ -54,32 +53,26 @@ class PushRuleRestServlet(RestServlet):
         self.notifier = hs.get_notifier()
         self._is_worker = hs.config.worker.worker_app is not None
         self._push_rules_handler = hs.get_push_rules_handler()
-        self._push_rule_linearizer = Linearizer(name="push_rules")
 
     async def on_PUT(self, request: SynapseRequest, path: str) -> Tuple[int, JsonDict]:
         if self._is_worker:
             raise Exception("Cannot handle PUT /push_rules on worker")
 
-        requester = await self.auth.get_user_by_req(request)
-        user_id = requester.user.to_string()
-
-        async with self._push_rule_linearizer.queue(user_id):
-            return await self.handle_put(request, path, user_id)
-
-    async def handle_put(
-        self, request: SynapseRequest, path: str, user_id: str
-    ) -> Tuple[int, JsonDict]:
         spec = _rule_spec_from_path(path.split("/"))
         try:
             priority_class = _priority_class_from_spec(spec)
         except InvalidRuleException as e:
             raise SynapseError(400, str(e))
 
+        requester = await self.auth.get_user_by_req(request)
+
         if "/" in spec.rule_id or "\\" in spec.rule_id:
             raise SynapseError(400, "rule_id may not contain slashes")
 
         content = parse_json_value_from_request(request)
 
+        user_id = requester.user.to_string()
+
         if spec.attr:
             try:
                 await self._push_rules_handler.set_rule_attr(user_id, spec, content)
@@ -133,20 +126,11 @@ class PushRuleRestServlet(RestServlet):
         if self._is_worker:
             raise Exception("Cannot handle DELETE /push_rules on worker")
 
+        spec = _rule_spec_from_path(path.split("/"))
+
         requester = await self.auth.get_user_by_req(request)
         user_id = requester.user.to_string()
 
-        async with self._push_rule_linearizer.queue(user_id):
-            return await self.handle_delete(request, path, user_id)
-
-    async def handle_delete(
-        self,
-        request: SynapseRequest,
-        path: str,
-        user_id: str,
-    ) -> Tuple[int, JsonDict]:
-        spec = _rule_spec_from_path(path.split("/"))
-
         namespaced_rule_id = f"global/{spec.template}/{spec.rule_id}"
 
         try:

synapse/rest/client/room_upgrade_rest_servlet.py

@@ -17,7 +17,7 @@ from typing import TYPE_CHECKING, Tuple
 
 from synapse.api.errors import Codes, ShadowBanError, SynapseError
 from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
-from synapse.handlers.worker_lock import NEW_EVENT_DURING_PURGE_LOCK_NAME
+from synapse.handlers.worker_lock import DELETE_ROOM_LOCK_NAME
 from synapse.http.server import HttpServer
 from synapse.http.servlet import (
     RestServlet,
@@ -81,7 +81,7 @@ class RoomUpgradeRestServlet(RestServlet):
 
         try:
             async with self._worker_lock_handler.acquire_read_write_lock(
-                NEW_EVENT_DURING_PURGE_LOCK_NAME, room_id, write=False
+                DELETE_ROOM_LOCK_NAME, room_id, write=False
             ):
                 new_room_id = await self._room_creation_handler.upgrade_room(
                     requester, room_id, new_version

synapse/rest/key/v2/remote_key_resource.py

@@ -14,7 +14,7 @@
 
 import logging
 import re
-from typing import TYPE_CHECKING, Dict, Mapping, Optional, Set, Tuple
+from typing import TYPE_CHECKING, Dict, Optional, Set, Tuple
 
 from signedjson.sign import sign_json
 
@@ -27,7 +27,6 @@ from synapse.http.servlet import (
     parse_integer,
     parse_json_object_from_request,
 )
-from synapse.storage.keys import FetchKeyResultForRemote
 from synapse.types import JsonDict
 from synapse.util import json_decoder
 from synapse.util.async_helpers import yieldable_gather_results
@@ -158,22 +157,14 @@ class RemoteKey(RestServlet):
     ) -> JsonDict:
         logger.info("Handling query for keys %r", query)
 
-        server_keys: Dict[Tuple[str, str], Optional[FetchKeyResultForRemote]] = {}
+        store_queries = []
         for server_name, key_ids in query.items():
-            if key_ids:
-                results: Mapping[
-                    str, Optional[FetchKeyResultForRemote]
-                ] = await self.store.get_server_keys_json_for_remote(
-                    server_name, key_ids
-                )
-            else:
-                results = await self.store.get_all_server_keys_json_for_remote(
-                    server_name
-                )
+            if not key_ids:
+                key_ids = (None,)
+            for key_id in key_ids:
+                store_queries.append((server_name, key_id, None))
 
-            server_keys.update(
-                ((server_name, key_id), res) for key_id, res in results.items()
-            )
+        cached = await self.store.get_server_keys_json_for_remote(store_queries)
 
         json_results: Set[bytes] = set()
 
@@ -182,20 +173,23 @@ class RemoteKey(RestServlet):
         # Map server_name->key_id->int. Note that the value of the int is unused.
         # XXX: why don't we just use a set?
         cache_misses: Dict[str, Dict[str, int]] = {}
-        for (server_name, key_id), key_result in server_keys.items():
-            if not query[server_name]:
+        for (server_name, key_id, _), key_results in cached.items():
+            results = [(result["ts_added_ms"], result) for result in key_results]
+
+            if key_id is None:
                 # all keys were requested. Just return what we have without worrying
                 # about validity
-                if key_result:
-                    json_results.add(key_result.key_json)
+                for _, result in results:
+                    # Cast to bytes since postgresql returns a memoryview.
+                    json_results.add(bytes(result["key_json"]))
                 continue
 
             miss = False
-            if key_result is None:
+            if not results:
                 miss = True
             else:
-                ts_added_ms = key_result.added_ts
-                ts_valid_until_ms = key_result.valid_until_ts
+                ts_added_ms, most_recent_result = max(results)
+                ts_valid_until_ms = most_recent_result["ts_valid_until_ms"]
                 req_key = query.get(server_name, {}).get(key_id, {})
                 req_valid_until = req_key.get("minimum_valid_until_ts")
                 if req_valid_until is not None:
@@ -241,8 +235,8 @@ class RemoteKey(RestServlet):
                         ts_valid_until_ms,
                         time_now_ms,
                     )
-
-                json_results.add(key_result.key_json)
+                # Cast to bytes since postgresql returns a memoryview.
+                json_results.add(bytes(most_recent_result["key_json"]))
 
             if miss and query_remote_on_cache_miss:
                 # only bother attempting to fetch keys from servers on our whitelist

synapse/server.py

@@ -142,7 +142,6 @@ from synapse.util.distributor import Distributor
 from synapse.util.macaroons import MacaroonGenerator
 from synapse.util.ratelimitutils import FederationRateLimiter
 from synapse.util.stringutils import random_string
-from synapse.util.task_scheduler import TaskScheduler
 
 logger = logging.getLogger(__name__)
 
@@ -361,7 +360,6 @@ class HomeServer(metaclass=abc.ABCMeta):
         """
         for i in self.REQUIRED_ON_BACKGROUND_TASK_STARTUP:
             getattr(self, "get_" + i + "_handler")()
-        self.get_task_scheduler()
 
     def get_reactor(self) -> ISynapseReactor:
         """
@@ -917,7 +915,3 @@ class HomeServer(metaclass=abc.ABCMeta):
     @cache_in_self
     def get_worker_locks_handler(self) -> WorkerLocksHandler:
         return WorkerLocksHandler(self)
-
-    @cache_in_self
-    def get_task_scheduler(self) -> TaskScheduler:
-        return TaskScheduler(self)

synapse/storage/background_updates.py

@@ -238,7 +238,6 @@ class BackgroundUpdater:
     def __init__(self, hs: "HomeServer", database: "DatabasePool"):
         self._clock = hs.get_clock()
         self.db_pool = database
-        self.hs = hs
 
         self._database_name = database.name()
 
@@ -759,11 +758,6 @@ class BackgroundUpdater:
                 logger.debug("[SQL] %s", sql)
                 c.execute(sql)
 
-                # override the global statement timeout to avoid accidentally squashing
-                # a long-running index creation process
-                timeout_sql = "SET SESSION statement_timeout = 0"
-                c.execute(timeout_sql)
-
                 sql = (
                     "CREATE %(unique)s INDEX CONCURRENTLY %(name)s"
                     " ON %(table)s"
@@ -784,12 +778,6 @@ class BackgroundUpdater:
                     logger.debug("[SQL] %s", sql)
                     c.execute(sql)
             finally:
-                # mypy ignore - `statement_timeout` is defined on PostgresEngine
-                # reset the global timeout to the default
-                default_timeout = self.db_pool.engine.statement_timeout  # type: ignore[attr-defined]
-                undo_timeout_sql = f"SET statement_timeout = {default_timeout}"
-                conn.cursor().execute(undo_timeout_sql)
-
                 conn.set_session(autocommit=False)  # type: ignore
 
         def create_index_sqlite(conn: Connection) -> None:

synapse/storage/controllers/persist_events.py

@@ -45,7 +45,7 @@ from twisted.internet import defer
 from synapse.api.constants import EventTypes, Membership
 from synapse.events import EventBase
 from synapse.events.snapshot import EventContext
-from synapse.handlers.worker_lock import NEW_EVENT_DURING_PURGE_LOCK_NAME
+from synapse.handlers.worker_lock import DELETE_ROOM_LOCK_NAME
 from synapse.logging.context import PreserveLoggingContext, make_deferred_yieldable
 from synapse.logging.opentracing import (
     SynapseTags,
@@ -357,7 +357,7 @@ class EventsPersistenceStorageController:
         # it. We might already have taken out the lock, but since this is just a
         # "read" lock its inherently reentrant.
         async with self.hs.get_worker_locks_handler().acquire_read_write_lock(
-            NEW_EVENT_DURING_PURGE_LOCK_NAME, room_id, write=False
+            DELETE_ROOM_LOCK_NAME, room_id, write=False
         ):
             if isinstance(task, _PersistEventsTask):
                 return await self._persist_event_batch(room_id, task)

synapse/storage/databases/main/__init__.py

@@ -70,7 +70,6 @@ from .state import StateStore
 from .stats import StatsStore
 from .stream import StreamWorkerStore
 from .tags import TagsStore
-from .task_scheduler import TaskSchedulerWorkerStore
 from .transactions import TransactionWorkerStore
 from .ui_auth import UIAuthStore
 from .user_directory import UserDirectoryStore
@@ -128,7 +127,6 @@ class DataStore(
     CacheInvalidationWorkerStore,
     LockStore,
     SessionStore,
-    TaskSchedulerWorkerStore,
 ):
     def __init__(
         self,
@@ -170,7 +168,6 @@ class DataStore(
         name: Optional[str] = None,
         guests: bool = True,
         deactivated: bool = False,
-        admins: Optional[bool] = None,
         order_by: str = UserSortOrder.NAME.value,
         direction: Direction = Direction.FORWARDS,
         approved: bool = True,
@@ -187,9 +184,6 @@ class DataStore(
             name: search for local part of user_id or display name
             guests: whether to in include guest users
             deactivated: whether to include deactivated users
-            admins: Optional flag to filter admins. If true, only admins are queried.
-                    if false, admins are excluded from the query. When it is
-                    none (the default), both admins and none-admins are queried.
             order_by: the sort order of the returned list
             direction: sort ascending or descending
             approved: whether to include approved users
@@ -226,12 +220,6 @@ class DataStore(
             if not deactivated:
                 filters.append("deactivated = 0")
 
-            if admins is not None:
-                if admins:
-                    filters.append("admin = 1")
-                else:
-                    filters.append("admin = 0")
-
             if not approved:
                 # We ignore NULL values for the approved flag because these should only
                 # be already existing users that we consider as already approved.

synapse/storage/databases/main/cache.py

@@ -18,8 +18,6 @@ import logging
 from typing import TYPE_CHECKING, Any, Collection, Iterable, List, Optional, Tuple
 
 from synapse.api.constants import EventTypes
-from synapse.config._base import Config
-from synapse.metrics.background_process_metrics import wrap_as_background_process
 from synapse.replication.tcp.streams import BackfillStream, CachesStream
 from synapse.replication.tcp.streams.events import (
     EventsStream,
@@ -54,21 +52,6 @@ PURGE_HISTORY_CACHE_NAME = "ph_cache_fake"
 # As above, but for invalidating room caches on room deletion
 DELETE_ROOM_CACHE_NAME = "dr_cache_fake"
 
-# How long between cache invalidation table cleanups, once we have caught up
-# with the backlog.
-REGULAR_CLEANUP_INTERVAL_MS = Config.parse_duration("1h")
-
-# How long between cache invalidation table cleanups, before we have caught
-# up with the backlog.
-CATCH_UP_CLEANUP_INTERVAL_MS = Config.parse_duration("1m")
-
-# Maximum number of cache invalidation rows to delete at once.
-CLEAN_UP_MAX_BATCH_SIZE = 20_000
-
-# Keep cache invalidations for 7 days
-# (This is likely to be quite excessive.)
-RETENTION_PERIOD_OF_CACHE_INVALIDATIONS_MS = Config.parse_duration("7d")
-
 
 class CacheInvalidationWorkerStore(SQLBaseStore):
     def __init__(
@@ -115,18 +98,6 @@ class CacheInvalidationWorkerStore(SQLBaseStore):
         else:
             self._cache_id_gen = None
 
-        # Occasionally clean up the cache invalidations stream table by deleting
-        # old rows.
-        # This is only applicable when Postgres is in use; this table is unused
-        # and not populated at all when SQLite is the active database engine.
-        if hs.config.worker.run_background_tasks and isinstance(
-            self.database_engine, PostgresEngine
-        ):
-            self.hs.get_clock().call_later(
-                CATCH_UP_CLEANUP_INTERVAL_MS / 1000,
-                self._clean_up_cache_invalidation_wrapper,
-            )
-
     async def get_all_updated_caches(
         self, instance_name: str, last_id: int, current_id: int, limit: int
     ) -> Tuple[List[Tuple[int, tuple]], int, bool]:
@@ -583,117 +554,3 @@ class CacheInvalidationWorkerStore(SQLBaseStore):
             return self._cache_id_gen.get_current_token_for_writer(instance_name)
         else:
             return 0
-
-    async def stream_introspection_token_invalidation(
-        self, key: Tuple[Optional[str]]
-    ) -> None:
-        """
-        Stream an invalidation request for the introspection token cache to workers
-
-        Args:
-            key: token_id of the introspection token to remove from the cache
-        """
-        await self.send_invalidation_to_replication(
-            "introspection_token_invalidation", key
-        )
-
-    @wrap_as_background_process("clean_up_old_cache_invalidations")
-    async def _clean_up_cache_invalidation_wrapper(self) -> None:
-        """
-        Clean up cache invalidation stream table entries occasionally.
-        If we are behind (i.e. there are entries old enough to
-        be deleted but too many of them to be deleted in one go),
-        then we run slightly more frequently.
-        """
-        delete_up_to: int = (
-            self.hs.get_clock().time_msec() - RETENTION_PERIOD_OF_CACHE_INVALIDATIONS_MS
-        )
-
-        in_backlog = await self._clean_up_batch_of_old_cache_invalidations(delete_up_to)
-
-        # Vary how long we wait before calling again depending on whether we
-        # are still sifting through backlog or we have caught up.
-        if in_backlog:
-            next_interval = CATCH_UP_CLEANUP_INTERVAL_MS
-        else:
-            next_interval = REGULAR_CLEANUP_INTERVAL_MS
-
-        self.hs.get_clock().call_later(
-            next_interval / 1000, self._clean_up_cache_invalidation_wrapper
-        )
-
-    async def _clean_up_batch_of_old_cache_invalidations(
-        self, delete_up_to_millisec: int
-    ) -> bool:
-        """
-        Remove old rows from the `cache_invalidation_stream_by_instance` table automatically (this table is unused in SQLite).
-
-        Up to `CLEAN_UP_BATCH_SIZE` rows will be deleted at once.
-
-        Returns true if and only if we were limited by batch size (i.e. we are in backlog:
-        there are more things to clean up).
-        """
-
-        def _clean_up_batch_of_old_cache_invalidations_txn(
-            txn: LoggingTransaction,
-        ) -> bool:
-            # First get the earliest stream ID
-            txn.execute(
-                """
-                SELECT stream_id FROM cache_invalidation_stream_by_instance
-                ORDER BY stream_id ASC
-                LIMIT 1
-                """
-            )
-            row = txn.fetchone()
-            if row is None:
-                return False
-            earliest_stream_id: int = row[0]
-
-            # Then find the last stream ID of the range we will delete
-            txn.execute(
-                """
-                SELECT stream_id FROM cache_invalidation_stream_by_instance
-                WHERE stream_id <= ? AND invalidation_ts <= ?
-                ORDER BY stream_id DESC
-                LIMIT 1
-                """,
-                (earliest_stream_id + CLEAN_UP_MAX_BATCH_SIZE, delete_up_to_millisec),
-            )
-            row = txn.fetchone()
-            if row is None:
-                return False
-            cutoff_stream_id: int = row[0]
-
-            # Determine whether we are caught up or still catching up
-            txn.execute(
-                """
-                SELECT invalidation_ts FROM cache_invalidation_stream_by_instance
-                WHERE stream_id > ?
-                ORDER BY stream_id ASC
-                LIMIT 1
-                """,
-                (cutoff_stream_id,),
-            )
-            row = txn.fetchone()
-            if row is None:
-                in_backlog = False
-            else:
-                # We are in backlog if the next row could have been deleted
-                # if we didn't have such a small batch size
-                in_backlog = row[0] <= delete_up_to_millisec
-
-            txn.execute(
-                """
-                DELETE FROM cache_invalidation_stream_by_instance
-                WHERE ? <= stream_id AND stream_id <= ?
-                """,
-                (earliest_stream_id, cutoff_stream_id),
-            )
-
-            return in_backlog
-
-        return await self.db_pool.runInteraction(
-            "clean_up_old_cache_invalidations",
-            _clean_up_batch_of_old_cache_invalidations_txn,
-        )

synapse/storage/databases/main/client_ips.py

@@ -579,11 +579,6 @@ class ClientIpWorkerStore(ClientIpBackgroundUpdateStore, MonthlyActiveUsersWorke
         device_id: Optional[str],
         now: Optional[int] = None,
     ) -> None:
-        # The sync proxy continuously triggers /sync even if the user is not
-        # present so should be excluded from user_ips entries.
-        if user_agent == "sync-v3-proxy-":
-            return
-
         if not now:
             now = int(self._clock.time_msec())
         key = (user_id, access_token, ip)

synapse/storage/databases/main/devices.py

@@ -28,12 +28,10 @@ from typing import (
     cast,
 )
 
-from canonicaljson import encode_canonical_json
 from typing_extensions import Literal
 
 from synapse.api.constants import EduTypes
 from synapse.api.errors import Codes, StoreError
-from synapse.config.homeserver import HomeServerConfig
 from synapse.logging.opentracing import (
     get_active_span_text_map,
     set_tag,
@@ -1190,42 +1188,8 @@ class DeviceWorkerStore(RoomMemberWorkerStore, EndToEndKeyWorkerStore):
         )
 
     def _store_dehydrated_device_txn(
-        self,
-        txn: LoggingTransaction,
-        user_id: str,
-        device_id: str,
-        device_data: str,
-        time: int,
-        keys: Optional[JsonDict] = None,
+        self, txn: LoggingTransaction, user_id: str, device_id: str, device_data: str
     ) -> Optional[str]:
-        # TODO: make keys non-optional once support for msc2697 is dropped
-        if keys:
-            device_keys = keys.get("device_keys", None)
-            if device_keys:
-                # Type ignore - this function is defined on EndToEndKeyStore which we do
-                # have access to due to hs.get_datastore() "magic"
-                self._set_e2e_device_keys_txn(  # type: ignore[attr-defined]
-                    txn, user_id, device_id, time, device_keys
-                )
-
-            one_time_keys = keys.get("one_time_keys", None)
-            if one_time_keys:
-                key_list = []
-                for key_id, key_obj in one_time_keys.items():
-                    algorithm, key_id = key_id.split(":")
-                    key_list.append(
-                        (
-                            algorithm,
-                            key_id,
-                            encode_canonical_json(key_obj).decode("ascii"),
-                        )
-                    )
-                self._add_e2e_one_time_keys_txn(txn, user_id, device_id, time, key_list)
-
-            fallback_keys = keys.get("fallback_keys", None)
-            if fallback_keys:
-                self._set_e2e_fallback_keys_txn(txn, user_id, device_id, fallback_keys)
-
         old_device_id = self.db_pool.simple_select_one_onecol_txn(
             txn,
             table="dehydrated_devices",
@@ -1239,16 +1203,10 @@ class DeviceWorkerStore(RoomMemberWorkerStore, EndToEndKeyWorkerStore):
             keyvalues={"user_id": user_id},
             values={"device_id": device_id, "device_data": device_data},
         )
-
         return old_device_id
 
     async def store_dehydrated_device(
-        self,
-        user_id: str,
-        device_id: str,
-        device_data: JsonDict,
-        time_now: int,
-        keys: Optional[dict] = None,
+        self, user_id: str, device_id: str, device_data: JsonDict
     ) -> Optional[str]:
         """Store a dehydrated device for a user.
 
@@ -1256,21 +1214,15 @@ class DeviceWorkerStore(RoomMemberWorkerStore, EndToEndKeyWorkerStore):
             user_id: the user that we are storing the device for
             device_id: the ID of the dehydrated device
             device_data: the dehydrated device information
-            time_now: current time at the request in milliseconds
-            keys: keys for the dehydrated device
-
         Returns:
             device id of the user's previous dehydrated device, if any
         """
-
         return await self.db_pool.runInteraction(
             "store_dehydrated_device_txn",
             self._store_dehydrated_device_txn,
             user_id,
             device_id,
             json_encoder.encode(device_data),
-            time_now,
-            keys,
         )
 
     async def remove_dehydrated_device(self, user_id: str, device_id: str) -> bool:
@@ -1664,7 +1616,6 @@ class DeviceStore(DeviceWorkerStore, DeviceBackgroundUpdateStore):
         self.device_id_exists_cache: LruCache[
             Tuple[str, str], Literal[True]
         ] = LruCache(cache_name="device_id_exists", max_size=10000)
-        self.config: HomeServerConfig = hs.config
 
     async def store_device(
         self,
@@ -1786,13 +1737,6 @@ class DeviceStore(DeviceWorkerStore, DeviceBackgroundUpdateStore):
         for device_id in device_ids:
             self.device_id_exists_cache.invalidate((user_id, device_id))
 
-        # TODO: don't nuke the entire cache once there is a way to associate
-        #  device_id -> introspection_token
-        if self.config.experimental.msc3861.enabled:
-            # mypy ignore - the token cache is defined on MSC3861DelegatedAuth
-            self.auth._token_cache.invalidate_all()  # type: ignore[attr-defined]
-            await self.stream_introspection_token_invalidation((None,))
-
     async def update_device(
         self, user_id: str, device_id: str, new_display_name: Optional[str] = None
     ) -> None:

synapse/storage/databases/main/end_to_end_keys.py

@@ -522,57 +522,36 @@ class EndToEndKeyWorkerStore(EndToEndKeyBackgroundStore, CacheInvalidationWorker
             new_keys: keys to add - each a tuple of (algorithm, key_id, key json)
         """
 
+        def _add_e2e_one_time_keys(txn: LoggingTransaction) -> None:
+            set_tag("user_id", user_id)
+            set_tag("device_id", device_id)
+            set_tag("new_keys", str(new_keys))
+            # We are protected from race between lookup and insertion due to
+            # a unique constraint. If there is a race of two calls to
+            # `add_e2e_one_time_keys` then they'll conflict and we will only
+            # insert one set.
+            self.db_pool.simple_insert_many_txn(
+                txn,
+                table="e2e_one_time_keys_json",
+                keys=(
+                    "user_id",
+                    "device_id",
+                    "algorithm",
+                    "key_id",
+                    "ts_added_ms",
+                    "key_json",
+                ),
+                values=[
+                    (user_id, device_id, algorithm, key_id, time_now, json_bytes)
+                    for algorithm, key_id, json_bytes in new_keys
+                ],
+            )
+            self._invalidate_cache_and_stream(
+                txn, self.count_e2e_one_time_keys, (user_id, device_id)
+            )
+
         await self.db_pool.runInteraction(
-            "add_e2e_one_time_keys_insert",
-            self._add_e2e_one_time_keys_txn,
-            user_id,
-            device_id,
-            time_now,
-            new_keys,
-        )
-
-    def _add_e2e_one_time_keys_txn(
-        self,
-        txn: LoggingTransaction,
-        user_id: str,
-        device_id: str,
-        time_now: int,
-        new_keys: Iterable[Tuple[str, str, str]],
-    ) -> None:
-        """Insert some new one time keys for a device. Errors if any of the keys already exist.
-
-        Args:
-             user_id: id of user to get keys for
-             device_id: id of device to get keys for
-             time_now: insertion time to record (ms since epoch)
-             new_keys: keys to add - each a tuple of (algorithm, key_id, key json) - note
-             that the key JSON must be in canonical JSON form
-        """
-        set_tag("user_id", user_id)
-        set_tag("device_id", device_id)
-        set_tag("new_keys", str(new_keys))
-        # We are protected from race between lookup and insertion due to
-        # a unique constraint. If there is a race of two calls to
-        # `add_e2e_one_time_keys` then they'll conflict and we will only
-        # insert one set.
-        self.db_pool.simple_insert_many_txn(
-            txn,
-            table="e2e_one_time_keys_json",
-            keys=(
-                "user_id",
-                "device_id",
-                "algorithm",
-                "key_id",
-                "ts_added_ms",
-                "key_json",
-            ),
-            values=[
-                (user_id, device_id, algorithm, key_id, time_now, json_bytes)
-                for algorithm, key_id, json_bytes in new_keys
-            ],
-        )
-        self._invalidate_cache_and_stream(
-            txn, self.count_e2e_one_time_keys, (user_id, device_id)
+            "add_e2e_one_time_keys_insert", _add_e2e_one_time_keys
         )
 
     @cached(max_entries=10000)
@@ -744,14 +723,6 @@ class EndToEndKeyWorkerStore(EndToEndKeyBackgroundStore, CacheInvalidationWorker
         device_id: str,
         fallback_keys: JsonDict,
     ) -> None:
-        """Set the user's e2e fallback keys.
-
-        Args:
-            user_id: the user whose keys are being set
-            device_id: the device whose keys are being set
-            fallback_keys: the keys to set.  This is a map from key ID (which is
-                    of the form "algorithm:id") to key data.
-        """
         # fallback_keys will usually only have one item in it, so using a for
         # loop (as opposed to calling simple_upsert_many_txn) won't be too bad
         # FIXME: make sure that only one key per algorithm is uploaded
@@ -1333,70 +1304,43 @@ class EndToEndKeyStore(EndToEndKeyWorkerStore, SQLBaseStore):
     ) -> bool:
         """Stores device keys for a device. Returns whether there was a change
         or the keys were already in the database.
-
-            Args:
-                user_id: user_id of the user to store keys for
-                device_id: device_id of the device to store keys for
-                time_now: time at the request to store the keys
-                device_keys: the keys to store
         """
 
+        def _set_e2e_device_keys_txn(txn: LoggingTransaction) -> bool:
+            set_tag("user_id", user_id)
+            set_tag("device_id", device_id)
+            set_tag("time_now", time_now)
+            set_tag("device_keys", str(device_keys))
+
+            old_key_json = self.db_pool.simple_select_one_onecol_txn(
+                txn,
+                table="e2e_device_keys_json",
+                keyvalues={"user_id": user_id, "device_id": device_id},
+                retcol="key_json",
+                allow_none=True,
+            )
+
+            # In py3 we need old_key_json to match new_key_json type. The DB
+            # returns unicode while encode_canonical_json returns bytes.
+            new_key_json = encode_canonical_json(device_keys).decode("utf-8")
+
+            if old_key_json == new_key_json:
+                log_kv({"Message": "Device key already stored."})
+                return False
+
+            self.db_pool.simple_upsert_txn(
+                txn,
+                table="e2e_device_keys_json",
+                keyvalues={"user_id": user_id, "device_id": device_id},
+                values={"ts_added_ms": time_now, "key_json": new_key_json},
+            )
+            log_kv({"message": "Device keys stored."})
+            return True
+
         return await self.db_pool.runInteraction(
-            "set_e2e_device_keys",
-            self._set_e2e_device_keys_txn,
-            user_id,
-            device_id,
-            time_now,
-            device_keys,
+            "set_e2e_device_keys", _set_e2e_device_keys_txn
         )
 
-    def _set_e2e_device_keys_txn(
-        self,
-        txn: LoggingTransaction,
-        user_id: str,
-        device_id: str,
-        time_now: int,
-        device_keys: JsonDict,
-    ) -> bool:
-        """Stores device keys for a device. Returns whether there was a change
-        or the keys were already in the database.
-
-        Args:
-             user_id: user_id of the user to store keys for
-             device_id: device_id of the device to store keys for
-             time_now: time at the request to store the keys
-             device_keys: the keys to store
-        """
-        set_tag("user_id", user_id)
-        set_tag("device_id", device_id)
-        set_tag("time_now", time_now)
-        set_tag("device_keys", str(device_keys))
-
-        old_key_json = self.db_pool.simple_select_one_onecol_txn(
-            txn,
-            table="e2e_device_keys_json",
-            keyvalues={"user_id": user_id, "device_id": device_id},
-            retcol="key_json",
-            allow_none=True,
-        )
-
-        # In py3 we need old_key_json to match new_key_json type. The DB
-        # returns unicode while encode_canonical_json returns bytes.
-        new_key_json = encode_canonical_json(device_keys).decode("utf-8")
-
-        if old_key_json == new_key_json:
-            log_kv({"Message": "Device key already stored."})
-            return False
-
-        self.db_pool.simple_upsert_txn(
-            txn,
-            table="e2e_device_keys_json",
-            keyvalues={"user_id": user_id, "device_id": device_id},
-            values={"ts_added_ms": time_now, "key_json": new_key_json},
-        )
-        log_kv({"message": "Device keys stored."})
-        return True
-
     async def delete_e2e_keys_by_device(self, user_id: str, device_id: str) -> None:
         def delete_e2e_keys_by_device_txn(txn: LoggingTransaction) -> None:
             log_kv(

synapse/storage/databases/main/event_federation.py

@@ -452,56 +452,33 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
         # sets.
         seen_chains: Set[int] = set()
 
-        # Fetch the chain cover index for the initial set of events we're
-        # considering.
-        def fetch_chain_info(events_to_fetch: Collection[str]) -> None:
-            sql = """
-                SELECT event_id, chain_id, sequence_number
-                FROM event_auth_chains
-                WHERE %s
-            """
-            for batch in batch_iter(events_to_fetch, 1000):
-                clause, args = make_in_list_sql_clause(
-                    txn.database_engine, "event_id", batch
-                )
-                txn.execute(sql % (clause,), args)
+        sql = """
+            SELECT event_id, chain_id, sequence_number
+            FROM event_auth_chains
+            WHERE %s
+        """
+        for batch in batch_iter(initial_events, 1000):
+            clause, args = make_in_list_sql_clause(
+                txn.database_engine, "event_id", batch
+            )
+            txn.execute(sql % (clause,), args)
 
-                for event_id, chain_id, sequence_number in txn:
-                    chain_info[event_id] = (chain_id, sequence_number)
-                    seen_chains.add(chain_id)
-                    chain_to_event.setdefault(chain_id, {})[sequence_number] = event_id
-
-        fetch_chain_info(initial_events)
+            for event_id, chain_id, sequence_number in txn:
+                chain_info[event_id] = (chain_id, sequence_number)
+                seen_chains.add(chain_id)
+                chain_to_event.setdefault(chain_id, {})[sequence_number] = event_id
 
         # Check that we actually have a chain ID for all the events.
         events_missing_chain_info = initial_events.difference(chain_info)
-
-        # The result set to return, i.e. the auth chain difference.
-        result: Set[str] = set()
-
         if events_missing_chain_info:
-            # For some reason we have events we haven't calculated the chain
-            # index for, so we need to handle those separately. This should only
-            # happen for older rooms where the server doesn't have all the auth
-            # events.
-            result = self._fixup_auth_chain_difference_sets(
-                txn,
+            # This can happen due to e.g. downgrade/upgrade of the server. We
+            # raise an exception and fall back to the previous algorithm.
+            logger.info(
+                "Unexpectedly found that events don't have chain IDs in room %s: %s",
                 room_id,
-                state_sets=state_sets,
-                events_missing_chain_info=events_missing_chain_info,
-                events_that_have_chain_index=chain_info,
+                events_missing_chain_info,
             )
-
-            # We now need to refetch any events that we have added to the state
-            # sets.
-            new_events_to_fetch = {
-                event_id
-                for state_set in state_sets
-                for event_id in state_set
-                if event_id not in initial_events
-            }
-
-            fetch_chain_info(new_events_to_fetch)
+            raise _NoChainCoverIndex(room_id)
 
         # Corresponds to `state_sets`, except as a map from chain ID to max
         # sequence number reachable from the state set.
@@ -510,8 +487,8 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
             chains: Dict[int, int] = {}
             set_to_chain.append(chains)
 
-            for state_id in state_set:
-                chain_id, seq_no = chain_info[state_id]
+            for event_id in state_set:
+                chain_id, seq_no = chain_info[event_id]
 
                 chains[chain_id] = max(seq_no, chains.get(chain_id, 0))
 
@@ -555,6 +532,7 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
         # from *any* state set and the minimum sequence number reachable from
         # *all* state sets. Events in that range are in the auth chain
         # difference.
+        result = set()
 
         # Mapping from chain ID to the range of sequence numbers that should be
         # pulled from the database.
@@ -610,122 +588,6 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
 
         return result
 
-    def _fixup_auth_chain_difference_sets(
-        self,
-        txn: LoggingTransaction,
-        room_id: str,
-        state_sets: List[Set[str]],
-        events_missing_chain_info: Set[str],
-        events_that_have_chain_index: Collection[str],
-    ) -> Set[str]:
-        """Helper for `_get_auth_chain_difference_using_cover_index_txn` to
-        handle the case where we haven't calculated the chain cover index for
-        all events.
-
-        This modifies `state_sets` so that they only include events that have a
-        chain cover index, and returns a set of event IDs that are part of the
-        auth difference.
-        """
-
-        # This works similarly to the handling of unpersisted events in
-        # `synapse.state.v2_get_auth_chain_difference`. We uses the observation
-        # that if you can split the set of events into two classes X and Y,
-        # where no events in Y have events in X in their auth chain, then we can
-        # calculate the auth difference by considering X and Y separately.
-        #
-        # We do this in three steps:
-        #   1. Compute the set of events without chain cover index belonging to
-        #      the auth difference.
-        #   2. Replacing the un-indexed events in the state_sets with their auth
-        #      events, recursively, until the state_sets contain only indexed
-        #      events. We can then calculate the auth difference of those state
-        #      sets using the chain cover index.
-        #   3. Add the results of 1 and 2 together.
-
-        # By construction we know that all events that we haven't persisted the
-        # chain cover index for are contained in
-        # `event_auth_chain_to_calculate`, so we pull out the events from those
-        # rather than doing recursive queries to walk the auth chain.
-        #
-        # We pull out those events with their auth events, which gives us enough
-        # information to construct the auth chain of an event up to auth events
-        # that have the chain cover index.
-        sql = """
-            SELECT tc.event_id, ea.auth_id, eac.chain_id IS NOT NULL
-            FROM event_auth_chain_to_calculate AS tc
-            LEFT JOIN event_auth AS ea USING (event_id)
-            LEFT JOIN event_auth_chains AS eac ON (ea.auth_id = eac.event_id)
-            WHERE tc.room_id = ?
-        """
-        txn.execute(sql, (room_id,))
-        event_to_auth_ids: Dict[str, Set[str]] = {}
-        events_that_have_chain_index = set(events_that_have_chain_index)
-        for event_id, auth_id, auth_id_has_chain in txn:
-            s = event_to_auth_ids.setdefault(event_id, set())
-            if auth_id is not None:
-                s.add(auth_id)
-                if auth_id_has_chain:
-                    events_that_have_chain_index.add(auth_id)
-
-        if events_missing_chain_info - event_to_auth_ids.keys():
-            # Uh oh, we somehow haven't correctly done the chain cover index,
-            # bail and fall back to the old method.
-            logger.info(
-                "Unexpectedly found that events don't have chain IDs in room %s: %s",
-                room_id,
-                events_missing_chain_info - event_to_auth_ids.keys(),
-            )
-            raise _NoChainCoverIndex(room_id)
-
-        # Create a map from event IDs we care about to their partial auth chain.
-        event_id_to_partial_auth_chain: Dict[str, Set[str]] = {}
-        for event_id, auth_ids in event_to_auth_ids.items():
-            if not any(event_id in state_set for state_set in state_sets):
-                continue
-
-            processing = set(auth_ids)
-            to_add = set()
-            while processing:
-                auth_id = processing.pop()
-                to_add.add(auth_id)
-
-                sub_auth_ids = event_to_auth_ids.get(auth_id)
-                if sub_auth_ids is None:
-                    continue
-
-                processing.update(sub_auth_ids - to_add)
-
-            event_id_to_partial_auth_chain[event_id] = to_add
-
-        # Now we do two things:
-        #   1. Update the state sets to only include indexed events; and
-        #   2. Create a new list containing the auth chains of the un-indexed
-        #      events
-        unindexed_state_sets: List[Set[str]] = []
-        for state_set in state_sets:
-            unindexed_state_set = set()
-            for event_id, auth_chain in event_id_to_partial_auth_chain.items():
-                if event_id not in state_set:
-                    continue
-
-                unindexed_state_set.add(event_id)
-
-                state_set.discard(event_id)
-                state_set.difference_update(auth_chain)
-                for auth_id in auth_chain:
-                    if auth_id in events_that_have_chain_index:
-                        state_set.add(auth_id)
-                    else:
-                        unindexed_state_set.add(auth_id)
-
-            unindexed_state_sets.append(unindexed_state_set)
-
-        # Calculate and return the auth difference of the un-indexed events.
-        union = unindexed_state_sets[0].union(*unindexed_state_sets[1:])
-        intersection = unindexed_state_sets[0].intersection(*unindexed_state_sets[1:])
-
-        return union - intersection
-
     def _get_auth_chain_difference_txn(
         self, txn: LoggingTransaction, state_sets: List[Set[str]]
     ) -> Set[str]:

synapse/storage/databases/main/filtering.py

@@ -13,9 +13,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from typing import TYPE_CHECKING, Optional, Tuple, Union, cast
+from typing import Optional, Tuple, Union, cast
 
 from canonicaljson import encode_canonical_json
+from typing_extensions import TYPE_CHECKING
 
 from synapse.api.errors import Codes, StoreError, SynapseError
 from synapse.storage._base import SQLBaseStore, db_to_json

synapse/storage/databases/main/keys.py

@@ -16,13 +16,14 @@
 import itertools
 import json
 import logging
-from typing import Dict, Iterable, Mapping, Optional, Tuple
+from typing import Any, Dict, Iterable, List, Mapping, Optional, Tuple
 
 from signedjson.key import decode_verify_key_bytes
 from unpaddedbase64 import decode_base64
 
-from synapse.storage.databases.main.cache import CacheInvalidationWorkerStore
-from synapse.storage.keys import FetchKeyResult, FetchKeyResultForRemote
+from synapse.storage._base import SQLBaseStore
+from synapse.storage.database import LoggingTransaction
+from synapse.storage.keys import FetchKeyResult
 from synapse.storage.types import Cursor
 from synapse.util.caches.descriptors import cached, cachedList
 from synapse.util.iterutils import batch_iter
@@ -33,7 +34,7 @@ logger = logging.getLogger(__name__)
 db_binary_type = memoryview
 
 
-class KeyStore(CacheInvalidationWorkerStore):
+class KeyStore(SQLBaseStore):
     """Persistence for signature verification keys"""
 
     @cached()
@@ -187,12 +188,7 @@ class KeyStore(CacheInvalidationWorkerStore):
         # invalidate takes a tuple corresponding to the params of
         # _get_server_keys_json. _get_server_keys_json only takes one
         # param, which is itself the 2-tuple (server_name, key_id).
-        await self.invalidate_cache_and_stream(
-            "_get_server_keys_json", ((server_name, key_id),)
-        )
-        await self.invalidate_cache_and_stream(
-            "get_server_key_json_for_remote", (server_name, key_id)
-        )
+        self._get_server_keys_json.invalidate((((server_name, key_id),)))
 
     @cached()
     def _get_server_keys_json(
@@ -257,87 +253,47 @@ class KeyStore(CacheInvalidationWorkerStore):
 
         return await self.db_pool.runInteraction("get_server_keys_json", _txn)
 
-    @cached()
-    def get_server_key_json_for_remote(
-        self,
-        server_name: str,
-        key_id: str,
-    ) -> Optional[FetchKeyResultForRemote]:
-        raise NotImplementedError()
-
-    @cachedList(
-        cached_method_name="get_server_key_json_for_remote", list_name="key_ids"
-    )
     async def get_server_keys_json_for_remote(
-        self, server_name: str, key_ids: Iterable[str]
-    ) -> Dict[str, Optional[FetchKeyResultForRemote]]:
-        """Fetch the cached keys for the given server/key IDs.
+        self, server_keys: Iterable[Tuple[str, Optional[str], Optional[str]]]
+    ) -> Dict[Tuple[str, Optional[str], Optional[str]], List[Dict[str, Any]]]:
+        """Retrieve the key json for a list of server_keys and key ids.
+        If no keys are found for a given server, key_id and source then
+        that server, key_id, and source triplet entry will be an empty list.
+        The JSON is returned as a byte array so that it can be efficiently
+        used in an HTTP response.
 
-        If we have multiple entries for a given key ID, returns the most recent.
+        Args:
+            server_keys: List of (server_name, key_id, source) triplets.
+
+        Returns:
+            A mapping from (server_name, key_id, source) triplets to a list of dicts
         """
-        rows = await self.db_pool.simple_select_many_batch(
-            table="server_keys_json",
-            column="key_id",
-            iterable=key_ids,
-            keyvalues={"server_name": server_name},
-            retcols=(
-                "key_id",
-                "from_server",
-                "ts_added_ms",
-                "ts_valid_until_ms",
-                "key_json",
-            ),
-            desc="get_server_keys_json_for_remote",
+
+        def _get_server_keys_json_txn(
+            txn: LoggingTransaction,
+        ) -> Dict[Tuple[str, Optional[str], Optional[str]], List[Dict[str, Any]]]:
+            results = {}
+            for server_name, key_id, from_server in server_keys:
+                keyvalues = {"server_name": server_name}
+                if key_id is not None:
+                    keyvalues["key_id"] = key_id
+                if from_server is not None:
+                    keyvalues["from_server"] = from_server
+                rows = self.db_pool.simple_select_list_txn(
+                    txn,
+                    "server_keys_json",
+                    keyvalues=keyvalues,
+                    retcols=(
+                        "key_id",
+                        "from_server",
+                        "ts_added_ms",
+                        "ts_valid_until_ms",
+                        "key_json",
+                    ),
+                )
+                results[(server_name, key_id, from_server)] = rows
+            return results
+
+        return await self.db_pool.runInteraction(
+            "get_server_keys_json", _get_server_keys_json_txn
         )
-
-        if not rows:
-            return {}
-
-        # We sort the rows so that the most recently added entry is picked up.
-        rows.sort(key=lambda r: r["ts_added_ms"])
-
-        return {
-            row["key_id"]: FetchKeyResultForRemote(
-                # Cast to bytes since postgresql returns a memoryview.
-                key_json=bytes(row["key_json"]),
-                valid_until_ts=row["ts_valid_until_ms"],
-                added_ts=row["ts_added_ms"],
-            )
-            for row in rows
-        }
-
-    async def get_all_server_keys_json_for_remote(
-        self,
-        server_name: str,
-    ) -> Dict[str, FetchKeyResultForRemote]:
-        """Fetch the cached keys for the given server.
-
-        If we have multiple entries for a given key ID, returns the most recent.
-        """
-        rows = await self.db_pool.simple_select_list(
-            table="server_keys_json",
-            keyvalues={"server_name": server_name},
-            retcols=(
-                "key_id",
-                "from_server",
-                "ts_added_ms",
-                "ts_valid_until_ms",
-                "key_json",
-            ),
-            desc="get_server_keys_json_for_remote",
-        )
-
-        if not rows:
-            return {}
-
-        rows.sort(key=lambda r: r["ts_added_ms"])
-
-        return {
-            row["key_id"]: FetchKeyResultForRemote(
-                # Cast to bytes since postgresql returns a memoryview.
-                key_json=bytes(row["key_json"]),
-                valid_until_ts=row["ts_valid_until_ms"],
-                added_ts=row["ts_added_ms"],
-            )
-            for row in rows
-        }

synapse/storage/databases/main/lock.py

@@ -26,6 +26,7 @@ from synapse.storage.database import (
     LoggingDatabaseConnection,
     LoggingTransaction,
 )
+from synapse.storage.engines import PostgresEngine
 from synapse.util import Clock
 from synapse.util.stringutils import random_string
 
@@ -95,10 +96,6 @@ class LockStore(SQLBaseStore):
 
         self._acquiring_locks: Set[Tuple[str, str]] = set()
 
-        self._clock.looping_call(
-            self._reap_stale_read_write_locks, _LOCK_TIMEOUT_MS / 10.0
-        )
-
     @wrap_as_background_process("LockStore._on_shutdown")
     async def _on_shutdown(self) -> None:
         """Called when the server is shutting down"""
@@ -219,7 +216,6 @@ class LockStore(SQLBaseStore):
                 lock_name,
                 lock_key,
                 write,
-                db_autocommit=True,
             )
         except self.database_engine.module.IntegrityError:
             return None
@@ -237,22 +233,61 @@ class LockStore(SQLBaseStore):
         # `worker_read_write_locks` and seeing if that fails any
         # constraints. If it doesn't then we have acquired the lock,
         # otherwise we haven't.
+        #
+        # Before that though we clear the table of any stale locks.
 
         now = self._clock.time_msec()
         token = random_string(6)
 
-        self.db_pool.simple_insert_txn(
-            txn,
-            table="worker_read_write_locks",
-            values={
-                "lock_name": lock_name,
-                "lock_key": lock_key,
-                "write_lock": write,
-                "instance_name": self._instance_name,
-                "token": token,
-                "last_renewed_ts": now,
-            },
-        )
+        delete_sql = """
+            DELETE FROM worker_read_write_locks
+                WHERE last_renewed_ts < ? AND lock_name = ? AND lock_key = ?;
+        """
+
+        insert_sql = """
+            INSERT INTO worker_read_write_locks (lock_name, lock_key, write_lock, instance_name, token, last_renewed_ts)
+            VALUES (?, ?, ?, ?, ?, ?)
+        """
+
+        if isinstance(self.database_engine, PostgresEngine):
+            # For Postgres we can send these queries at the same time.
+            txn.execute(
+                delete_sql + ";" + insert_sql,
+                (
+                    # DELETE args
+                    now - _LOCK_TIMEOUT_MS,
+                    lock_name,
+                    lock_key,
+                    # UPSERT args
+                    lock_name,
+                    lock_key,
+                    write,
+                    self._instance_name,
+                    token,
+                    now,
+                ),
+            )
+        else:
+            # For SQLite these need to be two queries.
+            txn.execute(
+                delete_sql,
+                (
+                    now - _LOCK_TIMEOUT_MS,
+                    lock_name,
+                    lock_key,
+                ),
+            )
+            txn.execute(
+                insert_sql,
+                (
+                    lock_name,
+                    lock_key,
+                    write,
+                    self._instance_name,
+                    token,
+                    now,
+                ),
+            )
 
         lock = Lock(
             self._reactor,
@@ -316,24 +351,6 @@ class LockStore(SQLBaseStore):
 
         return locks
 
-    @wrap_as_background_process("_reap_stale_read_write_locks")
-    async def _reap_stale_read_write_locks(self) -> None:
-        delete_sql = """
-            DELETE FROM worker_read_write_locks
-                WHERE last_renewed_ts < ?
-        """
-
-        def reap_stale_read_write_locks_txn(txn: LoggingTransaction) -> None:
-            txn.execute(delete_sql, (self._clock.time_msec() - _LOCK_TIMEOUT_MS,))
-            if txn.rowcount:
-                logger.info("Reaped %d stale locks", txn.rowcount)
-
-        await self.db_pool.runInteraction(
-            "_reap_stale_read_write_locks",
-            reap_stale_read_write_locks_txn,
-            db_autocommit=True,
-        )
-
 
 class Lock:
     """An async context manager that manages an acquired lock, ensuring it is