conditionally import psycopg2

CHANGES.md

@@ -1,124 +1,3 @@
-Synapse 1.61.1 (2022-06-28)
-===========================
-
-This patch release fixes a security issue regarding URL previews, affecting all prior versions of Synapse. Server administrators are encouraged to update Synapse as soon as possible. We are not aware of these vulnerabilities being exploited in the wild.
-
-Server administrators who are unable to update Synapse may use the workarounds described in the linked GitHub Security Advisory below.
-
-## Security advisory
-
-The following issue is fixed in 1.61.1.
-
-* [GHSA-22p3-qrh9-cx32](https://github.com/matrix-org/synapse/security/advisories/GHSA-22p3-qrh9-cx32) / [CVE-2022-31052](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-31052)
-
-  Synapse instances with the [`url_preview_enabled`](https://matrix-org.github.io/synapse/v1.61/usage/configuration/config_documentation.html#media-store) homeserver config option set to `true` are affected. URL previews of some web pages can lead to unbounded recursion, causing the request to either fail, or in some cases crash the running Synapse process.
-
-  Requesting URL previews requires authentication. Nevertheless, it is possible to exploit this maliciously, either by malicious users on the homeserver, or by remote users sending URLs that a local user's client may automatically request a URL preview for.
-
-  Homeservers with the `url_preview_enabled` configuration option set to `false` (the default) are unaffected. Instances with the `enable_media_repo` configuration option set to `false` are also unaffected, as this also disables URL preview functionality.
-
-  Fixed by [fa1308061802ac7b7d20e954ba7372c5ac292333](https://github.com/matrix-org/synapse/commit/fa1308061802ac7b7d20e954ba7372c5ac292333).
-
-Synapse 1.61.0 (2022-06-14)
-===========================
-
-This release removes support for the non-standard feature known both as 'groups' and as 'communities', which have been superseded by *Spaces*.
-
-See [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1610)
-for more details.
-
-Improved Documentation
-----------------------
-
-- Mention removed community/group worker endpoints in [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1610). Contributed by @olmari. ([\#13023](https://github.com/matrix-org/synapse/issues/13023))
-
-
-Synapse 1.61.0rc1 (2022-06-07)
-==============================
-
-Features
---------
-
-- Add new `media_retention` options to the homeserver config for routinely cleaning up non-recently accessed media. ([\#12732](https://github.com/matrix-org/synapse/issues/12732), [\#12972](https://github.com/matrix-org/synapse/issues/12972), [\#12977](https://github.com/matrix-org/synapse/issues/12977))
-- Experimental support for [MSC3772](https://github.com/matrix-org/matrix-spec-proposals/pull/3772): Push rule for mutually related events. ([\#12740](https://github.com/matrix-org/synapse/issues/12740), [\#12859](https://github.com/matrix-org/synapse/issues/12859))
-- Update to the `check_event_for_spam` module callback: Deprecate the current callback signature, replace it with a new signature that is both less ambiguous (replacing booleans with explicit allow/block) and more powerful (ability to return explicit error codes). ([\#12808](https://github.com/matrix-org/synapse/issues/12808))
-- Add storage and module API methods to get monthly active users (and their corresponding appservices) within an optionally specified time range. ([\#12838](https://github.com/matrix-org/synapse/issues/12838), [\#12917](https://github.com/matrix-org/synapse/issues/12917))
-- Support the new error code `ORG.MATRIX.MSC3823.USER_ACCOUNT_SUSPENDED` from [MSC3823](https://github.com/matrix-org/matrix-spec-proposals/pull/3823). ([\#12845](https://github.com/matrix-org/synapse/issues/12845), [\#12923](https://github.com/matrix-org/synapse/issues/12923))
-- Add a configurable background job to delete stale devices. ([\#12855](https://github.com/matrix-org/synapse/issues/12855))
-- Improve URL previews for pages with empty elements. ([\#12951](https://github.com/matrix-org/synapse/issues/12951))
-- Allow updating a user's password using the admin API without logging out their devices. Contributed by @jcgruenhage. ([\#12952](https://github.com/matrix-org/synapse/issues/12952))
-
-
-Bugfixes
---------
-
-- Always send an `access_token` in `/thirdparty/` requests to appservices, as required by the [Application Service API specification](https://spec.matrix.org/v1.1/application-service-api/#third-party-networks). ([\#12746](https://github.com/matrix-org/synapse/issues/12746))
-- Implement [MSC3816](https://github.com/matrix-org/matrix-spec-proposals/pull/3816): sending the root event in a thread should count as having 'participated' in it. ([\#12766](https://github.com/matrix-org/synapse/issues/12766))
-- Delete events from the `federation_inbound_events_staging` table when a room is purged through the admin API. ([\#12784](https://github.com/matrix-org/synapse/issues/12784))
-- Fix a bug where we did not correctly handle invalid device list updates over federation. Contributed by Carl Bordum Hansen. ([\#12829](https://github.com/matrix-org/synapse/issues/12829))
-- Fix a bug which allowed multiple async operations to access database locks concurrently. Contributed by @sumnerevans @ Beeper. ([\#12832](https://github.com/matrix-org/synapse/issues/12832))
-- Fix an issue introduced in Synapse 0.34 where the `/notifications` endpoint would only return notifications if a user registered at least one pusher. Contributed by Famedly. ([\#12840](https://github.com/matrix-org/synapse/issues/12840))
-- Fix a bug where servers using a Postgres database would fail to backfill from an insertion event when MSC2716 is enabled (`experimental_features.msc2716_enabled`). ([\#12843](https://github.com/matrix-org/synapse/issues/12843))
-- Fix [MSC3787](https://github.com/matrix-org/matrix-spec-proposals/pull/3787) rooms being omitted from room directory, room summary and space hierarchy responses. ([\#12858](https://github.com/matrix-org/synapse/issues/12858))
-- Fix a bug introduced in Synapse 1.54.0 which could sometimes cause exceptions when handling federated traffic. ([\#12877](https://github.com/matrix-org/synapse/issues/12877))
-- Fix a bug introduced in Synapse 1.59.0 which caused room deletion to fail with a foreign key violation error. ([\#12889](https://github.com/matrix-org/synapse/issues/12889))
-- Fix a long-standing bug which caused the `/messages` endpoint to return an incorrect `end` attribute when there were no more events. Contributed by @Vetchu. ([\#12903](https://github.com/matrix-org/synapse/issues/12903))
-- Fix a bug introduced in Synapse 1.58.0 where `/sync` would fail if the most recent event in a room was a redaction of an event that has since been purged. ([\#12905](https://github.com/matrix-org/synapse/issues/12905))
-- Fix a potential memory leak when generating thumbnails. ([\#12932](https://github.com/matrix-org/synapse/issues/12932))
-- Fix a long-standing bug where a URL preview would break if the image failed to download. ([\#12950](https://github.com/matrix-org/synapse/issues/12950))
-
-
-Improved Documentation
-----------------------
-
-- Fix typographical errors in documentation. ([\#12863](https://github.com/matrix-org/synapse/issues/12863))
-- Fix documentation incorrectly stating the `sendToDevice` endpoint can be directed at generic workers. Contributed by Nick @ Beeper. ([\#12867](https://github.com/matrix-org/synapse/issues/12867))
-
-
-Deprecations and Removals
--------------------------
-
-- Remove support for the non-standard groups/communities feature from Synapse. ([\#12553](https://github.com/matrix-org/synapse/issues/12553), [\#12558](https://github.com/matrix-org/synapse/issues/12558), [\#12563](https://github.com/matrix-org/synapse/issues/12563), [\#12895](https://github.com/matrix-org/synapse/issues/12895), [\#12897](https://github.com/matrix-org/synapse/issues/12897), [\#12899](https://github.com/matrix-org/synapse/issues/12899), [\#12900](https://github.com/matrix-org/synapse/issues/12900), [\#12936](https://github.com/matrix-org/synapse/issues/12936), [\#12966](https://github.com/matrix-org/synapse/issues/12966))
-- Remove contributed `kick_users.py` script. This is broken under Python 3, and is not added to the environment when `pip install`ing Synapse. ([\#12908](https://github.com/matrix-org/synapse/issues/12908))
-- Remove `contrib/jitsimeetbridge`. This was an unused experiment that hasn't been meaningfully changed since 2014. ([\#12909](https://github.com/matrix-org/synapse/issues/12909))
-- Remove unused `contrib/experiements/cursesio.py` script, which fails to run under Python 3. ([\#12910](https://github.com/matrix-org/synapse/issues/12910))
-- Remove unused `contrib/experiements/test_messaging.py` script. This fails to run on Python 3. ([\#12911](https://github.com/matrix-org/synapse/issues/12911))
-
-
-Internal Changes
-----------------
-
-- Test Synapse against Complement with workers. ([\#12810](https://github.com/matrix-org/synapse/issues/12810), [\#12933](https://github.com/matrix-org/synapse/issues/12933))
-- Reduce the amount of state we pull from the DB. ([\#12811](https://github.com/matrix-org/synapse/issues/12811), [\#12964](https://github.com/matrix-org/synapse/issues/12964))
-- Try other homeservers when re-syncing state for rooms with partial state. ([\#12812](https://github.com/matrix-org/synapse/issues/12812))
-- Resume state re-syncing for rooms with partial state after a Synapse restart. ([\#12813](https://github.com/matrix-org/synapse/issues/12813))
-- Remove Mutual Rooms' ([MSC2666](https://github.com/matrix-org/matrix-spec-proposals/pull/2666)) endpoint dependency on the User Directory. ([\#12836](https://github.com/matrix-org/synapse/issues/12836))
-- Experimental: expand `check_event_for_spam` with ability to return additional fields. This enables spam-checker implementations to experiment with mechanisms to give users more information about why they are blocked and whether any action is needed from them to be unblocked. ([\#12846](https://github.com/matrix-org/synapse/issues/12846))
-- Remove `dont_notify` from the `.m.rule.room.server_acl` rule. ([\#12849](https://github.com/matrix-org/synapse/issues/12849))
-- Remove the unstable `/hierarchy` endpoint from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946). ([\#12851](https://github.com/matrix-org/synapse/issues/12851))
-- Pull out less state when handling gaps in room DAG. ([\#12852](https://github.com/matrix-org/synapse/issues/12852), [\#12904](https://github.com/matrix-org/synapse/issues/12904))
-- Clean-up the push rules datastore. ([\#12856](https://github.com/matrix-org/synapse/issues/12856))
-- Correct a type annotation in the URL preview source code. ([\#12860](https://github.com/matrix-org/synapse/issues/12860))
-- Update `pyjwt` dependency to [2.4.0](https://github.com/jpadilla/pyjwt/releases/tag/2.4.0). ([\#12865](https://github.com/matrix-org/synapse/issues/12865))
-- Enable the `/account/whoami` endpoint on synapse worker processes. Contributed by Nick @ Beeper. ([\#12866](https://github.com/matrix-org/synapse/issues/12866))
-- Enable the `batch_send` endpoint on synapse worker processes. Contributed by Nick @ Beeper. ([\#12868](https://github.com/matrix-org/synapse/issues/12868))
-- Don't generate empty AS transactions when the AS is flagged as down. Contributed by Nick @ Beeper. ([\#12869](https://github.com/matrix-org/synapse/issues/12869))
-- Fix up the variable `state_store` naming. ([\#12871](https://github.com/matrix-org/synapse/issues/12871))
-- Faster room joins: when querying the current state of the room, wait for state to be populated. ([\#12872](https://github.com/matrix-org/synapse/issues/12872))
-- Avoid running queries which will never result in deletions. ([\#12879](https://github.com/matrix-org/synapse/issues/12879))
-- Use constants for EDU types. ([\#12884](https://github.com/matrix-org/synapse/issues/12884))
-- Reduce database load of `/sync` when presence is enabled. ([\#12885](https://github.com/matrix-org/synapse/issues/12885))
-- Refactor `have_seen_events` to reduce memory consumed when processing federation traffic. ([\#12886](https://github.com/matrix-org/synapse/issues/12886))
-- Refactor receipt linearization code. ([\#12888](https://github.com/matrix-org/synapse/issues/12888))
-- Add type annotations to `synapse.logging.opentracing`. ([\#12894](https://github.com/matrix-org/synapse/issues/12894))
-- Remove PyNaCl occurrences directly used in Synapse code. ([\#12902](https://github.com/matrix-org/synapse/issues/12902))
-- Bump types-jsonschema from 4.4.1 to 4.4.6. ([\#12912](https://github.com/matrix-org/synapse/issues/12912))
-- Rename storage classes. ([\#12913](https://github.com/matrix-org/synapse/issues/12913))
-- Preparation for database schema simplifications: stop reading from `event_edges.room_id`. ([\#12914](https://github.com/matrix-org/synapse/issues/12914))
-- Check if we are in a virtual environment before overriding the `PYTHONPATH` environment variable in the demo script. ([\#12916](https://github.com/matrix-org/synapse/issues/12916))
-- Improve the logging when signature checks on events fail. ([\#12925](https://github.com/matrix-org/synapse/issues/12925))
-
-
 Synapse 1.60.0 (2022-05-31)
 ===========================
 

changelog.d/12553.removal

@@ -0,0 +1 @@
+Remove support for the non-standard groups/communities feature from Synapse.

changelog.d/12558.removal

@@ -0,0 +1 @@
+Remove support for the non-standard groups/communities feature from Synapse.

changelog.d/12563.removal

@@ -0,0 +1 @@
+Remove support for the non-standard groups/communities feature from Synapse.

changelog.d/12732.feature

@@ -0,0 +1 @@
+Add new `media_retention` options to the homeserver config for routinely cleaning up non-recently accessed media.

changelog.d/12740.feature

@@ -0,0 +1 @@
+Experimental support for [MSC3772](https://github.com/matrix-org/matrix-spec-proposals/pull/3772): Push rule for mutually related events.

changelog.d/12746.bugfix

@@ -0,0 +1 @@
+Always send an `access_token` in `/thirdparty/` requests to appservices, as required by the [Matrix specification](https://spec.matrix.org/v1.1/application-service-api/#third-party-networks).

changelog.d/12766.bugfix

@@ -0,0 +1 @@
+Implement [MSC3816](https://github.com/matrix-org/matrix-spec-proposals/pull/3816): sending the root event in a thread should count as "participated" in it.

changelog.d/12784.bugfix

@@ -0,0 +1 @@
+Delete events from the `federation_inbound_events_staging` table when a room is purged through the admin API.

changelog.d/12808.feature

@@ -0,0 +1 @@
+Update to `check_event_for_spam`. Deprecate the current callback signature, replace it with a new signature that is both less ambiguous (replacing booleans with explicit allow/block) and more powerful (ability to return explicit error codes).

changelog.d/12810.misc

@@ -0,0 +1 @@
+Test Synapse against Complement with workers.

changelog.d/12811.misc

@@ -0,0 +1 @@
+Reduce the amount of state we pull from the DB.

changelog.d/12812.misc

@@ -0,0 +1 @@
+Try other homeservers when re-syncing state for rooms with partial state.

changelog.d/12813.misc

@@ -0,0 +1 @@
+Resume state re-syncing for rooms with partial state after a Synapse restart.

changelog.d/12829.bugfix

@@ -0,0 +1 @@
+Fix a bug where we did not correctly handle invalid device list updates over federation. Contributed by Carl Bordum Hansen.

changelog.d/12832.bugfix

@@ -0,0 +1 @@
+Fixed a bug which allowed multiple async operations to access database locks concurrently. Contributed by @sumnerevans @ Beeper.

changelog.d/12836.misc

@@ -0,0 +1 @@
+Remove Mutual Rooms ([MSC2666](https://github.com/matrix-org/matrix-spec-proposals/pull/2666)) endpoint dependency on the User Directory.

changelog.d/12838.feature

@@ -0,0 +1 @@
+Add storage and module API methods to get monthly active users (and their corresponding appservices) within an optionally specified time range.

changelog.d/12840.bugfix

@@ -0,0 +1 @@
+Fix an issue introduced in Synapse 0.34 where the `/notifications` endpoint would only return notifications if a user registered at least one pusher. Contributed by Famedly.

changelog.d/12843.bugfix

@@ -0,0 +1 @@
+Fix bug where servers using a Postgres database would fail to backfill from an insertion event when MSC2716 is enabled (`experimental_features.msc2716_enabled`).

changelog.d/12845.feature

@@ -0,0 +1 @@
+Support the new error code "ORG.MATRIX.MSC3823.USER_ACCOUNT_SUSPENDED" from [MSC3823](https://github.com/matrix-org/matrix-spec-proposals/pull/3823).

changelog.d/12846.misc

@@ -0,0 +1 @@
+Experimental: expand `check_event_for_spam` with ability to return additional fields. This enables spam-checker implementations to experiment with mechanisms to give users more information about why they are blocked and whether any action is needed from them to be unblocked.

changelog.d/12849.misc

@@ -0,0 +1 @@
+Remove `dont_notify` from the `.m.rule.room.server_acl` rule.

changelog.d/12851.misc

@@ -0,0 +1 @@
+Remove the unstable `/hierarchy` endpoint from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).

changelog.d/12852.misc

@@ -0,0 +1 @@
+Pull out less state when handling gaps in room DAG.

changelog.d/12855.feature

@@ -0,0 +1 @@
+Add a configurable background job to delete stale devices.

changelog.d/12856.misc

@@ -0,0 +1 @@
+Clean-up the push rules datastore.

changelog.d/12858.bugfix

@@ -0,0 +1 @@
+Fix [MSC3787](https://github.com/matrix-org/matrix-spec-proposals/pull/3787) rooms being omitted from room directory, room summary and space hierarchy responses.

changelog.d/12859.feature

@@ -0,0 +1 @@
+Experimental support for [MSC3772](https://github.com/matrix-org/matrix-spec-proposals/pull/3772): Push rule for mutually related events.

changelog.d/12860.misc

@@ -0,0 +1 @@
+Correct a type annotation in the URL preview source code.

changelog.d/12863.doc

@@ -0,0 +1 @@
+Fix typos in documentation.

changelog.d/12865.misc

@@ -0,0 +1 @@
+Update `pyjwt` dependency to [2.4.0](https://github.com/jpadilla/pyjwt/releases/tag/2.4.0).

changelog.d/12866.misc

@@ -0,0 +1 @@
+Enable the `/account/whoami` endpoint on synapse worker processes. Contributed by Nick @ Beeper.

changelog.d/12867.doc

@@ -0,0 +1 @@
+Fix documentation incorrectly stating the `sendToDevice` endpoint can be directed at generic workers. Contributed by Nick @ Beeper.

changelog.d/12868.misc

@@ -0,0 +1 @@
+Enable the `batch_send` endpoint on synapse worker processes. Contributed by Nick @ Beeper.

changelog.d/12869.misc

@@ -0,0 +1 @@
+Don't generate empty AS transactions when the AS is flagged as down. Contributed by Nick @ Beeper.

changelog.d/12871.misc

@@ -0,0 +1 @@
+Fix up the variable `state_store` naming.

changelog.d/12872.misc

@@ -0,0 +1 @@
+Faster room joins: when querying the current state of the room, wait for state to be populated.

changelog.d/12877.bugfix

@@ -0,0 +1 @@
+Fix a bug introduced in Synapse 1.54 which could sometimes cause exceptions when handling federated traffic.

changelog.d/12879.misc

@@ -0,0 +1 @@
+Avoid running queries which will never result in deletions.

changelog.d/12884.misc

@@ -0,0 +1 @@
+Use constants for EDU types.

changelog.d/12885.misc

@@ -0,0 +1 @@
+Reduce database load of `/sync` when presence is enabled.

changelog.d/12886.misc

@@ -0,0 +1 @@
+Refactor `have_seen_events` to reduce memory consumed when processing federation traffic.

changelog.d/12888.misc

@@ -0,0 +1 @@
+Refactor receipt linearization code.

changelog.d/12889.bugfix

@@ -0,0 +1 @@
+Fix a bug introduced in Synapse 1.59.0 which caused room deletion to fail with a foreign key violation.

changelog.d/12894.misc

@@ -0,0 +1 @@
+Add type annotations to `synapse.logging.opentracing`.

changelog.d/12895.removal

@@ -0,0 +1 @@
+Remove support for the non-standard groups/communities feature from Synapse.

changelog.d/12897.removal

@@ -0,0 +1 @@
+Remove support for the non-standard groups/communities feature from Synapse.

changelog.d/12899.removal

@@ -0,0 +1 @@
+Remove support for the non-standard groups/communities feature from Synapse.

changelog.d/12900.removal

@@ -0,0 +1 @@
+Remove support for the non-standard groups/communities feature from Synapse.

changelog.d/12902.misc

@@ -0,0 +1 @@
+Remove PyNaCl occurrences directly used in Synapse code.

changelog.d/12903.bugfix

@@ -0,0 +1 @@
+Fix a long-standing bug which caused the `/messages` endpoint to return an incorrect `end` attribute when there were no more events. Contributed by @Vetchu.

changelog.d/12904.misc

@@ -0,0 +1 @@
+Pull out less state when handling gaps in room DAG.

changelog.d/12905.bugfix

@@ -0,0 +1 @@
+Fix a bug introduced in Synapse 1.58.0 where `/sync` would fail if the most recent event in a room was a redaction of an event that has since been purged.

changelog.d/12908.removal

@@ -0,0 +1 @@
+Remove contributed `kick_users.py` script. This is broken under Python 3, and is not added to the environment when `pip install`ing Synapse.

changelog.d/12909.removal

@@ -0,0 +1 @@
+Remove `contrib/jitsimeetbridge`. This was an unused experiment that hasn't been meaningfully changed since 2014.

changelog.d/12910.removal

@@ -0,0 +1 @@
+Remove unused `contrib/experiements/cursesio.py` script, which fails to run under Python 3.

changelog.d/12911.removal

@@ -0,0 +1 @@
+Remove unused `contrib/experiements/test_messaging.py` script. This fails to run on Python 3.

changelog.d/12912.misc

@@ -0,0 +1 @@
+Bump types-jsonschema from 4.4.1 to 4.4.6.

changelog.d/12913.misc

@@ -0,0 +1 @@
+Rename storage classes.

changelog.d/12914.misc

@@ -0,0 +1 @@
+Preparation for database schema simplifications: stop reading from `event_edges.room_id`.

changelog.d/12916.misc

@@ -0,0 +1 @@
+Check if we are in a virtual environment before overriding the `PYTHONPATH` environment variable in the demo script.

changelog.d/12917.feature

@@ -0,0 +1 @@
+Add storage and module API methods to get monthly active users (and their corresponding appservices) within an optionally specified time range.

changelog.d/12923.feature

@@ -0,0 +1 @@
+Support the new error code "ORG.MATRIX.MSC3823.USER_ACCOUNT_SUSPENDED" from [MSC3823](https://github.com/matrix-org/matrix-spec-proposals/pull/3823).

changelog.d/12925.misc

@@ -0,0 +1 @@
+Improve the logging when signature checks on events fail.

changelog.d/12932.bugfix

@@ -0,0 +1 @@
+Fix potential memory leak when generating thumbnails.

changelog.d/12933.misc

@@ -0,0 +1 @@
+Test Synapse against Complement with workers.

changelog.d/12936.removal

@@ -0,0 +1 @@
+Remove support for the non-standard groups/communities feature from Synapse.

changelog.d/12950.bugfix

@@ -0,0 +1 @@
+Fix a long-standing bug where a URL preview would break if the image failed to download.

changelog.d/12951.feature

@@ -0,0 +1 @@
+Improve URL previews for pages with empty elements.

changelog.d/12952.feature

@@ -0,0 +1 @@
+Allow updating a user's password using the admin API without logging out their devices. Contributed by @jcgruenhage.

changelog.d/12964.misc

@@ -0,0 +1 @@
+Reduce the amount of state we pull from the DB.

changelog.d/12966.removal

@@ -0,0 +1 @@
+Remove support for the non-standard groups/communities feature from Synapse.

changelog.d/12974.bugfix

@@ -0,0 +1 @@
+Fix a bug introduced in v1.60.0 that causes synapse to fail to start when python module sqlite3 is not installed.

debian/changelog

@@ -1,21 +1,9 @@
-matrix-synapse-py3 (1.61.1) stable; urgency=medium
-
-  * New Synapse release 1.61.1.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 28 Jun 2022 14:33:46 +0100
-
-matrix-synapse-py3 (1.61.0) stable; urgency=medium
-
-  * New Synapse release 1.61.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 14 Jun 2022 11:44:19 +0100
-
-matrix-synapse-py3 (1.61.0~rc1) stable; urgency=medium
+matrix-synapse-py3 (1.61.0~rc1+nmu1) UNRELEASED; urgency=medium
 
+  * Non-maintainer upload.
   * Remove unused `jitsimeetbridge` experiment from `contrib` directory.
-  * New Synapse release 1.61.0rc1.
 
- -- Synapse Packaging team <packages@matrix.org>  Tue, 07 Jun 2022 12:42:31 +0100
+ -- Synapse Packaging team <packages@matrix.org>  Sun, 29 May 2022 14:44:45 +0100
 
 matrix-synapse-py3 (1.60.0) stable; urgency=medium
 

docs/upgrade.md

@@ -89,24 +89,6 @@ process, for example:
     dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
     ```
 
-# Upgrading to v1.61.0
-
-## Removal of deprecated community/groups
-
-This release of Synapse will remove deprecated community/groups from codebase.
-
-### Worker endpoints
-
-For those who have deployed workers, following worker endpoints will no longer
-exist and they can be removed from the reverse proxy configuration:
-
--   `^/_matrix/federation/v1/get_groups_publicised$`
--   `^/_matrix/client/(r0|v3|unstable)/joined_groups$`
--   `^/_matrix/client/(r0|v3|unstable)/publicised_groups$`
--   `^/_matrix/client/(r0|v3|unstable)/publicised_groups/`
--   `^/_matrix/federation/v1/groups/`
--   `^/_matrix/client/(r0|v3|unstable)/groups/`
-
 # Upgrading to v1.60.0
 
 ## Adding a new unique index to `state_group_edges` could fail if your database is corrupted

docs/usage/configuration/config_documentation.md

@@ -1583,12 +1583,6 @@ been accessed, the media's creation time is used instead. Both thumbnails
 and the original media will be removed. If either of these options are unset,
 then media of that type will not be purged.
 
-Local or cached remote media that has been
-[quarantined](../../admin_api/media_admin_api.md#quarantining-media-in-a-room)
-will not be deleted. Similarly, local media that has been marked as
-[protected from quarantine](../../admin_api/media_admin_api.md#protecting-media-from-being-quarantined)
-will not be deleted.
-
 Example configuration:
 ```yaml
 media_retention:

pyproject.toml

@@ -54,7 +54,7 @@ skip_gitignore = true
 
 [tool.poetry]
 name = "matrix-synapse"
-version = "1.61.1"
+version = "1.60.0"
 description = "Homeserver for the Matrix decentralised comms protocol"
 authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
 license = "Apache-2.0"

synapse/replication/slave/storage/_base.py

@@ -17,7 +17,7 @@ from typing import TYPE_CHECKING, Optional
 
 from synapse.storage.database import DatabasePool, LoggingDatabaseConnection
 from synapse.storage.databases.main.cache import CacheInvalidationWorkerStore
-from synapse.storage.engines import PostgresEngine
+from synapse.storage.engines.postgres import PostgresEngine
 from synapse.storage.util.id_generators import MultiWriterIdGenerator
 
 if TYPE_CHECKING:

synapse/rest/admin/media.py

@@ -83,7 +83,7 @@ class QuarantineMediaByUser(RestServlet):
         requester = await self.auth.get_user_by_req(request)
         await assert_user_is_admin(self.auth, requester.user)
 
-        logging.info("Quarantining media by user: %s", user_id)
+        logging.info("Quarantining local media by user: %s", user_id)
 
         # Quarantine all media this user has uploaded
         num_quarantined = await self.store.quarantine_media_ids_by_user(
@@ -112,7 +112,7 @@ class QuarantineMediaByID(RestServlet):
         requester = await self.auth.get_user_by_req(request)
         await assert_user_is_admin(self.auth, requester.user)
 
-        logging.info("Quarantining media by ID: %s/%s", server_name, media_id)
+        logging.info("Quarantining local media by ID: %s/%s", server_name, media_id)
 
         # Quarantine this media id
         await self.store.quarantine_media_by_id(
@@ -140,7 +140,9 @@ class UnquarantineMediaByID(RestServlet):
     ) -> Tuple[int, JsonDict]:
         await assert_requester_is_admin(self.auth, request)
 
-        logging.info("Remove from quarantine media by ID: %s/%s", server_name, media_id)
+        logging.info(
+            "Remove from quarantine local media by ID: %s/%s", server_name, media_id
+        )
 
         # Remove from quarantine this media id
         await self.store.quarantine_media_by_id(server_name, media_id, None)

synapse/rest/media/v1/media_repository.py

@@ -919,14 +919,10 @@ class MediaRepository:
             await self.delete_old_local_media(
                 before_ts=local_media_threshold_timestamp_ms,
                 keep_profiles=True,
-                delete_quarantined_media=False,
-                delete_protected_media=False,
             )
 
     async def delete_old_remote_media(self, before_ts: int) -> Dict[str, int]:
-        old_media = await self.store.get_remote_media_ids(
-            before_ts, include_quarantined_media=False
-        )
+        old_media = await self.store.get_remote_media_before(before_ts)
 
         deleted = 0
 
@@ -979,8 +975,6 @@ class MediaRepository:
         before_ts: int,
         size_gt: int = 0,
         keep_profiles: bool = True,
-        delete_quarantined_media: bool = False,
-        delete_protected_media: bool = False,
     ) -> Tuple[List[str], int]:
         """
         Delete local or remote media from this server by size and timestamp. Removes
@@ -988,22 +982,18 @@ class MediaRepository:
 
         Args:
             before_ts: Unix timestamp in ms.
-                Files that were last used before this timestamp will be deleted.
-            size_gt: Size of the media in bytes. Files that are larger will be deleted.
+                       Files that were last used before this timestamp will be deleted
+            size_gt: Size of the media in bytes. Files that are larger will be deleted
             keep_profiles: Switch to delete also files that are still used in image data
-                (e.g user profile, room avatar). If false these files will be deleted.
-            delete_quarantined_media: If True, media marked as quarantined will be deleted.
-            delete_protected_media: If True, media marked as protected will be deleted.
-
+                           (e.g user profile, room avatar)
+                           If false these files will be deleted
         Returns:
             A tuple of (list of deleted media IDs, total deleted media IDs).
         """
-        old_media = await self.store.get_local_media_ids(
+        old_media = await self.store.get_local_media_before(
             before_ts,
             size_gt,
             keep_profiles,
-            include_quarantined_media=delete_quarantined_media,
-            include_protected_media=delete_protected_media,
         )
         return await self._remove_local_media_from_disk(old_media)
 

synapse/rest/media/v1/preview_html.py

@@ -12,9 +12,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import codecs
+import itertools
 import logging
 import re
-from typing import TYPE_CHECKING, Dict, Generator, Iterable, List, Optional, Set, Union
+from typing import TYPE_CHECKING, Dict, Generator, Iterable, Optional, Set, Union
 
 if TYPE_CHECKING:
     from lxml import etree
@@ -275,7 +276,7 @@ def parse_html_description(tree: "etree.Element") -> Optional[str]:
 
     from lxml import etree
 
-    TAGS_TO_REMOVE = {
+    TAGS_TO_REMOVE = (
         "header",
         "nav",
         "aside",
@@ -290,42 +291,31 @@ def parse_html_description(tree: "etree.Element") -> Optional[str]:
         "img",
         "picture",
         etree.Comment,
-    }
+    )
 
     # Split all the text nodes into paragraphs (by splitting on new
     # lines)
     text_nodes = (
         re.sub(r"\s+", "\n", el).strip()
-        for el in _iterate_over_text(tree.find("body"), TAGS_TO_REMOVE)
+        for el in _iterate_over_text(tree.find("body"), *TAGS_TO_REMOVE)
     )
     return summarize_paragraphs(text_nodes)
 
 
 def _iterate_over_text(
-    tree: Optional["etree.Element"],
-    tags_to_ignore: Set[Union[str, "etree.Comment"]],
-    stack_limit: int = 1024,
+    tree: "etree.Element", *tags_to_ignore: Union[str, "etree.Comment"]
 ) -> Generator[str, None, None]:
     """Iterate over the tree returning text nodes in a depth first fashion,
     skipping text nodes inside certain tags.
-
-    Args:
-        tree: The parent element to iterate. Can be None if there isn't one.
-        tags_to_ignore: Set of tags to ignore
-        stack_limit: Maximum stack size limit for depth-first traversal.
-            Nodes will be dropped if this limit is hit, which may truncate the
-            textual result.
-            Intended to limit the maximum working memory when generating a preview.
     """
-
-    if tree is None:
-        return
-
-    # This is a stack whose items are elements to iterate over *or* strings
+    # This is basically a stack that we extend using itertools.chain.
+    # This will either consist of an element to iterate over *or* a string
     # to be returned.
-    elements: List[Union[str, "etree.Element"]] = [tree]
-    while elements:
-        el = elements.pop()
+    elements = iter([tree])
+    while True:
+        el = next(elements, None)
+        if el is None:
+            return
 
         if isinstance(el, str):
             yield el
@@ -339,22 +329,17 @@ def _iterate_over_text(
             if el.text:
                 yield el.text
 
-            # We add to the stack all the element's children, interspersed with
-            # each child's tail text (if it exists).
-            #
-            # We iterate in reverse order so that earlier pieces of text appear
-            # closer to the top of the stack.
-            for child in el.iterchildren(reversed=True):
-                if len(elements) > stack_limit:
-                    # We've hit our limit for working memory
-                    break
-
-                if child.tail:
-                    # The tail text of a node is text that comes *after* the node,
-                    # so we always include it even if we ignore the child node.
-                    elements.append(child.tail)
-
-                elements.append(child)
+            # We add to the stack all the elements children, interspersed with
+            # each child's tail text (if it exists). The tail text of a node
+            # is text that comes *after* the node, so we always include it even
+            # if we ignore the child node.
+            elements = itertools.chain(
+                itertools.chain.from_iterable(  # Basically a flatmap
+                    [child, child.tail] if child.tail else [child]
+                    for child in el.iterchildren()
+                ),
+                elements,
+            )
 
 
 def summarize_paragraphs(

synapse/storage/background_updates.py

@@ -628,7 +628,7 @@ class BackgroundUpdater:
                 logger.debug("[SQL] %s", sql)
                 c.execute(sql)
 
-        if isinstance(self.db_pool.engine, engines.PostgresEngine):
+        if isinstance(self.db_pool.engine, engines.postgres.PostgresEngine):
             runner: Optional[Callable[[Connection], None]] = create_index_psql
         elif psql_only:
             runner = None

synapse/storage/database.py

@@ -55,7 +55,9 @@ from synapse.logging.context import (
 from synapse.metrics import register_threadpool
 from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.storage.background_updates import BackgroundUpdater
-from synapse.storage.engines import BaseDatabaseEngine, PostgresEngine, Sqlite3Engine
+from synapse.storage.engines import BaseDatabaseEngine
+from synapse.storage.engines.postgres import PostgresEngine
+from synapse.storage.engines.sqlite import Sqlite3Engine
 from synapse.storage.types import Connection, Cursor
 from synapse.util.async_helpers import delay_cancellation
 from synapse.util.iterutils import batch_iter

synapse/storage/databases/main/__init__.py

@@ -24,7 +24,8 @@ from synapse.storage.database import (
     LoggingTransaction,
 )
 from synapse.storage.databases.main.stats import UserSortOrder
-from synapse.storage.engines import BaseDatabaseEngine, PostgresEngine
+from synapse.storage.engines import BaseDatabaseEngine
+from synapse.storage.engines.postgres import PostgresEngine
 from synapse.storage.types import Cursor
 from synapse.storage.util.id_generators import MultiWriterIdGenerator, StreamIdGenerator
 from synapse.types import JsonDict, get_domain_from_id

synapse/storage/databases/main/account_data.py

@@ -37,7 +37,7 @@ from synapse.storage.database import (
 )
 from synapse.storage.databases.main.cache import CacheInvalidationWorkerStore
 from synapse.storage.databases.main.push_rule import PushRulesWorkerStore
-from synapse.storage.engines import PostgresEngine
+from synapse.storage.engines.postgres import PostgresEngine
 from synapse.storage.util.id_generators import (
     AbstractStreamIdGenerator,
     AbstractStreamIdTracker,

synapse/storage/databases/main/cache.py

@@ -31,7 +31,7 @@ from synapse.storage.database import (
     LoggingDatabaseConnection,
     LoggingTransaction,
 )
-from synapse.storage.engines import PostgresEngine
+from synapse.storage.engines.postgres import PostgresEngine
 from synapse.util.caches.descriptors import _CachedFunction
 from synapse.util.iterutils import batch_iter
 

synapse/storage/databases/main/deviceinbox.py

@@ -36,7 +36,7 @@ from synapse.storage.database import (
     LoggingTransaction,
     make_in_list_sql_clause,
 )
-from synapse.storage.engines import PostgresEngine
+from synapse.storage.engines.postgres import PostgresEngine
 from synapse.storage.util.id_generators import (
     AbstractStreamIdGenerator,
     MultiWriterIdGenerator,

synapse/storage/databases/main/end_to_end_keys.py

@@ -42,7 +42,7 @@ from synapse.storage.database import (
     make_in_list_sql_clause,
 )
 from synapse.storage.databases.main.cache import CacheInvalidationWorkerStore
-from synapse.storage.engines import PostgresEngine
+from synapse.storage.engines.postgres import PostgresEngine
 from synapse.storage.util.id_generators import StreamIdGenerator
 from synapse.types import JsonDict
 from synapse.util import json_encoder

synapse/storage/databases/main/event_federation.py

@@ -42,7 +42,7 @@ from synapse.storage.database import (
 )
 from synapse.storage.databases.main.events_worker import EventsWorkerStore
 from synapse.storage.databases.main.signatures import SignatureWorkerStore
-from synapse.storage.engines import PostgresEngine
+from synapse.storage.engines.postgres import PostgresEngine
 from synapse.types import JsonDict
 from synapse.util import json_encoder
 from synapse.util.caches.descriptors import cached

synapse/storage/databases/main/events_worker.py

@@ -67,7 +67,7 @@ from synapse.storage.database import (
     LoggingDatabaseConnection,
     LoggingTransaction,
 )
-from synapse.storage.engines import PostgresEngine
+from synapse.storage.engines.postgres import PostgresEngine
 from synapse.storage.types import Cursor
 from synapse.storage.util.id_generators import (
     AbstractStreamIdTracker,

synapse/storage/databases/main/media_repository.py

@@ -251,36 +251,12 @@ class MediaRepositoryStore(MediaRepositoryBackgroundUpdateStore):
             "get_local_media_by_user_paginate_txn", get_local_media_by_user_paginate_txn
         )
 
-    async def get_local_media_ids(
+    async def get_local_media_before(
         self,
         before_ts: int,
         size_gt: int,
         keep_profiles: bool,
-        include_quarantined_media: bool,
-        include_protected_media: bool,
     ) -> List[str]:
-        """
-        Retrieve a list of media IDs from the local media store.
-
-        Args:
-            before_ts: Only retrieve IDs from media that was either last accessed
-                (or if never accessed, created) before the given UNIX timestamp in ms.
-            size_gt: Only retrieve IDs from media that has a size (in bytes) greater than
-                the given integer.
-            keep_profiles: If True, exclude media IDs from the results that are used in the
-                following situations:
-                    * global profile user avatar
-                    * per-room profile user avatar
-                    * room avatar
-                    * a user's avatar in the user directory
-            include_quarantined_media: If False, exclude media IDs from the results that have
-                been marked as quarantined.
-            include_protected_media: If False, exclude media IDs from the results that have
-                been marked as protected from quarantine.
-
-        Returns:
-            A list of local media IDs.
-        """
 
         # to find files that have never been accessed (last_access_ts IS NULL)
         # compare with `created_ts`
@@ -318,24 +294,12 @@ class MediaRepositoryStore(MediaRepositoryBackgroundUpdateStore):
             )
             sql += sql_keep
 
-        if include_quarantined_media is False:
-            # Do not include media that has been quarantined
-            sql += """
-                AND quarantined_by IS NULL
-            """
-
-        if include_protected_media is False:
-            # Do not include media that has been protected from quarantine
-            sql += """
-                AND NOT safe_from_quarantine
-            """
-
-        def _get_local_media_ids_txn(txn: LoggingTransaction) -> List[str]:
+        def _get_local_media_before_txn(txn: LoggingTransaction) -> List[str]:
             txn.execute(sql, (before_ts, before_ts, size_gt))
             return [row[0] for row in txn]
 
         return await self.db_pool.runInteraction(
-            "get_local_media_ids", _get_local_media_ids_txn
+            "get_local_media_before", _get_local_media_before_txn
         )
 
     async def store_local_media(
@@ -635,37 +599,15 @@ class MediaRepositoryStore(MediaRepositoryBackgroundUpdateStore):
             desc="store_remote_media_thumbnail",
         )
 
-    async def get_remote_media_ids(
-        self, before_ts: int, include_quarantined_media: bool
-    ) -> List[Dict[str, str]]:
-        """
-        Retrieve a list of server name, media ID tuples from the remote media cache.
-
-        Args:
-            before_ts: Only retrieve IDs from media that was either last accessed
-                (or if never accessed, created) before the given UNIX timestamp in ms.
-            include_quarantined_media: If False, exclude media IDs from the results that have
-                been marked as quarantined.
-
-        Returns:
-            A list of tuples containing:
-                * The server name of homeserver where the media originates from,
-                * The ID of the media.
-        """
+    async def get_remote_media_before(self, before_ts: int) -> List[Dict[str, str]]:
         sql = (
             "SELECT media_origin, media_id, filesystem_id"
             " FROM remote_media_cache"
             " WHERE last_access_ts < ?"
         )
 
-        if include_quarantined_media is False:
-            # Only include media that has not been quarantined
-            sql += """
-            AND quarantined_by IS NULL
-            """
-
         return await self.db_pool.execute(
-            "get_remote_media_ids", self.db_pool.cursor_to_dict, sql, before_ts
+            "get_remote_media_before", self.db_pool.cursor_to_dict, sql, before_ts
         )
 
     async def delete_remote_media(self, media_origin: str, media_id: str) -> None:

synapse/storage/databases/main/presence.py

@@ -23,7 +23,7 @@ from synapse.storage.database import (
     LoggingTransaction,
 )
 from synapse.storage.databases.main.cache import CacheInvalidationWorkerStore
-from synapse.storage.engines import PostgresEngine
+from synapse.storage.engines.postgres import PostgresEngine
 from synapse.storage.types import Connection
 from synapse.storage.util.id_generators import (
     AbstractStreamIdGenerator,

synapse/storage/databases/main/push_rule.py

@@ -31,7 +31,8 @@ from synapse.storage.databases.main.events_worker import EventsWorkerStore
 from synapse.storage.databases.main.pusher import PusherWorkerStore
 from synapse.storage.databases.main.receipts import ReceiptsWorkerStore
 from synapse.storage.databases.main.roommember import RoomMemberWorkerStore
-from synapse.storage.engines import PostgresEngine, Sqlite3Engine
+from synapse.storage.engines.postgres import PostgresEngine
+from synapse.storage.engines.sqlite import Sqlite3Engine
 from synapse.storage.push_rule import InconsistentRuleException, RuleNotFoundException
 from synapse.storage.util.id_generators import (
     AbstractStreamIdGenerator,

synapse/storage/databases/main/receipts.py

@@ -35,7 +35,7 @@ from synapse.storage.database import (
     LoggingDatabaseConnection,
     LoggingTransaction,
 )
-from synapse.storage.engines import PostgresEngine
+from synapse.storage.engines.postgres import PostgresEngine
 from synapse.storage.util.id_generators import (
     AbstractStreamIdTracker,
     MultiWriterIdGenerator,

synapse/storage/databases/main/relations.py

@@ -33,7 +33,7 @@ from synapse.events import EventBase
 from synapse.storage._base import SQLBaseStore
 from synapse.storage.database import LoggingTransaction, make_in_list_sql_clause
 from synapse.storage.databases.main.stream import generate_pagination_where_clause
-from synapse.storage.engines import PostgresEngine
+from synapse.storage.engines.postgres import PostgresEngine
 from synapse.types import JsonDict, RoomStreamToken, StreamKeyType, StreamToken
 from synapse.util.caches.descriptors import cached, cachedList
 

synapse/storage/databases/main/roommember.py

@@ -45,7 +45,7 @@ from synapse.storage.database import (
 )
 from synapse.storage.databases.main.cache import CacheInvalidationWorkerStore
 from synapse.storage.databases.main.events_worker import EventsWorkerStore
-from synapse.storage.engines import Sqlite3Engine
+from synapse.storage.engines.sqlite import Sqlite3Engine
 from synapse.storage.roommember import (
     GetRoomsForUserWithStreamOrdering,
     MemberSummary,

synapse/storage/databases/main/search.py

@@ -27,7 +27,9 @@ from synapse.storage.database import (
     LoggingTransaction,
 )
 from synapse.storage.databases.main.events_worker import EventRedactBehaviour
-from synapse.storage.engines import BaseDatabaseEngine, PostgresEngine, Sqlite3Engine
+from synapse.storage.engines import BaseDatabaseEngine
+from synapse.storage.engines.postgres import PostgresEngine
+from synapse.storage.engines.sqlite import Sqlite3Engine
 from synapse.types import JsonDict
 
 if TYPE_CHECKING:

synapse/storage/databases/main/stream.py

@@ -64,7 +64,8 @@ from synapse.storage.database import (
     make_in_list_sql_clause,
 )
 from synapse.storage.databases.main.events_worker import EventsWorkerStore
-from synapse.storage.engines import BaseDatabaseEngine, PostgresEngine
+from synapse.storage.engines import BaseDatabaseEngine
+from synapse.storage.engines.postgres import PostgresEngine
 from synapse.storage.util.id_generators import MultiWriterIdGenerator
 from synapse.types import PersistedEventPosition, RoomStreamToken
 from synapse.util.caches.descriptors import cached

synapse/storage/databases/main/user_directory.py

@@ -42,7 +42,8 @@ from synapse.storage.database import (
 )
 from synapse.storage.databases.main.state import StateFilter
 from synapse.storage.databases.main.state_deltas import StateDeltasStore
-from synapse.storage.engines import PostgresEngine, Sqlite3Engine
+from synapse.storage.engines.postgres import PostgresEngine
+from synapse.storage.engines.sqlite import Sqlite3Engine
 from synapse.types import (
     JsonDict,
     UserProfile,