changelog

Add a method for Synapse modules to carry out HTTP federation requests
Provides a fairly basic interface for Synapse modules to complete HTTP federation requests. Custom error types were used in order to prevent stabilising any of the internal MatrixFederationHttpClient code.
2023-02-23 23:40:07 +00:00 · 2023-02-23 23:40:07 +00:00 · 2023-02-23 13:25:56 -05:00 · 2023-02-23 13:14:17 -05:00 · 2023-02-23 12:54:03 -05:00 · 2023-02-22 15:30:41 -05:00
153 changed files with 553 additions and 618 deletions
@@ -21,4 +21,8 @@ aff1eb7c671b0a3813407321d2702ec46c71fa56
 0a00b7ff14890987f09112a2ae696c61001e6cf1

 # Convert tests/rest/admin/test_room.py to unix file endings (#7953).
-c4268e3da64f1abb5b31deaeb5769adb6510c0a7
+c4268e3da64f1abb5b31deaeb5769adb6510c0a7
+
+# Update black to 23.1.0 (#15103)
+9bb2eac71962970d02842bca441f4bcdbbf93a11
+
@@ -1,12 +1,3 @@
-Synapse 1.78.0 (2023-02-28)
-===========================
-
-Bugfixes
--------
-
- Fix a bug introduced in Synapse 1.76 where 5s delays would occasionally occur in deployments using workers. ([\#15150](https://github.com/matrix-org/synapse/issues/15150))
-
-
 Synapse 1.78.0rc1 (2023-02-21)
 ==============================

@@ -34,14 +34,6 @@ additional-css = [
    "docs/website_files/table-of-contents.css",
    "docs/website_files/remove-nav-buttons.css",
    "docs/website_files/indent-section-headers.css",
-    "docs/website_files/version-picker.css",
 ]
-additional-js = [
-    "docs/website_files/table-of-contents.js",
-    "docs/website_files/version-picker.js",
-    "docs/website_files/version.js",
-]
-theme = "docs/website_files/theme"
-
-[preprocessor.schema_versions]
-command = "./scripts-dev/schema_versions.py"
+additional-js = ["docs/website_files/table-of-contents.js"]
+theme = "docs/website_files/theme"
@@ -0,0 +1 @@
+Clarify which worker processes the ThirdPartyRules' [`on_new_event`](https://matrix-org.github.io/synapse/v1.78/modules/third_party_rules_callbacks.html#on_new_event) module API callback runs on.
@@ -0,0 +1 @@
+Remove the unspecced `room_alias` field from the [`/createRoom`](https://spec.matrix.org/v1.6/client-server-api/#post_matrixclientv3createroom) response.
@@ -0,0 +1 @@
+Refactor writing json data in `FileExfiltrationWriter`.
@@ -0,0 +1 @@
+Bump black from 22.12.0 to 23.1.0.
@@ -0,0 +1 @@
+Add media information to the command line [user data export tool](https://matrix-org.github.io/synapse/v1.79/usage/administration/admin_faq.html#how-can-i-export-user-data).
@@ -0,0 +1 @@
+Document using [Shibboleth](https://www.shibboleth.net/) as an OpenID Provider.
@@ -0,0 +1 @@
+Tighten the login ratelimit defaults.
@@ -0,0 +1 @@
+Correct reference to `federation_verify_certificates` in configuration documentation.
@@ -0,0 +1 @@
+Add a new `send_federation_http_request` method to the Module API to allow Synapse modules to make matrix federation requests over HTTP.
@@ -1,9 +1,3 @@
-matrix-synapse-py3 (1.78.0) stable; urgency=medium
-
-  * New Synapse release 1.78.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 28 Feb 2023 08:56:03 -0800
-
 matrix-synapse-py3 (1.78.0~rc1) stable; urgency=medium

  * Add `matrix-org-archive-keyring` package as recommended.
@@ -146,6 +146,9 @@ Note that this callback is called when the event has already been processed and
 into the room, which means this callback cannot be used to deny persisting the event. To
 deny an incoming event, see [`check_event_for_spam`](spam_checker_callbacks.md#check_event_for_spam) instead.

+For any given event, this callback will be called on every worker process, even if that worker will not end up
+acting on that event. This callback will not be called for events that are marked as rejected.
+
 If multiple modules implement this callback, Synapse runs them all in order.

 ### `check_can_shutdown_room`
@@ -590,6 +590,47 @@ oidc_providers:

 Note that the fields `client_id` and `client_secret` are taken from the CURL response above.

+### Shibboleth with OIDC Plugin
+
+[Shibboleth](https://www.shibboleth.net/) is an open Standard IdP solution widely used by Universities.
+
+1. Shibboleth needs the [OIDC Plugin](https://shibboleth.atlassian.net/wiki/spaces/IDPPLUGINS/pages/1376878976/OIDC+OP) installed and working correctly.
+2. Create a new config on the IdP Side, ensure that the `client_id` and `client_secret`
+   are randomly generated data.
+```json
+{
+    "client_id": "SOME-CLIENT-ID",
+    "client_secret": "SOME-SUPER-SECRET-SECRET",
+    "response_types": ["code"],
+    "grant_types": ["authorization_code"],
+    "scope": "openid profile email",
+    "redirect_uris": ["https://[synapse public baseurl]/_synapse/client/oidc/callback"]
+}
+```
+
+Synapse config:
+
+```yaml
+oidc_providers:
+  # Shibboleth IDP
+  #
+  - idp_id: shibboleth
+    idp_name: "Shibboleth Login"
+    discover: true
+    issuer: "https://YOUR-IDP-URL.TLD"
+    client_id: "YOUR_CLIENT_ID"
+    client_secret: "YOUR-CLIENT-SECRECT-FROM-YOUR-IDP"
+    scopes: ["openid", "profile", "email"]
+    allow_existing_users: true
+    user_profile_method: "userinfo_endpoint"
+    user_mapping_provider:
+      config:
+        subject_claim: "sub"
+        localpart_template: "{{ user.sub.split('@')[0] }}"
+        display_name_template: "{{ user.name }}"
+        email_template: "{{ user.email }}"
+```
+
 ### Twitch

 1. Setup a developer account on [Twitch](https://dev.twitch.tv/)
@@ -70,13 +70,55 @@ output-directory
 │       ├───state
 │       ├───invite_state
 │       └───knock_state
-└───user_data
-    ├───account_data
-    │   ├───global
-    │   └───<room_id>
-    ├───connections
-    ├───devices
-    └───profile
+├───user_data
+│   ├───account_data
+│   │   ├───global
+│   │   └───<room_id>
+│   ├───connections
+│   ├───devices
+│   └───profile
+└───media_ids
+    └───<media_id>
+```
+
+The `media_ids` folder contains only the metadata of the media uploaded by the user.
+It does not contain the media itself.
+Furthermore, only the `media_ids` that Synapse manages itself are exported.
+If another media repository (e.g. [matrix-media-repo](https://github.com/turt2live/matrix-media-repo))
+is used, the data must be exported separately.
+
+With the `media_ids` the media files can be downloaded.
+Media that have been sent in encrypted rooms are only retrieved in encrypted form.
+The following script can help with download the media files:
+
+```bash
+#!/usr/bin/env bash
+
+# Parameters
+#
+#   source_directory: Directory which contains the export with the media_ids.
+#   target_directory: Directory into which all files are to be downloaded.
+#   repository_url: Address of the media repository resp. media worker.
+#   serverName: Name of the server (`server_name` from homeserver.yaml).
+#
+#   Example:
+#       ./download_media.sh /tmp/export_data/media_ids/ /tmp/export_data/media_files/ http://localhost:8008 matrix.example.com
+
+source_directory=$1
+target_directory=$2
+repository_url=$3
+serverName=$4
+
+mkdir -p $target_directory
+
+for file in $source_directory/*; do
+    filename=$(basename ${file})
+    url=$repository_url/_matrix/media/v3/download/$serverName/$filename
+    echo "Downloading $filename - $url"
+    if ! wget -o /dev/null -P $target_directory $url; then
+        echo "Could not download $filename"
+    fi
+done
 ```

 Manually resetting passwords
@@ -87,7 +129,7 @@ can reset a user's password using the [admin API](../../admin_api/user_admin_api

 I have a problem with my server. Can I just delete my database and start again?
 ---
-Deleting your database is unlikely to make anything better. 
+Deleting your database is unlikely to make anything better.

 It's easy to make the mistake of thinking that you can start again from a clean
 slate by dropping your database, but things don't work like that in a federated
@@ -102,7 +144,7 @@ Come and seek help in https://matrix.to/#/#synapse:matrix.org.

 There are two exceptions when it might be sensible to delete your database and start again:
 * You have *never* joined any rooms which are federated with other servers. For
-instance, a local deployment which the outside world can't talk to. 
+instance, a local deployment which the outside world can't talk to.
 * You are changing the `server_name` in the homeserver configuration. In effect
 this makes your server a completely new one from the point of view of the network,
 so in this case it makes sense to start with a clean database.
@@ -115,7 +157,7 @@ Using the following curl command:
 curl -H 'Authorization: Bearer <access-token>' -X DELETE https://matrix.org/_matrix/client/r0/directory/room/<room-alias>
 ```
 `<access-token>` - can be obtained in riot by looking in the riot settings, down the bottom is:
-Access Token:\<click to reveal\> 
+Access Token:\<click to reveal\>

 `<room-alias>` - the room alias, eg. #my_room:matrix.org this possibly needs to be URL encoded also, for example  %23my_room%3Amatrix.org

@@ -152,13 +194,13 @@ What are the biggest rooms on my server?
 ---

 ```sql
-SELECT s.canonical_alias, g.room_id, count(*) AS num_rows 
-FROM 
-  state_groups_state AS g, 
-  room_stats_state AS s 
-WHERE g.room_id = s.room_id 
+SELECT s.canonical_alias, g.room_id, count(*) AS num_rows
+FROM
+  state_groups_state AS g,
+  room_stats_state AS s
+WHERE g.room_id = s.room_id
 GROUP BY s.canonical_alias, g.room_id
-ORDER BY num_rows desc 
+ORDER BY num_rows desc
 LIMIT 10;
 ```

@@ -1105,7 +1105,7 @@ This setting should only be used in very specific cases, such as
 federation over Tor hidden services and similar. For private networks
 of homeservers, you likely want to use a private CA instead.

-Only effective if `federation_verify_certicates` is `true`.
+Only effective if `federation_verify_certificates` is `true`.

 Example configuration:
 ```yaml
@@ -1518,11 +1518,11 @@ rc_registration_token_validity:

 This option specifies several limits for login:
 * `address` ratelimits login requests based on the client's IP
-      address. Defaults to `per_second: 0.17`, `burst_count: 3`.
+      address. Defaults to `per_second: 0.003`, `burst_count: 5`.

 * `account` ratelimits login requests based on the account the
-  client is attempting to log into. Defaults to `per_second: 0.17`,
-  `burst_count: 3`.
+  client is attempting to log into. Defaults to `per_second: 0.03`,
+  `burst_count: 5`.

 * `failed_attempts` ratelimits login requests based on the account the
  client is attempting to log into, based on the amount of failed login
@@ -24,11 +24,6 @@ Finally, we also stylise the chapter titles in the left sidebar by indenting the
 slightly so that they are more visually distinguishable from the section headers
 (the bold titles). This is done through the `indent-section-headers.css` file.

-In addition to these modifications, we have added a version picker to the documentation.
-Users can switch between documentations for different versions of Synapse.
-This functionality was implemented through the `version-picker.js` and
-`version-picker.css` files.
-
 More information can be found in mdbook's official documentation for
 [injecting page JS/CSS](https://rust-lang.github.io/mdBook/format/config.html)
 and
@@ -131,18 +131,6 @@
                            <i class="fa fa-search"></i>
                        </button>
                        {{/if}}
-                        <div class="version-picker">
-                            <div class="dropdown">
-                                <div class="select">
-                                    <span></span>
-                                    <i class="fa fa-chevron-down"></i>
-                                </div>
-                                <input type="hidden" name="version">
-                                <ul class="dropdown-menu">
-                                    <!-- Versions will be added dynamically in version-picker.js -->
-                                </ul>
-                            </div>
-                        </div>      
                    </div>

                    <h1 class="menu-title">{{ book_title }}</h1>
@@ -321,4 +309,4 @@
        {{/if}}

    </body>
-</html>
+</html>
@@ -1,78 +0,0 @@
-.version-picker {
-    display: flex;
-    align-items: center;
-}
-
-.version-picker .dropdown {
-    width: 130px;
-    max-height: 29px;
-    margin-left: 10px;
-    display: inline-block;
-    border-radius: 4px;
-    border: 1px solid var(--theme-popup-border);
-    position: relative;
-    font-size: 13px;
-    color: var(--fg);
-    height: 100%;
-    text-align: left;
-}
-.version-picker .dropdown .select {
-    cursor: pointer;
-    display: block;
-    padding: 5px 2px 5px 15px;
-}
-.version-picker .dropdown .select > i {
-    font-size: 10px;
-    color: var(--fg);
-    cursor: pointer;
-    float: right;
-    line-height: 20px !important;
-}
-.version-picker .dropdown:hover {
-    border: 1px solid var(--theme-popup-border);
-}
-.version-picker .dropdown:active {
-    background-color: var(--theme-popup-bg);
-}
-.version-picker .dropdown.active:hover,
-.version-picker .dropdown.active {
-    border: 1px solid var(--theme-popup-border);
-    border-radius: 2px 2px 0 0;
-    background-color: var(--theme-popup-bg);
-}
-.version-picker .dropdown.active .select > i {
-    transform: rotate(-180deg);
-}
-.version-picker .dropdown .dropdown-menu {
-    position: absolute;
-    background-color: var(--theme-popup-bg);
-    width: 100%;
-    left: -1px;
-    right: 1px;
-    margin-top: 1px;
-    border: 1px solid var(--theme-popup-border);
-    border-radius: 0 0 4px 4px;
-    overflow: hidden;
-    display: none;
-    max-height: 300px;
-    overflow-y: auto;
-    z-index: 9;
-}
-.version-picker .dropdown .dropdown-menu li {
-    font-size: 12px;
-    padding: 6px 20px;
-    cursor: pointer;
-} 
-.version-picker .dropdown .dropdown-menu {
-    padding: 0;
-    list-style: none;
-}
-.version-picker .dropdown .dropdown-menu li:hover {
-    background-color: var(--theme-hover);
-}
-.version-picker .dropdown .dropdown-menu li.active::before {
-    display: inline-block;
-    content: "✓";
-    margin-inline-start: -14px;
-    width: 14px;
-}
@@ -1,127 +0,0 @@
-
-const dropdown = document.querySelector('.version-picker .dropdown');
-const dropdownMenu = dropdown.querySelector('.dropdown-menu');
-
-fetchVersions(dropdown, dropdownMenu).then(() => {
-    initializeVersionDropdown(dropdown, dropdownMenu);
-});
-
-/**
- * Initialize the dropdown functionality for version selection.
- * 
- * @param {Element} dropdown - The dropdown element.
- * @param {Element} dropdownMenu - The dropdown menu element.
- */
-function initializeVersionDropdown(dropdown, dropdownMenu) {
-    // Toggle the dropdown menu on click
-    dropdown.addEventListener('click', function () {
-        this.setAttribute('tabindex', 1);
-        this.classList.toggle('active');
-        dropdownMenu.style.display = (dropdownMenu.style.display === 'block') ? 'none' : 'block';
-    });
-  
-    // Remove the 'active' class and hide the dropdown menu on focusout
-    dropdown.addEventListener('focusout', function () {
-        this.classList.remove('active');
-        dropdownMenu.style.display = 'none';
-    });
-  
-    // Handle item selection within the dropdown menu
-    const dropdownMenuItems = dropdownMenu.querySelectorAll('li');    
-    dropdownMenuItems.forEach(function (item) {
-        item.addEventListener('click', function () {
-            dropdownMenuItems.forEach(function (item) {
-                item.classList.remove('active');
-            });
-            this.classList.add('active');
-            dropdown.querySelector('span').textContent = this.textContent;
-            dropdown.querySelector('input').value = this.getAttribute('id');
-
-            window.location.href = changeVersion(window.location.href, this.textContent);
-        });
-    });
-};
-
-/**
- * This function fetches the available versions from a GitHub repository
- * and inserts them into the version picker.
- * 
- * @param {Element} dropdown - The dropdown element.
- * @param {Element} dropdownMenu - The dropdown menu element.
- * @returns {Promise<Array<string>>} A promise that resolves with an array of available versions.
- */
-function fetchVersions(dropdown, dropdownMenu) {
-    return new Promise((resolve, reject) => {
-        window.addEventListener("load", () => {
-
-            fetch("https://api.github.com/repos/matrix-org/synapse/git/trees/gh-pages", {
-                cache: "force-cache",
-            }).then(res => 
-                res.json()
-            ).then(resObject => {
-                const excluded = ['dev-docs', 'v1.91.0', 'v1.80.0', 'v1.69.0'];
-                const tree = resObject.tree.filter(item => item.type === "tree" && !excluded.includes(item.path));
-                const versions = tree.map(item => item.path).sort(sortVersions);
-
-                // Create a list of <li> items for versions
-                versions.forEach((version) => {
-                    const li = document.createElement("li");
-                    li.textContent = version;
-                    li.id = version;
-    
-                    if (window.SYNAPSE_VERSION === version) {
-                        li.classList.add('active');
-                        dropdown.querySelector('span').textContent = version;
-                        dropdown.querySelector('input').value = version;
-                    }
-    
-                    dropdownMenu.appendChild(li);
-                });
-
-                resolve(versions);
-
-            }).catch(ex => {
-                console.error("Failed to fetch version data", ex);
-                reject(ex);
-            })
-        });
-    });
-}
-
-/**
- * Custom sorting function to sort an array of version strings.
- *
- * @param {string} a - The first version string to compare.
- * @param {string} b - The second version string to compare.
- * @returns {number} - A negative number if a should come before b, a positive number if b should come before a, or 0 if they are equal.
- */
-function sortVersions(a, b) {
-    // Put 'develop' and 'latest' at the top
-    if (a === 'develop' || a === 'latest') return -1;
-    if (b === 'develop' || b === 'latest') return 1;
-
-    const versionA = (a.match(/v\d+(\.\d+)+/) || [])[0];
-    const versionB = (b.match(/v\d+(\.\d+)+/) || [])[0];
-
-    return versionB.localeCompare(versionA);
-}
-
-/**
- * Change the version in a URL path.
- *
- * @param {string} url - The original URL to be modified.
- * @param {string} newVersion - The new version to replace the existing version in the URL.
- * @returns {string} The updated URL with the new version.
- */
-function changeVersion(url, newVersion) {
-    const parsedURL = new URL(url);
-    const pathSegments = parsedURL.pathname.split('/');
-  
-    // Modify the version
-    pathSegments[2] = newVersion;
-
-    // Reconstruct the URL
-    parsedURL.pathname = pathSegments.join('/');
-  
-    return parsedURL.href;
-}
@@ -1 +0,0 @@
-window.SYNAPSE_VERSION = 'v1.78';
@@ -90,32 +90,46 @@ typecheck = ["mypy"]

 [[package]]
 name = "black"
-version = "22.12.0"
+version = "23.1.0"
 description = "The uncompromising code formatter."
 category = "dev"
 optional = false
 python-versions = ">=3.7"
 files = [
-    {file = "black-22.12.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9eedd20838bd5d75b80c9f5487dbcb06836a43833a37846cf1d8c1cc01cef59d"},
-    {file = "black-22.12.0-cp310-cp310-win_amd64.whl", hash = "sha256:159a46a4947f73387b4d83e87ea006dbb2337eab6c879620a3ba52699b1f4351"},
-    {file = "black-22.12.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d30b212bffeb1e252b31dd269dfae69dd17e06d92b87ad26e23890f3efea366f"},
-    {file = "black-22.12.0-cp311-cp311-win_amd64.whl", hash = "sha256:7412e75863aa5c5411886804678b7d083c7c28421210180d67dfd8cf1221e1f4"},
-    {file = "black-22.12.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c116eed0efb9ff870ded8b62fe9f28dd61ef6e9ddd28d83d7d264a38417dcee2"},
-    {file = "black-22.12.0-cp37-cp37m-win_amd64.whl", hash = "sha256:1f58cbe16dfe8c12b7434e50ff889fa479072096d79f0a7f25e4ab8e94cd8350"},
-    {file = "black-22.12.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:77d86c9f3db9b1bf6761244bc0b3572a546f5fe37917a044e02f3166d5aafa7d"},
-    {file = "black-22.12.0-cp38-cp38-win_amd64.whl", hash = "sha256:82d9fe8fee3401e02e79767016b4907820a7dc28d70d137eb397b92ef3cc5bfc"},
-    {file = "black-22.12.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:101c69b23df9b44247bd88e1d7e90154336ac4992502d4197bdac35dd7ee3320"},
-    {file = "black-22.12.0-cp39-cp39-win_amd64.whl", hash = "sha256:559c7a1ba9a006226f09e4916060982fd27334ae1998e7a38b3f33a37f7a2148"},
-    {file = "black-22.12.0-py3-none-any.whl", hash = "sha256:436cc9167dd28040ad90d3b404aec22cedf24a6e4d7de221bec2730ec0c97bcf"},
-    {file = "black-22.12.0.tar.gz", hash = "sha256:229351e5a18ca30f447bf724d007f890f97e13af070bb6ad4c0a441cd7596a2f"},
+    {file = "black-23.1.0-cp310-cp310-macosx_10_16_arm64.whl", hash = "sha256:b6a92a41ee34b883b359998f0c8e6eb8e99803aa8bf3123bf2b2e6fec505a221"},
+    {file = "black-23.1.0-cp310-cp310-macosx_10_16_universal2.whl", hash = "sha256:57c18c5165c1dbe291d5306e53fb3988122890e57bd9b3dcb75f967f13411a26"},
+    {file = "black-23.1.0-cp310-cp310-macosx_10_16_x86_64.whl", hash = "sha256:9880d7d419bb7e709b37e28deb5e68a49227713b623c72b2b931028ea65f619b"},
+    {file = "black-23.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e6663f91b6feca5d06f2ccd49a10f254f9298cc1f7f49c46e498a0771b507104"},
+    {file = "black-23.1.0-cp310-cp310-win_amd64.whl", hash = "sha256:9afd3f493666a0cd8f8df9a0200c6359ac53940cbde049dcb1a7eb6ee2dd7074"},
+    {file = "black-23.1.0-cp311-cp311-macosx_10_16_arm64.whl", hash = "sha256:bfffba28dc52a58f04492181392ee380e95262af14ee01d4bc7bb1b1c6ca8d27"},
+    {file = "black-23.1.0-cp311-cp311-macosx_10_16_universal2.whl", hash = "sha256:c1c476bc7b7d021321e7d93dc2cbd78ce103b84d5a4cf97ed535fbc0d6660648"},
+    {file = "black-23.1.0-cp311-cp311-macosx_10_16_x86_64.whl", hash = "sha256:382998821f58e5c8238d3166c492139573325287820963d2f7de4d518bd76958"},
+    {file = "black-23.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2bf649fda611c8550ca9d7592b69f0637218c2369b7744694c5e4902873b2f3a"},
+    {file = "black-23.1.0-cp311-cp311-win_amd64.whl", hash = "sha256:121ca7f10b4a01fd99951234abdbd97728e1240be89fde18480ffac16503d481"},
+    {file = "black-23.1.0-cp37-cp37m-macosx_10_16_x86_64.whl", hash = "sha256:a8471939da5e824b891b25751955be52ee7f8a30a916d570a5ba8e0f2eb2ecad"},
+    {file = "black-23.1.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8178318cb74f98bc571eef19068f6ab5613b3e59d4f47771582f04e175570ed8"},
+    {file = "black-23.1.0-cp37-cp37m-win_amd64.whl", hash = "sha256:a436e7881d33acaf2536c46a454bb964a50eff59b21b51c6ccf5a40601fbef24"},
+    {file = "black-23.1.0-cp38-cp38-macosx_10_16_arm64.whl", hash = "sha256:a59db0a2094d2259c554676403fa2fac3473ccf1354c1c63eccf7ae65aac8ab6"},
+    {file = "black-23.1.0-cp38-cp38-macosx_10_16_universal2.whl", hash = "sha256:0052dba51dec07ed029ed61b18183942043e00008ec65d5028814afaab9a22fd"},
+    {file = "black-23.1.0-cp38-cp38-macosx_10_16_x86_64.whl", hash = "sha256:49f7b39e30f326a34b5c9a4213213a6b221d7ae9d58ec70df1c4a307cf2a1580"},
+    {file = "black-23.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:162e37d49e93bd6eb6f1afc3e17a3d23a823042530c37c3c42eeeaf026f38468"},
+    {file = "black-23.1.0-cp38-cp38-win_amd64.whl", hash = "sha256:8b70eb40a78dfac24842458476135f9b99ab952dd3f2dab738c1881a9b38b753"},
+    {file = "black-23.1.0-cp39-cp39-macosx_10_16_arm64.whl", hash = "sha256:a29650759a6a0944e7cca036674655c2f0f63806ddecc45ed40b7b8aa314b651"},
+    {file = "black-23.1.0-cp39-cp39-macosx_10_16_universal2.whl", hash = "sha256:bb460c8561c8c1bec7824ecbc3ce085eb50005883a6203dcfb0122e95797ee06"},
+    {file = "black-23.1.0-cp39-cp39-macosx_10_16_x86_64.whl", hash = "sha256:c91dfc2c2a4e50df0026f88d2215e166616e0c80e86004d0003ece0488db2739"},
+    {file = "black-23.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2a951cc83ab535d248c89f300eccbd625e80ab880fbcfb5ac8afb5f01a258ac9"},
+    {file = "black-23.1.0-cp39-cp39-win_amd64.whl", hash = "sha256:0680d4380db3719ebcfb2613f34e86c8e6d15ffeabcf8ec59355c5e7b85bb555"},
+    {file = "black-23.1.0-py3-none-any.whl", hash = "sha256:7a0f701d314cfa0896b9001df70a530eb2472babb76086344e688829efd97d32"},
+    {file = "black-23.1.0.tar.gz", hash = "sha256:b0bd97bea8903f5a2ba7219257a44e3f1f9d00073d6cc1add68f0beec69692ac"},
 ]

 [package.dependencies]
 click = ">=8.0.0"
 mypy-extensions = ">=0.4.3"
+packaging = ">=22.0"
 pathspec = ">=0.9.0"
 platformdirs = ">=2"
-tomli = {version = ">=1.1.0", markers = "python_full_version < \"3.11.0a7\""}
+tomli = {version = ">=1.1.0", markers = "python_version < \"3.11\""}
 typed-ast = {version = ">=1.4.2", markers = "python_version < \"3.8\" and implementation_name == \"cpython\""}
 typing-extensions = {version = ">=3.10.0.0", markers = "python_version < \"3.10\""}

@@ -89,7 +89,7 @@ manifest-path = "rust/Cargo.toml"

 [tool.poetry]
 name = "matrix-synapse"
-version = "1.78.0"
+version = "1.78.0rc1"
 description = "Homeserver for the Matrix decentralised comms protocol"
 authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
 license = "Apache-2.0"
@@ -29,7 +29,6 @@ _Repr = Callable[[], str]
 def recursive_repr(fillvalue: str = ...) -> Callable[[_Repr], _Repr]: ...

 class SortedList(MutableSequence[_T]):
-
    DEFAULT_LOAD_FACTOR: int = ...
    def __init__(
        self,
@@ -47,7 +47,6 @@ def request_registration(
    _print: Callable[[str], None] = print,
    exit: Callable[[int], None] = sys.exit,
 ) -> None:
-
    url = "%s/_synapse/admin/v1/register" % (server_location.rstrip("/"),)

    # Get the nonce
@@ -154,7 +153,6 @@ def register_new_user(


 def main() -> None:
-
    logging.captureWarnings(True)

    parser = argparse.ArgumentParser(
@@ -1205,7 +1205,6 @@ class CursesProgress(Progress):
        if self.finished:
            status = "Time spent: %s (Done!)" % (duration_str,)
        else:
-
            if self.total_processed > 0:
                left = float(self.total_remaining) / self.total_processed

@@ -167,7 +167,6 @@ Worker = collections.namedtuple(


 def main() -> None:
-
    parser = argparse.ArgumentParser()

    parser.add_argument(
@@ -213,7 +213,7 @@ def handle_startup_exception(e: Exception) -> NoReturn:
 def redirect_stdio_to_logs() -> None:
    streams = [("stdout", LogLevel.info), ("stderr", LogLevel.error)]

-    for (stream, level) in streams:
+    for stream, level in streams:
        oldStream = getattr(sys, stream)
        loggingFile = LoggingFile(
            logger=twisted.logger.Logger(namespace=stream),
@@ -44,6 +44,7 @@ from synapse.storage.databases.main.event_push_actions import (
 )
 from synapse.storage.databases.main.events_worker import EventsWorkerStore
 from synapse.storage.databases.main.filtering import FilteringWorkerStore
+from synapse.storage.databases.main.media_repository import MediaRepositoryStore
 from synapse.storage.databases.main.profile import ProfileWorkerStore
 from synapse.storage.databases.main.push_rule import PushRulesWorkerStore
 from synapse.storage.databases.main.receipts import ReceiptsWorkerStore
@@ -86,6 +87,7 @@ class AdminCmdSlavedStore(
    RegistrationWorkerStore,
    RoomWorkerStore,
    ProfileWorkerStore,
+    MediaRepositoryStore,
 ):
    def __init__(
        self,
@@ -149,7 +151,7 @@ class FileExfiltrationWriter(ExfiltrationWriter):

        with open(events_file, "a") as f:
            for event in events:
-                print(json.dumps(event.get_pdu_json()), file=f)
+                json.dump(event.get_pdu_json(), fp=f)

    def write_state(
        self, room_id: str, event_id: str, state: StateMap[EventBase]
@@ -162,7 +164,7 @@ class FileExfiltrationWriter(ExfiltrationWriter):

        with open(event_file, "a") as f:
            for event in state.values():
-                print(json.dumps(event.get_pdu_json()), file=f)
+                json.dump(event.get_pdu_json(), fp=f)

    def write_invite(
        self, room_id: str, event: EventBase, state: StateMap[EventBase]
@@ -178,7 +180,7 @@ class FileExfiltrationWriter(ExfiltrationWriter):

        with open(invite_state, "a") as f:
            for event in state.values():
-                print(json.dumps(event), file=f)
+                json.dump(event, fp=f)

    def write_knock(
        self, room_id: str, event: EventBase, state: StateMap[EventBase]
@@ -194,7 +196,7 @@ class FileExfiltrationWriter(ExfiltrationWriter):

        with open(knock_state, "a") as f:
            for event in state.values():
-                print(json.dumps(event), file=f)
+                json.dump(event, fp=f)

    def write_profile(self, profile: JsonDict) -> None:
        user_directory = os.path.join(self.base_directory, "user_data")
@@ -202,7 +204,7 @@ class FileExfiltrationWriter(ExfiltrationWriter):
        profile_file = os.path.join(user_directory, "profile")

        with open(profile_file, "a") as f:
-            print(json.dumps(profile), file=f)
+            json.dump(profile, fp=f)

    def write_devices(self, devices: List[JsonDict]) -> None:
        user_directory = os.path.join(self.base_directory, "user_data")
@@ -211,7 +213,7 @@ class FileExfiltrationWriter(ExfiltrationWriter):

        for device in devices:
            with open(device_file, "a") as f:
-                print(json.dumps(device), file=f)
+                json.dump(device, fp=f)

    def write_connections(self, connections: List[JsonDict]) -> None:
        user_directory = os.path.join(self.base_directory, "user_data")
@@ -220,7 +222,7 @@ class FileExfiltrationWriter(ExfiltrationWriter):

        for connection in connections:
            with open(connection_file, "a") as f:
-                print(json.dumps(connection), file=f)
+                json.dump(connection, fp=f)

    def write_account_data(
        self, file_name: str, account_data: Mapping[str, JsonDict]
@@ -233,7 +235,15 @@ class FileExfiltrationWriter(ExfiltrationWriter):
        account_data_file = os.path.join(account_data_directory, file_name)

        with open(account_data_file, "a") as f:
-            print(json.dumps(account_data), file=f)
+            json.dump(account_data, fp=f)
+
+    def write_media_id(self, media_id: str, media_metadata: JsonDict) -> None:
+        file_directory = os.path.join(self.base_directory, "media_ids")
+        os.makedirs(file_directory, exist_ok=True)
+        media_id_file = os.path.join(file_directory, media_id)
+
+        with open(media_id_file, "w") as f:
+            json.dump(media_metadata, fp=f)

    def finished(self) -> str:
        return self.base_directory
@@ -219,7 +219,7 @@ def main() -> None:
    # memory space and don't need to repeat the work of loading the code!
    # Instead of using fork() directly, we use the multiprocessing library,
    # which uses fork() on Unix platforms.
-    for (func, worker_args) in zip(worker_functions, args_by_worker):
+    for func, worker_args in zip(worker_functions, args_by_worker):
        process = multiprocessing.Process(
            target=_worker_entrypoint, args=(func, proxy_reactor, worker_args)
        )
@@ -157,7 +157,6 @@ class GenericWorkerServer(HomeServer):
    DATASTORE_CLASS = GenericWorkerSlavedStore  # type: ignore

    def _listen_http(self, listener_config: ListenerConfig) -> None:
-
        assert listener_config.http_options is not None

        # We always include a health resource.
@@ -321,7 +321,6 @@ def setup(config_options: List[str]) -> SynapseHomeServer:
            and not config.registration.registrations_require_3pid
            and not config.registration.registration_requires_token
        ):
-
            raise ConfigError(
                "You have enabled open registration without any verification. This is a known vector for "
                "spam and abuse. If you would like to allow public registration, please consider adding email, "
@@ -22,7 +22,6 @@ from ._base import Config


 class ConsentConfig(Config):
-
    section = "consent"

    def __init__(self, *args: Any):
@@ -154,7 +154,6 @@ class DatabaseConfig(Config):
            logger.warning(NON_SQLITE_DATABASE_PATH_WARNING)

    def set_databasepath(self, database_path: str) -> None:
-
        if database_path != ":memory:":
            database_path = self.abspath(database_path)

@@ -56,7 +56,6 @@ from .workers import WorkerConfig


 class HomeServerConfig(RootConfig):
-
    config_classes = [
        ModulesConfig,
        ServerConfig,
@@ -46,7 +46,6 @@ class RatelimitConfig(Config):
    section = "ratelimiting"

    def read_config(self, config: JsonDict, **kwargs: Any) -> None:
-
        # Load the new-style messages config if it exists. Otherwise fall back
        # to the old method.
        if "rc_message" in config:
@@ -87,9 +86,18 @@ class RatelimitConfig(Config):
            defaults={"per_second": 0.1, "burst_count": 5},
        )

+        # It is reasonable to login with a bunch of devices at once (i.e. when
+        # setting up an account), but it is *not* valid to continually be
+        # logging into new devices.
        rc_login_config = config.get("rc_login", {})
-        self.rc_login_address = RatelimitSettings(rc_login_config.get("address", {}))
-        self.rc_login_account = RatelimitSettings(rc_login_config.get("account", {}))
+        self.rc_login_address = RatelimitSettings(
+            rc_login_config.get("address", {}),
+            defaults={"per_second": 0.003, "burst_count": 5},
+        )
+        self.rc_login_account = RatelimitSettings(
+            rc_login_config.get("account", {}),
+            defaults={"per_second": 0.003, "burst_count": 5},
+        )
        self.rc_login_failed_attempts = RatelimitSettings(
            rc_login_config.get("failed_attempts", {})
        )
@@ -116,7 +116,6 @@ class ContentRepositoryConfig(Config):
    section = "media"

    def read_config(self, config: JsonDict, **kwargs: Any) -> None:
-
        # Only enable the media repo if either the media repo is enabled or the
        # current worker app is the media repo.
        if (
@@ -735,7 +735,6 @@ class ServerConfig(Config):
        listeners: Optional[List[dict]],
        **kwargs: Any,
    ) -> str:
-
        _, bind_port = parse_and_validate_server_name(server_name)
        if bind_port is not None:
            unsecure_port = bind_port - 400
@@ -30,7 +30,6 @@ class TlsConfig(Config):
    section = "tls"

    def read_config(self, config: JsonDict, **kwargs: Any) -> None:
-
        self.tls_certificate_file = self.abspath(config.get("tls_certificate_path"))
        self.tls_private_key_file = self.abspath(config.get("tls_private_key_path"))

@@ -399,7 +399,7 @@ class Keyring:
        # We now convert the returned list of results into a map from server
        # name to key ID to FetchKeyResult, to return.
        to_return: Dict[str, Dict[str, FetchKeyResult]] = {}
-        for (request, results) in zip(deduped_requests, results_per_request):
+        for request, results in zip(deduped_requests, results_per_request):
            to_return_by_server = to_return.setdefault(request.server_name, {})
            for key_id, key_result in results.items():
                existing = to_return_by_server.get(key_id)
@@ -78,7 +78,6 @@ def load_legacy_third_party_event_rules(hs: "HomeServer") -> None:
        # correctly, we need to await its result. Therefore it doesn't make a lot of
        # sense to make it go through the run() wrapper.
        if f.__name__ == "check_event_allowed":
-
            # We need to wrap check_event_allowed because its old form would return either
            # a boolean or a dict, but now we want to return the dict separately from the
            # boolean.
@@ -100,7 +99,6 @@ def load_legacy_third_party_event_rules(hs: "HomeServer") -> None:
            return wrap_check_event_allowed

        if f.__name__ == "on_create_room":
-
            # We need to wrap on_create_room because its old form would return a boolean
            # if the room creation is denied, but now we just want it to raise an
            # exception.
@@ -314,7 +314,7 @@ class FederationRemoteSendQueue(AbstractFederationSender):
        # stream position.
        keyed_edus = {v: k for k, v in self.keyed_edu_changed.items()[i:j]}

-        for ((destination, edu_key), pos) in keyed_edus.items():
+        for (destination, edu_key), pos in keyed_edus.items():
            rows.append(
                (
                    pos,
@@ -329,7 +329,7 @@ class FederationRemoteSendQueue(AbstractFederationSender):
        j = self.edus.bisect_right(to_token) + 1
        edus = self.edus.items()[i:j]

-        for (pos, edu) in edus:
+        for pos, edu in edus:
            rows.append((pos, EduRow(edu)))

        # Sort rows based on pos
@@ -252,16 +252,19 @@ class AdminHandler:
        profile = await self.get_user(UserID.from_string(user_id))
        if profile is not None:
            writer.write_profile(profile)
+            logger.info("[%s] Written profile", user_id)

        # Get all devices the user has
        devices = await self._device_handler.get_devices_by_user(user_id)
        writer.write_devices(devices)
+        logger.info("[%s] Written %s devices", user_id, len(devices))

        # Get all connections the user has
        connections = await self.get_whois(UserID.from_string(user_id))
        writer.write_connections(
            connections["devices"][""]["sessions"][0]["connections"]
        )
+        logger.info("[%s] Written %s connections", user_id, len(connections))

        # Get all account data the user has global and in rooms
        global_data = await self._store.get_global_account_data_for_user(user_id)
@@ -269,6 +272,29 @@ class AdminHandler:
        writer.write_account_data("global", global_data)
        for room_id in by_room_data:
            writer.write_account_data(room_id, by_room_data[room_id])
+        logger.info(
+            "[%s] Written account data for %s rooms", user_id, len(by_room_data)
+        )
+
+        # Get all media ids the user has
+        limit = 100
+        start = 0
+        while True:
+            media_ids, total = await self._store.get_local_media_by_user_paginate(
+                start, limit, user_id
+            )
+            for media in media_ids:
+                writer.write_media_id(media["media_id"], media)
+
+            logger.info(
+                "[%s] Written %d media_ids of %s",
+                user_id,
+                (start + len(media_ids)),
+                total,
+            )
+            if (start + limit) >= total:
+                break
+            start += limit

        return writer.finished()

@@ -359,6 +385,18 @@ class ExfiltrationWriter(metaclass=abc.ABCMeta):
        """
        raise NotImplementedError()

+    @abc.abstractmethod
+    def write_media_id(self, media_id: str, media_metadata: JsonDict) -> None:
+        """Write the media's metadata of a user.
+        Exports only the metadata, as this can be fetched from the database via
+        read only. In order to access the files, a connection to the correct
+        media repository would be required.
+
+        Args:
+            media_id: ID of the media.
+            media_metadata: Metadata of one media file.
+        """
+
    @abc.abstractmethod
    def finished(self) -> Any:
        """Called when all data has successfully been exported and written.
@@ -737,7 +737,7 @@ class ApplicationServicesHandler:
        )

        ret = []
-        for (success, result) in results:
+        for success, result in results:
            if success:
                ret.extend(result)

@@ -815,7 +815,6 @@ class AuthHandler:
        now_ms = self._clock.time_msec()

        if existing_token.expiry_ts is not None and existing_token.expiry_ts < now_ms:
-
            raise SynapseError(
                HTTPStatus.FORBIDDEN,
                "The supplied refresh token has expired",
@@ -2259,7 +2258,6 @@ class PasswordAuthProvider:
    async def on_logged_out(
        self, user_id: str, device_id: Optional[str], access_token: str
    ) -> None:
-
        # call all of the on_logged_out callbacks
        for callback in self.on_logged_out_callbacks:
            try:
@@ -497,9 +497,11 @@ class DirectoryHandler:
                raise SynapseError(403, "Not allowed to publish room")

            # Check if publishing is blocked by a third party module
-            allowed_by_third_party_rules = await (
-                self.third_party_event_rules.check_visibility_can_be_modified(
-                    room_id, visibility
+            allowed_by_third_party_rules = (
+                await (
+                    self.third_party_event_rules.check_visibility_can_be_modified(
+                        room_id, visibility
+                    )
                )
            )
            if not allowed_by_third_party_rules:
@@ -188,7 +188,6 @@ class E2eRoomKeysHandler:

        # XXX: perhaps we should use a finer grained lock here?
        async with self._upload_linearizer.queue(user_id):
-
            # Check that the version we're trying to upload is the current version
            try:
                version_info = await self.store.get_e2e_room_keys_version_info(user_id)
@@ -236,7 +236,6 @@ class EventAuthHandler:
        # in any of them.
        allowed_rooms = await self.get_rooms_that_allow_join(state_ids)
        if not await self.is_user_in_rooms(allowed_rooms, user_id):
-
            # If this is a remote request, the user might be in an allowed room
            # that we do not know about.
            if get_domain_from_id(user_id) != self._server_name:
@@ -124,7 +124,6 @@ class InitialSyncHandler:
        as_client_event: bool = True,
        include_archived: bool = False,
    ) -> JsonDict:
-
        memberships = [Membership.INVITE, Membership.JOIN]
        if include_archived:
            memberships.append(Membership.LEAVE)
@@ -777,7 +777,6 @@ class PresenceHandler(BasePresenceHandler):
        )

        if self.unpersisted_users_changes:
-
            await self.store.update_presence(
                [
                    self.user_to_current_state[user_id]
@@ -823,7 +822,6 @@ class PresenceHandler(BasePresenceHandler):
        now = self.clock.time_msec()

        with Measure(self.clock, "presence_update_states"):
-
            # NOTE: We purposefully don't await between now and when we've
            # calculated what we want to do with the new states, to avoid races.

@@ -476,7 +476,7 @@ class RegistrationHandler:
                    # create room expects the localpart of the room alias
                    config["room_alias_name"] = room_alias.localpart

-                    info, _ = await room_creation_handler.create_room(
+                    room_id, _, _ = await room_creation_handler.create_room(
                        fake_requester,
                        config=config,
                        ratelimit=False,
@@ -490,7 +490,7 @@ class RegistrationHandler:
                                user_id, authenticated_entity=self._server_name
                            ),
                            target=UserID.from_string(user_id),
-                            room_id=info["room_id"],
+                            room_id=room_id,
                            # Since it was just created, there are no remote hosts.
                            remote_room_hosts=[],
                            action="join",
@@ -690,13 +690,14 @@ class RoomCreationHandler:
        config: JsonDict,
        ratelimit: bool = True,
        creator_join_profile: Optional[JsonDict] = None,
-    ) -> Tuple[dict, int]:
+    ) -> Tuple[str, Optional[RoomAlias], int]:
        """Creates a new room.

        Args:
-            requester:
-                The user who requested the room creation.
-            config : A dict of configuration options.
+            requester: The user who requested the room creation.
+            config: A dict of configuration options. This will be the body of
+                a /createRoom request; see
+                https://spec.matrix.org/latest/client-server-api/#post_matrixclientv3createroom
            ratelimit: set to False to disable the rate limiter

            creator_join_profile:
@@ -707,14 +708,17 @@ class RoomCreationHandler:
                `avatar_url` and/or `displayname`.

        Returns:
-                First, a dict containing the keys `room_id` and, if an alias
-                was, requested, `room_alias`. Secondly, the stream_id of the
-                last persisted event.
+            A 3-tuple containing:
+                - the room ID;
+                - if requested, the room alias, otherwise None; and
+                - the `stream_id` of the last persisted event.
        Raises:
-            SynapseError if the room ID couldn't be stored, 3pid invitation config
-            validation failed, or something went horribly wrong.
-            ResourceLimitError if server is blocked to some resource being
-            exceeded
+            SynapseError:
+                if the room ID couldn't be stored, 3pid invitation config
+                validation failed, or something went horribly wrong.
+            ResourceLimitError:
+                if server is blocked to some resource being
+                exceeded
        """
        user_id = requester.user.to_string()

@@ -864,9 +868,11 @@ class RoomCreationHandler:
        )

        # Check whether this visibility value is blocked by a third party module
-        allowed_by_third_party_rules = await (
-            self.third_party_event_rules.check_visibility_can_be_modified(
-                room_id, visibility
+        allowed_by_third_party_rules = (
+            await (
+                self.third_party_event_rules.check_visibility_can_be_modified(
+                    room_id, visibility
+                )
            )
        )
        if not allowed_by_third_party_rules:
@@ -1024,11 +1030,6 @@ class RoomCreationHandler:
            last_sent_event_id = member_event_id
            depth += 1

-        result = {"room_id": room_id}
-
-        if room_alias:
-            result["room_alias"] = room_alias.to_string()
-
        # Always wait for room creation to propagate before returning
        await self._replication.wait_for_stream_position(
            self.hs.config.worker.events_shard_config.get_instance(room_id),
@@ -1036,7 +1037,7 @@ class RoomCreationHandler:
            last_stream_id,
        )

-        return result, last_stream_id
+        return room_id, room_alias, last_stream_id

    async def _send_events_for_new_room(
        self,
@@ -1825,7 +1826,7 @@ class RoomShutdownHandler:
                new_room_user_id, authenticated_entity=requester_user_id
            )

-            info, stream_id = await self._room_creation_handler.create_room(
+            new_room_id, _, stream_id = await self._room_creation_handler.create_room(
                room_creator_requester,
                config={
                    "preset": RoomCreationPreset.PUBLIC_CHAT,
@@ -1834,7 +1835,6 @@ class RoomShutdownHandler:
                },
                ratelimit=False,
            )
-            new_room_id = info["room_id"]

            logger.info(
                "Shutting down room %r, joining to new room: %r", room_id, new_room_id
@@ -1887,6 +1887,7 @@ class RoomShutdownHandler:

                # Join users to new room
                if new_room_user_id:
+                    assert new_room_id is not None
                    await self.room_member_handler.update_membership(
                        requester=target_requester,
                        target=target_requester.user,
@@ -1919,6 +1920,7 @@ class RoomShutdownHandler:

            aliases_for_room = await self.store.get_aliases_for_room(room_id)

+            assert new_room_id is not None
            await self.store.update_aliases_for_room(
                room_id, new_room_id, requester_user_id
            )
@@ -374,7 +374,7 @@ class RoomBatchHandler:
        # correct stream_ordering as they are backfilled (which decrements).
        # Events are sorted by (topological_ordering, stream_ordering)
        # where topological_ordering is just depth.
-        for (event, context) in reversed(events_to_persist):
+        for event, context in reversed(events_to_persist):
            # This call can't raise `PartialStateConflictError` since we forbid
            # use of the historical batch API during partial state
            await self.event_creation_handler.handle_new_client_event(
@@ -1297,7 +1297,6 @@ class SyncHandler:
            return RoomNotifCounts.empty()

        with Measure(self.clock, "unread_notifs_for_room_id"):
-
            return await self.store.get_unread_event_push_actions_by_room_for_user(
                room_id,
                sync_config.user.to_string(),
@@ -524,6 +524,7 @@ def whitelisted_homeserver(destination: str) -> bool:

 # Start spans and scopes

+
 # Could use kwargs but I want these to be explicit
 def start_active_span(
    operation_name: str,
@@ -87,7 +87,6 @@ class LaterGauge(Collector):
    ]

    def collect(self) -> Iterable[Metric]:
-
        g = GaugeMetricFamily(self.name, self.desc, labels=self.labels)

        try:
@@ -139,7 +139,6 @@ def install_gc_manager() -> None:

 class PyPyGCStats(Collector):
    def collect(self) -> Iterable[Metric]:
-
        # @stats is a pretty-printer object with __str__() returning a nice table,
        # plus some fields that contain data from that table.
        # unfortunately, fields are pretty-printed themselves (i. e. '4.5MB').
@@ -37,7 +37,12 @@ from twisted.internet import defer
 from twisted.web.resource import Resource

 from synapse.api import errors
-from synapse.api.errors import SynapseError
+from synapse.api.errors import (
+    FederationDeniedError,
+    HttpResponseException,
+    RequestSendFailed,
+    SynapseError,
+)
 from synapse.events import EventBase
 from synapse.events.presence_router import (
    GET_INTERESTED_USERS_CALLBACK,
@@ -129,6 +134,14 @@ from synapse.util import Clock
 from synapse.util.async_helpers import maybe_awaitable
 from synapse.util.caches.descriptors import CachedFunction, cached as _cached
 from synapse.util.frozenutils import freeze
+from synapse.util.retryutils import NotRetryingDestination
+
+from .errors import (
+    FederationHttpDeniedException,
+    FederationHttpNotRetryingDestinationException,
+    FederationHttpRequestSendFailedException,
+    FederationHttpResponseException,
+)

 if TYPE_CHECKING:
    from synapse.app.generic_worker import GenericWorkerSlavedStore
@@ -1576,14 +1589,14 @@ class ModuleApi:
            )

        requester = create_requester(user_id)
-        room_id_and_alias, _ = await self._hs.get_room_creation_handler().create_room(
+        room_id, room_alias, _ = await self._hs.get_room_creation_handler().create_room(
            requester=requester,
            config=config,
            ratelimit=ratelimit,
            creator_join_profile=creator_join_profile,
        )
-
-        return room_id_and_alias["room_id"], room_id_and_alias.get("room_alias", None)
+        room_alias_str = room_alias.to_string() if room_alias else None
+        return room_id, room_alias_str

    async def set_displayname(
        self,
@@ -1612,6 +1625,123 @@ class ModuleApi:
            deactivation=deactivation,
        )

+    async def _try_federation_http_request(
+        self,
+        method: str,
+        remote_server_name: str,
+        path: str,
+        query_parameters: Optional[Dict[str, Any]],
+        body: Optional[JsonDict] = None,
+        timeout: Optional[int] = None,
+    ) -> Union[JsonDict, List]:
+        """
+        Send a federation request to a remote homeserver and return the response.
+
+        This method assumes the `method` argument is fully capitalised.
+
+        A helper method for self.send_federation_http_request, see that method for
+        more details.
+        """
+        assert method in ["GET", "PUT", "POST", "DELETE"]
+
+        fed_client = self._hs.get_federation_http_client()
+
+        if method == "GET":
+            return await fed_client.get_json(
+                destination=remote_server_name,
+                path=path,
+                args=query_parameters,
+                timeout=timeout,
+            )
+        elif method == "PUT":
+            return await fed_client.put_json(
+                destination=remote_server_name,
+                path=path,
+                args=query_parameters,
+                data=body,
+                timeout=timeout,
+            )
+        elif method == "POST":
+            return await fed_client.post_json(
+                destination=remote_server_name,
+                path=path,
+                args=query_parameters,
+                data=body,
+                timeout=timeout,
+            )
+        elif method == "DELETE":
+            return await fed_client.delete_json(
+                destination=remote_server_name,
+                path=path,
+                args=query_parameters,
+                timeout=timeout,
+            )
+
+        return {}
+
+    async def send_federation_http_request(
+        self,
+        method: str,
+        remote_server_name: str,
+        path: str,
+        query_parameters: Optional[Dict[str, Any]],
+        body: Optional[JsonDict] = None,
+        timeout: Optional[int] = None,
+    ) -> Union[JsonDict, List]:
+        """
+        Send an HTTP federation request to a remote homeserver.
+
+        Added in Synapse v1.79.0.
+
+        If the request is successful, the parsed response body will be returned. If
+        unsuccessful, an exception will be raised. Callers are expected to handle the
+        possible exception cases. See exception class docstrings for a more detailed
+        explanation of each.
+
+        Args:
+            method: The HTTP method to use. Must be one of: "GET", "PUT", "POST",
+                "DELETE".
+            remote_server_name: The remote server to send the request to. This method
+                 will resolve delegated homeserver URLs automatically (well-known etc).
+            path: The HTTP path for the request.
+            query_parameters: Any query parameters for the request.
+            body: The body of the request.
+            timeout: The timeout in seconds to wait before giving up on a request.
+
+        Returns:
+            The response to the request as a Python object.
+
+        Raises:
+            FederationHttpResponseException: If we get an HTTP response code >= 300
+                (except 429).
+            FederationHttpNotRetryingDestinationException: If the homeserver believes the
+                remote homeserver is down and is not yet ready to attempt to contact it.
+            FederationHttpDeniedException: If this destination is not on the local
+                homeserver's configured federation whitelist.
+            FederationHttpRequestSendFailedException: If there were problems connecting
+                to the remote, due to e.g. DNS failures, connection timeouts etc.
+        """
+        try:
+            return await self._try_federation_http_request(
+                method.upper(), remote_server_name, path, query_parameters, body, timeout
+            )
+        except HttpResponseException as e:
+            raise FederationHttpResponseException(
+                remote_server_name,
+                status_code=e.code,
+                msg=e.msg,
+                response_body=e.response,
+            )
+        except NotRetryingDestination:
+            raise FederationHttpNotRetryingDestinationException(remote_server_name)
+        except FederationDeniedError:
+            raise FederationHttpDeniedException(remote_server_name)
+        except RequestSendFailed as e:
+            raise FederationHttpRequestSendFailedException(
+                remote_server_name,
+                can_retry=e.can_retry,
+            )
+

 class PublicRoomListManager:
    """Contains methods for adding to, removing from and querying whether a room
@@ -13,6 +13,7 @@
 # limitations under the License.

 """Exception types which are exposed as part of the stable module API"""
+import attr

 from synapse.api.errors import (
    Codes,
@@ -24,6 +25,57 @@ from synapse.config._base import ConfigError
 from synapse.handlers.push_rules import InvalidRuleException
 from synapse.storage.push_rule import RuleNotFoundException

+
+@attr.s(auto_attribs=True)
+class FederationHttpResponseException(Exception):
+    """
+    Raised when an HTTP request over federation returns a status code > 300 (and not 429).
+    """
+
+    remote_server_name: str
+    # The HTTP status code of the response.
+    status_code: int
+    # A human-readable explanation for the error.
+    msg: str
+    # The non-parsed HTTP response body.
+    response_body: bytes
+
+
+@attr.s(auto_attribs=True)
+class FederationHttpNotRetryingDestinationException(Exception):
+    """
+    Raised when the local homeserver refuses to send traffic to a remote homeserver that
+    it believes is experiencing an outage.
+    """
+
+    remote_server_name: str
+
+
+@attr.s(auto_attribs=True)
+class FederationHttpDeniedException(Exception):
+    """
+    Raised when the local homeserver refuses to send federation traffic to a remote
+    homeserver. This is due to the remote homeserver not being on the configured
+    federation whitelist.
+    """
+
+    remote_server_name: str
+
+
+@attr.s(auto_attribs=True)
+class FederationHttpRequestSendFailedException(Exception):
+    """
+    Raised when there are problems connecting to the remote homeserver due to e.g.
+    DNS failures, connection timeouts, etc.
+    """
+
+    remote_server_name: str
+    # Whether the request can be retried with a chance of success. This will be True
+    # if the failure occurred due to e.g. timeouts, a disruption in the connection etc.
+    # Will be false in the case of e.g. a malformed response from the remote homeserver.
+    can_retry: bool
+
+
 __all__ = [
    "Codes",
    "InvalidClientCredentialsError",
@@ -32,4 +84,8 @@ __all__ = [
    "ConfigError",
    "InvalidRuleException",
    "RuleNotFoundException",
+    "FederationHttpResponseException",
+    "FederationHttpNotRetryingDestinationException",
+    "FederationHttpDeniedException",
+    "FederationHttpRequestSendFailedException",
 ]
@@ -330,7 +330,6 @@ class BulkPushRuleEvaluator:
        context: EventContext,
        event_id_to_event: Mapping[str, EventBase],
    ) -> None:
-
        if (
            not event.internal_metadata.is_notifiable()
            or event.internal_metadata.is_historical()
@@ -265,7 +265,6 @@ class ReplicationRemoveTagRestServlet(ReplicationEndpoint):

    @staticmethod
    async def _serialize_payload(user_id: str, room_id: str, tag: str) -> JsonDict:  # type: ignore[override]
-
        return {}

    async def _handle_request(  # type: ignore[override]
@@ -195,7 +195,6 @@ class ReplicationUploadKeysForUserRestServlet(ReplicationEndpoint):
    async def _serialize_payload(  # type: ignore[override]
        user_id: str, device_id: str, keys: JsonDict
    ) -> JsonDict:
-
        return {
            "user_id": user_id,
            "device_id": device_id,
@@ -328,7 +328,6 @@ class RedisDirectTcpReplicationClientFactory(SynapseRedisFactory):
        outbound_redis_connection: txredisapi.ConnectionHandler,
        channel_names: List[str],
    ):
-
        super().__init__(
            hs,
            uuid="subscriber",
@@ -238,24 +238,6 @@ class ReplicationStreamer:
                            except Exception:
                                logger.exception("Failed to replicate")

-                        # The last token we send may not match the current
-                        # token, in which case we want to send out a `POSITION`
-                        # to tell other workers the actual current position.
-                        if updates[-1][0] < current_token:
-                            logger.info(
-                                "Sending position: %s -> %s",
-                                stream.NAME,
-                                current_token,
-                            )
-                            self.command_handler.send_command(
-                                PositionCommand(
-                                    stream.NAME,
-                                    self._instance_name,
-                                    updates[-1][0],
-                                    current_token,
-                                )
-                            )
-
            logger.debug("No more pending updates, breaking poke loop")
        finally:
            self.pending_updates = False
@@ -139,7 +139,6 @@ class EventsStream(Stream):
        current_token: Token,
        target_row_count: int,
    ) -> StreamUpdateResult:
-
        # the events stream merges together three separate sources:
        #  * new events
        #  * current_state changes
@@ -75,7 +75,6 @@ class RoomRestV2Servlet(RestServlet):
    async def on_DELETE(
        self, request: SynapseRequest, room_id: str
    ) -> Tuple[int, JsonDict]:
-
        requester = await self._auth.get_user_by_req(request)
        await assert_user_is_admin(self._auth, requester)

@@ -144,7 +143,6 @@ class DeleteRoomStatusByRoomIdRestServlet(RestServlet):
    async def on_GET(
        self, request: SynapseRequest, room_id: str
    ) -> Tuple[int, JsonDict]:
-
        await assert_requester_is_admin(self._auth, request)

        if not RoomID.is_valid(room_id):
@@ -181,7 +179,6 @@ class DeleteRoomStatusByDeleteIdRestServlet(RestServlet):
    async def on_GET(
        self, request: SynapseRequest, delete_id: str
    ) -> Tuple[int, JsonDict]:
-
        await assert_requester_is_admin(self._auth, request)

        delete_status = self._pagination_handler.get_delete_status(delete_id)
@@ -438,7 +435,6 @@ class RoomStateRestServlet(RestServlet):


 class JoinRoomAliasServlet(ResolveRoomIdMixin, RestServlet):
-
    PATTERNS = admin_patterns("/join/(?P<room_identifier>[^/]*)$")

    def __init__(self, hs: "HomeServer"):
@@ -683,8 +683,12 @@ class AccountValidityRenewServlet(RestServlet):
        await assert_requester_is_admin(self.auth, request)

        if self.account_activity_handler.on_legacy_admin_request_callback:
-            expiration_ts = await (
-                self.account_activity_handler.on_legacy_admin_request_callback(request)
+            expiration_ts = (
+                await (
+                    self.account_activity_handler.on_legacy_admin_request_callback(
+                        request
+                    )
+                )
            )
        else:
            body = parse_json_object_from_request(request)
@@ -97,7 +97,6 @@ class AuthRestServlet(RestServlet):
        return None

    async def on_POST(self, request: Request, stagetype: str) -> None:
-
        session = parse_string(request, "session")
        if not session:
            raise SynapseError(400, "No session supplied")
@@ -79,7 +79,6 @@ class CreateFilterRestServlet(RestServlet):
    async def on_POST(
        self, request: SynapseRequest, user_id: str
    ) -> Tuple[int, JsonDict]:
-
        target_user = UserID.from_string(user_id)
        requester = await self.auth.get_user_by_req(request)

@@ -628,10 +628,12 @@ class RegisterRestServlet(RestServlet):
            if not password_hash:
                raise SynapseError(400, "Missing params: password", Codes.MISSING_PARAM)

-            desired_username = await (
-                self.password_auth_provider.get_username_for_registration(
-                    auth_result,
-                    params,
+            desired_username = (
+                await (
+                    self.password_auth_provider.get_username_for_registration(
+                        auth_result,
+                        params,
+                    )
                )
            )

@@ -682,9 +684,11 @@ class RegisterRestServlet(RestServlet):
                session_id
            )

-            display_name = await (
-                self.password_auth_provider.get_displayname_for_registration(
-                    auth_result, params
+            display_name = (
+                await (
+                    self.password_auth_provider.get_displayname_for_registration(
+                        auth_result, params
+                    )
                )
            )

@@ -160,11 +160,11 @@ class RoomCreateRestServlet(TransactionRestServlet):
    async def on_POST(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
        requester = await self.auth.get_user_by_req(request)

-        info, _ = await self._room_creation_handler.create_room(
+        room_id, _, _ = await self._room_creation_handler.create_room(
            requester, self.get_room_config(request)
        )

-        return 200, info
+        return 200, {"room_id": room_id}

    def get_room_config(self, request: Request) -> JsonDict:
        user_supplied_config = parse_json_object_from_request(request)
@@ -270,7 +270,6 @@ async def respond_with_responder(
        logger.debug("Responding to media request with responder %s", responder)
        add_file_headers(request, media_type, file_size, upload_name)
        try:
-
            await responder.write_to_consumer(request)
        except Exception as e:
            # The majority of the time this will be due to the client having gone
@@ -38,7 +38,6 @@ class ThumbnailError(Exception):


 class Thumbnailer:
-
    FORMATS = {"image/jpeg": "JPEG", "image/png": "PNG"}

    @staticmethod
@@ -178,7 +178,7 @@ class ServerNoticesManager:
                "avatar_url": self._config.servernotices.server_notices_mxid_avatar_url,
            }

-        info, _ = await self._room_creation_handler.create_room(
+        room_id, _, _ = await self._room_creation_handler.create_room(
            requester,
            config={
                "preset": RoomCreationPreset.PRIVATE_CHAT,
@@ -188,7 +188,6 @@ class ServerNoticesManager:
            ratelimit=False,
            creator_join_profile=join_profile,
        )
-        room_id = info["room_id"]

        self.maybe_get_notice_room_for_user.invalidate((user_id,))

@@ -721,8 +721,8 @@ class DeviceInboxWorkerStore(SQLBaseStore):
                        ],
                    )

-                for (user_id, messages_by_device) in edu["messages"].items():
-                    for (device_id, msg) in messages_by_device.items():
+                for user_id, messages_by_device in edu["messages"].items():
+                    for device_id, msg in messages_by_device.items():
                        with start_active_span("store_outgoing_to_device_message"):
                            set_tag(SynapseTags.TO_DEVICE_EDU_ID, edu["sender"])
                            set_tag(SynapseTags.TO_DEVICE_EDU_ID, edu["message_id"])
@@ -959,7 +959,6 @@ class DeviceInboxBackgroundUpdateStore(SQLBaseStore):
        def _remove_dead_devices_from_device_inbox_txn(
            txn: LoggingTransaction,
        ) -> Tuple[int, bool]:
-
            if "max_stream_id" in progress:
                max_stream_id = progress["max_stream_id"]
            else:
@@ -512,7 +512,7 @@ class DeviceWorkerStore(RoomMemberWorkerStore, EndToEndKeyWorkerStore):
            results.append(("org.matrix.signing_key_update", result))

        if issue_8631_logger.isEnabledFor(logging.DEBUG):
-            for (user_id, edu) in results:
+            for user_id, edu in results:
                issue_8631_logger.debug(
                    "device update to %s for %s from %s to %s: %s",
                    destination,
@@ -1316,7 +1316,7 @@ class DeviceWorkerStore(RoomMemberWorkerStore, EndToEndKeyWorkerStore):
                )
            """
            count = 0
-            for (destination, user_id, stream_id, device_id) in rows:
+            for destination, user_id, stream_id, device_id in rows:
                txn.execute(
                    delete_sql, (destination, user_id, stream_id, stream_id, device_id)
                )
@@ -108,7 +108,7 @@ class EndToEndRoomKeyStore(SQLBaseStore):
            raise StoreError(404, "No backup with that version exists")

        values = []
-        for (room_id, session_id, room_key) in room_keys:
+        for room_id, session_id, room_key in room_keys:
            values.append(
                (
                    user_id,
@@ -268,7 +268,7 @@ class EndToEndKeyWorkerStore(EndToEndKeyBackgroundStore, CacheInvalidationWorker
            )

            # add each cross-signing signature to the correct device in the result dict.
-            for (user_id, key_id, device_id, signature) in cross_sigs_result:
+            for user_id, key_id, device_id, signature in cross_sigs_result:
                target_device_result = result[user_id][device_id]
                # We've only looked up cross-signatures for non-deleted devices with key
                # data.
@@ -311,7 +311,7 @@ class EndToEndKeyWorkerStore(EndToEndKeyBackgroundStore, CacheInvalidationWorker
        # devices.
        user_list = []
        user_device_list = []
-        for (user_id, device_id) in query_list:
+        for user_id, device_id in query_list:
            if device_id is None:
                user_list.append(user_id)
            else:
@@ -353,7 +353,7 @@ class EndToEndKeyWorkerStore(EndToEndKeyBackgroundStore, CacheInvalidationWorker

            txn.execute(sql, query_params)

-            for (user_id, device_id, display_name, key_json) in txn:
+            for user_id, device_id, display_name, key_json in txn:
                assert device_id is not None
                if include_deleted_devices:
                    deleted_devices.remove((user_id, device_id))
@@ -382,7 +382,7 @@ class EndToEndKeyWorkerStore(EndToEndKeyBackgroundStore, CacheInvalidationWorker
        signature_query_clauses = []
        signature_query_params = []

-        for (user_id, device_id) in device_query:
+        for user_id, device_id in device_query:
            signature_query_clauses.append(
                "target_user_id = ? AND target_device_id = ? AND user_id = ?"
            )
@@ -1612,7 +1612,6 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
        latest_events: List[str],
        limit: int,
    ) -> List[str]:
-
        seen_events = set(earliest_events)
        front = set(latest_events) - seen_events
        event_results: List[str] = []
@@ -469,7 +469,6 @@ class PersistEventsStore:
        txn: LoggingTransaction,
        events: List[EventBase],
    ) -> None:
-
        # We only care about state events, so this if there are no state events.
        if not any(e.is_state() for e in events):
            return
@@ -709,7 +709,7 @@ class EventsBackgroundUpdatesStore(SQLBaseStore):

            nbrows = 0
            last_row_event_id = ""
-            for (event_id, event_json_raw) in results:
+            for event_id, event_json_raw in results:
                try:
                    event_json = db_to_json(event_json_raw)

@@ -1167,7 +1167,7 @@ class EventsBackgroundUpdatesStore(SQLBaseStore):
            results = list(txn)
            # (event_id, parent_id, rel_type) for each relation
            relations_to_insert: List[Tuple[str, str, str]] = []
-            for (event_id, event_json_raw) in results:
+            for event_id, event_json_raw in results:
                try:
                    event_json = db_to_json(event_json_raw)
                except Exception as e:
@@ -1493,7 +1493,7 @@ class EventsWorkerStore(SQLBaseStore):

            txn.execute(redactions_sql + clause, args)

-            for (redacter, redacted) in txn:
+            for redacter, redacted in txn:
                d = event_dict.get(redacted)
                if d:
                    d.redactions.append(redacter)
@@ -196,7 +196,6 @@ class MediaRepositoryStore(MediaRepositoryBackgroundUpdateStore):
        def get_local_media_by_user_paginate_txn(
            txn: LoggingTransaction,
        ) -> Tuple[List[Dict[str, Any]], int]:
-
            # Set ordering
            order_by_column = MediaSortOrder(order_by).value

@@ -344,7 +344,6 @@ class PusherWorkerStore(SQLBaseStore):
        last_user = progress.get("last_user", "")

        def _delete_pushers(txn: LoggingTransaction) -> int:
-
            sql = """
                SELECT name FROM users
                WHERE deactivated = ? and name > ?
@@ -392,7 +391,6 @@ class PusherWorkerStore(SQLBaseStore):
        last_pusher = progress.get("last_pusher", 0)

        def _delete_pushers(txn: LoggingTransaction) -> int:
-
            sql = """
                SELECT p.id, access_token FROM pushers AS p
                LEFT JOIN access_tokens AS a ON (p.access_token = a.id)
@@ -449,7 +447,6 @@ class PusherWorkerStore(SQLBaseStore):
        last_pusher = progress.get("last_pusher", 0)

        def _delete_pushers(txn: LoggingTransaction) -> int:
-
            sql = """
                SELECT p.id, p.user_name, p.app_id, p.pushkey
                FROM pushers AS p
@@ -887,7 +887,6 @@ class ReceiptsBackgroundUpdateStore(SQLBaseStore):
        def _populate_receipt_event_stream_ordering_txn(
            txn: LoggingTransaction,
        ) -> bool:
-
            if "max_stream_id" in progress:
                max_stream_id = progress["max_stream_id"]
            else:
@@ -2168,7 +2168,6 @@ class RoomStore(RoomBackgroundUpdateStore, RoomWorkerStore):
        def _get_event_report_txn(
            txn: LoggingTransaction, report_id: int
        ) -> Optional[Dict[str, Any]]:
-
            sql = """
                SELECT
                    er.id,
@@ -122,7 +122,6 @@ class SearchWorkerStore(SQLBaseStore):


 class SearchBackgroundUpdateStore(SearchWorkerStore):
-
    EVENT_SEARCH_UPDATE_NAME = "event_search"
    EVENT_SEARCH_ORDER_UPDATE_NAME = "event_search_order"
    EVENT_SEARCH_USE_GIN_POSTGRES_NAME = "event_search_postgres_gin"
@@ -615,7 +614,6 @@ class SearchStore(SearchBackgroundUpdateStore):
            """
            count_args = [search_query] + count_args
        elif isinstance(self.database_engine, Sqlite3Engine):
-
            # We use CROSS JOIN here to ensure we use the right indexes.
            # https://sqlite.org/optoverview.html#crossjoin
            #
@@ -490,7 +490,6 @@ class StateGroupWorkerStore(EventsWorkerStore, SQLBaseStore):


 class MainStateBackgroundUpdateStore(RoomMemberWorkerStore):
-
    CURRENT_STATE_INDEX_UPDATE_NAME = "current_state_members_idx"
    EVENT_STATE_GROUP_INDEX_UPDATE_NAME = "event_to_state_groups_sg_index"
    DELETE_CURRENT_STATE_UPDATE_NAME = "delete_old_current_state_events"
@@ -461,7 +461,7 @@ class StatsStore(StateDeltasStore):
        insert_cols = []
        qargs = []

-        for (key, val) in chain(
+        for key, val in chain(
            keyvalues.items(), absolutes.items(), additive_relatives.items()
        ):
            insert_cols.append(key)
@@ -87,6 +87,7 @@ MAX_STREAM_SIZE = 1000
 _STREAM_TOKEN = "stream"
 _TOPOLOGICAL_TOKEN = "topological"

+
 # Used as return values for pagination APIs
@attr.s(slots=True, frozen=True, auto_attribs=True)
 class _EventDictReturn:
@@ -573,7 +573,6 @@ class TransactionWorkerStore(CacheInvalidationWorkerStore):
        def get_destination_rooms_paginate_txn(
            txn: LoggingTransaction,
        ) -> Tuple[List[JsonDict], int]:
-
            if direction == Direction.BACKWARDS:
                order = "DESC"
            else:
@@ -98,7 +98,6 @@ class UserDirectoryBackgroundUpdateStore(StateDeltasStore):
    async def _populate_user_directory_createtables(
        self, progress: JsonDict, batch_size: int
    ) -> int:
-
        # Get all the rooms that we want to process.
        def _make_staging_area(txn: LoggingTransaction) -> None:
            sql = (
@@ -251,7 +251,6 @@ class StateGroupBackgroundUpdateStore(SQLBaseStore):


 class StateBackgroundUpdateStore(StateGroupBackgroundUpdateStore):
-
    STATE_GROUP_DEDUPLICATION_UPDATE_NAME = "state_group_state_deduplication"
    STATE_GROUP_INDEX_UPDATE_NAME = "state_group_state_type_index"
    STATE_GROUPS_ROOM_INDEX_UPDATE_NAME = "state_groups_room_id_idx"
@@ -257,14 +257,11 @@ class StateGroupDataStore(StateBackgroundUpdateStore, SQLBaseStore):
        member_filter, non_member_filter = state_filter.get_member_split()

        # Now we look them up in the member and non-member caches
-        (
-            non_member_state,
-            incomplete_groups_nm,
-        ) = self._get_state_for_groups_using_cache(
+        non_member_state, incomplete_groups_nm = self._get_state_for_groups_using_cache(
            groups, self._state_group_cache, state_filter=non_member_filter
        )

-        (member_state, incomplete_groups_m,) = self._get_state_for_groups_using_cache(
+        member_state, incomplete_groups_m = self._get_state_for_groups_using_cache(
            groups, self._state_group_members_cache, state_filter=member_filter
        )

@@ -563,7 +563,7 @@ def _apply_module_schemas(
    """
    # This is the old way for password_auth_provider modules to make changes
    # to the database. This should instead be done using the module API
-    for (mod, _config) in config.authproviders.password_providers:
+    for mod, _config in config.authproviders.password_providers:
        if not hasattr(mod, "get_db_schema_files"):
            continue
        modname = ".".join((mod.__module__, mod.__name__))
@@ -591,7 +591,7 @@ def _apply_module_schema_files(
        (modname,),
    )
    applied_deltas = {d for d, in cur}
-    for (name, stream) in names_and_streams:
+    for name, stream in names_and_streams:
        if name in applied_deltas:
            continue

@@ -120,7 +120,7 @@ class StateFilter:

    def to_types(self) -> Iterable[Tuple[str, Optional[str]]]:
        """The inverse to `from_types`."""
-        for (event_type, state_keys) in self.types.items():
+        for event_type, state_keys in self.types.items():
            if state_keys is None:
                yield event_type, None
            else:
@@ -98,7 +98,6 @@ class EvictionReason(Enum):

@attr.s(slots=True, auto_attribs=True)
 class CacheMetric:
-
    _cache: Sized
    _cache_type: str
    _cache_name: str
@@ -183,7 +183,7 @@ def check_requirements(extra: Optional[str] = None) -> None:
    deps_unfulfilled = []
    errors = []

-    for (requirement, must_be_installed) in dependencies:
+    for requirement, must_be_installed in dependencies:
        try:
            dist: metadata.Distribution = metadata.distribution(requirement.name)
        except metadata.PackageNotFoundError:
@@ -211,7 +211,6 @@ def _check_yield_points(
                result = Failure()

            if current_context() != expected_context:
-
                # This happens because the context is lost sometime *after* the
                # previous yield and *after* the current yield. E.g. the
                # deferred we waited on didn't follow the rules, or we forgot to
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Andrew Morgan	2ac38a1109	changelog	2023-02-23 23:40:07 +00:00
Andrew Morgan	29cc17fde7	Add a method for Synapse modules to carry out HTTP federation requests Provides a fairly basic interface for Synapse modules to complete HTTP federation requests. Custom error types were used in order to prevent stabilising any of the internal MatrixFederationHttpClient code.	2023-02-23 23:40:07 +00:00
Centzilius	1a1738eca2	Fix typo in federation_verify_certificates in config documentation. (#15139 )	2023-02-23 13:25:56 -05:00
Dirk Klimpel	a068ad7dd4	Add information on uploaded media to user export command. (#15107 )	2023-02-23 13:14:17 -05:00
Fly	452b009eb0	Documentation using Shibboleth with OIDC Plugin for SSO. (#15112 )	2023-02-23 12:54:03 -05:00
Patrick Cloke	adac949a41	Update .git-blame-ignore-revs for #15103 .	2023-02-22 15:30:41 -05:00
dependabot[bot]	9bb2eac719	Bump black from 22.12.0 to 23.1.0 (#15103 )	2023-02-22 15:29:09 -05:00
Patrick Cloke	4ed08ff72e	Tighten the default rate limit of creating new devices. (#15135 )	2023-02-22 14:37:18 -05:00
Dirk Klimpel	6def779a1a	Use `json.dump` in `FileExfiltrationWriter` (#15095 ) To directly write to the open file, instead of writing to an in-memory string first.	2023-02-22 14:29:39 -05:00
Andrew Morgan	91f8de7b56	Clarify the workers that the ThirdPartyRules' `on_new_event` callback will run on (#15071 )	2023-02-22 16:05:34 +00:00
David Robertson	647ff3ef65	Remove unused `room_alias` field from `/createRoom` response (#15093 ) * Change `create_room` return type * Don't return room alias from /createRoom * Update other callsites * Fix up mypy complaints It looks like new_room_user_id is None iff new_room_id is None. It's a shame we haven't expressed this in a way that mypy can understand. * Changelog	2023-02-22 11:07:28 +00:00
				`@@ -0,0 +1 @@`
				Clarify which worker processes the ThirdPartyRules' [`on_new_event`](https://matrix-org.github.io/synapse/v1.78/modules/third_party_rules_callbacks.html#on_new_event) module API callback runs on.
				`@@ -0,0 +1 @@`
				Remove the unspecced `room_alias` field from the [`/createRoom`](https://spec.matrix.org/v1.6/client-server-api/#post_matrixclientv3createroom) response.
				`@@ -0,0 +1 @@`
				Refactor writing json data in `FileExfiltrationWriter`.
				`@@ -0,0 +1 @@`
				`Add media information to the command line [user data export tool](https://matrix-org.github.io/synapse/v1.79/usage/administration/admin_faq.html#how-can-i-export-user-data).`
				`@@ -0,0 +1 @@`
				`Document using [Shibboleth](https://www.shibboleth.net/) as an OpenID Provider.`
				`@@ -0,0 +1 @@`
				Correct reference to `federation_verify_certificates` in configuration documentation.
				`@@ -0,0 +1 @@`
				Add a new `send_federation_http_request` method to the Module API to allow Synapse modules to make matrix federation requests over HTTP.